2023-03-03T07:35:30.480 INFO:root:teuthology version: 0.0.1.dev49+gc7c3cc8 2023-03-03T07:35:30.481 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_vshankar@teuthology --description fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} --name vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi --archive /home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192219 -- /home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192219/orig.config.yaml 2023-03-03T07:35:30.667 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-03-03T07:35:30.741 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192219 branch: wip-vshankar-testing-20230303.023823 description: fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} email: vshankar@redhat.com first_in_suite: false job_id: '7192219' kernel: flavor: default kdb: true sha1: 75534f74ac3dd06f86697ff862e8be2945b3c8e6 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-vshankar-testing-20230303.023823 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20230303.023823 sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b sleep_before_teardown: 0 suite: fs suite_branch: wip-vshankar-testing-20230303.023823 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi098.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaRdTmieDuhW/Jy25O4OW+z0Jazsj4LLoSTSi1R6l8MQAS5EVbl9ec3LToblzoQUHbcAH0gpcvT5pqOjR+nEL6xUcC7sxyanhz83kr8w43S4lduFjrFuKu9sq4nvRHjerv1mJg7efaEU+aBlyIwV5yNcV4+KPrO3MpykxS3a5UMu9LlxA9QdM92+a9m8ALEczNsGCJuAvnpohmMR2D5uX526xV7VR/WEd9lPzbpMtHvPJF2CfBH8iA+fh/Yqy8cPeDXVMC3ct38QadokM5v+P+kXT1FA+vUU9+4CqAAQ/xWBGb9t3su4dySRmFR7CNM/kdGwdLRIhvhQdmO35ZoQzCr5rF9T0mrzKh0aAUC7LhhZnEkGA1V/h17ID8xZNlDN1HDlix8+CYQaoUuyb3h8n3lnVWQuQzXOkmzdndYwjEM30KS7cRmMH5EpQszfFK1L5WKFcsYXPSRT1Oa8tI2+6j+X3oKQx9RqaVAmzJj+0W0MXbVu/jSkkFGwa6gc5mRP8= smithi100.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdlRjeXhUuVMVhbuUuPs9Rcclc5plYkzocTQkQjkpuKAPmjqQwGxv26ijTEOD/wP6UPiTEvDt+ABzrTOF764eu8U+bXxOvENK8SsmB3i+RAXC9ehZ0sSFcJQATAYYDjDk3b5Jm8Ob4sD7tQC6MI6h68QOQL9C5TpKfGB3iRkYOTqBVX2ubohCcHkL8NbEhnNH0ieqHON1Gc8ERgLeUZzrovgwDNlB3O+LLwVBlx/XhY8C4pz0kPTirD803jteDd4iK98x8xIFEPpiUtjlF70BS49jD7/dIuLGdXbUQLV1+xJDQzJux+X3x32DmAJvharMSYF6thaypeManV5tcXir/WUMu2obBDAFaOb8uwR/c89h6stqXSWrm/++wuhurjmrsr829GI3DlZr2qoNjIEga4kbePkkcDpmXQDCt1BADQ2g9vs10ko5620O3L77MZ/8Hup6qdQkhAMn72e8JbMkodMvkFmZcB15Kgv5tFiYYhH07SFNa7ZzWoavr095XEvk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: null - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestMisc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6 timestamp: 2023-03-03_04:39:14 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.31660 2023-03-03T07:35:30.741 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa; will attempt to use it 2023-03-03T07:35:30.742 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa/tasks 2023-03-03T07:35:30.742 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-03-03T07:35:30.744 INFO:teuthology.task.internal:Checking packages... 2023-03-03T07:35:30.774 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '472aaef4ec3b3a2c29f73ca67854785a910f1f9b' 2023-03-03T07:35:30.774 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-03-03T07:35:30.775 INFO:teuthology.packaging:ref: None 2023-03-03T07:35:30.775 INFO:teuthology.packaging:tag: None 2023-03-03T07:35:30.775 INFO:teuthology.packaging:branch: wip-vshankar-testing-20230303.023823 2023-03-03T07:35:30.775 INFO:teuthology.packaging:sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T07:35:30.776 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing-20230303.023823 2023-03-03T07:35:30.912 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-2739.g472aaef4 2023-03-03T07:35:30.913 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-03-03T07:35:30.937 INFO:teuthology.task.internal:no buildpackages task found 2023-03-03T07:35:30.938 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-03-03T07:35:30.948 INFO:teuthology.task.internal:Saving configuration 2023-03-03T07:35:30.967 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-03-03T07:35:30.975 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-03T07:35:31.005 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192219', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-03 07:30:19.157211', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCQX1UwKQf5IWbyJqcF9RcEkPkNmpDIsVNkWdLtBy7tj+LyOsTzSfpzmGmCGpmnkrQQpByw38Y8SOUCvjruM5I0='} 2023-03-03T07:35:31.036 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192219', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-03 07:30:19.158482', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPT6/lYq+uY1nJ6E9BvFcSR3nhaHy7RISdA+Jyf7q1fR0u3SjwgtZb1WjuDLyt+DAdK2k1u5M9womV26CIuuDd8='} 2023-03-03T07:35:31.037 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-03-03T07:35:31.053 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-03-03T07:35:31.054 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-03-03T07:35:31.054 INFO:teuthology.run_tasks:Running task console_log... 2023-03-03T07:35:31.142 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7faef2ccc5e0>, signals=[15]) 2023-03-03T07:35:31.143 INFO:teuthology.run_tasks:Running task internal.connect... 2023-03-03T07:35:31.163 INFO:teuthology.task.internal:Opening connections... 2023-03-03T07:35:31.164 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T07:35:31.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T07:35:31.234 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T07:35:31.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T07:35:31.302 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-03-03T07:35:31.335 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2023-03-03T07:35:31.356 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2023-03-03T07:35:31.357 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2023-03-03T07:35:31.416 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2023-03-03T07:35:31.416 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="8" 2023-03-03T07:35:31.417 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2023-03-03T07:35:31.417 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2023-03-03T07:35:31.418 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="8" 2023-03-03T07:35:31.418 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el8" 2023-03-03T07:35:31.418 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 8" 2023-03-03T07:35:31.418 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2023-03-03T07:35:31.418 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-03-03T07:35:31.419 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2023-03-03T07:35:31.419 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-03T07:35:31.419 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-03T07:35:31.419 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-03-03T07:35:31.420 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2023-03-03T07:35:31.449 DEBUG:teuthology.orchestra.run.smithi100:> uname -m 2023-03-03T07:35:31.470 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2023-03-03T07:35:31.470 DEBUG:teuthology.orchestra.run.smithi100:> cat /etc/os-release 2023-03-03T07:35:31.529 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="CentOS Stream" 2023-03-03T07:35:31.530 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="8" 2023-03-03T07:35:31.530 INFO:teuthology.orchestra.run.smithi100.stdout:ID="centos" 2023-03-03T07:35:31.530 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="rhel fedora" 2023-03-03T07:35:31.530 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="8" 2023-03-03T07:35:31.531 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el8" 2023-03-03T07:35:31.531 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="CentOS Stream 8" 2023-03-03T07:35:31.531 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31" 2023-03-03T07:35:31.531 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-03-03T07:35:31.531 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://centos.org/" 2023-03-03T07:35:31.531 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-03T07:35:31.531 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-03T07:35:31.532 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-03-03T07:35:31.532 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2023-03-03T07:35:31.557 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-03-03T07:35:31.651 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-03-03T07:35:31.701 INFO:teuthology.task.internal:Checking for old test directory... 2023-03-03T07:35:31.702 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2023-03-03T07:35:31.704 DEBUG:teuthology.orchestra.run.smithi100:> test '!' -e /home/ubuntu/cephtest 2023-03-03T07:35:31.724 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-03-03T07:35:31.736 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-03-03T07:35:31.737 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2023-03-03T07:35:31.765 DEBUG:teuthology.orchestra.run.smithi100:> test -z $(ls -A /var/lib/ceph) 2023-03-03T07:35:31.794 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-03-03T07:35:31.869 INFO:teuthology.run_tasks:Running task kernel... 2023-03-03T07:35:31.896 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '75534f74ac3dd06f86697ff862e8be2945b3c8e6'} 2023-03-03T07:35:31.897 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '75534f74ac3dd06f86697ff862e8be2945b3c8e6'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '75534f74ac3dd06f86697ff862e8be2945b3c8e6'}}, timeout 300 2023-03-03T07:35:31.897 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-03T07:35:31.898 DEBUG:teuthology.orchestra.run.smithi100:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-03T07:35:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T07:35:31.918 DEBUG:teuthology.task.kernel:sha1 for mon.a is 75534f74ac3dd06f86697ff862e8be2945b3c8e6 2023-03-03T07:35:31.919 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "75534f74ac3dd06f86697ff862e8be2945b3c8e6"... 2023-03-03T07:35:31.919 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2023-03-03T07:35:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T07:35:31.921 DEBUG:teuthology.task.kernel:sha1 for mon.b is 75534f74ac3dd06f86697ff862e8be2945b3c8e6 2023-03-03T07:35:31.921 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "75534f74ac3dd06f86697ff862e8be2945b3c8e6"... 2023-03-03T07:35:31.921 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2023-03-03T07:35:31.977 INFO:teuthology.orchestra.run.smithi098.stdout:4.18.0-408.el8.x86_64 2023-03-03T07:35:31.978 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-408.el8.x86_64 vs 75534f74ac3dd06f86697ff862e8be2945b3c8e6 2023-03-03T07:35:31.978 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-03-03T07:35:31.979 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=75534f74ac3dd06f86697ff862e8be2945b3c8e6 2023-03-03T07:35:31.982 INFO:teuthology.orchestra.run.smithi100.stdout:4.18.0-408.el8.x86_64 2023-03-03T07:35:31.982 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-408.el8.x86_64 vs 75534f74ac3dd06f86697ff862e8be2945b3c8e6 2023-03-03T07:35:31.982 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-03-03T07:35:31.983 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=75534f74ac3dd06f86697ff862e8be2945b3c8e6 2023-03-03T07:35:32.122 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum upgrade -y linux-firmware 2023-03-03T07:35:32.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum upgrade -y linux-firmware 2023-03-03T07:35:35.868 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-8 - AppStream 30 MB/s | 28 MB 00:00 2023-03-03T07:35:35.898 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-8 - AppStream 33 MB/s | 28 MB 00:00 2023-03-03T07:35:41.573 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-8 - BaseOS 42 MB/s | 27 MB 00:00 2023-03-03T07:35:41.801 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-8 - BaseOS 33 MB/s | 27 MB 00:00 2023-03-03T07:35:45.889 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-8 - Extras 112 kB/s | 18 kB 00:00 2023-03-03T07:35:46.077 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-8 - Extras 149 kB/s | 18 kB 00:00 2023-03-03T07:35:46.283 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-8 - PowerTools 21 MB/s | 5.7 MB 00:00 2023-03-03T07:35:46.494 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-8 - PowerTools 19 MB/s | 5.7 MB 00:00 2023-03-03T07:35:49.666 INFO:teuthology.orchestra.run.smithi098.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-03-03T07:35:50.102 INFO:teuthology.orchestra.run.smithi100.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-03-03T07:35:50.682 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 16 MB/s | 14 MB 00:00 2023-03-03T07:35:51.185 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 15 MB/s | 14 MB 00:00 2023-03-03T07:35:54.049 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 401 kB/s | 24 kB 00:00 2023-03-03T07:35:54.593 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 436 kB/s | 24 kB 00:00 2023-03-03T07:35:57.468 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2023-03-03T07:35:57.469 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2023-03-03T07:35:57.470 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2023-03-03T07:35:57.582 INFO:teuthology.task.kernel:Downloading kernel 75534f74ac3dd06f86697ff862e8be2945b3c8e6 on mon.a... 2023-03-03T07:35:57.583 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=75534f74ac3dd06f86697ff862e8be2945b3c8e6 2023-03-03T07:35:57.733 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/75534f74ac3dd06f86697ff862e8be2945b3c8e6/centos/8/flavors/default/x86_64/ 2023-03-03T07:35:57.734 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.2.0_ceph_g75534f74ac3d-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/75534f74ac3dd06f86697ff862e8be2945b3c8e6/centos/8/flavors/default/x86_64/ --input-file=- 2023-03-03T07:35:57.736 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi098.front.sepia.ceph.com to complete... 2023-03-03T07:35:58.031 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2023-03-03T07:35:58.032 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2023-03-03T07:35:58.032 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2023-03-03T07:35:58.139 INFO:teuthology.task.kernel:Downloading kernel 75534f74ac3dd06f86697ff862e8be2945b3c8e6 on mon.b... 2023-03-03T07:35:58.140 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=75534f74ac3dd06f86697ff862e8be2945b3c8e6 2023-03-03T07:35:58.291 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/75534f74ac3dd06f86697ff862e8be2945b3c8e6/centos/8/flavors/default/x86_64/ 2023-03-03T07:35:58.292 DEBUG:teuthology.orchestra.run.smithi100:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.2.0_ceph_g75534f74ac3d-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/75534f74ac3dd06f86697ff862e8be2945b3c8e6/centos/8/flavors/default/x86_64/ --input-file=- 2023-03-03T07:35:58.295 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi100.front.sepia.ceph.com to complete... 2023-03-03T07:36:16.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03 07:36:16 URL:https://3.chacra.ceph.com/r/kernel/testing/75534f74ac3dd06f86697ff862e8be2945b3c8e6/centos/8/flavors/default/x86_64/kernel-6.2.0_ceph_g75534f74ac3d-1.x86_64.rpm [383026056/383026056] -> "/tmp/kernel.x86_64.rpm" [1] 2023-03-03T07:36:16.756 INFO:teuthology.orchestra.run.smithi098.stderr:FINISHED --2023-03-03 07:36:16-- 2023-03-03T07:36:16.756 INFO:teuthology.orchestra.run.smithi098.stderr:Total wall clock time: 19s 2023-03-03T07:36:16.756 INFO:teuthology.orchestra.run.smithi098.stderr:Downloaded: 1 files, 365M in 19s (19.5 MB/s) 2023-03-03T07:36:16.757 INFO:teuthology.task.kernel:Installing kernel 75534f74ac3dd06f86697ff862e8be2945b3c8e6 on mon.a... 2023-03-03T07:36:16.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-03-03T07:36:18.581 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying... ######################################## 2023-03-03T07:36:19.239 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing... ######################################## 2023-03-03T07:36:19.384 INFO:teuthology.orchestra.run.smithi098.stdout:Updating / installing... 2023-03-03T07:36:23.175 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03 07:36:23 URL:https://3.chacra.ceph.com/r/kernel/testing/75534f74ac3dd06f86697ff862e8be2945b3c8e6/centos/8/flavors/default/x86_64/kernel-6.2.0_ceph_g75534f74ac3d-1.x86_64.rpm [383026056/383026056] -> "/tmp/kernel.x86_64.rpm" [1] 2023-03-03T07:36:23.175 INFO:teuthology.orchestra.run.smithi100.stderr:FINISHED --2023-03-03 07:36:23-- 2023-03-03T07:36:23.175 INFO:teuthology.orchestra.run.smithi100.stderr:Total wall clock time: 25s 2023-03-03T07:36:23.176 INFO:teuthology.orchestra.run.smithi100.stderr:Downloaded: 1 files, 365M in 24s (15.0 MB/s) 2023-03-03T07:36:23.177 INFO:teuthology.task.kernel:Installing kernel 75534f74ac3dd06f86697ff862e8be2945b3c8e6 on mon.b... 2023-03-03T07:36:23.177 DEBUG:teuthology.orchestra.run.smithi100:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-03-03T07:36:24.909 INFO:teuthology.orchestra.run.smithi100.stdout:Verifying... ######################################## 2023-03-03T07:36:25.746 INFO:teuthology.orchestra.run.smithi100.stdout:Preparing... ######################################## 2023-03-03T07:36:25.881 INFO:teuthology.orchestra.run.smithi100.stdout:Updating / installing... 2023-03-03T07:36:58.071 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-6.2.0_ceph_g75534f74ac3d-1 ######################################## 2023-03-03T07:37:04.497 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-6.2.0_ceph_g75534f74ac3d-1 ######################################## 2023-03-03T07:37:22.995 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-03-03T07:37:22.995 DEBUG:teuthology.orchestra.run.smithi098:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-03-03T07:37:23.025 INFO:teuthology.orchestra.run.smithi098.stdout:/boot/System.map-6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:23.026 INFO:teuthology.orchestra.run.smithi098.stdout:/boot/config-6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:23.026 INFO:teuthology.orchestra.run.smithi098.stdout:/boot/vmlinuz-6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:23.026 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:23.026 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel 2023-03-03T07:37:23.026 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch 2023-03-03T07:37:23.027 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86 2023-03-03T07:37:23.027 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto 2023-03-03T07:37:23.027 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-03-03T07:37:23.027 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-03-03T07:37:23.027 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-03-03T07:37:23.027 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-03-03T07:37:23.027 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-03-03T07:37:23.027 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-03-03T07:37:23.028 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-03-03T07:37:23.028 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/crc32c-intel.ko 2023-03-03T07:37:23.028 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-03-03T07:37:23.028 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-03-03T07:37:23.028 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-03-03T07:37:23.028 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-03-03T07:37:23.028 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/serpent-avx2.ko 2023-03-03T07:37:23.029 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-03-03T07:37:23.029 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-03-03T07:37:23.029 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-03-03T07:37:23.029 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-03-03T07:37:23.029 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-03-03T07:37:23.029 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-03-03T07:37:23.030 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-03-03T07:37:23.030 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kernel 2023-03-03T07:37:23.030 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kernel/cpu 2023-03-03T07:37:23.030 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kernel/cpu/mce 2023-03-03T07:37:23.030 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-03-03T07:37:23.030 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kvm 2023-03-03T07:37:23.031 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kvm/kvm-amd.ko 2023-03-03T07:37:23.031 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kvm/kvm-intel.ko 2023-03-03T07:37:23.031 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kvm/kvm.ko 2023-03-03T07:37:23.031 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto 2023-03-03T07:37:23.031 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/ansi_cprng.ko 2023-03-03T07:37:23.031 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/anubis.ko 2023-03-03T07:37:23.031 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/arc4.ko 2023-03-03T07:37:23.032 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx 2023-03-03T07:37:23.032 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/async_memcpy.ko 2023-03-03T07:37:23.032 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/async_pq.ko 2023-03-03T07:37:23.032 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/async_raid6_recov.ko 2023-03-03T07:37:23.032 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/async_tx.ko 2023-03-03T07:37:23.032 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/async_xor.ko 2023-03-03T07:37:23.032 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/raid6test.ko 2023-03-03T07:37:23.032 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/authenc.ko 2023-03-03T07:37:23.033 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/authencesn.ko 2023-03-03T07:37:23.033 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/blake2b_generic.ko 2023-03-03T07:37:23.033 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/blowfish_common.ko 2023-03-03T07:37:23.033 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/blowfish_generic.ko 2023-03-03T07:37:23.033 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/camellia_generic.ko 2023-03-03T07:37:23.033 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/cast5_generic.ko 2023-03-03T07:37:23.033 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/cast6_generic.ko 2023-03-03T07:37:23.034 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/cast_common.ko 2023-03-03T07:37:23.034 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/ccm.ko 2023-03-03T07:37:23.034 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/cmac.ko 2023-03-03T07:37:23.034 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/crc32_generic.ko 2023-03-03T07:37:23.034 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/crypto_engine.ko 2023-03-03T07:37:23.034 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/crypto_user.ko 2023-03-03T07:37:23.034 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/cts.ko 2023-03-03T07:37:23.035 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/des_generic.ko 2023-03-03T07:37:23.035 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/dh_generic.ko 2023-03-03T07:37:23.035 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/echainiv.ko 2023-03-03T07:37:23.035 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/essiv.ko 2023-03-03T07:37:23.035 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/fcrypt.ko 2023-03-03T07:37:23.035 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/khazad.ko 2023-03-03T07:37:23.035 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/lz4.ko 2023-03-03T07:37:23.035 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/lz4hc.ko 2023-03-03T07:37:23.036 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/md4.ko 2023-03-03T07:37:23.036 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/michael_mic.ko 2023-03-03T07:37:23.036 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/pcbc.ko 2023-03-03T07:37:23.036 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/pcrypt.ko 2023-03-03T07:37:23.036 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/rmd160.ko 2023-03-03T07:37:23.036 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/seed.ko 2023-03-03T07:37:23.036 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/serpent_generic.ko 2023-03-03T07:37:23.036 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/tcrypt.ko 2023-03-03T07:37:23.037 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/tea.ko 2023-03-03T07:37:23.037 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/twofish_common.ko 2023-03-03T07:37:23.037 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/twofish_generic.ko 2023-03-03T07:37:23.037 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/vmac.ko 2023-03-03T07:37:23.037 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/wp512.ko 2023-03-03T07:37:23.037 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/xcbc.ko 2023-03-03T07:37:23.037 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/xor.ko 2023-03-03T07:37:23.038 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/xxhash_generic.ko 2023-03-03T07:37:23.038 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers 2023-03-03T07:37:23.038 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi 2023-03-03T07:37:23.038 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/acpi_ipmi.ko 2023-03-03T07:37:23.038 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/acpi_pad.ko 2023-03-03T07:37:23.038 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/custom_method.ko 2023-03-03T07:37:23.038 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/ec_sys.ko 2023-03-03T07:37:23.038 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/platform_profile.ko 2023-03-03T07:37:23.039 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/sbs.ko 2023-03-03T07:37:23.039 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/sbshc.ko 2023-03-03T07:37:23.039 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/video.ko 2023-03-03T07:37:23.039 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata 2023-03-03T07:37:23.039 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/acard-ahci.ko 2023-03-03T07:37:23.039 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/ahci_platform.ko 2023-03-03T07:37:23.039 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/ata_generic.ko 2023-03-03T07:37:23.040 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/libahci_platform.ko 2023-03-03T07:37:23.040 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_acpi.ko 2023-03-03T07:37:23.040 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_ali.ko 2023-03-03T07:37:23.040 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_amd.ko 2023-03-03T07:37:23.040 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_artop.ko 2023-03-03T07:37:23.040 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_atiixp.ko 2023-03-03T07:37:23.040 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_atp867x.ko 2023-03-03T07:37:23.040 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_cmd640.ko 2023-03-03T07:37:23.041 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_cmd64x.ko 2023-03-03T07:37:23.041 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_cypress.ko 2023-03-03T07:37:23.041 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_efar.ko 2023-03-03T07:37:23.041 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_hpt366.ko 2023-03-03T07:37:23.041 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_hpt37x.ko 2023-03-03T07:37:23.041 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_hpt3x2n.ko 2023-03-03T07:37:23.041 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_hpt3x3.ko 2023-03-03T07:37:23.042 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_it8213.ko 2023-03-03T07:37:23.042 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_it821x.ko 2023-03-03T07:37:23.042 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_jmicron.ko 2023-03-03T07:37:23.042 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_marvell.ko 2023-03-03T07:37:23.042 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_mpiix.ko 2023-03-03T07:37:23.042 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_netcell.ko 2023-03-03T07:37:23.042 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_ninja32.ko 2023-03-03T07:37:23.042 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_ns87410.ko 2023-03-03T07:37:23.043 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_ns87415.ko 2023-03-03T07:37:23.043 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_oldpiix.ko 2023-03-03T07:37:23.043 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_opti.ko 2023-03-03T07:37:23.043 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_optidma.ko 2023-03-03T07:37:23.043 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_pcmcia.ko 2023-03-03T07:37:23.043 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_pdc2027x.ko 2023-03-03T07:37:23.043 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-03-03T07:37:23.043 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_piccolo.ko 2023-03-03T07:37:23.044 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_rdc.ko 2023-03-03T07:37:23.060 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_sch.ko 2023-03-03T07:37:23.060 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_serverworks.ko 2023-03-03T07:37:23.060 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_sil680.ko 2023-03-03T07:37:23.061 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_sis.ko 2023-03-03T07:37:23.061 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_sl82c105.ko 2023-03-03T07:37:23.061 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_triflex.ko 2023-03-03T07:37:23.061 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_via.ko 2023-03-03T07:37:23.061 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pdc_adma.ko 2023-03-03T07:37:23.061 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_inic162x.ko 2023-03-03T07:37:23.062 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_mv.ko 2023-03-03T07:37:23.062 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_nv.ko 2023-03-03T07:37:23.062 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_promise.ko 2023-03-03T07:37:23.062 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_qstor.ko 2023-03-03T07:37:23.062 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_sil.ko 2023-03-03T07:37:23.062 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_sil24.ko 2023-03-03T07:37:23.062 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_sis.ko 2023-03-03T07:37:23.062 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_svw.ko 2023-03-03T07:37:23.063 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_sx4.ko 2023-03-03T07:37:23.063 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_uli.ko 2023-03-03T07:37:23.063 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_via.ko 2023-03-03T07:37:23.063 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_vsc.ko 2023-03-03T07:37:23.063 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/auxdisplay 2023-03-03T07:37:23.063 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/auxdisplay/ks0108.ko 2023-03-03T07:37:23.063 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/base 2023-03-03T07:37:23.064 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/base/regmap 2023-03-03T07:37:23.064 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/base/regmap/regmap-i2c.ko 2023-03-03T07:37:23.064 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/bcma 2023-03-03T07:37:23.064 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/bcma/bcma.ko 2023-03-03T07:37:23.064 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block 2023-03-03T07:37:23.064 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/aoe 2023-03-03T07:37:23.064 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/aoe/aoe.ko 2023-03-03T07:37:23.064 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/brd.ko 2023-03-03T07:37:23.065 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/drbd 2023-03-03T07:37:23.065 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/drbd/drbd.ko 2023-03-03T07:37:23.065 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/floppy.ko 2023-03-03T07:37:23.065 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/loop.ko 2023-03-03T07:37:23.065 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/mtip32xx 2023-03-03T07:37:23.065 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-03-03T07:37:23.065 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/nbd.ko 2023-03-03T07:37:23.066 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/null_blk 2023-03-03T07:37:23.066 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/null_blk/null_blk.ko 2023-03-03T07:37:23.066 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/pktcdvd.ko 2023-03-03T07:37:23.066 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/rbd.ko 2023-03-03T07:37:23.066 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/virtio_blk.ko 2023-03-03T07:37:23.066 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/xen-blkback 2023-03-03T07:37:23.066 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-03-03T07:37:23.066 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/xen-blkfront.ko 2023-03-03T07:37:23.067 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/zram 2023-03-03T07:37:23.067 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/zram/zram.ko 2023-03-03T07:37:23.067 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char 2023-03-03T07:37:23.067 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hangcheck-timer.ko 2023-03-03T07:37:23.067 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random 2023-03-03T07:37:23.067 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random/amd-rng.ko 2023-03-03T07:37:23.067 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random/intel-rng.ko 2023-03-03T07:37:23.068 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-03-03T07:37:23.068 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random/via-rng.ko 2023-03-03T07:37:23.068 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random/virtio-rng.ko 2023-03-03T07:37:23.068 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi 2023-03-03T07:37:23.068 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-03-03T07:37:23.068 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-03-03T07:37:23.068 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-03-03T07:37:23.068 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_si.ko 2023-03-03T07:37:23.069 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-03-03T07:37:23.069 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-03-03T07:37:23.069 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/lp.ko 2023-03-03T07:37:23.069 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/mwave 2023-03-03T07:37:23.069 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/mwave/mwave.ko 2023-03-03T07:37:23.069 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/pcmcia 2023-03-03T07:37:23.069 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/pcmcia/cm4000_cs.ko 2023-03-03T07:37:23.070 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/pcmcia/cm4040_cs.ko 2023-03-03T07:37:23.070 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ppdev.ko 2023-03-03T07:37:23.070 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tlclk.ko 2023-03-03T07:37:23.070 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm 2023-03-03T07:37:23.070 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm.ko 2023-03-03T07:37:23.070 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm_atmel.ko 2023-03-03T07:37:23.070 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm_infineon.ko 2023-03-03T07:37:23.070 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm_nsc.ko 2023-03-03T07:37:23.071 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm_tis.ko 2023-03-03T07:37:23.071 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-03-03T07:37:23.071 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/uv_mmtimer.ko 2023-03-03T07:37:23.071 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/virtio_console.ko 2023-03-03T07:37:23.071 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq 2023-03-03T07:37:23.071 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-03-03T07:37:23.071 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-03-03T07:37:23.071 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/p4-clockmod.ko 2023-03-03T07:37:23.072 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-03-03T07:37:23.072 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/powernow-k8.ko 2023-03-03T07:37:23.072 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/speedstep-lib.ko 2023-03-03T07:37:23.072 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto 2023-03-03T07:37:23.072 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/ccp 2023-03-03T07:37:23.072 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-03-03T07:37:23.072 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/ccp/ccp.ko 2023-03-03T07:37:23.073 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/padlock-aes.ko 2023-03-03T07:37:23.073 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/padlock-sha.ko 2023-03-03T07:37:23.073 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/qat 2023-03-03T07:37:23.073 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/qat/qat_common 2023-03-03T07:37:23.073 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/qat/qat_common/intel_qat.ko 2023-03-03T07:37:23.073 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/qat/qat_dh895xcc 2023-03-03T07:37:23.073 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-03-03T07:37:23.073 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/virtio 2023-03-03T07:37:23.074 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-03-03T07:37:23.074 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dca 2023-03-03T07:37:23.074 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dca/dca.ko 2023-03-03T07:37:23.074 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/devfreq 2023-03-03T07:37:23.074 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-03-03T07:37:23.074 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dma 2023-03-03T07:37:23.074 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dma/dw 2023-03-03T07:37:23.074 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dma/dw/dw_dmac.ko 2023-03-03T07:37:23.075 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dma/ioat 2023-03-03T07:37:23.075 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dma/ioat/ioatdma.ko 2023-03-03T07:37:23.075 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac 2023-03-03T07:37:23.075 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/amd64_edac.ko 2023-03-03T07:37:23.075 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/e752x_edac.ko 2023-03-03T07:37:23.075 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/edac_mce_amd.ko 2023-03-03T07:37:23.075 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i3000_edac.ko 2023-03-03T07:37:23.076 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i3200_edac.ko 2023-03-03T07:37:23.076 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i5100_edac.ko 2023-03-03T07:37:23.076 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i5400_edac.ko 2023-03-03T07:37:23.076 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i7300_edac.ko 2023-03-03T07:37:23.076 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i7core_edac.ko 2023-03-03T07:37:23.076 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i82975x_edac.ko 2023-03-03T07:37:23.076 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/ie31200_edac.ko 2023-03-03T07:37:23.077 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/sb_edac.ko 2023-03-03T07:37:23.077 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/x38_edac.ko 2023-03-03T07:37:23.077 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/firmware 2023-03-03T07:37:23.077 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/firmware/edd.ko 2023-03-03T07:37:23.077 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/firmware/iscsi_ibft.ko 2023-03-03T07:37:23.077 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/gpio 2023-03-03T07:37:23.077 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/gpio/gpio-ich.ko 2023-03-03T07:37:23.077 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/gpio/gpio-viperboard.ko 2023-03-03T07:37:23.078 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid 2023-03-03T07:37:23.078 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-appleir.ko 2023-03-03T07:37:23.078 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-aureal.ko 2023-03-03T07:37:23.078 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-axff.ko 2023-03-03T07:37:23.078 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-dr.ko 2023-03-03T07:37:23.078 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-elecom.ko 2023-03-03T07:37:23.078 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-elo.ko 2023-03-03T07:37:23.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-emsff.ko 2023-03-03T07:37:23.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-gaff.ko 2023-03-03T07:37:23.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-gt683r.ko 2023-03-03T07:37:23.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-gyration.ko 2023-03-03T07:37:23.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-holtek-kbd.ko 2023-03-03T07:37:23.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-holtek-mouse.ko 2023-03-03T07:37:23.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-holtekff.ko 2023-03-03T07:37:23.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-hyperv.ko 2023-03-03T07:37:23.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-icade.ko 2023-03-03T07:37:23.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-keytouch.ko 2023-03-03T07:37:23.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-kye.ko 2023-03-03T07:37:23.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-lcpower.ko 2023-03-03T07:37:23.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-led.ko 2023-03-03T07:37:23.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-lenovo.ko 2023-03-03T07:37:23.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-logitech-dj.ko 2023-03-03T07:37:23.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-03-03T07:37:23.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-multitouch.ko 2023-03-03T07:37:23.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-ortek.ko 2023-03-03T07:37:23.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-penmount.ko 2023-03-03T07:37:23.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-petalynx.ko 2023-03-03T07:37:23.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-picolcd.ko 2023-03-03T07:37:23.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-pl.ko 2023-03-03T07:37:23.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-plantronics.ko 2023-03-03T07:37:23.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-primax.ko 2023-03-03T07:37:23.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-rmi.ko 2023-03-03T07:37:23.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-arvo.ko 2023-03-03T07:37:23.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-common.ko 2023-03-03T07:37:23.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-isku.ko 2023-03-03T07:37:23.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-kone.ko 2023-03-03T07:37:23.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-03-03T07:37:23.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-konepure.ko 2023-03-03T07:37:23.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-03-03T07:37:23.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-lua.ko 2023-03-03T07:37:23.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-pyra.ko 2023-03-03T07:37:23.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-ryos.ko 2023-03-03T07:37:23.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-savu.ko 2023-03-03T07:37:23.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat.ko 2023-03-03T07:37:23.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-saitek.ko 2023-03-03T07:37:23.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-samsung.ko 2023-03-03T07:37:23.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-sensor-hub.ko 2023-03-03T07:37:23.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-sjoy.ko 2023-03-03T07:37:23.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-sony.ko 2023-03-03T07:37:23.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-speedlink.ko 2023-03-03T07:37:23.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-steelseries.ko 2023-03-03T07:37:23.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-sunplus.ko 2023-03-03T07:37:23.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-thrustmaster.ko 2023-03-03T07:37:23.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-tivo.ko 2023-03-03T07:37:23.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-tmff.ko 2023-03-03T07:37:23.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-topseed.ko 2023-03-03T07:37:23.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-twinhan.ko 2023-03-03T07:37:23.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-uclogic.ko 2023-03-03T07:37:23.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-waltop.ko 2023-03-03T07:37:23.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-wiimote.ko 2023-03-03T07:37:23.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-xinmo.ko 2023-03-03T07:37:23.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-zpff.ko 2023-03-03T07:37:23.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-zydacron.ko 2023-03-03T07:37:23.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/uhid.ko 2023-03-03T07:37:23.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/wacom.ko 2023-03-03T07:37:23.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hv 2023-03-03T07:37:23.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hv/hv_balloon.ko 2023-03-03T07:37:23.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hv/hv_utils.ko 2023-03-03T07:37:23.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hv/hv_vmbus.ko 2023-03-03T07:37:23.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon 2023-03-03T07:37:23.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/abituguru.ko 2023-03-03T07:37:23.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/abituguru3.ko 2023-03-03T07:37:23.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/acpi_power_meter.ko 2023-03-03T07:37:23.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ad7414.ko 2023-03-03T07:37:23.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ad7418.ko 2023-03-03T07:37:23.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adc128d818.ko 2023-03-03T07:37:23.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adm1025.ko 2023-03-03T07:37:23.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adm1026.ko 2023-03-03T07:37:23.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adm1029.ko 2023-03-03T07:37:23.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adm1031.ko 2023-03-03T07:37:23.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adm9240.ko 2023-03-03T07:37:23.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ads7828.ko 2023-03-03T07:37:23.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7410.ko 2023-03-03T07:37:23.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7411.ko 2023-03-03T07:37:23.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7462.ko 2023-03-03T07:37:23.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7470.ko 2023-03-03T07:37:23.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7475.ko 2023-03-03T07:37:23.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7x10.ko 2023-03-03T07:37:23.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/amc6821.ko 2023-03-03T07:37:23.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/applesmc.ko 2023-03-03T07:37:23.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/asb100.ko 2023-03-03T07:37:23.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/asc7621.ko 2023-03-03T07:37:23.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/asus_atk0110.ko 2023-03-03T07:37:23.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/atxp1.ko 2023-03-03T07:37:23.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/coretemp.ko 2023-03-03T07:37:23.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-03-03T07:37:23.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/dme1737.ko 2023-03-03T07:37:23.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ds1621.ko 2023-03-03T07:37:23.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ds620.ko 2023-03-03T07:37:23.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/emc1403.ko 2023-03-03T07:37:23.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/emc6w201.ko 2023-03-03T07:37:23.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/f71805f.ko 2023-03-03T07:37:23.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/f71882fg.ko 2023-03-03T07:37:23.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/f75375s.ko 2023-03-03T07:37:23.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/fam15h_power.ko 2023-03-03T07:37:23.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/fschmd.ko 2023-03-03T07:37:23.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/g760a.ko 2023-03-03T07:37:23.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/g762.ko 2023-03-03T07:37:23.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/gl518sm.ko 2023-03-03T07:37:23.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/gl520sm.ko 2023-03-03T07:37:23.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/hwmon-vid.ko 2023-03-03T07:37:23.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/i5500_temp.ko 2023-03-03T07:37:23.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/i5k_amb.ko 2023-03-03T07:37:23.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ibmaem.ko 2023-03-03T07:37:23.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ibmpex.ko 2023-03-03T07:37:23.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ina209.ko 2023-03-03T07:37:23.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ina2xx.ko 2023-03-03T07:37:23.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/it87.ko 2023-03-03T07:37:23.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/k10temp.ko 2023-03-03T07:37:23.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/k8temp.ko 2023-03-03T07:37:23.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lineage-pem.ko 2023-03-03T07:37:23.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm63.ko 2023-03-03T07:37:23.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm73.ko 2023-03-03T07:37:23.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm75.ko 2023-03-03T07:37:23.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm77.ko 2023-03-03T07:37:23.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm78.ko 2023-03-03T07:37:23.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm80.ko 2023-03-03T07:37:23.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm83.ko 2023-03-03T07:37:23.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm85.ko 2023-03-03T07:37:23.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm87.ko 2023-03-03T07:37:23.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm90.ko 2023-03-03T07:37:23.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm92.ko 2023-03-03T07:37:23.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm93.ko 2023-03-03T07:37:23.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm95234.ko 2023-03-03T07:37:23.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm95241.ko 2023-03-03T07:37:23.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm95245.ko 2023-03-03T07:37:23.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc2945.ko 2023-03-03T07:37:23.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4151.ko 2023-03-03T07:37:23.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4215.ko 2023-03-03T07:37:23.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4222.ko 2023-03-03T07:37:23.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4245.ko 2023-03-03T07:37:23.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4260.ko 2023-03-03T07:37:23.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4261.ko 2023-03-03T07:37:23.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max16065.ko 2023-03-03T07:37:23.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max1619.ko 2023-03-03T07:37:23.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max1668.ko 2023-03-03T07:37:23.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max197.ko 2023-03-03T07:37:23.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max6639.ko 2023-03-03T07:37:23.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max6650.ko 2023-03-03T07:37:23.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max6697.ko 2023-03-03T07:37:23.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/mcp3021.ko 2023-03-03T07:37:23.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/nct6683.ko 2023-03-03T07:37:23.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/nct6775-core.ko 2023-03-03T07:37:23.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/nct6775.ko 2023-03-03T07:37:23.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/nct7802.ko 2023-03-03T07:37:23.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pc87360.ko 2023-03-03T07:37:23.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pc87427.ko 2023-03-03T07:37:23.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pcf8591.ko 2023-03-03T07:37:23.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus 2023-03-03T07:37:23.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-03-03T07:37:23.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-03-03T07:37:23.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-03-03T07:37:23.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/max16064.ko 2023-03-03T07:37:23.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/max34440.ko 2023-03-03T07:37:23.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/max8688.ko 2023-03-03T07:37:23.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-03-03T07:37:23.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-03-03T07:37:23.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-03-03T07:37:23.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-03-03T07:37:23.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-03-03T07:37:23.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-03-03T07:37:23.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/powr1220.ko 2023-03-03T07:37:23.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sch5627.ko 2023-03-03T07:37:23.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sch5636.ko 2023-03-03T07:37:23.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sch56xx-common.ko 2023-03-03T07:37:23.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sht15.ko 2023-03-03T07:37:23.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sht21.ko 2023-03-03T07:37:23.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/shtc1.ko 2023-03-03T07:37:23.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sis5595.ko 2023-03-03T07:37:23.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/smsc47b397.ko 2023-03-03T07:37:23.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/smsc47m1.ko 2023-03-03T07:37:23.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/smsc47m192.ko 2023-03-03T07:37:23.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/thmc50.ko 2023-03-03T07:37:23.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/tmp102.ko 2023-03-03T07:37:23.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/tmp103.ko 2023-03-03T07:37:23.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/tmp401.ko 2023-03-03T07:37:23.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/tmp421.ko 2023-03-03T07:37:23.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/via-cputemp.ko 2023-03-03T07:37:23.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/via686a.ko 2023-03-03T07:37:23.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/vt1211.ko 2023-03-03T07:37:23.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/vt8231.ko 2023-03-03T07:37:23.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83627ehf.ko 2023-03-03T07:37:23.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83627hf.ko 2023-03-03T07:37:23.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83781d.ko 2023-03-03T07:37:23.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83791d.ko 2023-03-03T07:37:23.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83792d.ko 2023-03-03T07:37:23.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83793.ko 2023-03-03T07:37:23.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83795.ko 2023-03-03T07:37:23.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83l785ts.ko 2023-03-03T07:37:23.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83l786ng.ko 2023-03-03T07:37:23.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c 2023-03-03T07:37:23.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/algos 2023-03-03T07:37:23.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-03-03T07:37:23.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-03-03T07:37:23.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses 2023-03-03T07:37:23.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-03-03T07:37:23.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-03-03T07:37:23.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-03-03T07:37:23.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-03-03T07:37:23.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-03-03T07:37:23.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-03-03T07:37:23.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-03-03T07:37:23.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-03-03T07:37:23.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-i801.ko 2023-03-03T07:37:23.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-isch.ko 2023-03-03T07:37:23.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-03-03T07:37:23.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-03-03T07:37:23.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-03-03T07:37:23.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-parport.ko 2023-03-03T07:37:23.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-03-03T07:37:23.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-03-03T07:37:23.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-03-03T07:37:23.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-03-03T07:37:23.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-03-03T07:37:23.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-03-03T07:37:23.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-via.ko 2023-03-03T07:37:23.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-03-03T07:37:23.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-03-03T07:37:23.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/i2c-dev.ko 2023-03-03T07:37:23.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/i2c-mux.ko 2023-03-03T07:37:23.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-03-03T07:37:23.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/i2c-smbus.ko 2023-03-03T07:37:23.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/i2c-stub.ko 2023-03-03T07:37:23.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband 2023-03-03T07:37:23.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core 2023-03-03T07:37:23.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/ib_cm.ko 2023-03-03T07:37:23.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/ib_core.ko 2023-03-03T07:37:23.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/ib_umad.ko 2023-03-03T07:37:23.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-03-03T07:37:23.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/iw_cm.ko 2023-03-03T07:37:23.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/rdma_cm.ko 2023-03-03T07:37:23.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-03-03T07:37:23.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw 2023-03-03T07:37:23.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/cxgb4 2023-03-03T07:37:23.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-03-03T07:37:23.111 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mlx4 2023-03-03T07:37:23.111 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-03-03T07:37:23.111 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mlx5 2023-03-03T07:37:23.111 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-03-03T07:37:23.111 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mthca 2023-03-03T07:37:23.111 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-03-03T07:37:23.111 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp 2023-03-03T07:37:23.112 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/ipoib 2023-03-03T07:37:23.112 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-03-03T07:37:23.112 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/iser 2023-03-03T07:37:23.112 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-03-03T07:37:23.112 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/isert 2023-03-03T07:37:23.112 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-03-03T07:37:23.112 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/srp 2023-03-03T07:37:23.112 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-03-03T07:37:23.113 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/srpt 2023-03-03T07:37:23.113 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-03-03T07:37:23.113 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input 2023-03-03T07:37:23.113 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport 2023-03-03T07:37:23.113 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-03-03T07:37:23.113 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport/fm801-gp.ko 2023-03-03T07:37:23.113 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport/gameport.ko 2023-03-03T07:37:23.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport/lightning.ko 2023-03-03T07:37:23.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport/ns558.ko 2023-03-03T07:37:23.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/joydev.ko 2023-03-03T07:37:23.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/keyboard 2023-03-03T07:37:23.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/keyboard/gpio_keys.ko 2023-03-03T07:37:23.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse 2023-03-03T07:37:23.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/appletouch.ko 2023-03-03T07:37:23.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/bcm5974.ko 2023-03-03T07:37:23.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/cyapatp.ko 2023-03-03T07:37:23.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/elan_i2c.ko 2023-03-03T07:37:23.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/sermouse.ko 2023-03-03T07:37:23.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-03-03T07:37:23.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/synaptics_usb.ko 2023-03-03T07:37:23.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/vsxxxaa.ko 2023-03-03T07:37:23.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/rmi4 2023-03-03T07:37:23.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/rmi4/rmi_core.ko 2023-03-03T07:37:23.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/serio 2023-03-03T07:37:23.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/serio/altera_ps2.ko 2023-03-03T07:37:23.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/serio/arc_ps2.ko 2023-03-03T07:37:23.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-03-03T07:37:23.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/serio/serio_raw.ko 2023-03-03T07:37:23.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/sparse-keymap.ko 2023-03-03T07:37:23.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/iommu 2023-03-03T07:37:23.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/iommu/amd 2023-03-03T07:37:23.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/iommu/amd/iommu_v2.ko 2023-03-03T07:37:23.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds 2023-03-03T07:37:23.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/leds-blinkm.ko 2023-03-03T07:37:23.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/leds-lm3530.ko 2023-03-03T07:37:23.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/leds-lp3944.ko 2023-03-03T07:37:23.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/leds-ss4200.ko 2023-03-03T07:37:23.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger 2023-03-03T07:37:23.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-03-03T07:37:23.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-03-03T07:37:23.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-03-03T07:37:23.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-03-03T07:37:23.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2023-03-03T07:37:23.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-03-03T07:37:23.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-03-03T07:37:23.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-03-03T07:37:23.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-03-03T07:37:23.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md 2023-03-03T07:37:23.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/bcache 2023-03-03T07:37:23.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/bcache/bcache.ko 2023-03-03T07:37:23.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-bio-prison.ko 2023-03-03T07:37:23.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-cache-smq.ko 2023-03-03T07:37:23.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-cache.ko 2023-03-03T07:37:23.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-crypt.ko 2023-03-03T07:37:23.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-delay.ko 2023-03-03T07:37:23.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-flakey.ko 2023-03-03T07:37:23.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-log-userspace.ko 2023-03-03T07:37:23.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-multipath.ko 2023-03-03T07:37:23.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-queue-length.ko 2023-03-03T07:37:23.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-raid.ko 2023-03-03T07:37:23.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-round-robin.ko 2023-03-03T07:37:23.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-service-time.ko 2023-03-03T07:37:23.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-switch.ko 2023-03-03T07:37:23.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-thin-pool.ko 2023-03-03T07:37:23.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-verity.ko 2023-03-03T07:37:23.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/faulty.ko 2023-03-03T07:37:23.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/linear.ko 2023-03-03T07:37:23.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/multipath.ko 2023-03-03T07:37:23.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/persistent-data 2023-03-03T07:37:23.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-03-03T07:37:23.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/raid0.ko 2023-03-03T07:37:23.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/raid1.ko 2023-03-03T07:37:23.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/raid10.ko 2023-03-03T07:37:23.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/raid456.ko 2023-03-03T07:37:23.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message 2023-03-03T07:37:23.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion 2023-03-03T07:37:23.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptbase.ko 2023-03-03T07:37:23.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptctl.ko 2023-03-03T07:37:23.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptfc.ko 2023-03-03T07:37:23.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptlan.ko 2023-03-03T07:37:23.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptsas.ko 2023-03-03T07:37:23.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptscsih.ko 2023-03-03T07:37:23.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptspi.ko 2023-03-03T07:37:23.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd 2023-03-03T07:37:23.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/lpc_ich.ko 2023-03-03T07:37:23.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/lpc_sch.ko 2023-03-03T07:37:23.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/mfd-core.ko 2023-03-03T07:37:23.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/sm501.ko 2023-03-03T07:37:23.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/viperboard.ko 2023-03-03T07:37:23.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/vx855.ko 2023-03-03T07:37:23.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/wl1273-core.ko 2023-03-03T07:37:23.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc 2023-03-03T07:37:23.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/altera-stapl 2023-03-03T07:37:23.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-03-03T07:37:23.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/apds9802als.ko 2023-03-03T07:37:23.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/apds990x.ko 2023-03-03T07:37:23.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/bh1770glc.ko 2023-03-03T07:37:23.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/cb710 2023-03-03T07:37:23.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/cb710/cb710.ko 2023-03-03T07:37:23.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/eeprom 2023-03-03T07:37:23.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/eeprom/at24.ko 2023-03-03T07:37:23.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/eeprom/eeprom.ko 2023-03-03T07:37:23.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-03-03T07:37:23.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/eeprom/max6875.ko 2023-03-03T07:37:23.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/enclosure.ko 2023-03-03T07:37:23.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/hpilo.ko 2023-03-03T07:37:23.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/isl29003.ko 2023-03-03T07:37:23.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/isl29020.ko 2023-03-03T07:37:23.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/lis3lv02d 2023-03-03T07:37:23.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-03-03T07:37:23.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-03-03T07:37:23.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/mei 2023-03-03T07:37:23.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/mei/mei-me.ko 2023-03-03T07:37:23.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/mei/mei-txe.ko 2023-03-03T07:37:23.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/mei/mei.ko 2023-03-03T07:37:23.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-gru 2023-03-03T07:37:23.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-gru/gru.ko 2023-03-03T07:37:23.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-xp 2023-03-03T07:37:23.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-xp/xp.ko 2023-03-03T07:37:23.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-xp/xpc.ko 2023-03-03T07:37:23.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-03-03T07:37:23.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/tifm_7xx1.ko 2023-03-03T07:37:23.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/tifm_core.ko 2023-03-03T07:37:23.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/tsl2550.ko 2023-03-03T07:37:23.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/vmw_balloon.ko 2023-03-03T07:37:23.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/vmw_vmci 2023-03-03T07:37:23.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-03-03T07:37:23.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net 2023-03-03T07:37:23.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/bonding 2023-03-03T07:37:23.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/bonding/bonding.ko 2023-03-03T07:37:23.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa 2023-03-03T07:37:23.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/b53 2023-03-03T07:37:23.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/b53/b53_common.ko 2023-03-03T07:37:23.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/bcm-sf2.ko 2023-03-03T07:37:23.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/mv88e6060.ko 2023-03-03T07:37:23.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/mv88e6xxx 2023-03-03T07:37:23.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-03-03T07:37:23.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dummy.ko 2023-03-03T07:37:23.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/eql.ko 2023-03-03T07:37:23.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet 2023-03-03T07:37:23.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/3com 2023-03-03T07:37:23.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-03-03T07:37:23.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-03-03T07:37:23.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-03-03T07:37:23.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-03-03T07:37:23.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/8390 2023-03-03T07:37:23.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/8390/8390.ko 2023-03-03T07:37:23.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-03-03T07:37:23.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-03-03T07:37:23.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-03-03T07:37:23.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/adaptec 2023-03-03T07:37:23.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-03-03T07:37:23.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/agere 2023-03-03T07:37:23.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/agere/et131x.ko 2023-03-03T07:37:23.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/alteon 2023-03-03T07:37:23.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-03-03T07:37:23.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/altera 2023-03-03T07:37:23.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-03-03T07:37:23.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/amd 2023-03-03T07:37:23.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-03-03T07:37:23.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-03-03T07:37:23.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-03-03T07:37:23.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros 2023-03-03T07:37:23.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/alx 2023-03-03T07:37:23.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-03-03T07:37:23.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atl1c 2023-03-03T07:37:23.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-03-03T07:37:23.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atl1e 2023-03-03T07:37:23.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-03-03T07:37:23.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atlx 2023-03-03T07:37:23.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-03-03T07:37:23.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-03-03T07:37:23.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom 2023-03-03T07:37:23.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-03-03T07:37:23.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-03-03T07:37:23.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-03-03T07:37:23.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-03-03T07:37:23.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-03-03T07:37:23.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/genet 2023-03-03T07:37:23.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-03-03T07:37:23.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-03-03T07:37:23.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/brocade 2023-03-03T07:37:23.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/brocade/bna 2023-03-03T07:37:23.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-03-03T07:37:23.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio 2023-03-03T07:37:23.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb 2023-03-03T07:37:23.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-03-03T07:37:23.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-03-03T07:37:23.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-03-03T07:37:23.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-03-03T07:37:23.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-03-03T07:37:23.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-03-03T07:37:23.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-03-03T07:37:23.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-03-03T07:37:23.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-03-03T07:37:23.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/cisco 2023-03-03T07:37:23.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/cisco/enic 2023-03-03T07:37:23.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-03-03T07:37:23.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec 2023-03-03T07:37:23.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip 2023-03-03T07:37:23.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-03-03T07:37:23.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-03-03T07:37:23.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-03-03T07:37:23.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-03-03T07:37:23.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-03-03T07:37:23.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-03-03T07:37:23.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dlink 2023-03-03T07:37:23.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-03-03T07:37:23.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-03-03T07:37:23.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dnet.ko 2023-03-03T07:37:23.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/emulex 2023-03-03T07:37:23.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/emulex/benet 2023-03-03T07:37:23.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-03-03T07:37:23.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/ethoc.ko 2023-03-03T07:37:23.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel 2023-03-03T07:37:23.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/e100.ko 2023-03-03T07:37:23.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/e1000 2023-03-03T07:37:23.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-03-03T07:37:23.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/e1000e 2023-03-03T07:37:23.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-03-03T07:37:23.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/fm10k 2023-03-03T07:37:23.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-03-03T07:37:23.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/i40e 2023-03-03T07:37:23.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-03-03T07:37:23.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/iavf 2023-03-03T07:37:23.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-03-03T07:37:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/igb 2023-03-03T07:37:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-03-03T07:37:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/igbvf 2023-03-03T07:37:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-03-03T07:37:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgb 2023-03-03T07:37:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgb/ixgb.ko 2023-03-03T07:37:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgbe 2023-03-03T07:37:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-03-03T07:37:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgbevf 2023-03-03T07:37:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-03-03T07:37:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/jme.ko 2023-03-03T07:37:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/marvell 2023-03-03T07:37:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-03-03T07:37:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/marvell/skge.ko 2023-03-03T07:37:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-03-03T07:37:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox 2023-03-03T07:37:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx4 2023-03-03T07:37:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-03-03T07:37:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-03-03T07:37:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx5 2023-03-03T07:37:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-03-03T07:37:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-03-03T07:37:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/micrel 2023-03-03T07:37:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-03-03T07:37:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/myricom 2023-03-03T07:37:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/myricom/myri10ge 2023-03-03T07:37:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-03-03T07:37:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/natsemi 2023-03-03T07:37:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-03-03T07:37:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-03-03T07:37:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/neterion 2023-03-03T07:37:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-03-03T07:37:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/nvidia 2023-03-03T07:37:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-03-03T07:37:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/packetengines 2023-03-03T07:37:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-03-03T07:37:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-03-03T07:37:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic 2023-03-03T07:37:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic/netxen 2023-03-03T07:37:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-03-03T07:37:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-03-03T07:37:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-03-03T07:37:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-03-03T07:37:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/rdc 2023-03-03T07:37:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-03-03T07:37:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/realtek 2023-03-03T07:37:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-03-03T07:37:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-03-03T07:37:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/realtek/atp.ko 2023-03-03T07:37:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-03-03T07:37:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/rocker 2023-03-03T07:37:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-03-03T07:37:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sfc 2023-03-03T07:37:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-03-03T07:37:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/silan 2023-03-03T07:37:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-03-03T07:37:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sis 2023-03-03T07:37:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sis/sis190.ko 2023-03-03T07:37:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sis/sis900.ko 2023-03-03T07:37:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/smsc 2023-03-03T07:37:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-03-03T07:37:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-03-03T07:37:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-03-03T07:37:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-03-03T07:37:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/stmicro 2023-03-03T07:37:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/stmicro/stmmac 2023-03-03T07:37:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-03-03T07:37:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-03-03T07:37:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sun 2023-03-03T07:37:23.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sun/cassini.ko 2023-03-03T07:37:23.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sun/niu.ko 2023-03-03T07:37:23.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sun/sungem.ko 2023-03-03T07:37:23.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-03-03T07:37:23.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/tehuti 2023-03-03T07:37:23.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-03-03T07:37:23.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/ti 2023-03-03T07:37:23.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/ti/tlan.ko 2023-03-03T07:37:23.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/via 2023-03-03T07:37:23.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-03-03T07:37:23.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-03-03T07:37:23.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/wiznet 2023-03-03T07:37:23.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-03-03T07:37:23.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-03-03T07:37:23.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/xircom 2023-03-03T07:37:23.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-03-03T07:37:23.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/geneve.ko 2023-03-03T07:37:23.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/hyperv 2023-03-03T07:37:23.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-03-03T07:37:23.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ieee802154 2023-03-03T07:37:23.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ieee802154/fakelb.ko 2023-03-03T07:37:23.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ifb.ko 2023-03-03T07:37:23.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ipvlan 2023-03-03T07:37:23.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ipvlan/ipvlan.ko 2023-03-03T07:37:23.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/macvlan.ko 2023-03-03T07:37:23.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/macvtap.ko 2023-03-03T07:37:23.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/mdio 2023-03-03T07:37:23.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/mdio.ko 2023-03-03T07:37:23.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-03-03T07:37:23.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-03-03T07:37:23.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/mii.ko 2023-03-03T07:37:23.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/net_failover.ko 2023-03-03T07:37:23.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/netconsole.ko 2023-03-03T07:37:23.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/nlmon.ko 2023-03-03T07:37:23.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/pcs 2023-03-03T07:37:23.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/pcs/pcs-altera-tse.ko 2023-03-03T07:37:23.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-03-03T07:37:23.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy 2023-03-03T07:37:23.157 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/amd.ko 2023-03-03T07:37:23.157 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-03-03T07:37:23.157 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-03-03T07:37:23.157 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/bcm7xxx.ko 2023-03-03T07:37:23.157 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/bcm87xx.ko 2023-03-03T07:37:23.157 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/broadcom.ko 2023-03-03T07:37:23.157 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/cicada.ko 2023-03-03T07:37:23.158 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/davicom.ko 2023-03-03T07:37:23.158 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/dp83640.ko 2023-03-03T07:37:23.158 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/et1011c.ko 2023-03-03T07:37:23.158 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/icplus.ko 2023-03-03T07:37:23.158 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/lxt.ko 2023-03-03T07:37:23.158 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/marvell.ko 2023-03-03T07:37:23.158 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/micrel.ko 2023-03-03T07:37:23.158 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/national.ko 2023-03-03T07:37:23.159 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/phylink.ko 2023-03-03T07:37:23.159 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/qsemi.ko 2023-03-03T07:37:23.159 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/realtek.ko 2023-03-03T07:37:23.159 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/smsc.ko 2023-03-03T07:37:23.159 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/ste10Xp.ko 2023-03-03T07:37:23.159 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/vitesse.ko 2023-03-03T07:37:23.159 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp 2023-03-03T07:37:23.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/bsd_comp.ko 2023-03-03T07:37:23.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/ppp_async.ko 2023-03-03T07:37:23.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/ppp_deflate.ko 2023-03-03T07:37:23.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/ppp_generic.ko 2023-03-03T07:37:23.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/ppp_mppe.ko 2023-03-03T07:37:23.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/ppp_synctty.ko 2023-03-03T07:37:23.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/pppoe.ko 2023-03-03T07:37:23.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/pppox.ko 2023-03-03T07:37:23.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/pptp.ko 2023-03-03T07:37:23.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/slip 2023-03-03T07:37:23.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/slip/slhc.ko 2023-03-03T07:37:23.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/slip/slip.ko 2023-03-03T07:37:23.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/sungem_phy.ko 2023-03-03T07:37:23.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/tap.ko 2023-03-03T07:37:23.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team 2023-03-03T07:37:23.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team.ko 2023-03-03T07:37:23.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team_mode_activebackup.ko 2023-03-03T07:37:23.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team_mode_broadcast.ko 2023-03-03T07:37:23.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-03-03T07:37:23.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team_mode_random.ko 2023-03-03T07:37:23.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-03-03T07:37:23.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/tun.ko 2023-03-03T07:37:23.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/veth.ko 2023-03-03T07:37:23.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/virtio_net.ko 2023-03-03T07:37:23.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/vmxnet3 2023-03-03T07:37:23.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-03-03T07:37:23.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/vxlan 2023-03-03T07:37:23.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/vxlan/vxlan.ko 2023-03-03T07:37:23.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/xen-netback 2023-03-03T07:37:23.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/xen-netback/xen-netback.ko 2023-03-03T07:37:23.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/xen-netfront.ko 2023-03-03T07:37:23.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ntb 2023-03-03T07:37:23.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ntb/ntb.ko 2023-03-03T07:37:23.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/nvme 2023-03-03T07:37:23.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/nvme/host 2023-03-03T07:37:23.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/nvme/host/nvme-core.ko 2023-03-03T07:37:23.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/nvme/host/nvme.ko 2023-03-03T07:37:23.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/parport 2023-03-03T07:37:23.165 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/parport/parport.ko 2023-03-03T07:37:23.165 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/parport/parport_cs.ko 2023-03-03T07:37:23.165 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/parport/parport_pc.ko 2023-03-03T07:37:23.165 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/parport/parport_serial.ko 2023-03-03T07:37:23.165 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci 2023-03-03T07:37:23.165 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci/hotplug 2023-03-03T07:37:23.165 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-03-03T07:37:23.166 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci/pcie 2023-03-03T07:37:23.166 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci/pcie/aer_inject.ko 2023-03-03T07:37:23.166 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci/xen-pcifront.ko 2023-03-03T07:37:23.166 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pcmcia 2023-03-03T07:37:23.166 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pcmcia/i82092.ko 2023-03-03T07:37:23.166 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pcmcia/pd6729.ko 2023-03-03T07:37:23.166 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pcmcia/yenta_socket.ko 2023-03-03T07:37:23.166 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pinctrl 2023-03-03T07:37:23.167 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pinctrl/intel 2023-03-03T07:37:23.167 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-03-03T07:37:23.167 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform 2023-03-03T07:37:23.167 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86 2023-03-03T07:37:23.167 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/acer-wmi.ko 2023-03-03T07:37:23.167 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/acerhdf.ko 2023-03-03T07:37:23.167 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/apple-gmux.ko 2023-03-03T07:37:23.168 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/asus-laptop.ko 2023-03-03T07:37:23.168 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-03-03T07:37:23.168 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/asus-wmi.ko 2023-03-03T07:37:23.168 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/classmate-laptop.ko 2023-03-03T07:37:23.168 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-03-03T07:37:23.168 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-03-03T07:37:23.168 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-03-03T07:37:23.169 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-03-03T07:37:23.169 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/hdaps.ko 2023-03-03T07:37:23.169 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/intel 2023-03-03T07:37:23.169 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-03-03T07:37:23.169 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/intel_ips.ko 2023-03-03T07:37:23.169 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/msi-wmi.ko 2023-03-03T07:37:23.170 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/mxm-wmi.ko 2023-03-03T07:37:23.170 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-03-03T07:37:23.170 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/samsung-laptop.ko 2023-03-03T07:37:23.170 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/samsung-q10.ko 2023-03-03T07:37:23.170 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/topstar-laptop.ko 2023-03-03T07:37:23.170 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-03-03T07:37:23.171 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/toshiba_haps.ko 2023-03-03T07:37:23.171 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/wmi-bmof.ko 2023-03-03T07:37:23.171 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/wmi.ko 2023-03-03T07:37:23.171 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/powercap 2023-03-03T07:37:23.171 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/powercap/intel_rapl_common.ko 2023-03-03T07:37:23.171 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/powercap/intel_rapl_msr.ko 2023-03-03T07:37:23.171 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps 2023-03-03T07:37:23.172 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps/clients 2023-03-03T07:37:23.172 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps/clients/pps-gpio.ko 2023-03-03T07:37:23.172 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps/clients/pps-ldisc.ko 2023-03-03T07:37:23.172 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps/clients/pps_parport.ko 2023-03-03T07:37:23.172 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps/pps_core.ko 2023-03-03T07:37:23.172 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ptp 2023-03-03T07:37:23.173 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ptp/ptp.ko 2023-03-03T07:37:23.173 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ptp/ptp_kvm.ko 2023-03-03T07:37:23.173 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc 2023-03-03T07:37:23.173 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-bq32k.ko 2023-03-03T07:37:23.173 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-bq4802.ko 2023-03-03T07:37:23.173 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1286.ko 2023-03-03T07:37:23.173 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1307.ko 2023-03-03T07:37:23.174 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1374.ko 2023-03-03T07:37:23.174 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1511.ko 2023-03-03T07:37:23.174 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1553.ko 2023-03-03T07:37:23.174 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1672.ko 2023-03-03T07:37:23.174 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1742.ko 2023-03-03T07:37:23.174 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds2404.ko 2023-03-03T07:37:23.174 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds3232.ko 2023-03-03T07:37:23.174 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-em3027.ko 2023-03-03T07:37:23.175 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-fm3130.ko 2023-03-03T07:37:23.175 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-isl12022.ko 2023-03-03T07:37:23.175 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-isl1208.ko 2023-03-03T07:37:23.175 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-m41t80.ko 2023-03-03T07:37:23.175 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-m48t35.ko 2023-03-03T07:37:23.175 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-m48t59.ko 2023-03-03T07:37:23.175 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-max6900.ko 2023-03-03T07:37:23.175 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-msm6242.ko 2023-03-03T07:37:23.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-pcf2127.ko 2023-03-03T07:37:23.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-pcf85063.ko 2023-03-03T07:37:23.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-pcf8523.ko 2023-03-03T07:37:23.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-pcf8563.ko 2023-03-03T07:37:23.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-pcf8583.ko 2023-03-03T07:37:23.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-rp5c01.ko 2023-03-03T07:37:23.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-rs5c372.ko 2023-03-03T07:37:23.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-03-03T07:37:23.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-rx8025.ko 2023-03-03T07:37:23.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-rx8581.ko 2023-03-03T07:37:23.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-03-03T07:37:23.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-v3020.ko 2023-03-03T07:37:23.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-x1205.ko 2023-03-03T07:37:23.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi 2023-03-03T07:37:23.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/3w-9xxx.ko 2023-03-03T07:37:23.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/3w-sas.ko 2023-03-03T07:37:23.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/3w-xxxx.ko 2023-03-03T07:37:23.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/BusLogic.ko 2023-03-03T07:37:23.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/a100u2w.ko 2023-03-03T07:37:23.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/aacraid 2023-03-03T07:37:23.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/aacraid/aacraid.ko 2023-03-03T07:37:23.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/advansys.ko 2023-03-03T07:37:23.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/aic7xxx 2023-03-03T07:37:23.179 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-03-03T07:37:23.179 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-03-03T07:37:23.179 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/am53c974.ko 2023-03-03T07:37:23.179 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/arcmsr 2023-03-03T07:37:23.179 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-03-03T07:37:23.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/atp870u.ko 2023-03-03T07:37:23.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/be2iscsi 2023-03-03T07:37:23.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-03-03T07:37:23.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bfa 2023-03-03T07:37:23.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bfa/bfa.ko 2023-03-03T07:37:23.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bnx2fc 2023-03-03T07:37:23.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-03-03T07:37:23.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bnx2i 2023-03-03T07:37:23.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-03-03T07:37:23.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/ch.ko 2023-03-03T07:37:23.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/csiostor 2023-03-03T07:37:23.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/csiostor/csiostor.ko 2023-03-03T07:37:23.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi 2023-03-03T07:37:23.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi/cxgb3i 2023-03-03T07:37:23.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-03-03T07:37:23.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi/cxgb4i 2023-03-03T07:37:23.183 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-03-03T07:37:23.183 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-03-03T07:37:23.183 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/dc395x.ko 2023-03-03T07:37:23.183 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/device_handler 2023-03-03T07:37:23.183 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-03-03T07:37:23.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-03-03T07:37:23.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-03-03T07:37:23.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-03-03T07:37:23.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/dmx3191d.ko 2023-03-03T07:37:23.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/esas2r 2023-03-03T07:37:23.185 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/esas2r/esas2r.ko 2023-03-03T07:37:23.185 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/esp_scsi.ko 2023-03-03T07:37:23.185 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/fcoe 2023-03-03T07:37:23.185 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/fcoe/fcoe.ko 2023-03-03T07:37:23.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-03-03T07:37:23.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/fnic 2023-03-03T07:37:23.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/fnic/fnic.ko 2023-03-03T07:37:23.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/hpsa.ko 2023-03-03T07:37:23.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/hptiop.ko 2023-03-03T07:37:23.187 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/hv_storvsc.ko 2023-03-03T07:37:23.187 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/initio.ko 2023-03-03T07:37:23.187 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/ipr.ko 2023-03-03T07:37:23.187 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/ips.ko 2023-03-03T07:37:23.187 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/isci 2023-03-03T07:37:23.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/isci/isci.ko 2023-03-03T07:37:23.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-03-03T07:37:23.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/iscsi_tcp.ko 2023-03-03T07:37:23.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libfc 2023-03-03T07:37:23.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libfc/libfc.ko 2023-03-03T07:37:23.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libiscsi.ko 2023-03-03T07:37:23.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libiscsi_tcp.ko 2023-03-03T07:37:23.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libsas 2023-03-03T07:37:23.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libsas/libsas.ko 2023-03-03T07:37:23.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/lpfc 2023-03-03T07:37:23.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/lpfc/lpfc.ko 2023-03-03T07:37:23.190 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/megaraid 2023-03-03T07:37:23.190 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/megaraid.ko 2023-03-03T07:37:23.190 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-03-03T07:37:23.190 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-03-03T07:37:23.191 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-03-03T07:37:23.191 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/mpt3sas 2023-03-03T07:37:23.191 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-03-03T07:37:23.191 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/mvsas 2023-03-03T07:37:23.191 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/mvsas/mvsas.ko 2023-03-03T07:37:23.191 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/mvumi.ko 2023-03-03T07:37:23.192 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/pm8001 2023-03-03T07:37:23.192 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-03-03T07:37:23.192 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/pmcraid.ko 2023-03-03T07:37:23.192 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla1280.ko 2023-03-03T07:37:23.192 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla2xxx 2023-03-03T07:37:23.192 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-03-03T07:37:23.192 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-03-03T07:37:23.193 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla4xxx 2023-03-03T07:37:23.193 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-03-03T07:37:23.193 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/raid_class.ko 2023-03-03T07:37:23.193 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_debug.ko 2023-03-03T07:37:23.194 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_transport_fc.ko 2023-03-03T07:37:23.194 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-03-03T07:37:23.194 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_transport_sas.ko 2023-03-03T07:37:23.194 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_transport_spi.ko 2023-03-03T07:37:23.194 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_transport_srp.ko 2023-03-03T07:37:23.194 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/ses.ko 2023-03-03T07:37:23.194 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/st.ko 2023-03-03T07:37:23.195 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/stex.ko 2023-03-03T07:37:23.195 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/sym53c8xx_2 2023-03-03T07:37:23.195 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-03-03T07:37:23.195 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/virtio_scsi.ko 2023-03-03T07:37:23.195 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/vmw_pvscsi.ko 2023-03-03T07:37:23.195 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/wd719x.ko 2023-03-03T07:37:23.195 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ssb 2023-03-03T07:37:23.195 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ssb/ssb.ko 2023-03-03T07:37:23.196 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target 2023-03-03T07:37:23.196 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/iscsi 2023-03-03T07:37:23.196 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-03-03T07:37:23.196 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/loopback 2023-03-03T07:37:23.196 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/loopback/tcm_loop.ko 2023-03-03T07:37:23.196 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/target_core_file.ko 2023-03-03T07:37:23.197 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/target_core_iblock.ko 2023-03-03T07:37:23.197 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/target_core_mod.ko 2023-03-03T07:37:23.197 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/target_core_pscsi.ko 2023-03-03T07:37:23.197 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/target_core_user.ko 2023-03-03T07:37:23.197 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/tcm_fc 2023-03-03T07:37:23.197 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-03-03T07:37:23.198 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal 2023-03-03T07:37:23.198 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel 2023-03-03T07:37:23.198 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal 2023-03-03T07:37:23.198 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-03-03T07:37:23.198 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-03-03T07:37:23.198 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-03-03T07:37:23.198 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-03-03T07:37:23.198 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-03-03T07:37:23.199 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-03-03T07:37:23.199 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-03-03T07:37:23.199 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-03-03T07:37:23.199 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-03-03T07:37:23.199 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-03-03T07:37:23.200 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-03-03T07:37:23.200 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-03-03T07:37:23.200 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-03-03T07:37:23.200 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-03-03T07:37:23.200 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-03-03T07:37:23.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty 2023-03-03T07:37:23.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/ipwireless 2023-03-03T07:37:23.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-03-03T07:37:23.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/n_gsm.ko 2023-03-03T07:37:23.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/n_hdlc.ko 2023-03-03T07:37:23.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/nozomi.ko 2023-03-03T07:37:23.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial 2023-03-03T07:37:23.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial/8250 2023-03-03T07:37:23.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-03-03T07:37:23.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial/arc_uart.ko 2023-03-03T07:37:23.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial/jsm 2023-03-03T07:37:23.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial/jsm/jsm.ko 2023-03-03T07:37:23.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/synclink_gt.ko 2023-03-03T07:37:23.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ufs 2023-03-03T07:37:23.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ufs/core 2023-03-03T07:37:23.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ufs/core/ufshcd-core.ko 2023-03-03T07:37:23.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ufs/host 2023-03-03T07:37:23.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-03-03T07:37:23.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio 2023-03-03T07:37:23.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio/uio.ko 2023-03-03T07:37:23.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio/uio_aec.ko 2023-03-03T07:37:23.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio/uio_cif.ko 2023-03-03T07:37:23.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio/uio_pci_generic.ko 2023-03-03T07:37:23.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio/uio_sercos3.ko 2023-03-03T07:37:23.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb 2023-03-03T07:37:23.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/class 2023-03-03T07:37:23.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/class/cdc-acm.ko 2023-03-03T07:37:23.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/class/cdc-wdm.ko 2023-03-03T07:37:23.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/class/usblp.ko 2023-03-03T07:37:23.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/class/usbtmc.ko 2023-03-03T07:37:23.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/host 2023-03-03T07:37:23.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/host/sl811-hcd.ko 2023-03-03T07:37:23.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/host/u132-hcd.ko 2023-03-03T07:37:23.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/image 2023-03-03T07:37:23.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/image/mdc800.ko 2023-03-03T07:37:23.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/image/microtek.ko 2023-03-03T07:37:23.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc 2023-03-03T07:37:23.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/adutux.ko 2023-03-03T07:37:23.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/appledisplay.ko 2023-03-03T07:37:23.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/emi26.ko 2023-03-03T07:37:23.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/emi62.ko 2023-03-03T07:37:23.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/ezusb.ko 2023-03-03T07:37:23.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/ftdi-elan.ko 2023-03-03T07:37:23.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/idmouse.ko 2023-03-03T07:37:23.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/iowarrior.ko 2023-03-03T07:37:23.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/isight_firmware.ko 2023-03-03T07:37:23.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/ldusb.ko 2023-03-03T07:37:23.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/legousbtower.ko 2023-03-03T07:37:23.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/sisusbvga 2023-03-03T07:37:23.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-03-03T07:37:23.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/trancevibrator.ko 2023-03-03T07:37:23.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/usb3503.ko 2023-03-03T07:37:23.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/usblcd.ko 2023-03-03T07:37:23.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/usbsevseg.ko 2023-03-03T07:37:23.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/uss720.ko 2023-03-03T07:37:23.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/yurex.ko 2023-03-03T07:37:23.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/phy 2023-03-03T07:37:23.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/phy/phy-generic.ko 2023-03-03T07:37:23.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial 2023-03-03T07:37:23.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/aircable.ko 2023-03-03T07:37:23.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ark3116.ko 2023-03-03T07:37:23.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/belkin_sa.ko 2023-03-03T07:37:23.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ch341.ko 2023-03-03T07:37:23.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/cp210x.ko 2023-03-03T07:37:23.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/cyberjack.ko 2023-03-03T07:37:23.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/cypress_m8.ko 2023-03-03T07:37:23.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/digi_acceleport.ko 2023-03-03T07:37:23.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/empeg.ko 2023-03-03T07:37:23.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ftdi_sio.ko 2023-03-03T07:37:23.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/garmin_gps.ko 2023-03-03T07:37:23.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/io_edgeport.ko 2023-03-03T07:37:23.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/io_ti.ko 2023-03-03T07:37:23.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ipaq.ko 2023-03-03T07:37:23.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ipw.ko 2023-03-03T07:37:23.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ir-usb.ko 2023-03-03T07:37:23.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-03-03T07:37:23.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/keyspan.ko 2023-03-03T07:37:23.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/keyspan_pda.ko 2023-03-03T07:37:23.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/kl5kusb105.ko 2023-03-03T07:37:23.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/kobil_sct.ko 2023-03-03T07:37:23.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/mct_u232.ko 2023-03-03T07:37:23.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/mos7720.ko 2023-03-03T07:37:23.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/mos7840.ko 2023-03-03T07:37:23.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/navman.ko 2023-03-03T07:37:23.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/omninet.ko 2023-03-03T07:37:23.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/opticon.ko 2023-03-03T07:37:23.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/option.ko 2023-03-03T07:37:23.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/oti6858.ko 2023-03-03T07:37:23.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/pl2303.ko 2023-03-03T07:37:23.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/qcaux.ko 2023-03-03T07:37:23.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/qcserial.ko 2023-03-03T07:37:23.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/quatech2.ko 2023-03-03T07:37:23.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/safe_serial.ko 2023-03-03T07:37:23.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/sierra.ko 2023-03-03T07:37:23.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/spcp8x5.ko 2023-03-03T07:37:23.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ssu100.ko 2023-03-03T07:37:23.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/symbolserial.ko 2023-03-03T07:37:23.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-03-03T07:37:23.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-03-03T07:37:23.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/usb_debug.ko 2023-03-03T07:37:23.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/usb_wwan.ko 2023-03-03T07:37:23.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/visor.ko 2023-03-03T07:37:23.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/whiteheat.ko 2023-03-03T07:37:23.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/xsens_mt.ko 2023-03-03T07:37:23.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage 2023-03-03T07:37:23.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/uas.ko 2023-03-03T07:37:23.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-alauda.ko 2023-03-03T07:37:23.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-cypress.ko 2023-03-03T07:37:23.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-datafab.ko 2023-03-03T07:37:23.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-03-03T07:37:23.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-freecom.ko 2023-03-03T07:37:23.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-isd200.ko 2023-03-03T07:37:23.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-03-03T07:37:23.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-karma.ko 2023-03-03T07:37:23.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-onetouch.ko 2023-03-03T07:37:23.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-realtek.ko 2023-03-03T07:37:23.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-sddr09.ko 2023-03-03T07:37:23.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-sddr55.ko 2023-03-03T07:37:23.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-usbat.ko 2023-03-03T07:37:23.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/usb-storage.ko 2023-03-03T07:37:23.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/usbip 2023-03-03T07:37:23.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/usbip/usbip-core.ko 2023-03-03T07:37:23.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/usbip/usbip-host.ko 2023-03-03T07:37:23.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-03-03T07:37:23.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio 2023-03-03T07:37:23.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio/pci 2023-03-03T07:37:23.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-03-03T07:37:23.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio/pci/vfio-pci.ko 2023-03-03T07:37:23.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio/vfio.ko 2023-03-03T07:37:23.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-03-03T07:37:23.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vhost 2023-03-03T07:37:23.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vhost/vhost.ko 2023-03-03T07:37:23.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vhost/vhost_iotlb.ko 2023-03-03T07:37:23.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vhost/vhost_net.ko 2023-03-03T07:37:23.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vhost/vhost_scsi.ko 2023-03-03T07:37:23.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/video 2023-03-03T07:37:23.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/video/backlight 2023-03-03T07:37:23.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/video/backlight/apple_bl.ko 2023-03-03T07:37:23.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/video/backlight/lcd.ko 2023-03-03T07:37:23.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/video/backlight/platform_lcd.ko 2023-03-03T07:37:23.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio 2023-03-03T07:37:23.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio.ko 2023-03-03T07:37:23.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_balloon.ko 2023-03-03T07:37:23.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_mmio.ko 2023-03-03T07:37:23.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_pci.ko 2023-03-03T07:37:23.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-03-03T07:37:23.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-03-03T07:37:23.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_ring.ko 2023-03-03T07:37:23.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1 2023-03-03T07:37:23.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/masters 2023-03-03T07:37:23.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/masters/ds1wm.ko 2023-03-03T07:37:23.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/masters/ds2482.ko 2023-03-03T07:37:23.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/masters/ds2490.ko 2023-03-03T07:37:23.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves 2023-03-03T07:37:23.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-03-03T07:37:23.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-03-03T07:37:23.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-03-03T07:37:23.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-03-03T07:37:23.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-03-03T07:37:23.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-03-03T07:37:23.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-03-03T07:37:23.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-03-03T07:37:23.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-03-03T07:37:23.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_smem.ko 2023-03-03T07:37:23.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_therm.ko 2023-03-03T07:37:23.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/wire.ko 2023-03-03T07:37:23.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog 2023-03-03T07:37:23.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/alim1535_wdt.ko 2023-03-03T07:37:23.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/alim7101_wdt.ko 2023-03-03T07:37:23.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/f71808e_wdt.ko 2023-03-03T07:37:23.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/hpwdt.ko 2023-03-03T07:37:23.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/i6300esb.ko 2023-03-03T07:37:23.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-03-03T07:37:23.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/iTCO_wdt.ko 2023-03-03T07:37:23.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/ib700wdt.ko 2023-03-03T07:37:23.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/ibmasr.ko 2023-03-03T07:37:23.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-03-03T07:37:23.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/it8712f_wdt.ko 2023-03-03T07:37:23.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/it87_wdt.ko 2023-03-03T07:37:23.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/machzwd.ko 2023-03-03T07:37:23.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/nv_tco.ko 2023-03-03T07:37:23.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/pcwd_pci.ko 2023-03-03T07:37:23.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/pcwd_usb.ko 2023-03-03T07:37:23.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-03-03T07:37:23.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/sch311x_wdt.ko 2023-03-03T07:37:23.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/softdog.ko 2023-03-03T07:37:23.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/sp5100_tco.ko 2023-03-03T07:37:23.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/via_wdt.ko 2023-03-03T07:37:23.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-03-03T07:37:23.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/w83877f_wdt.ko 2023-03-03T07:37:23.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/w83977f_wdt.ko 2023-03-03T07:37:23.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/wdt_pci.ko 2023-03-03T07:37:23.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/xen_wdt.ko 2023-03-03T07:37:23.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen 2023-03-03T07:37:23.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-acpi-processor.ko 2023-03-03T07:37:23.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-evtchn.ko 2023-03-03T07:37:23.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-gntalloc.ko 2023-03-03T07:37:23.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-gntdev.ko 2023-03-03T07:37:23.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-pciback 2023-03-03T07:37:23.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-03-03T07:37:23.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-privcmd.ko 2023-03-03T07:37:23.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xenfs 2023-03-03T07:37:23.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xenfs/xenfs.ko 2023-03-03T07:37:23.229 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs 2023-03-03T07:37:23.229 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/9p 2023-03-03T07:37:23.229 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/9p/9p.ko 2023-03-03T07:37:23.229 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/affs 2023-03-03T07:37:23.229 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/affs/affs.ko 2023-03-03T07:37:23.229 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/befs 2023-03-03T07:37:23.229 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/befs/befs.ko 2023-03-03T07:37:23.229 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/binfmt_misc.ko 2023-03-03T07:37:23.230 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/btrfs 2023-03-03T07:37:23.230 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/btrfs/btrfs.ko 2023-03-03T07:37:23.230 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cachefiles 2023-03-03T07:37:23.230 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cachefiles/cachefiles.ko 2023-03-03T07:37:23.230 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ceph 2023-03-03T07:37:23.230 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ceph/ceph.ko 2023-03-03T07:37:23.230 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cifs 2023-03-03T07:37:23.230 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cifs/cifs.ko 2023-03-03T07:37:23.231 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/coda 2023-03-03T07:37:23.231 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/coda/coda.ko 2023-03-03T07:37:23.231 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cramfs 2023-03-03T07:37:23.231 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cramfs/cramfs.ko 2023-03-03T07:37:23.231 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/dlm 2023-03-03T07:37:23.231 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/dlm/dlm.ko 2023-03-03T07:37:23.231 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ecryptfs 2023-03-03T07:37:23.231 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ecryptfs/ecryptfs.ko 2023-03-03T07:37:23.232 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/f2fs 2023-03-03T07:37:23.232 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/f2fs/f2fs.ko 2023-03-03T07:37:23.232 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fat 2023-03-03T07:37:23.232 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fat/fat.ko 2023-03-03T07:37:23.232 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fat/msdos.ko 2023-03-03T07:37:23.232 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fat/vfat.ko 2023-03-03T07:37:23.232 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fscache 2023-03-03T07:37:23.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fscache/fscache.ko 2023-03-03T07:37:23.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fuse 2023-03-03T07:37:23.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fuse/cuse.ko 2023-03-03T07:37:23.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fuse/fuse.ko 2023-03-03T07:37:23.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/gfs2 2023-03-03T07:37:23.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/gfs2/gfs2.ko 2023-03-03T07:37:23.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/hfs 2023-03-03T07:37:23.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/hfs/hfs.ko 2023-03-03T07:37:23.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/hfsplus 2023-03-03T07:37:23.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/hfsplus/hfsplus.ko 2023-03-03T07:37:23.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/isofs 2023-03-03T07:37:23.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/isofs/isofs.ko 2023-03-03T07:37:23.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/jfs 2023-03-03T07:37:23.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/jfs/jfs.ko 2023-03-03T07:37:23.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/lockd 2023-03-03T07:37:23.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/lockd/lockd.ko 2023-03-03T07:37:23.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/minix 2023-03-03T07:37:23.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/minix/minix.ko 2023-03-03T07:37:23.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/netfs 2023-03-03T07:37:23.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/netfs/netfs.ko 2023-03-03T07:37:23.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs 2023-03-03T07:37:23.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/blocklayout 2023-03-03T07:37:23.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-03-03T07:37:23.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/filelayout 2023-03-03T07:37:23.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-03-03T07:37:23.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/flexfilelayout 2023-03-03T07:37:23.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-03-03T07:37:23.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/nfs.ko 2023-03-03T07:37:23.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/nfsv3.ko 2023-03-03T07:37:23.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/nfsv4.ko 2023-03-03T07:37:23.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs_common 2023-03-03T07:37:23.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs_common/grace.ko 2023-03-03T07:37:23.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs_common/nfs_acl.ko 2023-03-03T07:37:23.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfsd 2023-03-03T07:37:23.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfsd/nfsd.ko 2023-03-03T07:37:23.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nilfs2 2023-03-03T07:37:23.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nilfs2/nilfs2.ko 2023-03-03T07:37:23.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls 2023-03-03T07:37:23.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-celtic.ko 2023-03-03T07:37:23.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-centeuro.ko 2023-03-03T07:37:23.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-croatian.ko 2023-03-03T07:37:23.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-cyrillic.ko 2023-03-03T07:37:23.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-gaelic.ko 2023-03-03T07:37:23.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-greek.ko 2023-03-03T07:37:23.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-iceland.ko 2023-03-03T07:37:23.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-inuit.ko 2023-03-03T07:37:23.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-roman.ko 2023-03-03T07:37:23.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-romanian.ko 2023-03-03T07:37:23.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-turkish.ko 2023-03-03T07:37:23.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp1250.ko 2023-03-03T07:37:23.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp1251.ko 2023-03-03T07:37:23.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp1255.ko 2023-03-03T07:37:23.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp737.ko 2023-03-03T07:37:23.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp775.ko 2023-03-03T07:37:23.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp850.ko 2023-03-03T07:37:23.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp852.ko 2023-03-03T07:37:23.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp855.ko 2023-03-03T07:37:23.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp857.ko 2023-03-03T07:37:23.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp860.ko 2023-03-03T07:37:23.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp861.ko 2023-03-03T07:37:23.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp862.ko 2023-03-03T07:37:23.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp863.ko 2023-03-03T07:37:23.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp864.ko 2023-03-03T07:37:23.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp865.ko 2023-03-03T07:37:23.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp866.ko 2023-03-03T07:37:23.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp869.ko 2023-03-03T07:37:23.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp874.ko 2023-03-03T07:37:23.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp932.ko 2023-03-03T07:37:23.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp936.ko 2023-03-03T07:37:23.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp949.ko 2023-03-03T07:37:23.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp950.ko 2023-03-03T07:37:23.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_euc-jp.ko 2023-03-03T07:37:23.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-1.ko 2023-03-03T07:37:23.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-13.ko 2023-03-03T07:37:23.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-14.ko 2023-03-03T07:37:23.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-15.ko 2023-03-03T07:37:23.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-2.ko 2023-03-03T07:37:23.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-3.ko 2023-03-03T07:37:23.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-4.ko 2023-03-03T07:37:23.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-5.ko 2023-03-03T07:37:23.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-6.ko 2023-03-03T07:37:23.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-7.ko 2023-03-03T07:37:23.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-9.ko 2023-03-03T07:37:23.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_koi8-r.ko 2023-03-03T07:37:23.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_koi8-ru.ko 2023-03-03T07:37:23.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_koi8-u.ko 2023-03-03T07:37:23.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_utf8.ko 2023-03-03T07:37:23.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2 2023-03-03T07:37:23.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/cluster 2023-03-03T07:37:23.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-03-03T07:37:23.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/dlm 2023-03-03T07:37:23.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-03-03T07:37:23.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/dlmfs 2023-03-03T07:37:23.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-03-03T07:37:23.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/ocfs2.ko 2023-03-03T07:37:23.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-03-03T07:37:23.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-03-03T07:37:23.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-03-03T07:37:23.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/overlayfs 2023-03-03T07:37:23.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/overlayfs/overlay.ko 2023-03-03T07:37:23.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/pstore 2023-03-03T07:37:23.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/pstore/ramoops.ko 2023-03-03T07:37:23.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/reiserfs 2023-03-03T07:37:23.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/reiserfs/reiserfs.ko 2023-03-03T07:37:23.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/romfs 2023-03-03T07:37:23.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/romfs/romfs.ko 2023-03-03T07:37:23.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/smbfs_common 2023-03-03T07:37:23.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/smbfs_common/cifs_arc4.ko 2023-03-03T07:37:23.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/smbfs_common/cifs_md4.ko 2023-03-03T07:37:23.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/squashfs 2023-03-03T07:37:23.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/squashfs/squashfs.ko 2023-03-03T07:37:23.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/sysv 2023-03-03T07:37:23.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/sysv/sysv.ko 2023-03-03T07:37:23.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/udf 2023-03-03T07:37:23.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/udf/udf.ko 2023-03-03T07:37:23.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ufs 2023-03-03T07:37:23.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ufs/ufs.ko 2023-03-03T07:37:23.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/xfs 2023-03-03T07:37:23.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/xfs/xfs.ko 2023-03-03T07:37:23.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel 2023-03-03T07:37:23.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel/rcu 2023-03-03T07:37:23.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel/rcu/rcutorture.ko 2023-03-03T07:37:23.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel/torture.ko 2023-03-03T07:37:23.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel/trace 2023-03-03T07:37:23.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-03-03T07:37:23.249 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib 2023-03-03T07:37:23.249 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/asn1_encoder.ko 2023-03-03T07:37:23.249 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/crc-itu-t.ko 2023-03-03T07:37:23.249 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/crc8.ko 2023-03-03T07:37:23.249 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/crypto 2023-03-03T07:37:23.249 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/crypto/libarc4.ko 2023-03-03T07:37:23.249 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/crypto/libdes.ko 2023-03-03T07:37:23.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/libcrc32c.ko 2023-03-03T07:37:23.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/lru_cache.ko 2023-03-03T07:37:23.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/lz4 2023-03-03T07:37:23.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/lz4/lz4_compress.ko 2023-03-03T07:37:23.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/lz4/lz4hc_compress.ko 2023-03-03T07:37:23.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/math 2023-03-03T07:37:23.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/math/cordic.ko 2023-03-03T07:37:23.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/raid6 2023-03-03T07:37:23.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/raid6/raid6_pq.ko 2023-03-03T07:37:23.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/reed_solomon 2023-03-03T07:37:23.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/reed_solomon/reed_solomon.ko 2023-03-03T07:37:23.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/ts_bm.ko 2023-03-03T07:37:23.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/ts_fsm.ko 2023-03-03T07:37:23.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/ts_kmp.ko 2023-03-03T07:37:23.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/zstd 2023-03-03T07:37:23.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/zstd/zstd_compress.ko 2023-03-03T07:37:23.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/mm 2023-03-03T07:37:23.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/mm/hwpoison-inject.ko 2023-03-03T07:37:23.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net 2023-03-03T07:37:23.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan 2023-03-03T07:37:23.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/6lowpan.ko 2023-03-03T07:37:23.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_dest.ko 2023-03-03T07:37:23.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_fragment.ko 2023-03-03T07:37:23.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_hop.ko 2023-03-03T07:37:23.253 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_ipv6.ko 2023-03-03T07:37:23.253 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_mobility.ko 2023-03-03T07:37:23.253 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_routing.ko 2023-03-03T07:37:23.253 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_udp.ko 2023-03-03T07:37:23.253 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802 2023-03-03T07:37:23.253 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802/garp.ko 2023-03-03T07:37:23.253 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802/mrp.ko 2023-03-03T07:37:23.253 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802/p8022.ko 2023-03-03T07:37:23.254 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802/psnap.ko 2023-03-03T07:37:23.254 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802/stp.ko 2023-03-03T07:37:23.254 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/8021q 2023-03-03T07:37:23.254 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/8021q/8021q.ko 2023-03-03T07:37:23.254 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/9p 2023-03-03T07:37:23.254 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/9p/9pnet.ko 2023-03-03T07:37:23.254 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/9p/9pnet_fd.ko 2023-03-03T07:37:23.254 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/9p/9pnet_rdma.ko 2023-03-03T07:37:23.255 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/9p/9pnet_virtio.ko 2023-03-03T07:37:23.255 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/batman-adv 2023-03-03T07:37:23.255 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/batman-adv/batman-adv.ko 2023-03-03T07:37:23.255 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge 2023-03-03T07:37:23.255 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/br_netfilter.ko 2023-03-03T07:37:23.255 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/bridge.ko 2023-03-03T07:37:23.255 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter 2023-03-03T07:37:23.256 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-03-03T07:37:23.256 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_among.ko 2023-03-03T07:37:23.256 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_arp.ko 2023-03-03T07:37:23.256 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-03-03T07:37:23.256 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-03-03T07:37:23.256 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_ip.ko 2023-03-03T07:37:23.256 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-03-03T07:37:23.256 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_limit.ko 2023-03-03T07:37:23.257 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_log.ko 2023-03-03T07:37:23.257 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_mark.ko 2023-03-03T07:37:23.257 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-03-03T07:37:23.257 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-03-03T07:37:23.257 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-03-03T07:37:23.257 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-03-03T07:37:23.257 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_snat.ko 2023-03-03T07:37:23.257 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_stp.ko 2023-03-03T07:37:23.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-03-03T07:37:23.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-03-03T07:37:23.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-03-03T07:37:23.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-03-03T07:37:23.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebtables.ko 2023-03-03T07:37:23.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-03-03T07:37:23.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-03-03T07:37:23.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-03-03T07:37:23.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ceph 2023-03-03T07:37:23.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ceph/libceph.ko 2023-03-03T07:37:23.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/core 2023-03-03T07:37:23.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/core/failover.ko 2023-03-03T07:37:23.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/core/pktgen.ko 2023-03-03T07:37:23.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dccp 2023-03-03T07:37:23.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dccp/dccp.ko 2023-03-03T07:37:23.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dccp/dccp_diag.ko 2023-03-03T07:37:23.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dccp/dccp_ipv4.ko 2023-03-03T07:37:23.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dccp/dccp_ipv6.ko 2023-03-03T07:37:23.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dns_resolver 2023-03-03T07:37:23.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dns_resolver/dns_resolver.ko 2023-03-03T07:37:23.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa 2023-03-03T07:37:23.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa/dsa_core.ko 2023-03-03T07:37:23.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa/tag_brcm.ko 2023-03-03T07:37:23.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa/tag_dsa.ko 2023-03-03T07:37:23.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa/tag_none.ko 2023-03-03T07:37:23.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa/tag_trailer.ko 2023-03-03T07:37:23.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ieee802154 2023-03-03T07:37:23.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ieee802154/6lowpan 2023-03-03T07:37:23.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-03-03T07:37:23.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ieee802154/ieee802154.ko 2023-03-03T07:37:23.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ieee802154/ieee802154_socket.ko 2023-03-03T07:37:23.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4 2023-03-03T07:37:23.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ah4.ko 2023-03-03T07:37:23.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/esp4.ko 2023-03-03T07:37:23.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/fou.ko 2023-03-03T07:37:23.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/gre.ko 2023-03-03T07:37:23.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/inet_diag.ko 2023-03-03T07:37:23.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ip_gre.ko 2023-03-03T07:37:23.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ip_tunnel.ko 2023-03-03T07:37:23.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ip_vti.ko 2023-03-03T07:37:23.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ipcomp.ko 2023-03-03T07:37:23.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ipip.ko 2023-03-03T07:37:23.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter 2023-03-03T07:37:23.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/arp_tables.ko 2023-03-03T07:37:23.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-03-03T07:37:23.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-03-03T07:37:23.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/ipt_CLUSTERIP.ko 2023-03-03T07:37:23.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-03-03T07:37:23.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-03-03T07:37:23.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-03-03T07:37:23.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-03-03T07:37:23.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-03-03T07:37:23.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-03-03T07:37:23.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-03-03T07:37:23.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/iptable_security.ko 2023-03-03T07:37:23.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-03-03T07:37:23.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-03-03T07:37:23.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-03-03T07:37:23.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-03-03T07:37:23.266 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-03-03T07:37:23.266 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-03-03T07:37:23.266 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-03-03T07:37:23.266 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-03-03T07:37:23.266 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-03-03T07:37:23.266 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-03-03T07:37:23.266 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_bic.ko 2023-03-03T07:37:23.266 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_dctcp.ko 2023-03-03T07:37:23.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_diag.ko 2023-03-03T07:37:23.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_highspeed.ko 2023-03-03T07:37:23.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_htcp.ko 2023-03-03T07:37:23.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_hybla.ko 2023-03-03T07:37:23.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_illinois.ko 2023-03-03T07:37:23.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_lp.ko 2023-03-03T07:37:23.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_scalable.ko 2023-03-03T07:37:23.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_vegas.ko 2023-03-03T07:37:23.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_veno.ko 2023-03-03T07:37:23.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_westwood.ko 2023-03-03T07:37:23.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_yeah.ko 2023-03-03T07:37:23.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tunnel4.ko 2023-03-03T07:37:23.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/udp_diag.ko 2023-03-03T07:37:23.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/udp_tunnel.ko 2023-03-03T07:37:23.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/xfrm4_tunnel.ko 2023-03-03T07:37:23.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6 2023-03-03T07:37:23.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/ah6.ko 2023-03-03T07:37:23.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/esp6.ko 2023-03-03T07:37:23.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/fou6.ko 2023-03-03T07:37:23.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/ip6_tunnel.ko 2023-03-03T07:37:23.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-03-03T07:37:23.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/ip6_vti.ko 2023-03-03T07:37:23.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/ipcomp6.ko 2023-03-03T07:37:23.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter 2023-03-03T07:37:23.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-03-03T07:37:23.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-03-03T07:37:23.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-03-03T07:37:23.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-03-03T07:37:23.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-03-03T07:37:23.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-03-03T07:37:23.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-03-03T07:37:23.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-03-03T07:37:23.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-03-03T07:37:23.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-03-03T07:37:23.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-03-03T07:37:23.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-03-03T07:37:23.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-03-03T07:37:23.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-03-03T07:37:23.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-03-03T07:37:23.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-03-03T07:37:23.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-03-03T07:37:23.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-03-03T07:37:23.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-03-03T07:37:23.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-03-03T07:37:23.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-03-03T07:37:23.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-03-03T07:37:23.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-03-03T07:37:23.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-03-03T07:37:23.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/sit.ko 2023-03-03T07:37:23.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/tunnel6.ko 2023-03-03T07:37:23.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/xfrm6_tunnel.ko 2023-03-03T07:37:23.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/key 2023-03-03T07:37:23.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/key/af_key.ko 2023-03-03T07:37:23.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp 2023-03-03T07:37:23.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_core.ko 2023-03-03T07:37:23.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_debugfs.ko 2023-03-03T07:37:23.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_eth.ko 2023-03-03T07:37:23.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_ip.ko 2023-03-03T07:37:23.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_ip6.ko 2023-03-03T07:37:23.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_netlink.ko 2023-03-03T07:37:23.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_ppp.ko 2023-03-03T07:37:23.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/llc 2023-03-03T07:37:23.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/llc/llc.ko 2023-03-03T07:37:23.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/mac802154 2023-03-03T07:37:23.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/mac802154/mac802154.ko 2023-03-03T07:37:23.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/mpls 2023-03-03T07:37:23.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/mpls/mpls_gso.ko 2023-03-03T07:37:23.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter 2023-03-03T07:37:23.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset 2023-03-03T07:37:23.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set.ko 2023-03-03T07:37:23.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-03-03T07:37:23.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-03-03T07:37:23.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-03-03T07:37:23.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-03-03T07:37:23.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-03-03T07:37:23.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-03-03T07:37:23.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-03-03T07:37:23.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-03-03T07:37:23.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-03-03T07:37:23.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-03-03T07:37:23.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-03-03T07:37:23.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-03-03T07:37:23.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-03-03T07:37:23.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-03-03T07:37:23.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-03-03T07:37:23.278 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs 2023-03-03T07:37:23.278 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs.ko 2023-03-03T07:37:23.278 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-03-03T07:37:23.278 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-03-03T07:37:23.278 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-03-03T07:37:23.278 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-03-03T07:37:23.278 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-03-03T07:37:23.278 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-03-03T07:37:23.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-03-03T07:37:23.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-03-03T07:37:23.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-03-03T07:37:23.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-03-03T07:37:23.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-03-03T07:37:23.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-03-03T07:37:23.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-03-03T07:37:23.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conncount.ko 2023-03-03T07:37:23.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack.ko 2023-03-03T07:37:23.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-03-03T07:37:23.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-03-03T07:37:23.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-03-03T07:37:23.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_h323.ko 2023-03-03T07:37:23.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_irc.ko 2023-03-03T07:37:23.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-03-03T07:37:23.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-03-03T07:37:23.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-03-03T07:37:23.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_sane.ko 2023-03-03T07:37:23.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_sip.ko 2023-03-03T07:37:23.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-03-03T07:37:23.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-03-03T07:37:23.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_dup_netdev.ko 2023-03-03T07:37:23.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_flow_table.ko 2023-03-03T07:37:23.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_flow_table_inet.ko 2023-03-03T07:37:23.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_log_syslog.ko 2023-03-03T07:37:23.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat.ko 2023-03-03T07:37:23.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat_amanda.ko 2023-03-03T07:37:23.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat_ftp.ko 2023-03-03T07:37:23.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat_irc.ko 2023-03-03T07:37:23.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat_sip.ko 2023-03-03T07:37:23.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat_tftp.ko 2023-03-03T07:37:23.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_synproxy_core.ko 2023-03-03T07:37:23.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_tables.ko 2023-03-03T07:37:23.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nfnetlink.ko 2023-03-03T07:37:23.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nfnetlink_acct.ko 2023-03-03T07:37:23.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nfnetlink_log.ko 2023-03-03T07:37:23.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nfnetlink_osf.ko 2023-03-03T07:37:23.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nfnetlink_queue.ko 2023-03-03T07:37:23.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_chain_nat.ko 2023-03-03T07:37:23.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_compat.ko 2023-03-03T07:37:23.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_ct.ko 2023-03-03T07:37:23.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_dup_netdev.ko 2023-03-03T07:37:23.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_fib.ko 2023-03-03T07:37:23.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_fwd_netdev.ko 2023-03-03T07:37:23.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_hash.ko 2023-03-03T07:37:23.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_limit.ko 2023-03-03T07:37:23.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_log.ko 2023-03-03T07:37:23.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_masq.ko 2023-03-03T07:37:23.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_nat.ko 2023-03-03T07:37:23.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_numgen.ko 2023-03-03T07:37:23.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_queue.ko 2023-03-03T07:37:23.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_quota.ko 2023-03-03T07:37:23.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_redir.ko 2023-03-03T07:37:23.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_reject.ko 2023-03-03T07:37:23.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_reject_inet.ko 2023-03-03T07:37:23.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_socket.ko 2023-03-03T07:37:23.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_tunnel.ko 2023-03-03T07:37:23.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_AUDIT.ko 2023-03-03T07:37:23.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_CHECKSUM.ko 2023-03-03T07:37:23.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_CLASSIFY.ko 2023-03-03T07:37:23.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-03-03T07:37:23.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_CT.ko 2023-03-03T07:37:23.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_DSCP.ko 2023-03-03T07:37:23.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_HL.ko 2023-03-03T07:37:23.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_HMARK.ko 2023-03-03T07:37:23.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_IDLETIMER.ko 2023-03-03T07:37:23.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_LED.ko 2023-03-03T07:37:23.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_LOG.ko 2023-03-03T07:37:23.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_MASQUERADE.ko 2023-03-03T07:37:23.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_NETMAP.ko 2023-03-03T07:37:23.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_NFLOG.ko 2023-03-03T07:37:23.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_NFQUEUE.ko 2023-03-03T07:37:23.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_RATEEST.ko 2023-03-03T07:37:23.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_REDIRECT.ko 2023-03-03T07:37:23.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_SECMARK.ko 2023-03-03T07:37:23.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_TCPMSS.ko 2023-03-03T07:37:23.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-03-03T07:37:23.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_TEE.ko 2023-03-03T07:37:23.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_TPROXY.ko 2023-03-03T07:37:23.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_TRACE.ko 2023-03-03T07:37:23.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_addrtype.ko 2023-03-03T07:37:23.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_bpf.ko 2023-03-03T07:37:23.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_cgroup.ko 2023-03-03T07:37:23.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_cluster.ko 2023-03-03T07:37:23.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_comment.ko 2023-03-03T07:37:23.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_connbytes.ko 2023-03-03T07:37:23.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_connlabel.ko 2023-03-03T07:37:23.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_connlimit.ko 2023-03-03T07:37:23.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_connmark.ko 2023-03-03T07:37:23.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_conntrack.ko 2023-03-03T07:37:23.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_cpu.ko 2023-03-03T07:37:23.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_dccp.ko 2023-03-03T07:37:23.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_devgroup.ko 2023-03-03T07:37:23.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_dscp.ko 2023-03-03T07:37:23.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_ecn.ko 2023-03-03T07:37:23.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_esp.ko 2023-03-03T07:37:23.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_hashlimit.ko 2023-03-03T07:37:23.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_helper.ko 2023-03-03T07:37:23.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_hl.ko 2023-03-03T07:37:23.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_ipcomp.ko 2023-03-03T07:37:23.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_iprange.ko 2023-03-03T07:37:23.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_ipvs.ko 2023-03-03T07:37:23.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_l2tp.ko 2023-03-03T07:37:23.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_length.ko 2023-03-03T07:37:23.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_limit.ko 2023-03-03T07:37:23.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_mac.ko 2023-03-03T07:37:23.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_mark.ko 2023-03-03T07:37:23.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_multiport.ko 2023-03-03T07:37:23.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_nat.ko 2023-03-03T07:37:23.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_nfacct.ko 2023-03-03T07:37:23.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_osf.ko 2023-03-03T07:37:23.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_owner.ko 2023-03-03T07:37:23.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_physdev.ko 2023-03-03T07:37:23.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_pkttype.ko 2023-03-03T07:37:23.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_policy.ko 2023-03-03T07:37:23.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_quota.ko 2023-03-03T07:37:23.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_rateest.ko 2023-03-03T07:37:23.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_realm.ko 2023-03-03T07:37:23.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_recent.ko 2023-03-03T07:37:23.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_sctp.ko 2023-03-03T07:37:23.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_set.ko 2023-03-03T07:37:23.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_socket.ko 2023-03-03T07:37:23.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_state.ko 2023-03-03T07:37:23.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_statistic.ko 2023-03-03T07:37:23.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_string.ko 2023-03-03T07:37:23.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_tcpmss.ko 2023-03-03T07:37:23.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_time.ko 2023-03-03T07:37:23.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_u32.ko 2023-03-03T07:37:23.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netlink 2023-03-03T07:37:23.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netlink/netlink_diag.ko 2023-03-03T07:37:23.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/nsh 2023-03-03T07:37:23.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/nsh/nsh.ko 2023-03-03T07:37:23.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/openvswitch 2023-03-03T07:37:23.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/openvswitch/openvswitch.ko 2023-03-03T07:37:23.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/openvswitch/vport-geneve.ko 2023-03-03T07:37:23.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/openvswitch/vport-gre.ko 2023-03-03T07:37:23.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/openvswitch/vport-vxlan.ko 2023-03-03T07:37:23.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/packet 2023-03-03T07:37:23.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/packet/af_packet_diag.ko 2023-03-03T07:37:23.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/rds 2023-03-03T07:37:23.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/rds/rds.ko 2023-03-03T07:37:23.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/rds/rds_rdma.ko 2023-03-03T07:37:23.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/rds/rds_tcp.ko 2023-03-03T07:37:23.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched 2023-03-03T07:37:23.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_csum.ko 2023-03-03T07:37:23.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_gact.ko 2023-03-03T07:37:23.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_ipt.ko 2023-03-03T07:37:23.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_mirred.ko 2023-03-03T07:37:23.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_nat.ko 2023-03-03T07:37:23.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_pedit.ko 2023-03-03T07:37:23.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_police.ko 2023-03-03T07:37:23.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_simple.ko 2023-03-03T07:37:23.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_skbedit.ko 2023-03-03T07:37:23.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_vlan.ko 2023-03-03T07:37:23.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_basic.ko 2023-03-03T07:37:23.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_bpf.ko 2023-03-03T07:37:23.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_flow.ko 2023-03-03T07:37:23.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_fw.ko 2023-03-03T07:37:23.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_route.ko 2023-03-03T07:37:23.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_rsvp.ko 2023-03-03T07:37:23.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_rsvp6.ko 2023-03-03T07:37:23.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_tcindex.ko 2023-03-03T07:37:23.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_u32.ko 2023-03-03T07:37:23.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_cmp.ko 2023-03-03T07:37:23.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_ipset.ko 2023-03-03T07:37:23.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_meta.ko 2023-03-03T07:37:23.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_nbyte.ko 2023-03-03T07:37:23.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_text.ko 2023-03-03T07:37:23.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_u32.ko 2023-03-03T07:37:23.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_cbq.ko 2023-03-03T07:37:23.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_choke.ko 2023-03-03T07:37:23.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_codel.ko 2023-03-03T07:37:23.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_drr.ko 2023-03-03T07:37:23.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_dsmark.ko 2023-03-03T07:37:23.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_fq.ko 2023-03-03T07:37:23.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_gred.ko 2023-03-03T07:37:23.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_hfsc.ko 2023-03-03T07:37:23.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_hhf.ko 2023-03-03T07:37:23.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_htb.ko 2023-03-03T07:37:23.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_ingress.ko 2023-03-03T07:37:23.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_mqprio.ko 2023-03-03T07:37:23.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_multiq.ko 2023-03-03T07:37:23.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_netem.ko 2023-03-03T07:37:23.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_pie.ko 2023-03-03T07:37:23.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_plug.ko 2023-03-03T07:37:23.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_prio.ko 2023-03-03T07:37:23.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_qfq.ko 2023-03-03T07:37:23.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_red.ko 2023-03-03T07:37:23.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_sfb.ko 2023-03-03T07:37:23.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_sfq.ko 2023-03-03T07:37:23.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_tbf.ko 2023-03-03T07:37:23.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_teql.ko 2023-03-03T07:37:23.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sctp 2023-03-03T07:37:23.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sctp/sctp.ko 2023-03-03T07:37:23.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sctp/sctp_diag.ko 2023-03-03T07:37:23.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc 2023-03-03T07:37:23.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/auth_gss 2023-03-03T07:37:23.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-03-03T07:37:23.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-03-03T07:37:23.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/sunrpc.ko 2023-03-03T07:37:23.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/xprtrdma 2023-03-03T07:37:23.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-03-03T07:37:23.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/tipc 2023-03-03T07:37:23.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/tipc/diag.ko 2023-03-03T07:37:23.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/tipc/tipc.ko 2023-03-03T07:37:23.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/unix 2023-03-03T07:37:23.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/unix/unix_diag.ko 2023-03-03T07:37:23.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock 2023-03-03T07:37:23.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-03-03T07:37:23.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-03-03T07:37:23.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock/vsock.ko 2023-03-03T07:37:23.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock/vsock_diag.ko 2023-03-03T07:37:23.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock/vsock_loopback.ko 2023-03-03T07:37:23.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/xfrm 2023-03-03T07:37:23.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/xfrm/xfrm_ipcomp.ko 2023-03-03T07:37:23.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security 2023-03-03T07:37:23.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security/keys 2023-03-03T07:37:23.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security/keys/encrypted-keys 2023-03-03T07:37:23.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-03-03T07:37:23.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security/keys/trusted-keys 2023-03-03T07:37:23.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security/keys/trusted-keys/trusted.ko 2023-03-03T07:37:23.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/virt 2023-03-03T07:37:23.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/virt/lib 2023-03-03T07:37:23.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/virt/lib/irqbypass.ko 2023-03-03T07:37:23.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.alias 2023-03-03T07:37:23.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.alias.bin 2023-03-03T07:37:23.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.builtin 2023-03-03T07:37:23.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.builtin.bin 2023-03-03T07:37:23.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.builtin.modinfo 2023-03-03T07:37:23.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.dep 2023-03-03T07:37:23.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.dep.bin 2023-03-03T07:37:23.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.devname 2023-03-03T07:37:23.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.order 2023-03-03T07:37:23.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.softdep 2023-03-03T07:37:23.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.symbols 2023-03-03T07:37:23.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.symbols.bin 2023-03-03T07:37:23.310 DEBUG:teuthology.task.kernel:get_image_version: 6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:23.310 DEBUG:teuthology.orchestra.run.smithi098:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-03-03T07:37:23.338 INFO:teuthology.orchestra.run.smithi098.stdout:postinstall scriptlet (using /bin/sh): 2023-03-03T07:37:23.338 INFO:teuthology.orchestra.run.smithi098.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.2.0-ceph-g75534f74ac3d -a -r /boot/System.map-6.2.0-ceph-g75534f74ac3d ]; then 2023-03-03T07:37:23.339 INFO:teuthology.orchestra.run.smithi098.stdout:cp /boot/vmlinuz-6.2.0-ceph-g75534f74ac3d /boot/.vmlinuz-6.2.0-ceph-g75534f74ac3d-rpm 2023-03-03T07:37:23.339 INFO:teuthology.orchestra.run.smithi098.stdout:cp /boot/System.map-6.2.0-ceph-g75534f74ac3d /boot/.System.map-6.2.0-ceph-g75534f74ac3d-rpm 2023-03-03T07:37:23.339 INFO:teuthology.orchestra.run.smithi098.stdout:rm -f /boot/vmlinuz-6.2.0-ceph-g75534f74ac3d /boot/System.map-6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:23.339 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/installkernel 6.2.0-ceph-g75534f74ac3d /boot/.vmlinuz-6.2.0-ceph-g75534f74ac3d-rpm /boot/.System.map-6.2.0-ceph-g75534f74ac3d-rpm 2023-03-03T07:37:23.339 INFO:teuthology.orchestra.run.smithi098.stdout:rm -f /boot/.vmlinuz-6.2.0-ceph-g75534f74ac3d-rpm /boot/.System.map-6.2.0-ceph-g75534f74ac3d-rpm 2023-03-03T07:37:23.339 INFO:teuthology.orchestra.run.smithi098.stdout:fi 2023-03-03T07:37:23.339 INFO:teuthology.orchestra.run.smithi098.stdout:preuninstall scriptlet (using /bin/sh): 2023-03-03T07:37:23.340 INFO:teuthology.orchestra.run.smithi098.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-03-03T07:37:23.340 INFO:teuthology.orchestra.run.smithi098.stdout:new-kernel-pkg --remove 6.2.0-ceph-g75534f74ac3d --rminitrd --initrdfile=/boot/initramfs-6.2.0-ceph-g75534f74ac3d.img 2023-03-03T07:37:23.340 INFO:teuthology.orchestra.run.smithi098.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-03-03T07:37:23.340 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-install remove 6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:23.340 INFO:teuthology.orchestra.run.smithi098.stdout:fi 2023-03-03T07:37:23.340 INFO:teuthology.orchestra.run.smithi098.stdout:postuninstall scriptlet (using /bin/sh): 2023-03-03T07:37:23.340 INFO:teuthology.orchestra.run.smithi098.stdout:if [ -x /sbin/update-bootloader ]; then 2023-03-03T07:37:23.340 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/update-bootloader --remove 6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:23.341 INFO:teuthology.orchestra.run.smithi098.stdout:fi 2023-03-03T07:37:23.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2023-03-03T07:37:23.420 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2023-03-03T07:37:23.421 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2023-03-03T07:37:23.421 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.02 2023-03-03T07:37:23.421 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 129.el8 2023-03-03T07:37:23.421 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2023-03-03T07:37:23.421 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Sun 28 Aug 2022 01:03:50 PM UTC 2023-03-03T07:37:23.421 INFO:teuthology.orchestra.run.smithi098.stdout:Group : System Environment/Base 2023-03-03T07:37:23.421 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 9265586 2023-03-03T07:37:23.422 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2023-03-03T07:37:23.422 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-03-03T07:37:23.422 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-03-03T07:37:23.422 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-03-03T07:37:23.422 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : kbuilder.bsys.centos.org 2023-03-03T07:37:23.422 INFO:teuthology.orchestra.run.smithi098.stdout:Relocations : (not relocatable) 2023-03-03T07:37:23.422 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : CentOS BuildSystem 2023-03-03T07:37:23.422 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2023-03-03T07:37:23.423 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2023-03-03T07:37:23.423 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2023-03-03T07:37:23.423 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2023-03-03T07:37:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:37:23.423 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-03-03T07:37:23.423 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2023-03-03T07:37:23.423 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2023-03-03T07:37:23.423 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2023-03-03T07:37:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:37:23.424 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2023-03-03T07:37:23.425 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-03-03T07:37:23.425 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:23.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-03-03T07:37:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2023-03-03T07:37:28.382 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi100.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-03-03T07:37:28.382 DEBUG:teuthology.orchestra.run.smithi100:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-03-03T07:37:28.479 INFO:teuthology.orchestra.run.smithi100.stdout:/boot/System.map-6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:28.479 INFO:teuthology.orchestra.run.smithi100.stdout:/boot/config-6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:28.480 INFO:teuthology.orchestra.run.smithi100.stdout:/boot/vmlinuz-6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:28.480 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:28.480 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel 2023-03-03T07:37:28.480 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch 2023-03-03T07:37:28.480 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86 2023-03-03T07:37:28.480 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto 2023-03-03T07:37:28.480 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-03-03T07:37:28.481 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-03-03T07:37:28.481 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-03-03T07:37:28.481 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-03-03T07:37:28.481 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-03-03T07:37:28.481 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-03-03T07:37:28.481 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-03-03T07:37:28.481 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/crc32c-intel.ko 2023-03-03T07:37:28.482 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-03-03T07:37:28.482 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-03-03T07:37:28.482 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-03-03T07:37:28.482 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-03-03T07:37:28.482 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/serpent-avx2.ko 2023-03-03T07:37:28.482 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-03-03T07:37:28.482 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-03-03T07:37:28.482 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-03-03T07:37:28.483 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-03-03T07:37:28.483 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-03-03T07:37:28.483 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-03-03T07:37:28.483 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-03-03T07:37:28.483 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kernel 2023-03-03T07:37:28.483 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kernel/cpu 2023-03-03T07:37:28.483 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kernel/cpu/mce 2023-03-03T07:37:28.484 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-03-03T07:37:28.484 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kvm 2023-03-03T07:37:28.484 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kvm/kvm-amd.ko 2023-03-03T07:37:28.484 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kvm/kvm-intel.ko 2023-03-03T07:37:28.484 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kvm/kvm.ko 2023-03-03T07:37:28.484 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto 2023-03-03T07:37:28.484 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/ansi_cprng.ko 2023-03-03T07:37:28.485 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/anubis.ko 2023-03-03T07:37:28.485 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/arc4.ko 2023-03-03T07:37:28.485 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx 2023-03-03T07:37:28.485 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/async_memcpy.ko 2023-03-03T07:37:28.485 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/async_pq.ko 2023-03-03T07:37:28.485 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/async_raid6_recov.ko 2023-03-03T07:37:28.485 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/async_tx.ko 2023-03-03T07:37:28.486 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/async_xor.ko 2023-03-03T07:37:28.486 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/raid6test.ko 2023-03-03T07:37:28.486 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/authenc.ko 2023-03-03T07:37:28.486 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/authencesn.ko 2023-03-03T07:37:28.486 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/blake2b_generic.ko 2023-03-03T07:37:28.486 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/blowfish_common.ko 2023-03-03T07:37:28.486 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/blowfish_generic.ko 2023-03-03T07:37:28.486 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/camellia_generic.ko 2023-03-03T07:37:28.487 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/cast5_generic.ko 2023-03-03T07:37:28.487 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/cast6_generic.ko 2023-03-03T07:37:28.487 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/cast_common.ko 2023-03-03T07:37:28.487 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/ccm.ko 2023-03-03T07:37:28.487 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/cmac.ko 2023-03-03T07:37:28.487 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/crc32_generic.ko 2023-03-03T07:37:28.487 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/crypto_engine.ko 2023-03-03T07:37:28.487 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/crypto_user.ko 2023-03-03T07:37:28.488 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/cts.ko 2023-03-03T07:37:28.488 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/des_generic.ko 2023-03-03T07:37:28.488 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/dh_generic.ko 2023-03-03T07:37:28.488 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/echainiv.ko 2023-03-03T07:37:28.488 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/essiv.ko 2023-03-03T07:37:28.488 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/fcrypt.ko 2023-03-03T07:37:28.488 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/khazad.ko 2023-03-03T07:37:28.489 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/lz4.ko 2023-03-03T07:37:28.489 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/lz4hc.ko 2023-03-03T07:37:28.489 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/md4.ko 2023-03-03T07:37:28.489 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/michael_mic.ko 2023-03-03T07:37:28.489 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/pcbc.ko 2023-03-03T07:37:28.489 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/pcrypt.ko 2023-03-03T07:37:28.489 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/rmd160.ko 2023-03-03T07:37:28.489 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/seed.ko 2023-03-03T07:37:28.490 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/serpent_generic.ko 2023-03-03T07:37:28.490 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/tcrypt.ko 2023-03-03T07:37:28.490 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/tea.ko 2023-03-03T07:37:28.490 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/twofish_common.ko 2023-03-03T07:37:28.490 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/twofish_generic.ko 2023-03-03T07:37:28.490 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/vmac.ko 2023-03-03T07:37:28.490 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/wp512.ko 2023-03-03T07:37:28.491 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/xcbc.ko 2023-03-03T07:37:28.491 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/xor.ko 2023-03-03T07:37:28.491 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/xxhash_generic.ko 2023-03-03T07:37:28.491 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers 2023-03-03T07:37:28.491 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi 2023-03-03T07:37:28.491 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/acpi_ipmi.ko 2023-03-03T07:37:28.491 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/acpi_pad.ko 2023-03-03T07:37:28.491 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/custom_method.ko 2023-03-03T07:37:28.492 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/ec_sys.ko 2023-03-03T07:37:28.492 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/platform_profile.ko 2023-03-03T07:37:28.492 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/sbs.ko 2023-03-03T07:37:28.492 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/sbshc.ko 2023-03-03T07:37:28.492 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/video.ko 2023-03-03T07:37:28.492 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata 2023-03-03T07:37:28.492 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/acard-ahci.ko 2023-03-03T07:37:28.492 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/ahci_platform.ko 2023-03-03T07:37:28.493 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/ata_generic.ko 2023-03-03T07:37:28.493 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/libahci_platform.ko 2023-03-03T07:37:28.493 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_acpi.ko 2023-03-03T07:37:28.493 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_ali.ko 2023-03-03T07:37:28.493 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_amd.ko 2023-03-03T07:37:28.493 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_artop.ko 2023-03-03T07:37:28.493 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_atiixp.ko 2023-03-03T07:37:28.494 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_atp867x.ko 2023-03-03T07:37:28.494 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_cmd640.ko 2023-03-03T07:37:28.494 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_cmd64x.ko 2023-03-03T07:37:28.494 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_cypress.ko 2023-03-03T07:37:28.494 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_efar.ko 2023-03-03T07:37:28.494 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_hpt366.ko 2023-03-03T07:37:28.495 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_hpt37x.ko 2023-03-03T07:37:28.495 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_hpt3x2n.ko 2023-03-03T07:37:28.495 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_hpt3x3.ko 2023-03-03T07:37:28.495 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_it8213.ko 2023-03-03T07:37:28.495 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_it821x.ko 2023-03-03T07:37:28.495 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_jmicron.ko 2023-03-03T07:37:28.495 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_marvell.ko 2023-03-03T07:37:28.495 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_mpiix.ko 2023-03-03T07:37:28.496 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_netcell.ko 2023-03-03T07:37:28.496 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_ninja32.ko 2023-03-03T07:37:28.496 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_ns87410.ko 2023-03-03T07:37:28.496 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_ns87415.ko 2023-03-03T07:37:28.496 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_oldpiix.ko 2023-03-03T07:37:28.496 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_opti.ko 2023-03-03T07:37:28.496 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_optidma.ko 2023-03-03T07:37:28.497 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_pcmcia.ko 2023-03-03T07:37:28.497 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_pdc2027x.ko 2023-03-03T07:37:28.497 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-03-03T07:37:28.497 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_piccolo.ko 2023-03-03T07:37:28.497 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_rdc.ko 2023-03-03T07:37:28.497 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_sch.ko 2023-03-03T07:37:28.498 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_serverworks.ko 2023-03-03T07:37:28.498 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_sil680.ko 2023-03-03T07:37:28.498 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_sis.ko 2023-03-03T07:37:28.498 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_sl82c105.ko 2023-03-03T07:37:28.498 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_triflex.ko 2023-03-03T07:37:28.498 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_via.ko 2023-03-03T07:37:28.499 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pdc_adma.ko 2023-03-03T07:37:28.499 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_inic162x.ko 2023-03-03T07:37:28.499 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_mv.ko 2023-03-03T07:37:28.499 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_nv.ko 2023-03-03T07:37:28.499 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_promise.ko 2023-03-03T07:37:28.500 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_qstor.ko 2023-03-03T07:37:28.500 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_sil.ko 2023-03-03T07:37:28.500 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_sil24.ko 2023-03-03T07:37:28.500 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_sis.ko 2023-03-03T07:37:28.500 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_svw.ko 2023-03-03T07:37:28.500 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_sx4.ko 2023-03-03T07:37:28.500 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_uli.ko 2023-03-03T07:37:28.501 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_via.ko 2023-03-03T07:37:28.501 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_vsc.ko 2023-03-03T07:37:28.501 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/auxdisplay 2023-03-03T07:37:28.501 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/auxdisplay/ks0108.ko 2023-03-03T07:37:28.501 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/base 2023-03-03T07:37:28.501 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/base/regmap 2023-03-03T07:37:28.501 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/base/regmap/regmap-i2c.ko 2023-03-03T07:37:28.502 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/bcma 2023-03-03T07:37:28.502 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/bcma/bcma.ko 2023-03-03T07:37:28.502 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block 2023-03-03T07:37:28.502 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/aoe 2023-03-03T07:37:28.502 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/aoe/aoe.ko 2023-03-03T07:37:28.502 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/brd.ko 2023-03-03T07:37:28.503 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/drbd 2023-03-03T07:37:28.503 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/drbd/drbd.ko 2023-03-03T07:37:28.503 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/floppy.ko 2023-03-03T07:37:28.503 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/loop.ko 2023-03-03T07:37:28.503 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/mtip32xx 2023-03-03T07:37:28.503 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-03-03T07:37:28.503 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/nbd.ko 2023-03-03T07:37:28.504 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/null_blk 2023-03-03T07:37:28.504 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/null_blk/null_blk.ko 2023-03-03T07:37:28.504 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/pktcdvd.ko 2023-03-03T07:37:28.504 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/rbd.ko 2023-03-03T07:37:28.504 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/virtio_blk.ko 2023-03-03T07:37:28.504 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/xen-blkback 2023-03-03T07:37:28.505 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-03-03T07:37:28.505 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/xen-blkfront.ko 2023-03-03T07:37:28.505 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/zram 2023-03-03T07:37:28.505 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/zram/zram.ko 2023-03-03T07:37:28.505 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char 2023-03-03T07:37:28.505 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hangcheck-timer.ko 2023-03-03T07:37:28.505 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random 2023-03-03T07:37:28.506 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random/amd-rng.ko 2023-03-03T07:37:28.506 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random/intel-rng.ko 2023-03-03T07:37:28.506 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-03-03T07:37:28.506 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random/via-rng.ko 2023-03-03T07:37:28.506 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random/virtio-rng.ko 2023-03-03T07:37:28.506 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi 2023-03-03T07:37:28.506 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-03-03T07:37:28.507 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-03-03T07:37:28.507 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-03-03T07:37:28.507 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_si.ko 2023-03-03T07:37:28.507 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-03-03T07:37:28.507 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-03-03T07:37:28.507 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/lp.ko 2023-03-03T07:37:28.507 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/mwave 2023-03-03T07:37:28.508 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/mwave/mwave.ko 2023-03-03T07:37:28.508 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/pcmcia 2023-03-03T07:37:28.508 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/pcmcia/cm4000_cs.ko 2023-03-03T07:37:28.508 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/pcmcia/cm4040_cs.ko 2023-03-03T07:37:28.508 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ppdev.ko 2023-03-03T07:37:28.508 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tlclk.ko 2023-03-03T07:37:28.508 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm 2023-03-03T07:37:28.509 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm.ko 2023-03-03T07:37:28.509 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm_atmel.ko 2023-03-03T07:37:28.509 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm_infineon.ko 2023-03-03T07:37:28.509 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm_nsc.ko 2023-03-03T07:37:28.509 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm_tis.ko 2023-03-03T07:37:28.509 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-03-03T07:37:28.510 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/uv_mmtimer.ko 2023-03-03T07:37:28.510 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/virtio_console.ko 2023-03-03T07:37:28.510 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq 2023-03-03T07:37:28.510 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-03-03T07:37:28.510 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-03-03T07:37:28.510 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/p4-clockmod.ko 2023-03-03T07:37:28.510 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-03-03T07:37:28.511 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/powernow-k8.ko 2023-03-03T07:37:28.511 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/speedstep-lib.ko 2023-03-03T07:37:28.511 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto 2023-03-03T07:37:28.511 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/ccp 2023-03-03T07:37:28.511 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-03-03T07:37:28.511 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/ccp/ccp.ko 2023-03-03T07:37:28.511 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/padlock-aes.ko 2023-03-03T07:37:28.512 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/padlock-sha.ko 2023-03-03T07:37:28.512 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/qat 2023-03-03T07:37:28.512 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/qat/qat_common 2023-03-03T07:37:28.512 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/qat/qat_common/intel_qat.ko 2023-03-03T07:37:28.512 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/qat/qat_dh895xcc 2023-03-03T07:37:28.512 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-03-03T07:37:28.512 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/virtio 2023-03-03T07:37:28.513 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-03-03T07:37:28.513 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dca 2023-03-03T07:37:28.513 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dca/dca.ko 2023-03-03T07:37:28.513 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/devfreq 2023-03-03T07:37:28.513 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-03-03T07:37:28.514 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dma 2023-03-03T07:37:28.514 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dma/dw 2023-03-03T07:37:28.514 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dma/dw/dw_dmac.ko 2023-03-03T07:37:28.514 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dma/ioat 2023-03-03T07:37:28.514 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dma/ioat/ioatdma.ko 2023-03-03T07:37:28.514 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac 2023-03-03T07:37:28.514 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/amd64_edac.ko 2023-03-03T07:37:28.514 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/e752x_edac.ko 2023-03-03T07:37:28.515 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/edac_mce_amd.ko 2023-03-03T07:37:28.515 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i3000_edac.ko 2023-03-03T07:37:28.515 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i3200_edac.ko 2023-03-03T07:37:28.515 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i5100_edac.ko 2023-03-03T07:37:28.515 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i5400_edac.ko 2023-03-03T07:37:28.515 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i7300_edac.ko 2023-03-03T07:37:28.515 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i7core_edac.ko 2023-03-03T07:37:28.516 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i82975x_edac.ko 2023-03-03T07:37:28.516 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/ie31200_edac.ko 2023-03-03T07:37:28.516 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/sb_edac.ko 2023-03-03T07:37:28.516 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/x38_edac.ko 2023-03-03T07:37:28.516 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/firmware 2023-03-03T07:37:28.516 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/firmware/edd.ko 2023-03-03T07:37:28.517 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/firmware/iscsi_ibft.ko 2023-03-03T07:37:28.517 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/gpio 2023-03-03T07:37:28.517 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/gpio/gpio-ich.ko 2023-03-03T07:37:28.517 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/gpio/gpio-viperboard.ko 2023-03-03T07:37:28.517 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid 2023-03-03T07:37:28.517 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-appleir.ko 2023-03-03T07:37:28.517 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-aureal.ko 2023-03-03T07:37:28.518 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-axff.ko 2023-03-03T07:37:28.518 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-dr.ko 2023-03-03T07:37:28.518 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-elecom.ko 2023-03-03T07:37:28.518 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-elo.ko 2023-03-03T07:37:28.518 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-emsff.ko 2023-03-03T07:37:28.518 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-gaff.ko 2023-03-03T07:37:28.518 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-gt683r.ko 2023-03-03T07:37:28.518 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-gyration.ko 2023-03-03T07:37:28.519 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-holtek-kbd.ko 2023-03-03T07:37:28.519 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-holtek-mouse.ko 2023-03-03T07:37:28.519 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-holtekff.ko 2023-03-03T07:37:28.519 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-hyperv.ko 2023-03-03T07:37:28.519 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-icade.ko 2023-03-03T07:37:28.519 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-keytouch.ko 2023-03-03T07:37:28.520 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-kye.ko 2023-03-03T07:37:28.520 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-lcpower.ko 2023-03-03T07:37:28.520 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-led.ko 2023-03-03T07:37:28.520 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-lenovo.ko 2023-03-03T07:37:28.520 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-logitech-dj.ko 2023-03-03T07:37:28.520 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-03-03T07:37:28.520 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-multitouch.ko 2023-03-03T07:37:28.520 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-ortek.ko 2023-03-03T07:37:28.521 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-penmount.ko 2023-03-03T07:37:28.521 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-petalynx.ko 2023-03-03T07:37:28.521 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-picolcd.ko 2023-03-03T07:37:28.521 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-pl.ko 2023-03-03T07:37:28.521 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-plantronics.ko 2023-03-03T07:37:28.521 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-primax.ko 2023-03-03T07:37:28.521 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-rmi.ko 2023-03-03T07:37:28.522 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-arvo.ko 2023-03-03T07:37:28.522 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-common.ko 2023-03-03T07:37:28.522 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-isku.ko 2023-03-03T07:37:28.522 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-kone.ko 2023-03-03T07:37:28.522 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-03-03T07:37:28.522 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-konepure.ko 2023-03-03T07:37:28.522 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-03-03T07:37:28.523 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-lua.ko 2023-03-03T07:37:28.523 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-pyra.ko 2023-03-03T07:37:28.523 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-ryos.ko 2023-03-03T07:37:28.523 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-savu.ko 2023-03-03T07:37:28.523 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat.ko 2023-03-03T07:37:28.523 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-saitek.ko 2023-03-03T07:37:28.523 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-samsung.ko 2023-03-03T07:37:28.524 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-sensor-hub.ko 2023-03-03T07:37:28.524 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-sjoy.ko 2023-03-03T07:37:28.524 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-sony.ko 2023-03-03T07:37:28.524 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-speedlink.ko 2023-03-03T07:37:28.524 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-steelseries.ko 2023-03-03T07:37:28.524 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-sunplus.ko 2023-03-03T07:37:28.525 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-thrustmaster.ko 2023-03-03T07:37:28.525 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-tivo.ko 2023-03-03T07:37:28.525 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-tmff.ko 2023-03-03T07:37:28.525 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-topseed.ko 2023-03-03T07:37:28.525 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-twinhan.ko 2023-03-03T07:37:28.525 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-uclogic.ko 2023-03-03T07:37:28.525 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-waltop.ko 2023-03-03T07:37:28.526 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-wiimote.ko 2023-03-03T07:37:28.526 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-xinmo.ko 2023-03-03T07:37:28.526 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-zpff.ko 2023-03-03T07:37:28.526 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-zydacron.ko 2023-03-03T07:37:28.526 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/uhid.ko 2023-03-03T07:37:28.526 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/wacom.ko 2023-03-03T07:37:28.526 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hv 2023-03-03T07:37:28.527 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hv/hv_balloon.ko 2023-03-03T07:37:28.527 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hv/hv_utils.ko 2023-03-03T07:37:28.527 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hv/hv_vmbus.ko 2023-03-03T07:37:28.527 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon 2023-03-03T07:37:28.527 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/abituguru.ko 2023-03-03T07:37:28.527 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/abituguru3.ko 2023-03-03T07:37:28.527 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/acpi_power_meter.ko 2023-03-03T07:37:28.527 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ad7414.ko 2023-03-03T07:37:28.528 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ad7418.ko 2023-03-03T07:37:28.528 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adc128d818.ko 2023-03-03T07:37:28.528 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adm1025.ko 2023-03-03T07:37:28.528 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adm1026.ko 2023-03-03T07:37:28.528 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adm1029.ko 2023-03-03T07:37:28.528 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adm1031.ko 2023-03-03T07:37:28.528 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adm9240.ko 2023-03-03T07:37:28.528 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ads7828.ko 2023-03-03T07:37:28.529 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7410.ko 2023-03-03T07:37:28.529 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7411.ko 2023-03-03T07:37:28.529 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7462.ko 2023-03-03T07:37:28.529 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7470.ko 2023-03-03T07:37:28.529 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7475.ko 2023-03-03T07:37:28.529 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7x10.ko 2023-03-03T07:37:28.530 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/amc6821.ko 2023-03-03T07:37:28.530 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/applesmc.ko 2023-03-03T07:37:28.530 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/asb100.ko 2023-03-03T07:37:28.530 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/asc7621.ko 2023-03-03T07:37:28.530 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/asus_atk0110.ko 2023-03-03T07:37:28.530 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/atxp1.ko 2023-03-03T07:37:28.530 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/coretemp.ko 2023-03-03T07:37:28.530 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-03-03T07:37:28.531 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/dme1737.ko 2023-03-03T07:37:28.531 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ds1621.ko 2023-03-03T07:37:28.531 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ds620.ko 2023-03-03T07:37:28.531 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/emc1403.ko 2023-03-03T07:37:28.531 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/emc6w201.ko 2023-03-03T07:37:28.531 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/f71805f.ko 2023-03-03T07:37:28.531 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/f71882fg.ko 2023-03-03T07:37:28.532 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/f75375s.ko 2023-03-03T07:37:28.532 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/fam15h_power.ko 2023-03-03T07:37:28.532 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/fschmd.ko 2023-03-03T07:37:28.532 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/g760a.ko 2023-03-03T07:37:28.532 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/g762.ko 2023-03-03T07:37:28.532 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/gl518sm.ko 2023-03-03T07:37:28.532 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/gl520sm.ko 2023-03-03T07:37:28.533 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/hwmon-vid.ko 2023-03-03T07:37:28.533 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/i5500_temp.ko 2023-03-03T07:37:28.533 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/i5k_amb.ko 2023-03-03T07:37:28.533 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ibmaem.ko 2023-03-03T07:37:28.533 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ibmpex.ko 2023-03-03T07:37:28.533 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ina209.ko 2023-03-03T07:37:28.534 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ina2xx.ko 2023-03-03T07:37:28.534 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/it87.ko 2023-03-03T07:37:28.534 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/k10temp.ko 2023-03-03T07:37:28.534 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/k8temp.ko 2023-03-03T07:37:28.534 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lineage-pem.ko 2023-03-03T07:37:28.534 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm63.ko 2023-03-03T07:37:28.534 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm73.ko 2023-03-03T07:37:28.535 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm75.ko 2023-03-03T07:37:28.535 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm77.ko 2023-03-03T07:37:28.535 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm78.ko 2023-03-03T07:37:28.535 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm80.ko 2023-03-03T07:37:28.535 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm83.ko 2023-03-03T07:37:28.535 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm85.ko 2023-03-03T07:37:28.535 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm87.ko 2023-03-03T07:37:28.536 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm90.ko 2023-03-03T07:37:28.536 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm92.ko 2023-03-03T07:37:28.536 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm93.ko 2023-03-03T07:37:28.536 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm95234.ko 2023-03-03T07:37:28.536 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm95241.ko 2023-03-03T07:37:28.536 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm95245.ko 2023-03-03T07:37:28.536 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc2945.ko 2023-03-03T07:37:28.537 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4151.ko 2023-03-03T07:37:28.537 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4215.ko 2023-03-03T07:37:28.537 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4222.ko 2023-03-03T07:37:28.537 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4245.ko 2023-03-03T07:37:28.537 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4260.ko 2023-03-03T07:37:28.537 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4261.ko 2023-03-03T07:37:28.537 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max16065.ko 2023-03-03T07:37:28.538 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max1619.ko 2023-03-03T07:37:28.538 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max1668.ko 2023-03-03T07:37:28.538 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max197.ko 2023-03-03T07:37:28.538 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max6639.ko 2023-03-03T07:37:28.538 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max6650.ko 2023-03-03T07:37:28.538 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max6697.ko 2023-03-03T07:37:28.538 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/mcp3021.ko 2023-03-03T07:37:28.539 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/nct6683.ko 2023-03-03T07:37:28.539 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/nct6775-core.ko 2023-03-03T07:37:28.539 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/nct6775.ko 2023-03-03T07:37:28.539 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/nct7802.ko 2023-03-03T07:37:28.539 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pc87360.ko 2023-03-03T07:37:28.539 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pc87427.ko 2023-03-03T07:37:28.539 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pcf8591.ko 2023-03-03T07:37:28.539 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus 2023-03-03T07:37:28.540 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-03-03T07:37:28.540 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-03-03T07:37:28.540 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-03-03T07:37:28.540 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/max16064.ko 2023-03-03T07:37:28.540 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/max34440.ko 2023-03-03T07:37:28.540 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/max8688.ko 2023-03-03T07:37:28.540 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-03-03T07:37:28.540 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-03-03T07:37:28.541 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-03-03T07:37:28.541 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-03-03T07:37:28.541 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-03-03T07:37:28.541 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-03-03T07:37:28.541 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/powr1220.ko 2023-03-03T07:37:28.541 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sch5627.ko 2023-03-03T07:37:28.541 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sch5636.ko 2023-03-03T07:37:28.541 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sch56xx-common.ko 2023-03-03T07:37:28.542 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sht15.ko 2023-03-03T07:37:28.542 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sht21.ko 2023-03-03T07:37:28.542 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/shtc1.ko 2023-03-03T07:37:28.542 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sis5595.ko 2023-03-03T07:37:28.542 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/smsc47b397.ko 2023-03-03T07:37:28.542 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/smsc47m1.ko 2023-03-03T07:37:28.542 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/smsc47m192.ko 2023-03-03T07:37:28.543 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/thmc50.ko 2023-03-03T07:37:28.543 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/tmp102.ko 2023-03-03T07:37:28.543 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/tmp103.ko 2023-03-03T07:37:28.543 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/tmp401.ko 2023-03-03T07:37:28.543 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/tmp421.ko 2023-03-03T07:37:28.543 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/via-cputemp.ko 2023-03-03T07:37:28.543 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/via686a.ko 2023-03-03T07:37:28.543 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/vt1211.ko 2023-03-03T07:37:28.544 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/vt8231.ko 2023-03-03T07:37:28.544 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83627ehf.ko 2023-03-03T07:37:28.544 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83627hf.ko 2023-03-03T07:37:28.544 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83781d.ko 2023-03-03T07:37:28.544 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83791d.ko 2023-03-03T07:37:28.544 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83792d.ko 2023-03-03T07:37:28.544 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83793.ko 2023-03-03T07:37:28.545 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83795.ko 2023-03-03T07:37:28.545 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83l785ts.ko 2023-03-03T07:37:28.545 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83l786ng.ko 2023-03-03T07:37:28.545 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c 2023-03-03T07:37:28.545 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/algos 2023-03-03T07:37:28.545 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-03-03T07:37:28.545 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-03-03T07:37:28.546 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses 2023-03-03T07:37:28.546 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-03-03T07:37:28.546 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-03-03T07:37:28.546 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-03-03T07:37:28.546 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-03-03T07:37:28.546 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-03-03T07:37:28.546 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-03-03T07:37:28.546 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-03-03T07:37:28.547 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-03-03T07:37:28.547 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-i801.ko 2023-03-03T07:37:28.547 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-isch.ko 2023-03-03T07:37:28.547 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-03-03T07:37:28.547 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-03-03T07:37:28.547 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-03-03T07:37:28.547 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-parport.ko 2023-03-03T07:37:28.547 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-03-03T07:37:28.548 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-03-03T07:37:28.548 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-03-03T07:37:28.548 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-03-03T07:37:28.548 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-03-03T07:37:28.548 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-03-03T07:37:28.548 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-via.ko 2023-03-03T07:37:28.548 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-03-03T07:37:28.549 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-03-03T07:37:28.549 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/i2c-dev.ko 2023-03-03T07:37:28.549 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/i2c-mux.ko 2023-03-03T07:37:28.549 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-03-03T07:37:28.549 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/i2c-smbus.ko 2023-03-03T07:37:28.549 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/i2c-stub.ko 2023-03-03T07:37:28.549 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband 2023-03-03T07:37:28.549 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core 2023-03-03T07:37:28.550 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/ib_cm.ko 2023-03-03T07:37:28.550 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/ib_core.ko 2023-03-03T07:37:28.550 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/ib_umad.ko 2023-03-03T07:37:28.550 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-03-03T07:37:28.550 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/iw_cm.ko 2023-03-03T07:37:28.550 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/rdma_cm.ko 2023-03-03T07:37:28.550 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-03-03T07:37:28.551 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw 2023-03-03T07:37:28.551 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/cxgb4 2023-03-03T07:37:28.551 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-03-03T07:37:28.551 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mlx4 2023-03-03T07:37:28.551 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-03-03T07:37:28.551 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mlx5 2023-03-03T07:37:28.551 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-03-03T07:37:28.551 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mthca 2023-03-03T07:37:28.552 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-03-03T07:37:28.552 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp 2023-03-03T07:37:28.552 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/ipoib 2023-03-03T07:37:28.552 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-03-03T07:37:28.552 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/iser 2023-03-03T07:37:28.552 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-03-03T07:37:28.552 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/isert 2023-03-03T07:37:28.552 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-03-03T07:37:28.553 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/srp 2023-03-03T07:37:28.553 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-03-03T07:37:28.553 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/srpt 2023-03-03T07:37:28.553 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-03-03T07:37:28.553 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input 2023-03-03T07:37:28.553 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport 2023-03-03T07:37:28.553 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-03-03T07:37:28.554 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport/fm801-gp.ko 2023-03-03T07:37:28.554 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport/gameport.ko 2023-03-03T07:37:28.554 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport/lightning.ko 2023-03-03T07:37:28.554 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport/ns558.ko 2023-03-03T07:37:28.554 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/joydev.ko 2023-03-03T07:37:28.554 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/keyboard 2023-03-03T07:37:28.554 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/keyboard/gpio_keys.ko 2023-03-03T07:37:28.555 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse 2023-03-03T07:37:28.555 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/appletouch.ko 2023-03-03T07:37:28.555 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/bcm5974.ko 2023-03-03T07:37:28.555 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/cyapatp.ko 2023-03-03T07:37:28.555 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/elan_i2c.ko 2023-03-03T07:37:28.555 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/sermouse.ko 2023-03-03T07:37:28.555 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-03-03T07:37:28.555 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/synaptics_usb.ko 2023-03-03T07:37:28.556 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/vsxxxaa.ko 2023-03-03T07:37:28.556 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/rmi4 2023-03-03T07:37:28.556 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/rmi4/rmi_core.ko 2023-03-03T07:37:28.556 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/serio 2023-03-03T07:37:28.556 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/serio/altera_ps2.ko 2023-03-03T07:37:28.556 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/serio/arc_ps2.ko 2023-03-03T07:37:28.556 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-03-03T07:37:28.556 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/serio/serio_raw.ko 2023-03-03T07:37:28.557 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/sparse-keymap.ko 2023-03-03T07:37:28.557 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/iommu 2023-03-03T07:37:28.557 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/iommu/amd 2023-03-03T07:37:28.557 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/iommu/amd/iommu_v2.ko 2023-03-03T07:37:28.557 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds 2023-03-03T07:37:28.557 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/leds-blinkm.ko 2023-03-03T07:37:28.557 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/leds-lm3530.ko 2023-03-03T07:37:28.558 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/leds-lp3944.ko 2023-03-03T07:37:28.558 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/leds-ss4200.ko 2023-03-03T07:37:28.558 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger 2023-03-03T07:37:28.558 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-03-03T07:37:28.558 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-03-03T07:37:28.558 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-03-03T07:37:28.558 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-03-03T07:37:28.558 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2023-03-03T07:37:28.559 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-03-03T07:37:28.559 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-03-03T07:37:28.559 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-03-03T07:37:28.559 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-03-03T07:37:28.559 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md 2023-03-03T07:37:28.559 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/bcache 2023-03-03T07:37:28.559 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/bcache/bcache.ko 2023-03-03T07:37:28.560 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-bio-prison.ko 2023-03-03T07:37:28.560 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-cache-smq.ko 2023-03-03T07:37:28.560 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-cache.ko 2023-03-03T07:37:28.560 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-crypt.ko 2023-03-03T07:37:28.560 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-delay.ko 2023-03-03T07:37:28.560 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-flakey.ko 2023-03-03T07:37:28.560 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-log-userspace.ko 2023-03-03T07:37:28.561 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-multipath.ko 2023-03-03T07:37:28.561 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-queue-length.ko 2023-03-03T07:37:28.561 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-raid.ko 2023-03-03T07:37:28.561 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-round-robin.ko 2023-03-03T07:37:28.561 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-service-time.ko 2023-03-03T07:37:28.561 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-switch.ko 2023-03-03T07:37:28.561 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-thin-pool.ko 2023-03-03T07:37:28.561 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-verity.ko 2023-03-03T07:37:28.562 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/faulty.ko 2023-03-03T07:37:28.562 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/linear.ko 2023-03-03T07:37:28.562 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/multipath.ko 2023-03-03T07:37:28.562 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/persistent-data 2023-03-03T07:37:28.562 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-03-03T07:37:28.562 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/raid0.ko 2023-03-03T07:37:28.562 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/raid1.ko 2023-03-03T07:37:28.563 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/raid10.ko 2023-03-03T07:37:28.563 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/raid456.ko 2023-03-03T07:37:28.563 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message 2023-03-03T07:37:28.563 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion 2023-03-03T07:37:28.563 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptbase.ko 2023-03-03T07:37:28.563 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptctl.ko 2023-03-03T07:37:28.563 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptfc.ko 2023-03-03T07:37:28.563 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptlan.ko 2023-03-03T07:37:28.564 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptsas.ko 2023-03-03T07:37:28.564 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptscsih.ko 2023-03-03T07:37:28.564 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptspi.ko 2023-03-03T07:37:28.564 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd 2023-03-03T07:37:28.564 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/lpc_ich.ko 2023-03-03T07:37:28.564 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/lpc_sch.ko 2023-03-03T07:37:28.564 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/mfd-core.ko 2023-03-03T07:37:28.565 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/sm501.ko 2023-03-03T07:37:28.565 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/viperboard.ko 2023-03-03T07:37:28.565 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/vx855.ko 2023-03-03T07:37:28.565 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/wl1273-core.ko 2023-03-03T07:37:28.565 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc 2023-03-03T07:37:28.565 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/altera-stapl 2023-03-03T07:37:28.565 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-03-03T07:37:28.566 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/apds9802als.ko 2023-03-03T07:37:28.566 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/apds990x.ko 2023-03-03T07:37:28.566 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/bh1770glc.ko 2023-03-03T07:37:28.566 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/cb710 2023-03-03T07:37:28.566 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/cb710/cb710.ko 2023-03-03T07:37:28.566 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/eeprom 2023-03-03T07:37:28.566 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/eeprom/at24.ko 2023-03-03T07:37:28.566 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/eeprom/eeprom.ko 2023-03-03T07:37:28.567 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-03-03T07:37:28.567 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/eeprom/max6875.ko 2023-03-03T07:37:28.567 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/enclosure.ko 2023-03-03T07:37:28.567 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/hpilo.ko 2023-03-03T07:37:28.567 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/isl29003.ko 2023-03-03T07:37:28.567 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/isl29020.ko 2023-03-03T07:37:28.567 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/lis3lv02d 2023-03-03T07:37:28.567 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-03-03T07:37:28.568 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-03-03T07:37:28.568 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/mei 2023-03-03T07:37:28.568 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/mei/mei-me.ko 2023-03-03T07:37:28.568 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/mei/mei-txe.ko 2023-03-03T07:37:28.568 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/mei/mei.ko 2023-03-03T07:37:28.568 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-gru 2023-03-03T07:37:28.568 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-gru/gru.ko 2023-03-03T07:37:28.569 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-xp 2023-03-03T07:37:28.569 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-xp/xp.ko 2023-03-03T07:37:28.569 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-xp/xpc.ko 2023-03-03T07:37:28.569 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-03-03T07:37:28.569 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/tifm_7xx1.ko 2023-03-03T07:37:28.569 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/tifm_core.ko 2023-03-03T07:37:28.569 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/tsl2550.ko 2023-03-03T07:37:28.569 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/vmw_balloon.ko 2023-03-03T07:37:28.570 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/vmw_vmci 2023-03-03T07:37:28.570 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-03-03T07:37:28.570 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net 2023-03-03T07:37:28.570 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/bonding 2023-03-03T07:37:28.570 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/bonding/bonding.ko 2023-03-03T07:37:28.570 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa 2023-03-03T07:37:28.570 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/b53 2023-03-03T07:37:28.570 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/b53/b53_common.ko 2023-03-03T07:37:28.571 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/bcm-sf2.ko 2023-03-03T07:37:28.571 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/mv88e6060.ko 2023-03-03T07:37:28.571 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/mv88e6xxx 2023-03-03T07:37:28.571 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-03-03T07:37:28.571 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dummy.ko 2023-03-03T07:37:28.571 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/eql.ko 2023-03-03T07:37:28.571 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet 2023-03-03T07:37:28.572 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/3com 2023-03-03T07:37:28.572 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-03-03T07:37:28.572 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-03-03T07:37:28.572 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-03-03T07:37:28.572 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-03-03T07:37:28.572 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/8390 2023-03-03T07:37:28.572 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/8390/8390.ko 2023-03-03T07:37:28.572 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-03-03T07:37:28.573 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-03-03T07:37:28.573 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-03-03T07:37:28.573 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/adaptec 2023-03-03T07:37:28.573 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-03-03T07:37:28.573 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/agere 2023-03-03T07:37:28.573 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/agere/et131x.ko 2023-03-03T07:37:28.573 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/alteon 2023-03-03T07:37:28.574 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-03-03T07:37:28.574 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/altera 2023-03-03T07:37:28.574 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-03-03T07:37:28.574 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/amd 2023-03-03T07:37:28.574 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-03-03T07:37:28.574 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-03-03T07:37:28.574 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-03-03T07:37:28.575 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros 2023-03-03T07:37:28.575 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/alx 2023-03-03T07:37:28.575 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-03-03T07:37:28.575 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atl1c 2023-03-03T07:37:28.575 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-03-03T07:37:28.575 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atl1e 2023-03-03T07:37:28.575 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-03-03T07:37:28.576 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atlx 2023-03-03T07:37:28.576 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-03-03T07:37:28.576 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-03-03T07:37:28.576 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom 2023-03-03T07:37:28.576 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-03-03T07:37:28.576 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-03-03T07:37:28.576 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-03-03T07:37:28.576 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-03-03T07:37:28.577 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-03-03T07:37:28.577 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/genet 2023-03-03T07:37:28.577 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-03-03T07:37:28.577 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-03-03T07:37:28.577 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/brocade 2023-03-03T07:37:28.577 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/brocade/bna 2023-03-03T07:37:28.577 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-03-03T07:37:28.577 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio 2023-03-03T07:37:28.578 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb 2023-03-03T07:37:28.578 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-03-03T07:37:28.578 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-03-03T07:37:28.578 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-03-03T07:37:28.578 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-03-03T07:37:28.578 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-03-03T07:37:28.578 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-03-03T07:37:28.579 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-03-03T07:37:28.579 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-03-03T07:37:28.579 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-03-03T07:37:28.579 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/cisco 2023-03-03T07:37:28.579 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/cisco/enic 2023-03-03T07:37:28.579 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-03-03T07:37:28.579 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec 2023-03-03T07:37:28.579 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip 2023-03-03T07:37:28.580 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-03-03T07:37:28.580 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-03-03T07:37:28.580 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-03-03T07:37:28.580 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-03-03T07:37:28.580 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-03-03T07:37:28.580 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-03-03T07:37:28.580 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dlink 2023-03-03T07:37:28.580 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-03-03T07:37:28.581 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-03-03T07:37:28.581 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dnet.ko 2023-03-03T07:37:28.581 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/emulex 2023-03-03T07:37:28.581 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/emulex/benet 2023-03-03T07:37:28.581 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-03-03T07:37:28.581 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/ethoc.ko 2023-03-03T07:37:28.581 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel 2023-03-03T07:37:28.582 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/e100.ko 2023-03-03T07:37:28.582 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/e1000 2023-03-03T07:37:28.582 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-03-03T07:37:28.582 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/e1000e 2023-03-03T07:37:28.582 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-03-03T07:37:28.582 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/fm10k 2023-03-03T07:37:28.582 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-03-03T07:37:28.582 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/i40e 2023-03-03T07:37:28.583 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-03-03T07:37:28.583 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/iavf 2023-03-03T07:37:28.583 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-03-03T07:37:28.583 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/igb 2023-03-03T07:37:28.583 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-03-03T07:37:28.583 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/igbvf 2023-03-03T07:37:28.583 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-03-03T07:37:28.583 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgb 2023-03-03T07:37:28.584 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgb/ixgb.ko 2023-03-03T07:37:28.584 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgbe 2023-03-03T07:37:28.584 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-03-03T07:37:28.584 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgbevf 2023-03-03T07:37:28.584 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-03-03T07:37:28.584 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/jme.ko 2023-03-03T07:37:28.584 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/marvell 2023-03-03T07:37:28.585 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-03-03T07:37:28.585 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/marvell/skge.ko 2023-03-03T07:37:28.585 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-03-03T07:37:28.585 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox 2023-03-03T07:37:28.585 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx4 2023-03-03T07:37:28.585 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-03-03T07:37:28.585 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-03-03T07:37:28.585 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx5 2023-03-03T07:37:28.586 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-03-03T07:37:28.586 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-03-03T07:37:28.586 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/micrel 2023-03-03T07:37:28.586 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-03-03T07:37:28.586 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/myricom 2023-03-03T07:37:28.586 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/myricom/myri10ge 2023-03-03T07:37:28.586 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-03-03T07:37:28.586 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/natsemi 2023-03-03T07:37:28.587 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-03-03T07:37:28.587 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-03-03T07:37:28.587 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/neterion 2023-03-03T07:37:28.587 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-03-03T07:37:28.587 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/nvidia 2023-03-03T07:37:28.587 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-03-03T07:37:28.587 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/packetengines 2023-03-03T07:37:28.587 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-03-03T07:37:28.588 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-03-03T07:37:28.588 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic 2023-03-03T07:37:28.588 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic/netxen 2023-03-03T07:37:28.588 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-03-03T07:37:28.588 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-03-03T07:37:28.588 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-03-03T07:37:28.588 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-03-03T07:37:28.589 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/rdc 2023-03-03T07:37:28.589 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-03-03T07:37:28.589 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/realtek 2023-03-03T07:37:28.589 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-03-03T07:37:28.589 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-03-03T07:37:28.589 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/realtek/atp.ko 2023-03-03T07:37:28.589 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-03-03T07:37:28.589 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/rocker 2023-03-03T07:37:28.590 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-03-03T07:37:28.590 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sfc 2023-03-03T07:37:28.590 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-03-03T07:37:28.590 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/silan 2023-03-03T07:37:28.590 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-03-03T07:37:28.590 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sis 2023-03-03T07:37:28.590 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sis/sis190.ko 2023-03-03T07:37:28.591 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sis/sis900.ko 2023-03-03T07:37:28.591 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/smsc 2023-03-03T07:37:28.591 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-03-03T07:37:28.591 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-03-03T07:37:28.591 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-03-03T07:37:28.591 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-03-03T07:37:28.591 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/stmicro 2023-03-03T07:37:28.591 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/stmicro/stmmac 2023-03-03T07:37:28.592 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-03-03T07:37:28.592 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-03-03T07:37:28.592 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sun 2023-03-03T07:37:28.592 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sun/cassini.ko 2023-03-03T07:37:28.592 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sun/niu.ko 2023-03-03T07:37:28.592 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sun/sungem.ko 2023-03-03T07:37:28.592 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-03-03T07:37:28.592 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/tehuti 2023-03-03T07:37:28.593 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-03-03T07:37:28.593 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/ti 2023-03-03T07:37:28.593 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/ti/tlan.ko 2023-03-03T07:37:28.593 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/via 2023-03-03T07:37:28.593 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-03-03T07:37:28.593 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-03-03T07:37:28.593 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/wiznet 2023-03-03T07:37:28.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-03-03T07:37:28.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-03-03T07:37:28.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/xircom 2023-03-03T07:37:28.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-03-03T07:37:28.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/geneve.ko 2023-03-03T07:37:28.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/hyperv 2023-03-03T07:37:28.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-03-03T07:37:28.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ieee802154 2023-03-03T07:37:28.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ieee802154/fakelb.ko 2023-03-03T07:37:28.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ifb.ko 2023-03-03T07:37:28.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ipvlan 2023-03-03T07:37:28.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ipvlan/ipvlan.ko 2023-03-03T07:37:28.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/macvlan.ko 2023-03-03T07:37:28.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/macvtap.ko 2023-03-03T07:37:28.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/mdio 2023-03-03T07:37:28.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/mdio.ko 2023-03-03T07:37:28.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-03-03T07:37:28.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-03-03T07:37:28.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/mii.ko 2023-03-03T07:37:28.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/net_failover.ko 2023-03-03T07:37:28.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/netconsole.ko 2023-03-03T07:37:28.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/nlmon.ko 2023-03-03T07:37:28.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/pcs 2023-03-03T07:37:28.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/pcs/pcs-altera-tse.ko 2023-03-03T07:37:28.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-03-03T07:37:28.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy 2023-03-03T07:37:28.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/amd.ko 2023-03-03T07:37:28.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-03-03T07:37:28.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-03-03T07:37:28.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/bcm7xxx.ko 2023-03-03T07:37:28.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/bcm87xx.ko 2023-03-03T07:37:28.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/broadcom.ko 2023-03-03T07:37:28.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/cicada.ko 2023-03-03T07:37:28.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/davicom.ko 2023-03-03T07:37:28.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/dp83640.ko 2023-03-03T07:37:28.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/et1011c.ko 2023-03-03T07:37:28.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/icplus.ko 2023-03-03T07:37:28.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/lxt.ko 2023-03-03T07:37:28.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/marvell.ko 2023-03-03T07:37:28.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/micrel.ko 2023-03-03T07:37:28.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/national.ko 2023-03-03T07:37:28.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/phylink.ko 2023-03-03T07:37:28.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/qsemi.ko 2023-03-03T07:37:28.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/realtek.ko 2023-03-03T07:37:28.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/smsc.ko 2023-03-03T07:37:28.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/ste10Xp.ko 2023-03-03T07:37:28.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/vitesse.ko 2023-03-03T07:37:28.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp 2023-03-03T07:37:28.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/bsd_comp.ko 2023-03-03T07:37:28.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/ppp_async.ko 2023-03-03T07:37:28.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/ppp_deflate.ko 2023-03-03T07:37:28.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/ppp_generic.ko 2023-03-03T07:37:28.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/ppp_mppe.ko 2023-03-03T07:37:28.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/ppp_synctty.ko 2023-03-03T07:37:28.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/pppoe.ko 2023-03-03T07:37:28.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/pppox.ko 2023-03-03T07:37:28.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/pptp.ko 2023-03-03T07:37:28.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/slip 2023-03-03T07:37:28.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/slip/slhc.ko 2023-03-03T07:37:28.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/slip/slip.ko 2023-03-03T07:37:28.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/sungem_phy.ko 2023-03-03T07:37:28.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/tap.ko 2023-03-03T07:37:28.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team 2023-03-03T07:37:28.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team.ko 2023-03-03T07:37:28.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team_mode_activebackup.ko 2023-03-03T07:37:28.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team_mode_broadcast.ko 2023-03-03T07:37:28.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-03-03T07:37:28.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team_mode_random.ko 2023-03-03T07:37:28.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-03-03T07:37:28.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/tun.ko 2023-03-03T07:37:28.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/veth.ko 2023-03-03T07:37:28.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/virtio_net.ko 2023-03-03T07:37:28.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/vmxnet3 2023-03-03T07:37:28.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-03-03T07:37:28.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/vxlan 2023-03-03T07:37:28.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/vxlan/vxlan.ko 2023-03-03T07:37:28.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/xen-netback 2023-03-03T07:37:28.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/xen-netback/xen-netback.ko 2023-03-03T07:37:28.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/xen-netfront.ko 2023-03-03T07:37:28.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ntb 2023-03-03T07:37:28.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ntb/ntb.ko 2023-03-03T07:37:28.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/nvme 2023-03-03T07:37:28.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/nvme/host 2023-03-03T07:37:28.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/nvme/host/nvme-core.ko 2023-03-03T07:37:28.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/nvme/host/nvme.ko 2023-03-03T07:37:28.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/parport 2023-03-03T07:37:28.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/parport/parport.ko 2023-03-03T07:37:28.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/parport/parport_cs.ko 2023-03-03T07:37:28.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/parport/parport_pc.ko 2023-03-03T07:37:28.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/parport/parport_serial.ko 2023-03-03T07:37:28.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci 2023-03-03T07:37:28.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci/hotplug 2023-03-03T07:37:28.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-03-03T07:37:28.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci/pcie 2023-03-03T07:37:28.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci/pcie/aer_inject.ko 2023-03-03T07:37:28.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci/xen-pcifront.ko 2023-03-03T07:37:28.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pcmcia 2023-03-03T07:37:28.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pcmcia/i82092.ko 2023-03-03T07:37:28.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pcmcia/pd6729.ko 2023-03-03T07:37:28.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pcmcia/yenta_socket.ko 2023-03-03T07:37:28.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pinctrl 2023-03-03T07:37:28.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pinctrl/intel 2023-03-03T07:37:28.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-03-03T07:37:28.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform 2023-03-03T07:37:28.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86 2023-03-03T07:37:28.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/acer-wmi.ko 2023-03-03T07:37:28.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/acerhdf.ko 2023-03-03T07:37:28.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/apple-gmux.ko 2023-03-03T07:37:28.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/asus-laptop.ko 2023-03-03T07:37:28.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-03-03T07:37:28.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/asus-wmi.ko 2023-03-03T07:37:28.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/classmate-laptop.ko 2023-03-03T07:37:28.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-03-03T07:37:28.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-03-03T07:37:28.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-03-03T07:37:28.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-03-03T07:37:28.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/hdaps.ko 2023-03-03T07:37:28.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/intel 2023-03-03T07:37:28.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-03-03T07:37:28.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/intel_ips.ko 2023-03-03T07:37:28.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/msi-wmi.ko 2023-03-03T07:37:28.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/mxm-wmi.ko 2023-03-03T07:37:28.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-03-03T07:37:28.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/samsung-laptop.ko 2023-03-03T07:37:28.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/samsung-q10.ko 2023-03-03T07:37:28.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/topstar-laptop.ko 2023-03-03T07:37:28.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-03-03T07:37:28.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/toshiba_haps.ko 2023-03-03T07:37:28.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/wmi-bmof.ko 2023-03-03T07:37:28.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/wmi.ko 2023-03-03T07:37:28.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/powercap 2023-03-03T07:37:28.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/powercap/intel_rapl_common.ko 2023-03-03T07:37:28.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/powercap/intel_rapl_msr.ko 2023-03-03T07:37:28.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps 2023-03-03T07:37:28.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps/clients 2023-03-03T07:37:28.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps/clients/pps-gpio.ko 2023-03-03T07:37:28.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps/clients/pps-ldisc.ko 2023-03-03T07:37:28.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps/clients/pps_parport.ko 2023-03-03T07:37:28.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps/pps_core.ko 2023-03-03T07:37:28.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ptp 2023-03-03T07:37:28.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ptp/ptp.ko 2023-03-03T07:37:28.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ptp/ptp_kvm.ko 2023-03-03T07:37:28.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc 2023-03-03T07:37:28.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-bq32k.ko 2023-03-03T07:37:28.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-bq4802.ko 2023-03-03T07:37:28.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1286.ko 2023-03-03T07:37:28.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1307.ko 2023-03-03T07:37:28.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1374.ko 2023-03-03T07:37:28.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1511.ko 2023-03-03T07:37:28.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1553.ko 2023-03-03T07:37:28.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1672.ko 2023-03-03T07:37:28.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1742.ko 2023-03-03T07:37:28.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds2404.ko 2023-03-03T07:37:28.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds3232.ko 2023-03-03T07:37:28.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-em3027.ko 2023-03-03T07:37:28.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-fm3130.ko 2023-03-03T07:37:28.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-isl12022.ko 2023-03-03T07:37:28.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-isl1208.ko 2023-03-03T07:37:28.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-m41t80.ko 2023-03-03T07:37:28.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-m48t35.ko 2023-03-03T07:37:28.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-m48t59.ko 2023-03-03T07:37:28.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-max6900.ko 2023-03-03T07:37:28.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-msm6242.ko 2023-03-03T07:37:28.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-pcf2127.ko 2023-03-03T07:37:28.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-pcf85063.ko 2023-03-03T07:37:28.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-pcf8523.ko 2023-03-03T07:37:28.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-pcf8563.ko 2023-03-03T07:37:28.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-pcf8583.ko 2023-03-03T07:37:28.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-rp5c01.ko 2023-03-03T07:37:28.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-rs5c372.ko 2023-03-03T07:37:28.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-03-03T07:37:28.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-rx8025.ko 2023-03-03T07:37:28.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-rx8581.ko 2023-03-03T07:37:28.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-03-03T07:37:28.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-v3020.ko 2023-03-03T07:37:28.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-x1205.ko 2023-03-03T07:37:28.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi 2023-03-03T07:37:28.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/3w-9xxx.ko 2023-03-03T07:37:28.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/3w-sas.ko 2023-03-03T07:37:28.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/3w-xxxx.ko 2023-03-03T07:37:28.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/BusLogic.ko 2023-03-03T07:37:28.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/a100u2w.ko 2023-03-03T07:37:28.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/aacraid 2023-03-03T07:37:28.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/aacraid/aacraid.ko 2023-03-03T07:37:28.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/advansys.ko 2023-03-03T07:37:28.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/aic7xxx 2023-03-03T07:37:28.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-03-03T07:37:28.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-03-03T07:37:28.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/am53c974.ko 2023-03-03T07:37:28.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/arcmsr 2023-03-03T07:37:28.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-03-03T07:37:28.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/atp870u.ko 2023-03-03T07:37:28.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/be2iscsi 2023-03-03T07:37:28.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-03-03T07:37:28.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bfa 2023-03-03T07:37:28.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bfa/bfa.ko 2023-03-03T07:37:28.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bnx2fc 2023-03-03T07:37:28.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-03-03T07:37:28.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bnx2i 2023-03-03T07:37:28.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-03-03T07:37:28.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/ch.ko 2023-03-03T07:37:28.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/csiostor 2023-03-03T07:37:28.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/csiostor/csiostor.ko 2023-03-03T07:37:28.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi 2023-03-03T07:37:28.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi/cxgb3i 2023-03-03T07:37:28.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-03-03T07:37:28.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi/cxgb4i 2023-03-03T07:37:28.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-03-03T07:37:28.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-03-03T07:37:28.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/dc395x.ko 2023-03-03T07:37:28.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/device_handler 2023-03-03T07:37:28.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-03-03T07:37:28.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-03-03T07:37:28.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-03-03T07:37:28.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-03-03T07:37:28.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/dmx3191d.ko 2023-03-03T07:37:28.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/esas2r 2023-03-03T07:37:28.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/esas2r/esas2r.ko 2023-03-03T07:37:28.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/esp_scsi.ko 2023-03-03T07:37:28.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/fcoe 2023-03-03T07:37:28.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/fcoe/fcoe.ko 2023-03-03T07:37:28.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-03-03T07:37:28.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/fnic 2023-03-03T07:37:28.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/fnic/fnic.ko 2023-03-03T07:37:28.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/hpsa.ko 2023-03-03T07:37:28.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/hptiop.ko 2023-03-03T07:37:28.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/hv_storvsc.ko 2023-03-03T07:37:28.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/initio.ko 2023-03-03T07:37:28.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/ipr.ko 2023-03-03T07:37:28.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/ips.ko 2023-03-03T07:37:28.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/isci 2023-03-03T07:37:28.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/isci/isci.ko 2023-03-03T07:37:28.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-03-03T07:37:28.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/iscsi_tcp.ko 2023-03-03T07:37:28.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libfc 2023-03-03T07:37:28.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libfc/libfc.ko 2023-03-03T07:37:28.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libiscsi.ko 2023-03-03T07:37:28.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libiscsi_tcp.ko 2023-03-03T07:37:28.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libsas 2023-03-03T07:37:28.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libsas/libsas.ko 2023-03-03T07:37:28.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/lpfc 2023-03-03T07:37:28.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/lpfc/lpfc.ko 2023-03-03T07:37:28.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/megaraid 2023-03-03T07:37:28.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/megaraid.ko 2023-03-03T07:37:28.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-03-03T07:37:28.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-03-03T07:37:28.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-03-03T07:37:28.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/mpt3sas 2023-03-03T07:37:28.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-03-03T07:37:28.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/mvsas 2023-03-03T07:37:28.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/mvsas/mvsas.ko 2023-03-03T07:37:28.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/mvumi.ko 2023-03-03T07:37:28.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/pm8001 2023-03-03T07:37:28.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-03-03T07:37:28.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/pmcraid.ko 2023-03-03T07:37:28.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla1280.ko 2023-03-03T07:37:28.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla2xxx 2023-03-03T07:37:28.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-03-03T07:37:28.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-03-03T07:37:28.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla4xxx 2023-03-03T07:37:28.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-03-03T07:37:28.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/raid_class.ko 2023-03-03T07:37:28.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_debug.ko 2023-03-03T07:37:28.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_transport_fc.ko 2023-03-03T07:37:28.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-03-03T07:37:28.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_transport_sas.ko 2023-03-03T07:37:28.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_transport_spi.ko 2023-03-03T07:37:28.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_transport_srp.ko 2023-03-03T07:37:28.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/ses.ko 2023-03-03T07:37:28.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/st.ko 2023-03-03T07:37:28.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/stex.ko 2023-03-03T07:37:28.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/sym53c8xx_2 2023-03-03T07:37:28.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-03-03T07:37:28.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/virtio_scsi.ko 2023-03-03T07:37:28.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/vmw_pvscsi.ko 2023-03-03T07:37:28.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/wd719x.ko 2023-03-03T07:37:28.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ssb 2023-03-03T07:37:28.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ssb/ssb.ko 2023-03-03T07:37:28.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target 2023-03-03T07:37:28.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/iscsi 2023-03-03T07:37:28.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-03-03T07:37:28.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/loopback 2023-03-03T07:37:28.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/loopback/tcm_loop.ko 2023-03-03T07:37:28.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/target_core_file.ko 2023-03-03T07:37:28.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/target_core_iblock.ko 2023-03-03T07:37:28.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/target_core_mod.ko 2023-03-03T07:37:28.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/target_core_pscsi.ko 2023-03-03T07:37:28.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/target_core_user.ko 2023-03-03T07:37:28.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/tcm_fc 2023-03-03T07:37:28.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-03-03T07:37:28.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal 2023-03-03T07:37:28.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel 2023-03-03T07:37:28.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal 2023-03-03T07:37:28.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-03-03T07:37:28.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-03-03T07:37:28.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-03-03T07:37:28.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-03-03T07:37:28.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-03-03T07:37:28.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-03-03T07:37:28.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-03-03T07:37:28.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-03-03T07:37:28.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-03-03T07:37:28.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-03-03T07:37:28.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-03-03T07:37:28.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-03-03T07:37:28.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-03-03T07:37:28.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-03-03T07:37:28.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-03-03T07:37:28.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty 2023-03-03T07:37:28.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/ipwireless 2023-03-03T07:37:28.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-03-03T07:37:28.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/n_gsm.ko 2023-03-03T07:37:28.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/n_hdlc.ko 2023-03-03T07:37:28.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/nozomi.ko 2023-03-03T07:37:28.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial 2023-03-03T07:37:28.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial/8250 2023-03-03T07:37:28.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-03-03T07:37:28.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial/arc_uart.ko 2023-03-03T07:37:28.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial/jsm 2023-03-03T07:37:28.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial/jsm/jsm.ko 2023-03-03T07:37:28.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/synclink_gt.ko 2023-03-03T07:37:28.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ufs 2023-03-03T07:37:28.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ufs/core 2023-03-03T07:37:28.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ufs/core/ufshcd-core.ko 2023-03-03T07:37:28.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ufs/host 2023-03-03T07:37:28.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-03-03T07:37:28.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio 2023-03-03T07:37:28.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio/uio.ko 2023-03-03T07:37:28.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio/uio_aec.ko 2023-03-03T07:37:28.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio/uio_cif.ko 2023-03-03T07:37:28.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio/uio_pci_generic.ko 2023-03-03T07:37:28.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio/uio_sercos3.ko 2023-03-03T07:37:28.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb 2023-03-03T07:37:28.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/class 2023-03-03T07:37:28.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/class/cdc-acm.ko 2023-03-03T07:37:28.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/class/cdc-wdm.ko 2023-03-03T07:37:28.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/class/usblp.ko 2023-03-03T07:37:28.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/class/usbtmc.ko 2023-03-03T07:37:28.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/host 2023-03-03T07:37:28.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/host/sl811-hcd.ko 2023-03-03T07:37:28.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/host/u132-hcd.ko 2023-03-03T07:37:28.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/image 2023-03-03T07:37:28.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/image/mdc800.ko 2023-03-03T07:37:28.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/image/microtek.ko 2023-03-03T07:37:28.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc 2023-03-03T07:37:28.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/adutux.ko 2023-03-03T07:37:28.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/appledisplay.ko 2023-03-03T07:37:28.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/emi26.ko 2023-03-03T07:37:28.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/emi62.ko 2023-03-03T07:37:28.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/ezusb.ko 2023-03-03T07:37:28.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/ftdi-elan.ko 2023-03-03T07:37:28.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/idmouse.ko 2023-03-03T07:37:28.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/iowarrior.ko 2023-03-03T07:37:28.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/isight_firmware.ko 2023-03-03T07:37:28.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/ldusb.ko 2023-03-03T07:37:28.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/legousbtower.ko 2023-03-03T07:37:28.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/sisusbvga 2023-03-03T07:37:28.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-03-03T07:37:28.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/trancevibrator.ko 2023-03-03T07:37:28.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/usb3503.ko 2023-03-03T07:37:28.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/usblcd.ko 2023-03-03T07:37:28.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/usbsevseg.ko 2023-03-03T07:37:28.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/uss720.ko 2023-03-03T07:37:28.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/yurex.ko 2023-03-03T07:37:28.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/phy 2023-03-03T07:37:28.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/phy/phy-generic.ko 2023-03-03T07:37:28.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial 2023-03-03T07:37:28.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/aircable.ko 2023-03-03T07:37:28.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ark3116.ko 2023-03-03T07:37:28.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/belkin_sa.ko 2023-03-03T07:37:28.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ch341.ko 2023-03-03T07:37:28.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/cp210x.ko 2023-03-03T07:37:28.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/cyberjack.ko 2023-03-03T07:37:28.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/cypress_m8.ko 2023-03-03T07:37:28.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/digi_acceleport.ko 2023-03-03T07:37:28.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/empeg.ko 2023-03-03T07:37:28.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ftdi_sio.ko 2023-03-03T07:37:28.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/garmin_gps.ko 2023-03-03T07:37:28.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/io_edgeport.ko 2023-03-03T07:37:28.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/io_ti.ko 2023-03-03T07:37:28.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ipaq.ko 2023-03-03T07:37:28.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ipw.ko 2023-03-03T07:37:28.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ir-usb.ko 2023-03-03T07:37:28.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-03-03T07:37:28.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/keyspan.ko 2023-03-03T07:37:28.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/keyspan_pda.ko 2023-03-03T07:37:28.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/kl5kusb105.ko 2023-03-03T07:37:28.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/kobil_sct.ko 2023-03-03T07:37:28.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/mct_u232.ko 2023-03-03T07:37:28.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/mos7720.ko 2023-03-03T07:37:28.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/mos7840.ko 2023-03-03T07:37:28.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/navman.ko 2023-03-03T07:37:28.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/omninet.ko 2023-03-03T07:37:28.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/opticon.ko 2023-03-03T07:37:28.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/option.ko 2023-03-03T07:37:28.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/oti6858.ko 2023-03-03T07:37:28.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/pl2303.ko 2023-03-03T07:37:28.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/qcaux.ko 2023-03-03T07:37:28.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/qcserial.ko 2023-03-03T07:37:28.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/quatech2.ko 2023-03-03T07:37:28.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/safe_serial.ko 2023-03-03T07:37:28.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/sierra.ko 2023-03-03T07:37:28.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/spcp8x5.ko 2023-03-03T07:37:28.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ssu100.ko 2023-03-03T07:37:28.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/symbolserial.ko 2023-03-03T07:37:28.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-03-03T07:37:28.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-03-03T07:37:28.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/usb_debug.ko 2023-03-03T07:37:28.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/usb_wwan.ko 2023-03-03T07:37:28.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/visor.ko 2023-03-03T07:37:28.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/whiteheat.ko 2023-03-03T07:37:28.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/xsens_mt.ko 2023-03-03T07:37:28.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage 2023-03-03T07:37:28.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/uas.ko 2023-03-03T07:37:28.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-alauda.ko 2023-03-03T07:37:28.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-cypress.ko 2023-03-03T07:37:28.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-datafab.ko 2023-03-03T07:37:28.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-03-03T07:37:28.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-freecom.ko 2023-03-03T07:37:28.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-isd200.ko 2023-03-03T07:37:28.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-03-03T07:37:28.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-karma.ko 2023-03-03T07:37:28.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-onetouch.ko 2023-03-03T07:37:28.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-realtek.ko 2023-03-03T07:37:28.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-sddr09.ko 2023-03-03T07:37:28.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-sddr55.ko 2023-03-03T07:37:28.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-usbat.ko 2023-03-03T07:37:28.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/usb-storage.ko 2023-03-03T07:37:28.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/usbip 2023-03-03T07:37:28.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/usbip/usbip-core.ko 2023-03-03T07:37:28.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/usbip/usbip-host.ko 2023-03-03T07:37:28.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-03-03T07:37:28.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio 2023-03-03T07:37:28.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio/pci 2023-03-03T07:37:28.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-03-03T07:37:28.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio/pci/vfio-pci.ko 2023-03-03T07:37:28.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio/vfio.ko 2023-03-03T07:37:28.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-03-03T07:37:28.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vhost 2023-03-03T07:37:28.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vhost/vhost.ko 2023-03-03T07:37:28.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vhost/vhost_iotlb.ko 2023-03-03T07:37:28.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vhost/vhost_net.ko 2023-03-03T07:37:28.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vhost/vhost_scsi.ko 2023-03-03T07:37:28.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/video 2023-03-03T07:37:28.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/video/backlight 2023-03-03T07:37:28.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/video/backlight/apple_bl.ko 2023-03-03T07:37:28.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/video/backlight/lcd.ko 2023-03-03T07:37:28.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/video/backlight/platform_lcd.ko 2023-03-03T07:37:28.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio 2023-03-03T07:37:28.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio.ko 2023-03-03T07:37:28.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_balloon.ko 2023-03-03T07:37:28.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_mmio.ko 2023-03-03T07:37:28.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_pci.ko 2023-03-03T07:37:28.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-03-03T07:37:28.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-03-03T07:37:28.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_ring.ko 2023-03-03T07:37:28.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1 2023-03-03T07:37:28.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/masters 2023-03-03T07:37:28.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/masters/ds1wm.ko 2023-03-03T07:37:28.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/masters/ds2482.ko 2023-03-03T07:37:28.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/masters/ds2490.ko 2023-03-03T07:37:28.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves 2023-03-03T07:37:28.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-03-03T07:37:28.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-03-03T07:37:28.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-03-03T07:37:28.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-03-03T07:37:28.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-03-03T07:37:28.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-03-03T07:37:28.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-03-03T07:37:28.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-03-03T07:37:28.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-03-03T07:37:28.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_smem.ko 2023-03-03T07:37:28.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_therm.ko 2023-03-03T07:37:28.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/wire.ko 2023-03-03T07:37:28.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog 2023-03-03T07:37:28.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/alim1535_wdt.ko 2023-03-03T07:37:28.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/alim7101_wdt.ko 2023-03-03T07:37:28.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/f71808e_wdt.ko 2023-03-03T07:37:28.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/hpwdt.ko 2023-03-03T07:37:28.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/i6300esb.ko 2023-03-03T07:37:28.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-03-03T07:37:28.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/iTCO_wdt.ko 2023-03-03T07:37:28.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/ib700wdt.ko 2023-03-03T07:37:28.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/ibmasr.ko 2023-03-03T07:37:28.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-03-03T07:37:28.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/it8712f_wdt.ko 2023-03-03T07:37:28.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/it87_wdt.ko 2023-03-03T07:37:28.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/machzwd.ko 2023-03-03T07:37:28.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/nv_tco.ko 2023-03-03T07:37:28.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/pcwd_pci.ko 2023-03-03T07:37:28.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/pcwd_usb.ko 2023-03-03T07:37:28.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-03-03T07:37:28.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/sch311x_wdt.ko 2023-03-03T07:37:28.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/softdog.ko 2023-03-03T07:37:28.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/sp5100_tco.ko 2023-03-03T07:37:28.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/via_wdt.ko 2023-03-03T07:37:28.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-03-03T07:37:28.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/w83877f_wdt.ko 2023-03-03T07:37:28.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/w83977f_wdt.ko 2023-03-03T07:37:28.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/wdt_pci.ko 2023-03-03T07:37:28.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/xen_wdt.ko 2023-03-03T07:37:28.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen 2023-03-03T07:37:28.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-acpi-processor.ko 2023-03-03T07:37:28.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-evtchn.ko 2023-03-03T07:37:28.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-gntalloc.ko 2023-03-03T07:37:28.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-gntdev.ko 2023-03-03T07:37:28.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-pciback 2023-03-03T07:37:28.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-03-03T07:37:28.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-privcmd.ko 2023-03-03T07:37:28.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xenfs 2023-03-03T07:37:28.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xenfs/xenfs.ko 2023-03-03T07:37:28.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs 2023-03-03T07:37:28.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/9p 2023-03-03T07:37:28.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/9p/9p.ko 2023-03-03T07:37:28.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/affs 2023-03-03T07:37:28.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/affs/affs.ko 2023-03-03T07:37:28.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/befs 2023-03-03T07:37:28.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/befs/befs.ko 2023-03-03T07:37:28.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/binfmt_misc.ko 2023-03-03T07:37:28.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/btrfs 2023-03-03T07:37:28.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/btrfs/btrfs.ko 2023-03-03T07:37:28.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cachefiles 2023-03-03T07:37:28.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cachefiles/cachefiles.ko 2023-03-03T07:37:28.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ceph 2023-03-03T07:37:28.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ceph/ceph.ko 2023-03-03T07:37:28.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cifs 2023-03-03T07:37:28.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cifs/cifs.ko 2023-03-03T07:37:28.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/coda 2023-03-03T07:37:28.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/coda/coda.ko 2023-03-03T07:37:28.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cramfs 2023-03-03T07:37:28.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cramfs/cramfs.ko 2023-03-03T07:37:28.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/dlm 2023-03-03T07:37:28.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/dlm/dlm.ko 2023-03-03T07:37:28.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ecryptfs 2023-03-03T07:37:28.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ecryptfs/ecryptfs.ko 2023-03-03T07:37:28.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/f2fs 2023-03-03T07:37:28.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/f2fs/f2fs.ko 2023-03-03T07:37:28.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fat 2023-03-03T07:37:28.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fat/fat.ko 2023-03-03T07:37:28.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fat/msdos.ko 2023-03-03T07:37:28.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fat/vfat.ko 2023-03-03T07:37:28.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fscache 2023-03-03T07:37:28.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fscache/fscache.ko 2023-03-03T07:37:28.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fuse 2023-03-03T07:37:28.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fuse/cuse.ko 2023-03-03T07:37:28.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fuse/fuse.ko 2023-03-03T07:37:28.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/gfs2 2023-03-03T07:37:28.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/gfs2/gfs2.ko 2023-03-03T07:37:28.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/hfs 2023-03-03T07:37:28.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/hfs/hfs.ko 2023-03-03T07:37:28.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/hfsplus 2023-03-03T07:37:28.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/hfsplus/hfsplus.ko 2023-03-03T07:37:28.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/isofs 2023-03-03T07:37:28.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/isofs/isofs.ko 2023-03-03T07:37:28.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/jfs 2023-03-03T07:37:28.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/jfs/jfs.ko 2023-03-03T07:37:28.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/lockd 2023-03-03T07:37:28.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/lockd/lockd.ko 2023-03-03T07:37:28.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/minix 2023-03-03T07:37:28.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/minix/minix.ko 2023-03-03T07:37:28.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/netfs 2023-03-03T07:37:28.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/netfs/netfs.ko 2023-03-03T07:37:28.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs 2023-03-03T07:37:28.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/blocklayout 2023-03-03T07:37:28.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-03-03T07:37:28.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/filelayout 2023-03-03T07:37:28.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-03-03T07:37:28.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/flexfilelayout 2023-03-03T07:37:28.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-03-03T07:37:28.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/nfs.ko 2023-03-03T07:37:28.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/nfsv3.ko 2023-03-03T07:37:28.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/nfsv4.ko 2023-03-03T07:37:28.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs_common 2023-03-03T07:37:28.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs_common/grace.ko 2023-03-03T07:37:28.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs_common/nfs_acl.ko 2023-03-03T07:37:28.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfsd 2023-03-03T07:37:28.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfsd/nfsd.ko 2023-03-03T07:37:28.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nilfs2 2023-03-03T07:37:28.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nilfs2/nilfs2.ko 2023-03-03T07:37:28.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls 2023-03-03T07:37:28.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-celtic.ko 2023-03-03T07:37:28.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-centeuro.ko 2023-03-03T07:37:28.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-croatian.ko 2023-03-03T07:37:28.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-cyrillic.ko 2023-03-03T07:37:28.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-gaelic.ko 2023-03-03T07:37:28.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-greek.ko 2023-03-03T07:37:28.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-iceland.ko 2023-03-03T07:37:28.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-inuit.ko 2023-03-03T07:37:28.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-roman.ko 2023-03-03T07:37:28.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-romanian.ko 2023-03-03T07:37:28.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-turkish.ko 2023-03-03T07:37:28.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp1250.ko 2023-03-03T07:37:28.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp1251.ko 2023-03-03T07:37:28.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp1255.ko 2023-03-03T07:37:28.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp737.ko 2023-03-03T07:37:28.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp775.ko 2023-03-03T07:37:28.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp850.ko 2023-03-03T07:37:28.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp852.ko 2023-03-03T07:37:28.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp855.ko 2023-03-03T07:37:28.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp857.ko 2023-03-03T07:37:28.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp860.ko 2023-03-03T07:37:28.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp861.ko 2023-03-03T07:37:28.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp862.ko 2023-03-03T07:37:28.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp863.ko 2023-03-03T07:37:28.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp864.ko 2023-03-03T07:37:28.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp865.ko 2023-03-03T07:37:28.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp866.ko 2023-03-03T07:37:28.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp869.ko 2023-03-03T07:37:28.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp874.ko 2023-03-03T07:37:28.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp932.ko 2023-03-03T07:37:28.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp936.ko 2023-03-03T07:37:28.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp949.ko 2023-03-03T07:37:28.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp950.ko 2023-03-03T07:37:28.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_euc-jp.ko 2023-03-03T07:37:28.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-1.ko 2023-03-03T07:37:28.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-13.ko 2023-03-03T07:37:28.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-14.ko 2023-03-03T07:37:28.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-15.ko 2023-03-03T07:37:28.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-2.ko 2023-03-03T07:37:28.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-3.ko 2023-03-03T07:37:28.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-4.ko 2023-03-03T07:37:28.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-5.ko 2023-03-03T07:37:28.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-6.ko 2023-03-03T07:37:28.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-7.ko 2023-03-03T07:37:28.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-9.ko 2023-03-03T07:37:28.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_koi8-r.ko 2023-03-03T07:37:28.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_koi8-ru.ko 2023-03-03T07:37:28.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_koi8-u.ko 2023-03-03T07:37:28.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_utf8.ko 2023-03-03T07:37:28.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2 2023-03-03T07:37:28.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/cluster 2023-03-03T07:37:28.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-03-03T07:37:28.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/dlm 2023-03-03T07:37:28.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-03-03T07:37:28.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/dlmfs 2023-03-03T07:37:28.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-03-03T07:37:28.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/ocfs2.ko 2023-03-03T07:37:28.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-03-03T07:37:28.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-03-03T07:37:28.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-03-03T07:37:28.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/overlayfs 2023-03-03T07:37:28.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/overlayfs/overlay.ko 2023-03-03T07:37:28.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/pstore 2023-03-03T07:37:28.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/pstore/ramoops.ko 2023-03-03T07:37:28.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/reiserfs 2023-03-03T07:37:28.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/reiserfs/reiserfs.ko 2023-03-03T07:37:28.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/romfs 2023-03-03T07:37:28.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/romfs/romfs.ko 2023-03-03T07:37:28.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/smbfs_common 2023-03-03T07:37:28.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/smbfs_common/cifs_arc4.ko 2023-03-03T07:37:28.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/smbfs_common/cifs_md4.ko 2023-03-03T07:37:28.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/squashfs 2023-03-03T07:37:28.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/squashfs/squashfs.ko 2023-03-03T07:37:28.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/sysv 2023-03-03T07:37:28.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/sysv/sysv.ko 2023-03-03T07:37:28.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/udf 2023-03-03T07:37:28.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/udf/udf.ko 2023-03-03T07:37:28.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ufs 2023-03-03T07:37:28.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ufs/ufs.ko 2023-03-03T07:37:28.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/xfs 2023-03-03T07:37:28.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/xfs/xfs.ko 2023-03-03T07:37:28.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel 2023-03-03T07:37:28.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel/rcu 2023-03-03T07:37:28.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel/rcu/rcutorture.ko 2023-03-03T07:37:28.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel/torture.ko 2023-03-03T07:37:28.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel/trace 2023-03-03T07:37:28.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-03-03T07:37:28.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib 2023-03-03T07:37:28.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/asn1_encoder.ko 2023-03-03T07:37:28.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/crc-itu-t.ko 2023-03-03T07:37:28.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/crc8.ko 2023-03-03T07:37:28.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/crypto 2023-03-03T07:37:28.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/crypto/libarc4.ko 2023-03-03T07:37:28.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/crypto/libdes.ko 2023-03-03T07:37:28.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/libcrc32c.ko 2023-03-03T07:37:28.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/lru_cache.ko 2023-03-03T07:37:28.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/lz4 2023-03-03T07:37:28.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/lz4/lz4_compress.ko 2023-03-03T07:37:28.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/lz4/lz4hc_compress.ko 2023-03-03T07:37:28.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/math 2023-03-03T07:37:28.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/math/cordic.ko 2023-03-03T07:37:28.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/raid6 2023-03-03T07:37:28.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/raid6/raid6_pq.ko 2023-03-03T07:37:28.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/reed_solomon 2023-03-03T07:37:28.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/reed_solomon/reed_solomon.ko 2023-03-03T07:37:28.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/ts_bm.ko 2023-03-03T07:37:28.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/ts_fsm.ko 2023-03-03T07:37:28.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/ts_kmp.ko 2023-03-03T07:37:28.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/zstd 2023-03-03T07:37:28.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/zstd/zstd_compress.ko 2023-03-03T07:37:28.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/mm 2023-03-03T07:37:28.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/mm/hwpoison-inject.ko 2023-03-03T07:37:28.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net 2023-03-03T07:37:28.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan 2023-03-03T07:37:28.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/6lowpan.ko 2023-03-03T07:37:28.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_dest.ko 2023-03-03T07:37:28.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_fragment.ko 2023-03-03T07:37:28.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_hop.ko 2023-03-03T07:37:28.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_ipv6.ko 2023-03-03T07:37:28.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_mobility.ko 2023-03-03T07:37:28.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_routing.ko 2023-03-03T07:37:28.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_udp.ko 2023-03-03T07:37:28.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802 2023-03-03T07:37:28.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802/garp.ko 2023-03-03T07:37:28.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802/mrp.ko 2023-03-03T07:37:28.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802/p8022.ko 2023-03-03T07:37:28.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802/psnap.ko 2023-03-03T07:37:28.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802/stp.ko 2023-03-03T07:37:28.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/8021q 2023-03-03T07:37:28.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/8021q/8021q.ko 2023-03-03T07:37:28.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/9p 2023-03-03T07:37:28.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/9p/9pnet.ko 2023-03-03T07:37:28.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/9p/9pnet_fd.ko 2023-03-03T07:37:28.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/9p/9pnet_rdma.ko 2023-03-03T07:37:28.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/9p/9pnet_virtio.ko 2023-03-03T07:37:28.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/batman-adv 2023-03-03T07:37:28.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/batman-adv/batman-adv.ko 2023-03-03T07:37:28.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge 2023-03-03T07:37:28.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/br_netfilter.ko 2023-03-03T07:37:28.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/bridge.ko 2023-03-03T07:37:28.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter 2023-03-03T07:37:28.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-03-03T07:37:28.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_among.ko 2023-03-03T07:37:28.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_arp.ko 2023-03-03T07:37:28.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-03-03T07:37:28.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-03-03T07:37:28.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_ip.ko 2023-03-03T07:37:28.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-03-03T07:37:28.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_limit.ko 2023-03-03T07:37:28.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_log.ko 2023-03-03T07:37:28.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_mark.ko 2023-03-03T07:37:28.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-03-03T07:37:28.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-03-03T07:37:28.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-03-03T07:37:28.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-03-03T07:37:28.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_snat.ko 2023-03-03T07:37:28.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_stp.ko 2023-03-03T07:37:28.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-03-03T07:37:28.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-03-03T07:37:28.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-03-03T07:37:28.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-03-03T07:37:28.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebtables.ko 2023-03-03T07:37:28.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-03-03T07:37:28.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-03-03T07:37:28.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-03-03T07:37:28.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ceph 2023-03-03T07:37:28.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ceph/libceph.ko 2023-03-03T07:37:28.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/core 2023-03-03T07:37:28.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/core/failover.ko 2023-03-03T07:37:28.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/core/pktgen.ko 2023-03-03T07:37:28.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dccp 2023-03-03T07:37:28.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dccp/dccp.ko 2023-03-03T07:37:28.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dccp/dccp_diag.ko 2023-03-03T07:37:28.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dccp/dccp_ipv4.ko 2023-03-03T07:37:28.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dccp/dccp_ipv6.ko 2023-03-03T07:37:28.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dns_resolver 2023-03-03T07:37:28.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dns_resolver/dns_resolver.ko 2023-03-03T07:37:28.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa 2023-03-03T07:37:28.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa/dsa_core.ko 2023-03-03T07:37:28.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa/tag_brcm.ko 2023-03-03T07:37:28.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa/tag_dsa.ko 2023-03-03T07:37:28.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa/tag_none.ko 2023-03-03T07:37:28.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa/tag_trailer.ko 2023-03-03T07:37:28.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ieee802154 2023-03-03T07:37:28.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ieee802154/6lowpan 2023-03-03T07:37:28.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-03-03T07:37:28.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ieee802154/ieee802154.ko 2023-03-03T07:37:28.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ieee802154/ieee802154_socket.ko 2023-03-03T07:37:28.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4 2023-03-03T07:37:28.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ah4.ko 2023-03-03T07:37:28.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/esp4.ko 2023-03-03T07:37:28.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/fou.ko 2023-03-03T07:37:28.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/gre.ko 2023-03-03T07:37:28.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/inet_diag.ko 2023-03-03T07:37:28.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ip_gre.ko 2023-03-03T07:37:28.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ip_tunnel.ko 2023-03-03T07:37:28.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ip_vti.ko 2023-03-03T07:37:28.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ipcomp.ko 2023-03-03T07:37:28.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ipip.ko 2023-03-03T07:37:28.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter 2023-03-03T07:37:28.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/arp_tables.ko 2023-03-03T07:37:28.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-03-03T07:37:28.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-03-03T07:37:28.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/ipt_CLUSTERIP.ko 2023-03-03T07:37:28.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-03-03T07:37:28.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-03-03T07:37:28.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-03-03T07:37:28.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-03-03T07:37:28.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-03-03T07:37:28.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-03-03T07:37:28.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-03-03T07:37:28.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/iptable_security.ko 2023-03-03T07:37:28.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-03-03T07:37:28.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-03-03T07:37:28.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-03-03T07:37:28.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-03-03T07:37:28.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-03-03T07:37:28.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-03-03T07:37:28.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-03-03T07:37:28.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-03-03T07:37:28.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-03-03T07:37:28.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-03-03T07:37:28.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_bic.ko 2023-03-03T07:37:28.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_dctcp.ko 2023-03-03T07:37:28.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_diag.ko 2023-03-03T07:37:28.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_highspeed.ko 2023-03-03T07:37:28.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_htcp.ko 2023-03-03T07:37:28.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_hybla.ko 2023-03-03T07:37:28.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_illinois.ko 2023-03-03T07:37:28.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_lp.ko 2023-03-03T07:37:28.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_scalable.ko 2023-03-03T07:37:28.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_vegas.ko 2023-03-03T07:37:28.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_veno.ko 2023-03-03T07:37:28.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_westwood.ko 2023-03-03T07:37:28.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_yeah.ko 2023-03-03T07:37:28.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tunnel4.ko 2023-03-03T07:37:28.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/udp_diag.ko 2023-03-03T07:37:28.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/udp_tunnel.ko 2023-03-03T07:37:28.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/xfrm4_tunnel.ko 2023-03-03T07:37:28.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6 2023-03-03T07:37:28.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/ah6.ko 2023-03-03T07:37:28.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/esp6.ko 2023-03-03T07:37:28.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/fou6.ko 2023-03-03T07:37:28.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/ip6_tunnel.ko 2023-03-03T07:37:28.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-03-03T07:37:28.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/ip6_vti.ko 2023-03-03T07:37:28.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/ipcomp6.ko 2023-03-03T07:37:28.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter 2023-03-03T07:37:28.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-03-03T07:37:28.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-03-03T07:37:28.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-03-03T07:37:28.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-03-03T07:37:28.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-03-03T07:37:28.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-03-03T07:37:28.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-03-03T07:37:28.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-03-03T07:37:28.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-03-03T07:37:28.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-03-03T07:37:28.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-03-03T07:37:28.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-03-03T07:37:28.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-03-03T07:37:28.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-03-03T07:37:28.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-03-03T07:37:28.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-03-03T07:37:28.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-03-03T07:37:28.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-03-03T07:37:28.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-03-03T07:37:28.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-03-03T07:37:28.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-03-03T07:37:28.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-03-03T07:37:28.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-03-03T07:37:28.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-03-03T07:37:28.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/sit.ko 2023-03-03T07:37:28.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/tunnel6.ko 2023-03-03T07:37:28.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/xfrm6_tunnel.ko 2023-03-03T07:37:28.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/key 2023-03-03T07:37:28.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/key/af_key.ko 2023-03-03T07:37:28.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp 2023-03-03T07:37:28.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_core.ko 2023-03-03T07:37:28.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_debugfs.ko 2023-03-03T07:37:28.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_eth.ko 2023-03-03T07:37:28.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_ip.ko 2023-03-03T07:37:28.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_ip6.ko 2023-03-03T07:37:28.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_netlink.ko 2023-03-03T07:37:28.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_ppp.ko 2023-03-03T07:37:28.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/llc 2023-03-03T07:37:28.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/llc/llc.ko 2023-03-03T07:37:28.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/mac802154 2023-03-03T07:37:28.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/mac802154/mac802154.ko 2023-03-03T07:37:28.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/mpls 2023-03-03T07:37:28.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/mpls/mpls_gso.ko 2023-03-03T07:37:28.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter 2023-03-03T07:37:28.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset 2023-03-03T07:37:28.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set.ko 2023-03-03T07:37:28.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-03-03T07:37:28.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-03-03T07:37:28.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-03-03T07:37:28.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-03-03T07:37:28.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-03-03T07:37:28.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-03-03T07:37:28.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-03-03T07:37:28.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-03-03T07:37:28.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-03-03T07:37:28.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-03-03T07:37:28.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-03-03T07:37:28.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-03-03T07:37:28.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-03-03T07:37:28.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-03-03T07:37:28.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-03-03T07:37:28.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs 2023-03-03T07:37:28.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs.ko 2023-03-03T07:37:28.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-03-03T07:37:28.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-03-03T07:37:28.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-03-03T07:37:28.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-03-03T07:37:28.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-03-03T07:37:28.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-03-03T07:37:28.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-03-03T07:37:28.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-03-03T07:37:28.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-03-03T07:37:28.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-03-03T07:37:28.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-03-03T07:37:28.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-03-03T07:37:28.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-03-03T07:37:28.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conncount.ko 2023-03-03T07:37:28.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack.ko 2023-03-03T07:37:28.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-03-03T07:37:28.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-03-03T07:37:28.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-03-03T07:37:28.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_h323.ko 2023-03-03T07:37:28.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_irc.ko 2023-03-03T07:37:28.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-03-03T07:37:28.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-03-03T07:37:28.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-03-03T07:37:28.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_sane.ko 2023-03-03T07:37:28.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_sip.ko 2023-03-03T07:37:28.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-03-03T07:37:28.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-03-03T07:37:28.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_dup_netdev.ko 2023-03-03T07:37:28.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_flow_table.ko 2023-03-03T07:37:28.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_flow_table_inet.ko 2023-03-03T07:37:28.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_log_syslog.ko 2023-03-03T07:37:28.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat.ko 2023-03-03T07:37:28.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat_amanda.ko 2023-03-03T07:37:28.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat_ftp.ko 2023-03-03T07:37:28.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat_irc.ko 2023-03-03T07:37:28.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat_sip.ko 2023-03-03T07:37:28.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat_tftp.ko 2023-03-03T07:37:28.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_synproxy_core.ko 2023-03-03T07:37:28.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_tables.ko 2023-03-03T07:37:28.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nfnetlink.ko 2023-03-03T07:37:28.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nfnetlink_acct.ko 2023-03-03T07:37:28.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nfnetlink_log.ko 2023-03-03T07:37:28.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nfnetlink_osf.ko 2023-03-03T07:37:28.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nfnetlink_queue.ko 2023-03-03T07:37:28.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_chain_nat.ko 2023-03-03T07:37:28.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_compat.ko 2023-03-03T07:37:28.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_ct.ko 2023-03-03T07:37:28.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_dup_netdev.ko 2023-03-03T07:37:28.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_fib.ko 2023-03-03T07:37:28.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_fwd_netdev.ko 2023-03-03T07:37:28.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_hash.ko 2023-03-03T07:37:28.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_limit.ko 2023-03-03T07:37:28.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_log.ko 2023-03-03T07:37:28.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_masq.ko 2023-03-03T07:37:28.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_nat.ko 2023-03-03T07:37:28.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_numgen.ko 2023-03-03T07:37:28.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_queue.ko 2023-03-03T07:37:28.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_quota.ko 2023-03-03T07:37:28.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_redir.ko 2023-03-03T07:37:28.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_reject.ko 2023-03-03T07:37:28.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_reject_inet.ko 2023-03-03T07:37:28.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_socket.ko 2023-03-03T07:37:28.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_tunnel.ko 2023-03-03T07:37:28.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_AUDIT.ko 2023-03-03T07:37:28.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_CHECKSUM.ko 2023-03-03T07:37:28.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_CLASSIFY.ko 2023-03-03T07:37:28.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-03-03T07:37:28.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_CT.ko 2023-03-03T07:37:28.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_DSCP.ko 2023-03-03T07:37:28.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_HL.ko 2023-03-03T07:37:28.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_HMARK.ko 2023-03-03T07:37:28.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_IDLETIMER.ko 2023-03-03T07:37:28.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_LED.ko 2023-03-03T07:37:28.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_LOG.ko 2023-03-03T07:37:28.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_MASQUERADE.ko 2023-03-03T07:37:28.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_NETMAP.ko 2023-03-03T07:37:28.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_NFLOG.ko 2023-03-03T07:37:28.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_NFQUEUE.ko 2023-03-03T07:37:28.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_RATEEST.ko 2023-03-03T07:37:28.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_REDIRECT.ko 2023-03-03T07:37:28.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_SECMARK.ko 2023-03-03T07:37:28.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_TCPMSS.ko 2023-03-03T07:37:28.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-03-03T07:37:28.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_TEE.ko 2023-03-03T07:37:28.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_TPROXY.ko 2023-03-03T07:37:28.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_TRACE.ko 2023-03-03T07:37:28.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_addrtype.ko 2023-03-03T07:37:28.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_bpf.ko 2023-03-03T07:37:28.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_cgroup.ko 2023-03-03T07:37:28.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_cluster.ko 2023-03-03T07:37:28.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_comment.ko 2023-03-03T07:37:28.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_connbytes.ko 2023-03-03T07:37:28.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_connlabel.ko 2023-03-03T07:37:28.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_connlimit.ko 2023-03-03T07:37:28.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_connmark.ko 2023-03-03T07:37:28.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_conntrack.ko 2023-03-03T07:37:28.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_cpu.ko 2023-03-03T07:37:28.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_dccp.ko 2023-03-03T07:37:28.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_devgroup.ko 2023-03-03T07:37:28.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_dscp.ko 2023-03-03T07:37:28.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_ecn.ko 2023-03-03T07:37:28.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_esp.ko 2023-03-03T07:37:28.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_hashlimit.ko 2023-03-03T07:37:28.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_helper.ko 2023-03-03T07:37:28.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_hl.ko 2023-03-03T07:37:28.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_ipcomp.ko 2023-03-03T07:37:28.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_iprange.ko 2023-03-03T07:37:28.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_ipvs.ko 2023-03-03T07:37:28.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_l2tp.ko 2023-03-03T07:37:28.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_length.ko 2023-03-03T07:37:28.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_limit.ko 2023-03-03T07:37:28.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_mac.ko 2023-03-03T07:37:28.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_mark.ko 2023-03-03T07:37:28.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_multiport.ko 2023-03-03T07:37:28.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_nat.ko 2023-03-03T07:37:28.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_nfacct.ko 2023-03-03T07:37:28.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_osf.ko 2023-03-03T07:37:28.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_owner.ko 2023-03-03T07:37:28.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_physdev.ko 2023-03-03T07:37:28.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_pkttype.ko 2023-03-03T07:37:28.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_policy.ko 2023-03-03T07:37:28.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_quota.ko 2023-03-03T07:37:28.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_rateest.ko 2023-03-03T07:37:28.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_realm.ko 2023-03-03T07:37:28.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_recent.ko 2023-03-03T07:37:28.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_sctp.ko 2023-03-03T07:37:28.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_set.ko 2023-03-03T07:37:28.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_socket.ko 2023-03-03T07:37:28.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_state.ko 2023-03-03T07:37:28.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_statistic.ko 2023-03-03T07:37:28.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_string.ko 2023-03-03T07:37:28.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_tcpmss.ko 2023-03-03T07:37:28.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_time.ko 2023-03-03T07:37:28.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_u32.ko 2023-03-03T07:37:28.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netlink 2023-03-03T07:37:28.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netlink/netlink_diag.ko 2023-03-03T07:37:28.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/nsh 2023-03-03T07:37:28.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/nsh/nsh.ko 2023-03-03T07:37:28.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/openvswitch 2023-03-03T07:37:28.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/openvswitch/openvswitch.ko 2023-03-03T07:37:28.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/openvswitch/vport-geneve.ko 2023-03-03T07:37:28.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/openvswitch/vport-gre.ko 2023-03-03T07:37:28.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/openvswitch/vport-vxlan.ko 2023-03-03T07:37:28.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/packet 2023-03-03T07:37:28.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/packet/af_packet_diag.ko 2023-03-03T07:37:28.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/rds 2023-03-03T07:37:28.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/rds/rds.ko 2023-03-03T07:37:28.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/rds/rds_rdma.ko 2023-03-03T07:37:28.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/rds/rds_tcp.ko 2023-03-03T07:37:28.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched 2023-03-03T07:37:28.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_csum.ko 2023-03-03T07:37:28.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_gact.ko 2023-03-03T07:37:28.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_ipt.ko 2023-03-03T07:37:28.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_mirred.ko 2023-03-03T07:37:28.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_nat.ko 2023-03-03T07:37:28.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_pedit.ko 2023-03-03T07:37:28.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_police.ko 2023-03-03T07:37:28.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_simple.ko 2023-03-03T07:37:28.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_skbedit.ko 2023-03-03T07:37:28.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_vlan.ko 2023-03-03T07:37:28.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_basic.ko 2023-03-03T07:37:28.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_bpf.ko 2023-03-03T07:37:28.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_flow.ko 2023-03-03T07:37:28.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_fw.ko 2023-03-03T07:37:28.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_route.ko 2023-03-03T07:37:28.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_rsvp.ko 2023-03-03T07:37:28.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_rsvp6.ko 2023-03-03T07:37:28.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_tcindex.ko 2023-03-03T07:37:28.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_u32.ko 2023-03-03T07:37:28.734 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_cmp.ko 2023-03-03T07:37:28.734 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_ipset.ko 2023-03-03T07:37:28.734 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_meta.ko 2023-03-03T07:37:28.734 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_nbyte.ko 2023-03-03T07:37:28.734 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_text.ko 2023-03-03T07:37:28.734 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_u32.ko 2023-03-03T07:37:28.734 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_cbq.ko 2023-03-03T07:37:28.734 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_choke.ko 2023-03-03T07:37:28.735 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_codel.ko 2023-03-03T07:37:28.735 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_drr.ko 2023-03-03T07:37:28.735 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_dsmark.ko 2023-03-03T07:37:28.735 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_fq.ko 2023-03-03T07:37:28.735 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_gred.ko 2023-03-03T07:37:28.735 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_hfsc.ko 2023-03-03T07:37:28.735 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_hhf.ko 2023-03-03T07:37:28.735 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_htb.ko 2023-03-03T07:37:28.736 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_ingress.ko 2023-03-03T07:37:28.736 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_mqprio.ko 2023-03-03T07:37:28.736 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_multiq.ko 2023-03-03T07:37:28.736 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_netem.ko 2023-03-03T07:37:28.736 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_pie.ko 2023-03-03T07:37:28.736 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_plug.ko 2023-03-03T07:37:28.736 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_prio.ko 2023-03-03T07:37:28.737 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_qfq.ko 2023-03-03T07:37:28.737 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_red.ko 2023-03-03T07:37:28.737 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_sfb.ko 2023-03-03T07:37:28.737 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_sfq.ko 2023-03-03T07:37:28.737 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_tbf.ko 2023-03-03T07:37:28.737 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_teql.ko 2023-03-03T07:37:28.737 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sctp 2023-03-03T07:37:28.737 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sctp/sctp.ko 2023-03-03T07:37:28.738 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sctp/sctp_diag.ko 2023-03-03T07:37:28.738 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc 2023-03-03T07:37:28.738 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/auth_gss 2023-03-03T07:37:28.738 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-03-03T07:37:28.738 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-03-03T07:37:28.738 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/sunrpc.ko 2023-03-03T07:37:28.738 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/xprtrdma 2023-03-03T07:37:28.738 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-03-03T07:37:28.739 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/tipc 2023-03-03T07:37:28.739 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/tipc/diag.ko 2023-03-03T07:37:28.739 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/tipc/tipc.ko 2023-03-03T07:37:28.739 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/unix 2023-03-03T07:37:28.739 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/unix/unix_diag.ko 2023-03-03T07:37:28.739 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock 2023-03-03T07:37:28.739 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-03-03T07:37:28.739 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-03-03T07:37:28.740 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock/vsock.ko 2023-03-03T07:37:28.740 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock/vsock_diag.ko 2023-03-03T07:37:28.740 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock/vsock_loopback.ko 2023-03-03T07:37:28.740 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/xfrm 2023-03-03T07:37:28.740 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/xfrm/xfrm_ipcomp.ko 2023-03-03T07:37:28.740 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security 2023-03-03T07:37:28.740 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security/keys 2023-03-03T07:37:28.741 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security/keys/encrypted-keys 2023-03-03T07:37:28.741 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-03-03T07:37:28.741 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security/keys/trusted-keys 2023-03-03T07:37:28.741 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security/keys/trusted-keys/trusted.ko 2023-03-03T07:37:28.741 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/virt 2023-03-03T07:37:28.741 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/virt/lib 2023-03-03T07:37:28.741 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/virt/lib/irqbypass.ko 2023-03-03T07:37:28.741 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.alias 2023-03-03T07:37:28.742 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.alias.bin 2023-03-03T07:37:28.742 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.builtin 2023-03-03T07:37:28.742 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.builtin.bin 2023-03-03T07:37:28.742 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.builtin.modinfo 2023-03-03T07:37:28.742 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.dep 2023-03-03T07:37:28.742 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.dep.bin 2023-03-03T07:37:28.742 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.devname 2023-03-03T07:37:28.742 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.order 2023-03-03T07:37:28.743 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.softdep 2023-03-03T07:37:28.743 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.symbols 2023-03-03T07:37:28.743 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.symbols.bin 2023-03-03T07:37:28.743 DEBUG:teuthology.task.kernel:get_image_version: 6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:28.744 DEBUG:teuthology.orchestra.run.smithi100:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-03-03T07:37:28.769 INFO:teuthology.orchestra.run.smithi100.stdout:postinstall scriptlet (using /bin/sh): 2023-03-03T07:37:28.769 INFO:teuthology.orchestra.run.smithi100.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.2.0-ceph-g75534f74ac3d -a -r /boot/System.map-6.2.0-ceph-g75534f74ac3d ]; then 2023-03-03T07:37:28.769 INFO:teuthology.orchestra.run.smithi100.stdout:cp /boot/vmlinuz-6.2.0-ceph-g75534f74ac3d /boot/.vmlinuz-6.2.0-ceph-g75534f74ac3d-rpm 2023-03-03T07:37:28.769 INFO:teuthology.orchestra.run.smithi100.stdout:cp /boot/System.map-6.2.0-ceph-g75534f74ac3d /boot/.System.map-6.2.0-ceph-g75534f74ac3d-rpm 2023-03-03T07:37:28.770 INFO:teuthology.orchestra.run.smithi100.stdout:rm -f /boot/vmlinuz-6.2.0-ceph-g75534f74ac3d /boot/System.map-6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:28.770 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/installkernel 6.2.0-ceph-g75534f74ac3d /boot/.vmlinuz-6.2.0-ceph-g75534f74ac3d-rpm /boot/.System.map-6.2.0-ceph-g75534f74ac3d-rpm 2023-03-03T07:37:28.770 INFO:teuthology.orchestra.run.smithi100.stdout:rm -f /boot/.vmlinuz-6.2.0-ceph-g75534f74ac3d-rpm /boot/.System.map-6.2.0-ceph-g75534f74ac3d-rpm 2023-03-03T07:37:28.770 INFO:teuthology.orchestra.run.smithi100.stdout:fi 2023-03-03T07:37:28.770 INFO:teuthology.orchestra.run.smithi100.stdout:preuninstall scriptlet (using /bin/sh): 2023-03-03T07:37:28.770 INFO:teuthology.orchestra.run.smithi100.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-03-03T07:37:28.770 INFO:teuthology.orchestra.run.smithi100.stdout:new-kernel-pkg --remove 6.2.0-ceph-g75534f74ac3d --rminitrd --initrdfile=/boot/initramfs-6.2.0-ceph-g75534f74ac3d.img 2023-03-03T07:37:28.771 INFO:teuthology.orchestra.run.smithi100.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-03-03T07:37:28.771 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-install remove 6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:28.771 INFO:teuthology.orchestra.run.smithi100.stdout:fi 2023-03-03T07:37:28.771 INFO:teuthology.orchestra.run.smithi100.stdout:postuninstall scriptlet (using /bin/sh): 2023-03-03T07:37:28.771 INFO:teuthology.orchestra.run.smithi100.stdout:if [ -x /sbin/update-bootloader ]; then 2023-03-03T07:37:28.771 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/update-bootloader --remove 6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:28.771 INFO:teuthology.orchestra.run.smithi100.stdout:fi 2023-03-03T07:37:28.772 DEBUG:teuthology.orchestra.run.smithi100:> sudo rpm -qi grub2-tools 2023-03-03T07:37:28.852 INFO:teuthology.orchestra.run.smithi100.stdout:Name : grub2-tools 2023-03-03T07:37:28.853 INFO:teuthology.orchestra.run.smithi100.stdout:Epoch : 1 2023-03-03T07:37:28.853 INFO:teuthology.orchestra.run.smithi100.stdout:Version : 2.02 2023-03-03T07:37:28.853 INFO:teuthology.orchestra.run.smithi100.stdout:Release : 129.el8 2023-03-03T07:37:28.853 INFO:teuthology.orchestra.run.smithi100.stdout:Architecture: x86_64 2023-03-03T07:37:28.853 INFO:teuthology.orchestra.run.smithi100.stdout:Install Date: Sun 28 Aug 2022 01:03:50 PM UTC 2023-03-03T07:37:28.854 INFO:teuthology.orchestra.run.smithi100.stdout:Group : System Environment/Base 2023-03-03T07:37:28.854 INFO:teuthology.orchestra.run.smithi100.stdout:Size : 9265586 2023-03-03T07:37:28.854 INFO:teuthology.orchestra.run.smithi100.stdout:License : GPLv3+ 2023-03-03T07:37:28.854 INFO:teuthology.orchestra.run.smithi100.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-03-03T07:37:28.854 INFO:teuthology.orchestra.run.smithi100.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-03-03T07:37:28.854 INFO:teuthology.orchestra.run.smithi100.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-03-03T07:37:28.854 INFO:teuthology.orchestra.run.smithi100.stdout:Build Host : kbuilder.bsys.centos.org 2023-03-03T07:37:28.854 INFO:teuthology.orchestra.run.smithi100.stdout:Relocations : (not relocatable) 2023-03-03T07:37:28.855 INFO:teuthology.orchestra.run.smithi100.stdout:Packager : CentOS BuildSystem 2023-03-03T07:37:28.855 INFO:teuthology.orchestra.run.smithi100.stdout:Vendor : CentOS 2023-03-03T07:37:28.855 INFO:teuthology.orchestra.run.smithi100.stdout:URL : http://www.gnu.org/software/grub/ 2023-03-03T07:37:28.855 INFO:teuthology.orchestra.run.smithi100.stdout:Summary : Support tools for GRUB. 2023-03-03T07:37:28.855 INFO:teuthology.orchestra.run.smithi100.stdout:Description : 2023-03-03T07:37:28.855 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:37:28.856 INFO:teuthology.orchestra.run.smithi100.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-03-03T07:37:28.857 INFO:teuthology.orchestra.run.smithi100.stdout:customizable bootloader with modular architecture. It supports a rich 2023-03-03T07:37:28.857 INFO:teuthology.orchestra.run.smithi100.stdout:variety of kernel formats, file systems, computer architectures and 2023-03-03T07:37:28.857 INFO:teuthology.orchestra.run.smithi100.stdout:hardware devices. 2023-03-03T07:37:28.857 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:37:28.857 INFO:teuthology.orchestra.run.smithi100.stdout:This subpackage provides tools for support of all platforms. 2023-03-03T07:37:28.858 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-03-03T07:37:28.858 INFO:teuthology.task.kernel:Updating grub on smithi100 to boot 6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:28.859 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-03-03T07:37:29.021 INFO:teuthology.orchestra.run.smithi098.stderr:done 2023-03-03T07:37:29.023 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2023-03-03T07:37:29.046 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.y7HMYm54o2 2023-03-03T07:37:29.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.y7HMYm54o2 2023-03-03T07:37:29.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.y7HMYm54o2 2023-03-03T07:37:29.253 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.y7HMYm54o2 is 6KB 2023-03-03T07:37:29.301 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.y7HMYm54o2 2023-03-03T07:37:29.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2023-03-03T07:37:29.392 INFO:teuthology.orchestra.run.smithi098.stdout:2ff282740cf84d4fa02ff60ed95fbbff-0-rescue.conf 2023-03-03T07:37:29.392 INFO:teuthology.orchestra.run.smithi098.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-305.3.1.el8.x86_64.conf 2023-03-03T07:37:29.392 INFO:teuthology.orchestra.run.smithi098.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-408.el8.x86_64.conf 2023-03-03T07:37:29.393 INFO:teuthology.orchestra.run.smithi098.stdout:2ff282740cf84d4fa02ff60ed95fbbff-6.2.0-ceph-g75534f74ac3d.conf 2023-03-03T07:37:29.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:29.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2023-03-03T07:37:33.234 INFO:teuthology.orchestra.run.smithi100.stderr:Generating grub configuration file ... 2023-03-03T07:37:35.721 INFO:teuthology.orchestra.run.smithi100.stderr:done 2023-03-03T07:37:35.723 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2023-03-03T07:37:35.746 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.ltj5df6loV 2023-03-03T07:37:35.747 DEBUG:teuthology.orchestra.run.smithi100:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ltj5df6loV 2023-03-03T07:37:35.825 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.ltj5df6loV 2023-03-03T07:37:35.979 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.ltj5df6loV is 6KB 2023-03-03T07:37:36.029 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.ltj5df6loV 2023-03-03T07:37:36.046 DEBUG:teuthology.orchestra.run.smithi100:> sudo /bin/ls /boot/loader/entries || true 2023-03-03T07:37:36.110 INFO:teuthology.orchestra.run.smithi100.stdout:2ff282740cf84d4fa02ff60ed95fbbff-0-rescue.conf 2023-03-03T07:37:36.111 INFO:teuthology.orchestra.run.smithi100.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-305.3.1.el8.x86_64.conf 2023-03-03T07:37:36.111 INFO:teuthology.orchestra.run.smithi100.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-408.el8.x86_64.conf 2023-03-03T07:37:36.111 INFO:teuthology.orchestra.run.smithi100.stdout:2ff282740cf84d4fa02ff60ed95fbbff-6.2.0-ceph-g75534f74ac3d.conf 2023-03-03T07:37:36.112 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-6.2.0-ceph-g75534f74ac3d 2023-03-03T07:37:36.252 DEBUG:teuthology.orchestra.run.smithi100:> sudo shutdown -r now 2023-03-03T07:37:59.573 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-03-03T07:37:59.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-03-03T07:37:59.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T07:38:06.261 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-03-03T07:38:06.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2023-03-03T07:38:06.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T07:38:17.989 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2023-03-03T07:38:24.645 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2023-03-03T07:38:48.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-03-03T07:38:48.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T07:38:51.081 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2023-03-03T07:38:54.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2023-03-03T07:38:54.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T07:38:57.705 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2023-03-03T07:39:21.105 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-03-03T07:39:21.106 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T07:39:21.507 DEBUG:teuthology.orchestra.run.smithi098:> true 2023-03-03T07:39:21.982 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-03-03T07:39:21.982 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.2.0-ceph-g75534f74ac3d"... 2023-03-03T07:39:21.982 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2023-03-03T07:39:22.048 INFO:teuthology.orchestra.run.smithi098.stdout:6.2.0-ceph-g75534f74ac3d 2023-03-03T07:39:22.049 DEBUG:teuthology.task.kernel:current kernel version is 6.2.0-ceph-g75534f74ac3d vs 6.2.0-ceph-g75534f74ac3d 2023-03-03T07:39:22.049 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-03-03T07:39:22.049 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-03-03T07:39:23.051 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-03-03T07:39:23.051 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-03-03T07:39:23.233 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2023-03-03T07:39:23.265 DEBUG:teuthology.parallel:result is None 2023-03-03T07:39:27.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2023-03-03T07:39:27.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T07:39:28.161 DEBUG:teuthology.orchestra.run.smithi100:> true 2023-03-03T07:39:28.655 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2023-03-03T07:39:28.656 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.2.0-ceph-g75534f74ac3d"... 2023-03-03T07:39:28.656 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2023-03-03T07:39:28.723 INFO:teuthology.orchestra.run.smithi100.stdout:6.2.0-ceph-g75534f74ac3d 2023-03-03T07:39:28.724 DEBUG:teuthology.task.kernel:current kernel version is 6.2.0-ceph-g75534f74ac3d vs 6.2.0-ceph-g75534f74ac3d 2023-03-03T07:39:28.724 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-03-03T07:39:28.724 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-03-03T07:39:29.725 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-03-03T07:39:29.726 DEBUG:teuthology.orchestra.run.smithi100:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-03-03T07:39:29.889 INFO:teuthology.orchestra.run.smithi100.stdout:ttyS1 2023-03-03T07:39:29.914 DEBUG:teuthology.parallel:result is None 2023-03-03T07:39:29.914 INFO:teuthology.run_tasks:Running task internal.base... 2023-03-03T07:39:29.922 INFO:teuthology.task.internal:Creating test directory... 2023-03-03T07:39:29.922 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-03T07:39:29.928 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-03T07:39:29.996 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-03-03T07:39:30.003 INFO:teuthology.run_tasks:Running task internal.archive... 2023-03-03T07:39:30.009 INFO:teuthology.task.internal:Creating archive directory... 2023-03-03T07:39:30.010 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-03T07:39:30.015 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-03T07:39:30.098 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-03-03T07:39:30.107 INFO:teuthology.task.internal:Enabling coredump saving... 2023-03-03T07:39:30.108 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-03T07:39:30.113 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-03T07:39:30.221 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T07:39:30.226 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T07:39:30.257 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T07:39:30.262 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T07:39:30.267 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-03-03T07:39:30.275 INFO:teuthology.task.internal:Configuring sudo... 2023-03-03T07:39:30.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-03T07:39:30.281 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-03T07:39:30.382 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-03-03T07:39:30.399 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-03-03T07:39:30.400 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-03T07:39:30.408 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-03T07:39:30.475 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T07:39:30.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T07:39:30.677 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T07:39:30.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T07:39:30.877 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:39:30.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-03T07:39:30.976 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T07:39:31.080 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T07:39:31.179 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T07:39:31.281 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T07:39:31.381 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:39:31.381 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-03T07:39:31.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2023-03-03T07:39:31.488 DEBUG:teuthology.orchestra.run.smithi100:> sudo service rsyslog restart 2023-03-03T07:39:31.634 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-03-03T07:39:31.699 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-03-03T07:39:32.107 INFO:teuthology.run_tasks:Running task internal.timer... 2023-03-03T07:39:32.117 INFO:teuthology.task.internal:Starting timer... 2023-03-03T07:39:32.117 INFO:teuthology.run_tasks:Running task pcp... 2023-03-03T07:39:32.128 INFO:teuthology.run_tasks:Running task selinux... 2023-03-03T07:39:32.137 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-03-03T07:39:32.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2023-03-03T07:39:32.311 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2023-03-03T07:39:32.317 DEBUG:teuthology.orchestra.run.smithi100:> sudo service auditd rotate 2023-03-03T07:39:32.480 INFO:teuthology.orchestra.run.smithi100.stdout:Rotating logs: 2023-03-03T07:39:32.486 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-03-03T07:39:32.487 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2023-03-03T07:39:32.563 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2023-03-03T07:39:32.564 DEBUG:teuthology.orchestra.run.smithi100:> /usr/sbin/getenforce 2023-03-03T07:39:32.639 INFO:teuthology.orchestra.run.smithi100.stdout:Permissive 2023-03-03T07:39:32.640 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi100.front.sepia.ceph.com': 'permissive'} 2023-03-03T07:39:32.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-03-03T07:39:32.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T07:39:32.743 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-03-03T07:39:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T07:39:32.845 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-03-03T07:39:32.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2023-03-03T07:39:32.947 DEBUG:teuthology.orchestra.run.smithi100:> sudo /usr/sbin/setenforce permissive 2023-03-03T07:39:33.054 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-03-03T07:39:33.073 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-03-03T07:39:33.074 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-03-03T07:39:33.098 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-03-03T07:39:33.100 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi100.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-03-03T07:51:41.427 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi100.front.sepia.ceph.com')] 2023-03-03T07:51:41.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-03-03T07:51:41.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T07:51:41.521 DEBUG:teuthology.orchestra.run.smithi098:> true 2023-03-03T07:51:41.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2023-03-03T07:51:41.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T07:51:41.764 DEBUG:teuthology.orchestra.run.smithi100:> true 2023-03-03T07:51:41.920 INFO:teuthology.run_tasks:Running task clock... 2023-03-03T07:51:41.966 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-03-03T07:51:41.967 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-03T07:51:41.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-03T07:51:41.973 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-03T07:51:41.973 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-03T07:51:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-03-03T07:51:42.079 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-03-03T07:51:42.117 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-03-03T07:51:42.123 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-03-03T07:51:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2023-03-03T07:51:42.238 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: ntpd: command not found 2023-03-03T07:51:42.272 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2023-03-03T07:51:42.275 INFO:teuthology.orchestra.run.smithi100.stdout:506 Cannot talk to daemon 2023-03-03T07:51:42.316 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-03-03T07:51:42.319 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-03-03T07:51:42.360 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-03-03T07:51:42.363 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-03-03T07:51:42.475 INFO:teuthology.orchestra.run.smithi098.stderr:bash: ntpq: command not found 2023-03-03T07:51:42.485 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-03-03T07:51:42.485 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2023-03-03T07:51:42.485 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T07:51:42.485 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T07:51:42.486 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T07:51:42.486 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T07:51:42.486 INFO:teuthology.orchestra.run.smithi100.stderr:bash: ntpq: command not found 2023-03-03T07:51:42.495 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-03-03T07:51:42.495 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2023-03-03T07:51:42.496 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T07:51:42.496 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T07:51:42.496 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T07:51:42.496 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T07:51:42.497 INFO:teuthology.run_tasks:Running task install... 2023-03-03T07:51:42.510 DEBUG:teuthology.task.install:project ceph 2023-03-03T07:51:42.511 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}} 2023-03-03T07:51:42.511 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'} 2023-03-03T07:51:42.511 INFO:teuthology.task.install:Using flavor: default 2023-03-03T07:51:42.517 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-03-03T07:51:42.517 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-03-03T07:51:42.518 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'tag': None, 'wait_for_package': False} 2023-03-03T07:51:42.519 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T07:51:42.521 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'tag': None, 'wait_for_package': False} 2023-03-03T07:51:42.522 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T07:51:42.660 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/ 2023-03-03T07:51:42.661 INFO:teuthology.task.install.rpm:Package version is 18.0.0-2739.g472aaef4 2023-03-03T07:51:42.665 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/ 2023-03-03T07:51:42.666 INFO:teuthology.task.install.rpm:Package version is 18.0.0-2739.g472aaef4 2023-03-03T07:51:42.994 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-03-03T07:51:42.994 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:51:42.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-03-03T07:51:42.998 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-03-03T07:51:42.999 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:51:42.999 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-03-03T07:51:43.100 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-03-03T07:51:43.100 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-03-03T07:51:43.101 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-03-03T07:51:43.102 DEBUG:teuthology.orchestra.run.smithi100:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-03-03T07:51:43.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-03-03T07:51:43.248 DEBUG:teuthology.orchestra.run.smithi100:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-03-03T07:51:43.383 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-03-03T07:51:43.384 DEBUG:teuthology.orchestra.run.smithi100:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-03-03T07:51:43.483 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2023-03-03T07:51:43.484 INFO:teuthology.orchestra.run.smithi100.stdout:check_obsoletes = 1 2023-03-03T07:51:43.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2023-03-03T07:51:43.489 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum clean all 2023-03-03T07:51:43.906 INFO:teuthology.orchestra.run.smithi100.stdout:71 files removed 2023-03-03T07:51:43.933 INFO:teuthology.orchestra.run.smithi098.stdout:71 files removed 2023-03-03T07:51:43.951 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-03-03T07:51:43.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-03-03T07:51:45.287 INFO:teuthology.orchestra.run.smithi100.stdout:ceph packages for x86_64 116 kB/s | 80 kB 00:00 2023-03-03T07:51:45.298 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 119 kB/s | 80 kB 00:00 2023-03-03T07:51:45.912 INFO:teuthology.orchestra.run.smithi100.stdout:ceph noarch packages 16 kB/s | 9.4 kB 00:00 2023-03-03T07:51:45.916 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 16 kB/s | 9.4 kB 00:00 2023-03-03T07:51:46.538 INFO:teuthology.orchestra.run.smithi100.stdout:ceph source packages 2.5 kB/s | 1.5 kB 00:00 2023-03-03T07:51:46.542 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 2.5 kB/s | 1.5 kB 00:00 2023-03-03T07:51:47.449 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-8 - AppStream 32 MB/s | 28 MB 00:00 2023-03-03T07:51:47.486 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-8 - AppStream 31 MB/s | 28 MB 00:00 2023-03-03T07:51:53.728 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-8 - BaseOS 39 MB/s | 27 MB 00:00 2023-03-03T07:51:53.999 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-8 - BaseOS 38 MB/s | 27 MB 00:00 2023-03-03T07:51:58.394 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-8 - Extras 104 kB/s | 18 kB 00:00 2023-03-03T07:51:58.640 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-8 - Extras 172 kB/s | 18 kB 00:00 2023-03-03T07:51:58.797 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-8 - PowerTools 21 MB/s | 5.7 MB 00:00 2023-03-03T07:51:59.042 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-8 - PowerTools 21 MB/s | 5.7 MB 00:00 2023-03-03T07:52:02.521 INFO:teuthology.orchestra.run.smithi098.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-03-03T07:52:02.725 INFO:teuthology.orchestra.run.smithi100.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-03-03T07:52:03.688 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 13 MB/s | 14 MB 00:01 2023-03-03T07:52:03.884 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 13 MB/s | 14 MB 00:01 2023-03-03T07:52:08.063 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 178 kB/s | 24 kB 00:00 2023-03-03T07:52:08.075 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 410 kB/s | 24 kB 00:00 2023-03-03T07:52:11.717 INFO:teuthology.orchestra.run.smithi100.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T07:52:11.717 INFO:teuthology.orchestra.run.smithi100.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T07:52:11.729 INFO:teuthology.orchestra.run.smithi100.stdout:Package NetworkManager-1:1.39.12-1.el8.x86_64 is already installed. 2023-03-03T07:52:11.730 INFO:teuthology.orchestra.run.smithi100.stdout:Package NetworkManager-initscripts-updown-1:1.39.12-1.el8.noarch is already installed. 2023-03-03T07:52:11.730 INFO:teuthology.orchestra.run.smithi100.stdout:Package iproute-5.18.0-1.el8.x86_64 is already installed. 2023-03-03T07:52:11.731 INFO:teuthology.orchestra.run.smithi100.stdout:Package util-linux-2.32.1-38.el8.x86_64 is already installed. 2023-03-03T07:52:11.732 INFO:teuthology.orchestra.run.smithi100.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-03-03T07:52:11.734 INFO:teuthology.orchestra.run.smithi100.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-03-03T07:52:11.737 INFO:teuthology.orchestra.run.smithi100.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-03-03T07:52:11.737 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T07:52:11.738 INFO:teuthology.orchestra.run.smithi100.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-03-03T07:52:11.738 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T07:52:11.750 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-1:1.39.12-1.el8.x86_64 is already installed. 2023-03-03T07:52:11.750 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-initscripts-updown-1:1.39.12-1.el8.noarch is already installed. 2023-03-03T07:52:11.751 INFO:teuthology.orchestra.run.smithi098.stdout:Package iproute-5.18.0-1.el8.x86_64 is already installed. 2023-03-03T07:52:11.752 INFO:teuthology.orchestra.run.smithi098.stdout:Package util-linux-2.32.1-38.el8.x86_64 is already installed. 2023-03-03T07:52:11.753 INFO:teuthology.orchestra.run.smithi098.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-03-03T07:52:11.755 INFO:teuthology.orchestra.run.smithi098.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-03-03T07:52:11.757 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-03-03T07:52:11.758 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-03-03T07:52:11.877 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2023-03-03T07:52:11.889 INFO:teuthology.orchestra.run.smithi100.stdout:========================================================================================================= 2023-03-03T07:52:11.889 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2023-03-03T07:52:11.890 INFO:teuthology.orchestra.run.smithi100.stdout:========================================================================================================= 2023-03-03T07:52:11.890 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2023-03-03T07:52:11.890 INFO:teuthology.orchestra.run.smithi100.stdout: bison x86_64 3.0.4-10.el8 CentOS-AppStream 688 k 2023-03-03T07:52:11.890 INFO:teuthology.orchestra.run.smithi100.stdout: ceph x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.4 k 2023-03-03T07:52:11.890 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 5.4 M 2023-03-03T07:52:11.890 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 845 k 2023-03-03T07:52:11.891 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 166 k 2023-03-03T07:52:11.891 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 1.5 M 2023-03-03T07:52:11.891 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 123 k 2023-03-03T07:52:11.891 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 1.7 M 2023-03-03T07:52:11.891 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 7.4 M 2023-03-03T07:52:11.891 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 47 k 2023-03-03T07:52:11.891 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 13 M 2023-03-03T07:52:11.892 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 47 M 2023-03-03T07:52:11.892 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 263 k 2023-03-03T07:52:11.892 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 83 k 2023-03-03T07:52:11.892 INFO:teuthology.orchestra.run.smithi100.stdout: cephfs-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 257 k 2023-03-03T07:52:11.892 INFO:teuthology.orchestra.run.smithi100.stdout: cephfs-top noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 18 k 2023-03-03T07:52:11.892 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-libelf-devel x86_64 0.188-3.el8 CentOS-BaseOS 61 k 2023-03-03T07:52:11.892 INFO:teuthology.orchestra.run.smithi100.stdout: flex x86_64 2.6.1-9.el8 CentOS-AppStream 320 k 2023-03-03T07:52:11.893 INFO:teuthology.orchestra.run.smithi100.stdout: libaio-devel x86_64 0.3.112-1.el8 CentOS-BaseOS 19 k 2023-03-03T07:52:11.893 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 30 k 2023-03-03T07:52:11.893 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 767 k 2023-03-03T07:52:11.893 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 139 k 2023-03-03T07:52:11.893 INFO:teuthology.orchestra.run.smithi100.stdout: libtool x86_64 2.4.6-25.el8 CentOS-AppStream 709 k 2023-03-03T07:52:11.893 INFO:teuthology.orchestra.run.smithi100.stdout: libuuid-devel x86_64 2.32.1-40.el8 CentOS-BaseOS 99 k 2023-03-03T07:52:11.893 INFO:teuthology.orchestra.run.smithi100.stdout: openssl-devel x86_64 1:1.1.1k-7.el8 CentOS-BaseOS 2.3 M 2023-03-03T07:52:11.894 INFO:teuthology.orchestra.run.smithi100.stdout: postgresql x86_64 10.19-1.module_el8.6.0+1047+4202cf9a CentOS-AppStream 1.5 M 2023-03-03T07:52:11.894 INFO:teuthology.orchestra.run.smithi100.stdout: postgresql-contrib x86_64 10.19-1.module_el8.6.0+1047+4202cf9a CentOS-AppStream 807 k 2023-03-03T07:52:11.894 INFO:teuthology.orchestra.run.smithi100.stdout: postgresql-server x86_64 10.19-1.module_el8.6.0+1047+4202cf9a CentOS-AppStream 5.1 M 2023-03-03T07:52:11.894 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 190 k 2023-03-03T07:52:11.894 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 371 k 2023-03-03T07:52:11.894 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 364 k 2023-03-03T07:52:11.894 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 107 k 2023-03-03T07:52:11.895 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 94 k 2023-03-03T07:52:11.895 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.4 M 2023-03-03T07:52:11.895 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 180 k 2023-03-03T07:52:11.895 INFO:teuthology.orchestra.run.smithi100.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 CentOS-BaseOS 72 k 2023-03-03T07:52:11.895 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrading: 2023-03-03T07:52:11.895 INFO:teuthology.orchestra.run.smithi100.stdout: NetworkManager-initscripts-updown noarch 1:1.40.12-1.el8 CentOS-BaseOS 142 k 2023-03-03T07:52:11.895 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-debuginfod-client x86_64 0.188-3.el8 CentOS-BaseOS 75 k 2023-03-03T07:52:11.896 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-libelf x86_64 0.188-3.el8 CentOS-BaseOS 231 k 2023-03-03T07:52:11.896 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-libs x86_64 0.188-3.el8 CentOS-BaseOS 299 k 2023-03-03T07:52:11.896 INFO:teuthology.orchestra.run.smithi100.stdout: libblkid x86_64 2.32.1-40.el8 CentOS-BaseOS 220 k 2023-03-03T07:52:11.896 INFO:teuthology.orchestra.run.smithi100.stdout: libfdisk x86_64 2.32.1-40.el8 CentOS-BaseOS 253 k 2023-03-03T07:52:11.896 INFO:teuthology.orchestra.run.smithi100.stdout: libmount x86_64 2.32.1-40.el8 CentOS-BaseOS 236 k 2023-03-03T07:52:11.896 INFO:teuthology.orchestra.run.smithi100.stdout: librados2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.7 M 2023-03-03T07:52:11.896 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.6 M 2023-03-03T07:52:11.897 INFO:teuthology.orchestra.run.smithi100.stdout: libsmartcols x86_64 2.32.1-40.el8 CentOS-BaseOS 179 k 2023-03-03T07:52:11.897 INFO:teuthology.orchestra.run.smithi100.stdout: libuuid x86_64 2.32.1-40.el8 CentOS-BaseOS 98 k 2023-03-03T07:52:11.897 INFO:teuthology.orchestra.run.smithi100.stdout: util-linux x86_64 2.32.1-40.el8 CentOS-BaseOS 2.5 M 2023-03-03T07:52:11.897 INFO:teuthology.orchestra.run.smithi100.stdout: util-linux-user x86_64 2.32.1-40.el8 CentOS-BaseOS 102 k 2023-03-03T07:52:11.897 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2023-03-03T07:52:11.897 INFO:teuthology.orchestra.run.smithi100.stdout: automake noarch 1.16.1-7.el8 CentOS-AppStream 713 k 2023-03-03T07:52:11.897 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 22 M 2023-03-03T07:52:11.898 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 23 k 2023-03-03T07:52:11.898 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 2.3 M 2023-03-03T07:52:11.898 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 236 k 2023-03-03T07:52:11.898 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 4.6 M 2023-03-03T07:52:11.898 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 18 M 2023-03-03T07:52:11.898 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 15 k 2023-03-03T07:52:11.898 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 24 k 2023-03-03T07:52:11.899 INFO:teuthology.orchestra.run.smithi100.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2023-03-03T07:52:11.899 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-03-03T07:52:11.899 INFO:teuthology.orchestra.run.smithi100.stdout: jq x86_64 1.6-6.el8 CentOS-AppStream 202 k 2023-03-03T07:52:11.899 INFO:teuthology.orchestra.run.smithi100.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 CentOS-BaseOS 48 k 2023-03-03T07:52:11.899 INFO:teuthology.orchestra.run.smithi100.stdout: krb5-devel x86_64 1.18.2-22.el8 CentOS-BaseOS 561 k 2023-03-03T07:52:11.899 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 137 k 2023-03-03T07:52:11.899 INFO:teuthology.orchestra.run.smithi100.stdout: libcom_err-devel x86_64 1.45.6-5.el8 CentOS-BaseOS 39 k 2023-03-03T07:52:11.899 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran x86_64 8.5.0-18.el8 CentOS-BaseOS 645 k 2023-03-03T07:52:11.900 INFO:teuthology.orchestra.run.smithi100.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-03-03T07:52:11.900 INFO:teuthology.orchestra.run.smithi100.stdout: libpq x86_64 13.5-1.el8 CentOS-AppStream 198 k 2023-03-03T07:52:11.900 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath x86_64 8.5.0-18.el8 CentOS-BaseOS 172 k 2023-03-03T07:52:11.900 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2023-03-03T07:52:11.900 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 474 k 2023-03-03T07:52:11.900 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2023-03-03T07:52:11.900 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.5 M 2023-03-03T07:52:11.901 INFO:teuthology.orchestra.run.smithi100.stdout: libselinux-devel x86_64 2.9-6.el8 CentOS-BaseOS 200 k 2023-03-03T07:52:11.901 INFO:teuthology.orchestra.run.smithi100.stdout: libsepol-devel x86_64 2.9-3.el8 CentOS-BaseOS 87 k 2023-03-03T07:52:11.901 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-03-03T07:52:11.901 INFO:teuthology.orchestra.run.smithi100.stdout: libverto-devel x86_64 0.3.2-2.el8 CentOS-BaseOS 18 k 2023-03-03T07:52:11.901 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2023-03-03T07:52:11.901 INFO:teuthology.orchestra.run.smithi100.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2023-03-03T07:52:11.901 INFO:teuthology.orchestra.run.smithi100.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2023-03-03T07:52:11.902 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2023-03-03T07:52:11.902 INFO:teuthology.orchestra.run.smithi100.stdout: pcre2-devel x86_64 10.32-3.el8 CentOS-BaseOS 605 k 2023-03-03T07:52:11.902 INFO:teuthology.orchestra.run.smithi100.stdout: pcre2-utf16 x86_64 10.32-3.el8 CentOS-BaseOS 229 k 2023-03-03T07:52:11.902 INFO:teuthology.orchestra.run.smithi100.stdout: pcre2-utf32 x86_64 10.32-3.el8 CentOS-BaseOS 220 k 2023-03-03T07:52:11.902 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-03-03T07:52:11.902 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2023-03-03T07:52:11.902 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-03-03T07:52:11.902 INFO:teuthology.orchestra.run.smithi100.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-03-03T07:52:11.903 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-03-03T07:52:11.903 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 45 k 2023-03-03T07:52:11.903 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 117 k 2023-03-03T07:52:11.903 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-03-03T07:52:11.903 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2023-03-03T07:52:11.903 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-03-03T07:52:11.903 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-03-03T07:52:11.903 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2023-03-03T07:52:11.904 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-03-03T07:52:11.904 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-03-03T07:52:11.904 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-03-03T07:52:11.904 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2023-03-03T07:52:11.904 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2023-03-03T07:52:11.904 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2023-03-03T07:52:11.904 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2023-03-03T07:52:11.905 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-03-03T07:52:11.905 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-03-03T07:52:11.905 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako noarch 1.0.6-14.el8 CentOS-AppStream 157 k 2023-03-03T07:52:11.905 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2023-03-03T07:52:11.905 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-03-03T07:52:11.905 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-03-03T07:52:11.905 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2023-03-03T07:52:11.906 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2023-03-03T07:52:11.906 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2023-03-03T07:52:11.906 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-03-03T07:52:11.906 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-03-03T07:52:11.906 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2023-03-03T07:52:11.906 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2023-03-03T07:52:11.906 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2023-03-03T07:52:11.906 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2023-03-03T07:52:11.907 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2023-03-03T07:52:11.907 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2023-03-03T07:52:11.907 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-03-03T07:52:11.907 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2023-03-03T07:52:11.907 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-03-03T07:52:11.907 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-03-03T07:52:11.907 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2023-03-03T07:52:11.908 INFO:teuthology.orchestra.run.smithi100.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-03-03T07:52:11.908 INFO:teuthology.orchestra.run.smithi100.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-03-03T07:52:11.908 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-03-03T07:52:11.908 INFO:teuthology.orchestra.run.smithi100.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-03-03T07:52:11.908 INFO:teuthology.orchestra.run.smithi100.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-03-03T07:52:11.908 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-03-03T07:52:11.908 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-03-03T07:52:11.908 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-03-03T07:52:11.909 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2023-03-03T07:52:11.909 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-03-03T07:52:11.909 INFO:teuthology.orchestra.run.smithi100.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2023-03-03T07:52:11.909 INFO:teuthology.orchestra.run.smithi100.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-03-03T07:52:11.909 INFO:teuthology.orchestra.run.smithi100.stdout: uuid x86_64 1.6.2-43.el8 CentOS-AppStream 64 k 2023-03-03T07:52:11.909 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-AppStream 71 k 2023-03-03T07:52:11.909 INFO:teuthology.orchestra.run.smithi100.stdout: zlib-devel x86_64 1.2.11-19.el8 CentOS-BaseOS 58 k 2023-03-03T07:52:11.910 INFO:teuthology.orchestra.run.smithi100.stdout:Enabling module streams: 2023-03-03T07:52:11.910 INFO:teuthology.orchestra.run.smithi100.stdout: postgresql 10 2023-03-03T07:52:11.910 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:52:11.910 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2023-03-03T07:52:11.910 INFO:teuthology.orchestra.run.smithi100.stdout:========================================================================================================= 2023-03-03T07:52:11.910 INFO:teuthology.orchestra.run.smithi100.stdout:Install 127 Packages 2023-03-03T07:52:11.910 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrade 13 Packages 2023-03-03T07:52:11.910 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:52:11.911 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2023-03-03T07:52:11.912 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================================= 2023-03-03T07:52:11.912 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2023-03-03T07:52:11.912 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================================= 2023-03-03T07:52:11.912 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2023-03-03T07:52:11.912 INFO:teuthology.orchestra.run.smithi098.stdout: bison x86_64 3.0.4-10.el8 CentOS-AppStream 688 k 2023-03-03T07:52:11.912 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.4 k 2023-03-03T07:52:11.912 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 5.4 M 2023-03-03T07:52:11.913 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 845 k 2023-03-03T07:52:11.913 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 166 k 2023-03-03T07:52:11.913 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 1.5 M 2023-03-03T07:52:11.913 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 123 k 2023-03-03T07:52:11.913 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 1.7 M 2023-03-03T07:52:11.913 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 7.4 M 2023-03-03T07:52:11.913 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 47 k 2023-03-03T07:52:11.914 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 13 M 2023-03-03T07:52:11.914 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 47 M 2023-03-03T07:52:11.914 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 263 k 2023-03-03T07:52:11.914 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 83 k 2023-03-03T07:52:11.914 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 257 k 2023-03-03T07:52:11.914 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 18 k 2023-03-03T07:52:11.914 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel x86_64 0.188-3.el8 CentOS-BaseOS 61 k 2023-03-03T07:52:11.914 INFO:teuthology.orchestra.run.smithi098.stdout: flex x86_64 2.6.1-9.el8 CentOS-AppStream 320 k 2023-03-03T07:52:11.915 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel x86_64 0.3.112-1.el8 CentOS-BaseOS 19 k 2023-03-03T07:52:11.915 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 30 k 2023-03-03T07:52:11.915 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 767 k 2023-03-03T07:52:11.915 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 139 k 2023-03-03T07:52:11.915 INFO:teuthology.orchestra.run.smithi098.stdout: libtool x86_64 2.4.6-25.el8 CentOS-AppStream 709 k 2023-03-03T07:52:11.915 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel x86_64 2.32.1-40.el8 CentOS-BaseOS 99 k 2023-03-03T07:52:11.915 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel x86_64 1:1.1.1k-7.el8 CentOS-BaseOS 2.3 M 2023-03-03T07:52:11.916 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql x86_64 10.19-1.module_el8.6.0+1047+4202cf9a CentOS-AppStream 1.5 M 2023-03-03T07:52:11.916 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib x86_64 10.19-1.module_el8.6.0+1047+4202cf9a CentOS-AppStream 807 k 2023-03-03T07:52:11.916 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server x86_64 10.19-1.module_el8.6.0+1047+4202cf9a CentOS-AppStream 5.1 M 2023-03-03T07:52:11.916 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 190 k 2023-03-03T07:52:11.916 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 371 k 2023-03-03T07:52:11.916 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 364 k 2023-03-03T07:52:11.916 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 107 k 2023-03-03T07:52:11.916 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 94 k 2023-03-03T07:52:11.917 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.4 M 2023-03-03T07:52:11.917 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 180 k 2023-03-03T07:52:11.917 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 CentOS-BaseOS 72 k 2023-03-03T07:52:11.917 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2023-03-03T07:52:11.917 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-initscripts-updown noarch 1:1.40.12-1.el8 CentOS-BaseOS 142 k 2023-03-03T07:52:11.917 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client x86_64 0.188-3.el8 CentOS-BaseOS 75 k 2023-03-03T07:52:11.917 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf x86_64 0.188-3.el8 CentOS-BaseOS 231 k 2023-03-03T07:52:11.918 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs x86_64 0.188-3.el8 CentOS-BaseOS 299 k 2023-03-03T07:52:11.918 INFO:teuthology.orchestra.run.smithi098.stdout: libblkid x86_64 2.32.1-40.el8 CentOS-BaseOS 220 k 2023-03-03T07:52:11.918 INFO:teuthology.orchestra.run.smithi098.stdout: libfdisk x86_64 2.32.1-40.el8 CentOS-BaseOS 253 k 2023-03-03T07:52:11.918 INFO:teuthology.orchestra.run.smithi098.stdout: libmount x86_64 2.32.1-40.el8 CentOS-BaseOS 236 k 2023-03-03T07:52:11.918 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.7 M 2023-03-03T07:52:11.918 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.6 M 2023-03-03T07:52:11.919 INFO:teuthology.orchestra.run.smithi098.stdout: libsmartcols x86_64 2.32.1-40.el8 CentOS-BaseOS 179 k 2023-03-03T07:52:11.919 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid x86_64 2.32.1-40.el8 CentOS-BaseOS 98 k 2023-03-03T07:52:11.919 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux x86_64 2.32.1-40.el8 CentOS-BaseOS 2.5 M 2023-03-03T07:52:11.919 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-user x86_64 2.32.1-40.el8 CentOS-BaseOS 102 k 2023-03-03T07:52:11.919 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2023-03-03T07:52:11.919 INFO:teuthology.orchestra.run.smithi098.stdout: automake noarch 1.16.1-7.el8 CentOS-AppStream 713 k 2023-03-03T07:52:11.919 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 22 M 2023-03-03T07:52:11.919 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 23 k 2023-03-03T07:52:11.920 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 2.3 M 2023-03-03T07:52:11.920 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 236 k 2023-03-03T07:52:11.920 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 4.6 M 2023-03-03T07:52:11.920 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 18 M 2023-03-03T07:52:11.920 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 15 k 2023-03-03T07:52:11.920 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 24 k 2023-03-03T07:52:11.920 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2023-03-03T07:52:11.920 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-03-03T07:52:11.921 INFO:teuthology.orchestra.run.smithi098.stdout: jq x86_64 1.6-6.el8 CentOS-AppStream 202 k 2023-03-03T07:52:11.921 INFO:teuthology.orchestra.run.smithi098.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 CentOS-BaseOS 48 k 2023-03-03T07:52:11.921 INFO:teuthology.orchestra.run.smithi098.stdout: krb5-devel x86_64 1.18.2-22.el8 CentOS-BaseOS 561 k 2023-03-03T07:52:11.921 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 137 k 2023-03-03T07:52:11.921 INFO:teuthology.orchestra.run.smithi098.stdout: libcom_err-devel x86_64 1.45.6-5.el8 CentOS-BaseOS 39 k 2023-03-03T07:52:11.921 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 8.5.0-18.el8 CentOS-BaseOS 645 k 2023-03-03T07:52:11.921 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-03-03T07:52:11.922 INFO:teuthology.orchestra.run.smithi098.stdout: libpq x86_64 13.5-1.el8 CentOS-AppStream 198 k 2023-03-03T07:52:11.922 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 8.5.0-18.el8 CentOS-BaseOS 172 k 2023-03-03T07:52:11.922 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2023-03-03T07:52:11.922 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 474 k 2023-03-03T07:52:11.922 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2023-03-03T07:52:11.922 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.5 M 2023-03-03T07:52:11.922 INFO:teuthology.orchestra.run.smithi098.stdout: libselinux-devel x86_64 2.9-6.el8 CentOS-BaseOS 200 k 2023-03-03T07:52:11.923 INFO:teuthology.orchestra.run.smithi098.stdout: libsepol-devel x86_64 2.9-3.el8 CentOS-BaseOS 87 k 2023-03-03T07:52:11.923 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-03-03T07:52:11.923 INFO:teuthology.orchestra.run.smithi098.stdout: libverto-devel x86_64 0.3.2-2.el8 CentOS-BaseOS 18 k 2023-03-03T07:52:11.923 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2023-03-03T07:52:11.923 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2023-03-03T07:52:11.923 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2023-03-03T07:52:11.923 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2023-03-03T07:52:11.923 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-devel x86_64 10.32-3.el8 CentOS-BaseOS 605 k 2023-03-03T07:52:11.924 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf16 x86_64 10.32-3.el8 CentOS-BaseOS 229 k 2023-03-03T07:52:11.924 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf32 x86_64 10.32-3.el8 CentOS-BaseOS 220 k 2023-03-03T07:52:11.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-03-03T07:52:11.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2023-03-03T07:52:11.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-03-03T07:52:11.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-03-03T07:52:11.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-03-03T07:52:11.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 45 k 2023-03-03T07:52:11.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 117 k 2023-03-03T07:52:11.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-03-03T07:52:11.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2023-03-03T07:52:11.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-03-03T07:52:11.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-03-03T07:52:11.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2023-03-03T07:52:11.926 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-03-03T07:52:11.926 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-03-03T07:52:11.926 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-03-03T07:52:11.926 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2023-03-03T07:52:11.926 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2023-03-03T07:52:11.926 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2023-03-03T07:52:11.926 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2023-03-03T07:52:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-03-03T07:52:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-03-03T07:52:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.0.6-14.el8 CentOS-AppStream 157 k 2023-03-03T07:52:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2023-03-03T07:52:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-03-03T07:52:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-03-03T07:52:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2023-03-03T07:52:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2023-03-03T07:52:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2023-03-03T07:52:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-03-03T07:52:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-03-03T07:52:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2023-03-03T07:52:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2023-03-03T07:52:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2023-03-03T07:52:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2023-03-03T07:52:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2023-03-03T07:52:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2023-03-03T07:52:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-03-03T07:52:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2023-03-03T07:52:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-03-03T07:52:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-03-03T07:52:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2023-03-03T07:52:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-03-03T07:52:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-03-03T07:52:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-03-03T07:52:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-03-03T07:52:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-03-03T07:52:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-03-03T07:52:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-03-03T07:52:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-03-03T07:52:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2023-03-03T07:52:11.931 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-03-03T07:52:11.931 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2023-03-03T07:52:11.931 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-03-03T07:52:11.931 INFO:teuthology.orchestra.run.smithi098.stdout: uuid x86_64 1.6.2-43.el8 CentOS-AppStream 64 k 2023-03-03T07:52:11.931 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-AppStream 71 k 2023-03-03T07:52:11.931 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel x86_64 1.2.11-19.el8 CentOS-BaseOS 58 k 2023-03-03T07:52:11.931 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling module streams: 2023-03-03T07:52:11.932 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql 10 2023-03-03T07:52:11.932 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:52:11.932 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2023-03-03T07:52:11.932 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================================= 2023-03-03T07:52:11.932 INFO:teuthology.orchestra.run.smithi098.stdout:Install 127 Packages 2023-03-03T07:52:11.932 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 13 Packages 2023-03-03T07:52:11.932 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:52:11.933 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 206 M 2023-03-03T07:52:11.933 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2023-03-03T07:52:11.933 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 206 M 2023-03-03T07:52:11.934 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2023-03-03T07:52:12.185 INFO:teuthology.orchestra.run.smithi100.stdout:(1/140): ceph-18.0.0-2739.g472aaef4.el8.x86_64. 24 kB/s | 6.4 kB 00:00 2023-03-03T07:52:12.206 INFO:teuthology.orchestra.run.smithi098.stdout:(1/140): ceph-18.0.0-2739.g472aaef4.el8.x86_64. 24 kB/s | 6.4 kB 00:00 2023-03-03T07:52:13.181 INFO:teuthology.orchestra.run.smithi098.stdout:(2/140): ceph-fuse-18.0.0-2739.g472aaef4.el8.x8 867 kB/s | 845 kB 00:00 2023-03-03T07:52:13.340 INFO:teuthology.orchestra.run.smithi098.stdout:(3/140): ceph-immutable-object-cache-18.0.0-273 1.0 MB/s | 166 kB 00:00 2023-03-03T07:52:13.560 INFO:teuthology.orchestra.run.smithi100.stdout:(2/140): ceph-fuse-18.0.0-2739.g472aaef4.el8.x8 615 kB/s | 845 kB 00:01 2023-03-03T07:52:14.085 INFO:teuthology.orchestra.run.smithi100.stdout:(3/140): ceph-immutable-object-cache-18.0.0-273 316 kB/s | 166 kB 00:00 2023-03-03T07:52:14.699 INFO:teuthology.orchestra.run.smithi098.stdout:(4/140): ceph-base-18.0.0-2739.g472aaef4.el8.x8 1.9 MB/s | 5.4 MB 00:02 2023-03-03T07:52:14.719 INFO:teuthology.orchestra.run.smithi100.stdout:(4/140): ceph-base-18.0.0-2739.g472aaef4.el8.x8 1.9 MB/s | 5.4 MB 00:02 2023-03-03T07:52:15.140 INFO:teuthology.orchestra.run.smithi098.stdout:(5/140): ceph-mds-18.0.0-2739.g472aaef4.el8.x86 1.3 MB/s | 2.3 MB 00:01 2023-03-03T07:52:15.211 INFO:teuthology.orchestra.run.smithi100.stdout:(5/140): ceph-mgr-18.0.0-2739.g472aaef4.el8.x86 3.1 MB/s | 1.5 MB 00:00 2023-03-03T07:52:15.216 INFO:teuthology.orchestra.run.smithi098.stdout:(6/140): ceph-mgr-18.0.0-2739.g472aaef4.el8.x86 3.0 MB/s | 1.5 MB 00:00 2023-03-03T07:52:16.436 INFO:teuthology.orchestra.run.smithi100.stdout:(6/140): ceph-mon-18.0.0-2739.g472aaef4.el8.x86 3.8 MB/s | 4.6 MB 00:01 2023-03-03T07:52:17.161 INFO:teuthology.orchestra.run.smithi100.stdout:(7/140): ceph-mds-18.0.0-2739.g472aaef4.el8.x86 757 kB/s | 2.3 MB 00:03 2023-03-03T07:52:18.366 INFO:teuthology.orchestra.run.smithi098.stdout:(7/140): ceph-mon-18.0.0-2739.g472aaef4.el8.x86 1.4 MB/s | 4.6 MB 00:03 2023-03-03T07:52:20.241 INFO:teuthology.orchestra.run.smithi098.stdout:(8/140): ceph-osd-18.0.0-2739.g472aaef4.el8.x86 3.5 MB/s | 18 MB 00:05 2023-03-03T07:52:20.325 INFO:teuthology.orchestra.run.smithi098.stdout:(9/140): ceph-selinux-18.0.0-2739.g472aaef4.el8 293 kB/s | 24 kB 00:00 2023-03-03T07:52:20.611 INFO:teuthology.orchestra.run.smithi100.stdout:(8/140): ceph-osd-18.0.0-2739.g472aaef4.el8.x86 4.2 MB/s | 18 MB 00:04 2023-03-03T07:52:20.686 INFO:teuthology.orchestra.run.smithi100.stdout:(9/140): ceph-selinux-18.0.0-2739.g472aaef4.el8 326 kB/s | 24 kB 00:00 2023-03-03T07:52:23.770 INFO:teuthology.orchestra.run.smithi100.stdout:(10/140): ceph-common-18.0.0-2739.g472aaef4.el8 1.8 MB/s | 22 MB 00:11 2023-03-03T07:52:23.920 INFO:teuthology.orchestra.run.smithi100.stdout:(11/140): cephfs-mirror-18.0.0-2739.g472aaef4.e 1.7 MB/s | 257 kB 00:00 2023-03-03T07:52:24.020 INFO:teuthology.orchestra.run.smithi100.stdout:(12/140): libcephfs-devel-18.0.0-2739.g472aaef4 305 kB/s | 30 kB 00:00 2023-03-03T07:52:24.208 INFO:teuthology.orchestra.run.smithi098.stdout:(10/140): ceph-common-18.0.0-2739.g472aaef4.el8 1.8 MB/s | 22 MB 00:12 2023-03-03T07:52:24.245 INFO:teuthology.orchestra.run.smithi100.stdout:(13/140): libcephfs2-18.0.0-2739.g472aaef4.el8. 3.3 MB/s | 767 kB 00:00 2023-03-03T07:52:24.337 INFO:teuthology.orchestra.run.smithi100.stdout:(14/140): libcephsqlite-18.0.0-2739.g472aaef4.e 1.5 MB/s | 137 kB 00:00 2023-03-03T07:52:24.424 INFO:teuthology.orchestra.run.smithi098.stdout:(11/140): ceph-radosgw-18.0.0-2739.g472aaef4.el 2.2 MB/s | 13 MB 00:06 2023-03-03T07:52:24.437 INFO:teuthology.orchestra.run.smithi100.stdout:(15/140): librados-devel-18.0.0-2739.g472aaef4. 1.4 MB/s | 139 kB 00:00 2023-03-03T07:52:24.492 INFO:teuthology.orchestra.run.smithi098.stdout:(12/140): cephfs-mirror-18.0.0-2739.g472aaef4.e 909 kB/s | 257 kB 00:00 2023-03-03T07:52:24.525 INFO:teuthology.orchestra.run.smithi098.stdout:(13/140): libcephfs-devel-18.0.0-2739.g472aaef4 305 kB/s | 30 kB 00:00 2023-03-03T07:52:24.634 INFO:teuthology.orchestra.run.smithi098.stdout:(14/140): libcephsqlite-18.0.0-2739.g472aaef4.e 1.2 MB/s | 137 kB 00:00 2023-03-03T07:52:24.734 INFO:teuthology.orchestra.run.smithi098.stdout:(15/140): librados-devel-18.0.0-2739.g472aaef4. 1.4 MB/s | 139 kB 00:00 2023-03-03T07:52:24.854 INFO:teuthology.orchestra.run.smithi100.stdout:(16/140): libradosstriper1-18.0.0-2739.g472aaef 1.1 MB/s | 474 kB 00:00 2023-03-03T07:52:24.968 INFO:teuthology.orchestra.run.smithi098.stdout:(16/140): libradosstriper1-18.0.0-2739.g472aaef 2.0 MB/s | 474 kB 00:00 2023-03-03T07:52:25.226 INFO:teuthology.orchestra.run.smithi098.stdout:(17/140): libcephfs2-18.0.0-2739.g472aaef4.el8. 1.0 MB/s | 767 kB 00:00 2023-03-03T07:52:25.310 INFO:teuthology.orchestra.run.smithi098.stdout:(18/140): python3-ceph-argparse-18.0.0-2739.g47 536 kB/s | 45 kB 00:00 2023-03-03T07:52:25.477 INFO:teuthology.orchestra.run.smithi098.stdout:(19/140): python3-ceph-common-18.0.0-2739.g472a 703 kB/s | 117 kB 00:00 2023-03-03T07:52:25.685 INFO:teuthology.orchestra.run.smithi098.stdout:(20/140): python3-cephfs-18.0.0-2739.g472aaef4. 914 kB/s | 190 kB 00:00 2023-03-03T07:52:26.052 INFO:teuthology.orchestra.run.smithi098.stdout:(21/140): python3-rados-18.0.0-2739.g472aaef4.e 1.0 MB/s | 371 kB 00:00 2023-03-03T07:52:26.361 INFO:teuthology.orchestra.run.smithi098.stdout:(22/140): python3-rbd-18.0.0-2739.g472aaef4.el8 1.2 MB/s | 364 kB 00:00 2023-03-03T07:52:26.520 INFO:teuthology.orchestra.run.smithi098.stdout:(23/140): python3-rgw-18.0.0-2739.g472aaef4.el8 674 kB/s | 107 kB 00:00 2023-03-03T07:52:26.612 INFO:teuthology.orchestra.run.smithi098.stdout:(24/140): rbd-fuse-18.0.0-2739.g472aaef4.el8.x8 1.0 MB/s | 94 kB 00:00 2023-03-03T07:52:27.038 INFO:teuthology.orchestra.run.smithi100.stdout:(17/140): librgw2-18.0.0-2739.g472aaef4.el8.x86 3.0 MB/s | 6.5 MB 00:02 2023-03-03T07:52:27.121 INFO:teuthology.orchestra.run.smithi100.stdout:(18/140): python3-ceph-argparse-18.0.0-2739.g47 537 kB/s | 45 kB 00:00 2023-03-03T07:52:27.213 INFO:teuthology.orchestra.run.smithi100.stdout:(19/140): python3-ceph-common-18.0.0-2739.g472a 1.2 MB/s | 117 kB 00:00 2023-03-03T07:52:27.306 INFO:teuthology.orchestra.run.smithi100.stdout:(20/140): python3-cephfs-18.0.0-2739.g472aaef4. 2.0 MB/s | 190 kB 00:00 2023-03-03T07:52:27.472 INFO:teuthology.orchestra.run.smithi100.stdout:(21/140): python3-rados-18.0.0-2739.g472aaef4.e 2.2 MB/s | 371 kB 00:00 2023-03-03T07:52:27.520 INFO:teuthology.orchestra.run.smithi098.stdout:(25/140): librgw2-18.0.0-2739.g472aaef4.el8.x86 2.5 MB/s | 6.5 MB 00:02 2023-03-03T07:52:27.621 INFO:teuthology.orchestra.run.smithi098.stdout:(26/140): rbd-nbd-18.0.0-2739.g472aaef4.el8.x86 1.8 MB/s | 180 kB 00:00 2023-03-03T07:52:27.631 INFO:teuthology.orchestra.run.smithi100.stdout:(22/140): python3-rbd-18.0.0-2739.g472aaef4.el8 2.2 MB/s | 364 kB 00:00 2023-03-03T07:52:27.696 INFO:teuthology.orchestra.run.smithi098.stdout:(27/140): ceph-grafana-dashboards-18.0.0-2739.g 309 kB/s | 23 kB 00:00 2023-03-03T07:52:27.763 INFO:teuthology.orchestra.run.smithi098.stdout:(28/140): ceph-mgr-cephadm-18.0.0-2739.g472aaef 1.8 MB/s | 123 kB 00:00 2023-03-03T07:52:27.815 INFO:teuthology.orchestra.run.smithi100.stdout:(23/140): ceph-radosgw-18.0.0-2739.g472aaef4.el 1.2 MB/s | 13 MB 00:10 2023-03-03T07:52:27.840 INFO:teuthology.orchestra.run.smithi100.stdout:(24/140): python3-rgw-18.0.0-2739.g472aaef4.el8 512 kB/s | 107 kB 00:00 2023-03-03T07:52:27.948 INFO:teuthology.orchestra.run.smithi100.stdout:(25/140): rbd-fuse-18.0.0-2739.g472aaef4.el8.x8 708 kB/s | 94 kB 00:00 2023-03-03T07:52:28.099 INFO:teuthology.orchestra.run.smithi100.stdout:(26/140): rbd-nbd-18.0.0-2739.g472aaef4.el8.x86 1.2 MB/s | 180 kB 00:00 2023-03-03T07:52:28.191 INFO:teuthology.orchestra.run.smithi100.stdout:(27/140): ceph-grafana-dashboards-18.0.0-2739.g 252 kB/s | 23 kB 00:00 2023-03-03T07:52:28.272 INFO:teuthology.orchestra.run.smithi098.stdout:(29/140): ceph-mgr-dashboard-18.0.0-2739.g472aa 3.4 MB/s | 1.7 MB 00:00 2023-03-03T07:52:28.324 INFO:teuthology.orchestra.run.smithi100.stdout:(28/140): ceph-mgr-cephadm-18.0.0-2739.g472aaef 921 kB/s | 123 kB 00:00 2023-03-03T07:52:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:(30/140): rbd-mirror-18.0.0-2739.g472aaef4.el8. 1.7 MB/s | 3.4 MB 00:02 2023-03-03T07:52:28.741 INFO:teuthology.orchestra.run.smithi100.stdout:(29/140): rbd-mirror-18.0.0-2739.g472aaef4.el8. 3.8 MB/s | 3.4 MB 00:00 2023-03-03T07:52:28.822 INFO:teuthology.orchestra.run.smithi098.stdout:(31/140): ceph-mgr-modules-core-18.0.0-2739.g47 1.5 MB/s | 236 kB 00:00 2023-03-03T07:52:28.906 INFO:teuthology.orchestra.run.smithi098.stdout:(32/140): ceph-mgr-rook-18.0.0-2739.g472aaef4.e 569 kB/s | 47 kB 00:00 2023-03-03T07:52:29.006 INFO:teuthology.orchestra.run.smithi098.stdout:(33/140): ceph-prometheus-alerts-18.0.0-2739.g4 146 kB/s | 15 kB 00:00 2023-03-03T07:52:29.182 INFO:teuthology.orchestra.run.smithi098.stdout:(34/140): ceph-volume-18.0.0-2739.g472aaef4.el8 1.5 MB/s | 263 kB 00:00 2023-03-03T07:52:29.257 INFO:teuthology.orchestra.run.smithi098.stdout:(35/140): cephadm-18.0.0-2739.g472aaef4.el8.noa 1.1 MB/s | 83 kB 00:00 2023-03-03T07:52:29.324 INFO:teuthology.orchestra.run.smithi098.stdout:(36/140): cephfs-top-18.0.0-2739.g472aaef4.el8. 272 kB/s | 18 kB 00:00 2023-03-03T07:52:29.399 INFO:teuthology.orchestra.run.smithi098.stdout:(37/140): automake-1.16.1-7.el8.noarch.rpm 9.3 MB/s | 713 kB 00:00 2023-03-03T07:52:29.441 INFO:teuthology.orchestra.run.smithi098.stdout:(38/140): bison-3.0.4-10.el8.x86_64.rpm 16 MB/s | 688 kB 00:00 2023-03-03T07:52:29.475 INFO:teuthology.orchestra.run.smithi098.stdout:(39/140): flex-2.6.1-9.el8.x86_64.rpm 9.4 MB/s | 320 kB 00:00 2023-03-03T07:52:29.500 INFO:teuthology.orchestra.run.smithi098.stdout:(40/140): jq-1.6-6.el8.x86_64.rpm 7.9 MB/s | 202 kB 00:00 2023-03-03T07:52:29.527 INFO:teuthology.orchestra.run.smithi098.stdout:(41/140): libpq-13.5-1.el8.x86_64.rpm 7.9 MB/s | 198 kB 00:00 2023-03-03T07:52:29.560 INFO:teuthology.orchestra.run.smithi098.stdout:(42/140): librdkafka-0.11.4-3.el8.x86_64.rpm 10 MB/s | 354 kB 00:00 2023-03-03T07:52:29.575 INFO:teuthology.orchestra.run.smithi100.stdout:(30/140): ceph-test-18.0.0-2739.g472aaef4.el8.x 5.2 MB/s | 47 MB 00:08 2023-03-03T07:52:29.601 INFO:teuthology.orchestra.run.smithi098.stdout:(43/140): libtool-2.4.6-25.el8.x86_64.rpm 17 MB/s | 709 kB 00:00 2023-03-03T07:52:29.626 INFO:teuthology.orchestra.run.smithi100.stdout:(31/140): ceph-mgr-dashboard-18.0.0-2739.g472aa 1.3 MB/s | 1.7 MB 00:01 2023-03-03T07:52:29.626 INFO:teuthology.orchestra.run.smithi098.stdout:(44/140): lttng-ust-2.8.1-11.el8.x86_64.rpm 10 MB/s | 259 kB 00:00 2023-03-03T07:52:29.651 INFO:teuthology.orchestra.run.smithi100.stdout:(32/140): ceph-mgr-modules-core-18.0.0-2739.g47 3.1 MB/s | 236 kB 00:00 2023-03-03T07:52:29.660 INFO:teuthology.orchestra.run.smithi098.stdout:(45/140): oniguruma-6.8.2-2.el8.x86_64.rpm 5.6 MB/s | 187 kB 00:00 2023-03-03T07:52:29.701 INFO:teuthology.orchestra.run.smithi100.stdout:(33/140): ceph-mgr-rook-18.0.0-2739.g472aaef4.e 633 kB/s | 47 kB 00:00 2023-03-03T07:52:29.726 INFO:teuthology.orchestra.run.smithi100.stdout:(34/140): ceph-prometheus-alerts-18.0.0-2739.g4 194 kB/s | 15 kB 00:00 2023-03-03T07:52:29.769 INFO:teuthology.orchestra.run.smithi098.stdout:(46/140): openblas-0.3.15-4.el8.x86_64.rpm 44 MB/s | 4.7 MB 00:00 2023-03-03T07:52:29.926 INFO:teuthology.orchestra.run.smithi100.stdout:(35/140): cephadm-18.0.0-2739.g472aaef4.el8.noa 414 kB/s | 83 kB 00:00 2023-03-03T07:52:29.952 INFO:teuthology.orchestra.run.smithi100.stdout:(36/140): ceph-volume-18.0.0-2739.g472aaef4.el8 1.0 MB/s | 263 kB 00:00 2023-03-03T07:52:29.987 INFO:teuthology.orchestra.run.smithi100.stdout:(37/140): cephfs-top-18.0.0-2739.g472aaef4.el8. 308 kB/s | 18 kB 00:00 2023-03-03T07:52:30.035 INFO:teuthology.orchestra.run.smithi098.stdout:(47/140): openblas-threads-0.3.15-4.el8.x86_64. 18 MB/s | 4.9 MB 00:00 2023-03-03T07:52:30.035 INFO:teuthology.orchestra.run.smithi100.stdout:(38/140): automake-1.16.1-7.el8.noarch.rpm 8.4 MB/s | 713 kB 00:00 2023-03-03T07:52:30.078 INFO:teuthology.orchestra.run.smithi100.stdout:(39/140): flex-2.6.1-9.el8.x86_64.rpm 7.9 MB/s | 320 kB 00:00 2023-03-03T07:52:30.111 INFO:teuthology.orchestra.run.smithi100.stdout:(40/140): jq-1.6-6.el8.x86_64.rpm 5.9 MB/s | 202 kB 00:00 2023-03-03T07:52:30.120 INFO:teuthology.orchestra.run.smithi098.stdout:(48/140): postgresql-10.19-1.module_el8.6.0+104 18 MB/s | 1.5 MB 00:00 2023-03-03T07:52:30.145 INFO:teuthology.orchestra.run.smithi100.stdout:(41/140): libpq-13.5-1.el8.x86_64.rpm 5.8 MB/s | 198 kB 00:00 2023-03-03T07:52:30.162 INFO:teuthology.orchestra.run.smithi098.stdout:(49/140): postgresql-contrib-10.19-1.module_el8 19 MB/s | 807 kB 00:00 2023-03-03T07:52:30.170 INFO:teuthology.orchestra.run.smithi100.stdout:(42/140): bison-3.0.4-10.el8.x86_64.rpm 3.7 MB/s | 688 kB 00:00 2023-03-03T07:52:30.212 INFO:teuthology.orchestra.run.smithi100.stdout:(43/140): librdkafka-0.11.4-3.el8.x86_64.rpm 5.2 MB/s | 354 kB 00:00 2023-03-03T07:52:30.246 INFO:teuthology.orchestra.run.smithi100.stdout:(44/140): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.7 MB/s | 259 kB 00:00 2023-03-03T07:52:30.280 INFO:teuthology.orchestra.run.smithi100.stdout:(45/140): libtool-2.4.6-25.el8.x86_64.rpm 6.4 MB/s | 709 kB 00:00 2023-03-03T07:52:30.295 INFO:teuthology.orchestra.run.smithi098.stdout:(50/140): postgresql-server-10.19-1.module_el8. 38 MB/s | 5.1 MB 00:00 2023-03-03T07:52:30.313 INFO:teuthology.orchestra.run.smithi100.stdout:(46/140): oniguruma-6.8.2-2.el8.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2023-03-03T07:52:30.387 INFO:teuthology.orchestra.run.smithi098.stdout:(51/140): ceph-mgr-diskprediction-local-18.0.0- 3.5 MB/s | 7.4 MB 00:02 2023-03-03T07:52:30.481 INFO:teuthology.orchestra.run.smithi100.stdout:(47/140): openblas-threads-0.3.15-4.el8.x86_64. 29 MB/s | 4.9 MB 00:00 2023-03-03T07:52:30.513 INFO:teuthology.orchestra.run.smithi098.stdout:(52/140): python3-babel-2.5.1-7.el8.noarch.rpm 22 MB/s | 4.8 MB 00:00 2023-03-03T07:52:30.538 INFO:teuthology.orchestra.run.smithi098.stdout:(53/140): python3-jsonpatch-1.21-2.el8.noarch.r 1.1 MB/s | 27 kB 00:00 2023-03-03T07:52:30.563 INFO:teuthology.orchestra.run.smithi098.stdout:(54/140): python3-jinja2-2.10.1-3.el8.noarch.rp 3.0 MB/s | 538 kB 00:00 2023-03-03T07:52:30.572 INFO:teuthology.orchestra.run.smithi100.stdout:(48/140): ceph-mgr-diskprediction-local-18.0.0- 4.0 MB/s | 7.4 MB 00:01 2023-03-03T07:52:30.589 INFO:teuthology.orchestra.run.smithi098.stdout:(55/140): python3-jsonpointer-1.10-11.el8.noarc 397 kB/s | 20 kB 00:00 2023-03-03T07:52:30.614 INFO:teuthology.orchestra.run.smithi098.stdout:(56/140): python3-markupsafe-0.23-19.el8.x86_64 1.5 MB/s | 39 kB 00:00 2023-03-03T07:52:30.639 INFO:teuthology.orchestra.run.smithi098.stdout:(57/140): python3-mako-1.0.6-14.el8.noarch.rpm 2.0 MB/s | 157 kB 00:00 2023-03-03T07:52:30.656 INFO:teuthology.orchestra.run.smithi100.stdout:(49/140): openblas-0.3.15-4.el8.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2023-03-03T07:52:30.681 INFO:teuthology.orchestra.run.smithi098.stdout:(58/140): python3-numpy-f2py-1.14.3-10.el8.x86_ 5.3 MB/s | 225 kB 00:00 2023-03-03T07:52:30.707 INFO:teuthology.orchestra.run.smithi098.stdout:(59/140): python3-prettytable-0.7.2-14.el8.noar 1.7 MB/s | 44 kB 00:00 2023-03-03T07:52:30.714 INFO:teuthology.orchestra.run.smithi100.stdout:(50/140): postgresql-10.19-1.module_el8.6.0+104 6.4 MB/s | 1.5 MB 00:00 2023-03-03T07:52:30.732 INFO:teuthology.orchestra.run.smithi098.stdout:(60/140): python3-pyOpenSSL-19.0.0-1.el8.noarch 4.1 MB/s | 103 kB 00:00 2023-03-03T07:52:30.766 INFO:teuthology.orchestra.run.smithi098.stdout:(61/140): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.7 MB/s | 126 kB 00:00 2023-03-03T07:52:30.781 INFO:teuthology.orchestra.run.smithi100.stdout:(51/140): postgresql-contrib-10.19-1.module_el8 3.8 MB/s | 807 kB 00:00 2023-03-03T07:52:30.799 INFO:teuthology.orchestra.run.smithi098.stdout:(62/140): python3-pyasn1-modules-0.3.7-6.el8.no 3.2 MB/s | 110 kB 00:00 2023-03-03T07:52:30.833 INFO:teuthology.orchestra.run.smithi098.stdout:(63/140): python3-pytz-2017.2-9.el8.noarch.rpm 1.6 MB/s | 54 kB 00:00 2023-03-03T07:52:30.848 INFO:teuthology.orchestra.run.smithi100.stdout:(52/140): python3-jinja2-2.10.1-3.el8.noarch.rp 7.9 MB/s | 538 kB 00:00 2023-03-03T07:52:30.874 INFO:teuthology.orchestra.run.smithi100.stdout:(53/140): python3-jsonpatch-1.21-2.el8.noarch.r 1.0 MB/s | 27 kB 00:00 2023-03-03T07:52:30.899 INFO:teuthology.orchestra.run.smithi100.stdout:(54/140): python3-jsonpointer-1.10-11.el8.noarc 807 kB/s | 20 kB 00:00 2023-03-03T07:52:30.916 INFO:teuthology.orchestra.run.smithi098.stdout:(64/140): python3-numpy-1.14.3-10.el8.x86_64.rp 12 MB/s | 3.7 MB 00:00 2023-03-03T07:52:30.933 INFO:teuthology.orchestra.run.smithi100.stdout:(55/140): python3-mako-1.0.6-14.el8.noarch.rpm 4.7 MB/s | 157 kB 00:00 2023-03-03T07:52:30.950 INFO:teuthology.orchestra.run.smithi100.stdout:(56/140): python3-markupsafe-0.23-19.el8.x86_64 2.3 MB/s | 39 kB 00:00 2023-03-03T07:52:30.958 INFO:teuthology.orchestra.run.smithi098.stdout:(65/140): python3-werkzeug-0.12.2-4.el8.noarch. 11 MB/s | 457 kB 00:00 2023-03-03T07:52:30.992 INFO:teuthology.orchestra.run.smithi098.stdout:(66/140): socat-1.7.4.1-1.el8.x86_64.rpm 9.5 MB/s | 323 kB 00:00 2023-03-03T07:52:31.018 INFO:teuthology.orchestra.run.smithi098.stdout:(67/140): uuid-1.6.2-43.el8.x86_64.rpm 2.5 MB/s | 64 kB 00:00 2023-03-03T07:52:31.042 INFO:teuthology.orchestra.run.smithi100.stdout:(57/140): postgresql-server-10.19-1.module_el8. 13 MB/s | 5.1 MB 00:00 2023-03-03T07:52:31.043 INFO:teuthology.orchestra.run.smithi098.stdout:(68/140): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 71 kB 00:00 2023-03-03T07:52:31.068 INFO:teuthology.orchestra.run.smithi098.stdout:(69/140): elfutils-libelf-devel-0.188-3.el8.x86 2.4 MB/s | 61 kB 00:00 2023-03-03T07:52:31.093 INFO:teuthology.orchestra.run.smithi098.stdout:(70/140): fuse-2.9.7-16.el8.x86_64.rpm 3.3 MB/s | 83 kB 00:00 2023-03-03T07:52:31.117 INFO:teuthology.orchestra.run.smithi100.stdout:(58/140): python3-babel-2.5.1-7.el8.noarch.rpm 12 MB/s | 4.8 MB 00:00 2023-03-03T07:52:31.119 INFO:teuthology.orchestra.run.smithi098.stdout:(71/140): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.9 MB/s | 48 kB 00:00 2023-03-03T07:52:31.134 INFO:teuthology.orchestra.run.smithi100.stdout:(59/140): python3-prettytable-0.7.2-14.el8.noar 2.6 MB/s | 44 kB 00:00 2023-03-03T07:52:31.153 INFO:teuthology.orchestra.run.smithi098.stdout:(72/140): krb5-devel-1.18.2-22.el8.x86_64.rpm 16 MB/s | 561 kB 00:00 2023-03-03T07:52:31.159 INFO:teuthology.orchestra.run.smithi100.stdout:(60/140): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.9 MB/s | 225 kB 00:00 2023-03-03T07:52:31.178 INFO:teuthology.orchestra.run.smithi098.stdout:(73/140): libaio-devel-0.3.112-1.el8.x86_64.rpm 756 kB/s | 19 kB 00:00 2023-03-03T07:52:31.184 INFO:teuthology.orchestra.run.smithi100.stdout:(61/140): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2023-03-03T07:52:31.202 INFO:teuthology.orchestra.run.smithi100.stdout:(62/140): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.9 MB/s | 126 kB 00:00 2023-03-03T07:52:31.203 INFO:teuthology.orchestra.run.smithi098.stdout:(74/140): libcom_err-devel-1.45.6-5.el8.x86_64. 1.5 MB/s | 39 kB 00:00 2023-03-03T07:52:31.227 INFO:teuthology.orchestra.run.smithi100.stdout:(63/140): python3-pyasn1-modules-0.3.7-6.el8.no 2.6 MB/s | 110 kB 00:00 2023-03-03T07:52:31.244 INFO:teuthology.orchestra.run.smithi100.stdout:(64/140): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-03-03T07:52:31.253 INFO:teuthology.orchestra.run.smithi098.stdout:(75/140): libgfortran-8.5.0-18.el8.x86_64.rpm 13 MB/s | 645 kB 00:00 2023-03-03T07:52:31.286 INFO:teuthology.orchestra.run.smithi100.stdout:(65/140): python3-werkzeug-0.12.2-4.el8.noarch. 11 MB/s | 457 kB 00:00 2023-03-03T07:52:31.328 INFO:teuthology.orchestra.run.smithi100.stdout:(66/140): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2023-03-03T07:52:31.345 INFO:teuthology.orchestra.run.smithi100.stdout:(67/140): uuid-1.6.2-43.el8.x86_64.rpm 3.8 MB/s | 64 kB 00:00 2023-03-03T07:52:31.370 INFO:teuthology.orchestra.run.smithi100.stdout:(68/140): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 71 kB 00:00 2023-03-03T07:52:31.387 INFO:teuthology.orchestra.run.smithi100.stdout:(69/140): elfutils-libelf-devel-0.188-3.el8.x86 3.7 MB/s | 61 kB 00:00 2023-03-03T07:52:31.412 INFO:teuthology.orchestra.run.smithi100.stdout:(70/140): fuse-2.9.7-16.el8.x86_64.rpm 3.3 MB/s | 83 kB 00:00 2023-03-03T07:52:31.429 INFO:teuthology.orchestra.run.smithi100.stdout:(71/140): keyutils-libs-devel-1.5.10-9.el8.x86_ 2.9 MB/s | 48 kB 00:00 2023-03-03T07:52:31.522 INFO:teuthology.orchestra.run.smithi100.stdout:(72/140): python3-numpy-1.14.3-10.el8.x86_64.rp 6.5 MB/s | 3.7 MB 00:00 2023-03-03T07:52:31.547 INFO:teuthology.orchestra.run.smithi100.stdout:(73/140): krb5-devel-1.18.2-22.el8.x86_64.rpm 4.7 MB/s | 561 kB 00:00 2023-03-03T07:52:31.572 INFO:teuthology.orchestra.run.smithi100.stdout:(74/140): libaio-devel-0.3.112-1.el8.x86_64.rpm 753 kB/s | 19 kB 00:00 2023-03-03T07:52:31.598 INFO:teuthology.orchestra.run.smithi100.stdout:(75/140): libcom_err-devel-1.45.6-5.el8.x86_64. 773 kB/s | 39 kB 00:00 2023-03-03T07:52:31.631 INFO:teuthology.orchestra.run.smithi100.stdout:(76/140): libquadmath-8.5.0-18.el8.x86_64.rpm 5.1 MB/s | 172 kB 00:00 2023-03-03T07:52:31.656 INFO:teuthology.orchestra.run.smithi100.stdout:(77/140): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2023-03-03T07:52:31.690 INFO:teuthology.orchestra.run.smithi100.stdout:(78/140): libgfortran-8.5.0-18.el8.x86_64.rpm 5.4 MB/s | 645 kB 00:00 2023-03-03T07:52:31.716 INFO:teuthology.orchestra.run.smithi100.stdout:(79/140): libselinux-devel-2.9-6.el8.x86_64.rpm 3.3 MB/s | 200 kB 00:00 2023-03-03T07:52:31.741 INFO:teuthology.orchestra.run.smithi100.stdout:(80/140): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-03-03T07:52:31.767 INFO:teuthology.orchestra.run.smithi100.stdout:(81/140): libuuid-devel-2.32.1-40.el8.x86_64.rp 1.9 MB/s | 99 kB 00:00 2023-03-03T07:52:31.791 INFO:teuthology.orchestra.run.smithi100.stdout:(82/140): libverto-devel-0.3.2-2.el8.x86_64.rpm 360 kB/s | 18 kB 00:00 2023-03-03T07:52:31.821 INFO:teuthology.orchestra.run.smithi098.stdout:(76/140): ceph-test-18.0.0-2739.g472aaef4.el8.x 4.1 MB/s | 47 MB 00:11 2023-03-03T07:52:31.850 INFO:teuthology.orchestra.run.smithi100.stdout:(83/140): pcre2-devel-10.32-3.el8.x86_64.rpm 10 MB/s | 605 kB 00:00 2023-03-03T07:52:31.855 INFO:teuthology.orchestra.run.smithi098.stdout:(77/140): libquadmath-8.5.0-18.el8.x86_64.rpm 286 kB/s | 172 kB 00:00 2023-03-03T07:52:31.884 INFO:teuthology.orchestra.run.smithi100.stdout:(84/140): pcre2-utf16-10.32-3.el8.x86_64.rpm 6.7 MB/s | 229 kB 00:00 2023-03-03T07:52:31.888 INFO:teuthology.orchestra.run.smithi098.stdout:(78/140): libselinux-devel-2.9-6.el8.x86_64.rpm 6.3 MB/s | 200 kB 00:00 2023-03-03T07:52:31.914 INFO:teuthology.orchestra.run.smithi098.stdout:(79/140): librabbitmq-0.9.0-3.el8.x86_64.rpm 509 kB/s | 47 kB 00:00 2023-03-03T07:52:31.917 INFO:teuthology.orchestra.run.smithi100.stdout:(85/140): pcre2-utf32-10.32-3.el8.x86_64.rpm 6.5 MB/s | 220 kB 00:00 2023-03-03T07:52:31.939 INFO:teuthology.orchestra.run.smithi098.stdout:(80/140): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-03-03T07:52:31.951 INFO:teuthology.orchestra.run.smithi100.stdout:(86/140): python3-cffi-1.11.5-5.el8.x86_64.rpm 7.0 MB/s | 237 kB 00:00 2023-03-03T07:52:32.018 INFO:teuthology.orchestra.run.smithi100.stdout:(87/140): openssl-devel-1.1.1k-7.el8.x86_64.rpm 9.3 MB/s | 2.3 MB 00:00 2023-03-03T07:52:32.022 INFO:teuthology.orchestra.run.smithi098.stdout:(81/140): libverto-devel-0.3.2-2.el8.x86_64.rpm 218 kB/s | 18 kB 00:00 2023-03-03T07:52:32.043 INFO:teuthology.orchestra.run.smithi100.stdout:(88/140): python3-cryptography-3.2.1-5.el8.x86_ 6.0 MB/s | 559 kB 00:00 2023-03-03T07:52:32.048 INFO:teuthology.orchestra.run.smithi098.stdout:(82/140): libuuid-devel-2.32.1-40.el8.x86_64.rp 741 kB/s | 99 kB 00:00 2023-03-03T07:52:32.068 INFO:teuthology.orchestra.run.smithi100.stdout:(89/140): python3-jwt-1.6.1-2.el8.noarch.rpm 1.7 MB/s | 43 kB 00:00 2023-03-03T07:52:32.094 INFO:teuthology.orchestra.run.smithi100.stdout:(90/140): python3-oauthlib-2.1.0-1.el8.noarch.r 3.0 MB/s | 155 kB 00:00 2023-03-03T07:52:32.119 INFO:teuthology.orchestra.run.smithi100.stdout:(91/140): python3-pycparser-2.14-14.el8.noarch. 2.1 MB/s | 109 kB 00:00 2023-03-03T07:52:32.136 INFO:teuthology.orchestra.run.smithi100.stdout:(92/140): python3-requests-oauthlib-1.0.0-1.el8 1.0 MB/s | 43 kB 00:00 2023-03-03T07:52:32.148 INFO:teuthology.orchestra.run.smithi098.stdout:(83/140): openssl-devel-1.1.1k-7.el8.x86_64.rpm 19 MB/s | 2.3 MB 00:00 2023-03-03T07:52:32.161 INFO:teuthology.orchestra.run.smithi100.stdout:(93/140): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.7 MB/s | 72 kB 00:00 2023-03-03T07:52:32.178 INFO:teuthology.orchestra.run.smithi100.stdout:(94/140): zlib-devel-1.2.11-19.el8.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2023-03-03T07:52:32.182 INFO:teuthology.orchestra.run.smithi098.stdout:(84/140): pcre2-devel-10.32-3.el8.x86_64.rpm 4.4 MB/s | 605 kB 00:00 2023-03-03T07:52:32.207 INFO:teuthology.orchestra.run.smithi098.stdout:(85/140): pcre2-utf16-10.32-3.el8.x86_64.rpm 3.9 MB/s | 229 kB 00:00 2023-03-03T07:52:32.232 INFO:teuthology.orchestra.run.smithi098.stdout:(86/140): pcre2-utf32-10.32-3.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-03-03T07:52:32.258 INFO:teuthology.orchestra.run.smithi098.stdout:(87/140): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.6 MB/s | 237 kB 00:00 2023-03-03T07:52:32.283 INFO:teuthology.orchestra.run.smithi098.stdout:(88/140): python3-jwt-1.6.1-2.el8.noarch.rpm 1.7 MB/s | 43 kB 00:00 2023-03-03T07:52:32.317 INFO:teuthology.orchestra.run.smithi098.stdout:(89/140): python3-oauthlib-2.1.0-1.el8.noarch.r 4.6 MB/s | 155 kB 00:00 2023-03-03T07:52:32.342 INFO:teuthology.orchestra.run.smithi098.stdout:(90/140): python3-pycparser-2.14-14.el8.noarch. 4.3 MB/s | 109 kB 00:00 2023-03-03T07:52:32.384 INFO:teuthology.orchestra.run.smithi098.stdout:(91/140): python3-cryptography-3.2.1-5.el8.x86_ 3.6 MB/s | 559 kB 00:00 2023-03-03T07:52:32.404 INFO:teuthology.orchestra.run.smithi100.stdout:(95/140): python3-scipy-1.0.0-21.module_el8.5.0 12 MB/s | 14 MB 00:01 2023-03-03T07:52:32.409 INFO:teuthology.orchestra.run.smithi098.stdout:(92/140): python3-requests-oauthlib-1.0.0-1.el8 639 kB/s | 43 kB 00:00 2023-03-03T07:52:32.420 INFO:teuthology.orchestra.run.smithi100.stdout:(96/140): liboath-2.6.2-3.el8.x86_64.rpm 243 kB/s | 59 kB 00:00 2023-03-03T07:52:32.453 INFO:teuthology.orchestra.run.smithi098.stdout:(93/140): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.0 MB/s | 72 kB 00:00 2023-03-03T07:52:32.477 INFO:teuthology.orchestra.run.smithi098.stdout:(94/140): zlib-devel-1.2.11-19.el8.x86_64.rpm 867 kB/s | 58 kB 00:00 2023-03-03T07:52:32.513 INFO:teuthology.orchestra.run.smithi100.stdout:(97/140): gperftools-libs-2.7-9.el8.x86_64.rpm 873 kB/s | 306 kB 00:00 2023-03-03T07:52:32.571 INFO:teuthology.orchestra.run.smithi100.stdout:(98/140): python3-asyncssh-2.7.0-2.el8.noarch.r 3.1 MB/s | 476 kB 00:00 2023-03-03T07:52:32.588 INFO:teuthology.orchestra.run.smithi100.stdout:(99/140): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 579 kB/s | 44 kB 00:00 2023-03-03T07:52:32.630 INFO:teuthology.orchestra.run.smithi100.stdout:(100/140): python3-beautifulsoup4-4.6.3-2.el8.1 3.1 MB/s | 185 kB 00:00 2023-03-03T07:52:32.647 INFO:teuthology.orchestra.run.smithi100.stdout:(101/140): python3-cachetools-3.1.1-4.el8.noarc 565 kB/s | 33 kB 00:00 2023-03-03T07:52:32.660 INFO:teuthology.orchestra.run.smithi098.stdout:(95/140): python3-scipy-1.0.0-21.module_el8.5.0 7.5 MB/s | 14 MB 00:01 2023-03-03T07:52:32.672 INFO:teuthology.orchestra.run.smithi100.stdout:(102/140): libunwind-1.3.1-3.el8.x86_64.rpm 280 kB/s | 75 kB 00:00 2023-03-03T07:52:32.690 INFO:teuthology.orchestra.run.smithi100.stdout:(103/140): python3-certifi-2018.10.15-7.el8.noa 277 kB/s | 16 kB 00:00 2023-03-03T07:52:32.719 INFO:teuthology.orchestra.run.smithi098.stdout:(96/140): liboath-2.6.2-3.el8.x86_64.rpm 243 kB/s | 59 kB 00:00 2023-03-03T07:52:32.723 INFO:teuthology.orchestra.run.smithi100.stdout:(104/140): python3-cheroot-8.5.2-1.el8.noarch.r 2.2 MB/s | 173 kB 00:00 2023-03-03T07:52:32.748 INFO:teuthology.orchestra.run.smithi100.stdout:(105/140): python3-google-auth-1.1.1-10.el8.noa 1.4 MB/s | 81 kB 00:00 2023-03-03T07:52:32.774 INFO:teuthology.orchestra.run.smithi100.stdout:(106/140): python3-jaraco-6.2-6.el8.noarch.rpm 220 kB/s | 11 kB 00:00 2023-03-03T07:52:32.777 INFO:teuthology.orchestra.run.smithi098.stdout:(97/140): gperftools-libs-2.7-9.el8.x86_64.rpm 946 kB/s | 306 kB 00:00 2023-03-03T07:52:32.799 INFO:teuthology.orchestra.run.smithi100.stdout:(107/140): python3-cherrypy-18.4.0-1.el8.noarch 3.0 MB/s | 384 kB 00:00 2023-03-03T07:52:32.803 INFO:teuthology.orchestra.run.smithi098.stdout:(98/140): libunwind-1.3.1-3.el8.x86_64.rpm 529 kB/s | 75 kB 00:00 2023-03-03T07:52:32.824 INFO:teuthology.orchestra.run.smithi100.stdout:(108/140): python3-jaraco-functools-2.0-4.el8.n 233 kB/s | 18 kB 00:00 2023-03-03T07:52:32.828 INFO:teuthology.orchestra.run.smithi098.stdout:(99/140): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 876 kB/s | 44 kB 00:00 2023-03-03T07:52:32.850 INFO:teuthology.orchestra.run.smithi100.stdout:(109/140): python3-logutils-0.3.5-11.el8.noarch 969 kB/s | 49 kB 00:00 2023-03-03T07:52:32.862 INFO:teuthology.orchestra.run.smithi098.stdout:(100/140): python3-asyncssh-2.7.0-2.el8.noarch. 3.3 MB/s | 476 kB 00:00 2023-03-03T07:52:32.875 INFO:teuthology.orchestra.run.smithi100.stdout:(110/140): python3-more-itertools-7.2.0-3.el8.n 1.1 MB/s | 59 kB 00:00 2023-03-03T07:52:32.887 INFO:teuthology.orchestra.run.smithi098.stdout:(101/140): python3-cachetools-3.1.1-4.el8.noarc 563 kB/s | 33 kB 00:00 2023-03-03T07:52:32.900 INFO:teuthology.orchestra.run.smithi100.stdout:(111/140): python3-natsort-7.1.1-2.el8.noarch.r 1.2 MB/s | 60 kB 00:00 2023-03-03T07:52:32.912 INFO:teuthology.orchestra.run.smithi098.stdout:(102/140): python3-beautifulsoup4-4.6.3-2.el8.1 1.7 MB/s | 185 kB 00:00 2023-03-03T07:52:32.934 INFO:teuthology.orchestra.run.smithi100.stdout:(112/140): python3-pecan-1.3.2-9.el8.noarch.rpm 4.7 MB/s | 283 kB 00:00 2023-03-03T07:52:32.938 INFO:teuthology.orchestra.run.smithi098.stdout:(103/140): python3-certifi-2018.10.15-7.el8.noa 216 kB/s | 16 kB 00:00 2023-03-03T07:52:32.968 INFO:teuthology.orchestra.run.smithi100.stdout:(113/140): python3-kubernetes-11.0.0-6.el8.noar 6.0 MB/s | 1.2 MB 00:00 2023-03-03T07:52:32.971 INFO:teuthology.orchestra.run.smithi098.stdout:(104/140): python3-cheroot-8.5.2-1.el8.noarch.r 2.0 MB/s | 173 kB 00:00 2023-03-03T07:52:32.993 INFO:teuthology.orchestra.run.smithi100.stdout:(114/140): python3-portend-2.6-1.el8.noarch.rpm 169 kB/s | 16 kB 00:00 2023-03-03T07:52:33.014 INFO:teuthology.orchestra.run.smithi098.stdout:(105/140): python3-google-auth-1.1.1-10.el8.noa 1.0 MB/s | 81 kB 00:00 2023-03-03T07:52:33.018 INFO:teuthology.orchestra.run.smithi100.stdout:(115/140): python3-repoze-lru-0.7-6.el8.noarch. 401 kB/s | 34 kB 00:00 2023-03-03T07:52:33.043 INFO:teuthology.orchestra.run.smithi100.stdout:(116/140): python3-routes-2.4.1-12.el8.noarch.r 2.5 MB/s | 196 kB 00:00 2023-03-03T07:52:33.047 INFO:teuthology.orchestra.run.smithi098.stdout:(106/140): python3-cherrypy-18.4.0-1.el8.noarch 2.8 MB/s | 384 kB 00:00 2023-03-03T07:52:33.069 INFO:teuthology.orchestra.run.smithi100.stdout:(117/140): python3-rsa-4.8-1.el8.noarch.rpm 816 kB/s | 61 kB 00:00 2023-03-03T07:52:33.072 INFO:teuthology.orchestra.run.smithi098.stdout:(107/140): python3-jaraco-6.2-6.el8.noarch.rpm 110 kB/s | 11 kB 00:00 2023-03-03T07:52:33.094 INFO:teuthology.orchestra.run.smithi100.stdout:(118/140): python3-simplegeneric-0.8.1-17.el8.n 248 kB/s | 19 kB 00:00 2023-03-03T07:52:33.098 INFO:teuthology.orchestra.run.smithi098.stdout:(108/140): python3-jaraco-functools-2.0-4.el8.n 210 kB/s | 18 kB 00:00 2023-03-03T07:52:33.119 INFO:teuthology.orchestra.run.smithi100.stdout:(119/140): python3-singledispatch-3.4.0.3-18.el 318 kB/s | 24 kB 00:00 2023-03-03T07:52:33.131 INFO:teuthology.orchestra.run.smithi098.stdout:(109/140): python3-logutils-0.3.5-11.el8.noarch 836 kB/s | 49 kB 00:00 2023-03-03T07:52:33.165 INFO:teuthology.orchestra.run.smithi098.stdout:(110/140): python3-more-itertools-7.2.0-3.el8.n 876 kB/s | 59 kB 00:00 2023-03-03T07:52:33.168 INFO:teuthology.orchestra.run.smithi100.stdout:(120/140): python3-tempora-1.14.1-5.el8.noarch. 296 kB/s | 29 kB 00:00 2023-03-03T07:52:33.195 INFO:teuthology.orchestra.run.smithi100.stdout:(121/140): python3-trustme-0.6.0-4.el8.noarch.r 274 kB/s | 27 kB 00:00 2023-03-03T07:52:33.207 INFO:teuthology.orchestra.run.smithi098.stdout:(111/140): python3-natsort-7.1.1-2.el8.noarch.r 803 kB/s | 60 kB 00:00 2023-03-03T07:52:33.220 INFO:teuthology.orchestra.run.smithi100.stdout:(122/140): python3-waitress-1.2.1-2.el8.1.noarc 2.3 MB/s | 241 kB 00:00 2023-03-03T07:52:33.240 INFO:teuthology.orchestra.run.smithi098.stdout:(112/140): python3-pecan-1.3.2-9.el8.noarch.rpm 3.7 MB/s | 283 kB 00:00 2023-03-03T07:52:33.245 INFO:teuthology.orchestra.run.smithi100.stdout:(123/140): python3-webob-1.8.5-1.el8.1.noarch.r 3.2 MB/s | 251 kB 00:00 2023-03-03T07:52:33.271 INFO:teuthology.orchestra.run.smithi100.stdout:(124/140): python3-websocket-client-0.56.0-5.el 809 kB/s | 61 kB 00:00 2023-03-03T07:52:33.282 INFO:teuthology.orchestra.run.smithi098.stdout:(113/140): python3-kubernetes-11.0.0-6.el8.noar 5.0 MB/s | 1.2 MB 00:00 2023-03-03T07:52:33.296 INFO:teuthology.orchestra.run.smithi100.stdout:(125/140): python3-webtest-2.0.33-1.el8.noarch. 1.1 MB/s | 85 kB 00:00 2023-03-03T07:52:33.308 INFO:teuthology.orchestra.run.smithi098.stdout:(114/140): python3-portend-2.6-1.el8.noarch.rpm 155 kB/s | 16 kB 00:00 2023-03-03T07:52:33.321 INFO:teuthology.orchestra.run.smithi100.stdout:(126/140): python3-zc-lockfile-2.0-2.el8.noarch 305 kB/s | 23 kB 00:00 2023-03-03T07:52:33.333 INFO:teuthology.orchestra.run.smithi098.stdout:(115/140): python3-repoze-lru-0.7-6.el8.noarch. 364 kB/s | 34 kB 00:00 2023-03-03T07:52:33.359 INFO:teuthology.orchestra.run.smithi098.stdout:(116/140): python3-routes-2.4.1-12.el8.noarch.r 2.5 MB/s | 196 kB 00:00 2023-03-03T07:52:33.384 INFO:teuthology.orchestra.run.smithi098.stdout:(117/140): python3-rsa-4.8-1.el8.noarch.rpm 814 kB/s | 61 kB 00:00 2023-03-03T07:52:33.409 INFO:teuthology.orchestra.run.smithi098.stdout:(118/140): python3-simplegeneric-0.8.1-17.el8.n 249 kB/s | 19 kB 00:00 2023-03-03T07:52:33.434 INFO:teuthology.orchestra.run.smithi098.stdout:(119/140): python3-singledispatch-3.4.0.3-18.el 317 kB/s | 24 kB 00:00 2023-03-03T07:52:33.460 INFO:teuthology.orchestra.run.smithi098.stdout:(120/140): python3-tempora-1.14.1-5.el8.noarch. 387 kB/s | 29 kB 00:00 2023-03-03T07:52:33.485 INFO:teuthology.orchestra.run.smithi098.stdout:(121/140): python3-trustme-0.6.0-4.el8.noarch.r 365 kB/s | 27 kB 00:00 2023-03-03T07:52:33.510 INFO:teuthology.orchestra.run.smithi098.stdout:(122/140): python3-waitress-1.2.1-2.el8.1.noarc 3.1 MB/s | 241 kB 00:00 2023-03-03T07:52:33.513 INFO:teuthology.orchestra.run.smithi100.stdout:(127/140): thrift-0.13.0-2.el8.x86_64.rpm 7.2 MB/s | 1.7 MB 00:00 2023-03-03T07:52:33.536 INFO:teuthology.orchestra.run.smithi098.stdout:(123/140): python3-webob-1.8.5-1.el8.1.noarch.r 3.3 MB/s | 251 kB 00:00 2023-03-03T07:52:33.539 INFO:teuthology.orchestra.run.smithi100.stdout:(128/140): NetworkManager-initscripts-updown-1. 5.6 MB/s | 142 kB 00:00 2023-03-03T07:52:33.564 INFO:teuthology.orchestra.run.smithi100.stdout:(129/140): elfutils-debuginfod-client-0.188-3.e 3.0 MB/s | 75 kB 00:00 2023-03-03T07:52:33.569 INFO:teuthology.orchestra.run.smithi098.stdout:(124/140): python3-websocket-client-0.56.0-5.el 729 kB/s | 61 kB 00:00 2023-03-03T07:52:33.594 INFO:teuthology.orchestra.run.smithi098.stdout:(125/140): python3-zc-lockfile-2.0-2.el8.noarch 394 kB/s | 23 kB 00:00 2023-03-03T07:52:33.598 INFO:teuthology.orchestra.run.smithi100.stdout:(130/140): elfutils-libelf-0.188-3.el8.x86_64.r 6.9 MB/s | 231 kB 00:00 2023-03-03T07:52:33.620 INFO:teuthology.orchestra.run.smithi098.stdout:(126/140): python3-webtest-2.0.33-1.el8.noarch. 782 kB/s | 85 kB 00:00 2023-03-03T07:52:33.631 INFO:teuthology.orchestra.run.smithi100.stdout:(131/140): elfutils-libs-0.188-3.el8.x86_64.rpm 8.8 MB/s | 299 kB 00:00 2023-03-03T07:52:33.665 INFO:teuthology.orchestra.run.smithi100.stdout:(132/140): libblkid-2.32.1-40.el8.x86_64.rpm 6.5 MB/s | 220 kB 00:00 2023-03-03T07:52:33.699 INFO:teuthology.orchestra.run.smithi100.stdout:(133/140): libfdisk-2.32.1-40.el8.x86_64.rpm 7.5 MB/s | 253 kB 00:00 2023-03-03T07:52:33.732 INFO:teuthology.orchestra.run.smithi100.stdout:(134/140): libmount-2.32.1-40.el8.x86_64.rpm 7.0 MB/s | 236 kB 00:00 2023-03-03T07:52:33.766 INFO:teuthology.orchestra.run.smithi100.stdout:(135/140): libsmartcols-2.32.1-40.el8.x86_64.rp 5.3 MB/s | 179 kB 00:00 2023-03-03T07:52:33.778 INFO:teuthology.orchestra.run.smithi098.stdout:(127/140): thrift-0.13.0-2.el8.x86_64.rpm 8.3 MB/s | 1.7 MB 00:00 2023-03-03T07:52:33.791 INFO:teuthology.orchestra.run.smithi100.stdout:(136/140): libuuid-2.32.1-40.el8.x86_64.rpm 3.9 MB/s | 98 kB 00:00 2023-03-03T07:52:33.804 INFO:teuthology.orchestra.run.smithi098.stdout:(128/140): NetworkManager-initscripts-updown-1. 5.7 MB/s | 142 kB 00:00 2023-03-03T07:52:33.829 INFO:teuthology.orchestra.run.smithi098.stdout:(129/140): elfutils-debuginfod-client-0.188-3.e 3.0 MB/s | 75 kB 00:00 2023-03-03T07:52:33.863 INFO:teuthology.orchestra.run.smithi098.stdout:(130/140): elfutils-libelf-0.188-3.el8.x86_64.r 6.9 MB/s | 231 kB 00:00 2023-03-03T07:52:33.875 INFO:teuthology.orchestra.run.smithi100.stdout:(137/140): util-linux-2.32.1-40.el8.x86_64.rpm 30 MB/s | 2.5 MB 00:00 2023-03-03T07:52:33.892 INFO:teuthology.orchestra.run.smithi100.stdout:(138/140): util-linux-user-2.32.1-40.el8.x86_64 6.2 MB/s | 102 kB 00:00 2023-03-03T07:52:33.897 INFO:teuthology.orchestra.run.smithi098.stdout:(131/140): elfutils-libs-0.188-3.el8.x86_64.rpm 8.7 MB/s | 299 kB 00:00 2023-03-03T07:52:33.930 INFO:teuthology.orchestra.run.smithi098.stdout:(132/140): libblkid-2.32.1-40.el8.x86_64.rpm 6.5 MB/s | 220 kB 00:00 2023-03-03T07:52:33.950 INFO:teuthology.orchestra.run.smithi100.stdout:(139/140): librbd1-18.0.0-2739.g472aaef4.el8.x8 5.7 MB/s | 3.6 MB 00:00 2023-03-03T07:52:33.964 INFO:teuthology.orchestra.run.smithi098.stdout:(133/140): libfdisk-2.32.1-40.el8.x86_64.rpm 7.5 MB/s | 253 kB 00:00 2023-03-03T07:52:33.997 INFO:teuthology.orchestra.run.smithi098.stdout:(134/140): libmount-2.32.1-40.el8.x86_64.rpm 7.0 MB/s | 236 kB 00:00 2023-03-03T07:52:34.031 INFO:teuthology.orchestra.run.smithi098.stdout:(135/140): libsmartcols-2.32.1-40.el8.x86_64.rp 5.3 MB/s | 179 kB 00:00 2023-03-03T07:52:34.056 INFO:teuthology.orchestra.run.smithi098.stdout:(136/140): libuuid-2.32.1-40.el8.x86_64.rpm 3.9 MB/s | 98 kB 00:00 2023-03-03T07:52:34.257 INFO:teuthology.orchestra.run.smithi098.stdout:(137/140): util-linux-2.32.1-40.el8.x86_64.rpm 12 MB/s | 2.5 MB 00:00 2023-03-03T07:52:34.282 INFO:teuthology.orchestra.run.smithi098.stdout:(138/140): util-linux-user-2.32.1-40.el8.x86_64 4.1 MB/s | 102 kB 00:00 2023-03-03T07:52:34.682 INFO:teuthology.orchestra.run.smithi098.stdout:(139/140): librbd1-18.0.0-2739.g472aaef4.el8.x8 3.4 MB/s | 3.6 MB 00:01 2023-03-03T07:52:34.799 INFO:teuthology.orchestra.run.smithi098.stdout:(140/140): librados2-18.0.0-2739.g472aaef4.el8. 3.0 MB/s | 3.7 MB 00:01 2023-03-03T07:52:34.803 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2023-03-03T07:52:34.803 INFO:teuthology.orchestra.run.smithi098.stdout:Total 9.0 MB/s | 206 MB 00:22 2023-03-03T07:52:35.026 INFO:teuthology.orchestra.run.smithi100.stdout:(140/140): librados2-18.0.0-2739.g472aaef4.el8. 2.1 MB/s | 3.7 MB 00:01 2023-03-03T07:52:35.030 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2023-03-03T07:52:35.030 INFO:teuthology.orchestra.run.smithi100.stdout:Total 8.9 MB/s | 206 MB 00:23 2023-03-03T07:52:37.903 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2023-03-03T07:52:38.128 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2023-03-03T07:52:38.191 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2023-03-03T07:52:38.192 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2023-03-03T07:52:38.421 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2023-03-03T07:52:38.422 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2023-03-03T07:52:39.905 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2023-03-03T07:52:39.910 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2023-03-03T07:52:40.018 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2023-03-03T07:52:40.048 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2023-03-03T07:52:45.244 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2023-03-03T07:52:45.307 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2023-03-03T07:52:45.497 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: thrift-0.13.0-2.el8.x86_64 1/1 2023-03-03T07:52:45.556 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: thrift-0.13.0-2.el8.x86_64 1/1 2023-03-03T07:52:45.678 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/153 2023-03-03T07:52:45.755 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/153 2023-03-03T07:52:45.830 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libuuid-2.32.1-40.el8.x86_64 2/153 2023-03-03T07:52:45.905 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : libuuid-2.32.1-40.el8.x86_64 2/153 2023-03-03T07:52:45.946 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libuuid-2.32.1-40.el8.x86_64 2/153 2023-03-03T07:52:46.030 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libuuid-2.32.1-40.el8.x86_64 2/153 2023-03-03T07:52:46.097 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libblkid-2.32.1-40.el8.x86_64 3/153 2023-03-03T07:52:46.205 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : libblkid-2.32.1-40.el8.x86_64 3/153 2023-03-03T07:52:46.209 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libblkid-2.32.1-40.el8.x86_64 3/153 2023-03-03T07:52:46.328 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libblkid-2.32.1-40.el8.x86_64 3/153 2023-03-03T07:52:46.419 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/153 2023-03-03T07:52:46.503 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/153 2023-03-03T07:52:46.618 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 5/153 2023-03-03T07:52:46.676 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 5/153 2023-03-03T07:52:46.789 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 6/153 2023-03-03T07:52:46.864 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 6/153 2023-03-03T07:52:47.290 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 6/153 2023-03-03T07:52:47.357 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 6/153 2023-03-03T07:52:47.455 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 7/153 2023-03-03T07:52:47.547 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 7/153 2023-03-03T07:52:47.957 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 7/153 2023-03-03T07:52:48.037 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 7/153 2023-03-03T07:52:48.130 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 8/153 2023-03-03T07:52:48.205 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 8/153 2023-03-03T07:52:48.311 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 8/153 2023-03-03T07:52:48.378 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 8/153 2023-03-03T07:52:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 9/153 2023-03-03T07:52:48.571 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 9/153 2023-03-03T07:52:48.622 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 9/153 2023-03-03T07:52:48.714 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 9/153 2023-03-03T07:52:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 10/153 2023-03-03T07:52:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 10/153 2023-03-03T07:52:48.954 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 11/153 2023-03-03T07:52:49.055 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 11/153 2023-03-03T07:52:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 11/153 2023-03-03T07:52:49.207 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 11/153 2023-03-03T07:52:49.329 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libelf-0.188-3.el8.x86_64 12/153 2023-03-03T07:52:49.454 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : elfutils-libelf-0.188-3.el8.x86_64 12/153 2023-03-03T07:52:49.535 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 13/153 2023-03-03T07:52:49.676 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 13/153 2023-03-03T07:52:49.749 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 14/153 2023-03-03T07:52:49.874 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 14/153 2023-03-03T07:52:50.241 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpq-13.5-1.el8.x86_64 15/153 2023-03-03T07:52:50.342 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libpq-13.5-1.el8.x86_64 15/153 2023-03-03T07:52:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-10.19-1.module_el8.6.0+1047+4202cf9a. 16/153 2023-03-03T07:52:50.570 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : postgresql-10.19-1.module_el8.6.0+1047+4202cf9a. 16/153 2023-03-03T07:52:50.630 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 17/153 2023-03-03T07:52:50.713 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 17/153 2023-03-03T07:52:50.763 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 17/153 2023-03-03T07:52:50.863 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 17/153 2023-03-03T07:52:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/153 2023-03-03T07:52:50.996 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/153 2023-03-03T07:52:51.108 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/153 2023-03-03T07:52:51.174 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/153 2023-03-03T07:52:51.322 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 19/153 2023-03-03T07:52:51.405 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 19/153 2023-03-03T07:52:51.465 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : zlib-devel-1.2.11-19.el8.x86_64 20/153 2023-03-03T07:52:51.573 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : zlib-devel-1.2.11-19.el8.x86_64 20/153 2023-03-03T07:52:51.596 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-8.5.0-18.el8.x86_64 21/153 2023-03-03T07:52:51.730 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libquadmath-8.5.0-18.el8.x86_64 21/153 2023-03-03T07:52:51.795 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libquadmath-8.5.0-18.el8.x86_64 21/153 2023-03-03T07:52:51.901 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libquadmath-8.5.0-18.el8.x86_64 21/153 2023-03-03T07:52:51.954 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-8.5.0-18.el8.x86_64 22/153 2023-03-03T07:52:52.029 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libgfortran-8.5.0-18.el8.x86_64 22/153 2023-03-03T07:52:52.699 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libgfortran-8.5.0-18.el8.x86_64 22/153 2023-03-03T07:52:52.780 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libgfortran-8.5.0-18.el8.x86_64 22/153 2023-03-03T07:52:52.863 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.15-4.el8.x86_64 23/153 2023-03-03T07:52:52.979 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-0.3.15-4.el8.x86_64 23/153 2023-03-03T07:52:53.609 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 23/153 2023-03-03T07:52:53.727 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 23/153 2023-03-03T07:52:53.771 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 24/153 2023-03-03T07:52:53.896 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 24/153 2023-03-03T07:52:54.668 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 24/153 2023-03-03T07:52:54.783 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 24/153 2023-03-03T07:52:55.065 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/153 2023-03-03T07:52:55.171 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/153 2023-03-03T07:52:55.306 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 26/153 2023-03-03T07:52:55.398 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 26/153 2023-03-03T07:52:55.548 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 27/153 2023-03-03T07:52:55.622 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 27/153 2023-03-03T07:52:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/153 2023-03-03T07:52:55.878 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/153 2023-03-03T07:52:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 29/153 2023-03-03T07:52:56.087 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 29/153 2023-03-03T07:52:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 30/153 2023-03-03T07:52:56.226 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 30/153 2023-03-03T07:52:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 31/153 2023-03-03T07:52:56.460 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 31/153 2023-03-03T07:52:57.563 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 32/153 2023-03-03T07:52:57.594 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 32/153 2023-03-03T07:52:58.030 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 33/153 2023-03-03T07:52:58.035 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 33/153 2023-03-03T07:52:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 34/153 2023-03-03T07:52:58.372 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 34/153 2023-03-03T07:52:58.576 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/153 2023-03-03T07:52:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/153 2023-03-03T07:53:00.809 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/153 2023-03-03T07:53:00.871 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/153 2023-03-03T07:53:01.186 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 36/153 2023-03-03T07:53:01.229 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 36/153 2023-03-03T07:53:01.389 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/153 2023-03-03T07:53:01.449 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/153 2023-03-03T07:53:01.623 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rsa-4.8-1.el8.noarch 38/153 2023-03-03T07:53:01.682 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.8-1.el8.noarch 38/153 2023-03-03T07:53:01.831 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : elfutils-libs-0.188-3.el8.x86_64 39/153 2023-03-03T07:53:01.841 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libs-0.188-3.el8.x86_64 39/153 2023-03-03T07:53:02.044 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : elfutils-debuginfod-client-0.188-3.el8.x86_64 40/153 2023-03-03T07:53:02.069 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-debuginfod-client-0.188-3.el8.x86_64 40/153 2023-03-03T07:53:02.211 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 41/153 2023-03-03T07:53:02.270 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 41/153 2023-03-03T07:53:02.394 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 42/153 2023-03-03T07:53:02.436 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 42/153 2023-03-03T07:53:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : libfdisk-2.32.1-40.el8.x86_64 43/153 2023-03-03T07:53:02.628 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libfdisk-2.32.1-40.el8.x86_64 43/153 2023-03-03T07:53:02.705 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libfdisk-2.32.1-40.el8.x86_64 43/153 2023-03-03T07:53:02.752 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libfdisk-2.32.1-40.el8.x86_64 43/153 2023-03-03T07:53:02.886 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : libmount-2.32.1-40.el8.x86_64 44/153 2023-03-03T07:53:02.919 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libmount-2.32.1-40.el8.x86_64 44/153 2023-03-03T07:53:02.988 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libmount-2.32.1-40.el8.x86_64 44/153 2023-03-03T07:53:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libmount-2.32.1-40.el8.x86_64 44/153 2023-03-03T07:53:03.132 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libuuid-devel-2.32.1-40.el8.x86_64 45/153 2023-03-03T07:53:03.166 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libuuid-devel-2.32.1-40.el8.x86_64 45/153 2023-03-03T07:53:03.311 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : libsmartcols-2.32.1-40.el8.x86_64 46/153 2023-03-03T07:53:03.336 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libsmartcols-2.32.1-40.el8.x86_64 46/153 2023-03-03T07:53:03.957 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libsmartcols-2.32.1-40.el8.x86_64 46/153 2023-03-03T07:53:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libsmartcols-2.32.1-40.el8.x86_64 46/153 2023-03-03T07:53:04.295 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : util-linux-2.32.1-40.el8.x86_64 47/153 2023-03-03T07:53:04.296 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-2.32.1-40.el8.x86_64 47/153 2023-03-03T07:53:04.346 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: util-linux-2.32.1-40.el8.x86_64 47/153 2023-03-03T07:53:04.348 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: util-linux-2.32.1-40.el8.x86_64 47/153 2023-03-03T07:53:04.554 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 48/153 2023-03-03T07:53:04.588 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 48/153 2023-03-03T07:53:04.837 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 49/153 2023-03-03T07:53:04.889 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 49/153 2023-03-03T07:53:05.061 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 50/153 2023-03-03T07:53:05.120 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 50/153 2023-03-03T07:53:05.225 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 51/153 2023-03-03T07:53:05.275 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 51/153 2023-03-03T07:53:05.433 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 52/153 2023-03-03T07:53:05.491 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 52/153 2023-03-03T07:53:05.721 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 53/153 2023-03-03T07:53:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 53/153 2023-03-03T07:53:05.964 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 54/153 2023-03-03T07:53:06.046 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 54/153 2023-03-03T07:53:06.187 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 55/153 2023-03-03T07:53:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 55/153 2023-03-03T07:53:06.398 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 56/153 2023-03-03T07:53:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 56/153 2023-03-03T07:53:06.586 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 57/153 2023-03-03T07:53:06.685 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 57/153 2023-03-03T07:53:06.811 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 58/153 2023-03-03T07:53:06.910 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 58/153 2023-03-03T07:53:07.052 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 59/153 2023-03-03T07:53:07.102 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 59/153 2023-03-03T07:53:07.247 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-portend-2.6-1.el8.noarch 60/153 2023-03-03T07:53:07.314 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-2.6-1.el8.noarch 60/153 2023-03-03T07:53:07.482 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 61/153 2023-03-03T07:53:07.542 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 61/153 2023-03-03T07:53:07.742 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 62/153 2023-03-03T07:53:07.794 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 62/153 2023-03-03T07:53:07.988 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 63/153 2023-03-03T07:53:08.074 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 63/153 2023-03-03T07:53:08.255 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 64/153 2023-03-03T07:53:08.366 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 64/153 2023-03-03T07:53:08.662 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 65/153 2023-03-03T07:53:08.793 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 65/153 2023-03-03T07:53:08.971 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 66/153 2023-03-03T07:53:09.078 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 66/153 2023-03-03T07:53:09.226 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 67/153 2023-03-03T07:53:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 67/153 2023-03-03T07:53:10.140 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 68/153 2023-03-03T07:53:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 68/153 2023-03-03T07:53:10.318 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 69/153 2023-03-03T07:53:10.445 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 69/153 2023-03-03T07:53:10.477 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 69/153 2023-03-03T07:53:10.602 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 69/153 2023-03-03T07:53:10.707 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 70/153 2023-03-03T07:53:10.857 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 70/153 2023-03-03T07:53:10.950 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 71/153 2023-03-03T07:53:11.117 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 71/153 2023-03-03T07:53:11.402 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 72/153 2023-03-03T07:53:11.580 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 72/153 2023-03-03T07:53:11.736 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 73/153 2023-03-03T07:53:11.878 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 73/153 2023-03-03T07:53:12.002 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 74/153 2023-03-03T07:53:12.153 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 74/153 2023-03-03T07:53:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 75/153 2023-03-03T07:53:12.439 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 75/153 2023-03-03T07:53:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 76/153 2023-03-03T07:53:12.698 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 76/153 2023-03-03T07:53:13.949 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 77/153 2023-03-03T07:53:14.135 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 77/153 2023-03-03T07:53:14.398 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 78/153 2023-03-03T07:53:14.616 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 78/153 2023-03-03T07:53:14.704 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 79/153 2023-03-03T07:53:14.946 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 79/153 2023-03-03T07:53:14.976 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 80/153 2023-03-03T07:53:15.229 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 80/153 2023-03-03T07:53:15.317 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 81/153 2023-03-03T07:53:15.587 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 81/153 2023-03-03T07:53:15.613 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/153 2023-03-03T07:53:15.856 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/153 2023-03-03T07:53:16.334 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/153 2023-03-03T07:53:16.550 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/153 2023-03-03T07:53:16.558 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 84/153 2023-03-03T07:53:16.750 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 84/153 2023-03-03T07:53:16.908 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 85/153 2023-03-03T07:53:17.099 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 85/153 2023-03-03T07:53:17.120 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 86/153 2023-03-03T07:53:17.287 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 87/153 2023-03-03T07:53:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 86/153 2023-03-03T07:53:17.490 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 87/153 2023-03-03T07:53:17.632 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 88/153 2023-03-03T07:53:17.827 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 88/153 2023-03-03T07:53:17.881 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libselinux-devel-2.9-6.el8.x86_64 89/153 2023-03-03T07:53:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 90/153 2023-03-03T07:53:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libselinux-devel-2.9-6.el8.x86_64 89/153 2023-03-03T07:53:18.226 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 90/153 2023-03-03T07:53:18.287 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 91/153 2023-03-03T07:53:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 91/153 2023-03-03T07:53:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 92/153 2023-03-03T07:53:18.737 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : fuse-2.9.7-16.el8.x86_64 93/153 2023-03-03T07:53:18.755 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 92/153 2023-03-03T07:53:18.914 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 94/153 2023-03-03T07:53:18.938 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.7-16.el8.x86_64 93/153 2023-03-03T07:53:19.117 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : uuid-1.6.2-43.el8.x86_64 95/153 2023-03-03T07:53:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 94/153 2023-03-03T07:53:19.265 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 95/153 2023-03-03T07:53:19.309 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : uuid-1.6.2-43.el8.x86_64 95/153 2023-03-03T07:53:19.462 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 95/153 2023-03-03T07:53:19.469 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 96/153 2023-03-03T07:53:19.634 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 97/153 2023-03-03T07:53:19.654 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 96/153 2023-03-03T07:53:19.818 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 97/153 2023-03-03T07:53:25.319 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 98/153 2023-03-03T07:53:25.425 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 98/153 2023-03-03T07:53:25.533 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 98/153 2023-03-03T07:53:25.684 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 98/153 2023-03-03T07:53:26.278 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 98/153 2023-03-03T07:53:26.422 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 98/153 2023-03-03T07:53:26.560 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 99/153 2023-03-03T07:53:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 99/153 2023-03-03T07:53:27.617 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 99/153 2023-03-03T07:53:27.782 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 100/153 2023-03-03T07:53:27.833 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 99/153 2023-03-03T07:53:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 100/153 2023-03-03T07:53:49.547 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 100/153 2023-03-03T07:53:49.547 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /sys 2023-03-03T07:53:49.547 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /proc 2023-03-03T07:53:49.547 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /dev 2023-03-03T07:53:49.547 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /run 2023-03-03T07:53:49.548 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /mnt 2023-03-03T07:53:49.548 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /var/tmp 2023-03-03T07:53:49.548 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /home 2023-03-03T07:53:49.548 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /tmp 2023-03-03T07:53:49.548 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /dev 2023-03-03T07:53:49.548 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:53:49.778 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 100/153 2023-03-03T07:53:49.779 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2023-03-03T07:53:49.779 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2023-03-03T07:53:49.779 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /dev 2023-03-03T07:53:49.779 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /run 2023-03-03T07:53:49.780 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2023-03-03T07:53:49.780 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2023-03-03T07:53:49.780 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2023-03-03T07:53:49.780 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2023-03-03T07:53:49.780 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /dev 2023-03-03T07:53:49.780 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:53:49.979 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 101/153 2023-03-03T07:53:50.223 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 101/153 2023-03-03T07:53:51.829 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 101/153 2023-03-03T07:53:52.064 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 102/153 2023-03-03T07:53:52.065 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 101/153 2023-03-03T07:53:52.300 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 102/153 2023-03-03T07:53:52.375 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 102/153 2023-03-03T07:53:52.570 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 103/153 2023-03-03T07:53:52.634 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 102/153 2023-03-03T07:53:52.830 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 103/153 2023-03-03T07:53:53.153 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 103/153 2023-03-03T07:53:53.345 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 104/153 2023-03-03T07:53:53.412 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 103/153 2023-03-03T07:53:53.430 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 104/153 2023-03-03T07:53:53.613 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 104/153 2023-03-03T07:53:53.646 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 105/153 2023-03-03T07:53:53.695 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 104/153 2023-03-03T07:53:53.868 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 106/153 2023-03-03T07:53:53.939 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 105/153 2023-03-03T07:53:54.019 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 107/153 2023-03-03T07:53:54.127 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 107/153 2023-03-03T07:53:54.161 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 106/153 2023-03-03T07:53:54.321 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 107/153 2023-03-03T07:53:54.445 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 107/153 2023-03-03T07:53:54.446 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : jq-1.6-6.el8.x86_64 108/153 2023-03-03T07:53:54.730 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : automake-1.16.1-7.el8.noarch 109/153 2023-03-03T07:53:54.757 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : jq-1.6-6.el8.x86_64 108/153 2023-03-03T07:53:55.049 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : automake-1.16.1-7.el8.noarch 109/153 2023-03-03T07:53:57.947 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 110/153 2023-03-03T07:53:58.133 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 110/153 2023-03-03T07:53:58.245 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 111/153 2023-03-03T07:53:58.286 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 110/153 2023-03-03T07:53:58.453 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 110/153 2023-03-03T07:53:58.477 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 112/153 2023-03-03T07:53:58.563 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 111/153 2023-03-03T07:53:58.685 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 113/153 2023-03-03T07:53:58.796 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 112/153 2023-03-03T07:53:58.987 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 113/153 2023-03-03T07:53:59.027 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 113/153 2023-03-03T07:53:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 113/153 2023-03-03T07:53:59.320 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 114/153 2023-03-03T07:53:59.574 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 114/153 2023-03-03T07:54:00.540 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 114/153 2023-03-03T07:54:00.768 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 115/153 2023-03-03T07:54:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 114/153 2023-03-03T07:54:00.930 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 115/153 2023-03-03T07:54:00.995 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 115/153 2023-03-03T07:54:01.157 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 115/153 2023-03-03T07:54:01.326 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 116/153 2023-03-03T07:54:01.516 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 117/153 2023-03-03T07:54:01.588 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 116/153 2023-03-03T07:54:01.614 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 117/153 2023-03-03T07:54:01.777 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 118/153 2023-03-03T07:54:01.809 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 117/153 2023-03-03T07:54:01.828 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 118/153 2023-03-03T07:54:01.908 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 117/153 2023-03-03T07:54:02.031 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 119/153 2023-03-03T07:54:02.104 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 118/153 2023-03-03T07:54:02.156 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 118/153 2023-03-03T07:54:02.209 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libtool-2.4.6-25.el8.x86_64 120/153 2023-03-03T07:54:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 119/153 2023-03-03T07:54:02.545 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libtool-2.4.6-25.el8.x86_64 120/153 2023-03-03T07:54:07.334 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 120/153 2023-03-03T07:54:07.650 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 120/153 2023-03-03T07:54:08.931 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 121/153 2023-03-03T07:54:09.176 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 122/153 2023-03-03T07:54:09.269 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 122/153 2023-03-03T07:54:09.280 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 121/153 2023-03-03T07:54:09.467 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 123/153 2023-03-03T07:54:09.547 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 122/153 2023-03-03T07:54:09.569 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-2739.g472aa 123/153 2023-03-03T07:54:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 122/153 2023-03-03T07:54:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 124/153 2023-03-03T07:54:09.886 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 123/153 2023-03-03T07:54:09.989 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-2739.g472aa 123/153 2023-03-03T07:54:10.203 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 124/153 2023-03-03T07:54:10.213 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 124/153 2023-03-03T07:54:10.409 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 125/153 2023-03-03T07:54:10.661 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 124/153 2023-03-03T07:54:10.745 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 125/153 2023-03-03T07:54:10.861 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 125/153 2023-03-03T07:54:11.125 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : postgresql-contrib-10.19-1.module_el8.6.0+1047+4 126/153 2023-03-03T07:54:11.202 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 125/153 2023-03-03T07:54:11.594 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-contrib-10.19-1.module_el8.6.0+1047+4 126/153 2023-03-03T07:54:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 127/153 2023-03-03T07:54:12.832 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openssl-devel-1:1.1.1k-7.el8.x86_64 128/153 2023-03-03T07:54:13.015 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 127/153 2023-03-03T07:54:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : util-linux-user-2.32.1-40.el8.x86_64 129/153 2023-03-03T07:54:13.222 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 130/153 2023-03-03T07:54:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openssl-devel-1:1.1.1k-7.el8.x86_64 128/153 2023-03-03T07:54:13.405 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 131/153 2023-03-03T07:54:13.575 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-user-2.32.1-40.el8.x86_64 129/153 2023-03-03T07:54:13.601 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : elfutils-libelf-devel-0.188-3.el8.x86_64 132/153 2023-03-03T07:54:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 130/153 2023-03-03T07:54:13.974 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 131/153 2023-03-03T07:54:14.178 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : elfutils-libelf-devel-0.188-3.el8.x86_64 132/153 2023-03-03T07:54:17.486 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: postgresql-server-10.19-1.module_el8.6.0+1047+42 133/153 2023-03-03T07:54:17.858 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : postgresql-server-10.19-1.module_el8.6.0+1047+42 133/153 2023-03-03T07:54:17.920 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: postgresql-server-10.19-1.module_el8.6.0+1047+42 133/153 2023-03-03T07:54:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-10.19-1.module_el8.6.0+1047+42 133/153 2023-03-03T07:54:18.151 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 134/153 2023-03-03T07:54:18.377 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-server-10.19-1.module_el8.6.0+1047+42 133/153 2023-03-03T07:54:18.384 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 135/153 2023-03-03T07:54:18.440 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-10.19-1.module_el8.6.0+1047+42 133/153 2023-03-03T07:54:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 136/153 2023-03-03T07:54:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 134/153 2023-03-03T07:54:18.749 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : NetworkManager-initscripts-updown-1:1.40.12-1.el 137/153 2023-03-03T07:54:18.800 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.12-1.el 137/153 2023-03-03T07:54:18.979 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 135/153 2023-03-03T07:54:18.981 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 138/153 2023-03-03T07:54:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flex-2.6.1-9.el8.x86_64 139/153 2023-03-03T07:54:19.212 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 136/153 2023-03-03T07:54:19.323 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 139/153 2023-03-03T07:54:19.351 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-initscripts-updown-1:1.40.12-1.el 137/153 2023-03-03T07:54:19.407 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.12-1.el 137/153 2023-03-03T07:54:19.523 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : bison-3.0.4-10.el8.x86_64 140/153 2023-03-03T07:54:19.572 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 140/153 2023-03-03T07:54:19.624 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 138/153 2023-03-03T07:54:19.742 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : elfutils-debuginfod-client-0.187-4.el8.x86_64 141/153 2023-03-03T07:54:19.834 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flex-2.6.1-9.el8.x86_64 139/153 2023-03-03T07:54:19.927 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : elfutils-libs-0.187-4.el8.x86_64 142/153 2023-03-03T07:54:19.928 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 143/153 2023-03-03T07:54:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 139/153 2023-03-03T07:54:20.176 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : bison-3.0.4-10.el8.x86_64 140/153 2023-03-03T07:54:20.177 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 143/153 2023-03-03T07:54:20.213 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.39.12-1.el 144/153 2023-03-03T07:54:20.226 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 140/153 2023-03-03T07:54:20.325 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : NetworkManager-initscripts-updown-1:1.39.12-1.el 144/153 2023-03-03T07:54:20.325 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 145/153 2023-03-03T07:54:20.420 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-debuginfod-client-0.187-4.el8.x86_64 141/153 2023-03-03T07:54:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libs-0.187-4.el8.x86_64 142/153 2023-03-03T07:54:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 143/153 2023-03-03T07:54:20.618 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 145/153 2023-03-03T07:54:20.792 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : util-linux-user-2.32.1-38.el8.x86_64 146/153 2023-03-03T07:54:20.871 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 143/153 2023-03-03T07:54:20.907 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.39.12-1.el 144/153 2023-03-03T07:54:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-initscripts-updown-1:1.39.12-1.el 144/153 2023-03-03T07:54:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 145/153 2023-03-03T07:54:21.034 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : util-linux-2.32.1-38.el8.x86_64 147/153 2023-03-03T07:54:21.034 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : libfdisk-2.32.1-38.el8.x86_64 148/153 2023-03-03T07:54:21.310 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libfdisk-2.32.1-38.el8.x86_64 148/153 2023-03-03T07:54:21.310 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : libmount-2.32.1-38.el8.x86_64 149/153 2023-03-03T07:54:21.312 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 145/153 2023-03-03T07:54:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-user-2.32.1-38.el8.x86_64 146/153 2023-03-03T07:54:21.560 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libmount-2.32.1-38.el8.x86_64 149/153 2023-03-03T07:54:21.560 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : libblkid-2.32.1-38.el8.x86_64 150/153 2023-03-03T07:54:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-2.32.1-38.el8.x86_64 147/153 2023-03-03T07:54:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libfdisk-2.32.1-38.el8.x86_64 148/153 2023-03-03T07:54:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libblkid-2.32.1-38.el8.x86_64 150/153 2023-03-03T07:54:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : libuuid-2.32.1-38.el8.x86_64 151/153 2023-03-03T07:54:21.979 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libfdisk-2.32.1-38.el8.x86_64 148/153 2023-03-03T07:54:21.980 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libmount-2.32.1-38.el8.x86_64 149/153 2023-03-03T07:54:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libuuid-2.32.1-38.el8.x86_64 151/153 2023-03-03T07:54:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : libsmartcols-2.32.1-38.el8.x86_64 152/153 2023-03-03T07:54:22.212 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libmount-2.32.1-38.el8.x86_64 149/153 2023-03-03T07:54:22.213 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libblkid-2.32.1-38.el8.x86_64 150/153 2023-03-03T07:54:22.309 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libsmartcols-2.32.1-38.el8.x86_64 152/153 2023-03-03T07:54:22.462 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libblkid-2.32.1-38.el8.x86_64 150/153 2023-03-03T07:54:22.463 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libuuid-2.32.1-38.el8.x86_64 151/153 2023-03-03T07:54:23.750 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : elfutils-libelf-0.187-4.el8.x86_64 153/153 2023-03-03T07:54:23.913 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libuuid-2.32.1-38.el8.x86_64 151/153 2023-03-03T07:54:23.913 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libsmartcols-2.32.1-38.el8.x86_64 152/153 2023-03-03T07:54:24.079 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: elfutils-libelf-0.187-4.el8.x86_64 153/153 2023-03-03T07:54:24.079 INFO:teuthology.orchestra.run.smithi100.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-03-03T07:54:24.080 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:24.163 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libsmartcols-2.32.1-38.el8.x86_64 152/153 2023-03-03T07:54:24.344 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libelf-0.187-4.el8.x86_64 153/153 2023-03-03T07:54:24.688 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: elfutils-libelf-0.187-4.el8.x86_64 153/153 2023-03-03T07:54:24.688 INFO:teuthology.orchestra.run.smithi098.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-03-03T07:54:24.689 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:25.003 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 1/153 2023-03-03T07:54:25.004 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2/153 2023-03-03T07:54:25.004 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 3/153 2023-03-03T07:54:25.004 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 4/153 2023-03-03T07:54:25.004 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 5/153 2023-03-03T07:54:25.005 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 6/153 2023-03-03T07:54:25.005 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 7/153 2023-03-03T07:54:25.005 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 8/153 2023-03-03T07:54:25.005 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 9/153 2023-03-03T07:54:25.006 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 10/153 2023-03-03T07:54:25.006 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 11/153 2023-03-03T07:54:25.006 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 12/153 2023-03-03T07:54:25.006 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 13/153 2023-03-03T07:54:25.006 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 14/153 2023-03-03T07:54:25.007 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 15/153 2023-03-03T07:54:25.007 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 16/153 2023-03-03T07:54:25.007 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 17/153 2023-03-03T07:54:25.007 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/153 2023-03-03T07:54:25.007 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 19/153 2023-03-03T07:54:25.008 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 20/153 2023-03-03T07:54:25.008 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 21/153 2023-03-03T07:54:25.008 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 22/153 2023-03-03T07:54:25.008 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 23/153 2023-03-03T07:54:25.008 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 24/153 2023-03-03T07:54:25.009 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 25/153 2023-03-03T07:54:25.009 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 26/153 2023-03-03T07:54:25.009 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 27/153 2023-03-03T07:54:25.009 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 28/153 2023-03-03T07:54:25.009 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 29/153 2023-03-03T07:54:25.019 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 30/153 2023-03-03T07:54:25.020 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 31/153 2023-03-03T07:54:25.020 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 32/153 2023-03-03T07:54:25.020 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 33/153 2023-03-03T07:54:25.020 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 34/153 2023-03-03T07:54:25.021 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 35/153 2023-03-03T07:54:25.021 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 36/153 2023-03-03T07:54:25.021 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 37/153 2023-03-03T07:54:25.021 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 38/153 2023-03-03T07:54:25.021 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : automake-1.16.1-7.el8.noarch 39/153 2023-03-03T07:54:25.022 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/153 2023-03-03T07:54:25.022 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/153 2023-03-03T07:54:25.022 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : jq-1.6-6.el8.x86_64 42/153 2023-03-03T07:54:25.022 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libpq-13.5-1.el8.x86_64 43/153 2023-03-03T07:54:25.022 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 44/153 2023-03-03T07:54:25.022 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 45/153 2023-03-03T07:54:25.023 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 46/153 2023-03-03T07:54:25.023 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 47/153 2023-03-03T07:54:25.023 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 48/153 2023-03-03T07:54:25.023 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 49/153 2023-03-03T07:54:25.023 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : postgresql-10.19-1.module_el8.6.0+1047+4202cf9a. 50/153 2023-03-03T07:54:25.024 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : postgresql-contrib-10.19-1.module_el8.6.0+1047+4 51/153 2023-03-03T07:54:25.024 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : postgresql-server-10.19-1.module_el8.6.0+1047+42 52/153 2023-03-03T07:54:25.024 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 53/153 2023-03-03T07:54:25.024 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 54/153 2023-03-03T07:54:25.025 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 55/153 2023-03-03T07:54:25.025 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 56/153 2023-03-03T07:54:25.025 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 57/153 2023-03-03T07:54:25.025 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 58/153 2023-03-03T07:54:25.025 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 59/153 2023-03-03T07:54:25.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 60/153 2023-03-03T07:54:25.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 61/153 2023-03-03T07:54:25.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 62/153 2023-03-03T07:54:25.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 63/153 2023-03-03T07:54:25.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 64/153 2023-03-03T07:54:25.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 65/153 2023-03-03T07:54:25.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 66/153 2023-03-03T07:54:25.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 67/153 2023-03-03T07:54:25.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 68/153 2023-03-03T07:54:25.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 69/153 2023-03-03T07:54:25.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/153 2023-03-03T07:54:25.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : elfutils-libelf-devel-0.188-3.el8.x86_64 71/153 2023-03-03T07:54:25.029 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 72/153 2023-03-03T07:54:25.029 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 73/153 2023-03-03T07:54:25.029 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 74/153 2023-03-03T07:54:25.029 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 75/153 2023-03-03T07:54:25.029 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 76/153 2023-03-03T07:54:25.030 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libgfortran-8.5.0-18.el8.x86_64 77/153 2023-03-03T07:54:25.030 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libquadmath-8.5.0-18.el8.x86_64 78/153 2023-03-03T07:54:25.030 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 79/153 2023-03-03T07:54:25.030 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libselinux-devel-2.9-6.el8.x86_64 80/153 2023-03-03T07:54:25.030 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 81/153 2023-03-03T07:54:25.031 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libuuid-devel-2.32.1-40.el8.x86_64 82/153 2023-03-03T07:54:25.031 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 83/153 2023-03-03T07:54:25.031 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8.x86_64 84/153 2023-03-03T07:54:25.031 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 85/153 2023-03-03T07:54:25.031 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 86/153 2023-03-03T07:54:25.032 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 87/153 2023-03-03T07:54:25.032 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 88/153 2023-03-03T07:54:25.032 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 89/153 2023-03-03T07:54:25.032 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 90/153 2023-03-03T07:54:25.033 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 91/153 2023-03-03T07:54:25.033 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 92/153 2023-03-03T07:54:25.033 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 93/153 2023-03-03T07:54:25.033 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 94/153 2023-03-03T07:54:25.033 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : zlib-devel-1.2.11-19.el8.x86_64 95/153 2023-03-03T07:54:25.034 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 96/153 2023-03-03T07:54:25.034 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 97/153 2023-03-03T07:54:25.034 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 98/153 2023-03-03T07:54:25.034 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 99/153 2023-03-03T07:54:25.034 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 100/153 2023-03-03T07:54:25.035 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 101/153 2023-03-03T07:54:25.035 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 102/153 2023-03-03T07:54:25.035 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 103/153 2023-03-03T07:54:25.035 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 104/153 2023-03-03T07:54:25.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 105/153 2023-03-03T07:54:25.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 106/153 2023-03-03T07:54:25.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 107/153 2023-03-03T07:54:25.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 108/153 2023-03-03T07:54:25.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 109/153 2023-03-03T07:54:25.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 110/153 2023-03-03T07:54:25.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 111/153 2023-03-03T07:54:25.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 112/153 2023-03-03T07:54:25.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 113/153 2023-03-03T07:54:25.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-portend-2.6-1.el8.noarch 114/153 2023-03-03T07:54:25.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 115/153 2023-03-03T07:54:25.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 116/153 2023-03-03T07:54:25.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 117/153 2023-03-03T07:54:25.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 118/153 2023-03-03T07:54:25.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 119/153 2023-03-03T07:54:25.039 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 120/153 2023-03-03T07:54:25.039 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 121/153 2023-03-03T07:54:25.039 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 122/153 2023-03-03T07:54:25.039 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 123/153 2023-03-03T07:54:25.039 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 124/153 2023-03-03T07:54:25.039 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 125/153 2023-03-03T07:54:25.040 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 126/153 2023-03-03T07:54:25.040 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 127/153 2023-03-03T07:54:25.040 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 128/153 2023-03-03T07:54:25.040 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 129/153 2023-03-03T07:54:25.040 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 130/153 2023-03-03T07:54:25.041 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 131/153 2023-03-03T07:54:25.041 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.12-1.el 132/153 2023-03-03T07:54:25.041 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : NetworkManager-initscripts-updown-1:1.39.12-1.el 133/153 2023-03-03T07:54:25.042 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : elfutils-debuginfod-client-0.188-3.el8.x86_64 134/153 2023-03-03T07:54:25.042 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : elfutils-debuginfod-client-0.187-4.el8.x86_64 135/153 2023-03-03T07:54:25.042 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : elfutils-libelf-0.188-3.el8.x86_64 136/153 2023-03-03T07:54:25.042 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : elfutils-libelf-0.187-4.el8.x86_64 137/153 2023-03-03T07:54:25.042 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : elfutils-libs-0.188-3.el8.x86_64 138/153 2023-03-03T07:54:25.043 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : elfutils-libs-0.187-4.el8.x86_64 139/153 2023-03-03T07:54:25.043 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libblkid-2.32.1-40.el8.x86_64 140/153 2023-03-03T07:54:25.043 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libblkid-2.32.1-38.el8.x86_64 141/153 2023-03-03T07:54:25.043 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libfdisk-2.32.1-40.el8.x86_64 142/153 2023-03-03T07:54:25.043 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libfdisk-2.32.1-38.el8.x86_64 143/153 2023-03-03T07:54:25.044 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libmount-2.32.1-40.el8.x86_64 144/153 2023-03-03T07:54:25.044 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libmount-2.32.1-38.el8.x86_64 145/153 2023-03-03T07:54:25.044 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libsmartcols-2.32.1-40.el8.x86_64 146/153 2023-03-03T07:54:25.044 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libsmartcols-2.32.1-38.el8.x86_64 147/153 2023-03-03T07:54:25.044 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libuuid-2.32.1-40.el8.x86_64 148/153 2023-03-03T07:54:25.045 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libuuid-2.32.1-38.el8.x86_64 149/153 2023-03-03T07:54:25.045 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : util-linux-2.32.1-40.el8.x86_64 150/153 2023-03-03T07:54:25.045 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : util-linux-2.32.1-38.el8.x86_64 151/153 2023-03-03T07:54:25.045 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : util-linux-user-2.32.1-40.el8.x86_64 152/153 2023-03-03T07:54:25.605 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 1/153 2023-03-03T07:54:25.606 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2/153 2023-03-03T07:54:25.606 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 3/153 2023-03-03T07:54:25.606 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 4/153 2023-03-03T07:54:25.606 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 5/153 2023-03-03T07:54:25.606 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 6/153 2023-03-03T07:54:25.606 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 7/153 2023-03-03T07:54:25.607 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 8/153 2023-03-03T07:54:25.607 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 9/153 2023-03-03T07:54:25.607 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 10/153 2023-03-03T07:54:25.607 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 11/153 2023-03-03T07:54:25.607 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 12/153 2023-03-03T07:54:25.607 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 13/153 2023-03-03T07:54:25.607 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 14/153 2023-03-03T07:54:25.607 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 15/153 2023-03-03T07:54:25.608 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 16/153 2023-03-03T07:54:25.608 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 17/153 2023-03-03T07:54:25.608 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/153 2023-03-03T07:54:25.608 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 19/153 2023-03-03T07:54:25.608 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 20/153 2023-03-03T07:54:25.608 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 21/153 2023-03-03T07:54:25.608 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 22/153 2023-03-03T07:54:25.609 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 23/153 2023-03-03T07:54:25.609 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 24/153 2023-03-03T07:54:25.609 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 25/153 2023-03-03T07:54:25.609 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 26/153 2023-03-03T07:54:25.609 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 27/153 2023-03-03T07:54:25.609 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 28/153 2023-03-03T07:54:25.609 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 29/153 2023-03-03T07:54:25.609 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 30/153 2023-03-03T07:54:25.610 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 31/153 2023-03-03T07:54:25.610 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 32/153 2023-03-03T07:54:25.610 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 33/153 2023-03-03T07:54:25.610 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 34/153 2023-03-03T07:54:25.610 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 35/153 2023-03-03T07:54:25.610 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 36/153 2023-03-03T07:54:25.610 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 37/153 2023-03-03T07:54:25.611 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 38/153 2023-03-03T07:54:25.611 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : automake-1.16.1-7.el8.noarch 39/153 2023-03-03T07:54:25.611 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/153 2023-03-03T07:54:25.611 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/153 2023-03-03T07:54:25.611 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : jq-1.6-6.el8.x86_64 42/153 2023-03-03T07:54:25.611 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpq-13.5-1.el8.x86_64 43/153 2023-03-03T07:54:25.611 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 44/153 2023-03-03T07:54:25.612 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 45/153 2023-03-03T07:54:25.612 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 46/153 2023-03-03T07:54:25.612 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 47/153 2023-03-03T07:54:25.612 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 48/153 2023-03-03T07:54:25.612 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 49/153 2023-03-03T07:54:25.612 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-10.19-1.module_el8.6.0+1047+4202cf9a. 50/153 2023-03-03T07:54:25.612 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-contrib-10.19-1.module_el8.6.0+1047+4 51/153 2023-03-03T07:54:25.612 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-server-10.19-1.module_el8.6.0+1047+42 52/153 2023-03-03T07:54:25.613 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 53/153 2023-03-03T07:54:25.613 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 54/153 2023-03-03T07:54:25.613 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 55/153 2023-03-03T07:54:25.613 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 56/153 2023-03-03T07:54:25.613 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 57/153 2023-03-03T07:54:25.613 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 58/153 2023-03-03T07:54:25.613 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 59/153 2023-03-03T07:54:25.614 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 60/153 2023-03-03T07:54:25.614 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 61/153 2023-03-03T07:54:25.614 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 62/153 2023-03-03T07:54:25.614 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 63/153 2023-03-03T07:54:25.614 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 64/153 2023-03-03T07:54:25.614 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 65/153 2023-03-03T07:54:25.614 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 66/153 2023-03-03T07:54:25.615 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 67/153 2023-03-03T07:54:25.615 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 68/153 2023-03-03T07:54:25.615 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 69/153 2023-03-03T07:54:25.615 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/153 2023-03-03T07:54:25.615 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-devel-0.188-3.el8.x86_64 71/153 2023-03-03T07:54:25.615 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 72/153 2023-03-03T07:54:25.615 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 73/153 2023-03-03T07:54:25.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 74/153 2023-03-03T07:54:25.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 75/153 2023-03-03T07:54:25.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 76/153 2023-03-03T07:54:25.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-8.5.0-18.el8.x86_64 77/153 2023-03-03T07:54:25.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-8.5.0-18.el8.x86_64 78/153 2023-03-03T07:54:25.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 79/153 2023-03-03T07:54:25.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libselinux-devel-2.9-6.el8.x86_64 80/153 2023-03-03T07:54:25.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 81/153 2023-03-03T07:54:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-devel-2.32.1-40.el8.x86_64 82/153 2023-03-03T07:54:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 83/153 2023-03-03T07:54:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8.x86_64 84/153 2023-03-03T07:54:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 85/153 2023-03-03T07:54:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 86/153 2023-03-03T07:54:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 87/153 2023-03-03T07:54:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 88/153 2023-03-03T07:54:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 89/153 2023-03-03T07:54:25.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 90/153 2023-03-03T07:54:25.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 91/153 2023-03-03T07:54:25.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 92/153 2023-03-03T07:54:25.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 93/153 2023-03-03T07:54:25.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 94/153 2023-03-03T07:54:25.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : zlib-devel-1.2.11-19.el8.x86_64 95/153 2023-03-03T07:54:25.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 96/153 2023-03-03T07:54:25.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 97/153 2023-03-03T07:54:25.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 98/153 2023-03-03T07:54:25.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 99/153 2023-03-03T07:54:25.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 100/153 2023-03-03T07:54:25.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 101/153 2023-03-03T07:54:25.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 102/153 2023-03-03T07:54:25.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 103/153 2023-03-03T07:54:25.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 104/153 2023-03-03T07:54:25.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 105/153 2023-03-03T07:54:25.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 106/153 2023-03-03T07:54:25.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 107/153 2023-03-03T07:54:25.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 108/153 2023-03-03T07:54:25.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 109/153 2023-03-03T07:54:25.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 110/153 2023-03-03T07:54:25.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 111/153 2023-03-03T07:54:25.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 112/153 2023-03-03T07:54:25.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 113/153 2023-03-03T07:54:25.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-2.6-1.el8.noarch 114/153 2023-03-03T07:54:25.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 115/153 2023-03-03T07:54:25.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 116/153 2023-03-03T07:54:25.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 117/153 2023-03-03T07:54:25.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 118/153 2023-03-03T07:54:25.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 119/153 2023-03-03T07:54:25.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 120/153 2023-03-03T07:54:25.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 121/153 2023-03-03T07:54:25.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 122/153 2023-03-03T07:54:25.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 123/153 2023-03-03T07:54:25.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 124/153 2023-03-03T07:54:25.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 125/153 2023-03-03T07:54:25.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 126/153 2023-03-03T07:54:25.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 127/153 2023-03-03T07:54:25.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 128/153 2023-03-03T07:54:25.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 129/153 2023-03-03T07:54:25.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 130/153 2023-03-03T07:54:25.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 131/153 2023-03-03T07:54:25.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.12-1.el 132/153 2023-03-03T07:54:25.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-initscripts-updown-1:1.39.12-1.el 133/153 2023-03-03T07:54:25.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.188-3.el8.x86_64 134/153 2023-03-03T07:54:25.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.187-4.el8.x86_64 135/153 2023-03-03T07:54:25.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.188-3.el8.x86_64 136/153 2023-03-03T07:54:25.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.187-4.el8.x86_64 137/153 2023-03-03T07:54:25.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.188-3.el8.x86_64 138/153 2023-03-03T07:54:25.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.187-4.el8.x86_64 139/153 2023-03-03T07:54:25.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libblkid-2.32.1-40.el8.x86_64 140/153 2023-03-03T07:54:25.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libblkid-2.32.1-38.el8.x86_64 141/153 2023-03-03T07:54:25.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libfdisk-2.32.1-40.el8.x86_64 142/153 2023-03-03T07:54:25.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libfdisk-2.32.1-38.el8.x86_64 143/153 2023-03-03T07:54:25.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libmount-2.32.1-40.el8.x86_64 144/153 2023-03-03T07:54:25.626 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libmount-2.32.1-38.el8.x86_64 145/153 2023-03-03T07:54:25.626 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsmartcols-2.32.1-40.el8.x86_64 146/153 2023-03-03T07:54:25.626 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsmartcols-2.32.1-38.el8.x86_64 147/153 2023-03-03T07:54:25.626 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-2.32.1-40.el8.x86_64 148/153 2023-03-03T07:54:25.626 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-2.32.1-38.el8.x86_64 149/153 2023-03-03T07:54:25.626 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-2.32.1-40.el8.x86_64 150/153 2023-03-03T07:54:25.627 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-2.32.1-38.el8.x86_64 151/153 2023-03-03T07:54:25.627 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-user-2.32.1-40.el8.x86_64 152/153 2023-03-03T07:54:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : util-linux-user-2.32.1-38.el8.x86_64 153/153 2023-03-03T07:54:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:27.057 INFO:teuthology.orchestra.run.smithi100.stdout:Upgraded: 2023-03-03T07:54:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: NetworkManager-initscripts-updown-1:1.40.12-1.el8.noarch 2023-03-03T07:54:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-debuginfod-client-0.188-3.el8.x86_64 2023-03-03T07:54:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-libelf-0.188-3.el8.x86_64 2023-03-03T07:54:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-libs-0.188-3.el8.x86_64 2023-03-03T07:54:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: libblkid-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: libfdisk-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: libmount-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: libsmartcols-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: libuuid-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: util-linux-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: util-linux-user-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.059 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2023-03-03T07:54:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: automake-1.16.1-7.el8.noarch 2023-03-03T07:54:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: bison-3.0.4-10.el8.x86_64 2023-03-03T07:54:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.063 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.063 INFO:teuthology.orchestra.run.smithi100.stdout: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.063 INFO:teuthology.orchestra.run.smithi100.stdout: cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.063 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-libelf-devel-0.188-3.el8.x86_64 2023-03-03T07:54:27.063 INFO:teuthology.orchestra.run.smithi100.stdout: flex-2.6.1-9.el8.x86_64 2023-03-03T07:54:27.063 INFO:teuthology.orchestra.run.smithi100.stdout: fuse-2.9.7-16.el8.x86_64 2023-03-03T07:54:27.063 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-03-03T07:54:27.063 INFO:teuthology.orchestra.run.smithi100.stdout: jq-1.6-6.el8.x86_64 2023-03-03T07:54:27.064 INFO:teuthology.orchestra.run.smithi100.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-03-03T07:54:27.064 INFO:teuthology.orchestra.run.smithi100.stdout: krb5-devel-1.18.2-22.el8.x86_64 2023-03-03T07:54:27.064 INFO:teuthology.orchestra.run.smithi100.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-03-03T07:54:27.064 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.064 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.064 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.064 INFO:teuthology.orchestra.run.smithi100.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2023-03-03T07:54:27.065 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran-8.5.0-18.el8.x86_64 2023-03-03T07:54:27.065 INFO:teuthology.orchestra.run.smithi100.stdout: liboath-2.6.2-3.el8.x86_64 2023-03-03T07:54:27.065 INFO:teuthology.orchestra.run.smithi100.stdout: libpq-13.5-1.el8.x86_64 2023-03-03T07:54:27.065 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath-8.5.0-18.el8.x86_64 2023-03-03T07:54:27.065 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-03-03T07:54:27.065 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.065 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.066 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-03-03T07:54:27.066 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.066 INFO:teuthology.orchestra.run.smithi100.stdout: libselinux-devel-2.9-6.el8.x86_64 2023-03-03T07:54:27.066 INFO:teuthology.orchestra.run.smithi100.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-03-03T07:54:27.066 INFO:teuthology.orchestra.run.smithi100.stdout: libtool-2.4.6-25.el8.x86_64 2023-03-03T07:54:27.066 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind-1.3.1-3.el8.x86_64 2023-03-03T07:54:27.066 INFO:teuthology.orchestra.run.smithi100.stdout: libuuid-devel-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.066 INFO:teuthology.orchestra.run.smithi100.stdout: libverto-devel-0.3.2-2.el8.x86_64 2023-03-03T07:54:27.067 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-03-03T07:54:27.067 INFO:teuthology.orchestra.run.smithi100.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-03-03T07:54:27.067 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-0.3.15-4.el8.x86_64 2023-03-03T07:54:27.067 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-03-03T07:54:27.067 INFO:teuthology.orchestra.run.smithi100.stdout: openssl-devel-1:1.1.1k-7.el8.x86_64 2023-03-03T07:54:27.067 INFO:teuthology.orchestra.run.smithi100.stdout: pcre2-devel-10.32-3.el8.x86_64 2023-03-03T07:54:27.067 INFO:teuthology.orchestra.run.smithi100.stdout: pcre2-utf16-10.32-3.el8.x86_64 2023-03-03T07:54:27.067 INFO:teuthology.orchestra.run.smithi100.stdout: pcre2-utf32-10.32-3.el8.x86_64 2023-03-03T07:54:27.068 INFO:teuthology.orchestra.run.smithi100.stdout: postgresql-10.19-1.module_el8.6.0+1047+4202cf9a.x86_64 2023-03-03T07:54:27.068 INFO:teuthology.orchestra.run.smithi100.stdout: postgresql-contrib-10.19-1.module_el8.6.0+1047+4202cf9a.x86_64 2023-03-03T07:54:27.068 INFO:teuthology.orchestra.run.smithi100.stdout: postgresql-server-10.19-1.module_el8.6.0+1047+4202cf9a.x86_64 2023-03-03T07:54:27.068 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-03-03T07:54:27.068 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel-2.5.1-7.el8.noarch 2023-03-03T07:54:27.068 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-03-03T07:54:27.068 INFO:teuthology.orchestra.run.smithi100.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-03-03T07:54:27.069 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-03-03T07:54:27.069 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.069 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.069 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.069 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-03-03T07:54:27.069 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-03-03T07:54:27.069 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-03-03T07:54:27.069 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-03-03T07:54:27.070 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-03-03T07:54:27.070 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-03-03T07:54:27.070 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-6.2-6.el8.noarch 2023-03-03T07:54:27.070 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-03-03T07:54:27.070 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-03-03T07:54:27.070 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-03-03T07:54:27.070 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-03-03T07:54:27.070 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-03-03T07:54:27.071 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-03-03T07:54:27.071 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-03-03T07:54:27.071 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako-1.0.6-14.el8.noarch 2023-03-03T07:54:27.071 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-03-03T07:54:27.071 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-03-03T07:54:27.071 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-03-03T07:54:27.071 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-03-03T07:54:27.072 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-03-03T07:54:27.072 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-03-03T07:54:27.072 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-03-03T07:54:27.072 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend-2.6-1.el8.noarch 2023-03-03T07:54:27.072 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-03-03T07:54:27.072 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-03-03T07:54:27.072 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-03-03T07:54:27.072 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-03-03T07:54:27.073 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser-2.14-14.el8.noarch 2023-03-03T07:54:27.073 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz-2017.2-9.el8.noarch 2023-03-03T07:54:27.073 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.073 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.073 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-03-03T07:54:27.074 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-03-03T07:54:27.074 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.074 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes-2.4.1-12.el8.noarch 2023-03-03T07:54:27.074 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa-4.8-1.el8.noarch 2023-03-03T07:54:27.074 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-03-03T07:54:27.074 INFO:teuthology.orchestra.run.smithi100.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-03-03T07:54:27.074 INFO:teuthology.orchestra.run.smithi100.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-03-03T07:54:27.074 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-03-03T07:54:27.075 INFO:teuthology.orchestra.run.smithi100.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-03-03T07:54:27.075 INFO:teuthology.orchestra.run.smithi100.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-03-03T07:54:27.075 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-03-03T07:54:27.075 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-03-03T07:54:27.075 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-03-03T07:54:27.075 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-03-03T07:54:27.075 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-03-03T07:54:27.075 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.076 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.076 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.076 INFO:teuthology.orchestra.run.smithi100.stdout: socat-1.7.4.1-1.el8.x86_64 2023-03-03T07:54:27.076 INFO:teuthology.orchestra.run.smithi100.stdout: thrift-0.13.0-2.el8.x86_64 2023-03-03T07:54:27.076 INFO:teuthology.orchestra.run.smithi100.stdout: uuid-1.6.2-43.el8.x86_64 2023-03-03T07:54:27.076 INFO:teuthology.orchestra.run.smithi100.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-03-03T07:54:27.076 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-03-03T07:54:27.077 INFO:teuthology.orchestra.run.smithi100.stdout: zlib-devel-1.2.11-19.el8.x86_64 2023-03-03T07:54:27.077 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:27.082 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2023-03-03T07:54:27.435 DEBUG:teuthology.parallel:result is None 2023-03-03T07:54:27.613 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-user-2.32.1-38.el8.x86_64 153/153 2023-03-03T07:54:27.614 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:27.614 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2023-03-03T07:54:27.614 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-initscripts-updown-1:1.40.12-1.el8.noarch 2023-03-03T07:54:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client-0.188-3.el8.x86_64 2023-03-03T07:54:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-0.188-3.el8.x86_64 2023-03-03T07:54:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs-0.188-3.el8.x86_64 2023-03-03T07:54:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: libblkid-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.617 INFO:teuthology.orchestra.run.smithi098.stdout: libfdisk-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: libmount-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: libsmartcols-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-user-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.619 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2023-03-03T07:54:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: automake-1.16.1-7.el8.noarch 2023-03-03T07:54:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: bison-3.0.4-10.el8.x86_64 2023-03-03T07:54:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T07:54:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel-0.188-3.el8.x86_64 2023-03-03T07:54:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: flex-2.6.1-9.el8.x86_64 2023-03-03T07:54:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: fuse-2.9.7-16.el8.x86_64 2023-03-03T07:54:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-03-03T07:54:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: jq-1.6-6.el8.x86_64 2023-03-03T07:54:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-03-03T07:54:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: krb5-devel-1.18.2-22.el8.x86_64 2023-03-03T07:54:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-03-03T07:54:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2023-03-03T07:54:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-8.5.0-18.el8.x86_64 2023-03-03T07:54:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.2-3.el8.x86_64 2023-03-03T07:54:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: libpq-13.5-1.el8.x86_64 2023-03-03T07:54:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-8.5.0-18.el8.x86_64 2023-03-03T07:54:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-03-03T07:54:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-03-03T07:54:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: libselinux-devel-2.9-6.el8.x86_64 2023-03-03T07:54:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-03-03T07:54:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: libtool-2.4.6-25.el8.x86_64 2023-03-03T07:54:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.3.1-3.el8.x86_64 2023-03-03T07:54:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel-2.32.1-40.el8.x86_64 2023-03-03T07:54:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: libverto-devel-0.3.2-2.el8.x86_64 2023-03-03T07:54:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-03-03T07:54:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-03-03T07:54:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.15-4.el8.x86_64 2023-03-03T07:54:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-03-03T07:54:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel-1:1.1.1k-7.el8.x86_64 2023-03-03T07:54:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-devel-10.32-3.el8.x86_64 2023-03-03T07:54:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf16-10.32-3.el8.x86_64 2023-03-03T07:54:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf32-10.32-3.el8.x86_64 2023-03-03T07:54:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-10.19-1.module_el8.6.0+1047+4202cf9a.x86_64 2023-03-03T07:54:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib-10.19-1.module_el8.6.0+1047+4202cf9a.x86_64 2023-03-03T07:54:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server-10.19-1.module_el8.6.0+1047+4202cf9a.x86_64 2023-03-03T07:54:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-03-03T07:54:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.5.1-7.el8.noarch 2023-03-03T07:54:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-03-03T07:54:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-03-03T07:54:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-03-03T07:54:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-03-03T07:54:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-03-03T07:54:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-03-03T07:54:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-03-03T07:54:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-03-03T07:54:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-03-03T07:54:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-6.2-6.el8.noarch 2023-03-03T07:54:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-03-03T07:54:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-03-03T07:54:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-03-03T07:54:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-03-03T07:54:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-03-03T07:54:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-03-03T07:54:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-03-03T07:54:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.0.6-14.el8.noarch 2023-03-03T07:54:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-03-03T07:54:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-03-03T07:54:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-03-03T07:54:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-03-03T07:54:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-03-03T07:54:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-03-03T07:54:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-03-03T07:54:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-2.6-1.el8.noarch 2023-03-03T07:54:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-03-03T07:54:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-03-03T07:54:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-03-03T07:54:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-03-03T07:54:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.14-14.el8.noarch 2023-03-03T07:54:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2017.2-9.el8.noarch 2023-03-03T07:54:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-03-03T07:54:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-03-03T07:54:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.4.1-12.el8.noarch 2023-03-03T07:54:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.8-1.el8.noarch 2023-03-03T07:54:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-03-03T07:54:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-03-03T07:54:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-03-03T07:54:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-03-03T07:54:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-03-03T07:54:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-03-03T07:54:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-03-03T07:54:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-03-03T07:54:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-03-03T07:54:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-03-03T07:54:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-03-03T07:54:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T07:54:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-1.el8.x86_64 2023-03-03T07:54:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.13.0-2.el8.x86_64 2023-03-03T07:54:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: uuid-1.6.2-43.el8.x86_64 2023-03-03T07:54:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-03-03T07:54:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-03-03T07:54:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel-1.2.11-19.el8.x86_64 2023-03-03T07:54:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:27.637 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2023-03-03T07:54:27.987 DEBUG:teuthology.parallel:result is None 2023-03-03T07:54:27.988 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T07:54:28.130 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-03-03T07:54:28.216 INFO:teuthology.orchestra.run.smithi098.stdout:18.0.0-2739.g472aaef4.el8 2023-03-03T07:54:28.217 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-2739.g472aaef4.el8 2023-03-03T07:54:28.217 INFO:teuthology.task.install:The correct ceph version 18.0.0-2739.g472aaef4 is installed. 2023-03-03T07:54:28.217 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T07:54:28.355 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-03-03T07:54:28.438 INFO:teuthology.orchestra.run.smithi100.stdout:18.0.0-2739.g472aaef4.el8 2023-03-03T07:54:28.438 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-2739.g472aaef4.el8 2023-03-03T07:54:28.439 INFO:teuthology.task.install:The correct ceph version 18.0.0-2739.g472aaef4 is installed. 2023-03-03T07:54:28.439 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-03-03T07:54:28.440 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:54:28.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-03T07:54:28.549 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:54:28.550 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-03T07:54:28.656 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-03-03T07:54:28.657 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:54:28.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2023-03-03T07:54:28.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-03T07:54:28.868 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:54:28.868 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/daemon-helper 2023-03-03T07:54:28.971 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-03T07:54:29.112 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-03-03T07:54:29.113 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:54:29.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-03T07:54:29.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-03T07:54:29.360 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:54:29.360 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-03T07:54:29.463 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-03T07:54:29.611 INFO:teuthology.run_tasks:Running task ceph... 2023-03-03T07:54:29.684 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-03-03T07:54:29.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2023-03-03T07:54:29.690 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 777 /var/log/ceph 2023-03-03T07:54:29.793 INFO:tasks.ceph:Disabling ceph logrotate... 2023-03-03T07:54:29.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2023-03-03T07:54:29.798 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/logrotate.d/ceph 2023-03-03T07:54:29.901 INFO:tasks.ceph:Creating extra log directories... 2023-03-03T07:54:29.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-03-03T07:54:29.908 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-03-03T07:54:30.020 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-03-03T07:54:30.020 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-03-03T07:54:30.020 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-03-03T07:54:30.021 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:54:30.021 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-03-03T07:54:30.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-03-03T07:54:30.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-03-03T07:54:30.454 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:54:30.454 DEBUG:teuthology.orchestra.run.smithi100:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-03-03T07:54:30.525 DEBUG:teuthology.orchestra.run.smithi100:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-03-03T07:54:30.777 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-03-03T07:54:30.887 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-03-03T07:54:30.887 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-03-03T07:54:30.887 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192219', 'branch': 'wip-vshankar-testing-20230303.023823', 'description': 'fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7192219', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '75534f74ac3dd06f86697ff862e8be2945b3c8e6'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20230303.023823'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20230303.023823', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20230303.023823', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi098.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaRdTmieDuhW/Jy25O4OW+z0Jazsj4LLoSTSi1R6l8MQAS5EVbl9ec3LToblzoQUHbcAH0gpcvT5pqOjR+nEL6xUcC7sxyanhz83kr8w43S4lduFjrFuKu9sq4nvRHjerv1mJg7efaEU+aBlyIwV5yNcV4+KPrO3MpykxS3a5UMu9LlxA9QdM92+a9m8ALEczNsGCJuAvnpohmMR2D5uX526xV7VR/WEd9lPzbpMtHvPJF2CfBH8iA+fh/Yqy8cPeDXVMC3ct38QadokM5v+P+kXT1FA+vUU9+4CqAAQ/xWBGb9t3su4dySRmFR7CNM/kdGwdLRIhvhQdmO35ZoQzCr5rF9T0mrzKh0aAUC7LhhZnEkGA1V/h17ID8xZNlDN1HDlix8+CYQaoUuyb3h8n3lnVWQuQzXOkmzdndYwjEM30KS7cRmMH5EpQszfFK1L5WKFcsYXPSRT1Oa8tI2+6j+X3oKQx9RqaVAmzJj+0W0MXbVu/jSkkFGwa6gc5mRP8=', 'smithi100.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdlRjeXhUuVMVhbuUuPs9Rcclc5plYkzocTQkQjkpuKAPmjqQwGxv26ijTEOD/wP6UPiTEvDt+ABzrTOF764eu8U+bXxOvENK8SsmB3i+RAXC9ehZ0sSFcJQATAYYDjDk3b5Jm8Ob4sD7tQC6MI6h68QOQL9C5TpKfGB3iRkYOTqBVX2ubohCcHkL8NbEhnNH0ieqHON1Gc8ERgLeUZzrovgwDNlB3O+LLwVBlx/XhY8C4pz0kPTirD803jteDd4iK98x8xIFEPpiUtjlF70BS49jD7/dIuLGdXbUQLV1+xJDQzJux+X3x32DmAJvharMSYF6thaypeManV5tcXir/WUMu2obBDAFaOb8uwR/c89h6stqXSWrm/++wuhurjmrsr829GI3DlZr2qoNjIEga4kbePkkcDpmXQDCt1BADQ2g9vs10ko5620O3L77MZ/8Hup6qdQkhAMn72e8JbMkodMvkFmZcB15Kgv5tFiYYhH07SFNa7ZzWoavr095XEvk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '75534f74ac3dd06f86697ff862e8be2945b3c8e6'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestMisc']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6', 'timestamp': '2023-03-03_04:39:14', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.31660'} 2023-03-03T07:54:30.888 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-03-03T07:54:30.893 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-03-03T07:54:30.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2023-03-03T07:54:30.968 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m0777 -- /var/run/ceph 2023-03-03T07:54:31.069 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:54:31.070 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2023-03-03T07:54:31.139 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-03T07:54:31.140 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2023-03-03T07:54:31.209 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-03T07:54:31.210 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T07:54:31.210 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 448 Links: 1 2023-03-03T07:54:31.210 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T07:54:31.210 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T07:54:31.210 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-03-03 07:54:24.935524184 +0000 2023-03-03T07:54:31.211 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-03-03 07:50:46.673902518 +0000 2023-03-03T07:54:31.211 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-03-03 07:50:46.673902518 +0000 2023-03-03T07:54:31.211 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2023-03-03 07:50:46.635902959 +0000 2023-03-03T07:54:31.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-03T07:54:31.307 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-03-03T07:54:31.308 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-03-03T07:54:31.308 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000332749 s, 1.5 MB/s 2023-03-03T07:54:31.312 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-03T07:54:31.387 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2023-03-03T07:54:31.458 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-03T07:54:31.458 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T07:54:31.458 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 454 Links: 1 2023-03-03T07:54:31.459 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T07:54:31.459 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T07:54:31.459 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-03-03 07:54:24.935524184 +0000 2023-03-03T07:54:31.459 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-03-03 07:50:47.764889857 +0000 2023-03-03T07:54:31.459 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-03-03 07:50:47.764889857 +0000 2023-03-03T07:54:31.459 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2023-03-03 07:50:47.734890205 +0000 2023-03-03T07:54:31.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-03T07:54:31.558 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-03-03T07:54:31.559 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-03-03T07:54:31.559 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.0003143 s, 1.6 MB/s 2023-03-03T07:54:31.563 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-03T07:54:31.639 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2023-03-03T07:54:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-03T07:54:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T07:54:31.710 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 460 Links: 1 2023-03-03T07:54:31.710 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T07:54:31.710 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T07:54:31.710 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-03-03 07:54:24.935524184 +0000 2023-03-03T07:54:31.710 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-03-03 07:50:48.840877371 +0000 2023-03-03T07:54:31.710 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-03-03 07:50:48.840877371 +0000 2023-03-03T07:54:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2023-03-03 07:50:48.808877742 +0000 2023-03-03T07:54:31.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-03T07:54:31.807 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-03-03T07:54:31.808 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-03-03T07:54:31.808 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000359597 s, 1.4 MB/s 2023-03-03T07:54:31.812 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-03T07:54:31.884 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2023-03-03T07:54:31.953 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-03T07:54:31.954 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T07:54:31.954 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 466 Links: 1 2023-03-03T07:54:31.954 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T07:54:31.954 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T07:54:31.954 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-03-03 07:54:24.936524172 +0000 2023-03-03T07:54:31.954 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-03-03 07:50:49.977864176 +0000 2023-03-03T07:54:31.955 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-03-03 07:50:49.977864176 +0000 2023-03-03T07:54:31.955 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2023-03-03 07:50:49.944864559 +0000 2023-03-03T07:54:31.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-03T07:54:32.056 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-03-03T07:54:32.056 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-03-03T07:54:32.057 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000362477 s, 1.4 MB/s 2023-03-03T07:54:32.063 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-03T07:54:32.136 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T07:54:32.137 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:54:32.137 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/scratch_devs of=/dev/stdout 2023-03-03T07:54:32.209 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-03T07:54:32.209 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_1 2023-03-03T07:54:32.283 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-03T07:54:32.283 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T07:54:32.283 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 448 Links: 1 2023-03-03T07:54:32.283 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T07:54:32.284 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T07:54:32.284 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2023-03-03 07:54:24.314448334 +0000 2023-03-03T07:54:32.284 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2023-03-03 07:50:46.689796140 +0000 2023-03-03T07:54:32.284 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2023-03-03 07:50:46.689796140 +0000 2023-03-03T07:54:32.284 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2023-03-03 07:50:46.652797093 +0000 2023-03-03T07:54:32.285 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-03T07:54:32.384 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2023-03-03T07:54:32.385 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2023-03-03T07:54:32.385 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000321164 s, 1.6 MB/s 2023-03-03T07:54:32.391 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-03T07:54:32.465 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_2 2023-03-03T07:54:32.535 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-03T07:54:32.536 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T07:54:32.536 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 454 Links: 1 2023-03-03T07:54:32.536 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T07:54:32.536 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T07:54:32.536 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2023-03-03 07:54:24.314448334 +0000 2023-03-03T07:54:32.537 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2023-03-03 07:50:47.798767569 +0000 2023-03-03T07:54:32.537 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2023-03-03 07:50:47.798767569 +0000 2023-03-03T07:54:32.537 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2023-03-03 07:50:47.763768471 +0000 2023-03-03T07:54:32.538 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-03T07:54:32.641 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2023-03-03T07:54:32.641 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2023-03-03T07:54:32.641 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000482628 s, 1.1 MB/s 2023-03-03T07:54:32.646 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-03T07:54:32.723 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_3 2023-03-03T07:54:32.791 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-03T07:54:32.792 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T07:54:32.792 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 460 Links: 1 2023-03-03T07:54:32.792 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T07:54:32.792 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T07:54:32.792 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2023-03-03 07:54:24.315448308 +0000 2023-03-03T07:54:32.792 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2023-03-03 07:50:48.873739875 +0000 2023-03-03T07:54:32.793 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2023-03-03 07:50:48.873739875 +0000 2023-03-03T07:54:32.793 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2023-03-03 07:50:48.842740674 +0000 2023-03-03T07:54:32.793 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-03T07:54:32.895 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2023-03-03T07:54:32.895 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2023-03-03T07:54:32.896 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000309482 s, 1.7 MB/s 2023-03-03T07:54:32.899 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-03T07:54:32.974 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_4 2023-03-03T07:54:33.046 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-03T07:54:33.047 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T07:54:33.047 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 466 Links: 1 2023-03-03T07:54:33.047 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T07:54:33.048 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T07:54:33.048 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2023-03-03 07:54:24.315448308 +0000 2023-03-03T07:54:33.048 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2023-03-03 07:50:49.957711949 +0000 2023-03-03T07:54:33.048 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2023-03-03 07:50:49.957711949 +0000 2023-03-03T07:54:33.048 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2023-03-03 07:50:49.918712954 +0000 2023-03-03T07:54:33.049 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-03T07:54:33.152 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2023-03-03T07:54:33.153 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2023-03-03T07:54:33.153 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000334044 s, 1.5 MB/s 2023-03-03T07:54:33.156 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-03T07:54:33.232 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T07:54:33.232 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi100.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-03-03T07:54:33.233 INFO:tasks.ceph:Generating config... 2023-03-03T07:54:33.234 INFO:tasks.ceph:[client] client mount timeout = 600 2023-03-03T07:54:33.234 INFO:tasks.ceph:[client] debug client = 20 2023-03-03T07:54:33.234 INFO:tasks.ceph:[client] debug ms = 1 2023-03-03T07:54:33.235 INFO:tasks.ceph:[client] pid file = 2023-03-03T07:54:33.235 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-03-03T07:54:33.235 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-03-03T07:54:33.235 INFO:tasks.ceph:[mds] debug mds = 20 2023-03-03T07:54:33.235 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-03-03T07:54:33.236 INFO:tasks.ceph:[mds] debug ms = 1 2023-03-03T07:54:33.236 INFO:tasks.ceph:[mds] mds debug frag = True 2023-03-03T07:54:33.236 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-03-03T07:54:33.236 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-03-03T07:54:33.236 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-03-03T07:54:33.237 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-03-03T07:54:33.237 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-03-03T07:54:33.237 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-03-03T07:54:33.237 INFO:tasks.ceph:[mgr] debug client = 20 2023-03-03T07:54:33.237 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-03-03T07:54:33.238 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-03-03T07:54:33.238 INFO:tasks.ceph:[mgr] debug ms = 1 2023-03-03T07:54:33.238 INFO:tasks.ceph:[mon] debug mon = 20 2023-03-03T07:54:33.238 INFO:tasks.ceph:[mon] debug ms = 1 2023-03-03T07:54:33.238 INFO:tasks.ceph:[mon] debug paxos = 20 2023-03-03T07:54:33.239 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-03-03T07:54:33.239 INFO:tasks.ceph:[osd] bdev async discard = True 2023-03-03T07:54:33.239 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-03-03T07:54:33.239 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-03-03T07:54:33.239 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-03-03T07:54:33.240 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-03-03T07:54:33.240 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-03-03T07:54:33.240 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-03-03T07:54:33.240 INFO:tasks.ceph:[osd] debug ms = 1 2023-03-03T07:54:33.240 INFO:tasks.ceph:[osd] debug osd = 20 2023-03-03T07:54:33.241 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-03-03T07:54:33.241 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-03-03T07:54:33.241 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-03-03T07:54:33.241 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-03-03T07:54:33.241 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-03-03T07:54:33.242 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-03-03T07:54:33.242 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-03-03T07:54:33.242 INFO:tasks.ceph:Setting up mon.a... 2023-03-03T07:54:33.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-03-03T07:54:33.384 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.keyring 2023-03-03T07:54:33.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-03-03T07:54:33.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-03T07:54:33.653 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.98'), ('mon.b', '172.21.15.100'), ('mon.c', '[v2:172.21.15.100:3301,v1:172.21.15.100:6790]')] 2023-03-03T07:54:33.654 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.98,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-03-03T07:54:33.655 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:54:33.655 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-03-03T07:54:33.728 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.98 --add b 172.21.15.100 --addv c '[v2:172.21.15.100:3301,v1:172.21.15.100:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-03-03T07:54:33.851 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-03-03T07:54:33.851 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: generated fsid fbbd322f-236a-44f8-96c5-be6963846c9b 2023-03-03T07:54:33.852 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = pacific 2023-03-03T07:54:33.852 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 0 2023-03-03T07:54:33.852 INFO:teuthology.orchestra.run.smithi098.stdout:fsid fbbd322f-236a-44f8-96c5-be6963846c9b 2023-03-03T07:54:33.852 INFO:teuthology.orchestra.run.smithi098.stdout:last_changed 2023-03-03T07:54:33.850995+0000 2023-03-03T07:54:33.852 INFO:teuthology.orchestra.run.smithi098.stdout:created 2023-03-03T07:54:33.850995+0000 2023-03-03T07:54:33.852 INFO:teuthology.orchestra.run.smithi098.stdout:min_mon_release 16 (pacific) 2023-03-03T07:54:33.852 INFO:teuthology.orchestra.run.smithi098.stdout:election_strategy: 1 2023-03-03T07:54:33.853 INFO:teuthology.orchestra.run.smithi098.stdout:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2023-03-03T07:54:33.853 INFO:teuthology.orchestra.run.smithi098.stdout:1: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.b 2023-03-03T07:54:33.853 INFO:teuthology.orchestra.run.smithi098.stdout:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2023-03-03T07:54:33.853 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-03-03T07:54:33.857 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-03-03T07:54:33.924 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID fbbd322f-236a-44f8-96c5-be6963846c9b... 2023-03-03T07:54:33.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-03-03T07:54:33.932 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-03-03T07:54:34.094 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2023-03-03T07:54:34.094 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2023-03-03T07:54:34.094 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-03-03T07:54:34.094 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2023-03-03T07:54:34.095 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.095 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2023-03-03T07:54:34.095 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.095 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2023-03-03T07:54:34.095 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.095 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2023-03-03T07:54:34.095 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2023-03-03T07:54:34.095 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.096 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2023-03-03T07:54:34.096 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2023-03-03T07:54:34.096 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.096 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2023-03-03T07:54:34.096 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2023-03-03T07:54:34.096 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.096 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2023-03-03T07:54:34.097 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2023-03-03T07:54:34.097 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.097 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2023-03-03T07:54:34.097 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.097 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2023-03-03T07:54:34.097 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2023-03-03T07:54:34.097 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2023-03-03T07:54:34.097 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2023-03-03T07:54:34.098 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2023-03-03T07:54:34.098 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2023-03-03T07:54:34.098 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2023-03-03T07:54:34.098 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-03-03T07:54:34.098 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2023-03-03T07:54:34.098 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2023-03-03T07:54:34.098 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.099 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-03-03T07:54:34.099 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.099 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2023-03-03T07:54:34.099 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.100 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2023-03-03T07:54:34.100 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.100 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2023-03-03T07:54:34.100 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2023-03-03T07:54:34.100 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2023-03-03T07:54:34.100 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = "172.21.15.98,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2023-03-03T07:54:34.100 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = fbbd322f-236a-44f8-96c5-be6963846c9b 2023-03-03T07:54:34.101 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.101 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2023-03-03T07:54:34.101 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2023-03-03T07:54:34.101 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.101 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2023-03-03T07:54:34.101 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2023-03-03T07:54:34.101 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.101 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2023-03-03T07:54:34.102 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2023-03-03T07:54:34.102 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.102 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2023-03-03T07:54:34.102 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2023-03-03T07:54:34.102 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2023-03-03T07:54:34.102 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.102 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2023-03-03T07:54:34.102 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2023-03-03T07:54:34.103 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.103 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2023-03-03T07:54:34.103 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.103 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2023-03-03T07:54:34.103 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.103 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2023-03-03T07:54:34.103 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.104 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2023-03-03T07:54:34.104 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2023-03-03T07:54:34.104 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.104 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2023-03-03T07:54:34.104 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2023-03-03T07:54:34.104 INFO:teuthology.orchestra.run.smithi098.stdout: bdev async discard = True 2023-03-03T07:54:34.104 INFO:teuthology.orchestra.run.smithi098.stdout: bdev enable discard = True 2023-03-03T07:54:34.104 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore allocator = bitmap 2023-03-03T07:54:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2023-03-03T07:54:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2023-03-03T07:54:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 1/20 2023-03-03T07:54:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 1/20 2023-03-03T07:54:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2023-03-03T07:54:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2023-03-03T07:54:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 4/10 2023-03-03T07:54:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2023-03-03T07:54:34.106 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2023-03-03T07:54:34.106 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2023-03-03T07:54:34.106 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2023-03-03T07:54:34.106 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2023-03-03T07:54:34.106 INFO:teuthology.orchestra.run.smithi098.stdout: osd op complaint time = 180 2023-03-03T07:54:34.106 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.106 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2023-03-03T07:54:34.106 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2023-03-03T07:54:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2023-03-03T07:54:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2023-03-03T07:54:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2023-03-03T07:54:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2023-03-03T07:54:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2023-03-03T07:54:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2023-03-03T07:54:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: debug client = 20 2023-03-03T07:54:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: debug finisher = 20 2023-03-03T07:54:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.108 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2023-03-03T07:54:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2023-03-03T07:54:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2023-03-03T07:54:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2023-03-03T07:54:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2023-03-03T07:54:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2023-03-03T07:54:34.109 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2023-03-03T07:54:34.109 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2023-03-03T07:54:34.109 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2023-03-03T07:54:34.109 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2023-03-03T07:54:34.109 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2023-03-03T07:54:34.109 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.109 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-03-03T07:54:34.109 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2023-03-03T07:54:34.110 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2023-03-03T07:54:34.110 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.110 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2023-03-03T07:54:34.110 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-03-03T07:54:34.110 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-03-03T07:54:34.110 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.110 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2023-03-03T07:54:34.110 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2023-03-03T07:54:34.111 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.111 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2023-03-03T07:54:34.111 INFO:teuthology.orchestra.run.smithi098.stdout: mon op complaint time = 120 2023-03-03T07:54:34.111 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:54:34.111 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2023-03-03T07:54:34.111 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2023-03-03T07:54:34.111 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2023-03-03T07:54:34.111 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2023-03-03T07:54:34.112 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-03-03T07:54:34.112 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-03-03T07:54:34.112 INFO:teuthology.orchestra.run.smithi098.stdout: client mount timeout = 600 2023-03-03T07:54:34.112 INFO:teuthology.orchestra.run.smithi098.stdout: debug client = 20 2023-03-03T07:54:34.112 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2023-03-03T07:54:34.112 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = "" 2023-03-03T07:54:34.112 INFO:teuthology.orchestra.run.smithi098.stdout: rados mon op timeout = 900 2023-03-03T07:54:34.112 INFO:teuthology.orchestra.run.smithi098.stdout: rados osd op timeout = 900 2023-03-03T07:54:34.113 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2023-03-03T07:54:34.113 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.b] 2023-03-03T07:54:34.113 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.c] 2023-03-03T07:54:34.113 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a] 2023-03-03T07:54:34.113 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c] 2023-03-03T07:54:34.113 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b] 2023-03-03T07:54:34.113 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d] 2023-03-03T07:54:34.114 INFO:teuthology.orchestra.run.smithi098.stdout:[mds] 2023-03-03T07:54:34.114 INFO:teuthology.orchestra.run.smithi098.stdout: debug mds = 20 2023-03-03T07:54:34.114 INFO:teuthology.orchestra.run.smithi098.stdout: debug mds balancer = 20 2023-03-03T07:54:34.114 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2023-03-03T07:54:34.114 INFO:teuthology.orchestra.run.smithi098.stdout: mds debug frag = True 2023-03-03T07:54:34.114 INFO:teuthology.orchestra.run.smithi098.stdout: mds debug scatterstat = True 2023-03-03T07:54:34.114 INFO:teuthology.orchestra.run.smithi098.stdout: mds op complaint time = 180 2023-03-03T07:54:34.114 INFO:teuthology.orchestra.run.smithi098.stdout: mds verify scatter = True 2023-03-03T07:54:34.115 INFO:teuthology.orchestra.run.smithi098.stdout: osd op complaint time = 180 2023-03-03T07:54:34.115 INFO:teuthology.orchestra.run.smithi098.stdout: rados mon op timeout = 900 2023-03-03T07:54:34.115 INFO:teuthology.orchestra.run.smithi098.stdout: rados osd op timeout = 900 2023-03-03T07:54:34.115 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2023-03-03T07:54:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: chdir = "" 2023-03-03T07:54:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-03-03T07:54:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: auth supported = cephx 2023-03-03T07:54:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: filestore xattr use omap = true 2023-03-03T07:54:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: mon clock drift allowed = 1.000 2023-03-03T07:54:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: osd crush chooseleaf type = 0 2023-03-03T07:54:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: auth debug = true 2023-03-03T07:54:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on old message = true 2023-03-03T07:54:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on bug = true 2023-03-03T07:54:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: mon max pg per osd = 10000 # >= luminous 2023-03-03T07:54:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: mon pg warn max object skew = 0 2023-03-03T07:54:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: # disable pg_autoscaler by default for new pools 2023-03-03T07:54:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: osd_pool_default_pg_autoscale_mode = off 2023-03-03T07:54:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default size = 2 2023-03-03T07:54:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow primary affinity = true 2023-03-03T07:54:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow pg remap = true 2023-03-03T07:54:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on legacy crush tunables = false 2023-03-03T07:54:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on crush straw calc version zero = false 2023-03-03T07:54:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on no sortbitwise = false 2023-03-03T07:54:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on osd down out interval zero = false 2023-03-03T07:54:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on too few osds = false 2023-03-03T07:54:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-03-03T07:54:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_no_redundancy = false 2023-03-03T07:54:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: mon_allow_pool_size_one = true 2023-03-03T07:54:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-03-03T07:54:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: osd default data pool replay window = 5 2023-03-03T07:54:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: mon allow pool delete = true 2023-03-03T07:54:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: mon cluster log file level = debug 2023-03-03T07:54:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: debug asserts on shutdown = true 2023-03-03T07:54:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: mon health detail to clog = false 2023-03-03T07:54:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: mon host = "172.21.15.98,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2023-03-03T07:54:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = fbbd322f-236a-44f8-96c5-be6963846c9b 2023-03-03T07:54:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.122 INFO:teuthology.orchestra.run.smithi100.stdout:[osd] 2023-03-03T07:54:34.122 INFO:teuthology.orchestra.run.smithi100.stdout: osd journal size = 100 2023-03-03T07:54:34.122 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.122 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub load threshold = 5.0 2023-03-03T07:54:34.122 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub max interval = 600 2023-03-03T07:54:34.122 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.122 INFO:teuthology.orchestra.run.smithi100.stdout: osd recover clone overlap = true 2023-03-03T07:54:34.122 INFO:teuthology.orchestra.run.smithi100.stdout: osd recovery max chunk = 1048576 2023-03-03T07:54:34.123 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.123 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug shutdown = true 2023-03-03T07:54:34.123 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug op order = true 2023-03-03T07:54:34.123 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify stray on activate = true 2023-03-03T07:54:34.123 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.123 INFO:teuthology.orchestra.run.smithi100.stdout: osd open classes on start = true 2023-03-03T07:54:34.123 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug pg log writeout = true 2023-03-03T07:54:34.123 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.124 INFO:teuthology.orchestra.run.smithi100.stdout: osd deep scrub update digest min age = 30 2023-03-03T07:54:34.124 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.124 INFO:teuthology.orchestra.run.smithi100.stdout: osd map max advance = 10 2023-03-03T07:54:34.124 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.124 INFO:teuthology.orchestra.run.smithi100.stdout: journal zero on create = true 2023-03-03T07:54:34.124 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.124 INFO:teuthology.orchestra.run.smithi100.stdout: filestore ondisk finisher threads = 3 2023-03-03T07:54:34.124 INFO:teuthology.orchestra.run.smithi100.stdout: filestore apply finisher threads = 3 2023-03-03T07:54:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: bdev debug aio = true 2023-03-03T07:54:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug misdirected ops = true 2023-03-03T07:54:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: bdev async discard = True 2023-03-03T07:54:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: bdev enable discard = True 2023-03-03T07:54:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore allocator = bitmap 2023-03-03T07:54:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore block size = 96636764160 2023-03-03T07:54:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore fsck on mount = True 2023-03-03T07:54:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluefs = 1/20 2023-03-03T07:54:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluestore = 1/20 2023-03-03T07:54:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2023-03-03T07:54:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: debug osd = 20 2023-03-03T07:54:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: debug rocksdb = 4/10 2023-03-03T07:54:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd backfillfull_ratio = 0.85 2023-03-03T07:54:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd full ratio = 0.9 2023-03-03T07:54:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd nearfull ratio = 0.8 2023-03-03T07:54:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: osd failsafe full ratio = 0.95 2023-03-03T07:54:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: osd objectstore = bluestore 2023-03-03T07:54:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2023-03-03T07:54:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.127 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr] 2023-03-03T07:54:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2023-03-03T07:54:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: debug mgr = 20 2023-03-03T07:54:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2023-03-03T07:54:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2023-03-03T07:54:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2023-03-03T07:54:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2023-03-03T07:54:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: mgr/telemetry/nag = false 2023-03-03T07:54:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: debug client = 20 2023-03-03T07:54:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: debug finisher = 20 2023-03-03T07:54:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.129 INFO:teuthology.orchestra.run.smithi100.stdout:[mon] 2023-03-03T07:54:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2023-03-03T07:54:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2023-03-03T07:54:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: debug paxos = 20 2023-03-03T07:54:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2023-03-03T07:54:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: mon data avail warn = 5 2023-03-03T07:54:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: mon mgr mkfs grace = 240 2023-03-03T07:54:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2023-03-03T07:54:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd reporter subtree level = osd 2023-03-03T07:54:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd prime pg temp = true 2023-03-03T07:54:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2023-03-03T07:54:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-03-03T07:54:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: auth mon ticket ttl = 660 # 11m 2023-03-03T07:54:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: auth service ticket ttl = 240 # 4m 2023-03-03T07:54:34.131 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.131 INFO:teuthology.orchestra.run.smithi100.stdout: # don't complain about insecure global_id in the test suite 2023-03-03T07:54:34.131 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-03-03T07:54:34.131 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-03-03T07:54:34.131 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.131 INFO:teuthology.orchestra.run.smithi100.stdout: # 1m isn't quite enough 2023-03-03T07:54:34.131 INFO:teuthology.orchestra.run.smithi100.stdout: mon_down_mkfs_grace = 2m 2023-03-03T07:54:34.131 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.132 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_filestore_osds = false 2023-03-03T07:54:34.132 INFO:teuthology.orchestra.run.smithi100.stdout: mon op complaint time = 120 2023-03-03T07:54:34.132 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-03-03T07:54:34.132 INFO:teuthology.orchestra.run.smithi100.stdout:[client] 2023-03-03T07:54:34.132 INFO:teuthology.orchestra.run.smithi100.stdout: rgw cache enabled = true 2023-03-03T07:54:34.132 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable ops log = true 2023-03-03T07:54:34.132 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable usage log = true 2023-03-03T07:54:34.132 INFO:teuthology.orchestra.run.smithi100.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-03-03T07:54:34.133 INFO:teuthology.orchestra.run.smithi100.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-03-03T07:54:34.133 INFO:teuthology.orchestra.run.smithi100.stdout: client mount timeout = 600 2023-03-03T07:54:34.133 INFO:teuthology.orchestra.run.smithi100.stdout: debug client = 20 2023-03-03T07:54:34.133 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2023-03-03T07:54:34.133 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = "" 2023-03-03T07:54:34.133 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2023-03-03T07:54:34.133 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2023-03-03T07:54:34.134 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.a] 2023-03-03T07:54:34.134 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.b] 2023-03-03T07:54:34.134 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.c] 2023-03-03T07:54:34.134 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.a] 2023-03-03T07:54:34.134 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.c] 2023-03-03T07:54:34.134 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.b] 2023-03-03T07:54:34.134 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.d] 2023-03-03T07:54:34.134 INFO:teuthology.orchestra.run.smithi100.stdout:[mds] 2023-03-03T07:54:34.135 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds = 20 2023-03-03T07:54:34.135 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds balancer = 20 2023-03-03T07:54:34.135 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2023-03-03T07:54:34.135 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug frag = True 2023-03-03T07:54:34.135 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug scatterstat = True 2023-03-03T07:54:34.135 INFO:teuthology.orchestra.run.smithi100.stdout: mds op complaint time = 180 2023-03-03T07:54:34.135 INFO:teuthology.orchestra.run.smithi100.stdout: mds verify scatter = True 2023-03-03T07:54:34.135 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2023-03-03T07:54:34.136 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2023-03-03T07:54:34.136 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2023-03-03T07:54:34.137 INFO:tasks.ceph:Creating admin key on mon.a... 2023-03-03T07:54:34.137 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-03-03T07:54:34.288 INFO:tasks.ceph:Copying monmap to all nodes... 2023-03-03T07:54:34.289 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:54:34.289 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-03-03T07:54:34.355 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:54:34.356 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-03-03T07:54:34.422 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T07:54:34.422 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:54:34.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.keyring 2023-03-03T07:54:34.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-03T07:54:34.599 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:54:34.599 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-03-03T07:54:34.706 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T07:54:34.706 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:54:34.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.keyring 2023-03-03T07:54:34.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-03T07:54:34.848 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:54:34.849 DEBUG:teuthology.orchestra.run.smithi100:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-03-03T07:54:34.955 INFO:tasks.ceph:Setting up mon nodes... 2023-03-03T07:54:34.955 INFO:tasks.ceph:Setting up mgr nodes... 2023-03-03T07:54:34.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-03-03T07:54:35.129 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-03-03T07:54:35.141 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-03-03T07:54:35.336 INFO:teuthology.orchestra.run.smithi100.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-03-03T07:54:35.348 INFO:tasks.ceph:Setting up mds nodes... 2023-03-03T07:54:35.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-03-03T07:54:35.526 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-03-03T07:54:35.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-03-03T07:54:35.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-03-03T07:54:35.813 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-03-03T07:54:35.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-03-03T07:54:35.927 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-03-03T07:54:36.103 INFO:teuthology.orchestra.run.smithi100.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-03-03T07:54:36.113 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-03-03T07:54:36.224 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-03-03T07:54:36.395 INFO:teuthology.orchestra.run.smithi100.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-03-03T07:54:36.404 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-03-03T07:54:36.507 INFO:tasks.ceph_client:Setting up client nodes... 2023-03-03T07:54:36.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-03-03T07:54:36.645 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-03-03T07:54:36.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-03-03T07:54:36.830 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-03-03T07:54:36.874 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-03-03T07:54:37.015 INFO:teuthology.orchestra.run.smithi100.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-03-03T07:54:37.059 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-03-03T07:54:37.198 INFO:teuthology.orchestra.run.smithi100.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-03-03T07:54:37.243 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-03-03T07:54:37.243 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi100.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-03-03T07:54:37.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-03-03T07:54:37.353 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T07:54:37.354 INFO:tasks.ceph:role: osd.0 2023-03-03T07:54:37.354 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T07:54:37.354 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-03-03T07:54:37.463 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T07:54:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T07:54:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T07:54:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T07:54:37.463 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T07:54:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2023-03-03T07:54:37.464 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T07:54:37.464 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T07:54:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T07:54:37.464 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T07:54:37.885 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2023-03-03T07:54:37.894 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2023-03-03T07:54:37.894 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-03-03T07:54:38.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-03-03T07:54:38.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-03-03T07:54:38.355 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T07:54:38.355 INFO:tasks.ceph:role: osd.1 2023-03-03T07:54:38.355 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T07:54:38.356 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-03-03T07:54:38.464 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T07:54:38.464 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T07:54:38.464 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T07:54:38.465 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T07:54:38.465 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T07:54:38.465 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2023-03-03T07:54:38.465 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T07:54:38.465 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T07:54:38.465 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T07:54:38.465 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T07:54:38.899 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2023-03-03T07:54:38.908 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2023-03-03T07:54:38.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-03-03T07:54:39.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-03-03T07:54:39.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-03-03T07:54:39.313 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T07:54:39.314 INFO:tasks.ceph:role: osd.2 2023-03-03T07:54:39.314 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T07:54:39.314 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-03-03T07:54:39.426 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T07:54:39.426 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T07:54:39.426 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T07:54:39.427 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T07:54:39.427 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T07:54:39.427 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2023-03-03T07:54:39.427 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T07:54:39.427 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T07:54:39.427 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T07:54:39.427 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T07:54:39.853 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2023-03-03T07:54:39.859 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2023-03-03T07:54:39.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-03-03T07:54:40.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-03-03T07:54:40.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-03-03T07:54:40.309 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T07:54:40.310 INFO:tasks.ceph:role: osd.3 2023-03-03T07:54:40.310 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T07:54:40.310 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-03-03T07:54:40.416 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T07:54:40.416 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T07:54:40.416 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T07:54:40.417 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T07:54:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T07:54:40.417 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2023-03-03T07:54:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T07:54:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T07:54:40.417 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T07:54:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T07:54:40.822 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2023-03-03T07:54:40.830 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2023-03-03T07:54:40.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-03-03T07:54:41.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-03-03T07:54:41.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T07:54:41.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:41.298+0000 7fe00fc9d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-03-03T07:54:41.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:41.300+0000 7fe00fc9d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-03-03T07:54:41.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:41.300+0000 7fe00fc9d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-03T07:54:41.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:41.300+0000 7fe00fc9d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-03T07:54:41.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:41.300+0000 7fe00fc9d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-03T07:54:41.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:41.307+0000 7fe00fc9d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-03-03T07:54:44.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-03-03T07:54:44.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T07:54:44.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:44.747+0000 7f581fa3f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-03-03T07:54:44.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:44.747+0000 7f581fa3f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-03-03T07:54:44.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:44.748+0000 7f581fa3f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-03T07:54:44.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:44.748+0000 7f581fa3f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-03T07:54:44.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:44.748+0000 7f581fa3f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-03T07:54:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:44.750+0000 7f581fa3f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-03-03T07:54:47.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-03-03T07:54:48.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T07:54:48.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:48.173+0000 7f5525f6b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-03-03T07:54:48.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:48.173+0000 7f5525f6b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-03-03T07:54:48.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:48.173+0000 7f5525f6b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-03T07:54:48.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:48.173+0000 7f5525f6b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-03T07:54:48.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:48.173+0000 7f5525f6b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-03T07:54:48.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:48.175+0000 7f5525f6b640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-03-03T07:54:51.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-03-03T07:54:51.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T07:54:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:51.602+0000 7f0af4d90640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-03-03T07:54:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:51.603+0000 7f0af4d90640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-03-03T07:54:51.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:51.603+0000 7f0af4d90640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-03T07:54:51.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:51.603+0000 7f0af4d90640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-03T07:54:51.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:51.603+0000 7f0af4d90640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-03T07:54:51.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:54:51.606+0000 7f0af4d90640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-03-03T07:54:54.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-03-03T07:54:54.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-03-03T07:54:54.744 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T07:54:54.745 INFO:tasks.ceph:role: osd.4 2023-03-03T07:54:54.746 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T07:54:54.746 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-03-03T07:54:54.855 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T07:54:54.855 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T07:54:54.855 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T07:54:54.855 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T07:54:54.855 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T07:54:54.856 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2023-03-03T07:54:54.856 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T07:54:54.856 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T07:54:54.856 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T07:54:54.856 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T07:54:55.295 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2023-03-03T07:54:55.301 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2023-03-03T07:54:55.301 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-03-03T07:54:55.528 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-03-03T07:54:55.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-03-03T07:54:55.738 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T07:54:55.738 INFO:tasks.ceph:role: osd.5 2023-03-03T07:54:55.738 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T07:54:55.739 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-03-03T07:54:55.844 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T07:54:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T07:54:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T07:54:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T07:54:55.844 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T07:54:55.845 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2023-03-03T07:54:55.845 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T07:54:55.845 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T07:54:55.846 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T07:54:55.846 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T07:54:56.280 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2023-03-03T07:54:56.287 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2023-03-03T07:54:56.287 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-03-03T07:54:56.451 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-03-03T07:54:56.559 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-03-03T07:54:56.659 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T07:54:56.660 INFO:tasks.ceph:role: osd.6 2023-03-03T07:54:56.660 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T07:54:56.660 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-03-03T07:54:56.765 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T07:54:56.766 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T07:54:56.766 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T07:54:56.766 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T07:54:56.766 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T07:54:56.767 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2023-03-03T07:54:56.767 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T07:54:56.767 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T07:54:56.767 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T07:54:56.767 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T07:54:57.204 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2023-03-03T07:54:57.211 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2023-03-03T07:54:57.212 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-03-03T07:54:57.431 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-03-03T07:54:57.541 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-03-03T07:54:57.640 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T07:54:57.640 INFO:tasks.ceph:role: osd.7 2023-03-03T07:54:57.641 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T07:54:57.641 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-03-03T07:54:57.746 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T07:54:57.747 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T07:54:57.747 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T07:54:57.747 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T07:54:57.747 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T07:54:57.747 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2023-03-03T07:54:57.747 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T07:54:57.748 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T07:54:57.748 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T07:54:57.748 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T07:54:58.153 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2023-03-03T07:54:58.160 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2023-03-03T07:54:58.161 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-03-03T07:54:58.334 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-03-03T07:54:58.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T07:54:58.596 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:54:58.595+0000 7f9f3d04f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-03-03T07:54:58.598 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:54:58.597+0000 7f9f3d04f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-03-03T07:54:58.599 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:54:58.597+0000 7f9f3d04f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-03T07:54:58.599 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:54:58.597+0000 7f9f3d04f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-03T07:54:58.599 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:54:58.597+0000 7f9f3d04f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-03T07:54:58.603 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:54:58.602+0000 7f9f3d04f640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-03-03T07:55:00.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:55:00.896 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:55:01.797 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-03-03T07:55:01.897 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T07:55:02.042 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:02.041+0000 7fdac2c11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-03-03T07:55:02.043 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:02.042+0000 7fdac2c11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-03-03T07:55:02.043 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:02.042+0000 7fdac2c11640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-03T07:55:02.043 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:02.042+0000 7fdac2c11640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-03T07:55:02.043 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:02.042+0000 7fdac2c11640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-03T07:55:02.044 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:02.043+0000 7fdac2c11640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-03-03T07:55:04.978 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-03-03T07:55:05.078 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T07:55:05.224 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:05.222+0000 7f99b900e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-03-03T07:55:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:05.222+0000 7f99b900e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-03-03T07:55:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:05.222+0000 7f99b900e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-03T07:55:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:05.222+0000 7f99b900e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-03T07:55:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:05.222+0000 7f99b900e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-03T07:55:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:05.224+0000 7f99b900e640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-03-03T07:55:08.149 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-03-03T07:55:08.249 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T07:55:08.395 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:08.393+0000 7f95016b3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-03-03T07:55:08.395 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:08.394+0000 7f95016b3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-03-03T07:55:08.396 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:08.394+0000 7f95016b3640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-03T07:55:08.396 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:08.394+0000 7f95016b3640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-03T07:55:08.396 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:08.394+0000 7f95016b3640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-03T07:55:08.397 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:08.396+0000 7f95016b3640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-03-03T07:55:11.318 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-03-03T07:55:11.420 INFO:tasks.ceph:Reading keys from all nodes... 2023-03-03T07:55:11.421 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:55:11.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-03-03T07:55:11.526 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:55:11.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-03-03T07:55:11.626 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:55:11.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-03-03T07:55:11.724 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:55:11.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-03-03T07:55:11.823 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:55:11.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-03-03T07:55:11.921 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:55:11.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-03-03T07:55:12.020 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:55:12.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-03-03T07:55:12.118 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:55:12.118 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-03-03T07:55:12.223 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:55:12.223 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-03-03T07:55:12.322 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:55:12.322 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-03-03T07:55:12.421 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:55:12.421 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-03-03T07:55:12.520 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:55:12.520 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-03-03T07:55:12.620 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:55:12.621 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-03-03T07:55:12.718 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:55:12.719 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-03-03T07:55:12.818 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:55:12.818 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-03-03T07:55:12.889 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:55:12.889 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-03-03T07:55:12.957 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:55:12.958 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-03-03T07:55:13.026 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:55:13.026 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-03-03T07:55:13.093 INFO:tasks.ceph:Adding keys to all mons... 2023-03-03T07:55:13.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo tee -a /etc/ceph/ceph.keyring 2023-03-03T07:55:13.099 DEBUG:teuthology.orchestra.run.smithi100:> sudo tee -a /etc/ceph/ceph.keyring 2023-03-03T07:55:13.188 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.y] 2023-03-03T07:55:13.189 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC7pwFkfYO5BxAAFW/elbdoqVXBkUIKrAvYsg== 2023-03-03T07:55:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a] 2023-03-03T07:55:13.189 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC7pwFkqFdgHxAAuwiGkaNnr5NplmhniDYAEQ== 2023-03-03T07:55:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c] 2023-03-03T07:55:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC7pwFkKoRwMBAAfkWGDUsqNgqfkscl0aBX1A== 2023-03-03T07:55:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2023-03-03T07:55:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDBpwFkTFzTERAAGHrFKuBlXIEfaYD7vs3ZHA== 2023-03-03T07:55:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2023-03-03T07:55:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDEpwFkVWObLBAAW4acTvNbmscIe18NXVgRYg== 2023-03-03T07:55:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2023-03-03T07:55:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDIpwFkK8lgChAAHVD5btBivUpcapSgDp2E0w== 2023-03-03T07:55:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.3] 2023-03-03T07:55:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDLpwFksrb8IxAAN+CP1O3DJ1zDy+qyGNor2Q== 2023-03-03T07:55:13.191 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.x] 2023-03-03T07:55:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC7pwFkvdMQFBAAMi+x7psn5CDpd//d6jN2VA== 2023-03-03T07:55:13.192 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b] 2023-03-03T07:55:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC8pwFk0x8kBhAA5JDo/wmI4650J9tUnvXVzw== 2023-03-03T07:55:13.192 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d] 2023-03-03T07:55:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC8pwFkK9SKFxAAFSCozGfyRwrv46QPitRxTQ== 2023-03-03T07:55:13.192 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.4] 2023-03-03T07:55:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDSpwFkkWGHIxAA4wejA0w05JPIUghEAVG/Rg== 2023-03-03T07:55:13.193 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.5] 2023-03-03T07:55:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDWpwFknHmGAhAA6VM16luaykZfJEzr+yDZZA== 2023-03-03T07:55:13.193 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.6] 2023-03-03T07:55:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDZpwFkcitNDRAAOz04YGTzx3Phenp0n8JeEA== 2023-03-03T07:55:13.193 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.7] 2023-03-03T07:55:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDcpwFkrjmIFxAAbWeB2+8VQgyTplW+fk47OA== 2023-03-03T07:55:13.194 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2023-03-03T07:55:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC8pwFk02lsJhAA0cH/evMnWdZN6DdBGCLjjg== 2023-03-03T07:55:13.194 INFO:teuthology.orchestra.run.smithi098.stdout:[client.2] 2023-03-03T07:55:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC8pwFkyKp3MRAA8Eh41TZNsO6/8VhAWCpNMQ== 2023-03-03T07:55:13.195 INFO:teuthology.orchestra.run.smithi098.stdout:[client.1] 2023-03-03T07:55:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC9pwFkRqXrABAArm/01O5MtSTQHWQWKqNQEg== 2023-03-03T07:55:13.195 INFO:teuthology.orchestra.run.smithi098.stdout:[client.3] 2023-03-03T07:55:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC9pwFk14bSCxAAy2nXc9G50W4lH5GtH84hpQ== 2023-03-03T07:55:13.196 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr.y] 2023-03-03T07:55:13.196 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC7pwFkfYO5BxAAFW/elbdoqVXBkUIKrAvYsg== 2023-03-03T07:55:13.196 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.a] 2023-03-03T07:55:13.196 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC7pwFkqFdgHxAAuwiGkaNnr5NplmhniDYAEQ== 2023-03-03T07:55:13.197 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.c] 2023-03-03T07:55:13.197 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC7pwFkKoRwMBAAfkWGDUsqNgqfkscl0aBX1A== 2023-03-03T07:55:13.197 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.0] 2023-03-03T07:55:13.197 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDBpwFkTFzTERAAGHrFKuBlXIEfaYD7vs3ZHA== 2023-03-03T07:55:13.197 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.1] 2023-03-03T07:55:13.197 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDEpwFkVWObLBAAW4acTvNbmscIe18NXVgRYg== 2023-03-03T07:55:13.197 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.2] 2023-03-03T07:55:13.198 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDIpwFkK8lgChAAHVD5btBivUpcapSgDp2E0w== 2023-03-03T07:55:13.198 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.3] 2023-03-03T07:55:13.198 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDLpwFksrb8IxAAN+CP1O3DJ1zDy+qyGNor2Q== 2023-03-03T07:55:13.198 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr.x] 2023-03-03T07:55:13.198 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC7pwFkvdMQFBAAMi+x7psn5CDpd//d6jN2VA== 2023-03-03T07:55:13.198 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.b] 2023-03-03T07:55:13.198 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC8pwFk0x8kBhAA5JDo/wmI4650J9tUnvXVzw== 2023-03-03T07:55:13.198 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.d] 2023-03-03T07:55:13.199 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC8pwFkK9SKFxAAFSCozGfyRwrv46QPitRxTQ== 2023-03-03T07:55:13.199 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.4] 2023-03-03T07:55:13.199 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDSpwFkkWGHIxAA4wejA0w05JPIUghEAVG/Rg== 2023-03-03T07:55:13.199 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.5] 2023-03-03T07:55:13.199 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDWpwFknHmGAhAA6VM16luaykZfJEzr+yDZZA== 2023-03-03T07:55:13.199 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.6] 2023-03-03T07:55:13.199 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDZpwFkcitNDRAAOz04YGTzx3Phenp0n8JeEA== 2023-03-03T07:55:13.199 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.7] 2023-03-03T07:55:13.200 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDcpwFkrjmIFxAAbWeB2+8VQgyTplW+fk47OA== 2023-03-03T07:55:13.200 INFO:teuthology.orchestra.run.smithi100.stdout:[client.0] 2023-03-03T07:55:13.200 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC8pwFk02lsJhAA0cH/evMnWdZN6DdBGCLjjg== 2023-03-03T07:55:13.200 INFO:teuthology.orchestra.run.smithi100.stdout:[client.2] 2023-03-03T07:55:13.200 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC8pwFkyKp3MRAA8Eh41TZNsO6/8VhAWCpNMQ== 2023-03-03T07:55:13.200 INFO:teuthology.orchestra.run.smithi100.stdout:[client.1] 2023-03-03T07:55:13.200 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC9pwFkRqXrABAArm/01O5MtSTQHWQWKqNQEg== 2023-03-03T07:55:13.201 INFO:teuthology.orchestra.run.smithi100.stdout:[client.3] 2023-03-03T07:55:13.201 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC9pwFk14bSCxAAy2nXc9G50W4lH5GtH84hpQ== 2023-03-03T07:55:13.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T07:55:13.207 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T07:55:13.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T07:55:13.375 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T07:55:13.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T07:55:13.532 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T07:55:13.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:13.690 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:13.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:13.845 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:13.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:13.999 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:14.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:14.155 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:14.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T07:55:14.308 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T07:55:14.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T07:55:14.465 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T07:55:14.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T07:55:14.621 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T07:55:14.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:14.779 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:14.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:14.933 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:15.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:15.088 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:15.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:15.242 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T07:55:15.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T07:55:15.395 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T07:55:15.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T07:55:15.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T07:55:15.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T07:55:15.709 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T07:55:15.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T07:55:15.868 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T07:55:16.019 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-03-03T07:55:16.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-03-03T07:55:16.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-03T07:55:16.287 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-03-03T07:55:16.387 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-03-03T07:55:16.486 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-03T07:55:16.657 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-03-03T07:55:16.757 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-03-03T07:55:16.857 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-03T07:55:17.030 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-03-03T07:55:17.130 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-03-03T07:55:17.135 DEBUG:teuthology.orchestra.run.smithi100:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-03-03T07:55:17.203 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-03-03T07:55:17.203 INFO:tasks.ceph.mon.a:Restarting daemon 2023-03-03T07:55:17.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-03-03T07:55:17.209 INFO:tasks.ceph.mon.a:Started 2023-03-03T07:55:17.209 INFO:tasks.ceph.mon.b:Restarting daemon 2023-03-03T07:55:17.210 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-03-03T07:55:17.215 INFO:tasks.ceph.mon.b:Started 2023-03-03T07:55:17.215 INFO:tasks.ceph.mon.c:Restarting daemon 2023-03-03T07:55:17.215 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-03-03T07:55:17.222 INFO:tasks.ceph.mon.c:Started 2023-03-03T07:55:17.222 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-03-03T07:55:17.222 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-03-03T07:55:17.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-03-03T07:55:17.228 INFO:tasks.ceph.mgr.y:Started 2023-03-03T07:55:17.229 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-03-03T07:55:17.229 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-03-03T07:55:17.234 INFO:tasks.ceph.mgr.x:Started 2023-03-03T07:55:17.235 INFO:tasks.ceph:Setting crush tunables to default 2023-03-03T07:55:17.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd crush tunables default 2023-03-03T07:55:17.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.478+0000 7f1d5e7ff700 1 Processor -- start 2023-03-03T07:55:17.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.478+0000 7f1d5e7ff700 1 -- start start 2023-03-03T07:55:17.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.479+0000 7f1d5e7ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1d580b26d0 0x7f1d580b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:17.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.479+0000 7f1d5e7ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1d580b5140 0x7f1d580d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:17.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.479+0000 7f1d5e7ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d580cd7d0 con 0x7f1d580b8d10 2023-03-03T07:55:17.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.480+0000 7f1d5e7ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d580b55c0 con 0x7f1d580b26d0 2023-03-03T07:55:17.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.480+0000 7f1d5e7ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d580b90e0 con 0x7f1d580b5140 2023-03-03T07:55:17.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.480+0000 7f1d577fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1d580b26d0 0x7f1d580b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:17.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.480+0000 7f1d57fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1d580b8d10 0x7f1d580cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55940/0 (socket says 172.21.15.98:55940) 2023-03-03T07:55:17.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.480+0000 7f1d57fff700 1 -- 172.21.15.98:0/1151078838 learned_addr learned my addr 172.21.15.98:0/1151078838 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:17.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.480+0000 7f1d577fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1d580b26d0 0x7f1d580b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53776/0 (socket says 172.21.15.98:53776) 2023-03-03T07:55:17.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.480+0000 7f1d577fe700 1 -- 172.21.15.98:0/1151078838 >> v2:172.21.15.98:3300/0 conn(0x7f1d580b26d0 msgr2=0x7f1d580b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-03-03T07:55:17.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.480+0000 7f1d577fe700 1 -- 172.21.15.98:0/1151078838 >> v2:172.21.15.98:3300/0 conn(0x7f1d580b26d0 msgr2=0x7f1d580b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-03-03T07:55:17.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.480+0000 7f1d577fe700 1 --2- 172.21.15.98:0/1151078838 >> v2:172.21.15.98:3300/0 conn(0x7f1d580b26d0 0x7f1d580b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-03T07:55:17.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.480+0000 7f1d577fe700 1 --2- 172.21.15.98:0/1151078838 >> v2:172.21.15.98:3300/0 conn(0x7f1d580b26d0 0x7f1d580b4c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-03T07:55:17.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.480+0000 7f1d5cd9c700 1 --2- 172.21.15.98:0/1151078838 >> v2:172.21.15.100:3300/0 conn(0x7f1d580b5140 0x7f1d580d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:17.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.481+0000 7f1d5cd9c700 1 -- 172.21.15.98:0/1151078838 >> v2:172.21.15.100:3300/0 conn(0x7f1d580b5140 msgr2=0x7f1d580d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-03-03T07:55:17.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.481+0000 7f1d5cd9c700 1 -- 172.21.15.98:0/1151078838 >> v2:172.21.15.100:3300/0 conn(0x7f1d580b5140 msgr2=0x7f1d580d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-03-03T07:55:17.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.481+0000 7f1d5cd9c700 1 --2- 172.21.15.98:0/1151078838 >> v2:172.21.15.100:3300/0 conn(0x7f1d580b5140 0x7f1d580d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-03T07:55:17.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.481+0000 7f1d5cd9c700 1 --2- 172.21.15.98:0/1151078838 >> v2:172.21.15.100:3300/0 conn(0x7f1d580b5140 0x7f1d580d13a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-03T07:55:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.675+0000 7f1d56ffd700 1 -- 172.21.15.98:0/1151078838 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368971157 0 0) 0x7f1d580cd7d0 con 0x7f1d580b8d10 2023-03-03T07:55:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.675+0000 7f1d56ffd700 1 -- 172.21.15.98:0/1151078838 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d400035b0 con 0x7f1d580b8d10 2023-03-03T07:55:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.675+0000 7f1d56ffd700 1 -- 172.21.15.98:0/1151078838 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1042268881 0 0) 0x7f1d400035b0 con 0x7f1d580b8d10 2023-03-03T07:55:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.675+0000 7f1d56ffd700 1 -- 172.21.15.98:0/1151078838 >> v2:172.21.15.100:3300/0 conn(0x7f1d580b5140 msgr2=0x7f1d580d13a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T07:55:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.675+0000 7f1d56ffd700 1 --2- 172.21.15.98:0/1151078838 >> v2:172.21.15.100:3300/0 conn(0x7f1d580b5140 0x7f1d580d13a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:17.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.675+0000 7f1d56ffd700 1 -- 172.21.15.98:0/1151078838 >> v2:172.21.15.98:3300/0 conn(0x7f1d580b26d0 msgr2=0x7f1d580b4c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T07:55:17.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.675+0000 7f1d56ffd700 1 --2- 172.21.15.98:0/1151078838 >> v2:172.21.15.98:3300/0 conn(0x7f1d580b26d0 0x7f1d580b4c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:17.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.676+0000 7f1d56ffd700 1 -- 172.21.15.98:0/1151078838 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d580cda20 con 0x7f1d580b8d10 2023-03-03T07:55:17.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.676+0000 7f1d56ffd700 1 -- 172.21.15.98:0/1151078838 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f1d48001670 con 0x7f1d580b8d10 2023-03-03T07:55:17.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.676+0000 7f1d56ffd700 1 -- 172.21.15.98:0/1151078838 >> v1:172.21.15.98:6789/0 conn(0x7f1d580b8d10 legacy=0x7f1d580cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:17.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.676+0000 7f1d56ffd700 1 --2- 172.21.15.98:0/1151078838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d40003a50 0x7f1d40003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:17.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.676+0000 7f1d56ffd700 1 --2- 172.21.15.98:0/1151078838 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d40004850 0x7f1d40004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:17.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.676+0000 7f1d56ffd700 1 --2- 172.21.15.98:0/1151078838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d40005200 0x7f1d4000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:17.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.676+0000 7f1d56ffd700 1 -- 172.21.15.98:0/1151078838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d580b55c0 con 0x7f1d40005200 2023-03-03T07:55:17.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.676+0000 7f1d56ffd700 1 -- 172.21.15.98:0/1151078838 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d580b90e0 con 0x7f1d40004850 2023-03-03T07:55:17.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.676+0000 7f1d56ffd700 1 -- 172.21.15.98:0/1151078838 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4000c1c0 con 0x7f1d40003a50 2023-03-03T07:55:17.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.677+0000 7f1d57fff700 1 --2- 172.21.15.98:0/1151078838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d40003a50 0x7f1d40003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:17.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.677+0000 7f1d5cd9c700 1 --2- 172.21.15.98:0/1151078838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d40005200 0x7f1d4000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:17.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.677+0000 7f1d577fe700 1 --2- 172.21.15.98:0/1151078838 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d40004850 0x7f1d40004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:17.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.677+0000 7f1d5cd9c700 1 -- 172.21.15.98:0/1151078838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d40003a50 msgr2=0x7f1d40003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:17.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.677+0000 7f1d5cd9c700 1 --2- 172.21.15.98:0/1151078838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d40003a50 0x7f1d40003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:17.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.677+0000 7f1d5cd9c700 1 -- 172.21.15.98:0/1151078838 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d40004850 msgr2=0x7f1d40004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:17.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.677+0000 7f1d5cd9c700 1 --2- 172.21.15.98:0/1151078838 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d40004850 0x7f1d40004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:17.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.677+0000 7f1d5cd9c700 1 -- 172.21.15.98:0/1151078838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d400035b0 con 0x7f1d40005200 2023-03-03T07:55:17.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.678+0000 7f1d5cd9c700 1 -- 172.21.15.98:0/1151078838 reap_dead start 2023-03-03T07:55:17.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.678+0000 7f1d5cd9c700 1 --2- 172.21.15.98:0/1151078838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d40005200 0x7f1d4000ba10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1d50009610 tx=0x7f1d40005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f58c147fa9225ce server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:17.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.678+0000 7f1d56ffd700 1 -- 172.21.15.98:0/1151078838 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d50007df0 con 0x7f1d40005200 2023-03-03T07:55:17.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.678+0000 7f1d56ffd700 1 -- 172.21.15.98:0/1151078838 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d50003a20 con 0x7f1d40005200 2023-03-03T07:55:17.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.678+0000 7f1d56ffd700 1 -- 172.21.15.98:0/1151078838 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d50003dd0 con 0x7f1d40005200 2023-03-03T07:55:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.679+0000 7f1d5e7ff700 1 -- 172.21.15.98:0/1151078838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d40005200 msgr2=0x7f1d4000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.679+0000 7f1d5e7ff700 1 --2- 172.21.15.98:0/1151078838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d40005200 0x7f1d4000ba10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1d50009610 tx=0x7f1d40005880 comp rx=0 tx=0).stop 2023-03-03T07:55:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.679+0000 7f1d5e7ff700 1 -- 172.21.15.98:0/1151078838 shutdown_connections 2023-03-03T07:55:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.679+0000 7f1d5e7ff700 1 --2- 172.21.15.98:0/1151078838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d40005200 0x7f1d4000ba10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.679+0000 7f1d5e7ff700 1 -- 172.21.15.98:0/1151078838 >> 172.21.15.98:0/1151078838 conn(0x7f1d58073e70 msgr2=0x7f1d5806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.679+0000 7f1d5e7ff700 1 -- 172.21.15.98:0/1151078838 shutdown_connections 2023-03-03T07:55:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.679+0000 7f1d5e7ff700 1 -- 172.21.15.98:0/1151078838 wait complete. 2023-03-03T07:55:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.681+0000 7f1d5e7ff700 1 Processor -- start 2023-03-03T07:55:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.681+0000 7f1d5e7ff700 1 -- start start 2023-03-03T07:55:17.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.683+0000 7f1d5e7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d40005200 0x7f1d581f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:17.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.683+0000 7f1d5e7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d581f5940 0x7f1d581f1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:17.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.683+0000 7f1d5e7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d581f3510 0x7f1d580b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:17.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.683+0000 7f1d5e7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d58150ea0 con 0x7f1d581f3510 2023-03-03T07:55:17.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.683+0000 7f1d5e7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5815e4f0 con 0x7f1d581f5940 2023-03-03T07:55:17.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.683+0000 7f1d5e7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d581f6170 con 0x7f1d40005200 2023-03-03T07:55:17.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.684+0000 7f1d57fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d40005200 0x7f1d581f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:17.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.684+0000 7f1d57fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d40005200 0x7f1d581f2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:48908/0 (socket says 172.21.15.98:48908) 2023-03-03T07:55:17.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.684+0000 7f1d57fff700 1 -- 172.21.15.98:0/4181614344 learned_addr learned my addr 172.21.15.98:0/4181614344 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.684+0000 7f1d577fe700 1 --2- 172.21.15.98:0/4181614344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d581f5940 0x7f1d581f1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.684+0000 7f1d577fe700 1 -- 172.21.15.98:0/4181614344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d40005200 msgr2=0x7f1d581f2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.684+0000 7f1d577fe700 1 --2- 172.21.15.98:0/4181614344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d40005200 0x7f1d581f2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.684+0000 7f1d577fe700 1 -- 172.21.15.98:0/4181614344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d581f3510 msgr2=0x7f1d580b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.684+0000 7f1d577fe700 1 --2- 172.21.15.98:0/4181614344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d581f3510 0x7f1d580b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.684+0000 7f1d577fe700 1 -- 172.21.15.98:0/4181614344 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d580d0e80 con 0x7f1d581f5940 2023-03-03T07:55:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.685+0000 7f1d577fe700 1 --2- 172.21.15.98:0/4181614344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d581f5940 0x7f1d581f1b80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c010890 tx=0x7f1d4c010ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:17.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.685+0000 7f1d54ff9700 1 -- 172.21.15.98:0/4181614344 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c006ea0 con 0x7f1d581f5940 2023-03-03T07:55:17.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.685+0000 7f1d54ff9700 1 -- 172.21.15.98:0/4181614344 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d4c015940 con 0x7f1d581f5940 2023-03-03T07:55:17.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.685+0000 7f1d54ff9700 1 -- 172.21.15.98:0/4181614344 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c015cf0 con 0x7f1d581f5940 2023-03-03T07:55:17.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.685+0000 7f1d5e7ff700 1 -- 172.21.15.98:0/4181614344 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d580cdd00 con 0x7f1d581f5940 2023-03-03T07:55:17.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.685+0000 7f1d5e7ff700 1 -- 172.21.15.98:0/4181614344 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d580cdfe0 con 0x7f1d581f5940 2023-03-03T07:55:17.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.685+0000 7f1d54ff9700 1 -- 172.21.15.98:0/4181614344 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f1d4c026070 con 0x7f1d581f5940 2023-03-03T07:55:17.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.686+0000 7f1d54ff9700 1 -- 172.21.15.98:0/4181614344 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1d4c00ecc0 con 0x7f1d581f5940 2023-03-03T07:55:17.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.686+0000 7f1d5e7ff700 1 -- 172.21.15.98:0/4181614344 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d5815e680 con 0x7f1d581f5940 2023-03-03T07:55:17.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.690+0000 7f1d54ff9700 1 -- 172.21.15.98:0/4181614344 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f1d4c01a020 con 0x7f1d581f5940 2023-03-03T07:55:17.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.768+0000 7f1d5e7ff700 1 -- 172.21.15.98:0/4181614344 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f1d5804e6e0 con 0x7f1d581f5940 2023-03-03T07:55:17.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.777+0000 7f1d54ff9700 1 -- 172.21.15.98:0/4181614344 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1d4c00f040 con 0x7f1d581f5940 2023-03-03T07:55:17.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.778+0000 7f1d5e7ff700 1 -- 172.21.15.98:0/4181614344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d581f5940 msgr2=0x7f1d581f1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:17.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.778+0000 7f1d5e7ff700 1 --2- 172.21.15.98:0/4181614344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d581f5940 0x7f1d581f1b80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c010890 tx=0x7f1d4c010ba0 comp rx=0 tx=0).stop 2023-03-03T07:55:17.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.778+0000 7f1d5e7ff700 1 -- 172.21.15.98:0/4181614344 shutdown_connections 2023-03-03T07:55:17.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.778+0000 7f1d5e7ff700 1 --2- 172.21.15.98:0/4181614344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d581f3510 0x7f1d580b6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:17.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.778+0000 7f1d5e7ff700 1 --2- 172.21.15.98:0/4181614344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d40005200 0x7f1d581f2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:17.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.778+0000 7f1d5e7ff700 1 --2- 172.21.15.98:0/4181614344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d581f5940 0x7f1d581f1b80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:17.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.778+0000 7f1d5e7ff700 1 -- 172.21.15.98:0/4181614344 >> 172.21.15.98:0/4181614344 conn(0x7f1d58073e70 msgr2=0x7f1d580b3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:17.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.778+0000 7f1d5e7ff700 1 -- 172.21.15.98:0/4181614344 shutdown_connections 2023-03-03T07:55:17.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:17.778+0000 7f1d5e7ff700 1 -- 172.21.15.98:0/4181614344 wait complete. 2023-03-03T07:55:17.782 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2023-03-03T07:55:17.808 INFO:tasks.ceph:check_enable_crimson: False 2023-03-03T07:55:17.809 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-03-03T07:55:17.809 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:55:17.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-03-03T07:55:17.915 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:55:17.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-03-03T07:55:18.017 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:55:18.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-03-03T07:55:18.124 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:55:18.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-03-03T07:55:18.233 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:55:18.233 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-03-03T07:55:18.333 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:55:18.334 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-03-03T07:55:18.434 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:55:18.434 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-03-03T07:55:18.534 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:55:18.534 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-03-03T07:55:18.634 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 18707639-bc5c-49a9-b8b6-d563deea2dea 0 2023-03-03T07:55:18.874 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.873+0000 7fb05b19d700 1 Processor -- start 2023-03-03T07:55:18.874 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.873+0000 7fb05b19d700 1 -- start start 2023-03-03T07:55:18.875 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.874+0000 7fb05b19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0540cf570 0x7fb05415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:18.876 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.874+0000 7fb05b19d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb05415db30 0x7fb0540b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:18.876 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.874+0000 7fb05b19d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0540cd780 con 0x7fb0541609b0 2023-03-03T07:55:18.876 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.874+0000 7fb05b19d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0540cebb0 con 0x7fb05415db30 2023-03-03T07:55:18.876 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.874+0000 7fb05b19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0540cda60 con 0x7fb0540cf570 2023-03-03T07:55:18.877 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.875+0000 7fb05a99c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb05415db30 0x7fb0540b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:18.877 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.875+0000 7fb05a99c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb05415db30 0x7fb0540b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53240/0 (socket says 172.21.15.100:53240) 2023-03-03T07:55:18.878 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.875+0000 7fb05999a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0540cf570 0x7fb05415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:18.878 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.875+0000 7fb05a99c700 1 -- 172.21.15.100:0/466227620 learned_addr learned my addr 172.21.15.100:0/466227620 (peer_addr_for_me v2:172.21.15.100:0/0) 2023-03-03T07:55:18.878 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.875+0000 7fb05a99c700 1 -- 172.21.15.100:0/466227620 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0540cf570 msgr2=0x7fb05415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:18.878 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.875+0000 7fb05a99c700 1 --2- 172.21.15.100:0/466227620 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0540cf570 0x7fb05415d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:18.878 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.875+0000 7fb05a99c700 1 -- 172.21.15.100:0/466227620 >> v1:172.21.15.100:6789/0 conn(0x7fb0541609b0 legacy=0x7fb054160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:18.878 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.875+0000 7fb05a99c700 1 -- 172.21.15.100:0/466227620 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0540cdd40 con 0x7fb05415db30 2023-03-03T07:55:18.878 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.876+0000 7fb05a99c700 1 --2- 172.21.15.100:0/466227620 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb05415db30 0x7fb0540b4aa0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb04c009e50 tx=0x7fb04c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2884327edc0e00ec server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:18.878 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.876+0000 7fb059199700 1 -- 172.21.15.100:0/466227620 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c00e070 con 0x7fb05415db30 2023-03-03T07:55:18.879 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.876+0000 7fb059199700 1 -- 172.21.15.100:0/466227620 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04c00b910 con 0x7fb05415db30 2023-03-03T07:55:18.879 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.876+0000 7fb059199700 1 -- 172.21.15.100:0/466227620 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c00bcc0 con 0x7fb05415db30 2023-03-03T07:55:18.880 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.877+0000 7fb05b19d700 1 -- 172.21.15.100:0/466227620 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb05415db30 msgr2=0x7fb0540b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:18.880 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.877+0000 7fb05b19d700 1 --2- 172.21.15.100:0/466227620 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb05415db30 0x7fb0540b4aa0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb04c009e50 tx=0x7fb04c00b330 comp rx=0 tx=0).stop 2023-03-03T07:55:18.880 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.877+0000 7fb05b19d700 1 -- 172.21.15.100:0/466227620 shutdown_connections 2023-03-03T07:55:18.880 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.877+0000 7fb05b19d700 1 --2- 172.21.15.100:0/466227620 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb05415db30 0x7fb0540b4aa0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:18.880 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.877+0000 7fb05b19d700 1 --2- 172.21.15.100:0/466227620 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0540cf570 0x7fb05415d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:18.880 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.877+0000 7fb05b19d700 1 -- 172.21.15.100:0/466227620 >> 172.21.15.100:0/466227620 conn(0x7fb05406b880 msgr2=0x7fb0540cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:18.880 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.877+0000 7fb05b19d700 1 -- 172.21.15.100:0/466227620 shutdown_connections 2023-03-03T07:55:18.881 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.877+0000 7fb05b19d700 1 -- 172.21.15.100:0/466227620 wait complete. 2023-03-03T07:55:18.881 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.879+0000 7fb05b19d700 1 Processor -- start 2023-03-03T07:55:18.881 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.879+0000 7fb05b19d700 1 -- start start 2023-03-03T07:55:18.882 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.880+0000 7fb05b19d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0540cf570 0x7fb054145000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:18.882 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.880+0000 7fb05b19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb05415db30 0x7fb0540b7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:18.882 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.880+0000 7fb05b19d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0541609b0 0x7fb054143340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:18.882 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.880+0000 7fb05b19d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb054142bd0 con 0x7fb0540cf570 2023-03-03T07:55:18.883 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.880+0000 7fb05b19d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb054142eb0 con 0x7fb0541609b0 2023-03-03T07:55:18.883 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.880+0000 7fb05b19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb054145540 con 0x7fb05415db30 2023-03-03T07:55:18.883 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.881+0000 7fb05999a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb05415db30 0x7fb0540b7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:18.883 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.881+0000 7fb05a99c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0541609b0 0x7fb054143340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:18.883 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.881+0000 7fb05999a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb05415db30 0x7fb0540b7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:33566/0 (socket says 172.21.15.100:33566) 2023-03-03T07:55:18.883 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.881+0000 7fb05999a700 1 -- 172.21.15.100:0/2689102660 learned_addr learned my addr 172.21.15.100:0/2689102660 (peer_addr_for_me v2:172.21.15.100:0/0) 2023-03-03T07:55:18.883 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.881+0000 7fb05a19b700 1 --2- 172.21.15.100:0/2689102660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0540cf570 0x7fb054145000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.881+0000 7fb05a99c700 1 -- 172.21.15.100:0/2689102660 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb05415db30 msgr2=0x7fb0540b7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.881+0000 7fb05a99c700 1 --2- 172.21.15.100:0/2689102660 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb05415db30 0x7fb0540b7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.881+0000 7fb05a99c700 1 -- 172.21.15.100:0/2689102660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0540cf570 msgr2=0x7fb054145000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.881+0000 7fb05a99c700 1 --2- 172.21.15.100:0/2689102660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0540cf570 0x7fb054145000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.881+0000 7fb05a99c700 1 -- 172.21.15.100:0/2689102660 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c009b30 con 0x7fb0541609b0 2023-03-03T07:55:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.881+0000 7fb05999a700 1 --2- 172.21.15.100:0/2689102660 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb05415db30 0x7fb0540b7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.882+0000 7fb05a99c700 1 --2- 172.21.15.100:0/2689102660 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0541609b0 0x7fb054143340 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb04c015000 tx=0x7fb04c004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:18.886 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.882+0000 7fb04affd700 1 -- 172.21.15.100:0/2689102660 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c004800 con 0x7fb0541609b0 2023-03-03T07:55:18.886 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.882+0000 7fb04affd700 1 -- 172.21.15.100:0/2689102660 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04c004960 con 0x7fb0541609b0 2023-03-03T07:55:18.886 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.882+0000 7fb04affd700 1 -- 172.21.15.100:0/2689102660 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c01c680 con 0x7fb0541609b0 2023-03-03T07:55:18.886 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.882+0000 7fb05b19d700 1 -- 172.21.15.100:0/2689102660 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb05415d0d0 con 0x7fb0541609b0 2023-03-03T07:55:18.886 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.882+0000 7fb05b19d700 1 -- 172.21.15.100:0/2689102660 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05415cdf0 con 0x7fb0541609b0 2023-03-03T07:55:18.886 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.882+0000 7fb04affd700 1 -- 172.21.15.100:0/2689102660 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fb04c022030 con 0x7fb0541609b0 2023-03-03T07:55:18.886 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.883+0000 7fb04affd700 1 -- 172.21.15.100:0/2689102660 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb04c025790 con 0x7fb0541609b0 2023-03-03T07:55:18.887 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.883+0000 7fb05b19d700 1 -- 172.21.15.100:0/2689102660 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0540b4fe0 con 0x7fb0541609b0 2023-03-03T07:55:18.887 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.886+0000 7fb04affd700 1 -- 172.21.15.100:0/2689102660 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fb04c020070 con 0x7fb0541609b0 2023-03-03T07:55:18.945 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:18.944+0000 7fcc07721200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-03T07:55:18.945 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:18.943+0000 7f8191787200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-03T07:55:18.989 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.988+0000 7fb05b19d700 1 -- 172.21.15.100:0/2689102660 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd new", "uuid": "18707639-bc5c-49a9-b8b6-d563deea2dea", "id": 0} v 0) v1 -- 0x7fb054068290 con 0x7fb0541609b0 2023-03-03T07:55:18.997 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.996+0000 7fb04affd700 1 -- 172.21.15.100:0/2689102660 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "18707639-bc5c-49a9-b8b6-d563deea2dea", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb04c01cd80 con 0x7fb0541609b0 2023-03-03T07:55:18.998 INFO:teuthology.orchestra.run.smithi100.stdout:0 2023-03-03T07:55:18.998 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.997+0000 7fb05b19d700 1 -- 172.21.15.100:0/2689102660 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0541609b0 msgr2=0x7fb054143340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:18.999 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.997+0000 7fb05b19d700 1 --2- 172.21.15.100:0/2689102660 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0541609b0 0x7fb054143340 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb04c015000 tx=0x7fb04c004ad0 comp rx=0 tx=0).stop 2023-03-03T07:55:18.999 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.997+0000 7fb05b19d700 1 -- 172.21.15.100:0/2689102660 shutdown_connections 2023-03-03T07:55:18.999 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.997+0000 7fb05b19d700 1 --2- 172.21.15.100:0/2689102660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0540cf570 0x7fb054145000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:18.999 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.997+0000 7fb05b19d700 1 --2- 172.21.15.100:0/2689102660 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0541609b0 0x7fb054143340 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:18.999 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.997+0000 7fb05b19d700 1 --2- 172.21.15.100:0/2689102660 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb05415db30 0x7fb0540b7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:18.999 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.997+0000 7fb05b19d700 1 -- 172.21.15.100:0/2689102660 >> 172.21.15.100:0/2689102660 conn(0x7fb05406b880 msgr2=0x7fb0540d1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:19.000 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.997+0000 7fb05b19d700 1 -- 172.21.15.100:0/2689102660 shutdown_connections 2023-03-03T07:55:19.000 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:18.997+0000 7fb05b19d700 1 -- 172.21.15.100:0/2689102660 wait complete. 2023-03-03T07:55:19.025 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 6e6ec2b2-e35f-4d13-a06a-a6f58af34e01 1 2023-03-03T07:55:19.052 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:19.049+0000 7f8191787200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-03T07:55:19.059 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:19.056+0000 7fcc07721200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-03T07:55:19.183 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:19.181+0000 7fcc07721200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-03T07:55:19.184 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:19.181+0000 7f8191787200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-03T07:55:19.263 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.262+0000 7faa715ff700 1 Processor -- start 2023-03-03T07:55:19.264 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.262+0000 7faa715ff700 1 -- start start 2023-03-03T07:55:19.266 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.263+0000 7faa715ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa6c0b26d0 0x7faa6c0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:19.266 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.263+0000 7faa715ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faa6c0b5140 0x7faa6c0d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:19.266 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.263+0000 7faa715ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6c0cd7d0 con 0x7faa6c0b8d10 2023-03-03T07:55:19.266 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.263+0000 7faa715ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa6c0b55c0 con 0x7faa6c0b5140 2023-03-03T07:55:19.266 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.263+0000 7faa715ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c0b90e0 con 0x7faa6c0b26d0 2023-03-03T07:55:19.266 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.264+0000 7faa6affd700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7faa6c0b8d10 0x7faa6c0cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:41934/0 (socket says 172.21.15.100:41934) 2023-03-03T07:55:19.267 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.264+0000 7faa6a7fc700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa6c0b26d0 0x7faa6c0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:19.267 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.264+0000 7faa6affd700 1 -- 172.21.15.100:0/501389506 learned_addr learned my addr 172.21.15.100:0/501389506 (peer_addr_for_me v1:172.21.15.100:0/0) 2023-03-03T07:55:19.267 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.264+0000 7faa6b7fe700 1 --2- 172.21.15.100:0/501389506 >> v2:172.21.15.98:3300/0 conn(0x7faa6c0b5140 0x7faa6c0d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:19.267 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.264+0000 7faa69ffb700 1 -- 172.21.15.100:0/501389506 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014329391 0 0) 0x7faa6c0cd7d0 con 0x7faa6c0b8d10 2023-03-03T07:55:19.267 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.264+0000 7faa6a7fc700 1 -- 172.21.15.100:0/501389506 >> v2:172.21.15.98:3300/0 conn(0x7faa6c0b5140 msgr2=0x7faa6c0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:19.267 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.264+0000 7faa6a7fc700 1 --2- 172.21.15.100:0/501389506 >> v2:172.21.15.98:3300/0 conn(0x7faa6c0b5140 0x7faa6c0d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.267 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.264+0000 7faa6a7fc700 1 -- 172.21.15.100:0/501389506 >> v1:172.21.15.100:6789/0 conn(0x7faa6c0b8d10 legacy=0x7faa6c0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:19.268 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.264+0000 7faa6a7fc700 1 -- 172.21.15.100:0/501389506 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6c0cda20 con 0x7faa6c0b26d0 2023-03-03T07:55:19.268 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.264+0000 7faa6b7fe700 1 --2- 172.21.15.100:0/501389506 >> v2:172.21.15.98:3300/0 conn(0x7faa6c0b5140 0x7faa6c0d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:19.268 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.265+0000 7faa6a7fc700 1 --2- 172.21.15.100:0/501389506 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa6c0b26d0 0x7faa6c0b4c00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7faa60009840 tx=0x7faa60009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e220a0e80fc8f526 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:19.269 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.265+0000 7faa69ffb700 1 -- 172.21.15.100:0/501389506 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa60011830 con 0x7faa6c0b26d0 2023-03-03T07:55:19.269 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.265+0000 7faa69ffb700 1 -- 172.21.15.100:0/501389506 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa60011990 con 0x7faa6c0b26d0 2023-03-03T07:55:19.270 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.265+0000 7faa69ffb700 1 -- 172.21.15.100:0/501389506 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa60011d20 con 0x7faa6c0b26d0 2023-03-03T07:55:19.270 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.266+0000 7faa715ff700 1 -- 172.21.15.100:0/501389506 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa6c0b26d0 msgr2=0x7faa6c0b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:19.270 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.266+0000 7faa715ff700 1 --2- 172.21.15.100:0/501389506 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa6c0b26d0 0x7faa6c0b4c00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7faa60009840 tx=0x7faa60009b50 comp rx=0 tx=0).stop 2023-03-03T07:55:19.270 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.266+0000 7faa715ff700 1 -- 172.21.15.100:0/501389506 shutdown_connections 2023-03-03T07:55:19.270 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.266+0000 7faa715ff700 1 --2- 172.21.15.100:0/501389506 >> v2:172.21.15.98:3300/0 conn(0x7faa6c0b5140 0x7faa6c0d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.270 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.266+0000 7faa715ff700 1 --2- 172.21.15.100:0/501389506 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa6c0b26d0 0x7faa6c0b4c00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.270 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.266+0000 7faa715ff700 1 -- 172.21.15.100:0/501389506 >> 172.21.15.100:0/501389506 conn(0x7faa6c073e70 msgr2=0x7faa6c06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:19.271 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.266+0000 7faa715ff700 1 -- 172.21.15.100:0/501389506 shutdown_connections 2023-03-03T07:55:19.271 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.266+0000 7faa715ff700 1 -- 172.21.15.100:0/501389506 wait complete. 2023-03-03T07:55:19.271 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.268+0000 7faa715ff700 1 Processor -- start 2023-03-03T07:55:19.271 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.268+0000 7faa715ff700 1 -- start start 2023-03-03T07:55:19.273 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.269+0000 7faa715ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa6c0b26d0 0x7faa6c0c6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:19.273 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.269+0000 7faa715ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa6c0b5140 0x7faa6c15f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:19.273 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.269+0000 7faa715ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa6c0b8d10 0x7faa6c156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:19.273 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.269+0000 7faa715ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c15ff20 con 0x7faa6c0b8d10 2023-03-03T07:55:19.273 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.269+0000 7faa715ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c157bc0 con 0x7faa6c0b26d0 2023-03-03T07:55:19.274 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.269+0000 7faa715ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c15f960 con 0x7faa6c0b5140 2023-03-03T07:55:19.274 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.269+0000 7faa6affd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa6c0b26d0 0x7faa6c0c6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:19.274 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.269+0000 7faa6a7fc700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa6c0b5140 0x7faa6c15f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:19.274 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.270+0000 7faa6affd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa6c0b26d0 0x7faa6c0c6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53254/0 (socket says 172.21.15.100:53254) 2023-03-03T07:55:19.274 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.270+0000 7faa6affd700 1 -- 172.21.15.100:0/1487576125 learned_addr learned my addr 172.21.15.100:0/1487576125 (peer_addr_for_me v2:172.21.15.100:0/0) 2023-03-03T07:55:19.274 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.270+0000 7faa6a7fc700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa6c0b5140 0x7faa6c15f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:33590/0 (socket says 172.21.15.100:33590) 2023-03-03T07:55:19.274 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.270+0000 7faa6b7fe700 1 --2- 172.21.15.100:0/1487576125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa6c0b8d10 0x7faa6c156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:19.274 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.270+0000 7faa6affd700 1 -- 172.21.15.100:0/1487576125 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa6c0b5140 msgr2=0x7faa6c15f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:19.275 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.270+0000 7faa6affd700 1 --2- 172.21.15.100:0/1487576125 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa6c0b5140 0x7faa6c15f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.275 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.270+0000 7faa6affd700 1 -- 172.21.15.100:0/1487576125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa6c0b8d10 msgr2=0x7faa6c156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:19.275 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.270+0000 7faa6affd700 1 --2- 172.21.15.100:0/1487576125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa6c0b8d10 0x7faa6c156a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.276 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.270+0000 7faa6affd700 1 -- 172.21.15.100:0/1487576125 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa600094a0 con 0x7faa6c0b26d0 2023-03-03T07:55:19.276 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.270+0000 7faa6affd700 1 --2- 172.21.15.100:0/1487576125 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa6c0b26d0 0x7faa6c0c6c70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faa5c00c550 tx=0x7faa5c00c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:19.276 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.271+0000 7faa537fe700 1 -- 172.21.15.100:0/1487576125 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c004c80 con 0x7faa6c0b26d0 2023-03-03T07:55:19.276 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.271+0000 7faa537fe700 1 -- 172.21.15.100:0/1487576125 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5c004e00 con 0x7faa6c0b26d0 2023-03-03T07:55:19.276 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.271+0000 7faa537fe700 1 -- 172.21.15.100:0/1487576125 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c007310 con 0x7faa6c0b26d0 2023-03-03T07:55:19.276 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.271+0000 7faa715ff700 1 -- 172.21.15.100:0/1487576125 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6c0d0e80 con 0x7faa6c0b26d0 2023-03-03T07:55:19.276 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.271+0000 7faa715ff700 1 -- 172.21.15.100:0/1487576125 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6c0cdd00 con 0x7faa6c0b26d0 2023-03-03T07:55:19.276 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.271+0000 7faa537fe700 1 -- 172.21.15.100:0/1487576125 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7faa5c01a020 con 0x7faa6c0b26d0 2023-03-03T07:55:19.277 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.271+0000 7faa537fe700 1 -- 172.21.15.100:0/1487576125 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7faa5c0133c0 con 0x7faa6c0b26d0 2023-03-03T07:55:19.277 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.272+0000 7faa715ff700 1 -- 172.21.15.100:0/1487576125 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa6c0cdfe0 con 0x7faa6c0b26d0 2023-03-03T07:55:19.277 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.274+0000 7faa537fe700 1 -- 172.21.15.100:0/1487576125 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7faa5c013580 con 0x7faa6c0b26d0 2023-03-03T07:55:19.365 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.363+0000 7faa715ff700 1 -- 172.21.15.100:0/1487576125 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6e6ec2b2-e35f-4d13-a06a-a6f58af34e01", "id": 1} v 0) v1 -- 0x7faa6c04e6e0 con 0x7faa6c0b26d0 2023-03-03T07:55:19.372 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.371+0000 7faa537fe700 1 -- 172.21.15.100:0/1487576125 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6e6ec2b2-e35f-4d13-a06a-a6f58af34e01", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7faa5c013a80 con 0x7faa6c0b26d0 2023-03-03T07:55:19.373 INFO:teuthology.orchestra.run.smithi100.stdout:1 2023-03-03T07:55:19.373 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.372+0000 7faa715ff700 1 -- 172.21.15.100:0/1487576125 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa6c0b26d0 msgr2=0x7faa6c0c6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:19.374 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.372+0000 7faa715ff700 1 --2- 172.21.15.100:0/1487576125 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa6c0b26d0 0x7faa6c0c6c70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faa5c00c550 tx=0x7faa5c00c860 comp rx=0 tx=0).stop 2023-03-03T07:55:19.374 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.372+0000 7faa715ff700 1 -- 172.21.15.100:0/1487576125 shutdown_connections 2023-03-03T07:55:19.374 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.372+0000 7faa715ff700 1 --2- 172.21.15.100:0/1487576125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa6c0b8d10 0x7faa6c156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.374 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.372+0000 7faa715ff700 1 --2- 172.21.15.100:0/1487576125 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa6c0b26d0 0x7faa6c0c6c70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.374 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.372+0000 7faa715ff700 1 --2- 172.21.15.100:0/1487576125 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa6c0b5140 0x7faa6c15f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.374 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.372+0000 7faa715ff700 1 -- 172.21.15.100:0/1487576125 >> 172.21.15.100:0/1487576125 conn(0x7faa6c073e70 msgr2=0x7faa6c0d13a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:19.375 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.372+0000 7faa715ff700 1 -- 172.21.15.100:0/1487576125 shutdown_connections 2023-03-03T07:55:19.375 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.372+0000 7faa715ff700 1 -- 172.21.15.100:0/1487576125 wait complete. 2023-03-03T07:55:19.399 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 8e6fc347-1bd9-4381-8410-2e0110bd5052 2 2023-03-03T07:55:19.448 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:19.446+0000 7f8191787200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-03T07:55:19.463 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:19.462+0000 7fcc07721200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-03T07:55:19.677 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.675+0000 7f7d3d99d700 1 Processor -- start 2023-03-03T07:55:19.677 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.676+0000 7f7d3d99d700 1 -- start start 2023-03-03T07:55:19.679 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.678+0000 7f7d3d99d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7d38160a20 0x7f7d38160df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:19.679 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.678+0000 7f7d3d99d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d380c3d40 0x7f7d380c4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:19.680 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.678+0000 7f7d3d99d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d380cc9c0 con 0x7f7d380b0040 2023-03-03T07:55:19.680 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.678+0000 7f7d3d99d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d380c46c0 con 0x7f7d38160a20 2023-03-03T07:55:19.680 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.678+0000 7f7d3d99d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380ce630 con 0x7f7d380c3d40 2023-03-03T07:55:19.681 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.679+0000 7f7d3c99b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7d38160a20 0x7f7d38160df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:19.681 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.679+0000 7f7d3d19c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d380c3d40 0x7f7d380c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:19.681 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.679+0000 7f7d3c99b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7d38160a20 0x7f7d38160df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53264/0 (socket says 172.21.15.100:53264) 2023-03-03T07:55:19.681 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.679+0000 7f7d3c99b700 1 -- 172.21.15.100:0/1059882476 learned_addr learned my addr 172.21.15.100:0/1059882476 (peer_addr_for_me v2:172.21.15.100:0/0) 2023-03-03T07:55:19.681 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.679+0000 7f7d3c99b700 1 -- 172.21.15.100:0/1059882476 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d380c3d40 msgr2=0x7f7d380c4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:19.682 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.679+0000 7f7d3c99b700 1 --2- 172.21.15.100:0/1059882476 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d380c3d40 0x7f7d380c4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.682 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.679+0000 7f7d3c99b700 1 -- 172.21.15.100:0/1059882476 >> v1:172.21.15.100:6789/0 conn(0x7f7d380b0040 legacy=0x7f7d380cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:19.682 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.679+0000 7f7d3c99b700 1 -- 172.21.15.100:0/1059882476 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d380cbdb0 con 0x7f7d38160a20 2023-03-03T07:55:19.682 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.680+0000 7f7d3c99b700 1 --2- 172.21.15.100:0/1059882476 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d38160a20 0x7f7d38160df0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0098f0 tx=0x7f7d2c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17fca531b790b0b6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:19.683 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.680+0000 7f7d377fe700 1 -- 172.21.15.100:0/1059882476 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c011830 con 0x7f7d38160a20 2023-03-03T07:55:19.683 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.680+0000 7f7d377fe700 1 -- 172.21.15.100:0/1059882476 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c011990 con 0x7f7d38160a20 2023-03-03T07:55:19.683 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.680+0000 7f7d377fe700 1 -- 172.21.15.100:0/1059882476 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c011d20 con 0x7f7d38160a20 2023-03-03T07:55:19.683 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.681+0000 7f7d3d99d700 1 -- 172.21.15.100:0/1059882476 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d38160a20 msgr2=0x7f7d38160df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:19.683 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.681+0000 7f7d3d99d700 1 --2- 172.21.15.100:0/1059882476 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d38160a20 0x7f7d38160df0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0098f0 tx=0x7f7d2c009c00 comp rx=0 tx=0).stop 2023-03-03T07:55:19.684 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.681+0000 7f7d3d99d700 1 -- 172.21.15.100:0/1059882476 shutdown_connections 2023-03-03T07:55:19.684 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.681+0000 7f7d3d99d700 1 --2- 172.21.15.100:0/1059882476 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d38160a20 0x7f7d38160df0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.684 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.681+0000 7f7d3d99d700 1 --2- 172.21.15.100:0/1059882476 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d380c3d40 0x7f7d380c4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.684 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.681+0000 7f7d3d99d700 1 -- 172.21.15.100:0/1059882476 >> 172.21.15.100:0/1059882476 conn(0x7f7d3809a0e0 msgr2=0x7f7d38073b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:19.684 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.681+0000 7f7d3d99d700 1 -- 172.21.15.100:0/1059882476 shutdown_connections 2023-03-03T07:55:19.685 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.681+0000 7f7d3d99d700 1 -- 172.21.15.100:0/1059882476 wait complete. 2023-03-03T07:55:19.685 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.683+0000 7f7d3d99d700 1 Processor -- start 2023-03-03T07:55:19.685 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.683+0000 7f7d3d99d700 1 -- start start 2023-03-03T07:55:19.686 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.684+0000 7f7d3d99d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d380c3d40 0x7f7d380c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:19.686 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.685+0000 7f7d3d99d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d38160a20 0x7f7d381fa930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:19.687 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.685+0000 7f7d3d99d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d38155150 0x7f7d380ae990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:19.687 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.685+0000 7f7d3d99d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d381fae70 con 0x7f7d38160a20 2023-03-03T07:55:19.687 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.685+0000 7f7d3d99d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380ca720 con 0x7f7d380c3d40 2023-03-03T07:55:19.687 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.685+0000 7f7d3d99d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380c9b10 con 0x7f7d38155150 2023-03-03T07:55:19.687 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.685+0000 7f7d3c99b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d380c3d40 0x7f7d380c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:19.687 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.685+0000 7f7d3d19c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d38155150 0x7f7d380ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:19.687 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.685+0000 7f7d3c99b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d380c3d40 0x7f7d380c92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53268/0 (socket says 172.21.15.100:53268) 2023-03-03T07:55:19.688 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.685+0000 7f7d3d19c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d38155150 0x7f7d380ae990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:33610/0 (socket says 172.21.15.100:33610) 2023-03-03T07:55:19.688 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.685+0000 7f7d3c99b700 1 -- 172.21.15.100:0/3124998943 learned_addr learned my addr 172.21.15.100:0/3124998943 (peer_addr_for_me v2:172.21.15.100:0/0) 2023-03-03T07:55:19.688 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.685+0000 7f7d37fff700 1 --2- 172.21.15.100:0/3124998943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d38160a20 0x7f7d381fa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:19.689 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.685+0000 7f7d3d19c700 1 -- 172.21.15.100:0/3124998943 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d380c3d40 msgr2=0x7f7d380c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:19.689 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.685+0000 7f7d3d19c700 1 --2- 172.21.15.100:0/3124998943 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d380c3d40 0x7f7d380c92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.689 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.686+0000 7f7d3d19c700 1 -- 172.21.15.100:0/3124998943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d38160a20 msgr2=0x7f7d381fa930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:19.689 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.686+0000 7f7d3d19c700 1 --2- 172.21.15.100:0/3124998943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d38160a20 0x7f7d381fa930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.690 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.686+0000 7f7d3d19c700 1 -- 172.21.15.100:0/3124998943 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c009550 con 0x7f7d38155150 2023-03-03T07:55:19.690 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.686+0000 7f7d3c99b700 1 --2- 172.21.15.100:0/3124998943 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d380c3d40 0x7f7d380c92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:19.690 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.686+0000 7f7d37fff700 1 --2- 172.21.15.100:0/3124998943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d38160a20 0x7f7d381fa930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:19.690 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.686+0000 7f7d3d19c700 1 --2- 172.21.15.100:0/3124998943 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d38155150 0x7f7d380ae990 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000efc0 tx=0x7f7d3000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:19.690 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.686+0000 7f7d357fa700 1 -- 172.21.15.100:0/3124998943 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3000cc90 con 0x7f7d38155150 2023-03-03T07:55:19.690 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.686+0000 7f7d357fa700 1 -- 172.21.15.100:0/3124998943 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3000f040 con 0x7f7d38155150 2023-03-03T07:55:19.690 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.686+0000 7f7d357fa700 1 -- 172.21.15.100:0/3124998943 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d300146c0 con 0x7f7d38155150 2023-03-03T07:55:19.691 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.686+0000 7f7d3d99d700 1 -- 172.21.15.100:0/3124998943 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d380cc370 con 0x7f7d38155150 2023-03-03T07:55:19.691 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.686+0000 7f7d3d99d700 1 -- 172.21.15.100:0/3124998943 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d380cc090 con 0x7f7d38155150 2023-03-03T07:55:19.691 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.687+0000 7f7d357fa700 1 -- 172.21.15.100:0/3124998943 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f7d30014820 con 0x7f7d38155150 2023-03-03T07:55:19.691 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.687+0000 7f7d357fa700 1 -- 172.21.15.100:0/3124998943 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f7d3001d630 con 0x7f7d38155150 2023-03-03T07:55:19.691 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.688+0000 7f7d3d99d700 1 -- 172.21.15.100:0/3124998943 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d380cc650 con 0x7f7d38155150 2023-03-03T07:55:19.694 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.693+0000 7f7d357fa700 1 -- 172.21.15.100:0/3124998943 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f7d3003bb00 con 0x7f7d38155150 2023-03-03T07:55:19.779 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.778+0000 7f7d3d99d700 1 -- 172.21.15.100:0/3124998943 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8e6fc347-1bd9-4381-8410-2e0110bd5052", "id": 2} v 0) v1 -- 0x7f7d38068290 con 0x7f7d38155150 2023-03-03T07:55:19.787 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.786+0000 7f7d357fa700 1 -- 172.21.15.100:0/3124998943 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8e6fc347-1bd9-4381-8410-2e0110bd5052", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7d3002a070 con 0x7f7d38155150 2023-03-03T07:55:19.788 INFO:teuthology.orchestra.run.smithi100.stdout:2 2023-03-03T07:55:19.788 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.787+0000 7f7d3d99d700 1 -- 172.21.15.100:0/3124998943 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d38155150 msgr2=0x7f7d380ae990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:19.789 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.787+0000 7f7d3d99d700 1 --2- 172.21.15.100:0/3124998943 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d38155150 0x7f7d380ae990 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000efc0 tx=0x7f7d3000c3a0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.789 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.787+0000 7f7d3d99d700 1 -- 172.21.15.100:0/3124998943 shutdown_connections 2023-03-03T07:55:19.789 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.787+0000 7f7d3d99d700 1 --2- 172.21.15.100:0/3124998943 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d38155150 0x7f7d380ae990 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.789 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.787+0000 7f7d3d99d700 1 --2- 172.21.15.100:0/3124998943 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d380c3d40 0x7f7d380c92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.789 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.787+0000 7f7d3d99d700 1 --2- 172.21.15.100:0/3124998943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d38160a20 0x7f7d381fa930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:19.789 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.787+0000 7f7d3d99d700 1 -- 172.21.15.100:0/3124998943 >> 172.21.15.100:0/3124998943 conn(0x7f7d3809a0e0 msgr2=0x7f7d3806b250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:19.790 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.787+0000 7f7d3d99d700 1 -- 172.21.15.100:0/3124998943 shutdown_connections 2023-03-03T07:55:19.790 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:19.787+0000 7f7d3d99d700 1 -- 172.21.15.100:0/3124998943 wait complete. 2023-03-03T07:55:19.814 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new cca208ef-d27f-4daf-b3c8-8bf22b693bd0 3 2023-03-03T07:55:20.092 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.089+0000 7f9e2f59e700 1 Processor -- start 2023-03-03T07:55:20.092 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.089+0000 7f9e2f59e700 1 -- start start 2023-03-03T07:55:20.093 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.091+0000 7f9e2f59e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9e301609b0 0x7f9e30160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:20.093 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.091+0000 7f9e2f59e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e301612c0 con 0x7f9e300b4aa0 2023-03-03T07:55:20.094 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.091+0000 7f9e2f59e700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3015dd00 con 0x7f9e300cf570 2023-03-03T07:55:20.094 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.091+0000 7f9e2f59e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e300cebb0 con 0x7f9e301609b0 2023-03-03T07:55:20.094 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.091+0000 7f9e2e59c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9e301609b0 0x7f9e30160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:20.094 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.091+0000 7f9e2e59c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9e301609b0 0x7f9e30160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:36070/0 (socket says 172.21.15.100:36070) 2023-03-03T07:55:20.094 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.091+0000 7f9e2e59c700 1 -- 172.21.15.100:0/1879660463 learned_addr learned my addr 172.21.15.100:0/1879660463 (peer_addr_for_me v2:172.21.15.100:0/0) 2023-03-03T07:55:20.095 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.092+0000 7f9e2e59c700 1 -- 172.21.15.100:0/1879660463 >> v1:172.21.15.100:6789/0 conn(0x7f9e300cf570 legacy=0x7f9e3015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.096 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.092+0000 7f9e2e59c700 1 -- 172.21.15.100:0/1879660463 >> v1:172.21.15.98:6789/0 conn(0x7f9e300b4aa0 legacy=0x7f9e300d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.096 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.092+0000 7f9e2e59c700 1 -- 172.21.15.100:0/1879660463 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e300cd780 con 0x7f9e301609b0 2023-03-03T07:55:20.096 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.092+0000 7f9e2e59c700 1 --2- 172.21.15.100:0/1879660463 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e301609b0 0x7f9e30160d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9e200098f0 tx=0x7f9e20009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70c6de235d2a1cb0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:20.096 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.092+0000 7f9e2dd9b700 1 -- 172.21.15.100:0/1879660463 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2000b8a0 con 0x7f9e301609b0 2023-03-03T07:55:20.096 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.093+0000 7f9e2dd9b700 1 -- 172.21.15.100:0/1879660463 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2000ba00 con 0x7f9e301609b0 2023-03-03T07:55:20.096 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.093+0000 7f9e2dd9b700 1 -- 172.21.15.100:0/1879660463 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2000bd90 con 0x7f9e301609b0 2023-03-03T07:55:20.096 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.093+0000 7f9e2f59e700 1 -- 172.21.15.100:0/1879660463 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e301609b0 msgr2=0x7f9e30160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.097 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.093+0000 7f9e2f59e700 1 --2- 172.21.15.100:0/1879660463 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e301609b0 0x7f9e30160d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9e200098f0 tx=0x7f9e20009c00 comp rx=0 tx=0).stop 2023-03-03T07:55:20.097 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.093+0000 7f9e2f59e700 1 -- 172.21.15.100:0/1879660463 shutdown_connections 2023-03-03T07:55:20.097 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.093+0000 7f9e2f59e700 1 --2- 172.21.15.100:0/1879660463 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e301609b0 0x7f9e30160d80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.097 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.093+0000 7f9e2f59e700 1 -- 172.21.15.100:0/1879660463 >> 172.21.15.100:0/1879660463 conn(0x7f9e3006b880 msgr2=0x7f9e300cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:20.098 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.093+0000 7f9e2f59e700 1 -- 172.21.15.100:0/1879660463 shutdown_connections 2023-03-03T07:55:20.099 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.093+0000 7f9e2f59e700 1 -- 172.21.15.100:0/1879660463 wait complete. 2023-03-03T07:55:20.100 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.095+0000 7f9e2f59e700 1 Processor -- start 2023-03-03T07:55:20.100 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.096+0000 7f9e2f59e700 1 -- start start 2023-03-03T07:55:20.101 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e2f59e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e300b4aa0 0x7f9e30151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:20.101 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e2f59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e300cf570 0x7f9e3015a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:20.101 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e2f59e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e301609b0 0x7f9e3015ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:20.101 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e2f59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e300b5e50 con 0x7f9e300cf570 2023-03-03T07:55:20.101 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e2f59e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301e84a0 con 0x7f9e301609b0 2023-03-03T07:55:20.101 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e2f59e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301e8780 con 0x7f9e300b4aa0 2023-03-03T07:55:20.102 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e2e59c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e300b4aa0 0x7f9e30151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:20.102 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e2ed9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e301609b0 0x7f9e3015ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:20.102 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e2e59c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e300b4aa0 0x7f9e30151410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:39460/0 (socket says 172.21.15.100:39460) 2023-03-03T07:55:20.102 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e2e59c700 1 -- 172.21.15.100:0/2638717095 learned_addr learned my addr 172.21.15.100:0/2638717095 (peer_addr_for_me v2:172.21.15.100:0/0) 2023-03-03T07:55:20.102 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e27fff700 1 --2- 172.21.15.100:0/2638717095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e300cf570 0x7f9e3015a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:20.102 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e2ed9d700 1 -- 172.21.15.100:0/2638717095 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e300b4aa0 msgr2=0x7f9e30151410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.102 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e2ed9d700 1 --2- 172.21.15.100:0/2638717095 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e300b4aa0 0x7f9e30151410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.103 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e2ed9d700 1 -- 172.21.15.100:0/2638717095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e300cf570 msgr2=0x7f9e3015a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.103 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e2ed9d700 1 --2- 172.21.15.100:0/2638717095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e300cf570 0x7f9e3015a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.103 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.097+0000 7f9e2ed9d700 1 -- 172.21.15.100:0/2638717095 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e20009550 con 0x7f9e301609b0 2023-03-03T07:55:20.103 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.098+0000 7f9e2e59c700 1 --2- 172.21.15.100:0/2638717095 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e300b4aa0 0x7f9e30151410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:20.103 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.098+0000 7f9e27fff700 1 --2- 172.21.15.100:0/2638717095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e300cf570 0x7f9e3015a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:20.103 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.098+0000 7f9e2ed9d700 1 --2- 172.21.15.100:0/2638717095 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e301609b0 0x7f9e3015ae70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800cb90 tx=0x7f9e2800cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:20.104 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.098+0000 7f9e26ffd700 1 -- 172.21.15.100:0/2638717095 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28003c00 con 0x7f9e301609b0 2023-03-03T07:55:20.104 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.098+0000 7f9e26ffd700 1 -- 172.21.15.100:0/2638717095 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e28003d60 con 0x7f9e301609b0 2023-03-03T07:55:20.104 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.098+0000 7f9e26ffd700 1 -- 172.21.15.100:0/2638717095 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2800aaa0 con 0x7f9e301609b0 2023-03-03T07:55:20.104 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.098+0000 7f9e2f59e700 1 -- 172.21.15.100:0/2638717095 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e300cdd40 con 0x7f9e301609b0 2023-03-03T07:55:20.105 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.098+0000 7f9e2f59e700 1 -- 172.21.15.100:0/2638717095 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e300cda60 con 0x7f9e301609b0 2023-03-03T07:55:20.105 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.099+0000 7f9e26ffd700 1 -- 172.21.15.100:0/2638717095 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f9e28010050 con 0x7f9e301609b0 2023-03-03T07:55:20.105 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.099+0000 7f9e26ffd700 1 -- 172.21.15.100:0/2638717095 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f9e28012420 con 0x7f9e301609b0 2023-03-03T07:55:20.106 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.100+0000 7f9e2f59e700 1 -- 172.21.15.100:0/2638717095 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e3015cdf0 con 0x7f9e301609b0 2023-03-03T07:55:20.106 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.103+0000 7f9e26ffd700 1 -- 172.21.15.100:0/2638717095 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f9e280050d0 con 0x7f9e301609b0 2023-03-03T07:55:20.194 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.192+0000 7f9e2f59e700 1 -- 172.21.15.100:0/2638717095 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cca208ef-d27f-4daf-b3c8-8bf22b693bd0", "id": 3} v 0) v1 -- 0x7f9e300b5fc0 con 0x7f9e301609b0 2023-03-03T07:55:20.202 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.199+0000 7f9e26ffd700 1 -- 172.21.15.100:0/2638717095 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cca208ef-d27f-4daf-b3c8-8bf22b693bd0", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9e28005330 con 0x7f9e301609b0 2023-03-03T07:55:20.202 INFO:teuthology.orchestra.run.smithi100.stdout:3 2023-03-03T07:55:20.203 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.201+0000 7f9e2f59e700 1 -- 172.21.15.100:0/2638717095 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e301609b0 msgr2=0x7f9e3015ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.203 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.201+0000 7f9e2f59e700 1 --2- 172.21.15.100:0/2638717095 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e301609b0 0x7f9e3015ae70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800cb90 tx=0x7f9e2800cea0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.204 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.201+0000 7f9e2f59e700 1 -- 172.21.15.100:0/2638717095 shutdown_connections 2023-03-03T07:55:20.204 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.201+0000 7f9e2f59e700 1 --2- 172.21.15.100:0/2638717095 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e301609b0 0x7f9e3015ae70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.204 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.201+0000 7f9e2f59e700 1 --2- 172.21.15.100:0/2638717095 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e300b4aa0 0x7f9e30151410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.204 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.201+0000 7f9e2f59e700 1 --2- 172.21.15.100:0/2638717095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e300cf570 0x7f9e3015a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.204 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.201+0000 7f9e2f59e700 1 -- 172.21.15.100:0/2638717095 >> 172.21.15.100:0/2638717095 conn(0x7f9e3006b880 msgr2=0x7f9e300b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:20.204 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.201+0000 7f9e2f59e700 1 -- 172.21.15.100:0/2638717095 shutdown_connections 2023-03-03T07:55:20.204 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.201+0000 7f9e2f59e700 1 -- 172.21.15.100:0/2638717095 wait complete. 2023-03-03T07:55:20.228 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 8f10aa62-7454-4d09-a38c-0ed613912eb5 4 2023-03-03T07:55:20.503 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.501+0000 7f8dfbfff700 1 Processor -- start 2023-03-03T07:55:20.504 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.501+0000 7f8dfbfff700 1 -- start start 2023-03-03T07:55:20.505 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.503+0000 7f8dfbfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dfc0cf570 0x7f8dfc15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:20.506 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.503+0000 7f8dfbfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8dfc15db30 0x7f8dfc0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:20.506 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.503+0000 7f8dfbfff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dfc0cd780 con 0x7f8dfc1609b0 2023-03-03T07:55:20.506 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.503+0000 7f8dfbfff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dfc0cebb0 con 0x7f8dfc15db30 2023-03-03T07:55:20.506 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.503+0000 7f8dfbfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc0cda60 con 0x7f8dfc0cf570 2023-03-03T07:55:20.506 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.503+0000 7f8dfa7fc700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dfc0cf570 0x7f8dfc15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:20.506 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.503+0000 7f8dfaffd700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f8dfc1609b0 0x7f8dfc160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:37932/0 (socket says 172.21.15.100:37932) 2023-03-03T07:55:20.507 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.503+0000 7f8dfaffd700 1 -- 172.21.15.100:0/3772618224 learned_addr learned my addr 172.21.15.100:0/3772618224 (peer_addr_for_me v1:172.21.15.100:0/0) 2023-03-03T07:55:20.507 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.503+0000 7f8dfb7fe700 1 --2- 172.21.15.100:0/3772618224 >> v2:172.21.15.98:3300/0 conn(0x7f8dfc15db30 0x7f8dfc0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:20.507 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:20.505+0000 7f8191787200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-03T07:55:20.508 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.504+0000 7f8dfa7fc700 1 -- 172.21.15.100:0/3772618224 >> v2:172.21.15.98:3300/0 conn(0x7f8dfc15db30 msgr2=0x7f8dfc0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:20.509 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.504+0000 7f8dfa7fc700 1 --2- 172.21.15.100:0/3772618224 >> v2:172.21.15.98:3300/0 conn(0x7f8dfc15db30 0x7f8dfc0b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.509 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.504+0000 7f8dfa7fc700 1 -- 172.21.15.100:0/3772618224 >> v1:172.21.15.100:6789/0 conn(0x7f8dfc1609b0 legacy=0x7f8dfc160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.509 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.504+0000 7f8df9ffb700 1 -- 172.21.15.100:0/3772618224 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917066658 0 0) 0x7f8dfc0cd780 con 0x7f8dfc1609b0 2023-03-03T07:55:20.509 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.504+0000 7f8dfa7fc700 1 -- 172.21.15.100:0/3772618224 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dfc0cdd40 con 0x7f8dfc0cf570 2023-03-03T07:55:20.509 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.504+0000 7f8dfa7fc700 1 --2- 172.21.15.100:0/3772618224 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dfc0cf570 0x7f8dfc15d5f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8df0009840 tx=0x7f8df0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caa96bbe471ebf4a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:20.509 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.504+0000 7f8df9ffb700 1 -- 172.21.15.100:0/3772618224 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df0011830 con 0x7f8dfc0cf570 2023-03-03T07:55:20.510 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.504+0000 7f8df9ffb700 1 -- 172.21.15.100:0/3772618224 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df0011990 con 0x7f8dfc0cf570 2023-03-03T07:55:20.510 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.504+0000 7f8df9ffb700 1 -- 172.21.15.100:0/3772618224 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df0011d20 con 0x7f8dfc0cf570 2023-03-03T07:55:20.510 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.505+0000 7f8dfbfff700 1 -- 172.21.15.100:0/3772618224 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dfc0cf570 msgr2=0x7f8dfc15d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.510 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.505+0000 7f8dfbfff700 1 --2- 172.21.15.100:0/3772618224 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dfc0cf570 0x7f8dfc15d5f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8df0009840 tx=0x7f8df0009b50 comp rx=0 tx=0).stop 2023-03-03T07:55:20.510 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.505+0000 7f8dfbfff700 1 -- 172.21.15.100:0/3772618224 shutdown_connections 2023-03-03T07:55:20.510 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.505+0000 7f8dfbfff700 1 --2- 172.21.15.100:0/3772618224 >> v2:172.21.15.98:3300/0 conn(0x7f8dfc15db30 0x7f8dfc0b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.510 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.505+0000 7f8dfbfff700 1 --2- 172.21.15.100:0/3772618224 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dfc0cf570 0x7f8dfc15d5f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.511 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.505+0000 7f8dfbfff700 1 -- 172.21.15.100:0/3772618224 >> 172.21.15.100:0/3772618224 conn(0x7f8dfc06b880 msgr2=0x7f8dfc0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:20.511 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.505+0000 7f8dfbfff700 1 -- 172.21.15.100:0/3772618224 shutdown_connections 2023-03-03T07:55:20.511 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.505+0000 7f8dfbfff700 1 -- 172.21.15.100:0/3772618224 wait complete. 2023-03-03T07:55:20.511 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.508+0000 7f8dfbfff700 1 Processor -- start 2023-03-03T07:55:20.512 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.509+0000 7f8dfbfff700 1 -- start start 2023-03-03T07:55:20.513 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.510+0000 7f8dfbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dfc0cf570 0x7f8dfc151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:20.513 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.510+0000 7f8dfbfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dfc15db30 0x7f8dfc0b7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:20.513 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.510+0000 7f8dfbfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dfc1609b0 0x7f8dfc0b5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:20.513 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.510+0000 7f8dfbfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc0b67f0 con 0x7f8dfc0cf570 2023-03-03T07:55:20.514 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.510+0000 7f8dfbfff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc0b6ad0 con 0x7f8dfc15db30 2023-03-03T07:55:20.514 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.510+0000 7f8dfbfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc0b6db0 con 0x7f8dfc1609b0 2023-03-03T07:55:20.514 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.510+0000 7f8dfa7fc700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dfc15db30 0x7f8dfc0b7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:20.514 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.510+0000 7f8dfaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dfc0cf570 0x7f8dfc151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:20.514 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.510+0000 7f8dfa7fc700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dfc15db30 0x7f8dfc0b7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:36090/0 (socket says 172.21.15.100:36090) 2023-03-03T07:55:20.514 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.510+0000 7f8dfa7fc700 1 -- 172.21.15.100:0/1378583027 learned_addr learned my addr 172.21.15.100:0/1378583027 (peer_addr_for_me v2:172.21.15.100:0/0) 2023-03-03T07:55:20.515 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.510+0000 7f8dfb7fe700 1 --2- 172.21.15.100:0/1378583027 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dfc1609b0 0x7f8dfc0b5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:20.515 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.511+0000 7f8dfa7fc700 1 -- 172.21.15.100:0/1378583027 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dfc1609b0 msgr2=0x7f8dfc0b5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.515 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.511+0000 7f8dfa7fc700 1 --2- 172.21.15.100:0/1378583027 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dfc1609b0 0x7f8dfc0b5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.515 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.511+0000 7f8dfa7fc700 1 -- 172.21.15.100:0/1378583027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dfc0cf570 msgr2=0x7f8dfc151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.515 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.511+0000 7f8dfa7fc700 1 --2- 172.21.15.100:0/1378583027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dfc0cf570 0x7f8dfc151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.515 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.511+0000 7f8dfa7fc700 1 -- 172.21.15.100:0/1378583027 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df00094a0 con 0x7f8dfc15db30 2023-03-03T07:55:20.516 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.511+0000 7f8dfaffd700 1 --2- 172.21.15.100:0/1378583027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dfc0cf570 0x7f8dfc151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:20.516 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.511+0000 7f8dfb7fe700 1 --2- 172.21.15.100:0/1378583027 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dfc1609b0 0x7f8dfc0b5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:20.517 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.511+0000 7f8dfa7fc700 1 --2- 172.21.15.100:0/1378583027 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dfc15db30 0x7f8dfc0b7890 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8df0009fd0 tx=0x7f8df0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:20.517 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.511+0000 7f8de37fe700 1 -- 172.21.15.100:0/1378583027 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df00036a0 con 0x7f8dfc15db30 2023-03-03T07:55:20.517 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.512+0000 7f8de37fe700 1 -- 172.21.15.100:0/1378583027 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df001f070 con 0x7f8dfc15db30 2023-03-03T07:55:20.517 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.512+0000 7f8dfbfff700 1 -- 172.21.15.100:0/1378583027 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dfc15d0d0 con 0x7f8dfc15db30 2023-03-03T07:55:20.517 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.512+0000 7f8de37fe700 1 -- 172.21.15.100:0/1378583027 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df0003a60 con 0x7f8dfc15db30 2023-03-03T07:55:20.518 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.512+0000 7f8dfbfff700 1 -- 172.21.15.100:0/1378583027 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dfc15cdf0 con 0x7f8dfc15db30 2023-03-03T07:55:20.518 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.512+0000 7f8de37fe700 1 -- 172.21.15.100:0/1378583027 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f8df000f050 con 0x7f8dfc15db30 2023-03-03T07:55:20.518 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.512+0000 7f8de37fe700 1 -- 172.21.15.100:0/1378583027 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f8df0021070 con 0x7f8dfc15db30 2023-03-03T07:55:20.518 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.513+0000 7f8dfbfff700 1 -- 172.21.15.100:0/1378583027 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dfc0b4fe0 con 0x7f8dfc15db30 2023-03-03T07:55:20.518 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.515+0000 7f8de37fe700 1 -- 172.21.15.100:0/1378583027 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f8df0015020 con 0x7f8dfc15db30 2023-03-03T07:55:20.581 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:20.578+0000 7fcc07721200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-03T07:55:20.606 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.603+0000 7f8dfbfff700 1 -- 172.21.15.100:0/1378583027 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8f10aa62-7454-4d09-a38c-0ed613912eb5", "id": 4} v 0) v1 -- 0x7f8dfc068290 con 0x7f8dfc15db30 2023-03-03T07:55:20.612 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.609+0000 7f8de37fe700 1 -- 172.21.15.100:0/1378583027 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8f10aa62-7454-4d09-a38c-0ed613912eb5", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8df0026480 con 0x7f8dfc15db30 2023-03-03T07:55:20.612 INFO:teuthology.orchestra.run.smithi100.stdout:4 2023-03-03T07:55:20.613 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.611+0000 7f8dfbfff700 1 -- 172.21.15.100:0/1378583027 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dfc15db30 msgr2=0x7f8dfc0b7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.613 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.611+0000 7f8dfbfff700 1 --2- 172.21.15.100:0/1378583027 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dfc15db30 0x7f8dfc0b7890 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8df0009fd0 tx=0x7f8df0004750 comp rx=0 tx=0).stop 2023-03-03T07:55:20.613 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.611+0000 7f8dfbfff700 1 -- 172.21.15.100:0/1378583027 shutdown_connections 2023-03-03T07:55:20.614 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.611+0000 7f8dfbfff700 1 --2- 172.21.15.100:0/1378583027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dfc0cf570 0x7f8dfc151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.614 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.611+0000 7f8dfbfff700 1 --2- 172.21.15.100:0/1378583027 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dfc1609b0 0x7f8dfc0b5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.614 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.611+0000 7f8dfbfff700 1 --2- 172.21.15.100:0/1378583027 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dfc15db30 0x7f8dfc0b7890 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.614 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.611+0000 7f8dfbfff700 1 -- 172.21.15.100:0/1378583027 >> 172.21.15.100:0/1378583027 conn(0x7f8dfc06b880 msgr2=0x7f8dfc0d2430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:20.614 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.611+0000 7f8dfbfff700 1 -- 172.21.15.100:0/1378583027 shutdown_connections 2023-03-03T07:55:20.614 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.611+0000 7f8dfbfff700 1 -- 172.21.15.100:0/1378583027 wait complete. 2023-03-03T07:55:20.641 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 265a0fef-4906-46a8-bf8e-505bdb8c3180 5 2023-03-03T07:55:20.735 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:20.733+0000 7f8191787200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-03T07:55:20.821 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:20.818+0000 7fcc07721200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-03T07:55:20.881 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.879+0000 7f23237ff700 1 Processor -- start 2023-03-03T07:55:20.882 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.879+0000 7f23237ff700 1 -- start start 2023-03-03T07:55:20.883 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.881+0000 7f23237ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f231c0b26d0 0x7f231c0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:20.883 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.881+0000 7f23237ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f231c0b5140 0x7f231c0d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:20.884 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.881+0000 7f23237ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231c0cd7d0 con 0x7f231c0b8d10 2023-03-03T07:55:20.884 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.881+0000 7f23237ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f231c0b55c0 con 0x7f231c0b26d0 2023-03-03T07:55:20.884 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.881+0000 7f23237ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231c0b90e0 con 0x7f231c0b5140 2023-03-03T07:55:20.884 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.881+0000 7f2321d9c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f231c0b5140 0x7f231c0d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:20.885 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.881+0000 7f232159b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f231c0b8d10 0x7f231c0cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.100:43418/0 (socket says 172.21.15.100:43418) 2023-03-03T07:55:20.885 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.881+0000 7f232159b700 1 -- 172.21.15.100:0/2928662379 learned_addr learned my addr 172.21.15.100:0/2928662379 (peer_addr_for_me v1:172.21.15.100:0/0) 2023-03-03T07:55:20.885 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.881+0000 7f2320d9a700 1 --2- 172.21.15.100:0/2928662379 >> v2:172.21.15.100:3300/0 conn(0x7f231c0b26d0 0x7f231c0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:20.885 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.882+0000 7f2321d9c700 1 -- 172.21.15.100:0/2928662379 >> v2:172.21.15.100:3300/0 conn(0x7f231c0b26d0 msgr2=0x7f231c0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:20.885 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.882+0000 7f2321d9c700 1 --2- 172.21.15.100:0/2928662379 >> v2:172.21.15.100:3300/0 conn(0x7f231c0b26d0 0x7f231c0b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.885 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.882+0000 7f2321d9c700 1 -- 172.21.15.100:0/2928662379 >> v1:172.21.15.98:6789/0 conn(0x7f231c0b8d10 legacy=0x7f231c0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.885 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.882+0000 7f2313fff700 1 -- 172.21.15.100:0/2928662379 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625539678 0 0) 0x7f231c0cd7d0 con 0x7f231c0b8d10 2023-03-03T07:55:20.886 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.882+0000 7f2321d9c700 1 -- 172.21.15.100:0/2928662379 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c0cda20 con 0x7f231c0b5140 2023-03-03T07:55:20.886 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.882+0000 7f2321d9c700 1 --2- 172.21.15.100:0/2928662379 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f231c0b5140 0x7f231c0d13a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2314009d90 tx=0x7f231400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60436fb5d5a9d5f3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:20.887 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.882+0000 7f2313fff700 1 -- 172.21.15.100:0/2928662379 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231400e070 con 0x7f231c0b5140 2023-03-03T07:55:20.887 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.882+0000 7f2313fff700 1 -- 172.21.15.100:0/2928662379 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231400bb20 con 0x7f231c0b5140 2023-03-03T07:55:20.887 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.882+0000 7f2313fff700 1 -- 172.21.15.100:0/2928662379 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2314012430 con 0x7f231c0b5140 2023-03-03T07:55:20.887 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.883+0000 7f23237ff700 1 -- 172.21.15.100:0/2928662379 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f231c0b5140 msgr2=0x7f231c0d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.887 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.883+0000 7f23237ff700 1 --2- 172.21.15.100:0/2928662379 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f231c0b5140 0x7f231c0d13a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2314009d90 tx=0x7f231400b330 comp rx=0 tx=0).stop 2023-03-03T07:55:20.887 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.883+0000 7f23237ff700 1 -- 172.21.15.100:0/2928662379 shutdown_connections 2023-03-03T07:55:20.888 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.883+0000 7f23237ff700 1 --2- 172.21.15.100:0/2928662379 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f231c0b5140 0x7f231c0d13a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.888 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.883+0000 7f23237ff700 1 --2- 172.21.15.100:0/2928662379 >> v2:172.21.15.100:3300/0 conn(0x7f231c0b26d0 0x7f231c0b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.888 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.883+0000 7f23237ff700 1 -- 172.21.15.100:0/2928662379 >> 172.21.15.100:0/2928662379 conn(0x7f231c073e70 msgr2=0x7f231c06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:20.888 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.883+0000 7f23237ff700 1 -- 172.21.15.100:0/2928662379 shutdown_connections 2023-03-03T07:55:20.888 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.883+0000 7f23237ff700 1 -- 172.21.15.100:0/2928662379 wait complete. 2023-03-03T07:55:20.890 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.885+0000 7f23237ff700 1 Processor -- start 2023-03-03T07:55:20.890 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.885+0000 7f23237ff700 1 -- start start 2023-03-03T07:55:20.890 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.886+0000 7f23237ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f231c0b26d0 0x7f231c1f61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:20.891 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.886+0000 7f23237ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f231c0b5140 0x7f231c0ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:20.891 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.886+0000 7f23237ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f231c0b8d10 0x7f231c1e84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:20.891 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.886+0000 7f23237ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c0ca960 con 0x7f231c0b5140 2023-03-03T07:55:20.891 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.886+0000 7f23237ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c0cac40 con 0x7f231c0b26d0 2023-03-03T07:55:20.891 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.886+0000 7f23237ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231c0caf20 con 0x7f231c0b8d10 2023-03-03T07:55:20.891 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.887+0000 7f232159b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f231c0b26d0 0x7f231c1f61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:20.891 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.887+0000 7f232159b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f231c0b26d0 0x7f231c1f61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:36104/0 (socket says 172.21.15.100:36104) 2023-03-03T07:55:20.892 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.887+0000 7f232159b700 1 -- 172.21.15.100:0/2951672473 learned_addr learned my addr 172.21.15.100:0/2951672473 (peer_addr_for_me v2:172.21.15.100:0/0) 2023-03-03T07:55:20.892 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.887+0000 7f2320d9a700 1 --2- 172.21.15.100:0/2951672473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f231c0b5140 0x7f231c0ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:20.892 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.887+0000 7f2321d9c700 1 --2- 172.21.15.100:0/2951672473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f231c0b8d10 0x7f231c1e84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:20.892 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.887+0000 7f232159b700 1 -- 172.21.15.100:0/2951672473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f231c0b8d10 msgr2=0x7f231c1e84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.892 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.887+0000 7f232159b700 1 --2- 172.21.15.100:0/2951672473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f231c0b8d10 0x7f231c1e84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.893 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.887+0000 7f232159b700 1 -- 172.21.15.100:0/2951672473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f231c0b5140 msgr2=0x7f231c0ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.893 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.887+0000 7f232159b700 1 --2- 172.21.15.100:0/2951672473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f231c0b5140 0x7f231c0ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.893 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.887+0000 7f232159b700 1 -- 172.21.15.100:0/2951672473 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2314009a70 con 0x7f231c0b26d0 2023-03-03T07:55:20.893 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.887+0000 7f2320d9a700 1 --2- 172.21.15.100:0/2951672473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f231c0b5140 0x7f231c0ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:20.893 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.888+0000 7f232159b700 1 --2- 172.21.15.100:0/2951672473 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f231c0b26d0 0x7f231c1f61e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f231800c550 tx=0x7f231800c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:20.894 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.888+0000 7f2311ffb700 1 -- 172.21.15.100:0/2951672473 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2318004c80 con 0x7f231c0b26d0 2023-03-03T07:55:20.894 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.888+0000 7f2311ffb700 1 -- 172.21.15.100:0/2951672473 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2318004e00 con 0x7f231c0b26d0 2023-03-03T07:55:20.894 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.888+0000 7f23237ff700 1 -- 172.21.15.100:0/2951672473 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231c0d0e80 con 0x7f231c0b26d0 2023-03-03T07:55:20.894 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.888+0000 7f2311ffb700 1 -- 172.21.15.100:0/2951672473 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2318007310 con 0x7f231c0b26d0 2023-03-03T07:55:20.895 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.888+0000 7f23237ff700 1 -- 172.21.15.100:0/2951672473 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231c0cdd00 con 0x7f231c0b26d0 2023-03-03T07:55:20.895 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.888+0000 7f2311ffb700 1 -- 172.21.15.100:0/2951672473 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f231801a020 con 0x7f231c0b26d0 2023-03-03T07:55:20.896 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.889+0000 7f2311ffb700 1 -- 172.21.15.100:0/2951672473 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f231801d3f0 con 0x7f231c0b26d0 2023-03-03T07:55:20.896 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.889+0000 7f23237ff700 1 -- 172.21.15.100:0/2951672473 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f231c0cdfe0 con 0x7f231c0b26d0 2023-03-03T07:55:20.896 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.892+0000 7f2311ffb700 1 -- 172.21.15.100:0/2951672473 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f2318007470 con 0x7f231c0b26d0 2023-03-03T07:55:20.983 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.981+0000 7f23237ff700 1 -- 172.21.15.100:0/2951672473 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd new", "uuid": "265a0fef-4906-46a8-bf8e-505bdb8c3180", "id": 5} v 0) v1 -- 0x7f231c04e6e0 con 0x7f231c0b26d0 2023-03-03T07:55:20.988 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.985+0000 7f2311ffb700 1 -- 172.21.15.100:0/2951672473 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "265a0fef-4906-46a8-bf8e-505bdb8c3180", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2318007610 con 0x7f231c0b26d0 2023-03-03T07:55:20.988 INFO:teuthology.orchestra.run.smithi100.stdout:5 2023-03-03T07:55:20.989 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.986+0000 7f23237ff700 1 -- 172.21.15.100:0/2951672473 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f231c0b26d0 msgr2=0x7f231c1f61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:20.989 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.986+0000 7f23237ff700 1 --2- 172.21.15.100:0/2951672473 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f231c0b26d0 0x7f231c1f61e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f231800c550 tx=0x7f231800c860 comp rx=0 tx=0).stop 2023-03-03T07:55:20.989 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.987+0000 7f23237ff700 1 -- 172.21.15.100:0/2951672473 shutdown_connections 2023-03-03T07:55:20.989 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.987+0000 7f23237ff700 1 --2- 172.21.15.100:0/2951672473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f231c0b8d10 0x7f231c1e84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.990 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.987+0000 7f23237ff700 1 --2- 172.21.15.100:0/2951672473 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f231c0b26d0 0x7f231c1f61e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.990 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.987+0000 7f23237ff700 1 --2- 172.21.15.100:0/2951672473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f231c0b5140 0x7f231c0ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:20.990 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.987+0000 7f23237ff700 1 -- 172.21.15.100:0/2951672473 >> 172.21.15.100:0/2951672473 conn(0x7f231c073e70 msgr2=0x7f231c0b68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:20.990 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.987+0000 7f23237ff700 1 -- 172.21.15.100:0/2951672473 shutdown_connections 2023-03-03T07:55:20.990 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:20.987+0000 7f23237ff700 1 -- 172.21.15.100:0/2951672473 wait complete. 2023-03-03T07:55:21.018 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 77a7054d-959a-437e-9c4b-ed58e78b58b4 6 2023-03-03T07:55:21.038 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:21.036+0000 7f8191787200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-03T07:55:21.151 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:21.149+0000 7fcc07721200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-03T07:55:21.156 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:21.154+0000 7f8191787200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-03T07:55:21.264 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.262+0000 7f3a289fe700 1 Processor -- start 2023-03-03T07:55:21.265 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.262+0000 7f3a289fe700 1 -- start start 2023-03-03T07:55:21.266 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.264+0000 7f3a289fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3a241609b0 0x7f3a24160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:21.266 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.264+0000 7f3a289fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3a240cf570 0x7f3a2415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:21.266 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.264+0000 7f3a289fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a2415db30 0x7f3a240b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:21.266 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.264+0000 7f3a289fe700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a240cebb0 con 0x7f3a240cf570 2023-03-03T07:55:21.267 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.264+0000 7f3a289fe700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a241612c0 con 0x7f3a241609b0 2023-03-03T07:55:21.267 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.264+0000 7f3a289fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a240cd780 con 0x7f3a2415db30 2023-03-03T07:55:21.268 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.264+0000 7f3a2359e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a2415db30 0x7f3a240b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:21.268 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.264+0000 7f3a22d9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3a241609b0 0x7f3a24160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:21.268 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.264+0000 7f3a2259c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3a240cf570 0x7f3a2415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:21.268 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.264+0000 7f3a2359e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a2415db30 0x7f3a240b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:39496/0 (socket says 172.21.15.100:39496) 2023-03-03T07:55:21.268 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.264+0000 7f3a22d9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3a241609b0 0x7f3a24160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:36108/0 (socket says 172.21.15.100:36108) 2023-03-03T07:55:21.269 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.264+0000 7f3a2359e700 1 -- 172.21.15.100:0/2040141339 learned_addr learned my addr 172.21.15.100:0/2040141339 (peer_addr_for_me v2:172.21.15.100:0/0) 2023-03-03T07:55:21.269 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.265+0000 7f3a2359e700 1 -- 172.21.15.100:0/2040141339 >> v2:172.21.15.100:3300/0 conn(0x7f3a241609b0 msgr2=0x7f3a24160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:21.269 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.265+0000 7f3a2359e700 1 --2- 172.21.15.100:0/2040141339 >> v2:172.21.15.100:3300/0 conn(0x7f3a241609b0 0x7f3a24160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.269 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.265+0000 7f3a2359e700 1 -- 172.21.15.100:0/2040141339 >> v2:172.21.15.98:3300/0 conn(0x7f3a240cf570 msgr2=0x7f3a2415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:21.269 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.265+0000 7f3a2359e700 1 --2- 172.21.15.100:0/2040141339 >> v2:172.21.15.98:3300/0 conn(0x7f3a240cf570 0x7f3a2415d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.269 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.265+0000 7f3a2359e700 1 -- 172.21.15.100:0/2040141339 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a240cda60 con 0x7f3a2415db30 2023-03-03T07:55:21.269 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.265+0000 7f3a22d9d700 1 --2- 172.21.15.100:0/2040141339 >> v2:172.21.15.100:3300/0 conn(0x7f3a241609b0 0x7f3a24160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:21.270 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.265+0000 7f3a2359e700 1 --2- 172.21.15.100:0/2040141339 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a2415db30 0x7f3a240b4aa0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3a240742b0 tx=0x7f3a1c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c157e405dd1a10e6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:21.270 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.265+0000 7f3a21d9b700 1 -- 172.21.15.100:0/2040141339 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c00e070 con 0x7f3a2415db30 2023-03-03T07:55:21.270 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.265+0000 7f3a21d9b700 1 -- 172.21.15.100:0/2040141339 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1c00b910 con 0x7f3a2415db30 2023-03-03T07:55:21.270 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.265+0000 7f3a21d9b700 1 -- 172.21.15.100:0/2040141339 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c00bcc0 con 0x7f3a2415db30 2023-03-03T07:55:21.270 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.266+0000 7f3a289fe700 1 -- 172.21.15.100:0/2040141339 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a2415db30 msgr2=0x7f3a240b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:21.270 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.266+0000 7f3a289fe700 1 --2- 172.21.15.100:0/2040141339 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a2415db30 0x7f3a240b4aa0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3a240742b0 tx=0x7f3a1c00b330 comp rx=0 tx=0).stop 2023-03-03T07:55:21.271 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.266+0000 7f3a289fe700 1 -- 172.21.15.100:0/2040141339 shutdown_connections 2023-03-03T07:55:21.271 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.266+0000 7f3a289fe700 1 --2- 172.21.15.100:0/2040141339 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a2415db30 0x7f3a240b4aa0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.272 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.266+0000 7f3a289fe700 1 --2- 172.21.15.100:0/2040141339 >> v2:172.21.15.100:3300/0 conn(0x7f3a241609b0 0x7f3a24160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.272 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.266+0000 7f3a289fe700 1 --2- 172.21.15.100:0/2040141339 >> v2:172.21.15.98:3300/0 conn(0x7f3a240cf570 0x7f3a2415d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.272 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.266+0000 7f3a289fe700 1 -- 172.21.15.100:0/2040141339 >> 172.21.15.100:0/2040141339 conn(0x7f3a2406b880 msgr2=0x7f3a240cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:21.272 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.266+0000 7f3a289fe700 1 -- 172.21.15.100:0/2040141339 shutdown_connections 2023-03-03T07:55:21.272 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.266+0000 7f3a289fe700 1 -- 172.21.15.100:0/2040141339 wait complete. 2023-03-03T07:55:21.272 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.268+0000 7f3a289fe700 1 Processor -- start 2023-03-03T07:55:21.272 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.268+0000 7f3a289fe700 1 -- start start 2023-03-03T07:55:21.274 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.269+0000 7f3a289fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a240cf570 0x7f3a241486d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:21.274 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a289fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a2415db30 0x7f3a2413e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:21.275 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a289fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a241609b0 0x7f3a240b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:21.275 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a289fe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a240b7da0 con 0x7f3a2415db30 2023-03-03T07:55:21.275 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a289fe700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2413e780 con 0x7f3a241609b0 2023-03-03T07:55:21.275 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a289fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a240cbcd0 con 0x7f3a240cf570 2023-03-03T07:55:21.275 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a22d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a240cf570 0x7f3a241486d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:21.275 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a22d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a240cf570 0x7f3a241486d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:39504/0 (socket says 172.21.15.100:39504) 2023-03-03T07:55:21.275 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a22d9d700 1 -- 172.21.15.100:0/1098836345 learned_addr learned my addr 172.21.15.100:0/1098836345 (peer_addr_for_me v2:172.21.15.100:0/0) 2023-03-03T07:55:21.275 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a2359e700 1 --2- 172.21.15.100:0/1098836345 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a241609b0 0x7f3a240b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:21.276 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a2259c700 1 --2- 172.21.15.100:0/1098836345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a2415db30 0x7f3a2413e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:21.276 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a2359e700 1 -- 172.21.15.100:0/1098836345 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a240cf570 msgr2=0x7f3a241486d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:21.276 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a2359e700 1 --2- 172.21.15.100:0/1098836345 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a240cf570 0x7f3a241486d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.276 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a2359e700 1 -- 172.21.15.100:0/1098836345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a2415db30 msgr2=0x7f3a2413e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:21.276 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a2359e700 1 --2- 172.21.15.100:0/1098836345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a2415db30 0x7f3a2413e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.276 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a2359e700 1 -- 172.21.15.100:0/1098836345 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a140094a0 con 0x7f3a241609b0 2023-03-03T07:55:21.276 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.270+0000 7f3a22d9d700 1 --2- 172.21.15.100:0/1098836345 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a240cf570 0x7f3a241486d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:21.277 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.271+0000 7f3a2359e700 1 --2- 172.21.15.100:0/1098836345 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a241609b0 0x7f3a240b2a80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3a240cf0c0 tx=0x7f3a1c004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:21.277 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.271+0000 7f3a0b7fe700 1 -- 172.21.15.100:0/1098836345 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c0099b0 con 0x7f3a241609b0 2023-03-03T07:55:21.277 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.271+0000 7f3a0b7fe700 1 -- 172.21.15.100:0/1098836345 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1c00ce80 con 0x7f3a241609b0 2023-03-03T07:55:21.277 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.271+0000 7f3a289fe700 1 -- 172.21.15.100:0/1098836345 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a1c009b30 con 0x7f3a241609b0 2023-03-03T07:55:21.277 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.271+0000 7f3a0b7fe700 1 -- 172.21.15.100:0/1098836345 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c004490 con 0x7f3a241609b0 2023-03-03T07:55:21.277 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.271+0000 7f3a289fe700 1 -- 172.21.15.100:0/1098836345 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2415cdf0 con 0x7f3a241609b0 2023-03-03T07:55:21.277 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.272+0000 7f3a0b7fe700 1 -- 172.21.15.100:0/1098836345 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f3a1c0045f0 con 0x7f3a241609b0 2023-03-03T07:55:21.277 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.272+0000 7f3a0b7fe700 1 -- 172.21.15.100:0/1098836345 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f3a1c01cc60 con 0x7f3a241609b0 2023-03-03T07:55:21.278 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.272+0000 7f3a289fe700 1 -- 172.21.15.100:0/1098836345 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a240cdd40 con 0x7f3a241609b0 2023-03-03T07:55:21.278 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.275+0000 7f3a0b7fe700 1 -- 172.21.15.100:0/1098836345 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f3a1c020030 con 0x7f3a241609b0 2023-03-03T07:55:21.281 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:21.279+0000 7fcc07721200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-03T07:55:21.364 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.361+0000 7f3a289fe700 1 -- 172.21.15.100:0/1098836345 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd new", "uuid": "77a7054d-959a-437e-9c4b-ed58e78b58b4", "id": 6} v 0) v1 -- 0x7f3a2415d0d0 con 0x7f3a241609b0 2023-03-03T07:55:21.369 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.366+0000 7f3a0b7fe700 1 -- 172.21.15.100:0/1098836345 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "77a7054d-959a-437e-9c4b-ed58e78b58b4", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3a1c01c440 con 0x7f3a241609b0 2023-03-03T07:55:21.369 INFO:teuthology.orchestra.run.smithi100.stdout:6 2023-03-03T07:55:21.370 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.367+0000 7f3a289fe700 1 -- 172.21.15.100:0/1098836345 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a241609b0 msgr2=0x7f3a240b2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:21.370 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.367+0000 7f3a289fe700 1 --2- 172.21.15.100:0/1098836345 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a241609b0 0x7f3a240b2a80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3a240cf0c0 tx=0x7f3a1c004ad0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.370 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.368+0000 7f3a289fe700 1 -- 172.21.15.100:0/1098836345 shutdown_connections 2023-03-03T07:55:21.370 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.368+0000 7f3a289fe700 1 --2- 172.21.15.100:0/1098836345 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a241609b0 0x7f3a240b2a80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.371 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.368+0000 7f3a289fe700 1 --2- 172.21.15.100:0/1098836345 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a240cf570 0x7f3a241486d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.371 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.368+0000 7f3a289fe700 1 --2- 172.21.15.100:0/1098836345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a2415db30 0x7f3a2413e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.371 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.368+0000 7f3a289fe700 1 -- 172.21.15.100:0/1098836345 >> 172.21.15.100:0/1098836345 conn(0x7f3a2406b880 msgr2=0x7f3a2415f100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:21.371 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.368+0000 7f3a289fe700 1 -- 172.21.15.100:0/1098836345 shutdown_connections 2023-03-03T07:55:21.371 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.368+0000 7f3a289fe700 1 -- 172.21.15.100:0/1098836345 wait complete. 2023-03-03T07:55:21.384 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:21.382+0000 7f8191787200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-03T07:55:21.395 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 9701bb5b-c322-4341-9dec-4b4176c1fbbb 7 2023-03-03T07:55:21.521 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:21.518+0000 7fcc07721200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-03T07:55:21.636 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.634+0000 7fdb5119d700 1 Processor -- start 2023-03-03T07:55:21.637 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.634+0000 7fdb5119d700 1 -- start start 2023-03-03T07:55:21.638 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.635+0000 7fdb5119d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb4c1609b0 0x7fdb4c160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:21.638 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.635+0000 7fdb5119d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdb4c0cf570 0x7fdb4c15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:21.638 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.636+0000 7fdb5119d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c1612c0 con 0x7fdb4c15db30 2023-03-03T07:55:21.639 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.636+0000 7fdb5119d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb4c0cebb0 con 0x7fdb4c0cf570 2023-03-03T07:55:21.639 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.636+0000 7fdb5119d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c0cd780 con 0x7fdb4c1609b0 2023-03-03T07:55:21.639 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.636+0000 7fdb4bfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb4c1609b0 0x7fdb4c160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:21.639 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.636+0000 7fdb4bfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb4c1609b0 0x7fdb4c160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:39506/0 (socket says 172.21.15.100:39506) 2023-03-03T07:55:21.639 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.636+0000 7fdb4bfff700 1 -- 172.21.15.100:0/1667483681 learned_addr learned my addr 172.21.15.100:0/1667483681 (peer_addr_for_me v2:172.21.15.100:0/0) 2023-03-03T07:55:21.640 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.636+0000 7fdb43fff700 1 --2- 172.21.15.100:0/1667483681 >> v2:172.21.15.98:3300/0 conn(0x7fdb4c0cf570 0x7fdb4c15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:21.641 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.636+0000 7fdb4bfff700 1 -- 172.21.15.100:0/1667483681 >> v2:172.21.15.98:3300/0 conn(0x7fdb4c0cf570 msgr2=0x7fdb4c15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:21.641 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.636+0000 7fdb4bfff700 1 --2- 172.21.15.100:0/1667483681 >> v2:172.21.15.98:3300/0 conn(0x7fdb4c0cf570 0x7fdb4c15d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.642 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.636+0000 7fdb4bfff700 1 -- 172.21.15.100:0/1667483681 >> v1:172.21.15.98:6789/0 conn(0x7fdb4c15db30 legacy=0x7fdb4c0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:21.642 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.636+0000 7fdb4bfff700 1 -- 172.21.15.100:0/1667483681 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c0cda60 con 0x7fdb4c1609b0 2023-03-03T07:55:21.642 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.636+0000 7fdb43fff700 1 --2- 172.21.15.100:0/1667483681 >> v2:172.21.15.98:3300/0 conn(0x7fdb4c0cf570 0x7fdb4c15d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:21.642 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.637+0000 7fdb4bfff700 1 --2- 172.21.15.100:0/1667483681 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb4c1609b0 0x7fdb4c160d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c0098f0 tx=0x7fdb3c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=add86941fe6df297 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:21.642 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.637+0000 7fdb4b7fe700 1 -- 172.21.15.100:0/1667483681 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c00b8a0 con 0x7fdb4c1609b0 2023-03-03T07:55:21.643 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.637+0000 7fdb4b7fe700 1 -- 172.21.15.100:0/1667483681 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3c00ba00 con 0x7fdb4c1609b0 2023-03-03T07:55:21.643 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.637+0000 7fdb4b7fe700 1 -- 172.21.15.100:0/1667483681 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c00bd90 con 0x7fdb4c1609b0 2023-03-03T07:55:21.643 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.637+0000 7fdb5119d700 1 -- 172.21.15.100:0/1667483681 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb4c1609b0 msgr2=0x7fdb4c160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:21.643 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.637+0000 7fdb5119d700 1 --2- 172.21.15.100:0/1667483681 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb4c1609b0 0x7fdb4c160d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c0098f0 tx=0x7fdb3c009c00 comp rx=0 tx=0).stop 2023-03-03T07:55:21.643 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.638+0000 7fdb5119d700 1 -- 172.21.15.100:0/1667483681 shutdown_connections 2023-03-03T07:55:21.644 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.638+0000 7fdb5119d700 1 --2- 172.21.15.100:0/1667483681 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb4c1609b0 0x7fdb4c160d80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.644 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.638+0000 7fdb5119d700 1 --2- 172.21.15.100:0/1667483681 >> v2:172.21.15.98:3300/0 conn(0x7fdb4c0cf570 0x7fdb4c15d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.644 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.638+0000 7fdb5119d700 1 -- 172.21.15.100:0/1667483681 >> 172.21.15.100:0/1667483681 conn(0x7fdb4c06b880 msgr2=0x7fdb4c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:21.644 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.638+0000 7fdb5119d700 1 -- 172.21.15.100:0/1667483681 shutdown_connections 2023-03-03T07:55:21.644 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.638+0000 7fdb5119d700 1 -- 172.21.15.100:0/1667483681 wait complete. 2023-03-03T07:55:21.646 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.640+0000 7fdb5119d700 1 Processor -- start 2023-03-03T07:55:21.646 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.640+0000 7fdb5119d700 1 -- start start 2023-03-03T07:55:21.646 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.641+0000 7fdb5119d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb4c0cf570 0x7fdb4c14ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:21.646 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.641+0000 7fdb5119d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb4c15db30 0x7fdb4c147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:21.647 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.641+0000 7fdb5119d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb4c1609b0 0x7fdb4c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:21.647 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.641+0000 7fdb5119d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c148020 con 0x7fdb4c1609b0 2023-03-03T07:55:21.647 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.641+0000 7fdb5119d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c0c9b70 con 0x7fdb4c0cf570 2023-03-03T07:55:21.647 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.641+0000 7fdb5119d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c0b5e50 con 0x7fdb4c15db30 2023-03-03T07:55:21.648 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.642+0000 7fdb4bfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb4c0cf570 0x7fdb4c14ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:21.648 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.642+0000 7fdb43fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb4c15db30 0x7fdb4c147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:21.648 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.642+0000 7fdb43fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb4c15db30 0x7fdb4c147ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:39510/0 (socket says 172.21.15.100:39510) 2023-03-03T07:55:21.648 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.642+0000 7fdb43fff700 1 -- 172.21.15.100:0/1605733131 learned_addr learned my addr 172.21.15.100:0/1605733131 (peer_addr_for_me v2:172.21.15.100:0/0) 2023-03-03T07:55:21.648 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.642+0000 7fdb4bfff700 1 --2- 172.21.15.100:0/1605733131 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb4c0cf570 0x7fdb4c14ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:36132/0 (socket says 172.21.15.100:36132) 2023-03-03T07:55:21.649 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.642+0000 7fdb5099c700 1 --2- 172.21.15.100:0/1605733131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb4c1609b0 0x7fdb4c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:21.649 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.642+0000 7fdb43fff700 1 -- 172.21.15.100:0/1605733131 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb4c0cf570 msgr2=0x7fdb4c14ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:21.649 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.642+0000 7fdb43fff700 1 --2- 172.21.15.100:0/1605733131 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb4c0cf570 0x7fdb4c14ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.649 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.642+0000 7fdb43fff700 1 -- 172.21.15.100:0/1605733131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb4c1609b0 msgr2=0x7fdb4c0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:21.650 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.642+0000 7fdb43fff700 1 --2- 172.21.15.100:0/1605733131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb4c1609b0 0x7fdb4c0d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.650 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.642+0000 7fdb43fff700 1 -- 172.21.15.100:0/1605733131 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c009550 con 0x7fdb4c15db30 2023-03-03T07:55:21.650 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.642+0000 7fdb4bfff700 1 --2- 172.21.15.100:0/1605733131 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb4c0cf570 0x7fdb4c14ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:21.650 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.642+0000 7fdb43fff700 1 --2- 172.21.15.100:0/1605733131 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb4c15db30 0x7fdb4c147ae0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdb3400e810 tx=0x7fdb3400eb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:21.650 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.643+0000 7fdb497fa700 1 -- 172.21.15.100:0/1605733131 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3400c070 con 0x7fdb4c15db30 2023-03-03T07:55:21.651 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.643+0000 7fdb497fa700 1 -- 172.21.15.100:0/1605733131 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb34004230 con 0x7fdb4c15db30 2023-03-03T07:55:21.651 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.643+0000 7fdb5119d700 1 -- 172.21.15.100:0/1605733131 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb4c15cdf0 con 0x7fdb4c15db30 2023-03-03T07:55:21.651 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.643+0000 7fdb497fa700 1 -- 172.21.15.100:0/1605733131 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb340036a0 con 0x7fdb4c15db30 2023-03-03T07:55:21.651 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.643+0000 7fdb5119d700 1 -- 172.21.15.100:0/1605733131 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4c0cdd40 con 0x7fdb4c15db30 2023-03-03T07:55:21.651 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.643+0000 7fdb497fa700 1 -- 172.21.15.100:0/1605733131 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fdb34019020 con 0x7fdb4c15db30 2023-03-03T07:55:21.651 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.643+0000 7fdb497fa700 1 -- 172.21.15.100:0/1605733131 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fdb34013410 con 0x7fdb4c15db30 2023-03-03T07:55:21.652 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.644+0000 7fdb5119d700 1 -- 172.21.15.100:0/1605733131 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb4c15d0d0 con 0x7fdb4c15db30 2023-03-03T07:55:21.652 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.647+0000 7fdb497fa700 1 -- 172.21.15.100:0/1605733131 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fdb3400f040 con 0x7fdb4c15db30 2023-03-03T07:55:21.741 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.737+0000 7fdb5119d700 1 -- 172.21.15.100:0/1605733131 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd new", "uuid": "9701bb5b-c322-4341-9dec-4b4176c1fbbb", "id": 7} v 0) v1 -- 0x7fdb4c068290 con 0x7fdb4c15db30 2023-03-03T07:55:21.748 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.745+0000 7fdb497fa700 1 -- 172.21.15.100:0/1605733131 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9701bb5b-c322-4341-9dec-4b4176c1fbbb", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdb34003bc0 con 0x7fdb4c15db30 2023-03-03T07:55:21.748 INFO:teuthology.orchestra.run.smithi100.stdout:7 2023-03-03T07:55:21.749 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.747+0000 7fdb5119d700 1 -- 172.21.15.100:0/1605733131 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb4c15db30 msgr2=0x7fdb4c147ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:21.750 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.747+0000 7fdb5119d700 1 --2- 172.21.15.100:0/1605733131 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb4c15db30 0x7fdb4c147ae0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdb3400e810 tx=0x7fdb3400eb20 comp rx=0 tx=0).stop 2023-03-03T07:55:21.750 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.747+0000 7fdb5119d700 1 -- 172.21.15.100:0/1605733131 shutdown_connections 2023-03-03T07:55:21.750 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.747+0000 7fdb5119d700 1 --2- 172.21.15.100:0/1605733131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb4c1609b0 0x7fdb4c0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.750 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.747+0000 7fdb5119d700 1 --2- 172.21.15.100:0/1605733131 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb4c0cf570 0x7fdb4c14ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.750 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.747+0000 7fdb5119d700 1 --2- 172.21.15.100:0/1605733131 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb4c15db30 0x7fdb4c147ae0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:21.750 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.747+0000 7fdb5119d700 1 -- 172.21.15.100:0/1605733131 >> 172.21.15.100:0/1605733131 conn(0x7fdb4c06b880 msgr2=0x7fdb4c0b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:21.750 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.747+0000 7fdb5119d700 1 -- 172.21.15.100:0/1605733131 shutdown_connections 2023-03-03T07:55:21.751 INFO:teuthology.orchestra.run.smithi100.stderr:2023-03-03T07:55:21.747+0000 7fdb5119d700 1 -- 172.21.15.100:0/1605733131 wait complete. 2023-03-03T07:55:21.789 INFO:tasks.ceph.osd.0:Restarting daemon 2023-03-03T07:55:21.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-03-03T07:55:21.795 INFO:tasks.ceph.osd.0:Started 2023-03-03T07:55:21.795 INFO:tasks.ceph.osd.1:Restarting daemon 2023-03-03T07:55:21.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-03-03T07:55:21.801 INFO:tasks.ceph.osd.1:Started 2023-03-03T07:55:21.801 INFO:tasks.ceph.osd.2:Restarting daemon 2023-03-03T07:55:21.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-03-03T07:55:21.807 INFO:tasks.ceph.osd.2:Started 2023-03-03T07:55:21.807 INFO:tasks.ceph.osd.3:Restarting daemon 2023-03-03T07:55:21.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-03-03T07:55:21.814 INFO:tasks.ceph.osd.3:Started 2023-03-03T07:55:21.815 INFO:tasks.ceph.osd.4:Restarting daemon 2023-03-03T07:55:21.815 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-03-03T07:55:21.820 INFO:tasks.ceph.osd.4:Started 2023-03-03T07:55:21.820 INFO:tasks.ceph.osd.5:Restarting daemon 2023-03-03T07:55:21.820 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-03-03T07:55:21.825 INFO:tasks.ceph.osd.5:Started 2023-03-03T07:55:21.825 INFO:tasks.ceph.osd.6:Restarting daemon 2023-03-03T07:55:21.825 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-03-03T07:55:21.832 INFO:tasks.ceph.osd.6:Started 2023-03-03T07:55:21.832 INFO:tasks.ceph.osd.7:Restarting daemon 2023-03-03T07:55:21.833 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-03-03T07:55:21.838 INFO:tasks.ceph.osd.7:Started 2023-03-03T07:55:21.839 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:55:22.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.175+0000 7f8a53fff700 1 Processor -- start 2023-03-03T07:55:22.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.175+0000 7f8a53fff700 1 -- start start 2023-03-03T07:55:22.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.176+0000 7f8a53fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8a540d0010 0x7f8a540d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:22.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.177+0000 7f8a53fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a540b4a00 0x7f8a540d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:22.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.177+0000 7f8a53fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8a540d1480 0x7f8a540b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:22.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.177+0000 7f8a53fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a540d1900 con 0x7f8a540d1480 2023-03-03T07:55:22.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.177+0000 7f8a53fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a540b0c50 con 0x7f8a540d0010 2023-03-03T07:55:22.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.177+0000 7f8a53fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540b0040 con 0x7f8a540b4a00 2023-03-03T07:55:22.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.177+0000 7f8a52ffd700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8a540d0010 0x7f8a540d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:22.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.177+0000 7f8a537fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8a540d1480 0x7f8a540b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:22.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.177+0000 7f8a527fc700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a540b4a00 0x7f8a540d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:22.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.177+0000 7f8a52ffd700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8a540d0010 0x7f8a540d03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:34228/0 (socket says 172.21.15.98:34228) 2023-03-03T07:55:22.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.177+0000 7f8a52ffd700 1 -- 172.21.15.98:0/4030411477 learned_addr learned my addr 172.21.15.98:0/4030411477 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:22.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.178+0000 7f8a537fe700 1 -- 172.21.15.98:0/4030411477 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a540b4a00 msgr2=0x7f8a540d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:22.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.178+0000 7f8a537fe700 1 --2- 172.21.15.98:0/4030411477 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a540b4a00 0x7f8a540d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.178+0000 7f8a537fe700 1 -- 172.21.15.98:0/4030411477 >> v2:172.21.15.100:3300/0 conn(0x7f8a540d0010 msgr2=0x7f8a540d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.178+0000 7f8a537fe700 1 --2- 172.21.15.98:0/4030411477 >> v2:172.21.15.100:3300/0 conn(0x7f8a540d0010 0x7f8a540d03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.178+0000 7f8a537fe700 1 -- 172.21.15.98:0/4030411477 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a540b0320 con 0x7f8a540d1480 2023-03-03T07:55:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.178+0000 7f8a52ffd700 1 --2- 172.21.15.98:0/4030411477 >> v2:172.21.15.100:3300/0 conn(0x7f8a540d0010 0x7f8a540d03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.178+0000 7f8a537fe700 1 --2- 172.21.15.98:0/4030411477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a540d1480 0x7f8a540b10e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8a5408c540 tx=0x7f8a4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d406e54ad1625b3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.178+0000 7f8a51ffb700 1 -- 172.21.15.98:0/4030411477 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c00e070 con 0x7f8a540d1480 2023-03-03T07:55:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.178+0000 7f8a51ffb700 1 -- 172.21.15.98:0/4030411477 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4c00bb20 con 0x7f8a540d1480 2023-03-03T07:55:22.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.178+0000 7f8a51ffb700 1 -- 172.21.15.98:0/4030411477 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c012430 con 0x7f8a540d1480 2023-03-03T07:55:22.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.179+0000 7f8a53fff700 1 -- 172.21.15.98:0/4030411477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a540d1480 msgr2=0x7f8a540b10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:22.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.179+0000 7f8a53fff700 1 --2- 172.21.15.98:0/4030411477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a540d1480 0x7f8a540b10e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8a5408c540 tx=0x7f8a4c00b330 comp rx=0 tx=0).stop 2023-03-03T07:55:22.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.179+0000 7f8a53fff700 1 -- 172.21.15.98:0/4030411477 shutdown_connections 2023-03-03T07:55:22.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.179+0000 7f8a53fff700 1 --2- 172.21.15.98:0/4030411477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a540d1480 0x7f8a540b10e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:22.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.179+0000 7f8a53fff700 1 --2- 172.21.15.98:0/4030411477 >> v2:172.21.15.100:3300/0 conn(0x7f8a540d0010 0x7f8a540d03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:22.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.179+0000 7f8a53fff700 1 --2- 172.21.15.98:0/4030411477 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a540b4a00 0x7f8a540d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:22.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.179+0000 7f8a53fff700 1 -- 172.21.15.98:0/4030411477 >> 172.21.15.98:0/4030411477 conn(0x7f8a54073b00 msgr2=0x7f8a540d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:22.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.179+0000 7f8a53fff700 1 -- 172.21.15.98:0/4030411477 shutdown_connections 2023-03-03T07:55:22.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.179+0000 7f8a53fff700 1 -- 172.21.15.98:0/4030411477 wait complete. 2023-03-03T07:55:22.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.182+0000 7f8a53fff700 1 Processor -- start 2023-03-03T07:55:22.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.182+0000 7f8a53fff700 1 -- start start 2023-03-03T07:55:22.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.184+0000 7f8a53fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a540b4a00 0x7f8a541f3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:22.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.184+0000 7f8a53fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a540d0010 0x7f8a541e6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:22.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.184+0000 7f8a53fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a540d1480 0x7f8a540b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:22.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.184+0000 7f8a53fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540b66e0 con 0x7f8a540b4a00 2023-03-03T07:55:22.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.184+0000 7f8a53fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540b7ac0 con 0x7f8a540d1480 2023-03-03T07:55:22.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.184+0000 7f8a53fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540b68d0 con 0x7f8a540d0010 2023-03-03T07:55:22.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.184+0000 7f8a52ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a540b4a00 0x7f8a541f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:22.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.184+0000 7f8a527fc700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a540d0010 0x7f8a541e6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:22.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.184+0000 7f8a52ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a540b4a00 0x7f8a541f3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53834/0 (socket says 172.21.15.98:53834) 2023-03-03T07:55:22.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.184+0000 7f8a52ffd700 1 -- 172.21.15.98:0/2985708340 learned_addr learned my addr 172.21.15.98:0/2985708340 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:22.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.184+0000 7f8a537fe700 1 --2- 172.21.15.98:0/2985708340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a540d1480 0x7f8a540b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:22.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.185+0000 7f8a52ffd700 1 -- 172.21.15.98:0/2985708340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a540d0010 msgr2=0x7f8a541e6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.185+0000 7f8a52ffd700 1 --2- 172.21.15.98:0/2985708340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a540d0010 0x7f8a541e6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.185+0000 7f8a52ffd700 1 -- 172.21.15.98:0/2985708340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a540d1480 msgr2=0x7f8a540b61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.185+0000 7f8a52ffd700 1 --2- 172.21.15.98:0/2985708340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a540d1480 0x7f8a540b61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.185+0000 7f8a52ffd700 1 -- 172.21.15.98:0/2985708340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a44009500 con 0x7f8a540b4a00 2023-03-03T07:55:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.185+0000 7f8a527fc700 1 --2- 172.21.15.98:0/2985708340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a540d0010 0x7f8a541e6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.185+0000 7f8a52ffd700 1 --2- 172.21.15.98:0/2985708340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a540b4a00 0x7f8a541f3920 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400ebb0 tx=0x7f8a4400eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.185+0000 7f8a3bfff700 1 -- 172.21.15.98:0/2985708340 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a44005a20 con 0x7f8a540b4a00 2023-03-03T07:55:22.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.185+0000 7f8a3bfff700 1 -- 172.21.15.98:0/2985708340 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a44005b80 con 0x7f8a540b4a00 2023-03-03T07:55:22.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.185+0000 7f8a3bfff700 1 -- 172.21.15.98:0/2985708340 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4400c850 con 0x7f8a540b4a00 2023-03-03T07:55:22.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.185+0000 7f8a53fff700 1 -- 172.21.15.98:0/2985708340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4c009a70 con 0x7f8a540b4a00 2023-03-03T07:55:22.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.186+0000 7f8a53fff700 1 -- 172.21.15.98:0/2985708340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a540b08e0 con 0x7f8a540b4a00 2023-03-03T07:55:22.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.187+0000 7f8a3bfff700 1 -- 172.21.15.98:0/2985708340 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f8a44019070 con 0x7f8a540b4a00 2023-03-03T07:55:22.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.187+0000 7f8a3bfff700 1 -- 172.21.15.98:0/2985708340 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f8a4400cca0 con 0x7f8a540b4a00 2023-03-03T07:55:22.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.187+0000 7f8a53fff700 1 -- 172.21.15.98:0/2985708340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a540b0600 con 0x7f8a540b4a00 2023-03-03T07:55:22.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.191+0000 7f8a3bfff700 1 -- 172.21.15.98:0/2985708340 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f8a44020060 con 0x7f8a540b4a00 2023-03-03T07:55:22.262 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:22.260+0000 7f8191787200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-03T07:55:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.264+0000 7f8a53fff700 1 -- 172.21.15.98:0/2985708340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a540729b0 con 0x7f8a540b4a00 2023-03-03T07:55:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.265+0000 7f8a3bfff700 1 -- 172.21.15.98:0/2985708340 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f8a44029020 con 0x7f8a540b4a00 2023-03-03T07:55:22.267 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:22.267 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":10,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:55:21.740708+0000","last_up_change":"0.000000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:55:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.266+0000 7f8a53fff700 1 -- 172.21.15.98:0/2985708340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a540b4a00 msgr2=0x7f8a541f3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.266+0000 7f8a53fff700 1 --2- 172.21.15.98:0/2985708340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a540b4a00 0x7f8a541f3920 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400ebb0 tx=0x7f8a4400eec0 comp rx=0 tx=0).stop 2023-03-03T07:55:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.266+0000 7f8a53fff700 1 -- 172.21.15.98:0/2985708340 shutdown_connections 2023-03-03T07:55:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.266+0000 7f8a53fff700 1 --2- 172.21.15.98:0/2985708340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a540b4a00 0x7f8a541f3920 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.266+0000 7f8a53fff700 1 --2- 172.21.15.98:0/2985708340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a540d1480 0x7f8a540b61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.266+0000 7f8a53fff700 1 --2- 172.21.15.98:0/2985708340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a540d0010 0x7f8a541e6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.266+0000 7f8a53fff700 1 -- 172.21.15.98:0/2985708340 >> 172.21.15.98:0/2985708340 conn(0x7f8a54073b00 msgr2=0x7f8a540b3900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.266+0000 7f8a53fff700 1 -- 172.21.15.98:0/2985708340 shutdown_connections 2023-03-03T07:55:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:22.266+0000 7f8a53fff700 1 -- 172.21.15.98:0/2985708340 wait complete. 2023-03-03T07:55:22.294 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-03-03T07:55:22.295 INFO:tasks.ceph:Waiting for OSDs to come up 2023-03-03T07:55:22.380 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:22.378+0000 7fcc07721200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-03T07:55:22.395 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:22.393+0000 7f8191787200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-03T07:55:22.510 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:22.508+0000 7fcc07721200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-03T07:55:22.603 INFO:tasks.ceph.osd.1.smithi098.stderr:2023-03-03T07:55:22.601+0000 7fadee4ce640 -1 Falling back to public interface 2023-03-03T07:55:22.608 INFO:tasks.ceph.osd.2.smithi098.stderr:2023-03-03T07:55:22.605+0000 7f38ea888640 -1 Falling back to public interface 2023-03-03T07:55:22.614 INFO:tasks.ceph.osd.7.smithi100.stderr:2023-03-03T07:55:22.612+0000 7f644b890640 -1 Falling back to public interface 2023-03-03T07:55:22.625 INFO:tasks.ceph.osd.5.smithi100.stderr:2023-03-03T07:55:22.623+0000 7f3e3b371640 -1 Falling back to public interface 2023-03-03T07:55:22.626 INFO:tasks.ceph.osd.6.smithi100.stderr:2023-03-03T07:55:22.623+0000 7f5875e65640 -1 Falling back to public interface 2023-03-03T07:55:22.632 INFO:tasks.ceph.osd.4.smithi100.stderr:2023-03-03T07:55:22.630+0000 7f6374d01640 -1 Falling back to public interface 2023-03-03T07:55:22.635 INFO:tasks.ceph.osd.3.smithi098.stderr:2023-03-03T07:55:22.633+0000 7f95e7c67640 -1 Falling back to public interface 2023-03-03T07:55:22.654 INFO:tasks.ceph.osd.0.smithi098.stderr:2023-03-03T07:55:22.651+0000 7f168d0ec640 -1 Falling back to public interface 2023-03-03T07:55:22.654 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:22.651+0000 7f8191787200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-03T07:55:22.770 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:22.767+0000 7fcc07721200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-03T07:55:23.003 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:23.001+0000 7f8191787200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-03T07:55:23.101 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-03T07:55:23.117 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:23.114+0000 7fcc07721200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-03T07:55:23.268 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:23.265+0000 7f8191787200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-03T07:55:23.335 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.332+0000 7f112759e700 1 Processor -- start 2023-03-03T07:55:23.336 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.332+0000 7f112759e700 1 -- start start 2023-03-03T07:55:23.336 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.333+0000 7f112759e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1128146360 0x7f1128160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:23.336 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.333+0000 7f112759e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11280d2590 0x7f11280b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:23.336 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.333+0000 7f112759e700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11281614c0 con 0x7f11280b4ab0 2023-03-03T07:55:23.336 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.333+0000 7f112759e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11280b5440 con 0x7f1128146360 2023-03-03T07:55:23.336 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.333+0000 7f112759e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11280d10f0 con 0x7f11280d2590 2023-03-03T07:55:23.338 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.334+0000 7f112659c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1128146360 0x7f1128160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:23.338 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.334+0000 7f112659c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1128146360 0x7f1128160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:34238/0 (socket says 172.21.15.98:34238) 2023-03-03T07:55:23.338 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.334+0000 7f112659c700 1 -- 172.21.15.98:0/1121666445 learned_addr learned my addr 172.21.15.98:0/1121666445 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:23.338 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.334+0000 7f1126d9d700 1 --2- 172.21.15.98:0/1121666445 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11280d2590 0x7f11280b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:23.338 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.334+0000 7f1126d9d700 1 -- 172.21.15.98:0/1121666445 >> v2:172.21.15.100:3300/0 conn(0x7f1128146360 msgr2=0x7f1128160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:23.339 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.334+0000 7f1126d9d700 1 --2- 172.21.15.98:0/1121666445 >> v2:172.21.15.100:3300/0 conn(0x7f1128146360 0x7f1128160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:23.339 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.334+0000 7f1126d9d700 1 -- 172.21.15.98:0/1121666445 >> v1:172.21.15.100:6789/0 conn(0x7f11280b4ab0 legacy=0x7f11280731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:23.339 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.334+0000 7f1126d9d700 1 -- 172.21.15.98:0/1121666445 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11280cdf20 con 0x7f11280d2590 2023-03-03T07:55:23.339 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.334+0000 7f112659c700 1 --2- 172.21.15.98:0/1121666445 >> v2:172.21.15.100:3300/0 conn(0x7f1128146360 0x7f1128160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:23.339 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.335+0000 7f1126d9d700 1 --2- 172.21.15.98:0/1121666445 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11280d2590 0x7f11280b4f00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1120009840 tx=0x7f1120009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae60a5a45cb5cf1c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:23.339 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.335+0000 7f1125d9b700 1 -- 172.21.15.98:0/1121666445 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1120011830 con 0x7f11280d2590 2023-03-03T07:55:23.339 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.335+0000 7f1125d9b700 1 -- 172.21.15.98:0/1121666445 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1120011990 con 0x7f11280d2590 2023-03-03T07:55:23.340 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.335+0000 7f1125d9b700 1 -- 172.21.15.98:0/1121666445 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1120011d20 con 0x7f11280d2590 2023-03-03T07:55:23.340 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.336+0000 7f112759e700 1 -- 172.21.15.98:0/1121666445 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11280d2590 msgr2=0x7f11280b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:23.340 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.336+0000 7f112759e700 1 --2- 172.21.15.98:0/1121666445 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11280d2590 0x7f11280b4f00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1120009840 tx=0x7f1120009b50 comp rx=0 tx=0).stop 2023-03-03T07:55:23.340 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.336+0000 7f112759e700 1 -- 172.21.15.98:0/1121666445 shutdown_connections 2023-03-03T07:55:23.340 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.336+0000 7f112759e700 1 --2- 172.21.15.98:0/1121666445 >> v2:172.21.15.100:3300/0 conn(0x7f1128146360 0x7f1128160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:23.340 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.336+0000 7f112759e700 1 --2- 172.21.15.98:0/1121666445 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11280d2590 0x7f11280b4f00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:23.340 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.336+0000 7f112759e700 1 -- 172.21.15.98:0/1121666445 >> 172.21.15.98:0/1121666445 conn(0x7f112806b8e0 msgr2=0x7f11280cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:23.340 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.336+0000 7f112759e700 1 -- 172.21.15.98:0/1121666445 shutdown_connections 2023-03-03T07:55:23.341 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.336+0000 7f112759e700 1 -- 172.21.15.98:0/1121666445 wait complete. 2023-03-03T07:55:23.341 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.339+0000 7f112759e700 1 Processor -- start 2023-03-03T07:55:23.341 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.339+0000 7f112759e700 1 -- start start 2023-03-03T07:55:23.343 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.340+0000 7f112759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11280b4ab0 0x7f1128157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:23.343 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.340+0000 7f112759e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11280d2590 0x7f11280c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:23.343 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.340+0000 7f112759e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1128146360 0x7f11280c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:23.343 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.340+0000 7f112759e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11280c78a0 con 0x7f11280b4ab0 2023-03-03T07:55:23.343 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.340+0000 7f112759e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11280c6c90 con 0x7f1128146360 2023-03-03T07:55:23.343 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.340+0000 7f112759e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11280c6f70 con 0x7f11280d2590 2023-03-03T07:55:23.344 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.341+0000 7f112659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11280b4ab0 0x7f1128157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:23.344 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.341+0000 7f112659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11280b4ab0 0x7f1128157ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53848/0 (socket says 172.21.15.98:53848) 2023-03-03T07:55:23.344 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.341+0000 7f112659c700 1 -- 172.21.15.98:0/4275871893 learned_addr learned my addr 172.21.15.98:0/4275871893 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:23.344 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.341+0000 7f1126d9d700 1 --2- 172.21.15.98:0/4275871893 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1128146360 0x7f11280c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:23.344 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.341+0000 7f111ffff700 1 --2- 172.21.15.98:0/4275871893 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11280d2590 0x7f11280c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:23.345 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.341+0000 7f112659c700 1 -- 172.21.15.98:0/4275871893 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11280d2590 msgr2=0x7f11280c7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:23.345 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.341+0000 7f112659c700 1 --2- 172.21.15.98:0/4275871893 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11280d2590 0x7f11280c7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:23.346 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.341+0000 7f112659c700 1 -- 172.21.15.98:0/4275871893 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1128146360 msgr2=0x7f11280c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:23.346 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.341+0000 7f112659c700 1 --2- 172.21.15.98:0/4275871893 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1128146360 0x7f11280c8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:23.346 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.341+0000 7f112659c700 1 -- 172.21.15.98:0/4275871893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1118009b30 con 0x7f11280b4ab0 2023-03-03T07:55:23.346 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.341+0000 7f112659c700 1 --2- 172.21.15.98:0/4275871893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11280b4ab0 0x7f1128157ab0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f111800efc0 tx=0x7f111800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:23.346 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.342+0000 7f111effd700 1 -- 172.21.15.98:0/4275871893 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111800cbb0 con 0x7f11280b4ab0 2023-03-03T07:55:23.346 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.342+0000 7f111effd700 1 -- 172.21.15.98:0/4275871893 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111800f040 con 0x7f11280b4ab0 2023-03-03T07:55:23.346 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.342+0000 7f112759e700 1 -- 172.21.15.98:0/4275871893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11200094a0 con 0x7f11280b4ab0 2023-03-03T07:55:23.347 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.342+0000 7f111effd700 1 -- 172.21.15.98:0/4275871893 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1118014480 con 0x7f11280b4ab0 2023-03-03T07:55:23.347 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.342+0000 7f112759e700 1 -- 172.21.15.98:0/4275871893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11280cd960 con 0x7f11280b4ab0 2023-03-03T07:55:23.347 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.342+0000 7f111effd700 1 -- 172.21.15.98:0/4275871893 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f11180145e0 con 0x7f11280b4ab0 2023-03-03T07:55:23.347 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.342+0000 7f111effd700 1 -- 172.21.15.98:0/4275871893 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f11180073f0 con 0x7f11280b4ab0 2023-03-03T07:55:23.347 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.343+0000 7f112759e700 1 -- 172.21.15.98:0/4275871893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11280cdc40 con 0x7f11280b4ab0 2023-03-03T07:55:23.350 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.348+0000 7f111effd700 1 -- 172.21.15.98:0/4275871893 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f1118003a20 con 0x7f11280b4ab0 2023-03-03T07:55:23.367 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:23.365+0000 7fcc07721200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-03T07:55:23.430 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:23.428+0000 7f8191787200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-03T07:55:23.444 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.442+0000 7f112759e700 1 -- 172.21.15.98:0/4275871893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1128068300 con 0x7f11280b4ab0 2023-03-03T07:55:23.445 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.443+0000 7f111effd700 1 -- 172.21.15.98:0/4275871893 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f1118003c70 con 0x7f11280b4ab0 2023-03-03T07:55:23.446 INFO:teuthology.misc.health.smithi098.stdout: 2023-03-03T07:55:23.446 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:55:21.740708+0000","last_up_change":"0.000000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:55:23.447 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.444+0000 7f112759e700 1 -- 172.21.15.98:0/4275871893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11280b4ab0 msgr2=0x7f1128157ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:23.447 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.444+0000 7f112759e700 1 --2- 172.21.15.98:0/4275871893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11280b4ab0 0x7f1128157ab0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f111800efc0 tx=0x7f111800c3a0 comp rx=0 tx=0).stop 2023-03-03T07:55:23.447 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.444+0000 7f112759e700 1 -- 172.21.15.98:0/4275871893 shutdown_connections 2023-03-03T07:55:23.448 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.444+0000 7f112759e700 1 --2- 172.21.15.98:0/4275871893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11280b4ab0 0x7f1128157ab0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:23.448 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.444+0000 7f112759e700 1 --2- 172.21.15.98:0/4275871893 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1128146360 0x7f11280c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:23.448 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.444+0000 7f112759e700 1 --2- 172.21.15.98:0/4275871893 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11280d2590 0x7f11280c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:23.448 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.444+0000 7f112759e700 1 -- 172.21.15.98:0/4275871893 >> 172.21.15.98:0/4275871893 conn(0x7f112806b8e0 msgr2=0x7f11280731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:23.448 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.444+0000 7f112759e700 1 -- 172.21.15.98:0/4275871893 shutdown_connections 2023-03-03T07:55:23.448 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:23.444+0000 7f112759e700 1 -- 172.21.15.98:0/4275871893 wait complete. 2023-03-03T07:55:23.468 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-03-03T07:55:23.509 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:23.506+0000 7fcc07721200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-03T07:55:23.578 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:23.576+0000 7f8191787200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-03T07:55:23.649 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:23.647+0000 7fcc07721200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-03T07:55:23.718 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:23.716+0000 7f8191787200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-03T07:55:23.791 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:23.789+0000 7fcc07721200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-03T07:55:24.862 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:24.860+0000 7f8191787200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-03T07:55:24.930 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:24.928+0000 7fcc07721200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-03T07:55:25.008 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:25.006+0000 7f8191787200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-03T07:55:25.077 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:25.074+0000 7fcc07721200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-03T07:55:25.156 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:25.153+0000 7f8191787200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-03T07:55:25.227 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:25.226+0000 7fcc07721200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-03T07:55:25.320 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:25.317+0000 7f8191787200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-03T07:55:25.399 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:25.398+0000 7fcc07721200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-03T07:55:25.674 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:25.672+0000 7f8191787200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-03T07:55:25.777 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:25.775+0000 7fcc07721200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-03T07:55:25.848 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:25.845+0000 7f8191787200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-03T07:55:25.927 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:25.925+0000 7fcc07721200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-03T07:55:26.149 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:26.147+0000 7f8191787200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-03T07:55:26.226 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T07:55:26.224+0000 7fcc07721200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-03T07:55:26.521 INFO:tasks.ceph.osd.1.smithi098.stderr:2023-03-03T07:55:26.518+0000 7fadee4ce640 -1 osd.1 0 log_to_monitors true 2023-03-03T07:55:26.548 INFO:tasks.ceph.osd.2.smithi098.stderr:2023-03-03T07:55:26.545+0000 7f38ea888640 -1 osd.2 0 log_to_monitors true 2023-03-03T07:55:26.553 INFO:tasks.ceph.osd.3.smithi098.stderr:2023-03-03T07:55:26.551+0000 7f95e7c67640 -1 osd.3 0 log_to_monitors true 2023-03-03T07:55:26.554 INFO:tasks.ceph.osd.0.smithi098.stderr:2023-03-03T07:55:26.552+0000 7f168d0ec640 -1 osd.0 0 log_to_monitors true 2023-03-03T07:55:26.611 INFO:tasks.ceph.osd.4.smithi100.stderr:2023-03-03T07:55:26.610+0000 7f6374d01640 -1 osd.4 0 log_to_monitors true 2023-03-03T07:55:26.616 INFO:tasks.ceph.osd.7.smithi100.stderr:2023-03-03T07:55:26.614+0000 7f644b890640 -1 osd.7 0 log_to_monitors true 2023-03-03T07:55:26.625 INFO:tasks.ceph.osd.6.smithi100.stderr:2023-03-03T07:55:26.624+0000 7f5875e65640 -1 osd.6 0 log_to_monitors true 2023-03-03T07:55:26.626 INFO:tasks.ceph.osd.5.smithi100.stderr:2023-03-03T07:55:26.625+0000 7f3e3b371640 -1 osd.5 0 log_to_monitors true 2023-03-03T07:55:27.174 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:27.172+0000 7f81463fa700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-03-03T07:55:27.175 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:55:27.173+0000 7f81463fa700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-03-03T07:55:30.276 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-03T07:55:30.406 INFO:tasks.ceph.osd.2.smithi098.stderr:2023-03-03T07:55:30.400+0000 7f38d8ff6700 -1 osd.2 0 waiting for initial osdmap 2023-03-03T07:55:30.414 INFO:tasks.ceph.osd.1.smithi098.stderr:2023-03-03T07:55:30.413+0000 7faddcbf6700 -1 osd.1 0 waiting for initial osdmap 2023-03-03T07:55:30.425 INFO:tasks.ceph.osd.2.smithi098.stderr:2023-03-03T07:55:30.423+0000 7f38d0ffb700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T07:55:30.432 INFO:tasks.ceph.osd.0.smithi098.stderr:2023-03-03T07:55:30.430+0000 7f167b7f6700 -1 osd.0 0 waiting for initial osdmap 2023-03-03T07:55:30.440 INFO:tasks.ceph.osd.1.smithi098.stderr:2023-03-03T07:55:30.439+0000 7fadd4bfb700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T07:55:30.452 INFO:tasks.ceph.osd.0.smithi098.stderr:2023-03-03T07:55:30.451+0000 7f16737fb700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T07:55:30.470 INFO:tasks.ceph.osd.6.smithi100.stderr:2023-03-03T07:55:30.468+0000 7f58645f6700 -1 osd.6 0 waiting for initial osdmap 2023-03-03T07:55:30.480 INFO:tasks.ceph.osd.7.smithi100.stderr:2023-03-03T07:55:30.478+0000 7f643b7f9700 -1 osd.7 0 waiting for initial osdmap 2023-03-03T07:55:30.487 INFO:tasks.ceph.osd.6.smithi100.stderr:2023-03-03T07:55:30.486+0000 7f585c5fb700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T07:55:30.499 INFO:tasks.ceph.osd.7.smithi100.stderr:2023-03-03T07:55:30.497+0000 7f6431ffb700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T07:55:30.515 INFO:tasks.ceph.osd.4.smithi100.stderr:2023-03-03T07:55:30.514+0000 7f63633f6700 -1 osd.4 0 waiting for initial osdmap 2023-03-03T07:55:30.519 INFO:tasks.ceph.osd.5.smithi100.stderr:2023-03-03T07:55:30.518+0000 7f3e299f6700 -1 osd.5 0 waiting for initial osdmap 2023-03-03T07:55:30.531 INFO:tasks.ceph.osd.4.smithi100.stderr:2023-03-03T07:55:30.529+0000 7f635b3fb700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T07:55:30.533 INFO:tasks.ceph.osd.5.smithi100.stderr:2023-03-03T07:55:30.532+0000 7f3e219fb700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T07:55:30.592 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.591+0000 7fe245d9d700 1 Processor -- start 2023-03-03T07:55:30.593 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.591+0000 7fe245d9d700 1 -- start start 2023-03-03T07:55:30.594 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.593+0000 7fe245d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2400d0010 0x7fe2400d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:30.595 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.593+0000 7fe245d9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe2400b4a00 0x7fe2400d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:30.595 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.593+0000 7fe245d9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2400b0c50 con 0x7fe2400d1480 2023-03-03T07:55:30.595 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.593+0000 7fe245d9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2400d1900 con 0x7fe2400b4a00 2023-03-03T07:55:30.595 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.593+0000 7fe245d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2400b0040 con 0x7fe2400d0010 2023-03-03T07:55:30.596 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.594+0000 7fe244d9b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2400d0010 0x7fe2400d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:30.597 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.594+0000 7fe244d9b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2400d0010 0x7fe2400d03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:49884/0 (socket says 172.21.15.98:49884) 2023-03-03T07:55:30.597 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.594+0000 7fe244d9b700 1 -- 172.21.15.98:0/1898770314 learned_addr learned my addr 172.21.15.98:0/1898770314 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:30.597 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.594+0000 7fe23ffff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe2400b4a00 0x7fe2400d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:30.597 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.594+0000 7fe244d9b700 1 -- 172.21.15.98:0/1898770314 >> v2:172.21.15.100:3300/0 conn(0x7fe2400b4a00 msgr2=0x7fe2400d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:30.597 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.594+0000 7fe244d9b700 1 --2- 172.21.15.98:0/1898770314 >> v2:172.21.15.100:3300/0 conn(0x7fe2400b4a00 0x7fe2400d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:30.597 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.594+0000 7fe244d9b700 1 -- 172.21.15.98:0/1898770314 >> v1:172.21.15.98:6789/0 conn(0x7fe2400d1480 legacy=0x7fe2400b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:30.597 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.594+0000 7fe244d9b700 1 -- 172.21.15.98:0/1898770314 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2400b0320 con 0x7fe2400d0010 2023-03-03T07:55:30.598 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.595+0000 7fe244d9b700 1 --2- 172.21.15.98:0/1898770314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2400d0010 0x7fe2400d03e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe234006fd0 tx=0x7fe234009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7454ff671f81087 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:30.598 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.595+0000 7fe23f7fe700 1 -- 172.21.15.98:0/1898770314 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe234005e30 con 0x7fe2400d0010 2023-03-03T07:55:30.598 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.595+0000 7fe23f7fe700 1 -- 172.21.15.98:0/1898770314 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe234004030 con 0x7fe2400d0010 2023-03-03T07:55:30.598 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.595+0000 7fe23f7fe700 1 -- 172.21.15.98:0/1898770314 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2340043e0 con 0x7fe2400d0010 2023-03-03T07:55:30.599 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.596+0000 7fe245d9d700 1 -- 172.21.15.98:0/1898770314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2400d0010 msgr2=0x7fe2400d03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:30.599 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.596+0000 7fe245d9d700 1 --2- 172.21.15.98:0/1898770314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2400d0010 0x7fe2400d03e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe234006fd0 tx=0x7fe234009bd0 comp rx=0 tx=0).stop 2023-03-03T07:55:30.599 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.596+0000 7fe245d9d700 1 -- 172.21.15.98:0/1898770314 shutdown_connections 2023-03-03T07:55:30.599 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.596+0000 7fe245d9d700 1 --2- 172.21.15.98:0/1898770314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2400d0010 0x7fe2400d03e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:30.599 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.596+0000 7fe245d9d700 1 --2- 172.21.15.98:0/1898770314 >> v2:172.21.15.100:3300/0 conn(0x7fe2400b4a00 0x7fe2400d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:30.599 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.596+0000 7fe245d9d700 1 -- 172.21.15.98:0/1898770314 >> 172.21.15.98:0/1898770314 conn(0x7fe240073b00 msgr2=0x7fe2400d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:30.600 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.596+0000 7fe245d9d700 1 -- 172.21.15.98:0/1898770314 shutdown_connections 2023-03-03T07:55:30.600 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.596+0000 7fe245d9d700 1 -- 172.21.15.98:0/1898770314 wait complete. 2023-03-03T07:55:30.600 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.598+0000 7fe245d9d700 1 Processor -- start 2023-03-03T07:55:30.601 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.598+0000 7fe245d9d700 1 -- start start 2023-03-03T07:55:30.601 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.599+0000 7fe245d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2400b4a00 0x7fe2400b1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:30.601 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.599+0000 7fe245d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2400d0010 0x7fe2400b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:30.601 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.599+0000 7fe245d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2400d1480 0x7fe2401f30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:30.601 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.599+0000 7fe245d9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2401e5190 con 0x7fe2400b4a00 2023-03-03T07:55:30.602 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.599+0000 7fe245d9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2400b2c10 con 0x7fe2400d1480 2023-03-03T07:55:30.602 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.599+0000 7fe245d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2401e45a0 con 0x7fe2400d0010 2023-03-03T07:55:30.603 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.600+0000 7fe244d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2400b4a00 0x7fe2400b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:30.603 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.600+0000 7fe244d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2400b4a00 0x7fe2400b1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44692/0 (socket says 172.21.15.98:44692) 2023-03-03T07:55:30.603 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.600+0000 7fe244d9b700 1 -- 172.21.15.98:0/656829142 learned_addr learned my addr 172.21.15.98:0/656829142 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:30.604 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.600+0000 7fe23ffff700 1 --2- 172.21.15.98:0/656829142 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2400d0010 0x7fe2400b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:30.604 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.600+0000 7fe24559c700 1 --2- 172.21.15.98:0/656829142 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2400d1480 0x7fe2401f30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:30.604 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.600+0000 7fe244d9b700 1 -- 172.21.15.98:0/656829142 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2400d0010 msgr2=0x7fe2400b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:30.604 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.600+0000 7fe244d9b700 1 --2- 172.21.15.98:0/656829142 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2400d0010 0x7fe2400b23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:30.604 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.600+0000 7fe244d9b700 1 -- 172.21.15.98:0/656829142 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2400d1480 msgr2=0x7fe2401f30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:30.604 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.600+0000 7fe244d9b700 1 --2- 172.21.15.98:0/656829142 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2400d1480 0x7fe2401f30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:30.604 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.600+0000 7fe244d9b700 1 -- 172.21.15.98:0/656829142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe234009500 con 0x7fe2400b4a00 2023-03-03T07:55:30.604 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.600+0000 7fe24559c700 1 --2- 172.21.15.98:0/656829142 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2400d1480 0x7fe2401f30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:30.605 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.601+0000 7fe244d9b700 1 --2- 172.21.15.98:0/656829142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2400b4a00 0x7fe2400b1410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe234009c10 tx=0x7fe234004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:30.605 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.601+0000 7fe23d7fa700 1 -- 172.21.15.98:0/656829142 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe234003c20 con 0x7fe2400b4a00 2023-03-03T07:55:30.605 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.601+0000 7fe23d7fa700 1 -- 172.21.15.98:0/656829142 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe234003d80 con 0x7fe2400b4a00 2023-03-03T07:55:30.605 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.601+0000 7fe23d7fa700 1 -- 172.21.15.98:0/656829142 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe234011660 con 0x7fe2400b4a00 2023-03-03T07:55:30.605 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.601+0000 7fe245d9d700 1 -- 172.21.15.98:0/656829142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2400b08e0 con 0x7fe2400b4a00 2023-03-03T07:55:30.605 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.601+0000 7fe245d9d700 1 -- 172.21.15.98:0/656829142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2400b0600 con 0x7fe2400b4a00 2023-03-03T07:55:30.606 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.603+0000 7fe245d9d700 1 -- 172.21.15.98:0/656829142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2400d0b30 con 0x7fe2400b4a00 2023-03-03T07:55:30.607 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.603+0000 7fe23d7fa700 1 -- 172.21.15.98:0/656829142 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7fe23402d070 con 0x7fe2400b4a00 2023-03-03T07:55:30.608 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.606+0000 7fe23d7fa700 1 --2- 172.21.15.98:0/656829142 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe2240682a0 0x7fe22406a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:30.608 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.606+0000 7fe23d7fa700 1 -- 172.21.15.98:0/656829142 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fe234086120 con 0x7fe2400b4a00 2023-03-03T07:55:30.608 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.607+0000 7fe23ffff700 1 --2- 172.21.15.98:0/656829142 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe2240682a0 0x7fe22406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:30.609 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.608+0000 7fe23ffff700 1 --2- 172.21.15.98:0/656829142 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe2240682a0 0x7fe22406a750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe2400b1950 tx=0x7fe22c009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:30.612 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.611+0000 7fe23d7fa700 1 -- 172.21.15.98:0/656829142 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe2340537d0 con 0x7fe2400b4a00 2023-03-03T07:55:30.869 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.868+0000 7fe245d9d700 1 -- 172.21.15.98:0/656829142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2400729b0 con 0x7fe2400b4a00 2023-03-03T07:55:30.870 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.869+0000 7fe23d7fa700 1 -- 172.21.15.98:0/656829142 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fe234056df0 con 0x7fe2400b4a00 2023-03-03T07:55:30.870 INFO:teuthology.misc.health.smithi098.stdout: 2023-03-03T07:55:30.871 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":12,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:55:28.240514+0000","last_up_change":"0.000000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:55:30.874 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.873+0000 7fe245d9d700 1 -- 172.21.15.98:0/656829142 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe2240682a0 msgr2=0x7fe22406a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:30.874 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.873+0000 7fe245d9d700 1 --2- 172.21.15.98:0/656829142 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe2240682a0 0x7fe22406a750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe2400b1950 tx=0x7fe22c009210 comp rx=0 tx=0).stop 2023-03-03T07:55:30.874 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.873+0000 7fe245d9d700 1 -- 172.21.15.98:0/656829142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2400b4a00 msgr2=0x7fe2400b1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:30.875 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.873+0000 7fe245d9d700 1 --2- 172.21.15.98:0/656829142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2400b4a00 0x7fe2400b1410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe234009c10 tx=0x7fe234004750 comp rx=0 tx=0).stop 2023-03-03T07:55:30.875 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.873+0000 7fe245d9d700 1 -- 172.21.15.98:0/656829142 shutdown_connections 2023-03-03T07:55:30.875 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.873+0000 7fe245d9d700 1 --2- 172.21.15.98:0/656829142 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe2240682a0 0x7fe22406a750 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:30.875 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.873+0000 7fe245d9d700 1 --2- 172.21.15.98:0/656829142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2400b4a00 0x7fe2400b1410 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:30.875 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.873+0000 7fe245d9d700 1 --2- 172.21.15.98:0/656829142 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2400d1480 0x7fe2401f30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:30.875 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.873+0000 7fe245d9d700 1 --2- 172.21.15.98:0/656829142 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2400d0010 0x7fe2400b23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:30.875 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.873+0000 7fe245d9d700 1 -- 172.21.15.98:0/656829142 >> 172.21.15.98:0/656829142 conn(0x7fe240073b00 msgr2=0x7fe2400c91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:30.876 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.873+0000 7fe245d9d700 1 -- 172.21.15.98:0/656829142 shutdown_connections 2023-03-03T07:55:30.876 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:30.873+0000 7fe245d9d700 1 -- 172.21.15.98:0/656829142 wait complete. 2023-03-03T07:55:30.898 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-03-03T07:55:30.933 INFO:tasks.ceph.osd.3.smithi098.stderr:2023-03-03T07:55:30.932+0000 7f95d63f6700 -1 osd.3 0 waiting for initial osdmap 2023-03-03T07:55:30.950 INFO:tasks.ceph.osd.3.smithi098.stderr:2023-03-03T07:55:30.948+0000 7f95ce3fb700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T07:55:31.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:55:31.052 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:55:37.705 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-03T07:55:37.925 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.924+0000 7fa96259d700 1 Processor -- start 2023-03-03T07:55:37.926 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.925+0000 7fa96259d700 1 -- start start 2023-03-03T07:55:37.930 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.926+0000 7fa96259d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa95c0b51c0 0x7fa95c160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:37.931 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.926+0000 7fa96259d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95c1614c0 con 0x7fa95c0d1580 2023-03-03T07:55:37.931 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.926+0000 7fa96259d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95c0d10f0 con 0x7fa95c146360 2023-03-03T07:55:37.931 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.926+0000 7fa96259d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa95c0cdf20 con 0x7fa95c0b51c0 2023-03-03T07:55:37.931 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.926+0000 7fa960d9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa95c0b51c0 0x7fa95c160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:37.932 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.926+0000 7fa961d9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa95c0d1580 0x7fa95c0d1980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46952/0 (socket says 172.21.15.98:46952) 2023-03-03T07:55:37.932 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.926+0000 7fa961d9c700 1 -- 172.21.15.98:0/2212799350 learned_addr learned my addr 172.21.15.98:0/2212799350 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:37.932 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.926+0000 7fa960d9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa95c0b51c0 0x7fa95c160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47594/0 (socket says 172.21.15.98:47594) 2023-03-03T07:55:37.932 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.927+0000 7fa960d9a700 1 -- 172.21.15.98:0/2212799350 >> v1:172.21.15.100:6789/0 conn(0x7fa95c146360 legacy=0x7fa95c0b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:37.933 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.927+0000 7fa960d9a700 1 -- 172.21.15.98:0/2212799350 >> v1:172.21.15.98:6789/0 conn(0x7fa95c0d1580 legacy=0x7fa95c0d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:37.933 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.927+0000 7fa960d9a700 1 -- 172.21.15.98:0/2212799350 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c0cd680 con 0x7fa95c0b51c0 2023-03-03T07:55:37.933 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.927+0000 7fa953fff700 1 -- 172.21.15.98:0/2212799350 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180981719 0 0) 0x7fa954001490 con 0x7fa95c0d1580 2023-03-03T07:55:37.933 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.927+0000 7fa960d9a700 1 --2- 172.21.15.98:0/2212799350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa95c0b51c0 0x7fa95c160f80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa94c009840 tx=0x7fa94c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90509b8576e4a4aa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:37.933 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.927+0000 7fa953fff700 1 -- 172.21.15.98:0/2212799350 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c011830 con 0x7fa95c0b51c0 2023-03-03T07:55:37.933 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.928+0000 7fa953fff700 1 -- 172.21.15.98:0/2212799350 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94c011990 con 0x7fa95c0b51c0 2023-03-03T07:55:37.934 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.928+0000 7fa953fff700 1 -- 172.21.15.98:0/2212799350 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c011d20 con 0x7fa95c0b51c0 2023-03-03T07:55:37.934 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.928+0000 7fa96259d700 1 -- 172.21.15.98:0/2212799350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa95c0b51c0 msgr2=0x7fa95c160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:37.934 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.928+0000 7fa96259d700 1 --2- 172.21.15.98:0/2212799350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa95c0b51c0 0x7fa95c160f80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa94c009840 tx=0x7fa94c009b50 comp rx=0 tx=0).stop 2023-03-03T07:55:37.934 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.928+0000 7fa96259d700 1 -- 172.21.15.98:0/2212799350 shutdown_connections 2023-03-03T07:55:37.934 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.928+0000 7fa96259d700 1 --2- 172.21.15.98:0/2212799350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa95c0b51c0 0x7fa95c160f80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:37.934 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.928+0000 7fa96259d700 1 -- 172.21.15.98:0/2212799350 >> 172.21.15.98:0/2212799350 conn(0x7fa95c06b8e0 msgr2=0x7fa95c0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:37.934 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.928+0000 7fa96259d700 1 -- 172.21.15.98:0/2212799350 shutdown_connections 2023-03-03T07:55:37.935 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.928+0000 7fa96259d700 1 -- 172.21.15.98:0/2212799350 wait complete. 2023-03-03T07:55:37.937 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.930+0000 7fa96259d700 1 Processor -- start 2023-03-03T07:55:37.938 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.930+0000 7fa96259d700 1 -- start start 2023-03-03T07:55:37.938 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.931+0000 7fa96259d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa95c0b51c0 0x7fa95c0d21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:37.938 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.931+0000 7fa96259d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa95c0d1580 0x7fa95c0b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:37.938 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.931+0000 7fa96259d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa95c146360 0x7fa95c0c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:37.938 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.931+0000 7fa96259d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0cc8d0 con 0x7fa95c0b51c0 2023-03-03T07:55:37.939 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.931+0000 7fa96259d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0cb410 con 0x7fa95c0d1580 2023-03-03T07:55:37.939 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.931+0000 7fa96259d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0ccb70 con 0x7fa95c146360 2023-03-03T07:55:37.939 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.932+0000 7fa96159b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa95c0b51c0 0x7fa95c0d21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:37.939 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.932+0000 7fa96159b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa95c0b51c0 0x7fa95c0d21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47596/0 (socket says 172.21.15.98:47596) 2023-03-03T07:55:37.940 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.932+0000 7fa96159b700 1 -- 172.21.15.98:0/2644058301 learned_addr learned my addr 172.21.15.98:0/2644058301 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:37.940 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.932+0000 7fa960d9a700 1 --2- 172.21.15.98:0/2644058301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa95c0d1580 0x7fa95c0b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:37.940 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.932+0000 7fa961d9c700 1 --2- 172.21.15.98:0/2644058301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa95c146360 0x7fa95c0c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:37.940 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.932+0000 7fa96159b700 1 -- 172.21.15.98:0/2644058301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa95c146360 msgr2=0x7fa95c0c5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:37.940 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.932+0000 7fa96159b700 1 --2- 172.21.15.98:0/2644058301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa95c146360 0x7fa95c0c5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:37.940 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.932+0000 7fa96159b700 1 -- 172.21.15.98:0/2644058301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa95c0d1580 msgr2=0x7fa95c0b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:37.940 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.932+0000 7fa96159b700 1 --2- 172.21.15.98:0/2644058301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa95c0d1580 0x7fa95c0b28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:37.941 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.932+0000 7fa96159b700 1 -- 172.21.15.98:0/2644058301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94c0094a0 con 0x7fa95c0b51c0 2023-03-03T07:55:37.941 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.933+0000 7fa96159b700 1 --2- 172.21.15.98:0/2644058301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa95c0b51c0 0x7fa95c0d21f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa95800c4e0 tx=0x7fa95800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:37.941 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.933+0000 7fa951ffb700 1 -- 172.21.15.98:0/2644058301 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9580052d0 con 0x7fa95c0b51c0 2023-03-03T07:55:37.941 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.933+0000 7fa951ffb700 1 -- 172.21.15.98:0/2644058301 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa958005430 con 0x7fa95c0b51c0 2023-03-03T07:55:37.941 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.933+0000 7fa96259d700 1 -- 172.21.15.98:0/2644058301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95c0cdc40 con 0x7fa95c0b51c0 2023-03-03T07:55:37.941 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.933+0000 7fa951ffb700 1 -- 172.21.15.98:0/2644058301 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9580038c0 con 0x7fa95c0b51c0 2023-03-03T07:55:37.942 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.933+0000 7fa96259d700 1 -- 172.21.15.98:0/2644058301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95c0cd960 con 0x7fa95c0b51c0 2023-03-03T07:55:37.942 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.934+0000 7fa96259d700 1 -- 172.21.15.98:0/2644058301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa95c0d0d80 con 0x7fa95c0b51c0 2023-03-03T07:55:37.942 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.935+0000 7fa951ffb700 1 -- 172.21.15.98:0/2644058301 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa958019030 con 0x7fa95c0b51c0 2023-03-03T07:55:37.942 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.937+0000 7fa951ffb700 1 --2- 172.21.15.98:0/2644058301 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9400682f0 0x7fa94006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:37.942 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.937+0000 7fa951ffb700 1 -- 172.21.15.98:0/2644058301 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa958084c00 con 0x7fa95c0b51c0 2023-03-03T07:55:37.943 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.937+0000 7fa960d9a700 1 --2- 172.21.15.98:0/2644058301 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9400682f0 0x7fa94006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:37.943 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.938+0000 7fa960d9a700 1 --2- 172.21.15.98:0/2644058301 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9400682f0 0x7fa94006a7a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa95c0b55b0 tx=0x7fa94c01c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:37.943 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:37.942+0000 7fa951ffb700 1 -- 172.21.15.98:0/2644058301 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa958051820 con 0x7fa95c0b51c0 2023-03-03T07:55:38.138 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:38.137+0000 7fa96259d700 1 -- 172.21.15.98:0/2644058301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa95c068300 con 0x7fa95c0b51c0 2023-03-03T07:55:38.139 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:38.138+0000 7fa951ffb700 1 -- 172.21.15.98:0/2644058301 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12478 (secure 0 0 0) 0x7fa958054e40 con 0x7fa95c0b51c0 2023-03-03T07:55:38.139 INFO:teuthology.misc.health.smithi098.stdout: 2023-03-03T07:55:38.139 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":16,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:55:34.454192+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:55:38.142 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:38.141+0000 7fa96259d700 1 -- 172.21.15.98:0/2644058301 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9400682f0 msgr2=0x7fa94006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:38.142 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:38.141+0000 7fa96259d700 1 --2- 172.21.15.98:0/2644058301 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9400682f0 0x7fa94006a7a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa95c0b55b0 tx=0x7fa94c01c040 comp rx=0 tx=0).stop 2023-03-03T07:55:38.142 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:38.141+0000 7fa96259d700 1 -- 172.21.15.98:0/2644058301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa95c0b51c0 msgr2=0x7fa95c0d21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:38.143 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:38.141+0000 7fa96259d700 1 --2- 172.21.15.98:0/2644058301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa95c0b51c0 0x7fa95c0d21f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa95800c4e0 tx=0x7fa95800c7f0 comp rx=0 tx=0).stop 2023-03-03T07:55:38.143 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:38.141+0000 7fa96259d700 1 -- 172.21.15.98:0/2644058301 shutdown_connections 2023-03-03T07:55:38.143 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:38.141+0000 7fa96259d700 1 --2- 172.21.15.98:0/2644058301 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9400682f0 0x7fa94006a7a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:38.143 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:38.141+0000 7fa96259d700 1 --2- 172.21.15.98:0/2644058301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa95c0b51c0 0x7fa95c0d21f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:38.143 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:38.141+0000 7fa96259d700 1 --2- 172.21.15.98:0/2644058301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa95c146360 0x7fa95c0c5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:38.143 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:38.141+0000 7fa96259d700 1 --2- 172.21.15.98:0/2644058301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa95c0d1580 0x7fa95c0b28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:38.143 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:38.141+0000 7fa96259d700 1 -- 172.21.15.98:0/2644058301 >> 172.21.15.98:0/2644058301 conn(0x7fa95c06b8e0 msgr2=0x7fa95c0b21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:38.144 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:38.141+0000 7fa96259d700 1 -- 172.21.15.98:0/2644058301 shutdown_connections 2023-03-03T07:55:38.144 INFO:teuthology.misc.health.smithi098.stderr:2023-03-03T07:55:38.141+0000 7fa96259d700 1 -- 172.21.15.98:0/2644058301 wait complete. 2023-03-03T07:55:38.166 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-03-03T07:55:38.166 INFO:tasks.ceph:Creating RBD pool 2023-03-03T07:55:38.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd pool create rbd 8 2023-03-03T07:55:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.401+0000 7f50cffff700 1 Processor -- start 2023-03-03T07:55:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.401+0000 7f50cffff700 1 -- start start 2023-03-03T07:55:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.403+0000 7f50cffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50c80b26d0 0x7f50c80b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.403+0000 7f50cffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f50c80b5140 0x7f50c80d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.403+0000 7f50cffff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c80cd7d0 con 0x7f50c80b8d10 2023-03-03T07:55:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.403+0000 7f50cffff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c80b55c0 con 0x7f50c80b5140 2023-03-03T07:55:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.403+0000 7f50cffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c80b90e0 con 0x7f50c80b26d0 2023-03-03T07:55:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.403+0000 7f50ce59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f50c80b5140 0x7f50c80d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.403+0000 7f50ce59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f50c80b5140 0x7f50c80d13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47608/0 (socket says 172.21.15.98:47608) 2023-03-03T07:55:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.403+0000 7f50ce59c700 1 -- 172.21.15.98:0/1118241683 learned_addr learned my addr 172.21.15.98:0/1118241683 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.403+0000 7f50cd59a700 1 --2- 172.21.15.98:0/1118241683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50c80b26d0 0x7f50c80b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.404+0000 7f50ce59c700 1 -- 172.21.15.98:0/1118241683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50c80b26d0 msgr2=0x7f50c80b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.404+0000 7f50ce59c700 1 --2- 172.21.15.98:0/1118241683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50c80b26d0 0x7f50c80b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.404+0000 7f50ce59c700 1 -- 172.21.15.98:0/1118241683 >> v1:172.21.15.100:6789/0 conn(0x7f50c80b8d10 legacy=0x7f50c80cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.404+0000 7f50ce59c700 1 -- 172.21.15.98:0/1118241683 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c80cda20 con 0x7f50c80b5140 2023-03-03T07:55:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.404+0000 7f50cd59a700 1 --2- 172.21.15.98:0/1118241683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50c80b26d0 0x7f50c80b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.404+0000 7f50ce59c700 1 --2- 172.21.15.98:0/1118241683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80b5140 0x7f50c80d13a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f50c0009e50 tx=0x7f50c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed04f2558b739279 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.404+0000 7f50ccd99700 1 -- 172.21.15.98:0/1118241683 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c000e070 con 0x7f50c80b5140 2023-03-03T07:55:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.404+0000 7f50ccd99700 1 -- 172.21.15.98:0/1118241683 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c000b9c0 con 0x7f50c80b5140 2023-03-03T07:55:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.404+0000 7f50ccd99700 1 -- 172.21.15.98:0/1118241683 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c000bd70 con 0x7f50c80b5140 2023-03-03T07:55:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.405+0000 7f50cffff700 1 -- 172.21.15.98:0/1118241683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80b5140 msgr2=0x7f50c80d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.405+0000 7f50cffff700 1 --2- 172.21.15.98:0/1118241683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80b5140 0x7f50c80d13a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f50c0009e50 tx=0x7f50c000b330 comp rx=0 tx=0).stop 2023-03-03T07:55:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.405+0000 7f50cffff700 1 -- 172.21.15.98:0/1118241683 shutdown_connections 2023-03-03T07:55:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.405+0000 7f50cffff700 1 --2- 172.21.15.98:0/1118241683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80b5140 0x7f50c80d13a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.405+0000 7f50cffff700 1 --2- 172.21.15.98:0/1118241683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50c80b26d0 0x7f50c80b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.405+0000 7f50cffff700 1 -- 172.21.15.98:0/1118241683 >> 172.21.15.98:0/1118241683 conn(0x7f50c8073e70 msgr2=0x7f50c806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.405+0000 7f50cffff700 1 -- 172.21.15.98:0/1118241683 shutdown_connections 2023-03-03T07:55:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.405+0000 7f50cffff700 1 -- 172.21.15.98:0/1118241683 wait complete. 2023-03-03T07:55:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.408+0000 7f50cffff700 1 Processor -- start 2023-03-03T07:55:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.408+0000 7f50cffff700 1 -- start start 2023-03-03T07:55:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.409+0000 7f50cffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80b26d0 0x7f50c8154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50cffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50c80b5140 0x7f50c815e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50cffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50c80b8d10 0x7f50c80b4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50cffff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c81e6830 con 0x7f50c80b26d0 2023-03-03T07:55:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50cffff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c81e69c0 con 0x7f50c80b5140 2023-03-03T07:55:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50cffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c81549c0 con 0x7f50c80b8d10 2023-03-03T07:55:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50cdd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80b26d0 0x7f50c8154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50cdd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80b26d0 0x7f50c8154480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47612/0 (socket says 172.21.15.98:47612) 2023-03-03T07:55:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50cdd9b700 1 -- 172.21.15.98:0/4001777092 learned_addr learned my addr 172.21.15.98:0/4001777092 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50cd59a700 1 --2- 172.21.15.98:0/4001777092 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50c80b5140 0x7f50c815e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50ce59c700 1 --2- 172.21.15.98:0/4001777092 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50c80b8d10 0x7f50c80b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50cdd9b700 1 -- 172.21.15.98:0/4001777092 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50c80b8d10 msgr2=0x7f50c80b4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50cdd9b700 1 --2- 172.21.15.98:0/4001777092 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50c80b8d10 0x7f50c80b4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50cdd9b700 1 -- 172.21.15.98:0/4001777092 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50c80b5140 msgr2=0x7f50c815e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50cdd9b700 1 --2- 172.21.15.98:0/4001777092 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50c80b5140 0x7f50c815e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50cdd9b700 1 -- 172.21.15.98:0/4001777092 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c0009b30 con 0x7f50c80b26d0 2023-03-03T07:55:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.410+0000 7f50ce59c700 1 --2- 172.21.15.98:0/4001777092 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50c80b8d10 0x7f50c80b4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.411+0000 7f50cdd9b700 1 --2- 172.21.15.98:0/4001777092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80b26d0 0x7f50c8154480 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f50c400c450 tx=0x7f50c400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.411+0000 7f50be7fc700 1 -- 172.21.15.98:0/4001777092 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c40052d0 con 0x7f50c80b26d0 2023-03-03T07:55:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.411+0000 7f50be7fc700 1 -- 172.21.15.98:0/4001777092 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c4005430 con 0x7f50c80b26d0 2023-03-03T07:55:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.411+0000 7f50be7fc700 1 -- 172.21.15.98:0/4001777092 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c40038c0 con 0x7f50c80b26d0 2023-03-03T07:55:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.411+0000 7f50cffff700 1 -- 172.21.15.98:0/4001777092 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c80d0e80 con 0x7f50c80b26d0 2023-03-03T07:55:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.411+0000 7f50cffff700 1 -- 172.21.15.98:0/4001777092 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c80cdd00 con 0x7f50c80b26d0 2023-03-03T07:55:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.412+0000 7f50cffff700 1 -- 172.21.15.98:0/4001777092 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c80cdfe0 con 0x7f50c80b26d0 2023-03-03T07:55:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.413+0000 7f50be7fc700 1 -- 172.21.15.98:0/4001777092 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f50c4019030 con 0x7f50c80b26d0 2023-03-03T07:55:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.415+0000 7f50be7fc700 1 --2- 172.21.15.98:0/4001777092 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50ac0682f0 0x7f50ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.415+0000 7f50be7fc700 1 -- 172.21.15.98:0/4001777092 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f50c4084b50 con 0x7f50c80b26d0 2023-03-03T07:55:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.415+0000 7f50cd59a700 1 --2- 172.21.15.98:0/4001777092 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50ac0682f0 0x7f50ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:38.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.415+0000 7f50cd59a700 1 --2- 172.21.15.98:0/4001777092 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50ac0682f0 0x7f50ac06a7a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f50c80b24c0 tx=0x7f50b8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:38.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.419+0000 7f50be7fc700 1 -- 172.21.15.98:0/4001777092 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f50c4051770 con 0x7f50c80b26d0 2023-03-03T07:55:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:38.617+0000 7f50cffff700 1 -- 172.21.15.98:0/4001777092 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f50c80cf4d0 con 0x7f50c80b26d0 2023-03-03T07:55:39.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.485+0000 7f50be7fc700 1 -- 172.21.15.98:0/4001777092 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) v1 ==== 109+0+0 (secure 0 0 0) 0x7f50c40170a0 con 0x7f50c80b26d0 2023-03-03T07:55:39.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.489+0000 7f50cffff700 1 -- 172.21.15.98:0/4001777092 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50ac0682f0 msgr2=0x7f50ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:39.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.489+0000 7f50cffff700 1 --2- 172.21.15.98:0/4001777092 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50ac0682f0 0x7f50ac06a7a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f50c80b24c0 tx=0x7f50b8008040 comp rx=0 tx=0).stop 2023-03-03T07:55:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.489+0000 7f50cffff700 1 -- 172.21.15.98:0/4001777092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80b26d0 msgr2=0x7f50c8154480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.489+0000 7f50cffff700 1 --2- 172.21.15.98:0/4001777092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80b26d0 0x7f50c8154480 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f50c400c450 tx=0x7f50c400c760 comp rx=0 tx=0).stop 2023-03-03T07:55:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.489+0000 7f50cffff700 1 -- 172.21.15.98:0/4001777092 shutdown_connections 2023-03-03T07:55:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.489+0000 7f50cffff700 1 --2- 172.21.15.98:0/4001777092 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50ac0682f0 0x7f50ac06a7a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.489+0000 7f50cffff700 1 --2- 172.21.15.98:0/4001777092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80b26d0 0x7f50c8154480 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.489+0000 7f50cffff700 1 --2- 172.21.15.98:0/4001777092 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50c80b8d10 0x7f50c80b4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.489+0000 7f50cffff700 1 --2- 172.21.15.98:0/4001777092 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50c80b5140 0x7f50c815e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:39.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.489+0000 7f50cffff700 1 -- 172.21.15.98:0/4001777092 >> 172.21.15.98:0/4001777092 conn(0x7f50c8073e70 msgr2=0x7f50c806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:39.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.489+0000 7f50cffff700 1 -- 172.21.15.98:0/4001777092 shutdown_connections 2023-03-03T07:55:39.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.489+0000 7f50cffff700 1 -- 172.21.15.98:0/4001777092 wait complete. 2023-03-03T07:55:39.492 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'rbd' created 2023-03-03T07:55:39.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-03-03T07:55:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.760+0000 7f970ffff700 1 Processor -- start 2023-03-03T07:55:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.760+0000 7f970ffff700 1 -- start start 2023-03-03T07:55:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.762+0000 7f970ffff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f970815cc40 0x7f97080cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.762+0000 7f970ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f970815d010 0x7f97080cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.762+0000 7f970ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f97080b1d80 0x7f97080b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.762+0000 7f970ffff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97080b2740 con 0x7f97080b1d80 2023-03-03T07:55:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.762+0000 7f970ffff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97081617b0 con 0x7f970815cc40 2023-03-03T07:55:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.762+0000 7f970ffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970806b900 con 0x7f970815d010 2023-03-03T07:55:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.762+0000 7f970e59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f97080b1d80 0x7f97080b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.762+0000 7f970e59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f97080b1d80 0x7f97080b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47616/0 (socket says 172.21.15.98:47616) 2023-03-03T07:55:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.762+0000 7f970e59c700 1 -- 172.21.15.98:0/2109926904 learned_addr learned my addr 172.21.15.98:0/2109926904 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.762+0000 7f970d59a700 1 --2- 172.21.15.98:0/2109926904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f970815d010 0x7f97080cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.762+0000 7f970dd9b700 1 --2- 172.21.15.98:0/2109926904 >> v2:172.21.15.100:3300/0 conn(0x7f970815cc40 0x7f97080cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.763+0000 7f970e59c700 1 -- 172.21.15.98:0/2109926904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f970815d010 msgr2=0x7f97080cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.763+0000 7f970e59c700 1 --2- 172.21.15.98:0/2109926904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f970815d010 0x7f97080cdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.763+0000 7f970e59c700 1 -- 172.21.15.98:0/2109926904 >> v2:172.21.15.100:3300/0 conn(0x7f970815cc40 msgr2=0x7f97080cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.763+0000 7f970e59c700 1 --2- 172.21.15.98:0/2109926904 >> v2:172.21.15.100:3300/0 conn(0x7f970815cc40 0x7f97080cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.763+0000 7f970e59c700 1 -- 172.21.15.98:0/2109926904 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97081609f0 con 0x7f97080b1d80 2023-03-03T07:55:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.763+0000 7f970e59c700 1 --2- 172.21.15.98:0/2109926904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97080b1d80 0x7f97080b2200 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f970806be50 tx=0x7f970000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c736df10abbc91 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.763+0000 7f970cd99700 1 -- 172.21.15.98:0/2109926904 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970000e070 con 0x7f97080b1d80 2023-03-03T07:55:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.763+0000 7f970cd99700 1 -- 172.21.15.98:0/2109926904 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970000b910 con 0x7f97080b1d80 2023-03-03T07:55:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.763+0000 7f970cd99700 1 -- 172.21.15.98:0/2109926904 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970000bcc0 con 0x7f97080b1d80 2023-03-03T07:55:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.764+0000 7f970ffff700 1 -- 172.21.15.98:0/2109926904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97080b1d80 msgr2=0x7f97080b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.764+0000 7f970ffff700 1 --2- 172.21.15.98:0/2109926904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97080b1d80 0x7f97080b2200 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f970806be50 tx=0x7f970000b330 comp rx=0 tx=0).stop 2023-03-03T07:55:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.764+0000 7f970ffff700 1 -- 172.21.15.98:0/2109926904 shutdown_connections 2023-03-03T07:55:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.764+0000 7f970ffff700 1 --2- 172.21.15.98:0/2109926904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97080b1d80 0x7f97080b2200 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.764+0000 7f970ffff700 1 --2- 172.21.15.98:0/2109926904 >> v2:172.21.15.100:3300/0 conn(0x7f970815cc40 0x7f97080cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.764+0000 7f970ffff700 1 --2- 172.21.15.98:0/2109926904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f970815d010 0x7f97080cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.764+0000 7f970ffff700 1 -- 172.21.15.98:0/2109926904 >> 172.21.15.98:0/2109926904 conn(0x7f9708073e10 msgr2=0x7f97080d31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.764+0000 7f970ffff700 1 -- 172.21.15.98:0/2109926904 shutdown_connections 2023-03-03T07:55:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.764+0000 7f970ffff700 1 -- 172.21.15.98:0/2109926904 wait complete. 2023-03-03T07:55:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.766+0000 7f970ffff700 1 Processor -- start 2023-03-03T07:55:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.766+0000 7f970ffff700 1 -- start start 2023-03-03T07:55:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.767+0000 7f970ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97080b1d80 0x7f97081f1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.767+0000 7f970ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f970815cc40 0x7f97080b2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.767+0000 7f970ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f970815d010 0x7f97080b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.767+0000 7f970ffff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97080b4080 con 0x7f970815d010 2023-03-03T07:55:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.767+0000 7f970ffff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97080cbd20 con 0x7f97080b1d80 2023-03-03T07:55:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.767+0000 7f970ffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97080b3470 con 0x7f970815cc40 2023-03-03T07:55:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.768+0000 7f970e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f970815d010 0x7f97080b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.768+0000 7f970e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f970815d010 0x7f97080b2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47626/0 (socket says 172.21.15.98:47626) 2023-03-03T07:55:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.768+0000 7f970e59c700 1 -- 172.21.15.98:0/3947231662 learned_addr learned my addr 172.21.15.98:0/3947231662 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.768+0000 7f970d59a700 1 --2- 172.21.15.98:0/3947231662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f970815cc40 0x7f97080b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.768+0000 7f970dd9b700 1 --2- 172.21.15.98:0/3947231662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97080b1d80 0x7f97081f1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.768+0000 7f970e59c700 1 -- 172.21.15.98:0/3947231662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f970815cc40 msgr2=0x7f97080b2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.768+0000 7f970e59c700 1 --2- 172.21.15.98:0/3947231662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f970815cc40 0x7f97080b2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.768+0000 7f970e59c700 1 -- 172.21.15.98:0/3947231662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97080b1d80 msgr2=0x7f97081f1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.768+0000 7f970e59c700 1 --2- 172.21.15.98:0/3947231662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97080b1d80 0x7f97081f1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.768+0000 7f970e59c700 1 -- 172.21.15.98:0/3947231662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9700009b30 con 0x7f970815d010 2023-03-03T07:55:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.768+0000 7f970e59c700 1 --2- 172.21.15.98:0/3947231662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f970815d010 0x7f97080b2e40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f970808c160 tx=0x7f9700004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.769+0000 7f96fe7fc700 1 -- 172.21.15.98:0/3947231662 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97000099b0 con 0x7f970815d010 2023-03-03T07:55:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.769+0000 7f96fe7fc700 1 -- 172.21.15.98:0/3947231662 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970000ce80 con 0x7f970815d010 2023-03-03T07:55:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.769+0000 7f96fe7fc700 1 -- 172.21.15.98:0/3947231662 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9700004490 con 0x7f970815d010 2023-03-03T07:55:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.769+0000 7f970ffff700 1 -- 172.21.15.98:0/3947231662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9708160fb0 con 0x7f970815d010 2023-03-03T07:55:39.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.769+0000 7f970ffff700 1 -- 172.21.15.98:0/3947231662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9708160cd0 con 0x7f970815d010 2023-03-03T07:55:39.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.770+0000 7f970ffff700 1 -- 172.21.15.98:0/3947231662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9708161290 con 0x7f970815d010 2023-03-03T07:55:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.772+0000 7f96fe7fc700 1 -- 172.21.15.98:0/3947231662 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f97000045f0 con 0x7f970815d010 2023-03-03T07:55:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.773+0000 7f96fe7fc700 1 --2- 172.21.15.98:0/3947231662 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f96ec0682f0 0x7f96ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.773+0000 7f96fe7fc700 1 -- 172.21.15.98:0/3947231662 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f9700086e00 con 0x7f970815d010 2023-03-03T07:55:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.773+0000 7f970dd9b700 1 --2- 172.21.15.98:0/3947231662 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f96ec0682f0 0x7f96ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.774+0000 7f970dd9b700 1 --2- 172.21.15.98:0/3947231662 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f96ec0682f0 0x7f96ec06a7a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9704009b60 tx=0x7f9704009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.778+0000 7f96fe7fc700 1 -- 172.21.15.98:0/3947231662 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9700053980 con 0x7f970815d010 2023-03-03T07:55:39.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:39.971+0000 7f970ffff700 1 -- 172.21.15.98:0/3947231662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f97080b41c0 con 0x7f970815d010 2023-03-03T07:55:40.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.492+0000 7f96fe7fc700 1 -- 172.21.15.98:0/3947231662 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v18) v1 ==== 173+0+0 (secure 0 0 0) 0x7f97000200c0 con 0x7f970815d010 2023-03-03T07:55:40.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.496+0000 7f970ffff700 1 -- 172.21.15.98:0/3947231662 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f96ec0682f0 msgr2=0x7f96ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:40.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.496+0000 7f970ffff700 1 --2- 172.21.15.98:0/3947231662 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f96ec0682f0 0x7f96ec06a7a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9704009b60 tx=0x7f9704009210 comp rx=0 tx=0).stop 2023-03-03T07:55:40.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.496+0000 7f970ffff700 1 -- 172.21.15.98:0/3947231662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f970815d010 msgr2=0x7f97080b2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:40.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.496+0000 7f970ffff700 1 --2- 172.21.15.98:0/3947231662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f970815d010 0x7f97080b2e40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f970808c160 tx=0x7f9700004ad0 comp rx=0 tx=0).stop 2023-03-03T07:55:40.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.496+0000 7f970ffff700 1 -- 172.21.15.98:0/3947231662 shutdown_connections 2023-03-03T07:55:40.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.496+0000 7f970ffff700 1 --2- 172.21.15.98:0/3947231662 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f96ec0682f0 0x7f96ec06a7a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:40.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.496+0000 7f970ffff700 1 --2- 172.21.15.98:0/3947231662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f970815d010 0x7f97080b2e40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:40.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.496+0000 7f970ffff700 1 --2- 172.21.15.98:0/3947231662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97080b1d80 0x7f97081f1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:40.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.496+0000 7f970ffff700 1 --2- 172.21.15.98:0/3947231662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f970815cc40 0x7f97080b2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:40.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.496+0000 7f970ffff700 1 -- 172.21.15.98:0/3947231662 >> 172.21.15.98:0/3947231662 conn(0x7f9708073e10 msgr2=0x7f970815b840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:40.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.496+0000 7f970ffff700 1 -- 172.21.15.98:0/3947231662 shutdown_connections 2023-03-03T07:55:40.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.496+0000 7f970ffff700 1 -- 172.21.15.98:0/3947231662 wait complete. 2023-03-03T07:55:40.500 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rbd' on pool 'rbd' 2023-03-03T07:55:40.526 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-03-03T07:55:40.527 INFO:tasks.ceph.mds.a:Restarting daemon 2023-03-03T07:55:40.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-03-03T07:55:40.533 INFO:tasks.ceph.mds.a:Started 2023-03-03T07:55:40.533 INFO:tasks.ceph.mds.c:Restarting daemon 2023-03-03T07:55:40.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-03-03T07:55:40.539 INFO:tasks.ceph.mds.c:Started 2023-03-03T07:55:40.539 INFO:tasks.ceph.mds.b:Restarting daemon 2023-03-03T07:55:40.540 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-03-03T07:55:40.545 INFO:tasks.ceph.mds.b:Started 2023-03-03T07:55:40.546 INFO:tasks.ceph.mds.d:Restarting daemon 2023-03-03T07:55:40.546 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-03-03T07:55:40.552 INFO:tasks.ceph.mds.d:Started 2023-03-03T07:55:40.552 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-03-03T07:55:40.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:55:40.737 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2023-03-03T07:55:40.737 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-03-03T07:55:40.742 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2023-03-03T07:55:40.747 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2023-03-03T07:55:40.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.817+0000 7fa6639ff700 1 Processor -- start 2023-03-03T07:55:40.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.817+0000 7fa6639ff700 1 -- start start 2023-03-03T07:55:40.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.818+0000 7fa6639ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa65c0cd8d0 0x7fa65c0b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:40.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.818+0000 7fa6639ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa65c0b6610 0x7fa65c0cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:40.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.819+0000 7fa6639ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa65c0cdd70 con 0x7fa65c15cc00 2023-03-03T07:55:40.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.819+0000 7fa6639ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65c0ce040 con 0x7fa65c0cd8d0 2023-03-03T07:55:40.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.819+0000 7fa6639ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65c073be0 con 0x7fa65c0b6610 2023-03-03T07:55:40.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.819+0000 7fa660f9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa65c0cd8d0 0x7fa65c0b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:40.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.819+0000 7fa66179b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa65c15cc00 0x7fa65c0d27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46956/0 (socket says 172.21.15.98:46956) 2023-03-03T07:55:40.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.819+0000 7fa66179b700 1 -- 172.21.15.98:0/166844055 learned_addr learned my addr 172.21.15.98:0/166844055 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:40.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.819+0000 7fa661f9c700 1 --2- 172.21.15.98:0/166844055 >> v2:172.21.15.100:3300/0 conn(0x7fa65c0b6610 0x7fa65c0cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:40.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.819+0000 7fa653fff700 1 -- 172.21.15.98:0/166844055 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006857655 0 0) 0x7fa65c0cdd70 con 0x7fa65c15cc00 2023-03-03T07:55:40.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.819+0000 7fa660f9a700 1 -- 172.21.15.98:0/166844055 >> v2:172.21.15.100:3300/0 conn(0x7fa65c0b6610 msgr2=0x7fa65c0cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:40.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.819+0000 7fa660f9a700 1 --2- 172.21.15.98:0/166844055 >> v2:172.21.15.100:3300/0 conn(0x7fa65c0b6610 0x7fa65c0cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:40.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.819+0000 7fa660f9a700 1 -- 172.21.15.98:0/166844055 >> v1:172.21.15.98:6789/0 conn(0x7fa65c15cc00 legacy=0x7fa65c0d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:40.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.819+0000 7fa660f9a700 1 -- 172.21.15.98:0/166844055 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65c0d1a00 con 0x7fa65c0cd8d0 2023-03-03T07:55:40.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.820+0000 7fa660f9a700 1 --2- 172.21.15.98:0/166844055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa65c0cd8d0 0x7fa65c0b60d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0d4360 tx=0x7fa64c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae21a3028c262607 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:40.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.820+0000 7fa653fff700 1 -- 172.21.15.98:0/166844055 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c011830 con 0x7fa65c0cd8d0 2023-03-03T07:55:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.820+0000 7fa653fff700 1 -- 172.21.15.98:0/166844055 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c011990 con 0x7fa65c0cd8d0 2023-03-03T07:55:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.820+0000 7fa653fff700 1 -- 172.21.15.98:0/166844055 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c011d20 con 0x7fa65c0cd8d0 2023-03-03T07:55:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.820+0000 7fa6639ff700 1 -- 172.21.15.98:0/166844055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa65c0cd8d0 msgr2=0x7fa65c0b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.820+0000 7fa6639ff700 1 --2- 172.21.15.98:0/166844055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa65c0cd8d0 0x7fa65c0b60d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0d4360 tx=0x7fa64c009b20 comp rx=0 tx=0).stop 2023-03-03T07:55:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.821+0000 7fa6639ff700 1 -- 172.21.15.98:0/166844055 shutdown_connections 2023-03-03T07:55:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.821+0000 7fa6639ff700 1 --2- 172.21.15.98:0/166844055 >> v2:172.21.15.100:3300/0 conn(0x7fa65c0b6610 0x7fa65c0cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.821+0000 7fa6639ff700 1 --2- 172.21.15.98:0/166844055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa65c0cd8d0 0x7fa65c0b60d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.821+0000 7fa6639ff700 1 -- 172.21.15.98:0/166844055 >> 172.21.15.98:0/166844055 conn(0x7fa65c06aee0 msgr2=0x7fa65c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:40.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.821+0000 7fa6639ff700 1 -- 172.21.15.98:0/166844055 shutdown_connections 2023-03-03T07:55:40.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.821+0000 7fa6639ff700 1 -- 172.21.15.98:0/166844055 wait complete. 2023-03-03T07:55:40.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.823+0000 7fa6639ff700 1 Processor -- start 2023-03-03T07:55:40.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.823+0000 7fa6639ff700 1 -- start start 2023-03-03T07:55:40.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.824+0000 7fa6639ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa65c0b6610 0x7fa65c0cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:40.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.824+0000 7fa6639ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa65c0cd8d0 0x7fa65c0cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:40.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.824+0000 7fa6639ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa65c15cc00 0x7fa65c1f1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:40.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.824+0000 7fa6639ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c0ccf10 con 0x7fa65c0b6610 2023-03-03T07:55:40.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.824+0000 7fa6639ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c0cbae0 con 0x7fa65c15cc00 2023-03-03T07:55:40.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.824+0000 7fa6639ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c0cc300 con 0x7fa65c0cd8d0 2023-03-03T07:55:40.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.824+0000 7fa66179b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa65c0b6610 0x7fa65c0cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:40.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.824+0000 7fa660f9a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa65c0cd8d0 0x7fa65c0cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.824+0000 7fa660f9a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa65c0cd8d0 0x7fa65c0cb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:39336/0 (socket says 172.21.15.98:39336) 2023-03-03T07:55:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.824+0000 7fa660f9a700 1 -- 172.21.15.98:0/812924839 learned_addr learned my addr 172.21.15.98:0/812924839 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.824+0000 7fa661f9c700 1 --2- 172.21.15.98:0/812924839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa65c15cc00 0x7fa65c1f1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.825+0000 7fa660f9a700 1 -- 172.21.15.98:0/812924839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa65c15cc00 msgr2=0x7fa65c1f1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.825+0000 7fa660f9a700 1 --2- 172.21.15.98:0/812924839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa65c15cc00 0x7fa65c1f1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.825+0000 7fa660f9a700 1 -- 172.21.15.98:0/812924839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa65c0b6610 msgr2=0x7fa65c0cbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.825+0000 7fa660f9a700 1 --2- 172.21.15.98:0/812924839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa65c0b6610 0x7fa65c0cbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.825+0000 7fa660f9a700 1 -- 172.21.15.98:0/812924839 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c0094a0 con 0x7fa65c0cd8d0 2023-03-03T07:55:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.825+0000 7fa660f9a700 1 --2- 172.21.15.98:0/812924839 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa65c0cd8d0 0x7fa65c0cb5a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa64c000c00 tx=0x7fa64c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.825+0000 7fa651ffb700 1 -- 172.21.15.98:0/812924839 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c004550 con 0x7fa65c0cd8d0 2023-03-03T07:55:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.825+0000 7fa651ffb700 1 -- 172.21.15.98:0/812924839 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c01e070 con 0x7fa65c0cd8d0 2023-03-03T07:55:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.825+0000 7fa6639ff700 1 -- 172.21.15.98:0/812924839 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65c0d1fc0 con 0x7fa65c0cd8d0 2023-03-03T07:55:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.825+0000 7fa651ffb700 1 -- 172.21.15.98:0/812924839 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c0038e0 con 0x7fa65c0cd8d0 2023-03-03T07:55:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.825+0000 7fa6639ff700 1 -- 172.21.15.98:0/812924839 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65c0d1ce0 con 0x7fa65c0cd8d0 2023-03-03T07:55:40.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.826+0000 7fa6639ff700 1 -- 172.21.15.98:0/812924839 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65c0d22a0 con 0x7fa65c0cd8d0 2023-03-03T07:55:40.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.828+0000 7fa651ffb700 1 -- 172.21.15.98:0/812924839 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa64c00f050 con 0x7fa65c0cd8d0 2023-03-03T07:55:40.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.829+0000 7fa651ffb700 1 --2- 172.21.15.98:0/812924839 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6400682f0 0x7fa64006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:40.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.829+0000 7fa651ffb700 1 -- 172.21.15.98:0/812924839 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa64c086830 con 0x7fa65c0cd8d0 2023-03-03T07:55:40.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.829+0000 7fa66179b700 1 --2- 172.21.15.98:0/812924839 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6400682f0 0x7fa64006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:40.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.830+0000 7fa66179b700 1 --2- 172.21.15.98:0/812924839 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6400682f0 0x7fa64006a7a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa658007fd0 tx=0x7fa658007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:40.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:40.834+0000 7fa651ffb700 1 -- 172.21.15.98:0/812924839 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa64c0532f0 con 0x7fa65c0cd8d0 2023-03-03T07:55:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.028+0000 7fa6639ff700 1 -- 172.21.15.98:0/812924839 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa65c0729b0 con 0x7fa65c0cd8d0 2023-03-03T07:55:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.030+0000 7fa651ffb700 1 -- 172.21.15.98:0/812924839 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+14277 (secure 0 0 0) 0x7fa64c056910 con 0x7fa65c0cd8d0 2023-03-03T07:55:41.031 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:41.031 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":18,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:55:40.484805+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:55:41.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.032+0000 7fa6639ff700 1 -- 172.21.15.98:0/812924839 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6400682f0 msgr2=0x7fa64006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:41.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.032+0000 7fa6639ff700 1 --2- 172.21.15.98:0/812924839 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6400682f0 0x7fa64006a7a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa658007fd0 tx=0x7fa658007de0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.032+0000 7fa6639ff700 1 -- 172.21.15.98:0/812924839 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa65c0cd8d0 msgr2=0x7fa65c0cb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:41.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.032+0000 7fa6639ff700 1 --2- 172.21.15.98:0/812924839 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa65c0cd8d0 0x7fa65c0cb5a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa64c000c00 tx=0x7fa64c004750 comp rx=0 tx=0).stop 2023-03-03T07:55:41.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.032+0000 7fa6639ff700 1 -- 172.21.15.98:0/812924839 shutdown_connections 2023-03-03T07:55:41.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.032+0000 7fa6639ff700 1 --2- 172.21.15.98:0/812924839 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6400682f0 0x7fa64006a7a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.032+0000 7fa6639ff700 1 --2- 172.21.15.98:0/812924839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa65c0b6610 0x7fa65c0cbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.032+0000 7fa6639ff700 1 --2- 172.21.15.98:0/812924839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa65c15cc00 0x7fa65c1f1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.032+0000 7fa6639ff700 1 --2- 172.21.15.98:0/812924839 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa65c0cd8d0 0x7fa65c0cb5a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.032+0000 7fa6639ff700 1 -- 172.21.15.98:0/812924839 >> 172.21.15.98:0/812924839 conn(0x7fa65c06aee0 msgr2=0x7fa65c15bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:41.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.032+0000 7fa6639ff700 1 -- 172.21.15.98:0/812924839 shutdown_connections 2023-03-03T07:55:41.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.032+0000 7fa6639ff700 1 -- 172.21.15.98:0/812924839 wait complete. 2023-03-03T07:55:41.061 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:55:41.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:55:41.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.310+0000 7fde8adff700 1 Processor -- start 2023-03-03T07:55:41.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.310+0000 7fde8adff700 1 -- start start 2023-03-03T07:55:41.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.312+0000 7fde8adff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8415ca70 0x7fde840cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:41.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.312+0000 7fde8adff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fde8406b4b0 0x7fde840cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:41.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.312+0000 7fde8adff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde840cf980 con 0x7fde840d00a0 2023-03-03T07:55:41.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.312+0000 7fde8adff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde840cd5b0 con 0x7fde8406b4b0 2023-03-03T07:55:41.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.312+0000 7fde8adff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde840d0510 con 0x7fde8415ca70 2023-03-03T07:55:41.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.312+0000 7fde83fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fde8406b4b0 0x7fde840cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:41.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.313+0000 7fde88b9b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8415ca70 0x7fde840cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:41.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.313+0000 7fde83fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fde8406b4b0 0x7fde840cfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:57850/0 (socket says 172.21.15.98:57850) 2023-03-03T07:55:41.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.313+0000 7fde8939c700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fde840d00a0 0x7fde840b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:42778/0 (socket says 172.21.15.98:42778) 2023-03-03T07:55:41.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.313+0000 7fde83fff700 1 -- 172.21.15.98:0/2042289099 learned_addr learned my addr 172.21.15.98:0/2042289099 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:41.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.313+0000 7fde88b9b700 1 -- 172.21.15.98:0/2042289099 >> v2:172.21.15.100:3300/0 conn(0x7fde8406b4b0 msgr2=0x7fde840cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:41.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.313+0000 7fde88b9b700 1 --2- 172.21.15.98:0/2042289099 >> v2:172.21.15.100:3300/0 conn(0x7fde8406b4b0 0x7fde840cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.313+0000 7fde88b9b700 1 -- 172.21.15.98:0/2042289099 >> v1:172.21.15.100:6789/0 conn(0x7fde840d00a0 legacy=0x7fde840b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:41.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.313+0000 7fde88b9b700 1 -- 172.21.15.98:0/2042289099 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde840d2fd0 con 0x7fde8415ca70 2023-03-03T07:55:41.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.313+0000 7fde83fff700 1 --2- 172.21.15.98:0/2042289099 >> v2:172.21.15.100:3300/0 conn(0x7fde8406b4b0 0x7fde840cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:41.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.313+0000 7fde837fe700 1 -- 172.21.15.98:0/2042289099 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311031762 0 0) 0x7fde840cf980 con 0x7fde840d00a0 2023-03-03T07:55:41.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.313+0000 7fde88b9b700 1 --2- 172.21.15.98:0/2042289099 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8415ca70 0x7fde840cf440 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fde78009840 tx=0x7fde78009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d01f6ede3ed8a34 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:41.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.314+0000 7fde837fe700 1 -- 172.21.15.98:0/2042289099 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78011830 con 0x7fde8415ca70 2023-03-03T07:55:41.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.314+0000 7fde837fe700 1 -- 172.21.15.98:0/2042289099 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde78011990 con 0x7fde8415ca70 2023-03-03T07:55:41.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.314+0000 7fde837fe700 1 -- 172.21.15.98:0/2042289099 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78011d20 con 0x7fde8415ca70 2023-03-03T07:55:41.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.314+0000 7fde8adff700 1 -- 172.21.15.98:0/2042289099 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8415ca70 msgr2=0x7fde840cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:41.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.314+0000 7fde8adff700 1 --2- 172.21.15.98:0/2042289099 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8415ca70 0x7fde840cf440 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fde78009840 tx=0x7fde78009b50 comp rx=0 tx=0).stop 2023-03-03T07:55:41.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.314+0000 7fde8adff700 1 -- 172.21.15.98:0/2042289099 shutdown_connections 2023-03-03T07:55:41.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.314+0000 7fde8adff700 1 --2- 172.21.15.98:0/2042289099 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8415ca70 0x7fde840cf440 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.314+0000 7fde8adff700 1 --2- 172.21.15.98:0/2042289099 >> v2:172.21.15.100:3300/0 conn(0x7fde8406b4b0 0x7fde840cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.314+0000 7fde8adff700 1 -- 172.21.15.98:0/2042289099 >> 172.21.15.98:0/2042289099 conn(0x7fde84073e70 msgr2=0x7fde8406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:41.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.314+0000 7fde8adff700 1 -- 172.21.15.98:0/2042289099 shutdown_connections 2023-03-03T07:55:41.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.315+0000 7fde8adff700 1 -- 172.21.15.98:0/2042289099 wait complete. 2023-03-03T07:55:41.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.317+0000 7fde8adff700 1 Processor -- start 2023-03-03T07:55:41.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.318+0000 7fde8adff700 1 -- start start 2023-03-03T07:55:41.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.319+0000 7fde8adff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde8406b4b0 0x7fde841f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:41.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.319+0000 7fde8adff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde840d00a0 0x7fde84157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:41.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.319+0000 7fde8adff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde8415ca70 0x7fde840ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:41.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.319+0000 7fde8adff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde841580e0 con 0x7fde8406b4b0 2023-03-03T07:55:41.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.319+0000 7fde8adff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde841f66b0 con 0x7fde8415ca70 2023-03-03T07:55:41.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.319+0000 7fde8adff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde84157e80 con 0x7fde840d00a0 2023-03-03T07:55:41.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.319+0000 7fde88b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde8406b4b0 0x7fde841f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:41.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.319+0000 7fde88b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde8406b4b0 0x7fde841f2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47666/0 (socket says 172.21.15.98:47666) 2023-03-03T07:55:41.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.319+0000 7fde83fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde840d00a0 0x7fde84157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:41.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.319+0000 7fde88b9b700 1 -- 172.21.15.98:0/2906920999 learned_addr learned my addr 172.21.15.98:0/2906920999 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:41.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.319+0000 7fde8939c700 1 --2- 172.21.15.98:0/2906920999 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde8415ca70 0x7fde840ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:41.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.320+0000 7fde83fff700 1 -- 172.21.15.98:0/2906920999 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde8415ca70 msgr2=0x7fde840ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:41.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.320+0000 7fde83fff700 1 --2- 172.21.15.98:0/2906920999 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde8415ca70 0x7fde840ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.320+0000 7fde83fff700 1 -- 172.21.15.98:0/2906920999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde8406b4b0 msgr2=0x7fde841f2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:41.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.320+0000 7fde83fff700 1 --2- 172.21.15.98:0/2906920999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde8406b4b0 0x7fde841f2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.320+0000 7fde83fff700 1 -- 172.21.15.98:0/2906920999 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde70009550 con 0x7fde840d00a0 2023-03-03T07:55:41.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.320+0000 7fde88b9b700 1 --2- 172.21.15.98:0/2906920999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde8406b4b0 0x7fde841f2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:41.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.320+0000 7fde83fff700 1 --2- 172.21.15.98:0/2906920999 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde840d00a0 0x7fde84157940 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fde840ceed0 tx=0x7fde7000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:41.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.320+0000 7fde817fa700 1 -- 172.21.15.98:0/2906920999 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde70004240 con 0x7fde840d00a0 2023-03-03T07:55:41.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.320+0000 7fde817fa700 1 -- 172.21.15.98:0/2906920999 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde700043a0 con 0x7fde840d00a0 2023-03-03T07:55:41.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.320+0000 7fde8adff700 1 -- 172.21.15.98:0/2906920999 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde780094a0 con 0x7fde840d00a0 2023-03-03T07:55:41.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.321+0000 7fde817fa700 1 -- 172.21.15.98:0/2906920999 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde700036a0 con 0x7fde840d00a0 2023-03-03T07:55:41.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.321+0000 7fde8adff700 1 -- 172.21.15.98:0/2906920999 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde840cd900 con 0x7fde840d00a0 2023-03-03T07:55:41.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.322+0000 7fde8adff700 1 -- 172.21.15.98:0/2906920999 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde840d1800 con 0x7fde840d00a0 2023-03-03T07:55:41.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.323+0000 7fde817fa700 1 -- 172.21.15.98:0/2906920999 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fde70003800 con 0x7fde840d00a0 2023-03-03T07:55:41.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.324+0000 7fde817fa700 1 --2- 172.21.15.98:0/2906920999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fde680682f0 0x7fde6806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:41.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.324+0000 7fde817fa700 1 -- 172.21.15.98:0/2906920999 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fde70085c70 con 0x7fde840d00a0 2023-03-03T07:55:41.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.324+0000 7fde88b9b700 1 --2- 172.21.15.98:0/2906920999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fde680682f0 0x7fde6806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:41.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.325+0000 7fde88b9b700 1 --2- 172.21.15.98:0/2906920999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fde680682f0 0x7fde6806a7a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fde78000c00 tx=0x7fde7801c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:41.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.328+0000 7fde817fa700 1 -- 172.21.15.98:0/2906920999 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fde70052730 con 0x7fde840d00a0 2023-03-03T07:55:41.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.525+0000 7fde8adff700 1 -- 172.21.15.98:0/2906920999 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde840cdbe0 con 0x7fde840d00a0 2023-03-03T07:55:41.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.526+0000 7fde817fa700 1 -- 172.21.15.98:0/2906920999 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7fde70055d50 con 0x7fde840d00a0 2023-03-03T07:55:41.527 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:55:41.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.528+0000 7fde8adff700 1 -- 172.21.15.98:0/2906920999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fde680682f0 msgr2=0x7fde6806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:41.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.528+0000 7fde8adff700 1 --2- 172.21.15.98:0/2906920999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fde680682f0 0x7fde6806a7a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fde78000c00 tx=0x7fde7801c040 comp rx=0 tx=0).stop 2023-03-03T07:55:41.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.528+0000 7fde8adff700 1 -- 172.21.15.98:0/2906920999 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde840d00a0 msgr2=0x7fde84157940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:41.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.528+0000 7fde8adff700 1 --2- 172.21.15.98:0/2906920999 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde840d00a0 0x7fde84157940 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fde840ceed0 tx=0x7fde7000ec80 comp rx=0 tx=0).stop 2023-03-03T07:55:41.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.528+0000 7fde8adff700 1 -- 172.21.15.98:0/2906920999 shutdown_connections 2023-03-03T07:55:41.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.528+0000 7fde8adff700 1 --2- 172.21.15.98:0/2906920999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fde680682f0 0x7fde6806a7a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.528+0000 7fde8adff700 1 --2- 172.21.15.98:0/2906920999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde8406b4b0 0x7fde841f2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.528+0000 7fde8adff700 1 --2- 172.21.15.98:0/2906920999 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde8415ca70 0x7fde840ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.528+0000 7fde8adff700 1 --2- 172.21.15.98:0/2906920999 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde840d00a0 0x7fde84157940 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.528+0000 7fde8adff700 1 -- 172.21.15.98:0/2906920999 >> 172.21.15.98:0/2906920999 conn(0x7fde84073e70 msgr2=0x7fde8415a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:41.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.529+0000 7fde8adff700 1 -- 172.21.15.98:0/2906920999 shutdown_connections 2023-03-03T07:55:41.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.529+0000 7fde8adff700 1 -- 172.21.15.98:0/2906920999 wait complete. 2023-03-03T07:55:41.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:55:41.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.810+0000 7f16f899d700 1 Processor -- start 2023-03-03T07:55:41.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.810+0000 7f16f899d700 1 -- start start 2023-03-03T07:55:41.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.812+0000 7f16f899d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f16f40cf570 0x7f16f415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:41.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.812+0000 7f16f899d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16f415db30 0x7f16f40b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:41.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.812+0000 7f16f899d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f40cd780 con 0x7f16f41609b0 2023-03-03T07:55:41.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.812+0000 7f16f899d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f40cebb0 con 0x7f16f40cf570 2023-03-03T07:55:41.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.812+0000 7f16f899d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f40cda60 con 0x7f16f415db30 2023-03-03T07:55:41.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.812+0000 7f16f2ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f16f40cf570 0x7f16f415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:41.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.812+0000 7f16f2ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f16f40cf570 0x7f16f415d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47672/0 (socket says 172.21.15.98:47672) 2023-03-03T07:55:41.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.812+0000 7f16f2ffd700 1 -- 172.21.15.98:0/810245205 learned_addr learned my addr 172.21.15.98:0/810245205 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:41.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.812+0000 7f16f3fff700 1 --2- 172.21.15.98:0/810245205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16f415db30 0x7f16f40b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:41.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.813+0000 7f16f2ffd700 1 -- 172.21.15.98:0/810245205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16f415db30 msgr2=0x7f16f40b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:41.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.813+0000 7f16f2ffd700 1 --2- 172.21.15.98:0/810245205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16f415db30 0x7f16f40b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.813+0000 7f16f2ffd700 1 -- 172.21.15.98:0/810245205 >> v1:172.21.15.100:6789/0 conn(0x7f16f41609b0 legacy=0x7f16f4160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:41.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.813+0000 7f16f2ffd700 1 -- 172.21.15.98:0/810245205 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f40cdd40 con 0x7f16f40cf570 2023-03-03T07:55:41.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.813+0000 7f16f2ffd700 1 --2- 172.21.15.98:0/810245205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16f40cf570 0x7f16f415d5f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f16e80098f0 tx=0x7f16e8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e599dc3c1ab8ca server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:41.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.813+0000 7f16f27fc700 1 -- 172.21.15.98:0/810245205 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e800b8a0 con 0x7f16f40cf570 2023-03-03T07:55:41.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.813+0000 7f16f27fc700 1 -- 172.21.15.98:0/810245205 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e800ba00 con 0x7f16f40cf570 2023-03-03T07:55:41.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.813+0000 7f16f27fc700 1 -- 172.21.15.98:0/810245205 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e800bd90 con 0x7f16f40cf570 2023-03-03T07:55:41.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.814+0000 7f16f899d700 1 -- 172.21.15.98:0/810245205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16f40cf570 msgr2=0x7f16f415d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:41.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.814+0000 7f16f899d700 1 --2- 172.21.15.98:0/810245205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16f40cf570 0x7f16f415d5f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f16e80098f0 tx=0x7f16e8009c00 comp rx=0 tx=0).stop 2023-03-03T07:55:41.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.814+0000 7f16f899d700 1 -- 172.21.15.98:0/810245205 shutdown_connections 2023-03-03T07:55:41.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.814+0000 7f16f899d700 1 --2- 172.21.15.98:0/810245205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16f415db30 0x7f16f40b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.814+0000 7f16f899d700 1 --2- 172.21.15.98:0/810245205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16f40cf570 0x7f16f415d5f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.814+0000 7f16f899d700 1 -- 172.21.15.98:0/810245205 >> 172.21.15.98:0/810245205 conn(0x7f16f406b880 msgr2=0x7f16f40cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:41.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.814+0000 7f16f899d700 1 -- 172.21.15.98:0/810245205 shutdown_connections 2023-03-03T07:55:41.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.814+0000 7f16f899d700 1 -- 172.21.15.98:0/810245205 wait complete. 2023-03-03T07:55:41.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.816+0000 7f16f899d700 1 Processor -- start 2023-03-03T07:55:41.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.816+0000 7f16f899d700 1 -- start start 2023-03-03T07:55:41.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.817+0000 7f16f899d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16f40cf570 0x7f16f40cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:41.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.817+0000 7f16f899d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16f415db30 0x7f16f40b7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:41.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.817+0000 7f16f899d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16f41609b0 0x7f16f4150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:41.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.817+0000 7f16f899d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f40b7850 con 0x7f16f41609b0 2023-03-03T07:55:41.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.817+0000 7f16f899d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f40cbcb0 con 0x7f16f415db30 2023-03-03T07:55:41.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.817+0000 7f16f899d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f40cbf90 con 0x7f16f40cf570 2023-03-03T07:55:41.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.818+0000 7f16f3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16f41609b0 0x7f16f4150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:41.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.818+0000 7f16f3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16f41609b0 0x7f16f4150e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47676/0 (socket says 172.21.15.98:47676) 2023-03-03T07:55:41.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.818+0000 7f16f3fff700 1 -- 172.21.15.98:0/3489558987 learned_addr learned my addr 172.21.15.98:0/3489558987 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:41.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.818+0000 7f16f2ffd700 1 --2- 172.21.15.98:0/3489558987 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16f415db30 0x7f16f40b7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:41.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.818+0000 7f16f37fe700 1 --2- 172.21.15.98:0/3489558987 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16f40cf570 0x7f16f40cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:41.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.818+0000 7f16f3fff700 1 -- 172.21.15.98:0/3489558987 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16f40cf570 msgr2=0x7f16f40cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:41.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.818+0000 7f16f3fff700 1 --2- 172.21.15.98:0/3489558987 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16f40cf570 0x7f16f40cb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.818+0000 7f16f3fff700 1 -- 172.21.15.98:0/3489558987 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16f415db30 msgr2=0x7f16f40b7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:41.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.818+0000 7f16f3fff700 1 --2- 172.21.15.98:0/3489558987 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16f415db30 0x7f16f40b7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:41.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.818+0000 7f16f3fff700 1 -- 172.21.15.98:0/3489558987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e8009550 con 0x7f16f41609b0 2023-03-03T07:55:41.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.819+0000 7f16f3fff700 1 --2- 172.21.15.98:0/3489558987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16f41609b0 0x7f16f4150e20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f16ec007e40 tx=0x7f16ec00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:41.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.819+0000 7f16dbfff700 1 -- 172.21.15.98:0/3489558987 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec00ed30 con 0x7f16f41609b0 2023-03-03T07:55:41.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.819+0000 7f16dbfff700 1 -- 172.21.15.98:0/3489558987 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ec014420 con 0x7f16f41609b0 2023-03-03T07:55:41.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.819+0000 7f16dbfff700 1 -- 172.21.15.98:0/3489558987 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec0147d0 con 0x7f16f41609b0 2023-03-03T07:55:41.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.819+0000 7f16f899d700 1 -- 172.21.15.98:0/3489558987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f415d0d0 con 0x7f16f41609b0 2023-03-03T07:55:41.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.819+0000 7f16f899d700 1 -- 172.21.15.98:0/3489558987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f415cdf0 con 0x7f16f41609b0 2023-03-03T07:55:41.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.820+0000 7f16f899d700 1 -- 172.21.15.98:0/3489558987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f40b4fe0 con 0x7f16f41609b0 2023-03-03T07:55:41.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.821+0000 7f16dbfff700 1 -- 172.21.15.98:0/3489558987 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f16ec012050 con 0x7f16f41609b0 2023-03-03T07:55:41.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.822+0000 7f16dbfff700 1 --2- 172.21.15.98:0/3489558987 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16e00682f0 0x7f16e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:41.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.823+0000 7f16dbfff700 1 -- 172.21.15.98:0/3489558987 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f16ec085ff0 con 0x7f16f41609b0 2023-03-03T07:55:41.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.823+0000 7f16f37fe700 1 --2- 172.21.15.98:0/3489558987 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16e00682f0 0x7f16e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:41.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.824+0000 7f16f37fe700 1 --2- 172.21.15.98:0/3489558987 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16e00682f0 0x7f16e006a7a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f16e4007fd0 tx=0x7f16e4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:41.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:41.827+0000 7f16dbfff700 1 -- 172.21.15.98:0/3489558987 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f16ec052a40 con 0x7f16f41609b0 2023-03-03T07:55:42.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.020+0000 7f16f899d700 1 -- 172.21.15.98:0/3489558987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f16f4068290 con 0x7f16f41609b0 2023-03-03T07:55:42.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.021+0000 7f16dbfff700 1 -- 172.21.15.98:0/3489558987 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f16ec056060 con 0x7f16f41609b0 2023-03-03T07:55:42.022 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:55:42.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.023+0000 7f16f899d700 1 -- 172.21.15.98:0/3489558987 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16e00682f0 msgr2=0x7f16e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:42.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.023+0000 7f16f899d700 1 --2- 172.21.15.98:0/3489558987 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16e00682f0 0x7f16e006a7a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f16e4007fd0 tx=0x7f16e4009040 comp rx=0 tx=0).stop 2023-03-03T07:55:42.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.023+0000 7f16f899d700 1 -- 172.21.15.98:0/3489558987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16f41609b0 msgr2=0x7f16f4150e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:42.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.023+0000 7f16f899d700 1 --2- 172.21.15.98:0/3489558987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16f41609b0 0x7f16f4150e20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f16ec007e40 tx=0x7f16ec00e3c0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.023+0000 7f16f899d700 1 -- 172.21.15.98:0/3489558987 shutdown_connections 2023-03-03T07:55:42.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.023+0000 7f16f899d700 1 --2- 172.21.15.98:0/3489558987 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16e00682f0 0x7f16e006a7a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.023+0000 7f16f899d700 1 --2- 172.21.15.98:0/3489558987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16f41609b0 0x7f16f4150e20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.023+0000 7f16f899d700 1 --2- 172.21.15.98:0/3489558987 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16f40cf570 0x7f16f40cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.023+0000 7f16f899d700 1 --2- 172.21.15.98:0/3489558987 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16f415db30 0x7f16f40b7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.023+0000 7f16f899d700 1 -- 172.21.15.98:0/3489558987 >> 172.21.15.98:0/3489558987 conn(0x7f16f406b880 msgr2=0x7f16f40b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:42.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.023+0000 7f16f899d700 1 -- 172.21.15.98:0/3489558987 shutdown_connections 2023-03-03T07:55:42.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.024+0000 7f16f899d700 1 -- 172.21.15.98:0/3489558987 wait complete. 2023-03-03T07:55:42.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:55:42.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.297+0000 7f0b653ff700 1 Processor -- start 2023-03-03T07:55:42.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.297+0000 7f0b653ff700 1 -- start start 2023-03-03T07:55:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.298+0000 7f0b653ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0b600b87c0 0x7f0b600cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.298+0000 7f0b653ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b600b5410 0x7f0b600d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.298+0000 7f0b653ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b600d0ef0 con 0x7f0b600b8b90 2023-03-03T07:55:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.298+0000 7f0b653ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b600d0600 con 0x7f0b600b87c0 2023-03-03T07:55:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.298+0000 7f0b653ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b600cd970 con 0x7f0b600b5410 2023-03-03T07:55:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.299+0000 7f0b5effd700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0b600b87c0 0x7f0b600cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.299+0000 7f0b5e7fc700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f0b600b8b90 0x7f0b600b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:42802/0 (socket says 172.21.15.98:42802) 2023-03-03T07:55:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.299+0000 7f0b5e7fc700 1 -- 172.21.15.98:0/2145545818 learned_addr learned my addr 172.21.15.98:0/2145545818 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.299+0000 7f0b5f7fe700 1 --2- 172.21.15.98:0/2145545818 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b600b5410 0x7f0b600d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.299+0000 7f0b5effd700 1 -- 172.21.15.98:0/2145545818 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b600b5410 msgr2=0x7f0b600d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.299+0000 7f0b5effd700 1 --2- 172.21.15.98:0/2145545818 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b600b5410 0x7f0b600d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.299+0000 7f0b5effd700 1 -- 172.21.15.98:0/2145545818 >> v1:172.21.15.100:6789/0 conn(0x7f0b600b8b90 legacy=0x7f0b600b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.299+0000 7f0b5effd700 1 -- 172.21.15.98:0/2145545818 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b600cdc50 con 0x7f0b600b87c0 2023-03-03T07:55:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.299+0000 7f0b5effd700 1 --2- 172.21.15.98:0/2145545818 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b600b87c0 0x7f0b600cffe0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009840 tx=0x7f0b50009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc503547fa760d2a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.300+0000 7f0b5dffb700 1 -- 172.21.15.98:0/2145545818 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50011830 con 0x7f0b600b87c0 2023-03-03T07:55:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.300+0000 7f0b5dffb700 1 -- 172.21.15.98:0/2145545818 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b50011990 con 0x7f0b600b87c0 2023-03-03T07:55:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.300+0000 7f0b5dffb700 1 -- 172.21.15.98:0/2145545818 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50011d20 con 0x7f0b600b87c0 2023-03-03T07:55:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.300+0000 7f0b653ff700 1 -- 172.21.15.98:0/2145545818 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b600b87c0 msgr2=0x7f0b600cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.300+0000 7f0b653ff700 1 --2- 172.21.15.98:0/2145545818 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b600b87c0 0x7f0b600cffe0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009840 tx=0x7f0b50009b50 comp rx=0 tx=0).stop 2023-03-03T07:55:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.300+0000 7f0b653ff700 1 -- 172.21.15.98:0/2145545818 shutdown_connections 2023-03-03T07:55:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.301+0000 7f0b653ff700 1 --2- 172.21.15.98:0/2145545818 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b600b87c0 0x7f0b600cffe0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.301+0000 7f0b653ff700 1 --2- 172.21.15.98:0/2145545818 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b600b5410 0x7f0b600d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.301+0000 7f0b653ff700 1 -- 172.21.15.98:0/2145545818 >> 172.21.15.98:0/2145545818 conn(0x7f0b6006aee0 msgr2=0x7f0b6006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.301+0000 7f0b653ff700 1 -- 172.21.15.98:0/2145545818 shutdown_connections 2023-03-03T07:55:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.301+0000 7f0b653ff700 1 -- 172.21.15.98:0/2145545818 wait complete. 2023-03-03T07:55:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.303+0000 7f0b653ff700 1 Processor -- start 2023-03-03T07:55:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.303+0000 7f0b653ff700 1 -- start start 2023-03-03T07:55:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.304+0000 7f0b653ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b600b5410 0x7f0b6015b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.304+0000 7f0b653ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b600b87c0 0x7f0b600b60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.304+0000 7f0b653ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b600b8b90 0x7f0b600b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.304+0000 7f0b653ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b601e5600 con 0x7f0b600b8b90 2023-03-03T07:55:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.304+0000 7f0b653ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b601e58e0 con 0x7f0b600b5410 2023-03-03T07:55:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.304+0000 7f0b653ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b601e5bc0 con 0x7f0b600b87c0 2023-03-03T07:55:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.304+0000 7f0b5effd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b600b5410 0x7f0b6015b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.304+0000 7f0b5effd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b600b5410 0x7f0b6015b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:57888/0 (socket says 172.21.15.98:57888) 2023-03-03T07:55:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.304+0000 7f0b5f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b600b8b90 0x7f0b600b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.304+0000 7f0b5e7fc700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b600b87c0 0x7f0b600b60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.304+0000 7f0b5effd700 1 -- 172.21.15.98:0/2870440896 learned_addr learned my addr 172.21.15.98:0/2870440896 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.305+0000 7f0b5effd700 1 -- 172.21.15.98:0/2870440896 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b600b87c0 msgr2=0x7f0b600b60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.305+0000 7f0b5effd700 1 --2- 172.21.15.98:0/2870440896 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b600b87c0 0x7f0b600b60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.305+0000 7f0b5effd700 1 -- 172.21.15.98:0/2870440896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b600b8b90 msgr2=0x7f0b600b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.305+0000 7f0b5effd700 1 --2- 172.21.15.98:0/2870440896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b600b8b90 0x7f0b600b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.305+0000 7f0b5effd700 1 -- 172.21.15.98:0/2870440896 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b500094a0 con 0x7f0b600b5410 2023-03-03T07:55:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.305+0000 7f0b5effd700 1 --2- 172.21.15.98:0/2870440896 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b600b5410 0x7f0b6015b6f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009fd0 tx=0x7f0b50004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.305+0000 7f0b477fe700 1 -- 172.21.15.98:0/2870440896 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b500036a0 con 0x7f0b600b5410 2023-03-03T07:55:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.306+0000 7f0b477fe700 1 -- 172.21.15.98:0/2870440896 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5001f070 con 0x7f0b600b5410 2023-03-03T07:55:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.306+0000 7f0b653ff700 1 -- 172.21.15.98:0/2870440896 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b600cdf30 con 0x7f0b600b5410 2023-03-03T07:55:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.306+0000 7f0b477fe700 1 -- 172.21.15.98:0/2870440896 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50003a60 con 0x7f0b600b5410 2023-03-03T07:55:42.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.306+0000 7f0b653ff700 1 -- 172.21.15.98:0/2870440896 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b60073a80 con 0x7f0b600b5410 2023-03-03T07:55:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.307+0000 7f0b653ff700 1 -- 172.21.15.98:0/2870440896 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b600ce0c0 con 0x7f0b600b5410 2023-03-03T07:55:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.309+0000 7f0b477fe700 1 -- 172.21.15.98:0/2870440896 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0b50048070 con 0x7f0b600b5410 2023-03-03T07:55:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.310+0000 7f0b477fe700 1 --2- 172.21.15.98:0/2870440896 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0b4c0682f0 0x7f0b4c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.310+0000 7f0b477fe700 1 -- 172.21.15.98:0/2870440896 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0b50021070 con 0x7f0b600b5410 2023-03-03T07:55:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.310+0000 7f0b5e7fc700 1 --2- 172.21.15.98:0/2870440896 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0b4c0682f0 0x7f0b4c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.311+0000 7f0b5e7fc700 1 --2- 172.21.15.98:0/2870440896 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0b4c0682f0 0x7f0b4c06a7a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0b6015c490 tx=0x7f0b54009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:42.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.315+0000 7f0b477fe700 1 -- 172.21.15.98:0/2870440896 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0b500548e0 con 0x7f0b600b5410 2023-03-03T07:55:42.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.551+0000 7f0b653ff700 1 -- 172.21.15.98:0/2870440896 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b600729b0 con 0x7f0b600b5410 2023-03-03T07:55:42.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.552+0000 7f0b477fe700 1 -- 172.21.15.98:0/2870440896 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3521 (secure 0 0 0) 0x7f0b5002bde0 con 0x7f0b600b5410 2023-03-03T07:55:42.554 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:42.554 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/246835374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":246835374},{"type":"v1","addr":"172.21.15.98:6836","nonce":246835374}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6833/959944231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":959944231},{"type":"v1","addr":"172.21.15.100:6833","nonce":959944231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/872788798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":872788798},{"type":"v1","addr":"172.21.15.100:6835","nonce":872788798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3969084152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3969084152},{"type":"v1","addr":"172.21.15.98:6837","nonce":3969084152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-03-03T07:55:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.555+0000 7f0b653ff700 1 -- 172.21.15.98:0/2870440896 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0b4c0682f0 msgr2=0x7f0b4c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.555+0000 7f0b653ff700 1 --2- 172.21.15.98:0/2870440896 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0b4c0682f0 0x7f0b4c06a7a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0b6015c490 tx=0x7f0b54009040 comp rx=0 tx=0).stop 2023-03-03T07:55:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.555+0000 7f0b653ff700 1 -- 172.21.15.98:0/2870440896 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b600b5410 msgr2=0x7f0b6015b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.555+0000 7f0b653ff700 1 --2- 172.21.15.98:0/2870440896 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b600b5410 0x7f0b6015b6f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009fd0 tx=0x7f0b50004750 comp rx=0 tx=0).stop 2023-03-03T07:55:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.555+0000 7f0b653ff700 1 -- 172.21.15.98:0/2870440896 shutdown_connections 2023-03-03T07:55:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.555+0000 7f0b653ff700 1 --2- 172.21.15.98:0/2870440896 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0b4c0682f0 0x7f0b4c06a7a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.555+0000 7f0b653ff700 1 --2- 172.21.15.98:0/2870440896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b600b8b90 0x7f0b600b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.555+0000 7f0b653ff700 1 --2- 172.21.15.98:0/2870440896 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b600b5410 0x7f0b6015b6f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.555+0000 7f0b653ff700 1 --2- 172.21.15.98:0/2870440896 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b600b87c0 0x7f0b600b60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.556+0000 7f0b653ff700 1 -- 172.21.15.98:0/2870440896 >> 172.21.15.98:0/2870440896 conn(0x7f0b6006aee0 msgr2=0x7f0b600b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.556+0000 7f0b653ff700 1 -- 172.21.15.98:0/2870440896 shutdown_connections 2023-03-03T07:55:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.556+0000 7f0b653ff700 1 -- 172.21.15.98:0/2870440896 wait complete. 2023-03-03T07:55:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 3 2023-03-03T07:55:42.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:55:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.828+0000 7f7016b9d700 1 Processor -- start 2023-03-03T07:55:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.828+0000 7f7016b9d700 1 -- start start 2023-03-03T07:55:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.830+0000 7f7016b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7010146360 0x7f7010160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.830+0000 7f7016b9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f70100b4ab0 0x7f70100b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.830+0000 7f7016b9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70100731e0 0x7f7010073660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.830+0000 7f7016b9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70100b5460 con 0x7f70100731e0 2023-03-03T07:55:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.830+0000 7f7016b9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70101614c0 con 0x7f70100b4ab0 2023-03-03T07:55:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.830+0000 7f7016b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70100d10f0 con 0x7f7010146360 2023-03-03T07:55:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.830+0000 7f701639c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70100731e0 0x7f7010073660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.830+0000 7f701639c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70100731e0 0x7f7010073660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47686/0 (socket says 172.21.15.98:47686) 2023-03-03T07:55:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.830+0000 7f701639c700 1 -- 172.21.15.98:0/3158661157 learned_addr learned my addr 172.21.15.98:0/3158661157 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:42.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.830+0000 7f7015b9b700 1 --2- 172.21.15.98:0/3158661157 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7010146360 0x7f7010160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:42.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.830+0000 7f701539a700 1 --2- 172.21.15.98:0/3158661157 >> v2:172.21.15.100:3300/0 conn(0x7f70100b4ab0 0x7f70100b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.831+0000 7f701639c700 1 -- 172.21.15.98:0/3158661157 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7010146360 msgr2=0x7f7010160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.831+0000 7f701639c700 1 --2- 172.21.15.98:0/3158661157 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7010146360 0x7f7010160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.831+0000 7f701639c700 1 -- 172.21.15.98:0/3158661157 >> v2:172.21.15.100:3300/0 conn(0x7f70100b4ab0 msgr2=0x7f70100b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.831+0000 7f701639c700 1 --2- 172.21.15.98:0/3158661157 >> v2:172.21.15.100:3300/0 conn(0x7f70100b4ab0 0x7f70100b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.831+0000 7f701639c700 1 -- 172.21.15.98:0/3158661157 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70100cdf20 con 0x7f70100731e0 2023-03-03T07:55:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.831+0000 7f701539a700 1 --2- 172.21.15.98:0/3158661157 >> v2:172.21.15.100:3300/0 conn(0x7f70100b4ab0 0x7f70100b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.831+0000 7f701639c700 1 --2- 172.21.15.98:0/3158661157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70100731e0 0x7f7010073660 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7010072a40 tx=0x7f700800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d64bea49cad903b2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.831+0000 7f7014b99700 1 -- 172.21.15.98:0/3158661157 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700800e070 con 0x7f70100731e0 2023-03-03T07:55:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.831+0000 7f7014b99700 1 -- 172.21.15.98:0/3158661157 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700800b910 con 0x7f70100731e0 2023-03-03T07:55:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.831+0000 7f7014b99700 1 -- 172.21.15.98:0/3158661157 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700800bcc0 con 0x7f70100731e0 2023-03-03T07:55:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.832+0000 7f7016b9d700 1 -- 172.21.15.98:0/3158661157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70100731e0 msgr2=0x7f7010073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.832+0000 7f7016b9d700 1 --2- 172.21.15.98:0/3158661157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70100731e0 0x7f7010073660 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7010072a40 tx=0x7f700800b330 comp rx=0 tx=0).stop 2023-03-03T07:55:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.832+0000 7f7016b9d700 1 -- 172.21.15.98:0/3158661157 shutdown_connections 2023-03-03T07:55:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.832+0000 7f7016b9d700 1 --2- 172.21.15.98:0/3158661157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70100731e0 0x7f7010073660 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.832+0000 7f7016b9d700 1 --2- 172.21.15.98:0/3158661157 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7010146360 0x7f7010160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.832+0000 7f7016b9d700 1 --2- 172.21.15.98:0/3158661157 >> v2:172.21.15.100:3300/0 conn(0x7f70100b4ab0 0x7f70100b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.832+0000 7f7016b9d700 1 -- 172.21.15.98:0/3158661157 >> 172.21.15.98:0/3158661157 conn(0x7f701006b8e0 msgr2=0x7f70100cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.832+0000 7f7016b9d700 1 -- 172.21.15.98:0/3158661157 shutdown_connections 2023-03-03T07:55:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.832+0000 7f7016b9d700 1 -- 172.21.15.98:0/3158661157 wait complete. 2023-03-03T07:55:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.834+0000 7f7016b9d700 1 Processor -- start 2023-03-03T07:55:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.834+0000 7f7016b9d700 1 -- start start 2023-03-03T07:55:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.835+0000 7f7016b9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70100731e0 0x7f7010154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.835+0000 7f7016b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70100b4ab0 0x7f70100c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.835+0000 7f7016b9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7010146360 0x7f70100c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.835+0000 7f7016b9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70100c6130 con 0x7f70100731e0 2023-03-03T07:55:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.835+0000 7f7016b9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70100c9510 con 0x7f7010146360 2023-03-03T07:55:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.835+0000 7f7016b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70100c97f0 con 0x7f70100b4ab0 2023-03-03T07:55:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.836+0000 7f7015b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70100731e0 0x7f7010154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.836+0000 7f7015b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70100731e0 0x7f7010154590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47702/0 (socket says 172.21.15.98:47702) 2023-03-03T07:55:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.836+0000 7f7015b9b700 1 -- 172.21.15.98:0/4189062375 learned_addr learned my addr 172.21.15.98:0/4189062375 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.836+0000 7f7015b9b700 1 -- 172.21.15.98:0/4189062375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70100b4ab0 msgr2=0x7f70100c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.836+0000 7f701539a700 1 --2- 172.21.15.98:0/4189062375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70100b4ab0 0x7f70100c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.836+0000 7f7015b9b700 1 --2- 172.21.15.98:0/4189062375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70100b4ab0 0x7f70100c5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.836+0000 7f7015b9b700 1 -- 172.21.15.98:0/4189062375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7010146360 msgr2=0x7f70100c57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.836+0000 7f701639c700 1 --2- 172.21.15.98:0/4189062375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7010146360 0x7f70100c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.836+0000 7f7015b9b700 1 --2- 172.21.15.98:0/4189062375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7010146360 0x7f70100c57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.836+0000 7f7015b9b700 1 -- 172.21.15.98:0/4189062375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7008009b30 con 0x7f70100731e0 2023-03-03T07:55:42.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.836+0000 7f701639c700 1 --2- 172.21.15.98:0/4189062375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7010146360 0x7f70100c57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:55:42.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.836+0000 7f7015b9b700 1 --2- 172.21.15.98:0/4189062375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70100731e0 0x7f7010154590 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f700c00e9b0 tx=0x7f700c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:42.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.837+0000 7f70067fc700 1 -- 172.21.15.98:0/4189062375 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c009710 con 0x7f70100731e0 2023-03-03T07:55:42.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.837+0000 7f70067fc700 1 -- 172.21.15.98:0/4189062375 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c00c850 con 0x7f70100731e0 2023-03-03T07:55:42.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.837+0000 7f70067fc700 1 -- 172.21.15.98:0/4189062375 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c00cbc0 con 0x7f70100731e0 2023-03-03T07:55:42.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.837+0000 7f7016b9d700 1 -- 172.21.15.98:0/4189062375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70100cd960 con 0x7f70100731e0 2023-03-03T07:55:42.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.837+0000 7f7016b9d700 1 -- 172.21.15.98:0/4189062375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70100cd680 con 0x7f70100731e0 2023-03-03T07:55:42.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.838+0000 7f7016b9d700 1 -- 172.21.15.98:0/4189062375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70100cdc40 con 0x7f70100731e0 2023-03-03T07:55:42.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.839+0000 7f70067fc700 1 -- 172.21.15.98:0/4189062375 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f700c019030 con 0x7f70100731e0 2023-03-03T07:55:42.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.841+0000 7f70067fc700 1 --2- 172.21.15.98:0/4189062375 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6ff40682f0 0x7f6ff406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:42.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.841+0000 7f70067fc700 1 -- 172.21.15.98:0/4189062375 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f700c0859a0 con 0x7f70100731e0 2023-03-03T07:55:42.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.841+0000 7f701539a700 1 --2- 172.21.15.98:0/4189062375 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6ff40682f0 0x7f6ff406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:42.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.842+0000 7f701539a700 1 --2- 172.21.15.98:0/4189062375 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6ff40682f0 0x7f6ff406a7a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f70100b87c0 tx=0x7f7000008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:42.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:42.846+0000 7f70067fc700 1 -- 172.21.15.98:0/4189062375 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f700c0523d0 con 0x7f70100731e0 2023-03-03T07:55:43.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.046+0000 7f7016b9d700 1 -- 172.21.15.98:0/4189062375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70100d0420 con 0x7f70100731e0 2023-03-03T07:55:43.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.047+0000 7f70067fc700 1 -- 172.21.15.98:0/4189062375 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+14264 (secure 0 0 0) 0x7f700c021090 con 0x7f70100731e0 2023-03-03T07:55:43.049 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:43.049 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":20,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:55:42.497504+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:55:43.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.050+0000 7f7016b9d700 1 -- 172.21.15.98:0/4189062375 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6ff40682f0 msgr2=0x7f6ff406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:43.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.050+0000 7f7016b9d700 1 --2- 172.21.15.98:0/4189062375 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6ff40682f0 0x7f6ff406a7a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f70100b87c0 tx=0x7f7000008040 comp rx=0 tx=0).stop 2023-03-03T07:55:43.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.050+0000 7f7016b9d700 1 -- 172.21.15.98:0/4189062375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70100731e0 msgr2=0x7f7010154590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.050+0000 7f7016b9d700 1 --2- 172.21.15.98:0/4189062375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70100731e0 0x7f7010154590 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f700c00e9b0 tx=0x7f700c00ed70 comp rx=0 tx=0).stop 2023-03-03T07:55:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.050+0000 7f7016b9d700 1 -- 172.21.15.98:0/4189062375 shutdown_connections 2023-03-03T07:55:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.050+0000 7f7016b9d700 1 --2- 172.21.15.98:0/4189062375 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6ff40682f0 0x7f6ff406a7a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.050+0000 7f7016b9d700 1 --2- 172.21.15.98:0/4189062375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70100731e0 0x7f7010154590 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.050+0000 7f7016b9d700 1 --2- 172.21.15.98:0/4189062375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7010146360 0x7f70100c57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.050+0000 7f7016b9d700 1 --2- 172.21.15.98:0/4189062375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70100b4ab0 0x7f70100c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.050+0000 7f7016b9d700 1 -- 172.21.15.98:0/4189062375 >> 172.21.15.98:0/4189062375 conn(0x7f701006b8e0 msgr2=0x7f70100735b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:43.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.050+0000 7f7016b9d700 1 -- 172.21.15.98:0/4189062375 shutdown_connections 2023-03-03T07:55:43.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.050+0000 7f7016b9d700 1 -- 172.21.15.98:0/4189062375 wait complete. 2023-03-03T07:55:43.078 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:55:43.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:55:43.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.323+0000 7ffa907ff700 1 Processor -- start 2023-03-03T07:55:43.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.324+0000 7ffa907ff700 1 -- start start 2023-03-03T07:55:43.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.325+0000 7ffa907ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ffa880b8d10 0x7ffa880cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:43.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.325+0000 7ffa907ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa880b5310 0x7ffa880d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:43.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.325+0000 7ffa907ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa880cd7d0 con 0x7ffa880b26d0 2023-03-03T07:55:43.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.325+0000 7ffa907ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa880b90e0 con 0x7ffa880b8d10 2023-03-03T07:55:43.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.325+0000 7ffa907ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa880cda20 con 0x7ffa880b5310 2023-03-03T07:55:43.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.325+0000 7ffa8dd9a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ffa880b26d0 0x7ffa880b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46968/0 (socket says 172.21.15.98:46968) 2023-03-03T07:55:43.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.325+0000 7ffa8dd9a700 1 -- 172.21.15.98:0/3755191994 learned_addr learned my addr 172.21.15.98:0/3755191994 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:43.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8e59b700 1 --2- 172.21.15.98:0/3755191994 >> v2:172.21.15.100:3300/0 conn(0x7ffa880b8d10 0x7ffa880cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:43.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8ed9c700 1 --2- 172.21.15.98:0/3755191994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa880b5310 0x7ffa880d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:43.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 -- 172.21.15.98:0/3755191994 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609613189 0 0) 0x7ffa880cd7d0 con 0x7ffa880b26d0 2023-03-03T07:55:43.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 -- 172.21.15.98:0/3755191994 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa740035b0 con 0x7ffa880b26d0 2023-03-03T07:55:43.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 -- 172.21.15.98:0/3755191994 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 621060926 0 0) 0x7ffa740035b0 con 0x7ffa880b26d0 2023-03-03T07:55:43.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 -- 172.21.15.98:0/3755191994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa880b5310 msgr2=0x7ffa880d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:43.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 --2- 172.21.15.98:0/3755191994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa880b5310 0x7ffa880d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 -- 172.21.15.98:0/3755191994 >> v2:172.21.15.100:3300/0 conn(0x7ffa880b8d10 msgr2=0x7ffa880cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:43.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 --2- 172.21.15.98:0/3755191994 >> v2:172.21.15.100:3300/0 conn(0x7ffa880b8d10 0x7ffa880cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 -- 172.21.15.98:0/3755191994 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa880cdd00 con 0x7ffa880b26d0 2023-03-03T07:55:43.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 -- 172.21.15.98:0/3755191994 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7ffa78001670 con 0x7ffa880b26d0 2023-03-03T07:55:43.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 -- 172.21.15.98:0/3755191994 >> v1:172.21.15.98:6789/0 conn(0x7ffa880b26d0 legacy=0x7ffa880b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:43.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 --2- 172.21.15.98:0/3755191994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa74003a00 0x7ffa74003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:43.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 --2- 172.21.15.98:0/3755191994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa880b5310 0x7ffa74004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:43.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 --2- 172.21.15.98:0/3755191994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa74004d90 0x7ffa74005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:43.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 -- 172.21.15.98:0/3755191994 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa880b90e0 con 0x7ffa74003a00 2023-03-03T07:55:43.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 -- 172.21.15.98:0/3755191994 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa880cda20 con 0x7ffa74004d90 2023-03-03T07:55:43.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.326+0000 7ffa8d599700 1 -- 172.21.15.98:0/3755191994 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7400ba60 con 0x7ffa880b5310 2023-03-03T07:55:43.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.327+0000 7ffa8e59b700 1 --2- 172.21.15.98:0/3755191994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa74003a00 0x7ffa74003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:43.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.327+0000 7ffa8dd9a700 1 --2- 172.21.15.98:0/3755191994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa880b5310 0x7ffa74004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:43.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.327+0000 7ffa8ed9c700 1 --2- 172.21.15.98:0/3755191994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa74004d90 0x7ffa74005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:43.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.327+0000 7ffa8e59b700 1 -- 172.21.15.98:0/3755191994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa880b5310 msgr2=0x7ffa74004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:43.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.327+0000 7ffa8e59b700 1 --2- 172.21.15.98:0/3755191994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa880b5310 0x7ffa74004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.327+0000 7ffa8e59b700 1 -- 172.21.15.98:0/3755191994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa74004d90 msgr2=0x7ffa74005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:43.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.327+0000 7ffa8e59b700 1 --2- 172.21.15.98:0/3755191994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa74004d90 0x7ffa74005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.327+0000 7ffa8e59b700 1 -- 172.21.15.98:0/3755191994 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa740035b0 con 0x7ffa74003a00 2023-03-03T07:55:43.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.328+0000 7ffa8e59b700 1 --2- 172.21.15.98:0/3755191994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa74003a00 0x7ffa74003e10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffa84009b60 tx=0x7ffa74005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b71a0cd942349f2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:43.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.328+0000 7ffa8d599700 1 -- 172.21.15.98:0/3755191994 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8400e6c0 con 0x7ffa74003a00 2023-03-03T07:55:43.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.328+0000 7ffa8d599700 1 -- 172.21.15.98:0/3755191994 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa8400e820 con 0x7ffa74003a00 2023-03-03T07:55:43.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.328+0000 7ffa8d599700 1 -- 172.21.15.98:0/3755191994 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8400eb90 con 0x7ffa74003a00 2023-03-03T07:55:43.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.328+0000 7ffa907ff700 1 -- 172.21.15.98:0/3755191994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa74003a00 msgr2=0x7ffa74003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.328+0000 7ffa907ff700 1 --2- 172.21.15.98:0/3755191994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa74003a00 0x7ffa74003e10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffa84009b60 tx=0x7ffa74005880 comp rx=0 tx=0).stop 2023-03-03T07:55:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.328+0000 7ffa8ed9c700 1 -- 172.21.15.98:0/3755191994 reap_dead start 2023-03-03T07:55:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.328+0000 7ffa907ff700 1 -- 172.21.15.98:0/3755191994 shutdown_connections 2023-03-03T07:55:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.328+0000 7ffa907ff700 1 -- 172.21.15.98:0/3755191994 >> 172.21.15.98:0/3755191994 conn(0x7ffa88073e70 msgr2=0x7ffa8806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.329+0000 7ffa907ff700 1 -- 172.21.15.98:0/3755191994 shutdown_connections 2023-03-03T07:55:43.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.329+0000 7ffa907ff700 1 -- 172.21.15.98:0/3755191994 wait complete. 2023-03-03T07:55:43.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.331+0000 7ffa907ff700 1 Processor -- start 2023-03-03T07:55:43.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.331+0000 7ffa907ff700 1 -- start start 2023-03-03T07:55:43.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.332+0000 7ffa907ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa74003a00 0x7ffa8815f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:43.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.332+0000 7ffa907ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa88151440 0x7ffa880b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.332+0000 7ffa907ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa880b4170 0x7ffa881f22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.332+0000 7ffa907ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa880b45f0 con 0x7ffa74003a00 2023-03-03T07:55:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.332+0000 7ffa907ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa881518b0 con 0x7ffa880b4170 2023-03-03T07:55:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.332+0000 7ffa907ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa88153810 con 0x7ffa88151440 2023-03-03T07:55:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.332+0000 7ffa8e59b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa74003a00 0x7ffa8815f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.332+0000 7ffa8e59b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa74003a00 0x7ffa8815f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47710/0 (socket says 172.21.15.98:47710) 2023-03-03T07:55:43.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.332+0000 7ffa8e59b700 1 -- 172.21.15.98:0/2665761708 learned_addr learned my addr 172.21.15.98:0/2665761708 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:43.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.332+0000 7ffa8dd9a700 1 --2- 172.21.15.98:0/2665761708 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa88151440 0x7ffa880b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:43.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.333+0000 7ffa8e59b700 1 -- 172.21.15.98:0/2665761708 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa88151440 msgr2=0x7ffa880b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:43.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.333+0000 7ffa8e59b700 1 --2- 172.21.15.98:0/2665761708 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa88151440 0x7ffa880b3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.333+0000 7ffa8ed9c700 1 --2- 172.21.15.98:0/2665761708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa880b4170 0x7ffa881f22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:43.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.333+0000 7ffa8e59b700 1 -- 172.21.15.98:0/2665761708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa880b4170 msgr2=0x7ffa881f22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:43.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.333+0000 7ffa8e59b700 1 --2- 172.21.15.98:0/2665761708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa880b4170 0x7ffa881f22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.333+0000 7ffa8e59b700 1 -- 172.21.15.98:0/2665761708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa880cdfe0 con 0x7ffa74003a00 2023-03-03T07:55:43.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.333+0000 7ffa8ed9c700 1 --2- 172.21.15.98:0/2665761708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa880b4170 0x7ffa881f22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:55:43.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.333+0000 7ffa8e59b700 1 --2- 172.21.15.98:0/2665761708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa74003a00 0x7ffa8815f7e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffa840090b0 tx=0x7ffa840047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:43.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.333+0000 7ffa7effd700 1 -- 172.21.15.98:0/2665761708 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa84021070 con 0x7ffa74003a00 2023-03-03T07:55:43.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.333+0000 7ffa7effd700 1 -- 172.21.15.98:0/2665761708 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa84004590 con 0x7ffa74003a00 2023-03-03T07:55:43.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.333+0000 7ffa7effd700 1 -- 172.21.15.98:0/2665761708 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8400e6c0 con 0x7ffa74003a00 2023-03-03T07:55:43.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.333+0000 7ffa907ff700 1 -- 172.21.15.98:0/2665761708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa880d0e80 con 0x7ffa74003a00 2023-03-03T07:55:43.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.334+0000 7ffa907ff700 1 -- 172.21.15.98:0/2665761708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa880d0500 con 0x7ffa74003a00 2023-03-03T07:55:43.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.335+0000 7ffa907ff700 1 -- 172.21.15.98:0/2665761708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa880b4570 con 0x7ffa74003a00 2023-03-03T07:55:43.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.336+0000 7ffa7effd700 1 -- 172.21.15.98:0/2665761708 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffa8402e030 con 0x7ffa74003a00 2023-03-03T07:55:43.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.338+0000 7ffa7effd700 1 --2- 172.21.15.98:0/2665761708 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa6c0682f0 0x7ffa6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:43.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.338+0000 7ffa7effd700 1 -- 172.21.15.98:0/2665761708 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ffa84086d10 con 0x7ffa74003a00 2023-03-03T07:55:43.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.338+0000 7ffa8dd9a700 1 --2- 172.21.15.98:0/2665761708 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa6c0682f0 0x7ffa6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:43.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.338+0000 7ffa8dd9a700 1 --2- 172.21.15.98:0/2665761708 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa6c0682f0 0x7ffa6c06a7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffa78000fb0 tx=0x7ffa78009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:43.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.342+0000 7ffa7effd700 1 -- 172.21.15.98:0/2665761708 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffa8408c020 con 0x7ffa74003a00 2023-03-03T07:55:43.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.534+0000 7ffa907ff700 1 -- 172.21.15.98:0/2665761708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffa8804e6e0 con 0x7ffa74003a00 2023-03-03T07:55:43.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.535+0000 7ffa7effd700 1 -- 172.21.15.98:0/2665761708 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffa840537d0 con 0x7ffa74003a00 2023-03-03T07:55:43.536 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:55:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.537+0000 7ffa907ff700 1 -- 172.21.15.98:0/2665761708 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa6c0682f0 msgr2=0x7ffa6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.537+0000 7ffa907ff700 1 --2- 172.21.15.98:0/2665761708 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa6c0682f0 0x7ffa6c06a7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffa78000fb0 tx=0x7ffa78009040 comp rx=0 tx=0).stop 2023-03-03T07:55:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.537+0000 7ffa907ff700 1 -- 172.21.15.98:0/2665761708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa74003a00 msgr2=0x7ffa8815f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.537+0000 7ffa907ff700 1 --2- 172.21.15.98:0/2665761708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa74003a00 0x7ffa8815f7e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffa840090b0 tx=0x7ffa840047c0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.537+0000 7ffa907ff700 1 -- 172.21.15.98:0/2665761708 shutdown_connections 2023-03-03T07:55:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.537+0000 7ffa907ff700 1 --2- 172.21.15.98:0/2665761708 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa6c0682f0 0x7ffa6c06a7a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.537+0000 7ffa907ff700 1 --2- 172.21.15.98:0/2665761708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa74003a00 0x7ffa8815f7e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.537+0000 7ffa907ff700 1 --2- 172.21.15.98:0/2665761708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa880b4170 0x7ffa881f22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.537+0000 7ffa907ff700 1 --2- 172.21.15.98:0/2665761708 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa88151440 0x7ffa880b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.537+0000 7ffa907ff700 1 -- 172.21.15.98:0/2665761708 >> 172.21.15.98:0/2665761708 conn(0x7ffa88073e70 msgr2=0x7ffa880cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.538+0000 7ffa907ff700 1 -- 172.21.15.98:0/2665761708 shutdown_connections 2023-03-03T07:55:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.538+0000 7ffa907ff700 1 -- 172.21.15.98:0/2665761708 wait complete. 2023-03-03T07:55:43.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:55:43.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.813+0000 7f39f1bff700 1 Processor -- start 2023-03-03T07:55:43.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.813+0000 7f39f1bff700 1 -- start start 2023-03-03T07:55:43.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.814+0000 7f39f1bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f39ec0b5310 0x7f39ec167850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:43.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.814+0000 7f39f1bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ec0cd7d0 con 0x7f39ec0b8d10 2023-03-03T07:55:43.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.814+0000 7f39f1bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ec0b90e0 con 0x7f39ec0b26d0 2023-03-03T07:55:43.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.814+0000 7f39f1bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39ec0cda20 con 0x7f39ec0b5310 2023-03-03T07:55:43.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.815+0000 7f39eaffd700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f39ec0b26d0 0x7f39ec0b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:42814/0 (socket says 172.21.15.98:42814) 2023-03-03T07:55:43.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.815+0000 7f39eaffd700 1 -- 172.21.15.98:0/3267881082 learned_addr learned my addr 172.21.15.98:0/3267881082 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:43.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.815+0000 7f39eb7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f39ec0b8d10 0x7f39ec0cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46972/0 (socket says 172.21.15.98:46972) 2023-03-03T07:55:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.815+0000 7f39ebfff700 1 --2- 172.21.15.98:0/3267881082 >> v2:172.21.15.98:3300/0 conn(0x7f39ec0b5310 0x7f39ec167850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.815+0000 7f39ea7fc700 1 -- 172.21.15.98:0/3267881082 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3661196587 0 0) 0x7f39ec0cd7d0 con 0x7f39ec0b8d10 2023-03-03T07:55:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.815+0000 7f39ebfff700 1 -- 172.21.15.98:0/3267881082 >> v1:172.21.15.100:6789/0 conn(0x7f39ec0b26d0 legacy=0x7f39ec0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.815+0000 7f39ebfff700 1 -- 172.21.15.98:0/3267881082 >> v1:172.21.15.98:6789/0 conn(0x7f39ec0b8d10 legacy=0x7f39ec0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.815+0000 7f39ebfff700 1 -- 172.21.15.98:0/3267881082 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ec0cdd00 con 0x7f39ec0b5310 2023-03-03T07:55:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.815+0000 7f39ebfff700 1 --2- 172.21.15.98:0/3267881082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39ec0b5310 0x7f39ec167850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f39e4009d90 tx=0x7f39e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=130732f1af26a37a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.816+0000 7f39ea7fc700 1 -- 172.21.15.98:0/3267881082 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e400e070 con 0x7f39ec0b5310 2023-03-03T07:55:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.816+0000 7f39ea7fc700 1 -- 172.21.15.98:0/3267881082 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e400bad0 con 0x7f39ec0b5310 2023-03-03T07:55:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.816+0000 7f39ea7fc700 1 -- 172.21.15.98:0/3267881082 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e400be80 con 0x7f39ec0b5310 2023-03-03T07:55:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.816+0000 7f39f1bff700 1 -- 172.21.15.98:0/3267881082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39ec0b5310 msgr2=0x7f39ec167850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.816+0000 7f39f1bff700 1 --2- 172.21.15.98:0/3267881082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39ec0b5310 0x7f39ec167850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f39e4009d90 tx=0x7f39e400b330 comp rx=0 tx=0).stop 2023-03-03T07:55:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.816+0000 7f39f1bff700 1 -- 172.21.15.98:0/3267881082 shutdown_connections 2023-03-03T07:55:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.816+0000 7f39f1bff700 1 --2- 172.21.15.98:0/3267881082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39ec0b5310 0x7f39ec167850 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.816+0000 7f39f1bff700 1 -- 172.21.15.98:0/3267881082 >> 172.21.15.98:0/3267881082 conn(0x7f39ec073e70 msgr2=0x7f39ec06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.816+0000 7f39f1bff700 1 -- 172.21.15.98:0/3267881082 shutdown_connections 2023-03-03T07:55:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.816+0000 7f39f1bff700 1 -- 172.21.15.98:0/3267881082 wait complete. 2023-03-03T07:55:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.818+0000 7f39f1bff700 1 Processor -- start 2023-03-03T07:55:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.819+0000 7f39f1bff700 1 -- start start 2023-03-03T07:55:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.819+0000 7f39f1bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39ec0b26d0 0x7f39ec157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39f1bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f39ec0b5310 0x7f39ec0c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39f1bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39ec0b8d10 0x7f39ec0b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39f1bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec0cd3b0 con 0x7f39ec0b26d0 2023-03-03T07:55:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39f1bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec0cd690 con 0x7f39ec0b5310 2023-03-03T07:55:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39f1bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec1573d0 con 0x7f39ec0b8d10 2023-03-03T07:55:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39eb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39ec0b26d0 0x7f39ec157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39eb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39ec0b26d0 0x7f39ec157eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47730/0 (socket says 172.21.15.98:47730) 2023-03-03T07:55:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39eb7fe700 1 -- 172.21.15.98:0/118115841 learned_addr learned my addr 172.21.15.98:0/118115841 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39eaffd700 1 --2- 172.21.15.98:0/118115841 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f39ec0b5310 0x7f39ec0c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39ebfff700 1 --2- 172.21.15.98:0/118115841 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39ec0b8d10 0x7f39ec0b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39eb7fe700 1 -- 172.21.15.98:0/118115841 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39ec0b8d10 msgr2=0x7f39ec0b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39eb7fe700 1 --2- 172.21.15.98:0/118115841 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39ec0b8d10 0x7f39ec0b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39eb7fe700 1 -- 172.21.15.98:0/118115841 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f39ec0b5310 msgr2=0x7f39ec0c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39eb7fe700 1 --2- 172.21.15.98:0/118115841 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f39ec0b5310 0x7f39ec0c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39eb7fe700 1 -- 172.21.15.98:0/118115841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e4009a70 con 0x7f39ec0b26d0 2023-03-03T07:55:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.820+0000 7f39ebfff700 1 --2- 172.21.15.98:0/118115841 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39ec0b8d10 0x7f39ec0b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.821+0000 7f39eb7fe700 1 --2- 172.21.15.98:0/118115841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39ec0b26d0 0x7f39ec157eb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00c550 tx=0x7f39dc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:43.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.821+0000 7f39d3fff700 1 -- 172.21.15.98:0/118115841 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc004c80 con 0x7f39ec0b26d0 2023-03-03T07:55:43.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.821+0000 7f39d3fff700 1 -- 172.21.15.98:0/118115841 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39dc004e00 con 0x7f39ec0b26d0 2023-03-03T07:55:43.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.821+0000 7f39d3fff700 1 -- 172.21.15.98:0/118115841 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc007310 con 0x7f39ec0b26d0 2023-03-03T07:55:43.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.821+0000 7f39f1bff700 1 -- 172.21.15.98:0/118115841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39ec0cdfe0 con 0x7f39ec0b26d0 2023-03-03T07:55:43.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.821+0000 7f39f1bff700 1 -- 172.21.15.98:0/118115841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39ec0d0e80 con 0x7f39ec0b26d0 2023-03-03T07:55:43.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.822+0000 7f39f1bff700 1 -- 172.21.15.98:0/118115841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39ec0b5fd0 con 0x7f39ec0b26d0 2023-03-03T07:55:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.823+0000 7f39d3fff700 1 -- 172.21.15.98:0/118115841 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f39dc01a020 con 0x7f39ec0b26d0 2023-03-03T07:55:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.824+0000 7f39d3fff700 1 --2- 172.21.15.98:0/118115841 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39d80682f0 0x7f39d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.824+0000 7f39d3fff700 1 -- 172.21.15.98:0/118115841 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f39dc085570 con 0x7f39ec0b26d0 2023-03-03T07:55:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.825+0000 7f39eaffd700 1 --2- 172.21.15.98:0/118115841 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39d80682f0 0x7f39d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.825+0000 7f39eaffd700 1 --2- 172.21.15.98:0/118115841 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39d80682f0 0x7f39d806a7a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f39e00012e0 tx=0x7f39e0001270 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:43.828+0000 7f39d3fff700 1 -- 172.21.15.98:0/118115841 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f39dc052030 con 0x7f39ec0b26d0 2023-03-03T07:55:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.022+0000 7f39f1bff700 1 -- 172.21.15.98:0/118115841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f39ec04e6e0 con 0x7f39ec0b26d0 2023-03-03T07:55:44.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.023+0000 7f39d3fff700 1 -- 172.21.15.98:0/118115841 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f39dc055650 con 0x7f39ec0b26d0 2023-03-03T07:55:44.142 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:55:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.025+0000 7f39f1bff700 1 -- 172.21.15.98:0/118115841 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39d80682f0 msgr2=0x7f39d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.025+0000 7f39f1bff700 1 --2- 172.21.15.98:0/118115841 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39d80682f0 0x7f39d806a7a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f39e00012e0 tx=0x7f39e0001270 comp rx=0 tx=0).stop 2023-03-03T07:55:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.025+0000 7f39f1bff700 1 -- 172.21.15.98:0/118115841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39ec0b26d0 msgr2=0x7f39ec157eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.025+0000 7f39f1bff700 1 --2- 172.21.15.98:0/118115841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39ec0b26d0 0x7f39ec157eb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00c550 tx=0x7f39dc00c860 comp rx=0 tx=0).stop 2023-03-03T07:55:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.025+0000 7f39f1bff700 1 -- 172.21.15.98:0/118115841 shutdown_connections 2023-03-03T07:55:44.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.025+0000 7f39f1bff700 1 --2- 172.21.15.98:0/118115841 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39d80682f0 0x7f39d806a7a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.025+0000 7f39f1bff700 1 --2- 172.21.15.98:0/118115841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39ec0b26d0 0x7f39ec157eb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.025+0000 7f39f1bff700 1 --2- 172.21.15.98:0/118115841 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39ec0b8d10 0x7f39ec0b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.025+0000 7f39f1bff700 1 --2- 172.21.15.98:0/118115841 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f39ec0b5310 0x7f39ec0c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.025+0000 7f39f1bff700 1 -- 172.21.15.98:0/118115841 >> 172.21.15.98:0/118115841 conn(0x7f39ec073e70 msgr2=0x7f39ec0b7770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:44.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.026+0000 7f39f1bff700 1 -- 172.21.15.98:0/118115841 shutdown_connections 2023-03-03T07:55:44.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.026+0000 7f39f1bff700 1 -- 172.21.15.98:0/118115841 wait complete. 2023-03-03T07:55:44.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-03-03T07:55:44.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.391+0000 7f92e139d700 1 Processor -- start 2023-03-03T07:55:44.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.392+0000 7f92e139d700 1 -- start start 2023-03-03T07:55:44.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.393+0000 7f92e139d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f92dc0b51c0 0x7f92dc160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:44.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.393+0000 7f92e139d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92dc0d1580 0x7f92dc0d19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:44.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.393+0000 7f92e139d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92dc1614c0 con 0x7f92dc146360 2023-03-03T07:55:44.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.393+0000 7f92e139d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92dc0d10f0 con 0x7f92dc0b51c0 2023-03-03T07:55:44.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.393+0000 7f92e139d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0cdf20 con 0x7f92dc0d1580 2023-03-03T07:55:44.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.393+0000 7f92db7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f92dc0b51c0 0x7f92dc160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:44.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.393+0000 7f92dbfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f92dc146360 0x7f92dc0b4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46984/0 (socket says 172.21.15.98:46984) 2023-03-03T07:55:44.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.393+0000 7f92dbfff700 1 -- 172.21.15.98:0/1828279501 learned_addr learned my addr 172.21.15.98:0/1828279501 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:44.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.393+0000 7f92e0b9c700 1 --2- 172.21.15.98:0/1828279501 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92dc0d1580 0x7f92dc0d19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:44.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.394+0000 7f92db7fe700 1 -- 172.21.15.98:0/1828279501 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92dc0d1580 msgr2=0x7f92dc0d19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:44.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.394+0000 7f92db7fe700 1 --2- 172.21.15.98:0/1828279501 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92dc0d1580 0x7f92dc0d19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.394+0000 7f92db7fe700 1 -- 172.21.15.98:0/1828279501 >> v1:172.21.15.98:6789/0 conn(0x7f92dc146360 legacy=0x7f92dc0b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:44.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.394+0000 7f92db7fe700 1 -- 172.21.15.98:0/1828279501 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc0cd680 con 0x7f92dc0b51c0 2023-03-03T07:55:44.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.394+0000 7f92e0b9c700 1 --2- 172.21.15.98:0/1828279501 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92dc0d1580 0x7f92dc0d19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:44.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.394+0000 7f92daffd700 1 -- 172.21.15.98:0/1828279501 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145290878 0 0) 0x7f92cc001110 con 0x7f92dc146360 2023-03-03T07:55:44.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.394+0000 7f92db7fe700 1 --2- 172.21.15.98:0/1828279501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc0b51c0 0x7f92dc160f80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009840 tx=0x7f92d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4b65e6b6f3cc39 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:44.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.394+0000 7f92daffd700 1 -- 172.21.15.98:0/1828279501 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0011830 con 0x7f92dc0b51c0 2023-03-03T07:55:44.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.395+0000 7f92daffd700 1 -- 172.21.15.98:0/1828279501 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d0011990 con 0x7f92dc0b51c0 2023-03-03T07:55:44.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.395+0000 7f92daffd700 1 -- 172.21.15.98:0/1828279501 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0011d20 con 0x7f92dc0b51c0 2023-03-03T07:55:44.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.395+0000 7f92e139d700 1 -- 172.21.15.98:0/1828279501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc0b51c0 msgr2=0x7f92dc160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:44.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.395+0000 7f92e139d700 1 --2- 172.21.15.98:0/1828279501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc0b51c0 0x7f92dc160f80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009840 tx=0x7f92d0009b50 comp rx=0 tx=0).stop 2023-03-03T07:55:44.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.395+0000 7f92e139d700 1 -- 172.21.15.98:0/1828279501 shutdown_connections 2023-03-03T07:55:44.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.395+0000 7f92e139d700 1 --2- 172.21.15.98:0/1828279501 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92dc0d1580 0x7f92dc0d19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.395+0000 7f92e139d700 1 --2- 172.21.15.98:0/1828279501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc0b51c0 0x7f92dc160f80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.395+0000 7f92e139d700 1 -- 172.21.15.98:0/1828279501 >> 172.21.15.98:0/1828279501 conn(0x7f92dc06b8e0 msgr2=0x7f92dc0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:44.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.395+0000 7f92e139d700 1 -- 172.21.15.98:0/1828279501 shutdown_connections 2023-03-03T07:55:44.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.395+0000 7f92e139d700 1 -- 172.21.15.98:0/1828279501 wait complete. 2023-03-03T07:55:44.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.397+0000 7f92e139d700 1 Processor -- start 2023-03-03T07:55:44.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.397+0000 7f92e139d700 1 -- start start 2023-03-03T07:55:44.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.398+0000 7f92e139d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92dc0b51c0 0x7f92dc15ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:44.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.398+0000 7f92e139d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92dc0d1580 0x7f92dc0c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:44.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.398+0000 7f92e139d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc146360 0x7f92dc0c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:44.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.398+0000 7f92e139d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc15e2d0 con 0x7f92dc146360 2023-03-03T07:55:44.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.398+0000 7f92e139d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0c9b00 con 0x7f92dc0b51c0 2023-03-03T07:55:44.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.398+0000 7f92e139d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc15e540 con 0x7f92dc0d1580 2023-03-03T07:55:44.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.399+0000 7f92e0b9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc146360 0x7f92dc0c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:44.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.399+0000 7f92e0b9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc146360 0x7f92dc0c94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47740/0 (socket says 172.21.15.98:47740) 2023-03-03T07:55:44.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.399+0000 7f92e0b9c700 1 -- 172.21.15.98:0/3516832966 learned_addr learned my addr 172.21.15.98:0/3516832966 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:44.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.399+0000 7f92db7fe700 1 --2- 172.21.15.98:0/3516832966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92dc0d1580 0x7f92dc0c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:44.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.399+0000 7f92dbfff700 1 --2- 172.21.15.98:0/3516832966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92dc0b51c0 0x7f92dc15ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:44.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.399+0000 7f92e0b9c700 1 -- 172.21.15.98:0/3516832966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92dc0d1580 msgr2=0x7f92dc0c8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:44.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.399+0000 7f92e0b9c700 1 --2- 172.21.15.98:0/3516832966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92dc0d1580 0x7f92dc0c8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.399+0000 7f92e0b9c700 1 -- 172.21.15.98:0/3516832966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92dc0b51c0 msgr2=0x7f92dc15ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:44.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.399+0000 7f92e0b9c700 1 --2- 172.21.15.98:0/3516832966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92dc0b51c0 0x7f92dc15ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.399+0000 7f92e0b9c700 1 -- 172.21.15.98:0/3516832966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d00094a0 con 0x7f92dc146360 2023-03-03T07:55:44.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.399+0000 7f92e0b9c700 1 --2- 172.21.15.98:0/3516832966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc146360 0x7f92dc0c94f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f92d4007e70 tx=0x7f92d400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.400+0000 7f92d8ff9700 1 -- 172.21.15.98:0/3516832966 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d400ddd0 con 0x7f92dc146360 2023-03-03T07:55:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.400+0000 7f92d8ff9700 1 -- 172.21.15.98:0/3516832966 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d400f040 con 0x7f92dc146360 2023-03-03T07:55:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.400+0000 7f92d8ff9700 1 -- 172.21.15.98:0/3516832966 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d4003c60 con 0x7f92dc146360 2023-03-03T07:55:44.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.400+0000 7f92e139d700 1 -- 172.21.15.98:0/3516832966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92dc0cdc40 con 0x7f92dc146360 2023-03-03T07:55:44.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.400+0000 7f92e139d700 1 -- 172.21.15.98:0/3516832966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92dc0cd960 con 0x7f92dc146360 2023-03-03T07:55:44.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.401+0000 7f92e139d700 1 -- 172.21.15.98:0/3516832966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92dc0d0d80 con 0x7f92dc146360 2023-03-03T07:55:44.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.402+0000 7f92d8ff9700 1 -- 172.21.15.98:0/3516832966 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f92d401c030 con 0x7f92dc146360 2023-03-03T07:55:44.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.403+0000 7f92d8ff9700 1 --2- 172.21.15.98:0/3516832966 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f92c80682f0 0x7f92c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:44.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.403+0000 7f92d8ff9700 1 -- 172.21.15.98:0/3516832966 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f92d4085970 con 0x7f92dc146360 2023-03-03T07:55:44.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.403+0000 7f92dbfff700 1 --2- 172.21.15.98:0/3516832966 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f92c80682f0 0x7f92c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:44.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.404+0000 7f92dbfff700 1 --2- 172.21.15.98:0/3516832966 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f92c80682f0 0x7f92c806a7a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f92cc000f60 tx=0x7f92cc007a40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:44.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.407+0000 7f92d8ff9700 1 -- 172.21.15.98:0/3516832966 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f92d4052430 con 0x7f92dc146360 2023-03-03T07:55:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.605+0000 7f92e139d700 1 -- 172.21.15.98:0/3516832966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f92dc15ea60 con 0x7f92dc146360 2023-03-03T07:55:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.606+0000 7f92d8ff9700 1 -- 172.21.15.98:0/3516832966 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v20) v1 ==== 84+0+126 (secure 0 0 0) 0x7f92dc15ea60 con 0x7f92dc146360 2023-03-03T07:55:44.608 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:44.608 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T07:55:44.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:55:44.608 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2023-03-03T07:55:44.609 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2023-03-03T07:55:44.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:55:44.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:55:44.610 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 2, 2023-03-03T07:55:44.610 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": "rbd" 2023-03-03T07:55:44.610 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:55:44.610 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T07:55:44.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.608+0000 7f92e139d700 1 -- 172.21.15.98:0/3516832966 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f92c80682f0 msgr2=0x7f92c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:44.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.608+0000 7f92e139d700 1 --2- 172.21.15.98:0/3516832966 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f92c80682f0 0x7f92c806a7a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f92cc000f60 tx=0x7f92cc007a40 comp rx=0 tx=0).stop 2023-03-03T07:55:44.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.608+0000 7f92e139d700 1 -- 172.21.15.98:0/3516832966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc146360 msgr2=0x7f92dc0c94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:44.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.608+0000 7f92e139d700 1 --2- 172.21.15.98:0/3516832966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc146360 0x7f92dc0c94f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f92d4007e70 tx=0x7f92d400d3b0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.608+0000 7f92e139d700 1 -- 172.21.15.98:0/3516832966 shutdown_connections 2023-03-03T07:55:44.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.608+0000 7f92e139d700 1 --2- 172.21.15.98:0/3516832966 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f92c80682f0 0x7f92c806a7a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.608+0000 7f92e139d700 1 --2- 172.21.15.98:0/3516832966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc146360 0x7f92dc0c94f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.608+0000 7f92e139d700 1 --2- 172.21.15.98:0/3516832966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92dc0b51c0 0x7f92dc15ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.608+0000 7f92e139d700 1 --2- 172.21.15.98:0/3516832966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92dc0d1580 0x7f92dc0c8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.608+0000 7f92e139d700 1 -- 172.21.15.98:0/3516832966 >> 172.21.15.98:0/3516832966 conn(0x7f92dc06b8e0 msgr2=0x7f92dc0b7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:44.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.608+0000 7f92e139d700 1 -- 172.21.15.98:0/3516832966 shutdown_connections 2023-03-03T07:55:44.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.608+0000 7f92e139d700 1 -- 172.21.15.98:0/3516832966 wait complete. 2023-03-03T07:55:44.636 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-03-03T07:55:44.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-03-03T07:55:44.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.887+0000 7faa159ff700 1 Processor -- start 2023-03-03T07:55:44.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.887+0000 7faa159ff700 1 -- start start 2023-03-03T07:55:44.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.888+0000 7faa159ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa100d1210 0x7faa100d1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:44.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.888+0000 7faa159ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa100d0ac0 con 0x7faa100cd770 2023-03-03T07:55:44.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.889+0000 7faa159ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa100cdbe0 con 0x7faa100b8880 2023-03-03T07:55:44.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.889+0000 7faa159ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa100b59c0 con 0x7faa100d1210 2023-03-03T07:55:44.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.889+0000 7faa0e7fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7faa100cd770 0x7faa100b52b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46988/0 (socket says 172.21.15.98:46988) 2023-03-03T07:55:44.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.889+0000 7faa0e7fc700 1 -- 172.21.15.98:0/2000406775 learned_addr learned my addr 172.21.15.98:0/2000406775 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:44.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.889+0000 7faa0f7fe700 1 --2- 172.21.15.98:0/2000406775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa100d1210 0x7faa100d1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:44.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.889+0000 7faa0dffb700 1 -- 172.21.15.98:0/2000406775 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570671293 0 0) 0x7faa100d0ac0 con 0x7faa100cd770 2023-03-03T07:55:44.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.889+0000 7faa0dffb700 1 -- 172.21.15.98:0/2000406775 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9f80035b0 con 0x7faa100cd770 2023-03-03T07:55:44.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.889+0000 7faa0dffb700 1 -- 172.21.15.98:0/2000406775 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986358399 0 0) 0x7faa100cdbe0 con 0x7faa100b8880 2023-03-03T07:55:44.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.889+0000 7faa0dffb700 1 -- 172.21.15.98:0/2000406775 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa100d0ac0 con 0x7faa100b8880 2023-03-03T07:55:44.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.890+0000 7faa0f7fe700 1 -- 172.21.15.98:0/2000406775 >> v1:172.21.15.100:6789/0 conn(0x7faa100b8880 legacy=0x7faa100b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:44.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.890+0000 7faa0f7fe700 1 -- 172.21.15.98:0/2000406775 >> v1:172.21.15.98:6789/0 conn(0x7faa100cd770 legacy=0x7faa100b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:44.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.890+0000 7faa0f7fe700 1 -- 172.21.15.98:0/2000406775 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa100cff70 con 0x7faa100d1210 2023-03-03T07:55:44.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.890+0000 7faa0f7fe700 1 --2- 172.21.15.98:0/2000406775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa100d1210 0x7faa100d1690 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faa08009dc0 tx=0x7faa0800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=334846894efd7826 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:44.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.890+0000 7faa0dffb700 1 -- 172.21.15.98:0/2000406775 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0800e070 con 0x7faa100d1210 2023-03-03T07:55:44.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.890+0000 7faa0dffb700 1 -- 172.21.15.98:0/2000406775 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0800bb20 con 0x7faa100d1210 2023-03-03T07:55:44.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.890+0000 7faa0dffb700 1 -- 172.21.15.98:0/2000406775 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa08012430 con 0x7faa100d1210 2023-03-03T07:55:44.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.891+0000 7faa159ff700 1 -- 172.21.15.98:0/2000406775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa100d1210 msgr2=0x7faa100d1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:44.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.891+0000 7faa159ff700 1 --2- 172.21.15.98:0/2000406775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa100d1210 0x7faa100d1690 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faa08009dc0 tx=0x7faa0800b330 comp rx=0 tx=0).stop 2023-03-03T07:55:44.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.891+0000 7faa159ff700 1 -- 172.21.15.98:0/2000406775 shutdown_connections 2023-03-03T07:55:44.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.891+0000 7faa159ff700 1 --2- 172.21.15.98:0/2000406775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa100d1210 0x7faa100d1690 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.891+0000 7faa159ff700 1 -- 172.21.15.98:0/2000406775 >> 172.21.15.98:0/2000406775 conn(0x7faa1006ad80 msgr2=0x7faa1006b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:44.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.891+0000 7faa159ff700 1 -- 172.21.15.98:0/2000406775 shutdown_connections 2023-03-03T07:55:44.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.891+0000 7faa159ff700 1 -- 172.21.15.98:0/2000406775 wait complete. 2023-03-03T07:55:44.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.893+0000 7faa159ff700 1 Processor -- start 2023-03-03T07:55:44.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.893+0000 7faa159ff700 1 -- start start 2023-03-03T07:55:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.894+0000 7faa159ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa100b8880 0x7faa1014c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:44.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.894+0000 7faa159ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa100cd770 0x7faa1015e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.894+0000 7faa159ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa100d1210 0x7faa1015eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.894+0000 7faa159ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa101f34a0 con 0x7faa100cd770 2023-03-03T07:55:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.894+0000 7faa159ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa101f3130 con 0x7faa100d1210 2023-03-03T07:55:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.894+0000 7faa159ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa101f36f0 con 0x7faa100b8880 2023-03-03T07:55:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.894+0000 7faa0e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa100cd770 0x7faa1015e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.894+0000 7faa0e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa100cd770 0x7faa1015e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47744/0 (socket says 172.21.15.98:47744) 2023-03-03T07:55:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.894+0000 7faa0e7fc700 1 -- 172.21.15.98:0/978748067 learned_addr learned my addr 172.21.15.98:0/978748067 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:44.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.895+0000 7faa0f7fe700 1 --2- 172.21.15.98:0/978748067 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa100d1210 0x7faa1015eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:44.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.895+0000 7faa0effd700 1 --2- 172.21.15.98:0/978748067 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa100b8880 0x7faa1014c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:44.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.895+0000 7faa0e7fc700 1 -- 172.21.15.98:0/978748067 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa100b8880 msgr2=0x7faa1014c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:44.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.895+0000 7faa0e7fc700 1 --2- 172.21.15.98:0/978748067 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa100b8880 0x7faa1014c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.895+0000 7faa0e7fc700 1 -- 172.21.15.98:0/978748067 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa100d1210 msgr2=0x7faa1015eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:44.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.895+0000 7faa0e7fc700 1 --2- 172.21.15.98:0/978748067 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa100d1210 0x7faa1015eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:44.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.895+0000 7faa0e7fc700 1 -- 172.21.15.98:0/978748067 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa08009a70 con 0x7faa100cd770 2023-03-03T07:55:44.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.895+0000 7faa0e7fc700 1 --2- 172.21.15.98:0/978748067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa100cd770 0x7faa1015e5b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faa04001eb0 tx=0x7faa0400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:44.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.895+0000 7fa9f7fff700 1 -- 172.21.15.98:0/978748067 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04005d70 con 0x7faa100cd770 2023-03-03T07:55:44.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.895+0000 7fa9f7fff700 1 -- 172.21.15.98:0/978748067 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa040070d0 con 0x7faa100cd770 2023-03-03T07:55:44.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.895+0000 7fa9f7fff700 1 -- 172.21.15.98:0/978748067 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04007460 con 0x7faa100cd770 2023-03-03T07:55:44.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.896+0000 7faa159ff700 1 -- 172.21.15.98:0/978748067 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa100d0530 con 0x7faa100cd770 2023-03-03T07:55:44.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.896+0000 7faa159ff700 1 -- 172.21.15.98:0/978748067 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa100d0250 con 0x7faa100cd770 2023-03-03T07:55:44.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.897+0000 7faa159ff700 1 -- 172.21.15.98:0/978748067 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa100d0cf0 con 0x7faa100cd770 2023-03-03T07:55:44.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.898+0000 7fa9f7fff700 1 -- 172.21.15.98:0/978748067 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faa04019030 con 0x7faa100cd770 2023-03-03T07:55:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.899+0000 7fa9f7fff700 1 --2- 172.21.15.98:0/978748067 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9fc0682f0 0x7fa9fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.899+0000 7fa9f7fff700 1 -- 172.21.15.98:0/978748067 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7faa04084cb0 con 0x7faa100cd770 2023-03-03T07:55:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.900+0000 7faa0effd700 1 --2- 172.21.15.98:0/978748067 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9fc0682f0 0x7fa9fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.900+0000 7faa0effd700 1 --2- 172.21.15.98:0/978748067 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9fc0682f0 0x7fa9fc06a7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faa100cd260 tx=0x7faa00009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:44.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:44.903+0000 7fa9f7fff700 1 -- 172.21.15.98:0/978748067 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faa04051770 con 0x7faa100cd770 2023-03-03T07:55:45.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.097+0000 7faa159ff700 1 -- 172.21.15.98:0/978748067 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7faa1004e6e0 con 0x7faa100cd770 2023-03-03T07:55:45.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.530+0000 7fa9f7fff700 1 -- 172.21.15.98:0/978748067 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v21) v1 ==== 138+0+0 (secure 0 0 0) 0x7faa040170b0 con 0x7faa100cd770 2023-03-03T07:55:45.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.537+0000 7faa159ff700 1 -- 172.21.15.98:0/978748067 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9fc0682f0 msgr2=0x7fa9fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:45.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.537+0000 7faa159ff700 1 --2- 172.21.15.98:0/978748067 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9fc0682f0 0x7fa9fc06a7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faa100cd260 tx=0x7faa00009040 comp rx=0 tx=0).stop 2023-03-03T07:55:45.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.537+0000 7faa159ff700 1 -- 172.21.15.98:0/978748067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa100cd770 msgr2=0x7faa1015e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:45.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.537+0000 7faa159ff700 1 --2- 172.21.15.98:0/978748067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa100cd770 0x7faa1015e5b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faa04001eb0 tx=0x7faa0400c7b0 comp rx=0 tx=0).stop 2023-03-03T07:55:45.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.538+0000 7faa159ff700 1 -- 172.21.15.98:0/978748067 shutdown_connections 2023-03-03T07:55:45.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.538+0000 7faa159ff700 1 --2- 172.21.15.98:0/978748067 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9fc0682f0 0x7fa9fc06a7a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:45.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.538+0000 7faa159ff700 1 --2- 172.21.15.98:0/978748067 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa100d1210 0x7faa1015eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:45.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.538+0000 7faa159ff700 1 --2- 172.21.15.98:0/978748067 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa100b8880 0x7faa1014c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:45.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.539+0000 7faa159ff700 1 --2- 172.21.15.98:0/978748067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa100cd770 0x7faa1015e5b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:45.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.539+0000 7faa159ff700 1 -- 172.21.15.98:0/978748067 >> 172.21.15.98:0/978748067 conn(0x7faa1006ad80 msgr2=0x7faa100d1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:45.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.539+0000 7faa159ff700 1 -- 172.21.15.98:0/978748067 shutdown_connections 2023-03-03T07:55:45.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.539+0000 7faa159ff700 1 -- 172.21.15.98:0/978748067 wait complete. 2023-03-03T07:55:45.542 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2023-03-03T07:55:45.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-03-03T07:55:45.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.821+0000 7f9f149ff700 1 Processor -- start 2023-03-03T07:55:45.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.821+0000 7f9f149ff700 1 -- start start 2023-03-03T07:55:45.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.822+0000 7f9f149ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9f100b8d10 0x7f9f100cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:45.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.822+0000 7f9f149ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f100b5310 0x7f9f100d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:45.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.822+0000 7f9f149ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f100cd7d0 con 0x7f9f100b26d0 2023-03-03T07:55:45.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.823+0000 7f9f149ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f100b90e0 con 0x7f9f100b8d10 2023-03-03T07:55:45.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.823+0000 7f9f149ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f100cda20 con 0x7f9f100b5310 2023-03-03T07:55:45.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.823+0000 7f9f0e59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9f100b8d10 0x7f9f100cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.823+0000 7f9f0e59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9f100b8d10 0x7f9f100cffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45174/0 (socket says 172.21.15.98:45174) 2023-03-03T07:55:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.823+0000 7f9f0e59c700 1 -- 172.21.15.98:0/4293816292 learned_addr learned my addr 172.21.15.98:0/4293816292 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.823+0000 7f9f0ed9d700 1 --2- 172.21.15.98:0/4293816292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f100b5310 0x7f9f100d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.823+0000 7f9f0e59c700 1 -- 172.21.15.98:0/4293816292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f100b5310 msgr2=0x7f9f100d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.823+0000 7f9f0e59c700 1 --2- 172.21.15.98:0/4293816292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f100b5310 0x7f9f100d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.823+0000 7f9f0e59c700 1 -- 172.21.15.98:0/4293816292 >> v1:172.21.15.100:6789/0 conn(0x7f9f100b26d0 legacy=0x7f9f100b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.823+0000 7f9f0e59c700 1 -- 172.21.15.98:0/4293816292 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f100cdd00 con 0x7f9f100b8d10 2023-03-03T07:55:45.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.824+0000 7f9f0e59c700 1 --2- 172.21.15.98:0/4293816292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f100b8d10 0x7f9f100cffc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9f000098f0 tx=0x7f9f00009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43827a49c84c4197 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:45.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.824+0000 7f9f0dd9b700 1 -- 172.21.15.98:0/4293816292 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0000b8a0 con 0x7f9f100b8d10 2023-03-03T07:55:45.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.824+0000 7f9f0dd9b700 1 -- 172.21.15.98:0/4293816292 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0000ba00 con 0x7f9f100b8d10 2023-03-03T07:55:45.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.824+0000 7f9f0dd9b700 1 -- 172.21.15.98:0/4293816292 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0000bd90 con 0x7f9f100b8d10 2023-03-03T07:55:45.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.824+0000 7f9f149ff700 1 -- 172.21.15.98:0/4293816292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f100b8d10 msgr2=0x7f9f100cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:45.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.824+0000 7f9f149ff700 1 --2- 172.21.15.98:0/4293816292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f100b8d10 0x7f9f100cffc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9f000098f0 tx=0x7f9f00009c00 comp rx=0 tx=0).stop 2023-03-03T07:55:45.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.824+0000 7f9f149ff700 1 -- 172.21.15.98:0/4293816292 shutdown_connections 2023-03-03T07:55:45.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.825+0000 7f9f149ff700 1 --2- 172.21.15.98:0/4293816292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f100b8d10 0x7f9f100cffc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:45.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.825+0000 7f9f149ff700 1 --2- 172.21.15.98:0/4293816292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f100b5310 0x7f9f100d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:45.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.825+0000 7f9f149ff700 1 -- 172.21.15.98:0/4293816292 >> 172.21.15.98:0/4293816292 conn(0x7f9f10073e70 msgr2=0x7f9f1006ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:45.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.825+0000 7f9f149ff700 1 -- 172.21.15.98:0/4293816292 shutdown_connections 2023-03-03T07:55:45.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.825+0000 7f9f149ff700 1 -- 172.21.15.98:0/4293816292 wait complete. 2023-03-03T07:55:45.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.827+0000 7f9f149ff700 1 Processor -- start 2023-03-03T07:55:45.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.827+0000 7f9f149ff700 1 -- start start 2023-03-03T07:55:45.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.828+0000 7f9f149ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f100b26d0 0x7f9f10151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:45.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.828+0000 7f9f149ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f100b5310 0x7f9f10156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:45.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.828+0000 7f9f149ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f100b8d10 0x7f9f100b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:45.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.828+0000 7f9f149ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f10152b10 con 0x7f9f100b5310 2023-03-03T07:55:45.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.828+0000 7f9f149ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f10156600 con 0x7f9f100b8d10 2023-03-03T07:55:45.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.828+0000 7f9f149ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f10152170 con 0x7f9f100b26d0 2023-03-03T07:55:45.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.828+0000 7f9f07fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f100b5310 0x7f9f10156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:45.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.828+0000 7f9f07fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f100b5310 0x7f9f10156a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45178/0 (socket says 172.21.15.98:45178) 2023-03-03T07:55:45.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.828+0000 7f9f07fff700 1 -- 172.21.15.98:0/969125725 learned_addr learned my addr 172.21.15.98:0/969125725 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:45.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.828+0000 7f9f0e59c700 1 --2- 172.21.15.98:0/969125725 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f100b26d0 0x7f9f10151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:45.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.828+0000 7f9f0ed9d700 1 --2- 172.21.15.98:0/969125725 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f100b8d10 0x7f9f100b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:45.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.829+0000 7f9f07fff700 1 -- 172.21.15.98:0/969125725 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f100b26d0 msgr2=0x7f9f10151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:45.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.829+0000 7f9f07fff700 1 --2- 172.21.15.98:0/969125725 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f100b26d0 0x7f9f10151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:45.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.829+0000 7f9f07fff700 1 -- 172.21.15.98:0/969125725 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f100b8d10 msgr2=0x7f9f100b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:45.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.829+0000 7f9f07fff700 1 --2- 172.21.15.98:0/969125725 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f100b8d10 0x7f9f100b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:45.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.829+0000 7f9f07fff700 1 -- 172.21.15.98:0/969125725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f00009550 con 0x7f9f100b5310 2023-03-03T07:55:45.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.829+0000 7f9f0ed9d700 1 --2- 172.21.15.98:0/969125725 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f100b8d10 0x7f9f100b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:45.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.829+0000 7f9f07fff700 1 --2- 172.21.15.98:0/969125725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f100b5310 0x7f9f10156a30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c450 tx=0x7f9ef800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:45.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.829+0000 7f9f06ffd700 1 -- 172.21.15.98:0/969125725 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef80052d0 con 0x7f9f100b5310 2023-03-03T07:55:45.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.829+0000 7f9f06ffd700 1 -- 172.21.15.98:0/969125725 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef8005430 con 0x7f9f100b5310 2023-03-03T07:55:45.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.829+0000 7f9f149ff700 1 -- 172.21.15.98:0/969125725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f100cdfe0 con 0x7f9f100b5310 2023-03-03T07:55:45.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.829+0000 7f9f06ffd700 1 -- 172.21.15.98:0/969125725 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef80038c0 con 0x7f9f100b5310 2023-03-03T07:55:45.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.829+0000 7f9f149ff700 1 -- 172.21.15.98:0/969125725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f100d0e80 con 0x7f9f100b5310 2023-03-03T07:55:45.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.831+0000 7f9f149ff700 1 -- 172.21.15.98:0/969125725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f100d0500 con 0x7f9f100b5310 2023-03-03T07:55:45.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.832+0000 7f9f06ffd700 1 -- 172.21.15.98:0/969125725 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9ef8019030 con 0x7f9f100b5310 2023-03-03T07:55:45.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.834+0000 7f9f06ffd700 1 --2- 172.21.15.98:0/969125725 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ef00682f0 0x7f9ef006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:45.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.834+0000 7f9f06ffd700 1 -- 172.21.15.98:0/969125725 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5580+0+0 (secure 0 0 0) 0x7f9ef8084de0 con 0x7f9f100b5310 2023-03-03T07:55:45.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.834+0000 7f9f0e59c700 1 --2- 172.21.15.98:0/969125725 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ef00682f0 0x7f9ef006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:45.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.835+0000 7f9f0e59c700 1 --2- 172.21.15.98:0/969125725 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ef00682f0 0x7f9ef006a7a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f00009c40 tx=0x7f9f00008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:45.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:45.839+0000 7f9f06ffd700 1 -- 172.21.15.98:0/969125725 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9ef80516d0 con 0x7f9f100b5310 2023-03-03T07:55:46.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.032+0000 7f9f149ff700 1 -- 172.21.15.98:0/969125725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9f1004e6e0 con 0x7f9f100b5310 2023-03-03T07:55:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.540+0000 7f9f06ffd700 1 -- 172.21.15.98:0/969125725 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v22) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9ef8054cf0 con 0x7f9f100b5310 2023-03-03T07:55:46.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.546+0000 7f9f149ff700 1 -- 172.21.15.98:0/969125725 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ef00682f0 msgr2=0x7f9ef006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:46.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.546+0000 7f9f149ff700 1 --2- 172.21.15.98:0/969125725 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ef00682f0 0x7f9ef006a7a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f00009c40 tx=0x7f9f00008030 comp rx=0 tx=0).stop 2023-03-03T07:55:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.546+0000 7f9f149ff700 1 -- 172.21.15.98:0/969125725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f100b5310 msgr2=0x7f9f10156a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.546+0000 7f9f149ff700 1 --2- 172.21.15.98:0/969125725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f100b5310 0x7f9f10156a30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c450 tx=0x7f9ef800c760 comp rx=0 tx=0).stop 2023-03-03T07:55:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.547+0000 7f9f149ff700 1 -- 172.21.15.98:0/969125725 shutdown_connections 2023-03-03T07:55:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.547+0000 7f9f149ff700 1 --2- 172.21.15.98:0/969125725 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ef00682f0 0x7f9ef006a7a0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f00009c40 tx=0x7f9f00008030 comp rx=0 tx=0).stop 2023-03-03T07:55:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.547+0000 7f9f149ff700 1 --2- 172.21.15.98:0/969125725 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f100b8d10 0x7f9f100b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.547+0000 7f9f149ff700 1 --2- 172.21.15.98:0/969125725 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f100b26d0 0x7f9f10151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.547+0000 7f9f149ff700 1 --2- 172.21.15.98:0/969125725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f100b5310 0x7f9f10156a30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.547+0000 7f9f149ff700 1 -- 172.21.15.98:0/969125725 >> 172.21.15.98:0/969125725 conn(0x7f9f10073e70 msgr2=0x7f9f100d12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:46.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.550+0000 7f9f149ff700 1 -- 172.21.15.98:0/969125725 shutdown_connections 2023-03-03T07:55:46.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.555+0000 7f9f149ff700 1 -- 172.21.15.98:0/969125725 wait complete. 2023-03-03T07:55:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2023-03-03T07:55:46.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-03-03T07:55:46.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.840+0000 7f3763bff700 1 Processor -- start 2023-03-03T07:55:46.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.840+0000 7f3763bff700 1 -- start start 2023-03-03T07:55:46.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.841+0000 7f3763bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c0b8d10 0x7f375c0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:46.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.842+0000 7f3763bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375c0cd7d0 con 0x7f375c0b5310 2023-03-03T07:55:46.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.842+0000 7f3763bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375c0b90e0 con 0x7f375c0b26d0 2023-03-03T07:55:46.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.842+0000 7f3763bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f375c0cda20 con 0x7f375c0b8d10 2023-03-03T07:55:46.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.842+0000 7f376219c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f375c0b5310 0x7f375c0d13a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60246/0 (socket says 172.21.15.98:60246) 2023-03-03T07:55:46.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.842+0000 7f376219c700 1 -- 172.21.15.98:0/1401401641 learned_addr learned my addr 172.21.15.98:0/1401401641 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:46.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.842+0000 7f376199b700 1 --2- 172.21.15.98:0/1401401641 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c0b8d10 0x7f375c0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:46.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.842+0000 7f3760999700 1 -- 172.21.15.98:0/1401401641 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349475756 0 0) 0x7f375c0cd7d0 con 0x7f375c0b5310 2023-03-03T07:55:46.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.842+0000 7f3760999700 1 -- 172.21.15.98:0/1401401641 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37480035b0 con 0x7f375c0b5310 2023-03-03T07:55:46.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.843+0000 7f3760999700 1 -- 172.21.15.98:0/1401401641 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186847421 0 0) 0x7f375c0b90e0 con 0x7f375c0b26d0 2023-03-03T07:55:46.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.843+0000 7f3760999700 1 -- 172.21.15.98:0/1401401641 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f375c0cd7d0 con 0x7f375c0b26d0 2023-03-03T07:55:46.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.843+0000 7f376199b700 1 -- 172.21.15.98:0/1401401641 >> v1:172.21.15.100:6789/0 conn(0x7f375c0b26d0 legacy=0x7f375c0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:46.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.843+0000 7f376199b700 1 -- 172.21.15.98:0/1401401641 >> v1:172.21.15.98:6789/0 conn(0x7f375c0b5310 legacy=0x7f375c0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:46.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.843+0000 7f376199b700 1 -- 172.21.15.98:0/1401401641 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c0cdd00 con 0x7f375c0b8d10 2023-03-03T07:55:46.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.843+0000 7f376199b700 1 --2- 172.21.15.98:0/1401401641 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c0b8d10 0x7f375c0cffc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3758009870 tx=0x7f3758009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9092ef06fb01222 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:46.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.843+0000 7f3760999700 1 -- 172.21.15.98:0/1401401641 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3758011830 con 0x7f375c0b8d10 2023-03-03T07:55:46.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.843+0000 7f3760999700 1 -- 172.21.15.98:0/1401401641 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3758011990 con 0x7f375c0b8d10 2023-03-03T07:55:46.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.843+0000 7f3760999700 1 -- 172.21.15.98:0/1401401641 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3758011d20 con 0x7f375c0b8d10 2023-03-03T07:55:46.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.844+0000 7f3763bff700 1 -- 172.21.15.98:0/1401401641 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c0b8d10 msgr2=0x7f375c0cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.844+0000 7f3763bff700 1 --2- 172.21.15.98:0/1401401641 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c0b8d10 0x7f375c0cffc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3758009870 tx=0x7f3758009b80 comp rx=0 tx=0).stop 2023-03-03T07:55:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.844+0000 7f3763bff700 1 -- 172.21.15.98:0/1401401641 shutdown_connections 2023-03-03T07:55:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.844+0000 7f3763bff700 1 --2- 172.21.15.98:0/1401401641 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c0b8d10 0x7f375c0cffc0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.844+0000 7f3763bff700 1 -- 172.21.15.98:0/1401401641 >> 172.21.15.98:0/1401401641 conn(0x7f375c073e70 msgr2=0x7f375c06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.844+0000 7f3763bff700 1 -- 172.21.15.98:0/1401401641 shutdown_connections 2023-03-03T07:55:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.844+0000 7f3763bff700 1 -- 172.21.15.98:0/1401401641 wait complete. 2023-03-03T07:55:46.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.846+0000 7f3763bff700 1 Processor -- start 2023-03-03T07:55:46.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.846+0000 7f3763bff700 1 -- start start 2023-03-03T07:55:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.847+0000 7f3763bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c0b26d0 0x7f375c0ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.847+0000 7f3763bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f375c0b5310 0x7f375c15f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.847+0000 7f3763bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f375c0b8d10 0x7f375c0ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.847+0000 7f3763bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c0cc0c0 con 0x7f375c0b8d10 2023-03-03T07:55:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.847+0000 7f3763bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c0cc320 con 0x7f375c0b5310 2023-03-03T07:55:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.847+0000 7f3763bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f375c145e40 con 0x7f375c0b26d0 2023-03-03T07:55:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.848+0000 7f376219c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f375c0b8d10 0x7f375c0ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:46.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.848+0000 7f376219c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f375c0b8d10 0x7f375c0ca250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45184/0 (socket says 172.21.15.98:45184) 2023-03-03T07:55:46.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.848+0000 7f376219c700 1 -- 172.21.15.98:0/4067288465 learned_addr learned my addr 172.21.15.98:0/4067288465 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:46.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.848+0000 7f376199b700 1 --2- 172.21.15.98:0/4067288465 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c0b26d0 0x7f375c0ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:46.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.848+0000 7f376119a700 1 --2- 172.21.15.98:0/4067288465 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f375c0b5310 0x7f375c15f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:46.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.848+0000 7f376219c700 1 -- 172.21.15.98:0/4067288465 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c0b26d0 msgr2=0x7f375c0ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:46.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.848+0000 7f376219c700 1 --2- 172.21.15.98:0/4067288465 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c0b26d0 0x7f375c0ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:46.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.848+0000 7f376219c700 1 -- 172.21.15.98:0/4067288465 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f375c0b5310 msgr2=0x7f375c15f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:46.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.848+0000 7f376219c700 1 --2- 172.21.15.98:0/4067288465 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f375c0b5310 0x7f375c15f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:46.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.848+0000 7f376219c700 1 -- 172.21.15.98:0/4067288465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37580094a0 con 0x7f375c0b8d10 2023-03-03T07:55:46.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.849+0000 7f376219c700 1 --2- 172.21.15.98:0/4067288465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f375c0b8d10 0x7f375c0ca250 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f375400caa0 tx=0x7f375400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:46.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.849+0000 7f37527fc700 1 -- 172.21.15.98:0/4067288465 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375400a9e0 con 0x7f375c0b8d10 2023-03-03T07:55:46.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.849+0000 7f37527fc700 1 -- 172.21.15.98:0/4067288465 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375400ab40 con 0x7f375c0b8d10 2023-03-03T07:55:46.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.849+0000 7f37527fc700 1 -- 172.21.15.98:0/4067288465 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3754012400 con 0x7f375c0b8d10 2023-03-03T07:55:46.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.849+0000 7f3763bff700 1 -- 172.21.15.98:0/4067288465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375c0cdfe0 con 0x7f375c0b8d10 2023-03-03T07:55:46.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.849+0000 7f3763bff700 1 -- 172.21.15.98:0/4067288465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375c0d0e80 con 0x7f375c0b8d10 2023-03-03T07:55:46.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.850+0000 7f3763bff700 1 -- 172.21.15.98:0/4067288465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f375c0d0500 con 0x7f375c0b8d10 2023-03-03T07:55:46.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.851+0000 7f37527fc700 1 -- 172.21.15.98:0/4067288465 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3754019030 con 0x7f375c0b8d10 2023-03-03T07:55:46.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.853+0000 7f37527fc700 1 --2- 172.21.15.98:0/4067288465 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37400682f0 0x7f374006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:46.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.853+0000 7f37527fc700 1 -- 172.21.15.98:0/4067288465 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f3754085650 con 0x7f375c0b8d10 2023-03-03T07:55:46.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.853+0000 7f376199b700 1 --2- 172.21.15.98:0/4067288465 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37400682f0 0x7f374006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:46.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.854+0000 7f376199b700 1 --2- 172.21.15.98:0/4067288465 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37400682f0 0x7f374006a7a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f37580097c0 tx=0x7f375801c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:46.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:46.857+0000 7f37527fc700 1 -- 172.21.15.98:0/4067288465 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f375404b5e0 con 0x7f375c0b8d10 2023-03-03T07:55:47.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.095+0000 7f3763bff700 1 -- 172.21.15.98:0/4067288465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f375c04e6e0 con 0x7f375c0b8d10 2023-03-03T07:55:47.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.104+0000 7f37527fc700 1 -- 172.21.15.98:0/4067288465 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f3754051f90 con 0x7f375c0b8d10 2023-03-03T07:55:47.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.112+0000 7f3763bff700 1 -- 172.21.15.98:0/4067288465 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37400682f0 msgr2=0x7f374006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:47.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.112+0000 7f3763bff700 1 --2- 172.21.15.98:0/4067288465 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37400682f0 0x7f374006a7a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f37580097c0 tx=0x7f375801c040 comp rx=0 tx=0).stop 2023-03-03T07:55:47.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.112+0000 7f3763bff700 1 -- 172.21.15.98:0/4067288465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f375c0b8d10 msgr2=0x7f375c0ca250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:47.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.112+0000 7f3763bff700 1 --2- 172.21.15.98:0/4067288465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f375c0b8d10 0x7f375c0ca250 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f375400caa0 tx=0x7f375400cdb0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.120+0000 7f3763bff700 1 -- 172.21.15.98:0/4067288465 shutdown_connections 2023-03-03T07:55:47.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.120+0000 7f3763bff700 1 --2- 172.21.15.98:0/4067288465 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37400682f0 0x7f374006a7a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.120+0000 7f3763bff700 1 --2- 172.21.15.98:0/4067288465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f375c0b8d10 0x7f375c0ca250 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.120+0000 7f3763bff700 1 --2- 172.21.15.98:0/4067288465 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c0b26d0 0x7f375c0ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.120+0000 7f3763bff700 1 --2- 172.21.15.98:0/4067288465 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f375c0b5310 0x7f375c15f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.120+0000 7f3763bff700 1 -- 172.21.15.98:0/4067288465 >> 172.21.15.98:0/4067288465 conn(0x7f375c073e70 msgr2=0x7f375c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:47.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.121+0000 7f3763bff700 1 -- 172.21.15.98:0/4067288465 shutdown_connections 2023-03-03T07:55:47.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.121+0000 7f3763bff700 1 -- 172.21.15.98:0/4067288465 wait complete. 2023-03-03T07:55:47.138 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 3 and data pool 4 2023-03-03T07:55:47.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:55:47.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.406+0000 7fa65319d700 1 Processor -- start 2023-03-03T07:55:47.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.406+0000 7fa65319d700 1 -- start start 2023-03-03T07:55:47.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.407+0000 7fa65319d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa64c146360 0x7fa64c160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:47.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.408+0000 7fa65319d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa64c0d2590 0x7fa64c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:47.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.408+0000 7fa65319d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c1614c0 con 0x7fa64c0b4ab0 2023-03-03T07:55:47.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.408+0000 7fa65319d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa64c0b5440 con 0x7fa64c146360 2023-03-03T07:55:47.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.408+0000 7fa65319d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c0d10f0 con 0x7fa64c0d2590 2023-03-03T07:55:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.408+0000 7fa65299c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa64c0d2590 0x7fa64c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.408+0000 7fa65299c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa64c0d2590 0x7fa64c0b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:42158/0 (socket says 172.21.15.98:42158) 2023-03-03T07:55:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.408+0000 7fa65299c700 1 -- 172.21.15.98:0/3881914775 learned_addr learned my addr 172.21.15.98:0/3881914775 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.408+0000 7fa65199a700 1 --1- 172.21.15.98:0/3881914775 >> v1:172.21.15.100:6789/0 conn(0x7fa64c0b4ab0 0x7fa64c0731e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:40478/0 (socket says 172.21.15.98:40478) 2023-03-03T07:55:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.408+0000 7fa65219b700 1 --2- 172.21.15.98:0/3881914775 >> v2:172.21.15.100:3300/0 conn(0x7fa64c146360 0x7fa64c160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.409+0000 7fa65219b700 1 -- 172.21.15.98:0/3881914775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa64c0d2590 msgr2=0x7fa64c0b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.409+0000 7fa65219b700 1 --2- 172.21.15.98:0/3881914775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa64c0d2590 0x7fa64c0b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.409+0000 7fa65219b700 1 -- 172.21.15.98:0/3881914775 >> v1:172.21.15.100:6789/0 conn(0x7fa64c0b4ab0 legacy=0x7fa64c0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.409+0000 7fa65219b700 1 -- 172.21.15.98:0/3881914775 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c0cdf20 con 0x7fa64c146360 2023-03-03T07:55:47.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.409+0000 7fa65219b700 1 --2- 172.21.15.98:0/3881914775 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa64c146360 0x7fa64c160f80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa648009840 tx=0x7fa648009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acaa65641b263935 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:47.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.409+0000 7fa651199700 1 -- 172.21.15.98:0/3881914775 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa648011830 con 0x7fa64c146360 2023-03-03T07:55:47.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.409+0000 7fa651199700 1 -- 172.21.15.98:0/3881914775 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa648011990 con 0x7fa64c146360 2023-03-03T07:55:47.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.409+0000 7fa651199700 1 -- 172.21.15.98:0/3881914775 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa648011d20 con 0x7fa64c146360 2023-03-03T07:55:47.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.410+0000 7fa65319d700 1 -- 172.21.15.98:0/3881914775 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa64c146360 msgr2=0x7fa64c160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:47.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.410+0000 7fa65319d700 1 --2- 172.21.15.98:0/3881914775 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa64c146360 0x7fa64c160f80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa648009840 tx=0x7fa648009b50 comp rx=0 tx=0).stop 2023-03-03T07:55:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.410+0000 7fa65319d700 1 -- 172.21.15.98:0/3881914775 shutdown_connections 2023-03-03T07:55:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.410+0000 7fa65319d700 1 --2- 172.21.15.98:0/3881914775 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa64c146360 0x7fa64c160f80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.410+0000 7fa65319d700 1 --2- 172.21.15.98:0/3881914775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa64c0d2590 0x7fa64c0b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.410+0000 7fa65319d700 1 -- 172.21.15.98:0/3881914775 >> 172.21.15.98:0/3881914775 conn(0x7fa64c06b8e0 msgr2=0x7fa64c0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:47.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.410+0000 7fa65319d700 1 -- 172.21.15.98:0/3881914775 shutdown_connections 2023-03-03T07:55:47.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.410+0000 7fa65319d700 1 -- 172.21.15.98:0/3881914775 wait complete. 2023-03-03T07:55:47.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.412+0000 7fa65319d700 1 Processor -- start 2023-03-03T07:55:47.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.412+0000 7fa65319d700 1 -- start start 2023-03-03T07:55:47.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.413+0000 7fa65319d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa64c0b4ab0 0x7fa64c14ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:47.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.413+0000 7fa65319d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa64c0d2590 0x7fa64c15db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:47.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.413+0000 7fa65319d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa64c146360 0x7fa64c0c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:47.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.413+0000 7fa65319d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c1604c0 con 0x7fa64c0d2590 2023-03-03T07:55:47.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.413+0000 7fa65319d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c150510 con 0x7fa64c146360 2023-03-03T07:55:47.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.413+0000 7fa65319d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c15e490 con 0x7fa64c0b4ab0 2023-03-03T07:55:47.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.414+0000 7fa65199a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa64c0d2590 0x7fa64c15db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:47.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.414+0000 7fa65199a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa64c0d2590 0x7fa64c15db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45200/0 (socket says 172.21.15.98:45200) 2023-03-03T07:55:47.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.414+0000 7fa65199a700 1 -- 172.21.15.98:0/65968443 learned_addr learned my addr 172.21.15.98:0/65968443 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:47.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.414+0000 7fa65219b700 1 --2- 172.21.15.98:0/65968443 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa64c0b4ab0 0x7fa64c14ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:47.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.414+0000 7fa65299c700 1 --2- 172.21.15.98:0/65968443 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa64c146360 0x7fa64c0c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:47.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.414+0000 7fa65199a700 1 -- 172.21.15.98:0/65968443 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa64c0b4ab0 msgr2=0x7fa64c14ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.414+0000 7fa65199a700 1 --2- 172.21.15.98:0/65968443 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa64c0b4ab0 0x7fa64c14ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.414+0000 7fa65199a700 1 -- 172.21.15.98:0/65968443 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa64c146360 msgr2=0x7fa64c0c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.414+0000 7fa65199a700 1 --2- 172.21.15.98:0/65968443 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa64c146360 0x7fa64c0c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.414+0000 7fa65199a700 1 -- 172.21.15.98:0/65968443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6480094a0 con 0x7fa64c0d2590 2023-03-03T07:55:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.414+0000 7fa65199a700 1 --2- 172.21.15.98:0/65968443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa64c0d2590 0x7fa64c15db20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00c530 tx=0x7fa63c00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:47.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.415+0000 7fa642ffd700 1 -- 172.21.15.98:0/65968443 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c00cea0 con 0x7fa64c0d2590 2023-03-03T07:55:47.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.415+0000 7fa642ffd700 1 -- 172.21.15.98:0/65968443 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63c005cb0 con 0x7fa64c0d2590 2023-03-03T07:55:47.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.415+0000 7fa642ffd700 1 -- 172.21.15.98:0/65968443 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c0072d0 con 0x7fa64c0d2590 2023-03-03T07:55:47.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.415+0000 7fa65319d700 1 -- 172.21.15.98:0/65968443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64c0cd960 con 0x7fa64c0d2590 2023-03-03T07:55:47.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.415+0000 7fa65319d700 1 -- 172.21.15.98:0/65968443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64c0cd680 con 0x7fa64c0d2590 2023-03-03T07:55:47.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.416+0000 7fa65319d700 1 -- 172.21.15.98:0/65968443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64c0cdc40 con 0x7fa64c0d2590 2023-03-03T07:55:47.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.417+0000 7fa642ffd700 1 -- 172.21.15.98:0/65968443 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa63c019030 con 0x7fa64c0d2590 2023-03-03T07:55:47.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.419+0000 7fa642ffd700 1 --2- 172.21.15.98:0/65968443 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6300682f0 0x7fa63006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:47.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.419+0000 7fa642ffd700 1 -- 172.21.15.98:0/65968443 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa63c084c80 con 0x7fa64c0d2590 2023-03-03T07:55:47.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.419+0000 7fa65219b700 1 --2- 172.21.15.98:0/65968443 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6300682f0 0x7fa63006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:47.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.420+0000 7fa65219b700 1 --2- 172.21.15.98:0/65968443 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6300682f0 0x7fa63006a7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa648000c00 tx=0x7fa64801c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:47.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.423+0000 7fa642ffd700 1 -- 172.21.15.98:0/65968443 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa63c04b5e0 con 0x7fa64c0d2590 2023-03-03T07:55:47.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.627+0000 7fa65319d700 1 -- 172.21.15.98:0/65968443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa64c1614c0 con 0x7fa64c0d2590 2023-03-03T07:55:47.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.628+0000 7fa642ffd700 1 -- 172.21.15.98:0/65968443 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+17920 (secure 0 0 0) 0x7fa64c1614c0 con 0x7fa64c0d2590 2023-03-03T07:55:47.631 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:47.631 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:55:47.096834+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:55:47.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.631+0000 7fa65319d700 1 -- 172.21.15.98:0/65968443 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6300682f0 msgr2=0x7fa63006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:47.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.631+0000 7fa65319d700 1 --2- 172.21.15.98:0/65968443 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6300682f0 0x7fa63006a7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa648000c00 tx=0x7fa64801c040 comp rx=0 tx=0).stop 2023-03-03T07:55:47.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.631+0000 7fa65319d700 1 -- 172.21.15.98:0/65968443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa64c0d2590 msgr2=0x7fa64c15db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:47.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.631+0000 7fa65319d700 1 --2- 172.21.15.98:0/65968443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa64c0d2590 0x7fa64c15db20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00c530 tx=0x7fa63c00c840 comp rx=0 tx=0).stop 2023-03-03T07:55:47.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.631+0000 7fa65319d700 1 -- 172.21.15.98:0/65968443 shutdown_connections 2023-03-03T07:55:47.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.631+0000 7fa65319d700 1 --2- 172.21.15.98:0/65968443 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6300682f0 0x7fa63006a7a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.631+0000 7fa65319d700 1 --2- 172.21.15.98:0/65968443 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa64c146360 0x7fa64c0c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.631+0000 7fa65319d700 1 --2- 172.21.15.98:0/65968443 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa64c0b4ab0 0x7fa64c14ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.631+0000 7fa65319d700 1 --2- 172.21.15.98:0/65968443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa64c0d2590 0x7fa64c15db20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.631+0000 7fa65319d700 1 -- 172.21.15.98:0/65968443 >> 172.21.15.98:0/65968443 conn(0x7fa64c06b8e0 msgr2=0x7fa64c0b21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:47.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.631+0000 7fa65319d700 1 -- 172.21.15.98:0/65968443 shutdown_connections 2023-03-03T07:55:47.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.631+0000 7fa65319d700 1 -- 172.21.15.98:0/65968443 wait complete. 2023-03-03T07:55:47.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:55:47.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.902+0000 7fd54d7ff700 1 Processor -- start 2023-03-03T07:55:47.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.903+0000 7fd54d7ff700 1 -- start start 2023-03-03T07:55:47.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.904+0000 7fd54d7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5480b87c0 0x7fd5480cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:47.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.904+0000 7fd54d7ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd5480b8b90 0x7fd5480b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:47.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.904+0000 7fd54d7ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd5480b5240 0x7fd5480b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:47.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.904+0000 7fd54d7ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5480d0ef0 con 0x7fd5480b5240 2023-03-03T07:55:47.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.904+0000 7fd54d7ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5480d0600 con 0x7fd5480b8b90 2023-03-03T07:55:47.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.904+0000 7fd54d7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5480cd970 con 0x7fd5480b87c0 2023-03-03T07:55:47.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.904+0000 7fd5477fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd5480b5240 0x7fd5480b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:47.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.905+0000 7fd5477fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd5480b5240 0x7fd5480b56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45210/0 (socket says 172.21.15.98:45210) 2023-03-03T07:55:47.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.905+0000 7fd5477fe700 1 -- 172.21.15.98:0/2467699001 learned_addr learned my addr 172.21.15.98:0/2467699001 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:47.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.905+0000 7fd546ffd700 1 --2- 172.21.15.98:0/2467699001 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5480b87c0 0x7fd5480cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:47.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.905+0000 7fd53ffff700 1 --2- 172.21.15.98:0/2467699001 >> v2:172.21.15.100:3300/0 conn(0x7fd5480b8b90 0x7fd5480b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:47.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.905+0000 7fd5477fe700 1 -- 172.21.15.98:0/2467699001 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5480b87c0 msgr2=0x7fd5480cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:47.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.905+0000 7fd5477fe700 1 --2- 172.21.15.98:0/2467699001 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5480b87c0 0x7fd5480cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.905+0000 7fd5477fe700 1 -- 172.21.15.98:0/2467699001 >> v2:172.21.15.100:3300/0 conn(0x7fd5480b8b90 msgr2=0x7fd5480b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:47.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.905+0000 7fd5477fe700 1 --2- 172.21.15.98:0/2467699001 >> v2:172.21.15.100:3300/0 conn(0x7fd5480b8b90 0x7fd5480b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.905+0000 7fd5477fe700 1 -- 172.21.15.98:0/2467699001 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5480cdc50 con 0x7fd5480b5240 2023-03-03T07:55:47.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.905+0000 7fd5477fe700 1 --2- 172.21.15.98:0/2467699001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5480b5240 0x7fd5480b56a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd5480b5d80 tx=0x7fd54000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78729d06777d7b2c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:47.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.905+0000 7fd5467fc700 1 -- 172.21.15.98:0/2467699001 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54000e070 con 0x7fd5480b5240 2023-03-03T07:55:47.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.905+0000 7fd5467fc700 1 -- 172.21.15.98:0/2467699001 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54000b9c0 con 0x7fd5480b5240 2023-03-03T07:55:47.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.905+0000 7fd5467fc700 1 -- 172.21.15.98:0/2467699001 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54000bd70 con 0x7fd5480b5240 2023-03-03T07:55:47.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.906+0000 7fd54d7ff700 1 -- 172.21.15.98:0/2467699001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5480b5240 msgr2=0x7fd5480b56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:47.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.906+0000 7fd54d7ff700 1 --2- 172.21.15.98:0/2467699001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5480b5240 0x7fd5480b56a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd5480b5d80 tx=0x7fd54000b330 comp rx=0 tx=0).stop 2023-03-03T07:55:47.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.906+0000 7fd54d7ff700 1 -- 172.21.15.98:0/2467699001 shutdown_connections 2023-03-03T07:55:47.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.906+0000 7fd54d7ff700 1 --2- 172.21.15.98:0/2467699001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5480b5240 0x7fd5480b56a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.906+0000 7fd54d7ff700 1 --2- 172.21.15.98:0/2467699001 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5480b87c0 0x7fd5480cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.906+0000 7fd54d7ff700 1 --2- 172.21.15.98:0/2467699001 >> v2:172.21.15.100:3300/0 conn(0x7fd5480b8b90 0x7fd5480b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.906+0000 7fd54d7ff700 1 -- 172.21.15.98:0/2467699001 >> 172.21.15.98:0/2467699001 conn(0x7fd54806aee0 msgr2=0x7fd54806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:47.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.906+0000 7fd54d7ff700 1 -- 172.21.15.98:0/2467699001 shutdown_connections 2023-03-03T07:55:47.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.906+0000 7fd54d7ff700 1 -- 172.21.15.98:0/2467699001 wait complete. 2023-03-03T07:55:47.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.909+0000 7fd54d7ff700 1 Processor -- start 2023-03-03T07:55:47.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.909+0000 7fd54d7ff700 1 -- start start 2023-03-03T07:55:47.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd54d7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5480b5240 0x7fd5480ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:47.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd54d7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5480b87c0 0x7fd5481570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:47.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd54d7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5480b8b90 0x7fd548159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:47.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd54d7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5480cad60 con 0x7fd5480b5240 2023-03-03T07:55:47.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd54d7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5480cb040 con 0x7fd5480b8b90 2023-03-03T07:55:47.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd54d7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5480cb320 con 0x7fd5480b87c0 2023-03-03T07:55:47.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd546ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5480b5240 0x7fd5480ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:47.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd546ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5480b5240 0x7fd5480ccd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45224/0 (socket says 172.21.15.98:45224) 2023-03-03T07:55:47.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd546ffd700 1 -- 172.21.15.98:0/3620923820 learned_addr learned my addr 172.21.15.98:0/3620923820 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:47.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd53ffff700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5480b87c0 0x7fd5481570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:47.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd5477fe700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5480b8b90 0x7fd548159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:47.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd546ffd700 1 -- 172.21.15.98:0/3620923820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5480b87c0 msgr2=0x7fd5481570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:47.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd546ffd700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5480b87c0 0x7fd5481570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd546ffd700 1 -- 172.21.15.98:0/3620923820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5480b8b90 msgr2=0x7fd548159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:47.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd546ffd700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5480b8b90 0x7fd548159fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:47.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd546ffd700 1 -- 172.21.15.98:0/3620923820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd540009b30 con 0x7fd5480b5240 2023-03-03T07:55:47.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd5477fe700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5480b8b90 0x7fd548159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:47.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.911+0000 7fd53ffff700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5480b87c0 0x7fd5481570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:47.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.912+0000 7fd546ffd700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5480b5240 0x7fd5480ccd50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd53800e810 tx=0x7fd53800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:47.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.912+0000 7fd53f7fe700 1 -- 172.21.15.98:0/3620923820 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd538013610 con 0x7fd5480b5240 2023-03-03T07:55:47.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.912+0000 7fd53f7fe700 1 -- 172.21.15.98:0/3620923820 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd538013770 con 0x7fd5480b5240 2023-03-03T07:55:47.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.912+0000 7fd53f7fe700 1 -- 172.21.15.98:0/3620923820 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd538013b00 con 0x7fd5480b5240 2023-03-03T07:55:47.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.912+0000 7fd54d7ff700 1 -- 172.21.15.98:0/3620923820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5480cdf30 con 0x7fd5480b5240 2023-03-03T07:55:47.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.912+0000 7fd54d7ff700 1 -- 172.21.15.98:0/3620923820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd548073a80 con 0x7fd5480b5240 2023-03-03T07:55:47.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.913+0000 7fd54d7ff700 1 -- 172.21.15.98:0/3620923820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5480ce0c0 con 0x7fd5480b5240 2023-03-03T07:55:47.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.915+0000 7fd53f7fe700 1 -- 172.21.15.98:0/3620923820 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd538023070 con 0x7fd5480b5240 2023-03-03T07:55:47.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.916+0000 7fd53f7fe700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5280682f0 0x7fd52806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.916+0000 7fd53f7fe700 1 -- 172.21.15.98:0/3620923820 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd5380859b0 con 0x7fd5480b5240 2023-03-03T07:55:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.916+0000 7fd53ffff700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5280682f0 0x7fd52806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.917+0000 7fd53ffff700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5280682f0 0x7fd52806a7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd5480b38a0 tx=0x7fd530008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:47.920+0000 7fd53f7fe700 1 -- 172.21.15.98:0/3620923820 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd538052120 con 0x7fd5480b5240 2023-03-03T07:55:48.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.118+0000 7fd54d7ff700 1 -- 172.21.15.98:0/3620923820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5480729b0 con 0x7fd5480b5240 2023-03-03T07:55:48.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.119+0000 7fd53f7fe700 1 -- 172.21.15.98:0/3620923820 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17907 (secure 0 0 0) 0x7fd538009440 con 0x7fd5480b5240 2023-03-03T07:55:48.120 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:48.121 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:55:48.099844+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:55:48.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.122+0000 7fd54d7ff700 1 -- 172.21.15.98:0/3620923820 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5280682f0 msgr2=0x7fd52806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:48.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.122+0000 7fd54d7ff700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5280682f0 0x7fd52806a7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd5480b38a0 tx=0x7fd530008040 comp rx=0 tx=0).stop 2023-03-03T07:55:48.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.122+0000 7fd54d7ff700 1 -- 172.21.15.98:0/3620923820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5480b5240 msgr2=0x7fd5480ccd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:48.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.122+0000 7fd54d7ff700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5480b5240 0x7fd5480ccd50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd53800e810 tx=0x7fd53800ebd0 comp rx=0 tx=0).stop 2023-03-03T07:55:48.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.122+0000 7fd54d7ff700 1 -- 172.21.15.98:0/3620923820 shutdown_connections 2023-03-03T07:55:48.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.122+0000 7fd54d7ff700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5280682f0 0x7fd52806a7a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.122+0000 7fd54d7ff700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5480b5240 0x7fd5480ccd50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.122+0000 7fd54d7ff700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5480b8b90 0x7fd548159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.122+0000 7fd54d7ff700 1 --2- 172.21.15.98:0/3620923820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5480b87c0 0x7fd5481570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.122+0000 7fd54d7ff700 1 -- 172.21.15.98:0/3620923820 >> 172.21.15.98:0/3620923820 conn(0x7fd54806aee0 msgr2=0x7fd5480d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.122+0000 7fd54d7ff700 1 -- 172.21.15.98:0/3620923820 shutdown_connections 2023-03-03T07:55:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.122+0000 7fd54d7ff700 1 -- 172.21.15.98:0/3620923820 wait complete. 2023-03-03T07:55:48.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-03-03T07:55:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.398+0000 7f80b53ff700 1 Processor -- start 2023-03-03T07:55:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.398+0000 7f80b53ff700 1 -- start start 2023-03-03T07:55:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.399+0000 7f80b53ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80b00b8d10 0x7f80b00cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.399+0000 7f80b53ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f80b00b5310 0x7f80b00d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.399+0000 7f80b53ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b00cd7d0 con 0x7f80b00b26d0 2023-03-03T07:55:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.399+0000 7f80b53ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b00b90e0 con 0x7f80b00b8d10 2023-03-03T07:55:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.399+0000 7f80b53ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b00cda20 con 0x7f80b00b5310 2023-03-03T07:55:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.400+0000 7f80aeffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80b00b8d10 0x7f80b00cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.400+0000 7f80ae7fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f80b00b26d0 0x7f80b00b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60254/0 (socket says 172.21.15.98:60254) 2023-03-03T07:55:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.400+0000 7f80ae7fc700 1 -- 172.21.15.98:0/1517176673 learned_addr learned my addr 172.21.15.98:0/1517176673 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.400+0000 7f80af7fe700 1 --2- 172.21.15.98:0/1517176673 >> v2:172.21.15.100:3300/0 conn(0x7f80b00b5310 0x7f80b00d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.400+0000 7f80aeffd700 1 -- 172.21.15.98:0/1517176673 >> v2:172.21.15.100:3300/0 conn(0x7f80b00b5310 msgr2=0x7f80b00d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.400+0000 7f80aeffd700 1 --2- 172.21.15.98:0/1517176673 >> v2:172.21.15.100:3300/0 conn(0x7f80b00b5310 0x7f80b00d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.400+0000 7f80aeffd700 1 -- 172.21.15.98:0/1517176673 >> v1:172.21.15.98:6789/0 conn(0x7f80b00b26d0 legacy=0x7f80b00b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.400+0000 7f80aeffd700 1 -- 172.21.15.98:0/1517176673 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b00cdd00 con 0x7f80b00b8d10 2023-03-03T07:55:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.400+0000 7f80adffb700 1 -- 172.21.15.98:0/1517176673 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133829405 0 0) 0x7f80b00cd7d0 con 0x7f80b00b26d0 2023-03-03T07:55:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.400+0000 7f80af7fe700 1 --2- 172.21.15.98:0/1517176673 >> v2:172.21.15.100:3300/0 conn(0x7f80b00b5310 0x7f80b00d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.400+0000 7f80aeffd700 1 --2- 172.21.15.98:0/1517176673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80b00b8d10 0x7f80b00cffc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f80b006b740 tx=0x7f80a0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7174be65895ad20 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.401+0000 7f80adffb700 1 -- 172.21.15.98:0/1517176673 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a0011830 con 0x7f80b00b8d10 2023-03-03T07:55:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.401+0000 7f80adffb700 1 -- 172.21.15.98:0/1517176673 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a0011990 con 0x7f80b00b8d10 2023-03-03T07:55:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.401+0000 7f80adffb700 1 -- 172.21.15.98:0/1517176673 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a0011d20 con 0x7f80b00b8d10 2023-03-03T07:55:48.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.401+0000 7f80b53ff700 1 -- 172.21.15.98:0/1517176673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80b00b8d10 msgr2=0x7f80b00cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:48.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.401+0000 7f80b53ff700 1 --2- 172.21.15.98:0/1517176673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80b00b8d10 0x7f80b00cffc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f80b006b740 tx=0x7f80a0009b20 comp rx=0 tx=0).stop 2023-03-03T07:55:48.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.401+0000 7f80b53ff700 1 -- 172.21.15.98:0/1517176673 shutdown_connections 2023-03-03T07:55:48.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.401+0000 7f80b53ff700 1 --2- 172.21.15.98:0/1517176673 >> v2:172.21.15.100:3300/0 conn(0x7f80b00b5310 0x7f80b00d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:48.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.401+0000 7f80b53ff700 1 --2- 172.21.15.98:0/1517176673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80b00b8d10 0x7f80b00cffc0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:48.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.401+0000 7f80b53ff700 1 -- 172.21.15.98:0/1517176673 >> 172.21.15.98:0/1517176673 conn(0x7f80b0073e70 msgr2=0x7f80b006ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:48.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.401+0000 7f80b53ff700 1 -- 172.21.15.98:0/1517176673 shutdown_connections 2023-03-03T07:55:48.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.401+0000 7f80b53ff700 1 -- 172.21.15.98:0/1517176673 wait complete. 2023-03-03T07:55:48.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.403+0000 7f80b53ff700 1 Processor -- start 2023-03-03T07:55:48.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.404+0000 7f80b53ff700 1 -- start start 2023-03-03T07:55:48.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.404+0000 7f80b53ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80b00b26d0 0x7f80b01526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:48.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.404+0000 7f80b53ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80b00b5310 0x7f80b0159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:48.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.405+0000 7f80b53ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80b00b8d10 0x7f80b015f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:48.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.405+0000 7f80b53ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b0152c30 con 0x7f80b00b8d10 2023-03-03T07:55:48.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.405+0000 7f80b53ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b015e450 con 0x7f80b00b5310 2023-03-03T07:55:48.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.405+0000 7f80b53ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b015e730 con 0x7f80b00b26d0 2023-03-03T07:55:48.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.405+0000 7f80af7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80b00b8d10 0x7f80b015f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:48.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.405+0000 7f80af7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80b00b8d10 0x7f80b015f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45234/0 (socket says 172.21.15.98:45234) 2023-03-03T07:55:48.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.405+0000 7f80af7fe700 1 -- 172.21.15.98:0/2829831392 learned_addr learned my addr 172.21.15.98:0/2829831392 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:48.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.405+0000 7f80ae7fc700 1 --2- 172.21.15.98:0/2829831392 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80b00b5310 0x7f80b0159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:48.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.405+0000 7f80aeffd700 1 --2- 172.21.15.98:0/2829831392 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80b00b26d0 0x7f80b01526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:48.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.405+0000 7f80af7fe700 1 -- 172.21.15.98:0/2829831392 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80b00b26d0 msgr2=0x7f80b01526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:48.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.405+0000 7f80af7fe700 1 --2- 172.21.15.98:0/2829831392 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80b00b26d0 0x7f80b01526f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:48.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.405+0000 7f80af7fe700 1 -- 172.21.15.98:0/2829831392 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80b00b5310 msgr2=0x7f80b0159020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:48.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.405+0000 7f80af7fe700 1 --2- 172.21.15.98:0/2829831392 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80b00b5310 0x7f80b0159020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:48.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.405+0000 7f80af7fe700 1 -- 172.21.15.98:0/2829831392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a00094a0 con 0x7f80b00b8d10 2023-03-03T07:55:48.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.405+0000 7f80aeffd700 1 --2- 172.21.15.98:0/2829831392 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80b00b26d0 0x7f80b01526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:48.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.406+0000 7f80af7fe700 1 --2- 172.21.15.98:0/2829831392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80b00b8d10 0x7f80b015f4f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f80a8007ec0 tx=0x7f80a800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:48.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.406+0000 7f80977fe700 1 -- 172.21.15.98:0/2829831392 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a800de90 con 0x7f80b00b8d10 2023-03-03T07:55:48.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.406+0000 7f80977fe700 1 -- 172.21.15.98:0/2829831392 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a800f040 con 0x7f80b00b8d10 2023-03-03T07:55:48.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.406+0000 7f80977fe700 1 -- 172.21.15.98:0/2829831392 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a8015640 con 0x7f80b00b8d10 2023-03-03T07:55:48.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.406+0000 7f80b53ff700 1 -- 172.21.15.98:0/2829831392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b00cdfe0 con 0x7f80b00b8d10 2023-03-03T07:55:48.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.406+0000 7f80b53ff700 1 -- 172.21.15.98:0/2829831392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b00d0e80 con 0x7f80b00b8d10 2023-03-03T07:55:48.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.407+0000 7f80b53ff700 1 -- 172.21.15.98:0/2829831392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b00d0500 con 0x7f80b00b8d10 2023-03-03T07:55:48.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.408+0000 7f80977fe700 1 -- 172.21.15.98:0/2829831392 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f80a801c030 con 0x7f80b00b8d10 2023-03-03T07:55:48.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.410+0000 7f80977fe700 1 --2- 172.21.15.98:0/2829831392 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f809c0682f0 0x7f809c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:48.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.410+0000 7f80977fe700 1 -- 172.21.15.98:0/2829831392 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f80a8085d90 con 0x7f80b00b8d10 2023-03-03T07:55:48.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.410+0000 7f80aeffd700 1 --2- 172.21.15.98:0/2829831392 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f809c0682f0 0x7f809c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:48.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.411+0000 7f80aeffd700 1 --2- 172.21.15.98:0/2829831392 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f809c0682f0 0x7f809c06a7a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f80a00090b0 tx=0x7f80a0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:48.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.414+0000 7f80977fe700 1 -- 172.21.15.98:0/2829831392 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f80a8052500 con 0x7f80b00b8d10 2023-03-03T07:55:48.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:48.655+0000 7f80b53ff700 1 -- 172.21.15.98:0/2829831392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f80b004e6e0 con 0x7f80b00b8d10 2023-03-03T07:55:49.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.555+0000 7f80977fe700 1 -- 172.21.15.98:0/2829831392 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7f80a8055b20 con 0x7f80b00b8d10 2023-03-03T07:55:49.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.558+0000 7f80b53ff700 1 -- 172.21.15.98:0/2829831392 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f809c0682f0 msgr2=0x7f809c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:49.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.558+0000 7f80b53ff700 1 --2- 172.21.15.98:0/2829831392 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f809c0682f0 0x7f809c06a7a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f80a00090b0 tx=0x7f80a0008030 comp rx=0 tx=0).stop 2023-03-03T07:55:49.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.558+0000 7f80b53ff700 1 -- 172.21.15.98:0/2829831392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80b00b8d10 msgr2=0x7f80b015f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:49.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.558+0000 7f80b53ff700 1 --2- 172.21.15.98:0/2829831392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80b00b8d10 0x7f80b015f4f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f80a8007ec0 tx=0x7f80a800d3b0 comp rx=0 tx=0).stop 2023-03-03T07:55:49.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.558+0000 7f80b53ff700 1 -- 172.21.15.98:0/2829831392 shutdown_connections 2023-03-03T07:55:49.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.558+0000 7f80b53ff700 1 --2- 172.21.15.98:0/2829831392 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f809c0682f0 0x7f809c06a7a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:49.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.558+0000 7f80b53ff700 1 --2- 172.21.15.98:0/2829831392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80b00b8d10 0x7f80b015f4f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:49.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.558+0000 7f80b53ff700 1 --2- 172.21.15.98:0/2829831392 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80b00b26d0 0x7f80b01526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:49.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.558+0000 7f80b53ff700 1 --2- 172.21.15.98:0/2829831392 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80b00b5310 0x7f80b0159020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:49.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.558+0000 7f80b53ff700 1 -- 172.21.15.98:0/2829831392 >> 172.21.15.98:0/2829831392 conn(0x7f80b0073e70 msgr2=0x7f80b00b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:49.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.558+0000 7f80b53ff700 1 -- 172.21.15.98:0/2829831392 shutdown_connections 2023-03-03T07:55:49.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.558+0000 7f80b53ff700 1 -- 172.21.15.98:0/2829831392 wait complete. 2023-03-03T07:55:49.586 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-03-03T07:55:49.586 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-03-03T07:55:49.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.832+0000 7f5c6ffff700 1 Processor -- start 2023-03-03T07:55:49.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.832+0000 7f5c6ffff700 1 -- start start 2023-03-03T07:55:49.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.834+0000 7f5c6ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5c701609b0 0x7f5c70160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:49.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.834+0000 7f5c6ffff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5c700cf570 0x7f5c7015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:49.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.834+0000 7f5c6ffff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c701612c0 con 0x7f5c7015db30 2023-03-03T07:55:49.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.834+0000 7f5c6ffff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c700cebb0 con 0x7f5c701609b0 2023-03-03T07:55:49.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.834+0000 7f5c6ffff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c700cd780 con 0x7f5c700cf570 2023-03-03T07:55:49.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.834+0000 7f5c6effd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5c701609b0 0x7f5c70160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:49.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.834+0000 7f5c6effd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5c701609b0 0x7f5c70160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45248/0 (socket says 172.21.15.98:45248) 2023-03-03T07:55:49.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.834+0000 7f5c6effd700 1 -- 172.21.15.98:0/3678245969 learned_addr learned my addr 172.21.15.98:0/3678245969 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:49.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.834+0000 7f5c6e7fc700 1 --2- 172.21.15.98:0/3678245969 >> v2:172.21.15.100:3300/0 conn(0x7f5c700cf570 0x7f5c7015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:49.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.835+0000 7f5c6effd700 1 -- 172.21.15.98:0/3678245969 >> v2:172.21.15.100:3300/0 conn(0x7f5c700cf570 msgr2=0x7f5c7015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:49.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.835+0000 7f5c6effd700 1 --2- 172.21.15.98:0/3678245969 >> v2:172.21.15.100:3300/0 conn(0x7f5c700cf570 0x7f5c7015d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:49.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.835+0000 7f5c6effd700 1 -- 172.21.15.98:0/3678245969 >> v1:172.21.15.100:6789/0 conn(0x7f5c7015db30 legacy=0x7f5c700b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:49.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.835+0000 7f5c6effd700 1 -- 172.21.15.98:0/3678245969 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c700cda60 con 0x7f5c701609b0 2023-03-03T07:55:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.835+0000 7f5c6effd700 1 --2- 172.21.15.98:0/3678245969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c701609b0 0x7f5c70160d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5c700cf0c0 tx=0x7f5c60009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657b62e37946b09a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.835+0000 7f5c6dffb700 1 -- 172.21.15.98:0/3678245969 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60011830 con 0x7f5c701609b0 2023-03-03T07:55:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.835+0000 7f5c6dffb700 1 -- 172.21.15.98:0/3678245969 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c60011990 con 0x7f5c701609b0 2023-03-03T07:55:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.835+0000 7f5c6dffb700 1 -- 172.21.15.98:0/3678245969 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60011d20 con 0x7f5c701609b0 2023-03-03T07:55:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.836+0000 7f5c6ffff700 1 -- 172.21.15.98:0/3678245969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c701609b0 msgr2=0x7f5c70160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.836+0000 7f5c6ffff700 1 --2- 172.21.15.98:0/3678245969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c701609b0 0x7f5c70160d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5c700cf0c0 tx=0x7f5c60009bd0 comp rx=0 tx=0).stop 2023-03-03T07:55:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.836+0000 7f5c6ffff700 1 -- 172.21.15.98:0/3678245969 shutdown_connections 2023-03-03T07:55:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.836+0000 7f5c6ffff700 1 --2- 172.21.15.98:0/3678245969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c701609b0 0x7f5c70160d80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.836+0000 7f5c6ffff700 1 --2- 172.21.15.98:0/3678245969 >> v2:172.21.15.100:3300/0 conn(0x7f5c700cf570 0x7f5c7015d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.836+0000 7f5c6ffff700 1 -- 172.21.15.98:0/3678245969 >> 172.21.15.98:0/3678245969 conn(0x7f5c7006b880 msgr2=0x7f5c700cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.836+0000 7f5c6ffff700 1 -- 172.21.15.98:0/3678245969 shutdown_connections 2023-03-03T07:55:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.836+0000 7f5c6ffff700 1 -- 172.21.15.98:0/3678245969 wait complete. 2023-03-03T07:55:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.838+0000 7f5c6ffff700 1 Processor -- start 2023-03-03T07:55:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.838+0000 7f5c6ffff700 1 -- start start 2023-03-03T07:55:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.839+0000 7f5c6ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5c700cf570 0x7f5c701e6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.839+0000 7f5c6ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c7015db30 0x7f5c700b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:49.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.839+0000 7f5c6ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c701609b0 0x7f5c700b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:49.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.839+0000 7f5c6ffff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700b8e80 con 0x7f5c701609b0 2023-03-03T07:55:49.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.839+0000 7f5c6ffff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c701e66d0 con 0x7f5c700cf570 2023-03-03T07:55:49.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.839+0000 7f5c6ffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c701e6900 con 0x7f5c7015db30 2023-03-03T07:55:49.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.840+0000 7f5c6f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c701609b0 0x7f5c700b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:49.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.840+0000 7f5c6f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c701609b0 0x7f5c700b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45258/0 (socket says 172.21.15.98:45258) 2023-03-03T07:55:49.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.840+0000 7f5c6f7fe700 1 -- 172.21.15.98:0/969724879 learned_addr learned my addr 172.21.15.98:0/969724879 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:49.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.840+0000 7f5c6effd700 1 --2- 172.21.15.98:0/969724879 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5c700cf570 0x7f5c701e6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:49.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.840+0000 7f5c6e7fc700 1 --2- 172.21.15.98:0/969724879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c7015db30 0x7f5c700b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:49.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.840+0000 7f5c6f7fe700 1 -- 172.21.15.98:0/969724879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c7015db30 msgr2=0x7f5c700b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:49.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.840+0000 7f5c6f7fe700 1 --2- 172.21.15.98:0/969724879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c7015db30 0x7f5c700b3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:49.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.840+0000 7f5c6f7fe700 1 -- 172.21.15.98:0/969724879 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5c700cf570 msgr2=0x7f5c701e6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:49.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.840+0000 7f5c6f7fe700 1 --2- 172.21.15.98:0/969724879 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5c700cf570 0x7f5c701e6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:49.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.840+0000 7f5c6f7fe700 1 -- 172.21.15.98:0/969724879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c60009550 con 0x7f5c701609b0 2023-03-03T07:55:49.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.840+0000 7f5c6f7fe700 1 --2- 172.21.15.98:0/969724879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c701609b0 0x7f5c700b44d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800cbc0 tx=0x7f5c6800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:49.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.841+0000 7f5c577fe700 1 -- 172.21.15.98:0/969724879 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c68003c00 con 0x7f5c701609b0 2023-03-03T07:55:49.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.841+0000 7f5c577fe700 1 -- 172.21.15.98:0/969724879 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c68003d60 con 0x7f5c701609b0 2023-03-03T07:55:49.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.841+0000 7f5c577fe700 1 -- 172.21.15.98:0/969724879 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6800ab00 con 0x7f5c701609b0 2023-03-03T07:55:49.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.841+0000 7f5c6ffff700 1 -- 172.21.15.98:0/969724879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7015cdf0 con 0x7f5c701609b0 2023-03-03T07:55:49.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.841+0000 7f5c6ffff700 1 -- 172.21.15.98:0/969724879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c700cdd40 con 0x7f5c701609b0 2023-03-03T07:55:49.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.842+0000 7f5c6ffff700 1 -- 172.21.15.98:0/969724879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c7015d0d0 con 0x7f5c701609b0 2023-03-03T07:55:49.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.843+0000 7f5c577fe700 1 -- 172.21.15.98:0/969724879 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5c68010050 con 0x7f5c701609b0 2023-03-03T07:55:49.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.844+0000 7f5c577fe700 1 --2- 172.21.15.98:0/969724879 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5c5c0682f0 0x7f5c5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:49.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.844+0000 7f5c577fe700 1 -- 172.21.15.98:0/969724879 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5c68085c30 con 0x7f5c701609b0 2023-03-03T07:55:49.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.845+0000 7f5c6effd700 1 --2- 172.21.15.98:0/969724879 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5c5c0682f0 0x7f5c5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:49.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.845+0000 7f5c6effd700 1 --2- 172.21.15.98:0/969724879 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5c5c0682f0 0x7f5c5c06a7a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5c60000c00 tx=0x7f5c60008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:49.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:49.849+0000 7f5c577fe700 1 -- 172.21.15.98:0/969724879 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5c68052340 con 0x7f5c701609b0 2023-03-03T07:55:50.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.083+0000 7f5c6ffff700 1 -- 172.21.15.98:0/969724879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5c70068290 con 0x7f5c701609b0 2023-03-03T07:55:50.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.562+0000 7f5c577fe700 1 -- 172.21.15.98:0/969724879 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5c68055960 con 0x7f5c701609b0 2023-03-03T07:55:50.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.565+0000 7f5c6ffff700 1 -- 172.21.15.98:0/969724879 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5c5c0682f0 msgr2=0x7f5c5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:50.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.565+0000 7f5c6ffff700 1 --2- 172.21.15.98:0/969724879 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5c5c0682f0 0x7f5c5c06a7a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5c60000c00 tx=0x7f5c60008030 comp rx=0 tx=0).stop 2023-03-03T07:55:50.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.565+0000 7f5c6ffff700 1 -- 172.21.15.98:0/969724879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c701609b0 msgr2=0x7f5c700b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:50.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.565+0000 7f5c6ffff700 1 --2- 172.21.15.98:0/969724879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c701609b0 0x7f5c700b44d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800cbc0 tx=0x7f5c6800ced0 comp rx=0 tx=0).stop 2023-03-03T07:55:50.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.565+0000 7f5c6ffff700 1 -- 172.21.15.98:0/969724879 shutdown_connections 2023-03-03T07:55:50.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.565+0000 7f5c6ffff700 1 --2- 172.21.15.98:0/969724879 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5c5c0682f0 0x7f5c5c06a7a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:50.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.565+0000 7f5c6ffff700 1 --2- 172.21.15.98:0/969724879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c701609b0 0x7f5c700b44d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:50.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.565+0000 7f5c6ffff700 1 --2- 172.21.15.98:0/969724879 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5c700cf570 0x7f5c701e6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:50.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.565+0000 7f5c6ffff700 1 --2- 172.21.15.98:0/969724879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c7015db30 0x7f5c700b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:50.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.565+0000 7f5c6ffff700 1 -- 172.21.15.98:0/969724879 >> 172.21.15.98:0/969724879 conn(0x7f5c7006b880 msgr2=0x7f5c7015f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:50.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.565+0000 7f5c6ffff700 1 -- 172.21.15.98:0/969724879 shutdown_connections 2023-03-03T07:55:50.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.565+0000 7f5c6ffff700 1 -- 172.21.15.98:0/969724879 wait complete. 2023-03-03T07:55:50.595 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-03-03T07:55:50.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:55:50.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.839+0000 7f05953ff700 1 Processor -- start 2023-03-03T07:55:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.839+0000 7f05953ff700 1 -- start start 2023-03-03T07:55:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.840+0000 7f05953ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f05900b87c0 0x7f05900cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.840+0000 7f05953ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900b8b90 0x7f05900b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.840+0000 7f05953ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05900d0ef0 con 0x7f05900b5240 2023-03-03T07:55:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.840+0000 7f05953ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05900d0600 con 0x7f05900b87c0 2023-03-03T07:55:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.840+0000 7f05953ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05900cd970 con 0x7f05900b8b90 2023-03-03T07:55:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.841+0000 7f058effd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f05900b87c0 0x7f05900cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.841+0000 7f058f7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f05900b5240 0x7f05901605b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60258/0 (socket says 172.21.15.98:60258) 2023-03-03T07:55:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.841+0000 7f058f7fe700 1 -- 172.21.15.98:0/2764227396 learned_addr learned my addr 172.21.15.98:0/2764227396 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.841+0000 7f058e7fc700 1 --2- 172.21.15.98:0/2764227396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900b8b90 0x7f05900b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.841+0000 7f058dffb700 1 -- 172.21.15.98:0/2764227396 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910387640 0 0) 0x7f05900d0ef0 con 0x7f05900b5240 2023-03-03T07:55:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.841+0000 7f058dffb700 1 -- 172.21.15.98:0/2764227396 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05780035b0 con 0x7f05900b5240 2023-03-03T07:55:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.841+0000 7f058effd700 1 -- 172.21.15.98:0/2764227396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900b8b90 msgr2=0x7f05900b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.841+0000 7f058effd700 1 --2- 172.21.15.98:0/2764227396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900b8b90 0x7f05900b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.841+0000 7f058effd700 1 -- 172.21.15.98:0/2764227396 >> v1:172.21.15.98:6789/0 conn(0x7f05900b5240 legacy=0x7f05901605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.841+0000 7f058effd700 1 -- 172.21.15.98:0/2764227396 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05900cdc50 con 0x7f05900b87c0 2023-03-03T07:55:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.841+0000 7f058e7fc700 1 --2- 172.21.15.98:0/2764227396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900b8b90 0x7f05900b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.842+0000 7f058effd700 1 --2- 172.21.15.98:0/2764227396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05900b87c0 0x7f05900cffe0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0580009840 tx=0x7f0580009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ac0cf6d05f2dbd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.842+0000 7f058dffb700 1 -- 172.21.15.98:0/2764227396 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580011830 con 0x7f05900b87c0 2023-03-03T07:55:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.842+0000 7f058dffb700 1 -- 172.21.15.98:0/2764227396 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0580011990 con 0x7f05900b87c0 2023-03-03T07:55:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.842+0000 7f058dffb700 1 -- 172.21.15.98:0/2764227396 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580011d20 con 0x7f05900b87c0 2023-03-03T07:55:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.842+0000 7f05953ff700 1 -- 172.21.15.98:0/2764227396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05900b87c0 msgr2=0x7f05900cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.842+0000 7f05953ff700 1 --2- 172.21.15.98:0/2764227396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05900b87c0 0x7f05900cffe0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0580009840 tx=0x7f0580009b50 comp rx=0 tx=0).stop 2023-03-03T07:55:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.843+0000 7f05953ff700 1 -- 172.21.15.98:0/2764227396 shutdown_connections 2023-03-03T07:55:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.843+0000 7f05953ff700 1 --2- 172.21.15.98:0/2764227396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05900b87c0 0x7f05900cffe0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.843+0000 7f05953ff700 1 --2- 172.21.15.98:0/2764227396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900b8b90 0x7f05900b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.843+0000 7f05953ff700 1 -- 172.21.15.98:0/2764227396 >> 172.21.15.98:0/2764227396 conn(0x7f059006aee0 msgr2=0x7f059006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.843+0000 7f05953ff700 1 -- 172.21.15.98:0/2764227396 shutdown_connections 2023-03-03T07:55:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.843+0000 7f05953ff700 1 -- 172.21.15.98:0/2764227396 wait complete. 2023-03-03T07:55:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.845+0000 7f05953ff700 1 Processor -- start 2023-03-03T07:55:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.845+0000 7f05953ff700 1 -- start start 2023-03-03T07:55:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.846+0000 7f05953ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900b5240 0x7f059015d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.846+0000 7f05953ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05900b87c0 0x7f059015d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.846+0000 7f05953ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05900b8b90 0x7f05901f2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.846+0000 7f05953ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059015dd60 con 0x7f05900b87c0 2023-03-03T07:55:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.846+0000 7f05953ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059015eb30 con 0x7f05900b8b90 2023-03-03T07:55:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.846+0000 7f05953ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059015dfb0 con 0x7f05900b5240 2023-03-03T07:55:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.846+0000 7f058e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05900b87c0 0x7f059015d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.846+0000 7f058e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05900b87c0 0x7f059015d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45266/0 (socket says 172.21.15.98:45266) 2023-03-03T07:55:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.846+0000 7f058e7fc700 1 -- 172.21.15.98:0/1329001365 learned_addr learned my addr 172.21.15.98:0/1329001365 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.846+0000 7f058effd700 1 --2- 172.21.15.98:0/1329001365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900b5240 0x7f059015d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.846+0000 7f058f7fe700 1 --2- 172.21.15.98:0/1329001365 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05900b8b90 0x7f05901f2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.847+0000 7f058e7fc700 1 -- 172.21.15.98:0/1329001365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900b5240 msgr2=0x7f059015d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.847+0000 7f058e7fc700 1 --2- 172.21.15.98:0/1329001365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900b5240 0x7f059015d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.847+0000 7f058e7fc700 1 -- 172.21.15.98:0/1329001365 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05900b8b90 msgr2=0x7f05901f2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.847+0000 7f058e7fc700 1 --2- 172.21.15.98:0/1329001365 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05900b8b90 0x7f05901f2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.847+0000 7f058e7fc700 1 -- 172.21.15.98:0/1329001365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05800094a0 con 0x7f05900b87c0 2023-03-03T07:55:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.847+0000 7f058f7fe700 1 --2- 172.21.15.98:0/1329001365 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05900b8b90 0x7f05901f2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.847+0000 7f058e7fc700 1 --2- 172.21.15.98:0/1329001365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05900b87c0 0x7f059015d820 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f058400e810 tx=0x7f058400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.847+0000 7f05777fe700 1 -- 172.21.15.98:0/1329001365 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058400c020 con 0x7f05900b87c0 2023-03-03T07:55:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.847+0000 7f05777fe700 1 -- 172.21.15.98:0/1329001365 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0584004250 con 0x7f05900b87c0 2023-03-03T07:55:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.847+0000 7f05953ff700 1 -- 172.21.15.98:0/1329001365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05900cdf30 con 0x7f05900b87c0 2023-03-03T07:55:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.847+0000 7f05777fe700 1 -- 172.21.15.98:0/1329001365 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05840036a0 con 0x7f05900b87c0 2023-03-03T07:55:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.847+0000 7f05953ff700 1 -- 172.21.15.98:0/1329001365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0590073a80 con 0x7f05900b87c0 2023-03-03T07:55:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.849+0000 7f05953ff700 1 -- 172.21.15.98:0/1329001365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05900ce0c0 con 0x7f05900b87c0 2023-03-03T07:55:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.850+0000 7f05777fe700 1 -- 172.21.15.98:0/1329001365 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0584003800 con 0x7f05900b87c0 2023-03-03T07:55:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.852+0000 7f05777fe700 1 --2- 172.21.15.98:0/1329001365 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f057c0682f0 0x7f057c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.852+0000 7f05777fe700 1 -- 172.21.15.98:0/1329001365 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0584085a00 con 0x7f05900b87c0 2023-03-03T07:55:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.852+0000 7f058effd700 1 --2- 172.21.15.98:0/1329001365 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f057c0682f0 0x7f057c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.853+0000 7f058effd700 1 --2- 172.21.15.98:0/1329001365 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f057c0682f0 0x7f057c06a7a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0580000c00 tx=0x7f0580008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:50.856+0000 7f05777fe700 1 -- 172.21.15.98:0/1329001365 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0584052170 con 0x7f05900b87c0 2023-03-03T07:55:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.089+0000 7f05953ff700 1 -- 172.21.15.98:0/1329001365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05900729b0 con 0x7f05900b87c0 2023-03-03T07:55:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.090+0000 7f05777fe700 1 -- 172.21.15.98:0/1329001365 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4639 (secure 0 0 0) 0x7f058400f040 con 0x7f05900b87c0 2023-03-03T07:55:51.092 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:51.093 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/246835374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":246835374},{"type":"v1","addr":"172.21.15.98:6836","nonce":246835374}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6833/959944231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":959944231},{"type":"v1","addr":"172.21.15.100:6833","nonce":959944231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/872788798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":872788798},{"type":"v1","addr":"172.21.15.100:6835","nonce":872788798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:55:47.097221+0000","modified":"2023-03-03T07:55:50.555384+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/3969084152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3969084152},{"type":"v1","addr":"172.21.15.98:6837","nonce":3969084152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:55:51.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.093+0000 7f05953ff700 1 -- 172.21.15.98:0/1329001365 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f057c0682f0 msgr2=0x7f057c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:51.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.093+0000 7f05953ff700 1 --2- 172.21.15.98:0/1329001365 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f057c0682f0 0x7f057c06a7a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0580000c00 tx=0x7f0580008030 comp rx=0 tx=0).stop 2023-03-03T07:55:51.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.093+0000 7f05953ff700 1 -- 172.21.15.98:0/1329001365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05900b87c0 msgr2=0x7f059015d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:51.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.093+0000 7f05953ff700 1 --2- 172.21.15.98:0/1329001365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05900b87c0 0x7f059015d820 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f058400e810 tx=0x7f058400eb20 comp rx=0 tx=0).stop 2023-03-03T07:55:51.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.093+0000 7f05953ff700 1 -- 172.21.15.98:0/1329001365 shutdown_connections 2023-03-03T07:55:51.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.093+0000 7f05953ff700 1 --2- 172.21.15.98:0/1329001365 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f057c0682f0 0x7f057c06a7a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.093+0000 7f05953ff700 1 --2- 172.21.15.98:0/1329001365 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05900b8b90 0x7f05901f2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.093+0000 7f05953ff700 1 --2- 172.21.15.98:0/1329001365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900b5240 0x7f059015d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.093+0000 7f05953ff700 1 --2- 172.21.15.98:0/1329001365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05900b87c0 0x7f059015d820 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.093+0000 7f05953ff700 1 -- 172.21.15.98:0/1329001365 >> 172.21.15.98:0/1329001365 conn(0x7f059006aee0 msgr2=0x7f05900d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:51.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.093+0000 7f05953ff700 1 -- 172.21.15.98:0/1329001365 shutdown_connections 2023-03-03T07:55:51.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.093+0000 7f05953ff700 1 -- 172.21.15.98:0/1329001365 wait complete. 2023-03-03T07:55:51.097 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 7 2023-03-03T07:55:51.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:55:51.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.364+0000 7f8307dff700 1 Processor -- start 2023-03-03T07:55:51.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.364+0000 7f8307dff700 1 -- start start 2023-03-03T07:55:51.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.365+0000 7f8307dff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83000cd8d0 0x7f83000b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:51.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.365+0000 7f8307dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83000b6610 0x7f83000cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:51.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.365+0000 7f8307dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83000cdd70 con 0x7f830015cc00 2023-03-03T07:55:51.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.366+0000 7f8307dff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83000ce040 con 0x7f83000cd8d0 2023-03-03T07:55:51.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.366+0000 7f8307dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8300073be0 con 0x7f83000b6610 2023-03-03T07:55:51.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.366+0000 7f8305b9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f830015cc00 0x7f83000d27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60274/0 (socket says 172.21.15.98:60274) 2023-03-03T07:55:51.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.366+0000 7f8305b9b700 1 -- 172.21.15.98:0/1709886941 learned_addr learned my addr 172.21.15.98:0/1709886941 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:51.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.366+0000 7f830539a700 1 --2- 172.21.15.98:0/1709886941 >> v2:172.21.15.98:3300/0 conn(0x7f83000cd8d0 0x7f83000b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:51.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.366+0000 7f830639c700 1 --2- 172.21.15.98:0/1709886941 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83000b6610 0x7f83000cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:51.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.366+0000 7f8304b99700 1 -- 172.21.15.98:0/1709886941 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64213906 0 0) 0x7f83000cdd70 con 0x7f830015cc00 2023-03-03T07:55:51.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.366+0000 7f8304b99700 1 -- 172.21.15.98:0/1709886941 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82ec0035b0 con 0x7f830015cc00 2023-03-03T07:55:51.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.366+0000 7f830539a700 1 -- 172.21.15.98:0/1709886941 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83000b6610 msgr2=0x7f83000cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:51.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.366+0000 7f830539a700 1 --2- 172.21.15.98:0/1709886941 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83000b6610 0x7f83000cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.366+0000 7f830539a700 1 -- 172.21.15.98:0/1709886941 >> v1:172.21.15.98:6789/0 conn(0x7f830015cc00 legacy=0x7f83000d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:51.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.366+0000 7f830539a700 1 -- 172.21.15.98:0/1709886941 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83000d1a00 con 0x7f83000cd8d0 2023-03-03T07:55:51.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.367+0000 7f830539a700 1 --2- 172.21.15.98:0/1709886941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83000cd8d0 0x7f83000b60d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009840 tx=0x7f82f0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf289a3868e36e5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:51.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.367+0000 7f8304b99700 1 -- 172.21.15.98:0/1709886941 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f0011830 con 0x7f83000cd8d0 2023-03-03T07:55:51.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.367+0000 7f8304b99700 1 -- 172.21.15.98:0/1709886941 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f0011990 con 0x7f83000cd8d0 2023-03-03T07:55:51.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.367+0000 7f8304b99700 1 -- 172.21.15.98:0/1709886941 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f0011d20 con 0x7f83000cd8d0 2023-03-03T07:55:51.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.368+0000 7f8307dff700 1 -- 172.21.15.98:0/1709886941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83000cd8d0 msgr2=0x7f83000b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:51.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.368+0000 7f8307dff700 1 --2- 172.21.15.98:0/1709886941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83000cd8d0 0x7f83000b60d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009840 tx=0x7f82f0009b50 comp rx=0 tx=0).stop 2023-03-03T07:55:51.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.368+0000 7f8307dff700 1 -- 172.21.15.98:0/1709886941 shutdown_connections 2023-03-03T07:55:51.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.368+0000 7f8307dff700 1 --2- 172.21.15.98:0/1709886941 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83000b6610 0x7f83000cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.368+0000 7f8307dff700 1 --2- 172.21.15.98:0/1709886941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83000cd8d0 0x7f83000b60d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.368+0000 7f8307dff700 1 -- 172.21.15.98:0/1709886941 >> 172.21.15.98:0/1709886941 conn(0x7f830006aee0 msgr2=0x7f830006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:51.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.368+0000 7f8307dff700 1 -- 172.21.15.98:0/1709886941 shutdown_connections 2023-03-03T07:55:51.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.368+0000 7f8307dff700 1 -- 172.21.15.98:0/1709886941 wait complete. 2023-03-03T07:55:51.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.370+0000 7f8307dff700 1 Processor -- start 2023-03-03T07:55:51.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.370+0000 7f8307dff700 1 -- start start 2023-03-03T07:55:51.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.371+0000 7f8307dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83000b6610 0x7f830014f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:51.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.371+0000 7f8307dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83000cd8d0 0x7f830015f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:51.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.371+0000 7f8307dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f830015cc00 0x7f830015f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:51.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.371+0000 7f8307dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830014fdb0 con 0x7f83000b6610 2023-03-03T07:55:51.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.371+0000 7f8307dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83000c7fb0 con 0x7f830015cc00 2023-03-03T07:55:51.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.371+0000 7f8307dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830015fce0 con 0x7f83000cd8d0 2023-03-03T07:55:51.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.371+0000 7f8305b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83000b6610 0x7f830014f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:51.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.372+0000 7f8305b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83000b6610 0x7f830014f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45288/0 (socket says 172.21.15.98:45288) 2023-03-03T07:55:51.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.372+0000 7f8305b9b700 1 -- 172.21.15.98:0/940067061 learned_addr learned my addr 172.21.15.98:0/940067061 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:51.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.372+0000 7f830539a700 1 --2- 172.21.15.98:0/940067061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83000cd8d0 0x7f830015f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:51.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.372+0000 7f830639c700 1 --2- 172.21.15.98:0/940067061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f830015cc00 0x7f830015f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:51.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.372+0000 7f8305b9b700 1 -- 172.21.15.98:0/940067061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83000cd8d0 msgr2=0x7f830015f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:51.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.372+0000 7f8305b9b700 1 --2- 172.21.15.98:0/940067061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83000cd8d0 0x7f830015f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.372+0000 7f8305b9b700 1 -- 172.21.15.98:0/940067061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f830015cc00 msgr2=0x7f830015f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:51.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.372+0000 7f8305b9b700 1 --2- 172.21.15.98:0/940067061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f830015cc00 0x7f830015f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.372+0000 7f8305b9b700 1 -- 172.21.15.98:0/940067061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f00094a0 con 0x7f83000b6610 2023-03-03T07:55:51.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.372+0000 7f8305b9b700 1 --2- 172.21.15.98:0/940067061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83000b6610 0x7f830014f870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00c730 tx=0x7f82fc00caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:51.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.372+0000 7f82f67fc700 1 -- 172.21.15.98:0/940067061 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc005d70 con 0x7f83000b6610 2023-03-03T07:55:51.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.373+0000 7f82f67fc700 1 -- 172.21.15.98:0/940067061 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82fc0070d0 con 0x7f83000b6610 2023-03-03T07:55:51.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.373+0000 7f82f67fc700 1 -- 172.21.15.98:0/940067061 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc007440 con 0x7f83000b6610 2023-03-03T07:55:51.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.373+0000 7f8307dff700 1 -- 172.21.15.98:0/940067061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83000d1fc0 con 0x7f83000b6610 2023-03-03T07:55:51.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.373+0000 7f8307dff700 1 -- 172.21.15.98:0/940067061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83000d1ce0 con 0x7f83000b6610 2023-03-03T07:55:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.374+0000 7f8307dff700 1 -- 172.21.15.98:0/940067061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83000d22a0 con 0x7f83000b6610 2023-03-03T07:55:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.375+0000 7f82f67fc700 1 -- 172.21.15.98:0/940067061 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f82fc019030 con 0x7f83000b6610 2023-03-03T07:55:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.376+0000 7f82f67fc700 1 --2- 172.21.15.98:0/940067061 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f82e40682f0 0x7f82e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.376+0000 7f82f67fc700 1 -- 172.21.15.98:0/940067061 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f82fc0850a0 con 0x7f83000b6610 2023-03-03T07:55:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.376+0000 7f830539a700 1 --2- 172.21.15.98:0/940067061 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f82e40682f0 0x7f82e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.377+0000 7f830539a700 1 --2- 172.21.15.98:0/940067061 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f82e40682f0 0x7f82e406a7a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f82f0000c00 tx=0x7f82f0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.380+0000 7f82f67fc700 1 -- 172.21.15.98:0/940067061 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f82fc04b5e0 con 0x7f83000b6610 2023-03-03T07:55:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.575+0000 7f8307dff700 1 -- 172.21.15.98:0/940067061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83000729b0 con 0x7f83000b6610 2023-03-03T07:55:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.577+0000 7f82f67fc700 1 -- 172.21.15.98:0/940067061 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17907 (secure 0 0 0) 0x7f82fc0518c0 con 0x7f83000b6610 2023-03-03T07:55:51.578 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:51.579 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:55:48.099844+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:55:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.580+0000 7f8307dff700 1 -- 172.21.15.98:0/940067061 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f82e40682f0 msgr2=0x7f82e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.580+0000 7f8307dff700 1 --2- 172.21.15.98:0/940067061 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f82e40682f0 0x7f82e406a7a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f82f0000c00 tx=0x7f82f0008030 comp rx=0 tx=0).stop 2023-03-03T07:55:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.580+0000 7f8307dff700 1 -- 172.21.15.98:0/940067061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83000b6610 msgr2=0x7f830014f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.580+0000 7f8307dff700 1 --2- 172.21.15.98:0/940067061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83000b6610 0x7f830014f870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00c730 tx=0x7f82fc00caf0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.580+0000 7f8307dff700 1 -- 172.21.15.98:0/940067061 shutdown_connections 2023-03-03T07:55:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.580+0000 7f8307dff700 1 --2- 172.21.15.98:0/940067061 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f82e40682f0 0x7f82e406a7a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.580+0000 7f8307dff700 1 --2- 172.21.15.98:0/940067061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83000b6610 0x7f830014f870 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.580+0000 7f8307dff700 1 --2- 172.21.15.98:0/940067061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f830015cc00 0x7f830015f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.580+0000 7f8307dff700 1 --2- 172.21.15.98:0/940067061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83000cd8d0 0x7f830015f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.580+0000 7f8307dff700 1 -- 172.21.15.98:0/940067061 >> 172.21.15.98:0/940067061 conn(0x7f830006aee0 msgr2=0x7f83000b6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.580+0000 7f8307dff700 1 -- 172.21.15.98:0/940067061 shutdown_connections 2023-03-03T07:55:51.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.580+0000 7f8307dff700 1 -- 172.21.15.98:0/940067061 wait complete. 2023-03-03T07:55:51.609 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-03-03T07:55:51.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T07:55:51.764 INFO:tasks.ceph.mds.c.smithi098.stderr:2023-03-03T07:55:51.762+0000 7f67237f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T07:55:51.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.852+0000 7f88871ff700 1 Processor -- start 2023-03-03T07:55:51.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.852+0000 7f88871ff700 1 -- start start 2023-03-03T07:55:51.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.854+0000 7f88871ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f888015cc00 0x7f88800cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:51.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.854+0000 7f88871ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88800cde10 con 0x7f88800d27c0 2023-03-03T07:55:51.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.854+0000 7f88871ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88800d1a00 con 0x7f88800d2c60 2023-03-03T07:55:51.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.854+0000 7f88871ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88800ce040 con 0x7f888015cc00 2023-03-03T07:55:51.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.854+0000 7f887ffff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f88800d27c0 0x7f88800b60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60276/0 (socket says 172.21.15.98:60276) 2023-03-03T07:55:51.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.854+0000 7f887ffff700 1 -- 172.21.15.98:0/3638319100 learned_addr learned my addr 172.21.15.98:0/3638319100 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:51.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f8884f9b700 1 --2- 172.21.15.98:0/3638319100 >> v2:172.21.15.100:3300/0 conn(0x7f888015cc00 0x7f88800cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:51.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070670281 0 0) 0x7f88800cde10 con 0x7f88800d27c0 2023-03-03T07:55:51.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88700035b0 con 0x7f88800d27c0 2023-03-03T07:55:51.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290065227 0 0) 0x7f88800d1a00 con 0x7f88800d2c60 2023-03-03T07:55:51.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88800cde10 con 0x7f88800d2c60 2023-03-03T07:55:51.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3095069476 0 0) 0x7f88700035b0 con 0x7f88800d27c0 2023-03-03T07:55:51.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 >> v2:172.21.15.100:3300/0 conn(0x7f888015cc00 msgr2=0x7f88800cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:51.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 --2- 172.21.15.98:0/3638319100 >> v2:172.21.15.100:3300/0 conn(0x7f888015cc00 0x7f88800cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 >> v1:172.21.15.100:6789/0 conn(0x7f88800d2c60 legacy=0x7f88800cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:51.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88800d1ce0 con 0x7f88800d27c0 2023-03-03T07:55:51.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f886c0015f0 con 0x7f88800d27c0 2023-03-03T07:55:51.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f8884f9b700 1 --2- 172.21.15.98:0/3638319100 >> v2:172.21.15.100:3300/0 conn(0x7f888015cc00 0x7f88800cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:51.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 >> v1:172.21.15.98:6789/0 conn(0x7f88800d27c0 legacy=0x7f88800b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:51.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 --2- 172.21.15.98:0/3638319100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8870003be0 0x7f8870003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:51.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 --2- 172.21.15.98:0/3638319100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870004850 0x7f8870004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:51.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 --2- 172.21.15.98:0/3638319100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8870005200 0x7f887000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:51.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88800ce040 con 0x7f8870004850 2023-03-03T07:55:51.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887000c0e0 con 0x7f8870003be0 2023-03-03T07:55:51.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.855+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887000c270 con 0x7f8870005200 2023-03-03T07:55:51.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.856+0000 7f887ffff700 1 --2- 172.21.15.98:0/3638319100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870004850 0x7f8870004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:51.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.856+0000 7f8884f9b700 1 --2- 172.21.15.98:0/3638319100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8870003be0 0x7f8870003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:51.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.856+0000 7f888579c700 1 --2- 172.21.15.98:0/3638319100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8870005200 0x7f887000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:51.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.856+0000 7f887ffff700 1 -- 172.21.15.98:0/3638319100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8870005200 msgr2=0x7f887000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:51.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.856+0000 7f887ffff700 1 --2- 172.21.15.98:0/3638319100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8870005200 0x7f887000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.856+0000 7f887ffff700 1 -- 172.21.15.98:0/3638319100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8870003be0 msgr2=0x7f8870003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:51.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.856+0000 7f887ffff700 1 --2- 172.21.15.98:0/3638319100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8870003be0 0x7f8870003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.856+0000 7f887ffff700 1 -- 172.21.15.98:0/3638319100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88700035b0 con 0x7f8870004850 2023-03-03T07:55:51.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.856+0000 7f888579c700 1 -- 172.21.15.98:0/3638319100 reap_dead start 2023-03-03T07:55:51.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.856+0000 7f887ffff700 1 --2- 172.21.15.98:0/3638319100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870004850 0x7f8870004cc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f886c007f80 tx=0x7f8870005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9148467bfaf83469 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:51.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.857+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c00cc50 con 0x7f8870004850 2023-03-03T07:55:51.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.857+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886c00cdb0 con 0x7f8870004850 2023-03-03T07:55:51.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.857+0000 7f887f7fe700 1 -- 172.21.15.98:0/3638319100 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c005ba0 con 0x7f8870004850 2023-03-03T07:55:51.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.857+0000 7f88871ff700 1 -- 172.21.15.98:0/3638319100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870004850 msgr2=0x7f8870004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:51.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.857+0000 7f88871ff700 1 --2- 172.21.15.98:0/3638319100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870004850 0x7f8870004cc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f886c007f80 tx=0x7f8870005880 comp rx=0 tx=0).stop 2023-03-03T07:55:51.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.857+0000 7f88871ff700 1 -- 172.21.15.98:0/3638319100 shutdown_connections 2023-03-03T07:55:51.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.857+0000 7f88871ff700 1 --2- 172.21.15.98:0/3638319100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870004850 0x7f8870004cc0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.857+0000 7f88871ff700 1 -- 172.21.15.98:0/3638319100 >> 172.21.15.98:0/3638319100 conn(0x7f888006aee0 msgr2=0x7f888006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:51.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.857+0000 7f88871ff700 1 -- 172.21.15.98:0/3638319100 shutdown_connections 2023-03-03T07:55:51.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.857+0000 7f88871ff700 1 -- 172.21.15.98:0/3638319100 wait complete. 2023-03-03T07:55:51.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.860+0000 7f88871ff700 1 Processor -- start 2023-03-03T07:55:51.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.860+0000 7f88871ff700 1 -- start start 2023-03-03T07:55:51.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.861+0000 7f88871ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870004850 0x7f8880149150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:51.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.861+0000 7f88871ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88800c91a0 0x7f88800cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:51.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.861+0000 7f88871ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88800ccd80 0x7f88801f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:51.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.861+0000 7f88871ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88800b1ee0 con 0x7f8870004850 2023-03-03T07:55:51.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.861+0000 7f88871ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88800b2780 con 0x7f88800ccd80 2023-03-03T07:55:51.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.861+0000 7f88871ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88800cd220 con 0x7f88800c91a0 2023-03-03T07:55:51.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.861+0000 7f8884f9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870004850 0x7f8880149150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:51.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.861+0000 7f8884f9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870004850 0x7f8880149150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45314/0 (socket says 172.21.15.98:45314) 2023-03-03T07:55:51.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.861+0000 7f8884f9b700 1 -- 172.21.15.98:0/1105108113 learned_addr learned my addr 172.21.15.98:0/1105108113 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:51.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.861+0000 7f887ffff700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88800c91a0 0x7f88800cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:51.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.861+0000 7f888579c700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88800ccd80 0x7f88801f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:51.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.862+0000 7f8884f9b700 1 -- 172.21.15.98:0/1105108113 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88800c91a0 msgr2=0x7f88800cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:51.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.862+0000 7f8884f9b700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88800c91a0 0x7f88800cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.862+0000 7f8884f9b700 1 -- 172.21.15.98:0/1105108113 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88800ccd80 msgr2=0x7f88801f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:51.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.862+0000 7f8884f9b700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88800ccd80 0x7f88801f21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:51.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.862+0000 7f8884f9b700 1 -- 172.21.15.98:0/1105108113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88800d22a0 con 0x7f8870004850 2023-03-03T07:55:51.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.862+0000 7f887ffff700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88800c91a0 0x7f88800cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:55:51.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.862+0000 7f888579c700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88800ccd80 0x7f88801f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:55:51.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.862+0000 7f8884f9b700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870004850 0x7f8880149150 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8874011d00 tx=0x7f887400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:51.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.862+0000 7f887d7fa700 1 -- 172.21.15.98:0/1105108113 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887400fad0 con 0x7f8870004850 2023-03-03T07:55:51.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.862+0000 7f887d7fa700 1 -- 172.21.15.98:0/1105108113 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887400fc30 con 0x7f8870004850 2023-03-03T07:55:51.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.862+0000 7f887d7fa700 1 -- 172.21.15.98:0/1105108113 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874004030 con 0x7f8870004850 2023-03-03T07:55:51.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.862+0000 7f88871ff700 1 -- 172.21.15.98:0/1105108113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88800d1fc0 con 0x7f8870004850 2023-03-03T07:55:51.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.862+0000 7f88871ff700 1 -- 172.21.15.98:0/1105108113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88800cd8d0 con 0x7f8870004850 2023-03-03T07:55:51.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.863+0000 7f88871ff700 1 -- 172.21.15.98:0/1105108113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88800729b0 con 0x7f8870004850 2023-03-03T07:55:51.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.865+0000 7f887d7fa700 1 -- 172.21.15.98:0/1105108113 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8874024070 con 0x7f8870004850 2023-03-03T07:55:51.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.866+0000 7f887d7fa700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f88640682f0 0x7f886406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:51.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.866+0000 7f887d7fa700 1 -- 172.21.15.98:0/1105108113 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8874086270 con 0x7f8870004850 2023-03-03T07:55:51.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.866+0000 7f887ffff700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f88640682f0 0x7f886406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:51.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.867+0000 7f887ffff700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f88640682f0 0x7f886406a7a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f886c007f80 tx=0x7f886c006040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:51.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:51.870+0000 7f887d7fa700 1 -- 172.21.15.98:0/1105108113 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f88740529e0 con 0x7f8870004850 2023-03-03T07:55:52.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.056+0000 7f88871ff700 1 -- 172.21.15.98:0/1105108113 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f88800b28c0 con 0x7f88640682f0 2023-03-03T07:55:52.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.067+0000 7f887d7fa700 1 -- 172.21.15.98:0/1105108113 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286645 (secure 0 0 0) 0x7f886c00ca50 con 0x7f88640682f0 2023-03-03T07:55:52.072 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:52.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.072+0000 7f88871ff700 1 -- 172.21.15.98:0/1105108113 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f88640682f0 msgr2=0x7f886406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:52.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.072+0000 7f88871ff700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f88640682f0 0x7f886406a7a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f886c007f80 tx=0x7f886c006040 comp rx=0 tx=0).stop 2023-03-03T07:55:52.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.072+0000 7f88871ff700 1 -- 172.21.15.98:0/1105108113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870004850 msgr2=0x7f8880149150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.072+0000 7f88871ff700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870004850 0x7f8880149150 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8874011d00 tx=0x7f887400f520 comp rx=0 tx=0).stop 2023-03-03T07:55:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.072+0000 7f88871ff700 1 -- 172.21.15.98:0/1105108113 shutdown_connections 2023-03-03T07:55:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.072+0000 7f88871ff700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f88640682f0 0x7f886406a7a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.072+0000 7f88871ff700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870004850 0x7f8880149150 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.072+0000 7f88871ff700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88800ccd80 0x7f88801f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.072+0000 7f88871ff700 1 --2- 172.21.15.98:0/1105108113 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88800c91a0 0x7f88800cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.072+0000 7f88871ff700 1 -- 172.21.15.98:0/1105108113 >> 172.21.15.98:0/1105108113 conn(0x7f888006aee0 msgr2=0x7f88800b82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:52.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.072+0000 7f88871ff700 1 -- 172.21.15.98:0/1105108113 shutdown_connections 2023-03-03T07:55:52.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.072+0000 7f88871ff700 1 -- 172.21.15.98:0/1105108113 wait complete. 2023-03-03T07:55:52.076 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T07:55:52.101 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-03-03T07:55:50.179188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":65324,"kb_used_data":2924,"kb_used_omap":0,"kb_used_meta":62336,"kb_avail":754909396,"statfs":{"total":773094113280,"available":773027221504,"internally_reserved":0,"allocated":2994176,"data_stored":1819864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63832064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.648841"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.129845+0000","last_change":"2023-03-03T07:55:47.123170+0000","last_active":"2023-03-03T07:55:47.129845+0000","last_peered":"2023-03-03T07:55:47.129845+0000","last_clean":"2023-03-03T07:55:47.129845+0000","last_became_active":"2023-03-03T07:55:47.122733+0000","last_became_peered":"2023-03-03T07:55:47.122733+0000","last_unstale":"2023-03-03T07:55:47.129845+0000","last_undegraded":"2023-03-03T07:55:47.129845+0000","last_fullsized":"2023-03-03T07:55:47.129845+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:46.105554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.137188+0000","last_change":"2023-03-03T07:55:47.130303+0000","last_active":"2023-03-03T07:55:47.137188+0000","last_peered":"2023-03-03T07:55:47.137188+0000","last_clean":"2023-03-03T07:55:47.137188+0000","last_became_active":"2023-03-03T07:55:47.129900+0000","last_became_peered":"2023-03-03T07:55:47.129900+0000","last_unstale":"2023-03-03T07:55:47.137188+0000","last_undegraded":"2023-03-03T07:55:47.137188+0000","last_fullsized":"2023-03-03T07:55:47.137188+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141057+0000","last_change":"2023-03-03T07:55:47.134218+0000","last_active":"2023-03-03T07:55:47.141057+0000","last_peered":"2023-03-03T07:55:47.141057+0000","last_clean":"2023-03-03T07:55:47.141057+0000","last_became_active":"2023-03-03T07:55:47.133821+0000","last_became_peered":"2023-03-03T07:55:47.133821+0000","last_unstale":"2023-03-03T07:55:47.141057+0000","last_undegraded":"2023-03-03T07:55:47.141057+0000","last_fullsized":"2023-03-03T07:55:47.141057+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:15.870956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.119362+0000","last_change":"2023-03-03T07:55:47.116584+0000","last_active":"2023-03-03T07:55:47.119362+0000","last_peered":"2023-03-03T07:55:47.119362+0000","last_clean":"2023-03-03T07:55:47.119362+0000","last_became_active":"2023-03-03T07:55:47.116307+0000","last_became_peered":"2023-03-03T07:55:47.116307+0000","last_unstale":"2023-03-03T07:55:47.119362+0000","last_undegraded":"2023-03-03T07:55:47.119362+0000","last_fullsized":"2023-03-03T07:55:47.119362+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:24.117627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138224+0000","last_change":"2023-03-03T07:55:47.120993+0000","last_active":"2023-03-03T07:55:47.138224+0000","last_peered":"2023-03-03T07:55:47.138224+0000","last_clean":"2023-03-03T07:55:47.138224+0000","last_became_active":"2023-03-03T07:55:47.119211+0000","last_became_peered":"2023-03-03T07:55:47.119211+0000","last_unstale":"2023-03-03T07:55:47.138224+0000","last_undegraded":"2023-03-03T07:55:47.138224+0000","last_fullsized":"2023-03-03T07:55:47.138224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:34:01.733177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141957+0000","last_change":"2023-03-03T07:55:47.139910+0000","last_active":"2023-03-03T07:55:47.141957+0000","last_peered":"2023-03-03T07:55:47.141957+0000","last_clean":"2023-03-03T07:55:47.141957+0000","last_became_active":"2023-03-03T07:55:47.138659+0000","last_became_peered":"2023-03-03T07:55:47.138659+0000","last_unstale":"2023-03-03T07:55:47.141957+0000","last_undegraded":"2023-03-03T07:55:47.141957+0000","last_fullsized":"2023-03-03T07:55:47.141957+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:42:18.110771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138374+0000","last_change":"2023-03-03T07:55:47.136724+0000","last_active":"2023-03-03T07:55:47.138374+0000","last_peered":"2023-03-03T07:55:47.138374+0000","last_clean":"2023-03-03T07:55:47.138374+0000","last_became_active":"2023-03-03T07:55:47.136066+0000","last_became_peered":"2023-03-03T07:55:47.136066+0000","last_unstale":"2023-03-03T07:55:47.138374+0000","last_undegraded":"2023-03-03T07:55:47.138374+0000","last_fullsized":"2023-03-03T07:55:47.138374+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.139340+0000","last_change":"2023-03-03T07:55:47.137112+0000","last_active":"2023-03-03T07:55:47.139340+0000","last_peered":"2023-03-03T07:55:47.139340+0000","last_clean":"2023-03-03T07:55:47.139340+0000","last_became_active":"2023-03-03T07:55:47.136698+0000","last_became_peered":"2023-03-03T07:55:47.136698+0000","last_unstale":"2023-03-03T07:55:47.139340+0000","last_undegraded":"2023-03-03T07:55:47.139340+0000","last_fullsized":"2023-03-03T07:55:47.139340+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138003+0000","last_change":"2023-03-03T07:55:47.123107+0000","last_active":"2023-03-03T07:55:47.138003+0000","last_peered":"2023-03-03T07:55:47.138003+0000","last_clean":"2023-03-03T07:55:47.138003+0000","last_became_active":"2023-03-03T07:55:47.122842+0000","last_became_peered":"2023-03-03T07:55:47.122842+0000","last_unstale":"2023-03-03T07:55:47.138003+0000","last_undegraded":"2023-03-03T07:55:47.138003+0000","last_fullsized":"2023-03-03T07:55:47.138003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:42:18.110771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.120154+0000","last_change":"2023-03-03T07:55:47.115719+0000","last_active":"2023-03-03T07:55:47.120154+0000","last_peered":"2023-03-03T07:55:47.120154+0000","last_clean":"2023-03-03T07:55:47.120154+0000","last_became_active":"2023-03-03T07:55:47.115404+0000","last_became_peered":"2023-03-03T07:55:47.115404+0000","last_unstale":"2023-03-03T07:55:47.120154+0000","last_undegraded":"2023-03-03T07:55:47.120154+0000","last_fullsized":"2023-03-03T07:55:47.120154+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:21.542695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141099+0000","last_change":"2023-03-03T07:55:47.135352+0000","last_active":"2023-03-03T07:55:47.141099+0000","last_peered":"2023-03-03T07:55:47.141099+0000","last_clean":"2023-03-03T07:55:47.141099+0000","last_became_active":"2023-03-03T07:55:47.135101+0000","last_became_peered":"2023-03-03T07:55:47.135101+0000","last_unstale":"2023-03-03T07:55:47.141099+0000","last_undegraded":"2023-03-03T07:55:47.141099+0000","last_fullsized":"2023-03-03T07:55:47.141099+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:46:04.349618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138518+0000","last_change":"2023-03-03T07:55:47.135595+0000","last_active":"2023-03-03T07:55:47.138518+0000","last_peered":"2023-03-03T07:55:47.138518+0000","last_clean":"2023-03-03T07:55:47.138518+0000","last_became_active":"2023-03-03T07:55:47.135349+0000","last_became_peered":"2023-03-03T07:55:47.135349+0000","last_unstale":"2023-03-03T07:55:47.138518+0000","last_undegraded":"2023-03-03T07:55:47.138518+0000","last_fullsized":"2023-03-03T07:55:47.138518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:24.117627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.129274+0000","last_change":"2023-03-03T07:55:47.125072+0000","last_active":"2023-03-03T07:55:47.129274+0000","last_peered":"2023-03-03T07:55:47.129274+0000","last_clean":"2023-03-03T07:55:47.129274+0000","last_became_active":"2023-03-03T07:55:47.124767+0000","last_became_peered":"2023-03-03T07:55:47.124767+0000","last_unstale":"2023-03-03T07:55:47.129274+0000","last_undegraded":"2023-03-03T07:55:47.129274+0000","last_fullsized":"2023-03-03T07:55:47.129274+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:34.910325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.130319+0000","last_change":"2023-03-03T07:55:47.128730+0000","last_active":"2023-03-03T07:55:47.130319+0000","last_peered":"2023-03-03T07:55:47.130319+0000","last_clean":"2023-03-03T07:55:47.130319+0000","last_became_active":"2023-03-03T07:55:47.127578+0000","last_became_peered":"2023-03-03T07:55:47.127578+0000","last_unstale":"2023-03-03T07:55:47.130319+0000","last_undegraded":"2023-03-03T07:55:47.130319+0000","last_fullsized":"2023-03-03T07:55:47.130319+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140723+0000","last_change":"2023-03-03T07:55:47.137768+0000","last_active":"2023-03-03T07:55:47.140723+0000","last_peered":"2023-03-03T07:55:47.140723+0000","last_clean":"2023-03-03T07:55:47.140723+0000","last_became_active":"2023-03-03T07:55:47.136957+0000","last_became_peered":"2023-03-03T07:55:47.136957+0000","last_unstale":"2023-03-03T07:55:47.140723+0000","last_undegraded":"2023-03-03T07:55:47.140723+0000","last_fullsized":"2023-03-03T07:55:47.140723+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141887+0000","last_change":"2023-03-03T07:55:47.139717+0000","last_active":"2023-03-03T07:55:47.141887+0000","last_peered":"2023-03-03T07:55:47.141887+0000","last_clean":"2023-03-03T07:55:47.141887+0000","last_became_active":"2023-03-03T07:55:47.138784+0000","last_became_peered":"2023-03-03T07:55:47.138784+0000","last_unstale":"2023-03-03T07:55:47.141887+0000","last_undegraded":"2023-03-03T07:55:47.141887+0000","last_fullsized":"2023-03-03T07:55:47.141887+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:34:01.733177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.137978+0000","last_change":"2023-03-03T07:55:47.121662+0000","last_active":"2023-03-03T07:55:47.137978+0000","last_peered":"2023-03-03T07:55:47.137978+0000","last_clean":"2023-03-03T07:55:47.137978+0000","last_became_active":"2023-03-03T07:55:47.118420+0000","last_became_peered":"2023-03-03T07:55:47.118420+0000","last_unstale":"2023-03-03T07:55:47.137978+0000","last_undegraded":"2023-03-03T07:55:47.137978+0000","last_fullsized":"2023-03-03T07:55:47.137978+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:36:18.587660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141133+0000","last_change":"2023-03-03T07:55:47.132197+0000","last_active":"2023-03-03T07:55:47.141133+0000","last_peered":"2023-03-03T07:55:47.141133+0000","last_clean":"2023-03-03T07:55:47.141133+0000","last_became_active":"2023-03-03T07:55:47.131599+0000","last_became_peered":"2023-03-03T07:55:47.131599+0000","last_unstale":"2023-03-03T07:55:47.141133+0000","last_undegraded":"2023-03-03T07:55:47.141133+0000","last_fullsized":"2023-03-03T07:55:47.141133+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:57:41.948092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.135166+0000","last_change":"2023-03-03T07:55:47.129930+0000","last_active":"2023-03-03T07:55:47.135166+0000","last_peered":"2023-03-03T07:55:47.135166+0000","last_clean":"2023-03-03T07:55:47.135166+0000","last_became_active":"2023-03-03T07:55:47.129569+0000","last_became_peered":"2023-03-03T07:55:47.129569+0000","last_unstale":"2023-03-03T07:55:47.135166+0000","last_undegraded":"2023-03-03T07:55:47.135166+0000","last_fullsized":"2023-03-03T07:55:47.135166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:40.146598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141800+0000","last_change":"2023-03-03T07:55:47.133303+0000","last_active":"2023-03-03T07:55:47.141800+0000","last_peered":"2023-03-03T07:55:47.141800+0000","last_clean":"2023-03-03T07:55:47.141800+0000","last_became_active":"2023-03-03T07:55:47.131841+0000","last_became_peered":"2023-03-03T07:55:47.131841+0000","last_unstale":"2023-03-03T07:55:47.141800+0000","last_undegraded":"2023-03-03T07:55:47.141800+0000","last_fullsized":"2023-03-03T07:55:47.141800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:13:04.772369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.137797+0000","last_change":"2023-03-03T07:55:47.134085+0000","last_active":"2023-03-03T07:55:47.137797+0000","last_peered":"2023-03-03T07:55:47.137797+0000","last_clean":"2023-03-03T07:55:47.137797+0000","last_became_active":"2023-03-03T07:55:47.133590+0000","last_became_peered":"2023-03-03T07:55:47.133590+0000","last_unstale":"2023-03-03T07:55:47.137797+0000","last_undegraded":"2023-03-03T07:55:47.137797+0000","last_fullsized":"2023-03-03T07:55:47.137797+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140449+0000","last_change":"2023-03-03T07:55:47.128927+0000","last_active":"2023-03-03T07:55:47.140449+0000","last_peered":"2023-03-03T07:55:47.140449+0000","last_clean":"2023-03-03T07:55:47.140449+0000","last_became_active":"2023-03-03T07:55:47.128642+0000","last_became_peered":"2023-03-03T07:55:47.128642+0000","last_unstale":"2023-03-03T07:55:47.140449+0000","last_undegraded":"2023-03-03T07:55:47.140449+0000","last_fullsized":"2023-03-03T07:55:47.140449+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.118379+0000","last_change":"2023-03-03T07:55:47.114026+0000","last_active":"2023-03-03T07:55:47.118379+0000","last_peered":"2023-03-03T07:55:47.118379+0000","last_clean":"2023-03-03T07:55:47.118379+0000","last_became_active":"2023-03-03T07:55:47.113732+0000","last_became_peered":"2023-03-03T07:55:47.113732+0000","last_unstale":"2023-03-03T07:55:47.118379+0000","last_undegraded":"2023-03-03T07:55:47.118379+0000","last_fullsized":"2023-03-03T07:55:47.118379+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:34:01.733177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140594+0000","last_change":"2023-03-03T07:55:47.137565+0000","last_active":"2023-03-03T07:55:47.140594+0000","last_peered":"2023-03-03T07:55:47.140594+0000","last_clean":"2023-03-03T07:55:47.140594+0000","last_became_active":"2023-03-03T07:55:47.136713+0000","last_became_peered":"2023-03-03T07:55:47.136713+0000","last_unstale":"2023-03-03T07:55:47.140594+0000","last_undegraded":"2023-03-03T07:55:47.140594+0000","last_fullsized":"2023-03-03T07:55:47.140594+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.133428+0000","last_change":"2023-03-03T07:55:47.124622+0000","last_active":"2023-03-03T07:55:47.133428+0000","last_peered":"2023-03-03T07:55:47.133428+0000","last_clean":"2023-03-03T07:55:47.133428+0000","last_became_active":"2023-03-03T07:55:47.124221+0000","last_became_peered":"2023-03-03T07:55:47.124221+0000","last_unstale":"2023-03-03T07:55:47.133428+0000","last_undegraded":"2023-03-03T07:55:47.133428+0000","last_fullsized":"2023-03-03T07:55:47.133428+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.133407+0000","last_change":"2023-03-03T07:55:47.123628+0000","last_active":"2023-03-03T07:55:47.133407+0000","last_peered":"2023-03-03T07:55:47.133407+0000","last_clean":"2023-03-03T07:55:47.133407+0000","last_became_active":"2023-03-03T07:55:47.123275+0000","last_became_peered":"2023-03-03T07:55:47.123275+0000","last_unstale":"2023-03-03T07:55:47.133407+0000","last_undegraded":"2023-03-03T07:55:47.133407+0000","last_fullsized":"2023-03-03T07:55:47.133407+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:21.542695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.129425+0000","last_change":"2023-03-03T07:55:47.127078+0000","last_active":"2023-03-03T07:55:47.129425+0000","last_peered":"2023-03-03T07:55:47.129425+0000","last_clean":"2023-03-03T07:55:47.129425+0000","last_became_active":"2023-03-03T07:55:47.126560+0000","last_became_peered":"2023-03-03T07:55:47.126560+0000","last_unstale":"2023-03-03T07:55:47.129425+0000","last_undegraded":"2023-03-03T07:55:47.129425+0000","last_fullsized":"2023-03-03T07:55:47.129425+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"23'1","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.132454+0000","last_change":"2023-03-03T07:55:46.566968+0000","last_active":"2023-03-03T07:55:47.132454+0000","last_peered":"2023-03-03T07:55:47.132454+0000","last_clean":"2023-03-03T07:55:47.132454+0000","last_became_active":"2023-03-03T07:55:46.566628+0000","last_became_peered":"2023-03-03T07:55:46.566628+0000","last_unstale":"2023-03-03T07:55:47.132454+0000","last_undegraded":"2023-03-03T07:55:47.132454+0000","last_fullsized":"2023-03-03T07:55:47.132454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:11.895405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.133105+0000","last_change":"2023-03-03T07:55:47.124330+0000","last_active":"2023-03-03T07:55:47.133105+0000","last_peered":"2023-03-03T07:55:47.133105+0000","last_clean":"2023-03-03T07:55:47.133105+0000","last_became_active":"2023-03-03T07:55:47.124015+0000","last_became_peered":"2023-03-03T07:55:47.124015+0000","last_unstale":"2023-03-03T07:55:47.133105+0000","last_undegraded":"2023-03-03T07:55:47.133105+0000","last_fullsized":"2023-03-03T07:55:47.133105+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:57:41.948092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.111029+0000","last_change":"2023-03-03T07:55:46.562055+0000","last_active":"2023-03-03T07:55:47.111029+0000","last_peered":"2023-03-03T07:55:47.111029+0000","last_clean":"2023-03-03T07:55:47.111029+0000","last_became_active":"2023-03-03T07:55:46.561625+0000","last_became_peered":"2023-03-03T07:55:46.561625+0000","last_unstale":"2023-03-03T07:55:47.111029+0000","last_undegraded":"2023-03-03T07:55:47.111029+0000","last_fullsized":"2023-03-03T07:55:47.111029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140601+0000","last_change":"2023-03-03T07:55:47.129628+0000","last_active":"2023-03-03T07:55:47.140601+0000","last_peered":"2023-03-03T07:55:47.140601+0000","last_clean":"2023-03-03T07:55:47.140601+0000","last_became_active":"2023-03-03T07:55:47.129356+0000","last_became_peered":"2023-03-03T07:55:47.129356+0000","last_unstale":"2023-03-03T07:55:47.140601+0000","last_undegraded":"2023-03-03T07:55:47.140601+0000","last_fullsized":"2023-03-03T07:55:47.140601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.102852+0000","last_change":"2023-03-03T07:55:46.559242+0000","last_active":"2023-03-03T07:55:47.102852+0000","last_peered":"2023-03-03T07:55:47.102852+0000","last_clean":"2023-03-03T07:55:47.102852+0000","last_became_active":"2023-03-03T07:55:46.558772+0000","last_became_peered":"2023-03-03T07:55:46.558772+0000","last_unstale":"2023-03-03T07:55:47.102852+0000","last_undegraded":"2023-03-03T07:55:47.102852+0000","last_fullsized":"2023-03-03T07:55:47.102852+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.119784+0000","last_change":"2023-03-03T07:55:47.114645+0000","last_active":"2023-03-03T07:55:47.119784+0000","last_peered":"2023-03-03T07:55:47.119784+0000","last_clean":"2023-03-03T07:55:47.119784+0000","last_became_active":"2023-03-03T07:55:47.114306+0000","last_became_peered":"2023-03-03T07:55:47.114306+0000","last_unstale":"2023-03-03T07:55:47.119784+0000","last_undegraded":"2023-03-03T07:55:47.119784+0000","last_fullsized":"2023-03-03T07:55:47.119784+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:29.066787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.109320+0000","last_change":"2023-03-03T07:55:46.567843+0000","last_active":"2023-03-03T07:55:47.109320+0000","last_peered":"2023-03-03T07:55:47.109320+0000","last_clean":"2023-03-03T07:55:47.109320+0000","last_became_active":"2023-03-03T07:55:46.567536+0000","last_became_peered":"2023-03-03T07:55:46.567536+0000","last_unstale":"2023-03-03T07:55:47.109320+0000","last_undegraded":"2023-03-03T07:55:47.109320+0000","last_fullsized":"2023-03-03T07:55:47.109320+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.132539+0000","last_change":"2023-03-03T07:55:47.126080+0000","last_active":"2023-03-03T07:55:47.132539+0000","last_peered":"2023-03-03T07:55:47.132539+0000","last_clean":"2023-03-03T07:55:47.132539+0000","last_became_active":"2023-03-03T07:55:47.125819+0000","last_became_peered":"2023-03-03T07:55:47.125819+0000","last_unstale":"2023-03-03T07:55:47.132539+0000","last_undegraded":"2023-03-03T07:55:47.132539+0000","last_fullsized":"2023-03-03T07:55:47.132539+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:42:18.110771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.107066+0000","last_change":"2023-03-03T07:55:46.564685+0000","last_active":"2023-03-03T07:55:47.107066+0000","last_peered":"2023-03-03T07:55:47.107066+0000","last_clean":"2023-03-03T07:55:47.107066+0000","last_became_active":"2023-03-03T07:55:46.564401+0000","last_became_peered":"2023-03-03T07:55:46.564401+0000","last_unstale":"2023-03-03T07:55:47.107066+0000","last_undegraded":"2023-03-03T07:55:47.107066+0000","last_fullsized":"2023-03-03T07:55:47.107066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.134139+0000","last_change":"2023-03-03T07:55:47.126626+0000","last_active":"2023-03-03T07:55:47.134139+0000","last_peered":"2023-03-03T07:55:47.134139+0000","last_clean":"2023-03-03T07:55:47.134139+0000","last_became_active":"2023-03-03T07:55:47.126341+0000","last_became_peered":"2023-03-03T07:55:47.126341+0000","last_unstale":"2023-03-03T07:55:47.134139+0000","last_undegraded":"2023-03-03T07:55:47.134139+0000","last_fullsized":"2023-03-03T07:55:47.134139+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"23'3","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.136059+0000","last_change":"2023-03-03T07:55:46.564428+0000","last_active":"2023-03-03T07:55:47.136059+0000","last_peered":"2023-03-03T07:55:47.136059+0000","last_clean":"2023-03-03T07:55:47.136059+0000","last_became_active":"2023-03-03T07:55:46.564082+0000","last_became_peered":"2023-03-03T07:55:46.564082+0000","last_unstale":"2023-03-03T07:55:47.136059+0000","last_undegraded":"2023-03-03T07:55:47.136059+0000","last_fullsized":"2023-03-03T07:55:47.136059+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:28:57.536524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138756+0000","last_change":"2023-03-03T07:55:47.135104+0000","last_active":"2023-03-03T07:55:47.138756+0000","last_peered":"2023-03-03T07:55:47.138756+0000","last_clean":"2023-03-03T07:55:47.138756+0000","last_became_active":"2023-03-03T07:55:47.134610+0000","last_became_peered":"2023-03-03T07:55:47.134610+0000","last_unstale":"2023-03-03T07:55:47.138756+0000","last_undegraded":"2023-03-03T07:55:47.138756+0000","last_fullsized":"2023-03-03T07:55:47.138756+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"23'6","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.143889+0000","last_change":"2023-03-03T07:55:46.563020+0000","last_active":"2023-03-03T07:55:47.143889+0000","last_peered":"2023-03-03T07:55:47.143889+0000","last_clean":"2023-03-03T07:55:47.143889+0000","last_became_active":"2023-03-03T07:55:46.562674+0000","last_became_peered":"2023-03-03T07:55:46.562674+0000","last_unstale":"2023-03-03T07:55:47.143889+0000","last_undegraded":"2023-03-03T07:55:47.143889+0000","last_fullsized":"2023-03-03T07:55:47.143889+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:30:20.455024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140330+0000","last_change":"2023-03-03T07:55:47.131440+0000","last_active":"2023-03-03T07:55:47.140330+0000","last_peered":"2023-03-03T07:55:47.140330+0000","last_clean":"2023-03-03T07:55:47.140330+0000","last_became_active":"2023-03-03T07:55:47.131159+0000","last_became_peered":"2023-03-03T07:55:47.131159+0000","last_unstale":"2023-03-03T07:55:47.140330+0000","last_undegraded":"2023-03-03T07:55:47.140330+0000","last_fullsized":"2023-03-03T07:55:47.140330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:29.837317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"23'2","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138652+0000","last_change":"2023-03-03T07:55:46.560188+0000","last_active":"2023-03-03T07:55:47.138652+0000","last_peered":"2023-03-03T07:55:47.138652+0000","last_clean":"2023-03-03T07:55:47.138652+0000","last_became_active":"2023-03-03T07:55:46.559853+0000","last_became_peered":"2023-03-03T07:55:46.559853+0000","last_unstale":"2023-03-03T07:55:47.138652+0000","last_undegraded":"2023-03-03T07:55:47.138652+0000","last_fullsized":"2023-03-03T07:55:47.138652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.137060+0000","last_change":"2023-03-03T07:55:47.131204+0000","last_active":"2023-03-03T07:55:47.137060+0000","last_peered":"2023-03-03T07:55:47.137060+0000","last_clean":"2023-03-03T07:55:47.137060+0000","last_became_active":"2023-03-03T07:55:47.130951+0000","last_became_peered":"2023-03-03T07:55:47.130951+0000","last_unstale":"2023-03-03T07:55:47.137060+0000","last_undegraded":"2023-03-03T07:55:47.137060+0000","last_fullsized":"2023-03-03T07:55:47.137060+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.108833+0000","last_change":"2023-03-03T07:55:46.567332+0000","last_active":"2023-03-03T07:55:47.108833+0000","last_peered":"2023-03-03T07:55:47.108833+0000","last_clean":"2023-03-03T07:55:47.108833+0000","last_became_active":"2023-03-03T07:55:46.567006+0000","last_became_peered":"2023-03-03T07:55:46.567006+0000","last_unstale":"2023-03-03T07:55:47.108833+0000","last_undegraded":"2023-03-03T07:55:47.108833+0000","last_fullsized":"2023-03-03T07:55:47.108833+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.129405+0000","last_change":"2023-03-03T07:55:47.126887+0000","last_active":"2023-03-03T07:55:47.129405+0000","last_peered":"2023-03-03T07:55:47.129405+0000","last_clean":"2023-03-03T07:55:47.129405+0000","last_became_active":"2023-03-03T07:55:47.125992+0000","last_became_peered":"2023-03-03T07:55:47.125992+0000","last_unstale":"2023-03-03T07:55:47.129405+0000","last_undegraded":"2023-03-03T07:55:47.129405+0000","last_fullsized":"2023-03-03T07:55:47.129405+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.119319+0000","last_change":"2023-03-03T07:55:46.559018+0000","last_active":"2023-03-03T07:55:47.119319+0000","last_peered":"2023-03-03T07:55:47.119319+0000","last_clean":"2023-03-03T07:55:47.119319+0000","last_became_active":"2023-03-03T07:55:46.558693+0000","last_became_peered":"2023-03-03T07:55:46.558693+0000","last_unstale":"2023-03-03T07:55:47.119319+0000","last_undegraded":"2023-03-03T07:55:47.119319+0000","last_fullsized":"2023-03-03T07:55:47.119319+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:28.058494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.137944+0000","last_change":"2023-03-03T07:55:47.123304+0000","last_active":"2023-03-03T07:55:47.137944+0000","last_peered":"2023-03-03T07:55:47.137944+0000","last_clean":"2023-03-03T07:55:47.137944+0000","last_became_active":"2023-03-03T07:55:47.118375+0000","last_became_peered":"2023-03-03T07:55:47.118375+0000","last_unstale":"2023-03-03T07:55:47.137944+0000","last_undegraded":"2023-03-03T07:55:47.137944+0000","last_fullsized":"2023-03-03T07:55:47.137944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:49:25.482566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"23'8","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.142364+0000","last_change":"2023-03-03T07:55:46.565523+0000","last_active":"2023-03-03T07:55:47.142364+0000","last_peered":"2023-03-03T07:55:47.142364+0000","last_clean":"2023-03-03T07:55:47.142364+0000","last_became_active":"2023-03-03T07:55:46.565158+0000","last_became_peered":"2023-03-03T07:55:46.565158+0000","last_unstale":"2023-03-03T07:55:47.142364+0000","last_undegraded":"2023-03-03T07:55:47.142364+0000","last_fullsized":"2023-03-03T07:55:47.142364+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:57.246539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.134491+0000","last_change":"2023-03-03T07:55:47.128607+0000","last_active":"2023-03-03T07:55:47.134491+0000","last_peered":"2023-03-03T07:55:47.134491+0000","last_clean":"2023-03-03T07:55:47.134491+0000","last_became_active":"2023-03-03T07:55:47.126465+0000","last_became_peered":"2023-03-03T07:55:47.126465+0000","last_unstale":"2023-03-03T07:55:47.134491+0000","last_undegraded":"2023-03-03T07:55:47.134491+0000","last_fullsized":"2023-03-03T07:55:47.134491+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:21.542695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"23'1","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141134+0000","last_change":"2023-03-03T07:55:46.560647+0000","last_active":"2023-03-03T07:55:47.141134+0000","last_peered":"2023-03-03T07:55:47.141134+0000","last_clean":"2023-03-03T07:55:47.141134+0000","last_became_active":"2023-03-03T07:55:46.559663+0000","last_became_peered":"2023-03-03T07:55:46.559663+0000","last_unstale":"2023-03-03T07:55:47.141134+0000","last_undegraded":"2023-03-03T07:55:47.141134+0000","last_fullsized":"2023-03-03T07:55:47.141134+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:54:12.785264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.129994+0000","last_change":"2023-03-03T07:55:47.126608+0000","last_active":"2023-03-03T07:55:47.129994+0000","last_peered":"2023-03-03T07:55:47.129994+0000","last_clean":"2023-03-03T07:55:47.129994+0000","last_became_active":"2023-03-03T07:55:47.126236+0000","last_became_peered":"2023-03-03T07:55:47.126236+0000","last_unstale":"2023-03-03T07:55:47.129994+0000","last_undegraded":"2023-03-03T07:55:47.129994+0000","last_fullsized":"2023-03-03T07:55:47.129994+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:03:11.053940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"23'2","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.142117+0000","last_change":"2023-03-03T07:55:46.558232+0000","last_active":"2023-03-03T07:55:47.142117+0000","last_peered":"2023-03-03T07:55:47.142117+0000","last_clean":"2023-03-03T07:55:47.142117+0000","last_became_active":"2023-03-03T07:55:46.557732+0000","last_became_peered":"2023-03-03T07:55:46.557732+0000","last_unstale":"2023-03-03T07:55:47.142117+0000","last_undegraded":"2023-03-03T07:55:47.142117+0000","last_fullsized":"2023-03-03T07:55:47.142117+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141631+0000","last_change":"2023-03-03T07:55:47.139286+0000","last_active":"2023-03-03T07:55:47.141631+0000","last_peered":"2023-03-03T07:55:47.141631+0000","last_clean":"2023-03-03T07:55:47.141631+0000","last_became_active":"2023-03-03T07:55:47.138539+0000","last_became_peered":"2023-03-03T07:55:47.138539+0000","last_unstale":"2023-03-03T07:55:47.141631+0000","last_undegraded":"2023-03-03T07:55:47.141631+0000","last_fullsized":"2023-03-03T07:55:47.141631+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:56.705526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"23'1","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.132951+0000","last_change":"2023-03-03T07:55:46.559244+0000","last_active":"2023-03-03T07:55:47.132951+0000","last_peered":"2023-03-03T07:55:47.132951+0000","last_clean":"2023-03-03T07:55:47.132951+0000","last_became_active":"2023-03-03T07:55:46.558832+0000","last_became_peered":"2023-03-03T07:55:46.558832+0000","last_unstale":"2023-03-03T07:55:47.132951+0000","last_undegraded":"2023-03-03T07:55:47.132951+0000","last_fullsized":"2023-03-03T07:55:47.132951+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:06:23.082465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.132403+0000","last_change":"2023-03-03T07:55:47.128420+0000","last_active":"2023-03-03T07:55:47.132403+0000","last_peered":"2023-03-03T07:55:47.132403+0000","last_clean":"2023-03-03T07:55:47.132403+0000","last_became_active":"2023-03-03T07:55:47.128117+0000","last_became_peered":"2023-03-03T07:55:47.128117+0000","last_unstale":"2023-03-03T07:55:47.132403+0000","last_undegraded":"2023-03-03T07:55:47.132403+0000","last_fullsized":"2023-03-03T07:55:47.132403+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:28:58.544816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.107330+0000","last_change":"2023-03-03T07:55:46.563195+0000","last_active":"2023-03-03T07:55:47.107330+0000","last_peered":"2023-03-03T07:55:47.107330+0000","last_clean":"2023-03-03T07:55:47.107330+0000","last_became_active":"2023-03-03T07:55:46.562774+0000","last_became_peered":"2023-03-03T07:55:46.562774+0000","last_unstale":"2023-03-03T07:55:47.107330+0000","last_undegraded":"2023-03-03T07:55:47.107330+0000","last_fullsized":"2023-03-03T07:55:47.107330+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:23.109334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140538+0000","last_change":"2023-03-03T07:55:47.132833+0000","last_active":"2023-03-03T07:55:47.140538+0000","last_peered":"2023-03-03T07:55:47.140538+0000","last_clean":"2023-03-03T07:55:47.140538+0000","last_became_active":"2023-03-03T07:55:47.132530+0000","last_became_peered":"2023-03-03T07:55:47.132530+0000","last_unstale":"2023-03-03T07:55:47.140538+0000","last_undegraded":"2023-03-03T07:55:47.140538+0000","last_fullsized":"2023-03-03T07:55:47.140538+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"23'2","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141011+0000","last_change":"2023-03-03T07:55:46.561556+0000","last_active":"2023-03-03T07:55:47.141011+0000","last_peered":"2023-03-03T07:55:47.141011+0000","last_clean":"2023-03-03T07:55:47.141011+0000","last_became_active":"2023-03-03T07:55:46.561273+0000","last_became_peered":"2023-03-03T07:55:46.561273+0000","last_unstale":"2023-03-03T07:55:47.141011+0000","last_undegraded":"2023-03-03T07:55:47.141011+0000","last_fullsized":"2023-03-03T07:55:47.141011+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141813+0000","last_change":"2023-03-03T07:55:47.138654+0000","last_active":"2023-03-03T07:55:47.141813+0000","last_peered":"2023-03-03T07:55:47.141813+0000","last_clean":"2023-03-03T07:55:47.141813+0000","last_became_active":"2023-03-03T07:55:47.131453+0000","last_became_peered":"2023-03-03T07:55:47.131453+0000","last_unstale":"2023-03-03T07:55:47.141813+0000","last_undegraded":"2023-03-03T07:55:47.141813+0000","last_fullsized":"2023-03-03T07:55:47.141813+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:36:18.587660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.105058+0000","last_change":"2023-03-03T07:55:46.563763+0000","last_active":"2023-03-03T07:55:47.105058+0000","last_peered":"2023-03-03T07:55:47.105058+0000","last_clean":"2023-03-03T07:55:47.105058+0000","last_became_active":"2023-03-03T07:55:46.563433+0000","last_became_peered":"2023-03-03T07:55:46.563433+0000","last_unstale":"2023-03-03T07:55:47.105058+0000","last_undegraded":"2023-03-03T07:55:47.105058+0000","last_fullsized":"2023-03-03T07:55:47.105058+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:12.972346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141078+0000","last_change":"2023-03-03T07:55:47.134804+0000","last_active":"2023-03-03T07:55:47.141078+0000","last_peered":"2023-03-03T07:55:47.141078+0000","last_clean":"2023-03-03T07:55:47.141078+0000","last_became_active":"2023-03-03T07:55:47.134504+0000","last_became_peered":"2023-03-03T07:55:47.134504+0000","last_unstale":"2023-03-03T07:55:47.141078+0000","last_undegraded":"2023-03-03T07:55:47.141078+0000","last_fullsized":"2023-03-03T07:55:47.141078+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.112117+0000","last_change":"2023-03-03T07:55:46.561627+0000","last_active":"2023-03-03T07:55:47.112117+0000","last_peered":"2023-03-03T07:55:47.112117+0000","last_clean":"2023-03-03T07:55:47.112117+0000","last_became_active":"2023-03-03T07:55:46.561129+0000","last_became_peered":"2023-03-03T07:55:46.561129+0000","last_unstale":"2023-03-03T07:55:47.112117+0000","last_undegraded":"2023-03-03T07:55:47.112117+0000","last_fullsized":"2023-03-03T07:55:47.112117+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:12.972346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.131137+0000","last_change":"2023-03-03T07:55:47.119813+0000","last_active":"2023-03-03T07:55:47.131137+0000","last_peered":"2023-03-03T07:55:47.131137+0000","last_clean":"2023-03-03T07:55:47.131137+0000","last_became_active":"2023-03-03T07:55:47.119517+0000","last_became_peered":"2023-03-03T07:55:47.119517+0000","last_unstale":"2023-03-03T07:55:47.131137+0000","last_undegraded":"2023-03-03T07:55:47.131137+0000","last_fullsized":"2023-03-03T07:55:47.131137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:56:43.323290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.102139+0000","last_change":"2023-03-03T07:55:46.557334+0000","last_active":"2023-03-03T07:55:47.102139+0000","last_peered":"2023-03-03T07:55:47.102139+0000","last_clean":"2023-03-03T07:55:47.102139+0000","last_became_active":"2023-03-03T07:55:46.556973+0000","last_became_peered":"2023-03-03T07:55:46.556973+0000","last_unstale":"2023-03-03T07:55:47.102139+0000","last_undegraded":"2023-03-03T07:55:47.102139+0000","last_fullsized":"2023-03-03T07:55:47.102139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.133390+0000","last_change":"2023-03-03T07:55:47.123429+0000","last_active":"2023-03-03T07:55:47.133390+0000","last_peered":"2023-03-03T07:55:47.133390+0000","last_clean":"2023-03-03T07:55:47.133390+0000","last_became_active":"2023-03-03T07:55:47.123131+0000","last_became_peered":"2023-03-03T07:55:47.123131+0000","last_unstale":"2023-03-03T07:55:47.133390+0000","last_undegraded":"2023-03-03T07:55:47.133390+0000","last_fullsized":"2023-03-03T07:55:47.133390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.560834+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T07:55:46.560834+0000","last_peered":"2023-03-03T07:55:46.560834+0000","last_clean":"2023-03-03T07:55:46.560834+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T07:55:46.560834+0000","last_undegraded":"2023-03-03T07:55:46.560834+0000","last_fullsized":"2023-03-03T07:55:46.560834+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.108422+0000","last_change":"2023-03-03T07:55:46.565316+0000","last_active":"2023-03-03T07:55:47.108422+0000","last_peered":"2023-03-03T07:55:47.108422+0000","last_clean":"2023-03-03T07:55:47.108422+0000","last_became_active":"2023-03-03T07:55:46.565027+0000","last_became_peered":"2023-03-03T07:55:46.565027+0000","last_unstale":"2023-03-03T07:55:47.108422+0000","last_undegraded":"2023-03-03T07:55:47.108422+0000","last_fullsized":"2023-03-03T07:55:47.108422+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:28.058494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.134097+0000","last_change":"2023-03-03T07:55:47.124467+0000","last_active":"2023-03-03T07:55:47.134097+0000","last_peered":"2023-03-03T07:55:47.134097+0000","last_clean":"2023-03-03T07:55:47.134097+0000","last_became_active":"2023-03-03T07:55:47.124149+0000","last_became_peered":"2023-03-03T07:55:47.124149+0000","last_unstale":"2023-03-03T07:55:47.134097+0000","last_undegraded":"2023-03-03T07:55:47.134097+0000","last_fullsized":"2023-03-03T07:55:47.134097+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.545932+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T07:55:46.545932+0000","last_peered":"2023-03-03T07:55:46.545932+0000","last_clean":"2023-03-03T07:55:46.545932+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T07:55:46.545932+0000","last_undegraded":"2023-03-03T07:55:46.545932+0000","last_fullsized":"2023-03-03T07:55:46.545932+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.113292+0000","last_change":"2023-03-03T07:55:46.559574+0000","last_active":"2023-03-03T07:55:47.113292+0000","last_peered":"2023-03-03T07:55:47.113292+0000","last_clean":"2023-03-03T07:55:47.113292+0000","last_became_active":"2023-03-03T07:55:46.559192+0000","last_became_peered":"2023-03-03T07:55:46.559192+0000","last_unstale":"2023-03-03T07:55:47.113292+0000","last_undegraded":"2023-03-03T07:55:47.113292+0000","last_fullsized":"2023-03-03T07:55:47.113292+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.133460+0000","last_change":"2023-03-03T07:55:47.127703+0000","last_active":"2023-03-03T07:55:47.133460+0000","last_peered":"2023-03-03T07:55:47.133460+0000","last_clean":"2023-03-03T07:55:47.133460+0000","last_became_active":"2023-03-03T07:55:47.126941+0000","last_became_peered":"2023-03-03T07:55:47.126941+0000","last_unstale":"2023-03-03T07:55:47.133460+0000","last_undegraded":"2023-03-03T07:55:47.133460+0000","last_fullsized":"2023-03-03T07:55:47.133460+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:03:11.053940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.539825+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T07:55:46.539825+0000","last_peered":"2023-03-03T07:55:46.539825+0000","last_clean":"2023-03-03T07:55:46.539825+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T07:55:46.539825+0000","last_undegraded":"2023-03-03T07:55:46.539825+0000","last_fullsized":"2023-03-03T07:55:46.539825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.110196+0000","last_change":"2023-03-03T07:55:46.560453+0000","last_active":"2023-03-03T07:55:47.110196+0000","last_peered":"2023-03-03T07:55:47.110196+0000","last_clean":"2023-03-03T07:55:47.110196+0000","last_became_active":"2023-03-03T07:55:46.560140+0000","last_became_peered":"2023-03-03T07:55:46.560140+0000","last_unstale":"2023-03-03T07:55:47.110196+0000","last_undegraded":"2023-03-03T07:55:47.110196+0000","last_fullsized":"2023-03-03T07:55:47.110196+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.133411+0000","last_change":"2023-03-03T07:55:47.124151+0000","last_active":"2023-03-03T07:55:47.133411+0000","last_peered":"2023-03-03T07:55:47.133411+0000","last_clean":"2023-03-03T07:55:47.133411+0000","last_became_active":"2023-03-03T07:55:47.123697+0000","last_became_peered":"2023-03-03T07:55:47.123697+0000","last_unstale":"2023-03-03T07:55:47.133411+0000","last_undegraded":"2023-03-03T07:55:47.133411+0000","last_fullsized":"2023-03-03T07:55:47.133411+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.546040+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T07:55:46.546040+0000","last_peered":"2023-03-03T07:55:46.546040+0000","last_clean":"2023-03-03T07:55:46.546040+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T07:55:46.546040+0000","last_undegraded":"2023-03-03T07:55:46.546040+0000","last_fullsized":"2023-03-03T07:55:46.546040+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.544672+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T07:55:46.544672+0000","last_peered":"2023-03-03T07:55:46.544672+0000","last_clean":"2023-03-03T07:55:46.544672+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T07:55:46.544672+0000","last_undegraded":"2023-03-03T07:55:46.544672+0000","last_fullsized":"2023-03-03T07:55:46.544672+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.102452+0000","last_change":"2023-03-03T07:55:46.559444+0000","last_active":"2023-03-03T07:55:47.102452+0000","last_peered":"2023-03-03T07:55:47.102452+0000","last_clean":"2023-03-03T07:55:47.102452+0000","last_became_active":"2023-03-03T07:55:46.559147+0000","last_became_peered":"2023-03-03T07:55:46.559147+0000","last_unstale":"2023-03-03T07:55:47.102452+0000","last_undegraded":"2023-03-03T07:55:47.102452+0000","last_fullsized":"2023-03-03T07:55:47.102452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:30:20.455024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.137452+0000","last_change":"2023-03-03T07:55:47.134945+0000","last_active":"2023-03-03T07:55:47.137452+0000","last_peered":"2023-03-03T07:55:47.137452+0000","last_clean":"2023-03-03T07:55:47.137452+0000","last_became_active":"2023-03-03T07:55:47.134662+0000","last_became_peered":"2023-03-03T07:55:47.134662+0000","last_unstale":"2023-03-03T07:55:47.137452+0000","last_undegraded":"2023-03-03T07:55:47.137452+0000","last_fullsized":"2023-03-03T07:55:47.137452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:54:13.793557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.544846+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T07:55:46.544846+0000","last_peered":"2023-03-03T07:55:46.544846+0000","last_clean":"2023-03-03T07:55:46.544846+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T07:55:46.544846+0000","last_undegraded":"2023-03-03T07:55:46.544846+0000","last_fullsized":"2023-03-03T07:55:46.544846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.108690+0000","last_change":"2023-03-03T07:55:46.567402+0000","last_active":"2023-03-03T07:55:47.108690+0000","last_peered":"2023-03-03T07:55:47.108690+0000","last_clean":"2023-03-03T07:55:47.108690+0000","last_became_active":"2023-03-03T07:55:46.567013+0000","last_became_peered":"2023-03-03T07:55:46.567013+0000","last_unstale":"2023-03-03T07:55:47.108690+0000","last_undegraded":"2023-03-03T07:55:47.108690+0000","last_fullsized":"2023-03-03T07:55:47.108690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140579+0000","last_change":"2023-03-03T07:55:47.125428+0000","last_active":"2023-03-03T07:55:47.140579+0000","last_peered":"2023-03-03T07:55:47.140579+0000","last_clean":"2023-03-03T07:55:47.140579+0000","last_became_active":"2023-03-03T07:55:47.125113+0000","last_became_peered":"2023-03-03T07:55:47.125113+0000","last_unstale":"2023-03-03T07:55:47.140579+0000","last_undegraded":"2023-03-03T07:55:47.140579+0000","last_fullsized":"2023-03-03T07:55:47.140579+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.545445+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T07:55:46.545445+0000","last_peered":"2023-03-03T07:55:46.545445+0000","last_clean":"2023-03-03T07:55:46.545445+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T07:55:46.545445+0000","last_undegraded":"2023-03-03T07:55:46.545445+0000","last_fullsized":"2023-03-03T07:55:46.545445+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.114036+0000","last_change":"2023-03-03T07:55:46.556310+0000","last_active":"2023-03-03T07:55:47.114036+0000","last_peered":"2023-03-03T07:55:47.114036+0000","last_clean":"2023-03-03T07:55:47.114036+0000","last_became_active":"2023-03-03T07:55:46.555567+0000","last_became_peered":"2023-03-03T07:55:46.555567+0000","last_unstale":"2023-03-03T07:55:47.114036+0000","last_undegraded":"2023-03-03T07:55:47.114036+0000","last_fullsized":"2023-03-03T07:55:47.114036+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138083+0000","last_change":"2023-03-03T07:55:47.119872+0000","last_active":"2023-03-03T07:55:47.138083+0000","last_peered":"2023-03-03T07:55:47.138083+0000","last_clean":"2023-03-03T07:55:47.138083+0000","last_became_active":"2023-03-03T07:55:47.119546+0000","last_became_peered":"2023-03-03T07:55:47.119546+0000","last_unstale":"2023-03-03T07:55:47.138083+0000","last_undegraded":"2023-03-03T07:55:47.138083+0000","last_fullsized":"2023-03-03T07:55:47.138083+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"23'2","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.139075+0000","last_change":"2023-03-03T07:55:46.559460+0000","last_active":"2023-03-03T07:55:47.139075+0000","last_peered":"2023-03-03T07:55:47.139075+0000","last_clean":"2023-03-03T07:55:47.139075+0000","last_became_active":"2023-03-03T07:55:46.559134+0000","last_became_peered":"2023-03-03T07:55:46.559134+0000","last_unstale":"2023-03-03T07:55:47.139075+0000","last_undegraded":"2023-03-03T07:55:47.139075+0000","last_fullsized":"2023-03-03T07:55:47.139075+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:11.895405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.561690+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T07:55:46.561690+0000","last_peered":"2023-03-03T07:55:46.561690+0000","last_clean":"2023-03-03T07:55:46.561690+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T07:55:46.561690+0000","last_undegraded":"2023-03-03T07:55:46.561690+0000","last_fullsized":"2023-03-03T07:55:46.561690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.134183+0000","last_change":"2023-03-03T07:55:47.127917+0000","last_active":"2023-03-03T07:55:47.134183+0000","last_peered":"2023-03-03T07:55:47.134183+0000","last_clean":"2023-03-03T07:55:47.134183+0000","last_became_active":"2023-03-03T07:55:47.127606+0000","last_became_peered":"2023-03-03T07:55:47.127606+0000","last_unstale":"2023-03-03T07:55:47.134183+0000","last_undegraded":"2023-03-03T07:55:47.134183+0000","last_fullsized":"2023-03-03T07:55:47.134183+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.555364+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T07:55:46.555364+0000","last_peered":"2023-03-03T07:55:46.555364+0000","last_clean":"2023-03-03T07:55:46.555364+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T07:55:46.555364+0000","last_undegraded":"2023-03-03T07:55:46.555364+0000","last_fullsized":"2023-03-03T07:55:46.555364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.109126+0000","last_change":"2023-03-03T07:55:46.565365+0000","last_active":"2023-03-03T07:55:47.109126+0000","last_peered":"2023-03-03T07:55:47.109126+0000","last_clean":"2023-03-03T07:55:47.109126+0000","last_became_active":"2023-03-03T07:55:46.565097+0000","last_became_peered":"2023-03-03T07:55:46.565097+0000","last_unstale":"2023-03-03T07:55:47.109126+0000","last_undegraded":"2023-03-03T07:55:47.109126+0000","last_fullsized":"2023-03-03T07:55:47.109126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:23.109334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141939+0000","last_change":"2023-03-03T07:55:47.139732+0000","last_active":"2023-03-03T07:55:47.141939+0000","last_peered":"2023-03-03T07:55:47.141939+0000","last_clean":"2023-03-03T07:55:47.141939+0000","last_became_active":"2023-03-03T07:55:47.138505+0000","last_became_peered":"2023-03-03T07:55:47.138505+0000","last_unstale":"2023-03-03T07:55:47.141939+0000","last_undegraded":"2023-03-03T07:55:47.141939+0000","last_fullsized":"2023-03-03T07:55:47.141939+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:49:25.482566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"23'1","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138125+0000","last_change":"2023-03-03T07:55:46.568574+0000","last_active":"2023-03-03T07:55:47.138125+0000","last_peered":"2023-03-03T07:55:47.138125+0000","last_clean":"2023-03-03T07:55:47.138125+0000","last_became_active":"2023-03-03T07:55:46.568139+0000","last_became_peered":"2023-03-03T07:55:46.568139+0000","last_unstale":"2023-03-03T07:55:47.138125+0000","last_undegraded":"2023-03-03T07:55:47.138125+0000","last_fullsized":"2023-03-03T07:55:47.138125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.134013+0000","last_change":"2023-03-03T07:55:47.128846+0000","last_active":"2023-03-03T07:55:47.134013+0000","last_peered":"2023-03-03T07:55:47.134013+0000","last_clean":"2023-03-03T07:55:47.134013+0000","last_became_active":"2023-03-03T07:55:47.128385+0000","last_became_peered":"2023-03-03T07:55:47.128385+0000","last_unstale":"2023-03-03T07:55:47.134013+0000","last_undegraded":"2023-03-03T07:55:47.134013+0000","last_fullsized":"2023-03-03T07:55:47.134013+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:00.511475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.135756+0000","last_change":"2023-03-03T07:55:46.560316+0000","last_active":"2023-03-03T07:55:47.135756+0000","last_peered":"2023-03-03T07:55:47.135756+0000","last_clean":"2023-03-03T07:55:47.135756+0000","last_became_active":"2023-03-03T07:55:46.560031+0000","last_became_peered":"2023-03-03T07:55:46.560031+0000","last_unstale":"2023-03-03T07:55:47.135756+0000","last_undegraded":"2023-03-03T07:55:47.135756+0000","last_fullsized":"2023-03-03T07:55:47.135756+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140798+0000","last_change":"2023-03-03T07:55:47.137604+0000","last_active":"2023-03-03T07:55:47.140798+0000","last_peered":"2023-03-03T07:55:47.140798+0000","last_clean":"2023-03-03T07:55:47.140798+0000","last_became_active":"2023-03-03T07:55:47.131552+0000","last_became_peered":"2023-03-03T07:55:47.131552+0000","last_unstale":"2023-03-03T07:55:47.140798+0000","last_undegraded":"2023-03-03T07:55:47.140798+0000","last_fullsized":"2023-03-03T07:55:47.140798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"23'2","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.135154+0000","last_change":"2023-03-03T07:55:46.560878+0000","last_active":"2023-03-03T07:55:47.135154+0000","last_peered":"2023-03-03T07:55:47.135154+0000","last_clean":"2023-03-03T07:55:47.135154+0000","last_became_active":"2023-03-03T07:55:46.560534+0000","last_became_peered":"2023-03-03T07:55:46.560534+0000","last_unstale":"2023-03-03T07:55:47.135154+0000","last_undegraded":"2023-03-03T07:55:47.135154+0000","last_fullsized":"2023-03-03T07:55:47.135154+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.132540+0000","last_change":"2023-03-03T07:55:47.122002+0000","last_active":"2023-03-03T07:55:47.132540+0000","last_peered":"2023-03-03T07:55:47.132540+0000","last_clean":"2023-03-03T07:55:47.132540+0000","last_became_active":"2023-03-03T07:55:47.121732+0000","last_became_peered":"2023-03-03T07:55:47.121732+0000","last_unstale":"2023-03-03T07:55:47.132540+0000","last_undegraded":"2023-03-03T07:55:47.132540+0000","last_fullsized":"2023-03-03T07:55:47.132540+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"23'1","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141645+0000","last_change":"2023-03-03T07:55:46.560027+0000","last_active":"2023-03-03T07:55:47.141645+0000","last_peered":"2023-03-03T07:55:47.141645+0000","last_clean":"2023-03-03T07:55:47.141645+0000","last_became_active":"2023-03-03T07:55:46.559508+0000","last_became_peered":"2023-03-03T07:55:46.559508+0000","last_unstale":"2023-03-03T07:55:47.141645+0000","last_undegraded":"2023-03-03T07:55:47.141645+0000","last_fullsized":"2023-03-03T07:55:47.141645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:23.109334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.132455+0000","last_change":"2023-03-03T07:55:47.121846+0000","last_active":"2023-03-03T07:55:47.132455+0000","last_peered":"2023-03-03T07:55:47.132455+0000","last_clean":"2023-03-03T07:55:47.132455+0000","last_became_active":"2023-03-03T07:55:47.121563+0000","last_became_peered":"2023-03-03T07:55:47.121563+0000","last_unstale":"2023-03-03T07:55:47.132455+0000","last_undegraded":"2023-03-03T07:55:47.132455+0000","last_fullsized":"2023-03-03T07:55:47.132455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:36:18.587660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"23'4","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140768+0000","last_change":"2023-03-03T07:55:46.567878+0000","last_active":"2023-03-03T07:55:47.140768+0000","last_peered":"2023-03-03T07:55:47.140768+0000","last_clean":"2023-03-03T07:55:47.140768+0000","last_became_active":"2023-03-03T07:55:46.567593+0000","last_became_peered":"2023-03-03T07:55:46.567593+0000","last_unstale":"2023-03-03T07:55:47.140768+0000","last_undegraded":"2023-03-03T07:55:47.140768+0000","last_fullsized":"2023-03-03T07:55:47.140768+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:28.058494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140213+0000","last_change":"2023-03-03T07:55:47.131373+0000","last_active":"2023-03-03T07:55:47.140213+0000","last_peered":"2023-03-03T07:55:47.140213+0000","last_clean":"2023-03-03T07:55:47.140213+0000","last_became_active":"2023-03-03T07:55:47.131037+0000","last_became_peered":"2023-03-03T07:55:47.131037+0000","last_unstale":"2023-03-03T07:55:47.140213+0000","last_undegraded":"2023-03-03T07:55:47.140213+0000","last_fullsized":"2023-03-03T07:55:47.140213+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:52:08.323505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.128848+0000","last_change":"2023-03-03T07:55:47.122842+0000","last_active":"2023-03-03T07:55:47.128848+0000","last_peered":"2023-03-03T07:55:47.128848+0000","last_clean":"2023-03-03T07:55:47.128848+0000","last_became_active":"2023-03-03T07:55:47.122580+0000","last_became_peered":"2023-03-03T07:55:47.122580+0000","last_unstale":"2023-03-03T07:55:47.128848+0000","last_undegraded":"2023-03-03T07:55:47.128848+0000","last_fullsized":"2023-03-03T07:55:47.128848+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:06:24.090758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.130081+0000","last_change":"2023-03-03T07:55:47.121284+0000","last_active":"2023-03-03T07:55:47.130081+0000","last_peered":"2023-03-03T07:55:47.130081+0000","last_clean":"2023-03-03T07:55:47.130081+0000","last_became_active":"2023-03-03T07:55:47.121012+0000","last_became_peered":"2023-03-03T07:55:47.121012+0000","last_unstale":"2023-03-03T07:55:47.130081+0000","last_undegraded":"2023-03-03T07:55:47.130081+0000","last_fullsized":"2023-03-03T07:55:47.130081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:15.870956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141235+0000","last_change":"2023-03-03T07:55:47.138325+0000","last_active":"2023-03-03T07:55:47.141235+0000","last_peered":"2023-03-03T07:55:47.141235+0000","last_clean":"2023-03-03T07:55:47.141235+0000","last_became_active":"2023-03-03T07:55:47.138019+0000","last_became_peered":"2023-03-03T07:55:47.138019+0000","last_unstale":"2023-03-03T07:55:47.141235+0000","last_undegraded":"2023-03-03T07:55:47.141235+0000","last_fullsized":"2023-03-03T07:55:47.141235+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:57.306479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.134158+0000","last_change":"2023-03-03T07:55:47.128792+0000","last_active":"2023-03-03T07:55:47.134158+0000","last_peered":"2023-03-03T07:55:47.134158+0000","last_clean":"2023-03-03T07:55:47.134158+0000","last_became_active":"2023-03-03T07:55:47.127248+0000","last_became_peered":"2023-03-03T07:55:47.127248+0000","last_unstale":"2023-03-03T07:55:47.134158+0000","last_undegraded":"2023-03-03T07:55:47.134158+0000","last_fullsized":"2023-03-03T07:55:47.134158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:49:25.482566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.133442+0000","last_change":"2023-03-03T07:55:47.127209+0000","last_active":"2023-03-03T07:55:47.133442+0000","last_peered":"2023-03-03T07:55:47.133442+0000","last_clean":"2023-03-03T07:55:47.133442+0000","last_became_active":"2023-03-03T07:55:47.126100+0000","last_became_peered":"2023-03-03T07:55:47.126100+0000","last_unstale":"2023-03-03T07:55:47.133442+0000","last_undegraded":"2023-03-03T07:55:47.133442+0000","last_fullsized":"2023-03-03T07:55:47.133442+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:46.306038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9004,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94362836,"statfs":{"total":96636764160,"available":96627544064,"internally_reserved":0,"allocated":823296,"data_stored":671596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7908,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363932,"statfs":{"total":96636764160,"available":96628666368,"internally_reserved":0,"allocated":225280,"data_stored":80209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7912,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363928,"statfs":{"total":96636764160,"available":96628662272,"internally_reserved":0,"allocated":229376,"data_stored":81228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8924,"kb_used_data":788,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94362916,"statfs":{"total":96636764160,"available":96627625984,"internally_reserved":0,"allocated":806912,"data_stored":663957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7908,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363932,"statfs":{"total":96636764160,"available":96628666368,"internally_reserved":0,"allocated":225280,"data_stored":80209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.407}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3620000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7848,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":94363992,"statfs":{"total":96636764160,"available":96628727808,"internally_reserved":0,"allocated":229376,"data_stored":81228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7912,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363928,"statfs":{"total":96636764160,"available":96628662272,"internally_reserved":0,"allocated":229376,"data_stored":81228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7908,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363932,"statfs":{"total":96636764160,"available":96628666368,"internally_reserved":0,"allocated":225280,"data_stored":80209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T07:55:52.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T07:55:52.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.384+0000 7f746179d700 1 Processor -- start 2023-03-03T07:55:52.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.385+0000 7f746179d700 1 -- start start 2023-03-03T07:55:52.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.386+0000 7f746179d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c146360 0x7f745c160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:52.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.386+0000 7f746179d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f745c0d2590 0x7f745c0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:52.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.386+0000 7f746179d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f745c1614c0 con 0x7f745c0b4ab0 2023-03-03T07:55:52.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.386+0000 7f746179d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f745c0b5460 con 0x7f745c0d2590 2023-03-03T07:55:52.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.386+0000 7f746179d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745c0d10f0 con 0x7f745c146360 2023-03-03T07:55:52.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.386+0000 7f7460f9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f745c0d2590 0x7f745c0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:52.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.387+0000 7f7460f9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f745c0d2590 0x7f745c0b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45318/0 (socket says 172.21.15.98:45318) 2023-03-03T07:55:52.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.387+0000 7f7460f9c700 1 -- 172.21.15.98:0/3678624914 learned_addr learned my addr 172.21.15.98:0/3678624914 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:52.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.387+0000 7f745bfff700 1 --2- 172.21.15.98:0/3678624914 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c146360 0x7f745c160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:52.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.387+0000 7f7460f9c700 1 -- 172.21.15.98:0/3678624914 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c146360 msgr2=0x7f745c160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:52.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.387+0000 7f7460f9c700 1 --2- 172.21.15.98:0/3678624914 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c146360 0x7f745c160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.387+0000 7f7460f9c700 1 -- 172.21.15.98:0/3678624914 >> v1:172.21.15.100:6789/0 conn(0x7f745c0b4ab0 legacy=0x7f745c0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:52.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.387+0000 7f7460f9c700 1 -- 172.21.15.98:0/3678624914 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c0cdf20 con 0x7f745c0d2590 2023-03-03T07:55:52.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.387+0000 7f7460f9c700 1 --2- 172.21.15.98:0/3678624914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0d2590 0x7f745c0b4f20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7454009e50 tx=0x7f745400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810be5cee40b691a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:52.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.387+0000 7f745affd700 1 -- 172.21.15.98:0/3678624914 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745400e070 con 0x7f745c0d2590 2023-03-03T07:55:52.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.388+0000 7f745affd700 1 -- 172.21.15.98:0/3678624914 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7454003a20 con 0x7f745c0d2590 2023-03-03T07:55:52.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.388+0000 7f745affd700 1 -- 172.21.15.98:0/3678624914 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454003dd0 con 0x7f745c0d2590 2023-03-03T07:55:52.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.388+0000 7f746179d700 1 -- 172.21.15.98:0/3678624914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0d2590 msgr2=0x7f745c0b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:52.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.388+0000 7f746179d700 1 --2- 172.21.15.98:0/3678624914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0d2590 0x7f745c0b4f20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7454009e50 tx=0x7f745400b330 comp rx=0 tx=0).stop 2023-03-03T07:55:52.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.388+0000 7f746179d700 1 -- 172.21.15.98:0/3678624914 shutdown_connections 2023-03-03T07:55:52.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.388+0000 7f746179d700 1 --2- 172.21.15.98:0/3678624914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0d2590 0x7f745c0b4f20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.388+0000 7f746179d700 1 --2- 172.21.15.98:0/3678624914 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c146360 0x7f745c160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.388+0000 7f746179d700 1 -- 172.21.15.98:0/3678624914 >> 172.21.15.98:0/3678624914 conn(0x7f745c06b8e0 msgr2=0x7f745c0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:52.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.388+0000 7f746179d700 1 -- 172.21.15.98:0/3678624914 shutdown_connections 2023-03-03T07:55:52.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.388+0000 7f746179d700 1 -- 172.21.15.98:0/3678624914 wait complete. 2023-03-03T07:55:52.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.390+0000 7f746179d700 1 Processor -- start 2023-03-03T07:55:52.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.391+0000 7f746179d700 1 -- start start 2023-03-03T07:55:52.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f746179d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0b4ab0 0x7f745c1f1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:52.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f746179d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f745c0d2590 0x7f745c143a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:52.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f746179d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c146360 0x7f745c0ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:52.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f746179d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c143580 con 0x7f745c0b4ab0 2023-03-03T07:55:52.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f746179d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c142970 con 0x7f745c0d2590 2023-03-03T07:55:52.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f746179d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745c142c50 con 0x7f745c146360 2023-03-03T07:55:52.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f745bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0b4ab0 0x7f745c1f1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:52.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f745bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0b4ab0 0x7f745c1f1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45328/0 (socket says 172.21.15.98:45328) 2023-03-03T07:55:52.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f745bfff700 1 -- 172.21.15.98:0/618392062 learned_addr learned my addr 172.21.15.98:0/618392062 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:52.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f745b7fe700 1 --2- 172.21.15.98:0/618392062 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f745c0d2590 0x7f745c143a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:52.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f7460f9c700 1 --2- 172.21.15.98:0/618392062 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c146360 0x7f745c0ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:52.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f745bfff700 1 -- 172.21.15.98:0/618392062 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c146360 msgr2=0x7f745c0ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:52.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f745bfff700 1 --2- 172.21.15.98:0/618392062 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c146360 0x7f745c0ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f745bfff700 1 -- 172.21.15.98:0/618392062 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f745c0d2590 msgr2=0x7f745c143a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:52.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f745bfff700 1 --2- 172.21.15.98:0/618392062 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f745c0d2590 0x7f745c143a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f745bfff700 1 -- 172.21.15.98:0/618392062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7454009b30 con 0x7f745c0b4ab0 2023-03-03T07:55:52.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.392+0000 7f7460f9c700 1 --2- 172.21.15.98:0/618392062 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c146360 0x7f745c0ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:55:52.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.393+0000 7f745bfff700 1 --2- 172.21.15.98:0/618392062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0b4ab0 0x7f745c1f1e80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f744c00e810 tx=0x7f744c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:52.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.393+0000 7f7458ff9700 1 -- 172.21.15.98:0/618392062 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c013610 con 0x7f745c0b4ab0 2023-03-03T07:55:52.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.393+0000 7f7458ff9700 1 -- 172.21.15.98:0/618392062 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744c013770 con 0x7f745c0b4ab0 2023-03-03T07:55:52.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.393+0000 7f7458ff9700 1 -- 172.21.15.98:0/618392062 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c013b20 con 0x7f745c0b4ab0 2023-03-03T07:55:52.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.393+0000 7f746179d700 1 -- 172.21.15.98:0/618392062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745c0cd960 con 0x7f745c0b4ab0 2023-03-03T07:55:52.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.393+0000 7f746179d700 1 -- 172.21.15.98:0/618392062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745c0cd680 con 0x7f745c0b4ab0 2023-03-03T07:55:52.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.394+0000 7f746179d700 1 -- 172.21.15.98:0/618392062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f745c0cdc40 con 0x7f745c0b4ab0 2023-03-03T07:55:52.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.395+0000 7f7458ff9700 1 -- 172.21.15.98:0/618392062 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f744c023070 con 0x7f745c0b4ab0 2023-03-03T07:55:52.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.397+0000 7f7458ff9700 1 --2- 172.21.15.98:0/618392062 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f74480682f0 0x7f744806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:52.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.397+0000 7f7458ff9700 1 -- 172.21.15.98:0/618392062 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f744c085a90 con 0x7f745c0b4ab0 2023-03-03T07:55:52.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.397+0000 7f745b7fe700 1 --2- 172.21.15.98:0/618392062 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f74480682f0 0x7f744806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:52.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.397+0000 7f745b7fe700 1 --2- 172.21.15.98:0/618392062 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f74480682f0 0x7f744806a7a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7450009870 tx=0x7f7450008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:52.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.400+0000 7f7458ff9700 1 -- 172.21.15.98:0/618392062 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f744c052180 con 0x7f745c0b4ab0 2023-03-03T07:55:52.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.592+0000 7f746179d700 1 -- 172.21.15.98:0/618392062 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f745c1436c0 con 0x7f74480682f0 2023-03-03T07:55:52.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.601+0000 7f7458ff9700 1 -- 172.21.15.98:0/618392062 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286645 (secure 0 0 0) 0x7f74500123e0 con 0x7f74480682f0 2023-03-03T07:55:52.607 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.605+0000 7f746179d700 1 -- 172.21.15.98:0/618392062 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f74480682f0 msgr2=0x7f744806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.605+0000 7f746179d700 1 --2- 172.21.15.98:0/618392062 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f74480682f0 0x7f744806a7a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7450009870 tx=0x7f7450008040 comp rx=0 tx=0).stop 2023-03-03T07:55:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.605+0000 7f746179d700 1 -- 172.21.15.98:0/618392062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0b4ab0 msgr2=0x7f745c1f1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:52.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.605+0000 7f746179d700 1 --2- 172.21.15.98:0/618392062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0b4ab0 0x7f745c1f1e80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f744c00e810 tx=0x7f744c00ebd0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.605+0000 7f746179d700 1 -- 172.21.15.98:0/618392062 shutdown_connections 2023-03-03T07:55:52.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.605+0000 7f746179d700 1 --2- 172.21.15.98:0/618392062 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f74480682f0 0x7f744806a7a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.605+0000 7f746179d700 1 --2- 172.21.15.98:0/618392062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0b4ab0 0x7f745c1f1e80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.605+0000 7f746179d700 1 --2- 172.21.15.98:0/618392062 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c146360 0x7f745c0ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.605+0000 7f746179d700 1 --2- 172.21.15.98:0/618392062 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f745c0d2590 0x7f745c143a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.605+0000 7f746179d700 1 -- 172.21.15.98:0/618392062 >> 172.21.15.98:0/618392062 conn(0x7f745c06b8e0 msgr2=0x7f745c0b3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:52.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.605+0000 7f746179d700 1 -- 172.21.15.98:0/618392062 shutdown_connections 2023-03-03T07:55:52.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.605+0000 7f746179d700 1 -- 172.21.15.98:0/618392062 wait complete. 2023-03-03T07:55:52.615 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T07:55:52.635 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-03-03T07:55:52.179633+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":65324,"kb_used_data":2924,"kb_used_omap":0,"kb_used_meta":62336,"kb_avail":754909396,"statfs":{"total":773094113280,"available":773027221504,"internally_reserved":0,"allocated":2994176,"data_stored":1819864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63832064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001357"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.129845+0000","last_change":"2023-03-03T07:55:47.123170+0000","last_active":"2023-03-03T07:55:47.129845+0000","last_peered":"2023-03-03T07:55:47.129845+0000","last_clean":"2023-03-03T07:55:47.129845+0000","last_became_active":"2023-03-03T07:55:47.122733+0000","last_became_peered":"2023-03-03T07:55:47.122733+0000","last_unstale":"2023-03-03T07:55:47.129845+0000","last_undegraded":"2023-03-03T07:55:47.129845+0000","last_fullsized":"2023-03-03T07:55:47.129845+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:46.105554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.137188+0000","last_change":"2023-03-03T07:55:47.130303+0000","last_active":"2023-03-03T07:55:47.137188+0000","last_peered":"2023-03-03T07:55:47.137188+0000","last_clean":"2023-03-03T07:55:47.137188+0000","last_became_active":"2023-03-03T07:55:47.129900+0000","last_became_peered":"2023-03-03T07:55:47.129900+0000","last_unstale":"2023-03-03T07:55:47.137188+0000","last_undegraded":"2023-03-03T07:55:47.137188+0000","last_fullsized":"2023-03-03T07:55:47.137188+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141057+0000","last_change":"2023-03-03T07:55:47.134218+0000","last_active":"2023-03-03T07:55:47.141057+0000","last_peered":"2023-03-03T07:55:47.141057+0000","last_clean":"2023-03-03T07:55:47.141057+0000","last_became_active":"2023-03-03T07:55:47.133821+0000","last_became_peered":"2023-03-03T07:55:47.133821+0000","last_unstale":"2023-03-03T07:55:47.141057+0000","last_undegraded":"2023-03-03T07:55:47.141057+0000","last_fullsized":"2023-03-03T07:55:47.141057+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:15.870956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.119362+0000","last_change":"2023-03-03T07:55:47.116584+0000","last_active":"2023-03-03T07:55:47.119362+0000","last_peered":"2023-03-03T07:55:47.119362+0000","last_clean":"2023-03-03T07:55:47.119362+0000","last_became_active":"2023-03-03T07:55:47.116307+0000","last_became_peered":"2023-03-03T07:55:47.116307+0000","last_unstale":"2023-03-03T07:55:47.119362+0000","last_undegraded":"2023-03-03T07:55:47.119362+0000","last_fullsized":"2023-03-03T07:55:47.119362+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:24.117627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138224+0000","last_change":"2023-03-03T07:55:47.120993+0000","last_active":"2023-03-03T07:55:47.138224+0000","last_peered":"2023-03-03T07:55:47.138224+0000","last_clean":"2023-03-03T07:55:47.138224+0000","last_became_active":"2023-03-03T07:55:47.119211+0000","last_became_peered":"2023-03-03T07:55:47.119211+0000","last_unstale":"2023-03-03T07:55:47.138224+0000","last_undegraded":"2023-03-03T07:55:47.138224+0000","last_fullsized":"2023-03-03T07:55:47.138224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:34:01.733177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141957+0000","last_change":"2023-03-03T07:55:47.139910+0000","last_active":"2023-03-03T07:55:47.141957+0000","last_peered":"2023-03-03T07:55:47.141957+0000","last_clean":"2023-03-03T07:55:47.141957+0000","last_became_active":"2023-03-03T07:55:47.138659+0000","last_became_peered":"2023-03-03T07:55:47.138659+0000","last_unstale":"2023-03-03T07:55:47.141957+0000","last_undegraded":"2023-03-03T07:55:47.141957+0000","last_fullsized":"2023-03-03T07:55:47.141957+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:42:18.110771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138374+0000","last_change":"2023-03-03T07:55:47.136724+0000","last_active":"2023-03-03T07:55:47.138374+0000","last_peered":"2023-03-03T07:55:47.138374+0000","last_clean":"2023-03-03T07:55:47.138374+0000","last_became_active":"2023-03-03T07:55:47.136066+0000","last_became_peered":"2023-03-03T07:55:47.136066+0000","last_unstale":"2023-03-03T07:55:47.138374+0000","last_undegraded":"2023-03-03T07:55:47.138374+0000","last_fullsized":"2023-03-03T07:55:47.138374+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.139340+0000","last_change":"2023-03-03T07:55:47.137112+0000","last_active":"2023-03-03T07:55:47.139340+0000","last_peered":"2023-03-03T07:55:47.139340+0000","last_clean":"2023-03-03T07:55:47.139340+0000","last_became_active":"2023-03-03T07:55:47.136698+0000","last_became_peered":"2023-03-03T07:55:47.136698+0000","last_unstale":"2023-03-03T07:55:47.139340+0000","last_undegraded":"2023-03-03T07:55:47.139340+0000","last_fullsized":"2023-03-03T07:55:47.139340+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138003+0000","last_change":"2023-03-03T07:55:47.123107+0000","last_active":"2023-03-03T07:55:47.138003+0000","last_peered":"2023-03-03T07:55:47.138003+0000","last_clean":"2023-03-03T07:55:47.138003+0000","last_became_active":"2023-03-03T07:55:47.122842+0000","last_became_peered":"2023-03-03T07:55:47.122842+0000","last_unstale":"2023-03-03T07:55:47.138003+0000","last_undegraded":"2023-03-03T07:55:47.138003+0000","last_fullsized":"2023-03-03T07:55:47.138003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:42:18.110771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.120154+0000","last_change":"2023-03-03T07:55:47.115719+0000","last_active":"2023-03-03T07:55:47.120154+0000","last_peered":"2023-03-03T07:55:47.120154+0000","last_clean":"2023-03-03T07:55:47.120154+0000","last_became_active":"2023-03-03T07:55:47.115404+0000","last_became_peered":"2023-03-03T07:55:47.115404+0000","last_unstale":"2023-03-03T07:55:47.120154+0000","last_undegraded":"2023-03-03T07:55:47.120154+0000","last_fullsized":"2023-03-03T07:55:47.120154+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:21.542695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141099+0000","last_change":"2023-03-03T07:55:47.135352+0000","last_active":"2023-03-03T07:55:47.141099+0000","last_peered":"2023-03-03T07:55:47.141099+0000","last_clean":"2023-03-03T07:55:47.141099+0000","last_became_active":"2023-03-03T07:55:47.135101+0000","last_became_peered":"2023-03-03T07:55:47.135101+0000","last_unstale":"2023-03-03T07:55:47.141099+0000","last_undegraded":"2023-03-03T07:55:47.141099+0000","last_fullsized":"2023-03-03T07:55:47.141099+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:46:04.349618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138518+0000","last_change":"2023-03-03T07:55:47.135595+0000","last_active":"2023-03-03T07:55:47.138518+0000","last_peered":"2023-03-03T07:55:47.138518+0000","last_clean":"2023-03-03T07:55:47.138518+0000","last_became_active":"2023-03-03T07:55:47.135349+0000","last_became_peered":"2023-03-03T07:55:47.135349+0000","last_unstale":"2023-03-03T07:55:47.138518+0000","last_undegraded":"2023-03-03T07:55:47.138518+0000","last_fullsized":"2023-03-03T07:55:47.138518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:24.117627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.129274+0000","last_change":"2023-03-03T07:55:47.125072+0000","last_active":"2023-03-03T07:55:47.129274+0000","last_peered":"2023-03-03T07:55:47.129274+0000","last_clean":"2023-03-03T07:55:47.129274+0000","last_became_active":"2023-03-03T07:55:47.124767+0000","last_became_peered":"2023-03-03T07:55:47.124767+0000","last_unstale":"2023-03-03T07:55:47.129274+0000","last_undegraded":"2023-03-03T07:55:47.129274+0000","last_fullsized":"2023-03-03T07:55:47.129274+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:34.910325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.130319+0000","last_change":"2023-03-03T07:55:47.128730+0000","last_active":"2023-03-03T07:55:47.130319+0000","last_peered":"2023-03-03T07:55:47.130319+0000","last_clean":"2023-03-03T07:55:47.130319+0000","last_became_active":"2023-03-03T07:55:47.127578+0000","last_became_peered":"2023-03-03T07:55:47.127578+0000","last_unstale":"2023-03-03T07:55:47.130319+0000","last_undegraded":"2023-03-03T07:55:47.130319+0000","last_fullsized":"2023-03-03T07:55:47.130319+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140723+0000","last_change":"2023-03-03T07:55:47.137768+0000","last_active":"2023-03-03T07:55:47.140723+0000","last_peered":"2023-03-03T07:55:47.140723+0000","last_clean":"2023-03-03T07:55:47.140723+0000","last_became_active":"2023-03-03T07:55:47.136957+0000","last_became_peered":"2023-03-03T07:55:47.136957+0000","last_unstale":"2023-03-03T07:55:47.140723+0000","last_undegraded":"2023-03-03T07:55:47.140723+0000","last_fullsized":"2023-03-03T07:55:47.140723+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141887+0000","last_change":"2023-03-03T07:55:47.139717+0000","last_active":"2023-03-03T07:55:47.141887+0000","last_peered":"2023-03-03T07:55:47.141887+0000","last_clean":"2023-03-03T07:55:47.141887+0000","last_became_active":"2023-03-03T07:55:47.138784+0000","last_became_peered":"2023-03-03T07:55:47.138784+0000","last_unstale":"2023-03-03T07:55:47.141887+0000","last_undegraded":"2023-03-03T07:55:47.141887+0000","last_fullsized":"2023-03-03T07:55:47.141887+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:34:01.733177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.137978+0000","last_change":"2023-03-03T07:55:47.121662+0000","last_active":"2023-03-03T07:55:47.137978+0000","last_peered":"2023-03-03T07:55:47.137978+0000","last_clean":"2023-03-03T07:55:47.137978+0000","last_became_active":"2023-03-03T07:55:47.118420+0000","last_became_peered":"2023-03-03T07:55:47.118420+0000","last_unstale":"2023-03-03T07:55:47.137978+0000","last_undegraded":"2023-03-03T07:55:47.137978+0000","last_fullsized":"2023-03-03T07:55:47.137978+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:36:18.587660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141133+0000","last_change":"2023-03-03T07:55:47.132197+0000","last_active":"2023-03-03T07:55:47.141133+0000","last_peered":"2023-03-03T07:55:47.141133+0000","last_clean":"2023-03-03T07:55:47.141133+0000","last_became_active":"2023-03-03T07:55:47.131599+0000","last_became_peered":"2023-03-03T07:55:47.131599+0000","last_unstale":"2023-03-03T07:55:47.141133+0000","last_undegraded":"2023-03-03T07:55:47.141133+0000","last_fullsized":"2023-03-03T07:55:47.141133+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:57:41.948092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.135166+0000","last_change":"2023-03-03T07:55:47.129930+0000","last_active":"2023-03-03T07:55:47.135166+0000","last_peered":"2023-03-03T07:55:47.135166+0000","last_clean":"2023-03-03T07:55:47.135166+0000","last_became_active":"2023-03-03T07:55:47.129569+0000","last_became_peered":"2023-03-03T07:55:47.129569+0000","last_unstale":"2023-03-03T07:55:47.135166+0000","last_undegraded":"2023-03-03T07:55:47.135166+0000","last_fullsized":"2023-03-03T07:55:47.135166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:40.146598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141800+0000","last_change":"2023-03-03T07:55:47.133303+0000","last_active":"2023-03-03T07:55:47.141800+0000","last_peered":"2023-03-03T07:55:47.141800+0000","last_clean":"2023-03-03T07:55:47.141800+0000","last_became_active":"2023-03-03T07:55:47.131841+0000","last_became_peered":"2023-03-03T07:55:47.131841+0000","last_unstale":"2023-03-03T07:55:47.141800+0000","last_undegraded":"2023-03-03T07:55:47.141800+0000","last_fullsized":"2023-03-03T07:55:47.141800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:13:04.772369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.137797+0000","last_change":"2023-03-03T07:55:47.134085+0000","last_active":"2023-03-03T07:55:47.137797+0000","last_peered":"2023-03-03T07:55:47.137797+0000","last_clean":"2023-03-03T07:55:47.137797+0000","last_became_active":"2023-03-03T07:55:47.133590+0000","last_became_peered":"2023-03-03T07:55:47.133590+0000","last_unstale":"2023-03-03T07:55:47.137797+0000","last_undegraded":"2023-03-03T07:55:47.137797+0000","last_fullsized":"2023-03-03T07:55:47.137797+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140449+0000","last_change":"2023-03-03T07:55:47.128927+0000","last_active":"2023-03-03T07:55:47.140449+0000","last_peered":"2023-03-03T07:55:47.140449+0000","last_clean":"2023-03-03T07:55:47.140449+0000","last_became_active":"2023-03-03T07:55:47.128642+0000","last_became_peered":"2023-03-03T07:55:47.128642+0000","last_unstale":"2023-03-03T07:55:47.140449+0000","last_undegraded":"2023-03-03T07:55:47.140449+0000","last_fullsized":"2023-03-03T07:55:47.140449+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.118379+0000","last_change":"2023-03-03T07:55:47.114026+0000","last_active":"2023-03-03T07:55:47.118379+0000","last_peered":"2023-03-03T07:55:47.118379+0000","last_clean":"2023-03-03T07:55:47.118379+0000","last_became_active":"2023-03-03T07:55:47.113732+0000","last_became_peered":"2023-03-03T07:55:47.113732+0000","last_unstale":"2023-03-03T07:55:47.118379+0000","last_undegraded":"2023-03-03T07:55:47.118379+0000","last_fullsized":"2023-03-03T07:55:47.118379+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:34:01.733177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140594+0000","last_change":"2023-03-03T07:55:47.137565+0000","last_active":"2023-03-03T07:55:47.140594+0000","last_peered":"2023-03-03T07:55:47.140594+0000","last_clean":"2023-03-03T07:55:47.140594+0000","last_became_active":"2023-03-03T07:55:47.136713+0000","last_became_peered":"2023-03-03T07:55:47.136713+0000","last_unstale":"2023-03-03T07:55:47.140594+0000","last_undegraded":"2023-03-03T07:55:47.140594+0000","last_fullsized":"2023-03-03T07:55:47.140594+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.133428+0000","last_change":"2023-03-03T07:55:47.124622+0000","last_active":"2023-03-03T07:55:47.133428+0000","last_peered":"2023-03-03T07:55:47.133428+0000","last_clean":"2023-03-03T07:55:47.133428+0000","last_became_active":"2023-03-03T07:55:47.124221+0000","last_became_peered":"2023-03-03T07:55:47.124221+0000","last_unstale":"2023-03-03T07:55:47.133428+0000","last_undegraded":"2023-03-03T07:55:47.133428+0000","last_fullsized":"2023-03-03T07:55:47.133428+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.133407+0000","last_change":"2023-03-03T07:55:47.123628+0000","last_active":"2023-03-03T07:55:47.133407+0000","last_peered":"2023-03-03T07:55:47.133407+0000","last_clean":"2023-03-03T07:55:47.133407+0000","last_became_active":"2023-03-03T07:55:47.123275+0000","last_became_peered":"2023-03-03T07:55:47.123275+0000","last_unstale":"2023-03-03T07:55:47.133407+0000","last_undegraded":"2023-03-03T07:55:47.133407+0000","last_fullsized":"2023-03-03T07:55:47.133407+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:21.542695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.129425+0000","last_change":"2023-03-03T07:55:47.127078+0000","last_active":"2023-03-03T07:55:47.129425+0000","last_peered":"2023-03-03T07:55:47.129425+0000","last_clean":"2023-03-03T07:55:47.129425+0000","last_became_active":"2023-03-03T07:55:47.126560+0000","last_became_peered":"2023-03-03T07:55:47.126560+0000","last_unstale":"2023-03-03T07:55:47.129425+0000","last_undegraded":"2023-03-03T07:55:47.129425+0000","last_fullsized":"2023-03-03T07:55:47.129425+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"23'1","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.132454+0000","last_change":"2023-03-03T07:55:46.566968+0000","last_active":"2023-03-03T07:55:47.132454+0000","last_peered":"2023-03-03T07:55:47.132454+0000","last_clean":"2023-03-03T07:55:47.132454+0000","last_became_active":"2023-03-03T07:55:46.566628+0000","last_became_peered":"2023-03-03T07:55:46.566628+0000","last_unstale":"2023-03-03T07:55:47.132454+0000","last_undegraded":"2023-03-03T07:55:47.132454+0000","last_fullsized":"2023-03-03T07:55:47.132454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:11.895405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.133105+0000","last_change":"2023-03-03T07:55:47.124330+0000","last_active":"2023-03-03T07:55:47.133105+0000","last_peered":"2023-03-03T07:55:47.133105+0000","last_clean":"2023-03-03T07:55:47.133105+0000","last_became_active":"2023-03-03T07:55:47.124015+0000","last_became_peered":"2023-03-03T07:55:47.124015+0000","last_unstale":"2023-03-03T07:55:47.133105+0000","last_undegraded":"2023-03-03T07:55:47.133105+0000","last_fullsized":"2023-03-03T07:55:47.133105+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:57:41.948092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.111029+0000","last_change":"2023-03-03T07:55:46.562055+0000","last_active":"2023-03-03T07:55:47.111029+0000","last_peered":"2023-03-03T07:55:47.111029+0000","last_clean":"2023-03-03T07:55:47.111029+0000","last_became_active":"2023-03-03T07:55:46.561625+0000","last_became_peered":"2023-03-03T07:55:46.561625+0000","last_unstale":"2023-03-03T07:55:47.111029+0000","last_undegraded":"2023-03-03T07:55:47.111029+0000","last_fullsized":"2023-03-03T07:55:47.111029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140601+0000","last_change":"2023-03-03T07:55:47.129628+0000","last_active":"2023-03-03T07:55:47.140601+0000","last_peered":"2023-03-03T07:55:47.140601+0000","last_clean":"2023-03-03T07:55:47.140601+0000","last_became_active":"2023-03-03T07:55:47.129356+0000","last_became_peered":"2023-03-03T07:55:47.129356+0000","last_unstale":"2023-03-03T07:55:47.140601+0000","last_undegraded":"2023-03-03T07:55:47.140601+0000","last_fullsized":"2023-03-03T07:55:47.140601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.102852+0000","last_change":"2023-03-03T07:55:46.559242+0000","last_active":"2023-03-03T07:55:47.102852+0000","last_peered":"2023-03-03T07:55:47.102852+0000","last_clean":"2023-03-03T07:55:47.102852+0000","last_became_active":"2023-03-03T07:55:46.558772+0000","last_became_peered":"2023-03-03T07:55:46.558772+0000","last_unstale":"2023-03-03T07:55:47.102852+0000","last_undegraded":"2023-03-03T07:55:47.102852+0000","last_fullsized":"2023-03-03T07:55:47.102852+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.119784+0000","last_change":"2023-03-03T07:55:47.114645+0000","last_active":"2023-03-03T07:55:47.119784+0000","last_peered":"2023-03-03T07:55:47.119784+0000","last_clean":"2023-03-03T07:55:47.119784+0000","last_became_active":"2023-03-03T07:55:47.114306+0000","last_became_peered":"2023-03-03T07:55:47.114306+0000","last_unstale":"2023-03-03T07:55:47.119784+0000","last_undegraded":"2023-03-03T07:55:47.119784+0000","last_fullsized":"2023-03-03T07:55:47.119784+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:29.066787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.109320+0000","last_change":"2023-03-03T07:55:46.567843+0000","last_active":"2023-03-03T07:55:47.109320+0000","last_peered":"2023-03-03T07:55:47.109320+0000","last_clean":"2023-03-03T07:55:47.109320+0000","last_became_active":"2023-03-03T07:55:46.567536+0000","last_became_peered":"2023-03-03T07:55:46.567536+0000","last_unstale":"2023-03-03T07:55:47.109320+0000","last_undegraded":"2023-03-03T07:55:47.109320+0000","last_fullsized":"2023-03-03T07:55:47.109320+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.132539+0000","last_change":"2023-03-03T07:55:47.126080+0000","last_active":"2023-03-03T07:55:47.132539+0000","last_peered":"2023-03-03T07:55:47.132539+0000","last_clean":"2023-03-03T07:55:47.132539+0000","last_became_active":"2023-03-03T07:55:47.125819+0000","last_became_peered":"2023-03-03T07:55:47.125819+0000","last_unstale":"2023-03-03T07:55:47.132539+0000","last_undegraded":"2023-03-03T07:55:47.132539+0000","last_fullsized":"2023-03-03T07:55:47.132539+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:42:18.110771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.107066+0000","last_change":"2023-03-03T07:55:46.564685+0000","last_active":"2023-03-03T07:55:47.107066+0000","last_peered":"2023-03-03T07:55:47.107066+0000","last_clean":"2023-03-03T07:55:47.107066+0000","last_became_active":"2023-03-03T07:55:46.564401+0000","last_became_peered":"2023-03-03T07:55:46.564401+0000","last_unstale":"2023-03-03T07:55:47.107066+0000","last_undegraded":"2023-03-03T07:55:47.107066+0000","last_fullsized":"2023-03-03T07:55:47.107066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.134139+0000","last_change":"2023-03-03T07:55:47.126626+0000","last_active":"2023-03-03T07:55:47.134139+0000","last_peered":"2023-03-03T07:55:47.134139+0000","last_clean":"2023-03-03T07:55:47.134139+0000","last_became_active":"2023-03-03T07:55:47.126341+0000","last_became_peered":"2023-03-03T07:55:47.126341+0000","last_unstale":"2023-03-03T07:55:47.134139+0000","last_undegraded":"2023-03-03T07:55:47.134139+0000","last_fullsized":"2023-03-03T07:55:47.134139+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"23'3","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.136059+0000","last_change":"2023-03-03T07:55:46.564428+0000","last_active":"2023-03-03T07:55:47.136059+0000","last_peered":"2023-03-03T07:55:47.136059+0000","last_clean":"2023-03-03T07:55:47.136059+0000","last_became_active":"2023-03-03T07:55:46.564082+0000","last_became_peered":"2023-03-03T07:55:46.564082+0000","last_unstale":"2023-03-03T07:55:47.136059+0000","last_undegraded":"2023-03-03T07:55:47.136059+0000","last_fullsized":"2023-03-03T07:55:47.136059+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:28:57.536524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138756+0000","last_change":"2023-03-03T07:55:47.135104+0000","last_active":"2023-03-03T07:55:47.138756+0000","last_peered":"2023-03-03T07:55:47.138756+0000","last_clean":"2023-03-03T07:55:47.138756+0000","last_became_active":"2023-03-03T07:55:47.134610+0000","last_became_peered":"2023-03-03T07:55:47.134610+0000","last_unstale":"2023-03-03T07:55:47.138756+0000","last_undegraded":"2023-03-03T07:55:47.138756+0000","last_fullsized":"2023-03-03T07:55:47.138756+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"23'6","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.143889+0000","last_change":"2023-03-03T07:55:46.563020+0000","last_active":"2023-03-03T07:55:47.143889+0000","last_peered":"2023-03-03T07:55:47.143889+0000","last_clean":"2023-03-03T07:55:47.143889+0000","last_became_active":"2023-03-03T07:55:46.562674+0000","last_became_peered":"2023-03-03T07:55:46.562674+0000","last_unstale":"2023-03-03T07:55:47.143889+0000","last_undegraded":"2023-03-03T07:55:47.143889+0000","last_fullsized":"2023-03-03T07:55:47.143889+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:30:20.455024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140330+0000","last_change":"2023-03-03T07:55:47.131440+0000","last_active":"2023-03-03T07:55:47.140330+0000","last_peered":"2023-03-03T07:55:47.140330+0000","last_clean":"2023-03-03T07:55:47.140330+0000","last_became_active":"2023-03-03T07:55:47.131159+0000","last_became_peered":"2023-03-03T07:55:47.131159+0000","last_unstale":"2023-03-03T07:55:47.140330+0000","last_undegraded":"2023-03-03T07:55:47.140330+0000","last_fullsized":"2023-03-03T07:55:47.140330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:29.837317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"23'2","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138652+0000","last_change":"2023-03-03T07:55:46.560188+0000","last_active":"2023-03-03T07:55:47.138652+0000","last_peered":"2023-03-03T07:55:47.138652+0000","last_clean":"2023-03-03T07:55:47.138652+0000","last_became_active":"2023-03-03T07:55:46.559853+0000","last_became_peered":"2023-03-03T07:55:46.559853+0000","last_unstale":"2023-03-03T07:55:47.138652+0000","last_undegraded":"2023-03-03T07:55:47.138652+0000","last_fullsized":"2023-03-03T07:55:47.138652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.137060+0000","last_change":"2023-03-03T07:55:47.131204+0000","last_active":"2023-03-03T07:55:47.137060+0000","last_peered":"2023-03-03T07:55:47.137060+0000","last_clean":"2023-03-03T07:55:47.137060+0000","last_became_active":"2023-03-03T07:55:47.130951+0000","last_became_peered":"2023-03-03T07:55:47.130951+0000","last_unstale":"2023-03-03T07:55:47.137060+0000","last_undegraded":"2023-03-03T07:55:47.137060+0000","last_fullsized":"2023-03-03T07:55:47.137060+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.108833+0000","last_change":"2023-03-03T07:55:46.567332+0000","last_active":"2023-03-03T07:55:47.108833+0000","last_peered":"2023-03-03T07:55:47.108833+0000","last_clean":"2023-03-03T07:55:47.108833+0000","last_became_active":"2023-03-03T07:55:46.567006+0000","last_became_peered":"2023-03-03T07:55:46.567006+0000","last_unstale":"2023-03-03T07:55:47.108833+0000","last_undegraded":"2023-03-03T07:55:47.108833+0000","last_fullsized":"2023-03-03T07:55:47.108833+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.129405+0000","last_change":"2023-03-03T07:55:47.126887+0000","last_active":"2023-03-03T07:55:47.129405+0000","last_peered":"2023-03-03T07:55:47.129405+0000","last_clean":"2023-03-03T07:55:47.129405+0000","last_became_active":"2023-03-03T07:55:47.125992+0000","last_became_peered":"2023-03-03T07:55:47.125992+0000","last_unstale":"2023-03-03T07:55:47.129405+0000","last_undegraded":"2023-03-03T07:55:47.129405+0000","last_fullsized":"2023-03-03T07:55:47.129405+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.119319+0000","last_change":"2023-03-03T07:55:46.559018+0000","last_active":"2023-03-03T07:55:47.119319+0000","last_peered":"2023-03-03T07:55:47.119319+0000","last_clean":"2023-03-03T07:55:47.119319+0000","last_became_active":"2023-03-03T07:55:46.558693+0000","last_became_peered":"2023-03-03T07:55:46.558693+0000","last_unstale":"2023-03-03T07:55:47.119319+0000","last_undegraded":"2023-03-03T07:55:47.119319+0000","last_fullsized":"2023-03-03T07:55:47.119319+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:28.058494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.137944+0000","last_change":"2023-03-03T07:55:47.123304+0000","last_active":"2023-03-03T07:55:47.137944+0000","last_peered":"2023-03-03T07:55:47.137944+0000","last_clean":"2023-03-03T07:55:47.137944+0000","last_became_active":"2023-03-03T07:55:47.118375+0000","last_became_peered":"2023-03-03T07:55:47.118375+0000","last_unstale":"2023-03-03T07:55:47.137944+0000","last_undegraded":"2023-03-03T07:55:47.137944+0000","last_fullsized":"2023-03-03T07:55:47.137944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:49:25.482566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"23'8","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.142364+0000","last_change":"2023-03-03T07:55:46.565523+0000","last_active":"2023-03-03T07:55:47.142364+0000","last_peered":"2023-03-03T07:55:47.142364+0000","last_clean":"2023-03-03T07:55:47.142364+0000","last_became_active":"2023-03-03T07:55:46.565158+0000","last_became_peered":"2023-03-03T07:55:46.565158+0000","last_unstale":"2023-03-03T07:55:47.142364+0000","last_undegraded":"2023-03-03T07:55:47.142364+0000","last_fullsized":"2023-03-03T07:55:47.142364+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:57.246539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.134491+0000","last_change":"2023-03-03T07:55:47.128607+0000","last_active":"2023-03-03T07:55:47.134491+0000","last_peered":"2023-03-03T07:55:47.134491+0000","last_clean":"2023-03-03T07:55:47.134491+0000","last_became_active":"2023-03-03T07:55:47.126465+0000","last_became_peered":"2023-03-03T07:55:47.126465+0000","last_unstale":"2023-03-03T07:55:47.134491+0000","last_undegraded":"2023-03-03T07:55:47.134491+0000","last_fullsized":"2023-03-03T07:55:47.134491+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:21.542695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"23'1","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141134+0000","last_change":"2023-03-03T07:55:46.560647+0000","last_active":"2023-03-03T07:55:47.141134+0000","last_peered":"2023-03-03T07:55:47.141134+0000","last_clean":"2023-03-03T07:55:47.141134+0000","last_became_active":"2023-03-03T07:55:46.559663+0000","last_became_peered":"2023-03-03T07:55:46.559663+0000","last_unstale":"2023-03-03T07:55:47.141134+0000","last_undegraded":"2023-03-03T07:55:47.141134+0000","last_fullsized":"2023-03-03T07:55:47.141134+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:54:12.785264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.129994+0000","last_change":"2023-03-03T07:55:47.126608+0000","last_active":"2023-03-03T07:55:47.129994+0000","last_peered":"2023-03-03T07:55:47.129994+0000","last_clean":"2023-03-03T07:55:47.129994+0000","last_became_active":"2023-03-03T07:55:47.126236+0000","last_became_peered":"2023-03-03T07:55:47.126236+0000","last_unstale":"2023-03-03T07:55:47.129994+0000","last_undegraded":"2023-03-03T07:55:47.129994+0000","last_fullsized":"2023-03-03T07:55:47.129994+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:03:11.053940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"23'2","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.142117+0000","last_change":"2023-03-03T07:55:46.558232+0000","last_active":"2023-03-03T07:55:47.142117+0000","last_peered":"2023-03-03T07:55:47.142117+0000","last_clean":"2023-03-03T07:55:47.142117+0000","last_became_active":"2023-03-03T07:55:46.557732+0000","last_became_peered":"2023-03-03T07:55:46.557732+0000","last_unstale":"2023-03-03T07:55:47.142117+0000","last_undegraded":"2023-03-03T07:55:47.142117+0000","last_fullsized":"2023-03-03T07:55:47.142117+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141631+0000","last_change":"2023-03-03T07:55:47.139286+0000","last_active":"2023-03-03T07:55:47.141631+0000","last_peered":"2023-03-03T07:55:47.141631+0000","last_clean":"2023-03-03T07:55:47.141631+0000","last_became_active":"2023-03-03T07:55:47.138539+0000","last_became_peered":"2023-03-03T07:55:47.138539+0000","last_unstale":"2023-03-03T07:55:47.141631+0000","last_undegraded":"2023-03-03T07:55:47.141631+0000","last_fullsized":"2023-03-03T07:55:47.141631+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:56.705526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"23'1","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.132951+0000","last_change":"2023-03-03T07:55:46.559244+0000","last_active":"2023-03-03T07:55:47.132951+0000","last_peered":"2023-03-03T07:55:47.132951+0000","last_clean":"2023-03-03T07:55:47.132951+0000","last_became_active":"2023-03-03T07:55:46.558832+0000","last_became_peered":"2023-03-03T07:55:46.558832+0000","last_unstale":"2023-03-03T07:55:47.132951+0000","last_undegraded":"2023-03-03T07:55:47.132951+0000","last_fullsized":"2023-03-03T07:55:47.132951+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:06:23.082465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.132403+0000","last_change":"2023-03-03T07:55:47.128420+0000","last_active":"2023-03-03T07:55:47.132403+0000","last_peered":"2023-03-03T07:55:47.132403+0000","last_clean":"2023-03-03T07:55:47.132403+0000","last_became_active":"2023-03-03T07:55:47.128117+0000","last_became_peered":"2023-03-03T07:55:47.128117+0000","last_unstale":"2023-03-03T07:55:47.132403+0000","last_undegraded":"2023-03-03T07:55:47.132403+0000","last_fullsized":"2023-03-03T07:55:47.132403+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:28:58.544816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.107330+0000","last_change":"2023-03-03T07:55:46.563195+0000","last_active":"2023-03-03T07:55:47.107330+0000","last_peered":"2023-03-03T07:55:47.107330+0000","last_clean":"2023-03-03T07:55:47.107330+0000","last_became_active":"2023-03-03T07:55:46.562774+0000","last_became_peered":"2023-03-03T07:55:46.562774+0000","last_unstale":"2023-03-03T07:55:47.107330+0000","last_undegraded":"2023-03-03T07:55:47.107330+0000","last_fullsized":"2023-03-03T07:55:47.107330+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:23.109334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140538+0000","last_change":"2023-03-03T07:55:47.132833+0000","last_active":"2023-03-03T07:55:47.140538+0000","last_peered":"2023-03-03T07:55:47.140538+0000","last_clean":"2023-03-03T07:55:47.140538+0000","last_became_active":"2023-03-03T07:55:47.132530+0000","last_became_peered":"2023-03-03T07:55:47.132530+0000","last_unstale":"2023-03-03T07:55:47.140538+0000","last_undegraded":"2023-03-03T07:55:47.140538+0000","last_fullsized":"2023-03-03T07:55:47.140538+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"23'2","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141011+0000","last_change":"2023-03-03T07:55:46.561556+0000","last_active":"2023-03-03T07:55:47.141011+0000","last_peered":"2023-03-03T07:55:47.141011+0000","last_clean":"2023-03-03T07:55:47.141011+0000","last_became_active":"2023-03-03T07:55:46.561273+0000","last_became_peered":"2023-03-03T07:55:46.561273+0000","last_unstale":"2023-03-03T07:55:47.141011+0000","last_undegraded":"2023-03-03T07:55:47.141011+0000","last_fullsized":"2023-03-03T07:55:47.141011+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141813+0000","last_change":"2023-03-03T07:55:47.138654+0000","last_active":"2023-03-03T07:55:47.141813+0000","last_peered":"2023-03-03T07:55:47.141813+0000","last_clean":"2023-03-03T07:55:47.141813+0000","last_became_active":"2023-03-03T07:55:47.131453+0000","last_became_peered":"2023-03-03T07:55:47.131453+0000","last_unstale":"2023-03-03T07:55:47.141813+0000","last_undegraded":"2023-03-03T07:55:47.141813+0000","last_fullsized":"2023-03-03T07:55:47.141813+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:36:18.587660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.105058+0000","last_change":"2023-03-03T07:55:46.563763+0000","last_active":"2023-03-03T07:55:47.105058+0000","last_peered":"2023-03-03T07:55:47.105058+0000","last_clean":"2023-03-03T07:55:47.105058+0000","last_became_active":"2023-03-03T07:55:46.563433+0000","last_became_peered":"2023-03-03T07:55:46.563433+0000","last_unstale":"2023-03-03T07:55:47.105058+0000","last_undegraded":"2023-03-03T07:55:47.105058+0000","last_fullsized":"2023-03-03T07:55:47.105058+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:12.972346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141078+0000","last_change":"2023-03-03T07:55:47.134804+0000","last_active":"2023-03-03T07:55:47.141078+0000","last_peered":"2023-03-03T07:55:47.141078+0000","last_clean":"2023-03-03T07:55:47.141078+0000","last_became_active":"2023-03-03T07:55:47.134504+0000","last_became_peered":"2023-03-03T07:55:47.134504+0000","last_unstale":"2023-03-03T07:55:47.141078+0000","last_undegraded":"2023-03-03T07:55:47.141078+0000","last_fullsized":"2023-03-03T07:55:47.141078+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.112117+0000","last_change":"2023-03-03T07:55:46.561627+0000","last_active":"2023-03-03T07:55:47.112117+0000","last_peered":"2023-03-03T07:55:47.112117+0000","last_clean":"2023-03-03T07:55:47.112117+0000","last_became_active":"2023-03-03T07:55:46.561129+0000","last_became_peered":"2023-03-03T07:55:46.561129+0000","last_unstale":"2023-03-03T07:55:47.112117+0000","last_undegraded":"2023-03-03T07:55:47.112117+0000","last_fullsized":"2023-03-03T07:55:47.112117+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:12.972346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.131137+0000","last_change":"2023-03-03T07:55:47.119813+0000","last_active":"2023-03-03T07:55:47.131137+0000","last_peered":"2023-03-03T07:55:47.131137+0000","last_clean":"2023-03-03T07:55:47.131137+0000","last_became_active":"2023-03-03T07:55:47.119517+0000","last_became_peered":"2023-03-03T07:55:47.119517+0000","last_unstale":"2023-03-03T07:55:47.131137+0000","last_undegraded":"2023-03-03T07:55:47.131137+0000","last_fullsized":"2023-03-03T07:55:47.131137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:56:43.323290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.102139+0000","last_change":"2023-03-03T07:55:46.557334+0000","last_active":"2023-03-03T07:55:47.102139+0000","last_peered":"2023-03-03T07:55:47.102139+0000","last_clean":"2023-03-03T07:55:47.102139+0000","last_became_active":"2023-03-03T07:55:46.556973+0000","last_became_peered":"2023-03-03T07:55:46.556973+0000","last_unstale":"2023-03-03T07:55:47.102139+0000","last_undegraded":"2023-03-03T07:55:47.102139+0000","last_fullsized":"2023-03-03T07:55:47.102139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.133390+0000","last_change":"2023-03-03T07:55:47.123429+0000","last_active":"2023-03-03T07:55:47.133390+0000","last_peered":"2023-03-03T07:55:47.133390+0000","last_clean":"2023-03-03T07:55:47.133390+0000","last_became_active":"2023-03-03T07:55:47.123131+0000","last_became_peered":"2023-03-03T07:55:47.123131+0000","last_unstale":"2023-03-03T07:55:47.133390+0000","last_undegraded":"2023-03-03T07:55:47.133390+0000","last_fullsized":"2023-03-03T07:55:47.133390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.560834+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T07:55:46.560834+0000","last_peered":"2023-03-03T07:55:46.560834+0000","last_clean":"2023-03-03T07:55:46.560834+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T07:55:46.560834+0000","last_undegraded":"2023-03-03T07:55:46.560834+0000","last_fullsized":"2023-03-03T07:55:46.560834+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.108422+0000","last_change":"2023-03-03T07:55:46.565316+0000","last_active":"2023-03-03T07:55:47.108422+0000","last_peered":"2023-03-03T07:55:47.108422+0000","last_clean":"2023-03-03T07:55:47.108422+0000","last_became_active":"2023-03-03T07:55:46.565027+0000","last_became_peered":"2023-03-03T07:55:46.565027+0000","last_unstale":"2023-03-03T07:55:47.108422+0000","last_undegraded":"2023-03-03T07:55:47.108422+0000","last_fullsized":"2023-03-03T07:55:47.108422+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:28.058494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.134097+0000","last_change":"2023-03-03T07:55:47.124467+0000","last_active":"2023-03-03T07:55:47.134097+0000","last_peered":"2023-03-03T07:55:47.134097+0000","last_clean":"2023-03-03T07:55:47.134097+0000","last_became_active":"2023-03-03T07:55:47.124149+0000","last_became_peered":"2023-03-03T07:55:47.124149+0000","last_unstale":"2023-03-03T07:55:47.134097+0000","last_undegraded":"2023-03-03T07:55:47.134097+0000","last_fullsized":"2023-03-03T07:55:47.134097+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.545932+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T07:55:46.545932+0000","last_peered":"2023-03-03T07:55:46.545932+0000","last_clean":"2023-03-03T07:55:46.545932+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T07:55:46.545932+0000","last_undegraded":"2023-03-03T07:55:46.545932+0000","last_fullsized":"2023-03-03T07:55:46.545932+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.113292+0000","last_change":"2023-03-03T07:55:46.559574+0000","last_active":"2023-03-03T07:55:47.113292+0000","last_peered":"2023-03-03T07:55:47.113292+0000","last_clean":"2023-03-03T07:55:47.113292+0000","last_became_active":"2023-03-03T07:55:46.559192+0000","last_became_peered":"2023-03-03T07:55:46.559192+0000","last_unstale":"2023-03-03T07:55:47.113292+0000","last_undegraded":"2023-03-03T07:55:47.113292+0000","last_fullsized":"2023-03-03T07:55:47.113292+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.133460+0000","last_change":"2023-03-03T07:55:47.127703+0000","last_active":"2023-03-03T07:55:47.133460+0000","last_peered":"2023-03-03T07:55:47.133460+0000","last_clean":"2023-03-03T07:55:47.133460+0000","last_became_active":"2023-03-03T07:55:47.126941+0000","last_became_peered":"2023-03-03T07:55:47.126941+0000","last_unstale":"2023-03-03T07:55:47.133460+0000","last_undegraded":"2023-03-03T07:55:47.133460+0000","last_fullsized":"2023-03-03T07:55:47.133460+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:03:11.053940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.539825+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T07:55:46.539825+0000","last_peered":"2023-03-03T07:55:46.539825+0000","last_clean":"2023-03-03T07:55:46.539825+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T07:55:46.539825+0000","last_undegraded":"2023-03-03T07:55:46.539825+0000","last_fullsized":"2023-03-03T07:55:46.539825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.110196+0000","last_change":"2023-03-03T07:55:46.560453+0000","last_active":"2023-03-03T07:55:47.110196+0000","last_peered":"2023-03-03T07:55:47.110196+0000","last_clean":"2023-03-03T07:55:47.110196+0000","last_became_active":"2023-03-03T07:55:46.560140+0000","last_became_peered":"2023-03-03T07:55:46.560140+0000","last_unstale":"2023-03-03T07:55:47.110196+0000","last_undegraded":"2023-03-03T07:55:47.110196+0000","last_fullsized":"2023-03-03T07:55:47.110196+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.133411+0000","last_change":"2023-03-03T07:55:47.124151+0000","last_active":"2023-03-03T07:55:47.133411+0000","last_peered":"2023-03-03T07:55:47.133411+0000","last_clean":"2023-03-03T07:55:47.133411+0000","last_became_active":"2023-03-03T07:55:47.123697+0000","last_became_peered":"2023-03-03T07:55:47.123697+0000","last_unstale":"2023-03-03T07:55:47.133411+0000","last_undegraded":"2023-03-03T07:55:47.133411+0000","last_fullsized":"2023-03-03T07:55:47.133411+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.546040+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T07:55:46.546040+0000","last_peered":"2023-03-03T07:55:46.546040+0000","last_clean":"2023-03-03T07:55:46.546040+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T07:55:46.546040+0000","last_undegraded":"2023-03-03T07:55:46.546040+0000","last_fullsized":"2023-03-03T07:55:46.546040+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.544672+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T07:55:46.544672+0000","last_peered":"2023-03-03T07:55:46.544672+0000","last_clean":"2023-03-03T07:55:46.544672+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T07:55:46.544672+0000","last_undegraded":"2023-03-03T07:55:46.544672+0000","last_fullsized":"2023-03-03T07:55:46.544672+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.102452+0000","last_change":"2023-03-03T07:55:46.559444+0000","last_active":"2023-03-03T07:55:47.102452+0000","last_peered":"2023-03-03T07:55:47.102452+0000","last_clean":"2023-03-03T07:55:47.102452+0000","last_became_active":"2023-03-03T07:55:46.559147+0000","last_became_peered":"2023-03-03T07:55:46.559147+0000","last_unstale":"2023-03-03T07:55:47.102452+0000","last_undegraded":"2023-03-03T07:55:47.102452+0000","last_fullsized":"2023-03-03T07:55:47.102452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:30:20.455024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.137452+0000","last_change":"2023-03-03T07:55:47.134945+0000","last_active":"2023-03-03T07:55:47.137452+0000","last_peered":"2023-03-03T07:55:47.137452+0000","last_clean":"2023-03-03T07:55:47.137452+0000","last_became_active":"2023-03-03T07:55:47.134662+0000","last_became_peered":"2023-03-03T07:55:47.134662+0000","last_unstale":"2023-03-03T07:55:47.137452+0000","last_undegraded":"2023-03-03T07:55:47.137452+0000","last_fullsized":"2023-03-03T07:55:47.137452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:54:13.793557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.544846+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T07:55:46.544846+0000","last_peered":"2023-03-03T07:55:46.544846+0000","last_clean":"2023-03-03T07:55:46.544846+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T07:55:46.544846+0000","last_undegraded":"2023-03-03T07:55:46.544846+0000","last_fullsized":"2023-03-03T07:55:46.544846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.108690+0000","last_change":"2023-03-03T07:55:46.567402+0000","last_active":"2023-03-03T07:55:47.108690+0000","last_peered":"2023-03-03T07:55:47.108690+0000","last_clean":"2023-03-03T07:55:47.108690+0000","last_became_active":"2023-03-03T07:55:46.567013+0000","last_became_peered":"2023-03-03T07:55:46.567013+0000","last_unstale":"2023-03-03T07:55:47.108690+0000","last_undegraded":"2023-03-03T07:55:47.108690+0000","last_fullsized":"2023-03-03T07:55:47.108690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140579+0000","last_change":"2023-03-03T07:55:47.125428+0000","last_active":"2023-03-03T07:55:47.140579+0000","last_peered":"2023-03-03T07:55:47.140579+0000","last_clean":"2023-03-03T07:55:47.140579+0000","last_became_active":"2023-03-03T07:55:47.125113+0000","last_became_peered":"2023-03-03T07:55:47.125113+0000","last_unstale":"2023-03-03T07:55:47.140579+0000","last_undegraded":"2023-03-03T07:55:47.140579+0000","last_fullsized":"2023-03-03T07:55:47.140579+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.545445+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T07:55:46.545445+0000","last_peered":"2023-03-03T07:55:46.545445+0000","last_clean":"2023-03-03T07:55:46.545445+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T07:55:46.545445+0000","last_undegraded":"2023-03-03T07:55:46.545445+0000","last_fullsized":"2023-03-03T07:55:46.545445+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.114036+0000","last_change":"2023-03-03T07:55:46.556310+0000","last_active":"2023-03-03T07:55:47.114036+0000","last_peered":"2023-03-03T07:55:47.114036+0000","last_clean":"2023-03-03T07:55:47.114036+0000","last_became_active":"2023-03-03T07:55:46.555567+0000","last_became_peered":"2023-03-03T07:55:46.555567+0000","last_unstale":"2023-03-03T07:55:47.114036+0000","last_undegraded":"2023-03-03T07:55:47.114036+0000","last_fullsized":"2023-03-03T07:55:47.114036+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138083+0000","last_change":"2023-03-03T07:55:47.119872+0000","last_active":"2023-03-03T07:55:47.138083+0000","last_peered":"2023-03-03T07:55:47.138083+0000","last_clean":"2023-03-03T07:55:47.138083+0000","last_became_active":"2023-03-03T07:55:47.119546+0000","last_became_peered":"2023-03-03T07:55:47.119546+0000","last_unstale":"2023-03-03T07:55:47.138083+0000","last_undegraded":"2023-03-03T07:55:47.138083+0000","last_fullsized":"2023-03-03T07:55:47.138083+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"23'2","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.139075+0000","last_change":"2023-03-03T07:55:46.559460+0000","last_active":"2023-03-03T07:55:47.139075+0000","last_peered":"2023-03-03T07:55:47.139075+0000","last_clean":"2023-03-03T07:55:47.139075+0000","last_became_active":"2023-03-03T07:55:46.559134+0000","last_became_peered":"2023-03-03T07:55:46.559134+0000","last_unstale":"2023-03-03T07:55:47.139075+0000","last_undegraded":"2023-03-03T07:55:47.139075+0000","last_fullsized":"2023-03-03T07:55:47.139075+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:11.895405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.561690+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T07:55:46.561690+0000","last_peered":"2023-03-03T07:55:46.561690+0000","last_clean":"2023-03-03T07:55:46.561690+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T07:55:46.561690+0000","last_undegraded":"2023-03-03T07:55:46.561690+0000","last_fullsized":"2023-03-03T07:55:46.561690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.134183+0000","last_change":"2023-03-03T07:55:47.127917+0000","last_active":"2023-03-03T07:55:47.134183+0000","last_peered":"2023-03-03T07:55:47.134183+0000","last_clean":"2023-03-03T07:55:47.134183+0000","last_became_active":"2023-03-03T07:55:47.127606+0000","last_became_peered":"2023-03-03T07:55:47.127606+0000","last_unstale":"2023-03-03T07:55:47.134183+0000","last_undegraded":"2023-03-03T07:55:47.134183+0000","last_fullsized":"2023-03-03T07:55:47.134183+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T07:55:46.555364+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T07:55:46.555364+0000","last_peered":"2023-03-03T07:55:46.555364+0000","last_clean":"2023-03-03T07:55:46.555364+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T07:55:46.555364+0000","last_undegraded":"2023-03-03T07:55:46.555364+0000","last_fullsized":"2023-03-03T07:55:46.555364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.109126+0000","last_change":"2023-03-03T07:55:46.565365+0000","last_active":"2023-03-03T07:55:47.109126+0000","last_peered":"2023-03-03T07:55:47.109126+0000","last_clean":"2023-03-03T07:55:47.109126+0000","last_became_active":"2023-03-03T07:55:46.565097+0000","last_became_peered":"2023-03-03T07:55:46.565097+0000","last_unstale":"2023-03-03T07:55:47.109126+0000","last_undegraded":"2023-03-03T07:55:47.109126+0000","last_fullsized":"2023-03-03T07:55:47.109126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:23.109334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141939+0000","last_change":"2023-03-03T07:55:47.139732+0000","last_active":"2023-03-03T07:55:47.141939+0000","last_peered":"2023-03-03T07:55:47.141939+0000","last_clean":"2023-03-03T07:55:47.141939+0000","last_became_active":"2023-03-03T07:55:47.138505+0000","last_became_peered":"2023-03-03T07:55:47.138505+0000","last_unstale":"2023-03-03T07:55:47.141939+0000","last_undegraded":"2023-03-03T07:55:47.141939+0000","last_fullsized":"2023-03-03T07:55:47.141939+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:49:25.482566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"23'1","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.138125+0000","last_change":"2023-03-03T07:55:46.568574+0000","last_active":"2023-03-03T07:55:47.138125+0000","last_peered":"2023-03-03T07:55:47.138125+0000","last_clean":"2023-03-03T07:55:47.138125+0000","last_became_active":"2023-03-03T07:55:46.568139+0000","last_became_peered":"2023-03-03T07:55:46.568139+0000","last_unstale":"2023-03-03T07:55:47.138125+0000","last_undegraded":"2023-03-03T07:55:47.138125+0000","last_fullsized":"2023-03-03T07:55:47.138125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.134013+0000","last_change":"2023-03-03T07:55:47.128846+0000","last_active":"2023-03-03T07:55:47.134013+0000","last_peered":"2023-03-03T07:55:47.134013+0000","last_clean":"2023-03-03T07:55:47.134013+0000","last_became_active":"2023-03-03T07:55:47.128385+0000","last_became_peered":"2023-03-03T07:55:47.128385+0000","last_unstale":"2023-03-03T07:55:47.134013+0000","last_undegraded":"2023-03-03T07:55:47.134013+0000","last_fullsized":"2023-03-03T07:55:47.134013+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:00.511475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.135756+0000","last_change":"2023-03-03T07:55:46.560316+0000","last_active":"2023-03-03T07:55:47.135756+0000","last_peered":"2023-03-03T07:55:47.135756+0000","last_clean":"2023-03-03T07:55:47.135756+0000","last_became_active":"2023-03-03T07:55:46.560031+0000","last_became_peered":"2023-03-03T07:55:46.560031+0000","last_unstale":"2023-03-03T07:55:47.135756+0000","last_undegraded":"2023-03-03T07:55:47.135756+0000","last_fullsized":"2023-03-03T07:55:47.135756+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140798+0000","last_change":"2023-03-03T07:55:47.137604+0000","last_active":"2023-03-03T07:55:47.140798+0000","last_peered":"2023-03-03T07:55:47.140798+0000","last_clean":"2023-03-03T07:55:47.140798+0000","last_became_active":"2023-03-03T07:55:47.131552+0000","last_became_peered":"2023-03-03T07:55:47.131552+0000","last_unstale":"2023-03-03T07:55:47.140798+0000","last_undegraded":"2023-03-03T07:55:47.140798+0000","last_fullsized":"2023-03-03T07:55:47.140798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"23'2","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.135154+0000","last_change":"2023-03-03T07:55:46.560878+0000","last_active":"2023-03-03T07:55:47.135154+0000","last_peered":"2023-03-03T07:55:47.135154+0000","last_clean":"2023-03-03T07:55:47.135154+0000","last_became_active":"2023-03-03T07:55:46.560534+0000","last_became_peered":"2023-03-03T07:55:46.560534+0000","last_unstale":"2023-03-03T07:55:47.135154+0000","last_undegraded":"2023-03-03T07:55:47.135154+0000","last_fullsized":"2023-03-03T07:55:47.135154+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.132540+0000","last_change":"2023-03-03T07:55:47.122002+0000","last_active":"2023-03-03T07:55:47.132540+0000","last_peered":"2023-03-03T07:55:47.132540+0000","last_clean":"2023-03-03T07:55:47.132540+0000","last_became_active":"2023-03-03T07:55:47.121732+0000","last_became_peered":"2023-03-03T07:55:47.121732+0000","last_unstale":"2023-03-03T07:55:47.132540+0000","last_undegraded":"2023-03-03T07:55:47.132540+0000","last_fullsized":"2023-03-03T07:55:47.132540+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"23'1","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141645+0000","last_change":"2023-03-03T07:55:46.560027+0000","last_active":"2023-03-03T07:55:47.141645+0000","last_peered":"2023-03-03T07:55:47.141645+0000","last_clean":"2023-03-03T07:55:47.141645+0000","last_became_active":"2023-03-03T07:55:46.559508+0000","last_became_peered":"2023-03-03T07:55:46.559508+0000","last_unstale":"2023-03-03T07:55:47.141645+0000","last_undegraded":"2023-03-03T07:55:47.141645+0000","last_fullsized":"2023-03-03T07:55:47.141645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:23.109334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.132455+0000","last_change":"2023-03-03T07:55:47.121846+0000","last_active":"2023-03-03T07:55:47.132455+0000","last_peered":"2023-03-03T07:55:47.132455+0000","last_clean":"2023-03-03T07:55:47.132455+0000","last_became_active":"2023-03-03T07:55:47.121563+0000","last_became_peered":"2023-03-03T07:55:47.121563+0000","last_unstale":"2023-03-03T07:55:47.132455+0000","last_undegraded":"2023-03-03T07:55:47.132455+0000","last_fullsized":"2023-03-03T07:55:47.132455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:36:18.587660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"23'4","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140768+0000","last_change":"2023-03-03T07:55:46.567878+0000","last_active":"2023-03-03T07:55:47.140768+0000","last_peered":"2023-03-03T07:55:47.140768+0000","last_clean":"2023-03-03T07:55:47.140768+0000","last_became_active":"2023-03-03T07:55:46.567593+0000","last_became_peered":"2023-03-03T07:55:46.567593+0000","last_unstale":"2023-03-03T07:55:47.140768+0000","last_undegraded":"2023-03-03T07:55:47.140768+0000","last_fullsized":"2023-03-03T07:55:47.140768+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:28.058494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.140213+0000","last_change":"2023-03-03T07:55:47.131373+0000","last_active":"2023-03-03T07:55:47.140213+0000","last_peered":"2023-03-03T07:55:47.140213+0000","last_clean":"2023-03-03T07:55:47.140213+0000","last_became_active":"2023-03-03T07:55:47.131037+0000","last_became_peered":"2023-03-03T07:55:47.131037+0000","last_unstale":"2023-03-03T07:55:47.140213+0000","last_undegraded":"2023-03-03T07:55:47.140213+0000","last_fullsized":"2023-03-03T07:55:47.140213+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:52:08.323505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.128848+0000","last_change":"2023-03-03T07:55:47.122842+0000","last_active":"2023-03-03T07:55:47.128848+0000","last_peered":"2023-03-03T07:55:47.128848+0000","last_clean":"2023-03-03T07:55:47.128848+0000","last_became_active":"2023-03-03T07:55:47.122580+0000","last_became_peered":"2023-03-03T07:55:47.122580+0000","last_unstale":"2023-03-03T07:55:47.128848+0000","last_undegraded":"2023-03-03T07:55:47.128848+0000","last_fullsized":"2023-03-03T07:55:47.128848+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:06:24.090758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.130081+0000","last_change":"2023-03-03T07:55:47.121284+0000","last_active":"2023-03-03T07:55:47.130081+0000","last_peered":"2023-03-03T07:55:47.130081+0000","last_clean":"2023-03-03T07:55:47.130081+0000","last_became_active":"2023-03-03T07:55:47.121012+0000","last_became_peered":"2023-03-03T07:55:47.121012+0000","last_unstale":"2023-03-03T07:55:47.130081+0000","last_undegraded":"2023-03-03T07:55:47.130081+0000","last_fullsized":"2023-03-03T07:55:47.130081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:15.870956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.141235+0000","last_change":"2023-03-03T07:55:47.138325+0000","last_active":"2023-03-03T07:55:47.141235+0000","last_peered":"2023-03-03T07:55:47.141235+0000","last_clean":"2023-03-03T07:55:47.141235+0000","last_became_active":"2023-03-03T07:55:47.138019+0000","last_became_peered":"2023-03-03T07:55:47.138019+0000","last_unstale":"2023-03-03T07:55:47.141235+0000","last_undegraded":"2023-03-03T07:55:47.141235+0000","last_fullsized":"2023-03-03T07:55:47.141235+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:57.306479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.134158+0000","last_change":"2023-03-03T07:55:47.128792+0000","last_active":"2023-03-03T07:55:47.134158+0000","last_peered":"2023-03-03T07:55:47.134158+0000","last_clean":"2023-03-03T07:55:47.134158+0000","last_became_active":"2023-03-03T07:55:47.127248+0000","last_became_peered":"2023-03-03T07:55:47.127248+0000","last_unstale":"2023-03-03T07:55:47.134158+0000","last_undegraded":"2023-03-03T07:55:47.134158+0000","last_fullsized":"2023-03-03T07:55:47.134158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:49:25.482566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.133442+0000","last_change":"2023-03-03T07:55:47.127209+0000","last_active":"2023-03-03T07:55:47.133442+0000","last_peered":"2023-03-03T07:55:47.133442+0000","last_clean":"2023-03-03T07:55:47.133442+0000","last_became_active":"2023-03-03T07:55:47.126100+0000","last_became_peered":"2023-03-03T07:55:47.126100+0000","last_unstale":"2023-03-03T07:55:47.133442+0000","last_undegraded":"2023-03-03T07:55:47.133442+0000","last_fullsized":"2023-03-03T07:55:47.133442+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:46.306038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9004,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94362836,"statfs":{"total":96636764160,"available":96627544064,"internally_reserved":0,"allocated":823296,"data_stored":671596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7908,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363932,"statfs":{"total":96636764160,"available":96628666368,"internally_reserved":0,"allocated":225280,"data_stored":80209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7912,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363928,"statfs":{"total":96636764160,"available":96628662272,"internally_reserved":0,"allocated":229376,"data_stored":81228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8924,"kb_used_data":788,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94362916,"statfs":{"total":96636764160,"available":96627625984,"internally_reserved":0,"allocated":806912,"data_stored":663957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7908,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363932,"statfs":{"total":96636764160,"available":96628666368,"internally_reserved":0,"allocated":225280,"data_stored":80209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.407}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3620000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7848,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":94363992,"statfs":{"total":96636764160,"available":96628727808,"internally_reserved":0,"allocated":229376,"data_stored":81228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7912,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363928,"statfs":{"total":96636764160,"available":96628662272,"internally_reserved":0,"allocated":229376,"data_stored":81228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7908,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363932,"statfs":{"total":96636764160,"available":96628666368,"internally_reserved":0,"allocated":225280,"data_stored":80209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T07:55:52.646 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-03-03T07:55:52.647 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-03-03T07:55:52.647 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-03-03T07:55:52.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-03-03T07:55:52.649 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-03-03T07:55:52.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.918+0000 7fdedc5ff700 1 Processor -- start 2023-03-03T07:55:52.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.918+0000 7fdedc5ff700 1 -- start start 2023-03-03T07:55:52.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.919+0000 7fdedc5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fded415cc00 0x7fded40cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:52.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.919+0000 7fdedc5ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fded40d27c0 0x7fded40d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:52.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.919+0000 7fdedc5ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded40cde10 con 0x7fded40b60d0 2023-03-03T07:55:52.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.919+0000 7fdedc5ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fded40ce040 con 0x7fded40d27c0 2023-03-03T07:55:52.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.919+0000 7fdedc5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded4073be0 con 0x7fded415cc00 2023-03-03T07:55:52.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.920+0000 7fded9b9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fded40d27c0 0x7fded40d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:52.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.920+0000 7fded9b9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fded40d27c0 0x7fded40d2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45332/0 (socket says 172.21.15.98:45332) 2023-03-03T07:55:52.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.920+0000 7fded9b9a700 1 -- 172.21.15.98:0/3576800525 learned_addr learned my addr 172.21.15.98:0/3576800525 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:52.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.920+0000 7fdeda39b700 1 --2- 172.21.15.98:0/3576800525 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fded415cc00 0x7fded40cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:52.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.920+0000 7fded9b9a700 1 -- 172.21.15.98:0/3576800525 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fded415cc00 msgr2=0x7fded40cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:52.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.920+0000 7fded9b9a700 1 --2- 172.21.15.98:0/3576800525 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fded415cc00 0x7fded40cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.920+0000 7fded9b9a700 1 -- 172.21.15.98:0/3576800525 >> v1:172.21.15.100:6789/0 conn(0x7fded40b60d0 legacy=0x7fded40b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:52.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.920+0000 7fded9b9a700 1 -- 172.21.15.98:0/3576800525 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded40d1a00 con 0x7fded40d27c0 2023-03-03T07:55:52.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.921+0000 7fded9b9a700 1 --2- 172.21.15.98:0/3576800525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded40d27c0 0x7fded40d2c30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdec40098f0 tx=0x7fdec4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e40febd21250c60 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:52.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.921+0000 7fded9399700 1 -- 172.21.15.98:0/3576800525 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec400b8a0 con 0x7fded40d27c0 2023-03-03T07:55:52.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.921+0000 7fded9399700 1 -- 172.21.15.98:0/3576800525 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec400ba00 con 0x7fded40d27c0 2023-03-03T07:55:52.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.921+0000 7fded9399700 1 -- 172.21.15.98:0/3576800525 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec400bd90 con 0x7fded40d27c0 2023-03-03T07:55:52.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.921+0000 7fdedc5ff700 1 -- 172.21.15.98:0/3576800525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded40d27c0 msgr2=0x7fded40d2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:52.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.921+0000 7fdedc5ff700 1 --2- 172.21.15.98:0/3576800525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded40d27c0 0x7fded40d2c30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdec40098f0 tx=0x7fdec4009c00 comp rx=0 tx=0).stop 2023-03-03T07:55:52.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.922+0000 7fdedc5ff700 1 -- 172.21.15.98:0/3576800525 shutdown_connections 2023-03-03T07:55:52.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.922+0000 7fdedc5ff700 1 --2- 172.21.15.98:0/3576800525 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fded415cc00 0x7fded40cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.922+0000 7fdedc5ff700 1 --2- 172.21.15.98:0/3576800525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded40d27c0 0x7fded40d2c30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.922+0000 7fdedc5ff700 1 -- 172.21.15.98:0/3576800525 >> 172.21.15.98:0/3576800525 conn(0x7fded406aee0 msgr2=0x7fded406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:52.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.922+0000 7fdedc5ff700 1 -- 172.21.15.98:0/3576800525 shutdown_connections 2023-03-03T07:55:52.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.922+0000 7fdedc5ff700 1 -- 172.21.15.98:0/3576800525 wait complete. 2023-03-03T07:55:52.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.924+0000 7fdedc5ff700 1 Processor -- start 2023-03-03T07:55:52.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.925+0000 7fdedc5ff700 1 -- start start 2023-03-03T07:55:52.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.926+0000 7fdedc5ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fded40b60d0 0x7fded414f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:52.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.926+0000 7fdedc5ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded40d27c0 0x7fded4152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:52.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.926+0000 7fdedc5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fded415cc00 0x7fded4153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:52.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.926+0000 7fdedc5ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded414fba0 con 0x7fded40d27c0 2023-03-03T07:55:52.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.926+0000 7fdedc5ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded40cd6b0 con 0x7fded40b60d0 2023-03-03T07:55:52.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.926+0000 7fdedc5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded41f35b0 con 0x7fded415cc00 2023-03-03T07:55:52.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.926+0000 7fded9b9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded40d27c0 0x7fded4152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:52.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.926+0000 7fded9b9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded40d27c0 0x7fded4152de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45340/0 (socket says 172.21.15.98:45340) 2023-03-03T07:55:52.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.926+0000 7fded9b9a700 1 -- 172.21.15.98:0/1259192214 learned_addr learned my addr 172.21.15.98:0/1259192214 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:52.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.927+0000 7fdeda39b700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fded40b60d0 0x7fded414f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:52.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.927+0000 7fded9b9a700 1 -- 172.21.15.98:0/1259192214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fded415cc00 msgr2=0x7fded4153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:52.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.927+0000 7fdedab9c700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fded415cc00 0x7fded4153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:52.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.927+0000 7fded9b9a700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fded415cc00 0x7fded4153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.927+0000 7fded9b9a700 1 -- 172.21.15.98:0/1259192214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fded40b60d0 msgr2=0x7fded414f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:52.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.927+0000 7fded9b9a700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fded40b60d0 0x7fded414f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:52.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.927+0000 7fded9b9a700 1 -- 172.21.15.98:0/1259192214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec4009550 con 0x7fded40d27c0 2023-03-03T07:55:52.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.927+0000 7fdeda39b700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fded40b60d0 0x7fded414f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:55:52.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.927+0000 7fdedab9c700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fded415cc00 0x7fded4153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:55:52.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.927+0000 7fded9b9a700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded40d27c0 0x7fded4152de0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdec4014040 tx=0x7fdec4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:52.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.927+0000 7fdecaffd700 1 -- 172.21.15.98:0/1259192214 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec40036a0 con 0x7fded40d27c0 2023-03-03T07:55:52.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.928+0000 7fdecaffd700 1 -- 172.21.15.98:0/1259192214 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec401e070 con 0x7fded40d27c0 2023-03-03T07:55:52.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.928+0000 7fdecaffd700 1 -- 172.21.15.98:0/1259192214 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec4003a40 con 0x7fded40d27c0 2023-03-03T07:55:52.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.928+0000 7fdedc5ff700 1 -- 172.21.15.98:0/1259192214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded40d1fc0 con 0x7fded40d27c0 2023-03-03T07:55:52.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.928+0000 7fdedc5ff700 1 -- 172.21.15.98:0/1259192214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded40d1ce0 con 0x7fded40d27c0 2023-03-03T07:55:52.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.929+0000 7fdedc5ff700 1 -- 172.21.15.98:0/1259192214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded40d22a0 con 0x7fded40d27c0 2023-03-03T07:55:52.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.930+0000 7fdecaffd700 1 -- 172.21.15.98:0/1259192214 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdec400f050 con 0x7fded40d27c0 2023-03-03T07:55:52.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.932+0000 7fdecaffd700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdeb80682f0 0x7fdeb806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:52.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.932+0000 7fdecaffd700 1 -- 172.21.15.98:0/1259192214 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdec4020070 con 0x7fded40d27c0 2023-03-03T07:55:52.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.932+0000 7fdeda39b700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdeb80682f0 0x7fdeb806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:52.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.933+0000 7fdeda39b700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdeb80682f0 0x7fdeb806a7a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fded0009a00 tx=0x7fded0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:52.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:52.937+0000 7fdecaffd700 1 -- 172.21.15.98:0/1259192214 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdec4053a40 con 0x7fded40d27c0 2023-03-03T07:55:53.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.180+0000 7fdedc5ff700 1 -- 172.21.15.98:0/1259192214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fded40729b0 con 0x7fded40d27c0 2023-03-03T07:55:53.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.184+0000 7fdecaffd700 1 -- 172.21.15.98:0/1259192214 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+166378 (secure 0 0 0) 0x7fdec4057060 con 0x7fded40d27c0 2023-03-03T07:55:53.187 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:53.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.187+0000 7fdedc5ff700 1 -- 172.21.15.98:0/1259192214 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdeb80682f0 msgr2=0x7fdeb806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:53.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.187+0000 7fdedc5ff700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdeb80682f0 0x7fdeb806a7a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fded0009a00 tx=0x7fded0008040 comp rx=0 tx=0).stop 2023-03-03T07:55:53.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.187+0000 7fdedc5ff700 1 -- 172.21.15.98:0/1259192214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded40d27c0 msgr2=0x7fded4152de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:53.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.187+0000 7fdedc5ff700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded40d27c0 0x7fded4152de0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdec4014040 tx=0x7fdec4004750 comp rx=0 tx=0).stop 2023-03-03T07:55:53.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.187+0000 7fdedc5ff700 1 -- 172.21.15.98:0/1259192214 shutdown_connections 2023-03-03T07:55:53.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.187+0000 7fdedc5ff700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdeb80682f0 0x7fdeb806a7a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.187+0000 7fdedc5ff700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fded415cc00 0x7fded4153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.187+0000 7fdedc5ff700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fded40b60d0 0x7fded414f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.187+0000 7fdedc5ff700 1 --2- 172.21.15.98:0/1259192214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded40d27c0 0x7fded4152de0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.187+0000 7fdedc5ff700 1 -- 172.21.15.98:0/1259192214 >> 172.21.15.98:0/1259192214 conn(0x7fded406aee0 msgr2=0x7fded406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:53.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.187+0000 7fdedc5ff700 1 -- 172.21.15.98:0/1259192214 shutdown_connections 2023-03-03T07:55:53.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.188+0000 7fdedc5ff700 1 -- 172.21.15.98:0/1259192214 wait complete. 2023-03-03T07:55:53.216 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"active_gid":4110,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":108777},{"type":"v1","addr":"172.21.15.98:6833","nonce":108777}]},"active_addr":"172.21.15.98:6833/108777","active_change":"2023-03-03T07:55:26.151884+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4107,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1647401399}]},{"addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":918927922}]},{"addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3559163180}]},{"addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1900692259}]}]} 2023-03-03T07:55:53.221 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-03-03T07:55:53.222 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-03-03T07:55:53.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:55:53.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.465+0000 7fae16fff700 1 Processor -- start 2023-03-03T07:55:53.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.465+0000 7fae16fff700 1 -- start start 2023-03-03T07:55:53.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.467+0000 7fae16fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fae1015cc00 0x7fae100cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:53.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.467+0000 7fae16fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae100d2c60 0x7fae100cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:53.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.467+0000 7fae16fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae100cde10 con 0x7fae100d27c0 2023-03-03T07:55:53.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.467+0000 7fae16fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae100ce040 con 0x7fae100d2c60 2023-03-03T07:55:53.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.467+0000 7fae16fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae100d30c0 con 0x7fae1015cc00 2023-03-03T07:55:53.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.467+0000 7fae1559c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae100d2c60 0x7fae100cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:53.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.467+0000 7fae1559c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae100d2c60 0x7fae100cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45346/0 (socket says 172.21.15.98:45346) 2023-03-03T07:55:53.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.467+0000 7fae1559c700 1 -- 172.21.15.98:0/126473760 learned_addr learned my addr 172.21.15.98:0/126473760 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:53.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.467+0000 7fae14d9b700 1 --2- 172.21.15.98:0/126473760 >> v2:172.21.15.100:3300/0 conn(0x7fae1015cc00 0x7fae100cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:53.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.467+0000 7fae1559c700 1 -- 172.21.15.98:0/126473760 >> v2:172.21.15.100:3300/0 conn(0x7fae1015cc00 msgr2=0x7fae100cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:53.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.467+0000 7fae1559c700 1 --2- 172.21.15.98:0/126473760 >> v2:172.21.15.100:3300/0 conn(0x7fae1015cc00 0x7fae100cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.467+0000 7fae1559c700 1 -- 172.21.15.98:0/126473760 >> v1:172.21.15.100:6789/0 conn(0x7fae100d27c0 legacy=0x7fae100b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:53.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.467+0000 7fae1559c700 1 -- 172.21.15.98:0/126473760 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae100d1a00 con 0x7fae100d2c60 2023-03-03T07:55:53.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.468+0000 7fae14d9b700 1 --2- 172.21.15.98:0/126473760 >> v2:172.21.15.100:3300/0 conn(0x7fae1015cc00 0x7fae100cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:53.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.468+0000 7fae1559c700 1 --2- 172.21.15.98:0/126473760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae100d2c60 0x7fae100cf4d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fae1008c200 tx=0x7fae0800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb05312e57c11c1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:53.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.468+0000 7fae0f7fe700 1 -- 172.21.15.98:0/126473760 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0800e070 con 0x7fae100d2c60 2023-03-03T07:55:53.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.468+0000 7fae0f7fe700 1 -- 172.21.15.98:0/126473760 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0800b9c0 con 0x7fae100d2c60 2023-03-03T07:55:53.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.468+0000 7fae0f7fe700 1 -- 172.21.15.98:0/126473760 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0800bd70 con 0x7fae100d2c60 2023-03-03T07:55:53.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.469+0000 7fae16fff700 1 -- 172.21.15.98:0/126473760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae100d2c60 msgr2=0x7fae100cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:53.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.469+0000 7fae16fff700 1 --2- 172.21.15.98:0/126473760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae100d2c60 0x7fae100cf4d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fae1008c200 tx=0x7fae0800b330 comp rx=0 tx=0).stop 2023-03-03T07:55:53.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.469+0000 7fae16fff700 1 -- 172.21.15.98:0/126473760 shutdown_connections 2023-03-03T07:55:53.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.469+0000 7fae16fff700 1 --2- 172.21.15.98:0/126473760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae100d2c60 0x7fae100cf4d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.469+0000 7fae16fff700 1 --2- 172.21.15.98:0/126473760 >> v2:172.21.15.100:3300/0 conn(0x7fae1015cc00 0x7fae100cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.469+0000 7fae16fff700 1 -- 172.21.15.98:0/126473760 >> 172.21.15.98:0/126473760 conn(0x7fae1006aee0 msgr2=0x7fae1006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:53.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.469+0000 7fae16fff700 1 -- 172.21.15.98:0/126473760 shutdown_connections 2023-03-03T07:55:53.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.469+0000 7fae16fff700 1 -- 172.21.15.98:0/126473760 wait complete. 2023-03-03T07:55:53.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.471+0000 7fae16fff700 1 Processor -- start 2023-03-03T07:55:53.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.471+0000 7fae16fff700 1 -- start start 2023-03-03T07:55:53.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.472+0000 7fae16fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae100d27c0 0x7fae1015fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:53.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.472+0000 7fae16fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae100d2c60 0x7fae100b8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:53.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.472+0000 7fae16fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae1015cc00 0x7fae1014e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:53.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.472+0000 7fae16fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1015e520 con 0x7fae100d2c60 2023-03-03T07:55:53.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.472+0000 7fae16fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae100b6f50 con 0x7fae100d27c0 2023-03-03T07:55:53.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.472+0000 7fae16fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae1015db80 con 0x7fae1015cc00 2023-03-03T07:55:53.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.472+0000 7fae0ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae100d2c60 0x7fae100b8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:53.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.472+0000 7fae0ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae100d2c60 0x7fae100b8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45350/0 (socket says 172.21.15.98:45350) 2023-03-03T07:55:53.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.472+0000 7fae0ffff700 1 -- 172.21.15.98:0/3994083589 learned_addr learned my addr 172.21.15.98:0/3994083589 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:53.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.473+0000 7fae14d9b700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae100d27c0 0x7fae1015fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:53.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.473+0000 7fae1559c700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae1015cc00 0x7fae1014e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:53.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.473+0000 7fae0ffff700 1 -- 172.21.15.98:0/3994083589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae1015cc00 msgr2=0x7fae1014e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:53.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.473+0000 7fae0ffff700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae1015cc00 0x7fae1014e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.473+0000 7fae0ffff700 1 -- 172.21.15.98:0/3994083589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae100d27c0 msgr2=0x7fae1015fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:53.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.473+0000 7fae0ffff700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae100d27c0 0x7fae1015fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.473+0000 7fae0ffff700 1 -- 172.21.15.98:0/3994083589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae08009b30 con 0x7fae100d2c60 2023-03-03T07:55:53.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.473+0000 7fae1559c700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae1015cc00 0x7fae1014e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:53.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.473+0000 7fae14d9b700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae100d27c0 0x7fae1015fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:53.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.473+0000 7fae0ffff700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae100d2c60 0x7fae100b8920 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fadfc00c4e0 tx=0x7fadfc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:53.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.473+0000 7fae0d7fa700 1 -- 172.21.15.98:0/3994083589 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc0052d0 con 0x7fae100d2c60 2023-03-03T07:55:53.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.473+0000 7fae0d7fa700 1 -- 172.21.15.98:0/3994083589 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadfc005430 con 0x7fae100d2c60 2023-03-03T07:55:53.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.473+0000 7fae0d7fa700 1 -- 172.21.15.98:0/3994083589 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc0038c0 con 0x7fae100d2c60 2023-03-03T07:55:53.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.473+0000 7fae16fff700 1 -- 172.21.15.98:0/3994083589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae100d1fc0 con 0x7fae100d2c60 2023-03-03T07:55:53.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.474+0000 7fae16fff700 1 -- 172.21.15.98:0/3994083589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae100d1ce0 con 0x7fae100d2c60 2023-03-03T07:55:53.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.475+0000 7fae16fff700 1 -- 172.21.15.98:0/3994083589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae100d22a0 con 0x7fae100d2c60 2023-03-03T07:55:53.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.476+0000 7fae0d7fa700 1 -- 172.21.15.98:0/3994083589 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fadfc019030 con 0x7fae100d2c60 2023-03-03T07:55:53.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.477+0000 7fae0d7fa700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fadf40682f0 0x7fadf406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:53.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.477+0000 7fae0d7fa700 1 -- 172.21.15.98:0/3994083589 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fadfc0850f0 con 0x7fae100d2c60 2023-03-03T07:55:53.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.477+0000 7fae14d9b700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fadf40682f0 0x7fadf406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:53.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.478+0000 7fae14d9b700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fadf40682f0 0x7fadf406a7a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fae0400c960 tx=0x7fae04008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:53.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.481+0000 7fae0d7fa700 1 -- 172.21.15.98:0/3994083589 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fadfc051860 con 0x7fae100d2c60 2023-03-03T07:55:53.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.676+0000 7fae16fff700 1 -- 172.21.15.98:0/3994083589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae100729b0 con 0x7fae100d2c60 2023-03-03T07:55:53.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.678+0000 7fae0d7fa700 1 -- 172.21.15.98:0/3994083589 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17907 (secure 0 0 0) 0x7fadfc054e80 con 0x7fae100d2c60 2023-03-03T07:55:53.679 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:53.680 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:55:48.099844+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:55:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.680+0000 7fae16fff700 1 -- 172.21.15.98:0/3994083589 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fadf40682f0 msgr2=0x7fadf406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.680+0000 7fae16fff700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fadf40682f0 0x7fadf406a7a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fae0400c960 tx=0x7fae04008040 comp rx=0 tx=0).stop 2023-03-03T07:55:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.680+0000 7fae16fff700 1 -- 172.21.15.98:0/3994083589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae100d2c60 msgr2=0x7fae100b8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.680+0000 7fae16fff700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae100d2c60 0x7fae100b8920 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fadfc00c4e0 tx=0x7fadfc00c8a0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.680+0000 7fae16fff700 1 -- 172.21.15.98:0/3994083589 shutdown_connections 2023-03-03T07:55:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.680+0000 7fae16fff700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fadf40682f0 0x7fadf406a7a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.680+0000 7fae16fff700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae1015cc00 0x7fae1014e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.680+0000 7fae16fff700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae100d27c0 0x7fae1015fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.680+0000 7fae16fff700 1 --2- 172.21.15.98:0/3994083589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae100d2c60 0x7fae100b8920 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.680+0000 7fae16fff700 1 -- 172.21.15.98:0/3994083589 >> 172.21.15.98:0/3994083589 conn(0x7fae1006aee0 msgr2=0x7fae1015b920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.680+0000 7fae16fff700 1 -- 172.21.15.98:0/3994083589 shutdown_connections 2023-03-03T07:55:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.681+0000 7fae16fff700 1 -- 172.21.15.98:0/3994083589 wait complete. 2023-03-03T07:55:53.710 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-03-03T07:55:53.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:55:53.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.955+0000 7f0e2f9ff700 1 Processor -- start 2023-03-03T07:55:53.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.955+0000 7f0e2f9ff700 1 -- start start 2023-03-03T07:55:53.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.956+0000 7f0e2f9ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0e2815ca40 0x7f0e280cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:53.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.956+0000 7f0e2f9ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0e280cdd10 0x7f0e280cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:53.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.956+0000 7f0e2f9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e280d00a0 0x7f0e280b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:53.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.956+0000 7f0e2f9ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e280d0540 con 0x7f0e2815ca40 2023-03-03T07:55:53.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.956+0000 7f0e2f9ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e280d1890 con 0x7f0e280cdd10 2023-03-03T07:55:53.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.956+0000 7f0e2f9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e280d1b70 con 0x7f0e280d00a0 2023-03-03T07:55:53.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.957+0000 7f0e2d79b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0e2815ca40 0x7f0e280cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:53.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.957+0000 7f0e2d79b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0e2815ca40 0x7f0e280cd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45364/0 (socket says 172.21.15.98:45364) 2023-03-03T07:55:53.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.957+0000 7f0e2d79b700 1 -- 172.21.15.98:0/4057435992 learned_addr learned my addr 172.21.15.98:0/4057435992 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:53.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.957+0000 7f0e2cf9a700 1 --2- 172.21.15.98:0/4057435992 >> v2:172.21.15.100:3300/0 conn(0x7f0e280cdd10 0x7f0e280cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:53.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.957+0000 7f0e2df9c700 1 --2- 172.21.15.98:0/4057435992 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e280d00a0 0x7f0e280b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:53.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.957+0000 7f0e2d79b700 1 -- 172.21.15.98:0/4057435992 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e280d00a0 msgr2=0x7f0e280b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:53.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.957+0000 7f0e2d79b700 1 --2- 172.21.15.98:0/4057435992 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e280d00a0 0x7f0e280b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.957+0000 7f0e2d79b700 1 -- 172.21.15.98:0/4057435992 >> v2:172.21.15.100:3300/0 conn(0x7f0e280cdd10 msgr2=0x7f0e280cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:53.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.957+0000 7f0e2d79b700 1 --2- 172.21.15.98:0/4057435992 >> v2:172.21.15.100:3300/0 conn(0x7f0e280cdd10 0x7f0e280cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.957+0000 7f0e2d79b700 1 -- 172.21.15.98:0/4057435992 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e280d1e50 con 0x7f0e2815ca40 2023-03-03T07:55:53.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.958+0000 7f0e2d79b700 1 --2- 172.21.15.98:0/4057435992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e2815ca40 0x7f0e280cd7d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0e280d0970 tx=0x7f0e24009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e50bfbc95758462 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:53.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.958+0000 7f0e1ffff700 1 -- 172.21.15.98:0/4057435992 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e24011830 con 0x7f0e2815ca40 2023-03-03T07:55:53.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.958+0000 7f0e1ffff700 1 -- 172.21.15.98:0/4057435992 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e24011990 con 0x7f0e2815ca40 2023-03-03T07:55:53.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.958+0000 7f0e1ffff700 1 -- 172.21.15.98:0/4057435992 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e24011d20 con 0x7f0e2815ca40 2023-03-03T07:55:53.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.958+0000 7f0e2f9ff700 1 -- 172.21.15.98:0/4057435992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e2815ca40 msgr2=0x7f0e280cd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:53.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.958+0000 7f0e2f9ff700 1 --2- 172.21.15.98:0/4057435992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e2815ca40 0x7f0e280cd7d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0e280d0970 tx=0x7f0e24009bd0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.959+0000 7f0e2f9ff700 1 -- 172.21.15.98:0/4057435992 shutdown_connections 2023-03-03T07:55:53.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.959+0000 7f0e2f9ff700 1 --2- 172.21.15.98:0/4057435992 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e280d00a0 0x7f0e280b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.959+0000 7f0e2f9ff700 1 --2- 172.21.15.98:0/4057435992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e2815ca40 0x7f0e280cd7d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.959+0000 7f0e2f9ff700 1 --2- 172.21.15.98:0/4057435992 >> v2:172.21.15.100:3300/0 conn(0x7f0e280cdd10 0x7f0e280cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.959+0000 7f0e2f9ff700 1 -- 172.21.15.98:0/4057435992 >> 172.21.15.98:0/4057435992 conn(0x7f0e28073e70 msgr2=0x7f0e2806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:53.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.959+0000 7f0e2f9ff700 1 -- 172.21.15.98:0/4057435992 shutdown_connections 2023-03-03T07:55:53.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.959+0000 7f0e2f9ff700 1 -- 172.21.15.98:0/4057435992 wait complete. 2023-03-03T07:55:53.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.961+0000 7f0e2f9ff700 1 Processor -- start 2023-03-03T07:55:53.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.961+0000 7f0e2f9ff700 1 -- start start 2023-03-03T07:55:53.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.962+0000 7f0e2f9ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e280d00a0 0x7f0e280c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:53.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.962+0000 7f0e2f9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e2815ca40 0x7f0e2814ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:53.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.962+0000 7f0e2f9ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e2814d210 0x7f0e280ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:53.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.962+0000 7f0e2f9ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e280c7050 con 0x7f0e2814d210 2023-03-03T07:55:53.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.962+0000 7f0e2f9ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e280a9f60 con 0x7f0e280d00a0 2023-03-03T07:55:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.962+0000 7f0e2f9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2814d690 con 0x7f0e2815ca40 2023-03-03T07:55:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.962+0000 7f0e2df9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e2814d210 0x7f0e280ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.962+0000 7f0e2df9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e2814d210 0x7f0e280ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45366/0 (socket says 172.21.15.98:45366) 2023-03-03T07:55:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.962+0000 7f0e2df9c700 1 -- 172.21.15.98:0/4291666565 learned_addr learned my addr 172.21.15.98:0/4291666565 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.963+0000 7f0e2d79b700 1 --2- 172.21.15.98:0/4291666565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e280d00a0 0x7f0e280c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.963+0000 7f0e2cf9a700 1 --2- 172.21.15.98:0/4291666565 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e2815ca40 0x7f0e2814ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.963+0000 7f0e2df9c700 1 -- 172.21.15.98:0/4291666565 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e2815ca40 msgr2=0x7f0e2814ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.963+0000 7f0e2df9c700 1 --2- 172.21.15.98:0/4291666565 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e2815ca40 0x7f0e2814ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.963+0000 7f0e2df9c700 1 -- 172.21.15.98:0/4291666565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e280d00a0 msgr2=0x7f0e280c95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.963+0000 7f0e2df9c700 1 --2- 172.21.15.98:0/4291666565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e280d00a0 0x7f0e280c95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.963+0000 7f0e2df9c700 1 -- 172.21.15.98:0/4291666565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e24009550 con 0x7f0e2814d210 2023-03-03T07:55:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.963+0000 7f0e2d79b700 1 --2- 172.21.15.98:0/4291666565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e280d00a0 0x7f0e280c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.963+0000 7f0e2df9c700 1 --2- 172.21.15.98:0/4291666565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e2814d210 0x7f0e280ca4f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0e20009dd0 tx=0x7f0e2000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:53.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.963+0000 7f0e1dffb700 1 -- 172.21.15.98:0/4291666565 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2000cd40 con 0x7f0e2814d210 2023-03-03T07:55:53.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.963+0000 7f0e1dffb700 1 -- 172.21.15.98:0/4291666565 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2000f040 con 0x7f0e2814d210 2023-03-03T07:55:53.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.963+0000 7f0e2f9ff700 1 -- 172.21.15.98:0/4291666565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2806b700 con 0x7f0e2814d210 2023-03-03T07:55:53.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.964+0000 7f0e1dffb700 1 -- 172.21.15.98:0/4291666565 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e200146c0 con 0x7f0e2814d210 2023-03-03T07:55:53.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.964+0000 7f0e2f9ff700 1 -- 172.21.15.98:0/4291666565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e280d2130 con 0x7f0e2814d210 2023-03-03T07:55:53.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.965+0000 7f0e2f9ff700 1 -- 172.21.15.98:0/4291666565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2806b890 con 0x7f0e2814d210 2023-03-03T07:55:53.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.966+0000 7f0e1dffb700 1 -- 172.21.15.98:0/4291666565 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0e20014820 con 0x7f0e2814d210 2023-03-03T07:55:53.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.967+0000 7f0e1dffb700 1 --2- 172.21.15.98:0/4291666565 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0e0c0682f0 0x7f0e0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:53.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.967+0000 7f0e1dffb700 1 -- 172.21.15.98:0/4291666565 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0e20086a30 con 0x7f0e2814d210 2023-03-03T07:55:53.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.968+0000 7f0e2d79b700 1 --2- 172.21.15.98:0/4291666565 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0e0c0682f0 0x7f0e0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:53.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.968+0000 7f0e2d79b700 1 --2- 172.21.15.98:0/4291666565 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0e0c0682f0 0x7f0e0c06a7a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009c10 tx=0x7f0e24008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:53.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:53.971+0000 7f0e1dffb700 1 -- 172.21.15.98:0/4291666565 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0e200531a0 con 0x7f0e2814d210 2023-03-03T07:55:54.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.166+0000 7f0e2f9ff700 1 -- 172.21.15.98:0/4291666565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e2804e6e0 con 0x7f0e2814d210 2023-03-03T07:55:54.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.168+0000 7f0e1dffb700 1 -- 172.21.15.98:0/4291666565 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17907 (secure 0 0 0) 0x7f0e2000cea0 con 0x7f0e2814d210 2023-03-03T07:55:54.169 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:54.169 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:55:48.099844+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:55:54.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.170+0000 7f0e2f9ff700 1 -- 172.21.15.98:0/4291666565 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0e0c0682f0 msgr2=0x7f0e0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.170+0000 7f0e2f9ff700 1 --2- 172.21.15.98:0/4291666565 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0e0c0682f0 0x7f0e0c06a7a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009c10 tx=0x7f0e24008030 comp rx=0 tx=0).stop 2023-03-03T07:55:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.170+0000 7f0e2f9ff700 1 -- 172.21.15.98:0/4291666565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e2814d210 msgr2=0x7f0e280ca4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.170+0000 7f0e2f9ff700 1 --2- 172.21.15.98:0/4291666565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e2814d210 0x7f0e280ca4f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0e20009dd0 tx=0x7f0e2000c3a0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.170+0000 7f0e2f9ff700 1 -- 172.21.15.98:0/4291666565 shutdown_connections 2023-03-03T07:55:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.170+0000 7f0e2f9ff700 1 --2- 172.21.15.98:0/4291666565 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0e0c0682f0 0x7f0e0c06a7a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.170+0000 7f0e2f9ff700 1 --2- 172.21.15.98:0/4291666565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e2814d210 0x7f0e280ca4f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.170+0000 7f0e2f9ff700 1 --2- 172.21.15.98:0/4291666565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e280d00a0 0x7f0e280c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.170+0000 7f0e2f9ff700 1 --2- 172.21.15.98:0/4291666565 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e2815ca40 0x7f0e2814ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.170+0000 7f0e2f9ff700 1 -- 172.21.15.98:0/4291666565 >> 172.21.15.98:0/4291666565 conn(0x7f0e28073e70 msgr2=0x7f0e2815b370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.170+0000 7f0e2f9ff700 1 -- 172.21.15.98:0/4291666565 shutdown_connections 2023-03-03T07:55:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.171+0000 7f0e2f9ff700 1 -- 172.21.15.98:0/4291666565 wait complete. 2023-03-03T07:55:54.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-03-03T07:55:54.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-03-03T07:55:54.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-03-03T07:55:54.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-03-03T07:55:54.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-03-03T07:55:54.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-03-03T07:55:54.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-03-03T07:55:54.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-03-03T07:55:54.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.648+0000 7fef513ff700 1 Processor -- start 2023-03-03T07:55:54.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.648+0000 7fef513ff700 1 -- start start 2023-03-03T07:55:54.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.651+0000 7fef513ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fef4c160ff0 0x7fef4c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.651+0000 7fef513ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef4c1613c0 con 0x7fef4c0ce1c0 2023-03-03T07:55:54.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.651+0000 7fef513ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef4c0ce980 con 0x7fef4c0b0f10 2023-03-03T07:55:54.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.651+0000 7fef513ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef4c0cacf0 con 0x7fef4c160ff0 2023-03-03T07:55:54.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.652+0000 7fef4b7fe700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fef4c0b0f10 0x7fef4c06b890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:40560/0 (socket says 172.21.15.98:40560) 2023-03-03T07:55:54.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.652+0000 7fef4b7fe700 1 -- 172.21.15.98:0/2543160582 learned_addr learned my addr 172.21.15.98:0/2543160582 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:54.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.654+0000 7fef49ffb700 1 -- 172.21.15.98:0/2543160582 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750967925 0 0) 0x7fef4c0ce980 con 0x7fef4c0b0f10 2023-03-03T07:55:54.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.654+0000 7fef49ffb700 1 -- 172.21.15.98:0/2543160582 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef340035b0 con 0x7fef4c0b0f10 2023-03-03T07:55:54.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.655+0000 7fef4affd700 1 --2- 172.21.15.98:0/2543160582 >> v2:172.21.15.98:3300/0 conn(0x7fef4c160ff0 0x7fef4c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.655+0000 7fef49ffb700 1 -- 172.21.15.98:0/2543160582 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2498945744 0 0) 0x7fef340035b0 con 0x7fef4c0b0f10 2023-03-03T07:55:54.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.655+0000 7fef49ffb700 1 -- 172.21.15.98:0/2543160582 >> v2:172.21.15.98:3300/0 conn(0x7fef4c160ff0 msgr2=0x7fef4c06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:54.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.655+0000 7fef49ffb700 1 --2- 172.21.15.98:0/2543160582 >> v2:172.21.15.98:3300/0 conn(0x7fef4c160ff0 0x7fef4c06d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.655+0000 7fef49ffb700 1 -- 172.21.15.98:0/2543160582 >> v1:172.21.15.98:6789/0 conn(0x7fef4c0ce1c0 legacy=0x7fef4c0b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:54.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.655+0000 7fef49ffb700 1 -- 172.21.15.98:0/2543160582 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c0cafd0 con 0x7fef4c0b0f10 2023-03-03T07:55:54.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.655+0000 7fef49ffb700 1 -- 172.21.15.98:0/2543160582 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7fef40001670 con 0x7fef4c0b0f10 2023-03-03T07:55:54.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.655+0000 7fef49ffb700 1 -- 172.21.15.98:0/2543160582 >> v1:172.21.15.100:6789/0 conn(0x7fef4c0b0f10 legacy=0x7fef4c06b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.655+0000 7fef49ffb700 1 --2- 172.21.15.98:0/2543160582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fef34003a00 0x7fef34003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.655+0000 7fef49ffb700 1 --2- 172.21.15.98:0/2543160582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef34004850 0x7fef34004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.655+0000 7fef49ffb700 1 --2- 172.21.15.98:0/2543160582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef34005200 0x7fef3400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.655+0000 7fef49ffb700 1 -- 172.21.15.98:0/2543160582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c0cacf0 con 0x7fef34004850 2023-03-03T07:55:54.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.655+0000 7fef49ffb700 1 -- 172.21.15.98:0/2543160582 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3400c110 con 0x7fef34005200 2023-03-03T07:55:54.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.655+0000 7fef49ffb700 1 -- 172.21.15.98:0/2543160582 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3401d2e0 con 0x7fef34003a00 2023-03-03T07:55:54.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.656+0000 7fef4b7fe700 1 --2- 172.21.15.98:0/2543160582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef34005200 0x7fef3400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.657+0000 7fef4b7fe700 1 -- 172.21.15.98:0/2543160582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fef34003a00 msgr2=0x7fef34003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:54.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.657+0000 7fef4b7fe700 1 --2- 172.21.15.98:0/2543160582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fef34003a00 0x7fef34003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.657+0000 7fef4b7fe700 1 -- 172.21.15.98:0/2543160582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef34004850 msgr2=0x7fef34004cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T07:55:54.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.657+0000 7fef4b7fe700 1 --2- 172.21.15.98:0/2543160582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef34004850 0x7fef34004cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.657+0000 7fef4b7fe700 1 -- 172.21.15.98:0/2543160582 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c0cafd0 con 0x7fef34005200 2023-03-03T07:55:54.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.657+0000 7fef4b7fe700 1 -- 172.21.15.98:0/2543160582 reap_dead start 2023-03-03T07:55:54.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.658+0000 7fef4b7fe700 1 --2- 172.21.15.98:0/2543160582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef34005200 0x7fef3400ba10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fef4000c370 tx=0x7fef34005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7545677d6a06e350 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.657+0000 7fb4d0dff700 1 Processor -- start 2023-03-03T07:55:54.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.657+0000 7fb4d0dff700 1 -- start start 2023-03-03T07:55:54.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.658+0000 7fef49ffb700 1 -- 172.21.15.98:0/2543160582 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef40005960 con 0x7fef34005200 2023-03-03T07:55:54.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.658+0000 7fef49ffb700 1 -- 172.21.15.98:0/2543160582 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef40005ae0 con 0x7fef34005200 2023-03-03T07:55:54.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.658+0000 7fef49ffb700 1 -- 172.21.15.98:0/2543160582 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef40005e90 con 0x7fef34005200 2023-03-03T07:55:54.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.659+0000 7fef513ff700 1 -- 172.21.15.98:0/2543160582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef34005200 msgr2=0x7fef3400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.659+0000 7fef513ff700 1 --2- 172.21.15.98:0/2543160582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef34005200 0x7fef3400ba10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fef4000c370 tx=0x7fef34005880 comp rx=0 tx=0).stop 2023-03-03T07:55:54.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.660+0000 7fb4d0dff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb4cc06fe40 0x7fb4cc06d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.661+0000 7f26b83ff700 1 Processor -- start 2023-03-03T07:55:54.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.662+0000 7f26b83ff700 1 -- start start 2023-03-03T07:55:54.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.660+0000 7fb4d0dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0ca910 0x7fb4cc1698d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.660+0000 7fb4d0dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4cc16d7b0 con 0x7fb4cc16dc40 2023-03-03T07:55:54.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.660+0000 7fb4d0dff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4cc169e10 con 0x7fb4cc06fe40 2023-03-03T07:55:54.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.660+0000 7fb4d0dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc0cae80 con 0x7fb4cc0ca910 2023-03-03T07:55:54.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.661+0000 7fb4c9d9b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0ca910 0x7fb4cc1698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.661+0000 7fb4c9d9b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0ca910 0x7fb4cc1698d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:42324/0 (socket says 172.21.15.98:42324) 2023-03-03T07:55:54.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.661+0000 7fb4c9d9b700 1 -- 172.21.15.98:0/1530472931 learned_addr learned my addr 172.21.15.98:0/1530472931 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:54.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.662+0000 7fb4c9d9b700 1 -- 172.21.15.98:0/1530472931 >> v2:172.21.15.98:3300/0 conn(0x7fb4cc06fe40 msgr2=0x7fb4cc06d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:54.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.662+0000 7fb4c9d9b700 1 --2- 172.21.15.98:0/1530472931 >> v2:172.21.15.98:3300/0 conn(0x7fb4cc06fe40 0x7fb4cc06d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.662+0000 7fb4c9d9b700 1 -- 172.21.15.98:0/1530472931 >> v1:172.21.15.100:6789/0 conn(0x7fb4cc16dc40 legacy=0x7fb4cc0ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.662+0000 7fb4c9d9b700 1 -- 172.21.15.98:0/1530472931 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc0cb160 con 0x7fb4cc0ca910 2023-03-03T07:55:54.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.663+0000 7fb4c9d9b700 1 --2- 172.21.15.98:0/1530472931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0ca910 0x7fb4cc1698d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00098f0 tx=0x7fb4c0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa26c83887ad858e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.663+0000 7fef513ff700 1 -- 172.21.15.98:0/2543160582 shutdown_connections 2023-03-03T07:55:54.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.663+0000 7fef513ff700 1 --2- 172.21.15.98:0/2543160582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef34005200 0x7fef3400ba10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.663+0000 7fef513ff700 1 -- 172.21.15.98:0/2543160582 >> 172.21.15.98:0/2543160582 conn(0x7fef4c0cb800 msgr2=0x7fef4c0cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.664+0000 7fb4c959a700 1 -- 172.21.15.98:0/1530472931 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0004230 con 0x7fb4cc0ca910 2023-03-03T07:55:54.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.664+0000 7fb4c959a700 1 -- 172.21.15.98:0/1530472931 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c0004390 con 0x7fb4cc0ca910 2023-03-03T07:55:54.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.664+0000 7fb4c959a700 1 -- 172.21.15.98:0/1530472931 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c00036a0 con 0x7fb4cc0ca910 2023-03-03T07:55:54.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.665+0000 7fb4d0dff700 1 -- 172.21.15.98:0/1530472931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0ca910 msgr2=0x7fb4cc1698d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.665+0000 7fb4d0dff700 1 --2- 172.21.15.98:0/1530472931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0ca910 0x7fb4cc1698d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00098f0 tx=0x7fb4c0009c00 comp rx=0 tx=0).stop 2023-03-03T07:55:54.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.665+0000 7fb4d0dff700 1 -- 172.21.15.98:0/1530472931 shutdown_connections 2023-03-03T07:55:54.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.665+0000 7fb4d0dff700 1 --2- 172.21.15.98:0/1530472931 >> v2:172.21.15.98:3300/0 conn(0x7fb4cc06fe40 0x7fb4cc06d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.665+0000 7fb4d0dff700 1 --2- 172.21.15.98:0/1530472931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0ca910 0x7fb4cc1698d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.665+0000 7fb4d0dff700 1 -- 172.21.15.98:0/1530472931 >> 172.21.15.98:0/1530472931 conn(0x7fb4cc0cb800 msgr2=0x7fb4cc0cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.666+0000 7f26b83ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f26b0158c40 0x7f26b00b0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.666+0000 7f26b83ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f26b00b0d40 0x7f26b0070770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.666+0000 7f26b83ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b015d540 con 0x7f26b0158870 2023-03-03T07:55:54.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.666+0000 7f26b83ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b00702e0 con 0x7f26b0158c40 2023-03-03T07:55:54.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.666+0000 7f26b83ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b00b1140 con 0x7f26b00b0d40 2023-03-03T07:55:54.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.666+0000 7f26b699c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f26b00b0d40 0x7f26b0070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.666+0000 7f26b699c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f26b00b0d40 0x7f26b0070770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:46340/0 (socket says 172.21.15.98:46340) 2023-03-03T07:55:54.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.666+0000 7f26b699c700 1 -- 172.21.15.98:0/74438699 learned_addr learned my addr 172.21.15.98:0/74438699 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:54.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.666+0000 7f26b599a700 1 --2- 172.21.15.98:0/74438699 >> v2:172.21.15.98:3300/0 conn(0x7f26b0158c40 0x7f26b00b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.667+0000 7f26b699c700 1 -- 172.21.15.98:0/74438699 >> v2:172.21.15.98:3300/0 conn(0x7f26b0158c40 msgr2=0x7f26b00b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:54.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.667+0000 7f26b699c700 1 --2- 172.21.15.98:0/74438699 >> v2:172.21.15.98:3300/0 conn(0x7f26b0158c40 0x7f26b00b0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.667+0000 7f26b699c700 1 -- 172.21.15.98:0/74438699 >> v1:172.21.15.100:6789/0 conn(0x7f26b0158870 legacy=0x7f26b015c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.667+0000 7f26b699c700 1 -- 172.21.15.98:0/74438699 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b015ccd0 con 0x7f26b00b0d40 2023-03-03T07:55:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.668+0000 7f26b699c700 1 --2- 172.21.15.98:0/74438699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26b00b0d40 0x7f26b0070770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f26b00719e0 tx=0x7f26a0009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab0819809566dd45 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.668+0000 7f26b5199700 1 -- 172.21.15.98:0/74438699 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a0011830 con 0x7f26b00b0d40 2023-03-03T07:55:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.669+0000 7f26b5199700 1 -- 172.21.15.98:0/74438699 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a0011990 con 0x7f26b00b0d40 2023-03-03T07:55:54.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.669+0000 7f26b5199700 1 -- 172.21.15.98:0/74438699 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a0011d20 con 0x7f26b00b0d40 2023-03-03T07:55:54.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.670+0000 7f26b83ff700 1 -- 172.21.15.98:0/74438699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26b00b0d40 msgr2=0x7f26b0070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.670+0000 7f26b83ff700 1 --2- 172.21.15.98:0/74438699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26b00b0d40 0x7f26b0070770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f26b00719e0 tx=0x7f26a0009bd0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.670+0000 7f26b83ff700 1 -- 172.21.15.98:0/74438699 shutdown_connections 2023-03-03T07:55:54.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.670+0000 7f26b83ff700 1 --2- 172.21.15.98:0/74438699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26b00b0d40 0x7f26b0070770 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.670+0000 7f26b83ff700 1 --2- 172.21.15.98:0/74438699 >> v2:172.21.15.98:3300/0 conn(0x7f26b0158c40 0x7f26b00b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.670+0000 7f26b83ff700 1 -- 172.21.15.98:0/74438699 >> 172.21.15.98:0/74438699 conn(0x7f26b00cb800 msgr2=0x7f26b00cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.671+0000 7f26b83ff700 1 -- 172.21.15.98:0/74438699 shutdown_connections 2023-03-03T07:55:54.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.671+0000 7f26b83ff700 1 -- 172.21.15.98:0/74438699 wait complete. 2023-03-03T07:55:54.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.665+0000 7fef513ff700 1 -- 172.21.15.98:0/2543160582 shutdown_connections 2023-03-03T07:55:54.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.666+0000 7fef513ff700 1 -- 172.21.15.98:0/2543160582 wait complete. 2023-03-03T07:55:54.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.669+0000 7fef513ff700 1 Processor -- start 2023-03-03T07:55:54.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.669+0000 7fef513ff700 1 -- start start 2023-03-03T07:55:54.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.671+0000 7fef513ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fef34005200 0x7fef4c14e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.671+0000 7fef513ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef4c165290 0x7fef4c162960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.671+0000 7fef513ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef4c162ea0 0x7fef4c0af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.671+0000 7fef513ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c14dcb0 con 0x7fef4c162ea0 2023-03-03T07:55:54.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.671+0000 7fef513ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c14d630 con 0x7fef4c165290 2023-03-03T07:55:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.671+0000 7fef513ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c1633e0 con 0x7fef34005200 2023-03-03T07:55:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.671+0000 7fef4a7fc700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef4c165290 0x7fef4c162960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.672+0000 7fef4a7fc700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef4c165290 0x7fef4c162960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:46348/0 (socket says 172.21.15.98:46348) 2023-03-03T07:55:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.672+0000 7fef4a7fc700 1 -- 172.21.15.98:0/1431471722 learned_addr learned my addr 172.21.15.98:0/1431471722 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.673+0000 7fef4a7fc700 1 -- 172.21.15.98:0/1431471722 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fef34005200 msgr2=0x7fef4c14e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.673+0000 7fef4a7fc700 1 --2- 172.21.15.98:0/1431471722 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fef34005200 0x7fef4c14e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.673+0000 7fef4a7fc700 1 -- 172.21.15.98:0/1431471722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef4c162ea0 msgr2=0x7fef4c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.673+0000 7fef4a7fc700 1 --2- 172.21.15.98:0/1431471722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef4c162ea0 0x7fef4c0af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.673+0000 7fef4a7fc700 1 -- 172.21.15.98:0/1431471722 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c0ce610 con 0x7fef4c165290 2023-03-03T07:55:54.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.673+0000 7fef4a7fc700 1 --2- 172.21.15.98:0/1431471722 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef4c165290 0x7fef4c162960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fef4400c700 tx=0x7fef4400cac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.674+0000 7fef337fe700 1 -- 172.21.15.98:0/1431471722 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef44003e10 con 0x7fef4c165290 2023-03-03T07:55:54.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.674+0000 7fef337fe700 1 -- 172.21.15.98:0/1431471722 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef440050d0 con 0x7fef4c165290 2023-03-03T07:55:54.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.674+0000 7fef337fe700 1 -- 172.21.15.98:0/1431471722 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef44005480 con 0x7fef4c165290 2023-03-03T07:55:54.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.675+0000 7fef513ff700 1 -- 172.21.15.98:0/1431471722 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef4c0cb2b0 con 0x7fef4c165290 2023-03-03T07:55:54.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.675+0000 7fef513ff700 1 -- 172.21.15.98:0/1431471722 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef4c0ca910 con 0x7fef4c165290 2023-03-03T07:55:54.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.674+0000 7f26b83ff700 1 Processor -- start 2023-03-03T07:55:54.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.674+0000 7f26b83ff700 1 -- start start 2023-03-03T07:55:54.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.672+0000 7fb4d0dff700 1 -- 172.21.15.98:0/1530472931 shutdown_connections 2023-03-03T07:55:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.672+0000 7fb4d0dff700 1 -- 172.21.15.98:0/1530472931 wait complete. 2023-03-03T07:55:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.675+0000 7fb4d0dff700 1 Processor -- start 2023-03-03T07:55:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.676+0000 7fb4d0dff700 1 -- start start 2023-03-03T07:55:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.677+0000 7fb4d0dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4cc06fe40 0x7fb4cc0c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.677+0000 7fb4d0dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0ca910 0x7fb4cc0c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.677+0000 7fb4d0dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4cc16dc40 0x7fb4cc088a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.677+0000 7fb4d0dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc169f80 con 0x7fb4cc06fe40 2023-03-03T07:55:54.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.677+0000 7fb4d0dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc16a540 con 0x7fb4cc16dc40 2023-03-03T07:55:54.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.677+0000 7fb4d0dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc08a170 con 0x7fb4cc0ca910 2023-03-03T07:55:54.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.678+0000 7fb4ca59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4cc06fe40 0x7fb4cc0c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.678+0000 7fb4ca59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4cc06fe40 0x7fb4cc0c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45418/0 (socket says 172.21.15.98:45418) 2023-03-03T07:55:54.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.678+0000 7fb4ca59c700 1 -- 172.21.15.98:0/4020553020 learned_addr learned my addr 172.21.15.98:0/4020553020 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:54.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.678+0000 7fb4cad9d700 1 --2- 172.21.15.98:0/4020553020 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4cc16dc40 0x7fb4cc088a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.680+0000 7fef513ff700 1 -- 172.21.15.98:0/1431471722 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fef4c04e6e0 con 0x7fef4c165290 2023-03-03T07:55:54.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.676+0000 7f26b83ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26b00b0d40 0x7f26b00b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.676+0000 7f26b83ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26b0158870 0x7f26b014f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.676+0000 7f26b83ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26b0158c40 0x7f26b015a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.676+0000 7f26b83ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b00b3af0 con 0x7f26b00b0d40 2023-03-03T07:55:54.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.676+0000 7f26b83ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b0146940 con 0x7f26b0158c40 2023-03-03T07:55:54.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.676+0000 7f26b83ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b00b2b80 con 0x7f26b0158870 2023-03-03T07:55:54.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.677+0000 7f26b619b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26b00b0d40 0x7f26b00b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.678+0000 7f26b619b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26b00b0d40 0x7f26b00b0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45404/0 (socket says 172.21.15.98:45404) 2023-03-03T07:55:54.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.678+0000 7f26b619b700 1 -- 172.21.15.98:0/1305760611 learned_addr learned my addr 172.21.15.98:0/1305760611 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:54.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.679+0000 7f26b619b700 1 -- 172.21.15.98:0/1305760611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26b0158870 msgr2=0x7f26b014f1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T07:55:54.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.679+0000 7f26b619b700 1 --2- 172.21.15.98:0/1305760611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26b0158870 0x7f26b014f1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.679+0000 7f26b619b700 1 -- 172.21.15.98:0/1305760611 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26b0158c40 msgr2=0x7f26b015a4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T07:55:54.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.679+0000 7f26b619b700 1 --2- 172.21.15.98:0/1305760611 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26b0158c40 0x7f26b015a4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.679+0000 7f26b619b700 1 -- 172.21.15.98:0/1305760611 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a0009550 con 0x7f26b00b0d40 2023-03-03T07:55:54.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.680+0000 7f26b619b700 1 --2- 172.21.15.98:0/1305760611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26b00b0d40 0x7f26b00b0750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f26ac00c450 tx=0x7f26ac00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.679+0000 7fb4cad9d700 1 -- 172.21.15.98:0/4020553020 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0ca910 msgr2=0x7fb4cc0c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.679+0000 7fb4cad9d700 1 --2- 172.21.15.98:0/4020553020 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0ca910 0x7fb4cc0c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.679+0000 7fb4cad9d700 1 -- 172.21.15.98:0/4020553020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4cc06fe40 msgr2=0x7fb4cc0c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.679+0000 7fb4cad9d700 1 --2- 172.21.15.98:0/4020553020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4cc06fe40 0x7fb4cc0c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.679+0000 7fb4cad9d700 1 -- 172.21.15.98:0/4020553020 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c0009550 con 0x7fb4cc16dc40 2023-03-03T07:55:54.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.679+0000 7fb4ca59c700 1 --2- 172.21.15.98:0/4020553020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4cc06fe40 0x7fb4cc0c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:54.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.680+0000 7fb4cad9d700 1 --2- 172.21.15.98:0/4020553020 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4cc16dc40 0x7fb4cc088a50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400c7f0 tx=0x7fb4c400cbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.681+0000 7f26a6ffd700 1 -- 172.21.15.98:0/1305760611 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac0050d0 con 0x7f26b00b0d40 2023-03-03T07:55:54.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.681+0000 7f26a6ffd700 1 -- 172.21.15.98:0/1305760611 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ac005230 con 0x7f26b00b0d40 2023-03-03T07:55:54.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.681+0000 7f26a6ffd700 1 -- 172.21.15.98:0/1305760611 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac003680 con 0x7f26b00b0d40 2023-03-03T07:55:54.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.682+0000 7f26b83ff700 1 -- 172.21.15.98:0/1305760611 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b015d260 con 0x7f26b00b0d40 2023-03-03T07:55:54.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.681+0000 7fb4baffd700 1 -- 172.21.15.98:0/4020553020 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4003e30 con 0x7fb4cc16dc40 2023-03-03T07:55:54.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.681+0000 7fb4baffd700 1 -- 172.21.15.98:0/4020553020 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c4005070 con 0x7fb4cc16dc40 2023-03-03T07:55:54.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.681+0000 7fb4baffd700 1 -- 172.21.15.98:0/4020553020 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4005400 con 0x7fb4cc16dc40 2023-03-03T07:55:54.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.682+0000 7f26b83ff700 1 -- 172.21.15.98:0/1305760611 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b015cf80 con 0x7f26b00b0d40 2023-03-03T07:55:54.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.682+0000 7fb4d0dff700 1 -- 172.21.15.98:0/4020553020 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4cc06cb50 con 0x7fb4cc16dc40 2023-03-03T07:55:54.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.682+0000 7fb4d0dff700 1 -- 172.21.15.98:0/4020553020 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4cc0cb440 con 0x7fb4cc16dc40 2023-03-03T07:55:54.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.683+0000 7fb4d0dff700 1 -- 172.21.15.98:0/4020553020 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb4cc06cce0 con 0x7fb4cc16dc40 2023-03-03T07:55:54.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.684+0000 7f26b83ff700 1 -- 172.21.15.98:0/1305760611 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f26b006ff70 con 0x7f26b00b0d40 2023-03-03T07:55:54.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.685+0000 7fef337fe700 1 -- 172.21.15.98:0/1431471722 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fef4401a030 con 0x7fef4c165290 2023-03-03T07:55:54.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.687+0000 7fef337fe700 1 --2- 172.21.15.98:0/1431471722 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fef380683c0 0x7fef3806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.687+0000 7fef337fe700 1 -- 172.21.15.98:0/1431471722 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fef440852d0 con 0x7fef4c165290 2023-03-03T07:55:54.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.688+0000 7fef337fe700 1 --2- 172.21.15.98:0/1431471722 >> [v2:172.21.15.98:6824/109121,v1:172.21.15.98:6825/109121] conn(0x7fef38070c70 0x7fef38073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.688+0000 7fef337fe700 1 -- 172.21.15.98:0/1431471722 --> [v2:172.21.15.98:6824/109121,v1:172.21.15.98:6825/109121] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef38073730 con 0x7fef38070c70 2023-03-03T07:55:54.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.688+0000 7fef337fe700 1 -- 172.21.15.98:0/1431471722 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef440878f0 con 0x7fef4c165290 2023-03-03T07:55:54.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.688+0000 7fef4affd700 1 --2- 172.21.15.98:0/1431471722 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fef380683c0 0x7fef3806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.686+0000 7fb4baffd700 1 -- 172.21.15.98:0/4020553020 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb4c4019030 con 0x7fb4cc16dc40 2023-03-03T07:55:54.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.688+0000 7fb4baffd700 1 --2- 172.21.15.98:0/4020553020 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb4ac0683c0 0x7fb4ac06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.688+0000 7fb4baffd700 1 -- 172.21.15.98:0/4020553020 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb4c40855c0 con 0x7fb4cc16dc40 2023-03-03T07:55:54.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.690+0000 7fef4b7fe700 1 --2- 172.21.15.98:0/1431471722 >> [v2:172.21.15.98:6824/109121,v1:172.21.15.98:6825/109121] conn(0x7fef38070c70 0x7fef38073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.690+0000 7fef4affd700 1 --2- 172.21.15.98:0/1431471722 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fef380683c0 0x7fef3806a870 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fef4c068f50 tx=0x7fef3c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.689+0000 7fb4baffd700 1 --2- 172.21.15.98:0/4020553020 >> [v2:172.21.15.98:6808/109122,v1:172.21.15.98:6809/109122] conn(0x7fb4ac070c70 0x7fb4ac073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.689+0000 7fb4baffd700 1 -- 172.21.15.98:0/4020553020 --> [v2:172.21.15.98:6808/109122,v1:172.21.15.98:6809/109122] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb4ac073730 con 0x7fb4ac070c70 2023-03-03T07:55:54.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.689+0000 7fb4baffd700 1 -- 172.21.15.98:0/4020553020 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4c4085970 con 0x7fb4cc16dc40 2023-03-03T07:55:54.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.690+0000 7f9ee23ff700 1 Processor -- start 2023-03-03T07:55:54.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.690+0000 7f9ee23ff700 1 -- start start 2023-03-03T07:55:54.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.690+0000 7f26a6ffd700 1 -- 172.21.15.98:0/1305760611 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f26ac02a030 con 0x7f26b00b0d40 2023-03-03T07:55:54.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.691+0000 7fb4ca59c700 1 --2- 172.21.15.98:0/4020553020 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb4ac0683c0 0x7fb4ac06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.691+0000 7fb4c9d9b700 1 --2- 172.21.15.98:0/4020553020 >> [v2:172.21.15.98:6808/109122,v1:172.21.15.98:6809/109122] conn(0x7fb4ac070c70 0x7fb4ac073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.692+0000 7f26a6ffd700 1 --2- 172.21.15.98:0/1305760611 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f26940683c0 0x7f269406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.693+0000 7f26a6ffd700 1 -- 172.21.15.98:0/1305760611 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f26ac096550 con 0x7f26b00b0d40 2023-03-03T07:55:54.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.693+0000 7f26a6ffd700 1 --2- 172.21.15.98:0/1305760611 >> [v2:172.21.15.98:6800/109118,v1:172.21.15.98:6801/109118] conn(0x7f2694070c70 0x7f2694073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.693+0000 7f26a6ffd700 1 -- 172.21.15.98:0/1305760611 --> [v2:172.21.15.98:6800/109118,v1:172.21.15.98:6801/109118] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2694073730 con 0x7f2694070c70 2023-03-03T07:55:54.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.693+0000 7f26a6ffd700 1 -- 172.21.15.98:0/1305760611 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26ac096900 con 0x7f26b00b0d40 2023-03-03T07:55:54.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.692+0000 7fb4c9d9b700 1 --2- 172.21.15.98:0/4020553020 >> [v2:172.21.15.98:6808/109122,v1:172.21.15.98:6809/109122] conn(0x7fb4ac070c70 0x7fb4ac073080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.692+0000 7fb4ca59c700 1 --2- 172.21.15.98:0/4020553020 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb4ac0683c0 0x7fb4ac06a870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc006fd0 tx=0x7fb4bc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.693+0000 7fef4b7fe700 1 --2- 172.21.15.98:0/1431471722 >> [v2:172.21.15.98:6824/109121,v1:172.21.15.98:6825/109121] conn(0x7fef38070c70 0x7fef38073080 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.694+0000 7f26b599a700 1 --2- 172.21.15.98:0/1305760611 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f26940683c0 0x7f269406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.694+0000 7f26b699c700 1 --2- 172.21.15.98:0/1305760611 >> [v2:172.21.15.98:6800/109118,v1:172.21.15.98:6801/109118] conn(0x7f2694070c70 0x7f2694073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.695+0000 7f26b599a700 1 --2- 172.21.15.98:0/1305760611 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f26940683c0 0x7f269406a870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f26b0146f20 tx=0x7f26a800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.695+0000 7f5fb71ff700 1 Processor -- start 2023-03-03T07:55:54.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.695+0000 7fef337fe700 1 -- 172.21.15.98:0/1431471722 <== osd.0 v2:172.21.15.98:6824/109121 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fef38073730 con 0x7fef38070c70 2023-03-03T07:55:54.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.695+0000 7fb4baffd700 1 -- 172.21.15.98:0/4020553020 <== osd.2 v2:172.21.15.98:6808/109122 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb4ac073730 con 0x7fb4ac070c70 2023-03-03T07:55:54.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.696+0000 7f9ee23ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9edc15ca70 0x7f9edc0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.696+0000 7f9ee23ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9edc06b4b0 0x7f9edc0cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.696+0000 7f9ee23ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9edc0cf980 con 0x7f9edc0d00a0 2023-03-03T07:55:54.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.696+0000 7f9ee23ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9edc0cd5b0 con 0x7f9edc06b4b0 2023-03-03T07:55:54.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.696+0000 7f9ee23ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9edc0d0540 con 0x7f9edc15ca70 2023-03-03T07:55:54.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.697+0000 7f9ee099c700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f9edc0d00a0 0x7f9edc0b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:40596/0 (socket says 172.21.15.98:40596) 2023-03-03T07:55:54.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.697+0000 7f9ee099c700 1 -- 172.21.15.98:0/1730268006 learned_addr learned my addr 172.21.15.98:0/1730268006 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:54.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.696+0000 7f5fb71ff700 1 -- start start 2023-03-03T07:55:54.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.698+0000 7f9edb7fe700 1 --2- 172.21.15.98:0/1730268006 >> v2:172.21.15.98:3300/0 conn(0x7f9edc06b4b0 0x7f9edc0cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.698+0000 7f5fb71ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5fb006d460 0x7f5fb006b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.698+0000 7f5fb71ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5fb00b0800 0x7f5fb00b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.698+0000 7f5fb71ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb00ce080 con 0x7f5fb00cc310 2023-03-03T07:55:54.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.698+0000 7f5fb71ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fb006c730 con 0x7f5fb00b0800 2023-03-03T07:55:54.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.698+0000 7f5fb71ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fb00b11c0 con 0x7f5fb006d460 2023-03-03T07:55:54.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.699+0000 7f9edaffd700 1 -- 172.21.15.98:0/1730268006 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93674056 0 0) 0x7f9edc0cf980 con 0x7f9edc0d00a0 2023-03-03T07:55:54.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.699+0000 7f9edaffd700 1 -- 172.21.15.98:0/1730268006 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec40035b0 con 0x7f9edc0d00a0 2023-03-03T07:55:54.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.699+0000 7f9edb7fe700 1 -- 172.21.15.98:0/1730268006 >> v2:172.21.15.100:3300/0 conn(0x7f9edc15ca70 msgr2=0x7f9edc0cf440 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-03T07:55:54.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.699+0000 7f9edb7fe700 1 --2- 172.21.15.98:0/1730268006 >> v2:172.21.15.100:3300/0 conn(0x7f9edc15ca70 0x7f9edc0cf440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.699+0000 7f9edb7fe700 1 -- 172.21.15.98:0/1730268006 >> v1:172.21.15.100:6789/0 conn(0x7f9edc0d00a0 legacy=0x7f9edc0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.700+0000 7f9edb7fe700 1 -- 172.21.15.98:0/1730268006 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc0d2fd0 con 0x7f9edc06b4b0 2023-03-03T07:55:54.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.700+0000 7f9edb7fe700 1 --2- 172.21.15.98:0/1730268006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9edc06b4b0 0x7f9edc0cfb60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9edc06c2c0 tx=0x7f9ed0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ff3f1f4718205a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.700+0000 7fc3c85ff700 1 Processor -- start 2023-03-03T07:55:54.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.700+0000 7f5fa7fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5fb006d460 0x7f5fb006b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.700+0000 7f5fa7fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5fb006d460 0x7f5fb006b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:46370/0 (socket says 172.21.15.98:46370) 2023-03-03T07:55:54.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.700+0000 7f5fa7fff700 1 -- 172.21.15.98:0/4131669926 learned_addr learned my addr 172.21.15.98:0/4131669926 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:54.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.701+0000 7f9edaffd700 1 -- 172.21.15.98:0/1730268006 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0011830 con 0x7f9edc06b4b0 2023-03-03T07:55:54.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.701+0000 7f9edaffd700 1 -- 172.21.15.98:0/1730268006 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed0011990 con 0x7f9edc06b4b0 2023-03-03T07:55:54.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.701+0000 7f9edaffd700 1 -- 172.21.15.98:0/1730268006 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0011d00 con 0x7f9edc06b4b0 2023-03-03T07:55:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.701+0000 7f5fb579c700 1 --2- 172.21.15.98:0/4131669926 >> v2:172.21.15.98:3300/0 conn(0x7f5fb00b0800 0x7f5fb00b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.701+0000 7fc3c85ff700 1 -- start start 2023-03-03T07:55:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.702+0000 7f9ee23ff700 1 -- 172.21.15.98:0/1730268006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9edc06b4b0 msgr2=0x7f9edc0cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.702+0000 7f9ee23ff700 1 --2- 172.21.15.98:0/1730268006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9edc06b4b0 0x7f9edc0cfb60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9edc06c2c0 tx=0x7f9ed0009b20 comp rx=0 tx=0).stop 2023-03-03T07:55:54.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.702+0000 7f9ee23ff700 1 -- 172.21.15.98:0/1730268006 shutdown_connections 2023-03-03T07:55:54.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.702+0000 7f9ee23ff700 1 --2- 172.21.15.98:0/1730268006 >> v2:172.21.15.100:3300/0 conn(0x7f9edc15ca70 0x7f9edc0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.702+0000 7f9ee23ff700 1 --2- 172.21.15.98:0/1730268006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9edc06b4b0 0x7f9edc0cfb60 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9edc06c2c0 tx=0x7f9ed0009b20 comp rx=0 tx=0).stop 2023-03-03T07:55:54.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.702+0000 7f9ee23ff700 1 -- 172.21.15.98:0/1730268006 >> 172.21.15.98:0/1730268006 conn(0x7f9edc073e70 msgr2=0x7f9edc06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.702+0000 7f5fa7fff700 1 -- 172.21.15.98:0/4131669926 >> v2:172.21.15.98:3300/0 conn(0x7f5fb00b0800 msgr2=0x7f5fb00b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:54.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.702+0000 7f5fa7fff700 1 --2- 172.21.15.98:0/4131669926 >> v2:172.21.15.98:3300/0 conn(0x7f5fb00b0800 0x7f5fb00b0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.702+0000 7f5fa7fff700 1 -- 172.21.15.98:0/4131669926 >> v1:172.21.15.98:6789/0 conn(0x7f5fb00cc310 legacy=0x7f5fb00ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.702+0000 7f5fa7fff700 1 -- 172.21.15.98:0/4131669926 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb006c170 con 0x7f5fb006d460 2023-03-03T07:55:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.701+0000 7fd8d5fff700 1 Processor -- start 2023-03-03T07:55:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.702+0000 7fd8d5fff700 1 -- start start 2023-03-03T07:55:54.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.703+0000 7f26b699c700 1 --2- 172.21.15.98:0/1305760611 >> [v2:172.21.15.98:6800/109118,v1:172.21.15.98:6801/109118] conn(0x7f2694070c70 0x7f2694073080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.703+0000 7f9ee23ff700 1 -- 172.21.15.98:0/1730268006 shutdown_connections 2023-03-03T07:55:54.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.703+0000 7fc3c85ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3c015ca70 0x7fc3c00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.703+0000 7fc3c85ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c00cf980 con 0x7fc3c00d0270 2023-03-03T07:55:54.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.703+0000 7fc3c85ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c00d1800 con 0x7fc3c006b4b0 2023-03-03T07:55:54.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.703+0000 7fc3c85ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c00cd5b0 con 0x7fc3c015ca70 2023-03-03T07:55:54.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.703+0000 7f5fa7fff700 1 --2- 172.21.15.98:0/4131669926 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fb006d460 0x7f5fb006b820 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5fb006f850 tx=0x7f5fa000be70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a754cb6ac4fe2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.705+0000 7f26a6ffd700 1 -- 172.21.15.98:0/1305760611 <== osd.1 v2:172.21.15.98:6800/109118 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f2694073730 con 0x7f2694070c70 2023-03-03T07:55:54.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.704+0000 7fc3c5b9a700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fc3c006b4b0 0x7fc3c00cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:40612/0 (socket says 172.21.15.98:40612) 2023-03-03T07:55:54.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.704+0000 7fc3c5b9a700 1 -- 172.21.15.98:0/3719423698 learned_addr learned my addr 172.21.15.98:0/3719423698 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:54.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.704+0000 7fc3c639b700 1 --2- 172.21.15.98:0/3719423698 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3c015ca70 0x7fc3c00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.704+0000 7f5fa77fe700 1 -- 172.21.15.98:0/4131669926 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa000dc70 con 0x7f5fb006d460 2023-03-03T07:55:54.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.704+0000 7f5fa77fe700 1 -- 172.21.15.98:0/4131669926 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa000ddd0 con 0x7f5fb006d460 2023-03-03T07:55:54.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.704+0000 7f5fa77fe700 1 -- 172.21.15.98:0/4131669926 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0010690 con 0x7f5fb006d460 2023-03-03T07:55:54.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.705+0000 7f5fb71ff700 1 -- 172.21.15.98:0/4131669926 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fb006d460 msgr2=0x7f5fb006b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.705+0000 7f5fb71ff700 1 --2- 172.21.15.98:0/4131669926 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fb006d460 0x7f5fb006b820 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5fb006f850 tx=0x7f5fa000be70 comp rx=0 tx=0).stop 2023-03-03T07:55:54.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.705+0000 7f5fb71ff700 1 -- 172.21.15.98:0/4131669926 shutdown_connections 2023-03-03T07:55:54.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.705+0000 7f5fb71ff700 1 --2- 172.21.15.98:0/4131669926 >> v2:172.21.15.98:3300/0 conn(0x7f5fb00b0800 0x7f5fb00b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.705+0000 7f5fb71ff700 1 --2- 172.21.15.98:0/4131669926 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fb006d460 0x7f5fb006b820 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.705+0000 7f5fb71ff700 1 -- 172.21.15.98:0/4131669926 >> 172.21.15.98:0/4131669926 conn(0x7f5fb00cb800 msgr2=0x7f5fb00cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.706+0000 7fc3c639b700 1 -- 172.21.15.98:0/3719423698 >> v1:172.21.15.100:6789/0 conn(0x7fc3c006b4b0 legacy=0x7fc3c00cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.706+0000 7fc3c639b700 1 -- 172.21.15.98:0/3719423698 >> v1:172.21.15.98:6789/0 conn(0x7fc3c00d0270 legacy=0x7fc3c00d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.706+0000 7fc3c639b700 1 -- 172.21.15.98:0/3719423698 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c00cd900 con 0x7fc3c015ca70 2023-03-03T07:55:54.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.704+0000 7fd8d5fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd8d006df90 0x7fd8d015c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.704+0000 7fd8d5fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8d015cb00 0x7fd8d0158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.704+0000 7fd8d5fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d015d540 con 0x7fd8d0158bd0 2023-03-03T07:55:54.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.704+0000 7fd8d5fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d0071780 con 0x7fd8d006df90 2023-03-03T07:55:54.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.704+0000 7fd8d5fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d00702e0 con 0x7fd8d015cb00 2023-03-03T07:55:54.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.705+0000 7fd8ceffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8d015cb00 0x7fd8d0158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.705+0000 7fd8ceffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8d015cb00 0x7fd8d0158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:42344/0 (socket says 172.21.15.98:42344) 2023-03-03T07:55:54.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.705+0000 7fd8ceffd700 1 -- 172.21.15.98:0/1943520183 learned_addr learned my addr 172.21.15.98:0/1943520183 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:54.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.706+0000 7fd8cf7fe700 1 --2- 172.21.15.98:0/1943520183 >> v2:172.21.15.98:3300/0 conn(0x7fd8d006df90 0x7fd8d015c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.707+0000 7fc3c639b700 1 --2- 172.21.15.98:0/3719423698 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3c015ca70 0x7fc3c00cf440 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc3b00097f0 tx=0x7fc3b0009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a097886df6c5592 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.703+0000 7f9ee23ff700 1 -- 172.21.15.98:0/1730268006 wait complete. 2023-03-03T07:55:54.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.706+0000 7fd8ceffd700 1 -- 172.21.15.98:0/1943520183 >> v2:172.21.15.98:3300/0 conn(0x7fd8d006df90 msgr2=0x7fd8d015c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:54.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.706+0000 7fd8ceffd700 1 --2- 172.21.15.98:0/1943520183 >> v2:172.21.15.98:3300/0 conn(0x7fd8d006df90 0x7fd8d015c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.706+0000 7fd8ceffd700 1 -- 172.21.15.98:0/1943520183 >> v1:172.21.15.100:6789/0 conn(0x7fd8d0158bd0 legacy=0x7fd8d00b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.706+0000 7fd8ceffd700 1 -- 172.21.15.98:0/1943520183 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d015f910 con 0x7fd8d015cb00 2023-03-03T07:55:54.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.707+0000 7f9ee23ff700 1 Processor -- start 2023-03-03T07:55:54.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.707+0000 7f5fb71ff700 1 -- 172.21.15.98:0/4131669926 shutdown_connections 2023-03-03T07:55:54.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.707+0000 7f9ee23ff700 1 -- start start 2023-03-03T07:55:54.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.709+0000 7f9ee23ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9edc0d00a0 0x7f9edc1ff020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.709+0000 7f9ee23ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9edc0b4bf0 0x7f9edc0b5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.709+0000 7f9ee23ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9edc0cc4c0 0x7f9edc0cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.709+0000 7f9ee23ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc0b55a0 con 0x7f9edc0d00a0 2023-03-03T07:55:54.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.709+0000 7f9ee23ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc0cd1d0 con 0x7f9edc0cc4c0 2023-03-03T07:55:54.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.709+0000 7f9ee23ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc202f00 con 0x7f9edc0b4bf0 2023-03-03T07:55:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.710+0000 7f9edb7fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9edc0b4bf0 0x7f9edc0b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.710+0000 7f9edb7fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9edc0b4bf0 0x7f9edc0b5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:42354/0 (socket says 172.21.15.98:42354) 2023-03-03T07:55:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.710+0000 7f9edb7fe700 1 -- 172.21.15.98:0/4011822506 learned_addr learned my addr 172.21.15.98:0/4011822506 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.710+0000 7f9edb7fe700 1 -- 172.21.15.98:0/4011822506 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9edc0cc4c0 msgr2=0x7f9edc0cc960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-03T07:55:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.710+0000 7f9edb7fe700 1 --2- 172.21.15.98:0/4011822506 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9edc0cc4c0 0x7f9edc0cc960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.710+0000 7f9edb7fe700 1 -- 172.21.15.98:0/4011822506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9edc0d00a0 msgr2=0x7f9edc1ff020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.710+0000 7f9edb7fe700 1 --2- 172.21.15.98:0/4011822506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9edc0d00a0 0x7f9edc1ff020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.710+0000 7f9edb7fe700 1 -- 172.21.15.98:0/4011822506 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed00094a0 con 0x7f9edc0b4bf0 2023-03-03T07:55:54.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.708+0000 7fc3c5399700 1 -- 172.21.15.98:0/3719423698 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0004230 con 0x7fc3c015ca70 2023-03-03T07:55:54.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.708+0000 7fc3c5399700 1 -- 172.21.15.98:0/3719423698 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b0004390 con 0x7fc3c015ca70 2023-03-03T07:55:54.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.708+0000 7fc3c5399700 1 -- 172.21.15.98:0/3719423698 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b00036a0 con 0x7fc3c015ca70 2023-03-03T07:55:54.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.711+0000 7fc3c85ff700 1 -- 172.21.15.98:0/3719423698 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3c015ca70 msgr2=0x7fc3c00cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.711+0000 7fc3c85ff700 1 --2- 172.21.15.98:0/3719423698 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3c015ca70 0x7fc3c00cf440 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc3b00097f0 tx=0x7fc3b0009b00 comp rx=0 tx=0).stop 2023-03-03T07:55:54.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.707+0000 7f5fb71ff700 1 -- 172.21.15.98:0/4131669926 wait complete. 2023-03-03T07:55:54.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.711+0000 7f5fb71ff700 1 Processor -- start 2023-03-03T07:55:54.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.712+0000 7fc3c85ff700 1 -- 172.21.15.98:0/3719423698 shutdown_connections 2023-03-03T07:55:54.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.712+0000 7fc3c85ff700 1 --2- 172.21.15.98:0/3719423698 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3c015ca70 0x7fc3c00cf440 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.712+0000 7fc3c85ff700 1 -- 172.21.15.98:0/3719423698 >> 172.21.15.98:0/3719423698 conn(0x7fc3c0073e70 msgr2=0x7fc3c006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.712+0000 7fc3c85ff700 1 -- 172.21.15.98:0/3719423698 shutdown_connections 2023-03-03T07:55:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.711+0000 7f5fb71ff700 1 -- start start 2023-03-03T07:55:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.713+0000 7f5fb71ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fb006d460 0x7f5fb01576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.713+0000 7f5fb71ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb00b0800 0x7f5fb00c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.713+0000 7f5fb71ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fb00cc310 0x7f5fb01e49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.713+0000 7f5fb71ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00b2d10 con 0x7f5fb00b0800 2023-03-03T07:55:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.713+0000 7f5fb71ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00b2100 con 0x7f5fb00cc310 2023-03-03T07:55:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.713+0000 7f5fb71ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00b23e0 con 0x7f5fb006d460 2023-03-03T07:55:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.714+0000 7f9edb7fe700 1 --2- 172.21.15.98:0/4011822506 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9edc0b4bf0 0x7f9edc0b5060 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00090b0 tx=0x7f9ed0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.714+0000 7fd8ceffd700 1 --2- 172.21.15.98:0/1943520183 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8d015cb00 0x7fd8d0158690 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00098f0 tx=0x7fd8c0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9dc8eaf16c89e04 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.714+0000 7f5fb4f9b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fb006d460 0x7f5fb01576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.714+0000 7f5fb4f9b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fb006d460 0x7f5fb01576c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:42356/0 (socket says 172.21.15.98:42356) 2023-03-03T07:55:54.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.714+0000 7f5fb4f9b700 1 -- 172.21.15.98:0/1275426016 learned_addr learned my addr 172.21.15.98:0/1275426016 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:54.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.715+0000 7f5fb579c700 1 --2- 172.21.15.98:0/1275426016 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fb00cc310 0x7f5fb01e49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.715+0000 7f5fa7fff700 1 --2- 172.21.15.98:0/1275426016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb00b0800 0x7f5fb00c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.716+0000 7f5fb4f9b700 1 -- 172.21.15.98:0/1275426016 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fb00cc310 msgr2=0x7f5fb01e49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.716+0000 7f5fb4f9b700 1 --2- 172.21.15.98:0/1275426016 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fb00cc310 0x7f5fb01e49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.716+0000 7f5fb4f9b700 1 -- 172.21.15.98:0/1275426016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb00b0800 msgr2=0x7f5fb00c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.716+0000 7f5fb4f9b700 1 --2- 172.21.15.98:0/1275426016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb00b0800 0x7f5fb00c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.716+0000 7f5fb4f9b700 1 -- 172.21.15.98:0/1275426016 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa000b7f0 con 0x7f5fb006d460 2023-03-03T07:55:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.715+0000 7fd8ce7fc700 1 -- 172.21.15.98:0/1943520183 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c0004230 con 0x7fd8d015cb00 2023-03-03T07:55:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.715+0000 7fd8ce7fc700 1 -- 172.21.15.98:0/1943520183 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c0004390 con 0x7fd8d015cb00 2023-03-03T07:55:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.715+0000 7fd8ce7fc700 1 -- 172.21.15.98:0/1943520183 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c00036a0 con 0x7fd8d015cb00 2023-03-03T07:55:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.716+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1943520183 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8d015cb00 msgr2=0x7fd8d0158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.716+0000 7fd8d5fff700 1 --2- 172.21.15.98:0/1943520183 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8d015cb00 0x7fd8d0158690 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00098f0 tx=0x7fd8c0009c00 comp rx=0 tx=0).stop 2023-03-03T07:55:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.717+0000 7f9ed8ff9700 1 -- 172.21.15.98:0/4011822506 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed00045d0 con 0x7f9edc0b4bf0 2023-03-03T07:55:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.717+0000 7f9ed8ff9700 1 -- 172.21.15.98:0/4011822506 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed001f070 con 0x7f9edc0b4bf0 2023-03-03T07:55:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.717+0000 7f9ed8ff9700 1 -- 172.21.15.98:0/4011822506 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed00038c0 con 0x7f9edc0b4bf0 2023-03-03T07:55:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.712+0000 7fc3c85ff700 1 -- 172.21.15.98:0/3719423698 wait complete. 2023-03-03T07:55:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.717+0000 7f5fb4f9b700 1 --2- 172.21.15.98:0/1275426016 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fb006d460 0x7f5fb01576c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00c450 tx=0x7f5fac00c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.716+0000 7fc3c85ff700 1 Processor -- start 2023-03-03T07:55:54.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.716+0000 7fc3c85ff700 1 -- start start 2023-03-03T07:55:54.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.717+0000 7f9ee23ff700 1 -- 172.21.15.98:0/4011822506 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9edc0cd900 con 0x7f9edc0b4bf0 2023-03-03T07:55:54.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7f2a889fe700 1 Processor -- start 2023-03-03T07:55:54.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7fc3c85ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c006b4b0 0x7fc3c01e6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7fc3c85ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3c00d0270 0x7fc3c00ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7fc3c85ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3c015ca70 0x7fc3c00ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7fc3c85ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c01e7300 con 0x7fc3c006b4b0 2023-03-03T07:55:54.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7fc3c85ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c00cce60 con 0x7fc3c015ca70 2023-03-03T07:55:54.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7fc3c85ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c00cd2f0 con 0x7fc3c00d0270 2023-03-03T07:55:54.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7f9ee23ff700 1 -- 172.21.15.98:0/4011822506 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9edc0d1800 con 0x7f9edc0b4bf0 2023-03-03T07:55:54.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7fc3c5b9a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3c00d0270 0x7fc3c00ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7fc3c5b9a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3c00d0270 0x7fc3c00ca250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:42364/0 (socket says 172.21.15.98:42364) 2023-03-03T07:55:54.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7fc3c5b9a700 1 -- 172.21.15.98:0/1611713710 learned_addr learned my addr 172.21.15.98:0/1611713710 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:54.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7fc3c6b9c700 1 --2- 172.21.15.98:0/1611713710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3c015ca70 0x7fc3c00ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.719+0000 7fc3c5b9a700 1 -- 172.21.15.98:0/1611713710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3c015ca70 msgr2=0x7fc3c00ca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.719+0000 7fc3c5b9a700 1 --2- 172.21.15.98:0/1611713710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3c015ca70 0x7fc3c00ca790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.719+0000 7fc3c5b9a700 1 -- 172.21.15.98:0/1611713710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c006b4b0 msgr2=0x7fc3c01e6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.719+0000 7fc3c5b9a700 1 --2- 172.21.15.98:0/1611713710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c006b4b0 0x7fc3c01e6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.719+0000 7fc3c5b9a700 1 -- 172.21.15.98:0/1611713710 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b00094a0 con 0x7fc3c00d0270 2023-03-03T07:55:54.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.720+0000 7fc3c5b9a700 1 --2- 172.21.15.98:0/1611713710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3c00d0270 0x7fc3c00ca250 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc001f60 tx=0x7fc3bc00a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7f5fa57fa700 1 -- 172.21.15.98:0/1275426016 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fac0052d0 con 0x7f5fb006d460 2023-03-03T07:55:54.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7f5fa57fa700 1 -- 172.21.15.98:0/1275426016 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fac005430 con 0x7f5fb006d460 2023-03-03T07:55:54.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7f5fa57fa700 1 -- 172.21.15.98:0/1275426016 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fac0038c0 con 0x7f5fb006d460 2023-03-03T07:55:54.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.720+0000 7fc3b6ffd700 1 -- 172.21.15.98:0/1611713710 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc00ade0 con 0x7fc3c00d0270 2023-03-03T07:55:54.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.720+0000 7fc3b6ffd700 1 -- 172.21.15.98:0/1611713710 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3bc0070d0 con 0x7fc3c00d0270 2023-03-03T07:55:54.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.720+0000 7fc3b6ffd700 1 -- 172.21.15.98:0/1611713710 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc007480 con 0x7fc3c00d0270 2023-03-03T07:55:54.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.720+0000 7f5fb71ff700 1 -- 172.21.15.98:0/1275426016 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb00cdd10 con 0x7f5fb006d460 2023-03-03T07:55:54.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.720+0000 7f5fb71ff700 1 -- 172.21.15.98:0/1275426016 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb006c450 con 0x7f5fb006d460 2023-03-03T07:55:54.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.718+0000 7f2a889fe700 1 -- start start 2023-03-03T07:55:54.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.719+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1943520183 shutdown_connections 2023-03-03T07:55:54.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.719+0000 7fd8d5fff700 1 --2- 172.21.15.98:0/1943520183 >> v2:172.21.15.98:3300/0 conn(0x7fd8d006df90 0x7fd8d015c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.719+0000 7fd8d5fff700 1 --2- 172.21.15.98:0/1943520183 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8d015cb00 0x7fd8d0158690 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.719+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1943520183 >> 172.21.15.98:0/1943520183 conn(0x7fd8d00cb800 msgr2=0x7fd8d00cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.719+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1943520183 shutdown_connections 2023-03-03T07:55:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.719+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1943520183 wait complete. 2023-03-03T07:55:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.721+0000 7f5fb71ff700 1 -- 172.21.15.98:0/1275426016 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5fb006bd60 con 0x7f5fb006d460 2023-03-03T07:55:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.721+0000 7f9ee23ff700 1 -- 172.21.15.98:0/4011822506 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9edc0cdbe0 con 0x7f9edc0b4bf0 2023-03-03T07:55:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.722+0000 7fc3c85ff700 1 -- 172.21.15.98:0/1611713710 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c00d1aa0 con 0x7fc3c00d0270 2023-03-03T07:55:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.722+0000 7fc3c85ff700 1 -- 172.21.15.98:0/1611713710 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c00cdbe0 con 0x7fc3c00d0270 2023-03-03T07:55:54.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.723+0000 7fd8d5fff700 1 Processor -- start 2023-03-03T07:55:54.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.723+0000 7fd8d5fff700 1 -- start start 2023-03-03T07:55:54.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.724+0000 7f9ed8ff9700 1 -- 172.21.15.98:0/4011822506 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9ed000f050 con 0x7f9edc0b4bf0 2023-03-03T07:55:54.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.726+0000 7f9ed8ff9700 1 --2- 172.21.15.98:0/4011822506 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ec80683c0 0x7f9ec806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.726+0000 7f9ed8ff9700 1 -- 172.21.15.98:0/4011822506 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9ed0021070 con 0x7f9edc0b4bf0 2023-03-03T07:55:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.726+0000 7f9ed8ff9700 1 --2- 172.21.15.98:0/4011822506 >> [v2:172.21.15.98:6816/109119,v1:172.21.15.98:6817/109119] conn(0x7f9ec8070c70 0x7f9ec8073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.726+0000 7f9ed8ff9700 1 -- 172.21.15.98:0/4011822506 --> [v2:172.21.15.98:6816/109119,v1:172.21.15.98:6817/109119] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ec8073730 con 0x7f9ec8070c70 2023-03-03T07:55:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.727+0000 7f9ed8ff9700 1 -- 172.21.15.98:0/4011822506 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ed0089860 con 0x7f9edc0b4bf0 2023-03-03T07:55:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.724+0000 7f2a889fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a8413eba0 0x7f2a840caf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.724+0000 7f2a889fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2a8406ac50 0x7f2a841585b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.724+0000 7f2a889fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2a840cef80 0x7f2a8406bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.724+0000 7f2a889fe700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a8413dee0 con 0x7f2a840cef80 2023-03-03T07:55:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.725+0000 7f2a889fe700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a8413e710 con 0x7f2a8406ac50 2023-03-03T07:55:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.725+0000 7f2a889fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a84158af0 con 0x7f2a8413eba0 2023-03-03T07:55:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.725+0000 7f2a8259c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2a8406ac50 0x7f2a841585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.726+0000 7f2a82d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a8413eba0 0x7f2a840caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.726+0000 7f2a8259c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2a8406ac50 0x7f2a841585b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:46396/0 (socket says 172.21.15.98:46396) 2023-03-03T07:55:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.726+0000 7f2a8259c700 1 -- 172.21.15.98:0/1987098014 learned_addr learned my addr 172.21.15.98:0/1987098014 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.727+0000 7f2a82d9d700 1 -- 172.21.15.98:0/1987098014 >> v2:172.21.15.100:3300/0 conn(0x7f2a8406ac50 msgr2=0x7f2a841585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.727+0000 7f2a82d9d700 1 --2- 172.21.15.98:0/1987098014 >> v2:172.21.15.100:3300/0 conn(0x7f2a8406ac50 0x7f2a841585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.727+0000 7f2a82d9d700 1 -- 172.21.15.98:0/1987098014 >> v2:172.21.15.98:3300/0 conn(0x7f2a840cef80 msgr2=0x7f2a8406bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.727+0000 7f2a82d9d700 1 --2- 172.21.15.98:0/1987098014 >> v2:172.21.15.98:3300/0 conn(0x7f2a840cef80 0x7f2a8406bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.727+0000 7f2a82d9d700 1 -- 172.21.15.98:0/1987098014 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a840cf790 con 0x7f2a8413eba0 2023-03-03T07:55:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.727+0000 7fd8d5fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8d006df90 0x7fd8d015bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.727+0000 7fd8d5fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8d0158bd0 0x7fd8d014c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.727+0000 7fd8d5fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d014a870 0x7fd8d015d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.727+0000 7fd8d5fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d015b600 con 0x7fd8d014a870 2023-03-03T07:55:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.727+0000 7fd8d5fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d014cee0 con 0x7fd8d0158bd0 2023-03-03T07:55:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.727+0000 7fd8d5fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d00b1c50 con 0x7fd8d006df90 2023-03-03T07:55:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.727+0000 7f9edbfff700 1 --2- 172.21.15.98:0/4011822506 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ec80683c0 0x7f9ec806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.728+0000 7fd8cf7fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8d006df90 0x7fd8d015bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.727+0000 7fc3b6ffd700 1 -- 172.21.15.98:0/1611713710 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc3bc01a030 con 0x7fc3c00d0270 2023-03-03T07:55:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.728+0000 7fd8cffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d014a870 0x7fd8d015d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.728+0000 7fd8cffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d014a870 0x7fd8d015d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45484/0 (socket says 172.21.15.98:45484) 2023-03-03T07:55:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.728+0000 7fd8cffff700 1 -- 172.21.15.98:0/1196265000 learned_addr learned my addr 172.21.15.98:0/1196265000 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.729+0000 7f9edbfff700 1 --2- 172.21.15.98:0/4011822506 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ec80683c0 0x7f9ec806a870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0b5ce0 tx=0x7f9ecc006cd0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.729+0000 7fd8ceffd700 1 --2- 172.21.15.98:0/1196265000 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8d0158bd0 0x7fd8d014c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.729+0000 7fd8cffff700 1 -- 172.21.15.98:0/1196265000 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8d006df90 msgr2=0x7fd8d015bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.729+0000 7fd8cffff700 1 --2- 172.21.15.98:0/1196265000 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8d006df90 0x7fd8d015bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.729+0000 7fd8cffff700 1 -- 172.21.15.98:0/1196265000 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8d0158bd0 msgr2=0x7fd8d014c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.729+0000 7fd8cffff700 1 --2- 172.21.15.98:0/1196265000 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8d0158bd0 0x7fd8d014c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.729+0000 7fd8cffff700 1 -- 172.21.15.98:0/1196265000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c0009550 con 0x7fd8d014a870 2023-03-03T07:55:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.725+0000 7f5fa57fa700 1 -- 172.21.15.98:0/1275426016 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5fac019030 con 0x7f5fb006d460 2023-03-03T07:55:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.727+0000 7f5fa57fa700 1 --2- 172.21.15.98:0/1275426016 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5f9c0683c0 0x7f5f9c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.728+0000 7f5fa57fa700 1 -- 172.21.15.98:0/1275426016 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5fac085730 con 0x7f5fb006d460 2023-03-03T07:55:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.728+0000 7f5fa57fa700 1 --2- 172.21.15.98:0/1275426016 >> [v2:172.21.15.100:6808/109379,v1:172.21.15.100:6810/109379] conn(0x7f5f9c070c70 0x7f5f9c073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.728+0000 7f5fa57fa700 1 -- 172.21.15.98:0/1275426016 --> [v2:172.21.15.100:6808/109379,v1:172.21.15.100:6810/109379] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f9c073730 con 0x7f5f9c070c70 2023-03-03T07:55:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.728+0000 7f5fa57fa700 1 -- 172.21.15.98:0/1275426016 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fac085ae0 con 0x7f5fb006d460 2023-03-03T07:55:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.729+0000 7f5fb579c700 1 --2- 172.21.15.98:0/1275426016 >> [v2:172.21.15.100:6808/109379,v1:172.21.15.100:6810/109379] conn(0x7f5f9c070c70 0x7f5f9c073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.729+0000 7f5fa7fff700 1 --2- 172.21.15.98:0/1275426016 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5f9c0683c0 0x7f5f9c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.729+0000 7fd8cf7fe700 1 --2- 172.21.15.98:0/1196265000 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8d006df90 0x7fd8d015bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:55:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.729+0000 7f2a82d9d700 1 --2- 172.21.15.98:0/1987098014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a8413eba0 0x7f2a840caf80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2a84073cc0 tx=0x7f2a74009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9b005a11fc5bec9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.730+0000 7fb4d0dff700 1 -- 172.21.15.98:0/4020553020 --> [v2:172.21.15.98:6808/109122,v1:172.21.15.98:6809/109122] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb4cc04e6e0 con 0x7fb4ac070c70 2023-03-03T07:55:54.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.730+0000 7fd8cffff700 1 --2- 172.21.15.98:0/1196265000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d014a870 0x7fd8d015d940 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800c870 tx=0x7fd8c800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.730+0000 7f9ee099c700 1 --2- 172.21.15.98:0/4011822506 >> [v2:172.21.15.98:6816/109119,v1:172.21.15.98:6817/109119] conn(0x7f9ec8070c70 0x7f9ec8073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.730+0000 7fb4baffd700 1 -- 172.21.15.98:0/4020553020 <== osd.2 v2:172.21.15.98:6808/109122 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb4cc04e6e0 con 0x7fb4ac070c70 2023-03-03T07:55:54.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.730+0000 7f2a81d9b700 1 -- 172.21.15.98:0/1987098014 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a74011830 con 0x7f2a8413eba0 2023-03-03T07:55:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.730+0000 7f2a81d9b700 1 -- 172.21.15.98:0/1987098014 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a74011990 con 0x7f2a8413eba0 2023-03-03T07:55:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.730+0000 7f2a81d9b700 1 -- 172.21.15.98:0/1987098014 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a74011d20 con 0x7f2a8413eba0 2023-03-03T07:55:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.730+0000 7fd8b7fff700 1 -- 172.21.15.98:0/1196265000 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8003e30 con 0x7fd8d014a870 2023-03-03T07:55:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.731+0000 7fd8b7fff700 1 -- 172.21.15.98:0/1196265000 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c800ce70 con 0x7fd8d014a870 2023-03-03T07:55:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.731+0000 7fd8b7fff700 1 -- 172.21.15.98:0/1196265000 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8005270 con 0x7fd8d014a870 2023-03-03T07:55:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.730+0000 7fc3b6ffd700 1 --2- 172.21.15.98:0/1611713710 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3a40683c0 0x7fc3a406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.730+0000 7fc3b6ffd700 1 -- 172.21.15.98:0/1611713710 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc3bc085cf0 con 0x7fc3c00d0270 2023-03-03T07:55:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.731+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1196265000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d006ff70 con 0x7fd8d014a870 2023-03-03T07:55:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.731+0000 7fc3c85ff700 1 --2- 172.21.15.98:0/1611713710 >> [v2:172.21.15.100:6819/109381,v1:172.21.15.100:6823/109381] conn(0x7fc3c00cc470 0x7fc3c00cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.731+0000 7fc3c85ff700 1 -- 172.21.15.98:0/1611713710 --> [v2:172.21.15.100:6819/109381,v1:172.21.15.100:6823/109381] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3c00cd140 con 0x7fc3c00cc470 2023-03-03T07:55:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.731+0000 7fc3c639b700 1 --2- 172.21.15.98:0/1611713710 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3a40683c0 0x7fc3a406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.731+0000 7f5fb579c700 1 --2- 172.21.15.98:0/1275426016 >> [v2:172.21.15.100:6808/109379,v1:172.21.15.100:6810/109379] conn(0x7f5f9c070c70 0x7f5f9c073080 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.731+0000 7f5fa7fff700 1 --2- 172.21.15.98:0/1275426016 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5f9c0683c0 0x7f5f9c06a870 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00c9840 tx=0x7f5fa0013000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.731+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1196265000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d015d260 con 0x7fd8d014a870 2023-03-03T07:55:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.731+0000 7fc3c6b9c700 1 --2- 172.21.15.98:0/1611713710 >> [v2:172.21.15.100:6819/109381,v1:172.21.15.100:6823/109381] conn(0x7fc3c00cc470 0x7fc3c00cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.731+0000 7f9ee099c700 1 --2- 172.21.15.98:0/4011822506 >> [v2:172.21.15.98:6816/109119,v1:172.21.15.98:6817/109119] conn(0x7f9ec8070c70 0x7f9ec8073080 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.732+0000 7f5fa57fa700 1 -- 172.21.15.98:0/1275426016 <== osd.5 v2:172.21.15.100:6808/109379 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f5f9c073730 con 0x7f5f9c070c70 2023-03-03T07:55:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.733+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1196265000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd8d00cc950 con 0x7fd8d014a870 2023-03-03T07:55:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.732+0000 7fc3c639b700 1 --2- 172.21.15.98:0/1611713710 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3a40683c0 0x7fc3a406a870 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0009b40 tx=0x7fc3b0010000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.733+0000 7fc3c6b9c700 1 --2- 172.21.15.98:0/1611713710 >> [v2:172.21.15.100:6819/109381,v1:172.21.15.100:6823/109381] conn(0x7fc3c00cc470 0x7fc3c00cc840 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.733+0000 7f9ed8ff9700 1 -- 172.21.15.98:0/4011822506 <== osd.3 v2:172.21.15.98:6816/109119 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f9ec8073730 con 0x7f9ec8070c70 2023-03-03T07:55:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.733+0000 7fef513ff700 1 -- 172.21.15.98:0/1431471722 --> [v2:172.21.15.98:6824/109121,v1:172.21.15.98:6825/109121] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fef4c14ddf0 con 0x7fef38070c70 2023-03-03T07:55:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.734+0000 7fb4d0dff700 1 -- 172.21.15.98:0/4020553020 >> [v2:172.21.15.98:6808/109122,v1:172.21.15.98:6809/109122] conn(0x7fb4ac070c70 msgr2=0x7fb4ac073080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.734+0000 7fb4d0dff700 1 --2- 172.21.15.98:0/4020553020 >> [v2:172.21.15.98:6808/109122,v1:172.21.15.98:6809/109122] conn(0x7fb4ac070c70 0x7fb4ac073080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.734+0000 7fef337fe700 1 -- 172.21.15.98:0/1431471722 <== osd.0 v2:172.21.15.98:6824/109121 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fef4c14ddf0 con 0x7fef38070c70 2023-03-03T07:55:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.734+0000 7fb4d0dff700 1 -- 172.21.15.98:0/4020553020 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb4ac0683c0 msgr2=0x7fb4ac06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.734+0000 7fb4d0dff700 1 --2- 172.21.15.98:0/4020553020 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb4ac0683c0 0x7fb4ac06a870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc006fd0 tx=0x7fb4bc008040 comp rx=0 tx=0).stop 2023-03-03T07:55:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.734+0000 7fb4d0dff700 1 -- 172.21.15.98:0/4020553020 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4cc16dc40 msgr2=0x7fb4cc088a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.734+0000 7fb4d0dff700 1 --2- 172.21.15.98:0/4020553020 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4cc16dc40 0x7fb4cc088a50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400c7f0 tx=0x7fb4c400cbb0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.734+0000 7f2a889fe700 1 -- 172.21.15.98:0/1987098014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a8413eba0 msgr2=0x7f2a840caf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.734+0000 7f2a889fe700 1 --2- 172.21.15.98:0/1987098014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a8413eba0 0x7f2a840caf80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2a84073cc0 tx=0x7f2a74009b20 comp rx=0 tx=0).stop 2023-03-03T07:55:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.735+0000 7fb4cad9d700 1 -- 172.21.15.98:0/4020553020 reap_dead start 2023-03-03T07:55:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.735+0000 7f2a889fe700 1 -- 172.21.15.98:0/1987098014 shutdown_connections 2023-03-03T07:55:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.735+0000 7f2a889fe700 1 --2- 172.21.15.98:0/1987098014 >> v2:172.21.15.98:3300/0 conn(0x7f2a840cef80 0x7f2a8406bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.735+0000 7f2a889fe700 1 --2- 172.21.15.98:0/1987098014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a8413eba0 0x7f2a840caf80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.735+0000 7f2a889fe700 1 --2- 172.21.15.98:0/1987098014 >> v2:172.21.15.100:3300/0 conn(0x7f2a8406ac50 0x7f2a841585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.735+0000 7f2a889fe700 1 -- 172.21.15.98:0/1987098014 >> 172.21.15.98:0/1987098014 conn(0x7f2a84073810 msgr2=0x7f2a8406a330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.735+0000 7f2a889fe700 1 -- 172.21.15.98:0/1987098014 shutdown_connections 2023-03-03T07:55:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.735+0000 7fb4d0dff700 1 -- 172.21.15.98:0/4020553020 shutdown_connections 2023-03-03T07:55:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.735+0000 7fb4d0dff700 1 -- 172.21.15.98:0/4020553020 >> 172.21.15.98:0/4020553020 conn(0x7fb4cc0cb800 msgr2=0x7fb4cc071400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.735+0000 7f2a889fe700 1 -- 172.21.15.98:0/1987098014 wait complete. 2023-03-03T07:55:54.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.735+0000 7fb4d0dff700 1 -- 172.21.15.98:0/4020553020 shutdown_connections 2023-03-03T07:55:54.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.735+0000 7fb4d0dff700 1 -- 172.21.15.98:0/4020553020 wait complete. 2023-03-03T07:55:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.735+0000 7fd8b7fff700 1 -- 172.21.15.98:0/1196265000 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd8c8019030 con 0x7fd8d014a870 2023-03-03T07:55:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.735+0000 7fef513ff700 1 -- 172.21.15.98:0/1431471722 >> [v2:172.21.15.98:6824/109121,v1:172.21.15.98:6825/109121] conn(0x7fef38070c70 msgr2=0x7fef38073080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.735+0000 7fef513ff700 1 --2- 172.21.15.98:0/1431471722 >> [v2:172.21.15.98:6824/109121,v1:172.21.15.98:6825/109121] conn(0x7fef38070c70 0x7fef38073080 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.736+0000 7fef513ff700 1 -- 172.21.15.98:0/1431471722 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fef380683c0 msgr2=0x7fef3806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.736+0000 7fef513ff700 1 --2- 172.21.15.98:0/1431471722 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fef380683c0 0x7fef3806a870 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fef4c068f50 tx=0x7fef3c008040 comp rx=0 tx=0).stop 2023-03-03T07:55:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.736+0000 7fef513ff700 1 -- 172.21.15.98:0/1431471722 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef4c165290 msgr2=0x7fef4c162960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.736+0000 7fef513ff700 1 --2- 172.21.15.98:0/1431471722 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef4c165290 0x7fef4c162960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fef4400c700 tx=0x7fef4400cac0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.736+0000 7fef4b7fe700 1 -- 172.21.15.98:0/1431471722 reap_dead start 2023-03-03T07:55:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.736+0000 7fef513ff700 1 -- 172.21.15.98:0/1431471722 shutdown_connections 2023-03-03T07:55:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.736+0000 7fef513ff700 1 -- 172.21.15.98:0/1431471722 >> 172.21.15.98:0/1431471722 conn(0x7fef4c0cb800 msgr2=0x7fef4c071420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.737+0000 7fd8b7fff700 1 --2- 172.21.15.98:0/1196265000 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8bc0683c0 0x7fd8bc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.738+0000 7f26b83ff700 1 -- 172.21.15.98:0/1305760611 --> [v2:172.21.15.98:6800/109118,v1:172.21.15.98:6801/109118] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f26b00b3c30 con 0x7f2694070c70 2023-03-03T07:55:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.738+0000 7f2a889fe700 1 Processor -- start 2023-03-03T07:55:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.738+0000 7fd8b7fff700 1 -- 172.21.15.98:0/1196265000 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd8c80853d0 con 0x7fd8d014a870 2023-03-03T07:55:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.738+0000 7fd8b7fff700 1 --2- 172.21.15.98:0/1196265000 >> [v2:172.21.15.100:6809/109382,v1:172.21.15.100:6812/109382] conn(0x7fd8bc070bf0 0x7fd8bc073000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.738+0000 7fd8b7fff700 1 -- 172.21.15.98:0/1196265000 --> [v2:172.21.15.100:6809/109382,v1:172.21.15.100:6812/109382] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8bc0736b0 con 0x7fd8bc070bf0 2023-03-03T07:55:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.739+0000 7f26a6ffd700 1 -- 172.21.15.98:0/1305760611 <== osd.1 v2:172.21.15.98:6800/109118 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f26a0011790 con 0x7f2694070c70 2023-03-03T07:55:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.739+0000 7f26b83ff700 1 -- 172.21.15.98:0/1305760611 >> [v2:172.21.15.98:6800/109118,v1:172.21.15.98:6801/109118] conn(0x7f2694070c70 msgr2=0x7f2694073080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.739+0000 7f26b83ff700 1 --2- 172.21.15.98:0/1305760611 >> [v2:172.21.15.98:6800/109118,v1:172.21.15.98:6801/109118] conn(0x7f2694070c70 0x7f2694073080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.739+0000 7fd8ceffd700 1 --2- 172.21.15.98:0/1196265000 >> [v2:172.21.15.100:6809/109382,v1:172.21.15.100:6812/109382] conn(0x7fd8bc070bf0 0x7fd8bc073000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.739+0000 7fd8cf7fe700 1 --2- 172.21.15.98:0/1196265000 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8bc0683c0 0x7fd8bc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.740+0000 7fd8b7fff700 1 -- 172.21.15.98:0/1196265000 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8c8051bc0 con 0x7fd8d014a870 2023-03-03T07:55:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.740+0000 7f26b83ff700 1 -- 172.21.15.98:0/1305760611 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f26940683c0 msgr2=0x7f269406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.740+0000 7f26b83ff700 1 --2- 172.21.15.98:0/1305760611 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f26940683c0 0x7f269406a870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f26b0146f20 tx=0x7f26a800b040 comp rx=0 tx=0).stop 2023-03-03T07:55:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.740+0000 7f26b83ff700 1 -- 172.21.15.98:0/1305760611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26b00b0d40 msgr2=0x7f26b00b0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.740+0000 7f26b83ff700 1 --2- 172.21.15.98:0/1305760611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26b00b0d40 0x7f26b00b0750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f26ac00c450 tx=0x7f26ac00c810 comp rx=0 tx=0).stop 2023-03-03T07:55:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.740+0000 7f26b699c700 1 -- 172.21.15.98:0/1305760611 reap_dead start 2023-03-03T07:55:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.740+0000 7fd8ceffd700 1 --2- 172.21.15.98:0/1196265000 >> [v2:172.21.15.100:6809/109382,v1:172.21.15.100:6812/109382] conn(0x7fd8bc070bf0 0x7fd8bc073000 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.740+0000 7f26b83ff700 1 -- 172.21.15.98:0/1305760611 shutdown_connections 2023-03-03T07:55:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.740+0000 7f26b83ff700 1 -- 172.21.15.98:0/1305760611 >> 172.21.15.98:0/1305760611 conn(0x7f26b00cb800 msgr2=0x7f26b00cd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.741+0000 7fd8cf7fe700 1 --2- 172.21.15.98:0/1196265000 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8bc0683c0 0x7fd8bc06a870 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd8d015e520 tx=0x7fd8c4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.741+0000 7f26b83ff700 1 -- 172.21.15.98:0/1305760611 shutdown_connections 2023-03-03T07:55:54.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.741+0000 7fc3b6ffd700 1 -- 172.21.15.98:0/1611713710 <== osd.4 v2:172.21.15.100:6819/109381 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc3c00cd140 con 0x7fc3c00cc470 2023-03-03T07:55:54.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.739+0000 7f2a889fe700 1 -- start start 2023-03-03T07:55:54.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.740+0000 7f2a889fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a8406ac50 0x7f2a8412fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.740+0000 7f2a889fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a840cef80 0x7f2a840b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.740+0000 7f2a889fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a8413eba0 0x7f2a84158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.740+0000 7f2a889fe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a841539a0 con 0x7f2a840cef80 2023-03-03T07:55:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.741+0000 7f2a889fe700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a84153c80 con 0x7f2a8413eba0 2023-03-03T07:55:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.741+0000 7f2a889fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a84153f60 con 0x7f2a8406ac50 2023-03-03T07:55:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.741+0000 7fef513ff700 1 -- 172.21.15.98:0/1431471722 shutdown_connections 2023-03-03T07:55:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.741+0000 7fef513ff700 1 -- 172.21.15.98:0/1431471722 wait complete. 2023-03-03T07:55:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.741+0000 7f26b83ff700 1 -- 172.21.15.98:0/1305760611 wait complete. 2023-03-03T07:55:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.743+0000 7fd8b7fff700 1 -- 172.21.15.98:0/1196265000 <== osd.6 v2:172.21.15.100:6809/109382 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd8bc0736b0 con 0x7fd8bc070bf0 2023-03-03T07:55:54.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.742+0000 7f2a8259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a840cef80 0x7f2a840b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.742+0000 7f2a8259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a840cef80 0x7f2a840b0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45490/0 (socket says 172.21.15.98:45490) 2023-03-03T07:55:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.742+0000 7f2a8259c700 1 -- 172.21.15.98:0/3843828460 learned_addr learned my addr 172.21.15.98:0/3843828460 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.742+0000 7f2a82d9d700 1 --2- 172.21.15.98:0/3843828460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a8406ac50 0x7f2a8412fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.742+0000 7f2a8359e700 1 --2- 172.21.15.98:0/3843828460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a8413eba0 0x7f2a84158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.743+0000 7f2a82d9d700 1 -- 172.21.15.98:0/3843828460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a8413eba0 msgr2=0x7f2a84158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.743+0000 7f2a82d9d700 1 --2- 172.21.15.98:0/3843828460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a8413eba0 0x7f2a84158cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.743+0000 7f2a82d9d700 1 -- 172.21.15.98:0/3843828460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a840cef80 msgr2=0x7f2a840b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.743+0000 7f2a82d9d700 1 --2- 172.21.15.98:0/3843828460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a840cef80 0x7f2a840b0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.743+0000 7f2a82d9d700 1 -- 172.21.15.98:0/3843828460 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a740094a0 con 0x7f2a8406ac50 2023-03-03T07:55:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.743+0000 7f2a82d9d700 1 --2- 172.21.15.98:0/3843828460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a8406ac50 0x7f2a8412fe60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2a74016000 tx=0x7f2a74004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.744+0000 7f2a6b7fe700 1 -- 172.21.15.98:0/3843828460 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a74004510 con 0x7f2a8406ac50 2023-03-03T07:55:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.744+0000 7f2a6b7fe700 1 -- 172.21.15.98:0/3843828460 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7401f070 con 0x7f2a8406ac50 2023-03-03T07:55:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.744+0000 7f2a6b7fe700 1 -- 172.21.15.98:0/3843828460 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a740038e0 con 0x7f2a8406ac50 2023-03-03T07:55:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.744+0000 7f2a889fe700 1 -- 172.21.15.98:0/3843828460 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a8413e3a0 con 0x7f2a8406ac50 2023-03-03T07:55:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.745+0000 7f2a889fe700 1 -- 172.21.15.98:0/3843828460 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8413e0c0 con 0x7f2a8406ac50 2023-03-03T07:55:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.746+0000 7f2a889fe700 1 -- 172.21.15.98:0/3843828460 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2a840cf420 con 0x7f2a8406ac50 2023-03-03T07:55:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.749+0000 7f2a6b7fe700 1 -- 172.21.15.98:0/3843828460 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2a7400f050 con 0x7f2a8406ac50 2023-03-03T07:55:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.751+0000 7f2a6b7fe700 1 --2- 172.21.15.98:0/3843828460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2a700683c0 0x7f2a7006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.751+0000 7f2a6b7fe700 1 -- 172.21.15.98:0/3843828460 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2a74021070 con 0x7f2a8406ac50 2023-03-03T07:55:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.752+0000 7f2a6b7fe700 1 --2- 172.21.15.98:0/3843828460 >> [v2:172.21.15.100:6800/109380,v1:172.21.15.100:6801/109380] conn(0x7f2a70070c70 0x7f2a70073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.752+0000 7f2a6b7fe700 1 -- 172.21.15.98:0/3843828460 --> [v2:172.21.15.100:6800/109380,v1:172.21.15.100:6801/109380] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a70073730 con 0x7f2a70070c70 2023-03-03T07:55:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.752+0000 7f2a6b7fe700 1 -- 172.21.15.98:0/3843828460 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a7408b2a0 con 0x7f2a8406ac50 2023-03-03T07:55:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.753+0000 7f2a8259c700 1 --2- 172.21.15.98:0/3843828460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2a700683c0 0x7f2a7006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.753+0000 7f2a8359e700 1 --2- 172.21.15.98:0/3843828460 >> [v2:172.21.15.100:6800/109380,v1:172.21.15.100:6801/109380] conn(0x7f2a70070c70 0x7f2a70073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.754+0000 7f2a8259c700 1 --2- 172.21.15.98:0/3843828460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2a700683c0 0x7f2a7006a870 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2a78006fd0 tx=0x7f2a780092c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.758+0000 7f5fb71ff700 1 -- 172.21.15.98:0/1275426016 --> [v2:172.21.15.100:6808/109379,v1:172.21.15.100:6810/109379] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5fb00b2520 con 0x7f5f9c070c70 2023-03-03T07:55:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.758+0000 7f5fa57fa700 1 -- 172.21.15.98:0/1275426016 <== osd.5 v2:172.21.15.100:6808/109379 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5fb00b2520 con 0x7f5f9c070c70 2023-03-03T07:55:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.759+0000 7f5fb71ff700 1 -- 172.21.15.98:0/1275426016 >> [v2:172.21.15.100:6808/109379,v1:172.21.15.100:6810/109379] conn(0x7f5f9c070c70 msgr2=0x7f5f9c073080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.759+0000 7f5fb71ff700 1 --2- 172.21.15.98:0/1275426016 >> [v2:172.21.15.100:6808/109379,v1:172.21.15.100:6810/109379] conn(0x7f5f9c070c70 0x7f5f9c073080 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.759+0000 7f5fb71ff700 1 -- 172.21.15.98:0/1275426016 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5f9c0683c0 msgr2=0x7f5f9c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.759+0000 7f5fb71ff700 1 --2- 172.21.15.98:0/1275426016 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5f9c0683c0 0x7f5f9c06a870 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00c9840 tx=0x7f5fa0013000 comp rx=0 tx=0).stop 2023-03-03T07:55:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.759+0000 7f5fb71ff700 1 -- 172.21.15.98:0/1275426016 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fb006d460 msgr2=0x7f5fb01576c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.759+0000 7f5fb71ff700 1 --2- 172.21.15.98:0/1275426016 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fb006d460 0x7f5fb01576c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00c450 tx=0x7f5fac00c760 comp rx=0 tx=0).stop 2023-03-03T07:55:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.759+0000 7f5fb579c700 1 -- 172.21.15.98:0/1275426016 reap_dead start 2023-03-03T07:55:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.760+0000 7f5fb71ff700 1 -- 172.21.15.98:0/1275426016 shutdown_connections 2023-03-03T07:55:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.760+0000 7f5fb71ff700 1 -- 172.21.15.98:0/1275426016 >> 172.21.15.98:0/1275426016 conn(0x7f5fb00cb800 msgr2=0x7f5fb0156b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.760+0000 7f5fb71ff700 1 -- 172.21.15.98:0/1275426016 shutdown_connections 2023-03-03T07:55:54.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.760+0000 7f5fb71ff700 1 -- 172.21.15.98:0/1275426016 wait complete. 2023-03-03T07:55:54.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.764+0000 7f2a8359e700 1 --2- 172.21.15.98:0/3843828460 >> [v2:172.21.15.100:6800/109380,v1:172.21.15.100:6801/109380] conn(0x7f2a70070c70 0x7f2a70073080 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:54.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.765+0000 7f2a6b7fe700 1 -- 172.21.15.98:0/3843828460 <== osd.7 v2:172.21.15.100:6800/109380 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f2a70073730 con 0x7f2a70070c70 2023-03-03T07:55:54.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.771+0000 7f9ee23ff700 1 -- 172.21.15.98:0/4011822506 --> [v2:172.21.15.98:6816/109119,v1:172.21.15.98:6817/109119] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9edc04e6e0 con 0x7f9ec8070c70 2023-03-03T07:55:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.772+0000 7f9ed8ff9700 1 -- 172.21.15.98:0/4011822506 <== osd.3 v2:172.21.15.98:6816/109119 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9edc04e6e0 con 0x7f9ec8070c70 2023-03-03T07:55:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.773+0000 7f9ee23ff700 1 -- 172.21.15.98:0/4011822506 >> [v2:172.21.15.98:6816/109119,v1:172.21.15.98:6817/109119] conn(0x7f9ec8070c70 msgr2=0x7f9ec8073080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.773+0000 7f9ee23ff700 1 --2- 172.21.15.98:0/4011822506 >> [v2:172.21.15.98:6816/109119,v1:172.21.15.98:6817/109119] conn(0x7f9ec8070c70 0x7f9ec8073080 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.774+0000 7f9ee23ff700 1 -- 172.21.15.98:0/4011822506 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ec80683c0 msgr2=0x7f9ec806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.774+0000 7f9ee23ff700 1 --2- 172.21.15.98:0/4011822506 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ec80683c0 0x7f9ec806a870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0b5ce0 tx=0x7f9ecc006cd0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.774+0000 7f9ee23ff700 1 -- 172.21.15.98:0/4011822506 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9edc0b4bf0 msgr2=0x7f9edc0b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.774+0000 7f9ee23ff700 1 --2- 172.21.15.98:0/4011822506 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9edc0b4bf0 0x7f9edc0b5060 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00090b0 tx=0x7f9ed0004750 comp rx=0 tx=0).stop 2023-03-03T07:55:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.774+0000 7f9ee099c700 1 -- 172.21.15.98:0/4011822506 reap_dead start 2023-03-03T07:55:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.775+0000 7f9ee23ff700 1 -- 172.21.15.98:0/4011822506 shutdown_connections 2023-03-03T07:55:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.775+0000 7f9ee23ff700 1 -- 172.21.15.98:0/4011822506 >> 172.21.15.98:0/4011822506 conn(0x7f9edc073e70 msgr2=0x7f9edc0c99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.776+0000 7f9ee23ff700 1 -- 172.21.15.98:0/4011822506 shutdown_connections 2023-03-03T07:55:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.777+0000 7fc3c85ff700 1 -- 172.21.15.98:0/1611713710 --> [v2:172.21.15.100:6819/109381,v1:172.21.15.100:6823/109381] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc3c00ccfd0 con 0x7fc3c00cc470 2023-03-03T07:55:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.777+0000 7f9ee23ff700 1 -- 172.21.15.98:0/4011822506 wait complete. 2023-03-03T07:55:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.778+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1196265000 --> [v2:172.21.15.100:6809/109382,v1:172.21.15.100:6812/109382] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd8d004e6e0 con 0x7fd8bc070bf0 2023-03-03T07:55:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.779+0000 7fd8b7fff700 1 -- 172.21.15.98:0/1196265000 <== osd.6 v2:172.21.15.100:6809/109382 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd8d004e6e0 con 0x7fd8bc070bf0 2023-03-03T07:55:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.780+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1196265000 >> [v2:172.21.15.100:6809/109382,v1:172.21.15.100:6812/109382] conn(0x7fd8bc070bf0 msgr2=0x7fd8bc073000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.780+0000 7fd8d5fff700 1 --2- 172.21.15.98:0/1196265000 >> [v2:172.21.15.100:6809/109382,v1:172.21.15.100:6812/109382] conn(0x7fd8bc070bf0 0x7fd8bc073000 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.781+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1196265000 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8bc0683c0 msgr2=0x7fd8bc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.781+0000 7fd8d5fff700 1 --2- 172.21.15.98:0/1196265000 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8bc0683c0 0x7fd8bc06a870 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd8d015e520 tx=0x7fd8c4008040 comp rx=0 tx=0).stop 2023-03-03T07:55:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.781+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1196265000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d014a870 msgr2=0x7fd8d015d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.781+0000 7fd8d5fff700 1 --2- 172.21.15.98:0/1196265000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d014a870 0x7fd8d015d940 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800c870 tx=0x7fd8c800cc30 comp rx=0 tx=0).stop 2023-03-03T07:55:54.809 INFO:teuthology.orchestra.run.smithi098.stdout:55834574855 2023-03-03T07:55:54.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-03-03T07:55:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.781+0000 7fd8cffff700 1 -- 172.21.15.98:0/1196265000 reap_dead start 2023-03-03T07:55:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.781+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1196265000 shutdown_connections 2023-03-03T07:55:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.781+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1196265000 >> 172.21.15.98:0/1196265000 conn(0x7fd8d00cb800 msgr2=0x7fd8d00ced40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.782+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1196265000 shutdown_connections 2023-03-03T07:55:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.782+0000 7fd8d5fff700 1 -- 172.21.15.98:0/1196265000 wait complete. 2023-03-03T07:55:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.783+0000 7fc3b6ffd700 1 -- 172.21.15.98:0/1611713710 <== osd.4 v2:172.21.15.100:6819/109381 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc3c00ccfd0 con 0x7fc3c00cc470 2023-03-03T07:55:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.784+0000 7fc3c85ff700 1 -- 172.21.15.98:0/1611713710 >> [v2:172.21.15.100:6819/109381,v1:172.21.15.100:6823/109381] conn(0x7fc3c00cc470 msgr2=0x7fc3c00cc840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.784+0000 7fc3c85ff700 1 --2- 172.21.15.98:0/1611713710 >> [v2:172.21.15.100:6819/109381,v1:172.21.15.100:6823/109381] conn(0x7fc3c00cc470 0x7fc3c00cc840 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.813 INFO:teuthology.orchestra.run.smithi098.stdout:55834574856 2023-03-03T07:55:54.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-03-03T07:55:54.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.787+0000 7fc3c85ff700 1 -- 172.21.15.98:0/1611713710 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3a40683c0 msgr2=0x7fc3a406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.787+0000 7fc3c85ff700 1 --2- 172.21.15.98:0/1611713710 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3a40683c0 0x7fc3a406a870 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0009b40 tx=0x7fc3b0010000 comp rx=0 tx=0).stop 2023-03-03T07:55:54.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.787+0000 7fc3c85ff700 1 -- 172.21.15.98:0/1611713710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3c00d0270 msgr2=0x7fc3c00ca250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.787+0000 7fc3c85ff700 1 --2- 172.21.15.98:0/1611713710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3c00d0270 0x7fc3c00ca250 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc001f60 tx=0x7fc3bc00a340 comp rx=0 tx=0).stop 2023-03-03T07:55:54.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.787+0000 7fc3c6b9c700 1 -- 172.21.15.98:0/1611713710 reap_dead start 2023-03-03T07:55:54.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.787+0000 7fc3c85ff700 1 -- 172.21.15.98:0/1611713710 shutdown_connections 2023-03-03T07:55:54.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.787+0000 7fc3c85ff700 1 -- 172.21.15.98:0/1611713710 >> 172.21.15.98:0/1611713710 conn(0x7fc3c0073e70 msgr2=0x7fc3c00b7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.788+0000 7fc3c85ff700 1 -- 172.21.15.98:0/1611713710 shutdown_connections 2023-03-03T07:55:54.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.788+0000 7fc3c85ff700 1 -- 172.21.15.98:0/1611713710 wait complete. 2023-03-03T07:55:54.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.792+0000 7f2a889fe700 1 -- 172.21.15.98:0/3843828460 --> [v2:172.21.15.100:6800/109380,v1:172.21.15.100:6801/109380] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2a84068290 con 0x7f2a70070c70 2023-03-03T07:55:54.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.792+0000 7f2a6b7fe700 1 -- 172.21.15.98:0/3843828460 <== osd.7 v2:172.21.15.100:6800/109380 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2a84068290 con 0x7f2a70070c70 2023-03-03T07:55:54.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.797+0000 7f2a889fe700 1 -- 172.21.15.98:0/3843828460 >> [v2:172.21.15.100:6800/109380,v1:172.21.15.100:6801/109380] conn(0x7f2a70070c70 msgr2=0x7f2a70073080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.797+0000 7f2a889fe700 1 --2- 172.21.15.98:0/3843828460 >> [v2:172.21.15.100:6800/109380,v1:172.21.15.100:6801/109380] conn(0x7f2a70070c70 0x7f2a70073080 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.798+0000 7f2a889fe700 1 -- 172.21.15.98:0/3843828460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2a700683c0 msgr2=0x7f2a7006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.798+0000 7f2a889fe700 1 --2- 172.21.15.98:0/3843828460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2a700683c0 0x7f2a7006a870 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2a78006fd0 tx=0x7f2a780092c0 comp rx=0 tx=0).stop 2023-03-03T07:55:54.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.798+0000 7f2a889fe700 1 -- 172.21.15.98:0/3843828460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a8406ac50 msgr2=0x7f2a8412fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:54.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.798+0000 7f2a889fe700 1 --2- 172.21.15.98:0/3843828460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a8406ac50 0x7f2a8412fe60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2a74016000 tx=0x7f2a74004750 comp rx=0 tx=0).stop 2023-03-03T07:55:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.798+0000 7f2a8359e700 1 -- 172.21.15.98:0/3843828460 reap_dead start 2023-03-03T07:55:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.798+0000 7f2a889fe700 1 -- 172.21.15.98:0/3843828460 shutdown_connections 2023-03-03T07:55:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.798+0000 7f2a889fe700 1 -- 172.21.15.98:0/3843828460 >> 172.21.15.98:0/3843828460 conn(0x7f2a84073810 msgr2=0x7f2a8412e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.798+0000 7f2a889fe700 1 -- 172.21.15.98:0/3843828460 shutdown_connections 2023-03-03T07:55:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:54.798+0000 7f2a889fe700 1 -- 172.21.15.98:0/3843828460 wait complete. 2023-03-03T07:55:54.819 INFO:teuthology.orchestra.run.smithi098.stdout:55834574855 2023-03-03T07:55:54.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-03-03T07:55:54.820 INFO:teuthology.orchestra.run.smithi098.stdout:55834574854 2023-03-03T07:55:54.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-03-03T07:55:54.821 INFO:teuthology.orchestra.run.smithi098.stdout:55834574854 2023-03-03T07:55:54.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-03-03T07:55:54.823 INFO:teuthology.orchestra.run.smithi098.stdout:55834574856 2023-03-03T07:55:54.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-03-03T07:55:54.826 INFO:teuthology.orchestra.run.smithi098.stdout:55834574854 2023-03-03T07:55:54.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-03-03T07:55:54.838 INFO:teuthology.orchestra.run.smithi098.stdout:55834574855 2023-03-03T07:55:54.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-03-03T07:55:55.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.265+0000 7f4db9bff700 1 Processor -- start 2023-03-03T07:55:55.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.265+0000 7f4db9bff700 1 -- start start 2023-03-03T07:55:55.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.267+0000 7f4db9bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4db40ce910 0x7f4db406b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.267+0000 7f4db9bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db416d7b0 con 0x7f4db416dc40 2023-03-03T07:55:55.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.267+0000 7f4db9bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db406d4d0 con 0x7f4db406d100 2023-03-03T07:55:55.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.267+0000 7f4db9bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db40cae80 con 0x7f4db40ce910 2023-03-03T07:55:55.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.269+0000 7f4db37fe700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f4db406d100 0x7f4db40b0bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:47776/0 (socket says 172.21.15.98:47776) 2023-03-03T07:55:55.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.269+0000 7f4db37fe700 1 -- 172.21.15.98:0/1480810201 learned_addr learned my addr 172.21.15.98:0/1480810201 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:55.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.270+0000 7f4db27fc700 1 -- 172.21.15.98:0/1480810201 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422513517 0 0) 0x7f4db406d4d0 con 0x7f4db406d100 2023-03-03T07:55:55.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.271+0000 7f4db27fc700 1 -- 172.21.15.98:0/1480810201 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d9c0035b0 con 0x7f4db406d100 2023-03-03T07:55:55.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.271+0000 7f4db27fc700 1 -- 172.21.15.98:0/1480810201 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3416591108 0 0) 0x7f4d9c0035b0 con 0x7f4db406d100 2023-03-03T07:55:55.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.271+0000 7f4db27fc700 1 -- 172.21.15.98:0/1480810201 >> v2:172.21.15.98:3300/0 conn(0x7f4db40ce910 msgr2=0x7f4db406b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:55.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.271+0000 7f4db27fc700 1 --2- 172.21.15.98:0/1480810201 >> v2:172.21.15.98:3300/0 conn(0x7f4db40ce910 0x7f4db406b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.271+0000 7f4db27fc700 1 -- 172.21.15.98:0/1480810201 >> v1:172.21.15.98:6789/0 conn(0x7f4db416dc40 legacy=0x7f4db40ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:55.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.271+0000 7f4db27fc700 1 -- 172.21.15.98:0/1480810201 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db40cb160 con 0x7f4db406d100 2023-03-03T07:55:55.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.271+0000 7f4db27fc700 1 -- 172.21.15.98:0/1480810201 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f4da4001670 con 0x7f4db406d100 2023-03-03T07:55:55.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.272+0000 7f4db27fc700 1 -- 172.21.15.98:0/1480810201 >> v1:172.21.15.100:6789/0 conn(0x7f4db406d100 legacy=0x7f4db40b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.272+0000 7f4db27fc700 1 --2- 172.21.15.98:0/1480810201 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d9c003a00 0x7f4d9c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.272+0000 7f4db27fc700 1 --2- 172.21.15.98:0/1480810201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c004850 0x7f4d9c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.272+0000 7f4db27fc700 1 --2- 172.21.15.98:0/1480810201 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c005320 0x7f4d9c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.272+0000 7f4db27fc700 1 -- 172.21.15.98:0/1480810201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db40cae80 con 0x7f4d9c004850 2023-03-03T07:55:55.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.272+0000 7f4db27fc700 1 -- 172.21.15.98:0/1480810201 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c00c1e0 con 0x7f4d9c003a00 2023-03-03T07:55:55.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.272+0000 7f4db27fc700 1 -- 172.21.15.98:0/1480810201 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c00c3a0 con 0x7f4d9c005320 2023-03-03T07:55:55.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.273+0000 7f4db3fff700 1 --2- 172.21.15.98:0/1480810201 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c005320 0x7f4d9c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.273+0000 7f4db2ffd700 1 --2- 172.21.15.98:0/1480810201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c004850 0x7f4d9c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.274+0000 7f4db3fff700 1 -- 172.21.15.98:0/1480810201 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d9c003a00 msgr2=0x7f4d9c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:55.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.274+0000 7f4db3fff700 1 --2- 172.21.15.98:0/1480810201 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d9c003a00 0x7f4d9c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.274+0000 7f4db3fff700 1 -- 172.21.15.98:0/1480810201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c004850 msgr2=0x7f4d9c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:55.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.274+0000 7f4db3fff700 1 --2- 172.21.15.98:0/1480810201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c004850 0x7f4d9c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.274+0000 7f4db3fff700 1 -- 172.21.15.98:0/1480810201 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db40cb160 con 0x7f4d9c005320 2023-03-03T07:55:55.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.274+0000 7f4db2ffd700 1 --2- 172.21.15.98:0/1480810201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c004850 0x7f4d9c004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:55.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.274+0000 7f4db3fff700 1 -- 172.21.15.98:0/1480810201 reap_dead start 2023-03-03T07:55:55.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.274+0000 7f4db3fff700 1 --2- 172.21.15.98:0/1480810201 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c005320 0x7f4d9c00ba10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4db406c950 tx=0x7f4d9c005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=674ef73545554688 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.275+0000 7f4db27fc700 1 -- 172.21.15.98:0/1480810201 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac0098b0 con 0x7f4d9c005320 2023-03-03T07:55:55.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.275+0000 7f4db27fc700 1 -- 172.21.15.98:0/1480810201 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dac003c20 con 0x7f4d9c005320 2023-03-03T07:55:55.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.275+0000 7f4db27fc700 1 -- 172.21.15.98:0/1480810201 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac00b8a0 con 0x7f4d9c005320 2023-03-03T07:55:55.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.276+0000 7f9dac9ff700 1 Processor -- start 2023-03-03T07:55:55.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.276+0000 7f4db9bff700 1 -- 172.21.15.98:0/1480810201 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c005320 msgr2=0x7f4d9c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.276+0000 7f4db9bff700 1 --2- 172.21.15.98:0/1480810201 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c005320 0x7f4d9c00ba10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4db406c950 tx=0x7f4d9c005880 comp rx=0 tx=0).stop 2023-03-03T07:55:55.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.276+0000 7f4db9bff700 1 -- 172.21.15.98:0/1480810201 shutdown_connections 2023-03-03T07:55:55.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.276+0000 7f4db9bff700 1 --2- 172.21.15.98:0/1480810201 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c005320 0x7f4d9c00ba10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.276+0000 7f4db9bff700 1 -- 172.21.15.98:0/1480810201 >> 172.21.15.98:0/1480810201 conn(0x7f4db40cb800 msgr2=0x7f4db40cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.276+0000 7f9dac9ff700 1 -- start start 2023-03-03T07:55:55.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.279+0000 7f9dac9ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9da80b0f10 0x7f9da806c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.279+0000 7f9dac9ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9da8164620 0x7f9da8164b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.279+0000 7f9dac9ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da8072170 con 0x7f9da8164250 2023-03-03T07:55:55.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.279+0000 7f9dac9ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da80cb5b0 con 0x7f9da8164620 2023-03-03T07:55:55.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.279+0000 7f9dac9ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da80c2440 con 0x7f9da80b0f10 2023-03-03T07:55:55.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.279+0000 7f9da5d9b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9da80b0f10 0x7f9da806c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.279+0000 7f9da5d9b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9da80b0f10 0x7f9da806c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:46912/0 (socket says 172.21.15.98:46912) 2023-03-03T07:55:55.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.279+0000 7f9da5d9b700 1 -- 172.21.15.98:0/2793536992 learned_addr learned my addr 172.21.15.98:0/2793536992 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:55.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.280+0000 7f9da5d9b700 1 -- 172.21.15.98:0/2793536992 >> v2:172.21.15.98:3300/0 conn(0x7f9da8164620 msgr2=0x7f9da8164b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:55.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.280+0000 7f9da5d9b700 1 --2- 172.21.15.98:0/2793536992 >> v2:172.21.15.98:3300/0 conn(0x7f9da8164620 0x7f9da8164b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.280+0000 7f9da5d9b700 1 -- 172.21.15.98:0/2793536992 >> v1:172.21.15.100:6789/0 conn(0x7f9da8164250 legacy=0x7f9da80b0800 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-03T07:55:55.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.280+0000 7f9da5d9b700 1 -- 172.21.15.98:0/2793536992 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da8071e90 con 0x7f9da80b0f10 2023-03-03T07:55:55.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.280+0000 7f9da5d9b700 1 --2- 172.21.15.98:0/2793536992 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9da80b0f10 0x7f9da806c8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9da80cc710 tx=0x7f9d98009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d633aaf65b77716 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.281+0000 7f4db9bff700 1 -- 172.21.15.98:0/1480810201 shutdown_connections 2023-03-03T07:55:55.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.281+0000 7f9da559a700 1 -- 172.21.15.98:0/2793536992 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d98004230 con 0x7f9da80b0f10 2023-03-03T07:55:55.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.281+0000 7f9da559a700 1 -- 172.21.15.98:0/2793536992 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d98004390 con 0x7f9da80b0f10 2023-03-03T07:55:55.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.281+0000 7f9da559a700 1 -- 172.21.15.98:0/2793536992 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d980036a0 con 0x7f9da80b0f10 2023-03-03T07:55:55.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.281+0000 7f4db9bff700 1 -- 172.21.15.98:0/1480810201 wait complete. 2023-03-03T07:55:55.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.282+0000 7f9dac9ff700 1 -- 172.21.15.98:0/2793536992 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9da80b0f10 msgr2=0x7f9da806c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.282+0000 7f9dac9ff700 1 --2- 172.21.15.98:0/2793536992 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9da80b0f10 0x7f9da806c8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9da80cc710 tx=0x7f9d98009bd0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.285+0000 7f4db9bff700 1 Processor -- start 2023-03-03T07:55:55.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.285+0000 7f4db9bff700 1 -- start start 2023-03-03T07:55:55.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.286+0000 7f4db9bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d9c005320 0x7f4db416c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.286+0000 7f4db9bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4db4070040 0x7f4db40cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.287+0000 7f4db9bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4db40704b0 0x7f4db4169e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.287+0000 7f4db9bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db416bfd0 con 0x7f4db40704b0 2023-03-03T07:55:55.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.287+0000 7f4db9bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db416b400 con 0x7f4d9c005320 2023-03-03T07:55:55.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.287+0000 7f4db9bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db416b6a0 con 0x7f4db4070040 2023-03-03T07:55:55.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.287+0000 7f9dac9ff700 1 -- 172.21.15.98:0/2793536992 shutdown_connections 2023-03-03T07:55:55.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.287+0000 7f9dac9ff700 1 --2- 172.21.15.98:0/2793536992 >> v2:172.21.15.98:3300/0 conn(0x7f9da8164620 0x7f9da8164b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.287+0000 7f9dac9ff700 1 --2- 172.21.15.98:0/2793536992 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9da80b0f10 0x7f9da806c8e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.287+0000 7f9dac9ff700 1 -- 172.21.15.98:0/2793536992 >> 172.21.15.98:0/2793536992 conn(0x7f9da80cb800 msgr2=0x7f9da80cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.288+0000 7f4db2ffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4db4070040 0x7f4db40cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.288+0000 7f4db2ffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4db4070040 0x7f4db40cce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:60886/0 (socket says 172.21.15.98:60886) 2023-03-03T07:55:55.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.288+0000 7f4db2ffd700 1 -- 172.21.15.98:0/1604298407 learned_addr learned my addr 172.21.15.98:0/1604298407 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:55.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.288+0000 7f4db3fff700 1 --2- 172.21.15.98:0/1604298407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4db40704b0 0x7f4db4169e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.289+0000 7f4db3fff700 1 -- 172.21.15.98:0/1604298407 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4db4070040 msgr2=0x7f4db40cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.289+0000 7f4db3fff700 1 --2- 172.21.15.98:0/1604298407 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4db4070040 0x7f4db40cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.289+0000 7f4db3fff700 1 -- 172.21.15.98:0/1604298407 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d9c005320 msgr2=0x7f4db416c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.289+0000 7f4db3fff700 1 --2- 172.21.15.98:0/1604298407 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d9c005320 0x7f4db416c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.289+0000 7f4db3fff700 1 -- 172.21.15.98:0/1604298407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db4169c90 con 0x7f4db40704b0 2023-03-03T07:55:55.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.290+0000 7f9dac9ff700 1 -- 172.21.15.98:0/2793536992 shutdown_connections 2023-03-03T07:55:55.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.290+0000 7f14a15ff700 1 Processor -- start 2023-03-03T07:55:55.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.290+0000 7f14a15ff700 1 -- start start 2023-03-03T07:55:55.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.292+0000 7f4db3fff700 1 --2- 172.21.15.98:0/1604298407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4db40704b0 0x7f4db4169e20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4db406fb30 tx=0x7f4dac004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.289+0000 7f6e3159d700 1 Processor -- start 2023-03-03T07:55:55.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.290+0000 7f9dac9ff700 1 -- 172.21.15.98:0/2793536992 wait complete. 2023-03-03T07:55:55.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.293+0000 7f9dac9ff700 1 Processor -- start 2023-03-03T07:55:55.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.294+0000 7f9dac9ff700 1 -- start start 2023-03-03T07:55:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.293+0000 7f4d9bfff700 1 -- 172.21.15.98:0/1604298407 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac0041a0 con 0x7f4db40704b0 2023-03-03T07:55:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.293+0000 7f4d9bfff700 1 -- 172.21.15.98:0/1604298407 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dac004320 con 0x7f4db40704b0 2023-03-03T07:55:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.293+0000 7f4d9bfff700 1 -- 172.21.15.98:0/1604298407 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac0046d0 con 0x7f4db40704b0 2023-03-03T07:55:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.293+0000 7f4db9bff700 1 -- 172.21.15.98:0/1604298407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db40cb440 con 0x7f4db40704b0 2023-03-03T07:55:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.293+0000 7f4db9bff700 1 -- 172.21.15.98:0/1604298407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db406cb50 con 0x7f4db40704b0 2023-03-03T07:55:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.293+0000 7f8a519ff700 1 Processor -- start 2023-03-03T07:55:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.293+0000 7f8a519ff700 1 -- start start 2023-03-03T07:55:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.290+0000 7f6e3159d700 1 -- start start 2023-03-03T07:55:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.293+0000 7f6e3159d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6e2c157a60 0x7f6e2c165010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.293+0000 7f6e3159d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6e2c06de90 0x7f6e2c0ce050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.293+0000 7f6e3159d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c161b50 con 0x7f6e2c06bd20 2023-03-03T07:55:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.293+0000 7f6e3159d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e2c06c2c0 con 0x7f6e2c06de90 2023-03-03T07:55:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.293+0000 7f6e3159d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e2c165550 con 0x7f6e2c157a60 2023-03-03T07:55:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.295+0000 7f9dac9ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9da80b0f10 0x7f9da8156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.295+0000 7f9dac9ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9da8164250 0x7f9da80c11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.295+0000 7f9dac9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da8164620 0x7f9da8161350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.295+0000 7f9dac9ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da8144550 con 0x7f9da80b0f10 2023-03-03T07:55:55.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.295+0000 7f9dac9ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da81447f0 con 0x7f9da8164250 2023-03-03T07:55:55.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.295+0000 7f9dac9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da8144ad0 con 0x7f9da8164620 2023-03-03T07:55:55.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.296+0000 7f8a519ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a4c0cc310 0x7f8a4c158c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.296+0000 7f8a519ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4c0ce080 con 0x7f8a4c06b820 2023-03-03T07:55:55.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.296+0000 7f8a519ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4c06c450 con 0x7f8a4c06d460 2023-03-03T07:55:55.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.296+0000 7f8a519ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c06c730 con 0x7f8a4c0cc310 2023-03-03T07:55:55.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.296+0000 7f9da6d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da8164620 0x7f9da8161350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.296+0000 7f9da6d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da8164620 0x7f9da8161350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:60894/0 (socket says 172.21.15.98:60894) 2023-03-03T07:55:55.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.296+0000 7f9da6d9d700 1 -- 172.21.15.98:0/3240270850 learned_addr learned my addr 172.21.15.98:0/3240270850 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:55.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.296+0000 7f4db9bff700 1 -- 172.21.15.98:0/1604298407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d940052f0 con 0x7f4db40704b0 2023-03-03T07:55:55.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.296+0000 7f8a4a7fc700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f8a4c06d460 0x7f8a4c0ce510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:47802/0 (socket says 172.21.15.98:47802) 2023-03-03T07:55:55.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.297+0000 7f8a4a7fc700 1 -- 172.21.15.98:0/1534967926 learned_addr learned my addr 172.21.15.98:0/1534967926 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:55.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.297+0000 7f8a4affd700 1 --2- 172.21.15.98:0/1534967926 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a4c0cc310 0x7f8a4c158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.297+0000 7f14a15ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f149c0b0ef0 0x7f149c15c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.297+0000 7f14a15ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f149c06d080 0x7f149c1589e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.297+0000 7f6e30d9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6e2c06de90 0x7f6e2c0ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.297+0000 7f6e30d9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6e2c06de90 0x7f6e2c0ce050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41580/0 (socket says 172.21.15.98:41580) 2023-03-03T07:55:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.297+0000 7f6e30d9c700 1 -- 172.21.15.98:0/424862114 learned_addr learned my addr 172.21.15.98:0/424862114 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.297+0000 7f14a15ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f149c158f20 con 0x7f149c0cc2f0 2023-03-03T07:55:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.297+0000 7f14a15ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f149c06d520 con 0x7f149c0b0ef0 2023-03-03T07:55:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.297+0000 7f14a15ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149c15d4c0 con 0x7f149c06d080 2023-03-03T07:55:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.298+0000 7f8a4affd700 1 -- 172.21.15.98:0/1534967926 >> v1:172.21.15.100:6789/0 conn(0x7f8a4c06d460 legacy=0x7f8a4c0ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.298+0000 7f8a4affd700 1 -- 172.21.15.98:0/1534967926 >> v1:172.21.15.98:6789/0 conn(0x7f8a4c06b820 legacy=0x7f8a4c0b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.298+0000 7f8a4affd700 1 -- 172.21.15.98:0/1534967926 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c0cdd10 con 0x7f8a4c0cc310 2023-03-03T07:55:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.298+0000 7f6e2bfff700 1 --2- 172.21.15.98:0/424862114 >> v2:172.21.15.100:3300/0 conn(0x7f6e2c157a60 0x7f6e2c165010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.298+0000 7f149affd700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f149c0cc2f0 0x7f149c0b07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:47808/0 (socket says 172.21.15.98:47808) 2023-03-03T07:55:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.298+0000 7f149affd700 1 -- 172.21.15.98:0/555771206 learned_addr learned my addr 172.21.15.98:0/555771206 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.298+0000 7f149a7fc700 1 --2- 172.21.15.98:0/555771206 >> v2:172.21.15.100:3300/0 conn(0x7f149c0b0ef0 0x7f149c15c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f149b7fe700 1 --2- 172.21.15.98:0/555771206 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f149c06d080 0x7f149c1589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f1499ffb700 1 -- 172.21.15.98:0/555771206 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220704963 0 0) 0x7f149c158f20 con 0x7f149c0cc2f0 2023-03-03T07:55:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f1499ffb700 1 -- 172.21.15.98:0/555771206 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14840035b0 con 0x7f149c0cc2f0 2023-03-03T07:55:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f149a7fc700 1 -- 172.21.15.98:0/555771206 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f149c06d080 msgr2=0x7f149c1589e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f149a7fc700 1 --2- 172.21.15.98:0/555771206 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f149c06d080 0x7f149c1589e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f149a7fc700 1 -- 172.21.15.98:0/555771206 >> v1:172.21.15.100:6789/0 conn(0x7f149c0cc2f0 legacy=0x7f149c0b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f149a7fc700 1 -- 172.21.15.98:0/555771206 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149c15cc20 con 0x7f149c0b0ef0 2023-03-03T07:55:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f725e1ff700 1 Processor -- start 2023-03-03T07:55:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f8a4affd700 1 --2- 172.21.15.98:0/1534967926 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a4c0cc310 0x7f8a4c158c00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009870 tx=0x7f8a3c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7500380beec017ba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f8a49ffb700 1 -- 172.21.15.98:0/1534967926 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c004230 con 0x7f8a4c0cc310 2023-03-03T07:55:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f8a49ffb700 1 -- 172.21.15.98:0/1534967926 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c004390 con 0x7f8a4c0cc310 2023-03-03T07:55:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f8a49ffb700 1 -- 172.21.15.98:0/1534967926 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c0036a0 con 0x7f8a4c0cc310 2023-03-03T07:55:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f6e2affd700 1 -- 172.21.15.98:0/424862114 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727150271 0 0) 0x7f6e2c161b50 con 0x7f6e2c06bd20 2023-03-03T07:55:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f6e2affd700 1 -- 172.21.15.98:0/424862114 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e180035b0 con 0x7f6e2c06bd20 2023-03-03T07:55:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f6e30d9c700 1 -- 172.21.15.98:0/424862114 >> v2:172.21.15.100:3300/0 conn(0x7f6e2c157a60 msgr2=0x7f6e2c165010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f6e30d9c700 1 --2- 172.21.15.98:0/424862114 >> v2:172.21.15.100:3300/0 conn(0x7f6e2c157a60 0x7f6e2c165010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f6e30d9c700 1 -- 172.21.15.98:0/424862114 >> v1:172.21.15.98:6789/0 conn(0x7f6e2c06bd20 legacy=0x7f6e2c06d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f6e30d9c700 1 -- 172.21.15.98:0/424862114 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2c161220 con 0x7f6e2c06de90 2023-03-03T07:55:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.300+0000 7f9da6d9d700 1 -- 172.21.15.98:0/3240270850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9da8164250 msgr2=0x7f9da80c11f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T07:55:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.300+0000 7f9da6d9d700 1 --2- 172.21.15.98:0/3240270850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9da8164250 0x7f9da80c11f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.300+0000 7f9da6d9d700 1 -- 172.21.15.98:0/3240270850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9da80b0f10 msgr2=0x7f9da8156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.300+0000 7f9da6d9d700 1 --2- 172.21.15.98:0/3240270850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9da80b0f10 0x7f9da8156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.300+0000 7f9da6d9d700 1 -- 172.21.15.98:0/3240270850 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d98009550 con 0x7f9da8164620 2023-03-03T07:55:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.300+0000 7f149a7fc700 1 --2- 172.21.15.98:0/555771206 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f149c0b0ef0 0x7f149c15c5d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f148c009840 tx=0x7f148c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ba22a47d88464fe server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.300+0000 7f6e30d9c700 1 --2- 172.21.15.98:0/424862114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e2c06de90 0x7f6e2c0ce050 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c06b180 tx=0x7f6e24009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202405913943732e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.301+0000 7f4d9bfff700 1 -- 172.21.15.98:0/1604298407 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4dac027030 con 0x7f4db40704b0 2023-03-03T07:55:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.301+0000 7f1499ffb700 1 -- 172.21.15.98:0/555771206 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c011830 con 0x7f149c0b0ef0 2023-03-03T07:55:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.301+0000 7f1499ffb700 1 -- 172.21.15.98:0/555771206 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148c011990 con 0x7f149c0b0ef0 2023-03-03T07:55:55.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.301+0000 7f1499ffb700 1 -- 172.21.15.98:0/555771206 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c011d20 con 0x7f149c0b0ef0 2023-03-03T07:55:55.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f14a15ff700 1 -- 172.21.15.98:0/555771206 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f149c0b0ef0 msgr2=0x7f149c15c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f14a15ff700 1 --2- 172.21.15.98:0/555771206 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f149c0b0ef0 0x7f149c15c5d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f148c009840 tx=0x7f148c009b50 comp rx=0 tx=0).stop 2023-03-03T07:55:55.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f14a15ff700 1 -- 172.21.15.98:0/555771206 shutdown_connections 2023-03-03T07:55:55.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f14a15ff700 1 --2- 172.21.15.98:0/555771206 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f149c06d080 0x7f149c1589e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f14a15ff700 1 --2- 172.21.15.98:0/555771206 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f149c0b0ef0 0x7f149c15c5d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f14a15ff700 1 -- 172.21.15.98:0/555771206 >> 172.21.15.98:0/555771206 conn(0x7f149c0cb7e0 msgr2=0x7f149c0cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.301+0000 7f6e2affd700 1 -- 172.21.15.98:0/424862114 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e24011830 con 0x7f6e2c06de90 2023-03-03T07:55:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.301+0000 7f6e2affd700 1 -- 172.21.15.98:0/424862114 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e24011990 con 0x7f6e2c06de90 2023-03-03T07:55:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.301+0000 7f6e2affd700 1 -- 172.21.15.98:0/424862114 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e24011d00 con 0x7f6e2c06de90 2023-03-03T07:55:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f6e3159d700 1 -- 172.21.15.98:0/424862114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e2c06de90 msgr2=0x7f6e2c0ce050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f6e3159d700 1 --2- 172.21.15.98:0/424862114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e2c06de90 0x7f6e2c0ce050 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c06b180 tx=0x7f6e24009bd0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f14a15ff700 1 -- 172.21.15.98:0/555771206 shutdown_connections 2023-03-03T07:55:55.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f14a15ff700 1 -- 172.21.15.98:0/555771206 wait complete. 2023-03-03T07:55:55.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.299+0000 7f725e1ff700 1 -- start start 2023-03-03T07:55:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f725e1ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f72580b26d0 0x7f72580b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f725e1ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f72580b5140 0x7f72580d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f725e1ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72580cd7d0 con 0x7f72580b8d10 2023-03-03T07:55:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f725e1ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72580b55c0 con 0x7f72580b26d0 2023-03-03T07:55:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f725e1ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72580b90e0 con 0x7f72580b5140 2023-03-03T07:55:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.303+0000 7f6e3159d700 1 -- 172.21.15.98:0/424862114 shutdown_connections 2023-03-03T07:55:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.303+0000 7f6e3159d700 1 --2- 172.21.15.98:0/424862114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e2c06de90 0x7f6e2c0ce050 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.303+0000 7f6e3159d700 1 --2- 172.21.15.98:0/424862114 >> v2:172.21.15.100:3300/0 conn(0x7f6e2c157a60 0x7f6e2c165010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.303+0000 7f6e3159d700 1 -- 172.21.15.98:0/424862114 >> 172.21.15.98:0/424862114 conn(0x7f6e2c073810 msgr2=0x7f6e2c06a330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.303+0000 7f4d9bfff700 1 --2- 172.21.15.98:0/1604298407 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4da00683c0 0x7f4da006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.303+0000 7f4d9bfff700 1 -- 172.21.15.98:0/1604298407 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4dac011070 con 0x7f4db40704b0 2023-03-03T07:55:55.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.303+0000 7f72577fe700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f72580b8d10 0x7f72580cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:47820/0 (socket says 172.21.15.98:47820) 2023-03-03T07:55:55.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.303+0000 7f72577fe700 1 -- 172.21.15.98:0/2016747713 learned_addr learned my addr 172.21.15.98:0/2016747713 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:55.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.303+0000 7f7257fff700 1 --2- 172.21.15.98:0/2016747713 >> v2:172.21.15.100:3300/0 conn(0x7f72580b5140 0x7f72580d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f8a519ff700 1 -- 172.21.15.98:0/1534967926 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a4c0cc310 msgr2=0x7f8a4c158c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.302+0000 7f8a519ff700 1 --2- 172.21.15.98:0/1534967926 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a4c0cc310 0x7f8a4c158c00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009870 tx=0x7f8a3c009b80 comp rx=0 tx=0).stop 2023-03-03T07:55:55.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.303+0000 7f9da6d9d700 1 --2- 172.21.15.98:0/3240270850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da8164620 0x7f9da8161350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9da000b9e0 tx=0x7f9da000bda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.304+0000 7f7256ffd700 1 --2- 172.21.15.98:0/2016747713 >> v2:172.21.15.98:3300/0 conn(0x7f72580b26d0 0x7f72580b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.304+0000 7f8a519ff700 1 -- 172.21.15.98:0/1534967926 shutdown_connections 2023-03-03T07:55:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.304+0000 7f8a519ff700 1 --2- 172.21.15.98:0/1534967926 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a4c0cc310 0x7f8a4c158c00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.304+0000 7f8a519ff700 1 -- 172.21.15.98:0/1534967926 >> 172.21.15.98:0/1534967926 conn(0x7f8a4c0cb800 msgr2=0x7f8a4c0cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.304+0000 7f8a519ff700 1 -- 172.21.15.98:0/1534967926 shutdown_connections 2023-03-03T07:55:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.304+0000 7f8a519ff700 1 -- 172.21.15.98:0/1534967926 wait complete. 2023-03-03T07:55:55.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.304+0000 7f9d96ffd700 1 -- 172.21.15.98:0/3240270850 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da000c9b0 con 0x7f9da8164620 2023-03-03T07:55:55.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.304+0000 7f9d96ffd700 1 -- 172.21.15.98:0/3240270850 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da000cb10 con 0x7f9da8164620 2023-03-03T07:55:55.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.304+0000 7f9d96ffd700 1 -- 172.21.15.98:0/3240270850 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da0014440 con 0x7f9da8164620 2023-03-03T07:55:55.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.304+0000 7f9dac9ff700 1 -- 172.21.15.98:0/3240270850 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da80cadb0 con 0x7f9da8164620 2023-03-03T07:55:55.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.305+0000 7f9dac9ff700 1 -- 172.21.15.98:0/3240270850 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da80caad0 con 0x7f9da8164620 2023-03-03T07:55:55.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.305+0000 7f7257fff700 1 -- 172.21.15.98:0/2016747713 >> v2:172.21.15.98:3300/0 conn(0x7f72580b26d0 msgr2=0x7f72580b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:55.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.305+0000 7f7257fff700 1 --2- 172.21.15.98:0/2016747713 >> v2:172.21.15.98:3300/0 conn(0x7f72580b26d0 0x7f72580b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.305+0000 7f7257fff700 1 -- 172.21.15.98:0/2016747713 >> v1:172.21.15.100:6789/0 conn(0x7f72580b8d10 legacy=0x7f72580cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.305+0000 7f7257fff700 1 -- 172.21.15.98:0/2016747713 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72580cda20 con 0x7f72580b5140 2023-03-03T07:55:55.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.305+0000 7f72567fc700 1 -- 172.21.15.98:0/2016747713 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962973564 0 0) 0x7f72580cd7d0 con 0x7f72580b8d10 2023-03-03T07:55:55.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.306+0000 7f7257fff700 1 --2- 172.21.15.98:0/2016747713 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72580b5140 0x7f72580d13a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f72580ca1e0 tx=0x7f725000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=366d2c6919494c7a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.308+0000 7f9dac9ff700 1 -- 172.21.15.98:0/3240270850 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8c0052f0 con 0x7f9da8164620 2023-03-03T07:55:55.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.306+0000 7f14a15ff700 1 Processor -- start 2023-03-03T07:55:55.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.307+0000 7f14a15ff700 1 -- start start 2023-03-03T07:55:55.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.308+0000 7f14a15ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f149c06d080 0x7f149c0c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.308+0000 7f14a15ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f149c0b0ef0 0x7f149c202ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.308+0000 7f14a15ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f149c0cc2f0 0x7f149c203010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.308+0000 7f14a15ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149c154820 con 0x7f149c0cc2f0 2023-03-03T07:55:55.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.308+0000 7f14a15ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149c0c37e0 con 0x7f149c0b0ef0 2023-03-03T07:55:55.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.308+0000 7f14a15ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149c0c4070 con 0x7f149c06d080 2023-03-03T07:55:55.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.307+0000 7f72567fc700 1 -- 172.21.15.98:0/2016747713 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725000e070 con 0x7f72580b5140 2023-03-03T07:55:55.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.307+0000 7f72567fc700 1 -- 172.21.15.98:0/2016747713 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725000bb20 con 0x7f72580b5140 2023-03-03T07:55:55.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.308+0000 7f72567fc700 1 -- 172.21.15.98:0/2016747713 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7250012430 con 0x7f72580b5140 2023-03-03T07:55:55.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.308+0000 7f725e1ff700 1 -- 172.21.15.98:0/2016747713 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72580b5140 msgr2=0x7f72580d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.308+0000 7f725e1ff700 1 --2- 172.21.15.98:0/2016747713 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72580b5140 0x7f72580d13a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f72580ca1e0 tx=0x7f725000b330 comp rx=0 tx=0).stop 2023-03-03T07:55:55.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.309+0000 7f725e1ff700 1 -- 172.21.15.98:0/2016747713 shutdown_connections 2023-03-03T07:55:55.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.309+0000 7f725e1ff700 1 --2- 172.21.15.98:0/2016747713 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72580b5140 0x7f72580d13a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.309+0000 7f725e1ff700 1 --2- 172.21.15.98:0/2016747713 >> v2:172.21.15.98:3300/0 conn(0x7f72580b26d0 0x7f72580b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.309+0000 7f725e1ff700 1 -- 172.21.15.98:0/2016747713 >> 172.21.15.98:0/2016747713 conn(0x7f7258073e70 msgr2=0x7f725806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.307+0000 7f8a519ff700 1 Processor -- start 2023-03-03T07:55:55.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.305+0000 7f6e3159d700 1 -- 172.21.15.98:0/424862114 shutdown_connections 2023-03-03T07:55:55.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.305+0000 7f6e3159d700 1 -- 172.21.15.98:0/424862114 wait complete. 2023-03-03T07:55:55.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.308+0000 7f6e3159d700 1 Processor -- start 2023-03-03T07:55:55.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.310+0000 7f149b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f149c0cc2f0 0x7f149c203010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.310+0000 7f149b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f149c0cc2f0 0x7f149c203010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41612/0 (socket says 172.21.15.98:41612) 2023-03-03T07:55:55.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.310+0000 7f149b7fe700 1 -- 172.21.15.98:0/2954731960 learned_addr learned my addr 172.21.15.98:0/2954731960 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:55.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f149b7fe700 1 -- 172.21.15.98:0/2954731960 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f149c06d080 msgr2=0x7f149c0c32a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T07:55:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f149b7fe700 1 --2- 172.21.15.98:0/2954731960 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f149c06d080 0x7f149c0c32a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f149b7fe700 1 -- 172.21.15.98:0/2954731960 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f149c0b0ef0 msgr2=0x7f149c202ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T07:55:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f149b7fe700 1 --2- 172.21.15.98:0/2954731960 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f149c0b0ef0 0x7f149c202ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f149b7fe700 1 -- 172.21.15.98:0/2954731960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c0094a0 con 0x7f149c0cc2f0 2023-03-03T07:55:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.308+0000 7f4db37fe700 1 --2- 172.21.15.98:0/1604298407 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4da00683c0 0x7f4da006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.310+0000 7f4db37fe700 1 --2- 172.21.15.98:0/1604298407 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4da00683c0 0x7f4da006a870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4db4070b00 tx=0x7f4da40015e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.307+0000 7f8a519ff700 1 -- start start 2023-03-03T07:55:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.309+0000 7f8a519ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a4c06b820 0x7f8a4c0b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.309+0000 7f8a519ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a4c06d460 0x7f8a4c1f9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.309+0000 7f8a519ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c0cc310 0x7f8a4c1fa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.309+0000 7f8a519ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c0b4620 con 0x7f8a4c0cc310 2023-03-03T07:55:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.310+0000 7f8a519ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c0b4060 con 0x7f8a4c06d460 2023-03-03T07:55:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.310+0000 7f8a519ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c0b4340 con 0x7f8a4c06b820 2023-03-03T07:55:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.310+0000 7f8a4a7fc700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a4c06d460 0x7f8a4c1f9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f8a4affd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a4c06b820 0x7f8a4c0b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f8a4a7fc700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a4c06d460 0x7f8a4c1f9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:46974/0 (socket says 172.21.15.98:46974) 2023-03-03T07:55:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f8a4a7fc700 1 -- 172.21.15.98:0/3711321404 learned_addr learned my addr 172.21.15.98:0/3711321404 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:55.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.309+0000 7f6e3159d700 1 -- start start 2023-03-03T07:55:55.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.310+0000 7f6e3159d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e2c06bd20 0x7f6e2c13bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.310+0000 7f6e3159d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e2c06de90 0x7f6e2c15f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f6e3159d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6e2c157a60 0x7f6e2c15fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f6e3159d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c15baa0 con 0x7f6e2c06de90 2023-03-03T07:55:55.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f6e3159d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c15b770 con 0x7f6e2c157a60 2023-03-03T07:55:55.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f6e3159d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c160430 con 0x7f6e2c06bd20 2023-03-03T07:55:55.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f6e2b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e2c06de90 0x7f6e2c15f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f6e2b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e2c06de90 0x7f6e2c15f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41636/0 (socket says 172.21.15.98:41636) 2023-03-03T07:55:55.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f6e2b7fe700 1 -- 172.21.15.98:0/2065812641 learned_addr learned my addr 172.21.15.98:0/2065812641 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:55.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f149b7fe700 1 --2- 172.21.15.98:0/2954731960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f149c0cc2f0 0x7f149c203010 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f149000d5d0 tx=0x7f149000d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.312+0000 7f8a4a7fc700 1 -- 172.21.15.98:0/3711321404 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a4c06b820 msgr2=0x7f8a4c0b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.312+0000 7f8a4a7fc700 1 --2- 172.21.15.98:0/3711321404 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a4c06b820 0x7f8a4c0b3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.312+0000 7f8a4a7fc700 1 -- 172.21.15.98:0/3711321404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c0cc310 msgr2=0x7f8a4c1fa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.312+0000 7f8a4a7fc700 1 --2- 172.21.15.98:0/3711321404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c0cc310 0x7f8a4c1fa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.312+0000 7f8a4a7fc700 1 -- 172.21.15.98:0/3711321404 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c0094a0 con 0x7f8a4c06d460 2023-03-03T07:55:55.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.312+0000 7f8a4a7fc700 1 --2- 172.21.15.98:0/3711321404 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a4c06d460 0x7f8a4c1f9d80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000c500 tx=0x7f8a4000c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.312+0000 7f6e2bfff700 1 --2- 172.21.15.98:0/2065812641 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e2c06bd20 0x7f6e2c13bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.312+0000 7f6e2b7fe700 1 -- 172.21.15.98:0/2065812641 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e2c06bd20 msgr2=0x7f6e2c13bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.312+0000 7f14837fe700 1 -- 172.21.15.98:0/2954731960 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14900125f0 con 0x7f149c0cc2f0 2023-03-03T07:55:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.312+0000 7f14837fe700 1 -- 172.21.15.98:0/2954731960 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1490012750 con 0x7f149c0cc2f0 2023-03-03T07:55:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.312+0000 7f14837fe700 1 -- 172.21.15.98:0/2954731960 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1490012b00 con 0x7f149c0cc2f0 2023-03-03T07:55:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.312+0000 7f14a15ff700 1 -- 172.21.15.98:0/2954731960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149c15d1e0 con 0x7f149c0cc2f0 2023-03-03T07:55:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.312+0000 7f14a15ff700 1 -- 172.21.15.98:0/2954731960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f149c15cf00 con 0x7f149c0cc2f0 2023-03-03T07:55:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.313+0000 7f8a33fff700 1 -- 172.21.15.98:0/3711321404 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a40005d70 con 0x7f8a4c06d460 2023-03-03T07:55:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.313+0000 7f8a33fff700 1 -- 172.21.15.98:0/3711321404 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a400070d0 con 0x7f8a4c06d460 2023-03-03T07:55:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.313+0000 7f8a33fff700 1 -- 172.21.15.98:0/3711321404 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a40007480 con 0x7f8a4c06d460 2023-03-03T07:55:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.313+0000 7f8a519ff700 1 -- 172.21.15.98:0/3711321404 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4c06be50 con 0x7f8a4c06d460 2023-03-03T07:55:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.313+0000 7f4d9bfff700 1 -- 172.21.15.98:0/1604298407 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4dac0534e0 con 0x7f4db40704b0 2023-03-03T07:55:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.312+0000 7f6e30d9c700 1 --2- 172.21.15.98:0/2065812641 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6e2c157a60 0x7f6e2c15fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.313+0000 7f6e2b7fe700 1 --2- 172.21.15.98:0/2065812641 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e2c06bd20 0x7f6e2c13bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.313+0000 7f6e2b7fe700 1 -- 172.21.15.98:0/2065812641 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6e2c157a60 msgr2=0x7f6e2c15fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.313+0000 7f6e2b7fe700 1 --2- 172.21.15.98:0/2065812641 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6e2c157a60 0x7f6e2c15fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.313+0000 7f6e2b7fe700 1 -- 172.21.15.98:0/2065812641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e24009550 con 0x7f6e2c06de90 2023-03-03T07:55:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.313+0000 7f6e2bfff700 1 --2- 172.21.15.98:0/2065812641 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e2c06bd20 0x7f6e2c13bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:55:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.314+0000 7f8a519ff700 1 -- 172.21.15.98:0/3711321404 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4c06bcc0 con 0x7f8a4c06d460 2023-03-03T07:55:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.314+0000 7f6e2b7fe700 1 --2- 172.21.15.98:0/2065812641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e2c06de90 0x7f6e2c15f5c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00cc50 tx=0x7f6e1c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.310+0000 7f725e1ff700 1 -- 172.21.15.98:0/2016747713 shutdown_connections 2023-03-03T07:55:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.311+0000 7f9d96ffd700 1 -- 172.21.15.98:0/3240270850 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9da001b030 con 0x7f9da8164620 2023-03-03T07:55:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.314+0000 7f9d96ffd700 1 --2- 172.21.15.98:0/3240270850 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9d88068390 0x7f9d8806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.314+0000 7f9d96ffd700 1 -- 172.21.15.98:0/3240270850 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9da0006e40 con 0x7f9da8164620 2023-03-03T07:55:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.315+0000 7f6e28ff9700 1 -- 172.21.15.98:0/2065812641 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c00ade0 con 0x7f6e2c06de90 2023-03-03T07:55:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.315+0000 7f6e28ff9700 1 -- 172.21.15.98:0/2065812641 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1c00ce50 con 0x7f6e2c06de90 2023-03-03T07:55:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.315+0000 7f6e28ff9700 1 -- 172.21.15.98:0/2065812641 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c003c20 con 0x7f6e2c06de90 2023-03-03T07:55:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.315+0000 7f6e3159d700 1 -- 172.21.15.98:0/2065812641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e2c1617e0 con 0x7f6e2c06de90 2023-03-03T07:55:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.315+0000 7f6e3159d700 1 -- 172.21.15.98:0/2065812641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2c161500 con 0x7f6e2c06de90 2023-03-03T07:55:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.316+0000 7f9da659c700 1 --2- 172.21.15.98:0/3240270850 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9d88068390 0x7f9d8806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.318+0000 7f9da659c700 1 --2- 172.21.15.98:0/3240270850 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9d88068390 0x7f9d8806a840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c009770 tx=0x7f9d9c006cd0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.314+0000 7f725e1ff700 1 -- 172.21.15.98:0/2016747713 wait complete. 2023-03-03T07:55:55.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.318+0000 7f725e1ff700 1 Processor -- start 2023-03-03T07:55:55.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.318+0000 7f8a33fff700 1 -- 172.21.15.98:0/3711321404 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8a40019030 con 0x7f8a4c06d460 2023-03-03T07:55:55.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.319+0000 7f9d96ffd700 1 -- 172.21.15.98:0/3240270850 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9da0052140 con 0x7f9da8164620 2023-03-03T07:55:55.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.318+0000 7f725e1ff700 1 -- start start 2023-03-03T07:55:55.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.319+0000 7f6e3159d700 1 -- 172.21.15.98:0/2065812641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e100052f0 con 0x7f6e2c06de90 2023-03-03T07:55:55.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.320+0000 7f14837fe700 1 -- 172.21.15.98:0/2954731960 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1490012c60 con 0x7f149c0cc2f0 2023-03-03T07:55:55.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.320+0000 7f725e1ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72580b26d0 0x7f725815e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.320+0000 7f725e1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72580b5140 0x7f72580b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.320+0000 7f725e1ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72580b8d10 0x7f72580b4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.320+0000 7f725e1ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72580ca980 con 0x7f72580b26d0 2023-03-03T07:55:55.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.320+0000 7f725e1ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725815ecb0 con 0x7f72580b8d10 2023-03-03T07:55:55.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.320+0000 7f725e1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725815ef90 con 0x7f72580b5140 2023-03-03T07:55:55.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.321+0000 7f7256ffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72580b5140 0x7f72580b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.321+0000 7f7256ffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72580b5140 0x7f72580b3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:60958/0 (socket says 172.21.15.98:60958) 2023-03-03T07:55:55.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.321+0000 7f7256ffd700 1 -- 172.21.15.98:0/1883901335 learned_addr learned my addr 172.21.15.98:0/1883901335 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:55.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.321+0000 7f7257fff700 1 --2- 172.21.15.98:0/1883901335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72580b8d10 0x7f72580b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.322+0000 7f7256ffd700 1 -- 172.21.15.98:0/1883901335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72580b8d10 msgr2=0x7f72580b4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.322+0000 7f7256ffd700 1 --2- 172.21.15.98:0/1883901335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72580b8d10 0x7f72580b4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.322+0000 7f7256ffd700 1 -- 172.21.15.98:0/1883901335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72580b26d0 msgr2=0x7f725815e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.322+0000 7f7256ffd700 1 --2- 172.21.15.98:0/1883901335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72580b26d0 0x7f725815e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.322+0000 7f7256ffd700 1 -- 172.21.15.98:0/1883901335 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7250009a70 con 0x7f72580b5140 2023-03-03T07:55:55.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.322+0000 7f8a33fff700 1 --2- 172.21.15.98:0/3711321404 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8a380683c0 0x7f8a3806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.322+0000 7f8a33fff700 1 -- 172.21.15.98:0/3711321404 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8a40085130 con 0x7f8a4c06d460 2023-03-03T07:55:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.322+0000 7f8a519ff700 1 -- 172.21.15.98:0/3711321404 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4c0b0f10 con 0x7f8a4c06d460 2023-03-03T07:55:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.322+0000 7f7256ffd700 1 --2- 172.21.15.98:0/1883901335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72580b5140 0x7f72580b3af0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f724c00e870 tx=0x7f724c00ec30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.322+0000 7f14837fe700 1 --2- 172.21.15.98:0/2954731960 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f14880683c0 0x7f148806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.323+0000 7f14837fe700 1 -- 172.21.15.98:0/2954731960 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1490085f90 con 0x7f149c0cc2f0 2023-03-03T07:55:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.323+0000 7f8a4affd700 1 --2- 172.21.15.98:0/3711321404 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8a380683c0 0x7f8a3806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.323+0000 7f6e28ff9700 1 -- 172.21.15.98:0/2065812641 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6e1c010050 con 0x7f6e2c06de90 2023-03-03T07:55:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.323+0000 7f14a15ff700 1 -- 172.21.15.98:0/2954731960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147c0052f0 con 0x7f149c0cc2f0 2023-03-03T07:55:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.323+0000 7f723ffff700 1 -- 172.21.15.98:0/1883901335 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c009590 con 0x7f72580b5140 2023-03-03T07:55:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.323+0000 7f723ffff700 1 -- 172.21.15.98:0/1883901335 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724c00ee70 con 0x7f72580b5140 2023-03-03T07:55:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.323+0000 7f723ffff700 1 -- 172.21.15.98:0/1883901335 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c00ca50 con 0x7f72580b5140 2023-03-03T07:55:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.323+0000 7f725e1ff700 1 -- 172.21.15.98:0/1883901335 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72580d0e80 con 0x7f72580b5140 2023-03-03T07:55:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.323+0000 7f725e1ff700 1 -- 172.21.15.98:0/1883901335 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72580cdd00 con 0x7f72580b5140 2023-03-03T07:55:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.325+0000 7f8a4affd700 1 --2- 172.21.15.98:0/3711321404 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8a380683c0 0x7f8a3806a870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0097c0 tx=0x7f8a3c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.325+0000 7f149affd700 1 --2- 172.21.15.98:0/2954731960 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f14880683c0 0x7f148806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.325+0000 7f6e28ff9700 1 --2- 172.21.15.98:0/2065812641 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6e0c0683c0 0x7f6e0c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.325+0000 7fe6b55ff700 1 Processor -- start 2023-03-03T07:55:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.326+0000 7fe6b55ff700 1 -- start start 2023-03-03T07:55:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.325+0000 7f6e28ff9700 1 -- 172.21.15.98:0/2065812641 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6e1c085ca0 con 0x7f6e2c06de90 2023-03-03T07:55:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.326+0000 7f6e2bfff700 1 --2- 172.21.15.98:0/2065812641 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6e0c0683c0 0x7f6e0c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.328+0000 7f6e2bfff700 1 --2- 172.21.15.98:0/2065812641 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6e0c0683c0 0x7f6e0c06a870 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6e14006fd0 tx=0x7f6e14008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.328+0000 7f149affd700 1 --2- 172.21.15.98:0/2954731960 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f14880683c0 0x7f148806a870 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1494002020 tx=0x7f1494001650 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.330+0000 7fe6b55ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b0158870 0x7fe6b015c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.326+0000 7f725e1ff700 1 -- 172.21.15.98:0/1883901335 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72380052f0 con 0x7f72580b5140 2023-03-03T07:55:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.331+0000 7f8a33fff700 1 -- 172.21.15.98:0/3711321404 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8a40051920 con 0x7f8a4c06d460 2023-03-03T07:55:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.330+0000 7fe6b55ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6b00b0f10 0x7fe6b0070770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.330+0000 7fe6b55ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b015d540 con 0x7fe6b015cb00 2023-03-03T07:55:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.330+0000 7fe6b55ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b00702e0 con 0x7fe6b00b0f10 2023-03-03T07:55:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.330+0000 7fe6b55ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b015cf80 con 0x7fe6b0158870 2023-03-03T07:55:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.331+0000 7fe6aeffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b0158870 0x7fe6b015c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.331+0000 7fe6aeffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b0158870 0x7fe6b015c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:60960/0 (socket says 172.21.15.98:60960) 2023-03-03T07:55:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.331+0000 7fe6aeffd700 1 -- 172.21.15.98:0/2547695922 learned_addr learned my addr 172.21.15.98:0/2547695922 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.332+0000 7fe6aeffd700 1 -- 172.21.15.98:0/2547695922 >> v2:172.21.15.98:3300/0 conn(0x7fe6b00b0f10 msgr2=0x7fe6b0070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.332+0000 7fe6aeffd700 1 --2- 172.21.15.98:0/2547695922 >> v2:172.21.15.98:3300/0 conn(0x7fe6b00b0f10 0x7fe6b0070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.332+0000 7fe6aeffd700 1 -- 172.21.15.98:0/2547695922 >> v1:172.21.15.100:6789/0 conn(0x7fe6b015cb00 legacy=0x7fe6b00b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.332+0000 7fe6aeffd700 1 -- 172.21.15.98:0/2547695922 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b015d260 con 0x7fe6b0158870 2023-03-03T07:55:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.333+0000 7fe6aeffd700 1 --2- 172.21.15.98:0/2547695922 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b0158870 0x7fe6b015c5c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0008970 tx=0x7fe6a0008c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93702ed50666a9c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.332+0000 7f6e28ff9700 1 -- 172.21.15.98:0/2065812641 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6e1c052490 con 0x7f6e2c06de90 2023-03-03T07:55:55.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.331+0000 7f723ffff700 1 -- 172.21.15.98:0/1883901335 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f724c019030 con 0x7f72580b5140 2023-03-03T07:55:55.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.333+0000 7fe6adffb700 1 -- 172.21.15.98:0/2547695922 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0004230 con 0x7fe6b0158870 2023-03-03T07:55:55.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.334+0000 7fe6adffb700 1 -- 172.21.15.98:0/2547695922 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a0004390 con 0x7fe6b0158870 2023-03-03T07:55:55.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.334+0000 7fe6adffb700 1 -- 172.21.15.98:0/2547695922 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0008ec0 con 0x7fe6b0158870 2023-03-03T07:55:55.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.335+0000 7fe6b55ff700 1 -- 172.21.15.98:0/2547695922 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b0158870 msgr2=0x7fe6b015c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.335+0000 7fe6b55ff700 1 --2- 172.21.15.98:0/2547695922 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b0158870 0x7fe6b015c5c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0008970 tx=0x7fe6a0008c80 comp rx=0 tx=0).stop 2023-03-03T07:55:55.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.335+0000 7fe6b55ff700 1 -- 172.21.15.98:0/2547695922 shutdown_connections 2023-03-03T07:55:55.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.335+0000 7fe6b55ff700 1 --2- 172.21.15.98:0/2547695922 >> v2:172.21.15.98:3300/0 conn(0x7fe6b00b0f10 0x7fe6b0070770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.335+0000 7fe6b55ff700 1 --2- 172.21.15.98:0/2547695922 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b0158870 0x7fe6b015c5c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.335+0000 7fe6b55ff700 1 -- 172.21.15.98:0/2547695922 >> 172.21.15.98:0/2547695922 conn(0x7fe6b00cb800 msgr2=0x7fe6b00cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.335+0000 7f723ffff700 1 --2- 172.21.15.98:0/1883901335 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f72440683c0 0x7f724406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.335+0000 7f723ffff700 1 -- 172.21.15.98:0/1883901335 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f724c085bf0 con 0x7f72580b5140 2023-03-03T07:55:55.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.336+0000 7f723ffff700 1 -- 172.21.15.98:0/1883901335 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f724c0b05e0 con 0x7f72580b5140 2023-03-03T07:55:55.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.338+0000 7f72577fe700 1 --2- 172.21.15.98:0/1883901335 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f72440683c0 0x7f724406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.339+0000 7f72577fe700 1 --2- 172.21.15.98:0/1883901335 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f72440683c0 0x7f724406a870 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7248001610 tx=0x7f72480079f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.343+0000 7f14837fe700 1 -- 172.21.15.98:0/2954731960 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1490052780 con 0x7f149c0cc2f0 2023-03-03T07:55:55.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.336+0000 7fe6b55ff700 1 -- 172.21.15.98:0/2547695922 shutdown_connections 2023-03-03T07:55:55.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.336+0000 7fe6b55ff700 1 -- 172.21.15.98:0/2547695922 wait complete. 2023-03-03T07:55:55.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.339+0000 7fe6b55ff700 1 Processor -- start 2023-03-03T07:55:55.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.340+0000 7fe6b55ff700 1 -- start start 2023-03-03T07:55:55.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.341+0000 7fe6b55ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6b00b0f10 0x7fe6b00af830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.341+0000 7fe6b55ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b0158870 0x7fe6b00afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.341+0000 7fe6b55ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b014e5a0 0x7fe6b014ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.341+0000 7fe6b55ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b014fa90 con 0x7fe6b0158870 2023-03-03T07:55:55.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.341+0000 7fe6b55ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b014ef50 con 0x7fe6b00b0f10 2023-03-03T07:55:55.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.341+0000 7fe6b55ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b014f1f0 con 0x7fe6b014e5a0 2023-03-03T07:55:55.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.342+0000 7fe6af7fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b014e5a0 0x7fe6b014ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.342+0000 7fe6af7fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b014e5a0 0x7fe6b014ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:60974/0 (socket says 172.21.15.98:60974) 2023-03-03T07:55:55.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.342+0000 7fe6af7fe700 1 -- 172.21.15.98:0/1425290079 learned_addr learned my addr 172.21.15.98:0/1425290079 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.343+0000 7fe6ae7fc700 1 --2- 172.21.15.98:0/1425290079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b0158870 0x7fe6b00afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.343+0000 7fe6aeffd700 1 --2- 172.21.15.98:0/1425290079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6b00b0f10 0x7fe6b00af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.343+0000 7fe6af7fe700 1 -- 172.21.15.98:0/1425290079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6b00b0f10 msgr2=0x7fe6b00af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.343+0000 7fe6af7fe700 1 --2- 172.21.15.98:0/1425290079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6b00b0f10 0x7fe6b00af830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.343+0000 7fe6af7fe700 1 -- 172.21.15.98:0/1425290079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b0158870 msgr2=0x7fe6b00afd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.343+0000 7fe6af7fe700 1 --2- 172.21.15.98:0/1425290079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b0158870 0x7fe6b00afd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.343+0000 7fe6af7fe700 1 -- 172.21.15.98:0/1425290079 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a00085d0 con 0x7fe6b014e5a0 2023-03-03T07:55:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.344+0000 7fe6af7fe700 1 --2- 172.21.15.98:0/1425290079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b014e5a0 0x7fe6b014ea10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800b9e0 tx=0x7fe6a800bda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.344+0000 7fe6977fe700 1 -- 172.21.15.98:0/1425290079 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a800c9b0 con 0x7fe6b014e5a0 2023-03-03T07:55:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.344+0000 7fe6977fe700 1 -- 172.21.15.98:0/1425290079 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a800cb30 con 0x7fe6b014e5a0 2023-03-03T07:55:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.344+0000 7fe6977fe700 1 -- 172.21.15.98:0/1425290079 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a8014440 con 0x7fe6b014e5a0 2023-03-03T07:55:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.344+0000 7fe6b55ff700 1 -- 172.21.15.98:0/1425290079 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b00cc950 con 0x7fe6b014e5a0 2023-03-03T07:55:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.345+0000 7fe6b55ff700 1 -- 172.21.15.98:0/1425290079 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b006ff70 con 0x7fe6b014e5a0 2023-03-03T07:55:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.346+0000 7fe6b55ff700 1 -- 172.21.15.98:0/1425290079 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b00ccae0 con 0x7fe6b014e5a0 2023-03-03T07:55:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.350+0000 7fe6977fe700 1 -- 172.21.15.98:0/1425290079 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe6a801b030 con 0x7fe6b014e5a0 2023-03-03T07:55:55.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.352+0000 7fe6977fe700 1 --2- 172.21.15.98:0/1425290079 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe69c0683c0 0x7fe69c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.352+0000 7fe6977fe700 1 -- 172.21.15.98:0/1425290079 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe6a8006e40 con 0x7fe6b014e5a0 2023-03-03T07:55:55.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.354+0000 7fe6aeffd700 1 --2- 172.21.15.98:0/1425290079 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe69c0683c0 0x7fe69c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.355+0000 7fe6aeffd700 1 --2- 172.21.15.98:0/1425290079 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe69c0683c0 0x7fe69c06a870 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0000c00 tx=0x7fe6a0015000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.356+0000 7fe6977fe700 1 -- 172.21.15.98:0/1425290079 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe6a80528d0 con 0x7fe6b014e5a0 2023-03-03T07:55:55.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.369+0000 7f6c709ff700 1 Processor -- start 2023-03-03T07:55:55.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.369+0000 7f6c709ff700 1 -- start start 2023-03-03T07:55:55.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.372+0000 7f6c709ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6c6c164250 0x7f6c6c06c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.372+0000 7f6c709ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6c6c0b0f10 0x7f6c6c164b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.373+0000 7f6c709ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c6c072170 con 0x7f6c6c164620 2023-03-03T07:55:55.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.373+0000 7f6c709ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c6c0cb5b0 con 0x7f6c6c0b0f10 2023-03-03T07:55:55.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.373+0000 7f6c709ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0c2440 con 0x7f6c6c164250 2023-03-03T07:55:55.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.373+0000 7f6c6a59c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6c6c164250 0x7f6c6c06c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.374+0000 7f6c6a59c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6c6c164250 0x7f6c6c06c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:60990/0 (socket says 172.21.15.98:60990) 2023-03-03T07:55:55.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.374+0000 7f6c6a59c700 1 -- 172.21.15.98:0/3531828270 learned_addr learned my addr 172.21.15.98:0/3531828270 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:55.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.375+0000 7f6c6a59c700 1 -- 172.21.15.98:0/3531828270 >> v2:172.21.15.98:3300/0 conn(0x7f6c6c0b0f10 msgr2=0x7f6c6c164b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T07:55:55.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.375+0000 7f6c6a59c700 1 --2- 172.21.15.98:0/3531828270 >> v2:172.21.15.98:3300/0 conn(0x7f6c6c0b0f10 0x7f6c6c164b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.375+0000 7f6c6a59c700 1 -- 172.21.15.98:0/3531828270 >> v1:172.21.15.98:6789/0 conn(0x7f6c6c164620 legacy=0x7f6c6c0b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.375+0000 7f6c6a59c700 1 -- 172.21.15.98:0/3531828270 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c071e90 con 0x7f6c6c164250 2023-03-03T07:55:55.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.380+0000 7f6c6a59c700 1 --2- 172.21.15.98:0/3531828270 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6c6c164250 0x7f6c6c06c8e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0098f0 tx=0x7f6c5c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea8df313f06f4283 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.381+0000 7f6c6959a700 1 -- 172.21.15.98:0/3531828270 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c00b8a0 con 0x7f6c6c164250 2023-03-03T07:55:55.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.381+0000 7f6c6959a700 1 -- 172.21.15.98:0/3531828270 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c5c00ba00 con 0x7f6c6c164250 2023-03-03T07:55:55.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.381+0000 7f6c6959a700 1 -- 172.21.15.98:0/3531828270 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c00bd90 con 0x7f6c6c164250 2023-03-03T07:55:55.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.382+0000 7f6c709ff700 1 -- 172.21.15.98:0/3531828270 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6c6c164250 msgr2=0x7f6c6c06c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.382+0000 7f6c709ff700 1 --2- 172.21.15.98:0/3531828270 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6c6c164250 0x7f6c6c06c8e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0098f0 tx=0x7f6c5c009c00 comp rx=0 tx=0).stop 2023-03-03T07:55:55.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.383+0000 7f6c709ff700 1 -- 172.21.15.98:0/3531828270 shutdown_connections 2023-03-03T07:55:55.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.383+0000 7f6c709ff700 1 --2- 172.21.15.98:0/3531828270 >> v2:172.21.15.98:3300/0 conn(0x7f6c6c0b0f10 0x7f6c6c164b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.383+0000 7f6c709ff700 1 --2- 172.21.15.98:0/3531828270 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6c6c164250 0x7f6c6c06c8e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.383+0000 7f6c709ff700 1 -- 172.21.15.98:0/3531828270 >> 172.21.15.98:0/3531828270 conn(0x7f6c6c0cb800 msgr2=0x7f6c6c0cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.384+0000 7f6c709ff700 1 -- 172.21.15.98:0/3531828270 shutdown_connections 2023-03-03T07:55:55.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.384+0000 7f6c709ff700 1 -- 172.21.15.98:0/3531828270 wait complete. 2023-03-03T07:55:55.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.387+0000 7f6c709ff700 1 Processor -- start 2023-03-03T07:55:55.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.388+0000 7f6c709ff700 1 -- start start 2023-03-03T07:55:55.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.389+0000 7f6c709ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c6c0b0f10 0x7f6c6c0c3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.389+0000 7f6c709ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c164250 0x7f6c6c15cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.390+0000 7f6c709ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6c6c164620 0x7f6c6c0c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.390+0000 7f6c709ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0c3ac0 con 0x7f6c6c164250 2023-03-03T07:55:55.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.390+0000 7f6c709ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0c7740 con 0x7f6c6c0b0f10 2023-03-03T07:55:55.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.390+0000 7f6c709ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0b4ab0 con 0x7f6c6c164620 2023-03-03T07:55:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.393+0000 7f6c6a59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c6c0b0f10 0x7f6c6c0c3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.394+0000 7f6c6a59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c6c0b0f10 0x7f6c6c0c3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:47010/0 (socket says 172.21.15.98:47010) 2023-03-03T07:55:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.394+0000 7f6c6a59c700 1 -- 172.21.15.98:0/3964328328 learned_addr learned my addr 172.21.15.98:0/3964328328 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.394+0000 7f6c6ad9d700 1 --2- 172.21.15.98:0/3964328328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6c6c164620 0x7f6c6c0c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.395+0000 7f6c6a59c700 1 -- 172.21.15.98:0/3964328328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6c6c164620 msgr2=0x7f6c6c0c4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.395+0000 7f6c6a59c700 1 --2- 172.21.15.98:0/3964328328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6c6c164620 0x7f6c6c0c4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.395+0000 7f6c6a59c700 1 -- 172.21.15.98:0/3964328328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c164250 msgr2=0x7f6c6c15cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.395+0000 7f6c6a59c700 1 --2- 172.21.15.98:0/3964328328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c164250 0x7f6c6c15cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.395+0000 7f6c6a59c700 1 -- 172.21.15.98:0/3964328328 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5c009550 con 0x7f6c6c0b0f10 2023-03-03T07:55:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.396+0000 7f6c6a59c700 1 --2- 172.21.15.98:0/3964328328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c6c0b0f10 0x7f6c6c0c3580 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c014040 tx=0x7f6c5c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.397+0000 7f6c5affd700 1 -- 172.21.15.98:0/3964328328 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c0038a0 con 0x7f6c6c0b0f10 2023-03-03T07:55:55.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.397+0000 7f6c5affd700 1 -- 172.21.15.98:0/3964328328 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c5c01e070 con 0x7f6c6c0b0f10 2023-03-03T07:55:55.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.397+0000 7f6c5affd700 1 -- 172.21.15.98:0/3964328328 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c003c00 con 0x7f6c6c0b0f10 2023-03-03T07:55:55.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.397+0000 7f6c709ff700 1 -- 172.21.15.98:0/3964328328 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6c0cadb0 con 0x7f6c6c0b0f10 2023-03-03T07:55:55.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.397+0000 7f6c709ff700 1 -- 172.21.15.98:0/3964328328 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6c0caad0 con 0x7f6c6c0b0f10 2023-03-03T07:55:55.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.404+0000 7f6c5affd700 1 -- 172.21.15.98:0/3964328328 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6c5c00f050 con 0x7f6c6c0b0f10 2023-03-03T07:55:55.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.407+0000 7f6c5affd700 1 --2- 172.21.15.98:0/3964328328 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6c4c0683c0 0x7f6c4c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:55.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.407+0000 7f6c5affd700 1 -- 172.21.15.98:0/3964328328 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6c5c020070 con 0x7f6c6c0b0f10 2023-03-03T07:55:55.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.408+0000 7f6c69d9b700 1 --2- 172.21.15.98:0/3964328328 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6c4c0683c0 0x7f6c4c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:55.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.410+0000 7f6c69d9b700 1 --2- 172.21.15.98:0/3964328328 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6c4c0683c0 0x7f6c4c06a870 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6c640012e0 tx=0x7f6c640016e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:55.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.410+0000 7f6c709ff700 1 -- 172.21.15.98:0/3964328328 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c500052f0 con 0x7f6c6c0b0f10 2023-03-03T07:55:55.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.417+0000 7f6c5affd700 1 -- 172.21.15.98:0/3964328328 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6c5c053b20 con 0x7f6c6c0b0f10 2023-03-03T07:55:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.760+0000 7f4db9bff700 1 -- 172.21.15.98:0/1604298407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f4d940055a0 con 0x7f4db40704b0 2023-03-03T07:55:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.761+0000 7f4d9bfff700 1 -- 172.21.15.98:0/1604298407 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4dac025090 con 0x7f4db40704b0 2023-03-03T07:55:55.766 INFO:teuthology.orchestra.run.smithi098.stdout:55834574855 2023-03-03T07:55:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.766+0000 7f4db9bff700 1 -- 172.21.15.98:0/1604298407 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4da00683c0 msgr2=0x7f4da006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.766+0000 7f4db9bff700 1 --2- 172.21.15.98:0/1604298407 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4da00683c0 0x7f4da006a870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4db4070b00 tx=0x7f4da40015e0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.766+0000 7f4db9bff700 1 -- 172.21.15.98:0/1604298407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4db40704b0 msgr2=0x7f4db4169e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.766+0000 7f4db9bff700 1 --2- 172.21.15.98:0/1604298407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4db40704b0 0x7f4db4169e20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4db406fb30 tx=0x7f4dac004be0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.767+0000 7f4db9bff700 1 -- 172.21.15.98:0/1604298407 shutdown_connections 2023-03-03T07:55:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.767+0000 7f4db9bff700 1 --2- 172.21.15.98:0/1604298407 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4da00683c0 0x7f4da006a870 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4db4070b00 tx=0x7f4da40015e0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.767+0000 7f4db9bff700 1 --2- 172.21.15.98:0/1604298407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4db40704b0 0x7f4db4169e20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.767+0000 7f4db9bff700 1 --2- 172.21.15.98:0/1604298407 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d9c005320 0x7f4db416c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.767+0000 7f4db9bff700 1 --2- 172.21.15.98:0/1604298407 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4db4070040 0x7f4db40cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.767+0000 7f4db9bff700 1 -- 172.21.15.98:0/1604298407 >> 172.21.15.98:0/1604298407 conn(0x7f4db40cb800 msgr2=0x7f4db40ce200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.768+0000 7f4db9bff700 1 -- 172.21.15.98:0/1604298407 shutdown_connections 2023-03-03T07:55:55.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.768+0000 7f4db9bff700 1 -- 172.21.15.98:0/1604298407 wait complete. 2023-03-03T07:55:55.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.770+0000 7f9dac9ff700 1 -- 172.21.15.98:0/3240270850 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9d8c0055a0 con 0x7f9da8164620 2023-03-03T07:55:55.777 INFO:teuthology.orchestra.run.smithi098.stdout:55834574854 2023-03-03T07:55:55.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.771+0000 7f9d96ffd700 1 -- 172.21.15.98:0/3240270850 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9da0055760 con 0x7f9da8164620 2023-03-03T07:55:55.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.777+0000 7f9dac9ff700 1 -- 172.21.15.98:0/3240270850 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9d88068390 msgr2=0x7f9d8806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.777+0000 7f9dac9ff700 1 --2- 172.21.15.98:0/3240270850 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9d88068390 0x7f9d8806a840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c009770 tx=0x7f9d9c006cd0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.777+0000 7f9dac9ff700 1 -- 172.21.15.98:0/3240270850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da8164620 msgr2=0x7f9da8161350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.777+0000 7f9dac9ff700 1 --2- 172.21.15.98:0/3240270850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da8164620 0x7f9da8161350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9da000b9e0 tx=0x7f9da000bda0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.778+0000 7f9dac9ff700 1 -- 172.21.15.98:0/3240270850 shutdown_connections 2023-03-03T07:55:55.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.778+0000 7f9dac9ff700 1 --2- 172.21.15.98:0/3240270850 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9d88068390 0x7f9d8806a840 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.778+0000 7f9dac9ff700 1 --2- 172.21.15.98:0/3240270850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9da80b0f10 0x7f9da8156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.778+0000 7f9dac9ff700 1 --2- 172.21.15.98:0/3240270850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da8164620 0x7f9da8161350 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.778+0000 7f9dac9ff700 1 --2- 172.21.15.98:0/3240270850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9da8164250 0x7f9da80c11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.778+0000 7f9dac9ff700 1 -- 172.21.15.98:0/3240270850 >> 172.21.15.98:0/3240270850 conn(0x7f9da80cb800 msgr2=0x7f9da806fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.778+0000 7f9dac9ff700 1 -- 172.21.15.98:0/3240270850 shutdown_connections 2023-03-03T07:55:55.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.778+0000 7f9dac9ff700 1 -- 172.21.15.98:0/3240270850 wait complete. 2023-03-03T07:55:55.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.781+0000 7f8a519ff700 1 -- 172.21.15.98:0/3711321404 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8a4c04e6e0 con 0x7f8a4c06d460 2023-03-03T07:55:55.786 INFO:teuthology.orchestra.run.smithi098.stdout:55834574854 2023-03-03T07:55:55.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.781+0000 7f8a33fff700 1 -- 172.21.15.98:0/3711321404 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8a40017030 con 0x7f8a4c06d460 2023-03-03T07:55:55.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.785+0000 7f6e3159d700 1 -- 172.21.15.98:0/2065812641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6e100055a0 con 0x7f6e2c06de90 2023-03-03T07:55:55.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.786+0000 7fe6b55ff700 1 -- 172.21.15.98:0/1425290079 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fe6b004e6e0 con 0x7fe6b014e5a0 2023-03-03T07:55:55.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.786+0000 7f6e28ff9700 1 -- 172.21.15.98:0/2065812641 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6e1c055ab0 con 0x7f6e2c06de90 2023-03-03T07:55:55.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.786+0000 7f8a519ff700 1 -- 172.21.15.98:0/3711321404 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8a380683c0 msgr2=0x7f8a3806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.786+0000 7f8a519ff700 1 --2- 172.21.15.98:0/3711321404 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8a380683c0 0x7f8a3806a870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0097c0 tx=0x7f8a3c008030 comp rx=0 tx=0).stop 2023-03-03T07:55:55.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.786+0000 7f8a519ff700 1 -- 172.21.15.98:0/3711321404 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a4c06d460 msgr2=0x7f8a4c1f9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.786+0000 7f8a519ff700 1 --2- 172.21.15.98:0/3711321404 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a4c06d460 0x7f8a4c1f9d80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000c500 tx=0x7f8a4000c810 comp rx=0 tx=0).stop 2023-03-03T07:55:55.789 INFO:teuthology.orchestra.run.smithi098.stdout:55834574853 2023-03-03T07:55:55.792 INFO:teuthology.orchestra.run.smithi098.stdout:55834574855 2023-03-03T07:55:55.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.789+0000 7f725e1ff700 1 -- 172.21.15.98:0/1883901335 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f7238005870 con 0x7f72580b5140 2023-03-03T07:55:55.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.790+0000 7f723ffff700 1 -- 172.21.15.98:0/1883901335 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f724c017020 con 0x7f72580b5140 2023-03-03T07:55:55.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.787+0000 7f8a519ff700 1 -- 172.21.15.98:0/3711321404 shutdown_connections 2023-03-03T07:55:55.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.787+0000 7f8a519ff700 1 --2- 172.21.15.98:0/3711321404 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8a380683c0 0x7f8a3806a870 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.787+0000 7f8a519ff700 1 --2- 172.21.15.98:0/3711321404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c0cc310 0x7f8a4c1fa2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.787+0000 7f8a519ff700 1 --2- 172.21.15.98:0/3711321404 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a4c06b820 0x7f8a4c0b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.787+0000 7f8a519ff700 1 --2- 172.21.15.98:0/3711321404 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a4c06d460 0x7f8a4c1f9d80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.787+0000 7f8a519ff700 1 -- 172.21.15.98:0/3711321404 >> 172.21.15.98:0/3711321404 conn(0x7f8a4c0cb800 msgr2=0x7f8a4c150b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.790+0000 7f8a519ff700 1 -- 172.21.15.98:0/3711321404 shutdown_connections 2023-03-03T07:55:55.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.790+0000 7f8a519ff700 1 -- 172.21.15.98:0/3711321404 wait complete. 2023-03-03T07:55:55.796 INFO:teuthology.orchestra.run.smithi098.stdout:55834574853 2023-03-03T07:55:55.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.787+0000 7fe6977fe700 1 -- 172.21.15.98:0/1425290079 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe6a8019020 con 0x7fe6b014e5a0 2023-03-03T07:55:55.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.791+0000 7fe6b55ff700 1 -- 172.21.15.98:0/1425290079 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe69c0683c0 msgr2=0x7fe69c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.791+0000 7fe6b55ff700 1 --2- 172.21.15.98:0/1425290079 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe69c0683c0 0x7fe69c06a870 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0000c00 tx=0x7fe6a0015000 comp rx=0 tx=0).stop 2023-03-03T07:55:55.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.791+0000 7fe6b55ff700 1 -- 172.21.15.98:0/1425290079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b014e5a0 msgr2=0x7fe6b014ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.791+0000 7fe6b55ff700 1 --2- 172.21.15.98:0/1425290079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b014e5a0 0x7fe6b014ea10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800b9e0 tx=0x7fe6a800bda0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.791+0000 7f6e3159d700 1 -- 172.21.15.98:0/2065812641 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6e0c0683c0 msgr2=0x7f6e0c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.791+0000 7f6e3159d700 1 --2- 172.21.15.98:0/2065812641 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6e0c0683c0 0x7f6e0c06a870 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6e14006fd0 tx=0x7f6e14008040 comp rx=0 tx=0).stop 2023-03-03T07:55:55.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.791+0000 7f6e3159d700 1 -- 172.21.15.98:0/2065812641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e2c06de90 msgr2=0x7f6e2c15f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.791+0000 7f6e3159d700 1 --2- 172.21.15.98:0/2065812641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e2c06de90 0x7f6e2c15f5c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00cc50 tx=0x7f6e1c00a340 comp rx=0 tx=0).stop 2023-03-03T07:55:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.793+0000 7fe6b55ff700 1 -- 172.21.15.98:0/1425290079 shutdown_connections 2023-03-03T07:55:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.793+0000 7fe6b55ff700 1 --2- 172.21.15.98:0/1425290079 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe69c0683c0 0x7fe69c06a870 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.793+0000 7fe6b55ff700 1 --2- 172.21.15.98:0/1425290079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b014e5a0 0x7fe6b014ea10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.793+0000 7fe6b55ff700 1 --2- 172.21.15.98:0/1425290079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6b00b0f10 0x7fe6b00af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.793+0000 7fe6b55ff700 1 --2- 172.21.15.98:0/1425290079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b0158870 0x7fe6b00afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.793+0000 7fe6b55ff700 1 -- 172.21.15.98:0/1425290079 >> 172.21.15.98:0/1425290079 conn(0x7fe6b00cb800 msgr2=0x7fe6b00cacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.794+0000 7fe6b55ff700 1 -- 172.21.15.98:0/1425290079 shutdown_connections 2023-03-03T07:55:55.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.794+0000 7f6e3159d700 1 -- 172.21.15.98:0/2065812641 shutdown_connections 2023-03-03T07:55:55.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.794+0000 7f6e3159d700 1 --2- 172.21.15.98:0/2065812641 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6e0c0683c0 0x7f6e0c06a870 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.794+0000 7f6e3159d700 1 --2- 172.21.15.98:0/2065812641 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6e2c157a60 0x7f6e2c15fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.794+0000 7f6e3159d700 1 --2- 172.21.15.98:0/2065812641 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e2c06bd20 0x7f6e2c13bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.794+0000 7f6e3159d700 1 --2- 172.21.15.98:0/2065812641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e2c06de90 0x7f6e2c15f5c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.794+0000 7f6e3159d700 1 -- 172.21.15.98:0/2065812641 >> 172.21.15.98:0/2065812641 conn(0x7f6e2c073810 msgr2=0x7f6e2c1623a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.794+0000 7fe6b55ff700 1 -- 172.21.15.98:0/1425290079 wait complete. 2023-03-03T07:55:55.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.794+0000 7f14a15ff700 1 -- 172.21.15.98:0/2954731960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f147c0055a0 con 0x7f149c0cc2f0 2023-03-03T07:55:55.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.795+0000 7f6e3159d700 1 -- 172.21.15.98:0/2065812641 shutdown_connections 2023-03-03T07:55:55.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.795+0000 7f6e3159d700 1 -- 172.21.15.98:0/2065812641 wait complete. 2023-03-03T07:55:55.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.795+0000 7f14837fe700 1 -- 172.21.15.98:0/2954731960 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1490055da0 con 0x7f149c0cc2f0 2023-03-03T07:55:55.807 INFO:teuthology.orchestra.run.smithi098.stdout:55834574853 2023-03-03T07:55:55.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.796+0000 7f725e1ff700 1 -- 172.21.15.98:0/1883901335 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f72440683c0 msgr2=0x7f724406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.796+0000 7f725e1ff700 1 --2- 172.21.15.98:0/1883901335 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f72440683c0 0x7f724406a870 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7248001610 tx=0x7f72480079f0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.796+0000 7f725e1ff700 1 -- 172.21.15.98:0/1883901335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72580b5140 msgr2=0x7f72580b3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.796+0000 7f725e1ff700 1 --2- 172.21.15.98:0/1883901335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72580b5140 0x7f72580b3af0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f724c00e870 tx=0x7f724c00ec30 comp rx=0 tx=0).stop 2023-03-03T07:55:55.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.797+0000 7f725e1ff700 1 -- 172.21.15.98:0/1883901335 shutdown_connections 2023-03-03T07:55:55.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.797+0000 7f725e1ff700 1 --2- 172.21.15.98:0/1883901335 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f72440683c0 0x7f724406a870 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.797+0000 7f725e1ff700 1 --2- 172.21.15.98:0/1883901335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72580b26d0 0x7f725815e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.797+0000 7f725e1ff700 1 --2- 172.21.15.98:0/1883901335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72580b8d10 0x7f72580b4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.797+0000 7f725e1ff700 1 --2- 172.21.15.98:0/1883901335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72580b5140 0x7f72580b3af0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.797+0000 7f725e1ff700 1 -- 172.21.15.98:0/1883901335 >> 172.21.15.98:0/1883901335 conn(0x7f7258073e70 msgr2=0x7f725815b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.798+0000 7f725e1ff700 1 -- 172.21.15.98:0/1883901335 shutdown_connections 2023-03-03T07:55:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.798+0000 7f725e1ff700 1 -- 172.21.15.98:0/1883901335 wait complete. 2023-03-03T07:55:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.800+0000 7f14a15ff700 1 -- 172.21.15.98:0/2954731960 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f14880683c0 msgr2=0x7f148806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.800+0000 7f14a15ff700 1 --2- 172.21.15.98:0/2954731960 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f14880683c0 0x7f148806a870 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1494002020 tx=0x7f1494001650 comp rx=0 tx=0).stop 2023-03-03T07:55:55.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.800+0000 7f14a15ff700 1 -- 172.21.15.98:0/2954731960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f149c0cc2f0 msgr2=0x7f149c203010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.800+0000 7f14a15ff700 1 --2- 172.21.15.98:0/2954731960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f149c0cc2f0 0x7f149c203010 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f149000d5d0 tx=0x7f149000d990 comp rx=0 tx=0).stop 2023-03-03T07:55:55.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.801+0000 7f14a15ff700 1 -- 172.21.15.98:0/2954731960 shutdown_connections 2023-03-03T07:55:55.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.801+0000 7f14a15ff700 1 --2- 172.21.15.98:0/2954731960 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f14880683c0 0x7f148806a870 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.801+0000 7f14a15ff700 1 --2- 172.21.15.98:0/2954731960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f149c0cc2f0 0x7f149c203010 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.801+0000 7f14a15ff700 1 --2- 172.21.15.98:0/2954731960 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f149c06d080 0x7f149c0c32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.801+0000 7f14a15ff700 1 --2- 172.21.15.98:0/2954731960 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f149c0b0ef0 0x7f149c202ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.801+0000 7f14a15ff700 1 -- 172.21.15.98:0/2954731960 >> 172.21.15.98:0/2954731960 conn(0x7f149c0cb7e0 msgr2=0x7f149c06bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.802+0000 7f14a15ff700 1 -- 172.21.15.98:0/2954731960 shutdown_connections 2023-03-03T07:55:55.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.803+0000 7f14a15ff700 1 -- 172.21.15.98:0/2954731960 wait complete. 2023-03-03T07:55:55.818 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.2 2023-03-03T07:55:55.834 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2023-03-03T07:55:55.837 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2023-03-03T07:55:55.846 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2023-03-03T07:55:55.848 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.6 2023-03-03T07:55:55.849 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2023-03-03T07:55:55.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.848+0000 7f6c709ff700 1 -- 172.21.15.98:0/3964328328 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f6c500055a0 con 0x7f6c6c0b0f10 2023-03-03T07:55:55.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.849+0000 7f6c5affd700 1 -- 172.21.15.98:0/3964328328 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6c5c015070 con 0x7f6c6c0b0f10 2023-03-03T07:55:55.851 INFO:teuthology.orchestra.run.smithi098.stdout:55834574854 2023-03-03T07:55:55.851 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.4 2023-03-03T07:55:55.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.851+0000 7f6c709ff700 1 -- 172.21.15.98:0/3964328328 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6c4c0683c0 msgr2=0x7f6c4c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.852+0000 7f6c709ff700 1 --2- 172.21.15.98:0/3964328328 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6c4c0683c0 0x7f6c4c06a870 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6c640012e0 tx=0x7f6c640016e0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.852+0000 7f6c709ff700 1 -- 172.21.15.98:0/3964328328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c6c0b0f10 msgr2=0x7f6c6c0c3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:55.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.852+0000 7f6c709ff700 1 --2- 172.21.15.98:0/3964328328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c6c0b0f10 0x7f6c6c0c3580 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c014040 tx=0x7f6c5c004750 comp rx=0 tx=0).stop 2023-03-03T07:55:55.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.852+0000 7f6c709ff700 1 -- 172.21.15.98:0/3964328328 shutdown_connections 2023-03-03T07:55:55.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.852+0000 7f6c709ff700 1 --2- 172.21.15.98:0/3964328328 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6c4c0683c0 0x7f6c4c06a870 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.852+0000 7f6c709ff700 1 --2- 172.21.15.98:0/3964328328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6c6c164620 0x7f6c6c0c4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.852+0000 7f6c709ff700 1 --2- 172.21.15.98:0/3964328328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c6c0b0f10 0x7f6c6c0c3580 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.852+0000 7f6c709ff700 1 --2- 172.21.15.98:0/3964328328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c164250 0x7f6c6c15cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:55.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.852+0000 7f6c709ff700 1 -- 172.21.15.98:0/3964328328 >> 172.21.15.98:0/3964328328 conn(0x7f6c6c0cb800 msgr2=0x7f6c6c0afe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:55.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.852+0000 7f6c709ff700 1 -- 172.21.15.98:0/3964328328 shutdown_connections 2023-03-03T07:55:55.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:55.852+0000 7f6c709ff700 1 -- 172.21.15.98:0/3964328328 wait complete. 2023-03-03T07:55:55.880 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.7 2023-03-03T07:55:56.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-03-03T07:55:56.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-03-03T07:55:56.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-03-03T07:55:56.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-03-03T07:55:56.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-03-03T07:55:56.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-03-03T07:55:56.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-03-03T07:55:56.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-03-03T07:55:57.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.220+0000 7fee1a9ff700 1 Processor -- start 2023-03-03T07:55:57.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.220+0000 7fee1a9ff700 1 -- start start 2023-03-03T07:55:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.224+0000 7fee1a9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fee1406d460 0x7fee140b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.224+0000 7fee1a9ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fee1406b820 0x7fee1415c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.224+0000 7fee1a9ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee140ce080 con 0x7fee140b0800 2023-03-03T07:55:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.224+0000 7fee1a9ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1406c730 con 0x7fee1406b820 2023-03-03T07:55:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.224+0000 7fee1a9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee140b1110 con 0x7fee1406d460 2023-03-03T07:55:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.225+0000 7fee137fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fee1406d460 0x7fee140b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.225+0000 7fee137fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fee1406d460 0x7fee140b0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:32782/0 (socket says 172.21.15.98:32782) 2023-03-03T07:55:57.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.225+0000 7fee137fe700 1 -- 172.21.15.98:0/1855021838 learned_addr learned my addr 172.21.15.98:0/1855021838 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:57.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.225+0000 7fee18f9c700 1 --2- 172.21.15.98:0/1855021838 >> v2:172.21.15.100:3300/0 conn(0x7fee1406b820 0x7fee1415c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.226+0000 7fee137fe700 1 -- 172.21.15.98:0/1855021838 >> v2:172.21.15.100:3300/0 conn(0x7fee1406b820 msgr2=0x7fee1415c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:57.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.226+0000 7fee137fe700 1 --2- 172.21.15.98:0/1855021838 >> v2:172.21.15.100:3300/0 conn(0x7fee1406b820 0x7fee1415c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.226+0000 7fee137fe700 1 -- 172.21.15.98:0/1855021838 >> v1:172.21.15.98:6789/0 conn(0x7fee140b0800 legacy=0x7fee140ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:57.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.226+0000 7fee137fe700 1 -- 172.21.15.98:0/1855021838 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1406c170 con 0x7fee1406d460 2023-03-03T07:55:57.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.227+0000 7fee137fe700 1 --2- 172.21.15.98:0/1855021838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fee1406d460 0x7fee140b0bd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00d370 tx=0x7fee0c00d680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2082ffaceeb3d202 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.228+0000 7fee12ffd700 1 -- 172.21.15.98:0/1855021838 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c010070 con 0x7fee1406d460 2023-03-03T07:55:57.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.228+0000 7fee12ffd700 1 -- 172.21.15.98:0/1855021838 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0c00dc20 con 0x7fee1406d460 2023-03-03T07:55:57.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.228+0000 7fee12ffd700 1 -- 172.21.15.98:0/1855021838 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c007b30 con 0x7fee1406d460 2023-03-03T07:55:57.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.229+0000 7fee1a9ff700 1 -- 172.21.15.98:0/1855021838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fee1406d460 msgr2=0x7fee140b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.229+0000 7fee1a9ff700 1 --2- 172.21.15.98:0/1855021838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fee1406d460 0x7fee140b0bd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00d370 tx=0x7fee0c00d680 comp rx=0 tx=0).stop 2023-03-03T07:55:57.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.230+0000 7fee1a9ff700 1 -- 172.21.15.98:0/1855021838 shutdown_connections 2023-03-03T07:55:57.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.230+0000 7fee1a9ff700 1 --2- 172.21.15.98:0/1855021838 >> v2:172.21.15.100:3300/0 conn(0x7fee1406b820 0x7fee1415c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.230+0000 7fee1a9ff700 1 --2- 172.21.15.98:0/1855021838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fee1406d460 0x7fee140b0bd0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.230+0000 7fee1a9ff700 1 -- 172.21.15.98:0/1855021838 >> 172.21.15.98:0/1855021838 conn(0x7fee140cb800 msgr2=0x7fee140cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.231+0000 7fee1a9ff700 1 -- 172.21.15.98:0/1855021838 shutdown_connections 2023-03-03T07:55:57.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.231+0000 7fee1a9ff700 1 -- 172.21.15.98:0/1855021838 wait complete. 2023-03-03T07:55:57.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.235+0000 7fee1a9ff700 1 Processor -- start 2023-03-03T07:55:57.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.235+0000 7fee1a9ff700 1 -- start start 2023-03-03T07:55:57.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.237+0000 7fee1a9ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fee1406b820 0x7fee140c48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.237+0000 7fee1a9ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee1406d460 0x7fee140b3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.237+0000 7fee1a9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fee140b0800 0x7fee141edda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.237+0000 7fee1a9ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee140b3f90 con 0x7fee1406d460 2023-03-03T07:55:57.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.237+0000 7fee1a9ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee140b4270 con 0x7fee1406b820 2023-03-03T07:55:57.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.237+0000 7fee1a9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee140b4550 con 0x7fee140b0800 2023-03-03T07:55:57.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.241+0000 7fee13fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fee1406b820 0x7fee140c48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.241+0000 7fee13fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fee1406b820 0x7fee140c48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:47036/0 (socket says 172.21.15.98:47036) 2023-03-03T07:55:57.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.241+0000 7fee13fff700 1 -- 172.21.15.98:0/2776757433 learned_addr learned my addr 172.21.15.98:0/2776757433 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:57.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.241+0000 7fee137fe700 1 --2- 172.21.15.98:0/2776757433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee1406d460 0x7fee140b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.241+0000 7fee18f9c700 1 --2- 172.21.15.98:0/2776757433 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fee140b0800 0x7fee141edda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.241+0000 7fee13fff700 1 -- 172.21.15.98:0/2776757433 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fee140b0800 msgr2=0x7fee141edda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.241+0000 7fee13fff700 1 --2- 172.21.15.98:0/2776757433 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fee140b0800 0x7fee141edda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.241+0000 7fee13fff700 1 -- 172.21.15.98:0/2776757433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee1406d460 msgr2=0x7fee140b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.241+0000 7fee13fff700 1 --2- 172.21.15.98:0/2776757433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee1406d460 0x7fee140b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.241+0000 7fee13fff700 1 -- 172.21.15.98:0/2776757433 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c00d050 con 0x7fee1406b820 2023-03-03T07:55:57.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.243+0000 7fee13fff700 1 --2- 172.21.15.98:0/2776757433 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fee1406b820 0x7fee140c48f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fee0400b470 tx=0x7fee0400b830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.245+0000 7fee10ff9700 1 -- 172.21.15.98:0/2776757433 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04011830 con 0x7fee1406b820 2023-03-03T07:55:57.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.245+0000 7fee10ff9700 1 -- 172.21.15.98:0/2776757433 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee04011990 con 0x7fee1406b820 2023-03-03T07:55:57.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.245+0000 7fee10ff9700 1 -- 172.21.15.98:0/2776757433 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04011d40 con 0x7fee1406b820 2023-03-03T07:55:57.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.245+0000 7fee1a9ff700 1 -- 172.21.15.98:0/2776757433 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee140cdd10 con 0x7fee1406b820 2023-03-03T07:55:57.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.245+0000 7fee1a9ff700 1 -- 172.21.15.98:0/2776757433 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1406c450 con 0x7fee1406b820 2023-03-03T07:55:57.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.249+0000 7fee1a9ff700 1 -- 172.21.15.98:0/2776757433 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf00052f0 con 0x7fee1406b820 2023-03-03T07:55:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.253+0000 7fee10ff9700 1 -- 172.21.15.98:0/2776757433 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fee04021070 con 0x7fee1406b820 2023-03-03T07:55:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.255+0000 7fee10ff9700 1 --2- 172.21.15.98:0/2776757433 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fedf8068390 0x7fedf806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.255+0000 7fee10ff9700 1 -- 172.21.15.98:0/2776757433 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fee04085e30 con 0x7fee1406b820 2023-03-03T07:55:57.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.258+0000 7fee137fe700 1 --2- 172.21.15.98:0/2776757433 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fedf8068390 0x7fedf806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.259+0000 7fee10ff9700 1 -- 172.21.15.98:0/2776757433 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fee040526d0 con 0x7fee1406b820 2023-03-03T07:55:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.291+0000 7fee137fe700 1 --2- 172.21.15.98:0/2776757433 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fedf8068390 0x7fedf806a840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fee0c000f80 tx=0x7fee0c012000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.318+0000 7f9ee0dff700 1 Processor -- start 2023-03-03T07:55:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.319+0000 7f9ee0dff700 1 -- start start 2023-03-03T07:55:57.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.319+0000 7f22693ff700 1 Processor -- start 2023-03-03T07:55:57.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.320+0000 7f22693ff700 1 -- start start 2023-03-03T07:55:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.326+0000 7f22693ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22640cc310 0x7f2264158c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.326+0000 7f22693ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f226406d460 0x7f22640ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.326+0000 7f22693ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22640ce080 con 0x7f226406b820 2023-03-03T07:55:57.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.326+0000 7f22693ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f226406c730 con 0x7f226406d460 2023-03-03T07:55:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.326+0000 7f22693ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22640cea50 con 0x7f22640cc310 2023-03-03T07:55:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.328+0000 7f2262ffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22640cc310 0x7f2264158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.328+0000 7f2262ffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22640cc310 0x7f2264158c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:32822/0 (socket says 172.21.15.98:32822) 2023-03-03T07:55:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.328+0000 7f2262ffd700 1 -- 172.21.15.98:0/3322294921 learned_addr learned my addr 172.21.15.98:0/3322294921 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.322+0000 7f9ee0dff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9edc0cc310 0x7f9edc158c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.322+0000 7f9ee0dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9edc06d460 0x7f9edc0ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.322+0000 7f9ee0dff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9edc0b0800 0x7f9edc0b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.322+0000 7f9ee0dff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9edc06c730 con 0x7f9edc0cc310 2023-03-03T07:55:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.322+0000 7f9ee0dff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9edc0b11e0 con 0x7f9edc0b0800 2023-03-03T07:55:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.322+0000 7f9ee0dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc0ce080 con 0x7f9edc06d460 2023-03-03T07:55:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.323+0000 7f9edad9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9edc0b0800 0x7f9edc0b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.323+0000 7f9edad9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9edc0b0800 0x7f9edc0b0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:47038/0 (socket says 172.21.15.98:47038) 2023-03-03T07:55:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.323+0000 7f9edad9d700 1 -- 172.21.15.98:0/3524506539 learned_addr learned my addr 172.21.15.98:0/3524506539 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.323+0000 7f9eda59c700 1 --2- 172.21.15.98:0/3524506539 >> v2:172.21.15.98:3300/0 conn(0x7f9edc0cc310 0x7f9edc158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.324+0000 7f9ed9d9b700 1 --2- 172.21.15.98:0/3524506539 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9edc06d460 0x7f9edc0ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.324+0000 7f9edad9d700 1 -- 172.21.15.98:0/3524506539 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9edc06d460 msgr2=0x7f9edc0ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:57.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.324+0000 7f9edad9d700 1 --2- 172.21.15.98:0/3524506539 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9edc06d460 0x7f9edc0ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.324+0000 7f9edad9d700 1 -- 172.21.15.98:0/3524506539 >> v2:172.21.15.98:3300/0 conn(0x7f9edc0cc310 msgr2=0x7f9edc158c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.324+0000 7f9edad9d700 1 --2- 172.21.15.98:0/3524506539 >> v2:172.21.15.98:3300/0 conn(0x7f9edc0cc310 0x7f9edc158c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.324+0000 7f9edad9d700 1 -- 172.21.15.98:0/3524506539 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc0cea50 con 0x7f9edc0b0800 2023-03-03T07:55:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.325+0000 7f9edad9d700 1 --2- 172.21.15.98:0/3524506539 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9edc0b0800 0x7f9edc0b0ca0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9edc06fbf0 tx=0x7f9ed400d430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59c6ff5e48348f81 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.326+0000 7f9ed959a700 1 -- 172.21.15.98:0/3524506539 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed400da10 con 0x7f9edc0b0800 2023-03-03T07:55:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.326+0000 7f9ed959a700 1 -- 172.21.15.98:0/3524506539 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed400db70 con 0x7f9edc0b0800 2023-03-03T07:55:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.326+0000 7f9ed959a700 1 -- 172.21.15.98:0/3524506539 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed40124d0 con 0x7f9edc0b0800 2023-03-03T07:55:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.327+0000 7f9ee0dff700 1 -- 172.21.15.98:0/3524506539 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9edc0b0800 msgr2=0x7f9edc0b0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.327+0000 7f9ee0dff700 1 --2- 172.21.15.98:0/3524506539 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9edc0b0800 0x7f9edc0b0ca0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9edc06fbf0 tx=0x7f9ed400d430 comp rx=0 tx=0).stop 2023-03-03T07:55:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.329+0000 7f22627fc700 1 --2- 172.21.15.98:0/3322294921 >> v2:172.21.15.98:3300/0 conn(0x7f226406d460 0x7f22640ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.330+0000 7f2261ffb700 1 -- 172.21.15.98:0/3322294921 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185411897 0 0) 0x7f22640ce080 con 0x7f226406b820 2023-03-03T07:55:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.330+0000 7f22627fc700 1 -- 172.21.15.98:0/3322294921 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22640cc310 msgr2=0x7f2264158c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.330+0000 7f22627fc700 1 --2- 172.21.15.98:0/3322294921 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22640cc310 0x7f2264158c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.330+0000 7f22627fc700 1 -- 172.21.15.98:0/3322294921 >> v1:172.21.15.98:6789/0 conn(0x7f226406b820 legacy=0x7f22640b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.330+0000 7f22627fc700 1 -- 172.21.15.98:0/3322294921 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226406c450 con 0x7f226406d460 2023-03-03T07:55:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.331+0000 7f9ee0dff700 1 -- 172.21.15.98:0/3524506539 shutdown_connections 2023-03-03T07:55:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.331+0000 7f9ee0dff700 1 --2- 172.21.15.98:0/3524506539 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9edc0b0800 0x7f9edc0b0ca0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.331+0000 7f9ee0dff700 1 --2- 172.21.15.98:0/3524506539 >> v2:172.21.15.98:3300/0 conn(0x7f9edc0cc310 0x7f9edc158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.331+0000 7f9ee0dff700 1 --2- 172.21.15.98:0/3524506539 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9edc06d460 0x7f9edc0ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.331+0000 7f9ee0dff700 1 -- 172.21.15.98:0/3524506539 >> 172.21.15.98:0/3524506539 conn(0x7f9edc0cb800 msgr2=0x7f9edc0cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.331+0000 7f9ee0dff700 1 -- 172.21.15.98:0/3524506539 shutdown_connections 2023-03-03T07:55:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.331+0000 7f9ee0dff700 1 -- 172.21.15.98:0/3524506539 wait complete. 2023-03-03T07:55:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.332+0000 7f22627fc700 1 --2- 172.21.15.98:0/3322294921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f226406d460 0x7f22640ce510 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2258009840 tx=0x7f2258009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e75a500c172ff40 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.332+0000 7f2261ffb700 1 -- 172.21.15.98:0/3322294921 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258011830 con 0x7f226406d460 2023-03-03T07:55:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.332+0000 7f2261ffb700 1 -- 172.21.15.98:0/3322294921 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2258011990 con 0x7f226406d460 2023-03-03T07:55:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.332+0000 7f2261ffb700 1 -- 172.21.15.98:0/3322294921 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258011d20 con 0x7f226406d460 2023-03-03T07:55:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.333+0000 7f22693ff700 1 -- 172.21.15.98:0/3322294921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f226406d460 msgr2=0x7f22640ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.333+0000 7f22693ff700 1 --2- 172.21.15.98:0/3322294921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f226406d460 0x7f22640ce510 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2258009840 tx=0x7f2258009b50 comp rx=0 tx=0).stop 2023-03-03T07:55:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.334+0000 7f22693ff700 1 -- 172.21.15.98:0/3322294921 shutdown_connections 2023-03-03T07:55:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.334+0000 7f22693ff700 1 --2- 172.21.15.98:0/3322294921 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22640cc310 0x7f2264158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.334+0000 7f22693ff700 1 --2- 172.21.15.98:0/3322294921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f226406d460 0x7f22640ce510 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.334+0000 7f22693ff700 1 -- 172.21.15.98:0/3322294921 >> 172.21.15.98:0/3322294921 conn(0x7f22640cb800 msgr2=0x7f22640cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.334+0000 7fd61dfff700 1 Processor -- start 2023-03-03T07:55:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.332+0000 7ff916bff700 1 Processor -- start 2023-03-03T07:55:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.333+0000 7ff916bff700 1 -- start start 2023-03-03T07:55:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.335+0000 7ff916bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9100b0800 0x7ff9100ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.335+0000 7ff916bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff91006b820 0x7ff91015c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.335+0000 7ff916bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9100ce080 con 0x7ff91006d460 2023-03-03T07:55:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.335+0000 7ff916bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff91006c730 con 0x7ff91006b820 2023-03-03T07:55:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.335+0000 7ff916bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9100cea50 con 0x7ff9100b0800 2023-03-03T07:55:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.334+0000 7f9ee0dff700 1 Processor -- start 2023-03-03T07:55:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.335+0000 7f9ee0dff700 1 -- start start 2023-03-03T07:55:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.336+0000 7f9ee0dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9edc06d460 0x7f9edc1fc880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7f9ee0dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9edc0b0800 0x7f9edc1e46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7f9ee0dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9edc0cc310 0x7f9edc0a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7f9ee0dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc1e5650 con 0x7f9edc0b0800 2023-03-03T07:55:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7f9ee0dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc0c73b0 con 0x7f9edc06d460 2023-03-03T07:55:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7f9ee0dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc1fcdc0 con 0x7f9edc0cc310 2023-03-03T07:55:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7f9eda59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9edc06d460 0x7f9edc1fc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7f9eda59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9edc06d460 0x7f9edc1fc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:47056/0 (socket says 172.21.15.98:47056) 2023-03-03T07:55:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7f9eda59c700 1 -- 172.21.15.98:0/3533720021 learned_addr learned my addr 172.21.15.98:0/3533720021 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.338+0000 7f9eda59c700 1 -- 172.21.15.98:0/3533720021 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9edc0cc310 msgr2=0x7f9edc0a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.338+0000 7f9eda59c700 1 --2- 172.21.15.98:0/3533720021 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9edc0cc310 0x7f9edc0a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.338+0000 7f9eda59c700 1 -- 172.21.15.98:0/3533720021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9edc0b0800 msgr2=0x7f9edc1e46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.338+0000 7f9eda59c700 1 --2- 172.21.15.98:0/3533720021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9edc0b0800 0x7f9edc1e46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.338+0000 7f9eda59c700 1 -- 172.21.15.98:0/3533720021 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed400bd40 con 0x7f9edc06d460 2023-03-03T07:55:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.339+0000 7f9eda59c700 1 --2- 172.21.15.98:0/3533720021 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9edc06d460 0x7f9edc1fc880 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00e870 tx=0x7f9ecc00ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.339+0000 7f9ecaffd700 1 -- 172.21.15.98:0/3533720021 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc009590 con 0x7f9edc06d460 2023-03-03T07:55:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.339+0000 7f9ecaffd700 1 -- 172.21.15.98:0/3533720021 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc00ee70 con 0x7f9edc06d460 2023-03-03T07:55:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.339+0000 7f9ecaffd700 1 -- 172.21.15.98:0/3533720021 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc00ca70 con 0x7f9edc06d460 2023-03-03T07:55:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7f4844bff700 1 Processor -- start 2023-03-03T07:55:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7f4844bff700 1 -- start start 2023-03-03T07:55:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.339+0000 7f4844bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f484006fe40 0x7f484006d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.334+0000 7fd61dfff700 1 -- start start 2023-03-03T07:55:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.336+0000 7fd61dfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd6180b8d10 0x7fd6180cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7fd61dfff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd6180b5310 0x7fd6180d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7fd61dfff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6180cd7d0 con 0x7fd6180b26d0 2023-03-03T07:55:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7fd61dfff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6180b90e0 con 0x7fd6180b8d10 2023-03-03T07:55:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7fd61dfff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6180cda20 con 0x7fd6180b5310 2023-03-03T07:55:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7fd616ffd700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fd6180b26d0 0x7fd6180b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:47824/0 (socket says 172.21.15.98:47824) 2023-03-03T07:55:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.338+0000 7fd616ffd700 1 -- 172.21.15.98:0/2059424858 learned_addr learned my addr 172.21.15.98:0/2059424858 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.338+0000 7fd617fff700 1 --2- 172.21.15.98:0/2059424858 >> v2:172.21.15.100:3300/0 conn(0x7fd6180b5310 0x7fd6180d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.338+0000 7fd6177fe700 1 --2- 172.21.15.98:0/2059424858 >> v2:172.21.15.98:3300/0 conn(0x7fd6180b8d10 0x7fd6180cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.339+0000 7fd6177fe700 1 -- 172.21.15.98:0/2059424858 >> v2:172.21.15.100:3300/0 conn(0x7fd6180b5310 msgr2=0x7fd6180d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.339+0000 7fd6177fe700 1 --2- 172.21.15.98:0/2059424858 >> v2:172.21.15.100:3300/0 conn(0x7fd6180b5310 0x7fd6180d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.339+0000 7fd6177fe700 1 -- 172.21.15.98:0/2059424858 >> v1:172.21.15.100:6789/0 conn(0x7fd6180b26d0 legacy=0x7fd6180b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.339+0000 7fd6177fe700 1 -- 172.21.15.98:0/2059424858 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6180cdd00 con 0x7fd6180b8d10 2023-03-03T07:55:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.336+0000 7ff91519c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff91006b820 0x7ff91015c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7ff91519c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff91006b820 0x7ff91015c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:47042/0 (socket says 172.21.15.98:47042) 2023-03-03T07:55:57.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7ff91519c700 1 -- 172.21.15.98:0/3240667753 learned_addr learned my addr 172.21.15.98:0/3240667753 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:57.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.340+0000 7f4844bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f48400ca910 0x7f484016dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.340+0000 7f22693ff700 1 -- 172.21.15.98:0/3322294921 shutdown_connections 2023-03-03T07:55:57.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.340+0000 7fd6177fe700 1 --2- 172.21.15.98:0/2059424858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6180b8d10 0x7fd6180cffc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd61806b740 tx=0x7fd610009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d7a412334eb42f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.340+0000 7f9ee0dff700 1 -- 172.21.15.98:0/3533720021 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9edc0cdd10 con 0x7f9edc06d460 2023-03-03T07:55:57.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.340+0000 7f9ee0dff700 1 -- 172.21.15.98:0/3533720021 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9edc06c450 con 0x7f9edc06d460 2023-03-03T07:55:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7ff91519c700 1 -- 172.21.15.98:0/3240667753 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9100b0800 msgr2=0x7ff9100ce510 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-03T07:55:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7ff91519c700 1 --2- 172.21.15.98:0/3240667753 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9100b0800 0x7ff9100ce510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7ff91519c700 1 -- 172.21.15.98:0/3240667753 >> v1:172.21.15.100:6789/0 conn(0x7ff91006d460 legacy=0x7ff9100b0bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-03T07:55:57.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.337+0000 7ff91519c700 1 -- 172.21.15.98:0/3240667753 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91006c170 con 0x7ff91006b820 2023-03-03T07:55:57.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.340+0000 7f4844bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484016d7b0 con 0x7f484016e180 2023-03-03T07:55:57.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.340+0000 7f4844bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f484016d480 con 0x7f484006fe40 2023-03-03T07:55:57.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.340+0000 7f4844bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48400cae80 con 0x7f48400ca910 2023-03-03T07:55:57.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.341+0000 7f483ed9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f484016e180 0x7f48400ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53678/0 (socket says 172.21.15.98:53678) 2023-03-03T07:55:57.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.341+0000 7f483ed9d700 1 -- 172.21.15.98:0/2758518017 learned_addr learned my addr 172.21.15.98:0/2758518017 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:57.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.341+0000 7fd6167fc700 1 -- 172.21.15.98:0/2059424858 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd610011830 con 0x7fd6180b8d10 2023-03-03T07:55:57.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.341+0000 7fd6167fc700 1 -- 172.21.15.98:0/2059424858 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd610011990 con 0x7fd6180b8d10 2023-03-03T07:55:57.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.341+0000 7fd6167fc700 1 -- 172.21.15.98:0/2059424858 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd610011d20 con 0x7fd6180b8d10 2023-03-03T07:55:57.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.342+0000 7fd61dfff700 1 -- 172.21.15.98:0/2059424858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6180b8d10 msgr2=0x7fd6180cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.342+0000 7fd61dfff700 1 --2- 172.21.15.98:0/2059424858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6180b8d10 0x7fd6180cffc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd61806b740 tx=0x7fd610009b20 comp rx=0 tx=0).stop 2023-03-03T07:55:57.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.343+0000 7fd61dfff700 1 -- 172.21.15.98:0/2059424858 shutdown_connections 2023-03-03T07:55:57.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.343+0000 7fd61dfff700 1 --2- 172.21.15.98:0/2059424858 >> v2:172.21.15.100:3300/0 conn(0x7fd6180b5310 0x7fd6180d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.343+0000 7fd61dfff700 1 --2- 172.21.15.98:0/2059424858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6180b8d10 0x7fd6180cffc0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.343+0000 7fd61dfff700 1 -- 172.21.15.98:0/2059424858 >> 172.21.15.98:0/2059424858 conn(0x7fd618073e70 msgr2=0x7fd61806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.343+0000 7fd61dfff700 1 -- 172.21.15.98:0/2059424858 shutdown_connections 2023-03-03T07:55:57.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.340+0000 7f22693ff700 1 -- 172.21.15.98:0/3322294921 wait complete. 2023-03-03T07:55:57.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.344+0000 7f22693ff700 1 Processor -- start 2023-03-03T07:55:57.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.344+0000 7f22693ff700 1 -- start start 2023-03-03T07:55:57.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.346+0000 7f22693ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f226406b820 0x7f22640a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.346+0000 7f22693ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f226406d460 0x7f22640a8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.346+0000 7f22693ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22640cc310 0x7f22640c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.346+0000 7f22693ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22640c7140 con 0x7f226406b820 2023-03-03T07:55:57.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.346+0000 7f22693ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22640c7420 con 0x7f226406d460 2023-03-03T07:55:57.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.346+0000 7f22693ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22640b21b0 con 0x7f22640cc310 2023-03-03T07:55:57.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.347+0000 7f2262ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f226406b820 0x7f22640a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.347+0000 7f2262ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f226406b820 0x7f22640a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41750/0 (socket says 172.21.15.98:41750) 2023-03-03T07:55:57.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.347+0000 7f2262ffd700 1 -- 172.21.15.98:0/1879150480 learned_addr learned my addr 172.21.15.98:0/1879150480 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:57.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.347+0000 7f22627fc700 1 --2- 172.21.15.98:0/1879150480 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f226406d460 0x7f22640a8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.347+0000 7f22637fe700 1 --2- 172.21.15.98:0/1879150480 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22640cc310 0x7f22640c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.348+0000 7f22627fc700 1 -- 172.21.15.98:0/1879150480 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22640cc310 msgr2=0x7f22640c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.348+0000 7f22627fc700 1 --2- 172.21.15.98:0/1879150480 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22640cc310 0x7f22640c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.348+0000 7f22627fc700 1 -- 172.21.15.98:0/1879150480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f226406b820 msgr2=0x7f22640a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.348+0000 7f22627fc700 1 --2- 172.21.15.98:0/1879150480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f226406b820 0x7f22640a84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.348+0000 7f22627fc700 1 -- 172.21.15.98:0/1879150480 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22580094a0 con 0x7f226406d460 2023-03-03T07:55:57.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.344+0000 7f9ecaffd700 1 -- 172.21.15.98:0/3533720021 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9ecc023070 con 0x7f9edc06d460 2023-03-03T07:55:57.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.346+0000 7f9ecaffd700 1 --2- 172.21.15.98:0/3533720021 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ebc068390 0x7f9ebc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.346+0000 7f9ecaffd700 1 -- 172.21.15.98:0/3533720021 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9ecc085e60 con 0x7f9edc06d460 2023-03-03T07:55:57.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.347+0000 7f9ed9d9b700 1 --2- 172.21.15.98:0/3533720021 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ebc068390 0x7f9ebc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.349+0000 7f9ed9d9b700 1 --2- 172.21.15.98:0/3533720021 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ebc068390 0x7f9ebc06a840 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009770 tx=0x7f9ed0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.344+0000 7f4835d9b700 1 --2- 172.21.15.98:0/2758518017 >> v2:172.21.15.100:3300/0 conn(0x7f48400ca910 0x7f484016dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.344+0000 7f483dd9b700 1 -- 172.21.15.98:0/2758518017 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510613054 0 0) 0x7f484016d7b0 con 0x7f484016e180 2023-03-03T07:55:57.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.344+0000 7f483dd9b700 1 -- 172.21.15.98:0/2758518017 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f482c0035b0 con 0x7f484016e180 2023-03-03T07:55:57.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.344+0000 7f483dd9b700 1 -- 172.21.15.98:0/2758518017 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2502637743 0 0) 0x7f482c0035b0 con 0x7f484016e180 2023-03-03T07:55:57.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.344+0000 7f483dd9b700 1 -- 172.21.15.98:0/2758518017 >> v2:172.21.15.100:3300/0 conn(0x7f48400ca910 msgr2=0x7f484016dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:57.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.344+0000 7f483dd9b700 1 --2- 172.21.15.98:0/2758518017 >> v2:172.21.15.100:3300/0 conn(0x7f48400ca910 0x7f484016dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.345+0000 7f483dd9b700 1 -- 172.21.15.98:0/2758518017 >> v2:172.21.15.98:3300/0 conn(0x7f484006fe40 msgr2=0x7f484006d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:57.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.345+0000 7f483dd9b700 1 --2- 172.21.15.98:0/2758518017 >> v2:172.21.15.98:3300/0 conn(0x7f484006fe40 0x7f484006d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.345+0000 7f483dd9b700 1 -- 172.21.15.98:0/2758518017 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48400cb160 con 0x7f484016e180 2023-03-03T07:55:57.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.345+0000 7f483dd9b700 1 -- 172.21.15.98:0/2758518017 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f48380019f0 con 0x7f484016e180 2023-03-03T07:55:57.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.345+0000 7f483dd9b700 1 -- 172.21.15.98:0/2758518017 >> v1:172.21.15.98:6789/0 conn(0x7f484016e180 legacy=0x7f48400ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.345+0000 7f483dd9b700 1 --2- 172.21.15.98:0/2758518017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f482c003a00 0x7f482c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.345+0000 7f483dd9b700 1 --2- 172.21.15.98:0/2758518017 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f482c004850 0x7f482c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.345+0000 7f483dd9b700 1 --2- 172.21.15.98:0/2758518017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f482c005200 0x7f482c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.345+0000 7f483dd9b700 1 -- 172.21.15.98:0/2758518017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484016d480 con 0x7f482c005200 2023-03-03T07:55:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.345+0000 7f483dd9b700 1 -- 172.21.15.98:0/2758518017 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48400cae80 con 0x7f482c004850 2023-03-03T07:55:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.345+0000 7f483dd9b700 1 -- 172.21.15.98:0/2758518017 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482c00c210 con 0x7f482c003a00 2023-03-03T07:55:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.346+0000 7f483ed9d700 1 --2- 172.21.15.98:0/2758518017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f482c005200 0x7f482c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.347+0000 7f483ed9d700 1 -- 172.21.15.98:0/2758518017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f482c003a00 msgr2=0x7f482c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.347+0000 7f483ed9d700 1 --2- 172.21.15.98:0/2758518017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f482c003a00 0x7f482c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.347+0000 7f483ed9d700 1 -- 172.21.15.98:0/2758518017 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f482c004850 msgr2=0x7f482c004cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T07:55:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.347+0000 7f483ed9d700 1 --2- 172.21.15.98:0/2758518017 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f482c004850 0x7f482c004cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.347+0000 7f483ed9d700 1 -- 172.21.15.98:0/2758518017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48400cb160 con 0x7f482c005200 2023-03-03T07:55:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.347+0000 7f483ed9d700 1 -- 172.21.15.98:0/2758518017 reap_dead start 2023-03-03T07:55:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.348+0000 7f483ed9d700 1 --2- 172.21.15.98:0/2758518017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f482c005200 0x7f482c00ba10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4838002230 tx=0x7f482c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f67af98b2d4253ca server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.343+0000 7fd61dfff700 1 -- 172.21.15.98:0/2059424858 wait complete. 2023-03-03T07:55:57.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.347+0000 7fd61dfff700 1 Processor -- start 2023-03-03T07:55:57.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.347+0000 7fd61dfff700 1 -- start start 2023-03-03T07:55:57.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.349+0000 7fd61dfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6180b26d0 0x7fd6181fc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.349+0000 7fd61dfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd6180b5310 0x7fd618153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.349+0000 7fd61dfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd6180b8d10 0x7fd618153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.349+0000 7fd61dfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6180c6ae0 con 0x7fd6180b26d0 2023-03-03T07:55:57.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.349+0000 7fd61dfff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd618155f10 con 0x7fd6180b8d10 2023-03-03T07:55:57.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.349+0000 7fd61dfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6181561f0 con 0x7fd6180b5310 2023-03-03T07:55:57.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.348+0000 7ff91519c700 1 --2- 172.21.15.98:0/3240667753 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91006b820 0x7ff91015c700 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff90800d370 tx=0x7ff90800d680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf62174f044b3cfc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7f22627fc700 1 --2- 172.21.15.98:0/1879150480 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f226406d460 0x7f22640a8a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2258016000 tx=0x7f2258004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7f9ee0dff700 1 -- 172.21.15.98:0/3533720021 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9edc06b820 con 0x7f9edc06d460 2023-03-03T07:55:57.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7fd617fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd6180b8d10 0x7fd618153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7fd617fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd6180b8d10 0x7fd618153750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:47092/0 (socket says 172.21.15.98:47092) 2023-03-03T07:55:57.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7fd617fff700 1 -- 172.21.15.98:0/2359514561 learned_addr learned my addr 172.21.15.98:0/2359514561 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:57.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7fd6177fe700 1 --2- 172.21.15.98:0/2359514561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6180b26d0 0x7fd6181fc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7fd616ffd700 1 --2- 172.21.15.98:0/2359514561 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd6180b5310 0x7fd618153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7f224b7fe700 1 -- 172.21.15.98:0/1879150480 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258004590 con 0x7f226406d460 2023-03-03T07:55:57.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7f224b7fe700 1 -- 172.21.15.98:0/1879150480 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225801f070 con 0x7f226406d460 2023-03-03T07:55:57.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7f224b7fe700 1 -- 172.21.15.98:0/1879150480 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22580038e0 con 0x7f226406d460 2023-03-03T07:55:57.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7f22693ff700 1 -- 172.21.15.98:0/1879150480 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226406bcc0 con 0x7f226406d460 2023-03-03T07:55:57.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.351+0000 7f22693ff700 1 -- 172.21.15.98:0/1879150480 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22640cdd10 con 0x7f226406d460 2023-03-03T07:55:57.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7ff90f7fe700 1 -- 172.21.15.98:0/3240667753 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff908010070 con 0x7ff91006b820 2023-03-03T07:55:57.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7ff90f7fe700 1 -- 172.21.15.98:0/3240667753 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff90800dcd0 con 0x7ff91006b820 2023-03-03T07:55:57.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7ff90f7fe700 1 -- 172.21.15.98:0/3240667753 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff908007d70 con 0x7ff91006b820 2023-03-03T07:55:57.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.351+0000 7f483dd9b700 1 -- 172.21.15.98:0/2758518017 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4838011880 con 0x7f482c005200 2023-03-03T07:55:57.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.351+0000 7f483dd9b700 1 -- 172.21.15.98:0/2758518017 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48380119e0 con 0x7f482c005200 2023-03-03T07:55:57.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.351+0000 7f483dd9b700 1 -- 172.21.15.98:0/2758518017 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4838011d70 con 0x7f482c005200 2023-03-03T07:55:57.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7fd617fff700 1 -- 172.21.15.98:0/2359514561 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd6180b5310 msgr2=0x7fd618153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.350+0000 7fd617fff700 1 --2- 172.21.15.98:0/2359514561 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd6180b5310 0x7fd618153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.351+0000 7fd617fff700 1 -- 172.21.15.98:0/2359514561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6180b26d0 msgr2=0x7fd6181fc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.351+0000 7fd617fff700 1 --2- 172.21.15.98:0/2359514561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6180b26d0 0x7fd6181fc930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.351+0000 7fd617fff700 1 -- 172.21.15.98:0/2359514561 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6100094a0 con 0x7fd6180b8d10 2023-03-03T07:55:57.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.351+0000 7fd617fff700 1 --2- 172.21.15.98:0/2359514561 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd6180b8d10 0x7fd618153750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd60000e7c0 tx=0x7fd60000ead0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.351+0000 7ff916bff700 1 -- 172.21.15.98:0/3240667753 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91006b820 msgr2=0x7ff91015c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.351+0000 7ff916bff700 1 --2- 172.21.15.98:0/3240667753 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91006b820 0x7ff91015c700 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff90800d370 tx=0x7ff90800d680 comp rx=0 tx=0).stop 2023-03-03T07:55:57.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.352+0000 7ff916bff700 1 -- 172.21.15.98:0/3240667753 shutdown_connections 2023-03-03T07:55:57.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.352+0000 7ff916bff700 1 --2- 172.21.15.98:0/3240667753 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91006b820 0x7ff91015c700 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.352+0000 7ff916bff700 1 --2- 172.21.15.98:0/3240667753 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9100b0800 0x7ff9100ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.352+0000 7ff916bff700 1 -- 172.21.15.98:0/3240667753 >> 172.21.15.98:0/3240667753 conn(0x7ff9100cb800 msgr2=0x7ff9100cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.352+0000 7f4844bff700 1 -- 172.21.15.98:0/2758518017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f482c005200 msgr2=0x7f482c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.352+0000 7f4844bff700 1 --2- 172.21.15.98:0/2758518017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f482c005200 0x7f482c00ba10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4838002230 tx=0x7f482c005880 comp rx=0 tx=0).stop 2023-03-03T07:55:57.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.352+0000 7ff916bff700 1 -- 172.21.15.98:0/3240667753 shutdown_connections 2023-03-03T07:55:57.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.352+0000 7f58a57ff700 1 Processor -- start 2023-03-03T07:55:57.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.353+0000 7f4844bff700 1 -- 172.21.15.98:0/2758518017 shutdown_connections 2023-03-03T07:55:57.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.353+0000 7f4844bff700 1 --2- 172.21.15.98:0/2758518017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f482c005200 0x7f482c00ba10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.353+0000 7f4844bff700 1 -- 172.21.15.98:0/2758518017 >> 172.21.15.98:0/2758518017 conn(0x7f48400cb800 msgr2=0x7f48400cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.353+0000 7f4844bff700 1 -- 172.21.15.98:0/2758518017 shutdown_connections 2023-03-03T07:55:57.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.353+0000 7f4844bff700 1 -- 172.21.15.98:0/2758518017 wait complete. 2023-03-03T07:55:57.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.354+0000 7f22693ff700 1 -- 172.21.15.98:0/1879150480 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22440052f0 con 0x7f226406d460 2023-03-03T07:55:57.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.352+0000 7ff916bff700 1 -- 172.21.15.98:0/3240667753 wait complete. 2023-03-03T07:55:57.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.353+0000 7f58a57ff700 1 -- start start 2023-03-03T07:55:57.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.355+0000 7f9ecaffd700 1 -- 172.21.15.98:0/3533720021 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9ecc052650 con 0x7f9edc06d460 2023-03-03T07:55:57.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.356+0000 7f4844bff700 1 Processor -- start 2023-03-03T07:55:57.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.356+0000 7fd60ffff700 1 -- 172.21.15.98:0/2359514561 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd600013680 con 0x7fd6180b8d10 2023-03-03T07:55:57.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.356+0000 7fd60ffff700 1 -- 172.21.15.98:0/2359514561 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6000137e0 con 0x7fd6180b8d10 2023-03-03T07:55:57.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.356+0000 7fd60ffff700 1 -- 172.21.15.98:0/2359514561 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd600013b50 con 0x7fd6180b8d10 2023-03-03T07:55:57.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.356+0000 7ff916bff700 1 Processor -- start 2023-03-03T07:55:57.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.355+0000 7f58a57ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f58a0164250 0x7f58a006c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.355+0000 7f58a57ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f58a0164620 0x7f58a00b0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.355+0000 7f58a57ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58a0071920 0x7f58a00b0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.355+0000 7f58a57ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a00cb5b0 con 0x7f58a0164250 2023-03-03T07:55:57.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.355+0000 7f58a57ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a0072170 con 0x7f58a0164620 2023-03-03T07:55:57.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.355+0000 7f58a57ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a00c2440 con 0x7f58a0071920 2023-03-03T07:55:57.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.356+0000 7f589f7fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58a0071920 0x7f58a00b0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.356+0000 7f589f7fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58a0071920 0x7f58a00b0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:32882/0 (socket says 172.21.15.98:32882) 2023-03-03T07:55:57.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.356+0000 7f589f7fe700 1 -- 172.21.15.98:0/2244908850 learned_addr learned my addr 172.21.15.98:0/2244908850 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:57.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.357+0000 7f58967fc700 1 --2- 172.21.15.98:0/2244908850 >> v2:172.21.15.100:3300/0 conn(0x7f58a0164620 0x7f58a00b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.358+0000 7f224b7fe700 1 -- 172.21.15.98:0/1879150480 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f225800f050 con 0x7f226406d460 2023-03-03T07:55:57.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.358+0000 7fd61dfff700 1 -- 172.21.15.98:0/2359514561 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6180cdfe0 con 0x7fd6180b8d10 2023-03-03T07:55:57.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.358+0000 7fd61dfff700 1 -- 172.21.15.98:0/2359514561 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6180d0e80 con 0x7fd6180b8d10 2023-03-03T07:55:57.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.359+0000 7f224b7fe700 1 --2- 172.21.15.98:0/1879150480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f22500683c0 0x7f225006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.359+0000 7f224b7fe700 1 -- 172.21.15.98:0/1879150480 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2258021070 con 0x7f226406d460 2023-03-03T07:55:57.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.357+0000 7ff916bff700 1 -- start start 2023-03-03T07:55:57.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.359+0000 7ff916bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91006b820 0x7ff910159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.360+0000 7ff916bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff91006d460 0x7ff91015aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.360+0000 7ff916bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9100b0800 0x7ff91015b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.360+0000 7ff916bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9101558d0 con 0x7ff9100b0800 2023-03-03T07:55:57.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.360+0000 7ff916bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91015b910 con 0x7ff91006b820 2023-03-03T07:55:57.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.360+0000 7ff916bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91015b630 con 0x7ff91006d460 2023-03-03T07:55:57.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.359+0000 7f589effd700 1 --2- 172.21.15.98:0/2244908850 >> v2:172.21.15.98:3300/0 conn(0x7f58a0164250 0x7f58a006c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.359+0000 7f589f7fe700 1 -- 172.21.15.98:0/2244908850 >> v2:172.21.15.100:3300/0 conn(0x7f58a0164620 msgr2=0x7f58a00b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:57.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.359+0000 7f589f7fe700 1 --2- 172.21.15.98:0/2244908850 >> v2:172.21.15.100:3300/0 conn(0x7f58a0164620 0x7f58a00b0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.359+0000 7f589f7fe700 1 -- 172.21.15.98:0/2244908850 >> v2:172.21.15.98:3300/0 conn(0x7f58a0164250 msgr2=0x7f58a006c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:57.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.359+0000 7f589f7fe700 1 --2- 172.21.15.98:0/2244908850 >> v2:172.21.15.98:3300/0 conn(0x7f58a0164250 0x7f58a006c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.359+0000 7f589f7fe700 1 -- 172.21.15.98:0/2244908850 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a0071e90 con 0x7f58a0071920 2023-03-03T07:55:57.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.360+0000 7ff91499b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91006b820 0x7ff910159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.360+0000 7ff91499b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91006b820 0x7ff910159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:47110/0 (socket says 172.21.15.98:47110) 2023-03-03T07:55:57.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.360+0000 7ff91499b700 1 -- 172.21.15.98:0/2884891096 learned_addr learned my addr 172.21.15.98:0/2884891096 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:57.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.360+0000 7f2262ffd700 1 --2- 172.21.15.98:0/1879150480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f22500683c0 0x7f225006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.361+0000 7ff90ffff700 1 --2- 172.21.15.98:0/2884891096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff91006d460 0x7ff91015aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.360+0000 7f4844bff700 1 -- start start 2023-03-03T07:55:57.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.361+0000 7f224b7fe700 1 -- 172.21.15.98:0/1879150480 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f22580541f0 con 0x7f226406d460 2023-03-03T07:55:57.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.362+0000 7ff91499b700 1 -- 172.21.15.98:0/2884891096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff91006d460 msgr2=0x7ff91015aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.362+0000 7ff91499b700 1 --2- 172.21.15.98:0/2884891096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff91006d460 0x7ff91015aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.362+0000 7ff91499b700 1 -- 172.21.15.98:0/2884891096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9100b0800 msgr2=0x7ff91015b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.362+0000 7ff91499b700 1 --2- 172.21.15.98:0/2884891096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9100b0800 0x7ff91015b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.362+0000 7ff91499b700 1 -- 172.21.15.98:0/2884891096 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90800d050 con 0x7ff91006b820 2023-03-03T07:55:57.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.362+0000 7f4844bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f482c005200 0x7f484016c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.362+0000 7f4844bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4840169ab0 0x7f484016b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.362+0000 7f4844bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48400c8140 0x7f48400c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.362+0000 7f4844bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484016a9e0 con 0x7f48400c8140 2023-03-03T07:55:57.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.362+0000 7f4844bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4840169f20 con 0x7f4840169ab0 2023-03-03T07:55:57.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.362+0000 7f4844bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48400b38f0 con 0x7f482c005200 2023-03-03T07:55:57.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.363+0000 7f483e59c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f482c005200 0x7f484016c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.363+0000 7f483e59c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f482c005200 0x7f484016c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:32892/0 (socket says 172.21.15.98:32892) 2023-03-03T07:55:57.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.363+0000 7f483e59c700 1 -- 172.21.15.98:0/1560902557 learned_addr learned my addr 172.21.15.98:0/1560902557 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:57.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.363+0000 7f4835d9b700 1 --2- 172.21.15.98:0/1560902557 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4840169ab0 0x7f484016b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.363+0000 7ff91499b700 1 --2- 172.21.15.98:0/2884891096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91006b820 0x7ff910159d70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff90000b470 tx=0x7ff90000b780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.361+0000 7f589f7fe700 1 --2- 172.21.15.98:0/2244908850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58a0071920 0x7f58a00b0d40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f58a00cc6e0 tx=0x7f589800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0f631672274bf7b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.362+0000 7f589e7fc700 1 -- 172.21.15.98:0/2244908850 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589800e070 con 0x7f58a0071920 2023-03-03T07:55:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.362+0000 7f589e7fc700 1 -- 172.21.15.98:0/2244908850 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589800b910 con 0x7f58a0071920 2023-03-03T07:55:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.362+0000 7f589e7fc700 1 -- 172.21.15.98:0/2244908850 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589800bcc0 con 0x7f58a0071920 2023-03-03T07:55:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.364+0000 7ff90d7fa700 1 -- 172.21.15.98:0/2884891096 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff900011830 con 0x7ff91006b820 2023-03-03T07:55:57.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.364+0000 7ff90d7fa700 1 -- 172.21.15.98:0/2884891096 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9000119b0 con 0x7ff91006b820 2023-03-03T07:55:57.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.364+0000 7ff90d7fa700 1 -- 172.21.15.98:0/2884891096 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff900011d60 con 0x7ff91006b820 2023-03-03T07:55:57.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.364+0000 7f483e59c700 1 -- 172.21.15.98:0/1560902557 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4840169ab0 msgr2=0x7f484016b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.364+0000 7f483e59c700 1 --2- 172.21.15.98:0/1560902557 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4840169ab0 0x7f484016b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.364+0000 7f483e59c700 1 -- 172.21.15.98:0/1560902557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48400c8140 msgr2=0x7f48400c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.364+0000 7f483e59c700 1 --2- 172.21.15.98:0/1560902557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48400c8140 0x7f48400c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.364+0000 7f483e59c700 1 -- 172.21.15.98:0/1560902557 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484006cb50 con 0x7f482c005200 2023-03-03T07:55:57.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.364+0000 7fd60ffff700 1 -- 172.21.15.98:0/2359514561 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd600019030 con 0x7fd6180b8d10 2023-03-03T07:55:57.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.364+0000 7ff916bff700 1 -- 172.21.15.98:0/2884891096 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9100cdd10 con 0x7ff91006b820 2023-03-03T07:55:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.365+0000 7f483e59c700 1 --2- 172.21.15.98:0/1560902557 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f482c005200 0x7f484016c530 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f483000b420 tx=0x7f483000b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.365+0000 7ff916bff700 1 -- 172.21.15.98:0/2884891096 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff91006c450 con 0x7ff91006b820 2023-03-03T07:55:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.364+0000 7f2262ffd700 1 --2- 172.21.15.98:0/1879150480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f22500683c0 0x7f225006a870 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f225c009e40 tx=0x7f225c00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.366+0000 7f48377fe700 1 -- 172.21.15.98:0/1560902557 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483000be60 con 0x7f482c005200 2023-03-03T07:55:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.366+0000 7f48377fe700 1 -- 172.21.15.98:0/1560902557 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4830004020 con 0x7f482c005200 2023-03-03T07:55:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.366+0000 7f48377fe700 1 -- 172.21.15.98:0/1560902557 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48300043b0 con 0x7f482c005200 2023-03-03T07:55:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.366+0000 7f4844bff700 1 -- 172.21.15.98:0/1560902557 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48400cb440 con 0x7f482c005200 2023-03-03T07:55:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.366+0000 7f4844bff700 1 -- 172.21.15.98:0/1560902557 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484006cce0 con 0x7f482c005200 2023-03-03T07:55:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.367+0000 7fd60ffff700 1 --2- 172.21.15.98:0/2359514561 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5f80683c0 0x7fd5f806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.367+0000 7fd60ffff700 1 -- 172.21.15.98:0/2359514561 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd600085a60 con 0x7fd6180b8d10 2023-03-03T07:55:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.368+0000 7fd61dfff700 1 -- 172.21.15.98:0/2359514561 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6180d0500 con 0x7fd6180b8d10 2023-03-03T07:55:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.370+0000 7ff90d7fa700 1 -- 172.21.15.98:0/2884891096 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff900021070 con 0x7ff91006b820 2023-03-03T07:55:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.371+0000 7f48377fe700 1 -- 172.21.15.98:0/1560902557 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f483000f050 con 0x7f482c005200 2023-03-03T07:55:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.370+0000 7fd6177fe700 1 --2- 172.21.15.98:0/2359514561 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5f80683c0 0x7fd5f806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.372+0000 7f4844bff700 1 -- 172.21.15.98:0/1560902557 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48240052f0 con 0x7f482c005200 2023-03-03T07:55:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.374+0000 7f48377fe700 1 --2- 172.21.15.98:0/1560902557 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48200683f0 0x7f482006a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.374+0000 7f48377fe700 1 -- 172.21.15.98:0/1560902557 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f48300855b0 con 0x7f482c005200 2023-03-03T07:55:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.375+0000 7f4835d9b700 1 --2- 172.21.15.98:0/1560902557 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48200683f0 0x7f482006a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.376+0000 7f4835d9b700 1 --2- 172.21.15.98:0/1560902557 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48200683f0 0x7f482006a8a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f484016bab0 tx=0x7f4828008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.371+0000 7ff916bff700 1 -- 172.21.15.98:0/2884891096 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff910158de0 con 0x7ff91006b820 2023-03-03T07:55:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.372+0000 7ff90d7fa700 1 --2- 172.21.15.98:0/2884891096 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff8f00683c0 0x7ff8f006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.372+0000 7ff90d7fa700 1 -- 172.21.15.98:0/2884891096 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff900085ae0 con 0x7ff91006b820 2023-03-03T07:55:57.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.373+0000 7ff90ffff700 1 --2- 172.21.15.98:0/2884891096 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff8f00683c0 0x7ff8f006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.375+0000 7ff90ffff700 1 --2- 172.21.15.98:0/2884891096 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff8f00683c0 0x7ff8f006a870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff910070f70 tx=0x7ff8f8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.370+0000 7f58a57ff700 1 -- 172.21.15.98:0/2244908850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58a0071920 msgr2=0x7f58a00b0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.370+0000 7f58a57ff700 1 --2- 172.21.15.98:0/2244908850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58a0071920 0x7f58a00b0d40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f58a00cc6e0 tx=0x7f589800b330 comp rx=0 tx=0).stop 2023-03-03T07:55:57.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.370+0000 7f58a57ff700 1 -- 172.21.15.98:0/2244908850 shutdown_connections 2023-03-03T07:55:57.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.370+0000 7f58a57ff700 1 --2- 172.21.15.98:0/2244908850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58a0071920 0x7f58a00b0d40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.370+0000 7f58a57ff700 1 --2- 172.21.15.98:0/2244908850 >> v2:172.21.15.98:3300/0 conn(0x7f58a0164250 0x7f58a006c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.370+0000 7f58a57ff700 1 --2- 172.21.15.98:0/2244908850 >> v2:172.21.15.100:3300/0 conn(0x7f58a0164620 0x7f58a00b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.370+0000 7f58a57ff700 1 -- 172.21.15.98:0/2244908850 >> 172.21.15.98:0/2244908850 conn(0x7f58a00cb800 msgr2=0x7f58a00cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.373+0000 7f58a57ff700 1 -- 172.21.15.98:0/2244908850 shutdown_connections 2023-03-03T07:55:57.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.373+0000 7f58a57ff700 1 -- 172.21.15.98:0/2244908850 wait complete. 2023-03-03T07:55:57.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.372+0000 7fd6177fe700 1 --2- 172.21.15.98:0/2359514561 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5f80683c0 0x7fd5f806a870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd6100090b0 tx=0x7fd610008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.373+0000 7fd60ffff700 1 -- 172.21.15.98:0/2359514561 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd600055c80 con 0x7fd6180b8d10 2023-03-03T07:55:57.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.377+0000 7f58a57ff700 1 Processor -- start 2023-03-03T07:55:57.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.378+0000 7f48377fe700 1 -- 172.21.15.98:0/1560902557 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4830051da0 con 0x7f482c005200 2023-03-03T07:55:57.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.378+0000 7ff90d7fa700 1 -- 172.21.15.98:0/2884891096 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff900055d00 con 0x7ff91006b820 2023-03-03T07:55:57.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.377+0000 7f58a57ff700 1 -- start start 2023-03-03T07:55:57.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.379+0000 7f58a57ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58a0071920 0x7f58a00c19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.379+0000 7f58a57ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a0164250 0x7f58a0153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.379+0000 7f58a57ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58a0164620 0x7f58a0154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.379+0000 7f58a57ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a00c5e50 con 0x7f58a0164250 2023-03-03T07:55:57.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.379+0000 7f58a57ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a01546c0 con 0x7f58a0164620 2023-03-03T07:55:57.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.379+0000 7f58a57ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a01549a0 con 0x7f58a0071920 2023-03-03T07:55:57.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.380+0000 7f589effd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58a0071920 0x7f58a00c19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.380+0000 7f589effd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58a0071920 0x7f58a00c19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:32902/0 (socket says 172.21.15.98:32902) 2023-03-03T07:55:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.380+0000 7f589effd700 1 -- 172.21.15.98:0/2276389269 learned_addr learned my addr 172.21.15.98:0/2276389269 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.380+0000 7f589f7fe700 1 --2- 172.21.15.98:0/2276389269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58a0164620 0x7f58a0154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.381+0000 7f58967fc700 1 --2- 172.21.15.98:0/2276389269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a0164250 0x7f58a0153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.382+0000 7f58967fc700 1 -- 172.21.15.98:0/2276389269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58a0071920 msgr2=0x7f58a00c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.382+0000 7f58967fc700 1 --2- 172.21.15.98:0/2276389269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58a0071920 0x7f58a00c19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.382+0000 7f58967fc700 1 -- 172.21.15.98:0/2276389269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58a0164620 msgr2=0x7f58a0154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.382+0000 7f58967fc700 1 --2- 172.21.15.98:0/2276389269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58a0164620 0x7f58a0154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.382+0000 7f58967fc700 1 -- 172.21.15.98:0/2276389269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5898009b30 con 0x7f58a0164250 2023-03-03T07:55:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.383+0000 7f58967fc700 1 --2- 172.21.15.98:0/2276389269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a0164250 0x7f58a0153b00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f589000e8d0 tx=0x7f589000ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.387+0000 7f5897fff700 1 -- 172.21.15.98:0/2276389269 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589000c690 con 0x7f58a0164250 2023-03-03T07:55:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.387+0000 7f5897fff700 1 -- 172.21.15.98:0/2276389269 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589000c7f0 con 0x7f58a0164250 2023-03-03T07:55:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.387+0000 7f5897fff700 1 -- 172.21.15.98:0/2276389269 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589000cb60 con 0x7f58a0164250 2023-03-03T07:55:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.387+0000 7f58a57ff700 1 -- 172.21.15.98:0/2276389269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a00cadb0 con 0x7f58a0164250 2023-03-03T07:55:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.387+0000 7f58a57ff700 1 -- 172.21.15.98:0/2276389269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a00caad0 con 0x7f58a0164250 2023-03-03T07:55:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.392+0000 7f58a57ff700 1 -- 172.21.15.98:0/2276389269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a00cb090 con 0x7f58a0164250 2023-03-03T07:55:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.400+0000 7f5897fff700 1 -- 172.21.15.98:0/2276389269 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5890019030 con 0x7f58a0164250 2023-03-03T07:55:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.402+0000 7f5897fff700 1 --2- 172.21.15.98:0/2276389269 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f58800683c0 0x7f588006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.402+0000 7f5897fff700 1 -- 172.21.15.98:0/2276389269 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5890085c60 con 0x7f58a0164250 2023-03-03T07:55:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.405+0000 7f5897fff700 1 -- 172.21.15.98:0/2276389269 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5890055df0 con 0x7f58a0164250 2023-03-03T07:55:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.405+0000 7f589effd700 1 --2- 172.21.15.98:0/2276389269 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f58800683c0 0x7f588006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.406+0000 7f589effd700 1 --2- 172.21.15.98:0/2276389269 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f58800683c0 0x7f588006a870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f58880046e0 tx=0x7f5888008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.421+0000 7f5ccf3ff700 1 Processor -- start 2023-03-03T07:55:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.421+0000 7f5ccf3ff700 1 -- start start 2023-03-03T07:55:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.424+0000 7f5ccf3ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5cc80b0800 0x7f5cc80ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.424+0000 7f5ccf3ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5cc806d460 0x7f5cc80b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.424+0000 7f5ccf3ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc80ce080 con 0x7f5cc806b820 2023-03-03T07:55:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.424+0000 7f5ccf3ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc806c730 con 0x7f5cc80b0800 2023-03-03T07:55:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.424+0000 7f5ccf3ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc80cea50 con 0x7f5cc806d460 2023-03-03T07:55:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.425+0000 7f5ccd19b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5cc80b0800 0x7f5cc80ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.425+0000 7f5ccd19b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5cc80b0800 0x7f5cc80ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41790/0 (socket says 172.21.15.98:41790) 2023-03-03T07:55:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.425+0000 7f5ccd19b700 1 -- 172.21.15.98:0/1336453985 learned_addr learned my addr 172.21.15.98:0/1336453985 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.426+0000 7f5ccd19b700 1 -- 172.21.15.98:0/1336453985 >> v2:172.21.15.100:3300/0 conn(0x7f5cc806d460 msgr2=0x7f5cc80b0bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-03T07:55:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.426+0000 7f5ccd19b700 1 --2- 172.21.15.98:0/1336453985 >> v2:172.21.15.100:3300/0 conn(0x7f5cc806d460 0x7f5cc80b0bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.426+0000 7f5ccd19b700 1 -- 172.21.15.98:0/1336453985 >> v1:172.21.15.100:6789/0 conn(0x7f5cc806b820 legacy=0x7f5cc815c700 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-03T07:55:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.426+0000 7f5ccd19b700 1 -- 172.21.15.98:0/1336453985 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc80b1110 con 0x7f5cc80b0800 2023-03-03T07:55:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.427+0000 7f5ccd19b700 1 --2- 172.21.15.98:0/1336453985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc80b0800 0x7f5cc80ce510 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5cc806f850 tx=0x7f5cc0010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62cf393e9a33352a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.429+0000 7f5cbffff700 1 -- 172.21.15.98:0/1336453985 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0003a20 con 0x7f5cc80b0800 2023-03-03T07:55:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.429+0000 7f5cbffff700 1 -- 172.21.15.98:0/1336453985 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc0003b80 con 0x7f5cc80b0800 2023-03-03T07:55:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.429+0000 7f5cbffff700 1 -- 172.21.15.98:0/1336453985 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc000a2e0 con 0x7f5cc80b0800 2023-03-03T07:55:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.430+0000 7f5ccf3ff700 1 -- 172.21.15.98:0/1336453985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc80b0800 msgr2=0x7f5cc80ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.430+0000 7f5ccf3ff700 1 --2- 172.21.15.98:0/1336453985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc80b0800 0x7f5cc80ce510 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5cc806f850 tx=0x7f5cc0010800 comp rx=0 tx=0).stop 2023-03-03T07:55:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.431+0000 7f5ccf3ff700 1 -- 172.21.15.98:0/1336453985 shutdown_connections 2023-03-03T07:55:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.431+0000 7f5ccf3ff700 1 --2- 172.21.15.98:0/1336453985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc80b0800 0x7f5cc80ce510 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.431+0000 7f5ccf3ff700 1 --2- 172.21.15.98:0/1336453985 >> v2:172.21.15.100:3300/0 conn(0x7f5cc806d460 0x7f5cc80b0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.431+0000 7f5ccf3ff700 1 -- 172.21.15.98:0/1336453985 >> 172.21.15.98:0/1336453985 conn(0x7f5cc80cb800 msgr2=0x7f5cc80cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.432+0000 7f5ccf3ff700 1 -- 172.21.15.98:0/1336453985 shutdown_connections 2023-03-03T07:55:57.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.432+0000 7f5ccf3ff700 1 -- 172.21.15.98:0/1336453985 wait complete. 2023-03-03T07:55:57.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.435+0000 7f5ccf3ff700 1 Processor -- start 2023-03-03T07:55:57.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.438+0000 7f5ccf3ff700 1 -- start start 2023-03-03T07:55:57.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.440+0000 7f5ccf3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5cc806b820 0x7f5cc80b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.440+0000 7f5ccf3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc806d460 0x7f5cc80c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.440+0000 7f5ccf3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cc80b0800 0x7f5cc80c3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.440+0000 7f5ccf3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc80b41c0 con 0x7f5cc806d460 2023-03-03T07:55:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.440+0000 7f5ccf3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc80b32d0 con 0x7f5cc806b820 2023-03-03T07:55:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.440+0000 7f5ccf3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc80b35b0 con 0x7f5cc80b0800 2023-03-03T07:55:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.442+0000 7f5ccd99c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cc80b0800 0x7f5cc80c3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.442+0000 7f5ccd99c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cc80b0800 0x7f5cc80c3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:32914/0 (socket says 172.21.15.98:32914) 2023-03-03T07:55:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.442+0000 7f5ccd99c700 1 -- 172.21.15.98:0/384734112 learned_addr learned my addr 172.21.15.98:0/384734112 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:57.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.443+0000 7f5ccc99a700 1 --2- 172.21.15.98:0/384734112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc806d460 0x7f5cc80c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.444+0000 7f5ccc99a700 1 -- 172.21.15.98:0/384734112 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cc80b0800 msgr2=0x7f5cc80c3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.444+0000 7f5ccc99a700 1 --2- 172.21.15.98:0/384734112 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cc80b0800 0x7f5cc80c3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.444+0000 7f5ccc99a700 1 -- 172.21.15.98:0/384734112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5cc806b820 msgr2=0x7f5cc80b4650 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T07:55:57.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.444+0000 7f5ccc99a700 1 --2- 172.21.15.98:0/384734112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5cc806b820 0x7f5cc80b4650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.444+0000 7f5ccc99a700 1 -- 172.21.15.98:0/384734112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc0007790 con 0x7f5cc806d460 2023-03-03T07:55:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.446+0000 7f5ccc99a700 1 --2- 172.21.15.98:0/384734112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc806d460 0x7f5cc80c8290 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8006fd0 tx=0x7f5cb800b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.447+0000 7f5cbdffb700 1 -- 172.21.15.98:0/384734112 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8011830 con 0x7f5cc806d460 2023-03-03T07:55:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.447+0000 7f5cbdffb700 1 -- 172.21.15.98:0/384734112 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb8011990 con 0x7f5cc806d460 2023-03-03T07:55:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.447+0000 7f5cbdffb700 1 -- 172.21.15.98:0/384734112 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8011d00 con 0x7f5cc806d460 2023-03-03T07:55:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.447+0000 7f5ccf3ff700 1 -- 172.21.15.98:0/384734112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc806c450 con 0x7f5cc806d460 2023-03-03T07:55:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.447+0000 7f5ccf3ff700 1 -- 172.21.15.98:0/384734112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc806c170 con 0x7f5cc806d460 2023-03-03T07:55:57.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.466+0000 7f5ccf3ff700 1 -- 172.21.15.98:0/384734112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca80052f0 con 0x7f5cc806d460 2023-03-03T07:55:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.468+0000 7f5cbdffb700 1 -- 172.21.15.98:0/384734112 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5cb8021070 con 0x7f5cc806d460 2023-03-03T07:55:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.471+0000 7f5cbdffb700 1 --2- 172.21.15.98:0/384734112 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5cac0683c0 0x7f5cac06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.471+0000 7f5cbdffb700 1 -- 172.21.15.98:0/384734112 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5cb8085b10 con 0x7f5cc806d460 2023-03-03T07:55:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.474+0000 7f5ccd19b700 1 --2- 172.21.15.98:0/384734112 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5cac0683c0 0x7f5cac06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.475+0000 7f5ccd19b700 1 --2- 172.21.15.98:0/384734112 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5cac0683c0 0x7f5cac06a870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0000f80 tx=0x7f5cc000d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.482+0000 7f5cbdffb700 1 -- 172.21.15.98:0/384734112 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5cb8052300 con 0x7f5cc806d460 2023-03-03T07:55:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.691+0000 7fee1a9ff700 1 -- 172.21.15.98:0/2776757433 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fedf0005870 con 0x7fee1406b820 2023-03-03T07:55:57.699 INFO:teuthology.orchestra.run.smithi098.stdout:55834574856 2023-03-03T07:55:57.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.694+0000 7fee10ff9700 1 -- 172.21.15.98:0/2776757433 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fee04015020 con 0x7fee1406b820 2023-03-03T07:55:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.700+0000 7fee1a9ff700 1 -- 172.21.15.98:0/2776757433 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fedf8068390 msgr2=0x7fedf806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.700+0000 7fee1a9ff700 1 --2- 172.21.15.98:0/2776757433 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fedf8068390 0x7fedf806a840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fee0c000f80 tx=0x7fee0c012000 comp rx=0 tx=0).stop 2023-03-03T07:55:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.700+0000 7fee1a9ff700 1 -- 172.21.15.98:0/2776757433 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fee1406b820 msgr2=0x7fee140c48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.700+0000 7fee1a9ff700 1 --2- 172.21.15.98:0/2776757433 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fee1406b820 0x7fee140c48f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fee0400b470 tx=0x7fee0400b830 comp rx=0 tx=0).stop 2023-03-03T07:55:57.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.704+0000 7fee1a9ff700 1 -- 172.21.15.98:0/2776757433 shutdown_connections 2023-03-03T07:55:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.704+0000 7fee1a9ff700 1 --2- 172.21.15.98:0/2776757433 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fedf8068390 0x7fedf806a840 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fee0c000f80 tx=0x7fee0c012000 comp rx=0 tx=0).stop 2023-03-03T07:55:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.704+0000 7fee1a9ff700 1 --2- 172.21.15.98:0/2776757433 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fee140b0800 0x7fee141edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.704+0000 7fee1a9ff700 1 --2- 172.21.15.98:0/2776757433 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fee1406b820 0x7fee140c48f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.704+0000 7fee1a9ff700 1 --2- 172.21.15.98:0/2776757433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee1406d460 0x7fee140b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.704+0000 7fee1a9ff700 1 -- 172.21.15.98:0/2776757433 >> 172.21.15.98:0/2776757433 conn(0x7fee140cb800 msgr2=0x7fee14070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.705+0000 7fee1a9ff700 1 -- 172.21.15.98:0/2776757433 shutdown_connections 2023-03-03T07:55:57.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.706+0000 7fee1a9ff700 1 -- 172.21.15.98:0/2776757433 wait complete. 2023-03-03T07:55:57.761 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2023-03-03T07:55:57.762 DEBUG:teuthology.parallel:result is None 2023-03-03T07:55:57.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.776+0000 7f9ee0dff700 1 -- 172.21.15.98:0/3533720021 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f9edc04e6e0 con 0x7f9edc06d460 2023-03-03T07:55:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.776+0000 7f9ecaffd700 1 -- 172.21.15.98:0/3533720021 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9ecc021020 con 0x7f9edc06d460 2023-03-03T07:55:57.779 INFO:teuthology.orchestra.run.smithi098.stdout:55834574855 2023-03-03T07:55:57.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.779+0000 7f9ee0dff700 1 -- 172.21.15.98:0/3533720021 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ebc068390 msgr2=0x7f9ebc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.779+0000 7f9ee0dff700 1 --2- 172.21.15.98:0/3533720021 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ebc068390 0x7f9ebc06a840 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009770 tx=0x7f9ed0008040 comp rx=0 tx=0).stop 2023-03-03T07:55:57.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.779+0000 7f9ee0dff700 1 -- 172.21.15.98:0/3533720021 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9edc06d460 msgr2=0x7f9edc1fc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.779+0000 7f9ee0dff700 1 --2- 172.21.15.98:0/3533720021 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9edc06d460 0x7f9edc1fc880 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00e870 tx=0x7f9ecc00ec30 comp rx=0 tx=0).stop 2023-03-03T07:55:57.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.780+0000 7f9ee0dff700 1 -- 172.21.15.98:0/3533720021 shutdown_connections 2023-03-03T07:55:57.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.780+0000 7f9ee0dff700 1 --2- 172.21.15.98:0/3533720021 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ebc068390 0x7f9ebc06a840 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.780+0000 7f9ee0dff700 1 --2- 172.21.15.98:0/3533720021 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9edc0cc310 0x7f9edc0a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.780+0000 7f9ee0dff700 1 --2- 172.21.15.98:0/3533720021 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9edc06d460 0x7f9edc1fc880 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.780+0000 7f9ee0dff700 1 --2- 172.21.15.98:0/3533720021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9edc0b0800 0x7f9edc1e46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.780+0000 7f9ee0dff700 1 -- 172.21.15.98:0/3533720021 >> 172.21.15.98:0/3533720021 conn(0x7f9edc0cb800 msgr2=0x7f9edc0af0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.780+0000 7f9ee0dff700 1 -- 172.21.15.98:0/3533720021 shutdown_connections 2023-03-03T07:55:57.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.780+0000 7f9ee0dff700 1 -- 172.21.15.98:0/3533720021 wait complete. 2023-03-03T07:55:57.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.794+0000 7f22693ff700 1 -- 172.21.15.98:0/1879150480 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f2244005870 con 0x7f226406d460 2023-03-03T07:55:57.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.795+0000 7f224b7fe700 1 -- 172.21.15.98:0/1879150480 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2258015090 con 0x7f226406d460 2023-03-03T07:55:57.797 INFO:teuthology.orchestra.run.smithi098.stdout:55834574855 2023-03-03T07:55:57.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.800+0000 7f22693ff700 1 -- 172.21.15.98:0/1879150480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f22500683c0 msgr2=0x7f225006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.800+0000 7f22693ff700 1 --2- 172.21.15.98:0/1879150480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f22500683c0 0x7f225006a870 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f225c009e40 tx=0x7f225c00b040 comp rx=0 tx=0).stop 2023-03-03T07:55:57.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.800+0000 7f22693ff700 1 -- 172.21.15.98:0/1879150480 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f226406d460 msgr2=0x7f22640a8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.800+0000 7f22693ff700 1 --2- 172.21.15.98:0/1879150480 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f226406d460 0x7f22640a8a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2258016000 tx=0x7f2258004750 comp rx=0 tx=0).stop 2023-03-03T07:55:57.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.800+0000 7f22693ff700 1 -- 172.21.15.98:0/1879150480 shutdown_connections 2023-03-03T07:55:57.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.800+0000 7f22693ff700 1 --2- 172.21.15.98:0/1879150480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f22500683c0 0x7f225006a870 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.800+0000 7f22693ff700 1 --2- 172.21.15.98:0/1879150480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f226406b820 0x7f22640a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.800+0000 7f22693ff700 1 --2- 172.21.15.98:0/1879150480 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22640cc310 0x7f22640c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.800+0000 7f22693ff700 1 --2- 172.21.15.98:0/1879150480 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f226406d460 0x7f22640a8a00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.800+0000 7f22693ff700 1 -- 172.21.15.98:0/1879150480 >> 172.21.15.98:0/1879150480 conn(0x7f22640cb800 msgr2=0x7f22640cc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.801+0000 7f22693ff700 1 -- 172.21.15.98:0/1879150480 shutdown_connections 2023-03-03T07:55:57.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.801+0000 7f22693ff700 1 -- 172.21.15.98:0/1879150480 wait complete. 2023-03-03T07:55:57.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.803+0000 7ff916bff700 1 -- 172.21.15.98:0/2884891096 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff91004e6e0 con 0x7ff91006b820 2023-03-03T07:55:57.805 INFO:teuthology.orchestra.run.smithi098.stdout:55834574854 2023-03-03T07:55:57.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.803+0000 7ff90d7fa700 1 -- 172.21.15.98:0/2884891096 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff900015090 con 0x7ff91006b820 2023-03-03T07:55:57.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.809+0000 7ff916bff700 1 -- 172.21.15.98:0/2884891096 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff8f00683c0 msgr2=0x7ff8f006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.809+0000 7ff916bff700 1 --2- 172.21.15.98:0/2884891096 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff8f00683c0 0x7ff8f006a870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff910070f70 tx=0x7ff8f8009190 comp rx=0 tx=0).stop 2023-03-03T07:55:57.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.809+0000 7ff916bff700 1 -- 172.21.15.98:0/2884891096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91006b820 msgr2=0x7ff910159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.809+0000 7ff916bff700 1 --2- 172.21.15.98:0/2884891096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91006b820 0x7ff910159d70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff90000b470 tx=0x7ff90000b780 comp rx=0 tx=0).stop 2023-03-03T07:55:57.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.810+0000 7ff916bff700 1 -- 172.21.15.98:0/2884891096 shutdown_connections 2023-03-03T07:55:57.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.810+0000 7ff916bff700 1 --2- 172.21.15.98:0/2884891096 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff8f00683c0 0x7ff8f006a870 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.810+0000 7ff916bff700 1 --2- 172.21.15.98:0/2884891096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9100b0800 0x7ff91015b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.810+0000 7ff916bff700 1 --2- 172.21.15.98:0/2884891096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91006b820 0x7ff910159d70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.810+0000 7ff916bff700 1 --2- 172.21.15.98:0/2884891096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff91006d460 0x7ff91015aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.810+0000 7ff916bff700 1 -- 172.21.15.98:0/2884891096 >> 172.21.15.98:0/2884891096 conn(0x7ff9100cb800 msgr2=0x7ff9100cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.810+0000 7ff916bff700 1 -- 172.21.15.98:0/2884891096 shutdown_connections 2023-03-03T07:55:57.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.810+0000 7ff916bff700 1 -- 172.21.15.98:0/2884891096 wait complete. 2023-03-03T07:55:57.814 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2023-03-03T07:55:57.815 DEBUG:teuthology.parallel:result is None 2023-03-03T07:55:57.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.815+0000 7fd61dfff700 1 -- 172.21.15.98:0/2359514561 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd61804e6e0 con 0x7fd6180b8d10 2023-03-03T07:55:57.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.816+0000 7fd60ffff700 1 -- 172.21.15.98:0/2359514561 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd600021030 con 0x7fd6180b8d10 2023-03-03T07:55:57.817 INFO:teuthology.orchestra.run.smithi098.stdout:55834574856 2023-03-03T07:55:57.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.820+0000 7fd61dfff700 1 -- 172.21.15.98:0/2359514561 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5f80683c0 msgr2=0x7fd5f806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.820+0000 7fd61dfff700 1 --2- 172.21.15.98:0/2359514561 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5f80683c0 0x7fd5f806a870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd6100090b0 tx=0x7fd610008030 comp rx=0 tx=0).stop 2023-03-03T07:55:57.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.820+0000 7fd61dfff700 1 -- 172.21.15.98:0/2359514561 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd6180b8d10 msgr2=0x7fd618153750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.820+0000 7fd61dfff700 1 --2- 172.21.15.98:0/2359514561 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd6180b8d10 0x7fd618153750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd60000e7c0 tx=0x7fd60000ead0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.820+0000 7fd61dfff700 1 -- 172.21.15.98:0/2359514561 shutdown_connections 2023-03-03T07:55:57.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.820+0000 7fd61dfff700 1 --2- 172.21.15.98:0/2359514561 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5f80683c0 0x7fd5f806a870 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.820+0000 7fd61dfff700 1 --2- 172.21.15.98:0/2359514561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6180b26d0 0x7fd6181fc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.820+0000 7fd61dfff700 1 --2- 172.21.15.98:0/2359514561 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd6180b8d10 0x7fd618153750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.820+0000 7fd61dfff700 1 --2- 172.21.15.98:0/2359514561 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd6180b5310 0x7fd618153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.820+0000 7fd61dfff700 1 -- 172.21.15.98:0/2359514561 >> 172.21.15.98:0/2359514561 conn(0x7fd618073e70 msgr2=0x7fd6180b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.822+0000 7fd61dfff700 1 -- 172.21.15.98:0/2359514561 shutdown_connections 2023-03-03T07:55:57.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.822+0000 7fd61dfff700 1 -- 172.21.15.98:0/2359514561 wait complete. 2023-03-03T07:55:57.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.826+0000 7f4844bff700 1 -- 172.21.15.98:0/1560902557 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f48240055a0 con 0x7f482c005200 2023-03-03T07:55:57.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.826+0000 7f48377fe700 1 -- 172.21.15.98:0/1560902557 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f48300553c0 con 0x7f482c005200 2023-03-03T07:55:57.829 INFO:teuthology.orchestra.run.smithi098.stdout:55834574854 2023-03-03T07:55:57.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.831+0000 7f4844bff700 1 -- 172.21.15.98:0/1560902557 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48200683f0 msgr2=0x7f482006a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.831+0000 7f4844bff700 1 --2- 172.21.15.98:0/1560902557 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48200683f0 0x7f482006a8a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f484016bab0 tx=0x7f4828008040 comp rx=0 tx=0).stop 2023-03-03T07:55:57.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.831+0000 7f4844bff700 1 -- 172.21.15.98:0/1560902557 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f482c005200 msgr2=0x7f484016c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.831+0000 7f4844bff700 1 --2- 172.21.15.98:0/1560902557 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f482c005200 0x7f484016c530 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f483000b420 tx=0x7f483000b730 comp rx=0 tx=0).stop 2023-03-03T07:55:57.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.831+0000 7f4844bff700 1 -- 172.21.15.98:0/1560902557 shutdown_connections 2023-03-03T07:55:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.831+0000 7f4844bff700 1 --2- 172.21.15.98:0/1560902557 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48200683f0 0x7f482006a8a0 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f484016bab0 tx=0x7f4828008040 comp rx=0 tx=0).stop 2023-03-03T07:55:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.831+0000 7f4844bff700 1 --2- 172.21.15.98:0/1560902557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48400c8140 0x7f48400c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.831+0000 7f4844bff700 1 --2- 172.21.15.98:0/1560902557 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f482c005200 0x7f484016c530 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.831+0000 7f4844bff700 1 --2- 172.21.15.98:0/1560902557 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4840169ab0 0x7f484016b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.831+0000 7f4844bff700 1 -- 172.21.15.98:0/1560902557 >> 172.21.15.98:0/1560902557 conn(0x7f48400cb800 msgr2=0x7f48400c5550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.834+0000 7f4844bff700 1 -- 172.21.15.98:0/1560902557 shutdown_connections 2023-03-03T07:55:57.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.834+0000 7f4844bff700 1 -- 172.21.15.98:0/1560902557 wait complete. 2023-03-03T07:55:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.845+0000 7f58a57ff700 1 -- 172.21.15.98:0/2276389269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f58a004e6e0 con 0x7f58a0164250 2023-03-03T07:55:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.846+0000 7f5897fff700 1 -- 172.21.15.98:0/2276389269 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5890021090 con 0x7f58a0164250 2023-03-03T07:55:57.851 INFO:teuthology.orchestra.run.smithi098.stdout:55834574854 2023-03-03T07:55:57.853 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2023-03-03T07:55:57.853 DEBUG:teuthology.parallel:result is None 2023-03-03T07:55:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.851+0000 7f58a57ff700 1 -- 172.21.15.98:0/2276389269 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f58800683c0 msgr2=0x7f588006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.851+0000 7f58a57ff700 1 --2- 172.21.15.98:0/2276389269 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f58800683c0 0x7f588006a870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f58880046e0 tx=0x7f5888008040 comp rx=0 tx=0).stop 2023-03-03T07:55:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.851+0000 7f58a57ff700 1 -- 172.21.15.98:0/2276389269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a0164250 msgr2=0x7f58a0153b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.851+0000 7f58a57ff700 1 --2- 172.21.15.98:0/2276389269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a0164250 0x7f58a0153b00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f589000e8d0 tx=0x7f589000ebe0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.851+0000 7f58a57ff700 1 -- 172.21.15.98:0/2276389269 shutdown_connections 2023-03-03T07:55:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.851+0000 7f58a57ff700 1 --2- 172.21.15.98:0/2276389269 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f58800683c0 0x7f588006a870 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.851+0000 7f58a57ff700 1 --2- 172.21.15.98:0/2276389269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58a0164620 0x7f58a0154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.851+0000 7f58a57ff700 1 --2- 172.21.15.98:0/2276389269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58a0071920 0x7f58a00c19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.851+0000 7f58a57ff700 1 --2- 172.21.15.98:0/2276389269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a0164250 0x7f58a0153b00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.851+0000 7f58a57ff700 1 -- 172.21.15.98:0/2276389269 >> 172.21.15.98:0/2276389269 conn(0x7f58a00cb800 msgr2=0x7f58a00cd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.851+0000 7f58a57ff700 1 -- 172.21.15.98:0/2276389269 shutdown_connections 2023-03-03T07:55:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.851+0000 7f58a57ff700 1 -- 172.21.15.98:0/2276389269 wait complete. 2023-03-03T07:55:57.857 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2023-03-03T07:55:57.857 DEBUG:teuthology.parallel:result is None 2023-03-03T07:55:57.859 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2023-03-03T07:55:57.859 DEBUG:teuthology.parallel:result is None 2023-03-03T07:55:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.874+0000 7f5ccf3ff700 1 -- 172.21.15.98:0/384734112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f5ca80055a0 con 0x7f5cc806d460 2023-03-03T07:55:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.874+0000 7f5cbdffb700 1 -- 172.21.15.98:0/384734112 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5cb8015090 con 0x7f5cc806d460 2023-03-03T07:55:57.876 INFO:teuthology.orchestra.run.smithi098.stdout:55834574855 2023-03-03T07:55:57.878 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2023-03-03T07:55:57.878 DEBUG:teuthology.parallel:result is None 2023-03-03T07:55:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.878+0000 7f5ccf3ff700 1 -- 172.21.15.98:0/384734112 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5cac0683c0 msgr2=0x7f5cac06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.878+0000 7f5ccf3ff700 1 --2- 172.21.15.98:0/384734112 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5cac0683c0 0x7f5cac06a870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0000f80 tx=0x7f5cc000d040 comp rx=0 tx=0).stop 2023-03-03T07:55:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.878+0000 7f5ccf3ff700 1 -- 172.21.15.98:0/384734112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc806d460 msgr2=0x7f5cc80c8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.878+0000 7f5ccf3ff700 1 --2- 172.21.15.98:0/384734112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc806d460 0x7f5cc80c8290 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8006fd0 tx=0x7f5cb800b7e0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.878+0000 7f5ccf3ff700 1 -- 172.21.15.98:0/384734112 shutdown_connections 2023-03-03T07:55:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.878+0000 7f5ccf3ff700 1 --2- 172.21.15.98:0/384734112 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5cac0683c0 0x7f5cac06a870 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.878+0000 7f5ccf3ff700 1 --2- 172.21.15.98:0/384734112 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cc80b0800 0x7f5cc80c3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.878+0000 7f5ccf3ff700 1 --2- 172.21.15.98:0/384734112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5cc806b820 0x7f5cc80b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.878+0000 7f5ccf3ff700 1 --2- 172.21.15.98:0/384734112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc806d460 0x7f5cc80c8290 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.878+0000 7f5ccf3ff700 1 -- 172.21.15.98:0/384734112 >> 172.21.15.98:0/384734112 conn(0x7f5cc80cb800 msgr2=0x7f5cc80affc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.878+0000 7f5ccf3ff700 1 -- 172.21.15.98:0/384734112 shutdown_connections 2023-03-03T07:55:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:57.878+0000 7f5ccf3ff700 1 -- 172.21.15.98:0/384734112 wait complete. 2023-03-03T07:55:57.894 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2023-03-03T07:55:57.894 DEBUG:teuthology.parallel:result is None 2023-03-03T07:55:57.909 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2023-03-03T07:55:57.910 DEBUG:teuthology.parallel:result is None 2023-03-03T07:55:57.910 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-03-03T07:55:57.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T07:55:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.190+0000 7fc1239ff700 1 Processor -- start 2023-03-03T07:55:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.191+0000 7fc1239ff700 1 -- start start 2023-03-03T07:55:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.192+0000 7fc1239ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc11c15cc00 0x7fc11c0cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.192+0000 7fc1239ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc11c0d2c60 0x7fc11c0cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.192+0000 7fc1239ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11c0cde10 con 0x7fc11c0d27c0 2023-03-03T07:55:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.192+0000 7fc1239ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc11c0ce040 con 0x7fc11c15cc00 2023-03-03T07:55:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.192+0000 7fc1239ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc11c0d30c0 con 0x7fc11c0d2c60 2023-03-03T07:55:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.192+0000 7fc12179b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc11c15cc00 0x7fc11c0cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.192+0000 7fc12179b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc11c15cc00 0x7fc11c0cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41818/0 (socket says 172.21.15.98:41818) 2023-03-03T07:55:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.192+0000 7fc12179b700 1 -- 172.21.15.98:0/2602939113 learned_addr learned my addr 172.21.15.98:0/2602939113 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.192+0000 7fc121f9c700 1 --2- 172.21.15.98:0/2602939113 >> v2:172.21.15.100:3300/0 conn(0x7fc11c0d2c60 0x7fc11c0cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.193+0000 7fc12179b700 1 -- 172.21.15.98:0/2602939113 >> v2:172.21.15.100:3300/0 conn(0x7fc11c0d2c60 msgr2=0x7fc11c0cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.193+0000 7fc12179b700 1 --2- 172.21.15.98:0/2602939113 >> v2:172.21.15.100:3300/0 conn(0x7fc11c0d2c60 0x7fc11c0cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.193+0000 7fc12179b700 1 -- 172.21.15.98:0/2602939113 >> v1:172.21.15.100:6789/0 conn(0x7fc11c0d27c0 legacy=0x7fc11c0b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.193+0000 7fc12179b700 1 -- 172.21.15.98:0/2602939113 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c0d1a00 con 0x7fc11c15cc00 2023-03-03T07:55:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.193+0000 7fc121f9c700 1 --2- 172.21.15.98:0/2602939113 >> v2:172.21.15.100:3300/0 conn(0x7fc11c0d2c60 0x7fc11c0cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.193+0000 7fc12179b700 1 --2- 172.21.15.98:0/2602939113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc11c15cc00 0x7fc11c0cd8d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0d4360 tx=0x7fc118009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3fc2a2cb0a17ef server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.193+0000 7fc113fff700 1 -- 172.21.15.98:0/2602939113 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11800b870 con 0x7fc11c15cc00 2023-03-03T07:55:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.193+0000 7fc113fff700 1 -- 172.21.15.98:0/2602939113 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11800b9d0 con 0x7fc11c15cc00 2023-03-03T07:55:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.193+0000 7fc113fff700 1 -- 172.21.15.98:0/2602939113 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11800bd60 con 0x7fc11c15cc00 2023-03-03T07:55:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.194+0000 7fc1239ff700 1 -- 172.21.15.98:0/2602939113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc11c15cc00 msgr2=0x7fc11c0cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.194+0000 7fc1239ff700 1 --2- 172.21.15.98:0/2602939113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc11c15cc00 0x7fc11c0cd8d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0d4360 tx=0x7fc118009bd0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.194+0000 7fc1239ff700 1 -- 172.21.15.98:0/2602939113 shutdown_connections 2023-03-03T07:55:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.194+0000 7fc1239ff700 1 --2- 172.21.15.98:0/2602939113 >> v2:172.21.15.100:3300/0 conn(0x7fc11c0d2c60 0x7fc11c0cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.194+0000 7fc1239ff700 1 --2- 172.21.15.98:0/2602939113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc11c15cc00 0x7fc11c0cd8d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.194+0000 7fc1239ff700 1 -- 172.21.15.98:0/2602939113 >> 172.21.15.98:0/2602939113 conn(0x7fc11c06aee0 msgr2=0x7fc11c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.194+0000 7fc1239ff700 1 -- 172.21.15.98:0/2602939113 shutdown_connections 2023-03-03T07:55:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.194+0000 7fc1239ff700 1 -- 172.21.15.98:0/2602939113 wait complete. 2023-03-03T07:55:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.196+0000 7fc1239ff700 1 Processor -- start 2023-03-03T07:55:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.196+0000 7fc1239ff700 1 -- start start 2023-03-03T07:55:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.197+0000 7fc1239ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc11c0d27c0 0x7fc11c1f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.197+0000 7fc1239ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc11c0d2c60 0x7fc11c0cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.197+0000 7fc1239ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc11c15cc00 0x7fc11c0c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.197+0000 7fc1239ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c1f2b80 con 0x7fc11c0d27c0 2023-03-03T07:55:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.197+0000 7fc1239ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c0b6e60 con 0x7fc11c0d2c60 2023-03-03T07:55:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.197+0000 7fc1239ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c0b7640 con 0x7fc11c15cc00 2023-03-03T07:55:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.198+0000 7fc12179b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc11c0d27c0 0x7fc11c1f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.198+0000 7fc12179b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc11c0d27c0 0x7fc11c1f2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41824/0 (socket says 172.21.15.98:41824) 2023-03-03T07:55:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.198+0000 7fc12179b700 1 -- 172.21.15.98:0/2411525927 learned_addr learned my addr 172.21.15.98:0/2411525927 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.198+0000 7fc121f9c700 1 --2- 172.21.15.98:0/2411525927 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc11c15cc00 0x7fc11c0c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.198+0000 7fc120f9a700 1 --2- 172.21.15.98:0/2411525927 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc11c0d2c60 0x7fc11c0cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.198+0000 7fc12179b700 1 -- 172.21.15.98:0/2411525927 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc11c15cc00 msgr2=0x7fc11c0c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.198+0000 7fc12179b700 1 --2- 172.21.15.98:0/2411525927 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc11c15cc00 0x7fc11c0c8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.198+0000 7fc12179b700 1 -- 172.21.15.98:0/2411525927 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc11c0d2c60 msgr2=0x7fc11c0cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.198+0000 7fc12179b700 1 --2- 172.21.15.98:0/2411525927 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc11c0d2c60 0x7fc11c0cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.198+0000 7fc12179b700 1 -- 172.21.15.98:0/2411525927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc118009550 con 0x7fc11c0d27c0 2023-03-03T07:55:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.198+0000 7fc12179b700 1 --2- 172.21.15.98:0/2411525927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc11c0d27c0 0x7fc11c1f2640 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc118000c00 tx=0x7fc118004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.199+0000 7fc111ffb700 1 -- 172.21.15.98:0/2411525927 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc118004520 con 0x7fc11c0d27c0 2023-03-03T07:55:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.199+0000 7fc111ffb700 1 -- 172.21.15.98:0/2411525927 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11801e070 con 0x7fc11c0d27c0 2023-03-03T07:55:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.199+0000 7fc111ffb700 1 -- 172.21.15.98:0/2411525927 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1180038a0 con 0x7fc11c0d27c0 2023-03-03T07:55:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.199+0000 7fc1239ff700 1 -- 172.21.15.98:0/2411525927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11c0d1fc0 con 0x7fc11c0d27c0 2023-03-03T07:55:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.199+0000 7fc1239ff700 1 -- 172.21.15.98:0/2411525927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11c0d1ce0 con 0x7fc11c0d27c0 2023-03-03T07:55:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.200+0000 7fc0ff7fe700 1 -- 172.21.15.98:0/2411525927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc11c0d22a0 con 0x7fc11c0d27c0 2023-03-03T07:55:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.201+0000 7fc111ffb700 1 -- 172.21.15.98:0/2411525927 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc11800f050 con 0x7fc11c0d27c0 2023-03-03T07:55:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.203+0000 7fc111ffb700 1 --2- 172.21.15.98:0/2411525927 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc1000682f0 0x7fc10006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.203+0000 7fc111ffb700 1 -- 172.21.15.98:0/2411525927 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc118020070 con 0x7fc11c0d27c0 2023-03-03T07:55:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.203+0000 7fc120f9a700 1 --2- 172.21.15.98:0/2411525927 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc1000682f0 0x7fc10006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.204+0000 7fc120f9a700 1 --2- 172.21.15.98:0/2411525927 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc1000682f0 0x7fc10006a7a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc10c000f60 tx=0x7fc10c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.207+0000 7fc111ffb700 1 -- 172.21.15.98:0/2411525927 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc118053b10 con 0x7fc11c0d27c0 2023-03-03T07:55:58.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.398+0000 7fc0ff7fe700 1 -- 172.21.15.98:0/2411525927 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc11c0729b0 con 0x7fc1000682f0 2023-03-03T07:55:58.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.406+0000 7fc111ffb700 1 -- 172.21.15.98:0/2411525927 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286503 (secure 0 0 0) 0x7fc11c0729b0 con 0x7fc1000682f0 2023-03-03T07:55:58.411 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:58.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.411+0000 7fc0ff7fe700 1 -- 172.21.15.98:0/2411525927 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc1000682f0 msgr2=0x7fc10006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:58.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.411+0000 7fc0ff7fe700 1 --2- 172.21.15.98:0/2411525927 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc1000682f0 0x7fc10006a7a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc10c000f60 tx=0x7fc10c009040 comp rx=0 tx=0).stop 2023-03-03T07:55:58.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.411+0000 7fc0ff7fe700 1 -- 172.21.15.98:0/2411525927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc11c0d27c0 msgr2=0x7fc11c1f2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:58.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.411+0000 7fc0ff7fe700 1 --2- 172.21.15.98:0/2411525927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc11c0d27c0 0x7fc11c1f2640 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc118000c00 tx=0x7fc118004750 comp rx=0 tx=0).stop 2023-03-03T07:55:58.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.411+0000 7fc0ff7fe700 1 -- 172.21.15.98:0/2411525927 shutdown_connections 2023-03-03T07:55:58.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.411+0000 7fc0ff7fe700 1 --2- 172.21.15.98:0/2411525927 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc1000682f0 0x7fc10006a7a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.411+0000 7fc0ff7fe700 1 --2- 172.21.15.98:0/2411525927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc11c0d27c0 0x7fc11c1f2640 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.411+0000 7fc0ff7fe700 1 --2- 172.21.15.98:0/2411525927 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc11c15cc00 0x7fc11c0c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.411+0000 7fc0ff7fe700 1 --2- 172.21.15.98:0/2411525927 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc11c0d2c60 0x7fc11c0cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.411+0000 7fc0ff7fe700 1 -- 172.21.15.98:0/2411525927 >> 172.21.15.98:0/2411525927 conn(0x7fc11c06aee0 msgr2=0x7fc11c159810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:58.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.411+0000 7fc0ff7fe700 1 -- 172.21.15.98:0/2411525927 shutdown_connections 2023-03-03T07:55:58.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.411+0000 7fc0ff7fe700 1 -- 172.21.15.98:0/2411525927 wait complete. 2023-03-03T07:55:58.415 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T07:55:58.440 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-03-03T07:55:58.181089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":66424,"kb_used_data":3192,"kb_used_omap":0,"kb_used_meta":63168,"kb_avail":754908296,"statfs":{"total":773094113280,"available":773026095104,"internally_reserved":0,"allocated":3268608,"data_stored":1941392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64684032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.075080"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.256834+0000","last_change":"2023-03-03T07:55:47.123170+0000","last_active":"2023-03-03T07:55:47.256834+0000","last_peered":"2023-03-03T07:55:47.256834+0000","last_clean":"2023-03-03T07:55:47.256834+0000","last_became_active":"2023-03-03T07:55:47.122733+0000","last_became_peered":"2023-03-03T07:55:47.122733+0000","last_unstale":"2023-03-03T07:55:47.256834+0000","last_undegraded":"2023-03-03T07:55:47.256834+0000","last_fullsized":"2023-03-03T07:55:47.256834+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:46.105554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263772+0000","last_change":"2023-03-03T07:55:47.130303+0000","last_active":"2023-03-03T07:55:47.263772+0000","last_peered":"2023-03-03T07:55:47.263772+0000","last_clean":"2023-03-03T07:55:47.263772+0000","last_became_active":"2023-03-03T07:55:47.129900+0000","last_became_peered":"2023-03-03T07:55:47.129900+0000","last_unstale":"2023-03-03T07:55:47.263772+0000","last_undegraded":"2023-03-03T07:55:47.263772+0000","last_fullsized":"2023-03-03T07:55:47.263772+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262173+0000","last_change":"2023-03-03T07:55:47.134218+0000","last_active":"2023-03-03T07:55:47.262173+0000","last_peered":"2023-03-03T07:55:47.262173+0000","last_clean":"2023-03-03T07:55:47.262173+0000","last_became_active":"2023-03-03T07:55:47.133821+0000","last_became_peered":"2023-03-03T07:55:47.133821+0000","last_unstale":"2023-03-03T07:55:47.262173+0000","last_undegraded":"2023-03-03T07:55:47.262173+0000","last_fullsized":"2023-03-03T07:55:47.262173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:15.870956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.265039+0000","last_change":"2023-03-03T07:55:47.116584+0000","last_active":"2023-03-03T07:55:47.265039+0000","last_peered":"2023-03-03T07:55:47.265039+0000","last_clean":"2023-03-03T07:55:47.265039+0000","last_became_active":"2023-03-03T07:55:47.116307+0000","last_became_peered":"2023-03-03T07:55:47.116307+0000","last_unstale":"2023-03-03T07:55:47.265039+0000","last_undegraded":"2023-03-03T07:55:47.265039+0000","last_fullsized":"2023-03-03T07:55:47.265039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:24.117627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259139+0000","last_change":"2023-03-03T07:55:47.120993+0000","last_active":"2023-03-03T07:55:47.259139+0000","last_peered":"2023-03-03T07:55:47.259139+0000","last_clean":"2023-03-03T07:55:47.259139+0000","last_became_active":"2023-03-03T07:55:47.119211+0000","last_became_peered":"2023-03-03T07:55:47.119211+0000","last_unstale":"2023-03-03T07:55:47.259139+0000","last_undegraded":"2023-03-03T07:55:47.259139+0000","last_fullsized":"2023-03-03T07:55:47.259139+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:34:01.733177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262076+0000","last_change":"2023-03-03T07:55:47.139910+0000","last_active":"2023-03-03T07:55:47.262076+0000","last_peered":"2023-03-03T07:55:47.262076+0000","last_clean":"2023-03-03T07:55:47.262076+0000","last_became_active":"2023-03-03T07:55:47.138659+0000","last_became_peered":"2023-03-03T07:55:47.138659+0000","last_unstale":"2023-03-03T07:55:47.262076+0000","last_undegraded":"2023-03-03T07:55:47.262076+0000","last_fullsized":"2023-03-03T07:55:47.262076+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:42:18.110771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263673+0000","last_change":"2023-03-03T07:55:47.136724+0000","last_active":"2023-03-03T07:55:47.263673+0000","last_peered":"2023-03-03T07:55:47.263673+0000","last_clean":"2023-03-03T07:55:47.263673+0000","last_became_active":"2023-03-03T07:55:47.136066+0000","last_became_peered":"2023-03-03T07:55:47.136066+0000","last_unstale":"2023-03-03T07:55:47.263673+0000","last_undegraded":"2023-03-03T07:55:47.263673+0000","last_fullsized":"2023-03-03T07:55:47.263673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263607+0000","last_change":"2023-03-03T07:55:47.137112+0000","last_active":"2023-03-03T07:55:47.263607+0000","last_peered":"2023-03-03T07:55:47.263607+0000","last_clean":"2023-03-03T07:55:47.263607+0000","last_became_active":"2023-03-03T07:55:47.136698+0000","last_became_peered":"2023-03-03T07:55:47.136698+0000","last_unstale":"2023-03-03T07:55:47.263607+0000","last_undegraded":"2023-03-03T07:55:47.263607+0000","last_fullsized":"2023-03-03T07:55:47.263607+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259277+0000","last_change":"2023-03-03T07:55:47.123107+0000","last_active":"2023-03-03T07:55:47.259277+0000","last_peered":"2023-03-03T07:55:47.259277+0000","last_clean":"2023-03-03T07:55:47.259277+0000","last_became_active":"2023-03-03T07:55:47.122842+0000","last_became_peered":"2023-03-03T07:55:47.122842+0000","last_unstale":"2023-03-03T07:55:47.259277+0000","last_undegraded":"2023-03-03T07:55:47.259277+0000","last_fullsized":"2023-03-03T07:55:47.259277+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:42:18.110771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263818+0000","last_change":"2023-03-03T07:55:47.115719+0000","last_active":"2023-03-03T07:55:47.263818+0000","last_peered":"2023-03-03T07:55:47.263818+0000","last_clean":"2023-03-03T07:55:47.263818+0000","last_became_active":"2023-03-03T07:55:47.115404+0000","last_became_peered":"2023-03-03T07:55:47.115404+0000","last_unstale":"2023-03-03T07:55:47.263818+0000","last_undegraded":"2023-03-03T07:55:47.263818+0000","last_fullsized":"2023-03-03T07:55:47.263818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:21.542695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262181+0000","last_change":"2023-03-03T07:55:47.135352+0000","last_active":"2023-03-03T07:55:47.262181+0000","last_peered":"2023-03-03T07:55:47.262181+0000","last_clean":"2023-03-03T07:55:47.262181+0000","last_became_active":"2023-03-03T07:55:47.135101+0000","last_became_peered":"2023-03-03T07:55:47.135101+0000","last_unstale":"2023-03-03T07:55:47.262181+0000","last_undegraded":"2023-03-03T07:55:47.262181+0000","last_fullsized":"2023-03-03T07:55:47.262181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:46:04.349618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.256802+0000","last_change":"2023-03-03T07:55:47.135595+0000","last_active":"2023-03-03T07:55:47.256802+0000","last_peered":"2023-03-03T07:55:47.256802+0000","last_clean":"2023-03-03T07:55:47.256802+0000","last_became_active":"2023-03-03T07:55:47.135349+0000","last_became_peered":"2023-03-03T07:55:47.135349+0000","last_unstale":"2023-03-03T07:55:47.256802+0000","last_undegraded":"2023-03-03T07:55:47.256802+0000","last_fullsized":"2023-03-03T07:55:47.256802+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:24.117627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.256771+0000","last_change":"2023-03-03T07:55:47.125072+0000","last_active":"2023-03-03T07:55:47.256771+0000","last_peered":"2023-03-03T07:55:47.256771+0000","last_clean":"2023-03-03T07:55:47.256771+0000","last_became_active":"2023-03-03T07:55:47.124767+0000","last_became_peered":"2023-03-03T07:55:47.124767+0000","last_unstale":"2023-03-03T07:55:47.256771+0000","last_undegraded":"2023-03-03T07:55:47.256771+0000","last_fullsized":"2023-03-03T07:55:47.256771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:34.910325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263657+0000","last_change":"2023-03-03T07:55:47.128730+0000","last_active":"2023-03-03T07:55:47.263657+0000","last_peered":"2023-03-03T07:55:47.263657+0000","last_clean":"2023-03-03T07:55:47.263657+0000","last_became_active":"2023-03-03T07:55:47.127578+0000","last_became_peered":"2023-03-03T07:55:47.127578+0000","last_unstale":"2023-03-03T07:55:47.263657+0000","last_undegraded":"2023-03-03T07:55:47.263657+0000","last_fullsized":"2023-03-03T07:55:47.263657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263702+0000","last_change":"2023-03-03T07:55:47.137768+0000","last_active":"2023-03-03T07:55:47.263702+0000","last_peered":"2023-03-03T07:55:47.263702+0000","last_clean":"2023-03-03T07:55:47.263702+0000","last_became_active":"2023-03-03T07:55:47.136957+0000","last_became_peered":"2023-03-03T07:55:47.136957+0000","last_unstale":"2023-03-03T07:55:47.263702+0000","last_undegraded":"2023-03-03T07:55:47.263702+0000","last_fullsized":"2023-03-03T07:55:47.263702+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262039+0000","last_change":"2023-03-03T07:55:47.139717+0000","last_active":"2023-03-03T07:55:47.262039+0000","last_peered":"2023-03-03T07:55:47.262039+0000","last_clean":"2023-03-03T07:55:47.262039+0000","last_became_active":"2023-03-03T07:55:47.138784+0000","last_became_peered":"2023-03-03T07:55:47.138784+0000","last_unstale":"2023-03-03T07:55:47.262039+0000","last_undegraded":"2023-03-03T07:55:47.262039+0000","last_fullsized":"2023-03-03T07:55:47.262039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:34:01.733177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259293+0000","last_change":"2023-03-03T07:55:47.121662+0000","last_active":"2023-03-03T07:55:47.259293+0000","last_peered":"2023-03-03T07:55:47.259293+0000","last_clean":"2023-03-03T07:55:47.259293+0000","last_became_active":"2023-03-03T07:55:47.118420+0000","last_became_peered":"2023-03-03T07:55:47.118420+0000","last_unstale":"2023-03-03T07:55:47.259293+0000","last_undegraded":"2023-03-03T07:55:47.259293+0000","last_fullsized":"2023-03-03T07:55:47.259293+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:36:18.587660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262199+0000","last_change":"2023-03-03T07:55:47.132197+0000","last_active":"2023-03-03T07:55:47.262199+0000","last_peered":"2023-03-03T07:55:47.262199+0000","last_clean":"2023-03-03T07:55:47.262199+0000","last_became_active":"2023-03-03T07:55:47.131599+0000","last_became_peered":"2023-03-03T07:55:47.131599+0000","last_unstale":"2023-03-03T07:55:47.262199+0000","last_undegraded":"2023-03-03T07:55:47.262199+0000","last_fullsized":"2023-03-03T07:55:47.262199+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:57:41.948092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.256819+0000","last_change":"2023-03-03T07:55:47.129930+0000","last_active":"2023-03-03T07:55:47.256819+0000","last_peered":"2023-03-03T07:55:47.256819+0000","last_clean":"2023-03-03T07:55:47.256819+0000","last_became_active":"2023-03-03T07:55:47.129569+0000","last_became_peered":"2023-03-03T07:55:47.129569+0000","last_unstale":"2023-03-03T07:55:47.256819+0000","last_undegraded":"2023-03-03T07:55:47.256819+0000","last_fullsized":"2023-03-03T07:55:47.256819+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:40.146598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262146+0000","last_change":"2023-03-03T07:55:47.133303+0000","last_active":"2023-03-03T07:55:47.262146+0000","last_peered":"2023-03-03T07:55:47.262146+0000","last_clean":"2023-03-03T07:55:47.262146+0000","last_became_active":"2023-03-03T07:55:47.131841+0000","last_became_peered":"2023-03-03T07:55:47.131841+0000","last_unstale":"2023-03-03T07:55:47.262146+0000","last_undegraded":"2023-03-03T07:55:47.262146+0000","last_fullsized":"2023-03-03T07:55:47.262146+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:13:04.772369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263745+0000","last_change":"2023-03-03T07:55:47.134085+0000","last_active":"2023-03-03T07:55:47.263745+0000","last_peered":"2023-03-03T07:55:47.263745+0000","last_clean":"2023-03-03T07:55:47.263745+0000","last_became_active":"2023-03-03T07:55:47.133590+0000","last_became_peered":"2023-03-03T07:55:47.133590+0000","last_unstale":"2023-03-03T07:55:47.263745+0000","last_undegraded":"2023-03-03T07:55:47.263745+0000","last_fullsized":"2023-03-03T07:55:47.263745+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259059+0000","last_change":"2023-03-03T07:55:47.128927+0000","last_active":"2023-03-03T07:55:47.259059+0000","last_peered":"2023-03-03T07:55:47.259059+0000","last_clean":"2023-03-03T07:55:47.259059+0000","last_became_active":"2023-03-03T07:55:47.128642+0000","last_became_peered":"2023-03-03T07:55:47.128642+0000","last_unstale":"2023-03-03T07:55:47.259059+0000","last_undegraded":"2023-03-03T07:55:47.259059+0000","last_fullsized":"2023-03-03T07:55:47.259059+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261331+0000","last_change":"2023-03-03T07:55:47.114026+0000","last_active":"2023-03-03T07:55:47.261331+0000","last_peered":"2023-03-03T07:55:47.261331+0000","last_clean":"2023-03-03T07:55:47.261331+0000","last_became_active":"2023-03-03T07:55:47.113732+0000","last_became_peered":"2023-03-03T07:55:47.113732+0000","last_unstale":"2023-03-03T07:55:47.261331+0000","last_undegraded":"2023-03-03T07:55:47.261331+0000","last_fullsized":"2023-03-03T07:55:47.261331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:34:01.733177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263655+0000","last_change":"2023-03-03T07:55:47.137565+0000","last_active":"2023-03-03T07:55:47.263655+0000","last_peered":"2023-03-03T07:55:47.263655+0000","last_clean":"2023-03-03T07:55:47.263655+0000","last_became_active":"2023-03-03T07:55:47.136713+0000","last_became_peered":"2023-03-03T07:55:47.136713+0000","last_unstale":"2023-03-03T07:55:47.263655+0000","last_undegraded":"2023-03-03T07:55:47.263655+0000","last_fullsized":"2023-03-03T07:55:47.263655+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259593+0000","last_change":"2023-03-03T07:55:47.124622+0000","last_active":"2023-03-03T07:55:47.259593+0000","last_peered":"2023-03-03T07:55:47.259593+0000","last_clean":"2023-03-03T07:55:47.259593+0000","last_became_active":"2023-03-03T07:55:47.124221+0000","last_became_peered":"2023-03-03T07:55:47.124221+0000","last_unstale":"2023-03-03T07:55:47.259593+0000","last_undegraded":"2023-03-03T07:55:47.259593+0000","last_fullsized":"2023-03-03T07:55:47.259593+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259563+0000","last_change":"2023-03-03T07:55:47.123628+0000","last_active":"2023-03-03T07:55:47.259563+0000","last_peered":"2023-03-03T07:55:47.259563+0000","last_clean":"2023-03-03T07:55:47.259563+0000","last_became_active":"2023-03-03T07:55:47.123275+0000","last_became_peered":"2023-03-03T07:55:47.123275+0000","last_unstale":"2023-03-03T07:55:47.259563+0000","last_undegraded":"2023-03-03T07:55:47.259563+0000","last_fullsized":"2023-03-03T07:55:47.259563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:21.542695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263729+0000","last_change":"2023-03-03T07:55:47.127078+0000","last_active":"2023-03-03T07:55:47.263729+0000","last_peered":"2023-03-03T07:55:47.263729+0000","last_clean":"2023-03-03T07:55:47.263729+0000","last_became_active":"2023-03-03T07:55:47.126560+0000","last_became_peered":"2023-03-03T07:55:47.126560+0000","last_unstale":"2023-03-03T07:55:47.263729+0000","last_undegraded":"2023-03-03T07:55:47.263729+0000","last_fullsized":"2023-03-03T07:55:47.263729+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262095+0000","last_change":"2023-03-03T07:55:46.566968+0000","last_active":"2023-03-03T07:55:47.262095+0000","last_peered":"2023-03-03T07:55:47.262095+0000","last_clean":"2023-03-03T07:55:47.262095+0000","last_became_active":"2023-03-03T07:55:46.566628+0000","last_became_peered":"2023-03-03T07:55:46.566628+0000","last_unstale":"2023-03-03T07:55:47.262095+0000","last_undegraded":"2023-03-03T07:55:47.262095+0000","last_fullsized":"2023-03-03T07:55:47.262095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:11.895405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261401+0000","last_change":"2023-03-03T07:55:47.124330+0000","last_active":"2023-03-03T07:55:47.261401+0000","last_peered":"2023-03-03T07:55:47.261401+0000","last_clean":"2023-03-03T07:55:47.261401+0000","last_became_active":"2023-03-03T07:55:47.124015+0000","last_became_peered":"2023-03-03T07:55:47.124015+0000","last_unstale":"2023-03-03T07:55:47.261401+0000","last_undegraded":"2023-03-03T07:55:47.261401+0000","last_fullsized":"2023-03-03T07:55:47.261401+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:57:41.948092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259432+0000","last_change":"2023-03-03T07:55:46.562055+0000","last_active":"2023-03-03T07:55:47.259432+0000","last_peered":"2023-03-03T07:55:47.259432+0000","last_clean":"2023-03-03T07:55:47.259432+0000","last_became_active":"2023-03-03T07:55:46.561625+0000","last_became_peered":"2023-03-03T07:55:46.561625+0000","last_unstale":"2023-03-03T07:55:47.259432+0000","last_undegraded":"2023-03-03T07:55:47.259432+0000","last_fullsized":"2023-03-03T07:55:47.259432+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259192+0000","last_change":"2023-03-03T07:55:47.129628+0000","last_active":"2023-03-03T07:55:47.259192+0000","last_peered":"2023-03-03T07:55:47.259192+0000","last_clean":"2023-03-03T07:55:47.259192+0000","last_became_active":"2023-03-03T07:55:47.129356+0000","last_became_peered":"2023-03-03T07:55:47.129356+0000","last_unstale":"2023-03-03T07:55:47.259192+0000","last_undegraded":"2023-03-03T07:55:47.259192+0000","last_fullsized":"2023-03-03T07:55:47.259192+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261268+0000","last_change":"2023-03-03T07:55:46.559242+0000","last_active":"2023-03-03T07:55:47.261268+0000","last_peered":"2023-03-03T07:55:47.261268+0000","last_clean":"2023-03-03T07:55:47.261268+0000","last_became_active":"2023-03-03T07:55:46.558772+0000","last_became_peered":"2023-03-03T07:55:46.558772+0000","last_unstale":"2023-03-03T07:55:47.261268+0000","last_undegraded":"2023-03-03T07:55:47.261268+0000","last_fullsized":"2023-03-03T07:55:47.261268+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.265064+0000","last_change":"2023-03-03T07:55:47.114645+0000","last_active":"2023-03-03T07:55:47.265064+0000","last_peered":"2023-03-03T07:55:47.265064+0000","last_clean":"2023-03-03T07:55:47.265064+0000","last_became_active":"2023-03-03T07:55:47.114306+0000","last_became_peered":"2023-03-03T07:55:47.114306+0000","last_unstale":"2023-03-03T07:55:47.265064+0000","last_undegraded":"2023-03-03T07:55:47.265064+0000","last_fullsized":"2023-03-03T07:55:47.265064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:29.066787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259309+0000","last_change":"2023-03-03T07:55:46.567843+0000","last_active":"2023-03-03T07:55:47.259309+0000","last_peered":"2023-03-03T07:55:47.259309+0000","last_clean":"2023-03-03T07:55:47.259309+0000","last_became_active":"2023-03-03T07:55:46.567536+0000","last_became_peered":"2023-03-03T07:55:46.567536+0000","last_unstale":"2023-03-03T07:55:47.259309+0000","last_undegraded":"2023-03-03T07:55:47.259309+0000","last_fullsized":"2023-03-03T07:55:47.259309+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261252+0000","last_change":"2023-03-03T07:55:47.126080+0000","last_active":"2023-03-03T07:55:47.261252+0000","last_peered":"2023-03-03T07:55:47.261252+0000","last_clean":"2023-03-03T07:55:47.261252+0000","last_became_active":"2023-03-03T07:55:47.125819+0000","last_became_peered":"2023-03-03T07:55:47.125819+0000","last_unstale":"2023-03-03T07:55:47.261252+0000","last_undegraded":"2023-03-03T07:55:47.261252+0000","last_fullsized":"2023-03-03T07:55:47.261252+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:42:18.110771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263807+0000","last_change":"2023-03-03T07:55:46.564685+0000","last_active":"2023-03-03T07:55:47.263807+0000","last_peered":"2023-03-03T07:55:47.263807+0000","last_clean":"2023-03-03T07:55:47.263807+0000","last_became_active":"2023-03-03T07:55:46.564401+0000","last_became_peered":"2023-03-03T07:55:46.564401+0000","last_unstale":"2023-03-03T07:55:47.263807+0000","last_undegraded":"2023-03-03T07:55:47.263807+0000","last_fullsized":"2023-03-03T07:55:47.263807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261322+0000","last_change":"2023-03-03T07:55:47.126626+0000","last_active":"2023-03-03T07:55:47.261322+0000","last_peered":"2023-03-03T07:55:47.261322+0000","last_clean":"2023-03-03T07:55:47.261322+0000","last_became_active":"2023-03-03T07:55:47.126341+0000","last_became_peered":"2023-03-03T07:55:47.126341+0000","last_unstale":"2023-03-03T07:55:47.261322+0000","last_undegraded":"2023-03-03T07:55:47.261322+0000","last_fullsized":"2023-03-03T07:55:47.261322+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"23'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263754+0000","last_change":"2023-03-03T07:55:46.564428+0000","last_active":"2023-03-03T07:55:47.263754+0000","last_peered":"2023-03-03T07:55:47.263754+0000","last_clean":"2023-03-03T07:55:47.263754+0000","last_became_active":"2023-03-03T07:55:46.564082+0000","last_became_peered":"2023-03-03T07:55:46.564082+0000","last_unstale":"2023-03-03T07:55:47.263754+0000","last_undegraded":"2023-03-03T07:55:47.263754+0000","last_fullsized":"2023-03-03T07:55:47.263754+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:28:57.536524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259449+0000","last_change":"2023-03-03T07:55:47.135104+0000","last_active":"2023-03-03T07:55:47.259449+0000","last_peered":"2023-03-03T07:55:47.259449+0000","last_clean":"2023-03-03T07:55:47.259449+0000","last_became_active":"2023-03-03T07:55:47.134610+0000","last_became_peered":"2023-03-03T07:55:47.134610+0000","last_unstale":"2023-03-03T07:55:47.259449+0000","last_undegraded":"2023-03-03T07:55:47.259449+0000","last_fullsized":"2023-03-03T07:55:47.259449+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"23'6","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262163+0000","last_change":"2023-03-03T07:55:46.563020+0000","last_active":"2023-03-03T07:55:47.262163+0000","last_peered":"2023-03-03T07:55:47.262163+0000","last_clean":"2023-03-03T07:55:47.262163+0000","last_became_active":"2023-03-03T07:55:46.562674+0000","last_became_peered":"2023-03-03T07:55:46.562674+0000","last_unstale":"2023-03-03T07:55:47.262163+0000","last_undegraded":"2023-03-03T07:55:47.262163+0000","last_fullsized":"2023-03-03T07:55:47.262163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:30:20.455024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262117+0000","last_change":"2023-03-03T07:55:47.131440+0000","last_active":"2023-03-03T07:55:47.262117+0000","last_peered":"2023-03-03T07:55:47.262117+0000","last_clean":"2023-03-03T07:55:47.262117+0000","last_became_active":"2023-03-03T07:55:47.131159+0000","last_became_peered":"2023-03-03T07:55:47.131159+0000","last_unstale":"2023-03-03T07:55:47.262117+0000","last_undegraded":"2023-03-03T07:55:47.262117+0000","last_fullsized":"2023-03-03T07:55:47.262117+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:29.837317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263734+0000","last_change":"2023-03-03T07:55:46.560188+0000","last_active":"2023-03-03T07:55:47.263734+0000","last_peered":"2023-03-03T07:55:47.263734+0000","last_clean":"2023-03-03T07:55:47.263734+0000","last_became_active":"2023-03-03T07:55:46.559853+0000","last_became_peered":"2023-03-03T07:55:46.559853+0000","last_unstale":"2023-03-03T07:55:47.263734+0000","last_undegraded":"2023-03-03T07:55:47.263734+0000","last_fullsized":"2023-03-03T07:55:47.263734+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261392+0000","last_change":"2023-03-03T07:55:47.131204+0000","last_active":"2023-03-03T07:55:47.261392+0000","last_peered":"2023-03-03T07:55:47.261392+0000","last_clean":"2023-03-03T07:55:47.261392+0000","last_became_active":"2023-03-03T07:55:47.130951+0000","last_became_peered":"2023-03-03T07:55:47.130951+0000","last_unstale":"2023-03-03T07:55:47.261392+0000","last_undegraded":"2023-03-03T07:55:47.261392+0000","last_fullsized":"2023-03-03T07:55:47.261392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259033+0000","last_change":"2023-03-03T07:55:46.567332+0000","last_active":"2023-03-03T07:55:47.259033+0000","last_peered":"2023-03-03T07:55:47.259033+0000","last_clean":"2023-03-03T07:55:47.259033+0000","last_became_active":"2023-03-03T07:55:46.567006+0000","last_became_peered":"2023-03-03T07:55:46.567006+0000","last_unstale":"2023-03-03T07:55:47.259033+0000","last_undegraded":"2023-03-03T07:55:47.259033+0000","last_fullsized":"2023-03-03T07:55:47.259033+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263740+0000","last_change":"2023-03-03T07:55:47.126887+0000","last_active":"2023-03-03T07:55:47.263740+0000","last_peered":"2023-03-03T07:55:47.263740+0000","last_clean":"2023-03-03T07:55:47.263740+0000","last_became_active":"2023-03-03T07:55:47.125992+0000","last_became_peered":"2023-03-03T07:55:47.125992+0000","last_unstale":"2023-03-03T07:55:47.263740+0000","last_undegraded":"2023-03-03T07:55:47.263740+0000","last_fullsized":"2023-03-03T07:55:47.263740+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263684+0000","last_change":"2023-03-03T07:55:46.559018+0000","last_active":"2023-03-03T07:55:47.263684+0000","last_peered":"2023-03-03T07:55:47.263684+0000","last_clean":"2023-03-03T07:55:47.263684+0000","last_became_active":"2023-03-03T07:55:46.558693+0000","last_became_peered":"2023-03-03T07:55:46.558693+0000","last_unstale":"2023-03-03T07:55:47.263684+0000","last_undegraded":"2023-03-03T07:55:47.263684+0000","last_fullsized":"2023-03-03T07:55:47.263684+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:28.058494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259175+0000","last_change":"2023-03-03T07:55:47.123304+0000","last_active":"2023-03-03T07:55:47.259175+0000","last_peered":"2023-03-03T07:55:47.259175+0000","last_clean":"2023-03-03T07:55:47.259175+0000","last_became_active":"2023-03-03T07:55:47.118375+0000","last_became_peered":"2023-03-03T07:55:47.118375+0000","last_unstale":"2023-03-03T07:55:47.259175+0000","last_undegraded":"2023-03-03T07:55:47.259175+0000","last_fullsized":"2023-03-03T07:55:47.259175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:49:25.482566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"23'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.264975+0000","last_change":"2023-03-03T07:55:46.565523+0000","last_active":"2023-03-03T07:55:47.264975+0000","last_peered":"2023-03-03T07:55:47.264975+0000","last_clean":"2023-03-03T07:55:47.264975+0000","last_became_active":"2023-03-03T07:55:46.565158+0000","last_became_peered":"2023-03-03T07:55:46.565158+0000","last_unstale":"2023-03-03T07:55:47.264975+0000","last_undegraded":"2023-03-03T07:55:47.264975+0000","last_fullsized":"2023-03-03T07:55:47.264975+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:57.246539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261430+0000","last_change":"2023-03-03T07:55:47.128607+0000","last_active":"2023-03-03T07:55:47.261430+0000","last_peered":"2023-03-03T07:55:47.261430+0000","last_clean":"2023-03-03T07:55:47.261430+0000","last_became_active":"2023-03-03T07:55:47.126465+0000","last_became_peered":"2023-03-03T07:55:47.126465+0000","last_unstale":"2023-03-03T07:55:47.261430+0000","last_undegraded":"2023-03-03T07:55:47.261430+0000","last_fullsized":"2023-03-03T07:55:47.261430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:21.542695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263798+0000","last_change":"2023-03-03T07:55:46.560647+0000","last_active":"2023-03-03T07:55:47.263798+0000","last_peered":"2023-03-03T07:55:47.263798+0000","last_clean":"2023-03-03T07:55:47.263798+0000","last_became_active":"2023-03-03T07:55:46.559663+0000","last_became_peered":"2023-03-03T07:55:46.559663+0000","last_unstale":"2023-03-03T07:55:47.263798+0000","last_undegraded":"2023-03-03T07:55:47.263798+0000","last_fullsized":"2023-03-03T07:55:47.263798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:54:12.785264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263622+0000","last_change":"2023-03-03T07:55:47.126608+0000","last_active":"2023-03-03T07:55:47.263622+0000","last_peered":"2023-03-03T07:55:47.263622+0000","last_clean":"2023-03-03T07:55:47.263622+0000","last_became_active":"2023-03-03T07:55:47.126236+0000","last_became_peered":"2023-03-03T07:55:47.126236+0000","last_unstale":"2023-03-03T07:55:47.263622+0000","last_undegraded":"2023-03-03T07:55:47.263622+0000","last_fullsized":"2023-03-03T07:55:47.263622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:03:11.053940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263787+0000","last_change":"2023-03-03T07:55:46.558232+0000","last_active":"2023-03-03T07:55:47.263787+0000","last_peered":"2023-03-03T07:55:47.263787+0000","last_clean":"2023-03-03T07:55:47.263787+0000","last_became_active":"2023-03-03T07:55:46.557732+0000","last_became_peered":"2023-03-03T07:55:46.557732+0000","last_unstale":"2023-03-03T07:55:47.263787+0000","last_undegraded":"2023-03-03T07:55:47.263787+0000","last_fullsized":"2023-03-03T07:55:47.263787+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262049+0000","last_change":"2023-03-03T07:55:47.139286+0000","last_active":"2023-03-03T07:55:47.262049+0000","last_peered":"2023-03-03T07:55:47.262049+0000","last_clean":"2023-03-03T07:55:47.262049+0000","last_became_active":"2023-03-03T07:55:47.138539+0000","last_became_peered":"2023-03-03T07:55:47.138539+0000","last_unstale":"2023-03-03T07:55:47.262049+0000","last_undegraded":"2023-03-03T07:55:47.262049+0000","last_fullsized":"2023-03-03T07:55:47.262049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:56.705526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263776+0000","last_change":"2023-03-03T07:55:46.559244+0000","last_active":"2023-03-03T07:55:47.263776+0000","last_peered":"2023-03-03T07:55:47.263776+0000","last_clean":"2023-03-03T07:55:47.263776+0000","last_became_active":"2023-03-03T07:55:46.558832+0000","last_became_peered":"2023-03-03T07:55:46.558832+0000","last_unstale":"2023-03-03T07:55:47.263776+0000","last_undegraded":"2023-03-03T07:55:47.263776+0000","last_fullsized":"2023-03-03T07:55:47.263776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:06:23.082465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.265028+0000","last_change":"2023-03-03T07:55:47.128420+0000","last_active":"2023-03-03T07:55:47.265028+0000","last_peered":"2023-03-03T07:55:47.265028+0000","last_clean":"2023-03-03T07:55:47.265028+0000","last_became_active":"2023-03-03T07:55:47.128117+0000","last_became_peered":"2023-03-03T07:55:47.128117+0000","last_unstale":"2023-03-03T07:55:47.265028+0000","last_undegraded":"2023-03-03T07:55:47.265028+0000","last_fullsized":"2023-03-03T07:55:47.265028+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:28:58.544816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263713+0000","last_change":"2023-03-03T07:55:46.563195+0000","last_active":"2023-03-03T07:55:47.263713+0000","last_peered":"2023-03-03T07:55:47.263713+0000","last_clean":"2023-03-03T07:55:47.263713+0000","last_became_active":"2023-03-03T07:55:46.562774+0000","last_became_peered":"2023-03-03T07:55:46.562774+0000","last_unstale":"2023-03-03T07:55:47.263713+0000","last_undegraded":"2023-03-03T07:55:47.263713+0000","last_fullsized":"2023-03-03T07:55:47.263713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:23.109334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259102+0000","last_change":"2023-03-03T07:55:47.132833+0000","last_active":"2023-03-03T07:55:47.259102+0000","last_peered":"2023-03-03T07:55:47.259102+0000","last_clean":"2023-03-03T07:55:47.259102+0000","last_became_active":"2023-03-03T07:55:47.132530+0000","last_became_peered":"2023-03-03T07:55:47.132530+0000","last_unstale":"2023-03-03T07:55:47.259102+0000","last_undegraded":"2023-03-03T07:55:47.259102+0000","last_fullsized":"2023-03-03T07:55:47.259102+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261380+0000","last_change":"2023-03-03T07:55:46.561556+0000","last_active":"2023-03-03T07:55:47.261380+0000","last_peered":"2023-03-03T07:55:47.261380+0000","last_clean":"2023-03-03T07:55:47.261380+0000","last_became_active":"2023-03-03T07:55:46.561273+0000","last_became_peered":"2023-03-03T07:55:46.561273+0000","last_unstale":"2023-03-03T07:55:47.261380+0000","last_undegraded":"2023-03-03T07:55:47.261380+0000","last_fullsized":"2023-03-03T07:55:47.261380+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262155+0000","last_change":"2023-03-03T07:55:47.138654+0000","last_active":"2023-03-03T07:55:47.262155+0000","last_peered":"2023-03-03T07:55:47.262155+0000","last_clean":"2023-03-03T07:55:47.262155+0000","last_became_active":"2023-03-03T07:55:47.131453+0000","last_became_peered":"2023-03-03T07:55:47.131453+0000","last_unstale":"2023-03-03T07:55:47.262155+0000","last_undegraded":"2023-03-03T07:55:47.262155+0000","last_fullsized":"2023-03-03T07:55:47.262155+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:36:18.587660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.265000+0000","last_change":"2023-03-03T07:55:46.563763+0000","last_active":"2023-03-03T07:55:47.265000+0000","last_peered":"2023-03-03T07:55:47.265000+0000","last_clean":"2023-03-03T07:55:47.265000+0000","last_became_active":"2023-03-03T07:55:46.563433+0000","last_became_peered":"2023-03-03T07:55:46.563433+0000","last_unstale":"2023-03-03T07:55:47.265000+0000","last_undegraded":"2023-03-03T07:55:47.265000+0000","last_fullsized":"2023-03-03T07:55:47.265000+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:12.972346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262191+0000","last_change":"2023-03-03T07:55:47.134804+0000","last_active":"2023-03-03T07:55:47.262191+0000","last_peered":"2023-03-03T07:55:47.262191+0000","last_clean":"2023-03-03T07:55:47.262191+0000","last_became_active":"2023-03-03T07:55:47.134504+0000","last_became_peered":"2023-03-03T07:55:47.134504+0000","last_unstale":"2023-03-03T07:55:47.262191+0000","last_undegraded":"2023-03-03T07:55:47.262191+0000","last_fullsized":"2023-03-03T07:55:47.262191+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262068+0000","last_change":"2023-03-03T07:55:46.561627+0000","last_active":"2023-03-03T07:55:47.262068+0000","last_peered":"2023-03-03T07:55:47.262068+0000","last_clean":"2023-03-03T07:55:47.262068+0000","last_became_active":"2023-03-03T07:55:46.561129+0000","last_became_peered":"2023-03-03T07:55:46.561129+0000","last_unstale":"2023-03-03T07:55:47.262068+0000","last_undegraded":"2023-03-03T07:55:47.262068+0000","last_fullsized":"2023-03-03T07:55:47.262068+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:12.972346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262208+0000","last_change":"2023-03-03T07:55:47.119813+0000","last_active":"2023-03-03T07:55:47.262208+0000","last_peered":"2023-03-03T07:55:47.262208+0000","last_clean":"2023-03-03T07:55:47.262208+0000","last_became_active":"2023-03-03T07:55:47.119517+0000","last_became_peered":"2023-03-03T07:55:47.119517+0000","last_unstale":"2023-03-03T07:55:47.262208+0000","last_undegraded":"2023-03-03T07:55:47.262208+0000","last_fullsized":"2023-03-03T07:55:47.262208+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:56:43.323290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261240+0000","last_change":"2023-03-03T07:55:46.557334+0000","last_active":"2023-03-03T07:55:47.261240+0000","last_peered":"2023-03-03T07:55:47.261240+0000","last_clean":"2023-03-03T07:55:47.261240+0000","last_became_active":"2023-03-03T07:55:46.556973+0000","last_became_peered":"2023-03-03T07:55:46.556973+0000","last_unstale":"2023-03-03T07:55:47.261240+0000","last_undegraded":"2023-03-03T07:55:47.261240+0000","last_fullsized":"2023-03-03T07:55:47.261240+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259625+0000","last_change":"2023-03-03T07:55:47.123429+0000","last_active":"2023-03-03T07:55:47.259625+0000","last_peered":"2023-03-03T07:55:47.259625+0000","last_clean":"2023-03-03T07:55:47.259625+0000","last_became_active":"2023-03-03T07:55:47.123131+0000","last_became_peered":"2023-03-03T07:55:47.123131+0000","last_unstale":"2023-03-03T07:55:47.259625+0000","last_undegraded":"2023-03-03T07:55:47.259625+0000","last_fullsized":"2023-03-03T07:55:47.259625+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.114110+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T07:55:47.114110+0000","last_peered":"2023-03-03T07:55:47.114110+0000","last_clean":"2023-03-03T07:55:47.114110+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T07:55:47.114110+0000","last_undegraded":"2023-03-03T07:55:47.114110+0000","last_fullsized":"2023-03-03T07:55:47.114110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263752+0000","last_change":"2023-03-03T07:55:46.565316+0000","last_active":"2023-03-03T07:55:47.263752+0000","last_peered":"2023-03-03T07:55:47.263752+0000","last_clean":"2023-03-03T07:55:47.263752+0000","last_became_active":"2023-03-03T07:55:46.565027+0000","last_became_peered":"2023-03-03T07:55:46.565027+0000","last_unstale":"2023-03-03T07:55:47.263752+0000","last_undegraded":"2023-03-03T07:55:47.263752+0000","last_fullsized":"2023-03-03T07:55:47.263752+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:28.058494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259465+0000","last_change":"2023-03-03T07:55:47.124467+0000","last_active":"2023-03-03T07:55:47.259465+0000","last_peered":"2023-03-03T07:55:47.259465+0000","last_clean":"2023-03-03T07:55:47.259465+0000","last_became_active":"2023-03-03T07:55:47.124149+0000","last_became_peered":"2023-03-03T07:55:47.124149+0000","last_unstale":"2023-03-03T07:55:47.259465+0000","last_undegraded":"2023-03-03T07:55:47.259465+0000","last_fullsized":"2023-03-03T07:55:47.259465+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.107246+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T07:55:47.107246+0000","last_peered":"2023-03-03T07:55:47.107246+0000","last_clean":"2023-03-03T07:55:47.107246+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T07:55:47.107246+0000","last_undegraded":"2023-03-03T07:55:47.107246+0000","last_fullsized":"2023-03-03T07:55:47.107246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262126+0000","last_change":"2023-03-03T07:55:46.559574+0000","last_active":"2023-03-03T07:55:47.262126+0000","last_peered":"2023-03-03T07:55:47.262126+0000","last_clean":"2023-03-03T07:55:47.262126+0000","last_became_active":"2023-03-03T07:55:46.559192+0000","last_became_peered":"2023-03-03T07:55:46.559192+0000","last_unstale":"2023-03-03T07:55:47.262126+0000","last_undegraded":"2023-03-03T07:55:47.262126+0000","last_fullsized":"2023-03-03T07:55:47.262126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259529+0000","last_change":"2023-03-03T07:55:47.127703+0000","last_active":"2023-03-03T07:55:47.259529+0000","last_peered":"2023-03-03T07:55:47.259529+0000","last_clean":"2023-03-03T07:55:47.259529+0000","last_became_active":"2023-03-03T07:55:47.126941+0000","last_became_peered":"2023-03-03T07:55:47.126941+0000","last_unstale":"2023-03-03T07:55:47.259529+0000","last_undegraded":"2023-03-03T07:55:47.259529+0000","last_fullsized":"2023-03-03T07:55:47.259529+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:03:11.053940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.120195+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T07:55:47.120195+0000","last_peered":"2023-03-03T07:55:47.120195+0000","last_clean":"2023-03-03T07:55:47.120195+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T07:55:47.120195+0000","last_undegraded":"2023-03-03T07:55:47.120195+0000","last_fullsized":"2023-03-03T07:55:47.120195+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259411+0000","last_change":"2023-03-03T07:55:46.560453+0000","last_active":"2023-03-03T07:55:47.259411+0000","last_peered":"2023-03-03T07:55:47.259411+0000","last_clean":"2023-03-03T07:55:47.259411+0000","last_became_active":"2023-03-03T07:55:46.560140+0000","last_became_peered":"2023-03-03T07:55:46.560140+0000","last_unstale":"2023-03-03T07:55:47.259411+0000","last_undegraded":"2023-03-03T07:55:47.259411+0000","last_fullsized":"2023-03-03T07:55:47.259411+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259609+0000","last_change":"2023-03-03T07:55:47.124151+0000","last_active":"2023-03-03T07:55:47.259609+0000","last_peered":"2023-03-03T07:55:47.259609+0000","last_clean":"2023-03-03T07:55:47.259609+0000","last_became_active":"2023-03-03T07:55:47.123697+0000","last_became_peered":"2023-03-03T07:55:47.123697+0000","last_unstale":"2023-03-03T07:55:47.259609+0000","last_undegraded":"2023-03-03T07:55:47.259609+0000","last_fullsized":"2023-03-03T07:55:47.259609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.107382+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T07:55:47.107382+0000","last_peered":"2023-03-03T07:55:47.107382+0000","last_clean":"2023-03-03T07:55:47.107382+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T07:55:47.107382+0000","last_undegraded":"2023-03-03T07:55:47.107382+0000","last_fullsized":"2023-03-03T07:55:47.107382+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.123508+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T07:55:47.123508+0000","last_peered":"2023-03-03T07:55:47.123508+0000","last_clean":"2023-03-03T07:55:47.123508+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T07:55:47.123508+0000","last_undegraded":"2023-03-03T07:55:47.123508+0000","last_fullsized":"2023-03-03T07:55:47.123508+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261341+0000","last_change":"2023-03-03T07:55:46.559444+0000","last_active":"2023-03-03T07:55:47.261341+0000","last_peered":"2023-03-03T07:55:47.261341+0000","last_clean":"2023-03-03T07:55:47.261341+0000","last_became_active":"2023-03-03T07:55:46.559147+0000","last_became_peered":"2023-03-03T07:55:46.559147+0000","last_unstale":"2023-03-03T07:55:47.261341+0000","last_undegraded":"2023-03-03T07:55:47.261341+0000","last_fullsized":"2023-03-03T07:55:47.261341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:30:20.455024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.265049+0000","last_change":"2023-03-03T07:55:47.134945+0000","last_active":"2023-03-03T07:55:47.265049+0000","last_peered":"2023-03-03T07:55:47.265049+0000","last_clean":"2023-03-03T07:55:47.265049+0000","last_became_active":"2023-03-03T07:55:47.134662+0000","last_became_peered":"2023-03-03T07:55:47.134662+0000","last_unstale":"2023-03-03T07:55:47.265049+0000","last_undegraded":"2023-03-03T07:55:47.265049+0000","last_fullsized":"2023-03-03T07:55:47.265049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:54:13.793557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.123980+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T07:55:47.123980+0000","last_peered":"2023-03-03T07:55:47.123980+0000","last_clean":"2023-03-03T07:55:47.123980+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T07:55:47.123980+0000","last_undegraded":"2023-03-03T07:55:47.123980+0000","last_fullsized":"2023-03-03T07:55:47.123980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259010+0000","last_change":"2023-03-03T07:55:46.567402+0000","last_active":"2023-03-03T07:55:47.259010+0000","last_peered":"2023-03-03T07:55:47.259010+0000","last_clean":"2023-03-03T07:55:47.259010+0000","last_became_active":"2023-03-03T07:55:46.567013+0000","last_became_peered":"2023-03-03T07:55:46.567013+0000","last_unstale":"2023-03-03T07:55:47.259010+0000","last_undegraded":"2023-03-03T07:55:47.259010+0000","last_fullsized":"2023-03-03T07:55:47.259010+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259226+0000","last_change":"2023-03-03T07:55:47.125428+0000","last_active":"2023-03-03T07:55:47.259226+0000","last_peered":"2023-03-03T07:55:47.259226+0000","last_clean":"2023-03-03T07:55:47.259226+0000","last_became_active":"2023-03-03T07:55:47.125113+0000","last_became_peered":"2023-03-03T07:55:47.125113+0000","last_unstale":"2023-03-03T07:55:47.259226+0000","last_undegraded":"2023-03-03T07:55:47.259226+0000","last_fullsized":"2023-03-03T07:55:47.259226+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.122691+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T07:55:47.122691+0000","last_peered":"2023-03-03T07:55:47.122691+0000","last_clean":"2023-03-03T07:55:47.122691+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T07:55:47.122691+0000","last_undegraded":"2023-03-03T07:55:47.122691+0000","last_fullsized":"2023-03-03T07:55:47.122691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259545+0000","last_change":"2023-03-03T07:55:46.556310+0000","last_active":"2023-03-03T07:55:47.259545+0000","last_peered":"2023-03-03T07:55:47.259545+0000","last_clean":"2023-03-03T07:55:47.259545+0000","last_became_active":"2023-03-03T07:55:46.555567+0000","last_became_peered":"2023-03-03T07:55:46.555567+0000","last_unstale":"2023-03-03T07:55:47.259545+0000","last_undegraded":"2023-03-03T07:55:47.259545+0000","last_fullsized":"2023-03-03T07:55:47.259545+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259158+0000","last_change":"2023-03-03T07:55:47.119872+0000","last_active":"2023-03-03T07:55:47.259158+0000","last_peered":"2023-03-03T07:55:47.259158+0000","last_clean":"2023-03-03T07:55:47.259158+0000","last_became_active":"2023-03-03T07:55:47.119546+0000","last_became_peered":"2023-03-03T07:55:47.119546+0000","last_unstale":"2023-03-03T07:55:47.259158+0000","last_undegraded":"2023-03-03T07:55:47.259158+0000","last_fullsized":"2023-03-03T07:55:47.259158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261420+0000","last_change":"2023-03-03T07:55:46.559460+0000","last_active":"2023-03-03T07:55:47.261420+0000","last_peered":"2023-03-03T07:55:47.261420+0000","last_clean":"2023-03-03T07:55:47.261420+0000","last_became_active":"2023-03-03T07:55:46.559134+0000","last_became_peered":"2023-03-03T07:55:46.559134+0000","last_unstale":"2023-03-03T07:55:47.261420+0000","last_undegraded":"2023-03-03T07:55:47.261420+0000","last_fullsized":"2023-03-03T07:55:47.261420+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:11.895405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.112543+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T07:55:47.112543+0000","last_peered":"2023-03-03T07:55:47.112543+0000","last_clean":"2023-03-03T07:55:47.112543+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T07:55:47.112543+0000","last_undegraded":"2023-03-03T07:55:47.112543+0000","last_fullsized":"2023-03-03T07:55:47.112543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261225+0000","last_change":"2023-03-03T07:55:47.127917+0000","last_active":"2023-03-03T07:55:47.261225+0000","last_peered":"2023-03-03T07:55:47.261225+0000","last_clean":"2023-03-03T07:55:47.261225+0000","last_became_active":"2023-03-03T07:55:47.127606+0000","last_became_peered":"2023-03-03T07:55:47.127606+0000","last_unstale":"2023-03-03T07:55:47.261225+0000","last_undegraded":"2023-03-03T07:55:47.261225+0000","last_fullsized":"2023-03-03T07:55:47.261225+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.105592+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T07:55:47.105592+0000","last_peered":"2023-03-03T07:55:47.105592+0000","last_clean":"2023-03-03T07:55:47.105592+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T07:55:47.105592+0000","last_undegraded":"2023-03-03T07:55:47.105592+0000","last_fullsized":"2023-03-03T07:55:47.105592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"24'1","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-03-03T07:55:49.262039+0000","last_change":"2023-03-03T07:55:46.565365+0000","last_active":"2023-03-03T07:55:49.262039+0000","last_peered":"2023-03-03T07:55:49.262039+0000","last_clean":"2023-03-03T07:55:49.262039+0000","last_became_active":"2023-03-03T07:55:46.565097+0000","last_became_peered":"2023-03-03T07:55:46.565097+0000","last_unstale":"2023-03-03T07:55:49.262039+0000","last_undegraded":"2023-03-03T07:55:49.262039+0000","last_fullsized":"2023-03-03T07:55:49.262039+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:23.109334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262057+0000","last_change":"2023-03-03T07:55:47.139732+0000","last_active":"2023-03-03T07:55:47.262057+0000","last_peered":"2023-03-03T07:55:47.262057+0000","last_clean":"2023-03-03T07:55:47.262057+0000","last_became_active":"2023-03-03T07:55:47.138505+0000","last_became_peered":"2023-03-03T07:55:47.138505+0000","last_unstale":"2023-03-03T07:55:47.262057+0000","last_undegraded":"2023-03-03T07:55:47.262057+0000","last_fullsized":"2023-03-03T07:55:47.262057+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:49:25.482566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259120+0000","last_change":"2023-03-03T07:55:46.568574+0000","last_active":"2023-03-03T07:55:47.259120+0000","last_peered":"2023-03-03T07:55:47.259120+0000","last_clean":"2023-03-03T07:55:47.259120+0000","last_became_active":"2023-03-03T07:55:46.568139+0000","last_became_peered":"2023-03-03T07:55:46.568139+0000","last_unstale":"2023-03-03T07:55:47.259120+0000","last_undegraded":"2023-03-03T07:55:47.259120+0000","last_fullsized":"2023-03-03T07:55:47.259120+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.256785+0000","last_change":"2023-03-03T07:55:47.128846+0000","last_active":"2023-03-03T07:55:47.256785+0000","last_peered":"2023-03-03T07:55:47.256785+0000","last_clean":"2023-03-03T07:55:47.256785+0000","last_became_active":"2023-03-03T07:55:47.128385+0000","last_became_peered":"2023-03-03T07:55:47.128385+0000","last_unstale":"2023-03-03T07:55:47.256785+0000","last_undegraded":"2023-03-03T07:55:47.256785+0000","last_fullsized":"2023-03-03T07:55:47.256785+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:00.511475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263692+0000","last_change":"2023-03-03T07:55:46.560316+0000","last_active":"2023-03-03T07:55:47.263692+0000","last_peered":"2023-03-03T07:55:47.263692+0000","last_clean":"2023-03-03T07:55:47.263692+0000","last_became_active":"2023-03-03T07:55:46.560031+0000","last_became_peered":"2023-03-03T07:55:46.560031+0000","last_unstale":"2023-03-03T07:55:47.263692+0000","last_undegraded":"2023-03-03T07:55:47.263692+0000","last_fullsized":"2023-03-03T07:55:47.263692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262135+0000","last_change":"2023-03-03T07:55:47.137604+0000","last_active":"2023-03-03T07:55:47.262135+0000","last_peered":"2023-03-03T07:55:47.262135+0000","last_clean":"2023-03-03T07:55:47.262135+0000","last_became_active":"2023-03-03T07:55:47.131552+0000","last_became_peered":"2023-03-03T07:55:47.131552+0000","last_unstale":"2023-03-03T07:55:47.262135+0000","last_undegraded":"2023-03-03T07:55:47.262135+0000","last_fullsized":"2023-03-03T07:55:47.262135+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263762+0000","last_change":"2023-03-03T07:55:46.560878+0000","last_active":"2023-03-03T07:55:47.263762+0000","last_peered":"2023-03-03T07:55:47.263762+0000","last_clean":"2023-03-03T07:55:47.263762+0000","last_became_active":"2023-03-03T07:55:46.560534+0000","last_became_peered":"2023-03-03T07:55:46.560534+0000","last_unstale":"2023-03-03T07:55:47.263762+0000","last_undegraded":"2023-03-03T07:55:47.263762+0000","last_fullsized":"2023-03-03T07:55:47.263762+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262107+0000","last_change":"2023-03-03T07:55:47.122002+0000","last_active":"2023-03-03T07:55:47.262107+0000","last_peered":"2023-03-03T07:55:47.262107+0000","last_clean":"2023-03-03T07:55:47.262107+0000","last_became_active":"2023-03-03T07:55:47.121732+0000","last_became_peered":"2023-03-03T07:55:47.121732+0000","last_unstale":"2023-03-03T07:55:47.262107+0000","last_undegraded":"2023-03-03T07:55:47.262107+0000","last_fullsized":"2023-03-03T07:55:47.262107+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261411+0000","last_change":"2023-03-03T07:55:46.560027+0000","last_active":"2023-03-03T07:55:47.261411+0000","last_peered":"2023-03-03T07:55:47.261411+0000","last_clean":"2023-03-03T07:55:47.261411+0000","last_became_active":"2023-03-03T07:55:46.559508+0000","last_became_peered":"2023-03-03T07:55:46.559508+0000","last_unstale":"2023-03-03T07:55:47.261411+0000","last_undegraded":"2023-03-03T07:55:47.261411+0000","last_fullsized":"2023-03-03T07:55:47.261411+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:23.109334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261313+0000","last_change":"2023-03-03T07:55:47.121846+0000","last_active":"2023-03-03T07:55:47.261313+0000","last_peered":"2023-03-03T07:55:47.261313+0000","last_clean":"2023-03-03T07:55:47.261313+0000","last_became_active":"2023-03-03T07:55:47.121563+0000","last_became_peered":"2023-03-03T07:55:47.121563+0000","last_unstale":"2023-03-03T07:55:47.261313+0000","last_undegraded":"2023-03-03T07:55:47.261313+0000","last_fullsized":"2023-03-03T07:55:47.261313+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:36:18.587660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"23'4","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259326+0000","last_change":"2023-03-03T07:55:46.567878+0000","last_active":"2023-03-03T07:55:47.259326+0000","last_peered":"2023-03-03T07:55:47.259326+0000","last_clean":"2023-03-03T07:55:47.259326+0000","last_became_active":"2023-03-03T07:55:46.567593+0000","last_became_peered":"2023-03-03T07:55:46.567593+0000","last_unstale":"2023-03-03T07:55:47.259326+0000","last_undegraded":"2023-03-03T07:55:47.259326+0000","last_fullsized":"2023-03-03T07:55:47.259326+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:28.058494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262023+0000","last_change":"2023-03-03T07:55:47.131373+0000","last_active":"2023-03-03T07:55:47.262023+0000","last_peered":"2023-03-03T07:55:47.262023+0000","last_clean":"2023-03-03T07:55:47.262023+0000","last_became_active":"2023-03-03T07:55:47.131037+0000","last_became_peered":"2023-03-03T07:55:47.131037+0000","last_unstale":"2023-03-03T07:55:47.262023+0000","last_undegraded":"2023-03-03T07:55:47.262023+0000","last_fullsized":"2023-03-03T07:55:47.262023+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:52:08.323505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.264990+0000","last_change":"2023-03-03T07:55:47.122842+0000","last_active":"2023-03-03T07:55:47.264990+0000","last_peered":"2023-03-03T07:55:47.264990+0000","last_clean":"2023-03-03T07:55:47.264990+0000","last_became_active":"2023-03-03T07:55:47.122580+0000","last_became_peered":"2023-03-03T07:55:47.122580+0000","last_unstale":"2023-03-03T07:55:47.264990+0000","last_undegraded":"2023-03-03T07:55:47.264990+0000","last_fullsized":"2023-03-03T07:55:47.264990+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:06:24.090758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261297+0000","last_change":"2023-03-03T07:55:47.121284+0000","last_active":"2023-03-03T07:55:47.261297+0000","last_peered":"2023-03-03T07:55:47.261297+0000","last_clean":"2023-03-03T07:55:47.261297+0000","last_became_active":"2023-03-03T07:55:47.121012+0000","last_became_peered":"2023-03-03T07:55:47.121012+0000","last_unstale":"2023-03-03T07:55:47.261297+0000","last_undegraded":"2023-03-03T07:55:47.261297+0000","last_fullsized":"2023-03-03T07:55:47.261297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:15.870956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262084+0000","last_change":"2023-03-03T07:55:47.138325+0000","last_active":"2023-03-03T07:55:47.262084+0000","last_peered":"2023-03-03T07:55:47.262084+0000","last_clean":"2023-03-03T07:55:47.262084+0000","last_became_active":"2023-03-03T07:55:47.138019+0000","last_became_peered":"2023-03-03T07:55:47.138019+0000","last_unstale":"2023-03-03T07:55:47.262084+0000","last_undegraded":"2023-03-03T07:55:47.262084+0000","last_fullsized":"2023-03-03T07:55:47.262084+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:57.306479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261371+0000","last_change":"2023-03-03T07:55:47.128792+0000","last_active":"2023-03-03T07:55:47.261371+0000","last_peered":"2023-03-03T07:55:47.261371+0000","last_clean":"2023-03-03T07:55:47.261371+0000","last_became_active":"2023-03-03T07:55:47.127248+0000","last_became_peered":"2023-03-03T07:55:47.127248+0000","last_unstale":"2023-03-03T07:55:47.261371+0000","last_undegraded":"2023-03-03T07:55:47.261371+0000","last_fullsized":"2023-03-03T07:55:47.261371+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:49:25.482566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259481+0000","last_change":"2023-03-03T07:55:47.127209+0000","last_active":"2023-03-03T07:55:47.259481+0000","last_peered":"2023-03-03T07:55:47.259481+0000","last_clean":"2023-03-03T07:55:47.259481+0000","last_became_active":"2023-03-03T07:55:47.126100+0000","last_became_peered":"2023-03-03T07:55:47.126100+0000","last_unstale":"2023-03-03T07:55:47.259481+0000","last_undegraded":"2023-03-03T07:55:47.259481+0000","last_fullsized":"2023-03-03T07:55:47.259481+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:46.306038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9120,"kb_used_data":856,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362720,"statfs":{"total":96636764160,"available":96627425280,"internally_reserved":0,"allocated":876544,"data_stored":687141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6829999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":93990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8064,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363776,"statfs":{"total":96636764160,"available":96628506624,"internally_reserved":0,"allocated":253952,"data_stored":95009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4970000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.897}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.843999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.613}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.907}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9092,"kb_used_data":828,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362748,"statfs":{"total":96636764160,"available":96627453952,"internally_reserved":0,"allocated":847872,"data_stored":684924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":93990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8068,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363772,"statfs":{"total":96636764160,"available":96628502528,"internally_reserved":0,"allocated":258048,"data_stored":95099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5820000000000007}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3190000000000008}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8629999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8190000000000008}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.625}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8084,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363756,"statfs":{"total":96636764160,"available":96628486144,"internally_reserved":0,"allocated":274432,"data_stored":96613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.1799999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5709999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4529999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8004,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363836,"statfs":{"total":96636764160,"available":96628568064,"internally_reserved":0,"allocated":258048,"data_stored":94626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T07:55:58.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T07:55:58.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.723+0000 7f49b159d700 1 Processor -- start 2023-03-03T07:55:58.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.723+0000 7f49b159d700 1 -- start start 2023-03-03T07:55:58.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.725+0000 7f49b159d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49ac0b51c0 0x7f49ac160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:58.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.725+0000 7f49b159d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac1614c0 con 0x7f49ac146360 2023-03-03T07:55:58.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.725+0000 7f49b159d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac0d10f0 con 0x7f49ac0d1580 2023-03-03T07:55:58.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.725+0000 7f49b159d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0cdf20 con 0x7f49ac0b51c0 2023-03-03T07:55:58.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.725+0000 7f49abfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f49ac146360 0x7f49ac0b4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53690/0 (socket says 172.21.15.98:53690) 2023-03-03T07:55:58.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.725+0000 7f49abfff700 1 -- 172.21.15.98:0/2252336828 learned_addr learned my addr 172.21.15.98:0/2252336828 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:58.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49a3fff700 1 --2- 172.21.15.98:0/2252336828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49ac0b51c0 0x7f49ac160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:58.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49ab7fe700 1 -- 172.21.15.98:0/2252336828 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844518205 0 0) 0x7f499c001110 con 0x7f49ac146360 2023-03-03T07:55:58.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49ab7fe700 1 -- 172.21.15.98:0/2252336828 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49980035b0 con 0x7f49ac146360 2023-03-03T07:55:58.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49ab7fe700 1 -- 172.21.15.98:0/2252336828 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1828460336 0 0) 0x7f49980035b0 con 0x7f49ac146360 2023-03-03T07:55:58.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49ab7fe700 1 -- 172.21.15.98:0/2252336828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49ac0b51c0 msgr2=0x7f49ac160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:58.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49ab7fe700 1 --2- 172.21.15.98:0/2252336828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49ac0b51c0 0x7f49ac160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49ab7fe700 1 -- 172.21.15.98:0/2252336828 >> v1:172.21.15.100:6789/0 conn(0x7f49ac0d1580 legacy=0x7f49ac0d19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:58.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49ab7fe700 1 -- 172.21.15.98:0/2252336828 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac0cd680 con 0x7f49ac146360 2023-03-03T07:55:58.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49ab7fe700 1 -- 172.21.15.98:0/2252336828 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f499c001800 con 0x7f49ac146360 2023-03-03T07:55:58.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49ab7fe700 1 -- 172.21.15.98:0/2252336828 >> v1:172.21.15.98:6789/0 conn(0x7f49ac146360 legacy=0x7f49ac0b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:58.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49ab7fe700 1 --2- 172.21.15.98:0/2252336828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4998003a30 0x7f4998003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:58.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49ab7fe700 1 --2- 172.21.15.98:0/2252336828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49ac0b51c0 0x7f4998004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:58.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49ab7fe700 1 --2- 172.21.15.98:0/2252336828 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4998004d90 0x7f4998005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:58.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49ab7fe700 1 -- 172.21.15.98:0/2252336828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0cdf20 con 0x7f4998003a30 2023-03-03T07:55:58.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49ab7fe700 1 -- 172.21.15.98:0/2252336828 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499800bac0 con 0x7f4998004d90 2023-03-03T07:55:58.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.726+0000 7f49ab7fe700 1 -- 172.21.15.98:0/2252336828 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499800bc50 con 0x7f49ac0b51c0 2023-03-03T07:55:58.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.727+0000 7f49a3fff700 1 --2- 172.21.15.98:0/2252336828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49ac0b51c0 0x7f4998004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:58.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.727+0000 7f49abfff700 1 --2- 172.21.15.98:0/2252336828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4998003a30 0x7f4998003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:58.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.727+0000 7f49b0d9c700 1 --2- 172.21.15.98:0/2252336828 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4998004d90 0x7f4998005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:58.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.727+0000 7f49a3fff700 1 -- 172.21.15.98:0/2252336828 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4998004d90 msgr2=0x7f4998005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:58.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.727+0000 7f49a3fff700 1 --2- 172.21.15.98:0/2252336828 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4998004d90 0x7f4998005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.727+0000 7f49a3fff700 1 -- 172.21.15.98:0/2252336828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4998003a30 msgr2=0x7f4998003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:58.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.727+0000 7f49a3fff700 1 --2- 172.21.15.98:0/2252336828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4998003a30 0x7f4998003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.727+0000 7f49a3fff700 1 -- 172.21.15.98:0/2252336828 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49980035b0 con 0x7f49ac0b51c0 2023-03-03T07:55:58.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.727+0000 7f49abfff700 1 --2- 172.21.15.98:0/2252336828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4998003a30 0x7f4998003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:55:58.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.727+0000 7f49a3fff700 1 --2- 172.21.15.98:0/2252336828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49ac0b51c0 0x7f4998004850 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4994009b30 tx=0x7f4998005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96917842f699725b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:58.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.728+0000 7f49ab7fe700 1 -- 172.21.15.98:0/2252336828 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499400e6c0 con 0x7f49ac0b51c0 2023-03-03T07:55:58.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.728+0000 7f49ab7fe700 1 -- 172.21.15.98:0/2252336828 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499400e820 con 0x7f49ac0b51c0 2023-03-03T07:55:58.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.728+0000 7f49ab7fe700 1 -- 172.21.15.98:0/2252336828 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499400eb90 con 0x7f49ac0b51c0 2023-03-03T07:55:58.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.728+0000 7f49b159d700 1 -- 172.21.15.98:0/2252336828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49ac0b51c0 msgr2=0x7f4998004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:58.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.728+0000 7f49b159d700 1 --2- 172.21.15.98:0/2252336828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49ac0b51c0 0x7f4998004850 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4994009b30 tx=0x7f4998005880 comp rx=0 tx=0).stop 2023-03-03T07:55:58.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.728+0000 7f49b0d9c700 1 -- 172.21.15.98:0/2252336828 reap_dead start 2023-03-03T07:55:58.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.728+0000 7f49b159d700 1 -- 172.21.15.98:0/2252336828 shutdown_connections 2023-03-03T07:55:58.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.728+0000 7f49b159d700 1 -- 172.21.15.98:0/2252336828 >> 172.21.15.98:0/2252336828 conn(0x7f49ac06b8e0 msgr2=0x7f49ac0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:58.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.728+0000 7f49b159d700 1 -- 172.21.15.98:0/2252336828 shutdown_connections 2023-03-03T07:55:58.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.728+0000 7f49b159d700 1 -- 172.21.15.98:0/2252336828 wait complete. 2023-03-03T07:55:58.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.731+0000 7f49b159d700 1 Processor -- start 2023-03-03T07:55:58.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.732+0000 7f49b159d700 1 -- start start 2023-03-03T07:55:58.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.733+0000 7f49b159d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49ac0b51c0 0x7f49ac15a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:58.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.733+0000 7f49b159d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49ac157b70 0x7f49ac0b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:58.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.733+0000 7f49b159d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac1557f0 0x7f49ac155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:58.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.733+0000 7f49b159d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac1561a0 con 0x7f49ac1557f0 2023-03-03T07:55:58.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.733+0000 7f49b159d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0b7870 con 0x7f49ac157b70 2023-03-03T07:55:58.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.733+0000 7f49b159d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0b7af0 con 0x7f49ac0b51c0 2023-03-03T07:55:58.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.733+0000 7f49b0d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac1557f0 0x7f49ac155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:58.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.733+0000 7f49abfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49ac0b51c0 0x7f49ac15a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:58.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.733+0000 7f49b0d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac1557f0 0x7f49ac155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41838/0 (socket says 172.21.15.98:41838) 2023-03-03T07:55:58.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.733+0000 7f49b0d9c700 1 -- 172.21.15.98:0/1816508090 learned_addr learned my addr 172.21.15.98:0/1816508090 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:58.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.733+0000 7f49a3fff700 1 --2- 172.21.15.98:0/1816508090 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49ac157b70 0x7f49ac0b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:58.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.734+0000 7f49b0d9c700 1 -- 172.21.15.98:0/1816508090 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49ac0b51c0 msgr2=0x7f49ac15a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:58.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.734+0000 7f49b0d9c700 1 --2- 172.21.15.98:0/1816508090 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49ac0b51c0 0x7f49ac15a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.734+0000 7f49b0d9c700 1 -- 172.21.15.98:0/1816508090 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49ac157b70 msgr2=0x7f49ac0b88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:58.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.734+0000 7f49b0d9c700 1 --2- 172.21.15.98:0/1816508090 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49ac157b70 0x7f49ac0b88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.734+0000 7f49b0d9c700 1 -- 172.21.15.98:0/1816508090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac0cdc40 con 0x7f49ac1557f0 2023-03-03T07:55:58.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.734+0000 7f49b0d9c700 1 --2- 172.21.15.98:0/1816508090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac1557f0 0x7f49ac155c60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0d2fe0 tx=0x7f49a400d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:58.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.734+0000 7f49a97fa700 1 -- 172.21.15.98:0/1816508090 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a400db90 con 0x7f49ac1557f0 2023-03-03T07:55:58.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.734+0000 7f49a97fa700 1 -- 172.21.15.98:0/1816508090 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a400dcf0 con 0x7f49ac1557f0 2023-03-03T07:55:58.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.734+0000 7f49b159d700 1 -- 172.21.15.98:0/1816508090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ac0cd960 con 0x7f49ac1557f0 2023-03-03T07:55:58.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.734+0000 7f49a97fa700 1 -- 172.21.15.98:0/1816508090 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4003c40 con 0x7f49ac1557f0 2023-03-03T07:55:58.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.734+0000 7f49b159d700 1 -- 172.21.15.98:0/1816508090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac0d0d80 con 0x7f49ac1557f0 2023-03-03T07:55:58.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.735+0000 7f49b159d700 1 -- 172.21.15.98:0/1816508090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ac157f40 con 0x7f49ac1557f0 2023-03-03T07:55:58.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.737+0000 7f49a97fa700 1 -- 172.21.15.98:0/1816508090 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f49a40050d0 con 0x7f49ac1557f0 2023-03-03T07:55:58.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.738+0000 7f49a97fa700 1 --2- 172.21.15.98:0/1816508090 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f498c0682f0 0x7f498c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:58.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.738+0000 7f49a97fa700 1 -- 172.21.15.98:0/1816508090 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f49a4085f90 con 0x7f49ac1557f0 2023-03-03T07:55:58.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.738+0000 7f49abfff700 1 --2- 172.21.15.98:0/1816508090 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f498c0682f0 0x7f498c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:58.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.739+0000 7f49abfff700 1 --2- 172.21.15.98:0/1816508090 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f498c0682f0 0x7f498c06a7a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f499c000c00 tx=0x7f499c007e00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:58.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.742+0000 7f49a97fa700 1 -- 172.21.15.98:0/1816508090 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f49a4052700 con 0x7f49ac1557f0 2023-03-03T07:55:58.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.929+0000 7f49b159d700 1 -- 172.21.15.98:0/1816508090 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f49ac1614c0 con 0x7f498c0682f0 2023-03-03T07:55:58.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.938+0000 7f49a97fa700 1 -- 172.21.15.98:0/1816508090 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286503 (secure 0 0 0) 0x7f49ac1614c0 con 0x7f498c0682f0 2023-03-03T07:55:58.942 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:58.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.943+0000 7f49b159d700 1 -- 172.21.15.98:0/1816508090 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f498c0682f0 msgr2=0x7f498c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:58.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.943+0000 7f49b159d700 1 --2- 172.21.15.98:0/1816508090 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f498c0682f0 0x7f498c06a7a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f499c000c00 tx=0x7f499c007e00 comp rx=0 tx=0).stop 2023-03-03T07:55:58.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.943+0000 7f49b159d700 1 -- 172.21.15.98:0/1816508090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac1557f0 msgr2=0x7f49ac155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:58.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.943+0000 7f49b159d700 1 --2- 172.21.15.98:0/1816508090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac1557f0 0x7f49ac155c60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0d2fe0 tx=0x7f49a400d660 comp rx=0 tx=0).stop 2023-03-03T07:55:58.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.943+0000 7f49b159d700 1 -- 172.21.15.98:0/1816508090 shutdown_connections 2023-03-03T07:55:58.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.943+0000 7f49b159d700 1 --2- 172.21.15.98:0/1816508090 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f498c0682f0 0x7f498c06a7a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.943+0000 7f49b159d700 1 --2- 172.21.15.98:0/1816508090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac1557f0 0x7f49ac155c60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.943+0000 7f49b159d700 1 --2- 172.21.15.98:0/1816508090 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49ac0b51c0 0x7f49ac15a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.943+0000 7f49b159d700 1 --2- 172.21.15.98:0/1816508090 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49ac157b70 0x7f49ac0b88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:58.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.943+0000 7f49b159d700 1 -- 172.21.15.98:0/1816508090 >> 172.21.15.98:0/1816508090 conn(0x7f49ac06b8e0 msgr2=0x7f49ac0b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:58.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.943+0000 7f49b159d700 1 -- 172.21.15.98:0/1816508090 shutdown_connections 2023-03-03T07:55:58.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:58.943+0000 7f49b159d700 1 -- 172.21.15.98:0/1816508090 wait complete. 2023-03-03T07:55:58.947 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T07:55:58.972 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-03-03T07:55:58.181089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":66424,"kb_used_data":3192,"kb_used_omap":0,"kb_used_meta":63168,"kb_avail":754908296,"statfs":{"total":773094113280,"available":773026095104,"internally_reserved":0,"allocated":3268608,"data_stored":1941392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64684032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.075080"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.256834+0000","last_change":"2023-03-03T07:55:47.123170+0000","last_active":"2023-03-03T07:55:47.256834+0000","last_peered":"2023-03-03T07:55:47.256834+0000","last_clean":"2023-03-03T07:55:47.256834+0000","last_became_active":"2023-03-03T07:55:47.122733+0000","last_became_peered":"2023-03-03T07:55:47.122733+0000","last_unstale":"2023-03-03T07:55:47.256834+0000","last_undegraded":"2023-03-03T07:55:47.256834+0000","last_fullsized":"2023-03-03T07:55:47.256834+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:46.105554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263772+0000","last_change":"2023-03-03T07:55:47.130303+0000","last_active":"2023-03-03T07:55:47.263772+0000","last_peered":"2023-03-03T07:55:47.263772+0000","last_clean":"2023-03-03T07:55:47.263772+0000","last_became_active":"2023-03-03T07:55:47.129900+0000","last_became_peered":"2023-03-03T07:55:47.129900+0000","last_unstale":"2023-03-03T07:55:47.263772+0000","last_undegraded":"2023-03-03T07:55:47.263772+0000","last_fullsized":"2023-03-03T07:55:47.263772+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262173+0000","last_change":"2023-03-03T07:55:47.134218+0000","last_active":"2023-03-03T07:55:47.262173+0000","last_peered":"2023-03-03T07:55:47.262173+0000","last_clean":"2023-03-03T07:55:47.262173+0000","last_became_active":"2023-03-03T07:55:47.133821+0000","last_became_peered":"2023-03-03T07:55:47.133821+0000","last_unstale":"2023-03-03T07:55:47.262173+0000","last_undegraded":"2023-03-03T07:55:47.262173+0000","last_fullsized":"2023-03-03T07:55:47.262173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:15.870956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.265039+0000","last_change":"2023-03-03T07:55:47.116584+0000","last_active":"2023-03-03T07:55:47.265039+0000","last_peered":"2023-03-03T07:55:47.265039+0000","last_clean":"2023-03-03T07:55:47.265039+0000","last_became_active":"2023-03-03T07:55:47.116307+0000","last_became_peered":"2023-03-03T07:55:47.116307+0000","last_unstale":"2023-03-03T07:55:47.265039+0000","last_undegraded":"2023-03-03T07:55:47.265039+0000","last_fullsized":"2023-03-03T07:55:47.265039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:24.117627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259139+0000","last_change":"2023-03-03T07:55:47.120993+0000","last_active":"2023-03-03T07:55:47.259139+0000","last_peered":"2023-03-03T07:55:47.259139+0000","last_clean":"2023-03-03T07:55:47.259139+0000","last_became_active":"2023-03-03T07:55:47.119211+0000","last_became_peered":"2023-03-03T07:55:47.119211+0000","last_unstale":"2023-03-03T07:55:47.259139+0000","last_undegraded":"2023-03-03T07:55:47.259139+0000","last_fullsized":"2023-03-03T07:55:47.259139+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:34:01.733177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262076+0000","last_change":"2023-03-03T07:55:47.139910+0000","last_active":"2023-03-03T07:55:47.262076+0000","last_peered":"2023-03-03T07:55:47.262076+0000","last_clean":"2023-03-03T07:55:47.262076+0000","last_became_active":"2023-03-03T07:55:47.138659+0000","last_became_peered":"2023-03-03T07:55:47.138659+0000","last_unstale":"2023-03-03T07:55:47.262076+0000","last_undegraded":"2023-03-03T07:55:47.262076+0000","last_fullsized":"2023-03-03T07:55:47.262076+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:42:18.110771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263673+0000","last_change":"2023-03-03T07:55:47.136724+0000","last_active":"2023-03-03T07:55:47.263673+0000","last_peered":"2023-03-03T07:55:47.263673+0000","last_clean":"2023-03-03T07:55:47.263673+0000","last_became_active":"2023-03-03T07:55:47.136066+0000","last_became_peered":"2023-03-03T07:55:47.136066+0000","last_unstale":"2023-03-03T07:55:47.263673+0000","last_undegraded":"2023-03-03T07:55:47.263673+0000","last_fullsized":"2023-03-03T07:55:47.263673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263607+0000","last_change":"2023-03-03T07:55:47.137112+0000","last_active":"2023-03-03T07:55:47.263607+0000","last_peered":"2023-03-03T07:55:47.263607+0000","last_clean":"2023-03-03T07:55:47.263607+0000","last_became_active":"2023-03-03T07:55:47.136698+0000","last_became_peered":"2023-03-03T07:55:47.136698+0000","last_unstale":"2023-03-03T07:55:47.263607+0000","last_undegraded":"2023-03-03T07:55:47.263607+0000","last_fullsized":"2023-03-03T07:55:47.263607+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259277+0000","last_change":"2023-03-03T07:55:47.123107+0000","last_active":"2023-03-03T07:55:47.259277+0000","last_peered":"2023-03-03T07:55:47.259277+0000","last_clean":"2023-03-03T07:55:47.259277+0000","last_became_active":"2023-03-03T07:55:47.122842+0000","last_became_peered":"2023-03-03T07:55:47.122842+0000","last_unstale":"2023-03-03T07:55:47.259277+0000","last_undegraded":"2023-03-03T07:55:47.259277+0000","last_fullsized":"2023-03-03T07:55:47.259277+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:42:18.110771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263818+0000","last_change":"2023-03-03T07:55:47.115719+0000","last_active":"2023-03-03T07:55:47.263818+0000","last_peered":"2023-03-03T07:55:47.263818+0000","last_clean":"2023-03-03T07:55:47.263818+0000","last_became_active":"2023-03-03T07:55:47.115404+0000","last_became_peered":"2023-03-03T07:55:47.115404+0000","last_unstale":"2023-03-03T07:55:47.263818+0000","last_undegraded":"2023-03-03T07:55:47.263818+0000","last_fullsized":"2023-03-03T07:55:47.263818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:21.542695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262181+0000","last_change":"2023-03-03T07:55:47.135352+0000","last_active":"2023-03-03T07:55:47.262181+0000","last_peered":"2023-03-03T07:55:47.262181+0000","last_clean":"2023-03-03T07:55:47.262181+0000","last_became_active":"2023-03-03T07:55:47.135101+0000","last_became_peered":"2023-03-03T07:55:47.135101+0000","last_unstale":"2023-03-03T07:55:47.262181+0000","last_undegraded":"2023-03-03T07:55:47.262181+0000","last_fullsized":"2023-03-03T07:55:47.262181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:46:04.349618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.256802+0000","last_change":"2023-03-03T07:55:47.135595+0000","last_active":"2023-03-03T07:55:47.256802+0000","last_peered":"2023-03-03T07:55:47.256802+0000","last_clean":"2023-03-03T07:55:47.256802+0000","last_became_active":"2023-03-03T07:55:47.135349+0000","last_became_peered":"2023-03-03T07:55:47.135349+0000","last_unstale":"2023-03-03T07:55:47.256802+0000","last_undegraded":"2023-03-03T07:55:47.256802+0000","last_fullsized":"2023-03-03T07:55:47.256802+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:24.117627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.256771+0000","last_change":"2023-03-03T07:55:47.125072+0000","last_active":"2023-03-03T07:55:47.256771+0000","last_peered":"2023-03-03T07:55:47.256771+0000","last_clean":"2023-03-03T07:55:47.256771+0000","last_became_active":"2023-03-03T07:55:47.124767+0000","last_became_peered":"2023-03-03T07:55:47.124767+0000","last_unstale":"2023-03-03T07:55:47.256771+0000","last_undegraded":"2023-03-03T07:55:47.256771+0000","last_fullsized":"2023-03-03T07:55:47.256771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:34.910325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263657+0000","last_change":"2023-03-03T07:55:47.128730+0000","last_active":"2023-03-03T07:55:47.263657+0000","last_peered":"2023-03-03T07:55:47.263657+0000","last_clean":"2023-03-03T07:55:47.263657+0000","last_became_active":"2023-03-03T07:55:47.127578+0000","last_became_peered":"2023-03-03T07:55:47.127578+0000","last_unstale":"2023-03-03T07:55:47.263657+0000","last_undegraded":"2023-03-03T07:55:47.263657+0000","last_fullsized":"2023-03-03T07:55:47.263657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263702+0000","last_change":"2023-03-03T07:55:47.137768+0000","last_active":"2023-03-03T07:55:47.263702+0000","last_peered":"2023-03-03T07:55:47.263702+0000","last_clean":"2023-03-03T07:55:47.263702+0000","last_became_active":"2023-03-03T07:55:47.136957+0000","last_became_peered":"2023-03-03T07:55:47.136957+0000","last_unstale":"2023-03-03T07:55:47.263702+0000","last_undegraded":"2023-03-03T07:55:47.263702+0000","last_fullsized":"2023-03-03T07:55:47.263702+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262039+0000","last_change":"2023-03-03T07:55:47.139717+0000","last_active":"2023-03-03T07:55:47.262039+0000","last_peered":"2023-03-03T07:55:47.262039+0000","last_clean":"2023-03-03T07:55:47.262039+0000","last_became_active":"2023-03-03T07:55:47.138784+0000","last_became_peered":"2023-03-03T07:55:47.138784+0000","last_unstale":"2023-03-03T07:55:47.262039+0000","last_undegraded":"2023-03-03T07:55:47.262039+0000","last_fullsized":"2023-03-03T07:55:47.262039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:34:01.733177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259293+0000","last_change":"2023-03-03T07:55:47.121662+0000","last_active":"2023-03-03T07:55:47.259293+0000","last_peered":"2023-03-03T07:55:47.259293+0000","last_clean":"2023-03-03T07:55:47.259293+0000","last_became_active":"2023-03-03T07:55:47.118420+0000","last_became_peered":"2023-03-03T07:55:47.118420+0000","last_unstale":"2023-03-03T07:55:47.259293+0000","last_undegraded":"2023-03-03T07:55:47.259293+0000","last_fullsized":"2023-03-03T07:55:47.259293+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:36:18.587660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262199+0000","last_change":"2023-03-03T07:55:47.132197+0000","last_active":"2023-03-03T07:55:47.262199+0000","last_peered":"2023-03-03T07:55:47.262199+0000","last_clean":"2023-03-03T07:55:47.262199+0000","last_became_active":"2023-03-03T07:55:47.131599+0000","last_became_peered":"2023-03-03T07:55:47.131599+0000","last_unstale":"2023-03-03T07:55:47.262199+0000","last_undegraded":"2023-03-03T07:55:47.262199+0000","last_fullsized":"2023-03-03T07:55:47.262199+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:57:41.948092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.256819+0000","last_change":"2023-03-03T07:55:47.129930+0000","last_active":"2023-03-03T07:55:47.256819+0000","last_peered":"2023-03-03T07:55:47.256819+0000","last_clean":"2023-03-03T07:55:47.256819+0000","last_became_active":"2023-03-03T07:55:47.129569+0000","last_became_peered":"2023-03-03T07:55:47.129569+0000","last_unstale":"2023-03-03T07:55:47.256819+0000","last_undegraded":"2023-03-03T07:55:47.256819+0000","last_fullsized":"2023-03-03T07:55:47.256819+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:40.146598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262146+0000","last_change":"2023-03-03T07:55:47.133303+0000","last_active":"2023-03-03T07:55:47.262146+0000","last_peered":"2023-03-03T07:55:47.262146+0000","last_clean":"2023-03-03T07:55:47.262146+0000","last_became_active":"2023-03-03T07:55:47.131841+0000","last_became_peered":"2023-03-03T07:55:47.131841+0000","last_unstale":"2023-03-03T07:55:47.262146+0000","last_undegraded":"2023-03-03T07:55:47.262146+0000","last_fullsized":"2023-03-03T07:55:47.262146+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:13:04.772369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263745+0000","last_change":"2023-03-03T07:55:47.134085+0000","last_active":"2023-03-03T07:55:47.263745+0000","last_peered":"2023-03-03T07:55:47.263745+0000","last_clean":"2023-03-03T07:55:47.263745+0000","last_became_active":"2023-03-03T07:55:47.133590+0000","last_became_peered":"2023-03-03T07:55:47.133590+0000","last_unstale":"2023-03-03T07:55:47.263745+0000","last_undegraded":"2023-03-03T07:55:47.263745+0000","last_fullsized":"2023-03-03T07:55:47.263745+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259059+0000","last_change":"2023-03-03T07:55:47.128927+0000","last_active":"2023-03-03T07:55:47.259059+0000","last_peered":"2023-03-03T07:55:47.259059+0000","last_clean":"2023-03-03T07:55:47.259059+0000","last_became_active":"2023-03-03T07:55:47.128642+0000","last_became_peered":"2023-03-03T07:55:47.128642+0000","last_unstale":"2023-03-03T07:55:47.259059+0000","last_undegraded":"2023-03-03T07:55:47.259059+0000","last_fullsized":"2023-03-03T07:55:47.259059+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261331+0000","last_change":"2023-03-03T07:55:47.114026+0000","last_active":"2023-03-03T07:55:47.261331+0000","last_peered":"2023-03-03T07:55:47.261331+0000","last_clean":"2023-03-03T07:55:47.261331+0000","last_became_active":"2023-03-03T07:55:47.113732+0000","last_became_peered":"2023-03-03T07:55:47.113732+0000","last_unstale":"2023-03-03T07:55:47.261331+0000","last_undegraded":"2023-03-03T07:55:47.261331+0000","last_fullsized":"2023-03-03T07:55:47.261331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:34:01.733177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263655+0000","last_change":"2023-03-03T07:55:47.137565+0000","last_active":"2023-03-03T07:55:47.263655+0000","last_peered":"2023-03-03T07:55:47.263655+0000","last_clean":"2023-03-03T07:55:47.263655+0000","last_became_active":"2023-03-03T07:55:47.136713+0000","last_became_peered":"2023-03-03T07:55:47.136713+0000","last_unstale":"2023-03-03T07:55:47.263655+0000","last_undegraded":"2023-03-03T07:55:47.263655+0000","last_fullsized":"2023-03-03T07:55:47.263655+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259593+0000","last_change":"2023-03-03T07:55:47.124622+0000","last_active":"2023-03-03T07:55:47.259593+0000","last_peered":"2023-03-03T07:55:47.259593+0000","last_clean":"2023-03-03T07:55:47.259593+0000","last_became_active":"2023-03-03T07:55:47.124221+0000","last_became_peered":"2023-03-03T07:55:47.124221+0000","last_unstale":"2023-03-03T07:55:47.259593+0000","last_undegraded":"2023-03-03T07:55:47.259593+0000","last_fullsized":"2023-03-03T07:55:47.259593+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259563+0000","last_change":"2023-03-03T07:55:47.123628+0000","last_active":"2023-03-03T07:55:47.259563+0000","last_peered":"2023-03-03T07:55:47.259563+0000","last_clean":"2023-03-03T07:55:47.259563+0000","last_became_active":"2023-03-03T07:55:47.123275+0000","last_became_peered":"2023-03-03T07:55:47.123275+0000","last_unstale":"2023-03-03T07:55:47.259563+0000","last_undegraded":"2023-03-03T07:55:47.259563+0000","last_fullsized":"2023-03-03T07:55:47.259563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:21.542695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263729+0000","last_change":"2023-03-03T07:55:47.127078+0000","last_active":"2023-03-03T07:55:47.263729+0000","last_peered":"2023-03-03T07:55:47.263729+0000","last_clean":"2023-03-03T07:55:47.263729+0000","last_became_active":"2023-03-03T07:55:47.126560+0000","last_became_peered":"2023-03-03T07:55:47.126560+0000","last_unstale":"2023-03-03T07:55:47.263729+0000","last_undegraded":"2023-03-03T07:55:47.263729+0000","last_fullsized":"2023-03-03T07:55:47.263729+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262095+0000","last_change":"2023-03-03T07:55:46.566968+0000","last_active":"2023-03-03T07:55:47.262095+0000","last_peered":"2023-03-03T07:55:47.262095+0000","last_clean":"2023-03-03T07:55:47.262095+0000","last_became_active":"2023-03-03T07:55:46.566628+0000","last_became_peered":"2023-03-03T07:55:46.566628+0000","last_unstale":"2023-03-03T07:55:47.262095+0000","last_undegraded":"2023-03-03T07:55:47.262095+0000","last_fullsized":"2023-03-03T07:55:47.262095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:11.895405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261401+0000","last_change":"2023-03-03T07:55:47.124330+0000","last_active":"2023-03-03T07:55:47.261401+0000","last_peered":"2023-03-03T07:55:47.261401+0000","last_clean":"2023-03-03T07:55:47.261401+0000","last_became_active":"2023-03-03T07:55:47.124015+0000","last_became_peered":"2023-03-03T07:55:47.124015+0000","last_unstale":"2023-03-03T07:55:47.261401+0000","last_undegraded":"2023-03-03T07:55:47.261401+0000","last_fullsized":"2023-03-03T07:55:47.261401+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:57:41.948092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259432+0000","last_change":"2023-03-03T07:55:46.562055+0000","last_active":"2023-03-03T07:55:47.259432+0000","last_peered":"2023-03-03T07:55:47.259432+0000","last_clean":"2023-03-03T07:55:47.259432+0000","last_became_active":"2023-03-03T07:55:46.561625+0000","last_became_peered":"2023-03-03T07:55:46.561625+0000","last_unstale":"2023-03-03T07:55:47.259432+0000","last_undegraded":"2023-03-03T07:55:47.259432+0000","last_fullsized":"2023-03-03T07:55:47.259432+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259192+0000","last_change":"2023-03-03T07:55:47.129628+0000","last_active":"2023-03-03T07:55:47.259192+0000","last_peered":"2023-03-03T07:55:47.259192+0000","last_clean":"2023-03-03T07:55:47.259192+0000","last_became_active":"2023-03-03T07:55:47.129356+0000","last_became_peered":"2023-03-03T07:55:47.129356+0000","last_unstale":"2023-03-03T07:55:47.259192+0000","last_undegraded":"2023-03-03T07:55:47.259192+0000","last_fullsized":"2023-03-03T07:55:47.259192+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261268+0000","last_change":"2023-03-03T07:55:46.559242+0000","last_active":"2023-03-03T07:55:47.261268+0000","last_peered":"2023-03-03T07:55:47.261268+0000","last_clean":"2023-03-03T07:55:47.261268+0000","last_became_active":"2023-03-03T07:55:46.558772+0000","last_became_peered":"2023-03-03T07:55:46.558772+0000","last_unstale":"2023-03-03T07:55:47.261268+0000","last_undegraded":"2023-03-03T07:55:47.261268+0000","last_fullsized":"2023-03-03T07:55:47.261268+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.265064+0000","last_change":"2023-03-03T07:55:47.114645+0000","last_active":"2023-03-03T07:55:47.265064+0000","last_peered":"2023-03-03T07:55:47.265064+0000","last_clean":"2023-03-03T07:55:47.265064+0000","last_became_active":"2023-03-03T07:55:47.114306+0000","last_became_peered":"2023-03-03T07:55:47.114306+0000","last_unstale":"2023-03-03T07:55:47.265064+0000","last_undegraded":"2023-03-03T07:55:47.265064+0000","last_fullsized":"2023-03-03T07:55:47.265064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:29.066787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259309+0000","last_change":"2023-03-03T07:55:46.567843+0000","last_active":"2023-03-03T07:55:47.259309+0000","last_peered":"2023-03-03T07:55:47.259309+0000","last_clean":"2023-03-03T07:55:47.259309+0000","last_became_active":"2023-03-03T07:55:46.567536+0000","last_became_peered":"2023-03-03T07:55:46.567536+0000","last_unstale":"2023-03-03T07:55:47.259309+0000","last_undegraded":"2023-03-03T07:55:47.259309+0000","last_fullsized":"2023-03-03T07:55:47.259309+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261252+0000","last_change":"2023-03-03T07:55:47.126080+0000","last_active":"2023-03-03T07:55:47.261252+0000","last_peered":"2023-03-03T07:55:47.261252+0000","last_clean":"2023-03-03T07:55:47.261252+0000","last_became_active":"2023-03-03T07:55:47.125819+0000","last_became_peered":"2023-03-03T07:55:47.125819+0000","last_unstale":"2023-03-03T07:55:47.261252+0000","last_undegraded":"2023-03-03T07:55:47.261252+0000","last_fullsized":"2023-03-03T07:55:47.261252+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:42:18.110771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263807+0000","last_change":"2023-03-03T07:55:46.564685+0000","last_active":"2023-03-03T07:55:47.263807+0000","last_peered":"2023-03-03T07:55:47.263807+0000","last_clean":"2023-03-03T07:55:47.263807+0000","last_became_active":"2023-03-03T07:55:46.564401+0000","last_became_peered":"2023-03-03T07:55:46.564401+0000","last_unstale":"2023-03-03T07:55:47.263807+0000","last_undegraded":"2023-03-03T07:55:47.263807+0000","last_fullsized":"2023-03-03T07:55:47.263807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261322+0000","last_change":"2023-03-03T07:55:47.126626+0000","last_active":"2023-03-03T07:55:47.261322+0000","last_peered":"2023-03-03T07:55:47.261322+0000","last_clean":"2023-03-03T07:55:47.261322+0000","last_became_active":"2023-03-03T07:55:47.126341+0000","last_became_peered":"2023-03-03T07:55:47.126341+0000","last_unstale":"2023-03-03T07:55:47.261322+0000","last_undegraded":"2023-03-03T07:55:47.261322+0000","last_fullsized":"2023-03-03T07:55:47.261322+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"23'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263754+0000","last_change":"2023-03-03T07:55:46.564428+0000","last_active":"2023-03-03T07:55:47.263754+0000","last_peered":"2023-03-03T07:55:47.263754+0000","last_clean":"2023-03-03T07:55:47.263754+0000","last_became_active":"2023-03-03T07:55:46.564082+0000","last_became_peered":"2023-03-03T07:55:46.564082+0000","last_unstale":"2023-03-03T07:55:47.263754+0000","last_undegraded":"2023-03-03T07:55:47.263754+0000","last_fullsized":"2023-03-03T07:55:47.263754+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:28:57.536524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259449+0000","last_change":"2023-03-03T07:55:47.135104+0000","last_active":"2023-03-03T07:55:47.259449+0000","last_peered":"2023-03-03T07:55:47.259449+0000","last_clean":"2023-03-03T07:55:47.259449+0000","last_became_active":"2023-03-03T07:55:47.134610+0000","last_became_peered":"2023-03-03T07:55:47.134610+0000","last_unstale":"2023-03-03T07:55:47.259449+0000","last_undegraded":"2023-03-03T07:55:47.259449+0000","last_fullsized":"2023-03-03T07:55:47.259449+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"23'6","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262163+0000","last_change":"2023-03-03T07:55:46.563020+0000","last_active":"2023-03-03T07:55:47.262163+0000","last_peered":"2023-03-03T07:55:47.262163+0000","last_clean":"2023-03-03T07:55:47.262163+0000","last_became_active":"2023-03-03T07:55:46.562674+0000","last_became_peered":"2023-03-03T07:55:46.562674+0000","last_unstale":"2023-03-03T07:55:47.262163+0000","last_undegraded":"2023-03-03T07:55:47.262163+0000","last_fullsized":"2023-03-03T07:55:47.262163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:30:20.455024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262117+0000","last_change":"2023-03-03T07:55:47.131440+0000","last_active":"2023-03-03T07:55:47.262117+0000","last_peered":"2023-03-03T07:55:47.262117+0000","last_clean":"2023-03-03T07:55:47.262117+0000","last_became_active":"2023-03-03T07:55:47.131159+0000","last_became_peered":"2023-03-03T07:55:47.131159+0000","last_unstale":"2023-03-03T07:55:47.262117+0000","last_undegraded":"2023-03-03T07:55:47.262117+0000","last_fullsized":"2023-03-03T07:55:47.262117+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:29.837317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263734+0000","last_change":"2023-03-03T07:55:46.560188+0000","last_active":"2023-03-03T07:55:47.263734+0000","last_peered":"2023-03-03T07:55:47.263734+0000","last_clean":"2023-03-03T07:55:47.263734+0000","last_became_active":"2023-03-03T07:55:46.559853+0000","last_became_peered":"2023-03-03T07:55:46.559853+0000","last_unstale":"2023-03-03T07:55:47.263734+0000","last_undegraded":"2023-03-03T07:55:47.263734+0000","last_fullsized":"2023-03-03T07:55:47.263734+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261392+0000","last_change":"2023-03-03T07:55:47.131204+0000","last_active":"2023-03-03T07:55:47.261392+0000","last_peered":"2023-03-03T07:55:47.261392+0000","last_clean":"2023-03-03T07:55:47.261392+0000","last_became_active":"2023-03-03T07:55:47.130951+0000","last_became_peered":"2023-03-03T07:55:47.130951+0000","last_unstale":"2023-03-03T07:55:47.261392+0000","last_undegraded":"2023-03-03T07:55:47.261392+0000","last_fullsized":"2023-03-03T07:55:47.261392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259033+0000","last_change":"2023-03-03T07:55:46.567332+0000","last_active":"2023-03-03T07:55:47.259033+0000","last_peered":"2023-03-03T07:55:47.259033+0000","last_clean":"2023-03-03T07:55:47.259033+0000","last_became_active":"2023-03-03T07:55:46.567006+0000","last_became_peered":"2023-03-03T07:55:46.567006+0000","last_unstale":"2023-03-03T07:55:47.259033+0000","last_undegraded":"2023-03-03T07:55:47.259033+0000","last_fullsized":"2023-03-03T07:55:47.259033+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263740+0000","last_change":"2023-03-03T07:55:47.126887+0000","last_active":"2023-03-03T07:55:47.263740+0000","last_peered":"2023-03-03T07:55:47.263740+0000","last_clean":"2023-03-03T07:55:47.263740+0000","last_became_active":"2023-03-03T07:55:47.125992+0000","last_became_peered":"2023-03-03T07:55:47.125992+0000","last_unstale":"2023-03-03T07:55:47.263740+0000","last_undegraded":"2023-03-03T07:55:47.263740+0000","last_fullsized":"2023-03-03T07:55:47.263740+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263684+0000","last_change":"2023-03-03T07:55:46.559018+0000","last_active":"2023-03-03T07:55:47.263684+0000","last_peered":"2023-03-03T07:55:47.263684+0000","last_clean":"2023-03-03T07:55:47.263684+0000","last_became_active":"2023-03-03T07:55:46.558693+0000","last_became_peered":"2023-03-03T07:55:46.558693+0000","last_unstale":"2023-03-03T07:55:47.263684+0000","last_undegraded":"2023-03-03T07:55:47.263684+0000","last_fullsized":"2023-03-03T07:55:47.263684+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:28.058494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259175+0000","last_change":"2023-03-03T07:55:47.123304+0000","last_active":"2023-03-03T07:55:47.259175+0000","last_peered":"2023-03-03T07:55:47.259175+0000","last_clean":"2023-03-03T07:55:47.259175+0000","last_became_active":"2023-03-03T07:55:47.118375+0000","last_became_peered":"2023-03-03T07:55:47.118375+0000","last_unstale":"2023-03-03T07:55:47.259175+0000","last_undegraded":"2023-03-03T07:55:47.259175+0000","last_fullsized":"2023-03-03T07:55:47.259175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:49:25.482566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"23'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.264975+0000","last_change":"2023-03-03T07:55:46.565523+0000","last_active":"2023-03-03T07:55:47.264975+0000","last_peered":"2023-03-03T07:55:47.264975+0000","last_clean":"2023-03-03T07:55:47.264975+0000","last_became_active":"2023-03-03T07:55:46.565158+0000","last_became_peered":"2023-03-03T07:55:46.565158+0000","last_unstale":"2023-03-03T07:55:47.264975+0000","last_undegraded":"2023-03-03T07:55:47.264975+0000","last_fullsized":"2023-03-03T07:55:47.264975+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:57.246539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261430+0000","last_change":"2023-03-03T07:55:47.128607+0000","last_active":"2023-03-03T07:55:47.261430+0000","last_peered":"2023-03-03T07:55:47.261430+0000","last_clean":"2023-03-03T07:55:47.261430+0000","last_became_active":"2023-03-03T07:55:47.126465+0000","last_became_peered":"2023-03-03T07:55:47.126465+0000","last_unstale":"2023-03-03T07:55:47.261430+0000","last_undegraded":"2023-03-03T07:55:47.261430+0000","last_fullsized":"2023-03-03T07:55:47.261430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:21.542695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263798+0000","last_change":"2023-03-03T07:55:46.560647+0000","last_active":"2023-03-03T07:55:47.263798+0000","last_peered":"2023-03-03T07:55:47.263798+0000","last_clean":"2023-03-03T07:55:47.263798+0000","last_became_active":"2023-03-03T07:55:46.559663+0000","last_became_peered":"2023-03-03T07:55:46.559663+0000","last_unstale":"2023-03-03T07:55:47.263798+0000","last_undegraded":"2023-03-03T07:55:47.263798+0000","last_fullsized":"2023-03-03T07:55:47.263798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:54:12.785264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263622+0000","last_change":"2023-03-03T07:55:47.126608+0000","last_active":"2023-03-03T07:55:47.263622+0000","last_peered":"2023-03-03T07:55:47.263622+0000","last_clean":"2023-03-03T07:55:47.263622+0000","last_became_active":"2023-03-03T07:55:47.126236+0000","last_became_peered":"2023-03-03T07:55:47.126236+0000","last_unstale":"2023-03-03T07:55:47.263622+0000","last_undegraded":"2023-03-03T07:55:47.263622+0000","last_fullsized":"2023-03-03T07:55:47.263622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:03:11.053940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263787+0000","last_change":"2023-03-03T07:55:46.558232+0000","last_active":"2023-03-03T07:55:47.263787+0000","last_peered":"2023-03-03T07:55:47.263787+0000","last_clean":"2023-03-03T07:55:47.263787+0000","last_became_active":"2023-03-03T07:55:46.557732+0000","last_became_peered":"2023-03-03T07:55:46.557732+0000","last_unstale":"2023-03-03T07:55:47.263787+0000","last_undegraded":"2023-03-03T07:55:47.263787+0000","last_fullsized":"2023-03-03T07:55:47.263787+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262049+0000","last_change":"2023-03-03T07:55:47.139286+0000","last_active":"2023-03-03T07:55:47.262049+0000","last_peered":"2023-03-03T07:55:47.262049+0000","last_clean":"2023-03-03T07:55:47.262049+0000","last_became_active":"2023-03-03T07:55:47.138539+0000","last_became_peered":"2023-03-03T07:55:47.138539+0000","last_unstale":"2023-03-03T07:55:47.262049+0000","last_undegraded":"2023-03-03T07:55:47.262049+0000","last_fullsized":"2023-03-03T07:55:47.262049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:56.705526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263776+0000","last_change":"2023-03-03T07:55:46.559244+0000","last_active":"2023-03-03T07:55:47.263776+0000","last_peered":"2023-03-03T07:55:47.263776+0000","last_clean":"2023-03-03T07:55:47.263776+0000","last_became_active":"2023-03-03T07:55:46.558832+0000","last_became_peered":"2023-03-03T07:55:46.558832+0000","last_unstale":"2023-03-03T07:55:47.263776+0000","last_undegraded":"2023-03-03T07:55:47.263776+0000","last_fullsized":"2023-03-03T07:55:47.263776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:06:23.082465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.265028+0000","last_change":"2023-03-03T07:55:47.128420+0000","last_active":"2023-03-03T07:55:47.265028+0000","last_peered":"2023-03-03T07:55:47.265028+0000","last_clean":"2023-03-03T07:55:47.265028+0000","last_became_active":"2023-03-03T07:55:47.128117+0000","last_became_peered":"2023-03-03T07:55:47.128117+0000","last_unstale":"2023-03-03T07:55:47.265028+0000","last_undegraded":"2023-03-03T07:55:47.265028+0000","last_fullsized":"2023-03-03T07:55:47.265028+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:28:58.544816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263713+0000","last_change":"2023-03-03T07:55:46.563195+0000","last_active":"2023-03-03T07:55:47.263713+0000","last_peered":"2023-03-03T07:55:47.263713+0000","last_clean":"2023-03-03T07:55:47.263713+0000","last_became_active":"2023-03-03T07:55:46.562774+0000","last_became_peered":"2023-03-03T07:55:46.562774+0000","last_unstale":"2023-03-03T07:55:47.263713+0000","last_undegraded":"2023-03-03T07:55:47.263713+0000","last_fullsized":"2023-03-03T07:55:47.263713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:23.109334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259102+0000","last_change":"2023-03-03T07:55:47.132833+0000","last_active":"2023-03-03T07:55:47.259102+0000","last_peered":"2023-03-03T07:55:47.259102+0000","last_clean":"2023-03-03T07:55:47.259102+0000","last_became_active":"2023-03-03T07:55:47.132530+0000","last_became_peered":"2023-03-03T07:55:47.132530+0000","last_unstale":"2023-03-03T07:55:47.259102+0000","last_undegraded":"2023-03-03T07:55:47.259102+0000","last_fullsized":"2023-03-03T07:55:47.259102+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261380+0000","last_change":"2023-03-03T07:55:46.561556+0000","last_active":"2023-03-03T07:55:47.261380+0000","last_peered":"2023-03-03T07:55:47.261380+0000","last_clean":"2023-03-03T07:55:47.261380+0000","last_became_active":"2023-03-03T07:55:46.561273+0000","last_became_peered":"2023-03-03T07:55:46.561273+0000","last_unstale":"2023-03-03T07:55:47.261380+0000","last_undegraded":"2023-03-03T07:55:47.261380+0000","last_fullsized":"2023-03-03T07:55:47.261380+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:39.138305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262155+0000","last_change":"2023-03-03T07:55:47.138654+0000","last_active":"2023-03-03T07:55:47.262155+0000","last_peered":"2023-03-03T07:55:47.262155+0000","last_clean":"2023-03-03T07:55:47.262155+0000","last_became_active":"2023-03-03T07:55:47.131453+0000","last_became_peered":"2023-03-03T07:55:47.131453+0000","last_unstale":"2023-03-03T07:55:47.262155+0000","last_undegraded":"2023-03-03T07:55:47.262155+0000","last_fullsized":"2023-03-03T07:55:47.262155+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:36:18.587660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.265000+0000","last_change":"2023-03-03T07:55:46.563763+0000","last_active":"2023-03-03T07:55:47.265000+0000","last_peered":"2023-03-03T07:55:47.265000+0000","last_clean":"2023-03-03T07:55:47.265000+0000","last_became_active":"2023-03-03T07:55:46.563433+0000","last_became_peered":"2023-03-03T07:55:46.563433+0000","last_unstale":"2023-03-03T07:55:47.265000+0000","last_undegraded":"2023-03-03T07:55:47.265000+0000","last_fullsized":"2023-03-03T07:55:47.265000+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:12.972346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262191+0000","last_change":"2023-03-03T07:55:47.134804+0000","last_active":"2023-03-03T07:55:47.262191+0000","last_peered":"2023-03-03T07:55:47.262191+0000","last_clean":"2023-03-03T07:55:47.262191+0000","last_became_active":"2023-03-03T07:55:47.134504+0000","last_became_peered":"2023-03-03T07:55:47.134504+0000","last_unstale":"2023-03-03T07:55:47.262191+0000","last_undegraded":"2023-03-03T07:55:47.262191+0000","last_fullsized":"2023-03-03T07:55:47.262191+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262068+0000","last_change":"2023-03-03T07:55:46.561627+0000","last_active":"2023-03-03T07:55:47.262068+0000","last_peered":"2023-03-03T07:55:47.262068+0000","last_clean":"2023-03-03T07:55:47.262068+0000","last_became_active":"2023-03-03T07:55:46.561129+0000","last_became_peered":"2023-03-03T07:55:46.561129+0000","last_unstale":"2023-03-03T07:55:47.262068+0000","last_undegraded":"2023-03-03T07:55:47.262068+0000","last_fullsized":"2023-03-03T07:55:47.262068+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:12.972346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262208+0000","last_change":"2023-03-03T07:55:47.119813+0000","last_active":"2023-03-03T07:55:47.262208+0000","last_peered":"2023-03-03T07:55:47.262208+0000","last_clean":"2023-03-03T07:55:47.262208+0000","last_became_active":"2023-03-03T07:55:47.119517+0000","last_became_peered":"2023-03-03T07:55:47.119517+0000","last_unstale":"2023-03-03T07:55:47.262208+0000","last_undegraded":"2023-03-03T07:55:47.262208+0000","last_fullsized":"2023-03-03T07:55:47.262208+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:56:43.323290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261240+0000","last_change":"2023-03-03T07:55:46.557334+0000","last_active":"2023-03-03T07:55:47.261240+0000","last_peered":"2023-03-03T07:55:47.261240+0000","last_clean":"2023-03-03T07:55:47.261240+0000","last_became_active":"2023-03-03T07:55:46.556973+0000","last_became_peered":"2023-03-03T07:55:46.556973+0000","last_unstale":"2023-03-03T07:55:47.261240+0000","last_undegraded":"2023-03-03T07:55:47.261240+0000","last_fullsized":"2023-03-03T07:55:47.261240+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259625+0000","last_change":"2023-03-03T07:55:47.123429+0000","last_active":"2023-03-03T07:55:47.259625+0000","last_peered":"2023-03-03T07:55:47.259625+0000","last_clean":"2023-03-03T07:55:47.259625+0000","last_became_active":"2023-03-03T07:55:47.123131+0000","last_became_peered":"2023-03-03T07:55:47.123131+0000","last_unstale":"2023-03-03T07:55:47.259625+0000","last_undegraded":"2023-03-03T07:55:47.259625+0000","last_fullsized":"2023-03-03T07:55:47.259625+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.114110+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T07:55:47.114110+0000","last_peered":"2023-03-03T07:55:47.114110+0000","last_clean":"2023-03-03T07:55:47.114110+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T07:55:47.114110+0000","last_undegraded":"2023-03-03T07:55:47.114110+0000","last_fullsized":"2023-03-03T07:55:47.114110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263752+0000","last_change":"2023-03-03T07:55:46.565316+0000","last_active":"2023-03-03T07:55:47.263752+0000","last_peered":"2023-03-03T07:55:47.263752+0000","last_clean":"2023-03-03T07:55:47.263752+0000","last_became_active":"2023-03-03T07:55:46.565027+0000","last_became_peered":"2023-03-03T07:55:46.565027+0000","last_unstale":"2023-03-03T07:55:47.263752+0000","last_undegraded":"2023-03-03T07:55:47.263752+0000","last_fullsized":"2023-03-03T07:55:47.263752+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:28.058494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259465+0000","last_change":"2023-03-03T07:55:47.124467+0000","last_active":"2023-03-03T07:55:47.259465+0000","last_peered":"2023-03-03T07:55:47.259465+0000","last_clean":"2023-03-03T07:55:47.259465+0000","last_became_active":"2023-03-03T07:55:47.124149+0000","last_became_peered":"2023-03-03T07:55:47.124149+0000","last_unstale":"2023-03-03T07:55:47.259465+0000","last_undegraded":"2023-03-03T07:55:47.259465+0000","last_fullsized":"2023-03-03T07:55:47.259465+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.107246+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T07:55:47.107246+0000","last_peered":"2023-03-03T07:55:47.107246+0000","last_clean":"2023-03-03T07:55:47.107246+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T07:55:47.107246+0000","last_undegraded":"2023-03-03T07:55:47.107246+0000","last_fullsized":"2023-03-03T07:55:47.107246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262126+0000","last_change":"2023-03-03T07:55:46.559574+0000","last_active":"2023-03-03T07:55:47.262126+0000","last_peered":"2023-03-03T07:55:47.262126+0000","last_clean":"2023-03-03T07:55:47.262126+0000","last_became_active":"2023-03-03T07:55:46.559192+0000","last_became_peered":"2023-03-03T07:55:46.559192+0000","last_unstale":"2023-03-03T07:55:47.262126+0000","last_undegraded":"2023-03-03T07:55:47.262126+0000","last_fullsized":"2023-03-03T07:55:47.262126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259529+0000","last_change":"2023-03-03T07:55:47.127703+0000","last_active":"2023-03-03T07:55:47.259529+0000","last_peered":"2023-03-03T07:55:47.259529+0000","last_clean":"2023-03-03T07:55:47.259529+0000","last_became_active":"2023-03-03T07:55:47.126941+0000","last_became_peered":"2023-03-03T07:55:47.126941+0000","last_unstale":"2023-03-03T07:55:47.259529+0000","last_undegraded":"2023-03-03T07:55:47.259529+0000","last_fullsized":"2023-03-03T07:55:47.259529+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:03:11.053940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.120195+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T07:55:47.120195+0000","last_peered":"2023-03-03T07:55:47.120195+0000","last_clean":"2023-03-03T07:55:47.120195+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T07:55:47.120195+0000","last_undegraded":"2023-03-03T07:55:47.120195+0000","last_fullsized":"2023-03-03T07:55:47.120195+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259411+0000","last_change":"2023-03-03T07:55:46.560453+0000","last_active":"2023-03-03T07:55:47.259411+0000","last_peered":"2023-03-03T07:55:47.259411+0000","last_clean":"2023-03-03T07:55:47.259411+0000","last_became_active":"2023-03-03T07:55:46.560140+0000","last_became_peered":"2023-03-03T07:55:46.560140+0000","last_unstale":"2023-03-03T07:55:47.259411+0000","last_undegraded":"2023-03-03T07:55:47.259411+0000","last_fullsized":"2023-03-03T07:55:47.259411+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259609+0000","last_change":"2023-03-03T07:55:47.124151+0000","last_active":"2023-03-03T07:55:47.259609+0000","last_peered":"2023-03-03T07:55:47.259609+0000","last_clean":"2023-03-03T07:55:47.259609+0000","last_became_active":"2023-03-03T07:55:47.123697+0000","last_became_peered":"2023-03-03T07:55:47.123697+0000","last_unstale":"2023-03-03T07:55:47.259609+0000","last_undegraded":"2023-03-03T07:55:47.259609+0000","last_fullsized":"2023-03-03T07:55:47.259609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:44.832850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.107382+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T07:55:47.107382+0000","last_peered":"2023-03-03T07:55:47.107382+0000","last_clean":"2023-03-03T07:55:47.107382+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T07:55:47.107382+0000","last_undegraded":"2023-03-03T07:55:47.107382+0000","last_fullsized":"2023-03-03T07:55:47.107382+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.123508+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T07:55:47.123508+0000","last_peered":"2023-03-03T07:55:47.123508+0000","last_clean":"2023-03-03T07:55:47.123508+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T07:55:47.123508+0000","last_undegraded":"2023-03-03T07:55:47.123508+0000","last_fullsized":"2023-03-03T07:55:47.123508+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261341+0000","last_change":"2023-03-03T07:55:46.559444+0000","last_active":"2023-03-03T07:55:47.261341+0000","last_peered":"2023-03-03T07:55:47.261341+0000","last_clean":"2023-03-03T07:55:47.261341+0000","last_became_active":"2023-03-03T07:55:46.559147+0000","last_became_peered":"2023-03-03T07:55:46.559147+0000","last_unstale":"2023-03-03T07:55:47.261341+0000","last_undegraded":"2023-03-03T07:55:47.261341+0000","last_fullsized":"2023-03-03T07:55:47.261341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:30:20.455024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.265049+0000","last_change":"2023-03-03T07:55:47.134945+0000","last_active":"2023-03-03T07:55:47.265049+0000","last_peered":"2023-03-03T07:55:47.265049+0000","last_clean":"2023-03-03T07:55:47.265049+0000","last_became_active":"2023-03-03T07:55:47.134662+0000","last_became_peered":"2023-03-03T07:55:47.134662+0000","last_unstale":"2023-03-03T07:55:47.265049+0000","last_undegraded":"2023-03-03T07:55:47.265049+0000","last_fullsized":"2023-03-03T07:55:47.265049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:54:13.793557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.123980+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T07:55:47.123980+0000","last_peered":"2023-03-03T07:55:47.123980+0000","last_clean":"2023-03-03T07:55:47.123980+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T07:55:47.123980+0000","last_undegraded":"2023-03-03T07:55:47.123980+0000","last_fullsized":"2023-03-03T07:55:47.123980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259010+0000","last_change":"2023-03-03T07:55:46.567402+0000","last_active":"2023-03-03T07:55:47.259010+0000","last_peered":"2023-03-03T07:55:47.259010+0000","last_clean":"2023-03-03T07:55:47.259010+0000","last_became_active":"2023-03-03T07:55:46.567013+0000","last_became_peered":"2023-03-03T07:55:46.567013+0000","last_unstale":"2023-03-03T07:55:47.259010+0000","last_undegraded":"2023-03-03T07:55:47.259010+0000","last_fullsized":"2023-03-03T07:55:47.259010+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259226+0000","last_change":"2023-03-03T07:55:47.125428+0000","last_active":"2023-03-03T07:55:47.259226+0000","last_peered":"2023-03-03T07:55:47.259226+0000","last_clean":"2023-03-03T07:55:47.259226+0000","last_became_active":"2023-03-03T07:55:47.125113+0000","last_became_peered":"2023-03-03T07:55:47.125113+0000","last_unstale":"2023-03-03T07:55:47.259226+0000","last_undegraded":"2023-03-03T07:55:47.259226+0000","last_fullsized":"2023-03-03T07:55:47.259226+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.122691+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T07:55:47.122691+0000","last_peered":"2023-03-03T07:55:47.122691+0000","last_clean":"2023-03-03T07:55:47.122691+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T07:55:47.122691+0000","last_undegraded":"2023-03-03T07:55:47.122691+0000","last_fullsized":"2023-03-03T07:55:47.122691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259545+0000","last_change":"2023-03-03T07:55:46.556310+0000","last_active":"2023-03-03T07:55:47.259545+0000","last_peered":"2023-03-03T07:55:47.259545+0000","last_clean":"2023-03-03T07:55:47.259545+0000","last_became_active":"2023-03-03T07:55:46.555567+0000","last_became_peered":"2023-03-03T07:55:46.555567+0000","last_unstale":"2023-03-03T07:55:47.259545+0000","last_undegraded":"2023-03-03T07:55:47.259545+0000","last_fullsized":"2023-03-03T07:55:47.259545+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:59:59.503182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259158+0000","last_change":"2023-03-03T07:55:47.119872+0000","last_active":"2023-03-03T07:55:47.259158+0000","last_peered":"2023-03-03T07:55:47.259158+0000","last_clean":"2023-03-03T07:55:47.259158+0000","last_became_active":"2023-03-03T07:55:47.119546+0000","last_became_peered":"2023-03-03T07:55:47.119546+0000","last_unstale":"2023-03-03T07:55:47.259158+0000","last_undegraded":"2023-03-03T07:55:47.259158+0000","last_fullsized":"2023-03-03T07:55:47.259158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:48.506069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261420+0000","last_change":"2023-03-03T07:55:46.559460+0000","last_active":"2023-03-03T07:55:47.261420+0000","last_peered":"2023-03-03T07:55:47.261420+0000","last_clean":"2023-03-03T07:55:47.261420+0000","last_became_active":"2023-03-03T07:55:46.559134+0000","last_became_peered":"2023-03-03T07:55:46.559134+0000","last_unstale":"2023-03-03T07:55:47.261420+0000","last_undegraded":"2023-03-03T07:55:47.261420+0000","last_fullsized":"2023-03-03T07:55:47.261420+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:11.895405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.112543+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T07:55:47.112543+0000","last_peered":"2023-03-03T07:55:47.112543+0000","last_clean":"2023-03-03T07:55:47.112543+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T07:55:47.112543+0000","last_undegraded":"2023-03-03T07:55:47.112543+0000","last_fullsized":"2023-03-03T07:55:47.112543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261225+0000","last_change":"2023-03-03T07:55:47.127917+0000","last_active":"2023-03-03T07:55:47.261225+0000","last_peered":"2023-03-03T07:55:47.261225+0000","last_clean":"2023-03-03T07:55:47.261225+0000","last_became_active":"2023-03-03T07:55:47.127606+0000","last_became_peered":"2023-03-03T07:55:47.127606+0000","last_unstale":"2023-03-03T07:55:47.261225+0000","last_undegraded":"2023-03-03T07:55:47.261225+0000","last_fullsized":"2023-03-03T07:55:47.261225+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:57:10.297255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.105592+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T07:55:47.105592+0000","last_peered":"2023-03-03T07:55:47.105592+0000","last_clean":"2023-03-03T07:55:47.105592+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T07:55:47.105592+0000","last_undegraded":"2023-03-03T07:55:47.105592+0000","last_fullsized":"2023-03-03T07:55:47.105592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"24'1","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-03-03T07:55:49.262039+0000","last_change":"2023-03-03T07:55:46.565365+0000","last_active":"2023-03-03T07:55:49.262039+0000","last_peered":"2023-03-03T07:55:49.262039+0000","last_clean":"2023-03-03T07:55:49.262039+0000","last_became_active":"2023-03-03T07:55:46.565097+0000","last_became_peered":"2023-03-03T07:55:46.565097+0000","last_unstale":"2023-03-03T07:55:49.262039+0000","last_undegraded":"2023-03-03T07:55:49.262039+0000","last_fullsized":"2023-03-03T07:55:49.262039+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:23.109334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262057+0000","last_change":"2023-03-03T07:55:47.139732+0000","last_active":"2023-03-03T07:55:47.262057+0000","last_peered":"2023-03-03T07:55:47.262057+0000","last_clean":"2023-03-03T07:55:47.262057+0000","last_became_active":"2023-03-03T07:55:47.138505+0000","last_became_peered":"2023-03-03T07:55:47.138505+0000","last_unstale":"2023-03-03T07:55:47.262057+0000","last_undegraded":"2023-03-03T07:55:47.262057+0000","last_fullsized":"2023-03-03T07:55:47.262057+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:49:25.482566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259120+0000","last_change":"2023-03-03T07:55:46.568574+0000","last_active":"2023-03-03T07:55:47.259120+0000","last_peered":"2023-03-03T07:55:47.259120+0000","last_clean":"2023-03-03T07:55:47.259120+0000","last_became_active":"2023-03-03T07:55:46.568139+0000","last_became_peered":"2023-03-03T07:55:46.568139+0000","last_unstale":"2023-03-03T07:55:47.259120+0000","last_undegraded":"2023-03-03T07:55:47.259120+0000","last_fullsized":"2023-03-03T07:55:47.259120+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.256785+0000","last_change":"2023-03-03T07:55:47.128846+0000","last_active":"2023-03-03T07:55:47.256785+0000","last_peered":"2023-03-03T07:55:47.256785+0000","last_clean":"2023-03-03T07:55:47.256785+0000","last_became_active":"2023-03-03T07:55:47.128385+0000","last_became_peered":"2023-03-03T07:55:47.128385+0000","last_unstale":"2023-03-03T07:55:47.256785+0000","last_undegraded":"2023-03-03T07:55:47.256785+0000","last_fullsized":"2023-03-03T07:55:47.256785+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:00.511475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263692+0000","last_change":"2023-03-03T07:55:46.560316+0000","last_active":"2023-03-03T07:55:47.263692+0000","last_peered":"2023-03-03T07:55:47.263692+0000","last_clean":"2023-03-03T07:55:47.263692+0000","last_became_active":"2023-03-03T07:55:46.560031+0000","last_became_peered":"2023-03-03T07:55:46.560031+0000","last_unstale":"2023-03-03T07:55:47.263692+0000","last_undegraded":"2023-03-03T07:55:47.263692+0000","last_fullsized":"2023-03-03T07:55:47.263692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:02:33.902033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262135+0000","last_change":"2023-03-03T07:55:47.137604+0000","last_active":"2023-03-03T07:55:47.262135+0000","last_peered":"2023-03-03T07:55:47.262135+0000","last_clean":"2023-03-03T07:55:47.262135+0000","last_became_active":"2023-03-03T07:55:47.131552+0000","last_became_peered":"2023-03-03T07:55:47.131552+0000","last_unstale":"2023-03-03T07:55:47.262135+0000","last_undegraded":"2023-03-03T07:55:47.262135+0000","last_fullsized":"2023-03-03T07:55:47.262135+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:25.771342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.263762+0000","last_change":"2023-03-03T07:55:46.560878+0000","last_active":"2023-03-03T07:55:47.263762+0000","last_peered":"2023-03-03T07:55:47.263762+0000","last_clean":"2023-03-03T07:55:47.263762+0000","last_became_active":"2023-03-03T07:55:46.560534+0000","last_became_peered":"2023-03-03T07:55:46.560534+0000","last_unstale":"2023-03-03T07:55:47.263762+0000","last_undegraded":"2023-03-03T07:55:47.263762+0000","last_fullsized":"2023-03-03T07:55:47.263762+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:45.097261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262107+0000","last_change":"2023-03-03T07:55:47.122002+0000","last_active":"2023-03-03T07:55:47.262107+0000","last_peered":"2023-03-03T07:55:47.262107+0000","last_clean":"2023-03-03T07:55:47.262107+0000","last_became_active":"2023-03-03T07:55:47.121732+0000","last_became_peered":"2023-03-03T07:55:47.121732+0000","last_unstale":"2023-03-03T07:55:47.262107+0000","last_undegraded":"2023-03-03T07:55:47.262107+0000","last_fullsized":"2023-03-03T07:55:47.262107+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:49:48.523585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261411+0000","last_change":"2023-03-03T07:55:46.560027+0000","last_active":"2023-03-03T07:55:47.261411+0000","last_peered":"2023-03-03T07:55:47.261411+0000","last_clean":"2023-03-03T07:55:47.261411+0000","last_became_active":"2023-03-03T07:55:46.559508+0000","last_became_peered":"2023-03-03T07:55:46.559508+0000","last_unstale":"2023-03-03T07:55:47.261411+0000","last_undegraded":"2023-03-03T07:55:47.261411+0000","last_fullsized":"2023-03-03T07:55:47.261411+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:23.109334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261313+0000","last_change":"2023-03-03T07:55:47.121846+0000","last_active":"2023-03-03T07:55:47.261313+0000","last_peered":"2023-03-03T07:55:47.261313+0000","last_clean":"2023-03-03T07:55:47.261313+0000","last_became_active":"2023-03-03T07:55:47.121563+0000","last_became_peered":"2023-03-03T07:55:47.121563+0000","last_unstale":"2023-03-03T07:55:47.261313+0000","last_undegraded":"2023-03-03T07:55:47.261313+0000","last_fullsized":"2023-03-03T07:55:47.261313+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:36:18.587660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"23'4","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259326+0000","last_change":"2023-03-03T07:55:46.567878+0000","last_active":"2023-03-03T07:55:47.259326+0000","last_peered":"2023-03-03T07:55:47.259326+0000","last_clean":"2023-03-03T07:55:47.259326+0000","last_became_active":"2023-03-03T07:55:46.567593+0000","last_became_peered":"2023-03-03T07:55:46.567593+0000","last_unstale":"2023-03-03T07:55:47.259326+0000","last_undegraded":"2023-03-03T07:55:47.259326+0000","last_fullsized":"2023-03-03T07:55:47.259326+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:45.519992+0000","last_clean_scrub_stamp":"2023-03-03T07:55:45.519992+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:08:28.058494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262023+0000","last_change":"2023-03-03T07:55:47.131373+0000","last_active":"2023-03-03T07:55:47.262023+0000","last_peered":"2023-03-03T07:55:47.262023+0000","last_clean":"2023-03-03T07:55:47.262023+0000","last_became_active":"2023-03-03T07:55:47.131037+0000","last_became_peered":"2023-03-03T07:55:47.131037+0000","last_unstale":"2023-03-03T07:55:47.262023+0000","last_undegraded":"2023-03-03T07:55:47.262023+0000","last_fullsized":"2023-03-03T07:55:47.262023+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:52:08.323505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.264990+0000","last_change":"2023-03-03T07:55:47.122842+0000","last_active":"2023-03-03T07:55:47.264990+0000","last_peered":"2023-03-03T07:55:47.264990+0000","last_clean":"2023-03-03T07:55:47.264990+0000","last_became_active":"2023-03-03T07:55:47.122580+0000","last_became_peered":"2023-03-03T07:55:47.122580+0000","last_unstale":"2023-03-03T07:55:47.264990+0000","last_undegraded":"2023-03-03T07:55:47.264990+0000","last_fullsized":"2023-03-03T07:55:47.264990+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:06:24.090758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261297+0000","last_change":"2023-03-03T07:55:47.121284+0000","last_active":"2023-03-03T07:55:47.261297+0000","last_peered":"2023-03-03T07:55:47.261297+0000","last_clean":"2023-03-03T07:55:47.261297+0000","last_became_active":"2023-03-03T07:55:47.121012+0000","last_became_peered":"2023-03-03T07:55:47.121012+0000","last_unstale":"2023-03-03T07:55:47.261297+0000","last_undegraded":"2023-03-03T07:55:47.261297+0000","last_fullsized":"2023-03-03T07:55:47.261297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:15.870956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.262084+0000","last_change":"2023-03-03T07:55:47.138325+0000","last_active":"2023-03-03T07:55:47.262084+0000","last_peered":"2023-03-03T07:55:47.262084+0000","last_clean":"2023-03-03T07:55:47.262084+0000","last_became_active":"2023-03-03T07:55:47.138019+0000","last_became_peered":"2023-03-03T07:55:47.138019+0000","last_unstale":"2023-03-03T07:55:47.262084+0000","last_undegraded":"2023-03-03T07:55:47.262084+0000","last_fullsized":"2023-03-03T07:55:47.262084+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:57.306479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.261371+0000","last_change":"2023-03-03T07:55:47.128792+0000","last_active":"2023-03-03T07:55:47.261371+0000","last_peered":"2023-03-03T07:55:47.261371+0000","last_clean":"2023-03-03T07:55:47.261371+0000","last_became_active":"2023-03-03T07:55:47.127248+0000","last_became_peered":"2023-03-03T07:55:47.127248+0000","last_unstale":"2023-03-03T07:55:47.261371+0000","last_undegraded":"2023-03-03T07:55:47.261371+0000","last_fullsized":"2023-03-03T07:55:47.261371+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:49:25.482566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T07:55:47.259481+0000","last_change":"2023-03-03T07:55:47.127209+0000","last_active":"2023-03-03T07:55:47.259481+0000","last_peered":"2023-03-03T07:55:47.259481+0000","last_clean":"2023-03-03T07:55:47.259481+0000","last_became_active":"2023-03-03T07:55:47.126100+0000","last_became_peered":"2023-03-03T07:55:47.126100+0000","last_unstale":"2023-03-03T07:55:47.259481+0000","last_undegraded":"2023-03-03T07:55:47.259481+0000","last_fullsized":"2023-03-03T07:55:47.259481+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:46.528285+0000","last_clean_scrub_stamp":"2023-03-03T07:55:46.528285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:46.306038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9120,"kb_used_data":856,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362720,"statfs":{"total":96636764160,"available":96627425280,"internally_reserved":0,"allocated":876544,"data_stored":687141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6829999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":93990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8064,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363776,"statfs":{"total":96636764160,"available":96628506624,"internally_reserved":0,"allocated":253952,"data_stored":95009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4970000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.897}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.843999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.613}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.907}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9092,"kb_used_data":828,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362748,"statfs":{"total":96636764160,"available":96627453952,"internally_reserved":0,"allocated":847872,"data_stored":684924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":93990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8068,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363772,"statfs":{"total":96636764160,"available":96628502528,"internally_reserved":0,"allocated":258048,"data_stored":95099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5820000000000007}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3190000000000008}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8629999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8190000000000008}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.625}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8084,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363756,"statfs":{"total":96636764160,"available":96628486144,"internally_reserved":0,"allocated":274432,"data_stored":96613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.1799999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5709999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4529999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8004,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363836,"statfs":{"total":96636764160,"available":96628568064,"internally_reserved":0,"allocated":258048,"data_stored":94626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T07:55:58.978 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-03-03T07:55:58.979 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-03-03T07:55:58.979 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-03-03T07:55:58.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-03-03T07:55:59.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.253+0000 7f502bfff700 1 Processor -- start 2023-03-03T07:55:59.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.254+0000 7f502bfff700 1 -- start start 2023-03-03T07:55:59.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.255+0000 7f502bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f502c146360 0x7f502c160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:59.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.255+0000 7f502bfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f502c0d2590 0x7f502c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:59.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.255+0000 7f502bfff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f502c1614c0 con 0x7f502c0b4ab0 2023-03-03T07:55:59.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.255+0000 7f502bfff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f502c0b5440 con 0x7f502c146360 2023-03-03T07:55:59.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.255+0000 7f502bfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f502c0d10f0 con 0x7f502c0d2590 2023-03-03T07:55:59.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.255+0000 7f502affd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f502c146360 0x7f502c160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:59.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.255+0000 7f502a7fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f502c0b4ab0 0x7f502c0731e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53700/0 (socket says 172.21.15.98:53700) 2023-03-03T07:55:59.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.255+0000 7f502a7fc700 1 -- 172.21.15.98:0/1317843685 learned_addr learned my addr 172.21.15.98:0/1317843685 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:55:59.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.256+0000 7f502b7fe700 1 --2- 172.21.15.98:0/1317843685 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f502c0d2590 0x7f502c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:59.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.256+0000 7f5029ffb700 1 -- 172.21.15.98:0/1317843685 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329138980 0 0) 0x7f5020001110 con 0x7f502c0b4ab0 2023-03-03T07:55:59.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.256+0000 7f502affd700 1 -- 172.21.15.98:0/1317843685 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f502c0d2590 msgr2=0x7f502c0b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:59.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.256+0000 7f502affd700 1 --2- 172.21.15.98:0/1317843685 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f502c0d2590 0x7f502c0b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.256+0000 7f502affd700 1 -- 172.21.15.98:0/1317843685 >> v1:172.21.15.98:6789/0 conn(0x7f502c0b4ab0 legacy=0x7f502c0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:59.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.256+0000 7f502affd700 1 -- 172.21.15.98:0/1317843685 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f502c0cdf20 con 0x7f502c146360 2023-03-03T07:55:59.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.256+0000 7f502affd700 1 --2- 172.21.15.98:0/1317843685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c146360 0x7f502c160f80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f501c009840 tx=0x7f501c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c355b4895232d1a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:59.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.256+0000 7f5029ffb700 1 -- 172.21.15.98:0/1317843685 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c011830 con 0x7f502c146360 2023-03-03T07:55:59.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.257+0000 7f5029ffb700 1 -- 172.21.15.98:0/1317843685 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501c011990 con 0x7f502c146360 2023-03-03T07:55:59.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.257+0000 7f5029ffb700 1 -- 172.21.15.98:0/1317843685 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c011d20 con 0x7f502c146360 2023-03-03T07:55:59.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.257+0000 7f502bfff700 1 -- 172.21.15.98:0/1317843685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c146360 msgr2=0x7f502c160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:59.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.257+0000 7f502bfff700 1 --2- 172.21.15.98:0/1317843685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c146360 0x7f502c160f80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f501c009840 tx=0x7f501c009b50 comp rx=0 tx=0).stop 2023-03-03T07:55:59.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.257+0000 7f502bfff700 1 -- 172.21.15.98:0/1317843685 shutdown_connections 2023-03-03T07:55:59.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.257+0000 7f502bfff700 1 --2- 172.21.15.98:0/1317843685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c146360 0x7f502c160f80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.257+0000 7f502bfff700 1 --2- 172.21.15.98:0/1317843685 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f502c0d2590 0x7f502c0b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.257+0000 7f502bfff700 1 -- 172.21.15.98:0/1317843685 >> 172.21.15.98:0/1317843685 conn(0x7f502c06b8e0 msgr2=0x7f502c0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:59.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.257+0000 7f502bfff700 1 -- 172.21.15.98:0/1317843685 shutdown_connections 2023-03-03T07:55:59.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.257+0000 7f502bfff700 1 -- 172.21.15.98:0/1317843685 wait complete. 2023-03-03T07:55:59.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.259+0000 7f502bfff700 1 Processor -- start 2023-03-03T07:55:59.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.260+0000 7f502bfff700 1 -- start start 2023-03-03T07:55:59.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.261+0000 7f502bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c0b4ab0 0x7f502c1f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:59.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.261+0000 7f502bfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f502c0d2590 0x7f502c152f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:59.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.261+0000 7f502bfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f502c146360 0x7f502c0c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:59.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.261+0000 7f502bfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502c151eb0 con 0x7f502c0b4ab0 2023-03-03T07:55:59.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.261+0000 7f502bfff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502c0b73b0 con 0x7f502c0d2590 2023-03-03T07:55:59.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.261+0000 7f502bfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f502c0b7690 con 0x7f502c146360 2023-03-03T07:55:59.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.261+0000 7f502affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c0b4ab0 0x7f502c1f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:59.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.261+0000 7f502affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c0b4ab0 0x7f502c1f1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41862/0 (socket says 172.21.15.98:41862) 2023-03-03T07:55:59.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.261+0000 7f502affd700 1 -- 172.21.15.98:0/1494442961 learned_addr learned my addr 172.21.15.98:0/1494442961 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:59.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.261+0000 7f502a7fc700 1 --2- 172.21.15.98:0/1494442961 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f502c0d2590 0x7f502c152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:59.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.261+0000 7f502b7fe700 1 --2- 172.21.15.98:0/1494442961 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f502c146360 0x7f502c0c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:59.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.261+0000 7f502affd700 1 -- 172.21.15.98:0/1494442961 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f502c146360 msgr2=0x7f502c0c6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:59.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.261+0000 7f502affd700 1 --2- 172.21.15.98:0/1494442961 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f502c146360 0x7f502c0c6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.261+0000 7f502affd700 1 -- 172.21.15.98:0/1494442961 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f502c0d2590 msgr2=0x7f502c152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:59.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.261+0000 7f502affd700 1 --2- 172.21.15.98:0/1494442961 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f502c0d2590 0x7f502c152f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.262+0000 7f502affd700 1 -- 172.21.15.98:0/1494442961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501c0094a0 con 0x7f502c0b4ab0 2023-03-03T07:55:59.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.262+0000 7f502affd700 1 --2- 172.21.15.98:0/1494442961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c0b4ab0 0x7f502c1f1c80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f501c009b90 tx=0x7f501c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:59.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.262+0000 7f50137fe700 1 -- 172.21.15.98:0/1494442961 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c0036a0 con 0x7f502c0b4ab0 2023-03-03T07:55:59.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.262+0000 7f50137fe700 1 -- 172.21.15.98:0/1494442961 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501c01e070 con 0x7f502c0b4ab0 2023-03-03T07:55:59.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.262+0000 7f50137fe700 1 -- 172.21.15.98:0/1494442961 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c003a60 con 0x7f502c0b4ab0 2023-03-03T07:55:59.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.262+0000 7f502bfff700 1 -- 172.21.15.98:0/1494442961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f502c0cd960 con 0x7f502c0b4ab0 2023-03-03T07:55:59.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.262+0000 7f502bfff700 1 -- 172.21.15.98:0/1494442961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f502c0cd680 con 0x7f502c0b4ab0 2023-03-03T07:55:59.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.263+0000 7f502bfff700 1 -- 172.21.15.98:0/1494442961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f502c0cdc40 con 0x7f502c0b4ab0 2023-03-03T07:55:59.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.264+0000 7f50137fe700 1 -- 172.21.15.98:0/1494442961 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f501c00f050 con 0x7f502c0b4ab0 2023-03-03T07:55:59.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.266+0000 7f50137fe700 1 --2- 172.21.15.98:0/1494442961 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50180682f0 0x7f501806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:59.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.266+0000 7f50137fe700 1 -- 172.21.15.98:0/1494442961 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f501c086c30 con 0x7f502c0b4ab0 2023-03-03T07:55:59.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.266+0000 7f502a7fc700 1 --2- 172.21.15.98:0/1494442961 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50180682f0 0x7f501806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:59.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.266+0000 7f502a7fc700 1 --2- 172.21.15.98:0/1494442961 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50180682f0 0x7f501806a7a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f502c145ca0 tx=0x7f5020007e70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:59.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.269+0000 7f50137fe700 1 -- 172.21.15.98:0/1494442961 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f501c0533a0 con 0x7f502c0b4ab0 2023-03-03T07:55:59.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.505+0000 7f502bfff700 1 -- 172.21.15.98:0/1494442961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f502c068300 con 0x7f502c0b4ab0 2023-03-03T07:55:59.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.505+0000 7f50137fe700 1 -- 172.21.15.98:0/1494442961 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f501c020040 con 0x7f502c0b4ab0 2023-03-03T07:55:59.507 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:59.507 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-03-03T07:55:59.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.508+0000 7f502bfff700 1 -- 172.21.15.98:0/1494442961 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50180682f0 msgr2=0x7f501806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:59.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.508+0000 7f502bfff700 1 --2- 172.21.15.98:0/1494442961 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50180682f0 0x7f501806a7a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f502c145ca0 tx=0x7f5020007e70 comp rx=0 tx=0).stop 2023-03-03T07:55:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.508+0000 7f502bfff700 1 -- 172.21.15.98:0/1494442961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c0b4ab0 msgr2=0x7f502c1f1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.508+0000 7f502bfff700 1 --2- 172.21.15.98:0/1494442961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c0b4ab0 0x7f502c1f1c80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f501c009b90 tx=0x7f501c004750 comp rx=0 tx=0).stop 2023-03-03T07:55:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.508+0000 7f502bfff700 1 -- 172.21.15.98:0/1494442961 shutdown_connections 2023-03-03T07:55:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.508+0000 7f502bfff700 1 --2- 172.21.15.98:0/1494442961 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50180682f0 0x7f501806a7a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.508+0000 7f502bfff700 1 --2- 172.21.15.98:0/1494442961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c0b4ab0 0x7f502c1f1c80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.508+0000 7f502bfff700 1 --2- 172.21.15.98:0/1494442961 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f502c146360 0x7f502c0c6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.508+0000 7f502bfff700 1 --2- 172.21.15.98:0/1494442961 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f502c0d2590 0x7f502c152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.508+0000 7f502bfff700 1 -- 172.21.15.98:0/1494442961 >> 172.21.15.98:0/1494442961 conn(0x7f502c06b8e0 msgr2=0x7f502c0d1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:59.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.508+0000 7f502bfff700 1 -- 172.21.15.98:0/1494442961 shutdown_connections 2023-03-03T07:55:59.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.508+0000 7f502bfff700 1 -- 172.21.15.98:0/1494442961 wait complete. 2023-03-03T07:55:59.536 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-03-03T07:55:59.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:55:59.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.781+0000 7f69c65ff700 1 Processor -- start 2023-03-03T07:55:59.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.782+0000 7f69c65ff700 1 -- start start 2023-03-03T07:55:59.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.783+0000 7f69c65ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c015cc00 0x7f69c00cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:59.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.783+0000 7f69c65ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f69c00d27c0 0x7f69c00d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:59.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.783+0000 7f69c65ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f69c00b60d0 0x7f69c00b6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:59.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.783+0000 7f69c65ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c00ce040 con 0x7f69c00b60d0 2023-03-03T07:55:59.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.783+0000 7f69c65ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c0073be0 con 0x7f69c00d27c0 2023-03-03T07:55:59.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.783+0000 7f69c65ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c00cde10 con 0x7f69c015cc00 2023-03-03T07:55:59.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.784+0000 7f69c4b9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f69c00b60d0 0x7f69c00b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:59.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.784+0000 7f69c4b9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f69c00b60d0 0x7f69c00b6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41864/0 (socket says 172.21.15.98:41864) 2023-03-03T07:55:59.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.784+0000 7f69c4b9c700 1 -- 172.21.15.98:0/855097252 learned_addr learned my addr 172.21.15.98:0/855097252 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:59.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.784+0000 7f69bffff700 1 --2- 172.21.15.98:0/855097252 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c015cc00 0x7f69c00cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:59.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.784+0000 7f69bf7fe700 1 --2- 172.21.15.98:0/855097252 >> v2:172.21.15.100:3300/0 conn(0x7f69c00d27c0 0x7f69c00d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:59.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.784+0000 7f69c4b9c700 1 -- 172.21.15.98:0/855097252 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c015cc00 msgr2=0x7f69c00cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:59.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.784+0000 7f69c4b9c700 1 --2- 172.21.15.98:0/855097252 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c015cc00 0x7f69c00cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.784+0000 7f69c4b9c700 1 -- 172.21.15.98:0/855097252 >> v2:172.21.15.100:3300/0 conn(0x7f69c00d27c0 msgr2=0x7f69c00d2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:55:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.784+0000 7f69c4b9c700 1 --2- 172.21.15.98:0/855097252 >> v2:172.21.15.100:3300/0 conn(0x7f69c00d27c0 0x7f69c00d2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.784+0000 7f69c4b9c700 1 -- 172.21.15.98:0/855097252 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c00d1a00 con 0x7f69c00b60d0 2023-03-03T07:55:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.784+0000 7f69c4b9c700 1 --2- 172.21.15.98:0/855097252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c00b60d0 0x7f69c00b6550 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f69c00b5d80 tx=0x7f69b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82eceb2894d4050c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.785+0000 7f69beffd700 1 -- 172.21.15.98:0/855097252 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b800e070 con 0x7f69c00b60d0 2023-03-03T07:55:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.785+0000 7f69beffd700 1 -- 172.21.15.98:0/855097252 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b800b9c0 con 0x7f69c00b60d0 2023-03-03T07:55:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.785+0000 7f69beffd700 1 -- 172.21.15.98:0/855097252 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b800bd70 con 0x7f69c00b60d0 2023-03-03T07:55:59.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.785+0000 7f69c65ff700 1 -- 172.21.15.98:0/855097252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c00b60d0 msgr2=0x7f69c00b6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:59.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.785+0000 7f69c65ff700 1 --2- 172.21.15.98:0/855097252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c00b60d0 0x7f69c00b6550 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f69c00b5d80 tx=0x7f69b800b330 comp rx=0 tx=0).stop 2023-03-03T07:55:59.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.785+0000 7f69c65ff700 1 -- 172.21.15.98:0/855097252 shutdown_connections 2023-03-03T07:55:59.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.785+0000 7f69c65ff700 1 --2- 172.21.15.98:0/855097252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c00b60d0 0x7f69c00b6550 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.785+0000 7f69c65ff700 1 --2- 172.21.15.98:0/855097252 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c015cc00 0x7f69c00cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.785+0000 7f69c65ff700 1 --2- 172.21.15.98:0/855097252 >> v2:172.21.15.100:3300/0 conn(0x7f69c00d27c0 0x7f69c00d2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.785+0000 7f69c65ff700 1 -- 172.21.15.98:0/855097252 >> 172.21.15.98:0/855097252 conn(0x7f69c006aee0 msgr2=0x7f69c006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:55:59.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.785+0000 7f69c65ff700 1 -- 172.21.15.98:0/855097252 shutdown_connections 2023-03-03T07:55:59.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.786+0000 7f69c65ff700 1 -- 172.21.15.98:0/855097252 wait complete. 2023-03-03T07:55:59.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.788+0000 7f69c65ff700 1 Processor -- start 2023-03-03T07:55:59.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.789+0000 7f69c65ff700 1 -- start start 2023-03-03T07:55:59.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.790+0000 7f69c65ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c00b60d0 0x7f69c015f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:59.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.790+0000 7f69c65ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69c00d27c0 0x7f69c00c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:59.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.790+0000 7f69c65ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c015cc00 0x7f69c015d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:59.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.790+0000 7f69c65ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c00c9250 con 0x7f69c00b60d0 2023-03-03T07:55:59.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.790+0000 7f69c65ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c00c8640 con 0x7f69c00d27c0 2023-03-03T07:55:59.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.790+0000 7f69c65ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c00c8920 con 0x7f69c015cc00 2023-03-03T07:55:59.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.790+0000 7f69bffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c00b60d0 0x7f69c015f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:59.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.790+0000 7f69bffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c00b60d0 0x7f69c015f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41876/0 (socket says 172.21.15.98:41876) 2023-03-03T07:55:59.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.790+0000 7f69bffff700 1 -- 172.21.15.98:0/1777087627 learned_addr learned my addr 172.21.15.98:0/1777087627 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:55:59.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.791+0000 7f69bf7fe700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69c00d27c0 0x7f69c00c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:59.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.791+0000 7f69c4b9c700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c015cc00 0x7f69c015d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:59.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.791+0000 7f69bffff700 1 -- 172.21.15.98:0/1777087627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c015cc00 msgr2=0x7f69c015d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:59.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.791+0000 7f69bffff700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c015cc00 0x7f69c015d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.791+0000 7f69bffff700 1 -- 172.21.15.98:0/1777087627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69c00d27c0 msgr2=0x7f69c00c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:59.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.791+0000 7f69bffff700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69c00d27c0 0x7f69c00c7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:55:59.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.791+0000 7f69bffff700 1 -- 172.21.15.98:0/1777087627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b8009b30 con 0x7f69c00b60d0 2023-03-03T07:55:59.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.791+0000 7f69c4b9c700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c015cc00 0x7f69c015d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:59.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.791+0000 7f69bf7fe700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69c00d27c0 0x7f69c00c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:55:59.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.791+0000 7f69bffff700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c00b60d0 0x7f69c015f4f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f69b000e810 tx=0x7f69b000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:59.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.791+0000 7f69bcff9700 1 -- 172.21.15.98:0/1777087627 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b0013610 con 0x7f69c00b60d0 2023-03-03T07:55:59.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.791+0000 7f69bcff9700 1 -- 172.21.15.98:0/1777087627 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b0013770 con 0x7f69c00b60d0 2023-03-03T07:55:59.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.791+0000 7f69c65ff700 1 -- 172.21.15.98:0/1777087627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c00d1fc0 con 0x7f69c00b60d0 2023-03-03T07:55:59.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.791+0000 7f69bcff9700 1 -- 172.21.15.98:0/1777087627 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b0013b20 con 0x7f69c00b60d0 2023-03-03T07:55:59.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.792+0000 7f69c65ff700 1 -- 172.21.15.98:0/1777087627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c00d1ce0 con 0x7f69c00b60d0 2023-03-03T07:55:59.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.793+0000 7f69c65ff700 1 -- 172.21.15.98:0/1777087627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c00d22a0 con 0x7f69c00b60d0 2023-03-03T07:55:59.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.794+0000 7f69bcff9700 1 -- 172.21.15.98:0/1777087627 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f69b0023070 con 0x7f69c00b60d0 2023-03-03T07:55:59.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.795+0000 7f69bcff9700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f69ac0682f0 0x7f69ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:55:59.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.796+0000 7f69bcff9700 1 -- 172.21.15.98:0/1777087627 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f69b0085b00 con 0x7f69c00b60d0 2023-03-03T07:55:59.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.796+0000 7f69bf7fe700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f69ac0682f0 0x7f69ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:55:59.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.796+0000 7f69bf7fe700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f69ac0682f0 0x7f69ac06a7a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f69c01f6920 tx=0x7f69b4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:55:59.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.799+0000 7f69bcff9700 1 -- 172.21.15.98:0/1777087627 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f69b0087e60 con 0x7f69c00b60d0 2023-03-03T07:55:59.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.994+0000 7f69c65ff700 1 -- 172.21.15.98:0/1777087627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69c00729b0 con 0x7f69c00b60d0 2023-03-03T07:55:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.995+0000 7f69bcff9700 1 -- 172.21.15.98:0/1777087627 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17907 (secure 0 0 0) 0x7f69b0055c20 con 0x7f69c00b60d0 2023-03-03T07:55:59.997 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:55:59.997 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:55:48.099844+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:55:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.997+0000 7f69c65ff700 1 -- 172.21.15.98:0/1777087627 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f69ac0682f0 msgr2=0x7f69ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.997+0000 7f69c65ff700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f69ac0682f0 0x7f69ac06a7a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f69c01f6920 tx=0x7f69b4008040 comp rx=0 tx=0).stop 2023-03-03T07:55:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.997+0000 7f69c65ff700 1 -- 172.21.15.98:0/1777087627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c00b60d0 msgr2=0x7f69c015f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:55:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.997+0000 7f69c65ff700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c00b60d0 0x7f69c015f4f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f69b000e810 tx=0x7f69b000ebd0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.998+0000 7f69c65ff700 1 -- 172.21.15.98:0/1777087627 shutdown_connections 2023-03-03T07:56:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.998+0000 7f69c65ff700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f69ac0682f0 0x7f69ac06a7a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.998+0000 7f69c65ff700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c00b60d0 0x7f69c015f4f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.998+0000 7f69c65ff700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c015cc00 0x7f69c015d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.998+0000 7f69c65ff700 1 --2- 172.21.15.98:0/1777087627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69c00d27c0 0x7f69c00c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.998+0000 7f69c65ff700 1 -- 172.21.15.98:0/1777087627 >> 172.21.15.98:0/1777087627 conn(0x7f69c006aee0 msgr2=0x7f69c00d0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.998+0000 7f69c65ff700 1 -- 172.21.15.98:0/1777087627 shutdown_connections 2023-03-03T07:56:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:55:59.998+0000 7f69c65ff700 1 -- 172.21.15.98:0/1777087627 wait complete. 2023-03-03T07:56:00.027 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:55:45.157666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:55:46.092091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:56:00.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:56:00.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.275+0000 7fe9a57ff700 1 Processor -- start 2023-03-03T07:56:00.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.275+0000 7fe9a57ff700 1 -- start start 2023-03-03T07:56:00.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.277+0000 7fe9a57ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9a00b8d10 0x7fe9a00cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:00.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.277+0000 7fe9a57ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe9a00b5310 0x7fe9a00d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.277+0000 7fe9a57ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00cd7d0 con 0x7fe9a00b26d0 2023-03-03T07:56:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.277+0000 7fe9a57ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9a00b90e0 con 0x7fe9a00b5310 2023-03-03T07:56:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.277+0000 7fe9a57ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a00cda20 con 0x7fe9a00b8d10 2023-03-03T07:56:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.277+0000 7fe99effd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9a00b8d10 0x7fe9a00cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.277+0000 7fe99effd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9a00b8d10 0x7fe9a00cffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:32990/0 (socket says 172.21.15.98:32990) 2023-03-03T07:56:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.277+0000 7fe99effd700 1 -- 172.21.15.98:0/1622182660 learned_addr learned my addr 172.21.15.98:0/1622182660 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.277+0000 7fe99f7fe700 1 --2- 172.21.15.98:0/1622182660 >> v2:172.21.15.100:3300/0 conn(0x7fe9a00b5310 0x7fe9a00d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.278+0000 7fe99effd700 1 -- 172.21.15.98:0/1622182660 >> v2:172.21.15.100:3300/0 conn(0x7fe9a00b5310 msgr2=0x7fe9a00d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.278+0000 7fe99effd700 1 --2- 172.21.15.98:0/1622182660 >> v2:172.21.15.100:3300/0 conn(0x7fe9a00b5310 0x7fe9a00d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.278+0000 7fe99dffb700 1 -- 172.21.15.98:0/1622182660 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822208412 0 0) 0x7fe9a00cd7d0 con 0x7fe9a00b26d0 2023-03-03T07:56:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.278+0000 7fe99effd700 1 -- 172.21.15.98:0/1622182660 >> v1:172.21.15.100:6789/0 conn(0x7fe9a00b26d0 legacy=0x7fe9a00b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.278+0000 7fe99effd700 1 -- 172.21.15.98:0/1622182660 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a00cdd00 con 0x7fe9a00b8d10 2023-03-03T07:56:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.278+0000 7fe99effd700 1 --2- 172.21.15.98:0/1622182660 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9a00b8d10 0x7fe9a00cffc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe9900098f0 tx=0x7fe990009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e15c453b5c3d7ec server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.278+0000 7fe99dffb700 1 -- 172.21.15.98:0/1622182660 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99000b8a0 con 0x7fe9a00b8d10 2023-03-03T07:56:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.278+0000 7fe99dffb700 1 -- 172.21.15.98:0/1622182660 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99000ba00 con 0x7fe9a00b8d10 2023-03-03T07:56:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.278+0000 7fe99dffb700 1 -- 172.21.15.98:0/1622182660 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99000bd90 con 0x7fe9a00b8d10 2023-03-03T07:56:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.279+0000 7fe9a57ff700 1 -- 172.21.15.98:0/1622182660 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9a00b8d10 msgr2=0x7fe9a00cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.279+0000 7fe9a57ff700 1 --2- 172.21.15.98:0/1622182660 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9a00b8d10 0x7fe9a00cffc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe9900098f0 tx=0x7fe990009c00 comp rx=0 tx=0).stop 2023-03-03T07:56:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.279+0000 7fe9a57ff700 1 -- 172.21.15.98:0/1622182660 shutdown_connections 2023-03-03T07:56:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.279+0000 7fe9a57ff700 1 --2- 172.21.15.98:0/1622182660 >> v2:172.21.15.100:3300/0 conn(0x7fe9a00b5310 0x7fe9a00d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.279+0000 7fe9a57ff700 1 --2- 172.21.15.98:0/1622182660 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9a00b8d10 0x7fe9a00cffc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.279+0000 7fe9a57ff700 1 -- 172.21.15.98:0/1622182660 >> 172.21.15.98:0/1622182660 conn(0x7fe9a0073e70 msgr2=0x7fe9a006ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.279+0000 7fe9a57ff700 1 -- 172.21.15.98:0/1622182660 shutdown_connections 2023-03-03T07:56:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.279+0000 7fe9a57ff700 1 -- 172.21.15.98:0/1622182660 wait complete. 2023-03-03T07:56:00.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.282+0000 7fe9a57ff700 1 Processor -- start 2023-03-03T07:56:00.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.282+0000 7fe9a57ff700 1 -- start start 2023-03-03T07:56:00.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.283+0000 7fe9a57ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe9a00b26d0 0x7fe9a014d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:00.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.283+0000 7fe9a57ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9a00b5310 0x7fe9a00ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:00.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.284+0000 7fe9a57ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9a00b8d10 0x7fe9a00a9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:00.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.284+0000 7fe9a57ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a00ab530 con 0x7fe9a00b8d10 2023-03-03T07:56:00.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.284+0000 7fe9a57ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a00cc990 con 0x7fe9a00b26d0 2023-03-03T07:56:00.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.284+0000 7fe9a57ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a014db00 con 0x7fe9a00b5310 2023-03-03T07:56:00.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.284+0000 7fe99f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9a00b8d10 0x7fe9a00a9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:00.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.284+0000 7fe99f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9a00b8d10 0x7fe9a00a9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41888/0 (socket says 172.21.15.98:41888) 2023-03-03T07:56:00.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.284+0000 7fe99f7fe700 1 -- 172.21.15.98:0/427452494 learned_addr learned my addr 172.21.15.98:0/427452494 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:00.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.284+0000 7fe99e7fc700 1 --2- 172.21.15.98:0/427452494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9a00b5310 0x7fe9a00ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:00.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.284+0000 7fe99effd700 1 --2- 172.21.15.98:0/427452494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe9a00b26d0 0x7fe9a014d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:00.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.284+0000 7fe99f7fe700 1 -- 172.21.15.98:0/427452494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9a00b5310 msgr2=0x7fe9a00ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:00.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.284+0000 7fe99f7fe700 1 --2- 172.21.15.98:0/427452494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9a00b5310 0x7fe9a00ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.284+0000 7fe99f7fe700 1 -- 172.21.15.98:0/427452494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe9a00b26d0 msgr2=0x7fe9a014d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:00.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.284+0000 7fe99f7fe700 1 --2- 172.21.15.98:0/427452494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe9a00b26d0 0x7fe9a014d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.284+0000 7fe99f7fe700 1 -- 172.21.15.98:0/427452494 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe990009550 con 0x7fe9a00b8d10 2023-03-03T07:56:00.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.285+0000 7fe99f7fe700 1 --2- 172.21.15.98:0/427452494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9a00b8d10 0x7fe9a00a9b30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe998007e90 tx=0x7fe99800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:00.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.285+0000 7fe9877fe700 1 -- 172.21.15.98:0/427452494 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99800ede0 con 0x7fe9a00b8d10 2023-03-03T07:56:00.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.285+0000 7fe9877fe700 1 -- 172.21.15.98:0/427452494 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe998014420 con 0x7fe9a00b8d10 2023-03-03T07:56:00.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.285+0000 7fe9a57ff700 1 -- 172.21.15.98:0/427452494 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a00cdfe0 con 0x7fe9a00b8d10 2023-03-03T07:56:00.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.285+0000 7fe9877fe700 1 -- 172.21.15.98:0/427452494 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9980147d0 con 0x7fe9a00b8d10 2023-03-03T07:56:00.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.285+0000 7fe9a57ff700 1 -- 172.21.15.98:0/427452494 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a00d0e80 con 0x7fe9a00b8d10 2023-03-03T07:56:00.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.286+0000 7fe9a57ff700 1 -- 172.21.15.98:0/427452494 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a00d0500 con 0x7fe9a00b8d10 2023-03-03T07:56:00.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.287+0000 7fe9877fe700 1 -- 172.21.15.98:0/427452494 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe998012050 con 0x7fe9a00b8d10 2023-03-03T07:56:00.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.288+0000 7fe9877fe700 1 --2- 172.21.15.98:0/427452494 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe98c068320 0x7fe98c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.289+0000 7fe9877fe700 1 -- 172.21.15.98:0/427452494 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe9980863a0 con 0x7fe9a00b8d10 2023-03-03T07:56:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.289+0000 7fe99effd700 1 --2- 172.21.15.98:0/427452494 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe98c068320 0x7fe98c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.290+0000 7fe99effd700 1 --2- 172.21.15.98:0/427452494 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe98c068320 0x7fe98c06a7d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe990000c00 tx=0x7fe990008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.293+0000 7fe9877fe700 1 -- 172.21.15.98:0/427452494 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe998052a80 con 0x7fe9a00b8d10 2023-03-03T07:56:00.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.486+0000 7fe9a57ff700 1 -- 172.21.15.98:0/427452494 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe9a00ccad0 con 0x7fe9a00b8d10 2023-03-03T07:56:00.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.486+0000 7fe9877fe700 1 -- 172.21.15.98:0/427452494 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe9a00ccad0 con 0x7fe9a00b8d10 2023-03-03T07:56:00.488 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:56:00.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.489+0000 7fe9a57ff700 1 -- 172.21.15.98:0/427452494 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe98c068320 msgr2=0x7fe98c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:00.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.489+0000 7fe9a57ff700 1 --2- 172.21.15.98:0/427452494 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe98c068320 0x7fe98c06a7d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe990000c00 tx=0x7fe990008030 comp rx=0 tx=0).stop 2023-03-03T07:56:00.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.489+0000 7fe9a57ff700 1 -- 172.21.15.98:0/427452494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9a00b8d10 msgr2=0x7fe9a00a9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:00.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.489+0000 7fe9a57ff700 1 --2- 172.21.15.98:0/427452494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9a00b8d10 0x7fe9a00a9b30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe998007e90 tx=0x7fe99800e3c0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.489+0000 7fe9a57ff700 1 -- 172.21.15.98:0/427452494 shutdown_connections 2023-03-03T07:56:00.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.489+0000 7fe9a57ff700 1 --2- 172.21.15.98:0/427452494 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe98c068320 0x7fe98c06a7d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.489+0000 7fe9a57ff700 1 --2- 172.21.15.98:0/427452494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9a00b8d10 0x7fe9a00a9b30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.489+0000 7fe9a57ff700 1 --2- 172.21.15.98:0/427452494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe9a00b26d0 0x7fe9a014d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.489+0000 7fe9a57ff700 1 --2- 172.21.15.98:0/427452494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9a00b5310 0x7fe9a00ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.489+0000 7fe9a57ff700 1 -- 172.21.15.98:0/427452494 >> 172.21.15.98:0/427452494 conn(0x7fe9a0073e70 msgr2=0x7fe9a00b6d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:00.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.489+0000 7fe9a57ff700 1 -- 172.21.15.98:0/427452494 shutdown_connections 2023-03-03T07:56:00.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.489+0000 7fe9a57ff700 1 -- 172.21.15.98:0/427452494 wait complete. 2023-03-03T07:56:00.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:56:00.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.765+0000 7ffbc299d700 1 Processor -- start 2023-03-03T07:56:00.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.765+0000 7ffbc299d700 1 -- start start 2023-03-03T07:56:00.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.766+0000 7ffbc299d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ffbbc0cf570 0x7ffbbc15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:00.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.767+0000 7ffbc299d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc0cd780 con 0x7ffbbc15db30 2023-03-03T07:56:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.767+0000 7ffbc299d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc0cda60 con 0x7ffbbc1609b0 2023-03-03T07:56:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.767+0000 7ffbc299d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbbc0cebb0 con 0x7ffbbc0cf570 2023-03-03T07:56:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.767+0000 7ffbc219c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ffbbc15db30 0x7ffbbc0b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53716/0 (socket says 172.21.15.98:53716) 2023-03-03T07:56:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.767+0000 7ffbc219c700 1 -- 172.21.15.98:0/1572880698 learned_addr learned my addr 172.21.15.98:0/1572880698 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.767+0000 7ffbc119a700 1 --2- 172.21.15.98:0/1572880698 >> v2:172.21.15.100:3300/0 conn(0x7ffbbc0cf570 0x7ffbbc15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.767+0000 7ffbc0999700 1 -- 172.21.15.98:0/1572880698 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899140454 0 0) 0x7ffbbc0cd780 con 0x7ffbbc15db30 2023-03-03T07:56:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.767+0000 7ffbc0999700 1 -- 172.21.15.98:0/1572880698 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba80035b0 con 0x7ffbbc15db30 2023-03-03T07:56:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.768+0000 7ffbc0999700 1 -- 172.21.15.98:0/1572880698 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527334935 0 0) 0x7ffbbc0cda60 con 0x7ffbbc1609b0 2023-03-03T07:56:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.768+0000 7ffbc0999700 1 -- 172.21.15.98:0/1572880698 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbbc0cd780 con 0x7ffbbc1609b0 2023-03-03T07:56:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.768+0000 7ffbc119a700 1 -- 172.21.15.98:0/1572880698 >> v1:172.21.15.100:6789/0 conn(0x7ffbbc1609b0 legacy=0x7ffbbc160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.768+0000 7ffbc119a700 1 -- 172.21.15.98:0/1572880698 >> v1:172.21.15.98:6789/0 conn(0x7ffbbc15db30 legacy=0x7ffbbc0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.768+0000 7ffbc119a700 1 -- 172.21.15.98:0/1572880698 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbbc0cdd40 con 0x7ffbbc0cf570 2023-03-03T07:56:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.768+0000 7ffbc119a700 1 --2- 172.21.15.98:0/1572880698 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffbbc0cf570 0x7ffbbc15d5f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffbac009840 tx=0x7ffbac009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd4cb655197d96a5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:00.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.768+0000 7ffbc0999700 1 -- 172.21.15.98:0/1572880698 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac011830 con 0x7ffbbc0cf570 2023-03-03T07:56:00.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.768+0000 7ffbc0999700 1 -- 172.21.15.98:0/1572880698 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbac011990 con 0x7ffbbc0cf570 2023-03-03T07:56:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.768+0000 7ffbc0999700 1 -- 172.21.15.98:0/1572880698 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac011d20 con 0x7ffbbc0cf570 2023-03-03T07:56:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.769+0000 7ffbc299d700 1 -- 172.21.15.98:0/1572880698 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffbbc0cf570 msgr2=0x7ffbbc15d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.769+0000 7ffbc299d700 1 --2- 172.21.15.98:0/1572880698 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffbbc0cf570 0x7ffbbc15d5f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffbac009840 tx=0x7ffbac009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.769+0000 7ffbc299d700 1 -- 172.21.15.98:0/1572880698 shutdown_connections 2023-03-03T07:56:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.769+0000 7ffbc299d700 1 --2- 172.21.15.98:0/1572880698 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffbbc0cf570 0x7ffbbc15d5f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.769+0000 7ffbc299d700 1 -- 172.21.15.98:0/1572880698 >> 172.21.15.98:0/1572880698 conn(0x7ffbbc06b880 msgr2=0x7ffbbc0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.769+0000 7ffbc299d700 1 -- 172.21.15.98:0/1572880698 shutdown_connections 2023-03-03T07:56:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.769+0000 7ffbc299d700 1 -- 172.21.15.98:0/1572880698 wait complete. 2023-03-03T07:56:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.771+0000 7ffbc299d700 1 Processor -- start 2023-03-03T07:56:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.771+0000 7ffbc299d700 1 -- start start 2023-03-03T07:56:00.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.772+0000 7ffbc299d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbbc0cf570 0x7ffbbc151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:00.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.772+0000 7ffbc299d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffbbc15db30 0x7ffbbc1f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:00.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.772+0000 7ffbc299d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffbbc1609b0 0x7ffbbc0d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:00.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.772+0000 7ffbc299d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc1f34f0 con 0x7ffbbc0cf570 2023-03-03T07:56:00.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.772+0000 7ffbc299d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc1519c0 con 0x7ffbbc1609b0 2023-03-03T07:56:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.772+0000 7ffbc299d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc1f32f0 con 0x7ffbbc15db30 2023-03-03T07:56:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.773+0000 7ffbc199b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbbc0cf570 0x7ffbbc151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.773+0000 7ffbc199b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbbc0cf570 0x7ffbbc151480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41898/0 (socket says 172.21.15.98:41898) 2023-03-03T07:56:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.773+0000 7ffbc199b700 1 -- 172.21.15.98:0/1985804041 learned_addr learned my addr 172.21.15.98:0/1985804041 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.773+0000 7ffbc219c700 1 --2- 172.21.15.98:0/1985804041 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffbbc1609b0 0x7ffbbc0d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.773+0000 7ffbc119a700 1 --2- 172.21.15.98:0/1985804041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffbbc15db30 0x7ffbbc1f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.773+0000 7ffbc199b700 1 -- 172.21.15.98:0/1985804041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffbbc15db30 msgr2=0x7ffbbc1f2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.773+0000 7ffbc199b700 1 --2- 172.21.15.98:0/1985804041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffbbc15db30 0x7ffbbc1f2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.773+0000 7ffbc199b700 1 -- 172.21.15.98:0/1985804041 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffbbc1609b0 msgr2=0x7ffbbc0d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.773+0000 7ffbc199b700 1 --2- 172.21.15.98:0/1985804041 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffbbc1609b0 0x7ffbbc0d1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.773+0000 7ffbc199b700 1 -- 172.21.15.98:0/1985804041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbac0094a0 con 0x7ffbbc0cf570 2023-03-03T07:56:00.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.773+0000 7ffbc199b700 1 --2- 172.21.15.98:0/1985804041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbbc0cf570 0x7ffbbc151480 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffbb800c500 tx=0x7ffbb800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:00.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.774+0000 7ffbb27fc700 1 -- 172.21.15.98:0/1985804041 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80052d0 con 0x7ffbbc0cf570 2023-03-03T07:56:00.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.774+0000 7ffbb27fc700 1 -- 172.21.15.98:0/1985804041 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb8005450 con 0x7ffbbc0cf570 2023-03-03T07:56:00.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.774+0000 7ffbb27fc700 1 -- 172.21.15.98:0/1985804041 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80038c0 con 0x7ffbbc0cf570 2023-03-03T07:56:00.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.774+0000 7ffbc299d700 1 -- 172.21.15.98:0/1985804041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbbc15d0d0 con 0x7ffbbc0cf570 2023-03-03T07:56:00.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.774+0000 7ffbc299d700 1 -- 172.21.15.98:0/1985804041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbbc15cdf0 con 0x7ffbbc0cf570 2023-03-03T07:56:00.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.775+0000 7ffbc299d700 1 -- 172.21.15.98:0/1985804041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbbc0b51b0 con 0x7ffbbc0cf570 2023-03-03T07:56:00.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.776+0000 7ffbb27fc700 1 -- 172.21.15.98:0/1985804041 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffbb8019030 con 0x7ffbbc0cf570 2023-03-03T07:56:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.777+0000 7ffbb27fc700 1 --2- 172.21.15.98:0/1985804041 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffba00682f0 0x7ffba006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.777+0000 7ffbb27fc700 1 -- 172.21.15.98:0/1985804041 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ffbb8085130 con 0x7ffbbc0cf570 2023-03-03T07:56:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.778+0000 7ffbc119a700 1 --2- 172.21.15.98:0/1985804041 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffba00682f0 0x7ffba006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.778+0000 7ffbc119a700 1 --2- 172.21.15.98:0/1985804041 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffba00682f0 0x7ffba006a7a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffbac009fd0 tx=0x7ffbac01c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:00.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.781+0000 7ffbb27fc700 1 -- 172.21.15.98:0/1985804041 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffbb80518a0 con 0x7ffbbc0cf570 2023-03-03T07:56:00.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.974+0000 7ffbc299d700 1 -- 172.21.15.98:0/1985804041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ffbbc068290 con 0x7ffbbc0cf570 2023-03-03T07:56:00.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.975+0000 7ffbb27fc700 1 -- 172.21.15.98:0/1985804041 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7ffbb8054ec0 con 0x7ffbbc0cf570 2023-03-03T07:56:00.978 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:56:00.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.977+0000 7ffbc299d700 1 -- 172.21.15.98:0/1985804041 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffba00682f0 msgr2=0x7ffba006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:00.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.977+0000 7ffbc299d700 1 --2- 172.21.15.98:0/1985804041 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffba00682f0 0x7ffba006a7a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffbac009fd0 tx=0x7ffbac01c040 comp rx=0 tx=0).stop 2023-03-03T07:56:00.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.977+0000 7ffbc299d700 1 -- 172.21.15.98:0/1985804041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbbc0cf570 msgr2=0x7ffbbc151480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:00.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.977+0000 7ffbc299d700 1 --2- 172.21.15.98:0/1985804041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbbc0cf570 0x7ffbbc151480 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffbb800c500 tx=0x7ffbb800c8c0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.977+0000 7ffbc299d700 1 -- 172.21.15.98:0/1985804041 shutdown_connections 2023-03-03T07:56:00.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.977+0000 7ffbc299d700 1 --2- 172.21.15.98:0/1985804041 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffba00682f0 0x7ffba006a7a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.977+0000 7ffbc299d700 1 --2- 172.21.15.98:0/1985804041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbbc0cf570 0x7ffbbc151480 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.977+0000 7ffbc299d700 1 --2- 172.21.15.98:0/1985804041 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffbbc1609b0 0x7ffbbc0d1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.977+0000 7ffbc299d700 1 --2- 172.21.15.98:0/1985804041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffbbc15db30 0x7ffbbc1f2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:00.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.977+0000 7ffbc299d700 1 -- 172.21.15.98:0/1985804041 >> 172.21.15.98:0/1985804041 conn(0x7ffbbc06b880 msgr2=0x7ffbbc15bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:00.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.977+0000 7ffbc299d700 1 -- 172.21.15.98:0/1985804041 shutdown_connections 2023-03-03T07:56:00.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:00.977+0000 7ffbc299d700 1 -- 172.21.15.98:0/1985804041 wait complete. 2023-03-03T07:56:01.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:56:01.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:56:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.253+0000 7f16ca1ff700 1 Processor -- start 2023-03-03T07:56:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.253+0000 7f16ca1ff700 1 -- start start 2023-03-03T07:56:01.260 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:56:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.255+0000 7f16ca1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16c40b26d0 0x7f16c40b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.255+0000 7f16ca1ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f16c40b5140 0x7f16c40d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.255+0000 7f16ca1ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c40cd7d0 con 0x7f16c40b8d10 2023-03-03T07:56:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.255+0000 7f16ca1ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c40b55c0 con 0x7f16c40b5140 2023-03-03T07:56:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.255+0000 7f16ca1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c40b90e0 con 0x7f16c40b26d0 2023-03-03T07:56:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.255+0000 7f16c37fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f16c40b8d10 0x7f16c40cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53730/0 (socket says 172.21.15.98:53730) 2023-03-03T07:56:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.255+0000 7f16c37fe700 1 -- 172.21.15.98:0/1667408280 learned_addr learned my addr 172.21.15.98:0/1667408280 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.255+0000 7f16c3fff700 1 --2- 172.21.15.98:0/1667408280 >> v2:172.21.15.98:3300/0 conn(0x7f16c40b5140 0x7f16c40d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.255+0000 7f16c2ffd700 1 --2- 172.21.15.98:0/1667408280 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16c40b26d0 0x7f16c40b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.255+0000 7f16c27fc700 1 -- 172.21.15.98:0/1667408280 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936815224 0 0) 0x7f16c40cd7d0 con 0x7f16c40b8d10 2023-03-03T07:56:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.256+0000 7f16c27fc700 1 -- 172.21.15.98:0/1667408280 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16ac0035b0 con 0x7f16c40b8d10 2023-03-03T07:56:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.256+0000 7f16c3fff700 1 -- 172.21.15.98:0/1667408280 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16c40b26d0 msgr2=0x7f16c40b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.256+0000 7f16c3fff700 1 --2- 172.21.15.98:0/1667408280 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16c40b26d0 0x7f16c40b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.256+0000 7f16c3fff700 1 -- 172.21.15.98:0/1667408280 >> v1:172.21.15.98:6789/0 conn(0x7f16c40b8d10 legacy=0x7f16c40cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.256+0000 7f16c3fff700 1 -- 172.21.15.98:0/1667408280 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c40cda20 con 0x7f16c40b5140 2023-03-03T07:56:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.256+0000 7f16c3fff700 1 --2- 172.21.15.98:0/1667408280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16c40b5140 0x7f16c40d13a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009d90 tx=0x7f16bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ea6a7c8a444bdf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.256+0000 7f16c27fc700 1 -- 172.21.15.98:0/1667408280 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16bc00e070 con 0x7f16c40b5140 2023-03-03T07:56:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.256+0000 7f16c27fc700 1 -- 172.21.15.98:0/1667408280 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16bc00bb20 con 0x7f16c40b5140 2023-03-03T07:56:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.256+0000 7f16c27fc700 1 -- 172.21.15.98:0/1667408280 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16bc00be90 con 0x7f16c40b5140 2023-03-03T07:56:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.257+0000 7f16ca1ff700 1 -- 172.21.15.98:0/1667408280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16c40b5140 msgr2=0x7f16c40d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.257+0000 7f16ca1ff700 1 --2- 172.21.15.98:0/1667408280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16c40b5140 0x7f16c40d13a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009d90 tx=0x7f16bc00b330 comp rx=0 tx=0).stop 2023-03-03T07:56:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.257+0000 7f16ca1ff700 1 -- 172.21.15.98:0/1667408280 shutdown_connections 2023-03-03T07:56:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.257+0000 7f16ca1ff700 1 --2- 172.21.15.98:0/1667408280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16c40b5140 0x7f16c40d13a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.257+0000 7f16ca1ff700 1 --2- 172.21.15.98:0/1667408280 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16c40b26d0 0x7f16c40b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.257+0000 7f16ca1ff700 1 -- 172.21.15.98:0/1667408280 >> 172.21.15.98:0/1667408280 conn(0x7f16c4073e70 msgr2=0x7f16c406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.257+0000 7f16ca1ff700 1 -- 172.21.15.98:0/1667408280 shutdown_connections 2023-03-03T07:56:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.257+0000 7f16ca1ff700 1 -- 172.21.15.98:0/1667408280 wait complete. 2023-03-03T07:56:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.259+0000 7f16ca1ff700 1 Processor -- start 2023-03-03T07:56:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.259+0000 7f16ca1ff700 1 -- start start 2023-03-03T07:56:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.260+0000 7f16ca1ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16c40b26d0 0x7f16c4154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.260+0000 7f16ca1ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16c40b5140 0x7f16c40cac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.260+0000 7f16ca1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16c40b8d10 0x7f16c40b3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.261+0000 7f16ca1ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c415e3d0 con 0x7f16c40b5140 2023-03-03T07:56:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.261+0000 7f16ca1ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c40cbc00 con 0x7f16c40b26d0 2023-03-03T07:56:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.261+0000 7f16ca1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c4154d90 con 0x7f16c40b8d10 2023-03-03T07:56:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.261+0000 7f16c2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16c40b5140 0x7f16c40cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.261+0000 7f16c2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16c40b5140 0x7f16c40cac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41920/0 (socket says 172.21.15.98:41920) 2023-03-03T07:56:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.261+0000 7f16c2ffd700 1 -- 172.21.15.98:0/657100678 learned_addr learned my addr 172.21.15.98:0/657100678 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.261+0000 7f16c37fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16c40b26d0 0x7f16c4154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.261+0000 7f16c3fff700 1 --2- 172.21.15.98:0/657100678 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16c40b8d10 0x7f16c40b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.261+0000 7f16c2ffd700 1 -- 172.21.15.98:0/657100678 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16c40b8d10 msgr2=0x7f16c40b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.261+0000 7f16c2ffd700 1 --2- 172.21.15.98:0/657100678 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16c40b8d10 0x7f16c40b3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.261+0000 7f16c2ffd700 1 -- 172.21.15.98:0/657100678 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16c40b26d0 msgr2=0x7f16c4154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.261+0000 7f16c2ffd700 1 --2- 172.21.15.98:0/657100678 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16c40b26d0 0x7f16c4154850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.261+0000 7f16c2ffd700 1 -- 172.21.15.98:0/657100678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc009a70 con 0x7f16c40b5140 2023-03-03T07:56:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.262+0000 7f16c2ffd700 1 --2- 172.21.15.98:0/657100678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16c40b5140 0x7f16c40cac60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f16b800e870 tx=0x7f16b800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.262+0000 7f16abfff700 1 -- 172.21.15.98:0/657100678 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8009590 con 0x7f16c40b5140 2023-03-03T07:56:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.262+0000 7f16abfff700 1 -- 172.21.15.98:0/657100678 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b8013680 con 0x7f16c40b5140 2023-03-03T07:56:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.262+0000 7f16ca1ff700 1 -- 172.21.15.98:0/657100678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c40d0e80 con 0x7f16c40b5140 2023-03-03T07:56:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.262+0000 7f16abfff700 1 -- 172.21.15.98:0/657100678 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b80139f0 con 0x7f16c40b5140 2023-03-03T07:56:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.262+0000 7f16ca1ff700 1 -- 172.21.15.98:0/657100678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c40cdd00 con 0x7f16c40b5140 2023-03-03T07:56:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.263+0000 7f16ca1ff700 1 -- 172.21.15.98:0/657100678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c40cdfe0 con 0x7f16c40b5140 2023-03-03T07:56:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.265+0000 7f16abfff700 1 -- 172.21.15.98:0/657100678 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f16b8019030 con 0x7f16c40b5140 2023-03-03T07:56:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.266+0000 7f16abfff700 1 --2- 172.21.15.98:0/657100678 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16b00682f0 0x7f16b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.266+0000 7f16abfff700 1 -- 172.21.15.98:0/657100678 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f16b8085a40 con 0x7f16c40b5140 2023-03-03T07:56:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.266+0000 7f16c37fe700 1 --2- 172.21.15.98:0/657100678 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16b00682f0 0x7f16b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.267+0000 7f16c37fe700 1 --2- 172.21.15.98:0/657100678 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16b00682f0 0x7f16b006a7a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f16b4007fd0 tx=0x7f16b4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.270+0000 7f16abfff700 1 -- 172.21.15.98:0/657100678 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f16b8052150 con 0x7f16c40b5140 2023-03-03T07:56:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.479+0000 7f16ca1ff700 1 -- 172.21.15.98:0/657100678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f16c40cf470 con 0x7f16c40b5140 2023-03-03T07:56:01.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.479+0000 7f16abfff700 1 -- 172.21.15.98:0/657100678 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f16b8055770 con 0x7f16c40b5140 2023-03-03T07:56:01.481 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:56:01.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.482+0000 7f16ca1ff700 1 -- 172.21.15.98:0/657100678 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16b00682f0 msgr2=0x7f16b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:01.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.482+0000 7f16ca1ff700 1 --2- 172.21.15.98:0/657100678 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16b00682f0 0x7f16b006a7a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f16b4007fd0 tx=0x7f16b4009040 comp rx=0 tx=0).stop 2023-03-03T07:56:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.482+0000 7f16ca1ff700 1 -- 172.21.15.98:0/657100678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16c40b5140 msgr2=0x7f16c40cac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.482+0000 7f16ca1ff700 1 --2- 172.21.15.98:0/657100678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16c40b5140 0x7f16c40cac60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f16b800e870 tx=0x7f16b800eb80 comp rx=0 tx=0).stop 2023-03-03T07:56:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.482+0000 7f16ca1ff700 1 -- 172.21.15.98:0/657100678 shutdown_connections 2023-03-03T07:56:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.482+0000 7f16ca1ff700 1 --2- 172.21.15.98:0/657100678 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16b00682f0 0x7f16b006a7a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.482+0000 7f16ca1ff700 1 --2- 172.21.15.98:0/657100678 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16c40b8d10 0x7f16c40b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.482+0000 7f16ca1ff700 1 --2- 172.21.15.98:0/657100678 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16c40b26d0 0x7f16c4154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.482+0000 7f16ca1ff700 1 --2- 172.21.15.98:0/657100678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16c40b5140 0x7f16c40cac60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.482+0000 7f16ca1ff700 1 -- 172.21.15.98:0/657100678 >> 172.21.15.98:0/657100678 conn(0x7f16c4073e70 msgr2=0x7f16c415b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.482+0000 7f16ca1ff700 1 -- 172.21.15.98:0/657100678 shutdown_connections 2023-03-03T07:56:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.482+0000 7f16ca1ff700 1 -- 172.21.15.98:0/657100678 wait complete. 2023-03-03T07:56:01.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:56:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.757+0000 7f823ffff700 1 Processor -- start 2023-03-03T07:56:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.758+0000 7f823ffff700 1 -- start start 2023-03-03T07:56:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.759+0000 7f823ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82380b8d10 0x7f82380cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.759+0000 7f823ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f82380b5310 0x7f82380d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.759+0000 7f823ffff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82380cd7d0 con 0x7f82380b26d0 2023-03-03T07:56:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.759+0000 7f823ffff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82380b90e0 con 0x7f82380b5310 2023-03-03T07:56:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.759+0000 7f823ffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82380cda20 con 0x7f82380b8d10 2023-03-03T07:56:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.759+0000 7f823e59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f82380b5310 0x7f82380d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.760+0000 7f823dd9b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82380b8d10 0x7f82380cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.760+0000 7f823d59a700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f82380b26d0 0x7f82380b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:47892/0 (socket says 172.21.15.98:47892) 2023-03-03T07:56:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.760+0000 7f823d59a700 1 -- 172.21.15.98:0/4272294593 learned_addr learned my addr 172.21.15.98:0/4272294593 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.760+0000 7f823e59c700 1 -- 172.21.15.98:0/4272294593 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82380b8d10 msgr2=0x7f82380cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.760+0000 7f823e59c700 1 --2- 172.21.15.98:0/4272294593 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82380b8d10 0x7f82380cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.760+0000 7f823e59c700 1 -- 172.21.15.98:0/4272294593 >> v1:172.21.15.100:6789/0 conn(0x7f82380b26d0 legacy=0x7f82380b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.760+0000 7f823e59c700 1 -- 172.21.15.98:0/4272294593 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82380cdd00 con 0x7f82380b5310 2023-03-03T07:56:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.760+0000 7f823e59c700 1 --2- 172.21.15.98:0/4272294593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82380b5310 0x7f82380d13a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8230009d90 tx=0x7f823000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0768fd1ade08941 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.761+0000 7f823cd99700 1 -- 172.21.15.98:0/4272294593 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823000e070 con 0x7f82380b5310 2023-03-03T07:56:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.761+0000 7f823cd99700 1 -- 172.21.15.98:0/4272294593 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823000bb20 con 0x7f82380b5310 2023-03-03T07:56:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.761+0000 7f823cd99700 1 -- 172.21.15.98:0/4272294593 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8230012430 con 0x7f82380b5310 2023-03-03T07:56:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.761+0000 7f823ffff700 1 -- 172.21.15.98:0/4272294593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82380b5310 msgr2=0x7f82380d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.761+0000 7f823ffff700 1 --2- 172.21.15.98:0/4272294593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82380b5310 0x7f82380d13a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8230009d90 tx=0x7f823000b330 comp rx=0 tx=0).stop 2023-03-03T07:56:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.761+0000 7f823ffff700 1 -- 172.21.15.98:0/4272294593 shutdown_connections 2023-03-03T07:56:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.761+0000 7f823ffff700 1 --2- 172.21.15.98:0/4272294593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82380b5310 0x7f82380d13a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.761+0000 7f823ffff700 1 --2- 172.21.15.98:0/4272294593 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82380b8d10 0x7f82380cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.761+0000 7f823ffff700 1 -- 172.21.15.98:0/4272294593 >> 172.21.15.98:0/4272294593 conn(0x7f8238073e70 msgr2=0x7f823806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.761+0000 7f823ffff700 1 -- 172.21.15.98:0/4272294593 shutdown_connections 2023-03-03T07:56:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.761+0000 7f823ffff700 1 -- 172.21.15.98:0/4272294593 wait complete. 2023-03-03T07:56:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.763+0000 7f823ffff700 1 Processor -- start 2023-03-03T07:56:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.764+0000 7f823ffff700 1 -- start start 2023-03-03T07:56:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82380b26d0 0x7f82381515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82380b5310 0x7f8238154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82380b8d10 0x7f82380b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823ffff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82381558f0 con 0x7f82380b8d10 2023-03-03T07:56:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823ffff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8238155bd0 con 0x7f82380b26d0 2023-03-03T07:56:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823ffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8238155eb0 con 0x7f82380b5310 2023-03-03T07:56:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823dd9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82380b26d0 0x7f82381515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82380b8d10 0x7f82380b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823d59a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82380b5310 0x7f8238154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823dd9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82380b26d0 0x7f82381515e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:47270/0 (socket says 172.21.15.98:47270) 2023-03-03T07:56:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823dd9b700 1 -- 172.21.15.98:0/2553459821 learned_addr learned my addr 172.21.15.98:0/2553459821 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823e59c700 1 -- 172.21.15.98:0/2553459821 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82380b5310 msgr2=0x7f8238154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823e59c700 1 --2- 172.21.15.98:0/2553459821 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82380b5310 0x7f8238154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823e59c700 1 -- 172.21.15.98:0/2553459821 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82380b26d0 msgr2=0x7f82381515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823e59c700 1 --2- 172.21.15.98:0/2553459821 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82380b26d0 0x7f82381515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.765+0000 7f823e59c700 1 -- 172.21.15.98:0/2553459821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8230009a70 con 0x7f82380b8d10 2023-03-03T07:56:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.766+0000 7f823e59c700 1 --2- 172.21.15.98:0/2553459821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82380b8d10 0x7f82380b3c70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8230016000 tx=0x7f8230004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.766+0000 7f822e7fc700 1 -- 172.21.15.98:0/2553459821 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8230012430 con 0x7f82380b8d10 2023-03-03T07:56:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.766+0000 7f822e7fc700 1 -- 172.21.15.98:0/2553459821 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8230020070 con 0x7f82380b8d10 2023-03-03T07:56:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.766+0000 7f822e7fc700 1 -- 172.21.15.98:0/2553459821 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82300098f0 con 0x7f82380b8d10 2023-03-03T07:56:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.766+0000 7f823ffff700 1 -- 172.21.15.98:0/2553459821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82380cdfe0 con 0x7f82380b8d10 2023-03-03T07:56:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.766+0000 7f823ffff700 1 -- 172.21.15.98:0/2553459821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82380d0e80 con 0x7f82380b8d10 2023-03-03T07:56:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.767+0000 7f823ffff700 1 -- 172.21.15.98:0/2553459821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82380d0500 con 0x7f82380b8d10 2023-03-03T07:56:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.769+0000 7f822e7fc700 1 -- 172.21.15.98:0/2553459821 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f823002e070 con 0x7f82380b8d10 2023-03-03T07:56:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.771+0000 7f822e7fc700 1 --2- 172.21.15.98:0/2553459821 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f821c0682f0 0x7f821c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.771+0000 7f822e7fc700 1 -- 172.21.15.98:0/2553459821 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8230022070 con 0x7f82380b8d10 2023-03-03T07:56:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.771+0000 7f823dd9b700 1 --2- 172.21.15.98:0/2553459821 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f821c0682f0 0x7f821c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.771+0000 7f823dd9b700 1 --2- 172.21.15.98:0/2553459821 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f821c0682f0 0x7f821c06a7a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f82380d0210 tx=0x7f8234009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.775+0000 7f822e7fc700 1 -- 172.21.15.98:0/2553459821 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8230054100 con 0x7f82380b8d10 2023-03-03T07:56:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.967+0000 7f823ffff700 1 -- 172.21.15.98:0/2553459821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f823804e6e0 con 0x7f82380b8d10 2023-03-03T07:56:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.968+0000 7f822e7fc700 1 -- 172.21.15.98:0/2553459821 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f823002c0a0 con 0x7f82380b8d10 2023-03-03T07:56:01.970 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:56:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.970+0000 7f823ffff700 1 -- 172.21.15.98:0/2553459821 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f821c0682f0 msgr2=0x7f821c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.970+0000 7f823ffff700 1 --2- 172.21.15.98:0/2553459821 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f821c0682f0 0x7f821c06a7a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f82380d0210 tx=0x7f8234009210 comp rx=0 tx=0).stop 2023-03-03T07:56:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.970+0000 7f823ffff700 1 -- 172.21.15.98:0/2553459821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82380b8d10 msgr2=0x7f82380b3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.970+0000 7f823ffff700 1 --2- 172.21.15.98:0/2553459821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82380b8d10 0x7f82380b3c70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8230016000 tx=0x7f8230004ad0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.970+0000 7f823ffff700 1 -- 172.21.15.98:0/2553459821 shutdown_connections 2023-03-03T07:56:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.970+0000 7f823ffff700 1 --2- 172.21.15.98:0/2553459821 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f821c0682f0 0x7f821c06a7a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.970+0000 7f823ffff700 1 --2- 172.21.15.98:0/2553459821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82380b8d10 0x7f82380b3c70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.970+0000 7f823ffff700 1 --2- 172.21.15.98:0/2553459821 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82380b26d0 0x7f82381515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.970+0000 7f823ffff700 1 --2- 172.21.15.98:0/2553459821 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82380b5310 0x7f8238154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.970+0000 7f823ffff700 1 -- 172.21.15.98:0/2553459821 >> 172.21.15.98:0/2553459821 conn(0x7f8238073e70 msgr2=0x7f82380b6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.970+0000 7f823ffff700 1 -- 172.21.15.98:0/2553459821 shutdown_connections 2023-03-03T07:56:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:01.970+0000 7f823ffff700 1 -- 172.21.15.98:0/2553459821 wait complete. 2023-03-03T07:56:01.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:56:02.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.244+0000 7f9c053ff700 1 Processor -- start 2023-03-03T07:56:02.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.244+0000 7f9c053ff700 1 -- start start 2023-03-03T07:56:02.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.245+0000 7f9c053ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c000cd8d0 0x7f9c000b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:02.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.246+0000 7f9c053ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c000d1a00 con 0x7f9c0015cc00 2023-03-03T07:56:02.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.246+0000 7f9c053ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c000d1ce0 con 0x7f9c000b6610 2023-03-03T07:56:02.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.246+0000 7f9c053ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c000ce040 con 0x7f9c000cd8d0 2023-03-03T07:56:02.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.246+0000 7f9bfeffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9c0015cc00 0x7f9c000d27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53742/0 (socket says 172.21.15.98:53742) 2023-03-03T07:56:02.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.246+0000 7f9bfeffd700 1 -- 172.21.15.98:0/1266462363 learned_addr learned my addr 172.21.15.98:0/1266462363 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:02.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.246+0000 7f9bfe7fc700 1 --2- 172.21.15.98:0/1266462363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c000cd8d0 0x7f9c000b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:02.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.246+0000 7f9bfdffb700 1 -- 172.21.15.98:0/1266462363 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985484496 0 0) 0x7f9c000d1a00 con 0x7f9c0015cc00 2023-03-03T07:56:02.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.246+0000 7f9bfdffb700 1 -- 172.21.15.98:0/1266462363 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be80035b0 con 0x7f9c0015cc00 2023-03-03T07:56:02.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.247+0000 7f9bfe7fc700 1 -- 172.21.15.98:0/1266462363 >> v1:172.21.15.100:6789/0 conn(0x7f9c000b6610 legacy=0x7f9c000cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:02.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.247+0000 7f9bfdffb700 1 -- 172.21.15.98:0/1266462363 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504286100 0 0) 0x7f9c000d1ce0 con 0x7f9c000b6610 2023-03-03T07:56:02.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.247+0000 7f9bfe7fc700 1 -- 172.21.15.98:0/1266462363 >> v1:172.21.15.98:6789/0 conn(0x7f9c0015cc00 legacy=0x7f9c000d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:02.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.247+0000 7f9bfe7fc700 1 -- 172.21.15.98:0/1266462363 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c000d1fc0 con 0x7f9c000cd8d0 2023-03-03T07:56:02.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.247+0000 7f9bfe7fc700 1 --2- 172.21.15.98:0/1266462363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c000cd8d0 0x7f9c000b60d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4009870 tx=0x7f9bf4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcf8ff548aa4daf4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:02.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.247+0000 7f9bfdffb700 1 -- 172.21.15.98:0/1266462363 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4011830 con 0x7f9c000cd8d0 2023-03-03T07:56:02.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.247+0000 7f9bfdffb700 1 -- 172.21.15.98:0/1266462363 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf4011990 con 0x7f9c000cd8d0 2023-03-03T07:56:02.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.247+0000 7f9bfdffb700 1 -- 172.21.15.98:0/1266462363 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4011d20 con 0x7f9c000cd8d0 2023-03-03T07:56:02.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.248+0000 7f9c053ff700 1 -- 172.21.15.98:0/1266462363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c000cd8d0 msgr2=0x7f9c000b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:02.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.248+0000 7f9c053ff700 1 --2- 172.21.15.98:0/1266462363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c000cd8d0 0x7f9c000b60d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4009870 tx=0x7f9bf4009b80 comp rx=0 tx=0).stop 2023-03-03T07:56:02.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.248+0000 7f9c053ff700 1 -- 172.21.15.98:0/1266462363 shutdown_connections 2023-03-03T07:56:02.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.248+0000 7f9c053ff700 1 --2- 172.21.15.98:0/1266462363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c000cd8d0 0x7f9c000b60d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:02.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.248+0000 7f9c053ff700 1 -- 172.21.15.98:0/1266462363 >> 172.21.15.98:0/1266462363 conn(0x7f9c0006aee0 msgr2=0x7f9c0006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:02.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.248+0000 7f9c053ff700 1 -- 172.21.15.98:0/1266462363 shutdown_connections 2023-03-03T07:56:02.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.248+0000 7f9c053ff700 1 -- 172.21.15.98:0/1266462363 wait complete. 2023-03-03T07:56:02.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.250+0000 7f9c053ff700 1 Processor -- start 2023-03-03T07:56:02.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.250+0000 7f9c053ff700 1 -- start start 2023-03-03T07:56:02.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.251+0000 7f9c053ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9c000b6610 0x7f9c0015f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:02.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.251+0000 7f9c053ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c000cd8d0 0x7f9c000b5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:02.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.251+0000 7f9c053ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c0015cc00 0x7f9c001f57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:02.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.251+0000 7f9c053ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c000cb9d0 con 0x7f9c0015cc00 2023-03-03T07:56:02.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.251+0000 7f9c053ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c00143150 con 0x7f9c000b6610 2023-03-03T07:56:02.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.251+0000 7f9c053ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c000cb3a0 con 0x7f9c000cd8d0 2023-03-03T07:56:02.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.252+0000 7f9bff7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c0015cc00 0x7f9c001f57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:02.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.252+0000 7f9bfe7fc700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c000cd8d0 0x7f9c000b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:02.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.252+0000 7f9bff7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c0015cc00 0x7f9c001f57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41938/0 (socket says 172.21.15.98:41938) 2023-03-03T07:56:02.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.252+0000 7f9bfe7fc700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c000cd8d0 0x7f9c000b5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:33036/0 (socket says 172.21.15.98:33036) 2023-03-03T07:56:02.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.252+0000 7f9bfe7fc700 1 -- 172.21.15.98:0/2858918111 learned_addr learned my addr 172.21.15.98:0/2858918111 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:02.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.252+0000 7f9bfeffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9c000b6610 0x7f9c0015f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:02.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.252+0000 7f9bff7fe700 1 -- 172.21.15.98:0/2858918111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c000cd8d0 msgr2=0x7f9c000b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:02.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.252+0000 7f9bff7fe700 1 --2- 172.21.15.98:0/2858918111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c000cd8d0 0x7f9c000b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:02.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.252+0000 7f9bff7fe700 1 -- 172.21.15.98:0/2858918111 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9c000b6610 msgr2=0x7f9c0015f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:02.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.252+0000 7f9bff7fe700 1 --2- 172.21.15.98:0/2858918111 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9c000b6610 0x7f9c0015f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:02.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.252+0000 7f9bff7fe700 1 -- 172.21.15.98:0/2858918111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf40094a0 con 0x7f9c0015cc00 2023-03-03T07:56:02.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.252+0000 7f9bfe7fc700 1 --2- 172.21.15.98:0/2858918111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c000cd8d0 0x7f9c000b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:56:02.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.253+0000 7f9bff7fe700 1 --2- 172.21.15.98:0/2858918111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c0015cc00 0x7f9c001f57a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9bf800c7b0 tx=0x7f9bf800cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:02.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.253+0000 7f9be77fe700 1 -- 172.21.15.98:0/2858918111 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8003e30 con 0x7f9c0015cc00 2023-03-03T07:56:02.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.253+0000 7f9be77fe700 1 -- 172.21.15.98:0/2858918111 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf80050d0 con 0x7f9c0015cc00 2023-03-03T07:56:02.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.253+0000 7f9be77fe700 1 -- 172.21.15.98:0/2858918111 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8005460 con 0x7f9c0015cc00 2023-03-03T07:56:02.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.253+0000 7f9c053ff700 1 -- 172.21.15.98:0/2858918111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c000cdd40 con 0x7f9c0015cc00 2023-03-03T07:56:02.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.253+0000 7f9c053ff700 1 -- 172.21.15.98:0/2858918111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c000d22a0 con 0x7f9c0015cc00 2023-03-03T07:56:02.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.254+0000 7f9c053ff700 1 -- 172.21.15.98:0/2858918111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c000d2ed0 con 0x7f9c0015cc00 2023-03-03T07:56:02.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.255+0000 7f9be77fe700 1 -- 172.21.15.98:0/2858918111 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9bf8019030 con 0x7f9c0015cc00 2023-03-03T07:56:02.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.256+0000 7f9be77fe700 1 --2- 172.21.15.98:0/2858918111 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9bec0682f0 0x7f9bec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:02.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.257+0000 7f9be77fe700 1 -- 172.21.15.98:0/2858918111 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9bf8085210 con 0x7f9c0015cc00 2023-03-03T07:56:02.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.257+0000 7f9bfeffd700 1 --2- 172.21.15.98:0/2858918111 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9bec0682f0 0x7f9bec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:02.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.257+0000 7f9bfeffd700 1 --2- 172.21.15.98:0/2858918111 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9bec0682f0 0x7f9bec06a7a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0007fd0 tx=0x7f9bf0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.260+0000 7f9be77fe700 1 -- 172.21.15.98:0/2858918111 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9bf8051980 con 0x7f9c0015cc00 2023-03-03T07:56:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.494+0000 7f9c053ff700 1 -- 172.21.15.98:0/2858918111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c000729b0 con 0x7f9c0015cc00 2023-03-03T07:56:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.495+0000 7f9be77fe700 1 -- 172.21.15.98:0/2858918111 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4639 (secure 0 0 0) 0x7f9bf8054fa0 con 0x7f9c0015cc00 2023-03-03T07:56:02.497 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:56:02.497 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/246835374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":246835374},{"type":"v1","addr":"172.21.15.98:6836","nonce":246835374}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6833/959944231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":959944231},{"type":"v1","addr":"172.21.15.100:6833","nonce":959944231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/872788798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":872788798},{"type":"v1","addr":"172.21.15.100:6835","nonce":872788798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:55:47.097221+0000","modified":"2023-03-03T07:55:50.555384+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/3969084152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3969084152},{"type":"v1","addr":"172.21.15.98:6837","nonce":3969084152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:56:02.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.498+0000 7f9c053ff700 1 -- 172.21.15.98:0/2858918111 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9bec0682f0 msgr2=0x7f9bec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:02.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.498+0000 7f9c053ff700 1 --2- 172.21.15.98:0/2858918111 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9bec0682f0 0x7f9bec06a7a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0007fd0 tx=0x7f9bf0009040 comp rx=0 tx=0).stop 2023-03-03T07:56:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.498+0000 7f9c053ff700 1 -- 172.21.15.98:0/2858918111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c0015cc00 msgr2=0x7f9c001f57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.498+0000 7f9c053ff700 1 --2- 172.21.15.98:0/2858918111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c0015cc00 0x7f9c001f57a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9bf800c7b0 tx=0x7f9bf800cac0 comp rx=0 tx=0).stop 2023-03-03T07:56:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.498+0000 7f9c053ff700 1 -- 172.21.15.98:0/2858918111 shutdown_connections 2023-03-03T07:56:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.498+0000 7f9c053ff700 1 --2- 172.21.15.98:0/2858918111 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9bec0682f0 0x7f9bec06a7a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.498+0000 7f9c053ff700 1 --2- 172.21.15.98:0/2858918111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c0015cc00 0x7f9c001f57a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.498+0000 7f9c053ff700 1 --2- 172.21.15.98:0/2858918111 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9c000b6610 0x7f9c0015f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.498+0000 7f9c053ff700 1 --2- 172.21.15.98:0/2858918111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c000cd8d0 0x7f9c000b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.498+0000 7f9c053ff700 1 -- 172.21.15.98:0/2858918111 >> 172.21.15.98:0/2858918111 conn(0x7f9c0006aee0 msgr2=0x7f9c000b6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.498+0000 7f9c053ff700 1 -- 172.21.15.98:0/2858918111 shutdown_connections 2023-03-03T07:56:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:02.498+0000 7f9c053ff700 1 -- 172.21.15.98:0/2858918111 wait complete. 2023-03-03T07:56:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 7 2023-03-03T07:56:02.929 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-03-03T07:55:47.097221+0000', 'modified': '2023-03-03T07:55:50.555384+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4285}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4285': {'gid': 4285, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.98:6837/3969084152', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3969084152}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3969084152}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-03-03T07:56:02.930 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-03-03T07:56:02.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-03-03T07:56:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.180+0000 7f03389ff700 1 Processor -- start 2023-03-03T07:56:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.180+0000 7f03389ff700 1 -- start start 2023-03-03T07:56:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.181+0000 7f03389ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f03340d0270 0x7f03340cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.181+0000 7f03389ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f033406b4b0 0x7f03340d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.181+0000 7f03389ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03340cf980 con 0x7f033415ca70 2023-03-03T07:56:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.181+0000 7f03389ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03340cd5b0 con 0x7f03340d0270 2023-03-03T07:56:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.181+0000 7f03389ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03340d1800 con 0x7f033406b4b0 2023-03-03T07:56:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.182+0000 7f033259c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f033415ca70 0x7f03340cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53744/0 (socket says 172.21.15.98:53744) 2023-03-03T07:56:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.182+0000 7f033259c700 1 -- 172.21.15.98:0/2296892586 learned_addr learned my addr 172.21.15.98:0/2296892586 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.182+0000 7f0331d9b700 1 --2- 172.21.15.98:0/2296892586 >> v2:172.21.15.98:3300/0 conn(0x7f03340d0270 0x7f03340cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.182+0000 7f0332d9d700 1 --2- 172.21.15.98:0/2296892586 >> v2:172.21.15.100:3300/0 conn(0x7f033406b4b0 0x7f03340d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.182+0000 7f033159a700 1 -- 172.21.15.98:0/2296892586 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542252264 0 0) 0x7f03340cf980 con 0x7f033415ca70 2023-03-03T07:56:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.182+0000 7f033159a700 1 -- 172.21.15.98:0/2296892586 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f031c0035b0 con 0x7f033415ca70 2023-03-03T07:56:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.182+0000 7f0331d9b700 1 -- 172.21.15.98:0/2296892586 >> v2:172.21.15.100:3300/0 conn(0x7f033406b4b0 msgr2=0x7f03340d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.182+0000 7f0331d9b700 1 --2- 172.21.15.98:0/2296892586 >> v2:172.21.15.100:3300/0 conn(0x7f033406b4b0 0x7f03340d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.182+0000 7f0331d9b700 1 -- 172.21.15.98:0/2296892586 >> v1:172.21.15.98:6789/0 conn(0x7f033415ca70 legacy=0x7f03340cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.182+0000 7f0331d9b700 1 -- 172.21.15.98:0/2296892586 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03340cd900 con 0x7f03340d0270 2023-03-03T07:56:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.183+0000 7f0331d9b700 1 --2- 172.21.15.98:0/2296892586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03340d0270 0x7f03340cf440 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f03340ca1d0 tx=0x7f0328009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a7e026a43a99f86 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.183+0000 7f033159a700 1 -- 172.21.15.98:0/2296892586 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0328011830 con 0x7f03340d0270 2023-03-03T07:56:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.183+0000 7f033159a700 1 -- 172.21.15.98:0/2296892586 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0328011990 con 0x7f03340d0270 2023-03-03T07:56:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.183+0000 7f033159a700 1 -- 172.21.15.98:0/2296892586 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0328011d20 con 0x7f03340d0270 2023-03-03T07:56:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.183+0000 7f03389ff700 1 -- 172.21.15.98:0/2296892586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03340d0270 msgr2=0x7f03340cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.183+0000 7f03389ff700 1 --2- 172.21.15.98:0/2296892586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03340d0270 0x7f03340cf440 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f03340ca1d0 tx=0x7f0328009b20 comp rx=0 tx=0).stop 2023-03-03T07:56:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.184+0000 7f03389ff700 1 -- 172.21.15.98:0/2296892586 shutdown_connections 2023-03-03T07:56:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.184+0000 7f03389ff700 1 --2- 172.21.15.98:0/2296892586 >> v2:172.21.15.100:3300/0 conn(0x7f033406b4b0 0x7f03340d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.184+0000 7f03389ff700 1 --2- 172.21.15.98:0/2296892586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03340d0270 0x7f03340cf440 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.184+0000 7f03389ff700 1 -- 172.21.15.98:0/2296892586 >> 172.21.15.98:0/2296892586 conn(0x7f0334073e70 msgr2=0x7f033406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.184+0000 7f03389ff700 1 -- 172.21.15.98:0/2296892586 shutdown_connections 2023-03-03T07:56:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.184+0000 7f03389ff700 1 -- 172.21.15.98:0/2296892586 wait complete. 2023-03-03T07:56:03.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.186+0000 7f03389ff700 1 Processor -- start 2023-03-03T07:56:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.186+0000 7f03389ff700 1 -- start start 2023-03-03T07:56:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.187+0000 7f03389ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f033406b4b0 0x7f03340ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.187+0000 7f03389ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03340d0270 0x7f033414a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.187+0000 7f03389ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f033415ca70 0x7f03340b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.187+0000 7f03389ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03341f6ac0 con 0x7f033406b4b0 2023-03-03T07:56:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.187+0000 7f03389ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03341f5f40 con 0x7f03340d0270 2023-03-03T07:56:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.187+0000 7f03389ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03341f6220 con 0x7f033415ca70 2023-03-03T07:56:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.187+0000 7f033259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f033406b4b0 0x7f03340ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.187+0000 7f033259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f033406b4b0 0x7f03340ca250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41954/0 (socket says 172.21.15.98:41954) 2023-03-03T07:56:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.187+0000 7f033259c700 1 -- 172.21.15.98:0/387093643 learned_addr learned my addr 172.21.15.98:0/387093643 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.188+0000 7f0332d9d700 1 --2- 172.21.15.98:0/387093643 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f033415ca70 0x7f03340b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.188+0000 7f0331d9b700 1 --2- 172.21.15.98:0/387093643 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03340d0270 0x7f033414a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.188+0000 7f033259c700 1 -- 172.21.15.98:0/387093643 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f033415ca70 msgr2=0x7f03340b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.188+0000 7f033259c700 1 --2- 172.21.15.98:0/387093643 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f033415ca70 0x7f03340b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.188+0000 7f033259c700 1 -- 172.21.15.98:0/387093643 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03340d0270 msgr2=0x7f033414a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.188+0000 7f033259c700 1 --2- 172.21.15.98:0/387093643 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03340d0270 0x7f033414a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.188+0000 7f033259c700 1 -- 172.21.15.98:0/387093643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03280094a0 con 0x7f033406b4b0 2023-03-03T07:56:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.188+0000 7f0331d9b700 1 --2- 172.21.15.98:0/387093643 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03340d0270 0x7f033414a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:56:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.188+0000 7f033259c700 1 --2- 172.21.15.98:0/387093643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f033406b4b0 0x7f03340ca250 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f032400c700 tx=0x7f032400cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.188+0000 7f0322ffd700 1 -- 172.21.15.98:0/387093643 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0324005d70 con 0x7f033406b4b0 2023-03-03T07:56:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.188+0000 7f0322ffd700 1 -- 172.21.15.98:0/387093643 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03240070d0 con 0x7f033406b4b0 2023-03-03T07:56:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.188+0000 7f03389ff700 1 -- 172.21.15.98:0/387093643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03340d1aa0 con 0x7f033406b4b0 2023-03-03T07:56:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.188+0000 7f0322ffd700 1 -- 172.21.15.98:0/387093643 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0324007460 con 0x7f033406b4b0 2023-03-03T07:56:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.188+0000 7f03389ff700 1 -- 172.21.15.98:0/387093643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03340cdbe0 con 0x7f033406b4b0 2023-03-03T07:56:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.191+0000 7f0322ffd700 1 -- 172.21.15.98:0/387093643 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0324019030 con 0x7f033406b4b0 2023-03-03T07:56:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.192+0000 7f0322ffd700 1 --2- 172.21.15.98:0/387093643 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f03140682f0 0x7f031406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.192+0000 7f0322ffd700 1 -- 172.21.15.98:0/387093643 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f03240852c0 con 0x7f033406b4b0 2023-03-03T07:56:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.192+0000 7f0331d9b700 1 --2- 172.21.15.98:0/387093643 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f03140682f0 0x7f031406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.193+0000 7f0331d9b700 1 --2- 172.21.15.98:0/387093643 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f03140682f0 0x7f031406a7a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f03340ca1d0 tx=0x7f0328008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.194+0000 7f034c27eb80 1 Processor -- start 2023-03-03T07:56:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.194+0000 7f034c27eb80 1 -- start start 2023-03-03T07:56:03.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.195+0000 7f034c27eb80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd17a70 0x55fcacd17e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.195+0000 7f034c27eb80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd16cd0 0x55fcacd66030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.195+0000 7f034c27eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd12560 0x55fcacd66570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.195+0000 7f034c27eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcacc8d4f0 con 0x55fcacd12560 2023-03-03T07:56:03.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.195+0000 7f034c27eb80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcacc02b60 con 0x55fcacd17a70 2023-03-03T07:56:03.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.195+0000 7f034c27eb80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fcacc71ad0 con 0x55fcacd16cd0 2023-03-03T07:56:03.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.196+0000 7f0332d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd12560 0x55fcacd66570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.196+0000 7f0332d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd12560 0x55fcacd66570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41962/0 (socket says 172.21.15.98:41962) 2023-03-03T07:56:03.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.196+0000 7f0332d9d700 1 -- 172.21.15.98:0/2758831609 learned_addr learned my addr 172.21.15.98:0/2758831609 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.196+0000 7f0331d9b700 1 --2- 172.21.15.98:0/2758831609 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd16cd0 0x55fcacd66030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.196+0000 7f033259c700 1 --2- 172.21.15.98:0/2758831609 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd17a70 0x55fcacd17e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.196+0000 7f0332d9d700 1 -- 172.21.15.98:0/2758831609 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd16cd0 msgr2=0x55fcacd66030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.196+0000 7f0332d9d700 1 --2- 172.21.15.98:0/2758831609 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd16cd0 0x55fcacd66030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.196+0000 7f0332d9d700 1 -- 172.21.15.98:0/2758831609 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd17a70 msgr2=0x55fcacd17e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.196+0000 7f0332d9d700 1 --2- 172.21.15.98:0/2758831609 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd17a70 0x55fcacd17e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.196+0000 7f0332d9d700 1 -- 172.21.15.98:0/2758831609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fcaca5f540 con 0x55fcacd12560 2023-03-03T07:56:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.197+0000 7f0332d9d700 1 --2- 172.21.15.98:0/2758831609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd12560 0x55fcacd66570 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f032c0073e0 tx=0x7f032c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55343bf3dbc88e87 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.197+0000 7f0313fff700 1 -- 172.21.15.98:0/2758831609 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032c019070 con 0x55fcacd12560 2023-03-03T07:56:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.197+0000 7f0313fff700 1 -- 172.21.15.98:0/2758831609 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032c00f050 con 0x55fcacd12560 2023-03-03T07:56:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.197+0000 7f0313fff700 1 -- 172.21.15.98:0/2758831609 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032c003a20 con 0x55fcacd12560 2023-03-03T07:56:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.197+0000 7f034c27eb80 1 -- 172.21.15.98:0/2758831609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd12560 msgr2=0x55fcacd66570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.197+0000 7f034c27eb80 1 --2- 172.21.15.98:0/2758831609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd12560 0x55fcacd66570 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f032c0073e0 tx=0x7f032c009750 comp rx=0 tx=0).stop 2023-03-03T07:56:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.198+0000 7f034c27eb80 1 -- 172.21.15.98:0/2758831609 shutdown_connections 2023-03-03T07:56:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.198+0000 7f034c27eb80 1 --2- 172.21.15.98:0/2758831609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd12560 0x55fcacd66570 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.198+0000 7f034c27eb80 1 --2- 172.21.15.98:0/2758831609 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd17a70 0x55fcacd17e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.198+0000 7f034c27eb80 1 --2- 172.21.15.98:0/2758831609 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd16cd0 0x55fcacd66030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.198+0000 7f034c27eb80 1 -- 172.21.15.98:0/2758831609 >> 172.21.15.98:0/2758831609 conn(0x55fcacd15c30 msgr2=0x55fcacd165c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:03.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.198+0000 7f034c27eb80 1 -- 172.21.15.98:0/2758831609 shutdown_connections 2023-03-03T07:56:03.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.198+0000 7f034c27eb80 1 -- 172.21.15.98:0/2758831609 wait complete. 2023-03-03T07:56:03.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.199+0000 7f034c27eb80 1 -- start start 2023-03-03T07:56:03.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.199+0000 7f034c27eb80 1 Processor -- start 2023-03-03T07:56:03.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.200+0000 7f034c27eb80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd12560 0x55fcacd3dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.200+0000 7f034c27eb80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd16cd0 0x55fcacd70200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.200+0000 7f034c27eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd70740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.200+0000 7f034c27eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcacc6ae40 con 0x55fcacd17a70 2023-03-03T07:56:03.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.200+0000 7f034c27eb80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcaca60140 con 0x55fcacd16cd0 2023-03-03T07:56:03.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.200+0000 7f034c27eb80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fcacd41480 con 0x55fcacd12560 2023-03-03T07:56:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.200+0000 7f034c27eb80 10 client.0 _ll_register_callbacks cb 0x55fcacc091c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T07:56:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.200+0000 7f0332d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd70740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.200+0000 7f0332d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd70740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41978/0 (socket says 172.21.15.98:41978) 2023-03-03T07:56:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.201+0000 7f0332d9d700 1 -- 172.21.15.98:0/572637255 learned_addr learned my addr 172.21.15.98:0/572637255 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.201+0000 7f0331d9b700 1 --2- 172.21.15.98:0/572637255 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd16cd0 0x55fcacd70200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.201+0000 7f033259c700 1 --2- 172.21.15.98:0/572637255 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd12560 0x55fcacd3dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.201+0000 7f0332d9d700 1 -- 172.21.15.98:0/572637255 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd12560 msgr2=0x55fcacd3dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:03.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.201+0000 7f0332d9d700 1 --2- 172.21.15.98:0/572637255 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd12560 0x55fcacd3dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.201+0000 7f0332d9d700 1 -- 172.21.15.98:0/572637255 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd16cd0 msgr2=0x55fcacd70200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:03.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.201+0000 7f0332d9d700 1 --2- 172.21.15.98:0/572637255 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd16cd0 0x55fcacd70200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.201+0000 7f0332d9d700 1 -- 172.21.15.98:0/572637255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f032c009d20 con 0x55fcacd17a70 2023-03-03T07:56:03.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.201+0000 7f0332d9d700 1 --2- 172.21.15.98:0/572637255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd70740 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f032c009790 tx=0x7f032c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae4a515d69211d7f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:03.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.201+0000 7f03117fa700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-03-03T07:56:03.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.202+0000 7f03117fa700 1 -- 172.21.15.98:0/572637255 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032c019040 con 0x55fcacd17a70 2023-03-03T07:56:03.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.202+0000 7f03117fa700 1 -- 172.21.15.98:0/572637255 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f032c003a20 con 0x55fcacd17a70 2023-03-03T07:56:03.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.202+0000 7f034c27eb80 1 -- 172.21.15.98:0/572637255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fcacd0ac50 con 0x55fcacd17a70 2023-03-03T07:56:03.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.202+0000 7f03117fa700 1 -- 172.21.15.98:0/572637255 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f032c009990 con 0x55fcacd17a70 2023-03-03T07:56:03.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.202+0000 7f034c27eb80 10 client.4566 fetch_fsmap learned FSMap version 7 2023-03-03T07:56:03.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.202+0000 7f034c27eb80 1 -- 172.21.15.98:0/572637255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55fcaca99b10 con 0x55fcacd17a70 2023-03-03T07:56:03.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.202+0000 7f03117fa700 1 -- 172.21.15.98:0/572637255 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55fcacd0ac50 con 0x55fcacd17a70 2023-03-03T07:56:03.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.202+0000 7f03117fa700 1 -- 172.21.15.98:0/572637255 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 7) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f032c003be0 con 0x55fcacd17a70 2023-03-03T07:56:03.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.202+0000 7f034c27eb80 10 client.4566 fetch_fsmap finished waiting for FSMap version 7 2023-03-03T07:56:03.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.202+0000 7f034c27eb80 10 client.4566 resolve_mds: resolved name 'c' to daemon mds.c 2023-03-03T07:56:03.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.202+0000 7f034c27eb80 20 client.4566 populate_metadata read hostname 'smithi098' 2023-03-03T07:56:03.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.202+0000 7f034c27eb80 1 --2- 172.21.15.98:0/572637255 >> [v2:172.21.15.98:6835/3969084152,v1:172.21.15.98:6837/3969084152] conn(0x55fcacd40900 0x55fcacd43590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.202+0000 7f034c27eb80 4 client.4566 mds_command: new command op to 4285 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-03T07:56:03.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.202+0000 7f034c27eb80 1 -- 172.21.15.98:0/572637255 --> [v2:172.21.15.98:6835/3969084152,v1:172.21.15.98:6837/3969084152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fcacc64020 con 0x55fcacd40900 2023-03-03T07:56:03.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.203+0000 7f033259c700 1 --2- 172.21.15.98:0/572637255 >> [v2:172.21.15.98:6835/3969084152,v1:172.21.15.98:6837/3969084152] conn(0x55fcacd40900 0x55fcacd43590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.203+0000 7f033259c700 1 --2- 172.21.15.98:0/572637255 >> [v2:172.21.15.98:6835/3969084152,v1:172.21.15.98:6837/3969084152] conn(0x55fcacd40900 0x55fcacd43590 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca462c77610d6104 server_cookie=988ccb8aa4dd5f34 in_seq=0 out_seq=0 2023-03-03T07:56:03.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.203+0000 7f03117fa700 10 client.4566 ms_handle_connect on v2:172.21.15.98:6835/3969084152 2023-03-03T07:56:03.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.204+0000 7f03117fa700 1 -- 172.21.15.98:0/572637255 <== mds.0 v2:172.21.15.98:6835/3969084152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55fcacc64020 con 0x55fcacd40900 2023-03-03T07:56:03.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.204+0000 7f03117fa700 10 client.4566 handle_command_reply: tid=0 2023-03-03T07:56:03.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.204+0000 7f034c27eb80 1 client.4566 shutdown 2023-03-03T07:56:03.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.204+0000 7f034c27eb80 1 -- 172.21.15.98:0/572637255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 msgr2=0x55fcacd70740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:03.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.204+0000 7f034c27eb80 1 --2- 172.21.15.98:0/572637255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd70740 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f032c009790 tx=0x7f032c004b40 comp rx=0 tx=0).stop 2023-03-03T07:56:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.204+0000 7f034c27eb80 1 -- 172.21.15.98:0/572637255 shutdown_connections 2023-03-03T07:56:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.204+0000 7f034c27eb80 1 --2- 172.21.15.98:0/572637255 >> [v2:172.21.15.98:6835/3969084152,v1:172.21.15.98:6837/3969084152] conn(0x55fcacd40900 0x55fcacd43590 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.204+0000 7f034c27eb80 1 --2- 172.21.15.98:0/572637255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd70740 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.204+0000 7f034c27eb80 1 --2- 172.21.15.98:0/572637255 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd12560 0x55fcacd3dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.204+0000 7f03117fa700 0 client.4566 ms_handle_reset on v2:172.21.15.98:6835/3969084152 2023-03-03T07:56:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.204+0000 7f034c27eb80 1 --2- 172.21.15.98:0/572637255 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd16cd0 0x55fcacd70200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.204+0000 7f034c27eb80 1 -- 172.21.15.98:0/572637255 >> 172.21.15.98:0/572637255 conn(0x55fcacd15c30 msgr2=0x55fcacd12bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:03.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.205+0000 7f034c27eb80 1 -- 172.21.15.98:0/572637255 shutdown_connections 2023-03-03T07:56:03.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.205+0000 7f034c27eb80 1 -- 172.21.15.98:0/572637255 wait complete. 2023-03-03T07:56:03.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.205+0000 7f034c27eb80 20 client.4566 trim_cache size 0 max 16384 2023-03-03T07:56:03.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.223+0000 7f034c27eb80 1 Processor -- start 2023-03-03T07:56:03.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.223+0000 7f034c27eb80 1 -- start start 2023-03-03T07:56:03.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.224+0000 7f034c27eb80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd12560 0x55fcacd56e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.224+0000 7f034c27eb80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd16cd0 0x55fcacd112c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.224+0000 7f034c27eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd3d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.224+0000 7f034c27eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcacd45050 con 0x55fcacd17a70 2023-03-03T07:56:03.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.224+0000 7f034c27eb80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcacd44530 con 0x55fcacd16cd0 2023-03-03T07:56:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.224+0000 7f034c27eb80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fcacd42e70 con 0x55fcacd12560 2023-03-03T07:56:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.224+0000 7f0332d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd3d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.224+0000 7f0331d9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd16cd0 0x55fcacd112c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.224+0000 7f033259c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd12560 0x55fcacd56e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.224+0000 7f0332d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd3d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41988/0 (socket says 172.21.15.98:41988) 2023-03-03T07:56:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.224+0000 7f0332d9d700 1 -- 172.21.15.98:0/19544645 learned_addr learned my addr 172.21.15.98:0/19544645 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.224+0000 7f0331d9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd16cd0 0x55fcacd112c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:47328/0 (socket says 172.21.15.98:47328) 2023-03-03T07:56:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.225+0000 7f0332d9d700 1 -- 172.21.15.98:0/19544645 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd12560 msgr2=0x55fcacd56e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.225+0000 7f0332d9d700 1 --2- 172.21.15.98:0/19544645 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd12560 0x55fcacd56e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.225+0000 7f0332d9d700 1 -- 172.21.15.98:0/19544645 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd16cd0 msgr2=0x55fcacd112c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.225+0000 7f0332d9d700 1 --2- 172.21.15.98:0/19544645 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd16cd0 0x55fcacd112c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.225+0000 7f0332d9d700 1 -- 172.21.15.98:0/19544645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fcacc32230 con 0x55fcacd17a70 2023-03-03T07:56:03.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.225+0000 7f0332d9d700 1 --2- 172.21.15.98:0/19544645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd3d380 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f032c000f80 tx=0x7f032c01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce227975ba21a065 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:03.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.225+0000 7f0320ff9700 1 -- 172.21.15.98:0/19544645 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032c015d30 con 0x55fcacd17a70 2023-03-03T07:56:03.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.225+0000 7f0320ff9700 1 -- 172.21.15.98:0/19544645 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032c009950 con 0x55fcacd17a70 2023-03-03T07:56:03.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.225+0000 7f0320ff9700 1 -- 172.21.15.98:0/19544645 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032c004900 con 0x55fcacd17a70 2023-03-03T07:56:03.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.226+0000 7f034c27eb80 1 -- 172.21.15.98:0/19544645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 msgr2=0x55fcacd3d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:03.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.226+0000 7f034c27eb80 1 --2- 172.21.15.98:0/19544645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd3d380 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f032c000f80 tx=0x7f032c01e000 comp rx=0 tx=0).stop 2023-03-03T07:56:03.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.226+0000 7f034c27eb80 1 -- 172.21.15.98:0/19544645 shutdown_connections 2023-03-03T07:56:03.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.226+0000 7f034c27eb80 1 --2- 172.21.15.98:0/19544645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd3d380 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.226+0000 7f034c27eb80 1 --2- 172.21.15.98:0/19544645 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd12560 0x55fcacd56e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.226+0000 7f034c27eb80 1 --2- 172.21.15.98:0/19544645 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd16cd0 0x55fcacd112c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.226+0000 7f034c27eb80 1 -- 172.21.15.98:0/19544645 >> 172.21.15.98:0/19544645 conn(0x55fcacd15c30 msgr2=0x55fcacd16000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:03.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.226+0000 7f034c27eb80 1 -- 172.21.15.98:0/19544645 shutdown_connections 2023-03-03T07:56:03.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.226+0000 7f034c27eb80 1 -- 172.21.15.98:0/19544645 wait complete. 2023-03-03T07:56:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.227+0000 7f034c27eb80 1 -- start start 2023-03-03T07:56:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.227+0000 7f034c27eb80 1 Processor -- start 2023-03-03T07:56:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.228+0000 7f034c27eb80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd12560 0x55fcacd70200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.228+0000 7f034c27eb80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd16cd0 0x55fcacd70740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.228+0000 7f034c27eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd6bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.228+0000 7f034c27eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcacdc5a90 con 0x55fcacd17a70 2023-03-03T07:56:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.228+0000 7f034c27eb80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcacd5d3a0 con 0x55fcacd12560 2023-03-03T07:56:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.228+0000 7f034c27eb80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fcacd43f30 con 0x55fcacd16cd0 2023-03-03T07:56:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.228+0000 7f034c27eb80 10 client.0 _ll_register_callbacks cb 0x55fcacc091c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T07:56:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.228+0000 7f0332d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd6bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.228+0000 7f0332d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd6bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41996/0 (socket says 172.21.15.98:41996) 2023-03-03T07:56:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.229+0000 7f0332d9d700 1 -- 172.21.15.98:0/2633586603 learned_addr learned my addr 172.21.15.98:0/2633586603 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.229+0000 7f033259c700 1 --2- 172.21.15.98:0/2633586603 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd12560 0x55fcacd70200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.229+0000 7f0331d9b700 1 --2- 172.21.15.98:0/2633586603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd16cd0 0x55fcacd70740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.229+0000 7f0332d9d700 1 -- 172.21.15.98:0/2633586603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd16cd0 msgr2=0x55fcacd70740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.229+0000 7f0332d9d700 1 --2- 172.21.15.98:0/2633586603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd16cd0 0x55fcacd70740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.229+0000 7f0332d9d700 1 -- 172.21.15.98:0/2633586603 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd12560 msgr2=0x55fcacd70200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.229+0000 7f0332d9d700 1 --2- 172.21.15.98:0/2633586603 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd12560 0x55fcacd70200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.229+0000 7f0332d9d700 1 -- 172.21.15.98:0/2633586603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fcaca99b10 con 0x55fcacd17a70 2023-03-03T07:56:03.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.229+0000 7f0332d9d700 1 --2- 172.21.15.98:0/2633586603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd6bb70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f032c000f80 tx=0x7f032c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1262cd38df93d1fd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:03.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.229+0000 7f03117fa700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-03-03T07:56:03.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.230+0000 7f03117fa700 1 -- 172.21.15.98:0/2633586603 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032c003c20 con 0x55fcacd17a70 2023-03-03T07:56:03.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.230+0000 7f03117fa700 1 -- 172.21.15.98:0/2633586603 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f032c047c10 con 0x55fcacd17a70 2023-03-03T07:56:03.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.230+0000 7f034c27eb80 1 -- 172.21.15.98:0/2633586603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f032c00f050 con 0x55fcacd17a70 2023-03-03T07:56:03.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.230+0000 7f03117fa700 1 -- 172.21.15.98:0/2633586603 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f032c047dd0 con 0x55fcacd17a70 2023-03-03T07:56:03.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.230+0000 7f034c27eb80 10 client.4572 fetch_fsmap learned FSMap version 7 2023-03-03T07:56:03.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.230+0000 7f034c27eb80 1 -- 172.21.15.98:0/2633586603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55fcaca82990 con 0x55fcacd17a70 2023-03-03T07:56:03.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.230+0000 7f03117fa700 1 -- 172.21.15.98:0/2633586603 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032c00f050 con 0x55fcacd17a70 2023-03-03T07:56:03.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.230+0000 7f03117fa700 1 -- 172.21.15.98:0/2633586603 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 7) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f032c044030 con 0x55fcacd17a70 2023-03-03T07:56:03.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.230+0000 7f034c27eb80 10 client.4572 fetch_fsmap finished waiting for FSMap version 7 2023-03-03T07:56:03.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.230+0000 7f034c27eb80 10 client.4572 resolve_mds: resolved name 'c' to daemon mds.c 2023-03-03T07:56:03.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.230+0000 7f034c27eb80 20 client.4572 populate_metadata read hostname 'smithi098' 2023-03-03T07:56:03.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.230+0000 7f034c27eb80 1 --2- 172.21.15.98:0/2633586603 >> [v2:172.21.15.98:6835/3969084152,v1:172.21.15.98:6837/3969084152] conn(0x55fcacdc52a0 0x55fcacd6c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.230+0000 7f034c27eb80 4 client.4572 mds_command: new command op to 4285 tid=0[{"prefix": "status"}] 2023-03-03T07:56:03.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.230+0000 7f034c27eb80 1 -- 172.21.15.98:0/2633586603 --> [v2:172.21.15.98:6835/3969084152,v1:172.21.15.98:6837/3969084152] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fcacabaca0 con 0x55fcacdc52a0 2023-03-03T07:56:03.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.231+0000 7f033259c700 1 --2- 172.21.15.98:0/2633586603 >> [v2:172.21.15.98:6835/3969084152,v1:172.21.15.98:6837/3969084152] conn(0x55fcacdc52a0 0x55fcacd6c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.231+0000 7f033259c700 1 --2- 172.21.15.98:0/2633586603 >> [v2:172.21.15.98:6835/3969084152,v1:172.21.15.98:6837/3969084152] conn(0x55fcacdc52a0 0x55fcacd6c0b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=691d71def3c9bb87 server_cookie=ce3e49ba7dbf2374 in_seq=0 out_seq=0 2023-03-03T07:56:03.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.231+0000 7f03117fa700 10 client.4572 ms_handle_connect on v2:172.21.15.98:6835/3969084152 2023-03-03T07:56:03.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.231+0000 7f03117fa700 1 -- 172.21.15.98:0/2633586603 <== mds.0 v2:172.21.15.98:6835/3969084152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55fcacabaca0 con 0x55fcacdc52a0 2023-03-03T07:56:03.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.231+0000 7f03117fa700 10 client.4572 handle_command_reply: tid=0 2023-03-03T07:56:03.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.231+0000 7f034c27eb80 1 client.4572 shutdown 2023-03-03T07:56:03.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.232+0000 7f034c27eb80 1 -- 172.21.15.98:0/2633586603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 msgr2=0x55fcacd6bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:03.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.232+0000 7f034c27eb80 1 --2- 172.21.15.98:0/2633586603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd6bb70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f032c000f80 tx=0x7f032c00b040 comp rx=0 tx=0).stop 2023-03-03T07:56:03.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.232+0000 7f034c27eb80 1 -- 172.21.15.98:0/2633586603 shutdown_connections 2023-03-03T07:56:03.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.232+0000 7f034c27eb80 1 --2- 172.21.15.98:0/2633586603 >> [v2:172.21.15.98:6835/3969084152,v1:172.21.15.98:6837/3969084152] conn(0x55fcacdc52a0 0x55fcacd6c0b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.232+0000 7f034c27eb80 1 --2- 172.21.15.98:0/2633586603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcacd17a70 0x55fcacd6bb70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.232+0000 7f034c27eb80 1 --2- 172.21.15.98:0/2633586603 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fcacd12560 0x55fcacd70200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.232+0000 7f03117fa700 0 client.4572 ms_handle_reset on v2:172.21.15.98:6835/3969084152 2023-03-03T07:56:03.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.232+0000 7f034c27eb80 1 --2- 172.21.15.98:0/2633586603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fcacd16cd0 0x55fcacd70740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.232+0000 7f034c27eb80 1 -- 172.21.15.98:0/2633586603 >> 172.21.15.98:0/2633586603 conn(0x55fcacd15c30 msgr2=0x55fcacd112c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:03.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.232+0000 7f034c27eb80 1 -- 172.21.15.98:0/2633586603 shutdown_connections 2023-03-03T07:56:03.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.232+0000 7f034c27eb80 1 -- 172.21.15.98:0/2633586603 wait complete. 2023-03-03T07:56:03.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.232+0000 7f034c27eb80 20 client.4572 trim_cache size 0 max 16384 2023-03-03T07:56:03.247 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T07:56:03.247 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "fbbd322f-236a-44f8-96c5-be6963846c9b", 2023-03-03T07:56:03.247 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2023-03-03T07:56:03.247 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4285, 2023-03-03T07:56:03.247 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-03-03T07:56:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-03-03T07:56:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-03-03T07:56:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 16.130851545999999, 2023-03-03T07:56:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 7, 2023-03-03T07:56:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 24, 2023-03-03T07:56:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 24, 2023-03-03T07:56:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 22.490303214000001 2023-03-03T07:56:03.249 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T07:56:03.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.233+0000 7f03389ff700 1 -- 172.21.15.98:0/387093643 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f03140682f0 msgr2=0x7f031406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:03.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.233+0000 7f03389ff700 1 --2- 172.21.15.98:0/387093643 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f03140682f0 0x7f031406a7a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f03340ca1d0 tx=0x7f0328008030 comp rx=0 tx=0).stop 2023-03-03T07:56:03.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.233+0000 7f03389ff700 1 -- 172.21.15.98:0/387093643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f033406b4b0 msgr2=0x7f03340ca250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:03.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.233+0000 7f03389ff700 1 --2- 172.21.15.98:0/387093643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f033406b4b0 0x7f03340ca250 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f032400c700 tx=0x7f032400cac0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.233+0000 7f03389ff700 1 -- 172.21.15.98:0/387093643 shutdown_connections 2023-03-03T07:56:03.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.233+0000 7f03389ff700 1 --2- 172.21.15.98:0/387093643 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f03140682f0 0x7f031406a7a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.233+0000 7f03389ff700 1 --2- 172.21.15.98:0/387093643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f033406b4b0 0x7f03340ca250 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.233+0000 7f03389ff700 1 --2- 172.21.15.98:0/387093643 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f033415ca70 0x7f03340b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.233+0000 7f03389ff700 1 --2- 172.21.15.98:0/387093643 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03340d0270 0x7f033414a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.233+0000 7f03389ff700 1 -- 172.21.15.98:0/387093643 >> 172.21.15.98:0/387093643 conn(0x7f0334073e70 msgr2=0x7f03340b5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:03.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.233+0000 7f03389ff700 1 -- 172.21.15.98:0/387093643 shutdown_connections 2023-03-03T07:56:03.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.233+0000 7f03389ff700 1 -- 172.21.15.98:0/387093643 wait complete. 2023-03-03T07:56:03.260 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-03-03T07:56:03.316 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-03-03T07:56:03.317 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-03-03T07:56:03.317 INFO:tasks.ceph_fuse:client.0 config is {} 2023-03-03T07:56:03.317 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-03-03T07:56:03.318 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-03-03T07:56:03.318 INFO:tasks.ceph_fuse:client.2 config is {} 2023-03-03T07:56:03.318 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-03-03T07:56:03.319 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-03-03T07:56:03.319 INFO:tasks.ceph_fuse:client.1 config is {} 2023-03-03T07:56:03.319 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-03-03T07:56:03.319 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-03-03T07:56:03.319 INFO:tasks.ceph_fuse:client.3 config is {} 2023-03-03T07:56:03.320 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-03-03T07:56:03.320 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-03-03T07:56:03.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:03.321 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2023-03-03T07:56:03.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:03.396 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link delete ceph-brx 2023-03-03T07:56:03.502 INFO:teuthology.orchestra.run.smithi100.stderr:Cannot find device "ceph-brx" 2023-03-03T07:56:03.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T07:56:03.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:03.508 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-03-03T07:56:03.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:03.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2023-03-03T07:56:03.681 INFO:teuthology.orchestra.run.smithi098.stderr:Cannot find device "ceph-brx" 2023-03-03T07:56:03.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T07:56:03.686 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-03-03T07:56:03.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T07:56:03.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.929+0000 7fe710b9d700 1 Processor -- start 2023-03-03T07:56:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.929+0000 7fe710b9d700 1 -- start start 2023-03-03T07:56:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.930+0000 7fe710b9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe70c160f80 0x7fe70c0d1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.930+0000 7fe710b9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70c0d10f0 con 0x7fe70c146360 2023-03-03T07:56:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.930+0000 7fe710b9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70c0cdf20 con 0x7fe70c0b51c0 2023-03-03T07:56:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.930+0000 7fe710b9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe70c0cd680 con 0x7fe70c160f80 2023-03-03T07:56:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.931+0000 7fe70b7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe70c146360 0x7fe70c0b4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53748/0 (socket says 172.21.15.98:53748) 2023-03-03T07:56:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.931+0000 7fe70b7fe700 1 -- 172.21.15.98:0/823226152 learned_addr learned my addr 172.21.15.98:0/823226152 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:03.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.931+0000 7fe70bfff700 1 --2- 172.21.15.98:0/823226152 >> v2:172.21.15.100:3300/0 conn(0x7fe70c160f80 0x7fe70c0d1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.931+0000 7fe70a7fc700 1 -- 172.21.15.98:0/823226152 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798293067 0 0) 0x7fe70c0d10f0 con 0x7fe70c146360 2023-03-03T07:56:03.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.931+0000 7fe70a7fc700 1 -- 172.21.15.98:0/823226152 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6f40035b0 con 0x7fe70c146360 2023-03-03T07:56:03.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.931+0000 7fe70a7fc700 1 -- 172.21.15.98:0/823226152 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853479715 0 0) 0x7fe70c0cdf20 con 0x7fe70c0b51c0 2023-03-03T07:56:03.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.931+0000 7fe70a7fc700 1 -- 172.21.15.98:0/823226152 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe70c0d10f0 con 0x7fe70c0b51c0 2023-03-03T07:56:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.931+0000 7fe70bfff700 1 -- 172.21.15.98:0/823226152 >> v1:172.21.15.100:6789/0 conn(0x7fe70c0b51c0 legacy=0x7fe70c0d1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.931+0000 7fe70bfff700 1 -- 172.21.15.98:0/823226152 >> v1:172.21.15.98:6789/0 conn(0x7fe70c146360 legacy=0x7fe70c0b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.931+0000 7fe70bfff700 1 -- 172.21.15.98:0/823226152 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c0cd960 con 0x7fe70c160f80 2023-03-03T07:56:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.932+0000 7fe70bfff700 1 --2- 172.21.15.98:0/823226152 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe70c160f80 0x7fe70c0d1c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe704009d90 tx=0x7fe70400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b651fd3ca84c32d7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.932+0000 7fe70a7fc700 1 -- 172.21.15.98:0/823226152 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70400e070 con 0x7fe70c160f80 2023-03-03T07:56:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.932+0000 7fe70a7fc700 1 -- 172.21.15.98:0/823226152 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70400bb20 con 0x7fe70c160f80 2023-03-03T07:56:03.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.932+0000 7fe70a7fc700 1 -- 172.21.15.98:0/823226152 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe704012430 con 0x7fe70c160f80 2023-03-03T07:56:03.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.932+0000 7fe710b9d700 1 -- 172.21.15.98:0/823226152 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe70c160f80 msgr2=0x7fe70c0d1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:03.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.932+0000 7fe710b9d700 1 --2- 172.21.15.98:0/823226152 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe70c160f80 0x7fe70c0d1c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe704009d90 tx=0x7fe70400b330 comp rx=0 tx=0).stop 2023-03-03T07:56:03.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.933+0000 7fe710b9d700 1 -- 172.21.15.98:0/823226152 shutdown_connections 2023-03-03T07:56:03.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.933+0000 7fe710b9d700 1 --2- 172.21.15.98:0/823226152 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe70c160f80 0x7fe70c0d1c90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.933+0000 7fe710b9d700 1 -- 172.21.15.98:0/823226152 >> 172.21.15.98:0/823226152 conn(0x7fe70c06b8e0 msgr2=0x7fe70c0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:03.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.933+0000 7fe710b9d700 1 -- 172.21.15.98:0/823226152 shutdown_connections 2023-03-03T07:56:03.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.933+0000 7fe710b9d700 1 -- 172.21.15.98:0/823226152 wait complete. 2023-03-03T07:56:03.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.935+0000 7fe710b9d700 1 Processor -- start 2023-03-03T07:56:03.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.936+0000 7fe710b9d700 1 -- start start 2023-03-03T07:56:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.937+0000 7fe710b9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b51c0 0x7fe70c156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.937+0000 7fe710b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe70c146360 0x7fe70c1f2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.937+0000 7fe710b9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe70c160f80 0x7fe70c0b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.937+0000 7fe710b9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c15ff90 con 0x7fe70c0b51c0 2023-03-03T07:56:03.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.937+0000 7fe710b9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c15d680 con 0x7fe70c160f80 2023-03-03T07:56:03.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.937+0000 7fe710b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c156fe0 con 0x7fe70c146360 2023-03-03T07:56:03.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.937+0000 7fe70b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b51c0 0x7fe70c156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.938+0000 7fe70b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b51c0 0x7fe70c156aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42002/0 (socket says 172.21.15.98:42002) 2023-03-03T07:56:03.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.938+0000 7fe70b7fe700 1 -- 172.21.15.98:0/44859855 learned_addr learned my addr 172.21.15.98:0/44859855 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:03.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.938+0000 7fe70affd700 1 --2- 172.21.15.98:0/44859855 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe70c146360 0x7fe70c1f2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.938+0000 7fe70bfff700 1 --2- 172.21.15.98:0/44859855 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe70c160f80 0x7fe70c0b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.938+0000 7fe70b7fe700 1 -- 172.21.15.98:0/44859855 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe70c146360 msgr2=0x7fe70c1f2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:03.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.938+0000 7fe70b7fe700 1 --2- 172.21.15.98:0/44859855 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe70c146360 0x7fe70c1f2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.938+0000 7fe70b7fe700 1 -- 172.21.15.98:0/44859855 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe70c160f80 msgr2=0x7fe70c0b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:03.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.938+0000 7fe70b7fe700 1 --2- 172.21.15.98:0/44859855 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe70c160f80 0x7fe70c0b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:03.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.938+0000 7fe70b7fe700 1 -- 172.21.15.98:0/44859855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe704009a70 con 0x7fe70c0b51c0 2023-03-03T07:56:03.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.938+0000 7fe70b7fe700 1 --2- 172.21.15.98:0/44859855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b51c0 0x7fe70c156aa0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00c730 tx=0x7fe6fc00ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:03.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.938+0000 7fe6f3fff700 1 -- 172.21.15.98:0/44859855 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc005d70 con 0x7fe70c0b51c0 2023-03-03T07:56:03.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.939+0000 7fe6f3fff700 1 -- 172.21.15.98:0/44859855 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6fc0070d0 con 0x7fe70c0b51c0 2023-03-03T07:56:03.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.939+0000 7fe6f3fff700 1 -- 172.21.15.98:0/44859855 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc007440 con 0x7fe70c0b51c0 2023-03-03T07:56:03.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.939+0000 7fe710b9d700 1 -- 172.21.15.98:0/44859855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe70c0d0d80 con 0x7fe70c0b51c0 2023-03-03T07:56:03.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.939+0000 7fe710b9d700 1 -- 172.21.15.98:0/44859855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe70c0cdc40 con 0x7fe70c0b51c0 2023-03-03T07:56:03.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.940+0000 7fe710b9d700 1 -- 172.21.15.98:0/44859855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe70c161380 con 0x7fe70c0b51c0 2023-03-03T07:56:03.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.941+0000 7fe6f3fff700 1 -- 172.21.15.98:0/44859855 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe6fc019030 con 0x7fe70c0b51c0 2023-03-03T07:56:03.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.942+0000 7fe6f3fff700 1 --2- 172.21.15.98:0/44859855 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6f80682c0 0x7fe6f806a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:03.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.943+0000 7fe6f3fff700 1 -- 172.21.15.98:0/44859855 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe6fc084fc0 con 0x7fe70c0b51c0 2023-03-03T07:56:03.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.943+0000 7fe70affd700 1 --2- 172.21.15.98:0/44859855 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6f80682c0 0x7fe6f806a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:03.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.943+0000 7fe70affd700 1 --2- 172.21.15.98:0/44859855 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6f80682c0 0x7fe6f806a770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe70c1f5a30 tx=0x7fe700009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:03.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:03.947+0000 7fe6f3fff700 1 -- 172.21.15.98:0/44859855 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe6fc04b5e0 con 0x7fe70c0b51c0 2023-03-03T07:56:04.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:04.181+0000 7fe710b9d700 1 -- 172.21.15.98:0/44859855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe70c068300 con 0x7fe70c0b51c0 2023-03-03T07:56:04.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:04.181+0000 7fe6f3fff700 1 -- 172.21.15.98:0/44859855 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe6fc017070 con 0x7fe70c0b51c0 2023-03-03T07:56:04.183 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T07:56:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:04.184+0000 7fe710b9d700 1 -- 172.21.15.98:0/44859855 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6f80682c0 msgr2=0x7fe6f806a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:04.184+0000 7fe710b9d700 1 --2- 172.21.15.98:0/44859855 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6f80682c0 0x7fe6f806a770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe70c1f5a30 tx=0x7fe700009040 comp rx=0 tx=0).stop 2023-03-03T07:56:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:04.184+0000 7fe710b9d700 1 -- 172.21.15.98:0/44859855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b51c0 msgr2=0x7fe70c156aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:04.184+0000 7fe710b9d700 1 --2- 172.21.15.98:0/44859855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b51c0 0x7fe70c156aa0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00c730 tx=0x7fe6fc00ca40 comp rx=0 tx=0).stop 2023-03-03T07:56:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:04.184+0000 7fe710b9d700 1 -- 172.21.15.98:0/44859855 shutdown_connections 2023-03-03T07:56:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:04.184+0000 7fe710b9d700 1 --2- 172.21.15.98:0/44859855 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6f80682c0 0x7fe6f806a770 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:04.184+0000 7fe710b9d700 1 --2- 172.21.15.98:0/44859855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b51c0 0x7fe70c156aa0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:04.184+0000 7fe710b9d700 1 --2- 172.21.15.98:0/44859855 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe70c160f80 0x7fe70c0b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:04.184+0000 7fe710b9d700 1 --2- 172.21.15.98:0/44859855 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe70c146360 0x7fe70c1f2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:04.184+0000 7fe710b9d700 1 -- 172.21.15.98:0/44859855 >> 172.21.15.98:0/44859855 conn(0x7fe70c06b8e0 msgr2=0x7fe70c0b7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:04.184+0000 7fe710b9d700 1 -- 172.21.15.98:0/44859855 shutdown_connections 2023-03-03T07:56:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:04.184+0000 7fe710b9d700 1 -- 172.21.15.98:0/44859855 wait complete. 2023-03-03T07:56:04.214 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T07:56:04.214 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T07:56:04.215 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2023-03-03T07:56:04.215 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-03-03T07:56:04.215 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-03-03T07:56:04.215 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T07:56:04.215 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-03-03T07:56:04.215 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T07:56:04.216 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T07:56:04.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:04.217 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2023-03-03T07:56:04.285 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T07:56:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T07:56:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T07:56:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-03-03T07:56:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:04.290 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:56:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2023-03-03T07:56:04.291 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:56:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2023-03-03T07:56:04.292 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:56:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:04.292 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T07:56:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T07:56:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42192sec preferred_lft 42192sec 2023-03-03T07:56:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2023-03-03T07:56:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:04.295 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-03-03T07:56:04.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:04.296 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-03-03T07:56:04.296 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-03-03T07:56:04.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2023-03-03T07:56:04.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2023-03-03T07:56:04.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2023-03-03T07:56:04.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-03-03T07:56:04.297 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-03-03T07:56:04.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:04.817 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T07:56:04.964 INFO:teuthology.orchestra.run.smithi098.stdout:1 2023-03-03T07:56:04.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:04.969 DEBUG:teuthology.orchestra.run.smithi098:> route 2023-03-03T07:56:05.060 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2023-03-03T07:56:05.060 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T07:56:05.060 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T07:56:05.060 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T07:56:05.060 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T07:56:05.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:05.062 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-03-03T07:56:05.062 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-03-03T07:56:05.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T07:56:05.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T07:56:05.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T07:56:05.063 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-03-03T07:56:05.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:05.829 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-03-03T07:56:05.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:05.900 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2023-03-03T07:56:05.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:05.973 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-03-03T07:56:05.973 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-03-03T07:56:05.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-03-03T07:56:05.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-03-03T07:56:05.974 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-03-03T07:56:06.179 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-03-03T07:56:06.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:06.180 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-03-03T07:56:06.181 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-03-03T07:56:06.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-03-03T07:56:06.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-03-03T07:56:06.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-03-03T07:56:06.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-03-03T07:56:06.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-03-03T07:56:06.182 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-03-03T07:56:06.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:06.556 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-03-03T07:56:06.556 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-03-03T07:56:06.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2023-03-03T07:56:06.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2023-03-03T07:56:06.556 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-03-03T07:56:06.791 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-03-03T07:56:06.792 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:56:06.792 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-03-03T07:56:06.860 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-03-03T07:56:06.861 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:56:06.862 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-03-03T07:56:06.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T07:56:07.030 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:56:07.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:56:07.128 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:56:07.132 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:56:07.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:07.133 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T07:56:07.202 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-03-03T07:56:07.202 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/) 2023-03-03T07:56:07.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T07:56:07.313 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:56:07.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:56:07.418 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:56:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:56:07.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:07.426 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T07:56:07.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:2023-03-03T07:56:07.436+0000 7f3b91dfa5c0 -1 init, newargv = 0x563417abcd40 newargc=15 2023-03-03T07:56:07.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[118098]: starting ceph client 2023-03-03T07:56:07.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[118098]: starting fuse 2023-03-03T07:56:07.499 INFO:teuthology.orchestra.run.smithi098.stdout:42 2023-03-03T07:56:07.500 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2023-03-03T07:56:07.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T07:56:07.501 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2023-03-03T07:56:07.501 DEBUG:teuthology.orchestra.run.smithi098:> import re 2023-03-03T07:56:07.501 DEBUG:teuthology.orchestra.run.smithi098:> import os 2023-03-03T07:56:07.501 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2023-03-03T07:56:07.502 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:56:07.502 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2023-03-03T07:56:07.502 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-03-03T07:56:07.502 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2023-03-03T07:56:07.503 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-03-03T07:56:07.503 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:56:07.503 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2023-03-03T07:56:07.503 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2023-03-03T07:56:07.504 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2023-03-03T07:56:07.504 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2023-03-03T07:56:07.504 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:56:07.504 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2023-03-03T07:56:07.504 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T07:56:07.505 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T07:56:07.505 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T07:56:07.505 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2023-03-03T07:56:07.505 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2023-03-03T07:56:07.506 DEBUG:teuthology.orchestra.run.smithi098:> return f 2023-03-03T07:56:07.506 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T07:56:07.506 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:56:07.506 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.0")) 2023-03-03T07:56:07.507 DEBUG:teuthology.orchestra.run.smithi098:> ' 2023-03-03T07:56:07.724 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.0.118098.asok 2023-03-03T07:56:07.901 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.118098.asok 2023-03-03T07:56:07.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:07.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.118098.asok status 2023-03-03T07:56:08.169 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T07:56:08.170 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2023-03-03T07:56:08.170 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T07:56:08.170 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T07:56:08.170 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "0", 2023-03-03T07:56:08.170 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2023-03-03T07:56:08.170 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-03-03T07:56:08.170 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "118098", 2023-03-03T07:56:08.171 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2023-03-03T07:56:08.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:56:08.171 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2023-03-03T07:56:08.171 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2023-03-03T07:56:08.171 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4587, 2023-03-03T07:56:08.171 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2023-03-03T07:56:08.171 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2023-03-03T07:56:08.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2023-03-03T07:56:08.172 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4587 2023-03-03T07:56:08.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:56:08.172 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T07:56:08.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T07:56:08.172 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-03-03T07:56:08.172 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 608951434 2023-03-03T07:56:08.172 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:56:08.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:56:08.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T07:56:08.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T07:56:08.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-03-03T07:56:08.173 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 608951434 2023-03-03T07:56:08.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:56:08.174 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.4587 192.168.0.1:0/608951434", 2023-03-03T07:56:08.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/608951434", 2023-03-03T07:56:08.174 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2023-03-03T07:56:08.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 7, 2023-03-03T07:56:08.174 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 24, 2023-03-03T07:56:08.174 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2023-03-03T07:56:08.174 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2023-03-03T07:56:08.174 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2023-03-03T07:56:08.175 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T07:56:08.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T07:56:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.435+0000 7f2b87fff700 1 Processor -- start 2023-03-03T07:56:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.436+0000 7f2b87fff700 1 -- start start 2023-03-03T07:56:08.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.437+0000 7f2b87fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b880b51c0 0x7f2b88160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:08.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.437+0000 7f2b87fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b881614c0 con 0x7f2b880d1580 2023-03-03T07:56:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.437+0000 7f2b87fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b880d10f0 con 0x7f2b88146360 2023-03-03T07:56:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.437+0000 7f2b87fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880cdf20 con 0x7f2b880b51c0 2023-03-03T07:56:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.437+0000 7f2b877fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2b880d1580 0x7f2b880d19e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57124/0 (socket says 172.21.15.98:57124) 2023-03-03T07:56:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.437+0000 7f2b877fe700 1 -- 172.21.15.98:0/3399666645 learned_addr learned my addr 172.21.15.98:0/3399666645 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.438+0000 7f2b867fc700 1 --2- 172.21.15.98:0/3399666645 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b880b51c0 0x7f2b88160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.438+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462359934 0 0) 0x7f2b80001490 con 0x7f2b880d1580 2023-03-03T07:56:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.438+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b700035b0 con 0x7f2b880d1580 2023-03-03T07:56:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.438+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780397761 0 0) 0x7f2b880d10f0 con 0x7f2b88146360 2023-03-03T07:56:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.438+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b80001490 con 0x7f2b88146360 2023-03-03T07:56:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.438+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1542331258 0 0) 0x7f2b700035b0 con 0x7f2b880d1580 2023-03-03T07:56:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.438+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b880b51c0 msgr2=0x7f2b88160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.438+0000 7f2b85ffb700 1 --2- 172.21.15.98:0/3399666645 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b880b51c0 0x7f2b88160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.438+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 >> v1:172.21.15.100:6789/0 conn(0x7f2b88146360 legacy=0x7f2b880b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.438+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b880cd680 con 0x7f2b880d1580 2023-03-03T07:56:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.438+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f2b80001b80 con 0x7f2b880d1580 2023-03-03T07:56:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.438+0000 7f2b867fc700 1 --2- 172.21.15.98:0/3399666645 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b880b51c0 0x7f2b88160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:56:08.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.438+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 >> v1:172.21.15.98:6789/0 conn(0x7f2b880d1580 legacy=0x7f2b880d19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:08.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.439+0000 7f2b85ffb700 1 --2- 172.21.15.98:0/3399666645 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b880b51c0 0x7f2b70003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:08.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.439+0000 7f2b85ffb700 1 --2- 172.21.15.98:0/3399666645 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b70004150 0x7f2b70004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:08.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.439+0000 7f2b85ffb700 1 --2- 172.21.15.98:0/3399666645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b70004d90 0x7f2b70005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:08.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.439+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880cdf20 con 0x7f2b70004d90 2023-03-03T07:56:08.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.439+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7000bba0 con 0x7f2b70004150 2023-03-03T07:56:08.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.439+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7000bd60 con 0x7f2b880b51c0 2023-03-03T07:56:08.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.439+0000 7f2b867fc700 1 --2- 172.21.15.98:0/3399666645 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b70004150 0x7f2b70004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:08.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.439+0000 7f2b86ffd700 1 --2- 172.21.15.98:0/3399666645 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b880b51c0 0x7f2b70003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:08.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.439+0000 7f2b877fe700 1 --2- 172.21.15.98:0/3399666645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b70004d90 0x7f2b70005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:08.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.439+0000 7f2b867fc700 1 -- 172.21.15.98:0/3399666645 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b880b51c0 msgr2=0x7f2b70003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:08.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.439+0000 7f2b867fc700 1 --2- 172.21.15.98:0/3399666645 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b880b51c0 0x7f2b70003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:08.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.439+0000 7f2b867fc700 1 -- 172.21.15.98:0/3399666645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b70004d90 msgr2=0x7f2b70005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:08.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.439+0000 7f2b867fc700 1 --2- 172.21.15.98:0/3399666645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b70004d90 0x7f2b70005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:08.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.439+0000 7f2b867fc700 1 -- 172.21.15.98:0/3399666645 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c0094a0 con 0x7f2b70004150 2023-03-03T07:56:08.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.440+0000 7f2b867fc700 1 --2- 172.21.15.98:0/3399666645 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b70004150 0x7f2b70004850 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009ce0 tx=0x7f2b70005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=368453b8f91899b2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:08.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.440+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c004030 con 0x7f2b70004150 2023-03-03T07:56:08.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.440+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7c004190 con 0x7f2b70004150 2023-03-03T07:56:08.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.440+0000 7f2b85ffb700 1 -- 172.21.15.98:0/3399666645 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c004540 con 0x7f2b70004150 2023-03-03T07:56:08.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.440+0000 7f2b87fff700 1 -- 172.21.15.98:0/3399666645 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b70004150 msgr2=0x7f2b70004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:08.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.440+0000 7f2b87fff700 1 --2- 172.21.15.98:0/3399666645 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b70004150 0x7f2b70004850 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009ce0 tx=0x7f2b70005880 comp rx=0 tx=0).stop 2023-03-03T07:56:08.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.441+0000 7f2b877fe700 1 -- 172.21.15.98:0/3399666645 reap_dead start 2023-03-03T07:56:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.441+0000 7f2b87fff700 1 -- 172.21.15.98:0/3399666645 shutdown_connections 2023-03-03T07:56:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.441+0000 7f2b87fff700 1 -- 172.21.15.98:0/3399666645 >> 172.21.15.98:0/3399666645 conn(0x7f2b8806b8e0 msgr2=0x7f2b880cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.441+0000 7f2b87fff700 1 -- 172.21.15.98:0/3399666645 shutdown_connections 2023-03-03T07:56:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.441+0000 7f2b87fff700 1 -- 172.21.15.98:0/3399666645 wait complete. 2023-03-03T07:56:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.443+0000 7f2b87fff700 1 Processor -- start 2023-03-03T07:56:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.444+0000 7f2b87fff700 1 -- start start 2023-03-03T07:56:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.445+0000 7f2b87fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b70004150 0x7f2b88157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.445+0000 7f2b87fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b880cab10 0x7f2b880b8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.445+0000 7f2b87fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880caf80 0x7f2b880b8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.445+0000 7f2b87fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880c6d00 con 0x7f2b880caf80 2023-03-03T07:56:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.445+0000 7f2b87fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880c6fe0 con 0x7f2b880cab10 2023-03-03T07:56:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.445+0000 7f2b87fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880c72c0 con 0x7f2b70004150 2023-03-03T07:56:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.445+0000 7f2b877fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880caf80 0x7f2b880b8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.445+0000 7f2b877fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880caf80 0x7f2b880b8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48908/0 (socket says 172.21.15.98:48908) 2023-03-03T07:56:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.445+0000 7f2b86ffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b70004150 0x7f2b88157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.445+0000 7f2b877fe700 1 -- 172.21.15.98:0/3352753470 learned_addr learned my addr 172.21.15.98:0/3352753470 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.445+0000 7f2b867fc700 1 --2- 172.21.15.98:0/3352753470 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b880cab10 0x7f2b880b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:08.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.446+0000 7f2b877fe700 1 -- 172.21.15.98:0/3352753470 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b70004150 msgr2=0x7f2b88157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:08.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.446+0000 7f2b877fe700 1 --2- 172.21.15.98:0/3352753470 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b70004150 0x7f2b88157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:08.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.446+0000 7f2b877fe700 1 -- 172.21.15.98:0/3352753470 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b880cab10 msgr2=0x7f2b880b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.446+0000 7f2b877fe700 1 --2- 172.21.15.98:0/3352753470 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b880cab10 0x7f2b880b8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.446+0000 7f2b877fe700 1 -- 172.21.15.98:0/3352753470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b880cdc40 con 0x7f2b880caf80 2023-03-03T07:56:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.446+0000 7f2b877fe700 1 --2- 172.21.15.98:0/3352753470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880caf80 0x7f2b880b8590 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2b880743c0 tx=0x7f2b80007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:08.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.446+0000 7f2b6ffff700 1 -- 172.21.15.98:0/3352753470 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b80007a00 con 0x7f2b880caf80 2023-03-03T07:56:08.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.446+0000 7f2b6ffff700 1 -- 172.21.15.98:0/3352753470 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b80007b80 con 0x7f2b880caf80 2023-03-03T07:56:08.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.446+0000 7f2b6ffff700 1 -- 172.21.15.98:0/3352753470 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b80013630 con 0x7f2b880caf80 2023-03-03T07:56:08.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.446+0000 7f2b87fff700 1 -- 172.21.15.98:0/3352753470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b880cd960 con 0x7f2b880caf80 2023-03-03T07:56:08.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.446+0000 7f2b87fff700 1 -- 172.21.15.98:0/3352753470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b880d0d80 con 0x7f2b880caf80 2023-03-03T07:56:08.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.448+0000 7f2b87fff700 1 -- 172.21.15.98:0/3352753470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b880cecc0 con 0x7f2b880caf80 2023-03-03T07:56:08.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.449+0000 7f2b6ffff700 1 -- 172.21.15.98:0/3352753470 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2b80013790 con 0x7f2b880caf80 2023-03-03T07:56:08.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.450+0000 7f2b6ffff700 1 --2- 172.21.15.98:0/3352753470 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2b740682f0 0x7f2b7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:08.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.450+0000 7f2b6ffff700 1 -- 172.21.15.98:0/3352753470 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2b800864c0 con 0x7f2b880caf80 2023-03-03T07:56:08.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.450+0000 7f2b86ffd700 1 --2- 172.21.15.98:0/3352753470 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2b740682f0 0x7f2b7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:08.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.451+0000 7f2b86ffd700 1 --2- 172.21.15.98:0/3352753470 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2b740682f0 0x7f2b7406a7a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b78007fd0 tx=0x7f2b78009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:08.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.454+0000 7f2b6ffff700 1 -- 172.21.15.98:0/3352753470 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2b80052c30 con 0x7f2b880caf80 2023-03-03T07:56:08.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.691+0000 7f2b87fff700 1 -- 172.21.15.98:0/3352753470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2b881614c0 con 0x7f2b880caf80 2023-03-03T07:56:08.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.691+0000 7f2b6ffff700 1 -- 172.21.15.98:0/3352753470 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2b881614c0 con 0x7f2b880caf80 2023-03-03T07:56:08.693 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T07:56:08.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.694+0000 7f2b87fff700 1 -- 172.21.15.98:0/3352753470 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2b740682f0 msgr2=0x7f2b7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:08.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.694+0000 7f2b87fff700 1 --2- 172.21.15.98:0/3352753470 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2b740682f0 0x7f2b7406a7a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b78007fd0 tx=0x7f2b78009040 comp rx=0 tx=0).stop 2023-03-03T07:56:08.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.694+0000 7f2b87fff700 1 -- 172.21.15.98:0/3352753470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880caf80 msgr2=0x7f2b880b8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:08.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.694+0000 7f2b87fff700 1 --2- 172.21.15.98:0/3352753470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880caf80 0x7f2b880b8590 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2b880743c0 tx=0x7f2b80007340 comp rx=0 tx=0).stop 2023-03-03T07:56:08.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.694+0000 7f2b87fff700 1 -- 172.21.15.98:0/3352753470 shutdown_connections 2023-03-03T07:56:08.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.694+0000 7f2b87fff700 1 --2- 172.21.15.98:0/3352753470 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2b740682f0 0x7f2b7406a7a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:08.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.694+0000 7f2b87fff700 1 --2- 172.21.15.98:0/3352753470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880caf80 0x7f2b880b8590 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:08.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.694+0000 7f2b87fff700 1 --2- 172.21.15.98:0/3352753470 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b70004150 0x7f2b88157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:08.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.694+0000 7f2b87fff700 1 --2- 172.21.15.98:0/3352753470 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b880cab10 0x7f2b880b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:08.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.694+0000 7f2b87fff700 1 -- 172.21.15.98:0/3352753470 >> 172.21.15.98:0/3352753470 conn(0x7f2b8806b8e0 msgr2=0x7f2b8815b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:08.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.694+0000 7f2b87fff700 1 -- 172.21.15.98:0/3352753470 shutdown_connections 2023-03-03T07:56:08.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:08.694+0000 7f2b87fff700 1 -- 172.21.15.98:0/3352753470 wait complete. 2023-03-03T07:56:08.722 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T07:56:08.723 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T07:56:08.723 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2023-03-03T07:56:08.723 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-03-03T07:56:08.723 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-03-03T07:56:08.724 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T07:56:08.724 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-03-03T07:56:08.724 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T07:56:08.724 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-03-03T07:56:08.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:08.725 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2023-03-03T07:56:08.792 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T07:56:08.793 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T07:56:08.793 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T07:56:08.793 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:08.794 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-03-03T07:56:08.794 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:08.794 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:56:08.794 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:08.794 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2023-03-03T07:56:08.795 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:56:08.795 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:08.795 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2023-03-03T07:56:08.795 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:56:08.795 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:08.795 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T07:56:08.795 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:08.796 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T07:56:08.796 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42187sec preferred_lft 42187sec 2023-03-03T07:56:08.796 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2023-03-03T07:56:08.796 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:08.796 INFO:teuthology.orchestra.run.smithi098.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T07:56:08.796 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 46:3f:9e:55:37:22 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:08.796 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-03-03T07:56:08.797 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:08.797 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::1857:c2ff:fe19:346f/64 scope link 2023-03-03T07:56:08.797 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:08.797 INFO:teuthology.orchestra.run.smithi098.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T07:56:08.797 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 46:3f:9e:55:37:22 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-03-03T07:56:08.797 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::443f:9eff:fe55:3722/64 scope link 2023-03-03T07:56:08.797 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:08.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:08.799 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T07:56:08.894 INFO:teuthology.orchestra.run.smithi098.stdout:1 2023-03-03T07:56:08.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:08.900 DEBUG:teuthology.orchestra.run.smithi098:> route 2023-03-03T07:56:08.972 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2023-03-03T07:56:08.972 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T07:56:08.973 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T07:56:08.973 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T07:56:08.973 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T07:56:08.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:08.975 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-03-03T07:56:08.976 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-03-03T07:56:08.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T07:56:08.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T07:56:08.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T07:56:08.978 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-03-03T07:56:09.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:09.218 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-03-03T07:56:09.287 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-03-03T07:56:09.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:09.289 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2023-03-03T07:56:09.355 INFO:teuthology.orchestra.run.smithi098.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-03-03T07:56:09.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:09.357 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-03-03T07:56:09.357 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-03-03T07:56:09.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-03-03T07:56:09.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-03-03T07:56:09.358 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-03-03T07:56:09.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:09.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-03-03T07:56:09.664 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T07:56:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T07:56:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T07:56:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-03-03T07:56:09.665 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:09.665 INFO:teuthology.orchestra.run.smithi098.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T07:56:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether c6:e3:51:b7:86:a5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-03-03T07:56:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-03-03T07:56:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c4e3:51ff:feb7:86a5/64 scope link 2023-03-03T07:56:09.666 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:09.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:09.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-03-03T07:56:09.770 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T07:56:09.770 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T07:56:09.770 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T07:56:09.770 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:09.770 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-03-03T07:56:09.771 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:09.771 INFO:teuthology.orchestra.run.smithi098.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T07:56:09.771 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether c6:e3:51:b7:86:a5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-03-03T07:56:09.771 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-03-03T07:56:09.771 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:09.771 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c4e3:51ff:feb7:86a5/64 scope link 2023-03-03T07:56:09.771 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:09.774 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-03-03T07:56:09.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:09.775 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-03-03T07:56:09.775 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-03-03T07:56:09.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-03-03T07:56:09.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-03-03T07:56:09.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-03-03T07:56:09.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-03-03T07:56:09.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-03-03T07:56:09.776 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-03-03T07:56:10.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:10.119 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-03-03T07:56:10.120 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-03-03T07:56:10.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.1 up 2023-03-03T07:56:10.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.1 master ceph-brx 2023-03-03T07:56:10.120 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-03-03T07:56:10.355 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2023-03-03T07:56:10.356 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:56:10.356 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-03-03T07:56:10.424 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-03-03T07:56:10.426 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:56:10.426 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-03-03T07:56:10.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T07:56:10.596 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:56:10.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:56:10.694 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:56:10.698 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:56:10.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:10.700 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T07:56:10.768 INFO:teuthology.orchestra.run.smithi098.stdout:42 2023-03-03T07:56:10.769 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2023-03-03T07:56:10.769 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2 --client_mountpoint=/) 2023-03-03T07:56:10.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T07:56:10.878 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:56:10.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:56:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:56:10.982 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:56:10.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:10.984 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T07:56:10.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi098.stderr:2023-03-03T07:56:10.992+0000 7f68ab0965c0 -1 init, newargv = 0x563b3cd019d0 newargc=15 2023-03-03T07:56:10.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi098.stderr:ceph-fuse[118822]: starting ceph client 2023-03-03T07:56:11.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi098.stderr:ceph-fuse[118822]: starting fuse 2023-03-03T07:56:11.061 INFO:teuthology.orchestra.run.smithi098.stdout:42 2023-03-03T07:56:11.062 INFO:teuthology.orchestra.run.smithi098.stdout:44 2023-03-03T07:56:11.063 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 44] 2023-03-03T07:56:11.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T07:56:11.063 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2023-03-03T07:56:11.064 DEBUG:teuthology.orchestra.run.smithi098:> import re 2023-03-03T07:56:11.064 DEBUG:teuthology.orchestra.run.smithi098:> import os 2023-03-03T07:56:11.065 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2023-03-03T07:56:11.065 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:56:11.065 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2023-03-03T07:56:11.066 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2023-03-03T07:56:11.066 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2023-03-03T07:56:11.066 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2023-03-03T07:56:11.066 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:56:11.067 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2023-03-03T07:56:11.067 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2023-03-03T07:56:11.069 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2023-03-03T07:56:11.069 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2023-03-03T07:56:11.070 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:56:11.071 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2023-03-03T07:56:11.072 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T07:56:11.072 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T07:56:11.073 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T07:56:11.073 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2023-03-03T07:56:11.073 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2023-03-03T07:56:11.074 DEBUG:teuthology.orchestra.run.smithi098:> return f 2023-03-03T07:56:11.074 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T07:56:11.075 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:56:11.075 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.2")) 2023-03-03T07:56:11.076 DEBUG:teuthology.orchestra.run.smithi098:> ' 2023-03-03T07:56:11.291 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.2.118822.asok 2023-03-03T07:56:11.469 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.118822.asok 2023-03-03T07:56:11.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:11.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.118822.asok status 2023-03-03T07:56:11.735 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T07:56:11.736 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2023-03-03T07:56:11.736 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T07:56:11.736 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T07:56:11.736 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "2", 2023-03-03T07:56:11.736 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2023-03-03T07:56:11.737 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2023-03-03T07:56:11.737 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "118822", 2023-03-03T07:56:11.737 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2023-03-03T07:56:11.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:56:11.737 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2023-03-03T07:56:11.737 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2023-03-03T07:56:11.737 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4602, 2023-03-03T07:56:11.738 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2023-03-03T07:56:11.738 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2023-03-03T07:56:11.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2023-03-03T07:56:11.738 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4602 2023-03-03T07:56:11.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:56:11.738 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T07:56:11.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T07:56:11.739 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.2:0", 2023-03-03T07:56:11.739 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2649207224 2023-03-03T07:56:11.739 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:56:11.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:56:11.739 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T07:56:11.739 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T07:56:11.739 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.2:0", 2023-03-03T07:56:11.739 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2649207224 2023-03-03T07:56:11.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:56:11.740 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.4602 192.168.0.2:0/2649207224", 2023-03-03T07:56:11.740 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.2:0/2649207224", 2023-03-03T07:56:11.740 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2023-03-03T07:56:11.740 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 7, 2023-03-03T07:56:11.740 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 24, 2023-03-03T07:56:11.740 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2023-03-03T07:56:11.740 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2023-03-03T07:56:11.741 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2023-03-03T07:56:11.741 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T07:56:11.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T07:56:12.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.001+0000 7feee3fff700 1 Processor -- start 2023-03-03T07:56:12.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.001+0000 7feee3fff700 1 -- start start 2023-03-03T07:56:12.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.002+0000 7feee3fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7feee415a570 0x7feee415ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:12.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.002+0000 7feee3fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feee4071ad0 0x7feee40b0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:12.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.003+0000 7feee3fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee4071640 con 0x7feee415a940 2023-03-03T07:56:12.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.003+0000 7feee3fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee4071f50 con 0x7feee415a570 2023-03-03T07:56:12.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.003+0000 7feee3fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee40b4c70 con 0x7feee4071ad0 2023-03-03T07:56:12.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.003+0000 7feedbfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7feee415a940 0x7feee40b4140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57126/0 (socket says 172.21.15.98:57126) 2023-03-03T07:56:12.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.003+0000 7feedbfff700 1 -- 172.21.15.98:0/1434585202 learned_addr learned my addr 172.21.15.98:0/1434585202 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:12.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.003+0000 7feee2ffd700 1 --2- 172.21.15.98:0/1434585202 >> v2:172.21.15.100:3300/0 conn(0x7feee415a570 0x7feee415ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:12.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.003+0000 7feee37fe700 1 --2- 172.21.15.98:0/1434585202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feee4071ad0 0x7feee40b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:12.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.003+0000 7feee27fc700 1 -- 172.21.15.98:0/1434585202 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268786186 0 0) 0x7feee4071640 con 0x7feee415a940 2023-03-03T07:56:12.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.004+0000 7feee27fc700 1 -- 172.21.15.98:0/1434585202 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed00035b0 con 0x7feee415a940 2023-03-03T07:56:12.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.004+0000 7feee2ffd700 1 -- 172.21.15.98:0/1434585202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feee4071ad0 msgr2=0x7feee40b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:12.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.004+0000 7feee2ffd700 1 --2- 172.21.15.98:0/1434585202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feee4071ad0 0x7feee40b0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:12.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.004+0000 7feee2ffd700 1 -- 172.21.15.98:0/1434585202 >> v1:172.21.15.98:6789/0 conn(0x7feee415a940 legacy=0x7feee40b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:12.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.004+0000 7feee2ffd700 1 -- 172.21.15.98:0/1434585202 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee40b4990 con 0x7feee415a570 2023-03-03T07:56:12.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.004+0000 7feee2ffd700 1 --2- 172.21.15.98:0/1434585202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feee415a570 0x7feee415ef60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feed4009840 tx=0x7feed4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d98d87614f337654 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:12.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.004+0000 7feee27fc700 1 -- 172.21.15.98:0/1434585202 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed4011830 con 0x7feee415a570 2023-03-03T07:56:12.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.004+0000 7feee27fc700 1 -- 172.21.15.98:0/1434585202 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed4011990 con 0x7feee415a570 2023-03-03T07:56:12.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.004+0000 7feee27fc700 1 -- 172.21.15.98:0/1434585202 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed4011d20 con 0x7feee415a570 2023-03-03T07:56:12.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.005+0000 7feee3fff700 1 -- 172.21.15.98:0/1434585202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feee415a570 msgr2=0x7feee415ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:12.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.005+0000 7feee3fff700 1 --2- 172.21.15.98:0/1434585202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feee415a570 0x7feee415ef60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feed4009840 tx=0x7feed4009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:12.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.005+0000 7feee3fff700 1 -- 172.21.15.98:0/1434585202 shutdown_connections 2023-03-03T07:56:12.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.005+0000 7feee3fff700 1 --2- 172.21.15.98:0/1434585202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feee415a570 0x7feee415ef60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:12.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.005+0000 7feee3fff700 1 --2- 172.21.15.98:0/1434585202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feee4071ad0 0x7feee40b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:12.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.005+0000 7feee3fff700 1 -- 172.21.15.98:0/1434585202 >> 172.21.15.98:0/1434585202 conn(0x7feee40cf950 msgr2=0x7feee40cb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:12.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.005+0000 7feee3fff700 1 -- 172.21.15.98:0/1434585202 shutdown_connections 2023-03-03T07:56:12.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.005+0000 7feee3fff700 1 -- 172.21.15.98:0/1434585202 wait complete. 2023-03-03T07:56:12.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.007+0000 7feee3fff700 1 Processor -- start 2023-03-03T07:56:12.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.007+0000 7feee3fff700 1 -- start start 2023-03-03T07:56:12.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.008+0000 7feee3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee4071ad0 0x7feee4158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:12.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.008+0000 7feee3fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feee415a570 0x7feee40cca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:12.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.009+0000 7feee3fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feee415a940 0x7feee41f1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:12.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.009+0000 7feee3fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee41f2210 con 0x7feee4071ad0 2023-03-03T07:56:12.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.009+0000 7feee3fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee41f24f0 con 0x7feee415a570 2023-03-03T07:56:12.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.009+0000 7feee3fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee41f27d0 con 0x7feee415a940 2023-03-03T07:56:12.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.009+0000 7feee2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee4071ad0 0x7feee4158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.009+0000 7feee2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee4071ad0 0x7feee4158650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48912/0 (socket says 172.21.15.98:48912) 2023-03-03T07:56:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.009+0000 7feee2ffd700 1 -- 172.21.15.98:0/2710610098 learned_addr learned my addr 172.21.15.98:0/2710610098 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.009+0000 7feedbfff700 1 --2- 172.21.15.98:0/2710610098 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feee415a570 0x7feee40cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.009+0000 7feee37fe700 1 --2- 172.21.15.98:0/2710610098 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feee415a940 0x7feee41f1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.009+0000 7feee2ffd700 1 -- 172.21.15.98:0/2710610098 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feee415a940 msgr2=0x7feee41f1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.009+0000 7feee2ffd700 1 --2- 172.21.15.98:0/2710610098 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feee415a940 0x7feee41f1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.009+0000 7feee2ffd700 1 -- 172.21.15.98:0/2710610098 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feee415a570 msgr2=0x7feee40cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:12.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.009+0000 7feee2ffd700 1 --2- 172.21.15.98:0/2710610098 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feee415a570 0x7feee40cca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:12.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.009+0000 7feee2ffd700 1 -- 172.21.15.98:0/2710610098 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed40094a0 con 0x7feee4071ad0 2023-03-03T07:56:12.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.009+0000 7feedbfff700 1 --2- 172.21.15.98:0/2710610098 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feee415a570 0x7feee40cca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:12.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.010+0000 7feee2ffd700 1 --2- 172.21.15.98:0/2710610098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee4071ad0 0x7feee4158650 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feed40090b0 tx=0x7feed4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:12.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.010+0000 7feedb7fe700 1 -- 172.21.15.98:0/2710610098 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed40036a0 con 0x7feee4071ad0 2023-03-03T07:56:12.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.010+0000 7feedb7fe700 1 -- 172.21.15.98:0/2710610098 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed401f070 con 0x7feee4071ad0 2023-03-03T07:56:12.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.010+0000 7feedb7fe700 1 -- 172.21.15.98:0/2710610098 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed4003a20 con 0x7feee4071ad0 2023-03-03T07:56:12.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.010+0000 7feee3fff700 1 -- 172.21.15.98:0/2710610098 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee40712d0 con 0x7feee4071ad0 2023-03-03T07:56:12.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.010+0000 7feee3fff700 1 -- 172.21.15.98:0/2710610098 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee4070ff0 con 0x7feee4071ad0 2023-03-03T07:56:12.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.011+0000 7feee3fff700 1 -- 172.21.15.98:0/2710610098 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee40b0cc0 con 0x7feee4071ad0 2023-03-03T07:56:12.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.012+0000 7feedb7fe700 1 -- 172.21.15.98:0/2710610098 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feed400f050 con 0x7feee4071ad0 2023-03-03T07:56:12.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.014+0000 7feedb7fe700 1 --2- 172.21.15.98:0/2710610098 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feec40682f0 0x7feec406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:12.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.014+0000 7feedb7fe700 1 -- 172.21.15.98:0/2710610098 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7feed4021070 con 0x7feee4071ad0 2023-03-03T07:56:12.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.014+0000 7feedbfff700 1 --2- 172.21.15.98:0/2710610098 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feec40682f0 0x7feec406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:12.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.015+0000 7feedbfff700 1 --2- 172.21.15.98:0/2710610098 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feec40682f0 0x7feec406a7a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7feee4145aa0 tx=0x7feecc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:12.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.018+0000 7feedb7fe700 1 -- 172.21.15.98:0/2710610098 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feed4053e20 con 0x7feee4071ad0 2023-03-03T07:56:12.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.259+0000 7feee3fff700 1 -- 172.21.15.98:0/2710610098 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7feee4068300 con 0x7feee4071ad0 2023-03-03T07:56:12.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.260+0000 7feedb7fe700 1 -- 172.21.15.98:0/2710610098 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7feed4057440 con 0x7feee4071ad0 2023-03-03T07:56:12.261 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T07:56:12.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.262+0000 7feee3fff700 1 -- 172.21.15.98:0/2710610098 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feec40682f0 msgr2=0x7feec406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:12.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.262+0000 7feee3fff700 1 --2- 172.21.15.98:0/2710610098 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feec40682f0 0x7feec406a7a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7feee4145aa0 tx=0x7feecc009040 comp rx=0 tx=0).stop 2023-03-03T07:56:12.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.263+0000 7feee3fff700 1 -- 172.21.15.98:0/2710610098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee4071ad0 msgr2=0x7feee4158650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:12.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.263+0000 7feee3fff700 1 --2- 172.21.15.98:0/2710610098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee4071ad0 0x7feee4158650 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feed40090b0 tx=0x7feed4004750 comp rx=0 tx=0).stop 2023-03-03T07:56:12.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.263+0000 7feee3fff700 1 -- 172.21.15.98:0/2710610098 shutdown_connections 2023-03-03T07:56:12.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.263+0000 7feee3fff700 1 --2- 172.21.15.98:0/2710610098 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feec40682f0 0x7feec406a7a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:12.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.263+0000 7feee3fff700 1 --2- 172.21.15.98:0/2710610098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee4071ad0 0x7feee4158650 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:12.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.263+0000 7feee3fff700 1 --2- 172.21.15.98:0/2710610098 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feee415a940 0x7feee41f1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:12.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.263+0000 7feee3fff700 1 --2- 172.21.15.98:0/2710610098 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feee415a570 0x7feee40cca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:12.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.263+0000 7feee3fff700 1 -- 172.21.15.98:0/2710610098 >> 172.21.15.98:0/2710610098 conn(0x7feee40cf950 msgr2=0x7feee40c45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:12.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.263+0000 7feee3fff700 1 -- 172.21.15.98:0/2710610098 shutdown_connections 2023-03-03T07:56:12.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:12.263+0000 7feee3fff700 1 -- 172.21.15.98:0/2710610098 wait complete. 2023-03-03T07:56:12.292 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T07:56:12.292 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T07:56:12.293 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2023-03-03T07:56:12.293 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-03-03T07:56:12.293 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-03-03T07:56:12.293 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T07:56:12.293 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-03-03T07:56:12.294 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T07:56:12.294 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-03-03T07:56:12.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:12.294 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2023-03-03T07:56:12.368 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T07:56:12.369 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T07:56:12.369 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T07:56:12.369 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:12.370 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2023-03-03T07:56:12.370 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:12.370 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:56:12.371 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:12.371 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2023-03-03T07:56:12.371 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:56:12.371 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:12.371 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2023-03-03T07:56:12.372 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:56:12.372 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:12.372 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T07:56:12.372 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:12.373 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T07:56:12.373 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42188sec preferred_lft 42188sec 2023-03-03T07:56:12.374 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2023-03-03T07:56:12.374 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:12.376 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-03-03T07:56:12.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:12.377 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2023-03-03T07:56:12.377 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2023-03-03T07:56:12.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link add name ceph-brx type bridge 2023-03-03T07:56:12.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip addr flush dev ceph-brx 2023-03-03T07:56:12.379 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set ceph-brx up 2023-03-03T07:56:12.379 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-03-03T07:56:12.379 DEBUG:teuthology.orchestra.run.smithi100:> ') 2023-03-03T07:56:12.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:12.973 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T07:56:13.128 INFO:teuthology.orchestra.run.smithi100.stdout:1 2023-03-03T07:56:13.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:13.134 DEBUG:teuthology.orchestra.run.smithi100:> route 2023-03-03T07:56:13.221 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2023-03-03T07:56:13.221 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T07:56:13.221 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T07:56:13.221 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T07:56:13.221 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T07:56:13.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:13.223 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2023-03-03T07:56:13.223 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2023-03-03T07:56:13.224 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T07:56:13.224 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T07:56:13.224 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T07:56:13.224 DEBUG:teuthology.orchestra.run.smithi100:> ') 2023-03-03T07:56:14.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:14.028 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2023-03-03T07:56:14.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:14.099 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list-id 2023-03-03T07:56:14.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:14.169 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2023-03-03T07:56:14.169 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2023-03-03T07:56:14.169 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-03-03T07:56:14.169 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-03-03T07:56:14.169 DEBUG:teuthology.orchestra.run.smithi100:> ') 2023-03-03T07:56:14.376 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-03-03T07:56:14.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:14.376 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2023-03-03T07:56:14.377 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2023-03-03T07:56:14.377 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-03-03T07:56:14.377 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-03-03T07:56:14.377 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-03-03T07:56:14.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-03-03T07:56:14.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-03-03T07:56:14.378 DEBUG:teuthology.orchestra.run.smithi100:> ') 2023-03-03T07:56:14.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:14.749 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2023-03-03T07:56:14.750 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2023-03-03T07:56:14.750 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2023-03-03T07:56:14.750 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set dev brx.0 master ceph-brx 2023-03-03T07:56:14.750 DEBUG:teuthology.orchestra.run.smithi100:> ') 2023-03-03T07:56:14.985 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-03-03T07:56:15.287 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:56:15.288 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-03-03T07:56:15.359 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-03-03T07:56:15.359 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:56:15.359 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-03-03T07:56:15.432 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T07:56:15.535 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:56:15.536 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:56:15.637 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:56:15.641 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:56:15.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:15.642 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T07:56:15.713 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-03-03T07:56:15.714 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_mountpoint=/) 2023-03-03T07:56:15.720 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T07:56:15.824 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:56:15.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:56:15.924 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:56:15.928 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:56:15.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:15.929 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T07:56:15.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2023-03-03T07:56:15.941+0000 7fac229135c0 -1 init, newargv = 0x562658f05d60 newargc=15 2023-03-03T07:56:15.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[115353]: starting ceph client 2023-03-03T07:56:15.954 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[115353]: starting fuse 2023-03-03T07:56:16.005 INFO:teuthology.orchestra.run.smithi100.stdout:42 2023-03-03T07:56:16.006 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2023-03-03T07:56:16.006 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T07:56:16.007 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2023-03-03T07:56:16.007 DEBUG:teuthology.orchestra.run.smithi100:> import re 2023-03-03T07:56:16.007 DEBUG:teuthology.orchestra.run.smithi100:> import os 2023-03-03T07:56:16.007 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2023-03-03T07:56:16.007 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:56:16.008 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2023-03-03T07:56:16.008 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-03-03T07:56:16.008 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2023-03-03T07:56:16.008 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-03-03T07:56:16.009 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:56:16.009 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2023-03-03T07:56:16.009 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2023-03-03T07:56:16.009 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2023-03-03T07:56:16.010 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2023-03-03T07:56:16.010 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:56:16.010 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2023-03-03T07:56:16.010 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T07:56:16.011 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T07:56:16.011 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T07:56:16.011 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2023-03-03T07:56:16.011 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2023-03-03T07:56:16.011 DEBUG:teuthology.orchestra.run.smithi100:> return f 2023-03-03T07:56:16.012 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T07:56:16.012 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:56:16.012 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.1")) 2023-03-03T07:56:16.012 DEBUG:teuthology.orchestra.run.smithi100:> ' 2023-03-03T07:56:16.230 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.1.115353.asok 2023-03-03T07:56:16.408 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.115353.asok 2023-03-03T07:56:16.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:16.409 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.115353.asok status 2023-03-03T07:56:16.683 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2023-03-03T07:56:16.684 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2023-03-03T07:56:16.684 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T07:56:16.684 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T07:56:16.684 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "1", 2023-03-03T07:56:16.685 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2023-03-03T07:56:16.685 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-03-03T07:56:16.685 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "115353", 2023-03-03T07:56:16.685 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2023-03-03T07:56:16.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:56:16.685 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2023-03-03T07:56:16.686 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2023-03-03T07:56:16.686 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 4617, 2023-03-03T07:56:16.686 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2023-03-03T07:56:16.686 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2023-03-03T07:56:16.686 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2023-03-03T07:56:16.686 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 4617 2023-03-03T07:56:16.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:56:16.687 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T07:56:16.687 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T07:56:16.687 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:0", 2023-03-03T07:56:16.687 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1764425242 2023-03-03T07:56:16.687 INFO:teuthology.orchestra.run.smithi100.stdout: } 2023-03-03T07:56:16.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:56:16.687 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T07:56:16.688 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T07:56:16.688 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:0", 2023-03-03T07:56:16.688 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1764425242 2023-03-03T07:56:16.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:56:16.688 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.4617 172.21.15.100:0/1764425242", 2023-03-03T07:56:16.688 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "172.21.15.100:0/1764425242", 2023-03-03T07:56:16.688 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2023-03-03T07:56:16.688 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 7, 2023-03-03T07:56:16.689 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 24, 2023-03-03T07:56:16.689 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2023-03-03T07:56:16.689 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2023-03-03T07:56:16.689 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2023-03-03T07:56:16.689 INFO:teuthology.orchestra.run.smithi100.stdout:} 2023-03-03T07:56:16.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T07:56:16.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.955+0000 7fc42619d700 1 Processor -- start 2023-03-03T07:56:16.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.956+0000 7fc42619d700 1 -- start start 2023-03-03T07:56:16.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.957+0000 7fc42619d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc4200b51c0 0x7fc420160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:16.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.957+0000 7fc42619d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc4200d1580 0x7fc4200d1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:16.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.957+0000 7fc42619d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4201614c0 con 0x7fc420146360 2023-03-03T07:56:16.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.957+0000 7fc42619d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4200d10f0 con 0x7fc4200d1580 2023-03-03T07:56:16.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.957+0000 7fc42619d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4200cdf20 con 0x7fc4200b51c0 2023-03-03T07:56:16.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.957+0000 7fc42519b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc420146360 0x7fc4200b4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58258/0 (socket says 172.21.15.98:58258) 2023-03-03T07:56:16.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.957+0000 7fc42519b700 1 -- 172.21.15.98:0/2927128414 learned_addr learned my addr 172.21.15.98:0/2927128414 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:16.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.957+0000 7fc42599c700 1 --2- 172.21.15.98:0/2927128414 >> v2:172.21.15.98:3300/0 conn(0x7fc4200d1580 0x7fc4200d1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:16.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.957+0000 7fc42499a700 1 --2- 172.21.15.98:0/2927128414 >> v2:172.21.15.100:3300/0 conn(0x7fc4200b51c0 0x7fc420160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:16.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.958+0000 7fc417fff700 1 -- 172.21.15.98:0/2927128414 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448789865 0 0) 0x7fc41c001110 con 0x7fc420146360 2023-03-03T07:56:16.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.958+0000 7fc417fff700 1 -- 172.21.15.98:0/2927128414 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc40c0035b0 con 0x7fc420146360 2023-03-03T07:56:16.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.958+0000 7fc42599c700 1 -- 172.21.15.98:0/2927128414 >> v2:172.21.15.100:3300/0 conn(0x7fc4200b51c0 msgr2=0x7fc420160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:16.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.958+0000 7fc42599c700 1 --2- 172.21.15.98:0/2927128414 >> v2:172.21.15.100:3300/0 conn(0x7fc4200b51c0 0x7fc420160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:16.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.958+0000 7fc42599c700 1 -- 172.21.15.98:0/2927128414 >> v1:172.21.15.98:6789/0 conn(0x7fc420146360 legacy=0x7fc4200b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:16.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.958+0000 7fc42599c700 1 -- 172.21.15.98:0/2927128414 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4200cd680 con 0x7fc4200d1580 2023-03-03T07:56:16.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.958+0000 7fc42599c700 1 --2- 172.21.15.98:0/2927128414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4200d1580 0x7fc4200d1980 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc42006bd20 tx=0x7fc41800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4efe02b650d183e1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:16.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.959+0000 7fc417fff700 1 -- 172.21.15.98:0/2927128414 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41800e070 con 0x7fc4200d1580 2023-03-03T07:56:16.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.959+0000 7fc417fff700 1 -- 172.21.15.98:0/2927128414 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41800bb20 con 0x7fc4200d1580 2023-03-03T07:56:16.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.959+0000 7fc417fff700 1 -- 172.21.15.98:0/2927128414 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418012430 con 0x7fc4200d1580 2023-03-03T07:56:16.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.959+0000 7fc42619d700 1 -- 172.21.15.98:0/2927128414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4200d1580 msgr2=0x7fc4200d1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:16.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.959+0000 7fc42619d700 1 --2- 172.21.15.98:0/2927128414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4200d1580 0x7fc4200d1980 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc42006bd20 tx=0x7fc41800b330 comp rx=0 tx=0).stop 2023-03-03T07:56:16.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.959+0000 7fc42619d700 1 -- 172.21.15.98:0/2927128414 shutdown_connections 2023-03-03T07:56:16.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.959+0000 7fc42619d700 1 --2- 172.21.15.98:0/2927128414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4200d1580 0x7fc4200d1980 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:16.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.959+0000 7fc42619d700 1 --2- 172.21.15.98:0/2927128414 >> v2:172.21.15.100:3300/0 conn(0x7fc4200b51c0 0x7fc420160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:16.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.959+0000 7fc42619d700 1 -- 172.21.15.98:0/2927128414 >> 172.21.15.98:0/2927128414 conn(0x7fc42006b8e0 msgr2=0x7fc4200cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:16.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.959+0000 7fc42619d700 1 -- 172.21.15.98:0/2927128414 shutdown_connections 2023-03-03T07:56:16.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.959+0000 7fc42619d700 1 -- 172.21.15.98:0/2927128414 wait complete. 2023-03-03T07:56:16.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.961+0000 7fc42619d700 1 Processor -- start 2023-03-03T07:56:16.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.962+0000 7fc42619d700 1 -- start start 2023-03-03T07:56:16.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.962+0000 7fc42619d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4200b51c0 0x7fc4201524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:16.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42619d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4200d1580 0x7fc4201579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:16.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42619d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc420146360 0x7fc42015a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:16.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42619d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4201f6740 con 0x7fc4200d1580 2023-03-03T07:56:16.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42619d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4201f5870 con 0x7fc420146360 2023-03-03T07:56:16.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42619d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc420158250 con 0x7fc4200b51c0 2023-03-03T07:56:16.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42499a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4200d1580 0x7fc4201579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:16.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42499a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4200d1580 0x7fc4201579a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49368/0 (socket says 172.21.15.98:49368) 2023-03-03T07:56:16.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42499a700 1 -- 172.21.15.98:0/70065477 learned_addr learned my addr 172.21.15.98:0/70065477 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:16.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42519b700 1 --2- 172.21.15.98:0/70065477 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4200b51c0 0x7fc4201524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:16.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42599c700 1 --2- 172.21.15.98:0/70065477 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc420146360 0x7fc42015a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:16.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42499a700 1 -- 172.21.15.98:0/70065477 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4200b51c0 msgr2=0x7fc4201524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:16.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42499a700 1 --2- 172.21.15.98:0/70065477 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4200b51c0 0x7fc4201524f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:16.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42499a700 1 -- 172.21.15.98:0/70065477 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc420146360 msgr2=0x7fc42015a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:16.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42499a700 1 --2- 172.21.15.98:0/70065477 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc420146360 0x7fc42015a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:16.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42499a700 1 -- 172.21.15.98:0/70065477 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc418009a70 con 0x7fc4200d1580 2023-03-03T07:56:16.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.963+0000 7fc42599c700 1 --2- 172.21.15.98:0/70065477 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc420146360 0x7fc42015a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:16.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.964+0000 7fc42499a700 1 --2- 172.21.15.98:0/70065477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4200d1580 0x7fc4201579a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc41000e870 tx=0x7fc41000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:16.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.964+0000 7fc415ffb700 1 -- 172.21.15.98:0/70065477 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc410009590 con 0x7fc4200d1580 2023-03-03T07:56:16.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.964+0000 7fc415ffb700 1 -- 172.21.15.98:0/70065477 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41000ee70 con 0x7fc4200d1580 2023-03-03T07:56:16.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.964+0000 7fc415ffb700 1 -- 172.21.15.98:0/70065477 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41000ca90 con 0x7fc4200d1580 2023-03-03T07:56:16.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.964+0000 7fc42619d700 1 -- 172.21.15.98:0/70065477 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4200cdc40 con 0x7fc4200d1580 2023-03-03T07:56:16.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.964+0000 7fc42619d700 1 -- 172.21.15.98:0/70065477 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4200cd960 con 0x7fc4200d1580 2023-03-03T07:56:16.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.965+0000 7fc42619d700 1 -- 172.21.15.98:0/70065477 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4200d0d80 con 0x7fc4200d1580 2023-03-03T07:56:16.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.966+0000 7fc415ffb700 1 -- 172.21.15.98:0/70065477 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc410019030 con 0x7fc4200d1580 2023-03-03T07:56:16.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.968+0000 7fc415ffb700 1 --2- 172.21.15.98:0/70065477 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc4040682f0 0x7fc40406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:16.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.968+0000 7fc415ffb700 1 -- 172.21.15.98:0/70065477 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc410085c10 con 0x7fc4200d1580 2023-03-03T07:56:16.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.968+0000 7fc42519b700 1 --2- 172.21.15.98:0/70065477 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc4040682f0 0x7fc40406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:16.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.969+0000 7fc42519b700 1 --2- 172.21.15.98:0/70065477 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc4040682f0 0x7fc40406a7a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc41c001270 tx=0x7fc41c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:16.973+0000 7fc415ffb700 1 -- 172.21.15.98:0/70065477 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc410052380 con 0x7fc4200d1580 2023-03-03T07:56:17.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:17.212+0000 7fc42619d700 1 -- 172.21.15.98:0/70065477 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc420068300 con 0x7fc4200d1580 2023-03-03T07:56:17.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:17.212+0000 7fc415ffb700 1 -- 172.21.15.98:0/70065477 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc4100559a0 con 0x7fc4200d1580 2023-03-03T07:56:17.214 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T07:56:17.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:17.215+0000 7fc42619d700 1 -- 172.21.15.98:0/70065477 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc4040682f0 msgr2=0x7fc40406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:17.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:17.215+0000 7fc42619d700 1 --2- 172.21.15.98:0/70065477 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc4040682f0 0x7fc40406a7a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc41c001270 tx=0x7fc41c009040 comp rx=0 tx=0).stop 2023-03-03T07:56:17.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:17.215+0000 7fc42619d700 1 -- 172.21.15.98:0/70065477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4200d1580 msgr2=0x7fc4201579a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:17.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:17.215+0000 7fc42619d700 1 --2- 172.21.15.98:0/70065477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4200d1580 0x7fc4201579a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc41000e870 tx=0x7fc41000ec30 comp rx=0 tx=0).stop 2023-03-03T07:56:17.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:17.215+0000 7fc42619d700 1 -- 172.21.15.98:0/70065477 shutdown_connections 2023-03-03T07:56:17.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:17.215+0000 7fc42619d700 1 --2- 172.21.15.98:0/70065477 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc4040682f0 0x7fc40406a7a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:17.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:17.215+0000 7fc42619d700 1 --2- 172.21.15.98:0/70065477 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc420146360 0x7fc42015a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:17.215+0000 7fc42619d700 1 --2- 172.21.15.98:0/70065477 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4200b51c0 0x7fc4201524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:17.215+0000 7fc42619d700 1 --2- 172.21.15.98:0/70065477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4200d1580 0x7fc4201579a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:17.215+0000 7fc42619d700 1 -- 172.21.15.98:0/70065477 >> 172.21.15.98:0/70065477 conn(0x7fc42006b8e0 msgr2=0x7fc4200d2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:17.215+0000 7fc42619d700 1 -- 172.21.15.98:0/70065477 shutdown_connections 2023-03-03T07:56:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:17.215+0000 7fc42619d700 1 -- 172.21.15.98:0/70065477 wait complete. 2023-03-03T07:56:17.243 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T07:56:17.244 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T07:56:17.244 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2023-03-03T07:56:17.244 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-03-03T07:56:17.244 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-03-03T07:56:17.245 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T07:56:17.245 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-03-03T07:56:17.245 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T07:56:17.245 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-03-03T07:56:17.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:17.246 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2023-03-03T07:56:17.318 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T07:56:17.318 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T07:56:17.318 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T07:56:17.318 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:17.318 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2023-03-03T07:56:17.319 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:17.319 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:56:17.319 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:17.319 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2023-03-03T07:56:17.319 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:56:17.319 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:17.319 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2023-03-03T07:56:17.319 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:56:17.320 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:17.320 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T07:56:17.320 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:17.320 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T07:56:17.320 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42183sec preferred_lft 42183sec 2023-03-03T07:56:17.320 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2023-03-03T07:56:17.320 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:17.321 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T07:56:17.321 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:5e:bd:c1:84:4d brd ff:ff:ff:ff:ff:ff 2023-03-03T07:56:17.321 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-03-03T07:56:17.321 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:17.321 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::9406:dcff:fe40:a4a4/64 scope link 2023-03-03T07:56:17.322 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:17.322 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T07:56:17.322 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:5e:bd:c1:84:4d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-03-03T07:56:17.322 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f45e:bdff:fec1:844d/64 scope link 2023-03-03T07:56:17.322 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:17.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:17.324 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T07:56:17.421 INFO:teuthology.orchestra.run.smithi100.stdout:1 2023-03-03T07:56:17.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:17.427 DEBUG:teuthology.orchestra.run.smithi100:> route 2023-03-03T07:56:17.499 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2023-03-03T07:56:17.500 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T07:56:17.500 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T07:56:17.500 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T07:56:17.500 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T07:56:17.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:17.502 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2023-03-03T07:56:17.502 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2023-03-03T07:56:17.502 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T07:56:17.502 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T07:56:17.502 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T07:56:17.503 DEBUG:teuthology.orchestra.run.smithi100:> ') 2023-03-03T07:56:17.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:17.744 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2023-03-03T07:56:17.815 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-03-03T07:56:17.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:17.816 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list-id 2023-03-03T07:56:17.884 INFO:teuthology.orchestra.run.smithi100.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-03-03T07:56:17.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:17.885 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2023-03-03T07:56:17.886 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2023-03-03T07:56:17.886 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-03-03T07:56:17.886 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-03-03T07:56:17.886 DEBUG:teuthology.orchestra.run.smithi100:> ') 2023-03-03T07:56:18.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:18.092 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-03-03T07:56:18.193 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T07:56:18.194 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T07:56:18.194 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T07:56:18.194 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:18.194 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2023-03-03T07:56:18.194 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:18.194 INFO:teuthology.orchestra.run.smithi100.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T07:56:18.195 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ae:cd:d0:fb:58:d2 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-03-03T07:56:18.195 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-03-03T07:56:18.195 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:18.195 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::accd:d0ff:fefb:58d2/64 scope link 2023-03-03T07:56:18.195 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:18.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:18.198 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-03-03T07:56:18.301 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T07:56:18.302 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T07:56:18.302 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T07:56:18.302 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:18.302 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2023-03-03T07:56:18.302 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:18.303 INFO:teuthology.orchestra.run.smithi100.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T07:56:18.303 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ae:cd:d0:fb:58:d2 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-03-03T07:56:18.303 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-03-03T07:56:18.303 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:18.303 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::accd:d0ff:fefb:58d2/64 scope link 2023-03-03T07:56:18.303 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:56:18.306 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-03-03T07:56:18.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:18.309 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2023-03-03T07:56:18.309 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2023-03-03T07:56:18.310 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-03-03T07:56:18.310 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-03-03T07:56:18.310 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-03-03T07:56:18.310 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-03-03T07:56:18.311 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-03-03T07:56:18.311 DEBUG:teuthology.orchestra.run.smithi100:> ') 2023-03-03T07:56:18.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:18.660 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2023-03-03T07:56:18.660 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2023-03-03T07:56:18.661 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.1 up 2023-03-03T07:56:18.661 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set dev brx.1 master ceph-brx 2023-03-03T07:56:18.661 DEBUG:teuthology.orchestra.run.smithi100:> ') 2023-03-03T07:56:18.895 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2023-03-03T07:56:18.895 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:56:18.895 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-03-03T07:56:18.963 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-03-03T07:56:18.964 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:56:18.964 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-03-03T07:56:19.031 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T07:56:19.132 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:56:19.133 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:56:19.231 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:56:19.235 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:56:19.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:19.237 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T07:56:19.305 INFO:teuthology.orchestra.run.smithi100.stdout:42 2023-03-03T07:56:19.306 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2023-03-03T07:56:19.307 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3 --client_mountpoint=/) 2023-03-03T07:56:19.312 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T07:56:19.417 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:56:19.418 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:56:19.521 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:56:19.525 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:56:19.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:19.527 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T07:56:19.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi100.stderr:2023-03-03T07:56:19.539+0000 7fa5d0b615c0 -1 init, newargv = 0x562674856930 newargc=15 2023-03-03T07:56:19.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi100.stderr:ceph-fuse[116028]: starting ceph client 2023-03-03T07:56:19.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi100.stderr:ceph-fuse[116028]: starting fuse 2023-03-03T07:56:19.602 INFO:teuthology.orchestra.run.smithi100.stdout:42 2023-03-03T07:56:19.602 INFO:teuthology.orchestra.run.smithi100.stdout:44 2023-03-03T07:56:19.603 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 44] 2023-03-03T07:56:19.603 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T07:56:19.603 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2023-03-03T07:56:19.604 DEBUG:teuthology.orchestra.run.smithi100:> import re 2023-03-03T07:56:19.604 DEBUG:teuthology.orchestra.run.smithi100:> import os 2023-03-03T07:56:19.604 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2023-03-03T07:56:19.604 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:56:19.605 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2023-03-03T07:56:19.605 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2023-03-03T07:56:19.605 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2023-03-03T07:56:19.605 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2023-03-03T07:56:19.606 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:56:19.606 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2023-03-03T07:56:19.606 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2023-03-03T07:56:19.606 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2023-03-03T07:56:19.606 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2023-03-03T07:56:19.607 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:56:19.607 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2023-03-03T07:56:19.607 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T07:56:19.607 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T07:56:19.608 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T07:56:19.608 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2023-03-03T07:56:19.608 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2023-03-03T07:56:19.608 DEBUG:teuthology.orchestra.run.smithi100:> return f 2023-03-03T07:56:19.609 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T07:56:19.609 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:56:19.609 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.3")) 2023-03-03T07:56:19.609 DEBUG:teuthology.orchestra.run.smithi100:> ' 2023-03-03T07:56:19.828 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.3.116028.asok 2023-03-03T07:56:20.006 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.116028.asok 2023-03-03T07:56:20.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:20.007 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.116028.asok status 2023-03-03T07:56:20.275 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2023-03-03T07:56:20.275 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2023-03-03T07:56:20.275 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T07:56:20.275 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T07:56:20.275 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "3", 2023-03-03T07:56:20.275 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2023-03-03T07:56:20.276 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2023-03-03T07:56:20.276 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "116028", 2023-03-03T07:56:20.276 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2023-03-03T07:56:20.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:56:20.276 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2023-03-03T07:56:20.276 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2023-03-03T07:56:20.276 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 4657, 2023-03-03T07:56:20.277 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2023-03-03T07:56:20.277 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2023-03-03T07:56:20.277 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2023-03-03T07:56:20.277 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 4657 2023-03-03T07:56:20.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:56:20.277 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T07:56:20.277 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T07:56:20.278 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.2:0", 2023-03-03T07:56:20.278 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 940400031 2023-03-03T07:56:20.278 INFO:teuthology.orchestra.run.smithi100.stdout: } 2023-03-03T07:56:20.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:56:20.278 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T07:56:20.278 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T07:56:20.278 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.2:0", 2023-03-03T07:56:20.278 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 940400031 2023-03-03T07:56:20.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:56:20.279 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.4657 192.168.0.2:0/940400031", 2023-03-03T07:56:20.279 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.2:0/940400031", 2023-03-03T07:56:20.279 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2023-03-03T07:56:20.279 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 7, 2023-03-03T07:56:20.279 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 24, 2023-03-03T07:56:20.279 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2023-03-03T07:56:20.279 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2023-03-03T07:56:20.280 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2023-03-03T07:56:20.280 INFO:teuthology.orchestra.run.smithi100.stdout:} 2023-03-03T07:56:20.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:20.297 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2023-03-03T07:56:20.297 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.0 2023-03-03T07:56:20.369 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2023-03-03T07:56:20.370 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-03-03T07:56:20.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:20.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-03-03T07:56:20.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:20.474 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2023-03-03T07:56:20.474 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.2 2023-03-03T07:56:20.543 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2023-03-03T07:56:20.544 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2023-03-03T07:56:20.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:20.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-03-03T07:56:20.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:20.648 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2023-03-03T07:56:20.648 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2023-03-03T07:56:20.719 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2023-03-03T07:56:20.719 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-03-03T07:56:20.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:20.720 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-03-03T07:56:20.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:20.822 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2023-03-03T07:56:20.822 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.3 2023-03-03T07:56:20.891 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2023-03-03T07:56:20.892 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2023-03-03T07:56:20.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:20.893 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-03-03T07:56:20.994 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-03-03T07:56:21.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:56:21.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.269+0000 7f765e59d700 1 Processor -- start 2023-03-03T07:56:21.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.269+0000 7f765e59d700 1 -- start start 2023-03-03T07:56:21.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.270+0000 7f765e59d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7658146360 0x7f7658160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:21.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.270+0000 7f765e59d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f76580b4ab0 0x7f76580b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:21.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.270+0000 7f765e59d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76581614c0 con 0x7f76580731e0 2023-03-03T07:56:21.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.271+0000 7f765e59d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76580b5460 con 0x7f76580b4ab0 2023-03-03T07:56:21.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.271+0000 7f765e59d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76580d10f0 con 0x7f7658146360 2023-03-03T07:56:21.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.271+0000 7f765dd9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f76580731e0 0x7f7658073660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58268/0 (socket says 172.21.15.98:58268) 2023-03-03T07:56:21.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.271+0000 7f765dd9c700 1 -- 172.21.15.98:0/1566394817 learned_addr learned my addr 172.21.15.98:0/1566394817 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:21.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.271+0000 7f765d59b700 1 --2- 172.21.15.98:0/1566394817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7658146360 0x7f7658160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:21.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.271+0000 7f765cd9a700 1 --2- 172.21.15.98:0/1566394817 >> v2:172.21.15.100:3300/0 conn(0x7f76580b4ab0 0x7f76580b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:21.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.271+0000 7f765d59b700 1 -- 172.21.15.98:0/1566394817 >> v2:172.21.15.100:3300/0 conn(0x7f76580b4ab0 msgr2=0x7f76580b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:21.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.271+0000 7f765d59b700 1 --2- 172.21.15.98:0/1566394817 >> v2:172.21.15.100:3300/0 conn(0x7f76580b4ab0 0x7f76580b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.271+0000 7f765d59b700 1 -- 172.21.15.98:0/1566394817 >> v1:172.21.15.98:6789/0 conn(0x7f76580731e0 legacy=0x7f7658073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:21.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.271+0000 7f764ffff700 1 -- 172.21.15.98:0/1566394817 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993296480 0 0) 0x7f7650001490 con 0x7f76580731e0 2023-03-03T07:56:21.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.271+0000 7f765d59b700 1 -- 172.21.15.98:0/1566394817 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76580cdf20 con 0x7f7658146360 2023-03-03T07:56:21.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.272+0000 7f765d59b700 1 --2- 172.21.15.98:0/1566394817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7658146360 0x7f7658160f80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7654009840 tx=0x7f7654009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d01255e60744ec67 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:21.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.272+0000 7f764ffff700 1 -- 172.21.15.98:0/1566394817 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654011830 con 0x7f7658146360 2023-03-03T07:56:21.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.272+0000 7f764ffff700 1 -- 172.21.15.98:0/1566394817 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7654011990 con 0x7f7658146360 2023-03-03T07:56:21.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.272+0000 7f764ffff700 1 -- 172.21.15.98:0/1566394817 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654011d20 con 0x7f7658146360 2023-03-03T07:56:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.273+0000 7f765e59d700 1 -- 172.21.15.98:0/1566394817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7658146360 msgr2=0x7f7658160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.273+0000 7f765e59d700 1 --2- 172.21.15.98:0/1566394817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7658146360 0x7f7658160f80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7654009840 tx=0x7f7654009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.273+0000 7f765e59d700 1 -- 172.21.15.98:0/1566394817 shutdown_connections 2023-03-03T07:56:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.273+0000 7f765e59d700 1 --2- 172.21.15.98:0/1566394817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7658146360 0x7f7658160f80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.273+0000 7f765e59d700 1 --2- 172.21.15.98:0/1566394817 >> v2:172.21.15.100:3300/0 conn(0x7f76580b4ab0 0x7f76580b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.273+0000 7f765e59d700 1 -- 172.21.15.98:0/1566394817 >> 172.21.15.98:0/1566394817 conn(0x7f765806b8e0 msgr2=0x7f76580cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.273+0000 7f765e59d700 1 -- 172.21.15.98:0/1566394817 shutdown_connections 2023-03-03T07:56:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.273+0000 7f765e59d700 1 -- 172.21.15.98:0/1566394817 wait complete. 2023-03-03T07:56:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.275+0000 7f765e59d700 1 Processor -- start 2023-03-03T07:56:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.275+0000 7f765e59d700 1 -- start start 2023-03-03T07:56:21.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.276+0000 7f765e59d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76580731e0 0x7f765815a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:21.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.276+0000 7f765e59d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76580b4ab0 0x7f76580cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:21.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.276+0000 7f765e59d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7658146360 0x7f7658149d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:21.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.276+0000 7f765e59d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76580cc3b0 con 0x7f76580731e0 2023-03-03T07:56:21.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.276+0000 7f765e59d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76580cb500 con 0x7f7658146360 2023-03-03T07:56:21.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.276+0000 7f765e59d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76580cb7a0 con 0x7f76580b4ab0 2023-03-03T07:56:21.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.276+0000 7f765d59b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76580731e0 0x7f765815a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:21.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.276+0000 7f765d59b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76580731e0 0x7f765815a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49378/0 (socket says 172.21.15.98:49378) 2023-03-03T07:56:21.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.276+0000 7f765d59b700 1 -- 172.21.15.98:0/2254504093 learned_addr learned my addr 172.21.15.98:0/2254504093 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:21.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.277+0000 7f765cd9a700 1 --2- 172.21.15.98:0/2254504093 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76580b4ab0 0x7f76580cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:21.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.277+0000 7f765dd9c700 1 --2- 172.21.15.98:0/2254504093 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7658146360 0x7f7658149d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:21.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.277+0000 7f765d59b700 1 -- 172.21.15.98:0/2254504093 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76580b4ab0 msgr2=0x7f76580cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:21.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.277+0000 7f765d59b700 1 --2- 172.21.15.98:0/2254504093 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76580b4ab0 0x7f76580cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.277+0000 7f765d59b700 1 -- 172.21.15.98:0/2254504093 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7658146360 msgr2=0x7f7658149d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:21.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.277+0000 7f765d59b700 1 --2- 172.21.15.98:0/2254504093 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7658146360 0x7f7658149d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.277+0000 7f765d59b700 1 -- 172.21.15.98:0/2254504093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76540094a0 con 0x7f76580731e0 2023-03-03T07:56:21.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.277+0000 7f765d59b700 1 --2- 172.21.15.98:0/2254504093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76580731e0 0x7f765815a360 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7654009fd0 tx=0x7f7654004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:21.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.278+0000 7f764dffb700 1 -- 172.21.15.98:0/2254504093 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76540036a0 con 0x7f76580731e0 2023-03-03T07:56:21.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.278+0000 7f764dffb700 1 -- 172.21.15.98:0/2254504093 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765401f070 con 0x7f76580731e0 2023-03-03T07:56:21.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.278+0000 7f764dffb700 1 -- 172.21.15.98:0/2254504093 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654003a60 con 0x7f76580731e0 2023-03-03T07:56:21.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.278+0000 7f765e59d700 1 -- 172.21.15.98:0/2254504093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76580cd960 con 0x7f76580731e0 2023-03-03T07:56:21.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.278+0000 7f765e59d700 1 -- 172.21.15.98:0/2254504093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76580cd680 con 0x7f76580731e0 2023-03-03T07:56:21.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.279+0000 7f765e59d700 1 -- 172.21.15.98:0/2254504093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76580cdc40 con 0x7f76580731e0 2023-03-03T07:56:21.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.280+0000 7f764dffb700 1 -- 172.21.15.98:0/2254504093 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f765400f050 con 0x7f76580731e0 2023-03-03T07:56:21.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.281+0000 7f764dffb700 1 --2- 172.21.15.98:0/2254504093 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f763c0682f0 0x7f763c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:21.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.281+0000 7f764dffb700 1 -- 172.21.15.98:0/2254504093 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7654021070 con 0x7f76580731e0 2023-03-03T07:56:21.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.281+0000 7f765cd9a700 1 --2- 172.21.15.98:0/2254504093 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f763c0682f0 0x7f763c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:21.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.282+0000 7f765cd9a700 1 --2- 172.21.15.98:0/2254504093 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f763c0682f0 0x7f763c06a7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7648009b40 tx=0x7f7648009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:21.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.286+0000 7f764dffb700 1 -- 172.21.15.98:0/2254504093 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7654053cb0 con 0x7f76580731e0 2023-03-03T07:56:21.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.481+0000 7f765e59d700 1 -- 172.21.15.98:0/2254504093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7658068300 con 0x7f76580731e0 2023-03-03T07:56:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.482+0000 7f764dffb700 1 -- 172.21.15.98:0/2254504093 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17907 (secure 0 0 0) 0x7f76540572d0 con 0x7f76580731e0 2023-03-03T07:56:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:56:21.484 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:55:48.099844+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:56:21.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.485+0000 7f765e59d700 1 -- 172.21.15.98:0/2254504093 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f763c0682f0 msgr2=0x7f763c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.485+0000 7f765e59d700 1 --2- 172.21.15.98:0/2254504093 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f763c0682f0 0x7f763c06a7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7648009b40 tx=0x7f7648009210 comp rx=0 tx=0).stop 2023-03-03T07:56:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.485+0000 7f765e59d700 1 -- 172.21.15.98:0/2254504093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76580731e0 msgr2=0x7f765815a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.485+0000 7f765e59d700 1 --2- 172.21.15.98:0/2254504093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76580731e0 0x7f765815a360 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7654009fd0 tx=0x7f7654004750 comp rx=0 tx=0).stop 2023-03-03T07:56:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.485+0000 7f765e59d700 1 -- 172.21.15.98:0/2254504093 shutdown_connections 2023-03-03T07:56:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.485+0000 7f765e59d700 1 --2- 172.21.15.98:0/2254504093 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f763c0682f0 0x7f763c06a7a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.485+0000 7f765e59d700 1 --2- 172.21.15.98:0/2254504093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76580731e0 0x7f765815a360 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.485+0000 7f765e59d700 1 --2- 172.21.15.98:0/2254504093 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7658146360 0x7f7658149d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.485+0000 7f765e59d700 1 --2- 172.21.15.98:0/2254504093 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76580b4ab0 0x7f76580cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.485+0000 7f765e59d700 1 -- 172.21.15.98:0/2254504093 >> 172.21.15.98:0/2254504093 conn(0x7f765806b8e0 msgr2=0x7f7658073660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.485+0000 7f765e59d700 1 -- 172.21.15.98:0/2254504093 shutdown_connections 2023-03-03T07:56:21.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.485+0000 7f765e59d700 1 -- 172.21.15.98:0/2254504093 wait complete. 2023-03-03T07:56:21.513 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:55:45.157666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:55:46.092091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:56:21.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:56:21.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.761+0000 7f5ff139d700 1 Processor -- start 2023-03-03T07:56:21.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.762+0000 7f5ff139d700 1 -- start start 2023-03-03T07:56:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.763+0000 7f5ff139d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fec0cf570 0x7f5fec15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.763+0000 7f5ff139d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5fec15db30 0x7f5fec0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.763+0000 7f5ff139d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec0cd780 con 0x7f5fec1609b0 2023-03-03T07:56:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.763+0000 7f5ff139d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fec0cebb0 con 0x7f5fec15db30 2023-03-03T07:56:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.763+0000 7f5ff139d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0cda60 con 0x7f5fec0cf570 2023-03-03T07:56:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.764+0000 7f5ff0b9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5fec15db30 0x7f5fec0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.764+0000 7f5ff0b9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5fec15db30 0x7f5fec0b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49380/0 (socket says 172.21.15.98:49380) 2023-03-03T07:56:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.764+0000 7f5ff0b9c700 1 -- 172.21.15.98:0/2229155562 learned_addr learned my addr 172.21.15.98:0/2229155562 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.764+0000 7f5fe3fff700 1 --2- 172.21.15.98:0/2229155562 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fec0cf570 0x7f5fec15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.764+0000 7f5ff0b9c700 1 -- 172.21.15.98:0/2229155562 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fec0cf570 msgr2=0x7f5fec15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.764+0000 7f5ff0b9c700 1 --2- 172.21.15.98:0/2229155562 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fec0cf570 0x7f5fec15d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.764+0000 7f5ff0b9c700 1 -- 172.21.15.98:0/2229155562 >> v1:172.21.15.100:6789/0 conn(0x7f5fec1609b0 legacy=0x7f5fec160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.764+0000 7f5ff0b9c700 1 -- 172.21.15.98:0/2229155562 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec0cdd40 con 0x7f5fec15db30 2023-03-03T07:56:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.765+0000 7f5ff0b9c700 1 --2- 172.21.15.98:0/2229155562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fec15db30 0x7f5fec0b4aa0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4009e50 tx=0x7f5fe400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed87a67293667a61 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.765+0000 7f5feb7fe700 1 -- 172.21.15.98:0/2229155562 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe400e070 con 0x7f5fec15db30 2023-03-03T07:56:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.765+0000 7f5feb7fe700 1 -- 172.21.15.98:0/2229155562 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe400b910 con 0x7f5fec15db30 2023-03-03T07:56:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.765+0000 7f5feb7fe700 1 -- 172.21.15.98:0/2229155562 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe400bcc0 con 0x7f5fec15db30 2023-03-03T07:56:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.765+0000 7f5ff139d700 1 -- 172.21.15.98:0/2229155562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fec15db30 msgr2=0x7f5fec0b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.765+0000 7f5ff139d700 1 --2- 172.21.15.98:0/2229155562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fec15db30 0x7f5fec0b4aa0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4009e50 tx=0x7f5fe400b330 comp rx=0 tx=0).stop 2023-03-03T07:56:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.766+0000 7f5ff139d700 1 -- 172.21.15.98:0/2229155562 shutdown_connections 2023-03-03T07:56:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.766+0000 7f5ff139d700 1 --2- 172.21.15.98:0/2229155562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fec15db30 0x7f5fec0b4aa0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.766+0000 7f5ff139d700 1 --2- 172.21.15.98:0/2229155562 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fec0cf570 0x7f5fec15d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.766+0000 7f5ff139d700 1 -- 172.21.15.98:0/2229155562 >> 172.21.15.98:0/2229155562 conn(0x7f5fec06b880 msgr2=0x7f5fec0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.766+0000 7f5ff139d700 1 -- 172.21.15.98:0/2229155562 shutdown_connections 2023-03-03T07:56:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.766+0000 7f5ff139d700 1 -- 172.21.15.98:0/2229155562 wait complete. 2023-03-03T07:56:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.768+0000 7f5ff139d700 1 Processor -- start 2023-03-03T07:56:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.768+0000 7f5ff139d700 1 -- start start 2023-03-03T07:56:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.769+0000 7f5ff139d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fec0cf570 0x7f5fec14e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.769+0000 7f5ff139d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fec15db30 0x7f5fec0b3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.769+0000 7f5ff139d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fec1609b0 0x7f5fec150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.769+0000 7f5ff139d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0b4430 con 0x7f5fec15db30 2023-03-03T07:56:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.769+0000 7f5ff139d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec1510d0 con 0x7f5fec1609b0 2023-03-03T07:56:21.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.769+0000 7f5ff139d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0b6080 con 0x7f5fec0cf570 2023-03-03T07:56:21.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.769+0000 7f5fe3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fec15db30 0x7f5fec0b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:21.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.769+0000 7f5fe3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fec15db30 0x7f5fec0b3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49386/0 (socket says 172.21.15.98:49386) 2023-03-03T07:56:21.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.769+0000 7f5fe3fff700 1 -- 172.21.15.98:0/721987930 learned_addr learned my addr 172.21.15.98:0/721987930 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:21.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.770+0000 7f5ff0b9c700 1 --2- 172.21.15.98:0/721987930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fec1609b0 0x7f5fec150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:21.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.770+0000 7f5febfff700 1 --2- 172.21.15.98:0/721987930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fec0cf570 0x7f5fec14e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:21.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.770+0000 7f5fe3fff700 1 -- 172.21.15.98:0/721987930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fec0cf570 msgr2=0x7f5fec14e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:21.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.770+0000 7f5fe3fff700 1 --2- 172.21.15.98:0/721987930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fec0cf570 0x7f5fec14e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.770+0000 7f5fe3fff700 1 -- 172.21.15.98:0/721987930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fec1609b0 msgr2=0x7f5fec150b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:21.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.770+0000 7f5fe3fff700 1 --2- 172.21.15.98:0/721987930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fec1609b0 0x7f5fec150b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.770+0000 7f5fe3fff700 1 -- 172.21.15.98:0/721987930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe4009b30 con 0x7f5fec15db30 2023-03-03T07:56:21.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.770+0000 7f5fe3fff700 1 --2- 172.21.15.98:0/721987930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fec15db30 0x7f5fec0b3490 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400e9b0 tx=0x7f5fd400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:21.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.770+0000 7f5fe97fa700 1 -- 172.21.15.98:0/721987930 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4009710 con 0x7f5fec15db30 2023-03-03T07:56:21.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.771+0000 7f5fe97fa700 1 -- 172.21.15.98:0/721987930 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd400ee50 con 0x7f5fec15db30 2023-03-03T07:56:21.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.771+0000 7f5fe97fa700 1 -- 172.21.15.98:0/721987930 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd400ca50 con 0x7f5fec15db30 2023-03-03T07:56:21.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.771+0000 7f5ff139d700 1 -- 172.21.15.98:0/721987930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fec15d0d0 con 0x7f5fec15db30 2023-03-03T07:56:21.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.771+0000 7f5ff139d700 1 -- 172.21.15.98:0/721987930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fec15cdf0 con 0x7f5fec15db30 2023-03-03T07:56:21.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.772+0000 7f5ff139d700 1 -- 172.21.15.98:0/721987930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fec0b4fe0 con 0x7f5fec15db30 2023-03-03T07:56:21.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.773+0000 7f5fe97fa700 1 -- 172.21.15.98:0/721987930 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5fd4019030 con 0x7f5fec15db30 2023-03-03T07:56:21.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.774+0000 7f5fe97fa700 1 --2- 172.21.15.98:0/721987930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5fcc0682f0 0x7f5fcc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:21.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.775+0000 7f5fe97fa700 1 -- 172.21.15.98:0/721987930 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5fd4085c00 con 0x7f5fec15db30 2023-03-03T07:56:21.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.775+0000 7f5febfff700 1 --2- 172.21.15.98:0/721987930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5fcc0682f0 0x7f5fcc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:21.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.775+0000 7f5febfff700 1 --2- 172.21.15.98:0/721987930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5fcc0682f0 0x7f5fcc06a7a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc007fd0 tx=0x7f5fdc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.778+0000 7f5fe97fa700 1 -- 172.21.15.98:0/721987930 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5fd4052370 con 0x7f5fec15db30 2023-03-03T07:56:21.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.973+0000 7f5ff139d700 1 -- 172.21.15.98:0/721987930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5fec0b61f0 con 0x7f5fec15db30 2023-03-03T07:56:21.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.973+0000 7f5fe97fa700 1 -- 172.21.15.98:0/721987930 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5fd4055990 con 0x7f5fec15db30 2023-03-03T07:56:21.976 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:56:21.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.976+0000 7f5ff139d700 1 -- 172.21.15.98:0/721987930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5fcc0682f0 msgr2=0x7f5fcc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:21.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.976+0000 7f5ff139d700 1 --2- 172.21.15.98:0/721987930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5fcc0682f0 0x7f5fcc06a7a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc007fd0 tx=0x7f5fdc009040 comp rx=0 tx=0).stop 2023-03-03T07:56:21.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.976+0000 7f5ff139d700 1 -- 172.21.15.98:0/721987930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fec15db30 msgr2=0x7f5fec0b3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:21.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.976+0000 7f5ff139d700 1 --2- 172.21.15.98:0/721987930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fec15db30 0x7f5fec0b3490 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400e9b0 tx=0x7f5fd400ecc0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.976+0000 7f5ff139d700 1 -- 172.21.15.98:0/721987930 shutdown_connections 2023-03-03T07:56:21.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.976+0000 7f5ff139d700 1 --2- 172.21.15.98:0/721987930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5fcc0682f0 0x7f5fcc06a7a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.976+0000 7f5ff139d700 1 --2- 172.21.15.98:0/721987930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fec1609b0 0x7f5fec150b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.976+0000 7f5ff139d700 1 --2- 172.21.15.98:0/721987930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5fec0cf570 0x7f5fec14e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.976+0000 7f5ff139d700 1 --2- 172.21.15.98:0/721987930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fec15db30 0x7f5fec0b3490 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:21.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.976+0000 7f5ff139d700 1 -- 172.21.15.98:0/721987930 >> 172.21.15.98:0/721987930 conn(0x7f5fec06b880 msgr2=0x7f5fec14f530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:21.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.976+0000 7f5ff139d700 1 -- 172.21.15.98:0/721987930 shutdown_connections 2023-03-03T07:56:21.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:21.976+0000 7f5ff139d700 1 -- 172.21.15.98:0/721987930 wait complete. 2023-03-03T07:56:22.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:56:22.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.253+0000 7f0d1a39d700 1 Processor -- start 2023-03-03T07:56:22.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.254+0000 7f0d1a39d700 1 -- start start 2023-03-03T07:56:22.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.255+0000 7f0d1a39d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d141609b0 0x7f0d14160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:22.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.255+0000 7f0d1a39d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0d140b4aa0 0x7f0d140b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:22.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.255+0000 7f0d1a39d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d141612c0 con 0x7f0d140cf570 2023-03-03T07:56:22.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.255+0000 7f0d1a39d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d140cebb0 con 0x7f0d141609b0 2023-03-03T07:56:22.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.255+0000 7f0d1a39d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d140b5460 con 0x7f0d140b4aa0 2023-03-03T07:56:22.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.255+0000 7f0d1939b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d141609b0 0x7f0d14160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:22.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.255+0000 7f0d18b9a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0d140cf570 0x7f0d1415d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58282/0 (socket says 172.21.15.98:58282) 2023-03-03T07:56:22.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.255+0000 7f0d18b9a700 1 -- 172.21.15.98:0/605108946 learned_addr learned my addr 172.21.15.98:0/605108946 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:22.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.256+0000 7f0d19b9c700 1 --2- 172.21.15.98:0/605108946 >> v2:172.21.15.100:3300/0 conn(0x7f0d140b4aa0 0x7f0d140b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:22.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.256+0000 7f0d1939b700 1 -- 172.21.15.98:0/605108946 >> v2:172.21.15.100:3300/0 conn(0x7f0d140b4aa0 msgr2=0x7f0d140b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:22.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.256+0000 7f0d1939b700 1 --2- 172.21.15.98:0/605108946 >> v2:172.21.15.100:3300/0 conn(0x7f0d140b4aa0 0x7f0d140b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.256+0000 7f0d1939b700 1 -- 172.21.15.98:0/605108946 >> v1:172.21.15.98:6789/0 conn(0x7f0d140cf570 legacy=0x7f0d1415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:22.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.256+0000 7f0d1939b700 1 -- 172.21.15.98:0/605108946 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1415dd00 con 0x7f0d141609b0 2023-03-03T07:56:22.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.256+0000 7f0d1939b700 1 --2- 172.21.15.98:0/605108946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d141609b0 0x7f0d14160d80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0d1406d4d0 tx=0x7f0d10009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf566ef9c9fb37b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:22.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.256+0000 7f0d0bfff700 1 -- 172.21.15.98:0/605108946 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10011830 con 0x7f0d141609b0 2023-03-03T07:56:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.256+0000 7f0d0bfff700 1 -- 172.21.15.98:0/605108946 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d10011990 con 0x7f0d141609b0 2023-03-03T07:56:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.256+0000 7f0d0bfff700 1 -- 172.21.15.98:0/605108946 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10011d20 con 0x7f0d141609b0 2023-03-03T07:56:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.257+0000 7f0d1a39d700 1 -- 172.21.15.98:0/605108946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d141609b0 msgr2=0x7f0d14160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.257+0000 7f0d1a39d700 1 --2- 172.21.15.98:0/605108946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d141609b0 0x7f0d14160d80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0d1406d4d0 tx=0x7f0d10009b20 comp rx=0 tx=0).stop 2023-03-03T07:56:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.257+0000 7f0d1a39d700 1 -- 172.21.15.98:0/605108946 shutdown_connections 2023-03-03T07:56:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.257+0000 7f0d1a39d700 1 --2- 172.21.15.98:0/605108946 >> v2:172.21.15.100:3300/0 conn(0x7f0d140b4aa0 0x7f0d140b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.257+0000 7f0d1a39d700 1 --2- 172.21.15.98:0/605108946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d141609b0 0x7f0d14160d80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.257+0000 7f0d1a39d700 1 -- 172.21.15.98:0/605108946 >> 172.21.15.98:0/605108946 conn(0x7f0d1406b880 msgr2=0x7f0d140cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.257+0000 7f0d1a39d700 1 -- 172.21.15.98:0/605108946 shutdown_connections 2023-03-03T07:56:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.257+0000 7f0d1a39d700 1 -- 172.21.15.98:0/605108946 wait complete. 2023-03-03T07:56:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.259+0000 7f0d1a39d700 1 Processor -- start 2023-03-03T07:56:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.260+0000 7f0d1a39d700 1 -- start start 2023-03-03T07:56:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.261+0000 7f0d1a39d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d140b4aa0 0x7f0d1414fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.261+0000 7f0d1a39d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d140cf570 0x7f0d141500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.261+0000 7f0d1a39d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d141609b0 0x7f0d141f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.261+0000 7f0d1a39d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d141514e0 con 0x7f0d141609b0 2023-03-03T07:56:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.261+0000 7f0d1a39d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d14150620 con 0x7f0d140cf570 2023-03-03T07:56:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.261+0000 7f0d1a39d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d141508d0 con 0x7f0d140b4aa0 2023-03-03T07:56:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.261+0000 7f0d19b9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d141609b0 0x7f0d141f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.261+0000 7f0d19b9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d141609b0 0x7f0d141f1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49404/0 (socket says 172.21.15.98:49404) 2023-03-03T07:56:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.261+0000 7f0d19b9c700 1 -- 172.21.15.98:0/1227920478 learned_addr learned my addr 172.21.15.98:0/1227920478 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.262+0000 7f0d1939b700 1 --2- 172.21.15.98:0/1227920478 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d140b4aa0 0x7f0d1414fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.262+0000 7f0d18b9a700 1 --2- 172.21.15.98:0/1227920478 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d140cf570 0x7f0d141500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.262+0000 7f0d19b9c700 1 -- 172.21.15.98:0/1227920478 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d140b4aa0 msgr2=0x7f0d1414fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.262+0000 7f0d19b9c700 1 --2- 172.21.15.98:0/1227920478 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d140b4aa0 0x7f0d1414fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.262+0000 7f0d19b9c700 1 -- 172.21.15.98:0/1227920478 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d140cf570 msgr2=0x7f0d141500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.262+0000 7f0d19b9c700 1 --2- 172.21.15.98:0/1227920478 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d140cf570 0x7f0d141500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.262+0000 7f0d19b9c700 1 -- 172.21.15.98:0/1227920478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d100094a0 con 0x7f0d141609b0 2023-03-03T07:56:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.262+0000 7f0d19b9c700 1 --2- 172.21.15.98:0/1227920478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d141609b0 0x7f0d141f1e50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c007e40 tx=0x7f0d0c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.263+0000 7f0d09ffb700 1 -- 172.21.15.98:0/1227920478 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c00ed30 con 0x7f0d141609b0 2023-03-03T07:56:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.263+0000 7f0d09ffb700 1 -- 172.21.15.98:0/1227920478 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c014420 con 0x7f0d141609b0 2023-03-03T07:56:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.263+0000 7f0d09ffb700 1 -- 172.21.15.98:0/1227920478 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c0147d0 con 0x7f0d141609b0 2023-03-03T07:56:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.263+0000 7f0d1a39d700 1 -- 172.21.15.98:0/1227920478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d140cda60 con 0x7f0d141609b0 2023-03-03T07:56:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.263+0000 7f0d1a39d700 1 -- 172.21.15.98:0/1227920478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d140cd780 con 0x7f0d141609b0 2023-03-03T07:56:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.264+0000 7f0d1a39d700 1 -- 172.21.15.98:0/1227920478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d140cdd40 con 0x7f0d141609b0 2023-03-03T07:56:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.265+0000 7f0d09ffb700 1 -- 172.21.15.98:0/1227920478 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0d0c012050 con 0x7f0d141609b0 2023-03-03T07:56:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.267+0000 7f0d09ffb700 1 --2- 172.21.15.98:0/1227920478 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0cf80682f0 0x7f0cf806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.267+0000 7f0d09ffb700 1 -- 172.21.15.98:0/1227920478 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0d0c0862a0 con 0x7f0d141609b0 2023-03-03T07:56:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.267+0000 7f0d1939b700 1 --2- 172.21.15.98:0/1227920478 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0cf80682f0 0x7f0cf806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.268+0000 7f0d1939b700 1 --2- 172.21.15.98:0/1227920478 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0cf80682f0 0x7f0cf806a7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0d100090b0 tx=0x7f0d1001c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.270+0000 7f0d09ffb700 1 -- 172.21.15.98:0/1227920478 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0d0c052a10 con 0x7f0d141609b0 2023-03-03T07:56:22.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.464+0000 7f0d1a39d700 1 -- 172.21.15.98:0/1227920478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f0d14068290 con 0x7f0d141609b0 2023-03-03T07:56:22.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.465+0000 7f0d09ffb700 1 -- 172.21.15.98:0/1227920478 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0d0c056030 con 0x7f0d141609b0 2023-03-03T07:56:22.467 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:56:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.467+0000 7f0d1a39d700 1 -- 172.21.15.98:0/1227920478 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0cf80682f0 msgr2=0x7f0cf806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.467+0000 7f0d1a39d700 1 --2- 172.21.15.98:0/1227920478 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0cf80682f0 0x7f0cf806a7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0d100090b0 tx=0x7f0d1001c040 comp rx=0 tx=0).stop 2023-03-03T07:56:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.467+0000 7f0d1a39d700 1 -- 172.21.15.98:0/1227920478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d141609b0 msgr2=0x7f0d141f1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.467+0000 7f0d1a39d700 1 --2- 172.21.15.98:0/1227920478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d141609b0 0x7f0d141f1e50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c007e40 tx=0x7f0d0c00e3c0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.467+0000 7f0d1a39d700 1 -- 172.21.15.98:0/1227920478 shutdown_connections 2023-03-03T07:56:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.467+0000 7f0d1a39d700 1 --2- 172.21.15.98:0/1227920478 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0cf80682f0 0x7f0cf806a7a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.467+0000 7f0d1a39d700 1 --2- 172.21.15.98:0/1227920478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d141609b0 0x7f0d141f1e50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.467+0000 7f0d1a39d700 1 --2- 172.21.15.98:0/1227920478 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d140b4aa0 0x7f0d1414fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.467+0000 7f0d1a39d700 1 --2- 172.21.15.98:0/1227920478 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d140cf570 0x7f0d141500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.467+0000 7f0d1a39d700 1 -- 172.21.15.98:0/1227920478 >> 172.21.15.98:0/1227920478 conn(0x7f0d1406b880 msgr2=0x7f0d140b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.467+0000 7f0d1a39d700 1 -- 172.21.15.98:0/1227920478 shutdown_connections 2023-03-03T07:56:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.468+0000 7f0d1a39d700 1 -- 172.21.15.98:0/1227920478 wait complete. 2023-03-03T07:56:22.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:56:22.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.743+0000 7f3e0d3ff700 1 Processor -- start 2023-03-03T07:56:22.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.743+0000 7f3e0d3ff700 1 -- start start 2023-03-03T07:56:22.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.744+0000 7f3e0d3ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3e080b26d0 0x7f3e080b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:22.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.744+0000 7f3e0d3ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3e080b5140 0x7f3e080d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:22.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.744+0000 7f3e0d3ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e080cd7d0 con 0x7f3e080b8d10 2023-03-03T07:56:22.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.745+0000 7f3e0d3ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e080b55c0 con 0x7f3e080b26d0 2023-03-03T07:56:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.745+0000 7f3e0d3ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e080b90e0 con 0x7f3e080b5140 2023-03-03T07:56:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.745+0000 7f3e06ffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3e080b8d10 0x7f3e080cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58292/0 (socket says 172.21.15.98:58292) 2023-03-03T07:56:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.745+0000 7f3e06ffd700 1 -- 172.21.15.98:0/1440628400 learned_addr learned my addr 172.21.15.98:0/1440628400 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.745+0000 7f3dfffff700 1 --2- 172.21.15.98:0/1440628400 >> v2:172.21.15.98:3300/0 conn(0x7f3e080b26d0 0x7f3e080b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.745+0000 7f3e077fe700 1 --2- 172.21.15.98:0/1440628400 >> v2:172.21.15.100:3300/0 conn(0x7f3e080b5140 0x7f3e080d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.745+0000 7f3e067fc700 1 -- 172.21.15.98:0/1440628400 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969980937 0 0) 0x7f3e080cd7d0 con 0x7f3e080b8d10 2023-03-03T07:56:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.745+0000 7f3e067fc700 1 -- 172.21.15.98:0/1440628400 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3df40035b0 con 0x7f3e080b8d10 2023-03-03T07:56:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.745+0000 7f3dfffff700 1 -- 172.21.15.98:0/1440628400 >> v2:172.21.15.100:3300/0 conn(0x7f3e080b5140 msgr2=0x7f3e080d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.745+0000 7f3dfffff700 1 --2- 172.21.15.98:0/1440628400 >> v2:172.21.15.100:3300/0 conn(0x7f3e080b5140 0x7f3e080d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.745+0000 7f3dfffff700 1 -- 172.21.15.98:0/1440628400 >> v1:172.21.15.98:6789/0 conn(0x7f3e080b8d10 legacy=0x7f3e080cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.745+0000 7f3dfffff700 1 -- 172.21.15.98:0/1440628400 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e080cda20 con 0x7f3e080b26d0 2023-03-03T07:56:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.745+0000 7f3e077fe700 1 --2- 172.21.15.98:0/1440628400 >> v2:172.21.15.100:3300/0 conn(0x7f3e080b5140 0x7f3e080d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.746+0000 7f3dfffff700 1 --2- 172.21.15.98:0/1440628400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e080b26d0 0x7f3e080b4c00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3e080ca1e0 tx=0x7f3df0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab3e8c0881c88a8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.746+0000 7f3e067fc700 1 -- 172.21.15.98:0/1440628400 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df0011830 con 0x7f3e080b26d0 2023-03-03T07:56:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.746+0000 7f3e067fc700 1 -- 172.21.15.98:0/1440628400 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df0011990 con 0x7f3e080b26d0 2023-03-03T07:56:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.746+0000 7f3e067fc700 1 -- 172.21.15.98:0/1440628400 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df0011d20 con 0x7f3e080b26d0 2023-03-03T07:56:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.747+0000 7f3e0d3ff700 1 -- 172.21.15.98:0/1440628400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e080b26d0 msgr2=0x7f3e080b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.747+0000 7f3e0d3ff700 1 --2- 172.21.15.98:0/1440628400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e080b26d0 0x7f3e080b4c00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3e080ca1e0 tx=0x7f3df0009b20 comp rx=0 tx=0).stop 2023-03-03T07:56:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.747+0000 7f3e0d3ff700 1 -- 172.21.15.98:0/1440628400 shutdown_connections 2023-03-03T07:56:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.747+0000 7f3e0d3ff700 1 --2- 172.21.15.98:0/1440628400 >> v2:172.21.15.100:3300/0 conn(0x7f3e080b5140 0x7f3e080d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.747+0000 7f3e0d3ff700 1 --2- 172.21.15.98:0/1440628400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e080b26d0 0x7f3e080b4c00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.747+0000 7f3e0d3ff700 1 -- 172.21.15.98:0/1440628400 >> 172.21.15.98:0/1440628400 conn(0x7f3e08073e70 msgr2=0x7f3e0806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.747+0000 7f3e0d3ff700 1 -- 172.21.15.98:0/1440628400 shutdown_connections 2023-03-03T07:56:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.747+0000 7f3e0d3ff700 1 -- 172.21.15.98:0/1440628400 wait complete. 2023-03-03T07:56:22.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.749+0000 7f3e0d3ff700 1 Processor -- start 2023-03-03T07:56:22.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.749+0000 7f3e0d3ff700 1 -- start start 2023-03-03T07:56:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.750+0000 7f3e0d3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3e080b26d0 0x7f3e080cb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.750+0000 7f3e0d3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3e080b5140 0x7f3e0815f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.750+0000 7f3e0d3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e080b8d10 0x7f3e080c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.750+0000 7f3e0d3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0815eef0 con 0x7f3e080b8d10 2023-03-03T07:56:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.750+0000 7f3e0d3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e081518c0 con 0x7f3e080b5140 2023-03-03T07:56:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.750+0000 7f3e0d3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0815e770 con 0x7f3e080b26d0 2023-03-03T07:56:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.750+0000 7f3e077fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e080b8d10 0x7f3e080c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.750+0000 7f3e077fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e080b8d10 0x7f3e080c8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49434/0 (socket says 172.21.15.98:49434) 2023-03-03T07:56:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.750+0000 7f3e077fe700 1 -- 172.21.15.98:0/3317577284 learned_addr learned my addr 172.21.15.98:0/3317577284 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.751+0000 7f3e06ffd700 1 --2- 172.21.15.98:0/3317577284 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3e080b26d0 0x7f3e080cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.751+0000 7f3dfffff700 1 --2- 172.21.15.98:0/3317577284 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3e080b5140 0x7f3e0815f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.751+0000 7f3e077fe700 1 -- 172.21.15.98:0/3317577284 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3e080b26d0 msgr2=0x7f3e080cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.751+0000 7f3e077fe700 1 --2- 172.21.15.98:0/3317577284 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3e080b26d0 0x7f3e080cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.751+0000 7f3e077fe700 1 -- 172.21.15.98:0/3317577284 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3e080b5140 msgr2=0x7f3e0815f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.751+0000 7f3e077fe700 1 --2- 172.21.15.98:0/3317577284 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3e080b5140 0x7f3e0815f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.751+0000 7f3e077fe700 1 -- 172.21.15.98:0/3317577284 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df00094a0 con 0x7f3e080b8d10 2023-03-03T07:56:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.751+0000 7f3e077fe700 1 --2- 172.21.15.98:0/3317577284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e080b8d10 0x7f3e080c8de0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e00007e70 tx=0x7f3e0000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.751+0000 7f3dff7fe700 1 -- 172.21.15.98:0/3317577284 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0000ddd0 con 0x7f3e080b8d10 2023-03-03T07:56:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.751+0000 7f3dff7fe700 1 -- 172.21.15.98:0/3317577284 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e0000f040 con 0x7f3e080b8d10 2023-03-03T07:56:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.751+0000 7f3dff7fe700 1 -- 172.21.15.98:0/3317577284 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e00003c60 con 0x7f3e080b8d10 2023-03-03T07:56:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.752+0000 7f3e0d3ff700 1 -- 172.21.15.98:0/3317577284 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e080d0e80 con 0x7f3e080b8d10 2023-03-03T07:56:22.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.752+0000 7f3e0d3ff700 1 -- 172.21.15.98:0/3317577284 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e080cdd00 con 0x7f3e080b8d10 2023-03-03T07:56:22.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.753+0000 7f3e0d3ff700 1 -- 172.21.15.98:0/3317577284 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e080cdfe0 con 0x7f3e080b8d10 2023-03-03T07:56:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.754+0000 7f3dff7fe700 1 -- 172.21.15.98:0/3317577284 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3e0001c030 con 0x7f3e080b8d10 2023-03-03T07:56:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.755+0000 7f3dff7fe700 1 --2- 172.21.15.98:0/3317577284 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3de80682f0 0x7f3de806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.755+0000 7f3dff7fe700 1 -- 172.21.15.98:0/3317577284 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3e00085cc0 con 0x7f3e080b8d10 2023-03-03T07:56:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.755+0000 7f3e06ffd700 1 --2- 172.21.15.98:0/3317577284 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3de80682f0 0x7f3de806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.756+0000 7f3e06ffd700 1 --2- 172.21.15.98:0/3317577284 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3de80682f0 0x7f3de806a7a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3df80011e0 tx=0x7f3df8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:22.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.759+0000 7f3dff7fe700 1 -- 172.21.15.98:0/3317577284 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3e0008c080 con 0x7f3e080b8d10 2023-03-03T07:56:22.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.954+0000 7f3e0d3ff700 1 -- 172.21.15.98:0/3317577284 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3e0804e6e0 con 0x7f3e080b8d10 2023-03-03T07:56:22.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.954+0000 7f3dff7fe700 1 -- 172.21.15.98:0/3317577284 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3e0001a020 con 0x7f3e080b8d10 2023-03-03T07:56:22.956 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:56:22.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.957+0000 7f3e0d3ff700 1 -- 172.21.15.98:0/3317577284 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3de80682f0 msgr2=0x7f3de806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.957+0000 7f3e0d3ff700 1 --2- 172.21.15.98:0/3317577284 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3de80682f0 0x7f3de806a7a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3df80011e0 tx=0x7f3df8009040 comp rx=0 tx=0).stop 2023-03-03T07:56:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.957+0000 7f3e0d3ff700 1 -- 172.21.15.98:0/3317577284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e080b8d10 msgr2=0x7f3e080c8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.957+0000 7f3e0d3ff700 1 --2- 172.21.15.98:0/3317577284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e080b8d10 0x7f3e080c8de0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e00007e70 tx=0x7f3e0000d3b0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.957+0000 7f3e0d3ff700 1 -- 172.21.15.98:0/3317577284 shutdown_connections 2023-03-03T07:56:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.957+0000 7f3e0d3ff700 1 --2- 172.21.15.98:0/3317577284 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3de80682f0 0x7f3de806a7a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.957+0000 7f3e0d3ff700 1 --2- 172.21.15.98:0/3317577284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e080b8d10 0x7f3e080c8de0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.957+0000 7f3e0d3ff700 1 --2- 172.21.15.98:0/3317577284 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3e080b26d0 0x7f3e080cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.957+0000 7f3e0d3ff700 1 --2- 172.21.15.98:0/3317577284 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3e080b5140 0x7f3e0815f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.957+0000 7f3e0d3ff700 1 -- 172.21.15.98:0/3317577284 >> 172.21.15.98:0/3317577284 conn(0x7f3e08073e70 msgr2=0x7f3e080b5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.957+0000 7f3e0d3ff700 1 -- 172.21.15.98:0/3317577284 shutdown_connections 2023-03-03T07:56:22.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:22.957+0000 7f3e0d3ff700 1 -- 172.21.15.98:0/3317577284 wait complete. 2023-03-03T07:56:22.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:56:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.232+0000 7fd3ba79d700 1 Processor -- start 2023-03-03T07:56:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.233+0000 7fd3ba79d700 1 -- start start 2023-03-03T07:56:23.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.234+0000 7fd3ba79d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3b40b85f0 0x7fd3b415d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:23.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.234+0000 7fd3ba79d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3b40b4a10 0x7fd3b40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:23.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.234+0000 7fd3ba79d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b40d0d20 con 0x7fd3b40d2790 2023-03-03T07:56:23.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.234+0000 7fd3ba79d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b40b53f0 con 0x7fd3b40b4a10 2023-03-03T07:56:23.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.234+0000 7fd3ba79d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b40b5870 con 0x7fd3b40b85f0 2023-03-03T07:56:23.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.234+0000 7fd3b9f9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3b40b4a10 0x7fd3b40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:23.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.235+0000 7fd3b979b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3b40b85f0 0x7fd3b415d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:23.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.235+0000 7fd3b9f9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3b40b4a10 0x7fd3b40b4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49448/0 (socket says 172.21.15.98:49448) 2023-03-03T07:56:23.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.235+0000 7fd3b9f9c700 1 -- 172.21.15.98:0/39990211 learned_addr learned my addr 172.21.15.98:0/39990211 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:23.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.235+0000 7fd3b9f9c700 1 -- 172.21.15.98:0/39990211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3b40b85f0 msgr2=0x7fd3b415d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:23.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.235+0000 7fd3b9f9c700 1 --2- 172.21.15.98:0/39990211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3b40b85f0 0x7fd3b415d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.235+0000 7fd3b9f9c700 1 -- 172.21.15.98:0/39990211 >> v1:172.21.15.100:6789/0 conn(0x7fd3b40d2790 legacy=0x7fd3b40cf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.235+0000 7fd3b9f9c700 1 -- 172.21.15.98:0/39990211 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b40cdc40 con 0x7fd3b40b4a10 2023-03-03T07:56:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.235+0000 7fd3b9f9c700 1 --2- 172.21.15.98:0/39990211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b40b4a10 0x7fd3b40b4eb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009e50 tx=0x7fd3ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e53330fd7306092 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.236+0000 7fd3abfff700 1 -- 172.21.15.98:0/39990211 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac00e070 con 0x7fd3b40b4a10 2023-03-03T07:56:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.236+0000 7fd3abfff700 1 -- 172.21.15.98:0/39990211 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ac00b910 con 0x7fd3b40b4a10 2023-03-03T07:56:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.236+0000 7fd3abfff700 1 -- 172.21.15.98:0/39990211 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac00bcc0 con 0x7fd3b40b4a10 2023-03-03T07:56:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.236+0000 7fd3ba79d700 1 -- 172.21.15.98:0/39990211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b40b4a10 msgr2=0x7fd3b40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.236+0000 7fd3ba79d700 1 --2- 172.21.15.98:0/39990211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b40b4a10 0x7fd3b40b4eb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009e50 tx=0x7fd3ac00b330 comp rx=0 tx=0).stop 2023-03-03T07:56:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.236+0000 7fd3ba79d700 1 -- 172.21.15.98:0/39990211 shutdown_connections 2023-03-03T07:56:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.236+0000 7fd3ba79d700 1 --2- 172.21.15.98:0/39990211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b40b4a10 0x7fd3b40b4eb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.236+0000 7fd3ba79d700 1 --2- 172.21.15.98:0/39990211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3b40b85f0 0x7fd3b415d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.236+0000 7fd3ba79d700 1 -- 172.21.15.98:0/39990211 >> 172.21.15.98:0/39990211 conn(0x7fd3b40d2f90 msgr2=0x7fd3b40d3390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.236+0000 7fd3ba79d700 1 -- 172.21.15.98:0/39990211 shutdown_connections 2023-03-03T07:56:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.236+0000 7fd3ba79d700 1 -- 172.21.15.98:0/39990211 wait complete. 2023-03-03T07:56:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.238+0000 7fd3ba79d700 1 Processor -- start 2023-03-03T07:56:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.239+0000 7fd3ba79d700 1 -- start start 2023-03-03T07:56:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3ba79d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3b40b4a10 0x7fd3b40cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3ba79d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b40b85f0 0x7fd3b40b66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3ba79d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3b40d2790 0x7fd3b41f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3ba79d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b41f2f30 con 0x7fd3b40b85f0 2023-03-03T07:56:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3ba79d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b40b6c30 con 0x7fd3b40b4a10 2023-03-03T07:56:23.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3ba79d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b41f2970 con 0x7fd3b40d2790 2023-03-03T07:56:23.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3b8f9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b40b85f0 0x7fd3b40b66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:23.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3b8f9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b40b85f0 0x7fd3b40b66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49458/0 (socket says 172.21.15.98:49458) 2023-03-03T07:56:23.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3b8f9a700 1 -- 172.21.15.98:0/3101896630 learned_addr learned my addr 172.21.15.98:0/3101896630 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:23.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3b9f9c700 1 --2- 172.21.15.98:0/3101896630 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3b40d2790 0x7fd3b41f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:23.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3b979b700 1 --2- 172.21.15.98:0/3101896630 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3b40b4a10 0x7fd3b40cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3b8f9a700 1 -- 172.21.15.98:0/3101896630 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3b40d2790 msgr2=0x7fd3b41f1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3b8f9a700 1 --2- 172.21.15.98:0/3101896630 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3b40d2790 0x7fd3b41f1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3b8f9a700 1 -- 172.21.15.98:0/3101896630 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3b40b4a10 msgr2=0x7fd3b40cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3b8f9a700 1 --2- 172.21.15.98:0/3101896630 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3b40b4a10 0x7fd3b40cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.240+0000 7fd3b8f9a700 1 -- 172.21.15.98:0/3101896630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac009b30 con 0x7fd3b40b85f0 2023-03-03T07:56:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.241+0000 7fd3b8f9a700 1 --2- 172.21.15.98:0/3101896630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b40b85f0 0x7fd3b40b66f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400c4b0 tx=0x7fd3a400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.241+0000 7fd3a9ffb700 1 -- 172.21.15.98:0/3101896630 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a40052d0 con 0x7fd3b40b85f0 2023-03-03T07:56:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.241+0000 7fd3a9ffb700 1 -- 172.21.15.98:0/3101896630 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a4005430 con 0x7fd3b40b85f0 2023-03-03T07:56:23.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.241+0000 7fd3a9ffb700 1 -- 172.21.15.98:0/3101896630 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a40038c0 con 0x7fd3b40b85f0 2023-03-03T07:56:23.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.241+0000 7fd3ba79d700 1 -- 172.21.15.98:0/3101896630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b40cd680 con 0x7fd3b40b85f0 2023-03-03T07:56:23.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.241+0000 7fd3ba79d700 1 -- 172.21.15.98:0/3101896630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b415dd10 con 0x7fd3b40b85f0 2023-03-03T07:56:23.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.242+0000 7fd3ba79d700 1 -- 172.21.15.98:0/3101896630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b40cd960 con 0x7fd3b40b85f0 2023-03-03T07:56:23.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.244+0000 7fd3a9ffb700 1 -- 172.21.15.98:0/3101896630 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd3a4019030 con 0x7fd3b40b85f0 2023-03-03T07:56:23.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.245+0000 7fd3a9ffb700 1 --2- 172.21.15.98:0/3101896630 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3980682f0 0x7fd39806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:23.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.245+0000 7fd3a9ffb700 1 -- 172.21.15.98:0/3101896630 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd3a4084f30 con 0x7fd3b40b85f0 2023-03-03T07:56:23.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.245+0000 7fd3b979b700 1 --2- 172.21.15.98:0/3101896630 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3980682f0 0x7fd39806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:23.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.246+0000 7fd3b979b700 1 --2- 172.21.15.98:0/3101896630 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3980682f0 0x7fd39806a7a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0009be0 tx=0x7fd3b0009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:23.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.249+0000 7fd3a9ffb700 1 -- 172.21.15.98:0/3101896630 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd3a40516a0 con 0x7fd3b40b85f0 2023-03-03T07:56:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.440+0000 7fd3ba79d700 1 -- 172.21.15.98:0/3101896630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd3b4068290 con 0x7fd3b40b85f0 2023-03-03T07:56:23.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.441+0000 7fd3a9ffb700 1 -- 172.21.15.98:0/3101896630 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd3a40170a0 con 0x7fd3b40b85f0 2023-03-03T07:56:23.442 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:56:23.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.443+0000 7fd3ba79d700 1 -- 172.21.15.98:0/3101896630 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3980682f0 msgr2=0x7fd39806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:23.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.443+0000 7fd3ba79d700 1 --2- 172.21.15.98:0/3101896630 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3980682f0 0x7fd39806a7a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0009be0 tx=0x7fd3b0009210 comp rx=0 tx=0).stop 2023-03-03T07:56:23.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.443+0000 7fd3ba79d700 1 -- 172.21.15.98:0/3101896630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b40b85f0 msgr2=0x7fd3b40b66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.443+0000 7fd3ba79d700 1 --2- 172.21.15.98:0/3101896630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b40b85f0 0x7fd3b40b66f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400c4b0 tx=0x7fd3a400c7c0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.443+0000 7fd3ba79d700 1 -- 172.21.15.98:0/3101896630 shutdown_connections 2023-03-03T07:56:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.443+0000 7fd3ba79d700 1 --2- 172.21.15.98:0/3101896630 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3980682f0 0x7fd39806a7a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.443+0000 7fd3ba79d700 1 --2- 172.21.15.98:0/3101896630 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3b40d2790 0x7fd3b41f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.443+0000 7fd3ba79d700 1 --2- 172.21.15.98:0/3101896630 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3b40b4a10 0x7fd3b40cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.443+0000 7fd3ba79d700 1 --2- 172.21.15.98:0/3101896630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b40b85f0 0x7fd3b40b66f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.443+0000 7fd3ba79d700 1 -- 172.21.15.98:0/3101896630 >> 172.21.15.98:0/3101896630 conn(0x7fd3b40d2f90 msgr2=0x7fd3b40b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.444+0000 7fd3ba79d700 1 -- 172.21.15.98:0/3101896630 shutdown_connections 2023-03-03T07:56:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.444+0000 7fd3ba79d700 1 -- 172.21.15.98:0/3101896630 wait complete. 2023-03-03T07:56:23.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:56:23.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.716+0000 7fbd9df9d700 1 Processor -- start 2023-03-03T07:56:23.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.716+0000 7fbd9df9d700 1 -- start start 2023-03-03T07:56:23.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.718+0000 7fbd9df9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fbd98146360 0x7fbd98160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.718+0000 7fbd9df9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbd980b4ab0 0x7fbd980b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.718+0000 7fbd9df9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd981614c0 con 0x7fbd980731e0 2023-03-03T07:56:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.718+0000 7fbd9df9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd980b5440 con 0x7fbd980b4ab0 2023-03-03T07:56:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.718+0000 7fbd9df9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd980d10f0 con 0x7fbd98146360 2023-03-03T07:56:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.718+0000 7fbd97fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbd980b4ab0 0x7fbd980b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.718+0000 7fbd97fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbd980b4ab0 0x7fbd980b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49474/0 (socket says 172.21.15.98:49474) 2023-03-03T07:56:23.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.718+0000 7fbd97fff700 1 -- 172.21.15.98:0/3927681920 learned_addr learned my addr 172.21.15.98:0/3927681920 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:23.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.718+0000 7fbd9cf9b700 1 --2- 172.21.15.98:0/3927681920 >> v2:172.21.15.100:3300/0 conn(0x7fbd98146360 0x7fbd98160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:23.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.719+0000 7fbd97fff700 1 -- 172.21.15.98:0/3927681920 >> v2:172.21.15.100:3300/0 conn(0x7fbd98146360 msgr2=0x7fbd98160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:23.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.719+0000 7fbd97fff700 1 --2- 172.21.15.98:0/3927681920 >> v2:172.21.15.100:3300/0 conn(0x7fbd98146360 0x7fbd98160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.719+0000 7fbd97fff700 1 -- 172.21.15.98:0/3927681920 >> v1:172.21.15.100:6789/0 conn(0x7fbd980731e0 legacy=0x7fbd98073600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:23.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.719+0000 7fbd97fff700 1 -- 172.21.15.98:0/3927681920 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd980cdf20 con 0x7fbd980b4ab0 2023-03-03T07:56:23.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.719+0000 7fbd977fe700 1 -- 172.21.15.98:0/3927681920 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504004009 0 0) 0x7fbd90001490 con 0x7fbd980731e0 2023-03-03T07:56:23.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.719+0000 7fbd97fff700 1 --2- 172.21.15.98:0/3927681920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd980b4ab0 0x7fbd980b4f00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbd980743f0 tx=0x7fbd84009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d00349afcfabc2bc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:23.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.719+0000 7fbd977fe700 1 -- 172.21.15.98:0/3927681920 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd84011830 con 0x7fbd980b4ab0 2023-03-03T07:56:23.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.720+0000 7fbd977fe700 1 -- 172.21.15.98:0/3927681920 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd84011990 con 0x7fbd980b4ab0 2023-03-03T07:56:23.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.720+0000 7fbd977fe700 1 -- 172.21.15.98:0/3927681920 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd84011d20 con 0x7fbd980b4ab0 2023-03-03T07:56:23.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.720+0000 7fbd9df9d700 1 -- 172.21.15.98:0/3927681920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd980b4ab0 msgr2=0x7fbd980b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:23.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.720+0000 7fbd9df9d700 1 --2- 172.21.15.98:0/3927681920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd980b4ab0 0x7fbd980b4f00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbd980743f0 tx=0x7fbd84009bd0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.720+0000 7fbd9df9d700 1 -- 172.21.15.98:0/3927681920 shutdown_connections 2023-03-03T07:56:23.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.720+0000 7fbd9df9d700 1 --2- 172.21.15.98:0/3927681920 >> v2:172.21.15.100:3300/0 conn(0x7fbd98146360 0x7fbd98160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.720+0000 7fbd9df9d700 1 --2- 172.21.15.98:0/3927681920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd980b4ab0 0x7fbd980b4f00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.720+0000 7fbd9df9d700 1 -- 172.21.15.98:0/3927681920 >> 172.21.15.98:0/3927681920 conn(0x7fbd9806b8e0 msgr2=0x7fbd980cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:23.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.720+0000 7fbd9df9d700 1 -- 172.21.15.98:0/3927681920 shutdown_connections 2023-03-03T07:56:23.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.720+0000 7fbd9df9d700 1 -- 172.21.15.98:0/3927681920 wait complete. 2023-03-03T07:56:23.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.722+0000 7fbd9df9d700 1 Processor -- start 2023-03-03T07:56:23.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.723+0000 7fbd9df9d700 1 -- start start 2023-03-03T07:56:23.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd9df9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbd980731e0 0x7fbd980cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:23.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd9df9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd980b4ab0 0x7fbd98151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:23.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd9df9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd98146360 0x7fbd980c5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:23.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd9df9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd98151fd0 con 0x7fbd980b4ab0 2023-03-03T07:56:23.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd9df9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd98152e10 con 0x7fbd980731e0 2023-03-03T07:56:23.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd9df9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd98152200 con 0x7fbd98146360 2023-03-03T07:56:23.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd9cf9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbd980731e0 0x7fbd980cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:23.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd97fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd980b4ab0 0x7fbd98151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:23.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd9cf9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbd980731e0 0x7fbd980cb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:44826/0 (socket says 172.21.15.98:44826) 2023-03-03T07:56:23.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd9cf9b700 1 -- 172.21.15.98:0/2807033618 learned_addr learned my addr 172.21.15.98:0/2807033618 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:23.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd9d79c700 1 --2- 172.21.15.98:0/2807033618 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd98146360 0x7fbd980c5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:23.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd97fff700 1 -- 172.21.15.98:0/2807033618 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd98146360 msgr2=0x7fbd980c5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:23.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd97fff700 1 --2- 172.21.15.98:0/2807033618 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd98146360 0x7fbd980c5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd97fff700 1 -- 172.21.15.98:0/2807033618 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbd980731e0 msgr2=0x7fbd980cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:23.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd97fff700 1 --2- 172.21.15.98:0/2807033618 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbd980731e0 0x7fbd980cb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.724+0000 7fbd97fff700 1 -- 172.21.15.98:0/2807033618 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd84009550 con 0x7fbd980b4ab0 2023-03-03T07:56:23.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.725+0000 7fbd97fff700 1 --2- 172.21.15.98:0/2807033618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd980b4ab0 0x7fbd98151a90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbd84016000 tx=0x7fbd84004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:23.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.725+0000 7fbd957fa700 1 -- 172.21.15.98:0/2807033618 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd840036a0 con 0x7fbd980b4ab0 2023-03-03T07:56:23.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.725+0000 7fbd957fa700 1 -- 172.21.15.98:0/2807033618 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd84003800 con 0x7fbd980b4ab0 2023-03-03T07:56:23.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.725+0000 7fbd957fa700 1 -- 172.21.15.98:0/2807033618 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd84003b70 con 0x7fbd980b4ab0 2023-03-03T07:56:23.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.725+0000 7fbd9df9d700 1 -- 172.21.15.98:0/2807033618 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd980cd960 con 0x7fbd980b4ab0 2023-03-03T07:56:23.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.725+0000 7fbd9df9d700 1 -- 172.21.15.98:0/2807033618 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd980cd680 con 0x7fbd980b4ab0 2023-03-03T07:56:23.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.726+0000 7fbd9df9d700 1 -- 172.21.15.98:0/2807033618 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd980cdc40 con 0x7fbd980b4ab0 2023-03-03T07:56:23.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.727+0000 7fbd957fa700 1 -- 172.21.15.98:0/2807033618 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbd8402d070 con 0x7fbd980b4ab0 2023-03-03T07:56:23.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.729+0000 7fbd957fa700 1 --2- 172.21.15.98:0/2807033618 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbd7c0682f0 0x7fbd7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:23.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.729+0000 7fbd957fa700 1 -- 172.21.15.98:0/2807033618 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fbd84021030 con 0x7fbd980b4ab0 2023-03-03T07:56:23.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.729+0000 7fbd9cf9b700 1 --2- 172.21.15.98:0/2807033618 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbd7c0682f0 0x7fbd7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:23.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.730+0000 7fbd9cf9b700 1 --2- 172.21.15.98:0/2807033618 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbd7c0682f0 0x7fbd7c06a7a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbd980b4a80 tx=0x7fbd8c009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:23.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.732+0000 7fbd957fa700 1 -- 172.21.15.98:0/2807033618 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbd84053b90 con 0x7fbd980b4ab0 2023-03-03T07:56:23.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.926+0000 7fbd9df9d700 1 -- 172.21.15.98:0/2807033618 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd980b4fa0 con 0x7fbd980b4ab0 2023-03-03T07:56:23.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.928+0000 7fbd957fa700 1 -- 172.21.15.98:0/2807033618 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17907 (secure 0 0 0) 0x7fbd84015090 con 0x7fbd980b4ab0 2023-03-03T07:56:23.929 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:56:23.930 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:55:48.099844+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:56:23.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.931+0000 7fbd9df9d700 1 -- 172.21.15.98:0/2807033618 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbd7c0682f0 msgr2=0x7fbd7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:23.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.931+0000 7fbd9df9d700 1 --2- 172.21.15.98:0/2807033618 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbd7c0682f0 0x7fbd7c06a7a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbd980b4a80 tx=0x7fbd8c009210 comp rx=0 tx=0).stop 2023-03-03T07:56:23.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.931+0000 7fbd9df9d700 1 -- 172.21.15.98:0/2807033618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd980b4ab0 msgr2=0x7fbd98151a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:23.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.931+0000 7fbd9df9d700 1 --2- 172.21.15.98:0/2807033618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd980b4ab0 0x7fbd98151a90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbd84016000 tx=0x7fbd84004750 comp rx=0 tx=0).stop 2023-03-03T07:56:23.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.931+0000 7fbd9df9d700 1 -- 172.21.15.98:0/2807033618 shutdown_connections 2023-03-03T07:56:23.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.931+0000 7fbd9df9d700 1 --2- 172.21.15.98:0/2807033618 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbd7c0682f0 0x7fbd7c06a7a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.931+0000 7fbd9df9d700 1 --2- 172.21.15.98:0/2807033618 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd98146360 0x7fbd980c5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.931+0000 7fbd9df9d700 1 --2- 172.21.15.98:0/2807033618 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbd980731e0 0x7fbd980cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.931+0000 7fbd9df9d700 1 --2- 172.21.15.98:0/2807033618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd980b4ab0 0x7fbd98151a90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:23.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.931+0000 7fbd9df9d700 1 -- 172.21.15.98:0/2807033618 >> 172.21.15.98:0/2807033618 conn(0x7fbd9806b8e0 msgr2=0x7fbd980735b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:23.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.931+0000 7fbd9df9d700 1 -- 172.21.15.98:0/2807033618 shutdown_connections 2023-03-03T07:56:23.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:23.931+0000 7fbd9df9d700 1 -- 172.21.15.98:0/2807033618 wait complete. 2023-03-03T07:56:23.959 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:55:45.157666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:55:46.092091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:56:23.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:56:24.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.208+0000 7f438a9ff700 1 Processor -- start 2023-03-03T07:56:24.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.208+0000 7f438a9ff700 1 -- start start 2023-03-03T07:56:24.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.209+0000 7f438a9ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43840b26d0 0x7f43840b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:24.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.209+0000 7f438a9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43840b5140 0x7f43840d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:24.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.210+0000 7f438a9ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43840cd7d0 con 0x7f43840b8d10 2023-03-03T07:56:24.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.210+0000 7f438a9ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43840b55c0 con 0x7f43840b26d0 2023-03-03T07:56:24.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.210+0000 7f438a9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43840b90e0 con 0x7f43840b5140 2023-03-03T07:56:24.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.210+0000 7f43837fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43840b26d0 0x7f43840b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:24.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.210+0000 7f43837fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43840b26d0 0x7f43840b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49492/0 (socket says 172.21.15.98:49492) 2023-03-03T07:56:24.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.210+0000 7f43837fe700 1 -- 172.21.15.98:0/1879440909 learned_addr learned my addr 172.21.15.98:0/1879440909 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:24.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.210+0000 7f4388f9c700 1 --2- 172.21.15.98:0/1879440909 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43840b5140 0x7f43840d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:24.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.210+0000 7f43837fe700 1 -- 172.21.15.98:0/1879440909 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43840b5140 msgr2=0x7f43840d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:24.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.210+0000 7f43837fe700 1 --2- 172.21.15.98:0/1879440909 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43840b5140 0x7f43840d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.210+0000 7f43837fe700 1 -- 172.21.15.98:0/1879440909 >> v1:172.21.15.100:6789/0 conn(0x7f43840b8d10 legacy=0x7f43840cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:24.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.210+0000 7f43837fe700 1 -- 172.21.15.98:0/1879440909 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43840cda20 con 0x7f43840b26d0 2023-03-03T07:56:24.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.211+0000 7f43837fe700 1 --2- 172.21.15.98:0/1879440909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43840b26d0 0x7f43840b4c00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f43780098f0 tx=0x7f4378009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdace3796c645115 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:24.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.211+0000 7f4382ffd700 1 -- 172.21.15.98:0/1879440909 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437800b8a0 con 0x7f43840b26d0 2023-03-03T07:56:24.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.211+0000 7f4382ffd700 1 -- 172.21.15.98:0/1879440909 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437800ba00 con 0x7f43840b26d0 2023-03-03T07:56:24.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.211+0000 7f4382ffd700 1 -- 172.21.15.98:0/1879440909 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437800bd90 con 0x7f43840b26d0 2023-03-03T07:56:24.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.212+0000 7f438a9ff700 1 -- 172.21.15.98:0/1879440909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43840b26d0 msgr2=0x7f43840b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:24.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.212+0000 7f438a9ff700 1 --2- 172.21.15.98:0/1879440909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43840b26d0 0x7f43840b4c00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f43780098f0 tx=0x7f4378009c00 comp rx=0 tx=0).stop 2023-03-03T07:56:24.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.212+0000 7f438a9ff700 1 -- 172.21.15.98:0/1879440909 shutdown_connections 2023-03-03T07:56:24.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.212+0000 7f438a9ff700 1 --2- 172.21.15.98:0/1879440909 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43840b5140 0x7f43840d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.212+0000 7f438a9ff700 1 --2- 172.21.15.98:0/1879440909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43840b26d0 0x7f43840b4c00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.212+0000 7f438a9ff700 1 -- 172.21.15.98:0/1879440909 >> 172.21.15.98:0/1879440909 conn(0x7f4384073e70 msgr2=0x7f438406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:24.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.212+0000 7f438a9ff700 1 -- 172.21.15.98:0/1879440909 shutdown_connections 2023-03-03T07:56:24.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.212+0000 7f438a9ff700 1 -- 172.21.15.98:0/1879440909 wait complete. 2023-03-03T07:56:24.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.214+0000 7f438a9ff700 1 Processor -- start 2023-03-03T07:56:24.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.214+0000 7f438a9ff700 1 -- start start 2023-03-03T07:56:24.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.215+0000 7f438a9ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43840b26d0 0x7f4384159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:24.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.215+0000 7f438a9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43840b5140 0x7f438415f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:24.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.215+0000 7f438a9ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43840b8d10 0x7f438415d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:24.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.215+0000 7f438a9ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43840cb080 con 0x7f43840b8d10 2023-03-03T07:56:24.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.215+0000 7f438a9ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43840cb360 con 0x7f43840b26d0 2023-03-03T07:56:24.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.215+0000 7f438a9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43840cb640 con 0x7f43840b5140 2023-03-03T07:56:24.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.215+0000 7f4388f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43840b8d10 0x7f438415d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:24.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.215+0000 7f4388f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43840b8d10 0x7f438415d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49502/0 (socket says 172.21.15.98:49502) 2023-03-03T07:56:24.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.215+0000 7f4388f9c700 1 -- 172.21.15.98:0/600892954 learned_addr learned my addr 172.21.15.98:0/600892954 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:24.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.216+0000 7f43837fe700 1 --2- 172.21.15.98:0/600892954 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43840b5140 0x7f438415f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:24.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.216+0000 7f4383fff700 1 --2- 172.21.15.98:0/600892954 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43840b26d0 0x7f4384159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:24.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.216+0000 7f4388f9c700 1 -- 172.21.15.98:0/600892954 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43840b5140 msgr2=0x7f438415f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:24.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.216+0000 7f4388f9c700 1 --2- 172.21.15.98:0/600892954 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43840b5140 0x7f438415f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.216+0000 7f4388f9c700 1 -- 172.21.15.98:0/600892954 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43840b26d0 msgr2=0x7f4384159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:24.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.216+0000 7f4388f9c700 1 --2- 172.21.15.98:0/600892954 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43840b26d0 0x7f4384159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.216+0000 7f4388f9c700 1 -- 172.21.15.98:0/600892954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4378009550 con 0x7f43840b8d10 2023-03-03T07:56:24.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.216+0000 7f4388f9c700 1 --2- 172.21.15.98:0/600892954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43840b8d10 0x7f438415d7e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f437c007e60 tx=0x7f437c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:24.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.216+0000 7f4380ff9700 1 -- 172.21.15.98:0/600892954 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c00ede0 con 0x7f43840b8d10 2023-03-03T07:56:24.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.216+0000 7f4380ff9700 1 -- 172.21.15.98:0/600892954 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437c014420 con 0x7f43840b8d10 2023-03-03T07:56:24.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.216+0000 7f4380ff9700 1 -- 172.21.15.98:0/600892954 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c0147d0 con 0x7f43840b8d10 2023-03-03T07:56:24.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.216+0000 7f438a9ff700 1 -- 172.21.15.98:0/600892954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43840d0e80 con 0x7f43840b8d10 2023-03-03T07:56:24.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.217+0000 7f438a9ff700 1 -- 172.21.15.98:0/600892954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43840cdd00 con 0x7f43840b8d10 2023-03-03T07:56:24.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.218+0000 7f438a9ff700 1 -- 172.21.15.98:0/600892954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43840cdfe0 con 0x7f43840b8d10 2023-03-03T07:56:24.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.219+0000 7f4380ff9700 1 -- 172.21.15.98:0/600892954 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f437c012050 con 0x7f43840b8d10 2023-03-03T07:56:24.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.221+0000 7f4380ff9700 1 --2- 172.21.15.98:0/600892954 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f43700682f0 0x7f437006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:24.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.221+0000 7f4380ff9700 1 -- 172.21.15.98:0/600892954 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f437c0863a0 con 0x7f43840b8d10 2023-03-03T07:56:24.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.221+0000 7f4383fff700 1 --2- 172.21.15.98:0/600892954 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f43700682f0 0x7f437006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.222+0000 7f4383fff700 1 --2- 172.21.15.98:0/600892954 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f43700682f0 0x7f437006a7a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4374000f60 tx=0x7f4374009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.225+0000 7f4380ff9700 1 -- 172.21.15.98:0/600892954 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f437c052b10 con 0x7f43840b8d10 2023-03-03T07:56:24.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.417+0000 7f438a9ff700 1 -- 172.21.15.98:0/600892954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f438404e6e0 con 0x7f43840b8d10 2023-03-03T07:56:24.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.418+0000 7f4380ff9700 1 -- 172.21.15.98:0/600892954 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f437c056130 con 0x7f43840b8d10 2023-03-03T07:56:24.419 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:56:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.420+0000 7f438a9ff700 1 -- 172.21.15.98:0/600892954 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f43700682f0 msgr2=0x7f437006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.420+0000 7f438a9ff700 1 --2- 172.21.15.98:0/600892954 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f43700682f0 0x7f437006a7a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4374000f60 tx=0x7f4374009040 comp rx=0 tx=0).stop 2023-03-03T07:56:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.420+0000 7f438a9ff700 1 -- 172.21.15.98:0/600892954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43840b8d10 msgr2=0x7f438415d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.420+0000 7f438a9ff700 1 --2- 172.21.15.98:0/600892954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43840b8d10 0x7f438415d7e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f437c007e60 tx=0x7f437c00e3c0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.420+0000 7f438a9ff700 1 -- 172.21.15.98:0/600892954 shutdown_connections 2023-03-03T07:56:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.420+0000 7f438a9ff700 1 --2- 172.21.15.98:0/600892954 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f43700682f0 0x7f437006a7a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.420+0000 7f438a9ff700 1 --2- 172.21.15.98:0/600892954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43840b8d10 0x7f438415d7e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.420+0000 7f438a9ff700 1 --2- 172.21.15.98:0/600892954 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43840b26d0 0x7f4384159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.420+0000 7f438a9ff700 1 --2- 172.21.15.98:0/600892954 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43840b5140 0x7f438415f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.420+0000 7f438a9ff700 1 -- 172.21.15.98:0/600892954 >> 172.21.15.98:0/600892954 conn(0x7f4384073e70 msgr2=0x7f438406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:24.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.421+0000 7f438a9ff700 1 -- 172.21.15.98:0/600892954 shutdown_connections 2023-03-03T07:56:24.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.421+0000 7f438a9ff700 1 -- 172.21.15.98:0/600892954 wait complete. 2023-03-03T07:56:24.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:56:24.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.696+0000 7f9afc3ff700 1 Processor -- start 2023-03-03T07:56:24.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.696+0000 7f9afc3ff700 1 -- start start 2023-03-03T07:56:24.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.697+0000 7f9afc3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9af415ca70 0x7f9af40cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:24.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.697+0000 7f9afc3ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9af40d0270 0x7f9af40d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:24.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.697+0000 7f9afc3ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af40cf980 con 0x7f9af406b4b0 2023-03-03T07:56:24.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.697+0000 7f9afc3ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9af40cd5b0 con 0x7f9af40d0270 2023-03-03T07:56:24.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.697+0000 7f9afc3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af40d1800 con 0x7f9af415ca70 2023-03-03T07:56:24.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.698+0000 7f9af999a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9af406b4b0 0x7f9af40cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58304/0 (socket says 172.21.15.98:58304) 2023-03-03T07:56:24.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.698+0000 7f9af999a700 1 -- 172.21.15.98:0/31205323 learned_addr learned my addr 172.21.15.98:0/31205323 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:24.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.698+0000 7f9afa19b700 1 --2- 172.21.15.98:0/31205323 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9af415ca70 0x7f9af40cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:24.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.698+0000 7f9afa99c700 1 --2- 172.21.15.98:0/31205323 >> v2:172.21.15.100:3300/0 conn(0x7f9af40d0270 0x7f9af40d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:24.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.698+0000 7f9af9199700 1 -- 172.21.15.98:0/31205323 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812608745 0 0) 0x7f9af40cf980 con 0x7f9af406b4b0 2023-03-03T07:56:24.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.698+0000 7f9af9199700 1 -- 172.21.15.98:0/31205323 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ae00035b0 con 0x7f9af406b4b0 2023-03-03T07:56:24.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9af9199700 1 -- 172.21.15.98:0/31205323 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3609009953 0 0) 0x7f9ae00035b0 con 0x7f9af406b4b0 2023-03-03T07:56:24.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9af9199700 1 -- 172.21.15.98:0/31205323 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9af415ca70 msgr2=0x7f9af40cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:24.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9af9199700 1 --2- 172.21.15.98:0/31205323 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9af415ca70 0x7f9af40cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9af9199700 1 -- 172.21.15.98:0/31205323 >> v2:172.21.15.100:3300/0 conn(0x7f9af40d0270 msgr2=0x7f9af40d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:24.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9af9199700 1 --2- 172.21.15.98:0/31205323 >> v2:172.21.15.100:3300/0 conn(0x7f9af40d0270 0x7f9af40d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9af9199700 1 -- 172.21.15.98:0/31205323 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af40cd900 con 0x7f9af406b4b0 2023-03-03T07:56:24.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9af9199700 1 -- 172.21.15.98:0/31205323 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f9ae4001670 con 0x7f9af406b4b0 2023-03-03T07:56:24.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9af9199700 1 -- 172.21.15.98:0/31205323 >> v1:172.21.15.98:6789/0 conn(0x7f9af406b4b0 legacy=0x7f9af40cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:24.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9af9199700 1 --2- 172.21.15.98:0/31205323 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9ae0003a00 0x7f9ae0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:24.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9af9199700 1 --2- 172.21.15.98:0/31205323 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9af415ca70 0x7f9ae0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:24.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9af9199700 1 --2- 172.21.15.98:0/31205323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae0004d90 0x7f9ae0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:24.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9af9199700 1 -- 172.21.15.98:0/31205323 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af40cd5b0 con 0x7f9ae0004d90 2023-03-03T07:56:24.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9af9199700 1 -- 172.21.15.98:0/31205323 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af40d1800 con 0x7f9ae0003a00 2023-03-03T07:56:24.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9af9199700 1 -- 172.21.15.98:0/31205323 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae000ba10 con 0x7f9af415ca70 2023-03-03T07:56:24.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9afa99c700 1 --2- 172.21.15.98:0/31205323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae0004d90 0x7f9ae0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:24.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9afa19b700 1 --2- 172.21.15.98:0/31205323 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9ae0003a00 0x7f9ae0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:24.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.699+0000 7f9af999a700 1 --2- 172.21.15.98:0/31205323 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9af415ca70 0x7f9ae0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:24.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.700+0000 7f9afa99c700 1 -- 172.21.15.98:0/31205323 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9af415ca70 msgr2=0x7f9ae0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:24.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.700+0000 7f9afa99c700 1 --2- 172.21.15.98:0/31205323 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9af415ca70 0x7f9ae0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.700+0000 7f9afa99c700 1 -- 172.21.15.98:0/31205323 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9ae0003a00 msgr2=0x7f9ae0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:24.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.700+0000 7f9afa99c700 1 --2- 172.21.15.98:0/31205323 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9ae0003a00 0x7f9ae0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.700+0000 7f9afa99c700 1 -- 172.21.15.98:0/31205323 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae00035b0 con 0x7f9ae0004d90 2023-03-03T07:56:24.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.700+0000 7f9afa99c700 1 --2- 172.21.15.98:0/31205323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae0004d90 0x7f9ae0005200 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009fc0 tx=0x7f9ae0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc26a795aaf20824 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:24.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.700+0000 7f9af9199700 1 -- 172.21.15.98:0/31205323 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec00ec30 con 0x7f9ae0004d90 2023-03-03T07:56:24.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.701+0000 7f9af9199700 1 -- 172.21.15.98:0/31205323 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec00f040 con 0x7f9ae0004d90 2023-03-03T07:56:24.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.701+0000 7f9af9199700 1 -- 172.21.15.98:0/31205323 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec014480 con 0x7f9ae0004d90 2023-03-03T07:56:24.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.701+0000 7f9afc3ff700 1 -- 172.21.15.98:0/31205323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae0004d90 msgr2=0x7f9ae0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:24.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.701+0000 7f9afc3ff700 1 --2- 172.21.15.98:0/31205323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae0004d90 0x7f9ae0005200 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009fc0 tx=0x7f9ae0005880 comp rx=0 tx=0).stop 2023-03-03T07:56:24.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.701+0000 7f9afa99c700 1 -- 172.21.15.98:0/31205323 reap_dead start 2023-03-03T07:56:24.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.701+0000 7f9afc3ff700 1 -- 172.21.15.98:0/31205323 shutdown_connections 2023-03-03T07:56:24.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.701+0000 7f9afc3ff700 1 -- 172.21.15.98:0/31205323 >> 172.21.15.98:0/31205323 conn(0x7f9af4073e70 msgr2=0x7f9af406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:24.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.701+0000 7f9afc3ff700 1 -- 172.21.15.98:0/31205323 shutdown_connections 2023-03-03T07:56:24.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.701+0000 7f9afc3ff700 1 -- 172.21.15.98:0/31205323 wait complete. 2023-03-03T07:56:24.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.703+0000 7f9afc3ff700 1 Processor -- start 2023-03-03T07:56:24.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.704+0000 7f9afc3ff700 1 -- start start 2023-03-03T07:56:24.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.705+0000 7f9afc3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af40b15a0 0x7f9af4159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:24.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.705+0000 7f9afc3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9af40b5610 0x7f9af40b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:24.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.705+0000 7f9afc3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9af40b3120 0x7f9af40cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:24.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.705+0000 7f9afc3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af415a8d0 con 0x7f9af40b15a0 2023-03-03T07:56:24.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.705+0000 7f9afc3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af40b5a80 con 0x7f9af40b3120 2023-03-03T07:56:24.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.705+0000 7f9afc3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af40b1970 con 0x7f9af40b5610 2023-03-03T07:56:24.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.705+0000 7f9afa19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af40b15a0 0x7f9af4159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:24.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.705+0000 7f9afa19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af40b15a0 0x7f9af4159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49524/0 (socket says 172.21.15.98:49524) 2023-03-03T07:56:24.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.705+0000 7f9afa19b700 1 -- 172.21.15.98:0/3834889223 learned_addr learned my addr 172.21.15.98:0/3834889223 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:24.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.705+0000 7f9afa99c700 1 --2- 172.21.15.98:0/3834889223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9af40b3120 0x7f9af40cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:24.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.705+0000 7f9af999a700 1 --2- 172.21.15.98:0/3834889223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9af40b5610 0x7f9af40b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.705+0000 7f9afa19b700 1 -- 172.21.15.98:0/3834889223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9af40b5610 msgr2=0x7f9af40b2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.705+0000 7f9afa19b700 1 --2- 172.21.15.98:0/3834889223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9af40b5610 0x7f9af40b2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.706+0000 7f9afa19b700 1 -- 172.21.15.98:0/3834889223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9af40b3120 msgr2=0x7f9af40cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.706+0000 7f9afa19b700 1 --2- 172.21.15.98:0/3834889223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9af40b3120 0x7f9af40cae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.706+0000 7f9afa19b700 1 -- 172.21.15.98:0/3834889223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af40d1aa0 con 0x7f9af40b15a0 2023-03-03T07:56:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.706+0000 7f9afa99c700 1 --2- 172.21.15.98:0/3834889223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9af40b3120 0x7f9af40cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.706+0000 7f9afa19b700 1 --2- 172.21.15.98:0/3834889223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af40b15a0 0x7f9af4159f80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9af0011aa0 tx=0x7f9af0011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.706+0000 7f9aeaffd700 1 -- 172.21.15.98:0/3834889223 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af000f940 con 0x7f9af40b15a0 2023-03-03T07:56:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.706+0000 7f9aeaffd700 1 -- 172.21.15.98:0/3834889223 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af000faa0 con 0x7f9af40b15a0 2023-03-03T07:56:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.706+0000 7f9afc3ff700 1 -- 172.21.15.98:0/3834889223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af40cdbe0 con 0x7f9af40b15a0 2023-03-03T07:56:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.706+0000 7f9aeaffd700 1 -- 172.21.15.98:0/3834889223 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af000fe50 con 0x7f9af40b15a0 2023-03-03T07:56:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.706+0000 7f9afc3ff700 1 -- 172.21.15.98:0/3834889223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af415ce40 con 0x7f9af40b15a0 2023-03-03T07:56:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.708+0000 7f9afc3ff700 1 -- 172.21.15.98:0/3834889223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af404e6e0 con 0x7f9af40b15a0 2023-03-03T07:56:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.709+0000 7f9aeaffd700 1 -- 172.21.15.98:0/3834889223 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9af0014440 con 0x7f9af40b15a0 2023-03-03T07:56:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.711+0000 7f9aeaffd700 1 --2- 172.21.15.98:0/3834889223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ad8068320 0x7f9ad806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.711+0000 7f9aeaffd700 1 -- 172.21.15.98:0/3834889223 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9af0086610 con 0x7f9af40b15a0 2023-03-03T07:56:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.711+0000 7f9af999a700 1 --2- 172.21.15.98:0/3834889223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ad8068320 0x7f9ad806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.712+0000 7f9af999a700 1 --2- 172.21.15.98:0/3834889223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ad8068320 0x7f9ad806a7d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40048d0 tx=0x7f9ae4007980 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:24.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.715+0000 7f9aeaffd700 1 -- 172.21.15.98:0/3834889223 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9af0052de0 con 0x7f9af40b15a0 2023-03-03T07:56:24.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.906+0000 7f9afc3ff700 1 -- 172.21.15.98:0/3834889223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9af40d0f40 con 0x7f9af40b15a0 2023-03-03T07:56:24.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.907+0000 7f9aeaffd700 1 -- 172.21.15.98:0/3834889223 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9af0056400 con 0x7f9af40b15a0 2023-03-03T07:56:24.909 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:56:24.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.910+0000 7f9afc3ff700 1 -- 172.21.15.98:0/3834889223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ad8068320 msgr2=0x7f9ad806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:24.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.910+0000 7f9afc3ff700 1 --2- 172.21.15.98:0/3834889223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ad8068320 0x7f9ad806a7d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40048d0 tx=0x7f9ae4007980 comp rx=0 tx=0).stop 2023-03-03T07:56:24.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.910+0000 7f9afc3ff700 1 -- 172.21.15.98:0/3834889223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af40b15a0 msgr2=0x7f9af4159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:24.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.910+0000 7f9afc3ff700 1 --2- 172.21.15.98:0/3834889223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af40b15a0 0x7f9af4159f80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9af0011aa0 tx=0x7f9af0011db0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.910+0000 7f9afc3ff700 1 -- 172.21.15.98:0/3834889223 shutdown_connections 2023-03-03T07:56:24.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.910+0000 7f9afc3ff700 1 --2- 172.21.15.98:0/3834889223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ad8068320 0x7f9ad806a7d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.910+0000 7f9afc3ff700 1 --2- 172.21.15.98:0/3834889223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af40b15a0 0x7f9af4159f80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.910+0000 7f9afc3ff700 1 --2- 172.21.15.98:0/3834889223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9af40b3120 0x7f9af40cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.910+0000 7f9afc3ff700 1 --2- 172.21.15.98:0/3834889223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9af40b5610 0x7f9af40b2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:24.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.910+0000 7f9afc3ff700 1 -- 172.21.15.98:0/3834889223 >> 172.21.15.98:0/3834889223 conn(0x7f9af4073e70 msgr2=0x7f9af415ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:24.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.910+0000 7f9afc3ff700 1 -- 172.21.15.98:0/3834889223 shutdown_connections 2023-03-03T07:56:24.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:24.910+0000 7f9afc3ff700 1 -- 172.21.15.98:0/3834889223 wait complete. 2023-03-03T07:56:24.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:56:25.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.185+0000 7fb612d9d700 1 Processor -- start 2023-03-03T07:56:25.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.185+0000 7fb612d9d700 1 -- start start 2023-03-03T07:56:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.187+0000 7fb612d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb60c0cf570 0x7fb60c15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.187+0000 7fb612d9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb60c15db30 0x7fb60c0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.187+0000 7fb612d9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb60c0cd780 con 0x7fb60c1609b0 2023-03-03T07:56:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.187+0000 7fb612d9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60c0cebb0 con 0x7fb60c15db30 2023-03-03T07:56:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.187+0000 7fb612d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c0cda60 con 0x7fb60c0cf570 2023-03-03T07:56:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.187+0000 7fb611d9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb60c1609b0 0x7fb60c160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39752/0 (socket says 172.21.15.98:39752) 2023-03-03T07:56:25.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.187+0000 7fb611d9b700 1 -- 172.21.15.98:0/4052280642 learned_addr learned my addr 172.21.15.98:0/4052280642 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:25.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.187+0000 7fb61159a700 1 --2- 172.21.15.98:0/4052280642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb60c0cf570 0x7fb60c15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:25.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.187+0000 7fb61259c700 1 --2- 172.21.15.98:0/4052280642 >> v2:172.21.15.100:3300/0 conn(0x7fb60c15db30 0x7fb60c0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:25.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.188+0000 7fb61159a700 1 -- 172.21.15.98:0/4052280642 >> v2:172.21.15.100:3300/0 conn(0x7fb60c15db30 msgr2=0x7fb60c0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:25.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.188+0000 7fb61159a700 1 --2- 172.21.15.98:0/4052280642 >> v2:172.21.15.100:3300/0 conn(0x7fb60c15db30 0x7fb60c0b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.188+0000 7fb610d99700 1 -- 172.21.15.98:0/4052280642 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538138467 0 0) 0x7fb60c0cd780 con 0x7fb60c1609b0 2023-03-03T07:56:25.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.188+0000 7fb61159a700 1 -- 172.21.15.98:0/4052280642 >> v1:172.21.15.98:6789/0 conn(0x7fb60c1609b0 legacy=0x7fb60c160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:25.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.188+0000 7fb61159a700 1 -- 172.21.15.98:0/4052280642 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c0cdd40 con 0x7fb60c0cf570 2023-03-03T07:56:25.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.188+0000 7fb61159a700 1 --2- 172.21.15.98:0/4052280642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb60c0cf570 0x7fb60c15d5f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc009840 tx=0x7fb5fc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6ee7b4b6a0a928 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:25.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.188+0000 7fb610d99700 1 -- 172.21.15.98:0/4052280642 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc011830 con 0x7fb60c0cf570 2023-03-03T07:56:25.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.188+0000 7fb610d99700 1 -- 172.21.15.98:0/4052280642 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5fc011990 con 0x7fb60c0cf570 2023-03-03T07:56:25.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.189+0000 7fb610d99700 1 -- 172.21.15.98:0/4052280642 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc011d20 con 0x7fb60c0cf570 2023-03-03T07:56:25.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.189+0000 7fb612d9d700 1 -- 172.21.15.98:0/4052280642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb60c0cf570 msgr2=0x7fb60c15d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:25.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.189+0000 7fb612d9d700 1 --2- 172.21.15.98:0/4052280642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb60c0cf570 0x7fb60c15d5f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc009840 tx=0x7fb5fc009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:25.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.189+0000 7fb612d9d700 1 -- 172.21.15.98:0/4052280642 shutdown_connections 2023-03-03T07:56:25.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.189+0000 7fb612d9d700 1 --2- 172.21.15.98:0/4052280642 >> v2:172.21.15.100:3300/0 conn(0x7fb60c15db30 0x7fb60c0b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.189+0000 7fb612d9d700 1 --2- 172.21.15.98:0/4052280642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb60c0cf570 0x7fb60c15d5f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.189+0000 7fb612d9d700 1 -- 172.21.15.98:0/4052280642 >> 172.21.15.98:0/4052280642 conn(0x7fb60c06b880 msgr2=0x7fb60c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:25.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.189+0000 7fb612d9d700 1 -- 172.21.15.98:0/4052280642 shutdown_connections 2023-03-03T07:56:25.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.189+0000 7fb612d9d700 1 -- 172.21.15.98:0/4052280642 wait complete. 2023-03-03T07:56:25.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.191+0000 7fb612d9d700 1 Processor -- start 2023-03-03T07:56:25.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.191+0000 7fb612d9d700 1 -- start start 2023-03-03T07:56:25.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.192+0000 7fb612d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb60c0cf570 0x7fb60c14f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:25.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.192+0000 7fb612d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb60c15db30 0x7fb60c0b8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:25.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.192+0000 7fb612d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb60c1609b0 0x7fb60c0b8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:25.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.192+0000 7fb612d9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c0d06e0 con 0x7fb60c1609b0 2023-03-03T07:56:25.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.192+0000 7fb612d9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c14fa30 con 0x7fb60c15db30 2023-03-03T07:56:25.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.192+0000 7fb612d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c0b6d40 con 0x7fb60c0cf570 2023-03-03T07:56:25.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.193+0000 7fb61259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb60c1609b0 0x7fb60c0b8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:25.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.193+0000 7fb61259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb60c1609b0 0x7fb60c0b8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40502/0 (socket says 172.21.15.98:40502) 2023-03-03T07:56:25.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.193+0000 7fb61259c700 1 -- 172.21.15.98:0/2315553281 learned_addr learned my addr 172.21.15.98:0/2315553281 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:25.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.193+0000 7fb611d9b700 1 --2- 172.21.15.98:0/2315553281 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb60c0cf570 0x7fb60c14f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:25.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.193+0000 7fb61159a700 1 --2- 172.21.15.98:0/2315553281 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb60c15db30 0x7fb60c0b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:25.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.193+0000 7fb61259c700 1 -- 172.21.15.98:0/2315553281 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb60c0cf570 msgr2=0x7fb60c14f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:25.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.193+0000 7fb61259c700 1 --2- 172.21.15.98:0/2315553281 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb60c0cf570 0x7fb60c14f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.193+0000 7fb61259c700 1 -- 172.21.15.98:0/2315553281 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb60c15db30 msgr2=0x7fb60c0b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:25.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.193+0000 7fb61259c700 1 --2- 172.21.15.98:0/2315553281 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb60c15db30 0x7fb60c0b8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.193+0000 7fb61259c700 1 -- 172.21.15.98:0/2315553281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc0094a0 con 0x7fb60c1609b0 2023-03-03T07:56:25.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.194+0000 7fb61259c700 1 --2- 172.21.15.98:0/2315553281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb60c1609b0 0x7fb60c0b8590 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb60400efc0 tx=0x7fb60400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:25.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.194+0000 7fb6027fc700 1 -- 172.21.15.98:0/2315553281 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb604009ea0 con 0x7fb60c1609b0 2023-03-03T07:56:25.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.194+0000 7fb6027fc700 1 -- 172.21.15.98:0/2315553281 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb60400f040 con 0x7fb60c1609b0 2023-03-03T07:56:25.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.194+0000 7fb612d9d700 1 -- 172.21.15.98:0/2315553281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60c15d0d0 con 0x7fb60c1609b0 2023-03-03T07:56:25.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.194+0000 7fb6027fc700 1 -- 172.21.15.98:0/2315553281 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60400ced0 con 0x7fb60c1609b0 2023-03-03T07:56:25.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.194+0000 7fb612d9d700 1 -- 172.21.15.98:0/2315553281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60c15cdf0 con 0x7fb60c1609b0 2023-03-03T07:56:25.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.195+0000 7fb612d9d700 1 -- 172.21.15.98:0/2315553281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb60c0b4fe0 con 0x7fb60c1609b0 2023-03-03T07:56:25.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.197+0000 7fb6027fc700 1 -- 172.21.15.98:0/2315553281 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb604014480 con 0x7fb60c1609b0 2023-03-03T07:56:25.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.199+0000 7fb6027fc700 1 --2- 172.21.15.98:0/2315553281 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb5f00682f0 0x7fb5f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:25.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.199+0000 7fb6027fc700 1 -- 172.21.15.98:0/2315553281 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb604086700 con 0x7fb60c1609b0 2023-03-03T07:56:25.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.199+0000 7fb611d9b700 1 --2- 172.21.15.98:0/2315553281 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb5f00682f0 0x7fb5f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:25.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.200+0000 7fb611d9b700 1 --2- 172.21.15.98:0/2315553281 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb5f00682f0 0x7fb5f006a7a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb608000f60 tx=0x7fb608007e70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:25.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.203+0000 7fb6027fc700 1 -- 172.21.15.98:0/2315553281 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb604052e10 con 0x7fb60c1609b0 2023-03-03T07:56:25.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.397+0000 7fb612d9d700 1 -- 172.21.15.98:0/2315553281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb60c068290 con 0x7fb60c1609b0 2023-03-03T07:56:25.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.397+0000 7fb6027fc700 1 -- 172.21.15.98:0/2315553281 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb604056430 con 0x7fb60c1609b0 2023-03-03T07:56:25.399 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:56:25.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.400+0000 7fb612d9d700 1 -- 172.21.15.98:0/2315553281 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb5f00682f0 msgr2=0x7fb5f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:25.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.400+0000 7fb612d9d700 1 --2- 172.21.15.98:0/2315553281 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb5f00682f0 0x7fb5f006a7a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb608000f60 tx=0x7fb608007e70 comp rx=0 tx=0).stop 2023-03-03T07:56:25.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.400+0000 7fb612d9d700 1 -- 172.21.15.98:0/2315553281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb60c1609b0 msgr2=0x7fb60c0b8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:25.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.400+0000 7fb612d9d700 1 --2- 172.21.15.98:0/2315553281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb60c1609b0 0x7fb60c0b8590 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb60400efc0 tx=0x7fb60400c3a0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.400+0000 7fb612d9d700 1 -- 172.21.15.98:0/2315553281 shutdown_connections 2023-03-03T07:56:25.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.400+0000 7fb612d9d700 1 --2- 172.21.15.98:0/2315553281 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb5f00682f0 0x7fb5f006a7a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.400+0000 7fb612d9d700 1 --2- 172.21.15.98:0/2315553281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb60c1609b0 0x7fb60c0b8590 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.400+0000 7fb612d9d700 1 --2- 172.21.15.98:0/2315553281 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb60c0cf570 0x7fb60c14f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.400+0000 7fb612d9d700 1 --2- 172.21.15.98:0/2315553281 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb60c15db30 0x7fb60c0b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.400+0000 7fb612d9d700 1 -- 172.21.15.98:0/2315553281 >> 172.21.15.98:0/2315553281 conn(0x7fb60c06b880 msgr2=0x7fb60c0b40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:25.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.400+0000 7fb612d9d700 1 -- 172.21.15.98:0/2315553281 shutdown_connections 2023-03-03T07:56:25.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.400+0000 7fb612d9d700 1 -- 172.21.15.98:0/2315553281 wait complete. 2023-03-03T07:56:25.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:56:25.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.676+0000 7fe8dc9ff700 1 Processor -- start 2023-03-03T07:56:25.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.676+0000 7fe8dc9ff700 1 -- start start 2023-03-03T07:56:25.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.677+0000 7fe8dc9ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe8d80b8d10 0x7fe8d80cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:25.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.677+0000 7fe8dc9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8d80b26d0 0x7fe8d80b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:25.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.677+0000 7fe8dc9ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe8d80b5140 0x7fe8d8160520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:25.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.677+0000 7fe8dc9ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8d80b55c0 con 0x7fe8d80b5140 2023-03-03T07:56:25.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.677+0000 7fe8dc9ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8d80cd7d0 con 0x7fe8d80b8d10 2023-03-03T07:56:25.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.677+0000 7fe8dc9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d80b90e0 con 0x7fe8d80b26d0 2023-03-03T07:56:25.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.678+0000 7fe8d6d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe8d80b5140 0x7fe8d8160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:25.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.678+0000 7fe8d6d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe8d80b5140 0x7fe8d8160520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40506/0 (socket says 172.21.15.98:40506) 2023-03-03T07:56:25.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.678+0000 7fe8d6d9d700 1 -- 172.21.15.98:0/3095276577 learned_addr learned my addr 172.21.15.98:0/3095276577 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:25.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.678+0000 7fe8d659c700 1 --2- 172.21.15.98:0/3095276577 >> v2:172.21.15.100:3300/0 conn(0x7fe8d80b8d10 0x7fe8d80cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:25.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.678+0000 7fe8d5d9b700 1 --2- 172.21.15.98:0/3095276577 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8d80b26d0 0x7fe8d80b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:25.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.678+0000 7fe8d6d9d700 1 -- 172.21.15.98:0/3095276577 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8d80b26d0 msgr2=0x7fe8d80b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:25.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.678+0000 7fe8d6d9d700 1 --2- 172.21.15.98:0/3095276577 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8d80b26d0 0x7fe8d80b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.678+0000 7fe8d6d9d700 1 -- 172.21.15.98:0/3095276577 >> v2:172.21.15.100:3300/0 conn(0x7fe8d80b8d10 msgr2=0x7fe8d80cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:25.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.678+0000 7fe8d6d9d700 1 --2- 172.21.15.98:0/3095276577 >> v2:172.21.15.100:3300/0 conn(0x7fe8d80b8d10 0x7fe8d80cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.678+0000 7fe8d6d9d700 1 -- 172.21.15.98:0/3095276577 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d80cda20 con 0x7fe8d80b5140 2023-03-03T07:56:25.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.678+0000 7fe8d659c700 1 --2- 172.21.15.98:0/3095276577 >> v2:172.21.15.100:3300/0 conn(0x7fe8d80b8d10 0x7fe8d80cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:56:25.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.678+0000 7fe8d5d9b700 1 --2- 172.21.15.98:0/3095276577 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8d80b26d0 0x7fe8d80b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:56:25.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.679+0000 7fe8d6d9d700 1 --2- 172.21.15.98:0/3095276577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8d80b5140 0x7fe8d8160520 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe8d808c040 tx=0x7fe8d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5245695a4cfb1ea6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:25.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.679+0000 7fe8d559a700 1 -- 172.21.15.98:0/3095276577 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d000e070 con 0x7fe8d80b5140 2023-03-03T07:56:25.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.679+0000 7fe8d559a700 1 -- 172.21.15.98:0/3095276577 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d000b9c0 con 0x7fe8d80b5140 2023-03-03T07:56:25.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.679+0000 7fe8d559a700 1 -- 172.21.15.98:0/3095276577 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d000bd70 con 0x7fe8d80b5140 2023-03-03T07:56:25.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.680+0000 7fe8dc9ff700 1 -- 172.21.15.98:0/3095276577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8d80b5140 msgr2=0x7fe8d8160520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:25.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.680+0000 7fe8dc9ff700 1 --2- 172.21.15.98:0/3095276577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8d80b5140 0x7fe8d8160520 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe8d808c040 tx=0x7fe8d000b330 comp rx=0 tx=0).stop 2023-03-03T07:56:25.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.680+0000 7fe8dc9ff700 1 -- 172.21.15.98:0/3095276577 shutdown_connections 2023-03-03T07:56:25.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.680+0000 7fe8dc9ff700 1 --2- 172.21.15.98:0/3095276577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8d80b5140 0x7fe8d8160520 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.680+0000 7fe8dc9ff700 1 --2- 172.21.15.98:0/3095276577 >> v2:172.21.15.100:3300/0 conn(0x7fe8d80b8d10 0x7fe8d80cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.680+0000 7fe8dc9ff700 1 --2- 172.21.15.98:0/3095276577 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8d80b26d0 0x7fe8d80b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.680+0000 7fe8dc9ff700 1 -- 172.21.15.98:0/3095276577 >> 172.21.15.98:0/3095276577 conn(0x7fe8d8073e70 msgr2=0x7fe8d806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:25.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.680+0000 7fe8dc9ff700 1 -- 172.21.15.98:0/3095276577 shutdown_connections 2023-03-03T07:56:25.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.680+0000 7fe8dc9ff700 1 -- 172.21.15.98:0/3095276577 wait complete. 2023-03-03T07:56:25.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.683+0000 7fe8dc9ff700 1 Processor -- start 2023-03-03T07:56:25.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.683+0000 7fe8dc9ff700 1 -- start start 2023-03-03T07:56:25.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.684+0000 7fe8dc9ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8d80b26d0 0x7fe8d80cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:25.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.684+0000 7fe8dc9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8d80b5140 0x7fe8d80b1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:25.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.684+0000 7fe8dc9ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8d80b8d10 0x7fe8d80ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:25.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.684+0000 7fe8dc9ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d80b1940 con 0x7fe8d80b26d0 2023-03-03T07:56:25.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.684+0000 7fe8dc9ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d80cc490 con 0x7fe8d80b8d10 2023-03-03T07:56:25.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.684+0000 7fe8dc9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d80cc770 con 0x7fe8d80b5140 2023-03-03T07:56:25.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.685+0000 7fe8d659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8d80b26d0 0x7fe8d80cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:25.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.685+0000 7fe8d659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8d80b26d0 0x7fe8d80cd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40516/0 (socket says 172.21.15.98:40516) 2023-03-03T07:56:25.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.685+0000 7fe8d659c700 1 -- 172.21.15.98:0/4107588059 learned_addr learned my addr 172.21.15.98:0/4107588059 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:25.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.685+0000 7fe8d5d9b700 1 --2- 172.21.15.98:0/4107588059 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8d80b5140 0x7fe8d80b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:25.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.685+0000 7fe8d6d9d700 1 --2- 172.21.15.98:0/4107588059 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8d80b8d10 0x7fe8d80ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:25.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.685+0000 7fe8d659c700 1 -- 172.21.15.98:0/4107588059 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8d80b5140 msgr2=0x7fe8d80b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.685+0000 7fe8d659c700 1 --2- 172.21.15.98:0/4107588059 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8d80b5140 0x7fe8d80b1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.685+0000 7fe8d659c700 1 -- 172.21.15.98:0/4107588059 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8d80b8d10 msgr2=0x7fe8d80ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.685+0000 7fe8d659c700 1 --2- 172.21.15.98:0/4107588059 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8d80b8d10 0x7fe8d80ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.685+0000 7fe8d659c700 1 -- 172.21.15.98:0/4107588059 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0009b30 con 0x7fe8d80b26d0 2023-03-03T07:56:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.685+0000 7fe8d6d9d700 1 --2- 172.21.15.98:0/4107588059 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8d80b8d10 0x7fe8d80ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:56:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.685+0000 7fe8d659c700 1 --2- 172.21.15.98:0/4107588059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8d80b26d0 0x7fe8d80cd250 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe8c800c7f0 tx=0x7fe8c800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.686+0000 7fe8c6ffd700 1 -- 172.21.15.98:0/4107588059 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8011800 con 0x7fe8d80b26d0 2023-03-03T07:56:25.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.686+0000 7fe8c6ffd700 1 -- 172.21.15.98:0/4107588059 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c8011960 con 0x7fe8d80b26d0 2023-03-03T07:56:25.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.686+0000 7fe8c6ffd700 1 -- 172.21.15.98:0/4107588059 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8011d10 con 0x7fe8d80b26d0 2023-03-03T07:56:25.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.686+0000 7fe8dc9ff700 1 -- 172.21.15.98:0/4107588059 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d80d0e80 con 0x7fe8d80b26d0 2023-03-03T07:56:25.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.686+0000 7fe8dc9ff700 1 -- 172.21.15.98:0/4107588059 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d80cdd00 con 0x7fe8d80b26d0 2023-03-03T07:56:25.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.687+0000 7fe8dc9ff700 1 -- 172.21.15.98:0/4107588059 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d80cdfe0 con 0x7fe8d80b26d0 2023-03-03T07:56:25.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.688+0000 7fe8c6ffd700 1 -- 172.21.15.98:0/4107588059 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe8c8021070 con 0x7fe8d80b26d0 2023-03-03T07:56:25.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.690+0000 7fe8c6ffd700 1 --2- 172.21.15.98:0/4107588059 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8b80682f0 0x7fe8b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:25.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.690+0000 7fe8c6ffd700 1 -- 172.21.15.98:0/4107588059 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe8c8085b90 con 0x7fe8d80b26d0 2023-03-03T07:56:25.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.690+0000 7fe8d5d9b700 1 --2- 172.21.15.98:0/4107588059 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8b80682f0 0x7fe8b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:25.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.690+0000 7fe8d5d9b700 1 --2- 172.21.15.98:0/4107588059 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8b80682f0 0x7fe8b806a7a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc006fd0 tx=0x7fe8cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:25.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.693+0000 7fe8c6ffd700 1 -- 172.21.15.98:0/4107588059 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe8c8052300 con 0x7fe8d80b26d0 2023-03-03T07:56:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.884+0000 7fe8dc9ff700 1 -- 172.21.15.98:0/4107588059 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe8d804e6e0 con 0x7fe8d80b26d0 2023-03-03T07:56:25.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.885+0000 7fe8c6ffd700 1 -- 172.21.15.98:0/4107588059 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe8c8055920 con 0x7fe8d80b26d0 2023-03-03T07:56:25.886 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:56:25.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.887+0000 7fe8dc9ff700 1 -- 172.21.15.98:0/4107588059 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8b80682f0 msgr2=0x7fe8b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:25.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.887+0000 7fe8dc9ff700 1 --2- 172.21.15.98:0/4107588059 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8b80682f0 0x7fe8b806a7a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc006fd0 tx=0x7fe8cc008040 comp rx=0 tx=0).stop 2023-03-03T07:56:25.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.887+0000 7fe8dc9ff700 1 -- 172.21.15.98:0/4107588059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8d80b26d0 msgr2=0x7fe8d80cd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:25.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.887+0000 7fe8dc9ff700 1 --2- 172.21.15.98:0/4107588059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8d80b26d0 0x7fe8d80cd250 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe8c800c7f0 tx=0x7fe8c800cbb0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.887+0000 7fe8dc9ff700 1 -- 172.21.15.98:0/4107588059 shutdown_connections 2023-03-03T07:56:25.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.887+0000 7fe8dc9ff700 1 --2- 172.21.15.98:0/4107588059 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8b80682f0 0x7fe8b806a7a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.887+0000 7fe8dc9ff700 1 --2- 172.21.15.98:0/4107588059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8d80b26d0 0x7fe8d80cd250 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.887+0000 7fe8dc9ff700 1 --2- 172.21.15.98:0/4107588059 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8d80b8d10 0x7fe8d80ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.887+0000 7fe8dc9ff700 1 --2- 172.21.15.98:0/4107588059 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8d80b5140 0x7fe8d80b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.887+0000 7fe8dc9ff700 1 -- 172.21.15.98:0/4107588059 >> 172.21.15.98:0/4107588059 conn(0x7fe8d8073e70 msgr2=0x7fe8d80cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.888+0000 7fe8dc9ff700 1 -- 172.21.15.98:0/4107588059 shutdown_connections 2023-03-03T07:56:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:25.888+0000 7fe8dc9ff700 1 -- 172.21.15.98:0/4107588059 wait complete. 2023-03-03T07:56:25.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:56:26.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.158+0000 7fa99e7ff700 1 Processor -- start 2023-03-03T07:56:26.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.159+0000 7fa99e7ff700 1 -- start start 2023-03-03T07:56:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.161+0000 7fa99e7ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa99815cc00 0x7fa9980cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.161+0000 7fa99e7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9980d27c0 0x7fa9980d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.161+0000 7fa99e7ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9980cde10 con 0x7fa9980b60d0 2023-03-03T07:56:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.161+0000 7fa99e7ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9980ce040 con 0x7fa99815cc00 2023-03-03T07:56:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.161+0000 7fa99e7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa998073be0 con 0x7fa9980d27c0 2023-03-03T07:56:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.161+0000 7fa99cd9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa9980b60d0 0x7fa9980b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39764/0 (socket says 172.21.15.98:39764) 2023-03-03T07:56:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.161+0000 7fa99cd9c700 1 -- 172.21.15.98:0/2854293682 learned_addr learned my addr 172.21.15.98:0/2854293682 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.161+0000 7fa997fff700 1 --2- 172.21.15.98:0/2854293682 >> v2:172.21.15.98:3300/0 conn(0x7fa99815cc00 0x7fa9980cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.161+0000 7fa98ffff700 1 --2- 172.21.15.98:0/2854293682 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9980d27c0 0x7fa9980d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.162+0000 7fa997fff700 1 -- 172.21.15.98:0/2854293682 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9980d27c0 msgr2=0x7fa9980d2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.162+0000 7fa997fff700 1 --2- 172.21.15.98:0/2854293682 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9980d27c0 0x7fa9980d2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.162+0000 7fa997fff700 1 -- 172.21.15.98:0/2854293682 >> v1:172.21.15.98:6789/0 conn(0x7fa9980b60d0 legacy=0x7fa9980b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.162+0000 7fa997fff700 1 -- 172.21.15.98:0/2854293682 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9980d1a00 con 0x7fa99815cc00 2023-03-03T07:56:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.162+0000 7fa9977fe700 1 -- 172.21.15.98:0/2854293682 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389328030 0 0) 0x7fa9980cde10 con 0x7fa9980b60d0 2023-03-03T07:56:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.162+0000 7fa997fff700 1 --2- 172.21.15.98:0/2854293682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa99815cc00 0x7fa9980cd8d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa988009840 tx=0x7fa988009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d9840f01759763 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.162+0000 7fa9977fe700 1 -- 172.21.15.98:0/2854293682 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988011830 con 0x7fa99815cc00 2023-03-03T07:56:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.163+0000 7fa9977fe700 1 -- 172.21.15.98:0/2854293682 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa988011990 con 0x7fa99815cc00 2023-03-03T07:56:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.163+0000 7fa9977fe700 1 -- 172.21.15.98:0/2854293682 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988011d20 con 0x7fa99815cc00 2023-03-03T07:56:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.163+0000 7fa99e7ff700 1 -- 172.21.15.98:0/2854293682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa99815cc00 msgr2=0x7fa9980cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.163+0000 7fa99e7ff700 1 --2- 172.21.15.98:0/2854293682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa99815cc00 0x7fa9980cd8d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa988009840 tx=0x7fa988009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.163+0000 7fa99e7ff700 1 -- 172.21.15.98:0/2854293682 shutdown_connections 2023-03-03T07:56:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.163+0000 7fa99e7ff700 1 --2- 172.21.15.98:0/2854293682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa99815cc00 0x7fa9980cd8d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.163+0000 7fa99e7ff700 1 --2- 172.21.15.98:0/2854293682 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9980d27c0 0x7fa9980d2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.163+0000 7fa99e7ff700 1 -- 172.21.15.98:0/2854293682 >> 172.21.15.98:0/2854293682 conn(0x7fa99806aee0 msgr2=0x7fa99806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.163+0000 7fa99e7ff700 1 -- 172.21.15.98:0/2854293682 shutdown_connections 2023-03-03T07:56:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.163+0000 7fa99e7ff700 1 -- 172.21.15.98:0/2854293682 wait complete. 2023-03-03T07:56:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.165+0000 7fa99e7ff700 1 Processor -- start 2023-03-03T07:56:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.165+0000 7fa99e7ff700 1 -- start start 2023-03-03T07:56:26.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.166+0000 7fa99e7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9980b60d0 0x7fa9981f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:26.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.166+0000 7fa99e7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9980d27c0 0x7fa9981f5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:26.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.166+0000 7fa99e7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa99815cc00 0x7fa9980b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:26.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.166+0000 7fa99e7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9981f38d0 con 0x7fa9980d27c0 2023-03-03T07:56:26.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.166+0000 7fa99e7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9980b4830 con 0x7fa99815cc00 2023-03-03T07:56:26.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.166+0000 7fa99e7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9980b4b10 con 0x7fa9980b60d0 2023-03-03T07:56:26.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.167+0000 7fa98ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9980d27c0 0x7fa9981f5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:26.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.167+0000 7fa98ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9980d27c0 0x7fa9981f5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40540/0 (socket says 172.21.15.98:40540) 2023-03-03T07:56:26.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.167+0000 7fa98ffff700 1 -- 172.21.15.98:0/3588025331 learned_addr learned my addr 172.21.15.98:0/3588025331 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:26.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.167+0000 7fa997fff700 1 --2- 172.21.15.98:0/3588025331 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9980b60d0 0x7fa9981f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:26.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.167+0000 7fa99cd9c700 1 --2- 172.21.15.98:0/3588025331 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa99815cc00 0x7fa9980b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:26.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.167+0000 7fa98ffff700 1 -- 172.21.15.98:0/3588025331 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9980b60d0 msgr2=0x7fa9981f55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:26.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.167+0000 7fa98ffff700 1 --2- 172.21.15.98:0/3588025331 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9980b60d0 0x7fa9981f55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.167+0000 7fa98ffff700 1 -- 172.21.15.98:0/3588025331 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa99815cc00 msgr2=0x7fa9980b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.167+0000 7fa98ffff700 1 --2- 172.21.15.98:0/3588025331 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa99815cc00 0x7fa9980b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.167+0000 7fa98ffff700 1 -- 172.21.15.98:0/3588025331 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9880094a0 con 0x7fa9980d27c0 2023-03-03T07:56:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.168+0000 7fa98ffff700 1 --2- 172.21.15.98:0/3588025331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9980d27c0 0x7fa9981f5ae0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa98000e870 tx=0x7fa98000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.168+0000 7fa9957fa700 1 -- 172.21.15.98:0/3588025331 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980009590 con 0x7fa9980d27c0 2023-03-03T07:56:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.168+0000 7fa9957fa700 1 -- 172.21.15.98:0/3588025331 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa980013680 con 0x7fa9980d27c0 2023-03-03T07:56:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.168+0000 7fa9957fa700 1 -- 172.21.15.98:0/3588025331 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9800139f0 con 0x7fa9980d27c0 2023-03-03T07:56:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.168+0000 7fa99e7ff700 1 -- 172.21.15.98:0/3588025331 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9980d1fc0 con 0x7fa9980d27c0 2023-03-03T07:56:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.168+0000 7fa99e7ff700 1 -- 172.21.15.98:0/3588025331 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9980d1ce0 con 0x7fa9980d27c0 2023-03-03T07:56:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.169+0000 7fa99e7ff700 1 -- 172.21.15.98:0/3588025331 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9980d22a0 con 0x7fa9980d27c0 2023-03-03T07:56:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.171+0000 7fa9957fa700 1 -- 172.21.15.98:0/3588025331 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa980019030 con 0x7fa9980d27c0 2023-03-03T07:56:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.172+0000 7fa9957fa700 1 --2- 172.21.15.98:0/3588025331 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9780682f0 0x7fa97806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.172+0000 7fa9957fa700 1 -- 172.21.15.98:0/3588025331 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa980085b00 con 0x7fa9980d27c0 2023-03-03T07:56:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.172+0000 7fa997fff700 1 --2- 172.21.15.98:0/3588025331 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9780682f0 0x7fa97806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:26.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.173+0000 7fa997fff700 1 --2- 172.21.15.98:0/3588025331 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9780682f0 0x7fa97806a7a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa988009fd0 tx=0x7fa988008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:26.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.176+0000 7fa9957fa700 1 -- 172.21.15.98:0/3588025331 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa980087e60 con 0x7fa9980d27c0 2023-03-03T07:56:26.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.375+0000 7fa99e7ff700 1 -- 172.21.15.98:0/3588025331 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9980729b0 con 0x7fa9980d27c0 2023-03-03T07:56:26.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.376+0000 7fa9957fa700 1 -- 172.21.15.98:0/3588025331 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17907 (secure 0 0 0) 0x7fa980052270 con 0x7fa9980d27c0 2023-03-03T07:56:26.378 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:56:26.378 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:55:48.099844+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:56:26.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.378+0000 7fa99e7ff700 1 -- 172.21.15.98:0/3588025331 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9780682f0 msgr2=0x7fa97806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:26.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.378+0000 7fa99e7ff700 1 --2- 172.21.15.98:0/3588025331 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9780682f0 0x7fa97806a7a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa988009fd0 tx=0x7fa988008030 comp rx=0 tx=0).stop 2023-03-03T07:56:26.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.378+0000 7fa99e7ff700 1 -- 172.21.15.98:0/3588025331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9980d27c0 msgr2=0x7fa9981f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:26.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.378+0000 7fa99e7ff700 1 --2- 172.21.15.98:0/3588025331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9980d27c0 0x7fa9981f5ae0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa98000e870 tx=0x7fa98000eb80 comp rx=0 tx=0).stop 2023-03-03T07:56:26.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.379+0000 7fa99e7ff700 1 -- 172.21.15.98:0/3588025331 shutdown_connections 2023-03-03T07:56:26.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.379+0000 7fa99e7ff700 1 --2- 172.21.15.98:0/3588025331 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9780682f0 0x7fa97806a7a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.379+0000 7fa99e7ff700 1 --2- 172.21.15.98:0/3588025331 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa99815cc00 0x7fa9980b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.379+0000 7fa99e7ff700 1 --2- 172.21.15.98:0/3588025331 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9980b60d0 0x7fa9981f55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.379+0000 7fa99e7ff700 1 --2- 172.21.15.98:0/3588025331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9980d27c0 0x7fa9981f5ae0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.379+0000 7fa99e7ff700 1 -- 172.21.15.98:0/3588025331 >> 172.21.15.98:0/3588025331 conn(0x7fa99806aee0 msgr2=0x7fa9980b6780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:26.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.379+0000 7fa99e7ff700 1 -- 172.21.15.98:0/3588025331 shutdown_connections 2023-03-03T07:56:26.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.379+0000 7fa99e7ff700 1 -- 172.21.15.98:0/3588025331 wait complete. 2023-03-03T07:56:26.408 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:55:45.157666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:55:46.092091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:56:26.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:56:26.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.661+0000 7f98c9fff700 1 Processor -- start 2023-03-03T07:56:26.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.661+0000 7f98c9fff700 1 -- start start 2023-03-03T07:56:26.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.663+0000 7f98c9fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f98c40b8d10 0x7f98c40cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:26.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.663+0000 7f98c9fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c40cd7d0 con 0x7f98c40b26d0 2023-03-03T07:56:26.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.663+0000 7f98c2ffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f98c40b26d0 0x7f98c40b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39780/0 (socket says 172.21.15.98:39780) 2023-03-03T07:56:26.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.663+0000 7f98c2ffd700 1 -- 172.21.15.98:0/1739418904 learned_addr learned my addr 172.21.15.98:0/1739418904 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:26.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.663+0000 7f98c9fff700 1 -- 172.21.15.98:0/1739418904 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c40b90e0 con 0x7f98c40b5310 2023-03-03T07:56:26.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.663+0000 7f98c9fff700 1 -- 172.21.15.98:0/1739418904 --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c40cda20 con 0x7f98c40b8d10 2023-03-03T07:56:26.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.663+0000 7f98c37fe700 1 --2- 172.21.15.98:0/1739418904 >> v2:172.21.15.100:3300/0 conn(0x7f98c40b8d10 0x7f98c40cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:26.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.664+0000 7f98c27fc700 1 -- 172.21.15.98:0/1739418904 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427180844 0 0) 0x7f98c40cd7d0 con 0x7f98c40b26d0 2023-03-03T07:56:26.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.664+0000 7f98c27fc700 1 -- 172.21.15.98:0/1739418904 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ac0035b0 con 0x7f98c40b26d0 2023-03-03T07:56:26.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.664+0000 7f98c37fe700 1 -- 172.21.15.98:0/1739418904 >> v1:172.21.15.100:6789/0 conn(0x7f98c40b5310 legacy=0x7f98c40d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:26.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.664+0000 7f98c37fe700 1 -- 172.21.15.98:0/1739418904 >> v1:172.21.15.98:6789/0 conn(0x7f98c40b26d0 legacy=0x7f98c40b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:26.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.664+0000 7f98c37fe700 1 -- 172.21.15.98:0/1739418904 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c40cdd00 con 0x7f98c40b8d10 2023-03-03T07:56:26.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.664+0000 7f98c37fe700 1 --2- 172.21.15.98:0/1739418904 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98c40b8d10 0x7f98c40cffc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009840 tx=0x7f98b4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cfcaa4b2c2e3bb6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:26.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.665+0000 7f98c27fc700 1 -- 172.21.15.98:0/1739418904 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4011830 con 0x7f98c40b8d10 2023-03-03T07:56:26.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.665+0000 7f98c27fc700 1 -- 172.21.15.98:0/1739418904 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b4011990 con 0x7f98c40b8d10 2023-03-03T07:56:26.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.665+0000 7f98c27fc700 1 -- 172.21.15.98:0/1739418904 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4011d00 con 0x7f98c40b8d10 2023-03-03T07:56:26.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.665+0000 7f98c9fff700 1 -- 172.21.15.98:0/1739418904 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98c40b8d10 msgr2=0x7f98c40cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:26.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.665+0000 7f98c9fff700 1 --2- 172.21.15.98:0/1739418904 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98c40b8d10 0x7f98c40cffc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009840 tx=0x7f98b4009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:26.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.665+0000 7f98c9fff700 1 -- 172.21.15.98:0/1739418904 shutdown_connections 2023-03-03T07:56:26.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.665+0000 7f98c9fff700 1 --2- 172.21.15.98:0/1739418904 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98c40b8d10 0x7f98c40cffc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.665+0000 7f98c9fff700 1 -- 172.21.15.98:0/1739418904 >> 172.21.15.98:0/1739418904 conn(0x7f98c4073e70 msgr2=0x7f98c406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:26.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.665+0000 7f98c9fff700 1 -- 172.21.15.98:0/1739418904 shutdown_connections 2023-03-03T07:56:26.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.665+0000 7f98c9fff700 1 -- 172.21.15.98:0/1739418904 wait complete. 2023-03-03T07:56:26.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.667+0000 7f98c9fff700 1 Processor -- start 2023-03-03T07:56:26.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.667+0000 7f98c9fff700 1 -- start start 2023-03-03T07:56:26.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.668+0000 7f98c9fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c40b26d0 0x7f98c41f5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:26.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.668+0000 7f98c9fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98c40b5310 0x7f98c40ac5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:26.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.668+0000 7f98c9fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98c40b8d10 0x7f98c40acae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:26.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.669+0000 7f98c9fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c40a9ee0 con 0x7f98c40b26d0 2023-03-03T07:56:26.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.669+0000 7f98c9fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c40aa1a0 con 0x7f98c40b8d10 2023-03-03T07:56:26.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.669+0000 7f98c9fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c40aa480 con 0x7f98c40b5310 2023-03-03T07:56:26.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.669+0000 7f98c37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c40b26d0 0x7f98c41f5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.669+0000 7f98c37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c40b26d0 0x7f98c41f5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40542/0 (socket says 172.21.15.98:40542) 2023-03-03T07:56:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.669+0000 7f98c37fe700 1 -- 172.21.15.98:0/3936333990 learned_addr learned my addr 172.21.15.98:0/3936333990 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.669+0000 7f98c2ffd700 1 --2- 172.21.15.98:0/3936333990 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98c40b5310 0x7f98c40ac5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.669+0000 7f98c3fff700 1 --2- 172.21.15.98:0/3936333990 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98c40b8d10 0x7f98c40acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.669+0000 7f98c37fe700 1 -- 172.21.15.98:0/3936333990 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98c40b5310 msgr2=0x7f98c40ac5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.669+0000 7f98c37fe700 1 --2- 172.21.15.98:0/3936333990 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98c40b5310 0x7f98c40ac5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.669+0000 7f98c37fe700 1 -- 172.21.15.98:0/3936333990 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98c40b8d10 msgr2=0x7f98c40acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:26.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.669+0000 7f98c37fe700 1 --2- 172.21.15.98:0/3936333990 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98c40b8d10 0x7f98c40acae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.669+0000 7f98c37fe700 1 -- 172.21.15.98:0/3936333990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b40094a0 con 0x7f98c40b26d0 2023-03-03T07:56:26.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.669+0000 7f98c2ffd700 1 --2- 172.21.15.98:0/3936333990 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98c40b5310 0x7f98c40ac5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:26.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.670+0000 7f98c37fe700 1 --2- 172.21.15.98:0/3936333990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c40b26d0 0x7f98c41f5520 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009fd0 tx=0x7f98b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:26.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.670+0000 7f98abfff700 1 -- 172.21.15.98:0/3936333990 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b40036a0 con 0x7f98c40b26d0 2023-03-03T07:56:26.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.670+0000 7f98abfff700 1 -- 172.21.15.98:0/3936333990 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b401f070 con 0x7f98c40b26d0 2023-03-03T07:56:26.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.670+0000 7f98abfff700 1 -- 172.21.15.98:0/3936333990 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4003a20 con 0x7f98c40b26d0 2023-03-03T07:56:26.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.670+0000 7f98c9fff700 1 -- 172.21.15.98:0/3936333990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c40cdfe0 con 0x7f98c40b26d0 2023-03-03T07:56:26.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.670+0000 7f98c9fff700 1 -- 172.21.15.98:0/3936333990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c40d0e80 con 0x7f98c40b26d0 2023-03-03T07:56:26.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.671+0000 7f98c9fff700 1 -- 172.21.15.98:0/3936333990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c40d0500 con 0x7f98c40b26d0 2023-03-03T07:56:26.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.672+0000 7f98abfff700 1 -- 172.21.15.98:0/3936333990 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f98b400f050 con 0x7f98c40b26d0 2023-03-03T07:56:26.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.674+0000 7f98abfff700 1 --2- 172.21.15.98:0/3936333990 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f98b00682f0 0x7f98b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:26.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.674+0000 7f98abfff700 1 -- 172.21.15.98:0/3936333990 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f98b4021070 con 0x7f98c40b26d0 2023-03-03T07:56:26.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.674+0000 7f98c2ffd700 1 --2- 172.21.15.98:0/3936333990 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f98b00682f0 0x7f98b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:26.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.675+0000 7f98c2ffd700 1 --2- 172.21.15.98:0/3936333990 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f98b00682f0 0x7f98b006a7a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f98b80011e0 tx=0x7f98b8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:26.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.678+0000 7f98abfff700 1 -- 172.21.15.98:0/3936333990 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f98b4053e30 con 0x7f98c40b26d0 2023-03-03T07:56:26.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.872+0000 7f98c9fff700 1 -- 172.21.15.98:0/3936333990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f98c404e6e0 con 0x7f98c40b26d0 2023-03-03T07:56:26.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.872+0000 7f98abfff700 1 -- 172.21.15.98:0/3936333990 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f98b4057450 con 0x7f98c40b26d0 2023-03-03T07:56:26.874 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:56:26.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.875+0000 7f98c9fff700 1 -- 172.21.15.98:0/3936333990 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f98b00682f0 msgr2=0x7f98b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:26.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.875+0000 7f98c9fff700 1 --2- 172.21.15.98:0/3936333990 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f98b00682f0 0x7f98b006a7a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f98b80011e0 tx=0x7f98b8009040 comp rx=0 tx=0).stop 2023-03-03T07:56:26.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.875+0000 7f98c9fff700 1 -- 172.21.15.98:0/3936333990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c40b26d0 msgr2=0x7f98c41f5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:26.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.875+0000 7f98c9fff700 1 --2- 172.21.15.98:0/3936333990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c40b26d0 0x7f98c41f5520 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009fd0 tx=0x7f98b4004750 comp rx=0 tx=0).stop 2023-03-03T07:56:26.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.875+0000 7f98c9fff700 1 -- 172.21.15.98:0/3936333990 shutdown_connections 2023-03-03T07:56:26.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.875+0000 7f98c9fff700 1 --2- 172.21.15.98:0/3936333990 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f98b00682f0 0x7f98b006a7a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.875+0000 7f98c9fff700 1 --2- 172.21.15.98:0/3936333990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c40b26d0 0x7f98c41f5520 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.875+0000 7f98c9fff700 1 --2- 172.21.15.98:0/3936333990 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98c40b8d10 0x7f98c40acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.875+0000 7f98c9fff700 1 --2- 172.21.15.98:0/3936333990 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98c40b5310 0x7f98c40ac5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:26.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.875+0000 7f98c9fff700 1 -- 172.21.15.98:0/3936333990 >> 172.21.15.98:0/3936333990 conn(0x7f98c4073e70 msgr2=0x7f98c415c630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:26.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.875+0000 7f98c9fff700 1 -- 172.21.15.98:0/3936333990 shutdown_connections 2023-03-03T07:56:26.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:26.875+0000 7f98c9fff700 1 -- 172.21.15.98:0/3936333990 wait complete. 2023-03-03T07:56:26.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:56:27.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.156+0000 7f56ba5ff700 1 Processor -- start 2023-03-03T07:56:27.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.156+0000 7f56ba5ff700 1 -- start start 2023-03-03T07:56:27.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.157+0000 7f56ba5ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f56b40b8d10 0x7f56b40cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:27.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.157+0000 7f56ba5ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f56b40b26d0 0x7f56b40b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:27.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.157+0000 7f56ba5ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b40cd7d0 con 0x7f56b40b5140 2023-03-03T07:56:27.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.157+0000 7f56ba5ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b40b55c0 con 0x7f56b40b26d0 2023-03-03T07:56:27.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.157+0000 7f56ba5ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b40b90e0 con 0x7f56b40b8d10 2023-03-03T07:56:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.158+0000 7f56b37fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f56b40b26d0 0x7f56b40b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.158+0000 7f56b37fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f56b40b26d0 0x7f56b40b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40544/0 (socket says 172.21.15.98:40544) 2023-03-03T07:56:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.158+0000 7f56b37fe700 1 -- 172.21.15.98:0/1160392662 learned_addr learned my addr 172.21.15.98:0/1160392662 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.158+0000 7f56b3fff700 1 --2- 172.21.15.98:0/1160392662 >> v2:172.21.15.100:3300/0 conn(0x7f56b40b8d10 0x7f56b40cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.158+0000 7f56b37fe700 1 -- 172.21.15.98:0/1160392662 >> v2:172.21.15.100:3300/0 conn(0x7f56b40b8d10 msgr2=0x7f56b40cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.158+0000 7f56b37fe700 1 --2- 172.21.15.98:0/1160392662 >> v2:172.21.15.100:3300/0 conn(0x7f56b40b8d10 0x7f56b40cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.158+0000 7f56b37fe700 1 -- 172.21.15.98:0/1160392662 >> v1:172.21.15.100:6789/0 conn(0x7f56b40b5140 legacy=0x7f56b4160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.158+0000 7f56b37fe700 1 -- 172.21.15.98:0/1160392662 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b40cda20 con 0x7f56b40b26d0 2023-03-03T07:56:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.159+0000 7f56b37fe700 1 --2- 172.21.15.98:0/1160392662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b40b26d0 0x7f56b40b4c00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f56b408c040 tx=0x7f56a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad4444b604fc417 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.159+0000 7f56b2ffd700 1 -- 172.21.15.98:0/1160392662 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a8011830 con 0x7f56b40b26d0 2023-03-03T07:56:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.159+0000 7f56b2ffd700 1 -- 172.21.15.98:0/1160392662 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a8011990 con 0x7f56b40b26d0 2023-03-03T07:56:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.159+0000 7f56b2ffd700 1 -- 172.21.15.98:0/1160392662 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a8011d20 con 0x7f56b40b26d0 2023-03-03T07:56:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.159+0000 7f56ba5ff700 1 -- 172.21.15.98:0/1160392662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b40b26d0 msgr2=0x7f56b40b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.159+0000 7f56ba5ff700 1 --2- 172.21.15.98:0/1160392662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b40b26d0 0x7f56b40b4c00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f56b408c040 tx=0x7f56a8009bd0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.160+0000 7f56ba5ff700 1 -- 172.21.15.98:0/1160392662 shutdown_connections 2023-03-03T07:56:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.160+0000 7f56ba5ff700 1 --2- 172.21.15.98:0/1160392662 >> v2:172.21.15.100:3300/0 conn(0x7f56b40b8d10 0x7f56b40cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.160+0000 7f56ba5ff700 1 --2- 172.21.15.98:0/1160392662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b40b26d0 0x7f56b40b4c00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.160+0000 7f56ba5ff700 1 -- 172.21.15.98:0/1160392662 >> 172.21.15.98:0/1160392662 conn(0x7f56b4073e70 msgr2=0x7f56b406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.160+0000 7f56ba5ff700 1 -- 172.21.15.98:0/1160392662 shutdown_connections 2023-03-03T07:56:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.160+0000 7f56ba5ff700 1 -- 172.21.15.98:0/1160392662 wait complete. 2023-03-03T07:56:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.162+0000 7f56ba5ff700 1 Processor -- start 2023-03-03T07:56:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.162+0000 7f56ba5ff700 1 -- start start 2023-03-03T07:56:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.163+0000 7f56ba5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56b40b26d0 0x7f56b4154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.163+0000 7f56ba5ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b40b5140 0x7f56b415f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.163+0000 7f56ba5ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56b40b8d10 0x7f56b4141200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.163+0000 7f56ba5ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b4154990 con 0x7f56b40b5140 2023-03-03T07:56:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.163+0000 7f56ba5ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b4154c70 con 0x7f56b40b8d10 2023-03-03T07:56:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.163+0000 7f56ba5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b4154f50 con 0x7f56b40b26d0 2023-03-03T07:56:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.163+0000 7f56b37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b40b5140 0x7f56b415f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.163+0000 7f56b37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b40b5140 0x7f56b415f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40554/0 (socket says 172.21.15.98:40554) 2023-03-03T07:56:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.163+0000 7f56b37fe700 1 -- 172.21.15.98:0/393541087 learned_addr learned my addr 172.21.15.98:0/393541087 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:27.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.163+0000 7f56b8b9c700 1 --2- 172.21.15.98:0/393541087 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56b40b8d10 0x7f56b4141200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:27.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.163+0000 7f56b3fff700 1 --2- 172.21.15.98:0/393541087 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56b40b26d0 0x7f56b4154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:27.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.164+0000 7f56b37fe700 1 -- 172.21.15.98:0/393541087 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56b40b26d0 msgr2=0x7f56b4154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:27.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.164+0000 7f56b37fe700 1 --2- 172.21.15.98:0/393541087 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56b40b26d0 0x7f56b4154140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.164+0000 7f56b37fe700 1 -- 172.21.15.98:0/393541087 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56b40b8d10 msgr2=0x7f56b4141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:27.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.164+0000 7f56b37fe700 1 --2- 172.21.15.98:0/393541087 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56b40b8d10 0x7f56b4141200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.164+0000 7f56b37fe700 1 -- 172.21.15.98:0/393541087 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a8009550 con 0x7f56b40b5140 2023-03-03T07:56:27.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.164+0000 7f56b37fe700 1 --2- 172.21.15.98:0/393541087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b40b5140 0x7f56b415f930 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009130 tx=0x7f56a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:27.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.164+0000 7f56b0ff9700 1 -- 172.21.15.98:0/393541087 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a8004510 con 0x7f56b40b5140 2023-03-03T07:56:27.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.164+0000 7f56b0ff9700 1 -- 172.21.15.98:0/393541087 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a801e070 con 0x7f56b40b5140 2023-03-03T07:56:27.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.164+0000 7f56b0ff9700 1 -- 172.21.15.98:0/393541087 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a80038a0 con 0x7f56b40b5140 2023-03-03T07:56:27.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.164+0000 7f56ba5ff700 1 -- 172.21.15.98:0/393541087 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b40d0e80 con 0x7f56b40b5140 2023-03-03T07:56:27.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.165+0000 7f56ba5ff700 1 -- 172.21.15.98:0/393541087 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b40cdd00 con 0x7f56b40b5140 2023-03-03T07:56:27.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.166+0000 7f56ba5ff700 1 -- 172.21.15.98:0/393541087 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b40cdfe0 con 0x7f56b40b5140 2023-03-03T07:56:27.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.167+0000 7f56b0ff9700 1 -- 172.21.15.98:0/393541087 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f56a800f050 con 0x7f56b40b5140 2023-03-03T07:56:27.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.168+0000 7f56b0ff9700 1 --2- 172.21.15.98:0/393541087 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f56a00682f0 0x7f56a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:27.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.168+0000 7f56b0ff9700 1 -- 172.21.15.98:0/393541087 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f56a8086be0 con 0x7f56b40b5140 2023-03-03T07:56:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.169+0000 7f56b3fff700 1 --2- 172.21.15.98:0/393541087 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f56a00682f0 0x7f56a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.169+0000 7f56b3fff700 1 --2- 172.21.15.98:0/393541087 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f56a00682f0 0x7f56a006a7a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f56a4009b40 tx=0x7f56a4009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.172+0000 7f56b0ff9700 1 -- 172.21.15.98:0/393541087 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f56a8053290 con 0x7f56b40b5140 2023-03-03T07:56:27.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.369+0000 7f56ba5ff700 1 -- 172.21.15.98:0/393541087 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f56b404e6e0 con 0x7f56b40b5140 2023-03-03T07:56:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.369+0000 7f56b0ff9700 1 -- 172.21.15.98:0/393541087 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v25) v1 ==== 92+0+10 (secure 0 0 0) 0x7f56a8020020 con 0x7f56b40b5140 2023-03-03T07:56:27.371 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:56:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.372+0000 7f56ba5ff700 1 -- 172.21.15.98:0/393541087 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f56a00682f0 msgr2=0x7f56a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.372+0000 7f56ba5ff700 1 --2- 172.21.15.98:0/393541087 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f56a00682f0 0x7f56a006a7a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f56a4009b40 tx=0x7f56a4009210 comp rx=0 tx=0).stop 2023-03-03T07:56:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.372+0000 7f56ba5ff700 1 -- 172.21.15.98:0/393541087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b40b5140 msgr2=0x7f56b415f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.372+0000 7f56ba5ff700 1 --2- 172.21.15.98:0/393541087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b40b5140 0x7f56b415f930 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009130 tx=0x7f56a8004750 comp rx=0 tx=0).stop 2023-03-03T07:56:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.372+0000 7f56ba5ff700 1 -- 172.21.15.98:0/393541087 shutdown_connections 2023-03-03T07:56:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.372+0000 7f56ba5ff700 1 --2- 172.21.15.98:0/393541087 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f56a00682f0 0x7f56a006a7a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.372+0000 7f56ba5ff700 1 --2- 172.21.15.98:0/393541087 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56b40b8d10 0x7f56b4141200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.372+0000 7f56ba5ff700 1 --2- 172.21.15.98:0/393541087 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56b40b26d0 0x7f56b4154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.372+0000 7f56ba5ff700 1 --2- 172.21.15.98:0/393541087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b40b5140 0x7f56b415f930 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.372+0000 7f56ba5ff700 1 -- 172.21.15.98:0/393541087 >> 172.21.15.98:0/393541087 conn(0x7f56b4073e70 msgr2=0x7f56b415c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.372+0000 7f56ba5ff700 1 -- 172.21.15.98:0/393541087 shutdown_connections 2023-03-03T07:56:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.372+0000 7f56ba5ff700 1 -- 172.21.15.98:0/393541087 wait complete. 2023-03-03T07:56:27.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:56:27.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.649+0000 7efe55fff700 1 Processor -- start 2023-03-03T07:56:27.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.649+0000 7efe55fff700 1 -- start start 2023-03-03T07:56:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.650+0000 7efe55fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efe500b8d10 0x7efe500cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.650+0000 7efe55fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efe500b26d0 0x7efe500b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.650+0000 7efe55fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe500b5140 0x7efe50160520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.651+0000 7efe55fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe500b55c0 con 0x7efe500b26d0 2023-03-03T07:56:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.651+0000 7efe55fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe500cd7d0 con 0x7efe500b8d10 2023-03-03T07:56:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.651+0000 7efe55fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe500b90e0 con 0x7efe500b5140 2023-03-03T07:56:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.651+0000 7efe4effd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efe500b26d0 0x7efe500b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.651+0000 7efe4effd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efe500b26d0 0x7efe500b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40568/0 (socket says 172.21.15.98:40568) 2023-03-03T07:56:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.651+0000 7efe4effd700 1 -- 172.21.15.98:0/320823687 learned_addr learned my addr 172.21.15.98:0/320823687 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.651+0000 7efe4f7fe700 1 --2- 172.21.15.98:0/320823687 >> v2:172.21.15.100:3300/0 conn(0x7efe500b8d10 0x7efe500cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.651+0000 7efe4ffff700 1 --2- 172.21.15.98:0/320823687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe500b5140 0x7efe50160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.651+0000 7efe4effd700 1 -- 172.21.15.98:0/320823687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe500b5140 msgr2=0x7efe50160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.651+0000 7efe4effd700 1 --2- 172.21.15.98:0/320823687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe500b5140 0x7efe50160520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.651+0000 7efe4effd700 1 -- 172.21.15.98:0/320823687 >> v2:172.21.15.100:3300/0 conn(0x7efe500b8d10 msgr2=0x7efe500cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:27.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.651+0000 7efe4effd700 1 --2- 172.21.15.98:0/320823687 >> v2:172.21.15.100:3300/0 conn(0x7efe500b8d10 0x7efe500cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.652+0000 7efe4effd700 1 -- 172.21.15.98:0/320823687 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe500cda20 con 0x7efe500b26d0 2023-03-03T07:56:27.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.652+0000 7efe4effd700 1 --2- 172.21.15.98:0/320823687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe500b26d0 0x7efe500b4c00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efe5008c040 tx=0x7efe44009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ef3ebada42deb7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:27.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.652+0000 7efe4e7fc700 1 -- 172.21.15.98:0/320823687 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe44011830 con 0x7efe500b26d0 2023-03-03T07:56:27.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.652+0000 7efe4e7fc700 1 -- 172.21.15.98:0/320823687 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe44011990 con 0x7efe500b26d0 2023-03-03T07:56:27.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.652+0000 7efe4e7fc700 1 -- 172.21.15.98:0/320823687 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe44011d20 con 0x7efe500b26d0 2023-03-03T07:56:27.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.653+0000 7efe55fff700 1 -- 172.21.15.98:0/320823687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe500b26d0 msgr2=0x7efe500b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:27.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.653+0000 7efe55fff700 1 --2- 172.21.15.98:0/320823687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe500b26d0 0x7efe500b4c00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efe5008c040 tx=0x7efe44009bd0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.653+0000 7efe55fff700 1 -- 172.21.15.98:0/320823687 shutdown_connections 2023-03-03T07:56:27.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.653+0000 7efe55fff700 1 --2- 172.21.15.98:0/320823687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe500b5140 0x7efe50160520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.653+0000 7efe55fff700 1 --2- 172.21.15.98:0/320823687 >> v2:172.21.15.100:3300/0 conn(0x7efe500b8d10 0x7efe500cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.653+0000 7efe55fff700 1 --2- 172.21.15.98:0/320823687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe500b26d0 0x7efe500b4c00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.653+0000 7efe55fff700 1 -- 172.21.15.98:0/320823687 >> 172.21.15.98:0/320823687 conn(0x7efe50073e70 msgr2=0x7efe5006ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:27.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.653+0000 7efe55fff700 1 -- 172.21.15.98:0/320823687 shutdown_connections 2023-03-03T07:56:27.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.653+0000 7efe55fff700 1 -- 172.21.15.98:0/320823687 wait complete. 2023-03-03T07:56:27.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.655+0000 7efe55fff700 1 Processor -- start 2023-03-03T07:56:27.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.655+0000 7efe55fff700 1 -- start start 2023-03-03T07:56:27.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.656+0000 7efe55fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe500b26d0 0x7efe500b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:27.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.656+0000 7efe55fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe500b5140 0x7efe500cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:27.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.656+0000 7efe55fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe500b8d10 0x7efe5015f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:27.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.656+0000 7efe55fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe50158a90 con 0x7efe500b8d10 2023-03-03T07:56:27.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.656+0000 7efe55fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe50157e80 con 0x7efe500b5140 2023-03-03T07:56:27.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.656+0000 7efe55fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe50158160 con 0x7efe500b26d0 2023-03-03T07:56:27.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.657+0000 7efe4ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe500b8d10 0x7efe5015f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:27.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.657+0000 7efe4ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe500b8d10 0x7efe5015f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40578/0 (socket says 172.21.15.98:40578) 2023-03-03T07:56:27.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.657+0000 7efe4ffff700 1 -- 172.21.15.98:0/1251952039 learned_addr learned my addr 172.21.15.98:0/1251952039 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:27.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.657+0000 7efe4f7fe700 1 --2- 172.21.15.98:0/1251952039 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe500b26d0 0x7efe500b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:27.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.657+0000 7efe4effd700 1 --2- 172.21.15.98:0/1251952039 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe500b5140 0x7efe500cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:27.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.657+0000 7efe4ffff700 1 -- 172.21.15.98:0/1251952039 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe500b26d0 msgr2=0x7efe500b14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:27.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.657+0000 7efe4ffff700 1 --2- 172.21.15.98:0/1251952039 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe500b26d0 0x7efe500b14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.657+0000 7efe4ffff700 1 -- 172.21.15.98:0/1251952039 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe500b5140 msgr2=0x7efe500cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:27.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.657+0000 7efe4ffff700 1 --2- 172.21.15.98:0/1251952039 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe500b5140 0x7efe500cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.657+0000 7efe4ffff700 1 -- 172.21.15.98:0/1251952039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe44009550 con 0x7efe500b8d10 2023-03-03T07:56:27.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.657+0000 7efe4ffff700 1 --2- 172.21.15.98:0/1251952039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe500b8d10 0x7efe5015f510 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efe4800efc0 tx=0x7efe4800c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:27.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.658+0000 7efe37fff700 1 -- 172.21.15.98:0/1251952039 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4800cad0 con 0x7efe500b8d10 2023-03-03T07:56:27.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.658+0000 7efe37fff700 1 -- 172.21.15.98:0/1251952039 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4800f040 con 0x7efe500b8d10 2023-03-03T07:56:27.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.658+0000 7efe37fff700 1 -- 172.21.15.98:0/1251952039 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4800ce70 con 0x7efe500b8d10 2023-03-03T07:56:27.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.658+0000 7efe55fff700 1 -- 172.21.15.98:0/1251952039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe500d0e80 con 0x7efe500b8d10 2023-03-03T07:56:27.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.658+0000 7efe55fff700 1 -- 172.21.15.98:0/1251952039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe500cdd00 con 0x7efe500b8d10 2023-03-03T07:56:27.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.659+0000 7efe55fff700 1 -- 172.21.15.98:0/1251952039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe500cdfe0 con 0x7efe500b8d10 2023-03-03T07:56:27.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.660+0000 7efe37fff700 1 -- 172.21.15.98:0/1251952039 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efe48014480 con 0x7efe500b8d10 2023-03-03T07:56:27.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.662+0000 7efe37fff700 1 --2- 172.21.15.98:0/1251952039 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efe3c0682f0 0x7efe3c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:27.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.662+0000 7efe37fff700 1 -- 172.21.15.98:0/1251952039 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7efe48086650 con 0x7efe500b8d10 2023-03-03T07:56:27.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.662+0000 7efe4f7fe700 1 --2- 172.21.15.98:0/1251952039 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efe3c0682f0 0x7efe3c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:27.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.663+0000 7efe4f7fe700 1 --2- 172.21.15.98:0/1251952039 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efe3c0682f0 0x7efe3c06a7a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efe40009b60 tx=0x7efe40009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:27.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.666+0000 7efe37fff700 1 -- 172.21.15.98:0/1251952039 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efe48052d20 con 0x7efe500b8d10 2023-03-03T07:56:27.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.858+0000 7efe55fff700 1 -- 172.21.15.98:0/1251952039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efe5004e6e0 con 0x7efe500b8d10 2023-03-03T07:56:27.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.858+0000 7efe37fff700 1 -- 172.21.15.98:0/1251952039 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7efe48056340 con 0x7efe500b8d10 2023-03-03T07:56:27.860 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:56:27.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.861+0000 7efe55fff700 1 -- 172.21.15.98:0/1251952039 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efe3c0682f0 msgr2=0x7efe3c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:27.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.861+0000 7efe55fff700 1 --2- 172.21.15.98:0/1251952039 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efe3c0682f0 0x7efe3c06a7a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efe40009b60 tx=0x7efe40009210 comp rx=0 tx=0).stop 2023-03-03T07:56:27.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.861+0000 7efe55fff700 1 -- 172.21.15.98:0/1251952039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe500b8d10 msgr2=0x7efe5015f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:27.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.861+0000 7efe55fff700 1 --2- 172.21.15.98:0/1251952039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe500b8d10 0x7efe5015f510 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efe4800efc0 tx=0x7efe4800c0c0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.861+0000 7efe55fff700 1 -- 172.21.15.98:0/1251952039 shutdown_connections 2023-03-03T07:56:27.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.861+0000 7efe55fff700 1 --2- 172.21.15.98:0/1251952039 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efe3c0682f0 0x7efe3c06a7a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.861+0000 7efe55fff700 1 --2- 172.21.15.98:0/1251952039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe500b8d10 0x7efe5015f510 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.861+0000 7efe55fff700 1 --2- 172.21.15.98:0/1251952039 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe500b26d0 0x7efe500b14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.861+0000 7efe55fff700 1 --2- 172.21.15.98:0/1251952039 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe500b5140 0x7efe500cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:27.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.861+0000 7efe55fff700 1 -- 172.21.15.98:0/1251952039 >> 172.21.15.98:0/1251952039 conn(0x7efe50073e70 msgr2=0x7efe500d10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:27.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.861+0000 7efe55fff700 1 -- 172.21.15.98:0/1251952039 shutdown_connections 2023-03-03T07:56:27.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:27.861+0000 7efe55fff700 1 -- 172.21.15.98:0/1251952039 wait complete. 2023-03-03T07:56:27.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:56:28.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.134+0000 7fefff39d700 1 Processor -- start 2023-03-03T07:56:28.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.134+0000 7fefff39d700 1 -- start start 2023-03-03T07:56:28.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.136+0000 7fefff39d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7feff80cf570 0x7feff815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:28.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.136+0000 7fefff39d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff80cd780 con 0x7feff81609b0 2023-03-03T07:56:28.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.136+0000 7fefff39d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff80cda60 con 0x7feff815db30 2023-03-03T07:56:28.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.136+0000 7fefff39d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff80cebb0 con 0x7feff80cf570 2023-03-03T07:56:28.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.136+0000 7feffe39b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7feff81609b0 0x7feff8160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39784/0 (socket says 172.21.15.98:39784) 2023-03-03T07:56:28.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.136+0000 7feffe39b700 1 -- 172.21.15.98:0/3624046395 learned_addr learned my addr 172.21.15.98:0/3624046395 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:28.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.136+0000 7feffdb9a700 1 --2- 172.21.15.98:0/3624046395 >> v2:172.21.15.100:3300/0 conn(0x7feff80cf570 0x7feff815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:28.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.137+0000 7feffd399700 1 -- 172.21.15.98:0/3624046395 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872485721 0 0) 0x7feff80cd780 con 0x7feff81609b0 2023-03-03T07:56:28.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.137+0000 7feffd399700 1 -- 172.21.15.98:0/3624046395 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe40035b0 con 0x7feff81609b0 2023-03-03T07:56:28.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.137+0000 7feffd399700 1 -- 172.21.15.98:0/3624046395 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784016751 0 0) 0x7feff80cda60 con 0x7feff815db30 2023-03-03T07:56:28.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.137+0000 7feffd399700 1 -- 172.21.15.98:0/3624046395 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff80cd780 con 0x7feff815db30 2023-03-03T07:56:28.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.137+0000 7feffdb9a700 1 -- 172.21.15.98:0/3624046395 >> v1:172.21.15.100:6789/0 conn(0x7feff815db30 legacy=0x7feff80b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:28.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.137+0000 7feffdb9a700 1 -- 172.21.15.98:0/3624046395 >> v1:172.21.15.98:6789/0 conn(0x7feff81609b0 legacy=0x7feff8160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:28.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.137+0000 7feffd399700 1 -- 172.21.15.98:0/3624046395 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3593607060 0 0) 0x7fefe40035b0 con 0x7feff81609b0 2023-03-03T07:56:28.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.137+0000 7feffdb9a700 1 -- 172.21.15.98:0/3624046395 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff80cdd40 con 0x7feff80cf570 2023-03-03T07:56:28.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.137+0000 7feffdb9a700 1 --2- 172.21.15.98:0/3624046395 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff80cf570 0x7feff815d5f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fefe8009840 tx=0x7fefe8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89a2fc64b046bb46 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:28.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.138+0000 7feffd399700 1 -- 172.21.15.98:0/3624046395 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff4001670 con 0x7feff80cf570 2023-03-03T07:56:28.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.138+0000 7feffd399700 1 -- 172.21.15.98:0/3624046395 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe8004030 con 0x7feff80cf570 2023-03-03T07:56:28.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.138+0000 7feffd399700 1 -- 172.21.15.98:0/3624046395 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe80043c0 con 0x7feff80cf570 2023-03-03T07:56:28.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.138+0000 7fefff39d700 1 -- 172.21.15.98:0/3624046395 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff80cf570 msgr2=0x7feff815d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:28.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.138+0000 7fefff39d700 1 --2- 172.21.15.98:0/3624046395 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff80cf570 0x7feff815d5f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fefe8009840 tx=0x7fefe8009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:28.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.138+0000 7fefff39d700 1 -- 172.21.15.98:0/3624046395 shutdown_connections 2023-03-03T07:56:28.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.138+0000 7fefff39d700 1 --2- 172.21.15.98:0/3624046395 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff80cf570 0x7feff815d5f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.138+0000 7fefff39d700 1 -- 172.21.15.98:0/3624046395 >> 172.21.15.98:0/3624046395 conn(0x7feff806b880 msgr2=0x7feff80cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:28.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.138+0000 7fefff39d700 1 -- 172.21.15.98:0/3624046395 shutdown_connections 2023-03-03T07:56:28.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.138+0000 7fefff39d700 1 -- 172.21.15.98:0/3624046395 wait complete. 2023-03-03T07:56:28.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.140+0000 7fefff39d700 1 Processor -- start 2023-03-03T07:56:28.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.141+0000 7fefff39d700 1 -- start start 2023-03-03T07:56:28.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.142+0000 7fefff39d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff80cf570 0x7feff80b8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:28.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.142+0000 7fefff39d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feff815db30 0x7feff814f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:28.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.142+0000 7fefff39d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feff81609b0 0x7feff80b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:28.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.142+0000 7fefff39d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff80b3b00 con 0x7feff81609b0 2023-03-03T07:56:28.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.142+0000 7fefff39d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff814fb50 con 0x7feff80cf570 2023-03-03T07:56:28.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.142+0000 7fefff39d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff80caa40 con 0x7feff815db30 2023-03-03T07:56:28.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.142+0000 7feffeb9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feff81609b0 0x7feff80b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:28.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.142+0000 7feffeb9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feff81609b0 0x7feff80b2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40580/0 (socket says 172.21.15.98:40580) 2023-03-03T07:56:28.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.142+0000 7feffeb9c700 1 -- 172.21.15.98:0/3504833140 learned_addr learned my addr 172.21.15.98:0/3504833140 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:28.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.142+0000 7feffe39b700 1 --2- 172.21.15.98:0/3504833140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff80cf570 0x7feff80b8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:28.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.143+0000 7feffdb9a700 1 --2- 172.21.15.98:0/3504833140 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feff815db30 0x7feff814f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:28.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.143+0000 7feffeb9c700 1 -- 172.21.15.98:0/3504833140 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feff815db30 msgr2=0x7feff814f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:28.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.143+0000 7feffeb9c700 1 --2- 172.21.15.98:0/3504833140 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feff815db30 0x7feff814f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.143+0000 7feffeb9c700 1 -- 172.21.15.98:0/3504833140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff80cf570 msgr2=0x7feff80b8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:28.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.143+0000 7feffeb9c700 1 --2- 172.21.15.98:0/3504833140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff80cf570 0x7feff80b8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.143+0000 7feffeb9c700 1 -- 172.21.15.98:0/3504833140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe80094a0 con 0x7feff81609b0 2023-03-03T07:56:28.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.143+0000 7feffeb9c700 1 --2- 172.21.15.98:0/3504833140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feff81609b0 0x7feff80b2a80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7feff000cbc0 tx=0x7feff000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:28.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.143+0000 7fefeeffd700 1 -- 172.21.15.98:0/3504833140 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0003c00 con 0x7feff81609b0 2023-03-03T07:56:28.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.143+0000 7fefeeffd700 1 -- 172.21.15.98:0/3504833140 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff0003d60 con 0x7feff81609b0 2023-03-03T07:56:28.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.143+0000 7fefeeffd700 1 -- 172.21.15.98:0/3504833140 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff000ab40 con 0x7feff81609b0 2023-03-03T07:56:28.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.143+0000 7fefff39d700 1 -- 172.21.15.98:0/3504833140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff815d0d0 con 0x7feff81609b0 2023-03-03T07:56:28.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.144+0000 7fefff39d700 1 -- 172.21.15.98:0/3504833140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff815cdf0 con 0x7feff81609b0 2023-03-03T07:56:28.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.145+0000 7fefff39d700 1 -- 172.21.15.98:0/3504833140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff80b51b0 con 0x7feff81609b0 2023-03-03T07:56:28.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.146+0000 7fefeeffd700 1 -- 172.21.15.98:0/3504833140 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feff0010050 con 0x7feff81609b0 2023-03-03T07:56:28.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.147+0000 7fefeeffd700 1 --2- 172.21.15.98:0/3504833140 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fefdc0682f0 0x7fefdc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:28.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.147+0000 7fefeeffd700 1 -- 172.21.15.98:0/3504833140 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7feff0085ce0 con 0x7feff81609b0 2023-03-03T07:56:28.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.147+0000 7feffe39b700 1 --2- 172.21.15.98:0/3504833140 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fefdc0682f0 0x7fefdc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:28.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.148+0000 7feffe39b700 1 --2- 172.21.15.98:0/3504833140 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fefdc0682f0 0x7fefdc06a7a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7feff4007fd0 tx=0x7feff4007dd0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:28.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.151+0000 7fefeeffd700 1 -- 172.21.15.98:0/3504833140 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feff008c080 con 0x7feff81609b0 2023-03-03T07:56:28.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.347+0000 7fefff39d700 1 -- 172.21.15.98:0/3504833140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feff80b3020 con 0x7feff81609b0 2023-03-03T07:56:28.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.348+0000 7fefeeffd700 1 -- 172.21.15.98:0/3504833140 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7feff80b3020 con 0x7feff81609b0 2023-03-03T07:56:28.349 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:56:28.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.350+0000 7fefff39d700 1 -- 172.21.15.98:0/3504833140 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fefdc0682f0 msgr2=0x7fefdc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:28.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.350+0000 7fefff39d700 1 --2- 172.21.15.98:0/3504833140 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fefdc0682f0 0x7fefdc06a7a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7feff4007fd0 tx=0x7feff4007dd0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.350+0000 7fefff39d700 1 -- 172.21.15.98:0/3504833140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feff81609b0 msgr2=0x7feff80b2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:28.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.350+0000 7fefff39d700 1 --2- 172.21.15.98:0/3504833140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feff81609b0 0x7feff80b2a80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7feff000cbc0 tx=0x7feff000ced0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.350+0000 7fefff39d700 1 -- 172.21.15.98:0/3504833140 shutdown_connections 2023-03-03T07:56:28.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.350+0000 7fefff39d700 1 --2- 172.21.15.98:0/3504833140 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fefdc0682f0 0x7fefdc06a7a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.350+0000 7fefff39d700 1 --2- 172.21.15.98:0/3504833140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feff81609b0 0x7feff80b2a80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.350+0000 7fefff39d700 1 --2- 172.21.15.98:0/3504833140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff80cf570 0x7feff80b8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.350+0000 7fefff39d700 1 --2- 172.21.15.98:0/3504833140 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feff815db30 0x7feff814f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.350+0000 7fefff39d700 1 -- 172.21.15.98:0/3504833140 >> 172.21.15.98:0/3504833140 conn(0x7feff806b880 msgr2=0x7feff8152280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:28.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.351+0000 7fefff39d700 1 -- 172.21.15.98:0/3504833140 shutdown_connections 2023-03-03T07:56:28.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.351+0000 7fefff39d700 1 -- 172.21.15.98:0/3504833140 wait complete. 2023-03-03T07:56:28.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:56:28.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.624+0000 7fcf8719d700 1 Processor -- start 2023-03-03T07:56:28.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.624+0000 7fcf8719d700 1 -- start start 2023-03-03T07:56:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.626+0000 7fcf8719d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf800b51c0 0x7fcf80160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.626+0000 7fcf8719d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcf800d1580 0x7fcf800d19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.626+0000 7fcf8719d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf801614c0 con 0x7fcf80146360 2023-03-03T07:56:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.626+0000 7fcf8719d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf800d10f0 con 0x7fcf800d1580 2023-03-03T07:56:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.626+0000 7fcf8719d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800cdf20 con 0x7fcf800b51c0 2023-03-03T07:56:28.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.626+0000 7fcf8699c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcf800d1580 0x7fcf800d19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:28.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.626+0000 7fcf8699c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcf800d1580 0x7fcf800d19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40592/0 (socket says 172.21.15.98:40592) 2023-03-03T07:56:28.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.626+0000 7fcf8699c700 1 -- 172.21.15.98:0/1896170182 learned_addr learned my addr 172.21.15.98:0/1896170182 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:28.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.626+0000 7fcf8599a700 1 --2- 172.21.15.98:0/1896170182 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf800b51c0 0x7fcf80160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:28.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.627+0000 7fcf8699c700 1 -- 172.21.15.98:0/1896170182 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf800b51c0 msgr2=0x7fcf80160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:28.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.627+0000 7fcf8699c700 1 --2- 172.21.15.98:0/1896170182 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf800b51c0 0x7fcf80160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.627+0000 7fcf8699c700 1 -- 172.21.15.98:0/1896170182 >> v1:172.21.15.100:6789/0 conn(0x7fcf80146360 legacy=0x7fcf800b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:28.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.627+0000 7fcf8699c700 1 -- 172.21.15.98:0/1896170182 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf800cd680 con 0x7fcf800d1580 2023-03-03T07:56:28.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.627+0000 7fcf8599a700 1 --2- 172.21.15.98:0/1896170182 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf800b51c0 0x7fcf80160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:28.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.627+0000 7fcf8699c700 1 --2- 172.21.15.98:0/1896170182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf800d1580 0x7fcf800d19e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009e50 tx=0x7fcf7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f443630a972ccd39 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:28.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.627+0000 7fcf85199700 1 -- 172.21.15.98:0/1896170182 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7800e070 con 0x7fcf800d1580 2023-03-03T07:56:28.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.627+0000 7fcf85199700 1 -- 172.21.15.98:0/1896170182 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7800b9c0 con 0x7fcf800d1580 2023-03-03T07:56:28.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.627+0000 7fcf85199700 1 -- 172.21.15.98:0/1896170182 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7800bd70 con 0x7fcf800d1580 2023-03-03T07:56:28.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.628+0000 7fcf8719d700 1 -- 172.21.15.98:0/1896170182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf800d1580 msgr2=0x7fcf800d19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:28.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.628+0000 7fcf8719d700 1 --2- 172.21.15.98:0/1896170182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf800d1580 0x7fcf800d19e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009e50 tx=0x7fcf7800b330 comp rx=0 tx=0).stop 2023-03-03T07:56:28.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.628+0000 7fcf8719d700 1 -- 172.21.15.98:0/1896170182 shutdown_connections 2023-03-03T07:56:28.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.628+0000 7fcf8719d700 1 --2- 172.21.15.98:0/1896170182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf800d1580 0x7fcf800d19e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.628+0000 7fcf8719d700 1 --2- 172.21.15.98:0/1896170182 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf800b51c0 0x7fcf80160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.628+0000 7fcf8719d700 1 -- 172.21.15.98:0/1896170182 >> 172.21.15.98:0/1896170182 conn(0x7fcf8006b8e0 msgr2=0x7fcf800cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:28.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.628+0000 7fcf8719d700 1 -- 172.21.15.98:0/1896170182 shutdown_connections 2023-03-03T07:56:28.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.628+0000 7fcf8719d700 1 -- 172.21.15.98:0/1896170182 wait complete. 2023-03-03T07:56:28.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.630+0000 7fcf8719d700 1 Processor -- start 2023-03-03T07:56:28.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.631+0000 7fcf8719d700 1 -- start start 2023-03-03T07:56:28.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.632+0000 7fcf8719d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf800b51c0 0x7fcf801f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:28.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.632+0000 7fcf8719d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcf800d1580 0x7fcf800b2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:28.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.632+0000 7fcf8719d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf80146360 0x7fcf800b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:28.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.632+0000 7fcf8719d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80152740 con 0x7fcf800b51c0 2023-03-03T07:56:28.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.632+0000 7fcf8719d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80153c50 con 0x7fcf800d1580 2023-03-03T07:56:28.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.632+0000 7fcf8719d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf801f2640 con 0x7fcf80146360 2023-03-03T07:56:28.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.632+0000 7fcf8619b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf800b51c0 0x7fcf801f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:28.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.632+0000 7fcf8619b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf800b51c0 0x7fcf801f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40606/0 (socket says 172.21.15.98:40606) 2023-03-03T07:56:28.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.632+0000 7fcf8619b700 1 -- 172.21.15.98:0/1612054930 learned_addr learned my addr 172.21.15.98:0/1612054930 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:28.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.632+0000 7fcf8699c700 1 --2- 172.21.15.98:0/1612054930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf80146360 0x7fcf800b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:28.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.633+0000 7fcf8619b700 1 -- 172.21.15.98:0/1612054930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf80146360 msgr2=0x7fcf800b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:28.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.633+0000 7fcf8619b700 1 --2- 172.21.15.98:0/1612054930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf80146360 0x7fcf800b4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.633+0000 7fcf8599a700 1 --2- 172.21.15.98:0/1612054930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcf800d1580 0x7fcf800b2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:28.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.633+0000 7fcf8619b700 1 -- 172.21.15.98:0/1612054930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcf800d1580 msgr2=0x7fcf800b2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:28.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.633+0000 7fcf8619b700 1 --2- 172.21.15.98:0/1612054930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcf800d1580 0x7fcf800b2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.633+0000 7fcf8619b700 1 -- 172.21.15.98:0/1612054930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf78009b30 con 0x7fcf800b51c0 2023-03-03T07:56:28.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.633+0000 7fcf8699c700 1 --2- 172.21.15.98:0/1612054930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf80146360 0x7fcf800b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:56:28.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.633+0000 7fcf8619b700 1 --2- 172.21.15.98:0/1612054930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf800b51c0 0x7fcf801f3250 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c00c4e0 tx=0x7fcf7c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:28.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.633+0000 7fcf76ffd700 1 -- 172.21.15.98:0/1612054930 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c0052d0 con 0x7fcf800b51c0 2023-03-03T07:56:28.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.633+0000 7fcf76ffd700 1 -- 172.21.15.98:0/1612054930 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7c005430 con 0x7fcf800b51c0 2023-03-03T07:56:28.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.633+0000 7fcf76ffd700 1 -- 172.21.15.98:0/1612054930 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c0038c0 con 0x7fcf800b51c0 2023-03-03T07:56:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.633+0000 7fcf8719d700 1 -- 172.21.15.98:0/1612054930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf800cdc40 con 0x7fcf800b51c0 2023-03-03T07:56:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.633+0000 7fcf8719d700 1 -- 172.21.15.98:0/1612054930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf800cd960 con 0x7fcf800b51c0 2023-03-03T07:56:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.634+0000 7fcf8719d700 1 -- 172.21.15.98:0/1612054930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf800d0d80 con 0x7fcf800b51c0 2023-03-03T07:56:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.636+0000 7fcf76ffd700 1 -- 172.21.15.98:0/1612054930 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcf7c019030 con 0x7fcf800b51c0 2023-03-03T07:56:28.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.638+0000 7fcf76ffd700 1 --2- 172.21.15.98:0/1612054930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcf640682f0 0x7fcf6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:28.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.638+0000 7fcf76ffd700 1 -- 172.21.15.98:0/1612054930 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fcf7c084fd0 con 0x7fcf800b51c0 2023-03-03T07:56:28.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.638+0000 7fcf8599a700 1 --2- 172.21.15.98:0/1612054930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcf640682f0 0x7fcf6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:28.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.639+0000 7fcf8599a700 1 --2- 172.21.15.98:0/1612054930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcf640682f0 0x7fcf6406a7a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcf80073850 tx=0x7fcf70008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:28.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.642+0000 7fcf76ffd700 1 -- 172.21.15.98:0/1612054930 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcf7c051700 con 0x7fcf800b51c0 2023-03-03T07:56:28.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.838+0000 7fcf8719d700 1 -- 172.21.15.98:0/1612054930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf801614c0 con 0x7fcf800b51c0 2023-03-03T07:56:28.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.839+0000 7fcf76ffd700 1 -- 172.21.15.98:0/1612054930 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18003 (secure 0 0 0) 0x7fcf801614c0 con 0x7fcf800b51c0 2023-03-03T07:56:28.842 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:56:28.842 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:56:27.498205+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:56:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.842+0000 7fcf8719d700 1 -- 172.21.15.98:0/1612054930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcf640682f0 msgr2=0x7fcf6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.842+0000 7fcf8719d700 1 --2- 172.21.15.98:0/1612054930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcf640682f0 0x7fcf6406a7a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcf80073850 tx=0x7fcf70008040 comp rx=0 tx=0).stop 2023-03-03T07:56:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.842+0000 7fcf8719d700 1 -- 172.21.15.98:0/1612054930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf800b51c0 msgr2=0x7fcf801f3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.842+0000 7fcf8719d700 1 --2- 172.21.15.98:0/1612054930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf800b51c0 0x7fcf801f3250 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c00c4e0 tx=0x7fcf7c00c8a0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.843+0000 7fcf8719d700 1 -- 172.21.15.98:0/1612054930 shutdown_connections 2023-03-03T07:56:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.843+0000 7fcf8719d700 1 --2- 172.21.15.98:0/1612054930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcf640682f0 0x7fcf6406a7a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.843+0000 7fcf8719d700 1 --2- 172.21.15.98:0/1612054930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf800b51c0 0x7fcf801f3250 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.843+0000 7fcf8719d700 1 --2- 172.21.15.98:0/1612054930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf80146360 0x7fcf800b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.843+0000 7fcf8719d700 1 --2- 172.21.15.98:0/1612054930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcf800d1580 0x7fcf800b2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.843+0000 7fcf8719d700 1 -- 172.21.15.98:0/1612054930 >> 172.21.15.98:0/1612054930 conn(0x7fcf8006b8e0 msgr2=0x7fcf800b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.843+0000 7fcf8719d700 1 -- 172.21.15.98:0/1612054930 shutdown_connections 2023-03-03T07:56:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:28.843+0000 7fcf8719d700 1 -- 172.21.15.98:0/1612054930 wait complete. 2023-03-03T07:56:28.872 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:55:45.157666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:55:46.092091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:56:28.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:56:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.119+0000 7f1db2dff700 1 Processor -- start 2023-03-03T07:56:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.119+0000 7f1db2dff700 1 -- start start 2023-03-03T07:56:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.120+0000 7f1db2dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1dac0b8d10 0x7f1dac0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.121+0000 7f1db2dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac0cd7d0 con 0x7f1dac0b5310 2023-03-03T07:56:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.121+0000 7f1db2dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac0b90e0 con 0x7f1dac0b26d0 2023-03-03T07:56:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.121+0000 7f1db2dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0cda20 con 0x7f1dac0b8d10 2023-03-03T07:56:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.121+0000 7f1db0b9b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1dac0b8d10 0x7f1dac0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.121+0000 7f1db139c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1dac0b5310 0x7f1dac0d13a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39796/0 (socket says 172.21.15.98:39796) 2023-03-03T07:56:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.121+0000 7f1db139c700 1 -- 172.21.15.98:0/1280598611 learned_addr learned my addr 172.21.15.98:0/1280598611 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.121+0000 7f1dab7fe700 1 -- 172.21.15.98:0/1280598611 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616334456 0 0) 0x7f1dac0cd7d0 con 0x7f1dac0b5310 2023-03-03T07:56:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.121+0000 7f1db0b9b700 1 -- 172.21.15.98:0/1280598611 >> v1:172.21.15.100:6789/0 conn(0x7f1dac0b26d0 legacy=0x7f1dac0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.121+0000 7f1db0b9b700 1 -- 172.21.15.98:0/1280598611 >> v1:172.21.15.98:6789/0 conn(0x7f1dac0b5310 legacy=0x7f1dac0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.122+0000 7f1db0b9b700 1 -- 172.21.15.98:0/1280598611 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac0cdd00 con 0x7f1dac0b8d10 2023-03-03T07:56:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.122+0000 7f1db0b9b700 1 --2- 172.21.15.98:0/1280598611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1dac0b8d10 0x7f1dac0cffc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1da00097f0 tx=0x7f1da0009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fd09a3a913a3f59 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:29.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.122+0000 7f1dab7fe700 1 -- 172.21.15.98:0/1280598611 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0004230 con 0x7f1dac0b8d10 2023-03-03T07:56:29.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.123+0000 7f1dab7fe700 1 -- 172.21.15.98:0/1280598611 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da0004390 con 0x7f1dac0b8d10 2023-03-03T07:56:29.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.123+0000 7f1dab7fe700 1 -- 172.21.15.98:0/1280598611 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da00036a0 con 0x7f1dac0b8d10 2023-03-03T07:56:29.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.123+0000 7f1db2dff700 1 -- 172.21.15.98:0/1280598611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1dac0b8d10 msgr2=0x7f1dac0cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:29.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.123+0000 7f1db2dff700 1 --2- 172.21.15.98:0/1280598611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1dac0b8d10 0x7f1dac0cffc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1da00097f0 tx=0x7f1da0009b00 comp rx=0 tx=0).stop 2023-03-03T07:56:29.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.123+0000 7f1db2dff700 1 -- 172.21.15.98:0/1280598611 shutdown_connections 2023-03-03T07:56:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.123+0000 7f1db2dff700 1 --2- 172.21.15.98:0/1280598611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1dac0b8d10 0x7f1dac0cffc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.123+0000 7f1db2dff700 1 -- 172.21.15.98:0/1280598611 >> 172.21.15.98:0/1280598611 conn(0x7f1dac073e70 msgr2=0x7f1dac06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.123+0000 7f1db2dff700 1 -- 172.21.15.98:0/1280598611 shutdown_connections 2023-03-03T07:56:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.123+0000 7f1db2dff700 1 -- 172.21.15.98:0/1280598611 wait complete. 2023-03-03T07:56:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.125+0000 7f1db2dff700 1 Processor -- start 2023-03-03T07:56:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.126+0000 7f1db2dff700 1 -- start start 2023-03-03T07:56:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db2dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dac0b26d0 0x7f1dac14eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db2dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1dac0b5310 0x7f1dac0cc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db2dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1dac0b8d10 0x7f1dac0ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db2dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0cb1e0 con 0x7f1dac0b26d0 2023-03-03T07:56:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db2dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0cb4c0 con 0x7f1dac0b5310 2023-03-03T07:56:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db2dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0cb7a0 con 0x7f1dac0b8d10 2023-03-03T07:56:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db0b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dac0b26d0 0x7f1dac14eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db0b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dac0b26d0 0x7f1dac14eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40620/0 (socket says 172.21.15.98:40620) 2023-03-03T07:56:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db0b9b700 1 -- 172.21.15.98:0/669073656 learned_addr learned my addr 172.21.15.98:0/669073656 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1dabfff700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1dac0b5310 0x7f1dac0cc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db139c700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1dac0b8d10 0x7f1dac0ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db0b9b700 1 -- 172.21.15.98:0/669073656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1dac0b8d10 msgr2=0x7f1dac0ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db0b9b700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1dac0b8d10 0x7f1dac0ca760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db0b9b700 1 -- 172.21.15.98:0/669073656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1dac0b5310 msgr2=0x7f1dac0cc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db0b9b700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1dac0b5310 0x7f1dac0cc280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db0b9b700 1 -- 172.21.15.98:0/669073656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da00094a0 con 0x7f1dac0b26d0 2023-03-03T07:56:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1db139c700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1dac0b8d10 0x7f1dac0ca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.127+0000 7f1dabfff700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1dac0b5310 0x7f1dac0cc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.128+0000 7f1db0b9b700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dac0b26d0 0x7f1dac14eab0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1da00097c0 tx=0x7f1da0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.128+0000 7f1da97fa700 1 -- 172.21.15.98:0/669073656 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0011630 con 0x7f1dac0b26d0 2023-03-03T07:56:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.128+0000 7f1da97fa700 1 -- 172.21.15.98:0/669073656 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da001f070 con 0x7f1dac0b26d0 2023-03-03T07:56:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.128+0000 7f1db2dff700 1 -- 172.21.15.98:0/669073656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac0cdfe0 con 0x7f1dac0b26d0 2023-03-03T07:56:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.128+0000 7f1da97fa700 1 -- 172.21.15.98:0/669073656 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da00119d0 con 0x7f1dac0b26d0 2023-03-03T07:56:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.128+0000 7f1db2dff700 1 -- 172.21.15.98:0/669073656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac0d0e80 con 0x7f1dac0b26d0 2023-03-03T07:56:29.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.129+0000 7f1db2dff700 1 -- 172.21.15.98:0/669073656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac0d0500 con 0x7f1dac0b26d0 2023-03-03T07:56:29.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.131+0000 7f1da97fa700 1 -- 172.21.15.98:0/669073656 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1da0017020 con 0x7f1dac0b26d0 2023-03-03T07:56:29.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.132+0000 7f1da97fa700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1d900682f0 0x7f1d9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:29.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.132+0000 7f1da97fa700 1 -- 172.21.15.98:0/669073656 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f1da0021070 con 0x7f1dac0b26d0 2023-03-03T07:56:29.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.132+0000 7f1dabfff700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1d900682f0 0x7f1d9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:29.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.133+0000 7f1dabfff700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1d900682f0 0x7f1d9006a7a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0ca690 tx=0x7f1d98009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:29.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.136+0000 7f1da97fa700 1 -- 172.21.15.98:0/669073656 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1da0054020 con 0x7f1dac0b26d0 2023-03-03T07:56:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.327+0000 7f1db2dff700 1 -- 172.21.15.98:0/669073656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1dac04e6e0 con 0x7f1dac0b26d0 2023-03-03T07:56:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.328+0000 7f1da97fa700 1 -- 172.21.15.98:0/669073656 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1da0057640 con 0x7f1dac0b26d0 2023-03-03T07:56:29.331 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:56:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.331+0000 7f1db2dff700 1 -- 172.21.15.98:0/669073656 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1d900682f0 msgr2=0x7f1d9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.331+0000 7f1db2dff700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1d900682f0 0x7f1d9006a7a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0ca690 tx=0x7f1d98009040 comp rx=0 tx=0).stop 2023-03-03T07:56:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.331+0000 7f1db2dff700 1 -- 172.21.15.98:0/669073656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dac0b26d0 msgr2=0x7f1dac14eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.331+0000 7f1db2dff700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dac0b26d0 0x7f1dac14eab0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1da00097c0 tx=0x7f1da0004750 comp rx=0 tx=0).stop 2023-03-03T07:56:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.331+0000 7f1db2dff700 1 -- 172.21.15.98:0/669073656 shutdown_connections 2023-03-03T07:56:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.331+0000 7f1db2dff700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1d900682f0 0x7f1d9006a7a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.331+0000 7f1db2dff700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dac0b26d0 0x7f1dac14eab0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.331+0000 7f1db2dff700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1dac0b8d10 0x7f1dac0ca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.331+0000 7f1db2dff700 1 --2- 172.21.15.98:0/669073656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1dac0b5310 0x7f1dac0cc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.331+0000 7f1db2dff700 1 -- 172.21.15.98:0/669073656 >> 172.21.15.98:0/669073656 conn(0x7f1dac073e70 msgr2=0x7f1dac0b6180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.331+0000 7f1db2dff700 1 -- 172.21.15.98:0/669073656 shutdown_connections 2023-03-03T07:56:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.331+0000 7f1db2dff700 1 -- 172.21.15.98:0/669073656 wait complete. 2023-03-03T07:56:29.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:56:29.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.607+0000 7fec8edff700 1 Processor -- start 2023-03-03T07:56:29.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.608+0000 7fec8edff700 1 -- start start 2023-03-03T07:56:29.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.609+0000 7fec8edff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fec880b26d0 0x7fec880b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:29.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.609+0000 7fec8edff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec880cd7d0 con 0x7fec880b5140 2023-03-03T07:56:29.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.609+0000 7fec8edff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec880b90e0 con 0x7fec880b8d10 2023-03-03T07:56:29.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.609+0000 7fec8edff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec880b55c0 con 0x7fec880b26d0 2023-03-03T07:56:29.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.609+0000 7fec8d39c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fec880b5140 0x7fec880d13a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39812/0 (socket says 172.21.15.98:39812) 2023-03-03T07:56:29.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.609+0000 7fec8d39c700 1 -- 172.21.15.98:0/2530914502 learned_addr learned my addr 172.21.15.98:0/2530914502 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:29.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.609+0000 7fec87fff700 1 --2- 172.21.15.98:0/2530914502 >> v2:172.21.15.98:3300/0 conn(0x7fec880b26d0 0x7fec880b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:29.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.610+0000 7fec877fe700 1 -- 172.21.15.98:0/2530914502 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283223631 0 0) 0x7fec880cd7d0 con 0x7fec880b5140 2023-03-03T07:56:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.610+0000 7fec877fe700 1 -- 172.21.15.98:0/2530914502 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec780035b0 con 0x7fec880b5140 2023-03-03T07:56:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.610+0000 7fec87fff700 1 -- 172.21.15.98:0/2530914502 >> v1:172.21.15.100:6789/0 conn(0x7fec880b8d10 legacy=0x7fec880cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.610+0000 7fec877fe700 1 -- 172.21.15.98:0/2530914502 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435269595 0 0) 0x7fec880b90e0 con 0x7fec880b8d10 2023-03-03T07:56:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.610+0000 7fec87fff700 1 -- 172.21.15.98:0/2530914502 >> v1:172.21.15.98:6789/0 conn(0x7fec880b5140 legacy=0x7fec880d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.610+0000 7fec87fff700 1 -- 172.21.15.98:0/2530914502 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec880cda20 con 0x7fec880b26d0 2023-03-03T07:56:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.610+0000 7fec87fff700 1 --2- 172.21.15.98:0/2530914502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec880b26d0 0x7fec880b4c00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fec74009840 tx=0x7fec74009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36f8794889e3e424 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.611+0000 7fec877fe700 1 -- 172.21.15.98:0/2530914502 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74011830 con 0x7fec880b26d0 2023-03-03T07:56:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.611+0000 7fec877fe700 1 -- 172.21.15.98:0/2530914502 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec74011990 con 0x7fec880b26d0 2023-03-03T07:56:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.611+0000 7fec877fe700 1 -- 172.21.15.98:0/2530914502 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74011d20 con 0x7fec880b26d0 2023-03-03T07:56:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.611+0000 7fec8edff700 1 -- 172.21.15.98:0/2530914502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec880b26d0 msgr2=0x7fec880b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.611+0000 7fec8edff700 1 --2- 172.21.15.98:0/2530914502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec880b26d0 0x7fec880b4c00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fec74009840 tx=0x7fec74009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.611+0000 7fec8edff700 1 -- 172.21.15.98:0/2530914502 shutdown_connections 2023-03-03T07:56:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.611+0000 7fec8edff700 1 --2- 172.21.15.98:0/2530914502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec880b26d0 0x7fec880b4c00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.611+0000 7fec8edff700 1 -- 172.21.15.98:0/2530914502 >> 172.21.15.98:0/2530914502 conn(0x7fec88073e70 msgr2=0x7fec8806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.611+0000 7fec8edff700 1 -- 172.21.15.98:0/2530914502 shutdown_connections 2023-03-03T07:56:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.612+0000 7fec8edff700 1 -- 172.21.15.98:0/2530914502 wait complete. 2023-03-03T07:56:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.613+0000 7fec8edff700 1 Processor -- start 2023-03-03T07:56:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.614+0000 7fec8edff700 1 -- start start 2023-03-03T07:56:29.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.614+0000 7fec8edff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec880b26d0 0x7fec881433d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.615+0000 7fec8edff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec880b5140 0x7fec8815d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.615+0000 7fec8edff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec880b8d10 0x7fec880cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.615+0000 7fec8edff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec88153060 con 0x7fec880b8d10 2023-03-03T07:56:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.615+0000 7fec8edff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec88155fe0 con 0x7fec880b5140 2023-03-03T07:56:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.615+0000 7fec8edff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8815d790 con 0x7fec880b26d0 2023-03-03T07:56:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.615+0000 7fec8d39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec880b8d10 0x7fec880cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.615+0000 7fec8d39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec880b8d10 0x7fec880cbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40634/0 (socket says 172.21.15.98:40634) 2023-03-03T07:56:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.615+0000 7fec8d39c700 1 -- 172.21.15.98:0/3049713524 learned_addr learned my addr 172.21.15.98:0/3049713524 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.615+0000 7fec8cb9b700 1 --2- 172.21.15.98:0/3049713524 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec880b26d0 0x7fec881433d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.615+0000 7fec87fff700 1 --2- 172.21.15.98:0/3049713524 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec880b5140 0x7fec8815d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.615+0000 7fec8d39c700 1 -- 172.21.15.98:0/3049713524 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec880b26d0 msgr2=0x7fec881433d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.615+0000 7fec8d39c700 1 --2- 172.21.15.98:0/3049713524 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec880b26d0 0x7fec881433d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.615+0000 7fec8d39c700 1 -- 172.21.15.98:0/3049713524 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec880b5140 msgr2=0x7fec8815d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.615+0000 7fec8d39c700 1 --2- 172.21.15.98:0/3049713524 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec880b5140 0x7fec8815d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.615+0000 7fec8d39c700 1 -- 172.21.15.98:0/3049713524 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec740094a0 con 0x7fec880b8d10 2023-03-03T07:56:29.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.616+0000 7fec8d39c700 1 --2- 172.21.15.98:0/3049713524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec880b8d10 0x7fec880cbf90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fec8000caa0 tx=0x7fec8000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:29.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.616+0000 7fec857fa700 1 -- 172.21.15.98:0/3049713524 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8000aa80 con 0x7fec880b8d10 2023-03-03T07:56:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.616+0000 7fec857fa700 1 -- 172.21.15.98:0/3049713524 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8000abe0 con 0x7fec880b8d10 2023-03-03T07:56:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.616+0000 7fec857fa700 1 -- 172.21.15.98:0/3049713524 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec80012400 con 0x7fec880b8d10 2023-03-03T07:56:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.616+0000 7fec8edff700 1 -- 172.21.15.98:0/3049713524 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec880d0e80 con 0x7fec880b8d10 2023-03-03T07:56:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.616+0000 7fec8edff700 1 -- 172.21.15.98:0/3049713524 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec880cdd00 con 0x7fec880b8d10 2023-03-03T07:56:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.617+0000 7fec8edff700 1 -- 172.21.15.98:0/3049713524 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec880cdfe0 con 0x7fec880b8d10 2023-03-03T07:56:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.619+0000 7fec857fa700 1 -- 172.21.15.98:0/3049713524 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fec80019030 con 0x7fec880b8d10 2023-03-03T07:56:29.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.620+0000 7fec857fa700 1 --2- 172.21.15.98:0/3049713524 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fec6c0682f0 0x7fec6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:29.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.621+0000 7fec857fa700 1 -- 172.21.15.98:0/3049713524 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fec800859f0 con 0x7fec880b8d10 2023-03-03T07:56:29.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.621+0000 7fec8cb9b700 1 --2- 172.21.15.98:0/3049713524 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fec6c0682f0 0x7fec6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:29.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.621+0000 7fec8cb9b700 1 --2- 172.21.15.98:0/3049713524 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fec6c0682f0 0x7fec6c06a7a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fec7c000f60 tx=0x7fec7c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:29.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.625+0000 7fec857fa700 1 -- 172.21.15.98:0/3049713524 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fec80052120 con 0x7fec880b8d10 2023-03-03T07:56:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.815+0000 7fec8edff700 1 -- 172.21.15.98:0/3049713524 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fec880cbcb0 con 0x7fec880b8d10 2023-03-03T07:56:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.816+0000 7fec857fa700 1 -- 172.21.15.98:0/3049713524 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7fec80017020 con 0x7fec880b8d10 2023-03-03T07:56:29.819 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:56:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.818+0000 7fec8edff700 1 -- 172.21.15.98:0/3049713524 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fec6c0682f0 msgr2=0x7fec6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.818+0000 7fec8edff700 1 --2- 172.21.15.98:0/3049713524 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fec6c0682f0 0x7fec6c06a7a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fec7c000f60 tx=0x7fec7c009040 comp rx=0 tx=0).stop 2023-03-03T07:56:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.818+0000 7fec8edff700 1 -- 172.21.15.98:0/3049713524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec880b8d10 msgr2=0x7fec880cbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.818+0000 7fec8edff700 1 --2- 172.21.15.98:0/3049713524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec880b8d10 0x7fec880cbf90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fec8000caa0 tx=0x7fec8000ce60 comp rx=0 tx=0).stop 2023-03-03T07:56:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.819+0000 7fec8edff700 1 -- 172.21.15.98:0/3049713524 shutdown_connections 2023-03-03T07:56:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.819+0000 7fec8edff700 1 --2- 172.21.15.98:0/3049713524 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fec6c0682f0 0x7fec6c06a7a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.819+0000 7fec8edff700 1 --2- 172.21.15.98:0/3049713524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec880b8d10 0x7fec880cbf90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.819+0000 7fec8edff700 1 --2- 172.21.15.98:0/3049713524 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec880b26d0 0x7fec881433d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.819+0000 7fec8edff700 1 --2- 172.21.15.98:0/3049713524 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec880b5140 0x7fec8815d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.819+0000 7fec8edff700 1 -- 172.21.15.98:0/3049713524 >> 172.21.15.98:0/3049713524 conn(0x7fec88073e70 msgr2=0x7fec880b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.819+0000 7fec8edff700 1 -- 172.21.15.98:0/3049713524 shutdown_connections 2023-03-03T07:56:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:29.819+0000 7fec8edff700 1 -- 172.21.15.98:0/3049713524 wait complete. 2023-03-03T07:56:29.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:56:30.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.093+0000 7effb719d700 1 Processor -- start 2023-03-03T07:56:30.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.094+0000 7effb719d700 1 -- start start 2023-03-03T07:56:30.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.095+0000 7effb719d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7effb00cf570 0x7effb015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:30.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.095+0000 7effb719d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7effb015db30 0x7effb00b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:30.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.095+0000 7effb719d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb00cd780 con 0x7effb01609b0 2023-03-03T07:56:30.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.095+0000 7effb719d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb00cebb0 con 0x7effb015db30 2023-03-03T07:56:30.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.095+0000 7effb719d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb00cda60 con 0x7effb00cf570 2023-03-03T07:56:30.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.096+0000 7effb699c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7effb015db30 0x7effb00b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:30.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.096+0000 7effb599a700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7effb00cf570 0x7effb015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:30.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.096+0000 7effb699c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7effb015db30 0x7effb00b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40636/0 (socket says 172.21.15.98:40636) 2023-03-03T07:56:30.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.096+0000 7effb699c700 1 -- 172.21.15.98:0/1821906271 learned_addr learned my addr 172.21.15.98:0/1821906271 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:30.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.096+0000 7effb619b700 1 --1- 172.21.15.98:0/1821906271 >> v1:172.21.15.100:6789/0 conn(0x7effb01609b0 0x7effb0160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:38980/0 (socket says 172.21.15.98:38980) 2023-03-03T07:56:30.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.096+0000 7effb699c700 1 -- 172.21.15.98:0/1821906271 >> v2:172.21.15.100:3300/0 conn(0x7effb00cf570 msgr2=0x7effb015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:30.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.096+0000 7effb699c700 1 --2- 172.21.15.98:0/1821906271 >> v2:172.21.15.100:3300/0 conn(0x7effb00cf570 0x7effb015d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.096+0000 7effb699c700 1 -- 172.21.15.98:0/1821906271 >> v1:172.21.15.100:6789/0 conn(0x7effb01609b0 legacy=0x7effb0160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:30.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.096+0000 7effb699c700 1 -- 172.21.15.98:0/1821906271 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb00cdd40 con 0x7effb015db30 2023-03-03T07:56:30.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.096+0000 7effb699c700 1 --2- 172.21.15.98:0/1821906271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effb015db30 0x7effb00b4aa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7effb00cf0c0 tx=0x7effa800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ed186998072cc51 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:30.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.097+0000 7effb5199700 1 -- 172.21.15.98:0/1821906271 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa800e070 con 0x7effb015db30 2023-03-03T07:56:30.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.097+0000 7effb5199700 1 -- 172.21.15.98:0/1821906271 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa800b910 con 0x7effb015db30 2023-03-03T07:56:30.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.097+0000 7effb5199700 1 -- 172.21.15.98:0/1821906271 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa800bcc0 con 0x7effb015db30 2023-03-03T07:56:30.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.097+0000 7effb719d700 1 -- 172.21.15.98:0/1821906271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effb015db30 msgr2=0x7effb00b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:30.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.097+0000 7effb719d700 1 --2- 172.21.15.98:0/1821906271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effb015db30 0x7effb00b4aa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7effb00cf0c0 tx=0x7effa800b330 comp rx=0 tx=0).stop 2023-03-03T07:56:30.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.097+0000 7effb719d700 1 -- 172.21.15.98:0/1821906271 shutdown_connections 2023-03-03T07:56:30.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.097+0000 7effb719d700 1 --2- 172.21.15.98:0/1821906271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effb015db30 0x7effb00b4aa0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.097+0000 7effb719d700 1 --2- 172.21.15.98:0/1821906271 >> v2:172.21.15.100:3300/0 conn(0x7effb00cf570 0x7effb015d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.097+0000 7effb719d700 1 -- 172.21.15.98:0/1821906271 >> 172.21.15.98:0/1821906271 conn(0x7effb006b880 msgr2=0x7effb00cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.097+0000 7effb719d700 1 -- 172.21.15.98:0/1821906271 shutdown_connections 2023-03-03T07:56:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.097+0000 7effb719d700 1 -- 172.21.15.98:0/1821906271 wait complete. 2023-03-03T07:56:30.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.099+0000 7effb719d700 1 Processor -- start 2023-03-03T07:56:30.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.100+0000 7effb719d700 1 -- start start 2023-03-03T07:56:30.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb719d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7effb00cf570 0x7effb00cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:30.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb719d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7effb015db30 0x7effb01482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:30.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb719d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effb01609b0 0x7effb00d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:30.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb719d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb00b7430 con 0x7effb01609b0 2023-03-03T07:56:30.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb719d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb00b7710 con 0x7effb015db30 2023-03-03T07:56:30.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb719d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb00b79f0 con 0x7effb00cf570 2023-03-03T07:56:30.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb699c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effb01609b0 0x7effb00d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:30.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb699c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effb01609b0 0x7effb00d17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40652/0 (socket says 172.21.15.98:40652) 2023-03-03T07:56:30.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb699c700 1 -- 172.21.15.98:0/852327409 learned_addr learned my addr 172.21.15.98:0/852327409 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:30.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb619b700 1 --2- 172.21.15.98:0/852327409 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7effb00cf570 0x7effb00cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:30.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb599a700 1 --2- 172.21.15.98:0/852327409 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7effb015db30 0x7effb01482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb699c700 1 -- 172.21.15.98:0/852327409 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7effb00cf570 msgr2=0x7effb00cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb699c700 1 --2- 172.21.15.98:0/852327409 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7effb00cf570 0x7effb00cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb699c700 1 -- 172.21.15.98:0/852327409 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7effb015db30 msgr2=0x7effb01482e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb699c700 1 --2- 172.21.15.98:0/852327409 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7effb015db30 0x7effb01482e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb699c700 1 -- 172.21.15.98:0/852327409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa8009b30 con 0x7effb01609b0 2023-03-03T07:56:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.101+0000 7effb599a700 1 --2- 172.21.15.98:0/852327409 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7effb015db30 0x7effb01482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:30.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.102+0000 7effb699c700 1 --2- 172.21.15.98:0/852327409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effb01609b0 0x7effb00d17d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7effa800b370 tx=0x7effa8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:30.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.102+0000 7effa6ffd700 1 -- 172.21.15.98:0/852327409 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa8004930 con 0x7effb01609b0 2023-03-03T07:56:30.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.102+0000 7effa6ffd700 1 -- 172.21.15.98:0/852327409 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa80099b0 con 0x7effb01609b0 2023-03-03T07:56:30.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.102+0000 7effa6ffd700 1 -- 172.21.15.98:0/852327409 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa801c660 con 0x7effb01609b0 2023-03-03T07:56:30.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.102+0000 7effb719d700 1 -- 172.21.15.98:0/852327409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb015d0d0 con 0x7effb01609b0 2023-03-03T07:56:30.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.102+0000 7effb719d700 1 -- 172.21.15.98:0/852327409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb015cdf0 con 0x7effb01609b0 2023-03-03T07:56:30.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.103+0000 7effb719d700 1 -- 172.21.15.98:0/852327409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb015d5f0 con 0x7effb01609b0 2023-03-03T07:56:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.104+0000 7effa6ffd700 1 -- 172.21.15.98:0/852327409 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7effa8022030 con 0x7effb01609b0 2023-03-03T07:56:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.106+0000 7effa6ffd700 1 --2- 172.21.15.98:0/852327409 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7eff940682f0 0x7eff9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.106+0000 7effa6ffd700 1 -- 172.21.15.98:0/852327409 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7effa8087620 con 0x7effb01609b0 2023-03-03T07:56:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.106+0000 7effb619b700 1 --2- 172.21.15.98:0/852327409 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7eff940682f0 0x7eff9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.106+0000 7effb619b700 1 --2- 172.21.15.98:0/852327409 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7eff940682f0 0x7eff9406a7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7effac000f60 tx=0x7effac007ed0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:30.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.110+0000 7effa6ffd700 1 -- 172.21.15.98:0/852327409 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7effa8053d50 con 0x7effb01609b0 2023-03-03T07:56:30.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.302+0000 7effb719d700 1 -- 172.21.15.98:0/852327409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7effb0067d20 con 0x7effb01609b0 2023-03-03T07:56:30.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.303+0000 7effa6ffd700 1 -- 172.21.15.98:0/852327409 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7effa8057370 con 0x7effb01609b0 2023-03-03T07:56:30.306 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:56:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.305+0000 7effb719d700 1 -- 172.21.15.98:0/852327409 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7eff940682f0 msgr2=0x7eff9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.305+0000 7effb719d700 1 --2- 172.21.15.98:0/852327409 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7eff940682f0 0x7eff9406a7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7effac000f60 tx=0x7effac007ed0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.305+0000 7effb719d700 1 -- 172.21.15.98:0/852327409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effb01609b0 msgr2=0x7effb00d17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.305+0000 7effb719d700 1 --2- 172.21.15.98:0/852327409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effb01609b0 0x7effb00d17d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7effa800b370 tx=0x7effa8004ad0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.306+0000 7effb719d700 1 -- 172.21.15.98:0/852327409 shutdown_connections 2023-03-03T07:56:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.306+0000 7effb719d700 1 --2- 172.21.15.98:0/852327409 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7eff940682f0 0x7eff9406a7a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.306+0000 7effb719d700 1 --2- 172.21.15.98:0/852327409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effb01609b0 0x7effb00d17d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.306+0000 7effb719d700 1 --2- 172.21.15.98:0/852327409 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7effb00cf570 0x7effb00cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.306+0000 7effb719d700 1 --2- 172.21.15.98:0/852327409 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7effb015db30 0x7effb01482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.306+0000 7effb719d700 1 -- 172.21.15.98:0/852327409 >> 172.21.15.98:0/852327409 conn(0x7effb006b880 msgr2=0x7effb00731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.306+0000 7effb719d700 1 -- 172.21.15.98:0/852327409 shutdown_connections 2023-03-03T07:56:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.306+0000 7effb719d700 1 -- 172.21.15.98:0/852327409 wait complete. 2023-03-03T07:56:30.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:56:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.585+0000 7f5166f9d700 1 Processor -- start 2023-03-03T07:56:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.585+0000 7f5166f9d700 1 -- start start 2023-03-03T07:56:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.586+0000 7f5166f9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f51601609b0 0x7f5160160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.586+0000 7f5166f9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51600cf570 0x7f516015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.586+0000 7f5166f9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51601612c0 con 0x7f516015db30 2023-03-03T07:56:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.586+0000 7f5166f9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51600cebb0 con 0x7f51601609b0 2023-03-03T07:56:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.586+0000 7f5166f9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51600cd780 con 0x7f51600cf570 2023-03-03T07:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.587+0000 7f516679c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f516015db30 0x7f51600b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39814/0 (socket says 172.21.15.98:39814) 2023-03-03T07:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.587+0000 7f516679c700 1 -- 172.21.15.98:0/2738662636 learned_addr learned my addr 172.21.15.98:0/2738662636 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.587+0000 7f516579a700 1 --2- 172.21.15.98:0/2738662636 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51600cf570 0x7f516015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.587+0000 7f5165f9b700 1 --2- 172.21.15.98:0/2738662636 >> v2:172.21.15.100:3300/0 conn(0x7f51601609b0 0x7f5160160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.587+0000 7f5164f99700 1 -- 172.21.15.98:0/2738662636 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104012668 0 0) 0x7f51601612c0 con 0x7f516015db30 2023-03-03T07:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.587+0000 7f5164f99700 1 -- 172.21.15.98:0/2738662636 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f514c0035b0 con 0x7f516015db30 2023-03-03T07:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.587+0000 7f5164f99700 1 -- 172.21.15.98:0/2738662636 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1356225512 0 0) 0x7f514c0035b0 con 0x7f516015db30 2023-03-03T07:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.587+0000 7f516579a700 1 -- 172.21.15.98:0/2738662636 >> v2:172.21.15.100:3300/0 conn(0x7f51601609b0 msgr2=0x7f5160160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.587+0000 7f516579a700 1 --2- 172.21.15.98:0/2738662636 >> v2:172.21.15.100:3300/0 conn(0x7f51601609b0 0x7f5160160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.587+0000 7f516579a700 1 -- 172.21.15.98:0/2738662636 >> v1:172.21.15.98:6789/0 conn(0x7f516015db30 legacy=0x7f51600b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.588+0000 7f516579a700 1 -- 172.21.15.98:0/2738662636 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51600cda60 con 0x7f51600cf570 2023-03-03T07:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.588+0000 7f516579a700 1 --2- 172.21.15.98:0/2738662636 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51600cf570 0x7f516015d5f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5150009840 tx=0x7f5150009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20673868afb46c3e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.588+0000 7f5164f99700 1 -- 172.21.15.98:0/2738662636 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51580019f0 con 0x7f51600cf570 2023-03-03T07:56:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.588+0000 7f5164f99700 1 -- 172.21.15.98:0/2738662636 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5150004030 con 0x7f51600cf570 2023-03-03T07:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.588+0000 7f5164f99700 1 -- 172.21.15.98:0/2738662636 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51500043c0 con 0x7f51600cf570 2023-03-03T07:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.589+0000 7f5166f9d700 1 -- 172.21.15.98:0/2738662636 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51600cf570 msgr2=0x7f516015d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.589+0000 7f5166f9d700 1 --2- 172.21.15.98:0/2738662636 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51600cf570 0x7f516015d5f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5150009840 tx=0x7f5150009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.589+0000 7f5166f9d700 1 -- 172.21.15.98:0/2738662636 shutdown_connections 2023-03-03T07:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.589+0000 7f5166f9d700 1 --2- 172.21.15.98:0/2738662636 >> v2:172.21.15.100:3300/0 conn(0x7f51601609b0 0x7f5160160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.589+0000 7f5166f9d700 1 --2- 172.21.15.98:0/2738662636 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51600cf570 0x7f516015d5f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.589+0000 7f5166f9d700 1 -- 172.21.15.98:0/2738662636 >> 172.21.15.98:0/2738662636 conn(0x7f516006b880 msgr2=0x7f51600cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.589+0000 7f5166f9d700 1 -- 172.21.15.98:0/2738662636 shutdown_connections 2023-03-03T07:56:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.589+0000 7f5166f9d700 1 -- 172.21.15.98:0/2738662636 wait complete. 2023-03-03T07:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.591+0000 7f5166f9d700 1 Processor -- start 2023-03-03T07:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.591+0000 7f5166f9d700 1 -- start start 2023-03-03T07:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.592+0000 7f5166f9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51600cf570 0x7f51600b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.592+0000 7f5166f9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f516015db30 0x7f5160073830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.592+0000 7f5166f9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51601609b0 0x7f51601485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.592+0000 7f5166f9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51600ccf90 con 0x7f51601609b0 2023-03-03T07:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.592+0000 7f5166f9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5160148ae0 con 0x7f516015db30 2023-03-03T07:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.592+0000 7f5166f9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51600b4920 con 0x7f51600cf570 2023-03-03T07:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.593+0000 7f516679c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51601609b0 0x7f51601485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.593+0000 7f516579a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f516015db30 0x7f5160073830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.593+0000 7f516579a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f516015db30 0x7f5160073830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:33890/0 (socket says 172.21.15.98:33890) 2023-03-03T07:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.593+0000 7f516579a700 1 -- 172.21.15.98:0/4084998775 learned_addr learned my addr 172.21.15.98:0/4084998775 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.593+0000 7f5165f9b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51600cf570 0x7f51600b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.593+0000 7f516579a700 1 -- 172.21.15.98:0/4084998775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51600cf570 msgr2=0x7f51600b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.593+0000 7f516579a700 1 --2- 172.21.15.98:0/4084998775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51600cf570 0x7f51600b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.593+0000 7f516579a700 1 -- 172.21.15.98:0/4084998775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51601609b0 msgr2=0x7f51601485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.593+0000 7f516579a700 1 --2- 172.21.15.98:0/4084998775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51601609b0 0x7f51601485a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.593+0000 7f516579a700 1 -- 172.21.15.98:0/4084998775 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51500094a0 con 0x7f516015db30 2023-03-03T07:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.593+0000 7f5165f9b700 1 --2- 172.21.15.98:0/4084998775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51600cf570 0x7f51600b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.593+0000 7f516679c700 1 --2- 172.21.15.98:0/4084998775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51601609b0 0x7f51601485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.593+0000 7f516579a700 1 --2- 172.21.15.98:0/4084998775 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f516015db30 0x7f5160073830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f51500090b0 tx=0x7f5150004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.594+0000 7f51567fc700 1 -- 172.21.15.98:0/4084998775 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150003c70 con 0x7f516015db30 2023-03-03T07:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.594+0000 7f51567fc700 1 -- 172.21.15.98:0/4084998775 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515001f070 con 0x7f516015db30 2023-03-03T07:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.594+0000 7f5166f9d700 1 -- 172.21.15.98:0/4084998775 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f516015cdf0 con 0x7f516015db30 2023-03-03T07:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.594+0000 7f51567fc700 1 -- 172.21.15.98:0/4084998775 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150011630 con 0x7f516015db30 2023-03-03T07:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.594+0000 7f5166f9d700 1 -- 172.21.15.98:0/4084998775 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51600cdd40 con 0x7f516015db30 2023-03-03T07:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.595+0000 7f5166f9d700 1 -- 172.21.15.98:0/4084998775 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516015d0d0 con 0x7f516015db30 2023-03-03T07:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.597+0000 7f51567fc700 1 -- 172.21.15.98:0/4084998775 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5150017020 con 0x7f516015db30 2023-03-03T07:56:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.598+0000 7f51567fc700 1 --2- 172.21.15.98:0/4084998775 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f51440682f0 0x7f514406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.598+0000 7f51567fc700 1 -- 172.21.15.98:0/4084998775 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f5150021070 con 0x7f516015db30 2023-03-03T07:56:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.598+0000 7f5165f9b700 1 --2- 172.21.15.98:0/4084998775 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f51440682f0 0x7f514406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.599+0000 7f5165f9b700 1 --2- 172.21.15.98:0/4084998775 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f51440682f0 0x7f514406a7a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f515c009f30 tx=0x7f515c009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.601+0000 7f51567fc700 1 -- 172.21.15.98:0/4084998775 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5150053e60 con 0x7f516015db30 2023-03-03T07:56:30.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.794+0000 7f5166f9d700 1 -- 172.21.15.98:0/4084998775 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5160068290 con 0x7f516015db30 2023-03-03T07:56:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.794+0000 7f51567fc700 1 -- 172.21.15.98:0/4084998775 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5150057480 con 0x7f516015db30 2023-03-03T07:56:30.797 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:56:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.797+0000 7f5166f9d700 1 -- 172.21.15.98:0/4084998775 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f51440682f0 msgr2=0x7f514406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.797+0000 7f5166f9d700 1 --2- 172.21.15.98:0/4084998775 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f51440682f0 0x7f514406a7a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f515c009f30 tx=0x7f515c009210 comp rx=0 tx=0).stop 2023-03-03T07:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.797+0000 7f5166f9d700 1 -- 172.21.15.98:0/4084998775 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f516015db30 msgr2=0x7f5160073830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.797+0000 7f5166f9d700 1 --2- 172.21.15.98:0/4084998775 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f516015db30 0x7f5160073830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f51500090b0 tx=0x7f5150004750 comp rx=0 tx=0).stop 2023-03-03T07:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.797+0000 7f5166f9d700 1 -- 172.21.15.98:0/4084998775 shutdown_connections 2023-03-03T07:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.797+0000 7f5166f9d700 1 --2- 172.21.15.98:0/4084998775 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f51440682f0 0x7f514406a7a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.797+0000 7f5166f9d700 1 --2- 172.21.15.98:0/4084998775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51601609b0 0x7f51601485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.797+0000 7f5166f9d700 1 --2- 172.21.15.98:0/4084998775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51600cf570 0x7f51600b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.797+0000 7f5166f9d700 1 --2- 172.21.15.98:0/4084998775 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f516015db30 0x7f5160073830 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.797+0000 7f5166f9d700 1 -- 172.21.15.98:0/4084998775 >> 172.21.15.98:0/4084998775 conn(0x7f516006b880 msgr2=0x7f51600d1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.797+0000 7f5166f9d700 1 -- 172.21.15.98:0/4084998775 shutdown_connections 2023-03-03T07:56:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:30.797+0000 7f5166f9d700 1 -- 172.21.15.98:0/4084998775 wait complete. 2023-03-03T07:56:30.914 INFO:tasks.cephfs_test_runner:Starting test: test_binary_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) 2023-03-03T07:56:30.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade' 2023-03-03T07:56:31.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.158+0000 7fd44cbff700 1 Processor -- start 2023-03-03T07:56:31.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.158+0000 7fd44cbff700 1 -- start start 2023-03-03T07:56:31.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.159+0000 7fd44cbff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd4480b87c0 0x7fd4480cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:31.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.159+0000 7fd44cbff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4480d0ef0 con 0x7fd4480b8b90 2023-03-03T07:56:31.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.160+0000 7fd44cbff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4480d0600 con 0x7fd4480b5410 2023-03-03T07:56:31.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.160+0000 7fd44cbff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4480cd970 con 0x7fd4480b87c0 2023-03-03T07:56:31.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.160+0000 7fd445d9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd4480b8b90 0x7fd4480b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39822/0 (socket says 172.21.15.98:39822) 2023-03-03T07:56:31.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.160+0000 7fd445d9b700 1 -- 172.21.15.98:0/2593075535 learned_addr learned my addr 172.21.15.98:0/2593075535 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.160+0000 7fd44659c700 1 --2- 172.21.15.98:0/2593075535 >> v2:172.21.15.100:3300/0 conn(0x7fd4480b87c0 0x7fd4480cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.160+0000 7fd44559a700 1 -- 172.21.15.98:0/2593075535 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285594482 0 0) 0x7fd4480d0ef0 con 0x7fd4480b8b90 2023-03-03T07:56:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.160+0000 7fd44559a700 1 -- 172.21.15.98:0/2593075535 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4300035b0 con 0x7fd4480b8b90 2023-03-03T07:56:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.161+0000 7fd44659c700 1 -- 172.21.15.98:0/2593075535 >> v1:172.21.15.100:6789/0 conn(0x7fd4480b5410 legacy=0x7fd4480d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.161+0000 7fd44659c700 1 -- 172.21.15.98:0/2593075535 >> v1:172.21.15.98:6789/0 conn(0x7fd4480b8b90 legacy=0x7fd4480b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.161+0000 7fd44659c700 1 -- 172.21.15.98:0/2593075535 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4480cdc50 con 0x7fd4480b87c0 2023-03-03T07:56:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.161+0000 7fd44659c700 1 --2- 172.21.15.98:0/2593075535 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd4480b87c0 0x7fd4480cffe0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd438009840 tx=0x7fd438009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e89e76e454b86f4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:31.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.161+0000 7fd44559a700 1 -- 172.21.15.98:0/2593075535 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd438011830 con 0x7fd4480b87c0 2023-03-03T07:56:31.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.161+0000 7fd44559a700 1 -- 172.21.15.98:0/2593075535 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd438011990 con 0x7fd4480b87c0 2023-03-03T07:56:31.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.161+0000 7fd44559a700 1 -- 172.21.15.98:0/2593075535 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd438011d20 con 0x7fd4480b87c0 2023-03-03T07:56:31.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.162+0000 7fd44cbff700 1 -- 172.21.15.98:0/2593075535 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd4480b87c0 msgr2=0x7fd4480cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:31.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.162+0000 7fd44cbff700 1 --2- 172.21.15.98:0/2593075535 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd4480b87c0 0x7fd4480cffe0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd438009840 tx=0x7fd438009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:31.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.162+0000 7fd44cbff700 1 -- 172.21.15.98:0/2593075535 shutdown_connections 2023-03-03T07:56:31.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.162+0000 7fd44cbff700 1 --2- 172.21.15.98:0/2593075535 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd4480b87c0 0x7fd4480cffe0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:31.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.162+0000 7fd44cbff700 1 -- 172.21.15.98:0/2593075535 >> 172.21.15.98:0/2593075535 conn(0x7fd44806aee0 msgr2=0x7fd44806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:31.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.162+0000 7fd44cbff700 1 -- 172.21.15.98:0/2593075535 shutdown_connections 2023-03-03T07:56:31.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.162+0000 7fd44cbff700 1 -- 172.21.15.98:0/2593075535 wait complete. 2023-03-03T07:56:31.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.164+0000 7fd44cbff700 1 Processor -- start 2023-03-03T07:56:31.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.164+0000 7fd44cbff700 1 -- start start 2023-03-03T07:56:31.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.165+0000 7fd44cbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4480b5410 0x7fd448152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:31.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.165+0000 7fd44cbff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4480b87c0 0x7fd448156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:31.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.166+0000 7fd44cbff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd4480b8b90 0x7fd448156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:31.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.166+0000 7fd44cbff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4480cb410 con 0x7fd4480b87c0 2023-03-03T07:56:31.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.166+0000 7fd44cbff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4481594f0 con 0x7fd4480b8b90 2023-03-03T07:56:31.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.166+0000 7fd44cbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4480caae0 con 0x7fd4480b5410 2023-03-03T07:56:31.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.166+0000 7fd445d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4480b87c0 0x7fd448156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:31.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.166+0000 7fd445d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4480b87c0 0x7fd448156660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40662/0 (socket says 172.21.15.98:40662) 2023-03-03T07:56:31.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.166+0000 7fd445d9b700 1 -- 172.21.15.98:0/1740526707 learned_addr learned my addr 172.21.15.98:0/1740526707 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:31.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.166+0000 7fd446d9d700 1 --2- 172.21.15.98:0/1740526707 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd4480b8b90 0x7fd448156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:31.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.166+0000 7fd44659c700 1 --2- 172.21.15.98:0/1740526707 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4480b5410 0x7fd448152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:31.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.166+0000 7fd445d9b700 1 -- 172.21.15.98:0/1740526707 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4480b5410 msgr2=0x7fd448152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:31.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.166+0000 7fd445d9b700 1 --2- 172.21.15.98:0/1740526707 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4480b5410 0x7fd448152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:31.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.166+0000 7fd445d9b700 1 -- 172.21.15.98:0/1740526707 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd4480b8b90 msgr2=0x7fd448156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:31.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.166+0000 7fd445d9b700 1 --2- 172.21.15.98:0/1740526707 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd4480b8b90 0x7fd448156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:31.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.166+0000 7fd445d9b700 1 -- 172.21.15.98:0/1740526707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4380094a0 con 0x7fd4480b87c0 2023-03-03T07:56:31.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.167+0000 7fd445d9b700 1 --2- 172.21.15.98:0/1740526707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4480b87c0 0x7fd448156660 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd43c005910 tx=0x7fd43c00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:31.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.167+0000 7fd436ffd700 1 -- 172.21.15.98:0/1740526707 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c005d70 con 0x7fd4480b87c0 2023-03-03T07:56:31.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.167+0000 7fd436ffd700 1 -- 172.21.15.98:0/1740526707 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43c0070d0 con 0x7fd4480b87c0 2023-03-03T07:56:31.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.167+0000 7fd436ffd700 1 -- 172.21.15.98:0/1740526707 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c007460 con 0x7fd4480b87c0 2023-03-03T07:56:31.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.167+0000 7fd44cbff700 1 -- 172.21.15.98:0/1740526707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4480cdf30 con 0x7fd4480b87c0 2023-03-03T07:56:31.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.167+0000 7fd44cbff700 1 -- 172.21.15.98:0/1740526707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd448073a80 con 0x7fd4480b87c0 2023-03-03T07:56:31.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.168+0000 7fd44cbff700 1 -- 172.21.15.98:0/1740526707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4480ce0c0 con 0x7fd4480b87c0 2023-03-03T07:56:31.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.170+0000 7fd436ffd700 1 -- 172.21.15.98:0/1740526707 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd43c019030 con 0x7fd4480b87c0 2023-03-03T07:56:31.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.171+0000 7fd436ffd700 1 --2- 172.21.15.98:0/1740526707 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd4280682f0 0x7fd42806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:31.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.171+0000 7fd436ffd700 1 -- 172.21.15.98:0/1740526707 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fd43c085100 con 0x7fd4480b87c0 2023-03-03T07:56:31.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.171+0000 7fd44659c700 1 --2- 172.21.15.98:0/1740526707 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd4280682f0 0x7fd42806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:31.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.172+0000 7fd44659c700 1 --2- 172.21.15.98:0/1740526707 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd4280682f0 0x7fd42806a7a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd4380090b0 tx=0x7fd438008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:31.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.175+0000 7fd436ffd700 1 -- 172.21.15.98:0/1740526707 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd43c0517a0 con 0x7fd4480b87c0 2023-03-03T07:56:31.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:56:31.411 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:56:31.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.450+0000 7fd44cbff700 1 -- 172.21.15.98:0/1740526707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]} v 0) v1 -- 0x7fd4480729b0 con 0x7fd4480b87c0 2023-03-03T07:56:31.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.524+0000 7fd436ffd700 1 -- 172.21.15.98:0/1740526707 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]}]=0 v66) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd43c017020 con 0x7fd4480b87c0 2023-03-03T07:56:31.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.526+0000 7fd44cbff700 1 -- 172.21.15.98:0/1740526707 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd4280682f0 msgr2=0x7fd42806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:31.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.526+0000 7fd44cbff700 1 --2- 172.21.15.98:0/1740526707 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd4280682f0 0x7fd42806a7a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd4380090b0 tx=0x7fd438008030 comp rx=0 tx=0).stop 2023-03-03T07:56:31.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.526+0000 7fd44cbff700 1 -- 172.21.15.98:0/1740526707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4480b87c0 msgr2=0x7fd448156660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:31.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.526+0000 7fd44cbff700 1 --2- 172.21.15.98:0/1740526707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4480b87c0 0x7fd448156660 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd43c005910 tx=0x7fd43c00c9e0 comp rx=0 tx=0).stop 2023-03-03T07:56:31.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.526+0000 7fd44cbff700 1 -- 172.21.15.98:0/1740526707 shutdown_connections 2023-03-03T07:56:31.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.526+0000 7fd44cbff700 1 --2- 172.21.15.98:0/1740526707 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd4280682f0 0x7fd42806a7a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:31.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.526+0000 7fd44cbff700 1 --2- 172.21.15.98:0/1740526707 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd4480b8b90 0x7fd448156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:31.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.526+0000 7fd44cbff700 1 --2- 172.21.15.98:0/1740526707 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4480b5410 0x7fd448152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:31.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.526+0000 7fd44cbff700 1 --2- 172.21.15.98:0/1740526707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4480b87c0 0x7fd448156660 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:31.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.526+0000 7fd44cbff700 1 -- 172.21.15.98:0/1740526707 >> 172.21.15.98:0/1740526707 conn(0x7fd44806aee0 msgr2=0x7fd4480b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:31.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.527+0000 7fd44cbff700 1 -- 172.21.15.98:0/1740526707 shutdown_connections 2023-03-03T07:56:31.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.527+0000 7fd44cbff700 1 -- 172.21.15.98:0/1740526707 wait complete. 2023-03-03T07:56:31.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-03-03T07:56:31.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.844+0000 7f2f3a1ff700 1 Processor -- start 2023-03-03T07:56:31.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.844+0000 7f2f3a1ff700 1 -- start start 2023-03-03T07:56:31.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.845+0000 7f2f3a1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f340b26d0 0x7f2f340b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:31.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.845+0000 7f2f3a1ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f340b5140 0x7f2f340d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:31.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.845+0000 7f2f3a1ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f340cd7d0 con 0x7f2f340b8d10 2023-03-03T07:56:31.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.845+0000 7f2f3a1ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f340b55c0 con 0x7f2f340b5140 2023-03-03T07:56:31.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.845+0000 7f2f3a1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f340b90e0 con 0x7f2f340b26d0 2023-03-03T07:56:31.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.846+0000 7f2f337fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2f340b8d10 0x7f2f340cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39824/0 (socket says 172.21.15.98:39824) 2023-03-03T07:56:31.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.846+0000 7f2f337fe700 1 -- 172.21.15.98:0/1131444216 learned_addr learned my addr 172.21.15.98:0/1131444216 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:31.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.846+0000 7f2f33fff700 1 --2- 172.21.15.98:0/1131444216 >> v2:172.21.15.98:3300/0 conn(0x7f2f340b5140 0x7f2f340d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:31.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.846+0000 7f2f2bfff700 1 --2- 172.21.15.98:0/1131444216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f340b26d0 0x7f2f340b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:31.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.846+0000 7f2f32ffd700 1 -- 172.21.15.98:0/1131444216 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886168835 0 0) 0x7f2f340cd7d0 con 0x7f2f340b8d10 2023-03-03T07:56:31.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.846+0000 7f2f33fff700 1 -- 172.21.15.98:0/1131444216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f340b26d0 msgr2=0x7f2f340b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:31.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.846+0000 7f2f33fff700 1 --2- 172.21.15.98:0/1131444216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f340b26d0 0x7f2f340b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.846+0000 7f2f33fff700 1 -- 172.21.15.98:0/1131444216 >> v1:172.21.15.98:6789/0 conn(0x7f2f340b8d10 legacy=0x7f2f340cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.846+0000 7f2f33fff700 1 -- 172.21.15.98:0/1131444216 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f340cda20 con 0x7f2f340b5140 2023-03-03T07:56:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.847+0000 7f2f33fff700 1 --2- 172.21.15.98:0/1131444216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f340b5140 0x7f2f340d13a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c009d90 tx=0x7f2f2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a194a1c2c3bffca7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.847+0000 7f2f32ffd700 1 -- 172.21.15.98:0/1131444216 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c00e070 con 0x7f2f340b5140 2023-03-03T07:56:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.847+0000 7f2f32ffd700 1 -- 172.21.15.98:0/1131444216 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2c00bb20 con 0x7f2f340b5140 2023-03-03T07:56:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.847+0000 7f2f32ffd700 1 -- 172.21.15.98:0/1131444216 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c012430 con 0x7f2f340b5140 2023-03-03T07:56:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.848+0000 7f2f3a1ff700 1 -- 172.21.15.98:0/1131444216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f340b5140 msgr2=0x7f2f340d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.848+0000 7f2f3a1ff700 1 --2- 172.21.15.98:0/1131444216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f340b5140 0x7f2f340d13a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c009d90 tx=0x7f2f2c00b330 comp rx=0 tx=0).stop 2023-03-03T07:56:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.848+0000 7f2f3a1ff700 1 -- 172.21.15.98:0/1131444216 shutdown_connections 2023-03-03T07:56:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.848+0000 7f2f3a1ff700 1 --2- 172.21.15.98:0/1131444216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f340b5140 0x7f2f340d13a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.848+0000 7f2f3a1ff700 1 --2- 172.21.15.98:0/1131444216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f340b26d0 0x7f2f340b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:31.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.848+0000 7f2f3a1ff700 1 -- 172.21.15.98:0/1131444216 >> 172.21.15.98:0/1131444216 conn(0x7f2f34073e70 msgr2=0x7f2f3406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:31.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.848+0000 7f2f3a1ff700 1 -- 172.21.15.98:0/1131444216 shutdown_connections 2023-03-03T07:56:31.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.848+0000 7f2f3a1ff700 1 -- 172.21.15.98:0/1131444216 wait complete. 2023-03-03T07:56:31.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.850+0000 7f2f3a1ff700 1 Processor -- start 2023-03-03T07:56:31.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.850+0000 7f2f3a1ff700 1 -- start start 2023-03-03T07:56:31.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.851+0000 7f2f3a1ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f340b26d0 0x7f2f3415f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:31.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.851+0000 7f2f3a1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f340b5140 0x7f2f3415c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:31.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.851+0000 7f2f3a1ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f340b8d10 0x7f2f3415c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:31.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.851+0000 7f2f3a1ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f340cb2a0 con 0x7f2f340b8d10 2023-03-03T07:56:31.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.851+0000 7f2f3a1ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f341f29e0 con 0x7f2f340b26d0 2023-03-03T07:56:31.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.851+0000 7f2f3a1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f341f2cc0 con 0x7f2f340b5140 2023-03-03T07:56:31.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.851+0000 7f2f33fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f340b8d10 0x7f2f3415c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:31.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.851+0000 7f2f33fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f340b8d10 0x7f2f3415c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40686/0 (socket says 172.21.15.98:40686) 2023-03-03T07:56:31.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.851+0000 7f2f33fff700 1 -- 172.21.15.98:0/4005435229 learned_addr learned my addr 172.21.15.98:0/4005435229 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:31.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.852+0000 7f2f2bfff700 1 --2- 172.21.15.98:0/4005435229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f340b5140 0x7f2f3415c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:31.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.852+0000 7f2f337fe700 1 --2- 172.21.15.98:0/4005435229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f340b26d0 0x7f2f3415f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:31.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.852+0000 7f2f33fff700 1 -- 172.21.15.98:0/4005435229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f340b5140 msgr2=0x7f2f3415c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:31.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.852+0000 7f2f33fff700 1 --2- 172.21.15.98:0/4005435229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f340b5140 0x7f2f3415c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:31.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.852+0000 7f2f33fff700 1 -- 172.21.15.98:0/4005435229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f340b26d0 msgr2=0x7f2f3415f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:31.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.852+0000 7f2f33fff700 1 --2- 172.21.15.98:0/4005435229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f340b26d0 0x7f2f3415f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:31.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.852+0000 7f2f33fff700 1 -- 172.21.15.98:0/4005435229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2c009a70 con 0x7f2f340b8d10 2023-03-03T07:56:31.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.852+0000 7f2f2bfff700 1 --2- 172.21.15.98:0/4005435229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f340b5140 0x7f2f3415c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:31.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.852+0000 7f2f33fff700 1 --2- 172.21.15.98:0/4005435229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f340b8d10 0x7f2f3415c850 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2f3408c2c0 tx=0x7f2f2c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:31.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.852+0000 7f2f30ff9700 1 -- 172.21.15.98:0/4005435229 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c012430 con 0x7f2f340b8d10 2023-03-03T07:56:31.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.853+0000 7f2f30ff9700 1 -- 172.21.15.98:0/4005435229 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2c020070 con 0x7f2f340b8d10 2023-03-03T07:56:31.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.853+0000 7f2f3a1ff700 1 -- 172.21.15.98:0/4005435229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f340d0e80 con 0x7f2f340b8d10 2023-03-03T07:56:31.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.853+0000 7f2f30ff9700 1 -- 172.21.15.98:0/4005435229 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c004280 con 0x7f2f340b8d10 2023-03-03T07:56:31.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.853+0000 7f2f3a1ff700 1 -- 172.21.15.98:0/4005435229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f340cdd00 con 0x7f2f340b8d10 2023-03-03T07:56:31.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.854+0000 7f2f3a1ff700 1 -- 172.21.15.98:0/4005435229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f340cdfe0 con 0x7f2f340b8d10 2023-03-03T07:56:31.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.855+0000 7f2f30ff9700 1 -- 172.21.15.98:0/4005435229 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2f2c0043e0 con 0x7f2f340b8d10 2023-03-03T07:56:31.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.856+0000 7f2f30ff9700 1 --2- 172.21.15.98:0/4005435229 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2f140682f0 0x7f2f1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:31.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.856+0000 7f2f30ff9700 1 -- 172.21.15.98:0/4005435229 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f2f2c022070 con 0x7f2f340b8d10 2023-03-03T07:56:31.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.857+0000 7f2f337fe700 1 --2- 172.21.15.98:0/4005435229 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2f140682f0 0x7f2f1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:31.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.857+0000 7f2f337fe700 1 --2- 172.21.15.98:0/4005435229 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2f140682f0 0x7f2f1406a7a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2f24000f60 tx=0x7f2f24007c90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:31.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:31.860+0000 7f2f30ff9700 1 -- 172.21.15.98:0/4005435229 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f2c054940 con 0x7f2f340b8d10 2023-03-03T07:56:32.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.054+0000 7f2f3a1ff700 1 -- 172.21.15.98:0/4005435229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2f340cb3e0 con 0x7f2f340b8d10 2023-03-03T07:56:32.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.058+0000 7f2f30ff9700 1 -- 172.21.15.98:0/4005435229 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2f2c057f60 con 0x7f2f340b8d10 2023-03-03T07:56:32.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.061+0000 7f2f3a1ff700 1 -- 172.21.15.98:0/4005435229 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2f140682f0 msgr2=0x7f2f1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:32.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.061+0000 7f2f3a1ff700 1 --2- 172.21.15.98:0/4005435229 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2f140682f0 0x7f2f1406a7a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2f24000f60 tx=0x7f2f24007c90 comp rx=0 tx=0).stop 2023-03-03T07:56:32.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.061+0000 7f2f3a1ff700 1 -- 172.21.15.98:0/4005435229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f340b8d10 msgr2=0x7f2f3415c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:32.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.061+0000 7f2f3a1ff700 1 --2- 172.21.15.98:0/4005435229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f340b8d10 0x7f2f3415c850 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2f3408c2c0 tx=0x7f2f2c004ad0 comp rx=0 tx=0).stop 2023-03-03T07:56:32.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.061+0000 7f2f3a1ff700 1 -- 172.21.15.98:0/4005435229 shutdown_connections 2023-03-03T07:56:32.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.061+0000 7f2f3a1ff700 1 --2- 172.21.15.98:0/4005435229 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2f140682f0 0x7f2f1406a7a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:32.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.061+0000 7f2f3a1ff700 1 --2- 172.21.15.98:0/4005435229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f340b8d10 0x7f2f3415c850 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:32.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.061+0000 7f2f3a1ff700 1 --2- 172.21.15.98:0/4005435229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f340b26d0 0x7f2f3415f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:32.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.061+0000 7f2f3a1ff700 1 --2- 172.21.15.98:0/4005435229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f340b5140 0x7f2f3415c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:32.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.061+0000 7f2f3a1ff700 1 -- 172.21.15.98:0/4005435229 >> 172.21.15.98:0/4005435229 conn(0x7f2f34073e70 msgr2=0x7f2f340b69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:32.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.061+0000 7f2f3a1ff700 1 -- 172.21.15.98:0/4005435229 shutdown_connections 2023-03-03T07:56:32.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.061+0000 7f2f3a1ff700 1 -- 172.21.15.98:0/4005435229 wait complete. 2023-03-03T07:56:32.090 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-03-03T07:56:32.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T07:56:32.225 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T07:56:32.360 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-03-03T07:56:32.361 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:56:32.361 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:56:32.429 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:56:32.429 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:56:32.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:56:32.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.741+0000 7feb3699d700 1 Processor -- start 2023-03-03T07:56:32.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.742+0000 7feb3699d700 1 -- start start 2023-03-03T07:56:32.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.743+0000 7feb3699d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb300b51c0 0x7feb30160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:32.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.743+0000 7feb3699d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb300d1580 0x7feb300d19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:32.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.743+0000 7feb3699d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb301614c0 con 0x7feb30146360 2023-03-03T07:56:32.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.743+0000 7feb3699d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb300d10f0 con 0x7feb300d1580 2023-03-03T07:56:32.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.743+0000 7feb3699d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb300cdf20 con 0x7feb300b51c0 2023-03-03T07:56:32.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.743+0000 7feb3619c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb300d1580 0x7feb300d19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:32.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.743+0000 7feb3619c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb300d1580 0x7feb300d19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40688/0 (socket says 172.21.15.98:40688) 2023-03-03T07:56:32.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.743+0000 7feb3619c700 1 -- 172.21.15.98:0/2290570851 learned_addr learned my addr 172.21.15.98:0/2290570851 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:32.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.744+0000 7feb3519a700 1 --2- 172.21.15.98:0/2290570851 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb300b51c0 0x7feb30160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:32.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.744+0000 7feb3619c700 1 -- 172.21.15.98:0/2290570851 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb300b51c0 msgr2=0x7feb30160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:32.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.744+0000 7feb3619c700 1 --2- 172.21.15.98:0/2290570851 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb300b51c0 0x7feb30160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:32.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.744+0000 7feb3619c700 1 -- 172.21.15.98:0/2290570851 >> v1:172.21.15.100:6789/0 conn(0x7feb30146360 legacy=0x7feb300b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:32.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.744+0000 7feb3619c700 1 -- 172.21.15.98:0/2290570851 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb300cd680 con 0x7feb300d1580 2023-03-03T07:56:32.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.744+0000 7feb3619c700 1 --2- 172.21.15.98:0/2290570851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb300d1580 0x7feb300d19e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feb28009e50 tx=0x7feb2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c600547de758005d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:32.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.744+0000 7feb34999700 1 -- 172.21.15.98:0/2290570851 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2800e070 con 0x7feb300d1580 2023-03-03T07:56:32.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.745+0000 7feb34999700 1 -- 172.21.15.98:0/2290570851 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2800b9c0 con 0x7feb300d1580 2023-03-03T07:56:32.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.745+0000 7feb34999700 1 -- 172.21.15.98:0/2290570851 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2800bd70 con 0x7feb300d1580 2023-03-03T07:56:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.745+0000 7feb3699d700 1 -- 172.21.15.98:0/2290570851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb300d1580 msgr2=0x7feb300d19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.745+0000 7feb3699d700 1 --2- 172.21.15.98:0/2290570851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb300d1580 0x7feb300d19e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feb28009e50 tx=0x7feb2800b330 comp rx=0 tx=0).stop 2023-03-03T07:56:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.745+0000 7feb3699d700 1 -- 172.21.15.98:0/2290570851 shutdown_connections 2023-03-03T07:56:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.745+0000 7feb3699d700 1 --2- 172.21.15.98:0/2290570851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb300d1580 0x7feb300d19e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.745+0000 7feb3699d700 1 --2- 172.21.15.98:0/2290570851 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb300b51c0 0x7feb30160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.745+0000 7feb3699d700 1 -- 172.21.15.98:0/2290570851 >> 172.21.15.98:0/2290570851 conn(0x7feb3006b8e0 msgr2=0x7feb300cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.745+0000 7feb3699d700 1 -- 172.21.15.98:0/2290570851 shutdown_connections 2023-03-03T07:56:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.745+0000 7feb3699d700 1 -- 172.21.15.98:0/2290570851 wait complete. 2023-03-03T07:56:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.747+0000 7feb3699d700 1 Processor -- start 2023-03-03T07:56:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.748+0000 7feb3699d700 1 -- start start 2023-03-03T07:56:32.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.749+0000 7feb3699d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb300b51c0 0x7feb300c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:32.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.749+0000 7feb3699d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb300d1580 0x7feb300c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:32.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.749+0000 7feb3699d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb30146360 0x7feb300ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:32.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.749+0000 7feb3699d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb300c66d0 con 0x7feb300b51c0 2023-03-03T07:56:32.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.749+0000 7feb3699d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb300cd290 con 0x7feb300d1580 2023-03-03T07:56:32.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.749+0000 7feb3699d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb300c75c0 con 0x7feb30146360 2023-03-03T07:56:32.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.749+0000 7feb3599b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb300b51c0 0x7feb300c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:32.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.749+0000 7feb3599b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb300b51c0 0x7feb300c6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40702/0 (socket says 172.21.15.98:40702) 2023-03-03T07:56:32.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.749+0000 7feb3599b700 1 -- 172.21.15.98:0/2990592030 learned_addr learned my addr 172.21.15.98:0/2990592030 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:32.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.749+0000 7feb3519a700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb300d1580 0x7feb300c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:32.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.749+0000 7feb3619c700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb30146360 0x7feb300ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:32.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.749+0000 7feb3599b700 1 -- 172.21.15.98:0/2990592030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb30146360 msgr2=0x7feb300ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:32.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.749+0000 7feb3599b700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb30146360 0x7feb300ccd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:32.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.750+0000 7feb3599b700 1 -- 172.21.15.98:0/2990592030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb300d1580 msgr2=0x7feb300c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:32.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.750+0000 7feb3599b700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb300d1580 0x7feb300c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:32.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.750+0000 7feb3599b700 1 -- 172.21.15.98:0/2990592030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb28009b30 con 0x7feb300b51c0 2023-03-03T07:56:32.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.750+0000 7feb3619c700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb30146360 0x7feb300ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:32.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.750+0000 7feb3519a700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb300d1580 0x7feb300c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:32.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.750+0000 7feb3599b700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb300b51c0 0x7feb300c6fe0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00c510 tx=0x7feb2c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:32.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.750+0000 7feb267fc700 1 -- 172.21.15.98:0/2990592030 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c0052d0 con 0x7feb300b51c0 2023-03-03T07:56:32.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.750+0000 7feb267fc700 1 -- 172.21.15.98:0/2990592030 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c005430 con 0x7feb300b51c0 2023-03-03T07:56:32.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.750+0000 7feb3699d700 1 -- 172.21.15.98:0/2990592030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb300cdc40 con 0x7feb300b51c0 2023-03-03T07:56:32.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.750+0000 7feb267fc700 1 -- 172.21.15.98:0/2990592030 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c0038c0 con 0x7feb300b51c0 2023-03-03T07:56:32.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.750+0000 7feb3699d700 1 -- 172.21.15.98:0/2990592030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb300cd960 con 0x7feb300b51c0 2023-03-03T07:56:32.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.751+0000 7feb3699d700 1 -- 172.21.15.98:0/2990592030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb300d0d80 con 0x7feb300b51c0 2023-03-03T07:56:32.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.753+0000 7feb267fc700 1 -- 172.21.15.98:0/2990592030 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feb2c019030 con 0x7feb300b51c0 2023-03-03T07:56:32.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.755+0000 7feb267fc700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feb14068340 0x7feb1406a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:32.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.755+0000 7feb267fc700 1 -- 172.21.15.98:0/2990592030 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7feb2c084f30 con 0x7feb300b51c0 2023-03-03T07:56:32.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.755+0000 7feb3519a700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feb14068340 0x7feb1406a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:32.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.756+0000 7feb3519a700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feb14068340 0x7feb1406a7f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7feb300cc510 tx=0x7feb20008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:32.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.759+0000 7feb267fc700 1 -- 172.21.15.98:0/2990592030 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feb2c0516e0 con 0x7feb300b51c0 2023-03-03T07:56:32.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.954+0000 7feb3699d700 1 -- 172.21.15.98:0/2990592030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb301614c0 con 0x7feb300b51c0 2023-03-03T07:56:32.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.955+0000 7feb267fc700 1 -- 172.21.15.98:0/2990592030 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18003 (secure 0 0 0) 0x7feb301614c0 con 0x7feb300b51c0 2023-03-03T07:56:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:56:32.958 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:56:27.498205+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:56:32.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.957+0000 7feb3699d700 1 -- 172.21.15.98:0/2990592030 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feb14068340 msgr2=0x7feb1406a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:32.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.957+0000 7feb3699d700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feb14068340 0x7feb1406a7f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7feb300cc510 tx=0x7feb20008040 comp rx=0 tx=0).stop 2023-03-03T07:56:32.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.957+0000 7feb3699d700 1 -- 172.21.15.98:0/2990592030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb300b51c0 msgr2=0x7feb300c6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:32.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.957+0000 7feb3699d700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb300b51c0 0x7feb300c6fe0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00c510 tx=0x7feb2c00c8d0 comp rx=0 tx=0).stop 2023-03-03T07:56:32.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.958+0000 7feb3699d700 1 -- 172.21.15.98:0/2990592030 shutdown_connections 2023-03-03T07:56:32.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.958+0000 7feb3699d700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feb14068340 0x7feb1406a7f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:32.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.958+0000 7feb3699d700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb300b51c0 0x7feb300c6fe0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:32.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.958+0000 7feb3699d700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb30146360 0x7feb300ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:32.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.958+0000 7feb3699d700 1 --2- 172.21.15.98:0/2990592030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb300d1580 0x7feb300c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:32.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.958+0000 7feb3699d700 1 -- 172.21.15.98:0/2990592030 >> 172.21.15.98:0/2990592030 conn(0x7feb3006b8e0 msgr2=0x7feb300b21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:32.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.958+0000 7feb3699d700 1 -- 172.21.15.98:0/2990592030 shutdown_connections 2023-03-03T07:56:32.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:32.958+0000 7feb3699d700 1 -- 172.21.15.98:0/2990592030 wait complete. 2023-03-03T07:56:32.988 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:55:45.157666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:55:46.092091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:56:32.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:56:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.234+0000 7ff67db9d700 1 Processor -- start 2023-03-03T07:56:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.235+0000 7ff67db9d700 1 -- start start 2023-03-03T07:56:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.236+0000 7ff67db9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6780b5e70 0x7ff6780b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.236+0000 7ff67db9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67815ce00 con 0x7ff6780cf570 2023-03-03T07:56:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.236+0000 7ff67db9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6780d0bf0 con 0x7ff6780b8910 2023-03-03T07:56:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.236+0000 7ff67db9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6780b8ce0 con 0x7ff6780b5e70 2023-03-03T07:56:33.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.236+0000 7ff67d39c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff6780cf570 0x7ff6780d13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39832/0 (socket says 172.21.15.98:39832) 2023-03-03T07:56:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.236+0000 7ff67d39c700 1 -- 172.21.15.98:0/2381044406 learned_addr learned my addr 172.21.15.98:0/2381044406 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.237+0000 7ff677fff700 1 --2- 172.21.15.98:0/2381044406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6780b5e70 0x7ff6780b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.237+0000 7ff6777fe700 1 -- 172.21.15.98:0/2381044406 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609424093 0 0) 0x7ff670001490 con 0x7ff6780cf570 2023-03-03T07:56:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.237+0000 7ff6777fe700 1 -- 172.21.15.98:0/2381044406 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6680035b0 con 0x7ff6780cf570 2023-03-03T07:56:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.237+0000 7ff6777fe700 1 -- 172.21.15.98:0/2381044406 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36026600 0 0) 0x7ff6780d0bf0 con 0x7ff6780b8910 2023-03-03T07:56:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.237+0000 7ff6777fe700 1 -- 172.21.15.98:0/2381044406 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff670001490 con 0x7ff6780b8910 2023-03-03T07:56:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.237+0000 7ff677fff700 1 -- 172.21.15.98:0/2381044406 >> v1:172.21.15.100:6789/0 conn(0x7ff6780b8910 legacy=0x7ff6781604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.237+0000 7ff677fff700 1 -- 172.21.15.98:0/2381044406 >> v1:172.21.15.98:6789/0 conn(0x7ff6780cf570 legacy=0x7ff6780d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.237+0000 7ff677fff700 1 -- 172.21.15.98:0/2381044406 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff678160cf0 con 0x7ff6780b5e70 2023-03-03T07:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.238+0000 7ff677fff700 1 --2- 172.21.15.98:0/2381044406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6780b5e70 0x7ff6780b4aa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff664009870 tx=0x7ff664009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2de37d1cad011e7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.238+0000 7ff6777fe700 1 -- 172.21.15.98:0/2381044406 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664011830 con 0x7ff6780b5e70 2023-03-03T07:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.238+0000 7ff6777fe700 1 -- 172.21.15.98:0/2381044406 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff664011990 con 0x7ff6780b5e70 2023-03-03T07:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.238+0000 7ff6777fe700 1 -- 172.21.15.98:0/2381044406 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664011d20 con 0x7ff6780b5e70 2023-03-03T07:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.239+0000 7ff67db9d700 1 -- 172.21.15.98:0/2381044406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6780b5e70 msgr2=0x7ff6780b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.239+0000 7ff67db9d700 1 --2- 172.21.15.98:0/2381044406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6780b5e70 0x7ff6780b4aa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff664009870 tx=0x7ff664009b80 comp rx=0 tx=0).stop 2023-03-03T07:56:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.239+0000 7ff67db9d700 1 -- 172.21.15.98:0/2381044406 shutdown_connections 2023-03-03T07:56:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.239+0000 7ff67db9d700 1 --2- 172.21.15.98:0/2381044406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6780b5e70 0x7ff6780b4aa0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.239+0000 7ff67db9d700 1 -- 172.21.15.98:0/2381044406 >> 172.21.15.98:0/2381044406 conn(0x7ff67806b880 msgr2=0x7ff6780cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.239+0000 7ff67db9d700 1 -- 172.21.15.98:0/2381044406 shutdown_connections 2023-03-03T07:56:33.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.239+0000 7ff67db9d700 1 -- 172.21.15.98:0/2381044406 wait complete. 2023-03-03T07:56:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.241+0000 7ff67db9d700 1 Processor -- start 2023-03-03T07:56:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.241+0000 7ff67db9d700 1 -- start start 2023-03-03T07:56:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.242+0000 7ff67db9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6780b5e70 0x7ff6781546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.242+0000 7ff67db9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6780b8910 0x7ff67815e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.242+0000 7ff67db9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6780cf570 0x7ff67815e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.242+0000 7ff67db9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6781f32b0 con 0x7ff6780cf570 2023-03-03T07:56:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.242+0000 7ff67db9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6780b3fc0 con 0x7ff6780b5e70 2023-03-03T07:56:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.242+0000 7ff67db9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff678154c30 con 0x7ff6780b8910 2023-03-03T07:56:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.243+0000 7ff67d39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6780cf570 0x7ff67815e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.243+0000 7ff67d39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6780cf570 0x7ff67815e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40708/0 (socket says 172.21.15.98:40708) 2023-03-03T07:56:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.243+0000 7ff67d39c700 1 -- 172.21.15.98:0/621686952 learned_addr learned my addr 172.21.15.98:0/621686952 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.243+0000 7ff677fff700 1 --2- 172.21.15.98:0/621686952 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6780b8910 0x7ff67815e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.243+0000 7ff67cb9b700 1 --2- 172.21.15.98:0/621686952 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6780b5e70 0x7ff6781546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.243+0000 7ff67d39c700 1 -- 172.21.15.98:0/621686952 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6780b8910 msgr2=0x7ff67815e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.243+0000 7ff67d39c700 1 --2- 172.21.15.98:0/621686952 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6780b8910 0x7ff67815e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.243+0000 7ff67d39c700 1 -- 172.21.15.98:0/621686952 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6780b5e70 msgr2=0x7ff6781546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.243+0000 7ff67d39c700 1 --2- 172.21.15.98:0/621686952 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6780b5e70 0x7ff6781546f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.243+0000 7ff67d39c700 1 -- 172.21.15.98:0/621686952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6640094a0 con 0x7ff6780cf570 2023-03-03T07:56:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.243+0000 7ff67cb9b700 1 --2- 172.21.15.98:0/621686952 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6780b5e70 0x7ff6781546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.243+0000 7ff67d39c700 1 --2- 172.21.15.98:0/621686952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6780cf570 0x7ff67815e760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff670001f50 tx=0x7ff67000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.244+0000 7ff6757fa700 1 -- 172.21.15.98:0/621686952 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67000ac20 con 0x7ff6780cf570 2023-03-03T07:56:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.244+0000 7ff6757fa700 1 -- 172.21.15.98:0/621686952 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67000ad80 con 0x7ff6780cf570 2023-03-03T07:56:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.244+0000 7ff67db9d700 1 -- 172.21.15.98:0/621686952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6781612b0 con 0x7ff6780cf570 2023-03-03T07:56:33.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.244+0000 7ff6757fa700 1 -- 172.21.15.98:0/621686952 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff670012600 con 0x7ff6780cf570 2023-03-03T07:56:33.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.244+0000 7ff67db9d700 1 -- 172.21.15.98:0/621686952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff678160fd0 con 0x7ff6780cf570 2023-03-03T07:56:33.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.245+0000 7ff67db9d700 1 -- 172.21.15.98:0/621686952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6780d0eb0 con 0x7ff6780cf570 2023-03-03T07:56:33.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.246+0000 7ff6757fa700 1 -- 172.21.15.98:0/621686952 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff670019030 con 0x7ff6780cf570 2023-03-03T07:56:33.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.248+0000 7ff6757fa700 1 --2- 172.21.15.98:0/621686952 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff65c0682f0 0x7ff65c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:33.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.248+0000 7ff6757fa700 1 -- 172.21.15.98:0/621686952 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7ff670085830 con 0x7ff6780cf570 2023-03-03T07:56:33.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.249+0000 7ff67cb9b700 1 --2- 172.21.15.98:0/621686952 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff65c0682f0 0x7ff65c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:33.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.249+0000 7ff67cb9b700 1 --2- 172.21.15.98:0/621686952 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff65c0682f0 0x7ff65c06a7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0011e0 tx=0x7ff66c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.252+0000 7ff6757fa700 1 -- 172.21.15.98:0/621686952 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff67004b5e0 con 0x7ff6780cf570 2023-03-03T07:56:33.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.445+0000 7ff67db9d700 1 -- 172.21.15.98:0/621686952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff67815ce00 con 0x7ff6780cf570 2023-03-03T07:56:33.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.445+0000 7ff6757fa700 1 -- 172.21.15.98:0/621686952 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff67815ce00 con 0x7ff6780cf570 2023-03-03T07:56:33.448 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:56:33.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.447+0000 7ff67db9d700 1 -- 172.21.15.98:0/621686952 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff65c0682f0 msgr2=0x7ff65c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:33.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.447+0000 7ff67db9d700 1 --2- 172.21.15.98:0/621686952 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff65c0682f0 0x7ff65c06a7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0011e0 tx=0x7ff66c009040 comp rx=0 tx=0).stop 2023-03-03T07:56:33.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.447+0000 7ff67db9d700 1 -- 172.21.15.98:0/621686952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6780cf570 msgr2=0x7ff67815e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:33.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.447+0000 7ff67db9d700 1 --2- 172.21.15.98:0/621686952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6780cf570 0x7ff67815e760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff670001f50 tx=0x7ff67000a060 comp rx=0 tx=0).stop 2023-03-03T07:56:33.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.448+0000 7ff67db9d700 1 -- 172.21.15.98:0/621686952 shutdown_connections 2023-03-03T07:56:33.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.448+0000 7ff67db9d700 1 --2- 172.21.15.98:0/621686952 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff65c0682f0 0x7ff65c06a7a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:33.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.448+0000 7ff67db9d700 1 --2- 172.21.15.98:0/621686952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6780cf570 0x7ff67815e760 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:33.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.448+0000 7ff67db9d700 1 --2- 172.21.15.98:0/621686952 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6780b5e70 0x7ff6781546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:33.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.448+0000 7ff67db9d700 1 --2- 172.21.15.98:0/621686952 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6780b8910 0x7ff67815e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:33.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.448+0000 7ff67db9d700 1 -- 172.21.15.98:0/621686952 >> 172.21.15.98:0/621686952 conn(0x7ff67806b880 msgr2=0x7ff6780731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:33.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.448+0000 7ff67db9d700 1 -- 172.21.15.98:0/621686952 shutdown_connections 2023-03-03T07:56:33.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.448+0000 7ff67db9d700 1 -- 172.21.15.98:0/621686952 wait complete. 2023-03-03T07:56:33.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:56:33.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.722+0000 7fe3997ff700 1 Processor -- start 2023-03-03T07:56:33.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.722+0000 7fe3997ff700 1 -- start start 2023-03-03T07:56:33.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.723+0000 7fe3997ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe3940b8d10 0x7fe3940cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:33.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.723+0000 7fe3997ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3940cd7d0 con 0x7fe3940b5310 2023-03-03T07:56:33.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.723+0000 7fe3997ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3940b90e0 con 0x7fe3940b26d0 2023-03-03T07:56:33.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.723+0000 7fe3997ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3940cda20 con 0x7fe3940b8d10 2023-03-03T07:56:33.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.724+0000 7fe3937fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe3940b5310 0x7fe3940d13a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39836/0 (socket says 172.21.15.98:39836) 2023-03-03T07:56:33.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.724+0000 7fe3937fe700 1 -- 172.21.15.98:0/4219916070 learned_addr learned my addr 172.21.15.98:0/4219916070 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:33.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.724+0000 7fe392ffd700 1 --2- 172.21.15.98:0/4219916070 >> v2:172.21.15.100:3300/0 conn(0x7fe3940b8d10 0x7fe3940cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:33.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.724+0000 7fe391ffb700 1 -- 172.21.15.98:0/4219916070 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534685707 0 0) 0x7fe3940cd7d0 con 0x7fe3940b5310 2023-03-03T07:56:33.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.724+0000 7fe391ffb700 1 -- 172.21.15.98:0/4219916070 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe37c0035b0 con 0x7fe3940b5310 2023-03-03T07:56:33.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.724+0000 7fe391ffb700 1 -- 172.21.15.98:0/4219916070 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834187361 0 0) 0x7fe3940b90e0 con 0x7fe3940b26d0 2023-03-03T07:56:33.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.725+0000 7fe391ffb700 1 -- 172.21.15.98:0/4219916070 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3940cd7d0 con 0x7fe3940b26d0 2023-03-03T07:56:33.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.725+0000 7fe392ffd700 1 -- 172.21.15.98:0/4219916070 >> v1:172.21.15.100:6789/0 conn(0x7fe3940b26d0 legacy=0x7fe3940b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:33.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.725+0000 7fe392ffd700 1 -- 172.21.15.98:0/4219916070 >> v1:172.21.15.98:6789/0 conn(0x7fe3940b5310 legacy=0x7fe3940d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:33.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.725+0000 7fe392ffd700 1 -- 172.21.15.98:0/4219916070 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3940cdd00 con 0x7fe3940b8d10 2023-03-03T07:56:33.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.725+0000 7fe392ffd700 1 --2- 172.21.15.98:0/4219916070 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3940b8d10 0x7fe3940cffc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe384009840 tx=0x7fe384009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71c34aefd1d1088 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:33.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.725+0000 7fe391ffb700 1 -- 172.21.15.98:0/4219916070 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe384011830 con 0x7fe3940b8d10 2023-03-03T07:56:33.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.725+0000 7fe391ffb700 1 -- 172.21.15.98:0/4219916070 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe384011990 con 0x7fe3940b8d10 2023-03-03T07:56:33.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.725+0000 7fe391ffb700 1 -- 172.21.15.98:0/4219916070 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe384011d20 con 0x7fe3940b8d10 2023-03-03T07:56:33.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.726+0000 7fe3997ff700 1 -- 172.21.15.98:0/4219916070 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3940b8d10 msgr2=0x7fe3940cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:33.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.726+0000 7fe3997ff700 1 --2- 172.21.15.98:0/4219916070 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3940b8d10 0x7fe3940cffc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe384009840 tx=0x7fe384009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:33.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.726+0000 7fe3997ff700 1 -- 172.21.15.98:0/4219916070 shutdown_connections 2023-03-03T07:56:33.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.726+0000 7fe3997ff700 1 --2- 172.21.15.98:0/4219916070 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3940b8d10 0x7fe3940cffc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:33.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.726+0000 7fe3997ff700 1 -- 172.21.15.98:0/4219916070 >> 172.21.15.98:0/4219916070 conn(0x7fe394073e70 msgr2=0x7fe39406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:33.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.726+0000 7fe3997ff700 1 -- 172.21.15.98:0/4219916070 shutdown_connections 2023-03-03T07:56:33.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.726+0000 7fe3997ff700 1 -- 172.21.15.98:0/4219916070 wait complete. 2023-03-03T07:56:33.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.728+0000 7fe3997ff700 1 Processor -- start 2023-03-03T07:56:33.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.728+0000 7fe3997ff700 1 -- start start 2023-03-03T07:56:33.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.729+0000 7fe3997ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3940b26d0 0x7fe3941524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:33.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe3997ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3940b5310 0x7fe394154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:33.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe3997ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3940b8d10 0x7fe394156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:33.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe3997ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe394152a10 con 0x7fe3940b26d0 2023-03-03T07:56:33.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe3997ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe394154b90 con 0x7fe3940b5310 2023-03-03T07:56:33.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe3997ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe394154e70 con 0x7fe3940b8d10 2023-03-03T07:56:33.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe392ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3940b26d0 0x7fe3941524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:33.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe392ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3940b26d0 0x7fe3941524d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40722/0 (socket says 172.21.15.98:40722) 2023-03-03T07:56:33.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe392ffd700 1 -- 172.21.15.98:0/1856882937 learned_addr learned my addr 172.21.15.98:0/1856882937 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:33.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe3927fc700 1 --2- 172.21.15.98:0/1856882937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3940b5310 0x7fe394154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:33.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe3937fe700 1 --2- 172.21.15.98:0/1856882937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3940b8d10 0x7fe394156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:33.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe392ffd700 1 -- 172.21.15.98:0/1856882937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3940b8d10 msgr2=0x7fe394156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe392ffd700 1 --2- 172.21.15.98:0/1856882937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3940b8d10 0x7fe394156670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe392ffd700 1 -- 172.21.15.98:0/1856882937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3940b5310 msgr2=0x7fe394154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe392ffd700 1 --2- 172.21.15.98:0/1856882937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3940b5310 0x7fe394154140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe392ffd700 1 -- 172.21.15.98:0/1856882937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3840094a0 con 0x7fe3940b26d0 2023-03-03T07:56:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.730+0000 7fe3927fc700 1 --2- 172.21.15.98:0/1856882937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3940b5310 0x7fe394154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:33.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.731+0000 7fe392ffd700 1 --2- 172.21.15.98:0/1856882937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3940b26d0 0x7fe3941524d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe3840090b0 tx=0x7fe384004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:33.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.731+0000 7fe37b7fe700 1 -- 172.21.15.98:0/1856882937 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38401a860 con 0x7fe3940b26d0 2023-03-03T07:56:33.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.731+0000 7fe37b7fe700 1 -- 172.21.15.98:0/1856882937 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38401a9c0 con 0x7fe3940b26d0 2023-03-03T07:56:33.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.731+0000 7fe3997ff700 1 -- 172.21.15.98:0/1856882937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3940cdfe0 con 0x7fe3940b26d0 2023-03-03T07:56:33.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.731+0000 7fe37b7fe700 1 -- 172.21.15.98:0/1856882937 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38401ad30 con 0x7fe3940b26d0 2023-03-03T07:56:33.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.731+0000 7fe3997ff700 1 -- 172.21.15.98:0/1856882937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3940d0e80 con 0x7fe3940b26d0 2023-03-03T07:56:33.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.732+0000 7fe3997ff700 1 -- 172.21.15.98:0/1856882937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe39404e6e0 con 0x7fe3940b26d0 2023-03-03T07:56:33.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.733+0000 7fe37b7fe700 1 -- 172.21.15.98:0/1856882937 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe38402b030 con 0x7fe3940b26d0 2023-03-03T07:56:33.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.735+0000 7fe37b7fe700 1 --2- 172.21.15.98:0/1856882937 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3800682f0 0x7fe38006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:33.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.735+0000 7fe37b7fe700 1 -- 172.21.15.98:0/1856882937 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fe384086ff0 con 0x7fe3940b26d0 2023-03-03T07:56:33.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.735+0000 7fe3927fc700 1 --2- 172.21.15.98:0/1856882937 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3800682f0 0x7fe38006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:33.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.736+0000 7fe3927fc700 1 --2- 172.21.15.98:0/1856882937 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3800682f0 0x7fe38006a7a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe3940b4020 tx=0x7fe388009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:33.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.739+0000 7fe37b7fe700 1 -- 172.21.15.98:0/1856882937 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe3840b85e0 con 0x7fe3940b26d0 2023-03-03T07:56:33.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.930+0000 7fe3997ff700 1 -- 172.21.15.98:0/1856882937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe3940d0480 con 0x7fe3940b26d0 2023-03-03T07:56:33.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.930+0000 7fe37b7fe700 1 -- 172.21.15.98:0/1856882937 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe384029020 con 0x7fe3940b26d0 2023-03-03T07:56:33.933 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:56:33.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.933+0000 7fe3997ff700 1 -- 172.21.15.98:0/1856882937 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3800682f0 msgr2=0x7fe38006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:33.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.933+0000 7fe3997ff700 1 --2- 172.21.15.98:0/1856882937 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3800682f0 0x7fe38006a7a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe3940b4020 tx=0x7fe388009040 comp rx=0 tx=0).stop 2023-03-03T07:56:33.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.933+0000 7fe3997ff700 1 -- 172.21.15.98:0/1856882937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3940b26d0 msgr2=0x7fe3941524d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.933+0000 7fe3997ff700 1 --2- 172.21.15.98:0/1856882937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3940b26d0 0x7fe3941524d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe3840090b0 tx=0x7fe384004750 comp rx=0 tx=0).stop 2023-03-03T07:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.933+0000 7fe3997ff700 1 -- 172.21.15.98:0/1856882937 shutdown_connections 2023-03-03T07:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.933+0000 7fe3997ff700 1 --2- 172.21.15.98:0/1856882937 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3800682f0 0x7fe38006a7a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.933+0000 7fe3997ff700 1 --2- 172.21.15.98:0/1856882937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3940b26d0 0x7fe3941524d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.933+0000 7fe3997ff700 1 --2- 172.21.15.98:0/1856882937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3940b8d10 0x7fe394156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.933+0000 7fe3997ff700 1 --2- 172.21.15.98:0/1856882937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3940b5310 0x7fe394154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.933+0000 7fe3997ff700 1 -- 172.21.15.98:0/1856882937 >> 172.21.15.98:0/1856882937 conn(0x7fe394073e70 msgr2=0x7fe3940b8390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:33.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.933+0000 7fe3997ff700 1 -- 172.21.15.98:0/1856882937 shutdown_connections 2023-03-03T07:56:33.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:33.933+0000 7fe3997ff700 1 -- 172.21.15.98:0/1856882937 wait complete. 2023-03-03T07:56:33.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:56:34.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.207+0000 7f2ab2dff700 1 Processor -- start 2023-03-03T07:56:34.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.207+0000 7f2ab2dff700 1 -- start start 2023-03-03T07:56:34.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.209+0000 7f2ab2dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2aac06b4b0 0x7f2aac0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:34.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.209+0000 7f2ab2dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aac0cf980 con 0x7f2aac15ca70 2023-03-03T07:56:34.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.209+0000 7f2ab2dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aac0d1800 con 0x7f2aac0d0270 2023-03-03T07:56:34.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.209+0000 7f2ab2dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0cd5b0 con 0x7f2aac06b4b0 2023-03-03T07:56:34.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.209+0000 7f2ab0b9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2aac15ca70 0x7f2aac0cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39838/0 (socket says 172.21.15.98:39838) 2023-03-03T07:56:34.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.209+0000 7f2ab0b9b700 1 -- 172.21.15.98:0/2828399310 learned_addr learned my addr 172.21.15.98:0/2828399310 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:34.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.209+0000 7f2ab139c700 1 --2- 172.21.15.98:0/2828399310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2aac06b4b0 0x7f2aac0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:34.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.209+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193827918 0 0) 0x7f2aac0cf980 con 0x7f2aac15ca70 2023-03-03T07:56:34.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a9c0035b0 con 0x7f2aac15ca70 2023-03-03T07:56:34.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751470267 0 0) 0x7f2aac0d1800 con 0x7f2aac0d0270 2023-03-03T07:56:34.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aac0cf980 con 0x7f2aac0d0270 2023-03-03T07:56:34.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3486587587 0 0) 0x7f2a9c0035b0 con 0x7f2aac15ca70 2023-03-03T07:56:34.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2aac06b4b0 msgr2=0x7f2aac0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:34.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 --2- 172.21.15.98:0/2828399310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2aac06b4b0 0x7f2aac0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 >> v1:172.21.15.100:6789/0 conn(0x7f2aac0d0270 legacy=0x7f2aac0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:34.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac0cd900 con 0x7f2aac15ca70 2023-03-03T07:56:34.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f2aa0001670 con 0x7f2aac15ca70 2023-03-03T07:56:34.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 >> v1:172.21.15.98:6789/0 conn(0x7f2aac15ca70 legacy=0x7f2aac0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:34.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 --2- 172.21.15.98:0/2828399310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c003bc0 0x7f2a9c003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:34.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 --2- 172.21.15.98:0/2828399310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a9c004850 0x7f2a9c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:34.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 --2- 172.21.15.98:0/2828399310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2aac06b4b0 0x7f2a9c005260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:34.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0cd5b0 con 0x7f2a9c003bc0 2023-03-03T07:56:34.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c00bbd0 con 0x7f2a9c004850 2023-03-03T07:56:34.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.210+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c00bd90 con 0x7f2aac06b4b0 2023-03-03T07:56:34.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.211+0000 7f2ab0b9b700 1 --2- 172.21.15.98:0/2828399310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c003bc0 0x7f2a9c003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:34.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.211+0000 7f2aabfff700 1 --2- 172.21.15.98:0/2828399310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a9c004850 0x7f2a9c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:34.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.211+0000 7f2ab139c700 1 --2- 172.21.15.98:0/2828399310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2aac06b4b0 0x7f2a9c005260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.211+0000 7f2ab0b9b700 1 -- 172.21.15.98:0/2828399310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2aac06b4b0 msgr2=0x7f2a9c005260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.211+0000 7f2ab0b9b700 1 --2- 172.21.15.98:0/2828399310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2aac06b4b0 0x7f2a9c005260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.211+0000 7f2ab0b9b700 1 -- 172.21.15.98:0/2828399310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a9c004850 msgr2=0x7f2a9c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.211+0000 7f2ab0b9b700 1 --2- 172.21.15.98:0/2828399310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a9c004850 0x7f2a9c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.211+0000 7f2ab0b9b700 1 -- 172.21.15.98:0/2828399310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c0035b0 con 0x7f2a9c003bc0 2023-03-03T07:56:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.212+0000 7f2ab0b9b700 1 --2- 172.21.15.98:0/2828399310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c003bc0 0x7f2a9c003fb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000c160 tx=0x7f2a9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b328eef1c1ae8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.212+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa000cca0 con 0x7f2a9c003bc0 2023-03-03T07:56:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.212+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa000ce20 con 0x7f2a9c003bc0 2023-03-03T07:56:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.212+0000 7f2aab7fe700 1 -- 172.21.15.98:0/2828399310 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0005ba0 con 0x7f2a9c003bc0 2023-03-03T07:56:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.212+0000 7f2ab2dff700 1 -- 172.21.15.98:0/2828399310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c003bc0 msgr2=0x7f2a9c003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.212+0000 7f2ab2dff700 1 --2- 172.21.15.98:0/2828399310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c003bc0 0x7f2a9c003fb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000c160 tx=0x7f2a9c005880 comp rx=0 tx=0).stop 2023-03-03T07:56:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.212+0000 7f2ab139c700 1 -- 172.21.15.98:0/2828399310 reap_dead start 2023-03-03T07:56:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.213+0000 7f2ab2dff700 1 -- 172.21.15.98:0/2828399310 shutdown_connections 2023-03-03T07:56:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.213+0000 7f2ab2dff700 1 -- 172.21.15.98:0/2828399310 >> 172.21.15.98:0/2828399310 conn(0x7f2aac073e70 msgr2=0x7f2aac06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.213+0000 7f2ab2dff700 1 -- 172.21.15.98:0/2828399310 shutdown_connections 2023-03-03T07:56:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.213+0000 7f2ab2dff700 1 -- 172.21.15.98:0/2828399310 wait complete. 2023-03-03T07:56:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.215+0000 7f2ab2dff700 1 Processor -- start 2023-03-03T07:56:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.215+0000 7f2ab2dff700 1 -- start start 2023-03-03T07:56:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.216+0000 7f2ab2dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a9c003bc0 0x7f2aac0cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.216+0000 7f2ab2dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2aac0b3c30 0x7f2aac15aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.216+0000 7f2ab2dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2aac0b40a0 0x7f2aac15af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.216+0000 7f2ab2dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac149740 con 0x7f2aac0b40a0 2023-03-03T07:56:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.216+0000 7f2ab2dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0b4e70 con 0x7f2aac0b3c30 2023-03-03T07:56:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.216+0000 7f2ab2dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0b45d0 con 0x7f2a9c003bc0 2023-03-03T07:56:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.216+0000 7f2ab139c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2aac0b40a0 0x7f2aac15af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.216+0000 7f2aabfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2aac0b3c30 0x7f2aac15aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.217+0000 7f2ab139c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2aac0b40a0 0x7f2aac15af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40742/0 (socket says 172.21.15.98:40742) 2023-03-03T07:56:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.217+0000 7f2ab139c700 1 -- 172.21.15.98:0/1334985648 learned_addr learned my addr 172.21.15.98:0/1334985648 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.217+0000 7f2ab0b9b700 1 --2- 172.21.15.98:0/1334985648 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a9c003bc0 0x7f2aac0cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.217+0000 7f2ab139c700 1 -- 172.21.15.98:0/1334985648 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a9c003bc0 msgr2=0x7f2aac0cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.217+0000 7f2ab139c700 1 --2- 172.21.15.98:0/1334985648 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a9c003bc0 0x7f2aac0cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.217+0000 7f2ab139c700 1 -- 172.21.15.98:0/1334985648 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2aac0b3c30 msgr2=0x7f2aac15aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.217+0000 7f2ab139c700 1 --2- 172.21.15.98:0/1334985648 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2aac0b3c30 0x7f2aac15aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.217+0000 7f2ab139c700 1 -- 172.21.15.98:0/1334985648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac0d1aa0 con 0x7f2aac0b40a0 2023-03-03T07:56:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.217+0000 7f2ab139c700 1 --2- 172.21.15.98:0/1334985648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2aac0b40a0 0x7f2aac15af40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0b1ff0 tx=0x7f2aa40114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.217+0000 7f2aa97fa700 1 -- 172.21.15.98:0/1334985648 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa400e070 con 0x7f2aac0b40a0 2023-03-03T07:56:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.218+0000 7f2aa97fa700 1 -- 172.21.15.98:0/1334985648 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa4003cf0 con 0x7f2aac0b40a0 2023-03-03T07:56:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.218+0000 7f2aa97fa700 1 -- 172.21.15.98:0/1334985648 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4003ea0 con 0x7f2aac0b40a0 2023-03-03T07:56:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.218+0000 7f2ab2dff700 1 -- 172.21.15.98:0/1334985648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aac0cdbe0 con 0x7f2aac0b40a0 2023-03-03T07:56:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.218+0000 7f2ab2dff700 1 -- 172.21.15.98:0/1334985648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aac15ce40 con 0x7f2aac0b40a0 2023-03-03T07:56:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.219+0000 7f2ab2dff700 1 -- 172.21.15.98:0/1334985648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aac1494a0 con 0x7f2aac0b40a0 2023-03-03T07:56:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.220+0000 7f2aa97fa700 1 -- 172.21.15.98:0/1334985648 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2aa4011960 con 0x7f2aac0b40a0 2023-03-03T07:56:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.221+0000 7f2aa97fa700 1 --2- 172.21.15.98:0/1334985648 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2a900682f0 0x7f2a9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.221+0000 7f2aa97fa700 1 -- 172.21.15.98:0/1334985648 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f2aa4086220 con 0x7f2aac0b40a0 2023-03-03T07:56:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.221+0000 7f2ab0b9b700 1 --2- 172.21.15.98:0/1334985648 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2a900682f0 0x7f2a9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.222+0000 7f2ab0b9b700 1 --2- 172.21.15.98:0/1334985648 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2a900682f0 0x7f2a9006a7a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000d010 tx=0x7f2aa0007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.225+0000 7f2aa97fa700 1 -- 172.21.15.98:0/1334985648 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2aac1494a0 con 0x7f2aac0b40a0 2023-03-03T07:56:34.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.419+0000 7f2ab2dff700 1 -- 172.21.15.98:0/1334985648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2aac1494a0 con 0x7f2aac0b40a0 2023-03-03T07:56:34.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.420+0000 7f2aa97fa700 1 -- 172.21.15.98:0/1334985648 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2aac1494a0 con 0x7f2aac0b40a0 2023-03-03T07:56:34.422 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:56:34.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.423+0000 7f2ab2dff700 1 -- 172.21.15.98:0/1334985648 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2a900682f0 msgr2=0x7f2a9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:34.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.423+0000 7f2ab2dff700 1 --2- 172.21.15.98:0/1334985648 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2a900682f0 0x7f2a9006a7a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000d010 tx=0x7f2aa0007000 comp rx=0 tx=0).stop 2023-03-03T07:56:34.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.423+0000 7f2ab2dff700 1 -- 172.21.15.98:0/1334985648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2aac0b40a0 msgr2=0x7f2aac15af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:34.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.423+0000 7f2ab2dff700 1 --2- 172.21.15.98:0/1334985648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2aac0b40a0 0x7f2aac15af40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0b1ff0 tx=0x7f2aa40114b0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.423+0000 7f2ab2dff700 1 -- 172.21.15.98:0/1334985648 shutdown_connections 2023-03-03T07:56:34.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.423+0000 7f2ab2dff700 1 --2- 172.21.15.98:0/1334985648 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2a900682f0 0x7f2a9006a7a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.423+0000 7f2ab2dff700 1 --2- 172.21.15.98:0/1334985648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2aac0b40a0 0x7f2aac15af40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.423+0000 7f2ab2dff700 1 --2- 172.21.15.98:0/1334985648 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a9c003bc0 0x7f2aac0cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.423+0000 7f2ab2dff700 1 --2- 172.21.15.98:0/1334985648 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2aac0b3c30 0x7f2aac15aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.423+0000 7f2ab2dff700 1 -- 172.21.15.98:0/1334985648 >> 172.21.15.98:0/1334985648 conn(0x7f2aac073e70 msgr2=0x7f2aac0b8090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:34.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.423+0000 7f2ab2dff700 1 -- 172.21.15.98:0/1334985648 shutdown_connections 2023-03-03T07:56:34.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.423+0000 7f2ab2dff700 1 -- 172.21.15.98:0/1334985648 wait complete. 2023-03-03T07:56:34.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:56:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.703+0000 7f4f055ff700 1 Processor -- start 2023-03-03T07:56:34.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.703+0000 7f4f055ff700 1 -- start start 2023-03-03T07:56:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.704+0000 7f4f055ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f000b26d0 0x7f4f000b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.704+0000 7f4f055ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4f000b5140 0x7f4f000d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.704+0000 7f4f055ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f000cd7d0 con 0x7f4f000b8d10 2023-03-03T07:56:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.704+0000 7f4f055ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f000b55c0 con 0x7f4f000b5140 2023-03-03T07:56:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.704+0000 7f4f055ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f000b90e0 con 0x7f4f000b26d0 2023-03-03T07:56:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.705+0000 7f4eff7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4f000b5140 0x7f4f000d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.705+0000 7f4eff7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4f000b5140 0x7f4f000d13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40756/0 (socket says 172.21.15.98:40756) 2023-03-03T07:56:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.705+0000 7f4eff7fe700 1 -- 172.21.15.98:0/4210049616 learned_addr learned my addr 172.21.15.98:0/4210049616 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.705+0000 7f4efe7fc700 1 --2- 172.21.15.98:0/4210049616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f000b26d0 0x7f4f000b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.705+0000 7f4eff7fe700 1 -- 172.21.15.98:0/4210049616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f000b26d0 msgr2=0x7f4f000b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.705+0000 7f4eff7fe700 1 --2- 172.21.15.98:0/4210049616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f000b26d0 0x7f4f000b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.705+0000 7f4eff7fe700 1 -- 172.21.15.98:0/4210049616 >> v1:172.21.15.100:6789/0 conn(0x7f4f000b8d10 legacy=0x7f4f000cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.705+0000 7f4eff7fe700 1 -- 172.21.15.98:0/4210049616 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f000cda20 con 0x7f4f000b5140 2023-03-03T07:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.706+0000 7f4eff7fe700 1 --2- 172.21.15.98:0/4210049616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f000b5140 0x7f4f000d13a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8009e50 tx=0x7f4ef800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d13f6dd5f1a7d2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.706+0000 7f4efdffb700 1 -- 172.21.15.98:0/4210049616 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef800e070 con 0x7f4f000b5140 2023-03-03T07:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.706+0000 7f4efdffb700 1 -- 172.21.15.98:0/4210049616 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef800b910 con 0x7f4f000b5140 2023-03-03T07:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.706+0000 7f4efdffb700 1 -- 172.21.15.98:0/4210049616 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef800bcc0 con 0x7f4f000b5140 2023-03-03T07:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.706+0000 7f4f055ff700 1 -- 172.21.15.98:0/4210049616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f000b5140 msgr2=0x7f4f000d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.706+0000 7f4f055ff700 1 --2- 172.21.15.98:0/4210049616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f000b5140 0x7f4f000d13a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8009e50 tx=0x7f4ef800b330 comp rx=0 tx=0).stop 2023-03-03T07:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.706+0000 7f4f055ff700 1 -- 172.21.15.98:0/4210049616 shutdown_connections 2023-03-03T07:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.706+0000 7f4f055ff700 1 --2- 172.21.15.98:0/4210049616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f000b5140 0x7f4f000d13a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.706+0000 7f4f055ff700 1 --2- 172.21.15.98:0/4210049616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f000b26d0 0x7f4f000b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.706+0000 7f4f055ff700 1 -- 172.21.15.98:0/4210049616 >> 172.21.15.98:0/4210049616 conn(0x7f4f00073e70 msgr2=0x7f4f0006ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.707+0000 7f4f055ff700 1 -- 172.21.15.98:0/4210049616 shutdown_connections 2023-03-03T07:56:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.707+0000 7f4f055ff700 1 -- 172.21.15.98:0/4210049616 wait complete. 2023-03-03T07:56:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.709+0000 7f4f055ff700 1 Processor -- start 2023-03-03T07:56:34.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.709+0000 7f4f055ff700 1 -- start start 2023-03-03T07:56:34.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.710+0000 7f4f055ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f000b26d0 0x7f4f0015d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:34.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.710+0000 7f4f055ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f000b5140 0x7f4f00151380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.710+0000 7f4f055ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4f000b8d10 0x7f4f000cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.710+0000 7f4f055ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f000c8070 con 0x7f4f000b26d0 2023-03-03T07:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.710+0000 7f4f055ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f000cc190 con 0x7f4f000b8d10 2023-03-03T07:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.710+0000 7f4f055ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f000c82c0 con 0x7f4f000b5140 2023-03-03T07:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.710+0000 7f4efeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f000b26d0 0x7f4f0015d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.710+0000 7f4efeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f000b26d0 0x7f4f0015d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40772/0 (socket says 172.21.15.98:40772) 2023-03-03T07:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.710+0000 7f4efeffd700 1 -- 172.21.15.98:0/473898291 learned_addr learned my addr 172.21.15.98:0/473898291 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.710+0000 7f4efe7fc700 1 --2- 172.21.15.98:0/473898291 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f000b5140 0x7f4f00151380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.710+0000 7f4eff7fe700 1 --2- 172.21.15.98:0/473898291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4f000b8d10 0x7f4f000cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.710+0000 7f4efeffd700 1 -- 172.21.15.98:0/473898291 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f000b5140 msgr2=0x7f4f00151380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.711+0000 7f4efeffd700 1 --2- 172.21.15.98:0/473898291 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f000b5140 0x7f4f00151380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.711+0000 7f4efeffd700 1 -- 172.21.15.98:0/473898291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4f000b8d10 msgr2=0x7f4f000cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.711+0000 7f4efeffd700 1 --2- 172.21.15.98:0/473898291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4f000b8d10 0x7f4f000cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.711+0000 7f4efeffd700 1 -- 172.21.15.98:0/473898291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef8009b30 con 0x7f4f000b26d0 2023-03-03T07:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.711+0000 7f4efeffd700 1 --2- 172.21.15.98:0/473898291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f000b26d0 0x7f4f0015d2f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000c450 tx=0x7f4ef000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.711+0000 7f4ee77fe700 1 -- 172.21.15.98:0/473898291 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef00052d0 con 0x7f4f000b26d0 2023-03-03T07:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.711+0000 7f4ee77fe700 1 -- 172.21.15.98:0/473898291 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef0005430 con 0x7f4f000b26d0 2023-03-03T07:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.711+0000 7f4ee77fe700 1 -- 172.21.15.98:0/473898291 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef00038c0 con 0x7f4f000b26d0 2023-03-03T07:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.711+0000 7f4f055ff700 1 -- 172.21.15.98:0/473898291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f000d0e80 con 0x7f4f000b26d0 2023-03-03T07:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.711+0000 7f4f055ff700 1 -- 172.21.15.98:0/473898291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f000cdd00 con 0x7f4f000b26d0 2023-03-03T07:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.712+0000 7f4f055ff700 1 -- 172.21.15.98:0/473898291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f000cdfe0 con 0x7f4f000b26d0 2023-03-03T07:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.714+0000 7f4ee77fe700 1 -- 172.21.15.98:0/473898291 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ef0019030 con 0x7f4f000b26d0 2023-03-03T07:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.715+0000 7f4ee77fe700 1 --2- 172.21.15.98:0/473898291 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4eec0682f0 0x7f4eec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.715+0000 7f4ee77fe700 1 -- 172.21.15.98:0/473898291 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f4ef0084f60 con 0x7f4f000b26d0 2023-03-03T07:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.715+0000 7f4efe7fc700 1 --2- 172.21.15.98:0/473898291 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4eec0682f0 0x7f4eec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.716+0000 7f4efe7fc700 1 --2- 172.21.15.98:0/473898291 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4eec0682f0 0x7f4eec06a7a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4f000b2500 tx=0x7f4ef4009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.719+0000 7f4ee77fe700 1 -- 172.21.15.98:0/473898291 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ef0051690 con 0x7f4f000b26d0 2023-03-03T07:56:34.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.912+0000 7f4f055ff700 1 -- 172.21.15.98:0/473898291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4f0004e6e0 con 0x7f4f000b26d0 2023-03-03T07:56:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.912+0000 7f4ee77fe700 1 -- 172.21.15.98:0/473898291 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4ef00170a0 con 0x7f4f000b26d0 2023-03-03T07:56:34.915 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:56:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.915+0000 7f4f055ff700 1 -- 172.21.15.98:0/473898291 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4eec0682f0 msgr2=0x7f4eec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.915+0000 7f4f055ff700 1 --2- 172.21.15.98:0/473898291 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4eec0682f0 0x7f4eec06a7a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4f000b2500 tx=0x7f4ef4009210 comp rx=0 tx=0).stop 2023-03-03T07:56:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.915+0000 7f4f055ff700 1 -- 172.21.15.98:0/473898291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f000b26d0 msgr2=0x7f4f0015d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.915+0000 7f4f055ff700 1 --2- 172.21.15.98:0/473898291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f000b26d0 0x7f4f0015d2f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000c450 tx=0x7f4ef000c760 comp rx=0 tx=0).stop 2023-03-03T07:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.915+0000 7f4f055ff700 1 -- 172.21.15.98:0/473898291 shutdown_connections 2023-03-03T07:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.915+0000 7f4f055ff700 1 --2- 172.21.15.98:0/473898291 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4eec0682f0 0x7f4eec06a7a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.915+0000 7f4f055ff700 1 --2- 172.21.15.98:0/473898291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f000b26d0 0x7f4f0015d2f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.915+0000 7f4f055ff700 1 --2- 172.21.15.98:0/473898291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4f000b8d10 0x7f4f000cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.915+0000 7f4f055ff700 1 --2- 172.21.15.98:0/473898291 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f000b5140 0x7f4f00151380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.915+0000 7f4f055ff700 1 -- 172.21.15.98:0/473898291 >> 172.21.15.98:0/473898291 conn(0x7f4f00073e70 msgr2=0x7f4f0006ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.915+0000 7f4f055ff700 1 -- 172.21.15.98:0/473898291 shutdown_connections 2023-03-03T07:56:34.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:34.915+0000 7f4f055ff700 1 -- 172.21.15.98:0/473898291 wait complete. 2023-03-03T07:56:34.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T07:56:35.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.193+0000 7f8c421ff700 1 Processor -- start 2023-03-03T07:56:35.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.193+0000 7f8c421ff700 1 -- start start 2023-03-03T07:56:35.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.195+0000 7f8c421ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8c3c0b8d10 0x7f8c3c0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.195+0000 7f8c421ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c3c0b26d0 0x7f8c3c0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.195+0000 7f8c421ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8c3c0b5140 0x7f8c3c160520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.195+0000 7f8c421ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c3c0b55c0 con 0x7f8c3c0b5140 2023-03-03T07:56:35.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.195+0000 7f8c421ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c3c0cd7d0 con 0x7f8c3c0b8d10 2023-03-03T07:56:35.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.195+0000 7f8c421ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3c0b90e0 con 0x7f8c3c0b26d0 2023-03-03T07:56:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.195+0000 7f8c3bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8c3c0b5140 0x7f8c3c160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.195+0000 7f8c3bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8c3c0b5140 0x7f8c3c160520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42124/0 (socket says 172.21.15.98:42124) 2023-03-03T07:56:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.195+0000 7f8c3bfff700 1 -- 172.21.15.98:0/2027040203 learned_addr learned my addr 172.21.15.98:0/2027040203 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.195+0000 7f8c3affd700 1 --2- 172.21.15.98:0/2027040203 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c3c0b26d0 0x7f8c3c0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.195+0000 7f8c3b7fe700 1 --2- 172.21.15.98:0/2027040203 >> v2:172.21.15.100:3300/0 conn(0x7f8c3c0b8d10 0x7f8c3c0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.196+0000 7f8c3bfff700 1 -- 172.21.15.98:0/2027040203 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c3c0b26d0 msgr2=0x7f8c3c0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.196+0000 7f8c3bfff700 1 --2- 172.21.15.98:0/2027040203 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c3c0b26d0 0x7f8c3c0b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.196+0000 7f8c3bfff700 1 -- 172.21.15.98:0/2027040203 >> v2:172.21.15.100:3300/0 conn(0x7f8c3c0b8d10 msgr2=0x7f8c3c0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.196+0000 7f8c3bfff700 1 --2- 172.21.15.98:0/2027040203 >> v2:172.21.15.100:3300/0 conn(0x7f8c3c0b8d10 0x7f8c3c0cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.196+0000 7f8c3bfff700 1 -- 172.21.15.98:0/2027040203 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3c0cda20 con 0x7f8c3c0b5140 2023-03-03T07:56:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.196+0000 7f8c3bfff700 1 --2- 172.21.15.98:0/2027040203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c0b5140 0x7f8c3c160520 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c08c040 tx=0x7f8c3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f32acccf331c1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.196+0000 7f8c3a7fc700 1 -- 172.21.15.98:0/2027040203 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3400e070 con 0x7f8c3c0b5140 2023-03-03T07:56:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.196+0000 7f8c3a7fc700 1 -- 172.21.15.98:0/2027040203 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3400b910 con 0x7f8c3c0b5140 2023-03-03T07:56:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.196+0000 7f8c3a7fc700 1 -- 172.21.15.98:0/2027040203 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3400bcc0 con 0x7f8c3c0b5140 2023-03-03T07:56:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.197+0000 7f8c421ff700 1 -- 172.21.15.98:0/2027040203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c0b5140 msgr2=0x7f8c3c160520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.197+0000 7f8c421ff700 1 --2- 172.21.15.98:0/2027040203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c0b5140 0x7f8c3c160520 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c08c040 tx=0x7f8c3400b330 comp rx=0 tx=0).stop 2023-03-03T07:56:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.197+0000 7f8c421ff700 1 -- 172.21.15.98:0/2027040203 shutdown_connections 2023-03-03T07:56:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.197+0000 7f8c421ff700 1 --2- 172.21.15.98:0/2027040203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c0b5140 0x7f8c3c160520 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.197+0000 7f8c421ff700 1 --2- 172.21.15.98:0/2027040203 >> v2:172.21.15.100:3300/0 conn(0x7f8c3c0b8d10 0x7f8c3c0cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.197+0000 7f8c421ff700 1 --2- 172.21.15.98:0/2027040203 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c3c0b26d0 0x7f8c3c0b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.197+0000 7f8c421ff700 1 -- 172.21.15.98:0/2027040203 >> 172.21.15.98:0/2027040203 conn(0x7f8c3c073e70 msgr2=0x7f8c3c06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.197+0000 7f8c421ff700 1 -- 172.21.15.98:0/2027040203 shutdown_connections 2023-03-03T07:56:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.197+0000 7f8c421ff700 1 -- 172.21.15.98:0/2027040203 wait complete. 2023-03-03T07:56:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.199+0000 7f8c421ff700 1 Processor -- start 2023-03-03T07:56:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.199+0000 7f8c421ff700 1 -- start start 2023-03-03T07:56:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.200+0000 7f8c421ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c0b26d0 0x7f8c3c1f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.200+0000 7f8c421ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c3c0b5140 0x7f8c3c15b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.200+0000 7f8c421ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c3c0b8d10 0x7f8c3c0d17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.200+0000 7f8c421ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3c0ca6d0 con 0x7f8c3c0b26d0 2023-03-03T07:56:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.200+0000 7f8c421ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3c1f2f10 con 0x7f8c3c0b8d10 2023-03-03T07:56:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.200+0000 7f8c421ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3c0cc2c0 con 0x7f8c3c0b5140 2023-03-03T07:56:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.201+0000 7f8c3b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c0b26d0 0x7f8c3c1f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.201+0000 7f8c3b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c0b26d0 0x7f8c3c1f29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42136/0 (socket says 172.21.15.98:42136) 2023-03-03T07:56:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.201+0000 7f8c3b7fe700 1 -- 172.21.15.98:0/812698051 learned_addr learned my addr 172.21.15.98:0/812698051 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.201+0000 7f8c3bfff700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c3c0b8d10 0x7f8c3c0d17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.201+0000 7f8c3affd700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c3c0b5140 0x7f8c3c15b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.201+0000 7f8c3b7fe700 1 -- 172.21.15.98:0/812698051 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c3c0b5140 msgr2=0x7f8c3c15b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.201+0000 7f8c3b7fe700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c3c0b5140 0x7f8c3c15b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.201+0000 7f8c3b7fe700 1 -- 172.21.15.98:0/812698051 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c3c0b8d10 msgr2=0x7f8c3c0d17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.201+0000 7f8c3b7fe700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c3c0b8d10 0x7f8c3c0d17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.201+0000 7f8c3b7fe700 1 -- 172.21.15.98:0/812698051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c34009b30 con 0x7f8c3c0b26d0 2023-03-03T07:56:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.201+0000 7f8c3affd700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c3c0b5140 0x7f8c3c15b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.201+0000 7f8c3bfff700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c3c0b8d10 0x7f8c3c0d17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.201+0000 7f8c3b7fe700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c0b26d0 0x7f8c3c1f29d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c00d800 tx=0x7f8c2c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.202+0000 7f8c23fff700 1 -- 172.21.15.98:0/812698051 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c0125f0 con 0x7f8c3c0b26d0 2023-03-03T07:56:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.202+0000 7f8c23fff700 1 -- 172.21.15.98:0/812698051 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2c012750 con 0x7f8c3c0b26d0 2023-03-03T07:56:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.202+0000 7f8c421ff700 1 -- 172.21.15.98:0/812698051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c3c0d0e80 con 0x7f8c3c0b26d0 2023-03-03T07:56:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.202+0000 7f8c23fff700 1 -- 172.21.15.98:0/812698051 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c012b00 con 0x7f8c3c0b26d0 2023-03-03T07:56:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.202+0000 7f8c421ff700 1 -- 172.21.15.98:0/812698051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c3c0cdd00 con 0x7f8c3c0b26d0 2023-03-03T07:56:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.203+0000 7f8c421ff700 1 -- 172.21.15.98:0/812698051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c3c0cdfe0 con 0x7f8c3c0b26d0 2023-03-03T07:56:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.204+0000 7f8c23fff700 1 -- 172.21.15.98:0/812698051 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8c2c012c60 con 0x7f8c3c0b26d0 2023-03-03T07:56:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.206+0000 7f8c23fff700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c280682f0 0x7f8c2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.206+0000 7f8c23fff700 1 -- 172.21.15.98:0/812698051 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f8c2c086260 con 0x7f8c3c0b26d0 2023-03-03T07:56:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.206+0000 7f8c3affd700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c280682f0 0x7f8c2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.207+0000 7f8c3affd700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c280682f0 0x7f8c2806a7a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0b5960 tx=0x7f8c30008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.210+0000 7f8c23fff700 1 -- 172.21.15.98:0/812698051 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8c2c052990 con 0x7f8c3c0b26d0 2023-03-03T07:56:35.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.419+0000 7f8c421ff700 1 -- 172.21.15.98:0/812698051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8c3c0ca940 con 0x7f8c3c0b26d0 2023-03-03T07:56:35.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.419+0000 7f8c23fff700 1 -- 172.21.15.98:0/812698051 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8c2c055fb0 con 0x7f8c3c0b26d0 2023-03-03T07:56:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.422+0000 7f8c421ff700 1 -- 172.21.15.98:0/812698051 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c280682f0 msgr2=0x7f8c2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.422+0000 7f8c421ff700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c280682f0 0x7f8c2806a7a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0b5960 tx=0x7f8c30008040 comp rx=0 tx=0).stop 2023-03-03T07:56:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.422+0000 7f8c421ff700 1 -- 172.21.15.98:0/812698051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c0b26d0 msgr2=0x7f8c3c1f29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.422+0000 7f8c421ff700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c0b26d0 0x7f8c3c1f29d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c00d800 tx=0x7f8c2c00dbc0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.422+0000 7f8c421ff700 1 -- 172.21.15.98:0/812698051 shutdown_connections 2023-03-03T07:56:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.422+0000 7f8c421ff700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c280682f0 0x7f8c2806a7a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.422+0000 7f8c421ff700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c0b26d0 0x7f8c3c1f29d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.422+0000 7f8c421ff700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c3c0b8d10 0x7f8c3c0d17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.422+0000 7f8c421ff700 1 --2- 172.21.15.98:0/812698051 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c3c0b5140 0x7f8c3c15b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.422+0000 7f8c421ff700 1 -- 172.21.15.98:0/812698051 >> 172.21.15.98:0/812698051 conn(0x7f8c3c073e70 msgr2=0x7f8c3c0cfc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:35.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.422+0000 7f8c421ff700 1 -- 172.21.15.98:0/812698051 shutdown_connections 2023-03-03T07:56:35.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.422+0000 7f8c421ff700 1 -- 172.21.15.98:0/812698051 wait complete. 2023-03-03T07:56:35.426 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T07:56:35.451 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:56:35.451 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:56:35.521 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:56:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.763+0000 7f0927bff700 1 Processor -- start 2023-03-03T07:56:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.763+0000 7f0927bff700 1 -- start start 2023-03-03T07:56:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.764+0000 7f0927bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f092015cc00 0x7f09200cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.764+0000 7f0927bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f09200d2c30 0x7f09200cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.764+0000 7f0927bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09200cde10 con 0x7f09200d27c0 2023-03-03T07:56:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.764+0000 7f0927bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09200ce040 con 0x7f09200d2c30 2023-03-03T07:56:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.764+0000 7f0927bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09200d30b0 con 0x7f092015cc00 2023-03-03T07:56:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.765+0000 7f092519a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f09200d27c0 0x7f09200b60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54056/0 (socket says 172.21.15.98:54056) 2023-03-03T07:56:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.765+0000 7f092519a700 1 -- 172.21.15.98:0/1749215842 learned_addr learned my addr 172.21.15.98:0/1749215842 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.765+0000 7f092619c700 1 --2- 172.21.15.98:0/1749215842 >> v2:172.21.15.100:3300/0 conn(0x7f09200d2c30 0x7f09200cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.765+0000 7f092599b700 1 --2- 172.21.15.98:0/1749215842 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f092015cc00 0x7f09200cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.765+0000 7f0924999700 1 -- 172.21.15.98:0/1749215842 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414767186 0 0) 0x7f09200cde10 con 0x7f09200d27c0 2023-03-03T07:56:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.765+0000 7f0924999700 1 -- 172.21.15.98:0/1749215842 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f090c0035b0 con 0x7f09200d27c0 2023-03-03T07:56:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.765+0000 7f0924999700 1 -- 172.21.15.98:0/1749215842 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2175643256 0 0) 0x7f090c0035b0 con 0x7f09200d27c0 2023-03-03T07:56:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.765+0000 7f0924999700 1 -- 172.21.15.98:0/1749215842 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f092015cc00 msgr2=0x7f09200cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.765+0000 7f0924999700 1 --2- 172.21.15.98:0/1749215842 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f092015cc00 0x7f09200cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.765+0000 7f0924999700 1 -- 172.21.15.98:0/1749215842 >> v2:172.21.15.100:3300/0 conn(0x7f09200d2c30 msgr2=0x7f09200cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.765+0000 7f0924999700 1 --2- 172.21.15.98:0/1749215842 >> v2:172.21.15.100:3300/0 conn(0x7f09200d2c30 0x7f09200cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.765+0000 7f0924999700 1 -- 172.21.15.98:0/1749215842 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09200d1a00 con 0x7f09200d27c0 2023-03-03T07:56:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.765+0000 7f0924999700 1 -- 172.21.15.98:0/1749215842 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f0910001670 con 0x7f09200d27c0 2023-03-03T07:56:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.765+0000 7f092599b700 1 --2- 172.21.15.98:0/1749215842 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f092015cc00 0x7f09200cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:56:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.766+0000 7f0924999700 1 -- 172.21.15.98:0/1749215842 >> v1:172.21.15.98:6789/0 conn(0x7f09200d27c0 legacy=0x7f09200b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.766+0000 7f0924999700 1 --2- 172.21.15.98:0/1749215842 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f092015cc00 0x7f090c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.766+0000 7f0924999700 1 --2- 172.21.15.98:0/1749215842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003f40 0x7f090c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.766+0000 7f0924999700 1 --2- 172.21.15.98:0/1749215842 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f090c004d90 0x7f090c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.766+0000 7f0924999700 1 -- 172.21.15.98:0/1749215842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09200ce040 con 0x7f090c003f40 2023-03-03T07:56:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.766+0000 7f0924999700 1 -- 172.21.15.98:0/1749215842 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09200d30b0 con 0x7f090c004d90 2023-03-03T07:56:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.766+0000 7f0924999700 1 -- 172.21.15.98:0/1749215842 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090c00ba10 con 0x7f092015cc00 2023-03-03T07:56:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.766+0000 7f092519a700 1 --2- 172.21.15.98:0/1749215842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003f40 0x7f090c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.766+0000 7f092619c700 1 --2- 172.21.15.98:0/1749215842 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f090c004d90 0x7f090c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.766+0000 7f092599b700 1 --2- 172.21.15.98:0/1749215842 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f092015cc00 0x7f090c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.766+0000 7f092519a700 1 -- 172.21.15.98:0/1749215842 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f092015cc00 msgr2=0x7f090c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.766+0000 7f092519a700 1 --2- 172.21.15.98:0/1749215842 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f092015cc00 0x7f090c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.766+0000 7f092519a700 1 -- 172.21.15.98:0/1749215842 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f090c004d90 msgr2=0x7f090c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.767+0000 7f092519a700 1 --2- 172.21.15.98:0/1749215842 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f090c004d90 0x7f090c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.767+0000 7f092519a700 1 -- 172.21.15.98:0/1749215842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c0094a0 con 0x7f090c003f40 2023-03-03T07:56:35.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.767+0000 7f092519a700 1 --2- 172.21.15.98:0/1749215842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003f40 0x7f090c004850 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f091000c320 tx=0x7f090c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afe02bce0d39d524 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:35.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.767+0000 7f0924999700 1 -- 172.21.15.98:0/1749215842 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091000ce60 con 0x7f090c003f40 2023-03-03T07:56:35.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.767+0000 7f0924999700 1 -- 172.21.15.98:0/1749215842 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0910005960 con 0x7f090c003f40 2023-03-03T07:56:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.767+0000 7f0924999700 1 -- 172.21.15.98:0/1749215842 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0910005cf0 con 0x7f090c003f40 2023-03-03T07:56:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.768+0000 7f0927bff700 1 -- 172.21.15.98:0/1749215842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003f40 msgr2=0x7f090c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.768+0000 7f0927bff700 1 --2- 172.21.15.98:0/1749215842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003f40 0x7f090c004850 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f091000c320 tx=0x7f090c005880 comp rx=0 tx=0).stop 2023-03-03T07:56:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.768+0000 7f092619c700 1 -- 172.21.15.98:0/1749215842 reap_dead start 2023-03-03T07:56:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.768+0000 7f0927bff700 1 -- 172.21.15.98:0/1749215842 shutdown_connections 2023-03-03T07:56:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.768+0000 7f0927bff700 1 -- 172.21.15.98:0/1749215842 >> 172.21.15.98:0/1749215842 conn(0x7f092006aee0 msgr2=0x7f092006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.768+0000 7f0927bff700 1 -- 172.21.15.98:0/1749215842 shutdown_connections 2023-03-03T07:56:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.768+0000 7f0927bff700 1 -- 172.21.15.98:0/1749215842 wait complete. 2023-03-03T07:56:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.771+0000 7f0927bff700 1 Processor -- start 2023-03-03T07:56:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.771+0000 7f0927bff700 1 -- start start 2023-03-03T07:56:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.772+0000 7f0927bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003f40 0x7f09200b6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.772+0000 7f0927bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f092014a350 0x7f092014a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.772+0000 7f0927bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09200cb4c0 0x7f09200cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.772+0000 7f0927bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09200ccde0 con 0x7f090c003f40 2023-03-03T07:56:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.772+0000 7f0927bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09200b9090 con 0x7f09200cb4c0 2023-03-03T07:56:35.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.772+0000 7f0927bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09200cd1e0 con 0x7f092014a350 2023-03-03T07:56:35.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.773+0000 7f092599b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003f40 0x7f09200b6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.773+0000 7f092599b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003f40 0x7f09200b6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42156/0 (socket says 172.21.15.98:42156) 2023-03-03T07:56:35.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.773+0000 7f092599b700 1 -- 172.21.15.98:0/750404669 learned_addr learned my addr 172.21.15.98:0/750404669 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:35.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.773+0000 7f092619c700 1 --2- 172.21.15.98:0/750404669 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09200cb4c0 0x7f09200cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.773+0000 7f092519a700 1 --2- 172.21.15.98:0/750404669 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f092014a350 0x7f092014a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.773+0000 7f092599b700 1 -- 172.21.15.98:0/750404669 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f092014a350 msgr2=0x7f092014a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.773+0000 7f092599b700 1 --2- 172.21.15.98:0/750404669 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f092014a350 0x7f092014a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.773+0000 7f092599b700 1 -- 172.21.15.98:0/750404669 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09200cb4c0 msgr2=0x7f09200cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.773+0000 7f092599b700 1 --2- 172.21.15.98:0/750404669 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09200cb4c0 0x7f09200cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.773+0000 7f092599b700 1 -- 172.21.15.98:0/750404669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09200d1fc0 con 0x7f090c003f40 2023-03-03T07:56:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.773+0000 7f092519a700 1 --2- 172.21.15.98:0/750404669 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f092014a350 0x7f092014a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:56:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.774+0000 7f092599b700 1 --2- 172.21.15.98:0/750404669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003f40 0x7f09200b6d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f091c011cd0 tx=0x7f091c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.774+0000 7f09167fc700 1 -- 172.21.15.98:0/750404669 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091c011e70 con 0x7f090c003f40 2023-03-03T07:56:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.774+0000 7f09167fc700 1 -- 172.21.15.98:0/750404669 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091c00fce0 con 0x7f090c003f40 2023-03-03T07:56:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.774+0000 7f09167fc700 1 -- 172.21.15.98:0/750404669 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091c004240 con 0x7f090c003f40 2023-03-03T07:56:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.774+0000 7f0927bff700 1 -- 172.21.15.98:0/750404669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09200d1ce0 con 0x7f090c003f40 2023-03-03T07:56:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.774+0000 7f0927bff700 1 -- 172.21.15.98:0/750404669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09200d22a0 con 0x7f090c003f40 2023-03-03T07:56:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.775+0000 7f0927bff700 1 -- 172.21.15.98:0/750404669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09200729b0 con 0x7f090c003f40 2023-03-03T07:56:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.776+0000 7f09167fc700 1 -- 172.21.15.98:0/750404669 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f091c024070 con 0x7f090c003f40 2023-03-03T07:56:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.777+0000 7f09167fc700 1 --2- 172.21.15.98:0/750404669 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f09040682f0 0x7f090406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.777+0000 7f09167fc700 1 -- 172.21.15.98:0/750404669 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f091c0862d0 con 0x7f090c003f40 2023-03-03T07:56:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.778+0000 7f092519a700 1 --2- 172.21.15.98:0/750404669 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f09040682f0 0x7f090406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.778+0000 7f092519a700 1 --2- 172.21.15.98:0/750404669 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f09040682f0 0x7f090406a7a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0910015040 tx=0x7f091000a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.781+0000 7f09167fc700 1 -- 172.21.15.98:0/750404669 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f091c052a00 con 0x7f090c003f40 2023-03-03T07:56:35.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.974+0000 7f0927bff700 1 -- 172.21.15.98:0/750404669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f092004e730 con 0x7f090c003f40 2023-03-03T07:56:35.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.975+0000 7f09167fc700 1 -- 172.21.15.98:0/750404669 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18003 (secure 0 0 0) 0x7f091c022090 con 0x7f090c003f40 2023-03-03T07:56:35.978 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:56:35.978 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:56:27.498205+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:56:35.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.978+0000 7f0927bff700 1 -- 172.21.15.98:0/750404669 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f09040682f0 msgr2=0x7f090406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:35.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.978+0000 7f0927bff700 1 --2- 172.21.15.98:0/750404669 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f09040682f0 0x7f090406a7a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0910015040 tx=0x7f091000a000 comp rx=0 tx=0).stop 2023-03-03T07:56:35.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.978+0000 7f0927bff700 1 -- 172.21.15.98:0/750404669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003f40 msgr2=0x7f09200b6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:35.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.978+0000 7f0927bff700 1 --2- 172.21.15.98:0/750404669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003f40 0x7f09200b6d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f091c011cd0 tx=0x7f091c00f520 comp rx=0 tx=0).stop 2023-03-03T07:56:35.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.978+0000 7f0927bff700 1 -- 172.21.15.98:0/750404669 shutdown_connections 2023-03-03T07:56:35.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.978+0000 7f0927bff700 1 --2- 172.21.15.98:0/750404669 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f09040682f0 0x7f090406a7a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.978+0000 7f0927bff700 1 --2- 172.21.15.98:0/750404669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003f40 0x7f09200b6d50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.978+0000 7f0927bff700 1 --2- 172.21.15.98:0/750404669 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09200cb4c0 0x7f09200cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.978+0000 7f0927bff700 1 --2- 172.21.15.98:0/750404669 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f092014a350 0x7f092014a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:35.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.978+0000 7f0927bff700 1 -- 172.21.15.98:0/750404669 >> 172.21.15.98:0/750404669 conn(0x7f092006aee0 msgr2=0x7f09200ce250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:35.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.978+0000 7f0927bff700 1 -- 172.21.15.98:0/750404669 shutdown_connections 2023-03-03T07:56:35.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:35.979+0000 7f0927bff700 1 -- 172.21.15.98:0/750404669 wait complete. 2023-03-03T07:56:36.008 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:55:45.157666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:55:46.092091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:56:36.009 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:56:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.256+0000 7fe44d9ff700 1 Processor -- start 2023-03-03T07:56:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.256+0000 7fe44d9ff700 1 -- start start 2023-03-03T07:56:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.258+0000 7fe44d9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4480d0270 0x7fe4480cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.258+0000 7fe44d9ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4480cf980 con 0x7fe44806b4b0 2023-03-03T07:56:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.258+0000 7fe44d9ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4480d1800 con 0x7fe44815ca70 2023-03-03T07:56:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.258+0000 7fe44d9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4480cd5b0 con 0x7fe4480d0270 2023-03-03T07:56:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.258+0000 7fe4477fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe44806b4b0 0x7fe4480d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54064/0 (socket says 172.21.15.98:54064) 2023-03-03T07:56:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.258+0000 7fe4477fe700 1 -- 172.21.15.98:0/1723201646 learned_addr learned my addr 172.21.15.98:0/1723201646 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.258+0000 7fe4467fc700 1 --2- 172.21.15.98:0/1723201646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4480d0270 0x7fe4480cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.259+0000 7fe445ffb700 1 -- 172.21.15.98:0/1723201646 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304126140 0 0) 0x7fe4480cf980 con 0x7fe44806b4b0 2023-03-03T07:56:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.259+0000 7fe445ffb700 1 -- 172.21.15.98:0/1723201646 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4300035b0 con 0x7fe44806b4b0 2023-03-03T07:56:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.259+0000 7fe445ffb700 1 -- 172.21.15.98:0/1723201646 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335941715 0 0) 0x7fe4480d1800 con 0x7fe44815ca70 2023-03-03T07:56:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.259+0000 7fe445ffb700 1 -- 172.21.15.98:0/1723201646 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4480cf980 con 0x7fe44815ca70 2023-03-03T07:56:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.259+0000 7fe4467fc700 1 -- 172.21.15.98:0/1723201646 >> v1:172.21.15.100:6789/0 conn(0x7fe44815ca70 legacy=0x7fe4480cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.259+0000 7fe4467fc700 1 -- 172.21.15.98:0/1723201646 >> v1:172.21.15.98:6789/0 conn(0x7fe44806b4b0 legacy=0x7fe4480d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.259+0000 7fe4467fc700 1 -- 172.21.15.98:0/1723201646 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4480cd900 con 0x7fe4480d0270 2023-03-03T07:56:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.259+0000 7fe4467fc700 1 --2- 172.21.15.98:0/1723201646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4480d0270 0x7fe4480cf440 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe43c009870 tx=0x7fe43c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53e91c540ae288f3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.259+0000 7fe445ffb700 1 -- 172.21.15.98:0/1723201646 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c011830 con 0x7fe4480d0270 2023-03-03T07:56:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.260+0000 7fe445ffb700 1 -- 172.21.15.98:0/1723201646 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43c011990 con 0x7fe4480d0270 2023-03-03T07:56:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.260+0000 7fe445ffb700 1 -- 172.21.15.98:0/1723201646 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c011d20 con 0x7fe4480d0270 2023-03-03T07:56:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.260+0000 7fe44d9ff700 1 -- 172.21.15.98:0/1723201646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4480d0270 msgr2=0x7fe4480cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.260+0000 7fe44d9ff700 1 --2- 172.21.15.98:0/1723201646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4480d0270 0x7fe4480cf440 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe43c009870 tx=0x7fe43c009b80 comp rx=0 tx=0).stop 2023-03-03T07:56:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.260+0000 7fe44d9ff700 1 -- 172.21.15.98:0/1723201646 shutdown_connections 2023-03-03T07:56:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.260+0000 7fe44d9ff700 1 --2- 172.21.15.98:0/1723201646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4480d0270 0x7fe4480cf440 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.260+0000 7fe44d9ff700 1 -- 172.21.15.98:0/1723201646 >> 172.21.15.98:0/1723201646 conn(0x7fe448073e70 msgr2=0x7fe44806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.260+0000 7fe44d9ff700 1 -- 172.21.15.98:0/1723201646 shutdown_connections 2023-03-03T07:56:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.260+0000 7fe44d9ff700 1 -- 172.21.15.98:0/1723201646 wait complete. 2023-03-03T07:56:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.262+0000 7fe44d9ff700 1 Processor -- start 2023-03-03T07:56:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.262+0000 7fe44d9ff700 1 -- start start 2023-03-03T07:56:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.263+0000 7fe44d9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe44806b4b0 0x7fe4480cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.263+0000 7fe44d9ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4480d0270 0x7fe4480c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.263+0000 7fe44d9ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe44815ca70 0x7fe4480b7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.263+0000 7fe44d9ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4480cb2a0 con 0x7fe4480d0270 2023-03-03T07:56:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.263+0000 7fe44d9ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4481f2f90 con 0x7fe44815ca70 2023-03-03T07:56:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.263+0000 7fe44d9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4480cb4f0 con 0x7fe44806b4b0 2023-03-03T07:56:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.264+0000 7fe4467fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4480d0270 0x7fe4480c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.264+0000 7fe4467fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4480d0270 0x7fe4480c93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42164/0 (socket says 172.21.15.98:42164) 2023-03-03T07:56:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.264+0000 7fe4467fc700 1 -- 172.21.15.98:0/3433314744 learned_addr learned my addr 172.21.15.98:0/3433314744 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.264+0000 7fe446ffd700 1 --2- 172.21.15.98:0/3433314744 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe44806b4b0 0x7fe4480cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.264+0000 7fe4477fe700 1 --2- 172.21.15.98:0/3433314744 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe44815ca70 0x7fe4480b7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.264+0000 7fe4467fc700 1 -- 172.21.15.98:0/3433314744 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe44806b4b0 msgr2=0x7fe4480cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.264+0000 7fe4467fc700 1 --2- 172.21.15.98:0/3433314744 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe44806b4b0 0x7fe4480cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.264+0000 7fe4467fc700 1 -- 172.21.15.98:0/3433314744 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe44815ca70 msgr2=0x7fe4480b7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.264+0000 7fe4467fc700 1 --2- 172.21.15.98:0/3433314744 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe44815ca70 0x7fe4480b7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.264+0000 7fe4467fc700 1 -- 172.21.15.98:0/3433314744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43c0094a0 con 0x7fe4480d0270 2023-03-03T07:56:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.265+0000 7fe4467fc700 1 --2- 172.21.15.98:0/3433314744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4480d0270 0x7fe4480c93c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe43c0090b0 tx=0x7fe43c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.265+0000 7fe42ffff700 1 -- 172.21.15.98:0/3433314744 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c01a6e0 con 0x7fe4480d0270 2023-03-03T07:56:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.265+0000 7fe42ffff700 1 -- 172.21.15.98:0/3433314744 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43c01a840 con 0x7fe4480d0270 2023-03-03T07:56:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.265+0000 7fe42ffff700 1 -- 172.21.15.98:0/3433314744 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c01abb0 con 0x7fe4480d0270 2023-03-03T07:56:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.265+0000 7fe44d9ff700 1 -- 172.21.15.98:0/3433314744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4480d1aa0 con 0x7fe4480d0270 2023-03-03T07:56:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.265+0000 7fe44d9ff700 1 -- 172.21.15.98:0/3433314744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4480cdbe0 con 0x7fe4480d0270 2023-03-03T07:56:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.266+0000 7fe44d9ff700 1 -- 172.21.15.98:0/3433314744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe44815ce40 con 0x7fe4480d0270 2023-03-03T07:56:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.267+0000 7fe42ffff700 1 -- 172.21.15.98:0/3433314744 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe43c00f050 con 0x7fe4480d0270 2023-03-03T07:56:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.269+0000 7fe42ffff700 1 --2- 172.21.15.98:0/3433314744 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe4340682f0 0x7fe43406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.269+0000 7fe42ffff700 1 -- 172.21.15.98:0/3433314744 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fe43c086a70 con 0x7fe4480d0270 2023-03-03T07:56:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.270+0000 7fe446ffd700 1 --2- 172.21.15.98:0/3433314744 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe4340682f0 0x7fe43406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.270+0000 7fe446ffd700 1 --2- 172.21.15.98:0/3433314744 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe4340682f0 0x7fe43406a7a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe438001220 tx=0x7fe438009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.274+0000 7fe42ffff700 1 -- 172.21.15.98:0/3433314744 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe43c053250 con 0x7fe4480d0270 2023-03-03T07:56:36.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.466+0000 7fe44d9ff700 1 -- 172.21.15.98:0/3433314744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe44806ce10 con 0x7fe4480d0270 2023-03-03T07:56:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.467+0000 7fe42ffff700 1 -- 172.21.15.98:0/3433314744 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe43c0290a0 con 0x7fe4480d0270 2023-03-03T07:56:36.469 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:56:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.469+0000 7fe44d9ff700 1 -- 172.21.15.98:0/3433314744 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe4340682f0 msgr2=0x7fe43406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.469+0000 7fe44d9ff700 1 --2- 172.21.15.98:0/3433314744 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe4340682f0 0x7fe43406a7a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe438001220 tx=0x7fe438009040 comp rx=0 tx=0).stop 2023-03-03T07:56:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.469+0000 7fe44d9ff700 1 -- 172.21.15.98:0/3433314744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4480d0270 msgr2=0x7fe4480c93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.469+0000 7fe44d9ff700 1 --2- 172.21.15.98:0/3433314744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4480d0270 0x7fe4480c93c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe43c0090b0 tx=0x7fe43c004750 comp rx=0 tx=0).stop 2023-03-03T07:56:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.470+0000 7fe44d9ff700 1 -- 172.21.15.98:0/3433314744 shutdown_connections 2023-03-03T07:56:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.470+0000 7fe44d9ff700 1 --2- 172.21.15.98:0/3433314744 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe4340682f0 0x7fe43406a7a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.470+0000 7fe44d9ff700 1 --2- 172.21.15.98:0/3433314744 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe44815ca70 0x7fe4480b7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.470+0000 7fe44d9ff700 1 --2- 172.21.15.98:0/3433314744 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe44806b4b0 0x7fe4480cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.470+0000 7fe44d9ff700 1 --2- 172.21.15.98:0/3433314744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4480d0270 0x7fe4480c93c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.470+0000 7fe44d9ff700 1 -- 172.21.15.98:0/3433314744 >> 172.21.15.98:0/3433314744 conn(0x7fe448073e70 msgr2=0x7fe44815b190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.470+0000 7fe44d9ff700 1 -- 172.21.15.98:0/3433314744 shutdown_connections 2023-03-03T07:56:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.470+0000 7fe44d9ff700 1 -- 172.21.15.98:0/3433314744 wait complete. 2023-03-03T07:56:36.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:56:36.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.745+0000 7f18249fe700 1 Processor -- start 2023-03-03T07:56:36.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.745+0000 7f18249fe700 1 -- start start 2023-03-03T07:56:36.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.747+0000 7f18249fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f18200cf570 0x7f182015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:36.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.747+0000 7f18249fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f182015db30 0x7f18200b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:36.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.747+0000 7f18249fe700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18200cd780 con 0x7f18201609b0 2023-03-03T07:56:36.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.747+0000 7f18249fe700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18200cebb0 con 0x7f182015db30 2023-03-03T07:56:36.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.747+0000 7f18249fe700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18200cda60 con 0x7f18200cf570 2023-03-03T07:56:36.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.747+0000 7f181f59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f182015db30 0x7f18200b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:36.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.747+0000 7f181ed9d700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f18201609b0 0x7f1820160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:36998/0 (socket says 172.21.15.98:36998) 2023-03-03T07:56:36.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.747+0000 7f181ed9d700 1 -- 172.21.15.98:0/1456299526 learned_addr learned my addr 172.21.15.98:0/1456299526 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:36.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.747+0000 7f181e59c700 1 --2- 172.21.15.98:0/1456299526 >> v2:172.21.15.100:3300/0 conn(0x7f18200cf570 0x7f182015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:36.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.748+0000 7f181f59e700 1 -- 172.21.15.98:0/1456299526 >> v2:172.21.15.100:3300/0 conn(0x7f18200cf570 msgr2=0x7f182015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:36.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.748+0000 7f181f59e700 1 --2- 172.21.15.98:0/1456299526 >> v2:172.21.15.100:3300/0 conn(0x7f18200cf570 0x7f182015d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.748+0000 7f181f59e700 1 -- 172.21.15.98:0/1456299526 >> v1:172.21.15.100:6789/0 conn(0x7f18201609b0 legacy=0x7f1820160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:36.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.748+0000 7f181f59e700 1 -- 172.21.15.98:0/1456299526 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18200cdd40 con 0x7f182015db30 2023-03-03T07:56:36.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.748+0000 7f181f59e700 1 --2- 172.21.15.98:0/1456299526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f182015db30 0x7f18200b4aa0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f18200cf0c0 tx=0x7f181800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79d45b78e25d713 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:36.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.748+0000 7f181dd9b700 1 -- 172.21.15.98:0/1456299526 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181800e070 con 0x7f182015db30 2023-03-03T07:56:36.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.748+0000 7f181dd9b700 1 -- 172.21.15.98:0/1456299526 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181800bb20 con 0x7f182015db30 2023-03-03T07:56:36.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.748+0000 7f181dd9b700 1 -- 172.21.15.98:0/1456299526 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1818012430 con 0x7f182015db30 2023-03-03T07:56:36.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.749+0000 7f18249fe700 1 -- 172.21.15.98:0/1456299526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f182015db30 msgr2=0x7f18200b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:36.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.749+0000 7f18249fe700 1 --2- 172.21.15.98:0/1456299526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f182015db30 0x7f18200b4aa0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f18200cf0c0 tx=0x7f181800b330 comp rx=0 tx=0).stop 2023-03-03T07:56:36.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.749+0000 7f18249fe700 1 -- 172.21.15.98:0/1456299526 shutdown_connections 2023-03-03T07:56:36.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.749+0000 7f18249fe700 1 --2- 172.21.15.98:0/1456299526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f182015db30 0x7f18200b4aa0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.749+0000 7f18249fe700 1 --2- 172.21.15.98:0/1456299526 >> v2:172.21.15.100:3300/0 conn(0x7f18200cf570 0x7f182015d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.749+0000 7f18249fe700 1 -- 172.21.15.98:0/1456299526 >> 172.21.15.98:0/1456299526 conn(0x7f182006b880 msgr2=0x7f18200cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:36.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.749+0000 7f18249fe700 1 -- 172.21.15.98:0/1456299526 shutdown_connections 2023-03-03T07:56:36.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.749+0000 7f18249fe700 1 -- 172.21.15.98:0/1456299526 wait complete. 2023-03-03T07:56:36.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.751+0000 7f18249fe700 1 Processor -- start 2023-03-03T07:56:36.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.751+0000 7f18249fe700 1 -- start start 2023-03-03T07:56:36.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.752+0000 7f18249fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18200cf570 0x7f18200cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:36.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.752+0000 7f18249fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f182015db30 0x7f18201482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:36.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.752+0000 7f18249fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18201609b0 0x7f18200d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:36.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.752+0000 7f18249fe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18200b7430 con 0x7f18200cf570 2023-03-03T07:56:36.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.752+0000 7f18249fe700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18200b7710 con 0x7f182015db30 2023-03-03T07:56:36.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.752+0000 7f18249fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18200b79f0 con 0x7f18201609b0 2023-03-03T07:56:36.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.753+0000 7f181ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18200cf570 0x7f18200cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:36.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.753+0000 7f181ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18200cf570 0x7f18200cb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42186/0 (socket says 172.21.15.98:42186) 2023-03-03T07:56:36.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.753+0000 7f181ed9d700 1 -- 172.21.15.98:0/2243032777 learned_addr learned my addr 172.21.15.98:0/2243032777 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:36.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.753+0000 7f181e59c700 1 --2- 172.21.15.98:0/2243032777 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f182015db30 0x7f18201482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:36.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.753+0000 7f181f59e700 1 --2- 172.21.15.98:0/2243032777 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18201609b0 0x7f18200d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:36.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.753+0000 7f181ed9d700 1 -- 172.21.15.98:0/2243032777 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18201609b0 msgr2=0x7f18200d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:36.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.753+0000 7f181ed9d700 1 --2- 172.21.15.98:0/2243032777 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18201609b0 0x7f18200d17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.753+0000 7f181ed9d700 1 -- 172.21.15.98:0/2243032777 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f182015db30 msgr2=0x7f18201482e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:36.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.753+0000 7f181ed9d700 1 --2- 172.21.15.98:0/2243032777 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f182015db30 0x7f18201482e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.753+0000 7f181ed9d700 1 -- 172.21.15.98:0/2243032777 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1818009a70 con 0x7f18200cf570 2023-03-03T07:56:36.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.753+0000 7f181ed9d700 1 --2- 172.21.15.98:0/2243032777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18200cf570 0x7f18200cb020 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f181000c480 tx=0x7f181000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:36.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.754+0000 7f18077fe700 1 -- 172.21.15.98:0/2243032777 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810005d70 con 0x7f18200cf570 2023-03-03T07:56:36.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.754+0000 7f18077fe700 1 -- 172.21.15.98:0/2243032777 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18100070d0 con 0x7f18200cf570 2023-03-03T07:56:36.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.754+0000 7f18077fe700 1 -- 172.21.15.98:0/2243032777 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810007440 con 0x7f18200cf570 2023-03-03T07:56:36.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.754+0000 7f18249fe700 1 -- 172.21.15.98:0/2243032777 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182015d0d0 con 0x7f18200cf570 2023-03-03T07:56:36.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.754+0000 7f18249fe700 1 -- 172.21.15.98:0/2243032777 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182015cdf0 con 0x7f18200cf570 2023-03-03T07:56:36.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.755+0000 7f18249fe700 1 -- 172.21.15.98:0/2243032777 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182015d5f0 con 0x7f18200cf570 2023-03-03T07:56:36.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.756+0000 7f18077fe700 1 -- 172.21.15.98:0/2243032777 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1810019030 con 0x7f18200cf570 2023-03-03T07:56:36.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.757+0000 7f18077fe700 1 --2- 172.21.15.98:0/2243032777 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f180c0682f0 0x7f180c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:36.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.758+0000 7f18077fe700 1 -- 172.21.15.98:0/2243032777 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f18100850e0 con 0x7f18200cf570 2023-03-03T07:56:36.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.758+0000 7f181e59c700 1 --2- 172.21.15.98:0/2243032777 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f180c0682f0 0x7f180c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:36.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.758+0000 7f181e59c700 1 --2- 172.21.15.98:0/2243032777 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f180c0682f0 0x7f180c06a7a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f18200d2640 tx=0x7f1814008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.761+0000 7f18077fe700 1 -- 172.21.15.98:0/2243032777 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f181004b5e0 con 0x7f18200cf570 2023-03-03T07:56:36.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.953+0000 7f18249fe700 1 -- 172.21.15.98:0/2243032777 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f1820067d20 con 0x7f18200cf570 2023-03-03T07:56:36.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.953+0000 7f18077fe700 1 -- 172.21.15.98:0/2243032777 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f18100518c0 con 0x7f18200cf570 2023-03-03T07:56:36.956 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:56:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.956+0000 7f18249fe700 1 -- 172.21.15.98:0/2243032777 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f180c0682f0 msgr2=0x7f180c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.956+0000 7f18249fe700 1 --2- 172.21.15.98:0/2243032777 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f180c0682f0 0x7f180c06a7a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f18200d2640 tx=0x7f1814008040 comp rx=0 tx=0).stop 2023-03-03T07:56:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.956+0000 7f18249fe700 1 -- 172.21.15.98:0/2243032777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18200cf570 msgr2=0x7f18200cb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:36.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.956+0000 7f18249fe700 1 --2- 172.21.15.98:0/2243032777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18200cf570 0x7f18200cb020 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f181000c480 tx=0x7f181000c840 comp rx=0 tx=0).stop 2023-03-03T07:56:36.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.956+0000 7f18249fe700 1 -- 172.21.15.98:0/2243032777 shutdown_connections 2023-03-03T07:56:36.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.956+0000 7f18249fe700 1 --2- 172.21.15.98:0/2243032777 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f180c0682f0 0x7f180c06a7a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.956+0000 7f18249fe700 1 --2- 172.21.15.98:0/2243032777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18200cf570 0x7f18200cb020 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.956+0000 7f18249fe700 1 --2- 172.21.15.98:0/2243032777 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18201609b0 0x7f18200d17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.956+0000 7f18249fe700 1 --2- 172.21.15.98:0/2243032777 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f182015db30 0x7f18201482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:36.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.956+0000 7f18249fe700 1 -- 172.21.15.98:0/2243032777 >> 172.21.15.98:0/2243032777 conn(0x7f182006b880 msgr2=0x7f18200731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:36.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.956+0000 7f18249fe700 1 -- 172.21.15.98:0/2243032777 shutdown_connections 2023-03-03T07:56:36.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:36.956+0000 7f18249fe700 1 -- 172.21.15.98:0/2243032777 wait complete. 2023-03-03T07:56:36.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:56:37.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.230+0000 7f81e759e700 1 Processor -- start 2023-03-03T07:56:37.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.231+0000 7f81e759e700 1 -- start start 2023-03-03T07:56:37.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.232+0000 7f81e759e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81e80cf570 0x7f81e815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:37.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.232+0000 7f81e759e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f81e815db30 0x7f81e80b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:37.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.232+0000 7f81e759e700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e80cd780 con 0x7f81e81609b0 2023-03-03T07:56:37.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.232+0000 7f81e759e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81e80cebb0 con 0x7f81e815db30 2023-03-03T07:56:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.232+0000 7f81e759e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e80cda60 con 0x7f81e80cf570 2023-03-03T07:56:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.233+0000 7f81dffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81e80cf570 0x7f81e815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.233+0000 7f81e6d9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f81e815db30 0x7f81e80b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:37.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.233+0000 7f81e6d9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f81e815db30 0x7f81e80b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:49142/0 (socket says 172.21.15.98:49142) 2023-03-03T07:56:37.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.233+0000 7f81e6d9d700 1 -- 172.21.15.98:0/1372698412 learned_addr learned my addr 172.21.15.98:0/1372698412 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:37.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.233+0000 7f81e6d9d700 1 -- 172.21.15.98:0/1372698412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81e80cf570 msgr2=0x7f81e815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:37.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.233+0000 7f81e6d9d700 1 --2- 172.21.15.98:0/1372698412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81e80cf570 0x7f81e815d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.233+0000 7f81e6d9d700 1 -- 172.21.15.98:0/1372698412 >> v1:172.21.15.100:6789/0 conn(0x7f81e81609b0 legacy=0x7f81e8160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:37.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.233+0000 7f81e6d9d700 1 -- 172.21.15.98:0/1372698412 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e80cdd40 con 0x7f81e815db30 2023-03-03T07:56:37.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.234+0000 7f81e6d9d700 1 --2- 172.21.15.98:0/1372698412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81e815db30 0x7f81e80b4aa0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009e50 tx=0x7f81e000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f6e47dd84a7c211 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:37.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.234+0000 7f81e5d9b700 1 -- 172.21.15.98:0/1372698412 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e000e070 con 0x7f81e815db30 2023-03-03T07:56:37.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.234+0000 7f81e5d9b700 1 -- 172.21.15.98:0/1372698412 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e000b910 con 0x7f81e815db30 2023-03-03T07:56:37.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.234+0000 7f81e5d9b700 1 -- 172.21.15.98:0/1372698412 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e000bcc0 con 0x7f81e815db30 2023-03-03T07:56:37.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.234+0000 7f81e759e700 1 -- 172.21.15.98:0/1372698412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81e815db30 msgr2=0x7f81e80b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:37.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.234+0000 7f81e759e700 1 --2- 172.21.15.98:0/1372698412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81e815db30 0x7f81e80b4aa0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009e50 tx=0x7f81e000b330 comp rx=0 tx=0).stop 2023-03-03T07:56:37.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.234+0000 7f81e759e700 1 -- 172.21.15.98:0/1372698412 shutdown_connections 2023-03-03T07:56:37.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.235+0000 7f81e759e700 1 --2- 172.21.15.98:0/1372698412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81e815db30 0x7f81e80b4aa0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.235+0000 7f81e759e700 1 --2- 172.21.15.98:0/1372698412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81e80cf570 0x7f81e815d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.235+0000 7f81e759e700 1 -- 172.21.15.98:0/1372698412 >> 172.21.15.98:0/1372698412 conn(0x7f81e806b880 msgr2=0x7f81e80cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:37.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.235+0000 7f81e759e700 1 -- 172.21.15.98:0/1372698412 shutdown_connections 2023-03-03T07:56:37.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.235+0000 7f81e759e700 1 -- 172.21.15.98:0/1372698412 wait complete. 2023-03-03T07:56:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.237+0000 7f81e759e700 1 Processor -- start 2023-03-03T07:56:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.237+0000 7f81e759e700 1 -- start start 2023-03-03T07:56:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.238+0000 7f81e759e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81e80cf570 0x7f81e814e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.238+0000 7f81e759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e815db30 0x7f81e80b3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.238+0000 7f81e759e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81e81609b0 0x7f81e8150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:37.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.238+0000 7f81e759e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e80b4430 con 0x7f81e815db30 2023-03-03T07:56:37.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.238+0000 7f81e759e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e81510d0 con 0x7f81e81609b0 2023-03-03T07:56:37.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.238+0000 7f81e759e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e80b6080 con 0x7f81e80cf570 2023-03-03T07:56:37.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.238+0000 7f81dffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e815db30 0x7f81e80b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:37.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.238+0000 7f81dffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e815db30 0x7f81e80b3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42188/0 (socket says 172.21.15.98:42188) 2023-03-03T07:56:37.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.238+0000 7f81dffff700 1 -- 172.21.15.98:0/2786136606 learned_addr learned my addr 172.21.15.98:0/2786136606 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:37.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.238+0000 7f81e659c700 1 --2- 172.21.15.98:0/2786136606 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81e80cf570 0x7f81e814e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:37.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.238+0000 7f81e6d9d700 1 --2- 172.21.15.98:0/2786136606 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81e81609b0 0x7f81e8150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:37.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.239+0000 7f81dffff700 1 -- 172.21.15.98:0/2786136606 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81e80cf570 msgr2=0x7f81e814e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:37.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.239+0000 7f81dffff700 1 --2- 172.21.15.98:0/2786136606 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81e80cf570 0x7f81e814e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.239+0000 7f81dffff700 1 -- 172.21.15.98:0/2786136606 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81e81609b0 msgr2=0x7f81e8150b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:37.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.239+0000 7f81dffff700 1 --2- 172.21.15.98:0/2786136606 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81e81609b0 0x7f81e8150b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.239+0000 7f81dffff700 1 -- 172.21.15.98:0/2786136606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e0009b30 con 0x7f81e815db30 2023-03-03T07:56:37.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.239+0000 7f81dffff700 1 --2- 172.21.15.98:0/2786136606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e815db30 0x7f81e80b3490 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f81d000e9b0 tx=0x7f81d000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:37.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.239+0000 7f81deffd700 1 -- 172.21.15.98:0/2786136606 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d0009710 con 0x7f81e815db30 2023-03-03T07:56:37.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.239+0000 7f81deffd700 1 -- 172.21.15.98:0/2786136606 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d000ee50 con 0x7f81e815db30 2023-03-03T07:56:37.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.239+0000 7f81deffd700 1 -- 172.21.15.98:0/2786136606 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d000ca50 con 0x7f81e815db30 2023-03-03T07:56:37.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.239+0000 7f81e759e700 1 -- 172.21.15.98:0/2786136606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e815d0d0 con 0x7f81e815db30 2023-03-03T07:56:37.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.240+0000 7f81e759e700 1 -- 172.21.15.98:0/2786136606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e815cdf0 con 0x7f81e815db30 2023-03-03T07:56:37.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.241+0000 7f81e759e700 1 -- 172.21.15.98:0/2786136606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e80b4fe0 con 0x7f81e815db30 2023-03-03T07:56:37.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.242+0000 7f81deffd700 1 -- 172.21.15.98:0/2786136606 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f81d0019030 con 0x7f81e815db30 2023-03-03T07:56:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.244+0000 7f81deffd700 1 --2- 172.21.15.98:0/2786136606 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f81c80682f0 0x7f81c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.244+0000 7f81deffd700 1 -- 172.21.15.98:0/2786136606 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f81d0085c40 con 0x7f81e815db30 2023-03-03T07:56:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.244+0000 7f81e659c700 1 --2- 172.21.15.98:0/2786136606 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f81c80682f0 0x7f81c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.245+0000 7f81e659c700 1 --2- 172.21.15.98:0/2786136606 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f81c80682f0 0x7f81c806a7a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f81d8007fd0 tx=0x7f81d8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:37.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.248+0000 7f81deffd700 1 -- 172.21.15.98:0/2786136606 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f81d0052370 con 0x7f81e815db30 2023-03-03T07:56:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.443+0000 7f81e759e700 1 -- 172.21.15.98:0/2786136606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f81e80b5e50 con 0x7f81e815db30 2023-03-03T07:56:37.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.444+0000 7f81deffd700 1 -- 172.21.15.98:0/2786136606 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f81e80b5e50 con 0x7f81e815db30 2023-03-03T07:56:37.447 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:56:37.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.447+0000 7f81e759e700 1 -- 172.21.15.98:0/2786136606 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f81c80682f0 msgr2=0x7f81c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:37.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.447+0000 7f81e759e700 1 --2- 172.21.15.98:0/2786136606 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f81c80682f0 0x7f81c806a7a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f81d8007fd0 tx=0x7f81d8009040 comp rx=0 tx=0).stop 2023-03-03T07:56:37.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.447+0000 7f81e759e700 1 -- 172.21.15.98:0/2786136606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e815db30 msgr2=0x7f81e80b3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:37.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.447+0000 7f81e759e700 1 --2- 172.21.15.98:0/2786136606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e815db30 0x7f81e80b3490 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f81d000e9b0 tx=0x7f81d000ecc0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.448+0000 7f81e759e700 1 -- 172.21.15.98:0/2786136606 shutdown_connections 2023-03-03T07:56:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.448+0000 7f81e759e700 1 --2- 172.21.15.98:0/2786136606 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f81c80682f0 0x7f81c806a7a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.448+0000 7f81e759e700 1 --2- 172.21.15.98:0/2786136606 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81e81609b0 0x7f81e8150b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.448+0000 7f81e759e700 1 --2- 172.21.15.98:0/2786136606 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81e80cf570 0x7f81e814e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.448+0000 7f81e759e700 1 --2- 172.21.15.98:0/2786136606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e815db30 0x7f81e80b3490 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.448+0000 7f81e759e700 1 -- 172.21.15.98:0/2786136606 >> 172.21.15.98:0/2786136606 conn(0x7f81e806b880 msgr2=0x7f81e814f530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.448+0000 7f81e759e700 1 -- 172.21.15.98:0/2786136606 shutdown_connections 2023-03-03T07:56:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.448+0000 7f81e759e700 1 -- 172.21.15.98:0/2786136606 wait complete. 2023-03-03T07:56:37.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:56:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.722+0000 7efc3ffff700 1 Processor -- start 2023-03-03T07:56:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.723+0000 7efc3ffff700 1 -- start start 2023-03-03T07:56:37.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.724+0000 7efc3ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc380b26d0 0x7efc380b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:37.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.724+0000 7efc3ffff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc380cd7d0 con 0x7efc380b5140 2023-03-03T07:56:37.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.724+0000 7efc3ffff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc380b90e0 con 0x7efc380b8d10 2023-03-03T07:56:37.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.724+0000 7efc3ffff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc380b55c0 con 0x7efc380b26d0 2023-03-03T07:56:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.725+0000 7efc3d59a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc380b26d0 0x7efc380b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.725+0000 7efc3d59a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc380b26d0 0x7efc380b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42190/0 (socket says 172.21.15.98:42190) 2023-03-03T07:56:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.725+0000 7efc3d59a700 1 -- 172.21.15.98:0/1742587054 learned_addr learned my addr 172.21.15.98:0/1742587054 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.725+0000 7efc3e59c700 1 --1- 172.21.15.98:0/1742587054 >> v1:172.21.15.98:6789/0 conn(0x7efc380b5140 0x7efc380d13a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54070/0 (socket says 172.21.15.98:54070) 2023-03-03T07:56:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.725+0000 7efc3d59a700 1 -- 172.21.15.98:0/1742587054 >> v1:172.21.15.100:6789/0 conn(0x7efc380b8d10 legacy=0x7efc380cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.725+0000 7efc3d59a700 1 -- 172.21.15.98:0/1742587054 >> v1:172.21.15.98:6789/0 conn(0x7efc380b5140 legacy=0x7efc380d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.725+0000 7efc3d59a700 1 -- 172.21.15.98:0/1742587054 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc380cda20 con 0x7efc380b26d0 2023-03-03T07:56:37.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.725+0000 7efc3d59a700 1 --2- 172.21.15.98:0/1742587054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc380b26d0 0x7efc380b4c00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7efc280098f0 tx=0x7efc28009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7c790537ec3fc5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:37.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.726+0000 7efc3cd99700 1 -- 172.21.15.98:0/1742587054 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2800b8a0 con 0x7efc380b26d0 2023-03-03T07:56:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.726+0000 7efc3cd99700 1 -- 172.21.15.98:0/1742587054 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc2800ba00 con 0x7efc380b26d0 2023-03-03T07:56:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.726+0000 7efc3cd99700 1 -- 172.21.15.98:0/1742587054 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2800bd90 con 0x7efc380b26d0 2023-03-03T07:56:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.726+0000 7efc3ffff700 1 -- 172.21.15.98:0/1742587054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc380b26d0 msgr2=0x7efc380b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.726+0000 7efc3ffff700 1 --2- 172.21.15.98:0/1742587054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc380b26d0 0x7efc380b4c00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7efc280098f0 tx=0x7efc28009c00 comp rx=0 tx=0).stop 2023-03-03T07:56:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.727+0000 7efc3ffff700 1 -- 172.21.15.98:0/1742587054 shutdown_connections 2023-03-03T07:56:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.727+0000 7efc3ffff700 1 --2- 172.21.15.98:0/1742587054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc380b26d0 0x7efc380b4c00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.727+0000 7efc3ffff700 1 -- 172.21.15.98:0/1742587054 >> 172.21.15.98:0/1742587054 conn(0x7efc38073e70 msgr2=0x7efc3806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.727+0000 7efc3ffff700 1 -- 172.21.15.98:0/1742587054 shutdown_connections 2023-03-03T07:56:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.727+0000 7efc3ffff700 1 -- 172.21.15.98:0/1742587054 wait complete. 2023-03-03T07:56:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.729+0000 7efc3ffff700 1 Processor -- start 2023-03-03T07:56:37.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.730+0000 7efc3ffff700 1 -- start start 2023-03-03T07:56:37.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.731+0000 7efc3ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc380b26d0 0x7efc380c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:37.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.731+0000 7efc3ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc380b5140 0x7efc380ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:37.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.731+0000 7efc3ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc380b8d10 0x7efc380cae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:37.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.731+0000 7efc3ffff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3815dde0 con 0x7efc380b5140 2023-03-03T07:56:37.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.731+0000 7efc3ffff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc380c9000 con 0x7efc380b26d0 2023-03-03T07:56:37.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.731+0000 7efc3ffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc380cc310 con 0x7efc380b8d10 2023-03-03T07:56:37.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.732+0000 7efc3d59a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc380b5140 0x7efc380ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:37.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.732+0000 7efc3d59a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc380b5140 0x7efc380ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42196/0 (socket says 172.21.15.98:42196) 2023-03-03T07:56:37.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.732+0000 7efc3d59a700 1 -- 172.21.15.98:0/169447680 learned_addr learned my addr 172.21.15.98:0/169447680 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:37.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.732+0000 7efc3dd9b700 1 --2- 172.21.15.98:0/169447680 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc380b26d0 0x7efc380c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.732+0000 7efc3e59c700 1 --2- 172.21.15.98:0/169447680 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc380b8d10 0x7efc380cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.732+0000 7efc3d59a700 1 -- 172.21.15.98:0/169447680 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc380b8d10 msgr2=0x7efc380cae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.732+0000 7efc3d59a700 1 --2- 172.21.15.98:0/169447680 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc380b8d10 0x7efc380cae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.732+0000 7efc3d59a700 1 -- 172.21.15.98:0/169447680 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc380b26d0 msgr2=0x7efc380c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.732+0000 7efc3d59a700 1 --2- 172.21.15.98:0/169447680 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc380b26d0 0x7efc380c8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.732+0000 7efc3d59a700 1 -- 172.21.15.98:0/169447680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc28009550 con 0x7efc380b5140 2023-03-03T07:56:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.732+0000 7efc3d59a700 1 --2- 172.21.15.98:0/169447680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc380b5140 0x7efc380ca930 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7efc28009130 tx=0x7efc28004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:37.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.733+0000 7efc2e7fc700 1 -- 172.21.15.98:0/169447680 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc280038a0 con 0x7efc380b5140 2023-03-03T07:56:37.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.733+0000 7efc2e7fc700 1 -- 172.21.15.98:0/169447680 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc2801e070 con 0x7efc380b5140 2023-03-03T07:56:37.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.733+0000 7efc2e7fc700 1 -- 172.21.15.98:0/169447680 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc28003c20 con 0x7efc380b5140 2023-03-03T07:56:37.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.733+0000 7efc3ffff700 1 -- 172.21.15.98:0/169447680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc380d0e80 con 0x7efc380b5140 2023-03-03T07:56:37.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.733+0000 7efc3ffff700 1 -- 172.21.15.98:0/169447680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc380cdd00 con 0x7efc380b5140 2023-03-03T07:56:37.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.734+0000 7efc3ffff700 1 -- 172.21.15.98:0/169447680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc380cdfe0 con 0x7efc380b5140 2023-03-03T07:56:37.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.735+0000 7efc2e7fc700 1 -- 172.21.15.98:0/169447680 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efc2800f050 con 0x7efc380b5140 2023-03-03T07:56:37.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.737+0000 7efc2e7fc700 1 --2- 172.21.15.98:0/169447680 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc1c0682f0 0x7efc1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:37.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.737+0000 7efc2e7fc700 1 -- 172.21.15.98:0/169447680 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7efc28020070 con 0x7efc380b5140 2023-03-03T07:56:37.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.737+0000 7efc3dd9b700 1 --2- 172.21.15.98:0/169447680 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc1c0682f0 0x7efc1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:37.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.737+0000 7efc3dd9b700 1 --2- 172.21.15.98:0/169447680 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc1c0682f0 0x7efc1c06a7a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efc34000f60 tx=0x7efc34009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:37.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.741+0000 7efc2e7fc700 1 -- 172.21.15.98:0/169447680 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efc28053bb0 con 0x7efc380b5140 2023-03-03T07:56:37.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.932+0000 7efc3ffff700 1 -- 172.21.15.98:0/169447680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efc3804e6e0 con 0x7efc380b5140 2023-03-03T07:56:37.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.933+0000 7efc2e7fc700 1 -- 172.21.15.98:0/169447680 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7efc2802a020 con 0x7efc380b5140 2023-03-03T07:56:37.935 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:56:37.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.935+0000 7efc3ffff700 1 -- 172.21.15.98:0/169447680 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc1c0682f0 msgr2=0x7efc1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:37.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.935+0000 7efc3ffff700 1 --2- 172.21.15.98:0/169447680 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc1c0682f0 0x7efc1c06a7a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efc34000f60 tx=0x7efc34009040 comp rx=0 tx=0).stop 2023-03-03T07:56:37.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.935+0000 7efc3ffff700 1 -- 172.21.15.98:0/169447680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc380b5140 msgr2=0x7efc380ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:37.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.935+0000 7efc3ffff700 1 --2- 172.21.15.98:0/169447680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc380b5140 0x7efc380ca930 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7efc28009130 tx=0x7efc28004750 comp rx=0 tx=0).stop 2023-03-03T07:56:37.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.935+0000 7efc3ffff700 1 -- 172.21.15.98:0/169447680 shutdown_connections 2023-03-03T07:56:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.935+0000 7efc3ffff700 1 --2- 172.21.15.98:0/169447680 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc1c0682f0 0x7efc1c06a7a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.935+0000 7efc3ffff700 1 --2- 172.21.15.98:0/169447680 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc380b8d10 0x7efc380cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.935+0000 7efc3ffff700 1 --2- 172.21.15.98:0/169447680 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc380b26d0 0x7efc380c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.935+0000 7efc3ffff700 1 --2- 172.21.15.98:0/169447680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc380b5140 0x7efc380ca930 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.935+0000 7efc3ffff700 1 -- 172.21.15.98:0/169447680 >> 172.21.15.98:0/169447680 conn(0x7efc38073e70 msgr2=0x7efc380b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.936+0000 7efc3ffff700 1 -- 172.21.15.98:0/169447680 shutdown_connections 2023-03-03T07:56:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:37.936+0000 7efc3ffff700 1 -- 172.21.15.98:0/169447680 wait complete. 2023-03-03T07:56:37.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T07:56:38.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.209+0000 7fe6167ff700 1 Processor -- start 2023-03-03T07:56:38.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.209+0000 7fe6167ff700 1 -- start start 2023-03-03T07:56:38.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.210+0000 7fe6167ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6100b26d0 0x7fe6100b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:38.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.210+0000 7fe6167ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe6100b5140 0x7fe6100d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:38.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.210+0000 7fe6167ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6100cd7d0 con 0x7fe6100b8d10 2023-03-03T07:56:38.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.210+0000 7fe6167ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6100b55c0 con 0x7fe6100b26d0 2023-03-03T07:56:38.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.210+0000 7fe6167ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6100b90e0 con 0x7fe6100b5140 2023-03-03T07:56:38.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.211+0000 7fe60ffff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe6100b8d10 0x7fe6100cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54084/0 (socket says 172.21.15.98:54084) 2023-03-03T07:56:38.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.211+0000 7fe60ffff700 1 -- 172.21.15.98:0/2542323955 learned_addr learned my addr 172.21.15.98:0/2542323955 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:38.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.211+0000 7fe60f7fe700 1 --2- 172.21.15.98:0/2542323955 >> v2:172.21.15.98:3300/0 conn(0x7fe6100b26d0 0x7fe6100b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:38.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.211+0000 7fe614d9c700 1 --2- 172.21.15.98:0/2542323955 >> v2:172.21.15.100:3300/0 conn(0x7fe6100b5140 0x7fe6100d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:38.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.211+0000 7fe60f7fe700 1 -- 172.21.15.98:0/2542323955 >> v2:172.21.15.100:3300/0 conn(0x7fe6100b5140 msgr2=0x7fe6100d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:38.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.211+0000 7fe60f7fe700 1 --2- 172.21.15.98:0/2542323955 >> v2:172.21.15.100:3300/0 conn(0x7fe6100b5140 0x7fe6100d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:38.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.211+0000 7fe60f7fe700 1 -- 172.21.15.98:0/2542323955 >> v1:172.21.15.98:6789/0 conn(0x7fe6100b8d10 legacy=0x7fe6100cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:38.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.211+0000 7fe60f7fe700 1 -- 172.21.15.98:0/2542323955 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6100cda20 con 0x7fe6100b26d0 2023-03-03T07:56:38.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.212+0000 7fe60f7fe700 1 --2- 172.21.15.98:0/2542323955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6100b26d0 0x7fe6100b4c00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe6100ca1e0 tx=0x7fe604009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8a549f334a2280 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:38.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.212+0000 7fe60effd700 1 -- 172.21.15.98:0/2542323955 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604011830 con 0x7fe6100b26d0 2023-03-03T07:56:38.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.212+0000 7fe60effd700 1 -- 172.21.15.98:0/2542323955 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe604011990 con 0x7fe6100b26d0 2023-03-03T07:56:38.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.212+0000 7fe60effd700 1 -- 172.21.15.98:0/2542323955 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604011d20 con 0x7fe6100b26d0 2023-03-03T07:56:38.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.213+0000 7fe6167ff700 1 -- 172.21.15.98:0/2542323955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6100b26d0 msgr2=0x7fe6100b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:38.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.213+0000 7fe6167ff700 1 --2- 172.21.15.98:0/2542323955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6100b26d0 0x7fe6100b4c00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe6100ca1e0 tx=0x7fe604009b20 comp rx=0 tx=0).stop 2023-03-03T07:56:38.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.213+0000 7fe6167ff700 1 -- 172.21.15.98:0/2542323955 shutdown_connections 2023-03-03T07:56:38.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.213+0000 7fe6167ff700 1 --2- 172.21.15.98:0/2542323955 >> v2:172.21.15.100:3300/0 conn(0x7fe6100b5140 0x7fe6100d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:38.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.213+0000 7fe6167ff700 1 --2- 172.21.15.98:0/2542323955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6100b26d0 0x7fe6100b4c00 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:38.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.213+0000 7fe6167ff700 1 -- 172.21.15.98:0/2542323955 >> 172.21.15.98:0/2542323955 conn(0x7fe610073e70 msgr2=0x7fe61006ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:38.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.213+0000 7fe6167ff700 1 -- 172.21.15.98:0/2542323955 shutdown_connections 2023-03-03T07:56:38.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.213+0000 7fe6167ff700 1 -- 172.21.15.98:0/2542323955 wait complete. 2023-03-03T07:56:38.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.216+0000 7fe6167ff700 1 Processor -- start 2023-03-03T07:56:38.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.216+0000 7fe6167ff700 1 -- start start 2023-03-03T07:56:38.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.217+0000 7fe6167ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6100b26d0 0x7fe610152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:38.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.217+0000 7fe6167ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6100b5140 0x7fe610154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:38.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.217+0000 7fe6167ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6100b8d10 0x7fe610156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:38.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.217+0000 7fe6167ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe610155930 con 0x7fe6100b8d10 2023-03-03T07:56:38.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.217+0000 7fe6167ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe610152c50 con 0x7fe6100b26d0 2023-03-03T07:56:38.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.217+0000 7fe6167ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe610154cb0 con 0x7fe6100b5140 2023-03-03T07:56:38.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.218+0000 7fe614d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6100b8d10 0x7fe610156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:38.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.218+0000 7fe60ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6100b26d0 0x7fe610152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:38.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.218+0000 7fe60f7fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6100b5140 0x7fe610154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:38.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.218+0000 7fe614d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6100b8d10 0x7fe610156790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42216/0 (socket says 172.21.15.98:42216) 2023-03-03T07:56:38.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.218+0000 7fe614d9c700 1 -- 172.21.15.98:0/1782986120 learned_addr learned my addr 172.21.15.98:0/1782986120 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.218+0000 7fe614d9c700 1 -- 172.21.15.98:0/1782986120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6100b5140 msgr2=0x7fe610154260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.218+0000 7fe614d9c700 1 --2- 172.21.15.98:0/1782986120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6100b5140 0x7fe610154260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.218+0000 7fe614d9c700 1 -- 172.21.15.98:0/1782986120 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6100b26d0 msgr2=0x7fe610152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.218+0000 7fe614d9c700 1 --2- 172.21.15.98:0/1782986120 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6100b26d0 0x7fe610152710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.218+0000 7fe614d9c700 1 -- 172.21.15.98:0/1782986120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6040094a0 con 0x7fe6100b8d10 2023-03-03T07:56:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.219+0000 7fe614d9c700 1 --2- 172.21.15.98:0/1782986120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6100b8d10 0x7fe610156790 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe60800efc0 tx=0x7fe60800c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.219+0000 7fe60cff9700 1 -- 172.21.15.98:0/1782986120 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60800cad0 con 0x7fe6100b8d10 2023-03-03T07:56:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.219+0000 7fe60cff9700 1 -- 172.21.15.98:0/1782986120 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60800f040 con 0x7fe6100b8d10 2023-03-03T07:56:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.219+0000 7fe60cff9700 1 -- 172.21.15.98:0/1782986120 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60800ce70 con 0x7fe6100b8d10 2023-03-03T07:56:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.219+0000 7fe6167ff700 1 -- 172.21.15.98:0/1782986120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6100d0e80 con 0x7fe6100b8d10 2023-03-03T07:56:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.219+0000 7fe6167ff700 1 -- 172.21.15.98:0/1782986120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6100cdd00 con 0x7fe6100b8d10 2023-03-03T07:56:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.220+0000 7fe6167ff700 1 -- 172.21.15.98:0/1782986120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6100cdfe0 con 0x7fe6100b8d10 2023-03-03T07:56:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.221+0000 7fe60cff9700 1 -- 172.21.15.98:0/1782986120 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe608014480 con 0x7fe6100b8d10 2023-03-03T07:56:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.223+0000 7fe60cff9700 1 --2- 172.21.15.98:0/1782986120 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe5fc0682f0 0x7fe5fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.223+0000 7fe60cff9700 1 -- 172.21.15.98:0/1782986120 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fe608086650 con 0x7fe6100b8d10 2023-03-03T07:56:38.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.223+0000 7fe60ffff700 1 --2- 172.21.15.98:0/1782986120 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe5fc0682f0 0x7fe5fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:38.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.224+0000 7fe60ffff700 1 --2- 172.21.15.98:0/1782986120 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe5fc0682f0 0x7fe5fc06a7a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe600005910 tx=0x7fe600009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:38.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.226+0000 7fe60cff9700 1 -- 172.21.15.98:0/1782986120 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe608052d20 con 0x7fe6100b8d10 2023-03-03T07:56:38.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.435+0000 7fe6167ff700 1 -- 172.21.15.98:0/1782986120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe610155a70 con 0x7fe6100b8d10 2023-03-03T07:56:38.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.436+0000 7fe60cff9700 1 -- 172.21.15.98:0/1782986120 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe608056340 con 0x7fe6100b8d10 2023-03-03T07:56:38.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.438+0000 7fe6167ff700 1 -- 172.21.15.98:0/1782986120 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe5fc0682f0 msgr2=0x7fe5fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:38.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.438+0000 7fe6167ff700 1 --2- 172.21.15.98:0/1782986120 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe5fc0682f0 0x7fe5fc06a7a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe600005910 tx=0x7fe600009040 comp rx=0 tx=0).stop 2023-03-03T07:56:38.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.438+0000 7fe6167ff700 1 -- 172.21.15.98:0/1782986120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6100b8d10 msgr2=0x7fe610156790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:38.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.438+0000 7fe6167ff700 1 --2- 172.21.15.98:0/1782986120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6100b8d10 0x7fe610156790 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe60800efc0 tx=0x7fe60800c0c0 comp rx=0 tx=0).stop 2023-03-03T07:56:38.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.439+0000 7fe6167ff700 1 -- 172.21.15.98:0/1782986120 shutdown_connections 2023-03-03T07:56:38.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.439+0000 7fe6167ff700 1 --2- 172.21.15.98:0/1782986120 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe5fc0682f0 0x7fe5fc06a7a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:38.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.439+0000 7fe6167ff700 1 --2- 172.21.15.98:0/1782986120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6100b8d10 0x7fe610156790 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:38.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.439+0000 7fe6167ff700 1 --2- 172.21.15.98:0/1782986120 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6100b26d0 0x7fe610152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:38.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.439+0000 7fe6167ff700 1 --2- 172.21.15.98:0/1782986120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6100b5140 0x7fe610154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:38.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.439+0000 7fe6167ff700 1 -- 172.21.15.98:0/1782986120 >> 172.21.15.98:0/1782986120 conn(0x7fe610073e70 msgr2=0x7fe6100b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:38.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.439+0000 7fe6167ff700 1 -- 172.21.15.98:0/1782986120 shutdown_connections 2023-03-03T07:56:38.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:38.439+0000 7fe6167ff700 1 -- 172.21.15.98:0/1782986120 wait complete. 2023-03-03T07:56:38.443 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T07:56:38.468 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2023-03-03T07:56:38.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:38.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-03-03T07:56:38.570 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T07:56:38.571 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T07:56:38.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:38.572 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-03-03T07:56:38.648 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:56:38.648 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:56:38.722 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:56:38.723 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:56:38.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:56:39.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.036+0000 7f93289ff700 1 Processor -- start 2023-03-03T07:56:39.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.036+0000 7f93289ff700 1 -- start start 2023-03-03T07:56:39.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.037+0000 7f93289ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93240b87c0 0x7f93240cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:39.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.037+0000 7f93289ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f93240b5410 0x7f93240d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:39.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.038+0000 7f93289ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93240d0ef0 con 0x7f93240b8b90 2023-03-03T07:56:39.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.038+0000 7f93289ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93240d0600 con 0x7f93240b87c0 2023-03-03T07:56:39.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.038+0000 7f93289ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93240cd970 con 0x7f93240b5410 2023-03-03T07:56:39.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.038+0000 7f932259c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93240b87c0 0x7f93240cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:39.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.038+0000 7f932259c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93240b87c0 0x7f93240cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42224/0 (socket says 172.21.15.98:42224) 2023-03-03T07:56:39.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.038+0000 7f932259c700 1 -- 172.21.15.98:0/2713792758 learned_addr learned my addr 172.21.15.98:0/2713792758 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:39.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.038+0000 7f9322d9d700 1 --2- 172.21.15.98:0/2713792758 >> v2:172.21.15.100:3300/0 conn(0x7f93240b5410 0x7f93240d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:39.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.039+0000 7f932259c700 1 -- 172.21.15.98:0/2713792758 >> v2:172.21.15.100:3300/0 conn(0x7f93240b5410 msgr2=0x7f93240d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:39.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.039+0000 7f932259c700 1 --2- 172.21.15.98:0/2713792758 >> v2:172.21.15.100:3300/0 conn(0x7f93240b5410 0x7f93240d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.039+0000 7f932259c700 1 -- 172.21.15.98:0/2713792758 >> v1:172.21.15.100:6789/0 conn(0x7f93240b8b90 legacy=0x7f93240b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:39.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.039+0000 7f932259c700 1 -- 172.21.15.98:0/2713792758 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93240cdc50 con 0x7f93240b87c0 2023-03-03T07:56:39.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.039+0000 7f932159a700 1 -- 172.21.15.98:0/2713792758 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698540545 0 0) 0x7f93240d0ef0 con 0x7f93240b8b90 2023-03-03T07:56:39.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.039+0000 7f932259c700 1 --2- 172.21.15.98:0/2713792758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93240b87c0 0x7f93240cffe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f93240c6d50 tx=0x7f9314009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=840d6dfd0b2c9521 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:39.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.039+0000 7f932159a700 1 -- 172.21.15.98:0/2713792758 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314011830 con 0x7f93240b87c0 2023-03-03T07:56:39.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.039+0000 7f932159a700 1 -- 172.21.15.98:0/2713792758 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9314011990 con 0x7f93240b87c0 2023-03-03T07:56:39.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.039+0000 7f932159a700 1 -- 172.21.15.98:0/2713792758 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314011d20 con 0x7f93240b87c0 2023-03-03T07:56:39.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.040+0000 7f93289ff700 1 -- 172.21.15.98:0/2713792758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93240b87c0 msgr2=0x7f93240cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:39.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.040+0000 7f93289ff700 1 --2- 172.21.15.98:0/2713792758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93240b87c0 0x7f93240cffe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f93240c6d50 tx=0x7f9314009bd0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.040+0000 7f93289ff700 1 -- 172.21.15.98:0/2713792758 shutdown_connections 2023-03-03T07:56:39.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.040+0000 7f93289ff700 1 --2- 172.21.15.98:0/2713792758 >> v2:172.21.15.100:3300/0 conn(0x7f93240b5410 0x7f93240d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.040+0000 7f93289ff700 1 --2- 172.21.15.98:0/2713792758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93240b87c0 0x7f93240cffe0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.040+0000 7f93289ff700 1 -- 172.21.15.98:0/2713792758 >> 172.21.15.98:0/2713792758 conn(0x7f932406aee0 msgr2=0x7f932406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:39.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.040+0000 7f93289ff700 1 -- 172.21.15.98:0/2713792758 shutdown_connections 2023-03-03T07:56:39.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.040+0000 7f93289ff700 1 -- 172.21.15.98:0/2713792758 wait complete. 2023-03-03T07:56:39.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.042+0000 7f93289ff700 1 Processor -- start 2023-03-03T07:56:39.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.042+0000 7f93289ff700 1 -- start start 2023-03-03T07:56:39.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.043+0000 7f93289ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93240b5410 0x7f93240c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:39.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.043+0000 7f93289ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93240b87c0 0x7f93240c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:39.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.043+0000 7f93289ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93240b8b90 0x7f93240c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:39.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.043+0000 7f93289ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93240c8570 con 0x7f93240b87c0 2023-03-03T07:56:39.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.043+0000 7f93289ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93240c78f0 con 0x7f93240b8b90 2023-03-03T07:56:39.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.043+0000 7f93289ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93240c7bd0 con 0x7f93240b5410 2023-03-03T07:56:39.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.043+0000 7f9321d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93240b87c0 0x7f93240c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:39.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.044+0000 7f9321d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93240b87c0 0x7f93240c6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42232/0 (socket says 172.21.15.98:42232) 2023-03-03T07:56:39.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.044+0000 7f9321d9b700 1 -- 172.21.15.98:0/2127103561 learned_addr learned my addr 172.21.15.98:0/2127103561 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:39.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.044+0000 7f9322d9d700 1 --2- 172.21.15.98:0/2127103561 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93240b8b90 0x7f93240c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:39.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.044+0000 7f932259c700 1 --2- 172.21.15.98:0/2127103561 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93240b5410 0x7f93240c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:39.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.044+0000 7f9321d9b700 1 -- 172.21.15.98:0/2127103561 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93240b5410 msgr2=0x7f93240c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:39.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.044+0000 7f9321d9b700 1 --2- 172.21.15.98:0/2127103561 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93240b5410 0x7f93240c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.044+0000 7f9321d9b700 1 -- 172.21.15.98:0/2127103561 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93240b8b90 msgr2=0x7f93240c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:39.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.044+0000 7f9321d9b700 1 --2- 172.21.15.98:0/2127103561 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93240b8b90 0x7f93240c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.044+0000 7f9321d9b700 1 -- 172.21.15.98:0/2127103561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9314009550 con 0x7f93240b87c0 2023-03-03T07:56:39.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.044+0000 7f9321d9b700 1 --2- 172.21.15.98:0/2127103561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93240b87c0 0x7f93240c6d80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f931800c450 tx=0x7f931800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:39.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.044+0000 7f9312ffd700 1 -- 172.21.15.98:0/2127103561 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93180054e0 con 0x7f93240b87c0 2023-03-03T07:56:39.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.045+0000 7f9312ffd700 1 -- 172.21.15.98:0/2127103561 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931800ce60 con 0x7f93240b87c0 2023-03-03T07:56:39.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.045+0000 7f9312ffd700 1 -- 172.21.15.98:0/2127103561 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93180038a0 con 0x7f93240b87c0 2023-03-03T07:56:39.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.045+0000 7f93289ff700 1 -- 172.21.15.98:0/2127103561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93240cdf30 con 0x7f93240b87c0 2023-03-03T07:56:39.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.045+0000 7f93289ff700 1 -- 172.21.15.98:0/2127103561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9324073a80 con 0x7f93240b87c0 2023-03-03T07:56:39.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.046+0000 7f93289ff700 1 -- 172.21.15.98:0/2127103561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93240ce0c0 con 0x7f93240b87c0 2023-03-03T07:56:39.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.047+0000 7f9312ffd700 1 -- 172.21.15.98:0/2127103561 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9318019030 con 0x7f93240b87c0 2023-03-03T07:56:39.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.048+0000 7f9312ffd700 1 --2- 172.21.15.98:0/2127103561 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f93040682f0 0x7f930406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:39.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.049+0000 7f9312ffd700 1 -- 172.21.15.98:0/2127103561 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f9318084f90 con 0x7f93240b87c0 2023-03-03T07:56:39.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.049+0000 7f932259c700 1 --2- 172.21.15.98:0/2127103561 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f93040682f0 0x7f930406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:39.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.050+0000 7f932259c700 1 --2- 172.21.15.98:0/2127103561 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f93040682f0 0x7f930406a7a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9314009c10 tx=0x7f9314008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:39.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.053+0000 7f9312ffd700 1 -- 172.21.15.98:0/2127103561 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f93180516c0 con 0x7f93240b87c0 2023-03-03T07:56:39.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.246+0000 7f93289ff700 1 -- 172.21.15.98:0/2127103561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93240729b0 con 0x7f93240b87c0 2023-03-03T07:56:39.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.247+0000 7f9312ffd700 1 -- 172.21.15.98:0/2127103561 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18003 (secure 0 0 0) 0x7f9318017020 con 0x7f93240b87c0 2023-03-03T07:56:39.249 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:56:39.250 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:56:27.498205+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:56:39.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.250+0000 7f93289ff700 1 -- 172.21.15.98:0/2127103561 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f93040682f0 msgr2=0x7f930406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:39.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.250+0000 7f93289ff700 1 --2- 172.21.15.98:0/2127103561 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f93040682f0 0x7f930406a7a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9314009c10 tx=0x7f9314008030 comp rx=0 tx=0).stop 2023-03-03T07:56:39.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.250+0000 7f93289ff700 1 -- 172.21.15.98:0/2127103561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93240b87c0 msgr2=0x7f93240c6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:39.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.250+0000 7f93289ff700 1 --2- 172.21.15.98:0/2127103561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93240b87c0 0x7f93240c6d80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f931800c450 tx=0x7f931800c760 comp rx=0 tx=0).stop 2023-03-03T07:56:39.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.250+0000 7f93289ff700 1 -- 172.21.15.98:0/2127103561 shutdown_connections 2023-03-03T07:56:39.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.250+0000 7f93289ff700 1 --2- 172.21.15.98:0/2127103561 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f93040682f0 0x7f930406a7a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.250+0000 7f93289ff700 1 --2- 172.21.15.98:0/2127103561 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93240b8b90 0x7f93240c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.250+0000 7f93289ff700 1 --2- 172.21.15.98:0/2127103561 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93240b5410 0x7f93240c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.250+0000 7f93289ff700 1 --2- 172.21.15.98:0/2127103561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93240b87c0 0x7f93240c6d80 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.250+0000 7f93289ff700 1 -- 172.21.15.98:0/2127103561 >> 172.21.15.98:0/2127103561 conn(0x7f932406aee0 msgr2=0x7f93240b4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:39.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.250+0000 7f93289ff700 1 -- 172.21.15.98:0/2127103561 shutdown_connections 2023-03-03T07:56:39.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.250+0000 7f93289ff700 1 -- 172.21.15.98:0/2127103561 wait complete. 2023-03-03T07:56:39.280 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:55:45.157666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:55:46.092091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:56:39.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:56:39.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.527+0000 7fdd009fe700 1 Processor -- start 2023-03-03T07:56:39.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.527+0000 7fdd009fe700 1 -- start start 2023-03-03T07:56:39.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.529+0000 7fdd009fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdcfc0cafd0 0x7fdcfc15b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:39.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.529+0000 7fdd009fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcfc0cb3a0 0x7fdcfc0b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:39.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.529+0000 7fdd009fe700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcfc06bea0 con 0x7fdcfc0ce380 2023-03-03T07:56:39.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.529+0000 7fdd009fe700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcfc15ba30 con 0x7fdcfc0cafd0 2023-03-03T07:56:39.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.529+0000 7fdd009fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc0b2020 con 0x7fdcfc0cb3a0 2023-03-03T07:56:39.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.529+0000 7fdcfb59e700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdcfc0ce380 0x7fdcfc0b3150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54090/0 (socket says 172.21.15.98:54090) 2023-03-03T07:56:39.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.529+0000 7fdcfb59e700 1 -- 172.21.15.98:0/1536838438 learned_addr learned my addr 172.21.15.98:0/1536838438 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:39.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.529+0000 7fdcfa59c700 1 --2- 172.21.15.98:0/1536838438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcfc0cb3a0 0x7fdcfc0b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:39.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.529+0000 7fdcfad9d700 1 --2- 172.21.15.98:0/1536838438 >> v2:172.21.15.100:3300/0 conn(0x7fdcfc0cafd0 0x7fdcfc15b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:39.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.530+0000 7fdcf9d9b700 1 -- 172.21.15.98:0/1536838438 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924495590 0 0) 0x7fdcfc06bea0 con 0x7fdcfc0ce380 2023-03-03T07:56:39.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.530+0000 7fdcf9d9b700 1 -- 172.21.15.98:0/1536838438 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce40035b0 con 0x7fdcfc0ce380 2023-03-03T07:56:39.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.530+0000 7fdcfa59c700 1 -- 172.21.15.98:0/1536838438 >> v2:172.21.15.100:3300/0 conn(0x7fdcfc0cafd0 msgr2=0x7fdcfc15b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:39.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.530+0000 7fdcfa59c700 1 --2- 172.21.15.98:0/1536838438 >> v2:172.21.15.100:3300/0 conn(0x7fdcfc0cafd0 0x7fdcfc15b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.530+0000 7fdcfa59c700 1 -- 172.21.15.98:0/1536838438 >> v1:172.21.15.98:6789/0 conn(0x7fdcfc0ce380 legacy=0x7fdcfc0b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:39.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.530+0000 7fdcf9d9b700 1 -- 172.21.15.98:0/1536838438 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3444620709 0 0) 0x7fdce40035b0 con 0x7fdcfc0ce380 2023-03-03T07:56:39.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.530+0000 7fdcfa59c700 1 -- 172.21.15.98:0/1536838438 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc0b2230 con 0x7fdcfc0cb3a0 2023-03-03T07:56:39.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.530+0000 7fdcfa59c700 1 --2- 172.21.15.98:0/1536838438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcfc0cb3a0 0x7fdcfc0b1ae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0009840 tx=0x7fdcf0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8ee1109878c3840 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:39.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.531+0000 7fdcf9d9b700 1 -- 172.21.15.98:0/1536838438 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf40019f0 con 0x7fdcfc0cb3a0 2023-03-03T07:56:39.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.531+0000 7fdcf9d9b700 1 -- 172.21.15.98:0/1536838438 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf0004030 con 0x7fdcfc0cb3a0 2023-03-03T07:56:39.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.531+0000 7fdcf9d9b700 1 -- 172.21.15.98:0/1536838438 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf00043c0 con 0x7fdcfc0cb3a0 2023-03-03T07:56:39.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.531+0000 7fdd009fe700 1 -- 172.21.15.98:0/1536838438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcfc0cb3a0 msgr2=0x7fdcfc0b1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:39.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.531+0000 7fdd009fe700 1 --2- 172.21.15.98:0/1536838438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcfc0cb3a0 0x7fdcfc0b1ae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0009840 tx=0x7fdcf0009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:39.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.531+0000 7fdd009fe700 1 -- 172.21.15.98:0/1536838438 shutdown_connections 2023-03-03T07:56:39.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.531+0000 7fdd009fe700 1 --2- 172.21.15.98:0/1536838438 >> v2:172.21.15.100:3300/0 conn(0x7fdcfc0cafd0 0x7fdcfc15b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.531+0000 7fdd009fe700 1 --2- 172.21.15.98:0/1536838438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcfc0cb3a0 0x7fdcfc0b1ae0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.531+0000 7fdd009fe700 1 -- 172.21.15.98:0/1536838438 >> 172.21.15.98:0/1536838438 conn(0x7fdcfc0733c0 msgr2=0x7fdcfc06a330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:39.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.531+0000 7fdd009fe700 1 -- 172.21.15.98:0/1536838438 shutdown_connections 2023-03-03T07:56:39.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.532+0000 7fdd009fe700 1 -- 172.21.15.98:0/1536838438 wait complete. 2023-03-03T07:56:39.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.534+0000 7fdd009fe700 1 Processor -- start 2023-03-03T07:56:39.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.535+0000 7fdd009fe700 1 -- start start 2023-03-03T07:56:39.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.536+0000 7fdd009fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdcfc0cafd0 0x7fdcfc070600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:39.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.536+0000 7fdd009fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcfc0cb3a0 0x7fdcfc15e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:39.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.536+0000 7fdd009fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcfc0ce380 0x7fdcfc15a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:39.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.536+0000 7fdd009fe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc15eca0 con 0x7fdcfc0ce380 2023-03-03T07:56:39.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.536+0000 7fdd009fe700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc070c90 con 0x7fdcfc0cafd0 2023-03-03T07:56:39.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.536+0000 7fdd009fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc070f70 con 0x7fdcfc0cb3a0 2023-03-03T07:56:39.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.536+0000 7fdcfb59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcfc0ce380 0x7fdcfc15a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:39.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.537+0000 7fdcfb59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcfc0ce380 0x7fdcfc15a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42236/0 (socket says 172.21.15.98:42236) 2023-03-03T07:56:39.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.537+0000 7fdcfb59e700 1 -- 172.21.15.98:0/3378141392 learned_addr learned my addr 172.21.15.98:0/3378141392 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:39.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.537+0000 7fdcfa59c700 1 --2- 172.21.15.98:0/3378141392 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcfc0cb3a0 0x7fdcfc15e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:39.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.537+0000 7fdcfad9d700 1 --2- 172.21.15.98:0/3378141392 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdcfc0cafd0 0x7fdcfc070600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:39.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.537+0000 7fdcfb59e700 1 -- 172.21.15.98:0/3378141392 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcfc0cb3a0 msgr2=0x7fdcfc15e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:39.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.537+0000 7fdcfb59e700 1 --2- 172.21.15.98:0/3378141392 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcfc0cb3a0 0x7fdcfc15e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.537+0000 7fdcfb59e700 1 -- 172.21.15.98:0/3378141392 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdcfc0cafd0 msgr2=0x7fdcfc070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:39.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.537+0000 7fdcfb59e700 1 --2- 172.21.15.98:0/3378141392 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdcfc0cafd0 0x7fdcfc070600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.537+0000 7fdcfb59e700 1 -- 172.21.15.98:0/3378141392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf00094a0 con 0x7fdcfc0ce380 2023-03-03T07:56:39.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.537+0000 7fdcfb59e700 1 --2- 172.21.15.98:0/3378141392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcfc0ce380 0x7fdcfc15a970 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdcf400cc00 tx=0x7fdcf400cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:39.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.537+0000 7fdce37fe700 1 -- 172.21.15.98:0/3378141392 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf400aa80 con 0x7fdcfc0ce380 2023-03-03T07:56:39.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.538+0000 7fdce37fe700 1 -- 172.21.15.98:0/3378141392 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf400ac00 con 0x7fdcfc0ce380 2023-03-03T07:56:39.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.538+0000 7fdce37fe700 1 -- 172.21.15.98:0/3378141392 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf40070d0 con 0x7fdcfc0ce380 2023-03-03T07:56:39.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.538+0000 7fdd009fe700 1 -- 172.21.15.98:0/3378141392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcfc0b27f0 con 0x7fdcfc0ce380 2023-03-03T07:56:39.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.538+0000 7fdd009fe700 1 -- 172.21.15.98:0/3378141392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcfc0b2510 con 0x7fdcfc0ce380 2023-03-03T07:56:39.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.539+0000 7fdd009fe700 1 -- 172.21.15.98:0/3378141392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcfc0ce800 con 0x7fdcfc0ce380 2023-03-03T07:56:39.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.540+0000 7fdce37fe700 1 -- 172.21.15.98:0/3378141392 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdcf4010070 con 0x7fdcfc0ce380 2023-03-03T07:56:39.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.541+0000 7fdce37fe700 1 --2- 172.21.15.98:0/3378141392 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdce80682f0 0x7fdce806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:39.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.541+0000 7fdce37fe700 1 -- 172.21.15.98:0/3378141392 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fdcf4085670 con 0x7fdcfc0ce380 2023-03-03T07:56:39.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.542+0000 7fdcfad9d700 1 --2- 172.21.15.98:0/3378141392 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdce80682f0 0x7fdce806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:39.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.542+0000 7fdcfad9d700 1 --2- 172.21.15.98:0/3378141392 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdce80682f0 0x7fdce806a7a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc15a6a0 tx=0x7fdcec009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:39.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.545+0000 7fdce37fe700 1 -- 172.21.15.98:0/3378141392 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdcf4051da0 con 0x7fdcfc0ce380 2023-03-03T07:56:39.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.739+0000 7fdd009fe700 1 -- 172.21.15.98:0/3378141392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdcfc068290 con 0x7fdcfc0ce380 2023-03-03T07:56:39.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.740+0000 7fdce37fe700 1 -- 172.21.15.98:0/3378141392 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdcf4018020 con 0x7fdcfc0ce380 2023-03-03T07:56:39.742 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:56:39.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.742+0000 7fdd009fe700 1 -- 172.21.15.98:0/3378141392 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdce80682f0 msgr2=0x7fdce806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:39.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.742+0000 7fdd009fe700 1 --2- 172.21.15.98:0/3378141392 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdce80682f0 0x7fdce806a7a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc15a6a0 tx=0x7fdcec009210 comp rx=0 tx=0).stop 2023-03-03T07:56:39.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.742+0000 7fdd009fe700 1 -- 172.21.15.98:0/3378141392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcfc0ce380 msgr2=0x7fdcfc15a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:39.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.742+0000 7fdd009fe700 1 --2- 172.21.15.98:0/3378141392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcfc0ce380 0x7fdcfc15a970 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdcf400cc00 tx=0x7fdcf400cf10 comp rx=0 tx=0).stop 2023-03-03T07:56:39.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.743+0000 7fdd009fe700 1 -- 172.21.15.98:0/3378141392 shutdown_connections 2023-03-03T07:56:39.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.743+0000 7fdd009fe700 1 --2- 172.21.15.98:0/3378141392 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdce80682f0 0x7fdce806a7a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.743+0000 7fdd009fe700 1 --2- 172.21.15.98:0/3378141392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcfc0ce380 0x7fdcfc15a970 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.743+0000 7fdd009fe700 1 --2- 172.21.15.98:0/3378141392 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdcfc0cafd0 0x7fdcfc070600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.743+0000 7fdd009fe700 1 --2- 172.21.15.98:0/3378141392 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcfc0cb3a0 0x7fdcfc15e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:39.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.743+0000 7fdd009fe700 1 -- 172.21.15.98:0/3378141392 >> 172.21.15.98:0/3378141392 conn(0x7fdcfc0733c0 msgr2=0x7fdcfc06c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:39.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.743+0000 7fdd009fe700 1 -- 172.21.15.98:0/3378141392 shutdown_connections 2023-03-03T07:56:39.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:39.743+0000 7fdd009fe700 1 -- 172.21.15.98:0/3378141392 wait complete. 2023-03-03T07:56:39.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:56:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.018+0000 7f55223ff700 1 Processor -- start 2023-03-03T07:56:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.018+0000 7f55223ff700 1 -- start start 2023-03-03T07:56:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.020+0000 7f55223ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f551c0d0270 0x7f551c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.020+0000 7f55223ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551c0cf980 con 0x7f551c15ca70 2023-03-03T07:56:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.020+0000 7f55223ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551c0d1800 con 0x7f551c06b4b0 2023-03-03T07:56:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.020+0000 7f55223ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f551c0cd5b0 con 0x7f551c0d0270 2023-03-03T07:56:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.020+0000 7f551bfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f551c15ca70 0x7f551c0cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54094/0 (socket says 172.21.15.98:54094) 2023-03-03T07:56:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.020+0000 7f551bfff700 1 -- 172.21.15.98:0/2361243 learned_addr learned my addr 172.21.15.98:0/2361243 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.020+0000 7f551b7fe700 1 --2- 172.21.15.98:0/2361243 >> v2:172.21.15.100:3300/0 conn(0x7f551c0d0270 0x7f551c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.020+0000 7f551affd700 1 -- 172.21.15.98:0/2361243 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085492488 0 0) 0x7f551c0cf980 con 0x7f551c15ca70 2023-03-03T07:56:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.020+0000 7f551affd700 1 -- 172.21.15.98:0/2361243 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55040035b0 con 0x7f551c15ca70 2023-03-03T07:56:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.021+0000 7f551affd700 1 -- 172.21.15.98:0/2361243 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229943636 0 0) 0x7f551c0d1800 con 0x7f551c06b4b0 2023-03-03T07:56:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.021+0000 7f551affd700 1 -- 172.21.15.98:0/2361243 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f551c0cf980 con 0x7f551c06b4b0 2023-03-03T07:56:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.021+0000 7f551affd700 1 -- 172.21.15.98:0/2361243 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3456239480 0 0) 0x7f55040035b0 con 0x7f551c15ca70 2023-03-03T07:56:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.021+0000 7f551b7fe700 1 -- 172.21.15.98:0/2361243 >> v1:172.21.15.100:6789/0 conn(0x7f551c06b4b0 legacy=0x7f551c0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.021+0000 7f551b7fe700 1 -- 172.21.15.98:0/2361243 >> v1:172.21.15.98:6789/0 conn(0x7f551c15ca70 legacy=0x7f551c0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.021+0000 7f551b7fe700 1 -- 172.21.15.98:0/2361243 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551c0cd900 con 0x7f551c0d0270 2023-03-03T07:56:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.021+0000 7f551b7fe700 1 --2- 172.21.15.98:0/2361243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f551c0d0270 0x7f551c0cf440 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5510009840 tx=0x7f5510009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ba2110fcb9637c0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.022+0000 7f551affd700 1 -- 172.21.15.98:0/2361243 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c001670 con 0x7f551c0d0270 2023-03-03T07:56:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.022+0000 7f551affd700 1 -- 172.21.15.98:0/2361243 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5510004030 con 0x7f551c0d0270 2023-03-03T07:56:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.022+0000 7f551affd700 1 -- 172.21.15.98:0/2361243 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55100043a0 con 0x7f551c0d0270 2023-03-03T07:56:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.022+0000 7f55223ff700 1 -- 172.21.15.98:0/2361243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f551c0d0270 msgr2=0x7f551c0cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.022+0000 7f55223ff700 1 --2- 172.21.15.98:0/2361243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f551c0d0270 0x7f551c0cf440 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5510009840 tx=0x7f5510009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.023+0000 7f55223ff700 1 -- 172.21.15.98:0/2361243 shutdown_connections 2023-03-03T07:56:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.023+0000 7f55223ff700 1 --2- 172.21.15.98:0/2361243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f551c0d0270 0x7f551c0cf440 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.023+0000 7f55223ff700 1 -- 172.21.15.98:0/2361243 >> 172.21.15.98:0/2361243 conn(0x7f551c073e70 msgr2=0x7f551c06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.023+0000 7f55223ff700 1 -- 172.21.15.98:0/2361243 shutdown_connections 2023-03-03T07:56:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.023+0000 7f55223ff700 1 -- 172.21.15.98:0/2361243 wait complete. 2023-03-03T07:56:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.025+0000 7f55223ff700 1 Processor -- start 2023-03-03T07:56:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.025+0000 7f55223ff700 1 -- start start 2023-03-03T07:56:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.026+0000 7f55223ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f551c06b4b0 0x7f551c0c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.026+0000 7f55223ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f551c0d0270 0x7f551c15fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.026+0000 7f55223ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f551c15ca70 0x7f551c1f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.026+0000 7f55223ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c15ffc0 con 0x7f551c15ca70 2023-03-03T07:56:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.026+0000 7f55223ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c15d200 con 0x7f551c06b4b0 2023-03-03T07:56:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.026+0000 7f55223ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f551c152b40 con 0x7f551c0d0270 2023-03-03T07:56:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.026+0000 7f552099c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f551c15ca70 0x7f551c1f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.026+0000 7f552099c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f551c15ca70 0x7f551c1f22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42244/0 (socket says 172.21.15.98:42244) 2023-03-03T07:56:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.026+0000 7f552099c700 1 -- 172.21.15.98:0/2623672144 learned_addr learned my addr 172.21.15.98:0/2623672144 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.026+0000 7f551bfff700 1 --2- 172.21.15.98:0/2623672144 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f551c06b4b0 0x7f551c0c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.026+0000 7f551b7fe700 1 --2- 172.21.15.98:0/2623672144 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f551c0d0270 0x7f551c15fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.027+0000 7f552099c700 1 -- 172.21.15.98:0/2623672144 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f551c0d0270 msgr2=0x7f551c15fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.027+0000 7f552099c700 1 --2- 172.21.15.98:0/2623672144 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f551c0d0270 0x7f551c15fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.027+0000 7f552099c700 1 -- 172.21.15.98:0/2623672144 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f551c06b4b0 msgr2=0x7f551c0c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.027+0000 7f552099c700 1 --2- 172.21.15.98:0/2623672144 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f551c06b4b0 0x7f551c0c8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.027+0000 7f552099c700 1 -- 172.21.15.98:0/2623672144 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55100094a0 con 0x7f551c15ca70 2023-03-03T07:56:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.027+0000 7f552099c700 1 --2- 172.21.15.98:0/2623672144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f551c15ca70 0x7f551c1f22b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f551400cc50 tx=0x7f551400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.027+0000 7f5518ff9700 1 -- 172.21.15.98:0/2623672144 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5514003c00 con 0x7f551c15ca70 2023-03-03T07:56:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.027+0000 7f5518ff9700 1 -- 172.21.15.98:0/2623672144 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5514003d60 con 0x7f551c15ca70 2023-03-03T07:56:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.027+0000 7f5518ff9700 1 -- 172.21.15.98:0/2623672144 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551400aad0 con 0x7f551c15ca70 2023-03-03T07:56:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.027+0000 7f55223ff700 1 -- 172.21.15.98:0/2623672144 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551c0d1aa0 con 0x7f551c15ca70 2023-03-03T07:56:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.027+0000 7f55223ff700 1 -- 172.21.15.98:0/2623672144 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551c0cdbe0 con 0x7f551c15ca70 2023-03-03T07:56:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.028+0000 7f55223ff700 1 -- 172.21.15.98:0/2623672144 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f551c15ce40 con 0x7f551c15ca70 2023-03-03T07:56:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.030+0000 7f5518ff9700 1 -- 172.21.15.98:0/2623672144 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5514010050 con 0x7f551c15ca70 2023-03-03T07:56:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.031+0000 7f5518ff9700 1 --2- 172.21.15.98:0/2623672144 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f55080682f0 0x7f550806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.031+0000 7f5518ff9700 1 -- 172.21.15.98:0/2623672144 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f5514085ce0 con 0x7f551c15ca70 2023-03-03T07:56:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.031+0000 7f551bfff700 1 --2- 172.21.15.98:0/2623672144 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f55080682f0 0x7f550806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.032+0000 7f551bfff700 1 --2- 172.21.15.98:0/2623672144 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f55080682f0 0x7f550806a7a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f550c001380 tx=0x7f550c007b80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:40.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.035+0000 7f5518ff9700 1 -- 172.21.15.98:0/2623672144 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5514052410 con 0x7f551c15ca70 2023-03-03T07:56:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.229+0000 7f55223ff700 1 -- 172.21.15.98:0/2623672144 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f551c04e6e0 con 0x7f551c15ca70 2023-03-03T07:56:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.230+0000 7f5518ff9700 1 -- 172.21.15.98:0/2623672144 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5514055a30 con 0x7f551c15ca70 2023-03-03T07:56:40.233 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:56:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.233+0000 7f55223ff700 1 -- 172.21.15.98:0/2623672144 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f55080682f0 msgr2=0x7f550806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.233+0000 7f55223ff700 1 --2- 172.21.15.98:0/2623672144 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f55080682f0 0x7f550806a7a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f550c001380 tx=0x7f550c007b80 comp rx=0 tx=0).stop 2023-03-03T07:56:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.233+0000 7f55223ff700 1 -- 172.21.15.98:0/2623672144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f551c15ca70 msgr2=0x7f551c1f22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.233+0000 7f55223ff700 1 --2- 172.21.15.98:0/2623672144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f551c15ca70 0x7f551c1f22b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f551400cc50 tx=0x7f551400cf60 comp rx=0 tx=0).stop 2023-03-03T07:56:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.233+0000 7f55223ff700 1 -- 172.21.15.98:0/2623672144 shutdown_connections 2023-03-03T07:56:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.233+0000 7f55223ff700 1 --2- 172.21.15.98:0/2623672144 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f55080682f0 0x7f550806a7a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.233+0000 7f55223ff700 1 --2- 172.21.15.98:0/2623672144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f551c15ca70 0x7f551c1f22b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.233+0000 7f55223ff700 1 --2- 172.21.15.98:0/2623672144 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f551c06b4b0 0x7f551c0c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.233+0000 7f55223ff700 1 --2- 172.21.15.98:0/2623672144 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f551c0d0270 0x7f551c15fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.233+0000 7f55223ff700 1 -- 172.21.15.98:0/2623672144 >> 172.21.15.98:0/2623672144 conn(0x7f551c073e70 msgr2=0x7f551c0b6350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.233+0000 7f55223ff700 1 -- 172.21.15.98:0/2623672144 shutdown_connections 2023-03-03T07:56:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.233+0000 7f55223ff700 1 -- 172.21.15.98:0/2623672144 wait complete. 2023-03-03T07:56:40.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:56:40.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.510+0000 7f34039ff700 1 Processor -- start 2023-03-03T07:56:40.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.510+0000 7f34039ff700 1 -- start start 2023-03-03T07:56:40.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.511+0000 7f34039ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33fc0b8d10 0x7f33fc0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:40.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.511+0000 7f34039ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33fc0cd7d0 con 0x7f33fc0b5310 2023-03-03T07:56:40.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.511+0000 7f34039ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33fc0b90e0 con 0x7f33fc0b26d0 2023-03-03T07:56:40.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.511+0000 7f34039ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc0cda20 con 0x7f33fc0b8d10 2023-03-03T07:56:40.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.512+0000 7f3401f9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f33fc0b5310 0x7f33fc0d13a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54104/0 (socket says 172.21.15.98:54104) 2023-03-03T07:56:40.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.512+0000 7f3401f9c700 1 -- 172.21.15.98:0/3928757165 learned_addr learned my addr 172.21.15.98:0/3928757165 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:40.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.512+0000 7f340179b700 1 --2- 172.21.15.98:0/3928757165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33fc0b8d10 0x7f33fc0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:40.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.512+0000 7f33f3fff700 1 -- 172.21.15.98:0/3928757165 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873721718 0 0) 0x7f33fc0cd7d0 con 0x7f33fc0b5310 2023-03-03T07:56:40.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.512+0000 7f33f3fff700 1 -- 172.21.15.98:0/3928757165 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e80035b0 con 0x7f33fc0b5310 2023-03-03T07:56:40.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.513+0000 7f340179b700 1 -- 172.21.15.98:0/3928757165 >> v1:172.21.15.100:6789/0 conn(0x7f33fc0b26d0 legacy=0x7f33fc0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:40.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.513+0000 7f33f3fff700 1 -- 172.21.15.98:0/3928757165 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1398767397 0 0) 0x7f33e80035b0 con 0x7f33fc0b5310 2023-03-03T07:56:40.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.513+0000 7f340179b700 1 -- 172.21.15.98:0/3928757165 >> v1:172.21.15.98:6789/0 conn(0x7f33fc0b5310 legacy=0x7f33fc0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:40.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.513+0000 7f340179b700 1 -- 172.21.15.98:0/3928757165 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33fc0cdd00 con 0x7f33fc0b8d10 2023-03-03T07:56:40.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.513+0000 7f340179b700 1 --2- 172.21.15.98:0/3928757165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33fc0b8d10 0x7f33fc0cffc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f33f8009870 tx=0x7f33f8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9220406d3e4d0996 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:40.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.514+0000 7f33f3fff700 1 -- 172.21.15.98:0/3928757165 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f40019f0 con 0x7f33fc0b8d10 2023-03-03T07:56:40.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.514+0000 7f33f3fff700 1 -- 172.21.15.98:0/3928757165 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f8004030 con 0x7f33fc0b8d10 2023-03-03T07:56:40.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.514+0000 7f33f3fff700 1 -- 172.21.15.98:0/3928757165 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f80043c0 con 0x7f33fc0b8d10 2023-03-03T07:56:40.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.514+0000 7f34039ff700 1 -- 172.21.15.98:0/3928757165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33fc0b8d10 msgr2=0x7f33fc0cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:40.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.514+0000 7f34039ff700 1 --2- 172.21.15.98:0/3928757165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33fc0b8d10 0x7f33fc0cffc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f33f8009870 tx=0x7f33f8009b80 comp rx=0 tx=0).stop 2023-03-03T07:56:40.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.514+0000 7f34039ff700 1 -- 172.21.15.98:0/3928757165 shutdown_connections 2023-03-03T07:56:40.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.514+0000 7f34039ff700 1 --2- 172.21.15.98:0/3928757165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33fc0b8d10 0x7f33fc0cffc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:40.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.514+0000 7f34039ff700 1 -- 172.21.15.98:0/3928757165 >> 172.21.15.98:0/3928757165 conn(0x7f33fc073e70 msgr2=0x7f33fc06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:40.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.515+0000 7f34039ff700 1 -- 172.21.15.98:0/3928757165 shutdown_connections 2023-03-03T07:56:40.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.515+0000 7f34039ff700 1 -- 172.21.15.98:0/3928757165 wait complete. 2023-03-03T07:56:40.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.516+0000 7f34039ff700 1 Processor -- start 2023-03-03T07:56:40.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.517+0000 7f34039ff700 1 -- start start 2023-03-03T07:56:40.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f34039ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33fc0b26d0 0x7f33fc0cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:40.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f34039ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33fc0b5310 0x7f33fc14adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:40.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f34039ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc0b8d10 0x7f33fc0c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:40.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f34039ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc0cd7c0 con 0x7f33fc0b8d10 2023-03-03T07:56:40.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f34039ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc14b2f0 con 0x7f33fc0b26d0 2023-03-03T07:56:40.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f34039ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc0c9800 con 0x7f33fc0b5310 2023-03-03T07:56:40.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f3401f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc0b8d10 0x7f33fc0c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:40.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f3401f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc0b8d10 0x7f33fc0c92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42258/0 (socket says 172.21.15.98:42258) 2023-03-03T07:56:40.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f3401f9c700 1 -- 172.21.15.98:0/374116946 learned_addr learned my addr 172.21.15.98:0/374116946 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:40.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f340179b700 1 --2- 172.21.15.98:0/374116946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33fc0b26d0 0x7f33fc0cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:40.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f3400f9a700 1 --2- 172.21.15.98:0/374116946 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33fc0b5310 0x7f33fc14adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:40.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f3401f9c700 1 -- 172.21.15.98:0/374116946 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33fc0b5310 msgr2=0x7f33fc14adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:40.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f3401f9c700 1 --2- 172.21.15.98:0/374116946 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33fc0b5310 0x7f33fc14adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:40.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f3401f9c700 1 -- 172.21.15.98:0/374116946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33fc0b26d0 msgr2=0x7f33fc0cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:40.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f3401f9c700 1 --2- 172.21.15.98:0/374116946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33fc0b26d0 0x7f33fc0cd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:40.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f3401f9c700 1 -- 172.21.15.98:0/374116946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f80094a0 con 0x7f33fc0b8d10 2023-03-03T07:56:40.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.518+0000 7f3400f9a700 1 --2- 172.21.15.98:0/374116946 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33fc0b5310 0x7f33fc14adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:40.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.519+0000 7f3401f9c700 1 --2- 172.21.15.98:0/374116946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc0b8d10 0x7f33fc0c92c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f33f400cbe0 tx=0x7f33f400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:40.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.519+0000 7f33f1ffb700 1 -- 172.21.15.98:0/374116946 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f400ab50 con 0x7f33fc0b8d10 2023-03-03T07:56:40.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.519+0000 7f33f1ffb700 1 -- 172.21.15.98:0/374116946 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f400acb0 con 0x7f33fc0b8d10 2023-03-03T07:56:40.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.519+0000 7f33f1ffb700 1 -- 172.21.15.98:0/374116946 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f4007310 con 0x7f33fc0b8d10 2023-03-03T07:56:40.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.519+0000 7f34039ff700 1 -- 172.21.15.98:0/374116946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33fc0cdfe0 con 0x7f33fc0b8d10 2023-03-03T07:56:40.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.519+0000 7f34039ff700 1 -- 172.21.15.98:0/374116946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33fc0d0e80 con 0x7f33fc0b8d10 2023-03-03T07:56:40.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.520+0000 7f34039ff700 1 -- 172.21.15.98:0/374116946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33fc0d0500 con 0x7f33fc0b8d10 2023-03-03T07:56:40.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.522+0000 7f33f1ffb700 1 -- 172.21.15.98:0/374116946 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f33f4010070 con 0x7f33fc0b8d10 2023-03-03T07:56:40.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.523+0000 7f33f1ffb700 1 --2- 172.21.15.98:0/374116946 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33e00682f0 0x7f33e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:40.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.523+0000 7f33f1ffb700 1 -- 172.21.15.98:0/374116946 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f33f4085740 con 0x7f33fc0b8d10 2023-03-03T07:56:40.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.523+0000 7f340179b700 1 --2- 172.21.15.98:0/374116946 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33e00682f0 0x7f33e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:40.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.524+0000 7f340179b700 1 --2- 172.21.15.98:0/374116946 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33e00682f0 0x7f33e006a7a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f33f80097c0 tx=0x7f33f8008000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:40.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.526+0000 7f33f1ffb700 1 -- 172.21.15.98:0/374116946 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f33f4051e70 con 0x7f33fc0b8d10 2023-03-03T07:56:40.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.723+0000 7f34039ff700 1 -- 172.21.15.98:0/374116946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f33fc0ceb30 con 0x7f33fc0b8d10 2023-03-03T07:56:40.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.724+0000 7f33f1ffb700 1 -- 172.21.15.98:0/374116946 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f33f4055490 con 0x7f33fc0b8d10 2023-03-03T07:56:40.726 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:56:40.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.726+0000 7f34039ff700 1 -- 172.21.15.98:0/374116946 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33e00682f0 msgr2=0x7f33e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:40.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.726+0000 7f34039ff700 1 --2- 172.21.15.98:0/374116946 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33e00682f0 0x7f33e006a7a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f33f80097c0 tx=0x7f33f8008000 comp rx=0 tx=0).stop 2023-03-03T07:56:40.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.726+0000 7f34039ff700 1 -- 172.21.15.98:0/374116946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc0b8d10 msgr2=0x7f33fc0c92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:40.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.726+0000 7f34039ff700 1 --2- 172.21.15.98:0/374116946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc0b8d10 0x7f33fc0c92c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f33f400cbe0 tx=0x7f33f400cef0 comp rx=0 tx=0).stop 2023-03-03T07:56:40.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.726+0000 7f34039ff700 1 -- 172.21.15.98:0/374116946 shutdown_connections 2023-03-03T07:56:40.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.726+0000 7f34039ff700 1 --2- 172.21.15.98:0/374116946 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33e00682f0 0x7f33e006a7a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:40.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.726+0000 7f34039ff700 1 --2- 172.21.15.98:0/374116946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc0b8d10 0x7f33fc0c92c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:40.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.726+0000 7f34039ff700 1 --2- 172.21.15.98:0/374116946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33fc0b26d0 0x7f33fc0cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:40.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.726+0000 7f34039ff700 1 --2- 172.21.15.98:0/374116946 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33fc0b5310 0x7f33fc14adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:40.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.726+0000 7f34039ff700 1 -- 172.21.15.98:0/374116946 >> 172.21.15.98:0/374116946 conn(0x7f33fc073e70 msgr2=0x7f33fc15b850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:40.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.727+0000 7f34039ff700 1 -- 172.21.15.98:0/374116946 shutdown_connections 2023-03-03T07:56:40.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:40.727+0000 7f34039ff700 1 -- 172.21.15.98:0/374116946 wait complete. 2023-03-03T07:56:40.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:56:41.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.001+0000 7f24c09ff700 1 Processor -- start 2023-03-03T07:56:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.001+0000 7f24c09ff700 1 -- start start 2023-03-03T07:56:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.002+0000 7f24c09ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24bc0b8d10 0x7f24bc0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.002+0000 7f24c09ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f24bc0b26d0 0x7f24bc0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.002+0000 7f24c09ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f24bc0b5140 0x7f24bc160520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.002+0000 7f24c09ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24bc0b55c0 con 0x7f24bc0b26d0 2023-03-03T07:56:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.002+0000 7f24c09ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24bc0cd7d0 con 0x7f24bc0b5140 2023-03-03T07:56:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.002+0000 7f24c09ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc0b90e0 con 0x7f24bc0b8d10 2023-03-03T07:56:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.003+0000 7f24b9d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f24bc0b26d0 0x7f24bc0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.003+0000 7f24b9d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f24bc0b26d0 0x7f24bc0b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42270/0 (socket says 172.21.15.98:42270) 2023-03-03T07:56:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.003+0000 7f24b9d9b700 1 -- 172.21.15.98:0/1352732513 learned_addr learned my addr 172.21.15.98:0/1352732513 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.003+0000 7f24bad9d700 1 --2- 172.21.15.98:0/1352732513 >> v2:172.21.15.100:3300/0 conn(0x7f24bc0b5140 0x7f24bc160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.003+0000 7f24ba59c700 1 --2- 172.21.15.98:0/1352732513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24bc0b8d10 0x7f24bc0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:41.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.003+0000 7f24b9d9b700 1 -- 172.21.15.98:0/1352732513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24bc0b8d10 msgr2=0x7f24bc0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:41.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.003+0000 7f24b9d9b700 1 --2- 172.21.15.98:0/1352732513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24bc0b8d10 0x7f24bc0cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.003+0000 7f24b9d9b700 1 -- 172.21.15.98:0/1352732513 >> v2:172.21.15.100:3300/0 conn(0x7f24bc0b5140 msgr2=0x7f24bc160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:41.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.003+0000 7f24b9d9b700 1 --2- 172.21.15.98:0/1352732513 >> v2:172.21.15.100:3300/0 conn(0x7f24bc0b5140 0x7f24bc160520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.003+0000 7f24b9d9b700 1 -- 172.21.15.98:0/1352732513 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24bc0cda20 con 0x7f24bc0b26d0 2023-03-03T07:56:41.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.004+0000 7f24b9d9b700 1 --2- 172.21.15.98:0/1352732513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24bc0b26d0 0x7f24bc0b4c00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f24bc08c5e0 tx=0x7f24b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fead1172adf8bb9d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:41.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.004+0000 7f24b959a700 1 -- 172.21.15.98:0/1352732513 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b000b870 con 0x7f24bc0b26d0 2023-03-03T07:56:41.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.004+0000 7f24b959a700 1 -- 172.21.15.98:0/1352732513 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b000b9d0 con 0x7f24bc0b26d0 2023-03-03T07:56:41.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.004+0000 7f24b959a700 1 -- 172.21.15.98:0/1352732513 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b000bd60 con 0x7f24bc0b26d0 2023-03-03T07:56:41.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.005+0000 7f24c09ff700 1 -- 172.21.15.98:0/1352732513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24bc0b26d0 msgr2=0x7f24bc0b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:41.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.005+0000 7f24c09ff700 1 --2- 172.21.15.98:0/1352732513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24bc0b26d0 0x7f24bc0b4c00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f24bc08c5e0 tx=0x7f24b0009bd0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.005+0000 7f24c09ff700 1 -- 172.21.15.98:0/1352732513 shutdown_connections 2023-03-03T07:56:41.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.005+0000 7f24c09ff700 1 --2- 172.21.15.98:0/1352732513 >> v2:172.21.15.100:3300/0 conn(0x7f24bc0b5140 0x7f24bc160520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.005+0000 7f24c09ff700 1 --2- 172.21.15.98:0/1352732513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24bc0b8d10 0x7f24bc0cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.005+0000 7f24c09ff700 1 --2- 172.21.15.98:0/1352732513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24bc0b26d0 0x7f24bc0b4c00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.005+0000 7f24c09ff700 1 -- 172.21.15.98:0/1352732513 >> 172.21.15.98:0/1352732513 conn(0x7f24bc073e70 msgr2=0x7f24bc06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:41.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.005+0000 7f24c09ff700 1 -- 172.21.15.98:0/1352732513 shutdown_connections 2023-03-03T07:56:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.005+0000 7f24c09ff700 1 -- 172.21.15.98:0/1352732513 wait complete. 2023-03-03T07:56:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.007+0000 7f24c09ff700 1 Processor -- start 2023-03-03T07:56:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.007+0000 7f24c09ff700 1 -- start start 2023-03-03T07:56:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.008+0000 7f24c09ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24bc0b26d0 0x7f24bc151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.008+0000 7f24c09ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24bc0b5140 0x7f24bc156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.008+0000 7f24c09ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24bc0b8d10 0x7f24bc0b6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.008+0000 7f24c09ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc1580b0 con 0x7f24bc0b26d0 2023-03-03T07:56:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.008+0000 7f24c09ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc157010 con 0x7f24bc0b8d10 2023-03-03T07:56:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.008+0000 7f24c09ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc1572f0 con 0x7f24bc0b5140 2023-03-03T07:56:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.009+0000 7f24ba59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24bc0b26d0 0x7f24bc151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.009+0000 7f24ba59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24bc0b26d0 0x7f24bc151a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42276/0 (socket says 172.21.15.98:42276) 2023-03-03T07:56:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.009+0000 7f24ba59c700 1 -- 172.21.15.98:0/1660023655 learned_addr learned my addr 172.21.15.98:0/1660023655 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.009+0000 7f24b9d9b700 1 --2- 172.21.15.98:0/1660023655 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24bc0b5140 0x7f24bc156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.009+0000 7f24bad9d700 1 --2- 172.21.15.98:0/1660023655 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24bc0b8d10 0x7f24bc0b6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.009+0000 7f24ba59c700 1 -- 172.21.15.98:0/1660023655 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24bc0b5140 msgr2=0x7f24bc156750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.009+0000 7f24ba59c700 1 --2- 172.21.15.98:0/1660023655 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24bc0b5140 0x7f24bc156750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.009+0000 7f24ba59c700 1 -- 172.21.15.98:0/1660023655 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24bc0b8d10 msgr2=0x7f24bc0b6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.009+0000 7f24ba59c700 1 --2- 172.21.15.98:0/1660023655 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24bc0b8d10 0x7f24bc0b6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.009+0000 7f24ba59c700 1 -- 172.21.15.98:0/1660023655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b0009550 con 0x7f24bc0b26d0 2023-03-03T07:56:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.009+0000 7f24bad9d700 1 --2- 172.21.15.98:0/1660023655 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24bc0b8d10 0x7f24bc0b6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:56:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.009+0000 7f24ba59c700 1 --2- 172.21.15.98:0/1660023655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24bc0b26d0 0x7f24bc151a20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f24ac00d800 tx=0x7f24ac00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.010+0000 7f24aaffd700 1 -- 172.21.15.98:0/1660023655 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0125f0 con 0x7f24bc0b26d0 2023-03-03T07:56:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.010+0000 7f24aaffd700 1 -- 172.21.15.98:0/1660023655 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ac012750 con 0x7f24bc0b26d0 2023-03-03T07:56:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.010+0000 7f24c09ff700 1 -- 172.21.15.98:0/1660023655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24bc0d0e80 con 0x7f24bc0b26d0 2023-03-03T07:56:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.010+0000 7f24aaffd700 1 -- 172.21.15.98:0/1660023655 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac012b00 con 0x7f24bc0b26d0 2023-03-03T07:56:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.010+0000 7f24c09ff700 1 -- 172.21.15.98:0/1660023655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24bc0cdd00 con 0x7f24bc0b26d0 2023-03-03T07:56:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.011+0000 7f24c09ff700 1 -- 172.21.15.98:0/1660023655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24bc0cdfe0 con 0x7f24bc0b26d0 2023-03-03T07:56:41.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.012+0000 7f24aaffd700 1 -- 172.21.15.98:0/1660023655 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f24ac012c60 con 0x7f24bc0b26d0 2023-03-03T07:56:41.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.013+0000 7f24aaffd700 1 --2- 172.21.15.98:0/1660023655 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f249c0682f0 0x7f249c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:41.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.013+0000 7f24aaffd700 1 -- 172.21.15.98:0/1660023655 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f24ac086280 con 0x7f24bc0b26d0 2023-03-03T07:56:41.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.014+0000 7f24b9d9b700 1 --2- 172.21.15.98:0/1660023655 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f249c0682f0 0x7f249c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:41.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.015+0000 7f24b9d9b700 1 --2- 172.21.15.98:0/1660023655 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f249c0682f0 0x7f249c06a7a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f24b0014040 tx=0x7f24b0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:41.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.018+0000 7f24aaffd700 1 -- 172.21.15.98:0/1660023655 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f24ac0529b0 con 0x7f24bc0b26d0 2023-03-03T07:56:41.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.212+0000 7f24c09ff700 1 -- 172.21.15.98:0/1660023655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f24bc04e6e0 con 0x7f24bc0b26d0 2023-03-03T07:56:41.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.213+0000 7f24aaffd700 1 -- 172.21.15.98:0/1660023655 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f24ac055fd0 con 0x7f24bc0b26d0 2023-03-03T07:56:41.215 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:56:41.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.215+0000 7f24c09ff700 1 -- 172.21.15.98:0/1660023655 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f249c0682f0 msgr2=0x7f249c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:41.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.215+0000 7f24c09ff700 1 --2- 172.21.15.98:0/1660023655 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f249c0682f0 0x7f249c06a7a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f24b0014040 tx=0x7f24b0008030 comp rx=0 tx=0).stop 2023-03-03T07:56:41.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.215+0000 7f24c09ff700 1 -- 172.21.15.98:0/1660023655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24bc0b26d0 msgr2=0x7f24bc151a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:41.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.215+0000 7f24c09ff700 1 --2- 172.21.15.98:0/1660023655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24bc0b26d0 0x7f24bc151a20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f24ac00d800 tx=0x7f24ac00dbc0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.215+0000 7f24c09ff700 1 -- 172.21.15.98:0/1660023655 shutdown_connections 2023-03-03T07:56:41.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.215+0000 7f24c09ff700 1 --2- 172.21.15.98:0/1660023655 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f249c0682f0 0x7f249c06a7a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.215+0000 7f24c09ff700 1 --2- 172.21.15.98:0/1660023655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24bc0b26d0 0x7f24bc151a20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.215+0000 7f24c09ff700 1 --2- 172.21.15.98:0/1660023655 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24bc0b8d10 0x7f24bc0b6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.215+0000 7f24c09ff700 1 --2- 172.21.15.98:0/1660023655 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24bc0b5140 0x7f24bc156750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.215+0000 7f24c09ff700 1 -- 172.21.15.98:0/1660023655 >> 172.21.15.98:0/1660023655 conn(0x7f24bc073e70 msgr2=0x7f24bc0cfc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:41.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.215+0000 7f24c09ff700 1 -- 172.21.15.98:0/1660023655 shutdown_connections 2023-03-03T07:56:41.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.215+0000 7f24c09ff700 1 -- 172.21.15.98:0/1660023655 wait complete. 2023-03-03T07:56:41.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T07:56:41.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.488+0000 7fb6fbbff700 1 Processor -- start 2023-03-03T07:56:41.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.488+0000 7fb6fbbff700 1 -- start start 2023-03-03T07:56:41.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.489+0000 7fb6fbbff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb6f40b26d0 0x7fb6f40b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:41.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.489+0000 7fb6fbbff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb6f40b5140 0x7fb6f40d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:41.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.489+0000 7fb6fbbff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f40cd7d0 con 0x7fb6f40b8d10 2023-03-03T07:56:41.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.489+0000 7fb6fbbff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f40b55c0 con 0x7fb6f40b5140 2023-03-03T07:56:41.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.489+0000 7fb6fbbff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f40b90e0 con 0x7fb6f40b26d0 2023-03-03T07:56:41.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.490+0000 7fb6fa19c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb6f40b5140 0x7fb6f40d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:41.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.490+0000 7fb6f919a700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb6f40b26d0 0x7fb6f40b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:41.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.490+0000 7fb6fa19c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb6f40b5140 0x7fb6f40d13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42280/0 (socket says 172.21.15.98:42280) 2023-03-03T07:56:41.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.490+0000 7fb6fa19c700 1 -- 172.21.15.98:0/264768200 learned_addr learned my addr 172.21.15.98:0/264768200 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:41.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.490+0000 7fb6fa19c700 1 -- 172.21.15.98:0/264768200 >> v2:172.21.15.100:3300/0 conn(0x7fb6f40b26d0 msgr2=0x7fb6f40b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.490+0000 7fb6fa19c700 1 --2- 172.21.15.98:0/264768200 >> v2:172.21.15.100:3300/0 conn(0x7fb6f40b26d0 0x7fb6f40b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.490+0000 7fb6fa19c700 1 -- 172.21.15.98:0/264768200 >> v1:172.21.15.100:6789/0 conn(0x7fb6f40b8d10 legacy=0x7fb6f40cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.490+0000 7fb6fa19c700 1 -- 172.21.15.98:0/264768200 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f40cda20 con 0x7fb6f40b5140 2023-03-03T07:56:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.490+0000 7fb6fa19c700 1 --2- 172.21.15.98:0/264768200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f40b5140 0x7fb6f40d13a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb6f408bfa0 tx=0x7fb6ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4229c1c17e3950e6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.491+0000 7fb6f8999700 1 -- 172.21.15.98:0/264768200 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec00e070 con 0x7fb6f40b5140 2023-03-03T07:56:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.491+0000 7fb6f8999700 1 -- 172.21.15.98:0/264768200 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ec00b910 con 0x7fb6f40b5140 2023-03-03T07:56:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.491+0000 7fb6f8999700 1 -- 172.21.15.98:0/264768200 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec00bcc0 con 0x7fb6f40b5140 2023-03-03T07:56:41.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.491+0000 7fb6fbbff700 1 -- 172.21.15.98:0/264768200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f40b5140 msgr2=0x7fb6f40d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:41.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.491+0000 7fb6fbbff700 1 --2- 172.21.15.98:0/264768200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f40b5140 0x7fb6f40d13a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb6f408bfa0 tx=0x7fb6ec00b330 comp rx=0 tx=0).stop 2023-03-03T07:56:41.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.491+0000 7fb6fbbff700 1 -- 172.21.15.98:0/264768200 shutdown_connections 2023-03-03T07:56:41.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.491+0000 7fb6fbbff700 1 --2- 172.21.15.98:0/264768200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f40b5140 0x7fb6f40d13a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.491+0000 7fb6fbbff700 1 --2- 172.21.15.98:0/264768200 >> v2:172.21.15.100:3300/0 conn(0x7fb6f40b26d0 0x7fb6f40b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.491+0000 7fb6fbbff700 1 -- 172.21.15.98:0/264768200 >> 172.21.15.98:0/264768200 conn(0x7fb6f4073e70 msgr2=0x7fb6f406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:41.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.492+0000 7fb6fbbff700 1 -- 172.21.15.98:0/264768200 shutdown_connections 2023-03-03T07:56:41.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.492+0000 7fb6fbbff700 1 -- 172.21.15.98:0/264768200 wait complete. 2023-03-03T07:56:41.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.494+0000 7fb6fbbff700 1 Processor -- start 2023-03-03T07:56:41.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.495+0000 7fb6fbbff700 1 -- start start 2023-03-03T07:56:41.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.496+0000 7fb6fbbff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f40b26d0 0x7fb6f40cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:41.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.496+0000 7fb6fbbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6f40b5140 0x7fb6f40b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:41.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.496+0000 7fb6fbbff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb6f40b8d10 0x7fb6f40ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:41.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.496+0000 7fb6fbbff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f40cb8b0 con 0x7fb6f40b26d0 2023-03-03T07:56:41.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.496+0000 7fb6fbbff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f40cc210 con 0x7fb6f40b8d10 2023-03-03T07:56:41.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.496+0000 7fb6fbbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f40cac30 con 0x7fb6f40b5140 2023-03-03T07:56:41.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.496+0000 7fb6f999b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f40b26d0 0x7fb6f40cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:41.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.497+0000 7fb6f999b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f40b26d0 0x7fb6f40cbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42290/0 (socket says 172.21.15.98:42290) 2023-03-03T07:56:41.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.497+0000 7fb6f999b700 1 -- 172.21.15.98:0/2450107755 learned_addr learned my addr 172.21.15.98:0/2450107755 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:41.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.497+0000 7fb6fa19c700 1 --2- 172.21.15.98:0/2450107755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb6f40b8d10 0x7fb6f40ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:41.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.497+0000 7fb6f919a700 1 --2- 172.21.15.98:0/2450107755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6f40b5140 0x7fb6f40b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:41.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.497+0000 7fb6f999b700 1 -- 172.21.15.98:0/2450107755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6f40b5140 msgr2=0x7fb6f40b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:41.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.497+0000 7fb6f999b700 1 --2- 172.21.15.98:0/2450107755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6f40b5140 0x7fb6f40b4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.497+0000 7fb6f999b700 1 -- 172.21.15.98:0/2450107755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb6f40b8d10 msgr2=0x7fb6f40ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:41.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.497+0000 7fb6f999b700 1 --2- 172.21.15.98:0/2450107755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb6f40b8d10 0x7fb6f40ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.497+0000 7fb6f999b700 1 -- 172.21.15.98:0/2450107755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ec009b30 con 0x7fb6f40b26d0 2023-03-03T07:56:41.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.497+0000 7fb6f999b700 1 --2- 172.21.15.98:0/2450107755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f40b26d0 0x7fb6f40cbcd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000c450 tx=0x7fb6f000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:41.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.498+0000 7fb6ea7fc700 1 -- 172.21.15.98:0/2450107755 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f00052d0 con 0x7fb6f40b26d0 2023-03-03T07:56:41.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.498+0000 7fb6ea7fc700 1 -- 172.21.15.98:0/2450107755 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f0005430 con 0x7fb6f40b26d0 2023-03-03T07:56:41.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.498+0000 7fb6ea7fc700 1 -- 172.21.15.98:0/2450107755 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f00038c0 con 0x7fb6f40b26d0 2023-03-03T07:56:41.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.498+0000 7fb6fbbff700 1 -- 172.21.15.98:0/2450107755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f40d0e80 con 0x7fb6f40b26d0 2023-03-03T07:56:41.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.498+0000 7fb6fbbff700 1 -- 172.21.15.98:0/2450107755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f40cdd00 con 0x7fb6f40b26d0 2023-03-03T07:56:41.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.499+0000 7fb6fbbff700 1 -- 172.21.15.98:0/2450107755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f40cdfe0 con 0x7fb6f40b26d0 2023-03-03T07:56:41.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.500+0000 7fb6ea7fc700 1 -- 172.21.15.98:0/2450107755 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb6f0019030 con 0x7fb6f40b26d0 2023-03-03T07:56:41.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.501+0000 7fb6ea7fc700 1 --2- 172.21.15.98:0/2450107755 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb6d80682f0 0x7fb6d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:41.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.501+0000 7fb6ea7fc700 1 -- 172.21.15.98:0/2450107755 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fb6f0085100 con 0x7fb6f40b26d0 2023-03-03T07:56:41.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.502+0000 7fb6f919a700 1 --2- 172.21.15.98:0/2450107755 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb6d80682f0 0x7fb6d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:41.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.502+0000 7fb6f919a700 1 --2- 172.21.15.98:0/2450107755 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb6d80682f0 0x7fb6d806a7a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40b6c30 tx=0x7fb6e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:41.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.505+0000 7fb6ea7fc700 1 -- 172.21.15.98:0/2450107755 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb6f00517a0 con 0x7fb6f40b26d0 2023-03-03T07:56:41.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.715+0000 7fb6fbbff700 1 -- 172.21.15.98:0/2450107755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb6f40cbb20 con 0x7fb6f40b26d0 2023-03-03T07:56:41.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.716+0000 7fb6ea7fc700 1 -- 172.21.15.98:0/2450107755 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb6f0017080 con 0x7fb6f40b26d0 2023-03-03T07:56:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.718+0000 7fb6fbbff700 1 -- 172.21.15.98:0/2450107755 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb6d80682f0 msgr2=0x7fb6d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.718+0000 7fb6fbbff700 1 --2- 172.21.15.98:0/2450107755 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb6d80682f0 0x7fb6d806a7a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40b6c30 tx=0x7fb6e4008040 comp rx=0 tx=0).stop 2023-03-03T07:56:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.718+0000 7fb6fbbff700 1 -- 172.21.15.98:0/2450107755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f40b26d0 msgr2=0x7fb6f40cbcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.718+0000 7fb6fbbff700 1 --2- 172.21.15.98:0/2450107755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f40b26d0 0x7fb6f40cbcd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000c450 tx=0x7fb6f000c810 comp rx=0 tx=0).stop 2023-03-03T07:56:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.718+0000 7fb6fbbff700 1 -- 172.21.15.98:0/2450107755 shutdown_connections 2023-03-03T07:56:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.718+0000 7fb6fbbff700 1 --2- 172.21.15.98:0/2450107755 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb6d80682f0 0x7fb6d806a7a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.718+0000 7fb6fbbff700 1 --2- 172.21.15.98:0/2450107755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f40b26d0 0x7fb6f40cbcd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.718+0000 7fb6fbbff700 1 --2- 172.21.15.98:0/2450107755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb6f40b8d10 0x7fb6f40ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.718+0000 7fb6fbbff700 1 --2- 172.21.15.98:0/2450107755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6f40b5140 0x7fb6f40b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:41.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.718+0000 7fb6fbbff700 1 -- 172.21.15.98:0/2450107755 >> 172.21.15.98:0/2450107755 conn(0x7fb6f4073e70 msgr2=0x7fb6f40b8390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:41.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.719+0000 7fb6fbbff700 1 -- 172.21.15.98:0/2450107755 shutdown_connections 2023-03-03T07:56:41.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:41.719+0000 7fb6fbbff700 1 -- 172.21.15.98:0/2450107755 wait complete. 2023-03-03T07:56:41.723 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T07:56:41.748 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:56:41.749 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:56:41.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:56:42.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.063+0000 7efdc599d700 1 Processor -- start 2023-03-03T07:56:42.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.063+0000 7efdc599d700 1 -- start start 2023-03-03T07:56:42.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.064+0000 7efdc599d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdc0146360 0x7efdc0160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:42.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.064+0000 7efdc599d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efdc00b4ab0 0x7efdc00b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:42.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.065+0000 7efdc599d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efdc00731e0 0x7efdc0073660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:42.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.065+0000 7efdc599d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc00b5460 con 0x7efdc00731e0 2023-03-03T07:56:42.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.065+0000 7efdc599d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc01614c0 con 0x7efdc00b4ab0 2023-03-03T07:56:42.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.065+0000 7efdc599d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc00d10f0 con 0x7efdc0146360 2023-03-03T07:56:42.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.065+0000 7efdc519c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efdc00731e0 0x7efdc0073660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:42.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.065+0000 7efdc519c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efdc00731e0 0x7efdc0073660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42300/0 (socket says 172.21.15.98:42300) 2023-03-03T07:56:42.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.065+0000 7efdc519c700 1 -- 172.21.15.98:0/3445444451 learned_addr learned my addr 172.21.15.98:0/3445444451 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.065+0000 7efdbffff700 1 --2- 172.21.15.98:0/3445444451 >> v2:172.21.15.100:3300/0 conn(0x7efdc00b4ab0 0x7efdc00b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.065+0000 7efdc499b700 1 --2- 172.21.15.98:0/3445444451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdc0146360 0x7efdc0160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.065+0000 7efdc519c700 1 -- 172.21.15.98:0/3445444451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdc0146360 msgr2=0x7efdc0160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.065+0000 7efdc519c700 1 --2- 172.21.15.98:0/3445444451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdc0146360 0x7efdc0160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.065+0000 7efdc519c700 1 -- 172.21.15.98:0/3445444451 >> v2:172.21.15.100:3300/0 conn(0x7efdc00b4ab0 msgr2=0x7efdc00b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.065+0000 7efdc519c700 1 --2- 172.21.15.98:0/3445444451 >> v2:172.21.15.100:3300/0 conn(0x7efdc00b4ab0 0x7efdc00b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.065+0000 7efdc519c700 1 -- 172.21.15.98:0/3445444451 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc00cdf20 con 0x7efdc00731e0 2023-03-03T07:56:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.066+0000 7efdc519c700 1 --2- 172.21.15.98:0/3445444451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc00731e0 0x7efdc0073660 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efdc0072a40 tx=0x7efdb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa09c5cd270dcfdd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.066+0000 7efdbf7fe700 1 -- 172.21.15.98:0/3445444451 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb800e070 con 0x7efdc00731e0 2023-03-03T07:56:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.066+0000 7efdbf7fe700 1 -- 172.21.15.98:0/3445444451 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb800b9c0 con 0x7efdc00731e0 2023-03-03T07:56:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.066+0000 7efdbf7fe700 1 -- 172.21.15.98:0/3445444451 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb800bd70 con 0x7efdc00731e0 2023-03-03T07:56:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.067+0000 7efdc599d700 1 -- 172.21.15.98:0/3445444451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc00731e0 msgr2=0x7efdc0073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.067+0000 7efdc599d700 1 --2- 172.21.15.98:0/3445444451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc00731e0 0x7efdc0073660 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efdc0072a40 tx=0x7efdb800b330 comp rx=0 tx=0).stop 2023-03-03T07:56:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.067+0000 7efdc599d700 1 -- 172.21.15.98:0/3445444451 shutdown_connections 2023-03-03T07:56:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.067+0000 7efdc599d700 1 --2- 172.21.15.98:0/3445444451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc00731e0 0x7efdc0073660 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.067+0000 7efdc599d700 1 --2- 172.21.15.98:0/3445444451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdc0146360 0x7efdc0160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.067+0000 7efdc599d700 1 --2- 172.21.15.98:0/3445444451 >> v2:172.21.15.100:3300/0 conn(0x7efdc00b4ab0 0x7efdc00b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.067+0000 7efdc599d700 1 -- 172.21.15.98:0/3445444451 >> 172.21.15.98:0/3445444451 conn(0x7efdc006b8e0 msgr2=0x7efdc00cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.067+0000 7efdc599d700 1 -- 172.21.15.98:0/3445444451 shutdown_connections 2023-03-03T07:56:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.067+0000 7efdc599d700 1 -- 172.21.15.98:0/3445444451 wait complete. 2023-03-03T07:56:42.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.069+0000 7efdc599d700 1 Processor -- start 2023-03-03T07:56:42.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.069+0000 7efdc599d700 1 -- start start 2023-03-03T07:56:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.070+0000 7efdc599d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdc00731e0 0x7efdc01f5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.070+0000 7efdc599d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efdc00b4ab0 0x7efdc0151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.070+0000 7efdc599d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc0146360 0x7efdc00b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.070+0000 7efdc599d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc015eee0 con 0x7efdc0146360 2023-03-03T07:56:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.070+0000 7efdc599d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc0152100 con 0x7efdc00b4ab0 2023-03-03T07:56:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.070+0000 7efdc599d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc015e260 con 0x7efdc00731e0 2023-03-03T07:56:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.070+0000 7efdc519c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc0146360 0x7efdc00b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.070+0000 7efdc519c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc0146360 0x7efdc00b3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42312/0 (socket says 172.21.15.98:42312) 2023-03-03T07:56:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.070+0000 7efdc519c700 1 -- 172.21.15.98:0/507081900 learned_addr learned my addr 172.21.15.98:0/507081900 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.071+0000 7efdc499b700 1 --2- 172.21.15.98:0/507081900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdc00731e0 0x7efdc01f5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.071+0000 7efdbffff700 1 --2- 172.21.15.98:0/507081900 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efdc00b4ab0 0x7efdc0151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.071+0000 7efdc519c700 1 -- 172.21.15.98:0/507081900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdc00731e0 msgr2=0x7efdc01f5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.071+0000 7efdc519c700 1 --2- 172.21.15.98:0/507081900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdc00731e0 0x7efdc01f5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.071+0000 7efdc519c700 1 -- 172.21.15.98:0/507081900 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efdc00b4ab0 msgr2=0x7efdc0151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.071+0000 7efdc519c700 1 --2- 172.21.15.98:0/507081900 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efdc00b4ab0 0x7efdc0151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.071+0000 7efdc519c700 1 -- 172.21.15.98:0/507081900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb8009b30 con 0x7efdc0146360 2023-03-03T07:56:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.071+0000 7efdc519c700 1 --2- 172.21.15.98:0/507081900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc0146360 0x7efdc00b3f20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efdc00747b0 tx=0x7efdb8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.071+0000 7efdbd7fa700 1 -- 172.21.15.98:0/507081900 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb8004250 con 0x7efdc0146360 2023-03-03T07:56:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.071+0000 7efdbd7fa700 1 -- 172.21.15.98:0/507081900 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb80043d0 con 0x7efdc0146360 2023-03-03T07:56:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.071+0000 7efdbd7fa700 1 -- 172.21.15.98:0/507081900 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb8004780 con 0x7efdc0146360 2023-03-03T07:56:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.071+0000 7efdc599d700 1 -- 172.21.15.98:0/507081900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc00cd960 con 0x7efdc0146360 2023-03-03T07:56:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.072+0000 7efdc599d700 1 -- 172.21.15.98:0/507081900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc00cd680 con 0x7efdc0146360 2023-03-03T07:56:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.073+0000 7efdc599d700 1 -- 172.21.15.98:0/507081900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc00cdc40 con 0x7efdc0146360 2023-03-03T07:56:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.074+0000 7efdbd7fa700 1 -- 172.21.15.98:0/507081900 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efdb802b070 con 0x7efdc0146360 2023-03-03T07:56:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.075+0000 7efdbd7fa700 1 --2- 172.21.15.98:0/507081900 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efda40682f0 0x7efda406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.075+0000 7efdbd7fa700 1 -- 172.21.15.98:0/507081900 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7efdb80871b0 con 0x7efdc0146360 2023-03-03T07:56:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.075+0000 7efdc499b700 1 --2- 172.21.15.98:0/507081900 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efda40682f0 0x7efda406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.076+0000 7efdc499b700 1 --2- 172.21.15.98:0/507081900 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efda40682f0 0x7efda406a7a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efdc015b700 tx=0x7efdb4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:42.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.079+0000 7efdbd7fa700 1 -- 172.21.15.98:0/507081900 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efdb80538e0 con 0x7efdc0146360 2023-03-03T07:56:42.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.273+0000 7efdc599d700 1 -- 172.21.15.98:0/507081900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdc015f020 con 0x7efdc0146360 2023-03-03T07:56:42.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.274+0000 7efdbd7fa700 1 -- 172.21.15.98:0/507081900 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18003 (secure 0 0 0) 0x7efdc015f020 con 0x7efdc0146360 2023-03-03T07:56:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:56:42.277 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:56:27.498205+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:56:42.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.277+0000 7efdc599d700 1 -- 172.21.15.98:0/507081900 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efda40682f0 msgr2=0x7efda406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:42.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.277+0000 7efdc599d700 1 --2- 172.21.15.98:0/507081900 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efda40682f0 0x7efda406a7a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efdc015b700 tx=0x7efdb4008040 comp rx=0 tx=0).stop 2023-03-03T07:56:42.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.277+0000 7efdc599d700 1 -- 172.21.15.98:0/507081900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc0146360 msgr2=0x7efdc00b3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:42.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.277+0000 7efdc599d700 1 --2- 172.21.15.98:0/507081900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc0146360 0x7efdc00b3f20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efdc00747b0 tx=0x7efdb8004ad0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.277+0000 7efdc599d700 1 -- 172.21.15.98:0/507081900 shutdown_connections 2023-03-03T07:56:42.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.277+0000 7efdc599d700 1 --2- 172.21.15.98:0/507081900 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efda40682f0 0x7efda406a7a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.277+0000 7efdc599d700 1 --2- 172.21.15.98:0/507081900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc0146360 0x7efdc00b3f20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.277+0000 7efdc599d700 1 --2- 172.21.15.98:0/507081900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdc00731e0 0x7efdc01f5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.277+0000 7efdc599d700 1 --2- 172.21.15.98:0/507081900 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efdc00b4ab0 0x7efdc0151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.277+0000 7efdc599d700 1 -- 172.21.15.98:0/507081900 >> 172.21.15.98:0/507081900 conn(0x7efdc006b8e0 msgr2=0x7efdc00d2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:42.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.277+0000 7efdc599d700 1 -- 172.21.15.98:0/507081900 shutdown_connections 2023-03-03T07:56:42.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.277+0000 7efdc599d700 1 -- 172.21.15.98:0/507081900 wait complete. 2023-03-03T07:56:42.306 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:55:45.157666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:55:46.092091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:56:42.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:56:42.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.551+0000 7f4198b9d700 1 Processor -- start 2023-03-03T07:56:42.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.552+0000 7f4198b9d700 1 -- start start 2023-03-03T07:56:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.553+0000 7f4198b9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f41941609b0 0x7f4194160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.553+0000 7f4198b9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f41940cf570 0x7f419415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.553+0000 7f4198b9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41941612c0 con 0x7f419415db30 2023-03-03T07:56:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.553+0000 7f4198b9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41940cebb0 con 0x7f41941609b0 2023-03-03T07:56:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.553+0000 7f4198b9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41940cd780 con 0x7f41940cf570 2023-03-03T07:56:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.553+0000 7f41937fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f41941609b0 0x7f4194160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.554+0000 7f41937fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f41941609b0 0x7f4194160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42324/0 (socket says 172.21.15.98:42324) 2023-03-03T07:56:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.554+0000 7f41937fe700 1 -- 172.21.15.98:0/1449373135 learned_addr learned my addr 172.21.15.98:0/1449373135 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.554+0000 7f418bfff700 1 --2- 172.21.15.98:0/1449373135 >> v2:172.21.15.100:3300/0 conn(0x7f41940cf570 0x7f419415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.554+0000 7f41937fe700 1 -- 172.21.15.98:0/1449373135 >> v2:172.21.15.100:3300/0 conn(0x7f41940cf570 msgr2=0x7f419415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.554+0000 7f41937fe700 1 --2- 172.21.15.98:0/1449373135 >> v2:172.21.15.100:3300/0 conn(0x7f41940cf570 0x7f419415d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.554+0000 7f41937fe700 1 -- 172.21.15.98:0/1449373135 >> v1:172.21.15.100:6789/0 conn(0x7f419415db30 legacy=0x7f41940b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.554+0000 7f41937fe700 1 -- 172.21.15.98:0/1449373135 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41940cda60 con 0x7f41941609b0 2023-03-03T07:56:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.554+0000 7f41937fe700 1 --2- 172.21.15.98:0/1449373135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41941609b0 0x7f4194160d80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f41940cf0c0 tx=0x7f4184009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e481b8c86f11bffc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.555+0000 7f4192ffd700 1 -- 172.21.15.98:0/1449373135 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4184004230 con 0x7f41941609b0 2023-03-03T07:56:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.555+0000 7f4192ffd700 1 -- 172.21.15.98:0/1449373135 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4184004390 con 0x7f41941609b0 2023-03-03T07:56:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.555+0000 7f4192ffd700 1 -- 172.21.15.98:0/1449373135 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41840036a0 con 0x7f41941609b0 2023-03-03T07:56:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.555+0000 7f4198b9d700 1 -- 172.21.15.98:0/1449373135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41941609b0 msgr2=0x7f4194160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.555+0000 7f4198b9d700 1 --2- 172.21.15.98:0/1449373135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41941609b0 0x7f4194160d80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f41940cf0c0 tx=0x7f4184009bd0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.555+0000 7f4198b9d700 1 -- 172.21.15.98:0/1449373135 shutdown_connections 2023-03-03T07:56:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.555+0000 7f4198b9d700 1 --2- 172.21.15.98:0/1449373135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41941609b0 0x7f4194160d80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.555+0000 7f4198b9d700 1 --2- 172.21.15.98:0/1449373135 >> v2:172.21.15.100:3300/0 conn(0x7f41940cf570 0x7f419415d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.555+0000 7f4198b9d700 1 -- 172.21.15.98:0/1449373135 >> 172.21.15.98:0/1449373135 conn(0x7f419406b880 msgr2=0x7f41940cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.555+0000 7f4198b9d700 1 -- 172.21.15.98:0/1449373135 shutdown_connections 2023-03-03T07:56:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.556+0000 7f4198b9d700 1 -- 172.21.15.98:0/1449373135 wait complete. 2023-03-03T07:56:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.558+0000 7f4198b9d700 1 Processor -- start 2023-03-03T07:56:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.558+0000 7f4198b9d700 1 -- start start 2023-03-03T07:56:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.560+0000 7f4198b9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41940cf570 0x7f41940c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.560+0000 7f4198b9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f419415db30 0x7f41940d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.560+0000 7f4198b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41941609b0 0x7f41940b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.560+0000 7f4198b9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41940d26f0 con 0x7f419415db30 2023-03-03T07:56:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.560+0000 7f4198b9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41940b44e0 con 0x7f41940cf570 2023-03-03T07:56:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.560+0000 7f4198b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41940b5430 con 0x7f41941609b0 2023-03-03T07:56:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.560+0000 7f418bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f419415db30 0x7f41940d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.560+0000 7f418bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f419415db30 0x7f41940d21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42340/0 (socket says 172.21.15.98:42340) 2023-03-03T07:56:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.560+0000 7f418bfff700 1 -- 172.21.15.98:0/828250998 learned_addr learned my addr 172.21.15.98:0/828250998 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.560+0000 7f41937fe700 1 --2- 172.21.15.98:0/828250998 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41940cf570 0x7f41940c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.560+0000 7f4193fff700 1 --2- 172.21.15.98:0/828250998 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41941609b0 0x7f41940b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.561+0000 7f418bfff700 1 -- 172.21.15.98:0/828250998 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41941609b0 msgr2=0x7f41940b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.561+0000 7f418bfff700 1 --2- 172.21.15.98:0/828250998 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41941609b0 0x7f41940b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.561+0000 7f418bfff700 1 -- 172.21.15.98:0/828250998 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41940cf570 msgr2=0x7f41940c5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.561+0000 7f418bfff700 1 --2- 172.21.15.98:0/828250998 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41940cf570 0x7f41940c5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.561+0000 7f418bfff700 1 -- 172.21.15.98:0/828250998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4184009550 con 0x7f419415db30 2023-03-03T07:56:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.561+0000 7f418bfff700 1 --2- 172.21.15.98:0/828250998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f419415db30 0x7f41940d21b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f417c00e840 tx=0x7f417c00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.561+0000 7f4190ff9700 1 -- 172.21.15.98:0/828250998 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c013610 con 0x7f419415db30 2023-03-03T07:56:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.562+0000 7f4190ff9700 1 -- 172.21.15.98:0/828250998 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c013770 con 0x7f419415db30 2023-03-03T07:56:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.562+0000 7f4198b9d700 1 -- 172.21.15.98:0/828250998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419415cdf0 con 0x7f419415db30 2023-03-03T07:56:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.562+0000 7f4190ff9700 1 -- 172.21.15.98:0/828250998 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c013b20 con 0x7f419415db30 2023-03-03T07:56:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.562+0000 7f4198b9d700 1 -- 172.21.15.98:0/828250998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41940cdd40 con 0x7f419415db30 2023-03-03T07:56:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.563+0000 7f4198b9d700 1 -- 172.21.15.98:0/828250998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419415d0d0 con 0x7f419415db30 2023-03-03T07:56:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.564+0000 7f4190ff9700 1 -- 172.21.15.98:0/828250998 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f417c023070 con 0x7f419415db30 2023-03-03T07:56:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.566+0000 7f4190ff9700 1 --2- 172.21.15.98:0/828250998 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f41740682f0 0x7f417406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.566+0000 7f4190ff9700 1 -- 172.21.15.98:0/828250998 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f417c0882a0 con 0x7f419415db30 2023-03-03T07:56:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.566+0000 7f41937fe700 1 --2- 172.21.15.98:0/828250998 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f41740682f0 0x7f417406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.567+0000 7f41937fe700 1 --2- 172.21.15.98:0/828250998 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f41740682f0 0x7f417406a7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4184009130 tx=0x7f4184008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.570+0000 7f4190ff9700 1 -- 172.21.15.98:0/828250998 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f417c052010 con 0x7f419415db30 2023-03-03T07:56:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.765+0000 7f4198b9d700 1 -- 172.21.15.98:0/828250998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4194149690 con 0x7f419415db30 2023-03-03T07:56:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.765+0000 7f4190ff9700 1 -- 172.21.15.98:0/828250998 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4194149690 con 0x7f419415db30 2023-03-03T07:56:42.768 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:56:42.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.768+0000 7f4198b9d700 1 -- 172.21.15.98:0/828250998 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f41740682f0 msgr2=0x7f417406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:42.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.768+0000 7f4198b9d700 1 --2- 172.21.15.98:0/828250998 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f41740682f0 0x7f417406a7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4184009130 tx=0x7f4184008030 comp rx=0 tx=0).stop 2023-03-03T07:56:42.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.768+0000 7f4198b9d700 1 -- 172.21.15.98:0/828250998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f419415db30 msgr2=0x7f41940d21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:42.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.768+0000 7f4198b9d700 1 --2- 172.21.15.98:0/828250998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f419415db30 0x7f41940d21b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f417c00e840 tx=0x7f417c00eb50 comp rx=0 tx=0).stop 2023-03-03T07:56:42.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.768+0000 7f4198b9d700 1 -- 172.21.15.98:0/828250998 shutdown_connections 2023-03-03T07:56:42.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.768+0000 7f4198b9d700 1 --2- 172.21.15.98:0/828250998 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f41740682f0 0x7f417406a7a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.768+0000 7f4198b9d700 1 --2- 172.21.15.98:0/828250998 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41941609b0 0x7f41940b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.768+0000 7f4198b9d700 1 --2- 172.21.15.98:0/828250998 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41940cf570 0x7f41940c5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.768+0000 7f4198b9d700 1 --2- 172.21.15.98:0/828250998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f419415db30 0x7f41940d21b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:42.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.768+0000 7f4198b9d700 1 -- 172.21.15.98:0/828250998 >> 172.21.15.98:0/828250998 conn(0x7f419406b880 msgr2=0x7f41940b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:42.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.768+0000 7f4198b9d700 1 -- 172.21.15.98:0/828250998 shutdown_connections 2023-03-03T07:56:42.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:42.769+0000 7f4198b9d700 1 -- 172.21.15.98:0/828250998 wait complete. 2023-03-03T07:56:42.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:56:43.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.041+0000 7ff44659d700 1 Processor -- start 2023-03-03T07:56:43.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.042+0000 7ff44659d700 1 -- start start 2023-03-03T07:56:43.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.043+0000 7ff44659d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff4400cf570 0x7ff44015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:43.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.043+0000 7ff44659d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4400cd780 con 0x7ff4401609b0 2023-03-03T07:56:43.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.043+0000 7ff44659d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4400cda60 con 0x7ff44015db30 2023-03-03T07:56:43.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.043+0000 7ff44659d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4400cebb0 con 0x7ff4400cf570 2023-03-03T07:56:43.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.043+0000 7ff44559b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff4401609b0 0x7ff440160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54108/0 (socket says 172.21.15.98:54108) 2023-03-03T07:56:43.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.043+0000 7ff44559b700 1 -- 172.21.15.98:0/2218295839 learned_addr learned my addr 172.21.15.98:0/2218295839 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:43.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.044+0000 7ff444d9a700 1 --2- 172.21.15.98:0/2218295839 >> v2:172.21.15.100:3300/0 conn(0x7ff4400cf570 0x7ff44015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:43.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.044+0000 7ff437fff700 1 -- 172.21.15.98:0/2218295839 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39726806 0 0) 0x7ff4400cd780 con 0x7ff4401609b0 2023-03-03T07:56:43.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.044+0000 7ff437fff700 1 -- 172.21.15.98:0/2218295839 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff42c0035b0 con 0x7ff4401609b0 2023-03-03T07:56:43.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.044+0000 7ff437fff700 1 -- 172.21.15.98:0/2218295839 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4247239595 0 0) 0x7ff42c0035b0 con 0x7ff4401609b0 2023-03-03T07:56:43.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.044+0000 7ff444d9a700 1 -- 172.21.15.98:0/2218295839 >> v1:172.21.15.100:6789/0 conn(0x7ff44015db30 legacy=0x7ff4400b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:43.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.044+0000 7ff444d9a700 1 -- 172.21.15.98:0/2218295839 >> v1:172.21.15.98:6789/0 conn(0x7ff4401609b0 legacy=0x7ff440160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:43.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.044+0000 7ff444d9a700 1 -- 172.21.15.98:0/2218295839 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4400cdd40 con 0x7ff4400cf570 2023-03-03T07:56:43.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.044+0000 7ff444d9a700 1 --2- 172.21.15.98:0/2218295839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4400cf570 0x7ff44015d5f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff430009840 tx=0x7ff430009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec008ede0148aa63 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.045+0000 7ff437fff700 1 -- 172.21.15.98:0/2218295839 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c001670 con 0x7ff4400cf570 2023-03-03T07:56:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.045+0000 7ff437fff700 1 -- 172.21.15.98:0/2218295839 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff430004030 con 0x7ff4400cf570 2023-03-03T07:56:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.045+0000 7ff437fff700 1 -- 172.21.15.98:0/2218295839 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4300043c0 con 0x7ff4400cf570 2023-03-03T07:56:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.045+0000 7ff44659d700 1 -- 172.21.15.98:0/2218295839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4400cf570 msgr2=0x7ff44015d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:43.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.045+0000 7ff44659d700 1 --2- 172.21.15.98:0/2218295839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4400cf570 0x7ff44015d5f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff430009840 tx=0x7ff430009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:43.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.045+0000 7ff44659d700 1 -- 172.21.15.98:0/2218295839 shutdown_connections 2023-03-03T07:56:43.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.045+0000 7ff44659d700 1 --2- 172.21.15.98:0/2218295839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4400cf570 0x7ff44015d5f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:43.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.045+0000 7ff44659d700 1 -- 172.21.15.98:0/2218295839 >> 172.21.15.98:0/2218295839 conn(0x7ff44006b880 msgr2=0x7ff4400cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:43.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.046+0000 7ff44659d700 1 -- 172.21.15.98:0/2218295839 shutdown_connections 2023-03-03T07:56:43.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.046+0000 7ff44659d700 1 -- 172.21.15.98:0/2218295839 wait complete. 2023-03-03T07:56:43.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.048+0000 7ff44659d700 1 Processor -- start 2023-03-03T07:56:43.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.048+0000 7ff44659d700 1 -- start start 2023-03-03T07:56:43.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.049+0000 7ff44659d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4400cf570 0x7ff440151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:43.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.049+0000 7ff44659d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff44015db30 0x7ff4400ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:43.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.049+0000 7ff44659d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4401609b0 0x7ff4400d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:43.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.049+0000 7ff44659d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44014dc10 con 0x7ff44015db30 2023-03-03T07:56:43.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.049+0000 7ff44659d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400ca630 con 0x7ff4401609b0 2023-03-03T07:56:43.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.049+0000 7ff44659d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff440150aa0 con 0x7ff4400cf570 2023-03-03T07:56:43.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.049+0000 7ff444d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff44015db30 0x7ff4400ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:43.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.049+0000 7ff444d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff44015db30 0x7ff4400ca0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42354/0 (socket says 172.21.15.98:42354) 2023-03-03T07:56:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.049+0000 7ff444d9a700 1 -- 172.21.15.98:0/1755389044 learned_addr learned my addr 172.21.15.98:0/1755389044 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.049+0000 7ff44559b700 1 --2- 172.21.15.98:0/1755389044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4400cf570 0x7ff440151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.050+0000 7ff445d9c700 1 --2- 172.21.15.98:0/1755389044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4401609b0 0x7ff4400d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.050+0000 7ff444d9a700 1 -- 172.21.15.98:0/1755389044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4400cf570 msgr2=0x7ff440151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.050+0000 7ff444d9a700 1 --2- 172.21.15.98:0/1755389044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4400cf570 0x7ff440151870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.050+0000 7ff444d9a700 1 -- 172.21.15.98:0/1755389044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4401609b0 msgr2=0x7ff4400d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.050+0000 7ff444d9a700 1 --2- 172.21.15.98:0/1755389044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4401609b0 0x7ff4400d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:43.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.050+0000 7ff444d9a700 1 -- 172.21.15.98:0/1755389044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4300094a0 con 0x7ff44015db30 2023-03-03T07:56:43.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.050+0000 7ff444d9a700 1 --2- 172.21.15.98:0/1755389044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff44015db30 0x7ff4400ca0f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff43000a010 tx=0x7ff430004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:43.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.050+0000 7ff435ffb700 1 -- 172.21.15.98:0/1755389044 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff430011630 con 0x7ff44015db30 2023-03-03T07:56:43.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.050+0000 7ff435ffb700 1 -- 172.21.15.98:0/1755389044 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff43001f070 con 0x7ff44015db30 2023-03-03T07:56:43.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.050+0000 7ff435ffb700 1 -- 172.21.15.98:0/1755389044 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4300119d0 con 0x7ff44015db30 2023-03-03T07:56:43.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.050+0000 7ff44659d700 1 -- 172.21.15.98:0/1755389044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44015d0d0 con 0x7ff44015db30 2023-03-03T07:56:43.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.051+0000 7ff44659d700 1 -- 172.21.15.98:0/1755389044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44015cdf0 con 0x7ff44015db30 2023-03-03T07:56:43.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.052+0000 7ff44659d700 1 -- 172.21.15.98:0/1755389044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4400b51b0 con 0x7ff44015db30 2023-03-03T07:56:43.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.053+0000 7ff435ffb700 1 -- 172.21.15.98:0/1755389044 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff430017020 con 0x7ff44015db30 2023-03-03T07:56:43.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.054+0000 7ff435ffb700 1 --2- 172.21.15.98:0/1755389044 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4240682f0 0x7ff42406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:43.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.054+0000 7ff435ffb700 1 -- 172.21.15.98:0/1755389044 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7ff430021070 con 0x7ff44015db30 2023-03-03T07:56:43.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.054+0000 7ff44559b700 1 --2- 172.21.15.98:0/1755389044 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4240682f0 0x7ff42406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:43.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.055+0000 7ff44559b700 1 --2- 172.21.15.98:0/1755389044 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4240682f0 0x7ff42406a7a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0011e0 tx=0x7ff43c007ba0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:43.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.058+0000 7ff435ffb700 1 -- 172.21.15.98:0/1755389044 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff430053e40 con 0x7ff44015db30 2023-03-03T07:56:43.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.257+0000 7ff44659d700 1 -- 172.21.15.98:0/1755389044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff4400679a0 con 0x7ff44015db30 2023-03-03T07:56:43.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.257+0000 7ff435ffb700 1 -- 172.21.15.98:0/1755389044 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff430024380 con 0x7ff44015db30 2023-03-03T07:56:43.260 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:56:43.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.260+0000 7ff44659d700 1 -- 172.21.15.98:0/1755389044 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4240682f0 msgr2=0x7ff42406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:43.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.260+0000 7ff44659d700 1 --2- 172.21.15.98:0/1755389044 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4240682f0 0x7ff42406a7a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0011e0 tx=0x7ff43c007ba0 comp rx=0 tx=0).stop 2023-03-03T07:56:43.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.260+0000 7ff44659d700 1 -- 172.21.15.98:0/1755389044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff44015db30 msgr2=0x7ff4400ca0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.260+0000 7ff44659d700 1 --2- 172.21.15.98:0/1755389044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff44015db30 0x7ff4400ca0f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff43000a010 tx=0x7ff430004750 comp rx=0 tx=0).stop 2023-03-03T07:56:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.260+0000 7ff44659d700 1 -- 172.21.15.98:0/1755389044 shutdown_connections 2023-03-03T07:56:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.260+0000 7ff44659d700 1 --2- 172.21.15.98:0/1755389044 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4240682f0 0x7ff42406a7a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.260+0000 7ff44659d700 1 --2- 172.21.15.98:0/1755389044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4401609b0 0x7ff4400d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.260+0000 7ff44659d700 1 --2- 172.21.15.98:0/1755389044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4400cf570 0x7ff440151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.260+0000 7ff44659d700 1 --2- 172.21.15.98:0/1755389044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff44015db30 0x7ff4400ca0f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.260+0000 7ff44659d700 1 -- 172.21.15.98:0/1755389044 >> 172.21.15.98:0/1755389044 conn(0x7ff44006b880 msgr2=0x7ff44014f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.260+0000 7ff44659d700 1 -- 172.21.15.98:0/1755389044 shutdown_connections 2023-03-03T07:56:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.260+0000 7ff44659d700 1 -- 172.21.15.98:0/1755389044 wait complete. 2023-03-03T07:56:43.289 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:56:43.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.535+0000 7f0045dff700 1 Processor -- start 2023-03-03T07:56:43.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.535+0000 7f0045dff700 1 -- start start 2023-03-03T07:56:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.537+0000 7f0045dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00400cde20 0x7f0040160c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.537+0000 7f0045dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00400cd990 con 0x7f00400d2a60 2023-03-03T07:56:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.537+0000 7f0045dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00400d2fd0 con 0x7f00400b8ae0 2023-03-03T07:56:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.537+0000 7f0045dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00400d1aa0 con 0x7f00400cde20 2023-03-03T07:56:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.537+0000 7f003ffff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f00400d2a60 0x7f00400cf490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54116/0 (socket says 172.21.15.98:54116) 2023-03-03T07:56:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.537+0000 7f003ffff700 1 -- 172.21.15.98:0/1039181155 learned_addr learned my addr 172.21.15.98:0/1039181155 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.537+0000 7f003effd700 1 --2- 172.21.15.98:0/1039181155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00400cde20 0x7f0040160c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:43.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.538+0000 7f003e7fc700 1 -- 172.21.15.98:0/1039181155 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422752204 0 0) 0x7f00400cd990 con 0x7f00400d2a60 2023-03-03T07:56:43.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.538+0000 7f003e7fc700 1 -- 172.21.15.98:0/1039181155 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00280035b0 con 0x7f00400d2a60 2023-03-03T07:56:43.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.538+0000 7f003effd700 1 -- 172.21.15.98:0/1039181155 >> v1:172.21.15.100:6789/0 conn(0x7f00400b8ae0 legacy=0x7f0040160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:43.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.538+0000 7f003effd700 1 -- 172.21.15.98:0/1039181155 >> v1:172.21.15.98:6789/0 conn(0x7f00400d2a60 legacy=0x7f00400cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:43.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.538+0000 7f003effd700 1 -- 172.21.15.98:0/1039181155 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00400d1d80 con 0x7f00400cde20 2023-03-03T07:56:43.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.538+0000 7f003effd700 1 --2- 172.21.15.98:0/1039181155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00400cde20 0x7f0040160c50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0034009870 tx=0x7f0034009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c4e625f4b01b9fb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:43.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.539+0000 7f003e7fc700 1 -- 172.21.15.98:0/1039181155 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0034011830 con 0x7f00400cde20 2023-03-03T07:56:43.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.539+0000 7f003e7fc700 1 -- 172.21.15.98:0/1039181155 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0034011990 con 0x7f00400cde20 2023-03-03T07:56:43.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.539+0000 7f003e7fc700 1 -- 172.21.15.98:0/1039181155 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0034011d20 con 0x7f00400cde20 2023-03-03T07:56:43.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.539+0000 7f0045dff700 1 -- 172.21.15.98:0/1039181155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00400cde20 msgr2=0x7f0040160c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:43.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.539+0000 7f0045dff700 1 --2- 172.21.15.98:0/1039181155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00400cde20 0x7f0040160c50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0034009870 tx=0x7f0034009b80 comp rx=0 tx=0).stop 2023-03-03T07:56:43.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.539+0000 7f0045dff700 1 -- 172.21.15.98:0/1039181155 shutdown_connections 2023-03-03T07:56:43.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.539+0000 7f0045dff700 1 --2- 172.21.15.98:0/1039181155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00400cde20 0x7f0040160c50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:43.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.539+0000 7f0045dff700 1 -- 172.21.15.98:0/1039181155 >> 172.21.15.98:0/1039181155 conn(0x7f0040073e10 msgr2=0x7f004006ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:43.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.539+0000 7f0045dff700 1 -- 172.21.15.98:0/1039181155 shutdown_connections 2023-03-03T07:56:43.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.539+0000 7f0045dff700 1 -- 172.21.15.98:0/1039181155 wait complete. 2023-03-03T07:56:43.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.542+0000 7f0045dff700 1 Processor -- start 2023-03-03T07:56:43.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.542+0000 7f0045dff700 1 -- start start 2023-03-03T07:56:43.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.544+0000 7f0045dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00400b8ae0 0x7f0040157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:43.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.544+0000 7f0045dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00400cde20 0x7f00401f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:43.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.544+0000 7f0045dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00400d2a60 0x7f00401f2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:43.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.544+0000 7f0045dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00401e6cb0 con 0x7f00400cde20 2023-03-03T07:56:43.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.544+0000 7f0045dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00401e6f50 con 0x7f00400b8ae0 2023-03-03T07:56:43.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.544+0000 7f0045dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00401e7230 con 0x7f00400d2a60 2023-03-03T07:56:43.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.544+0000 7f003effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00400cde20 0x7f00401f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:43.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.544+0000 7f003f7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00400b8ae0 0x7f0040157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:43.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.544+0000 7f003f7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00400b8ae0 0x7f0040157770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:49296/0 (socket says 172.21.15.98:49296) 2023-03-03T07:56:43.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.544+0000 7f003effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00400cde20 0x7f00401f2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42356/0 (socket says 172.21.15.98:42356) 2023-03-03T07:56:43.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.544+0000 7f003f7fe700 1 -- 172.21.15.98:0/1379401480 learned_addr learned my addr 172.21.15.98:0/1379401480 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:43.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.544+0000 7f003ffff700 1 --2- 172.21.15.98:0/1379401480 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00400d2a60 0x7f00401f2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:43.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.545+0000 7f003effd700 1 -- 172.21.15.98:0/1379401480 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00400d2a60 msgr2=0x7f00401f2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:43.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.545+0000 7f003effd700 1 --2- 172.21.15.98:0/1379401480 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00400d2a60 0x7f00401f2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:43.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.545+0000 7f003effd700 1 -- 172.21.15.98:0/1379401480 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00400b8ae0 msgr2=0x7f0040157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:43.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.545+0000 7f003effd700 1 --2- 172.21.15.98:0/1379401480 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00400b8ae0 0x7f0040157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:43.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.545+0000 7f003effd700 1 -- 172.21.15.98:0/1379401480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00340094a0 con 0x7f00400cde20 2023-03-03T07:56:43.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.545+0000 7f003effd700 1 --2- 172.21.15.98:0/1379401480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00400cde20 0x7f00401f2760 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0034016000 tx=0x7f0034004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:43.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.545+0000 7f0027fff700 1 -- 172.21.15.98:0/1379401480 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00340036a0 con 0x7f00400cde20 2023-03-03T07:56:43.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.545+0000 7f0027fff700 1 -- 172.21.15.98:0/1379401480 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003401f070 con 0x7f00400cde20 2023-03-03T07:56:43.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.546+0000 7f0027fff700 1 -- 172.21.15.98:0/1379401480 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0034003a00 con 0x7f00400cde20 2023-03-03T07:56:43.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.546+0000 7f0045dff700 1 -- 172.21.15.98:0/1379401480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f004006bbd0 con 0x7f00400cde20 2023-03-03T07:56:43.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.546+0000 7f0045dff700 1 -- 172.21.15.98:0/1379401480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00400d2060 con 0x7f00400cde20 2023-03-03T07:56:43.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.547+0000 7f0045dff700 1 -- 172.21.15.98:0/1379401480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f004004e6e0 con 0x7f00400cde20 2023-03-03T07:56:43.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.548+0000 7f0027fff700 1 -- 172.21.15.98:0/1379401480 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f003400f050 con 0x7f00400cde20 2023-03-03T07:56:43.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.549+0000 7f0027fff700 1 --2- 172.21.15.98:0/1379401480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f002c0682f0 0x7f002c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:43.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.549+0000 7f0027fff700 1 -- 172.21.15.98:0/1379401480 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f0034021070 con 0x7f00400cde20 2023-03-03T07:56:43.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.550+0000 7f003f7fe700 1 --2- 172.21.15.98:0/1379401480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f002c0682f0 0x7f002c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:43.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.550+0000 7f003f7fe700 1 --2- 172.21.15.98:0/1379401480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f002c0682f0 0x7f002c06a7a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0030001880 tx=0x7f0030009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:43.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.553+0000 7f0027fff700 1 -- 172.21.15.98:0/1379401480 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0034053d70 con 0x7f00400cde20 2023-03-03T07:56:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.747+0000 7f0045dff700 1 -- 172.21.15.98:0/1379401480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f00400d2630 con 0x7f00400cde20 2023-03-03T07:56:43.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.748+0000 7f0027fff700 1 -- 172.21.15.98:0/1379401480 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0034015020 con 0x7f00400cde20 2023-03-03T07:56:43.750 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:56:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.750+0000 7f0045dff700 1 -- 172.21.15.98:0/1379401480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f002c0682f0 msgr2=0x7f002c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.750+0000 7f0045dff700 1 --2- 172.21.15.98:0/1379401480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f002c0682f0 0x7f002c06a7a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0030001880 tx=0x7f0030009040 comp rx=0 tx=0).stop 2023-03-03T07:56:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.750+0000 7f0045dff700 1 -- 172.21.15.98:0/1379401480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00400cde20 msgr2=0x7f00401f2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.750+0000 7f0045dff700 1 --2- 172.21.15.98:0/1379401480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00400cde20 0x7f00401f2760 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0034016000 tx=0x7f0034004750 comp rx=0 tx=0).stop 2023-03-03T07:56:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.751+0000 7f0045dff700 1 -- 172.21.15.98:0/1379401480 shutdown_connections 2023-03-03T07:56:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.751+0000 7f0045dff700 1 --2- 172.21.15.98:0/1379401480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f002c0682f0 0x7f002c06a7a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.751+0000 7f0045dff700 1 --2- 172.21.15.98:0/1379401480 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00400d2a60 0x7f00401f2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.751+0000 7f0045dff700 1 --2- 172.21.15.98:0/1379401480 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00400b8ae0 0x7f0040157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.751+0000 7f0045dff700 1 --2- 172.21.15.98:0/1379401480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00400cde20 0x7f00401f2760 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.751+0000 7f0045dff700 1 -- 172.21.15.98:0/1379401480 >> 172.21.15.98:0/1379401480 conn(0x7f0040073e10 msgr2=0x7f004015cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.751+0000 7f0045dff700 1 -- 172.21.15.98:0/1379401480 shutdown_connections 2023-03-03T07:56:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:43.751+0000 7f0045dff700 1 -- 172.21.15.98:0/1379401480 wait complete. 2023-03-03T07:56:43.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:56:44.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.025+0000 7f495699d700 1 Processor -- start 2023-03-03T07:56:44.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.026+0000 7f495699d700 1 -- start start 2023-03-03T07:56:44.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.027+0000 7f495699d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f49501609b0 0x7f4950160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:44.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.027+0000 7f495699d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f49500cf570 0x7f495015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.027+0000 7f495699d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f495015db30 0x7f49500b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.027+0000 7f495699d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49500cebb0 con 0x7f49500cf570 2023-03-03T07:56:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.027+0000 7f495699d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49501612c0 con 0x7f49501609b0 2023-03-03T07:56:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.027+0000 7f495699d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49500cd780 con 0x7f495015db30 2023-03-03T07:56:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.027+0000 7f495519a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f49500cf570 0x7f495015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.027+0000 7f495519a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f49500cf570 0x7f495015d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42366/0 (socket says 172.21.15.98:42366) 2023-03-03T07:56:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.027+0000 7f495519a700 1 -- 172.21.15.98:0/3078248268 learned_addr learned my addr 172.21.15.98:0/3078248268 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.028+0000 7f495619c700 1 --2- 172.21.15.98:0/3078248268 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f495015db30 0x7f49500b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.028+0000 7f495599b700 1 --2- 172.21.15.98:0/3078248268 >> v2:172.21.15.100:3300/0 conn(0x7f49501609b0 0x7f4950160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.028+0000 7f495519a700 1 -- 172.21.15.98:0/3078248268 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f495015db30 msgr2=0x7f49500b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.028+0000 7f495519a700 1 --2- 172.21.15.98:0/3078248268 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f495015db30 0x7f49500b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.028+0000 7f495519a700 1 -- 172.21.15.98:0/3078248268 >> v2:172.21.15.100:3300/0 conn(0x7f49501609b0 msgr2=0x7f4950160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.028+0000 7f495519a700 1 --2- 172.21.15.98:0/3078248268 >> v2:172.21.15.100:3300/0 conn(0x7f49501609b0 0x7f4950160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.028+0000 7f495519a700 1 -- 172.21.15.98:0/3078248268 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49500cda60 con 0x7f49500cf570 2023-03-03T07:56:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.028+0000 7f495599b700 1 --2- 172.21.15.98:0/3078248268 >> v2:172.21.15.100:3300/0 conn(0x7f49501609b0 0x7f4950160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:56:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.028+0000 7f495619c700 1 --2- 172.21.15.98:0/3078248268 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f495015db30 0x7f49500b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:56:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.028+0000 7f495519a700 1 --2- 172.21.15.98:0/3078248268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49500cf570 0x7f495015d5f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f49500742b0 tx=0x7f4940009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd67b2a6109cde server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.028+0000 7f4954999700 1 -- 172.21.15.98:0/3078248268 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494000b870 con 0x7f49500cf570 2023-03-03T07:56:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.029+0000 7f4954999700 1 -- 172.21.15.98:0/3078248268 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494000b9d0 con 0x7f49500cf570 2023-03-03T07:56:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.029+0000 7f4954999700 1 -- 172.21.15.98:0/3078248268 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494000bd60 con 0x7f49500cf570 2023-03-03T07:56:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.029+0000 7f495699d700 1 -- 172.21.15.98:0/3078248268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49500cf570 msgr2=0x7f495015d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.029+0000 7f495699d700 1 --2- 172.21.15.98:0/3078248268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49500cf570 0x7f495015d5f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f49500742b0 tx=0x7f4940009bd0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.029+0000 7f495699d700 1 -- 172.21.15.98:0/3078248268 shutdown_connections 2023-03-03T07:56:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.029+0000 7f495699d700 1 --2- 172.21.15.98:0/3078248268 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f495015db30 0x7f49500b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.029+0000 7f495699d700 1 --2- 172.21.15.98:0/3078248268 >> v2:172.21.15.100:3300/0 conn(0x7f49501609b0 0x7f4950160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.029+0000 7f495699d700 1 --2- 172.21.15.98:0/3078248268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49500cf570 0x7f495015d5f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.029+0000 7f495699d700 1 -- 172.21.15.98:0/3078248268 >> 172.21.15.98:0/3078248268 conn(0x7f495006b880 msgr2=0x7f49500cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.029+0000 7f495699d700 1 -- 172.21.15.98:0/3078248268 shutdown_connections 2023-03-03T07:56:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.029+0000 7f495699d700 1 -- 172.21.15.98:0/3078248268 wait complete. 2023-03-03T07:56:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.031+0000 7f495699d700 1 Processor -- start 2023-03-03T07:56:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.032+0000 7f495699d700 1 -- start start 2023-03-03T07:56:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.032+0000 7f495699d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49500cf570 0x7f49500cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.033+0000 7f495699d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f495015db30 0x7f49500cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.033+0000 7f495699d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49501609b0 0x7f49500ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.033+0000 7f495699d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495014e4e0 con 0x7f49501609b0 2023-03-03T07:56:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.033+0000 7f495699d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49500ca8b0 con 0x7f495015db30 2023-03-03T07:56:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.033+0000 7f495699d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49500cab10 con 0x7f49500cf570 2023-03-03T07:56:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.033+0000 7f495619c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49501609b0 0x7f49500ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.033+0000 7f495619c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49501609b0 0x7f49500ca370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42368/0 (socket says 172.21.15.98:42368) 2023-03-03T07:56:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.033+0000 7f495619c700 1 -- 172.21.15.98:0/9369822 learned_addr learned my addr 172.21.15.98:0/9369822 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.033+0000 7f495519a700 1 --2- 172.21.15.98:0/9369822 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f495015db30 0x7f49500cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.033+0000 7f495599b700 1 --2- 172.21.15.98:0/9369822 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49500cf570 0x7f49500cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.033+0000 7f495619c700 1 -- 172.21.15.98:0/9369822 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49500cf570 msgr2=0x7f49500cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.033+0000 7f495619c700 1 --2- 172.21.15.98:0/9369822 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49500cf570 0x7f49500cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.033+0000 7f495619c700 1 -- 172.21.15.98:0/9369822 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f495015db30 msgr2=0x7f49500cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.033+0000 7f495619c700 1 --2- 172.21.15.98:0/9369822 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f495015db30 0x7f49500cbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.033+0000 7f495619c700 1 -- 172.21.15.98:0/9369822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4940009550 con 0x7f49501609b0 2023-03-03T07:56:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.034+0000 7f495619c700 1 --2- 172.21.15.98:0/9369822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49501609b0 0x7f49500ca370 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f494800cef0 tx=0x7f494800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.034+0000 7f49467fc700 1 -- 172.21.15.98:0/9369822 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494800ad30 con 0x7f49501609b0 2023-03-03T07:56:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.034+0000 7f49467fc700 1 -- 172.21.15.98:0/9369822 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494800ae90 con 0x7f49501609b0 2023-03-03T07:56:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.034+0000 7f495699d700 1 -- 172.21.15.98:0/9369822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f495015cdf0 con 0x7f49501609b0 2023-03-03T07:56:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.034+0000 7f49467fc700 1 -- 172.21.15.98:0/9369822 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4948015680 con 0x7f49501609b0 2023-03-03T07:56:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.034+0000 7f495699d700 1 -- 172.21.15.98:0/9369822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49500cdd40 con 0x7f49501609b0 2023-03-03T07:56:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.035+0000 7f495699d700 1 -- 172.21.15.98:0/9369822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f495015d0d0 con 0x7f49501609b0 2023-03-03T07:56:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.037+0000 7f49467fc700 1 -- 172.21.15.98:0/9369822 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f494801c030 con 0x7f49501609b0 2023-03-03T07:56:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.038+0000 7f49467fc700 1 --2- 172.21.15.98:0/9369822 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49340682f0 0x7f493406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.038+0000 7f49467fc700 1 -- 172.21.15.98:0/9369822 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f4948086140 con 0x7f49501609b0 2023-03-03T07:56:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.039+0000 7f495599b700 1 --2- 172.21.15.98:0/9369822 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49340682f0 0x7f493406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.039+0000 7f495599b700 1 --2- 172.21.15.98:0/9369822 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49340682f0 0x7f493406a7a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f494c006fd0 tx=0x7f494c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.042+0000 7f49467fc700 1 -- 172.21.15.98:0/9369822 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4948052870 con 0x7f49501609b0 2023-03-03T07:56:44.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.235+0000 7f495699d700 1 -- 172.21.15.98:0/9369822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4950068290 con 0x7f49501609b0 2023-03-03T07:56:44.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.236+0000 7f49467fc700 1 -- 172.21.15.98:0/9369822 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f494801a020 con 0x7f49501609b0 2023-03-03T07:56:44.239 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:56:44.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.239+0000 7f495699d700 1 -- 172.21.15.98:0/9369822 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49340682f0 msgr2=0x7f493406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:44.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.239+0000 7f495699d700 1 --2- 172.21.15.98:0/9369822 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49340682f0 0x7f493406a7a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f494c006fd0 tx=0x7f494c008040 comp rx=0 tx=0).stop 2023-03-03T07:56:44.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.239+0000 7f495699d700 1 -- 172.21.15.98:0/9369822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49501609b0 msgr2=0x7f49500ca370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:44.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.239+0000 7f495699d700 1 --2- 172.21.15.98:0/9369822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49501609b0 0x7f49500ca370 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f494800cef0 tx=0x7f494800cf20 comp rx=0 tx=0).stop 2023-03-03T07:56:44.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.239+0000 7f495699d700 1 -- 172.21.15.98:0/9369822 shutdown_connections 2023-03-03T07:56:44.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.239+0000 7f495699d700 1 --2- 172.21.15.98:0/9369822 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49340682f0 0x7f493406a7a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.239+0000 7f495699d700 1 --2- 172.21.15.98:0/9369822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49501609b0 0x7f49500ca370 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.239+0000 7f495699d700 1 --2- 172.21.15.98:0/9369822 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49500cf570 0x7f49500cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.239+0000 7f495699d700 1 --2- 172.21.15.98:0/9369822 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f495015db30 0x7f49500cbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.239+0000 7f495699d700 1 -- 172.21.15.98:0/9369822 >> 172.21.15.98:0/9369822 conn(0x7f495006b880 msgr2=0x7f49500b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:44.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.239+0000 7f495699d700 1 -- 172.21.15.98:0/9369822 shutdown_connections 2023-03-03T07:56:44.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.239+0000 7f495699d700 1 -- 172.21.15.98:0/9369822 wait complete. 2023-03-03T07:56:44.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T07:56:44.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.512+0000 7f3630fff700 1 Processor -- start 2023-03-03T07:56:44.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.512+0000 7f3630fff700 1 -- start start 2023-03-03T07:56:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.514+0000 7f3630fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f362c0b0800 0x7f362c0b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.515+0000 7f3630fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f362c0b11a0 con 0x7f362c0c93d0 2023-03-03T07:56:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.515+0000 7f3630fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f362c0c9940 con 0x7f362c15a890 2023-03-03T07:56:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.515+0000 7f3630fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f362c0b22e0 con 0x7f362c0b0800 2023-03-03T07:56:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.515+0000 7f3629d9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f362c0c93d0 0x7f362c0b1bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54118/0 (socket says 172.21.15.98:54118) 2023-03-03T07:56:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.515+0000 7f3629d9b700 1 -- 172.21.15.98:0/3949775733 learned_addr learned my addr 172.21.15.98:0/3949775733 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.515+0000 7f362ad9d700 1 --2- 172.21.15.98:0/3949775733 >> v2:172.21.15.98:3300/0 conn(0x7f362c0b0800 0x7f362c0b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.515+0000 7f362959a700 1 -- 172.21.15.98:0/3949775733 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678808614 0 0) 0x7f362c0b11a0 con 0x7f362c0c93d0 2023-03-03T07:56:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.515+0000 7f362959a700 1 -- 172.21.15.98:0/3949775733 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36140035b0 con 0x7f362c0c93d0 2023-03-03T07:56:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.516+0000 7f362ad9d700 1 -- 172.21.15.98:0/3949775733 >> v1:172.21.15.100:6789/0 conn(0x7f362c15a890 legacy=0x7f362c071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.516+0000 7f362959a700 1 -- 172.21.15.98:0/3949775733 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791423222 0 0) 0x7f362c0c9940 con 0x7f362c15a890 2023-03-03T07:56:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.516+0000 7f362ad9d700 1 -- 172.21.15.98:0/3949775733 >> v1:172.21.15.98:6789/0 conn(0x7f362c0c93d0 legacy=0x7f362c0b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:44.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.516+0000 7f362ad9d700 1 -- 172.21.15.98:0/3949775733 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362c0c9c20 con 0x7f362c0b0800 2023-03-03T07:56:44.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.516+0000 7f362ad9d700 1 --2- 172.21.15.98:0/3949775733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c0b0800 0x7f362c0b0c60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3624009d90 tx=0x7f362400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d922ec0ea2e7588e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:44.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.516+0000 7f362959a700 1 -- 172.21.15.98:0/3949775733 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362400e070 con 0x7f362c0b0800 2023-03-03T07:56:44.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.516+0000 7f362959a700 1 -- 172.21.15.98:0/3949775733 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362400bb20 con 0x7f362c0b0800 2023-03-03T07:56:44.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.516+0000 7f362959a700 1 -- 172.21.15.98:0/3949775733 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362400bed0 con 0x7f362c0b0800 2023-03-03T07:56:44.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.517+0000 7f3630fff700 1 -- 172.21.15.98:0/3949775733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c0b0800 msgr2=0x7f362c0b0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:44.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.517+0000 7f3630fff700 1 --2- 172.21.15.98:0/3949775733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c0b0800 0x7f362c0b0c60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3624009d90 tx=0x7f362400b330 comp rx=0 tx=0).stop 2023-03-03T07:56:44.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.517+0000 7f3630fff700 1 -- 172.21.15.98:0/3949775733 shutdown_connections 2023-03-03T07:56:44.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.517+0000 7f3630fff700 1 --2- 172.21.15.98:0/3949775733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c0b0800 0x7f362c0b0c60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.517+0000 7f3630fff700 1 -- 172.21.15.98:0/3949775733 >> 172.21.15.98:0/3949775733 conn(0x7f362c0cb800 msgr2=0x7f362c0cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:44.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.517+0000 7f3630fff700 1 -- 172.21.15.98:0/3949775733 shutdown_connections 2023-03-03T07:56:44.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.517+0000 7f3630fff700 1 -- 172.21.15.98:0/3949775733 wait complete. 2023-03-03T07:56:44.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.519+0000 7f3630fff700 1 Processor -- start 2023-03-03T07:56:44.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.519+0000 7f3630fff700 1 -- start start 2023-03-03T07:56:44.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.520+0000 7f3630fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c0b0800 0x7f362c151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:44.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.520+0000 7f3630fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f362c0c93d0 0x7f362c0c2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:44.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.520+0000 7f3630fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f362c15a890 0x7f362c0c32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:44.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.520+0000 7f3630fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c1485e0 con 0x7f362c0b0800 2023-03-03T07:56:44.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.520+0000 7f3630fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c152150 con 0x7f362c15a890 2023-03-03T07:56:44.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.520+0000 7f3630fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362c0c7930 con 0x7f362c0c93d0 2023-03-03T07:56:44.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.520+0000 7f362a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c0b0800 0x7f362c151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:44.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.520+0000 7f362a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c0b0800 0x7f362c151c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42384/0 (socket says 172.21.15.98:42384) 2023-03-03T07:56:44.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.520+0000 7f362a59c700 1 -- 172.21.15.98:0/2666967208 learned_addr learned my addr 172.21.15.98:0/2666967208 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:44.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.521+0000 7f3629d9b700 1 --2- 172.21.15.98:0/2666967208 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f362c0c93d0 0x7f362c0c2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:44.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.521+0000 7f362ad9d700 1 --2- 172.21.15.98:0/2666967208 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f362c15a890 0x7f362c0c32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:44.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.521+0000 7f362a59c700 1 -- 172.21.15.98:0/2666967208 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f362c0c93d0 msgr2=0x7f362c0c2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:44.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.521+0000 7f362a59c700 1 --2- 172.21.15.98:0/2666967208 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f362c0c93d0 0x7f362c0c2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.521+0000 7f362a59c700 1 -- 172.21.15.98:0/2666967208 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f362c15a890 msgr2=0x7f362c0c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:44.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.521+0000 7f362a59c700 1 --2- 172.21.15.98:0/2666967208 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f362c15a890 0x7f362c0c32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.521+0000 7f362a59c700 1 -- 172.21.15.98:0/2666967208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3624009a70 con 0x7f362c0b0800 2023-03-03T07:56:44.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.521+0000 7f362ad9d700 1 --2- 172.21.15.98:0/2666967208 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f362c15a890 0x7f362c0c32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:44.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.521+0000 7f362a59c700 1 --2- 172.21.15.98:0/2666967208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c0b0800 0x7f362c151c10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f361c00c550 tx=0x7f361c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:44.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.521+0000 7f361affd700 1 -- 172.21.15.98:0/2666967208 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c0054e0 con 0x7f362c0b0800 2023-03-03T07:56:44.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.521+0000 7f361affd700 1 -- 172.21.15.98:0/2666967208 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361c003680 con 0x7f362c0b0800 2023-03-03T07:56:44.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.521+0000 7f361affd700 1 -- 172.21.15.98:0/2666967208 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c003a30 con 0x7f362c0b0800 2023-03-03T07:56:44.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.521+0000 7f3630fff700 1 -- 172.21.15.98:0/2666967208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362c0ca1e0 con 0x7f362c0b0800 2023-03-03T07:56:44.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.522+0000 7f3630fff700 1 -- 172.21.15.98:0/2666967208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362c0c9f00 con 0x7f362c0b0800 2023-03-03T07:56:44.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.523+0000 7f3630fff700 1 -- 172.21.15.98:0/2666967208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362c0ca4c0 con 0x7f362c0b0800 2023-03-03T07:56:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.524+0000 7f361affd700 1 -- 172.21.15.98:0/2666967208 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f361c019030 con 0x7f362c0b0800 2023-03-03T07:56:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.525+0000 7f361affd700 1 --2- 172.21.15.98:0/2666967208 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f360c0682f0 0x7f360c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.525+0000 7f361affd700 1 -- 172.21.15.98:0/2666967208 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f361c085110 con 0x7f362c0b0800 2023-03-03T07:56:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.525+0000 7f3629d9b700 1 --2- 172.21.15.98:0/2666967208 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f360c0682f0 0x7f360c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.526+0000 7f3629d9b700 1 --2- 172.21.15.98:0/2666967208 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f360c0682f0 0x7f360c06a7a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f362c1f12b0 tx=0x7f3620009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.529+0000 7f361affd700 1 -- 172.21.15.98:0/2666967208 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f361c051840 con 0x7f362c0b0800 2023-03-03T07:56:44.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.741+0000 7f3630fff700 1 -- 172.21.15.98:0/2666967208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f362c0b1760 con 0x7f362c0b0800 2023-03-03T07:56:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.741+0000 7f361affd700 1 -- 172.21.15.98:0/2666967208 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7f361c017080 con 0x7f362c0b0800 2023-03-03T07:56:44.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.744+0000 7f3630fff700 1 -- 172.21.15.98:0/2666967208 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f360c0682f0 msgr2=0x7f360c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:44.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.744+0000 7f3630fff700 1 --2- 172.21.15.98:0/2666967208 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f360c0682f0 0x7f360c06a7a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f362c1f12b0 tx=0x7f3620009040 comp rx=0 tx=0).stop 2023-03-03T07:56:44.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.744+0000 7f3630fff700 1 -- 172.21.15.98:0/2666967208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c0b0800 msgr2=0x7f362c151c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:44.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.744+0000 7f3630fff700 1 --2- 172.21.15.98:0/2666967208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c0b0800 0x7f362c151c10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f361c00c550 tx=0x7f361c00c910 comp rx=0 tx=0).stop 2023-03-03T07:56:44.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.744+0000 7f3630fff700 1 -- 172.21.15.98:0/2666967208 shutdown_connections 2023-03-03T07:56:44.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.744+0000 7f3630fff700 1 --2- 172.21.15.98:0/2666967208 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f360c0682f0 0x7f360c06a7a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.744+0000 7f3630fff700 1 --2- 172.21.15.98:0/2666967208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f362c0b0800 0x7f362c151c10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.744+0000 7f3630fff700 1 --2- 172.21.15.98:0/2666967208 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f362c15a890 0x7f362c0c32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.744+0000 7f3630fff700 1 --2- 172.21.15.98:0/2666967208 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f362c0c93d0 0x7f362c0c2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.744+0000 7f3630fff700 1 -- 172.21.15.98:0/2666967208 >> 172.21.15.98:0/2666967208 conn(0x7f362c0cb800 msgr2=0x7f362c0b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.744+0000 7f3630fff700 1 -- 172.21.15.98:0/2666967208 shutdown_connections 2023-03-03T07:56:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:44.744+0000 7f3630fff700 1 -- 172.21.15.98:0/2666967208 wait complete. 2023-03-03T07:56:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T07:56:44.774 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2023-03-03T07:56:44.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:44.776 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-03-03T07:56:44.882 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T07:56:44.883 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T07:56:44.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:44.884 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-03-03T07:56:45.179 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:56:45.179 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:56:45.249 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:56:45.249 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:56:45.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:56:45.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.564+0000 7f46c61ff700 1 Processor -- start 2023-03-03T07:56:45.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.565+0000 7f46c61ff700 1 -- start start 2023-03-03T07:56:45.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.566+0000 7f46c61ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46c00b8b90 0x7f46c00cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:45.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.566+0000 7f46c61ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c00d0ef0 con 0x7f46c00b87c0 2023-03-03T07:56:45.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.566+0000 7f46c61ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c00d0600 con 0x7f46c00b5410 2023-03-03T07:56:45.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.566+0000 7f46c61ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c00cd970 con 0x7f46c00b8b90 2023-03-03T07:56:45.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.566+0000 7f46bf7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f46c00b87c0 0x7f46c00b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46874/0 (socket says 172.21.15.98:46874) 2023-03-03T07:56:45.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.566+0000 7f46bf7fe700 1 -- 172.21.15.98:0/1469080275 learned_addr learned my addr 172.21.15.98:0/1469080275 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:45.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.567+0000 7f46beffd700 1 --2- 172.21.15.98:0/1469080275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46c00b8b90 0x7f46c00cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:45.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.567+0000 7f46be7fc700 1 -- 172.21.15.98:0/1469080275 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111998395 0 0) 0x7f46c00d0ef0 con 0x7f46c00b87c0 2023-03-03T07:56:45.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.567+0000 7f46be7fc700 1 -- 172.21.15.98:0/1469080275 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a80035b0 con 0x7f46c00b87c0 2023-03-03T07:56:45.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.567+0000 7f46be7fc700 1 -- 172.21.15.98:0/1469080275 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205175262 0 0) 0x7f46c00d0600 con 0x7f46c00b5410 2023-03-03T07:56:45.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.567+0000 7f46be7fc700 1 -- 172.21.15.98:0/1469080275 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c00d0ef0 con 0x7f46c00b5410 2023-03-03T07:56:45.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.567+0000 7f46beffd700 1 -- 172.21.15.98:0/1469080275 >> v1:172.21.15.100:6789/0 conn(0x7f46c00b5410 legacy=0x7f46c00d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:45.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.567+0000 7f46beffd700 1 -- 172.21.15.98:0/1469080275 >> v1:172.21.15.98:6789/0 conn(0x7f46c00b87c0 legacy=0x7f46c00b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:45.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.567+0000 7f46beffd700 1 -- 172.21.15.98:0/1469080275 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c00cdc50 con 0x7f46c00b8b90 2023-03-03T07:56:45.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.568+0000 7f46beffd700 1 --2- 172.21.15.98:0/1469080275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46c00b8b90 0x7f46c00cffe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f46b4009870 tx=0x7f46b4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e3882854721c465 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:45.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.568+0000 7f46be7fc700 1 -- 172.21.15.98:0/1469080275 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b4011830 con 0x7f46c00b8b90 2023-03-03T07:56:45.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.568+0000 7f46be7fc700 1 -- 172.21.15.98:0/1469080275 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b4011990 con 0x7f46c00b8b90 2023-03-03T07:56:45.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.568+0000 7f46be7fc700 1 -- 172.21.15.98:0/1469080275 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b4011d20 con 0x7f46c00b8b90 2023-03-03T07:56:45.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.569+0000 7f46c61ff700 1 -- 172.21.15.98:0/1469080275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46c00b8b90 msgr2=0x7f46c00cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:45.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.569+0000 7f46c61ff700 1 --2- 172.21.15.98:0/1469080275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46c00b8b90 0x7f46c00cffe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f46b4009870 tx=0x7f46b4009b80 comp rx=0 tx=0).stop 2023-03-03T07:56:45.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.569+0000 7f46c61ff700 1 -- 172.21.15.98:0/1469080275 shutdown_connections 2023-03-03T07:56:45.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.569+0000 7f46c61ff700 1 --2- 172.21.15.98:0/1469080275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46c00b8b90 0x7f46c00cffe0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:45.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.569+0000 7f46c61ff700 1 -- 172.21.15.98:0/1469080275 >> 172.21.15.98:0/1469080275 conn(0x7f46c006aee0 msgr2=0x7f46c006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:45.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.569+0000 7f46c61ff700 1 -- 172.21.15.98:0/1469080275 shutdown_connections 2023-03-03T07:56:45.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.569+0000 7f46c61ff700 1 -- 172.21.15.98:0/1469080275 wait complete. 2023-03-03T07:56:45.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.571+0000 7f46c61ff700 1 Processor -- start 2023-03-03T07:56:45.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.571+0000 7f46c61ff700 1 -- start start 2023-03-03T07:56:45.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.572+0000 7f46c61ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46c00b5410 0x7f46c014e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:45.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.572+0000 7f46c61ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46c00b87c0 0x7f46c00ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:45.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.572+0000 7f46c61ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46c00b8b90 0x7f46c00cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:45.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.572+0000 7f46c61ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c01e5770 con 0x7f46c00b87c0 2023-03-03T07:56:45.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.572+0000 7f46c61ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c01e6c00 con 0x7f46c00b8b90 2023-03-03T07:56:45.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.572+0000 7f46c61ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c01e5490 con 0x7f46c00b5410 2023-03-03T07:56:45.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.572+0000 7f46beffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46c00b87c0 0x7f46c00ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:45.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.572+0000 7f46beffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46c00b87c0 0x7f46c00ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51490/0 (socket says 172.21.15.98:51490) 2023-03-03T07:56:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.573+0000 7f46beffd700 1 -- 172.21.15.98:0/2121952396 learned_addr learned my addr 172.21.15.98:0/2121952396 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.573+0000 7f46bf7fe700 1 --2- 172.21.15.98:0/2121952396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46c00b5410 0x7f46c014e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.573+0000 7f46bffff700 1 --2- 172.21.15.98:0/2121952396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46c00b8b90 0x7f46c00cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.573+0000 7f46beffd700 1 -- 172.21.15.98:0/2121952396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46c00b5410 msgr2=0x7f46c014e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.573+0000 7f46beffd700 1 --2- 172.21.15.98:0/2121952396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46c00b5410 0x7f46c014e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.573+0000 7f46beffd700 1 -- 172.21.15.98:0/2121952396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46c00b8b90 msgr2=0x7f46c00cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.573+0000 7f46beffd700 1 --2- 172.21.15.98:0/2121952396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46c00b8b90 0x7f46c00cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.573+0000 7f46beffd700 1 -- 172.21.15.98:0/2121952396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b40094a0 con 0x7f46c00b87c0 2023-03-03T07:56:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.573+0000 7f46beffd700 1 --2- 172.21.15.98:0/2121952396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46c00b87c0 0x7f46c00ca810 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f46b40090b0 tx=0x7f46b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.574+0000 7f46a7fff700 1 -- 172.21.15.98:0/2121952396 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b401a860 con 0x7f46c00b87c0 2023-03-03T07:56:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.574+0000 7f46a7fff700 1 -- 172.21.15.98:0/2121952396 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b401a9c0 con 0x7f46c00b87c0 2023-03-03T07:56:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.574+0000 7f46c61ff700 1 -- 172.21.15.98:0/2121952396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c00cdf30 con 0x7f46c00b87c0 2023-03-03T07:56:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.574+0000 7f46a7fff700 1 -- 172.21.15.98:0/2121952396 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b401ad30 con 0x7f46c00b87c0 2023-03-03T07:56:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.574+0000 7f46c61ff700 1 -- 172.21.15.98:0/2121952396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c0073a80 con 0x7f46c00b87c0 2023-03-03T07:56:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.575+0000 7f46c61ff700 1 -- 172.21.15.98:0/2121952396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c00ce0c0 con 0x7f46c00b87c0 2023-03-03T07:56:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.576+0000 7f46a7fff700 1 -- 172.21.15.98:0/2121952396 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f46b402b030 con 0x7f46c00b87c0 2023-03-03T07:56:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.577+0000 7f46a7fff700 1 --2- 172.21.15.98:0/2121952396 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46ac0682f0 0x7f46ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.578+0000 7f46a7fff700 1 -- 172.21.15.98:0/2121952396 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f46b4086ff0 con 0x7f46c00b87c0 2023-03-03T07:56:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.578+0000 7f46bf7fe700 1 --2- 172.21.15.98:0/2121952396 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46ac0682f0 0x7f46ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.578+0000 7f46bf7fe700 1 --2- 172.21.15.98:0/2121952396 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46ac0682f0 0x7f46ac06a7a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f46b0000f60 tx=0x7f46b0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:45.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.582+0000 7f46a7fff700 1 -- 172.21.15.98:0/2121952396 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f46b40b85e0 con 0x7f46c00b87c0 2023-03-03T07:56:45.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.778+0000 7f46c61ff700 1 -- 172.21.15.98:0/2121952396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46c00729b0 con 0x7f46c00b87c0 2023-03-03T07:56:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.779+0000 7f46a7fff700 1 -- 172.21.15.98:0/2121952396 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18003 (secure 0 0 0) 0x7f46b4029020 con 0x7f46c00b87c0 2023-03-03T07:56:45.782 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:56:45.782 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:56:27.498205+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:56:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.782+0000 7f46c61ff700 1 -- 172.21.15.98:0/2121952396 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46ac0682f0 msgr2=0x7f46ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.782+0000 7f46c61ff700 1 --2- 172.21.15.98:0/2121952396 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46ac0682f0 0x7f46ac06a7a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f46b0000f60 tx=0x7f46b0009040 comp rx=0 tx=0).stop 2023-03-03T07:56:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.782+0000 7f46c61ff700 1 -- 172.21.15.98:0/2121952396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46c00b87c0 msgr2=0x7f46c00ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.782+0000 7f46c61ff700 1 --2- 172.21.15.98:0/2121952396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46c00b87c0 0x7f46c00ca810 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f46b40090b0 tx=0x7f46b4004750 comp rx=0 tx=0).stop 2023-03-03T07:56:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.782+0000 7f46c61ff700 1 -- 172.21.15.98:0/2121952396 shutdown_connections 2023-03-03T07:56:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.782+0000 7f46c61ff700 1 --2- 172.21.15.98:0/2121952396 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46ac0682f0 0x7f46ac06a7a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.782+0000 7f46c61ff700 1 --2- 172.21.15.98:0/2121952396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46c00b8b90 0x7f46c00cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.782+0000 7f46c61ff700 1 --2- 172.21.15.98:0/2121952396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46c00b5410 0x7f46c014e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.782+0000 7f46c61ff700 1 --2- 172.21.15.98:0/2121952396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46c00b87c0 0x7f46c00ca810 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.782+0000 7f46c61ff700 1 -- 172.21.15.98:0/2121952396 >> 172.21.15.98:0/2121952396 conn(0x7f46c006aee0 msgr2=0x7f46c00d22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.782+0000 7f46c61ff700 1 -- 172.21.15.98:0/2121952396 shutdown_connections 2023-03-03T07:56:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:45.782+0000 7f46c61ff700 1 -- 172.21.15.98:0/2121952396 wait complete. 2023-03-03T07:56:45.812 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:55:45.157666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:55:46.092091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:56:45.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:56:46.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.057+0000 7f95a7dff700 1 Processor -- start 2023-03-03T07:56:46.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.057+0000 7f95a7dff700 1 -- start start 2023-03-03T07:56:46.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.058+0000 7f95a7dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a006b4b0 0x7f95a00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:46.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.058+0000 7f95a7dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a00cf980 con 0x7f95a00d0270 2023-03-03T07:56:46.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.059+0000 7f95a7dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a00d1800 con 0x7f95a015ca70 2023-03-03T07:56:46.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.059+0000 7f95a7dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00cd5b0 con 0x7f95a006b4b0 2023-03-03T07:56:46.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.059+0000 7f95a539a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f95a00d0270 0x7f95a00d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46884/0 (socket says 172.21.15.98:46884) 2023-03-03T07:56:46.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.059+0000 7f95a539a700 1 -- 172.21.15.98:0/2014272249 learned_addr learned my addr 172.21.15.98:0/2014272249 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:46.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.059+0000 7f95a639c700 1 --2- 172.21.15.98:0/2014272249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a006b4b0 0x7f95a00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:46.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.059+0000 7f95a4b99700 1 -- 172.21.15.98:0/2014272249 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724490733 0 0) 0x7f95a00cf980 con 0x7f95a00d0270 2023-03-03T07:56:46.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.059+0000 7f95a4b99700 1 -- 172.21.15.98:0/2014272249 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f958c0035b0 con 0x7f95a00d0270 2023-03-03T07:56:46.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.059+0000 7f95a4b99700 1 -- 172.21.15.98:0/2014272249 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625419593 0 0) 0x7f95a00d1800 con 0x7f95a015ca70 2023-03-03T07:56:46.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.060+0000 7f95a4b99700 1 -- 172.21.15.98:0/2014272249 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95a00cf980 con 0x7f95a015ca70 2023-03-03T07:56:46.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.060+0000 7f95a4b99700 1 -- 172.21.15.98:0/2014272249 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 488795406 0 0) 0x7f958c0035b0 con 0x7f95a00d0270 2023-03-03T07:56:46.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.060+0000 7f95a639c700 1 -- 172.21.15.98:0/2014272249 >> v1:172.21.15.100:6789/0 conn(0x7f95a015ca70 legacy=0x7f95a00cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:46.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.060+0000 7f95a639c700 1 -- 172.21.15.98:0/2014272249 >> v1:172.21.15.98:6789/0 conn(0x7f95a00d0270 legacy=0x7f95a00d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:46.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.060+0000 7f95a639c700 1 -- 172.21.15.98:0/2014272249 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a00cd900 con 0x7f95a006b4b0 2023-03-03T07:56:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.060+0000 7f95a639c700 1 --2- 172.21.15.98:0/2014272249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a006b4b0 0x7f95a00cf440 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9598009dc0 tx=0x7f959800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cae80cf165bc98e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.060+0000 7f95a4b99700 1 -- 172.21.15.98:0/2014272249 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959800e070 con 0x7f95a006b4b0 2023-03-03T07:56:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.060+0000 7f95a4b99700 1 -- 172.21.15.98:0/2014272249 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9598003a20 con 0x7f95a006b4b0 2023-03-03T07:56:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.060+0000 7f95a4b99700 1 -- 172.21.15.98:0/2014272249 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9590001670 con 0x7f95a006b4b0 2023-03-03T07:56:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.061+0000 7f95a7dff700 1 -- 172.21.15.98:0/2014272249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a006b4b0 msgr2=0x7f95a00cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.061+0000 7f95a7dff700 1 --2- 172.21.15.98:0/2014272249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a006b4b0 0x7f95a00cf440 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9598009dc0 tx=0x7f959800b330 comp rx=0 tx=0).stop 2023-03-03T07:56:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.061+0000 7f95a7dff700 1 -- 172.21.15.98:0/2014272249 shutdown_connections 2023-03-03T07:56:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.061+0000 7f95a7dff700 1 --2- 172.21.15.98:0/2014272249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a006b4b0 0x7f95a00cf440 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.061+0000 7f95a7dff700 1 -- 172.21.15.98:0/2014272249 >> 172.21.15.98:0/2014272249 conn(0x7f95a0073e70 msgr2=0x7f95a006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.061+0000 7f95a7dff700 1 -- 172.21.15.98:0/2014272249 shutdown_connections 2023-03-03T07:56:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.061+0000 7f95a7dff700 1 -- 172.21.15.98:0/2014272249 wait complete. 2023-03-03T07:56:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.063+0000 7f95a7dff700 1 Processor -- start 2023-03-03T07:56:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.063+0000 7f95a7dff700 1 -- start start 2023-03-03T07:56:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.064+0000 7f95a7dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a006b4b0 0x7f95a01576f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.064+0000 7f95a7dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95a00d0270 0x7f95a015f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.064+0000 7f95a7dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a015ca70 0x7f95a00d2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.064+0000 7f95a7dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00ca2f0 con 0x7f95a015ca70 2023-03-03T07:56:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.064+0000 7f95a7dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00ca5d0 con 0x7f95a00d0270 2023-03-03T07:56:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.064+0000 7f95a7dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00ca8b0 con 0x7f95a006b4b0 2023-03-03T07:56:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.065+0000 7f95a639c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a015ca70 0x7f95a00d2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.065+0000 7f95a639c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a015ca70 0x7f95a00d2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51500/0 (socket says 172.21.15.98:51500) 2023-03-03T07:56:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.065+0000 7f95a639c700 1 -- 172.21.15.98:0/392879035 learned_addr learned my addr 172.21.15.98:0/392879035 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.065+0000 7f95a5b9b700 1 --2- 172.21.15.98:0/392879035 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a006b4b0 0x7f95a01576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.065+0000 7f95a539a700 1 --2- 172.21.15.98:0/392879035 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95a00d0270 0x7f95a015f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.065+0000 7f95a639c700 1 -- 172.21.15.98:0/392879035 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a006b4b0 msgr2=0x7f95a01576f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.065+0000 7f95a639c700 1 --2- 172.21.15.98:0/392879035 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a006b4b0 0x7f95a01576f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.065+0000 7f95a639c700 1 -- 172.21.15.98:0/392879035 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95a00d0270 msgr2=0x7f95a015f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.065+0000 7f95a639c700 1 --2- 172.21.15.98:0/392879035 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95a00d0270 0x7f95a015f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.065+0000 7f95a639c700 1 -- 172.21.15.98:0/392879035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9598009a70 con 0x7f95a015ca70 2023-03-03T07:56:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.066+0000 7f95a639c700 1 --2- 172.21.15.98:0/392879035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a015ca70 0x7f95a00d2f10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9598009d90 tx=0x7f9598004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.066+0000 7f95967fc700 1 -- 172.21.15.98:0/392879035 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95980042c0 con 0x7f95a015ca70 2023-03-03T07:56:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.066+0000 7f95967fc700 1 -- 172.21.15.98:0/392879035 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9598004440 con 0x7f95a015ca70 2023-03-03T07:56:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.066+0000 7f95967fc700 1 -- 172.21.15.98:0/392879035 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95980047f0 con 0x7f95a015ca70 2023-03-03T07:56:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.066+0000 7f95a7dff700 1 -- 172.21.15.98:0/392879035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a00d1aa0 con 0x7f95a015ca70 2023-03-03T07:56:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.066+0000 7f95a7dff700 1 -- 172.21.15.98:0/392879035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a00cdbe0 con 0x7f95a015ca70 2023-03-03T07:56:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.067+0000 7f95a7dff700 1 -- 172.21.15.98:0/392879035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a015ce40 con 0x7f95a015ca70 2023-03-03T07:56:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.068+0000 7f95967fc700 1 -- 172.21.15.98:0/392879035 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9598018040 con 0x7f95a015ca70 2023-03-03T07:56:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.070+0000 7f95967fc700 1 --2- 172.21.15.98:0/392879035 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f95840682f0 0x7f958406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:46.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.070+0000 7f95967fc700 1 -- 172.21.15.98:0/392879035 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f95980871d0 con 0x7f95a015ca70 2023-03-03T07:56:46.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.070+0000 7f95a5b9b700 1 --2- 172.21.15.98:0/392879035 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f95840682f0 0x7f958406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:46.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.071+0000 7f95a5b9b700 1 --2- 172.21.15.98:0/392879035 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f95840682f0 0x7f958406a7a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f959c001220 tx=0x7f959c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:46.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.074+0000 7f95967fc700 1 -- 172.21.15.98:0/392879035 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9598053900 con 0x7f95a015ca70 2023-03-03T07:56:46.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.266+0000 7f95a7dff700 1 -- 172.21.15.98:0/392879035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f95a004e6e0 con 0x7f95a015ca70 2023-03-03T07:56:46.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.267+0000 7f95967fc700 1 -- 172.21.15.98:0/392879035 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9598056f20 con 0x7f95a015ca70 2023-03-03T07:56:46.712 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:56:46.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.269+0000 7f95a7dff700 1 -- 172.21.15.98:0/392879035 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f95840682f0 msgr2=0x7f958406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:46.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.269+0000 7f95a7dff700 1 --2- 172.21.15.98:0/392879035 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f95840682f0 0x7f958406a7a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f959c001220 tx=0x7f959c009040 comp rx=0 tx=0).stop 2023-03-03T07:56:46.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.269+0000 7f95a7dff700 1 -- 172.21.15.98:0/392879035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a015ca70 msgr2=0x7f95a00d2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:46.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.269+0000 7f95a7dff700 1 --2- 172.21.15.98:0/392879035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a015ca70 0x7f95a00d2f10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9598009d90 tx=0x7f9598004ad0 comp rx=0 tx=0).stop 2023-03-03T07:56:46.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.269+0000 7f95a7dff700 1 -- 172.21.15.98:0/392879035 shutdown_connections 2023-03-03T07:56:46.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.269+0000 7f95a7dff700 1 --2- 172.21.15.98:0/392879035 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f95840682f0 0x7f958406a7a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:46.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.269+0000 7f95a7dff700 1 --2- 172.21.15.98:0/392879035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a015ca70 0x7f95a00d2f10 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:46.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.269+0000 7f95a7dff700 1 --2- 172.21.15.98:0/392879035 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a006b4b0 0x7f95a01576f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:46.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.269+0000 7f95a7dff700 1 --2- 172.21.15.98:0/392879035 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95a00d0270 0x7f95a015f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:46.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.269+0000 7f95a7dff700 1 -- 172.21.15.98:0/392879035 >> 172.21.15.98:0/392879035 conn(0x7f95a0073e70 msgr2=0x7f95a00b7770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:46.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.269+0000 7f95a7dff700 1 -- 172.21.15.98:0/392879035 shutdown_connections 2023-03-03T07:56:46.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.270+0000 7f95a7dff700 1 -- 172.21.15.98:0/392879035 wait complete. 2023-03-03T07:56:46.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:56:46.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.961+0000 7fe3435ff700 1 Processor -- start 2023-03-03T07:56:46.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.961+0000 7fe3435ff700 1 -- start start 2023-03-03T07:56:46.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.962+0000 7fe3435ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe33c0b8d10 0x7fe33c0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:46.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.963+0000 7fe3435ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe33c0cd7d0 con 0x7fe33c0b26d0 2023-03-03T07:56:46.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.963+0000 7fe3435ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe33c0b90e0 con 0x7fe33c0b5310 2023-03-03T07:56:46.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.963+0000 7fe3435ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe33c0cda20 con 0x7fe33c0b8d10 2023-03-03T07:56:46.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.963+0000 7fe340b9a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe33c0b26d0 0x7fe33c0b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46896/0 (socket says 172.21.15.98:46896) 2023-03-03T07:56:46.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.963+0000 7fe340b9a700 1 -- 172.21.15.98:0/5846768 learned_addr learned my addr 172.21.15.98:0/5846768 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.963+0000 7fe34139b700 1 --2- 172.21.15.98:0/5846768 >> v2:172.21.15.100:3300/0 conn(0x7fe33c0b8d10 0x7fe33c0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.964+0000 7fe333fff700 1 -- 172.21.15.98:0/5846768 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743780605 0 0) 0x7fe33c0cd7d0 con 0x7fe33c0b26d0 2023-03-03T07:56:46.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.964+0000 7fe333fff700 1 -- 172.21.15.98:0/5846768 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3280035b0 con 0x7fe33c0b26d0 2023-03-03T07:56:46.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.964+0000 7fe333fff700 1 -- 172.21.15.98:0/5846768 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914506841 0 0) 0x7fe33c0b90e0 con 0x7fe33c0b5310 2023-03-03T07:56:46.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.964+0000 7fe333fff700 1 -- 172.21.15.98:0/5846768 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe33c0cd7d0 con 0x7fe33c0b5310 2023-03-03T07:56:46.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.964+0000 7fe34139b700 1 -- 172.21.15.98:0/5846768 >> v1:172.21.15.100:6789/0 conn(0x7fe33c0b5310 legacy=0x7fe33c0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:46.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.964+0000 7fe34139b700 1 -- 172.21.15.98:0/5846768 >> v1:172.21.15.98:6789/0 conn(0x7fe33c0b26d0 legacy=0x7fe33c0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:46.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.964+0000 7fe34139b700 1 -- 172.21.15.98:0/5846768 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c0cdd00 con 0x7fe33c0b8d10 2023-03-03T07:56:46.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.964+0000 7fe34139b700 1 --2- 172.21.15.98:0/5846768 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe33c0b8d10 0x7fe33c0cffc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe338009840 tx=0x7fe338009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3819c4bb674d74a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:46.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.964+0000 7fe333fff700 1 -- 172.21.15.98:0/5846768 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338011830 con 0x7fe33c0b8d10 2023-03-03T07:56:46.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.965+0000 7fe333fff700 1 -- 172.21.15.98:0/5846768 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe338011990 con 0x7fe33c0b8d10 2023-03-03T07:56:46.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.965+0000 7fe333fff700 1 -- 172.21.15.98:0/5846768 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338011d20 con 0x7fe33c0b8d10 2023-03-03T07:56:46.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.965+0000 7fe3435ff700 1 -- 172.21.15.98:0/5846768 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe33c0b8d10 msgr2=0x7fe33c0cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:46.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.965+0000 7fe3435ff700 1 --2- 172.21.15.98:0/5846768 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe33c0b8d10 0x7fe33c0cffc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe338009840 tx=0x7fe338009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:46.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.965+0000 7fe3435ff700 1 -- 172.21.15.98:0/5846768 shutdown_connections 2023-03-03T07:56:46.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.965+0000 7fe3435ff700 1 --2- 172.21.15.98:0/5846768 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe33c0b8d10 0x7fe33c0cffc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:46.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.965+0000 7fe3435ff700 1 -- 172.21.15.98:0/5846768 >> 172.21.15.98:0/5846768 conn(0x7fe33c073e70 msgr2=0x7fe33c06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:46.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.965+0000 7fe3435ff700 1 -- 172.21.15.98:0/5846768 shutdown_connections 2023-03-03T07:56:46.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.965+0000 7fe3435ff700 1 -- 172.21.15.98:0/5846768 wait complete. 2023-03-03T07:56:46.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.967+0000 7fe3435ff700 1 Processor -- start 2023-03-03T07:56:46.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.967+0000 7fe3435ff700 1 -- start start 2023-03-03T07:56:46.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.968+0000 7fe3435ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe33c0b26d0 0x7fe33c0c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:46.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.968+0000 7fe3435ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe33c0b5310 0x7fe33c0cbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:46.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.968+0000 7fe3435ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe33c0b8d10 0x7fe33c0cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:46.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.968+0000 7fe3435ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c0cc820 con 0x7fe33c0b26d0 2023-03-03T07:56:46.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.968+0000 7fe3435ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c0ccb00 con 0x7fe33c0b8d10 2023-03-03T07:56:46.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.968+0000 7fe3435ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c15d7e0 con 0x7fe33c0b5310 2023-03-03T07:56:46.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.969+0000 7fe34139b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe33c0b26d0 0x7fe33c0c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:46.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.969+0000 7fe34139b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe33c0b26d0 0x7fe33c0c8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51512/0 (socket says 172.21.15.98:51512) 2023-03-03T07:56:46.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.969+0000 7fe34139b700 1 -- 172.21.15.98:0/3188600220 learned_addr learned my addr 172.21.15.98:0/3188600220 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:46.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.969+0000 7fe341b9c700 1 --2- 172.21.15.98:0/3188600220 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe33c0b8d10 0x7fe33c0cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:46.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.969+0000 7fe340b9a700 1 --2- 172.21.15.98:0/3188600220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe33c0b5310 0x7fe33c0cbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:46.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.969+0000 7fe34139b700 1 -- 172.21.15.98:0/3188600220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe33c0b5310 msgr2=0x7fe33c0cbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:46.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.969+0000 7fe34139b700 1 --2- 172.21.15.98:0/3188600220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe33c0b5310 0x7fe33c0cbcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:46.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.969+0000 7fe34139b700 1 -- 172.21.15.98:0/3188600220 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe33c0b8d10 msgr2=0x7fe33c0cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:46.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.969+0000 7fe34139b700 1 --2- 172.21.15.98:0/3188600220 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe33c0b8d10 0x7fe33c0cc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:46.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.969+0000 7fe34139b700 1 -- 172.21.15.98:0/3188600220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3380094a0 con 0x7fe33c0b26d0 2023-03-03T07:56:46.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.970+0000 7fe34139b700 1 --2- 172.21.15.98:0/3188600220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe33c0b26d0 0x7fe33c0c8ac0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe3380090b0 tx=0x7fe338004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:46.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.970+0000 7fe331ffb700 1 -- 172.21.15.98:0/3188600220 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33801a860 con 0x7fe33c0b26d0 2023-03-03T07:56:46.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.970+0000 7fe331ffb700 1 -- 172.21.15.98:0/3188600220 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33801a9c0 con 0x7fe33c0b26d0 2023-03-03T07:56:46.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.970+0000 7fe3435ff700 1 -- 172.21.15.98:0/3188600220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33c0cdfe0 con 0x7fe33c0b26d0 2023-03-03T07:56:46.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.970+0000 7fe331ffb700 1 -- 172.21.15.98:0/3188600220 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33801ad30 con 0x7fe33c0b26d0 2023-03-03T07:56:46.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.970+0000 7fe3435ff700 1 -- 172.21.15.98:0/3188600220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33c0d0e80 con 0x7fe33c0b26d0 2023-03-03T07:56:46.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.971+0000 7fe3435ff700 1 -- 172.21.15.98:0/3188600220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe33c04e6e0 con 0x7fe33c0b26d0 2023-03-03T07:56:46.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.972+0000 7fe331ffb700 1 -- 172.21.15.98:0/3188600220 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe33802b030 con 0x7fe33c0b26d0 2023-03-03T07:56:46.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.974+0000 7fe331ffb700 1 --2- 172.21.15.98:0/3188600220 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3200682f0 0x7fe32006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:46.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.974+0000 7fe331ffb700 1 -- 172.21.15.98:0/3188600220 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fe3380870e0 con 0x7fe33c0b26d0 2023-03-03T07:56:46.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.974+0000 7fe340b9a700 1 --2- 172.21.15.98:0/3188600220 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3200682f0 0x7fe32006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:46.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.975+0000 7fe340b9a700 1 --2- 172.21.15.98:0/3188600220 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3200682f0 0x7fe32006a7a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe33c154fb0 tx=0x7fe32c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:46.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:46.979+0000 7fe331ffb700 1 -- 172.21.15.98:0/3188600220 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe338053810 con 0x7fe33c0b26d0 2023-03-03T07:56:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.171+0000 7fe3435ff700 1 -- 172.21.15.98:0/3188600220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe33c0d0480 con 0x7fe33c0b26d0 2023-03-03T07:56:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.172+0000 7fe331ffb700 1 -- 172.21.15.98:0/3188600220 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe338029020 con 0x7fe33c0b26d0 2023-03-03T07:56:47.175 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:56:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.175+0000 7fe3435ff700 1 -- 172.21.15.98:0/3188600220 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3200682f0 msgr2=0x7fe32006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.175+0000 7fe3435ff700 1 --2- 172.21.15.98:0/3188600220 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3200682f0 0x7fe32006a7a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe33c154fb0 tx=0x7fe32c009040 comp rx=0 tx=0).stop 2023-03-03T07:56:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.175+0000 7fe3435ff700 1 -- 172.21.15.98:0/3188600220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe33c0b26d0 msgr2=0x7fe33c0c8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.175+0000 7fe3435ff700 1 --2- 172.21.15.98:0/3188600220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe33c0b26d0 0x7fe33c0c8ac0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe3380090b0 tx=0x7fe338004750 comp rx=0 tx=0).stop 2023-03-03T07:56:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.175+0000 7fe3435ff700 1 -- 172.21.15.98:0/3188600220 shutdown_connections 2023-03-03T07:56:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.175+0000 7fe3435ff700 1 --2- 172.21.15.98:0/3188600220 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3200682f0 0x7fe32006a7a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.175+0000 7fe3435ff700 1 --2- 172.21.15.98:0/3188600220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe33c0b26d0 0x7fe33c0c8ac0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.175+0000 7fe3435ff700 1 --2- 172.21.15.98:0/3188600220 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe33c0b8d10 0x7fe33c0cc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.175+0000 7fe3435ff700 1 --2- 172.21.15.98:0/3188600220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe33c0b5310 0x7fe33c0cbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.175+0000 7fe3435ff700 1 -- 172.21.15.98:0/3188600220 >> 172.21.15.98:0/3188600220 conn(0x7fe33c073e70 msgr2=0x7fe33c15c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.175+0000 7fe3435ff700 1 -- 172.21.15.98:0/3188600220 shutdown_connections 2023-03-03T07:56:47.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.175+0000 7fe3435ff700 1 -- 172.21.15.98:0/3188600220 wait complete. 2023-03-03T07:56:47.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:56:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.451+0000 7f47389ff700 1 Processor -- start 2023-03-03T07:56:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.451+0000 7f47389ff700 1 -- start start 2023-03-03T07:56:47.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.453+0000 7f47389ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f473415ca70 0x7f47340cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:47.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.453+0000 7f47389ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f473406b4b0 0x7f47340cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:47.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.453+0000 7f47389ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f47340d00a0 0x7f47340b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:47.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.453+0000 7f47389ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47340cd5b0 con 0x7f47340d00a0 2023-03-03T07:56:47.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.453+0000 7f47389ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47340cf980 con 0x7f473415ca70 2023-03-03T07:56:47.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.453+0000 7f47389ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47340d0510 con 0x7f473406b4b0 2023-03-03T07:56:47.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.453+0000 7f4732d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f47340d00a0 0x7f47340b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:47.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.453+0000 7f4732d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f47340d00a0 0x7f47340b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51528/0 (socket says 172.21.15.98:51528) 2023-03-03T07:56:47.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.453+0000 7f4732d9d700 1 -- 172.21.15.98:0/3888415767 learned_addr learned my addr 172.21.15.98:0/3888415767 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:47.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.453+0000 7f473259c700 1 --2- 172.21.15.98:0/3888415767 >> v2:172.21.15.100:3300/0 conn(0x7f473415ca70 0x7f47340cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:47.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.453+0000 7f4731d9b700 1 --2- 172.21.15.98:0/3888415767 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f473406b4b0 0x7f47340cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:47.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.454+0000 7f4732d9d700 1 -- 172.21.15.98:0/3888415767 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f473406b4b0 msgr2=0x7f47340cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:47.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.454+0000 7f4732d9d700 1 --2- 172.21.15.98:0/3888415767 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f473406b4b0 0x7f47340cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.454+0000 7f4732d9d700 1 -- 172.21.15.98:0/3888415767 >> v2:172.21.15.100:3300/0 conn(0x7f473415ca70 msgr2=0x7f47340cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:47.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.454+0000 7f4732d9d700 1 --2- 172.21.15.98:0/3888415767 >> v2:172.21.15.100:3300/0 conn(0x7f473415ca70 0x7f47340cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.454+0000 7f4732d9d700 1 -- 172.21.15.98:0/3888415767 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47340d1800 con 0x7f47340d00a0 2023-03-03T07:56:47.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.454+0000 7f4732d9d700 1 --2- 172.21.15.98:0/3888415767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47340d00a0 0x7f47340b4c00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f473406c2c0 tx=0x7f472c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feacf653200fc602 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:47.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.454+0000 7f473159a700 1 -- 172.21.15.98:0/3888415767 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472c00e070 con 0x7f47340d00a0 2023-03-03T07:56:47.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.454+0000 7f473159a700 1 -- 172.21.15.98:0/3888415767 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472c00b980 con 0x7f47340d00a0 2023-03-03T07:56:47.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.454+0000 7f473159a700 1 -- 172.21.15.98:0/3888415767 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472c00bd30 con 0x7f47340d00a0 2023-03-03T07:56:47.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.455+0000 7f47389ff700 1 -- 172.21.15.98:0/3888415767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47340d00a0 msgr2=0x7f47340b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:47.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.455+0000 7f47389ff700 1 --2- 172.21.15.98:0/3888415767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47340d00a0 0x7f47340b4c00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f473406c2c0 tx=0x7f472c00b330 comp rx=0 tx=0).stop 2023-03-03T07:56:47.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.455+0000 7f47389ff700 1 -- 172.21.15.98:0/3888415767 shutdown_connections 2023-03-03T07:56:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.455+0000 7f47389ff700 1 --2- 172.21.15.98:0/3888415767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47340d00a0 0x7f47340b4c00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.455+0000 7f47389ff700 1 --2- 172.21.15.98:0/3888415767 >> v2:172.21.15.100:3300/0 conn(0x7f473415ca70 0x7f47340cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.455+0000 7f47389ff700 1 --2- 172.21.15.98:0/3888415767 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f473406b4b0 0x7f47340cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.455+0000 7f47389ff700 1 -- 172.21.15.98:0/3888415767 >> 172.21.15.98:0/3888415767 conn(0x7f4734073e70 msgr2=0x7f473406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.455+0000 7f47389ff700 1 -- 172.21.15.98:0/3888415767 shutdown_connections 2023-03-03T07:56:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.455+0000 7f47389ff700 1 -- 172.21.15.98:0/3888415767 wait complete. 2023-03-03T07:56:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.457+0000 7f47389ff700 1 Processor -- start 2023-03-03T07:56:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.457+0000 7f47389ff700 1 -- start start 2023-03-03T07:56:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.458+0000 7f47389ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f473406b4b0 0x7f47340b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.458+0000 7f47389ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47340d00a0 0x7f4734151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f47389ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f473415ca70 0x7f473415e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f47389ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473415f360 con 0x7f473406b4b0 2023-03-03T07:56:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f47389ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473415f640 con 0x7f473415ca70 2023-03-03T07:56:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f47389ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473415f920 con 0x7f47340d00a0 2023-03-03T07:56:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f473259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f473406b4b0 0x7f47340b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f473259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f473406b4b0 0x7f47340b5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51534/0 (socket says 172.21.15.98:51534) 2023-03-03T07:56:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f473259c700 1 -- 172.21.15.98:0/2974304343 learned_addr learned my addr 172.21.15.98:0/2974304343 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f473259c700 1 -- 172.21.15.98:0/2974304343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47340d00a0 msgr2=0x7f4734151ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f4731d9b700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47340d00a0 0x7f4734151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f4732d9d700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f473415ca70 0x7f473415e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f473259c700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47340d00a0 0x7f4734151ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f473259c700 1 -- 172.21.15.98:0/2974304343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f473415ca70 msgr2=0x7f473415e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f473259c700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f473415ca70 0x7f473415e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f473259c700 1 -- 172.21.15.98:0/2974304343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c009b30 con 0x7f473406b4b0 2023-03-03T07:56:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f4731d9b700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47340d00a0 0x7f4734151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:56:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.459+0000 7f4732d9d700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f473415ca70 0x7f473415e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:56:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.460+0000 7f473259c700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f473406b4b0 0x7f47340b5400 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f472400d800 tx=0x7f472400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:47.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.460+0000 7f4722ffd700 1 -- 172.21.15.98:0/2974304343 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47240125f0 con 0x7f473406b4b0 2023-03-03T07:56:47.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.460+0000 7f4722ffd700 1 -- 172.21.15.98:0/2974304343 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4724012750 con 0x7f473406b4b0 2023-03-03T07:56:47.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.460+0000 7f4722ffd700 1 -- 172.21.15.98:0/2974304343 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4724012b00 con 0x7f473406b4b0 2023-03-03T07:56:47.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.460+0000 7f47389ff700 1 -- 172.21.15.98:0/2974304343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47340cdbe0 con 0x7f473406b4b0 2023-03-03T07:56:47.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.460+0000 7f47389ff700 1 -- 172.21.15.98:0/2974304343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47340cd900 con 0x7f473406b4b0 2023-03-03T07:56:47.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.461+0000 7f47389ff700 1 -- 172.21.15.98:0/2974304343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47340d1aa0 con 0x7f473406b4b0 2023-03-03T07:56:47.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.462+0000 7f4722ffd700 1 -- 172.21.15.98:0/2974304343 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4724012c60 con 0x7f473406b4b0 2023-03-03T07:56:47.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.464+0000 7f4722ffd700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f47140682f0 0x7f471406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.464+0000 7f4722ffd700 1 -- 172.21.15.98:0/2974304343 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f4724086260 con 0x7f473406b4b0 2023-03-03T07:56:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.464+0000 7f4731d9b700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f47140682f0 0x7f471406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.465+0000 7f4731d9b700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f47140682f0 0x7f471406a7a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f47340b6320 tx=0x7f4728008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:47.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.468+0000 7f4722ffd700 1 -- 172.21.15.98:0/2974304343 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4724052990 con 0x7f473406b4b0 2023-03-03T07:56:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.661+0000 7f47389ff700 1 -- 172.21.15.98:0/2974304343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f473404e6e0 con 0x7f473406b4b0 2023-03-03T07:56:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.661+0000 7f4722ffd700 1 -- 172.21.15.98:0/2974304343 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4724055fb0 con 0x7f473406b4b0 2023-03-03T07:56:47.664 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:56:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.664+0000 7f47389ff700 1 -- 172.21.15.98:0/2974304343 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f47140682f0 msgr2=0x7f471406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.664+0000 7f47389ff700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f47140682f0 0x7f471406a7a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f47340b6320 tx=0x7f4728008040 comp rx=0 tx=0).stop 2023-03-03T07:56:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.664+0000 7f47389ff700 1 -- 172.21.15.98:0/2974304343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f473406b4b0 msgr2=0x7f47340b5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.664+0000 7f47389ff700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f473406b4b0 0x7f47340b5400 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f472400d800 tx=0x7f472400dbc0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.664+0000 7f47389ff700 1 -- 172.21.15.98:0/2974304343 shutdown_connections 2023-03-03T07:56:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.664+0000 7f47389ff700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f47140682f0 0x7f471406a7a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.664+0000 7f47389ff700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f473406b4b0 0x7f47340b5400 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.664+0000 7f47389ff700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f473415ca70 0x7f473415e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.664+0000 7f47389ff700 1 --2- 172.21.15.98:0/2974304343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47340d00a0 0x7f4734151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.664+0000 7f47389ff700 1 -- 172.21.15.98:0/2974304343 >> 172.21.15.98:0/2974304343 conn(0x7f4734073e70 msgr2=0x7f47340b63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.665+0000 7f47389ff700 1 -- 172.21.15.98:0/2974304343 shutdown_connections 2023-03-03T07:56:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.665+0000 7f47389ff700 1 -- 172.21.15.98:0/2974304343 wait complete. 2023-03-03T07:56:47.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:56:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.936+0000 7f71771ff700 1 Processor -- start 2023-03-03T07:56:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.936+0000 7f71771ff700 1 -- start start 2023-03-03T07:56:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.938+0000 7f71771ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f71700b8d10 0x7f71700cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.938+0000 7f71771ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71700b26d0 0x7f71700b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.938+0000 7f71771ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f71700b5140 0x7f7170160520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.938+0000 7f71771ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71700b55c0 con 0x7f71700b5140 2023-03-03T07:56:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.938+0000 7f71771ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71700cd7d0 con 0x7f71700b8d10 2023-03-03T07:56:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.938+0000 7f71771ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71700b90e0 con 0x7f71700b26d0 2023-03-03T07:56:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.938+0000 7f717579c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f71700b5140 0x7f7170160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.938+0000 7f717579c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f71700b5140 0x7f7170160520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51538/0 (socket says 172.21.15.98:51538) 2023-03-03T07:56:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.938+0000 7f717579c700 1 -- 172.21.15.98:0/194269897 learned_addr learned my addr 172.21.15.98:0/194269897 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.939+0000 7f716ffff700 1 --2- 172.21.15.98:0/194269897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71700b26d0 0x7f71700b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:47.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.939+0000 7f7174f9b700 1 --2- 172.21.15.98:0/194269897 >> v2:172.21.15.100:3300/0 conn(0x7f71700b8d10 0x7f71700cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:47.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.939+0000 7f717579c700 1 -- 172.21.15.98:0/194269897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71700b26d0 msgr2=0x7f71700b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:47.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.939+0000 7f717579c700 1 --2- 172.21.15.98:0/194269897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71700b26d0 0x7f71700b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.939+0000 7f717579c700 1 -- 172.21.15.98:0/194269897 >> v2:172.21.15.100:3300/0 conn(0x7f71700b8d10 msgr2=0x7f71700cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:47.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.939+0000 7f717579c700 1 --2- 172.21.15.98:0/194269897 >> v2:172.21.15.100:3300/0 conn(0x7f71700b8d10 0x7f71700cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.939+0000 7f717579c700 1 -- 172.21.15.98:0/194269897 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71700cda20 con 0x7f71700b5140 2023-03-03T07:56:47.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.939+0000 7f717579c700 1 --2- 172.21.15.98:0/194269897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71700b5140 0x7f7170160520 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f717008c040 tx=0x7f716800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501725787eded715 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.939+0000 7f716f7fe700 1 -- 172.21.15.98:0/194269897 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716800e070 con 0x7f71700b5140 2023-03-03T07:56:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.939+0000 7f716f7fe700 1 -- 172.21.15.98:0/194269897 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f716800b9c0 con 0x7f71700b5140 2023-03-03T07:56:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.939+0000 7f716f7fe700 1 -- 172.21.15.98:0/194269897 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716800bd70 con 0x7f71700b5140 2023-03-03T07:56:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.940+0000 7f71771ff700 1 -- 172.21.15.98:0/194269897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71700b5140 msgr2=0x7f7170160520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:47.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.940+0000 7f71771ff700 1 --2- 172.21.15.98:0/194269897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71700b5140 0x7f7170160520 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f717008c040 tx=0x7f716800b330 comp rx=0 tx=0).stop 2023-03-03T07:56:47.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.940+0000 7f71771ff700 1 -- 172.21.15.98:0/194269897 shutdown_connections 2023-03-03T07:56:47.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.940+0000 7f71771ff700 1 --2- 172.21.15.98:0/194269897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71700b5140 0x7f7170160520 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.940+0000 7f71771ff700 1 --2- 172.21.15.98:0/194269897 >> v2:172.21.15.100:3300/0 conn(0x7f71700b8d10 0x7f71700cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.940+0000 7f71771ff700 1 --2- 172.21.15.98:0/194269897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71700b26d0 0x7f71700b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.940+0000 7f71771ff700 1 -- 172.21.15.98:0/194269897 >> 172.21.15.98:0/194269897 conn(0x7f7170073e70 msgr2=0x7f717006ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:47.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.940+0000 7f71771ff700 1 -- 172.21.15.98:0/194269897 shutdown_connections 2023-03-03T07:56:47.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.940+0000 7f71771ff700 1 -- 172.21.15.98:0/194269897 wait complete. 2023-03-03T07:56:47.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.942+0000 7f71771ff700 1 Processor -- start 2023-03-03T07:56:47.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.942+0000 7f71771ff700 1 -- start start 2023-03-03T07:56:47.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.943+0000 7f71771ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71700b26d0 0x7f71700cac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:47.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.943+0000 7f71771ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71700b5140 0x7f717015d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:47.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.943+0000 7f71771ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71700b8d10 0x7f717015e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:47.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.943+0000 7f71771ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71700c78d0 con 0x7f71700b5140 2023-03-03T07:56:47.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.943+0000 7f71771ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71700cb4b0 con 0x7f71700b26d0 2023-03-03T07:56:47.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.943+0000 7f71771ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71700cb790 con 0x7f71700b8d10 2023-03-03T07:56:47.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.944+0000 7f716ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71700b5140 0x7f717015d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:47.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.944+0000 7f716ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71700b5140 0x7f717015d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51542/0 (socket says 172.21.15.98:51542) 2023-03-03T07:56:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.944+0000 7f716ffff700 1 -- 172.21.15.98:0/3348845486 learned_addr learned my addr 172.21.15.98:0/3348845486 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.944+0000 7f717579c700 1 --2- 172.21.15.98:0/3348845486 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71700b8d10 0x7f717015e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.944+0000 7f7174f9b700 1 --2- 172.21.15.98:0/3348845486 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71700b26d0 0x7f71700cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.944+0000 7f716ffff700 1 -- 172.21.15.98:0/3348845486 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71700b8d10 msgr2=0x7f717015e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.944+0000 7f716ffff700 1 --2- 172.21.15.98:0/3348845486 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71700b8d10 0x7f717015e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.944+0000 7f716ffff700 1 -- 172.21.15.98:0/3348845486 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71700b26d0 msgr2=0x7f71700cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.944+0000 7f716ffff700 1 --2- 172.21.15.98:0/3348845486 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71700b26d0 0x7f71700cac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.944+0000 7f716ffff700 1 -- 172.21.15.98:0/3348845486 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7168009b30 con 0x7f71700b5140 2023-03-03T07:56:47.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.944+0000 7f717579c700 1 --2- 172.21.15.98:0/3348845486 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71700b8d10 0x7f717015e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:47.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.944+0000 7f716ffff700 1 --2- 172.21.15.98:0/3348845486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71700b5140 0x7f717015d260 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f715c00d800 tx=0x7f715c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:47.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.945+0000 7f716d7fa700 1 -- 172.21.15.98:0/3348845486 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715c0125f0 con 0x7f71700b5140 2023-03-03T07:56:47.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.945+0000 7f716d7fa700 1 -- 172.21.15.98:0/3348845486 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715c012750 con 0x7f71700b5140 2023-03-03T07:56:47.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.945+0000 7f716d7fa700 1 -- 172.21.15.98:0/3348845486 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715c012b00 con 0x7f71700b5140 2023-03-03T07:56:47.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.945+0000 7f71771ff700 1 -- 172.21.15.98:0/3348845486 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71700d0e80 con 0x7f71700b5140 2023-03-03T07:56:47.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.945+0000 7f71771ff700 1 -- 172.21.15.98:0/3348845486 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71700cdd00 con 0x7f71700b5140 2023-03-03T07:56:47.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.946+0000 7f71771ff700 1 -- 172.21.15.98:0/3348845486 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71700cdfe0 con 0x7f71700b5140 2023-03-03T07:56:47.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.947+0000 7f716d7fa700 1 -- 172.21.15.98:0/3348845486 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f715c012c60 con 0x7f71700b5140 2023-03-03T07:56:47.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.948+0000 7f716d7fa700 1 --2- 172.21.15.98:0/3348845486 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f71540682f0 0x7f715406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:47.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.948+0000 7f716d7fa700 1 -- 172.21.15.98:0/3348845486 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f715c0862b0 con 0x7f71700b5140 2023-03-03T07:56:47.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.949+0000 7f7174f9b700 1 --2- 172.21.15.98:0/3348845486 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f71540682f0 0x7f715406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:47.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.949+0000 7f7174f9b700 1 --2- 172.21.15.98:0/3348845486 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f71540682f0 0x7f715406a7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f716400b930 tx=0x7f7164008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:47.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:47.952+0000 7f716d7fa700 1 -- 172.21.15.98:0/3348845486 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f715c0529e0 con 0x7f71700b5140 2023-03-03T07:56:48.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.145+0000 7f71771ff700 1 -- 172.21.15.98:0/3348845486 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f71700c7730 con 0x7f71700b5140 2023-03-03T07:56:48.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.146+0000 7f716d7fa700 1 -- 172.21.15.98:0/3348845486 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f71700c7730 con 0x7f71700b5140 2023-03-03T07:56:48.149 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:56:48.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.148+0000 7f71771ff700 1 -- 172.21.15.98:0/3348845486 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f71540682f0 msgr2=0x7f715406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:48.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.148+0000 7f71771ff700 1 --2- 172.21.15.98:0/3348845486 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f71540682f0 0x7f715406a7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f716400b930 tx=0x7f7164008040 comp rx=0 tx=0).stop 2023-03-03T07:56:48.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.148+0000 7f71771ff700 1 -- 172.21.15.98:0/3348845486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71700b5140 msgr2=0x7f717015d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:48.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.148+0000 7f71771ff700 1 --2- 172.21.15.98:0/3348845486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71700b5140 0x7f717015d260 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f715c00d800 tx=0x7f715c00dbc0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.148+0000 7f71771ff700 1 -- 172.21.15.98:0/3348845486 shutdown_connections 2023-03-03T07:56:48.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.148+0000 7f71771ff700 1 --2- 172.21.15.98:0/3348845486 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f71540682f0 0x7f715406a7a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.148+0000 7f71771ff700 1 --2- 172.21.15.98:0/3348845486 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71700b8d10 0x7f717015e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.148+0000 7f71771ff700 1 --2- 172.21.15.98:0/3348845486 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71700b26d0 0x7f71700cac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.148+0000 7f71771ff700 1 --2- 172.21.15.98:0/3348845486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71700b5140 0x7f717015d260 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.149+0000 7f71771ff700 1 -- 172.21.15.98:0/3348845486 >> 172.21.15.98:0/3348845486 conn(0x7f7170073e70 msgr2=0x7f71700cfc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:48.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.149+0000 7f71771ff700 1 -- 172.21.15.98:0/3348845486 shutdown_connections 2023-03-03T07:56:48.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.149+0000 7f71771ff700 1 -- 172.21.15.98:0/3348845486 wait complete. 2023-03-03T07:56:48.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T07:56:48.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.420+0000 7f412f9ff700 1 Processor -- start 2023-03-03T07:56:48.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.420+0000 7f412f9ff700 1 -- start start 2023-03-03T07:56:48.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.420+0000 7f412f9ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f412815ca70 0x7f41280cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:48.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.420+0000 7f412f9ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f412806b4b0 0x7f41280cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:48.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.421+0000 7f412f9ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41280cf980 con 0x7f41280d00a0 2023-03-03T07:56:48.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.421+0000 7f412f9ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41280cd5b0 con 0x7f412806b4b0 2023-03-03T07:56:48.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.421+0000 7f412f9ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41280d0540 con 0x7f412815ca70 2023-03-03T07:56:48.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.421+0000 7f412cf9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f412806b4b0 0x7f41280cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:48.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.421+0000 7f412d79b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f412815ca70 0x7f41280cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:48.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.421+0000 7f412d79b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f412815ca70 0x7f41280cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:43696/0 (socket says 172.21.15.98:43696) 2023-03-03T07:56:48.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.421+0000 7f412cf9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f412806b4b0 0x7f41280cfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51556/0 (socket says 172.21.15.98:51556) 2023-03-03T07:56:48.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.421+0000 7f412d79b700 1 -- 172.21.15.98:0/1263608613 learned_addr learned my addr 172.21.15.98:0/1263608613 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:48.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.421+0000 7f412cf9a700 1 -- 172.21.15.98:0/1263608613 >> v2:172.21.15.100:3300/0 conn(0x7f412815ca70 msgr2=0x7f41280cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:48.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.421+0000 7f412cf9a700 1 --2- 172.21.15.98:0/1263608613 >> v2:172.21.15.100:3300/0 conn(0x7f412815ca70 0x7f41280cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.421+0000 7f412cf9a700 1 -- 172.21.15.98:0/1263608613 >> v1:172.21.15.100:6789/0 conn(0x7f41280d00a0 legacy=0x7f41280b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:48.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.422+0000 7f412cf9a700 1 -- 172.21.15.98:0/1263608613 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41280d2fd0 con 0x7f412806b4b0 2023-03-03T07:56:48.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.422+0000 7f412d79b700 1 --2- 172.21.15.98:0/1263608613 >> v2:172.21.15.100:3300/0 conn(0x7f412815ca70 0x7f41280cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:56:48.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.422+0000 7f412cf9a700 1 --2- 172.21.15.98:0/1263608613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412806b4b0 0x7f41280cfb60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f412806c2c0 tx=0x7f4118009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f449ece9f19e065b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:48.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.422+0000 7f411ffff700 1 -- 172.21.15.98:0/1263608613 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118011830 con 0x7f412806b4b0 2023-03-03T07:56:48.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.422+0000 7f411ffff700 1 -- 172.21.15.98:0/1263608613 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4118011990 con 0x7f412806b4b0 2023-03-03T07:56:48.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.422+0000 7f411ffff700 1 -- 172.21.15.98:0/1263608613 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118011d20 con 0x7f412806b4b0 2023-03-03T07:56:48.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.423+0000 7f412f9ff700 1 -- 172.21.15.98:0/1263608613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412806b4b0 msgr2=0x7f41280cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:48.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.423+0000 7f412f9ff700 1 --2- 172.21.15.98:0/1263608613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412806b4b0 0x7f41280cfb60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f412806c2c0 tx=0x7f4118009b20 comp rx=0 tx=0).stop 2023-03-03T07:56:48.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.423+0000 7f412f9ff700 1 -- 172.21.15.98:0/1263608613 shutdown_connections 2023-03-03T07:56:48.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.423+0000 7f412f9ff700 1 --2- 172.21.15.98:0/1263608613 >> v2:172.21.15.100:3300/0 conn(0x7f412815ca70 0x7f41280cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.423+0000 7f412f9ff700 1 --2- 172.21.15.98:0/1263608613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412806b4b0 0x7f41280cfb60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.423+0000 7f412f9ff700 1 -- 172.21.15.98:0/1263608613 >> 172.21.15.98:0/1263608613 conn(0x7f4128073e70 msgr2=0x7f412806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:48.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.423+0000 7f412f9ff700 1 -- 172.21.15.98:0/1263608613 shutdown_connections 2023-03-03T07:56:48.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.423+0000 7f412f9ff700 1 -- 172.21.15.98:0/1263608613 wait complete. 2023-03-03T07:56:48.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.425+0000 7f412f9ff700 1 Processor -- start 2023-03-03T07:56:48.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.425+0000 7f412f9ff700 1 -- start start 2023-03-03T07:56:48.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412f9ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f412806b4b0 0x7f41281f4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:48.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412f9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41280d00a0 0x7f41280c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:48.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412f9ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412815ca70 0x7f412815f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:48.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412f9ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41281f5430 con 0x7f412815ca70 2023-03-03T07:56:48.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412f9ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41281584c0 con 0x7f412806b4b0 2023-03-03T07:56:48.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412f9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412815cfe0 con 0x7f41280d00a0 2023-03-03T07:56:48.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412df9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412815ca70 0x7f412815f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:48.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412df9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412815ca70 0x7f412815f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51564/0 (socket says 172.21.15.98:51564) 2023-03-03T07:56:48.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412df9c700 1 -- 172.21.15.98:0/3384025223 learned_addr learned my addr 172.21.15.98:0/3384025223 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:48.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412d79b700 1 --2- 172.21.15.98:0/3384025223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f412806b4b0 0x7f41281f4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:48.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412cf9a700 1 --2- 172.21.15.98:0/3384025223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41280d00a0 0x7f41280c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:48.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412df9c700 1 -- 172.21.15.98:0/3384025223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41280d00a0 msgr2=0x7f41280c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:48.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412df9c700 1 --2- 172.21.15.98:0/3384025223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41280d00a0 0x7f41280c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412df9c700 1 -- 172.21.15.98:0/3384025223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f412806b4b0 msgr2=0x7f41281f4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412df9c700 1 --2- 172.21.15.98:0/3384025223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f412806b4b0 0x7f41281f4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412df9c700 1 -- 172.21.15.98:0/3384025223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4124009550 con 0x7f412815ca70 2023-03-03T07:56:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.427+0000 7f412df9c700 1 --2- 172.21.15.98:0/3384025223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412815ca70 0x7f412815f170 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f412000cc80 tx=0x7f412000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.428+0000 7f411dffb700 1 -- 172.21.15.98:0/3384025223 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4120003c00 con 0x7f412815ca70 2023-03-03T07:56:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.428+0000 7f411dffb700 1 -- 172.21.15.98:0/3384025223 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4120003d60 con 0x7f412815ca70 2023-03-03T07:56:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.428+0000 7f411dffb700 1 -- 172.21.15.98:0/3384025223 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412000aaa0 con 0x7f412815ca70 2023-03-03T07:56:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.428+0000 7f412f9ff700 1 -- 172.21.15.98:0/3384025223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41180094a0 con 0x7f412815ca70 2023-03-03T07:56:48.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.428+0000 7f412f9ff700 1 -- 172.21.15.98:0/3384025223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41280cd900 con 0x7f412815ca70 2023-03-03T07:56:48.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.429+0000 7f412f9ff700 1 -- 172.21.15.98:0/3384025223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41280d1800 con 0x7f412815ca70 2023-03-03T07:56:48.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.431+0000 7f411dffb700 1 -- 172.21.15.98:0/3384025223 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4120010050 con 0x7f412815ca70 2023-03-03T07:56:48.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.432+0000 7f411dffb700 1 --2- 172.21.15.98:0/3384025223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f410c0682f0 0x7f410c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:48.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.432+0000 7f411dffb700 1 -- 172.21.15.98:0/3384025223 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f4120085c00 con 0x7f412815ca70 2023-03-03T07:56:48.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.432+0000 7f412d79b700 1 --2- 172.21.15.98:0/3384025223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f410c0682f0 0x7f410c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:48.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.433+0000 7f412d79b700 1 --2- 172.21.15.98:0/3384025223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f410c0682f0 0x7f410c06a7a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f412815eee0 tx=0x7f41240092c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:48.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.437+0000 7f411dffb700 1 -- 172.21.15.98:0/3384025223 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4120052330 con 0x7f412815ca70 2023-03-03T07:56:48.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.645+0000 7f412f9ff700 1 -- 172.21.15.98:0/3384025223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f41280cdbe0 con 0x7f412815ca70 2023-03-03T07:56:48.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.645+0000 7f411dffb700 1 -- 172.21.15.98:0/3384025223 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4120055950 con 0x7f412815ca70 2023-03-03T07:56:48.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.648+0000 7f412f9ff700 1 -- 172.21.15.98:0/3384025223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f410c0682f0 msgr2=0x7f410c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:48.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.648+0000 7f412f9ff700 1 --2- 172.21.15.98:0/3384025223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f410c0682f0 0x7f410c06a7a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f412815eee0 tx=0x7f41240092c0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.648+0000 7f412f9ff700 1 -- 172.21.15.98:0/3384025223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412815ca70 msgr2=0x7f412815f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.648+0000 7f412f9ff700 1 --2- 172.21.15.98:0/3384025223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412815ca70 0x7f412815f170 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f412000cc80 tx=0x7f412000cf90 comp rx=0 tx=0).stop 2023-03-03T07:56:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.648+0000 7f412f9ff700 1 -- 172.21.15.98:0/3384025223 shutdown_connections 2023-03-03T07:56:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.648+0000 7f412f9ff700 1 --2- 172.21.15.98:0/3384025223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f410c0682f0 0x7f410c06a7a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.648+0000 7f412f9ff700 1 --2- 172.21.15.98:0/3384025223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412815ca70 0x7f412815f170 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.648+0000 7f412f9ff700 1 --2- 172.21.15.98:0/3384025223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f412806b4b0 0x7f41281f4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.648+0000 7f412f9ff700 1 --2- 172.21.15.98:0/3384025223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41280d00a0 0x7f41280c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.648+0000 7f412f9ff700 1 -- 172.21.15.98:0/3384025223 >> 172.21.15.98:0/3384025223 conn(0x7f4128073e70 msgr2=0x7f41280b80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:48.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.648+0000 7f412f9ff700 1 -- 172.21.15.98:0/3384025223 shutdown_connections 2023-03-03T07:56:48.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.648+0000 7f412f9ff700 1 -- 172.21.15.98:0/3384025223 wait complete. 2023-03-03T07:56:48.652 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T07:56:48.677 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:56:48.678 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:56:48.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:56:48.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.987+0000 7f426a3ff700 1 Processor -- start 2023-03-03T07:56:48.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.987+0000 7f426a3ff700 1 -- start start 2023-03-03T07:56:48.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.987+0000 7f426a3ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f42640cd8d0 0x7f42640b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.987+0000 7f426a3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42640b6610 0x7f42640cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.987+0000 7f426a3ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42640cdd70 con 0x7f426415cc00 2023-03-03T07:56:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.987+0000 7f426a3ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42640ce040 con 0x7f42640cd8d0 2023-03-03T07:56:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.987+0000 7f426a3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4264073be0 con 0x7f42640b6610 2023-03-03T07:56:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.988+0000 7f425bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f42640cd8d0 0x7f42640b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.988+0000 7f425bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f42640cd8d0 0x7f42640b60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51580/0 (socket says 172.21.15.98:51580) 2023-03-03T07:56:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.988+0000 7f426899c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42640b6610 0x7f42640cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.988+0000 7f425bfff700 1 -- 172.21.15.98:0/2472796640 learned_addr learned my addr 172.21.15.98:0/2472796640 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.988+0000 7f4263fff700 1 --1- 172.21.15.98:0/2472796640 >> v1:172.21.15.100:6789/0 conn(0x7f426415cc00 0x7f42640d27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:40114/0 (socket says 172.21.15.98:40114) 2023-03-03T07:56:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.988+0000 7f425bfff700 1 -- 172.21.15.98:0/2472796640 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42640b6610 msgr2=0x7f42640cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.988+0000 7f425bfff700 1 --2- 172.21.15.98:0/2472796640 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42640b6610 0x7f42640cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.988+0000 7f425bfff700 1 -- 172.21.15.98:0/2472796640 >> v1:172.21.15.100:6789/0 conn(0x7f426415cc00 legacy=0x7f42640d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.988+0000 7f425bfff700 1 -- 172.21.15.98:0/2472796640 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42640d1a00 con 0x7f42640cd8d0 2023-03-03T07:56:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.990+0000 7f425bfff700 1 --2- 172.21.15.98:0/2472796640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640cd8d0 0x7f42640b60d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f424c0098f0 tx=0x7f424c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6017f9fa8bdb1cb9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.990+0000 7f42637fe700 1 -- 172.21.15.98:0/2472796640 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c011830 con 0x7f42640cd8d0 2023-03-03T07:56:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.990+0000 7f42637fe700 1 -- 172.21.15.98:0/2472796640 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f424c011990 con 0x7f42640cd8d0 2023-03-03T07:56:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.990+0000 7f42637fe700 1 -- 172.21.15.98:0/2472796640 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c011d20 con 0x7f42640cd8d0 2023-03-03T07:56:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.990+0000 7f426a3ff700 1 -- 172.21.15.98:0/2472796640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640cd8d0 msgr2=0x7f42640b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.990+0000 7f426a3ff700 1 --2- 172.21.15.98:0/2472796640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640cd8d0 0x7f42640b60d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f424c0098f0 tx=0x7f424c009c00 comp rx=0 tx=0).stop 2023-03-03T07:56:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.991+0000 7f426a3ff700 1 -- 172.21.15.98:0/2472796640 shutdown_connections 2023-03-03T07:56:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.991+0000 7f426a3ff700 1 --2- 172.21.15.98:0/2472796640 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42640b6610 0x7f42640cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.991+0000 7f426a3ff700 1 --2- 172.21.15.98:0/2472796640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640cd8d0 0x7f42640b60d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.991+0000 7f426a3ff700 1 -- 172.21.15.98:0/2472796640 >> 172.21.15.98:0/2472796640 conn(0x7f426406aee0 msgr2=0x7f426406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:48.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.991+0000 7f426a3ff700 1 -- 172.21.15.98:0/2472796640 shutdown_connections 2023-03-03T07:56:48.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.991+0000 7f426a3ff700 1 -- 172.21.15.98:0/2472796640 wait complete. 2023-03-03T07:56:48.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.992+0000 7f426a3ff700 1 Processor -- start 2023-03-03T07:56:48.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.992+0000 7f426a3ff700 1 -- start start 2023-03-03T07:56:48.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.994+0000 7f426a3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42640b6610 0x7f42641f57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:48.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.994+0000 7f426a3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640cd8d0 0x7f426415d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:48.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.994+0000 7f426a3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f426415cc00 0x7f42640c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:48.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.994+0000 7f426a3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426415e2c0 con 0x7f42640cd8d0 2023-03-03T07:56:48.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.994+0000 7f426a3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42641f5d00 con 0x7f42640b6610 2023-03-03T07:56:49.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.994+0000 7f426a3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42640b2e80 con 0x7f426415cc00 2023-03-03T07:56:49.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.994+0000 7f425bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640cd8d0 0x7f426415d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.994+0000 7f425bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640cd8d0 0x7f426415d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51594/0 (socket says 172.21.15.98:51594) 2023-03-03T07:56:49.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.994+0000 7f425bfff700 1 -- 172.21.15.98:0/1814952274 learned_addr learned my addr 172.21.15.98:0/1814952274 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:49.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.995+0000 7f426899c700 1 --2- 172.21.15.98:0/1814952274 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f426415cc00 0x7f42640c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.995+0000 7f4263fff700 1 --2- 172.21.15.98:0/1814952274 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42640b6610 0x7f42641f57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.995+0000 7f425bfff700 1 -- 172.21.15.98:0/1814952274 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f426415cc00 msgr2=0x7f42640c8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:49.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.995+0000 7f425bfff700 1 --2- 172.21.15.98:0/1814952274 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f426415cc00 0x7f42640c8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.995+0000 7f425bfff700 1 -- 172.21.15.98:0/1814952274 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42640b6610 msgr2=0x7f42641f57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:49.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.995+0000 7f425bfff700 1 --2- 172.21.15.98:0/1814952274 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42640b6610 0x7f42641f57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.995+0000 7f425bfff700 1 -- 172.21.15.98:0/1814952274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424c009550 con 0x7f42640cd8d0 2023-03-03T07:56:49.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.995+0000 7f4263fff700 1 --2- 172.21.15.98:0/1814952274 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42640b6610 0x7f42641f57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:49.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.995+0000 7f425bfff700 1 --2- 172.21.15.98:0/1814952274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640cd8d0 0x7f426415d9b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f424c000c00 tx=0x7f424c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:49.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.995+0000 7f42617fa700 1 -- 172.21.15.98:0/1814952274 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c004480 con 0x7f42640cd8d0 2023-03-03T07:56:49.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.995+0000 7f42617fa700 1 -- 172.21.15.98:0/1814952274 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f424c01f070 con 0x7f42640cd8d0 2023-03-03T07:56:49.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.995+0000 7f426a3ff700 1 -- 172.21.15.98:0/1814952274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42640d1fc0 con 0x7f42640cd8d0 2023-03-03T07:56:49.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.995+0000 7f42617fa700 1 -- 172.21.15.98:0/1814952274 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c0038c0 con 0x7f42640cd8d0 2023-03-03T07:56:49.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.995+0000 7f426a3ff700 1 -- 172.21.15.98:0/1814952274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42640d1ce0 con 0x7f42640cd8d0 2023-03-03T07:56:49.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.997+0000 7f426a3ff700 1 -- 172.21.15.98:0/1814952274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42640d22a0 con 0x7f42640cd8d0 2023-03-03T07:56:49.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.998+0000 7f42617fa700 1 -- 172.21.15.98:0/1814952274 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f424c02a020 con 0x7f42640cd8d0 2023-03-03T07:56:49.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.998+0000 7f42617fa700 1 --2- 172.21.15.98:0/1814952274 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f42440682f0 0x7f424406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:49.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.998+0000 7f42617fa700 1 -- 172.21.15.98:0/1814952274 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f424c0873c0 con 0x7f42640cd8d0 2023-03-03T07:56:49.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:48.998+0000 7f4263fff700 1 --2- 172.21.15.98:0/1814952274 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f42440682f0 0x7f424406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.000+0000 7f4263fff700 1 --2- 172.21.15.98:0/1814952274 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f42440682f0 0x7f424406a7a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4254000f60 tx=0x7f4254007dd0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:49.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.003+0000 7f42617fa700 1 -- 172.21.15.98:0/1814952274 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f424c053af0 con 0x7f42640cd8d0 2023-03-03T07:56:49.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.201+0000 7f426a3ff700 1 -- 172.21.15.98:0/1814952274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42640729b0 con 0x7f42640cd8d0 2023-03-03T07:56:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.202+0000 7f42617fa700 1 -- 172.21.15.98:0/1814952274 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18003 (secure 0 0 0) 0x7f424c057110 con 0x7f42640cd8d0 2023-03-03T07:56:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:56:49.205 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:56:27.498205+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:56:49.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.205+0000 7f426a3ff700 1 -- 172.21.15.98:0/1814952274 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f42440682f0 msgr2=0x7f424406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:49.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.205+0000 7f426a3ff700 1 --2- 172.21.15.98:0/1814952274 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f42440682f0 0x7f424406a7a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4254000f60 tx=0x7f4254007dd0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.205+0000 7f426a3ff700 1 -- 172.21.15.98:0/1814952274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640cd8d0 msgr2=0x7f426415d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:49.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.205+0000 7f426a3ff700 1 --2- 172.21.15.98:0/1814952274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640cd8d0 0x7f426415d9b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f424c000c00 tx=0x7f424c004750 comp rx=0 tx=0).stop 2023-03-03T07:56:49.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.205+0000 7f426a3ff700 1 -- 172.21.15.98:0/1814952274 shutdown_connections 2023-03-03T07:56:49.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.205+0000 7f426a3ff700 1 --2- 172.21.15.98:0/1814952274 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f42440682f0 0x7f424406a7a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.205+0000 7f426a3ff700 1 --2- 172.21.15.98:0/1814952274 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f426415cc00 0x7f42640c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.205+0000 7f426a3ff700 1 --2- 172.21.15.98:0/1814952274 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42640b6610 0x7f42641f57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.205+0000 7f426a3ff700 1 --2- 172.21.15.98:0/1814952274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640cd8d0 0x7f426415d9b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.205+0000 7f426a3ff700 1 -- 172.21.15.98:0/1814952274 >> 172.21.15.98:0/1814952274 conn(0x7f426406aee0 msgr2=0x7f42640b6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:49.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.206+0000 7f426a3ff700 1 -- 172.21.15.98:0/1814952274 shutdown_connections 2023-03-03T07:56:49.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.206+0000 7f426a3ff700 1 -- 172.21.15.98:0/1814952274 wait complete. 2023-03-03T07:56:49.235 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:55:45.157666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:55:46.092091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:56:49.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:56:49.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.483+0000 7ffaf43ff700 1 Processor -- start 2023-03-03T07:56:49.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.483+0000 7ffaf43ff700 1 -- start start 2023-03-03T07:56:49.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.485+0000 7ffaf43ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ffaec0b8d10 0x7ffaec0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:49.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.485+0000 7ffaf43ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffaec0b26d0 0x7ffaec0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:49.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.485+0000 7ffaf43ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0cd7d0 con 0x7ffaec0b5140 2023-03-03T07:56:49.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.485+0000 7ffaf43ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaec0b55c0 con 0x7ffaec0b26d0 2023-03-03T07:56:49.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.485+0000 7ffaf43ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaec0b90e0 con 0x7ffaec0b8d10 2023-03-03T07:56:49.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.485+0000 7ffaf199a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffaec0b26d0 0x7ffaec0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.485+0000 7ffaf299c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ffaec0b5140 0x7ffaec160520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46912/0 (socket says 172.21.15.98:46912) 2023-03-03T07:56:49.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.485+0000 7ffaf299c700 1 -- 172.21.15.98:0/940126784 learned_addr learned my addr 172.21.15.98:0/940126784 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:49.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.485+0000 7ffaf219b700 1 --2- 172.21.15.98:0/940126784 >> v2:172.21.15.100:3300/0 conn(0x7ffaec0b8d10 0x7ffaec0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.486+0000 7ffaf199a700 1 -- 172.21.15.98:0/940126784 >> v2:172.21.15.100:3300/0 conn(0x7ffaec0b8d10 msgr2=0x7ffaec0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:49.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.486+0000 7ffaf199a700 1 --2- 172.21.15.98:0/940126784 >> v2:172.21.15.100:3300/0 conn(0x7ffaec0b8d10 0x7ffaec0cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.486+0000 7ffaf199a700 1 -- 172.21.15.98:0/940126784 >> v1:172.21.15.98:6789/0 conn(0x7ffaec0b5140 legacy=0x7ffaec160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:49.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.486+0000 7ffaf199a700 1 -- 172.21.15.98:0/940126784 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec0cda20 con 0x7ffaec0b26d0 2023-03-03T07:56:49.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.486+0000 7ffaf219b700 1 --2- 172.21.15.98:0/940126784 >> v2:172.21.15.100:3300/0 conn(0x7ffaec0b8d10 0x7ffaec0cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:56:49.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.486+0000 7ffaf1199700 1 -- 172.21.15.98:0/940126784 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249216092 0 0) 0x7ffaec0cd7d0 con 0x7ffaec0b5140 2023-03-03T07:56:49.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.486+0000 7ffaf199a700 1 --2- 172.21.15.98:0/940126784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec0b26d0 0x7ffaec0b4c00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ffaec08c040 tx=0x7ffadc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1a3bb6622455c4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:49.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.486+0000 7ffaf1199700 1 -- 172.21.15.98:0/940126784 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc011830 con 0x7ffaec0b26d0 2023-03-03T07:56:49.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.486+0000 7ffaf1199700 1 -- 172.21.15.98:0/940126784 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffadc011990 con 0x7ffaec0b26d0 2023-03-03T07:56:49.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.486+0000 7ffaf1199700 1 -- 172.21.15.98:0/940126784 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc011d20 con 0x7ffaec0b26d0 2023-03-03T07:56:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.487+0000 7ffaf43ff700 1 -- 172.21.15.98:0/940126784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec0b26d0 msgr2=0x7ffaec0b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.487+0000 7ffaf43ff700 1 --2- 172.21.15.98:0/940126784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec0b26d0 0x7ffaec0b4c00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ffaec08c040 tx=0x7ffadc009b20 comp rx=0 tx=0).stop 2023-03-03T07:56:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.487+0000 7ffaf43ff700 1 -- 172.21.15.98:0/940126784 shutdown_connections 2023-03-03T07:56:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.487+0000 7ffaf43ff700 1 --2- 172.21.15.98:0/940126784 >> v2:172.21.15.100:3300/0 conn(0x7ffaec0b8d10 0x7ffaec0cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.487+0000 7ffaf43ff700 1 --2- 172.21.15.98:0/940126784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec0b26d0 0x7ffaec0b4c00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.487+0000 7ffaf43ff700 1 -- 172.21.15.98:0/940126784 >> 172.21.15.98:0/940126784 conn(0x7ffaec073e70 msgr2=0x7ffaec06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:49.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.487+0000 7ffaf43ff700 1 -- 172.21.15.98:0/940126784 shutdown_connections 2023-03-03T07:56:49.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.487+0000 7ffaf43ff700 1 -- 172.21.15.98:0/940126784 wait complete. 2023-03-03T07:56:49.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.489+0000 7ffaf43ff700 1 Processor -- start 2023-03-03T07:56:49.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.489+0000 7ffaf43ff700 1 -- start start 2023-03-03T07:56:49.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.490+0000 7ffaf43ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffaec0b26d0 0x7ffaec15f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:49.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.490+0000 7ffaf43ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaec0b5140 0x7ffaec1f5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:49.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.490+0000 7ffaf43ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec0b8d10 0x7ffaec1f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:49.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.490+0000 7ffaf43ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec15eff0 con 0x7ffaec0b8d10 2023-03-03T07:56:49.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.490+0000 7ffaf43ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec15f950 con 0x7ffaec0b26d0 2023-03-03T07:56:49.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.490+0000 7ffaf43ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec15e370 con 0x7ffaec0b5140 2023-03-03T07:56:49.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.491+0000 7ffaf299c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec0b8d10 0x7ffaec1f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.491+0000 7ffaf299c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec0b8d10 0x7ffaec1f6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51608/0 (socket says 172.21.15.98:51608) 2023-03-03T07:56:49.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.491+0000 7ffaf299c700 1 -- 172.21.15.98:0/2136463796 learned_addr learned my addr 172.21.15.98:0/2136463796 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:49.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.491+0000 7ffaf219b700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffaec0b26d0 0x7ffaec15f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.491+0000 7ffaf199a700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaec0b5140 0x7ffaec1f5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.491+0000 7ffaf299c700 1 -- 172.21.15.98:0/2136463796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaec0b5140 msgr2=0x7ffaec1f5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:49.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.491+0000 7ffaf299c700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaec0b5140 0x7ffaec1f5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.491+0000 7ffaf299c700 1 -- 172.21.15.98:0/2136463796 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffaec0b26d0 msgr2=0x7ffaec15f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:49.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.491+0000 7ffaf299c700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffaec0b26d0 0x7ffaec15f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.491+0000 7ffaf299c700 1 -- 172.21.15.98:0/2136463796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae80094a0 con 0x7ffaec0b8d10 2023-03-03T07:56:49.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.491+0000 7ffaf199a700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaec0b5140 0x7ffaec1f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:49.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.491+0000 7ffaf219b700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffaec0b26d0 0x7ffaec15f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:49.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.491+0000 7ffaf299c700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec0b8d10 0x7ffaec1f6020 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ffae400cba0 tx=0x7ffae400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:49.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.492+0000 7ffae2ffd700 1 -- 172.21.15.98:0/2136463796 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae400ab80 con 0x7ffaec0b8d10 2023-03-03T07:56:49.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.492+0000 7ffae2ffd700 1 -- 172.21.15.98:0/2136463796 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae400ad00 con 0x7ffaec0b8d10 2023-03-03T07:56:49.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.492+0000 7ffae2ffd700 1 -- 172.21.15.98:0/2136463796 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae4007310 con 0x7ffaec0b8d10 2023-03-03T07:56:49.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.492+0000 7ffaf43ff700 1 -- 172.21.15.98:0/2136463796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffadc0094a0 con 0x7ffaec0b8d10 2023-03-03T07:56:49.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.492+0000 7ffaf43ff700 1 -- 172.21.15.98:0/2136463796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec0d0e80 con 0x7ffaec0b8d10 2023-03-03T07:56:49.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.493+0000 7ffaf43ff700 1 -- 172.21.15.98:0/2136463796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaec0cdd00 con 0x7ffaec0b8d10 2023-03-03T07:56:49.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.494+0000 7ffae2ffd700 1 -- 172.21.15.98:0/2136463796 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffae4010070 con 0x7ffaec0b8d10 2023-03-03T07:56:49.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.495+0000 7ffae2ffd700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffad00682f0 0x7ffad006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:49.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.495+0000 7ffae2ffd700 1 -- 172.21.15.98:0/2136463796 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7ffae4085720 con 0x7ffaec0b8d10 2023-03-03T07:56:49.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.496+0000 7ffaf219b700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffad00682f0 0x7ffad006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.496+0000 7ffaf219b700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffad00682f0 0x7ffad006a7a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009d20 tx=0x7ffae8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:49.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.499+0000 7ffae2ffd700 1 -- 172.21.15.98:0/2136463796 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffae4051e50 con 0x7ffaec0b8d10 2023-03-03T07:56:49.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.703+0000 7ffaf43ff700 1 -- 172.21.15.98:0/2136463796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffaec0cdfe0 con 0x7ffaec0b8d10 2023-03-03T07:56:49.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.703+0000 7ffae2ffd700 1 -- 172.21.15.98:0/2136463796 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffae4018020 con 0x7ffaec0b8d10 2023-03-03T07:56:49.706 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:56:49.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.706+0000 7ffaf43ff700 1 -- 172.21.15.98:0/2136463796 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffad00682f0 msgr2=0x7ffad006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:49.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.706+0000 7ffaf43ff700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffad00682f0 0x7ffad006a7a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009d20 tx=0x7ffae8008040 comp rx=0 tx=0).stop 2023-03-03T07:56:49.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.706+0000 7ffaf43ff700 1 -- 172.21.15.98:0/2136463796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec0b8d10 msgr2=0x7ffaec1f6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:49.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.706+0000 7ffaf43ff700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec0b8d10 0x7ffaec1f6020 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ffae400cba0 tx=0x7ffae400cf60 comp rx=0 tx=0).stop 2023-03-03T07:56:49.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.706+0000 7ffaf43ff700 1 -- 172.21.15.98:0/2136463796 shutdown_connections 2023-03-03T07:56:49.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.706+0000 7ffaf43ff700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffad00682f0 0x7ffad006a7a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.706+0000 7ffaf43ff700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec0b8d10 0x7ffaec1f6020 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.706+0000 7ffaf43ff700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffaec0b26d0 0x7ffaec15f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.706+0000 7ffaf43ff700 1 --2- 172.21.15.98:0/2136463796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaec0b5140 0x7ffaec1f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.706+0000 7ffaf43ff700 1 -- 172.21.15.98:0/2136463796 >> 172.21.15.98:0/2136463796 conn(0x7ffaec073e70 msgr2=0x7ffaec15c410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:49.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.707+0000 7ffaf43ff700 1 -- 172.21.15.98:0/2136463796 shutdown_connections 2023-03-03T07:56:49.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.707+0000 7ffaf43ff700 1 -- 172.21.15.98:0/2136463796 wait complete. 2023-03-03T07:56:49.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:56:49.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.984+0000 7f4906d9d700 1 Processor -- start 2023-03-03T07:56:49.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.984+0000 7f4906d9d700 1 -- start start 2023-03-03T07:56:49.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.986+0000 7f4906d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f49000cf570 0x7f490015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:49.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.986+0000 7f4906d9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f490015db30 0x7f49000b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:49.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.986+0000 7f4906d9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49000cd780 con 0x7f49001609b0 2023-03-03T07:56:49.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.986+0000 7f4906d9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49000cebb0 con 0x7f49000cf570 2023-03-03T07:56:49.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.986+0000 7f4906d9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49000cda60 con 0x7f490015db30 2023-03-03T07:56:49.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.986+0000 7f4905d9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f49001609b0 0x7f4900160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46922/0 (socket says 172.21.15.98:46922) 2023-03-03T07:56:49.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.986+0000 7f4905d9b700 1 -- 172.21.15.98:0/2094796801 learned_addr learned my addr 172.21.15.98:0/2094796801 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:49.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.986+0000 7f490559a700 1 --2- 172.21.15.98:0/2094796801 >> v2:172.21.15.98:3300/0 conn(0x7f49000cf570 0x7f490015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.986+0000 7f490659c700 1 --2- 172.21.15.98:0/2094796801 >> v2:172.21.15.100:3300/0 conn(0x7f490015db30 0x7f49000b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.987+0000 7f490559a700 1 -- 172.21.15.98:0/2094796801 >> v2:172.21.15.100:3300/0 conn(0x7f490015db30 msgr2=0x7f49000b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:49.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.987+0000 7f4904d99700 1 -- 172.21.15.98:0/2094796801 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309151372 0 0) 0x7f49000cd780 con 0x7f49001609b0 2023-03-03T07:56:49.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.987+0000 7f490559a700 1 --2- 172.21.15.98:0/2094796801 >> v2:172.21.15.100:3300/0 conn(0x7f490015db30 0x7f49000b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.987+0000 7f490559a700 1 -- 172.21.15.98:0/2094796801 >> v1:172.21.15.98:6789/0 conn(0x7f49001609b0 legacy=0x7f4900160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:49.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.987+0000 7f490559a700 1 -- 172.21.15.98:0/2094796801 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49000cdd40 con 0x7f49000cf570 2023-03-03T07:56:49.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.987+0000 7f490559a700 1 --2- 172.21.15.98:0/2094796801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49000cf570 0x7f490015d5f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f49000ca080 tx=0x7f48f0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b9c6e7ae7c9e63 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:49.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.987+0000 7f4904d99700 1 -- 172.21.15.98:0/2094796801 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0011830 con 0x7f49000cf570 2023-03-03T07:56:49.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.987+0000 7f4904d99700 1 -- 172.21.15.98:0/2094796801 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f0011990 con 0x7f49000cf570 2023-03-03T07:56:49.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.987+0000 7f4904d99700 1 -- 172.21.15.98:0/2094796801 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0011d20 con 0x7f49000cf570 2023-03-03T07:56:49.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.988+0000 7f4906d9d700 1 -- 172.21.15.98:0/2094796801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49000cf570 msgr2=0x7f490015d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:49.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.988+0000 7f4906d9d700 1 --2- 172.21.15.98:0/2094796801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49000cf570 0x7f490015d5f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f49000ca080 tx=0x7f48f0009b20 comp rx=0 tx=0).stop 2023-03-03T07:56:49.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.988+0000 7f4906d9d700 1 -- 172.21.15.98:0/2094796801 shutdown_connections 2023-03-03T07:56:49.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.988+0000 7f4906d9d700 1 --2- 172.21.15.98:0/2094796801 >> v2:172.21.15.100:3300/0 conn(0x7f490015db30 0x7f49000b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.988+0000 7f4906d9d700 1 --2- 172.21.15.98:0/2094796801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49000cf570 0x7f490015d5f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.988+0000 7f4906d9d700 1 -- 172.21.15.98:0/2094796801 >> 172.21.15.98:0/2094796801 conn(0x7f490006b880 msgr2=0x7f49000cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:49.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.988+0000 7f4906d9d700 1 -- 172.21.15.98:0/2094796801 shutdown_connections 2023-03-03T07:56:49.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.988+0000 7f4906d9d700 1 -- 172.21.15.98:0/2094796801 wait complete. 2023-03-03T07:56:49.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.990+0000 7f4906d9d700 1 Processor -- start 2023-03-03T07:56:49.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.990+0000 7f4906d9d700 1 -- start start 2023-03-03T07:56:49.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.991+0000 7f4906d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49000cf570 0x7f490014c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:49.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.991+0000 7f4906d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f490015db30 0x7f49000b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:49.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.991+0000 7f4906d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49001609b0 0x7f49000b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:49.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.991+0000 7f4906d9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490014cdb0 con 0x7f49000cf570 2023-03-03T07:56:49.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.991+0000 7f4906d9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49000cbbd0 con 0x7f49001609b0 2023-03-03T07:56:49.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.991+0000 7f4906d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49000cbeb0 con 0x7f490015db30 2023-03-03T07:56:49.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.992+0000 7f4905d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49000cf570 0x7f490014c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.992+0000 7f4905d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49000cf570 0x7f490014c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51622/0 (socket says 172.21.15.98:51622) 2023-03-03T07:56:49.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.992+0000 7f4905d9b700 1 -- 172.21.15.98:0/899044191 learned_addr learned my addr 172.21.15.98:0/899044191 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:49.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.992+0000 7f490659c700 1 --2- 172.21.15.98:0/899044191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49001609b0 0x7f49000b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.992+0000 7f490559a700 1 --2- 172.21.15.98:0/899044191 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f490015db30 0x7f49000b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.992+0000 7f4905d9b700 1 -- 172.21.15.98:0/899044191 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f490015db30 msgr2=0x7f49000b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:49.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.992+0000 7f4905d9b700 1 --2- 172.21.15.98:0/899044191 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f490015db30 0x7f49000b3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.992+0000 7f4905d9b700 1 -- 172.21.15.98:0/899044191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49001609b0 msgr2=0x7f49000b4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:49.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.992+0000 7f4905d9b700 1 --2- 172.21.15.98:0/899044191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49001609b0 0x7f49000b4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:49.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.992+0000 7f4905d9b700 1 -- 172.21.15.98:0/899044191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f00094a0 con 0x7f49000cf570 2023-03-03T07:56:49.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.992+0000 7f490659c700 1 --2- 172.21.15.98:0/899044191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49001609b0 0x7f49000b4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:56:49.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.993+0000 7f4905d9b700 1 --2- 172.21.15.98:0/899044191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49000cf570 0x7f490014c870 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00c550 tx=0x7f48fc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:49.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.993+0000 7f48f67fc700 1 -- 172.21.15.98:0/899044191 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc004c80 con 0x7f49000cf570 2023-03-03T07:56:49.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.993+0000 7f48f67fc700 1 -- 172.21.15.98:0/899044191 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48fc004e00 con 0x7f49000cf570 2023-03-03T07:56:49.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.993+0000 7f4906d9d700 1 -- 172.21.15.98:0/899044191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490015d0d0 con 0x7f49000cf570 2023-03-03T07:56:49.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.993+0000 7f48f67fc700 1 -- 172.21.15.98:0/899044191 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc007310 con 0x7f49000cf570 2023-03-03T07:56:49.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.993+0000 7f4906d9d700 1 -- 172.21.15.98:0/899044191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490015cdf0 con 0x7f49000cf570 2023-03-03T07:56:49.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.994+0000 7f4906d9d700 1 -- 172.21.15.98:0/899044191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4900068290 con 0x7f49000cf570 2023-03-03T07:56:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.995+0000 7f48f67fc700 1 -- 172.21.15.98:0/899044191 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f48fc01a020 con 0x7f49000cf570 2023-03-03T07:56:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.996+0000 7f48f67fc700 1 --2- 172.21.15.98:0/899044191 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48e40682f0 0x7f48e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.997+0000 7f48f67fc700 1 -- 172.21.15.98:0/899044191 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f48fc085860 con 0x7f49000cf570 2023-03-03T07:56:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.997+0000 7f490559a700 1 --2- 172.21.15.98:0/899044191 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48e40682f0 0x7f48e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:49.998+0000 7f490559a700 1 --2- 172.21.15.98:0/899044191 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48e40682f0 0x7f48e406a7a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f48f00090b0 tx=0x7f48f0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:50.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.001+0000 7f48f67fc700 1 -- 172.21.15.98:0/899044191 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f48fc051f90 con 0x7f49000cf570 2023-03-03T07:56:50.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.193+0000 7f4906d9d700 1 -- 172.21.15.98:0/899044191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f49000d0430 con 0x7f49000cf570 2023-03-03T07:56:50.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.193+0000 7f48f67fc700 1 -- 172.21.15.98:0/899044191 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f48fc0555b0 con 0x7f49000cf570 2023-03-03T07:56:50.195 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:56:50.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.196+0000 7f4906d9d700 1 -- 172.21.15.98:0/899044191 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48e40682f0 msgr2=0x7f48e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:50.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.196+0000 7f4906d9d700 1 --2- 172.21.15.98:0/899044191 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48e40682f0 0x7f48e406a7a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f48f00090b0 tx=0x7f48f0008030 comp rx=0 tx=0).stop 2023-03-03T07:56:50.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.196+0000 7f4906d9d700 1 -- 172.21.15.98:0/899044191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49000cf570 msgr2=0x7f490014c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:50.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.196+0000 7f4906d9d700 1 --2- 172.21.15.98:0/899044191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49000cf570 0x7f490014c870 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00c550 tx=0x7f48fc00c860 comp rx=0 tx=0).stop 2023-03-03T07:56:50.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.196+0000 7f4906d9d700 1 -- 172.21.15.98:0/899044191 shutdown_connections 2023-03-03T07:56:50.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.196+0000 7f4906d9d700 1 --2- 172.21.15.98:0/899044191 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48e40682f0 0x7f48e406a7a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.196+0000 7f4906d9d700 1 --2- 172.21.15.98:0/899044191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49000cf570 0x7f490014c870 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.196+0000 7f4906d9d700 1 --2- 172.21.15.98:0/899044191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49001609b0 0x7f49000b4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.196+0000 7f4906d9d700 1 --2- 172.21.15.98:0/899044191 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f490015db30 0x7f49000b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.196+0000 7f4906d9d700 1 -- 172.21.15.98:0/899044191 >> 172.21.15.98:0/899044191 conn(0x7f490006b880 msgr2=0x7f4900152310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.196+0000 7f4906d9d700 1 -- 172.21.15.98:0/899044191 shutdown_connections 2023-03-03T07:56:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.196+0000 7f4906d9d700 1 -- 172.21.15.98:0/899044191 wait complete. 2023-03-03T07:56:50.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:56:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.471+0000 7f8448d9d700 1 Processor -- start 2023-03-03T07:56:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.472+0000 7f8448d9d700 1 -- start start 2023-03-03T07:56:50.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.473+0000 7f8448d9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f84441609b0 0x7f8444160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:50.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.473+0000 7f8448d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f84440cf570 0x7f844415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:50.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.473+0000 7f8448d9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84441612c0 con 0x7f844415db30 2023-03-03T07:56:50.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.473+0000 7f8448d9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84440cebb0 con 0x7f84440cf570 2023-03-03T07:56:50.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.473+0000 7f8448d9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84440cd780 con 0x7f84441609b0 2023-03-03T07:56:50.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.473+0000 7f8442ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f84440cf570 0x7f844415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:50.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.473+0000 7f8442ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f84440cf570 0x7f844415d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51638/0 (socket says 172.21.15.98:51638) 2023-03-03T07:56:50.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.473+0000 7f8442ffd700 1 -- 172.21.15.98:0/1721255332 learned_addr learned my addr 172.21.15.98:0/1721255332 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:50.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.473+0000 7f84437fe700 1 --2- 172.21.15.98:0/1721255332 >> v2:172.21.15.100:3300/0 conn(0x7f84441609b0 0x7f8444160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:50.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.474+0000 7f8442ffd700 1 -- 172.21.15.98:0/1721255332 >> v2:172.21.15.100:3300/0 conn(0x7f84441609b0 msgr2=0x7f8444160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:50.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.474+0000 7f8442ffd700 1 --2- 172.21.15.98:0/1721255332 >> v2:172.21.15.100:3300/0 conn(0x7f84441609b0 0x7f8444160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.474+0000 7f8442ffd700 1 -- 172.21.15.98:0/1721255332 >> v1:172.21.15.100:6789/0 conn(0x7f844415db30 legacy=0x7f84440b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:50.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.474+0000 7f8442ffd700 1 -- 172.21.15.98:0/1721255332 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84440cda60 con 0x7f84440cf570 2023-03-03T07:56:50.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.474+0000 7f8442ffd700 1 --2- 172.21.15.98:0/1721255332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84440cf570 0x7f844415d5f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f84440742b0 tx=0x7f8438009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d2fd73f6afc23a2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:50.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.474+0000 7f84427fc700 1 -- 172.21.15.98:0/1721255332 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8438011830 con 0x7f84440cf570 2023-03-03T07:56:50.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.474+0000 7f84427fc700 1 -- 172.21.15.98:0/1721255332 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8438011990 con 0x7f84440cf570 2023-03-03T07:56:50.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.474+0000 7f84427fc700 1 -- 172.21.15.98:0/1721255332 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8438011d20 con 0x7f84440cf570 2023-03-03T07:56:50.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.475+0000 7f8448d9d700 1 -- 172.21.15.98:0/1721255332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84440cf570 msgr2=0x7f844415d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:50.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.475+0000 7f8448d9d700 1 --2- 172.21.15.98:0/1721255332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84440cf570 0x7f844415d5f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f84440742b0 tx=0x7f8438009bd0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.475+0000 7f8448d9d700 1 -- 172.21.15.98:0/1721255332 shutdown_connections 2023-03-03T07:56:50.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.475+0000 7f8448d9d700 1 --2- 172.21.15.98:0/1721255332 >> v2:172.21.15.100:3300/0 conn(0x7f84441609b0 0x7f8444160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.475+0000 7f8448d9d700 1 --2- 172.21.15.98:0/1721255332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84440cf570 0x7f844415d5f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.475+0000 7f8448d9d700 1 -- 172.21.15.98:0/1721255332 >> 172.21.15.98:0/1721255332 conn(0x7f844406b880 msgr2=0x7f84440cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:50.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.475+0000 7f8448d9d700 1 -- 172.21.15.98:0/1721255332 shutdown_connections 2023-03-03T07:56:50.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.475+0000 7f8448d9d700 1 -- 172.21.15.98:0/1721255332 wait complete. 2023-03-03T07:56:50.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.477+0000 7f8448d9d700 1 Processor -- start 2023-03-03T07:56:50.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.477+0000 7f8448d9d700 1 -- start start 2023-03-03T07:56:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.478+0000 7f8448d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84440cf570 0x7f844414bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.478+0000 7f8448d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f844415db30 0x7f84440c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.478+0000 7f8448d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84441609b0 0x7f84441e6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.478+0000 7f8448d9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84440d15d0 con 0x7f84440cf570 2023-03-03T07:56:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.478+0000 7f8448d9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84440b5300 con 0x7f84441609b0 2023-03-03T07:56:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.478+0000 7f8448d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84440c9700 con 0x7f844415db30 2023-03-03T07:56:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.479+0000 7f84437fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84440cf570 0x7f844414bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.479+0000 7f84437fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84440cf570 0x7f844414bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51644/0 (socket says 172.21.15.98:51644) 2023-03-03T07:56:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.479+0000 7f84437fe700 1 -- 172.21.15.98:0/2442342558 learned_addr learned my addr 172.21.15.98:0/2442342558 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.479+0000 7f8442ffd700 1 --2- 172.21.15.98:0/2442342558 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f844415db30 0x7f84440c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.479+0000 7f8443fff700 1 --2- 172.21.15.98:0/2442342558 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84441609b0 0x7f84441e6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.479+0000 7f84437fe700 1 -- 172.21.15.98:0/2442342558 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f844415db30 msgr2=0x7f84440c5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.479+0000 7f84437fe700 1 --2- 172.21.15.98:0/2442342558 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f844415db30 0x7f84440c5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.479+0000 7f84437fe700 1 -- 172.21.15.98:0/2442342558 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84441609b0 msgr2=0x7f84441e6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.479+0000 7f84437fe700 1 --2- 172.21.15.98:0/2442342558 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84441609b0 0x7f84441e6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.479+0000 7f84437fe700 1 -- 172.21.15.98:0/2442342558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8438009550 con 0x7f84440cf570 2023-03-03T07:56:50.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.479+0000 7f8443fff700 1 --2- 172.21.15.98:0/2442342558 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84441609b0 0x7f84441e6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:50.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.479+0000 7f84437fe700 1 --2- 172.21.15.98:0/2442342558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84440cf570 0x7f844414bf20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f843400e8a0 tx=0x7f843400ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:50.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.480+0000 7f842bfff700 1 -- 172.21.15.98:0/2442342558 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8434009590 con 0x7f84440cf570 2023-03-03T07:56:50.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.480+0000 7f842bfff700 1 -- 172.21.15.98:0/2442342558 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8434013680 con 0x7f84440cf570 2023-03-03T07:56:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.480+0000 7f842bfff700 1 -- 172.21.15.98:0/2442342558 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84340139f0 con 0x7f84440cf570 2023-03-03T07:56:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.480+0000 7f8448d9d700 1 -- 172.21.15.98:0/2442342558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844415cdf0 con 0x7f84440cf570 2023-03-03T07:56:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.480+0000 7f8448d9d700 1 -- 172.21.15.98:0/2442342558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84440cdd40 con 0x7f84440cf570 2023-03-03T07:56:50.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.481+0000 7f8448d9d700 1 -- 172.21.15.98:0/2442342558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844415d0d0 con 0x7f84440cf570 2023-03-03T07:56:50.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.482+0000 7f842bfff700 1 -- 172.21.15.98:0/2442342558 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8434019030 con 0x7f84440cf570 2023-03-03T07:56:50.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.483+0000 7f842bfff700 1 --2- 172.21.15.98:0/2442342558 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f84300682f0 0x7f843006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.483+0000 7f842bfff700 1 -- 172.21.15.98:0/2442342558 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f8434085bc0 con 0x7f84440cf570 2023-03-03T07:56:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.484+0000 7f8442ffd700 1 --2- 172.21.15.98:0/2442342558 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f84300682f0 0x7f843006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.484+0000 7f8442ffd700 1 --2- 172.21.15.98:0/2442342558 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f84300682f0 0x7f843006a7a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8438000c00 tx=0x7f8438008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:50.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.487+0000 7f842bfff700 1 -- 172.21.15.98:0/2442342558 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8434052290 con 0x7f84440cf570 2023-03-03T07:56:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.683+0000 7f8448d9d700 1 -- 172.21.15.98:0/2442342558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f84440b55e0 con 0x7f84440cf570 2023-03-03T07:56:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.683+0000 7f842bfff700 1 -- 172.21.15.98:0/2442342558 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f84340558b0 con 0x7f84440cf570 2023-03-03T07:56:50.685 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:56:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.686+0000 7f8448d9d700 1 -- 172.21.15.98:0/2442342558 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f84300682f0 msgr2=0x7f843006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.686+0000 7f8448d9d700 1 --2- 172.21.15.98:0/2442342558 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f84300682f0 0x7f843006a7a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8438000c00 tx=0x7f8438008030 comp rx=0 tx=0).stop 2023-03-03T07:56:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.686+0000 7f8448d9d700 1 -- 172.21.15.98:0/2442342558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84440cf570 msgr2=0x7f844414bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.686+0000 7f8448d9d700 1 --2- 172.21.15.98:0/2442342558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84440cf570 0x7f844414bf20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f843400e8a0 tx=0x7f843400ebb0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.686+0000 7f8448d9d700 1 -- 172.21.15.98:0/2442342558 shutdown_connections 2023-03-03T07:56:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.686+0000 7f8448d9d700 1 --2- 172.21.15.98:0/2442342558 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f84300682f0 0x7f843006a7a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.686+0000 7f8448d9d700 1 --2- 172.21.15.98:0/2442342558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84440cf570 0x7f844414bf20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.686+0000 7f8448d9d700 1 --2- 172.21.15.98:0/2442342558 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84441609b0 0x7f84441e6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.686+0000 7f8448d9d700 1 --2- 172.21.15.98:0/2442342558 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f844415db30 0x7f84440c5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.686+0000 7f8448d9d700 1 -- 172.21.15.98:0/2442342558 >> 172.21.15.98:0/2442342558 conn(0x7f844406b880 msgr2=0x7f84440d08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.686+0000 7f8448d9d700 1 -- 172.21.15.98:0/2442342558 shutdown_connections 2023-03-03T07:56:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.686+0000 7f8448d9d700 1 -- 172.21.15.98:0/2442342558 wait complete. 2023-03-03T07:56:50.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:56:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.963+0000 7f750bfff700 1 Processor -- start 2023-03-03T07:56:50.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.963+0000 7f750bfff700 1 -- start start 2023-03-03T07:56:50.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.965+0000 7f750bfff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f750c1609b0 0x7f750c160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:50.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.965+0000 7f750bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f750c0cf570 0x7f750c15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:50.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.965+0000 7f750bfff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750c1612c0 con 0x7f750c15db30 2023-03-03T07:56:50.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.965+0000 7f750bfff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750c0cebb0 con 0x7f750c0cf570 2023-03-03T07:56:50.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.965+0000 7f750bfff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750c0cd780 con 0x7f750c1609b0 2023-03-03T07:56:50.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.965+0000 7f7503fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f750c0cf570 0x7f750c15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:50.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.965+0000 7f7503fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f750c0cf570 0x7f750c15d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51648/0 (socket says 172.21.15.98:51648) 2023-03-03T07:56:50.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.965+0000 7f7503fff700 1 -- 172.21.15.98:0/3244044223 learned_addr learned my addr 172.21.15.98:0/3244044223 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:50.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.965+0000 7f750affd700 1 --2- 172.21.15.98:0/3244044223 >> v2:172.21.15.100:3300/0 conn(0x7f750c1609b0 0x7f750c160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:50.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.966+0000 7f7503fff700 1 -- 172.21.15.98:0/3244044223 >> v2:172.21.15.100:3300/0 conn(0x7f750c1609b0 msgr2=0x7f750c160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.966+0000 7f7503fff700 1 --2- 172.21.15.98:0/3244044223 >> v2:172.21.15.100:3300/0 conn(0x7f750c1609b0 0x7f750c160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.966+0000 7f7503fff700 1 -- 172.21.15.98:0/3244044223 >> v1:172.21.15.100:6789/0 conn(0x7f750c15db30 legacy=0x7f750c0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.966+0000 7f7503fff700 1 -- 172.21.15.98:0/3244044223 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750c0cda60 con 0x7f750c0cf570 2023-03-03T07:56:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.966+0000 7f7503fff700 1 --2- 172.21.15.98:0/3244044223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f750c0cf570 0x7f750c15d5f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f750c0742b0 tx=0x7f74f4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac311c1ee79b4c5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.966+0000 7f750a7fc700 1 -- 172.21.15.98:0/3244044223 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f4011830 con 0x7f750c0cf570 2023-03-03T07:56:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.966+0000 7f750a7fc700 1 -- 172.21.15.98:0/3244044223 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f4011990 con 0x7f750c0cf570 2023-03-03T07:56:50.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.966+0000 7f750a7fc700 1 -- 172.21.15.98:0/3244044223 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f4011d20 con 0x7f750c0cf570 2023-03-03T07:56:50.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.967+0000 7f750bfff700 1 -- 172.21.15.98:0/3244044223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f750c0cf570 msgr2=0x7f750c15d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:50.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.967+0000 7f750bfff700 1 --2- 172.21.15.98:0/3244044223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f750c0cf570 0x7f750c15d5f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f750c0742b0 tx=0x7f74f4009bd0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.967+0000 7f750bfff700 1 -- 172.21.15.98:0/3244044223 shutdown_connections 2023-03-03T07:56:50.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.967+0000 7f750bfff700 1 --2- 172.21.15.98:0/3244044223 >> v2:172.21.15.100:3300/0 conn(0x7f750c1609b0 0x7f750c160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.967+0000 7f750bfff700 1 --2- 172.21.15.98:0/3244044223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f750c0cf570 0x7f750c15d5f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.967+0000 7f750bfff700 1 -- 172.21.15.98:0/3244044223 >> 172.21.15.98:0/3244044223 conn(0x7f750c06b880 msgr2=0x7f750c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:50.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.967+0000 7f750bfff700 1 -- 172.21.15.98:0/3244044223 shutdown_connections 2023-03-03T07:56:50.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.967+0000 7f750bfff700 1 -- 172.21.15.98:0/3244044223 wait complete. 2023-03-03T07:56:50.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.969+0000 7f750bfff700 1 Processor -- start 2023-03-03T07:56:50.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.969+0000 7f750bfff700 1 -- start start 2023-03-03T07:56:50.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.970+0000 7f750bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f750c0cf570 0x7f750c14bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:50.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.970+0000 7f750bfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f750c15db30 0x7f750c0c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:50.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.971+0000 7f750bfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f750c1609b0 0x7f750c1e6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:50.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.971+0000 7f750bfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c0d15d0 con 0x7f750c0cf570 2023-03-03T07:56:50.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.971+0000 7f750bfff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c0b5300 con 0x7f750c15db30 2023-03-03T07:56:50.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.971+0000 7f750bfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750c0c9700 con 0x7f750c1609b0 2023-03-03T07:56:50.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.971+0000 7f750affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f750c0cf570 0x7f750c14bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:50.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.971+0000 7f750affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f750c0cf570 0x7f750c14bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51660/0 (socket says 172.21.15.98:51660) 2023-03-03T07:56:50.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.971+0000 7f750affd700 1 -- 172.21.15.98:0/574639480 learned_addr learned my addr 172.21.15.98:0/574639480 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:50.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.971+0000 7f7503fff700 1 --2- 172.21.15.98:0/574639480 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f750c15db30 0x7f750c0c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:50.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.971+0000 7f750b7fe700 1 --2- 172.21.15.98:0/574639480 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f750c1609b0 0x7f750c1e6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:50.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.971+0000 7f750affd700 1 -- 172.21.15.98:0/574639480 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f750c1609b0 msgr2=0x7f750c1e6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:50.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.971+0000 7f750affd700 1 --2- 172.21.15.98:0/574639480 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f750c1609b0 0x7f750c1e6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.971+0000 7f750affd700 1 -- 172.21.15.98:0/574639480 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f750c15db30 msgr2=0x7f750c0c5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:50.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.971+0000 7f750affd700 1 --2- 172.21.15.98:0/574639480 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f750c15db30 0x7f750c0c5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:50.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.971+0000 7f750affd700 1 -- 172.21.15.98:0/574639480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f4009550 con 0x7f750c0cf570 2023-03-03T07:56:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.972+0000 7f750affd700 1 --2- 172.21.15.98:0/574639480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f750c0cf570 0x7f750c14bf20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f74fc00e8a0 tx=0x7f74fc00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.972+0000 7f75037fe700 1 -- 172.21.15.98:0/574639480 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc009590 con 0x7f750c0cf570 2023-03-03T07:56:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.972+0000 7f75037fe700 1 -- 172.21.15.98:0/574639480 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74fc00eea0 con 0x7f750c0cf570 2023-03-03T07:56:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.972+0000 7f75037fe700 1 -- 172.21.15.98:0/574639480 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc00ca50 con 0x7f750c0cf570 2023-03-03T07:56:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.972+0000 7f750bfff700 1 -- 172.21.15.98:0/574639480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750c15cdf0 con 0x7f750c0cf570 2023-03-03T07:56:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.972+0000 7f750bfff700 1 -- 172.21.15.98:0/574639480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750c0cdd40 con 0x7f750c0cf570 2023-03-03T07:56:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.973+0000 7f750bfff700 1 -- 172.21.15.98:0/574639480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f750c15d0d0 con 0x7f750c0cf570 2023-03-03T07:56:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.974+0000 7f75037fe700 1 -- 172.21.15.98:0/574639480 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f74fc019030 con 0x7f750c0cf570 2023-03-03T07:56:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.977+0000 7f75037fe700 1 --2- 172.21.15.98:0/574639480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f74ec0682f0 0x7f74ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.977+0000 7f75037fe700 1 -- 172.21.15.98:0/574639480 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f74fc085ca0 con 0x7f750c0cf570 2023-03-03T07:56:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.977+0000 7f7503fff700 1 --2- 172.21.15.98:0/574639480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f74ec0682f0 0x7f74ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.978+0000 7f7503fff700 1 --2- 172.21.15.98:0/574639480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f74ec0682f0 0x7f74ec06a7a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f74f4000c00 tx=0x7f74f4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:50.981+0000 7f75037fe700 1 -- 172.21.15.98:0/574639480 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f74fc052370 con 0x7f750c0cf570 2023-03-03T07:56:51.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.179+0000 7f750bfff700 1 -- 172.21.15.98:0/574639480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f750c0b55e0 con 0x7f750c0cf570 2023-03-03T07:56:51.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.179+0000 7f75037fe700 1 -- 172.21.15.98:0/574639480 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f74fc0170a0 con 0x7f750c0cf570 2023-03-03T07:56:51.181 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:56:51.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.182+0000 7f750bfff700 1 -- 172.21.15.98:0/574639480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f74ec0682f0 msgr2=0x7f74ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:51.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.182+0000 7f750bfff700 1 --2- 172.21.15.98:0/574639480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f74ec0682f0 0x7f74ec06a7a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f74f4000c00 tx=0x7f74f4008030 comp rx=0 tx=0).stop 2023-03-03T07:56:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.182+0000 7f750bfff700 1 -- 172.21.15.98:0/574639480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f750c0cf570 msgr2=0x7f750c14bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.182+0000 7f750bfff700 1 --2- 172.21.15.98:0/574639480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f750c0cf570 0x7f750c14bf20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f74fc00e8a0 tx=0x7f74fc00ec60 comp rx=0 tx=0).stop 2023-03-03T07:56:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.182+0000 7f750bfff700 1 -- 172.21.15.98:0/574639480 shutdown_connections 2023-03-03T07:56:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.182+0000 7f750bfff700 1 --2- 172.21.15.98:0/574639480 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f74ec0682f0 0x7f74ec06a7a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.182+0000 7f750bfff700 1 --2- 172.21.15.98:0/574639480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f750c0cf570 0x7f750c14bf20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.182+0000 7f750bfff700 1 --2- 172.21.15.98:0/574639480 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f750c1609b0 0x7f750c1e6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.182+0000 7f750bfff700 1 --2- 172.21.15.98:0/574639480 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f750c15db30 0x7f750c0c5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.182+0000 7f750bfff700 1 -- 172.21.15.98:0/574639480 >> 172.21.15.98:0/574639480 conn(0x7f750c06b880 msgr2=0x7f750c0d08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.182+0000 7f750bfff700 1 -- 172.21.15.98:0/574639480 shutdown_connections 2023-03-03T07:56:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.182+0000 7f750bfff700 1 -- 172.21.15.98:0/574639480 wait complete. 2023-03-03T07:56:51.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T07:56:51.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.455+0000 7fe0989fe700 1 Processor -- start 2023-03-03T07:56:51.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.455+0000 7fe0989fe700 1 -- start start 2023-03-03T07:56:51.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.457+0000 7fe0989fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0941609b0 0x7fe094160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:51.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.457+0000 7fe0989fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe0940b4aa0 0x7fe0940b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:51.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.457+0000 7fe0989fe700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0941612c0 con 0x7fe0940cf570 2023-03-03T07:56:51.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.457+0000 7fe0989fe700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0940cebb0 con 0x7fe0940b4aa0 2023-03-03T07:56:51.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.457+0000 7fe0989fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940b5460 con 0x7fe0941609b0 2023-03-03T07:56:51.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.458+0000 7fe092d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0941609b0 0x7fe094160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:51.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.458+0000 7fe09359e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe0940b4aa0 0x7fe0940b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:51.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.458+0000 7fe092d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0941609b0 0x7fe094160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:34974/0 (socket says 172.21.15.98:34974) 2023-03-03T07:56:51.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.458+0000 7fe09359e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe0940b4aa0 0x7fe0940b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:43800/0 (socket says 172.21.15.98:43800) 2023-03-03T07:56:51.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.458+0000 7fe092d9d700 1 -- 172.21.15.98:0/454470324 learned_addr learned my addr 172.21.15.98:0/454470324 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:51.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.458+0000 7fe092d9d700 1 -- 172.21.15.98:0/454470324 >> v2:172.21.15.100:3300/0 conn(0x7fe0940b4aa0 msgr2=0x7fe0940b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:51.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.458+0000 7fe092d9d700 1 --2- 172.21.15.98:0/454470324 >> v2:172.21.15.100:3300/0 conn(0x7fe0940b4aa0 0x7fe0940b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:51.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.458+0000 7fe091d9b700 1 -- 172.21.15.98:0/454470324 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037108358 0 0) 0x7fe0941612c0 con 0x7fe0940cf570 2023-03-03T07:56:51.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.458+0000 7fe092d9d700 1 -- 172.21.15.98:0/454470324 >> v1:172.21.15.100:6789/0 conn(0x7fe0940cf570 legacy=0x7fe09415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:51.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.458+0000 7fe092d9d700 1 -- 172.21.15.98:0/454470324 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09415dd00 con 0x7fe0941609b0 2023-03-03T07:56:51.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.459+0000 7fe092d9d700 1 --2- 172.21.15.98:0/454470324 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0941609b0 0x7fe094160d80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe0840098f0 tx=0x7fe084009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef2c5bf2d5e3b4b6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.459+0000 7fe091d9b700 1 -- 172.21.15.98:0/454470324 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084011830 con 0x7fe0941609b0 2023-03-03T07:56:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.459+0000 7fe091d9b700 1 -- 172.21.15.98:0/454470324 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe084011990 con 0x7fe0941609b0 2023-03-03T07:56:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.459+0000 7fe091d9b700 1 -- 172.21.15.98:0/454470324 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084011d20 con 0x7fe0941609b0 2023-03-03T07:56:51.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.460+0000 7fe0989fe700 1 -- 172.21.15.98:0/454470324 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0941609b0 msgr2=0x7fe094160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:51.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.460+0000 7fe0989fe700 1 --2- 172.21.15.98:0/454470324 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0941609b0 0x7fe094160d80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe0840098f0 tx=0x7fe084009c00 comp rx=0 tx=0).stop 2023-03-03T07:56:51.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.460+0000 7fe0989fe700 1 -- 172.21.15.98:0/454470324 shutdown_connections 2023-03-03T07:56:51.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.460+0000 7fe0989fe700 1 --2- 172.21.15.98:0/454470324 >> v2:172.21.15.100:3300/0 conn(0x7fe0940b4aa0 0x7fe0940b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:51.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.460+0000 7fe0989fe700 1 --2- 172.21.15.98:0/454470324 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0941609b0 0x7fe094160d80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:51.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.460+0000 7fe0989fe700 1 -- 172.21.15.98:0/454470324 >> 172.21.15.98:0/454470324 conn(0x7fe09406b880 msgr2=0x7fe0940cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:51.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.460+0000 7fe0989fe700 1 -- 172.21.15.98:0/454470324 shutdown_connections 2023-03-03T07:56:51.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.460+0000 7fe0989fe700 1 -- 172.21.15.98:0/454470324 wait complete. 2023-03-03T07:56:51.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.462+0000 7fe0989fe700 1 Processor -- start 2023-03-03T07:56:51.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.462+0000 7fe0989fe700 1 -- start start 2023-03-03T07:56:51.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.463+0000 7fe0989fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0940b4aa0 0x7fe0940b7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:51.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.463+0000 7fe0989fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0940cf570 0x7fe0940cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.463+0000 7fe0989fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0941609b0 0x7fe0940b7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.463+0000 7fe0989fe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940b7c40 con 0x7fe0940b4aa0 2023-03-03T07:56:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.463+0000 7fe0989fe700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940cbb90 con 0x7fe0940cf570 2023-03-03T07:56:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.463+0000 7fe0989fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940cbe70 con 0x7fe0941609b0 2023-03-03T07:56:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.464+0000 7fe092d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0940b4aa0 0x7fe0940b7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.464+0000 7fe092d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0940b4aa0 0x7fe0940b7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51674/0 (socket says 172.21.15.98:51674) 2023-03-03T07:56:51.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.464+0000 7fe092d9d700 1 -- 172.21.15.98:0/2229190936 learned_addr learned my addr 172.21.15.98:0/2229190936 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:51.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.464+0000 7fe09259c700 1 --2- 172.21.15.98:0/2229190936 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0940cf570 0x7fe0940cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:51.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.464+0000 7fe09359e700 1 --2- 172.21.15.98:0/2229190936 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0941609b0 0x7fe0940b7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:51.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.464+0000 7fe092d9d700 1 -- 172.21.15.98:0/2229190936 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0941609b0 msgr2=0x7fe0940b7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:51.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.464+0000 7fe092d9d700 1 --2- 172.21.15.98:0/2229190936 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0941609b0 0x7fe0940b7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:51.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.464+0000 7fe092d9d700 1 -- 172.21.15.98:0/2229190936 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0940cf570 msgr2=0x7fe0940cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:51.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.464+0000 7fe092d9d700 1 --2- 172.21.15.98:0/2229190936 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0940cf570 0x7fe0940cb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:51.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.464+0000 7fe092d9d700 1 -- 172.21.15.98:0/2229190936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe084009550 con 0x7fe0940b4aa0 2023-03-03T07:56:51.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.464+0000 7fe09259c700 1 --2- 172.21.15.98:0/2229190936 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0940cf570 0x7fe0940cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:51.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.465+0000 7fe092d9d700 1 --2- 172.21.15.98:0/2229190936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0940b4aa0 0x7fe0940b7700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe084009c40 tx=0x7fe084004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.465+0000 7fe07b7fe700 1 -- 172.21.15.98:0/2229190936 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0840036a0 con 0x7fe0940b4aa0 2023-03-03T07:56:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.465+0000 7fe07b7fe700 1 -- 172.21.15.98:0/2229190936 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe084003800 con 0x7fe0940b4aa0 2023-03-03T07:56:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.465+0000 7fe07b7fe700 1 -- 172.21.15.98:0/2229190936 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084003b70 con 0x7fe0940b4aa0 2023-03-03T07:56:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.465+0000 7fe0989fe700 1 -- 172.21.15.98:0/2229190936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0940cda60 con 0x7fe0940b4aa0 2023-03-03T07:56:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.465+0000 7fe0989fe700 1 -- 172.21.15.98:0/2229190936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0940cd780 con 0x7fe0940b4aa0 2023-03-03T07:56:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.466+0000 7fe0989fe700 1 -- 172.21.15.98:0/2229190936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0940cdd40 con 0x7fe0940b4aa0 2023-03-03T07:56:51.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.467+0000 7fe07b7fe700 1 -- 172.21.15.98:0/2229190936 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe08402d070 con 0x7fe0940b4aa0 2023-03-03T07:56:51.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.468+0000 7fe07b7fe700 1 --2- 172.21.15.98:0/2229190936 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe0800682f0 0x7fe08006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:51.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.469+0000 7fe07b7fe700 1 -- 172.21.15.98:0/2229190936 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fe084021030 con 0x7fe0940b4aa0 2023-03-03T07:56:51.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.469+0000 7fe09259c700 1 --2- 172.21.15.98:0/2229190936 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe0800682f0 0x7fe08006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:51.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.469+0000 7fe09259c700 1 --2- 172.21.15.98:0/2229190936 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe0800682f0 0x7fe08006a7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe0940b27a0 tx=0x7fe088009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:51.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.472+0000 7fe07b7fe700 1 -- 172.21.15.98:0/2229190936 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe084053b90 con 0x7fe0940b4aa0 2023-03-03T07:56:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.681+0000 7fe0989fe700 1 -- 172.21.15.98:0/2229190936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe094068290 con 0x7fe0940b4aa0 2023-03-03T07:56:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.682+0000 7fe07b7fe700 1 -- 172.21.15.98:0/2229190936 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe0840571b0 con 0x7fe0940b4aa0 2023-03-03T07:56:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.685+0000 7fe0989fe700 1 -- 172.21.15.98:0/2229190936 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe0800682f0 msgr2=0x7fe08006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.685+0000 7fe0989fe700 1 --2- 172.21.15.98:0/2229190936 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe0800682f0 0x7fe08006a7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe0940b27a0 tx=0x7fe088009040 comp rx=0 tx=0).stop 2023-03-03T07:56:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.685+0000 7fe0989fe700 1 -- 172.21.15.98:0/2229190936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0940b4aa0 msgr2=0x7fe0940b7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.685+0000 7fe0989fe700 1 --2- 172.21.15.98:0/2229190936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0940b4aa0 0x7fe0940b7700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe084009c40 tx=0x7fe084004750 comp rx=0 tx=0).stop 2023-03-03T07:56:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.685+0000 7fe0989fe700 1 -- 172.21.15.98:0/2229190936 shutdown_connections 2023-03-03T07:56:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.685+0000 7fe0989fe700 1 --2- 172.21.15.98:0/2229190936 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe0800682f0 0x7fe08006a7a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.685+0000 7fe0989fe700 1 --2- 172.21.15.98:0/2229190936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0940b4aa0 0x7fe0940b7700 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.685+0000 7fe0989fe700 1 --2- 172.21.15.98:0/2229190936 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0941609b0 0x7fe0940b7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.685+0000 7fe0989fe700 1 --2- 172.21.15.98:0/2229190936 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0940cf570 0x7fe0940cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.685+0000 7fe0989fe700 1 -- 172.21.15.98:0/2229190936 >> 172.21.15.98:0/2229190936 conn(0x7fe09406b880 msgr2=0x7fe0940731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.685+0000 7fe0989fe700 1 -- 172.21.15.98:0/2229190936 shutdown_connections 2023-03-03T07:56:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:51.685+0000 7fe0989fe700 1 -- 172.21.15.98:0/2229190936 wait complete. 2023-03-03T07:56:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T07:56:51.714 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2023-03-03T07:56:51.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:51.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2023-03-03T07:56:51.815 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T07:56:51.816 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T07:56:51.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:51.817 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-03-03T07:56:51.886 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:56:51.886 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:56:51.960 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:56:51.960 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:56:52.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:56:52.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.278+0000 7fed0759e700 1 Processor -- start 2023-03-03T07:56:52.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.278+0000 7fed0759e700 1 -- start start 2023-03-03T07:56:52.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.280+0000 7fed0759e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fed080b51c0 0x7fed08160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:52.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.280+0000 7fed0759e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed081614c0 con 0x7fed08146360 2023-03-03T07:56:52.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.280+0000 7fed0759e700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed080d10f0 con 0x7fed080d1580 2023-03-03T07:56:52.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.280+0000 7fed0759e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed080cdf20 con 0x7fed080b51c0 2023-03-03T07:56:52.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.280+0000 7fed0659c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fed08146360 0x7fed080b4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46936/0 (socket says 172.21.15.98:46936) 2023-03-03T07:56:52.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.280+0000 7fed0659c700 1 -- 172.21.15.98:0/118513399 learned_addr learned my addr 172.21.15.98:0/118513399 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:52.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.280+0000 7fed05d9b700 1 --2- 172.21.15.98:0/118513399 >> v2:172.21.15.100:3300/0 conn(0x7fed080b51c0 0x7fed08160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:52.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.281+0000 7fed0559a700 1 -- 172.21.15.98:0/118513399 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050309116 0 0) 0x7fecf8001110 con 0x7fed08146360 2023-03-03T07:56:52.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.281+0000 7fed0559a700 1 -- 172.21.15.98:0/118513399 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf00035b0 con 0x7fed08146360 2023-03-03T07:56:52.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.281+0000 7fed0559a700 1 -- 172.21.15.98:0/118513399 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887271098 0 0) 0x7fed080d10f0 con 0x7fed080d1580 2023-03-03T07:56:52.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.281+0000 7fed0559a700 1 -- 172.21.15.98:0/118513399 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf8001110 con 0x7fed080d1580 2023-03-03T07:56:52.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.281+0000 7fed05d9b700 1 -- 172.21.15.98:0/118513399 >> v1:172.21.15.100:6789/0 conn(0x7fed080d1580 legacy=0x7fed080d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:52.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.281+0000 7fed05d9b700 1 -- 172.21.15.98:0/118513399 >> v1:172.21.15.98:6789/0 conn(0x7fed08146360 legacy=0x7fed080b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:52.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.281+0000 7fed05d9b700 1 -- 172.21.15.98:0/118513399 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed080cd680 con 0x7fed080b51c0 2023-03-03T07:56:52.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.281+0000 7fed05d9b700 1 --2- 172.21.15.98:0/118513399 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed080b51c0 0x7fed08160f80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009840 tx=0x7fecfc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49713f4cae21c38b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:52.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.282+0000 7fed0559a700 1 -- 172.21.15.98:0/118513399 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc011830 con 0x7fed080b51c0 2023-03-03T07:56:52.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.282+0000 7fed0559a700 1 -- 172.21.15.98:0/118513399 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecfc011990 con 0x7fed080b51c0 2023-03-03T07:56:52.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.282+0000 7fed0559a700 1 -- 172.21.15.98:0/118513399 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc011d00 con 0x7fed080b51c0 2023-03-03T07:56:52.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.282+0000 7fed0759e700 1 -- 172.21.15.98:0/118513399 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed080b51c0 msgr2=0x7fed08160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:52.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.282+0000 7fed0759e700 1 --2- 172.21.15.98:0/118513399 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed080b51c0 0x7fed08160f80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009840 tx=0x7fecfc009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:52.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.282+0000 7fed0759e700 1 -- 172.21.15.98:0/118513399 shutdown_connections 2023-03-03T07:56:52.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.282+0000 7fed0759e700 1 --2- 172.21.15.98:0/118513399 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed080b51c0 0x7fed08160f80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.282+0000 7fed0759e700 1 -- 172.21.15.98:0/118513399 >> 172.21.15.98:0/118513399 conn(0x7fed0806b8e0 msgr2=0x7fed080cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:52.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.283+0000 7fed0759e700 1 -- 172.21.15.98:0/118513399 shutdown_connections 2023-03-03T07:56:52.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.283+0000 7fed0759e700 1 -- 172.21.15.98:0/118513399 wait complete. 2023-03-03T07:56:52.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.285+0000 7fed0759e700 1 Processor -- start 2023-03-03T07:56:52.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.285+0000 7fed0759e700 1 -- start start 2023-03-03T07:56:52.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.286+0000 7fed0759e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed080b51c0 0x7fed080c6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:52.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.286+0000 7fed0759e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed080d1580 0x7fed0815dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:52.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.286+0000 7fed0759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed08146360 0x7fed0815e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:52.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.286+0000 7fed0759e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed080cba90 con 0x7fed08146360 2023-03-03T07:56:52.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.286+0000 7fed0759e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed080caba0 con 0x7fed080d1580 2023-03-03T07:56:52.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.286+0000 7fed0759e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed080cae80 con 0x7fed080b51c0 2023-03-03T07:56:52.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.286+0000 7fed06d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed08146360 0x7fed0815e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:52.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.286+0000 7fed06d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed08146360 0x7fed0815e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51680/0 (socket says 172.21.15.98:51680) 2023-03-03T07:56:52.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.286+0000 7fed06d9d700 1 -- 172.21.15.98:0/1078448241 learned_addr learned my addr 172.21.15.98:0/1078448241 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:52.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.286+0000 7fed0659c700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed080b51c0 0x7fed080c6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:52.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.286+0000 7fed05d9b700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed080d1580 0x7fed0815dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:52.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.287+0000 7fed06d9d700 1 -- 172.21.15.98:0/1078448241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed080b51c0 msgr2=0x7fed080c6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:52.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.287+0000 7fed06d9d700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed080b51c0 0x7fed080c6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.287+0000 7fed06d9d700 1 -- 172.21.15.98:0/1078448241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed080d1580 msgr2=0x7fed0815dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:52.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.287+0000 7fed06d9d700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed080d1580 0x7fed0815dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.287+0000 7fed06d9d700 1 -- 172.21.15.98:0/1078448241 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc0094a0 con 0x7fed08146360 2023-03-03T07:56:52.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.287+0000 7fed0659c700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed080b51c0 0x7fed080c6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:56:52.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.287+0000 7fed05d9b700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed080d1580 0x7fed0815dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:52.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.287+0000 7fed06d9d700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed08146360 0x7fed0815e510 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fed0000cc20 tx=0x7fed0000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:52.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.287+0000 7fecf6ffd700 1 -- 172.21.15.98:0/1078448241 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0000ac20 con 0x7fed08146360 2023-03-03T07:56:52.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.287+0000 7fecf6ffd700 1 -- 172.21.15.98:0/1078448241 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0000ad80 con 0x7fed08146360 2023-03-03T07:56:52.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.287+0000 7fecf6ffd700 1 -- 172.21.15.98:0/1078448241 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00003c20 con 0x7fed08146360 2023-03-03T07:56:52.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.287+0000 7fed0759e700 1 -- 172.21.15.98:0/1078448241 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed080cdc40 con 0x7fed08146360 2023-03-03T07:56:52.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.288+0000 7fed0759e700 1 -- 172.21.15.98:0/1078448241 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed080cd960 con 0x7fed08146360 2023-03-03T07:56:52.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.289+0000 7fed0759e700 1 -- 172.21.15.98:0/1078448241 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed080d0d80 con 0x7fed08146360 2023-03-03T07:56:52.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.290+0000 7fecf6ffd700 1 -- 172.21.15.98:0/1078448241 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fed00010050 con 0x7fed08146360 2023-03-03T07:56:52.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.291+0000 7fecf6ffd700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fece80682f0 0x7fece806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:52.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.291+0000 7fecf6ffd700 1 -- 172.21.15.98:0/1078448241 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fed00085ea0 con 0x7fed08146360 2023-03-03T07:56:52.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.291+0000 7fed0659c700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fece80682f0 0x7fece806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:52.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.292+0000 7fed0659c700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fece80682f0 0x7fece806a7a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fecf8001370 tx=0x7fecf8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:52.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.295+0000 7fecf6ffd700 1 -- 172.21.15.98:0/1078448241 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fed00052550 con 0x7fed08146360 2023-03-03T07:56:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.488+0000 7fed0759e700 1 -- 172.21.15.98:0/1078448241 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed08068300 con 0x7fed08146360 2023-03-03T07:56:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.489+0000 7fecf6ffd700 1 -- 172.21.15.98:0/1078448241 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18003 (secure 0 0 0) 0x7fed0005b020 con 0x7fed08146360 2023-03-03T07:56:52.493 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:56:52.493 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:56:27.498205+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:56:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.492+0000 7fed0759e700 1 -- 172.21.15.98:0/1078448241 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fece80682f0 msgr2=0x7fece806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.492+0000 7fed0759e700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fece80682f0 0x7fece806a7a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fecf8001370 tx=0x7fecf8009040 comp rx=0 tx=0).stop 2023-03-03T07:56:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.492+0000 7fed0759e700 1 -- 172.21.15.98:0/1078448241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed08146360 msgr2=0x7fed0815e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.492+0000 7fed0759e700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed08146360 0x7fed0815e510 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fed0000cc20 tx=0x7fed0000a060 comp rx=0 tx=0).stop 2023-03-03T07:56:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.492+0000 7fed0759e700 1 -- 172.21.15.98:0/1078448241 shutdown_connections 2023-03-03T07:56:52.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.492+0000 7fed0759e700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fece80682f0 0x7fece806a7a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.492+0000 7fed0759e700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed08146360 0x7fed0815e510 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.492+0000 7fed0759e700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed080b51c0 0x7fed080c6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.492+0000 7fed0759e700 1 --2- 172.21.15.98:0/1078448241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed080d1580 0x7fed0815dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.492+0000 7fed0759e700 1 -- 172.21.15.98:0/1078448241 >> 172.21.15.98:0/1078448241 conn(0x7fed0806b8e0 msgr2=0x7fed080d2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:52.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.493+0000 7fed0759e700 1 -- 172.21.15.98:0/1078448241 shutdown_connections 2023-03-03T07:56:52.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.493+0000 7fed0759e700 1 -- 172.21.15.98:0/1078448241 wait complete. 2023-03-03T07:56:52.522 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:55:45.157666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:55:46.092091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:56:52.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:56:52.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.775+0000 7fa9ba5ff700 1 Processor -- start 2023-03-03T07:56:52.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.775+0000 7fa9ba5ff700 1 -- start start 2023-03-03T07:56:52.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.777+0000 7fa9ba5ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9b40b8d10 0x7fa9b40cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:52.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.777+0000 7fa9ba5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9b40b5310 0x7fa9b40d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:52.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.777+0000 7fa9ba5ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b40cd7d0 con 0x7fa9b40b26d0 2023-03-03T07:56:52.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.777+0000 7fa9ba5ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b40b90e0 con 0x7fa9b40b8d10 2023-03-03T07:56:52.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.777+0000 7fa9ba5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b40cda20 con 0x7fa9b40b5310 2023-03-03T07:56:52.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.777+0000 7fa9b3fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9b40b8d10 0x7fa9b40cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:52.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.777+0000 7fa9abfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa9b40b26d0 0x7fa9b40b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46938/0 (socket says 172.21.15.98:46938) 2023-03-03T07:56:52.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.777+0000 7fa9abfff700 1 -- 172.21.15.98:0/3568429806 learned_addr learned my addr 172.21.15.98:0/3568429806 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:52.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.777+0000 7fa9b8b9c700 1 --2- 172.21.15.98:0/3568429806 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9b40b5310 0x7fa9b40d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:52.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.778+0000 7fa9b3fff700 1 -- 172.21.15.98:0/3568429806 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9b40b5310 msgr2=0x7fa9b40d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:52.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.778+0000 7fa9b3fff700 1 --2- 172.21.15.98:0/3568429806 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9b40b5310 0x7fa9b40d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.778+0000 7fa9b3fff700 1 -- 172.21.15.98:0/3568429806 >> v1:172.21.15.98:6789/0 conn(0x7fa9b40b26d0 legacy=0x7fa9b40b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:52.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.778+0000 7fa9b3fff700 1 -- 172.21.15.98:0/3568429806 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b40cdd00 con 0x7fa9b40b8d10 2023-03-03T07:56:52.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.778+0000 7fa9b37fe700 1 -- 172.21.15.98:0/3568429806 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660469563 0 0) 0x7fa9b40cd7d0 con 0x7fa9b40b26d0 2023-03-03T07:56:52.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.778+0000 7fa9b3fff700 1 --2- 172.21.15.98:0/3568429806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40b8d10 0x7fa9b40cffc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4009840 tx=0x7fa9a4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a8c24ef636eb9a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:52.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.778+0000 7fa9b37fe700 1 -- 172.21.15.98:0/3568429806 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4011830 con 0x7fa9b40b8d10 2023-03-03T07:56:52.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.778+0000 7fa9b37fe700 1 -- 172.21.15.98:0/3568429806 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a4011990 con 0x7fa9b40b8d10 2023-03-03T07:56:52.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.778+0000 7fa9b37fe700 1 -- 172.21.15.98:0/3568429806 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4011d20 con 0x7fa9b40b8d10 2023-03-03T07:56:52.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.779+0000 7fa9ba5ff700 1 -- 172.21.15.98:0/3568429806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40b8d10 msgr2=0x7fa9b40cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:52.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.779+0000 7fa9ba5ff700 1 --2- 172.21.15.98:0/3568429806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40b8d10 0x7fa9b40cffc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4009840 tx=0x7fa9a4009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:52.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.779+0000 7fa9ba5ff700 1 -- 172.21.15.98:0/3568429806 shutdown_connections 2023-03-03T07:56:52.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.779+0000 7fa9ba5ff700 1 --2- 172.21.15.98:0/3568429806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40b8d10 0x7fa9b40cffc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.779+0000 7fa9ba5ff700 1 --2- 172.21.15.98:0/3568429806 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9b40b5310 0x7fa9b40d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.779+0000 7fa9ba5ff700 1 -- 172.21.15.98:0/3568429806 >> 172.21.15.98:0/3568429806 conn(0x7fa9b4073e70 msgr2=0x7fa9b406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:52.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.779+0000 7fa9ba5ff700 1 -- 172.21.15.98:0/3568429806 shutdown_connections 2023-03-03T07:56:52.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.779+0000 7fa9ba5ff700 1 -- 172.21.15.98:0/3568429806 wait complete. 2023-03-03T07:56:52.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.781+0000 7fa9ba5ff700 1 Processor -- start 2023-03-03T07:56:52.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.781+0000 7fa9ba5ff700 1 -- start start 2023-03-03T07:56:52.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.782+0000 7fa9ba5ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40b26d0 0x7fa9b4150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:52.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.782+0000 7fa9ba5ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9b40b5310 0x7fa9b415e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:52.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.782+0000 7fa9ba5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9b40b8d10 0x7fa9b415da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:52.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.782+0000 7fa9ba5ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b415dfb0 con 0x7fa9b40b26d0 2023-03-03T07:56:52.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.782+0000 7fa9ba5ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b415ec60 con 0x7fa9b40b5310 2023-03-03T07:56:52.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.782+0000 7fa9ba5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b415ef40 con 0x7fa9b40b8d10 2023-03-03T07:56:52.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.783+0000 7fa9b3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40b26d0 0x7fa9b4150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:52.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.783+0000 7fa9b3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40b26d0 0x7fa9b4150f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51690/0 (socket says 172.21.15.98:51690) 2023-03-03T07:56:52.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.783+0000 7fa9b3fff700 1 -- 172.21.15.98:0/2107393074 learned_addr learned my addr 172.21.15.98:0/2107393074 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:52.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.783+0000 7fa9abfff700 1 --2- 172.21.15.98:0/2107393074 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9b40b5310 0x7fa9b415e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:52.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.783+0000 7fa9b8b9c700 1 --2- 172.21.15.98:0/2107393074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9b40b8d10 0x7fa9b415da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:52.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.783+0000 7fa9b3fff700 1 -- 172.21.15.98:0/2107393074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9b40b8d10 msgr2=0x7fa9b415da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:52.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.783+0000 7fa9b3fff700 1 --2- 172.21.15.98:0/2107393074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9b40b8d10 0x7fa9b415da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.783+0000 7fa9b3fff700 1 -- 172.21.15.98:0/2107393074 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9b40b5310 msgr2=0x7fa9b415e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:52.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.783+0000 7fa9b3fff700 1 --2- 172.21.15.98:0/2107393074 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9b40b5310 0x7fa9b415e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.783+0000 7fa9b3fff700 1 -- 172.21.15.98:0/2107393074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a40094a0 con 0x7fa9b40b26d0 2023-03-03T07:56:52.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.784+0000 7fa9b3fff700 1 --2- 172.21.15.98:0/2107393074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40b26d0 0x7fa9b4150f80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4000c00 tx=0x7fa9a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:52.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.784+0000 7fa9b17fa700 1 -- 172.21.15.98:0/2107393074 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4004550 con 0x7fa9b40b26d0 2023-03-03T07:56:52.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.784+0000 7fa9b17fa700 1 -- 172.21.15.98:0/2107393074 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a401e070 con 0x7fa9b40b26d0 2023-03-03T07:56:52.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.784+0000 7fa9ba5ff700 1 -- 172.21.15.98:0/2107393074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b40cdfe0 con 0x7fa9b40b26d0 2023-03-03T07:56:52.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.784+0000 7fa9b17fa700 1 -- 172.21.15.98:0/2107393074 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a40038e0 con 0x7fa9b40b26d0 2023-03-03T07:56:52.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.784+0000 7fa9ba5ff700 1 -- 172.21.15.98:0/2107393074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b40d0e80 con 0x7fa9b40b26d0 2023-03-03T07:56:52.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.785+0000 7fa9ba5ff700 1 -- 172.21.15.98:0/2107393074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b40d0500 con 0x7fa9b40b26d0 2023-03-03T07:56:52.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.786+0000 7fa9b17fa700 1 -- 172.21.15.98:0/2107393074 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa9a400f050 con 0x7fa9b40b26d0 2023-03-03T07:56:52.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.787+0000 7fa9b17fa700 1 --2- 172.21.15.98:0/2107393074 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9940682f0 0x7fa99406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:52.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.787+0000 7fa9b17fa700 1 -- 172.21.15.98:0/2107393074 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fa9a4086d00 con 0x7fa9b40b26d0 2023-03-03T07:56:52.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.788+0000 7fa9abfff700 1 --2- 172.21.15.98:0/2107393074 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9940682f0 0x7fa99406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:52.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.788+0000 7fa9abfff700 1 --2- 172.21.15.98:0/2107393074 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9940682f0 0x7fa99406a7a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40b1dd0 tx=0x7fa99c007e70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:52.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.792+0000 7fa9b17fa700 1 -- 172.21.15.98:0/2107393074 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa9a4053430 con 0x7fa9b40b26d0 2023-03-03T07:56:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.986+0000 7fa9ba5ff700 1 -- 172.21.15.98:0/2107393074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa9b404e6e0 con 0x7fa9b40b26d0 2023-03-03T07:56:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.987+0000 7fa9b17fa700 1 -- 172.21.15.98:0/2107393074 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa9a40150a0 con 0x7fa9b40b26d0 2023-03-03T07:56:52.989 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:56:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.989+0000 7fa9ba5ff700 1 -- 172.21.15.98:0/2107393074 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9940682f0 msgr2=0x7fa99406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.989+0000 7fa9ba5ff700 1 --2- 172.21.15.98:0/2107393074 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9940682f0 0x7fa99406a7a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40b1dd0 tx=0x7fa99c007e70 comp rx=0 tx=0).stop 2023-03-03T07:56:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.989+0000 7fa9ba5ff700 1 -- 172.21.15.98:0/2107393074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40b26d0 msgr2=0x7fa9b4150f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.989+0000 7fa9ba5ff700 1 --2- 172.21.15.98:0/2107393074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40b26d0 0x7fa9b4150f80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4000c00 tx=0x7fa9a4004750 comp rx=0 tx=0).stop 2023-03-03T07:56:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.989+0000 7fa9ba5ff700 1 -- 172.21.15.98:0/2107393074 shutdown_connections 2023-03-03T07:56:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.989+0000 7fa9ba5ff700 1 --2- 172.21.15.98:0/2107393074 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa9940682f0 0x7fa99406a7a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.989+0000 7fa9ba5ff700 1 --2- 172.21.15.98:0/2107393074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40b26d0 0x7fa9b4150f80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.989+0000 7fa9ba5ff700 1 --2- 172.21.15.98:0/2107393074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9b40b8d10 0x7fa9b415da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.989+0000 7fa9ba5ff700 1 --2- 172.21.15.98:0/2107393074 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9b40b5310 0x7fa9b415e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:52.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.989+0000 7fa9ba5ff700 1 -- 172.21.15.98:0/2107393074 >> 172.21.15.98:0/2107393074 conn(0x7fa9b4073e70 msgr2=0x7fa9b406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:52.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.990+0000 7fa9ba5ff700 1 -- 172.21.15.98:0/2107393074 shutdown_connections 2023-03-03T07:56:52.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:52.990+0000 7fa9ba5ff700 1 -- 172.21.15.98:0/2107393074 wait complete. 2023-03-03T07:56:53.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:56:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.265+0000 7f4a75b9d700 1 Processor -- start 2023-03-03T07:56:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.266+0000 7f4a75b9d700 1 -- start start 2023-03-03T07:56:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.267+0000 7f4a75b9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4a701609b0 0x7f4a70160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.267+0000 7f4a75b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a700cf570 0x7f4a7015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.267+0000 7f4a75b9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4a7015db30 0x7f4a700b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.267+0000 7f4a75b9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a700cebb0 con 0x7f4a701609b0 2023-03-03T07:56:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.267+0000 7f4a75b9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a701612c0 con 0x7f4a7015db30 2023-03-03T07:56:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.267+0000 7f4a75b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700cd780 con 0x7f4a700cf570 2023-03-03T07:56:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.267+0000 7f4a74b9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4a701609b0 0x7f4a70160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.267+0000 7f4a74b9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4a701609b0 0x7f4a70160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51692/0 (socket says 172.21.15.98:51692) 2023-03-03T07:56:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.268+0000 7f4a74b9b700 1 -- 172.21.15.98:0/3862579171 learned_addr learned my addr 172.21.15.98:0/3862579171 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.268+0000 7f4a6ffff700 1 --2- 172.21.15.98:0/3862579171 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a700cf570 0x7f4a7015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.268+0000 7f4a7539c700 1 --2- 172.21.15.98:0/3862579171 >> v2:172.21.15.100:3300/0 conn(0x7f4a7015db30 0x7f4a700b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.268+0000 7f4a74b9b700 1 -- 172.21.15.98:0/3862579171 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a700cf570 msgr2=0x7f4a7015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:53.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.268+0000 7f4a74b9b700 1 --2- 172.21.15.98:0/3862579171 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a700cf570 0x7f4a7015d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.268+0000 7f4a74b9b700 1 -- 172.21.15.98:0/3862579171 >> v2:172.21.15.100:3300/0 conn(0x7f4a7015db30 msgr2=0x7f4a700b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:53.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.268+0000 7f4a74b9b700 1 --2- 172.21.15.98:0/3862579171 >> v2:172.21.15.100:3300/0 conn(0x7f4a7015db30 0x7f4a700b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.268+0000 7f4a74b9b700 1 -- 172.21.15.98:0/3862579171 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a700cda60 con 0x7f4a701609b0 2023-03-03T07:56:53.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.268+0000 7f4a74b9b700 1 --2- 172.21.15.98:0/3862579171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a701609b0 0x7f4a70160d80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4a700b5bd0 tx=0x7f4a64009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e1649c1fa5afd2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:53.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.269+0000 7f4a6f7fe700 1 -- 172.21.15.98:0/3862579171 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a64011830 con 0x7f4a701609b0 2023-03-03T07:56:53.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.269+0000 7f4a6f7fe700 1 -- 172.21.15.98:0/3862579171 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a64011990 con 0x7f4a701609b0 2023-03-03T07:56:53.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.269+0000 7f4a6f7fe700 1 -- 172.21.15.98:0/3862579171 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a64011d20 con 0x7f4a701609b0 2023-03-03T07:56:53.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.269+0000 7f4a75b9d700 1 -- 172.21.15.98:0/3862579171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a701609b0 msgr2=0x7f4a70160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.269+0000 7f4a75b9d700 1 --2- 172.21.15.98:0/3862579171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a701609b0 0x7f4a70160d80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4a700b5bd0 tx=0x7f4a64009bd0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.269+0000 7f4a75b9d700 1 -- 172.21.15.98:0/3862579171 shutdown_connections 2023-03-03T07:56:53.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.269+0000 7f4a75b9d700 1 --2- 172.21.15.98:0/3862579171 >> v2:172.21.15.100:3300/0 conn(0x7f4a7015db30 0x7f4a700b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.269+0000 7f4a75b9d700 1 --2- 172.21.15.98:0/3862579171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a701609b0 0x7f4a70160d80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.269+0000 7f4a75b9d700 1 --2- 172.21.15.98:0/3862579171 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a700cf570 0x7f4a7015d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.269+0000 7f4a75b9d700 1 -- 172.21.15.98:0/3862579171 >> 172.21.15.98:0/3862579171 conn(0x7f4a7006b880 msgr2=0x7f4a700cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:53.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.270+0000 7f4a75b9d700 1 -- 172.21.15.98:0/3862579171 shutdown_connections 2023-03-03T07:56:53.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.270+0000 7f4a75b9d700 1 -- 172.21.15.98:0/3862579171 wait complete. 2023-03-03T07:56:53.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.271+0000 7f4a75b9d700 1 Processor -- start 2023-03-03T07:56:53.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.272+0000 7f4a75b9d700 1 -- start start 2023-03-03T07:56:53.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.273+0000 7f4a75b9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a700cf570 0x7f4a700cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.273+0000 7f4a75b9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a7015db30 0x7f4a700ca360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.273+0000 7f4a75b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a701609b0 0x7f4a70150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.273+0000 7f4a75b9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700c6b50 con 0x7f4a7015db30 2023-03-03T07:56:53.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.273+0000 7f4a75b9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700b3250 con 0x7f4a700cf570 2023-03-03T07:56:53.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.273+0000 7f4a75b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700ccb40 con 0x7f4a701609b0 2023-03-03T07:56:53.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.273+0000 7f4a6ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a7015db30 0x7f4a700ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.273+0000 7f4a6ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a7015db30 0x7f4a700ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51708/0 (socket says 172.21.15.98:51708) 2023-03-03T07:56:53.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.273+0000 7f4a6ffff700 1 -- 172.21.15.98:0/4166151522 learned_addr learned my addr 172.21.15.98:0/4166151522 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:53.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.273+0000 7f4a74b9b700 1 --2- 172.21.15.98:0/4166151522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a700cf570 0x7f4a700cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.273+0000 7f4a7539c700 1 --2- 172.21.15.98:0/4166151522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a701609b0 0x7f4a70150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.274+0000 7f4a6ffff700 1 -- 172.21.15.98:0/4166151522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a701609b0 msgr2=0x7f4a70150440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.274+0000 7f4a6ffff700 1 --2- 172.21.15.98:0/4166151522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a701609b0 0x7f4a70150440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.274+0000 7f4a6ffff700 1 -- 172.21.15.98:0/4166151522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a700cf570 msgr2=0x7f4a700cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.274+0000 7f4a6ffff700 1 --2- 172.21.15.98:0/4166151522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a700cf570 0x7f4a700cc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.274+0000 7f4a6ffff700 1 -- 172.21.15.98:0/4166151522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a64009550 con 0x7f4a7015db30 2023-03-03T07:56:53.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.274+0000 7f4a6ffff700 1 --2- 172.21.15.98:0/4166151522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a7015db30 0x7f4a700ca360 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00e930 tx=0x7f4a5c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:53.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.274+0000 7f4a6d7fa700 1 -- 172.21.15.98:0/4166151522 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c013680 con 0x7f4a7015db30 2023-03-03T07:56:53.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.274+0000 7f4a6d7fa700 1 -- 172.21.15.98:0/4166151522 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5c0137e0 con 0x7f4a7015db30 2023-03-03T07:56:53.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.274+0000 7f4a6d7fa700 1 -- 172.21.15.98:0/4166151522 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c013b70 con 0x7f4a7015db30 2023-03-03T07:56:53.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.274+0000 7f4a75b9d700 1 -- 172.21.15.98:0/4166151522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7015cdf0 con 0x7f4a7015db30 2023-03-03T07:56:53.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.275+0000 7f4a75b9d700 1 -- 172.21.15.98:0/4166151522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a700cdd40 con 0x7f4a7015db30 2023-03-03T07:56:53.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.276+0000 7f4a75b9d700 1 -- 172.21.15.98:0/4166151522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a7015d0d0 con 0x7f4a7015db30 2023-03-03T07:56:53.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.277+0000 7f4a6d7fa700 1 -- 172.21.15.98:0/4166151522 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4a5c019030 con 0x7f4a7015db30 2023-03-03T07:56:53.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.278+0000 7f4a6d7fa700 1 --2- 172.21.15.98:0/4166151522 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4a540682f0 0x7f4a5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.278+0000 7f4a6d7fa700 1 -- 172.21.15.98:0/4166151522 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f4a5c085b10 con 0x7f4a7015db30 2023-03-03T07:56:53.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.278+0000 7f4a74b9b700 1 --2- 172.21.15.98:0/4166151522 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4a540682f0 0x7f4a5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.279+0000 7f4a74b9b700 1 --2- 172.21.15.98:0/4166151522 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4a540682f0 0x7f4a5406a7a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4a64009840 tx=0x7f4a64008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:53.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.282+0000 7f4a6d7fa700 1 -- 172.21.15.98:0/4166151522 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4a5c052330 con 0x7f4a7015db30 2023-03-03T07:56:53.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.476+0000 7f4a75b9d700 1 -- 172.21.15.98:0/4166151522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4a700b33c0 con 0x7f4a7015db30 2023-03-03T07:56:53.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.476+0000 7f4a6d7fa700 1 -- 172.21.15.98:0/4166151522 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4a5c017020 con 0x7f4a7015db30 2023-03-03T07:56:53.478 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:56:53.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.479+0000 7f4a75b9d700 1 -- 172.21.15.98:0/4166151522 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4a540682f0 msgr2=0x7f4a5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.479+0000 7f4a75b9d700 1 --2- 172.21.15.98:0/4166151522 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4a540682f0 0x7f4a5406a7a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4a64009840 tx=0x7f4a64008030 comp rx=0 tx=0).stop 2023-03-03T07:56:53.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.479+0000 7f4a75b9d700 1 -- 172.21.15.98:0/4166151522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a7015db30 msgr2=0x7f4a700ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.479+0000 7f4a75b9d700 1 --2- 172.21.15.98:0/4166151522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a7015db30 0x7f4a700ca360 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00e930 tx=0x7f4a5c00ec40 comp rx=0 tx=0).stop 2023-03-03T07:56:53.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.479+0000 7f4a75b9d700 1 -- 172.21.15.98:0/4166151522 shutdown_connections 2023-03-03T07:56:53.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.479+0000 7f4a75b9d700 1 --2- 172.21.15.98:0/4166151522 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4a540682f0 0x7f4a5406a7a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.479+0000 7f4a75b9d700 1 --2- 172.21.15.98:0/4166151522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a701609b0 0x7f4a70150440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.479+0000 7f4a75b9d700 1 --2- 172.21.15.98:0/4166151522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a700cf570 0x7f4a700cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.479+0000 7f4a75b9d700 1 --2- 172.21.15.98:0/4166151522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a7015db30 0x7f4a700ca360 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.479+0000 7f4a75b9d700 1 -- 172.21.15.98:0/4166151522 >> 172.21.15.98:0/4166151522 conn(0x7f4a7006b880 msgr2=0x7f4a7015f910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:53.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.479+0000 7f4a75b9d700 1 -- 172.21.15.98:0/4166151522 shutdown_connections 2023-03-03T07:56:53.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.479+0000 7f4a75b9d700 1 -- 172.21.15.98:0/4166151522 wait complete. 2023-03-03T07:56:53.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:56:53.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.754+0000 7fcee49fe700 1 Processor -- start 2023-03-03T07:56:53.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.754+0000 7fcee49fe700 1 -- start start 2023-03-03T07:56:53.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.756+0000 7fcee49fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcee01609b0 0x7fcee0160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.756+0000 7fcee49fe700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee01612c0 con 0x7fcee00b4aa0 2023-03-03T07:56:53.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.756+0000 7fcee49fe700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee015dd00 con 0x7fcee00cf570 2023-03-03T07:56:53.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.756+0000 7fcee49fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00cebb0 con 0x7fcee01609b0 2023-03-03T07:56:53.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.756+0000 7fcedf59e700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcee00b4aa0 0x7fcee00d0bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46952/0 (socket says 172.21.15.98:46952) 2023-03-03T07:56:53.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.756+0000 7fceded9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcee01609b0 0x7fcee0160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.756+0000 7fcedf59e700 1 -- 172.21.15.98:0/3599512468 learned_addr learned my addr 172.21.15.98:0/3599512468 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:53.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.756+0000 7fced7fff700 1 --1- 172.21.15.98:0/3599512468 >> v1:172.21.15.100:6789/0 conn(0x7fcee00cf570 0x7fcee015d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:40152/0 (socket says 172.21.15.98:40152) 2023-03-03T07:56:53.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.757+0000 7fcede59c700 1 -- 172.21.15.98:0/3599512468 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121828896 0 0) 0x7fcee015dd00 con 0x7fcee00cf570 2023-03-03T07:56:53.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.757+0000 7fcede59c700 1 -- 172.21.15.98:0/3599512468 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcecc0035b0 con 0x7fcee00cf570 2023-03-03T07:56:53.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.757+0000 7fcede59c700 1 -- 172.21.15.98:0/3599512468 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536396880 0 0) 0x7fcee01612c0 con 0x7fcee00b4aa0 2023-03-03T07:56:53.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.757+0000 7fceded9d700 1 -- 172.21.15.98:0/3599512468 >> v1:172.21.15.100:6789/0 conn(0x7fcee00cf570 legacy=0x7fcee015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.757+0000 7fceded9d700 1 -- 172.21.15.98:0/3599512468 >> v1:172.21.15.98:6789/0 conn(0x7fcee00b4aa0 legacy=0x7fcee00d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.757+0000 7fceded9d700 1 -- 172.21.15.98:0/3599512468 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee00cd780 con 0x7fcee01609b0 2023-03-03T07:56:53.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.757+0000 7fceded9d700 1 --2- 172.21.15.98:0/3599512468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcee01609b0 0x7fcee0160d80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fced0009870 tx=0x7fced0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b14738aab0b9fbd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:53.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.758+0000 7fcede59c700 1 -- 172.21.15.98:0/3599512468 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0011830 con 0x7fcee01609b0 2023-03-03T07:56:53.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.758+0000 7fcede59c700 1 -- 172.21.15.98:0/3599512468 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced0011990 con 0x7fcee01609b0 2023-03-03T07:56:53.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.758+0000 7fcede59c700 1 -- 172.21.15.98:0/3599512468 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0011d20 con 0x7fcee01609b0 2023-03-03T07:56:53.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.758+0000 7fcee49fe700 1 -- 172.21.15.98:0/3599512468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcee01609b0 msgr2=0x7fcee0160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.758+0000 7fcee49fe700 1 --2- 172.21.15.98:0/3599512468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcee01609b0 0x7fcee0160d80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fced0009870 tx=0x7fced0009b80 comp rx=0 tx=0).stop 2023-03-03T07:56:53.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.758+0000 7fcee49fe700 1 -- 172.21.15.98:0/3599512468 shutdown_connections 2023-03-03T07:56:53.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.758+0000 7fcee49fe700 1 --2- 172.21.15.98:0/3599512468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcee01609b0 0x7fcee0160d80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.758+0000 7fcee49fe700 1 -- 172.21.15.98:0/3599512468 >> 172.21.15.98:0/3599512468 conn(0x7fcee006b880 msgr2=0x7fcee00cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:53.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.759+0000 7fcee49fe700 1 -- 172.21.15.98:0/3599512468 shutdown_connections 2023-03-03T07:56:53.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.759+0000 7fcee49fe700 1 -- 172.21.15.98:0/3599512468 wait complete. 2023-03-03T07:56:53.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.761+0000 7fcee49fe700 1 Processor -- start 2023-03-03T07:56:53.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.761+0000 7fcee49fe700 1 -- start start 2023-03-03T07:56:53.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.762+0000 7fcee49fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcee00b4aa0 0x7fcee0147fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.762+0000 7fcee49fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee00cf570 0x7fcee00c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.762+0000 7fcee49fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcee01609b0 0x7fcee00b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.762+0000 7fcee49fe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00b6d40 con 0x7fcee00cf570 2023-03-03T07:56:53.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.762+0000 7fcee49fe700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee0148500 con 0x7fcee01609b0 2023-03-03T07:56:53.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.762+0000 7fcee49fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00b5e50 con 0x7fcee00b4aa0 2023-03-03T07:56:53.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.762+0000 7fced7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee00cf570 0x7fcee00c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.762+0000 7fced7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee00cf570 0x7fcee00c69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51718/0 (socket says 172.21.15.98:51718) 2023-03-03T07:56:53.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.762+0000 7fced7fff700 1 -- 172.21.15.98:0/3900948467 learned_addr learned my addr 172.21.15.98:0/3900948467 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:53.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.762+0000 7fceded9d700 1 --2- 172.21.15.98:0/3900948467 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcee00b4aa0 0x7fcee0147fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.762+0000 7fcedf59e700 1 --2- 172.21.15.98:0/3900948467 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcee01609b0 0x7fcee00b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.763+0000 7fced7fff700 1 -- 172.21.15.98:0/3900948467 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcee00b4aa0 msgr2=0x7fcee0147fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.763+0000 7fced7fff700 1 --2- 172.21.15.98:0/3900948467 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcee00b4aa0 0x7fcee0147fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.763+0000 7fced7fff700 1 -- 172.21.15.98:0/3900948467 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcee01609b0 msgr2=0x7fcee00b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.763+0000 7fced7fff700 1 --2- 172.21.15.98:0/3900948467 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcee01609b0 0x7fcee00b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.763+0000 7fced7fff700 1 -- 172.21.15.98:0/3900948467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced00094a0 con 0x7fcee00cf570 2023-03-03T07:56:53.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.763+0000 7fced7fff700 1 --2- 172.21.15.98:0/3900948467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee00cf570 0x7fcee00c69b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcec800c700 tx=0x7fcec800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:53.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.763+0000 7fced77fe700 1 -- 172.21.15.98:0/3900948467 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec8005cb0 con 0x7fcee00cf570 2023-03-03T07:56:53.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.763+0000 7fced77fe700 1 -- 172.21.15.98:0/3900948467 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec8005e10 con 0x7fcee00cf570 2023-03-03T07:56:53.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.763+0000 7fced77fe700 1 -- 172.21.15.98:0/3900948467 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec80072d0 con 0x7fcee00cf570 2023-03-03T07:56:53.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.763+0000 7fcee49fe700 1 -- 172.21.15.98:0/3900948467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee00cdd40 con 0x7fcee00cf570 2023-03-03T07:56:53.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.763+0000 7fcee49fe700 1 -- 172.21.15.98:0/3900948467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee00cda60 con 0x7fcee00cf570 2023-03-03T07:56:53.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.764+0000 7fcee49fe700 1 -- 172.21.15.98:0/3900948467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee015cdf0 con 0x7fcee00cf570 2023-03-03T07:56:53.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.766+0000 7fced77fe700 1 -- 172.21.15.98:0/3900948467 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcec8019030 con 0x7fcee00cf570 2023-03-03T07:56:53.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.767+0000 7fced77fe700 1 --2- 172.21.15.98:0/3900948467 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcec00682f0 0x7fcec006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.767+0000 7fced77fe700 1 -- 172.21.15.98:0/3900948467 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fcec8084f60 con 0x7fcee00cf570 2023-03-03T07:56:53.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.767+0000 7fceded9d700 1 --2- 172.21.15.98:0/3900948467 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcec00682f0 0x7fcec006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.768+0000 7fceded9d700 1 --2- 172.21.15.98:0/3900948467 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcec00682f0 0x7fcec006a7a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fced00090b0 tx=0x7fced001c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:53.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.771+0000 7fced77fe700 1 -- 172.21.15.98:0/3900948467 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcec804b5e0 con 0x7fcee00cf570 2023-03-03T07:56:53.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.962+0000 7fcee49fe700 1 -- 172.21.15.98:0/3900948467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcee0068290 con 0x7fcee00cf570 2023-03-03T07:56:53.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.963+0000 7fced77fe700 1 -- 172.21.15.98:0/3900948467 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcec8017020 con 0x7fcee00cf570 2023-03-03T07:56:53.965 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:56:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.965+0000 7fcee49fe700 1 -- 172.21.15.98:0/3900948467 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcec00682f0 msgr2=0x7fcec006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.965+0000 7fcee49fe700 1 --2- 172.21.15.98:0/3900948467 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcec00682f0 0x7fcec006a7a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fced00090b0 tx=0x7fced001c040 comp rx=0 tx=0).stop 2023-03-03T07:56:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.965+0000 7fcee49fe700 1 -- 172.21.15.98:0/3900948467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee00cf570 msgr2=0x7fcee00c69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.965+0000 7fcee49fe700 1 --2- 172.21.15.98:0/3900948467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee00cf570 0x7fcee00c69b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcec800c700 tx=0x7fcec800ca10 comp rx=0 tx=0).stop 2023-03-03T07:56:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.965+0000 7fcee49fe700 1 -- 172.21.15.98:0/3900948467 shutdown_connections 2023-03-03T07:56:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.965+0000 7fcee49fe700 1 --2- 172.21.15.98:0/3900948467 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcec00682f0 0x7fcec006a7a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.965+0000 7fcee49fe700 1 --2- 172.21.15.98:0/3900948467 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcee01609b0 0x7fcee00b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.965+0000 7fcee49fe700 1 --2- 172.21.15.98:0/3900948467 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcee00b4aa0 0x7fcee0147fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.965+0000 7fcee49fe700 1 --2- 172.21.15.98:0/3900948467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee00cf570 0x7fcee00c69b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.965+0000 7fcee49fe700 1 -- 172.21.15.98:0/3900948467 >> 172.21.15.98:0/3900948467 conn(0x7fcee006b880 msgr2=0x7fcee00b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.966+0000 7fcee49fe700 1 -- 172.21.15.98:0/3900948467 shutdown_connections 2023-03-03T07:56:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:53.966+0000 7fcee49fe700 1 -- 172.21.15.98:0/3900948467 wait complete. 2023-03-03T07:56:53.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:56:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.241+0000 7f129abff700 1 Processor -- start 2023-03-03T07:56:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.241+0000 7f129abff700 1 -- start start 2023-03-03T07:56:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.242+0000 7f129abff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f129406b4b0 0x7f12940cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.242+0000 7f129abff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12940cf980 con 0x7f12940d0270 2023-03-03T07:56:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.243+0000 7f129abff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12940d1800 con 0x7f129415ca70 2023-03-03T07:56:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.243+0000 7f129abff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12940cd5b0 con 0x7f129406b4b0 2023-03-03T07:56:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.243+0000 7f1293fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f12940d0270 0x7f12940d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46968/0 (socket says 172.21.15.98:46968) 2023-03-03T07:56:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.243+0000 7f1293fff700 1 -- 172.21.15.98:0/3449188076 learned_addr learned my addr 172.21.15.98:0/3449188076 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:54.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.243+0000 7f129919c700 1 --2- 172.21.15.98:0/3449188076 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f129406b4b0 0x7f12940cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:54.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.243+0000 7f12937fe700 1 -- 172.21.15.98:0/3449188076 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384995278 0 0) 0x7f12940cf980 con 0x7f12940d0270 2023-03-03T07:56:54.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.243+0000 7f12937fe700 1 -- 172.21.15.98:0/3449188076 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12840035b0 con 0x7f12940d0270 2023-03-03T07:56:54.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.243+0000 7f12937fe700 1 -- 172.21.15.98:0/3449188076 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204110015 0 0) 0x7f12940d1800 con 0x7f129415ca70 2023-03-03T07:56:54.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.243+0000 7f12937fe700 1 -- 172.21.15.98:0/3449188076 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12940cf980 con 0x7f129415ca70 2023-03-03T07:56:54.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.244+0000 7f129919c700 1 -- 172.21.15.98:0/3449188076 >> v1:172.21.15.100:6789/0 conn(0x7f129415ca70 legacy=0x7f12940cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:54.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.244+0000 7f129919c700 1 -- 172.21.15.98:0/3449188076 >> v1:172.21.15.98:6789/0 conn(0x7f12940d0270 legacy=0x7f12940d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:54.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.244+0000 7f129919c700 1 -- 172.21.15.98:0/3449188076 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12940cd900 con 0x7f129406b4b0 2023-03-03T07:56:54.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.244+0000 7f129919c700 1 --2- 172.21.15.98:0/3449188076 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f129406b4b0 0x7f12940cf440 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f128c009dc0 tx=0x7f128c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9884d17e8d9ab20e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:54.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.244+0000 7f12937fe700 1 -- 172.21.15.98:0/3449188076 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c00e070 con 0x7f129406b4b0 2023-03-03T07:56:54.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.244+0000 7f12937fe700 1 -- 172.21.15.98:0/3449188076 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c00bb20 con 0x7f129406b4b0 2023-03-03T07:56:54.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.245+0000 7f12937fe700 1 -- 172.21.15.98:0/3449188076 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c012430 con 0x7f129406b4b0 2023-03-03T07:56:54.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.245+0000 7f129abff700 1 -- 172.21.15.98:0/3449188076 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f129406b4b0 msgr2=0x7f12940cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:54.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.245+0000 7f129abff700 1 --2- 172.21.15.98:0/3449188076 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f129406b4b0 0x7f12940cf440 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f128c009dc0 tx=0x7f128c00b330 comp rx=0 tx=0).stop 2023-03-03T07:56:54.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.245+0000 7f129abff700 1 -- 172.21.15.98:0/3449188076 shutdown_connections 2023-03-03T07:56:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.245+0000 7f129abff700 1 --2- 172.21.15.98:0/3449188076 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f129406b4b0 0x7f12940cf440 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.245+0000 7f129abff700 1 -- 172.21.15.98:0/3449188076 >> 172.21.15.98:0/3449188076 conn(0x7f1294073e70 msgr2=0x7f129406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.245+0000 7f129abff700 1 -- 172.21.15.98:0/3449188076 shutdown_connections 2023-03-03T07:56:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.245+0000 7f129abff700 1 -- 172.21.15.98:0/3449188076 wait complete. 2023-03-03T07:56:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.247+0000 7f129abff700 1 Processor -- start 2023-03-03T07:56:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.247+0000 7f129abff700 1 -- start start 2023-03-03T07:56:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.248+0000 7f129abff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f129406b4b0 0x7f12940ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.248+0000 7f129abff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12940d0270 0x7f1294155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:54.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.248+0000 7f129abff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f129415ca70 0x7f1294156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:54.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.248+0000 7f129abff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12941579a0 con 0x7f12940d0270 2023-03-03T07:56:54.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.248+0000 7f129abff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129414ede0 con 0x7f129415ca70 2023-03-03T07:56:54.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.248+0000 7f129abff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1294156900 con 0x7f129406b4b0 2023-03-03T07:56:54.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.249+0000 7f1293fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12940d0270 0x7f1294155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:54.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.249+0000 7f1293fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12940d0270 0x7f1294155d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51732/0 (socket says 172.21.15.98:51732) 2023-03-03T07:56:54.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.249+0000 7f1293fff700 1 -- 172.21.15.98:0/2503081621 learned_addr learned my addr 172.21.15.98:0/2503081621 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:54.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.249+0000 7f129899b700 1 --2- 172.21.15.98:0/2503081621 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f129406b4b0 0x7f12940ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:54.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.249+0000 7f129919c700 1 --2- 172.21.15.98:0/2503081621 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f129415ca70 0x7f1294156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:54.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.249+0000 7f1293fff700 1 -- 172.21.15.98:0/2503081621 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f129406b4b0 msgr2=0x7f12940ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:54.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.249+0000 7f1293fff700 1 --2- 172.21.15.98:0/2503081621 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f129406b4b0 0x7f12940ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.249+0000 7f1293fff700 1 -- 172.21.15.98:0/2503081621 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f129415ca70 msgr2=0x7f1294156260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:54.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.249+0000 7f1293fff700 1 --2- 172.21.15.98:0/2503081621 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f129415ca70 0x7f1294156260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.249+0000 7f1293fff700 1 -- 172.21.15.98:0/2503081621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c009a70 con 0x7f12940d0270 2023-03-03T07:56:54.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.249+0000 7f129899b700 1 --2- 172.21.15.98:0/2503081621 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f129406b4b0 0x7f12940ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:54.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.250+0000 7f1293fff700 1 --2- 172.21.15.98:0/2503081621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12940d0270 0x7f1294155d20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f128000c500 tx=0x7f128000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:54.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.250+0000 7f12917fa700 1 -- 172.21.15.98:0/2503081621 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1280005d70 con 0x7f12940d0270 2023-03-03T07:56:54.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.250+0000 7f12917fa700 1 -- 172.21.15.98:0/2503081621 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12800070d0 con 0x7f12940d0270 2023-03-03T07:56:54.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.250+0000 7f12917fa700 1 -- 172.21.15.98:0/2503081621 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1280007460 con 0x7f12940d0270 2023-03-03T07:56:54.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.250+0000 7f129abff700 1 -- 172.21.15.98:0/2503081621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12940d1aa0 con 0x7f12940d0270 2023-03-03T07:56:54.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.250+0000 7f129abff700 1 -- 172.21.15.98:0/2503081621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12940cdbe0 con 0x7f12940d0270 2023-03-03T07:56:54.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.251+0000 7f129abff700 1 -- 172.21.15.98:0/2503081621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129415ce40 con 0x7f12940d0270 2023-03-03T07:56:54.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.252+0000 7f12917fa700 1 -- 172.21.15.98:0/2503081621 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1280019030 con 0x7f12940d0270 2023-03-03T07:56:54.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.254+0000 7f12917fa700 1 --2- 172.21.15.98:0/2503081621 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f12780682f0 0x7f127806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:54.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.254+0000 7f12917fa700 1 -- 172.21.15.98:0/2503081621 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f1280085140 con 0x7f12940d0270 2023-03-03T07:56:54.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.254+0000 7f129899b700 1 --2- 172.21.15.98:0/2503081621 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f12780682f0 0x7f127806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:54.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.255+0000 7f129899b700 1 --2- 172.21.15.98:0/2503081621 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f12780682f0 0x7f127806a7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1288007fd0 tx=0x7f1288009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:54.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.257+0000 7f12917fa700 1 -- 172.21.15.98:0/2503081621 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f12800517e0 con 0x7f12940d0270 2023-03-03T07:56:54.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.449+0000 7f129abff700 1 -- 172.21.15.98:0/2503081621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f12941576c0 con 0x7f12940d0270 2023-03-03T07:56:54.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.450+0000 7f12917fa700 1 -- 172.21.15.98:0/2503081621 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1280017020 con 0x7f12940d0270 2023-03-03T07:56:54.452 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:56:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.452+0000 7f129abff700 1 -- 172.21.15.98:0/2503081621 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f12780682f0 msgr2=0x7f127806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.452+0000 7f129abff700 1 --2- 172.21.15.98:0/2503081621 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f12780682f0 0x7f127806a7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1288007fd0 tx=0x7f1288009040 comp rx=0 tx=0).stop 2023-03-03T07:56:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.452+0000 7f129abff700 1 -- 172.21.15.98:0/2503081621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12940d0270 msgr2=0x7f1294155d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.452+0000 7f129abff700 1 --2- 172.21.15.98:0/2503081621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12940d0270 0x7f1294155d20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f128000c500 tx=0x7f128000c810 comp rx=0 tx=0).stop 2023-03-03T07:56:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.452+0000 7f129abff700 1 -- 172.21.15.98:0/2503081621 shutdown_connections 2023-03-03T07:56:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.452+0000 7f129abff700 1 --2- 172.21.15.98:0/2503081621 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f12780682f0 0x7f127806a7a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.452+0000 7f129abff700 1 --2- 172.21.15.98:0/2503081621 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f129415ca70 0x7f1294156260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.452+0000 7f129abff700 1 --2- 172.21.15.98:0/2503081621 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f129406b4b0 0x7f12940ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.452+0000 7f129abff700 1 --2- 172.21.15.98:0/2503081621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12940d0270 0x7f1294155d20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.452+0000 7f129abff700 1 -- 172.21.15.98:0/2503081621 >> 172.21.15.98:0/2503081621 conn(0x7f1294073e70 msgr2=0x7f12941597f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.452+0000 7f129abff700 1 -- 172.21.15.98:0/2503081621 shutdown_connections 2023-03-03T07:56:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.452+0000 7f129abff700 1 -- 172.21.15.98:0/2503081621 wait complete. 2023-03-03T07:56:54.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T07:56:54.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.725+0000 7f6710b9d700 1 Processor -- start 2023-03-03T07:56:54.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.726+0000 7f6710b9d700 1 -- start start 2023-03-03T07:56:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.727+0000 7f6710b9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f670c1609b0 0x7f670c160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.727+0000 7f6710b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f670c0b4aa0 0x7f670c0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.727+0000 7f6710b9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f670c1612c0 con 0x7f670c0cf570 2023-03-03T07:56:54.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.727+0000 7f6710b9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f670c0cebb0 con 0x7f670c1609b0 2023-03-03T07:56:54.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.727+0000 7f6710b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670c0b5460 con 0x7f670c0b4aa0 2023-03-03T07:56:54.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.727+0000 7f670b7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f670c1609b0 0x7f670c160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:54.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.727+0000 7f670affd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f670c0cf570 0x7f670c15d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46978/0 (socket says 172.21.15.98:46978) 2023-03-03T07:56:54.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.727+0000 7f670affd700 1 -- 172.21.15.98:0/1384875930 learned_addr learned my addr 172.21.15.98:0/1384875930 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:54.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.728+0000 7f670bfff700 1 --2- 172.21.15.98:0/1384875930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f670c0b4aa0 0x7f670c0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:54.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.728+0000 7f670b7fe700 1 -- 172.21.15.98:0/1384875930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f670c0b4aa0 msgr2=0x7f670c0b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:54.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.728+0000 7f670b7fe700 1 --2- 172.21.15.98:0/1384875930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f670c0b4aa0 0x7f670c0b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.728+0000 7f670b7fe700 1 -- 172.21.15.98:0/1384875930 >> v1:172.21.15.98:6789/0 conn(0x7f670c0cf570 legacy=0x7f670c15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.728+0000 7f670b7fe700 1 -- 172.21.15.98:0/1384875930 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c15dd00 con 0x7f670c1609b0 2023-03-03T07:56:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.728+0000 7f670a7fc700 1 -- 172.21.15.98:0/1384875930 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 479376958 0 0) 0x7f670c1612c0 con 0x7f670c0cf570 2023-03-03T07:56:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.728+0000 7f670b7fe700 1 --2- 172.21.15.98:0/1384875930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f670c1609b0 0x7f670c160d80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f66fc009840 tx=0x7f66fc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f47bcec20872f2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.728+0000 7f670a7fc700 1 -- 172.21.15.98:0/1384875930 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66fc011830 con 0x7f670c1609b0 2023-03-03T07:56:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.729+0000 7f670a7fc700 1 -- 172.21.15.98:0/1384875930 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66fc011990 con 0x7f670c1609b0 2023-03-03T07:56:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.729+0000 7f670a7fc700 1 -- 172.21.15.98:0/1384875930 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66fc011d20 con 0x7f670c1609b0 2023-03-03T07:56:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.729+0000 7f6710b9d700 1 -- 172.21.15.98:0/1384875930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f670c1609b0 msgr2=0x7f670c160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.729+0000 7f6710b9d700 1 --2- 172.21.15.98:0/1384875930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f670c1609b0 0x7f670c160d80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f66fc009840 tx=0x7f66fc009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.729+0000 7f6710b9d700 1 -- 172.21.15.98:0/1384875930 shutdown_connections 2023-03-03T07:56:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.729+0000 7f6710b9d700 1 --2- 172.21.15.98:0/1384875930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f670c1609b0 0x7f670c160d80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.729+0000 7f6710b9d700 1 --2- 172.21.15.98:0/1384875930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f670c0b4aa0 0x7f670c0b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.729+0000 7f6710b9d700 1 -- 172.21.15.98:0/1384875930 >> 172.21.15.98:0/1384875930 conn(0x7f670c06b880 msgr2=0x7f670c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.729+0000 7f6710b9d700 1 -- 172.21.15.98:0/1384875930 shutdown_connections 2023-03-03T07:56:54.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.729+0000 7f6710b9d700 1 -- 172.21.15.98:0/1384875930 wait complete. 2023-03-03T07:56:54.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.731+0000 7f6710b9d700 1 Processor -- start 2023-03-03T07:56:54.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.732+0000 7f6710b9d700 1 -- start start 2023-03-03T07:56:54.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.732+0000 7f6710b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f670c0b4aa0 0x7f670c0c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:54.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.732+0000 7f6710b9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f670c0cf570 0x7f670c0c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.733+0000 7f6710b9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f670c1609b0 0x7f670c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.733+0000 7f6710b9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670c0ca6e0 con 0x7f670c1609b0 2023-03-03T07:56:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.733+0000 7f6710b9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670c0ca0f0 con 0x7f670c0cf570 2023-03-03T07:56:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.733+0000 7f6710b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670c0c9700 con 0x7f670c0b4aa0 2023-03-03T07:56:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.733+0000 7f670bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f670c1609b0 0x7f670c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.733+0000 7f670bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f670c1609b0 0x7f670c0cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51762/0 (socket says 172.21.15.98:51762) 2023-03-03T07:56:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.733+0000 7f670bfff700 1 -- 172.21.15.98:0/3815188615 learned_addr learned my addr 172.21.15.98:0/3815188615 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.733+0000 7f670b7fe700 1 --2- 172.21.15.98:0/3815188615 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f670c0b4aa0 0x7f670c0c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.733+0000 7f670affd700 1 --2- 172.21.15.98:0/3815188615 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f670c0cf570 0x7f670c0c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.733+0000 7f670bfff700 1 -- 172.21.15.98:0/3815188615 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f670c0b4aa0 msgr2=0x7f670c0c81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.733+0000 7f670bfff700 1 --2- 172.21.15.98:0/3815188615 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f670c0b4aa0 0x7f670c0c81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.733+0000 7f670bfff700 1 -- 172.21.15.98:0/3815188615 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f670c0cf570 msgr2=0x7f670c0c91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.733+0000 7f670bfff700 1 --2- 172.21.15.98:0/3815188615 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f670c0cf570 0x7f670c0c91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.733+0000 7f670bfff700 1 -- 172.21.15.98:0/3815188615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66fc0094a0 con 0x7f670c1609b0 2023-03-03T07:56:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.733+0000 7f670b7fe700 1 --2- 172.21.15.98:0/3815188615 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f670c0b4aa0 0x7f670c0c81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.734+0000 7f670bfff700 1 --2- 172.21.15.98:0/3815188615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f670c1609b0 0x7f670c0cc600 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6704007e40 tx=0x7f670400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.734+0000 7f66f3fff700 1 -- 172.21.15.98:0/3815188615 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670400eea0 con 0x7f670c1609b0 2023-03-03T07:56:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.734+0000 7f66f3fff700 1 -- 172.21.15.98:0/3815188615 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6704014420 con 0x7f670c1609b0 2023-03-03T07:56:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.734+0000 7f66f3fff700 1 -- 172.21.15.98:0/3815188615 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67040147b0 con 0x7f670c1609b0 2023-03-03T07:56:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.734+0000 7f6710b9d700 1 -- 172.21.15.98:0/3815188615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f670c0cda60 con 0x7f670c1609b0 2023-03-03T07:56:54.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.734+0000 7f6710b9d700 1 -- 172.21.15.98:0/3815188615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f670c0cd780 con 0x7f670c1609b0 2023-03-03T07:56:54.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.735+0000 7f6710b9d700 1 -- 172.21.15.98:0/3815188615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f670c0cdd40 con 0x7f670c1609b0 2023-03-03T07:56:54.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.737+0000 7f66f3fff700 1 -- 172.21.15.98:0/3815188615 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6704012050 con 0x7f670c1609b0 2023-03-03T07:56:54.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.738+0000 7f66f3fff700 1 --2- 172.21.15.98:0/3815188615 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f66f80682f0 0x7f66f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:54.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.738+0000 7f66f3fff700 1 -- 172.21.15.98:0/3815188615 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f6704086490 con 0x7f670c1609b0 2023-03-03T07:56:54.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.738+0000 7f670b7fe700 1 --2- 172.21.15.98:0/3815188615 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f66f80682f0 0x7f66f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:54.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.739+0000 7f670b7fe700 1 --2- 172.21.15.98:0/3815188615 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f66f80682f0 0x7f66f806a7a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f66fc0090b0 tx=0x7f66fc01c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:54.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.742+0000 7f66f3fff700 1 -- 172.21.15.98:0/3815188615 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6704052bc0 con 0x7f670c1609b0 2023-03-03T07:56:54.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.951+0000 7f6710b9d700 1 -- 172.21.15.98:0/3815188615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f670c0ca820 con 0x7f670c1609b0 2023-03-03T07:56:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.952+0000 7f66f3fff700 1 -- 172.21.15.98:0/3815188615 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7f670c0ca820 con 0x7f670c1609b0 2023-03-03T07:56:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.954+0000 7f6710b9d700 1 -- 172.21.15.98:0/3815188615 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f66f80682f0 msgr2=0x7f66f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.954+0000 7f6710b9d700 1 --2- 172.21.15.98:0/3815188615 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f66f80682f0 0x7f66f806a7a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f66fc0090b0 tx=0x7f66fc01c040 comp rx=0 tx=0).stop 2023-03-03T07:56:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.954+0000 7f6710b9d700 1 -- 172.21.15.98:0/3815188615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f670c1609b0 msgr2=0x7f670c0cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.954+0000 7f6710b9d700 1 --2- 172.21.15.98:0/3815188615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f670c1609b0 0x7f670c0cc600 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6704007e40 tx=0x7f670400e3c0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.954+0000 7f6710b9d700 1 -- 172.21.15.98:0/3815188615 shutdown_connections 2023-03-03T07:56:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.954+0000 7f6710b9d700 1 --2- 172.21.15.98:0/3815188615 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f66f80682f0 0x7f66f806a7a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.954+0000 7f6710b9d700 1 --2- 172.21.15.98:0/3815188615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f670c1609b0 0x7f670c0cc600 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.954+0000 7f6710b9d700 1 --2- 172.21.15.98:0/3815188615 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f670c0b4aa0 0x7f670c0c81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.954+0000 7f6710b9d700 1 --2- 172.21.15.98:0/3815188615 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f670c0cf570 0x7f670c0c91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.954+0000 7f6710b9d700 1 -- 172.21.15.98:0/3815188615 >> 172.21.15.98:0/3815188615 conn(0x7f670c06b880 msgr2=0x7f670c15df20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.955+0000 7f6710b9d700 1 -- 172.21.15.98:0/3815188615 shutdown_connections 2023-03-03T07:56:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:54.955+0000 7f6710b9d700 1 -- 172.21.15.98:0/3815188615 wait complete. 2023-03-03T07:56:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T07:56:54.982 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:56:54.983 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:56:55.060 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:56:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.303+0000 7f77895ff700 1 Processor -- start 2023-03-03T07:56:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.303+0000 7f77895ff700 1 -- start start 2023-03-03T07:56:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.304+0000 7f77895ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f77840b87c0 0x7f77840cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.304+0000 7f77895ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f77840b8b90 0x7f77840b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.305+0000 7f77895ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77840d0ef0 con 0x7f77840b5240 2023-03-03T07:56:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.305+0000 7f77895ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77840d0600 con 0x7f77840b8b90 2023-03-03T07:56:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.305+0000 7f77895ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77840cd970 con 0x7f77840b87c0 2023-03-03T07:56:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.305+0000 7f77827fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f77840b8b90 0x7f77840b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.305+0000 7f77837fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f77840b5240 0x7f77841605b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34300/0 (socket says 172.21.15.98:34300) 2023-03-03T07:56:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.305+0000 7f77837fe700 1 -- 172.21.15.98:0/2801706945 learned_addr learned my addr 172.21.15.98:0/2801706945 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.305+0000 7f7782ffd700 1 --2- 172.21.15.98:0/2801706945 >> v2:172.21.15.100:3300/0 conn(0x7f77840b87c0 0x7f77840cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.305+0000 7f77827fc700 1 -- 172.21.15.98:0/2801706945 >> v2:172.21.15.100:3300/0 conn(0x7f77840b87c0 msgr2=0x7f77840cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.305+0000 7f77827fc700 1 --2- 172.21.15.98:0/2801706945 >> v2:172.21.15.100:3300/0 conn(0x7f77840b87c0 0x7f77840cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.305+0000 7f77827fc700 1 -- 172.21.15.98:0/2801706945 >> v1:172.21.15.98:6789/0 conn(0x7f77840b5240 legacy=0x7f77841605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.305+0000 7f77827fc700 1 -- 172.21.15.98:0/2801706945 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77840cdc50 con 0x7f77840b8b90 2023-03-03T07:56:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.305+0000 7f7781ffb700 1 -- 172.21.15.98:0/2801706945 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694013042 0 0) 0x7f77840d0ef0 con 0x7f77840b5240 2023-03-03T07:56:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.306+0000 7f77827fc700 1 --2- 172.21.15.98:0/2801706945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77840b8b90 0x7f77840b4d00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7784073840 tx=0x7f7778009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b1dfc708edcf18 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.306+0000 7f7781ffb700 1 -- 172.21.15.98:0/2801706945 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778011830 con 0x7f77840b8b90 2023-03-03T07:56:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.306+0000 7f7781ffb700 1 -- 172.21.15.98:0/2801706945 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7778011990 con 0x7f77840b8b90 2023-03-03T07:56:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.306+0000 7f7781ffb700 1 -- 172.21.15.98:0/2801706945 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778011d20 con 0x7f77840b8b90 2023-03-03T07:56:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.307+0000 7f77895ff700 1 -- 172.21.15.98:0/2801706945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77840b8b90 msgr2=0x7f77840b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.307+0000 7f77895ff700 1 --2- 172.21.15.98:0/2801706945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77840b8b90 0x7f77840b4d00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7784073840 tx=0x7f7778009b20 comp rx=0 tx=0).stop 2023-03-03T07:56:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.307+0000 7f77895ff700 1 -- 172.21.15.98:0/2801706945 shutdown_connections 2023-03-03T07:56:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.307+0000 7f77895ff700 1 --2- 172.21.15.98:0/2801706945 >> v2:172.21.15.100:3300/0 conn(0x7f77840b87c0 0x7f77840cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.307+0000 7f77895ff700 1 --2- 172.21.15.98:0/2801706945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77840b8b90 0x7f77840b4d00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.307+0000 7f77895ff700 1 -- 172.21.15.98:0/2801706945 >> 172.21.15.98:0/2801706945 conn(0x7f778406aee0 msgr2=0x7f778406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.307+0000 7f77895ff700 1 -- 172.21.15.98:0/2801706945 shutdown_connections 2023-03-03T07:56:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.307+0000 7f77895ff700 1 -- 172.21.15.98:0/2801706945 wait complete. 2023-03-03T07:56:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.310+0000 7f77895ff700 1 Processor -- start 2023-03-03T07:56:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.310+0000 7f77895ff700 1 -- start start 2023-03-03T07:56:55.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.311+0000 7f77895ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77840b5240 0x7f77841f1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.311+0000 7f77895ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77840b87c0 0x7f77840b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.311+0000 7f77895ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f77840b8b90 0x7f77840cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.311+0000 7f77895ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77840c8240 con 0x7f77840b87c0 2023-03-03T07:56:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.311+0000 7f77895ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77840c8520 con 0x7f77840b8b90 2023-03-03T07:56:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.311+0000 7f77895ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77840c8800 con 0x7f77840b5240 2023-03-03T07:56:55.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.312+0000 7f77827fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77840b87c0 0x7f77840b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:55.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.312+0000 7f77827fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77840b87c0 0x7f77840b1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45638/0 (socket says 172.21.15.98:45638) 2023-03-03T07:56:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.312+0000 7f77827fc700 1 -- 172.21.15.98:0/3925481271 learned_addr learned my addr 172.21.15.98:0/3925481271 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.312+0000 7f7782ffd700 1 --2- 172.21.15.98:0/3925481271 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77840b5240 0x7f77841f1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.312+0000 7f77837fe700 1 --2- 172.21.15.98:0/3925481271 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f77840b8b90 0x7f77840cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.312+0000 7f77827fc700 1 -- 172.21.15.98:0/3925481271 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77840b5240 msgr2=0x7f77841f1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.312+0000 7f77827fc700 1 --2- 172.21.15.98:0/3925481271 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77840b5240 0x7f77841f1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.312+0000 7f77827fc700 1 -- 172.21.15.98:0/3925481271 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f77840b8b90 msgr2=0x7f77840cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.312+0000 7f77827fc700 1 --2- 172.21.15.98:0/3925481271 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f77840b8b90 0x7f77840cbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.312+0000 7f77827fc700 1 -- 172.21.15.98:0/3925481271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77780094a0 con 0x7f77840b87c0 2023-03-03T07:56:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.312+0000 7f77827fc700 1 --2- 172.21.15.98:0/3925481271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77840b87c0 0x7f77840b1d20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7778014040 tx=0x7f7778004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.313+0000 7f776b7fe700 1 -- 172.21.15.98:0/3925481271 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77780038a0 con 0x7f77840b87c0 2023-03-03T07:56:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.313+0000 7f776b7fe700 1 -- 172.21.15.98:0/3925481271 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777801e070 con 0x7f77840b87c0 2023-03-03T07:56:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.313+0000 7f776b7fe700 1 -- 172.21.15.98:0/3925481271 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778003c20 con 0x7f77840b87c0 2023-03-03T07:56:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.313+0000 7f77895ff700 1 -- 172.21.15.98:0/3925481271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77840cdf30 con 0x7f77840b87c0 2023-03-03T07:56:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.313+0000 7f77895ff700 1 -- 172.21.15.98:0/3925481271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7784073a80 con 0x7f77840b87c0 2023-03-03T07:56:55.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.314+0000 7f77895ff700 1 -- 172.21.15.98:0/3925481271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77840ce0c0 con 0x7f77840b87c0 2023-03-03T07:56:55.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.315+0000 7f776b7fe700 1 -- 172.21.15.98:0/3925481271 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f777800f050 con 0x7f77840b87c0 2023-03-03T07:56:55.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.316+0000 7f776b7fe700 1 --2- 172.21.15.98:0/3925481271 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f77700682f0 0x7f777006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:55.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.316+0000 7f776b7fe700 1 -- 172.21.15.98:0/3925481271 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f7778086dc0 con 0x7f77840b87c0 2023-03-03T07:56:55.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.317+0000 7f7782ffd700 1 --2- 172.21.15.98:0/3925481271 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f77700682f0 0x7f777006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:55.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.318+0000 7f7782ffd700 1 --2- 172.21.15.98:0/3925481271 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f77700682f0 0x7f777006a7a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7774009930 tx=0x7f7774008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:55.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.321+0000 7f776b7fe700 1 -- 172.21.15.98:0/3925481271 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f77780534f0 con 0x7f77840b87c0 2023-03-03T07:56:55.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.516+0000 7f77895ff700 1 -- 172.21.15.98:0/3925481271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77840729b0 con 0x7f77840b87c0 2023-03-03T07:56:55.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.518+0000 7f776b7fe700 1 -- 172.21.15.98:0/3925481271 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18003 (secure 0 0 0) 0x7f7778015090 con 0x7f77840b87c0 2023-03-03T07:56:55.519 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:56:55.519 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:56:27.498205+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:56:55.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.520+0000 7f77895ff700 1 -- 172.21.15.98:0/3925481271 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f77700682f0 msgr2=0x7f777006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:55.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.520+0000 7f77895ff700 1 --2- 172.21.15.98:0/3925481271 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f77700682f0 0x7f777006a7a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7774009930 tx=0x7f7774008040 comp rx=0 tx=0).stop 2023-03-03T07:56:55.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.520+0000 7f77895ff700 1 -- 172.21.15.98:0/3925481271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77840b87c0 msgr2=0x7f77840b1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:55.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.520+0000 7f77895ff700 1 --2- 172.21.15.98:0/3925481271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77840b87c0 0x7f77840b1d20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7778014040 tx=0x7f7778004750 comp rx=0 tx=0).stop 2023-03-03T07:56:55.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.521+0000 7f77895ff700 1 -- 172.21.15.98:0/3925481271 shutdown_connections 2023-03-03T07:56:55.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.521+0000 7f77895ff700 1 --2- 172.21.15.98:0/3925481271 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f77700682f0 0x7f777006a7a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:55.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.521+0000 7f77895ff700 1 --2- 172.21.15.98:0/3925481271 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f77840b8b90 0x7f77840cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:55.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.521+0000 7f77895ff700 1 --2- 172.21.15.98:0/3925481271 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77840b5240 0x7f77841f1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:55.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.521+0000 7f77895ff700 1 --2- 172.21.15.98:0/3925481271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77840b87c0 0x7f77840b1d20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:55.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.521+0000 7f77895ff700 1 -- 172.21.15.98:0/3925481271 >> 172.21.15.98:0/3925481271 conn(0x7f778406aee0 msgr2=0x7f77840d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:55.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.521+0000 7f77895ff700 1 -- 172.21.15.98:0/3925481271 shutdown_connections 2023-03-03T07:56:55.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.521+0000 7f77895ff700 1 -- 172.21.15.98:0/3925481271 wait complete. 2023-03-03T07:56:55.550 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:55:45.157666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:55:46.092091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:56:55.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:56:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.798+0000 7f31aef9d700 1 Processor -- start 2023-03-03T07:56:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.799+0000 7f31aef9d700 1 -- start start 2023-03-03T07:56:55.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.800+0000 7f31aef9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f31a81609b0 0x7f31a8160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:55.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.800+0000 7f31aef9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a81612c0 con 0x7f31a80cf570 2023-03-03T07:56:55.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.800+0000 7f31aef9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a815dd00 con 0x7f31a80b4aa0 2023-03-03T07:56:55.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.800+0000 7f31aef9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a80cebb0 con 0x7f31a81609b0 2023-03-03T07:56:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.800+0000 7f31ad79a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f31a80cf570 0x7f31a815d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34312/0 (socket says 172.21.15.98:34312) 2023-03-03T07:56:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.800+0000 7f31ad79a700 1 -- 172.21.15.98:0/3631713282 learned_addr learned my addr 172.21.15.98:0/3631713282 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.801+0000 7f31adf9b700 1 --2- 172.21.15.98:0/3631713282 >> v2:172.21.15.100:3300/0 conn(0x7f31a81609b0 0x7f31a8160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.801+0000 7f31acf99700 1 -- 172.21.15.98:0/3631713282 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272356446 0 0) 0x7f31a81612c0 con 0x7f31a80cf570 2023-03-03T07:56:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.801+0000 7f31acf99700 1 -- 172.21.15.98:0/3631713282 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31940035b0 con 0x7f31a80cf570 2023-03-03T07:56:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.801+0000 7f31acf99700 1 -- 172.21.15.98:0/3631713282 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466130612 0 0) 0x7f31a815dd00 con 0x7f31a80b4aa0 2023-03-03T07:56:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.801+0000 7f31acf99700 1 -- 172.21.15.98:0/3631713282 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31a81612c0 con 0x7f31a80b4aa0 2023-03-03T07:56:55.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.801+0000 7f31acf99700 1 -- 172.21.15.98:0/3631713282 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3008596581 0 0) 0x7f31940035b0 con 0x7f31a80cf570 2023-03-03T07:56:55.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.801+0000 7f31adf9b700 1 -- 172.21.15.98:0/3631713282 >> v1:172.21.15.100:6789/0 conn(0x7f31a80b4aa0 legacy=0x7f31a80d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:55.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.801+0000 7f31adf9b700 1 -- 172.21.15.98:0/3631713282 >> v1:172.21.15.98:6789/0 conn(0x7f31a80cf570 legacy=0x7f31a815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:55.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.801+0000 7f31adf9b700 1 -- 172.21.15.98:0/3631713282 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a80cd780 con 0x7f31a81609b0 2023-03-03T07:56:55.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.802+0000 7f31adf9b700 1 --2- 172.21.15.98:0/3631713282 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a81609b0 0x7f31a8160d80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f31a4009840 tx=0x7f31a4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=369aa0c0e8f909f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:55.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.802+0000 7f31acf99700 1 -- 172.21.15.98:0/3631713282 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198001670 con 0x7f31a81609b0 2023-03-03T07:56:55.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.802+0000 7f31acf99700 1 -- 172.21.15.98:0/3631713282 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a4004030 con 0x7f31a81609b0 2023-03-03T07:56:55.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.802+0000 7f31acf99700 1 -- 172.21.15.98:0/3631713282 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a40043c0 con 0x7f31a81609b0 2023-03-03T07:56:55.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.802+0000 7f31aef9d700 1 -- 172.21.15.98:0/3631713282 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a81609b0 msgr2=0x7f31a8160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:55.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.802+0000 7f31aef9d700 1 --2- 172.21.15.98:0/3631713282 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a81609b0 0x7f31a8160d80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f31a4009840 tx=0x7f31a4009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:55.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.803+0000 7f31aef9d700 1 -- 172.21.15.98:0/3631713282 shutdown_connections 2023-03-03T07:56:55.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.803+0000 7f31aef9d700 1 --2- 172.21.15.98:0/3631713282 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a81609b0 0x7f31a8160d80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:55.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.803+0000 7f31aef9d700 1 -- 172.21.15.98:0/3631713282 >> 172.21.15.98:0/3631713282 conn(0x7f31a806b880 msgr2=0x7f31a80cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:55.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.803+0000 7f31aef9d700 1 -- 172.21.15.98:0/3631713282 shutdown_connections 2023-03-03T07:56:55.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.803+0000 7f31aef9d700 1 -- 172.21.15.98:0/3631713282 wait complete. 2023-03-03T07:56:55.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.805+0000 7f31aef9d700 1 Processor -- start 2023-03-03T07:56:55.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.805+0000 7f31aef9d700 1 -- start start 2023-03-03T07:56:55.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.806+0000 7f31aef9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a80b4aa0 0x7f31a8150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:55.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.806+0000 7f31aef9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80cf570 0x7f31a80b6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:55.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.806+0000 7f31aef9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a81609b0 0x7f31a8150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:55.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.806+0000 7f31aef9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80b8730 con 0x7f31a81609b0 2023-03-03T07:56:55.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.806+0000 7f31aef9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80cb180 con 0x7f31a80b4aa0 2023-03-03T07:56:55.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.806+0000 7f31aef9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80b8960 con 0x7f31a80cf570 2023-03-03T07:56:55.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.806+0000 7f31ae79c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a81609b0 0x7f31a8150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:55.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.807+0000 7f31adf9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a80b4aa0 0x7f31a8150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.807+0000 7f31ae79c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a81609b0 0x7f31a8150450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45646/0 (socket says 172.21.15.98:45646) 2023-03-03T07:56:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.807+0000 7f31ad79a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80cf570 0x7f31a80b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.807+0000 7f31ae79c700 1 -- 172.21.15.98:0/1118299943 learned_addr learned my addr 172.21.15.98:0/1118299943 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.807+0000 7f31ae79c700 1 -- 172.21.15.98:0/1118299943 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80cf570 msgr2=0x7f31a80b6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.807+0000 7f31ae79c700 1 --2- 172.21.15.98:0/1118299943 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80cf570 0x7f31a80b6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.807+0000 7f31ae79c700 1 -- 172.21.15.98:0/1118299943 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a80b4aa0 msgr2=0x7f31a8150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.807+0000 7f31ae79c700 1 --2- 172.21.15.98:0/1118299943 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a80b4aa0 0x7f31a8150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:55.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.807+0000 7f31ae79c700 1 -- 172.21.15.98:0/1118299943 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a40094a0 con 0x7f31a81609b0 2023-03-03T07:56:55.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.807+0000 7f31ae79c700 1 --2- 172.21.15.98:0/1118299943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a81609b0 0x7f31a8150450 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f31a000cc20 tx=0x7f31a000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:55.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.808+0000 7f319e7fc700 1 -- 172.21.15.98:0/1118299943 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a0003c00 con 0x7f31a81609b0 2023-03-03T07:56:55.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.808+0000 7f319e7fc700 1 -- 172.21.15.98:0/1118299943 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a0003d60 con 0x7f31a81609b0 2023-03-03T07:56:55.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.808+0000 7f31aef9d700 1 -- 172.21.15.98:0/1118299943 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a80cdd40 con 0x7f31a81609b0 2023-03-03T07:56:55.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.808+0000 7f319e7fc700 1 -- 172.21.15.98:0/1118299943 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a000aae0 con 0x7f31a81609b0 2023-03-03T07:56:55.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.808+0000 7f31aef9d700 1 -- 172.21.15.98:0/1118299943 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a80cda60 con 0x7f31a81609b0 2023-03-03T07:56:55.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.809+0000 7f31aef9d700 1 -- 172.21.15.98:0/1118299943 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a8068290 con 0x7f31a81609b0 2023-03-03T07:56:55.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.811+0000 7f319e7fc700 1 -- 172.21.15.98:0/1118299943 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f31a0010050 con 0x7f31a81609b0 2023-03-03T07:56:55.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.812+0000 7f319e7fc700 1 --2- 172.21.15.98:0/1118299943 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f318c0682f0 0x7f318c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:55.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.812+0000 7f319e7fc700 1 -- 172.21.15.98:0/1118299943 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f31a0085cd0 con 0x7f31a81609b0 2023-03-03T07:56:55.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.812+0000 7f31adf9b700 1 --2- 172.21.15.98:0/1118299943 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f318c0682f0 0x7f318c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:55.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.813+0000 7f31adf9b700 1 --2- 172.21.15.98:0/1118299943 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f318c0682f0 0x7f318c06a7a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f31a40090b0 tx=0x7f31a4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:55.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:55.816+0000 7f319e7fc700 1 -- 172.21.15.98:0/1118299943 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f31a0052400 con 0x7f31a81609b0 2023-03-03T07:56:56.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.007+0000 7f31aef9d700 1 -- 172.21.15.98:0/1118299943 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f31a80b4f90 con 0x7f31a81609b0 2023-03-03T07:56:56.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.008+0000 7f319e7fc700 1 -- 172.21.15.98:0/1118299943 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31a0055a20 con 0x7f31a81609b0 2023-03-03T07:56:56.009 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:56:56.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.010+0000 7f31aef9d700 1 -- 172.21.15.98:0/1118299943 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f318c0682f0 msgr2=0x7f318c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:56.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.010+0000 7f31aef9d700 1 --2- 172.21.15.98:0/1118299943 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f318c0682f0 0x7f318c06a7a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f31a40090b0 tx=0x7f31a4008030 comp rx=0 tx=0).stop 2023-03-03T07:56:56.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.010+0000 7f31aef9d700 1 -- 172.21.15.98:0/1118299943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a81609b0 msgr2=0x7f31a8150450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:56.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.010+0000 7f31aef9d700 1 --2- 172.21.15.98:0/1118299943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a81609b0 0x7f31a8150450 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f31a000cc20 tx=0x7f31a000cf30 comp rx=0 tx=0).stop 2023-03-03T07:56:56.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.010+0000 7f31aef9d700 1 -- 172.21.15.98:0/1118299943 shutdown_connections 2023-03-03T07:56:56.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.010+0000 7f31aef9d700 1 --2- 172.21.15.98:0/1118299943 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f318c0682f0 0x7f318c06a7a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.010+0000 7f31aef9d700 1 --2- 172.21.15.98:0/1118299943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a81609b0 0x7f31a8150450 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.010+0000 7f31aef9d700 1 --2- 172.21.15.98:0/1118299943 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a80b4aa0 0x7f31a8150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.010+0000 7f31aef9d700 1 --2- 172.21.15.98:0/1118299943 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80cf570 0x7f31a80b6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.011+0000 7f31aef9d700 1 -- 172.21.15.98:0/1118299943 >> 172.21.15.98:0/1118299943 conn(0x7f31a806b880 msgr2=0x7f31a80737a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:56.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.011+0000 7f31aef9d700 1 -- 172.21.15.98:0/1118299943 shutdown_connections 2023-03-03T07:56:56.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.011+0000 7f31aef9d700 1 -- 172.21.15.98:0/1118299943 wait complete. 2023-03-03T07:56:56.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:56:56.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.289+0000 7f058e79d700 1 Processor -- start 2023-03-03T07:56:56.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.290+0000 7f058e79d700 1 -- start start 2023-03-03T07:56:56.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.291+0000 7f058e79d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f05880cf570 0x7f058815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:56.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.291+0000 7f058e79d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05880cd780 con 0x7f058815db30 2023-03-03T07:56:56.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.291+0000 7f058e79d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05880cda60 con 0x7f05881609b0 2023-03-03T07:56:56.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.291+0000 7f058e79d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05880cebb0 con 0x7f05880cf570 2023-03-03T07:56:56.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.291+0000 7f058df9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f058815db30 0x7f05880b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34326/0 (socket says 172.21.15.98:34326) 2023-03-03T07:56:56.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.291+0000 7f058df9c700 1 -- 172.21.15.98:0/328363383 learned_addr learned my addr 172.21.15.98:0/328363383 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:56.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.291+0000 7f058cf9a700 1 --2- 172.21.15.98:0/328363383 >> v2:172.21.15.100:3300/0 conn(0x7f05880cf570 0x7f058815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:56.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.292+0000 7f057ffff700 1 -- 172.21.15.98:0/328363383 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809801628 0 0) 0x7f05880cd780 con 0x7f058815db30 2023-03-03T07:56:56.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.292+0000 7f057ffff700 1 -- 172.21.15.98:0/328363383 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05740035b0 con 0x7f058815db30 2023-03-03T07:56:56.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.292+0000 7f057ffff700 1 -- 172.21.15.98:0/328363383 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770784680 0 0) 0x7f05880cda60 con 0x7f05881609b0 2023-03-03T07:56:56.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.292+0000 7f058cf9a700 1 -- 172.21.15.98:0/328363383 >> v1:172.21.15.100:6789/0 conn(0x7f05881609b0 legacy=0x7f0588160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:56.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.292+0000 7f058cf9a700 1 -- 172.21.15.98:0/328363383 >> v1:172.21.15.98:6789/0 conn(0x7f058815db30 legacy=0x7f05880b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:56.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.292+0000 7f058cf9a700 1 -- 172.21.15.98:0/328363383 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05880cdd40 con 0x7f05880cf570 2023-03-03T07:56:56.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.292+0000 7f058cf9a700 1 --2- 172.21.15.98:0/328363383 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05880cf570 0x7f058815d5f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0578009840 tx=0x7f0578009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=844942873a3aa342 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:56.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.293+0000 7f057ffff700 1 -- 172.21.15.98:0/328363383 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0578011830 con 0x7f05880cf570 2023-03-03T07:56:56.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.293+0000 7f057ffff700 1 -- 172.21.15.98:0/328363383 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0578011990 con 0x7f05880cf570 2023-03-03T07:56:56.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.293+0000 7f057ffff700 1 -- 172.21.15.98:0/328363383 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0578011d20 con 0x7f05880cf570 2023-03-03T07:56:56.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.293+0000 7f058e79d700 1 -- 172.21.15.98:0/328363383 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05880cf570 msgr2=0x7f058815d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:56.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.293+0000 7f058e79d700 1 --2- 172.21.15.98:0/328363383 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05880cf570 0x7f058815d5f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0578009840 tx=0x7f0578009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:56.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.293+0000 7f058e79d700 1 -- 172.21.15.98:0/328363383 shutdown_connections 2023-03-03T07:56:56.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.293+0000 7f058e79d700 1 --2- 172.21.15.98:0/328363383 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05880cf570 0x7f058815d5f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.293+0000 7f058e79d700 1 -- 172.21.15.98:0/328363383 >> 172.21.15.98:0/328363383 conn(0x7f058806b880 msgr2=0x7f05880cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:56.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.294+0000 7f058e79d700 1 -- 172.21.15.98:0/328363383 shutdown_connections 2023-03-03T07:56:56.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.294+0000 7f058e79d700 1 -- 172.21.15.98:0/328363383 wait complete. 2023-03-03T07:56:56.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.296+0000 7f058e79d700 1 Processor -- start 2023-03-03T07:56:56.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.296+0000 7f058e79d700 1 -- start start 2023-03-03T07:56:56.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.297+0000 7f058e79d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05880cf570 0x7f058814ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:56.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.297+0000 7f058e79d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f058815db30 0x7f058815bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:56.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.297+0000 7f058e79d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05881609b0 0x7f05880c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:56.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.297+0000 7f058e79d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05880b78f0 con 0x7f05881609b0 2023-03-03T07:56:56.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.297+0000 7f058e79d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05880b7bd0 con 0x7f058815db30 2023-03-03T07:56:56.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.297+0000 7f058e79d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05880b7eb0 con 0x7f05880cf570 2023-03-03T07:56:56.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.297+0000 7f058df9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05881609b0 0x7f05880c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:56.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.298+0000 7f058df9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05881609b0 0x7f05880c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45654/0 (socket says 172.21.15.98:45654) 2023-03-03T07:56:56.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.298+0000 7f058cf9a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f058815db30 0x7f058815bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:56.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.298+0000 7f058df9c700 1 -- 172.21.15.98:0/4264681146 learned_addr learned my addr 172.21.15.98:0/4264681146 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:56.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.298+0000 7f058d79b700 1 --2- 172.21.15.98:0/4264681146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05880cf570 0x7f058814ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:56.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.298+0000 7f058df9c700 1 -- 172.21.15.98:0/4264681146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05880cf570 msgr2=0x7f058814ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:56.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.298+0000 7f058df9c700 1 --2- 172.21.15.98:0/4264681146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05880cf570 0x7f058814ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.298+0000 7f058df9c700 1 -- 172.21.15.98:0/4264681146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f058815db30 msgr2=0x7f058815bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:56.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.298+0000 7f058df9c700 1 --2- 172.21.15.98:0/4264681146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f058815db30 0x7f058815bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.298+0000 7f058df9c700 1 -- 172.21.15.98:0/4264681146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05780094a0 con 0x7f05881609b0 2023-03-03T07:56:56.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.298+0000 7f058df9c700 1 --2- 172.21.15.98:0/4264681146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05881609b0 0x7f05880c6980 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f058000cae0 tx=0x7f058000cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:56.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.299+0000 7f057dffb700 1 -- 172.21.15.98:0/4264681146 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058000a9e0 con 0x7f05881609b0 2023-03-03T07:56:56.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.299+0000 7f057dffb700 1 -- 172.21.15.98:0/4264681146 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058000ab40 con 0x7f05881609b0 2023-03-03T07:56:56.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.299+0000 7f058e79d700 1 -- 172.21.15.98:0/4264681146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058815d0d0 con 0x7f05881609b0 2023-03-03T07:56:56.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.299+0000 7f057dffb700 1 -- 172.21.15.98:0/4264681146 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580012400 con 0x7f05881609b0 2023-03-03T07:56:56.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.299+0000 7f058e79d700 1 -- 172.21.15.98:0/4264681146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058815cdf0 con 0x7f05881609b0 2023-03-03T07:56:56.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.300+0000 7f058e79d700 1 -- 172.21.15.98:0/4264681146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05880b51b0 con 0x7f05881609b0 2023-03-03T07:56:56.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.301+0000 7f057dffb700 1 -- 172.21.15.98:0/4264681146 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0580019030 con 0x7f05881609b0 2023-03-03T07:56:56.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.303+0000 7f057dffb700 1 --2- 172.21.15.98:0/4264681146 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f056c068320 0x7f056c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:56.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.303+0000 7f057dffb700 1 -- 172.21.15.98:0/4264681146 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f0580085860 con 0x7f05881609b0 2023-03-03T07:56:56.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.304+0000 7f058d79b700 1 --2- 172.21.15.98:0/4264681146 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f056c068320 0x7f056c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:56.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.304+0000 7f058d79b700 1 --2- 172.21.15.98:0/4264681146 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f056c068320 0x7f056c06a7d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0584000f60 tx=0x7f0584009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:56.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.307+0000 7f057dffb700 1 -- 172.21.15.98:0/4264681146 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0580051f30 con 0x7f05881609b0 2023-03-03T07:56:56.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.500+0000 7f058e79d700 1 -- 172.21.15.98:0/4264681146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f0588068290 con 0x7f05881609b0 2023-03-03T07:56:56.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.500+0000 7f057dffb700 1 -- 172.21.15.98:0/4264681146 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0580055550 con 0x7f05881609b0 2023-03-03T07:56:56.502 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:56:56.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.503+0000 7f058e79d700 1 -- 172.21.15.98:0/4264681146 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f056c068320 msgr2=0x7f056c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:56.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.503+0000 7f058e79d700 1 --2- 172.21.15.98:0/4264681146 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f056c068320 0x7f056c06a7d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0584000f60 tx=0x7f0584009040 comp rx=0 tx=0).stop 2023-03-03T07:56:56.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.503+0000 7f058e79d700 1 -- 172.21.15.98:0/4264681146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05881609b0 msgr2=0x7f05880c6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:56.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.503+0000 7f058e79d700 1 --2- 172.21.15.98:0/4264681146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05881609b0 0x7f05880c6980 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f058000cae0 tx=0x7f058000cdf0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.503+0000 7f058e79d700 1 -- 172.21.15.98:0/4264681146 shutdown_connections 2023-03-03T07:56:56.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.503+0000 7f058e79d700 1 --2- 172.21.15.98:0/4264681146 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f056c068320 0x7f056c06a7d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.503+0000 7f058e79d700 1 --2- 172.21.15.98:0/4264681146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05881609b0 0x7f05880c6980 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.503+0000 7f058e79d700 1 --2- 172.21.15.98:0/4264681146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05880cf570 0x7f058814ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.503+0000 7f058e79d700 1 --2- 172.21.15.98:0/4264681146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f058815db30 0x7f058815bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.503+0000 7f058e79d700 1 -- 172.21.15.98:0/4264681146 >> 172.21.15.98:0/4264681146 conn(0x7f058806b880 msgr2=0x7f058815a930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:56.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.503+0000 7f058e79d700 1 -- 172.21.15.98:0/4264681146 shutdown_connections 2023-03-03T07:56:56.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.503+0000 7f058e79d700 1 -- 172.21.15.98:0/4264681146 wait complete. 2023-03-03T07:56:56.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:56:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.783+0000 7fdd16b9d700 1 Processor -- start 2023-03-03T07:56:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.784+0000 7fdd16b9d700 1 -- start start 2023-03-03T07:56:56.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.785+0000 7fdd16b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd101609b0 0x7fdd10160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:56.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.785+0000 7fdd16b9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdd100b4aa0 0x7fdd100b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:56.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.785+0000 7fdd16b9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd101612c0 con 0x7fdd100cf570 2023-03-03T07:56:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.785+0000 7fdd16b9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd100cebb0 con 0x7fdd100b4aa0 2023-03-03T07:56:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.785+0000 7fdd16b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd100b5460 con 0x7fdd101609b0 2023-03-03T07:56:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.785+0000 7fdd1639c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdd100b4aa0 0x7fdd100b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.786+0000 7fdd1639c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdd100b4aa0 0x7fdd100b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45666/0 (socket says 172.21.15.98:45666) 2023-03-03T07:56:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.786+0000 7fdd1639c700 1 -- 172.21.15.98:0/3442455811 learned_addr learned my addr 172.21.15.98:0/3442455811 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.786+0000 7fdd15b9b700 1 --2- 172.21.15.98:0/3442455811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd101609b0 0x7fdd10160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.786+0000 7fdd1639c700 1 -- 172.21.15.98:0/3442455811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd101609b0 msgr2=0x7fdd10160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.786+0000 7fdd1639c700 1 --2- 172.21.15.98:0/3442455811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd101609b0 0x7fdd10160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.786+0000 7fdd1639c700 1 -- 172.21.15.98:0/3442455811 >> v1:172.21.15.100:6789/0 conn(0x7fdd100cf570 legacy=0x7fdd1015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.786+0000 7fdd1639c700 1 -- 172.21.15.98:0/3442455811 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1015dd00 con 0x7fdd100b4aa0 2023-03-03T07:56:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.786+0000 7fdd15b9b700 1 --2- 172.21.15.98:0/3442455811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd101609b0 0x7fdd10160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.786+0000 7fdd1639c700 1 --2- 172.21.15.98:0/3442455811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd100b4aa0 0x7fdd100b4f20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdd08009e50 tx=0x7fdd0800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a11be4f3da2e5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.787+0000 7fdd14b99700 1 -- 172.21.15.98:0/3442455811 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0800e070 con 0x7fdd100b4aa0 2023-03-03T07:56:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.787+0000 7fdd14b99700 1 -- 172.21.15.98:0/3442455811 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0800b9c0 con 0x7fdd100b4aa0 2023-03-03T07:56:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.787+0000 7fdd14b99700 1 -- 172.21.15.98:0/3442455811 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0800bd70 con 0x7fdd100b4aa0 2023-03-03T07:56:56.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.787+0000 7fdd16b9d700 1 -- 172.21.15.98:0/3442455811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd100b4aa0 msgr2=0x7fdd100b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:56.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.787+0000 7fdd16b9d700 1 --2- 172.21.15.98:0/3442455811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd100b4aa0 0x7fdd100b4f20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdd08009e50 tx=0x7fdd0800b330 comp rx=0 tx=0).stop 2023-03-03T07:56:56.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.787+0000 7fdd16b9d700 1 -- 172.21.15.98:0/3442455811 shutdown_connections 2023-03-03T07:56:56.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.787+0000 7fdd16b9d700 1 --2- 172.21.15.98:0/3442455811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd100b4aa0 0x7fdd100b4f20 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.787+0000 7fdd16b9d700 1 --2- 172.21.15.98:0/3442455811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd101609b0 0x7fdd10160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.787+0000 7fdd16b9d700 1 -- 172.21.15.98:0/3442455811 >> 172.21.15.98:0/3442455811 conn(0x7fdd1006b880 msgr2=0x7fdd100cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:56.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.788+0000 7fdd16b9d700 1 -- 172.21.15.98:0/3442455811 shutdown_connections 2023-03-03T07:56:56.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.788+0000 7fdd16b9d700 1 -- 172.21.15.98:0/3442455811 wait complete. 2023-03-03T07:56:56.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.790+0000 7fdd16b9d700 1 Processor -- start 2023-03-03T07:56:56.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.790+0000 7fdd16b9d700 1 -- start start 2023-03-03T07:56:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.791+0000 7fdd16b9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd100b4aa0 0x7fdd10151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.791+0000 7fdd16b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd100cf570 0x7fdd1015c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.791+0000 7fdd16b9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd101609b0 0x7fdd10150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.791+0000 7fdd16b9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd100cb860 con 0x7fdd100b4aa0 2023-03-03T07:56:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.791+0000 7fdd16b9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd100c9530 con 0x7fdd101609b0 2023-03-03T07:56:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.791+0000 7fdd16b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd100cb1a0 con 0x7fdd100cf570 2023-03-03T07:56:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.791+0000 7fdd15b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd100b4aa0 0x7fdd10151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.791+0000 7fdd15b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd100b4aa0 0x7fdd10151050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45674/0 (socket says 172.21.15.98:45674) 2023-03-03T07:56:56.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.791+0000 7fdd15b9b700 1 -- 172.21.15.98:0/1935228225 learned_addr learned my addr 172.21.15.98:0/1935228225 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:56.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.792+0000 7fdd1539a700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd100cf570 0x7fdd1015c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:56.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.792+0000 7fdd1639c700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd101609b0 0x7fdd10150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:56.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.792+0000 7fdd15b9b700 1 -- 172.21.15.98:0/1935228225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd100cf570 msgr2=0x7fdd1015c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:56.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.792+0000 7fdd15b9b700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd100cf570 0x7fdd1015c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.792+0000 7fdd15b9b700 1 -- 172.21.15.98:0/1935228225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd101609b0 msgr2=0x7fdd10150420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:56.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.792+0000 7fdd15b9b700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd101609b0 0x7fdd10150420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.792+0000 7fdd15b9b700 1 -- 172.21.15.98:0/1935228225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd08009b30 con 0x7fdd100b4aa0 2023-03-03T07:56:56.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.792+0000 7fdd1639c700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd101609b0 0x7fdd10150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:56:56.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.792+0000 7fdd1539a700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd100cf570 0x7fdd1015c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:56:56.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.792+0000 7fdd15b9b700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd100b4aa0 0x7fdd10151050 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00e810 tx=0x7fdd0c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:56.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.792+0000 7fdd067fc700 1 -- 172.21.15.98:0/1935228225 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c00c020 con 0x7fdd100b4aa0 2023-03-03T07:56:56.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.792+0000 7fdd067fc700 1 -- 172.21.15.98:0/1935228225 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0c004250 con 0x7fdd100b4aa0 2023-03-03T07:56:56.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.792+0000 7fdd16b9d700 1 -- 172.21.15.98:0/1935228225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd100cda60 con 0x7fdd100b4aa0 2023-03-03T07:56:56.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.792+0000 7fdd067fc700 1 -- 172.21.15.98:0/1935228225 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c0036a0 con 0x7fdd100b4aa0 2023-03-03T07:56:56.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.793+0000 7fdd16b9d700 1 -- 172.21.15.98:0/1935228225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd100cd780 con 0x7fdd100b4aa0 2023-03-03T07:56:56.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.794+0000 7fdd16b9d700 1 -- 172.21.15.98:0/1935228225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd100cdd40 con 0x7fdd100b4aa0 2023-03-03T07:56:56.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.795+0000 7fdd067fc700 1 -- 172.21.15.98:0/1935228225 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdd0c003800 con 0x7fdd100b4aa0 2023-03-03T07:56:56.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.796+0000 7fdd067fc700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdcf40682f0 0x7fdcf406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:56.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.796+0000 7fdd067fc700 1 -- 172.21.15.98:0/1935228225 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fdd0c085ad0 con 0x7fdd100b4aa0 2023-03-03T07:56:56.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.796+0000 7fdd1539a700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdcf40682f0 0x7fdcf406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:56.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.797+0000 7fdd1539a700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdcf40682f0 0x7fdcf406a7a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdd101518c0 tx=0x7fdd00009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:56.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.800+0000 7fdd067fc700 1 -- 172.21.15.98:0/1935228225 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdd0c052200 con 0x7fdd100b4aa0 2023-03-03T07:56:56.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.992+0000 7fdd16b9d700 1 -- 172.21.15.98:0/1935228225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdd10068290 con 0x7fdd100b4aa0 2023-03-03T07:56:56.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.993+0000 7fdd067fc700 1 -- 172.21.15.98:0/1935228225 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdd0c00f040 con 0x7fdd100b4aa0 2023-03-03T07:56:56.994 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:56:56.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.995+0000 7fdd16b9d700 1 -- 172.21.15.98:0/1935228225 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdcf40682f0 msgr2=0x7fdcf406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.995+0000 7fdd16b9d700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdcf40682f0 0x7fdcf406a7a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdd101518c0 tx=0x7fdd00009040 comp rx=0 tx=0).stop 2023-03-03T07:56:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.995+0000 7fdd16b9d700 1 -- 172.21.15.98:0/1935228225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd100b4aa0 msgr2=0x7fdd10151050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.995+0000 7fdd16b9d700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd100b4aa0 0x7fdd10151050 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00e810 tx=0x7fdd0c00ebd0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.995+0000 7fdd16b9d700 1 -- 172.21.15.98:0/1935228225 shutdown_connections 2023-03-03T07:56:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.995+0000 7fdd16b9d700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdcf40682f0 0x7fdcf406a7a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.995+0000 7fdd16b9d700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd100b4aa0 0x7fdd10151050 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.995+0000 7fdd16b9d700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd101609b0 0x7fdd10150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.995+0000 7fdd16b9d700 1 --2- 172.21.15.98:0/1935228225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd100cf570 0x7fdd1015c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.995+0000 7fdd16b9d700 1 -- 172.21.15.98:0/1935228225 >> 172.21.15.98:0/1935228225 conn(0x7fdd1006b880 msgr2=0x7fdd100d1040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.995+0000 7fdd16b9d700 1 -- 172.21.15.98:0/1935228225 shutdown_connections 2023-03-03T07:56:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:56.995+0000 7fdd16b9d700 1 -- 172.21.15.98:0/1935228225 wait complete. 2023-03-03T07:56:57.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:56:57.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.272+0000 7fe5c5f9d700 1 Processor -- start 2023-03-03T07:56:57.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.273+0000 7fe5c5f9d700 1 -- start start 2023-03-03T07:56:57.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.274+0000 7fe5c5f9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe5c00cf570 0x7fe5c015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:57.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.274+0000 7fe5c5f9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe5c015db30 0x7fe5c00b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:57.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.274+0000 7fe5c5f9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c00cd780 con 0x7fe5c01609b0 2023-03-03T07:56:57.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.274+0000 7fe5c5f9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c00cebb0 con 0x7fe5c00cf570 2023-03-03T07:56:57.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.274+0000 7fe5c5f9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c00cda60 con 0x7fe5c015db30 2023-03-03T07:56:57.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.274+0000 7fe5bffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe5c00cf570 0x7fe5c015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:57.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.275+0000 7fe5bffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe5c00cf570 0x7fe5c015d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45684/0 (socket says 172.21.15.98:45684) 2023-03-03T07:56:57.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.275+0000 7fe5bffff700 1 -- 172.21.15.98:0/458192264 learned_addr learned my addr 172.21.15.98:0/458192264 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:57.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.275+0000 7fe5c579c700 1 --2- 172.21.15.98:0/458192264 >> v2:172.21.15.100:3300/0 conn(0x7fe5c015db30 0x7fe5c00b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.275+0000 7fe5bffff700 1 -- 172.21.15.98:0/458192264 >> v2:172.21.15.100:3300/0 conn(0x7fe5c015db30 msgr2=0x7fe5c00b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.275+0000 7fe5bffff700 1 --2- 172.21.15.98:0/458192264 >> v2:172.21.15.100:3300/0 conn(0x7fe5c015db30 0x7fe5c00b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.275+0000 7fe5bffff700 1 -- 172.21.15.98:0/458192264 >> v1:172.21.15.100:6789/0 conn(0x7fe5c01609b0 legacy=0x7fe5c0160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.275+0000 7fe5bffff700 1 -- 172.21.15.98:0/458192264 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c00cdd40 con 0x7fe5c00cf570 2023-03-03T07:56:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.276+0000 7fe5bffff700 1 --2- 172.21.15.98:0/458192264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c00cf570 0x7fe5c015d5f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe5c00ca080 tx=0x7fe5ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77a768f14119ad4a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.276+0000 7fe5bf7fe700 1 -- 172.21.15.98:0/458192264 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac011830 con 0x7fe5c00cf570 2023-03-03T07:56:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.276+0000 7fe5bf7fe700 1 -- 172.21.15.98:0/458192264 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ac011990 con 0x7fe5c00cf570 2023-03-03T07:56:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.276+0000 7fe5bf7fe700 1 -- 172.21.15.98:0/458192264 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac011d20 con 0x7fe5c00cf570 2023-03-03T07:56:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.276+0000 7fe5c5f9d700 1 -- 172.21.15.98:0/458192264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c00cf570 msgr2=0x7fe5c015d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.276+0000 7fe5c5f9d700 1 --2- 172.21.15.98:0/458192264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c00cf570 0x7fe5c015d5f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe5c00ca080 tx=0x7fe5ac009bd0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.277+0000 7fe5c5f9d700 1 -- 172.21.15.98:0/458192264 shutdown_connections 2023-03-03T07:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.277+0000 7fe5c5f9d700 1 --2- 172.21.15.98:0/458192264 >> v2:172.21.15.100:3300/0 conn(0x7fe5c015db30 0x7fe5c00b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.277+0000 7fe5c5f9d700 1 --2- 172.21.15.98:0/458192264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c00cf570 0x7fe5c015d5f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.277+0000 7fe5c5f9d700 1 -- 172.21.15.98:0/458192264 >> 172.21.15.98:0/458192264 conn(0x7fe5c006b880 msgr2=0x7fe5c00cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.277+0000 7fe5c5f9d700 1 -- 172.21.15.98:0/458192264 shutdown_connections 2023-03-03T07:56:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.277+0000 7fe5c5f9d700 1 -- 172.21.15.98:0/458192264 wait complete. 2023-03-03T07:56:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.279+0000 7fe5c5f9d700 1 Processor -- start 2023-03-03T07:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.279+0000 7fe5c5f9d700 1 -- start start 2023-03-03T07:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.280+0000 7fe5c5f9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c00cf570 0x7fe5c01f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.280+0000 7fe5c5f9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe5c015db30 0x7fe5c00c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.280+0000 7fe5c5f9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe5c01609b0 0x7fe5c014f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.280+0000 7fe5c5f9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c00c7090 con 0x7fe5c00cf570 2023-03-03T07:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.280+0000 7fe5c5f9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c01f2fb0 con 0x7fe5c015db30 2023-03-03T07:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.280+0000 7fe5c5f9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c01f3290 con 0x7fe5c01609b0 2023-03-03T07:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.280+0000 7fe5c4f9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c00cf570 0x7fe5c01f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.280+0000 7fe5c4f9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c00cf570 0x7fe5c01f6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45700/0 (socket says 172.21.15.98:45700) 2023-03-03T07:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.280+0000 7fe5c4f9b700 1 -- 172.21.15.98:0/1478579466 learned_addr learned my addr 172.21.15.98:0/1478579466 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.280+0000 7fe5bffff700 1 --2- 172.21.15.98:0/1478579466 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe5c015db30 0x7fe5c00c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.280+0000 7fe5c579c700 1 --2- 172.21.15.98:0/1478579466 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe5c01609b0 0x7fe5c014f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.281+0000 7fe5c4f9b700 1 -- 172.21.15.98:0/1478579466 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe5c01609b0 msgr2=0x7fe5c014f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.281+0000 7fe5c4f9b700 1 --2- 172.21.15.98:0/1478579466 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe5c01609b0 0x7fe5c014f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.281+0000 7fe5c4f9b700 1 -- 172.21.15.98:0/1478579466 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe5c015db30 msgr2=0x7fe5c00c6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.281+0000 7fe5c4f9b700 1 --2- 172.21.15.98:0/1478579466 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe5c015db30 0x7fe5c00c6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.281+0000 7fe5c4f9b700 1 -- 172.21.15.98:0/1478579466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ac009550 con 0x7fe5c00cf570 2023-03-03T07:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.281+0000 7fe5c4f9b700 1 --2- 172.21.15.98:0/1478579466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c00cf570 0x7fe5c01f6d80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400c450 tx=0x7fe5b400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.281+0000 7fe5bd7fa700 1 -- 172.21.15.98:0/1478579466 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b40052d0 con 0x7fe5c00cf570 2023-03-03T07:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.281+0000 7fe5bd7fa700 1 -- 172.21.15.98:0/1478579466 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b4005430 con 0x7fe5c00cf570 2023-03-03T07:56:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.281+0000 7fe5bd7fa700 1 -- 172.21.15.98:0/1478579466 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b40038c0 con 0x7fe5c00cf570 2023-03-03T07:56:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.281+0000 7fe5c5f9d700 1 -- 172.21.15.98:0/1478579466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c015d0d0 con 0x7fe5c00cf570 2023-03-03T07:56:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.282+0000 7fe5c5f9d700 1 -- 172.21.15.98:0/1478579466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c015cdf0 con 0x7fe5c00cf570 2023-03-03T07:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.283+0000 7fe5c5f9d700 1 -- 172.21.15.98:0/1478579466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c00b4fe0 con 0x7fe5c00cf570 2023-03-03T07:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.284+0000 7fe5bd7fa700 1 -- 172.21.15.98:0/1478579466 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe5b4019030 con 0x7fe5c00cf570 2023-03-03T07:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.286+0000 7fe5bd7fa700 1 --2- 172.21.15.98:0/1478579466 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe5a40682f0 0x7fe5a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.286+0000 7fe5bd7fa700 1 -- 172.21.15.98:0/1478579466 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fe5b4084f60 con 0x7fe5c00cf570 2023-03-03T07:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.286+0000 7fe5bffff700 1 --2- 172.21.15.98:0/1478579466 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe5a40682f0 0x7fe5a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.287+0000 7fe5bffff700 1 --2- 172.21.15.98:0/1478579466 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe5a40682f0 0x7fe5a406a7a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac000c00 tx=0x7fe5ac008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.290+0000 7fe5bd7fa700 1 -- 172.21.15.98:0/1478579466 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe5b4051690 con 0x7fe5c00cf570 2023-03-03T07:56:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.484+0000 7fe5c5f9d700 1 -- 172.21.15.98:0/1478579466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe5c0068290 con 0x7fe5c00cf570 2023-03-03T07:56:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.484+0000 7fe5bd7fa700 1 -- 172.21.15.98:0/1478579466 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe5b40170a0 con 0x7fe5c00cf570 2023-03-03T07:56:57.486 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:56:57.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.487+0000 7fe5c5f9d700 1 -- 172.21.15.98:0/1478579466 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe5a40682f0 msgr2=0x7fe5a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:57.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.487+0000 7fe5c5f9d700 1 --2- 172.21.15.98:0/1478579466 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe5a40682f0 0x7fe5a406a7a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac000c00 tx=0x7fe5ac008030 comp rx=0 tx=0).stop 2023-03-03T07:56:57.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.487+0000 7fe5c5f9d700 1 -- 172.21.15.98:0/1478579466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c00cf570 msgr2=0x7fe5c01f6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:57.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.487+0000 7fe5c5f9d700 1 --2- 172.21.15.98:0/1478579466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c00cf570 0x7fe5c01f6d80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400c450 tx=0x7fe5b400c760 comp rx=0 tx=0).stop 2023-03-03T07:56:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.487+0000 7fe5c5f9d700 1 -- 172.21.15.98:0/1478579466 shutdown_connections 2023-03-03T07:56:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.487+0000 7fe5c5f9d700 1 --2- 172.21.15.98:0/1478579466 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe5a40682f0 0x7fe5a406a7a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.487+0000 7fe5c5f9d700 1 --2- 172.21.15.98:0/1478579466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c00cf570 0x7fe5c01f6d80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.487+0000 7fe5c5f9d700 1 --2- 172.21.15.98:0/1478579466 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe5c01609b0 0x7fe5c014f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.487+0000 7fe5c5f9d700 1 --2- 172.21.15.98:0/1478579466 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe5c015db30 0x7fe5c00c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.487+0000 7fe5c5f9d700 1 -- 172.21.15.98:0/1478579466 >> 172.21.15.98:0/1478579466 conn(0x7fe5c006b880 msgr2=0x7fe5c0152310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.487+0000 7fe5c5f9d700 1 -- 172.21.15.98:0/1478579466 shutdown_connections 2023-03-03T07:56:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.488+0000 7fe5c5f9d700 1 -- 172.21.15.98:0/1478579466 wait complete. 2023-03-03T07:56:57.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T07:56:57.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.763+0000 7f641b1ff700 1 Processor -- start 2023-03-03T07:56:57.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.763+0000 7f641b1ff700 1 -- start start 2023-03-03T07:56:57.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.765+0000 7f641b1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f641415ca70 0x7f64140cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:57.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.765+0000 7f641b1ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f641406b4b0 0x7f64140cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:57.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.765+0000 7f641b1ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64140cf980 con 0x7f64140d00a0 2023-03-03T07:56:57.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.765+0000 7f641b1ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64140cd5b0 con 0x7f641406b4b0 2023-03-03T07:56:57.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.765+0000 7f641b1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64140d0510 con 0x7f641415ca70 2023-03-03T07:56:57.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.765+0000 7f6413fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f641406b4b0 0x7f64140cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:57.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.765+0000 7f6413fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f641406b4b0 0x7f64140cfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45702/0 (socket says 172.21.15.98:45702) 2023-03-03T07:56:57.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.765+0000 7f6413fff700 1 -- 172.21.15.98:0/2882002248 learned_addr learned my addr 172.21.15.98:0/2882002248 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:57.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.765+0000 7f6418f9b700 1 --2- 172.21.15.98:0/2882002248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f641415ca70 0x7f64140cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:57.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.766+0000 7f6413fff700 1 -- 172.21.15.98:0/2882002248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f641415ca70 msgr2=0x7f64140cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:57.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.766+0000 7f6413fff700 1 --2- 172.21.15.98:0/2882002248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f641415ca70 0x7f64140cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.766+0000 7f6413fff700 1 -- 172.21.15.98:0/2882002248 >> v1:172.21.15.100:6789/0 conn(0x7f64140d00a0 legacy=0x7f64140b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:57.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.766+0000 7f6413fff700 1 -- 172.21.15.98:0/2882002248 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64140d2fd0 con 0x7f641406b4b0 2023-03-03T07:56:57.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.766+0000 7f6413fff700 1 --2- 172.21.15.98:0/2882002248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f641406b4b0 0x7f64140cfb60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f64000098f0 tx=0x7f6400009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20fbbde85b2f43f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:57.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.766+0000 7f64137fe700 1 -- 172.21.15.98:0/2882002248 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640000b8a0 con 0x7f641406b4b0 2023-03-03T07:56:57.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.766+0000 7f64137fe700 1 -- 172.21.15.98:0/2882002248 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640000ba00 con 0x7f641406b4b0 2023-03-03T07:56:57.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.766+0000 7f64137fe700 1 -- 172.21.15.98:0/2882002248 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640000bd90 con 0x7f641406b4b0 2023-03-03T07:56:57.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.767+0000 7f641b1ff700 1 -- 172.21.15.98:0/2882002248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f641406b4b0 msgr2=0x7f64140cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:57.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.767+0000 7f641b1ff700 1 --2- 172.21.15.98:0/2882002248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f641406b4b0 0x7f64140cfb60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f64000098f0 tx=0x7f6400009c00 comp rx=0 tx=0).stop 2023-03-03T07:56:57.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.767+0000 7f641b1ff700 1 -- 172.21.15.98:0/2882002248 shutdown_connections 2023-03-03T07:56:57.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.767+0000 7f641b1ff700 1 --2- 172.21.15.98:0/2882002248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f641415ca70 0x7f64140cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.767+0000 7f641b1ff700 1 --2- 172.21.15.98:0/2882002248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f641406b4b0 0x7f64140cfb60 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.767+0000 7f641b1ff700 1 -- 172.21.15.98:0/2882002248 >> 172.21.15.98:0/2882002248 conn(0x7f6414073e70 msgr2=0x7f641406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:57.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.767+0000 7f641b1ff700 1 -- 172.21.15.98:0/2882002248 shutdown_connections 2023-03-03T07:56:57.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.767+0000 7f641b1ff700 1 -- 172.21.15.98:0/2882002248 wait complete. 2023-03-03T07:56:57.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.769+0000 7f641b1ff700 1 Processor -- start 2023-03-03T07:56:57.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.769+0000 7f641b1ff700 1 -- start start 2023-03-03T07:56:57.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.770+0000 7f641b1ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f641406b4b0 0x7f64140c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:57.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.770+0000 7f641b1ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64140d00a0 0x7f64140a9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:57.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.770+0000 7f641b1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f641415ca70 0x7f64140b68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:57.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.770+0000 7f641b1ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64140c95e0 con 0x7f64140d00a0 2023-03-03T07:56:57.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.770+0000 7f641b1ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64140ccac0 con 0x7f641406b4b0 2023-03-03T07:56:57.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.770+0000 7f641b1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64140cc540 con 0x7f641415ca70 2023-03-03T07:56:57.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.771+0000 7f6413fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64140d00a0 0x7f64140a9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:57.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.771+0000 7f6413fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64140d00a0 0x7f64140a9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45718/0 (socket says 172.21.15.98:45718) 2023-03-03T07:56:57.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.771+0000 7f6413fff700 1 -- 172.21.15.98:0/4032142786 learned_addr learned my addr 172.21.15.98:0/4032142786 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:57.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.771+0000 7f6418f9b700 1 --2- 172.21.15.98:0/4032142786 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f641406b4b0 0x7f64140c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:57.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.771+0000 7f641979c700 1 --2- 172.21.15.98:0/4032142786 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f641415ca70 0x7f64140b68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:57.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.771+0000 7f6413fff700 1 -- 172.21.15.98:0/4032142786 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f641415ca70 msgr2=0x7f64140b68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:57.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.771+0000 7f6413fff700 1 --2- 172.21.15.98:0/4032142786 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f641415ca70 0x7f64140b68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.771+0000 7f6413fff700 1 -- 172.21.15.98:0/4032142786 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f641406b4b0 msgr2=0x7f64140c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:57.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.771+0000 7f6413fff700 1 --2- 172.21.15.98:0/4032142786 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f641406b4b0 0x7f64140c90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.771+0000 7f6413fff700 1 -- 172.21.15.98:0/4032142786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6400009550 con 0x7f64140d00a0 2023-03-03T07:56:57.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.772+0000 7f6413fff700 1 --2- 172.21.15.98:0/4032142786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64140d00a0 0x7f64140a9b30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6400000c00 tx=0x7f6400004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:57.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.772+0000 7f64117fa700 1 -- 172.21.15.98:0/4032142786 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6400004510 con 0x7f64140d00a0 2023-03-03T07:56:57.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.772+0000 7f64117fa700 1 -- 172.21.15.98:0/4032142786 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64000036a0 con 0x7f64140d00a0 2023-03-03T07:56:57.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.772+0000 7f64117fa700 1 -- 172.21.15.98:0/4032142786 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6400003a30 con 0x7f64140d00a0 2023-03-03T07:56:57.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.772+0000 7f641b1ff700 1 -- 172.21.15.98:0/4032142786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64140cd900 con 0x7f64140d00a0 2023-03-03T07:56:57.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.772+0000 7f641b1ff700 1 -- 172.21.15.98:0/4032142786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64140d1800 con 0x7f64140d00a0 2023-03-03T07:56:57.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.773+0000 7f641b1ff700 1 -- 172.21.15.98:0/4032142786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64140cdbe0 con 0x7f64140d00a0 2023-03-03T07:56:57.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.774+0000 7f64117fa700 1 -- 172.21.15.98:0/4032142786 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f640000f050 con 0x7f64140d00a0 2023-03-03T07:56:57.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.776+0000 7f64117fa700 1 --2- 172.21.15.98:0/4032142786 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63f80682f0 0x7f63f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.776+0000 7f64117fa700 1 -- 172.21.15.98:0/4032142786 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f64000873a0 con 0x7f64140d00a0 2023-03-03T07:56:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.777+0000 7f6418f9b700 1 --2- 172.21.15.98:0/4032142786 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63f80682f0 0x7f63f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.777+0000 7f6418f9b700 1 --2- 172.21.15.98:0/4032142786 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63f80682f0 0x7f63f806a7a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6408006b20 tx=0x7f6408008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:57.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.781+0000 7f64117fa700 1 -- 172.21.15.98:0/4032142786 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6400053a50 con 0x7f64140d00a0 2023-03-03T07:56:57.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.988+0000 7f641b1ff700 1 -- 172.21.15.98:0/4032142786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f64140ce4c0 con 0x7f64140d00a0 2023-03-03T07:56:57.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.988+0000 7f64117fa700 1 -- 172.21.15.98:0/4032142786 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6400057070 con 0x7f64140d00a0 2023-03-03T07:56:57.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.991+0000 7f641b1ff700 1 -- 172.21.15.98:0/4032142786 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63f80682f0 msgr2=0x7f63f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:57.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.991+0000 7f641b1ff700 1 --2- 172.21.15.98:0/4032142786 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63f80682f0 0x7f63f806a7a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6408006b20 tx=0x7f6408008040 comp rx=0 tx=0).stop 2023-03-03T07:56:57.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.991+0000 7f641b1ff700 1 -- 172.21.15.98:0/4032142786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64140d00a0 msgr2=0x7f64140a9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:57.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.991+0000 7f641b1ff700 1 --2- 172.21.15.98:0/4032142786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64140d00a0 0x7f64140a9b30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6400000c00 tx=0x7f6400004750 comp rx=0 tx=0).stop 2023-03-03T07:56:57.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.991+0000 7f641b1ff700 1 -- 172.21.15.98:0/4032142786 shutdown_connections 2023-03-03T07:56:57.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.991+0000 7f641b1ff700 1 --2- 172.21.15.98:0/4032142786 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63f80682f0 0x7f63f806a7a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.991+0000 7f641b1ff700 1 --2- 172.21.15.98:0/4032142786 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f641415ca70 0x7f64140b68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.991+0000 7f641b1ff700 1 --2- 172.21.15.98:0/4032142786 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f641406b4b0 0x7f64140c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.991+0000 7f641b1ff700 1 --2- 172.21.15.98:0/4032142786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64140d00a0 0x7f64140a9b30 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:57.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.991+0000 7f641b1ff700 1 -- 172.21.15.98:0/4032142786 >> 172.21.15.98:0/4032142786 conn(0x7f6414073e70 msgr2=0x7f641406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:57.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.991+0000 7f641b1ff700 1 -- 172.21.15.98:0/4032142786 shutdown_connections 2023-03-03T07:56:57.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:57.991+0000 7f641b1ff700 1 -- 172.21.15.98:0/4032142786 wait complete. 2023-03-03T07:56:57.994 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T07:56:58.020 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2023-03-03T07:56:58.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:58.021 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2023-03-03T07:56:58.128 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T07:56:58.128 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T07:56:58.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:56:58.129 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-03-03T07:56:58.198 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-03-03T07:56:58.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:56:58.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.442+0000 7f8dcb39d700 1 Processor -- start 2023-03-03T07:56:58.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.442+0000 7f8dcb39d700 1 -- start start 2023-03-03T07:56:58.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.444+0000 7f8dcb39d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dc4146360 0x7f8dc4160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:58.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.444+0000 7f8dcb39d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8dc40b4ab0 0x7f8dc40b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:58.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.444+0000 7f8dcb39d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc41614c0 con 0x7f8dc40731e0 2023-03-03T07:56:58.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.444+0000 7f8dcb39d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc40b5460 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.444+0000 7f8dcb39d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc40d10f0 con 0x7f8dc4146360 2023-03-03T07:56:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.444+0000 7f8dca39b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dc4146360 0x7f8dc4160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.444+0000 7f8dcab9c700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f8dc40731e0 0x7f8dc4073660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:60216/0 (socket says 172.21.15.98:60216) 2023-03-03T07:56:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.445+0000 7f8dcab9c700 1 -- 172.21.15.98:0/264157622 learned_addr learned my addr 172.21.15.98:0/264157622 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.445+0000 7f8dc9b9a700 1 --2- 172.21.15.98:0/264157622 >> v2:172.21.15.100:3300/0 conn(0x7f8dc40b4ab0 0x7f8dc40b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.445+0000 7f8dc9b9a700 1 -- 172.21.15.98:0/264157622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dc4146360 msgr2=0x7f8dc4160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.445+0000 7f8dc9b9a700 1 --2- 172.21.15.98:0/264157622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dc4146360 0x7f8dc4160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.445+0000 7f8dc9b9a700 1 -- 172.21.15.98:0/264157622 >> v1:172.21.15.100:6789/0 conn(0x7f8dc40731e0 legacy=0x7f8dc4073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.445+0000 7f8dc9b9a700 1 -- 172.21.15.98:0/264157622 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc40cdf20 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.445+0000 7f8dc9399700 1 -- 172.21.15.98:0/264157622 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119939444 0 0) 0x7f8dbc001490 con 0x7f8dc40731e0 2023-03-03T07:56:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.445+0000 7f8dca39b700 1 --2- 172.21.15.98:0/264157622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dc4146360 0x7f8dc4160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:56:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.445+0000 7f8dc9b9a700 1 --2- 172.21.15.98:0/264157622 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dc40b4ab0 0x7f8dc40b4f20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009840 tx=0x7f8db4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15cb717067dbb407 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.446+0000 7f8dc9399700 1 -- 172.21.15.98:0/264157622 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db4011830 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.446+0000 7f8dc9399700 1 -- 172.21.15.98:0/264157622 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db4011990 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.446+0000 7f8dc9399700 1 -- 172.21.15.98:0/264157622 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db4011d20 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.446+0000 7f8dcb39d700 1 -- 172.21.15.98:0/264157622 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dc40b4ab0 msgr2=0x7f8dc40b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:58.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.446+0000 7f8dcb39d700 1 --2- 172.21.15.98:0/264157622 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dc40b4ab0 0x7f8dc40b4f20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009840 tx=0x7f8db4009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:58.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.446+0000 7f8dcb39d700 1 -- 172.21.15.98:0/264157622 shutdown_connections 2023-03-03T07:56:58.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.446+0000 7f8dcb39d700 1 --2- 172.21.15.98:0/264157622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dc4146360 0x7f8dc4160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:58.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.446+0000 7f8dcb39d700 1 --2- 172.21.15.98:0/264157622 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dc40b4ab0 0x7f8dc40b4f20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:58.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.446+0000 7f8dcb39d700 1 -- 172.21.15.98:0/264157622 >> 172.21.15.98:0/264157622 conn(0x7f8dc406b8e0 msgr2=0x7f8dc40cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:58.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.446+0000 7f8dcb39d700 1 -- 172.21.15.98:0/264157622 shutdown_connections 2023-03-03T07:56:58.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.446+0000 7f8dcb39d700 1 -- 172.21.15.98:0/264157622 wait complete. 2023-03-03T07:56:58.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.448+0000 7f8dcb39d700 1 Processor -- start 2023-03-03T07:56:58.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.449+0000 7f8dcb39d700 1 -- start start 2023-03-03T07:56:58.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.450+0000 7f8dcb39d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dc40731e0 0x7f8dc40c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:58.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.450+0000 7f8dcb39d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dc40b4ab0 0x7f8dc4159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:58.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.450+0000 7f8dcb39d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dc4146360 0x7f8dc415a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:58.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.450+0000 7f8dcb39d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc40c7bb0 con 0x7f8dc4146360 2023-03-03T07:56:58.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.450+0000 7f8dcb39d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc40ca230 con 0x7f8dc40731e0 2023-03-03T07:56:58.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.450+0000 7f8dcb39d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc40c78d0 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.450+0000 7f8dc9b9a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dc40b4ab0 0x7f8dc4159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:58.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.450+0000 7f8dca39b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dc40731e0 0x7f8dc40c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:58.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.450+0000 7f8dcab9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dc4146360 0x7f8dc415a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:58.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.450+0000 7f8dc9b9a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dc40b4ab0 0x7f8dc4159f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:59896/0 (socket says 172.21.15.98:59896) 2023-03-03T07:56:58.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.450+0000 7f8dca39b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dc40731e0 0x7f8dc40c6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:34778/0 (socket says 172.21.15.98:34778) 2023-03-03T07:56:58.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.450+0000 7f8dc9b9a700 1 -- 172.21.15.98:0/700033451 learned_addr learned my addr 172.21.15.98:0/700033451 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:58.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.451+0000 7f8dc9b9a700 1 -- 172.21.15.98:0/700033451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dc40731e0 msgr2=0x7f8dc40c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.451+0000 7f8dc9b9a700 1 --2- 172.21.15.98:0/700033451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dc40731e0 0x7f8dc40c6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.451+0000 7f8dc9b9a700 1 -- 172.21.15.98:0/700033451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dc4146360 msgr2=0x7f8dc415a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.451+0000 7f8dc9b9a700 1 --2- 172.21.15.98:0/700033451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dc4146360 0x7f8dc415a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.451+0000 7f8dc9b9a700 1 -- 172.21.15.98:0/700033451 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc00094a0 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.451+0000 7f8dc9b9a700 1 --2- 172.21.15.98:0/700033451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dc40b4ab0 0x7f8dc4159f00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009b90 tx=0x7f8db4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.451+0000 7f8dbaffd700 1 -- 172.21.15.98:0/700033451 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db40036a0 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.451+0000 7f8dbaffd700 1 -- 172.21.15.98:0/700033451 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db401e070 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.451+0000 7f8dbaffd700 1 -- 172.21.15.98:0/700033451 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db4003a60 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.451+0000 7f8dcb39d700 1 -- 172.21.15.98:0/700033451 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db40094a0 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.452+0000 7f8dcb39d700 1 -- 172.21.15.98:0/700033451 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc40cd960 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.453+0000 7f8dcb39d700 1 -- 172.21.15.98:0/700033451 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc40cd680 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.454+0000 7f8dbaffd700 1 -- 172.21.15.98:0/700033451 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8db400f050 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.455+0000 7f8dbaffd700 1 --2- 172.21.15.98:0/700033451 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8da80682f0 0x7f8da806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.455+0000 7f8dbaffd700 1 -- 172.21.15.98:0/700033451 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f8db4086c30 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.455+0000 7f8dca39b700 1 --2- 172.21.15.98:0/700033451 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8da80682f0 0x7f8da806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:58.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.456+0000 7f8dca39b700 1 --2- 172.21.15.98:0/700033451 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8da80682f0 0x7f8da806a7a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009cd0 tx=0x7f8dc0009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:58.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.461+0000 7f8dbaffd700 1 -- 172.21.15.98:0/700033451 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8db4053360 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.701+0000 7f8dcb39d700 1 -- 172.21.15.98:0/700033451 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dc40cdc40 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.702+0000 7f8dbaffd700 1 -- 172.21.15.98:0/700033451 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4639 (secure 0 0 0) 0x7f8db4020090 con 0x7f8dc40b4ab0 2023-03-03T07:56:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:56:58.705 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/246835374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":246835374},{"type":"v1","addr":"172.21.15.98:6836","nonce":246835374}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6833/959944231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":959944231},{"type":"v1","addr":"172.21.15.100:6833","nonce":959944231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/872788798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":872788798},{"type":"v1","addr":"172.21.15.100:6835","nonce":872788798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:55:47.097221+0000","modified":"2023-03-03T07:55:50.555384+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/3969084152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3969084152},{"type":"v1","addr":"172.21.15.98:6837","nonce":3969084152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:56:58.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.705+0000 7f8dcb39d700 1 -- 172.21.15.98:0/700033451 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8da80682f0 msgr2=0x7f8da806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:58.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.705+0000 7f8dcb39d700 1 --2- 172.21.15.98:0/700033451 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8da80682f0 0x7f8da806a7a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009cd0 tx=0x7f8dc0009210 comp rx=0 tx=0).stop 2023-03-03T07:56:58.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.705+0000 7f8dcb39d700 1 -- 172.21.15.98:0/700033451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dc40b4ab0 msgr2=0x7f8dc4159f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:58.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.705+0000 7f8dcb39d700 1 --2- 172.21.15.98:0/700033451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dc40b4ab0 0x7f8dc4159f00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009b90 tx=0x7f8db4004750 comp rx=0 tx=0).stop 2023-03-03T07:56:58.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.705+0000 7f8dcb39d700 1 -- 172.21.15.98:0/700033451 shutdown_connections 2023-03-03T07:56:58.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.705+0000 7f8dcb39d700 1 --2- 172.21.15.98:0/700033451 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8da80682f0 0x7f8da806a7a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:58.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.705+0000 7f8dcb39d700 1 --2- 172.21.15.98:0/700033451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dc4146360 0x7f8dc415a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:58.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.705+0000 7f8dcb39d700 1 --2- 172.21.15.98:0/700033451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dc40731e0 0x7f8dc40c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:58.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.705+0000 7f8dcb39d700 1 --2- 172.21.15.98:0/700033451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dc40b4ab0 0x7f8dc4159f00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:58.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.705+0000 7f8dcb39d700 1 -- 172.21.15.98:0/700033451 >> 172.21.15.98:0/700033451 conn(0x7f8dc406b8e0 msgr2=0x7f8dc40b3770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:58.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.705+0000 7f8dcb39d700 1 -- 172.21.15.98:0/700033451 shutdown_connections 2023-03-03T07:56:58.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.705+0000 7f8dcb39d700 1 -- 172.21.15.98:0/700033451 wait complete. 2023-03-03T07:56:58.709 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 7 2023-03-03T07:56:58.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:56:58.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.976+0000 7f8f88dff700 1 Processor -- start 2023-03-03T07:56:58.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.976+0000 7f8f88dff700 1 -- start start 2023-03-03T07:56:58.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.977+0000 7f8f88dff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8f840cd8d0 0x7f8f840b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:58.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.977+0000 7f8f88dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8f840b6610 0x7f8f840cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:58.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.977+0000 7f8f88dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f840cdd70 con 0x7f8f8415cc00 2023-03-03T07:56:58.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.977+0000 7f8f88dff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f840ce040 con 0x7f8f840cd8d0 2023-03-03T07:56:58.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.977+0000 7f8f88dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f84073be0 con 0x7f8f840b6610 2023-03-03T07:56:58.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.978+0000 7f8f81d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8f840cd8d0 0x7f8f840b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:58.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.978+0000 7f8f8259c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8f8415cc00 0x7f8f840d27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34340/0 (socket says 172.21.15.98:34340) 2023-03-03T07:56:58.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.978+0000 7f8f8259c700 1 -- 172.21.15.98:0/49139104 learned_addr learned my addr 172.21.15.98:0/49139104 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:58.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.978+0000 7f8f82d9d700 1 --2- 172.21.15.98:0/49139104 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8f840b6610 0x7f8f840cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:58.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.978+0000 7f8f81d9b700 1 -- 172.21.15.98:0/49139104 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8f840b6610 msgr2=0x7f8f840cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:58.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.978+0000 7f8f81d9b700 1 --2- 172.21.15.98:0/49139104 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8f840b6610 0x7f8f840cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:58.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.978+0000 7f8f81d9b700 1 -- 172.21.15.98:0/49139104 >> v1:172.21.15.98:6789/0 conn(0x7f8f8415cc00 legacy=0x7f8f840d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:58.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.978+0000 7f8f81d9b700 1 -- 172.21.15.98:0/49139104 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f840d1a00 con 0x7f8f840cd8d0 2023-03-03T07:56:58.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.978+0000 7f8f8159a700 1 -- 172.21.15.98:0/49139104 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703027459 0 0) 0x7f8f840cdd70 con 0x7f8f8415cc00 2023-03-03T07:56:58.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.979+0000 7f8f81d9b700 1 --2- 172.21.15.98:0/49139104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f840cd8d0 0x7f8f840b60d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009840 tx=0x7f8f78009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f107a0d0970067 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:58.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.979+0000 7f8f8159a700 1 -- 172.21.15.98:0/49139104 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f78011830 con 0x7f8f840cd8d0 2023-03-03T07:56:58.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.979+0000 7f8f8159a700 1 -- 172.21.15.98:0/49139104 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f78011990 con 0x7f8f840cd8d0 2023-03-03T07:56:58.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.979+0000 7f8f8159a700 1 -- 172.21.15.98:0/49139104 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f78011d20 con 0x7f8f840cd8d0 2023-03-03T07:56:58.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.979+0000 7f8f88dff700 1 -- 172.21.15.98:0/49139104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f840cd8d0 msgr2=0x7f8f840b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:58.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.979+0000 7f8f88dff700 1 --2- 172.21.15.98:0/49139104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f840cd8d0 0x7f8f840b60d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009840 tx=0x7f8f78009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:58.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.980+0000 7f8f88dff700 1 -- 172.21.15.98:0/49139104 shutdown_connections 2023-03-03T07:56:58.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.980+0000 7f8f88dff700 1 --2- 172.21.15.98:0/49139104 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8f840b6610 0x7f8f840cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:58.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.980+0000 7f8f88dff700 1 --2- 172.21.15.98:0/49139104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f840cd8d0 0x7f8f840b60d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:58.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.980+0000 7f8f88dff700 1 -- 172.21.15.98:0/49139104 >> 172.21.15.98:0/49139104 conn(0x7f8f8406aee0 msgr2=0x7f8f8406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:58.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.980+0000 7f8f88dff700 1 -- 172.21.15.98:0/49139104 shutdown_connections 2023-03-03T07:56:58.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.980+0000 7f8f88dff700 1 -- 172.21.15.98:0/49139104 wait complete. 2023-03-03T07:56:58.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.982+0000 7f8f88dff700 1 Processor -- start 2023-03-03T07:56:58.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.982+0000 7f8f88dff700 1 -- start start 2023-03-03T07:56:58.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.983+0000 7f8f88dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f840b6610 0x7f8f841549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:58.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.983+0000 7f8f88dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8f840cd8d0 0x7f8f840b36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:58.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.983+0000 7f8f88dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8f8415cc00 0x7f8f84150680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:58.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.983+0000 7f8f88dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8415d340 con 0x7f8f840b6610 2023-03-03T07:56:58.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.983+0000 7f8f88dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8415d620 con 0x7f8f840cd8d0 2023-03-03T07:56:58.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.983+0000 7f8f88dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8415d900 con 0x7f8f8415cc00 2023-03-03T07:56:58.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.983+0000 7f8f8259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f840b6610 0x7f8f841549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:58.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.983+0000 7f8f8259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f840b6610 0x7f8f841549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45750/0 (socket says 172.21.15.98:45750) 2023-03-03T07:56:58.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.983+0000 7f8f8259c700 1 -- 172.21.15.98:0/4244852148 learned_addr learned my addr 172.21.15.98:0/4244852148 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:58.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.984+0000 7f8f8259c700 1 -- 172.21.15.98:0/4244852148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8f8415cc00 msgr2=0x7f8f84150680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:58.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.984+0000 7f8f8259c700 1 --2- 172.21.15.98:0/4244852148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8f8415cc00 0x7f8f84150680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:58.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.984+0000 7f8f8259c700 1 -- 172.21.15.98:0/4244852148 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8f840cd8d0 msgr2=0x7f8f840b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:58.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.984+0000 7f8f8259c700 1 --2- 172.21.15.98:0/4244852148 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8f840cd8d0 0x7f8f840b36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:58.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.984+0000 7f8f8259c700 1 -- 172.21.15.98:0/4244852148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f780094a0 con 0x7f8f840b6610 2023-03-03T07:56:58.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.984+0000 7f8f8259c700 1 --2- 172.21.15.98:0/4244852148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f840b6610 0x7f8f841549b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8f740011e0 tx=0x7f8f7400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:58.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.984+0000 7f8f72ffd700 1 -- 172.21.15.98:0/4244852148 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74004c80 con 0x7f8f840b6610 2023-03-03T07:56:58.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.984+0000 7f8f72ffd700 1 -- 172.21.15.98:0/4244852148 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f74004e00 con 0x7f8f840b6610 2023-03-03T07:56:58.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.984+0000 7f8f72ffd700 1 -- 172.21.15.98:0/4244852148 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74007310 con 0x7f8f840b6610 2023-03-03T07:56:58.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.984+0000 7f8f88dff700 1 -- 172.21.15.98:0/4244852148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f840d1fc0 con 0x7f8f840b6610 2023-03-03T07:56:58.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.984+0000 7f8f88dff700 1 -- 172.21.15.98:0/4244852148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f840d1ce0 con 0x7f8f840b6610 2023-03-03T07:56:58.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.986+0000 7f8f88dff700 1 -- 172.21.15.98:0/4244852148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f840d22a0 con 0x7f8f840b6610 2023-03-03T07:56:58.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.987+0000 7f8f72ffd700 1 -- 172.21.15.98:0/4244852148 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8f7401a020 con 0x7f8f840b6610 2023-03-03T07:56:58.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.989+0000 7f8f72ffd700 1 --2- 172.21.15.98:0/4244852148 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8f640682f0 0x7f8f6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:58.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.989+0000 7f8f72ffd700 1 -- 172.21.15.98:0/4244852148 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f8f740859a0 con 0x7f8f840b6610 2023-03-03T07:56:58.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.989+0000 7f8f81d9b700 1 --2- 172.21.15.98:0/4244852148 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8f640682f0 0x7f8f6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:58.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.990+0000 7f8f81d9b700 1 --2- 172.21.15.98:0/4244852148 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8f640682f0 0x7f8f6406a7a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8f84159370 tx=0x7f8f78010000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:58.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:58.993+0000 7f8f72ffd700 1 -- 172.21.15.98:0/4244852148 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8f74052180 con 0x7f8f840b6610 2023-03-03T07:56:59.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.194+0000 7f8f88dff700 1 -- 172.21.15.98:0/4244852148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f840729b0 con 0x7f8f840b6610 2023-03-03T07:56:59.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.196+0000 7f8f72ffd700 1 -- 172.21.15.98:0/4244852148 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18003 (secure 0 0 0) 0x7f8f740557a0 con 0x7f8f840b6610 2023-03-03T07:56:59.197 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:56:59.197 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:56:27.498205+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:56:59.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.198+0000 7f8f88dff700 1 -- 172.21.15.98:0/4244852148 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8f640682f0 msgr2=0x7f8f6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:59.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.198+0000 7f8f88dff700 1 --2- 172.21.15.98:0/4244852148 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8f640682f0 0x7f8f6406a7a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8f84159370 tx=0x7f8f78010000 comp rx=0 tx=0).stop 2023-03-03T07:56:59.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.198+0000 7f8f88dff700 1 -- 172.21.15.98:0/4244852148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f840b6610 msgr2=0x7f8f841549b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:59.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.198+0000 7f8f88dff700 1 --2- 172.21.15.98:0/4244852148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f840b6610 0x7f8f841549b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8f740011e0 tx=0x7f8f7400cae0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.198+0000 7f8f88dff700 1 -- 172.21.15.98:0/4244852148 shutdown_connections 2023-03-03T07:56:59.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.198+0000 7f8f88dff700 1 --2- 172.21.15.98:0/4244852148 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8f640682f0 0x7f8f6406a7a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.198+0000 7f8f88dff700 1 --2- 172.21.15.98:0/4244852148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f840b6610 0x7f8f841549b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.198+0000 7f8f88dff700 1 --2- 172.21.15.98:0/4244852148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8f8415cc00 0x7f8f84150680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.198+0000 7f8f88dff700 1 --2- 172.21.15.98:0/4244852148 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8f840cd8d0 0x7f8f840b36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.198+0000 7f8f88dff700 1 -- 172.21.15.98:0/4244852148 >> 172.21.15.98:0/4244852148 conn(0x7f8f8406aee0 msgr2=0x7f8f840b6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:59.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.198+0000 7f8f88dff700 1 -- 172.21.15.98:0/4244852148 shutdown_connections 2023-03-03T07:56:59.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.198+0000 7f8f88dff700 1 -- 172.21.15.98:0/4244852148 wait complete. 2023-03-03T07:56:59.227 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:55:45.157666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:55:46.092091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:56:59.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:56:59.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.475+0000 7fc8bd5ff700 1 Processor -- start 2023-03-03T07:56:59.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.476+0000 7fc8bd5ff700 1 -- start start 2023-03-03T07:56:59.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.477+0000 7fc8bd5ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc8b80d0270 0x7fc8b80cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:59.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.477+0000 7fc8bd5ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b80cf980 con 0x7fc8b806b4b0 2023-03-03T07:56:59.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.477+0000 7fc8bd5ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b80d1800 con 0x7fc8b815ca70 2023-03-03T07:56:59.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.477+0000 7fc8bd5ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b80cd5b0 con 0x7fc8b80d0270 2023-03-03T07:56:59.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.477+0000 7fc8b77fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc8b806b4b0 0x7fc8b80d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34352/0 (socket says 172.21.15.98:34352) 2023-03-03T07:56:59.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.477+0000 7fc8b77fe700 1 -- 172.21.15.98:0/2317071249 learned_addr learned my addr 172.21.15.98:0/2317071249 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:56:59.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.477+0000 7fc8b67fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc8b80d0270 0x7fc8b80cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:59.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.478+0000 7fc8b67fc700 1 -- 172.21.15.98:0/2317071249 >> v1:172.21.15.100:6789/0 conn(0x7fc8b815ca70 legacy=0x7fc8b80cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.478+0000 7fc8b67fc700 1 -- 172.21.15.98:0/2317071249 >> v1:172.21.15.98:6789/0 conn(0x7fc8b806b4b0 legacy=0x7fc8b80d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.478+0000 7fc8b67fc700 1 -- 172.21.15.98:0/2317071249 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b80cd900 con 0x7fc8b80d0270 2023-03-03T07:56:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.478+0000 7fc8b67fc700 1 --2- 172.21.15.98:0/2317071249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b80d0270 0x7fc8b80cf440 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009840 tx=0x7fc8ac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b450b9ff70da82 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.479+0000 7fc8b5ffb700 1 -- 172.21.15.98:0/2317071249 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac011830 con 0x7fc8b80d0270 2023-03-03T07:56:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.479+0000 7fc8b5ffb700 1 -- 172.21.15.98:0/2317071249 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ac011990 con 0x7fc8b80d0270 2023-03-03T07:56:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.479+0000 7fc8b5ffb700 1 -- 172.21.15.98:0/2317071249 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac011d20 con 0x7fc8b80d0270 2023-03-03T07:56:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.479+0000 7fc8bd5ff700 1 -- 172.21.15.98:0/2317071249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b80d0270 msgr2=0x7fc8b80cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.479+0000 7fc8bd5ff700 1 --2- 172.21.15.98:0/2317071249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b80d0270 0x7fc8b80cf440 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009840 tx=0x7fc8ac009b50 comp rx=0 tx=0).stop 2023-03-03T07:56:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.479+0000 7fc8bd5ff700 1 -- 172.21.15.98:0/2317071249 shutdown_connections 2023-03-03T07:56:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.479+0000 7fc8bd5ff700 1 --2- 172.21.15.98:0/2317071249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b80d0270 0x7fc8b80cf440 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.479+0000 7fc8bd5ff700 1 -- 172.21.15.98:0/2317071249 >> 172.21.15.98:0/2317071249 conn(0x7fc8b8073e70 msgr2=0x7fc8b806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.479+0000 7fc8bd5ff700 1 -- 172.21.15.98:0/2317071249 shutdown_connections 2023-03-03T07:56:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.479+0000 7fc8bd5ff700 1 -- 172.21.15.98:0/2317071249 wait complete. 2023-03-03T07:56:59.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.481+0000 7fc8bd5ff700 1 Processor -- start 2023-03-03T07:56:59.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.482+0000 7fc8bd5ff700 1 -- start start 2023-03-03T07:56:59.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.482+0000 7fc8bd5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8b806b4b0 0x7fc8b81535c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:59.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.483+0000 7fc8bd5ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b80d0270 0x7fc8b815f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:59.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.483+0000 7fc8bd5ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b815ca70 0x7fc8b80c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:59.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.483+0000 7fc8bd5ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b80cb520 con 0x7fc8b815ca70 2023-03-03T07:56:59.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.483+0000 7fc8bd5ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b80cb800 con 0x7fc8b80d0270 2023-03-03T07:56:59.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.483+0000 7fc8bd5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b80cbae0 con 0x7fc8b806b4b0 2023-03-03T07:56:59.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.483+0000 7fc8b77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b815ca70 0x7fc8b80c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:59.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.483+0000 7fc8b77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b815ca70 0x7fc8b80c6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45776/0 (socket says 172.21.15.98:45776) 2023-03-03T07:56:59.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.483+0000 7fc8b77fe700 1 -- 172.21.15.98:0/2396293535 learned_addr learned my addr 172.21.15.98:0/2396293535 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:59.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.483+0000 7fc8b67fc700 1 --2- 172.21.15.98:0/2396293535 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b80d0270 0x7fc8b815f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:59.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.483+0000 7fc8b6ffd700 1 --2- 172.21.15.98:0/2396293535 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8b806b4b0 0x7fc8b81535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:59.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.483+0000 7fc8b77fe700 1 -- 172.21.15.98:0/2396293535 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8b806b4b0 msgr2=0x7fc8b81535c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:59.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.484+0000 7fc8b77fe700 1 --2- 172.21.15.98:0/2396293535 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8b806b4b0 0x7fc8b81535c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.484+0000 7fc8b77fe700 1 -- 172.21.15.98:0/2396293535 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b80d0270 msgr2=0x7fc8b815f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:59.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.484+0000 7fc8b77fe700 1 --2- 172.21.15.98:0/2396293535 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b80d0270 0x7fc8b815f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.484+0000 7fc8b77fe700 1 -- 172.21.15.98:0/2396293535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ac0094a0 con 0x7fc8b815ca70 2023-03-03T07:56:59.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.484+0000 7fc8b67fc700 1 --2- 172.21.15.98:0/2396293535 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b80d0270 0x7fc8b815f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:56:59.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.484+0000 7fc8b77fe700 1 --2- 172.21.15.98:0/2396293535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b815ca70 0x7fc8b80c6c90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc8b000cbc0 tx=0x7fc8b000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:59.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.484+0000 7fc89f7fe700 1 -- 172.21.15.98:0/2396293535 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b000a9e0 con 0x7fc8b815ca70 2023-03-03T07:56:59.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.484+0000 7fc89f7fe700 1 -- 172.21.15.98:0/2396293535 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b000ab40 con 0x7fc8b815ca70 2023-03-03T07:56:59.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.484+0000 7fc89f7fe700 1 -- 172.21.15.98:0/2396293535 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0012400 con 0x7fc8b815ca70 2023-03-03T07:56:59.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.484+0000 7fc8bd5ff700 1 -- 172.21.15.98:0/2396293535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b80d1aa0 con 0x7fc8b815ca70 2023-03-03T07:56:59.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.484+0000 7fc8bd5ff700 1 -- 172.21.15.98:0/2396293535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b80cdbe0 con 0x7fc8b815ca70 2023-03-03T07:56:59.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.485+0000 7fc8bd5ff700 1 -- 172.21.15.98:0/2396293535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b815ce40 con 0x7fc8b815ca70 2023-03-03T07:56:59.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.487+0000 7fc89f7fe700 1 -- 172.21.15.98:0/2396293535 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc8b0019030 con 0x7fc8b815ca70 2023-03-03T07:56:59.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.488+0000 7fc89f7fe700 1 --2- 172.21.15.98:0/2396293535 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc8a40682f0 0x7fc8a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:59.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.488+0000 7fc89f7fe700 1 -- 172.21.15.98:0/2396293535 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fc8b0085760 con 0x7fc8b815ca70 2023-03-03T07:56:59.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.488+0000 7fc8b6ffd700 1 --2- 172.21.15.98:0/2396293535 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc8a40682f0 0x7fc8a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:59.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.489+0000 7fc8b6ffd700 1 --2- 172.21.15.98:0/2396293535 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc8a40682f0 0x7fc8a406a7a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc8b80c6bf0 tx=0x7fc8a8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:59.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.492+0000 7fc89f7fe700 1 -- 172.21.15.98:0/2396293535 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc8b004b5e0 con 0x7fc8b815ca70 2023-03-03T07:56:59.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.686+0000 7fc8bd5ff700 1 -- 172.21.15.98:0/2396293535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8b80cbfa0 con 0x7fc8b815ca70 2023-03-03T07:56:59.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.687+0000 7fc89f7fe700 1 -- 172.21.15.98:0/2396293535 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc8b0051f40 con 0x7fc8b815ca70 2023-03-03T07:56:59.689 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:56:59.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.689+0000 7fc8bd5ff700 1 -- 172.21.15.98:0/2396293535 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc8a40682f0 msgr2=0x7fc8a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:59.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.689+0000 7fc8bd5ff700 1 --2- 172.21.15.98:0/2396293535 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc8a40682f0 0x7fc8a406a7a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc8b80c6bf0 tx=0x7fc8a8009040 comp rx=0 tx=0).stop 2023-03-03T07:56:59.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.689+0000 7fc8bd5ff700 1 -- 172.21.15.98:0/2396293535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b815ca70 msgr2=0x7fc8b80c6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:59.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.689+0000 7fc8bd5ff700 1 --2- 172.21.15.98:0/2396293535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b815ca70 0x7fc8b80c6c90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc8b000cbc0 tx=0x7fc8b000ced0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.689+0000 7fc8bd5ff700 1 -- 172.21.15.98:0/2396293535 shutdown_connections 2023-03-03T07:56:59.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.689+0000 7fc8bd5ff700 1 --2- 172.21.15.98:0/2396293535 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc8a40682f0 0x7fc8a406a7a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.689+0000 7fc8bd5ff700 1 --2- 172.21.15.98:0/2396293535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b815ca70 0x7fc8b80c6c90 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.689+0000 7fc8bd5ff700 1 --2- 172.21.15.98:0/2396293535 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8b806b4b0 0x7fc8b81535c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.689+0000 7fc8bd5ff700 1 --2- 172.21.15.98:0/2396293535 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b80d0270 0x7fc8b815f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.689+0000 7fc8bd5ff700 1 -- 172.21.15.98:0/2396293535 >> 172.21.15.98:0/2396293535 conn(0x7fc8b8073e70 msgr2=0x7fc8b80b7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:59.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.690+0000 7fc8bd5ff700 1 -- 172.21.15.98:0/2396293535 shutdown_connections 2023-03-03T07:56:59.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.690+0000 7fc8bd5ff700 1 -- 172.21.15.98:0/2396293535 wait complete. 2023-03-03T07:56:59.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:56:59.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.964+0000 7f3b7319d700 1 Processor -- start 2023-03-03T07:56:59.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.964+0000 7f3b7319d700 1 -- start start 2023-03-03T07:56:59.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.965+0000 7f3b7319d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3b6c1609b0 0x7f3b6c160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:59.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.965+0000 7f3b7319d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3b6c0cf570 0x7f3b6c15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:59.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.966+0000 7f3b7319d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b6c15db30 0x7f3b6c0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.966+0000 7f3b7319d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6c0cebb0 con 0x7f3b6c1609b0 2023-03-03T07:56:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.966+0000 7f3b7319d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6c1612c0 con 0x7f3b6c0cf570 2023-03-03T07:56:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.966+0000 7f3b7319d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c0cd780 con 0x7f3b6c15db30 2023-03-03T07:56:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.966+0000 7f3b7219b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3b6c1609b0 0x7f3b6c160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.966+0000 7f3b7219b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3b6c1609b0 0x7f3b6c160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45786/0 (socket says 172.21.15.98:45786) 2023-03-03T07:56:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.966+0000 7f3b7219b700 1 -- 172.21.15.98:0/2088654719 learned_addr learned my addr 172.21.15.98:0/2088654719 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.966+0000 7f3b7299c700 1 --2- 172.21.15.98:0/2088654719 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b6c15db30 0x7f3b6c0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.966+0000 7f3b7199a700 1 --2- 172.21.15.98:0/2088654719 >> v2:172.21.15.100:3300/0 conn(0x7f3b6c0cf570 0x7f3b6c15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:59.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.967+0000 7f3b7219b700 1 -- 172.21.15.98:0/2088654719 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b6c15db30 msgr2=0x7f3b6c0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:59.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.967+0000 7f3b7219b700 1 --2- 172.21.15.98:0/2088654719 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b6c15db30 0x7f3b6c0b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.967+0000 7f3b7219b700 1 -- 172.21.15.98:0/2088654719 >> v2:172.21.15.100:3300/0 conn(0x7f3b6c0cf570 msgr2=0x7f3b6c15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:56:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.967+0000 7f3b7219b700 1 --2- 172.21.15.98:0/2088654719 >> v2:172.21.15.100:3300/0 conn(0x7f3b6c0cf570 0x7f3b6c15d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.967+0000 7f3b7219b700 1 -- 172.21.15.98:0/2088654719 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6c0cda60 con 0x7f3b6c1609b0 2023-03-03T07:56:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.967+0000 7f3b7219b700 1 --2- 172.21.15.98:0/2088654719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b6c1609b0 0x7f3b6c160d80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0cf0c0 tx=0x7f3b68009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d436f1a8f58ec005 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.967+0000 7f3b71199700 1 -- 172.21.15.98:0/2088654719 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b68011830 con 0x7f3b6c1609b0 2023-03-03T07:56:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.967+0000 7f3b71199700 1 -- 172.21.15.98:0/2088654719 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b68011990 con 0x7f3b6c1609b0 2023-03-03T07:56:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.967+0000 7f3b71199700 1 -- 172.21.15.98:0/2088654719 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b68011d20 con 0x7f3b6c1609b0 2023-03-03T07:56:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.968+0000 7f3b7319d700 1 -- 172.21.15.98:0/2088654719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b6c1609b0 msgr2=0x7f3b6c160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.968+0000 7f3b7319d700 1 --2- 172.21.15.98:0/2088654719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b6c1609b0 0x7f3b6c160d80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0cf0c0 tx=0x7f3b68009bd0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.968+0000 7f3b7319d700 1 -- 172.21.15.98:0/2088654719 shutdown_connections 2023-03-03T07:56:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.968+0000 7f3b7319d700 1 --2- 172.21.15.98:0/2088654719 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b6c15db30 0x7f3b6c0b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.968+0000 7f3b7319d700 1 --2- 172.21.15.98:0/2088654719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b6c1609b0 0x7f3b6c160d80 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.968+0000 7f3b7319d700 1 --2- 172.21.15.98:0/2088654719 >> v2:172.21.15.100:3300/0 conn(0x7f3b6c0cf570 0x7f3b6c15d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.968+0000 7f3b7319d700 1 -- 172.21.15.98:0/2088654719 >> 172.21.15.98:0/2088654719 conn(0x7f3b6c06b880 msgr2=0x7f3b6c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.968+0000 7f3b7319d700 1 -- 172.21.15.98:0/2088654719 shutdown_connections 2023-03-03T07:56:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.968+0000 7f3b7319d700 1 -- 172.21.15.98:0/2088654719 wait complete. 2023-03-03T07:56:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.970+0000 7f3b7319d700 1 Processor -- start 2023-03-03T07:56:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.970+0000 7f3b7319d700 1 -- start start 2023-03-03T07:56:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.971+0000 7f3b7319d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b6c0cf570 0x7f3b6c150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.971+0000 7f3b7319d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b6c15db30 0x7f3b6c0b5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.971+0000 7f3b7319d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b6c1609b0 0x7f3b6c0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.971+0000 7f3b7319d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c13b520 con 0x7f3b6c0cf570 2023-03-03T07:56:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.971+0000 7f3b7319d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c1e6620 con 0x7f3b6c15db30 2023-03-03T07:56:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.971+0000 7f3b7319d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c0d1f90 con 0x7f3b6c1609b0 2023-03-03T07:56:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.972+0000 7f3b7219b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b6c0cf570 0x7f3b6c150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.972+0000 7f3b7219b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b6c0cf570 0x7f3b6c150b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45792/0 (socket says 172.21.15.98:45792) 2023-03-03T07:56:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.972+0000 7f3b7219b700 1 -- 172.21.15.98:0/3516194425 learned_addr learned my addr 172.21.15.98:0/3516194425 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:56:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.972+0000 7f3b7299c700 1 --2- 172.21.15.98:0/3516194425 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b6c1609b0 0x7f3b6c0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.972+0000 7f3b7199a700 1 --2- 172.21.15.98:0/3516194425 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b6c15db30 0x7f3b6c0b5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.972+0000 7f3b7219b700 1 -- 172.21.15.98:0/3516194425 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b6c1609b0 msgr2=0x7f3b6c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.972+0000 7f3b7219b700 1 --2- 172.21.15.98:0/3516194425 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b6c1609b0 0x7f3b6c0b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.972+0000 7f3b7219b700 1 -- 172.21.15.98:0/3516194425 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b6c15db30 msgr2=0x7f3b6c0b5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.972+0000 7f3b7219b700 1 --2- 172.21.15.98:0/3516194425 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b6c15db30 0x7f3b6c0b5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.972+0000 7f3b7219b700 1 -- 172.21.15.98:0/3516194425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b68009550 con 0x7f3b6c0cf570 2023-03-03T07:56:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.972+0000 7f3b7219b700 1 --2- 172.21.15.98:0/3516194425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b6c0cf570 0x7f3b6c150b60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3b68014040 tx=0x7f3b68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.973+0000 7f3b62ffd700 1 -- 172.21.15.98:0/3516194425 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b68004450 con 0x7f3b6c0cf570 2023-03-03T07:56:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.973+0000 7f3b62ffd700 1 -- 172.21.15.98:0/3516194425 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b6801e070 con 0x7f3b6c0cf570 2023-03-03T07:56:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.973+0000 7f3b62ffd700 1 -- 172.21.15.98:0/3516194425 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b680038e0 con 0x7f3b6c0cf570 2023-03-03T07:56:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.973+0000 7f3b7319d700 1 -- 172.21.15.98:0/3516194425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6c15cdf0 con 0x7f3b6c0cf570 2023-03-03T07:56:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.973+0000 7f3b7319d700 1 -- 172.21.15.98:0/3516194425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6c0cdd40 con 0x7f3b6c0cf570 2023-03-03T07:56:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.974+0000 7f3b7319d700 1 -- 172.21.15.98:0/3516194425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6c15d0d0 con 0x7f3b6c0cf570 2023-03-03T07:56:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.975+0000 7f3b62ffd700 1 -- 172.21.15.98:0/3516194425 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3b6800f050 con 0x7f3b6c0cf570 2023-03-03T07:56:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.976+0000 7f3b62ffd700 1 --2- 172.21.15.98:0/3516194425 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b500682f0 0x7f3b5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.976+0000 7f3b62ffd700 1 -- 172.21.15.98:0/3516194425 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f3b68086a60 con 0x7f3b6c0cf570 2023-03-03T07:56:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.977+0000 7f3b7199a700 1 --2- 172.21.15.98:0/3516194425 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b500682f0 0x7f3b5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.977+0000 7f3b7199a700 1 --2- 172.21.15.98:0/3516194425 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b500682f0 0x7f3b5006a7a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c151b70 tx=0x7f3b5c009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:59.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:56:59.981+0000 7f3b62ffd700 1 -- 172.21.15.98:0/3516194425 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3b68053190 con 0x7f3b6c0cf570 2023-03-03T07:57:00.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.172+0000 7f3b7319d700 1 -- 172.21.15.98:0/3516194425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3b6c068290 con 0x7f3b6c0cf570 2023-03-03T07:57:00.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.172+0000 7f3b62ffd700 1 -- 172.21.15.98:0/3516194425 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3b680200a0 con 0x7f3b6c0cf570 2023-03-03T07:57:00.174 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:57:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.175+0000 7f3b7319d700 1 -- 172.21.15.98:0/3516194425 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b500682f0 msgr2=0x7f3b5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.175+0000 7f3b7319d700 1 --2- 172.21.15.98:0/3516194425 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b500682f0 0x7f3b5006a7a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c151b70 tx=0x7f3b5c009210 comp rx=0 tx=0).stop 2023-03-03T07:57:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.175+0000 7f3b7319d700 1 -- 172.21.15.98:0/3516194425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b6c0cf570 msgr2=0x7f3b6c150b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.175+0000 7f3b7319d700 1 --2- 172.21.15.98:0/3516194425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b6c0cf570 0x7f3b6c150b60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3b68014040 tx=0x7f3b68004750 comp rx=0 tx=0).stop 2023-03-03T07:57:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.175+0000 7f3b7319d700 1 -- 172.21.15.98:0/3516194425 shutdown_connections 2023-03-03T07:57:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.175+0000 7f3b7319d700 1 --2- 172.21.15.98:0/3516194425 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b500682f0 0x7f3b5006a7a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.175+0000 7f3b7319d700 1 --2- 172.21.15.98:0/3516194425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b6c0cf570 0x7f3b6c150b60 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.175+0000 7f3b7319d700 1 --2- 172.21.15.98:0/3516194425 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b6c1609b0 0x7f3b6c0b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.175+0000 7f3b7319d700 1 --2- 172.21.15.98:0/3516194425 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b6c15db30 0x7f3b6c0b5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.175+0000 7f3b7319d700 1 -- 172.21.15.98:0/3516194425 >> 172.21.15.98:0/3516194425 conn(0x7f3b6c06b880 msgr2=0x7f3b6c15b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:00.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.175+0000 7f3b7319d700 1 -- 172.21.15.98:0/3516194425 shutdown_connections 2023-03-03T07:57:00.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.175+0000 7f3b7319d700 1 -- 172.21.15.98:0/3516194425 wait complete. 2023-03-03T07:57:00.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:57:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.452+0000 7f40701ff700 1 Processor -- start 2023-03-03T07:57:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.452+0000 7f40701ff700 1 -- start start 2023-03-03T07:57:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.453+0000 7f40701ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f406815ca70 0x7f40680cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.453+0000 7f40701ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f406806b4b0 0x7f40680cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.453+0000 7f40701ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40680d00a0 0x7f40680b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.453+0000 7f40701ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40680cd5b0 con 0x7f406815ca70 2023-03-03T07:57:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.453+0000 7f40701ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40680cf980 con 0x7f406806b4b0 2023-03-03T07:57:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.453+0000 7f40701ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40680d0510 con 0x7f40680d00a0 2023-03-03T07:57:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.454+0000 7f406df9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f406815ca70 0x7f40680cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.454+0000 7f406df9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f406815ca70 0x7f40680cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45796/0 (socket says 172.21.15.98:45796) 2023-03-03T07:57:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.454+0000 7f406df9b700 1 -- 172.21.15.98:0/4258631582 learned_addr learned my addr 172.21.15.98:0/4258631582 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.454+0000 7f406d79a700 1 --2- 172.21.15.98:0/4258631582 >> v2:172.21.15.100:3300/0 conn(0x7f406806b4b0 0x7f40680cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.454+0000 7f406e79c700 1 --2- 172.21.15.98:0/4258631582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40680d00a0 0x7f40680b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.454+0000 7f406df9b700 1 -- 172.21.15.98:0/4258631582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40680d00a0 msgr2=0x7f40680b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.454+0000 7f406df9b700 1 --2- 172.21.15.98:0/4258631582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40680d00a0 0x7f40680b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.454+0000 7f406df9b700 1 -- 172.21.15.98:0/4258631582 >> v2:172.21.15.100:3300/0 conn(0x7f406806b4b0 msgr2=0x7f40680cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:00.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.454+0000 7f406df9b700 1 --2- 172.21.15.98:0/4258631582 >> v2:172.21.15.100:3300/0 conn(0x7f406806b4b0 0x7f40680cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.454+0000 7f406df9b700 1 -- 172.21.15.98:0/4258631582 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40680d1800 con 0x7f406815ca70 2023-03-03T07:57:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.454+0000 7f406e79c700 1 --2- 172.21.15.98:0/4258631582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40680d00a0 0x7f40680b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.455+0000 7f406df9b700 1 --2- 172.21.15.98:0/4258631582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f406815ca70 0x7f40680cf440 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4068160560 tx=0x7f4064009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db3b6466c3a1a2e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.455+0000 7f406cf99700 1 -- 172.21.15.98:0/4258631582 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4064011830 con 0x7f406815ca70 2023-03-03T07:57:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.455+0000 7f406cf99700 1 -- 172.21.15.98:0/4258631582 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4064011990 con 0x7f406815ca70 2023-03-03T07:57:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.455+0000 7f406cf99700 1 -- 172.21.15.98:0/4258631582 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4064011d20 con 0x7f406815ca70 2023-03-03T07:57:00.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.456+0000 7f40701ff700 1 -- 172.21.15.98:0/4258631582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f406815ca70 msgr2=0x7f40680cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.456+0000 7f40701ff700 1 --2- 172.21.15.98:0/4258631582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f406815ca70 0x7f40680cf440 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4068160560 tx=0x7f4064009c00 comp rx=0 tx=0).stop 2023-03-03T07:57:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.456+0000 7f40701ff700 1 -- 172.21.15.98:0/4258631582 shutdown_connections 2023-03-03T07:57:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.456+0000 7f40701ff700 1 --2- 172.21.15.98:0/4258631582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40680d00a0 0x7f40680b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.456+0000 7f40701ff700 1 --2- 172.21.15.98:0/4258631582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f406815ca70 0x7f40680cf440 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.456+0000 7f40701ff700 1 --2- 172.21.15.98:0/4258631582 >> v2:172.21.15.100:3300/0 conn(0x7f406806b4b0 0x7f40680cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.456+0000 7f40701ff700 1 -- 172.21.15.98:0/4258631582 >> 172.21.15.98:0/4258631582 conn(0x7f4068073e70 msgr2=0x7f406806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.456+0000 7f40701ff700 1 -- 172.21.15.98:0/4258631582 shutdown_connections 2023-03-03T07:57:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.456+0000 7f40701ff700 1 -- 172.21.15.98:0/4258631582 wait complete. 2023-03-03T07:57:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.458+0000 7f40701ff700 1 Processor -- start 2023-03-03T07:57:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.458+0000 7f40701ff700 1 -- start start 2023-03-03T07:57:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.459+0000 7f40701ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f406806b4b0 0x7f4068154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.459+0000 7f40701ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40680d00a0 0x7f40680ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.459+0000 7f40701ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f406815ca70 0x7f40680caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.459+0000 7f40701ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40680c9be0 con 0x7f406815ca70 2023-03-03T07:57:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.459+0000 7f40701ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40680ca790 con 0x7f40680d00a0 2023-03-03T07:57:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.459+0000 7f40701ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40680c9900 con 0x7f406806b4b0 2023-03-03T07:57:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.459+0000 7f406e79c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f406815ca70 0x7f40680caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.459+0000 7f406e79c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f406815ca70 0x7f40680caa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45812/0 (socket says 172.21.15.98:45812) 2023-03-03T07:57:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.459+0000 7f406e79c700 1 -- 172.21.15.98:0/3349623954 learned_addr learned my addr 172.21.15.98:0/3349623954 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.460+0000 7f406df9b700 1 --2- 172.21.15.98:0/3349623954 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f406806b4b0 0x7f4068154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.460+0000 7f406d79a700 1 --2- 172.21.15.98:0/3349623954 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40680d00a0 0x7f40680ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.460+0000 7f406e79c700 1 -- 172.21.15.98:0/3349623954 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f406806b4b0 msgr2=0x7f4068154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:00.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.460+0000 7f406e79c700 1 --2- 172.21.15.98:0/3349623954 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f406806b4b0 0x7f4068154350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.460+0000 7f406e79c700 1 -- 172.21.15.98:0/3349623954 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40680d00a0 msgr2=0x7f40680ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:00.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.460+0000 7f406e79c700 1 --2- 172.21.15.98:0/3349623954 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40680d00a0 0x7f40680ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.460+0000 7f406e79c700 1 -- 172.21.15.98:0/3349623954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4064009550 con 0x7f406815ca70 2023-03-03T07:57:00.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.460+0000 7f406e79c700 1 --2- 172.21.15.98:0/3349623954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f406815ca70 0x7f40680caa10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4060009fa0 tx=0x7f406000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:00.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.460+0000 7f405e7fc700 1 -- 172.21.15.98:0/3349623954 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406000ddb0 con 0x7f406815ca70 2023-03-03T07:57:00.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.460+0000 7f405e7fc700 1 -- 172.21.15.98:0/3349623954 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406000f040 con 0x7f406815ca70 2023-03-03T07:57:00.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.460+0000 7f405e7fc700 1 -- 172.21.15.98:0/3349623954 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060003c60 con 0x7f406815ca70 2023-03-03T07:57:00.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.461+0000 7f40701ff700 1 -- 172.21.15.98:0/3349623954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40680cdbe0 con 0x7f406815ca70 2023-03-03T07:57:00.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.461+0000 7f40701ff700 1 -- 172.21.15.98:0/3349623954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40680cd900 con 0x7f406815ca70 2023-03-03T07:57:00.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.462+0000 7f40701ff700 1 -- 172.21.15.98:0/3349623954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40680cca30 con 0x7f406815ca70 2023-03-03T07:57:00.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.463+0000 7f405e7fc700 1 -- 172.21.15.98:0/3349623954 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f406001c030 con 0x7f406815ca70 2023-03-03T07:57:00.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.464+0000 7f405e7fc700 1 --2- 172.21.15.98:0/3349623954 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f404c0682f0 0x7f404c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:00.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.464+0000 7f405e7fc700 1 -- 172.21.15.98:0/3349623954 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f4060085cf0 con 0x7f406815ca70 2023-03-03T07:57:00.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.464+0000 7f406df9b700 1 --2- 172.21.15.98:0/3349623954 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f404c0682f0 0x7f404c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:00.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.465+0000 7f406df9b700 1 --2- 172.21.15.98:0/3349623954 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f404c0682f0 0x7f404c06a7a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4064014040 tx=0x7f4064008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:00.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.468+0000 7f405e7fc700 1 -- 172.21.15.98:0/3349623954 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f40680cca30 con 0x7f406815ca70 2023-03-03T07:57:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.663+0000 7f40701ff700 1 -- 172.21.15.98:0/3349623954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f40680cca30 con 0x7f406815ca70 2023-03-03T07:57:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.664+0000 7f405e7fc700 1 -- 172.21.15.98:0/3349623954 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f406001a020 con 0x7f406815ca70 2023-03-03T07:57:00.666 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:57:00.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.666+0000 7f40701ff700 1 -- 172.21.15.98:0/3349623954 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f404c0682f0 msgr2=0x7f404c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:00.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.666+0000 7f40701ff700 1 --2- 172.21.15.98:0/3349623954 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f404c0682f0 0x7f404c06a7a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4064014040 tx=0x7f4064008030 comp rx=0 tx=0).stop 2023-03-03T07:57:00.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.666+0000 7f40701ff700 1 -- 172.21.15.98:0/3349623954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f406815ca70 msgr2=0x7f40680caa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:00.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.666+0000 7f40701ff700 1 --2- 172.21.15.98:0/3349623954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f406815ca70 0x7f40680caa10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4060009fa0 tx=0x7f406000d3b0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.667+0000 7f40701ff700 1 -- 172.21.15.98:0/3349623954 shutdown_connections 2023-03-03T07:57:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.667+0000 7f40701ff700 1 --2- 172.21.15.98:0/3349623954 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f404c0682f0 0x7f404c06a7a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.667+0000 7f40701ff700 1 --2- 172.21.15.98:0/3349623954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f406815ca70 0x7f40680caa10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.667+0000 7f40701ff700 1 --2- 172.21.15.98:0/3349623954 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f406806b4b0 0x7f4068154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.667+0000 7f40701ff700 1 --2- 172.21.15.98:0/3349623954 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40680d00a0 0x7f40680ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.667+0000 7f40701ff700 1 -- 172.21.15.98:0/3349623954 >> 172.21.15.98:0/3349623954 conn(0x7f4068073e70 msgr2=0x7f406806aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.667+0000 7f40701ff700 1 -- 172.21.15.98:0/3349623954 shutdown_connections 2023-03-03T07:57:00.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.667+0000 7f40701ff700 1 -- 172.21.15.98:0/3349623954 wait complete. 2023-03-03T07:57:00.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:57:00.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.943+0000 7fb7fefff700 1 Processor -- start 2023-03-03T07:57:00.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.943+0000 7fb7fefff700 1 -- start start 2023-03-03T07:57:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.945+0000 7fb7fefff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7f80b26d0 0x7fb7f80b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.945+0000 7fb7fefff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb7f80b5140 0x7fb7f80d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.945+0000 7fb7fefff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f80cd7d0 con 0x7fb7f80b8d10 2023-03-03T07:57:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.945+0000 7fb7fefff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f80b55c0 con 0x7fb7f80b5140 2023-03-03T07:57:00.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.945+0000 7fb7fefff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80b90e0 con 0x7fb7f80b26d0 2023-03-03T07:57:00.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.945+0000 7fb7fcd9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb7f80b8d10 0x7fb7f80cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34358/0 (socket says 172.21.15.98:34358) 2023-03-03T07:57:00.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.945+0000 7fb7fcd9b700 1 -- 172.21.15.98:0/2809335877 learned_addr learned my addr 172.21.15.98:0/2809335877 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:00.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.945+0000 7fb7fd59c700 1 --2- 172.21.15.98:0/2809335877 >> v2:172.21.15.98:3300/0 conn(0x7fb7f80b5140 0x7fb7f80d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:00.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.945+0000 7fb7f7fff700 1 --2- 172.21.15.98:0/2809335877 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7f80b26d0 0x7fb7f80b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:00.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.946+0000 7fb7f77fe700 1 -- 172.21.15.98:0/2809335877 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999214189 0 0) 0x7fb7f80cd7d0 con 0x7fb7f80b8d10 2023-03-03T07:57:00.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.946+0000 7fb7f77fe700 1 -- 172.21.15.98:0/2809335877 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e80035b0 con 0x7fb7f80b8d10 2023-03-03T07:57:00.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.946+0000 7fb7fd59c700 1 -- 172.21.15.98:0/2809335877 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7f80b26d0 msgr2=0x7fb7f80b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:00.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.946+0000 7fb7fd59c700 1 --2- 172.21.15.98:0/2809335877 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7f80b26d0 0x7fb7f80b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.946+0000 7fb7fd59c700 1 -- 172.21.15.98:0/2809335877 >> v1:172.21.15.98:6789/0 conn(0x7fb7f80b8d10 legacy=0x7fb7f80cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:00.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.946+0000 7fb7fd59c700 1 -- 172.21.15.98:0/2809335877 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f80cda20 con 0x7fb7f80b5140 2023-03-03T07:57:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.946+0000 7fb7fd59c700 1 --2- 172.21.15.98:0/2809335877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f80b5140 0x7fb7f80d13a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009d90 tx=0x7fb7f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a4452a98e0f0b47 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.946+0000 7fb7f77fe700 1 -- 172.21.15.98:0/2809335877 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f000e070 con 0x7fb7f80b5140 2023-03-03T07:57:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.946+0000 7fb7f77fe700 1 -- 172.21.15.98:0/2809335877 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f000bb20 con 0x7fb7f80b5140 2023-03-03T07:57:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.946+0000 7fb7f77fe700 1 -- 172.21.15.98:0/2809335877 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0012430 con 0x7fb7f80b5140 2023-03-03T07:57:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.947+0000 7fb7fefff700 1 -- 172.21.15.98:0/2809335877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f80b5140 msgr2=0x7fb7f80d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.947+0000 7fb7fefff700 1 --2- 172.21.15.98:0/2809335877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f80b5140 0x7fb7f80d13a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009d90 tx=0x7fb7f000b330 comp rx=0 tx=0).stop 2023-03-03T07:57:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.947+0000 7fb7fefff700 1 -- 172.21.15.98:0/2809335877 shutdown_connections 2023-03-03T07:57:00.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.947+0000 7fb7fefff700 1 --2- 172.21.15.98:0/2809335877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f80b5140 0x7fb7f80d13a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.947+0000 7fb7fefff700 1 --2- 172.21.15.98:0/2809335877 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7f80b26d0 0x7fb7f80b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.947+0000 7fb7fefff700 1 -- 172.21.15.98:0/2809335877 >> 172.21.15.98:0/2809335877 conn(0x7fb7f8073e70 msgr2=0x7fb7f806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:00.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.947+0000 7fb7fefff700 1 -- 172.21.15.98:0/2809335877 shutdown_connections 2023-03-03T07:57:00.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.947+0000 7fb7fefff700 1 -- 172.21.15.98:0/2809335877 wait complete. 2023-03-03T07:57:00.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.949+0000 7fb7fefff700 1 Processor -- start 2023-03-03T07:57:00.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.950+0000 7fb7fefff700 1 -- start start 2023-03-03T07:57:00.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.950+0000 7fb7fefff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f80b26d0 0x7fb7f80cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:00.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7fefff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7f80b5140 0x7fb7f815cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:00.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7fefff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7f80b8d10 0x7fb7f814a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7fefff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80cc840 con 0x7fb7f80b26d0 2023-03-03T07:57:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7fefff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f8154b30 con 0x7fb7f80b5140 2023-03-03T07:57:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7fefff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80cbbc0 con 0x7fb7f80b8d10 2023-03-03T07:57:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7fcd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f80b26d0 0x7fb7f80cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7fcd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f80b26d0 0x7fb7f80cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45820/0 (socket says 172.21.15.98:45820) 2023-03-03T07:57:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7fcd9b700 1 -- 172.21.15.98:0/2063773244 learned_addr learned my addr 172.21.15.98:0/2063773244 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7f7fff700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7f80b5140 0x7fb7f815cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:00.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7fd59c700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7f80b8d10 0x7fb7f814a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:00.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7fcd9b700 1 -- 172.21.15.98:0/2063773244 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7f80b8d10 msgr2=0x7fb7f814a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:00.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7fcd9b700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7f80b8d10 0x7fb7f814a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7fcd9b700 1 -- 172.21.15.98:0/2063773244 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7f80b5140 msgr2=0x7fb7f815cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:00.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7fcd9b700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7f80b5140 0x7fb7f815cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:00.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7fcd9b700 1 -- 172.21.15.98:0/2063773244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f0009a70 con 0x7fb7f80b26d0 2023-03-03T07:57:00.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7f7fff700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7f80b5140 0x7fb7f815cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:00.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.951+0000 7fb7fd59c700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7f80b8d10 0x7fb7f814a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:00.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.952+0000 7fb7fcd9b700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f80b26d0 0x7fb7f80cb5e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec001eb0 tx=0x7fb7ec00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:00.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.952+0000 7fb7f57fa700 1 -- 172.21.15.98:0/2063773244 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec005d70 con 0x7fb7f80b26d0 2023-03-03T07:57:00.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.952+0000 7fb7f57fa700 1 -- 172.21.15.98:0/2063773244 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7ec0070d0 con 0x7fb7f80b26d0 2023-03-03T07:57:00.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.952+0000 7fb7fefff700 1 -- 172.21.15.98:0/2063773244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f80d0e80 con 0x7fb7f80b26d0 2023-03-03T07:57:00.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.952+0000 7fb7f57fa700 1 -- 172.21.15.98:0/2063773244 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec007460 con 0x7fb7f80b26d0 2023-03-03T07:57:00.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.952+0000 7fb7fefff700 1 -- 172.21.15.98:0/2063773244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f80cdd00 con 0x7fb7f80b26d0 2023-03-03T07:57:00.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.953+0000 7fb7fefff700 1 -- 172.21.15.98:0/2063773244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f80cdfe0 con 0x7fb7f80b26d0 2023-03-03T07:57:00.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.954+0000 7fb7f57fa700 1 -- 172.21.15.98:0/2063773244 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb7ec019030 con 0x7fb7f80b26d0 2023-03-03T07:57:00.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.956+0000 7fb7f57fa700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb7dc0682f0 0x7fb7dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:00.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.957+0000 7fb7f57fa700 1 -- 172.21.15.98:0/2063773244 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fb7ec0851c0 con 0x7fb7f80b26d0 2023-03-03T07:57:00.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.957+0000 7fb7f7fff700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb7dc0682f0 0x7fb7dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:00.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.957+0000 7fb7f7fff700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb7dc0682f0 0x7fb7dc06a7a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4009b90 tx=0x7fb7e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:00.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:00.961+0000 7fb7f57fa700 1 -- 172.21.15.98:0/2063773244 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb7ec0518f0 con 0x7fb7f80b26d0 2023-03-03T07:57:01.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.154+0000 7fb7fefff700 1 -- 172.21.15.98:0/2063773244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb7f804e6e0 con 0x7fb7f80b26d0 2023-03-03T07:57:01.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.154+0000 7fb7f57fa700 1 -- 172.21.15.98:0/2063773244 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb7ec054f10 con 0x7fb7f80b26d0 2023-03-03T07:57:01.157 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:57:01.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.157+0000 7fb7fefff700 1 -- 172.21.15.98:0/2063773244 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb7dc0682f0 msgr2=0x7fb7dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:01.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.157+0000 7fb7fefff700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb7dc0682f0 0x7fb7dc06a7a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4009b90 tx=0x7fb7e4008040 comp rx=0 tx=0).stop 2023-03-03T07:57:01.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.157+0000 7fb7fefff700 1 -- 172.21.15.98:0/2063773244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f80b26d0 msgr2=0x7fb7f80cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:01.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.157+0000 7fb7fefff700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f80b26d0 0x7fb7f80cb5e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec001eb0 tx=0x7fb7ec00c860 comp rx=0 tx=0).stop 2023-03-03T07:57:01.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.157+0000 7fb7fefff700 1 -- 172.21.15.98:0/2063773244 shutdown_connections 2023-03-03T07:57:01.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.157+0000 7fb7fefff700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb7dc0682f0 0x7fb7dc06a7a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.157+0000 7fb7fefff700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f80b26d0 0x7fb7f80cb5e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.157+0000 7fb7fefff700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7f80b8d10 0x7fb7f814a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.157+0000 7fb7fefff700 1 --2- 172.21.15.98:0/2063773244 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7f80b5140 0x7fb7f815cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.157+0000 7fb7fefff700 1 -- 172.21.15.98:0/2063773244 >> 172.21.15.98:0/2063773244 conn(0x7fb7f8073e70 msgr2=0x7fb7f815b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.157+0000 7fb7fefff700 1 -- 172.21.15.98:0/2063773244 shutdown_connections 2023-03-03T07:57:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.157+0000 7fb7fefff700 1 -- 172.21.15.98:0/2063773244 wait complete. 2023-03-03T07:57:01.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:57:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.430+0000 7f544539d700 1 Processor -- start 2023-03-03T07:57:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.430+0000 7f544539d700 1 -- start start 2023-03-03T07:57:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.432+0000 7f544539d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5440146360 0x7f5440160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.432+0000 7f544539d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f54400d2590 0x7f54400b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.432+0000 7f544539d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54401614c0 con 0x7f54400b4ab0 2023-03-03T07:57:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.432+0000 7f544539d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54400b5460 con 0x7f54400d2590 2023-03-03T07:57:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.433+0000 7f544539d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54400d10f0 con 0x7f5440146360 2023-03-03T07:57:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.433+0000 7f543f7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f54400b4ab0 0x7f54400731e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34374/0 (socket says 172.21.15.98:34374) 2023-03-03T07:57:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.433+0000 7f543f7fe700 1 -- 172.21.15.98:0/2312511880 learned_addr learned my addr 172.21.15.98:0/2312511880 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.433+0000 7f5444b9c700 1 --2- 172.21.15.98:0/2312511880 >> v2:172.21.15.98:3300/0 conn(0x7f54400d2590 0x7f54400b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.433+0000 7f543ffff700 1 --2- 172.21.15.98:0/2312511880 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5440146360 0x7f5440160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.433+0000 7f543effd700 1 -- 172.21.15.98:0/2312511880 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098883855 0 0) 0x7f5434001110 con 0x7f54400b4ab0 2023-03-03T07:57:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.433+0000 7f543effd700 1 -- 172.21.15.98:0/2312511880 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54280035b0 con 0x7f54400b4ab0 2023-03-03T07:57:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.433+0000 7f5444b9c700 1 -- 172.21.15.98:0/2312511880 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5440146360 msgr2=0x7f5440160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.433+0000 7f5444b9c700 1 --2- 172.21.15.98:0/2312511880 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5440146360 0x7f5440160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.433+0000 7f5444b9c700 1 -- 172.21.15.98:0/2312511880 >> v1:172.21.15.98:6789/0 conn(0x7f54400b4ab0 legacy=0x7f54400731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.433+0000 7f5444b9c700 1 -- 172.21.15.98:0/2312511880 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54400cdf20 con 0x7f54400d2590 2023-03-03T07:57:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.433+0000 7f543ffff700 1 --2- 172.21.15.98:0/2312511880 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5440146360 0x7f5440160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.434+0000 7f5444b9c700 1 --2- 172.21.15.98:0/2312511880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54400d2590 0x7f54400b4f20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5438009d90 tx=0x7f543800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d324558d204b4ad3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.434+0000 7f543effd700 1 -- 172.21.15.98:0/2312511880 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543800e070 con 0x7f54400d2590 2023-03-03T07:57:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.434+0000 7f543effd700 1 -- 172.21.15.98:0/2312511880 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543800bb20 con 0x7f54400d2590 2023-03-03T07:57:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.434+0000 7f543effd700 1 -- 172.21.15.98:0/2312511880 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5438012430 con 0x7f54400d2590 2023-03-03T07:57:01.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.435+0000 7f544539d700 1 -- 172.21.15.98:0/2312511880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54400d2590 msgr2=0x7f54400b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:01.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.435+0000 7f544539d700 1 --2- 172.21.15.98:0/2312511880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54400d2590 0x7f54400b4f20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5438009d90 tx=0x7f543800b330 comp rx=0 tx=0).stop 2023-03-03T07:57:01.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.435+0000 7f544539d700 1 -- 172.21.15.98:0/2312511880 shutdown_connections 2023-03-03T07:57:01.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.435+0000 7f544539d700 1 --2- 172.21.15.98:0/2312511880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54400d2590 0x7f54400b4f20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:01.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.435+0000 7f544539d700 1 --2- 172.21.15.98:0/2312511880 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5440146360 0x7f5440160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:01.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.435+0000 7f544539d700 1 -- 172.21.15.98:0/2312511880 >> 172.21.15.98:0/2312511880 conn(0x7f544006b8e0 msgr2=0x7f54400cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:01.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.435+0000 7f544539d700 1 -- 172.21.15.98:0/2312511880 shutdown_connections 2023-03-03T07:57:01.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.435+0000 7f544539d700 1 -- 172.21.15.98:0/2312511880 wait complete. 2023-03-03T07:57:01.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.438+0000 7f544539d700 1 Processor -- start 2023-03-03T07:57:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.438+0000 7f544539d700 1 -- start start 2023-03-03T07:57:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.439+0000 7f544539d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54400b4ab0 0x7f5440155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.439+0000 7f544539d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54400d2590 0x7f54400cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.439+0000 7f544539d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5440146360 0x7f54400cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.439+0000 7f544539d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54400c6ed0 con 0x7f54400b4ab0 2023-03-03T07:57:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.439+0000 7f544539d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54400c71b0 con 0x7f54400d2590 2023-03-03T07:57:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.439+0000 7f544539d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54400c7490 con 0x7f5440146360 2023-03-03T07:57:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.440+0000 7f543ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54400b4ab0 0x7f5440155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.440+0000 7f543ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54400b4ab0 0x7f5440155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45842/0 (socket says 172.21.15.98:45842) 2023-03-03T07:57:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.440+0000 7f543ffff700 1 -- 172.21.15.98:0/412574712 learned_addr learned my addr 172.21.15.98:0/412574712 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.440+0000 7f5444b9c700 1 --2- 172.21.15.98:0/412574712 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5440146360 0x7f54400cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.440+0000 7f543f7fe700 1 --2- 172.21.15.98:0/412574712 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54400d2590 0x7f54400cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:01.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.440+0000 7f543ffff700 1 -- 172.21.15.98:0/412574712 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5440146360 msgr2=0x7f54400cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:01.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.440+0000 7f543ffff700 1 --2- 172.21.15.98:0/412574712 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5440146360 0x7f54400cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:01.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.440+0000 7f543ffff700 1 -- 172.21.15.98:0/412574712 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54400d2590 msgr2=0x7f54400cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:01.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.440+0000 7f543ffff700 1 --2- 172.21.15.98:0/412574712 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54400d2590 0x7f54400cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:01.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.440+0000 7f543ffff700 1 -- 172.21.15.98:0/412574712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5438009a70 con 0x7f54400b4ab0 2023-03-03T07:57:01.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.440+0000 7f5444b9c700 1 --2- 172.21.15.98:0/412574712 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5440146360 0x7f54400cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:01.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.441+0000 7f543ffff700 1 --2- 172.21.15.98:0/412574712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54400b4ab0 0x7f5440155180 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f543000e760 tx=0x7f543000ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:01.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.441+0000 7f543cff9700 1 -- 172.21.15.98:0/412574712 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543000c070 con 0x7f54400b4ab0 2023-03-03T07:57:01.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.441+0000 7f543cff9700 1 -- 172.21.15.98:0/412574712 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5430013610 con 0x7f54400b4ab0 2023-03-03T07:57:01.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.441+0000 7f543cff9700 1 -- 172.21.15.98:0/412574712 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54300139c0 con 0x7f54400b4ab0 2023-03-03T07:57:01.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.441+0000 7f544539d700 1 -- 172.21.15.98:0/412574712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54400cd960 con 0x7f54400b4ab0 2023-03-03T07:57:01.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.441+0000 7f544539d700 1 -- 172.21.15.98:0/412574712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54400cd680 con 0x7f54400b4ab0 2023-03-03T07:57:01.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.442+0000 7f544539d700 1 -- 172.21.15.98:0/412574712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54400cdc40 con 0x7f54400b4ab0 2023-03-03T07:57:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.443+0000 7f543cff9700 1 -- 172.21.15.98:0/412574712 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5430023070 con 0x7f54400b4ab0 2023-03-03T07:57:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.444+0000 7f543cff9700 1 --2- 172.21.15.98:0/412574712 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f542c0682f0 0x7f542c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.445+0000 7f543cff9700 1 -- 172.21.15.98:0/412574712 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f5430085af0 con 0x7f54400b4ab0 2023-03-03T07:57:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.445+0000 7f543f7fe700 1 --2- 172.21.15.98:0/412574712 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f542c0682f0 0x7f542c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.445+0000 7f543f7fe700 1 --2- 172.21.15.98:0/412574712 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f542c0682f0 0x7f542c06a7a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5434007fd0 tx=0x7f5434007c40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.448+0000 7f543cff9700 1 -- 172.21.15.98:0/412574712 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5430052190 con 0x7f54400b4ab0 2023-03-03T07:57:01.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:57:01.572 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:57:01.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.701+0000 7f544539d700 1 -- 172.21.15.98:0/412574712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54400c7dc0 con 0x7f54400b4ab0 2023-03-03T07:57:01.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.701+0000 7f543cff9700 1 -- 172.21.15.98:0/412574712 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4639 (secure 0 0 0) 0x7f54300557b0 con 0x7f54400b4ab0 2023-03-03T07:57:01.703 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:01.703 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/246835374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":246835374},{"type":"v1","addr":"172.21.15.98:6836","nonce":246835374}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6833/959944231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":959944231},{"type":"v1","addr":"172.21.15.100:6833","nonce":959944231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/872788798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":872788798},{"type":"v1","addr":"172.21.15.100:6835","nonce":872788798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:55:47.097221+0000","modified":"2023-03-03T07:55:50.555384+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/3969084152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3969084152},{"type":"v1","addr":"172.21.15.98:6837","nonce":3969084152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:57:01.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.704+0000 7f544539d700 1 -- 172.21.15.98:0/412574712 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f542c0682f0 msgr2=0x7f542c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:01.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.704+0000 7f544539d700 1 --2- 172.21.15.98:0/412574712 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f542c0682f0 0x7f542c06a7a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5434007fd0 tx=0x7f5434007c40 comp rx=0 tx=0).stop 2023-03-03T07:57:01.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.704+0000 7f544539d700 1 -- 172.21.15.98:0/412574712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54400b4ab0 msgr2=0x7f5440155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:01.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.704+0000 7f544539d700 1 --2- 172.21.15.98:0/412574712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54400b4ab0 0x7f5440155180 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f543000e760 tx=0x7f543000ea70 comp rx=0 tx=0).stop 2023-03-03T07:57:01.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.704+0000 7f544539d700 1 -- 172.21.15.98:0/412574712 shutdown_connections 2023-03-03T07:57:01.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.704+0000 7f544539d700 1 --2- 172.21.15.98:0/412574712 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f542c0682f0 0x7f542c06a7a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:01.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.704+0000 7f544539d700 1 --2- 172.21.15.98:0/412574712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54400b4ab0 0x7f5440155180 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:01.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.704+0000 7f544539d700 1 --2- 172.21.15.98:0/412574712 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5440146360 0x7f54400cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:01.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.704+0000 7f544539d700 1 --2- 172.21.15.98:0/412574712 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54400d2590 0x7f54400cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:01.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.704+0000 7f544539d700 1 -- 172.21.15.98:0/412574712 >> 172.21.15.98:0/412574712 conn(0x7f544006b8e0 msgr2=0x7f54400b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:01.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.705+0000 7f544539d700 1 -- 172.21.15.98:0/412574712 shutdown_connections 2023-03-03T07:57:01.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:01.705+0000 7f544539d700 1 -- 172.21.15.98:0/412574712 wait complete. 2023-03-03T07:57:01.708 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 7 2023-03-03T07:57:01.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:57:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.015+0000 7f4dcffff700 1 Processor -- start 2023-03-03T07:57:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.015+0000 7f4dcffff700 1 -- start start 2023-03-03T07:57:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.017+0000 7f4dcffff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4dd0146360 0x7f4dd0160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.017+0000 7f4dcffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4dd00b4ab0 0x7f4dd00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.017+0000 7f4dcffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4dd00731e0 0x7f4dd0073660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.017+0000 7f4dcffff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd01614c0 con 0x7f4dd00731e0 2023-03-03T07:57:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.017+0000 7f4dcffff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd00b5440 con 0x7f4dd0146360 2023-03-03T07:57:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.017+0000 7f4dcffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00d10f0 con 0x7f4dd00b4ab0 2023-03-03T07:57:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.017+0000 7f4dcf7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4dd00731e0 0x7f4dd0073660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.017+0000 7f4dceffd700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4dd0146360 0x7f4dd0160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.017+0000 7f4dce7fc700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4dd00b4ab0 0x7f4dd00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.017+0000 7f4dcf7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4dd00731e0 0x7f4dd0073660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45850/0 (socket says 172.21.15.98:45850) 2023-03-03T07:57:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.017+0000 7f4dcf7fe700 1 -- 172.21.15.98:0/1010751897 learned_addr learned my addr 172.21.15.98:0/1010751897 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.017+0000 7f4dceffd700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4dd0146360 0x7f4dd0160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:34878/0 (socket says 172.21.15.98:34878) 2023-03-03T07:57:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.018+0000 7f4dcf7fe700 1 -- 172.21.15.98:0/1010751897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4dd00b4ab0 msgr2=0x7f4dd00b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.018+0000 7f4dcf7fe700 1 --2- 172.21.15.98:0/1010751897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4dd00b4ab0 0x7f4dd00b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.018+0000 7f4dcf7fe700 1 -- 172.21.15.98:0/1010751897 >> v2:172.21.15.100:3300/0 conn(0x7f4dd0146360 msgr2=0x7f4dd0160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.018+0000 7f4dcf7fe700 1 --2- 172.21.15.98:0/1010751897 >> v2:172.21.15.100:3300/0 conn(0x7f4dd0146360 0x7f4dd0160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.018+0000 7f4dcf7fe700 1 -- 172.21.15.98:0/1010751897 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd00cdf20 con 0x7f4dd00731e0 2023-03-03T07:57:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.018+0000 7f4dcf7fe700 1 --2- 172.21.15.98:0/1010751897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dd00731e0 0x7f4dd0073660 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00743f0 tx=0x7f4dc800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a644d36f483e0cf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.018+0000 7f4dcdffb700 1 -- 172.21.15.98:0/1010751897 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc800e070 con 0x7f4dd00731e0 2023-03-03T07:57:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.018+0000 7f4dcdffb700 1 -- 172.21.15.98:0/1010751897 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc800b910 con 0x7f4dd00731e0 2023-03-03T07:57:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.018+0000 7f4dcdffb700 1 -- 172.21.15.98:0/1010751897 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc800bcc0 con 0x7f4dd00731e0 2023-03-03T07:57:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.019+0000 7f4dcffff700 1 -- 172.21.15.98:0/1010751897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dd00731e0 msgr2=0x7f4dd0073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.019+0000 7f4dcffff700 1 --2- 172.21.15.98:0/1010751897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dd00731e0 0x7f4dd0073660 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00743f0 tx=0x7f4dc800b330 comp rx=0 tx=0).stop 2023-03-03T07:57:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.019+0000 7f4dcffff700 1 -- 172.21.15.98:0/1010751897 shutdown_connections 2023-03-03T07:57:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.019+0000 7f4dcffff700 1 --2- 172.21.15.98:0/1010751897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dd00731e0 0x7f4dd0073660 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.019+0000 7f4dcffff700 1 --2- 172.21.15.98:0/1010751897 >> v2:172.21.15.100:3300/0 conn(0x7f4dd0146360 0x7f4dd0160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.019+0000 7f4dcffff700 1 --2- 172.21.15.98:0/1010751897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4dd00b4ab0 0x7f4dd00b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.019+0000 7f4dcffff700 1 -- 172.21.15.98:0/1010751897 >> 172.21.15.98:0/1010751897 conn(0x7f4dd006b8e0 msgr2=0x7f4dd00cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.019+0000 7f4dcffff700 1 -- 172.21.15.98:0/1010751897 shutdown_connections 2023-03-03T07:57:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.019+0000 7f4dcffff700 1 -- 172.21.15.98:0/1010751897 wait complete. 2023-03-03T07:57:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.021+0000 7f4dcffff700 1 Processor -- start 2023-03-03T07:57:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.021+0000 7f4dcffff700 1 -- start start 2023-03-03T07:57:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.022+0000 7f4dcffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4dd00731e0 0x7f4dd00ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.022+0000 7f4dcffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4dd00b4ab0 0x7f4dd00c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.022+0000 7f4dcffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dd0146360 0x7f4dd00ca410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.022+0000 7f4dcffff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00c63d0 con 0x7f4dd0146360 2023-03-03T07:57:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.022+0000 7f4dcffff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd0151bf0 con 0x7f4dd00b4ab0 2023-03-03T07:57:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.022+0000 7f4dcffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00b8550 con 0x7f4dd00731e0 2023-03-03T07:57:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.023+0000 7f4dcf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dd0146360 0x7f4dd00ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.023+0000 7f4dcf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dd0146360 0x7f4dd00ca410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45866/0 (socket says 172.21.15.98:45866) 2023-03-03T07:57:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.023+0000 7f4dceffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4dd00731e0 0x7f4dd00ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.023+0000 7f4dcf7fe700 1 -- 172.21.15.98:0/3320282542 learned_addr learned my addr 172.21.15.98:0/3320282542 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.023+0000 7f4dce7fc700 1 --2- 172.21.15.98:0/3320282542 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4dd00b4ab0 0x7f4dd00c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.023+0000 7f4dcf7fe700 1 -- 172.21.15.98:0/3320282542 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4dd00731e0 msgr2=0x7f4dd00ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.023+0000 7f4dcf7fe700 1 --2- 172.21.15.98:0/3320282542 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4dd00731e0 0x7f4dd00ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.023+0000 7f4dcf7fe700 1 -- 172.21.15.98:0/3320282542 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4dd00b4ab0 msgr2=0x7f4dd00c5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.023+0000 7f4dcf7fe700 1 --2- 172.21.15.98:0/3320282542 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4dd00b4ab0 0x7f4dd00c5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.023+0000 7f4dcf7fe700 1 -- 172.21.15.98:0/3320282542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc8009b30 con 0x7f4dd0146360 2023-03-03T07:57:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.024+0000 7f4dcf7fe700 1 --2- 172.21.15.98:0/3320282542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dd0146360 0x7f4dd00ca410 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00747b0 tx=0x7f4dc8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.024+0000 7f4db7fff700 1 -- 172.21.15.98:0/3320282542 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc80099b0 con 0x7f4dd0146360 2023-03-03T07:57:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.024+0000 7f4db7fff700 1 -- 172.21.15.98:0/3320282542 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc800ce80 con 0x7f4dd0146360 2023-03-03T07:57:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.024+0000 7f4db7fff700 1 -- 172.21.15.98:0/3320282542 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8004490 con 0x7f4dd0146360 2023-03-03T07:57:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.024+0000 7f4dcffff700 1 -- 172.21.15.98:0/3320282542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd00cd960 con 0x7f4dd0146360 2023-03-03T07:57:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.024+0000 7f4dcffff700 1 -- 172.21.15.98:0/3320282542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd00cd680 con 0x7f4dd0146360 2023-03-03T07:57:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.025+0000 7f4dcffff700 1 -- 172.21.15.98:0/3320282542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd00cdc40 con 0x7f4dd0146360 2023-03-03T07:57:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.026+0000 7f4db7fff700 1 -- 172.21.15.98:0/3320282542 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4dc80045f0 con 0x7f4dd0146360 2023-03-03T07:57:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.028+0000 7f4db7fff700 1 --2- 172.21.15.98:0/3320282542 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4dbc0682f0 0x7f4dbc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.028+0000 7f4db7fff700 1 -- 172.21.15.98:0/3320282542 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f4dc8086e60 con 0x7f4dd0146360 2023-03-03T07:57:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.029+0000 7f4dceffd700 1 --2- 172.21.15.98:0/3320282542 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4dbc0682f0 0x7f4dbc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.029+0000 7f4dceffd700 1 --2- 172.21.15.98:0/3320282542 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4dbc0682f0 0x7f4dbc06a7a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0009ac0 tx=0x7f4dc0009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.033+0000 7f4db7fff700 1 -- 172.21.15.98:0/3320282542 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4dc8053590 con 0x7f4dd0146360 2023-03-03T07:57:02.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.227+0000 7f4dcffff700 1 -- 172.21.15.98:0/3320282542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4dd00cefa0 con 0x7f4dd0146360 2023-03-03T07:57:02.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.228+0000 7f4db7fff700 1 -- 172.21.15.98:0/3320282542 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18003 (secure 0 0 0) 0x7f4dc801f090 con 0x7f4dd0146360 2023-03-03T07:57:02.230 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:02.230 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:56:27.498205+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:57:02.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.231+0000 7f4dcffff700 1 -- 172.21.15.98:0/3320282542 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4dbc0682f0 msgr2=0x7f4dbc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:02.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.231+0000 7f4dcffff700 1 --2- 172.21.15.98:0/3320282542 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4dbc0682f0 0x7f4dbc06a7a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0009ac0 tx=0x7f4dc0009210 comp rx=0 tx=0).stop 2023-03-03T07:57:02.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.231+0000 7f4dcffff700 1 -- 172.21.15.98:0/3320282542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dd0146360 msgr2=0x7f4dd00ca410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:02.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.231+0000 7f4dcffff700 1 --2- 172.21.15.98:0/3320282542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dd0146360 0x7f4dd00ca410 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00747b0 tx=0x7f4dc8004ad0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.231+0000 7f4dcffff700 1 -- 172.21.15.98:0/3320282542 shutdown_connections 2023-03-03T07:57:02.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.231+0000 7f4dcffff700 1 --2- 172.21.15.98:0/3320282542 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4dbc0682f0 0x7f4dbc06a7a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.231+0000 7f4dcffff700 1 --2- 172.21.15.98:0/3320282542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dd0146360 0x7f4dd00ca410 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.231+0000 7f4dcffff700 1 --2- 172.21.15.98:0/3320282542 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4dd00731e0 0x7f4dd00ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.231+0000 7f4dcffff700 1 --2- 172.21.15.98:0/3320282542 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4dd00b4ab0 0x7f4dd00c5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.231+0000 7f4dcffff700 1 -- 172.21.15.98:0/3320282542 >> 172.21.15.98:0/3320282542 conn(0x7f4dd006b8e0 msgr2=0x7f4dd00735b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:02.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.231+0000 7f4dcffff700 1 -- 172.21.15.98:0/3320282542 shutdown_connections 2023-03-03T07:57:02.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.231+0000 7f4dcffff700 1 -- 172.21.15.98:0/3320282542 wait complete. 2023-03-03T07:57:02.260 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-03-03T07:57:02.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:57:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.505+0000 7f8de43ff700 1 Processor -- start 2023-03-03T07:57:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.505+0000 7f8de43ff700 1 -- start start 2023-03-03T07:57:02.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.506+0000 7f8de43ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8ddc15cc00 0x7f8ddc0cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.506+0000 7f8de43ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8ddc0d27c0 0x7f8ddc0d2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.506+0000 7f8de43ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ddc0b60d0 0x7f8ddc0b6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.506+0000 7f8de43ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ddc0ce040 con 0x7f8ddc15cc00 2023-03-03T07:57:02.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.506+0000 7f8de43ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ddc073be0 con 0x7f8ddc0d27c0 2023-03-03T07:57:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.506+0000 7f8de43ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc0cde10 con 0x7f8ddc0b60d0 2023-03-03T07:57:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.507+0000 7f8de219b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8ddc15cc00 0x7f8ddc0cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.507+0000 7f8de219b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8ddc15cc00 0x7f8ddc0cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45870/0 (socket says 172.21.15.98:45870) 2023-03-03T07:57:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.507+0000 7f8de219b700 1 -- 172.21.15.98:0/2314871811 learned_addr learned my addr 172.21.15.98:0/2314871811 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.507+0000 7f8de299c700 1 --2- 172.21.15.98:0/2314871811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ddc0b60d0 0x7f8ddc0b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.507+0000 7f8de199a700 1 --2- 172.21.15.98:0/2314871811 >> v2:172.21.15.100:3300/0 conn(0x7f8ddc0d27c0 0x7f8ddc0d2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.507+0000 7f8de219b700 1 -- 172.21.15.98:0/2314871811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ddc0b60d0 msgr2=0x7f8ddc0b6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.507+0000 7f8de219b700 1 --2- 172.21.15.98:0/2314871811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ddc0b60d0 0x7f8ddc0b6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.507+0000 7f8de219b700 1 -- 172.21.15.98:0/2314871811 >> v2:172.21.15.100:3300/0 conn(0x7f8ddc0d27c0 msgr2=0x7f8ddc0d2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.507+0000 7f8de219b700 1 --2- 172.21.15.98:0/2314871811 >> v2:172.21.15.100:3300/0 conn(0x7f8ddc0d27c0 0x7f8ddc0d2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.507+0000 7f8de219b700 1 -- 172.21.15.98:0/2314871811 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc0d1a00 con 0x7f8ddc15cc00 2023-03-03T07:57:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.507+0000 7f8de199a700 1 --2- 172.21.15.98:0/2314871811 >> v2:172.21.15.100:3300/0 conn(0x7f8ddc0d27c0 0x7f8ddc0d2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.508+0000 7f8de219b700 1 --2- 172.21.15.98:0/2314871811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ddc15cc00 0x7f8ddc0cd8d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc0b5e70 tx=0x7f8dd8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a041e31f0a473589 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.508+0000 7f8de1199700 1 -- 172.21.15.98:0/2314871811 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd800b870 con 0x7f8ddc15cc00 2023-03-03T07:57:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.508+0000 7f8de1199700 1 -- 172.21.15.98:0/2314871811 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd800b9d0 con 0x7f8ddc15cc00 2023-03-03T07:57:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.508+0000 7f8de1199700 1 -- 172.21.15.98:0/2314871811 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd800bd60 con 0x7f8ddc15cc00 2023-03-03T07:57:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.508+0000 7f8de43ff700 1 -- 172.21.15.98:0/2314871811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ddc15cc00 msgr2=0x7f8ddc0cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:02.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.508+0000 7f8de43ff700 1 --2- 172.21.15.98:0/2314871811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ddc15cc00 0x7f8ddc0cd8d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc0b5e70 tx=0x7f8dd8009bd0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.509+0000 7f8de43ff700 1 -- 172.21.15.98:0/2314871811 shutdown_connections 2023-03-03T07:57:02.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.509+0000 7f8de43ff700 1 --2- 172.21.15.98:0/2314871811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ddc0b60d0 0x7f8ddc0b6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.509+0000 7f8de43ff700 1 --2- 172.21.15.98:0/2314871811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ddc15cc00 0x7f8ddc0cd8d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.509+0000 7f8de43ff700 1 --2- 172.21.15.98:0/2314871811 >> v2:172.21.15.100:3300/0 conn(0x7f8ddc0d27c0 0x7f8ddc0d2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.509+0000 7f8de43ff700 1 -- 172.21.15.98:0/2314871811 >> 172.21.15.98:0/2314871811 conn(0x7f8ddc06aee0 msgr2=0x7f8ddc06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.509+0000 7f8de43ff700 1 -- 172.21.15.98:0/2314871811 shutdown_connections 2023-03-03T07:57:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.509+0000 7f8de43ff700 1 -- 172.21.15.98:0/2314871811 wait complete. 2023-03-03T07:57:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.511+0000 7f8de43ff700 1 Processor -- start 2023-03-03T07:57:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.511+0000 7f8de43ff700 1 -- start start 2023-03-03T07:57:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.512+0000 7f8de43ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ddc0b60d0 0x7f8ddc15f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.512+0000 7f8de43ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ddc0d27c0 0x7f8ddc15faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.512+0000 7f8de43ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8ddc15cc00 0x7f8ddc0ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.512+0000 7f8de43ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc1f5390 con 0x7f8ddc0d27c0 2023-03-03T07:57:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.512+0000 7f8de43ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc1f5f10 con 0x7f8ddc15cc00 2023-03-03T07:57:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.512+0000 7f8de43ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc1f55e0 con 0x7f8ddc0b60d0 2023-03-03T07:57:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.512+0000 7f8de199a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ddc0d27c0 0x7f8ddc15faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:02.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.512+0000 7f8de199a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ddc0d27c0 0x7f8ddc15faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45874/0 (socket says 172.21.15.98:45874) 2023-03-03T07:57:02.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.512+0000 7f8de199a700 1 -- 172.21.15.98:0/4252886193 learned_addr learned my addr 172.21.15.98:0/4252886193 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:02.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.513+0000 7f8de219b700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ddc0b60d0 0x7f8ddc15f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:02.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.513+0000 7f8de299c700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8ddc15cc00 0x7f8ddc0ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:02.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.513+0000 7f8de199a700 1 -- 172.21.15.98:0/4252886193 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ddc0b60d0 msgr2=0x7f8ddc15f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:02.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.513+0000 7f8de199a700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ddc0b60d0 0x7f8ddc15f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.513+0000 7f8de199a700 1 -- 172.21.15.98:0/4252886193 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8ddc15cc00 msgr2=0x7f8ddc0ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:02.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.513+0000 7f8de199a700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8ddc15cc00 0x7f8ddc0ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.513+0000 7f8de199a700 1 -- 172.21.15.98:0/4252886193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd8009550 con 0x7f8ddc0d27c0 2023-03-03T07:57:02.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.513+0000 7f8de299c700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8ddc15cc00 0x7f8ddc0ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:57:02.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.513+0000 7f8de219b700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ddc0b60d0 0x7f8ddc15f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:57:02.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.513+0000 7f8de199a700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ddc0d27c0 0x7f8ddc15faa0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00e840 tx=0x7f8dcc00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:02.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.513+0000 7f8dd2ffd700 1 -- 172.21.15.98:0/4252886193 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc004230 con 0x7f8ddc0d27c0 2023-03-03T07:57:02.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.514+0000 7f8dd2ffd700 1 -- 172.21.15.98:0/4252886193 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dcc004390 con 0x7f8ddc0d27c0 2023-03-03T07:57:02.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.514+0000 7f8dd2ffd700 1 -- 172.21.15.98:0/4252886193 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc0036a0 con 0x7f8ddc0d27c0 2023-03-03T07:57:02.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.514+0000 7f8de43ff700 1 -- 172.21.15.98:0/4252886193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ddc0d1fc0 con 0x7f8ddc0d27c0 2023-03-03T07:57:02.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.514+0000 7f8de43ff700 1 -- 172.21.15.98:0/4252886193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ddc0d1ce0 con 0x7f8ddc0d27c0 2023-03-03T07:57:02.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.515+0000 7f8de43ff700 1 -- 172.21.15.98:0/4252886193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ddc0d22a0 con 0x7f8ddc0d27c0 2023-03-03T07:57:02.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.516+0000 7f8dd2ffd700 1 -- 172.21.15.98:0/4252886193 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8dcc022070 con 0x7f8ddc0d27c0 2023-03-03T07:57:02.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.517+0000 7f8dd2ffd700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8dc00682f0 0x7f8dc006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:02.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.517+0000 7f8dd2ffd700 1 -- 172.21.15.98:0/4252886193 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f8dcc085830 con 0x7f8ddc0d27c0 2023-03-03T07:57:02.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.518+0000 7f8de219b700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8dc00682f0 0x7f8dc006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:02.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.518+0000 7f8de219b700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8dc00682f0 0x7f8dc006a7a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc153720 tx=0x7f8dd8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:02.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.521+0000 7f8dd2ffd700 1 -- 172.21.15.98:0/4252886193 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8dcc051f60 con 0x7f8ddc0d27c0 2023-03-03T07:57:02.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.760+0000 7f8de43ff700 1 -- 172.21.15.98:0/4252886193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ddc0729b0 con 0x7f8ddc0d27c0 2023-03-03T07:57:02.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.761+0000 7f8dd2ffd700 1 -- 172.21.15.98:0/4252886193 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4639 (secure 0 0 0) 0x7f8dcc0094c0 con 0x7f8ddc0d27c0 2023-03-03T07:57:02.763 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:02.763 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/246835374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":246835374},{"type":"v1","addr":"172.21.15.98:6836","nonce":246835374}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6833/959944231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":959944231},{"type":"v1","addr":"172.21.15.100:6833","nonce":959944231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/872788798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":872788798},{"type":"v1","addr":"172.21.15.100:6835","nonce":872788798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:55:47.097221+0000","modified":"2023-03-03T07:55:50.555384+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/3969084152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3969084152},{"type":"v1","addr":"172.21.15.98:6837","nonce":3969084152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:57:02.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.763+0000 7f8de43ff700 1 -- 172.21.15.98:0/4252886193 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8dc00682f0 msgr2=0x7f8dc006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:02.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.763+0000 7f8de43ff700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8dc00682f0 0x7f8dc006a7a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc153720 tx=0x7f8dd8008030 comp rx=0 tx=0).stop 2023-03-03T07:57:02.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.763+0000 7f8de43ff700 1 -- 172.21.15.98:0/4252886193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ddc0d27c0 msgr2=0x7f8ddc15faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:02.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.763+0000 7f8de43ff700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ddc0d27c0 0x7f8ddc15faa0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00e840 tx=0x7f8dcc00ec00 comp rx=0 tx=0).stop 2023-03-03T07:57:02.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.763+0000 7f8de43ff700 1 -- 172.21.15.98:0/4252886193 shutdown_connections 2023-03-03T07:57:02.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.763+0000 7f8de43ff700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8dc00682f0 0x7f8dc006a7a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.763+0000 7f8de43ff700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8ddc15cc00 0x7f8ddc0ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.763+0000 7f8de43ff700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ddc0b60d0 0x7f8ddc15f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.763+0000 7f8de43ff700 1 --2- 172.21.15.98:0/4252886193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ddc0d27c0 0x7f8ddc15faa0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:02.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.763+0000 7f8de43ff700 1 -- 172.21.15.98:0/4252886193 >> 172.21.15.98:0/4252886193 conn(0x7f8ddc06aee0 msgr2=0x7f8ddc0b69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:02.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.763+0000 7f8de43ff700 1 -- 172.21.15.98:0/4252886193 shutdown_connections 2023-03-03T07:57:02.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:02.763+0000 7f8de43ff700 1 -- 172.21.15.98:0/4252886193 wait complete. 2023-03-03T07:57:02.767 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 7 2023-03-03T07:57:02.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:57:03.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.037+0000 7f78fefff700 1 Processor -- start 2023-03-03T07:57:03.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.037+0000 7f78fefff700 1 -- start start 2023-03-03T07:57:03.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.038+0000 7f78fefff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78f80b8b90 0x7f78f80cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.038+0000 7f78fefff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f78f80b5410 0x7f78f80d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.039+0000 7f78fefff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f80d0ef0 con 0x7f78f80b87c0 2023-03-03T07:57:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.039+0000 7f78fefff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78f80d0600 con 0x7f78f80b5410 2023-03-03T07:57:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.039+0000 7f78fefff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f80cd970 con 0x7f78f80b8b90 2023-03-03T07:57:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.039+0000 7f78fcd9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f78f80b87c0 0x7f78f80b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34380/0 (socket says 172.21.15.98:34380) 2023-03-03T07:57:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.039+0000 7f78fcd9b700 1 -- 172.21.15.98:0/1353888424 learned_addr learned my addr 172.21.15.98:0/1353888424 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.039+0000 7f78f7fff700 1 --2- 172.21.15.98:0/1353888424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78f80b8b90 0x7f78f80cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.039+0000 7f78f77fe700 1 -- 172.21.15.98:0/1353888424 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1575618610 0 0) 0x7f78f80d0ef0 con 0x7f78f80b87c0 2023-03-03T07:57:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.039+0000 7f78fd59c700 1 --2- 172.21.15.98:0/1353888424 >> v2:172.21.15.100:3300/0 conn(0x7f78f80b5410 0x7f78f80d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.039+0000 7f78f77fe700 1 -- 172.21.15.98:0/1353888424 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78e80035b0 con 0x7f78f80b87c0 2023-03-03T07:57:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78f77fe700 1 -- 172.21.15.98:0/1353888424 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4288823344 0 0) 0x7f78e80035b0 con 0x7f78f80b87c0 2023-03-03T07:57:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78f77fe700 1 -- 172.21.15.98:0/1353888424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78f80b8b90 msgr2=0x7f78f80cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78f77fe700 1 --2- 172.21.15.98:0/1353888424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78f80b8b90 0x7f78f80cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78f77fe700 1 -- 172.21.15.98:0/1353888424 >> v2:172.21.15.100:3300/0 conn(0x7f78f80b5410 msgr2=0x7f78f80d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78f77fe700 1 --2- 172.21.15.98:0/1353888424 >> v2:172.21.15.100:3300/0 conn(0x7f78f80b5410 0x7f78f80d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78f77fe700 1 -- 172.21.15.98:0/1353888424 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f80cdc50 con 0x7f78f80b87c0 2023-03-03T07:57:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78f77fe700 1 -- 172.21.15.98:0/1353888424 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f78ec001670 con 0x7f78f80b87c0 2023-03-03T07:57:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78f77fe700 1 -- 172.21.15.98:0/1353888424 >> v1:172.21.15.98:6789/0 conn(0x7f78f80b87c0 legacy=0x7f78f80b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:03.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78f77fe700 1 --2- 172.21.15.98:0/1353888424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e8003a00 0x7f78e8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:03.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78f77fe700 1 --2- 172.21.15.98:0/1353888424 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78e8004850 0x7f78e8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:03.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78f77fe700 1 --2- 172.21.15.98:0/1353888424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78f80b8b90 0x7f78e8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:03.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78f77fe700 1 -- 172.21.15.98:0/1353888424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f80d0600 con 0x7f78e8003a00 2023-03-03T07:57:03.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78f77fe700 1 -- 172.21.15.98:0/1353888424 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f80cd970 con 0x7f78e8004850 2023-03-03T07:57:03.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78f77fe700 1 -- 172.21.15.98:0/1353888424 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e800ba10 con 0x7f78f80b8b90 2023-03-03T07:57:03.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78fcd9b700 1 --2- 172.21.15.98:0/1353888424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e8003a00 0x7f78e8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:03.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78fd59c700 1 --2- 172.21.15.98:0/1353888424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78f80b8b90 0x7f78e8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:03.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.040+0000 7f78f7fff700 1 --2- 172.21.15.98:0/1353888424 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78e8004850 0x7f78e8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:03.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.041+0000 7f78fcd9b700 1 -- 172.21.15.98:0/1353888424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78f80b8b90 msgr2=0x7f78e8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:03.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.041+0000 7f78fcd9b700 1 --2- 172.21.15.98:0/1353888424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78f80b8b90 0x7f78e8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.041+0000 7f78fcd9b700 1 -- 172.21.15.98:0/1353888424 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78e8004850 msgr2=0x7f78e8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:03.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.041+0000 7f78fcd9b700 1 --2- 172.21.15.98:0/1353888424 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78e8004850 0x7f78e8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.041+0000 7f78fcd9b700 1 -- 172.21.15.98:0/1353888424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e80035b0 con 0x7f78e8003a00 2023-03-03T07:57:03.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.041+0000 7f78fcd9b700 1 --2- 172.21.15.98:0/1353888424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e8003a00 0x7f78e8003e10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f78ec00c240 tx=0x7f78e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ceb45268065726a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:03.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.041+0000 7f78f77fe700 1 -- 172.21.15.98:0/1353888424 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ec00cd40 con 0x7f78e8003a00 2023-03-03T07:57:03.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.041+0000 7f78f77fe700 1 -- 172.21.15.98:0/1353888424 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ec005960 con 0x7f78e8003a00 2023-03-03T07:57:03.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.041+0000 7f78f77fe700 1 -- 172.21.15.98:0/1353888424 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ec005d10 con 0x7f78e8003a00 2023-03-03T07:57:03.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.042+0000 7f78fefff700 1 -- 172.21.15.98:0/1353888424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e8003a00 msgr2=0x7f78e8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:03.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.042+0000 7f78fefff700 1 --2- 172.21.15.98:0/1353888424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e8003a00 0x7f78e8003e10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f78ec00c240 tx=0x7f78e8005880 comp rx=0 tx=0).stop 2023-03-03T07:57:03.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.042+0000 7f78fd59c700 1 -- 172.21.15.98:0/1353888424 reap_dead start 2023-03-03T07:57:03.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.042+0000 7f78fefff700 1 -- 172.21.15.98:0/1353888424 shutdown_connections 2023-03-03T07:57:03.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.042+0000 7f78fefff700 1 -- 172.21.15.98:0/1353888424 >> 172.21.15.98:0/1353888424 conn(0x7f78f806aee0 msgr2=0x7f78f806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:03.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.042+0000 7f78fefff700 1 -- 172.21.15.98:0/1353888424 shutdown_connections 2023-03-03T07:57:03.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.042+0000 7f78fefff700 1 -- 172.21.15.98:0/1353888424 wait complete. 2023-03-03T07:57:03.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.044+0000 7f78fefff700 1 Processor -- start 2023-03-03T07:57:03.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.045+0000 7f78fefff700 1 -- start start 2023-03-03T07:57:03.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.046+0000 7f78fefff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78e8003a00 0x7f78f815ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:03.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.046+0000 7f78fefff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78f815f790 0x7f78f81f2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:03.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.046+0000 7f78fefff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f81f2b50 0x7f78f80c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:03.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.046+0000 7f78fefff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f815e0f0 con 0x7f78f81f2b50 2023-03-03T07:57:03.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.046+0000 7f78fefff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f815d470 con 0x7f78e8003a00 2023-03-03T07:57:03.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.046+0000 7f78fefff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f815d750 con 0x7f78f815f790 2023-03-03T07:57:03.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.046+0000 7f78fd59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f81f2b50 0x7f78f80c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:03.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.046+0000 7f78fd59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f81f2b50 0x7f78f80c8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45886/0 (socket says 172.21.15.98:45886) 2023-03-03T07:57:03.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.046+0000 7f78fd59c700 1 -- 172.21.15.98:0/195685703 learned_addr learned my addr 172.21.15.98:0/195685703 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:03.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.046+0000 7f78f7fff700 1 --2- 172.21.15.98:0/195685703 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78f815f790 0x7f78f81f2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:03.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.046+0000 7f78fcd9b700 1 --2- 172.21.15.98:0/195685703 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78e8003a00 0x7f78f815ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:03.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.047+0000 7f78fd59c700 1 -- 172.21.15.98:0/195685703 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78f815f790 msgr2=0x7f78f81f2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:03.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.047+0000 7f78fd59c700 1 --2- 172.21.15.98:0/195685703 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78f815f790 0x7f78f81f2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.047+0000 7f78fd59c700 1 -- 172.21.15.98:0/195685703 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78e8003a00 msgr2=0x7f78f815ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:03.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.047+0000 7f78fd59c700 1 --2- 172.21.15.98:0/195685703 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78e8003a00 0x7f78f815ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.047+0000 7f78fd59c700 1 -- 172.21.15.98:0/195685703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f80cdf30 con 0x7f78f81f2b50 2023-03-03T07:57:03.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.047+0000 7f78fcd9b700 1 --2- 172.21.15.98:0/195685703 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78e8003a00 0x7f78f815ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:03.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.047+0000 7f78fd59c700 1 --2- 172.21.15.98:0/195685703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f81f2b50 0x7f78f80c8e20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f78f80b5e70 tx=0x7f78f00114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:03.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.047+0000 7f78f57fa700 1 -- 172.21.15.98:0/195685703 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f000e070 con 0x7f78f81f2b50 2023-03-03T07:57:03.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.047+0000 7f78f57fa700 1 -- 172.21.15.98:0/195685703 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f0003cf0 con 0x7f78f81f2b50 2023-03-03T07:57:03.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.047+0000 7f78f57fa700 1 -- 172.21.15.98:0/195685703 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f0003ea0 con 0x7f78f81f2b50 2023-03-03T07:57:03.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.047+0000 7f78fefff700 1 -- 172.21.15.98:0/195685703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f8073a80 con 0x7f78f81f2b50 2023-03-03T07:57:03.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.047+0000 7f78fefff700 1 -- 172.21.15.98:0/195685703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f80ce0c0 con 0x7f78f81f2b50 2023-03-03T07:57:03.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.049+0000 7f78fefff700 1 -- 172.21.15.98:0/195685703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f80729b0 con 0x7f78f81f2b50 2023-03-03T07:57:03.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.050+0000 7f78f57fa700 1 -- 172.21.15.98:0/195685703 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f78f0011a10 con 0x7f78f81f2b50 2023-03-03T07:57:03.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.052+0000 7f78f57fa700 1 --2- 172.21.15.98:0/195685703 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f78dc0682f0 0x7f78dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:03.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.052+0000 7f78f57fa700 1 -- 172.21.15.98:0/195685703 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f78f00863c0 con 0x7f78f81f2b50 2023-03-03T07:57:03.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.052+0000 7f78fcd9b700 1 --2- 172.21.15.98:0/195685703 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f78dc0682f0 0x7f78dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:03.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.053+0000 7f78fcd9b700 1 --2- 172.21.15.98:0/195685703 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f78dc0682f0 0x7f78dc06a7a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f78ec007fd0 tx=0x7f78ec00a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:03.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.056+0000 7f78f57fa700 1 -- 172.21.15.98:0/195685703 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f78f0052af0 con 0x7f78f81f2b50 2023-03-03T07:57:03.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.291+0000 7f78fefff700 1 -- 172.21.15.98:0/195685703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78f815e230 con 0x7f78f81f2b50 2023-03-03T07:57:03.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.292+0000 7f78f57fa700 1 -- 172.21.15.98:0/195685703 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4639 (secure 0 0 0) 0x7f78f815e230 con 0x7f78f81f2b50 2023-03-03T07:57:03.294 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:03.294 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/246835374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":246835374},{"type":"v1","addr":"172.21.15.98:6836","nonce":246835374}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6833/959944231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":959944231},{"type":"v1","addr":"172.21.15.100:6833","nonce":959944231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/872788798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":872788798},{"type":"v1","addr":"172.21.15.100:6835","nonce":872788798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:55:47.097221+0000","modified":"2023-03-03T07:55:50.555384+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/3969084152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3969084152},{"type":"v1","addr":"172.21.15.98:6837","nonce":3969084152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:57:03.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.295+0000 7f78fefff700 1 -- 172.21.15.98:0/195685703 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f78dc0682f0 msgr2=0x7f78dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:03.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.295+0000 7f78fefff700 1 --2- 172.21.15.98:0/195685703 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f78dc0682f0 0x7f78dc06a7a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f78ec007fd0 tx=0x7f78ec00a000 comp rx=0 tx=0).stop 2023-03-03T07:57:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.295+0000 7f78fefff700 1 -- 172.21.15.98:0/195685703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f81f2b50 msgr2=0x7f78f80c8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.295+0000 7f78fefff700 1 --2- 172.21.15.98:0/195685703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f81f2b50 0x7f78f80c8e20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f78f80b5e70 tx=0x7f78f00114b0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.295+0000 7f78fefff700 1 -- 172.21.15.98:0/195685703 shutdown_connections 2023-03-03T07:57:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.295+0000 7f78fefff700 1 --2- 172.21.15.98:0/195685703 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f78dc0682f0 0x7f78dc06a7a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.295+0000 7f78fefff700 1 --2- 172.21.15.98:0/195685703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f81f2b50 0x7f78f80c8e20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.295+0000 7f78fefff700 1 --2- 172.21.15.98:0/195685703 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78e8003a00 0x7f78f815ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.295+0000 7f78fefff700 1 --2- 172.21.15.98:0/195685703 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78f815f790 0x7f78f81f2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.295+0000 7f78fefff700 1 -- 172.21.15.98:0/195685703 >> 172.21.15.98:0/195685703 conn(0x7f78f806aee0 msgr2=0x7f78f80b7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:03.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.295+0000 7f78fefff700 1 -- 172.21.15.98:0/195685703 shutdown_connections 2023-03-03T07:57:03.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.295+0000 7f78fefff700 1 -- 172.21.15.98:0/195685703 wait complete. 2023-03-03T07:57:03.298 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 7 2023-03-03T07:57:03.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:57:03.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.566+0000 7f6a887ff700 1 Processor -- start 2023-03-03T07:57:03.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.566+0000 7f6a887ff700 1 -- start start 2023-03-03T07:57:03.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.567+0000 7f6a887ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a800d23e0 0x7f6a800cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:03.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.567+0000 7f6a887ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a800b5b20 con 0x7f6a800b87c0 2023-03-03T07:57:03.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.567+0000 7f6a887ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a800d0ef0 con 0x7f6a800b8b90 2023-03-03T07:57:03.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.567+0000 7f6a887ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a800d0600 con 0x7f6a800d23e0 2023-03-03T07:57:03.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.568+0000 7f6a8659b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6a800b87c0 0x7f6a800b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34386/0 (socket says 172.21.15.98:34386) 2023-03-03T07:57:03.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.568+0000 7f6a8659b700 1 -- 172.21.15.98:0/804276897 learned_addr learned my addr 172.21.15.98:0/804276897 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:03.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.568+0000 7f6a86d9c700 1 --2- 172.21.15.98:0/804276897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a800d23e0 0x7f6a800cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:03.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.568+0000 7f6a85599700 1 -- 172.21.15.98:0/804276897 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669502132 0 0) 0x7f6a800b5b20 con 0x7f6a800b87c0 2023-03-03T07:57:03.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.568+0000 7f6a85599700 1 -- 172.21.15.98:0/804276897 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a6c0035b0 con 0x7f6a800b87c0 2023-03-03T07:57:03.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.568+0000 7f6a85599700 1 -- 172.21.15.98:0/804276897 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944865632 0 0) 0x7f6a800d0ef0 con 0x7f6a800b8b90 2023-03-03T07:57:03.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.568+0000 7f6a85599700 1 -- 172.21.15.98:0/804276897 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a800b5b20 con 0x7f6a800b8b90 2023-03-03T07:57:03.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.568+0000 7f6a86d9c700 1 -- 172.21.15.98:0/804276897 >> v1:172.21.15.100:6789/0 conn(0x7f6a800b8b90 legacy=0x7f6a800b5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:03.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.568+0000 7f6a86d9c700 1 -- 172.21.15.98:0/804276897 >> v1:172.21.15.98:6789/0 conn(0x7f6a800b87c0 legacy=0x7f6a800b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:03.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.568+0000 7f6a86d9c700 1 -- 172.21.15.98:0/804276897 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a800cd970 con 0x7f6a800d23e0 2023-03-03T07:57:03.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.569+0000 7f6a86d9c700 1 --2- 172.21.15.98:0/804276897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a800d23e0 0x7f6a800cffe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009dc0 tx=0x7f6a7800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5c647b7e20658c6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:03.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.569+0000 7f6a85599700 1 -- 172.21.15.98:0/804276897 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7800e070 con 0x7f6a800d23e0 2023-03-03T07:57:03.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.569+0000 7f6a85599700 1 -- 172.21.15.98:0/804276897 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7800bb20 con 0x7f6a800d23e0 2023-03-03T07:57:03.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.569+0000 7f6a85599700 1 -- 172.21.15.98:0/804276897 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a78012430 con 0x7f6a800d23e0 2023-03-03T07:57:03.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.570+0000 7f6a887ff700 1 -- 172.21.15.98:0/804276897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a800d23e0 msgr2=0x7f6a800cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:03.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.570+0000 7f6a887ff700 1 --2- 172.21.15.98:0/804276897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a800d23e0 0x7f6a800cffe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009dc0 tx=0x7f6a7800b330 comp rx=0 tx=0).stop 2023-03-03T07:57:03.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.570+0000 7f6a887ff700 1 -- 172.21.15.98:0/804276897 shutdown_connections 2023-03-03T07:57:03.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.570+0000 7f6a887ff700 1 --2- 172.21.15.98:0/804276897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a800d23e0 0x7f6a800cffe0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.570+0000 7f6a887ff700 1 -- 172.21.15.98:0/804276897 >> 172.21.15.98:0/804276897 conn(0x7f6a8006aee0 msgr2=0x7f6a8006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:03.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.570+0000 7f6a887ff700 1 -- 172.21.15.98:0/804276897 shutdown_connections 2023-03-03T07:57:03.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.570+0000 7f6a887ff700 1 -- 172.21.15.98:0/804276897 wait complete. 2023-03-03T07:57:03.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.573+0000 7f6a887ff700 1 Processor -- start 2023-03-03T07:57:03.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.573+0000 7f6a887ff700 1 -- start start 2023-03-03T07:57:03.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.574+0000 7f6a887ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a800b87c0 0x7f6a801480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:03.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.574+0000 7f6a887ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a800b8b90 0x7f6a801477f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:03.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.574+0000 7f6a887ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a800d23e0 0x7f6a8015c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:03.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.574+0000 7f6a887ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8015d1a0 con 0x7f6a800b8b90 2023-03-03T07:57:03.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.574+0000 7f6a887ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8015d480 con 0x7f6a800b87c0 2023-03-03T07:57:03.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.574+0000 7f6a887ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8015d760 con 0x7f6a800d23e0 2023-03-03T07:57:03.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.575+0000 7f6a85d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a800b8b90 0x7f6a801477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:03.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.575+0000 7f6a85d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a800b8b90 0x7f6a801477f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45888/0 (socket says 172.21.15.98:45888) 2023-03-03T07:57:03.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.575+0000 7f6a85d9a700 1 -- 172.21.15.98:0/3301999225 learned_addr learned my addr 172.21.15.98:0/3301999225 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:03.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.575+0000 7f6a8659b700 1 --2- 172.21.15.98:0/3301999225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a800b87c0 0x7f6a801480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:03.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.575+0000 7f6a86d9c700 1 --2- 172.21.15.98:0/3301999225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a800d23e0 0x7f6a8015c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:03.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.575+0000 7f6a85d9a700 1 -- 172.21.15.98:0/3301999225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a800d23e0 msgr2=0x7f6a8015c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:03.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.575+0000 7f6a85d9a700 1 --2- 172.21.15.98:0/3301999225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a800d23e0 0x7f6a8015c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.575+0000 7f6a85d9a700 1 -- 172.21.15.98:0/3301999225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a800b87c0 msgr2=0x7f6a801480b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.575+0000 7f6a85d9a700 1 --2- 172.21.15.98:0/3301999225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a800b87c0 0x7f6a801480b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.575+0000 7f6a85d9a700 1 -- 172.21.15.98:0/3301999225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a78009a70 con 0x7f6a800b8b90 2023-03-03T07:57:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.575+0000 7f6a85d9a700 1 --2- 172.21.15.98:0/3301999225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a800b8b90 0x7f6a801477f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000c4f0 tx=0x7f6a7000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.576+0000 7f6a76ffd700 1 -- 172.21.15.98:0/3301999225 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a700052d0 con 0x7f6a800b8b90 2023-03-03T07:57:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.576+0000 7f6a76ffd700 1 -- 172.21.15.98:0/3301999225 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a70005430 con 0x7f6a800b8b90 2023-03-03T07:57:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.576+0000 7f6a76ffd700 1 -- 172.21.15.98:0/3301999225 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a70003880 con 0x7f6a800b8b90 2023-03-03T07:57:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.576+0000 7f6a887ff700 1 -- 172.21.15.98:0/3301999225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a80073a80 con 0x7f6a800b8b90 2023-03-03T07:57:03.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.576+0000 7f6a887ff700 1 -- 172.21.15.98:0/3301999225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a800cdc50 con 0x7f6a800b8b90 2023-03-03T07:57:03.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.577+0000 7f6a887ff700 1 -- 172.21.15.98:0/3301999225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a800cdf30 con 0x7f6a800b8b90 2023-03-03T07:57:03.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.578+0000 7f6a76ffd700 1 -- 172.21.15.98:0/3301999225 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6a70019030 con 0x7f6a800b8b90 2023-03-03T07:57:03.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.580+0000 7f6a76ffd700 1 --2- 172.21.15.98:0/3301999225 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6a640682f0 0x7f6a6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:03.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.580+0000 7f6a76ffd700 1 -- 172.21.15.98:0/3301999225 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f6a700850a0 con 0x7f6a800b8b90 2023-03-03T07:57:03.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.580+0000 7f6a8659b700 1 --2- 172.21.15.98:0/3301999225 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6a640682f0 0x7f6a6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:03.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.581+0000 7f6a8659b700 1 --2- 172.21.15.98:0/3301999225 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6a640682f0 0x7f6a6406a7a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c0011e0 tx=0x7f6a7c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:03.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.584+0000 7f6a76ffd700 1 -- 172.21.15.98:0/3301999225 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6a7004b5e0 con 0x7f6a800b8b90 2023-03-03T07:57:03.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.778+0000 7f6a887ff700 1 -- 172.21.15.98:0/3301999225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a800729b0 con 0x7f6a800b8b90 2023-03-03T07:57:03.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.779+0000 7f6a76ffd700 1 -- 172.21.15.98:0/3301999225 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18003 (secure 0 0 0) 0x7f6a70051880 con 0x7f6a800b8b90 2023-03-03T07:57:03.781 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:03.781 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:56:27.498205+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:55:45.157666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T07:55:46.092091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:57:03.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.782+0000 7f6a887ff700 1 -- 172.21.15.98:0/3301999225 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6a640682f0 msgr2=0x7f6a6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:03.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.782+0000 7f6a887ff700 1 --2- 172.21.15.98:0/3301999225 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6a640682f0 0x7f6a6406a7a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c0011e0 tx=0x7f6a7c009040 comp rx=0 tx=0).stop 2023-03-03T07:57:03.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.782+0000 7f6a887ff700 1 -- 172.21.15.98:0/3301999225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a800b8b90 msgr2=0x7f6a801477f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.782+0000 7f6a887ff700 1 --2- 172.21.15.98:0/3301999225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a800b8b90 0x7f6a801477f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000c4f0 tx=0x7f6a7000c8b0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.782+0000 7f6a887ff700 1 -- 172.21.15.98:0/3301999225 shutdown_connections 2023-03-03T07:57:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.782+0000 7f6a887ff700 1 --2- 172.21.15.98:0/3301999225 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6a640682f0 0x7f6a6406a7a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.782+0000 7f6a887ff700 1 --2- 172.21.15.98:0/3301999225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a800d23e0 0x7f6a8015c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.782+0000 7f6a887ff700 1 --2- 172.21.15.98:0/3301999225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a800b87c0 0x7f6a801480b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.782+0000 7f6a887ff700 1 --2- 172.21.15.98:0/3301999225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a800b8b90 0x7f6a801477f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:03.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.782+0000 7f6a887ff700 1 -- 172.21.15.98:0/3301999225 >> 172.21.15.98:0/3301999225 conn(0x7f6a8006aee0 msgr2=0x7f6a800b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:03.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.782+0000 7f6a887ff700 1 -- 172.21.15.98:0/3301999225 shutdown_connections 2023-03-03T07:57:03.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:03.782+0000 7f6a887ff700 1 -- 172.21.15.98:0/3301999225 wait complete. 2023-03-03T07:57:03.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-03-03T07:57:04.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.055+0000 7fd4a07ff700 1 Processor -- start 2023-03-03T07:57:04.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.055+0000 7fd4a07ff700 1 -- start start 2023-03-03T07:57:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.056+0000 7fd4a07ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4980d0270 0x7fd4980cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.056+0000 7fd4a07ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd49806b4b0 0x7fd4980d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.056+0000 7fd4a07ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4980cf980 con 0x7fd49815ca70 2023-03-03T07:57:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.056+0000 7fd4a07ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4980cd5b0 con 0x7fd49806b4b0 2023-03-03T07:57:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.056+0000 7fd4a07ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980d1800 con 0x7fd4980d0270 2023-03-03T07:57:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.057+0000 7fd49e59b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd49815ca70 0x7fd4980cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34390/0 (socket says 172.21.15.98:34390) 2023-03-03T07:57:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.057+0000 7fd49e59b700 1 -- 172.21.15.98:0/2186792622 learned_addr learned my addr 172.21.15.98:0/2186792622 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.057+0000 7fd49dd9a700 1 --2- 172.21.15.98:0/2186792622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4980d0270 0x7fd4980cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.057+0000 7fd49ed9c700 1 --2- 172.21.15.98:0/2186792622 >> v2:172.21.15.100:3300/0 conn(0x7fd49806b4b0 0x7fd4980d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.057+0000 7fd49d599700 1 -- 172.21.15.98:0/2186792622 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052388250 0 0) 0x7fd4980cf980 con 0x7fd49815ca70 2023-03-03T07:57:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.057+0000 7fd49d599700 1 -- 172.21.15.98:0/2186792622 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4840035b0 con 0x7fd49815ca70 2023-03-03T07:57:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.057+0000 7fd49ed9c700 1 -- 172.21.15.98:0/2186792622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4980d0270 msgr2=0x7fd4980cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.057+0000 7fd49ed9c700 1 --2- 172.21.15.98:0/2186792622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4980d0270 0x7fd4980cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.057+0000 7fd49ed9c700 1 -- 172.21.15.98:0/2186792622 >> v1:172.21.15.98:6789/0 conn(0x7fd49815ca70 legacy=0x7fd4980cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.058+0000 7fd49ed9c700 1 -- 172.21.15.98:0/2186792622 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4980cd900 con 0x7fd49806b4b0 2023-03-03T07:57:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.058+0000 7fd49ed9c700 1 --2- 172.21.15.98:0/2186792622 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd49806b4b0 0x7fd4980d1fc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd490009d90 tx=0x7fd49000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3773433c02b9df85 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.058+0000 7fd49d599700 1 -- 172.21.15.98:0/2186792622 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49000e070 con 0x7fd49806b4b0 2023-03-03T07:57:04.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.058+0000 7fd49d599700 1 -- 172.21.15.98:0/2186792622 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49000bb20 con 0x7fd49806b4b0 2023-03-03T07:57:04.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.058+0000 7fd49d599700 1 -- 172.21.15.98:0/2186792622 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490012430 con 0x7fd49806b4b0 2023-03-03T07:57:04.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.059+0000 7fd4a07ff700 1 -- 172.21.15.98:0/2186792622 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd49806b4b0 msgr2=0x7fd4980d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:04.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.059+0000 7fd4a07ff700 1 --2- 172.21.15.98:0/2186792622 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd49806b4b0 0x7fd4980d1fc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd490009d90 tx=0x7fd49000b330 comp rx=0 tx=0).stop 2023-03-03T07:57:04.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.059+0000 7fd4a07ff700 1 -- 172.21.15.98:0/2186792622 shutdown_connections 2023-03-03T07:57:04.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.059+0000 7fd4a07ff700 1 --2- 172.21.15.98:0/2186792622 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd49806b4b0 0x7fd4980d1fc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:04.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.059+0000 7fd4a07ff700 1 --2- 172.21.15.98:0/2186792622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4980d0270 0x7fd4980cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:04.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.059+0000 7fd4a07ff700 1 -- 172.21.15.98:0/2186792622 >> 172.21.15.98:0/2186792622 conn(0x7fd498073e70 msgr2=0x7fd49806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:04.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.059+0000 7fd4a07ff700 1 -- 172.21.15.98:0/2186792622 shutdown_connections 2023-03-03T07:57:04.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.059+0000 7fd4a07ff700 1 -- 172.21.15.98:0/2186792622 wait complete. 2023-03-03T07:57:04.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.061+0000 7fd4a07ff700 1 Processor -- start 2023-03-03T07:57:04.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.061+0000 7fd4a07ff700 1 -- start start 2023-03-03T07:57:04.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.062+0000 7fd4a07ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd49806b4b0 0x7fd4980ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.062+0000 7fd4a07ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4980d0270 0x7fd49815f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.062+0000 7fd4a07ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd49815ca70 0x7fd4980d30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.062+0000 7fd4a07ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49814f140 con 0x7fd49806b4b0 2023-03-03T07:57:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.062+0000 7fd4a07ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980b2be0 con 0x7fd49815ca70 2023-03-03T07:57:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.062+0000 7fd4a07ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980b2e30 con 0x7fd4980d0270 2023-03-03T07:57:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.062+0000 7fd49e59b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd49806b4b0 0x7fd4980ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.063+0000 7fd49e59b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd49806b4b0 0x7fd4980ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45902/0 (socket says 172.21.15.98:45902) 2023-03-03T07:57:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.063+0000 7fd49e59b700 1 -- 172.21.15.98:0/876974264 learned_addr learned my addr 172.21.15.98:0/876974264 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.063+0000 7fd49ed9c700 1 --2- 172.21.15.98:0/876974264 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd49815ca70 0x7fd4980d30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.063+0000 7fd49dd9a700 1 --2- 172.21.15.98:0/876974264 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4980d0270 0x7fd49815f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.063+0000 7fd49e59b700 1 -- 172.21.15.98:0/876974264 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4980d0270 msgr2=0x7fd49815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.063+0000 7fd49e59b700 1 --2- 172.21.15.98:0/876974264 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4980d0270 0x7fd49815f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.063+0000 7fd49e59b700 1 -- 172.21.15.98:0/876974264 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd49815ca70 msgr2=0x7fd4980d30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.063+0000 7fd49e59b700 1 --2- 172.21.15.98:0/876974264 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd49815ca70 0x7fd4980d30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:04.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.063+0000 7fd49e59b700 1 -- 172.21.15.98:0/876974264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd490009a70 con 0x7fd49806b4b0 2023-03-03T07:57:04.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.063+0000 7fd49dd9a700 1 --2- 172.21.15.98:0/876974264 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4980d0270 0x7fd49815f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:04.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.063+0000 7fd49e59b700 1 --2- 172.21.15.98:0/876974264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd49806b4b0 0x7fd4980ccae0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd49400c700 tx=0x7fd49400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:04.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.064+0000 7fd48effd700 1 -- 172.21.15.98:0/876974264 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd494005d70 con 0x7fd49806b4b0 2023-03-03T07:57:04.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.064+0000 7fd48effd700 1 -- 172.21.15.98:0/876974264 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4940070d0 con 0x7fd49806b4b0 2023-03-03T07:57:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.064+0000 7fd48effd700 1 -- 172.21.15.98:0/876974264 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd494007460 con 0x7fd49806b4b0 2023-03-03T07:57:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.064+0000 7fd4a07ff700 1 -- 172.21.15.98:0/876974264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4980d1aa0 con 0x7fd49806b4b0 2023-03-03T07:57:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.064+0000 7fd4a07ff700 1 -- 172.21.15.98:0/876974264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4980cdbe0 con 0x7fd49806b4b0 2023-03-03T07:57:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.065+0000 7fd4a07ff700 1 -- 172.21.15.98:0/876974264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49815ce40 con 0x7fd49806b4b0 2023-03-03T07:57:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.066+0000 7fd48effd700 1 -- 172.21.15.98:0/876974264 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd494019030 con 0x7fd49806b4b0 2023-03-03T07:57:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.067+0000 7fd48effd700 1 --2- 172.21.15.98:0/876974264 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd47c0682f0 0x7fd47c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.067+0000 7fd48effd700 1 -- 172.21.15.98:0/876974264 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fd494085190 con 0x7fd49806b4b0 2023-03-03T07:57:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.068+0000 7fd49dd9a700 1 --2- 172.21.15.98:0/876974264 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd47c0682f0 0x7fd47c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.068+0000 7fd49dd9a700 1 --2- 172.21.15.98:0/876974264 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd47c0682f0 0x7fd47c06a7a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd4980d2e60 tx=0x7fd488008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.071+0000 7fd48effd700 1 -- 172.21.15.98:0/876974264 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd4940518c0 con 0x7fd49806b4b0 2023-03-03T07:57:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.308+0000 7fd4a07ff700 1 -- 172.21.15.98:0/876974264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd49804e6e0 con 0x7fd49806b4b0 2023-03-03T07:57:04.651 INFO:tasks.ceph.mon.a.smithi098.stderr:2023-03-03T07:57:04.650+0000 7fb71abf9700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-03-03T07:57:04.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.659+0000 7fd48effd700 1 -- 172.21.15.98:0/876974264 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd494054ee0 con 0x7fd49806b4b0 2023-03-03T07:57:04.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.662+0000 7fd4a07ff700 1 -- 172.21.15.98:0/876974264 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd47c0682f0 msgr2=0x7fd47c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:04.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.662+0000 7fd4a07ff700 1 --2- 172.21.15.98:0/876974264 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd47c0682f0 0x7fd47c06a7a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd4980d2e60 tx=0x7fd488008040 comp rx=0 tx=0).stop 2023-03-03T07:57:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.662+0000 7fd4a07ff700 1 -- 172.21.15.98:0/876974264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd49806b4b0 msgr2=0x7fd4980ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.662+0000 7fd4a07ff700 1 --2- 172.21.15.98:0/876974264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd49806b4b0 0x7fd4980ccae0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd49400c700 tx=0x7fd49400ca10 comp rx=0 tx=0).stop 2023-03-03T07:57:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.662+0000 7fd4a07ff700 1 -- 172.21.15.98:0/876974264 shutdown_connections 2023-03-03T07:57:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.662+0000 7fd4a07ff700 1 --2- 172.21.15.98:0/876974264 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd47c0682f0 0x7fd47c06a7a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.662+0000 7fd4a07ff700 1 --2- 172.21.15.98:0/876974264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd49806b4b0 0x7fd4980ccae0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.662+0000 7fd4a07ff700 1 --2- 172.21.15.98:0/876974264 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd49815ca70 0x7fd4980d30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.662+0000 7fd4a07ff700 1 --2- 172.21.15.98:0/876974264 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4980d0270 0x7fd49815f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.662+0000 7fd4a07ff700 1 -- 172.21.15.98:0/876974264 >> 172.21.15.98:0/876974264 conn(0x7fd498073e70 msgr2=0x7fd4981598a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.662+0000 7fd4a07ff700 1 -- 172.21.15.98:0/876974264 shutdown_connections 2023-03-03T07:57:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.662+0000 7fd4a07ff700 1 -- 172.21.15.98:0/876974264 wait complete. 2023-03-03T07:57:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-03-03T07:57:04.673 INFO:tasks.ceph.mds.c.smithi098.stderr: -4076> 2023-03-03T07:55:51.762+0000 7f67237f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T07:57:04.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-03-03T07:57:04.750 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-03-03T07:57:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.939+0000 7f96d15ff700 1 Processor -- start 2023-03-03T07:57:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.940+0000 7f96d15ff700 1 -- start start 2023-03-03T07:57:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.941+0000 7f96d15ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f96cc0b8d10 0x7f96cc0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.941+0000 7f96d15ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f96cc0b26d0 0x7f96cc0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.941+0000 7f96d15ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc0cd7d0 con 0x7f96cc0b5140 2023-03-03T07:57:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.941+0000 7f96d15ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96cc0b55c0 con 0x7f96cc0b8d10 2023-03-03T07:57:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.941+0000 7f96d15ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96cc0b90e0 con 0x7f96cc0b26d0 2023-03-03T07:57:04.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.942+0000 7f96caffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f96cc0b8d10 0x7f96cc0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:04.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.942+0000 7f96cb7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f96cc0b5140 0x7f96cc160520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55062/0 (socket says 172.21.15.98:55062) 2023-03-03T07:57:04.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.942+0000 7f96cb7fe700 1 -- 172.21.15.98:0/3354923781 learned_addr learned my addr 172.21.15.98:0/3354923781 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:04.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.942+0000 7f96c3fff700 1 --2- 172.21.15.98:0/3354923781 >> v2:172.21.15.100:3300/0 conn(0x7f96cc0b26d0 0x7f96cc0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.942+0000 7f96caffd700 1 -- 172.21.15.98:0/3354923781 >> v2:172.21.15.100:3300/0 conn(0x7f96cc0b26d0 msgr2=0x7f96cc0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.942+0000 7f96caffd700 1 --2- 172.21.15.98:0/3354923781 >> v2:172.21.15.100:3300/0 conn(0x7f96cc0b26d0 0x7f96cc0b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.942+0000 7f96caffd700 1 -- 172.21.15.98:0/3354923781 >> v1:172.21.15.98:6789/0 conn(0x7f96cc0b5140 legacy=0x7f96cc160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.942+0000 7f96caffd700 1 -- 172.21.15.98:0/3354923781 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc0cda20 con 0x7f96cc0b8d10 2023-03-03T07:57:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.942+0000 7f96ca7fc700 1 -- 172.21.15.98:0/3354923781 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725103852 0 0) 0x7f96cc0cd7d0 con 0x7f96cc0b5140 2023-03-03T07:57:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.942+0000 7f96caffd700 1 --2- 172.21.15.98:0/3354923781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96cc0b8d10 0x7f96cc0cffc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f96cc08bfa0 tx=0x7f96bc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3671f1640b76ae0d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:04.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.943+0000 7f96ca7fc700 1 -- 172.21.15.98:0/3354923781 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc011830 con 0x7f96cc0b8d10 2023-03-03T07:57:04.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.943+0000 7f96ca7fc700 1 -- 172.21.15.98:0/3354923781 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96bc011990 con 0x7f96cc0b8d10 2023-03-03T07:57:04.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.943+0000 7f96ca7fc700 1 -- 172.21.15.98:0/3354923781 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc011d20 con 0x7f96cc0b8d10 2023-03-03T07:57:04.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.943+0000 7f96d15ff700 1 -- 172.21.15.98:0/3354923781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96cc0b8d10 msgr2=0x7f96cc0cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:04.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.943+0000 7f96d15ff700 1 --2- 172.21.15.98:0/3354923781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96cc0b8d10 0x7f96cc0cffc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f96cc08bfa0 tx=0x7f96bc009b20 comp rx=0 tx=0).stop 2023-03-03T07:57:04.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.943+0000 7f96d15ff700 1 -- 172.21.15.98:0/3354923781 shutdown_connections 2023-03-03T07:57:04.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.943+0000 7f96d15ff700 1 --2- 172.21.15.98:0/3354923781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96cc0b8d10 0x7f96cc0cffc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:04.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.943+0000 7f96d15ff700 1 --2- 172.21.15.98:0/3354923781 >> v2:172.21.15.100:3300/0 conn(0x7f96cc0b26d0 0x7f96cc0b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:04.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.943+0000 7f96d15ff700 1 -- 172.21.15.98:0/3354923781 >> 172.21.15.98:0/3354923781 conn(0x7f96cc073e70 msgr2=0x7f96cc06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:04.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.944+0000 7f96d15ff700 1 -- 172.21.15.98:0/3354923781 shutdown_connections 2023-03-03T07:57:04.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.944+0000 7f96d15ff700 1 -- 172.21.15.98:0/3354923781 wait complete. 2023-03-03T07:57:04.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.945+0000 7f96d15ff700 1 Processor -- start 2023-03-03T07:57:04.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.946+0000 7f96d15ff700 1 -- start start 2023-03-03T07:57:04.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96d15ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96cc0b26d0 0x7f96cc15cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:04.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96d15ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96cc0b5140 0x7f96cc153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:04.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96d15ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96cc0b8d10 0x7f96cc15e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:04.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96d15ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc15f080 con 0x7f96cc0b26d0 2023-03-03T07:57:04.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96d15ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc15edc0 con 0x7f96cc0b5140 2023-03-03T07:57:04.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96d15ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc15d680 con 0x7f96cc0b8d10 2023-03-03T07:57:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96caffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96cc0b26d0 0x7f96cc15cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96caffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96cc0b26d0 0x7f96cc15cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47728/0 (socket says 172.21.15.98:47728) 2023-03-03T07:57:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96caffd700 1 -- 172.21.15.98:0/2990583463 learned_addr learned my addr 172.21.15.98:0/2990583463 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96cb7fe700 1 --2- 172.21.15.98:0/2990583463 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96cc0b8d10 0x7f96cc15e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96c3fff700 1 --2- 172.21.15.98:0/2990583463 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96cc0b5140 0x7f96cc153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96caffd700 1 -- 172.21.15.98:0/2990583463 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96cc0b8d10 msgr2=0x7f96cc15e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96caffd700 1 --2- 172.21.15.98:0/2990583463 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96cc0b8d10 0x7f96cc15e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96caffd700 1 -- 172.21.15.98:0/2990583463 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96cc0b5140 msgr2=0x7f96cc153740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96caffd700 1 --2- 172.21.15.98:0/2990583463 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96cc0b5140 0x7f96cc153740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96caffd700 1 -- 172.21.15.98:0/2990583463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96bc0094a0 con 0x7f96cc0b26d0 2023-03-03T07:57:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.947+0000 7f96c3fff700 1 --2- 172.21.15.98:0/2990583463 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96cc0b5140 0x7f96cc153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.948+0000 7f96caffd700 1 --2- 172.21.15.98:0/2990583463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96cc0b26d0 0x7f96cc15cff0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009b60 tx=0x7f96bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.948+0000 7f96c37fe700 1 -- 172.21.15.98:0/2990583463 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc0036a0 con 0x7f96cc0b26d0 2023-03-03T07:57:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.948+0000 7f96c37fe700 1 -- 172.21.15.98:0/2990583463 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96bc01e070 con 0x7f96cc0b26d0 2023-03-03T07:57:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.948+0000 7f96c37fe700 1 -- 172.21.15.98:0/2990583463 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc003a20 con 0x7f96cc0b26d0 2023-03-03T07:57:04.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.948+0000 7f96d15ff700 1 -- 172.21.15.98:0/2990583463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96cc0d0e80 con 0x7f96cc0b26d0 2023-03-03T07:57:04.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.948+0000 7f96d15ff700 1 -- 172.21.15.98:0/2990583463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96cc0cdd00 con 0x7f96cc0b26d0 2023-03-03T07:57:04.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.949+0000 7f96d15ff700 1 -- 172.21.15.98:0/2990583463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96cc0cdfe0 con 0x7f96cc0b26d0 2023-03-03T07:57:04.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.950+0000 7f96c37fe700 1 -- 172.21.15.98:0/2990583463 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f96bc00f050 con 0x7f96cc0b26d0 2023-03-03T07:57:04.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.951+0000 7f96c37fe700 1 --2- 172.21.15.98:0/2990583463 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f96ac0682f0 0x7f96ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:04.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.952+0000 7f96c37fe700 1 -- 172.21.15.98:0/2990583463 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6206+0+0 (secure 0 0 0) 0x7f96bc086e60 con 0x7f96cc0b26d0 2023-03-03T07:57:04.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.952+0000 7f96c3fff700 1 --2- 172.21.15.98:0/2990583463 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f96ac0682f0 0x7f96ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:04.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.952+0000 7f96c3fff700 1 --2- 172.21.15.98:0/2990583463 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f96ac0682f0 0x7f96ac06a7a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f96cc147a30 tx=0x7f96b4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:04.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:04.956+0000 7f96c37fe700 1 -- 172.21.15.98:0/2990583463 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f96bc053540 con 0x7f96cc0b26d0 2023-03-03T07:57:05.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.191+0000 7f96d15ff700 1 -- 172.21.15.98:0/2990583463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f96cc06ce30 con 0x7f96cc0b26d0 2023-03-03T07:57:05.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.663+0000 7f96c37fe700 1 -- 172.21.15.98:0/2990583463 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7f96bc0200a0 con 0x7f96cc0b26d0 2023-03-03T07:57:05.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.667+0000 7f96d15ff700 1 -- 172.21.15.98:0/2990583463 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f96ac0682f0 msgr2=0x7f96ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:05.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.667+0000 7f96d15ff700 1 --2- 172.21.15.98:0/2990583463 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f96ac0682f0 0x7f96ac06a7a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f96cc147a30 tx=0x7f96b4008040 comp rx=0 tx=0).stop 2023-03-03T07:57:05.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.667+0000 7f96d15ff700 1 -- 172.21.15.98:0/2990583463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96cc0b26d0 msgr2=0x7f96cc15cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:05.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.667+0000 7f96d15ff700 1 --2- 172.21.15.98:0/2990583463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96cc0b26d0 0x7f96cc15cff0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009b60 tx=0x7f96bc004750 comp rx=0 tx=0).stop 2023-03-03T07:57:05.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.667+0000 7f96d15ff700 1 -- 172.21.15.98:0/2990583463 shutdown_connections 2023-03-03T07:57:05.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.667+0000 7f96d15ff700 1 --2- 172.21.15.98:0/2990583463 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f96ac0682f0 0x7f96ac06a7a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:05.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.667+0000 7f96d15ff700 1 --2- 172.21.15.98:0/2990583463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96cc0b26d0 0x7f96cc15cff0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:05.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.667+0000 7f96d15ff700 1 --2- 172.21.15.98:0/2990583463 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96cc0b8d10 0x7f96cc15e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:05.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.667+0000 7f96d15ff700 1 --2- 172.21.15.98:0/2990583463 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96cc0b5140 0x7f96cc153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:05.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.667+0000 7f96d15ff700 1 -- 172.21.15.98:0/2990583463 >> 172.21.15.98:0/2990583463 conn(0x7f96cc073e70 msgr2=0x7f96cc0b7220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:05.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.667+0000 7f96d15ff700 1 -- 172.21.15.98:0/2990583463 shutdown_connections 2023-03-03T07:57:05.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.667+0000 7f96d15ff700 1 -- 172.21.15.98:0/2990583463 wait complete. 2023-03-03T07:57:05.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-03-03T07:57:05.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.949+0000 7f15773ff700 1 Processor -- start 2023-03-03T07:57:05.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.949+0000 7f15773ff700 1 -- start start 2023-03-03T07:57:05.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.950+0000 7f15773ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f157015d010 0x7f15700b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:05.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.951+0000 7f15773ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15700b22c0 0x7f15700d0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:05.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.951+0000 7f15773ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15700b2720 con 0x7f157015cc40 2023-03-03T07:57:05.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.951+0000 7f15773ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15701617b0 con 0x7f157015d010 2023-03-03T07:57:05.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.951+0000 7f15773ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157006b900 con 0x7f15700b22c0 2023-03-03T07:57:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.951+0000 7f157599c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15700b22c0 0x7f15700d0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.951+0000 7f157499a700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f157015d010 0x7f15700b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.951+0000 7f157519b700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f157015cc40 0x7f15700cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:49724/0 (socket says 172.21.15.98:49724) 2023-03-03T07:57:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.951+0000 7f157519b700 1 -- 172.21.15.98:0/3449558895 learned_addr learned my addr 172.21.15.98:0/3449558895 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.951+0000 7f157599c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15700b22c0 0x7f15700d0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:43790/0 (socket says 172.21.15.98:43790) 2023-03-03T07:57:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.952+0000 7f1567fff700 1 -- 172.21.15.98:0/3449558895 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898039941 0 0) 0x7f15700b2720 con 0x7f157015cc40 2023-03-03T07:57:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.952+0000 7f157599c700 1 -- 172.21.15.98:0/3449558895 >> v2:172.21.15.100:3300/0 conn(0x7f157015d010 msgr2=0x7f15700b1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.952+0000 7f157599c700 1 --2- 172.21.15.98:0/3449558895 >> v2:172.21.15.100:3300/0 conn(0x7f157015d010 0x7f15700b1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.952+0000 7f157599c700 1 -- 172.21.15.98:0/3449558895 >> v1:172.21.15.100:6789/0 conn(0x7f157015cc40 legacy=0x7f15700cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.952+0000 7f157599c700 1 -- 172.21.15.98:0/3449558895 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15701609f0 con 0x7f15700b22c0 2023-03-03T07:57:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.952+0000 7f157499a700 1 --2- 172.21.15.98:0/3449558895 >> v2:172.21.15.100:3300/0 conn(0x7f157015d010 0x7f15700b1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.952+0000 7f157599c700 1 --2- 172.21.15.98:0/3449558895 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15700b22c0 0x7f15700d0f30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1568009d90 tx=0x7f156800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5f8027782f6a4e3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:05.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.953+0000 7f1567fff700 1 -- 172.21.15.98:0/3449558895 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156800e070 con 0x7f15700b22c0 2023-03-03T07:57:05.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.953+0000 7f1567fff700 1 -- 172.21.15.98:0/3449558895 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156800bb20 con 0x7f15700b22c0 2023-03-03T07:57:05.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.953+0000 7f1567fff700 1 -- 172.21.15.98:0/3449558895 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568012430 con 0x7f15700b22c0 2023-03-03T07:57:05.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.953+0000 7f15773ff700 1 -- 172.21.15.98:0/3449558895 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15700b22c0 msgr2=0x7f15700d0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:05.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.953+0000 7f15773ff700 1 --2- 172.21.15.98:0/3449558895 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15700b22c0 0x7f15700d0f30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1568009d90 tx=0x7f156800b330 comp rx=0 tx=0).stop 2023-03-03T07:57:05.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.953+0000 7f15773ff700 1 -- 172.21.15.98:0/3449558895 shutdown_connections 2023-03-03T07:57:05.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.953+0000 7f15773ff700 1 --2- 172.21.15.98:0/3449558895 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15700b22c0 0x7f15700d0f30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:05.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.953+0000 7f15773ff700 1 --2- 172.21.15.98:0/3449558895 >> v2:172.21.15.100:3300/0 conn(0x7f157015d010 0x7f15700b1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:05.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.953+0000 7f15773ff700 1 -- 172.21.15.98:0/3449558895 >> 172.21.15.98:0/3449558895 conn(0x7f1570073e10 msgr2=0x7f15700d31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:05.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.953+0000 7f15773ff700 1 -- 172.21.15.98:0/3449558895 shutdown_connections 2023-03-03T07:57:05.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.953+0000 7f15773ff700 1 -- 172.21.15.98:0/3449558895 wait complete. 2023-03-03T07:57:05.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.955+0000 7f15773ff700 1 Processor -- start 2023-03-03T07:57:05.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.956+0000 7f15773ff700 1 -- start start 2023-03-03T07:57:05.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.956+0000 7f15773ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15700b22c0 0x7f157014e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:05.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.956+0000 7f15773ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f157015cc40 0x7f157015e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:05.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.957+0000 7f15773ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157015d010 0x7f157015e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:05.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.957+0000 7f15773ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157015faf0 con 0x7f157015d010 2023-03-03T07:57:05.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.957+0000 7f15773ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1570160610 con 0x7f157015cc40 2023-03-03T07:57:05.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.957+0000 7f15773ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157015eee0 con 0x7f15700b22c0 2023-03-03T07:57:05.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.957+0000 7f157599c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157015d010 0x7f157015e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:05.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.957+0000 7f157599c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157015d010 0x7f157015e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47744/0 (socket says 172.21.15.98:47744) 2023-03-03T07:57:05.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.957+0000 7f157599c700 1 -- 172.21.15.98:0/424198117 learned_addr learned my addr 172.21.15.98:0/424198117 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:05.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.957+0000 7f157519b700 1 --2- 172.21.15.98:0/424198117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15700b22c0 0x7f157014e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:05.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.957+0000 7f157499a700 1 --2- 172.21.15.98:0/424198117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f157015cc40 0x7f157015e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:05.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.957+0000 7f157599c700 1 -- 172.21.15.98:0/424198117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15700b22c0 msgr2=0x7f157014e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:05.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.958+0000 7f157599c700 1 --2- 172.21.15.98:0/424198117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15700b22c0 0x7f157014e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:05.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.958+0000 7f157599c700 1 -- 172.21.15.98:0/424198117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f157015cc40 msgr2=0x7f157015e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:05.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.958+0000 7f157599c700 1 --2- 172.21.15.98:0/424198117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f157015cc40 0x7f157015e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:05.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.958+0000 7f157599c700 1 -- 172.21.15.98:0/424198117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15600094a0 con 0x7f157015d010 2023-03-03T07:57:05.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.958+0000 7f157499a700 1 --2- 172.21.15.98:0/424198117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f157015cc40 0x7f157015e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:05.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.958+0000 7f157599c700 1 --2- 172.21.15.98:0/424198117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157015d010 0x7f157015e8b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f157006ba40 tx=0x7f1568004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:05.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.958+0000 7f1565ffb700 1 -- 172.21.15.98:0/424198117 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568012430 con 0x7f157015d010 2023-03-03T07:57:05.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.958+0000 7f1565ffb700 1 -- 172.21.15.98:0/424198117 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1568020070 con 0x7f157015d010 2023-03-03T07:57:05.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.958+0000 7f1565ffb700 1 -- 172.21.15.98:0/424198117 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568004280 con 0x7f157015d010 2023-03-03T07:57:05.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.958+0000 7f15773ff700 1 -- 172.21.15.98:0/424198117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1568009a70 con 0x7f157015d010 2023-03-03T07:57:05.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.959+0000 7f15773ff700 1 -- 172.21.15.98:0/424198117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1570160fb0 con 0x7f157015d010 2023-03-03T07:57:05.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.960+0000 7f15773ff700 1 -- 172.21.15.98:0/424198117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1570160cd0 con 0x7f157015d010 2023-03-03T07:57:05.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.961+0000 7f1565ffb700 1 -- 172.21.15.98:0/424198117 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f156802e070 con 0x7f157015d010 2023-03-03T07:57:05.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.962+0000 7f1565ffb700 1 --2- 172.21.15.98:0/424198117 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15540682f0 0x7f155406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:05.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.962+0000 7f1565ffb700 1 -- 172.21.15.98:0/424198117 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6206+0+0 (secure 0 0 0) 0x7f1568022070 con 0x7f157015d010 2023-03-03T07:57:05.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.962+0000 7f157519b700 1 --2- 172.21.15.98:0/424198117 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15540682f0 0x7f155406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:05.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.963+0000 7f157519b700 1 --2- 172.21.15.98:0/424198117 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15540682f0 0x7f155406a7a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f156c007fd0 tx=0x7f156c007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:05.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:05.966+0000 7f1565ffb700 1 -- 172.21.15.98:0/424198117 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f15680541f0 con 0x7f157015d010 2023-03-03T07:57:06.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.160+0000 7f15773ff700 1 -- 172.21.15.98:0/424198117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1570161290 con 0x7f157015d010 2023-03-03T07:57:06.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.671+0000 7f1565ffb700 1 -- 172.21.15.98:0/424198117 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1568057810 con 0x7f157015d010 2023-03-03T07:57:06.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.673+0000 7f15773ff700 1 -- 172.21.15.98:0/424198117 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15540682f0 msgr2=0x7f155406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:06.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.673+0000 7f15773ff700 1 --2- 172.21.15.98:0/424198117 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15540682f0 0x7f155406a7a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f156c007fd0 tx=0x7f156c007de0 comp rx=0 tx=0).stop 2023-03-03T07:57:06.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.673+0000 7f15773ff700 1 -- 172.21.15.98:0/424198117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157015d010 msgr2=0x7f157015e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:06.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.673+0000 7f15773ff700 1 --2- 172.21.15.98:0/424198117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157015d010 0x7f157015e8b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f157006ba40 tx=0x7f1568004ad0 comp rx=0 tx=0).stop 2023-03-03T07:57:06.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.674+0000 7f15773ff700 1 -- 172.21.15.98:0/424198117 shutdown_connections 2023-03-03T07:57:06.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.674+0000 7f15773ff700 1 --2- 172.21.15.98:0/424198117 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15540682f0 0x7f155406a7a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:06.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.674+0000 7f15773ff700 1 --2- 172.21.15.98:0/424198117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157015d010 0x7f157015e8b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:06.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.674+0000 7f15773ff700 1 --2- 172.21.15.98:0/424198117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15700b22c0 0x7f157014e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:06.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.674+0000 7f15773ff700 1 --2- 172.21.15.98:0/424198117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f157015cc40 0x7f157015e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:06.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.674+0000 7f15773ff700 1 -- 172.21.15.98:0/424198117 >> 172.21.15.98:0/424198117 conn(0x7f1570073e10 msgr2=0x7f15700cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:06.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.674+0000 7f15773ff700 1 -- 172.21.15.98:0/424198117 shutdown_connections 2023-03-03T07:57:06.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.674+0000 7f15773ff700 1 -- 172.21.15.98:0/424198117 wait complete. 2023-03-03T07:57:06.678 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' removed 2023-03-03T07:57:06.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-03-03T07:57:06.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.953+0000 7fc8f959d700 1 Processor -- start 2023-03-03T07:57:06.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.954+0000 7fc8f959d700 1 -- start start 2023-03-03T07:57:06.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.955+0000 7fc8f959d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc8f40b3550 0x7fc8f40cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.955+0000 7fc8f959d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8f40cee50 0x7fc8f40cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.955+0000 7fc8f959d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f40b3c80 con 0x7fc8f4073ed0 2023-03-03T07:57:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.955+0000 7fc8f959d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f40b4370 con 0x7fc8f40b3550 2023-03-03T07:57:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.955+0000 7fc8f959d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f40b3ed0 con 0x7fc8f40cee50 2023-03-03T07:57:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.956+0000 7fc8f8d9c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8f40cee50 0x7fc8f40cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.956+0000 7fc8f8d9c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8f40cee50 0x7fc8f40cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:43800/0 (socket says 172.21.15.98:43800) 2023-03-03T07:57:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.956+0000 7fc8f8d9c700 1 -- 172.21.15.98:0/818545495 learned_addr learned my addr 172.21.15.98:0/818545495 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.956+0000 7fc8f3fff700 1 --2- 172.21.15.98:0/818545495 >> v2:172.21.15.100:3300/0 conn(0x7fc8f40b3550 0x7fc8f40cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.956+0000 7fc8f8d9c700 1 -- 172.21.15.98:0/818545495 >> v2:172.21.15.100:3300/0 conn(0x7fc8f40b3550 msgr2=0x7fc8f40cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.956+0000 7fc8f8d9c700 1 --2- 172.21.15.98:0/818545495 >> v2:172.21.15.100:3300/0 conn(0x7fc8f40b3550 0x7fc8f40cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.956+0000 7fc8f8d9c700 1 -- 172.21.15.98:0/818545495 >> v1:172.21.15.100:6789/0 conn(0x7fc8f4073ed0 legacy=0x7fc8f4160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.956+0000 7fc8f8d9c700 1 -- 172.21.15.98:0/818545495 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f40cd640 con 0x7fc8f40cee50 2023-03-03T07:57:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.957+0000 7fc8f8d9c700 1 --2- 172.21.15.98:0/818545495 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8f40cee50 0x7fc8f40cff40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009e50 tx=0x7fc8ec00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f24ebf2a5699cc78 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.957+0000 7fc8f37fe700 1 -- 172.21.15.98:0/818545495 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec00e070 con 0x7fc8f40cee50 2023-03-03T07:57:06.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.957+0000 7fc8f37fe700 1 -- 172.21.15.98:0/818545495 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ec00b910 con 0x7fc8f40cee50 2023-03-03T07:57:06.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.957+0000 7fc8f37fe700 1 -- 172.21.15.98:0/818545495 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec00bcc0 con 0x7fc8f40cee50 2023-03-03T07:57:06.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.958+0000 7fc8f959d700 1 -- 172.21.15.98:0/818545495 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8f40cee50 msgr2=0x7fc8f40cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.958+0000 7fc8f959d700 1 --2- 172.21.15.98:0/818545495 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8f40cee50 0x7fc8f40cff40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009e50 tx=0x7fc8ec00b330 comp rx=0 tx=0).stop 2023-03-03T07:57:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.958+0000 7fc8f959d700 1 -- 172.21.15.98:0/818545495 shutdown_connections 2023-03-03T07:57:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.958+0000 7fc8f959d700 1 --2- 172.21.15.98:0/818545495 >> v2:172.21.15.100:3300/0 conn(0x7fc8f40b3550 0x7fc8f40cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.958+0000 7fc8f959d700 1 --2- 172.21.15.98:0/818545495 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8f40cee50 0x7fc8f40cff40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.958+0000 7fc8f959d700 1 -- 172.21.15.98:0/818545495 >> 172.21.15.98:0/818545495 conn(0x7fc8f4073b00 msgr2=0x7fc8f40d2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.958+0000 7fc8f959d700 1 -- 172.21.15.98:0/818545495 shutdown_connections 2023-03-03T07:57:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.958+0000 7fc8f959d700 1 -- 172.21.15.98:0/818545495 wait complete. 2023-03-03T07:57:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.960+0000 7fc8f959d700 1 Processor -- start 2023-03-03T07:57:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.960+0000 7fc8f959d700 1 -- start start 2023-03-03T07:57:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.961+0000 7fc8f959d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8f4073ed0 0x7fc8f40b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.961+0000 7fc8f959d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8f40b3550 0x7fc8f41595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.961+0000 7fc8f959d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f40cee50 0x7fc8f41f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.961+0000 7fc8f959d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f40afd40 con 0x7fc8f40cee50 2023-03-03T07:57:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.961+0000 7fc8f959d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f40affe0 con 0x7fc8f40b3550 2023-03-03T07:57:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.961+0000 7fc8f959d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f40b02c0 con 0x7fc8f4073ed0 2023-03-03T07:57:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.961+0000 7fc8f8d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f40cee50 0x7fc8f41f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.962+0000 7fc8f3fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8f4073ed0 0x7fc8f40b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.962+0000 7fc8f8d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f40cee50 0x7fc8f41f2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47750/0 (socket says 172.21.15.98:47750) 2023-03-03T07:57:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.962+0000 7fc8ebfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8f40b3550 0x7fc8f41595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.962+0000 7fc8f8d9c700 1 -- 172.21.15.98:0/1460976204 learned_addr learned my addr 172.21.15.98:0/1460976204 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.962+0000 7fc8f3fff700 1 -- 172.21.15.98:0/1460976204 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8f40b3550 msgr2=0x7fc8f41595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.962+0000 7fc8f3fff700 1 --2- 172.21.15.98:0/1460976204 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8f40b3550 0x7fc8f41595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.962+0000 7fc8f3fff700 1 -- 172.21.15.98:0/1460976204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f40cee50 msgr2=0x7fc8f41f2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.962+0000 7fc8f3fff700 1 --2- 172.21.15.98:0/1460976204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f40cee50 0x7fc8f41f2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.962+0000 7fc8f3fff700 1 -- 172.21.15.98:0/1460976204 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec009b30 con 0x7fc8f4073ed0 2023-03-03T07:57:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.962+0000 7fc8f3fff700 1 --2- 172.21.15.98:0/1460976204 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8f4073ed0 0x7fc8f40b0610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400e900 tx=0x7fc8e400ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.963+0000 7fc8f17fa700 1 -- 172.21.15.98:0/1460976204 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4013680 con 0x7fc8f4073ed0 2023-03-03T07:57:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.963+0000 7fc8f17fa700 1 -- 172.21.15.98:0/1460976204 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e40137e0 con 0x7fc8f4073ed0 2023-03-03T07:57:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.963+0000 7fc8f959d700 1 -- 172.21.15.98:0/1460976204 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f40d3d60 con 0x7fc8f4073ed0 2023-03-03T07:57:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.963+0000 7fc8f17fa700 1 -- 172.21.15.98:0/1460976204 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4013b50 con 0x7fc8f4073ed0 2023-03-03T07:57:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.963+0000 7fc8f959d700 1 -- 172.21.15.98:0/1460976204 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f40cd920 con 0x7fc8f4073ed0 2023-03-03T07:57:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.964+0000 7fc8f959d700 1 -- 172.21.15.98:0/1460976204 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f40d3ef0 con 0x7fc8f4073ed0 2023-03-03T07:57:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.966+0000 7fc8f17fa700 1 -- 172.21.15.98:0/1460976204 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc8e4019030 con 0x7fc8f4073ed0 2023-03-03T07:57:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.967+0000 7fc8f17fa700 1 --2- 172.21.15.98:0/1460976204 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc8d40682f0 0x7fc8d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.967+0000 7fc8f17fa700 1 -- 172.21.15.98:0/1460976204 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5768+0+0 (secure 0 0 0) 0x7fc8e4085db0 con 0x7fc8f4073ed0 2023-03-03T07:57:06.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.967+0000 7fc8ebfff700 1 --2- 172.21.15.98:0/1460976204 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc8d40682f0 0x7fc8d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:06.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.968+0000 7fc8ebfff700 1 --2- 172.21.15.98:0/1460976204 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc8d40682f0 0x7fc8d406a7a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc8f415a010 tx=0x7fc8dc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:06.972+0000 7fc8f17fa700 1 -- 172.21.15.98:0/1460976204 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc8f40d3ef0 con 0x7fc8f4073ed0 2023-03-03T07:57:07.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:07.172+0000 7fc8f959d700 1 -- 172.21.15.98:0/1460976204 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc8f40d3ef0 con 0x7fc8f4073ed0 2023-03-03T07:57:07.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:07.685+0000 7fc8f17fa700 1 -- 172.21.15.98:0/1460976204 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc8f40d3ef0 con 0x7fc8f4073ed0 2023-03-03T07:57:07.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:07.689+0000 7fc8f959d700 1 -- 172.21.15.98:0/1460976204 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc8d40682f0 msgr2=0x7fc8d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:07.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:07.689+0000 7fc8f959d700 1 --2- 172.21.15.98:0/1460976204 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc8d40682f0 0x7fc8d406a7a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc8f415a010 tx=0x7fc8dc009040 comp rx=0 tx=0).stop 2023-03-03T07:57:07.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:07.689+0000 7fc8f959d700 1 -- 172.21.15.98:0/1460976204 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8f4073ed0 msgr2=0x7fc8f40b0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:07.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:07.689+0000 7fc8f959d700 1 --2- 172.21.15.98:0/1460976204 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8f4073ed0 0x7fc8f40b0610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400e900 tx=0x7fc8e400ec10 comp rx=0 tx=0).stop 2023-03-03T07:57:07.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:07.689+0000 7fc8f959d700 1 -- 172.21.15.98:0/1460976204 shutdown_connections 2023-03-03T07:57:07.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:07.689+0000 7fc8f959d700 1 --2- 172.21.15.98:0/1460976204 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc8d40682f0 0x7fc8d406a7a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:07.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:07.689+0000 7fc8f959d700 1 --2- 172.21.15.98:0/1460976204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f40cee50 0x7fc8f41f2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:07.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:07.689+0000 7fc8f959d700 1 --2- 172.21.15.98:0/1460976204 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8f4073ed0 0x7fc8f40b0610 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:07.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:07.689+0000 7fc8f959d700 1 --2- 172.21.15.98:0/1460976204 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8f40b3550 0x7fc8f41595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:07.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:07.689+0000 7fc8f959d700 1 -- 172.21.15.98:0/1460976204 >> 172.21.15.98:0/1460976204 conn(0x7fc8f4073b00 msgr2=0x7fc8f41e6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:07.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:07.689+0000 7fc8f959d700 1 -- 172.21.15.98:0/1460976204 shutdown_connections 2023-03-03T07:57:07.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:07.690+0000 7fc8f959d700 1 -- 172.21.15.98:0/1460976204 wait complete. 2023-03-03T07:57:07.693 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' removed 2023-03-03T07:57:07.720 INFO:tasks.ceph.mds.a:Restarting daemon 2023-03-03T07:57:07.720 INFO:tasks.ceph.mds.a:Stopping old one... 2023-03-03T07:57:07.721 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-03-03T07:57:07.721 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T07:57:07.721 INFO:tasks.ceph.mds.c:Restarting daemon 2023-03-03T07:57:07.721 INFO:tasks.ceph.mds.c:Stopping old one... 2023-03-03T07:57:07.722 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-03-03T07:57:07.722 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T07:57:07.722 INFO:tasks.ceph.mds.b:Restarting daemon 2023-03-03T07:57:07.722 INFO:tasks.ceph.mds.b:Stopping old one... 2023-03-03T07:57:07.723 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-03-03T07:57:07.723 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T07:57:07.723 INFO:tasks.ceph.mds.d:Restarting daemon 2023-03-03T07:57:07.723 INFO:tasks.ceph.mds.d:Stopping old one... 2023-03-03T07:57:07.724 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-03-03T07:57:07.724 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T07:57:07.743 INFO:tasks.ceph.mds.a:Stopped 2023-03-03T07:57:07.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-03-03T07:57:07.745 INFO:tasks.ceph.mds.c:Stopped 2023-03-03T07:57:07.745 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-03-03T07:57:07.754 INFO:tasks.ceph.mds.b:Stopped 2023-03-03T07:57:07.754 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-03-03T07:57:07.755 INFO:tasks.ceph.mds.d:Stopped 2023-03-03T07:57:07.755 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-03-03T07:57:07.789 INFO:tasks.ceph.mds.a:Started 2023-03-03T07:57:07.790 DEBUG:teuthology.parallel:result is None 2023-03-03T07:57:07.792 INFO:tasks.ceph.mds.c:Started 2023-03-03T07:57:07.793 DEBUG:teuthology.parallel:result is None 2023-03-03T07:57:07.799 INFO:tasks.ceph.mds.b:Started 2023-03-03T07:57:07.800 DEBUG:teuthology.parallel:result is None 2023-03-03T07:57:07.804 INFO:tasks.ceph.mds.d:Started 2023-03-03T07:57:07.804 DEBUG:teuthology.parallel:result is None 2023-03-03T07:57:07.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-03-03T07:57:07.999 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2023-03-03T07:57:08.000 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2023-03-03T07:57:08.009 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2023-03-03T07:57:08.010 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-03-03T07:57:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.093+0000 7f67373ff700 1 Processor -- start 2023-03-03T07:57:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.094+0000 7f67373ff700 1 -- start start 2023-03-03T07:57:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.095+0000 7f67373ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f673015ca70 0x7f67300cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.095+0000 7f67373ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67300d0270 0x7f67300d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.095+0000 7f67373ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67300cf980 con 0x7f673006b4b0 2023-03-03T07:57:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.095+0000 7f67373ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67300cd5b0 con 0x7f673015ca70 2023-03-03T07:57:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.095+0000 7f67373ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67300d1800 con 0x7f67300d0270 2023-03-03T07:57:08.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.095+0000 7f673499a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f673006b4b0 0x7f67300cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55076/0 (socket says 172.21.15.98:55076) 2023-03-03T07:57:08.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.095+0000 7f673499a700 1 -- 172.21.15.98:0/1745045186 learned_addr learned my addr 172.21.15.98:0/1745045186 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:08.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.095+0000 7f673519b700 1 --2- 172.21.15.98:0/1745045186 >> v2:172.21.15.100:3300/0 conn(0x7f673015ca70 0x7f67300cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:08.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.095+0000 7f673599c700 1 --2- 172.21.15.98:0/1745045186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67300d0270 0x7f67300d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:08.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.096+0000 7f6727fff700 1 -- 172.21.15.98:0/1745045186 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089587169 0 0) 0x7f67300cf980 con 0x7f673006b4b0 2023-03-03T07:57:08.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.096+0000 7f673519b700 1 -- 172.21.15.98:0/1745045186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67300d0270 msgr2=0x7f67300d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:08.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.096+0000 7f673519b700 1 --2- 172.21.15.98:0/1745045186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67300d0270 0x7f67300d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:08.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.096+0000 7f673519b700 1 -- 172.21.15.98:0/1745045186 >> v1:172.21.15.98:6789/0 conn(0x7f673006b4b0 legacy=0x7f67300cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:08.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.096+0000 7f673519b700 1 -- 172.21.15.98:0/1745045186 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67300cd900 con 0x7f673015ca70 2023-03-03T07:57:08.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.096+0000 7f673599c700 1 --2- 172.21.15.98:0/1745045186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67300d0270 0x7f67300d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:08.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.096+0000 7f673519b700 1 --2- 172.21.15.98:0/1745045186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f673015ca70 0x7f67300cf440 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f672c009840 tx=0x7f672c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d266d863727ec58 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:08.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.097+0000 7f6727fff700 1 -- 172.21.15.98:0/1745045186 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c011830 con 0x7f673015ca70 2023-03-03T07:57:08.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.097+0000 7f6727fff700 1 -- 172.21.15.98:0/1745045186 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672c011990 con 0x7f673015ca70 2023-03-03T07:57:08.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.097+0000 7f6727fff700 1 -- 172.21.15.98:0/1745045186 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c011d20 con 0x7f673015ca70 2023-03-03T07:57:08.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.097+0000 7f67373ff700 1 -- 172.21.15.98:0/1745045186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f673015ca70 msgr2=0x7f67300cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.097+0000 7f67373ff700 1 --2- 172.21.15.98:0/1745045186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f673015ca70 0x7f67300cf440 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f672c009840 tx=0x7f672c009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.097+0000 7f67373ff700 1 -- 172.21.15.98:0/1745045186 shutdown_connections 2023-03-03T07:57:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.097+0000 7f67373ff700 1 --2- 172.21.15.98:0/1745045186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f673015ca70 0x7f67300cf440 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.097+0000 7f67373ff700 1 --2- 172.21.15.98:0/1745045186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67300d0270 0x7f67300d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.097+0000 7f67373ff700 1 -- 172.21.15.98:0/1745045186 >> 172.21.15.98:0/1745045186 conn(0x7f6730073e70 msgr2=0x7f673006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.097+0000 7f67373ff700 1 -- 172.21.15.98:0/1745045186 shutdown_connections 2023-03-03T07:57:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.097+0000 7f67373ff700 1 -- 172.21.15.98:0/1745045186 wait complete. 2023-03-03T07:57:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.100+0000 7f67373ff700 1 Processor -- start 2023-03-03T07:57:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.101+0000 7f67373ff700 1 -- start start 2023-03-03T07:57:08.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.102+0000 7f67373ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f673006b4b0 0x7f67301f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.102+0000 7f67373ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67300d0270 0x7f67300b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.102+0000 7f67373ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f673015ca70 0x7f67300b7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.102+0000 7f67373ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67300b2ff0 con 0x7f67300d0270 2023-03-03T07:57:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.102+0000 7f67373ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67301f3470 con 0x7f673015ca70 2023-03-03T07:57:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.102+0000 7f67373ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67301f2eb0 con 0x7f673006b4b0 2023-03-03T07:57:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.102+0000 7f673499a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67300d0270 0x7f67300b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.102+0000 7f673499a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67300d0270 0x7f67300b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47796/0 (socket says 172.21.15.98:47796) 2023-03-03T07:57:08.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.102+0000 7f673499a700 1 -- 172.21.15.98:0/3175676815 learned_addr learned my addr 172.21.15.98:0/3175676815 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:08.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.103+0000 7f673519b700 1 --2- 172.21.15.98:0/3175676815 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f673006b4b0 0x7f67301f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.103+0000 7f673599c700 1 --2- 172.21.15.98:0/3175676815 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f673015ca70 0x7f67300b7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.103+0000 7f673499a700 1 -- 172.21.15.98:0/3175676815 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f673006b4b0 msgr2=0x7f67301f24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.103+0000 7f673499a700 1 --2- 172.21.15.98:0/3175676815 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f673006b4b0 0x7f67301f24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.103+0000 7f673499a700 1 -- 172.21.15.98:0/3175676815 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f673015ca70 msgr2=0x7f67300b7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.103+0000 7f673499a700 1 --2- 172.21.15.98:0/3175676815 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f673015ca70 0x7f67300b7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.103+0000 7f673499a700 1 -- 172.21.15.98:0/3175676815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6728009a70 con 0x7f67300d0270 2023-03-03T07:57:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.103+0000 7f673499a700 1 --2- 172.21.15.98:0/3175676815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67300d0270 0x7f67300b3480 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f672000c520 tx=0x7f672000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.103+0000 7f6725ffb700 1 -- 172.21.15.98:0/3175676815 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720004c80 con 0x7f67300d0270 2023-03-03T07:57:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.103+0000 7f6725ffb700 1 -- 172.21.15.98:0/3175676815 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6720004de0 con 0x7f67300d0270 2023-03-03T07:57:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.103+0000 7f67373ff700 1 -- 172.21.15.98:0/3175676815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672c0094a0 con 0x7f67300d0270 2023-03-03T07:57:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.103+0000 7f6725ffb700 1 -- 172.21.15.98:0/3175676815 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67200072d0 con 0x7f67300d0270 2023-03-03T07:57:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.104+0000 7f67373ff700 1 -- 172.21.15.98:0/3175676815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67300d1aa0 con 0x7f67300d0270 2023-03-03T07:57:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.105+0000 7f67373ff700 1 -- 172.21.15.98:0/3175676815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67300cdbe0 con 0x7f67300d0270 2023-03-03T07:57:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.106+0000 7f6725ffb700 1 -- 172.21.15.98:0/3175676815 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f672001a020 con 0x7f67300d0270 2023-03-03T07:57:08.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.107+0000 7f6725ffb700 1 --2- 172.21.15.98:0/3175676815 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f67140682c0 0x7f671406a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:08.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.107+0000 7f6725ffb700 1 -- 172.21.15.98:0/3175676815 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f6720085660 con 0x7f67300d0270 2023-03-03T07:57:08.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.108+0000 7f673519b700 1 --2- 172.21.15.98:0/3175676815 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f67140682c0 0x7f671406a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:08.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.109+0000 7f673519b700 1 --2- 172.21.15.98:0/3175676815 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f67140682c0 0x7f671406a770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f672c0090b0 tx=0x7f672c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:08.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.112+0000 7f6725ffb700 1 -- 172.21.15.98:0/3175676815 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f67200520f0 con 0x7f67300d0270 2023-03-03T07:57:08.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:08.322+0000 7f67373ff700 1 -- 172.21.15.98:0/3175676815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f673015ce40 con 0x7f67300d0270 2023-03-03T07:57:09.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.045+0000 7f6725ffb700 1 -- 172.21.15.98:0/3175676815 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v34) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6720055710 con 0x7f67300d0270 2023-03-03T07:57:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.048+0000 7f67373ff700 1 -- 172.21.15.98:0/3175676815 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f67140682c0 msgr2=0x7f671406a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.048+0000 7f67373ff700 1 --2- 172.21.15.98:0/3175676815 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f67140682c0 0x7f671406a770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f672c0090b0 tx=0x7f672c008030 comp rx=0 tx=0).stop 2023-03-03T07:57:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.049+0000 7f67373ff700 1 -- 172.21.15.98:0/3175676815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67300d0270 msgr2=0x7f67300b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.049+0000 7f67373ff700 1 --2- 172.21.15.98:0/3175676815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67300d0270 0x7f67300b3480 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f672000c520 tx=0x7f672000c8e0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.049+0000 7f67373ff700 1 -- 172.21.15.98:0/3175676815 shutdown_connections 2023-03-03T07:57:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.049+0000 7f67373ff700 1 --2- 172.21.15.98:0/3175676815 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f67140682c0 0x7f671406a770 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.049+0000 7f67373ff700 1 --2- 172.21.15.98:0/3175676815 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f673015ca70 0x7f67300b7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.049+0000 7f67373ff700 1 --2- 172.21.15.98:0/3175676815 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f673006b4b0 0x7f67301f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.049+0000 7f67373ff700 1 --2- 172.21.15.98:0/3175676815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67300d0270 0x7f67300b3480 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.049+0000 7f67373ff700 1 -- 172.21.15.98:0/3175676815 >> 172.21.15.98:0/3175676815 conn(0x7f6730073e70 msgr2=0x7f673015f350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.049+0000 7f67373ff700 1 -- 172.21.15.98:0/3175676815 shutdown_connections 2023-03-03T07:57:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.049+0000 7f67373ff700 1 -- 172.21.15.98:0/3175676815 wait complete. 2023-03-03T07:57:09.052 INFO:teuthology.orchestra.run.smithi098.stderr: removed all blocklist entries 2023-03-03T07:57:09.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-03-03T07:57:09.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.329+0000 7f49d13ff700 1 Processor -- start 2023-03-03T07:57:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.329+0000 7f49d13ff700 1 -- start start 2023-03-03T07:57:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.330+0000 7f49d13ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49cc0b8b90 0x7f49cc0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.330+0000 7f49d13ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49cc0d0ef0 con 0x7f49cc0b5410 2023-03-03T07:57:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.331+0000 7f49d13ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49cc0d0600 con 0x7f49cc0b87c0 2023-03-03T07:57:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.331+0000 7f49d13ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0cd970 con 0x7f49cc0b8b90 2023-03-03T07:57:09.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.331+0000 7f49cb7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f49cc0b5410 0x7f49cc0d13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55088/0 (socket says 172.21.15.98:55088) 2023-03-03T07:57:09.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.331+0000 7f49cb7fe700 1 -- 172.21.15.98:0/56128261 learned_addr learned my addr 172.21.15.98:0/56128261 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:09.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.331+0000 7f49ca7fc700 1 --2- 172.21.15.98:0/56128261 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49cc0b8b90 0x7f49cc0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:09.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.331+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989341903 0 0) 0x7f49cc0d0ef0 con 0x7f49cc0b5410 2023-03-03T07:57:09.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.331+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b40035b0 con 0x7f49cc0b5410 2023-03-03T07:57:09.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918267753 0 0) 0x7f49cc0d0600 con 0x7f49cc0b87c0 2023-03-03T07:57:09.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49cc0d0ef0 con 0x7f49cc0b87c0 2023-03-03T07:57:09.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2202356095 0 0) 0x7f49b40035b0 con 0x7f49cc0b5410 2023-03-03T07:57:09.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49cc0b8b90 msgr2=0x7f49cc0cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:09.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49c9ffb700 1 --2- 172.21.15.98:0/56128261 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49cc0b8b90 0x7f49cc0cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 >> v1:172.21.15.100:6789/0 conn(0x7f49cc0b87c0 legacy=0x7f49cc0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:09.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc0cdc50 con 0x7f49cc0b5410 2023-03-03T07:57:09.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f49c40019f0 con 0x7f49cc0b5410 2023-03-03T07:57:09.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49ca7fc700 1 --2- 172.21.15.98:0/56128261 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49cc0b8b90 0x7f49cc0cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:09.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 >> v1:172.21.15.98:6789/0 conn(0x7f49cc0b5410 legacy=0x7f49cc0d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:09.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49c9ffb700 1 --2- 172.21.15.98:0/56128261 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49cc0b8b90 0x7f49b4003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:09.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49c9ffb700 1 --2- 172.21.15.98:0/56128261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49b4004100 0x7f49b4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:09.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49c9ffb700 1 --2- 172.21.15.98:0/56128261 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49b4004d90 0x7f49b4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:09.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0cd970 con 0x7f49b4004100 2023-03-03T07:57:09.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b400bac0 con 0x7f49b4004d90 2023-03-03T07:57:09.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b400bc80 con 0x7f49cc0b8b90 2023-03-03T07:57:09.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49caffd700 1 --2- 172.21.15.98:0/56128261 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49cc0b8b90 0x7f49b4003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:09.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.332+0000 7f49ca7fc700 1 --2- 172.21.15.98:0/56128261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49b4004100 0x7f49b4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:09.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.333+0000 7f49cb7fe700 1 --2- 172.21.15.98:0/56128261 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49b4004d90 0x7f49b4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:09.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.333+0000 7f49caffd700 1 -- 172.21.15.98:0/56128261 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49b4004d90 msgr2=0x7f49b4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:09.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.333+0000 7f49caffd700 1 --2- 172.21.15.98:0/56128261 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49b4004d90 0x7f49b4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.333+0000 7f49caffd700 1 -- 172.21.15.98:0/56128261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49b4004100 msgr2=0x7f49b4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:09.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.333+0000 7f49caffd700 1 --2- 172.21.15.98:0/56128261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49b4004100 0x7f49b4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.333+0000 7f49caffd700 1 -- 172.21.15.98:0/56128261 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c00094a0 con 0x7f49cc0b8b90 2023-03-03T07:57:09.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.333+0000 7f49ca7fc700 1 --2- 172.21.15.98:0/56128261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49b4004100 0x7f49b4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:09.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.333+0000 7f49caffd700 1 --2- 172.21.15.98:0/56128261 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49cc0b8b90 0x7f49b4003bc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f49bc007fd0 tx=0x7f49b4005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2895fe007c8aec9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:09.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.334+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc001430 con 0x7f49cc0b8b90 2023-03-03T07:57:09.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.334+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49bc0052d0 con 0x7f49cc0b8b90 2023-03-03T07:57:09.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.334+0000 7f49c9ffb700 1 -- 172.21.15.98:0/56128261 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc001d40 con 0x7f49cc0b8b90 2023-03-03T07:57:09.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.334+0000 7f49d13ff700 1 -- 172.21.15.98:0/56128261 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49cc0b8b90 msgr2=0x7f49b4003bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:09.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.334+0000 7f49d13ff700 1 --2- 172.21.15.98:0/56128261 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49cc0b8b90 0x7f49b4003bc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f49bc007fd0 tx=0x7f49b4005880 comp rx=0 tx=0).stop 2023-03-03T07:57:09.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.334+0000 7f49cb7fe700 1 -- 172.21.15.98:0/56128261 reap_dead start 2023-03-03T07:57:09.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.334+0000 7f49d13ff700 1 -- 172.21.15.98:0/56128261 shutdown_connections 2023-03-03T07:57:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.334+0000 7f49d13ff700 1 -- 172.21.15.98:0/56128261 >> 172.21.15.98:0/56128261 conn(0x7f49cc06aee0 msgr2=0x7f49cc06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.334+0000 7f49d13ff700 1 -- 172.21.15.98:0/56128261 shutdown_connections 2023-03-03T07:57:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.334+0000 7f49d13ff700 1 -- 172.21.15.98:0/56128261 wait complete. 2023-03-03T07:57:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.336+0000 7f49d13ff700 1 Processor -- start 2023-03-03T07:57:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.337+0000 7f49d13ff700 1 -- start start 2023-03-03T07:57:09.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.338+0000 7f49d13ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49cc0b8b90 0x7f49cc156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:09.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.338+0000 7f49d13ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49cc0cabd0 0x7f49cc0cb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:09.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.338+0000 7f49d13ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49cc15e0b0 0x7f49cc15e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:09.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.338+0000 7f49d13ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc15ea90 con 0x7f49cc15e0b0 2023-03-03T07:57:09.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.338+0000 7f49d13ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0cb580 con 0x7f49cc0b8b90 2023-03-03T07:57:09.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.338+0000 7f49d13ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0ccf90 con 0x7f49cc0cabd0 2023-03-03T07:57:09.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.338+0000 7f49caffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49cc0b8b90 0x7f49cc156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:09.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.338+0000 7f49cb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49cc15e0b0 0x7f49cc15e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:09.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.338+0000 7f49ca7fc700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49cc0cabd0 0x7f49cc0cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:09.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.338+0000 7f49caffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49cc0b8b90 0x7f49cc156950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:40816/0 (socket says 172.21.15.98:40816) 2023-03-03T07:57:09.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.338+0000 7f49caffd700 1 -- 172.21.15.98:0/4255338913 learned_addr learned my addr 172.21.15.98:0/4255338913 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:09.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.339+0000 7f49cb7fe700 1 -- 172.21.15.98:0/4255338913 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49cc0cabd0 msgr2=0x7f49cc0cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:09.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.339+0000 7f49cb7fe700 1 --2- 172.21.15.98:0/4255338913 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49cc0cabd0 0x7f49cc0cb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.339+0000 7f49cb7fe700 1 -- 172.21.15.98:0/4255338913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49cc0b8b90 msgr2=0x7f49cc156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:09.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.339+0000 7f49cb7fe700 1 --2- 172.21.15.98:0/4255338913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49cc0b8b90 0x7f49cc156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.339+0000 7f49cb7fe700 1 -- 172.21.15.98:0/4255338913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc0cdf30 con 0x7f49cc15e0b0 2023-03-03T07:57:09.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.339+0000 7f49cb7fe700 1 --2- 172.21.15.98:0/4255338913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49cc15e0b0 0x7f49cc15e550 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f49cc06c3a0 tx=0x7f49c400a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:09.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.339+0000 7f49b37fe700 1 -- 172.21.15.98:0/4255338913 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c400ad30 con 0x7f49cc15e0b0 2023-03-03T07:57:09.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.339+0000 7f49b37fe700 1 -- 172.21.15.98:0/4255338913 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c40133f0 con 0x7f49cc15e0b0 2023-03-03T07:57:09.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.339+0000 7f49b37fe700 1 -- 172.21.15.98:0/4255338913 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c40137a0 con 0x7f49cc15e0b0 2023-03-03T07:57:09.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.339+0000 7f49d13ff700 1 -- 172.21.15.98:0/4255338913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49cc073a80 con 0x7f49cc15e0b0 2023-03-03T07:57:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.340+0000 7f49d13ff700 1 -- 172.21.15.98:0/4255338913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49cc0ce0c0 con 0x7f49cc15e0b0 2023-03-03T07:57:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.341+0000 7f49d13ff700 1 -- 172.21.15.98:0/4255338913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49cc0729b0 con 0x7f49cc15e0b0 2023-03-03T07:57:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.342+0000 7f49b37fe700 1 -- 172.21.15.98:0/4255338913 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f49c4013900 con 0x7f49cc15e0b0 2023-03-03T07:57:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.344+0000 7f49b37fe700 1 --2- 172.21.15.98:0/4255338913 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49b80682f0 0x7f49b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.344+0000 7f49b37fe700 1 -- 172.21.15.98:0/4255338913 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f49c40862d0 con 0x7f49cc15e0b0 2023-03-03T07:57:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.344+0000 7f49caffd700 1 --2- 172.21.15.98:0/4255338913 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49b80682f0 0x7f49b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:09.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.345+0000 7f49caffd700 1 --2- 172.21.15.98:0/4255338913 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49b80682f0 0x7f49b806a7a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f49bc005040 tx=0x7f49bc00a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:09.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.348+0000 7f49b37fe700 1 -- 172.21.15.98:0/4255338913 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f49c4052d90 con 0x7f49cc15e0b0 2023-03-03T07:57:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.614+0000 7f49d13ff700 1 -- 172.21.15.98:0/4255338913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f49cc04e6e0 con 0x7f49cc15e0b0 2023-03-03T07:57:09.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.615+0000 7f49b37fe700 1 -- 172.21.15.98:0/4255338913 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f49c40563b0 con 0x7f49cc15e0b0 2023-03-03T07:57:09.617 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:09.617 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T07:57:09.617 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_dump": [ 2023-03-03T07:57:09.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.618 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.a", 2023-03-03T07:57:09.618 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkqFdgHxAAuwiGkaNnr5NplmhniDYAEQ==", 2023-03-03T07:57:09.618 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.618 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:57:09.618 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T07:57:09.618 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T07:57:09.619 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:57:09.619 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.620 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.b", 2023-03-03T07:57:09.620 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFk0x8kBhAA5JDo/wmI4650J9tUnvXVzw==", 2023-03-03T07:57:09.620 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.620 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:57:09.620 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T07:57:09.621 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T07:57:09.621 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:57:09.621 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.621 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.c", 2023-03-03T07:57:09.622 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkKoRwMBAAfkWGDUsqNgqfkscl0aBX1A==", 2023-03-03T07:57:09.622 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.622 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:57:09.622 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T07:57:09.622 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T07:57:09.622 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:57:09.623 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.623 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.d", 2023-03-03T07:57:09.623 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFkK9SKFxAAFSCozGfyRwrv46QPitRxTQ==", 2023-03-03T07:57:09.623 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.624 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:57:09.624 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T07:57:09.624 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T07:57:09.624 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:57:09.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.625 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.0", 2023-03-03T07:57:09.625 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDBpwFkTFzTERAAGHrFKuBlXIEfaYD7vs3ZHA==", 2023-03-03T07:57:09.625 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.625 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:57:09.625 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:57:09.625 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:57:09.625 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.1", 2023-03-03T07:57:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDEpwFkVWObLBAAW4acTvNbmscIe18NXVgRYg==", 2023-03-03T07:57:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:57:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:57:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:57:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.2", 2023-03-03T07:57:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDIpwFkK8lgChAAHVD5btBivUpcapSgDp2E0w==", 2023-03-03T07:57:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:57:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:57:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:57:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.3", 2023-03-03T07:57:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDLpwFksrb8IxAAN+CP1O3DJ1zDy+qyGNor2Q==", 2023-03-03T07:57:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:57:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:57:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:57:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.4", 2023-03-03T07:57:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDSpwFkkWGHIxAA4wejA0w05JPIUghEAVG/Rg==", 2023-03-03T07:57:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:57:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:57:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:57:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.5", 2023-03-03T07:57:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDWpwFknHmGAhAA6VM16luaykZfJEzr+yDZZA==", 2023-03-03T07:57:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:57:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:57:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:57:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.6", 2023-03-03T07:57:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDZpwFkcitNDRAAOz04YGTzx3Phenp0n8JeEA==", 2023-03-03T07:57:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:57:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:57:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:57:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.7", 2023-03-03T07:57:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDcpwFkrjmIFxAAbWeB2+8VQgyTplW+fk47OA==", 2023-03-03T07:57:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:57:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:57:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:57:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.0", 2023-03-03T07:57:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFk02lsJhAA0cH/evMnWdZN6DdBGCLjjg==", 2023-03-03T07:57:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:57:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T07:57:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T07:57:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T07:57:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.1", 2023-03-03T07:57:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC9pwFkRqXrABAArm/01O5MtSTQHWQWKqNQEg==", 2023-03-03T07:57:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:57:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T07:57:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T07:57:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T07:57:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.2", 2023-03-03T07:57:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFkyKp3MRAA8Eh41TZNsO6/8VhAWCpNMQ==", 2023-03-03T07:57:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:57:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T07:57:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T07:57:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T07:57:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.3", 2023-03-03T07:57:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC9pwFk14bSCxAAy2nXc9G50W4lH5GtH84hpQ==", 2023-03-03T07:57:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:57:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T07:57:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T07:57:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T07:57:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.admin", 2023-03-03T07:57:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC6pwFkokmiEBAAFwi8rhPmk4Zyk95p760HDg==", 2023-03-03T07:57:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T07:57:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T07:57:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T07:57:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:57:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.bootstrap-mds", 2023-03-03T07:57:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDlpwFkRmMBJxAAGDl5ZJtycs1NOZjUdRaXKg==", 2023-03-03T07:57:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile bootstrap-mds" 2023-03-03T07:57:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.bootstrap-mgr", 2023-03-03T07:57:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDlpwFkFHABJxAA/aZ0doRNSBRnnkuWvStU3w==", 2023-03-03T07:57:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile bootstrap-mgr" 2023-03-03T07:57:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.bootstrap-osd", 2023-03-03T07:57:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDlpwFk4XkBJxAAKi7cJ9fTcxMBPeNXp9cPJg==", 2023-03-03T07:57:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile bootstrap-osd" 2023-03-03T07:57:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.bootstrap-rbd", 2023-03-03T07:57:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDlpwFkdYMBJxAAsGS9sknleQ5NvocZZMeQIw==", 2023-03-03T07:57:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile bootstrap-rbd" 2023-03-03T07:57:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-03-03T07:57:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDlpwFkH40BJxAAzVB6nJEXg9pPJUXpOXmHPQ==", 2023-03-03T07:57:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-03-03T07:57:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.bootstrap-rgw", 2023-03-03T07:57:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDlpwFkrJcBJxAAh85BF2Q+nraw7xlpORl/SA==", 2023-03-03T07:57:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile bootstrap-rgw" 2023-03-03T07:57:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mgr.x", 2023-03-03T07:57:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkvdMQFBAAMi+x7psn5CDpd//d6jN2VA==", 2023-03-03T07:57:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T07:57:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile mgr", 2023-03-03T07:57:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:57:09.649 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:09.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:09.649 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mgr.y", 2023-03-03T07:57:09.649 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkfYO5BxAAFW/elbdoqVXBkUIKrAvYsg==", 2023-03-03T07:57:09.649 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:57:09.649 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T07:57:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile mgr", 2023-03-03T07:57:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:57:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:09.650 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T07:57:09.650 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T07:57:09.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.617+0000 7f49d13ff700 1 -- 172.21.15.98:0/4255338913 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49b80682f0 msgr2=0x7f49b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:09.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.617+0000 7f49d13ff700 1 --2- 172.21.15.98:0/4255338913 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49b80682f0 0x7f49b806a7a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f49bc005040 tx=0x7f49bc00a000 comp rx=0 tx=0).stop 2023-03-03T07:57:09.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.617+0000 7f49d13ff700 1 -- 172.21.15.98:0/4255338913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49cc15e0b0 msgr2=0x7f49cc15e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:09.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.617+0000 7f49d13ff700 1 --2- 172.21.15.98:0/4255338913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49cc15e0b0 0x7f49cc15e550 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f49cc06c3a0 tx=0x7f49c400a5b0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.618+0000 7f49d13ff700 1 -- 172.21.15.98:0/4255338913 shutdown_connections 2023-03-03T07:57:09.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.618+0000 7f49d13ff700 1 --2- 172.21.15.98:0/4255338913 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49b80682f0 0x7f49b806a7a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.618+0000 7f49d13ff700 1 --2- 172.21.15.98:0/4255338913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49cc15e0b0 0x7f49cc15e550 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.618+0000 7f49d13ff700 1 --2- 172.21.15.98:0/4255338913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49cc0b8b90 0x7f49cc156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.618+0000 7f49d13ff700 1 --2- 172.21.15.98:0/4255338913 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49cc0cabd0 0x7f49cc0cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.618+0000 7f49d13ff700 1 -- 172.21.15.98:0/4255338913 >> 172.21.15.98:0/4255338913 conn(0x7f49cc06aee0 msgr2=0x7f49cc0b75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:09.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.618+0000 7f49d13ff700 1 -- 172.21.15.98:0/4255338913 shutdown_connections 2023-03-03T07:57:09.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.618+0000 7f49d13ff700 1 -- 172.21.15.98:0/4255338913 wait complete. 2023-03-03T07:57:09.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-03-03T07:57:09.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.903+0000 7f9141dff700 1 Processor -- start 2023-03-03T07:57:09.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.903+0000 7f9141dff700 1 -- start start 2023-03-03T07:57:09.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.904+0000 7f9141dff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f913c15ca70 0x7f913c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:09.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.905+0000 7f9141dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c0cf980 con 0x7f913c0d0270 2023-03-03T07:57:09.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.905+0000 7f9141dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c0d1800 con 0x7f913c06b4b0 2023-03-03T07:57:09.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.905+0000 7f9141dff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913c0cd5b0 con 0x7f913c15ca70 2023-03-03T07:57:09.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.905+0000 7f913b7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f913c15ca70 0x7f913c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:09.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.905+0000 7f913bfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f913c0d0270 0x7f913c0d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55104/0 (socket says 172.21.15.98:55104) 2023-03-03T07:57:09.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.905+0000 7f913bfff700 1 -- 172.21.15.98:0/525769331 learned_addr learned my addr 172.21.15.98:0/525769331 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:09.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.905+0000 7f913b7fe700 1 -- 172.21.15.98:0/525769331 >> v1:172.21.15.100:6789/0 conn(0x7f913c06b4b0 legacy=0x7f913c0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:09.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.905+0000 7f913b7fe700 1 -- 172.21.15.98:0/525769331 >> v1:172.21.15.98:6789/0 conn(0x7f913c0d0270 legacy=0x7f913c0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:09.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.906+0000 7f913b7fe700 1 -- 172.21.15.98:0/525769331 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c0cd900 con 0x7f913c15ca70 2023-03-03T07:57:09.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.905+0000 7f913affd700 1 -- 172.21.15.98:0/525769331 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530740624 0 0) 0x7f913c0cf980 con 0x7f913c0d0270 2023-03-03T07:57:09.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.906+0000 7f913b7fe700 1 --2- 172.21.15.98:0/525769331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f913c15ca70 0x7f913c0cf440 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f912c009840 tx=0x7f912c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3282c57504d8a9ee server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:09.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.906+0000 7f913affd700 1 -- 172.21.15.98:0/525769331 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c011830 con 0x7f913c15ca70 2023-03-03T07:57:09.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.906+0000 7f913affd700 1 -- 172.21.15.98:0/525769331 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912c011990 con 0x7f913c15ca70 2023-03-03T07:57:09.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.906+0000 7f913affd700 1 -- 172.21.15.98:0/525769331 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c011d20 con 0x7f913c15ca70 2023-03-03T07:57:09.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.907+0000 7f9141dff700 1 -- 172.21.15.98:0/525769331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f913c15ca70 msgr2=0x7f913c0cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:09.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.907+0000 7f9141dff700 1 --2- 172.21.15.98:0/525769331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f913c15ca70 0x7f913c0cf440 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f912c009840 tx=0x7f912c009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:09.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.907+0000 7f9141dff700 1 -- 172.21.15.98:0/525769331 shutdown_connections 2023-03-03T07:57:09.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.907+0000 7f9141dff700 1 --2- 172.21.15.98:0/525769331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f913c15ca70 0x7f913c0cf440 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.907+0000 7f9141dff700 1 -- 172.21.15.98:0/525769331 >> 172.21.15.98:0/525769331 conn(0x7f913c073e70 msgr2=0x7f913c06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:09.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.907+0000 7f9141dff700 1 -- 172.21.15.98:0/525769331 shutdown_connections 2023-03-03T07:57:09.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.907+0000 7f9141dff700 1 -- 172.21.15.98:0/525769331 wait complete. 2023-03-03T07:57:09.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.909+0000 7f9141dff700 1 Processor -- start 2023-03-03T07:57:09.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.909+0000 7f9141dff700 1 -- start start 2023-03-03T07:57:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.910+0000 7f9141dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f913c06b4b0 0x7f913c15a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.910+0000 7f9141dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f913c0d0270 0x7f913c0b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.910+0000 7f9141dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f913c15ca70 0x7f913c0b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.910+0000 7f9141dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c15d090 con 0x7f913c15ca70 2023-03-03T07:57:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.910+0000 7f9141dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0ab480 con 0x7f913c0d0270 2023-03-03T07:57:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.910+0000 7f9141dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c15ac90 con 0x7f913c06b4b0 2023-03-03T07:57:09.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.910+0000 7f913bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f913c15ca70 0x7f913c0b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:09.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.910+0000 7f913bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f913c15ca70 0x7f913c0b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47826/0 (socket says 172.21.15.98:47826) 2023-03-03T07:57:09.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.910+0000 7f913bfff700 1 -- 172.21.15.98:0/417618679 learned_addr learned my addr 172.21.15.98:0/417618679 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:09.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.910+0000 7f913b7fe700 1 --2- 172.21.15.98:0/417618679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f913c06b4b0 0x7f913c15a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:09.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.910+0000 7f9133fff700 1 --2- 172.21.15.98:0/417618679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f913c0d0270 0x7f913c0b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:09.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.911+0000 7f913bfff700 1 -- 172.21.15.98:0/417618679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f913c06b4b0 msgr2=0x7f913c15a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:09.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.911+0000 7f913bfff700 1 --2- 172.21.15.98:0/417618679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f913c06b4b0 0x7f913c15a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.911+0000 7f913bfff700 1 -- 172.21.15.98:0/417618679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f913c0d0270 msgr2=0x7f913c0b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:09.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.911+0000 7f913bfff700 1 --2- 172.21.15.98:0/417618679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f913c0d0270 0x7f913c0b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:09.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.911+0000 7f913bfff700 1 -- 172.21.15.98:0/417618679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912c0094a0 con 0x7f913c15ca70 2023-03-03T07:57:09.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.911+0000 7f913bfff700 1 --2- 172.21.15.98:0/417618679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f913c15ca70 0x7f913c0b3c90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f913400cbe0 tx=0x7f913400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:09.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.911+0000 7f9138ff9700 1 -- 172.21.15.98:0/417618679 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913400aaa0 con 0x7f913c15ca70 2023-03-03T07:57:09.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.911+0000 7f9138ff9700 1 -- 172.21.15.98:0/417618679 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913400ac20 con 0x7f913c15ca70 2023-03-03T07:57:09.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.911+0000 7f9141dff700 1 -- 172.21.15.98:0/417618679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913c0d1aa0 con 0x7f913c15ca70 2023-03-03T07:57:09.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.911+0000 7f9138ff9700 1 -- 172.21.15.98:0/417618679 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91340070d0 con 0x7f913c15ca70 2023-03-03T07:57:09.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.912+0000 7f9141dff700 1 -- 172.21.15.98:0/417618679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c0cdbe0 con 0x7f913c15ca70 2023-03-03T07:57:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.913+0000 7f9141dff700 1 -- 172.21.15.98:0/417618679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913c15ce40 con 0x7f913c15ca70 2023-03-03T07:57:09.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.914+0000 7f9138ff9700 1 -- 172.21.15.98:0/417618679 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9134010070 con 0x7f913c15ca70 2023-03-03T07:57:09.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.915+0000 7f9138ff9700 1 --2- 172.21.15.98:0/417618679 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f911c0682f0 0x7f911c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:09.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.915+0000 7f9138ff9700 1 -- 172.21.15.98:0/417618679 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9134085370 con 0x7f913c15ca70 2023-03-03T07:57:09.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.916+0000 7f913b7fe700 1 --2- 172.21.15.98:0/417618679 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f911c0682f0 0x7f911c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:09.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.916+0000 7f913b7fe700 1 --2- 172.21.15.98:0/417618679 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f911c0682f0 0x7f911c06a7a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f912c014040 tx=0x7f912c01c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:09.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:09.919+0000 7f9138ff9700 1 -- 172.21.15.98:0/417618679 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9134051e30 con 0x7f913c15ca70 2023-03-03T07:57:10.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.186+0000 7f9141dff700 1 -- 172.21.15.98:0/417618679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f913c04e6e0 con 0x7f913c15ca70 2023-03-03T07:57:10.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.190+0000 7f9138ff9700 1 -- 172.21.15.98:0/417618679 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) v1 ==== 90+0+0 (secure 0 0 0) 0x7f91340180a0 con 0x7f913c15ca70 2023-03-03T07:57:10.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.193+0000 7f9141dff700 1 -- 172.21.15.98:0/417618679 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f911c0682f0 msgr2=0x7f911c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:10.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.193+0000 7f9141dff700 1 --2- 172.21.15.98:0/417618679 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f911c0682f0 0x7f911c06a7a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f912c014040 tx=0x7f912c01c040 comp rx=0 tx=0).stop 2023-03-03T07:57:10.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.193+0000 7f9141dff700 1 -- 172.21.15.98:0/417618679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f913c15ca70 msgr2=0x7f913c0b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:10.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.193+0000 7f9141dff700 1 --2- 172.21.15.98:0/417618679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f913c15ca70 0x7f913c0b3c90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f913400cbe0 tx=0x7f913400cef0 comp rx=0 tx=0).stop 2023-03-03T07:57:10.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.193+0000 7f9141dff700 1 -- 172.21.15.98:0/417618679 shutdown_connections 2023-03-03T07:57:10.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.193+0000 7f9141dff700 1 --2- 172.21.15.98:0/417618679 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f911c0682f0 0x7f911c06a7a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:10.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.193+0000 7f9141dff700 1 --2- 172.21.15.98:0/417618679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f913c15ca70 0x7f913c0b3c90 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:10.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.193+0000 7f9141dff700 1 --2- 172.21.15.98:0/417618679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f913c06b4b0 0x7f913c15a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:10.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.193+0000 7f9141dff700 1 --2- 172.21.15.98:0/417618679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f913c0d0270 0x7f913c0b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:10.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.193+0000 7f9141dff700 1 -- 172.21.15.98:0/417618679 >> 172.21.15.98:0/417618679 conn(0x7f913c073e70 msgr2=0x7f913c0c9840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:10.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.193+0000 7f9141dff700 1 -- 172.21.15.98:0/417618679 shutdown_connections 2023-03-03T07:57:10.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.193+0000 7f9141dff700 1 -- 172.21.15.98:0/417618679 wait complete. 2023-03-03T07:57:10.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-03-03T07:57:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.465+0000 7fe8dffff700 1 Processor -- start 2023-03-03T07:57:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.466+0000 7fe8dffff700 1 -- start start 2023-03-03T07:57:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.467+0000 7fe8dffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe8e01609b0 0x7fe8e0160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.467+0000 7fe8dffff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e01612c0 con 0x7fe8e00cf570 2023-03-03T07:57:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.467+0000 7fe8dffff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e015dd00 con 0x7fe8e00b4aa0 2023-03-03T07:57:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.467+0000 7fe8dffff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e00cebb0 con 0x7fe8e01609b0 2023-03-03T07:57:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.467+0000 7fe8d7fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe8e00cf570 0x7fe8e015d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55112/0 (socket says 172.21.15.98:55112) 2023-03-03T07:57:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.467+0000 7fe8d7fff700 1 -- 172.21.15.98:0/1220714070 learned_addr learned my addr 172.21.15.98:0/1220714070 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.467+0000 7fe8deffd700 1 --2- 172.21.15.98:0/1220714070 >> v2:172.21.15.98:3300/0 conn(0x7fe8e01609b0 0x7fe8e0160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.468+0000 7fe8de7fc700 1 -- 172.21.15.98:0/1220714070 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237405291 0 0) 0x7fe8e01612c0 con 0x7fe8e00cf570 2023-03-03T07:57:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.468+0000 7fe8de7fc700 1 -- 172.21.15.98:0/1220714070 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8cc0035b0 con 0x7fe8e00cf570 2023-03-03T07:57:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.468+0000 7fe8deffd700 1 -- 172.21.15.98:0/1220714070 >> v1:172.21.15.100:6789/0 conn(0x7fe8e00b4aa0 legacy=0x7fe8e00d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.468+0000 7fe8deffd700 1 -- 172.21.15.98:0/1220714070 >> v1:172.21.15.98:6789/0 conn(0x7fe8e00cf570 legacy=0x7fe8e015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.468+0000 7fe8deffd700 1 -- 172.21.15.98:0/1220714070 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e00cd780 con 0x7fe8e01609b0 2023-03-03T07:57:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.468+0000 7fe8deffd700 1 --2- 172.21.15.98:0/1220714070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8e01609b0 0x7fe8e0160d80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0009840 tx=0x7fe8d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6601346e1a6b661 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.469+0000 7fe8de7fc700 1 -- 172.21.15.98:0/1220714070 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0011830 con 0x7fe8e01609b0 2023-03-03T07:57:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.469+0000 7fe8de7fc700 1 -- 172.21.15.98:0/1220714070 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d0011990 con 0x7fe8e01609b0 2023-03-03T07:57:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.469+0000 7fe8de7fc700 1 -- 172.21.15.98:0/1220714070 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0011d20 con 0x7fe8e01609b0 2023-03-03T07:57:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.469+0000 7fe8dffff700 1 -- 172.21.15.98:0/1220714070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8e01609b0 msgr2=0x7fe8e0160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.469+0000 7fe8dffff700 1 --2- 172.21.15.98:0/1220714070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8e01609b0 0x7fe8e0160d80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0009840 tx=0x7fe8d0009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.469+0000 7fe8dffff700 1 -- 172.21.15.98:0/1220714070 shutdown_connections 2023-03-03T07:57:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.469+0000 7fe8dffff700 1 --2- 172.21.15.98:0/1220714070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8e01609b0 0x7fe8e0160d80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.469+0000 7fe8dffff700 1 -- 172.21.15.98:0/1220714070 >> 172.21.15.98:0/1220714070 conn(0x7fe8e006b880 msgr2=0x7fe8e00cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.469+0000 7fe8dffff700 1 -- 172.21.15.98:0/1220714070 shutdown_connections 2023-03-03T07:57:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.469+0000 7fe8dffff700 1 -- 172.21.15.98:0/1220714070 wait complete. 2023-03-03T07:57:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.471+0000 7fe8dffff700 1 Processor -- start 2023-03-03T07:57:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.471+0000 7fe8dffff700 1 -- start start 2023-03-03T07:57:10.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.472+0000 7fe8dffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8e00b4aa0 0x7fe8e01f67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.472+0000 7fe8dffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e00cf570 0x7fe8e014f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.473+0000 7fe8dffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e01609b0 0x7fe8e01516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.473+0000 7fe8dffff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e01f2810 con 0x7fe8e00b4aa0 2023-03-03T07:57:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.473+0000 7fe8dffff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e01f6d30 con 0x7fe8e00cf570 2023-03-03T07:57:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.473+0000 7fe8dffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e01f2a50 con 0x7fe8e01609b0 2023-03-03T07:57:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.473+0000 7fe8deffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8e00b4aa0 0x7fe8e01f67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.473+0000 7fe8d7fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e00cf570 0x7fe8e014f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.473+0000 7fe8df7fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e01609b0 0x7fe8e01516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.473+0000 7fe8deffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8e00b4aa0 0x7fe8e01f67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47840/0 (socket says 172.21.15.98:47840) 2023-03-03T07:57:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.473+0000 7fe8deffd700 1 -- 172.21.15.98:0/1844048089 learned_addr learned my addr 172.21.15.98:0/1844048089 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.473+0000 7fe8deffd700 1 -- 172.21.15.98:0/1844048089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e01609b0 msgr2=0x7fe8e01516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.473+0000 7fe8deffd700 1 --2- 172.21.15.98:0/1844048089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e01609b0 0x7fe8e01516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.473+0000 7fe8deffd700 1 -- 172.21.15.98:0/1844048089 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e00cf570 msgr2=0x7fe8e014f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.473+0000 7fe8deffd700 1 --2- 172.21.15.98:0/1844048089 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e00cf570 0x7fe8e014f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.473+0000 7fe8deffd700 1 -- 172.21.15.98:0/1844048089 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d00094a0 con 0x7fe8e00b4aa0 2023-03-03T07:57:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.474+0000 7fe8deffd700 1 --2- 172.21.15.98:0/1844048089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8e00b4aa0 0x7fe8e01f67f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00090b0 tx=0x7fe8d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.474+0000 7fe8dcff9700 1 -- 172.21.15.98:0/1844048089 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d00036a0 con 0x7fe8e00b4aa0 2023-03-03T07:57:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.474+0000 7fe8dcff9700 1 -- 172.21.15.98:0/1844048089 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d001e070 con 0x7fe8e00b4aa0 2023-03-03T07:57:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.474+0000 7fe8dcff9700 1 -- 172.21.15.98:0/1844048089 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0003a60 con 0x7fe8e00b4aa0 2023-03-03T07:57:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.474+0000 7fe8dffff700 1 -- 172.21.15.98:0/1844048089 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e00cdd40 con 0x7fe8e00b4aa0 2023-03-03T07:57:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.474+0000 7fe8dffff700 1 -- 172.21.15.98:0/1844048089 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e00cda60 con 0x7fe8e00b4aa0 2023-03-03T07:57:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.475+0000 7fe8dffff700 1 -- 172.21.15.98:0/1844048089 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e015cdf0 con 0x7fe8e00b4aa0 2023-03-03T07:57:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.476+0000 7fe8dcff9700 1 -- 172.21.15.98:0/1844048089 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe8d000f050 con 0x7fe8e00b4aa0 2023-03-03T07:57:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.478+0000 7fe8dcff9700 1 --2- 172.21.15.98:0/1844048089 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8c00682f0 0x7fe8c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.478+0000 7fe8dcff9700 1 -- 172.21.15.98:0/1844048089 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe8d0086910 con 0x7fe8e00b4aa0 2023-03-03T07:57:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.478+0000 7fe8d7fff700 1 --2- 172.21.15.98:0/1844048089 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8c00682f0 0x7fe8c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:10.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.479+0000 7fe8d7fff700 1 --2- 172.21.15.98:0/1844048089 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8c00682f0 0x7fe8c006a7a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00d2500 tx=0x7fe8c8007dc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:10.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.482+0000 7fe8dcff9700 1 -- 172.21.15.98:0/1844048089 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe8d00533d0 con 0x7fe8e00b4aa0 2023-03-03T07:57:10.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.749+0000 7fe8dffff700 1 -- 172.21.15.98:0/1844048089 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fe8e0068290 con 0x7fe8e00b4aa0 2023-03-03T07:57:10.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.753+0000 7fe8dcff9700 1 -- 172.21.15.98:0/1844048089 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe8d0020070 con 0x7fe8e00b4aa0 2023-03-03T07:57:10.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.755+0000 7fe8dffff700 1 -- 172.21.15.98:0/1844048089 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8c00682f0 msgr2=0x7fe8c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:10.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.755+0000 7fe8dffff700 1 --2- 172.21.15.98:0/1844048089 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8c00682f0 0x7fe8c006a7a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00d2500 tx=0x7fe8c8007dc0 comp rx=0 tx=0).stop 2023-03-03T07:57:10.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.755+0000 7fe8dffff700 1 -- 172.21.15.98:0/1844048089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8e00b4aa0 msgr2=0x7fe8e01f67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:10.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.755+0000 7fe8dffff700 1 --2- 172.21.15.98:0/1844048089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8e00b4aa0 0x7fe8e01f67f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00090b0 tx=0x7fe8d0004750 comp rx=0 tx=0).stop 2023-03-03T07:57:10.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.755+0000 7fe8dffff700 1 -- 172.21.15.98:0/1844048089 shutdown_connections 2023-03-03T07:57:10.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.755+0000 7fe8dffff700 1 --2- 172.21.15.98:0/1844048089 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8c00682f0 0x7fe8c006a7a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:10.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.755+0000 7fe8dffff700 1 --2- 172.21.15.98:0/1844048089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8e00b4aa0 0x7fe8e01f67f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:10.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.755+0000 7fe8dffff700 1 --2- 172.21.15.98:0/1844048089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e01609b0 0x7fe8e01516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:10.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.755+0000 7fe8dffff700 1 --2- 172.21.15.98:0/1844048089 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e00cf570 0x7fe8e014f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:10.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.755+0000 7fe8dffff700 1 -- 172.21.15.98:0/1844048089 >> 172.21.15.98:0/1844048089 conn(0x7fe8e006b880 msgr2=0x7fe8e015e790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:10.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.756+0000 7fe8dffff700 1 -- 172.21.15.98:0/1844048089 shutdown_connections 2023-03-03T07:57:10.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:10.756+0000 7fe8dffff700 1 -- 172.21.15.98:0/1844048089 wait complete. 2023-03-03T07:57:10.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-03-03T07:57:11.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.031+0000 7f230abff700 1 Processor -- start 2023-03-03T07:57:11.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.031+0000 7f230abff700 1 -- start start 2023-03-03T07:57:11.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.033+0000 7f230abff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f23040b8d10 0x7f23040cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.033+0000 7f230abff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23040cd7d0 con 0x7f23040b5310 2023-03-03T07:57:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.033+0000 7f230abff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23040b90e0 con 0x7f23040b26d0 2023-03-03T07:57:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.033+0000 7f230abff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23040cda20 con 0x7f23040b8d10 2023-03-03T07:57:11.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.034+0000 7f230919c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f23040b5310 0x7f23040d13a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55114/0 (socket says 172.21.15.98:55114) 2023-03-03T07:57:11.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.034+0000 7f230899b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f23040b8d10 0x7f23040cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.034+0000 7f230919c700 1 -- 172.21.15.98:0/2660188908 learned_addr learned my addr 172.21.15.98:0/2660188908 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.034+0000 7f23037fe700 1 -- 172.21.15.98:0/2660188908 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8673320 0 0) 0x7f23040cd7d0 con 0x7f23040b5310 2023-03-03T07:57:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.034+0000 7f23037fe700 1 -- 172.21.15.98:0/2660188908 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f40035b0 con 0x7f23040b5310 2023-03-03T07:57:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.034+0000 7f230899b700 1 -- 172.21.15.98:0/2660188908 >> v1:172.21.15.100:6789/0 conn(0x7f23040b26d0 legacy=0x7f23040b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.034+0000 7f230899b700 1 -- 172.21.15.98:0/2660188908 >> v1:172.21.15.98:6789/0 conn(0x7f23040b5310 legacy=0x7f23040d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.034+0000 7f230899b700 1 -- 172.21.15.98:0/2660188908 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23040cdd00 con 0x7f23040b8d10 2023-03-03T07:57:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.035+0000 7f230899b700 1 --2- 172.21.15.98:0/2660188908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23040b8d10 0x7f23040cffc0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009840 tx=0x7f22f8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2f9684f3c9eed6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.035+0000 7f23037fe700 1 -- 172.21.15.98:0/2660188908 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f8011830 con 0x7f23040b8d10 2023-03-03T07:57:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.035+0000 7f23037fe700 1 -- 172.21.15.98:0/2660188908 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f8011990 con 0x7f23040b8d10 2023-03-03T07:57:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.035+0000 7f23037fe700 1 -- 172.21.15.98:0/2660188908 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f8011d20 con 0x7f23040b8d10 2023-03-03T07:57:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.035+0000 7f230abff700 1 -- 172.21.15.98:0/2660188908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23040b8d10 msgr2=0x7f23040cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.035+0000 7f230abff700 1 --2- 172.21.15.98:0/2660188908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23040b8d10 0x7f23040cffc0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009840 tx=0x7f22f8009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.036+0000 7f230abff700 1 -- 172.21.15.98:0/2660188908 shutdown_connections 2023-03-03T07:57:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.036+0000 7f230abff700 1 --2- 172.21.15.98:0/2660188908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23040b8d10 0x7f23040cffc0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.036+0000 7f230abff700 1 -- 172.21.15.98:0/2660188908 >> 172.21.15.98:0/2660188908 conn(0x7f2304073e70 msgr2=0x7f230406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.036+0000 7f230abff700 1 -- 172.21.15.98:0/2660188908 shutdown_connections 2023-03-03T07:57:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.036+0000 7f230abff700 1 -- 172.21.15.98:0/2660188908 wait complete. 2023-03-03T07:57:11.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.038+0000 7f230abff700 1 Processor -- start 2023-03-03T07:57:11.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.038+0000 7f230abff700 1 -- start start 2023-03-03T07:57:11.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.039+0000 7f230abff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23040b26d0 0x7f23040ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:11.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.039+0000 7f230abff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23040b5310 0x7f23041553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:11.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.039+0000 7f230abff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23040b8d10 0x7f23041558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:11.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.039+0000 7f230abff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2304157ca0 con 0x7f23040b26d0 2023-03-03T07:57:11.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.039+0000 7f230abff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23041f63b0 con 0x7f23040b8d10 2023-03-03T07:57:11.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.039+0000 7f230abff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2304157eb0 con 0x7f23040b5310 2023-03-03T07:57:11.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.039+0000 7f230899b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23040b26d0 0x7f23040ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:11.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.039+0000 7f230899b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23040b26d0 0x7f23040ccdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47854/0 (socket says 172.21.15.98:47854) 2023-03-03T07:57:11.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.039+0000 7f230899b700 1 -- 172.21.15.98:0/2996516377 learned_addr learned my addr 172.21.15.98:0/2996516377 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:11.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.039+0000 7f230899b700 1 -- 172.21.15.98:0/2996516377 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23040b5310 msgr2=0x7f23041553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:11.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.040+0000 7f230899b700 1 --2- 172.21.15.98:0/2996516377 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23040b5310 0x7f23041553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.040+0000 7f230899b700 1 -- 172.21.15.98:0/2996516377 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23040b8d10 msgr2=0x7f23041558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:11.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.040+0000 7f230899b700 1 --2- 172.21.15.98:0/2996516377 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23040b8d10 0x7f23041558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.040+0000 7f230899b700 1 -- 172.21.15.98:0/2996516377 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f80094a0 con 0x7f23040b26d0 2023-03-03T07:57:11.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.040+0000 7f230899b700 1 --2- 172.21.15.98:0/2996516377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23040b26d0 0x7f23040ccdf0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009fd0 tx=0x7f22f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:11.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.040+0000 7f23017fa700 1 -- 172.21.15.98:0/2996516377 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f80038a0 con 0x7f23040b26d0 2023-03-03T07:57:11.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.040+0000 7f23017fa700 1 -- 172.21.15.98:0/2996516377 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f801f070 con 0x7f23040b26d0 2023-03-03T07:57:11.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.040+0000 7f23017fa700 1 -- 172.21.15.98:0/2996516377 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f8003c40 con 0x7f23040b26d0 2023-03-03T07:57:11.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.040+0000 7f230abff700 1 -- 172.21.15.98:0/2996516377 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23040cdfe0 con 0x7f23040b26d0 2023-03-03T07:57:11.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.040+0000 7f230abff700 1 -- 172.21.15.98:0/2996516377 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23040d0e80 con 0x7f23040b26d0 2023-03-03T07:57:11.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.042+0000 7f230abff700 1 -- 172.21.15.98:0/2996516377 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23040d0500 con 0x7f23040b26d0 2023-03-03T07:57:11.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.043+0000 7f23017fa700 1 -- 172.21.15.98:0/2996516377 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f22f800f050 con 0x7f23040b26d0 2023-03-03T07:57:11.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.044+0000 7f23017fa700 1 --2- 172.21.15.98:0/2996516377 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f22e8068320 0x7f22e806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:11.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.044+0000 7f23017fa700 1 -- 172.21.15.98:0/2996516377 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f22f8021070 con 0x7f23040b26d0 2023-03-03T07:57:11.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.044+0000 7f2303fff700 1 --2- 172.21.15.98:0/2996516377 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f22e8068320 0x7f22e806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:11.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.045+0000 7f2303fff700 1 --2- 172.21.15.98:0/2996516377 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f22e8068320 0x7f22e806a7d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f22f0000f60 tx=0x7f22f0007a40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:11.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.048+0000 7f23017fa700 1 -- 172.21.15.98:0/2996516377 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f22f8053e30 con 0x7f23040b26d0 2023-03-03T07:57:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.317+0000 7f230abff700 1 -- 172.21.15.98:0/2996516377 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f230406b7a0 con 0x7f23040b26d0 2023-03-03T07:57:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.321+0000 7f23017fa700 1 -- 172.21.15.98:0/2996516377 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f22f8057450 con 0x7f23040b26d0 2023-03-03T07:57:11.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.324+0000 7f230abff700 1 -- 172.21.15.98:0/2996516377 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f22e8068320 msgr2=0x7f22e806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:11.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.324+0000 7f230abff700 1 --2- 172.21.15.98:0/2996516377 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f22e8068320 0x7f22e806a7d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f22f0000f60 tx=0x7f22f0007a40 comp rx=0 tx=0).stop 2023-03-03T07:57:11.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.324+0000 7f230abff700 1 -- 172.21.15.98:0/2996516377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23040b26d0 msgr2=0x7f23040ccdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.324+0000 7f230abff700 1 --2- 172.21.15.98:0/2996516377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23040b26d0 0x7f23040ccdf0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009fd0 tx=0x7f22f8004750 comp rx=0 tx=0).stop 2023-03-03T07:57:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.324+0000 7f230abff700 1 -- 172.21.15.98:0/2996516377 shutdown_connections 2023-03-03T07:57:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.324+0000 7f230abff700 1 --2- 172.21.15.98:0/2996516377 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f22e8068320 0x7f22e806a7d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.324+0000 7f230abff700 1 --2- 172.21.15.98:0/2996516377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23040b26d0 0x7f23040ccdf0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.324+0000 7f230abff700 1 --2- 172.21.15.98:0/2996516377 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23040b8d10 0x7f23041558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.324+0000 7f230abff700 1 --2- 172.21.15.98:0/2996516377 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23040b5310 0x7f23041553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.324+0000 7f230abff700 1 -- 172.21.15.98:0/2996516377 >> 172.21.15.98:0/2996516377 conn(0x7f2304073e70 msgr2=0x7f230406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.324+0000 7f230abff700 1 -- 172.21.15.98:0/2996516377 shutdown_connections 2023-03-03T07:57:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.324+0000 7f230abff700 1 -- 172.21.15.98:0/2996516377 wait complete. 2023-03-03T07:57:11.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-03-03T07:57:11.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.599+0000 7f462bbff700 1 Processor -- start 2023-03-03T07:57:11.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.599+0000 7f462bbff700 1 -- start start 2023-03-03T07:57:11.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.600+0000 7f462bbff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f46240b8d10 0x7f46240cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:11.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.600+0000 7f462bbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46240b26d0 0x7f46240b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:11.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.600+0000 7f462bbff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46240cd7d0 con 0x7f46240b5140 2023-03-03T07:57:11.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.600+0000 7f462bbff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46240b55c0 con 0x7f46240b8d10 2023-03-03T07:57:11.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.600+0000 7f462bbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46240b90e0 con 0x7f46240b26d0 2023-03-03T07:57:11.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.600+0000 7f462a19c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f46240b5140 0x7f4624160520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55116/0 (socket says 172.21.15.98:55116) 2023-03-03T07:57:11.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.600+0000 7f462a19c700 1 -- 172.21.15.98:0/8357535 learned_addr learned my addr 172.21.15.98:0/8357535 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:11.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.601+0000 7f462919a700 1 --2- 172.21.15.98:0/8357535 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46240b26d0 0x7f46240b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.601+0000 7f462999b700 1 --2- 172.21.15.98:0/8357535 >> v2:172.21.15.100:3300/0 conn(0x7f46240b8d10 0x7f46240cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.601+0000 7f4628999700 1 -- 172.21.15.98:0/8357535 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190636074 0 0) 0x7f46240cd7d0 con 0x7f46240b5140 2023-03-03T07:57:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.601+0000 7f4628999700 1 -- 172.21.15.98:0/8357535 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46100035b0 con 0x7f46240b5140 2023-03-03T07:57:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.601+0000 7f462919a700 1 -- 172.21.15.98:0/8357535 >> v2:172.21.15.100:3300/0 conn(0x7f46240b8d10 msgr2=0x7f46240cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.601+0000 7f462919a700 1 --2- 172.21.15.98:0/8357535 >> v2:172.21.15.100:3300/0 conn(0x7f46240b8d10 0x7f46240cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.601+0000 7f462919a700 1 -- 172.21.15.98:0/8357535 >> v1:172.21.15.98:6789/0 conn(0x7f46240b5140 legacy=0x7f4624160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:11.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.601+0000 7f462919a700 1 -- 172.21.15.98:0/8357535 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46240cda20 con 0x7f46240b26d0 2023-03-03T07:57:11.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.602+0000 7f462919a700 1 --2- 172.21.15.98:0/8357535 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46240b26d0 0x7f46240b4c00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4614009840 tx=0x7f4614009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bab1f2c406ad4559 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:11.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.602+0000 7f4628999700 1 -- 172.21.15.98:0/8357535 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4614011830 con 0x7f46240b26d0 2023-03-03T07:57:11.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.602+0000 7f4628999700 1 -- 172.21.15.98:0/8357535 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4614011990 con 0x7f46240b26d0 2023-03-03T07:57:11.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.602+0000 7f4628999700 1 -- 172.21.15.98:0/8357535 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4614011d20 con 0x7f46240b26d0 2023-03-03T07:57:11.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.602+0000 7f462bbff700 1 -- 172.21.15.98:0/8357535 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46240b26d0 msgr2=0x7f46240b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:11.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.602+0000 7f462bbff700 1 --2- 172.21.15.98:0/8357535 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46240b26d0 0x7f46240b4c00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4614009840 tx=0x7f4614009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:11.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.603+0000 7f462bbff700 1 -- 172.21.15.98:0/8357535 shutdown_connections 2023-03-03T07:57:11.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.603+0000 7f462bbff700 1 --2- 172.21.15.98:0/8357535 >> v2:172.21.15.100:3300/0 conn(0x7f46240b8d10 0x7f46240cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.603+0000 7f462bbff700 1 --2- 172.21.15.98:0/8357535 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46240b26d0 0x7f46240b4c00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.603+0000 7f462bbff700 1 -- 172.21.15.98:0/8357535 >> 172.21.15.98:0/8357535 conn(0x7f4624073e70 msgr2=0x7f462406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.603+0000 7f462bbff700 1 -- 172.21.15.98:0/8357535 shutdown_connections 2023-03-03T07:57:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.603+0000 7f462bbff700 1 -- 172.21.15.98:0/8357535 wait complete. 2023-03-03T07:57:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.605+0000 7f462bbff700 1 Processor -- start 2023-03-03T07:57:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.605+0000 7f462bbff700 1 -- start start 2023-03-03T07:57:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.606+0000 7f462bbff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46240b26d0 0x7f46241f34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.606+0000 7f462bbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46240b5140 0x7f4624154950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.606+0000 7f462bbff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46240b8d10 0x7f462415c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.606+0000 7f462bbff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4624156eb0 con 0x7f46240b8d10 2023-03-03T07:57:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.606+0000 7f462bbff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46241f31f0 con 0x7f46240b26d0 2023-03-03T07:57:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.606+0000 7f462bbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4624156510 con 0x7f46240b5140 2023-03-03T07:57:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.606+0000 7f462a19c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46240b8d10 0x7f462415c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.606+0000 7f462a19c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46240b8d10 0x7f462415c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47858/0 (socket says 172.21.15.98:47858) 2023-03-03T07:57:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.606+0000 7f462a19c700 1 -- 172.21.15.98:0/2418030876 learned_addr learned my addr 172.21.15.98:0/2418030876 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.606+0000 7f462999b700 1 --2- 172.21.15.98:0/2418030876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46240b26d0 0x7f46241f34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.606+0000 7f462919a700 1 --2- 172.21.15.98:0/2418030876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46240b5140 0x7f4624154950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.607+0000 7f462a19c700 1 -- 172.21.15.98:0/2418030876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46240b5140 msgr2=0x7f4624154950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.607+0000 7f462a19c700 1 --2- 172.21.15.98:0/2418030876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46240b5140 0x7f4624154950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.607+0000 7f462a19c700 1 -- 172.21.15.98:0/2418030876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46240b26d0 msgr2=0x7f46241f34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:11.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.607+0000 7f462a19c700 1 --2- 172.21.15.98:0/2418030876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46240b26d0 0x7f46241f34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.607+0000 7f462a19c700 1 -- 172.21.15.98:0/2418030876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46140094a0 con 0x7f46240b8d10 2023-03-03T07:57:11.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.607+0000 7f462a19c700 1 --2- 172.21.15.98:0/2418030876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46240b8d10 0x7f462415c260 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f461c00caa0 tx=0x7f461c00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:11.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.607+0000 7f461a7fc700 1 -- 172.21.15.98:0/2418030876 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c00a9e0 con 0x7f46240b8d10 2023-03-03T07:57:11.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.607+0000 7f461a7fc700 1 -- 172.21.15.98:0/2418030876 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461c00ab40 con 0x7f46240b8d10 2023-03-03T07:57:11.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.607+0000 7f461a7fc700 1 -- 172.21.15.98:0/2418030876 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c012400 con 0x7f46240b8d10 2023-03-03T07:57:11.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.607+0000 7f462bbff700 1 -- 172.21.15.98:0/2418030876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46240d0e80 con 0x7f46240b8d10 2023-03-03T07:57:11.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.608+0000 7f462bbff700 1 -- 172.21.15.98:0/2418030876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46240cdd00 con 0x7f46240b8d10 2023-03-03T07:57:11.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.609+0000 7f462bbff700 1 -- 172.21.15.98:0/2418030876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46240cdfe0 con 0x7f46240b8d10 2023-03-03T07:57:11.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.610+0000 7f461a7fc700 1 -- 172.21.15.98:0/2418030876 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f461c019030 con 0x7f46240b8d10 2023-03-03T07:57:11.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.612+0000 7f461a7fc700 1 --2- 172.21.15.98:0/2418030876 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46080682f0 0x7f460806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:11.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.612+0000 7f461a7fc700 1 -- 172.21.15.98:0/2418030876 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f461c0854e0 con 0x7f46240b8d10 2023-03-03T07:57:11.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.612+0000 7f462999b700 1 --2- 172.21.15.98:0/2418030876 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46080682f0 0x7f460806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:11.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.613+0000 7f462999b700 1 --2- 172.21.15.98:0/2418030876 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46080682f0 0x7f460806a7a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4620009be0 tx=0x7f4620009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:11.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.615+0000 7f461a7fc700 1 -- 172.21.15.98:0/2418030876 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f461c052060 con 0x7f46240b8d10 2023-03-03T07:57:11.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.886+0000 7f462bbff700 1 -- 172.21.15.98:0/2418030876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f4624156ff0 con 0x7f46240b8d10 2023-03-03T07:57:11.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.891+0000 7f461a7fc700 1 -- 172.21.15.98:0/2418030876 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f4624156ff0 con 0x7f46240b8d10 2023-03-03T07:57:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.893+0000 7f462bbff700 1 -- 172.21.15.98:0/2418030876 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46080682f0 msgr2=0x7f460806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.893+0000 7f462bbff700 1 --2- 172.21.15.98:0/2418030876 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46080682f0 0x7f460806a7a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4620009be0 tx=0x7f4620009210 comp rx=0 tx=0).stop 2023-03-03T07:57:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.893+0000 7f462bbff700 1 -- 172.21.15.98:0/2418030876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46240b8d10 msgr2=0x7f462415c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.893+0000 7f462bbff700 1 --2- 172.21.15.98:0/2418030876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46240b8d10 0x7f462415c260 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f461c00caa0 tx=0x7f461c00cdb0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.893+0000 7f462bbff700 1 -- 172.21.15.98:0/2418030876 shutdown_connections 2023-03-03T07:57:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.893+0000 7f462bbff700 1 --2- 172.21.15.98:0/2418030876 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46080682f0 0x7f460806a7a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.893+0000 7f462bbff700 1 --2- 172.21.15.98:0/2418030876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46240b8d10 0x7f462415c260 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.893+0000 7f462bbff700 1 --2- 172.21.15.98:0/2418030876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46240b26d0 0x7f46241f34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.893+0000 7f462bbff700 1 --2- 172.21.15.98:0/2418030876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46240b5140 0x7f4624154950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.893+0000 7f462bbff700 1 -- 172.21.15.98:0/2418030876 >> 172.21.15.98:0/2418030876 conn(0x7f4624073e70 msgr2=0x7f46240b80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.893+0000 7f462bbff700 1 -- 172.21.15.98:0/2418030876 shutdown_connections 2023-03-03T07:57:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:11.894+0000 7f462bbff700 1 -- 172.21.15.98:0/2418030876 wait complete. 2023-03-03T07:57:11.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-03-03T07:57:12.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.174+0000 7f36babff700 1 Processor -- start 2023-03-03T07:57:12.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.174+0000 7f36babff700 1 -- start start 2023-03-03T07:57:12.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.175+0000 7f36babff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36b40d0270 0x7f36b40cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:12.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.176+0000 7f36babff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b40cf980 con 0x7f36b406b4b0 2023-03-03T07:57:12.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.176+0000 7f36babff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b40d1800 con 0x7f36b415ca70 2023-03-03T07:57:12.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.176+0000 7f36babff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b40cd5b0 con 0x7f36b40d0270 2023-03-03T07:57:12.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.176+0000 7f36b919c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f36b406b4b0 0x7f36b40d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55132/0 (socket says 172.21.15.98:55132) 2023-03-03T07:57:12.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.176+0000 7f36abfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36b40d0270 0x7f36b40cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:12.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.176+0000 7f36b919c700 1 -- 172.21.15.98:0/2801088389 learned_addr learned my addr 172.21.15.98:0/2801088389 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:12.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.176+0000 7f36abfff700 1 -- 172.21.15.98:0/2801088389 >> v1:172.21.15.100:6789/0 conn(0x7f36b415ca70 legacy=0x7f36b40cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:12.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.176+0000 7f36abfff700 1 -- 172.21.15.98:0/2801088389 >> v1:172.21.15.98:6789/0 conn(0x7f36b406b4b0 legacy=0x7f36b40d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:12.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.176+0000 7f36abfff700 1 -- 172.21.15.98:0/2801088389 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b40cd900 con 0x7f36b40d0270 2023-03-03T07:57:12.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.176+0000 7f36ab7fe700 1 -- 172.21.15.98:0/2801088389 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819337126 0 0) 0x7f36b40cf980 con 0x7f36b406b4b0 2023-03-03T07:57:12.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.177+0000 7f36abfff700 1 --2- 172.21.15.98:0/2801088389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b40d0270 0x7f36b40cf440 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f36a4009840 tx=0x7f36a4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a7b7358f3b6919a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:12.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.177+0000 7f36ab7fe700 1 -- 172.21.15.98:0/2801088389 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a4011830 con 0x7f36b40d0270 2023-03-03T07:57:12.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.177+0000 7f36ab7fe700 1 -- 172.21.15.98:0/2801088389 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a4011990 con 0x7f36b40d0270 2023-03-03T07:57:12.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.177+0000 7f36ab7fe700 1 -- 172.21.15.98:0/2801088389 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a4011d20 con 0x7f36b40d0270 2023-03-03T07:57:12.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.178+0000 7f36babff700 1 -- 172.21.15.98:0/2801088389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b40d0270 msgr2=0x7f36b40cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:12.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.178+0000 7f36babff700 1 --2- 172.21.15.98:0/2801088389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b40d0270 0x7f36b40cf440 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f36a4009840 tx=0x7f36a4009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:12.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.178+0000 7f36babff700 1 -- 172.21.15.98:0/2801088389 shutdown_connections 2023-03-03T07:57:12.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.178+0000 7f36babff700 1 --2- 172.21.15.98:0/2801088389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b40d0270 0x7f36b40cf440 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.178+0000 7f36babff700 1 -- 172.21.15.98:0/2801088389 >> 172.21.15.98:0/2801088389 conn(0x7f36b4073e70 msgr2=0x7f36b406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.178+0000 7f36babff700 1 -- 172.21.15.98:0/2801088389 shutdown_connections 2023-03-03T07:57:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.178+0000 7f36babff700 1 -- 172.21.15.98:0/2801088389 wait complete. 2023-03-03T07:57:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.181+0000 7f36babff700 1 Processor -- start 2023-03-03T07:57:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.181+0000 7f36babff700 1 -- start start 2023-03-03T07:57:12.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.182+0000 7f36babff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b406b4b0 0x7f36b414b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:12.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.182+0000 7f36babff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b40d0270 0x7f36b40b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:12.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.182+0000 7f36babff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b415ca70 0x7f36b40d1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:12.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.182+0000 7f36babff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b414bb50 con 0x7f36b406b4b0 2023-03-03T07:57:12.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.182+0000 7f36babff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b415d430 con 0x7f36b415ca70 2023-03-03T07:57:12.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.182+0000 7f36babff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b415d710 con 0x7f36b40d0270 2023-03-03T07:57:12.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.183+0000 7f36b899b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b406b4b0 0x7f36b414b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:12.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.183+0000 7f36b899b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b406b4b0 0x7f36b414b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47886/0 (socket says 172.21.15.98:47886) 2023-03-03T07:57:12.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.183+0000 7f36b899b700 1 -- 172.21.15.98:0/811478905 learned_addr learned my addr 172.21.15.98:0/811478905 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:12.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.183+0000 7f36abfff700 1 --2- 172.21.15.98:0/811478905 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b40d0270 0x7f36b40b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:12.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.183+0000 7f36b919c700 1 --2- 172.21.15.98:0/811478905 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b415ca70 0x7f36b40d1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:12.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.183+0000 7f36b899b700 1 -- 172.21.15.98:0/811478905 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b40d0270 msgr2=0x7f36b40b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:12.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.183+0000 7f36b899b700 1 --2- 172.21.15.98:0/811478905 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b40d0270 0x7f36b40b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:12.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.183+0000 7f36b899b700 1 -- 172.21.15.98:0/811478905 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b415ca70 msgr2=0x7f36b40d1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:12.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.183+0000 7f36b899b700 1 --2- 172.21.15.98:0/811478905 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b415ca70 0x7f36b40d1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:12.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.183+0000 7f36b899b700 1 -- 172.21.15.98:0/811478905 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a40094a0 con 0x7f36b406b4b0 2023-03-03T07:57:12.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.184+0000 7f36b899b700 1 --2- 172.21.15.98:0/811478905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b406b4b0 0x7f36b414b610 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f36b000c450 tx=0x7f36b000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:12.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.184+0000 7f36a97fa700 1 -- 172.21.15.98:0/811478905 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b00052d0 con 0x7f36b406b4b0 2023-03-03T07:57:12.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.184+0000 7f36a97fa700 1 -- 172.21.15.98:0/811478905 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b0005430 con 0x7f36b406b4b0 2023-03-03T07:57:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.184+0000 7f36babff700 1 -- 172.21.15.98:0/811478905 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b40d1aa0 con 0x7f36b406b4b0 2023-03-03T07:57:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.184+0000 7f36a97fa700 1 -- 172.21.15.98:0/811478905 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b00038c0 con 0x7f36b406b4b0 2023-03-03T07:57:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.184+0000 7f36babff700 1 -- 172.21.15.98:0/811478905 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b40cdbe0 con 0x7f36b406b4b0 2023-03-03T07:57:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.185+0000 7f36babff700 1 -- 172.21.15.98:0/811478905 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b415ce40 con 0x7f36b406b4b0 2023-03-03T07:57:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.187+0000 7f36a97fa700 1 -- 172.21.15.98:0/811478905 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f36b0019030 con 0x7f36b406b4b0 2023-03-03T07:57:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.188+0000 7f36a97fa700 1 --2- 172.21.15.98:0/811478905 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36a00682f0 0x7f36a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.188+0000 7f36a97fa700 1 -- 172.21.15.98:0/811478905 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f36b0084c20 con 0x7f36b406b4b0 2023-03-03T07:57:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.188+0000 7f36abfff700 1 --2- 172.21.15.98:0/811478905 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36a00682f0 0x7f36a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.189+0000 7f36abfff700 1 --2- 172.21.15.98:0/811478905 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36a00682f0 0x7f36a006a7a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f36a4009840 tx=0x7f36a401c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:12.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.192+0000 7f36a97fa700 1 -- 172.21.15.98:0/811478905 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f36b00516e0 con 0x7f36b406b4b0 2023-03-03T07:57:12.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.460+0000 7f36babff700 1 -- 172.21.15.98:0/811478905 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f36b40d3210 con 0x7f36b406b4b0 2023-03-03T07:57:12.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.464+0000 7f36a97fa700 1 -- 172.21.15.98:0/811478905 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f36b0054d00 con 0x7f36b406b4b0 2023-03-03T07:57:12.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.466+0000 7f36babff700 1 -- 172.21.15.98:0/811478905 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36a00682f0 msgr2=0x7f36a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:12.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.466+0000 7f36babff700 1 --2- 172.21.15.98:0/811478905 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36a00682f0 0x7f36a006a7a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f36a4009840 tx=0x7f36a401c040 comp rx=0 tx=0).stop 2023-03-03T07:57:12.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.466+0000 7f36babff700 1 -- 172.21.15.98:0/811478905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b406b4b0 msgr2=0x7f36b414b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:12.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.466+0000 7f36babff700 1 --2- 172.21.15.98:0/811478905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b406b4b0 0x7f36b414b610 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f36b000c450 tx=0x7f36b000c810 comp rx=0 tx=0).stop 2023-03-03T07:57:12.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.467+0000 7f36babff700 1 -- 172.21.15.98:0/811478905 shutdown_connections 2023-03-03T07:57:12.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.467+0000 7f36babff700 1 --2- 172.21.15.98:0/811478905 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36a00682f0 0x7f36a006a7a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:12.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.467+0000 7f36babff700 1 --2- 172.21.15.98:0/811478905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b406b4b0 0x7f36b414b610 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:12.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.467+0000 7f36babff700 1 --2- 172.21.15.98:0/811478905 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b415ca70 0x7f36b40d1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:12.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.467+0000 7f36babff700 1 --2- 172.21.15.98:0/811478905 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b40d0270 0x7f36b40b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:12.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.467+0000 7f36babff700 1 -- 172.21.15.98:0/811478905 >> 172.21.15.98:0/811478905 conn(0x7f36b4073e70 msgr2=0x7f36b40b5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:12.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.467+0000 7f36babff700 1 -- 172.21.15.98:0/811478905 shutdown_connections 2023-03-03T07:57:12.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.467+0000 7f36babff700 1 -- 172.21.15.98:0/811478905 wait complete. 2023-03-03T07:57:12.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-03-03T07:57:12.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.744+0000 7f63bc7ff700 1 Processor -- start 2023-03-03T07:57:12.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.744+0000 7f63bc7ff700 1 -- start start 2023-03-03T07:57:12.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.746+0000 7f63bc7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63b40d0270 0x7f63b40cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:12.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.746+0000 7f63bc7ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f63b406b4b0 0x7f63b40d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:12.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.746+0000 7f63bc7ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b40cf980 con 0x7f63b415ca70 2023-03-03T07:57:12.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.746+0000 7f63bc7ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b40cd5b0 con 0x7f63b406b4b0 2023-03-03T07:57:12.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.746+0000 7f63bc7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b40d1800 con 0x7f63b40d0270 2023-03-03T07:57:12.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.746+0000 7f63ba59b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f63b415ca70 0x7f63b40cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55148/0 (socket says 172.21.15.98:55148) 2023-03-03T07:57:12.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.746+0000 7f63ba59b700 1 -- 172.21.15.98:0/3756942653 learned_addr learned my addr 172.21.15.98:0/3756942653 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:12.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.746+0000 7f63b9d9a700 1 --2- 172.21.15.98:0/3756942653 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63b40d0270 0x7f63b40cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:12.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.746+0000 7f63bad9c700 1 --2- 172.21.15.98:0/3756942653 >> v2:172.21.15.100:3300/0 conn(0x7f63b406b4b0 0x7f63b40d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:12.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.746+0000 7f63b9599700 1 -- 172.21.15.98:0/3756942653 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524237497 0 0) 0x7f63b40cf980 con 0x7f63b415ca70 2023-03-03T07:57:12.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.747+0000 7f63b9599700 1 -- 172.21.15.98:0/3756942653 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a00035b0 con 0x7f63b415ca70 2023-03-03T07:57:12.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.747+0000 7f63b9d9a700 1 -- 172.21.15.98:0/3756942653 >> v2:172.21.15.100:3300/0 conn(0x7f63b406b4b0 msgr2=0x7f63b40d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:12.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.747+0000 7f63b9d9a700 1 --2- 172.21.15.98:0/3756942653 >> v2:172.21.15.100:3300/0 conn(0x7f63b406b4b0 0x7f63b40d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:12.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.747+0000 7f63b9d9a700 1 -- 172.21.15.98:0/3756942653 >> v1:172.21.15.98:6789/0 conn(0x7f63b415ca70 legacy=0x7f63b40cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:12.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.747+0000 7f63b9d9a700 1 -- 172.21.15.98:0/3756942653 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b40cd900 con 0x7f63b40d0270 2023-03-03T07:57:12.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.747+0000 7f63b9d9a700 1 --2- 172.21.15.98:0/3756942653 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63b40d0270 0x7f63b40cf440 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f63a4009840 tx=0x7f63a4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b55b25921095596b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:12.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.747+0000 7f63b9599700 1 -- 172.21.15.98:0/3756942653 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a4011830 con 0x7f63b40d0270 2023-03-03T07:57:12.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.748+0000 7f63b9599700 1 -- 172.21.15.98:0/3756942653 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a4011990 con 0x7f63b40d0270 2023-03-03T07:57:12.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.748+0000 7f63b9599700 1 -- 172.21.15.98:0/3756942653 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a4011d20 con 0x7f63b40d0270 2023-03-03T07:57:12.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.748+0000 7f63bc7ff700 1 -- 172.21.15.98:0/3756942653 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63b40d0270 msgr2=0x7f63b40cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:12.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.748+0000 7f63bc7ff700 1 --2- 172.21.15.98:0/3756942653 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63b40d0270 0x7f63b40cf440 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f63a4009840 tx=0x7f63a4009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:12.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.748+0000 7f63bc7ff700 1 -- 172.21.15.98:0/3756942653 shutdown_connections 2023-03-03T07:57:12.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.748+0000 7f63bc7ff700 1 --2- 172.21.15.98:0/3756942653 >> v2:172.21.15.100:3300/0 conn(0x7f63b406b4b0 0x7f63b40d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:12.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.748+0000 7f63bc7ff700 1 --2- 172.21.15.98:0/3756942653 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63b40d0270 0x7f63b40cf440 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:12.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.748+0000 7f63bc7ff700 1 -- 172.21.15.98:0/3756942653 >> 172.21.15.98:0/3756942653 conn(0x7f63b4073e70 msgr2=0x7f63b406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:12.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.748+0000 7f63bc7ff700 1 -- 172.21.15.98:0/3756942653 shutdown_connections 2023-03-03T07:57:12.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.748+0000 7f63bc7ff700 1 -- 172.21.15.98:0/3756942653 wait complete. 2023-03-03T07:57:12.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.750+0000 7f63bc7ff700 1 Processor -- start 2023-03-03T07:57:12.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.750+0000 7f63bc7ff700 1 -- start start 2023-03-03T07:57:12.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.751+0000 7f63bc7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63b406b4b0 0x7f63b40cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.751+0000 7f63bc7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63b40d0270 0x7f63b40b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.751+0000 7f63bc7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b415ca70 0x7f63b413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.751+0000 7f63bc7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b40c7580 con 0x7f63b415ca70 2023-03-03T07:57:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.751+0000 7f63bc7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b40b4640 con 0x7f63b40d0270 2023-03-03T07:57:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.751+0000 7f63bc7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b40b5590 con 0x7f63b406b4b0 2023-03-03T07:57:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.752+0000 7f63bad9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b415ca70 0x7f63b413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.752+0000 7f63bad9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b415ca70 0x7f63b413f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47900/0 (socket says 172.21.15.98:47900) 2023-03-03T07:57:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.752+0000 7f63bad9c700 1 -- 172.21.15.98:0/450656759 learned_addr learned my addr 172.21.15.98:0/450656759 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.752+0000 7f63ba59b700 1 --2- 172.21.15.98:0/450656759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63b406b4b0 0x7f63b40cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.752+0000 7f63b9d9a700 1 --2- 172.21.15.98:0/450656759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63b40d0270 0x7f63b40b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.752+0000 7f63bad9c700 1 -- 172.21.15.98:0/450656759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63b406b4b0 msgr2=0x7f63b40cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.752+0000 7f63bad9c700 1 --2- 172.21.15.98:0/450656759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63b406b4b0 0x7f63b40cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.752+0000 7f63bad9c700 1 -- 172.21.15.98:0/450656759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63b40d0270 msgr2=0x7f63b40b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.752+0000 7f63bad9c700 1 --2- 172.21.15.98:0/450656759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63b40d0270 0x7f63b40b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:12.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.752+0000 7f63bad9c700 1 -- 172.21.15.98:0/450656759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a40094a0 con 0x7f63b415ca70 2023-03-03T07:57:12.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.752+0000 7f63b9d9a700 1 --2- 172.21.15.98:0/450656759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63b40d0270 0x7f63b40b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.752+0000 7f63bad9c700 1 --2- 172.21.15.98:0/450656759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b415ca70 0x7f63b413f2d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f63ac00efc0 tx=0x7f63ac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.753+0000 7f63aaffd700 1 -- 172.21.15.98:0/450656759 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac009ba0 con 0x7f63b415ca70 2023-03-03T07:57:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.753+0000 7f63aaffd700 1 -- 172.21.15.98:0/450656759 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ac00f040 con 0x7f63b415ca70 2023-03-03T07:57:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.753+0000 7f63aaffd700 1 -- 172.21.15.98:0/450656759 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac014480 con 0x7f63b415ca70 2023-03-03T07:57:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.753+0000 7f63bc7ff700 1 -- 172.21.15.98:0/450656759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b40d1aa0 con 0x7f63b415ca70 2023-03-03T07:57:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.753+0000 7f63bc7ff700 1 -- 172.21.15.98:0/450656759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b40cdbe0 con 0x7f63b415ca70 2023-03-03T07:57:12.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.754+0000 7f63bc7ff700 1 -- 172.21.15.98:0/450656759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b415ce40 con 0x7f63b415ca70 2023-03-03T07:57:12.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.755+0000 7f63aaffd700 1 -- 172.21.15.98:0/450656759 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f63ac0145e0 con 0x7f63b415ca70 2023-03-03T07:57:12.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.756+0000 7f63aaffd700 1 --2- 172.21.15.98:0/450656759 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63980682f0 0x7f639806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.757+0000 7f63aaffd700 1 -- 172.21.15.98:0/450656759 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f63ac086430 con 0x7f63b415ca70 2023-03-03T07:57:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.757+0000 7f63ba59b700 1 --2- 172.21.15.98:0/450656759 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63980682f0 0x7f639806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.758+0000 7f63ba59b700 1 --2- 172.21.15.98:0/450656759 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63980682f0 0x7f639806a7a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f63b00011e0 tx=0x7f63b0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:12.761+0000 7f63aaffd700 1 -- 172.21.15.98:0/450656759 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f63ac052ef0 con 0x7f63b415ca70 2023-03-03T07:57:13.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.029+0000 7f63bc7ff700 1 -- 172.21.15.98:0/450656759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f63b40b2a50 con 0x7f63b415ca70 2023-03-03T07:57:13.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.034+0000 7f63aaffd700 1 -- 172.21.15.98:0/450656759 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f63ac056510 con 0x7f63b415ca70 2023-03-03T07:57:13.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.036+0000 7f63bc7ff700 1 -- 172.21.15.98:0/450656759 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63980682f0 msgr2=0x7f639806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:13.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.036+0000 7f63bc7ff700 1 --2- 172.21.15.98:0/450656759 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63980682f0 0x7f639806a7a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f63b00011e0 tx=0x7f63b0009040 comp rx=0 tx=0).stop 2023-03-03T07:57:13.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.036+0000 7f63bc7ff700 1 -- 172.21.15.98:0/450656759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b415ca70 msgr2=0x7f63b413f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:13.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.036+0000 7f63bc7ff700 1 --2- 172.21.15.98:0/450656759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b415ca70 0x7f63b413f2d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f63ac00efc0 tx=0x7f63ac00c3a0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.036+0000 7f63bc7ff700 1 -- 172.21.15.98:0/450656759 shutdown_connections 2023-03-03T07:57:13.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.036+0000 7f63bc7ff700 1 --2- 172.21.15.98:0/450656759 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63980682f0 0x7f639806a7a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.036+0000 7f63bc7ff700 1 --2- 172.21.15.98:0/450656759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b415ca70 0x7f63b413f2d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.036+0000 7f63bc7ff700 1 --2- 172.21.15.98:0/450656759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63b406b4b0 0x7f63b40cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.036+0000 7f63bc7ff700 1 --2- 172.21.15.98:0/450656759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63b40d0270 0x7f63b40b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.036+0000 7f63bc7ff700 1 -- 172.21.15.98:0/450656759 >> 172.21.15.98:0/450656759 conn(0x7f63b4073e70 msgr2=0x7f63b40b61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:13.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.036+0000 7f63bc7ff700 1 -- 172.21.15.98:0/450656759 shutdown_connections 2023-03-03T07:57:13.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.037+0000 7f63bc7ff700 1 -- 172.21.15.98:0/450656759 wait complete. 2023-03-03T07:57:13.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:57:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.309+0000 7fb3ca99d700 1 Processor -- start 2023-03-03T07:57:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.310+0000 7fb3ca99d700 1 -- start start 2023-03-03T07:57:13.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.311+0000 7fb3ca99d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3c40b51c0 0x7fb3c4160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.311+0000 7fb3ca99d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3c41614c0 con 0x7fb3c40d1580 2023-03-03T07:57:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.311+0000 7fb3ca99d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3c40d10f0 con 0x7fb3c4146360 2023-03-03T07:57:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.311+0000 7fb3ca99d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c40cdf20 con 0x7fb3c40b51c0 2023-03-03T07:57:13.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.311+0000 7fb3c919a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3c40b51c0 0x7fb3c4160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.311+0000 7fb3ca19c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb3c40d1580 0x7fb3c40d1980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55154/0 (socket says 172.21.15.98:55154) 2023-03-03T07:57:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.311+0000 7fb3ca19c700 1 -- 172.21.15.98:0/936993438 learned_addr learned my addr 172.21.15.98:0/936993438 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.312+0000 7fb3c8999700 1 -- 172.21.15.98:0/936993438 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974386621 0 0) 0x7fb3bc001490 con 0x7fb3c40d1580 2023-03-03T07:57:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.312+0000 7fb3c8999700 1 -- 172.21.15.98:0/936993438 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3b00035b0 con 0x7fb3c40d1580 2023-03-03T07:57:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.312+0000 7fb3c919a700 1 -- 172.21.15.98:0/936993438 >> v1:172.21.15.100:6789/0 conn(0x7fb3c4146360 legacy=0x7fb3c40b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.312+0000 7fb3c8999700 1 -- 172.21.15.98:0/936993438 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674179470 0 0) 0x7fb3c40d10f0 con 0x7fb3c4146360 2023-03-03T07:57:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.312+0000 7fb3c919a700 1 -- 172.21.15.98:0/936993438 >> v1:172.21.15.98:6789/0 conn(0x7fb3c40d1580 legacy=0x7fb3c40d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.312+0000 7fb3c919a700 1 -- 172.21.15.98:0/936993438 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c40cd680 con 0x7fb3c40b51c0 2023-03-03T07:57:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.313+0000 7fb3c919a700 1 --2- 172.21.15.98:0/936993438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40b51c0 0x7fb3c4160f80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4009840 tx=0x7fb3b4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eead98b15354bec5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.313+0000 7fb3c8999700 1 -- 172.21.15.98:0/936993438 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b4011830 con 0x7fb3c40b51c0 2023-03-03T07:57:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.313+0000 7fb3c8999700 1 -- 172.21.15.98:0/936993438 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b4011990 con 0x7fb3c40b51c0 2023-03-03T07:57:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.313+0000 7fb3c8999700 1 -- 172.21.15.98:0/936993438 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b4011d20 con 0x7fb3c40b51c0 2023-03-03T07:57:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.313+0000 7fb3ca99d700 1 -- 172.21.15.98:0/936993438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40b51c0 msgr2=0x7fb3c4160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.313+0000 7fb3ca99d700 1 --2- 172.21.15.98:0/936993438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40b51c0 0x7fb3c4160f80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4009840 tx=0x7fb3b4009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.313+0000 7fb3ca99d700 1 -- 172.21.15.98:0/936993438 shutdown_connections 2023-03-03T07:57:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.314+0000 7fb3ca99d700 1 --2- 172.21.15.98:0/936993438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40b51c0 0x7fb3c4160f80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.314+0000 7fb3ca99d700 1 -- 172.21.15.98:0/936993438 >> 172.21.15.98:0/936993438 conn(0x7fb3c406b8e0 msgr2=0x7fb3c40cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:13.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.314+0000 7fb3ca99d700 1 -- 172.21.15.98:0/936993438 shutdown_connections 2023-03-03T07:57:13.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.314+0000 7fb3ca99d700 1 -- 172.21.15.98:0/936993438 wait complete. 2023-03-03T07:57:13.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.316+0000 7fb3ca99d700 1 Processor -- start 2023-03-03T07:57:13.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.317+0000 7fb3ca99d700 1 -- start start 2023-03-03T07:57:13.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.318+0000 7fb3ca99d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb3c40b51c0 0x7fb3c41f5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:13.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.318+0000 7fb3ca99d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40d1580 0x7fb3c414e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:13.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.318+0000 7fb3ca99d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb3c4146360 0x7fb3c415dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:13.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.318+0000 7fb3ca99d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c414dbf0 con 0x7fb3c40d1580 2023-03-03T07:57:13.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.318+0000 7fb3ca99d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c414cfe0 con 0x7fb3c4146360 2023-03-03T07:57:13.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.318+0000 7fb3ca99d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c414d2c0 con 0x7fb3c40b51c0 2023-03-03T07:57:13.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.318+0000 7fb3c919a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40d1580 0x7fb3c414e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:13.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.319+0000 7fb3c919a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40d1580 0x7fb3c414e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47918/0 (socket says 172.21.15.98:47918) 2023-03-03T07:57:13.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.319+0000 7fb3c919a700 1 -- 172.21.15.98:0/3944058335 learned_addr learned my addr 172.21.15.98:0/3944058335 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:13.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.319+0000 7fb3c999b700 1 --2- 172.21.15.98:0/3944058335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb3c40b51c0 0x7fb3c41f5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.319+0000 7fb3ca19c700 1 --2- 172.21.15.98:0/3944058335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb3c4146360 0x7fb3c415dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.319+0000 7fb3c919a700 1 -- 172.21.15.98:0/3944058335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb3c40b51c0 msgr2=0x7fb3c41f5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.319+0000 7fb3c919a700 1 --2- 172.21.15.98:0/3944058335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb3c40b51c0 0x7fb3c41f5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.319+0000 7fb3c919a700 1 -- 172.21.15.98:0/3944058335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb3c4146360 msgr2=0x7fb3c415dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.319+0000 7fb3c919a700 1 --2- 172.21.15.98:0/3944058335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb3c4146360 0x7fb3c415dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.319+0000 7fb3c919a700 1 -- 172.21.15.98:0/3944058335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b40094a0 con 0x7fb3c40d1580 2023-03-03T07:57:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.319+0000 7fb3ca19c700 1 --2- 172.21.15.98:0/3944058335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb3c4146360 0x7fb3c415dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:57:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.319+0000 7fb3c919a700 1 --2- 172.21.15.98:0/3944058335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40d1580 0x7fb3c414e080 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb3b40090b0 tx=0x7fb3b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.320+0000 7fb3ba7fc700 1 -- 172.21.15.98:0/3944058335 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b401a910 con 0x7fb3c40d1580 2023-03-03T07:57:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.320+0000 7fb3ba7fc700 1 -- 172.21.15.98:0/3944058335 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b401aa70 con 0x7fb3c40d1580 2023-03-03T07:57:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.320+0000 7fb3ba7fc700 1 -- 172.21.15.98:0/3944058335 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b401ade0 con 0x7fb3c40d1580 2023-03-03T07:57:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.320+0000 7fb3ca99d700 1 -- 172.21.15.98:0/3944058335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c40cdc40 con 0x7fb3c40d1580 2023-03-03T07:57:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.320+0000 7fb3ca99d700 1 -- 172.21.15.98:0/3944058335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c40cd960 con 0x7fb3c40d1580 2023-03-03T07:57:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.321+0000 7fb3ca99d700 1 -- 172.21.15.98:0/3944058335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c40d0d80 con 0x7fb3c40d1580 2023-03-03T07:57:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.322+0000 7fb3ba7fc700 1 -- 172.21.15.98:0/3944058335 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb3b400f050 con 0x7fb3c40d1580 2023-03-03T07:57:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.323+0000 7fb3ba7fc700 1 --2- 172.21.15.98:0/3944058335 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb3a80682f0 0x7fb3a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.323+0000 7fb3ba7fc700 1 -- 172.21.15.98:0/3944058335 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb3b40868c0 con 0x7fb3c40d1580 2023-03-03T07:57:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.324+0000 7fb3c999b700 1 --2- 172.21.15.98:0/3944058335 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb3a80682f0 0x7fb3a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.324+0000 7fb3c999b700 1 --2- 172.21.15.98:0/3944058335 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb3a80682f0 0x7fb3a806a7a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0000f60 tx=0x7fb3c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:13.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.327+0000 7fb3ba7fc700 1 -- 172.21.15.98:0/3944058335 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb3b4053430 con 0x7fb3c40d1580 2023-03-03T07:57:13.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.520+0000 7fb3ca99d700 1 -- 172.21.15.98:0/3944058335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3c4145ff0 con 0x7fb3c40d1580 2023-03-03T07:57:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.521+0000 7fb3ba7fc700 1 -- 172.21.15.98:0/3944058335 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+14267 (secure 0 0 0) 0x7fb3b4029090 con 0x7fb3c40d1580 2023-03-03T07:57:13.523 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:13.523 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":34,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:57:09.038047+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:57:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.523+0000 7fb3ca99d700 1 -- 172.21.15.98:0/3944058335 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb3a80682f0 msgr2=0x7fb3a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.523+0000 7fb3ca99d700 1 --2- 172.21.15.98:0/3944058335 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb3a80682f0 0x7fb3a806a7a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0000f60 tx=0x7fb3c0009040 comp rx=0 tx=0).stop 2023-03-03T07:57:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.523+0000 7fb3ca99d700 1 -- 172.21.15.98:0/3944058335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40d1580 msgr2=0x7fb3c414e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.523+0000 7fb3ca99d700 1 --2- 172.21.15.98:0/3944058335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40d1580 0x7fb3c414e080 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb3b40090b0 tx=0x7fb3b4004750 comp rx=0 tx=0).stop 2023-03-03T07:57:13.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.524+0000 7fb3ca99d700 1 -- 172.21.15.98:0/3944058335 shutdown_connections 2023-03-03T07:57:13.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.524+0000 7fb3ca99d700 1 --2- 172.21.15.98:0/3944058335 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb3a80682f0 0x7fb3a806a7a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.524+0000 7fb3ca99d700 1 --2- 172.21.15.98:0/3944058335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb3c4146360 0x7fb3c415dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.524+0000 7fb3ca99d700 1 --2- 172.21.15.98:0/3944058335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb3c40b51c0 0x7fb3c41f5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.524+0000 7fb3ca99d700 1 --2- 172.21.15.98:0/3944058335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40d1580 0x7fb3c414e080 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.524+0000 7fb3ca99d700 1 -- 172.21.15.98:0/3944058335 >> 172.21.15.98:0/3944058335 conn(0x7fb3c406b8e0 msgr2=0x7fb3c40b44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:13.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.524+0000 7fb3ca99d700 1 -- 172.21.15.98:0/3944058335 shutdown_connections 2023-03-03T07:57:13.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.524+0000 7fb3ca99d700 1 -- 172.21.15.98:0/3944058335 wait complete. 2023-03-03T07:57:13.552 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:57:13.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:57:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.805+0000 7f7bfa19d700 1 Processor -- start 2023-03-03T07:57:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.805+0000 7f7bfa19d700 1 -- start start 2023-03-03T07:57:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.806+0000 7f7bfa19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bf41609b0 0x7f7bf4160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.806+0000 7f7bfa19d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7bf40cf570 0x7f7bf415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.806+0000 7f7bfa19d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf41612c0 con 0x7f7bf415db30 2023-03-03T07:57:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.806+0000 7f7bfa19d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf40cebb0 con 0x7f7bf40cf570 2023-03-03T07:57:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.806+0000 7f7bfa19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40cd780 con 0x7f7bf41609b0 2023-03-03T07:57:13.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.807+0000 7f7bf899a700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7bf40cf570 0x7f7bf415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.807+0000 7f7bf919b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bf41609b0 0x7f7bf4160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.807+0000 7f7bf919b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bf41609b0 0x7f7bf4160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:44018/0 (socket says 172.21.15.98:44018) 2023-03-03T07:57:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.807+0000 7f7bf919b700 1 -- 172.21.15.98:0/1331352157 learned_addr learned my addr 172.21.15.98:0/1331352157 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.807+0000 7f7bf899a700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7bf40cf570 0x7f7bf415d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:40920/0 (socket says 172.21.15.98:40920) 2023-03-03T07:57:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.808+0000 7f7bf899a700 1 -- 172.21.15.98:0/1331352157 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bf41609b0 msgr2=0x7f7bf4160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.808+0000 7f7bf899a700 1 --2- 172.21.15.98:0/1331352157 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bf41609b0 0x7f7bf4160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.808+0000 7f7bf899a700 1 -- 172.21.15.98:0/1331352157 >> v1:172.21.15.100:6789/0 conn(0x7f7bf415db30 legacy=0x7f7bf40b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.808+0000 7f7bf899a700 1 -- 172.21.15.98:0/1331352157 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf40cda60 con 0x7f7bf40cf570 2023-03-03T07:57:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.808+0000 7f7bebfff700 1 -- 172.21.15.98:0/1331352157 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781411919 0 0) 0x7f7bf41612c0 con 0x7f7bf415db30 2023-03-03T07:57:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.808+0000 7f7bf899a700 1 --2- 172.21.15.98:0/1331352157 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7bf40cf570 0x7f7bf415d5f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009840 tx=0x7f7be4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a040b870711951f1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.808+0000 7f7bebfff700 1 -- 172.21.15.98:0/1331352157 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4011830 con 0x7f7bf40cf570 2023-03-03T07:57:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.808+0000 7f7bebfff700 1 -- 172.21.15.98:0/1331352157 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be4011990 con 0x7f7bf40cf570 2023-03-03T07:57:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.808+0000 7f7bebfff700 1 -- 172.21.15.98:0/1331352157 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4011d20 con 0x7f7bf40cf570 2023-03-03T07:57:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.809+0000 7f7bfa19d700 1 -- 172.21.15.98:0/1331352157 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7bf40cf570 msgr2=0x7f7bf415d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.809+0000 7f7bfa19d700 1 --2- 172.21.15.98:0/1331352157 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7bf40cf570 0x7f7bf415d5f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009840 tx=0x7f7be4009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.809+0000 7f7bfa19d700 1 -- 172.21.15.98:0/1331352157 shutdown_connections 2023-03-03T07:57:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.809+0000 7f7bfa19d700 1 --2- 172.21.15.98:0/1331352157 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bf41609b0 0x7f7bf4160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.809+0000 7f7bfa19d700 1 --2- 172.21.15.98:0/1331352157 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7bf40cf570 0x7f7bf415d5f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.809+0000 7f7bfa19d700 1 -- 172.21.15.98:0/1331352157 >> 172.21.15.98:0/1331352157 conn(0x7f7bf406b880 msgr2=0x7f7bf40cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.809+0000 7f7bfa19d700 1 -- 172.21.15.98:0/1331352157 shutdown_connections 2023-03-03T07:57:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.809+0000 7f7bfa19d700 1 -- 172.21.15.98:0/1331352157 wait complete. 2023-03-03T07:57:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.811+0000 7f7bfa19d700 1 Processor -- start 2023-03-03T07:57:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.811+0000 7f7bfa19d700 1 -- start start 2023-03-03T07:57:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.812+0000 7f7bfa19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bf40cf570 0x7f7bf41f19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.812+0000 7f7bfa19d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf415db30 0x7f7bf40c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.812+0000 7f7bfa19d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7bf41609b0 0x7f7bf40c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.812+0000 7f7bfa19d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40b2f10 con 0x7f7bf415db30 2023-03-03T07:57:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.812+0000 7f7bfa19d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40b31f0 con 0x7f7bf41609b0 2023-03-03T07:57:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.812+0000 7f7bfa19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40b34d0 con 0x7f7bf40cf570 2023-03-03T07:57:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.813+0000 7f7bf899a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf415db30 0x7f7bf40c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.813+0000 7f7bf899a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf415db30 0x7f7bf40c75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47922/0 (socket says 172.21.15.98:47922) 2023-03-03T07:57:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.813+0000 7f7bf899a700 1 -- 172.21.15.98:0/549326771 learned_addr learned my addr 172.21.15.98:0/549326771 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.813+0000 7f7bf999c700 1 --2- 172.21.15.98:0/549326771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7bf41609b0 0x7f7bf40c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.813+0000 7f7bf919b700 1 --2- 172.21.15.98:0/549326771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bf40cf570 0x7f7bf41f19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.813+0000 7f7bf899a700 1 -- 172.21.15.98:0/549326771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bf40cf570 msgr2=0x7f7bf41f19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.813+0000 7f7bf899a700 1 --2- 172.21.15.98:0/549326771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bf40cf570 0x7f7bf41f19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.813+0000 7f7bf899a700 1 -- 172.21.15.98:0/549326771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7bf41609b0 msgr2=0x7f7bf40c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.813+0000 7f7bf899a700 1 --2- 172.21.15.98:0/549326771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7bf41609b0 0x7f7bf40c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.813+0000 7f7bf899a700 1 -- 172.21.15.98:0/549326771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be40094a0 con 0x7f7bf415db30 2023-03-03T07:57:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.813+0000 7f7bf999c700 1 --2- 172.21.15.98:0/549326771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7bf41609b0 0x7f7bf40c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.813+0000 7f7bf899a700 1 --2- 172.21.15.98:0/549326771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf415db30 0x7f7bf40c75a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7be4000c00 tx=0x7f7be4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.814+0000 7f7be9ffb700 1 -- 172.21.15.98:0/549326771 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be40038a0 con 0x7f7bf415db30 2023-03-03T07:57:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.814+0000 7f7be9ffb700 1 -- 172.21.15.98:0/549326771 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be401f070 con 0x7f7bf415db30 2023-03-03T07:57:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.814+0000 7f7be9ffb700 1 -- 172.21.15.98:0/549326771 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4003c00 con 0x7f7bf415db30 2023-03-03T07:57:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.814+0000 7f7bfa19d700 1 -- 172.21.15.98:0/549326771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf415cdf0 con 0x7f7bf415db30 2023-03-03T07:57:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.814+0000 7f7bfa19d700 1 -- 172.21.15.98:0/549326771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf40cdd40 con 0x7f7bf415db30 2023-03-03T07:57:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.815+0000 7f7bfa19d700 1 -- 172.21.15.98:0/549326771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf415d0d0 con 0x7f7bf415db30 2023-03-03T07:57:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.816+0000 7f7be9ffb700 1 -- 172.21.15.98:0/549326771 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7be400f050 con 0x7f7bf415db30 2023-03-03T07:57:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.817+0000 7f7be9ffb700 1 --2- 172.21.15.98:0/549326771 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7bd80682f0 0x7f7bd806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.817+0000 7f7be9ffb700 1 -- 172.21.15.98:0/549326771 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7be4021070 con 0x7f7bf415db30 2023-03-03T07:57:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.818+0000 7f7bf919b700 1 --2- 172.21.15.98:0/549326771 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7bd80682f0 0x7f7bd806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.818+0000 7f7bf919b700 1 --2- 172.21.15.98:0/549326771 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7bd80682f0 0x7f7bd806a7a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0009ac0 tx=0x7f7bf0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:13.821+0000 7f7be9ffb700 1 -- 172.21.15.98:0/549326771 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7be4053e50 con 0x7f7bf415db30 2023-03-03T07:57:14.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.012+0000 7f7bfa19d700 1 -- 172.21.15.98:0/549326771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7bf4068290 con 0x7f7bf415db30 2023-03-03T07:57:14.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.012+0000 7f7be9ffb700 1 -- 172.21.15.98:0/549326771 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7be402b0a0 con 0x7f7bf415db30 2023-03-03T07:57:14.014 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:57:14.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.015+0000 7f7bfa19d700 1 -- 172.21.15.98:0/549326771 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7bd80682f0 msgr2=0x7f7bd806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.015+0000 7f7bfa19d700 1 --2- 172.21.15.98:0/549326771 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7bd80682f0 0x7f7bd806a7a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0009ac0 tx=0x7f7bf0008040 comp rx=0 tx=0).stop 2023-03-03T07:57:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.015+0000 7f7bfa19d700 1 -- 172.21.15.98:0/549326771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf415db30 msgr2=0x7f7bf40c75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.015+0000 7f7bfa19d700 1 --2- 172.21.15.98:0/549326771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf415db30 0x7f7bf40c75a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7be4000c00 tx=0x7f7be4004750 comp rx=0 tx=0).stop 2023-03-03T07:57:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.015+0000 7f7bfa19d700 1 -- 172.21.15.98:0/549326771 shutdown_connections 2023-03-03T07:57:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.015+0000 7f7bfa19d700 1 --2- 172.21.15.98:0/549326771 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7bd80682f0 0x7f7bd806a7a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.015+0000 7f7bfa19d700 1 --2- 172.21.15.98:0/549326771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7bf41609b0 0x7f7bf40c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.015+0000 7f7bfa19d700 1 --2- 172.21.15.98:0/549326771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bf40cf570 0x7f7bf41f19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.015+0000 7f7bfa19d700 1 --2- 172.21.15.98:0/549326771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf415db30 0x7f7bf40c75a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.015+0000 7f7bfa19d700 1 -- 172.21.15.98:0/549326771 >> 172.21.15.98:0/549326771 conn(0x7f7bf406b880 msgr2=0x7f7bf40b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.015+0000 7f7bfa19d700 1 -- 172.21.15.98:0/549326771 shutdown_connections 2023-03-03T07:57:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.015+0000 7f7bfa19d700 1 -- 172.21.15.98:0/549326771 wait complete. 2023-03-03T07:57:14.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:57:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.293+0000 7f15b01ff700 1 Processor -- start 2023-03-03T07:57:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.294+0000 7f15b01ff700 1 -- start start 2023-03-03T07:57:14.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.295+0000 7f15b01ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a80b8d10 0x7f15a80cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:14.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.295+0000 7f15b01ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15a80b5310 0x7f15a80d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:14.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.295+0000 7f15b01ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a80cd7d0 con 0x7f15a80b26d0 2023-03-03T07:57:14.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.295+0000 7f15b01ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a80b90e0 con 0x7f15a80b5310 2023-03-03T07:57:14.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.295+0000 7f15b01ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a80cda20 con 0x7f15a80b8d10 2023-03-03T07:57:14.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.296+0000 7f15ad79a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f15a80b26d0 0x7f15a80b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55158/0 (socket says 172.21.15.98:55158) 2023-03-03T07:57:14.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.296+0000 7f15ad79a700 1 -- 172.21.15.98:0/3058202653 learned_addr learned my addr 172.21.15.98:0/3058202653 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:14.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.296+0000 7f15adf9b700 1 --2- 172.21.15.98:0/3058202653 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a80b8d10 0x7f15a80cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:14.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.296+0000 7f15ae79c700 1 --2- 172.21.15.98:0/3058202653 >> v2:172.21.15.98:3300/0 conn(0x7f15a80b5310 0x7f15a80d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:14.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.296+0000 7f15acf99700 1 -- 172.21.15.98:0/3058202653 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727289190 0 0) 0x7f15a80cd7d0 con 0x7f15a80b26d0 2023-03-03T07:57:14.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.296+0000 7f15adf9b700 1 -- 172.21.15.98:0/3058202653 >> v2:172.21.15.98:3300/0 conn(0x7f15a80b5310 msgr2=0x7f15a80d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:14.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.296+0000 7f15adf9b700 1 --2- 172.21.15.98:0/3058202653 >> v2:172.21.15.98:3300/0 conn(0x7f15a80b5310 0x7f15a80d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.296+0000 7f15adf9b700 1 -- 172.21.15.98:0/3058202653 >> v1:172.21.15.98:6789/0 conn(0x7f15a80b26d0 legacy=0x7f15a80b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:14.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.296+0000 7f15adf9b700 1 -- 172.21.15.98:0/3058202653 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a80cdd00 con 0x7f15a80b8d10 2023-03-03T07:57:14.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.297+0000 7f15adf9b700 1 --2- 172.21.15.98:0/3058202653 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a80b8d10 0x7f15a80cffc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009840 tx=0x7f15a4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfb5db8a19ec7970 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:14.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.297+0000 7f15acf99700 1 -- 172.21.15.98:0/3058202653 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a4011830 con 0x7f15a80b8d10 2023-03-03T07:57:14.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.297+0000 7f15acf99700 1 -- 172.21.15.98:0/3058202653 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a4011990 con 0x7f15a80b8d10 2023-03-03T07:57:14.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.297+0000 7f15acf99700 1 -- 172.21.15.98:0/3058202653 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a4011d20 con 0x7f15a80b8d10 2023-03-03T07:57:14.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.297+0000 7f15b01ff700 1 -- 172.21.15.98:0/3058202653 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a80b8d10 msgr2=0x7f15a80cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:14.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.297+0000 7f15b01ff700 1 --2- 172.21.15.98:0/3058202653 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a80b8d10 0x7f15a80cffc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009840 tx=0x7f15a4009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:14.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.297+0000 7f15b01ff700 1 -- 172.21.15.98:0/3058202653 shutdown_connections 2023-03-03T07:57:14.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.297+0000 7f15b01ff700 1 --2- 172.21.15.98:0/3058202653 >> v2:172.21.15.98:3300/0 conn(0x7f15a80b5310 0x7f15a80d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.297+0000 7f15b01ff700 1 --2- 172.21.15.98:0/3058202653 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a80b8d10 0x7f15a80cffc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.297+0000 7f15b01ff700 1 -- 172.21.15.98:0/3058202653 >> 172.21.15.98:0/3058202653 conn(0x7f15a8073e70 msgr2=0x7f15a806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:14.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.298+0000 7f15b01ff700 1 -- 172.21.15.98:0/3058202653 shutdown_connections 2023-03-03T07:57:14.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.298+0000 7f15b01ff700 1 -- 172.21.15.98:0/3058202653 wait complete. 2023-03-03T07:57:14.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.300+0000 7f15b01ff700 1 Processor -- start 2023-03-03T07:57:14.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.300+0000 7f15b01ff700 1 -- start start 2023-03-03T07:57:14.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.301+0000 7f15b01ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15a80b26d0 0x7f15a814d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:14.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.301+0000 7f15b01ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a80b5310 0x7f15a80ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:14.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.301+0000 7f15b01ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a80b8d10 0x7f15a80a9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:14.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.301+0000 7f15b01ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a80ab530 con 0x7f15a80b5310 2023-03-03T07:57:14.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.301+0000 7f15b01ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a80cc990 con 0x7f15a80b26d0 2023-03-03T07:57:14.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.301+0000 7f15b01ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a814db00 con 0x7f15a80b8d10 2023-03-03T07:57:14.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.301+0000 7f15ad79a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a80b5310 0x7f15a80ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:14.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.301+0000 7f15ad79a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a80b5310 0x7f15a80ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47944/0 (socket says 172.21.15.98:47944) 2023-03-03T07:57:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.301+0000 7f15ad79a700 1 -- 172.21.15.98:0/3631296999 learned_addr learned my addr 172.21.15.98:0/3631296999 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.301+0000 7f15adf9b700 1 --2- 172.21.15.98:0/3631296999 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15a80b26d0 0x7f15a814d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.301+0000 7f15ae79c700 1 --2- 172.21.15.98:0/3631296999 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a80b8d10 0x7f15a80a9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.302+0000 7f15ad79a700 1 -- 172.21.15.98:0/3631296999 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a80b8d10 msgr2=0x7f15a80a9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.302+0000 7f15ad79a700 1 --2- 172.21.15.98:0/3631296999 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a80b8d10 0x7f15a80a9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.302+0000 7f15ad79a700 1 -- 172.21.15.98:0/3631296999 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15a80b26d0 msgr2=0x7f15a814d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.302+0000 7f15ad79a700 1 --2- 172.21.15.98:0/3631296999 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15a80b26d0 0x7f15a814d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.302+0000 7f15ad79a700 1 -- 172.21.15.98:0/3631296999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40094a0 con 0x7f15a80b5310 2023-03-03T07:57:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.302+0000 7f15ae79c700 1 --2- 172.21.15.98:0/3631296999 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a80b8d10 0x7f15a80a9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.302+0000 7f15ad79a700 1 --2- 172.21.15.98:0/3631296999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a80b5310 0x7f15a80ccdb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f159800c580 tx=0x7f159800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.302+0000 7f159e7fc700 1 -- 172.21.15.98:0/3631296999 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1598004c80 con 0x7f15a80b5310 2023-03-03T07:57:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.302+0000 7f15b01ff700 1 -- 172.21.15.98:0/3631296999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a80cdfe0 con 0x7f15a80b5310 2023-03-03T07:57:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.302+0000 7f159e7fc700 1 -- 172.21.15.98:0/3631296999 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1598004e00 con 0x7f15a80b5310 2023-03-03T07:57:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.302+0000 7f159e7fc700 1 -- 172.21.15.98:0/3631296999 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1598007310 con 0x7f15a80b5310 2023-03-03T07:57:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.303+0000 7f15b01ff700 1 -- 172.21.15.98:0/3631296999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a80d0e80 con 0x7f15a80b5310 2023-03-03T07:57:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.304+0000 7f15b01ff700 1 -- 172.21.15.98:0/3631296999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a80d0500 con 0x7f15a80b5310 2023-03-03T07:57:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.305+0000 7f159e7fc700 1 -- 172.21.15.98:0/3631296999 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f159801a020 con 0x7f15a80b5310 2023-03-03T07:57:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.306+0000 7f159e7fc700 1 --2- 172.21.15.98:0/3631296999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f158c068320 0x7f158c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.306+0000 7f159e7fc700 1 -- 172.21.15.98:0/3631296999 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1598085640 con 0x7f15a80b5310 2023-03-03T07:57:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.306+0000 7f15adf9b700 1 --2- 172.21.15.98:0/3631296999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f158c068320 0x7f158c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:14.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.307+0000 7f15adf9b700 1 --2- 172.21.15.98:0/3631296999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f158c068320 0x7f158c06a7d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f15a40090b0 tx=0x7f15a4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:14.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.310+0000 7f159e7fc700 1 -- 172.21.15.98:0/3631296999 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f15980520a0 con 0x7f15a80b5310 2023-03-03T07:57:14.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.501+0000 7f15b01ff700 1 -- 172.21.15.98:0/3631296999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f15a80ccad0 con 0x7f15a80b5310 2023-03-03T07:57:14.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.502+0000 7f159e7fc700 1 -- 172.21.15.98:0/3631296999 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v34) v1 ==== 92+0+10 (secure 0 0 0) 0x7f15a80ccad0 con 0x7f15a80b5310 2023-03-03T07:57:14.503 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:57:14.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.504+0000 7f15b01ff700 1 -- 172.21.15.98:0/3631296999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f158c068320 msgr2=0x7f158c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:14.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.504+0000 7f15b01ff700 1 --2- 172.21.15.98:0/3631296999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f158c068320 0x7f158c06a7d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f15a40090b0 tx=0x7f15a4008030 comp rx=0 tx=0).stop 2023-03-03T07:57:14.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.504+0000 7f15b01ff700 1 -- 172.21.15.98:0/3631296999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a80b5310 msgr2=0x7f15a80ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:14.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.504+0000 7f15b01ff700 1 --2- 172.21.15.98:0/3631296999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a80b5310 0x7f15a80ccdb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f159800c580 tx=0x7f159800c890 comp rx=0 tx=0).stop 2023-03-03T07:57:14.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.504+0000 7f15b01ff700 1 -- 172.21.15.98:0/3631296999 shutdown_connections 2023-03-03T07:57:14.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.504+0000 7f15b01ff700 1 --2- 172.21.15.98:0/3631296999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f158c068320 0x7f158c06a7d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.504+0000 7f15b01ff700 1 --2- 172.21.15.98:0/3631296999 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a80b8d10 0x7f15a80a9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.504+0000 7f15b01ff700 1 --2- 172.21.15.98:0/3631296999 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15a80b26d0 0x7f15a814d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.504+0000 7f15b01ff700 1 --2- 172.21.15.98:0/3631296999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a80b5310 0x7f15a80ccdb0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.504+0000 7f15b01ff700 1 -- 172.21.15.98:0/3631296999 >> 172.21.15.98:0/3631296999 conn(0x7f15a8073e70 msgr2=0x7f15a80b6d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.505+0000 7f15b01ff700 1 -- 172.21.15.98:0/3631296999 shutdown_connections 2023-03-03T07:57:14.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.505+0000 7f15b01ff700 1 -- 172.21.15.98:0/3631296999 wait complete. 2023-03-03T07:57:14.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-03-03T07:57:14.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.779+0000 7efe1139d700 1 Processor -- start 2023-03-03T07:57:14.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.780+0000 7efe1139d700 1 -- start start 2023-03-03T07:57:14.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.781+0000 7efe1139d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efe0c06c7c0 0x7efe0c1599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:14.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.781+0000 7efe1139d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efe0c159ee0 0x7efe0c0cd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:14.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.781+0000 7efe1139d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe0c0cd000 con 0x7efe0c1595d0 2023-03-03T07:57:14.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.781+0000 7efe1139d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe0c06cd80 con 0x7efe0c06c7c0 2023-03-03T07:57:14.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.781+0000 7efe1139d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe0c06d060 con 0x7efe0c159ee0 2023-03-03T07:57:14.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.782+0000 7efe03fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efe0c06c7c0 0x7efe0c1599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:14.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.782+0000 7efe0bfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efe0c1595d0 0x7efe0c0c9d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55166/0 (socket says 172.21.15.98:55166) 2023-03-03T07:57:14.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.782+0000 7efe0bfff700 1 -- 172.21.15.98:0/1138543282 learned_addr learned my addr 172.21.15.98:0/1138543282 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:14.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.782+0000 7efe10b9c700 1 --2- 172.21.15.98:0/1138543282 >> v2:172.21.15.100:3300/0 conn(0x7efe0c159ee0 0x7efe0c0cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:14.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.782+0000 7efe03fff700 1 -- 172.21.15.98:0/1138543282 >> v2:172.21.15.100:3300/0 conn(0x7efe0c159ee0 msgr2=0x7efe0c0cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:14.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.782+0000 7efe03fff700 1 --2- 172.21.15.98:0/1138543282 >> v2:172.21.15.100:3300/0 conn(0x7efe0c159ee0 0x7efe0c0cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.782+0000 7efe03fff700 1 -- 172.21.15.98:0/1138543282 >> v1:172.21.15.98:6789/0 conn(0x7efe0c1595d0 legacy=0x7efe0c0c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:14.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.782+0000 7efe0b7fe700 1 -- 172.21.15.98:0/1138543282 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118815230 0 0) 0x7efe0c0cd000 con 0x7efe0c1595d0 2023-03-03T07:57:14.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.782+0000 7efe03fff700 1 -- 172.21.15.98:0/1138543282 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c06d340 con 0x7efe0c06c7c0 2023-03-03T07:57:14.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.783+0000 7efe03fff700 1 --2- 172.21.15.98:0/1138543282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c06c7c0 0x7efe0c1599a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7efe0c0b0780 tx=0x7efdf4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de95ba8ef5c28f24 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:14.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.783+0000 7efe0b7fe700 1 -- 172.21.15.98:0/1138543282 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf4011830 con 0x7efe0c06c7c0 2023-03-03T07:57:14.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.783+0000 7efe0b7fe700 1 -- 172.21.15.98:0/1138543282 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf4011990 con 0x7efe0c06c7c0 2023-03-03T07:57:14.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.783+0000 7efe0b7fe700 1 -- 172.21.15.98:0/1138543282 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf4011d20 con 0x7efe0c06c7c0 2023-03-03T07:57:14.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.783+0000 7efe1139d700 1 -- 172.21.15.98:0/1138543282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c06c7c0 msgr2=0x7efe0c1599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:14.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.783+0000 7efe1139d700 1 --2- 172.21.15.98:0/1138543282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c06c7c0 0x7efe0c1599a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7efe0c0b0780 tx=0x7efdf4009b20 comp rx=0 tx=0).stop 2023-03-03T07:57:14.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.784+0000 7efe1139d700 1 -- 172.21.15.98:0/1138543282 shutdown_connections 2023-03-03T07:57:14.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.784+0000 7efe1139d700 1 --2- 172.21.15.98:0/1138543282 >> v2:172.21.15.100:3300/0 conn(0x7efe0c159ee0 0x7efe0c0cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.784+0000 7efe1139d700 1 --2- 172.21.15.98:0/1138543282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c06c7c0 0x7efe0c1599a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.784+0000 7efe1139d700 1 -- 172.21.15.98:0/1138543282 >> 172.21.15.98:0/1138543282 conn(0x7efe0c0cf950 msgr2=0x7efe0c0cb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.784+0000 7efe1139d700 1 -- 172.21.15.98:0/1138543282 shutdown_connections 2023-03-03T07:57:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.784+0000 7efe1139d700 1 -- 172.21.15.98:0/1138543282 wait complete. 2023-03-03T07:57:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.787+0000 7efe1139d700 1 Processor -- start 2023-03-03T07:57:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.787+0000 7efe1139d700 1 -- start start 2023-03-03T07:57:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.788+0000 7efe1139d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c06c7c0 0x7efe0c1599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:14.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.788+0000 7efe1139d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe0c1595d0 0x7efe0c071510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:14.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.788+0000 7efe1139d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe0c159ee0 0x7efe0c0b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:14.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.788+0000 7efe1139d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c0b0370 con 0x7efe0c06c7c0 2023-03-03T07:57:14.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.788+0000 7efe1139d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c071a50 con 0x7efe0c159ee0 2023-03-03T07:57:14.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.788+0000 7efe1139d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c154030 con 0x7efe0c1595d0 2023-03-03T07:57:14.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.789+0000 7efe0bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c06c7c0 0x7efe0c1599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:14.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.789+0000 7efe0bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c06c7c0 0x7efe0c1599a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47964/0 (socket says 172.21.15.98:47964) 2023-03-03T07:57:14.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.789+0000 7efe0bfff700 1 -- 172.21.15.98:0/202981753 learned_addr learned my addr 172.21.15.98:0/202981753 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:14.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.789+0000 7efe10b9c700 1 --2- 172.21.15.98:0/202981753 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe0c159ee0 0x7efe0c0b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:14.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.789+0000 7efe03fff700 1 --2- 172.21.15.98:0/202981753 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe0c1595d0 0x7efe0c071510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:14.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.789+0000 7efe0bfff700 1 -- 172.21.15.98:0/202981753 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe0c1595d0 msgr2=0x7efe0c071510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:14.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.789+0000 7efe0bfff700 1 --2- 172.21.15.98:0/202981753 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe0c1595d0 0x7efe0c071510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.789+0000 7efe0bfff700 1 -- 172.21.15.98:0/202981753 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe0c159ee0 msgr2=0x7efe0c0b34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:14.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.789+0000 7efe0bfff700 1 --2- 172.21.15.98:0/202981753 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe0c159ee0 0x7efe0c0b34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:14.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.789+0000 7efe0bfff700 1 -- 172.21.15.98:0/202981753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf40094a0 con 0x7efe0c06c7c0 2023-03-03T07:57:14.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.789+0000 7efe03fff700 1 --2- 172.21.15.98:0/202981753 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe0c1595d0 0x7efe0c071510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:14.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.789+0000 7efe0bfff700 1 --2- 172.21.15.98:0/202981753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c06c7c0 0x7efe0c1599a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7efdfc0011e0 tx=0x7efdfc00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:14.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.790+0000 7efe097fa700 1 -- 172.21.15.98:0/202981753 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc004c80 con 0x7efe0c06c7c0 2023-03-03T07:57:14.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.790+0000 7efe097fa700 1 -- 172.21.15.98:0/202981753 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdfc004de0 con 0x7efe0c06c7c0 2023-03-03T07:57:14.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.790+0000 7efe097fa700 1 -- 172.21.15.98:0/202981753 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc007310 con 0x7efe0c06c7c0 2023-03-03T07:57:14.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.790+0000 7efe1139d700 1 -- 172.21.15.98:0/202981753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe0c15a340 con 0x7efe0c06c7c0 2023-03-03T07:57:14.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.790+0000 7efe1139d700 1 -- 172.21.15.98:0/202981753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe0c06d620 con 0x7efe0c06c7c0 2023-03-03T07:57:14.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.791+0000 7efe1139d700 1 -- 172.21.15.98:0/202981753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe0c15a4d0 con 0x7efe0c06c7c0 2023-03-03T07:57:14.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.792+0000 7efe097fa700 1 -- 172.21.15.98:0/202981753 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efdfc01a020 con 0x7efe0c06c7c0 2023-03-03T07:57:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.794+0000 7efe097fa700 1 --2- 172.21.15.98:0/202981753 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efdec0682f0 0x7efdec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.794+0000 7efe097fa700 1 -- 172.21.15.98:0/202981753 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efdfc0855f0 con 0x7efe0c06c7c0 2023-03-03T07:57:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.794+0000 7efe03fff700 1 --2- 172.21.15.98:0/202981753 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efdec0682f0 0x7efdec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.795+0000 7efe03fff700 1 --2- 172.21.15.98:0/202981753 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efdec0682f0 0x7efdec06a7a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efdf40090b0 tx=0x7efdf4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:14.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.798+0000 7efe097fa700 1 -- 172.21.15.98:0/202981753 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efdfc0520b0 con 0x7efe0c06c7c0 2023-03-03T07:57:14.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.992+0000 7efe1139d700 1 -- 172.21.15.98:0/202981753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7efe0c0b04b0 con 0x7efe0c06c7c0 2023-03-03T07:57:14.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.993+0000 7efe097fa700 1 -- 172.21.15.98:0/202981753 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v34) v1 ==== 84+0+126 (secure 0 0 0) 0x7efdfc0556d0 con 0x7efe0c06c7c0 2023-03-03T07:57:14.995 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:14.995 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T07:57:14.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:14.995 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2023-03-03T07:57:14.995 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2023-03-03T07:57:14.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:14.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:14.996 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 2, 2023-03-03T07:57:14.996 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": "rbd" 2023-03-03T07:57:14.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:14.996 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T07:57:14.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.995+0000 7efe1139d700 1 -- 172.21.15.98:0/202981753 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efdec0682f0 msgr2=0x7efdec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:14.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.995+0000 7efe1139d700 1 --2- 172.21.15.98:0/202981753 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efdec0682f0 0x7efdec06a7a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efdf40090b0 tx=0x7efdf4008030 comp rx=0 tx=0).stop 2023-03-03T07:57:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.996+0000 7efe1139d700 1 -- 172.21.15.98:0/202981753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c06c7c0 msgr2=0x7efe0c1599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.996+0000 7efe1139d700 1 --2- 172.21.15.98:0/202981753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c06c7c0 0x7efe0c1599a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7efdfc0011e0 tx=0x7efdfc00ca30 comp rx=0 tx=0).stop 2023-03-03T07:57:14.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.996+0000 7efe1139d700 1 -- 172.21.15.98:0/202981753 shutdown_connections 2023-03-03T07:57:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.996+0000 7efe1139d700 1 --2- 172.21.15.98:0/202981753 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efdec0682f0 0x7efdec06a7a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.996+0000 7efe1139d700 1 --2- 172.21.15.98:0/202981753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c06c7c0 0x7efe0c1599a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.996+0000 7efe1139d700 1 --2- 172.21.15.98:0/202981753 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe0c159ee0 0x7efe0c0b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.996+0000 7efe1139d700 1 --2- 172.21.15.98:0/202981753 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe0c1595d0 0x7efe0c071510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.996+0000 7efe1139d700 1 -- 172.21.15.98:0/202981753 >> 172.21.15.98:0/202981753 conn(0x7efe0c0cf950 msgr2=0x7efe0c0b3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.996+0000 7efe1139d700 1 -- 172.21.15.98:0/202981753 shutdown_connections 2023-03-03T07:57:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:14.996+0000 7efe1139d700 1 -- 172.21.15.98:0/202981753 wait complete. 2023-03-03T07:57:15.024 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-03-03T07:57:15.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-03-03T07:57:15.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.274+0000 7f36e19ff700 1 Processor -- start 2023-03-03T07:57:15.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.274+0000 7f36e19ff700 1 -- start start 2023-03-03T07:57:15.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.276+0000 7f36e19ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36dc0b8880 0x7f36dc0cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:15.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.276+0000 7f36e19ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36dc0b5720 0x7f36dc15bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:15.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.276+0000 7f36e19ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36dc0d0ac0 con 0x7f36dc0b4ba0 2023-03-03T07:57:15.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.276+0000 7f36e19ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36dc0cff70 con 0x7f36dc0b5720 2023-03-03T07:57:15.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.276+0000 7f36e19ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc0d0250 con 0x7f36dc0b8880 2023-03-03T07:57:15.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.276+0000 7f36db7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36dc0b5720 0x7f36dc15bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:15.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.276+0000 7f36d3fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f36dc0b4ba0 0x7f36dc0b5010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38742/0 (socket says 172.21.15.98:38742) 2023-03-03T07:57:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.276+0000 7f36d3fff700 1 -- 172.21.15.98:0/2463818114 learned_addr learned my addr 172.21.15.98:0/2463818114 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.277+0000 7f36daffd700 1 --2- 172.21.15.98:0/2463818114 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36dc0b8880 0x7f36dc0cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.277+0000 7f36db7fe700 1 -- 172.21.15.98:0/2463818114 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36dc0b8880 msgr2=0x7f36dc0cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.277+0000 7f36db7fe700 1 --2- 172.21.15.98:0/2463818114 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36dc0b8880 0x7f36dc0cd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.277+0000 7f36db7fe700 1 -- 172.21.15.98:0/2463818114 >> v1:172.21.15.98:6789/0 conn(0x7f36dc0b4ba0 legacy=0x7f36dc0b5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.277+0000 7f36db7fe700 1 -- 172.21.15.98:0/2463818114 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36dc0d0530 con 0x7f36dc0b5720 2023-03-03T07:57:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.277+0000 7f36daffd700 1 --2- 172.21.15.98:0/2463818114 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36dc0b8880 0x7f36dc0cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.277+0000 7f36da7fc700 1 -- 172.21.15.98:0/2463818114 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361327831 0 0) 0x7f36dc0d0ac0 con 0x7f36dc0b4ba0 2023-03-03T07:57:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.277+0000 7f36db7fe700 1 --2- 172.21.15.98:0/2463818114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36dc0b5720 0x7f36dc15bf70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f36d4009d90 tx=0x7f36d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1df25bbfad2cf93 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.278+0000 7f36da7fc700 1 -- 172.21.15.98:0/2463818114 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d400e070 con 0x7f36dc0b5720 2023-03-03T07:57:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.278+0000 7f36da7fc700 1 -- 172.21.15.98:0/2463818114 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d400bbd0 con 0x7f36dc0b5720 2023-03-03T07:57:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.278+0000 7f36da7fc700 1 -- 172.21.15.98:0/2463818114 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d4012430 con 0x7f36dc0b5720 2023-03-03T07:57:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.278+0000 7f36e19ff700 1 -- 172.21.15.98:0/2463818114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36dc0b5720 msgr2=0x7f36dc15bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.278+0000 7f36e19ff700 1 --2- 172.21.15.98:0/2463818114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36dc0b5720 0x7f36dc15bf70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f36d4009d90 tx=0x7f36d400b330 comp rx=0 tx=0).stop 2023-03-03T07:57:15.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.278+0000 7f36e19ff700 1 -- 172.21.15.98:0/2463818114 shutdown_connections 2023-03-03T07:57:15.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.278+0000 7f36e19ff700 1 --2- 172.21.15.98:0/2463818114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36dc0b5720 0x7f36dc15bf70 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:15.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.278+0000 7f36e19ff700 1 --2- 172.21.15.98:0/2463818114 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36dc0b8880 0x7f36dc0cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:15.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.278+0000 7f36e19ff700 1 -- 172.21.15.98:0/2463818114 >> 172.21.15.98:0/2463818114 conn(0x7f36dc06ad80 msgr2=0x7f36dc06b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:15.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.278+0000 7f36e19ff700 1 -- 172.21.15.98:0/2463818114 shutdown_connections 2023-03-03T07:57:15.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.278+0000 7f36e19ff700 1 -- 172.21.15.98:0/2463818114 wait complete. 2023-03-03T07:57:15.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.281+0000 7f36e19ff700 1 Processor -- start 2023-03-03T07:57:15.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.282+0000 7f36e19ff700 1 -- start start 2023-03-03T07:57:15.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.283+0000 7f36e19ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36dc0b4ba0 0x7f36dc1494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.283+0000 7f36e19ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36dc0b5720 0x7f36dc1f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.283+0000 7f36e19ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36dc0b8880 0x7f36dc159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.283+0000 7f36e19ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc149a30 con 0x7f36dc0b8880 2023-03-03T07:57:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.283+0000 7f36e19ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc15bb30 con 0x7f36dc0b4ba0 2023-03-03T07:57:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.283+0000 7f36e19ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc15be10 con 0x7f36dc0b5720 2023-03-03T07:57:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.283+0000 7f36db7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36dc0b8880 0x7f36dc159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.283+0000 7f36daffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36dc0b4ba0 0x7f36dc1494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.283+0000 7f36db7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36dc0b8880 0x7f36dc159960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45426/0 (socket says 172.21.15.98:45426) 2023-03-03T07:57:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.283+0000 7f36db7fe700 1 -- 172.21.15.98:0/4048516892 learned_addr learned my addr 172.21.15.98:0/4048516892 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.283+0000 7f36d3fff700 1 --2- 172.21.15.98:0/4048516892 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36dc0b5720 0x7f36dc1f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.284+0000 7f36db7fe700 1 -- 172.21.15.98:0/4048516892 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36dc0b5720 msgr2=0x7f36dc1f2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.284+0000 7f36db7fe700 1 --2- 172.21.15.98:0/4048516892 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36dc0b5720 0x7f36dc1f2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.284+0000 7f36db7fe700 1 -- 172.21.15.98:0/4048516892 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36dc0b4ba0 msgr2=0x7f36dc1494f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.284+0000 7f36db7fe700 1 --2- 172.21.15.98:0/4048516892 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36dc0b4ba0 0x7f36dc1494f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.284+0000 7f36db7fe700 1 -- 172.21.15.98:0/4048516892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d4009a70 con 0x7f36dc0b8880 2023-03-03T07:57:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.284+0000 7f36daffd700 1 --2- 172.21.15.98:0/4048516892 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36dc0b4ba0 0x7f36dc1494f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.284+0000 7f36db7fe700 1 --2- 172.21.15.98:0/4048516892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36dc0b8880 0x7f36dc159960 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f36dc08b830 tx=0x7f36d4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.285+0000 7f36d8ff9700 1 -- 172.21.15.98:0/4048516892 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d4012430 con 0x7f36dc0b8880 2023-03-03T07:57:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.285+0000 7f36d8ff9700 1 -- 172.21.15.98:0/4048516892 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d4020070 con 0x7f36dc0b8880 2023-03-03T07:57:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.285+0000 7f36d8ff9700 1 -- 172.21.15.98:0/4048516892 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d4004280 con 0x7f36dc0b8880 2023-03-03T07:57:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.285+0000 7f36e19ff700 1 -- 172.21.15.98:0/4048516892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36dc0b8c50 con 0x7f36dc0b8880 2023-03-03T07:57:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.285+0000 7f36e19ff700 1 -- 172.21.15.98:0/4048516892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36dc0d0cf0 con 0x7f36dc0b8880 2023-03-03T07:57:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.286+0000 7f36e19ff700 1 -- 172.21.15.98:0/4048516892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36dc0cde70 con 0x7f36dc0b8880 2023-03-03T07:57:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.287+0000 7f36d8ff9700 1 -- 172.21.15.98:0/4048516892 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f36d40043e0 con 0x7f36dc0b8880 2023-03-03T07:57:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.288+0000 7f36d8ff9700 1 --2- 172.21.15.98:0/4048516892 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36bc0682f0 0x7f36bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.288+0000 7f36d8ff9700 1 -- 172.21.15.98:0/4048516892 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f36d4022070 con 0x7f36dc0b8880 2023-03-03T07:57:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.289+0000 7f36daffd700 1 --2- 172.21.15.98:0/4048516892 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36bc0682f0 0x7f36bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:15.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.289+0000 7f36daffd700 1 --2- 172.21.15.98:0/4048516892 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36bc0682f0 0x7f36bc06a7a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f36dc0cd7e0 tx=0x7f36cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:15.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.293+0000 7f36d8ff9700 1 -- 172.21.15.98:0/4048516892 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f36d4089e60 con 0x7f36dc0b8880 2023-03-03T07:57:15.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.485+0000 7f36e19ff700 1 -- 172.21.15.98:0/4048516892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f36dc04e6e0 con 0x7f36dc0b8880 2023-03-03T07:57:15.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.720+0000 7f36d8ff9700 1 -- 172.21.15.98:0/4048516892 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7f36d402c0b0 con 0x7f36dc0b8880 2023-03-03T07:57:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.724+0000 7f36e19ff700 1 -- 172.21.15.98:0/4048516892 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36bc0682f0 msgr2=0x7f36bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.724+0000 7f36e19ff700 1 --2- 172.21.15.98:0/4048516892 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36bc0682f0 0x7f36bc06a7a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f36dc0cd7e0 tx=0x7f36cc008040 comp rx=0 tx=0).stop 2023-03-03T07:57:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.724+0000 7f36e19ff700 1 -- 172.21.15.98:0/4048516892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36dc0b8880 msgr2=0x7f36dc159960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.724+0000 7f36e19ff700 1 --2- 172.21.15.98:0/4048516892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36dc0b8880 0x7f36dc159960 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f36dc08b830 tx=0x7f36d4004ad0 comp rx=0 tx=0).stop 2023-03-03T07:57:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.725+0000 7f36e19ff700 1 -- 172.21.15.98:0/4048516892 shutdown_connections 2023-03-03T07:57:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.725+0000 7f36e19ff700 1 --2- 172.21.15.98:0/4048516892 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36bc0682f0 0x7f36bc06a7a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.725+0000 7f36e19ff700 1 --2- 172.21.15.98:0/4048516892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36dc0b8880 0x7f36dc159960 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.725+0000 7f36e19ff700 1 --2- 172.21.15.98:0/4048516892 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36dc0b4ba0 0x7f36dc1494f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.725+0000 7f36e19ff700 1 --2- 172.21.15.98:0/4048516892 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36dc0b5720 0x7f36dc1f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.725+0000 7f36e19ff700 1 -- 172.21.15.98:0/4048516892 >> 172.21.15.98:0/4048516892 conn(0x7f36dc06ad80 msgr2=0x7f36dc0b5010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.725+0000 7f36e19ff700 1 -- 172.21.15.98:0/4048516892 shutdown_connections 2023-03-03T07:57:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:15.725+0000 7f36e19ff700 1 -- 172.21.15.98:0/4048516892 wait complete. 2023-03-03T07:57:15.729 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2023-03-03T07:57:15.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-03-03T07:57:16.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.009+0000 7fd0869ff700 1 Processor -- start 2023-03-03T07:57:16.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.009+0000 7fd0869ff700 1 -- start start 2023-03-03T07:57:16.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.010+0000 7fd0869ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd08015c830 0x7fd08006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:16.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.010+0000 7fd0869ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd08015cc00 0x7fd0800d0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:16.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.011+0000 7fd0869ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd0800b4c70 0x7fd0800b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:16.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.011+0000 7fd0869ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0800b5610 con 0x7fd08015c830 2023-03-03T07:57:16.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.011+0000 7fd0869ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0800d0540 con 0x7fd0800b4c70 2023-03-03T07:57:16.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.011+0000 7fd0869ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800cd970 con 0x7fd08015cc00 2023-03-03T07:57:16.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.011+0000 7fd07ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd08015c830 0x7fd08006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:16.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.011+0000 7fd07ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd08015c830 0x7fd08006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45440/0 (socket says 172.21.15.98:45440) 2023-03-03T07:57:16.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.011+0000 7fd07ffff700 1 -- 172.21.15.98:0/4119634833 learned_addr learned my addr 172.21.15.98:0/4119634833 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:16.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.011+0000 7fd07f7fe700 1 --2- 172.21.15.98:0/4119634833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd08015cc00 0x7fd0800d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:16.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.011+0000 7fd084f9c700 1 --2- 172.21.15.98:0/4119634833 >> v2:172.21.15.100:3300/0 conn(0x7fd0800b4c70 0x7fd0800b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:16.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.011+0000 7fd07ffff700 1 -- 172.21.15.98:0/4119634833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd08015cc00 msgr2=0x7fd0800d0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:16.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.011+0000 7fd07ffff700 1 --2- 172.21.15.98:0/4119634833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd08015cc00 0x7fd0800d0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:16.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.011+0000 7fd07ffff700 1 -- 172.21.15.98:0/4119634833 >> v2:172.21.15.100:3300/0 conn(0x7fd0800b4c70 msgr2=0x7fd0800b50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:16.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.011+0000 7fd07ffff700 1 --2- 172.21.15.98:0/4119634833 >> v2:172.21.15.100:3300/0 conn(0x7fd0800b4c70 0x7fd0800b50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:16.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.011+0000 7fd07ffff700 1 -- 172.21.15.98:0/4119634833 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0800cdc50 con 0x7fd08015c830 2023-03-03T07:57:16.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.011+0000 7fd084f9c700 1 --2- 172.21.15.98:0/4119634833 >> v2:172.21.15.100:3300/0 conn(0x7fd0800b4c70 0x7fd0800b50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:16.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.012+0000 7fd07ffff700 1 --2- 172.21.15.98:0/4119634833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd08015c830 0x7fd08006b9a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd0800d0ca0 tx=0x7fd070009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d083d775bd6acf2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:16.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.012+0000 7fd07effd700 1 -- 172.21.15.98:0/4119634833 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070011830 con 0x7fd08015c830 2023-03-03T07:57:16.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.012+0000 7fd07effd700 1 -- 172.21.15.98:0/4119634833 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd070011990 con 0x7fd08015c830 2023-03-03T07:57:16.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.012+0000 7fd07effd700 1 -- 172.21.15.98:0/4119634833 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070011d20 con 0x7fd08015c830 2023-03-03T07:57:16.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.013+0000 7fd0869ff700 1 -- 172.21.15.98:0/4119634833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd08015c830 msgr2=0x7fd08006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:16.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.013+0000 7fd0869ff700 1 --2- 172.21.15.98:0/4119634833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd08015c830 0x7fd08006b9a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd0800d0ca0 tx=0x7fd070009bd0 comp rx=0 tx=0).stop 2023-03-03T07:57:16.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.013+0000 7fd0869ff700 1 -- 172.21.15.98:0/4119634833 shutdown_connections 2023-03-03T07:57:16.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.013+0000 7fd0869ff700 1 --2- 172.21.15.98:0/4119634833 >> v2:172.21.15.100:3300/0 conn(0x7fd0800b4c70 0x7fd0800b50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:16.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.013+0000 7fd0869ff700 1 --2- 172.21.15.98:0/4119634833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd08015c830 0x7fd08006b9a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:16.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.013+0000 7fd0869ff700 1 --2- 172.21.15.98:0/4119634833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd08015cc00 0x7fd0800d0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:16.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.013+0000 7fd0869ff700 1 -- 172.21.15.98:0/4119634833 >> 172.21.15.98:0/4119634833 conn(0x7fd080073e70 msgr2=0x7fd08006ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:16.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.013+0000 7fd0869ff700 1 -- 172.21.15.98:0/4119634833 shutdown_connections 2023-03-03T07:57:16.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.013+0000 7fd0869ff700 1 -- 172.21.15.98:0/4119634833 wait complete. 2023-03-03T07:57:16.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.015+0000 7fd0869ff700 1 Processor -- start 2023-03-03T07:57:16.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.015+0000 7fd0869ff700 1 -- start start 2023-03-03T07:57:16.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.016+0000 7fd0869ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd0800b4c70 0x7fd0800ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:16.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.016+0000 7fd0869ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd08015c830 0x7fd0800b8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:16.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.016+0000 7fd0869ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd08015cc00 0x7fd080154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:16.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.016+0000 7fd0869ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800ca9b0 con 0x7fd08015cc00 2023-03-03T07:57:16.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.016+0000 7fd0869ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800b8e90 con 0x7fd0800b4c70 2023-03-03T07:57:16.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.016+0000 7fd0869ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800ccbf0 con 0x7fd08015c830 2023-03-03T07:57:16.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.017+0000 7fd084f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd08015cc00 0x7fd080154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:16.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.017+0000 7fd084f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd08015cc00 0x7fd080154e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45444/0 (socket says 172.21.15.98:45444) 2023-03-03T07:57:16.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.017+0000 7fd084f9c700 1 -- 172.21.15.98:0/3717608274 learned_addr learned my addr 172.21.15.98:0/3717608274 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:16.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.017+0000 7fd07ffff700 1 --2- 172.21.15.98:0/3717608274 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd0800b4c70 0x7fd0800ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:16.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.017+0000 7fd07f7fe700 1 --2- 172.21.15.98:0/3717608274 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd08015c830 0x7fd0800b8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:16.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.017+0000 7fd084f9c700 1 -- 172.21.15.98:0/3717608274 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd08015c830 msgr2=0x7fd0800b8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:16.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.017+0000 7fd084f9c700 1 --2- 172.21.15.98:0/3717608274 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd08015c830 0x7fd0800b8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:16.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.017+0000 7fd084f9c700 1 -- 172.21.15.98:0/3717608274 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd0800b4c70 msgr2=0x7fd0800ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:16.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.017+0000 7fd084f9c700 1 --2- 172.21.15.98:0/3717608274 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd0800b4c70 0x7fd0800ca470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:16.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.017+0000 7fd084f9c700 1 -- 172.21.15.98:0/3717608274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd070009550 con 0x7fd08015cc00 2023-03-03T07:57:16.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.017+0000 7fd084f9c700 1 --2- 172.21.15.98:0/3717608274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd08015cc00 0x7fd080154e80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd078007e70 tx=0x7fd07800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:16.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.018+0000 7fd07cff9700 1 -- 172.21.15.98:0/3717608274 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07800dd20 con 0x7fd08015cc00 2023-03-03T07:57:16.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.018+0000 7fd07cff9700 1 -- 172.21.15.98:0/3717608274 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07800f040 con 0x7fd08015cc00 2023-03-03T07:57:16.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.018+0000 7fd07cff9700 1 -- 172.21.15.98:0/3717608274 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07800dea0 con 0x7fd08015cc00 2023-03-03T07:57:16.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.018+0000 7fd0869ff700 1 -- 172.21.15.98:0/3717608274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0800d0e60 con 0x7fd08015cc00 2023-03-03T07:57:16.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.018+0000 7fd0869ff700 1 -- 172.21.15.98:0/3717608274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0800cdf30 con 0x7fd08015cc00 2023-03-03T07:57:16.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.019+0000 7fd0869ff700 1 -- 172.21.15.98:0/3717608274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0800d1140 con 0x7fd08015cc00 2023-03-03T07:57:16.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.020+0000 7fd07cff9700 1 -- 172.21.15.98:0/3717608274 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd07801c030 con 0x7fd08015cc00 2023-03-03T07:57:16.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.022+0000 7fd07cff9700 1 --2- 172.21.15.98:0/3717608274 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd06c0682f0 0x7fd06c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:16.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.022+0000 7fd07cff9700 1 -- 172.21.15.98:0/3717608274 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5580+0+0 (secure 0 0 0) 0x7fd0780859f0 con 0x7fd08015cc00 2023-03-03T07:57:16.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.022+0000 7fd07ffff700 1 --2- 172.21.15.98:0/3717608274 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd06c0682f0 0x7fd06c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:16.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.023+0000 7fd07ffff700 1 --2- 172.21.15.98:0/3717608274 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd06c0682f0 0x7fd06c06a7a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd070014040 tx=0x7fd070008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:16.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.026+0000 7fd07cff9700 1 -- 172.21.15.98:0/3717608274 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd0780522e0 con 0x7fd08015cc00 2023-03-03T07:57:16.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.221+0000 7fd0869ff700 1 -- 172.21.15.98:0/3717608274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd08004e6e0 con 0x7fd08015cc00 2023-03-03T07:57:16.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.735+0000 7fd07cff9700 1 -- 172.21.15.98:0/3717608274 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v36) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd078055900 con 0x7fd08015cc00 2023-03-03T07:57:16.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.741+0000 7fd0869ff700 1 -- 172.21.15.98:0/3717608274 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd06c0682f0 msgr2=0x7fd06c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:16.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.741+0000 7fd0869ff700 1 --2- 172.21.15.98:0/3717608274 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd06c0682f0 0x7fd06c06a7a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd070014040 tx=0x7fd070008030 comp rx=0 tx=0).stop 2023-03-03T07:57:16.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.741+0000 7fd0869ff700 1 -- 172.21.15.98:0/3717608274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd08015cc00 msgr2=0x7fd080154e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:16.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.741+0000 7fd0869ff700 1 --2- 172.21.15.98:0/3717608274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd08015cc00 0x7fd080154e80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd078007e70 tx=0x7fd07800d3b0 comp rx=0 tx=0).stop 2023-03-03T07:57:16.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.743+0000 7fd0869ff700 1 -- 172.21.15.98:0/3717608274 shutdown_connections 2023-03-03T07:57:16.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.743+0000 7fd0869ff700 1 --2- 172.21.15.98:0/3717608274 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd06c0682f0 0x7fd06c06a7a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:16.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.743+0000 7fd0869ff700 1 --2- 172.21.15.98:0/3717608274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd08015cc00 0x7fd080154e80 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:16.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.743+0000 7fd0869ff700 1 --2- 172.21.15.98:0/3717608274 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd0800b4c70 0x7fd0800ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:16.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.743+0000 7fd0869ff700 1 --2- 172.21.15.98:0/3717608274 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd08015c830 0x7fd0800b8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:16.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.743+0000 7fd0869ff700 1 -- 172.21.15.98:0/3717608274 >> 172.21.15.98:0/3717608274 conn(0x7fd080073e70 msgr2=0x7fd08006ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:16.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.750+0000 7fd0869ff700 1 -- 172.21.15.98:0/3717608274 shutdown_connections 2023-03-03T07:57:16.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:16.751+0000 7fd0869ff700 1 -- 172.21.15.98:0/3717608274 wait complete. 2023-03-03T07:57:16.756 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2023-03-03T07:57:16.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-03-03T07:57:17.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.034+0000 7f127edff700 1 Processor -- start 2023-03-03T07:57:17.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.035+0000 7f127edff700 1 -- start start 2023-03-03T07:57:17.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.036+0000 7f127edff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f12780b26d0 0x7f12780b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:17.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.036+0000 7f127edff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12780b5140 0x7f12780d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:17.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.036+0000 7f127edff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12780cd7d0 con 0x7f12780b8d10 2023-03-03T07:57:17.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.036+0000 7f127edff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12780b55c0 con 0x7f12780b26d0 2023-03-03T07:57:17.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.036+0000 7f127edff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12780b90e0 con 0x7f12780b5140 2023-03-03T07:57:17.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.037+0000 7f127d39c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12780b5140 0x7f12780d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:17.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.037+0000 7f1277fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f12780b26d0 0x7f12780b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:17.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.037+0000 7f127d39c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12780b5140 0x7f12780d13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:47012/0 (socket says 172.21.15.98:47012) 2023-03-03T07:57:17.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.037+0000 7f127d39c700 1 -- 172.21.15.98:0/1596913933 learned_addr learned my addr 172.21.15.98:0/1596913933 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:17.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.037+0000 7f1277fff700 1 -- 172.21.15.98:0/1596913933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12780b5140 msgr2=0x7f12780d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:17.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.037+0000 7f1277fff700 1 --2- 172.21.15.98:0/1596913933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12780b5140 0x7f12780d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.037+0000 7f1277fff700 1 -- 172.21.15.98:0/1596913933 >> v1:172.21.15.100:6789/0 conn(0x7f12780b8d10 legacy=0x7f12780cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:17.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.037+0000 7f1277fff700 1 -- 172.21.15.98:0/1596913933 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12780cda20 con 0x7f12780b26d0 2023-03-03T07:57:17.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.037+0000 7f127d39c700 1 --2- 172.21.15.98:0/1596913933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12780b5140 0x7f12780d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.037+0000 7f12777fe700 1 -- 172.21.15.98:0/1596913933 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701289664 0 0) 0x7f12780cd7d0 con 0x7f12780b8d10 2023-03-03T07:57:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.038+0000 7f1277fff700 1 --2- 172.21.15.98:0/1596913933 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12780b26d0 0x7f12780b4c00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1264009840 tx=0x7f1264009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7522ea0aba0f8d6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.038+0000 7f12777fe700 1 -- 172.21.15.98:0/1596913933 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1264011830 con 0x7f12780b26d0 2023-03-03T07:57:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.038+0000 7f12777fe700 1 -- 172.21.15.98:0/1596913933 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1264011990 con 0x7f12780b26d0 2023-03-03T07:57:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.038+0000 7f12777fe700 1 -- 172.21.15.98:0/1596913933 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1264011d20 con 0x7f12780b26d0 2023-03-03T07:57:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.038+0000 7f127edff700 1 -- 172.21.15.98:0/1596913933 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12780b26d0 msgr2=0x7f12780b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.038+0000 7f127edff700 1 --2- 172.21.15.98:0/1596913933 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12780b26d0 0x7f12780b4c00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1264009840 tx=0x7f1264009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:17.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.039+0000 7f127edff700 1 -- 172.21.15.98:0/1596913933 shutdown_connections 2023-03-03T07:57:17.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.039+0000 7f127edff700 1 --2- 172.21.15.98:0/1596913933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12780b5140 0x7f12780d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.039+0000 7f127edff700 1 --2- 172.21.15.98:0/1596913933 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12780b26d0 0x7f12780b4c00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.039+0000 7f127edff700 1 -- 172.21.15.98:0/1596913933 >> 172.21.15.98:0/1596913933 conn(0x7f1278073e70 msgr2=0x7f127806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:17.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.039+0000 7f127edff700 1 -- 172.21.15.98:0/1596913933 shutdown_connections 2023-03-03T07:57:17.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.039+0000 7f127edff700 1 -- 172.21.15.98:0/1596913933 wait complete. 2023-03-03T07:57:17.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.041+0000 7f127edff700 1 Processor -- start 2023-03-03T07:57:17.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.041+0000 7f127edff700 1 -- start start 2023-03-03T07:57:17.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.042+0000 7f127edff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12780b26d0 0x7f12781518b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:17.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.042+0000 7f127edff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12780b5140 0x7f12780cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:17.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.042+0000 7f127edff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780b8d10 0x7f12780b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:17.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.042+0000 7f127edff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12781526d0 con 0x7f12780b8d10 2023-03-03T07:57:17.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.042+0000 7f127edff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1278151df0 con 0x7f12780b26d0 2023-03-03T07:57:17.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.042+0000 7f127edff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12781551f0 con 0x7f12780b5140 2023-03-03T07:57:17.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.042+0000 7f127d39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780b8d10 0x7f12780b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:17.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.042+0000 7f127d39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780b8d10 0x7f12780b40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45448/0 (socket says 172.21.15.98:45448) 2023-03-03T07:57:17.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.042+0000 7f127d39c700 1 -- 172.21.15.98:0/3377917584 learned_addr learned my addr 172.21.15.98:0/3377917584 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:17.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.042+0000 7f127cb9b700 1 --2- 172.21.15.98:0/3377917584 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12780b26d0 0x7f12781518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:17.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.043+0000 7f1277fff700 1 --2- 172.21.15.98:0/3377917584 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12780b5140 0x7f12780cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:17.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.043+0000 7f127d39c700 1 -- 172.21.15.98:0/3377917584 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12780b5140 msgr2=0x7f12780cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:17.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.043+0000 7f127d39c700 1 --2- 172.21.15.98:0/3377917584 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12780b5140 0x7f12780cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.043+0000 7f127d39c700 1 -- 172.21.15.98:0/3377917584 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12780b26d0 msgr2=0x7f12781518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:17.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.043+0000 7f127d39c700 1 --2- 172.21.15.98:0/3377917584 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12780b26d0 0x7f12781518b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.043+0000 7f127d39c700 1 -- 172.21.15.98:0/3377917584 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1270009b30 con 0x7f12780b8d10 2023-03-03T07:57:17.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.043+0000 7f1277fff700 1 --2- 172.21.15.98:0/3377917584 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12780b5140 0x7f12780cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:17.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.043+0000 7f127d39c700 1 --2- 172.21.15.98:0/3377917584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780b8d10 0x7f12780b40c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f127000efc0 tx=0x7f127000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:17.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.043+0000 7f12757fa700 1 -- 172.21.15.98:0/3377917584 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270018070 con 0x7f12780b8d10 2023-03-03T07:57:17.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.044+0000 7f12757fa700 1 -- 172.21.15.98:0/3377917584 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127000f040 con 0x7f12780b8d10 2023-03-03T07:57:17.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.044+0000 7f12757fa700 1 -- 172.21.15.98:0/3377917584 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270003c40 con 0x7f12780b8d10 2023-03-03T07:57:17.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.044+0000 7f127edff700 1 -- 172.21.15.98:0/3377917584 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12640094a0 con 0x7f12780b8d10 2023-03-03T07:57:17.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.044+0000 7f127edff700 1 -- 172.21.15.98:0/3377917584 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12780d0e80 con 0x7f12780b8d10 2023-03-03T07:57:17.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.045+0000 7f127edff700 1 -- 172.21.15.98:0/3377917584 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12780cdd00 con 0x7f12780b8d10 2023-03-03T07:57:17.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.046+0000 7f12757fa700 1 -- 172.21.15.98:0/3377917584 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f127000c930 con 0x7f12780b8d10 2023-03-03T07:57:17.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.047+0000 7f12757fa700 1 --2- 172.21.15.98:0/3377917584 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f125c0682f0 0x7f125c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:17.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.047+0000 7f12757fa700 1 -- 172.21.15.98:0/3377917584 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f1270086000 con 0x7f12780b8d10 2023-03-03T07:57:17.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.048+0000 7f127cb9b700 1 --2- 172.21.15.98:0/3377917584 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f125c0682f0 0x7f125c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:17.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.048+0000 7f127cb9b700 1 --2- 172.21.15.98:0/3377917584 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f125c0682f0 0x7f125c06a7a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f126c000f60 tx=0x7f126c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:17.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.052+0000 7f12757fa700 1 -- 172.21.15.98:0/3377917584 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f12700527d0 con 0x7f12780b8d10 2023-03-03T07:57:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.290+0000 7f127edff700 1 -- 172.21.15.98:0/3377917584 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f12780cdfe0 con 0x7f12780b8d10 2023-03-03T07:57:17.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.298+0000 7f12757fa700 1 -- 172.21.15.98:0/3377917584 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v11) v1 ==== 172+0+0 (secure 0 0 0) 0x7f1270009ce0 con 0x7f12780b8d10 2023-03-03T07:57:17.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.304+0000 7f127edff700 1 -- 172.21.15.98:0/3377917584 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f125c0682f0 msgr2=0x7f125c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:17.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.304+0000 7f127edff700 1 --2- 172.21.15.98:0/3377917584 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f125c0682f0 0x7f125c06a7a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f126c000f60 tx=0x7f126c009040 comp rx=0 tx=0).stop 2023-03-03T07:57:17.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.304+0000 7f127edff700 1 -- 172.21.15.98:0/3377917584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780b8d10 msgr2=0x7f12780b40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:17.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.304+0000 7f127edff700 1 --2- 172.21.15.98:0/3377917584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780b8d10 0x7f12780b40c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f127000efc0 tx=0x7f127000c3a0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.305+0000 7f127edff700 1 -- 172.21.15.98:0/3377917584 shutdown_connections 2023-03-03T07:57:17.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.305+0000 7f127edff700 1 --2- 172.21.15.98:0/3377917584 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f125c0682f0 0x7f125c06a7a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.305+0000 7f127edff700 1 --2- 172.21.15.98:0/3377917584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780b8d10 0x7f12780b40c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.305+0000 7f127edff700 1 --2- 172.21.15.98:0/3377917584 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12780b26d0 0x7f12781518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.305+0000 7f127edff700 1 --2- 172.21.15.98:0/3377917584 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12780b5140 0x7f12780cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.305+0000 7f127edff700 1 -- 172.21.15.98:0/3377917584 >> 172.21.15.98:0/3377917584 conn(0x7f1278073e70 msgr2=0x7f12780b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:17.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.307+0000 7f127edff700 1 -- 172.21.15.98:0/3377917584 shutdown_connections 2023-03-03T07:57:17.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.316+0000 7f127edff700 1 -- 172.21.15.98:0/3377917584 wait complete. 2023-03-03T07:57:17.406 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 5 and data pool 6 2023-03-03T07:57:17.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:57:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.654+0000 7f08603ff700 1 Processor -- start 2023-03-03T07:57:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.654+0000 7f08603ff700 1 -- start start 2023-03-03T07:57:17.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.655+0000 7f08603ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08580b87c0 0x7f08580cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:17.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.656+0000 7f08603ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08580b8b90 0x7f08580b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.656+0000 7f08603ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f08580b5240 0x7f08580b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.656+0000 7f08603ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08580d0ef0 con 0x7f08580b87c0 2023-03-03T07:57:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.656+0000 7f08603ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08580d0600 con 0x7f08580b5240 2023-03-03T07:57:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.656+0000 7f08603ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08580cd970 con 0x7f08580b8b90 2023-03-03T07:57:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.656+0000 7f085e19b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08580b87c0 0x7f08580cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.656+0000 7f085e19b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08580b87c0 0x7f08580cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45452/0 (socket says 172.21.15.98:45452) 2023-03-03T07:57:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.656+0000 7f085e19b700 1 -- 172.21.15.98:0/2905947896 learned_addr learned my addr 172.21.15.98:0/2905947896 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:17.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.656+0000 7f085e19b700 1 -- 172.21.15.98:0/2905947896 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08580b8b90 msgr2=0x7f08580b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.656+0000 7f085e19b700 1 --2- 172.21.15.98:0/2905947896 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08580b8b90 0x7f08580b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.656+0000 7f085e19b700 1 -- 172.21.15.98:0/2905947896 >> v2:172.21.15.100:3300/0 conn(0x7f08580b5240 msgr2=0x7f08580b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.656+0000 7f085e19b700 1 --2- 172.21.15.98:0/2905947896 >> v2:172.21.15.100:3300/0 conn(0x7f08580b5240 0x7f08580b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.656+0000 7f085e19b700 1 -- 172.21.15.98:0/2905947896 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08580cdc50 con 0x7f08580b87c0 2023-03-03T07:57:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.657+0000 7f085e19b700 1 --2- 172.21.15.98:0/2905947896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08580b87c0 0x7f08580cffe0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f08580d31a0 tx=0x7f085400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bd8ee6e0ef4de0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.657+0000 7f085d199700 1 -- 172.21.15.98:0/2905947896 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085400e070 con 0x7f08580b87c0 2023-03-03T07:57:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.657+0000 7f085d199700 1 -- 172.21.15.98:0/2905947896 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085400b9c0 con 0x7f08580b87c0 2023-03-03T07:57:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.657+0000 7f085d199700 1 -- 172.21.15.98:0/2905947896 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085400bd70 con 0x7f08580b87c0 2023-03-03T07:57:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.658+0000 7f08603ff700 1 -- 172.21.15.98:0/2905947896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08580b87c0 msgr2=0x7f08580cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.658+0000 7f08603ff700 1 --2- 172.21.15.98:0/2905947896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08580b87c0 0x7f08580cffe0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f08580d31a0 tx=0x7f085400b330 comp rx=0 tx=0).stop 2023-03-03T07:57:17.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.658+0000 7f08603ff700 1 -- 172.21.15.98:0/2905947896 shutdown_connections 2023-03-03T07:57:17.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.658+0000 7f08603ff700 1 --2- 172.21.15.98:0/2905947896 >> v2:172.21.15.100:3300/0 conn(0x7f08580b5240 0x7f08580b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.658+0000 7f08603ff700 1 --2- 172.21.15.98:0/2905947896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08580b87c0 0x7f08580cffe0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.658+0000 7f08603ff700 1 --2- 172.21.15.98:0/2905947896 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08580b8b90 0x7f08580b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.658+0000 7f08603ff700 1 -- 172.21.15.98:0/2905947896 >> 172.21.15.98:0/2905947896 conn(0x7f085806aee0 msgr2=0x7f085806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:17.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.658+0000 7f08603ff700 1 -- 172.21.15.98:0/2905947896 shutdown_connections 2023-03-03T07:57:17.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.658+0000 7f08603ff700 1 -- 172.21.15.98:0/2905947896 wait complete. 2023-03-03T07:57:17.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.661+0000 7f08603ff700 1 Processor -- start 2023-03-03T07:57:17.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.661+0000 7f08603ff700 1 -- start start 2023-03-03T07:57:17.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.662+0000 7f08603ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08580b5240 0x7f085814d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:17.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.662+0000 7f08603ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08580b87c0 0x7f08580c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:17.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.662+0000 7f08603ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08580b8b90 0x7f08580c7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:17.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.662+0000 7f08603ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08580b7870 con 0x7f08580b5240 2023-03-03T07:57:17.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.662+0000 7f08603ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08580b6bf0 con 0x7f08580b8b90 2023-03-03T07:57:17.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.662+0000 7f08603ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08580b6ed0 con 0x7f08580b87c0 2023-03-03T07:57:17.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.663+0000 7f085e19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08580b5240 0x7f085814d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:17.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.663+0000 7f085e19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08580b5240 0x7f085814d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45464/0 (socket says 172.21.15.98:45464) 2023-03-03T07:57:17.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.663+0000 7f085e19b700 1 -- 172.21.15.98:0/1943427771 learned_addr learned my addr 172.21.15.98:0/1943427771 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:17.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.663+0000 7f085e99c700 1 --2- 172.21.15.98:0/1943427771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08580b8b90 0x7f08580c7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:17.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.663+0000 7f085d99a700 1 --2- 172.21.15.98:0/1943427771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08580b87c0 0x7f08580c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:17.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.663+0000 7f085e19b700 1 -- 172.21.15.98:0/1943427771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08580b87c0 msgr2=0x7f08580c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:17.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.663+0000 7f085e19b700 1 --2- 172.21.15.98:0/1943427771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08580b87c0 0x7f08580c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.663+0000 7f085e19b700 1 -- 172.21.15.98:0/1943427771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08580b8b90 msgr2=0x7f08580c7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:17.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.663+0000 7f085e19b700 1 --2- 172.21.15.98:0/1943427771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08580b8b90 0x7f08580c7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.663+0000 7f085e19b700 1 -- 172.21.15.98:0/1943427771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0854009b30 con 0x7f08580b5240 2023-03-03T07:57:17.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.663+0000 7f085d99a700 1 --2- 172.21.15.98:0/1943427771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08580b87c0 0x7f08580c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:17.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.664+0000 7f085e19b700 1 --2- 172.21.15.98:0/1943427771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08580b5240 0x7f085814d590 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f085400b370 tx=0x7f0854004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:17.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.664+0000 7f084effd700 1 -- 172.21.15.98:0/1943427771 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0854004250 con 0x7f08580b5240 2023-03-03T07:57:17.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.664+0000 7f084effd700 1 -- 172.21.15.98:0/1943427771 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08540043b0 con 0x7f08580b5240 2023-03-03T07:57:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.664+0000 7f084effd700 1 -- 172.21.15.98:0/1943427771 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0854004760 con 0x7f08580b5240 2023-03-03T07:57:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.664+0000 7f08603ff700 1 -- 172.21.15.98:0/1943427771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08580cdf30 con 0x7f08580b5240 2023-03-03T07:57:17.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.664+0000 7f08603ff700 1 -- 172.21.15.98:0/1943427771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0858073a80 con 0x7f08580b5240 2023-03-03T07:57:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.665+0000 7f08603ff700 1 -- 172.21.15.98:0/1943427771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08580ce0c0 con 0x7f08580b5240 2023-03-03T07:57:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.666+0000 7f084effd700 1 -- 172.21.15.98:0/1943427771 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f085402c070 con 0x7f08580b5240 2023-03-03T07:57:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.667+0000 7f084effd700 1 --2- 172.21.15.98:0/1943427771 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f083c0682f0 0x7f083c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.668+0000 7f084effd700 1 -- 172.21.15.98:0/1943427771 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f08540875f0 con 0x7f08580b5240 2023-03-03T07:57:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.668+0000 7f085d99a700 1 --2- 172.21.15.98:0/1943427771 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f083c0682f0 0x7f083c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.669+0000 7f085d99a700 1 --2- 172.21.15.98:0/1943427771 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f083c0682f0 0x7f083c06a7a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0848006fd0 tx=0x7f0848008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.672+0000 7f084effd700 1 -- 172.21.15.98:0/1943427771 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0854053d60 con 0x7f08580b5240 2023-03-03T07:57:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.868+0000 7f08603ff700 1 -- 172.21.15.98:0/1943427771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08580729b0 con 0x7f08580b5240 2023-03-03T07:57:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.870+0000 7f084effd700 1 -- 172.21.15.98:0/1943427771 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+17992 (secure 0 0 0) 0x7f0854057380 con 0x7f08580b5240 2023-03-03T07:57:17.871 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:17.871 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":37,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:57:17.291173+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:57:15.545608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-03-03T07:57:16.283131+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:57:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.872+0000 7f08603ff700 1 -- 172.21.15.98:0/1943427771 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f083c0682f0 msgr2=0x7f083c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.872+0000 7f08603ff700 1 --2- 172.21.15.98:0/1943427771 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f083c0682f0 0x7f083c06a7a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0848006fd0 tx=0x7f0848008040 comp rx=0 tx=0).stop 2023-03-03T07:57:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.872+0000 7f08603ff700 1 -- 172.21.15.98:0/1943427771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08580b5240 msgr2=0x7f085814d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.872+0000 7f08603ff700 1 --2- 172.21.15.98:0/1943427771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08580b5240 0x7f085814d590 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f085400b370 tx=0x7f0854004ad0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.872+0000 7f08603ff700 1 -- 172.21.15.98:0/1943427771 shutdown_connections 2023-03-03T07:57:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.872+0000 7f08603ff700 1 --2- 172.21.15.98:0/1943427771 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f083c0682f0 0x7f083c06a7a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.872+0000 7f08603ff700 1 --2- 172.21.15.98:0/1943427771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08580b5240 0x7f085814d590 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.872+0000 7f08603ff700 1 --2- 172.21.15.98:0/1943427771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08580b8b90 0x7f08580c7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.872+0000 7f08603ff700 1 --2- 172.21.15.98:0/1943427771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08580b87c0 0x7f08580c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:17.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.872+0000 7f08603ff700 1 -- 172.21.15.98:0/1943427771 >> 172.21.15.98:0/1943427771 conn(0x7f085806aee0 msgr2=0x7f08580b7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:17.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.872+0000 7f08603ff700 1 -- 172.21.15.98:0/1943427771 shutdown_connections 2023-03-03T07:57:17.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:17.872+0000 7f08603ff700 1 -- 172.21.15.98:0/1943427771 wait complete. 2023-03-03T07:57:17.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:57:18.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.143+0000 7f1ab83ff700 1 Processor -- start 2023-03-03T07:57:18.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.144+0000 7f1ab83ff700 1 -- start start 2023-03-03T07:57:18.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.145+0000 7f1ab83ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1ab00cd8d0 0x7f1ab00b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:18.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.145+0000 7f1ab83ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab00cdd70 con 0x7f1ab015cc00 2023-03-03T07:57:18.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.145+0000 7f1ab83ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab00d1a00 con 0x7f1ab00b6610 2023-03-03T07:57:18.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.145+0000 7f1ab83ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab00ce040 con 0x7f1ab00cd8d0 2023-03-03T07:57:18.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.145+0000 7f1ab619b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1ab015cc00 0x7f1ab00d27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38756/0 (socket says 172.21.15.98:38756) 2023-03-03T07:57:18.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.145+0000 7f1ab619b700 1 -- 172.21.15.98:0/1989912964 learned_addr learned my addr 172.21.15.98:0/1989912964 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:18.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.146+0000 7f1ab599a700 1 --2- 172.21.15.98:0/1989912964 >> v2:172.21.15.100:3300/0 conn(0x7f1ab00cd8d0 0x7f1ab00b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:18.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.146+0000 7f1ab5199700 1 -- 172.21.15.98:0/1989912964 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156242955 0 0) 0x7f1ab00cdd70 con 0x7f1ab015cc00 2023-03-03T07:57:18.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.146+0000 7f1ab5199700 1 -- 172.21.15.98:0/1989912964 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a9c0035b0 con 0x7f1ab015cc00 2023-03-03T07:57:18.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.146+0000 7f1ab5199700 1 -- 172.21.15.98:0/1989912964 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1324541993 0 0) 0x7f1a9c0035b0 con 0x7f1ab015cc00 2023-03-03T07:57:18.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.146+0000 7f1ab5199700 1 -- 172.21.15.98:0/1989912964 >> v2:172.21.15.100:3300/0 conn(0x7f1ab00cd8d0 msgr2=0x7f1ab00b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:18.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.146+0000 7f1ab5199700 1 --2- 172.21.15.98:0/1989912964 >> v2:172.21.15.100:3300/0 conn(0x7f1ab00cd8d0 0x7f1ab00b60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:18.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.146+0000 7f1ab5199700 1 -- 172.21.15.98:0/1989912964 >> v1:172.21.15.100:6789/0 conn(0x7f1ab00b6610 legacy=0x7f1ab00cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:18.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.146+0000 7f1ab5199700 1 -- 172.21.15.98:0/1989912964 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab00d1ce0 con 0x7f1ab015cc00 2023-03-03T07:57:18.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.146+0000 7f1ab5199700 1 -- 172.21.15.98:0/1989912964 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f1aac001670 con 0x7f1ab015cc00 2023-03-03T07:57:18.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.147+0000 7f1ab5199700 1 -- 172.21.15.98:0/1989912964 >> v1:172.21.15.98:6789/0 conn(0x7f1ab015cc00 legacy=0x7f1ab00d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:18.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.147+0000 7f1ab5199700 1 --2- 172.21.15.98:0/1989912964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a9c003a00 0x7f1a9c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:18.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.147+0000 7f1ab5199700 1 --2- 172.21.15.98:0/1989912964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a9c004850 0x7f1a9c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:18.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.147+0000 7f1ab5199700 1 --2- 172.21.15.98:0/1989912964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a9c005200 0x7f1a9c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:18.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.147+0000 7f1ab5199700 1 -- 172.21.15.98:0/1989912964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab00ce040 con 0x7f1a9c005200 2023-03-03T07:57:18.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.147+0000 7f1ab5199700 1 -- 172.21.15.98:0/1989912964 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c00c080 con 0x7f1a9c004850 2023-03-03T07:57:18.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.147+0000 7f1ab5199700 1 -- 172.21.15.98:0/1989912964 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c00c240 con 0x7f1a9c003a00 2023-03-03T07:57:18.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.147+0000 7f1ab699c700 1 --2- 172.21.15.98:0/1989912964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a9c005200 0x7f1a9c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:18.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.147+0000 7f1ab619b700 1 --2- 172.21.15.98:0/1989912964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a9c003a00 0x7f1a9c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:18.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.147+0000 7f1ab599a700 1 --2- 172.21.15.98:0/1989912964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a9c004850 0x7f1a9c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:18.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.147+0000 7f1ab699c700 1 -- 172.21.15.98:0/1989912964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a9c003a00 msgr2=0x7f1a9c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:18.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.147+0000 7f1ab699c700 1 --2- 172.21.15.98:0/1989912964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a9c003a00 0x7f1a9c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:18.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.148+0000 7f1ab699c700 1 -- 172.21.15.98:0/1989912964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a9c004850 msgr2=0x7f1a9c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:18.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.148+0000 7f1ab699c700 1 --2- 172.21.15.98:0/1989912964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a9c004850 0x7f1a9c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:18.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.148+0000 7f1ab699c700 1 -- 172.21.15.98:0/1989912964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c0035b0 con 0x7f1a9c005200 2023-03-03T07:57:18.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.148+0000 7f1ab699c700 1 -- 172.21.15.98:0/1989912964 reap_dead start 2023-03-03T07:57:18.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.148+0000 7f1ab699c700 1 --2- 172.21.15.98:0/1989912964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a9c005200 0x7f1a9c00ba10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800c190 tx=0x7f1a9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe0714830da51d0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:18.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.148+0000 7f1ab5199700 1 -- 172.21.15.98:0/1989912964 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8001db0 con 0x7f1a9c005200 2023-03-03T07:57:18.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.148+0000 7f1ab5199700 1 -- 172.21.15.98:0/1989912964 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa800ccf0 con 0x7f1a9c005200 2023-03-03T07:57:18.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.148+0000 7f1ab5199700 1 -- 172.21.15.98:0/1989912964 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8003c40 con 0x7f1a9c005200 2023-03-03T07:57:18.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.149+0000 7f1ab83ff700 1 -- 172.21.15.98:0/1989912964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a9c005200 msgr2=0x7f1a9c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:18.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.149+0000 7f1ab83ff700 1 --2- 172.21.15.98:0/1989912964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a9c005200 0x7f1a9c00ba10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800c190 tx=0x7f1a9c005880 comp rx=0 tx=0).stop 2023-03-03T07:57:18.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.149+0000 7f1ab83ff700 1 -- 172.21.15.98:0/1989912964 shutdown_connections 2023-03-03T07:57:18.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.149+0000 7f1ab83ff700 1 --2- 172.21.15.98:0/1989912964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a9c005200 0x7f1a9c00ba10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:18.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.149+0000 7f1ab83ff700 1 -- 172.21.15.98:0/1989912964 >> 172.21.15.98:0/1989912964 conn(0x7f1ab006aee0 msgr2=0x7f1ab006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:18.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.149+0000 7f1ab83ff700 1 -- 172.21.15.98:0/1989912964 shutdown_connections 2023-03-03T07:57:18.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.149+0000 7f1ab83ff700 1 -- 172.21.15.98:0/1989912964 wait complete. 2023-03-03T07:57:18.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.151+0000 7f1ab83ff700 1 Processor -- start 2023-03-03T07:57:18.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.151+0000 7f1ab83ff700 1 -- start start 2023-03-03T07:57:18.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.152+0000 7f1ab83ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a9c005200 0x7f1ab0154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:18.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.152+0000 7f1ab83ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab01f2260 0x7f1ab014efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:18.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.152+0000 7f1ab83ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1ab0155be0 0x7f1ab0156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:18.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.152+0000 7f1ab83ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab0154790 con 0x7f1ab01f2260 2023-03-03T07:57:18.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.152+0000 7f1ab83ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab014f4f0 con 0x7f1ab0155be0 2023-03-03T07:57:18.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.152+0000 7f1ab83ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab015f850 con 0x7f1a9c005200 2023-03-03T07:57:18.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.152+0000 7f1ab599a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab01f2260 0x7f1ab014efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:18.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.152+0000 7f1ab599a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab01f2260 0x7f1ab014efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45490/0 (socket says 172.21.15.98:45490) 2023-03-03T07:57:18.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.152+0000 7f1ab599a700 1 -- 172.21.15.98:0/1700985108 learned_addr learned my addr 172.21.15.98:0/1700985108 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:18.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.153+0000 7f1ab699c700 1 --2- 172.21.15.98:0/1700985108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1ab0155be0 0x7f1ab0156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:18.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.153+0000 7f1ab619b700 1 --2- 172.21.15.98:0/1700985108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a9c005200 0x7f1ab0154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:18.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.153+0000 7f1ab599a700 1 -- 172.21.15.98:0/1700985108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a9c005200 msgr2=0x7f1ab0154250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:18.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.153+0000 7f1ab599a700 1 --2- 172.21.15.98:0/1700985108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a9c005200 0x7f1ab0154250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:18.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.153+0000 7f1ab599a700 1 -- 172.21.15.98:0/1700985108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1ab0155be0 msgr2=0x7f1ab0156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:18.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.153+0000 7f1ab599a700 1 --2- 172.21.15.98:0/1700985108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1ab0155be0 0x7f1ab0156080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:18.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.153+0000 7f1ab599a700 1 -- 172.21.15.98:0/1700985108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab00d22a0 con 0x7f1ab01f2260 2023-03-03T07:57:18.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.153+0000 7f1ab599a700 1 --2- 172.21.15.98:0/1700985108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab01f2260 0x7f1ab014efb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00119a0 tx=0x7f1aa0011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:18.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.153+0000 7f1aa6ffd700 1 -- 172.21.15.98:0/1700985108 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa000f940 con 0x7f1ab01f2260 2023-03-03T07:57:18.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.153+0000 7f1aa6ffd700 1 -- 172.21.15.98:0/1700985108 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa000faa0 con 0x7f1ab01f2260 2023-03-03T07:57:18.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.153+0000 7f1aa6ffd700 1 -- 172.21.15.98:0/1700985108 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa000fe30 con 0x7f1ab01f2260 2023-03-03T07:57:18.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.154+0000 7f1ab83ff700 1 -- 172.21.15.98:0/1700985108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab00d1fc0 con 0x7f1ab01f2260 2023-03-03T07:57:18.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.154+0000 7f1ab83ff700 1 -- 172.21.15.98:0/1700985108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab00b60d0 con 0x7f1ab01f2260 2023-03-03T07:57:18.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.155+0000 7f1ab83ff700 1 -- 172.21.15.98:0/1700985108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab015e410 con 0x7f1ab01f2260 2023-03-03T07:57:18.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.156+0000 7f1aa6ffd700 1 -- 172.21.15.98:0/1700985108 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1aa0014440 con 0x7f1ab01f2260 2023-03-03T07:57:18.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.157+0000 7f1aa6ffd700 1 --2- 172.21.15.98:0/1700985108 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1a940682f0 0x7f1a9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:18.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.157+0000 7f1aa6ffd700 1 -- 172.21.15.98:0/1700985108 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1aa00865c0 con 0x7f1ab01f2260 2023-03-03T07:57:18.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.158+0000 7f1ab619b700 1 --2- 172.21.15.98:0/1700985108 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1a940682f0 0x7f1a9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:18.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.158+0000 7f1ab619b700 1 --2- 172.21.15.98:0/1700985108 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1a940682f0 0x7f1a9406a7a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00b5d50 tx=0x7f1aac009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:18.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.161+0000 7f1aa6ffd700 1 -- 172.21.15.98:0/1700985108 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1aa0052d80 con 0x7f1ab01f2260 2023-03-03T07:57:18.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.358+0000 7f1ab83ff700 1 -- 172.21.15.98:0/1700985108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ab00729b0 con 0x7f1ab01f2260 2023-03-03T07:57:18.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.359+0000 7f1aa6ffd700 1 -- 172.21.15.98:0/1700985108 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+17979 (secure 0 0 0) 0x7f1aa00563a0 con 0x7f1ab01f2260 2023-03-03T07:57:18.361 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:18.361 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":38,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:57:18.294120+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:57:15.545608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-03-03T07:57:16.283131+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:57:18.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.362+0000 7f1ab83ff700 1 -- 172.21.15.98:0/1700985108 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1a940682f0 msgr2=0x7f1a9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:18.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.362+0000 7f1ab83ff700 1 --2- 172.21.15.98:0/1700985108 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1a940682f0 0x7f1a9406a7a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00b5d50 tx=0x7f1aac009040 comp rx=0 tx=0).stop 2023-03-03T07:57:18.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.362+0000 7f1ab83ff700 1 -- 172.21.15.98:0/1700985108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab01f2260 msgr2=0x7f1ab014efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.362+0000 7f1ab83ff700 1 --2- 172.21.15.98:0/1700985108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab01f2260 0x7f1ab014efb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00119a0 tx=0x7f1aa0011d60 comp rx=0 tx=0).stop 2023-03-03T07:57:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.362+0000 7f1ab83ff700 1 -- 172.21.15.98:0/1700985108 shutdown_connections 2023-03-03T07:57:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.362+0000 7f1ab83ff700 1 --2- 172.21.15.98:0/1700985108 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1a940682f0 0x7f1a9406a7a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.362+0000 7f1ab83ff700 1 --2- 172.21.15.98:0/1700985108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1ab0155be0 0x7f1ab0156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.362+0000 7f1ab83ff700 1 --2- 172.21.15.98:0/1700985108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a9c005200 0x7f1ab0154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.362+0000 7f1ab83ff700 1 --2- 172.21.15.98:0/1700985108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab01f2260 0x7f1ab014efb0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.362+0000 7f1ab83ff700 1 -- 172.21.15.98:0/1700985108 >> 172.21.15.98:0/1700985108 conn(0x7f1ab006aee0 msgr2=0x7f1ab015bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.362+0000 7f1ab83ff700 1 -- 172.21.15.98:0/1700985108 shutdown_connections 2023-03-03T07:57:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.362+0000 7f1ab83ff700 1 -- 172.21.15.98:0/1700985108 wait complete. 2023-03-03T07:57:18.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-03-03T07:57:18.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.638+0000 7fe0aed9d700 1 Processor -- start 2023-03-03T07:57:18.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.638+0000 7fe0aed9d700 1 -- start start 2023-03-03T07:57:18.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.640+0000 7fe0aed9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0a80cf570 0x7fe0a815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:18.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.640+0000 7fe0aed9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe0a815db30 0x7fe0a80b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:18.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.640+0000 7fe0aed9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a80cd780 con 0x7fe0a81609b0 2023-03-03T07:57:18.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.640+0000 7fe0aed9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a80cebb0 con 0x7fe0a815db30 2023-03-03T07:57:18.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.640+0000 7fe0aed9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a80cda60 con 0x7fe0a80cf570 2023-03-03T07:57:18.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.640+0000 7fe0ae59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe0a815db30 0x7fe0a80b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:18.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.641+0000 7fe0ad59a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0a80cf570 0x7fe0a815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:18.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.641+0000 7fe0ae59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe0a815db30 0x7fe0a80b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45506/0 (socket says 172.21.15.98:45506) 2023-03-03T07:57:18.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.641+0000 7fe0ae59c700 1 -- 172.21.15.98:0/3828835277 learned_addr learned my addr 172.21.15.98:0/3828835277 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:18.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.641+0000 7fe0add9b700 1 --1- 172.21.15.98:0/3828835277 >> v1:172.21.15.100:6789/0 conn(0x7fe0a81609b0 0x7fe0a8160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:45298/0 (socket says 172.21.15.98:45298) 2023-03-03T07:57:18.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.641+0000 7fe0ae59c700 1 -- 172.21.15.98:0/3828835277 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0a80cf570 msgr2=0x7fe0a815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:18.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.641+0000 7fe0ae59c700 1 --2- 172.21.15.98:0/3828835277 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0a80cf570 0x7fe0a815d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:18.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.641+0000 7fe0ae59c700 1 -- 172.21.15.98:0/3828835277 >> v1:172.21.15.100:6789/0 conn(0x7fe0a81609b0 legacy=0x7fe0a8160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:18.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.641+0000 7fe0ae59c700 1 -- 172.21.15.98:0/3828835277 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a80cdd40 con 0x7fe0a815db30 2023-03-03T07:57:18.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.641+0000 7fe0ae59c700 1 --2- 172.21.15.98:0/3828835277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0a815db30 0x7fe0a80b4aa0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0009e50 tx=0x7fe0a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a92f0c35fe18b5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:18.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.642+0000 7fe0acd99700 1 -- 172.21.15.98:0/3828835277 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a000e070 con 0x7fe0a815db30 2023-03-03T07:57:18.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.642+0000 7fe0acd99700 1 -- 172.21.15.98:0/3828835277 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a000b910 con 0x7fe0a815db30 2023-03-03T07:57:18.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.642+0000 7fe0acd99700 1 -- 172.21.15.98:0/3828835277 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a000bcc0 con 0x7fe0a815db30 2023-03-03T07:57:18.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.642+0000 7fe0aed9d700 1 -- 172.21.15.98:0/3828835277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0a815db30 msgr2=0x7fe0a80b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:18.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.642+0000 7fe0aed9d700 1 --2- 172.21.15.98:0/3828835277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0a815db30 0x7fe0a80b4aa0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0009e50 tx=0x7fe0a000b330 comp rx=0 tx=0).stop 2023-03-03T07:57:18.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.642+0000 7fe0aed9d700 1 -- 172.21.15.98:0/3828835277 shutdown_connections 2023-03-03T07:57:18.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.642+0000 7fe0aed9d700 1 --2- 172.21.15.98:0/3828835277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0a815db30 0x7fe0a80b4aa0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:18.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.642+0000 7fe0aed9d700 1 --2- 172.21.15.98:0/3828835277 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0a80cf570 0x7fe0a815d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:18.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.642+0000 7fe0aed9d700 1 -- 172.21.15.98:0/3828835277 >> 172.21.15.98:0/3828835277 conn(0x7fe0a806b880 msgr2=0x7fe0a80cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:18.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.642+0000 7fe0aed9d700 1 -- 172.21.15.98:0/3828835277 shutdown_connections 2023-03-03T07:57:18.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.642+0000 7fe0aed9d700 1 -- 172.21.15.98:0/3828835277 wait complete. 2023-03-03T07:57:18.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.644+0000 7fe0aed9d700 1 Processor -- start 2023-03-03T07:57:18.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.645+0000 7fe0aed9d700 1 -- start start 2023-03-03T07:57:18.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0aed9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0a80cf570 0x7fe0a80b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:18.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0aed9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0a815db30 0x7fe0a80c8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:18.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0aed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0a81609b0 0x7fe0a8151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:18.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0aed9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a80ca990 con 0x7fe0a81609b0 2023-03-03T07:57:18.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0aed9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a80b3900 con 0x7fe0a80cf570 2023-03-03T07:57:18.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0aed9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a80c8370 con 0x7fe0a815db30 2023-03-03T07:57:18.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0ae59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0a81609b0 0x7fe0a8151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:18.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0ae59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0a81609b0 0x7fe0a8151180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45514/0 (socket says 172.21.15.98:45514) 2023-03-03T07:57:18.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0ae59c700 1 -- 172.21.15.98:0/90278371 learned_addr learned my addr 172.21.15.98:0/90278371 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:18.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0add9b700 1 --2- 172.21.15.98:0/90278371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0a80cf570 0x7fe0a80b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:18.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0ad59a700 1 --2- 172.21.15.98:0/90278371 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0a815db30 0x7fe0a80c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:18.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0ae59c700 1 -- 172.21.15.98:0/90278371 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0a815db30 msgr2=0x7fe0a80c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:18.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0ae59c700 1 --2- 172.21.15.98:0/90278371 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0a815db30 0x7fe0a80c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:18.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0ae59c700 1 -- 172.21.15.98:0/90278371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0a80cf570 msgr2=0x7fe0a80b2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:18.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0ae59c700 1 --2- 172.21.15.98:0/90278371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0a80cf570 0x7fe0a80b2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:18.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.646+0000 7fe0ae59c700 1 -- 172.21.15.98:0/90278371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a0009b30 con 0x7fe0a81609b0 2023-03-03T07:57:18.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.647+0000 7fe0ae59c700 1 --2- 172.21.15.98:0/90278371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0a81609b0 0x7fe0a8151180 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8074380 tx=0x7fe0a0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:18.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.647+0000 7fe09e7fc700 1 -- 172.21.15.98:0/90278371 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a00099b0 con 0x7fe0a81609b0 2023-03-03T07:57:18.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.647+0000 7fe09e7fc700 1 -- 172.21.15.98:0/90278371 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a00092e0 con 0x7fe0a81609b0 2023-03-03T07:57:18.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.647+0000 7fe09e7fc700 1 -- 172.21.15.98:0/90278371 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a0004450 con 0x7fe0a81609b0 2023-03-03T07:57:18.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.647+0000 7fe0aed9d700 1 -- 172.21.15.98:0/90278371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a815d0d0 con 0x7fe0a81609b0 2023-03-03T07:57:18.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.647+0000 7fe0aed9d700 1 -- 172.21.15.98:0/90278371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a815cdf0 con 0x7fe0a81609b0 2023-03-03T07:57:18.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.648+0000 7fe0aed9d700 1 -- 172.21.15.98:0/90278371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a80b4fe0 con 0x7fe0a81609b0 2023-03-03T07:57:18.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.650+0000 7fe09e7fc700 1 -- 172.21.15.98:0/90278371 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe0a002c070 con 0x7fe0a81609b0 2023-03-03T07:57:18.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.651+0000 7fe09e7fc700 1 --2- 172.21.15.98:0/90278371 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe08c0682f0 0x7fe08c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:18.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.651+0000 7fe09e7fc700 1 -- 172.21.15.98:0/90278371 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe0a00873b0 con 0x7fe0a81609b0 2023-03-03T07:57:18.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.651+0000 7fe0add9b700 1 --2- 172.21.15.98:0/90278371 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe08c0682f0 0x7fe08c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:18.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.652+0000 7fe0add9b700 1 --2- 172.21.15.98:0/90278371 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe08c0682f0 0x7fe08c06a7a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4000f60 tx=0x7fe0a4007ed0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:18.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.655+0000 7fe09e7fc700 1 -- 172.21.15.98:0/90278371 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe0a0053aa0 con 0x7fe0a81609b0 2023-03-03T07:57:18.701 INFO:tasks.ceph.mds.c.smithi098.stderr:2023-03-03T07:57:18.700+0000 7fe6279f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T07:57:18.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:18.896+0000 7fe0aed9d700 1 -- 172.21.15.98:0/90278371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe0a8068290 con 0x7fe0a81609b0 2023-03-03T07:57:19.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:19.741+0000 7fe09e7fc700 1 -- 172.21.15.98:0/90278371 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v13) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe0a00570c0 con 0x7fe0a81609b0 2023-03-03T07:57:19.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:19.745+0000 7fe0aed9d700 1 -- 172.21.15.98:0/90278371 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe08c0682f0 msgr2=0x7fe08c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:19.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:19.745+0000 7fe0aed9d700 1 --2- 172.21.15.98:0/90278371 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe08c0682f0 0x7fe08c06a7a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4000f60 tx=0x7fe0a4007ed0 comp rx=0 tx=0).stop 2023-03-03T07:57:19.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:19.745+0000 7fe0aed9d700 1 -- 172.21.15.98:0/90278371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0a81609b0 msgr2=0x7fe0a8151180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:19.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:19.745+0000 7fe0aed9d700 1 --2- 172.21.15.98:0/90278371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0a81609b0 0x7fe0a8151180 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8074380 tx=0x7fe0a0004ad0 comp rx=0 tx=0).stop 2023-03-03T07:57:19.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:19.745+0000 7fe0aed9d700 1 -- 172.21.15.98:0/90278371 shutdown_connections 2023-03-03T07:57:19.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:19.745+0000 7fe0aed9d700 1 --2- 172.21.15.98:0/90278371 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe08c0682f0 0x7fe08c06a7a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:19.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:19.745+0000 7fe0aed9d700 1 --2- 172.21.15.98:0/90278371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0a81609b0 0x7fe0a8151180 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:19.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:19.745+0000 7fe0aed9d700 1 --2- 172.21.15.98:0/90278371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0a80cf570 0x7fe0a80b2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:19.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:19.745+0000 7fe0aed9d700 1 --2- 172.21.15.98:0/90278371 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0a815db30 0x7fe0a80c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:19.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:19.745+0000 7fe0aed9d700 1 -- 172.21.15.98:0/90278371 >> 172.21.15.98:0/90278371 conn(0x7fe0a806b880 msgr2=0x7fe0a815b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:19.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:19.745+0000 7fe0aed9d700 1 -- 172.21.15.98:0/90278371 shutdown_connections 2023-03-03T07:57:19.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:19.745+0000 7fe0aed9d700 1 -- 172.21.15.98:0/90278371 wait complete. 2023-03-03T07:57:19.777 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-03-03T07:57:19.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-03-03T07:57:20.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.029+0000 7ff4c6dff700 1 Processor -- start 2023-03-03T07:57:20.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.029+0000 7ff4c6dff700 1 -- start start 2023-03-03T07:57:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.031+0000 7ff4c6dff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff4c00d0270 0x7ff4c00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.031+0000 7ff4c6dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c00cf980 con 0x7ff4c006b4b0 2023-03-03T07:57:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.031+0000 7ff4c6dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c00d1800 con 0x7ff4c015ca70 2023-03-03T07:57:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.031+0000 7ff4c6dff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c00cd5b0 con 0x7ff4c00d0270 2023-03-03T07:57:20.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.031+0000 7ff4c539c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff4c006b4b0 0x7ff4c00d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38770/0 (socket says 172.21.15.98:38770) 2023-03-03T07:57:20.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.031+0000 7ff4c539c700 1 -- 172.21.15.98:0/900842459 learned_addr learned my addr 172.21.15.98:0/900842459 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:20.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.031+0000 7ff4bffff700 1 --2- 172.21.15.98:0/900842459 >> v2:172.21.15.100:3300/0 conn(0x7ff4c00d0270 0x7ff4c00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:20.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.032+0000 7ff4bf7fe700 1 -- 172.21.15.98:0/900842459 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552319596 0 0) 0x7ff4c00cf980 con 0x7ff4c006b4b0 2023-03-03T07:57:20.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.032+0000 7ff4bf7fe700 1 -- 172.21.15.98:0/900842459 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b00035b0 con 0x7ff4c006b4b0 2023-03-03T07:57:20.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.032+0000 7ff4bf7fe700 1 -- 172.21.15.98:0/900842459 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619858965 0 0) 0x7ff4c00d1800 con 0x7ff4c015ca70 2023-03-03T07:57:20.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.032+0000 7ff4bf7fe700 1 -- 172.21.15.98:0/900842459 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c00cf980 con 0x7ff4c015ca70 2023-03-03T07:57:20.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.032+0000 7ff4bffff700 1 -- 172.21.15.98:0/900842459 >> v1:172.21.15.100:6789/0 conn(0x7ff4c015ca70 legacy=0x7ff4c00cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:20.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.032+0000 7ff4bffff700 1 -- 172.21.15.98:0/900842459 >> v1:172.21.15.98:6789/0 conn(0x7ff4c006b4b0 legacy=0x7ff4c00d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:20.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.032+0000 7ff4bffff700 1 -- 172.21.15.98:0/900842459 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c00cd900 con 0x7ff4c00d0270 2023-03-03T07:57:20.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.032+0000 7ff4bffff700 1 --2- 172.21.15.98:0/900842459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4c00d0270 0x7ff4c00cf440 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009840 tx=0x7ff4ac009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d912f5a773332812 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:20.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.032+0000 7ff4bf7fe700 1 -- 172.21.15.98:0/900842459 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac011830 con 0x7ff4c00d0270 2023-03-03T07:57:20.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.033+0000 7ff4bf7fe700 1 -- 172.21.15.98:0/900842459 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4ac011990 con 0x7ff4c00d0270 2023-03-03T07:57:20.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.033+0000 7ff4bf7fe700 1 -- 172.21.15.98:0/900842459 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac011d20 con 0x7ff4c00d0270 2023-03-03T07:57:20.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.033+0000 7ff4c6dff700 1 -- 172.21.15.98:0/900842459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4c00d0270 msgr2=0x7ff4c00cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:20.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.033+0000 7ff4c6dff700 1 --2- 172.21.15.98:0/900842459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4c00d0270 0x7ff4c00cf440 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009840 tx=0x7ff4ac009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:20.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.033+0000 7ff4c6dff700 1 -- 172.21.15.98:0/900842459 shutdown_connections 2023-03-03T07:57:20.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.033+0000 7ff4c6dff700 1 --2- 172.21.15.98:0/900842459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4c00d0270 0x7ff4c00cf440 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:20.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.033+0000 7ff4c6dff700 1 -- 172.21.15.98:0/900842459 >> 172.21.15.98:0/900842459 conn(0x7ff4c0073e70 msgr2=0x7ff4c006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:20.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.033+0000 7ff4c6dff700 1 -- 172.21.15.98:0/900842459 shutdown_connections 2023-03-03T07:57:20.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.033+0000 7ff4c6dff700 1 -- 172.21.15.98:0/900842459 wait complete. 2023-03-03T07:57:20.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.036+0000 7ff4c6dff700 1 Processor -- start 2023-03-03T07:57:20.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.036+0000 7ff4c6dff700 1 -- start start 2023-03-03T07:57:20.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.038+0000 7ff4c6dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c006b4b0 0x7ff4c0153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:20.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.038+0000 7ff4c6dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4c00d0270 0x7ff4c00c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:20.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.038+0000 7ff4c6dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4c015ca70 0x7ff4c00b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:20.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.038+0000 7ff4c6dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c01f7680 con 0x7ff4c006b4b0 2023-03-03T07:57:20.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.038+0000 7ff4c6dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00c65f0 con 0x7ff4c00d0270 2023-03-03T07:57:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.038+0000 7ff4c6dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c01f6a70 con 0x7ff4c015ca70 2023-03-03T07:57:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.038+0000 7ff4c4b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c006b4b0 0x7ff4c0153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.038+0000 7ff4c4b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c006b4b0 0x7ff4c0153ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45526/0 (socket says 172.21.15.98:45526) 2023-03-03T07:57:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.038+0000 7ff4c4b9b700 1 -- 172.21.15.98:0/325247579 learned_addr learned my addr 172.21.15.98:0/325247579 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.038+0000 7ff4bffff700 1 --2- 172.21.15.98:0/325247579 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4c00d0270 0x7ff4c00c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.038+0000 7ff4c539c700 1 --2- 172.21.15.98:0/325247579 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4c015ca70 0x7ff4c00b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.039+0000 7ff4c4b9b700 1 -- 172.21.15.98:0/325247579 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4c015ca70 msgr2=0x7ff4c00b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.039+0000 7ff4c4b9b700 1 --2- 172.21.15.98:0/325247579 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4c015ca70 0x7ff4c00b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.039+0000 7ff4c4b9b700 1 -- 172.21.15.98:0/325247579 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4c00d0270 msgr2=0x7ff4c00c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.039+0000 7ff4c4b9b700 1 --2- 172.21.15.98:0/325247579 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4c00d0270 0x7ff4c00c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.039+0000 7ff4c4b9b700 1 -- 172.21.15.98:0/325247579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac0094a0 con 0x7ff4c006b4b0 2023-03-03T07:57:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.039+0000 7ff4c539c700 1 --2- 172.21.15.98:0/325247579 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4c015ca70 0x7ff4c00b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:20.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.039+0000 7ff4c4b9b700 1 --2- 172.21.15.98:0/325247579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c006b4b0 0x7ff4c0153ac0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400cc50 tx=0x7ff4b400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:20.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.039+0000 7ff4bd7fa700 1 -- 172.21.15.98:0/325247579 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4003c00 con 0x7ff4c006b4b0 2023-03-03T07:57:20.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.039+0000 7ff4bd7fa700 1 -- 172.21.15.98:0/325247579 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b4003d60 con 0x7ff4c006b4b0 2023-03-03T07:57:20.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.039+0000 7ff4bd7fa700 1 -- 172.21.15.98:0/325247579 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b400abc0 con 0x7ff4c006b4b0 2023-03-03T07:57:20.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.039+0000 7ff4c6dff700 1 -- 172.21.15.98:0/325247579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c00d1aa0 con 0x7ff4c006b4b0 2023-03-03T07:57:20.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.039+0000 7ff4c6dff700 1 -- 172.21.15.98:0/325247579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c00cdbe0 con 0x7ff4c006b4b0 2023-03-03T07:57:20.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.040+0000 7ff4c6dff700 1 -- 172.21.15.98:0/325247579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c015ce40 con 0x7ff4c006b4b0 2023-03-03T07:57:20.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.042+0000 7ff4bd7fa700 1 -- 172.21.15.98:0/325247579 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff4b4010050 con 0x7ff4c006b4b0 2023-03-03T07:57:20.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.043+0000 7ff4bd7fa700 1 --2- 172.21.15.98:0/325247579 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4a40682f0 0x7ff4a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:20.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.043+0000 7ff4bd7fa700 1 -- 172.21.15.98:0/325247579 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff4b4085ce0 con 0x7ff4c006b4b0 2023-03-03T07:57:20.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.043+0000 7ff4bffff700 1 --2- 172.21.15.98:0/325247579 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4a40682f0 0x7ff4a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:20.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.044+0000 7ff4bffff700 1 --2- 172.21.15.98:0/325247579 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4a40682f0 0x7ff4a406a7a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009840 tx=0x7ff4ac01c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:20.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.047+0000 7ff4bd7fa700 1 -- 172.21.15.98:0/325247579 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff4b4052450 con 0x7ff4c006b4b0 2023-03-03T07:57:20.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.284+0000 7ff4c6dff700 1 -- 172.21.15.98:0/325247579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff4c004e6e0 con 0x7ff4c006b4b0 2023-03-03T07:57:20.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.744+0000 7ff4bd7fa700 1 -- 172.21.15.98:0/325247579 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff4b4055a70 con 0x7ff4c006b4b0 2023-03-03T07:57:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.747+0000 7ff4c6dff700 1 -- 172.21.15.98:0/325247579 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4a40682f0 msgr2=0x7ff4a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.747+0000 7ff4c6dff700 1 --2- 172.21.15.98:0/325247579 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4a40682f0 0x7ff4a406a7a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009840 tx=0x7ff4ac01c040 comp rx=0 tx=0).stop 2023-03-03T07:57:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.747+0000 7ff4c6dff700 1 -- 172.21.15.98:0/325247579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c006b4b0 msgr2=0x7ff4c0153ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.747+0000 7ff4c6dff700 1 --2- 172.21.15.98:0/325247579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c006b4b0 0x7ff4c0153ac0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400cc50 tx=0x7ff4b400cf60 comp rx=0 tx=0).stop 2023-03-03T07:57:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.747+0000 7ff4c6dff700 1 -- 172.21.15.98:0/325247579 shutdown_connections 2023-03-03T07:57:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.747+0000 7ff4c6dff700 1 --2- 172.21.15.98:0/325247579 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4a40682f0 0x7ff4a406a7a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.747+0000 7ff4c6dff700 1 --2- 172.21.15.98:0/325247579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c006b4b0 0x7ff4c0153ac0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.747+0000 7ff4c6dff700 1 --2- 172.21.15.98:0/325247579 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4c015ca70 0x7ff4c00b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.747+0000 7ff4c6dff700 1 --2- 172.21.15.98:0/325247579 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4c00d0270 0x7ff4c00c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.747+0000 7ff4c6dff700 1 -- 172.21.15.98:0/325247579 >> 172.21.15.98:0/325247579 conn(0x7ff4c0073e70 msgr2=0x7ff4c01591e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.747+0000 7ff4c6dff700 1 -- 172.21.15.98:0/325247579 shutdown_connections 2023-03-03T07:57:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:20.747+0000 7ff4c6dff700 1 -- 172.21.15.98:0/325247579 wait complete. 2023-03-03T07:57:20.776 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-03-03T07:57:20.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:57:21.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.022+0000 7f53a739d700 1 Processor -- start 2023-03-03T07:57:21.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.022+0000 7f53a739d700 1 -- start start 2023-03-03T07:57:21.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.023+0000 7f53a739d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f53a0146360 0x7f53a0160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:21.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.023+0000 7f53a739d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53a00d2590 0x7f53a00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:21.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.023+0000 7f53a739d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a01614c0 con 0x7f53a00b4ab0 2023-03-03T07:57:21.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.023+0000 7f53a739d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a00b5440 con 0x7f53a0146360 2023-03-03T07:57:21.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.023+0000 7f53a739d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a00d10f0 con 0x7f53a00d2590 2023-03-03T07:57:21.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.024+0000 7f53a639b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f53a0146360 0x7f53a0160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:21.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.024+0000 7f53a639b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f53a0146360 0x7f53a0160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45530/0 (socket says 172.21.15.98:45530) 2023-03-03T07:57:21.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.024+0000 7f53a639b700 1 -- 172.21.15.98:0/3374142774 learned_addr learned my addr 172.21.15.98:0/3374142774 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:21.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.024+0000 7f53a6b9c700 1 --2- 172.21.15.98:0/3374142774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53a00d2590 0x7f53a00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:21.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.024+0000 7f53a639b700 1 -- 172.21.15.98:0/3374142774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53a00d2590 msgr2=0x7f53a00b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:21.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.024+0000 7f53a639b700 1 --2- 172.21.15.98:0/3374142774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53a00d2590 0x7f53a00b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.024+0000 7f53a639b700 1 -- 172.21.15.98:0/3374142774 >> v1:172.21.15.100:6789/0 conn(0x7f53a00b4ab0 legacy=0x7f53a00731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:21.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.024+0000 7f53a639b700 1 -- 172.21.15.98:0/3374142774 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a00cdf20 con 0x7f53a0146360 2023-03-03T07:57:21.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.024+0000 7f53a6b9c700 1 --2- 172.21.15.98:0/3374142774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53a00d2590 0x7f53a00b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:21.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.025+0000 7f53a639b700 1 --2- 172.21.15.98:0/3374142774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a0146360 0x7f53a0160f80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f539c0098f0 tx=0x7f539c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f70953e32eb8810 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:21.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.025+0000 7f53a5399700 1 -- 172.21.15.98:0/3374142774 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c00b8a0 con 0x7f53a0146360 2023-03-03T07:57:21.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.025+0000 7f53a5399700 1 -- 172.21.15.98:0/3374142774 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539c00ba00 con 0x7f53a0146360 2023-03-03T07:57:21.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.025+0000 7f53a5399700 1 -- 172.21.15.98:0/3374142774 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c00bd90 con 0x7f53a0146360 2023-03-03T07:57:21.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.026+0000 7f53a739d700 1 -- 172.21.15.98:0/3374142774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a0146360 msgr2=0x7f53a0160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:21.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.026+0000 7f53a739d700 1 --2- 172.21.15.98:0/3374142774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a0146360 0x7f53a0160f80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f539c0098f0 tx=0x7f539c009c00 comp rx=0 tx=0).stop 2023-03-03T07:57:21.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.026+0000 7f53a739d700 1 -- 172.21.15.98:0/3374142774 shutdown_connections 2023-03-03T07:57:21.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.026+0000 7f53a739d700 1 --2- 172.21.15.98:0/3374142774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a0146360 0x7f53a0160f80 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.026+0000 7f53a739d700 1 --2- 172.21.15.98:0/3374142774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53a00d2590 0x7f53a00b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.026+0000 7f53a739d700 1 -- 172.21.15.98:0/3374142774 >> 172.21.15.98:0/3374142774 conn(0x7f53a006b8e0 msgr2=0x7f53a00cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.026+0000 7f53a739d700 1 -- 172.21.15.98:0/3374142774 shutdown_connections 2023-03-03T07:57:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.026+0000 7f53a739d700 1 -- 172.21.15.98:0/3374142774 wait complete. 2023-03-03T07:57:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.029+0000 7f53a739d700 1 Processor -- start 2023-03-03T07:57:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.029+0000 7f53a739d700 1 -- start start 2023-03-03T07:57:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.030+0000 7f53a739d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53a00b4ab0 0x7f53a00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.030+0000 7f53a739d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a00d2590 0x7f53a0155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.030+0000 7f53a739d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53a0146360 0x7f53a0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:21.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.030+0000 7f53a739d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a00b7980 con 0x7f53a00d2590 2023-03-03T07:57:21.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.030+0000 7f53a739d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a00b7e10 con 0x7f53a0146360 2023-03-03T07:57:21.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.030+0000 7f53a739d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a00b6d70 con 0x7f53a00b4ab0 2023-03-03T07:57:21.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.031+0000 7f53a5b9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a00d2590 0x7f53a0155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.031+0000 7f53a5b9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a00d2590 0x7f53a0155750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45544/0 (socket says 172.21.15.98:45544) 2023-03-03T07:57:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.031+0000 7f53a5b9a700 1 -- 172.21.15.98:0/1842308826 learned_addr learned my addr 172.21.15.98:0/1842308826 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.031+0000 7f53a6b9c700 1 --2- 172.21.15.98:0/1842308826 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53a0146360 0x7f53a0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.031+0000 7f53a639b700 1 --2- 172.21.15.98:0/1842308826 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53a00b4ab0 0x7f53a00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.031+0000 7f53a5b9a700 1 -- 172.21.15.98:0/1842308826 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53a00b4ab0 msgr2=0x7f53a00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.031+0000 7f53a5b9a700 1 --2- 172.21.15.98:0/1842308826 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53a00b4ab0 0x7f53a00c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.031+0000 7f53a5b9a700 1 -- 172.21.15.98:0/1842308826 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53a0146360 msgr2=0x7f53a0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.031+0000 7f53a5b9a700 1 --2- 172.21.15.98:0/1842308826 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53a0146360 0x7f53a0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.031+0000 7f53a5b9a700 1 -- 172.21.15.98:0/1842308826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539c009550 con 0x7f53a00d2590 2023-03-03T07:57:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.031+0000 7f53a639b700 1 --2- 172.21.15.98:0/1842308826 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53a00b4ab0 0x7f53a00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.032+0000 7f53a5b9a700 1 --2- 172.21.15.98:0/1842308826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a00d2590 0x7f53a0155750 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f539000c430 tx=0x7f539000c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.032+0000 7f5396ffd700 1 -- 172.21.15.98:0/1842308826 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53900052d0 con 0x7f53a00d2590 2023-03-03T07:57:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.032+0000 7f5396ffd700 1 -- 172.21.15.98:0/1842308826 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5390005450 con 0x7f53a00d2590 2023-03-03T07:57:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.032+0000 7f5396ffd700 1 -- 172.21.15.98:0/1842308826 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5390006aa0 con 0x7f53a00d2590 2023-03-03T07:57:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.032+0000 7f53a739d700 1 -- 172.21.15.98:0/1842308826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a00cd960 con 0x7f53a00d2590 2023-03-03T07:57:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.032+0000 7f53a739d700 1 -- 172.21.15.98:0/1842308826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a00cd680 con 0x7f53a00d2590 2023-03-03T07:57:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.033+0000 7f53a739d700 1 -- 172.21.15.98:0/1842308826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a00cdc40 con 0x7f53a00d2590 2023-03-03T07:57:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.035+0000 7f5396ffd700 1 -- 172.21.15.98:0/1842308826 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5390010050 con 0x7f53a00d2590 2023-03-03T07:57:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.036+0000 7f5396ffd700 1 --2- 172.21.15.98:0/1842308826 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f53840682f0 0x7f538406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.036+0000 7f5396ffd700 1 -- 172.21.15.98:0/1842308826 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5390084c40 con 0x7f53a00d2590 2023-03-03T07:57:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.036+0000 7f53a639b700 1 --2- 172.21.15.98:0/1842308826 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f53840682f0 0x7f538406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.037+0000 7f53a639b700 1 --2- 172.21.15.98:0/1842308826 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f53840682f0 0x7f538406a7a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f539c014040 tx=0x7f539c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.040+0000 7f5396ffd700 1 -- 172.21.15.98:0/1842308826 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f539004a5e0 con 0x7f53a00d2590 2023-03-03T07:57:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.275+0000 7f53a739d700 1 -- 172.21.15.98:0/1842308826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53a01614c0 con 0x7f53a00d2590 2023-03-03T07:57:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.276+0000 7f5396ffd700 1 -- 172.21.15.98:0/1842308826 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f53a01614c0 con 0x7f53a00d2590 2023-03-03T07:57:21.278 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:21.278 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3849166495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3849166495},{"type":"v1","addr":"172.21.15.100:6834","nonce":3849166495}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/4285491249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":4285491249},{"type":"v1","addr":"172.21.15.100:6835","nonce":4285491249}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":5208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3888598890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3888598890},{"type":"v1","addr":"172.21.15.98:6836","nonce":3888598890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:57:17.291546+0000","modified":"2023-03-03T07:57:20.741013+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5227},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5227":{"gid":5227,"name":"c","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.98:6837/1163251191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1163251191},{"type":"v1","addr":"172.21.15.98:6837","nonce":1163251191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T07:57:21.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.278+0000 7f53a739d700 1 -- 172.21.15.98:0/1842308826 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f53840682f0 msgr2=0x7f538406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:21.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.278+0000 7f53a739d700 1 --2- 172.21.15.98:0/1842308826 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f53840682f0 0x7f538406a7a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f539c014040 tx=0x7f539c008030 comp rx=0 tx=0).stop 2023-03-03T07:57:21.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.278+0000 7f53a739d700 1 -- 172.21.15.98:0/1842308826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a00d2590 msgr2=0x7f53a0155750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:21.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.278+0000 7f53a739d700 1 --2- 172.21.15.98:0/1842308826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a00d2590 0x7f53a0155750 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f539000c430 tx=0x7f539000c740 comp rx=0 tx=0).stop 2023-03-03T07:57:21.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.278+0000 7f53a739d700 1 -- 172.21.15.98:0/1842308826 shutdown_connections 2023-03-03T07:57:21.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.278+0000 7f53a739d700 1 --2- 172.21.15.98:0/1842308826 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f53840682f0 0x7f538406a7a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.278+0000 7f53a739d700 1 --2- 172.21.15.98:0/1842308826 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53a0146360 0x7f53a0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.278+0000 7f53a739d700 1 --2- 172.21.15.98:0/1842308826 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53a00b4ab0 0x7f53a00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.278+0000 7f53a739d700 1 --2- 172.21.15.98:0/1842308826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a00d2590 0x7f53a0155750 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.278+0000 7f53a739d700 1 -- 172.21.15.98:0/1842308826 >> 172.21.15.98:0/1842308826 conn(0x7f53a006b8e0 msgr2=0x7f53a00d1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:21.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.279+0000 7f53a739d700 1 -- 172.21.15.98:0/1842308826 shutdown_connections 2023-03-03T07:57:21.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.279+0000 7f53a739d700 1 -- 172.21.15.98:0/1842308826 wait complete. 2023-03-03T07:57:21.284 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 14 2023-03-03T07:57:21.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:57:21.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.553+0000 7f39987ff700 1 Processor -- start 2023-03-03T07:57:21.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.553+0000 7f39987ff700 1 -- start start 2023-03-03T07:57:21.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.555+0000 7f39987ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f39900b8b90 0x7f39900cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:21.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.555+0000 7f39987ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f39900b5410 0x7f39900d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:21.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.555+0000 7f39987ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39900d0ef0 con 0x7f39900b87c0 2023-03-03T07:57:21.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.555+0000 7f39987ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39900d0600 con 0x7f39900b8b90 2023-03-03T07:57:21.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.555+0000 7f39987ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39900cd970 con 0x7f39900b5410 2023-03-03T07:57:21.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.555+0000 7f399659b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f39900b87c0 0x7f39900b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38784/0 (socket says 172.21.15.98:38784) 2023-03-03T07:57:21.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.555+0000 7f399659b700 1 -- 172.21.15.98:0/1690549856 learned_addr learned my addr 172.21.15.98:0/1690549856 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:21.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.555+0000 7f3995d9a700 1 --2- 172.21.15.98:0/1690549856 >> v2:172.21.15.98:3300/0 conn(0x7f39900b8b90 0x7f39900cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:21.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.555+0000 7f3996d9c700 1 --2- 172.21.15.98:0/1690549856 >> v2:172.21.15.100:3300/0 conn(0x7f39900b5410 0x7f39900d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:21.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.555+0000 7f3995599700 1 -- 172.21.15.98:0/1690549856 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62069125 0 0) 0x7f39900d0ef0 con 0x7f39900b87c0 2023-03-03T07:57:21.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.556+0000 7f3995599700 1 -- 172.21.15.98:0/1690549856 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f397c0035b0 con 0x7f39900b87c0 2023-03-03T07:57:21.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.556+0000 7f3995d9a700 1 -- 172.21.15.98:0/1690549856 >> v2:172.21.15.100:3300/0 conn(0x7f39900b5410 msgr2=0x7f39900d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:21.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.556+0000 7f3995d9a700 1 --2- 172.21.15.98:0/1690549856 >> v2:172.21.15.100:3300/0 conn(0x7f39900b5410 0x7f39900d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.556+0000 7f3995d9a700 1 -- 172.21.15.98:0/1690549856 >> v1:172.21.15.98:6789/0 conn(0x7f39900b87c0 legacy=0x7f39900b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:21.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.556+0000 7f3995d9a700 1 -- 172.21.15.98:0/1690549856 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39900cdc50 con 0x7f39900b8b90 2023-03-03T07:57:21.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.556+0000 7f3995d9a700 1 --2- 172.21.15.98:0/1690549856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39900b8b90 0x7f39900cffe0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f39900c6d50 tx=0x7f3980009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab02e8f5ec454ca server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:21.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.556+0000 7f3995599700 1 -- 172.21.15.98:0/1690549856 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980011830 con 0x7f39900b8b90 2023-03-03T07:57:21.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.556+0000 7f3995599700 1 -- 172.21.15.98:0/1690549856 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3980011990 con 0x7f39900b8b90 2023-03-03T07:57:21.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.556+0000 7f3995599700 1 -- 172.21.15.98:0/1690549856 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980011d20 con 0x7f39900b8b90 2023-03-03T07:57:21.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.557+0000 7f39987ff700 1 -- 172.21.15.98:0/1690549856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39900b8b90 msgr2=0x7f39900cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:21.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.557+0000 7f39987ff700 1 --2- 172.21.15.98:0/1690549856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39900b8b90 0x7f39900cffe0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f39900c6d50 tx=0x7f3980009b20 comp rx=0 tx=0).stop 2023-03-03T07:57:21.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.557+0000 7f39987ff700 1 -- 172.21.15.98:0/1690549856 shutdown_connections 2023-03-03T07:57:21.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.557+0000 7f39987ff700 1 --2- 172.21.15.98:0/1690549856 >> v2:172.21.15.100:3300/0 conn(0x7f39900b5410 0x7f39900d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.557+0000 7f39987ff700 1 --2- 172.21.15.98:0/1690549856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39900b8b90 0x7f39900cffe0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.557+0000 7f39987ff700 1 -- 172.21.15.98:0/1690549856 >> 172.21.15.98:0/1690549856 conn(0x7f399006aee0 msgr2=0x7f399006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:21.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.557+0000 7f39987ff700 1 -- 172.21.15.98:0/1690549856 shutdown_connections 2023-03-03T07:57:21.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.557+0000 7f39987ff700 1 -- 172.21.15.98:0/1690549856 wait complete. 2023-03-03T07:57:21.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.559+0000 7f39987ff700 1 Processor -- start 2023-03-03T07:57:21.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.559+0000 7f39987ff700 1 -- start start 2023-03-03T07:57:21.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.560+0000 7f39987ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39900b5410 0x7f3990149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:21.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.561+0000 7f39987ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f39900b87c0 0x7f39901f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:21.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.561+0000 7f39987ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39900b8b90 0x7f399015da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:21.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.561+0000 7f39987ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39900ca3e0 con 0x7f39900b8b90 2023-03-03T07:57:21.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.561+0000 7f39987ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399015b8b0 con 0x7f39900b87c0 2023-03-03T07:57:21.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.561+0000 7f39987ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39901f3100 con 0x7f39900b5410 2023-03-03T07:57:21.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.561+0000 7f3996d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39900b8b90 0x7f399015da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:21.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.561+0000 7f3996d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39900b8b90 0x7f399015da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45550/0 (socket says 172.21.15.98:45550) 2023-03-03T07:57:21.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.561+0000 7f3996d9c700 1 -- 172.21.15.98:0/2926657420 learned_addr learned my addr 172.21.15.98:0/2926657420 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:21.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.561+0000 7f3995d9a700 1 --2- 172.21.15.98:0/2926657420 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f39900b87c0 0x7f39901f2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:21.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.561+0000 7f399659b700 1 --2- 172.21.15.98:0/2926657420 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39900b5410 0x7f3990149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:21.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.561+0000 7f3996d9c700 1 -- 172.21.15.98:0/2926657420 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39900b5410 msgr2=0x7f3990149f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:21.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.561+0000 7f3996d9c700 1 --2- 172.21.15.98:0/2926657420 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39900b5410 0x7f3990149f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.561+0000 7f3996d9c700 1 -- 172.21.15.98:0/2926657420 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f39900b87c0 msgr2=0x7f39901f2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:21.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.561+0000 7f3996d9c700 1 --2- 172.21.15.98:0/2926657420 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f39900b87c0 0x7f39901f2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.561+0000 7f3996d9c700 1 -- 172.21.15.98:0/2926657420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39800094a0 con 0x7f39900b8b90 2023-03-03T07:57:21.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.562+0000 7f3996d9c700 1 --2- 172.21.15.98:0/2926657420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39900b8b90 0x7f399015da70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3988007e40 tx=0x7f398800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:21.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.562+0000 7f3986ffd700 1 -- 172.21.15.98:0/2926657420 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398800ecd0 con 0x7f39900b8b90 2023-03-03T07:57:21.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.562+0000 7f3986ffd700 1 -- 172.21.15.98:0/2926657420 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398800ee50 con 0x7f39900b8b90 2023-03-03T07:57:21.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.562+0000 7f39987ff700 1 -- 172.21.15.98:0/2926657420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39900cdf30 con 0x7f39900b8b90 2023-03-03T07:57:21.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.562+0000 7f3986ffd700 1 -- 172.21.15.98:0/2926657420 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988014660 con 0x7f39900b8b90 2023-03-03T07:57:21.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.562+0000 7f39987ff700 1 -- 172.21.15.98:0/2926657420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3990073a80 con 0x7f39900b8b90 2023-03-03T07:57:21.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.563+0000 7f39987ff700 1 -- 172.21.15.98:0/2926657420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39900ce0c0 con 0x7f39900b8b90 2023-03-03T07:57:21.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.565+0000 7f3986ffd700 1 -- 172.21.15.98:0/2926657420 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3988012050 con 0x7f39900b8b90 2023-03-03T07:57:21.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.567+0000 7f3986ffd700 1 --2- 172.21.15.98:0/2926657420 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39740682f0 0x7f397406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:21.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.567+0000 7f3986ffd700 1 -- 172.21.15.98:0/2926657420 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3988086380 con 0x7f39900b8b90 2023-03-03T07:57:21.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.567+0000 7f399659b700 1 --2- 172.21.15.98:0/2926657420 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39740682f0 0x7f397406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:21.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.568+0000 7f399659b700 1 --2- 172.21.15.98:0/2926657420 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39740682f0 0x7f397406a7a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f398c007fd0 tx=0x7f398c007dc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:21.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.571+0000 7f3986ffd700 1 -- 172.21.15.98:0/2926657420 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3988052a70 con 0x7f39900b8b90 2023-03-03T07:57:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.764+0000 7f39987ff700 1 -- 172.21.15.98:0/2926657420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39900729b0 con 0x7f39900b8b90 2023-03-03T07:57:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.766+0000 7f3986ffd700 1 -- 172.21.15.98:0/2926657420 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+17979 (secure 0 0 0) 0x7f3988056090 con 0x7f39900b8b90 2023-03-03T07:57:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:21.768 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":38,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:57:18.294120+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:57:15.545608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-03-03T07:57:16.283131+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:57:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.768+0000 7f39987ff700 1 -- 172.21.15.98:0/2926657420 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39740682f0 msgr2=0x7f397406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.768+0000 7f39987ff700 1 --2- 172.21.15.98:0/2926657420 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39740682f0 0x7f397406a7a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f398c007fd0 tx=0x7f398c007dc0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.768+0000 7f39987ff700 1 -- 172.21.15.98:0/2926657420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39900b8b90 msgr2=0x7f399015da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.768+0000 7f39987ff700 1 --2- 172.21.15.98:0/2926657420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39900b8b90 0x7f399015da70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3988007e40 tx=0x7f398800e3c0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.769+0000 7f39987ff700 1 -- 172.21.15.98:0/2926657420 shutdown_connections 2023-03-03T07:57:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.769+0000 7f39987ff700 1 --2- 172.21.15.98:0/2926657420 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39740682f0 0x7f397406a7a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.769+0000 7f39987ff700 1 --2- 172.21.15.98:0/2926657420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39900b8b90 0x7f399015da70 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.769+0000 7f39987ff700 1 --2- 172.21.15.98:0/2926657420 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39900b5410 0x7f3990149f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.769+0000 7f39987ff700 1 --2- 172.21.15.98:0/2926657420 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f39900b87c0 0x7f39901f2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.769+0000 7f39987ff700 1 -- 172.21.15.98:0/2926657420 >> 172.21.15.98:0/2926657420 conn(0x7f399006aee0 msgr2=0x7f39900b2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.769+0000 7f39987ff700 1 -- 172.21.15.98:0/2926657420 shutdown_connections 2023-03-03T07:57:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:21.769+0000 7f39987ff700 1 -- 172.21.15.98:0/2926657420 wait complete. 2023-03-03T07:57:21.797 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-03-03T07:57:21.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T07:57:22.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.042+0000 7fa1c15ff700 1 Processor -- start 2023-03-03T07:57:22.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.042+0000 7fa1c15ff700 1 -- start start 2023-03-03T07:57:22.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.043+0000 7fa1c15ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa1bc0b87c0 0x7fa1bc0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.043+0000 7fa1c15ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1bc0b5410 0x7fa1bc0d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.044+0000 7fa1c15ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc0d0ef0 con 0x7fa1bc0b8b90 2023-03-03T07:57:22.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.044+0000 7fa1c15ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1bc0d0600 con 0x7fa1bc0b87c0 2023-03-03T07:57:22.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.044+0000 7fa1c15ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc0cd970 con 0x7fa1bc0b5410 2023-03-03T07:57:22.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.044+0000 7fa1ba7fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa1bc0b8b90 0x7fa1bc0b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38796/0 (socket says 172.21.15.98:38796) 2023-03-03T07:57:22.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.044+0000 7fa1ba7fc700 1 -- 172.21.15.98:0/2141652366 learned_addr learned my addr 172.21.15.98:0/2141652366 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:22.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.044+0000 7fa1baffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa1bc0b87c0 0x7fa1bc0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.044+0000 7fa1baffd700 1 -- 172.21.15.98:0/2141652366 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1bc0b5410 msgr2=0x7fa1bc0d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:22.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.044+0000 7fa1bb7fe700 1 --2- 172.21.15.98:0/2141652366 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1bc0b5410 0x7fa1bc0d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.044+0000 7fa1baffd700 1 --2- 172.21.15.98:0/2141652366 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1bc0b5410 0x7fa1bc0d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.044+0000 7fa1baffd700 1 -- 172.21.15.98:0/2141652366 >> v1:172.21.15.98:6789/0 conn(0x7fa1bc0b8b90 legacy=0x7fa1bc0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.044+0000 7fa1baffd700 1 -- 172.21.15.98:0/2141652366 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc0cdc50 con 0x7fa1bc0b87c0 2023-03-03T07:57:22.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.044+0000 7fa1b9ffb700 1 -- 172.21.15.98:0/2141652366 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143816243 0 0) 0x7fa1bc0d0ef0 con 0x7fa1bc0b8b90 2023-03-03T07:57:22.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.045+0000 7fa1bb7fe700 1 --2- 172.21.15.98:0/2141652366 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1bc0b5410 0x7fa1bc0d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:57:22.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.045+0000 7fa1baffd700 1 --2- 172.21.15.98:0/2141652366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1bc0b87c0 0x7fa1bc0cffe0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac009840 tx=0x7fa1ac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d0fa9f48aad952 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:22.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.045+0000 7fa1b9ffb700 1 -- 172.21.15.98:0/2141652366 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac011830 con 0x7fa1bc0b87c0 2023-03-03T07:57:22.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.045+0000 7fa1b9ffb700 1 -- 172.21.15.98:0/2141652366 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1ac011990 con 0x7fa1bc0b87c0 2023-03-03T07:57:22.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.045+0000 7fa1b9ffb700 1 -- 172.21.15.98:0/2141652366 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac011d20 con 0x7fa1bc0b87c0 2023-03-03T07:57:22.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.046+0000 7fa1c15ff700 1 -- 172.21.15.98:0/2141652366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1bc0b87c0 msgr2=0x7fa1bc0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.046+0000 7fa1c15ff700 1 --2- 172.21.15.98:0/2141652366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1bc0b87c0 0x7fa1bc0cffe0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac009840 tx=0x7fa1ac009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:22.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.046+0000 7fa1c15ff700 1 -- 172.21.15.98:0/2141652366 shutdown_connections 2023-03-03T07:57:22.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.046+0000 7fa1c15ff700 1 --2- 172.21.15.98:0/2141652366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1bc0b87c0 0x7fa1bc0cffe0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.046+0000 7fa1c15ff700 1 --2- 172.21.15.98:0/2141652366 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1bc0b5410 0x7fa1bc0d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.046+0000 7fa1c15ff700 1 -- 172.21.15.98:0/2141652366 >> 172.21.15.98:0/2141652366 conn(0x7fa1bc06aee0 msgr2=0x7fa1bc06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:22.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.046+0000 7fa1c15ff700 1 -- 172.21.15.98:0/2141652366 shutdown_connections 2023-03-03T07:57:22.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.046+0000 7fa1c15ff700 1 -- 172.21.15.98:0/2141652366 wait complete. 2023-03-03T07:57:22.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.048+0000 7fa1c15ff700 1 Processor -- start 2023-03-03T07:57:22.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.048+0000 7fa1c15ff700 1 -- start start 2023-03-03T07:57:22.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.049+0000 7fa1c15ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1bc0b5410 0x7fa1bc15b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.049+0000 7fa1c15ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1bc0b87c0 0x7fa1bc1f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.049+0000 7fa1c15ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1bc0b8b90 0x7fa1bc0ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.049+0000 7fa1c15ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc0ca5b0 con 0x7fa1bc0b8b90 2023-03-03T07:57:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.049+0000 7fa1c15ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc0ca790 con 0x7fa1bc0b5410 2023-03-03T07:57:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.049+0000 7fa1c15ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc15bce0 con 0x7fa1bc0b87c0 2023-03-03T07:57:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.050+0000 7fa1bb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1bc0b8b90 0x7fa1bc0ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.050+0000 7fa1baffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1bc0b5410 0x7fa1bc15b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.050+0000 7fa1bb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1bc0b8b90 0x7fa1bc0ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45570/0 (socket says 172.21.15.98:45570) 2023-03-03T07:57:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.050+0000 7fa1bb7fe700 1 -- 172.21.15.98:0/3810428980 learned_addr learned my addr 172.21.15.98:0/3810428980 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.050+0000 7fa1ba7fc700 1 --2- 172.21.15.98:0/3810428980 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1bc0b87c0 0x7fa1bc1f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.050+0000 7fa1bb7fe700 1 -- 172.21.15.98:0/3810428980 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1bc0b87c0 msgr2=0x7fa1bc1f3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.050+0000 7fa1bb7fe700 1 --2- 172.21.15.98:0/3810428980 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1bc0b87c0 0x7fa1bc1f3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.050+0000 7fa1bb7fe700 1 -- 172.21.15.98:0/3810428980 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1bc0b5410 msgr2=0x7fa1bc15b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.050+0000 7fa1bb7fe700 1 --2- 172.21.15.98:0/3810428980 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1bc0b5410 0x7fa1bc15b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.050+0000 7fa1bb7fe700 1 -- 172.21.15.98:0/3810428980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac0094a0 con 0x7fa1bc0b8b90 2023-03-03T07:57:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.051+0000 7fa1bb7fe700 1 --2- 172.21.15.98:0/3810428980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1bc0b8b90 0x7fa1bc0ccbd0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa1b400cef0 tx=0x7fa1b400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.051+0000 7fa1a37fe700 1 -- 172.21.15.98:0/3810428980 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b400ad00 con 0x7fa1bc0b8b90 2023-03-03T07:57:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.051+0000 7fa1a37fe700 1 -- 172.21.15.98:0/3810428980 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b400ae60 con 0x7fa1bc0b8b90 2023-03-03T07:57:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.051+0000 7fa1a37fe700 1 -- 172.21.15.98:0/3810428980 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4015680 con 0x7fa1bc0b8b90 2023-03-03T07:57:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.051+0000 7fa1c15ff700 1 -- 172.21.15.98:0/3810428980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1bc0cdf30 con 0x7fa1bc0b8b90 2023-03-03T07:57:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.051+0000 7fa1c15ff700 1 -- 172.21.15.98:0/3810428980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1bc073a80 con 0x7fa1bc0b8b90 2023-03-03T07:57:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.052+0000 7fa1c15ff700 1 -- 172.21.15.98:0/3810428980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1bc0ce0c0 con 0x7fa1bc0b8b90 2023-03-03T07:57:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.053+0000 7fa1a37fe700 1 -- 172.21.15.98:0/3810428980 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa1b401c030 con 0x7fa1bc0b8b90 2023-03-03T07:57:22.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.055+0000 7fa1a37fe700 1 --2- 172.21.15.98:0/3810428980 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa1a80682f0 0x7fa1a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.055+0000 7fa1a37fe700 1 -- 172.21.15.98:0/3810428980 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa1b40860d0 con 0x7fa1bc0b8b90 2023-03-03T07:57:22.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.055+0000 7fa1baffd700 1 --2- 172.21.15.98:0/3810428980 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa1a80682f0 0x7fa1a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.056+0000 7fa1baffd700 1 --2- 172.21.15.98:0/3810428980 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa1a80682f0 0x7fa1a806a7a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0090b0 tx=0x7fa1ac008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:22.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.058+0000 7fa1a37fe700 1 -- 172.21.15.98:0/3810428980 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa1b4052840 con 0x7fa1bc0b8b90 2023-03-03T07:57:22.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.250+0000 7fa1c15ff700 1 -- 172.21.15.98:0/3810428980 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa1bc0729b0 con 0x7fa1a80682f0 2023-03-03T07:57:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.262+0000 7fa1a37fe700 1 -- 172.21.15.98:0/3810428980 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+307606 (secure 0 0 0) 0x7fa1bc0729b0 con 0x7fa1a80682f0 2023-03-03T07:57:22.266 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.267+0000 7fa1c15ff700 1 -- 172.21.15.98:0/3810428980 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa1a80682f0 msgr2=0x7fa1a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.267+0000 7fa1c15ff700 1 --2- 172.21.15.98:0/3810428980 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa1a80682f0 0x7fa1a806a7a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0090b0 tx=0x7fa1ac008030 comp rx=0 tx=0).stop 2023-03-03T07:57:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.267+0000 7fa1c15ff700 1 -- 172.21.15.98:0/3810428980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1bc0b8b90 msgr2=0x7fa1bc0ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.267+0000 7fa1c15ff700 1 --2- 172.21.15.98:0/3810428980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1bc0b8b90 0x7fa1bc0ccbd0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa1b400cef0 tx=0x7fa1b400cf20 comp rx=0 tx=0).stop 2023-03-03T07:57:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.267+0000 7fa1c15ff700 1 -- 172.21.15.98:0/3810428980 shutdown_connections 2023-03-03T07:57:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.267+0000 7fa1c15ff700 1 --2- 172.21.15.98:0/3810428980 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa1a80682f0 0x7fa1a806a7a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.267+0000 7fa1c15ff700 1 --2- 172.21.15.98:0/3810428980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1bc0b8b90 0x7fa1bc0ccbd0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.267+0000 7fa1c15ff700 1 --2- 172.21.15.98:0/3810428980 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1bc0b5410 0x7fa1bc15b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.267+0000 7fa1c15ff700 1 --2- 172.21.15.98:0/3810428980 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1bc0b87c0 0x7fa1bc1f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.267+0000 7fa1c15ff700 1 -- 172.21.15.98:0/3810428980 >> 172.21.15.98:0/3810428980 conn(0x7fa1bc06aee0 msgr2=0x7fa1bc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.267+0000 7fa1c15ff700 1 -- 172.21.15.98:0/3810428980 shutdown_connections 2023-03-03T07:57:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.267+0000 7fa1c15ff700 1 -- 172.21.15.98:0/3810428980 wait complete. 2023-03-03T07:57:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T07:57:22.295 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2023-03-03T07:57:22.197578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":46,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68656,"kb_used_data":4272,"kb_used_omap":0,"kb_used_meta":64320,"kb_avail":754906064,"statfs":{"total":773094113280,"available":773023809536,"internally_reserved":0,"allocated":4374528,"data_stored":2473028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65863680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000901"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.748481+0000","last_change":"2023-03-03T07:57:16.748481+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.748481+0000","last_undegraded":"2023-03-03T07:57:16.748481+0000","last_fullsized":"2023-03-03T07:57:16.748481+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:39:27.674460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746045+0000","last_change":"2023-03-03T07:57:16.746045+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746045+0000","last_undegraded":"2023-03-03T07:57:16.746045+0000","last_fullsized":"2023-03-03T07:57:16.746045+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.739842+0000","last_change":"2023-03-03T07:57:16.739842+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.739842+0000","last_undegraded":"2023-03-03T07:57:16.739842+0000","last_fullsized":"2023-03-03T07:57:16.739842+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.740784+0000","last_change":"2023-03-03T07:57:16.740784+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.740784+0000","last_undegraded":"2023-03-03T07:57:16.740784+0000","last_fullsized":"2023-03-03T07:57:16.740784+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:07.498551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741327+0000","last_change":"2023-03-03T07:57:16.741327+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741327+0000","last_undegraded":"2023-03-03T07:57:16.741327+0000","last_fullsized":"2023-03-03T07:57:16.741327+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:37:50.701337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738165+0000","last_change":"2023-03-03T07:57:16.738165+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738165+0000","last_undegraded":"2023-03-03T07:57:16.738165+0000","last_fullsized":"2023-03-03T07:57:16.738165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:25.355832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.749480+0000","last_change":"2023-03-03T07:57:16.749480+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.749480+0000","last_undegraded":"2023-03-03T07:57:16.749480+0000","last_fullsized":"2023-03-03T07:57:16.749480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:44.621837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747051+0000","last_change":"2023-03-03T07:57:16.747051+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747051+0000","last_undegraded":"2023-03-03T07:57:16.747051+0000","last_fullsized":"2023-03-03T07:57:16.747051+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:01.871020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.739643+0000","last_change":"2023-03-03T07:57:16.739643+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.739643+0000","last_undegraded":"2023-03-03T07:57:16.739643+0000","last_fullsized":"2023-03-03T07:57:16.739643+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747834+0000","last_change":"2023-03-03T07:57:16.747834+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747834+0000","last_undegraded":"2023-03-03T07:57:16.747834+0000","last_fullsized":"2023-03-03T07:57:16.747834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.748481+0000","last_change":"2023-03-03T07:57:16.748481+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.748481+0000","last_undegraded":"2023-03-03T07:57:16.748481+0000","last_fullsized":"2023-03-03T07:57:16.748481+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.742043+0000","last_change":"2023-03-03T07:57:16.742043+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.742043+0000","last_undegraded":"2023-03-03T07:57:16.742043+0000","last_fullsized":"2023-03-03T07:57:16.742043+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746702+0000","last_change":"2023-03-03T07:57:16.746702+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746702+0000","last_undegraded":"2023-03-03T07:57:16.746702+0000","last_fullsized":"2023-03-03T07:57:16.746702+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.730661+0000","last_change":"2023-03-03T07:57:16.730661+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.730661+0000","last_undegraded":"2023-03-03T07:57:16.730661+0000","last_fullsized":"2023-03-03T07:57:16.730661+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:48:58.988637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747212+0000","last_change":"2023-03-03T07:57:16.747212+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747212+0000","last_undegraded":"2023-03-03T07:57:16.747212+0000","last_fullsized":"2023-03-03T07:57:16.747212+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:24:04.359311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746985+0000","last_change":"2023-03-03T07:57:16.746985+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746985+0000","last_undegraded":"2023-03-03T07:57:16.746985+0000","last_fullsized":"2023-03-03T07:57:16.746985+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738184+0000","last_change":"2023-03-03T07:57:16.738184+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738184+0000","last_undegraded":"2023-03-03T07:57:16.738184+0000","last_fullsized":"2023-03-03T07:57:16.738184+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737683+0000","last_change":"2023-03-03T07:57:16.737683+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737683+0000","last_undegraded":"2023-03-03T07:57:16.737683+0000","last_fullsized":"2023-03-03T07:57:16.737683+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:56:10.139056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.740159+0000","last_change":"2023-03-03T07:57:16.740159+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.740159+0000","last_undegraded":"2023-03-03T07:57:16.740159+0000","last_fullsized":"2023-03-03T07:57:16.740159+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:00:23.450170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737107+0000","last_change":"2023-03-03T07:57:16.737107+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737107+0000","last_undegraded":"2023-03-03T07:57:16.737107+0000","last_fullsized":"2023-03-03T07:57:16.737107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:15.531132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746180+0000","last_change":"2023-03-03T07:57:16.746180+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746180+0000","last_undegraded":"2023-03-03T07:57:16.746180+0000","last_fullsized":"2023-03-03T07:57:16.746180+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.732520+0000","last_change":"2023-03-03T07:57:16.732520+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.732520+0000","last_undegraded":"2023-03-03T07:57:16.732520+0000","last_fullsized":"2023-03-03T07:57:16.732520+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:39:27.674460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.739235+0000","last_change":"2023-03-03T07:57:16.739235+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.739235+0000","last_undegraded":"2023-03-03T07:57:16.739235+0000","last_fullsized":"2023-03-03T07:57:16.739235+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738079+0000","last_change":"2023-03-03T07:57:16.738079+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738079+0000","last_undegraded":"2023-03-03T07:57:16.738079+0000","last_fullsized":"2023-03-03T07:57:16.738079+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:57:04.542617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737878+0000","last_change":"2023-03-03T07:57:16.737878+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737878+0000","last_undegraded":"2023-03-03T07:57:16.737878+0000","last_fullsized":"2023-03-03T07:57:16.737878+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:57:04.542617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.736707+0000","last_change":"2023-03-03T07:57:16.736707+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.736707+0000","last_undegraded":"2023-03-03T07:57:16.736707+0000","last_fullsized":"2023-03-03T07:57:16.736707+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.736973+0000","last_change":"2023-03-03T07:57:16.736973+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.736973+0000","last_undegraded":"2023-03-03T07:57:16.736973+0000","last_fullsized":"2023-03-03T07:57:16.736973+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:20.464414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746809+0000","last_change":"2023-03-03T07:57:16.746809+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746809+0000","last_undegraded":"2023-03-03T07:57:16.746809+0000","last_fullsized":"2023-03-03T07:57:16.746809+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.736370+0000","last_change":"2023-03-03T07:57:16.736370+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.736370+0000","last_undegraded":"2023-03-03T07:57:16.736370+0000","last_fullsized":"2023-03-03T07:57:16.736370+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:07.498551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.739548+0000","last_change":"2023-03-03T07:57:16.739548+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.739548+0000","last_undegraded":"2023-03-03T07:57:16.739548+0000","last_fullsized":"2023-03-03T07:57:16.739548+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:43.532953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738752+0000","last_change":"2023-03-03T07:57:16.738752+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738752+0000","last_undegraded":"2023-03-03T07:57:16.738752+0000","last_fullsized":"2023-03-03T07:57:16.738752+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737178+0000","last_change":"2023-03-03T07:57:16.737178+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737178+0000","last_undegraded":"2023-03-03T07:57:16.737178+0000","last_fullsized":"2023-03-03T07:57:16.737178+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.655223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.742046+0000","last_change":"2023-03-03T07:57:16.742046+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.742046+0000","last_undegraded":"2023-03-03T07:57:16.742046+0000","last_fullsized":"2023-03-03T07:57:16.742046+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.749016+0000","last_change":"2023-03-03T07:57:16.745475+0000","last_active":"2023-03-03T07:57:16.749016+0000","last_peered":"2023-03-03T07:57:16.749016+0000","last_clean":"2023-03-03T07:57:16.749016+0000","last_became_active":"2023-03-03T07:57:16.745159+0000","last_became_peered":"2023-03-03T07:57:16.745159+0000","last_unstale":"2023-03-03T07:57:16.749016+0000","last_undegraded":"2023-03-03T07:57:16.749016+0000","last_fullsized":"2023-03-03T07:57:16.749016+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:45:49.372030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738564+0000","last_change":"2023-03-03T07:57:16.738564+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738564+0000","last_undegraded":"2023-03-03T07:57:16.738564+0000","last_fullsized":"2023-03-03T07:57:16.738564+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.758087+0000","last_change":"2023-03-03T07:57:16.755153+0000","last_active":"2023-03-03T07:57:16.758087+0000","last_peered":"2023-03-03T07:57:16.758087+0000","last_clean":"2023-03-03T07:57:16.758087+0000","last_became_active":"2023-03-03T07:57:16.754738+0000","last_became_peered":"2023-03-03T07:57:16.754738+0000","last_unstale":"2023-03-03T07:57:16.758087+0000","last_undegraded":"2023-03-03T07:57:16.758087+0000","last_fullsized":"2023-03-03T07:57:16.758087+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:14.522208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.731582+0000","last_change":"2023-03-03T07:57:16.731582+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.731582+0000","last_undegraded":"2023-03-03T07:57:16.731582+0000","last_fullsized":"2023-03-03T07:57:16.731582+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:11.090994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.751260+0000","last_change":"2023-03-03T07:57:16.746651+0000","last_active":"2023-03-03T07:57:16.751260+0000","last_peered":"2023-03-03T07:57:16.751260+0000","last_clean":"2023-03-03T07:57:16.751260+0000","last_became_active":"2023-03-03T07:57:16.746390+0000","last_became_peered":"2023-03-03T07:57:16.746390+0000","last_unstale":"2023-03-03T07:57:16.751260+0000","last_undegraded":"2023-03-03T07:57:16.751260+0000","last_fullsized":"2023-03-03T07:57:16.751260+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:06.489627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747957+0000","last_change":"2023-03-03T07:57:16.747957+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747957+0000","last_undegraded":"2023-03-03T07:57:16.747957+0000","last_fullsized":"2023-03-03T07:57:16.747957+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:11.090994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754237+0000","last_change":"2023-03-03T07:57:16.743191+0000","last_active":"2023-03-03T07:57:16.754237+0000","last_peered":"2023-03-03T07:57:16.754237+0000","last_clean":"2023-03-03T07:57:16.754237+0000","last_became_active":"2023-03-03T07:57:16.741858+0000","last_became_peered":"2023-03-03T07:57:16.741858+0000","last_unstale":"2023-03-03T07:57:16.754237+0000","last_undegraded":"2023-03-03T07:57:16.754237+0000","last_fullsized":"2023-03-03T07:57:16.754237+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:00.944189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741626+0000","last_change":"2023-03-03T07:57:16.741626+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741626+0000","last_undegraded":"2023-03-03T07:57:16.741626+0000","last_fullsized":"2023-03-03T07:57:16.741626+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754833+0000","last_change":"2023-03-03T07:57:16.753759+0000","last_active":"2023-03-03T07:57:16.754833+0000","last_peered":"2023-03-03T07:57:16.754833+0000","last_clean":"2023-03-03T07:57:16.754833+0000","last_became_active":"2023-03-03T07:57:16.753449+0000","last_became_peered":"2023-03-03T07:57:16.753449+0000","last_unstale":"2023-03-03T07:57:16.754833+0000","last_undegraded":"2023-03-03T07:57:16.754833+0000","last_fullsized":"2023-03-03T07:57:16.754833+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:56:09.130132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746580+0000","last_change":"2023-03-03T07:57:16.746580+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746580+0000","last_undegraded":"2023-03-03T07:57:16.746580+0000","last_fullsized":"2023-03-03T07:57:16.746580+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.746943+0000","last_change":"2023-03-03T07:57:16.740511+0000","last_active":"2023-03-03T07:57:16.746943+0000","last_peered":"2023-03-03T07:57:16.746943+0000","last_clean":"2023-03-03T07:57:16.746943+0000","last_became_active":"2023-03-03T07:57:16.740221+0000","last_became_peered":"2023-03-03T07:57:16.740221+0000","last_unstale":"2023-03-03T07:57:16.746943+0000","last_undegraded":"2023-03-03T07:57:16.746943+0000","last_fullsized":"2023-03-03T07:57:16.746943+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:10.938115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.731963+0000","last_change":"2023-03-03T07:57:16.731963+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.731963+0000","last_undegraded":"2023-03-03T07:57:16.731963+0000","last_fullsized":"2023-03-03T07:57:16.731963+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.760256+0000","last_change":"2023-03-03T07:57:16.756171+0000","last_active":"2023-03-03T07:57:16.760256+0000","last_peered":"2023-03-03T07:57:16.760256+0000","last_clean":"2023-03-03T07:57:16.760256+0000","last_became_active":"2023-03-03T07:57:16.755894+0000","last_became_peered":"2023-03-03T07:57:16.755894+0000","last_unstale":"2023-03-03T07:57:16.760256+0000","last_undegraded":"2023-03-03T07:57:16.760256+0000","last_fullsized":"2023-03-03T07:57:16.760256+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:19.455490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747501+0000","last_change":"2023-03-03T07:57:16.747501+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747501+0000","last_undegraded":"2023-03-03T07:57:16.747501+0000","last_fullsized":"2023-03-03T07:57:16.747501+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:48:58.988637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754386+0000","last_change":"2023-03-03T07:57:16.744111+0000","last_active":"2023-03-03T07:57:16.754386+0000","last_peered":"2023-03-03T07:57:16.754386+0000","last_clean":"2023-03-03T07:57:16.754386+0000","last_became_active":"2023-03-03T07:57:16.743778+0000","last_became_peered":"2023-03-03T07:57:16.743778+0000","last_unstale":"2023-03-03T07:57:16.754386+0000","last_undegraded":"2023-03-03T07:57:16.754386+0000","last_fullsized":"2023-03-03T07:57:16.754386+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:52:59.206045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738898+0000","last_change":"2023-03-03T07:57:16.738898+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738898+0000","last_undegraded":"2023-03-03T07:57:16.738898+0000","last_fullsized":"2023-03-03T07:57:16.738898+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:11.947039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.752195+0000","last_change":"2023-03-03T07:57:16.744161+0000","last_active":"2023-03-03T07:57:16.752195+0000","last_peered":"2023-03-03T07:57:16.752195+0000","last_clean":"2023-03-03T07:57:16.752195+0000","last_became_active":"2023-03-03T07:57:16.743909+0000","last_became_peered":"2023-03-03T07:57:16.743909+0000","last_unstale":"2023-03-03T07:57:16.752195+0000","last_undegraded":"2023-03-03T07:57:16.752195+0000","last_fullsized":"2023-03-03T07:57:16.752195+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:43:42.175275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741492+0000","last_change":"2023-03-03T07:57:16.741492+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741492+0000","last_undegraded":"2023-03-03T07:57:16.741492+0000","last_fullsized":"2023-03-03T07:57:16.741492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.756982+0000","last_change":"2023-03-03T07:57:16.754673+0000","last_active":"2023-03-03T07:57:16.756982+0000","last_peered":"2023-03-03T07:57:16.756982+0000","last_clean":"2023-03-03T07:57:16.756982+0000","last_became_active":"2023-03-03T07:57:16.754415+0000","last_became_peered":"2023-03-03T07:57:16.754415+0000","last_unstale":"2023-03-03T07:57:16.756982+0000","last_undegraded":"2023-03-03T07:57:16.756982+0000","last_fullsized":"2023-03-03T07:57:16.756982+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:06.489627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.742445+0000","last_change":"2023-03-03T07:57:16.742445+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.742445+0000","last_undegraded":"2023-03-03T07:57:16.742445+0000","last_fullsized":"2023-03-03T07:57:16.742445+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.762321+0000","last_change":"2023-03-03T07:57:16.757208+0000","last_active":"2023-03-03T07:57:16.762321+0000","last_peered":"2023-03-03T07:57:16.762321+0000","last_clean":"2023-03-03T07:57:16.762321+0000","last_became_active":"2023-03-03T07:57:16.756910+0000","last_became_peered":"2023-03-03T07:57:16.756910+0000","last_unstale":"2023-03-03T07:57:16.762321+0000","last_undegraded":"2023-03-03T07:57:16.762321+0000","last_fullsized":"2023-03-03T07:57:16.762321+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:14.522208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.731414+0000","last_change":"2023-03-03T07:57:16.731414+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.731414+0000","last_undegraded":"2023-03-03T07:57:16.731414+0000","last_fullsized":"2023-03-03T07:57:16.731414+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:01.871020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.755113+0000","last_change":"2023-03-03T07:57:16.749713+0000","last_active":"2023-03-03T07:57:16.755113+0000","last_peered":"2023-03-03T07:57:16.755113+0000","last_clean":"2023-03-03T07:57:16.755113+0000","last_became_active":"2023-03-03T07:57:16.749304+0000","last_became_peered":"2023-03-03T07:57:16.749304+0000","last_unstale":"2023-03-03T07:57:16.755113+0000","last_undegraded":"2023-03-03T07:57:16.755113+0000","last_fullsized":"2023-03-03T07:57:16.755113+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:19:07.613725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.740047+0000","last_change":"2023-03-03T07:57:16.740047+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.740047+0000","last_undegraded":"2023-03-03T07:57:16.740047+0000","last_fullsized":"2023-03-03T07:57:16.740047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:21:12.720267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.756281+0000","last_change":"2023-03-03T07:57:16.753448+0000","last_active":"2023-03-03T07:57:16.756281+0000","last_peered":"2023-03-03T07:57:16.756281+0000","last_clean":"2023-03-03T07:57:16.756281+0000","last_became_active":"2023-03-03T07:57:16.753183+0000","last_became_peered":"2023-03-03T07:57:16.753183+0000","last_unstale":"2023-03-03T07:57:16.756281+0000","last_undegraded":"2023-03-03T07:57:16.756281+0000","last_fullsized":"2023-03-03T07:57:16.756281+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:54.897881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.745842+0000","last_change":"2023-03-03T07:57:16.745842+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.745842+0000","last_undegraded":"2023-03-03T07:57:16.745842+0000","last_fullsized":"2023-03-03T07:57:16.745842+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:24:04.359311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.760221+0000","last_change":"2023-03-03T07:57:16.754628+0000","last_active":"2023-03-03T07:57:16.760221+0000","last_peered":"2023-03-03T07:57:16.760221+0000","last_clean":"2023-03-03T07:57:16.760221+0000","last_became_active":"2023-03-03T07:57:16.754370+0000","last_became_peered":"2023-03-03T07:57:16.754370+0000","last_unstale":"2023-03-03T07:57:16.760221+0000","last_undegraded":"2023-03-03T07:57:16.760221+0000","last_fullsized":"2023-03-03T07:57:16.760221+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:14.522208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.739793+0000","last_change":"2023-03-03T07:57:16.739793+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.739793+0000","last_undegraded":"2023-03-03T07:57:16.739793+0000","last_fullsized":"2023-03-03T07:57:16.739793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:21:12.720267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.747551+0000","last_change":"2023-03-03T07:57:16.745348+0000","last_active":"2023-03-03T07:57:16.747551+0000","last_peered":"2023-03-03T07:57:16.747551+0000","last_clean":"2023-03-03T07:57:16.747551+0000","last_became_active":"2023-03-03T07:57:16.745100+0000","last_became_peered":"2023-03-03T07:57:16.745100+0000","last_unstale":"2023-03-03T07:57:16.747551+0000","last_undegraded":"2023-03-03T07:57:16.747551+0000","last_fullsized":"2023-03-03T07:57:16.747551+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:17:28.990710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746997+0000","last_change":"2023-03-03T07:57:16.746997+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746997+0000","last_undegraded":"2023-03-03T07:57:16.746997+0000","last_fullsized":"2023-03-03T07:57:16.746997+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:35:32.174190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.743384+0000","last_change":"2023-03-03T07:57:16.736066+0000","last_active":"2023-03-03T07:57:16.743384+0000","last_peered":"2023-03-03T07:57:16.743384+0000","last_clean":"2023-03-03T07:57:16.743384+0000","last_became_active":"2023-03-03T07:57:16.735650+0000","last_became_peered":"2023-03-03T07:57:16.735650+0000","last_unstale":"2023-03-03T07:57:16.743384+0000","last_undegraded":"2023-03-03T07:57:16.743384+0000","last_fullsized":"2023-03-03T07:57:16.743384+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:00:22.441246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.745199+0000","last_change":"2023-03-03T07:57:16.745199+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.745199+0000","last_undegraded":"2023-03-03T07:57:16.745199+0000","last_fullsized":"2023-03-03T07:57:16.745199+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:44.621837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.762270+0000","last_change":"2023-03-03T07:57:16.756645+0000","last_active":"2023-03-03T07:57:16.762270+0000","last_peered":"2023-03-03T07:57:16.762270+0000","last_clean":"2023-03-03T07:57:16.762270+0000","last_became_active":"2023-03-03T07:57:16.756201+0000","last_became_peered":"2023-03-03T07:57:16.756201+0000","last_unstale":"2023-03-03T07:57:16.762270+0000","last_undegraded":"2023-03-03T07:57:16.762270+0000","last_fullsized":"2023-03-03T07:57:16.762270+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:42.524029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747344+0000","last_change":"2023-03-03T07:57:16.747344+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747344+0000","last_undegraded":"2023-03-03T07:57:16.747344+0000","last_fullsized":"2023-03-03T07:57:16.747344+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:13:17.532289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754469+0000","last_change":"2023-03-03T07:57:16.747099+0000","last_active":"2023-03-03T07:57:16.754469+0000","last_peered":"2023-03-03T07:57:16.754469+0000","last_clean":"2023-03-03T07:57:16.754469+0000","last_became_active":"2023-03-03T07:57:16.746766+0000","last_became_peered":"2023-03-03T07:57:16.746766+0000","last_unstale":"2023-03-03T07:57:16.754469+0000","last_undegraded":"2023-03-03T07:57:16.754469+0000","last_fullsized":"2023-03-03T07:57:16.754469+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:25.646299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.731014+0000","last_change":"2023-03-03T07:57:16.731014+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.731014+0000","last_undegraded":"2023-03-03T07:57:16.731014+0000","last_fullsized":"2023-03-03T07:57:16.731014+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:35:32.174190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.753777+0000","last_change":"2023-03-03T07:57:16.748443+0000","last_active":"2023-03-03T07:57:16.753777+0000","last_peered":"2023-03-03T07:57:16.753777+0000","last_clean":"2023-03-03T07:57:16.753777+0000","last_became_active":"2023-03-03T07:57:16.748172+0000","last_became_peered":"2023-03-03T07:57:16.748172+0000","last_unstale":"2023-03-03T07:57:16.753777+0000","last_undegraded":"2023-03-03T07:57:16.753777+0000","last_fullsized":"2023-03-03T07:57:16.753777+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:07:16.498493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738225+0000","last_change":"2023-03-03T07:57:16.738225+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738225+0000","last_undegraded":"2023-03-03T07:57:16.738225+0000","last_fullsized":"2023-03-03T07:57:16.738225+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:43.532953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.751352+0000","last_change":"2023-03-03T07:57:16.746019+0000","last_active":"2023-03-03T07:57:16.751352+0000","last_peered":"2023-03-03T07:57:16.751352+0000","last_clean":"2023-03-03T07:57:16.751352+0000","last_became_active":"2023-03-03T07:57:16.745766+0000","last_became_peered":"2023-03-03T07:57:16.745766+0000","last_unstale":"2023-03-03T07:57:16.751352+0000","last_undegraded":"2023-03-03T07:57:16.751352+0000","last_fullsized":"2023-03-03T07:57:16.751352+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:42.524029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.743223+0000","last_change":"2023-03-03T07:57:16.743223+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.743223+0000","last_undegraded":"2023-03-03T07:57:16.743223+0000","last_fullsized":"2023-03-03T07:57:16.743223+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.756514+0000","last_change":"2023-03-03T07:57:16.753514+0000","last_active":"2023-03-03T07:57:16.756514+0000","last_peered":"2023-03-03T07:57:16.756514+0000","last_clean":"2023-03-03T07:57:16.756514+0000","last_became_active":"2023-03-03T07:57:16.753206+0000","last_became_peered":"2023-03-03T07:57:16.753206+0000","last_unstale":"2023-03-03T07:57:16.756514+0000","last_undegraded":"2023-03-03T07:57:16.756514+0000","last_fullsized":"2023-03-03T07:57:16.756514+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:43:26.659371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741849+0000","last_change":"2023-03-03T07:57:16.741849+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741849+0000","last_undegraded":"2023-03-03T07:57:16.741849+0000","last_fullsized":"2023-03-03T07:57:16.741849+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.758038+0000","last_change":"2023-03-03T07:57:16.754775+0000","last_active":"2023-03-03T07:57:16.758038+0000","last_peered":"2023-03-03T07:57:16.758038+0000","last_clean":"2023-03-03T07:57:16.758038+0000","last_became_active":"2023-03-03T07:57:16.754315+0000","last_became_peered":"2023-03-03T07:57:16.754315+0000","last_unstale":"2023-03-03T07:57:16.758038+0000","last_undegraded":"2023-03-03T07:57:16.758038+0000","last_fullsized":"2023-03-03T07:57:16.758038+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:14:32.071416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746117+0000","last_change":"2023-03-03T07:57:16.746117+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746117+0000","last_undegraded":"2023-03-03T07:57:16.746117+0000","last_fullsized":"2023-03-03T07:57:16.746117+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.747475+0000","last_change":"2023-03-03T07:57:16.742812+0000","last_active":"2023-03-03T07:57:16.747475+0000","last_peered":"2023-03-03T07:57:16.747475+0000","last_clean":"2023-03-03T07:57:16.747475+0000","last_became_active":"2023-03-03T07:57:16.742529+0000","last_became_peered":"2023-03-03T07:57:16.742529+0000","last_unstale":"2023-03-03T07:57:16.747475+0000","last_undegraded":"2023-03-03T07:57:16.747475+0000","last_fullsized":"2023-03-03T07:57:16.747475+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:37:49.692413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746443+0000","last_change":"2023-03-03T07:57:16.746443+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746443+0000","last_undegraded":"2023-03-03T07:57:16.746443+0000","last_fullsized":"2023-03-03T07:57:16.746443+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.756542+0000","last_change":"2023-03-03T07:57:16.753472+0000","last_active":"2023-03-03T07:57:16.756542+0000","last_peered":"2023-03-03T07:57:16.756542+0000","last_clean":"2023-03-03T07:57:16.756542+0000","last_became_active":"2023-03-03T07:57:16.753183+0000","last_became_peered":"2023-03-03T07:57:16.753183+0000","last_unstale":"2023-03-03T07:57:16.756542+0000","last_undegraded":"2023-03-03T07:57:16.756542+0000","last_fullsized":"2023-03-03T07:57:16.756542+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:53.094916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741199+0000","last_change":"2023-03-03T07:57:16.741199+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741199+0000","last_undegraded":"2023-03-03T07:57:16.741199+0000","last_fullsized":"2023-03-03T07:57:16.741199+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.756071+0000","last_change":"2023-03-03T07:57:16.751997+0000","last_active":"2023-03-03T07:57:16.756071+0000","last_peered":"2023-03-03T07:57:16.756071+0000","last_clean":"2023-03-03T07:57:16.756071+0000","last_became_active":"2023-03-03T07:57:16.751593+0000","last_became_peered":"2023-03-03T07:57:16.751593+0000","last_unstale":"2023-03-03T07:57:16.756071+0000","last_undegraded":"2023-03-03T07:57:16.756071+0000","last_fullsized":"2023-03-03T07:57:16.756071+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:19.455490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.727392+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T07:57:15.727392+0000","last_peered":"2023-03-03T07:57:15.727392+0000","last_clean":"2023-03-03T07:57:15.727392+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T07:57:15.727392+0000","last_undegraded":"2023-03-03T07:57:15.727392+0000","last_fullsized":"2023-03-03T07:57:15.727392+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.744979+0000","last_change":"2023-03-03T07:57:16.744979+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.744979+0000","last_undegraded":"2023-03-03T07:57:16.744979+0000","last_fullsized":"2023-03-03T07:57:16.744979+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.752649+0000","last_change":"2023-03-03T07:57:16.749605+0000","last_active":"2023-03-03T07:57:16.752649+0000","last_peered":"2023-03-03T07:57:16.752649+0000","last_clean":"2023-03-03T07:57:16.752649+0000","last_became_active":"2023-03-03T07:57:16.749336+0000","last_became_peered":"2023-03-03T07:57:16.749336+0000","last_unstale":"2023-03-03T07:57:16.752649+0000","last_undegraded":"2023-03-03T07:57:16.752649+0000","last_fullsized":"2023-03-03T07:57:16.752649+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:42.524029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.722841+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T07:57:15.722841+0000","last_peered":"2023-03-03T07:57:15.722841+0000","last_clean":"2023-03-03T07:57:15.722841+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T07:57:15.722841+0000","last_undegraded":"2023-03-03T07:57:15.722841+0000","last_fullsized":"2023-03-03T07:57:15.722841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741077+0000","last_change":"2023-03-03T07:57:16.741077+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741077+0000","last_undegraded":"2023-03-03T07:57:16.741077+0000","last_fullsized":"2023-03-03T07:57:16.741077+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.753872+0000","last_change":"2023-03-03T07:57:16.750917+0000","last_active":"2023-03-03T07:57:16.753872+0000","last_peered":"2023-03-03T07:57:16.753872+0000","last_clean":"2023-03-03T07:57:16.753872+0000","last_became_active":"2023-03-03T07:57:16.750615+0000","last_became_peered":"2023-03-03T07:57:16.750615+0000","last_unstale":"2023-03-03T07:57:16.753872+0000","last_undegraded":"2023-03-03T07:57:16.753872+0000","last_fullsized":"2023-03-03T07:57:16.753872+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:49:51.007473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.732651+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T07:57:15.732651+0000","last_peered":"2023-03-03T07:57:15.732651+0000","last_clean":"2023-03-03T07:57:15.732651+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T07:57:15.732651+0000","last_undegraded":"2023-03-03T07:57:15.732651+0000","last_fullsized":"2023-03-03T07:57:15.732651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.745525+0000","last_change":"2023-03-03T07:57:16.745525+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.745525+0000","last_undegraded":"2023-03-03T07:57:16.745525+0000","last_fullsized":"2023-03-03T07:57:16.745525+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.751970+0000","last_change":"2023-03-03T07:57:16.748170+0000","last_active":"2023-03-03T07:57:16.751970+0000","last_peered":"2023-03-03T07:57:16.751970+0000","last_clean":"2023-03-03T07:57:16.751970+0000","last_became_active":"2023-03-03T07:57:16.747842+0000","last_became_peered":"2023-03-03T07:57:16.747842+0000","last_unstale":"2023-03-03T07:57:16.751970+0000","last_undegraded":"2023-03-03T07:57:16.751970+0000","last_fullsized":"2023-03-03T07:57:16.751970+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:57:03.533693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.723040+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T07:57:15.723040+0000","last_peered":"2023-03-03T07:57:15.723040+0000","last_clean":"2023-03-03T07:57:15.723040+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T07:57:15.723040+0000","last_undegraded":"2023-03-03T07:57:15.723040+0000","last_fullsized":"2023-03-03T07:57:15.723040+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":176,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.728933+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T07:57:15.728933+0000","last_peered":"2023-03-03T07:57:15.728933+0000","last_clean":"2023-03-03T07:57:15.728933+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T07:57:15.728933+0000","last_undegraded":"2023-03-03T07:57:15.728933+0000","last_fullsized":"2023-03-03T07:57:15.728933+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737432+0000","last_change":"2023-03-03T07:57:16.737432+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737432+0000","last_undegraded":"2023-03-03T07:57:16.737432+0000","last_fullsized":"2023-03-03T07:57:16.737432+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.762247+0000","last_change":"2023-03-03T07:57:16.756473+0000","last_active":"2023-03-03T07:57:16.762247+0000","last_peered":"2023-03-03T07:57:16.762247+0000","last_clean":"2023-03-03T07:57:16.762247+0000","last_became_active":"2023-03-03T07:57:16.755809+0000","last_became_peered":"2023-03-03T07:57:16.755809+0000","last_unstale":"2023-03-03T07:57:16.762247+0000","last_undegraded":"2023-03-03T07:57:16.762247+0000","last_fullsized":"2023-03-03T07:57:16.762247+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:19.455490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.729041+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T07:57:15.729041+0000","last_peered":"2023-03-03T07:57:15.729041+0000","last_clean":"2023-03-03T07:57:15.729041+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T07:57:15.729041+0000","last_undegraded":"2023-03-03T07:57:15.729041+0000","last_fullsized":"2023-03-03T07:57:15.729041+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737491+0000","last_change":"2023-03-03T07:57:16.737491+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737491+0000","last_undegraded":"2023-03-03T07:57:16.737491+0000","last_fullsized":"2023-03-03T07:57:16.737491+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:59.426058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754211+0000","last_change":"2023-03-03T07:57:16.752378+0000","last_active":"2023-03-03T07:57:16.754211+0000","last_peered":"2023-03-03T07:57:16.754211+0000","last_clean":"2023-03-03T07:57:16.754211+0000","last_became_active":"2023-03-03T07:57:16.752028+0000","last_became_peered":"2023-03-03T07:57:16.752028+0000","last_unstale":"2023-03-03T07:57:16.754211+0000","last_undegraded":"2023-03-03T07:57:16.754211+0000","last_fullsized":"2023-03-03T07:57:16.754211+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:14:02.921691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.729041+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T07:57:15.729041+0000","last_peered":"2023-03-03T07:57:15.729041+0000","last_clean":"2023-03-03T07:57:15.729041+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T07:57:15.729041+0000","last_undegraded":"2023-03-03T07:57:15.729041+0000","last_fullsized":"2023-03-03T07:57:15.729041+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.731292+0000","last_change":"2023-03-03T07:57:16.731292+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.731292+0000","last_undegraded":"2023-03-03T07:57:16.731292+0000","last_fullsized":"2023-03-03T07:57:16.731292+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754275+0000","last_change":"2023-03-03T07:57:16.742160+0000","last_active":"2023-03-03T07:57:16.754275+0000","last_peered":"2023-03-03T07:57:16.754275+0000","last_clean":"2023-03-03T07:57:16.754275+0000","last_became_active":"2023-03-03T07:57:16.741796+0000","last_became_peered":"2023-03-03T07:57:16.741796+0000","last_unstale":"2023-03-03T07:57:16.754275+0000","last_undegraded":"2023-03-03T07:57:16.754275+0000","last_fullsized":"2023-03-03T07:57:16.754275+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:55.349529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.727435+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T07:57:15.727435+0000","last_peered":"2023-03-03T07:57:15.727435+0000","last_clean":"2023-03-03T07:57:15.727435+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T07:57:15.727435+0000","last_undegraded":"2023-03-03T07:57:15.727435+0000","last_fullsized":"2023-03-03T07:57:15.727435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737604+0000","last_change":"2023-03-03T07:57:16.737604+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737604+0000","last_undegraded":"2023-03-03T07:57:16.737604+0000","last_fullsized":"2023-03-03T07:57:16.737604+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:25.355832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.752225+0000","last_change":"2023-03-03T07:57:16.744051+0000","last_active":"2023-03-03T07:57:16.752225+0000","last_peered":"2023-03-03T07:57:16.752225+0000","last_clean":"2023-03-03T07:57:16.752225+0000","last_became_active":"2023-03-03T07:57:16.743785+0000","last_became_peered":"2023-03-03T07:57:16.743785+0000","last_unstale":"2023-03-03T07:57:16.752225+0000","last_undegraded":"2023-03-03T07:57:16.752225+0000","last_fullsized":"2023-03-03T07:57:16.752225+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:48:13.119986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.726936+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T07:57:15.726936+0000","last_peered":"2023-03-03T07:57:15.726936+0000","last_clean":"2023-03-03T07:57:15.726936+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T07:57:15.726936+0000","last_undegraded":"2023-03-03T07:57:15.726936+0000","last_fullsized":"2023-03-03T07:57:15.726936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9424,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362416,"statfs":{"total":96636764160,"available":96627113984,"internally_reserved":0,"allocated":991232,"data_stored":750757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.624},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":0.60399999999999998}]},{"osd":1,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":6.6829999999999998,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":0.64800000000000002}]},{"osd":2,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.54300000000000004}]},{"osd":3,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.46000000000000002}]},{"osd":4,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.441},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.57199999999999995}]},{"osd":5,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.70399999999999996}]},{"osd":6,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.499}]}]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8200,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363640,"statfs":{"total":96636764160,"available":96628367360,"internally_reserved":0,"allocated":393216,"data_stored":159370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.79000000000000004}]},{"osd":1,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.68100000000000005}]},{"osd":2,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.754},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.66700000000000004}]},{"osd":3,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.89800000000000002}]},{"osd":4,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":4.0099999999999998,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":4.1859999999999999,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.70999999999999996}]},{"osd":5,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.82299999999999995}]},{"osd":7,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.63100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8332,"kb_used_data":388,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363508,"statfs":{"total":96636764160,"available":96628232192,"internally_reserved":0,"allocated":397312,"data_stored":160389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.972,"5min":3.972,"15min":3.972},"last":0.48899999999999999}]},{"osd":2,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":11.897,"5min":11.897,"15min":11.897},"last":0.44400000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":11.804,"5min":11.804,"15min":11.804},"last":0.51300000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":11.843999999999999,"5min":11.843999999999999,"15min":11.843999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":3.8740000000000001,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.43099999999999999}]},{"osd":4,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.72699999999999998}]},{"osd":5,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":3.8300000000000001,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.73899999999999999}]},{"osd":6,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":3.907,"5min":3.907,"15min":3.907},"last":0.748},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.78000000000000003}]},{"osd":7,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":3.9399999999999999,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":0.64800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9356,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362484,"statfs":{"total":96636764160,"available":96627183616,"internally_reserved":0,"allocated":987136,"data_stored":749738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.36599999999999999}]},{"osd":2,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.47199999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.629}]},{"osd":4,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.95999999999999996}]},{"osd":5,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.91300000000000003}]},{"osd":6,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.872}]},{"osd":7,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.88100000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8200,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363640,"statfs":{"total":96636764160,"available":96628367360,"internally_reserved":0,"allocated":393216,"data_stored":159370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":5.7889999999999997,"5min":5.7889999999999997,"15min":5.7889999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":2.6309999999999998,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":0.505}]},{"osd":1,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":5.8470000000000004,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.60799999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.961,"5min":12.961,"15min":12.961},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.1449999999999996,"5min":6.1449999999999996,"15min":6.1449999999999996},"last":0.55500000000000005}]},{"osd":4,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":6.484,"5min":6.484,"15min":6.484},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":6.5510000000000002,"5min":6.5510000000000002,"15min":6.5510000000000002},"last":0.96099999999999997}]},{"osd":5,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":5.7119999999999997,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":0.871},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":6.6369999999999996,"5min":6.6369999999999996,"15min":6.6369999999999996},"last":0.92900000000000005}]},{"osd":6,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":6.625,"5min":6.625,"15min":6.625},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":5.7539999999999996,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.91400000000000003}]},{"osd":7,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":6.7009999999999996,"5min":6.7009999999999996,"15min":6.7009999999999996},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":6.6840000000000002,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.98999999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8332,"kb_used_data":388,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363508,"statfs":{"total":96636764160,"available":96628232192,"internally_reserved":0,"allocated":397312,"data_stored":160389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":9.5820000000000007,"5min":9.5820000000000007,"15min":9.5820000000000007},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":9.7840000000000007,"5min":9.7840000000000007,"15min":9.7840000000000007},"last":0.35499999999999998}]},{"osd":1,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.702,"5min":9.702,"15min":9.702},"last":0.60999999999999999}]},{"osd":2,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":8.3190000000000008,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.133,"5min":1.133,"15min":1.133},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":9.734,"5min":9.734,"15min":9.734},"last":0.68000000000000005}]},{"osd":4,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":9.8629999999999995,"5min":9.8629999999999995,"15min":9.8629999999999995},"last":0.871},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.99199999999999999}]},{"osd":5,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":9.6600000000000001,"5min":9.6600000000000001,"15min":9.6600000000000001},"last":0.90100000000000002}]},{"osd":6,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":9.8190000000000008,"5min":9.8190000000000008,"15min":9.8190000000000008},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.91700000000000004}]},{"osd":7,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":9.625,"5min":9.625,"15min":9.625},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":9.8949999999999996,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":1.026}]}]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8472,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363368,"statfs":{"total":96636764160,"available":96628088832,"internally_reserved":0,"allocated":409600,"data_stored":167017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":7.1799999999999997,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.51300000000000001}]},{"osd":1,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":9.5709999999999997,"5min":9.5709999999999997,"15min":9.5709999999999997},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":5.266,"5min":5.266,"15min":5.266},"last":0.70399999999999996}]},{"osd":2,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":8.4529999999999994,"5min":8.4529999999999994,"15min":8.4529999999999994},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":6.6829999999999998,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.84999999999999998}]},{"osd":5,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.748}]},{"osd":6,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.82499999999999996}]},{"osd":7,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.92900000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8340,"kb_used_data":396,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363500,"statfs":{"total":96636764160,"available":96628224000,"internally_reserved":0,"allocated":405504,"data_stored":165998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69699999999999995}]},{"osd":1,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.76400000000000001}]},{"osd":2,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.72599999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.63200000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.80000000000000004}]},{"osd":6,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.85099999999999998}]},{"osd":7,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.747},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.82899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T07:57:22.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T07:57:22.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.579+0000 7f6d90fff700 1 Processor -- start 2023-03-03T07:57:22.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.580+0000 7f6d90fff700 1 -- start start 2023-03-03T07:57:22.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.581+0000 7f6d90fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d8c15cc00 0x7f6d8c0cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.581+0000 7f6d90fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6d8c0d2c30 0x7f6d8c0cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.581+0000 7f6d90fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c0cde10 con 0x7f6d8c0d27c0 2023-03-03T07:57:22.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.581+0000 7f6d90fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d8c0ce040 con 0x7f6d8c0d2c30 2023-03-03T07:57:22.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.581+0000 7f6d90fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c0d30b0 con 0x7f6d8c15cc00 2023-03-03T07:57:22.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.582+0000 7f6d8ad9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6d8c0d2c30 0x7f6d8c0cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.582+0000 7f6d8a59c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d8c15cc00 0x7f6d8c0cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.582+0000 7f6d89d9b700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f6d8c0d27c0 0x7f6d8c0b60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:45318/0 (socket says 172.21.15.98:45318) 2023-03-03T07:57:22.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.582+0000 7f6d8a59c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d8c15cc00 0x7f6d8c0cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:47132/0 (socket says 172.21.15.98:47132) 2023-03-03T07:57:22.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.582+0000 7f6d89d9b700 1 -- 172.21.15.98:0/4048469356 learned_addr learned my addr 172.21.15.98:0/4048469356 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:22.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.582+0000 7f6d8959a700 1 -- 172.21.15.98:0/4048469356 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012199185 0 0) 0x7f6d8c0cde10 con 0x7f6d8c0d27c0 2023-03-03T07:57:22.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.583+0000 7f6d8959a700 1 -- 172.21.15.98:0/4048469356 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d740035b0 con 0x7f6d8c0d27c0 2023-03-03T07:57:22.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.583+0000 7f6d8a59c700 1 -- 172.21.15.98:0/4048469356 >> v2:172.21.15.100:3300/0 conn(0x7f6d8c0d2c30 msgr2=0x7f6d8c0cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:22.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.583+0000 7f6d8a59c700 1 --2- 172.21.15.98:0/4048469356 >> v2:172.21.15.100:3300/0 conn(0x7f6d8c0d2c30 0x7f6d8c0cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.583+0000 7f6d8a59c700 1 -- 172.21.15.98:0/4048469356 >> v1:172.21.15.100:6789/0 conn(0x7f6d8c0d27c0 legacy=0x7f6d8c0b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.583+0000 7f6d8a59c700 1 -- 172.21.15.98:0/4048469356 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c0d1a00 con 0x7f6d8c15cc00 2023-03-03T07:57:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.583+0000 7f6d8ad9d700 1 --2- 172.21.15.98:0/4048469356 >> v2:172.21.15.100:3300/0 conn(0x7f6d8c0d2c30 0x7f6d8c0cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.583+0000 7f6d8a59c700 1 --2- 172.21.15.98:0/4048469356 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d8c15cc00 0x7f6d8c0cd8d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c009840 tx=0x7f6d7c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfdeb0382a8f4ff5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.583+0000 7f6d8959a700 1 -- 172.21.15.98:0/4048469356 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c011830 con 0x7f6d8c15cc00 2023-03-03T07:57:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.583+0000 7f6d8959a700 1 -- 172.21.15.98:0/4048469356 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7c011990 con 0x7f6d8c15cc00 2023-03-03T07:57:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.583+0000 7f6d8959a700 1 -- 172.21.15.98:0/4048469356 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c011d20 con 0x7f6d8c15cc00 2023-03-03T07:57:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.584+0000 7f6d90fff700 1 -- 172.21.15.98:0/4048469356 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d8c15cc00 msgr2=0x7f6d8c0cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.584+0000 7f6d90fff700 1 --2- 172.21.15.98:0/4048469356 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d8c15cc00 0x7f6d8c0cd8d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c009840 tx=0x7f6d7c009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:22.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.584+0000 7f6d90fff700 1 -- 172.21.15.98:0/4048469356 shutdown_connections 2023-03-03T07:57:22.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.584+0000 7f6d90fff700 1 --2- 172.21.15.98:0/4048469356 >> v2:172.21.15.100:3300/0 conn(0x7f6d8c0d2c30 0x7f6d8c0cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.584+0000 7f6d90fff700 1 --2- 172.21.15.98:0/4048469356 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d8c15cc00 0x7f6d8c0cd8d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.584+0000 7f6d90fff700 1 -- 172.21.15.98:0/4048469356 >> 172.21.15.98:0/4048469356 conn(0x7f6d8c06aee0 msgr2=0x7f6d8c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:22.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.584+0000 7f6d90fff700 1 -- 172.21.15.98:0/4048469356 shutdown_connections 2023-03-03T07:57:22.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.584+0000 7f6d90fff700 1 -- 172.21.15.98:0/4048469356 wait complete. 2023-03-03T07:57:22.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.586+0000 7f6d90fff700 1 Processor -- start 2023-03-03T07:57:22.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.586+0000 7f6d90fff700 1 -- start start 2023-03-03T07:57:22.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.587+0000 7f6d90fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d8c0d27c0 0x7f6d8c1f6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.587+0000 7f6d90fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d8c0d2c30 0x7f6d8c1f4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.587+0000 7f6d90fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d8c15cc00 0x7f6d8c0cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.587+0000 7f6d90fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c15b000 con 0x7f6d8c0d27c0 2023-03-03T07:57:22.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.587+0000 7f6d90fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c1f6660 con 0x7f6d8c15cc00 2023-03-03T07:57:22.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.587+0000 7f6d90fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c15a0a0 con 0x7f6d8c0d2c30 2023-03-03T07:57:22.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.588+0000 7f6d8a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d8c0d27c0 0x7f6d8c1f6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.588+0000 7f6d8a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d8c0d27c0 0x7f6d8c1f6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45576/0 (socket says 172.21.15.98:45576) 2023-03-03T07:57:22.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.588+0000 7f6d8a59c700 1 -- 172.21.15.98:0/1749282571 learned_addr learned my addr 172.21.15.98:0/1749282571 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:22.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.588+0000 7f6d89d9b700 1 --2- 172.21.15.98:0/1749282571 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d8c0d2c30 0x7f6d8c1f4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.588+0000 7f6d8ad9d700 1 --2- 172.21.15.98:0/1749282571 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d8c15cc00 0x7f6d8c0cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.588+0000 7f6d8a59c700 1 -- 172.21.15.98:0/1749282571 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d8c0d2c30 msgr2=0x7f6d8c1f4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.588+0000 7f6d8a59c700 1 --2- 172.21.15.98:0/1749282571 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d8c0d2c30 0x7f6d8c1f4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.588+0000 7f6d8a59c700 1 -- 172.21.15.98:0/1749282571 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d8c15cc00 msgr2=0x7f6d8c0cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.588+0000 7f6d8a59c700 1 --2- 172.21.15.98:0/1749282571 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d8c15cc00 0x7f6d8c0cb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.588+0000 7f6d8a59c700 1 -- 172.21.15.98:0/1749282571 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d84009a70 con 0x7f6d8c0d27c0 2023-03-03T07:57:22.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.589+0000 7f6d8a59c700 1 --2- 172.21.15.98:0/1749282571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d8c0d27c0 0x7f6d8c1f6120 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c0090b0 tx=0x7f6d7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:22.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.589+0000 7f6d7affd700 1 -- 172.21.15.98:0/1749282571 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c004590 con 0x7f6d8c0d27c0 2023-03-03T07:57:22.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.589+0000 7f6d7affd700 1 -- 172.21.15.98:0/1749282571 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7c0036a0 con 0x7f6d8c0d27c0 2023-03-03T07:57:22.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.589+0000 7f6d7affd700 1 -- 172.21.15.98:0/1749282571 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c003a10 con 0x7f6d8c0d27c0 2023-03-03T07:57:22.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.589+0000 7f6d90fff700 1 -- 172.21.15.98:0/1749282571 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7c0094a0 con 0x7f6d8c0d27c0 2023-03-03T07:57:22.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.589+0000 7f6d90fff700 1 -- 172.21.15.98:0/1749282571 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8c0d1fc0 con 0x7f6d8c0d27c0 2023-03-03T07:57:22.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.590+0000 7f6d90fff700 1 -- 172.21.15.98:0/1749282571 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d8c0d1ce0 con 0x7f6d8c0d27c0 2023-03-03T07:57:22.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.591+0000 7f6d7affd700 1 -- 172.21.15.98:0/1749282571 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6d7c00f050 con 0x7f6d8c0d27c0 2023-03-03T07:57:22.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.594+0000 7f6d7affd700 1 --2- 172.21.15.98:0/1749282571 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6d6c0682f0 0x7f6d6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.594+0000 7f6d7affd700 1 -- 172.21.15.98:0/1749282571 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6d7c086ef0 con 0x7f6d8c0d27c0 2023-03-03T07:57:22.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.594+0000 7f6d89d9b700 1 --2- 172.21.15.98:0/1749282571 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6d6c0682f0 0x7f6d6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.595+0000 7f6d89d9b700 1 --2- 172.21.15.98:0/1749282571 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6d6c0682f0 0x7f6d6c06a7a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c0cb780 tx=0x7f6d80007dc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:22.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.598+0000 7f6d7affd700 1 -- 172.21.15.98:0/1749282571 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6d7c053660 con 0x7f6d8c0d27c0 2023-03-03T07:57:22.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.789+0000 7f6d90fff700 1 -- 172.21.15.98:0/1749282571 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6d8c0729b0 con 0x7f6d6c0682f0 2023-03-03T07:57:22.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.798+0000 7f6d7affd700 1 -- 172.21.15.98:0/1749282571 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+307606 (secure 0 0 0) 0x7f6d8c0729b0 con 0x7f6d6c0682f0 2023-03-03T07:57:22.803 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:22.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.803+0000 7f6d90fff700 1 -- 172.21.15.98:0/1749282571 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6d6c0682f0 msgr2=0x7f6d6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.803+0000 7f6d90fff700 1 --2- 172.21.15.98:0/1749282571 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6d6c0682f0 0x7f6d6c06a7a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c0cb780 tx=0x7f6d80007dc0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.803+0000 7f6d90fff700 1 -- 172.21.15.98:0/1749282571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d8c0d27c0 msgr2=0x7f6d8c1f6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.803+0000 7f6d90fff700 1 --2- 172.21.15.98:0/1749282571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d8c0d27c0 0x7f6d8c1f6120 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c0090b0 tx=0x7f6d7c004750 comp rx=0 tx=0).stop 2023-03-03T07:57:22.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.804+0000 7f6d90fff700 1 -- 172.21.15.98:0/1749282571 shutdown_connections 2023-03-03T07:57:22.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.804+0000 7f6d90fff700 1 --2- 172.21.15.98:0/1749282571 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6d6c0682f0 0x7f6d6c06a7a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.804+0000 7f6d90fff700 1 --2- 172.21.15.98:0/1749282571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d8c0d27c0 0x7f6d8c1f6120 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.804+0000 7f6d90fff700 1 --2- 172.21.15.98:0/1749282571 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d8c15cc00 0x7f6d8c0cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.804+0000 7f6d90fff700 1 --2- 172.21.15.98:0/1749282571 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d8c0d2c30 0x7f6d8c1f4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.804+0000 7f6d90fff700 1 -- 172.21.15.98:0/1749282571 >> 172.21.15.98:0/1749282571 conn(0x7f6d8c06aee0 msgr2=0x7f6d8c14f890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:22.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.804+0000 7f6d90fff700 1 -- 172.21.15.98:0/1749282571 shutdown_connections 2023-03-03T07:57:22.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:22.804+0000 7f6d90fff700 1 -- 172.21.15.98:0/1749282571 wait complete. 2023-03-03T07:57:22.807 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T07:57:22.832 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2023-03-03T07:57:22.197578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":46,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68656,"kb_used_data":4272,"kb_used_omap":0,"kb_used_meta":64320,"kb_avail":754906064,"statfs":{"total":773094113280,"available":773023809536,"internally_reserved":0,"allocated":4374528,"data_stored":2473028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65863680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000901"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.748481+0000","last_change":"2023-03-03T07:57:16.748481+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.748481+0000","last_undegraded":"2023-03-03T07:57:16.748481+0000","last_fullsized":"2023-03-03T07:57:16.748481+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:39:27.674460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746045+0000","last_change":"2023-03-03T07:57:16.746045+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746045+0000","last_undegraded":"2023-03-03T07:57:16.746045+0000","last_fullsized":"2023-03-03T07:57:16.746045+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.739842+0000","last_change":"2023-03-03T07:57:16.739842+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.739842+0000","last_undegraded":"2023-03-03T07:57:16.739842+0000","last_fullsized":"2023-03-03T07:57:16.739842+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.740784+0000","last_change":"2023-03-03T07:57:16.740784+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.740784+0000","last_undegraded":"2023-03-03T07:57:16.740784+0000","last_fullsized":"2023-03-03T07:57:16.740784+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:07.498551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741327+0000","last_change":"2023-03-03T07:57:16.741327+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741327+0000","last_undegraded":"2023-03-03T07:57:16.741327+0000","last_fullsized":"2023-03-03T07:57:16.741327+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:37:50.701337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738165+0000","last_change":"2023-03-03T07:57:16.738165+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738165+0000","last_undegraded":"2023-03-03T07:57:16.738165+0000","last_fullsized":"2023-03-03T07:57:16.738165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:25.355832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.749480+0000","last_change":"2023-03-03T07:57:16.749480+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.749480+0000","last_undegraded":"2023-03-03T07:57:16.749480+0000","last_fullsized":"2023-03-03T07:57:16.749480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:44.621837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747051+0000","last_change":"2023-03-03T07:57:16.747051+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747051+0000","last_undegraded":"2023-03-03T07:57:16.747051+0000","last_fullsized":"2023-03-03T07:57:16.747051+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:01.871020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.739643+0000","last_change":"2023-03-03T07:57:16.739643+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.739643+0000","last_undegraded":"2023-03-03T07:57:16.739643+0000","last_fullsized":"2023-03-03T07:57:16.739643+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747834+0000","last_change":"2023-03-03T07:57:16.747834+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747834+0000","last_undegraded":"2023-03-03T07:57:16.747834+0000","last_fullsized":"2023-03-03T07:57:16.747834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.748481+0000","last_change":"2023-03-03T07:57:16.748481+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.748481+0000","last_undegraded":"2023-03-03T07:57:16.748481+0000","last_fullsized":"2023-03-03T07:57:16.748481+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.742043+0000","last_change":"2023-03-03T07:57:16.742043+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.742043+0000","last_undegraded":"2023-03-03T07:57:16.742043+0000","last_fullsized":"2023-03-03T07:57:16.742043+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746702+0000","last_change":"2023-03-03T07:57:16.746702+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746702+0000","last_undegraded":"2023-03-03T07:57:16.746702+0000","last_fullsized":"2023-03-03T07:57:16.746702+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.730661+0000","last_change":"2023-03-03T07:57:16.730661+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.730661+0000","last_undegraded":"2023-03-03T07:57:16.730661+0000","last_fullsized":"2023-03-03T07:57:16.730661+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:48:58.988637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747212+0000","last_change":"2023-03-03T07:57:16.747212+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747212+0000","last_undegraded":"2023-03-03T07:57:16.747212+0000","last_fullsized":"2023-03-03T07:57:16.747212+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:24:04.359311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746985+0000","last_change":"2023-03-03T07:57:16.746985+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746985+0000","last_undegraded":"2023-03-03T07:57:16.746985+0000","last_fullsized":"2023-03-03T07:57:16.746985+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738184+0000","last_change":"2023-03-03T07:57:16.738184+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738184+0000","last_undegraded":"2023-03-03T07:57:16.738184+0000","last_fullsized":"2023-03-03T07:57:16.738184+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737683+0000","last_change":"2023-03-03T07:57:16.737683+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737683+0000","last_undegraded":"2023-03-03T07:57:16.737683+0000","last_fullsized":"2023-03-03T07:57:16.737683+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:56:10.139056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.740159+0000","last_change":"2023-03-03T07:57:16.740159+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.740159+0000","last_undegraded":"2023-03-03T07:57:16.740159+0000","last_fullsized":"2023-03-03T07:57:16.740159+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:00:23.450170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737107+0000","last_change":"2023-03-03T07:57:16.737107+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737107+0000","last_undegraded":"2023-03-03T07:57:16.737107+0000","last_fullsized":"2023-03-03T07:57:16.737107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:15.531132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746180+0000","last_change":"2023-03-03T07:57:16.746180+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746180+0000","last_undegraded":"2023-03-03T07:57:16.746180+0000","last_fullsized":"2023-03-03T07:57:16.746180+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.732520+0000","last_change":"2023-03-03T07:57:16.732520+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.732520+0000","last_undegraded":"2023-03-03T07:57:16.732520+0000","last_fullsized":"2023-03-03T07:57:16.732520+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:39:27.674460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.739235+0000","last_change":"2023-03-03T07:57:16.739235+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.739235+0000","last_undegraded":"2023-03-03T07:57:16.739235+0000","last_fullsized":"2023-03-03T07:57:16.739235+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738079+0000","last_change":"2023-03-03T07:57:16.738079+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738079+0000","last_undegraded":"2023-03-03T07:57:16.738079+0000","last_fullsized":"2023-03-03T07:57:16.738079+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:57:04.542617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737878+0000","last_change":"2023-03-03T07:57:16.737878+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737878+0000","last_undegraded":"2023-03-03T07:57:16.737878+0000","last_fullsized":"2023-03-03T07:57:16.737878+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:57:04.542617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.736707+0000","last_change":"2023-03-03T07:57:16.736707+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.736707+0000","last_undegraded":"2023-03-03T07:57:16.736707+0000","last_fullsized":"2023-03-03T07:57:16.736707+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.736973+0000","last_change":"2023-03-03T07:57:16.736973+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.736973+0000","last_undegraded":"2023-03-03T07:57:16.736973+0000","last_fullsized":"2023-03-03T07:57:16.736973+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:20.464414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746809+0000","last_change":"2023-03-03T07:57:16.746809+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746809+0000","last_undegraded":"2023-03-03T07:57:16.746809+0000","last_fullsized":"2023-03-03T07:57:16.746809+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.736370+0000","last_change":"2023-03-03T07:57:16.736370+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.736370+0000","last_undegraded":"2023-03-03T07:57:16.736370+0000","last_fullsized":"2023-03-03T07:57:16.736370+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:07.498551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.739548+0000","last_change":"2023-03-03T07:57:16.739548+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.739548+0000","last_undegraded":"2023-03-03T07:57:16.739548+0000","last_fullsized":"2023-03-03T07:57:16.739548+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:43.532953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738752+0000","last_change":"2023-03-03T07:57:16.738752+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738752+0000","last_undegraded":"2023-03-03T07:57:16.738752+0000","last_fullsized":"2023-03-03T07:57:16.738752+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737178+0000","last_change":"2023-03-03T07:57:16.737178+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737178+0000","last_undegraded":"2023-03-03T07:57:16.737178+0000","last_fullsized":"2023-03-03T07:57:16.737178+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.655223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.742046+0000","last_change":"2023-03-03T07:57:16.742046+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.742046+0000","last_undegraded":"2023-03-03T07:57:16.742046+0000","last_fullsized":"2023-03-03T07:57:16.742046+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.749016+0000","last_change":"2023-03-03T07:57:16.745475+0000","last_active":"2023-03-03T07:57:16.749016+0000","last_peered":"2023-03-03T07:57:16.749016+0000","last_clean":"2023-03-03T07:57:16.749016+0000","last_became_active":"2023-03-03T07:57:16.745159+0000","last_became_peered":"2023-03-03T07:57:16.745159+0000","last_unstale":"2023-03-03T07:57:16.749016+0000","last_undegraded":"2023-03-03T07:57:16.749016+0000","last_fullsized":"2023-03-03T07:57:16.749016+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:45:49.372030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738564+0000","last_change":"2023-03-03T07:57:16.738564+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738564+0000","last_undegraded":"2023-03-03T07:57:16.738564+0000","last_fullsized":"2023-03-03T07:57:16.738564+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.758087+0000","last_change":"2023-03-03T07:57:16.755153+0000","last_active":"2023-03-03T07:57:16.758087+0000","last_peered":"2023-03-03T07:57:16.758087+0000","last_clean":"2023-03-03T07:57:16.758087+0000","last_became_active":"2023-03-03T07:57:16.754738+0000","last_became_peered":"2023-03-03T07:57:16.754738+0000","last_unstale":"2023-03-03T07:57:16.758087+0000","last_undegraded":"2023-03-03T07:57:16.758087+0000","last_fullsized":"2023-03-03T07:57:16.758087+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:14.522208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.731582+0000","last_change":"2023-03-03T07:57:16.731582+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.731582+0000","last_undegraded":"2023-03-03T07:57:16.731582+0000","last_fullsized":"2023-03-03T07:57:16.731582+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:11.090994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.751260+0000","last_change":"2023-03-03T07:57:16.746651+0000","last_active":"2023-03-03T07:57:16.751260+0000","last_peered":"2023-03-03T07:57:16.751260+0000","last_clean":"2023-03-03T07:57:16.751260+0000","last_became_active":"2023-03-03T07:57:16.746390+0000","last_became_peered":"2023-03-03T07:57:16.746390+0000","last_unstale":"2023-03-03T07:57:16.751260+0000","last_undegraded":"2023-03-03T07:57:16.751260+0000","last_fullsized":"2023-03-03T07:57:16.751260+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:06.489627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747957+0000","last_change":"2023-03-03T07:57:16.747957+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747957+0000","last_undegraded":"2023-03-03T07:57:16.747957+0000","last_fullsized":"2023-03-03T07:57:16.747957+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:11.090994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754237+0000","last_change":"2023-03-03T07:57:16.743191+0000","last_active":"2023-03-03T07:57:16.754237+0000","last_peered":"2023-03-03T07:57:16.754237+0000","last_clean":"2023-03-03T07:57:16.754237+0000","last_became_active":"2023-03-03T07:57:16.741858+0000","last_became_peered":"2023-03-03T07:57:16.741858+0000","last_unstale":"2023-03-03T07:57:16.754237+0000","last_undegraded":"2023-03-03T07:57:16.754237+0000","last_fullsized":"2023-03-03T07:57:16.754237+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:00.944189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741626+0000","last_change":"2023-03-03T07:57:16.741626+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741626+0000","last_undegraded":"2023-03-03T07:57:16.741626+0000","last_fullsized":"2023-03-03T07:57:16.741626+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754833+0000","last_change":"2023-03-03T07:57:16.753759+0000","last_active":"2023-03-03T07:57:16.754833+0000","last_peered":"2023-03-03T07:57:16.754833+0000","last_clean":"2023-03-03T07:57:16.754833+0000","last_became_active":"2023-03-03T07:57:16.753449+0000","last_became_peered":"2023-03-03T07:57:16.753449+0000","last_unstale":"2023-03-03T07:57:16.754833+0000","last_undegraded":"2023-03-03T07:57:16.754833+0000","last_fullsized":"2023-03-03T07:57:16.754833+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:56:09.130132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746580+0000","last_change":"2023-03-03T07:57:16.746580+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746580+0000","last_undegraded":"2023-03-03T07:57:16.746580+0000","last_fullsized":"2023-03-03T07:57:16.746580+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.746943+0000","last_change":"2023-03-03T07:57:16.740511+0000","last_active":"2023-03-03T07:57:16.746943+0000","last_peered":"2023-03-03T07:57:16.746943+0000","last_clean":"2023-03-03T07:57:16.746943+0000","last_became_active":"2023-03-03T07:57:16.740221+0000","last_became_peered":"2023-03-03T07:57:16.740221+0000","last_unstale":"2023-03-03T07:57:16.746943+0000","last_undegraded":"2023-03-03T07:57:16.746943+0000","last_fullsized":"2023-03-03T07:57:16.746943+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:10.938115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.731963+0000","last_change":"2023-03-03T07:57:16.731963+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.731963+0000","last_undegraded":"2023-03-03T07:57:16.731963+0000","last_fullsized":"2023-03-03T07:57:16.731963+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.760256+0000","last_change":"2023-03-03T07:57:16.756171+0000","last_active":"2023-03-03T07:57:16.760256+0000","last_peered":"2023-03-03T07:57:16.760256+0000","last_clean":"2023-03-03T07:57:16.760256+0000","last_became_active":"2023-03-03T07:57:16.755894+0000","last_became_peered":"2023-03-03T07:57:16.755894+0000","last_unstale":"2023-03-03T07:57:16.760256+0000","last_undegraded":"2023-03-03T07:57:16.760256+0000","last_fullsized":"2023-03-03T07:57:16.760256+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:19.455490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747501+0000","last_change":"2023-03-03T07:57:16.747501+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747501+0000","last_undegraded":"2023-03-03T07:57:16.747501+0000","last_fullsized":"2023-03-03T07:57:16.747501+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:48:58.988637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754386+0000","last_change":"2023-03-03T07:57:16.744111+0000","last_active":"2023-03-03T07:57:16.754386+0000","last_peered":"2023-03-03T07:57:16.754386+0000","last_clean":"2023-03-03T07:57:16.754386+0000","last_became_active":"2023-03-03T07:57:16.743778+0000","last_became_peered":"2023-03-03T07:57:16.743778+0000","last_unstale":"2023-03-03T07:57:16.754386+0000","last_undegraded":"2023-03-03T07:57:16.754386+0000","last_fullsized":"2023-03-03T07:57:16.754386+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:52:59.206045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738898+0000","last_change":"2023-03-03T07:57:16.738898+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738898+0000","last_undegraded":"2023-03-03T07:57:16.738898+0000","last_fullsized":"2023-03-03T07:57:16.738898+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:11.947039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.752195+0000","last_change":"2023-03-03T07:57:16.744161+0000","last_active":"2023-03-03T07:57:16.752195+0000","last_peered":"2023-03-03T07:57:16.752195+0000","last_clean":"2023-03-03T07:57:16.752195+0000","last_became_active":"2023-03-03T07:57:16.743909+0000","last_became_peered":"2023-03-03T07:57:16.743909+0000","last_unstale":"2023-03-03T07:57:16.752195+0000","last_undegraded":"2023-03-03T07:57:16.752195+0000","last_fullsized":"2023-03-03T07:57:16.752195+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:43:42.175275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741492+0000","last_change":"2023-03-03T07:57:16.741492+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741492+0000","last_undegraded":"2023-03-03T07:57:16.741492+0000","last_fullsized":"2023-03-03T07:57:16.741492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.756982+0000","last_change":"2023-03-03T07:57:16.754673+0000","last_active":"2023-03-03T07:57:16.756982+0000","last_peered":"2023-03-03T07:57:16.756982+0000","last_clean":"2023-03-03T07:57:16.756982+0000","last_became_active":"2023-03-03T07:57:16.754415+0000","last_became_peered":"2023-03-03T07:57:16.754415+0000","last_unstale":"2023-03-03T07:57:16.756982+0000","last_undegraded":"2023-03-03T07:57:16.756982+0000","last_fullsized":"2023-03-03T07:57:16.756982+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:06.489627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.742445+0000","last_change":"2023-03-03T07:57:16.742445+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.742445+0000","last_undegraded":"2023-03-03T07:57:16.742445+0000","last_fullsized":"2023-03-03T07:57:16.742445+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.762321+0000","last_change":"2023-03-03T07:57:16.757208+0000","last_active":"2023-03-03T07:57:16.762321+0000","last_peered":"2023-03-03T07:57:16.762321+0000","last_clean":"2023-03-03T07:57:16.762321+0000","last_became_active":"2023-03-03T07:57:16.756910+0000","last_became_peered":"2023-03-03T07:57:16.756910+0000","last_unstale":"2023-03-03T07:57:16.762321+0000","last_undegraded":"2023-03-03T07:57:16.762321+0000","last_fullsized":"2023-03-03T07:57:16.762321+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:14.522208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.731414+0000","last_change":"2023-03-03T07:57:16.731414+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.731414+0000","last_undegraded":"2023-03-03T07:57:16.731414+0000","last_fullsized":"2023-03-03T07:57:16.731414+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:01.871020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.755113+0000","last_change":"2023-03-03T07:57:16.749713+0000","last_active":"2023-03-03T07:57:16.755113+0000","last_peered":"2023-03-03T07:57:16.755113+0000","last_clean":"2023-03-03T07:57:16.755113+0000","last_became_active":"2023-03-03T07:57:16.749304+0000","last_became_peered":"2023-03-03T07:57:16.749304+0000","last_unstale":"2023-03-03T07:57:16.755113+0000","last_undegraded":"2023-03-03T07:57:16.755113+0000","last_fullsized":"2023-03-03T07:57:16.755113+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:19:07.613725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.740047+0000","last_change":"2023-03-03T07:57:16.740047+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.740047+0000","last_undegraded":"2023-03-03T07:57:16.740047+0000","last_fullsized":"2023-03-03T07:57:16.740047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:21:12.720267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.756281+0000","last_change":"2023-03-03T07:57:16.753448+0000","last_active":"2023-03-03T07:57:16.756281+0000","last_peered":"2023-03-03T07:57:16.756281+0000","last_clean":"2023-03-03T07:57:16.756281+0000","last_became_active":"2023-03-03T07:57:16.753183+0000","last_became_peered":"2023-03-03T07:57:16.753183+0000","last_unstale":"2023-03-03T07:57:16.756281+0000","last_undegraded":"2023-03-03T07:57:16.756281+0000","last_fullsized":"2023-03-03T07:57:16.756281+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:54.897881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.745842+0000","last_change":"2023-03-03T07:57:16.745842+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.745842+0000","last_undegraded":"2023-03-03T07:57:16.745842+0000","last_fullsized":"2023-03-03T07:57:16.745842+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:24:04.359311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.760221+0000","last_change":"2023-03-03T07:57:16.754628+0000","last_active":"2023-03-03T07:57:16.760221+0000","last_peered":"2023-03-03T07:57:16.760221+0000","last_clean":"2023-03-03T07:57:16.760221+0000","last_became_active":"2023-03-03T07:57:16.754370+0000","last_became_peered":"2023-03-03T07:57:16.754370+0000","last_unstale":"2023-03-03T07:57:16.760221+0000","last_undegraded":"2023-03-03T07:57:16.760221+0000","last_fullsized":"2023-03-03T07:57:16.760221+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:14.522208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.739793+0000","last_change":"2023-03-03T07:57:16.739793+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.739793+0000","last_undegraded":"2023-03-03T07:57:16.739793+0000","last_fullsized":"2023-03-03T07:57:16.739793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:21:12.720267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.747551+0000","last_change":"2023-03-03T07:57:16.745348+0000","last_active":"2023-03-03T07:57:16.747551+0000","last_peered":"2023-03-03T07:57:16.747551+0000","last_clean":"2023-03-03T07:57:16.747551+0000","last_became_active":"2023-03-03T07:57:16.745100+0000","last_became_peered":"2023-03-03T07:57:16.745100+0000","last_unstale":"2023-03-03T07:57:16.747551+0000","last_undegraded":"2023-03-03T07:57:16.747551+0000","last_fullsized":"2023-03-03T07:57:16.747551+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:17:28.990710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746997+0000","last_change":"2023-03-03T07:57:16.746997+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746997+0000","last_undegraded":"2023-03-03T07:57:16.746997+0000","last_fullsized":"2023-03-03T07:57:16.746997+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:35:32.174190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.743384+0000","last_change":"2023-03-03T07:57:16.736066+0000","last_active":"2023-03-03T07:57:16.743384+0000","last_peered":"2023-03-03T07:57:16.743384+0000","last_clean":"2023-03-03T07:57:16.743384+0000","last_became_active":"2023-03-03T07:57:16.735650+0000","last_became_peered":"2023-03-03T07:57:16.735650+0000","last_unstale":"2023-03-03T07:57:16.743384+0000","last_undegraded":"2023-03-03T07:57:16.743384+0000","last_fullsized":"2023-03-03T07:57:16.743384+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:00:22.441246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.745199+0000","last_change":"2023-03-03T07:57:16.745199+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.745199+0000","last_undegraded":"2023-03-03T07:57:16.745199+0000","last_fullsized":"2023-03-03T07:57:16.745199+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:44.621837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.762270+0000","last_change":"2023-03-03T07:57:16.756645+0000","last_active":"2023-03-03T07:57:16.762270+0000","last_peered":"2023-03-03T07:57:16.762270+0000","last_clean":"2023-03-03T07:57:16.762270+0000","last_became_active":"2023-03-03T07:57:16.756201+0000","last_became_peered":"2023-03-03T07:57:16.756201+0000","last_unstale":"2023-03-03T07:57:16.762270+0000","last_undegraded":"2023-03-03T07:57:16.762270+0000","last_fullsized":"2023-03-03T07:57:16.762270+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:42.524029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747344+0000","last_change":"2023-03-03T07:57:16.747344+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747344+0000","last_undegraded":"2023-03-03T07:57:16.747344+0000","last_fullsized":"2023-03-03T07:57:16.747344+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:13:17.532289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754469+0000","last_change":"2023-03-03T07:57:16.747099+0000","last_active":"2023-03-03T07:57:16.754469+0000","last_peered":"2023-03-03T07:57:16.754469+0000","last_clean":"2023-03-03T07:57:16.754469+0000","last_became_active":"2023-03-03T07:57:16.746766+0000","last_became_peered":"2023-03-03T07:57:16.746766+0000","last_unstale":"2023-03-03T07:57:16.754469+0000","last_undegraded":"2023-03-03T07:57:16.754469+0000","last_fullsized":"2023-03-03T07:57:16.754469+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:25.646299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.731014+0000","last_change":"2023-03-03T07:57:16.731014+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.731014+0000","last_undegraded":"2023-03-03T07:57:16.731014+0000","last_fullsized":"2023-03-03T07:57:16.731014+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:35:32.174190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.753777+0000","last_change":"2023-03-03T07:57:16.748443+0000","last_active":"2023-03-03T07:57:16.753777+0000","last_peered":"2023-03-03T07:57:16.753777+0000","last_clean":"2023-03-03T07:57:16.753777+0000","last_became_active":"2023-03-03T07:57:16.748172+0000","last_became_peered":"2023-03-03T07:57:16.748172+0000","last_unstale":"2023-03-03T07:57:16.753777+0000","last_undegraded":"2023-03-03T07:57:16.753777+0000","last_fullsized":"2023-03-03T07:57:16.753777+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:07:16.498493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738225+0000","last_change":"2023-03-03T07:57:16.738225+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738225+0000","last_undegraded":"2023-03-03T07:57:16.738225+0000","last_fullsized":"2023-03-03T07:57:16.738225+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:43.532953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.751352+0000","last_change":"2023-03-03T07:57:16.746019+0000","last_active":"2023-03-03T07:57:16.751352+0000","last_peered":"2023-03-03T07:57:16.751352+0000","last_clean":"2023-03-03T07:57:16.751352+0000","last_became_active":"2023-03-03T07:57:16.745766+0000","last_became_peered":"2023-03-03T07:57:16.745766+0000","last_unstale":"2023-03-03T07:57:16.751352+0000","last_undegraded":"2023-03-03T07:57:16.751352+0000","last_fullsized":"2023-03-03T07:57:16.751352+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:42.524029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.743223+0000","last_change":"2023-03-03T07:57:16.743223+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.743223+0000","last_undegraded":"2023-03-03T07:57:16.743223+0000","last_fullsized":"2023-03-03T07:57:16.743223+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.756514+0000","last_change":"2023-03-03T07:57:16.753514+0000","last_active":"2023-03-03T07:57:16.756514+0000","last_peered":"2023-03-03T07:57:16.756514+0000","last_clean":"2023-03-03T07:57:16.756514+0000","last_became_active":"2023-03-03T07:57:16.753206+0000","last_became_peered":"2023-03-03T07:57:16.753206+0000","last_unstale":"2023-03-03T07:57:16.756514+0000","last_undegraded":"2023-03-03T07:57:16.756514+0000","last_fullsized":"2023-03-03T07:57:16.756514+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:43:26.659371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741849+0000","last_change":"2023-03-03T07:57:16.741849+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741849+0000","last_undegraded":"2023-03-03T07:57:16.741849+0000","last_fullsized":"2023-03-03T07:57:16.741849+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.758038+0000","last_change":"2023-03-03T07:57:16.754775+0000","last_active":"2023-03-03T07:57:16.758038+0000","last_peered":"2023-03-03T07:57:16.758038+0000","last_clean":"2023-03-03T07:57:16.758038+0000","last_became_active":"2023-03-03T07:57:16.754315+0000","last_became_peered":"2023-03-03T07:57:16.754315+0000","last_unstale":"2023-03-03T07:57:16.758038+0000","last_undegraded":"2023-03-03T07:57:16.758038+0000","last_fullsized":"2023-03-03T07:57:16.758038+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:14:32.071416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746117+0000","last_change":"2023-03-03T07:57:16.746117+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746117+0000","last_undegraded":"2023-03-03T07:57:16.746117+0000","last_fullsized":"2023-03-03T07:57:16.746117+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.747475+0000","last_change":"2023-03-03T07:57:16.742812+0000","last_active":"2023-03-03T07:57:16.747475+0000","last_peered":"2023-03-03T07:57:16.747475+0000","last_clean":"2023-03-03T07:57:16.747475+0000","last_became_active":"2023-03-03T07:57:16.742529+0000","last_became_peered":"2023-03-03T07:57:16.742529+0000","last_unstale":"2023-03-03T07:57:16.747475+0000","last_undegraded":"2023-03-03T07:57:16.747475+0000","last_fullsized":"2023-03-03T07:57:16.747475+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:37:49.692413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746443+0000","last_change":"2023-03-03T07:57:16.746443+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746443+0000","last_undegraded":"2023-03-03T07:57:16.746443+0000","last_fullsized":"2023-03-03T07:57:16.746443+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.756542+0000","last_change":"2023-03-03T07:57:16.753472+0000","last_active":"2023-03-03T07:57:16.756542+0000","last_peered":"2023-03-03T07:57:16.756542+0000","last_clean":"2023-03-03T07:57:16.756542+0000","last_became_active":"2023-03-03T07:57:16.753183+0000","last_became_peered":"2023-03-03T07:57:16.753183+0000","last_unstale":"2023-03-03T07:57:16.756542+0000","last_undegraded":"2023-03-03T07:57:16.756542+0000","last_fullsized":"2023-03-03T07:57:16.756542+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:53.094916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741199+0000","last_change":"2023-03-03T07:57:16.741199+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741199+0000","last_undegraded":"2023-03-03T07:57:16.741199+0000","last_fullsized":"2023-03-03T07:57:16.741199+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.756071+0000","last_change":"2023-03-03T07:57:16.751997+0000","last_active":"2023-03-03T07:57:16.756071+0000","last_peered":"2023-03-03T07:57:16.756071+0000","last_clean":"2023-03-03T07:57:16.756071+0000","last_became_active":"2023-03-03T07:57:16.751593+0000","last_became_peered":"2023-03-03T07:57:16.751593+0000","last_unstale":"2023-03-03T07:57:16.756071+0000","last_undegraded":"2023-03-03T07:57:16.756071+0000","last_fullsized":"2023-03-03T07:57:16.756071+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:19.455490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.727392+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T07:57:15.727392+0000","last_peered":"2023-03-03T07:57:15.727392+0000","last_clean":"2023-03-03T07:57:15.727392+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T07:57:15.727392+0000","last_undegraded":"2023-03-03T07:57:15.727392+0000","last_fullsized":"2023-03-03T07:57:15.727392+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.744979+0000","last_change":"2023-03-03T07:57:16.744979+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.744979+0000","last_undegraded":"2023-03-03T07:57:16.744979+0000","last_fullsized":"2023-03-03T07:57:16.744979+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.752649+0000","last_change":"2023-03-03T07:57:16.749605+0000","last_active":"2023-03-03T07:57:16.752649+0000","last_peered":"2023-03-03T07:57:16.752649+0000","last_clean":"2023-03-03T07:57:16.752649+0000","last_became_active":"2023-03-03T07:57:16.749336+0000","last_became_peered":"2023-03-03T07:57:16.749336+0000","last_unstale":"2023-03-03T07:57:16.752649+0000","last_undegraded":"2023-03-03T07:57:16.752649+0000","last_fullsized":"2023-03-03T07:57:16.752649+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:42.524029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.722841+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T07:57:15.722841+0000","last_peered":"2023-03-03T07:57:15.722841+0000","last_clean":"2023-03-03T07:57:15.722841+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T07:57:15.722841+0000","last_undegraded":"2023-03-03T07:57:15.722841+0000","last_fullsized":"2023-03-03T07:57:15.722841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741077+0000","last_change":"2023-03-03T07:57:16.741077+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741077+0000","last_undegraded":"2023-03-03T07:57:16.741077+0000","last_fullsized":"2023-03-03T07:57:16.741077+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.753872+0000","last_change":"2023-03-03T07:57:16.750917+0000","last_active":"2023-03-03T07:57:16.753872+0000","last_peered":"2023-03-03T07:57:16.753872+0000","last_clean":"2023-03-03T07:57:16.753872+0000","last_became_active":"2023-03-03T07:57:16.750615+0000","last_became_peered":"2023-03-03T07:57:16.750615+0000","last_unstale":"2023-03-03T07:57:16.753872+0000","last_undegraded":"2023-03-03T07:57:16.753872+0000","last_fullsized":"2023-03-03T07:57:16.753872+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:49:51.007473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.732651+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T07:57:15.732651+0000","last_peered":"2023-03-03T07:57:15.732651+0000","last_clean":"2023-03-03T07:57:15.732651+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T07:57:15.732651+0000","last_undegraded":"2023-03-03T07:57:15.732651+0000","last_fullsized":"2023-03-03T07:57:15.732651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.745525+0000","last_change":"2023-03-03T07:57:16.745525+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.745525+0000","last_undegraded":"2023-03-03T07:57:16.745525+0000","last_fullsized":"2023-03-03T07:57:16.745525+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.751970+0000","last_change":"2023-03-03T07:57:16.748170+0000","last_active":"2023-03-03T07:57:16.751970+0000","last_peered":"2023-03-03T07:57:16.751970+0000","last_clean":"2023-03-03T07:57:16.751970+0000","last_became_active":"2023-03-03T07:57:16.747842+0000","last_became_peered":"2023-03-03T07:57:16.747842+0000","last_unstale":"2023-03-03T07:57:16.751970+0000","last_undegraded":"2023-03-03T07:57:16.751970+0000","last_fullsized":"2023-03-03T07:57:16.751970+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:57:03.533693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.723040+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T07:57:15.723040+0000","last_peered":"2023-03-03T07:57:15.723040+0000","last_clean":"2023-03-03T07:57:15.723040+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T07:57:15.723040+0000","last_undegraded":"2023-03-03T07:57:15.723040+0000","last_fullsized":"2023-03-03T07:57:15.723040+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":176,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.728933+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T07:57:15.728933+0000","last_peered":"2023-03-03T07:57:15.728933+0000","last_clean":"2023-03-03T07:57:15.728933+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T07:57:15.728933+0000","last_undegraded":"2023-03-03T07:57:15.728933+0000","last_fullsized":"2023-03-03T07:57:15.728933+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737432+0000","last_change":"2023-03-03T07:57:16.737432+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737432+0000","last_undegraded":"2023-03-03T07:57:16.737432+0000","last_fullsized":"2023-03-03T07:57:16.737432+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.762247+0000","last_change":"2023-03-03T07:57:16.756473+0000","last_active":"2023-03-03T07:57:16.762247+0000","last_peered":"2023-03-03T07:57:16.762247+0000","last_clean":"2023-03-03T07:57:16.762247+0000","last_became_active":"2023-03-03T07:57:16.755809+0000","last_became_peered":"2023-03-03T07:57:16.755809+0000","last_unstale":"2023-03-03T07:57:16.762247+0000","last_undegraded":"2023-03-03T07:57:16.762247+0000","last_fullsized":"2023-03-03T07:57:16.762247+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:19.455490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.729041+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T07:57:15.729041+0000","last_peered":"2023-03-03T07:57:15.729041+0000","last_clean":"2023-03-03T07:57:15.729041+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T07:57:15.729041+0000","last_undegraded":"2023-03-03T07:57:15.729041+0000","last_fullsized":"2023-03-03T07:57:15.729041+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737491+0000","last_change":"2023-03-03T07:57:16.737491+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737491+0000","last_undegraded":"2023-03-03T07:57:16.737491+0000","last_fullsized":"2023-03-03T07:57:16.737491+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:59.426058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754211+0000","last_change":"2023-03-03T07:57:16.752378+0000","last_active":"2023-03-03T07:57:16.754211+0000","last_peered":"2023-03-03T07:57:16.754211+0000","last_clean":"2023-03-03T07:57:16.754211+0000","last_became_active":"2023-03-03T07:57:16.752028+0000","last_became_peered":"2023-03-03T07:57:16.752028+0000","last_unstale":"2023-03-03T07:57:16.754211+0000","last_undegraded":"2023-03-03T07:57:16.754211+0000","last_fullsized":"2023-03-03T07:57:16.754211+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:14:02.921691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.729041+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T07:57:15.729041+0000","last_peered":"2023-03-03T07:57:15.729041+0000","last_clean":"2023-03-03T07:57:15.729041+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T07:57:15.729041+0000","last_undegraded":"2023-03-03T07:57:15.729041+0000","last_fullsized":"2023-03-03T07:57:15.729041+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.731292+0000","last_change":"2023-03-03T07:57:16.731292+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.731292+0000","last_undegraded":"2023-03-03T07:57:16.731292+0000","last_fullsized":"2023-03-03T07:57:16.731292+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754275+0000","last_change":"2023-03-03T07:57:16.742160+0000","last_active":"2023-03-03T07:57:16.754275+0000","last_peered":"2023-03-03T07:57:16.754275+0000","last_clean":"2023-03-03T07:57:16.754275+0000","last_became_active":"2023-03-03T07:57:16.741796+0000","last_became_peered":"2023-03-03T07:57:16.741796+0000","last_unstale":"2023-03-03T07:57:16.754275+0000","last_undegraded":"2023-03-03T07:57:16.754275+0000","last_fullsized":"2023-03-03T07:57:16.754275+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:55.349529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.727435+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T07:57:15.727435+0000","last_peered":"2023-03-03T07:57:15.727435+0000","last_clean":"2023-03-03T07:57:15.727435+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T07:57:15.727435+0000","last_undegraded":"2023-03-03T07:57:15.727435+0000","last_fullsized":"2023-03-03T07:57:15.727435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737604+0000","last_change":"2023-03-03T07:57:16.737604+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737604+0000","last_undegraded":"2023-03-03T07:57:16.737604+0000","last_fullsized":"2023-03-03T07:57:16.737604+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:25.355832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.752225+0000","last_change":"2023-03-03T07:57:16.744051+0000","last_active":"2023-03-03T07:57:16.752225+0000","last_peered":"2023-03-03T07:57:16.752225+0000","last_clean":"2023-03-03T07:57:16.752225+0000","last_became_active":"2023-03-03T07:57:16.743785+0000","last_became_peered":"2023-03-03T07:57:16.743785+0000","last_unstale":"2023-03-03T07:57:16.752225+0000","last_undegraded":"2023-03-03T07:57:16.752225+0000","last_fullsized":"2023-03-03T07:57:16.752225+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:48:13.119986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.726936+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T07:57:15.726936+0000","last_peered":"2023-03-03T07:57:15.726936+0000","last_clean":"2023-03-03T07:57:15.726936+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T07:57:15.726936+0000","last_undegraded":"2023-03-03T07:57:15.726936+0000","last_fullsized":"2023-03-03T07:57:15.726936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9424,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362416,"statfs":{"total":96636764160,"available":96627113984,"internally_reserved":0,"allocated":991232,"data_stored":750757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.624},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":0.60399999999999998}]},{"osd":1,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":6.6829999999999998,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":0.64800000000000002}]},{"osd":2,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.54300000000000004}]},{"osd":3,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.46000000000000002}]},{"osd":4,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.441},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.57199999999999995}]},{"osd":5,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.70399999999999996}]},{"osd":6,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.499}]}]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8200,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363640,"statfs":{"total":96636764160,"available":96628367360,"internally_reserved":0,"allocated":393216,"data_stored":159370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.79000000000000004}]},{"osd":1,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.68100000000000005}]},{"osd":2,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.754},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.66700000000000004}]},{"osd":3,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.89800000000000002}]},{"osd":4,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":4.0099999999999998,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":4.1859999999999999,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.70999999999999996}]},{"osd":5,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.82299999999999995}]},{"osd":7,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.63100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8332,"kb_used_data":388,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363508,"statfs":{"total":96636764160,"available":96628232192,"internally_reserved":0,"allocated":397312,"data_stored":160389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.972,"5min":3.972,"15min":3.972},"last":0.48899999999999999}]},{"osd":2,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":11.897,"5min":11.897,"15min":11.897},"last":0.44400000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":11.804,"5min":11.804,"15min":11.804},"last":0.51300000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":11.843999999999999,"5min":11.843999999999999,"15min":11.843999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":3.8740000000000001,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.43099999999999999}]},{"osd":4,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.72699999999999998}]},{"osd":5,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":3.8300000000000001,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.73899999999999999}]},{"osd":6,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":3.907,"5min":3.907,"15min":3.907},"last":0.748},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.78000000000000003}]},{"osd":7,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":3.9399999999999999,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":0.64800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9356,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362484,"statfs":{"total":96636764160,"available":96627183616,"internally_reserved":0,"allocated":987136,"data_stored":749738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.36599999999999999}]},{"osd":2,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.47199999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.629}]},{"osd":4,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.95999999999999996}]},{"osd":5,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.91300000000000003}]},{"osd":6,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.872}]},{"osd":7,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.88100000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8200,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363640,"statfs":{"total":96636764160,"available":96628367360,"internally_reserved":0,"allocated":393216,"data_stored":159370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":5.7889999999999997,"5min":5.7889999999999997,"15min":5.7889999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":2.6309999999999998,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":0.505}]},{"osd":1,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":5.8470000000000004,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.60799999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.961,"5min":12.961,"15min":12.961},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.1449999999999996,"5min":6.1449999999999996,"15min":6.1449999999999996},"last":0.55500000000000005}]},{"osd":4,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":6.484,"5min":6.484,"15min":6.484},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":6.5510000000000002,"5min":6.5510000000000002,"15min":6.5510000000000002},"last":0.96099999999999997}]},{"osd":5,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":5.7119999999999997,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":0.871},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":6.6369999999999996,"5min":6.6369999999999996,"15min":6.6369999999999996},"last":0.92900000000000005}]},{"osd":6,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":6.625,"5min":6.625,"15min":6.625},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":5.7539999999999996,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.91400000000000003}]},{"osd":7,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":6.7009999999999996,"5min":6.7009999999999996,"15min":6.7009999999999996},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":6.6840000000000002,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.98999999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8332,"kb_used_data":388,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363508,"statfs":{"total":96636764160,"available":96628232192,"internally_reserved":0,"allocated":397312,"data_stored":160389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":9.5820000000000007,"5min":9.5820000000000007,"15min":9.5820000000000007},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":9.7840000000000007,"5min":9.7840000000000007,"15min":9.7840000000000007},"last":0.35499999999999998}]},{"osd":1,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.702,"5min":9.702,"15min":9.702},"last":0.60999999999999999}]},{"osd":2,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":8.3190000000000008,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.133,"5min":1.133,"15min":1.133},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":9.734,"5min":9.734,"15min":9.734},"last":0.68000000000000005}]},{"osd":4,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":9.8629999999999995,"5min":9.8629999999999995,"15min":9.8629999999999995},"last":0.871},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.99199999999999999}]},{"osd":5,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":9.6600000000000001,"5min":9.6600000000000001,"15min":9.6600000000000001},"last":0.90100000000000002}]},{"osd":6,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":9.8190000000000008,"5min":9.8190000000000008,"15min":9.8190000000000008},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.91700000000000004}]},{"osd":7,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":9.625,"5min":9.625,"15min":9.625},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":9.8949999999999996,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":1.026}]}]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8472,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363368,"statfs":{"total":96636764160,"available":96628088832,"internally_reserved":0,"allocated":409600,"data_stored":167017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":7.1799999999999997,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.51300000000000001}]},{"osd":1,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":9.5709999999999997,"5min":9.5709999999999997,"15min":9.5709999999999997},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":5.266,"5min":5.266,"15min":5.266},"last":0.70399999999999996}]},{"osd":2,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":8.4529999999999994,"5min":8.4529999999999994,"15min":8.4529999999999994},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":6.6829999999999998,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.84999999999999998}]},{"osd":5,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.748}]},{"osd":6,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.82499999999999996}]},{"osd":7,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.92900000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8340,"kb_used_data":396,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363500,"statfs":{"total":96636764160,"available":96628224000,"internally_reserved":0,"allocated":405504,"data_stored":165998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69699999999999995}]},{"osd":1,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.76400000000000001}]},{"osd":2,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.72599999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.63200000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.80000000000000004}]},{"osd":6,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.85099999999999998}]},{"osd":7,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.747},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.82899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T07:57:22.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T07:57:23.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.115+0000 7f7bb1fff700 1 Processor -- start 2023-03-03T07:57:23.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.115+0000 7f7bb1fff700 1 -- start start 2023-03-03T07:57:23.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.116+0000 7f7bb1fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7bac15cc00 0x7f7bac0cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:23.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.117+0000 7f7bb1fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bac0d27c0 0x7f7bac0d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:23.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.117+0000 7f7bb1fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7bac0b60d0 0x7f7bac0b6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:23.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.117+0000 7f7bb1fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bac0ce040 con 0x7f7bac15cc00 2023-03-03T07:57:23.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.117+0000 7f7bb1fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bac073be0 con 0x7f7bac0b60d0 2023-03-03T07:57:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.117+0000 7f7bb1fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac0cde10 con 0x7f7bac0d27c0 2023-03-03T07:57:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.117+0000 7f7bab7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7bac15cc00 0x7f7bac0cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.117+0000 7f7bab7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7bac15cc00 0x7f7bac0cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45580/0 (socket says 172.21.15.98:45580) 2023-03-03T07:57:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.117+0000 7f7bab7fe700 1 -- 172.21.15.98:0/805816596 learned_addr learned my addr 172.21.15.98:0/805816596 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.117+0000 7f7baaffd700 1 --2- 172.21.15.98:0/805816596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bac0d27c0 0x7f7bac0d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.117+0000 7f7babfff700 1 --2- 172.21.15.98:0/805816596 >> v2:172.21.15.100:3300/0 conn(0x7f7bac0b60d0 0x7f7bac0b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.117+0000 7f7bab7fe700 1 -- 172.21.15.98:0/805816596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bac0d27c0 msgr2=0x7f7bac0d2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.118+0000 7f7bab7fe700 1 --2- 172.21.15.98:0/805816596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bac0d27c0 0x7f7bac0d2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.118+0000 7f7bab7fe700 1 -- 172.21.15.98:0/805816596 >> v2:172.21.15.100:3300/0 conn(0x7f7bac0b60d0 msgr2=0x7f7bac0b6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.118+0000 7f7bab7fe700 1 --2- 172.21.15.98:0/805816596 >> v2:172.21.15.100:3300/0 conn(0x7f7bac0b60d0 0x7f7bac0b6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.118+0000 7f7bab7fe700 1 -- 172.21.15.98:0/805816596 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bac0d1a00 con 0x7f7bac15cc00 2023-03-03T07:57:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.118+0000 7f7babfff700 1 --2- 172.21.15.98:0/805816596 >> v2:172.21.15.100:3300/0 conn(0x7f7bac0b60d0 0x7f7bac0b6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.118+0000 7f7bab7fe700 1 --2- 172.21.15.98:0/805816596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bac15cc00 0x7f7bac0cd8d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7bac06bf10 tx=0x7f7b9c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1beca7d8ac4cee06 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.118+0000 7f7baa7fc700 1 -- 172.21.15.98:0/805816596 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c011830 con 0x7f7bac15cc00 2023-03-03T07:57:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.118+0000 7f7baa7fc700 1 -- 172.21.15.98:0/805816596 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9c011990 con 0x7f7bac15cc00 2023-03-03T07:57:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.118+0000 7f7baa7fc700 1 -- 172.21.15.98:0/805816596 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c011d20 con 0x7f7bac15cc00 2023-03-03T07:57:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.119+0000 7f7bb1fff700 1 -- 172.21.15.98:0/805816596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bac15cc00 msgr2=0x7f7bac0cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.119+0000 7f7bb1fff700 1 --2- 172.21.15.98:0/805816596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bac15cc00 0x7f7bac0cd8d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7bac06bf10 tx=0x7f7b9c009bd0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.119+0000 7f7bb1fff700 1 -- 172.21.15.98:0/805816596 shutdown_connections 2023-03-03T07:57:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.119+0000 7f7bb1fff700 1 --2- 172.21.15.98:0/805816596 >> v2:172.21.15.100:3300/0 conn(0x7f7bac0b60d0 0x7f7bac0b6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.119+0000 7f7bb1fff700 1 --2- 172.21.15.98:0/805816596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bac15cc00 0x7f7bac0cd8d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.119+0000 7f7bb1fff700 1 --2- 172.21.15.98:0/805816596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bac0d27c0 0x7f7bac0d2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.119+0000 7f7bb1fff700 1 -- 172.21.15.98:0/805816596 >> 172.21.15.98:0/805816596 conn(0x7f7bac06aee0 msgr2=0x7f7bac06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.119+0000 7f7bb1fff700 1 -- 172.21.15.98:0/805816596 shutdown_connections 2023-03-03T07:57:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.119+0000 7f7bb1fff700 1 -- 172.21.15.98:0/805816596 wait complete. 2023-03-03T07:57:23.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.121+0000 7f7bb1fff700 1 Processor -- start 2023-03-03T07:57:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.121+0000 7f7bb1fff700 1 -- start start 2023-03-03T07:57:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.122+0000 7f7bb1fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bac0b60d0 0x7f7bac1f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.122+0000 7f7bb1fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bac0d27c0 0x7f7bac15da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.122+0000 7f7bb1fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7bac15cc00 0x7f7bac0ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.122+0000 7f7bb1fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac0b3e60 con 0x7f7bac0b60d0 2023-03-03T07:57:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.122+0000 7f7bb1fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac15ed30 con 0x7f7bac15cc00 2023-03-03T07:57:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.122+0000 7f7bb1fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac15efa0 con 0x7f7bac0d27c0 2023-03-03T07:57:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.123+0000 7f7bab7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bac0b60d0 0x7f7bac1f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.123+0000 7f7bab7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bac0b60d0 0x7f7bac1f55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45596/0 (socket says 172.21.15.98:45596) 2023-03-03T07:57:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.123+0000 7f7bab7fe700 1 -- 172.21.15.98:0/2875537267 learned_addr learned my addr 172.21.15.98:0/2875537267 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.123+0000 7f7babfff700 1 --2- 172.21.15.98:0/2875537267 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7bac15cc00 0x7f7bac0ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.123+0000 7f7baaffd700 1 --2- 172.21.15.98:0/2875537267 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bac0d27c0 0x7f7bac15da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.123+0000 7f7bab7fe700 1 -- 172.21.15.98:0/2875537267 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bac0d27c0 msgr2=0x7f7bac15da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.123+0000 7f7bab7fe700 1 --2- 172.21.15.98:0/2875537267 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bac0d27c0 0x7f7bac15da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.123+0000 7f7bab7fe700 1 -- 172.21.15.98:0/2875537267 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7bac15cc00 msgr2=0x7f7bac0ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.123+0000 7f7bab7fe700 1 --2- 172.21.15.98:0/2875537267 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7bac15cc00 0x7f7bac0ca350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.123+0000 7f7bab7fe700 1 -- 172.21.15.98:0/2875537267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c009550 con 0x7f7bac0b60d0 2023-03-03T07:57:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.123+0000 7f7baaffd700 1 --2- 172.21.15.98:0/2875537267 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bac0d27c0 0x7f7bac15da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:57:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.123+0000 7f7bab7fe700 1 --2- 172.21.15.98:0/2875537267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bac0b60d0 0x7f7bac1f55a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c009c10 tx=0x7f7b9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.124+0000 7f7b93fff700 1 -- 172.21.15.98:0/2875537267 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c004380 con 0x7f7bac0b60d0 2023-03-03T07:57:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.124+0000 7f7b93fff700 1 -- 172.21.15.98:0/2875537267 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9c01f070 con 0x7f7bac0b60d0 2023-03-03T07:57:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.124+0000 7f7b93fff700 1 -- 172.21.15.98:0/2875537267 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c0036a0 con 0x7f7bac0b60d0 2023-03-03T07:57:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.124+0000 7f7bb1fff700 1 -- 172.21.15.98:0/2875537267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bac0d1fc0 con 0x7f7bac0b60d0 2023-03-03T07:57:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.124+0000 7f7bb1fff700 1 -- 172.21.15.98:0/2875537267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bac0d1ce0 con 0x7f7bac0b60d0 2023-03-03T07:57:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.125+0000 7f7bb1fff700 1 -- 172.21.15.98:0/2875537267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bac0d22a0 con 0x7f7bac0b60d0 2023-03-03T07:57:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.126+0000 7f7b93fff700 1 -- 172.21.15.98:0/2875537267 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7b9c02a020 con 0x7f7bac0b60d0 2023-03-03T07:57:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.128+0000 7f7b93fff700 1 --2- 172.21.15.98:0/2875537267 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7b980682f0 0x7f7b9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.128+0000 7f7b93fff700 1 -- 172.21.15.98:0/2875537267 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7b9c087230 con 0x7f7bac0b60d0 2023-03-03T07:57:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.128+0000 7f7baaffd700 1 --2- 172.21.15.98:0/2875537267 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7b980682f0 0x7f7b9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.129+0000 7f7baaffd700 1 --2- 172.21.15.98:0/2875537267 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7b980682f0 0x7f7b9806a7a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0b6eb0 tx=0x7f7ba0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.131+0000 7f7b93fff700 1 -- 172.21.15.98:0/2875537267 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7b9c0539a0 con 0x7f7bac0b60d0 2023-03-03T07:57:23.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.322+0000 7f7bb1fff700 1 -- 172.21.15.98:0/2875537267 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7bac0729b0 con 0x7f7b980682f0 2023-03-03T07:57:23.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.331+0000 7f7b93fff700 1 -- 172.21.15.98:0/2875537267 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+307606 (secure 0 0 0) 0x7f7bac0729b0 con 0x7f7b980682f0 2023-03-03T07:57:23.337 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:23.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.337+0000 7f7bb1fff700 1 -- 172.21.15.98:0/2875537267 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7b980682f0 msgr2=0x7f7b9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:23.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.337+0000 7f7bb1fff700 1 --2- 172.21.15.98:0/2875537267 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7b980682f0 0x7f7b9806a7a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0b6eb0 tx=0x7f7ba0008040 comp rx=0 tx=0).stop 2023-03-03T07:57:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.337+0000 7f7bb1fff700 1 -- 172.21.15.98:0/2875537267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bac0b60d0 msgr2=0x7f7bac1f55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.337+0000 7f7bb1fff700 1 --2- 172.21.15.98:0/2875537267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bac0b60d0 0x7f7bac1f55a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c009c10 tx=0x7f7b9c004750 comp rx=0 tx=0).stop 2023-03-03T07:57:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.337+0000 7f7bb1fff700 1 -- 172.21.15.98:0/2875537267 shutdown_connections 2023-03-03T07:57:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.337+0000 7f7bb1fff700 1 --2- 172.21.15.98:0/2875537267 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7b980682f0 0x7f7b9806a7a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.337+0000 7f7bb1fff700 1 --2- 172.21.15.98:0/2875537267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bac0b60d0 0x7f7bac1f55a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.337+0000 7f7bb1fff700 1 --2- 172.21.15.98:0/2875537267 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7bac15cc00 0x7f7bac0ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.337+0000 7f7bb1fff700 1 --2- 172.21.15.98:0/2875537267 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7bac0d27c0 0x7f7bac15da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.337+0000 7f7bb1fff700 1 -- 172.21.15.98:0/2875537267 >> 172.21.15.98:0/2875537267 conn(0x7f7bac06aee0 msgr2=0x7f7bac0b7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:23.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.337+0000 7f7bb1fff700 1 -- 172.21.15.98:0/2875537267 shutdown_connections 2023-03-03T07:57:23.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.337+0000 7f7bb1fff700 1 -- 172.21.15.98:0/2875537267 wait complete. 2023-03-03T07:57:23.342 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T07:57:23.366 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2023-03-03T07:57:22.197578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":46,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68656,"kb_used_data":4272,"kb_used_omap":0,"kb_used_meta":64320,"kb_avail":754906064,"statfs":{"total":773094113280,"available":773023809536,"internally_reserved":0,"allocated":4374528,"data_stored":2473028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65863680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000901"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.748481+0000","last_change":"2023-03-03T07:57:16.748481+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.748481+0000","last_undegraded":"2023-03-03T07:57:16.748481+0000","last_fullsized":"2023-03-03T07:57:16.748481+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:39:27.674460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746045+0000","last_change":"2023-03-03T07:57:16.746045+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746045+0000","last_undegraded":"2023-03-03T07:57:16.746045+0000","last_fullsized":"2023-03-03T07:57:16.746045+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.739842+0000","last_change":"2023-03-03T07:57:16.739842+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.739842+0000","last_undegraded":"2023-03-03T07:57:16.739842+0000","last_fullsized":"2023-03-03T07:57:16.739842+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.740784+0000","last_change":"2023-03-03T07:57:16.740784+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.740784+0000","last_undegraded":"2023-03-03T07:57:16.740784+0000","last_fullsized":"2023-03-03T07:57:16.740784+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:07.498551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741327+0000","last_change":"2023-03-03T07:57:16.741327+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741327+0000","last_undegraded":"2023-03-03T07:57:16.741327+0000","last_fullsized":"2023-03-03T07:57:16.741327+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:37:50.701337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738165+0000","last_change":"2023-03-03T07:57:16.738165+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738165+0000","last_undegraded":"2023-03-03T07:57:16.738165+0000","last_fullsized":"2023-03-03T07:57:16.738165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:25.355832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.749480+0000","last_change":"2023-03-03T07:57:16.749480+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.749480+0000","last_undegraded":"2023-03-03T07:57:16.749480+0000","last_fullsized":"2023-03-03T07:57:16.749480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:44.621837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747051+0000","last_change":"2023-03-03T07:57:16.747051+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747051+0000","last_undegraded":"2023-03-03T07:57:16.747051+0000","last_fullsized":"2023-03-03T07:57:16.747051+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:01.871020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.739643+0000","last_change":"2023-03-03T07:57:16.739643+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.739643+0000","last_undegraded":"2023-03-03T07:57:16.739643+0000","last_fullsized":"2023-03-03T07:57:16.739643+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747834+0000","last_change":"2023-03-03T07:57:16.747834+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747834+0000","last_undegraded":"2023-03-03T07:57:16.747834+0000","last_fullsized":"2023-03-03T07:57:16.747834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.748481+0000","last_change":"2023-03-03T07:57:16.748481+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.748481+0000","last_undegraded":"2023-03-03T07:57:16.748481+0000","last_fullsized":"2023-03-03T07:57:16.748481+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.742043+0000","last_change":"2023-03-03T07:57:16.742043+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.742043+0000","last_undegraded":"2023-03-03T07:57:16.742043+0000","last_fullsized":"2023-03-03T07:57:16.742043+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746702+0000","last_change":"2023-03-03T07:57:16.746702+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746702+0000","last_undegraded":"2023-03-03T07:57:16.746702+0000","last_fullsized":"2023-03-03T07:57:16.746702+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.730661+0000","last_change":"2023-03-03T07:57:16.730661+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.730661+0000","last_undegraded":"2023-03-03T07:57:16.730661+0000","last_fullsized":"2023-03-03T07:57:16.730661+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:48:58.988637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747212+0000","last_change":"2023-03-03T07:57:16.747212+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747212+0000","last_undegraded":"2023-03-03T07:57:16.747212+0000","last_fullsized":"2023-03-03T07:57:16.747212+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:24:04.359311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746985+0000","last_change":"2023-03-03T07:57:16.746985+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746985+0000","last_undegraded":"2023-03-03T07:57:16.746985+0000","last_fullsized":"2023-03-03T07:57:16.746985+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738184+0000","last_change":"2023-03-03T07:57:16.738184+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738184+0000","last_undegraded":"2023-03-03T07:57:16.738184+0000","last_fullsized":"2023-03-03T07:57:16.738184+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737683+0000","last_change":"2023-03-03T07:57:16.737683+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737683+0000","last_undegraded":"2023-03-03T07:57:16.737683+0000","last_fullsized":"2023-03-03T07:57:16.737683+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:56:10.139056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.740159+0000","last_change":"2023-03-03T07:57:16.740159+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.740159+0000","last_undegraded":"2023-03-03T07:57:16.740159+0000","last_fullsized":"2023-03-03T07:57:16.740159+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:00:23.450170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737107+0000","last_change":"2023-03-03T07:57:16.737107+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737107+0000","last_undegraded":"2023-03-03T07:57:16.737107+0000","last_fullsized":"2023-03-03T07:57:16.737107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:15.531132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746180+0000","last_change":"2023-03-03T07:57:16.746180+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746180+0000","last_undegraded":"2023-03-03T07:57:16.746180+0000","last_fullsized":"2023-03-03T07:57:16.746180+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.732520+0000","last_change":"2023-03-03T07:57:16.732520+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.732520+0000","last_undegraded":"2023-03-03T07:57:16.732520+0000","last_fullsized":"2023-03-03T07:57:16.732520+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:39:27.674460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.739235+0000","last_change":"2023-03-03T07:57:16.739235+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.739235+0000","last_undegraded":"2023-03-03T07:57:16.739235+0000","last_fullsized":"2023-03-03T07:57:16.739235+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738079+0000","last_change":"2023-03-03T07:57:16.738079+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738079+0000","last_undegraded":"2023-03-03T07:57:16.738079+0000","last_fullsized":"2023-03-03T07:57:16.738079+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:57:04.542617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737878+0000","last_change":"2023-03-03T07:57:16.737878+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737878+0000","last_undegraded":"2023-03-03T07:57:16.737878+0000","last_fullsized":"2023-03-03T07:57:16.737878+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:57:04.542617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.736707+0000","last_change":"2023-03-03T07:57:16.736707+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.736707+0000","last_undegraded":"2023-03-03T07:57:16.736707+0000","last_fullsized":"2023-03-03T07:57:16.736707+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.736973+0000","last_change":"2023-03-03T07:57:16.736973+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.736973+0000","last_undegraded":"2023-03-03T07:57:16.736973+0000","last_fullsized":"2023-03-03T07:57:16.736973+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:20.464414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746809+0000","last_change":"2023-03-03T07:57:16.746809+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746809+0000","last_undegraded":"2023-03-03T07:57:16.746809+0000","last_fullsized":"2023-03-03T07:57:16.746809+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.736370+0000","last_change":"2023-03-03T07:57:16.736370+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.736370+0000","last_undegraded":"2023-03-03T07:57:16.736370+0000","last_fullsized":"2023-03-03T07:57:16.736370+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:07.498551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.739548+0000","last_change":"2023-03-03T07:57:16.739548+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.739548+0000","last_undegraded":"2023-03-03T07:57:16.739548+0000","last_fullsized":"2023-03-03T07:57:16.739548+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:43.532953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738752+0000","last_change":"2023-03-03T07:57:16.738752+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738752+0000","last_undegraded":"2023-03-03T07:57:16.738752+0000","last_fullsized":"2023-03-03T07:57:16.738752+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737178+0000","last_change":"2023-03-03T07:57:16.737178+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737178+0000","last_undegraded":"2023-03-03T07:57:16.737178+0000","last_fullsized":"2023-03-03T07:57:16.737178+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.655223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.742046+0000","last_change":"2023-03-03T07:57:16.742046+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.742046+0000","last_undegraded":"2023-03-03T07:57:16.742046+0000","last_fullsized":"2023-03-03T07:57:16.742046+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.749016+0000","last_change":"2023-03-03T07:57:16.745475+0000","last_active":"2023-03-03T07:57:16.749016+0000","last_peered":"2023-03-03T07:57:16.749016+0000","last_clean":"2023-03-03T07:57:16.749016+0000","last_became_active":"2023-03-03T07:57:16.745159+0000","last_became_peered":"2023-03-03T07:57:16.745159+0000","last_unstale":"2023-03-03T07:57:16.749016+0000","last_undegraded":"2023-03-03T07:57:16.749016+0000","last_fullsized":"2023-03-03T07:57:16.749016+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:45:49.372030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738564+0000","last_change":"2023-03-03T07:57:16.738564+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738564+0000","last_undegraded":"2023-03-03T07:57:16.738564+0000","last_fullsized":"2023-03-03T07:57:16.738564+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.758087+0000","last_change":"2023-03-03T07:57:16.755153+0000","last_active":"2023-03-03T07:57:16.758087+0000","last_peered":"2023-03-03T07:57:16.758087+0000","last_clean":"2023-03-03T07:57:16.758087+0000","last_became_active":"2023-03-03T07:57:16.754738+0000","last_became_peered":"2023-03-03T07:57:16.754738+0000","last_unstale":"2023-03-03T07:57:16.758087+0000","last_undegraded":"2023-03-03T07:57:16.758087+0000","last_fullsized":"2023-03-03T07:57:16.758087+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:14.522208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.731582+0000","last_change":"2023-03-03T07:57:16.731582+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.731582+0000","last_undegraded":"2023-03-03T07:57:16.731582+0000","last_fullsized":"2023-03-03T07:57:16.731582+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:11.090994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.751260+0000","last_change":"2023-03-03T07:57:16.746651+0000","last_active":"2023-03-03T07:57:16.751260+0000","last_peered":"2023-03-03T07:57:16.751260+0000","last_clean":"2023-03-03T07:57:16.751260+0000","last_became_active":"2023-03-03T07:57:16.746390+0000","last_became_peered":"2023-03-03T07:57:16.746390+0000","last_unstale":"2023-03-03T07:57:16.751260+0000","last_undegraded":"2023-03-03T07:57:16.751260+0000","last_fullsized":"2023-03-03T07:57:16.751260+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:06.489627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747957+0000","last_change":"2023-03-03T07:57:16.747957+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747957+0000","last_undegraded":"2023-03-03T07:57:16.747957+0000","last_fullsized":"2023-03-03T07:57:16.747957+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:11.090994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754237+0000","last_change":"2023-03-03T07:57:16.743191+0000","last_active":"2023-03-03T07:57:16.754237+0000","last_peered":"2023-03-03T07:57:16.754237+0000","last_clean":"2023-03-03T07:57:16.754237+0000","last_became_active":"2023-03-03T07:57:16.741858+0000","last_became_peered":"2023-03-03T07:57:16.741858+0000","last_unstale":"2023-03-03T07:57:16.754237+0000","last_undegraded":"2023-03-03T07:57:16.754237+0000","last_fullsized":"2023-03-03T07:57:16.754237+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:00.944189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741626+0000","last_change":"2023-03-03T07:57:16.741626+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741626+0000","last_undegraded":"2023-03-03T07:57:16.741626+0000","last_fullsized":"2023-03-03T07:57:16.741626+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754833+0000","last_change":"2023-03-03T07:57:16.753759+0000","last_active":"2023-03-03T07:57:16.754833+0000","last_peered":"2023-03-03T07:57:16.754833+0000","last_clean":"2023-03-03T07:57:16.754833+0000","last_became_active":"2023-03-03T07:57:16.753449+0000","last_became_peered":"2023-03-03T07:57:16.753449+0000","last_unstale":"2023-03-03T07:57:16.754833+0000","last_undegraded":"2023-03-03T07:57:16.754833+0000","last_fullsized":"2023-03-03T07:57:16.754833+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:56:09.130132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746580+0000","last_change":"2023-03-03T07:57:16.746580+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746580+0000","last_undegraded":"2023-03-03T07:57:16.746580+0000","last_fullsized":"2023-03-03T07:57:16.746580+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.746943+0000","last_change":"2023-03-03T07:57:16.740511+0000","last_active":"2023-03-03T07:57:16.746943+0000","last_peered":"2023-03-03T07:57:16.746943+0000","last_clean":"2023-03-03T07:57:16.746943+0000","last_became_active":"2023-03-03T07:57:16.740221+0000","last_became_peered":"2023-03-03T07:57:16.740221+0000","last_unstale":"2023-03-03T07:57:16.746943+0000","last_undegraded":"2023-03-03T07:57:16.746943+0000","last_fullsized":"2023-03-03T07:57:16.746943+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:10.938115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.731963+0000","last_change":"2023-03-03T07:57:16.731963+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.731963+0000","last_undegraded":"2023-03-03T07:57:16.731963+0000","last_fullsized":"2023-03-03T07:57:16.731963+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.760256+0000","last_change":"2023-03-03T07:57:16.756171+0000","last_active":"2023-03-03T07:57:16.760256+0000","last_peered":"2023-03-03T07:57:16.760256+0000","last_clean":"2023-03-03T07:57:16.760256+0000","last_became_active":"2023-03-03T07:57:16.755894+0000","last_became_peered":"2023-03-03T07:57:16.755894+0000","last_unstale":"2023-03-03T07:57:16.760256+0000","last_undegraded":"2023-03-03T07:57:16.760256+0000","last_fullsized":"2023-03-03T07:57:16.760256+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:19.455490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747501+0000","last_change":"2023-03-03T07:57:16.747501+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747501+0000","last_undegraded":"2023-03-03T07:57:16.747501+0000","last_fullsized":"2023-03-03T07:57:16.747501+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:48:58.988637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754386+0000","last_change":"2023-03-03T07:57:16.744111+0000","last_active":"2023-03-03T07:57:16.754386+0000","last_peered":"2023-03-03T07:57:16.754386+0000","last_clean":"2023-03-03T07:57:16.754386+0000","last_became_active":"2023-03-03T07:57:16.743778+0000","last_became_peered":"2023-03-03T07:57:16.743778+0000","last_unstale":"2023-03-03T07:57:16.754386+0000","last_undegraded":"2023-03-03T07:57:16.754386+0000","last_fullsized":"2023-03-03T07:57:16.754386+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:52:59.206045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738898+0000","last_change":"2023-03-03T07:57:16.738898+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738898+0000","last_undegraded":"2023-03-03T07:57:16.738898+0000","last_fullsized":"2023-03-03T07:57:16.738898+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:11.947039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.752195+0000","last_change":"2023-03-03T07:57:16.744161+0000","last_active":"2023-03-03T07:57:16.752195+0000","last_peered":"2023-03-03T07:57:16.752195+0000","last_clean":"2023-03-03T07:57:16.752195+0000","last_became_active":"2023-03-03T07:57:16.743909+0000","last_became_peered":"2023-03-03T07:57:16.743909+0000","last_unstale":"2023-03-03T07:57:16.752195+0000","last_undegraded":"2023-03-03T07:57:16.752195+0000","last_fullsized":"2023-03-03T07:57:16.752195+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:43:42.175275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741492+0000","last_change":"2023-03-03T07:57:16.741492+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741492+0000","last_undegraded":"2023-03-03T07:57:16.741492+0000","last_fullsized":"2023-03-03T07:57:16.741492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.756982+0000","last_change":"2023-03-03T07:57:16.754673+0000","last_active":"2023-03-03T07:57:16.756982+0000","last_peered":"2023-03-03T07:57:16.756982+0000","last_clean":"2023-03-03T07:57:16.756982+0000","last_became_active":"2023-03-03T07:57:16.754415+0000","last_became_peered":"2023-03-03T07:57:16.754415+0000","last_unstale":"2023-03-03T07:57:16.756982+0000","last_undegraded":"2023-03-03T07:57:16.756982+0000","last_fullsized":"2023-03-03T07:57:16.756982+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:06.489627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.742445+0000","last_change":"2023-03-03T07:57:16.742445+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.742445+0000","last_undegraded":"2023-03-03T07:57:16.742445+0000","last_fullsized":"2023-03-03T07:57:16.742445+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.762321+0000","last_change":"2023-03-03T07:57:16.757208+0000","last_active":"2023-03-03T07:57:16.762321+0000","last_peered":"2023-03-03T07:57:16.762321+0000","last_clean":"2023-03-03T07:57:16.762321+0000","last_became_active":"2023-03-03T07:57:16.756910+0000","last_became_peered":"2023-03-03T07:57:16.756910+0000","last_unstale":"2023-03-03T07:57:16.762321+0000","last_undegraded":"2023-03-03T07:57:16.762321+0000","last_fullsized":"2023-03-03T07:57:16.762321+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:14.522208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.731414+0000","last_change":"2023-03-03T07:57:16.731414+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.731414+0000","last_undegraded":"2023-03-03T07:57:16.731414+0000","last_fullsized":"2023-03-03T07:57:16.731414+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:01.871020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.755113+0000","last_change":"2023-03-03T07:57:16.749713+0000","last_active":"2023-03-03T07:57:16.755113+0000","last_peered":"2023-03-03T07:57:16.755113+0000","last_clean":"2023-03-03T07:57:16.755113+0000","last_became_active":"2023-03-03T07:57:16.749304+0000","last_became_peered":"2023-03-03T07:57:16.749304+0000","last_unstale":"2023-03-03T07:57:16.755113+0000","last_undegraded":"2023-03-03T07:57:16.755113+0000","last_fullsized":"2023-03-03T07:57:16.755113+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:19:07.613725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.740047+0000","last_change":"2023-03-03T07:57:16.740047+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.740047+0000","last_undegraded":"2023-03-03T07:57:16.740047+0000","last_fullsized":"2023-03-03T07:57:16.740047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:21:12.720267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.756281+0000","last_change":"2023-03-03T07:57:16.753448+0000","last_active":"2023-03-03T07:57:16.756281+0000","last_peered":"2023-03-03T07:57:16.756281+0000","last_clean":"2023-03-03T07:57:16.756281+0000","last_became_active":"2023-03-03T07:57:16.753183+0000","last_became_peered":"2023-03-03T07:57:16.753183+0000","last_unstale":"2023-03-03T07:57:16.756281+0000","last_undegraded":"2023-03-03T07:57:16.756281+0000","last_fullsized":"2023-03-03T07:57:16.756281+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:54.897881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.745842+0000","last_change":"2023-03-03T07:57:16.745842+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.745842+0000","last_undegraded":"2023-03-03T07:57:16.745842+0000","last_fullsized":"2023-03-03T07:57:16.745842+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:24:04.359311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.760221+0000","last_change":"2023-03-03T07:57:16.754628+0000","last_active":"2023-03-03T07:57:16.760221+0000","last_peered":"2023-03-03T07:57:16.760221+0000","last_clean":"2023-03-03T07:57:16.760221+0000","last_became_active":"2023-03-03T07:57:16.754370+0000","last_became_peered":"2023-03-03T07:57:16.754370+0000","last_unstale":"2023-03-03T07:57:16.760221+0000","last_undegraded":"2023-03-03T07:57:16.760221+0000","last_fullsized":"2023-03-03T07:57:16.760221+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:14.522208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.739793+0000","last_change":"2023-03-03T07:57:16.739793+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.739793+0000","last_undegraded":"2023-03-03T07:57:16.739793+0000","last_fullsized":"2023-03-03T07:57:16.739793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:21:12.720267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.747551+0000","last_change":"2023-03-03T07:57:16.745348+0000","last_active":"2023-03-03T07:57:16.747551+0000","last_peered":"2023-03-03T07:57:16.747551+0000","last_clean":"2023-03-03T07:57:16.747551+0000","last_became_active":"2023-03-03T07:57:16.745100+0000","last_became_peered":"2023-03-03T07:57:16.745100+0000","last_unstale":"2023-03-03T07:57:16.747551+0000","last_undegraded":"2023-03-03T07:57:16.747551+0000","last_fullsized":"2023-03-03T07:57:16.747551+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:17:28.990710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746997+0000","last_change":"2023-03-03T07:57:16.746997+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746997+0000","last_undegraded":"2023-03-03T07:57:16.746997+0000","last_fullsized":"2023-03-03T07:57:16.746997+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:35:32.174190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.743384+0000","last_change":"2023-03-03T07:57:16.736066+0000","last_active":"2023-03-03T07:57:16.743384+0000","last_peered":"2023-03-03T07:57:16.743384+0000","last_clean":"2023-03-03T07:57:16.743384+0000","last_became_active":"2023-03-03T07:57:16.735650+0000","last_became_peered":"2023-03-03T07:57:16.735650+0000","last_unstale":"2023-03-03T07:57:16.743384+0000","last_undegraded":"2023-03-03T07:57:16.743384+0000","last_fullsized":"2023-03-03T07:57:16.743384+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:00:22.441246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.745199+0000","last_change":"2023-03-03T07:57:16.745199+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.745199+0000","last_undegraded":"2023-03-03T07:57:16.745199+0000","last_fullsized":"2023-03-03T07:57:16.745199+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:44.621837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.762270+0000","last_change":"2023-03-03T07:57:16.756645+0000","last_active":"2023-03-03T07:57:16.762270+0000","last_peered":"2023-03-03T07:57:16.762270+0000","last_clean":"2023-03-03T07:57:16.762270+0000","last_became_active":"2023-03-03T07:57:16.756201+0000","last_became_peered":"2023-03-03T07:57:16.756201+0000","last_unstale":"2023-03-03T07:57:16.762270+0000","last_undegraded":"2023-03-03T07:57:16.762270+0000","last_fullsized":"2023-03-03T07:57:16.762270+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:42.524029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.747344+0000","last_change":"2023-03-03T07:57:16.747344+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.747344+0000","last_undegraded":"2023-03-03T07:57:16.747344+0000","last_fullsized":"2023-03-03T07:57:16.747344+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:13:17.532289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754469+0000","last_change":"2023-03-03T07:57:16.747099+0000","last_active":"2023-03-03T07:57:16.754469+0000","last_peered":"2023-03-03T07:57:16.754469+0000","last_clean":"2023-03-03T07:57:16.754469+0000","last_became_active":"2023-03-03T07:57:16.746766+0000","last_became_peered":"2023-03-03T07:57:16.746766+0000","last_unstale":"2023-03-03T07:57:16.754469+0000","last_undegraded":"2023-03-03T07:57:16.754469+0000","last_fullsized":"2023-03-03T07:57:16.754469+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:25.646299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.731014+0000","last_change":"2023-03-03T07:57:16.731014+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.731014+0000","last_undegraded":"2023-03-03T07:57:16.731014+0000","last_fullsized":"2023-03-03T07:57:16.731014+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:35:32.174190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.753777+0000","last_change":"2023-03-03T07:57:16.748443+0000","last_active":"2023-03-03T07:57:16.753777+0000","last_peered":"2023-03-03T07:57:16.753777+0000","last_clean":"2023-03-03T07:57:16.753777+0000","last_became_active":"2023-03-03T07:57:16.748172+0000","last_became_peered":"2023-03-03T07:57:16.748172+0000","last_unstale":"2023-03-03T07:57:16.753777+0000","last_undegraded":"2023-03-03T07:57:16.753777+0000","last_fullsized":"2023-03-03T07:57:16.753777+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:07:16.498493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.738225+0000","last_change":"2023-03-03T07:57:16.738225+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.738225+0000","last_undegraded":"2023-03-03T07:57:16.738225+0000","last_fullsized":"2023-03-03T07:57:16.738225+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:43.532953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.751352+0000","last_change":"2023-03-03T07:57:16.746019+0000","last_active":"2023-03-03T07:57:16.751352+0000","last_peered":"2023-03-03T07:57:16.751352+0000","last_clean":"2023-03-03T07:57:16.751352+0000","last_became_active":"2023-03-03T07:57:16.745766+0000","last_became_peered":"2023-03-03T07:57:16.745766+0000","last_unstale":"2023-03-03T07:57:16.751352+0000","last_undegraded":"2023-03-03T07:57:16.751352+0000","last_fullsized":"2023-03-03T07:57:16.751352+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:42.524029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.743223+0000","last_change":"2023-03-03T07:57:16.743223+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.743223+0000","last_undegraded":"2023-03-03T07:57:16.743223+0000","last_fullsized":"2023-03-03T07:57:16.743223+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.756514+0000","last_change":"2023-03-03T07:57:16.753514+0000","last_active":"2023-03-03T07:57:16.756514+0000","last_peered":"2023-03-03T07:57:16.756514+0000","last_clean":"2023-03-03T07:57:16.756514+0000","last_became_active":"2023-03-03T07:57:16.753206+0000","last_became_peered":"2023-03-03T07:57:16.753206+0000","last_unstale":"2023-03-03T07:57:16.756514+0000","last_undegraded":"2023-03-03T07:57:16.756514+0000","last_fullsized":"2023-03-03T07:57:16.756514+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:43:26.659371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741849+0000","last_change":"2023-03-03T07:57:16.741849+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741849+0000","last_undegraded":"2023-03-03T07:57:16.741849+0000","last_fullsized":"2023-03-03T07:57:16.741849+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.758038+0000","last_change":"2023-03-03T07:57:16.754775+0000","last_active":"2023-03-03T07:57:16.758038+0000","last_peered":"2023-03-03T07:57:16.758038+0000","last_clean":"2023-03-03T07:57:16.758038+0000","last_became_active":"2023-03-03T07:57:16.754315+0000","last_became_peered":"2023-03-03T07:57:16.754315+0000","last_unstale":"2023-03-03T07:57:16.758038+0000","last_undegraded":"2023-03-03T07:57:16.758038+0000","last_fullsized":"2023-03-03T07:57:16.758038+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:14:32.071416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746117+0000","last_change":"2023-03-03T07:57:16.746117+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746117+0000","last_undegraded":"2023-03-03T07:57:16.746117+0000","last_fullsized":"2023-03-03T07:57:16.746117+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.747475+0000","last_change":"2023-03-03T07:57:16.742812+0000","last_active":"2023-03-03T07:57:16.747475+0000","last_peered":"2023-03-03T07:57:16.747475+0000","last_clean":"2023-03-03T07:57:16.747475+0000","last_became_active":"2023-03-03T07:57:16.742529+0000","last_became_peered":"2023-03-03T07:57:16.742529+0000","last_unstale":"2023-03-03T07:57:16.747475+0000","last_undegraded":"2023-03-03T07:57:16.747475+0000","last_fullsized":"2023-03-03T07:57:16.747475+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:37:49.692413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.746443+0000","last_change":"2023-03-03T07:57:16.746443+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.746443+0000","last_undegraded":"2023-03-03T07:57:16.746443+0000","last_fullsized":"2023-03-03T07:57:16.746443+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.756542+0000","last_change":"2023-03-03T07:57:16.753472+0000","last_active":"2023-03-03T07:57:16.756542+0000","last_peered":"2023-03-03T07:57:16.756542+0000","last_clean":"2023-03-03T07:57:16.756542+0000","last_became_active":"2023-03-03T07:57:16.753183+0000","last_became_peered":"2023-03-03T07:57:16.753183+0000","last_unstale":"2023-03-03T07:57:16.756542+0000","last_undegraded":"2023-03-03T07:57:16.756542+0000","last_fullsized":"2023-03-03T07:57:16.756542+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:53.094916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741199+0000","last_change":"2023-03-03T07:57:16.741199+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741199+0000","last_undegraded":"2023-03-03T07:57:16.741199+0000","last_fullsized":"2023-03-03T07:57:16.741199+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.756071+0000","last_change":"2023-03-03T07:57:16.751997+0000","last_active":"2023-03-03T07:57:16.756071+0000","last_peered":"2023-03-03T07:57:16.756071+0000","last_clean":"2023-03-03T07:57:16.756071+0000","last_became_active":"2023-03-03T07:57:16.751593+0000","last_became_peered":"2023-03-03T07:57:16.751593+0000","last_unstale":"2023-03-03T07:57:16.756071+0000","last_undegraded":"2023-03-03T07:57:16.756071+0000","last_fullsized":"2023-03-03T07:57:16.756071+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:19.455490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.727392+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T07:57:15.727392+0000","last_peered":"2023-03-03T07:57:15.727392+0000","last_clean":"2023-03-03T07:57:15.727392+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T07:57:15.727392+0000","last_undegraded":"2023-03-03T07:57:15.727392+0000","last_fullsized":"2023-03-03T07:57:15.727392+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.744979+0000","last_change":"2023-03-03T07:57:16.744979+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.744979+0000","last_undegraded":"2023-03-03T07:57:16.744979+0000","last_fullsized":"2023-03-03T07:57:16.744979+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.752649+0000","last_change":"2023-03-03T07:57:16.749605+0000","last_active":"2023-03-03T07:57:16.752649+0000","last_peered":"2023-03-03T07:57:16.752649+0000","last_clean":"2023-03-03T07:57:16.752649+0000","last_became_active":"2023-03-03T07:57:16.749336+0000","last_became_peered":"2023-03-03T07:57:16.749336+0000","last_unstale":"2023-03-03T07:57:16.752649+0000","last_undegraded":"2023-03-03T07:57:16.752649+0000","last_fullsized":"2023-03-03T07:57:16.752649+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:42.524029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.722841+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T07:57:15.722841+0000","last_peered":"2023-03-03T07:57:15.722841+0000","last_clean":"2023-03-03T07:57:15.722841+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T07:57:15.722841+0000","last_undegraded":"2023-03-03T07:57:15.722841+0000","last_fullsized":"2023-03-03T07:57:15.722841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.741077+0000","last_change":"2023-03-03T07:57:16.741077+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.741077+0000","last_undegraded":"2023-03-03T07:57:16.741077+0000","last_fullsized":"2023-03-03T07:57:16.741077+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.753872+0000","last_change":"2023-03-03T07:57:16.750917+0000","last_active":"2023-03-03T07:57:16.753872+0000","last_peered":"2023-03-03T07:57:16.753872+0000","last_clean":"2023-03-03T07:57:16.753872+0000","last_became_active":"2023-03-03T07:57:16.750615+0000","last_became_peered":"2023-03-03T07:57:16.750615+0000","last_unstale":"2023-03-03T07:57:16.753872+0000","last_undegraded":"2023-03-03T07:57:16.753872+0000","last_fullsized":"2023-03-03T07:57:16.753872+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:49:51.007473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.732651+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T07:57:15.732651+0000","last_peered":"2023-03-03T07:57:15.732651+0000","last_clean":"2023-03-03T07:57:15.732651+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T07:57:15.732651+0000","last_undegraded":"2023-03-03T07:57:15.732651+0000","last_fullsized":"2023-03-03T07:57:15.732651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.745525+0000","last_change":"2023-03-03T07:57:16.745525+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.745525+0000","last_undegraded":"2023-03-03T07:57:16.745525+0000","last_fullsized":"2023-03-03T07:57:16.745525+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.751970+0000","last_change":"2023-03-03T07:57:16.748170+0000","last_active":"2023-03-03T07:57:16.751970+0000","last_peered":"2023-03-03T07:57:16.751970+0000","last_clean":"2023-03-03T07:57:16.751970+0000","last_became_active":"2023-03-03T07:57:16.747842+0000","last_became_peered":"2023-03-03T07:57:16.747842+0000","last_unstale":"2023-03-03T07:57:16.751970+0000","last_undegraded":"2023-03-03T07:57:16.751970+0000","last_fullsized":"2023-03-03T07:57:16.751970+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:57:03.533693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.723040+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T07:57:15.723040+0000","last_peered":"2023-03-03T07:57:15.723040+0000","last_clean":"2023-03-03T07:57:15.723040+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T07:57:15.723040+0000","last_undegraded":"2023-03-03T07:57:15.723040+0000","last_fullsized":"2023-03-03T07:57:15.723040+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":176,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.728933+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T07:57:15.728933+0000","last_peered":"2023-03-03T07:57:15.728933+0000","last_clean":"2023-03-03T07:57:15.728933+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T07:57:15.728933+0000","last_undegraded":"2023-03-03T07:57:15.728933+0000","last_fullsized":"2023-03-03T07:57:15.728933+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737432+0000","last_change":"2023-03-03T07:57:16.737432+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737432+0000","last_undegraded":"2023-03-03T07:57:16.737432+0000","last_fullsized":"2023-03-03T07:57:16.737432+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.762247+0000","last_change":"2023-03-03T07:57:16.756473+0000","last_active":"2023-03-03T07:57:16.762247+0000","last_peered":"2023-03-03T07:57:16.762247+0000","last_clean":"2023-03-03T07:57:16.762247+0000","last_became_active":"2023-03-03T07:57:16.755809+0000","last_became_peered":"2023-03-03T07:57:16.755809+0000","last_unstale":"2023-03-03T07:57:16.762247+0000","last_undegraded":"2023-03-03T07:57:16.762247+0000","last_fullsized":"2023-03-03T07:57:16.762247+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:19.455490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.729041+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T07:57:15.729041+0000","last_peered":"2023-03-03T07:57:15.729041+0000","last_clean":"2023-03-03T07:57:15.729041+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T07:57:15.729041+0000","last_undegraded":"2023-03-03T07:57:15.729041+0000","last_fullsized":"2023-03-03T07:57:15.729041+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737491+0000","last_change":"2023-03-03T07:57:16.737491+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737491+0000","last_undegraded":"2023-03-03T07:57:16.737491+0000","last_fullsized":"2023-03-03T07:57:16.737491+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:59.426058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754211+0000","last_change":"2023-03-03T07:57:16.752378+0000","last_active":"2023-03-03T07:57:16.754211+0000","last_peered":"2023-03-03T07:57:16.754211+0000","last_clean":"2023-03-03T07:57:16.754211+0000","last_became_active":"2023-03-03T07:57:16.752028+0000","last_became_peered":"2023-03-03T07:57:16.752028+0000","last_unstale":"2023-03-03T07:57:16.754211+0000","last_undegraded":"2023-03-03T07:57:16.754211+0000","last_fullsized":"2023-03-03T07:57:16.754211+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:14:02.921691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.729041+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T07:57:15.729041+0000","last_peered":"2023-03-03T07:57:15.729041+0000","last_clean":"2023-03-03T07:57:15.729041+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T07:57:15.729041+0000","last_undegraded":"2023-03-03T07:57:15.729041+0000","last_fullsized":"2023-03-03T07:57:15.729041+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.731292+0000","last_change":"2023-03-03T07:57:16.731292+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.731292+0000","last_undegraded":"2023-03-03T07:57:16.731292+0000","last_fullsized":"2023-03-03T07:57:16.731292+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.754275+0000","last_change":"2023-03-03T07:57:16.742160+0000","last_active":"2023-03-03T07:57:16.754275+0000","last_peered":"2023-03-03T07:57:16.754275+0000","last_clean":"2023-03-03T07:57:16.754275+0000","last_became_active":"2023-03-03T07:57:16.741796+0000","last_became_peered":"2023-03-03T07:57:16.741796+0000","last_unstale":"2023-03-03T07:57:16.754275+0000","last_undegraded":"2023-03-03T07:57:16.754275+0000","last_fullsized":"2023-03-03T07:57:16.754275+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:55.349529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.727435+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T07:57:15.727435+0000","last_peered":"2023-03-03T07:57:15.727435+0000","last_clean":"2023-03-03T07:57:15.727435+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T07:57:15.727435+0000","last_undegraded":"2023-03-03T07:57:15.727435+0000","last_fullsized":"2023-03-03T07:57:15.727435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2023-03-03T07:57:16.737604+0000","last_change":"2023-03-03T07:57:16.737604+0000","last_active":"2023-03-03T07:57:16.718506+0000","last_peered":"2023-03-03T07:57:16.718506+0000","last_clean":"2023-03-03T07:57:16.718506+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:57:16.737604+0000","last_undegraded":"2023-03-03T07:57:16.737604+0000","last_fullsized":"2023-03-03T07:57:16.737604+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:25.355832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-03-03T07:57:16.752225+0000","last_change":"2023-03-03T07:57:16.744051+0000","last_active":"2023-03-03T07:57:16.752225+0000","last_peered":"2023-03-03T07:57:16.752225+0000","last_clean":"2023-03-03T07:57:16.752225+0000","last_became_active":"2023-03-03T07:57:16.743785+0000","last_became_peered":"2023-03-03T07:57:16.743785+0000","last_unstale":"2023-03-03T07:57:16.752225+0000","last_undegraded":"2023-03-03T07:57:16.752225+0000","last_fullsized":"2023-03-03T07:57:16.752225+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:48:13.119986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-03-03T07:57:15.726936+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T07:57:15.726936+0000","last_peered":"2023-03-03T07:57:15.726936+0000","last_clean":"2023-03-03T07:57:15.726936+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T07:57:15.726936+0000","last_undegraded":"2023-03-03T07:57:15.726936+0000","last_fullsized":"2023-03-03T07:57:15.726936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9424,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362416,"statfs":{"total":96636764160,"available":96627113984,"internally_reserved":0,"allocated":991232,"data_stored":750757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.624},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":0.60399999999999998}]},{"osd":1,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":6.6829999999999998,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":0.64800000000000002}]},{"osd":2,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.54300000000000004}]},{"osd":3,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.46000000000000002}]},{"osd":4,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.441},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.57199999999999995}]},{"osd":5,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.70399999999999996}]},{"osd":6,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.499}]}]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8200,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363640,"statfs":{"total":96636764160,"available":96628367360,"internally_reserved":0,"allocated":393216,"data_stored":159370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.79000000000000004}]},{"osd":1,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.68100000000000005}]},{"osd":2,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.754},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.66700000000000004}]},{"osd":3,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.89800000000000002}]},{"osd":4,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":4.0099999999999998,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":4.1859999999999999,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.70999999999999996}]},{"osd":5,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.82299999999999995}]},{"osd":7,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.63100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8332,"kb_used_data":388,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363508,"statfs":{"total":96636764160,"available":96628232192,"internally_reserved":0,"allocated":397312,"data_stored":160389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.972,"5min":3.972,"15min":3.972},"last":0.48899999999999999}]},{"osd":2,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":11.897,"5min":11.897,"15min":11.897},"last":0.44400000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":11.804,"5min":11.804,"15min":11.804},"last":0.51300000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":11.843999999999999,"5min":11.843999999999999,"15min":11.843999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":3.8740000000000001,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.43099999999999999}]},{"osd":4,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.72699999999999998}]},{"osd":5,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":3.8300000000000001,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.73899999999999999}]},{"osd":6,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":3.907,"5min":3.907,"15min":3.907},"last":0.748},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.78000000000000003}]},{"osd":7,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":3.9399999999999999,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":0.64800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9356,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362484,"statfs":{"total":96636764160,"available":96627183616,"internally_reserved":0,"allocated":987136,"data_stored":749738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.36599999999999999}]},{"osd":2,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.47199999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.629}]},{"osd":4,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.95999999999999996}]},{"osd":5,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.91300000000000003}]},{"osd":6,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.872}]},{"osd":7,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.88100000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8200,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363640,"statfs":{"total":96636764160,"available":96628367360,"internally_reserved":0,"allocated":393216,"data_stored":159370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":5.7889999999999997,"5min":5.7889999999999997,"15min":5.7889999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":2.6309999999999998,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":0.505}]},{"osd":1,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":5.8470000000000004,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.60799999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.961,"5min":12.961,"15min":12.961},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.1449999999999996,"5min":6.1449999999999996,"15min":6.1449999999999996},"last":0.55500000000000005}]},{"osd":4,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":6.484,"5min":6.484,"15min":6.484},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":6.5510000000000002,"5min":6.5510000000000002,"15min":6.5510000000000002},"last":0.96099999999999997}]},{"osd":5,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":5.7119999999999997,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":0.871},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":6.6369999999999996,"5min":6.6369999999999996,"15min":6.6369999999999996},"last":0.92900000000000005}]},{"osd":6,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":6.625,"5min":6.625,"15min":6.625},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":5.7539999999999996,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.91400000000000003}]},{"osd":7,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":6.7009999999999996,"5min":6.7009999999999996,"15min":6.7009999999999996},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":6.6840000000000002,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.98999999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8332,"kb_used_data":388,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363508,"statfs":{"total":96636764160,"available":96628232192,"internally_reserved":0,"allocated":397312,"data_stored":160389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":9.5820000000000007,"5min":9.5820000000000007,"15min":9.5820000000000007},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":9.7840000000000007,"5min":9.7840000000000007,"15min":9.7840000000000007},"last":0.35499999999999998}]},{"osd":1,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.702,"5min":9.702,"15min":9.702},"last":0.60999999999999999}]},{"osd":2,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":8.3190000000000008,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.133,"5min":1.133,"15min":1.133},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":9.734,"5min":9.734,"15min":9.734},"last":0.68000000000000005}]},{"osd":4,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":9.8629999999999995,"5min":9.8629999999999995,"15min":9.8629999999999995},"last":0.871},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.99199999999999999}]},{"osd":5,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":9.6600000000000001,"5min":9.6600000000000001,"15min":9.6600000000000001},"last":0.90100000000000002}]},{"osd":6,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":9.8190000000000008,"5min":9.8190000000000008,"15min":9.8190000000000008},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.91700000000000004}]},{"osd":7,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":9.625,"5min":9.625,"15min":9.625},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":9.8949999999999996,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":1.026}]}]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8472,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363368,"statfs":{"total":96636764160,"available":96628088832,"internally_reserved":0,"allocated":409600,"data_stored":167017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":7.1799999999999997,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.51300000000000001}]},{"osd":1,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":9.5709999999999997,"5min":9.5709999999999997,"15min":9.5709999999999997},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":5.266,"5min":5.266,"15min":5.266},"last":0.70399999999999996}]},{"osd":2,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":8.4529999999999994,"5min":8.4529999999999994,"15min":8.4529999999999994},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":6.6829999999999998,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.84999999999999998}]},{"osd":5,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.748}]},{"osd":6,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.82499999999999996}]},{"osd":7,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.92900000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8340,"kb_used_data":396,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363500,"statfs":{"total":96636764160,"available":96628224000,"internally_reserved":0,"allocated":405504,"data_stored":165998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69699999999999995}]},{"osd":1,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.76400000000000001}]},{"osd":2,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.72599999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.63200000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.80000000000000004}]},{"osd":6,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.85099999999999998}]},{"osd":7,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.747},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.82899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T07:57:23.373 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3f is not active+clean 2023-03-03T07:57:23.374 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.748481+0000', 'last_change': '2023-03-03T07:57:16.748481+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.748481+0000', 'last_undegraded': '2023-03-03T07:57:16.748481+0000', 'last_fullsized': '2023-03-03T07:57:16.748481+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T10:39:27.674460+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-03-03T07:57:23.375 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3e is not active+clean 2023-03-03T07:57:23.375 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.746045+0000', 'last_change': '2023-03-03T07:57:16.746045+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.746045+0000', 'last_undegraded': '2023-03-03T07:57:16.746045+0000', 'last_fullsized': '2023-03-03T07:57:16.746045+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-03-03T07:57:23.375 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3d is not active+clean 2023-03-03T07:57:23.376 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.739842+0000', 'last_change': '2023-03-03T07:57:16.739842+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.739842+0000', 'last_undegraded': '2023-03-03T07:57:16.739842+0000', 'last_fullsized': '2023-03-03T07:57:16.739842+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-03-03T07:57:23.376 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3c is not active+clean 2023-03-03T07:57:23.377 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.740784+0000', 'last_change': '2023-03-03T07:57:16.740784+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.740784+0000', 'last_undegraded': '2023-03-03T07:57:16.740784+0000', 'last_fullsized': '2023-03-03T07:57:16.740784+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T15:33:07.498551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-03-03T07:57:23.377 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3b is not active+clean 2023-03-03T07:57:23.377 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.741327+0000', 'last_change': '2023-03-03T07:57:16.741327+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.741327+0000', 'last_undegraded': '2023-03-03T07:57:16.741327+0000', 'last_fullsized': '2023-03-03T07:57:16.741327+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T17:37:50.701337+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-03-03T07:57:23.378 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3a is not active+clean 2023-03-03T07:57:23.378 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.738165+0000', 'last_change': '2023-03-03T07:57:16.738165+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.738165+0000', 'last_undegraded': '2023-03-03T07:57:16.738165+0000', 'last_fullsized': '2023-03-03T07:57:16.738165+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T08:15:25.355832+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-03-03T07:57:23.378 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.39 is not active+clean 2023-03-03T07:57:23.378 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.749480+0000', 'last_change': '2023-03-03T07:57:16.749480+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.749480+0000', 'last_undegraded': '2023-03-03T07:57:16.749480+0000', 'last_fullsized': '2023-03-03T07:57:16.749480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T16:55:44.621837+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-03-03T07:57:23.379 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.38 is not active+clean 2023-03-03T07:57:23.379 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.747051+0000', 'last_change': '2023-03-03T07:57:16.747051+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.747051+0000', 'last_undegraded': '2023-03-03T07:57:16.747051+0000', 'last_fullsized': '2023-03-03T07:57:16.747051+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T14:03:01.871020+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-03-03T07:57:23.379 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.37 is not active+clean 2023-03-03T07:57:23.380 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.739643+0000', 'last_change': '2023-03-03T07:57:16.739643+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.739643+0000', 'last_undegraded': '2023-03-03T07:57:16.739643+0000', 'last_fullsized': '2023-03-03T07:57:16.739643+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-03-03T07:57:23.381 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.36 is not active+clean 2023-03-03T07:57:23.381 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.747834+0000', 'last_change': '2023-03-03T07:57:16.747834+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.747834+0000', 'last_undegraded': '2023-03-03T07:57:16.747834+0000', 'last_fullsized': '2023-03-03T07:57:16.747834+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-03-03T07:57:23.381 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.35 is not active+clean 2023-03-03T07:57:23.382 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.748481+0000', 'last_change': '2023-03-03T07:57:16.748481+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.748481+0000', 'last_undegraded': '2023-03-03T07:57:16.748481+0000', 'last_fullsized': '2023-03-03T07:57:16.748481+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-03-03T07:57:23.382 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.34 is not active+clean 2023-03-03T07:57:23.382 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.742043+0000', 'last_change': '2023-03-03T07:57:16.742043+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.742043+0000', 'last_undegraded': '2023-03-03T07:57:16.742043+0000', 'last_fullsized': '2023-03-03T07:57:16.742043+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-03-03T07:57:23.382 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.33 is not active+clean 2023-03-03T07:57:23.383 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.746702+0000', 'last_change': '2023-03-03T07:57:16.746702+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.746702+0000', 'last_undegraded': '2023-03-03T07:57:16.746702+0000', 'last_fullsized': '2023-03-03T07:57:16.746702+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-03-03T07:57:23.383 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.32 is not active+clean 2023-03-03T07:57:23.383 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.730661+0000', 'last_change': '2023-03-03T07:57:16.730661+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.730661+0000', 'last_undegraded': '2023-03-03T07:57:16.730661+0000', 'last_fullsized': '2023-03-03T07:57:16.730661+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T19:48:58.988637+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-03-03T07:57:23.384 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.31 is not active+clean 2023-03-03T07:57:23.384 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.747212+0000', 'last_change': '2023-03-03T07:57:16.747212+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.747212+0000', 'last_undegraded': '2023-03-03T07:57:16.747212+0000', 'last_fullsized': '2023-03-03T07:57:16.747212+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T10:24:04.359311+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-03-03T07:57:23.384 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2023-03-03T07:57:23.385 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.746985+0000', 'last_change': '2023-03-03T07:57:16.746985+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.746985+0000', 'last_undegraded': '2023-03-03T07:57:16.746985+0000', 'last_fullsized': '2023-03-03T07:57:16.746985+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-03-03T07:57:23.385 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2023-03-03T07:57:23.385 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.738184+0000', 'last_change': '2023-03-03T07:57:16.738184+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.738184+0000', 'last_undegraded': '2023-03-03T07:57:16.738184+0000', 'last_fullsized': '2023-03-03T07:57:16.738184+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-03-03T07:57:23.386 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2e is not active+clean 2023-03-03T07:57:23.386 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.737683+0000', 'last_change': '2023-03-03T07:57:16.737683+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.737683+0000', 'last_undegraded': '2023-03-03T07:57:16.737683+0000', 'last_fullsized': '2023-03-03T07:57:16.737683+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T18:56:10.139056+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-03-03T07:57:23.386 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2d is not active+clean 2023-03-03T07:57:23.386 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.740159+0000', 'last_change': '2023-03-03T07:57:16.740159+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.740159+0000', 'last_undegraded': '2023-03-03T07:57:16.740159+0000', 'last_fullsized': '2023-03-03T07:57:16.740159+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T16:00:23.450170+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-03-03T07:57:23.387 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2c is not active+clean 2023-03-03T07:57:23.387 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.737107+0000', 'last_change': '2023-03-03T07:57:16.737107+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.737107+0000', 'last_undegraded': '2023-03-03T07:57:16.737107+0000', 'last_fullsized': '2023-03-03T07:57:16.737107+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T14:32:15.531132+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-03-03T07:57:23.387 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2b is not active+clean 2023-03-03T07:57:23.388 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.746180+0000', 'last_change': '2023-03-03T07:57:16.746180+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.746180+0000', 'last_undegraded': '2023-03-03T07:57:16.746180+0000', 'last_fullsized': '2023-03-03T07:57:16.746180+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-03-03T07:57:23.388 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2a is not active+clean 2023-03-03T07:57:23.388 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.732520+0000', 'last_change': '2023-03-03T07:57:16.732520+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.732520+0000', 'last_undegraded': '2023-03-03T07:57:16.732520+0000', 'last_fullsized': '2023-03-03T07:57:16.732520+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T10:39:27.674460+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-03-03T07:57:23.389 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2023-03-03T07:57:23.389 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.739235+0000', 'last_change': '2023-03-03T07:57:16.739235+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.739235+0000', 'last_undegraded': '2023-03-03T07:57:16.739235+0000', 'last_fullsized': '2023-03-03T07:57:16.739235+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-03-03T07:57:23.389 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.28 is not active+clean 2023-03-03T07:57:23.390 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.738079+0000', 'last_change': '2023-03-03T07:57:16.738079+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.738079+0000', 'last_undegraded': '2023-03-03T07:57:16.738079+0000', 'last_fullsized': '2023-03-03T07:57:16.738079+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T10:57:04.542617+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-03-03T07:57:23.390 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.27 is not active+clean 2023-03-03T07:57:23.390 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.737878+0000', 'last_change': '2023-03-03T07:57:16.737878+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.737878+0000', 'last_undegraded': '2023-03-03T07:57:16.737878+0000', 'last_fullsized': '2023-03-03T07:57:16.737878+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T10:57:04.542617+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-03-03T07:57:23.391 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.26 is not active+clean 2023-03-03T07:57:23.391 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.736707+0000', 'last_change': '2023-03-03T07:57:16.736707+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.736707+0000', 'last_undegraded': '2023-03-03T07:57:16.736707+0000', 'last_fullsized': '2023-03-03T07:57:16.736707+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-03-03T07:57:23.391 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.25 is not active+clean 2023-03-03T07:57:23.391 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.736973+0000', 'last_change': '2023-03-03T07:57:16.736973+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.736973+0000', 'last_undegraded': '2023-03-03T07:57:16.736973+0000', 'last_fullsized': '2023-03-03T07:57:16.736973+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T16:21:20.464414+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-03-03T07:57:23.392 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.24 is not active+clean 2023-03-03T07:57:23.392 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.746809+0000', 'last_change': '2023-03-03T07:57:16.746809+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.746809+0000', 'last_undegraded': '2023-03-03T07:57:16.746809+0000', 'last_fullsized': '2023-03-03T07:57:16.746809+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-03-03T07:57:23.392 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.23 is not active+clean 2023-03-03T07:57:23.393 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.736370+0000', 'last_change': '2023-03-03T07:57:16.736370+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.736370+0000', 'last_undegraded': '2023-03-03T07:57:16.736370+0000', 'last_fullsized': '2023-03-03T07:57:16.736370+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T15:33:07.498551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-03-03T07:57:23.393 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.22 is not active+clean 2023-03-03T07:57:23.393 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.739548+0000', 'last_change': '2023-03-03T07:57:16.739548+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.739548+0000', 'last_undegraded': '2023-03-03T07:57:16.739548+0000', 'last_fullsized': '2023-03-03T07:57:16.739548+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T12:10:43.532953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-03-03T07:57:23.394 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2023-03-03T07:57:23.394 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.738752+0000', 'last_change': '2023-03-03T07:57:16.738752+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.738752+0000', 'last_undegraded': '2023-03-03T07:57:16.738752+0000', 'last_fullsized': '2023-03-03T07:57:16.738752+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-03-03T07:57:23.394 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.20 is not active+clean 2023-03-03T07:57:23.394 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.737178+0000', 'last_change': '2023-03-03T07:57:16.737178+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.737178+0000', 'last_undegraded': '2023-03-03T07:57:16.737178+0000', 'last_fullsized': '2023-03-03T07:57:16.737178+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T15:37:26.655223+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-03-03T07:57:23.395 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2023-03-03T07:57:23.395 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.742046+0000', 'last_change': '2023-03-03T07:57:16.742046+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.742046+0000', 'last_undegraded': '2023-03-03T07:57:16.742046+0000', 'last_fullsized': '2023-03-03T07:57:16.742046+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-03-03T07:57:23.395 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2023-03-03T07:57:23.396 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.738564+0000', 'last_change': '2023-03-03T07:57:16.738564+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.738564+0000', 'last_undegraded': '2023-03-03T07:57:16.738564+0000', 'last_fullsized': '2023-03-03T07:57:16.738564+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-03-03T07:57:23.396 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2023-03-03T07:57:23.396 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.731582+0000', 'last_change': '2023-03-03T07:57:16.731582+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.731582+0000', 'last_undegraded': '2023-03-03T07:57:16.731582+0000', 'last_fullsized': '2023-03-03T07:57:16.731582+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T18:27:11.090994+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-03-03T07:57:23.397 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2023-03-03T07:57:23.397 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.747957+0000', 'last_change': '2023-03-03T07:57:16.747957+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.747957+0000', 'last_undegraded': '2023-03-03T07:57:16.747957+0000', 'last_fullsized': '2023-03-03T07:57:16.747957+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T18:27:11.090994+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-03-03T07:57:23.397 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2023-03-03T07:57:23.398 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.741626+0000', 'last_change': '2023-03-03T07:57:16.741626+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.741626+0000', 'last_undegraded': '2023-03-03T07:57:16.741626+0000', 'last_fullsized': '2023-03-03T07:57:16.741626+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-03-03T07:57:23.398 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2023-03-03T07:57:23.398 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.746580+0000', 'last_change': '2023-03-03T07:57:16.746580+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.746580+0000', 'last_undegraded': '2023-03-03T07:57:16.746580+0000', 'last_fullsized': '2023-03-03T07:57:16.746580+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-03-03T07:57:23.399 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2023-03-03T07:57:23.399 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.731963+0000', 'last_change': '2023-03-03T07:57:16.731963+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.731963+0000', 'last_undegraded': '2023-03-03T07:57:16.731963+0000', 'last_fullsized': '2023-03-03T07:57:16.731963+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-03-03T07:57:23.399 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2023-03-03T07:57:23.399 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.747501+0000', 'last_change': '2023-03-03T07:57:16.747501+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.747501+0000', 'last_undegraded': '2023-03-03T07:57:16.747501+0000', 'last_fullsized': '2023-03-03T07:57:16.747501+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T19:48:58.988637+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-03-03T07:57:23.400 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2023-03-03T07:57:23.400 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.738898+0000', 'last_change': '2023-03-03T07:57:16.738898+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.738898+0000', 'last_undegraded': '2023-03-03T07:57:16.738898+0000', 'last_fullsized': '2023-03-03T07:57:16.738898+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T10:41:11.947039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-03-03T07:57:23.400 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2023-03-03T07:57:23.401 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.741492+0000', 'last_change': '2023-03-03T07:57:16.741492+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.741492+0000', 'last_undegraded': '2023-03-03T07:57:16.741492+0000', 'last_fullsized': '2023-03-03T07:57:16.741492+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-03-03T07:57:23.401 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2023-03-03T07:57:23.401 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.742445+0000', 'last_change': '2023-03-03T07:57:16.742445+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.742445+0000', 'last_undegraded': '2023-03-03T07:57:16.742445+0000', 'last_fullsized': '2023-03-03T07:57:16.742445+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-03-03T07:57:23.402 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2023-03-03T07:57:23.402 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.731414+0000', 'last_change': '2023-03-03T07:57:16.731414+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.731414+0000', 'last_undegraded': '2023-03-03T07:57:16.731414+0000', 'last_fullsized': '2023-03-03T07:57:16.731414+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T14:03:01.871020+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-03-03T07:57:23.402 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2023-03-03T07:57:23.402 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.740047+0000', 'last_change': '2023-03-03T07:57:16.740047+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.740047+0000', 'last_undegraded': '2023-03-03T07:57:16.740047+0000', 'last_fullsized': '2023-03-03T07:57:16.740047+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T12:21:12.720267+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-03-03T07:57:23.403 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2023-03-03T07:57:23.403 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.745842+0000', 'last_change': '2023-03-03T07:57:16.745842+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.745842+0000', 'last_undegraded': '2023-03-03T07:57:16.745842+0000', 'last_fullsized': '2023-03-03T07:57:16.745842+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T10:24:04.359311+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-03-03T07:57:23.403 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2023-03-03T07:57:23.404 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.739793+0000', 'last_change': '2023-03-03T07:57:16.739793+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.739793+0000', 'last_undegraded': '2023-03-03T07:57:16.739793+0000', 'last_fullsized': '2023-03-03T07:57:16.739793+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T12:21:12.720267+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-03-03T07:57:23.404 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2023-03-03T07:57:23.404 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.746997+0000', 'last_change': '2023-03-03T07:57:16.746997+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.746997+0000', 'last_undegraded': '2023-03-03T07:57:16.746997+0000', 'last_fullsized': '2023-03-03T07:57:16.746997+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T15:35:32.174190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-03-03T07:57:23.405 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2023-03-03T07:57:23.405 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.745199+0000', 'last_change': '2023-03-03T07:57:16.745199+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.745199+0000', 'last_undegraded': '2023-03-03T07:57:16.745199+0000', 'last_fullsized': '2023-03-03T07:57:16.745199+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T16:55:44.621837+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-03-03T07:57:23.405 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2023-03-03T07:57:23.406 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.747344+0000', 'last_change': '2023-03-03T07:57:16.747344+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.747344+0000', 'last_undegraded': '2023-03-03T07:57:16.747344+0000', 'last_fullsized': '2023-03-03T07:57:16.747344+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T18:13:17.532289+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-03-03T07:57:23.406 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2023-03-03T07:57:23.406 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.731014+0000', 'last_change': '2023-03-03T07:57:16.731014+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.731014+0000', 'last_undegraded': '2023-03-03T07:57:16.731014+0000', 'last_fullsized': '2023-03-03T07:57:16.731014+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T15:35:32.174190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-03-03T07:57:23.407 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2023-03-03T07:57:23.407 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.738225+0000', 'last_change': '2023-03-03T07:57:16.738225+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.738225+0000', 'last_undegraded': '2023-03-03T07:57:16.738225+0000', 'last_fullsized': '2023-03-03T07:57:16.738225+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T12:10:43.532953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-03-03T07:57:23.407 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2023-03-03T07:57:23.407 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.743223+0000', 'last_change': '2023-03-03T07:57:16.743223+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.743223+0000', 'last_undegraded': '2023-03-03T07:57:16.743223+0000', 'last_fullsized': '2023-03-03T07:57:16.743223+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-03-03T07:57:23.408 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2023-03-03T07:57:23.408 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.741849+0000', 'last_change': '2023-03-03T07:57:16.741849+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.741849+0000', 'last_undegraded': '2023-03-03T07:57:16.741849+0000', 'last_fullsized': '2023-03-03T07:57:16.741849+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-03-03T07:57:23.408 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2023-03-03T07:57:23.409 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.746117+0000', 'last_change': '2023-03-03T07:57:16.746117+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.746117+0000', 'last_undegraded': '2023-03-03T07:57:16.746117+0000', 'last_fullsized': '2023-03-03T07:57:16.746117+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-03-03T07:57:23.409 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2023-03-03T07:57:23.409 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.746443+0000', 'last_change': '2023-03-03T07:57:16.746443+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.746443+0000', 'last_undegraded': '2023-03-03T07:57:16.746443+0000', 'last_fullsized': '2023-03-03T07:57:16.746443+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-03-03T07:57:23.410 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2023-03-03T07:57:23.410 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.741199+0000', 'last_change': '2023-03-03T07:57:16.741199+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.741199+0000', 'last_undegraded': '2023-03-03T07:57:16.741199+0000', 'last_fullsized': '2023-03-03T07:57:16.741199+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-03-03T07:57:23.410 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2023-03-03T07:57:23.410 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.744979+0000', 'last_change': '2023-03-03T07:57:16.744979+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.744979+0000', 'last_undegraded': '2023-03-03T07:57:16.744979+0000', 'last_fullsized': '2023-03-03T07:57:16.744979+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-03-03T07:57:23.411 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2023-03-03T07:57:23.411 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.741077+0000', 'last_change': '2023-03-03T07:57:16.741077+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.741077+0000', 'last_undegraded': '2023-03-03T07:57:16.741077+0000', 'last_fullsized': '2023-03-03T07:57:16.741077+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-03-03T07:57:23.411 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2023-03-03T07:57:23.412 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.745525+0000', 'last_change': '2023-03-03T07:57:16.745525+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.745525+0000', 'last_undegraded': '2023-03-03T07:57:16.745525+0000', 'last_fullsized': '2023-03-03T07:57:16.745525+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-03-03T07:57:23.412 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2023-03-03T07:57:23.412 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.737432+0000', 'last_change': '2023-03-03T07:57:16.737432+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.737432+0000', 'last_undegraded': '2023-03-03T07:57:16.737432+0000', 'last_fullsized': '2023-03-03T07:57:16.737432+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-03-03T07:57:23.413 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2023-03-03T07:57:23.413 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.737491+0000', 'last_change': '2023-03-03T07:57:16.737491+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.737491+0000', 'last_undegraded': '2023-03-03T07:57:16.737491+0000', 'last_fullsized': '2023-03-03T07:57:16.737491+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T10:42:59.426058+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-03-03T07:57:23.413 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2023-03-03T07:57:23.414 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.731292+0000', 'last_change': '2023-03-03T07:57:16.731292+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.731292+0000', 'last_undegraded': '2023-03-03T07:57:16.731292+0000', 'last_fullsized': '2023-03-03T07:57:16.731292+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-03-03T07:57:23.414 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2023-03-03T07:57:23.414 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2023-03-03T07:57:16.737604+0000', 'last_change': '2023-03-03T07:57:16.737604+0000', 'last_active': '2023-03-03T07:57:16.718506+0000', 'last_peered': '2023-03-03T07:57:16.718506+0000', 'last_clean': '2023-03-03T07:57:16.718506+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-03-03T07:57:16.737604+0000', 'last_undegraded': '2023-03-03T07:57:16.737604+0000', 'last_fullsized': '2023-03-03T07:57:16.737604+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'last_clean_scrub_stamp': '2023-03-03T07:57:16.718506+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-03-04T08:15:25.355832+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-03-03T07:57:23.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-03-03T07:57:23.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.658+0000 7f9059bff700 1 Processor -- start 2023-03-03T07:57:23.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.659+0000 7f9059bff700 1 -- start start 2023-03-03T07:57:23.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.660+0000 7f9059bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f90540b8b90 0x7f90540cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:23.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.660+0000 7f9059bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90540d0ef0 con 0x7f90540b87c0 2023-03-03T07:57:23.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.660+0000 7f9059bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90540d0600 con 0x7f90540b5410 2023-03-03T07:57:23.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.660+0000 7f9059bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90540cd970 con 0x7f90540b8b90 2023-03-03T07:57:23.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.661+0000 7f90537fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f90540b87c0 0x7f90540b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38806/0 (socket says 172.21.15.98:38806) 2023-03-03T07:57:23.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.661+0000 7f90537fe700 1 -- 172.21.15.98:0/2827047688 learned_addr learned my addr 172.21.15.98:0/2827047688 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:23.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.661+0000 7f9052ffd700 1 --2- 172.21.15.98:0/2827047688 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f90540b8b90 0x7f90540cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:23.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.661+0000 7f90527fc700 1 -- 172.21.15.98:0/2827047688 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455062422 0 0) 0x7f90540d0ef0 con 0x7f90540b87c0 2023-03-03T07:57:23.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.661+0000 7f90527fc700 1 -- 172.21.15.98:0/2827047688 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f903c0035b0 con 0x7f90540b87c0 2023-03-03T07:57:23.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.661+0000 7f90527fc700 1 -- 172.21.15.98:0/2827047688 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582079820 0 0) 0x7f90540d0600 con 0x7f90540b5410 2023-03-03T07:57:23.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.661+0000 7f90527fc700 1 -- 172.21.15.98:0/2827047688 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90540d0ef0 con 0x7f90540b5410 2023-03-03T07:57:23.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.661+0000 7f9052ffd700 1 -- 172.21.15.98:0/2827047688 >> v1:172.21.15.100:6789/0 conn(0x7f90540b5410 legacy=0x7f90540d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:23.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.661+0000 7f9052ffd700 1 -- 172.21.15.98:0/2827047688 >> v1:172.21.15.98:6789/0 conn(0x7f90540b87c0 legacy=0x7f90540b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:23.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.661+0000 7f90527fc700 1 -- 172.21.15.98:0/2827047688 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1753772559 0 0) 0x7f903c0035b0 con 0x7f90540b87c0 2023-03-03T07:57:23.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.661+0000 7f9052ffd700 1 -- 172.21.15.98:0/2827047688 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90540cdc50 con 0x7f90540b8b90 2023-03-03T07:57:23.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.662+0000 7f9052ffd700 1 --2- 172.21.15.98:0/2827047688 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f90540b8b90 0x7f90540cffe0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9048009870 tx=0x7f9048009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ae90dde0979069d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:23.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.662+0000 7f90527fc700 1 -- 172.21.15.98:0/2827047688 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9044001670 con 0x7f90540b8b90 2023-03-03T07:57:23.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.662+0000 7f90527fc700 1 -- 172.21.15.98:0/2827047688 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9048004030 con 0x7f90540b8b90 2023-03-03T07:57:23.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.662+0000 7f90527fc700 1 -- 172.21.15.98:0/2827047688 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90480043c0 con 0x7f90540b8b90 2023-03-03T07:57:23.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.663+0000 7f9059bff700 1 -- 172.21.15.98:0/2827047688 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f90540b8b90 msgr2=0x7f90540cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:23.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.663+0000 7f9059bff700 1 --2- 172.21.15.98:0/2827047688 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f90540b8b90 0x7f90540cffe0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9048009870 tx=0x7f9048009b80 comp rx=0 tx=0).stop 2023-03-03T07:57:23.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.663+0000 7f9059bff700 1 -- 172.21.15.98:0/2827047688 shutdown_connections 2023-03-03T07:57:23.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.663+0000 7f9059bff700 1 --2- 172.21.15.98:0/2827047688 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f90540b8b90 0x7f90540cffe0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.663+0000 7f9059bff700 1 -- 172.21.15.98:0/2827047688 >> 172.21.15.98:0/2827047688 conn(0x7f905406aee0 msgr2=0x7f905406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:23.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.663+0000 7f9059bff700 1 -- 172.21.15.98:0/2827047688 shutdown_connections 2023-03-03T07:57:23.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.663+0000 7f9059bff700 1 -- 172.21.15.98:0/2827047688 wait complete. 2023-03-03T07:57:23.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.665+0000 7f9059bff700 1 Processor -- start 2023-03-03T07:57:23.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.665+0000 7f9059bff700 1 -- start start 2023-03-03T07:57:23.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.666+0000 7f9059bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90540b5410 0x7f90540ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:23.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.666+0000 7f9059bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f90540b87c0 0x7f90540cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:23.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.666+0000 7f9059bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f90540b8b90 0x7f90541f53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:23.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.666+0000 7f9059bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90540cba40 con 0x7f90540b5410 2023-03-03T07:57:23.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.666+0000 7f9059bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90540cb5b0 con 0x7f90540b8b90 2023-03-03T07:57:23.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.666+0000 7f9059bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90540ca9a0 con 0x7f90540b87c0 2023-03-03T07:57:23.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.666+0000 7f90537fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90540b5410 0x7f90540ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:23.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.666+0000 7f90537fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90540b5410 0x7f90540ca350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45598/0 (socket says 172.21.15.98:45598) 2023-03-03T07:57:23.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.666+0000 7f90537fe700 1 -- 172.21.15.98:0/3952025381 learned_addr learned my addr 172.21.15.98:0/3952025381 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:23.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.667+0000 7f9052ffd700 1 --2- 172.21.15.98:0/3952025381 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f90540b87c0 0x7f90540cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:23.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.667+0000 7f9053fff700 1 --2- 172.21.15.98:0/3952025381 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f90540b8b90 0x7f90541f53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:23.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.667+0000 7f90537fe700 1 -- 172.21.15.98:0/3952025381 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f90540b87c0 msgr2=0x7f90540cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:23.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.667+0000 7f90537fe700 1 --2- 172.21.15.98:0/3952025381 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f90540b87c0 0x7f90540cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.667+0000 7f90537fe700 1 -- 172.21.15.98:0/3952025381 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f90540b8b90 msgr2=0x7f90541f53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.667+0000 7f90537fe700 1 --2- 172.21.15.98:0/3952025381 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f90540b8b90 0x7f90541f53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.667+0000 7f90537fe700 1 -- 172.21.15.98:0/3952025381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90480094a0 con 0x7f90540b5410 2023-03-03T07:57:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.667+0000 7f90537fe700 1 --2- 172.21.15.98:0/3952025381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90540b5410 0x7f90540ca350 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f904400c730 tx=0x7f904400caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.667+0000 7f903bfff700 1 -- 172.21.15.98:0/3952025381 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9044004c80 con 0x7f90540b5410 2023-03-03T07:57:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.668+0000 7f903bfff700 1 -- 172.21.15.98:0/3952025381 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9044004e00 con 0x7f90540b5410 2023-03-03T07:57:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.668+0000 7f903bfff700 1 -- 172.21.15.98:0/3952025381 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9044007310 con 0x7f90540b5410 2023-03-03T07:57:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.668+0000 7f9059bff700 1 -- 172.21.15.98:0/3952025381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90540cdf30 con 0x7f90540b5410 2023-03-03T07:57:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.668+0000 7f9059bff700 1 -- 172.21.15.98:0/3952025381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9054073a80 con 0x7f90540b5410 2023-03-03T07:57:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.669+0000 7f9059bff700 1 -- 172.21.15.98:0/3952025381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90540ce0c0 con 0x7f90540b5410 2023-03-03T07:57:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.670+0000 7f903bfff700 1 -- 172.21.15.98:0/3952025381 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f904401a020 con 0x7f90540b5410 2023-03-03T07:57:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.671+0000 7f903bfff700 1 --2- 172.21.15.98:0/3952025381 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f90400682f0 0x7f904006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.671+0000 7f903bfff700 1 -- 172.21.15.98:0/3952025381 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9044085af0 con 0x7f90540b5410 2023-03-03T07:57:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.672+0000 7f9052ffd700 1 --2- 172.21.15.98:0/3952025381 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f90400682f0 0x7f904006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.673+0000 7f9052ffd700 1 --2- 172.21.15.98:0/3952025381 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f90400682f0 0x7f904006a7a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f90480097c0 tx=0x7f9048008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.675+0000 7f903bfff700 1 -- 172.21.15.98:0/3952025381 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9044052260 con 0x7f90540b5410 2023-03-03T07:57:23.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.947+0000 7f9059bff700 1 -- 172.21.15.98:0/3952025381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f90540729b0 con 0x7f90540b5410 2023-03-03T07:57:23.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.948+0000 7f903bfff700 1 -- 172.21.15.98:0/3952025381 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1033 (secure 0 0 0) 0x7f9044055880 con 0x7f90540b5410 2023-03-03T07:57:23.949 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:23.950 INFO:teuthology.orchestra.run.smithi098.stdout:{"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":126,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":38,"num_osds":8,"num_up_osds":8,"osd_up_since":1677830131,"num_in_osds":8,"osd_in_since":1677830121,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":41}],"num_pgs":105,"num_pools":4,"num_objects":2,"data_bytes":590368,"bytes_used":70303744,"bytes_avail":773023809536,"bytes_total":773094113280,"inactive_pgs_ratio":0.60952383279800415},"fsmap":{"epoch":14,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"c","status":"up:active","gid":5227}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2023-03-03T07:56:52.192060+0000","services":{}},"progress_events":{}} 2023-03-03T07:57:23.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.950+0000 7f9059bff700 1 -- 172.21.15.98:0/3952025381 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f90400682f0 msgr2=0x7f904006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:23.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.950+0000 7f9059bff700 1 --2- 172.21.15.98:0/3952025381 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f90400682f0 0x7f904006a7a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f90480097c0 tx=0x7f9048008030 comp rx=0 tx=0).stop 2023-03-03T07:57:23.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.950+0000 7f9059bff700 1 -- 172.21.15.98:0/3952025381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90540b5410 msgr2=0x7f90540ca350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:23.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.950+0000 7f9059bff700 1 --2- 172.21.15.98:0/3952025381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90540b5410 0x7f90540ca350 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f904400c730 tx=0x7f904400caf0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.950+0000 7f9059bff700 1 -- 172.21.15.98:0/3952025381 shutdown_connections 2023-03-03T07:57:23.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.950+0000 7f9059bff700 1 --2- 172.21.15.98:0/3952025381 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f90400682f0 0x7f904006a7a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.950+0000 7f9059bff700 1 --2- 172.21.15.98:0/3952025381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90540b5410 0x7f90540ca350 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.950+0000 7f9059bff700 1 --2- 172.21.15.98:0/3952025381 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f90540b8b90 0x7f90541f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.950+0000 7f9059bff700 1 --2- 172.21.15.98:0/3952025381 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f90540b87c0 0x7f90540cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:23.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.950+0000 7f9059bff700 1 -- 172.21.15.98:0/3952025381 >> 172.21.15.98:0/3952025381 conn(0x7f905406aee0 msgr2=0x7f90540d22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:23.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.951+0000 7f9059bff700 1 -- 172.21.15.98:0/3952025381 shutdown_connections 2023-03-03T07:57:23.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:23.951+0000 7f9059bff700 1 -- 172.21.15.98:0/3952025381 wait complete. 2023-03-03T07:57:23.978 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-03-03T07:57:23.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T07:57:24.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.223+0000 7f11cc1ff700 1 Processor -- start 2023-03-03T07:57:24.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.224+0000 7f11cc1ff700 1 -- start start 2023-03-03T07:57:24.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.225+0000 7f11cc1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11c40cd8d0 0x7f11c40b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.225+0000 7f11cc1ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f11c40b6610 0x7f11c40cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.225+0000 7f11cc1ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c40d1a00 con 0x7f11c415cc00 2023-03-03T07:57:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.225+0000 7f11cc1ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c40ce040 con 0x7f11c40b6610 2023-03-03T07:57:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.225+0000 7f11cc1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c4073be0 con 0x7f11c40cd8d0 2023-03-03T07:57:24.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.226+0000 7f11ca79c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f11c40b6610 0x7f11c40cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:24.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.226+0000 7f11ca79c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f11c40b6610 0x7f11c40cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45600/0 (socket says 172.21.15.98:45600) 2023-03-03T07:57:24.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.226+0000 7f11ca79c700 1 -- 172.21.15.98:0/2657072933 learned_addr learned my addr 172.21.15.98:0/2657072933 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:24.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.226+0000 7f11c979a700 1 --2- 172.21.15.98:0/2657072933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11c40cd8d0 0x7f11c40b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:24.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.226+0000 7f11ca79c700 1 -- 172.21.15.98:0/2657072933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11c40cd8d0 msgr2=0x7f11c40b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:24.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.226+0000 7f11ca79c700 1 --2- 172.21.15.98:0/2657072933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11c40cd8d0 0x7f11c40b60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:24.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.226+0000 7f11ca79c700 1 -- 172.21.15.98:0/2657072933 >> v1:172.21.15.100:6789/0 conn(0x7f11c415cc00 legacy=0x7f11c40d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:24.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.226+0000 7f11ca79c700 1 -- 172.21.15.98:0/2657072933 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40d1ce0 con 0x7f11c40b6610 2023-03-03T07:57:24.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.227+0000 7f11ca79c700 1 --2- 172.21.15.98:0/2657072933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c40b6610 0x7f11c40cf4d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009e50 tx=0x7f11bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701b16f7c49746ed server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:24.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.227+0000 7f11c8f99700 1 -- 172.21.15.98:0/2657072933 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc00e070 con 0x7f11c40b6610 2023-03-03T07:57:24.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.227+0000 7f11c8f99700 1 -- 172.21.15.98:0/2657072933 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11bc00b910 con 0x7f11c40b6610 2023-03-03T07:57:24.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.227+0000 7f11c8f99700 1 -- 172.21.15.98:0/2657072933 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc00bcc0 con 0x7f11c40b6610 2023-03-03T07:57:24.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.227+0000 7f11cc1ff700 1 -- 172.21.15.98:0/2657072933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c40b6610 msgr2=0x7f11c40cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:24.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.227+0000 7f11cc1ff700 1 --2- 172.21.15.98:0/2657072933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c40b6610 0x7f11c40cf4d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009e50 tx=0x7f11bc00b330 comp rx=0 tx=0).stop 2023-03-03T07:57:24.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.228+0000 7f11cc1ff700 1 -- 172.21.15.98:0/2657072933 shutdown_connections 2023-03-03T07:57:24.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.228+0000 7f11cc1ff700 1 --2- 172.21.15.98:0/2657072933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c40b6610 0x7f11c40cf4d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:24.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.228+0000 7f11cc1ff700 1 --2- 172.21.15.98:0/2657072933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11c40cd8d0 0x7f11c40b60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:24.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.228+0000 7f11cc1ff700 1 -- 172.21.15.98:0/2657072933 >> 172.21.15.98:0/2657072933 conn(0x7f11c406aee0 msgr2=0x7f11c406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:24.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.228+0000 7f11cc1ff700 1 -- 172.21.15.98:0/2657072933 shutdown_connections 2023-03-03T07:57:24.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.228+0000 7f11cc1ff700 1 -- 172.21.15.98:0/2657072933 wait complete. 2023-03-03T07:57:24.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.230+0000 7f11cc1ff700 1 Processor -- start 2023-03-03T07:57:24.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.231+0000 7f11cc1ff700 1 -- start start 2023-03-03T07:57:24.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.232+0000 7f11cc1ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11c40b6610 0x7f11c41563e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:24.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.232+0000 7f11cc1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11c40cd8d0 0x7f11c40ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:24.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.232+0000 7f11cc1ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c415cc00 0x7f11c40ca890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:24.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.232+0000 7f11cc1ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40cb2d0 con 0x7f11c415cc00 2023-03-03T07:57:24.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.232+0000 7f11cc1ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40b5470 con 0x7f11c40b6610 2023-03-03T07:57:24.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.232+0000 7f11cc1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c415dbb0 con 0x7f11c40cd8d0 2023-03-03T07:57:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.233+0000 7f11ca79c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c415cc00 0x7f11c40ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.233+0000 7f11ca79c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c415cc00 0x7f11c40ca890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45606/0 (socket says 172.21.15.98:45606) 2023-03-03T07:57:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.233+0000 7f11ca79c700 1 -- 172.21.15.98:0/742878340 learned_addr learned my addr 172.21.15.98:0/742878340 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.233+0000 7f11c979a700 1 --2- 172.21.15.98:0/742878340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11c40cd8d0 0x7f11c40ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.233+0000 7f11c9f9b700 1 --2- 172.21.15.98:0/742878340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11c40b6610 0x7f11c41563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.233+0000 7f11ca79c700 1 -- 172.21.15.98:0/742878340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11c40cd8d0 msgr2=0x7f11c40ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.233+0000 7f11ca79c700 1 --2- 172.21.15.98:0/742878340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11c40cd8d0 0x7f11c40ca350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.233+0000 7f11ca79c700 1 -- 172.21.15.98:0/742878340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11c40b6610 msgr2=0x7f11c41563e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.233+0000 7f11ca79c700 1 --2- 172.21.15.98:0/742878340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11c40b6610 0x7f11c41563e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.233+0000 7f11ca79c700 1 -- 172.21.15.98:0/742878340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc009b30 con 0x7f11c415cc00 2023-03-03T07:57:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.233+0000 7f11c9f9b700 1 --2- 172.21.15.98:0/742878340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11c40b6610 0x7f11c41563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.233+0000 7f11ca79c700 1 --2- 172.21.15.98:0/742878340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c415cc00 0x7f11c40ca890 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f11c40d0a60 tx=0x7f11bc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.234+0000 7f11ba7fc700 1 -- 172.21.15.98:0/742878340 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc004870 con 0x7f11c415cc00 2023-03-03T07:57:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.234+0000 7f11ba7fc700 1 -- 172.21.15.98:0/742878340 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11bc0099b0 con 0x7f11c415cc00 2023-03-03T07:57:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.234+0000 7f11ba7fc700 1 -- 172.21.15.98:0/742878340 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc01b670 con 0x7f11c415cc00 2023-03-03T07:57:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.234+0000 7f11cc1ff700 1 -- 172.21.15.98:0/742878340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c40d22a0 con 0x7f11c415cc00 2023-03-03T07:57:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.234+0000 7f11cc1ff700 1 -- 172.21.15.98:0/742878340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c40d1fc0 con 0x7f11c415cc00 2023-03-03T07:57:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.235+0000 7f11cc1ff700 1 -- 172.21.15.98:0/742878340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c40cdd40 con 0x7f11c415cc00 2023-03-03T07:57:24.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.237+0000 7f11ba7fc700 1 -- 172.21.15.98:0/742878340 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f11bc02b070 con 0x7f11c415cc00 2023-03-03T07:57:24.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.238+0000 7f11ba7fc700 1 --2- 172.21.15.98:0/742878340 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f11a80682f0 0x7f11a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:24.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.238+0000 7f11ba7fc700 1 -- 172.21.15.98:0/742878340 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f11bc087540 con 0x7f11c415cc00 2023-03-03T07:57:24.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.238+0000 7f11c9f9b700 1 --2- 172.21.15.98:0/742878340 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f11a80682f0 0x7f11a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:24.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.239+0000 7f11c9f9b700 1 --2- 172.21.15.98:0/742878340 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f11a80682f0 0x7f11a806a7a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f11c0000f60 tx=0x7f11c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:24.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.242+0000 7f11ba7fc700 1 -- 172.21.15.98:0/742878340 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f11bc053cb0 con 0x7f11c415cc00 2023-03-03T07:57:24.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.450+0000 7f11cc1ff700 1 -- 172.21.15.98:0/742878340 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11c40729b0 con 0x7f11a80682f0 2023-03-03T07:57:24.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.459+0000 7f11ba7fc700 1 -- 172.21.15.98:0/742878340 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310563 (secure 0 0 0) 0x7f11c40729b0 con 0x7f11a80682f0 2023-03-03T07:57:24.464 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:24.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.464+0000 7f11cc1ff700 1 -- 172.21.15.98:0/742878340 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f11a80682f0 msgr2=0x7f11a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:24.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.464+0000 7f11cc1ff700 1 --2- 172.21.15.98:0/742878340 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f11a80682f0 0x7f11a806a7a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f11c0000f60 tx=0x7f11c0009040 comp rx=0 tx=0).stop 2023-03-03T07:57:24.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.464+0000 7f11cc1ff700 1 -- 172.21.15.98:0/742878340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c415cc00 msgr2=0x7f11c40ca890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:24.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.464+0000 7f11cc1ff700 1 --2- 172.21.15.98:0/742878340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c415cc00 0x7f11c40ca890 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f11c40d0a60 tx=0x7f11bc004ad0 comp rx=0 tx=0).stop 2023-03-03T07:57:24.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.464+0000 7f11cc1ff700 1 -- 172.21.15.98:0/742878340 shutdown_connections 2023-03-03T07:57:24.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.464+0000 7f11cc1ff700 1 --2- 172.21.15.98:0/742878340 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f11a80682f0 0x7f11a806a7a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:24.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.464+0000 7f11cc1ff700 1 --2- 172.21.15.98:0/742878340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c415cc00 0x7f11c40ca890 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:24.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.464+0000 7f11cc1ff700 1 --2- 172.21.15.98:0/742878340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11c40b6610 0x7f11c41563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:24.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.464+0000 7f11cc1ff700 1 --2- 172.21.15.98:0/742878340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11c40cd8d0 0x7f11c40ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:24.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.464+0000 7f11cc1ff700 1 -- 172.21.15.98:0/742878340 >> 172.21.15.98:0/742878340 conn(0x7f11c406aee0 msgr2=0x7f11c40b2170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:24.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.464+0000 7f11cc1ff700 1 -- 172.21.15.98:0/742878340 shutdown_connections 2023-03-03T07:57:24.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:24.464+0000 7f11cc1ff700 1 -- 172.21.15.98:0/742878340 wait complete. 2023-03-03T07:57:24.468 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T07:57:24.493 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2023-03-03T07:57:24.197928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":70072,"kb_used_data":4600,"kb_used_omap":0,"kb_used_meta":65408,"kb_avail":754904648,"statfs":{"total":773094113280,"available":773022359552,"internally_reserved":0,"allocated":4710400,"data_stored":2627704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66977792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.469038"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.331107+0000","last_change":"2023-03-03T07:57:17.326551+0000","last_active":"2023-03-03T07:57:17.331107+0000","last_peered":"2023-03-03T07:57:17.331107+0000","last_clean":"2023-03-03T07:57:17.331107+0000","last_became_active":"2023-03-03T07:57:17.326309+0000","last_became_peered":"2023-03-03T07:57:17.326309+0000","last_unstale":"2023-03-03T07:57:17.331107+0000","last_undegraded":"2023-03-03T07:57:17.331107+0000","last_fullsized":"2023-03-03T07:57:17.331107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:39:27.674460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.327863+0000","last_change":"2023-03-03T07:57:17.323983+0000","last_active":"2023-03-03T07:57:17.327863+0000","last_peered":"2023-03-03T07:57:17.327863+0000","last_clean":"2023-03-03T07:57:17.327863+0000","last_became_active":"2023-03-03T07:57:17.323715+0000","last_became_peered":"2023-03-03T07:57:17.323715+0000","last_unstale":"2023-03-03T07:57:17.327863+0000","last_undegraded":"2023-03-03T07:57:17.327863+0000","last_fullsized":"2023-03-03T07:57:17.327863+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.329262+0000","last_change":"2023-03-03T07:57:17.324163+0000","last_active":"2023-03-03T07:57:17.329262+0000","last_peered":"2023-03-03T07:57:17.329262+0000","last_clean":"2023-03-03T07:57:17.329262+0000","last_became_active":"2023-03-03T07:57:17.323910+0000","last_became_peered":"2023-03-03T07:57:17.323910+0000","last_unstale":"2023-03-03T07:57:17.329262+0000","last_undegraded":"2023-03-03T07:57:17.329262+0000","last_fullsized":"2023-03-03T07:57:17.329262+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335677+0000","last_change":"2023-03-03T07:57:17.334114+0000","last_active":"2023-03-03T07:57:17.335677+0000","last_peered":"2023-03-03T07:57:17.335677+0000","last_clean":"2023-03-03T07:57:17.335677+0000","last_became_active":"2023-03-03T07:57:17.333860+0000","last_became_peered":"2023-03-03T07:57:17.333860+0000","last_unstale":"2023-03-03T07:57:17.335677+0000","last_undegraded":"2023-03-03T07:57:17.335677+0000","last_fullsized":"2023-03-03T07:57:17.335677+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:07.498551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335495+0000","last_change":"2023-03-03T07:57:17.332370+0000","last_active":"2023-03-03T07:57:17.335495+0000","last_peered":"2023-03-03T07:57:17.335495+0000","last_clean":"2023-03-03T07:57:17.335495+0000","last_became_active":"2023-03-03T07:57:17.332113+0000","last_became_peered":"2023-03-03T07:57:17.332113+0000","last_unstale":"2023-03-03T07:57:17.335495+0000","last_undegraded":"2023-03-03T07:57:17.335495+0000","last_fullsized":"2023-03-03T07:57:17.335495+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:37:50.701337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.320582+0000","last_change":"2023-03-03T07:57:17.316803+0000","last_active":"2023-03-03T07:57:17.320582+0000","last_peered":"2023-03-03T07:57:17.320582+0000","last_clean":"2023-03-03T07:57:17.320582+0000","last_became_active":"2023-03-03T07:57:17.316519+0000","last_became_peered":"2023-03-03T07:57:17.316519+0000","last_unstale":"2023-03-03T07:57:17.320582+0000","last_undegraded":"2023-03-03T07:57:17.320582+0000","last_fullsized":"2023-03-03T07:57:17.320582+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:25.355832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.327143+0000","last_change":"2023-03-03T07:57:17.321448+0000","last_active":"2023-03-03T07:57:17.327143+0000","last_peered":"2023-03-03T07:57:17.327143+0000","last_clean":"2023-03-03T07:57:17.327143+0000","last_became_active":"2023-03-03T07:57:17.321158+0000","last_became_peered":"2023-03-03T07:57:17.321158+0000","last_unstale":"2023-03-03T07:57:17.327143+0000","last_undegraded":"2023-03-03T07:57:17.327143+0000","last_fullsized":"2023-03-03T07:57:17.327143+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:44.621837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.331718+0000","last_change":"2023-03-03T07:57:17.327725+0000","last_active":"2023-03-03T07:57:17.331718+0000","last_peered":"2023-03-03T07:57:17.331718+0000","last_clean":"2023-03-03T07:57:17.331718+0000","last_became_active":"2023-03-03T07:57:17.327468+0000","last_became_peered":"2023-03-03T07:57:17.327468+0000","last_unstale":"2023-03-03T07:57:17.331718+0000","last_undegraded":"2023-03-03T07:57:17.331718+0000","last_fullsized":"2023-03-03T07:57:17.331718+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:01.871020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.329272+0000","last_change":"2023-03-03T07:57:17.322904+0000","last_active":"2023-03-03T07:57:17.329272+0000","last_peered":"2023-03-03T07:57:17.329272+0000","last_clean":"2023-03-03T07:57:17.329272+0000","last_became_active":"2023-03-03T07:57:17.322466+0000","last_became_peered":"2023-03-03T07:57:17.322466+0000","last_unstale":"2023-03-03T07:57:17.329272+0000","last_undegraded":"2023-03-03T07:57:17.329272+0000","last_fullsized":"2023-03-03T07:57:17.329272+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.317840+0000","last_change":"2023-03-03T07:57:17.312858+0000","last_active":"2023-03-03T07:57:17.317840+0000","last_peered":"2023-03-03T07:57:17.317840+0000","last_clean":"2023-03-03T07:57:17.317840+0000","last_became_active":"2023-03-03T07:57:17.312552+0000","last_became_peered":"2023-03-03T07:57:17.312552+0000","last_unstale":"2023-03-03T07:57:17.317840+0000","last_undegraded":"2023-03-03T07:57:17.317840+0000","last_fullsized":"2023-03-03T07:57:17.317840+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.334638+0000","last_change":"2023-03-03T07:57:17.328548+0000","last_active":"2023-03-03T07:57:17.334638+0000","last_peered":"2023-03-03T07:57:17.334638+0000","last_clean":"2023-03-03T07:57:17.334638+0000","last_became_active":"2023-03-03T07:57:17.328277+0000","last_became_peered":"2023-03-03T07:57:17.328277+0000","last_unstale":"2023-03-03T07:57:17.334638+0000","last_undegraded":"2023-03-03T07:57:17.334638+0000","last_fullsized":"2023-03-03T07:57:17.334638+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.328669+0000","last_change":"2023-03-03T07:57:17.320812+0000","last_active":"2023-03-03T07:57:17.328669+0000","last_peered":"2023-03-03T07:57:17.328669+0000","last_clean":"2023-03-03T07:57:17.328669+0000","last_became_active":"2023-03-03T07:57:17.320504+0000","last_became_peered":"2023-03-03T07:57:17.320504+0000","last_unstale":"2023-03-03T07:57:17.328669+0000","last_undegraded":"2023-03-03T07:57:17.328669+0000","last_fullsized":"2023-03-03T07:57:17.328669+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.317802+0000","last_change":"2023-03-03T07:57:17.312825+0000","last_active":"2023-03-03T07:57:17.317802+0000","last_peered":"2023-03-03T07:57:17.317802+0000","last_clean":"2023-03-03T07:57:17.317802+0000","last_became_active":"2023-03-03T07:57:17.312529+0000","last_became_peered":"2023-03-03T07:57:17.312529+0000","last_unstale":"2023-03-03T07:57:17.317802+0000","last_undegraded":"2023-03-03T07:57:17.317802+0000","last_fullsized":"2023-03-03T07:57:17.317802+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335390+0000","last_change":"2023-03-03T07:57:17.328800+0000","last_active":"2023-03-03T07:57:17.335390+0000","last_peered":"2023-03-03T07:57:17.335390+0000","last_clean":"2023-03-03T07:57:17.335390+0000","last_became_active":"2023-03-03T07:57:17.328177+0000","last_became_peered":"2023-03-03T07:57:17.328177+0000","last_unstale":"2023-03-03T07:57:17.335390+0000","last_undegraded":"2023-03-03T07:57:17.335390+0000","last_fullsized":"2023-03-03T07:57:17.335390+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:48:58.988637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.330279+0000","last_change":"2023-03-03T07:57:17.322362+0000","last_active":"2023-03-03T07:57:17.330279+0000","last_peered":"2023-03-03T07:57:17.330279+0000","last_clean":"2023-03-03T07:57:17.330279+0000","last_became_active":"2023-03-03T07:57:17.322087+0000","last_became_peered":"2023-03-03T07:57:17.322087+0000","last_unstale":"2023-03-03T07:57:17.330279+0000","last_undegraded":"2023-03-03T07:57:17.330279+0000","last_fullsized":"2023-03-03T07:57:17.330279+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:24:04.359311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.328654+0000","last_change":"2023-03-03T07:57:17.322950+0000","last_active":"2023-03-03T07:57:17.328654+0000","last_peered":"2023-03-03T07:57:17.328654+0000","last_clean":"2023-03-03T07:57:17.328654+0000","last_became_active":"2023-03-03T07:57:17.322228+0000","last_became_peered":"2023-03-03T07:57:17.322228+0000","last_unstale":"2023-03-03T07:57:17.328654+0000","last_undegraded":"2023-03-03T07:57:17.328654+0000","last_fullsized":"2023-03-03T07:57:17.328654+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.325795+0000","last_change":"2023-03-03T07:57:17.321739+0000","last_active":"2023-03-03T07:57:17.325795+0000","last_peered":"2023-03-03T07:57:17.325795+0000","last_clean":"2023-03-03T07:57:17.325795+0000","last_became_active":"2023-03-03T07:57:17.321373+0000","last_became_peered":"2023-03-03T07:57:17.321373+0000","last_unstale":"2023-03-03T07:57:17.325795+0000","last_undegraded":"2023-03-03T07:57:17.325795+0000","last_fullsized":"2023-03-03T07:57:17.325795+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.332188+0000","last_change":"2023-03-03T07:57:17.328635+0000","last_active":"2023-03-03T07:57:17.332188+0000","last_peered":"2023-03-03T07:57:17.332188+0000","last_clean":"2023-03-03T07:57:17.332188+0000","last_became_active":"2023-03-03T07:57:17.328129+0000","last_became_peered":"2023-03-03T07:57:17.328129+0000","last_unstale":"2023-03-03T07:57:17.332188+0000","last_undegraded":"2023-03-03T07:57:17.332188+0000","last_fullsized":"2023-03-03T07:57:17.332188+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:56:10.139056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.321964+0000","last_change":"2023-03-03T07:57:17.319110+0000","last_active":"2023-03-03T07:57:17.321964+0000","last_peered":"2023-03-03T07:57:17.321964+0000","last_clean":"2023-03-03T07:57:17.321964+0000","last_became_active":"2023-03-03T07:57:17.318212+0000","last_became_peered":"2023-03-03T07:57:17.318212+0000","last_unstale":"2023-03-03T07:57:17.321964+0000","last_undegraded":"2023-03-03T07:57:17.321964+0000","last_fullsized":"2023-03-03T07:57:17.321964+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:00:23.450170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.332401+0000","last_change":"2023-03-03T07:57:17.328451+0000","last_active":"2023-03-03T07:57:17.332401+0000","last_peered":"2023-03-03T07:57:17.332401+0000","last_clean":"2023-03-03T07:57:17.332401+0000","last_became_active":"2023-03-03T07:57:17.328048+0000","last_became_peered":"2023-03-03T07:57:17.328048+0000","last_unstale":"2023-03-03T07:57:17.332401+0000","last_undegraded":"2023-03-03T07:57:17.332401+0000","last_fullsized":"2023-03-03T07:57:17.332401+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:15.531132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.320596+0000","last_change":"2023-03-03T07:57:17.315034+0000","last_active":"2023-03-03T07:57:17.320596+0000","last_peered":"2023-03-03T07:57:17.320596+0000","last_clean":"2023-03-03T07:57:17.320596+0000","last_became_active":"2023-03-03T07:57:17.314760+0000","last_became_peered":"2023-03-03T07:57:17.314760+0000","last_unstale":"2023-03-03T07:57:17.320596+0000","last_undegraded":"2023-03-03T07:57:17.320596+0000","last_fullsized":"2023-03-03T07:57:17.320596+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.336425+0000","last_change":"2023-03-03T07:57:17.334238+0000","last_active":"2023-03-03T07:57:17.336425+0000","last_peered":"2023-03-03T07:57:17.336425+0000","last_clean":"2023-03-03T07:57:17.336425+0000","last_became_active":"2023-03-03T07:57:17.333913+0000","last_became_peered":"2023-03-03T07:57:17.333913+0000","last_unstale":"2023-03-03T07:57:17.336425+0000","last_undegraded":"2023-03-03T07:57:17.336425+0000","last_fullsized":"2023-03-03T07:57:17.336425+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:39:27.674460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.332819+0000","last_change":"2023-03-03T07:57:17.323585+0000","last_active":"2023-03-03T07:57:17.332819+0000","last_peered":"2023-03-03T07:57:17.332819+0000","last_clean":"2023-03-03T07:57:17.332819+0000","last_became_active":"2023-03-03T07:57:17.323266+0000","last_became_peered":"2023-03-03T07:57:17.323266+0000","last_unstale":"2023-03-03T07:57:17.332819+0000","last_undegraded":"2023-03-03T07:57:17.332819+0000","last_fullsized":"2023-03-03T07:57:17.332819+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.319841+0000","last_change":"2023-03-03T07:57:17.310071+0000","last_active":"2023-03-03T07:57:17.319841+0000","last_peered":"2023-03-03T07:57:17.319841+0000","last_clean":"2023-03-03T07:57:17.319841+0000","last_became_active":"2023-03-03T07:57:17.309133+0000","last_became_peered":"2023-03-03T07:57:17.309133+0000","last_unstale":"2023-03-03T07:57:17.319841+0000","last_undegraded":"2023-03-03T07:57:17.319841+0000","last_fullsized":"2023-03-03T07:57:17.319841+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:57:04.542617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.320681+0000","last_change":"2023-03-03T07:57:17.316824+0000","last_active":"2023-03-03T07:57:17.320681+0000","last_peered":"2023-03-03T07:57:17.320681+0000","last_clean":"2023-03-03T07:57:17.320681+0000","last_became_active":"2023-03-03T07:57:17.316537+0000","last_became_peered":"2023-03-03T07:57:17.316537+0000","last_unstale":"2023-03-03T07:57:17.320681+0000","last_undegraded":"2023-03-03T07:57:17.320681+0000","last_fullsized":"2023-03-03T07:57:17.320681+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:57:04.542617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.320908+0000","last_change":"2023-03-03T07:57:17.317046+0000","last_active":"2023-03-03T07:57:17.320908+0000","last_peered":"2023-03-03T07:57:17.320908+0000","last_clean":"2023-03-03T07:57:17.320908+0000","last_became_active":"2023-03-03T07:57:17.316775+0000","last_became_peered":"2023-03-03T07:57:17.316775+0000","last_unstale":"2023-03-03T07:57:17.320908+0000","last_undegraded":"2023-03-03T07:57:17.320908+0000","last_fullsized":"2023-03-03T07:57:17.320908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.325925+0000","last_change":"2023-03-03T07:57:17.324059+0000","last_active":"2023-03-03T07:57:17.325925+0000","last_peered":"2023-03-03T07:57:17.325925+0000","last_clean":"2023-03-03T07:57:17.325925+0000","last_became_active":"2023-03-03T07:57:17.323486+0000","last_became_peered":"2023-03-03T07:57:17.323486+0000","last_unstale":"2023-03-03T07:57:17.325925+0000","last_undegraded":"2023-03-03T07:57:17.325925+0000","last_fullsized":"2023-03-03T07:57:17.325925+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:20.464414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.322541+0000","last_change":"2023-03-03T07:57:17.319788+0000","last_active":"2023-03-03T07:57:17.322541+0000","last_peered":"2023-03-03T07:57:17.322541+0000","last_clean":"2023-03-03T07:57:17.322541+0000","last_became_active":"2023-03-03T07:57:17.319506+0000","last_became_peered":"2023-03-03T07:57:17.319506+0000","last_unstale":"2023-03-03T07:57:17.322541+0000","last_undegraded":"2023-03-03T07:57:17.322541+0000","last_fullsized":"2023-03-03T07:57:17.322541+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.321276+0000","last_change":"2023-03-03T07:57:17.318816+0000","last_active":"2023-03-03T07:57:17.321276+0000","last_peered":"2023-03-03T07:57:17.321276+0000","last_clean":"2023-03-03T07:57:17.321276+0000","last_became_active":"2023-03-03T07:57:17.318429+0000","last_became_peered":"2023-03-03T07:57:17.318429+0000","last_unstale":"2023-03-03T07:57:17.321276+0000","last_undegraded":"2023-03-03T07:57:17.321276+0000","last_fullsized":"2023-03-03T07:57:17.321276+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:07.498551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.321241+0000","last_change":"2023-03-03T07:57:17.318945+0000","last_active":"2023-03-03T07:57:17.321241+0000","last_peered":"2023-03-03T07:57:17.321241+0000","last_clean":"2023-03-03T07:57:17.321241+0000","last_became_active":"2023-03-03T07:57:17.318619+0000","last_became_peered":"2023-03-03T07:57:17.318619+0000","last_unstale":"2023-03-03T07:57:17.321241+0000","last_undegraded":"2023-03-03T07:57:17.321241+0000","last_fullsized":"2023-03-03T07:57:17.321241+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:43.532953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.332800+0000","last_change":"2023-03-03T07:57:17.323441+0000","last_active":"2023-03-03T07:57:17.332800+0000","last_peered":"2023-03-03T07:57:17.332800+0000","last_clean":"2023-03-03T07:57:17.332800+0000","last_became_active":"2023-03-03T07:57:17.323125+0000","last_became_peered":"2023-03-03T07:57:17.323125+0000","last_unstale":"2023-03-03T07:57:17.332800+0000","last_undegraded":"2023-03-03T07:57:17.332800+0000","last_fullsized":"2023-03-03T07:57:17.332800+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.320990+0000","last_change":"2023-03-03T07:57:17.317951+0000","last_active":"2023-03-03T07:57:17.320990+0000","last_peered":"2023-03-03T07:57:17.320990+0000","last_clean":"2023-03-03T07:57:17.320990+0000","last_became_active":"2023-03-03T07:57:17.317678+0000","last_became_peered":"2023-03-03T07:57:17.317678+0000","last_unstale":"2023-03-03T07:57:17.320990+0000","last_undegraded":"2023-03-03T07:57:17.320990+0000","last_fullsized":"2023-03-03T07:57:17.320990+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.655223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.324564+0000","last_change":"2023-03-03T07:57:17.321833+0000","last_active":"2023-03-03T07:57:17.324564+0000","last_peered":"2023-03-03T07:57:17.324564+0000","last_clean":"2023-03-03T07:57:17.324564+0000","last_became_active":"2023-03-03T07:57:17.321543+0000","last_became_peered":"2023-03-03T07:57:17.321543+0000","last_unstale":"2023-03-03T07:57:17.324564+0000","last_undegraded":"2023-03-03T07:57:17.324564+0000","last_fullsized":"2023-03-03T07:57:17.324564+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.334383+0000","last_change":"2023-03-03T07:57:16.745475+0000","last_active":"2023-03-03T07:57:17.334383+0000","last_peered":"2023-03-03T07:57:17.334383+0000","last_clean":"2023-03-03T07:57:17.334383+0000","last_became_active":"2023-03-03T07:57:16.745159+0000","last_became_peered":"2023-03-03T07:57:16.745159+0000","last_unstale":"2023-03-03T07:57:17.334383+0000","last_undegraded":"2023-03-03T07:57:17.334383+0000","last_fullsized":"2023-03-03T07:57:17.334383+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:45:49.372030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.330939+0000","last_change":"2023-03-03T07:57:17.325195+0000","last_active":"2023-03-03T07:57:17.330939+0000","last_peered":"2023-03-03T07:57:17.330939+0000","last_clean":"2023-03-03T07:57:17.330939+0000","last_became_active":"2023-03-03T07:57:17.324904+0000","last_became_peered":"2023-03-03T07:57:17.324904+0000","last_unstale":"2023-03-03T07:57:17.330939+0000","last_undegraded":"2023-03-03T07:57:17.330939+0000","last_fullsized":"2023-03-03T07:57:17.330939+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.323043+0000","last_change":"2023-03-03T07:57:16.755153+0000","last_active":"2023-03-03T07:57:17.323043+0000","last_peered":"2023-03-03T07:57:17.323043+0000","last_clean":"2023-03-03T07:57:17.323043+0000","last_became_active":"2023-03-03T07:57:16.754738+0000","last_became_peered":"2023-03-03T07:57:16.754738+0000","last_unstale":"2023-03-03T07:57:17.323043+0000","last_undegraded":"2023-03-03T07:57:17.323043+0000","last_fullsized":"2023-03-03T07:57:17.323043+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:14.522208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335341+0000","last_change":"2023-03-03T07:57:17.327812+0000","last_active":"2023-03-03T07:57:17.335341+0000","last_peered":"2023-03-03T07:57:17.335341+0000","last_clean":"2023-03-03T07:57:17.335341+0000","last_became_active":"2023-03-03T07:57:17.327531+0000","last_became_peered":"2023-03-03T07:57:17.327531+0000","last_unstale":"2023-03-03T07:57:17.335341+0000","last_undegraded":"2023-03-03T07:57:17.335341+0000","last_fullsized":"2023-03-03T07:57:17.335341+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:11.090994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.333541+0000","last_change":"2023-03-03T07:57:16.746651+0000","last_active":"2023-03-03T07:57:17.333541+0000","last_peered":"2023-03-03T07:57:17.333541+0000","last_clean":"2023-03-03T07:57:17.333541+0000","last_became_active":"2023-03-03T07:57:16.746390+0000","last_became_peered":"2023-03-03T07:57:16.746390+0000","last_unstale":"2023-03-03T07:57:17.333541+0000","last_undegraded":"2023-03-03T07:57:17.333541+0000","last_fullsized":"2023-03-03T07:57:17.333541+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:06.489627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.327422+0000","last_change":"2023-03-03T07:57:17.324462+0000","last_active":"2023-03-03T07:57:17.327422+0000","last_peered":"2023-03-03T07:57:17.327422+0000","last_clean":"2023-03-03T07:57:17.327422+0000","last_became_active":"2023-03-03T07:57:17.324090+0000","last_became_peered":"2023-03-03T07:57:17.324090+0000","last_unstale":"2023-03-03T07:57:17.327422+0000","last_undegraded":"2023-03-03T07:57:17.327422+0000","last_fullsized":"2023-03-03T07:57:17.327422+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:11.090994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.333541+0000","last_change":"2023-03-03T07:57:16.743191+0000","last_active":"2023-03-03T07:57:17.333541+0000","last_peered":"2023-03-03T07:57:17.333541+0000","last_clean":"2023-03-03T07:57:17.333541+0000","last_became_active":"2023-03-03T07:57:16.741858+0000","last_became_peered":"2023-03-03T07:57:16.741858+0000","last_unstale":"2023-03-03T07:57:17.333541+0000","last_undegraded":"2023-03-03T07:57:17.333541+0000","last_fullsized":"2023-03-03T07:57:17.333541+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:00.944189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.324965+0000","last_change":"2023-03-03T07:57:17.319886+0000","last_active":"2023-03-03T07:57:17.324965+0000","last_peered":"2023-03-03T07:57:17.324965+0000","last_clean":"2023-03-03T07:57:17.324965+0000","last_became_active":"2023-03-03T07:57:17.319614+0000","last_became_peered":"2023-03-03T07:57:17.319614+0000","last_unstale":"2023-03-03T07:57:17.324965+0000","last_undegraded":"2023-03-03T07:57:17.324965+0000","last_fullsized":"2023-03-03T07:57:17.324965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.325981+0000","last_change":"2023-03-03T07:57:16.753759+0000","last_active":"2023-03-03T07:57:17.325981+0000","last_peered":"2023-03-03T07:57:17.325981+0000","last_clean":"2023-03-03T07:57:17.325981+0000","last_became_active":"2023-03-03T07:57:16.753449+0000","last_became_peered":"2023-03-03T07:57:16.753449+0000","last_unstale":"2023-03-03T07:57:17.325981+0000","last_undegraded":"2023-03-03T07:57:17.325981+0000","last_fullsized":"2023-03-03T07:57:17.325981+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:56:09.130132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.322448+0000","last_change":"2023-03-03T07:57:17.317169+0000","last_active":"2023-03-03T07:57:17.322448+0000","last_peered":"2023-03-03T07:57:17.322448+0000","last_clean":"2023-03-03T07:57:17.322448+0000","last_became_active":"2023-03-03T07:57:17.316817+0000","last_became_peered":"2023-03-03T07:57:17.316817+0000","last_unstale":"2023-03-03T07:57:17.322448+0000","last_undegraded":"2023-03-03T07:57:17.322448+0000","last_fullsized":"2023-03-03T07:57:17.322448+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.336157+0000","last_change":"2023-03-03T07:57:16.740511+0000","last_active":"2023-03-03T07:57:17.336157+0000","last_peered":"2023-03-03T07:57:17.336157+0000","last_clean":"2023-03-03T07:57:17.336157+0000","last_became_active":"2023-03-03T07:57:16.740221+0000","last_became_peered":"2023-03-03T07:57:16.740221+0000","last_unstale":"2023-03-03T07:57:17.336157+0000","last_undegraded":"2023-03-03T07:57:17.336157+0000","last_fullsized":"2023-03-03T07:57:17.336157+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:10.938115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.336403+0000","last_change":"2023-03-03T07:57:17.334046+0000","last_active":"2023-03-03T07:57:17.336403+0000","last_peered":"2023-03-03T07:57:17.336403+0000","last_clean":"2023-03-03T07:57:17.336403+0000","last_became_active":"2023-03-03T07:57:17.333764+0000","last_became_peered":"2023-03-03T07:57:17.333764+0000","last_unstale":"2023-03-03T07:57:17.336403+0000","last_undegraded":"2023-03-03T07:57:17.336403+0000","last_fullsized":"2023-03-03T07:57:17.336403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.303992+0000","last_change":"2023-03-03T07:57:16.756171+0000","last_active":"2023-03-03T07:57:17.303992+0000","last_peered":"2023-03-03T07:57:17.303992+0000","last_clean":"2023-03-03T07:57:17.303992+0000","last_became_active":"2023-03-03T07:57:16.755894+0000","last_became_peered":"2023-03-03T07:57:16.755894+0000","last_unstale":"2023-03-03T07:57:17.303992+0000","last_undegraded":"2023-03-03T07:57:17.303992+0000","last_fullsized":"2023-03-03T07:57:17.303992+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:19.455490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.332685+0000","last_change":"2023-03-03T07:57:17.330901+0000","last_active":"2023-03-03T07:57:17.332685+0000","last_peered":"2023-03-03T07:57:17.332685+0000","last_clean":"2023-03-03T07:57:17.332685+0000","last_became_active":"2023-03-03T07:57:17.330613+0000","last_became_peered":"2023-03-03T07:57:17.330613+0000","last_unstale":"2023-03-03T07:57:17.332685+0000","last_undegraded":"2023-03-03T07:57:17.332685+0000","last_fullsized":"2023-03-03T07:57:17.332685+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:48:58.988637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.312998+0000","last_change":"2023-03-03T07:57:16.744111+0000","last_active":"2023-03-03T07:57:17.312998+0000","last_peered":"2023-03-03T07:57:17.312998+0000","last_clean":"2023-03-03T07:57:17.312998+0000","last_became_active":"2023-03-03T07:57:16.743778+0000","last_became_peered":"2023-03-03T07:57:16.743778+0000","last_unstale":"2023-03-03T07:57:17.312998+0000","last_undegraded":"2023-03-03T07:57:17.312998+0000","last_fullsized":"2023-03-03T07:57:17.312998+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:52:59.206045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.334869+0000","last_change":"2023-03-03T07:57:17.331846+0000","last_active":"2023-03-03T07:57:17.334869+0000","last_peered":"2023-03-03T07:57:17.334869+0000","last_clean":"2023-03-03T07:57:17.334869+0000","last_became_active":"2023-03-03T07:57:17.331542+0000","last_became_peered":"2023-03-03T07:57:17.331542+0000","last_unstale":"2023-03-03T07:57:17.334869+0000","last_undegraded":"2023-03-03T07:57:17.334869+0000","last_fullsized":"2023-03-03T07:57:17.334869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:11.947039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.314856+0000","last_change":"2023-03-03T07:57:16.744161+0000","last_active":"2023-03-03T07:57:17.314856+0000","last_peered":"2023-03-03T07:57:17.314856+0000","last_clean":"2023-03-03T07:57:17.314856+0000","last_became_active":"2023-03-03T07:57:16.743909+0000","last_became_peered":"2023-03-03T07:57:16.743909+0000","last_unstale":"2023-03-03T07:57:17.314856+0000","last_undegraded":"2023-03-03T07:57:17.314856+0000","last_fullsized":"2023-03-03T07:57:17.314856+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:43:42.175275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.333622+0000","last_change":"2023-03-03T07:57:17.324785+0000","last_active":"2023-03-03T07:57:17.333622+0000","last_peered":"2023-03-03T07:57:17.333622+0000","last_clean":"2023-03-03T07:57:17.333622+0000","last_became_active":"2023-03-03T07:57:17.324510+0000","last_became_peered":"2023-03-03T07:57:17.324510+0000","last_unstale":"2023-03-03T07:57:17.333622+0000","last_undegraded":"2023-03-03T07:57:17.333622+0000","last_fullsized":"2023-03-03T07:57:17.333622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.304107+0000","last_change":"2023-03-03T07:57:16.754673+0000","last_active":"2023-03-03T07:57:17.304107+0000","last_peered":"2023-03-03T07:57:17.304107+0000","last_clean":"2023-03-03T07:57:17.304107+0000","last_became_active":"2023-03-03T07:57:16.754415+0000","last_became_peered":"2023-03-03T07:57:16.754415+0000","last_unstale":"2023-03-03T07:57:17.304107+0000","last_undegraded":"2023-03-03T07:57:17.304107+0000","last_fullsized":"2023-03-03T07:57:17.304107+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:06.489627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.329305+0000","last_change":"2023-03-03T07:57:17.323654+0000","last_active":"2023-03-03T07:57:17.329305+0000","last_peered":"2023-03-03T07:57:17.329305+0000","last_clean":"2023-03-03T07:57:17.329305+0000","last_became_active":"2023-03-03T07:57:17.323356+0000","last_became_peered":"2023-03-03T07:57:17.323356+0000","last_unstale":"2023-03-03T07:57:17.329305+0000","last_undegraded":"2023-03-03T07:57:17.329305+0000","last_fullsized":"2023-03-03T07:57:17.329305+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.326392+0000","last_change":"2023-03-03T07:57:16.757208+0000","last_active":"2023-03-03T07:57:17.326392+0000","last_peered":"2023-03-03T07:57:17.326392+0000","last_clean":"2023-03-03T07:57:17.326392+0000","last_became_active":"2023-03-03T07:57:16.756910+0000","last_became_peered":"2023-03-03T07:57:16.756910+0000","last_unstale":"2023-03-03T07:57:17.326392+0000","last_undegraded":"2023-03-03T07:57:17.326392+0000","last_fullsized":"2023-03-03T07:57:17.326392+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:14.522208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335575+0000","last_change":"2023-03-03T07:57:17.330250+0000","last_active":"2023-03-03T07:57:17.335575+0000","last_peered":"2023-03-03T07:57:17.335575+0000","last_clean":"2023-03-03T07:57:17.335575+0000","last_became_active":"2023-03-03T07:57:17.329997+0000","last_became_peered":"2023-03-03T07:57:17.329997+0000","last_unstale":"2023-03-03T07:57:17.335575+0000","last_undegraded":"2023-03-03T07:57:17.335575+0000","last_fullsized":"2023-03-03T07:57:17.335575+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:01.871020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":17,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.337068+0000","last_change":"2023-03-03T07:57:16.749713+0000","last_active":"2023-03-03T07:57:17.337068+0000","last_peered":"2023-03-03T07:57:17.337068+0000","last_clean":"2023-03-03T07:57:17.337068+0000","last_became_active":"2023-03-03T07:57:16.749304+0000","last_became_peered":"2023-03-03T07:57:16.749304+0000","last_unstale":"2023-03-03T07:57:17.337068+0000","last_undegraded":"2023-03-03T07:57:17.337068+0000","last_fullsized":"2023-03-03T07:57:17.337068+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:19:07.613725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.331274+0000","last_change":"2023-03-03T07:57:17.325082+0000","last_active":"2023-03-03T07:57:17.331274+0000","last_peered":"2023-03-03T07:57:17.331274+0000","last_clean":"2023-03-03T07:57:17.331274+0000","last_became_active":"2023-03-03T07:57:17.324787+0000","last_became_peered":"2023-03-03T07:57:17.324787+0000","last_unstale":"2023-03-03T07:57:17.331274+0000","last_undegraded":"2023-03-03T07:57:17.331274+0000","last_fullsized":"2023-03-03T07:57:17.331274+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:21:12.720267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.326087+0000","last_change":"2023-03-03T07:57:16.753448+0000","last_active":"2023-03-03T07:57:17.326087+0000","last_peered":"2023-03-03T07:57:17.326087+0000","last_clean":"2023-03-03T07:57:17.326087+0000","last_became_active":"2023-03-03T07:57:16.753183+0000","last_became_peered":"2023-03-03T07:57:16.753183+0000","last_unstale":"2023-03-03T07:57:17.326087+0000","last_undegraded":"2023-03-03T07:57:17.326087+0000","last_fullsized":"2023-03-03T07:57:17.326087+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:54.897881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.327376+0000","last_change":"2023-03-03T07:57:17.319586+0000","last_active":"2023-03-03T07:57:17.327376+0000","last_peered":"2023-03-03T07:57:17.327376+0000","last_clean":"2023-03-03T07:57:17.327376+0000","last_became_active":"2023-03-03T07:57:17.319343+0000","last_became_peered":"2023-03-03T07:57:17.319343+0000","last_unstale":"2023-03-03T07:57:17.327376+0000","last_undegraded":"2023-03-03T07:57:17.327376+0000","last_fullsized":"2023-03-03T07:57:17.327376+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:24:04.359311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.304263+0000","last_change":"2023-03-03T07:57:16.754628+0000","last_active":"2023-03-03T07:57:17.304263+0000","last_peered":"2023-03-03T07:57:17.304263+0000","last_clean":"2023-03-03T07:57:17.304263+0000","last_became_active":"2023-03-03T07:57:16.754370+0000","last_became_peered":"2023-03-03T07:57:16.754370+0000","last_unstale":"2023-03-03T07:57:17.304263+0000","last_undegraded":"2023-03-03T07:57:17.304263+0000","last_fullsized":"2023-03-03T07:57:17.304263+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:14.522208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.325593+0000","last_change":"2023-03-03T07:57:17.322380+0000","last_active":"2023-03-03T07:57:17.325593+0000","last_peered":"2023-03-03T07:57:17.325593+0000","last_clean":"2023-03-03T07:57:17.325593+0000","last_became_active":"2023-03-03T07:57:17.322085+0000","last_became_peered":"2023-03-03T07:57:17.322085+0000","last_unstale":"2023-03-03T07:57:17.325593+0000","last_undegraded":"2023-03-03T07:57:17.325593+0000","last_fullsized":"2023-03-03T07:57:17.325593+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:21:12.720267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.304795+0000","last_change":"2023-03-03T07:57:16.745348+0000","last_active":"2023-03-03T07:57:17.304795+0000","last_peered":"2023-03-03T07:57:17.304795+0000","last_clean":"2023-03-03T07:57:17.304795+0000","last_became_active":"2023-03-03T07:57:16.745100+0000","last_became_peered":"2023-03-03T07:57:16.745100+0000","last_unstale":"2023-03-03T07:57:17.304795+0000","last_undegraded":"2023-03-03T07:57:17.304795+0000","last_fullsized":"2023-03-03T07:57:17.304795+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:17:28.990710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.325155+0000","last_change":"2023-03-03T07:57:17.318561+0000","last_active":"2023-03-03T07:57:17.325155+0000","last_peered":"2023-03-03T07:57:17.325155+0000","last_clean":"2023-03-03T07:57:17.325155+0000","last_became_active":"2023-03-03T07:57:17.318287+0000","last_became_peered":"2023-03-03T07:57:17.318287+0000","last_unstale":"2023-03-03T07:57:17.325155+0000","last_undegraded":"2023-03-03T07:57:17.325155+0000","last_fullsized":"2023-03-03T07:57:17.325155+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:35:32.174190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":19,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.336687+0000","last_change":"2023-03-03T07:57:16.736066+0000","last_active":"2023-03-03T07:57:17.336687+0000","last_peered":"2023-03-03T07:57:17.336687+0000","last_clean":"2023-03-03T07:57:17.336687+0000","last_became_active":"2023-03-03T07:57:16.735650+0000","last_became_peered":"2023-03-03T07:57:16.735650+0000","last_unstale":"2023-03-03T07:57:17.336687+0000","last_undegraded":"2023-03-03T07:57:17.336687+0000","last_fullsized":"2023-03-03T07:57:17.336687+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:00:22.441246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.327965+0000","last_change":"2023-03-03T07:57:17.323207+0000","last_active":"2023-03-03T07:57:17.327965+0000","last_peered":"2023-03-03T07:57:17.327965+0000","last_clean":"2023-03-03T07:57:17.327965+0000","last_became_active":"2023-03-03T07:57:17.322950+0000","last_became_peered":"2023-03-03T07:57:17.322950+0000","last_unstale":"2023-03-03T07:57:17.327965+0000","last_undegraded":"2023-03-03T07:57:17.327965+0000","last_fullsized":"2023-03-03T07:57:17.327965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:44.621837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.334024+0000","last_change":"2023-03-03T07:57:16.756645+0000","last_active":"2023-03-03T07:57:17.334024+0000","last_peered":"2023-03-03T07:57:17.334024+0000","last_clean":"2023-03-03T07:57:17.334024+0000","last_became_active":"2023-03-03T07:57:16.756201+0000","last_became_peered":"2023-03-03T07:57:16.756201+0000","last_unstale":"2023-03-03T07:57:17.334024+0000","last_undegraded":"2023-03-03T07:57:17.334024+0000","last_fullsized":"2023-03-03T07:57:17.334024+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:42.524029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.322596+0000","last_change":"2023-03-03T07:57:17.319019+0000","last_active":"2023-03-03T07:57:17.322596+0000","last_peered":"2023-03-03T07:57:17.322596+0000","last_clean":"2023-03-03T07:57:17.322596+0000","last_became_active":"2023-03-03T07:57:17.318759+0000","last_became_peered":"2023-03-03T07:57:17.318759+0000","last_unstale":"2023-03-03T07:57:17.322596+0000","last_undegraded":"2023-03-03T07:57:17.322596+0000","last_fullsized":"2023-03-03T07:57:17.322596+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:13:17.532289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.333446+0000","last_change":"2023-03-03T07:57:16.747099+0000","last_active":"2023-03-03T07:57:17.333446+0000","last_peered":"2023-03-03T07:57:17.333446+0000","last_clean":"2023-03-03T07:57:17.333446+0000","last_became_active":"2023-03-03T07:57:16.746766+0000","last_became_peered":"2023-03-03T07:57:16.746766+0000","last_unstale":"2023-03-03T07:57:17.333446+0000","last_undegraded":"2023-03-03T07:57:17.333446+0000","last_fullsized":"2023-03-03T07:57:17.333446+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:25.646299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335717+0000","last_change":"2023-03-03T07:57:17.330772+0000","last_active":"2023-03-03T07:57:17.335717+0000","last_peered":"2023-03-03T07:57:17.335717+0000","last_clean":"2023-03-03T07:57:17.335717+0000","last_became_active":"2023-03-03T07:57:17.330534+0000","last_became_peered":"2023-03-03T07:57:17.330534+0000","last_unstale":"2023-03-03T07:57:17.335717+0000","last_undegraded":"2023-03-03T07:57:17.335717+0000","last_fullsized":"2023-03-03T07:57:17.335717+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:35:32.174190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.328035+0000","last_change":"2023-03-03T07:57:16.748443+0000","last_active":"2023-03-03T07:57:17.328035+0000","last_peered":"2023-03-03T07:57:17.328035+0000","last_clean":"2023-03-03T07:57:17.328035+0000","last_became_active":"2023-03-03T07:57:16.748172+0000","last_became_peered":"2023-03-03T07:57:16.748172+0000","last_unstale":"2023-03-03T07:57:17.328035+0000","last_undegraded":"2023-03-03T07:57:17.328035+0000","last_fullsized":"2023-03-03T07:57:17.328035+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:07:16.498493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.324459+0000","last_change":"2023-03-03T07:57:17.320055+0000","last_active":"2023-03-03T07:57:17.324459+0000","last_peered":"2023-03-03T07:57:17.324459+0000","last_clean":"2023-03-03T07:57:17.324459+0000","last_became_active":"2023-03-03T07:57:17.319759+0000","last_became_peered":"2023-03-03T07:57:17.319759+0000","last_unstale":"2023-03-03T07:57:17.324459+0000","last_undegraded":"2023-03-03T07:57:17.324459+0000","last_fullsized":"2023-03-03T07:57:17.324459+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:43.532953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.303222+0000","last_change":"2023-03-03T07:57:16.746019+0000","last_active":"2023-03-03T07:57:17.303222+0000","last_peered":"2023-03-03T07:57:17.303222+0000","last_clean":"2023-03-03T07:57:17.303222+0000","last_became_active":"2023-03-03T07:57:16.745766+0000","last_became_peered":"2023-03-03T07:57:16.745766+0000","last_unstale":"2023-03-03T07:57:17.303222+0000","last_undegraded":"2023-03-03T07:57:17.303222+0000","last_fullsized":"2023-03-03T07:57:17.303222+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:42.524029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.328673+0000","last_change":"2023-03-03T07:57:17.323174+0000","last_active":"2023-03-03T07:57:17.328673+0000","last_peered":"2023-03-03T07:57:17.328673+0000","last_clean":"2023-03-03T07:57:17.328673+0000","last_became_active":"2023-03-03T07:57:17.322647+0000","last_became_peered":"2023-03-03T07:57:17.322647+0000","last_unstale":"2023-03-03T07:57:17.328673+0000","last_undegraded":"2023-03-03T07:57:17.328673+0000","last_fullsized":"2023-03-03T07:57:17.328673+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.318767+0000","last_change":"2023-03-03T07:57:16.753514+0000","last_active":"2023-03-03T07:57:17.318767+0000","last_peered":"2023-03-03T07:57:17.318767+0000","last_clean":"2023-03-03T07:57:17.318767+0000","last_became_active":"2023-03-03T07:57:16.753206+0000","last_became_peered":"2023-03-03T07:57:16.753206+0000","last_unstale":"2023-03-03T07:57:17.318767+0000","last_undegraded":"2023-03-03T07:57:17.318767+0000","last_fullsized":"2023-03-03T07:57:17.318767+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:43:26.659371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.322268+0000","last_change":"2023-03-03T07:57:17.319507+0000","last_active":"2023-03-03T07:57:17.322268+0000","last_peered":"2023-03-03T07:57:17.322268+0000","last_clean":"2023-03-03T07:57:17.322268+0000","last_became_active":"2023-03-03T07:57:17.319238+0000","last_became_peered":"2023-03-03T07:57:17.319238+0000","last_unstale":"2023-03-03T07:57:17.322268+0000","last_undegraded":"2023-03-03T07:57:17.322268+0000","last_fullsized":"2023-03-03T07:57:17.322268+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.298919+0000","last_change":"2023-03-03T07:57:16.754775+0000","last_active":"2023-03-03T07:57:17.298919+0000","last_peered":"2023-03-03T07:57:17.298919+0000","last_clean":"2023-03-03T07:57:17.298919+0000","last_became_active":"2023-03-03T07:57:16.754315+0000","last_became_peered":"2023-03-03T07:57:16.754315+0000","last_unstale":"2023-03-03T07:57:17.298919+0000","last_undegraded":"2023-03-03T07:57:17.298919+0000","last_fullsized":"2023-03-03T07:57:17.298919+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:14:32.071416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.320588+0000","last_change":"2023-03-03T07:57:17.316091+0000","last_active":"2023-03-03T07:57:17.320588+0000","last_peered":"2023-03-03T07:57:17.320588+0000","last_clean":"2023-03-03T07:57:17.320588+0000","last_became_active":"2023-03-03T07:57:17.315658+0000","last_became_peered":"2023-03-03T07:57:17.315658+0000","last_unstale":"2023-03-03T07:57:17.320588+0000","last_undegraded":"2023-03-03T07:57:17.320588+0000","last_fullsized":"2023-03-03T07:57:17.320588+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.306600+0000","last_change":"2023-03-03T07:57:16.742812+0000","last_active":"2023-03-03T07:57:17.306600+0000","last_peered":"2023-03-03T07:57:17.306600+0000","last_clean":"2023-03-03T07:57:17.306600+0000","last_became_active":"2023-03-03T07:57:16.742529+0000","last_became_peered":"2023-03-03T07:57:16.742529+0000","last_unstale":"2023-03-03T07:57:17.306600+0000","last_undegraded":"2023-03-03T07:57:17.306600+0000","last_fullsized":"2023-03-03T07:57:17.306600+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:37:49.692413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.332674+0000","last_change":"2023-03-03T07:57:17.330640+0000","last_active":"2023-03-03T07:57:17.332674+0000","last_peered":"2023-03-03T07:57:17.332674+0000","last_clean":"2023-03-03T07:57:17.332674+0000","last_became_active":"2023-03-03T07:57:17.330390+0000","last_became_peered":"2023-03-03T07:57:17.330390+0000","last_unstale":"2023-03-03T07:57:17.332674+0000","last_undegraded":"2023-03-03T07:57:17.332674+0000","last_fullsized":"2023-03-03T07:57:17.332674+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.299395+0000","last_change":"2023-03-03T07:57:16.753472+0000","last_active":"2023-03-03T07:57:17.299395+0000","last_peered":"2023-03-03T07:57:17.299395+0000","last_clean":"2023-03-03T07:57:17.299395+0000","last_became_active":"2023-03-03T07:57:16.753183+0000","last_became_peered":"2023-03-03T07:57:16.753183+0000","last_unstale":"2023-03-03T07:57:17.299395+0000","last_undegraded":"2023-03-03T07:57:17.299395+0000","last_fullsized":"2023-03-03T07:57:17.299395+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:53.094916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.331292+0000","last_change":"2023-03-03T07:57:17.328971+0000","last_active":"2023-03-03T07:57:17.331292+0000","last_peered":"2023-03-03T07:57:17.331292+0000","last_clean":"2023-03-03T07:57:17.331292+0000","last_became_active":"2023-03-03T07:57:17.328679+0000","last_became_peered":"2023-03-03T07:57:17.328679+0000","last_unstale":"2023-03-03T07:57:17.331292+0000","last_undegraded":"2023-03-03T07:57:17.331292+0000","last_fullsized":"2023-03-03T07:57:17.331292+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.307500+0000","last_change":"2023-03-03T07:57:16.751997+0000","last_active":"2023-03-03T07:57:17.307500+0000","last_peered":"2023-03-03T07:57:17.307500+0000","last_clean":"2023-03-03T07:57:17.307500+0000","last_became_active":"2023-03-03T07:57:16.751593+0000","last_became_peered":"2023-03-03T07:57:16.751593+0000","last_unstale":"2023-03-03T07:57:17.307500+0000","last_undegraded":"2023-03-03T07:57:17.307500+0000","last_fullsized":"2023-03-03T07:57:17.307500+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:19.455490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.303300+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T07:57:17.303300+0000","last_peered":"2023-03-03T07:57:17.303300+0000","last_clean":"2023-03-03T07:57:17.303300+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T07:57:17.303300+0000","last_undegraded":"2023-03-03T07:57:17.303300+0000","last_fullsized":"2023-03-03T07:57:17.303300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335374+0000","last_change":"2023-03-03T07:57:17.333984+0000","last_active":"2023-03-03T07:57:17.335374+0000","last_peered":"2023-03-03T07:57:17.335374+0000","last_clean":"2023-03-03T07:57:17.335374+0000","last_became_active":"2023-03-03T07:57:17.333692+0000","last_became_peered":"2023-03-03T07:57:17.333692+0000","last_unstale":"2023-03-03T07:57:17.335374+0000","last_undegraded":"2023-03-03T07:57:17.335374+0000","last_fullsized":"2023-03-03T07:57:17.335374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.309367+0000","last_change":"2023-03-03T07:57:16.749605+0000","last_active":"2023-03-03T07:57:17.309367+0000","last_peered":"2023-03-03T07:57:17.309367+0000","last_clean":"2023-03-03T07:57:17.309367+0000","last_became_active":"2023-03-03T07:57:16.749336+0000","last_became_peered":"2023-03-03T07:57:16.749336+0000","last_unstale":"2023-03-03T07:57:17.309367+0000","last_undegraded":"2023-03-03T07:57:17.309367+0000","last_fullsized":"2023-03-03T07:57:17.309367+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:42.524029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.303849+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T07:57:17.303849+0000","last_peered":"2023-03-03T07:57:17.303849+0000","last_clean":"2023-03-03T07:57:17.303849+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T07:57:17.303849+0000","last_undegraded":"2023-03-03T07:57:17.303849+0000","last_fullsized":"2023-03-03T07:57:17.303849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.334440+0000","last_change":"2023-03-03T07:57:17.324746+0000","last_active":"2023-03-03T07:57:17.334440+0000","last_peered":"2023-03-03T07:57:17.334440+0000","last_clean":"2023-03-03T07:57:17.334440+0000","last_became_active":"2023-03-03T07:57:17.323971+0000","last_became_peered":"2023-03-03T07:57:17.323971+0000","last_unstale":"2023-03-03T07:57:17.334440+0000","last_undegraded":"2023-03-03T07:57:17.334440+0000","last_fullsized":"2023-03-03T07:57:17.334440+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"37'1","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:18.031859+0000","last_change":"2023-03-03T07:57:16.750917+0000","last_active":"2023-03-03T07:57:18.031859+0000","last_peered":"2023-03-03T07:57:18.031859+0000","last_clean":"2023-03-03T07:57:18.031859+0000","last_became_active":"2023-03-03T07:57:16.750615+0000","last_became_peered":"2023-03-03T07:57:16.750615+0000","last_unstale":"2023-03-03T07:57:18.031859+0000","last_undegraded":"2023-03-03T07:57:18.031859+0000","last_fullsized":"2023-03-03T07:57:18.031859+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:49:51.007473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.300085+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T07:57:17.300085+0000","last_peered":"2023-03-03T07:57:17.300085+0000","last_clean":"2023-03-03T07:57:17.300085+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T07:57:17.300085+0000","last_undegraded":"2023-03-03T07:57:17.300085+0000","last_fullsized":"2023-03-03T07:57:17.300085+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.327122+0000","last_change":"2023-03-03T07:57:17.320314+0000","last_active":"2023-03-03T07:57:17.327122+0000","last_peered":"2023-03-03T07:57:17.327122+0000","last_clean":"2023-03-03T07:57:17.327122+0000","last_became_active":"2023-03-03T07:57:17.320033+0000","last_became_peered":"2023-03-03T07:57:17.320033+0000","last_unstale":"2023-03-03T07:57:17.327122+0000","last_undegraded":"2023-03-03T07:57:17.327122+0000","last_fullsized":"2023-03-03T07:57:17.327122+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.307831+0000","last_change":"2023-03-03T07:57:16.748170+0000","last_active":"2023-03-03T07:57:17.307831+0000","last_peered":"2023-03-03T07:57:17.307831+0000","last_clean":"2023-03-03T07:57:17.307831+0000","last_became_active":"2023-03-03T07:57:16.747842+0000","last_became_peered":"2023-03-03T07:57:16.747842+0000","last_unstale":"2023-03-03T07:57:17.307831+0000","last_undegraded":"2023-03-03T07:57:17.307831+0000","last_fullsized":"2023-03-03T07:57:17.307831+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:57:03.533693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.302674+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T07:57:17.302674+0000","last_peered":"2023-03-03T07:57:17.302674+0000","last_clean":"2023-03-03T07:57:17.302674+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T07:57:17.302674+0000","last_undegraded":"2023-03-03T07:57:17.302674+0000","last_fullsized":"2023-03-03T07:57:17.302674+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.307302+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T07:57:17.307302+0000","last_peered":"2023-03-03T07:57:17.307302+0000","last_clean":"2023-03-03T07:57:17.307302+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T07:57:17.307302+0000","last_undegraded":"2023-03-03T07:57:17.307302+0000","last_fullsized":"2023-03-03T07:57:17.307302+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.315023+0000","last_change":"2023-03-03T07:57:17.311231+0000","last_active":"2023-03-03T07:57:17.315023+0000","last_peered":"2023-03-03T07:57:17.315023+0000","last_clean":"2023-03-03T07:57:17.315023+0000","last_became_active":"2023-03-03T07:57:17.310950+0000","last_became_peered":"2023-03-03T07:57:17.310950+0000","last_unstale":"2023-03-03T07:57:17.315023+0000","last_undegraded":"2023-03-03T07:57:17.315023+0000","last_fullsized":"2023-03-03T07:57:17.315023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.324947+0000","last_change":"2023-03-03T07:57:16.756473+0000","last_active":"2023-03-03T07:57:17.324947+0000","last_peered":"2023-03-03T07:57:17.324947+0000","last_clean":"2023-03-03T07:57:17.324947+0000","last_became_active":"2023-03-03T07:57:16.755809+0000","last_became_peered":"2023-03-03T07:57:16.755809+0000","last_unstale":"2023-03-03T07:57:17.324947+0000","last_undegraded":"2023-03-03T07:57:17.324947+0000","last_fullsized":"2023-03-03T07:57:17.324947+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:19.455490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.307386+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T07:57:17.307386+0000","last_peered":"2023-03-03T07:57:17.307386+0000","last_clean":"2023-03-03T07:57:17.307386+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T07:57:17.307386+0000","last_undegraded":"2023-03-03T07:57:17.307386+0000","last_fullsized":"2023-03-03T07:57:17.307386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.324457+0000","last_change":"2023-03-03T07:57:17.316375+0000","last_active":"2023-03-03T07:57:17.324457+0000","last_peered":"2023-03-03T07:57:17.324457+0000","last_clean":"2023-03-03T07:57:17.324457+0000","last_became_active":"2023-03-03T07:57:17.316069+0000","last_became_peered":"2023-03-03T07:57:17.316069+0000","last_unstale":"2023-03-03T07:57:17.324457+0000","last_undegraded":"2023-03-03T07:57:17.324457+0000","last_fullsized":"2023-03-03T07:57:17.324457+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:59.426058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.305735+0000","last_change":"2023-03-03T07:57:16.752378+0000","last_active":"2023-03-03T07:57:17.305735+0000","last_peered":"2023-03-03T07:57:17.305735+0000","last_clean":"2023-03-03T07:57:17.305735+0000","last_became_active":"2023-03-03T07:57:16.752028+0000","last_became_peered":"2023-03-03T07:57:16.752028+0000","last_unstale":"2023-03-03T07:57:17.305735+0000","last_undegraded":"2023-03-03T07:57:17.305735+0000","last_fullsized":"2023-03-03T07:57:17.305735+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:14:02.921691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.308253+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T07:57:17.308253+0000","last_peered":"2023-03-03T07:57:17.308253+0000","last_clean":"2023-03-03T07:57:17.308253+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T07:57:17.308253+0000","last_undegraded":"2023-03-03T07:57:17.308253+0000","last_fullsized":"2023-03-03T07:57:17.308253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335408+0000","last_change":"2023-03-03T07:57:17.328978+0000","last_active":"2023-03-03T07:57:17.335408+0000","last_peered":"2023-03-03T07:57:17.335408+0000","last_clean":"2023-03-03T07:57:17.335408+0000","last_became_active":"2023-03-03T07:57:17.328542+0000","last_became_peered":"2023-03-03T07:57:17.328542+0000","last_unstale":"2023-03-03T07:57:17.335408+0000","last_undegraded":"2023-03-03T07:57:17.335408+0000","last_fullsized":"2023-03-03T07:57:17.335408+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.312893+0000","last_change":"2023-03-03T07:57:16.742160+0000","last_active":"2023-03-03T07:57:17.312893+0000","last_peered":"2023-03-03T07:57:17.312893+0000","last_clean":"2023-03-03T07:57:17.312893+0000","last_became_active":"2023-03-03T07:57:16.741796+0000","last_became_peered":"2023-03-03T07:57:16.741796+0000","last_unstale":"2023-03-03T07:57:17.312893+0000","last_undegraded":"2023-03-03T07:57:17.312893+0000","last_fullsized":"2023-03-03T07:57:17.312893+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:55.349529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.304389+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T07:57:17.304389+0000","last_peered":"2023-03-03T07:57:17.304389+0000","last_clean":"2023-03-03T07:57:17.304389+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T07:57:17.304389+0000","last_undegraded":"2023-03-03T07:57:17.304389+0000","last_fullsized":"2023-03-03T07:57:17.304389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.333773+0000","last_change":"2023-03-03T07:57:17.325655+0000","last_active":"2023-03-03T07:57:17.333773+0000","last_peered":"2023-03-03T07:57:17.333773+0000","last_clean":"2023-03-03T07:57:17.333773+0000","last_became_active":"2023-03-03T07:57:17.325418+0000","last_became_peered":"2023-03-03T07:57:17.325418+0000","last_unstale":"2023-03-03T07:57:17.333773+0000","last_undegraded":"2023-03-03T07:57:17.333773+0000","last_fullsized":"2023-03-03T07:57:17.333773+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:25.355832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.309427+0000","last_change":"2023-03-03T07:57:16.744051+0000","last_active":"2023-03-03T07:57:17.309427+0000","last_peered":"2023-03-03T07:57:17.309427+0000","last_clean":"2023-03-03T07:57:17.309427+0000","last_became_active":"2023-03-03T07:57:16.743785+0000","last_became_peered":"2023-03-03T07:57:16.743785+0000","last_unstale":"2023-03-03T07:57:17.309427+0000","last_undegraded":"2023-03-03T07:57:17.309427+0000","last_fullsized":"2023-03-03T07:57:17.309427+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:48:13.119986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.306114+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T07:57:17.306114+0000","last_peered":"2023-03-03T07:57:17.306114+0000","last_clean":"2023-03-03T07:57:17.306114+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T07:57:17.306114+0000","last_undegraded":"2023-03-03T07:57:17.306114+0000","last_fullsized":"2023-03-03T07:57:17.306114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9604,"kb_used_data":1020,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362236,"statfs":{"total":96636764160,"available":96626929664,"internally_reserved":0,"allocated":1044480,"data_stored":772784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":0.58199999999999996}]},{"osd":1,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":6.6829999999999998,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":0.68600000000000005}]},{"osd":2,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.71999999999999997}]},{"osd":3,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.61099999999999999}]},{"osd":4,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.745},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.90300000000000002}]},{"osd":5,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.89600000000000002}]},{"osd":6,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.82599999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8368,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363472,"statfs":{"total":96636764160,"available":96628195328,"internally_reserved":0,"allocated":434176,"data_stored":180345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.69799999999999995}]},{"osd":1,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":1.0029999999999999}]},{"osd":2,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.871},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.98099999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.77200000000000002}]},{"osd":4,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":4.0099999999999998,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":4.1859999999999999,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.60299999999999998}]},{"osd":5,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.74099999999999999}]},{"osd":7,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.82199999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574872,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":180888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.972,"5min":3.972,"15min":3.972},"last":0.439}]},{"osd":2,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":11.897,"5min":11.897,"15min":11.897},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":11.804,"5min":11.804,"15min":11.804},"last":0.56499999999999995}]},{"osd":3,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":11.843999999999999,"5min":11.843999999999999,"15min":11.843999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":3.8740000000000001,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.503}]},{"osd":4,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.876},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.88400000000000001}]},{"osd":5,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":3.8300000000000001,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.77900000000000003}]},{"osd":6,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":3.907,"5min":3.907,"15min":3.907},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.91500000000000004}]},{"osd":7,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":3.9399999999999999,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":0.72699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9524,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":8512,"kb_avail":94362316,"statfs":{"total":96636764160,"available":96627011584,"internally_reserved":0,"allocated":1028096,"data_stored":770181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8716288},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.51200000000000001}]},{"osd":2,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.46000000000000002}]},{"osd":3,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.53900000000000003}]},{"osd":4,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.77500000000000002}]},{"osd":6,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.624}]},{"osd":7,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.754}]}]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8368,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363472,"statfs":{"total":96636764160,"available":96628195328,"internally_reserved":0,"allocated":434176,"data_stored":179849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":5.7889999999999997,"5min":5.7889999999999997,"15min":5.7889999999999997},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":2.6309999999999998,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":5.8470000000000004,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.625}]},{"osd":3,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.961,"5min":12.961,"15min":12.961},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.1449999999999996,"5min":6.1449999999999996,"15min":6.1449999999999996},"last":0.53100000000000003}]},{"osd":4,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":6.484,"5min":6.484,"15min":6.484},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":6.5510000000000002,"5min":6.5510000000000002,"15min":6.5510000000000002},"last":0.91000000000000003}]},{"osd":5,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":5.7119999999999997,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":6.6369999999999996,"5min":6.6369999999999996,"15min":6.6369999999999996},"last":0.89700000000000002}]},{"osd":6,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":6.625,"5min":6.625,"15min":6.625},"last":0.872},{"interface":"front","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":5.7539999999999996,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.91700000000000004}]},{"osd":7,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":6.7009999999999996,"5min":6.7009999999999996,"15min":6.7009999999999996},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":6.6840000000000002,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.995}]}]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8576,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94363264,"statfs":{"total":96636764160,"available":96627982336,"internally_reserved":0,"allocated":450560,"data_stored":182380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":9.5820000000000007,"5min":9.5820000000000007,"15min":9.5820000000000007},"last":0.317},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":9.7840000000000007,"5min":9.7840000000000007,"15min":9.7840000000000007},"last":0.44900000000000001}]},{"osd":1,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.702,"5min":9.702,"15min":9.702},"last":0.47399999999999998}]},{"osd":2,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":8.3190000000000008,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.133,"5min":1.133,"15min":1.133},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":9.734,"5min":9.734,"15min":9.734},"last":0.51000000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":9.8629999999999995,"5min":9.8629999999999995,"15min":9.8629999999999995},"last":0.754},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.80600000000000005}]},{"osd":5,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":9.6600000000000001,"5min":9.6600000000000001,"15min":9.6600000000000001},"last":0.73199999999999998}]},{"osd":6,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":9.8190000000000008,"5min":9.8190000000000008,"15min":9.8190000000000008},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.72199999999999998}]},{"osd":7,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":9.625,"5min":9.625,"15min":9.625},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":9.8949999999999996,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":0.70799999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8632,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363208,"statfs":{"total":96636764160,"available":96627924992,"internally_reserved":0,"allocated":442368,"data_stored":181386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":7.1799999999999997,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.86199999999999999}]},{"osd":1,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":9.5709999999999997,"5min":9.5709999999999997,"15min":9.5709999999999997},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":5.266,"5min":5.266,"15min":5.266},"last":0.88}]},{"osd":2,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":8.4529999999999994,"5min":8.4529999999999994,"15min":8.4529999999999994},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":6.6829999999999998,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.83799999999999997}]},{"osd":3,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.84999999999999998}]},{"osd":5,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.74299999999999999}]},{"osd":6,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.77800000000000002}]},{"osd":7,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.72499999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":179891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.70999999999999996}]},{"osd":1,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.45500000000000002}]},{"osd":2,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.65600000000000003}]},{"osd":3,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.76000000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.82399999999999995}]},{"osd":6,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.73899999999999999}]},{"osd":7,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T07:57:27.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T07:57:27.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.752+0000 7fbebc9fe700 1 Processor -- start 2023-03-03T07:57:27.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.753+0000 7fbebc9fe700 1 -- start start 2023-03-03T07:57:27.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.754+0000 7fbebc9fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fbeb8146360 0x7fbeb8160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:27.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.754+0000 7fbebc9fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbeb80d2590 0x7fbeb80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:27.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.754+0000 7fbebc9fe700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb81614c0 con 0x7fbeb80b4ab0 2023-03-03T07:57:27.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.754+0000 7fbebc9fe700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeb80b5440 con 0x7fbeb80d2590 2023-03-03T07:57:27.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.754+0000 7fbebc9fe700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeb80d10f0 con 0x7fbeb8146360 2023-03-03T07:57:27.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.754+0000 7fbeaffff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbeb80b4ab0 0x7fbeb80731e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47678/0 (socket says 172.21.15.98:47678) 2023-03-03T07:57:27.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.754+0000 7fbeaffff700 1 -- 172.21.15.98:0/4001503313 learned_addr learned my addr 172.21.15.98:0/4001503313 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:27.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.754+0000 7fbeb759e700 1 --2- 172.21.15.98:0/4001503313 >> v2:172.21.15.98:3300/0 conn(0x7fbeb80d2590 0x7fbeb80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:27.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.755+0000 7fbeb6d9d700 1 --2- 172.21.15.98:0/4001503313 >> v2:172.21.15.100:3300/0 conn(0x7fbeb8146360 0x7fbeb8160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:27.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.755+0000 7fbeb659c700 1 -- 172.21.15.98:0/4001503313 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196722761 0 0) 0x7fbea0001110 con 0x7fbeb80b4ab0 2023-03-03T07:57:27.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.755+0000 7fbeb659c700 1 -- 172.21.15.98:0/4001503313 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea40035b0 con 0x7fbeb80b4ab0 2023-03-03T07:57:27.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.755+0000 7fbeb759e700 1 -- 172.21.15.98:0/4001503313 >> v2:172.21.15.100:3300/0 conn(0x7fbeb8146360 msgr2=0x7fbeb8160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:27.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.755+0000 7fbeb759e700 1 --2- 172.21.15.98:0/4001503313 >> v2:172.21.15.100:3300/0 conn(0x7fbeb8146360 0x7fbeb8160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:27.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.755+0000 7fbeb759e700 1 -- 172.21.15.98:0/4001503313 >> v1:172.21.15.98:6789/0 conn(0x7fbeb80b4ab0 legacy=0x7fbeb80731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:27.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.755+0000 7fbeb759e700 1 -- 172.21.15.98:0/4001503313 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb80cdf20 con 0x7fbeb80d2590 2023-03-03T07:57:27.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.755+0000 7fbeb759e700 1 --2- 172.21.15.98:0/4001503313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb80d2590 0x7fbeb80b4f00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80743f0 tx=0x7fbeb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998c4639aa7e63d0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:27.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.756+0000 7fbeb659c700 1 -- 172.21.15.98:0/4001503313 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb000e070 con 0x7fbeb80d2590 2023-03-03T07:57:27.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.756+0000 7fbeb659c700 1 -- 172.21.15.98:0/4001503313 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb000bb20 con 0x7fbeb80d2590 2023-03-03T07:57:27.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.756+0000 7fbeb659c700 1 -- 172.21.15.98:0/4001503313 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0012430 con 0x7fbeb80d2590 2023-03-03T07:57:27.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.756+0000 7fbebc9fe700 1 -- 172.21.15.98:0/4001503313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb80d2590 msgr2=0x7fbeb80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:27.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.756+0000 7fbebc9fe700 1 --2- 172.21.15.98:0/4001503313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb80d2590 0x7fbeb80b4f00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80743f0 tx=0x7fbeb000b330 comp rx=0 tx=0).stop 2023-03-03T07:57:27.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.756+0000 7fbebc9fe700 1 -- 172.21.15.98:0/4001503313 shutdown_connections 2023-03-03T07:57:27.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.756+0000 7fbebc9fe700 1 --2- 172.21.15.98:0/4001503313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb80d2590 0x7fbeb80b4f00 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:27.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.756+0000 7fbebc9fe700 1 --2- 172.21.15.98:0/4001503313 >> v2:172.21.15.100:3300/0 conn(0x7fbeb8146360 0x7fbeb8160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:27.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.756+0000 7fbebc9fe700 1 -- 172.21.15.98:0/4001503313 >> 172.21.15.98:0/4001503313 conn(0x7fbeb806b8e0 msgr2=0x7fbeb80cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:27.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.756+0000 7fbebc9fe700 1 -- 172.21.15.98:0/4001503313 shutdown_connections 2023-03-03T07:57:27.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.756+0000 7fbebc9fe700 1 -- 172.21.15.98:0/4001503313 wait complete. 2023-03-03T07:57:27.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.758+0000 7fbebc9fe700 1 Processor -- start 2023-03-03T07:57:27.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.759+0000 7fbebc9fe700 1 -- start start 2023-03-03T07:57:27.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.760+0000 7fbebc9fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbeb80b4ab0 0x7fbeb8159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:27.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.760+0000 7fbebc9fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb80d2590 0x7fbeb8151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:27.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.760+0000 7fbebc9fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbeb8146360 0x7fbeb8152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:27.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.760+0000 7fbebc9fe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb8152280 con 0x7fbeb80d2590 2023-03-03T07:57:27.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.760+0000 7fbebc9fe700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb8152560 con 0x7fbeb80b4ab0 2023-03-03T07:57:27.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.760+0000 7fbebc9fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb80c8860 con 0x7fbeb8146360 2023-03-03T07:57:27.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.760+0000 7fbeb6d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbeb80b4ab0 0x7fbeb8159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:27.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.760+0000 7fbeaffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb80d2590 0x7fbeb8151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:27.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.760+0000 7fbeb6d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbeb80b4ab0 0x7fbeb8159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:58024/0 (socket says 172.21.15.98:58024) 2023-03-03T07:57:27.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.760+0000 7fbeb6d9d700 1 -- 172.21.15.98:0/2229897388 learned_addr learned my addr 172.21.15.98:0/2229897388 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:27.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.760+0000 7fbeb759e700 1 --2- 172.21.15.98:0/2229897388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbeb8146360 0x7fbeb8152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:27.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.761+0000 7fbeaffff700 1 -- 172.21.15.98:0/2229897388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbeb8146360 msgr2=0x7fbeb8152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.761+0000 7fbeaffff700 1 --2- 172.21.15.98:0/2229897388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbeb8146360 0x7fbeb8152840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.761+0000 7fbeaffff700 1 -- 172.21.15.98:0/2229897388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbeb80b4ab0 msgr2=0x7fbeb8159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.761+0000 7fbeaffff700 1 --2- 172.21.15.98:0/2229897388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbeb80b4ab0 0x7fbeb8159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.761+0000 7fbeaffff700 1 -- 172.21.15.98:0/2229897388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb0009a70 con 0x7fbeb80d2590 2023-03-03T07:57:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.761+0000 7fbeaffff700 1 --2- 172.21.15.98:0/2229897388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb80d2590 0x7fbeb8151c20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbea000c430 tx=0x7fbea000c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.761+0000 7fbeaf7fe700 1 -- 172.21.15.98:0/2229897388 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea0005b70 con 0x7fbeb80d2590 2023-03-03T07:57:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.761+0000 7fbeaf7fe700 1 -- 172.21.15.98:0/2229897388 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea0005cd0 con 0x7fbeb80d2590 2023-03-03T07:57:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.761+0000 7fbebc9fe700 1 -- 172.21.15.98:0/2229897388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb80cd960 con 0x7fbeb80d2590 2023-03-03T07:57:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.761+0000 7fbeaf7fe700 1 -- 172.21.15.98:0/2229897388 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea00072f0 con 0x7fbeb80d2590 2023-03-03T07:57:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.762+0000 7fbebc9fe700 1 -- 172.21.15.98:0/2229897388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb80cd680 con 0x7fbeb80d2590 2023-03-03T07:57:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.763+0000 7fbebc9fe700 1 -- 172.21.15.98:0/2229897388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb80cdc40 con 0x7fbeb80d2590 2023-03-03T07:57:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.764+0000 7fbeaf7fe700 1 -- 172.21.15.98:0/2229897388 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbea0019030 con 0x7fbeb80d2590 2023-03-03T07:57:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.766+0000 7fbeaf7fe700 1 --2- 172.21.15.98:0/2229897388 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbe980682f0 0x7fbe9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.766+0000 7fbeaf7fe700 1 -- 172.21.15.98:0/2229897388 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fbea0084ce0 con 0x7fbeb80d2590 2023-03-03T07:57:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.766+0000 7fbeb6d9d700 1 --2- 172.21.15.98:0/2229897388 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbe980682f0 0x7fbe9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.767+0000 7fbeb6d9d700 1 --2- 172.21.15.98:0/2229897388 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbe980682f0 0x7fbe9806a7a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbea8009ca0 tx=0x7fbea8009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:27.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.770+0000 7fbeaf7fe700 1 -- 172.21.15.98:0/2229897388 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbea0051430 con 0x7fbeb80d2590 2023-03-03T07:57:27.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.959+0000 7fbebc9fe700 1 -- 172.21.15.98:0/2229897388 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbeb81526d0 con 0x7fbe980682f0 2023-03-03T07:57:27.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.969+0000 7fbeaf7fe700 1 -- 172.21.15.98:0/2229897388 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310563 (secure 0 0 0) 0x7fbeb81526d0 con 0x7fbe980682f0 2023-03-03T07:57:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:27.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.974+0000 7fbebc9fe700 1 -- 172.21.15.98:0/2229897388 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbe980682f0 msgr2=0x7fbe9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:27.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.974+0000 7fbebc9fe700 1 --2- 172.21.15.98:0/2229897388 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbe980682f0 0x7fbe9806a7a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbea8009ca0 tx=0x7fbea8009210 comp rx=0 tx=0).stop 2023-03-03T07:57:27.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.974+0000 7fbebc9fe700 1 -- 172.21.15.98:0/2229897388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb80d2590 msgr2=0x7fbeb8151c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:27.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.974+0000 7fbebc9fe700 1 --2- 172.21.15.98:0/2229897388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb80d2590 0x7fbeb8151c20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbea000c430 tx=0x7fbea000c740 comp rx=0 tx=0).stop 2023-03-03T07:57:27.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.975+0000 7fbebc9fe700 1 -- 172.21.15.98:0/2229897388 shutdown_connections 2023-03-03T07:57:27.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.975+0000 7fbebc9fe700 1 --2- 172.21.15.98:0/2229897388 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbe980682f0 0x7fbe9806a7a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:27.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.975+0000 7fbebc9fe700 1 --2- 172.21.15.98:0/2229897388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbeb8146360 0x7fbeb8152840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:27.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.975+0000 7fbebc9fe700 1 --2- 172.21.15.98:0/2229897388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbeb80b4ab0 0x7fbeb8159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:27.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.975+0000 7fbebc9fe700 1 --2- 172.21.15.98:0/2229897388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb80d2590 0x7fbeb8151c20 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:27.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.975+0000 7fbebc9fe700 1 -- 172.21.15.98:0/2229897388 >> 172.21.15.98:0/2229897388 conn(0x7fbeb806b8e0 msgr2=0x7fbeb80731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:27.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.975+0000 7fbebc9fe700 1 -- 172.21.15.98:0/2229897388 shutdown_connections 2023-03-03T07:57:27.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:27.975+0000 7fbebc9fe700 1 -- 172.21.15.98:0/2229897388 wait complete. 2023-03-03T07:57:27.979 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T07:57:28.004 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2023-03-03T07:57:27.259757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":70072,"kb_used_data":4600,"kb_used_omap":0,"kb_used_meta":65408,"kb_avail":754904648,"statfs":{"total":773094113280,"available":773022359552,"internally_reserved":0,"allocated":4710400,"data_stored":2627704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66977792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.062773"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.331107+0000","last_change":"2023-03-03T07:57:17.326551+0000","last_active":"2023-03-03T07:57:17.331107+0000","last_peered":"2023-03-03T07:57:17.331107+0000","last_clean":"2023-03-03T07:57:17.331107+0000","last_became_active":"2023-03-03T07:57:17.326309+0000","last_became_peered":"2023-03-03T07:57:17.326309+0000","last_unstale":"2023-03-03T07:57:17.331107+0000","last_undegraded":"2023-03-03T07:57:17.331107+0000","last_fullsized":"2023-03-03T07:57:17.331107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:39:27.674460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.327863+0000","last_change":"2023-03-03T07:57:17.323983+0000","last_active":"2023-03-03T07:57:17.327863+0000","last_peered":"2023-03-03T07:57:17.327863+0000","last_clean":"2023-03-03T07:57:17.327863+0000","last_became_active":"2023-03-03T07:57:17.323715+0000","last_became_peered":"2023-03-03T07:57:17.323715+0000","last_unstale":"2023-03-03T07:57:17.327863+0000","last_undegraded":"2023-03-03T07:57:17.327863+0000","last_fullsized":"2023-03-03T07:57:17.327863+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.329262+0000","last_change":"2023-03-03T07:57:17.324163+0000","last_active":"2023-03-03T07:57:17.329262+0000","last_peered":"2023-03-03T07:57:17.329262+0000","last_clean":"2023-03-03T07:57:17.329262+0000","last_became_active":"2023-03-03T07:57:17.323910+0000","last_became_peered":"2023-03-03T07:57:17.323910+0000","last_unstale":"2023-03-03T07:57:17.329262+0000","last_undegraded":"2023-03-03T07:57:17.329262+0000","last_fullsized":"2023-03-03T07:57:17.329262+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335677+0000","last_change":"2023-03-03T07:57:17.334114+0000","last_active":"2023-03-03T07:57:17.335677+0000","last_peered":"2023-03-03T07:57:17.335677+0000","last_clean":"2023-03-03T07:57:17.335677+0000","last_became_active":"2023-03-03T07:57:17.333860+0000","last_became_peered":"2023-03-03T07:57:17.333860+0000","last_unstale":"2023-03-03T07:57:17.335677+0000","last_undegraded":"2023-03-03T07:57:17.335677+0000","last_fullsized":"2023-03-03T07:57:17.335677+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:07.498551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335495+0000","last_change":"2023-03-03T07:57:17.332370+0000","last_active":"2023-03-03T07:57:17.335495+0000","last_peered":"2023-03-03T07:57:17.335495+0000","last_clean":"2023-03-03T07:57:17.335495+0000","last_became_active":"2023-03-03T07:57:17.332113+0000","last_became_peered":"2023-03-03T07:57:17.332113+0000","last_unstale":"2023-03-03T07:57:17.335495+0000","last_undegraded":"2023-03-03T07:57:17.335495+0000","last_fullsized":"2023-03-03T07:57:17.335495+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:37:50.701337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.320582+0000","last_change":"2023-03-03T07:57:17.316803+0000","last_active":"2023-03-03T07:57:17.320582+0000","last_peered":"2023-03-03T07:57:17.320582+0000","last_clean":"2023-03-03T07:57:17.320582+0000","last_became_active":"2023-03-03T07:57:17.316519+0000","last_became_peered":"2023-03-03T07:57:17.316519+0000","last_unstale":"2023-03-03T07:57:17.320582+0000","last_undegraded":"2023-03-03T07:57:17.320582+0000","last_fullsized":"2023-03-03T07:57:17.320582+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:25.355832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.327143+0000","last_change":"2023-03-03T07:57:17.321448+0000","last_active":"2023-03-03T07:57:17.327143+0000","last_peered":"2023-03-03T07:57:17.327143+0000","last_clean":"2023-03-03T07:57:17.327143+0000","last_became_active":"2023-03-03T07:57:17.321158+0000","last_became_peered":"2023-03-03T07:57:17.321158+0000","last_unstale":"2023-03-03T07:57:17.327143+0000","last_undegraded":"2023-03-03T07:57:17.327143+0000","last_fullsized":"2023-03-03T07:57:17.327143+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:44.621837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.331718+0000","last_change":"2023-03-03T07:57:17.327725+0000","last_active":"2023-03-03T07:57:17.331718+0000","last_peered":"2023-03-03T07:57:17.331718+0000","last_clean":"2023-03-03T07:57:17.331718+0000","last_became_active":"2023-03-03T07:57:17.327468+0000","last_became_peered":"2023-03-03T07:57:17.327468+0000","last_unstale":"2023-03-03T07:57:17.331718+0000","last_undegraded":"2023-03-03T07:57:17.331718+0000","last_fullsized":"2023-03-03T07:57:17.331718+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:01.871020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.329272+0000","last_change":"2023-03-03T07:57:17.322904+0000","last_active":"2023-03-03T07:57:17.329272+0000","last_peered":"2023-03-03T07:57:17.329272+0000","last_clean":"2023-03-03T07:57:17.329272+0000","last_became_active":"2023-03-03T07:57:17.322466+0000","last_became_peered":"2023-03-03T07:57:17.322466+0000","last_unstale":"2023-03-03T07:57:17.329272+0000","last_undegraded":"2023-03-03T07:57:17.329272+0000","last_fullsized":"2023-03-03T07:57:17.329272+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.317840+0000","last_change":"2023-03-03T07:57:17.312858+0000","last_active":"2023-03-03T07:57:17.317840+0000","last_peered":"2023-03-03T07:57:17.317840+0000","last_clean":"2023-03-03T07:57:17.317840+0000","last_became_active":"2023-03-03T07:57:17.312552+0000","last_became_peered":"2023-03-03T07:57:17.312552+0000","last_unstale":"2023-03-03T07:57:17.317840+0000","last_undegraded":"2023-03-03T07:57:17.317840+0000","last_fullsized":"2023-03-03T07:57:17.317840+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.334638+0000","last_change":"2023-03-03T07:57:17.328548+0000","last_active":"2023-03-03T07:57:17.334638+0000","last_peered":"2023-03-03T07:57:17.334638+0000","last_clean":"2023-03-03T07:57:17.334638+0000","last_became_active":"2023-03-03T07:57:17.328277+0000","last_became_peered":"2023-03-03T07:57:17.328277+0000","last_unstale":"2023-03-03T07:57:17.334638+0000","last_undegraded":"2023-03-03T07:57:17.334638+0000","last_fullsized":"2023-03-03T07:57:17.334638+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.328669+0000","last_change":"2023-03-03T07:57:17.320812+0000","last_active":"2023-03-03T07:57:17.328669+0000","last_peered":"2023-03-03T07:57:17.328669+0000","last_clean":"2023-03-03T07:57:17.328669+0000","last_became_active":"2023-03-03T07:57:17.320504+0000","last_became_peered":"2023-03-03T07:57:17.320504+0000","last_unstale":"2023-03-03T07:57:17.328669+0000","last_undegraded":"2023-03-03T07:57:17.328669+0000","last_fullsized":"2023-03-03T07:57:17.328669+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.317802+0000","last_change":"2023-03-03T07:57:17.312825+0000","last_active":"2023-03-03T07:57:17.317802+0000","last_peered":"2023-03-03T07:57:17.317802+0000","last_clean":"2023-03-03T07:57:17.317802+0000","last_became_active":"2023-03-03T07:57:17.312529+0000","last_became_peered":"2023-03-03T07:57:17.312529+0000","last_unstale":"2023-03-03T07:57:17.317802+0000","last_undegraded":"2023-03-03T07:57:17.317802+0000","last_fullsized":"2023-03-03T07:57:17.317802+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335390+0000","last_change":"2023-03-03T07:57:17.328800+0000","last_active":"2023-03-03T07:57:17.335390+0000","last_peered":"2023-03-03T07:57:17.335390+0000","last_clean":"2023-03-03T07:57:17.335390+0000","last_became_active":"2023-03-03T07:57:17.328177+0000","last_became_peered":"2023-03-03T07:57:17.328177+0000","last_unstale":"2023-03-03T07:57:17.335390+0000","last_undegraded":"2023-03-03T07:57:17.335390+0000","last_fullsized":"2023-03-03T07:57:17.335390+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:48:58.988637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.330279+0000","last_change":"2023-03-03T07:57:17.322362+0000","last_active":"2023-03-03T07:57:17.330279+0000","last_peered":"2023-03-03T07:57:17.330279+0000","last_clean":"2023-03-03T07:57:17.330279+0000","last_became_active":"2023-03-03T07:57:17.322087+0000","last_became_peered":"2023-03-03T07:57:17.322087+0000","last_unstale":"2023-03-03T07:57:17.330279+0000","last_undegraded":"2023-03-03T07:57:17.330279+0000","last_fullsized":"2023-03-03T07:57:17.330279+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:24:04.359311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.328654+0000","last_change":"2023-03-03T07:57:17.322950+0000","last_active":"2023-03-03T07:57:17.328654+0000","last_peered":"2023-03-03T07:57:17.328654+0000","last_clean":"2023-03-03T07:57:17.328654+0000","last_became_active":"2023-03-03T07:57:17.322228+0000","last_became_peered":"2023-03-03T07:57:17.322228+0000","last_unstale":"2023-03-03T07:57:17.328654+0000","last_undegraded":"2023-03-03T07:57:17.328654+0000","last_fullsized":"2023-03-03T07:57:17.328654+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.325795+0000","last_change":"2023-03-03T07:57:17.321739+0000","last_active":"2023-03-03T07:57:17.325795+0000","last_peered":"2023-03-03T07:57:17.325795+0000","last_clean":"2023-03-03T07:57:17.325795+0000","last_became_active":"2023-03-03T07:57:17.321373+0000","last_became_peered":"2023-03-03T07:57:17.321373+0000","last_unstale":"2023-03-03T07:57:17.325795+0000","last_undegraded":"2023-03-03T07:57:17.325795+0000","last_fullsized":"2023-03-03T07:57:17.325795+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.332188+0000","last_change":"2023-03-03T07:57:17.328635+0000","last_active":"2023-03-03T07:57:17.332188+0000","last_peered":"2023-03-03T07:57:17.332188+0000","last_clean":"2023-03-03T07:57:17.332188+0000","last_became_active":"2023-03-03T07:57:17.328129+0000","last_became_peered":"2023-03-03T07:57:17.328129+0000","last_unstale":"2023-03-03T07:57:17.332188+0000","last_undegraded":"2023-03-03T07:57:17.332188+0000","last_fullsized":"2023-03-03T07:57:17.332188+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:56:10.139056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.321964+0000","last_change":"2023-03-03T07:57:17.319110+0000","last_active":"2023-03-03T07:57:17.321964+0000","last_peered":"2023-03-03T07:57:17.321964+0000","last_clean":"2023-03-03T07:57:17.321964+0000","last_became_active":"2023-03-03T07:57:17.318212+0000","last_became_peered":"2023-03-03T07:57:17.318212+0000","last_unstale":"2023-03-03T07:57:17.321964+0000","last_undegraded":"2023-03-03T07:57:17.321964+0000","last_fullsized":"2023-03-03T07:57:17.321964+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:00:23.450170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.332401+0000","last_change":"2023-03-03T07:57:17.328451+0000","last_active":"2023-03-03T07:57:17.332401+0000","last_peered":"2023-03-03T07:57:17.332401+0000","last_clean":"2023-03-03T07:57:17.332401+0000","last_became_active":"2023-03-03T07:57:17.328048+0000","last_became_peered":"2023-03-03T07:57:17.328048+0000","last_unstale":"2023-03-03T07:57:17.332401+0000","last_undegraded":"2023-03-03T07:57:17.332401+0000","last_fullsized":"2023-03-03T07:57:17.332401+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:15.531132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.320596+0000","last_change":"2023-03-03T07:57:17.315034+0000","last_active":"2023-03-03T07:57:17.320596+0000","last_peered":"2023-03-03T07:57:17.320596+0000","last_clean":"2023-03-03T07:57:17.320596+0000","last_became_active":"2023-03-03T07:57:17.314760+0000","last_became_peered":"2023-03-03T07:57:17.314760+0000","last_unstale":"2023-03-03T07:57:17.320596+0000","last_undegraded":"2023-03-03T07:57:17.320596+0000","last_fullsized":"2023-03-03T07:57:17.320596+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.336425+0000","last_change":"2023-03-03T07:57:17.334238+0000","last_active":"2023-03-03T07:57:17.336425+0000","last_peered":"2023-03-03T07:57:17.336425+0000","last_clean":"2023-03-03T07:57:17.336425+0000","last_became_active":"2023-03-03T07:57:17.333913+0000","last_became_peered":"2023-03-03T07:57:17.333913+0000","last_unstale":"2023-03-03T07:57:17.336425+0000","last_undegraded":"2023-03-03T07:57:17.336425+0000","last_fullsized":"2023-03-03T07:57:17.336425+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:39:27.674460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.332819+0000","last_change":"2023-03-03T07:57:17.323585+0000","last_active":"2023-03-03T07:57:17.332819+0000","last_peered":"2023-03-03T07:57:17.332819+0000","last_clean":"2023-03-03T07:57:17.332819+0000","last_became_active":"2023-03-03T07:57:17.323266+0000","last_became_peered":"2023-03-03T07:57:17.323266+0000","last_unstale":"2023-03-03T07:57:17.332819+0000","last_undegraded":"2023-03-03T07:57:17.332819+0000","last_fullsized":"2023-03-03T07:57:17.332819+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.319841+0000","last_change":"2023-03-03T07:57:17.310071+0000","last_active":"2023-03-03T07:57:17.319841+0000","last_peered":"2023-03-03T07:57:17.319841+0000","last_clean":"2023-03-03T07:57:17.319841+0000","last_became_active":"2023-03-03T07:57:17.309133+0000","last_became_peered":"2023-03-03T07:57:17.309133+0000","last_unstale":"2023-03-03T07:57:17.319841+0000","last_undegraded":"2023-03-03T07:57:17.319841+0000","last_fullsized":"2023-03-03T07:57:17.319841+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:57:04.542617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.320681+0000","last_change":"2023-03-03T07:57:17.316824+0000","last_active":"2023-03-03T07:57:17.320681+0000","last_peered":"2023-03-03T07:57:17.320681+0000","last_clean":"2023-03-03T07:57:17.320681+0000","last_became_active":"2023-03-03T07:57:17.316537+0000","last_became_peered":"2023-03-03T07:57:17.316537+0000","last_unstale":"2023-03-03T07:57:17.320681+0000","last_undegraded":"2023-03-03T07:57:17.320681+0000","last_fullsized":"2023-03-03T07:57:17.320681+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:57:04.542617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.320908+0000","last_change":"2023-03-03T07:57:17.317046+0000","last_active":"2023-03-03T07:57:17.320908+0000","last_peered":"2023-03-03T07:57:17.320908+0000","last_clean":"2023-03-03T07:57:17.320908+0000","last_became_active":"2023-03-03T07:57:17.316775+0000","last_became_peered":"2023-03-03T07:57:17.316775+0000","last_unstale":"2023-03-03T07:57:17.320908+0000","last_undegraded":"2023-03-03T07:57:17.320908+0000","last_fullsized":"2023-03-03T07:57:17.320908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.325925+0000","last_change":"2023-03-03T07:57:17.324059+0000","last_active":"2023-03-03T07:57:17.325925+0000","last_peered":"2023-03-03T07:57:17.325925+0000","last_clean":"2023-03-03T07:57:17.325925+0000","last_became_active":"2023-03-03T07:57:17.323486+0000","last_became_peered":"2023-03-03T07:57:17.323486+0000","last_unstale":"2023-03-03T07:57:17.325925+0000","last_undegraded":"2023-03-03T07:57:17.325925+0000","last_fullsized":"2023-03-03T07:57:17.325925+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:20.464414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.322541+0000","last_change":"2023-03-03T07:57:17.319788+0000","last_active":"2023-03-03T07:57:17.322541+0000","last_peered":"2023-03-03T07:57:17.322541+0000","last_clean":"2023-03-03T07:57:17.322541+0000","last_became_active":"2023-03-03T07:57:17.319506+0000","last_became_peered":"2023-03-03T07:57:17.319506+0000","last_unstale":"2023-03-03T07:57:17.322541+0000","last_undegraded":"2023-03-03T07:57:17.322541+0000","last_fullsized":"2023-03-03T07:57:17.322541+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.321276+0000","last_change":"2023-03-03T07:57:17.318816+0000","last_active":"2023-03-03T07:57:17.321276+0000","last_peered":"2023-03-03T07:57:17.321276+0000","last_clean":"2023-03-03T07:57:17.321276+0000","last_became_active":"2023-03-03T07:57:17.318429+0000","last_became_peered":"2023-03-03T07:57:17.318429+0000","last_unstale":"2023-03-03T07:57:17.321276+0000","last_undegraded":"2023-03-03T07:57:17.321276+0000","last_fullsized":"2023-03-03T07:57:17.321276+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:07.498551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.321241+0000","last_change":"2023-03-03T07:57:17.318945+0000","last_active":"2023-03-03T07:57:17.321241+0000","last_peered":"2023-03-03T07:57:17.321241+0000","last_clean":"2023-03-03T07:57:17.321241+0000","last_became_active":"2023-03-03T07:57:17.318619+0000","last_became_peered":"2023-03-03T07:57:17.318619+0000","last_unstale":"2023-03-03T07:57:17.321241+0000","last_undegraded":"2023-03-03T07:57:17.321241+0000","last_fullsized":"2023-03-03T07:57:17.321241+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:43.532953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.332800+0000","last_change":"2023-03-03T07:57:17.323441+0000","last_active":"2023-03-03T07:57:17.332800+0000","last_peered":"2023-03-03T07:57:17.332800+0000","last_clean":"2023-03-03T07:57:17.332800+0000","last_became_active":"2023-03-03T07:57:17.323125+0000","last_became_peered":"2023-03-03T07:57:17.323125+0000","last_unstale":"2023-03-03T07:57:17.332800+0000","last_undegraded":"2023-03-03T07:57:17.332800+0000","last_fullsized":"2023-03-03T07:57:17.332800+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.320990+0000","last_change":"2023-03-03T07:57:17.317951+0000","last_active":"2023-03-03T07:57:17.320990+0000","last_peered":"2023-03-03T07:57:17.320990+0000","last_clean":"2023-03-03T07:57:17.320990+0000","last_became_active":"2023-03-03T07:57:17.317678+0000","last_became_peered":"2023-03-03T07:57:17.317678+0000","last_unstale":"2023-03-03T07:57:17.320990+0000","last_undegraded":"2023-03-03T07:57:17.320990+0000","last_fullsized":"2023-03-03T07:57:17.320990+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.655223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.324564+0000","last_change":"2023-03-03T07:57:17.321833+0000","last_active":"2023-03-03T07:57:17.324564+0000","last_peered":"2023-03-03T07:57:17.324564+0000","last_clean":"2023-03-03T07:57:17.324564+0000","last_became_active":"2023-03-03T07:57:17.321543+0000","last_became_peered":"2023-03-03T07:57:17.321543+0000","last_unstale":"2023-03-03T07:57:17.324564+0000","last_undegraded":"2023-03-03T07:57:17.324564+0000","last_fullsized":"2023-03-03T07:57:17.324564+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.334383+0000","last_change":"2023-03-03T07:57:16.745475+0000","last_active":"2023-03-03T07:57:17.334383+0000","last_peered":"2023-03-03T07:57:17.334383+0000","last_clean":"2023-03-03T07:57:17.334383+0000","last_became_active":"2023-03-03T07:57:16.745159+0000","last_became_peered":"2023-03-03T07:57:16.745159+0000","last_unstale":"2023-03-03T07:57:17.334383+0000","last_undegraded":"2023-03-03T07:57:17.334383+0000","last_fullsized":"2023-03-03T07:57:17.334383+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:45:49.372030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.330939+0000","last_change":"2023-03-03T07:57:17.325195+0000","last_active":"2023-03-03T07:57:17.330939+0000","last_peered":"2023-03-03T07:57:17.330939+0000","last_clean":"2023-03-03T07:57:17.330939+0000","last_became_active":"2023-03-03T07:57:17.324904+0000","last_became_peered":"2023-03-03T07:57:17.324904+0000","last_unstale":"2023-03-03T07:57:17.330939+0000","last_undegraded":"2023-03-03T07:57:17.330939+0000","last_fullsized":"2023-03-03T07:57:17.330939+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.323043+0000","last_change":"2023-03-03T07:57:16.755153+0000","last_active":"2023-03-03T07:57:17.323043+0000","last_peered":"2023-03-03T07:57:17.323043+0000","last_clean":"2023-03-03T07:57:17.323043+0000","last_became_active":"2023-03-03T07:57:16.754738+0000","last_became_peered":"2023-03-03T07:57:16.754738+0000","last_unstale":"2023-03-03T07:57:17.323043+0000","last_undegraded":"2023-03-03T07:57:17.323043+0000","last_fullsized":"2023-03-03T07:57:17.323043+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:14.522208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335341+0000","last_change":"2023-03-03T07:57:17.327812+0000","last_active":"2023-03-03T07:57:17.335341+0000","last_peered":"2023-03-03T07:57:17.335341+0000","last_clean":"2023-03-03T07:57:17.335341+0000","last_became_active":"2023-03-03T07:57:17.327531+0000","last_became_peered":"2023-03-03T07:57:17.327531+0000","last_unstale":"2023-03-03T07:57:17.335341+0000","last_undegraded":"2023-03-03T07:57:17.335341+0000","last_fullsized":"2023-03-03T07:57:17.335341+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:11.090994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.333541+0000","last_change":"2023-03-03T07:57:16.746651+0000","last_active":"2023-03-03T07:57:17.333541+0000","last_peered":"2023-03-03T07:57:17.333541+0000","last_clean":"2023-03-03T07:57:17.333541+0000","last_became_active":"2023-03-03T07:57:16.746390+0000","last_became_peered":"2023-03-03T07:57:16.746390+0000","last_unstale":"2023-03-03T07:57:17.333541+0000","last_undegraded":"2023-03-03T07:57:17.333541+0000","last_fullsized":"2023-03-03T07:57:17.333541+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:06.489627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.327422+0000","last_change":"2023-03-03T07:57:17.324462+0000","last_active":"2023-03-03T07:57:17.327422+0000","last_peered":"2023-03-03T07:57:17.327422+0000","last_clean":"2023-03-03T07:57:17.327422+0000","last_became_active":"2023-03-03T07:57:17.324090+0000","last_became_peered":"2023-03-03T07:57:17.324090+0000","last_unstale":"2023-03-03T07:57:17.327422+0000","last_undegraded":"2023-03-03T07:57:17.327422+0000","last_fullsized":"2023-03-03T07:57:17.327422+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:11.090994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.333541+0000","last_change":"2023-03-03T07:57:16.743191+0000","last_active":"2023-03-03T07:57:17.333541+0000","last_peered":"2023-03-03T07:57:17.333541+0000","last_clean":"2023-03-03T07:57:17.333541+0000","last_became_active":"2023-03-03T07:57:16.741858+0000","last_became_peered":"2023-03-03T07:57:16.741858+0000","last_unstale":"2023-03-03T07:57:17.333541+0000","last_undegraded":"2023-03-03T07:57:17.333541+0000","last_fullsized":"2023-03-03T07:57:17.333541+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:00.944189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.324965+0000","last_change":"2023-03-03T07:57:17.319886+0000","last_active":"2023-03-03T07:57:17.324965+0000","last_peered":"2023-03-03T07:57:17.324965+0000","last_clean":"2023-03-03T07:57:17.324965+0000","last_became_active":"2023-03-03T07:57:17.319614+0000","last_became_peered":"2023-03-03T07:57:17.319614+0000","last_unstale":"2023-03-03T07:57:17.324965+0000","last_undegraded":"2023-03-03T07:57:17.324965+0000","last_fullsized":"2023-03-03T07:57:17.324965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:01.953113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.325981+0000","last_change":"2023-03-03T07:57:16.753759+0000","last_active":"2023-03-03T07:57:17.325981+0000","last_peered":"2023-03-03T07:57:17.325981+0000","last_clean":"2023-03-03T07:57:17.325981+0000","last_became_active":"2023-03-03T07:57:16.753449+0000","last_became_peered":"2023-03-03T07:57:16.753449+0000","last_unstale":"2023-03-03T07:57:17.325981+0000","last_undegraded":"2023-03-03T07:57:17.325981+0000","last_fullsized":"2023-03-03T07:57:17.325981+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:56:09.130132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.322448+0000","last_change":"2023-03-03T07:57:17.317169+0000","last_active":"2023-03-03T07:57:17.322448+0000","last_peered":"2023-03-03T07:57:17.322448+0000","last_clean":"2023-03-03T07:57:17.322448+0000","last_became_active":"2023-03-03T07:57:17.316817+0000","last_became_peered":"2023-03-03T07:57:17.316817+0000","last_unstale":"2023-03-03T07:57:17.322448+0000","last_undegraded":"2023-03-03T07:57:17.322448+0000","last_fullsized":"2023-03-03T07:57:17.322448+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.336157+0000","last_change":"2023-03-03T07:57:16.740511+0000","last_active":"2023-03-03T07:57:17.336157+0000","last_peered":"2023-03-03T07:57:17.336157+0000","last_clean":"2023-03-03T07:57:17.336157+0000","last_became_active":"2023-03-03T07:57:16.740221+0000","last_became_peered":"2023-03-03T07:57:16.740221+0000","last_unstale":"2023-03-03T07:57:17.336157+0000","last_undegraded":"2023-03-03T07:57:17.336157+0000","last_fullsized":"2023-03-03T07:57:17.336157+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:10.938115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.336403+0000","last_change":"2023-03-03T07:57:17.334046+0000","last_active":"2023-03-03T07:57:17.336403+0000","last_peered":"2023-03-03T07:57:17.336403+0000","last_clean":"2023-03-03T07:57:17.336403+0000","last_became_active":"2023-03-03T07:57:17.333764+0000","last_became_peered":"2023-03-03T07:57:17.333764+0000","last_unstale":"2023-03-03T07:57:17.336403+0000","last_undegraded":"2023-03-03T07:57:17.336403+0000","last_fullsized":"2023-03-03T07:57:17.336403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:39:39.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.303992+0000","last_change":"2023-03-03T07:57:16.756171+0000","last_active":"2023-03-03T07:57:17.303992+0000","last_peered":"2023-03-03T07:57:17.303992+0000","last_clean":"2023-03-03T07:57:17.303992+0000","last_became_active":"2023-03-03T07:57:16.755894+0000","last_became_peered":"2023-03-03T07:57:16.755894+0000","last_unstale":"2023-03-03T07:57:17.303992+0000","last_undegraded":"2023-03-03T07:57:17.303992+0000","last_fullsized":"2023-03-03T07:57:17.303992+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:19.455490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.332685+0000","last_change":"2023-03-03T07:57:17.330901+0000","last_active":"2023-03-03T07:57:17.332685+0000","last_peered":"2023-03-03T07:57:17.332685+0000","last_clean":"2023-03-03T07:57:17.332685+0000","last_became_active":"2023-03-03T07:57:17.330613+0000","last_became_peered":"2023-03-03T07:57:17.330613+0000","last_unstale":"2023-03-03T07:57:17.332685+0000","last_undegraded":"2023-03-03T07:57:17.332685+0000","last_fullsized":"2023-03-03T07:57:17.332685+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:48:58.988637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.312998+0000","last_change":"2023-03-03T07:57:16.744111+0000","last_active":"2023-03-03T07:57:17.312998+0000","last_peered":"2023-03-03T07:57:17.312998+0000","last_clean":"2023-03-03T07:57:17.312998+0000","last_became_active":"2023-03-03T07:57:16.743778+0000","last_became_peered":"2023-03-03T07:57:16.743778+0000","last_unstale":"2023-03-03T07:57:17.312998+0000","last_undegraded":"2023-03-03T07:57:17.312998+0000","last_fullsized":"2023-03-03T07:57:17.312998+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:52:59.206045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.334869+0000","last_change":"2023-03-03T07:57:17.331846+0000","last_active":"2023-03-03T07:57:17.334869+0000","last_peered":"2023-03-03T07:57:17.334869+0000","last_clean":"2023-03-03T07:57:17.334869+0000","last_became_active":"2023-03-03T07:57:17.331542+0000","last_became_peered":"2023-03-03T07:57:17.331542+0000","last_unstale":"2023-03-03T07:57:17.334869+0000","last_undegraded":"2023-03-03T07:57:17.334869+0000","last_fullsized":"2023-03-03T07:57:17.334869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:11.947039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.314856+0000","last_change":"2023-03-03T07:57:16.744161+0000","last_active":"2023-03-03T07:57:17.314856+0000","last_peered":"2023-03-03T07:57:17.314856+0000","last_clean":"2023-03-03T07:57:17.314856+0000","last_became_active":"2023-03-03T07:57:16.743909+0000","last_became_peered":"2023-03-03T07:57:16.743909+0000","last_unstale":"2023-03-03T07:57:17.314856+0000","last_undegraded":"2023-03-03T07:57:17.314856+0000","last_fullsized":"2023-03-03T07:57:17.314856+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:43:42.175275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.333622+0000","last_change":"2023-03-03T07:57:17.324785+0000","last_active":"2023-03-03T07:57:17.333622+0000","last_peered":"2023-03-03T07:57:17.333622+0000","last_clean":"2023-03-03T07:57:17.333622+0000","last_became_active":"2023-03-03T07:57:17.324510+0000","last_became_peered":"2023-03-03T07:57:17.324510+0000","last_unstale":"2023-03-03T07:57:17.333622+0000","last_undegraded":"2023-03-03T07:57:17.333622+0000","last_fullsized":"2023-03-03T07:57:17.333622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:00.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.304107+0000","last_change":"2023-03-03T07:57:16.754673+0000","last_active":"2023-03-03T07:57:17.304107+0000","last_peered":"2023-03-03T07:57:17.304107+0000","last_clean":"2023-03-03T07:57:17.304107+0000","last_became_active":"2023-03-03T07:57:16.754415+0000","last_became_peered":"2023-03-03T07:57:16.754415+0000","last_unstale":"2023-03-03T07:57:17.304107+0000","last_undegraded":"2023-03-03T07:57:17.304107+0000","last_fullsized":"2023-03-03T07:57:17.304107+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:06.489627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.329305+0000","last_change":"2023-03-03T07:57:17.323654+0000","last_active":"2023-03-03T07:57:17.329305+0000","last_peered":"2023-03-03T07:57:17.329305+0000","last_clean":"2023-03-03T07:57:17.329305+0000","last_became_active":"2023-03-03T07:57:17.323356+0000","last_became_peered":"2023-03-03T07:57:17.323356+0000","last_unstale":"2023-03-03T07:57:17.329305+0000","last_undegraded":"2023-03-03T07:57:17.329305+0000","last_fullsized":"2023-03-03T07:57:17.329305+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:44.827206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.326392+0000","last_change":"2023-03-03T07:57:16.757208+0000","last_active":"2023-03-03T07:57:17.326392+0000","last_peered":"2023-03-03T07:57:17.326392+0000","last_clean":"2023-03-03T07:57:17.326392+0000","last_became_active":"2023-03-03T07:57:16.756910+0000","last_became_peered":"2023-03-03T07:57:16.756910+0000","last_unstale":"2023-03-03T07:57:17.326392+0000","last_undegraded":"2023-03-03T07:57:17.326392+0000","last_fullsized":"2023-03-03T07:57:17.326392+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:14.522208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335575+0000","last_change":"2023-03-03T07:57:17.330250+0000","last_active":"2023-03-03T07:57:17.335575+0000","last_peered":"2023-03-03T07:57:17.335575+0000","last_clean":"2023-03-03T07:57:17.335575+0000","last_became_active":"2023-03-03T07:57:17.329997+0000","last_became_peered":"2023-03-03T07:57:17.329997+0000","last_unstale":"2023-03-03T07:57:17.335575+0000","last_undegraded":"2023-03-03T07:57:17.335575+0000","last_fullsized":"2023-03-03T07:57:17.335575+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:01.871020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":17,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.337068+0000","last_change":"2023-03-03T07:57:16.749713+0000","last_active":"2023-03-03T07:57:17.337068+0000","last_peered":"2023-03-03T07:57:17.337068+0000","last_clean":"2023-03-03T07:57:17.337068+0000","last_became_active":"2023-03-03T07:57:16.749304+0000","last_became_peered":"2023-03-03T07:57:16.749304+0000","last_unstale":"2023-03-03T07:57:17.337068+0000","last_undegraded":"2023-03-03T07:57:17.337068+0000","last_fullsized":"2023-03-03T07:57:17.337068+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:19:07.613725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.331274+0000","last_change":"2023-03-03T07:57:17.325082+0000","last_active":"2023-03-03T07:57:17.331274+0000","last_peered":"2023-03-03T07:57:17.331274+0000","last_clean":"2023-03-03T07:57:17.331274+0000","last_became_active":"2023-03-03T07:57:17.324787+0000","last_became_peered":"2023-03-03T07:57:17.324787+0000","last_unstale":"2023-03-03T07:57:17.331274+0000","last_undegraded":"2023-03-03T07:57:17.331274+0000","last_fullsized":"2023-03-03T07:57:17.331274+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:21:12.720267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.326087+0000","last_change":"2023-03-03T07:57:16.753448+0000","last_active":"2023-03-03T07:57:17.326087+0000","last_peered":"2023-03-03T07:57:17.326087+0000","last_clean":"2023-03-03T07:57:17.326087+0000","last_became_active":"2023-03-03T07:57:16.753183+0000","last_became_peered":"2023-03-03T07:57:16.753183+0000","last_unstale":"2023-03-03T07:57:17.326087+0000","last_undegraded":"2023-03-03T07:57:17.326087+0000","last_fullsized":"2023-03-03T07:57:17.326087+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:54.897881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.327376+0000","last_change":"2023-03-03T07:57:17.319586+0000","last_active":"2023-03-03T07:57:17.327376+0000","last_peered":"2023-03-03T07:57:17.327376+0000","last_clean":"2023-03-03T07:57:17.327376+0000","last_became_active":"2023-03-03T07:57:17.319343+0000","last_became_peered":"2023-03-03T07:57:17.319343+0000","last_unstale":"2023-03-03T07:57:17.327376+0000","last_undegraded":"2023-03-03T07:57:17.327376+0000","last_fullsized":"2023-03-03T07:57:17.327376+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:24:04.359311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.304263+0000","last_change":"2023-03-03T07:57:16.754628+0000","last_active":"2023-03-03T07:57:17.304263+0000","last_peered":"2023-03-03T07:57:17.304263+0000","last_clean":"2023-03-03T07:57:17.304263+0000","last_became_active":"2023-03-03T07:57:16.754370+0000","last_became_peered":"2023-03-03T07:57:16.754370+0000","last_unstale":"2023-03-03T07:57:17.304263+0000","last_undegraded":"2023-03-03T07:57:17.304263+0000","last_fullsized":"2023-03-03T07:57:17.304263+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:14.522208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.325593+0000","last_change":"2023-03-03T07:57:17.322380+0000","last_active":"2023-03-03T07:57:17.325593+0000","last_peered":"2023-03-03T07:57:17.325593+0000","last_clean":"2023-03-03T07:57:17.325593+0000","last_became_active":"2023-03-03T07:57:17.322085+0000","last_became_peered":"2023-03-03T07:57:17.322085+0000","last_unstale":"2023-03-03T07:57:17.325593+0000","last_undegraded":"2023-03-03T07:57:17.325593+0000","last_fullsized":"2023-03-03T07:57:17.325593+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:21:12.720267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.304795+0000","last_change":"2023-03-03T07:57:16.745348+0000","last_active":"2023-03-03T07:57:17.304795+0000","last_peered":"2023-03-03T07:57:17.304795+0000","last_clean":"2023-03-03T07:57:17.304795+0000","last_became_active":"2023-03-03T07:57:16.745100+0000","last_became_peered":"2023-03-03T07:57:16.745100+0000","last_unstale":"2023-03-03T07:57:17.304795+0000","last_undegraded":"2023-03-03T07:57:17.304795+0000","last_fullsized":"2023-03-03T07:57:17.304795+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:17:28.990710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.325155+0000","last_change":"2023-03-03T07:57:17.318561+0000","last_active":"2023-03-03T07:57:17.325155+0000","last_peered":"2023-03-03T07:57:17.325155+0000","last_clean":"2023-03-03T07:57:17.325155+0000","last_became_active":"2023-03-03T07:57:17.318287+0000","last_became_peered":"2023-03-03T07:57:17.318287+0000","last_unstale":"2023-03-03T07:57:17.325155+0000","last_undegraded":"2023-03-03T07:57:17.325155+0000","last_fullsized":"2023-03-03T07:57:17.325155+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:35:32.174190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":19,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.336687+0000","last_change":"2023-03-03T07:57:16.736066+0000","last_active":"2023-03-03T07:57:17.336687+0000","last_peered":"2023-03-03T07:57:17.336687+0000","last_clean":"2023-03-03T07:57:17.336687+0000","last_became_active":"2023-03-03T07:57:16.735650+0000","last_became_peered":"2023-03-03T07:57:16.735650+0000","last_unstale":"2023-03-03T07:57:17.336687+0000","last_undegraded":"2023-03-03T07:57:17.336687+0000","last_fullsized":"2023-03-03T07:57:17.336687+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:00:22.441246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.327965+0000","last_change":"2023-03-03T07:57:17.323207+0000","last_active":"2023-03-03T07:57:17.327965+0000","last_peered":"2023-03-03T07:57:17.327965+0000","last_clean":"2023-03-03T07:57:17.327965+0000","last_became_active":"2023-03-03T07:57:17.322950+0000","last_became_peered":"2023-03-03T07:57:17.322950+0000","last_unstale":"2023-03-03T07:57:17.327965+0000","last_undegraded":"2023-03-03T07:57:17.327965+0000","last_fullsized":"2023-03-03T07:57:17.327965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:44.621837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.334024+0000","last_change":"2023-03-03T07:57:16.756645+0000","last_active":"2023-03-03T07:57:17.334024+0000","last_peered":"2023-03-03T07:57:17.334024+0000","last_clean":"2023-03-03T07:57:17.334024+0000","last_became_active":"2023-03-03T07:57:16.756201+0000","last_became_peered":"2023-03-03T07:57:16.756201+0000","last_unstale":"2023-03-03T07:57:17.334024+0000","last_undegraded":"2023-03-03T07:57:17.334024+0000","last_fullsized":"2023-03-03T07:57:17.334024+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:42.524029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.322596+0000","last_change":"2023-03-03T07:57:17.319019+0000","last_active":"2023-03-03T07:57:17.322596+0000","last_peered":"2023-03-03T07:57:17.322596+0000","last_clean":"2023-03-03T07:57:17.322596+0000","last_became_active":"2023-03-03T07:57:17.318759+0000","last_became_peered":"2023-03-03T07:57:17.318759+0000","last_unstale":"2023-03-03T07:57:17.322596+0000","last_undegraded":"2023-03-03T07:57:17.322596+0000","last_fullsized":"2023-03-03T07:57:17.322596+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:13:17.532289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.333446+0000","last_change":"2023-03-03T07:57:16.747099+0000","last_active":"2023-03-03T07:57:17.333446+0000","last_peered":"2023-03-03T07:57:17.333446+0000","last_clean":"2023-03-03T07:57:17.333446+0000","last_became_active":"2023-03-03T07:57:16.746766+0000","last_became_peered":"2023-03-03T07:57:16.746766+0000","last_unstale":"2023-03-03T07:57:17.333446+0000","last_undegraded":"2023-03-03T07:57:17.333446+0000","last_fullsized":"2023-03-03T07:57:17.333446+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:25.646299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335717+0000","last_change":"2023-03-03T07:57:17.330772+0000","last_active":"2023-03-03T07:57:17.335717+0000","last_peered":"2023-03-03T07:57:17.335717+0000","last_clean":"2023-03-03T07:57:17.335717+0000","last_became_active":"2023-03-03T07:57:17.330534+0000","last_became_peered":"2023-03-03T07:57:17.330534+0000","last_unstale":"2023-03-03T07:57:17.335717+0000","last_undegraded":"2023-03-03T07:57:17.335717+0000","last_fullsized":"2023-03-03T07:57:17.335717+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:35:32.174190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.328035+0000","last_change":"2023-03-03T07:57:16.748443+0000","last_active":"2023-03-03T07:57:17.328035+0000","last_peered":"2023-03-03T07:57:17.328035+0000","last_clean":"2023-03-03T07:57:17.328035+0000","last_became_active":"2023-03-03T07:57:16.748172+0000","last_became_peered":"2023-03-03T07:57:16.748172+0000","last_unstale":"2023-03-03T07:57:17.328035+0000","last_undegraded":"2023-03-03T07:57:17.328035+0000","last_fullsized":"2023-03-03T07:57:17.328035+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:07:16.498493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.324459+0000","last_change":"2023-03-03T07:57:17.320055+0000","last_active":"2023-03-03T07:57:17.324459+0000","last_peered":"2023-03-03T07:57:17.324459+0000","last_clean":"2023-03-03T07:57:17.324459+0000","last_became_active":"2023-03-03T07:57:17.319759+0000","last_became_peered":"2023-03-03T07:57:17.319759+0000","last_unstale":"2023-03-03T07:57:17.324459+0000","last_undegraded":"2023-03-03T07:57:17.324459+0000","last_fullsized":"2023-03-03T07:57:17.324459+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:43.532953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.303222+0000","last_change":"2023-03-03T07:57:16.746019+0000","last_active":"2023-03-03T07:57:17.303222+0000","last_peered":"2023-03-03T07:57:17.303222+0000","last_clean":"2023-03-03T07:57:17.303222+0000","last_became_active":"2023-03-03T07:57:16.745766+0000","last_became_peered":"2023-03-03T07:57:16.745766+0000","last_unstale":"2023-03-03T07:57:17.303222+0000","last_undegraded":"2023-03-03T07:57:17.303222+0000","last_fullsized":"2023-03-03T07:57:17.303222+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:42.524029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.328673+0000","last_change":"2023-03-03T07:57:17.323174+0000","last_active":"2023-03-03T07:57:17.328673+0000","last_peered":"2023-03-03T07:57:17.328673+0000","last_clean":"2023-03-03T07:57:17.328673+0000","last_became_active":"2023-03-03T07:57:17.322647+0000","last_became_peered":"2023-03-03T07:57:17.322647+0000","last_unstale":"2023-03-03T07:57:17.328673+0000","last_undegraded":"2023-03-03T07:57:17.328673+0000","last_fullsized":"2023-03-03T07:57:17.328673+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:20:39.338427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.318767+0000","last_change":"2023-03-03T07:57:16.753514+0000","last_active":"2023-03-03T07:57:17.318767+0000","last_peered":"2023-03-03T07:57:17.318767+0000","last_clean":"2023-03-03T07:57:17.318767+0000","last_became_active":"2023-03-03T07:57:16.753206+0000","last_became_peered":"2023-03-03T07:57:16.753206+0000","last_unstale":"2023-03-03T07:57:17.318767+0000","last_undegraded":"2023-03-03T07:57:17.318767+0000","last_fullsized":"2023-03-03T07:57:17.318767+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:43:26.659371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.322268+0000","last_change":"2023-03-03T07:57:17.319507+0000","last_active":"2023-03-03T07:57:17.322268+0000","last_peered":"2023-03-03T07:57:17.322268+0000","last_clean":"2023-03-03T07:57:17.322268+0000","last_became_active":"2023-03-03T07:57:17.319238+0000","last_became_peered":"2023-03-03T07:57:17.319238+0000","last_unstale":"2023-03-03T07:57:17.322268+0000","last_undegraded":"2023-03-03T07:57:17.322268+0000","last_fullsized":"2023-03-03T07:57:17.322268+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:41:26.001415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.298919+0000","last_change":"2023-03-03T07:57:16.754775+0000","last_active":"2023-03-03T07:57:17.298919+0000","last_peered":"2023-03-03T07:57:17.298919+0000","last_clean":"2023-03-03T07:57:17.298919+0000","last_became_active":"2023-03-03T07:57:16.754315+0000","last_became_peered":"2023-03-03T07:57:16.754315+0000","last_unstale":"2023-03-03T07:57:17.298919+0000","last_undegraded":"2023-03-03T07:57:17.298919+0000","last_fullsized":"2023-03-03T07:57:17.298919+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:14:32.071416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.320588+0000","last_change":"2023-03-03T07:57:17.316091+0000","last_active":"2023-03-03T07:57:17.320588+0000","last_peered":"2023-03-03T07:57:17.320588+0000","last_clean":"2023-03-03T07:57:17.320588+0000","last_became_active":"2023-03-03T07:57:17.315658+0000","last_became_peered":"2023-03-03T07:57:17.315658+0000","last_unstale":"2023-03-03T07:57:17.320588+0000","last_undegraded":"2023-03-03T07:57:17.320588+0000","last_fullsized":"2023-03-03T07:57:17.320588+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.306600+0000","last_change":"2023-03-03T07:57:16.742812+0000","last_active":"2023-03-03T07:57:17.306600+0000","last_peered":"2023-03-03T07:57:17.306600+0000","last_clean":"2023-03-03T07:57:17.306600+0000","last_became_active":"2023-03-03T07:57:16.742529+0000","last_became_peered":"2023-03-03T07:57:16.742529+0000","last_unstale":"2023-03-03T07:57:17.306600+0000","last_undegraded":"2023-03-03T07:57:17.306600+0000","last_fullsized":"2023-03-03T07:57:17.306600+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:37:49.692413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.332674+0000","last_change":"2023-03-03T07:57:17.330640+0000","last_active":"2023-03-03T07:57:17.332674+0000","last_peered":"2023-03-03T07:57:17.332674+0000","last_clean":"2023-03-03T07:57:17.332674+0000","last_became_active":"2023-03-03T07:57:17.330390+0000","last_became_peered":"2023-03-03T07:57:17.330390+0000","last_unstale":"2023-03-03T07:57:17.332674+0000","last_undegraded":"2023-03-03T07:57:17.332674+0000","last_fullsized":"2023-03-03T07:57:17.332674+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.299395+0000","last_change":"2023-03-03T07:57:16.753472+0000","last_active":"2023-03-03T07:57:17.299395+0000","last_peered":"2023-03-03T07:57:17.299395+0000","last_clean":"2023-03-03T07:57:17.299395+0000","last_became_active":"2023-03-03T07:57:16.753183+0000","last_became_peered":"2023-03-03T07:57:16.753183+0000","last_unstale":"2023-03-03T07:57:17.299395+0000","last_undegraded":"2023-03-03T07:57:17.299395+0000","last_fullsized":"2023-03-03T07:57:17.299395+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:53.094916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.331292+0000","last_change":"2023-03-03T07:57:17.328971+0000","last_active":"2023-03-03T07:57:17.331292+0000","last_peered":"2023-03-03T07:57:17.331292+0000","last_clean":"2023-03-03T07:57:17.331292+0000","last_became_active":"2023-03-03T07:57:17.328679+0000","last_became_peered":"2023-03-03T07:57:17.328679+0000","last_unstale":"2023-03-03T07:57:17.331292+0000","last_undegraded":"2023-03-03T07:57:17.331292+0000","last_fullsized":"2023-03-03T07:57:17.331292+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:11.164758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.307500+0000","last_change":"2023-03-03T07:57:16.751997+0000","last_active":"2023-03-03T07:57:17.307500+0000","last_peered":"2023-03-03T07:57:17.307500+0000","last_clean":"2023-03-03T07:57:17.307500+0000","last_became_active":"2023-03-03T07:57:16.751593+0000","last_became_peered":"2023-03-03T07:57:16.751593+0000","last_unstale":"2023-03-03T07:57:17.307500+0000","last_undegraded":"2023-03-03T07:57:17.307500+0000","last_fullsized":"2023-03-03T07:57:17.307500+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:19.455490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.303300+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T07:57:17.303300+0000","last_peered":"2023-03-03T07:57:17.303300+0000","last_clean":"2023-03-03T07:57:17.303300+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T07:57:17.303300+0000","last_undegraded":"2023-03-03T07:57:17.303300+0000","last_fullsized":"2023-03-03T07:57:17.303300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335374+0000","last_change":"2023-03-03T07:57:17.333984+0000","last_active":"2023-03-03T07:57:17.335374+0000","last_peered":"2023-03-03T07:57:17.335374+0000","last_clean":"2023-03-03T07:57:17.335374+0000","last_became_active":"2023-03-03T07:57:17.333692+0000","last_became_peered":"2023-03-03T07:57:17.333692+0000","last_unstale":"2023-03-03T07:57:17.335374+0000","last_undegraded":"2023-03-03T07:57:17.335374+0000","last_fullsized":"2023-03-03T07:57:17.335374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.309367+0000","last_change":"2023-03-03T07:57:16.749605+0000","last_active":"2023-03-03T07:57:17.309367+0000","last_peered":"2023-03-03T07:57:17.309367+0000","last_clean":"2023-03-03T07:57:17.309367+0000","last_became_active":"2023-03-03T07:57:16.749336+0000","last_became_peered":"2023-03-03T07:57:16.749336+0000","last_unstale":"2023-03-03T07:57:17.309367+0000","last_undegraded":"2023-03-03T07:57:17.309367+0000","last_fullsized":"2023-03-03T07:57:17.309367+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:10:42.524029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.303849+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T07:57:17.303849+0000","last_peered":"2023-03-03T07:57:17.303849+0000","last_clean":"2023-03-03T07:57:17.303849+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T07:57:17.303849+0000","last_undegraded":"2023-03-03T07:57:17.303849+0000","last_fullsized":"2023-03-03T07:57:17.303849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.334440+0000","last_change":"2023-03-03T07:57:17.324746+0000","last_active":"2023-03-03T07:57:17.334440+0000","last_peered":"2023-03-03T07:57:17.334440+0000","last_clean":"2023-03-03T07:57:17.334440+0000","last_became_active":"2023-03-03T07:57:17.323971+0000","last_became_peered":"2023-03-03T07:57:17.323971+0000","last_unstale":"2023-03-03T07:57:17.334440+0000","last_undegraded":"2023-03-03T07:57:17.334440+0000","last_fullsized":"2023-03-03T07:57:17.334440+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"37'1","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:18.031859+0000","last_change":"2023-03-03T07:57:16.750917+0000","last_active":"2023-03-03T07:57:18.031859+0000","last_peered":"2023-03-03T07:57:18.031859+0000","last_clean":"2023-03-03T07:57:18.031859+0000","last_became_active":"2023-03-03T07:57:16.750615+0000","last_became_peered":"2023-03-03T07:57:16.750615+0000","last_unstale":"2023-03-03T07:57:18.031859+0000","last_undegraded":"2023-03-03T07:57:18.031859+0000","last_fullsized":"2023-03-03T07:57:18.031859+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:49:51.007473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.300085+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T07:57:17.300085+0000","last_peered":"2023-03-03T07:57:17.300085+0000","last_clean":"2023-03-03T07:57:17.300085+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T07:57:17.300085+0000","last_undegraded":"2023-03-03T07:57:17.300085+0000","last_fullsized":"2023-03-03T07:57:17.300085+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.327122+0000","last_change":"2023-03-03T07:57:17.320314+0000","last_active":"2023-03-03T07:57:17.327122+0000","last_peered":"2023-03-03T07:57:17.327122+0000","last_clean":"2023-03-03T07:57:17.327122+0000","last_became_active":"2023-03-03T07:57:17.320033+0000","last_became_peered":"2023-03-03T07:57:17.320033+0000","last_unstale":"2023-03-03T07:57:17.327122+0000","last_undegraded":"2023-03-03T07:57:17.327122+0000","last_fullsized":"2023-03-03T07:57:17.327122+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:26.550536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.307831+0000","last_change":"2023-03-03T07:57:16.748170+0000","last_active":"2023-03-03T07:57:17.307831+0000","last_peered":"2023-03-03T07:57:17.307831+0000","last_clean":"2023-03-03T07:57:17.307831+0000","last_became_active":"2023-03-03T07:57:16.747842+0000","last_became_peered":"2023-03-03T07:57:16.747842+0000","last_unstale":"2023-03-03T07:57:17.307831+0000","last_undegraded":"2023-03-03T07:57:17.307831+0000","last_fullsized":"2023-03-03T07:57:17.307831+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:57:03.533693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.302674+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T07:57:17.302674+0000","last_peered":"2023-03-03T07:57:17.302674+0000","last_clean":"2023-03-03T07:57:17.302674+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T07:57:17.302674+0000","last_undegraded":"2023-03-03T07:57:17.302674+0000","last_fullsized":"2023-03-03T07:57:17.302674+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.307302+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T07:57:17.307302+0000","last_peered":"2023-03-03T07:57:17.307302+0000","last_clean":"2023-03-03T07:57:17.307302+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T07:57:17.307302+0000","last_undegraded":"2023-03-03T07:57:17.307302+0000","last_fullsized":"2023-03-03T07:57:17.307302+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.315023+0000","last_change":"2023-03-03T07:57:17.311231+0000","last_active":"2023-03-03T07:57:17.315023+0000","last_peered":"2023-03-03T07:57:17.315023+0000","last_clean":"2023-03-03T07:57:17.315023+0000","last_became_active":"2023-03-03T07:57:17.310950+0000","last_became_peered":"2023-03-03T07:57:17.310950+0000","last_unstale":"2023-03-03T07:57:17.315023+0000","last_undegraded":"2023-03-03T07:57:17.315023+0000","last_fullsized":"2023-03-03T07:57:17.315023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:56.358453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.324947+0000","last_change":"2023-03-03T07:57:16.756473+0000","last_active":"2023-03-03T07:57:17.324947+0000","last_peered":"2023-03-03T07:57:17.324947+0000","last_clean":"2023-03-03T07:57:17.324947+0000","last_became_active":"2023-03-03T07:57:16.755809+0000","last_became_peered":"2023-03-03T07:57:16.755809+0000","last_unstale":"2023-03-03T07:57:17.324947+0000","last_undegraded":"2023-03-03T07:57:17.324947+0000","last_fullsized":"2023-03-03T07:57:17.324947+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:19.455490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.307386+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T07:57:17.307386+0000","last_peered":"2023-03-03T07:57:17.307386+0000","last_clean":"2023-03-03T07:57:17.307386+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T07:57:17.307386+0000","last_undegraded":"2023-03-03T07:57:17.307386+0000","last_fullsized":"2023-03-03T07:57:17.307386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.324457+0000","last_change":"2023-03-03T07:57:17.316375+0000","last_active":"2023-03-03T07:57:17.324457+0000","last_peered":"2023-03-03T07:57:17.324457+0000","last_clean":"2023-03-03T07:57:17.324457+0000","last_became_active":"2023-03-03T07:57:17.316069+0000","last_became_peered":"2023-03-03T07:57:17.316069+0000","last_unstale":"2023-03-03T07:57:17.324457+0000","last_undegraded":"2023-03-03T07:57:17.324457+0000","last_fullsized":"2023-03-03T07:57:17.324457+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:59.426058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.305735+0000","last_change":"2023-03-03T07:57:16.752378+0000","last_active":"2023-03-03T07:57:17.305735+0000","last_peered":"2023-03-03T07:57:17.305735+0000","last_clean":"2023-03-03T07:57:17.305735+0000","last_became_active":"2023-03-03T07:57:16.752028+0000","last_became_peered":"2023-03-03T07:57:16.752028+0000","last_unstale":"2023-03-03T07:57:17.305735+0000","last_undegraded":"2023-03-03T07:57:17.305735+0000","last_fullsized":"2023-03-03T07:57:17.305735+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:14:02.921691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.308253+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T07:57:17.308253+0000","last_peered":"2023-03-03T07:57:17.308253+0000","last_clean":"2023-03-03T07:57:17.308253+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T07:57:17.308253+0000","last_undegraded":"2023-03-03T07:57:17.308253+0000","last_fullsized":"2023-03-03T07:57:17.308253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.335408+0000","last_change":"2023-03-03T07:57:17.328978+0000","last_active":"2023-03-03T07:57:17.335408+0000","last_peered":"2023-03-03T07:57:17.335408+0000","last_clean":"2023-03-03T07:57:17.335408+0000","last_became_active":"2023-03-03T07:57:17.328542+0000","last_became_peered":"2023-03-03T07:57:17.328542+0000","last_unstale":"2023-03-03T07:57:17.335408+0000","last_undegraded":"2023-03-03T07:57:17.335408+0000","last_fullsized":"2023-03-03T07:57:17.335408+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:58.191359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.312893+0000","last_change":"2023-03-03T07:57:16.742160+0000","last_active":"2023-03-03T07:57:17.312893+0000","last_peered":"2023-03-03T07:57:17.312893+0000","last_clean":"2023-03-03T07:57:17.312893+0000","last_became_active":"2023-03-03T07:57:16.741796+0000","last_became_peered":"2023-03-03T07:57:16.741796+0000","last_unstale":"2023-03-03T07:57:17.312893+0000","last_undegraded":"2023-03-03T07:57:17.312893+0000","last_fullsized":"2023-03-03T07:57:17.312893+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:55.349529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.304389+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T07:57:17.304389+0000","last_peered":"2023-03-03T07:57:17.304389+0000","last_clean":"2023-03-03T07:57:17.304389+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T07:57:17.304389+0000","last_undegraded":"2023-03-03T07:57:17.304389+0000","last_fullsized":"2023-03-03T07:57:17.304389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.333773+0000","last_change":"2023-03-03T07:57:17.325655+0000","last_active":"2023-03-03T07:57:17.333773+0000","last_peered":"2023-03-03T07:57:17.333773+0000","last_clean":"2023-03-03T07:57:17.333773+0000","last_became_active":"2023-03-03T07:57:17.325418+0000","last_became_peered":"2023-03-03T07:57:17.325418+0000","last_unstale":"2023-03-03T07:57:17.333773+0000","last_undegraded":"2023-03-03T07:57:17.333773+0000","last_fullsized":"2023-03-03T07:57:17.333773+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:16.718506+0000","last_clean_scrub_stamp":"2023-03-03T07:57:16.718506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:25.355832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.309427+0000","last_change":"2023-03-03T07:57:16.744051+0000","last_active":"2023-03-03T07:57:17.309427+0000","last_peered":"2023-03-03T07:57:17.309427+0000","last_clean":"2023-03-03T07:57:17.309427+0000","last_became_active":"2023-03-03T07:57:16.743785+0000","last_became_peered":"2023-03-03T07:57:16.743785+0000","last_unstale":"2023-03-03T07:57:17.309427+0000","last_undegraded":"2023-03-03T07:57:17.309427+0000","last_fullsized":"2023-03-03T07:57:17.309427+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:57:15.709582+0000","last_clean_scrub_stamp":"2023-03-03T07:57:15.709582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:48:13.119986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-03-03T07:57:17.306114+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T07:57:17.306114+0000","last_peered":"2023-03-03T07:57:17.306114+0000","last_clean":"2023-03-03T07:57:17.306114+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T07:57:17.306114+0000","last_undegraded":"2023-03-03T07:57:17.306114+0000","last_fullsized":"2023-03-03T07:57:17.306114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9604,"kb_used_data":1020,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362236,"statfs":{"total":96636764160,"available":96626929664,"internally_reserved":0,"allocated":1044480,"data_stored":772784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":0.58199999999999996}]},{"osd":1,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":6.6829999999999998,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":0.68600000000000005}]},{"osd":2,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.71999999999999997}]},{"osd":3,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.61099999999999999}]},{"osd":4,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.745},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.90300000000000002}]},{"osd":5,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.89600000000000002}]},{"osd":6,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.82599999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8368,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363472,"statfs":{"total":96636764160,"available":96628195328,"internally_reserved":0,"allocated":434176,"data_stored":180345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.69799999999999995}]},{"osd":1,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":1.0029999999999999}]},{"osd":2,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.871},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.98099999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.77200000000000002}]},{"osd":4,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":4.0099999999999998,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":4.1859999999999999,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.60299999999999998}]},{"osd":5,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.74099999999999999}]},{"osd":7,"last update":"Fri Mar 3 07:56:33 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.82199999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574872,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":180888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.972,"5min":3.972,"15min":3.972},"last":0.439}]},{"osd":2,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":11.897,"5min":11.897,"15min":11.897},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":11.804,"5min":11.804,"15min":11.804},"last":0.56499999999999995}]},{"osd":3,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":11.843999999999999,"5min":11.843999999999999,"15min":11.843999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":3.8740000000000001,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.503}]},{"osd":4,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.876},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.88400000000000001}]},{"osd":5,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":3.8300000000000001,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.77900000000000003}]},{"osd":6,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":3.907,"5min":3.907,"15min":3.907},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.91500000000000004}]},{"osd":7,"last update":"Fri Mar 3 07:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":3.9399999999999999,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":0.72699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9524,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":8512,"kb_avail":94362316,"statfs":{"total":96636764160,"available":96627011584,"internally_reserved":0,"allocated":1028096,"data_stored":770181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8716288},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.51200000000000001}]},{"osd":2,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.46000000000000002}]},{"osd":3,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.53900000000000003}]},{"osd":4,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.77500000000000002}]},{"osd":6,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.624}]},{"osd":7,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.754}]}]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8368,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363472,"statfs":{"total":96636764160,"available":96628195328,"internally_reserved":0,"allocated":434176,"data_stored":179849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":5.7889999999999997,"5min":5.7889999999999997,"15min":5.7889999999999997},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":2.6309999999999998,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":5.8470000000000004,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.625}]},{"osd":3,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.961,"5min":12.961,"15min":12.961},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.1449999999999996,"5min":6.1449999999999996,"15min":6.1449999999999996},"last":0.53100000000000003}]},{"osd":4,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":6.484,"5min":6.484,"15min":6.484},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":6.5510000000000002,"5min":6.5510000000000002,"15min":6.5510000000000002},"last":0.91000000000000003}]},{"osd":5,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":5.7119999999999997,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":6.6369999999999996,"5min":6.6369999999999996,"15min":6.6369999999999996},"last":0.89700000000000002}]},{"osd":6,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":6.625,"5min":6.625,"15min":6.625},"last":0.872},{"interface":"front","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":5.7539999999999996,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.91700000000000004}]},{"osd":7,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":6.7009999999999996,"5min":6.7009999999999996,"15min":6.7009999999999996},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":6.6840000000000002,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.995}]}]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8576,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94363264,"statfs":{"total":96636764160,"available":96627982336,"internally_reserved":0,"allocated":450560,"data_stored":182380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":9.5820000000000007,"5min":9.5820000000000007,"15min":9.5820000000000007},"last":0.317},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":9.7840000000000007,"5min":9.7840000000000007,"15min":9.7840000000000007},"last":0.44900000000000001}]},{"osd":1,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.702,"5min":9.702,"15min":9.702},"last":0.47399999999999998}]},{"osd":2,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":8.3190000000000008,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.133,"5min":1.133,"15min":1.133},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":9.734,"5min":9.734,"15min":9.734},"last":0.51000000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":9.8629999999999995,"5min":9.8629999999999995,"15min":9.8629999999999995},"last":0.754},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.80600000000000005}]},{"osd":5,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":9.6600000000000001,"5min":9.6600000000000001,"15min":9.6600000000000001},"last":0.73199999999999998}]},{"osd":6,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":9.8190000000000008,"5min":9.8190000000000008,"15min":9.8190000000000008},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.72199999999999998}]},{"osd":7,"last update":"Fri Mar 3 07:56:32 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":9.625,"5min":9.625,"15min":9.625},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":9.8949999999999996,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":0.70799999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8632,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363208,"statfs":{"total":96636764160,"available":96627924992,"internally_reserved":0,"allocated":442368,"data_stored":181386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":7.1799999999999997,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.86199999999999999}]},{"osd":1,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":9.5709999999999997,"5min":9.5709999999999997,"15min":9.5709999999999997},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":5.266,"5min":5.266,"15min":5.266},"last":0.88}]},{"osd":2,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":8.4529999999999994,"5min":8.4529999999999994,"15min":8.4529999999999994},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":6.6829999999999998,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.83799999999999997}]},{"osd":3,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.84999999999999998}]},{"osd":5,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.74299999999999999}]},{"osd":6,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.77800000000000002}]},{"osd":7,"last update":"Fri Mar 3 07:56:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.72499999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":179891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.70999999999999996}]},{"osd":1,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.45500000000000002}]},{"osd":2,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.65600000000000003}]},{"osd":3,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.76000000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.82399999999999995}]},{"osd":6,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.73899999999999999}]},{"osd":7,"last update":"Fri Mar 3 07:56:31 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T07:57:28.011 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-03-03T07:57:28.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2023-03-03T07:57:28.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.298+0000 7fa48eb9d700 1 Processor -- start 2023-03-03T07:57:28.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.299+0000 7fa48eb9d700 1 -- start start 2023-03-03T07:57:28.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.300+0000 7fa48eb9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa4880cf570 0x7fa48815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:28.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.300+0000 7fa48eb9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4880cd780 con 0x7fa4881609b0 2023-03-03T07:57:28.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.300+0000 7fa48eb9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4880cda60 con 0x7fa48815db30 2023-03-03T07:57:28.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.300+0000 7fa48eb9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4880cebb0 con 0x7fa4880cf570 2023-03-03T07:57:28.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.300+0000 7fa48db9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa4881609b0 0x7fa488160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47694/0 (socket says 172.21.15.98:47694) 2023-03-03T07:57:28.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.300+0000 7fa48db9b700 1 -- 172.21.15.98:0/2859205829 learned_addr learned my addr 172.21.15.98:0/2859205829 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:28.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.300+0000 7fa48d39a700 1 --2- 172.21.15.98:0/2859205829 >> v2:172.21.15.98:3300/0 conn(0x7fa4880cf570 0x7fa48815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:28.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.301+0000 7fa48cb99700 1 -- 172.21.15.98:0/2859205829 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866084866 0 0) 0x7fa4880cd780 con 0x7fa4881609b0 2023-03-03T07:57:28.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.301+0000 7fa48d39a700 1 -- 172.21.15.98:0/2859205829 >> v1:172.21.15.100:6789/0 conn(0x7fa48815db30 legacy=0x7fa4880b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:28.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.301+0000 7fa48d39a700 1 -- 172.21.15.98:0/2859205829 >> v1:172.21.15.98:6789/0 conn(0x7fa4881609b0 legacy=0x7fa488160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:28.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.301+0000 7fa48d39a700 1 -- 172.21.15.98:0/2859205829 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4880cdd40 con 0x7fa4880cf570 2023-03-03T07:57:28.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.301+0000 7fa48d39a700 1 --2- 172.21.15.98:0/2859205829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4880cf570 0x7fa48815d5f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa478009840 tx=0x7fa478009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7852c5912341c58 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:28.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.302+0000 7fa48cb99700 1 -- 172.21.15.98:0/2859205829 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa478004230 con 0x7fa4880cf570 2023-03-03T07:57:28.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.302+0000 7fa48cb99700 1 -- 172.21.15.98:0/2859205829 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa478004390 con 0x7fa4880cf570 2023-03-03T07:57:28.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.302+0000 7fa48cb99700 1 -- 172.21.15.98:0/2859205829 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4780036a0 con 0x7fa4880cf570 2023-03-03T07:57:28.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.302+0000 7fa48eb9d700 1 -- 172.21.15.98:0/2859205829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4880cf570 msgr2=0x7fa48815d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:28.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.302+0000 7fa48eb9d700 1 --2- 172.21.15.98:0/2859205829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4880cf570 0x7fa48815d5f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa478009840 tx=0x7fa478009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:28.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.302+0000 7fa48eb9d700 1 -- 172.21.15.98:0/2859205829 shutdown_connections 2023-03-03T07:57:28.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.302+0000 7fa48eb9d700 1 --2- 172.21.15.98:0/2859205829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4880cf570 0x7fa48815d5f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:28.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.302+0000 7fa48eb9d700 1 -- 172.21.15.98:0/2859205829 >> 172.21.15.98:0/2859205829 conn(0x7fa48806b880 msgr2=0x7fa4880cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:28.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.302+0000 7fa48eb9d700 1 -- 172.21.15.98:0/2859205829 shutdown_connections 2023-03-03T07:57:28.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.302+0000 7fa48eb9d700 1 -- 172.21.15.98:0/2859205829 wait complete. 2023-03-03T07:57:28.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.304+0000 7fa48eb9d700 1 Processor -- start 2023-03-03T07:57:28.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.305+0000 7fa48eb9d700 1 -- start start 2023-03-03T07:57:28.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.306+0000 7fa48eb9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4880cf570 0x7fa4880b5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:28.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.306+0000 7fa48eb9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48815db30 0x7fa488151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:28.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.306+0000 7fa48eb9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4881609b0 0x7fa48814f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:28.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.306+0000 7fa48eb9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4880b8640 con 0x7fa4881609b0 2023-03-03T07:57:28.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.306+0000 7fa48eb9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4880c76c0 con 0x7fa4880cf570 2023-03-03T07:57:28.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.306+0000 7fa48eb9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4880b6390 con 0x7fa48815db30 2023-03-03T07:57:28.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.306+0000 7fa48db9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4880cf570 0x7fa4880b5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:28.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.306+0000 7fa48db9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4880cf570 0x7fa4880b5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:58026/0 (socket says 172.21.15.98:58026) 2023-03-03T07:57:28.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.306+0000 7fa48e39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4881609b0 0x7fa48814f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:28.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.306+0000 7fa48db9b700 1 -- 172.21.15.98:0/2431491448 learned_addr learned my addr 172.21.15.98:0/2431491448 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:28.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.306+0000 7fa48d39a700 1 --2- 172.21.15.98:0/2431491448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48815db30 0x7fa488151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:28.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.306+0000 7fa48e39c700 1 -- 172.21.15.98:0/2431491448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48815db30 msgr2=0x7fa488151080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:28.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.306+0000 7fa48e39c700 1 --2- 172.21.15.98:0/2431491448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48815db30 0x7fa488151080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:28.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.307+0000 7fa48e39c700 1 -- 172.21.15.98:0/2431491448 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4880cf570 msgr2=0x7fa4880b5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:28.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.307+0000 7fa48e39c700 1 --2- 172.21.15.98:0/2431491448 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4880cf570 0x7fa4880b5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:28.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.307+0000 7fa48e39c700 1 -- 172.21.15.98:0/2431491448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4780094a0 con 0x7fa4881609b0 2023-03-03T07:57:28.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.307+0000 7fa48e39c700 1 --2- 172.21.15.98:0/2431491448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4881609b0 0x7fa48814f1a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa48808c450 tx=0x7fa48000cdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:28.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.307+0000 7fa47e7fc700 1 -- 172.21.15.98:0/2431491448 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48000a9e0 con 0x7fa4881609b0 2023-03-03T07:57:28.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.307+0000 7fa47e7fc700 1 -- 172.21.15.98:0/2431491448 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48000ab40 con 0x7fa4881609b0 2023-03-03T07:57:28.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.307+0000 7fa47e7fc700 1 -- 172.21.15.98:0/2431491448 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480012420 con 0x7fa4881609b0 2023-03-03T07:57:28.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.307+0000 7fa48eb9d700 1 -- 172.21.15.98:0/2431491448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa48815d0d0 con 0x7fa4881609b0 2023-03-03T07:57:28.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.307+0000 7fa48eb9d700 1 -- 172.21.15.98:0/2431491448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48815cdf0 con 0x7fa4881609b0 2023-03-03T07:57:28.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.309+0000 7fa48eb9d700 1 -- 172.21.15.98:0/2431491448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4880b51b0 con 0x7fa4881609b0 2023-03-03T07:57:28.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.310+0000 7fa47e7fc700 1 -- 172.21.15.98:0/2431491448 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa480022070 con 0x7fa4881609b0 2023-03-03T07:57:28.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.311+0000 7fa47e7fc700 1 --2- 172.21.15.98:0/2431491448 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa46c0682f0 0x7fa46c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:28.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.311+0000 7fa47e7fc700 1 -- 172.21.15.98:0/2431491448 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fa480086180 con 0x7fa4881609b0 2023-03-03T07:57:28.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.311+0000 7fa48db9b700 1 --2- 172.21.15.98:0/2431491448 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa46c0682f0 0x7fa46c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:28.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.312+0000 7fa48db9b700 1 --2- 172.21.15.98:0/2431491448 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa46c0682f0 0x7fa46c06a7a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa484000f60 tx=0x7fa484007f30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:28.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.315+0000 7fa47e7fc700 1 -- 172.21.15.98:0/2431491448 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa4800528d0 con 0x7fa4881609b0 2023-03-03T07:57:28.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.587+0000 7fa48eb9d700 1 -- 172.21.15.98:0/2431491448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7fa4880c7490 con 0x7fa4881609b0 2023-03-03T07:57:28.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.591+0000 7fa47e7fc700 1 -- 172.21.15.98:0/2431491448 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v9) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa4880c7490 con 0x7fa4881609b0 2023-03-03T07:57:28.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.594+0000 7fa48eb9d700 1 -- 172.21.15.98:0/2431491448 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa46c0682f0 msgr2=0x7fa46c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:28.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.594+0000 7fa48eb9d700 1 --2- 172.21.15.98:0/2431491448 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa46c0682f0 0x7fa46c06a7a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa484000f60 tx=0x7fa484007f30 comp rx=0 tx=0).stop 2023-03-03T07:57:28.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.594+0000 7fa48eb9d700 1 -- 172.21.15.98:0/2431491448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4881609b0 msgr2=0x7fa48814f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:28.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.594+0000 7fa48eb9d700 1 --2- 172.21.15.98:0/2431491448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4881609b0 0x7fa48814f1a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa48808c450 tx=0x7fa48000cdc0 comp rx=0 tx=0).stop 2023-03-03T07:57:28.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.594+0000 7fa48eb9d700 1 -- 172.21.15.98:0/2431491448 shutdown_connections 2023-03-03T07:57:28.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.594+0000 7fa48eb9d700 1 --2- 172.21.15.98:0/2431491448 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa46c0682f0 0x7fa46c06a7a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:28.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.594+0000 7fa48eb9d700 1 --2- 172.21.15.98:0/2431491448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4881609b0 0x7fa48814f1a0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:28.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.594+0000 7fa48eb9d700 1 --2- 172.21.15.98:0/2431491448 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4880cf570 0x7fa4880b5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:28.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.594+0000 7fa48eb9d700 1 --2- 172.21.15.98:0/2431491448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48815db30 0x7fa488151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:28.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.594+0000 7fa48eb9d700 1 -- 172.21.15.98:0/2431491448 >> 172.21.15.98:0/2431491448 conn(0x7fa48806b880 msgr2=0x7fa488152210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:28.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.594+0000 7fa48eb9d700 1 -- 172.21.15.98:0/2431491448 shutdown_connections 2023-03-03T07:57:28.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.594+0000 7fa48eb9d700 1 -- 172.21.15.98:0/2431491448 wait complete. 2023-03-03T07:57:28.597 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.0 2023-03-03T07:57:28.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:57:28.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.869+0000 7f175c7ff700 1 Processor -- start 2023-03-03T07:57:28.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.869+0000 7f175c7ff700 1 -- start start 2023-03-03T07:57:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.870+0000 7f175c7ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f17540b87c0 0x7f17540cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.870+0000 7f175c7ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f17540b5410 0x7f17540d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.870+0000 7f175c7ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17540d0ef0 con 0x7f17540b8b90 2023-03-03T07:57:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.870+0000 7f175c7ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17540d0600 con 0x7f17540b5410 2023-03-03T07:57:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.870+0000 7f175c7ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17540cd970 con 0x7f17540b87c0 2023-03-03T07:57:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.871+0000 7f175ad9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f17540b5410 0x7f17540d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.871+0000 7f175ad9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f17540b5410 0x7f17540d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57662/0 (socket says 172.21.15.98:57662) 2023-03-03T07:57:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.871+0000 7f175ad9c700 1 -- 172.21.15.98:0/3426317905 learned_addr learned my addr 172.21.15.98:0/3426317905 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.871+0000 7f175a59b700 1 --2- 172.21.15.98:0/3426317905 >> v2:172.21.15.100:3300/0 conn(0x7f17540b87c0 0x7f17540cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.871+0000 7f175ad9c700 1 -- 172.21.15.98:0/3426317905 >> v2:172.21.15.100:3300/0 conn(0x7f17540b87c0 msgr2=0x7f17540cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.871+0000 7f175ad9c700 1 --2- 172.21.15.98:0/3426317905 >> v2:172.21.15.100:3300/0 conn(0x7f17540b87c0 0x7f17540cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.871+0000 7f175ad9c700 1 -- 172.21.15.98:0/3426317905 >> v1:172.21.15.100:6789/0 conn(0x7f17540b8b90 legacy=0x7f17540b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.871+0000 7f175ad9c700 1 -- 172.21.15.98:0/3426317905 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17540cdc50 con 0x7f17540b5410 2023-03-03T07:57:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.872+0000 7f175ad9c700 1 --2- 172.21.15.98:0/3426317905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17540b5410 0x7f17540d13d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1754073840 tx=0x7f174c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca4a824c9a65863 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.872+0000 7f1759599700 1 -- 172.21.15.98:0/3426317905 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c00e070 con 0x7f17540b5410 2023-03-03T07:57:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.872+0000 7f1759599700 1 -- 172.21.15.98:0/3426317905 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174c003a20 con 0x7f17540b5410 2023-03-03T07:57:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.872+0000 7f1759599700 1 -- 172.21.15.98:0/3426317905 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c003dd0 con 0x7f17540b5410 2023-03-03T07:57:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.872+0000 7f175c7ff700 1 -- 172.21.15.98:0/3426317905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17540b5410 msgr2=0x7f17540d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.872+0000 7f175c7ff700 1 --2- 172.21.15.98:0/3426317905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17540b5410 0x7f17540d13d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1754073840 tx=0x7f174c00b330 comp rx=0 tx=0).stop 2023-03-03T07:57:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.872+0000 7f175c7ff700 1 -- 172.21.15.98:0/3426317905 shutdown_connections 2023-03-03T07:57:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.872+0000 7f175c7ff700 1 --2- 172.21.15.98:0/3426317905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17540b5410 0x7f17540d13d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.872+0000 7f175c7ff700 1 --2- 172.21.15.98:0/3426317905 >> v2:172.21.15.100:3300/0 conn(0x7f17540b87c0 0x7f17540cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.873+0000 7f175c7ff700 1 -- 172.21.15.98:0/3426317905 >> 172.21.15.98:0/3426317905 conn(0x7f175406aee0 msgr2=0x7f175406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.873+0000 7f175c7ff700 1 -- 172.21.15.98:0/3426317905 shutdown_connections 2023-03-03T07:57:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.873+0000 7f175c7ff700 1 -- 172.21.15.98:0/3426317905 wait complete. 2023-03-03T07:57:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.875+0000 7f175c7ff700 1 Processor -- start 2023-03-03T07:57:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.875+0000 7f175c7ff700 1 -- start start 2023-03-03T07:57:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.876+0000 7f175c7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17540b5410 0x7f175414f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.876+0000 7f175c7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17540b87c0 0x7f17540cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.876+0000 7f175c7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17540b8b90 0x7f17541f5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.876+0000 7f175c7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17540ccd50 con 0x7f17540b8b90 2023-03-03T07:57:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.876+0000 7f175c7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175414fc20 con 0x7f17540b87c0 2023-03-03T07:57:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.876+0000 7f175c7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17540cc790 con 0x7f17540b5410 2023-03-03T07:57:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.876+0000 7f175ad9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17540b8b90 0x7f17541f5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.876+0000 7f175ad9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17540b8b90 0x7f17541f5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57674/0 (socket says 172.21.15.98:57674) 2023-03-03T07:57:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.876+0000 7f175ad9c700 1 -- 172.21.15.98:0/2176110902 learned_addr learned my addr 172.21.15.98:0/2176110902 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.876+0000 7f1759d9a700 1 --2- 172.21.15.98:0/2176110902 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17540b87c0 0x7f17540cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.876+0000 7f175a59b700 1 --2- 172.21.15.98:0/2176110902 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17540b5410 0x7f175414f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.877+0000 7f175ad9c700 1 -- 172.21.15.98:0/2176110902 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17540b5410 msgr2=0x7f175414f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.877+0000 7f175ad9c700 1 --2- 172.21.15.98:0/2176110902 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17540b5410 0x7f175414f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.877+0000 7f175ad9c700 1 -- 172.21.15.98:0/2176110902 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17540b87c0 msgr2=0x7f17540cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.877+0000 7f175ad9c700 1 --2- 172.21.15.98:0/2176110902 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17540b87c0 0x7f17540cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.877+0000 7f175ad9c700 1 -- 172.21.15.98:0/2176110902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c009b30 con 0x7f17540b8b90 2023-03-03T07:57:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.877+0000 7f175ad9c700 1 --2- 172.21.15.98:0/2176110902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17540b8b90 0x7f17541f5cf0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f174c015040 tx=0x7f174c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.877+0000 7f174affd700 1 -- 172.21.15.98:0/2176110902 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c004440 con 0x7f17540b8b90 2023-03-03T07:57:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.877+0000 7f174affd700 1 -- 172.21.15.98:0/2176110902 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174c0045c0 con 0x7f17540b8b90 2023-03-03T07:57:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.877+0000 7f175c7ff700 1 -- 172.21.15.98:0/2176110902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17540cdf30 con 0x7f17540b8b90 2023-03-03T07:57:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.877+0000 7f174affd700 1 -- 172.21.15.98:0/2176110902 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c0094d0 con 0x7f17540b8b90 2023-03-03T07:57:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.878+0000 7f175c7ff700 1 -- 172.21.15.98:0/2176110902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1754073a80 con 0x7f17540b8b90 2023-03-03T07:57:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.879+0000 7f175c7ff700 1 -- 172.21.15.98:0/2176110902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17540ce0c0 con 0x7f17540b8b90 2023-03-03T07:57:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.880+0000 7f174affd700 1 -- 172.21.15.98:0/2176110902 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f174c022030 con 0x7f17540b8b90 2023-03-03T07:57:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.881+0000 7f174affd700 1 --2- 172.21.15.98:0/2176110902 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f17380682f0 0x7f173806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.881+0000 7f174affd700 1 -- 172.21.15.98:0/2176110902 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f174c086f80 con 0x7f17540b8b90 2023-03-03T07:57:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.882+0000 7f175a59b700 1 --2- 172.21.15.98:0/2176110902 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f17380682f0 0x7f173806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.882+0000 7f175a59b700 1 --2- 172.21.15.98:0/2176110902 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f17380682f0 0x7f173806a7a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1750006b20 tx=0x7f1750008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:28.885+0000 7f174affd700 1 -- 172.21.15.98:0/2176110902 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f174c0536d0 con 0x7f17540b8b90 2023-03-03T07:57:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.119+0000 7f175c7ff700 1 -- 172.21.15.98:0/2176110902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17540729b0 con 0x7f17540b8b90 2023-03-03T07:57:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.120+0000 7f174affd700 1 -- 172.21.15.98:0/2176110902 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f174c056cf0 con 0x7f17540b8b90 2023-03-03T07:57:29.121 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:29.122 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3849166495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3849166495},{"type":"v1","addr":"172.21.15.100:6834","nonce":3849166495}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/4285491249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":4285491249},{"type":"v1","addr":"172.21.15.100:6835","nonce":4285491249}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":5208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3888598890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3888598890},{"type":"v1","addr":"172.21.15.98:6836","nonce":3888598890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:57:17.291546+0000","modified":"2023-03-03T07:57:20.741013+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5227},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5227":{"gid":5227,"name":"c","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.98:6837/1163251191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1163251191},{"type":"v1","addr":"172.21.15.98:6837","nonce":1163251191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T07:57:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.122+0000 7f175c7ff700 1 -- 172.21.15.98:0/2176110902 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f17380682f0 msgr2=0x7f173806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.122+0000 7f175c7ff700 1 --2- 172.21.15.98:0/2176110902 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f17380682f0 0x7f173806a7a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1750006b20 tx=0x7f1750008040 comp rx=0 tx=0).stop 2023-03-03T07:57:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.122+0000 7f175c7ff700 1 -- 172.21.15.98:0/2176110902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17540b8b90 msgr2=0x7f17541f5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.122+0000 7f175c7ff700 1 --2- 172.21.15.98:0/2176110902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17540b8b90 0x7f17541f5cf0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f174c015040 tx=0x7f174c004ad0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.122+0000 7f175c7ff700 1 -- 172.21.15.98:0/2176110902 shutdown_connections 2023-03-03T07:57:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.122+0000 7f175c7ff700 1 --2- 172.21.15.98:0/2176110902 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f17380682f0 0x7f173806a7a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.122+0000 7f175c7ff700 1 --2- 172.21.15.98:0/2176110902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17540b8b90 0x7f17541f5cf0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.122+0000 7f175c7ff700 1 --2- 172.21.15.98:0/2176110902 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17540b5410 0x7f175414f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.122+0000 7f175c7ff700 1 --2- 172.21.15.98:0/2176110902 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17540b87c0 0x7f17540cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.122+0000 7f175c7ff700 1 -- 172.21.15.98:0/2176110902 >> 172.21.15.98:0/2176110902 conn(0x7f175406aee0 msgr2=0x7f17540b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.122+0000 7f175c7ff700 1 -- 172.21.15.98:0/2176110902 shutdown_connections 2023-03-03T07:57:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.122+0000 7f175c7ff700 1 -- 172.21.15.98:0/2176110902 wait complete. 2023-03-03T07:57:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 14 2023-03-03T07:57:29.554 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-03-03T07:57:17.291546+0000', 'modified': '2023-03-03T07:57:20.741013+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5227}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5227': {'gid': 5227, 'name': 'c', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.98:6837/1163251191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 1163251191}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 1163251191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-03-03T07:57:29.554 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-03-03T07:57:29.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-03-03T07:57:29.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.802+0000 7fcd3139d700 1 Processor -- start 2023-03-03T07:57:29.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.803+0000 7fcd3139d700 1 -- start start 2023-03-03T07:57:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.804+0000 7fcd3139d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fcd2c1609b0 0x7fcd2c160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.804+0000 7fcd3139d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcd2c0cf570 0x7fcd2c15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.804+0000 7fcd3139d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcd2c15db30 0x7fcd2c0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.804+0000 7fcd3139d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd2c0cebb0 con 0x7fcd2c15db30 2023-03-03T07:57:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.804+0000 7fcd3139d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd2c1612c0 con 0x7fcd2c1609b0 2023-03-03T07:57:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.804+0000 7fcd3139d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0cd780 con 0x7fcd2c0cf570 2023-03-03T07:57:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.805+0000 7fcd30b9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcd2c15db30 0x7fcd2c0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.805+0000 7fcd30b9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcd2c15db30 0x7fcd2c0b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57688/0 (socket says 172.21.15.98:57688) 2023-03-03T07:57:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.805+0000 7fcd30b9c700 1 -- 172.21.15.98:0/1277817385 learned_addr learned my addr 172.21.15.98:0/1277817385 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.805+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/1277817385 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcd2c0cf570 0x7fcd2c15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.805+0000 7fcd2bfff700 1 --2- 172.21.15.98:0/1277817385 >> v2:172.21.15.100:3300/0 conn(0x7fcd2c1609b0 0x7fcd2c160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.805+0000 7fcd30b9c700 1 -- 172.21.15.98:0/1277817385 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcd2c0cf570 msgr2=0x7fcd2c15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.805+0000 7fcd30b9c700 1 --2- 172.21.15.98:0/1277817385 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcd2c0cf570 0x7fcd2c15d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.805+0000 7fcd30b9c700 1 -- 172.21.15.98:0/1277817385 >> v2:172.21.15.100:3300/0 conn(0x7fcd2c1609b0 msgr2=0x7fcd2c160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.805+0000 7fcd30b9c700 1 --2- 172.21.15.98:0/1277817385 >> v2:172.21.15.100:3300/0 conn(0x7fcd2c1609b0 0x7fcd2c160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.805+0000 7fcd30b9c700 1 -- 172.21.15.98:0/1277817385 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2c0cda60 con 0x7fcd2c15db30 2023-03-03T07:57:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.805+0000 7fcd30b9c700 1 --2- 172.21.15.98:0/1277817385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c15db30 0x7fcd2c0b4aa0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0742b0 tx=0x7fcd2400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f7087280bb2e83d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.806+0000 7fcd2affd700 1 -- 172.21.15.98:0/1277817385 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2400e070 con 0x7fcd2c15db30 2023-03-03T07:57:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.806+0000 7fcd2affd700 1 -- 172.21.15.98:0/1277817385 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2400b9c0 con 0x7fcd2c15db30 2023-03-03T07:57:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.806+0000 7fcd2affd700 1 -- 172.21.15.98:0/1277817385 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2400bd70 con 0x7fcd2c15db30 2023-03-03T07:57:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.806+0000 7fcd3139d700 1 -- 172.21.15.98:0/1277817385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c15db30 msgr2=0x7fcd2c0b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.806+0000 7fcd3139d700 1 --2- 172.21.15.98:0/1277817385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c15db30 0x7fcd2c0b4aa0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0742b0 tx=0x7fcd2400b330 comp rx=0 tx=0).stop 2023-03-03T07:57:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.806+0000 7fcd3139d700 1 -- 172.21.15.98:0/1277817385 shutdown_connections 2023-03-03T07:57:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.806+0000 7fcd3139d700 1 --2- 172.21.15.98:0/1277817385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c15db30 0x7fcd2c0b4aa0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.806+0000 7fcd3139d700 1 --2- 172.21.15.98:0/1277817385 >> v2:172.21.15.100:3300/0 conn(0x7fcd2c1609b0 0x7fcd2c160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.806+0000 7fcd3139d700 1 --2- 172.21.15.98:0/1277817385 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcd2c0cf570 0x7fcd2c15d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.806+0000 7fcd3139d700 1 -- 172.21.15.98:0/1277817385 >> 172.21.15.98:0/1277817385 conn(0x7fcd2c06b880 msgr2=0x7fcd2c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.806+0000 7fcd3139d700 1 -- 172.21.15.98:0/1277817385 shutdown_connections 2023-03-03T07:57:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.807+0000 7fcd3139d700 1 -- 172.21.15.98:0/1277817385 wait complete. 2023-03-03T07:57:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.808+0000 7fcd3139d700 1 Processor -- start 2023-03-03T07:57:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.809+0000 7fcd3139d700 1 -- start start 2023-03-03T07:57:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.810+0000 7fcd3139d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c0cf570 0x7fcd2c150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.810+0000 7fcd3139d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcd2c15db30 0x7fcd2c0c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.810+0000 7fcd3139d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcd2c1609b0 0x7fcd2c15ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.810+0000 7fcd3139d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0c8000 con 0x7fcd2c0cf570 2023-03-03T07:57:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.810+0000 7fcd3139d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c1513e0 con 0x7fcd2c1609b0 2023-03-03T07:57:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.810+0000 7fcd3139d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0b74a0 con 0x7fcd2c15db30 2023-03-03T07:57:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.810+0000 7fcd2bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c0cf570 0x7fcd2c150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.810+0000 7fcd2bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c0cf570 0x7fcd2c150ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57702/0 (socket says 172.21.15.98:57702) 2023-03-03T07:57:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.810+0000 7fcd2bfff700 1 -- 172.21.15.98:0/2932835361 learned_addr learned my addr 172.21.15.98:0/2932835361 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.810+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/2932835361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcd2c15db30 0x7fcd2c0c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.810+0000 7fcd30b9c700 1 --2- 172.21.15.98:0/2932835361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcd2c1609b0 0x7fcd2c15ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.811+0000 7fcd2bfff700 1 -- 172.21.15.98:0/2932835361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcd2c15db30 msgr2=0x7fcd2c0c8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.811+0000 7fcd2bfff700 1 --2- 172.21.15.98:0/2932835361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcd2c15db30 0x7fcd2c0c8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.811+0000 7fcd2bfff700 1 -- 172.21.15.98:0/2932835361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcd2c1609b0 msgr2=0x7fcd2c15ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.811+0000 7fcd2bfff700 1 --2- 172.21.15.98:0/2932835361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcd2c1609b0 0x7fcd2c15ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.811+0000 7fcd2bfff700 1 -- 172.21.15.98:0/2932835361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd24009b30 con 0x7fcd2c0cf570 2023-03-03T07:57:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.811+0000 7fcd2bfff700 1 --2- 172.21.15.98:0/2932835361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c0cf570 0x7fcd2c150ea0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c00d800 tx=0x7fcd1c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.811+0000 7fcd28ff9700 1 -- 172.21.15.98:0/2932835361 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c0125f0 con 0x7fcd2c0cf570 2023-03-03T07:57:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.811+0000 7fcd28ff9700 1 -- 172.21.15.98:0/2932835361 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1c012750 con 0x7fcd2c0cf570 2023-03-03T07:57:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.811+0000 7fcd28ff9700 1 -- 172.21.15.98:0/2932835361 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c012b00 con 0x7fcd2c0cf570 2023-03-03T07:57:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.811+0000 7fcd3139d700 1 -- 172.21.15.98:0/2932835361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2c15cdf0 con 0x7fcd2c0cf570 2023-03-03T07:57:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.811+0000 7fcd3139d700 1 -- 172.21.15.98:0/2932835361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2c0cdd40 con 0x7fcd2c0cf570 2023-03-03T07:57:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.814+0000 7fcd28ff9700 1 -- 172.21.15.98:0/2932835361 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcd1c012c60 con 0x7fcd2c0cf570 2023-03-03T07:57:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.815+0000 7fcd28ff9700 1 --2- 172.21.15.98:0/2932835361 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcd180682f0 0x7fcd1806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.816+0000 7fcd28ff9700 1 -- 172.21.15.98:0/2932835361 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fcd1c086260 con 0x7fcd2c0cf570 2023-03-03T07:57:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.816+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/2932835361 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcd180682f0 0x7fcd1806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.816+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/2932835361 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcd180682f0 0x7fcd1806a7a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0b70a0 tx=0x7fcd20008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.817+0000 7fcd45f16b80 1 Processor -- start 2023-03-03T07:57:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.818+0000 7fcd45f16b80 1 -- start start 2023-03-03T07:57:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd45f16b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 0x559ce8fb9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd45f16b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 0x559ce8fe07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd45f16b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8f9aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd45f16b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce8f0f4f0 con 0x559ce8f9a5c0 2023-03-03T07:57:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd45f16b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce8e84b60 con 0x559ce8fb9ad0 2023-03-03T07:57:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd45f16b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ce8ef3ad0 con 0x559ce8f9bff0 2023-03-03T07:57:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd30b9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8f9aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd2bfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 0x559ce8fb9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd30b9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8f9aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57708/0 (socket says 172.21.15.98:57708) 2023-03-03T07:57:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd30b9c700 1 -- 172.21.15.98:0/608049559 learned_addr learned my addr 172.21.15.98:0/608049559 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/608049559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 0x559ce8fe07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd30b9c700 1 -- 172.21.15.98:0/608049559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 msgr2=0x559ce8fe07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd30b9c700 1 --2- 172.21.15.98:0/608049559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 0x559ce8fe07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd30b9c700 1 -- 172.21.15.98:0/608049559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 msgr2=0x559ce8fb9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd30b9c700 1 --2- 172.21.15.98:0/608049559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 0x559ce8fb9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.819+0000 7fcd30b9c700 1 -- 172.21.15.98:0/608049559 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ce8ce1540 con 0x559ce8f9a5c0 2023-03-03T07:57:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.820+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/608049559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 0x559ce8fe07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.820+0000 7fcd30b9c700 1 --2- 172.21.15.98:0/608049559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8f9aa30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009e20 tx=0x7fcd24010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f71f0fe85429908 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.820+0000 7fcd127fc700 1 -- 172.21.15.98:0/608049559 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2400bd30 con 0x559ce8f9a5c0 2023-03-03T07:57:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.820+0000 7fcd127fc700 1 -- 172.21.15.98:0/608049559 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2400be90 con 0x559ce8f9a5c0 2023-03-03T07:57:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.820+0000 7fcd127fc700 1 -- 172.21.15.98:0/608049559 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd24003c60 con 0x559ce8f9a5c0 2023-03-03T07:57:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.821+0000 7fcd45f16b80 1 -- 172.21.15.98:0/608049559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 msgr2=0x559ce8f9aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.821+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/608049559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8f9aa30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009e20 tx=0x7fcd24010000 comp rx=0 tx=0).stop 2023-03-03T07:57:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.821+0000 7fcd45f16b80 1 -- 172.21.15.98:0/608049559 shutdown_connections 2023-03-03T07:57:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.821+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/608049559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8f9aa30 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.821+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/608049559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 0x559ce8fb9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.821+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/608049559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 0x559ce8fe07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.821+0000 7fcd45f16b80 1 -- 172.21.15.98:0/608049559 >> 172.21.15.98:0/608049559 conn(0x559ce8f97e10 msgr2=0x559ce8f987a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.821+0000 7fcd45f16b80 1 -- 172.21.15.98:0/608049559 shutdown_connections 2023-03-03T07:57:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.821+0000 7fcd45f16b80 1 -- 172.21.15.98:0/608049559 wait complete. 2023-03-03T07:57:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.822+0000 7fcd45f16b80 1 -- start start 2023-03-03T07:57:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.822+0000 7fcd45f16b80 1 Processor -- start 2023-03-03T07:57:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.823+0000 7fcd45f16b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8fd7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.823+0000 7fcd45f16b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 0x559ce8fe7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.823+0000 7fcd45f16b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 0x559ce8fa3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.823+0000 7fcd45f16b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce8eece40 con 0x559ce8f9a5c0 2023-03-03T07:57:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.823+0000 7fcd45f16b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce8ce2140 con 0x559ce8fb9ad0 2023-03-03T07:57:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.823+0000 7fcd45f16b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ce8fd8e40 con 0x559ce8f9bff0 2023-03-03T07:57:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.824+0000 7fcd45f16b80 10 client.0 _ll_register_callbacks cb 0x559ce8e8b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T07:57:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.824+0000 7fcd2bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8fd7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.824+0000 7fcd2bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8fd7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57716/0 (socket says 172.21.15.98:57716) 2023-03-03T07:57:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.824+0000 7fcd2bfff700 1 -- 172.21.15.98:0/345135539 learned_addr learned my addr 172.21.15.98:0/345135539 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.824+0000 7fcd30b9c700 1 --2- 172.21.15.98:0/345135539 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 0x559ce8fa3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.824+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/345135539 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 0x559ce8fe7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.824+0000 7fcd2bfff700 1 -- 172.21.15.98:0/345135539 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 msgr2=0x559ce8fe7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.824+0000 7fcd2bfff700 1 --2- 172.21.15.98:0/345135539 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 0x559ce8fe7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.824+0000 7fcd2bfff700 1 -- 172.21.15.98:0/345135539 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 msgr2=0x559ce8fa3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.824+0000 7fcd2bfff700 1 --2- 172.21.15.98:0/345135539 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 0x559ce8fa3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.824+0000 7fcd2bfff700 1 -- 172.21.15.98:0/345135539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2c0cda60 con 0x559ce8f9a5c0 2023-03-03T07:57:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.824+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/345135539 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 0x559ce8fe7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:57:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.824+0000 7fcd2bfff700 1 --2- 172.21.15.98:0/345135539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8fd7770 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c052b70 tx=0x7fcd1c05c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebd270e4daa68ef server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.825+0000 7fcd0affd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-03-03T07:57:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.825+0000 7fcd0affd700 1 -- 172.21.15.98:0/345135539 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c055b60 con 0x559ce8f9a5c0 2023-03-03T07:57:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.825+0000 7fcd0affd700 1 -- 172.21.15.98:0/345135539 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fcd1c055cc0 con 0x559ce8f9a5c0 2023-03-03T07:57:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.825+0000 7fcd45f16b80 1 -- 172.21.15.98:0/345135539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559ce8f8cea0 con 0x559ce8f9a5c0 2023-03-03T07:57:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.825+0000 7fcd0affd700 1 -- 172.21.15.98:0/345135539 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd1c05e270 con 0x559ce8f9a5c0 2023-03-03T07:57:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.825+0000 7fcd45f16b80 10 client.5424 fetch_fsmap learned FSMap version 14 2023-03-03T07:57:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.825+0000 7fcd45f16b80 1 -- 172.21.15.98:0/345135539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) v3 -- 0x559ce8d1bb10 con 0x559ce8f9a5c0 2023-03-03T07:57:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.825+0000 7fcd0affd700 1 -- 172.21.15.98:0/345135539 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559ce8f8cea0 con 0x559ce8f9a5c0 2023-03-03T07:57:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.825+0000 7fcd0affd700 1 -- 172.21.15.98:0/345135539 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 14) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fcd1c084250 con 0x559ce8f9a5c0 2023-03-03T07:57:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.825+0000 7fcd45f16b80 10 client.5424 fetch_fsmap finished waiting for FSMap version 14 2023-03-03T07:57:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.825+0000 7fcd45f16b80 10 client.5424 resolve_mds: resolved name 'c' to daemon mds.c 2023-03-03T07:57:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.825+0000 7fcd45f16b80 20 client.5424 populate_metadata read hostname 'smithi098' 2023-03-03T07:57:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.825+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/345135539 >> [v2:172.21.15.98:6835/1163251191,v1:172.21.15.98:6837/1163251191] conn(0x559ce8fa67c0 0x559ce8ffca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.825+0000 7fcd45f16b80 4 client.5424 mds_command: new command op to 5227 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-03T07:57:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.825+0000 7fcd45f16b80 1 -- 172.21.15.98:0/345135539 --> [v2:172.21.15.98:6835/1163251191,v1:172.21.15.98:6837/1163251191] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559ce8ee6020 con 0x559ce8fa67c0 2023-03-03T07:57:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.826+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/345135539 >> [v2:172.21.15.98:6835/1163251191,v1:172.21.15.98:6837/1163251191] conn(0x559ce8fa67c0 0x559ce8ffca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.826+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/345135539 >> [v2:172.21.15.98:6835/1163251191,v1:172.21.15.98:6837/1163251191] conn(0x559ce8fa67c0 0x559ce8ffca50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0a4679eb08fd84a server_cookie=3717a69407149e97 in_seq=0 out_seq=0 2023-03-03T07:57:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.826+0000 7fcd0affd700 10 client.5424 ms_handle_connect on v2:172.21.15.98:6835/1163251191 2023-03-03T07:57:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.827+0000 7fcd0affd700 1 -- 172.21.15.98:0/345135539 <== mds.0 v2:172.21.15.98:6835/1163251191 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x559ce8ee6020 con 0x559ce8fa67c0 2023-03-03T07:57:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.827+0000 7fcd0affd700 10 client.5424 handle_command_reply: tid=0 2023-03-03T07:57:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.827+0000 7fcd45f16b80 1 client.5424 shutdown 2023-03-03T07:57:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.828+0000 7fcd45f16b80 1 -- 172.21.15.98:0/345135539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 msgr2=0x559ce8fd7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.828+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/345135539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8fd7770 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c052b70 tx=0x7fcd1c05c330 comp rx=0 tx=0).stop 2023-03-03T07:57:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.828+0000 7fcd45f16b80 1 -- 172.21.15.98:0/345135539 shutdown_connections 2023-03-03T07:57:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.828+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/345135539 >> [v2:172.21.15.98:6835/1163251191,v1:172.21.15.98:6837/1163251191] conn(0x559ce8fa67c0 0x559ce8ffca50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.828+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/345135539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8fd7770 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.828+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/345135539 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 0x559ce8fa3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.828+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/345135539 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 0x559ce8fe7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.828+0000 7fcd45f16b80 1 -- 172.21.15.98:0/345135539 >> 172.21.15.98:0/345135539 conn(0x559ce8f97e10 msgr2=0x559ce8f98db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.828+0000 7fcd0affd700 0 client.5424 ms_handle_reset on v2:172.21.15.98:6835/1163251191 2023-03-03T07:57:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.828+0000 7fcd45f16b80 1 -- 172.21.15.98:0/345135539 shutdown_connections 2023-03-03T07:57:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.828+0000 7fcd45f16b80 1 -- 172.21.15.98:0/345135539 wait complete. 2023-03-03T07:57:29.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.828+0000 7fcd45f16b80 20 client.5424 trim_cache size 0 max 16384 2023-03-03T07:57:29.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.847+0000 7fcd45f16b80 1 Processor -- start 2023-03-03T07:57:29.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.847+0000 7fcd45f16b80 1 -- start start 2023-03-03T07:57:29.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.848+0000 7fcd45f16b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8f94ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.848+0000 7fcd45f16b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 0x559ce8f97000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.848+0000 7fcd45f16b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 0x559ce903dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.849+0000 7fcd45f16b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce8fd9710 con 0x559ce8f9a5c0 2023-03-03T07:57:29.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.849+0000 7fcd45f16b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce8fc8630 con 0x559ce8fb9ad0 2023-03-03T07:57:29.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.849+0000 7fcd45f16b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ce8f9b680 con 0x559ce8f9bff0 2023-03-03T07:57:29.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.849+0000 7fcd2bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8f94ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.849+0000 7fcd2bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8f94ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57718/0 (socket says 172.21.15.98:57718) 2023-03-03T07:57:29.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.849+0000 7fcd2bfff700 1 -- 172.21.15.98:0/1269159050 learned_addr learned my addr 172.21.15.98:0/1269159050 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:29.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.849+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/1269159050 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 0x559ce8f97000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.849+0000 7fcd2bfff700 1 -- 172.21.15.98:0/1269159050 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 msgr2=0x559ce8f97000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:29.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.849+0000 7fcd2bfff700 1 --2- 172.21.15.98:0/1269159050 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 0x559ce8f97000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.849+0000 7fcd30b9c700 1 --2- 172.21.15.98:0/1269159050 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 0x559ce903dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.849+0000 7fcd2bfff700 1 -- 172.21.15.98:0/1269159050 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 msgr2=0x559ce903dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:29.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.849+0000 7fcd2bfff700 1 --2- 172.21.15.98:0/1269159050 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 0x559ce903dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.849+0000 7fcd2bfff700 1 -- 172.21.15.98:0/1269159050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ce8eb4230 con 0x559ce8f9a5c0 2023-03-03T07:57:29.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.849+0000 7fcd30b9c700 1 --2- 172.21.15.98:0/1269159050 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 0x559ce903dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:57:29.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.850+0000 7fcd2bfff700 1 --2- 172.21.15.98:0/1269159050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8f94ca0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c007fd0 tx=0x7fcd1c05a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda2986304fd8553 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:29.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.850+0000 7fcd12ffd700 1 -- 172.21.15.98:0/1269159050 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c084710 con 0x559ce8f9a5c0 2023-03-03T07:57:29.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.850+0000 7fcd12ffd700 1 -- 172.21.15.98:0/1269159050 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1c084250 con 0x559ce8f9a5c0 2023-03-03T07:57:29.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.850+0000 7fcd12ffd700 1 -- 172.21.15.98:0/1269159050 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c0843c0 con 0x559ce8f9a5c0 2023-03-03T07:57:29.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.850+0000 7fcd45f16b80 1 -- 172.21.15.98:0/1269159050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 msgr2=0x559ce8f94ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:29.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.850+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/1269159050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8f94ca0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c007fd0 tx=0x7fcd1c05a000 comp rx=0 tx=0).stop 2023-03-03T07:57:29.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.850+0000 7fcd45f16b80 1 -- 172.21.15.98:0/1269159050 shutdown_connections 2023-03-03T07:57:29.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.850+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/1269159050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce8f94ca0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.850+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/1269159050 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8fb9ad0 0x559ce903dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.850+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/1269159050 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8f9bff0 0x559ce8f97000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.850+0000 7fcd45f16b80 1 -- 172.21.15.98:0/1269159050 >> 172.21.15.98:0/1269159050 conn(0x559ce8f97e10 msgr2=0x559ce8f99d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:29.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.851+0000 7fcd45f16b80 1 -- 172.21.15.98:0/1269159050 shutdown_connections 2023-03-03T07:57:29.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.851+0000 7fcd45f16b80 1 -- 172.21.15.98:0/1269159050 wait complete. 2023-03-03T07:57:29.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.851+0000 7fcd45f16b80 1 -- start start 2023-03-03T07:57:29.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.852+0000 7fcd45f16b80 1 Processor -- start 2023-03-03T07:57:29.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd45f16b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce903d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd45f16b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8f9bff0 0x559ce8ff9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd45f16b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8fb9ad0 0x559ce8fd66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd45f16b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce8ff9740 con 0x559ce8f9a5c0 2023-03-03T07:57:29.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd45f16b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce8ff1e50 con 0x559ce8f9bff0 2023-03-03T07:57:29.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd45f16b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ce8fe36a0 con 0x559ce8fb9ad0 2023-03-03T07:57:29.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd45f16b80 10 client.0 _ll_register_callbacks cb 0x559ce8e8b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T07:57:29.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd2bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce903d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd2bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce903d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57722/0 (socket says 172.21.15.98:57722) 2023-03-03T07:57:29.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd2bfff700 1 -- 172.21.15.98:0/3441890232 learned_addr learned my addr 172.21.15.98:0/3441890232 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:29.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd30b9c700 1 --2- 172.21.15.98:0/3441890232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8fb9ad0 0x559ce8fd66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/3441890232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8f9bff0 0x559ce8ff9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd2bfff700 1 -- 172.21.15.98:0/3441890232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8fb9ad0 msgr2=0x559ce8fd66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:29.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd2bfff700 1 --2- 172.21.15.98:0/3441890232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8fb9ad0 0x559ce8fd66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd2bfff700 1 -- 172.21.15.98:0/3441890232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8f9bff0 msgr2=0x559ce8ff9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:29.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd2bfff700 1 --2- 172.21.15.98:0/3441890232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8f9bff0 0x559ce8ff9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd2bfff700 1 -- 172.21.15.98:0/3441890232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559ce8d1bb10 con 0x559ce8f9a5c0 2023-03-03T07:57:29.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.854+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/3441890232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8f9bff0 0x559ce8ff9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:29.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.853+0000 7fcd30b9c700 1 --2- 172.21.15.98:0/3441890232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8fb9ad0 0x559ce8fd66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:29.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.854+0000 7fcd2bfff700 1 --2- 172.21.15.98:0/3441890232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce903d120 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c0565a0 tx=0x7fcd1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d8e6745ca0508fb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:29.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.854+0000 7fcd0affd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-03-03T07:57:29.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.854+0000 7fcd0affd700 1 -- 172.21.15.98:0/3441890232 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c055b60 con 0x559ce8f9a5c0 2023-03-03T07:57:29.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.854+0000 7fcd0affd700 1 -- 172.21.15.98:0/3441890232 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fcd1c055cc0 con 0x559ce8f9a5c0 2023-03-03T07:57:29.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.854+0000 7fcd45f16b80 1 -- 172.21.15.98:0/3441890232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcd2400be90 con 0x559ce8f9a5c0 2023-03-03T07:57:29.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.854+0000 7fcd0affd700 1 -- 172.21.15.98:0/3441890232 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd1c05e270 con 0x559ce8f9a5c0 2023-03-03T07:57:29.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.854+0000 7fcd45f16b80 10 client.5430 fetch_fsmap learned FSMap version 14 2023-03-03T07:57:29.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.854+0000 7fcd45f16b80 1 -- 172.21.15.98:0/3441890232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) v3 -- 0x559ce8d04990 con 0x559ce8f9a5c0 2023-03-03T07:57:29.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.854+0000 7fcd0affd700 1 -- 172.21.15.98:0/3441890232 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2400be90 con 0x559ce8f9a5c0 2023-03-03T07:57:29.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.855+0000 7fcd0affd700 1 -- 172.21.15.98:0/3441890232 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 14) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fcd1c06db80 con 0x559ce8f9a5c0 2023-03-03T07:57:29.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.855+0000 7fcd45f16b80 10 client.5430 fetch_fsmap finished waiting for FSMap version 14 2023-03-03T07:57:29.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.855+0000 7fcd45f16b80 10 client.5430 resolve_mds: resolved name 'c' to daemon mds.c 2023-03-03T07:57:29.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.855+0000 7fcd45f16b80 20 client.5430 populate_metadata read hostname 'smithi098' 2023-03-03T07:57:29.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.855+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/3441890232 >> [v2:172.21.15.98:6835/1163251191,v1:172.21.15.98:6837/1163251191] conn(0x559ce8fa2760 0x559ce8fa2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:29.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.855+0000 7fcd45f16b80 4 client.5430 mds_command: new command op to 5227 tid=0[{"prefix": "status"}] 2023-03-03T07:57:29.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.855+0000 7fcd45f16b80 1 -- 172.21.15.98:0/3441890232 --> [v2:172.21.15.98:6835/1163251191,v1:172.21.15.98:6837/1163251191] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559ce8d3cca0 con 0x559ce8fa2760 2023-03-03T07:57:29.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.855+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/3441890232 >> [v2:172.21.15.98:6835/1163251191,v1:172.21.15.98:6837/1163251191] conn(0x559ce8fa2760 0x559ce8fa2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:29.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.855+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/3441890232 >> [v2:172.21.15.98:6835/1163251191,v1:172.21.15.98:6837/1163251191] conn(0x559ce8fa2760 0x559ce8fa2b30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d0ea7bda6147f72 server_cookie=751c1b6ebc2b7b75 in_seq=0 out_seq=0 2023-03-03T07:57:29.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.855+0000 7fcd0affd700 10 client.5430 ms_handle_connect on v2:172.21.15.98:6835/1163251191 2023-03-03T07:57:29.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.856+0000 7fcd0affd700 1 -- 172.21.15.98:0/3441890232 <== mds.0 v2:172.21.15.98:6835/1163251191 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x559ce8d3cca0 con 0x559ce8fa2760 2023-03-03T07:57:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.856+0000 7fcd0affd700 10 client.5430 handle_command_reply: tid=0 2023-03-03T07:57:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.856+0000 7fcd45f16b80 1 client.5430 shutdown 2023-03-03T07:57:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.856+0000 7fcd45f16b80 1 -- 172.21.15.98:0/3441890232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 msgr2=0x559ce903d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.856+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/3441890232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce903d120 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c0565a0 tx=0x7fcd1c004750 comp rx=0 tx=0).stop 2023-03-03T07:57:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.856+0000 7fcd45f16b80 1 -- 172.21.15.98:0/3441890232 shutdown_connections 2023-03-03T07:57:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.856+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/3441890232 >> [v2:172.21.15.98:6835/1163251191,v1:172.21.15.98:6837/1163251191] conn(0x559ce8fa2760 0x559ce8fa2b30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.856+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/3441890232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ce8f9a5c0 0x559ce903d120 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.856+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/3441890232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ce8fb9ad0 0x559ce8fd66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.856+0000 7fcd45f16b80 1 --2- 172.21.15.98:0/3441890232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ce8f9bff0 0x559ce8ff9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.856+0000 7fcd45f16b80 1 -- 172.21.15.98:0/3441890232 >> 172.21.15.98:0/3441890232 conn(0x559ce8f97e10 msgr2=0x559ce8fd7870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:29.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.856+0000 7fcd0affd700 0 client.5430 ms_handle_reset on v2:172.21.15.98:6835/1163251191 2023-03-03T07:57:29.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.856+0000 7fcd45f16b80 1 -- 172.21.15.98:0/3441890232 shutdown_connections 2023-03-03T07:57:29.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.856+0000 7fcd45f16b80 1 -- 172.21.15.98:0/3441890232 wait complete. 2023-03-03T07:57:29.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.857+0000 7fcd45f16b80 20 client.5430 trim_cache size 0 max 16384 2023-03-03T07:57:29.870 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T07:57:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "fbbd322f-236a-44f8-96c5-be6963846c9b", 2023-03-03T07:57:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2023-03-03T07:57:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 5227, 2023-03-03T07:57:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-03-03T07:57:29.871 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-03-03T07:57:29.871 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-03-03T07:57:29.871 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 12.561246105, 2023-03-03T07:57:29.871 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 14, 2023-03-03T07:57:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 40, 2023-03-03T07:57:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 38, 2023-03-03T07:57:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 21.838405350999999 2023-03-03T07:57:29.872 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T07:57:29.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.857+0000 7fcd3139d700 1 -- 172.21.15.98:0/2932835361 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcd180682f0 msgr2=0x7fcd1806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:29.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.857+0000 7fcd3139d700 1 --2- 172.21.15.98:0/2932835361 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcd180682f0 0x7fcd1806a7a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0b70a0 tx=0x7fcd20008040 comp rx=0 tx=0).stop 2023-03-03T07:57:29.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.857+0000 7fcd3139d700 1 -- 172.21.15.98:0/2932835361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c0cf570 msgr2=0x7fcd2c150ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:29.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.857+0000 7fcd3139d700 1 --2- 172.21.15.98:0/2932835361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c0cf570 0x7fcd2c150ea0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c00d800 tx=0x7fcd1c00dbc0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.857+0000 7fcd3139d700 1 -- 172.21.15.98:0/2932835361 shutdown_connections 2023-03-03T07:57:29.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.857+0000 7fcd3139d700 1 --2- 172.21.15.98:0/2932835361 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcd180682f0 0x7fcd1806a7a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.857+0000 7fcd3139d700 1 --2- 172.21.15.98:0/2932835361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c0cf570 0x7fcd2c150ea0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.857+0000 7fcd3139d700 1 --2- 172.21.15.98:0/2932835361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcd2c1609b0 0x7fcd2c15ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.857+0000 7fcd3139d700 1 --2- 172.21.15.98:0/2932835361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcd2c15db30 0x7fcd2c0c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:29.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.857+0000 7fcd3139d700 1 -- 172.21.15.98:0/2932835361 >> 172.21.15.98:0/2932835361 conn(0x7fcd2c06b880 msgr2=0x7fcd2c152380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:29.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.858+0000 7fcd3139d700 1 -- 172.21.15.98:0/2932835361 shutdown_connections 2023-03-03T07:57:29.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:29.858+0000 7fcd3139d700 1 -- 172.21.15.98:0/2932835361 wait complete. 2023-03-03T07:57:29.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T07:57:30.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.131+0000 7f058179d700 1 Processor -- start 2023-03-03T07:57:30.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.131+0000 7f058179d700 1 -- start start 2023-03-03T07:57:30.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.133+0000 7f058179d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f057c146360 0x7f057c160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:30.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.133+0000 7f058179d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f057c0d2590 0x7f057c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:30.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.133+0000 7f058179d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c1614c0 con 0x7f057c0b4ab0 2023-03-03T07:57:30.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.133+0000 7f058179d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f057c0b5440 con 0x7f057c146360 2023-03-03T07:57:30.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.133+0000 7f058179d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057c0d10f0 con 0x7f057c0d2590 2023-03-03T07:57:30.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.133+0000 7f057bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f057c146360 0x7f057c160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:30.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.133+0000 7f057bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f057c146360 0x7f057c160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57728/0 (socket says 172.21.15.98:57728) 2023-03-03T07:57:30.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.133+0000 7f057bfff700 1 -- 172.21.15.98:0/344332189 learned_addr learned my addr 172.21.15.98:0/344332189 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:30.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.134+0000 7f0580f9c700 1 --2- 172.21.15.98:0/344332189 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f057c0d2590 0x7f057c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:30.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.134+0000 7f057bfff700 1 -- 172.21.15.98:0/344332189 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f057c0d2590 msgr2=0x7f057c0b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:30.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.134+0000 7f057bfff700 1 --2- 172.21.15.98:0/344332189 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f057c0d2590 0x7f057c0b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:30.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.134+0000 7f057bfff700 1 -- 172.21.15.98:0/344332189 >> v1:172.21.15.100:6789/0 conn(0x7f057c0b4ab0 legacy=0x7f057c0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:30.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.134+0000 7f057bfff700 1 -- 172.21.15.98:0/344332189 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c0cdf20 con 0x7f057c146360 2023-03-03T07:57:30.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.134+0000 7f057bfff700 1 --2- 172.21.15.98:0/344332189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f057c146360 0x7f057c160f80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f056c0098f0 tx=0x7f056c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a951e628da75fc13 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:30.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.134+0000 7f057b7fe700 1 -- 172.21.15.98:0/344332189 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c00b8a0 con 0x7f057c146360 2023-03-03T07:57:30.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.135+0000 7f057b7fe700 1 -- 172.21.15.98:0/344332189 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c00ba00 con 0x7f057c146360 2023-03-03T07:57:30.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.135+0000 7f057b7fe700 1 -- 172.21.15.98:0/344332189 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c00bd90 con 0x7f057c146360 2023-03-03T07:57:30.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.135+0000 7f058179d700 1 -- 172.21.15.98:0/344332189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f057c146360 msgr2=0x7f057c160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:30.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.135+0000 7f058179d700 1 --2- 172.21.15.98:0/344332189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f057c146360 0x7f057c160f80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f056c0098f0 tx=0x7f056c009c00 comp rx=0 tx=0).stop 2023-03-03T07:57:30.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.135+0000 7f058179d700 1 -- 172.21.15.98:0/344332189 shutdown_connections 2023-03-03T07:57:30.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.135+0000 7f058179d700 1 --2- 172.21.15.98:0/344332189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f057c146360 0x7f057c160f80 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:30.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.135+0000 7f058179d700 1 --2- 172.21.15.98:0/344332189 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f057c0d2590 0x7f057c0b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:30.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.135+0000 7f058179d700 1 -- 172.21.15.98:0/344332189 >> 172.21.15.98:0/344332189 conn(0x7f057c06b8e0 msgr2=0x7f057c0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:30.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.135+0000 7f058179d700 1 -- 172.21.15.98:0/344332189 shutdown_connections 2023-03-03T07:57:30.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.135+0000 7f058179d700 1 -- 172.21.15.98:0/344332189 wait complete. 2023-03-03T07:57:30.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.138+0000 7f058179d700 1 Processor -- start 2023-03-03T07:57:30.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.138+0000 7f058179d700 1 -- start start 2023-03-03T07:57:30.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.140+0000 7f058179d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f057c0b4ab0 0x7f057c0c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:30.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.140+0000 7f058179d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f057c0d2590 0x7f057c15e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:30.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.140+0000 7f058179d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f057c146360 0x7f057c15e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:30.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.140+0000 7f058179d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c15def0 con 0x7f057c0b4ab0 2023-03-03T07:57:30.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.140+0000 7f058179d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c1f3b00 con 0x7f057c0d2590 2023-03-03T07:57:30.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.140+0000 7f058179d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057c1f2810 con 0x7f057c146360 2023-03-03T07:57:30.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.140+0000 7f057bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f057c0b4ab0 0x7f057c0c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:30.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.140+0000 7f057bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f057c0b4ab0 0x7f057c0c6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57736/0 (socket says 172.21.15.98:57736) 2023-03-03T07:57:30.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.140+0000 7f057bfff700 1 -- 172.21.15.98:0/1837946984 learned_addr learned my addr 172.21.15.98:0/1837946984 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:30.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.140+0000 7f0573fff700 1 --2- 172.21.15.98:0/1837946984 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f057c0d2590 0x7f057c15e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:30.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.140+0000 7f0580f9c700 1 --2- 172.21.15.98:0/1837946984 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f057c146360 0x7f057c15e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:30.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.141+0000 7f057bfff700 1 -- 172.21.15.98:0/1837946984 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f057c146360 msgr2=0x7f057c15e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:30.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.141+0000 7f057bfff700 1 --2- 172.21.15.98:0/1837946984 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f057c146360 0x7f057c15e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:30.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.141+0000 7f057bfff700 1 -- 172.21.15.98:0/1837946984 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f057c0d2590 msgr2=0x7f057c15e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.141+0000 7f057bfff700 1 --2- 172.21.15.98:0/1837946984 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f057c0d2590 0x7f057c15e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.141+0000 7f057bfff700 1 -- 172.21.15.98:0/1837946984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c009550 con 0x7f057c0b4ab0 2023-03-03T07:57:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.141+0000 7f057bfff700 1 --2- 172.21.15.98:0/1837946984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f057c0b4ab0 0x7f057c0c6250 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f056c009c40 tx=0x7f056c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.141+0000 7f05797fa700 1 -- 172.21.15.98:0/1837946984 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c004590 con 0x7f057c0b4ab0 2023-03-03T07:57:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.141+0000 7f05797fa700 1 -- 172.21.15.98:0/1837946984 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c01e070 con 0x7f057c0b4ab0 2023-03-03T07:57:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.141+0000 7f05797fa700 1 -- 172.21.15.98:0/1837946984 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c0038e0 con 0x7f057c0b4ab0 2023-03-03T07:57:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.141+0000 7f058179d700 1 -- 172.21.15.98:0/1837946984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057c0cd960 con 0x7f057c0b4ab0 2023-03-03T07:57:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.142+0000 7f058179d700 1 -- 172.21.15.98:0/1837946984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057c0cd680 con 0x7f057c0b4ab0 2023-03-03T07:57:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.143+0000 7f058179d700 1 -- 172.21.15.98:0/1837946984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057c0cdc40 con 0x7f057c0b4ab0 2023-03-03T07:57:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.144+0000 7f05797fa700 1 -- 172.21.15.98:0/1837946984 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f056c00f050 con 0x7f057c0b4ab0 2023-03-03T07:57:30.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.145+0000 7f05797fa700 1 --2- 172.21.15.98:0/1837946984 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f055c0682f0 0x7f055c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:30.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.145+0000 7f05797fa700 1 -- 172.21.15.98:0/1837946984 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f056c020070 con 0x7f057c0b4ab0 2023-03-03T07:57:30.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.145+0000 7f0573fff700 1 --2- 172.21.15.98:0/1837946984 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f055c0682f0 0x7f055c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:30.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.146+0000 7f0573fff700 1 --2- 172.21.15.98:0/1837946984 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f055c0682f0 0x7f055c06a7a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f057c15bb30 tx=0x7f0564009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:30.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.149+0000 7f05797fa700 1 -- 172.21.15.98:0/1837946984 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f056c0539a0 con 0x7f057c0b4ab0 2023-03-03T07:57:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.389+0000 7f058179d700 1 -- 172.21.15.98:0/1837946984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f057c068300 con 0x7f057c0b4ab0 2023-03-03T07:57:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.390+0000 7f05797fa700 1 -- 172.21.15.98:0/1837946984 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+73 (secure 0 0 0) 0x7f056c056fc0 con 0x7f057c0b4ab0 2023-03-03T07:57:30.392 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T07:57:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.393+0000 7f058179d700 1 -- 172.21.15.98:0/1837946984 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f055c0682f0 msgr2=0x7f055c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.393+0000 7f058179d700 1 --2- 172.21.15.98:0/1837946984 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f055c0682f0 0x7f055c06a7a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f057c15bb30 tx=0x7f0564009040 comp rx=0 tx=0).stop 2023-03-03T07:57:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.393+0000 7f058179d700 1 -- 172.21.15.98:0/1837946984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f057c0b4ab0 msgr2=0x7f057c0c6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.393+0000 7f058179d700 1 --2- 172.21.15.98:0/1837946984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f057c0b4ab0 0x7f057c0c6250 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f056c009c40 tx=0x7f056c004750 comp rx=0 tx=0).stop 2023-03-03T07:57:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.393+0000 7f058179d700 1 -- 172.21.15.98:0/1837946984 shutdown_connections 2023-03-03T07:57:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.393+0000 7f058179d700 1 --2- 172.21.15.98:0/1837946984 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f055c0682f0 0x7f055c06a7a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.393+0000 7f058179d700 1 --2- 172.21.15.98:0/1837946984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f057c0b4ab0 0x7f057c0c6250 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.393+0000 7f058179d700 1 --2- 172.21.15.98:0/1837946984 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f057c146360 0x7f057c15e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.393+0000 7f058179d700 1 --2- 172.21.15.98:0/1837946984 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f057c0d2590 0x7f057c15e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.393+0000 7f058179d700 1 -- 172.21.15.98:0/1837946984 >> 172.21.15.98:0/1837946984 conn(0x7f057c06b8e0 msgr2=0x7f057c15fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.393+0000 7f058179d700 1 -- 172.21.15.98:0/1837946984 shutdown_connections 2023-03-03T07:57:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:30.393+0000 7f058179d700 1 -- 172.21.15.98:0/1837946984 wait complete. 2023-03-03T07:57:30.422 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T07:57:30.423 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T07:57:30.424 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2023-03-03T07:57:30.424 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-03-03T07:57:30.425 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-03-03T07:57:30.425 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T07:57:30.426 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-03-03T07:57:30.427 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T07:57:30.427 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T07:57:30.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:30.428 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2023-03-03T07:57:30.497 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T07:57:30.497 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T07:57:30.497 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T07:57:30.498 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:57:30.498 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-03-03T07:57:30.498 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:57:30.498 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:57:30.499 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:57:30.499 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2023-03-03T07:57:30.499 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:57:30.500 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:57:30.500 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2023-03-03T07:57:30.500 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:57:30.501 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2023-03-03T07:57:30.501 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T07:57:30.501 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2023-03-03T07:57:30.502 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T07:57:30.502 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42105sec preferred_lft 42105sec 2023-03-03T07:57:30.502 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2023-03-03T07:57:30.503 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:57:30.503 INFO:teuthology.orchestra.run.smithi098.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T07:57:30.503 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 46:3f:9e:55:37:22 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:57:30.504 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-03-03T07:57:30.504 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:57:30.504 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::1857:c2ff:fe19:346f/64 scope link 2023-03-03T07:57:30.504 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:57:30.505 INFO:teuthology.orchestra.run.smithi098.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T07:57:30.505 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 46:3f:9e:55:37:22 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-03-03T07:57:30.505 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::443f:9eff:fe55:3722/64 scope link 2023-03-03T07:57:30.505 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:57:30.506 INFO:teuthology.orchestra.run.smithi098.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T07:57:30.506 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether d6:19:97:a6:7c:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-03-03T07:57:30.506 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::d419:97ff:fea6:7c83/64 scope link 2023-03-03T07:57:30.507 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:57:30.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:30.509 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T07:57:30.606 INFO:teuthology.orchestra.run.smithi098.stdout:1 2023-03-03T07:57:30.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:30.612 DEBUG:teuthology.orchestra.run.smithi098:> route 2023-03-03T07:57:30.685 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2023-03-03T07:57:30.686 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T07:57:30.686 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T07:57:30.686 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T07:57:30.686 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T07:57:30.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:30.689 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-03-03T07:57:30.690 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-03-03T07:57:30.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T07:57:30.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T07:57:30.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T07:57:30.694 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-03-03T07:57:30.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:30.939 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-03-03T07:57:31.010 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-03-03T07:57:31.010 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-03-03T07:57:31.011 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T07:57:31.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:31.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2023-03-03T07:57:31.117 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-03-03T07:57:31.117 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:57:31.118 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-03-03T07:57:31.186 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-03-03T07:57:31.187 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:57:31.187 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-03-03T07:57:31.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T07:57:31.360 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:57:31.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:57:31.460 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:57:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:57:31.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:31.467 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T07:57:31.536 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-03-03T07:57:31.537 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/) 2023-03-03T07:57:31.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T07:57:31.647 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:57:31.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:57:31.752 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:57:31.756 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:57:31.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:31.757 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T07:57:31.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:2023-03-03T07:57:31.767+0000 7fb6da2675c0 -1 init, newargv = 0x55a6a7e09ff0 newargc=15 2023-03-03T07:57:31.769 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[125617]: starting ceph client 2023-03-03T07:57:31.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:57:31.777 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:57:31.779 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[125617]: starting fuse 2023-03-03T07:57:31.834 INFO:teuthology.orchestra.run.smithi098.stdout:42 2023-03-03T07:57:31.834 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2023-03-03T07:57:31.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T07:57:31.835 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2023-03-03T07:57:31.836 DEBUG:teuthology.orchestra.run.smithi098:> import re 2023-03-03T07:57:31.836 DEBUG:teuthology.orchestra.run.smithi098:> import os 2023-03-03T07:57:31.836 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2023-03-03T07:57:31.837 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:57:31.837 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2023-03-03T07:57:31.837 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-03-03T07:57:31.837 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2023-03-03T07:57:31.838 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-03-03T07:57:31.838 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:57:31.838 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2023-03-03T07:57:31.839 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2023-03-03T07:57:31.839 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2023-03-03T07:57:31.839 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2023-03-03T07:57:31.839 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:57:31.840 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2023-03-03T07:57:31.840 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T07:57:31.840 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T07:57:31.841 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T07:57:31.841 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2023-03-03T07:57:31.841 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2023-03-03T07:57:31.842 DEBUG:teuthology.orchestra.run.smithi098:> return f 2023-03-03T07:57:31.842 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T07:57:31.842 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:57:31.842 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.0")) 2023-03-03T07:57:31.843 DEBUG:teuthology.orchestra.run.smithi098:> ' 2023-03-03T07:57:32.061 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.0.125617.asok 2023-03-03T07:57:32.238 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.125617.asok 2023-03-03T07:57:32.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:32.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.125617.asok status 2023-03-03T07:57:32.471 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T07:57:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2023-03-03T07:57:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T07:57:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T07:57:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "0", 2023-03-03T07:57:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2023-03-03T07:57:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-03-03T07:57:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "125617", 2023-03-03T07:57:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2023-03-03T07:57:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2023-03-03T07:57:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2023-03-03T07:57:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 5442, 2023-03-03T07:57:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2023-03-03T07:57:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2023-03-03T07:57:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2023-03-03T07:57:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5442 2023-03-03T07:57:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T07:57:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T07:57:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-03-03T07:57:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 25310936 2023-03-03T07:57:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T07:57:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T07:57:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-03-03T07:57:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 25310936 2023-03-03T07:57:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:57:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.5442 192.168.0.1:0/25310936", 2023-03-03T07:57:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/25310936", 2023-03-03T07:57:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2023-03-03T07:57:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 14, 2023-03-03T07:57:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 40, 2023-03-03T07:57:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2023-03-03T07:57:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2023-03-03T07:57:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2023-03-03T07:57:32.477 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T07:57:32.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:32.493 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2023-03-03T07:57:32.493 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.0 2023-03-03T07:57:32.561 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2023-03-03T07:57:32.562 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-03-03T07:57:32.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:32.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-03-03T07:57:32.666 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T07:57:32.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.916+0000 7f711a3ff700 1 Processor -- start 2023-03-03T07:57:32.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.917+0000 7f711a3ff700 1 -- start start 2023-03-03T07:57:32.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.918+0000 7f711a3ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f71140b8b90 0x7f71140cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:32.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.918+0000 7f711a3ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71140d0ef0 con 0x7f71140b87c0 2023-03-03T07:57:32.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.918+0000 7f711a3ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71140d0600 con 0x7f71140b5410 2023-03-03T07:57:32.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.918+0000 7f711a3ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71140cd970 con 0x7f71140b8b90 2023-03-03T07:57:32.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.918+0000 7f71137fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f71140b8b90 0x7f71140cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:32.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.918+0000 7f7113fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f71140b87c0 0x7f71140b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47696/0 (socket says 172.21.15.98:47696) 2023-03-03T07:57:32.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.918+0000 7f7113fff700 1 -- 172.21.15.98:0/1502221590 learned_addr learned my addr 172.21.15.98:0/1502221590 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:32.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.919+0000 7f71137fe700 1 -- 172.21.15.98:0/1502221590 >> v1:172.21.15.100:6789/0 conn(0x7f71140b5410 legacy=0x7f71140d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:32.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.919+0000 7f71137fe700 1 -- 172.21.15.98:0/1502221590 >> v1:172.21.15.98:6789/0 conn(0x7f71140b87c0 legacy=0x7f71140b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:32.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.919+0000 7f71137fe700 1 -- 172.21.15.98:0/1502221590 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71140cdc50 con 0x7f71140b8b90 2023-03-03T07:57:32.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.919+0000 7f7112ffd700 1 -- 172.21.15.98:0/1502221590 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667065575 0 0) 0x7f71140d0ef0 con 0x7f71140b87c0 2023-03-03T07:57:32.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.919+0000 7f71137fe700 1 --2- 172.21.15.98:0/1502221590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71140b8b90 0x7f71140cffe0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7108009840 tx=0x7f7108009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9e8f99b352b725 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:32.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.920+0000 7f7112ffd700 1 -- 172.21.15.98:0/1502221590 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7108011830 con 0x7f71140b8b90 2023-03-03T07:57:32.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.920+0000 7f7112ffd700 1 -- 172.21.15.98:0/1502221590 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7108011990 con 0x7f71140b8b90 2023-03-03T07:57:32.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.920+0000 7f7112ffd700 1 -- 172.21.15.98:0/1502221590 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7108011d20 con 0x7f71140b8b90 2023-03-03T07:57:32.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.920+0000 7f711a3ff700 1 -- 172.21.15.98:0/1502221590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71140b8b90 msgr2=0x7f71140cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:32.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.920+0000 7f711a3ff700 1 --2- 172.21.15.98:0/1502221590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71140b8b90 0x7f71140cffe0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7108009840 tx=0x7f7108009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:32.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.920+0000 7f711a3ff700 1 -- 172.21.15.98:0/1502221590 shutdown_connections 2023-03-03T07:57:32.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.920+0000 7f711a3ff700 1 --2- 172.21.15.98:0/1502221590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71140b8b90 0x7f71140cffe0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:32.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.920+0000 7f711a3ff700 1 -- 172.21.15.98:0/1502221590 >> 172.21.15.98:0/1502221590 conn(0x7f711406aee0 msgr2=0x7f711406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:32.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.920+0000 7f711a3ff700 1 -- 172.21.15.98:0/1502221590 shutdown_connections 2023-03-03T07:57:32.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.920+0000 7f711a3ff700 1 -- 172.21.15.98:0/1502221590 wait complete. 2023-03-03T07:57:32.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.923+0000 7f711a3ff700 1 Processor -- start 2023-03-03T07:57:32.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.924+0000 7f711a3ff700 1 -- start start 2023-03-03T07:57:32.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.925+0000 7f711a3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71140b5410 0x7f711415a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:32.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.925+0000 7f711a3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71140b87c0 0x7f71141e6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:32.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.925+0000 7f711a3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71140b8b90 0x7f71141e7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:32.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.925+0000 7f711a3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71141e8000 con 0x7f71140b5410 2023-03-03T07:57:32.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.925+0000 7f711a3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71141e7d20 con 0x7f71140b87c0 2023-03-03T07:57:32.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.925+0000 7f711a3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71140cbee0 con 0x7f71140b8b90 2023-03-03T07:57:32.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.925+0000 7f7113fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71140b5410 0x7f711415a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:32.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.925+0000 7f7113fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71140b5410 0x7f711415a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57752/0 (socket says 172.21.15.98:57752) 2023-03-03T07:57:32.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.925+0000 7f7113fff700 1 -- 172.21.15.98:0/2485806723 learned_addr learned my addr 172.21.15.98:0/2485806723 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:32.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.926+0000 7f71137fe700 1 --2- 172.21.15.98:0/2485806723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71140b87c0 0x7f71141e6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:32.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.926+0000 7f711899c700 1 --2- 172.21.15.98:0/2485806723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71140b8b90 0x7f71141e7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:32.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.926+0000 7f7113fff700 1 -- 172.21.15.98:0/2485806723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71140b8b90 msgr2=0x7f71141e7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:32.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.926+0000 7f7113fff700 1 --2- 172.21.15.98:0/2485806723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71140b8b90 0x7f71141e7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:32.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.926+0000 7f7113fff700 1 -- 172.21.15.98:0/2485806723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71140b87c0 msgr2=0x7f71141e6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:32.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.926+0000 7f7113fff700 1 --2- 172.21.15.98:0/2485806723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71140b87c0 0x7f71141e6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:32.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.926+0000 7f7113fff700 1 -- 172.21.15.98:0/2485806723 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71080094a0 con 0x7f71140b5410 2023-03-03T07:57:32.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.926+0000 7f71137fe700 1 --2- 172.21.15.98:0/2485806723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71140b87c0 0x7f71141e6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:32.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.926+0000 7f7113fff700 1 --2- 172.21.15.98:0/2485806723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71140b5410 0x7f711415a4d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f710400c520 tx=0x7f710400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:32.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.926+0000 7f7110ff9700 1 -- 172.21.15.98:0/2485806723 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7104004c80 con 0x7f71140b5410 2023-03-03T07:57:32.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.926+0000 7f7110ff9700 1 -- 172.21.15.98:0/2485806723 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7104004e00 con 0x7f71140b5410 2023-03-03T07:57:32.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.926+0000 7f7110ff9700 1 -- 172.21.15.98:0/2485806723 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7104007310 con 0x7f71140b5410 2023-03-03T07:57:32.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.926+0000 7f711a3ff700 1 -- 172.21.15.98:0/2485806723 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71140cdf30 con 0x7f71140b5410 2023-03-03T07:57:32.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.927+0000 7f711a3ff700 1 -- 172.21.15.98:0/2485806723 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7114073a80 con 0x7f71140b5410 2023-03-03T07:57:32.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.928+0000 7f711a3ff700 1 -- 172.21.15.98:0/2485806723 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71140ce0c0 con 0x7f71140b5410 2023-03-03T07:57:32.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.929+0000 7f7110ff9700 1 -- 172.21.15.98:0/2485806723 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f710401a020 con 0x7f71140b5410 2023-03-03T07:57:32.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.931+0000 7f7110ff9700 1 --2- 172.21.15.98:0/2485806723 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f71000682f0 0x7f710006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:32.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.931+0000 7f7110ff9700 1 -- 172.21.15.98:0/2485806723 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f71040859a0 con 0x7f71140b5410 2023-03-03T07:57:32.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.932+0000 7f71137fe700 1 --2- 172.21.15.98:0/2485806723 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f71000682f0 0x7f710006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:32.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.932+0000 7f71137fe700 1 --2- 172.21.15.98:0/2485806723 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f71000682f0 0x7f710006a7a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f71140b8fd0 tx=0x7f7108008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:32.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:32.936+0000 7f7110ff9700 1 -- 172.21.15.98:0/2485806723 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f71040520f0 con 0x7f71140b5410 2023-03-03T07:57:33.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:33.176+0000 7f711a3ff700 1 -- 172.21.15.98:0/2485806723 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f71140729b0 con 0x7f71140b5410 2023-03-03T07:57:33.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:33.177+0000 7f7110ff9700 1 -- 172.21.15.98:0/2485806723 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7104055710 con 0x7f71140b5410 2023-03-03T07:57:33.180 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T07:57:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:33.180+0000 7f711a3ff700 1 -- 172.21.15.98:0/2485806723 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f71000682f0 msgr2=0x7f710006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:33.180+0000 7f711a3ff700 1 --2- 172.21.15.98:0/2485806723 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f71000682f0 0x7f710006a7a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f71140b8fd0 tx=0x7f7108008030 comp rx=0 tx=0).stop 2023-03-03T07:57:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:33.180+0000 7f711a3ff700 1 -- 172.21.15.98:0/2485806723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71140b5410 msgr2=0x7f711415a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:33.180+0000 7f711a3ff700 1 --2- 172.21.15.98:0/2485806723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71140b5410 0x7f711415a4d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f710400c520 tx=0x7f710400c8e0 comp rx=0 tx=0).stop 2023-03-03T07:57:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:33.180+0000 7f711a3ff700 1 -- 172.21.15.98:0/2485806723 shutdown_connections 2023-03-03T07:57:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:33.180+0000 7f711a3ff700 1 --2- 172.21.15.98:0/2485806723 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f71000682f0 0x7f710006a7a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:33.180+0000 7f711a3ff700 1 --2- 172.21.15.98:0/2485806723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71140b5410 0x7f711415a4d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:33.180+0000 7f711a3ff700 1 --2- 172.21.15.98:0/2485806723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71140b8b90 0x7f71141e7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:33.180+0000 7f711a3ff700 1 --2- 172.21.15.98:0/2485806723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71140b87c0 0x7f71141e6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:33.180+0000 7f711a3ff700 1 -- 172.21.15.98:0/2485806723 >> 172.21.15.98:0/2485806723 conn(0x7f711406aee0 msgr2=0x7f71140d1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:33.180+0000 7f711a3ff700 1 -- 172.21.15.98:0/2485806723 shutdown_connections 2023-03-03T07:57:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:33.180+0000 7f711a3ff700 1 -- 172.21.15.98:0/2485806723 wait complete. 2023-03-03T07:57:33.208 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T07:57:33.208 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T07:57:33.210 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2023-03-03T07:57:33.211 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-03-03T07:57:33.212 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-03-03T07:57:33.212 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T07:57:33.213 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-03-03T07:57:33.214 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T07:57:33.214 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-03-03T07:57:33.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:33.216 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2023-03-03T07:57:33.290 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T07:57:33.290 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T07:57:33.291 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T07:57:33.291 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:57:33.291 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2023-03-03T07:57:33.291 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:57:33.292 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:57:33.292 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:57:33.292 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2023-03-03T07:57:33.292 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:57:33.292 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:57:33.293 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2023-03-03T07:57:33.293 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:57:33.293 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:57:33.293 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T07:57:33.293 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:57:33.293 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T07:57:33.293 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42107sec preferred_lft 42107sec 2023-03-03T07:57:33.294 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2023-03-03T07:57:33.294 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:57:33.294 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T07:57:33.295 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ee:78:63:4a:ea:32 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:57:33.295 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-03-03T07:57:33.296 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:57:33.296 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::9406:dcff:fe40:a4a4/64 scope link 2023-03-03T07:57:33.296 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:57:33.296 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T07:57:33.296 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:5e:bd:c1:84:4d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-03-03T07:57:33.296 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f45e:bdff:fec1:844d/64 scope link 2023-03-03T07:57:33.297 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:57:33.297 INFO:teuthology.orchestra.run.smithi100.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T07:57:33.297 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ee:78:63:4a:ea:32 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-03-03T07:57:33.297 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec78:63ff:fe4a:ea32/64 scope link 2023-03-03T07:57:33.298 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:57:33.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:33.299 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T07:57:33.399 INFO:teuthology.orchestra.run.smithi100.stdout:1 2023-03-03T07:57:33.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:33.405 DEBUG:teuthology.orchestra.run.smithi100:> route 2023-03-03T07:57:33.477 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2023-03-03T07:57:33.478 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T07:57:33.478 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T07:57:33.478 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T07:57:33.478 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T07:57:33.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:33.479 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2023-03-03T07:57:33.479 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2023-03-03T07:57:33.480 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T07:57:33.481 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T07:57:33.481 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T07:57:33.482 DEBUG:teuthology.orchestra.run.smithi100:> ') 2023-03-03T07:57:33.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:33.721 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2023-03-03T07:57:33.790 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-03-03T07:57:33.790 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-03-03T07:57:33.791 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-03-03T07:57:33.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:33.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2023-03-03T07:57:33.893 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-03-03T07:57:33.894 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:57:33.894 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-03-03T07:57:33.963 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-03-03T07:57:33.963 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:57:33.964 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-03-03T07:57:34.031 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T07:57:34.131 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:57:34.132 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:57:34.229 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:57:34.233 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:57:34.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:34.234 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T07:57:34.303 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-03-03T07:57:34.304 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_mountpoint=/) 2023-03-03T07:57:34.311 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T07:57:34.416 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:57:34.417 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:57:34.517 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:57:34.521 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:57:34.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:34.523 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T07:57:34.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2023-03-03T07:57:34.531+0000 7fb0521855c0 -1 init, newargv = 0x55f966aa6d50 newargc=15 2023-03-03T07:57:34.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[117008]: starting ceph client 2023-03-03T07:57:34.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[117008]: starting fuse 2023-03-03T07:57:34.596 INFO:teuthology.orchestra.run.smithi100.stdout:42 2023-03-03T07:57:34.597 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2023-03-03T07:57:34.597 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T07:57:34.598 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2023-03-03T07:57:34.598 DEBUG:teuthology.orchestra.run.smithi100:> import re 2023-03-03T07:57:34.598 DEBUG:teuthology.orchestra.run.smithi100:> import os 2023-03-03T07:57:34.598 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2023-03-03T07:57:34.599 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:57:34.599 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2023-03-03T07:57:34.600 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-03-03T07:57:34.600 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2023-03-03T07:57:34.600 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-03-03T07:57:34.601 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:57:34.601 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2023-03-03T07:57:34.601 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2023-03-03T07:57:34.602 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2023-03-03T07:57:34.602 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2023-03-03T07:57:34.602 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:57:34.602 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2023-03-03T07:57:34.603 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T07:57:34.603 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T07:57:34.603 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T07:57:34.604 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2023-03-03T07:57:34.604 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2023-03-03T07:57:34.604 DEBUG:teuthology.orchestra.run.smithi100:> return f 2023-03-03T07:57:34.605 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T07:57:34.605 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:57:34.605 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.1")) 2023-03-03T07:57:34.605 DEBUG:teuthology.orchestra.run.smithi100:> ' 2023-03-03T07:57:34.822 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.1.117008.asok 2023-03-03T07:57:35.000 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.117008.asok 2023-03-03T07:57:35.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:35.001 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.117008.asok status 2023-03-03T07:57:35.270 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2023-03-03T07:57:35.270 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2023-03-03T07:57:35.270 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T07:57:35.271 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T07:57:35.271 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "1", 2023-03-03T07:57:35.271 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2023-03-03T07:57:35.271 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-03-03T07:57:35.271 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "117008", 2023-03-03T07:57:35.271 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2023-03-03T07:57:35.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:57:35.272 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2023-03-03T07:57:35.272 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2023-03-03T07:57:35.272 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 5099, 2023-03-03T07:57:35.272 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2023-03-03T07:57:35.272 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2023-03-03T07:57:35.272 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2023-03-03T07:57:35.272 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 5099 2023-03-03T07:57:35.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:57:35.273 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T07:57:35.273 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T07:57:35.273 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2023-03-03T07:57:35.273 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 156595 2023-03-03T07:57:35.273 INFO:teuthology.orchestra.run.smithi100.stdout: } 2023-03-03T07:57:35.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:57:35.273 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T07:57:35.274 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T07:57:35.274 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2023-03-03T07:57:35.274 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 156595 2023-03-03T07:57:35.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:57:35.274 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.5099 192.168.0.1:0/156595", 2023-03-03T07:57:35.274 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.1:0/156595", 2023-03-03T07:57:35.274 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2023-03-03T07:57:35.275 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 14, 2023-03-03T07:57:35.275 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 40, 2023-03-03T07:57:35.275 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2023-03-03T07:57:35.275 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2023-03-03T07:57:35.275 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2023-03-03T07:57:35.275 INFO:teuthology.orchestra.run.smithi100.stdout:} 2023-03-03T07:57:35.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:35.292 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2023-03-03T07:57:35.292 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2023-03-03T07:57:35.361 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2023-03-03T07:57:35.362 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-03-03T07:57:35.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:35.362 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-03-03T07:57:35.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-03-03T07:57:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.723+0000 7f617219d700 1 Processor -- start 2023-03-03T07:57:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.723+0000 7f617219d700 1 -- start start 2023-03-03T07:57:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.724+0000 7f617219d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f616c0d25e0 0x7f616c0cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.724+0000 7f617219d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c160ef0 con 0x7f616c06b750 2023-03-03T07:57:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.724+0000 7f617219d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c161420 con 0x7f616c0b5130 2023-03-03T07:57:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.724+0000 7f617219d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616c161140 con 0x7f616c0d25e0 2023-03-03T07:57:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.725+0000 7f617099a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f616c06b750 0x7f616c0b4a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50110/0 (socket says 172.21.15.98:50110) 2023-03-03T07:57:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.725+0000 7f617099a700 1 -- 172.21.15.98:0/414704596 learned_addr learned my addr 172.21.15.98:0/414704596 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.725+0000 7f617119b700 1 --2- 172.21.15.98:0/414704596 >> v2:172.21.15.100:3300/0 conn(0x7f616c0d25e0 0x7f616c0cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.725+0000 7f6163fff700 1 -- 172.21.15.98:0/414704596 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132489313 0 0) 0x7f616c160ef0 con 0x7f616c06b750 2023-03-03T07:57:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.725+0000 7f6163fff700 1 -- 172.21.15.98:0/414704596 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61580035b0 con 0x7f616c06b750 2023-03-03T07:57:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.725+0000 7f6163fff700 1 -- 172.21.15.98:0/414704596 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851658824 0 0) 0x7f616c161420 con 0x7f616c0b5130 2023-03-03T07:57:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.725+0000 7f617119b700 1 -- 172.21.15.98:0/414704596 >> v1:172.21.15.100:6789/0 conn(0x7f616c0b5130 legacy=0x7f616c0d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.725+0000 7f617119b700 1 -- 172.21.15.98:0/414704596 >> v1:172.21.15.98:6789/0 conn(0x7f616c06b750 legacy=0x7f616c0b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.725+0000 7f617119b700 1 -- 172.21.15.98:0/414704596 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c0cd720 con 0x7f616c0d25e0 2023-03-03T07:57:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.726+0000 7f617119b700 1 --2- 172.21.15.98:0/414704596 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f616c0d25e0 0x7f616c0cf2d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6168009840 tx=0x7f6168009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12c398af83ac463 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.726+0000 7f6163fff700 1 -- 172.21.15.98:0/414704596 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168011830 con 0x7f616c0d25e0 2023-03-03T07:57:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.726+0000 7f6163fff700 1 -- 172.21.15.98:0/414704596 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6168011990 con 0x7f616c0d25e0 2023-03-03T07:57:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.726+0000 7f6163fff700 1 -- 172.21.15.98:0/414704596 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168011d20 con 0x7f616c0d25e0 2023-03-03T07:57:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.727+0000 7f617219d700 1 -- 172.21.15.98:0/414704596 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f616c0d25e0 msgr2=0x7f616c0cf2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.727+0000 7f617219d700 1 --2- 172.21.15.98:0/414704596 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f616c0d25e0 0x7f616c0cf2d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6168009840 tx=0x7f6168009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.727+0000 7f617219d700 1 -- 172.21.15.98:0/414704596 shutdown_connections 2023-03-03T07:57:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.727+0000 7f617219d700 1 --2- 172.21.15.98:0/414704596 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f616c0d25e0 0x7f616c0cf2d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.727+0000 7f617219d700 1 -- 172.21.15.98:0/414704596 >> 172.21.15.98:0/414704596 conn(0x7f616c0d3be0 msgr2=0x7f616c06ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.727+0000 7f617219d700 1 -- 172.21.15.98:0/414704596 shutdown_connections 2023-03-03T07:57:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.727+0000 7f617219d700 1 -- 172.21.15.98:0/414704596 wait complete. 2023-03-03T07:57:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.729+0000 7f617219d700 1 Processor -- start 2023-03-03T07:57:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.729+0000 7f617219d700 1 -- start start 2023-03-03T07:57:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.730+0000 7f617219d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f616c06b750 0x7f616c0ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.730+0000 7f617219d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f616c0b5130 0x7f616c1f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.730+0000 7f617219d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f616c0d25e0 0x7f616c1f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.730+0000 7f617219d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c1f33e0 con 0x7f616c0b5130 2023-03-03T07:57:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.730+0000 7f617219d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0ca650 con 0x7f616c06b750 2023-03-03T07:57:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.730+0000 7f617219d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0ca930 con 0x7f616c0d25e0 2023-03-03T07:57:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.730+0000 7f617099a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f616c0b5130 0x7f616c1f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.730+0000 7f617099a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f616c0b5130 0x7f616c1f2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36948/0 (socket says 172.21.15.98:36948) 2023-03-03T07:57:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.730+0000 7f617099a700 1 -- 172.21.15.98:0/687812963 learned_addr learned my addr 172.21.15.98:0/687812963 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.730+0000 7f617119b700 1 --2- 172.21.15.98:0/687812963 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f616c06b750 0x7f616c0ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.731+0000 7f617199c700 1 --2- 172.21.15.98:0/687812963 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f616c0d25e0 0x7f616c1f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.731+0000 7f617099a700 1 -- 172.21.15.98:0/687812963 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f616c0d25e0 msgr2=0x7f616c1f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.731+0000 7f617099a700 1 --2- 172.21.15.98:0/687812963 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f616c0d25e0 0x7f616c1f2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.731+0000 7f617099a700 1 -- 172.21.15.98:0/687812963 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f616c06b750 msgr2=0x7f616c0ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.731+0000 7f617099a700 1 --2- 172.21.15.98:0/687812963 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f616c06b750 0x7f616c0ca070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.731+0000 7f617099a700 1 -- 172.21.15.98:0/687812963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61680094a0 con 0x7f616c0b5130 2023-03-03T07:57:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.731+0000 7f617099a700 1 --2- 172.21.15.98:0/687812963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f616c0b5130 0x7f616c1f2420 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f615c00c420 tx=0x7f615c00c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.731+0000 7f6161ffb700 1 -- 172.21.15.98:0/687812963 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c005d70 con 0x7f616c0b5130 2023-03-03T07:57:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.731+0000 7f6161ffb700 1 -- 172.21.15.98:0/687812963 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615c0070d0 con 0x7f616c0b5130 2023-03-03T07:57:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.731+0000 7f617219d700 1 -- 172.21.15.98:0/687812963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c1466a0 con 0x7f616c0b5130 2023-03-03T07:57:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.732+0000 7f6161ffb700 1 -- 172.21.15.98:0/687812963 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c007440 con 0x7f616c0b5130 2023-03-03T07:57:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.732+0000 7f617219d700 1 -- 172.21.15.98:0/687812963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c0cda00 con 0x7f616c0b5130 2023-03-03T07:57:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.733+0000 7f617219d700 1 -- 172.21.15.98:0/687812963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616c0d29b0 con 0x7f616c0b5130 2023-03-03T07:57:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.734+0000 7f6161ffb700 1 -- 172.21.15.98:0/687812963 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f615c019030 con 0x7f616c0b5130 2023-03-03T07:57:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.735+0000 7f6161ffb700 1 --2- 172.21.15.98:0/687812963 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f61500682f0 0x7f615006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.735+0000 7f6161ffb700 1 -- 172.21.15.98:0/687812963 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f615c084f20 con 0x7f616c0b5130 2023-03-03T07:57:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.736+0000 7f617119b700 1 --2- 172.21.15.98:0/687812963 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f61500682f0 0x7f615006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.736+0000 7f617119b700 1 --2- 172.21.15.98:0/687812963 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f61500682f0 0x7f615006a7a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6168009fd0 tx=0x7f616801c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.740+0000 7f6161ffb700 1 -- 172.21.15.98:0/687812963 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f615c04b5e0 con 0x7f616c0b5130 2023-03-03T07:57:35.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:35.978+0000 7f617219d700 1 -- 172.21.15.98:0/687812963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f616c068290 con 0x7f616c0b5130 2023-03-03T07:57:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.478+0000 7f6161ffb700 1 -- 172.21.15.98:0/687812963 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v15) v1 ==== 136+0+0 (secure 0 0 0) 0x7f615c0170c0 con 0x7f616c0b5130 2023-03-03T07:57:36.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.481+0000 7f617219d700 1 -- 172.21.15.98:0/687812963 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f61500682f0 msgr2=0x7f615006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:36.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.481+0000 7f617219d700 1 --2- 172.21.15.98:0/687812963 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f61500682f0 0x7f615006a7a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6168009fd0 tx=0x7f616801c040 comp rx=0 tx=0).stop 2023-03-03T07:57:36.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.481+0000 7f617219d700 1 -- 172.21.15.98:0/687812963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f616c0b5130 msgr2=0x7f616c1f2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:36.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.481+0000 7f617219d700 1 --2- 172.21.15.98:0/687812963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f616c0b5130 0x7f616c1f2420 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f615c00c420 tx=0x7f615c00c730 comp rx=0 tx=0).stop 2023-03-03T07:57:36.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.481+0000 7f617219d700 1 -- 172.21.15.98:0/687812963 shutdown_connections 2023-03-03T07:57:36.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.481+0000 7f617219d700 1 --2- 172.21.15.98:0/687812963 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f61500682f0 0x7f615006a7a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:36.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.481+0000 7f617219d700 1 --2- 172.21.15.98:0/687812963 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f616c0d25e0 0x7f616c1f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:36.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.481+0000 7f617219d700 1 --2- 172.21.15.98:0/687812963 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f616c06b750 0x7f616c0ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.481+0000 7f617219d700 1 --2- 172.21.15.98:0/687812963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f616c0b5130 0x7f616c1f2420 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.481+0000 7f617219d700 1 -- 172.21.15.98:0/687812963 >> 172.21.15.98:0/687812963 conn(0x7f616c0d3be0 msgr2=0x7f616c0b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.481+0000 7f617219d700 1 -- 172.21.15.98:0/687812963 shutdown_connections 2023-03-03T07:57:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.481+0000 7f617219d700 1 -- 172.21.15.98:0/687812963 wait complete. 2023-03-03T07:57:36.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-03-03T07:57:36.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.757+0000 7f36e61ff700 1 Processor -- start 2023-03-03T07:57:36.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.757+0000 7f36e61ff700 1 -- start start 2023-03-03T07:57:36.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.758+0000 7f36e61ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36e00b5310 0x7f36e0167850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:36.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.758+0000 7f36e61ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e00cd7d0 con 0x7f36e00b8d10 2023-03-03T07:57:36.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.758+0000 7f36e61ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e00b90e0 con 0x7f36e00b26d0 2023-03-03T07:57:36.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.758+0000 7f36e61ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e00cda20 con 0x7f36e00b5310 2023-03-03T07:57:36.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.759+0000 7f36df7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f36e00b8d10 0x7f36e00cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50114/0 (socket says 172.21.15.98:50114) 2023-03-03T07:57:36.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.759+0000 7f36df7fe700 1 -- 172.21.15.98:0/1774648128 learned_addr learned my addr 172.21.15.98:0/1774648128 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:36.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.759+0000 7f36dffff700 1 --2- 172.21.15.98:0/1774648128 >> v2:172.21.15.98:3300/0 conn(0x7f36e00b5310 0x7f36e0167850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:36.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.759+0000 7f36dffff700 1 -- 172.21.15.98:0/1774648128 >> v1:172.21.15.100:6789/0 conn(0x7f36e00b26d0 legacy=0x7f36e00b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:36.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.759+0000 7f36dffff700 1 -- 172.21.15.98:0/1774648128 >> v1:172.21.15.98:6789/0 conn(0x7f36e00b8d10 legacy=0x7f36e00cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:36.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.759+0000 7f36dffff700 1 -- 172.21.15.98:0/1774648128 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e00cdd00 con 0x7f36e00b5310 2023-03-03T07:57:36.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.759+0000 7f36de7fc700 1 -- 172.21.15.98:0/1774648128 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686987647 0 0) 0x7f36e00cd7d0 con 0x7f36e00b8d10 2023-03-03T07:57:36.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.760+0000 7f36dffff700 1 --2- 172.21.15.98:0/1774648128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36e00b5310 0x7f36e0167850 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009d90 tx=0x7f36d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3dccc367a55b0f0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:36.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.760+0000 7f36de7fc700 1 -- 172.21.15.98:0/1774648128 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d800e070 con 0x7f36e00b5310 2023-03-03T07:57:36.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.760+0000 7f36de7fc700 1 -- 172.21.15.98:0/1774648128 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d800bad0 con 0x7f36e00b5310 2023-03-03T07:57:36.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.760+0000 7f36de7fc700 1 -- 172.21.15.98:0/1774648128 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d800be80 con 0x7f36e00b5310 2023-03-03T07:57:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.760+0000 7f36e61ff700 1 -- 172.21.15.98:0/1774648128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36e00b5310 msgr2=0x7f36e0167850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.760+0000 7f36e61ff700 1 --2- 172.21.15.98:0/1774648128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36e00b5310 0x7f36e0167850 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009d90 tx=0x7f36d800b330 comp rx=0 tx=0).stop 2023-03-03T07:57:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.761+0000 7f36e61ff700 1 -- 172.21.15.98:0/1774648128 shutdown_connections 2023-03-03T07:57:36.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.761+0000 7f36e61ff700 1 --2- 172.21.15.98:0/1774648128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36e00b5310 0x7f36e0167850 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:36.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.761+0000 7f36e61ff700 1 -- 172.21.15.98:0/1774648128 >> 172.21.15.98:0/1774648128 conn(0x7f36e0073e70 msgr2=0x7f36e006ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:36.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.761+0000 7f36e61ff700 1 -- 172.21.15.98:0/1774648128 shutdown_connections 2023-03-03T07:57:36.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.761+0000 7f36e61ff700 1 -- 172.21.15.98:0/1774648128 wait complete. 2023-03-03T07:57:36.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.763+0000 7f36e61ff700 1 Processor -- start 2023-03-03T07:57:36.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.763+0000 7f36e61ff700 1 -- start start 2023-03-03T07:57:36.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.764+0000 7f36e61ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36e00b26d0 0x7f36e0157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:36.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.764+0000 7f36e61ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36e00b5310 0x7f36e00c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:36.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.764+0000 7f36e61ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36e00b8d10 0x7f36e00b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:36.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.764+0000 7f36e61ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e00cd3b0 con 0x7f36e00b8d10 2023-03-03T07:57:36.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.764+0000 7f36e61ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e00cd690 con 0x7f36e00b26d0 2023-03-03T07:57:36.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.764+0000 7f36e61ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e01573d0 con 0x7f36e00b5310 2023-03-03T07:57:36.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.764+0000 7f36dffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36e00b8d10 0x7f36e00b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:36.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.764+0000 7f36dffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36e00b8d10 0x7f36e00b43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36964/0 (socket says 172.21.15.98:36964) 2023-03-03T07:57:36.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.764+0000 7f36dffff700 1 -- 172.21.15.98:0/1304088625 learned_addr learned my addr 172.21.15.98:0/1304088625 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:36.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.765+0000 7f36df7fe700 1 --2- 172.21.15.98:0/1304088625 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36e00b26d0 0x7f36e0157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:36.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.765+0000 7f36deffd700 1 --2- 172.21.15.98:0/1304088625 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36e00b5310 0x7f36e00c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:36.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.765+0000 7f36dffff700 1 -- 172.21.15.98:0/1304088625 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36e00b5310 msgr2=0x7f36e00c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:36.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.765+0000 7f36dffff700 1 --2- 172.21.15.98:0/1304088625 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36e00b5310 0x7f36e00c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:36.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.765+0000 7f36dffff700 1 -- 172.21.15.98:0/1304088625 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36e00b26d0 msgr2=0x7f36e0157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:36.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.765+0000 7f36dffff700 1 --2- 172.21.15.98:0/1304088625 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36e00b26d0 0x7f36e0157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:36.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.765+0000 7f36dffff700 1 -- 172.21.15.98:0/1304088625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d8009a70 con 0x7f36e00b8d10 2023-03-03T07:57:36.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.765+0000 7f36dffff700 1 --2- 172.21.15.98:0/1304088625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36e00b8d10 0x7f36e00b43c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f36d8004000 tx=0x7f36d8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:36.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.765+0000 7f36c7fff700 1 -- 172.21.15.98:0/1304088625 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d8012630 con 0x7f36e00b8d10 2023-03-03T07:57:36.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.766+0000 7f36c7fff700 1 -- 172.21.15.98:0/1304088625 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d8012790 con 0x7f36e00b8d10 2023-03-03T07:57:36.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.766+0000 7f36c7fff700 1 -- 172.21.15.98:0/1304088625 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d8012b40 con 0x7f36e00b8d10 2023-03-03T07:57:36.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.766+0000 7f36e61ff700 1 -- 172.21.15.98:0/1304088625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e00cdfe0 con 0x7f36e00b8d10 2023-03-03T07:57:36.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.766+0000 7f36e61ff700 1 -- 172.21.15.98:0/1304088625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e00d0e80 con 0x7f36e00b8d10 2023-03-03T07:57:36.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.767+0000 7f36e61ff700 1 -- 172.21.15.98:0/1304088625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e00b5fd0 con 0x7f36e00b8d10 2023-03-03T07:57:36.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.768+0000 7f36c7fff700 1 -- 172.21.15.98:0/1304088625 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f36d802b070 con 0x7f36e00b8d10 2023-03-03T07:57:36.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.769+0000 7f36c7fff700 1 --2- 172.21.15.98:0/1304088625 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36cc0682f0 0x7f36cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:36.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.769+0000 7f36c7fff700 1 -- 172.21.15.98:0/1304088625 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f36d8087ae0 con 0x7f36e00b8d10 2023-03-03T07:57:36.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.770+0000 7f36df7fe700 1 --2- 172.21.15.98:0/1304088625 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36cc0682f0 0x7f36cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:36.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.771+0000 7f36df7fe700 1 --2- 172.21.15.98:0/1304088625 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36cc0682f0 0x7f36cc06a7a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f36d0000f60 tx=0x7f36d0007cf0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.774+0000 7f36c7fff700 1 -- 172.21.15.98:0/1304088625 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f36d8054230 con 0x7f36e00b8d10 2023-03-03T07:57:36.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.994+0000 7f36e61ff700 1 -- 172.21.15.98:0/1304088625 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f36e004e6e0 con 0x7f36cc0682f0 2023-03-03T07:57:36.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.996+0000 7f36c7fff700 1 -- 172.21.15.98:0/1304088625 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f36e004e6e0 con 0x7f36cc0682f0 2023-03-03T07:57:36.997 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T07:57:36.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:57:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephfs" 2023-03-03T07:57:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:57:36.998 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T07:57:36.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.998+0000 7f36e61ff700 1 -- 172.21.15.98:0/1304088625 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36cc0682f0 msgr2=0x7f36cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:37.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.998+0000 7f36e61ff700 1 --2- 172.21.15.98:0/1304088625 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36cc0682f0 0x7f36cc06a7a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f36d0000f60 tx=0x7f36d0007cf0 comp rx=0 tx=0).stop 2023-03-03T07:57:37.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.998+0000 7f36e61ff700 1 -- 172.21.15.98:0/1304088625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36e00b8d10 msgr2=0x7f36e00b43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:37.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.998+0000 7f36e61ff700 1 --2- 172.21.15.98:0/1304088625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36e00b8d10 0x7f36e00b43c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f36d8004000 tx=0x7f36d8004ad0 comp rx=0 tx=0).stop 2023-03-03T07:57:37.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.998+0000 7f36e61ff700 1 -- 172.21.15.98:0/1304088625 shutdown_connections 2023-03-03T07:57:37.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.998+0000 7f36e61ff700 1 --2- 172.21.15.98:0/1304088625 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36cc0682f0 0x7f36cc06a7a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:37.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.998+0000 7f36e61ff700 1 --2- 172.21.15.98:0/1304088625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36e00b8d10 0x7f36e00b43c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:37.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.998+0000 7f36e61ff700 1 --2- 172.21.15.98:0/1304088625 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36e00b26d0 0x7f36e0157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:37.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.998+0000 7f36e61ff700 1 --2- 172.21.15.98:0/1304088625 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36e00b5310 0x7f36e00c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:37.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.998+0000 7f36e61ff700 1 -- 172.21.15.98:0/1304088625 >> 172.21.15.98:0/1304088625 conn(0x7f36e0073e70 msgr2=0x7f36e00b7770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:37.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.998+0000 7f36e61ff700 1 -- 172.21.15.98:0/1304088625 shutdown_connections 2023-03-03T07:57:37.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:36.999+0000 7f36e61ff700 1 -- 172.21.15.98:0/1304088625 wait complete. 2023-03-03T07:57:37.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-03-03T07:57:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.277+0000 7fd676dff700 1 Processor -- start 2023-03-03T07:57:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.277+0000 7fd676dff700 1 -- start start 2023-03-03T07:57:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.278+0000 7fd676dff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd67015ca70 0x7fd6700cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.278+0000 7fd676dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd6700d0270 0x7fd6700d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.278+0000 7fd676dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6700cf980 con 0x7fd67006b4b0 2023-03-03T07:57:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.278+0000 7fd676dff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6700cd5b0 con 0x7fd67015ca70 2023-03-03T07:57:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.278+0000 7fd676dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6700d1800 con 0x7fd6700d0270 2023-03-03T07:57:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.279+0000 7fd674b9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd67015ca70 0x7fd6700cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.279+0000 7fd674b9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd67015ca70 0x7fd6700cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36980/0 (socket says 172.21.15.98:36980) 2023-03-03T07:57:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.279+0000 7fd674b9b700 1 -- 172.21.15.98:0/3857316494 learned_addr learned my addr 172.21.15.98:0/3857316494 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.279+0000 7fd66ffff700 1 --1- 172.21.15.98:0/3857316494 >> v1:172.21.15.98:6789/0 conn(0x7fd67006b4b0 0x7fd6700cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50128/0 (socket says 172.21.15.98:50128) 2023-03-03T07:57:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.279+0000 7fd67539c700 1 --2- 172.21.15.98:0/3857316494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd6700d0270 0x7fd6700d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.279+0000 7fd674b9b700 1 -- 172.21.15.98:0/3857316494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd6700d0270 msgr2=0x7fd6700d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.279+0000 7fd674b9b700 1 --2- 172.21.15.98:0/3857316494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd6700d0270 0x7fd6700d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.279+0000 7fd674b9b700 1 -- 172.21.15.98:0/3857316494 >> v1:172.21.15.98:6789/0 conn(0x7fd67006b4b0 legacy=0x7fd6700cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.279+0000 7fd66f7fe700 1 -- 172.21.15.98:0/3857316494 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778782168 0 0) 0x7fd6700cf980 con 0x7fd67006b4b0 2023-03-03T07:57:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.279+0000 7fd674b9b700 1 -- 172.21.15.98:0/3857316494 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6700cd900 con 0x7fd67015ca70 2023-03-03T07:57:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.279+0000 7fd67539c700 1 --2- 172.21.15.98:0/3857316494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd6700d0270 0x7fd6700d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:57:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.280+0000 7fd674b9b700 1 --2- 172.21.15.98:0/3857316494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd67015ca70 0x7fd6700cf440 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd6640098f0 tx=0x7fd664009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1faa75dfe19814e2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.280+0000 7fd66f7fe700 1 -- 172.21.15.98:0/3857316494 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66400b8a0 con 0x7fd67015ca70 2023-03-03T07:57:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.280+0000 7fd66f7fe700 1 -- 172.21.15.98:0/3857316494 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66400ba00 con 0x7fd67015ca70 2023-03-03T07:57:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.280+0000 7fd66f7fe700 1 -- 172.21.15.98:0/3857316494 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66400bd90 con 0x7fd67015ca70 2023-03-03T07:57:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.281+0000 7fd676dff700 1 -- 172.21.15.98:0/3857316494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd67015ca70 msgr2=0x7fd6700cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.281+0000 7fd676dff700 1 --2- 172.21.15.98:0/3857316494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd67015ca70 0x7fd6700cf440 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd6640098f0 tx=0x7fd664009c00 comp rx=0 tx=0).stop 2023-03-03T07:57:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.281+0000 7fd676dff700 1 -- 172.21.15.98:0/3857316494 shutdown_connections 2023-03-03T07:57:37.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.281+0000 7fd676dff700 1 --2- 172.21.15.98:0/3857316494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd67015ca70 0x7fd6700cf440 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:37.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.281+0000 7fd676dff700 1 --2- 172.21.15.98:0/3857316494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd6700d0270 0x7fd6700d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:37.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.281+0000 7fd676dff700 1 -- 172.21.15.98:0/3857316494 >> 172.21.15.98:0/3857316494 conn(0x7fd670073e70 msgr2=0x7fd67006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:37.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.281+0000 7fd676dff700 1 -- 172.21.15.98:0/3857316494 shutdown_connections 2023-03-03T07:57:37.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.281+0000 7fd676dff700 1 -- 172.21.15.98:0/3857316494 wait complete. 2023-03-03T07:57:37.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.283+0000 7fd676dff700 1 Processor -- start 2023-03-03T07:57:37.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.283+0000 7fd676dff700 1 -- start start 2023-03-03T07:57:37.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.284+0000 7fd676dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd67006b4b0 0x7fd67015e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:37.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.284+0000 7fd676dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6700d0270 0x7fd67015e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:37.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.284+0000 7fd676dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd67015ca70 0x7fd6700cc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:37.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.284+0000 7fd676dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6700cd3c0 con 0x7fd6700d0270 2023-03-03T07:57:37.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.284+0000 7fd676dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd670149280 con 0x7fd67015ca70 2023-03-03T07:57:37.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.284+0000 7fd676dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67014a000 con 0x7fd67006b4b0 2023-03-03T07:57:37.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.285+0000 7fd66ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6700d0270 0x7fd67015e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:37.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.285+0000 7fd66ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6700d0270 0x7fd67015e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36996/0 (socket says 172.21.15.98:36996) 2023-03-03T07:57:37.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.285+0000 7fd66ffff700 1 -- 172.21.15.98:0/433158764 learned_addr learned my addr 172.21.15.98:0/433158764 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:37.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.285+0000 7fd674b9b700 1 --2- 172.21.15.98:0/433158764 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd67006b4b0 0x7fd67015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:37.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.285+0000 7fd67539c700 1 --2- 172.21.15.98:0/433158764 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd67015ca70 0x7fd6700cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:37.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.285+0000 7fd66ffff700 1 -- 172.21.15.98:0/433158764 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd67006b4b0 msgr2=0x7fd67015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:37.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.285+0000 7fd66ffff700 1 --2- 172.21.15.98:0/433158764 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd67006b4b0 0x7fd67015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:37.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.285+0000 7fd66ffff700 1 -- 172.21.15.98:0/433158764 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd67015ca70 msgr2=0x7fd6700cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:37.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.285+0000 7fd66ffff700 1 --2- 172.21.15.98:0/433158764 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd67015ca70 0x7fd6700cc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:37.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.285+0000 7fd66ffff700 1 -- 172.21.15.98:0/433158764 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd664009550 con 0x7fd6700d0270 2023-03-03T07:57:37.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.286+0000 7fd66ffff700 1 --2- 172.21.15.98:0/433158764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6700d0270 0x7fd67015e970 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00c520 tx=0x7fd65c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:37.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.286+0000 7fd66d7fa700 1 -- 172.21.15.98:0/433158764 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c004c80 con 0x7fd6700d0270 2023-03-03T07:57:37.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.286+0000 7fd66d7fa700 1 -- 172.21.15.98:0/433158764 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65c004de0 con 0x7fd6700d0270 2023-03-03T07:57:37.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.286+0000 7fd66d7fa700 1 -- 172.21.15.98:0/433158764 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c0072d0 con 0x7fd6700d0270 2023-03-03T07:57:37.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.286+0000 7fd676dff700 1 -- 172.21.15.98:0/433158764 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6700d1aa0 con 0x7fd6700d0270 2023-03-03T07:57:37.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.286+0000 7fd676dff700 1 -- 172.21.15.98:0/433158764 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6700cdbe0 con 0x7fd6700d0270 2023-03-03T07:57:37.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.287+0000 7fd676dff700 1 -- 172.21.15.98:0/433158764 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd67015ce40 con 0x7fd6700d0270 2023-03-03T07:57:37.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.288+0000 7fd66d7fa700 1 -- 172.21.15.98:0/433158764 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd65c01a020 con 0x7fd6700d0270 2023-03-03T07:57:37.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.289+0000 7fd66d7fa700 1 --2- 172.21.15.98:0/433158764 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd6540682f0 0x7fd65406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:37.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.290+0000 7fd66d7fa700 1 -- 172.21.15.98:0/433158764 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fd65c0857c0 con 0x7fd6700d0270 2023-03-03T07:57:37.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.290+0000 7fd674b9b700 1 --2- 172.21.15.98:0/433158764 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd6540682f0 0x7fd65406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:37.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.291+0000 7fd674b9b700 1 --2- 172.21.15.98:0/433158764 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd6540682f0 0x7fd65406a7a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd664009c40 tx=0x7fd664008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:37.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.293+0000 7fd66d7fa700 1 -- 172.21.15.98:0/433158764 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd65c0520b0 con 0x7fd6700d0270 2023-03-03T07:57:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.489+0000 7fd676dff700 1 -- 172.21.15.98:0/433158764 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd67004e730 con 0x7fd6700d0270 2023-03-03T07:57:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.490+0000 7fd66d7fa700 1 -- 172.21.15.98:0/433158764 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd65c0556d0 con 0x7fd6700d0270 2023-03-03T07:57:37.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.493+0000 7fd676dff700 1 -- 172.21.15.98:0/433158764 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd6540682f0 msgr2=0x7fd65406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:37.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.493+0000 7fd676dff700 1 --2- 172.21.15.98:0/433158764 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd6540682f0 0x7fd65406a7a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd664009c40 tx=0x7fd664008030 comp rx=0 tx=0).stop 2023-03-03T07:57:37.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.493+0000 7fd676dff700 1 -- 172.21.15.98:0/433158764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6700d0270 msgr2=0x7fd67015e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.493+0000 7fd676dff700 1 --2- 172.21.15.98:0/433158764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6700d0270 0x7fd67015e970 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00c520 tx=0x7fd65c00c830 comp rx=0 tx=0).stop 2023-03-03T07:57:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.493+0000 7fd676dff700 1 -- 172.21.15.98:0/433158764 shutdown_connections 2023-03-03T07:57:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.493+0000 7fd676dff700 1 --2- 172.21.15.98:0/433158764 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd6540682f0 0x7fd65406a7a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.493+0000 7fd676dff700 1 --2- 172.21.15.98:0/433158764 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd67015ca70 0x7fd6700cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.493+0000 7fd676dff700 1 --2- 172.21.15.98:0/433158764 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd67006b4b0 0x7fd67015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.493+0000 7fd676dff700 1 --2- 172.21.15.98:0/433158764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6700d0270 0x7fd67015e970 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.493+0000 7fd676dff700 1 -- 172.21.15.98:0/433158764 >> 172.21.15.98:0/433158764 conn(0x7fd670073e70 msgr2=0x7fd6700b7010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.493+0000 7fd676dff700 1 -- 172.21.15.98:0/433158764 shutdown_connections 2023-03-03T07:57:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:37.493+0000 7fd676dff700 1 -- 172.21.15.98:0/433158764 wait complete. 2023-03-03T07:57:37.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:37.522 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/group_0000000000532435/subvolume_0000000000240822) 2023-03-03T07:57:37.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:37.637 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2023-03-03T07:57:37.716 INFO:teuthology.orchestra.run.smithi098.stdout:cephfs_data 2023-03-03T07:57:37.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:37.718 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/group_0000000000532435/subvolume_0000000000240822) 2023-03-03T07:57:37.841 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:57:37.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/mnt.0/./volumes/group_0000000000532435/subvolume_0000000000240822/.meta 2023-03-03T07:57:37.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000240822 group_0000000000532435 2023-03-03T07:57:38.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.202+0000 7f7c5959d700 1 Processor -- start 2023-03-03T07:57:38.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.202+0000 7f7c5959d700 1 -- start start 2023-03-03T07:57:38.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.203+0000 7f7c5959d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7c540d25e0 0x7f7c540cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.204+0000 7f7c5959d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c54160ef0 con 0x7f7c5406b750 2023-03-03T07:57:38.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.204+0000 7f7c5959d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c54161420 con 0x7f7c540b5130 2023-03-03T07:57:38.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.204+0000 7f7c5959d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c54161140 con 0x7f7c540d25e0 2023-03-03T07:57:38.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.204+0000 7f7c4bfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7c5406b750 0x7f7c540b4a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50144/0 (socket says 172.21.15.98:50144) 2023-03-03T07:57:38.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.204+0000 7f7c4bfff700 1 -- 172.21.15.98:0/4046776998 learned_addr learned my addr 172.21.15.98:0/4046776998 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:38.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.204+0000 7f7c53fff700 1 --2- 172.21.15.98:0/4046776998 >> v2:172.21.15.100:3300/0 conn(0x7f7c540d25e0 0x7f7c540cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.204+0000 7f7c537fe700 1 -- 172.21.15.98:0/4046776998 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442368521 0 0) 0x7f7c54160ef0 con 0x7f7c5406b750 2023-03-03T07:57:38.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c537fe700 1 -- 172.21.15.98:0/4046776998 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c400035b0 con 0x7f7c5406b750 2023-03-03T07:57:38.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c537fe700 1 -- 172.21.15.98:0/4046776998 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3925984874 0 0) 0x7f7c400035b0 con 0x7f7c5406b750 2023-03-03T07:57:38.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c537fe700 1 -- 172.21.15.98:0/4046776998 >> v2:172.21.15.100:3300/0 conn(0x7f7c540d25e0 msgr2=0x7f7c540cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c537fe700 1 --2- 172.21.15.98:0/4046776998 >> v2:172.21.15.100:3300/0 conn(0x7f7c540d25e0 0x7f7c540cf2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c537fe700 1 -- 172.21.15.98:0/4046776998 >> v1:172.21.15.100:6789/0 conn(0x7f7c540b5130 legacy=0x7f7c540d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c537fe700 1 -- 172.21.15.98:0/4046776998 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c540cd720 con 0x7f7c5406b750 2023-03-03T07:57:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c537fe700 1 -- 172.21.15.98:0/4046776998 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f7c3c001670 con 0x7f7c5406b750 2023-03-03T07:57:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c537fe700 1 -- 172.21.15.98:0/4046776998 >> v1:172.21.15.98:6789/0 conn(0x7f7c5406b750 legacy=0x7f7c540b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c537fe700 1 --2- 172.21.15.98:0/4046776998 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c40003a00 0x7f7c40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c537fe700 1 --2- 172.21.15.98:0/4046776998 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c40004850 0x7f7c40004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c537fe700 1 --2- 172.21.15.98:0/4046776998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c40005250 0x7f7c4000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c537fe700 1 -- 172.21.15.98:0/4046776998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c54161140 con 0x7f7c40005250 2023-03-03T07:57:38.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c537fe700 1 -- 172.21.15.98:0/4046776998 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4000c190 con 0x7f7c40003a00 2023-03-03T07:57:38.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c537fe700 1 -- 172.21.15.98:0/4046776998 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4000c350 con 0x7f7c40004850 2023-03-03T07:57:38.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c53fff700 1 --2- 172.21.15.98:0/4046776998 >> v2:172.21.15.100:3300/0 conn(0x7f7c540d25e0 0x7f7c540cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:38.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.205+0000 7f7c58d9c700 1 --2- 172.21.15.98:0/4046776998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c40005250 0x7f7c4000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.206+0000 7f7c4bfff700 1 --2- 172.21.15.98:0/4046776998 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c40004850 0x7f7c40004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.206+0000 7f7c53fff700 1 --2- 172.21.15.98:0/4046776998 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c40003a00 0x7f7c40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.206+0000 7f7c58d9c700 1 -- 172.21.15.98:0/4046776998 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c40004850 msgr2=0x7f7c40004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:38.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.206+0000 7f7c58d9c700 1 --2- 172.21.15.98:0/4046776998 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c40004850 0x7f7c40004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.206+0000 7f7c58d9c700 1 -- 172.21.15.98:0/4046776998 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c40003a00 msgr2=0x7f7c40003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:38.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.206+0000 7f7c58d9c700 1 --2- 172.21.15.98:0/4046776998 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c40003a00 0x7f7c40003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.206+0000 7f7c58d9c700 1 -- 172.21.15.98:0/4046776998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c400035b0 con 0x7f7c40005250 2023-03-03T07:57:38.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.206+0000 7f7c58d9c700 1 -- 172.21.15.98:0/4046776998 reap_dead start 2023-03-03T07:57:38.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.206+0000 7f7c58d9c700 1 --2- 172.21.15.98:0/4046776998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c40005250 0x7f7c4000ba10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7c40005830 tx=0x7f7c40005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3acc58d590e9e6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:38.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.206+0000 7f7c537fe700 1 -- 172.21.15.98:0/4046776998 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c001e70 con 0x7f7c40005250 2023-03-03T07:57:38.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.207+0000 7f7c537fe700 1 -- 172.21.15.98:0/4046776998 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4c00cdc0 con 0x7f7c40005250 2023-03-03T07:57:38.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.207+0000 7f7c537fe700 1 -- 172.21.15.98:0/4046776998 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c003c40 con 0x7f7c40005250 2023-03-03T07:57:38.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.207+0000 7f7c5959d700 1 -- 172.21.15.98:0/4046776998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c40005250 msgr2=0x7f7c4000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:38.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.207+0000 7f7c5959d700 1 --2- 172.21.15.98:0/4046776998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c40005250 0x7f7c4000ba10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7c40005830 tx=0x7f7c40005880 comp rx=0 tx=0).stop 2023-03-03T07:57:38.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.207+0000 7f7c5959d700 1 -- 172.21.15.98:0/4046776998 shutdown_connections 2023-03-03T07:57:38.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.207+0000 7f7c5959d700 1 --2- 172.21.15.98:0/4046776998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c40005250 0x7f7c4000ba10 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.207+0000 7f7c5959d700 1 -- 172.21.15.98:0/4046776998 >> 172.21.15.98:0/4046776998 conn(0x7f7c540d3be0 msgr2=0x7f7c5406ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:38.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.207+0000 7f7c5959d700 1 -- 172.21.15.98:0/4046776998 shutdown_connections 2023-03-03T07:57:38.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.207+0000 7f7c5959d700 1 -- 172.21.15.98:0/4046776998 wait complete. 2023-03-03T07:57:38.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.209+0000 7f7c5959d700 1 Processor -- start 2023-03-03T07:57:38.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.210+0000 7f7c5959d700 1 -- start start 2023-03-03T07:57:38.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.210+0000 7f7c5959d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c40005250 0x7f7c540ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.211+0000 7f7c5959d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c54158690 0x7f7c540c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.211+0000 7f7c5959d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c540c9220 0x7f7c540cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.211+0000 7f7c5959d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c540c6340 con 0x7f7c540c9220 2023-03-03T07:57:38.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.211+0000 7f7c5959d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c540c84f0 con 0x7f7c54158690 2023-03-03T07:57:38.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.211+0000 7f7c5959d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c540ca650 con 0x7f7c40005250 2023-03-03T07:57:38.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.211+0000 7f7c58d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c540c9220 0x7f7c540cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.211+0000 7f7c53fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c40005250 0x7f7c540ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.211+0000 7f7c58d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c540c9220 0x7f7c540cad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37016/0 (socket says 172.21.15.98:37016) 2023-03-03T07:57:38.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.211+0000 7f7c58d9c700 1 -- 172.21.15.98:0/865045178 learned_addr learned my addr 172.21.15.98:0/865045178 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:38.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.211+0000 7f7c4bfff700 1 --2- 172.21.15.98:0/865045178 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c54158690 0x7f7c540c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.211+0000 7f7c58d9c700 1 -- 172.21.15.98:0/865045178 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c40005250 msgr2=0x7f7c540ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:38.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.211+0000 7f7c58d9c700 1 --2- 172.21.15.98:0/865045178 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c40005250 0x7f7c540ca070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.211+0000 7f7c58d9c700 1 -- 172.21.15.98:0/865045178 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c54158690 msgr2=0x7f7c540c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:38.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.211+0000 7f7c58d9c700 1 --2- 172.21.15.98:0/865045178 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c54158690 0x7f7c540c5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.211+0000 7f7c58d9c700 1 -- 172.21.15.98:0/865045178 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c541466a0 con 0x7f7c540c9220 2023-03-03T07:57:38.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.212+0000 7f7c58d9c700 1 --2- 172.21.15.98:0/865045178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c540c9220 0x7f7c540cad30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7c5408c1e0 tx=0x7f7c4c0015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:38.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.212+0000 7f7c517fa700 1 -- 172.21.15.98:0/865045178 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c01a050 con 0x7f7c540c9220 2023-03-03T07:57:38.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.212+0000 7f7c517fa700 1 -- 172.21.15.98:0/865045178 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4c018030 con 0x7f7c540c9220 2023-03-03T07:57:38.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.212+0000 7f7c5959d700 1 -- 172.21.15.98:0/865045178 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c540cda00 con 0x7f7c540c9220 2023-03-03T07:57:38.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.212+0000 7f7c517fa700 1 -- 172.21.15.98:0/865045178 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c005310 con 0x7f7c540c9220 2023-03-03T07:57:38.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.212+0000 7f7c5959d700 1 -- 172.21.15.98:0/865045178 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c540d29b0 con 0x7f7c540c9220 2023-03-03T07:57:38.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.213+0000 7f7c5959d700 1 -- 172.21.15.98:0/865045178 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c54068290 con 0x7f7c540c9220 2023-03-03T07:57:38.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.214+0000 7f7c517fa700 1 -- 172.21.15.98:0/865045178 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7c4c010050 con 0x7f7c540c9220 2023-03-03T07:57:38.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.216+0000 7f7c517fa700 1 --2- 172.21.15.98:0/865045178 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7c340682f0 0x7f7c3406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.216+0000 7f7c517fa700 1 -- 172.21.15.98:0/865045178 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f7c4c086680 con 0x7f7c540c9220 2023-03-03T07:57:38.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.216+0000 7f7c53fff700 1 --2- 172.21.15.98:0/865045178 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7c340682f0 0x7f7c3406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.217+0000 7f7c53fff700 1 --2- 172.21.15.98:0/865045178 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7c340682f0 0x7f7c3406a7a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009470 tx=0x7f7c44008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:38.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.221+0000 7f7c517fa700 1 -- 172.21.15.98:0/865045178 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7c4c052dd0 con 0x7f7c540c9220 2023-03-03T07:57:38.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.448+0000 7f7c5959d700 1 -- 172.21.15.98:0/865045178 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000240822", "group_name": "group_0000000000532435", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c5406b530 con 0x7f7c340682f0 2023-03-03T07:57:38.455 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:57:38.454+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:57:38.456 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:57:38.454+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:57:38.456 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:57:38.454+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:57:38.456 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:57:38.454+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:57:38.456 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:57:38.454+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:57:38.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.495+0000 7f7c517fa700 1 -- 172.21.15.98:0/865045178 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f7c5406b530 con 0x7f7c340682f0 2023-03-03T07:57:38.496 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/group_0000000000532435/subvolume_0000000000240822 2023-03-03T07:57:38.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.497+0000 7f7c5959d700 1 -- 172.21.15.98:0/865045178 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7c340682f0 msgr2=0x7f7c3406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:38.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.497+0000 7f7c5959d700 1 --2- 172.21.15.98:0/865045178 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7c340682f0 0x7f7c3406a7a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009470 tx=0x7f7c44008040 comp rx=0 tx=0).stop 2023-03-03T07:57:38.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.497+0000 7f7c5959d700 1 -- 172.21.15.98:0/865045178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c540c9220 msgr2=0x7f7c540cad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:38.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.497+0000 7f7c5959d700 1 --2- 172.21.15.98:0/865045178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c540c9220 0x7f7c540cad30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7c5408c1e0 tx=0x7f7c4c0015b0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.497+0000 7f7c5959d700 1 -- 172.21.15.98:0/865045178 shutdown_connections 2023-03-03T07:57:38.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.497+0000 7f7c5959d700 1 --2- 172.21.15.98:0/865045178 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7c340682f0 0x7f7c3406a7a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.498+0000 7f7c5959d700 1 --2- 172.21.15.98:0/865045178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c540c9220 0x7f7c540cad30 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.498+0000 7f7c5959d700 1 --2- 172.21.15.98:0/865045178 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c40005250 0x7f7c540ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.498+0000 7f7c5959d700 1 --2- 172.21.15.98:0/865045178 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c54158690 0x7f7c540c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.498+0000 7f7c5959d700 1 -- 172.21.15.98:0/865045178 >> 172.21.15.98:0/865045178 conn(0x7f7c540d3be0 msgr2=0x7f7c540b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:38.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.498+0000 7f7c5959d700 1 -- 172.21.15.98:0/865045178 shutdown_connections 2023-03-03T07:57:38.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.498+0000 7f7c5959d700 1 -- 172.21.15.98:0/865045178 wait complete. 2023-03-03T07:57:38.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000240822 group_0000000000532435 2023-03-03T07:57:38.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.776+0000 7f68c0fff700 1 Processor -- start 2023-03-03T07:57:38.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.776+0000 7f68c0fff700 1 -- start start 2023-03-03T07:57:38.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.778+0000 7f68c0fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68bc0b8880 0x7f68bc0cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.778+0000 7f68c0fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f68bc0b5720 0x7f68bc15bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.778+0000 7f68c0fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68bc0d0ac0 con 0x7f68bc0b4ba0 2023-03-03T07:57:38.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.778+0000 7f68c0fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68bc0cff70 con 0x7f68bc0b5720 2023-03-03T07:57:38.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.778+0000 7f68c0fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc0d0250 con 0x7f68bc0b8880 2023-03-03T07:57:38.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.778+0000 7f68b3fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f68bc0b4ba0 0x7f68bc0b5010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50160/0 (socket says 172.21.15.98:50160) 2023-03-03T07:57:38.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.778+0000 7f68b3fff700 1 -- 172.21.15.98:0/74529075 learned_addr learned my addr 172.21.15.98:0/74529075 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.778+0000 7f68bad9d700 1 --2- 172.21.15.98:0/74529075 >> v2:172.21.15.100:3300/0 conn(0x7f68bc0b5720 0x7f68bc15bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.778+0000 7f68ba59c700 1 --2- 172.21.15.98:0/74529075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68bc0b8880 0x7f68bc0cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 -- 172.21.15.98:0/74529075 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087445330 0 0) 0x7f68bc0d0ac0 con 0x7f68bc0b4ba0 2023-03-03T07:57:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 -- 172.21.15.98:0/74529075 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a80035b0 con 0x7f68bc0b4ba0 2023-03-03T07:57:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 -- 172.21.15.98:0/74529075 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1491843369 0 0) 0x7f68a80035b0 con 0x7f68bc0b4ba0 2023-03-03T07:57:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 -- 172.21.15.98:0/74529075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68bc0b8880 msgr2=0x7f68bc0cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 --2- 172.21.15.98:0/74529075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68bc0b8880 0x7f68bc0cd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 -- 172.21.15.98:0/74529075 >> v2:172.21.15.100:3300/0 conn(0x7f68bc0b5720 msgr2=0x7f68bc15bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 --2- 172.21.15.98:0/74529075 >> v2:172.21.15.100:3300/0 conn(0x7f68bc0b5720 0x7f68bc15bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 -- 172.21.15.98:0/74529075 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68bc0d0530 con 0x7f68bc0b4ba0 2023-03-03T07:57:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 -- 172.21.15.98:0/74529075 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f68a4001670 con 0x7f68bc0b4ba0 2023-03-03T07:57:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 -- 172.21.15.98:0/74529075 >> v1:172.21.15.98:6789/0 conn(0x7f68bc0b4ba0 legacy=0x7f68bc0b5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 --2- 172.21.15.98:0/74529075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68bc0b8880 0x7f68a8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 --2- 172.21.15.98:0/74529075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68a8003fd0 0x7f68a8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 --2- 172.21.15.98:0/74529075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a8004d90 0x7f68a8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 -- 172.21.15.98:0/74529075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc0cff70 con 0x7f68a8004d90 2023-03-03T07:57:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 -- 172.21.15.98:0/74529075 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc0d0250 con 0x7f68a8003fd0 2023-03-03T07:57:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68b9d9b700 1 -- 172.21.15.98:0/74529075 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a800ba90 con 0x7f68bc0b8880 2023-03-03T07:57:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.779+0000 7f68bad9d700 1 --2- 172.21.15.98:0/74529075 >> v2:172.21.15.100:3300/0 conn(0x7f68bc0b5720 0x7f68bc15bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.780+0000 7f68ba59c700 1 --2- 172.21.15.98:0/74529075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68bc0b8880 0x7f68a8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.780+0000 7f68b3fff700 1 --2- 172.21.15.98:0/74529075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68a8003fd0 0x7f68a8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.780+0000 7f68bad9d700 1 --2- 172.21.15.98:0/74529075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a8004d90 0x7f68a8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.780+0000 7f68ba59c700 1 -- 172.21.15.98:0/74529075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68a8003fd0 msgr2=0x7f68a8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.780+0000 7f68ba59c700 1 --2- 172.21.15.98:0/74529075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68a8003fd0 0x7f68a8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.780+0000 7f68ba59c700 1 -- 172.21.15.98:0/74529075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a8004d90 msgr2=0x7f68a8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.780+0000 7f68ba59c700 1 --2- 172.21.15.98:0/74529075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a8004d90 0x7f68a8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.780+0000 7f68ba59c700 1 -- 172.21.15.98:0/74529075 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a80035b0 con 0x7f68bc0b8880 2023-03-03T07:57:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.780+0000 7f68b3fff700 1 --2- 172.21.15.98:0/74529075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68a8003fd0 0x7f68a8004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.780+0000 7f68bad9d700 1 --2- 172.21.15.98:0/74529075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a8004d90 0x7f68a8005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.780+0000 7f68ba59c700 1 --2- 172.21.15.98:0/74529075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68bc0b8880 0x7f68a8003a00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009b90 tx=0x7f68a8005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd5be6ef43c125c0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.781+0000 7f68b9d9b700 1 -- 172.21.15.98:0/74529075 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac00e6c0 con 0x7f68bc0b8880 2023-03-03T07:57:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.781+0000 7f68b9d9b700 1 -- 172.21.15.98:0/74529075 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ac00e820 con 0x7f68bc0b8880 2023-03-03T07:57:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.781+0000 7f68b9d9b700 1 -- 172.21.15.98:0/74529075 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac00eb90 con 0x7f68bc0b8880 2023-03-03T07:57:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.781+0000 7f68c0fff700 1 -- 172.21.15.98:0/74529075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68bc0b8880 msgr2=0x7f68a8003a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.781+0000 7f68c0fff700 1 --2- 172.21.15.98:0/74529075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68bc0b8880 0x7f68a8003a00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009b90 tx=0x7f68a8005880 comp rx=0 tx=0).stop 2023-03-03T07:57:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.781+0000 7f68bad9d700 1 -- 172.21.15.98:0/74529075 reap_dead start 2023-03-03T07:57:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.781+0000 7f68c0fff700 1 -- 172.21.15.98:0/74529075 shutdown_connections 2023-03-03T07:57:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.781+0000 7f68c0fff700 1 -- 172.21.15.98:0/74529075 >> 172.21.15.98:0/74529075 conn(0x7f68bc06ad80 msgr2=0x7f68bc06b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.781+0000 7f68c0fff700 1 -- 172.21.15.98:0/74529075 shutdown_connections 2023-03-03T07:57:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.782+0000 7f68c0fff700 1 -- 172.21.15.98:0/74529075 wait complete. 2023-03-03T07:57:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.784+0000 7f68c0fff700 1 Processor -- start 2023-03-03T07:57:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.784+0000 7f68c0fff700 1 -- start start 2023-03-03T07:57:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.785+0000 7f68c0fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc0b8880 0x7f68bc157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.785+0000 7f68c0fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68bc158950 0x7f68bc158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.785+0000 7f68c0fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68bc0b2bb0 0x7f68bc0b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.785+0000 7f68c0fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc159300 con 0x7f68bc0b8880 2023-03-03T07:57:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.785+0000 7f68c0fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc157d90 con 0x7f68bc0b2bb0 2023-03-03T07:57:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.785+0000 7f68c0fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc0b3830 con 0x7f68bc158950 2023-03-03T07:57:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.785+0000 7f68ba59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc0b8880 0x7f68bc157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.785+0000 7f68ba59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc0b8880 0x7f68bc157850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37058/0 (socket says 172.21.15.98:37058) 2023-03-03T07:57:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.785+0000 7f68ba59c700 1 -- 172.21.15.98:0/3998329223 learned_addr learned my addr 172.21.15.98:0/3998329223 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.785+0000 7f68b3fff700 1 --2- 172.21.15.98:0/3998329223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68bc158950 0x7f68bc158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.785+0000 7f68bad9d700 1 --2- 172.21.15.98:0/3998329223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68bc0b2bb0 0x7f68bc0b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.786+0000 7f68ba59c700 1 -- 172.21.15.98:0/3998329223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68bc158950 msgr2=0x7f68bc158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.786+0000 7f68ba59c700 1 --2- 172.21.15.98:0/3998329223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68bc158950 0x7f68bc158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.786+0000 7f68ba59c700 1 -- 172.21.15.98:0/3998329223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68bc0b2bb0 msgr2=0x7f68bc0b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.786+0000 7f68ba59c700 1 --2- 172.21.15.98:0/3998329223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68bc0b2bb0 0x7f68bc0b3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.786+0000 7f68ba59c700 1 -- 172.21.15.98:0/3998329223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68bc0b8c50 con 0x7f68bc0b8880 2023-03-03T07:57:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.786+0000 7f68ba59c700 1 --2- 172.21.15.98:0/3998329223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc0b8880 0x7f68bc157850 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f68a800bc20 tx=0x7f68ac0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.786+0000 7f68b2ffd700 1 -- 172.21.15.98:0/3998329223 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac021070 con 0x7f68bc0b8880 2023-03-03T07:57:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.786+0000 7f68b2ffd700 1 -- 172.21.15.98:0/3998329223 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ac004330 con 0x7f68bc0b8880 2023-03-03T07:57:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.786+0000 7f68b2ffd700 1 -- 172.21.15.98:0/3998329223 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac00e6c0 con 0x7f68bc0b8880 2023-03-03T07:57:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.786+0000 7f68c0fff700 1 -- 172.21.15.98:0/3998329223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68bc0d0cf0 con 0x7f68bc0b8880 2023-03-03T07:57:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.786+0000 7f68c0fff700 1 -- 172.21.15.98:0/3998329223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68bc0cde70 con 0x7f68bc0b8880 2023-03-03T07:57:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.788+0000 7f68c0fff700 1 -- 172.21.15.98:0/3998329223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68bc04e6e0 con 0x7f68bc0b8880 2023-03-03T07:57:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.788+0000 7f68b2ffd700 1 -- 172.21.15.98:0/3998329223 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f68ac02e030 con 0x7f68bc0b8880 2023-03-03T07:57:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.790+0000 7f68b2ffd700 1 --2- 172.21.15.98:0/3998329223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f689c0682f0 0x7f689c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.790+0000 7f68b2ffd700 1 -- 172.21.15.98:0/3998329223 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f68ac086f00 con 0x7f68bc0b8880 2023-03-03T07:57:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.790+0000 7f68b3fff700 1 --2- 172.21.15.98:0/3998329223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f689c0682f0 0x7f689c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.791+0000 7f68b3fff700 1 --2- 172.21.15.98:0/3998329223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f689c0682f0 0x7f689c06a7a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f68a4000ff0 tx=0x7f68a4007040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:38.794+0000 7f68b2ffd700 1 -- 172.21.15.98:0/3998329223 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f68ac053700 con 0x7f68bc0b8880 2023-03-03T07:57:39.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:39.019+0000 7f68c0fff700 1 -- 172.21.15.98:0/3998329223 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000240822", "group_name": "group_0000000000532435", "target": ["mon-mgr", ""]}) v1 -- 0x7f68bc0cd530 con 0x7f689c0682f0 2023-03-03T07:57:39.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:39.023+0000 7f68b2ffd700 1 -- 172.21.15.98:0/3998329223 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f68bc0cd530 con 0x7f689c0682f0 2023-03-03T07:57:39.025 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/group_0000000000532435/subvolume_0000000000240822 2023-03-03T07:57:39.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:39.026+0000 7f68c0fff700 1 -- 172.21.15.98:0/3998329223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f689c0682f0 msgr2=0x7f689c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:39.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:39.026+0000 7f68c0fff700 1 --2- 172.21.15.98:0/3998329223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f689c0682f0 0x7f689c06a7a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f68a4000ff0 tx=0x7f68a4007040 comp rx=0 tx=0).stop 2023-03-03T07:57:39.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:39.026+0000 7f68c0fff700 1 -- 172.21.15.98:0/3998329223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc0b8880 msgr2=0x7f68bc157850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:39.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:39.026+0000 7f68c0fff700 1 --2- 172.21.15.98:0/3998329223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc0b8880 0x7f68bc157850 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f68a800bc20 tx=0x7f68ac0047c0 comp rx=0 tx=0).stop 2023-03-03T07:57:39.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:39.026+0000 7f68c0fff700 1 -- 172.21.15.98:0/3998329223 shutdown_connections 2023-03-03T07:57:39.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:39.026+0000 7f68c0fff700 1 --2- 172.21.15.98:0/3998329223 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f689c0682f0 0x7f689c06a7a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:39.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:39.026+0000 7f68c0fff700 1 --2- 172.21.15.98:0/3998329223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68bc0b8880 0x7f68bc157850 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:39.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:39.026+0000 7f68c0fff700 1 --2- 172.21.15.98:0/3998329223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68bc0b2bb0 0x7f68bc0b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:39.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:39.026+0000 7f68c0fff700 1 --2- 172.21.15.98:0/3998329223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68bc158950 0x7f68bc158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:39.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:39.026+0000 7f68c0fff700 1 -- 172.21.15.98:0/3998329223 >> 172.21.15.98:0/3998329223 conn(0x7f68bc06ad80 msgr2=0x7f68bc0b3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:39.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:39.026+0000 7f68c0fff700 1 -- 172.21.15.98:0/3998329223 shutdown_connections 2023-03-03T07:57:39.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:39.026+0000 7f68c0fff700 1 -- 172.21.15.98:0/3998329223 wait complete. 2023-03-03T07:57:39.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:39.055 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/b61019ecfcc6180b777b9642cbba587c.meta) 2023-03-03T07:57:43.038 INFO:teuthology.orchestra.run.smithi098.stdout:[GLOBAL] 2023-03-03T07:57:43.039 INFO:teuthology.orchestra.run.smithi098.stdout:version = 1 2023-03-03T07:57:43.039 INFO:teuthology.orchestra.run.smithi098.stdout:type = subvolume 2023-03-03T07:57:43.039 INFO:teuthology.orchestra.run.smithi098.stdout:path = /volumes/group_0000000000532435/subvolume_0000000000240822 2023-03-03T07:57:43.039 INFO:teuthology.orchestra.run.smithi098.stdout:state = complete 2023-03-03T07:57:43.039 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:43.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000240822 group_0000000000532435 2023-03-03T07:57:43.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.300+0000 7f87f59ff700 1 Processor -- start 2023-03-03T07:57:43.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.300+0000 7f87f59ff700 1 -- start start 2023-03-03T07:57:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.302+0000 7f87f59ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f87f015d010 0x7f87f00b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.302+0000 7f87f59ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f87f00b22c0 0x7f87f00d0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.302+0000 7f87f59ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f00b26e0 con 0x7f87f015cc40 2023-03-03T07:57:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.302+0000 7f87f59ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f01617b0 con 0x7f87f015d010 2023-03-03T07:57:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.302+0000 7f87f59ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f006b900 con 0x7f87f00b22c0 2023-03-03T07:57:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.302+0000 7f87ee7fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f87f015d010 0x7f87f00b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.302+0000 7f87eeffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f87f015cc40 0x7f87f00cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50172/0 (socket says 172.21.15.98:50172) 2023-03-03T07:57:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.302+0000 7f87eeffd700 1 -- 172.21.15.98:0/3974539244 learned_addr learned my addr 172.21.15.98:0/3974539244 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.303+0000 7f87ef7fe700 1 --2- 172.21.15.98:0/3974539244 >> v2:172.21.15.100:3300/0 conn(0x7f87f00b22c0 0x7f87f00d0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.303+0000 7f87ee7fc700 1 -- 172.21.15.98:0/3974539244 >> v2:172.21.15.100:3300/0 conn(0x7f87f00b22c0 msgr2=0x7f87f00d0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.303+0000 7f87ee7fc700 1 --2- 172.21.15.98:0/3974539244 >> v2:172.21.15.100:3300/0 conn(0x7f87f00b22c0 0x7f87f00d0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.303+0000 7f87ee7fc700 1 -- 172.21.15.98:0/3974539244 >> v1:172.21.15.98:6789/0 conn(0x7f87f015cc40 legacy=0x7f87f00cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.303+0000 7f87ee7fc700 1 -- 172.21.15.98:0/3974539244 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f01609f0 con 0x7f87f015d010 2023-03-03T07:57:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.303+0000 7f87ee7fc700 1 --2- 172.21.15.98:0/3974539244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f015d010 0x7f87f00b1d80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f87f00b5d40 tx=0x7f87e4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c0ebb56c62ae04 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.303+0000 7f87edffb700 1 -- 172.21.15.98:0/3974539244 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e4011830 con 0x7f87f015d010 2023-03-03T07:57:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.304+0000 7f87edffb700 1 -- 172.21.15.98:0/3974539244 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e4011990 con 0x7f87f015d010 2023-03-03T07:57:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.304+0000 7f87edffb700 1 -- 172.21.15.98:0/3974539244 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e4011d20 con 0x7f87f015d010 2023-03-03T07:57:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.304+0000 7f87f59ff700 1 -- 172.21.15.98:0/3974539244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f015d010 msgr2=0x7f87f00b1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.304+0000 7f87f59ff700 1 --2- 172.21.15.98:0/3974539244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f015d010 0x7f87f00b1d80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f87f00b5d40 tx=0x7f87e4009b20 comp rx=0 tx=0).stop 2023-03-03T07:57:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.304+0000 7f87f59ff700 1 -- 172.21.15.98:0/3974539244 shutdown_connections 2023-03-03T07:57:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.304+0000 7f87f59ff700 1 --2- 172.21.15.98:0/3974539244 >> v2:172.21.15.100:3300/0 conn(0x7f87f00b22c0 0x7f87f00d0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.304+0000 7f87f59ff700 1 --2- 172.21.15.98:0/3974539244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f015d010 0x7f87f00b1d80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.304+0000 7f87f59ff700 1 -- 172.21.15.98:0/3974539244 >> 172.21.15.98:0/3974539244 conn(0x7f87f0073e10 msgr2=0x7f87f00d31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.304+0000 7f87f59ff700 1 -- 172.21.15.98:0/3974539244 shutdown_connections 2023-03-03T07:57:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.304+0000 7f87f59ff700 1 -- 172.21.15.98:0/3974539244 wait complete. 2023-03-03T07:57:43.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.306+0000 7f87f59ff700 1 Processor -- start 2023-03-03T07:57:43.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.306+0000 7f87f59ff700 1 -- start start 2023-03-03T07:57:43.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.307+0000 7f87f59ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00b22c0 0x7f87f014e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:43.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87f59ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f87f015cc40 0x7f87f00b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:43.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87f59ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87f015d010 0x7f87f00b86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:43.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87f59ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f015eb10 con 0x7f87f00b22c0 2023-03-03T07:57:43.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87f59ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f014e700 con 0x7f87f015d010 2023-03-03T07:57:43.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87f59ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f015de90 con 0x7f87f015cc40 2023-03-03T07:57:43.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87eeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00b22c0 0x7f87f014e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:43.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87eeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00b22c0 0x7f87f014e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37072/0 (socket says 172.21.15.98:37072) 2023-03-03T07:57:43.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87eeffd700 1 -- 172.21.15.98:0/1047662306 learned_addr learned my addr 172.21.15.98:0/1047662306 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:43.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87ee7fc700 1 --2- 172.21.15.98:0/1047662306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f87f015cc40 0x7f87f00b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:43.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87ef7fe700 1 --2- 172.21.15.98:0/1047662306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87f015d010 0x7f87f00b86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87eeffd700 1 -- 172.21.15.98:0/1047662306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f87f015cc40 msgr2=0x7f87f00b81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87eeffd700 1 --2- 172.21.15.98:0/1047662306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f87f015cc40 0x7f87f00b81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87eeffd700 1 -- 172.21.15.98:0/1047662306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87f015d010 msgr2=0x7f87f00b86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87eeffd700 1 --2- 172.21.15.98:0/1047662306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87f015d010 0x7f87f00b86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87eeffd700 1 -- 172.21.15.98:0/1047662306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e40094a0 con 0x7f87f00b22c0 2023-03-03T07:57:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.308+0000 7f87ef7fe700 1 --2- 172.21.15.98:0/1047662306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87f015d010 0x7f87f00b86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.309+0000 7f87eeffd700 1 --2- 172.21.15.98:0/1047662306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00b22c0 0x7f87f014e1c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f87e000c550 tx=0x7f87e000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.309+0000 7f87d7fff700 1 -- 172.21.15.98:0/1047662306 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0005d70 con 0x7f87f00b22c0 2023-03-03T07:57:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.309+0000 7f87d7fff700 1 -- 172.21.15.98:0/1047662306 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e00070d0 con 0x7f87f00b22c0 2023-03-03T07:57:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.309+0000 7f87d7fff700 1 -- 172.21.15.98:0/1047662306 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0007440 con 0x7f87f00b22c0 2023-03-03T07:57:43.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.309+0000 7f87f59ff700 1 -- 172.21.15.98:0/1047662306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f0160fb0 con 0x7f87f00b22c0 2023-03-03T07:57:43.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.309+0000 7f87f59ff700 1 -- 172.21.15.98:0/1047662306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f0160cd0 con 0x7f87f00b22c0 2023-03-03T07:57:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.310+0000 7f87f59ff700 1 -- 172.21.15.98:0/1047662306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f0161290 con 0x7f87f00b22c0 2023-03-03T07:57:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.311+0000 7f87d7fff700 1 -- 172.21.15.98:0/1047662306 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 6) v1 ==== 86455+0+0 (secure 0 0 0) 0x7f87e0019030 con 0x7f87f00b22c0 2023-03-03T07:57:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.313+0000 7f87d7fff700 1 --2- 172.21.15.98:0/1047662306 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f87dc0682b0 0x7f87dc06a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.313+0000 7f87d7fff700 1 -- 172.21.15.98:0/1047662306 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f87e0084f50 con 0x7f87f00b22c0 2023-03-03T07:57:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.313+0000 7f87ee7fc700 1 --2- 172.21.15.98:0/1047662306 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f87dc0682b0 0x7f87dc06a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.314+0000 7f87ee7fc700 1 --2- 172.21.15.98:0/1047662306 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f87dc0682b0 0x7f87dc06a760 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f87f00b5d40 tx=0x7f87e401c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.317+0000 7f87d7fff700 1 -- 172.21.15.98:0/1047662306 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f87e004b5e0 con 0x7f87f00b22c0 2023-03-03T07:57:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.538+0000 7f87f59ff700 1 -- 172.21.15.98:0/1047662306 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000240822", "group_name": "group_0000000000532435", "target": ["mon-mgr", ""]}) v1 -- 0x7f87f0152d70 con 0x7f87dc0682b0 2023-03-03T07:57:43.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.554+0000 7f87d7fff700 1 -- 172.21.15.98:0/1047662306 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f87f0152d70 con 0x7f87dc0682b0 2023-03-03T07:57:43.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.557+0000 7f87f59ff700 1 -- 172.21.15.98:0/1047662306 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f87dc0682b0 msgr2=0x7f87dc06a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:43.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.557+0000 7f87f59ff700 1 --2- 172.21.15.98:0/1047662306 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f87dc0682b0 0x7f87dc06a760 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f87f00b5d40 tx=0x7f87e401c040 comp rx=0 tx=0).stop 2023-03-03T07:57:43.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.557+0000 7f87f59ff700 1 -- 172.21.15.98:0/1047662306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00b22c0 msgr2=0x7f87f014e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:43.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.557+0000 7f87f59ff700 1 --2- 172.21.15.98:0/1047662306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00b22c0 0x7f87f014e1c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f87e000c550 tx=0x7f87e000c860 comp rx=0 tx=0).stop 2023-03-03T07:57:43.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.557+0000 7f87f59ff700 1 -- 172.21.15.98:0/1047662306 shutdown_connections 2023-03-03T07:57:43.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.557+0000 7f87f59ff700 1 --2- 172.21.15.98:0/1047662306 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f87dc0682b0 0x7f87dc06a760 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:43.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.557+0000 7f87f59ff700 1 --2- 172.21.15.98:0/1047662306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00b22c0 0x7f87f014e1c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:43.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.557+0000 7f87f59ff700 1 --2- 172.21.15.98:0/1047662306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87f015d010 0x7f87f00b86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:43.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.557+0000 7f87f59ff700 1 --2- 172.21.15.98:0/1047662306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f87f015cc40 0x7f87f00b81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:43.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.557+0000 7f87f59ff700 1 -- 172.21.15.98:0/1047662306 >> 172.21.15.98:0/1047662306 conn(0x7f87f0073e10 msgr2=0x7f87f00d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:43.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.557+0000 7f87f59ff700 1 -- 172.21.15.98:0/1047662306 shutdown_connections 2023-03-03T07:57:43.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.557+0000 7f87f59ff700 1 -- 172.21.15.98:0/1047662306 wait complete. 2023-03-03T07:57:43.561 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:57:43.559+0000 7f810f411700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:57:43.561 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:57:43.559+0000 7f810f411700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:57:43.561 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:57:43.559+0000 7f810f411700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:57:43.561 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:57:43.559+0000 7f810f411700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:57:43.561 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:57:43.559+0000 7f810f411700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:57:43.581 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:57:43.580+0000 7f810ec10700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:57:43.581 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:57:43.580+0000 7f810ec10700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:57:43.581 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:57:43.580+0000 7f810ec10700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:57:43.581 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:57:43.580+0000 7f810ec10700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:57:43.581 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:57:43.580+0000 7f810ec10700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:57:43.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:43.595 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-03-03T07:57:43.732 INFO:teuthology.orchestra.run.smithi098.stdout:2 2023-03-03T07:57:43.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000532435 2023-03-03T07:57:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.979+0000 7fe1bd9ff700 1 Processor -- start 2023-03-03T07:57:43.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.979+0000 7fe1bd9ff700 1 -- start start 2023-03-03T07:57:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.981+0000 7fe1bd9ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe1b80d0270 0x7fe1b80cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.981+0000 7fe1bd9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1b806b4b0 0x7fe1b80d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.981+0000 7fe1bd9ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b80cf980 con 0x7fe1b815ca70 2023-03-03T07:57:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.981+0000 7fe1bd9ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b80cd5b0 con 0x7fe1b80d0270 2023-03-03T07:57:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.981+0000 7fe1bd9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b80d1800 con 0x7fe1b806b4b0 2023-03-03T07:57:43.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.981+0000 7fe1b77fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1b806b4b0 0x7fe1b80d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:43.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.981+0000 7fe1b67fc700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe1b80d0270 0x7fe1b80cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:43.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.981+0000 7fe1b67fc700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe1b80d0270 0x7fe1b80cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:42134/0 (socket says 172.21.15.98:42134) 2023-03-03T07:57:43.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.981+0000 7fe1b67fc700 1 -- 172.21.15.98:0/4064561545 learned_addr learned my addr 172.21.15.98:0/4064561545 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:43.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.982+0000 7fe1b67fc700 1 -- 172.21.15.98:0/4064561545 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1b806b4b0 msgr2=0x7fe1b80d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.982+0000 7fe1b67fc700 1 --2- 172.21.15.98:0/4064561545 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1b806b4b0 0x7fe1b80d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.982+0000 7fe1b67fc700 1 -- 172.21.15.98:0/4064561545 >> v1:172.21.15.100:6789/0 conn(0x7fe1b815ca70 legacy=0x7fe1b80cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.982+0000 7fe1b67fc700 1 -- 172.21.15.98:0/4064561545 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b80cd900 con 0x7fe1b80d0270 2023-03-03T07:57:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.982+0000 7fe1b67fc700 1 --2- 172.21.15.98:0/4064561545 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1b80d0270 0x7fe1b80cf440 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0098f0 tx=0x7fe1ac009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d9b9558cc676f67 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.982+0000 7fe1b5ffb700 1 -- 172.21.15.98:0/4064561545 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac011830 con 0x7fe1b80d0270 2023-03-03T07:57:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.983+0000 7fe1b5ffb700 1 -- 172.21.15.98:0/4064561545 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ac011990 con 0x7fe1b80d0270 2023-03-03T07:57:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.983+0000 7fe1b5ffb700 1 -- 172.21.15.98:0/4064561545 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac011d20 con 0x7fe1b80d0270 2023-03-03T07:57:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.983+0000 7fe1bd9ff700 1 -- 172.21.15.98:0/4064561545 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1b80d0270 msgr2=0x7fe1b80cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.983+0000 7fe1bd9ff700 1 --2- 172.21.15.98:0/4064561545 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1b80d0270 0x7fe1b80cf440 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0098f0 tx=0x7fe1ac009c00 comp rx=0 tx=0).stop 2023-03-03T07:57:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.983+0000 7fe1bd9ff700 1 -- 172.21.15.98:0/4064561545 shutdown_connections 2023-03-03T07:57:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.983+0000 7fe1bd9ff700 1 --2- 172.21.15.98:0/4064561545 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1b806b4b0 0x7fe1b80d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.983+0000 7fe1bd9ff700 1 --2- 172.21.15.98:0/4064561545 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1b80d0270 0x7fe1b80cf440 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.983+0000 7fe1bd9ff700 1 -- 172.21.15.98:0/4064561545 >> 172.21.15.98:0/4064561545 conn(0x7fe1b8073e70 msgr2=0x7fe1b806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.983+0000 7fe1bd9ff700 1 -- 172.21.15.98:0/4064561545 shutdown_connections 2023-03-03T07:57:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.983+0000 7fe1bd9ff700 1 -- 172.21.15.98:0/4064561545 wait complete. 2023-03-03T07:57:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.985+0000 7fe1bd9ff700 1 Processor -- start 2023-03-03T07:57:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.985+0000 7fe1bd9ff700 1 -- start start 2023-03-03T07:57:43.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.986+0000 7fe1bd9ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b806b4b0 0x7fe1b8151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:43.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.986+0000 7fe1bd9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1b80d0270 0x7fe1b80b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:43.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.987+0000 7fe1bd9ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1b815ca70 0x7fe1b80b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:43.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.987+0000 7fe1bd9ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b80c9730 con 0x7fe1b806b4b0 2023-03-03T07:57:43.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.987+0000 7fe1bd9ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b80c9450 con 0x7fe1b815ca70 2023-03-03T07:57:43.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.987+0000 7fe1bd9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b815d600 con 0x7fe1b80d0270 2023-03-03T07:57:43.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.987+0000 7fe1b6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b806b4b0 0x7fe1b8151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:43.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.987+0000 7fe1b6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b806b4b0 0x7fe1b8151570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37110/0 (socket says 172.21.15.98:37110) 2023-03-03T07:57:43.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.987+0000 7fe1b6ffd700 1 -- 172.21.15.98:0/1029951426 learned_addr learned my addr 172.21.15.98:0/1029951426 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:43.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.987+0000 7fe1b67fc700 1 --2- 172.21.15.98:0/1029951426 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1b80d0270 0x7fe1b80b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:43.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.987+0000 7fe1b77fe700 1 --2- 172.21.15.98:0/1029951426 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1b815ca70 0x7fe1b80b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:43.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.987+0000 7fe1b6ffd700 1 -- 172.21.15.98:0/1029951426 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1b80d0270 msgr2=0x7fe1b80b2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:43.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.987+0000 7fe1b6ffd700 1 --2- 172.21.15.98:0/1029951426 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1b80d0270 0x7fe1b80b2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:43.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.987+0000 7fe1b6ffd700 1 -- 172.21.15.98:0/1029951426 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1b815ca70 msgr2=0x7fe1b80b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:43.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.987+0000 7fe1b6ffd700 1 --2- 172.21.15.98:0/1029951426 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1b815ca70 0x7fe1b80b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:43.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.987+0000 7fe1b6ffd700 1 -- 172.21.15.98:0/1029951426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ac009550 con 0x7fe1b806b4b0 2023-03-03T07:57:43.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.987+0000 7fe1b77fe700 1 --2- 172.21.15.98:0/1029951426 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1b815ca70 0x7fe1b80b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:43.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.988+0000 7fe1b6ffd700 1 --2- 172.21.15.98:0/1029951426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b806b4b0 0x7fe1b8151570 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800c4e0 tx=0x7fe1a800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:43.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.988+0000 7fe19ffff700 1 -- 172.21.15.98:0/1029951426 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a80052d0 con 0x7fe1b806b4b0 2023-03-03T07:57:43.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.988+0000 7fe19ffff700 1 -- 172.21.15.98:0/1029951426 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a8005430 con 0x7fe1b806b4b0 2023-03-03T07:57:43.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.988+0000 7fe19ffff700 1 -- 172.21.15.98:0/1029951426 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a80038c0 con 0x7fe1b806b4b0 2023-03-03T07:57:43.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.988+0000 7fe1bd9ff700 1 -- 172.21.15.98:0/1029951426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b80d1aa0 con 0x7fe1b806b4b0 2023-03-03T07:57:43.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.988+0000 7fe1bd9ff700 1 -- 172.21.15.98:0/1029951426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b80cdbe0 con 0x7fe1b806b4b0 2023-03-03T07:57:43.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.989+0000 7fe1bd9ff700 1 -- 172.21.15.98:0/1029951426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b815ce40 con 0x7fe1b806b4b0 2023-03-03T07:57:43.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.990+0000 7fe19ffff700 1 -- 172.21.15.98:0/1029951426 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 6) v1 ==== 86455+0+0 (secure 0 0 0) 0x7fe1a8019030 con 0x7fe1b806b4b0 2023-03-03T07:57:43.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.992+0000 7fe19ffff700 1 --2- 172.21.15.98:0/1029951426 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe1a40682b0 0x7fe1a406a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:43.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.992+0000 7fe19ffff700 1 -- 172.21.15.98:0/1029951426 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fe1a8085120 con 0x7fe1b806b4b0 2023-03-03T07:57:43.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.992+0000 7fe1b67fc700 1 --2- 172.21.15.98:0/1029951426 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe1a40682b0 0x7fe1a406a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:43.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.993+0000 7fe1b67fc700 1 --2- 172.21.15.98:0/1029951426 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe1a40682b0 0x7fe1a406a760 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0098f0 tx=0x7fe1ac008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:43.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:43.996+0000 7fe19ffff700 1 -- 172.21.15.98:0/1029951426 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe1a8051830 con 0x7fe1b806b4b0 2023-03-03T07:57:44.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.217+0000 7fe1bd9ff700 1 -- 172.21.15.98:0/1029951426 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "group_name": "group_0000000000532435", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1b804e6e0 con 0x7fe1a40682b0 2023-03-03T07:57:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.223+0000 7fe19ffff700 1 -- 172.21.15.98:0/1029951426 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe1b804e6e0 con 0x7fe1a40682b0 2023-03-03T07:57:44.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.225+0000 7fe1bd9ff700 1 -- 172.21.15.98:0/1029951426 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe1a40682b0 msgr2=0x7fe1a406a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:44.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.225+0000 7fe1bd9ff700 1 --2- 172.21.15.98:0/1029951426 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe1a40682b0 0x7fe1a406a760 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0098f0 tx=0x7fe1ac008030 comp rx=0 tx=0).stop 2023-03-03T07:57:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.225+0000 7fe1bd9ff700 1 -- 172.21.15.98:0/1029951426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b806b4b0 msgr2=0x7fe1b8151570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.225+0000 7fe1bd9ff700 1 --2- 172.21.15.98:0/1029951426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b806b4b0 0x7fe1b8151570 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800c4e0 tx=0x7fe1a800c7f0 comp rx=0 tx=0).stop 2023-03-03T07:57:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.225+0000 7fe1bd9ff700 1 -- 172.21.15.98:0/1029951426 shutdown_connections 2023-03-03T07:57:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.225+0000 7fe1bd9ff700 1 --2- 172.21.15.98:0/1029951426 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe1a40682b0 0x7fe1a406a760 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.225+0000 7fe1bd9ff700 1 --2- 172.21.15.98:0/1029951426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b806b4b0 0x7fe1b8151570 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.225+0000 7fe1bd9ff700 1 --2- 172.21.15.98:0/1029951426 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1b815ca70 0x7fe1b80b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.225+0000 7fe1bd9ff700 1 --2- 172.21.15.98:0/1029951426 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1b80d0270 0x7fe1b80b2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.225+0000 7fe1bd9ff700 1 -- 172.21.15.98:0/1029951426 >> 172.21.15.98:0/1029951426 conn(0x7fe1b8073e70 msgr2=0x7fe1b80d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.225+0000 7fe1bd9ff700 1 -- 172.21.15.98:0/1029951426 shutdown_connections 2023-03-03T07:57:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.225+0000 7fe1bd9ff700 1 -- 172.21.15.98:0/1029951426 wait complete. 2023-03-03T07:57:44.254 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-03-03T07:57:44.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T07:57:44.394 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T07:57:44.539 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-03-03T07:57:44.540 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:57:44.540 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:57:44.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:57:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.854+0000 7f99d1dff700 1 Processor -- start 2023-03-03T07:57:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.854+0000 7f99d1dff700 1 -- start start 2023-03-03T07:57:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.855+0000 7f99d1dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99cc0b87c0 0x7f99cc0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.855+0000 7f99d1dff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f99cc0b5410 0x7f99cc0d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.856+0000 7f99d1dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99cc0d0ef0 con 0x7f99cc0b8b90 2023-03-03T07:57:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.856+0000 7f99d1dff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99cc0d0600 con 0x7f99cc0b5410 2023-03-03T07:57:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.856+0000 7f99d1dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc0cd970 con 0x7f99cc0b87c0 2023-03-03T07:57:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.856+0000 7f99cb7fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99cc0b87c0 0x7f99cc0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.856+0000 7f99cb7fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99cc0b87c0 0x7f99cc0cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:52264/0 (socket says 172.21.15.98:52264) 2023-03-03T07:57:44.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.856+0000 7f99cb7fe700 1 -- 172.21.15.98:0/3271191377 learned_addr learned my addr 172.21.15.98:0/3271191377 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:44.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.856+0000 7f99cbfff700 1 --2- 172.21.15.98:0/3271191377 >> v2:172.21.15.100:3300/0 conn(0x7f99cc0b5410 0x7f99cc0d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:44.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.857+0000 7f99cb7fe700 1 -- 172.21.15.98:0/3271191377 >> v2:172.21.15.100:3300/0 conn(0x7f99cc0b5410 msgr2=0x7f99cc0d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:44.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.857+0000 7f99cb7fe700 1 --2- 172.21.15.98:0/3271191377 >> v2:172.21.15.100:3300/0 conn(0x7f99cc0b5410 0x7f99cc0d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:44.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.857+0000 7f99cb7fe700 1 -- 172.21.15.98:0/3271191377 >> v1:172.21.15.100:6789/0 conn(0x7f99cc0b8b90 legacy=0x7f99cc0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:44.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.857+0000 7f99cb7fe700 1 -- 172.21.15.98:0/3271191377 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99cc0cdc50 con 0x7f99cc0b87c0 2023-03-03T07:57:44.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.857+0000 7f99cbfff700 1 --2- 172.21.15.98:0/3271191377 >> v2:172.21.15.100:3300/0 conn(0x7f99cc0b5410 0x7f99cc0d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.857+0000 7f99cb7fe700 1 --2- 172.21.15.98:0/3271191377 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99cc0b87c0 0x7f99cc0cffe0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f99bc0098f0 tx=0x7f99bc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9377205495fcf8bb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.857+0000 7f99ca7fc700 1 -- 172.21.15.98:0/3271191377 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc011830 con 0x7f99cc0b87c0 2023-03-03T07:57:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.857+0000 7f99ca7fc700 1 -- 172.21.15.98:0/3271191377 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99bc011990 con 0x7f99cc0b87c0 2023-03-03T07:57:44.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.857+0000 7f99ca7fc700 1 -- 172.21.15.98:0/3271191377 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc011d20 con 0x7f99cc0b87c0 2023-03-03T07:57:44.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.858+0000 7f99d1dff700 1 -- 172.21.15.98:0/3271191377 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99cc0b87c0 msgr2=0x7f99cc0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:44.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.858+0000 7f99d1dff700 1 --2- 172.21.15.98:0/3271191377 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99cc0b87c0 0x7f99cc0cffe0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f99bc0098f0 tx=0x7f99bc009c00 comp rx=0 tx=0).stop 2023-03-03T07:57:44.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.858+0000 7f99d1dff700 1 -- 172.21.15.98:0/3271191377 shutdown_connections 2023-03-03T07:57:44.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.858+0000 7f99d1dff700 1 --2- 172.21.15.98:0/3271191377 >> v2:172.21.15.100:3300/0 conn(0x7f99cc0b5410 0x7f99cc0d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:44.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.858+0000 7f99d1dff700 1 --2- 172.21.15.98:0/3271191377 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99cc0b87c0 0x7f99cc0cffe0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:44.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.858+0000 7f99d1dff700 1 -- 172.21.15.98:0/3271191377 >> 172.21.15.98:0/3271191377 conn(0x7f99cc06aee0 msgr2=0x7f99cc06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:44.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.858+0000 7f99d1dff700 1 -- 172.21.15.98:0/3271191377 shutdown_connections 2023-03-03T07:57:44.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.858+0000 7f99d1dff700 1 -- 172.21.15.98:0/3271191377 wait complete. 2023-03-03T07:57:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.860+0000 7f99d1dff700 1 Processor -- start 2023-03-03T07:57:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.860+0000 7f99d1dff700 1 -- start start 2023-03-03T07:57:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99d1dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99cc0b5410 0x7f99cc0c7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99d1dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99cc0b87c0 0x7f99cc0c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99d1dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99cc0b8b90 0x7f99cc15e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99d1dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc0c8f20 con 0x7f99cc0b5410 2023-03-03T07:57:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99d1dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc15c850 con 0x7f99cc0b8b90 2023-03-03T07:57:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99d1dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc0c82a0 con 0x7f99cc0b87c0 2023-03-03T07:57:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99cb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99cc0b5410 0x7f99cc0c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99cb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99cc0b5410 0x7f99cc0c7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37124/0 (socket says 172.21.15.98:37124) 2023-03-03T07:57:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99cb7fe700 1 -- 172.21.15.98:0/2770284516 learned_addr learned my addr 172.21.15.98:0/2770284516 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99caffd700 1 --2- 172.21.15.98:0/2770284516 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99cc0b87c0 0x7f99cc0c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99cbfff700 1 --2- 172.21.15.98:0/2770284516 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99cc0b8b90 0x7f99cc15e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99cb7fe700 1 -- 172.21.15.98:0/2770284516 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99cc0b87c0 msgr2=0x7f99cc0c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99cb7fe700 1 --2- 172.21.15.98:0/2770284516 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99cc0b87c0 0x7f99cc0c7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99cb7fe700 1 -- 172.21.15.98:0/2770284516 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99cc0b8b90 msgr2=0x7f99cc15e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99cb7fe700 1 --2- 172.21.15.98:0/2770284516 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99cc0b8b90 0x7f99cc15e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.862+0000 7f99cb7fe700 1 -- 172.21.15.98:0/2770284516 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c4009a70 con 0x7f99cc0b5410 2023-03-03T07:57:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.863+0000 7f99cbfff700 1 --2- 172.21.15.98:0/2770284516 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99cc0b8b90 0x7f99cc15e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.863+0000 7f99cb7fe700 1 --2- 172.21.15.98:0/2770284516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99cc0b5410 0x7f99cc0c7730 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f99bc000c00 tx=0x7f99bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.863+0000 7f99b3fff700 1 -- 172.21.15.98:0/2770284516 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc0043c0 con 0x7f99cc0b5410 2023-03-03T07:57:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.863+0000 7f99b3fff700 1 -- 172.21.15.98:0/2770284516 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99bc01f070 con 0x7f99cc0b5410 2023-03-03T07:57:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.863+0000 7f99b3fff700 1 -- 172.21.15.98:0/2770284516 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc0038c0 con 0x7f99cc0b5410 2023-03-03T07:57:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.863+0000 7f99d1dff700 1 -- 172.21.15.98:0/2770284516 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99bc009550 con 0x7f99cc0b5410 2023-03-03T07:57:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.864+0000 7f99d1dff700 1 -- 172.21.15.98:0/2770284516 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99cc0cdf30 con 0x7f99cc0b5410 2023-03-03T07:57:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.865+0000 7f99d1dff700 1 -- 172.21.15.98:0/2770284516 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99cc0ce0c0 con 0x7f99cc0b5410 2023-03-03T07:57:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.866+0000 7f99b3fff700 1 -- 172.21.15.98:0/2770284516 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f99bc02a020 con 0x7f99cc0b5410 2023-03-03T07:57:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.867+0000 7f99b3fff700 1 --2- 172.21.15.98:0/2770284516 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f99b80683c0 0x7f99b806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.867+0000 7f99b3fff700 1 -- 172.21.15.98:0/2770284516 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f99bc087430 con 0x7f99cc0b5410 2023-03-03T07:57:44.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.867+0000 7f99caffd700 1 --2- 172.21.15.98:0/2770284516 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f99b80683c0 0x7f99b806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:44.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.868+0000 7f99caffd700 1 --2- 172.21.15.98:0/2770284516 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f99b80683c0 0x7f99b806a870 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f99cc0c76c0 tx=0x7f99c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:44.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:44.871+0000 7f99b3fff700 1 -- 172.21.15.98:0/2770284516 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f99bc053ab0 con 0x7f99cc0b5410 2023-03-03T07:57:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.066+0000 7f99d1dff700 1 -- 172.21.15.98:0/2770284516 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99cc0729b0 con 0x7f99cc0b5410 2023-03-03T07:57:45.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.068+0000 7f99b3fff700 1 -- 172.21.15.98:0/2770284516 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+17972 (secure 0 0 0) 0x7f99bc0570d0 con 0x7f99cc0b5410 2023-03-03T07:57:45.069 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:45.070 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":40,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:57:28.259781+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:57:15.545608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-03-03T07:57:16.283131+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.31","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:57:45.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.070+0000 7f99d1dff700 1 -- 172.21.15.98:0/2770284516 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f99b80683c0 msgr2=0x7f99b806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:45.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.070+0000 7f99d1dff700 1 --2- 172.21.15.98:0/2770284516 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f99b80683c0 0x7f99b806a870 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f99cc0c76c0 tx=0x7f99c0009040 comp rx=0 tx=0).stop 2023-03-03T07:57:45.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.070+0000 7f99d1dff700 1 -- 172.21.15.98:0/2770284516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99cc0b5410 msgr2=0x7f99cc0c7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:45.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.070+0000 7f99d1dff700 1 --2- 172.21.15.98:0/2770284516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99cc0b5410 0x7f99cc0c7730 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f99bc000c00 tx=0x7f99bc004750 comp rx=0 tx=0).stop 2023-03-03T07:57:45.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.070+0000 7f99d1dff700 1 -- 172.21.15.98:0/2770284516 shutdown_connections 2023-03-03T07:57:45.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.070+0000 7f99d1dff700 1 --2- 172.21.15.98:0/2770284516 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f99b80683c0 0x7f99b806a870 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.070+0000 7f99d1dff700 1 --2- 172.21.15.98:0/2770284516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99cc0b5410 0x7f99cc0c7730 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.070+0000 7f99d1dff700 1 --2- 172.21.15.98:0/2770284516 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99cc0b8b90 0x7f99cc15e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.070+0000 7f99d1dff700 1 --2- 172.21.15.98:0/2770284516 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99cc0b87c0 0x7f99cc0c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.070+0000 7f99d1dff700 1 -- 172.21.15.98:0/2770284516 >> 172.21.15.98:0/2770284516 conn(0x7f99cc06aee0 msgr2=0x7f99cc0b1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:45.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.071+0000 7f99d1dff700 1 -- 172.21.15.98:0/2770284516 shutdown_connections 2023-03-03T07:57:45.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.071+0000 7f99d1dff700 1 -- 172.21.15.98:0/2770284516 wait complete. 2023-03-03T07:57:45.099 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:57:15.545608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:57:16.283131+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:57:45.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:57:45.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.350+0000 7f53b69ff700 1 Processor -- start 2023-03-03T07:57:45.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.350+0000 7f53b69ff700 1 -- start start 2023-03-03T07:57:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.352+0000 7f53b69ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b015ca70 0x7f53b00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.352+0000 7f53b69ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f53b006b4b0 0x7f53b00cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.352+0000 7f53b69ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b00cf980 con 0x7f53b00d00a0 2023-03-03T07:57:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.352+0000 7f53b69ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b00cd5b0 con 0x7f53b006b4b0 2023-03-03T07:57:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.352+0000 7f53b69ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b00d0510 con 0x7f53b015ca70 2023-03-03T07:57:45.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.352+0000 7f53af7fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f53b006b4b0 0x7f53b00cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:45.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.352+0000 7f53affff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b015ca70 0x7f53b00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:45.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.352+0000 7f53b4f9c700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f53b00d00a0 0x7f53b00b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:43868/0 (socket says 172.21.15.98:43868) 2023-03-03T07:57:45.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.352+0000 7f53b4f9c700 1 -- 172.21.15.98:0/2293640898 learned_addr learned my addr 172.21.15.98:0/2293640898 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:45.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.353+0000 7f53aeffd700 1 -- 172.21.15.98:0/2293640898 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912403264 0 0) 0x7f53b00cf980 con 0x7f53b00d00a0 2023-03-03T07:57:45.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.353+0000 7f53aeffd700 1 -- 172.21.15.98:0/2293640898 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53980035b0 con 0x7f53b00d00a0 2023-03-03T07:57:45.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.353+0000 7f53affff700 1 -- 172.21.15.98:0/2293640898 >> v2:172.21.15.100:3300/0 conn(0x7f53b006b4b0 msgr2=0x7f53b00cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.353+0000 7f53affff700 1 --2- 172.21.15.98:0/2293640898 >> v2:172.21.15.100:3300/0 conn(0x7f53b006b4b0 0x7f53b00cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.353+0000 7f53affff700 1 -- 172.21.15.98:0/2293640898 >> v1:172.21.15.100:6789/0 conn(0x7f53b00d00a0 legacy=0x7f53b00b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.353+0000 7f53affff700 1 -- 172.21.15.98:0/2293640898 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b00d2fd0 con 0x7f53b015ca70 2023-03-03T07:57:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.353+0000 7f53af7fe700 1 --2- 172.21.15.98:0/2293640898 >> v2:172.21.15.100:3300/0 conn(0x7f53b006b4b0 0x7f53b00cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.353+0000 7f53affff700 1 --2- 172.21.15.98:0/2293640898 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b015ca70 0x7f53b00cf440 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009840 tx=0x7f53a0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4e7c4a964a5e63 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.354+0000 7f53aeffd700 1 -- 172.21.15.98:0/2293640898 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a0011830 con 0x7f53b015ca70 2023-03-03T07:57:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.354+0000 7f53aeffd700 1 -- 172.21.15.98:0/2293640898 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a0011990 con 0x7f53b015ca70 2023-03-03T07:57:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.354+0000 7f53aeffd700 1 -- 172.21.15.98:0/2293640898 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a0011d20 con 0x7f53b015ca70 2023-03-03T07:57:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.354+0000 7f53b69ff700 1 -- 172.21.15.98:0/2293640898 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b015ca70 msgr2=0x7f53b00cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:45.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.354+0000 7f53b69ff700 1 --2- 172.21.15.98:0/2293640898 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b015ca70 0x7f53b00cf440 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009840 tx=0x7f53a0009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:45.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.354+0000 7f53b69ff700 1 -- 172.21.15.98:0/2293640898 shutdown_connections 2023-03-03T07:57:45.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.354+0000 7f53b69ff700 1 --2- 172.21.15.98:0/2293640898 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b015ca70 0x7f53b00cf440 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.354+0000 7f53b69ff700 1 --2- 172.21.15.98:0/2293640898 >> v2:172.21.15.100:3300/0 conn(0x7f53b006b4b0 0x7f53b00cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.354+0000 7f53b69ff700 1 -- 172.21.15.98:0/2293640898 >> 172.21.15.98:0/2293640898 conn(0x7f53b0073e70 msgr2=0x7f53b006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:45.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.354+0000 7f53b69ff700 1 -- 172.21.15.98:0/2293640898 shutdown_connections 2023-03-03T07:57:45.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.354+0000 7f53b69ff700 1 -- 172.21.15.98:0/2293640898 wait complete. 2023-03-03T07:57:45.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.356+0000 7f53b69ff700 1 Processor -- start 2023-03-03T07:57:45.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.357+0000 7f53b69ff700 1 -- start start 2023-03-03T07:57:45.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.358+0000 7f53b69ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b006b4b0 0x7f53b0151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:45.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.358+0000 7f53b69ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b00d00a0 0x7f53b0155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:45.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.358+0000 7f53b69ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53b015ca70 0x7f53b00b7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:45.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.358+0000 7f53b69ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b01f22d0 con 0x7f53b006b4b0 2023-03-03T07:57:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.358+0000 7f53b69ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b0156d00 con 0x7f53b015ca70 2023-03-03T07:57:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.358+0000 7f53b69ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b0152260 con 0x7f53b00d00a0 2023-03-03T07:57:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.358+0000 7f53affff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b006b4b0 0x7f53b0151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.358+0000 7f53affff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b006b4b0 0x7f53b0151d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42388/0 (socket says 172.21.15.98:42388) 2023-03-03T07:57:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.358+0000 7f53affff700 1 -- 172.21.15.98:0/3004148186 learned_addr learned my addr 172.21.15.98:0/3004148186 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.358+0000 7f53af7fe700 1 --2- 172.21.15.98:0/3004148186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b00d00a0 0x7f53b0155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.358+0000 7f53b4f9c700 1 --2- 172.21.15.98:0/3004148186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53b015ca70 0x7f53b00b7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.359+0000 7f53affff700 1 -- 172.21.15.98:0/3004148186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b00d00a0 msgr2=0x7f53b0155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.359+0000 7f53affff700 1 --2- 172.21.15.98:0/3004148186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b00d00a0 0x7f53b0155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.359+0000 7f53affff700 1 -- 172.21.15.98:0/3004148186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53b015ca70 msgr2=0x7f53b00b7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.359+0000 7f53affff700 1 --2- 172.21.15.98:0/3004148186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53b015ca70 0x7f53b00b7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.359+0000 7f53affff700 1 -- 172.21.15.98:0/3004148186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a40094a0 con 0x7f53b006b4b0 2023-03-03T07:57:45.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.359+0000 7f53affff700 1 --2- 172.21.15.98:0/3004148186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b006b4b0 0x7f53b0151d20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f53a00090b0 tx=0x7f53a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:45.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.359+0000 7f53acff9700 1 -- 172.21.15.98:0/3004148186 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a0004380 con 0x7f53b006b4b0 2023-03-03T07:57:45.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.359+0000 7f53acff9700 1 -- 172.21.15.98:0/3004148186 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a001f070 con 0x7f53b006b4b0 2023-03-03T07:57:45.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.359+0000 7f53acff9700 1 -- 172.21.15.98:0/3004148186 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a00036a0 con 0x7f53b006b4b0 2023-03-03T07:57:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.359+0000 7f53b69ff700 1 -- 172.21.15.98:0/3004148186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a00094a0 con 0x7f53b006b4b0 2023-03-03T07:57:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.360+0000 7f53b69ff700 1 -- 172.21.15.98:0/3004148186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b00cd900 con 0x7f53b006b4b0 2023-03-03T07:57:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.361+0000 7f53b69ff700 1 -- 172.21.15.98:0/3004148186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b00d1800 con 0x7f53b006b4b0 2023-03-03T07:57:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.362+0000 7f53acff9700 1 -- 172.21.15.98:0/3004148186 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f53a002a020 con 0x7f53b006b4b0 2023-03-03T07:57:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.363+0000 7f53acff9700 1 --2- 172.21.15.98:0/3004148186 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f539c0683c0 0x7f539c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.363+0000 7f53acff9700 1 -- 172.21.15.98:0/3004148186 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f53a0087170 con 0x7f53b006b4b0 2023-03-03T07:57:45.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.363+0000 7f53af7fe700 1 --2- 172.21.15.98:0/3004148186 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f539c0683c0 0x7f539c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:45.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.364+0000 7f53af7fe700 1 --2- 172.21.15.98:0/3004148186 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f539c0683c0 0x7f539c06a870 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f53b01f58b0 tx=0x7f53a4009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:45.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.367+0000 7f53acff9700 1 -- 172.21.15.98:0/3004148186 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f53a0053770 con 0x7f53b006b4b0 2023-03-03T07:57:45.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.559+0000 7f53b69ff700 1 -- 172.21.15.98:0/3004148186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f53b00cdbe0 con 0x7f53b006b4b0 2023-03-03T07:57:45.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.559+0000 7f53acff9700 1 -- 172.21.15.98:0/3004148186 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f53a0056d90 con 0x7f53b006b4b0 2023-03-03T07:57:45.561 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:57:45.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.562+0000 7f53b69ff700 1 -- 172.21.15.98:0/3004148186 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f539c0683c0 msgr2=0x7f539c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:45.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.562+0000 7f53b69ff700 1 --2- 172.21.15.98:0/3004148186 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f539c0683c0 0x7f539c06a870 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f53b01f58b0 tx=0x7f53a4009210 comp rx=0 tx=0).stop 2023-03-03T07:57:45.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.562+0000 7f53b69ff700 1 -- 172.21.15.98:0/3004148186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b006b4b0 msgr2=0x7f53b0151d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:45.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.562+0000 7f53b69ff700 1 --2- 172.21.15.98:0/3004148186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b006b4b0 0x7f53b0151d20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f53a00090b0 tx=0x7f53a0004750 comp rx=0 tx=0).stop 2023-03-03T07:57:45.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.562+0000 7f53b69ff700 1 -- 172.21.15.98:0/3004148186 shutdown_connections 2023-03-03T07:57:45.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.562+0000 7f53b69ff700 1 --2- 172.21.15.98:0/3004148186 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f539c0683c0 0x7f539c06a870 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.562+0000 7f53b69ff700 1 --2- 172.21.15.98:0/3004148186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b006b4b0 0x7f53b0151d20 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.562+0000 7f53b69ff700 1 --2- 172.21.15.98:0/3004148186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53b015ca70 0x7f53b00b7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.562+0000 7f53b69ff700 1 --2- 172.21.15.98:0/3004148186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b00d00a0 0x7f53b0155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.563+0000 7f53b69ff700 1 -- 172.21.15.98:0/3004148186 >> 172.21.15.98:0/3004148186 conn(0x7f53b0073e70 msgr2=0x7f53b01585f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.563+0000 7f53b69ff700 1 -- 172.21.15.98:0/3004148186 shutdown_connections 2023-03-03T07:57:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.563+0000 7f53b69ff700 1 -- 172.21.15.98:0/3004148186 wait complete. 2023-03-03T07:57:45.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:57:45.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.843+0000 7f86875ff700 1 Processor -- start 2023-03-03T07:57:45.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.843+0000 7f86875ff700 1 -- start start 2023-03-03T07:57:45.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.845+0000 7f86875ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f868015ca70 0x7f86800cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:45.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.845+0000 7f86875ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f868006b4b0 0x7f86800cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:45.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.845+0000 7f86875ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f86800d00a0 0x7f86800b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:45.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.845+0000 7f86875ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86800cd5b0 con 0x7f868015ca70 2023-03-03T07:57:45.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.845+0000 7f86875ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86800cf980 con 0x7f86800d00a0 2023-03-03T07:57:45.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.845+0000 7f86875ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86800d0510 con 0x7f868006b4b0 2023-03-03T07:57:45.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.845+0000 7f868539b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f868015ca70 0x7f86800cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:45.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.845+0000 7f868539b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f868015ca70 0x7f86800cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42396/0 (socket says 172.21.15.98:42396) 2023-03-03T07:57:45.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.845+0000 7f868539b700 1 -- 172.21.15.98:0/847633569 learned_addr learned my addr 172.21.15.98:0/847633569 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:45.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.846+0000 7f8685b9c700 1 --2- 172.21.15.98:0/847633569 >> v2:172.21.15.100:3300/0 conn(0x7f86800d00a0 0x7f86800b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:45.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.845+0000 7f8684b9a700 1 --2- 172.21.15.98:0/847633569 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f868006b4b0 0x7f86800cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:45.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.846+0000 7f868539b700 1 -- 172.21.15.98:0/847633569 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f868006b4b0 msgr2=0x7f86800cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:45.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.846+0000 7f868539b700 1 --2- 172.21.15.98:0/847633569 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f868006b4b0 0x7f86800cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.846+0000 7f868539b700 1 -- 172.21.15.98:0/847633569 >> v2:172.21.15.100:3300/0 conn(0x7f86800d00a0 msgr2=0x7f86800b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:45.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.846+0000 7f868539b700 1 --2- 172.21.15.98:0/847633569 >> v2:172.21.15.100:3300/0 conn(0x7f86800d00a0 0x7f86800b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.846+0000 7f868539b700 1 -- 172.21.15.98:0/847633569 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86800d1800 con 0x7f868015ca70 2023-03-03T07:57:45.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.846+0000 7f8685b9c700 1 --2- 172.21.15.98:0/847633569 >> v2:172.21.15.100:3300/0 conn(0x7f86800d00a0 0x7f86800b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:57:45.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.846+0000 7f8684b9a700 1 --2- 172.21.15.98:0/847633569 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f868006b4b0 0x7f86800cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:57:45.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.846+0000 7f868539b700 1 --2- 172.21.15.98:0/847633569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f868015ca70 0x7f86800cf440 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f868008b220 tx=0x7f867c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=517f3275a59810a8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:45.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.846+0000 7f8677fff700 1 -- 172.21.15.98:0/847633569 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c00b860 con 0x7f868015ca70 2023-03-03T07:57:45.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.846+0000 7f8677fff700 1 -- 172.21.15.98:0/847633569 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867c00b9c0 con 0x7f868015ca70 2023-03-03T07:57:45.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.846+0000 7f8677fff700 1 -- 172.21.15.98:0/847633569 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c00bd50 con 0x7f868015ca70 2023-03-03T07:57:45.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.847+0000 7f86875ff700 1 -- 172.21.15.98:0/847633569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f868015ca70 msgr2=0x7f86800cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:45.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.847+0000 7f86875ff700 1 --2- 172.21.15.98:0/847633569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f868015ca70 0x7f86800cf440 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f868008b220 tx=0x7f867c009c00 comp rx=0 tx=0).stop 2023-03-03T07:57:45.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.847+0000 7f86875ff700 1 -- 172.21.15.98:0/847633569 shutdown_connections 2023-03-03T07:57:45.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.847+0000 7f86875ff700 1 --2- 172.21.15.98:0/847633569 >> v2:172.21.15.100:3300/0 conn(0x7f86800d00a0 0x7f86800b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.847+0000 7f86875ff700 1 --2- 172.21.15.98:0/847633569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f868015ca70 0x7f86800cf440 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.847+0000 7f86875ff700 1 --2- 172.21.15.98:0/847633569 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f868006b4b0 0x7f86800cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.847+0000 7f86875ff700 1 -- 172.21.15.98:0/847633569 >> 172.21.15.98:0/847633569 conn(0x7f8680073e70 msgr2=0x7f868006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:45.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.847+0000 7f86875ff700 1 -- 172.21.15.98:0/847633569 shutdown_connections 2023-03-03T07:57:45.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.847+0000 7f86875ff700 1 -- 172.21.15.98:0/847633569 wait complete. 2023-03-03T07:57:45.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.850+0000 7f86875ff700 1 Processor -- start 2023-03-03T07:57:45.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.850+0000 7f86875ff700 1 -- start start 2023-03-03T07:57:45.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.851+0000 7f86875ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f868006b4b0 0x7f86800b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:45.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.851+0000 7f86875ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86800d00a0 0x7f86800cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:45.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.852+0000 7f86875ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f868015ca70 0x7f86801e60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:45.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.852+0000 7f86875ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868015fe50 con 0x7f868006b4b0 2023-03-03T07:57:45.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.852+0000 7f86875ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86801570d0 con 0x7f868015ca70 2023-03-03T07:57:45.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.852+0000 7f86875ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86801564c0 con 0x7f86800d00a0 2023-03-03T07:57:45.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.852+0000 7f868539b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f868006b4b0 0x7f86800b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:45.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.852+0000 7f868539b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f868006b4b0 0x7f86800b3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42398/0 (socket says 172.21.15.98:42398) 2023-03-03T07:57:45.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.852+0000 7f868539b700 1 -- 172.21.15.98:0/1216066200 learned_addr learned my addr 172.21.15.98:0/1216066200 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:45.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.852+0000 7f8684b9a700 1 --2- 172.21.15.98:0/1216066200 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86800d00a0 0x7f86800cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:45.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.852+0000 7f8685b9c700 1 --2- 172.21.15.98:0/1216066200 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f868015ca70 0x7f86801e60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:45.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.852+0000 7f868539b700 1 -- 172.21.15.98:0/1216066200 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86800d00a0 msgr2=0x7f86800cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:45.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.852+0000 7f868539b700 1 --2- 172.21.15.98:0/1216066200 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86800d00a0 0x7f86800cbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.852+0000 7f868539b700 1 -- 172.21.15.98:0/1216066200 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f868015ca70 msgr2=0x7f86801e60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:45.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.852+0000 7f868539b700 1 --2- 172.21.15.98:0/1216066200 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f868015ca70 0x7f86801e60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:45.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.852+0000 7f868539b700 1 -- 172.21.15.98:0/1216066200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c009550 con 0x7f868006b4b0 2023-03-03T07:57:45.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.853+0000 7f868539b700 1 --2- 172.21.15.98:0/1216066200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f868006b4b0 0x7f86800b3b60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f867c014040 tx=0x7f867c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:45.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.853+0000 7f8675ffb700 1 -- 172.21.15.98:0/1216066200 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c0036a0 con 0x7f868006b4b0 2023-03-03T07:57:45.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.853+0000 7f8675ffb700 1 -- 172.21.15.98:0/1216066200 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867c01e070 con 0x7f868006b4b0 2023-03-03T07:57:45.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.853+0000 7f8675ffb700 1 -- 172.21.15.98:0/1216066200 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c003a60 con 0x7f868006b4b0 2023-03-03T07:57:45.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.853+0000 7f86875ff700 1 -- 172.21.15.98:0/1216066200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86800cdbe0 con 0x7f868006b4b0 2023-03-03T07:57:45.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.853+0000 7f86875ff700 1 -- 172.21.15.98:0/1216066200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86800cd900 con 0x7f868006b4b0 2023-03-03T07:57:45.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.854+0000 7f86875ff700 1 -- 172.21.15.98:0/1216066200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86800d1aa0 con 0x7f868006b4b0 2023-03-03T07:57:45.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.855+0000 7f8675ffb700 1 -- 172.21.15.98:0/1216066200 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f867c00f050 con 0x7f868006b4b0 2023-03-03T07:57:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.857+0000 7f8675ffb700 1 --2- 172.21.15.98:0/1216066200 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f86640683c0 0x7f866406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.857+0000 7f8675ffb700 1 -- 172.21.15.98:0/1216066200 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f867c020070 con 0x7f868006b4b0 2023-03-03T07:57:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.857+0000 7f8684b9a700 1 --2- 172.21.15.98:0/1216066200 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f86640683c0 0x7f866406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.857+0000 7f8684b9a700 1 --2- 172.21.15.98:0/1216066200 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f86640683c0 0x7f866406a870 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8680160100 tx=0x7f8670008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:45.861+0000 7f8675ffb700 1 -- 172.21.15.98:0/1216066200 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f867c053a40 con 0x7f868006b4b0 2023-03-03T07:57:46.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.052+0000 7f86875ff700 1 -- 172.21.15.98:0/1216066200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f86800cea60 con 0x7f868006b4b0 2023-03-03T07:57:46.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.053+0000 7f8675ffb700 1 -- 172.21.15.98:0/1216066200 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v40) v1 ==== 92+0+10 (secure 0 0 0) 0x7f867c057060 con 0x7f868006b4b0 2023-03-03T07:57:46.054 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:57:46.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.055+0000 7f86875ff700 1 -- 172.21.15.98:0/1216066200 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f86640683c0 msgr2=0x7f866406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:46.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.055+0000 7f86875ff700 1 --2- 172.21.15.98:0/1216066200 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f86640683c0 0x7f866406a870 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8680160100 tx=0x7f8670008040 comp rx=0 tx=0).stop 2023-03-03T07:57:46.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.055+0000 7f86875ff700 1 -- 172.21.15.98:0/1216066200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f868006b4b0 msgr2=0x7f86800b3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:46.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.055+0000 7f86875ff700 1 --2- 172.21.15.98:0/1216066200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f868006b4b0 0x7f86800b3b60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f867c014040 tx=0x7f867c004750 comp rx=0 tx=0).stop 2023-03-03T07:57:46.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.055+0000 7f86875ff700 1 -- 172.21.15.98:0/1216066200 shutdown_connections 2023-03-03T07:57:46.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.055+0000 7f86875ff700 1 --2- 172.21.15.98:0/1216066200 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f86640683c0 0x7f866406a870 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.055+0000 7f86875ff700 1 --2- 172.21.15.98:0/1216066200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f868006b4b0 0x7f86800b3b60 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.055+0000 7f86875ff700 1 --2- 172.21.15.98:0/1216066200 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f868015ca70 0x7f86801e60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.055+0000 7f86875ff700 1 --2- 172.21.15.98:0/1216066200 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86800d00a0 0x7f86800cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.055+0000 7f86875ff700 1 -- 172.21.15.98:0/1216066200 >> 172.21.15.98:0/1216066200 conn(0x7f8680073e70 msgr2=0x7f86800b7430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:46.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.056+0000 7f86875ff700 1 -- 172.21.15.98:0/1216066200 shutdown_connections 2023-03-03T07:57:46.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.056+0000 7f86875ff700 1 -- 172.21.15.98:0/1216066200 wait complete. 2023-03-03T07:57:46.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:57:46.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.333+0000 7ff81239d700 1 Processor -- start 2023-03-03T07:57:46.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.334+0000 7ff81239d700 1 -- start start 2023-03-03T07:57:46.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.335+0000 7ff81239d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff80c1609b0 0x7ff80c160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:46.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.335+0000 7ff81239d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff80c0b4aa0 0x7ff80c0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:46.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.335+0000 7ff81239d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80c1612c0 con 0x7ff80c0cf570 2023-03-03T07:57:46.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.336+0000 7ff81239d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff80c0cebb0 con 0x7ff80c1609b0 2023-03-03T07:57:46.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.336+0000 7ff81239d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c0b5460 con 0x7ff80c0b4aa0 2023-03-03T07:57:46.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.336+0000 7ff81139b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff80c1609b0 0x7ff80c160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:46.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.336+0000 7ff811b9c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff80c0b4aa0 0x7ff80c0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:46.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.336+0000 7ff81139b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff80c1609b0 0x7ff80c160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:60866/0 (socket says 172.21.15.98:60866) 2023-03-03T07:57:46.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.336+0000 7ff81139b700 1 -- 172.21.15.98:0/424530199 learned_addr learned my addr 172.21.15.98:0/424530199 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:46.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.336+0000 7ff810b9a700 1 --1- 172.21.15.98:0/424530199 >> v1:172.21.15.100:6789/0 conn(0x7ff80c0cf570 0x7ff80c15d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:43878/0 (socket says 172.21.15.98:43878) 2023-03-03T07:57:46.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.336+0000 7ff811b9c700 1 -- 172.21.15.98:0/424530199 >> v2:172.21.15.100:3300/0 conn(0x7ff80c1609b0 msgr2=0x7ff80c160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:46.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.336+0000 7ff811b9c700 1 --2- 172.21.15.98:0/424530199 >> v2:172.21.15.100:3300/0 conn(0x7ff80c1609b0 0x7ff80c160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.337+0000 7ff811b9c700 1 -- 172.21.15.98:0/424530199 >> v1:172.21.15.100:6789/0 conn(0x7ff80c0cf570 legacy=0x7ff80c15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:46.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.337+0000 7ff811b9c700 1 -- 172.21.15.98:0/424530199 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c15dd00 con 0x7ff80c0b4aa0 2023-03-03T07:57:46.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.337+0000 7ff81139b700 1 --2- 172.21.15.98:0/424530199 >> v2:172.21.15.100:3300/0 conn(0x7ff80c1609b0 0x7ff80c160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:46.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.337+0000 7ff811b9c700 1 --2- 172.21.15.98:0/424530199 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff80c0b4aa0 0x7ff80c0b4f20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff804009d90 tx=0x7ff80400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3dbf9068e07150f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:46.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.337+0000 7ff803fff700 1 -- 172.21.15.98:0/424530199 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80400e070 con 0x7ff80c0b4aa0 2023-03-03T07:57:46.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.337+0000 7ff803fff700 1 -- 172.21.15.98:0/424530199 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80400bb20 con 0x7ff80c0b4aa0 2023-03-03T07:57:46.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.337+0000 7ff803fff700 1 -- 172.21.15.98:0/424530199 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff804012430 con 0x7ff80c0b4aa0 2023-03-03T07:57:46.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.338+0000 7ff81239d700 1 -- 172.21.15.98:0/424530199 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff80c0b4aa0 msgr2=0x7ff80c0b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:46.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.338+0000 7ff81239d700 1 --2- 172.21.15.98:0/424530199 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff80c0b4aa0 0x7ff80c0b4f20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff804009d90 tx=0x7ff80400b330 comp rx=0 tx=0).stop 2023-03-03T07:57:46.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.338+0000 7ff81239d700 1 -- 172.21.15.98:0/424530199 shutdown_connections 2023-03-03T07:57:46.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.338+0000 7ff81239d700 1 --2- 172.21.15.98:0/424530199 >> v2:172.21.15.100:3300/0 conn(0x7ff80c1609b0 0x7ff80c160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.338+0000 7ff81239d700 1 --2- 172.21.15.98:0/424530199 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff80c0b4aa0 0x7ff80c0b4f20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.338+0000 7ff81239d700 1 -- 172.21.15.98:0/424530199 >> 172.21.15.98:0/424530199 conn(0x7ff80c06b880 msgr2=0x7ff80c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:46.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.338+0000 7ff81239d700 1 -- 172.21.15.98:0/424530199 shutdown_connections 2023-03-03T07:57:46.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.338+0000 7ff81239d700 1 -- 172.21.15.98:0/424530199 wait complete. 2023-03-03T07:57:46.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.340+0000 7ff81239d700 1 Processor -- start 2023-03-03T07:57:46.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.340+0000 7ff81239d700 1 -- start start 2023-03-03T07:57:46.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.341+0000 7ff81239d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff80c0b4aa0 0x7ff80c1f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:46.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.341+0000 7ff81239d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c0cf570 0x7ff80c0c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:46.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.341+0000 7ff81239d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff80c1609b0 0x7ff80c0c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:46.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.341+0000 7ff81239d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c0c9ce0 con 0x7ff80c0cf570 2023-03-03T07:57:46.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.341+0000 7ff81239d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c0cb650 con 0x7ff80c0b4aa0 2023-03-03T07:57:46.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.341+0000 7ff81239d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c0c9720 con 0x7ff80c1609b0 2023-03-03T07:57:46.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.342+0000 7ff810b9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c0cf570 0x7ff80c0c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:46.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.342+0000 7ff810b9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c0cf570 0x7ff80c0c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42402/0 (socket says 172.21.15.98:42402) 2023-03-03T07:57:46.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.342+0000 7ff810b9a700 1 -- 172.21.15.98:0/517250121 learned_addr learned my addr 172.21.15.98:0/517250121 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:46.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.342+0000 7ff81139b700 1 --2- 172.21.15.98:0/517250121 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff80c0b4aa0 0x7ff80c1f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:46.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.342+0000 7ff811b9c700 1 --2- 172.21.15.98:0/517250121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff80c1609b0 0x7ff80c0c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:46.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.342+0000 7ff810b9a700 1 -- 172.21.15.98:0/517250121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff80c1609b0 msgr2=0x7ff80c0c59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:46.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.342+0000 7ff810b9a700 1 --2- 172.21.15.98:0/517250121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff80c1609b0 0x7ff80c0c59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.342+0000 7ff810b9a700 1 -- 172.21.15.98:0/517250121 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff80c0b4aa0 msgr2=0x7ff80c1f24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:46.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.342+0000 7ff810b9a700 1 --2- 172.21.15.98:0/517250121 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff80c0b4aa0 0x7ff80c1f24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.342+0000 7ff810b9a700 1 -- 172.21.15.98:0/517250121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff808009550 con 0x7ff80c0cf570 2023-03-03T07:57:46.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.342+0000 7ff81139b700 1 --2- 172.21.15.98:0/517250121 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff80c0b4aa0 0x7ff80c1f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:46.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.343+0000 7ff810b9a700 1 --2- 172.21.15.98:0/517250121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c0cf570 0x7ff80c0c6ba0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00c520 tx=0x7ff7fc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.343+0000 7ff801ffb700 1 -- 172.21.15.98:0/517250121 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc005cb0 con 0x7ff80c0cf570 2023-03-03T07:57:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.343+0000 7ff801ffb700 1 -- 172.21.15.98:0/517250121 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7fc005e10 con 0x7ff80c0cf570 2023-03-03T07:57:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.343+0000 7ff801ffb700 1 -- 172.21.15.98:0/517250121 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc0072d0 con 0x7ff80c0cf570 2023-03-03T07:57:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.343+0000 7ff81239d700 1 -- 172.21.15.98:0/517250121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff804009a70 con 0x7ff80c0cf570 2023-03-03T07:57:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.343+0000 7ff81239d700 1 -- 172.21.15.98:0/517250121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80c0cda60 con 0x7ff80c0cf570 2023-03-03T07:57:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.344+0000 7ff81239d700 1 -- 172.21.15.98:0/517250121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80c0cd780 con 0x7ff80c0cf570 2023-03-03T07:57:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.345+0000 7ff801ffb700 1 -- 172.21.15.98:0/517250121 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7ff7fc019030 con 0x7ff80c0cf570 2023-03-03T07:57:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.346+0000 7ff801ffb700 1 --2- 172.21.15.98:0/517250121 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff7f00683c0 0x7ff7f006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.347+0000 7ff801ffb700 1 -- 172.21.15.98:0/517250121 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7ff7fc0850c0 con 0x7ff80c0cf570 2023-03-03T07:57:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.347+0000 7ff81139b700 1 --2- 172.21.15.98:0/517250121 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff7f00683c0 0x7ff7f006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.348+0000 7ff81139b700 1 --2- 172.21.15.98:0/517250121 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff7f00683c0 0x7ff7f006a870 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff808009d10 tx=0x7ff808008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.350+0000 7ff801ffb700 1 -- 172.21.15.98:0/517250121 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff7fc04b5e0 con 0x7ff80c0cf570 2023-03-03T07:57:46.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.544+0000 7ff81239d700 1 -- 172.21.15.98:0/517250121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff80c0cdd40 con 0x7ff80c0cf570 2023-03-03T07:57:46.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.545+0000 7ff801ffb700 1 -- 172.21.15.98:0/517250121 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v40) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff7fc0170a0 con 0x7ff80c0cf570 2023-03-03T07:57:46.547 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:57:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.548+0000 7ff81239d700 1 -- 172.21.15.98:0/517250121 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff7f00683c0 msgr2=0x7ff7f006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.548+0000 7ff81239d700 1 --2- 172.21.15.98:0/517250121 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff7f00683c0 0x7ff7f006a870 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff808009d10 tx=0x7ff808008040 comp rx=0 tx=0).stop 2023-03-03T07:57:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.548+0000 7ff81239d700 1 -- 172.21.15.98:0/517250121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c0cf570 msgr2=0x7ff80c0c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.548+0000 7ff81239d700 1 --2- 172.21.15.98:0/517250121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c0cf570 0x7ff80c0c6ba0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00c520 tx=0x7ff7fc00c830 comp rx=0 tx=0).stop 2023-03-03T07:57:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.548+0000 7ff81239d700 1 -- 172.21.15.98:0/517250121 shutdown_connections 2023-03-03T07:57:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.548+0000 7ff81239d700 1 --2- 172.21.15.98:0/517250121 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff7f00683c0 0x7ff7f006a870 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.548+0000 7ff81239d700 1 --2- 172.21.15.98:0/517250121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff80c1609b0 0x7ff80c0c59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.548+0000 7ff81239d700 1 --2- 172.21.15.98:0/517250121 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff80c0b4aa0 0x7ff80c1f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.548+0000 7ff81239d700 1 --2- 172.21.15.98:0/517250121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c0cf570 0x7ff80c0c6ba0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.548+0000 7ff81239d700 1 -- 172.21.15.98:0/517250121 >> 172.21.15.98:0/517250121 conn(0x7ff80c06b880 msgr2=0x7ff80c151dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:46.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.548+0000 7ff81239d700 1 -- 172.21.15.98:0/517250121 shutdown_connections 2023-03-03T07:57:46.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.548+0000 7ff81239d700 1 -- 172.21.15.98:0/517250121 wait complete. 2023-03-03T07:57:46.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:57:46.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.825+0000 7f17ccbff700 1 Processor -- start 2023-03-03T07:57:46.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.825+0000 7f17ccbff700 1 -- start start 2023-03-03T07:57:46.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.826+0000 7f17ccbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17c80b8d10 0x7f17c80cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:46.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.826+0000 7f17ccbff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f17c80b26d0 0x7f17c80b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:46.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.827+0000 7f17ccbff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c80cd7d0 con 0x7f17c80b5140 2023-03-03T07:57:46.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.827+0000 7f17ccbff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c80b55c0 con 0x7f17c80b26d0 2023-03-03T07:57:46.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.827+0000 7f17ccbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c80b90e0 con 0x7f17c80b8d10 2023-03-03T07:57:46.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.827+0000 7f17c659c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17c80b8d10 0x7f17c80cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.827+0000 7f17c5d9b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f17c80b26d0 0x7f17c80b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.827+0000 7f17c5d9b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f17c80b26d0 0x7f17c80b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:60878/0 (socket says 172.21.15.98:60878) 2023-03-03T07:57:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.827+0000 7f17c5d9b700 1 -- 172.21.15.98:0/4209860652 learned_addr learned my addr 172.21.15.98:0/4209860652 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.828+0000 7f17c5d9b700 1 -- 172.21.15.98:0/4209860652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17c80b8d10 msgr2=0x7f17c80cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.828+0000 7f17c5d9b700 1 --2- 172.21.15.98:0/4209860652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17c80b8d10 0x7f17c80cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.828+0000 7f17c5d9b700 1 -- 172.21.15.98:0/4209860652 >> v1:172.21.15.100:6789/0 conn(0x7f17c80b5140 legacy=0x7f17c8160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.828+0000 7f17c5d9b700 1 -- 172.21.15.98:0/4209860652 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c80cda20 con 0x7f17c80b26d0 2023-03-03T07:57:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.828+0000 7f17c659c700 1 --2- 172.21.15.98:0/4209860652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17c80b8d10 0x7f17c80cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.828+0000 7f17c559a700 1 -- 172.21.15.98:0/4209860652 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308073563 0 0) 0x7f17c80cd7d0 con 0x7f17c80b5140 2023-03-03T07:57:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.828+0000 7f17c5d9b700 1 --2- 172.21.15.98:0/4209860652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17c80b26d0 0x7f17c80b4c00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0098f0 tx=0x7f17bc009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1627f9f05c74a966 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.828+0000 7f17c559a700 1 -- 172.21.15.98:0/4209860652 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc011830 con 0x7f17c80b26d0 2023-03-03T07:57:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.828+0000 7f17c559a700 1 -- 172.21.15.98:0/4209860652 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17bc011990 con 0x7f17c80b26d0 2023-03-03T07:57:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.828+0000 7f17c559a700 1 -- 172.21.15.98:0/4209860652 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc011d20 con 0x7f17c80b26d0 2023-03-03T07:57:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.829+0000 7f17ccbff700 1 -- 172.21.15.98:0/4209860652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17c80b26d0 msgr2=0x7f17c80b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.829+0000 7f17ccbff700 1 --2- 172.21.15.98:0/4209860652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17c80b26d0 0x7f17c80b4c00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0098f0 tx=0x7f17bc009c00 comp rx=0 tx=0).stop 2023-03-03T07:57:46.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.829+0000 7f17ccbff700 1 -- 172.21.15.98:0/4209860652 shutdown_connections 2023-03-03T07:57:46.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.829+0000 7f17ccbff700 1 --2- 172.21.15.98:0/4209860652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17c80b8d10 0x7f17c80cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.829+0000 7f17ccbff700 1 --2- 172.21.15.98:0/4209860652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17c80b26d0 0x7f17c80b4c00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.829+0000 7f17ccbff700 1 -- 172.21.15.98:0/4209860652 >> 172.21.15.98:0/4209860652 conn(0x7f17c8073e70 msgr2=0x7f17c806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:46.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.829+0000 7f17ccbff700 1 -- 172.21.15.98:0/4209860652 shutdown_connections 2023-03-03T07:57:46.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.829+0000 7f17ccbff700 1 -- 172.21.15.98:0/4209860652 wait complete. 2023-03-03T07:57:46.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.831+0000 7f17ccbff700 1 Processor -- start 2023-03-03T07:57:46.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.831+0000 7f17ccbff700 1 -- start start 2023-03-03T07:57:46.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.832+0000 7f17ccbff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17c80b26d0 0x7f17c81f5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:46.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.832+0000 7f17ccbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17c80b5140 0x7f17c815ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:46.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.832+0000 7f17ccbff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c80b8d10 0x7f17c815f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:46.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.832+0000 7f17ccbff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c815ef60 con 0x7f17c80b8d10 2023-03-03T07:57:46.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.832+0000 7f17ccbff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81f6b60 con 0x7f17c80b26d0 2023-03-03T07:57:46.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.832+0000 7f17ccbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81f62c0 con 0x7f17c80b5140 2023-03-03T07:57:46.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.833+0000 7f17c6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c80b8d10 0x7f17c815f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:46.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.833+0000 7f17c6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c80b8d10 0x7f17c815f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42418/0 (socket says 172.21.15.98:42418) 2023-03-03T07:57:46.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.833+0000 7f17c6d9d700 1 -- 172.21.15.98:0/2183073513 learned_addr learned my addr 172.21.15.98:0/2183073513 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:46.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.833+0000 7f17c5d9b700 1 --2- 172.21.15.98:0/2183073513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17c80b5140 0x7f17c815ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:46.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.833+0000 7f17c659c700 1 --2- 172.21.15.98:0/2183073513 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17c80b26d0 0x7f17c81f5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:46.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.833+0000 7f17c6d9d700 1 -- 172.21.15.98:0/2183073513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17c80b5140 msgr2=0x7f17c815ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:46.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.833+0000 7f17c6d9d700 1 --2- 172.21.15.98:0/2183073513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17c80b5140 0x7f17c815ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.833+0000 7f17c6d9d700 1 -- 172.21.15.98:0/2183073513 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17c80b26d0 msgr2=0x7f17c81f5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:46.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.833+0000 7f17c6d9d700 1 --2- 172.21.15.98:0/2183073513 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17c80b26d0 0x7f17c81f5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:46.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.833+0000 7f17c6d9d700 1 -- 172.21.15.98:0/2183073513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b80094a0 con 0x7f17c80b8d10 2023-03-03T07:57:46.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.834+0000 7f17c6d9d700 1 --2- 172.21.15.98:0/2183073513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c80b8d10 0x7f17c815f890 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f17c000c7d0 tx=0x7f17c000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:46.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.834+0000 7f17b6ffd700 1 -- 172.21.15.98:0/2183073513 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c0003e30 con 0x7f17c80b8d10 2023-03-03T07:57:46.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.834+0000 7f17b6ffd700 1 -- 172.21.15.98:0/2183073513 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c0005070 con 0x7f17c80b8d10 2023-03-03T07:57:46.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.834+0000 7f17b6ffd700 1 -- 172.21.15.98:0/2183073513 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c0005420 con 0x7f17c80b8d10 2023-03-03T07:57:46.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.834+0000 7f17ccbff700 1 -- 172.21.15.98:0/2183073513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17bc009550 con 0x7f17c80b8d10 2023-03-03T07:57:46.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.834+0000 7f17ccbff700 1 -- 172.21.15.98:0/2183073513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c80d0e80 con 0x7f17c80b8d10 2023-03-03T07:57:46.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.835+0000 7f17ccbff700 1 -- 172.21.15.98:0/2183073513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c80cdd00 con 0x7f17c80b8d10 2023-03-03T07:57:46.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.836+0000 7f17b6ffd700 1 -- 172.21.15.98:0/2183073513 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f17c0019030 con 0x7f17c80b8d10 2023-03-03T07:57:46.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.838+0000 7f17b6ffd700 1 --2- 172.21.15.98:0/2183073513 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f17a80683c0 0x7f17a806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:46.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.838+0000 7f17b6ffd700 1 -- 172.21.15.98:0/2183073513 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f17c0085520 con 0x7f17c80b8d10 2023-03-03T07:57:46.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.838+0000 7f17c659c700 1 --2- 172.21.15.98:0/2183073513 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f17a80683c0 0x7f17a806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:46.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.838+0000 7f17c659c700 1 --2- 172.21.15.98:0/2183073513 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f17a80683c0 0x7f17a806a870 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009d00 tx=0x7f17b8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:46.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:46.841+0000 7f17b6ffd700 1 -- 172.21.15.98:0/2183073513 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f17c0051b10 con 0x7f17c80b8d10 2023-03-03T07:57:47.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.032+0000 7f17ccbff700 1 -- 172.21.15.98:0/2183073513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f17c80cdfe0 con 0x7f17c80b8d10 2023-03-03T07:57:47.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.032+0000 7f17b6ffd700 1 -- 172.21.15.98:0/2183073513 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v40) v1 ==== 100+0+11 (secure 0 0 0) 0x7f17c0017070 con 0x7f17c80b8d10 2023-03-03T07:57:47.034 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:57:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.035+0000 7f17ccbff700 1 -- 172.21.15.98:0/2183073513 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f17a80683c0 msgr2=0x7f17a806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.035+0000 7f17ccbff700 1 --2- 172.21.15.98:0/2183073513 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f17a80683c0 0x7f17a806a870 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009d00 tx=0x7f17b8008040 comp rx=0 tx=0).stop 2023-03-03T07:57:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.035+0000 7f17ccbff700 1 -- 172.21.15.98:0/2183073513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c80b8d10 msgr2=0x7f17c815f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:47.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.035+0000 7f17ccbff700 1 --2- 172.21.15.98:0/2183073513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c80b8d10 0x7f17c815f890 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f17c000c7d0 tx=0x7f17c000cb90 comp rx=0 tx=0).stop 2023-03-03T07:57:47.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.035+0000 7f17ccbff700 1 -- 172.21.15.98:0/2183073513 shutdown_connections 2023-03-03T07:57:47.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.035+0000 7f17ccbff700 1 --2- 172.21.15.98:0/2183073513 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f17a80683c0 0x7f17a806a870 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:47.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.035+0000 7f17ccbff700 1 --2- 172.21.15.98:0/2183073513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c80b8d10 0x7f17c815f890 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:47.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.035+0000 7f17ccbff700 1 --2- 172.21.15.98:0/2183073513 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17c80b26d0 0x7f17c81f5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:47.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.035+0000 7f17ccbff700 1 --2- 172.21.15.98:0/2183073513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17c80b5140 0x7f17c815ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:47.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.035+0000 7f17ccbff700 1 -- 172.21.15.98:0/2183073513 >> 172.21.15.98:0/2183073513 conn(0x7f17c8073e70 msgr2=0x7f17c80b5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:47.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.035+0000 7f17ccbff700 1 -- 172.21.15.98:0/2183073513 shutdown_connections 2023-03-03T07:57:47.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.035+0000 7f17ccbff700 1 -- 172.21.15.98:0/2183073513 wait complete. 2023-03-03T07:57:47.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T07:57:47.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.312+0000 7f60431ff700 1 Processor -- start 2023-03-03T07:57:47.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.313+0000 7f60431ff700 1 -- start start 2023-03-03T07:57:47.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.314+0000 7f60431ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f603c0b8d10 0x7f603c0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:47.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.314+0000 7f60431ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f603c0b26d0 0x7f603c0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:47.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.314+0000 7f60431ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c0cd7d0 con 0x7f603c0b5140 2023-03-03T07:57:47.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.314+0000 7f60431ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f603c0b55c0 con 0x7f603c0b8d10 2023-03-03T07:57:47.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.314+0000 7f60431ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c0b90e0 con 0x7f603c0b26d0 2023-03-03T07:57:47.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.315+0000 7f6040f9b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f603c0b8d10 0x7f603c0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.315+0000 7f603bfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f603c0b26d0 0x7f603c0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.315+0000 7f603bfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f603c0b26d0 0x7f603c0b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:49614/0 (socket says 172.21.15.98:49614) 2023-03-03T07:57:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.315+0000 7f603bfff700 1 -- 172.21.15.98:0/3445976094 learned_addr learned my addr 172.21.15.98:0/3445976094 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.315+0000 7f604179c700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f603c0b5140 0x7f603c160520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:43898/0 (socket says 172.21.15.98:43898) 2023-03-03T07:57:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.315+0000 7f603bfff700 1 -- 172.21.15.98:0/3445976094 >> v2:172.21.15.100:3300/0 conn(0x7f603c0b8d10 msgr2=0x7f603c0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.315+0000 7f603bfff700 1 --2- 172.21.15.98:0/3445976094 >> v2:172.21.15.100:3300/0 conn(0x7f603c0b8d10 0x7f603c0cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.315+0000 7f603bfff700 1 -- 172.21.15.98:0/3445976094 >> v1:172.21.15.100:6789/0 conn(0x7f603c0b5140 legacy=0x7f603c160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.315+0000 7f603bfff700 1 -- 172.21.15.98:0/3445976094 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c0cda20 con 0x7f603c0b26d0 2023-03-03T07:57:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.316+0000 7f603bfff700 1 --2- 172.21.15.98:0/3445976094 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f603c0b26d0 0x7f603c0b4c00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f60280098f0 tx=0x7f6028009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18d48b2b70a7d832 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.316+0000 7f603b7fe700 1 -- 172.21.15.98:0/3445976094 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028011830 con 0x7f603c0b26d0 2023-03-03T07:57:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.316+0000 7f603b7fe700 1 -- 172.21.15.98:0/3445976094 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6028011990 con 0x7f603c0b26d0 2023-03-03T07:57:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.316+0000 7f603b7fe700 1 -- 172.21.15.98:0/3445976094 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028011d20 con 0x7f603c0b26d0 2023-03-03T07:57:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.316+0000 7f60431ff700 1 -- 172.21.15.98:0/3445976094 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f603c0b26d0 msgr2=0x7f603c0b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.316+0000 7f60431ff700 1 --2- 172.21.15.98:0/3445976094 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f603c0b26d0 0x7f603c0b4c00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f60280098f0 tx=0x7f6028009c00 comp rx=0 tx=0).stop 2023-03-03T07:57:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.317+0000 7f60431ff700 1 -- 172.21.15.98:0/3445976094 shutdown_connections 2023-03-03T07:57:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.317+0000 7f60431ff700 1 --2- 172.21.15.98:0/3445976094 >> v2:172.21.15.100:3300/0 conn(0x7f603c0b8d10 0x7f603c0cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.317+0000 7f60431ff700 1 --2- 172.21.15.98:0/3445976094 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f603c0b26d0 0x7f603c0b4c00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:47.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.317+0000 7f60431ff700 1 -- 172.21.15.98:0/3445976094 >> 172.21.15.98:0/3445976094 conn(0x7f603c073e70 msgr2=0x7f603c06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:47.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.317+0000 7f60431ff700 1 -- 172.21.15.98:0/3445976094 shutdown_connections 2023-03-03T07:57:47.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.317+0000 7f60431ff700 1 -- 172.21.15.98:0/3445976094 wait complete. 2023-03-03T07:57:47.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.319+0000 7f60431ff700 1 Processor -- start 2023-03-03T07:57:47.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.319+0000 7f60431ff700 1 -- start start 2023-03-03T07:57:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.320+0000 7f60431ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f603c0b26d0 0x7f603c15e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.320+0000 7f60431ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f603c0b5140 0x7f603c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.320+0000 7f60431ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f603c0b8d10 0x7f603c0b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.320+0000 7f60431ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c15ec30 con 0x7f603c0b5140 2023-03-03T07:57:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.320+0000 7f60431ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c15e950 con 0x7f603c0b8d10 2023-03-03T07:57:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.320+0000 7f60431ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c15d0e0 con 0x7f603c0b26d0 2023-03-03T07:57:47.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.321+0000 7f603bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f603c0b5140 0x7f603c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:47.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.321+0000 7f603bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f603c0b5140 0x7f603c150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42422/0 (socket says 172.21.15.98:42422) 2023-03-03T07:57:47.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.321+0000 7f603bfff700 1 -- 172.21.15.98:0/2275016044 learned_addr learned my addr 172.21.15.98:0/2275016044 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:47.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.321+0000 7f6040f9b700 1 --2- 172.21.15.98:0/2275016044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f603c0b26d0 0x7f603c15e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:47.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.321+0000 7f604179c700 1 --2- 172.21.15.98:0/2275016044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f603c0b8d10 0x7f603c0b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:47.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.321+0000 7f603bfff700 1 -- 172.21.15.98:0/2275016044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f603c0b26d0 msgr2=0x7f603c15e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:47.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.321+0000 7f603bfff700 1 --2- 172.21.15.98:0/2275016044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f603c0b26d0 0x7f603c15e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:47.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.321+0000 7f603bfff700 1 -- 172.21.15.98:0/2275016044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f603c0b8d10 msgr2=0x7f603c0b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:47.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.321+0000 7f603bfff700 1 --2- 172.21.15.98:0/2275016044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f603c0b8d10 0x7f603c0b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:47.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.321+0000 7f603bfff700 1 -- 172.21.15.98:0/2275016044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6028009550 con 0x7f603c0b5140 2023-03-03T07:57:47.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.321+0000 7f603bfff700 1 --2- 172.21.15.98:0/2275016044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f603c0b5140 0x7f603c150ad0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6028009c40 tx=0x7f6028004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.322+0000 7f60397fa700 1 -- 172.21.15.98:0/2275016044 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60280036a0 con 0x7f603c0b5140 2023-03-03T07:57:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.322+0000 7f60397fa700 1 -- 172.21.15.98:0/2275016044 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602801e070 con 0x7f603c0b5140 2023-03-03T07:57:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.322+0000 7f60397fa700 1 -- 172.21.15.98:0/2275016044 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028003a40 con 0x7f603c0b5140 2023-03-03T07:57:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.322+0000 7f60431ff700 1 -- 172.21.15.98:0/2275016044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603c0d0e80 con 0x7f603c0b5140 2023-03-03T07:57:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.322+0000 7f60431ff700 1 -- 172.21.15.98:0/2275016044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603c0cdd00 con 0x7f603c0b5140 2023-03-03T07:57:47.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.323+0000 7f60431ff700 1 -- 172.21.15.98:0/2275016044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603c0cdfe0 con 0x7f603c0b5140 2023-03-03T07:57:47.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.324+0000 7f60397fa700 1 -- 172.21.15.98:0/2275016044 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f602800f050 con 0x7f603c0b5140 2023-03-03T07:57:47.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.325+0000 7f60397fa700 1 --2- 172.21.15.98:0/2275016044 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f60200683c0 0x7f602006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:47.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.326+0000 7f60397fa700 1 -- 172.21.15.98:0/2275016044 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f6028086ea0 con 0x7f603c0b5140 2023-03-03T07:57:47.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.326+0000 7f6040f9b700 1 --2- 172.21.15.98:0/2275016044 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f60200683c0 0x7f602006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:47.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.327+0000 7f6040f9b700 1 --2- 172.21.15.98:0/2275016044 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f60200683c0 0x7f602006a870 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6030009b70 tx=0x7f6030008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:47.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.330+0000 7f60397fa700 1 -- 172.21.15.98:0/2275016044 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6028053520 con 0x7f603c0b5140 2023-03-03T07:57:47.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.542+0000 7f60431ff700 1 -- 172.21.15.98:0/2275016044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f603c04e6e0 con 0x7f603c0b5140 2023-03-03T07:57:47.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.542+0000 7f60397fa700 1 -- 172.21.15.98:0/2275016044 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v40) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6028020020 con 0x7f603c0b5140 2023-03-03T07:57:47.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.545+0000 7f60431ff700 1 -- 172.21.15.98:0/2275016044 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f60200683c0 msgr2=0x7f602006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.545+0000 7f60431ff700 1 --2- 172.21.15.98:0/2275016044 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f60200683c0 0x7f602006a870 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6030009b70 tx=0x7f6030008040 comp rx=0 tx=0).stop 2023-03-03T07:57:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.545+0000 7f60431ff700 1 -- 172.21.15.98:0/2275016044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f603c0b5140 msgr2=0x7f603c150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.545+0000 7f60431ff700 1 --2- 172.21.15.98:0/2275016044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f603c0b5140 0x7f603c150ad0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6028009c40 tx=0x7f6028004750 comp rx=0 tx=0).stop 2023-03-03T07:57:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.545+0000 7f60431ff700 1 -- 172.21.15.98:0/2275016044 shutdown_connections 2023-03-03T07:57:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.545+0000 7f60431ff700 1 --2- 172.21.15.98:0/2275016044 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f60200683c0 0x7f602006a870 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.545+0000 7f60431ff700 1 --2- 172.21.15.98:0/2275016044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f603c0b8d10 0x7f603c0b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:47.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.545+0000 7f60431ff700 1 --2- 172.21.15.98:0/2275016044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f603c0b26d0 0x7f603c15e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:47.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.545+0000 7f60431ff700 1 --2- 172.21.15.98:0/2275016044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f603c0b5140 0x7f603c150ad0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:47.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.545+0000 7f60431ff700 1 -- 172.21.15.98:0/2275016044 >> 172.21.15.98:0/2275016044 conn(0x7f603c073e70 msgr2=0x7f603c0b7140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:47.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.545+0000 7f60431ff700 1 -- 172.21.15.98:0/2275016044 shutdown_connections 2023-03-03T07:57:47.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:47.545+0000 7f60431ff700 1 -- 172.21.15.98:0/2275016044 wait complete. 2023-03-03T07:57:47.548 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T07:57:47.575 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2023-03-03T07:57:47.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:47.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-03-03T07:57:47.671 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[125617]: fuse finished with error 0 and tester_r 0 2023-03-03T07:57:47.677 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T07:57:47.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:47.678 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-03-03T07:57:47.783 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:57:47.783 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:57:47.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:57:48.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.099+0000 7fada53ff700 1 Processor -- start 2023-03-03T07:57:48.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.100+0000 7fada53ff700 1 -- start start 2023-03-03T07:57:48.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.101+0000 7fada53ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fada00b8b90 0x7fada00cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:48.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.101+0000 7fada53ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fada00b5410 0x7fada00d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:48.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.101+0000 7fada53ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada00d0ef0 con 0x7fada00b87c0 2023-03-03T07:57:48.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.101+0000 7fada53ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada00d0600 con 0x7fada00b5410 2023-03-03T07:57:48.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.101+0000 7fada53ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada00cd970 con 0x7fada00b8b90 2023-03-03T07:57:48.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.101+0000 7fad9e7fc700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fada00b8b90 0x7fada00cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:48.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.101+0000 7fad9f7fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fada00b5410 0x7fada00d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:48.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.102+0000 7fad9f7fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fada00b5410 0x7fada00d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:60920/0 (socket says 172.21.15.98:60920) 2023-03-03T07:57:48.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.102+0000 7fad9e7fc700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fada00b8b90 0x7fada00cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:49630/0 (socket says 172.21.15.98:49630) 2023-03-03T07:57:48.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.102+0000 7fad9f7fe700 1 -- 172.21.15.98:0/1364265306 learned_addr learned my addr 172.21.15.98:0/1364265306 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:48.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.102+0000 7fad9effd700 1 --1- 172.21.15.98:0/1364265306 >> v1:172.21.15.100:6789/0 conn(0x7fada00b87c0 0x7fada00b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:43900/0 (socket says 172.21.15.98:43900) 2023-03-03T07:57:48.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.102+0000 7fad9e7fc700 1 -- 172.21.15.98:0/1364265306 >> v2:172.21.15.100:3300/0 conn(0x7fada00b5410 msgr2=0x7fada00d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:48.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.102+0000 7fad9e7fc700 1 --2- 172.21.15.98:0/1364265306 >> v2:172.21.15.100:3300/0 conn(0x7fada00b5410 0x7fada00d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.102+0000 7fad9e7fc700 1 -- 172.21.15.98:0/1364265306 >> v1:172.21.15.100:6789/0 conn(0x7fada00b87c0 legacy=0x7fada00b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:48.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.102+0000 7fad9e7fc700 1 -- 172.21.15.98:0/1364265306 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada00cdc50 con 0x7fada00b8b90 2023-03-03T07:57:48.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.102+0000 7fad9f7fe700 1 --2- 172.21.15.98:0/1364265306 >> v2:172.21.15.100:3300/0 conn(0x7fada00b5410 0x7fada00d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:48.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.102+0000 7fad9dffb700 1 -- 172.21.15.98:0/1364265306 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83319659 0 0) 0x7fada00d0ef0 con 0x7fada00b87c0 2023-03-03T07:57:48.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.102+0000 7fad9e7fc700 1 --2- 172.21.15.98:0/1364265306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fada00b8b90 0x7fada00cffe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fad94009840 tx=0x7fad94009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c48f50b727be5402 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:48.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.103+0000 7fad9dffb700 1 -- 172.21.15.98:0/1364265306 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad94011830 con 0x7fada00b8b90 2023-03-03T07:57:48.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.103+0000 7fad9dffb700 1 -- 172.21.15.98:0/1364265306 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad94011990 con 0x7fada00b8b90 2023-03-03T07:57:48.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.103+0000 7fad9dffb700 1 -- 172.21.15.98:0/1364265306 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad94011d20 con 0x7fada00b8b90 2023-03-03T07:57:48.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.103+0000 7fada53ff700 1 -- 172.21.15.98:0/1364265306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fada00b8b90 msgr2=0x7fada00cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:48.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.103+0000 7fada53ff700 1 --2- 172.21.15.98:0/1364265306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fada00b8b90 0x7fada00cffe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fad94009840 tx=0x7fad94009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:48.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.103+0000 7fada53ff700 1 -- 172.21.15.98:0/1364265306 shutdown_connections 2023-03-03T07:57:48.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.103+0000 7fada53ff700 1 --2- 172.21.15.98:0/1364265306 >> v2:172.21.15.100:3300/0 conn(0x7fada00b5410 0x7fada00d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.103+0000 7fada53ff700 1 --2- 172.21.15.98:0/1364265306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fada00b8b90 0x7fada00cffe0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.103+0000 7fada53ff700 1 -- 172.21.15.98:0/1364265306 >> 172.21.15.98:0/1364265306 conn(0x7fada006aee0 msgr2=0x7fada006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:48.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.103+0000 7fada53ff700 1 -- 172.21.15.98:0/1364265306 shutdown_connections 2023-03-03T07:57:48.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.104+0000 7fada53ff700 1 -- 172.21.15.98:0/1364265306 wait complete. 2023-03-03T07:57:48.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.105+0000 7fada53ff700 1 Processor -- start 2023-03-03T07:57:48.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.106+0000 7fada53ff700 1 -- start start 2023-03-03T07:57:48.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fada53ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fada00b5410 0x7fada01e55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:48.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fada53ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fada00b87c0 0x7fada01e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:48.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fada53ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fada00b8b90 0x7fada01f53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:48.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fada53ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada01f2ed0 con 0x7fada00b5410 2023-03-03T07:57:48.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fada53ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada00cb3a0 con 0x7fada00b87c0 2023-03-03T07:57:48.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fada53ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada01e8120 con 0x7fada00b8b90 2023-03-03T07:57:48.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fad9effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fada00b5410 0x7fada01e55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:48.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fad9e7fc700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fada00b87c0 0x7fada01e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:48.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fad9e7fc700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fada00b87c0 0x7fada01e78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:60932/0 (socket says 172.21.15.98:60932) 2023-03-03T07:57:48.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fad9f7fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fada00b8b90 0x7fada01f53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:48.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fad9e7fc700 1 -- 172.21.15.98:0/3935950201 learned_addr learned my addr 172.21.15.98:0/3935950201 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:48.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fad9effd700 1 -- 172.21.15.98:0/3935950201 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fada00b8b90 msgr2=0x7fada01f53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:48.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fad9effd700 1 --2- 172.21.15.98:0/3935950201 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fada00b8b90 0x7fada01f53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fad9effd700 1 -- 172.21.15.98:0/3935950201 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fada00b87c0 msgr2=0x7fada01e78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:48.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fad9effd700 1 --2- 172.21.15.98:0/3935950201 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fada00b87c0 0x7fada01e78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.107+0000 7fad9effd700 1 -- 172.21.15.98:0/3935950201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad98009b30 con 0x7fada00b5410 2023-03-03T07:57:48.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.108+0000 7fad9f7fe700 1 --2- 172.21.15.98:0/3935950201 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fada00b8b90 0x7fada01f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:48.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.108+0000 7fad9e7fc700 1 --2- 172.21.15.98:0/3935950201 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fada00b87c0 0x7fada01e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:48.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.108+0000 7fad9effd700 1 --2- 172.21.15.98:0/3935950201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fada00b5410 0x7fada01e55f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fad9000c4b0 tx=0x7fad9000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:48.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.108+0000 7fad877fe700 1 -- 172.21.15.98:0/3935950201 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90004c80 con 0x7fada00b5410 2023-03-03T07:57:48.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.108+0000 7fad877fe700 1 -- 172.21.15.98:0/3935950201 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad90004e00 con 0x7fada00b5410 2023-03-03T07:57:48.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.108+0000 7fad877fe700 1 -- 172.21.15.98:0/3935950201 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90007310 con 0x7fada00b5410 2023-03-03T07:57:48.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.108+0000 7fada53ff700 1 -- 172.21.15.98:0/3935950201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad940094a0 con 0x7fada00b5410 2023-03-03T07:57:48.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.108+0000 7fada53ff700 1 -- 172.21.15.98:0/3935950201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada00cdf30 con 0x7fada00b5410 2023-03-03T07:57:48.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.110+0000 7fada53ff700 1 -- 172.21.15.98:0/3935950201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada00ce0c0 con 0x7fada00b5410 2023-03-03T07:57:48.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.111+0000 7fad877fe700 1 -- 172.21.15.98:0/3935950201 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fad9001a020 con 0x7fada00b5410 2023-03-03T07:57:48.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.113+0000 7fad877fe700 1 --2- 172.21.15.98:0/3935950201 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fad8c0683c0 0x7fad8c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:48.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.113+0000 7fad877fe700 1 -- 172.21.15.98:0/3935950201 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fad90085960 con 0x7fada00b5410 2023-03-03T07:57:48.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.113+0000 7fad9e7fc700 1 --2- 172.21.15.98:0/3935950201 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fad8c0683c0 0x7fad8c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:48.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.114+0000 7fad9e7fc700 1 --2- 172.21.15.98:0/3935950201 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fad8c0683c0 0x7fad8c06a870 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fad94000c00 tx=0x7fad9401c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:48.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.117+0000 7fad877fe700 1 -- 172.21.15.98:0/3935950201 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fad90052200 con 0x7fada00b5410 2023-03-03T07:57:48.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.315+0000 7fada53ff700 1 -- 172.21.15.98:0/3935950201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fada00729b0 con 0x7fada00b5410 2023-03-03T07:57:48.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.316+0000 7fad877fe700 1 -- 172.21.15.98:0/3935950201 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+17972 (secure 0 0 0) 0x7fad90055820 con 0x7fada00b5410 2023-03-03T07:57:48.322 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:48.322 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":40,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:57:28.259781+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:57:15.545608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-03-03T07:57:16.283131+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.31","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:57:48.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.319+0000 7fada53ff700 1 -- 172.21.15.98:0/3935950201 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fad8c0683c0 msgr2=0x7fad8c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:48.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.319+0000 7fada53ff700 1 --2- 172.21.15.98:0/3935950201 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fad8c0683c0 0x7fad8c06a870 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fad94000c00 tx=0x7fad9401c040 comp rx=0 tx=0).stop 2023-03-03T07:57:48.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.319+0000 7fada53ff700 1 -- 172.21.15.98:0/3935950201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fada00b5410 msgr2=0x7fada01e55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:48.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.319+0000 7fada53ff700 1 --2- 172.21.15.98:0/3935950201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fada00b5410 0x7fada01e55f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fad9000c4b0 tx=0x7fad9000c7c0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.319+0000 7fada53ff700 1 -- 172.21.15.98:0/3935950201 shutdown_connections 2023-03-03T07:57:48.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.319+0000 7fada53ff700 1 --2- 172.21.15.98:0/3935950201 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fad8c0683c0 0x7fad8c06a870 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.319+0000 7fada53ff700 1 --2- 172.21.15.98:0/3935950201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fada00b5410 0x7fada01e55f0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.319+0000 7fada53ff700 1 --2- 172.21.15.98:0/3935950201 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fada00b8b90 0x7fada01f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.319+0000 7fada53ff700 1 --2- 172.21.15.98:0/3935950201 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fada00b87c0 0x7fada01e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.319+0000 7fada53ff700 1 -- 172.21.15.98:0/3935950201 >> 172.21.15.98:0/3935950201 conn(0x7fada006aee0 msgr2=0x7fada00b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:48.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.319+0000 7fada53ff700 1 -- 172.21.15.98:0/3935950201 shutdown_connections 2023-03-03T07:57:48.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.319+0000 7fada53ff700 1 -- 172.21.15.98:0/3935950201 wait complete. 2023-03-03T07:57:48.347 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:57:15.545608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:57:16.283131+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:57:48.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:57:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.597+0000 7f7409b9d700 1 Processor -- start 2023-03-03T07:57:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.598+0000 7f7409b9d700 1 -- start start 2023-03-03T07:57:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.599+0000 7f7409b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f74040b8910 0x7f74041604a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.599+0000 7f7409b9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74040b5e70 0x7f74040b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.599+0000 7f7409b9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f74040cf570 0x7f74040b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.599+0000 7f7409b9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f740415ce00 con 0x7f74040b5e70 2023-03-03T07:57:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.599+0000 7f7409b9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74040d0bf0 con 0x7f74040cf570 2023-03-03T07:57:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.599+0000 7f7409b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74040b8ce0 con 0x7f74040b8910 2023-03-03T07:57:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.599+0000 7f7403fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74040b5e70 0x7f74040b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.600+0000 7f7403fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74040b5e70 0x7f74040b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42438/0 (socket says 172.21.15.98:42438) 2023-03-03T07:57:48.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.600+0000 7f7403fff700 1 -- 172.21.15.98:0/553269598 learned_addr learned my addr 172.21.15.98:0/553269598 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:48.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.600+0000 7f740939c700 1 --2- 172.21.15.98:0/553269598 >> v2:172.21.15.100:3300/0 conn(0x7f74040cf570 0x7f74040b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:48.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.600+0000 7f7408b9b700 1 --2- 172.21.15.98:0/553269598 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f74040b8910 0x7f74041604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:48.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.600+0000 7f7403fff700 1 -- 172.21.15.98:0/553269598 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f74040b8910 msgr2=0x7f74041604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:48.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.600+0000 7f7403fff700 1 --2- 172.21.15.98:0/553269598 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f74040b8910 0x7f74041604a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.600+0000 7f7403fff700 1 -- 172.21.15.98:0/553269598 >> v2:172.21.15.100:3300/0 conn(0x7f74040cf570 msgr2=0x7f74040b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:48.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.600+0000 7f7403fff700 1 --2- 172.21.15.98:0/553269598 >> v2:172.21.15.100:3300/0 conn(0x7f74040cf570 0x7f74040b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.600+0000 7f7403fff700 1 -- 172.21.15.98:0/553269598 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7404160cf0 con 0x7f74040b5e70 2023-03-03T07:57:48.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.600+0000 7f740939c700 1 --2- 172.21.15.98:0/553269598 >> v2:172.21.15.100:3300/0 conn(0x7f74040cf570 0x7f74040b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:48.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.600+0000 7f7408b9b700 1 --2- 172.21.15.98:0/553269598 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f74040b8910 0x7f74041604a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:48.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.600+0000 7f7403fff700 1 --2- 172.21.15.98:0/553269598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74040b5e70 0x7f74040b4aa0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f74040d0ba0 tx=0x7f73f0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50525507598c83c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.601+0000 7f74037fe700 1 -- 172.21.15.98:0/553269598 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f000b870 con 0x7f74040b5e70 2023-03-03T07:57:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.601+0000 7f74037fe700 1 -- 172.21.15.98:0/553269598 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f000b9d0 con 0x7f74040b5e70 2023-03-03T07:57:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.601+0000 7f74037fe700 1 -- 172.21.15.98:0/553269598 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f000bd60 con 0x7f74040b5e70 2023-03-03T07:57:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.601+0000 7f7409b9d700 1 -- 172.21.15.98:0/553269598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74040b5e70 msgr2=0x7f74040b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.601+0000 7f7409b9d700 1 --2- 172.21.15.98:0/553269598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74040b5e70 0x7f74040b4aa0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f74040d0ba0 tx=0x7f73f0009bd0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.601+0000 7f7409b9d700 1 -- 172.21.15.98:0/553269598 shutdown_connections 2023-03-03T07:57:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.601+0000 7f7409b9d700 1 --2- 172.21.15.98:0/553269598 >> v2:172.21.15.100:3300/0 conn(0x7f74040cf570 0x7f74040b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.601+0000 7f7409b9d700 1 --2- 172.21.15.98:0/553269598 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f74040b8910 0x7f74041604a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.601+0000 7f7409b9d700 1 --2- 172.21.15.98:0/553269598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74040b5e70 0x7f74040b4aa0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.601+0000 7f7409b9d700 1 -- 172.21.15.98:0/553269598 >> 172.21.15.98:0/553269598 conn(0x7f740406b880 msgr2=0x7f74040cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.602+0000 7f7409b9d700 1 -- 172.21.15.98:0/553269598 shutdown_connections 2023-03-03T07:57:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.602+0000 7f7409b9d700 1 -- 172.21.15.98:0/553269598 wait complete. 2023-03-03T07:57:48.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.604+0000 7f7409b9d700 1 Processor -- start 2023-03-03T07:57:48.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.604+0000 7f7409b9d700 1 -- start start 2023-03-03T07:57:48.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.605+0000 7f7409b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f74040b5e70 0x7f74041544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:48.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.605+0000 7f7409b9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f74040b8910 0x7f74041f3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:48.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.605+0000 7f7409b9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74040cf570 0x7f74041f4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:48.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.605+0000 7f7409b9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74041f3da0 con 0x7f74040cf570 2023-03-03T07:57:48.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.605+0000 7f7409b9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74041f4080 con 0x7f74040b8910 2023-03-03T07:57:48.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.605+0000 7f7409b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74041f4360 con 0x7f74040b5e70 2023-03-03T07:57:48.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.605+0000 7f740939c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74040cf570 0x7f74041f4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:48.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.605+0000 7f740939c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74040cf570 0x7f74041f4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42454/0 (socket says 172.21.15.98:42454) 2023-03-03T07:57:48.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.605+0000 7f740939c700 1 -- 172.21.15.98:0/3450502085 learned_addr learned my addr 172.21.15.98:0/3450502085 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:48.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.605+0000 7f7408b9b700 1 --2- 172.21.15.98:0/3450502085 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f74040b5e70 0x7f74041544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:48.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.606+0000 7f7403fff700 1 --2- 172.21.15.98:0/3450502085 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f74040b8910 0x7f74041f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:48.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.606+0000 7f740939c700 1 -- 172.21.15.98:0/3450502085 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f74040b5e70 msgr2=0x7f74041544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.606+0000 7f740939c700 1 --2- 172.21.15.98:0/3450502085 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f74040b5e70 0x7f74041544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.606+0000 7f740939c700 1 -- 172.21.15.98:0/3450502085 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f74040b8910 msgr2=0x7f74041f3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.606+0000 7f740939c700 1 --2- 172.21.15.98:0/3450502085 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f74040b8910 0x7f74041f3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.606+0000 7f740939c700 1 -- 172.21.15.98:0/3450502085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f0009550 con 0x7f74040cf570 2023-03-03T07:57:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.606+0000 7f7403fff700 1 --2- 172.21.15.98:0/3450502085 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f74040b8910 0x7f74041f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.606+0000 7f740939c700 1 --2- 172.21.15.98:0/3450502085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74040cf570 0x7f74041f4e40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f73fc007ef0 tx=0x7f73fc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.606+0000 7f74017fa700 1 -- 172.21.15.98:0/3450502085 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73fc00de30 con 0x7f74040cf570 2023-03-03T07:57:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.606+0000 7f74017fa700 1 -- 172.21.15.98:0/3450502085 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73fc00f040 con 0x7f74040cf570 2023-03-03T07:57:48.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.606+0000 7f74017fa700 1 -- 172.21.15.98:0/3450502085 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73fc003c60 con 0x7f74040cf570 2023-03-03T07:57:48.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.606+0000 7f7409b9d700 1 -- 172.21.15.98:0/3450502085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74041612b0 con 0x7f74040cf570 2023-03-03T07:57:48.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.607+0000 7f7409b9d700 1 -- 172.21.15.98:0/3450502085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7404160fd0 con 0x7f74040cf570 2023-03-03T07:57:48.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.608+0000 7f7409b9d700 1 -- 172.21.15.98:0/3450502085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74040d0eb0 con 0x7f74040cf570 2023-03-03T07:57:48.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.609+0000 7f74017fa700 1 -- 172.21.15.98:0/3450502085 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f73fc01c030 con 0x7f74040cf570 2023-03-03T07:57:48.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.610+0000 7f74017fa700 1 --2- 172.21.15.98:0/3450502085 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f73e80683c0 0x7f73e806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:48.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.610+0000 7f74017fa700 1 -- 172.21.15.98:0/3450502085 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f73fc085ed0 con 0x7f74040cf570 2023-03-03T07:57:48.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.611+0000 7f7408b9b700 1 --2- 172.21.15.98:0/3450502085 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f73e80683c0 0x7f73e806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:48.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.611+0000 7f7408b9b700 1 --2- 172.21.15.98:0/3450502085 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f73e80683c0 0x7f73e806a870 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f73f80045f0 tx=0x7f73f8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:48.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.614+0000 7f74017fa700 1 -- 172.21.15.98:0/3450502085 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f73fc052550 con 0x7f74040cf570 2023-03-03T07:57:48.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.811+0000 7f7409b9d700 1 -- 172.21.15.98:0/3450502085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7404068290 con 0x7f74040cf570 2023-03-03T07:57:48.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.812+0000 7f74017fa700 1 -- 172.21.15.98:0/3450502085 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f73fc01a020 con 0x7f74040cf570 2023-03-03T07:57:48.814 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:57:48.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.814+0000 7f7409b9d700 1 -- 172.21.15.98:0/3450502085 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f73e80683c0 msgr2=0x7f73e806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:48.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.814+0000 7f7409b9d700 1 --2- 172.21.15.98:0/3450502085 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f73e80683c0 0x7f73e806a870 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f73f80045f0 tx=0x7f73f8008040 comp rx=0 tx=0).stop 2023-03-03T07:57:48.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.814+0000 7f7409b9d700 1 -- 172.21.15.98:0/3450502085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74040cf570 msgr2=0x7f74041f4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:48.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.814+0000 7f7409b9d700 1 --2- 172.21.15.98:0/3450502085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74040cf570 0x7f74041f4e40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f73fc007ef0 tx=0x7f73fc00d3b0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.815+0000 7f7409b9d700 1 -- 172.21.15.98:0/3450502085 shutdown_connections 2023-03-03T07:57:48.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.815+0000 7f7409b9d700 1 --2- 172.21.15.98:0/3450502085 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f73e80683c0 0x7f73e806a870 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.815+0000 7f7409b9d700 1 --2- 172.21.15.98:0/3450502085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74040cf570 0x7f74041f4e40 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.815+0000 7f7409b9d700 1 --2- 172.21.15.98:0/3450502085 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f74040b5e70 0x7f74041544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.815+0000 7f7409b9d700 1 --2- 172.21.15.98:0/3450502085 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f74040b8910 0x7f74041f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:48.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.815+0000 7f7409b9d700 1 -- 172.21.15.98:0/3450502085 >> 172.21.15.98:0/3450502085 conn(0x7f740406b880 msgr2=0x7f740415c560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:48.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.815+0000 7f7409b9d700 1 -- 172.21.15.98:0/3450502085 shutdown_connections 2023-03-03T07:57:48.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:48.815+0000 7f7409b9d700 1 -- 172.21.15.98:0/3450502085 wait complete. 2023-03-03T07:57:48.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:57:49.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.091+0000 7fe7611ff700 1 Processor -- start 2023-03-03T07:57:49.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.091+0000 7fe7611ff700 1 -- start start 2023-03-03T07:57:49.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.093+0000 7fe7611ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe75c0b8d10 0x7fe75c0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:49.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.093+0000 7fe7611ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c0cd7d0 con 0x7fe75c0b26d0 2023-03-03T07:57:49.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.093+0000 7fe7611ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c0b90e0 con 0x7fe75c0b5310 2023-03-03T07:57:49.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.093+0000 7fe7611ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe75c0cda20 con 0x7fe75c0b8d10 2023-03-03T07:57:49.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.093+0000 7fe75a59c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe75c0b26d0 0x7fe75c0b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44228/0 (socket says 172.21.15.98:44228) 2023-03-03T07:57:49.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.093+0000 7fe75a59c700 1 -- 172.21.15.98:0/1283581388 learned_addr learned my addr 172.21.15.98:0/1283581388 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:49.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.093+0000 7fe75ad9d700 1 --2- 172.21.15.98:0/1283581388 >> v2:172.21.15.100:3300/0 conn(0x7fe75c0b8d10 0x7fe75c0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:49.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.094+0000 7fe759d9b700 1 -- 172.21.15.98:0/1283581388 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495847255 0 0) 0x7fe75c0cd7d0 con 0x7fe75c0b26d0 2023-03-03T07:57:49.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.094+0000 7fe759d9b700 1 -- 172.21.15.98:0/1283581388 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7440035b0 con 0x7fe75c0b26d0 2023-03-03T07:57:49.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.094+0000 7fe75ad9d700 1 -- 172.21.15.98:0/1283581388 >> v1:172.21.15.100:6789/0 conn(0x7fe75c0b5310 legacy=0x7fe75c0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:49.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.094+0000 7fe75ad9d700 1 -- 172.21.15.98:0/1283581388 >> v1:172.21.15.98:6789/0 conn(0x7fe75c0b26d0 legacy=0x7fe75c0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:49.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.094+0000 7fe75ad9d700 1 -- 172.21.15.98:0/1283581388 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c0cdd00 con 0x7fe75c0b8d10 2023-03-03T07:57:49.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.094+0000 7fe75ad9d700 1 --2- 172.21.15.98:0/1283581388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe75c0b8d10 0x7fe75c0cffc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009840 tx=0x7fe74c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2139f356426d7c2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:49.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.095+0000 7fe759d9b700 1 -- 172.21.15.98:0/1283581388 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c011830 con 0x7fe75c0b8d10 2023-03-03T07:57:49.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.095+0000 7fe759d9b700 1 -- 172.21.15.98:0/1283581388 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c011990 con 0x7fe75c0b8d10 2023-03-03T07:57:49.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.095+0000 7fe759d9b700 1 -- 172.21.15.98:0/1283581388 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c011d20 con 0x7fe75c0b8d10 2023-03-03T07:57:49.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.095+0000 7fe7611ff700 1 -- 172.21.15.98:0/1283581388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe75c0b8d10 msgr2=0x7fe75c0cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:49.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.095+0000 7fe7611ff700 1 --2- 172.21.15.98:0/1283581388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe75c0b8d10 0x7fe75c0cffc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009840 tx=0x7fe74c009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:49.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.095+0000 7fe7611ff700 1 -- 172.21.15.98:0/1283581388 shutdown_connections 2023-03-03T07:57:49.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.095+0000 7fe7611ff700 1 --2- 172.21.15.98:0/1283581388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe75c0b8d10 0x7fe75c0cffc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.095+0000 7fe7611ff700 1 -- 172.21.15.98:0/1283581388 >> 172.21.15.98:0/1283581388 conn(0x7fe75c073e70 msgr2=0x7fe75c06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:49.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.095+0000 7fe7611ff700 1 -- 172.21.15.98:0/1283581388 shutdown_connections 2023-03-03T07:57:49.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.095+0000 7fe7611ff700 1 -- 172.21.15.98:0/1283581388 wait complete. 2023-03-03T07:57:49.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.097+0000 7fe7611ff700 1 Processor -- start 2023-03-03T07:57:49.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.098+0000 7fe7611ff700 1 -- start start 2023-03-03T07:57:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.098+0000 7fe7611ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe75c0b26d0 0x7fe75c0ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.098+0000 7fe7611ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75c0b5310 0x7fe75c15aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:49.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.099+0000 7fe7611ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe75c0b8d10 0x7fe75c1e75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:49.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.099+0000 7fe7611ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c15af60 con 0x7fe75c0b5310 2023-03-03T07:57:49.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.099+0000 7fe7611ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c15b9a0 con 0x7fe75c0b26d0 2023-03-03T07:57:49.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.099+0000 7fe7611ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c0cae10 con 0x7fe75c0b8d10 2023-03-03T07:57:49.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.099+0000 7fe75a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75c0b5310 0x7fe75c15aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:49.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.099+0000 7fe75a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75c0b5310 0x7fe75c15aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42468/0 (socket says 172.21.15.98:42468) 2023-03-03T07:57:49.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.099+0000 7fe75a59c700 1 -- 172.21.15.98:0/4099993226 learned_addr learned my addr 172.21.15.98:0/4099993226 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:49.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.099+0000 7fe75ad9d700 1 --2- 172.21.15.98:0/4099993226 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe75c0b26d0 0x7fe75c0ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:49.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.099+0000 7fe75b59e700 1 --2- 172.21.15.98:0/4099993226 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe75c0b8d10 0x7fe75c1e75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:49.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.099+0000 7fe75a59c700 1 -- 172.21.15.98:0/4099993226 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe75c0b8d10 msgr2=0x7fe75c1e75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:49.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.099+0000 7fe75a59c700 1 --2- 172.21.15.98:0/4099993226 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe75c0b8d10 0x7fe75c1e75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.099+0000 7fe75a59c700 1 -- 172.21.15.98:0/4099993226 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe75c0b26d0 msgr2=0x7fe75c0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:49.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.099+0000 7fe75a59c700 1 --2- 172.21.15.98:0/4099993226 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe75c0b26d0 0x7fe75c0ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.099+0000 7fe75a59c700 1 -- 172.21.15.98:0/4099993226 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c0094a0 con 0x7fe75c0b5310 2023-03-03T07:57:49.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.099+0000 7fe75ad9d700 1 --2- 172.21.15.98:0/4099993226 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe75c0b26d0 0x7fe75c0ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:49.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.100+0000 7fe75a59c700 1 --2- 172.21.15.98:0/4099993226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75c0b5310 0x7fe75c15aa20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe750001eb0 tx=0x7fe75000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:49.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.100+0000 7fe7437fe700 1 -- 172.21.15.98:0/4099993226 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750005d70 con 0x7fe75c0b5310 2023-03-03T07:57:49.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.100+0000 7fe7437fe700 1 -- 172.21.15.98:0/4099993226 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7500070d0 con 0x7fe75c0b5310 2023-03-03T07:57:49.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.100+0000 7fe7437fe700 1 -- 172.21.15.98:0/4099993226 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750007460 con 0x7fe75c0b5310 2023-03-03T07:57:49.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.100+0000 7fe7611ff700 1 -- 172.21.15.98:0/4099993226 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75c0cdfe0 con 0x7fe75c0b5310 2023-03-03T07:57:49.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.100+0000 7fe7611ff700 1 -- 172.21.15.98:0/4099993226 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75c0d0e80 con 0x7fe75c0b5310 2023-03-03T07:57:49.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.102+0000 7fe7611ff700 1 -- 172.21.15.98:0/4099993226 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75c0d0500 con 0x7fe75c0b5310 2023-03-03T07:57:49.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.103+0000 7fe7437fe700 1 -- 172.21.15.98:0/4099993226 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fe750019030 con 0x7fe75c0b5310 2023-03-03T07:57:49.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.104+0000 7fe7437fe700 1 --2- 172.21.15.98:0/4099993226 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe7480683c0 0x7fe74806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:49.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.104+0000 7fe7437fe700 1 -- 172.21.15.98:0/4099993226 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fe750085160 con 0x7fe75c0b5310 2023-03-03T07:57:49.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.104+0000 7fe75ad9d700 1 --2- 172.21.15.98:0/4099993226 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe7480683c0 0x7fe74806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:49.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.105+0000 7fe75ad9d700 1 --2- 172.21.15.98:0/4099993226 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe7480683c0 0x7fe74806a870 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe74c0090b0 tx=0x7fe74c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:49.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.108+0000 7fe7437fe700 1 -- 172.21.15.98:0/4099993226 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe7500517e0 con 0x7fe75c0b5310 2023-03-03T07:57:49.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.300+0000 7fe7611ff700 1 -- 172.21.15.98:0/4099993226 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe75c04e6e0 con 0x7fe75c0b5310 2023-03-03T07:57:49.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.301+0000 7fe7437fe700 1 -- 172.21.15.98:0/4099993226 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v40) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe750054e00 con 0x7fe75c0b5310 2023-03-03T07:57:49.303 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:57:49.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.304+0000 7fe7611ff700 1 -- 172.21.15.98:0/4099993226 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe7480683c0 msgr2=0x7fe74806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:49.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.304+0000 7fe7611ff700 1 --2- 172.21.15.98:0/4099993226 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe7480683c0 0x7fe74806a870 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe74c0090b0 tx=0x7fe74c008030 comp rx=0 tx=0).stop 2023-03-03T07:57:49.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.304+0000 7fe7611ff700 1 -- 172.21.15.98:0/4099993226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75c0b5310 msgr2=0x7fe75c15aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:49.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.304+0000 7fe7611ff700 1 --2- 172.21.15.98:0/4099993226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75c0b5310 0x7fe75c15aa20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe750001eb0 tx=0x7fe75000c7b0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.304+0000 7fe7611ff700 1 -- 172.21.15.98:0/4099993226 shutdown_connections 2023-03-03T07:57:49.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.304+0000 7fe7611ff700 1 --2- 172.21.15.98:0/4099993226 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe7480683c0 0x7fe74806a870 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.304+0000 7fe7611ff700 1 --2- 172.21.15.98:0/4099993226 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe75c0b8d10 0x7fe75c1e75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.304+0000 7fe7611ff700 1 --2- 172.21.15.98:0/4099993226 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe75c0b26d0 0x7fe75c0ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.304+0000 7fe7611ff700 1 --2- 172.21.15.98:0/4099993226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75c0b5310 0x7fe75c15aa20 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.304+0000 7fe7611ff700 1 -- 172.21.15.98:0/4099993226 >> 172.21.15.98:0/4099993226 conn(0x7fe75c073e70 msgr2=0x7fe75c15c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:49.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.304+0000 7fe7611ff700 1 -- 172.21.15.98:0/4099993226 shutdown_connections 2023-03-03T07:57:49.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.304+0000 7fe7611ff700 1 -- 172.21.15.98:0/4099993226 wait complete. 2023-03-03T07:57:49.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:57:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.584+0000 7f3959b9d700 1 Processor -- start 2023-03-03T07:57:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.585+0000 7f3959b9d700 1 -- start start 2023-03-03T07:57:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.586+0000 7f3959b9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f395415a5c0 0x7f39540ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.586+0000 7f3959b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f395415a990 0x7f39540cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:49.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.586+0000 7f3959b9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f39540d0c60 0x7f39540d1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:49.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.586+0000 7f3959b9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f395406e6d0 con 0x7f39540d0c60 2023-03-03T07:57:49.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.586+0000 7f3959b9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f395406e9b0 con 0x7f395415a5c0 2023-03-03T07:57:49.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.586+0000 7f3959b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395406ec90 con 0x7f395415a990 2023-03-03T07:57:49.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.586+0000 7f395939c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f39540d0c60 0x7f39540d1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.586+0000 7f395939c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f39540d0c60 0x7f39540d1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42476/0 (socket says 172.21.15.98:42476) 2023-03-03T07:57:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.587+0000 7f395939c700 1 -- 172.21.15.98:0/1298384149 learned_addr learned my addr 172.21.15.98:0/1298384149 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.587+0000 7f3958b9b700 1 --2- 172.21.15.98:0/1298384149 >> v2:172.21.15.100:3300/0 conn(0x7f395415a5c0 0x7f39540ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.587+0000 7f3953fff700 1 --2- 172.21.15.98:0/1298384149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f395415a990 0x7f39540cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.587+0000 7f395939c700 1 -- 172.21.15.98:0/1298384149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f395415a990 msgr2=0x7f39540cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.587+0000 7f395939c700 1 --2- 172.21.15.98:0/1298384149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f395415a990 0x7f39540cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.587+0000 7f395939c700 1 -- 172.21.15.98:0/1298384149 >> v2:172.21.15.100:3300/0 conn(0x7f395415a5c0 msgr2=0x7f39540ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.587+0000 7f395939c700 1 --2- 172.21.15.98:0/1298384149 >> v2:172.21.15.100:3300/0 conn(0x7f395415a5c0 0x7f39540ce940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.587+0000 7f395939c700 1 -- 172.21.15.98:0/1298384149 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395406ef70 con 0x7f39540d0c60 2023-03-03T07:57:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.587+0000 7f3958b9b700 1 --2- 172.21.15.98:0/1298384149 >> v2:172.21.15.100:3300/0 conn(0x7f395415a5c0 0x7f39540ce940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.587+0000 7f395939c700 1 --2- 172.21.15.98:0/1298384149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39540d0c60 0x7f39540d1080 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3954071810 tx=0x7f394c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd56bc2f1d6183a3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.588+0000 7f39537fe700 1 -- 172.21.15.98:0/1298384149 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394c00e070 con 0x7f39540d0c60 2023-03-03T07:57:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.588+0000 7f39537fe700 1 -- 172.21.15.98:0/1298384149 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f394c00b910 con 0x7f39540d0c60 2023-03-03T07:57:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.588+0000 7f39537fe700 1 -- 172.21.15.98:0/1298384149 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394c00bcc0 con 0x7f39540d0c60 2023-03-03T07:57:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.588+0000 7f3959b9d700 1 -- 172.21.15.98:0/1298384149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39540d0c60 msgr2=0x7f39540d1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.588+0000 7f3959b9d700 1 --2- 172.21.15.98:0/1298384149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39540d0c60 0x7f39540d1080 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3954071810 tx=0x7f394c00b330 comp rx=0 tx=0).stop 2023-03-03T07:57:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.588+0000 7f3959b9d700 1 -- 172.21.15.98:0/1298384149 shutdown_connections 2023-03-03T07:57:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.588+0000 7f3959b9d700 1 --2- 172.21.15.98:0/1298384149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39540d0c60 0x7f39540d1080 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.588+0000 7f3959b9d700 1 --2- 172.21.15.98:0/1298384149 >> v2:172.21.15.100:3300/0 conn(0x7f395415a5c0 0x7f39540ce940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.588+0000 7f3959b9d700 1 --2- 172.21.15.98:0/1298384149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f395415a990 0x7f39540cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.588+0000 7f3959b9d700 1 -- 172.21.15.98:0/1298384149 >> 172.21.15.98:0/1298384149 conn(0x7f395409a0e0 msgr2=0x7f3954073b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:49.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.589+0000 7f3959b9d700 1 -- 172.21.15.98:0/1298384149 shutdown_connections 2023-03-03T07:57:49.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.589+0000 7f3959b9d700 1 -- 172.21.15.98:0/1298384149 wait complete. 2023-03-03T07:57:49.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.591+0000 7f3959b9d700 1 Processor -- start 2023-03-03T07:57:49.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.591+0000 7f3959b9d700 1 -- start start 2023-03-03T07:57:49.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.592+0000 7f3959b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39540d0c60 0x7f39540cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:49.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.592+0000 7f3959b9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f395415a5c0 0x7f395406bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:49.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.592+0000 7f3959b9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f395415a990 0x7f39540b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:49.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.592+0000 7f3959b9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39540ceb90 con 0x7f395415a5c0 2023-03-03T07:57:49.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.592+0000 7f3959b9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395406c400 con 0x7f395415a990 2023-03-03T07:57:49.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.592+0000 7f3959b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39540cdf10 con 0x7f39540d0c60 2023-03-03T07:57:49.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.592+0000 7f3953fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f395415a5c0 0x7f395406bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:49.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.592+0000 7f3953fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f395415a5c0 0x7f395406bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42488/0 (socket says 172.21.15.98:42488) 2023-03-03T07:57:49.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.592+0000 7f3953fff700 1 -- 172.21.15.98:0/3563952096 learned_addr learned my addr 172.21.15.98:0/3563952096 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:49.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.592+0000 7f3958b9b700 1 --2- 172.21.15.98:0/3563952096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39540d0c60 0x7f39540cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:49.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.592+0000 7f395939c700 1 --2- 172.21.15.98:0/3563952096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f395415a990 0x7f39540b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:49.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.593+0000 7f3953fff700 1 -- 172.21.15.98:0/3563952096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39540d0c60 msgr2=0x7f39540cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:49.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.593+0000 7f3953fff700 1 --2- 172.21.15.98:0/3563952096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39540d0c60 0x7f39540cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.593+0000 7f3953fff700 1 -- 172.21.15.98:0/3563952096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f395415a990 msgr2=0x7f39540b5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:49.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.593+0000 7f3953fff700 1 --2- 172.21.15.98:0/3563952096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f395415a990 0x7f39540b5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.593+0000 7f3953fff700 1 -- 172.21.15.98:0/3563952096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39480094a0 con 0x7f395415a5c0 2023-03-03T07:57:49.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.593+0000 7f3953fff700 1 --2- 172.21.15.98:0/3563952096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f395415a5c0 0x7f395406bec0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f394000e850 tx=0x7f394000eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:49.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.593+0000 7f39517fa700 1 -- 172.21.15.98:0/3563952096 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39400095b0 con 0x7f395415a5c0 2023-03-03T07:57:49.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.593+0000 7f39517fa700 1 -- 172.21.15.98:0/3563952096 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3940013680 con 0x7f395415a5c0 2023-03-03T07:57:49.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.593+0000 7f39517fa700 1 -- 172.21.15.98:0/3563952096 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3940013a10 con 0x7f395415a5c0 2023-03-03T07:57:49.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.593+0000 7f3959b9d700 1 -- 172.21.15.98:0/3563952096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394c009b30 con 0x7f395415a5c0 2023-03-03T07:57:49.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.594+0000 7f3959b9d700 1 -- 172.21.15.98:0/3563952096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395406f770 con 0x7f395415a5c0 2023-03-03T07:57:49.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.595+0000 7f3959b9d700 1 -- 172.21.15.98:0/3563952096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f395406f250 con 0x7f395415a5c0 2023-03-03T07:57:49.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.596+0000 7f39517fa700 1 -- 172.21.15.98:0/3563952096 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f3940019030 con 0x7f395415a5c0 2023-03-03T07:57:49.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.598+0000 7f39517fa700 1 --2- 172.21.15.98:0/3563952096 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39380683c0 0x7f393806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:49.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.598+0000 7f39517fa700 1 -- 172.21.15.98:0/3563952096 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f3940085be0 con 0x7f395415a5c0 2023-03-03T07:57:49.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.598+0000 7f3958b9b700 1 --2- 172.21.15.98:0/3563952096 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39380683c0 0x7f393806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:49.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.599+0000 7f3958b9b700 1 --2- 172.21.15.98:0/3563952096 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39380683c0 0x7f393806a870 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3948009470 tx=0x7f3948009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:49.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.602+0000 7f39517fa700 1 -- 172.21.15.98:0/3563952096 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3940052200 con 0x7f395415a5c0 2023-03-03T07:57:49.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.803+0000 7f3959b9d700 1 -- 172.21.15.98:0/3563952096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f395406f900 con 0x7f395415a5c0 2023-03-03T07:57:49.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.804+0000 7f39517fa700 1 -- 172.21.15.98:0/3563952096 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v40) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3940055820 con 0x7f395415a5c0 2023-03-03T07:57:49.806 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:57:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.806+0000 7f3959b9d700 1 -- 172.21.15.98:0/3563952096 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39380683c0 msgr2=0x7f393806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.806+0000 7f3959b9d700 1 --2- 172.21.15.98:0/3563952096 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39380683c0 0x7f393806a870 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3948009470 tx=0x7f3948009210 comp rx=0 tx=0).stop 2023-03-03T07:57:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.806+0000 7f3959b9d700 1 -- 172.21.15.98:0/3563952096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f395415a5c0 msgr2=0x7f395406bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.806+0000 7f3959b9d700 1 --2- 172.21.15.98:0/3563952096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f395415a5c0 0x7f395406bec0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f394000e850 tx=0x7f394000eb60 comp rx=0 tx=0).stop 2023-03-03T07:57:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.806+0000 7f3959b9d700 1 -- 172.21.15.98:0/3563952096 shutdown_connections 2023-03-03T07:57:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.806+0000 7f3959b9d700 1 --2- 172.21.15.98:0/3563952096 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f39380683c0 0x7f393806a870 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.806+0000 7f3959b9d700 1 --2- 172.21.15.98:0/3563952096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f395415a990 0x7f39540b5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.806+0000 7f3959b9d700 1 --2- 172.21.15.98:0/3563952096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39540d0c60 0x7f39540cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.806+0000 7f3959b9d700 1 --2- 172.21.15.98:0/3563952096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f395415a5c0 0x7f395406bec0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.806+0000 7f3959b9d700 1 -- 172.21.15.98:0/3563952096 >> 172.21.15.98:0/3563952096 conn(0x7f395409a0e0 msgr2=0x7f39540af8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.807+0000 7f3959b9d700 1 -- 172.21.15.98:0/3563952096 shutdown_connections 2023-03-03T07:57:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:49.807+0000 7f3959b9d700 1 -- 172.21.15.98:0/3563952096 wait complete. 2023-03-03T07:57:49.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:57:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.084+0000 7f293d7ff700 1 Processor -- start 2023-03-03T07:57:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.084+0000 7f293d7ff700 1 -- start start 2023-03-03T07:57:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.085+0000 7f293d7ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f293806b4b0 0x7f29380cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.085+0000 7f293d7ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29380cf980 con 0x7f29380d0270 2023-03-03T07:57:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.085+0000 7f293d7ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29380d1800 con 0x7f293815ca70 2023-03-03T07:57:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.086+0000 7f293d7ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29380cd5b0 con 0x7f293806b4b0 2023-03-03T07:57:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.086+0000 7f29367fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f29380d0270 0x7f29380d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44230/0 (socket says 172.21.15.98:44230) 2023-03-03T07:57:50.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.086+0000 7f29367fc700 1 -- 172.21.15.98:0/814988296 learned_addr learned my addr 172.21.15.98:0/814988296 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:50.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.086+0000 7f29377fe700 1 --2- 172.21.15.98:0/814988296 >> v2:172.21.15.98:3300/0 conn(0x7f293806b4b0 0x7f29380cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:50.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.086+0000 7f2935ffb700 1 -- 172.21.15.98:0/814988296 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288889614 0 0) 0x7f29380cf980 con 0x7f29380d0270 2023-03-03T07:57:50.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.086+0000 7f2935ffb700 1 -- 172.21.15.98:0/814988296 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29200035b0 con 0x7f29380d0270 2023-03-03T07:57:50.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.086+0000 7f29377fe700 1 -- 172.21.15.98:0/814988296 >> v1:172.21.15.100:6789/0 conn(0x7f293815ca70 legacy=0x7f29380cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:50.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.086+0000 7f29377fe700 1 -- 172.21.15.98:0/814988296 >> v1:172.21.15.98:6789/0 conn(0x7f29380d0270 legacy=0x7f29380d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:50.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.086+0000 7f29377fe700 1 -- 172.21.15.98:0/814988296 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29380cd900 con 0x7f293806b4b0 2023-03-03T07:57:50.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.087+0000 7f29377fe700 1 --2- 172.21.15.98:0/814988296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f293806b4b0 0x7f29380cf440 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2930009d90 tx=0x7f293000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7cbc87f4884b19c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:50.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.087+0000 7f2935ffb700 1 -- 172.21.15.98:0/814988296 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293000e070 con 0x7f293806b4b0 2023-03-03T07:57:50.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.087+0000 7f2935ffb700 1 -- 172.21.15.98:0/814988296 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293000bb20 con 0x7f293806b4b0 2023-03-03T07:57:50.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.087+0000 7f2935ffb700 1 -- 172.21.15.98:0/814988296 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2930012430 con 0x7f293806b4b0 2023-03-03T07:57:50.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.088+0000 7f293d7ff700 1 -- 172.21.15.98:0/814988296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f293806b4b0 msgr2=0x7f29380cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:50.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.088+0000 7f293d7ff700 1 --2- 172.21.15.98:0/814988296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f293806b4b0 0x7f29380cf440 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2930009d90 tx=0x7f293000b330 comp rx=0 tx=0).stop 2023-03-03T07:57:50.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.088+0000 7f293d7ff700 1 -- 172.21.15.98:0/814988296 shutdown_connections 2023-03-03T07:57:50.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.088+0000 7f293d7ff700 1 --2- 172.21.15.98:0/814988296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f293806b4b0 0x7f29380cf440 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.088+0000 7f293d7ff700 1 -- 172.21.15.98:0/814988296 >> 172.21.15.98:0/814988296 conn(0x7f2938073e70 msgr2=0x7f293806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:50.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.088+0000 7f293d7ff700 1 -- 172.21.15.98:0/814988296 shutdown_connections 2023-03-03T07:57:50.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.088+0000 7f293d7ff700 1 -- 172.21.15.98:0/814988296 wait complete. 2023-03-03T07:57:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.090+0000 7f293d7ff700 1 Processor -- start 2023-03-03T07:57:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.090+0000 7f293d7ff700 1 -- start start 2023-03-03T07:57:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.091+0000 7f293d7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293806b4b0 0x7f293815fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.091+0000 7f293d7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29380d0270 0x7f29380b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.091+0000 7f293d7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f293815ca70 0x7f293814e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.091+0000 7f293d7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293815f690 con 0x7f293815ca70 2023-03-03T07:57:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.091+0000 7f293d7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293815fff0 con 0x7f29380d0270 2023-03-03T07:57:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.091+0000 7f293d7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293814efb0 con 0x7f293806b4b0 2023-03-03T07:57:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.091+0000 7f29377fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f293815ca70 0x7f293814e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.091+0000 7f29377fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f293815ca70 0x7f293814e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42504/0 (socket says 172.21.15.98:42504) 2023-03-03T07:57:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.091+0000 7f29377fe700 1 -- 172.21.15.98:0/3163351801 learned_addr learned my addr 172.21.15.98:0/3163351801 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.092+0000 7f29367fc700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29380d0270 0x7f29380b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.092+0000 7f2936ffd700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293806b4b0 0x7f293815fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.092+0000 7f29377fe700 1 -- 172.21.15.98:0/3163351801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293806b4b0 msgr2=0x7f293815fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.092+0000 7f29377fe700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293806b4b0 0x7f293815fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.092+0000 7f29377fe700 1 -- 172.21.15.98:0/3163351801 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29380d0270 msgr2=0x7f29380b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.092+0000 7f29377fe700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29380d0270 0x7f29380b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.092+0000 7f29377fe700 1 -- 172.21.15.98:0/3163351801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2930009a70 con 0x7f293815ca70 2023-03-03T07:57:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.092+0000 7f29367fc700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29380d0270 0x7f29380b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:57:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.092+0000 7f2936ffd700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293806b4b0 0x7f293815fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:57:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.092+0000 7f29377fe700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f293815ca70 0x7f293814e430 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f29380b5d70 tx=0x7f2930004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.092+0000 7f291f7fe700 1 -- 172.21.15.98:0/3163351801 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2930012430 con 0x7f293815ca70 2023-03-03T07:57:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.093+0000 7f291f7fe700 1 -- 172.21.15.98:0/3163351801 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2930020070 con 0x7f293815ca70 2023-03-03T07:57:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.093+0000 7f293d7ff700 1 -- 172.21.15.98:0/3163351801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29380d1aa0 con 0x7f293815ca70 2023-03-03T07:57:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.093+0000 7f291f7fe700 1 -- 172.21.15.98:0/3163351801 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2930004280 con 0x7f293815ca70 2023-03-03T07:57:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.093+0000 7f293d7ff700 1 -- 172.21.15.98:0/3163351801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29380cdbe0 con 0x7f293815ca70 2023-03-03T07:57:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.094+0000 7f293d7ff700 1 -- 172.21.15.98:0/3163351801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293815ce40 con 0x7f293815ca70 2023-03-03T07:57:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.095+0000 7f291f7fe700 1 -- 172.21.15.98:0/3163351801 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f29300043e0 con 0x7f293815ca70 2023-03-03T07:57:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.097+0000 7f291f7fe700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f29240683c0 0x7f292406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.097+0000 7f291f7fe700 1 -- 172.21.15.98:0/3163351801 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f2930022070 con 0x7f293815ca70 2023-03-03T07:57:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.097+0000 7f2936ffd700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f29240683c0 0x7f292406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:50.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.098+0000 7f2936ffd700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f29240683c0 0x7f292406a870 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f29380d2350 tx=0x7f2928009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:50.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.101+0000 7f291f7fe700 1 -- 172.21.15.98:0/3163351801 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2930054930 con 0x7f293815ca70 2023-03-03T07:57:50.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.296+0000 7f293d7ff700 1 -- 172.21.15.98:0/3163351801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f293804e6e0 con 0x7f293815ca70 2023-03-03T07:57:50.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.296+0000 7f291f7fe700 1 -- 172.21.15.98:0/3163351801 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v40) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2930057f50 con 0x7f293815ca70 2023-03-03T07:57:50.299 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:57:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.299+0000 7f293d7ff700 1 -- 172.21.15.98:0/3163351801 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f29240683c0 msgr2=0x7f292406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.299+0000 7f293d7ff700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f29240683c0 0x7f292406a870 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f29380d2350 tx=0x7f2928009040 comp rx=0 tx=0).stop 2023-03-03T07:57:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.299+0000 7f293d7ff700 1 -- 172.21.15.98:0/3163351801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f293815ca70 msgr2=0x7f293814e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.299+0000 7f293d7ff700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f293815ca70 0x7f293814e430 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f29380b5d70 tx=0x7f2930004ad0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.299+0000 7f293d7ff700 1 -- 172.21.15.98:0/3163351801 shutdown_connections 2023-03-03T07:57:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.299+0000 7f293d7ff700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f29240683c0 0x7f292406a870 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.299+0000 7f293d7ff700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f293815ca70 0x7f293814e430 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.299+0000 7f293d7ff700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293806b4b0 0x7f293815fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.299+0000 7f293d7ff700 1 --2- 172.21.15.98:0/3163351801 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29380d0270 0x7f29380b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.299+0000 7f293d7ff700 1 -- 172.21.15.98:0/3163351801 >> 172.21.15.98:0/3163351801 conn(0x7f2938073e70 msgr2=0x7f29380d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:50.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.299+0000 7f293d7ff700 1 -- 172.21.15.98:0/3163351801 shutdown_connections 2023-03-03T07:57:50.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.299+0000 7f293d7ff700 1 -- 172.21.15.98:0/3163351801 wait complete. 2023-03-03T07:57:50.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T07:57:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.573+0000 7f10c41ff700 1 Processor -- start 2023-03-03T07:57:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.573+0000 7f10c41ff700 1 -- start start 2023-03-03T07:57:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.574+0000 7f10c41ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10bc15ca70 0x7f10bc0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.575+0000 7f10c41ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f10bc0d0270 0x7f10bc0d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.575+0000 7f10c41ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10bc0cf980 con 0x7f10bc06b4b0 2023-03-03T07:57:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.575+0000 7f10c41ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10bc0cd5b0 con 0x7f10bc0d0270 2023-03-03T07:57:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.575+0000 7f10c41ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0d1800 con 0x7f10bc15ca70 2023-03-03T07:57:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.575+0000 7f10c179a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f10bc06b4b0 0x7f10bc0cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44232/0 (socket says 172.21.15.98:44232) 2023-03-03T07:57:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.575+0000 7f10c179a700 1 -- 172.21.15.98:0/2048636622 learned_addr learned my addr 172.21.15.98:0/2048636622 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.575+0000 7f10c1f9b700 1 --2- 172.21.15.98:0/2048636622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10bc15ca70 0x7f10bc0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.575+0000 7f10c279c700 1 --2- 172.21.15.98:0/2048636622 >> v2:172.21.15.100:3300/0 conn(0x7f10bc0d0270 0x7f10bc0d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.575+0000 7f10c0f99700 1 -- 172.21.15.98:0/2048636622 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328434969 0 0) 0x7f10bc0cf980 con 0x7f10bc06b4b0 2023-03-03T07:57:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.576+0000 7f10c0f99700 1 -- 172.21.15.98:0/2048636622 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a80035b0 con 0x7f10bc06b4b0 2023-03-03T07:57:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.576+0000 7f10c1f9b700 1 -- 172.21.15.98:0/2048636622 >> v2:172.21.15.100:3300/0 conn(0x7f10bc0d0270 msgr2=0x7f10bc0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.576+0000 7f10c1f9b700 1 --2- 172.21.15.98:0/2048636622 >> v2:172.21.15.100:3300/0 conn(0x7f10bc0d0270 0x7f10bc0d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.576+0000 7f10c1f9b700 1 -- 172.21.15.98:0/2048636622 >> v1:172.21.15.98:6789/0 conn(0x7f10bc06b4b0 legacy=0x7f10bc0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.576+0000 7f10c1f9b700 1 -- 172.21.15.98:0/2048636622 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10bc0cd900 con 0x7f10bc15ca70 2023-03-03T07:57:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.576+0000 7f10c1f9b700 1 --2- 172.21.15.98:0/2048636622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10bc15ca70 0x7f10bc0cf440 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f10b8009840 tx=0x7f10b8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bece0cbe9820f53 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.576+0000 7f10c0f99700 1 -- 172.21.15.98:0/2048636622 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b8011830 con 0x7f10bc15ca70 2023-03-03T07:57:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.576+0000 7f10c0f99700 1 -- 172.21.15.98:0/2048636622 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b8011990 con 0x7f10bc15ca70 2023-03-03T07:57:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.576+0000 7f10c0f99700 1 -- 172.21.15.98:0/2048636622 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b8011d20 con 0x7f10bc15ca70 2023-03-03T07:57:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.577+0000 7f10c41ff700 1 -- 172.21.15.98:0/2048636622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10bc15ca70 msgr2=0x7f10bc0cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.577+0000 7f10c41ff700 1 --2- 172.21.15.98:0/2048636622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10bc15ca70 0x7f10bc0cf440 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f10b8009840 tx=0x7f10b8009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.577+0000 7f10c41ff700 1 -- 172.21.15.98:0/2048636622 shutdown_connections 2023-03-03T07:57:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.577+0000 7f10c41ff700 1 --2- 172.21.15.98:0/2048636622 >> v2:172.21.15.100:3300/0 conn(0x7f10bc0d0270 0x7f10bc0d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.577+0000 7f10c41ff700 1 --2- 172.21.15.98:0/2048636622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10bc15ca70 0x7f10bc0cf440 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.577+0000 7f10c41ff700 1 -- 172.21.15.98:0/2048636622 >> 172.21.15.98:0/2048636622 conn(0x7f10bc073e70 msgr2=0x7f10bc06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.577+0000 7f10c41ff700 1 -- 172.21.15.98:0/2048636622 shutdown_connections 2023-03-03T07:57:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.577+0000 7f10c41ff700 1 -- 172.21.15.98:0/2048636622 wait complete. 2023-03-03T07:57:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.579+0000 7f10c41ff700 1 Processor -- start 2023-03-03T07:57:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.579+0000 7f10c41ff700 1 -- start start 2023-03-03T07:57:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.580+0000 7f10c41ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10bc06b4b0 0x7f10bc0c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.580+0000 7f10c41ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10bc0d0270 0x7f10bc1f1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:50.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.580+0000 7f10c41ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10bc15ca70 0x7f10bc0cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:50.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.580+0000 7f10c41ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0cb8b0 con 0x7f10bc0d0270 2023-03-03T07:57:50.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.580+0000 7f10c41ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0b3360 con 0x7f10bc15ca70 2023-03-03T07:57:50.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.580+0000 7f10c41ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0c82c0 con 0x7f10bc06b4b0 2023-03-03T07:57:50.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.581+0000 7f10c179a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10bc0d0270 0x7f10bc1f1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:50.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.581+0000 7f10c179a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10bc0d0270 0x7f10bc1f1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42520/0 (socket says 172.21.15.98:42520) 2023-03-03T07:57:50.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.581+0000 7f10c179a700 1 -- 172.21.15.98:0/2825323658 learned_addr learned my addr 172.21.15.98:0/2825323658 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:50.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.581+0000 7f10c1f9b700 1 --2- 172.21.15.98:0/2825323658 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10bc06b4b0 0x7f10bc0c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:50.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.581+0000 7f10c279c700 1 --2- 172.21.15.98:0/2825323658 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10bc15ca70 0x7f10bc0cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:50.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.581+0000 7f10c179a700 1 -- 172.21.15.98:0/2825323658 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10bc06b4b0 msgr2=0x7f10bc0c7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:50.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.581+0000 7f10c179a700 1 --2- 172.21.15.98:0/2825323658 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10bc06b4b0 0x7f10bc0c7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.581+0000 7f10c179a700 1 -- 172.21.15.98:0/2825323658 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10bc15ca70 msgr2=0x7f10bc0cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:50.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.581+0000 7f10c179a700 1 --2- 172.21.15.98:0/2825323658 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10bc15ca70 0x7f10bc0cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.581+0000 7f10c179a700 1 -- 172.21.15.98:0/2825323658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b80094a0 con 0x7f10bc0d0270 2023-03-03T07:57:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.582+0000 7f10c179a700 1 --2- 172.21.15.98:0/2825323658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10bc0d0270 0x7f10bc1f1ac0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00c430 tx=0x7f10ac00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.582+0000 7f10b27fc700 1 -- 172.21.15.98:0/2825323658 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac005b70 con 0x7f10bc0d0270 2023-03-03T07:57:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.582+0000 7f10b27fc700 1 -- 172.21.15.98:0/2825323658 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10ac005cf0 con 0x7f10bc0d0270 2023-03-03T07:57:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.582+0000 7f10b27fc700 1 -- 172.21.15.98:0/2825323658 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac007310 con 0x7f10bc0d0270 2023-03-03T07:57:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.582+0000 7f10c41ff700 1 -- 172.21.15.98:0/2825323658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10bc0d1aa0 con 0x7f10bc0d0270 2023-03-03T07:57:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.582+0000 7f10c41ff700 1 -- 172.21.15.98:0/2825323658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10bc0cdbe0 con 0x7f10bc0d0270 2023-03-03T07:57:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.583+0000 7f10c41ff700 1 -- 172.21.15.98:0/2825323658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10bc15ce40 con 0x7f10bc0d0270 2023-03-03T07:57:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.584+0000 7f10b27fc700 1 -- 172.21.15.98:0/2825323658 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f10ac019030 con 0x7f10bc0d0270 2023-03-03T07:57:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.586+0000 7f10b27fc700 1 --2- 172.21.15.98:0/2825323658 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f10a00683c0 0x7f10a006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.586+0000 7f10b27fc700 1 -- 172.21.15.98:0/2825323658 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f10ac0850d0 con 0x7f10bc0d0270 2023-03-03T07:57:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.586+0000 7f10c1f9b700 1 --2- 172.21.15.98:0/2825323658 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f10a00683c0 0x7f10a006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.587+0000 7f10c1f9b700 1 --2- 172.21.15.98:0/2825323658 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f10a00683c0 0x7f10a006a870 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f10b8000c00 tx=0x7f10b8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:50.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.589+0000 7f10b27fc700 1 -- 172.21.15.98:0/2825323658 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f10ac051810 con 0x7f10bc0d0270 2023-03-03T07:57:50.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.799+0000 7f10c41ff700 1 -- 172.21.15.98:0/2825323658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f10bc0b3500 con 0x7f10bc0d0270 2023-03-03T07:57:50.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.800+0000 7f10b27fc700 1 -- 172.21.15.98:0/2825323658 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v40) v1 ==== 80+0+0 (secure 0 0 0) 0x7f10ac017080 con 0x7f10bc0d0270 2023-03-03T07:57:50.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.802+0000 7f10c41ff700 1 -- 172.21.15.98:0/2825323658 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f10a00683c0 msgr2=0x7f10a006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:50.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.802+0000 7f10c41ff700 1 --2- 172.21.15.98:0/2825323658 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f10a00683c0 0x7f10a006a870 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f10b8000c00 tx=0x7f10b8008030 comp rx=0 tx=0).stop 2023-03-03T07:57:50.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.802+0000 7f10c41ff700 1 -- 172.21.15.98:0/2825323658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10bc0d0270 msgr2=0x7f10bc1f1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:50.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.802+0000 7f10c41ff700 1 --2- 172.21.15.98:0/2825323658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10bc0d0270 0x7f10bc1f1ac0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00c430 tx=0x7f10ac00c740 comp rx=0 tx=0).stop 2023-03-03T07:57:50.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.802+0000 7f10c41ff700 1 -- 172.21.15.98:0/2825323658 shutdown_connections 2023-03-03T07:57:50.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.802+0000 7f10c41ff700 1 --2- 172.21.15.98:0/2825323658 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f10a00683c0 0x7f10a006a870 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.802+0000 7f10c41ff700 1 --2- 172.21.15.98:0/2825323658 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10bc15ca70 0x7f10bc0cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.802+0000 7f10c41ff700 1 --2- 172.21.15.98:0/2825323658 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10bc06b4b0 0x7f10bc0c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.802+0000 7f10c41ff700 1 --2- 172.21.15.98:0/2825323658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10bc0d0270 0x7f10bc1f1ac0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:50.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.802+0000 7f10c41ff700 1 -- 172.21.15.98:0/2825323658 >> 172.21.15.98:0/2825323658 conn(0x7f10bc073e70 msgr2=0x7f10bc0d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:50.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.802+0000 7f10c41ff700 1 -- 172.21.15.98:0/2825323658 shutdown_connections 2023-03-03T07:57:50.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:50.802+0000 7f10c41ff700 1 -- 172.21.15.98:0/2825323658 wait complete. 2023-03-03T07:57:50.807 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T07:57:50.832 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2023-03-03T07:57:50.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:50.833 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-03-03T07:57:50.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[117008]: fuse finished with error 0 and tester_r 0 2023-03-03T07:57:50.942 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T07:57:50.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:50.944 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-03-03T07:57:51.134 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:57:51.134 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:57:51.204 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T07:57:51.204 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T07:57:51.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:51.205 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-03-03T07:57:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-03-03T07:57:51.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T07:57:51.275 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:57:51.276 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:57:51.343 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T07:57:51.344 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T07:57:51.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:57:51.344 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-03-03T07:57:51.411 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-03-03T07:57:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T07:57:51.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:57:51.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.658+0000 7f540899d700 1 Processor -- start 2023-03-03T07:57:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.658+0000 7f540899d700 1 -- start start 2023-03-03T07:57:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.660+0000 7f540899d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f54040b50d0 0x7f54040cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.660+0000 7f540899d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5404160fa0 0x7f54040d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.660+0000 7f540899d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5404161480 con 0x7f54041461f0 2023-03-03T07:57:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.660+0000 7f540899d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54040d0cb0 con 0x7f5404160fa0 2023-03-03T07:57:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.660+0000 7f540899d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540415cf30 con 0x7f54040b50d0 2023-03-03T07:57:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.660+0000 7f54037fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f54041461f0 0x7f54040b49c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44242/0 (socket says 172.21.15.98:44242) 2023-03-03T07:57:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.660+0000 7f54037fe700 1 -- 172.21.15.98:0/3542481236 learned_addr learned my addr 172.21.15.98:0/3542481236 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.660+0000 7f5403fff700 1 --2- 172.21.15.98:0/3542481236 >> v2:172.21.15.98:3300/0 conn(0x7f5404160fa0 0x7f54040d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.661+0000 7f5402ffd700 1 --2- 172.21.15.98:0/3542481236 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f54040b50d0 0x7f54040cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.661+0000 7f54027fc700 1 -- 172.21.15.98:0/3542481236 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022176335 0 0) 0x7f5404161480 con 0x7f54041461f0 2023-03-03T07:57:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.661+0000 7f54027fc700 1 -- 172.21.15.98:0/3542481236 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53ec0035b0 con 0x7f54041461f0 2023-03-03T07:57:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.661+0000 7f5403fff700 1 -- 172.21.15.98:0/3542481236 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f54040b50d0 msgr2=0x7f54040cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.661+0000 7f5403fff700 1 --2- 172.21.15.98:0/3542481236 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f54040b50d0 0x7f54040cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.661+0000 7f5403fff700 1 -- 172.21.15.98:0/3542481236 >> v1:172.21.15.98:6789/0 conn(0x7f54041461f0 legacy=0x7f54040b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.661+0000 7f5403fff700 1 -- 172.21.15.98:0/3542481236 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54040cd6c0 con 0x7f5404160fa0 2023-03-03T07:57:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.661+0000 7f5403fff700 1 --2- 172.21.15.98:0/3542481236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5404160fa0 0x7f54040d10d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f53fc009d90 tx=0x7f53fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebd8f3e631ad1f0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.661+0000 7f54027fc700 1 -- 172.21.15.98:0/3542481236 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc00e070 con 0x7f5404160fa0 2023-03-03T07:57:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.662+0000 7f54027fc700 1 -- 172.21.15.98:0/3542481236 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc00bb20 con 0x7f5404160fa0 2023-03-03T07:57:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.662+0000 7f54027fc700 1 -- 172.21.15.98:0/3542481236 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc012430 con 0x7f5404160fa0 2023-03-03T07:57:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.662+0000 7f540899d700 1 -- 172.21.15.98:0/3542481236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5404160fa0 msgr2=0x7f54040d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.662+0000 7f540899d700 1 --2- 172.21.15.98:0/3542481236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5404160fa0 0x7f54040d10d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f53fc009d90 tx=0x7f53fc00b330 comp rx=0 tx=0).stop 2023-03-03T07:57:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.662+0000 7f540899d700 1 -- 172.21.15.98:0/3542481236 shutdown_connections 2023-03-03T07:57:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.662+0000 7f540899d700 1 --2- 172.21.15.98:0/3542481236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5404160fa0 0x7f54040d10d0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.662+0000 7f540899d700 1 --2- 172.21.15.98:0/3542481236 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f54040b50d0 0x7f54040cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.662+0000 7f540899d700 1 -- 172.21.15.98:0/3542481236 >> 172.21.15.98:0/3542481236 conn(0x7f540406b900 msgr2=0x7f54040cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.662+0000 7f540899d700 1 -- 172.21.15.98:0/3542481236 shutdown_connections 2023-03-03T07:57:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.662+0000 7f540899d700 1 -- 172.21.15.98:0/3542481236 wait complete. 2023-03-03T07:57:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.664+0000 7f540899d700 1 Processor -- start 2023-03-03T07:57:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.665+0000 7f540899d700 1 -- start start 2023-03-03T07:57:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.666+0000 7f540899d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f54040b50d0 0x7f54040b8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.666+0000 7f540899d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54041461f0 0x7f5404155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.666+0000 7f540899d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5404160fa0 0x7f54041f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.666+0000 7f540899d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54041f64b0 con 0x7f5404160fa0 2023-03-03T07:57:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.666+0000 7f540899d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54041f6790 con 0x7f54041461f0 2023-03-03T07:57:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.666+0000 7f540899d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54041f6a70 con 0x7f54040b50d0 2023-03-03T07:57:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.666+0000 7f5402ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54041461f0 0x7f5404155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.666+0000 7f54037fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f54040b50d0 0x7f54040b8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.666+0000 7f5402ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54041461f0 0x7f5404155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:32788/0 (socket says 172.21.15.98:32788) 2023-03-03T07:57:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.666+0000 7f54037fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f54040b50d0 0x7f54040b8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:49720/0 (socket says 172.21.15.98:49720) 2023-03-03T07:57:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.666+0000 7f5402ffd700 1 -- 172.21.15.98:0/3453055757 learned_addr learned my addr 172.21.15.98:0/3453055757 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.666+0000 7f5403fff700 1 --2- 172.21.15.98:0/3453055757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5404160fa0 0x7f54041f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.667+0000 7f5402ffd700 1 -- 172.21.15.98:0/3453055757 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f54040b50d0 msgr2=0x7f54040b8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.667+0000 7f5402ffd700 1 --2- 172.21.15.98:0/3453055757 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f54040b50d0 0x7f54040b8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.667+0000 7f5402ffd700 1 -- 172.21.15.98:0/3453055757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5404160fa0 msgr2=0x7f54041f5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.667+0000 7f5402ffd700 1 --2- 172.21.15.98:0/3453055757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5404160fa0 0x7f54041f5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.667+0000 7f5402ffd700 1 -- 172.21.15.98:0/3453055757 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc009a70 con 0x7f54041461f0 2023-03-03T07:57:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.667+0000 7f54037fe700 1 --2- 172.21.15.98:0/3453055757 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f54040b50d0 0x7f54040b8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.667+0000 7f5402ffd700 1 --2- 172.21.15.98:0/3453055757 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54041461f0 0x7f5404155fc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f53f800e840 tx=0x7f53f800eb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.667+0000 7f53ebfff700 1 -- 172.21.15.98:0/3453055757 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f8009590 con 0x7f54041461f0 2023-03-03T07:57:51.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.667+0000 7f53ebfff700 1 -- 172.21.15.98:0/3453055757 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f8013680 con 0x7f54041461f0 2023-03-03T07:57:51.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.667+0000 7f540899d700 1 -- 172.21.15.98:0/3453055757 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54040cdc80 con 0x7f54041461f0 2023-03-03T07:57:51.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.667+0000 7f53ebfff700 1 -- 172.21.15.98:0/3453055757 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f8013a30 con 0x7f54041461f0 2023-03-03T07:57:51.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.668+0000 7f540899d700 1 -- 172.21.15.98:0/3453055757 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54040cd9a0 con 0x7f54041461f0 2023-03-03T07:57:51.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.669+0000 7f540899d700 1 -- 172.21.15.98:0/3453055757 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54041465c0 con 0x7f54041461f0 2023-03-03T07:57:51.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.670+0000 7f53ebfff700 1 -- 172.21.15.98:0/3453055757 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f53f8019030 con 0x7f54041461f0 2023-03-03T07:57:51.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.671+0000 7f53ebfff700 1 --2- 172.21.15.98:0/3453055757 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f53f00683c0 0x7f53f006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:51.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.671+0000 7f53ebfff700 1 -- 172.21.15.98:0/3453055757 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f53f8085c30 con 0x7f54041461f0 2023-03-03T07:57:51.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.671+0000 7f54037fe700 1 --2- 172.21.15.98:0/3453055757 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f53f00683c0 0x7f53f006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:51.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.672+0000 7f54037fe700 1 --2- 172.21.15.98:0/3453055757 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f53f00683c0 0x7f53f006a870 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f53f4007fd0 tx=0x7f53f4007dc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.676+0000 7f53ebfff700 1 -- 172.21.15.98:0/3453055757 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f53f8052230 con 0x7f54041461f0 2023-03-03T07:57:51.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.914+0000 7f540899d700 1 -- 172.21.15.98:0/3453055757 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5404068290 con 0x7f54041461f0 2023-03-03T07:57:51.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.915+0000 7f53ebfff700 1 -- 172.21.15.98:0/3453055757 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f53f8055850 con 0x7f54041461f0 2023-03-03T07:57:51.917 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:51.918 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3849166495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3849166495},{"type":"v1","addr":"172.21.15.100:6834","nonce":3849166495}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/4285491249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":4285491249},{"type":"v1","addr":"172.21.15.100:6835","nonce":4285491249}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":5208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3888598890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3888598890},{"type":"v1","addr":"172.21.15.98:6836","nonce":3888598890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:57:17.291546+0000","modified":"2023-03-03T07:57:20.741013+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5227},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5227":{"gid":5227,"name":"c","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.98:6837/1163251191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1163251191},{"type":"v1","addr":"172.21.15.98:6837","nonce":1163251191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T07:57:51.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.917+0000 7f540899d700 1 -- 172.21.15.98:0/3453055757 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f53f00683c0 msgr2=0x7f53f006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:51.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.917+0000 7f540899d700 1 --2- 172.21.15.98:0/3453055757 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f53f00683c0 0x7f53f006a870 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f53f4007fd0 tx=0x7f53f4007dc0 comp rx=0 tx=0).stop 2023-03-03T07:57:51.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.917+0000 7f540899d700 1 -- 172.21.15.98:0/3453055757 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54041461f0 msgr2=0x7f5404155fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:51.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.917+0000 7f540899d700 1 --2- 172.21.15.98:0/3453055757 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54041461f0 0x7f5404155fc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f53f800e840 tx=0x7f53f800eb50 comp rx=0 tx=0).stop 2023-03-03T07:57:51.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.917+0000 7f540899d700 1 -- 172.21.15.98:0/3453055757 shutdown_connections 2023-03-03T07:57:51.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.917+0000 7f540899d700 1 --2- 172.21.15.98:0/3453055757 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f53f00683c0 0x7f53f006a870 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:51.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.917+0000 7f540899d700 1 --2- 172.21.15.98:0/3453055757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5404160fa0 0x7f54041f5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:51.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.917+0000 7f540899d700 1 --2- 172.21.15.98:0/3453055757 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f54040b50d0 0x7f54040b8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:51.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.917+0000 7f540899d700 1 --2- 172.21.15.98:0/3453055757 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54041461f0 0x7f5404155fc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:51.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.917+0000 7f540899d700 1 -- 172.21.15.98:0/3453055757 >> 172.21.15.98:0/3453055757 conn(0x7f540406b900 msgr2=0x7f54040731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:51.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.918+0000 7f540899d700 1 -- 172.21.15.98:0/3453055757 shutdown_connections 2023-03-03T07:57:51.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:51.918+0000 7f540899d700 1 -- 172.21.15.98:0/3453055757 wait complete. 2023-03-03T07:57:51.922 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 15 2023-03-03T07:57:51.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:57:52.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.190+0000 7f184dd9d700 1 Processor -- start 2023-03-03T07:57:52.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.190+0000 7f184dd9d700 1 -- start start 2023-03-03T07:57:52.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.192+0000 7f184dd9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f18481595d0 0x7f18480c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:52.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.192+0000 7f184dd9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f184815a0b0 0x7f18480cd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:52.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.192+0000 7f184dd9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18480cd000 con 0x7f184806c7c0 2023-03-03T07:57:52.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.192+0000 7f184dd9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f184806cd80 con 0x7f18481595d0 2023-03-03T07:57:52.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.192+0000 7f184dd9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f184806d060 con 0x7f184815a0b0 2023-03-03T07:57:52.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.192+0000 7f1847fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f184806c7c0 0x7f18481599a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44250/0 (socket says 172.21.15.98:44250) 2023-03-03T07:57:52.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.192+0000 7f1847fff700 1 -- 172.21.15.98:0/3034170415 learned_addr learned my addr 172.21.15.98:0/3034170415 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:52.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.192+0000 7f184cd9b700 1 --2- 172.21.15.98:0/3034170415 >> v2:172.21.15.98:3300/0 conn(0x7f18481595d0 0x7f18480c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:52.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.192+0000 7f184d59c700 1 --2- 172.21.15.98:0/3034170415 >> v2:172.21.15.100:3300/0 conn(0x7f184815a0b0 0x7f18480cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:52.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.193+0000 7f184cd9b700 1 -- 172.21.15.98:0/3034170415 >> v2:172.21.15.100:3300/0 conn(0x7f184815a0b0 msgr2=0x7f18480cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:52.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.193+0000 7f184cd9b700 1 --2- 172.21.15.98:0/3034170415 >> v2:172.21.15.100:3300/0 conn(0x7f184815a0b0 0x7f18480cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.193+0000 7f184cd9b700 1 -- 172.21.15.98:0/3034170415 >> v1:172.21.15.98:6789/0 conn(0x7f184806c7c0 legacy=0x7f18481599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:52.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.193+0000 7f184cd9b700 1 -- 172.21.15.98:0/3034170415 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184806d340 con 0x7f18481595d0 2023-03-03T07:57:52.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.193+0000 7f18477fe700 1 -- 172.21.15.98:0/3034170415 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677340235 0 0) 0x7f18480cd000 con 0x7f184806c7c0 2023-03-03T07:57:52.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.193+0000 7f184cd9b700 1 --2- 172.21.15.98:0/3034170415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18481595d0 0x7f18480c9d20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f18480b0780 tx=0x7f183c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab6a845b0e9d9f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:52.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.193+0000 7f18477fe700 1 -- 172.21.15.98:0/3034170415 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c011830 con 0x7f18481595d0 2023-03-03T07:57:52.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.193+0000 7f18477fe700 1 -- 172.21.15.98:0/3034170415 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f183c011990 con 0x7f18481595d0 2023-03-03T07:57:52.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.193+0000 7f18477fe700 1 -- 172.21.15.98:0/3034170415 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c011d20 con 0x7f18481595d0 2023-03-03T07:57:52.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.194+0000 7f184dd9d700 1 -- 172.21.15.98:0/3034170415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18481595d0 msgr2=0x7f18480c9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:52.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.194+0000 7f184dd9d700 1 --2- 172.21.15.98:0/3034170415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18481595d0 0x7f18480c9d20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f18480b0780 tx=0x7f183c009b20 comp rx=0 tx=0).stop 2023-03-03T07:57:52.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.194+0000 7f184dd9d700 1 -- 172.21.15.98:0/3034170415 shutdown_connections 2023-03-03T07:57:52.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.194+0000 7f184dd9d700 1 --2- 172.21.15.98:0/3034170415 >> v2:172.21.15.100:3300/0 conn(0x7f184815a0b0 0x7f18480cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.194+0000 7f184dd9d700 1 --2- 172.21.15.98:0/3034170415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18481595d0 0x7f18480c9d20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.194+0000 7f184dd9d700 1 -- 172.21.15.98:0/3034170415 >> 172.21.15.98:0/3034170415 conn(0x7f18480cf950 msgr2=0x7f18480cb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:52.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.194+0000 7f184dd9d700 1 -- 172.21.15.98:0/3034170415 shutdown_connections 2023-03-03T07:57:52.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.194+0000 7f184dd9d700 1 -- 172.21.15.98:0/3034170415 wait complete. 2023-03-03T07:57:52.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.196+0000 7f184dd9d700 1 Processor -- start 2023-03-03T07:57:52.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.196+0000 7f184dd9d700 1 -- start start 2023-03-03T07:57:52.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.197+0000 7f184dd9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f184806c7c0 0x7f184814a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:52.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.197+0000 7f184dd9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18481595d0 0x7f18480c37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:52.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.197+0000 7f184dd9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f184815a0b0 0x7f18480ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:52.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.197+0000 7f184dd9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184814a920 con 0x7f18481595d0 2023-03-03T07:57:52.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.197+0000 7f184dd9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18480c4e30 con 0x7f184806c7c0 2023-03-03T07:57:52.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.197+0000 7f184dd9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18480c4220 con 0x7f184815a0b0 2023-03-03T07:57:52.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.198+0000 7f1847fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18481595d0 0x7f18480c37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:52.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.198+0000 7f184cd9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f184806c7c0 0x7f184814a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:52.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.198+0000 7f184cd9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f184806c7c0 0x7f184814a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:32814/0 (socket says 172.21.15.98:32814) 2023-03-03T07:57:52.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.198+0000 7f184cd9b700 1 -- 172.21.15.98:0/513212727 learned_addr learned my addr 172.21.15.98:0/513212727 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:52.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.198+0000 7f184d59c700 1 --2- 172.21.15.98:0/513212727 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f184815a0b0 0x7f18480ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:52.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.198+0000 7f184cd9b700 1 -- 172.21.15.98:0/513212727 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f184815a0b0 msgr2=0x7f18480ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:52.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.198+0000 7f184cd9b700 1 --2- 172.21.15.98:0/513212727 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f184815a0b0 0x7f18480ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.198+0000 7f184cd9b700 1 -- 172.21.15.98:0/513212727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18481595d0 msgr2=0x7f18480c37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:52.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.198+0000 7f184cd9b700 1 --2- 172.21.15.98:0/513212727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18481595d0 0x7f18480c37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.198+0000 7f184cd9b700 1 -- 172.21.15.98:0/513212727 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183c0094a0 con 0x7f184806c7c0 2023-03-03T07:57:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.198+0000 7f184d59c700 1 --2- 172.21.15.98:0/513212727 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f184815a0b0 0x7f18480ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.198+0000 7f1847fff700 1 --2- 172.21.15.98:0/513212727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18481595d0 0x7f18480c37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.199+0000 7f184cd9b700 1 --2- 172.21.15.98:0/513212727 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f184806c7c0 0x7f184814a3e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f183c0090b0 tx=0x7f183c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.199+0000 7f18457fa700 1 -- 172.21.15.98:0/513212727 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c004510 con 0x7f184806c7c0 2023-03-03T07:57:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.199+0000 7f18457fa700 1 -- 172.21.15.98:0/513212727 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f183c01f070 con 0x7f184806c7c0 2023-03-03T07:57:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.199+0000 7f18457fa700 1 -- 172.21.15.98:0/513212727 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c0038c0 con 0x7f184806c7c0 2023-03-03T07:57:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.199+0000 7f184dd9d700 1 -- 172.21.15.98:0/513212727 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184815a510 con 0x7f184806c7c0 2023-03-03T07:57:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.199+0000 7f184dd9d700 1 -- 172.21.15.98:0/513212727 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f184806d620 con 0x7f184806c7c0 2023-03-03T07:57:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.200+0000 7f184dd9d700 1 -- 172.21.15.98:0/513212727 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18480ca260 con 0x7f184806c7c0 2023-03-03T07:57:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.202+0000 7f18457fa700 1 -- 172.21.15.98:0/513212727 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f183c00f050 con 0x7f184806c7c0 2023-03-03T07:57:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.203+0000 7f18457fa700 1 --2- 172.21.15.98:0/513212727 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f182c0683c0 0x7f182c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.203+0000 7f18457fa700 1 -- 172.21.15.98:0/513212727 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f183c021070 con 0x7f184806c7c0 2023-03-03T07:57:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.203+0000 7f1847fff700 1 --2- 172.21.15.98:0/513212727 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f182c0683c0 0x7f182c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.204+0000 7f1847fff700 1 --2- 172.21.15.98:0/513212727 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f182c0683c0 0x7f182c06a870 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1848159e60 tx=0x7f1834007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.208+0000 7f18457fa700 1 -- 172.21.15.98:0/513212727 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f183c053c60 con 0x7f184806c7c0 2023-03-03T07:57:52.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.407+0000 7f184dd9d700 1 -- 172.21.15.98:0/513212727 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1848068300 con 0x7f184806c7c0 2023-03-03T07:57:52.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.410+0000 7f18457fa700 1 -- 172.21.15.98:0/513212727 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+17972 (secure 0 0 0) 0x7f183c057280 con 0x7f184806c7c0 2023-03-03T07:57:52.412 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:52.412 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":40,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:57:28.259781+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:57:15.545608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-03-03T07:57:16.283131+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.31","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:57:52.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.412+0000 7f184dd9d700 1 -- 172.21.15.98:0/513212727 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f182c0683c0 msgr2=0x7f182c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:52.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.412+0000 7f184dd9d700 1 --2- 172.21.15.98:0/513212727 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f182c0683c0 0x7f182c06a870 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1848159e60 tx=0x7f1834007de0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.412+0000 7f184dd9d700 1 -- 172.21.15.98:0/513212727 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f184806c7c0 msgr2=0x7f184814a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:52.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.412+0000 7f184dd9d700 1 --2- 172.21.15.98:0/513212727 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f184806c7c0 0x7f184814a3e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f183c0090b0 tx=0x7f183c004750 comp rx=0 tx=0).stop 2023-03-03T07:57:52.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.412+0000 7f184dd9d700 1 -- 172.21.15.98:0/513212727 shutdown_connections 2023-03-03T07:57:52.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.412+0000 7f184dd9d700 1 --2- 172.21.15.98:0/513212727 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f182c0683c0 0x7f182c06a870 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.412+0000 7f184dd9d700 1 --2- 172.21.15.98:0/513212727 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f184815a0b0 0x7f18480ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.412+0000 7f184dd9d700 1 --2- 172.21.15.98:0/513212727 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f184806c7c0 0x7f184814a3e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.412+0000 7f184dd9d700 1 --2- 172.21.15.98:0/513212727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18481595d0 0x7f18480c37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.412+0000 7f184dd9d700 1 -- 172.21.15.98:0/513212727 >> 172.21.15.98:0/513212727 conn(0x7f18480cf950 msgr2=0x7f18480713f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:52.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.412+0000 7f184dd9d700 1 -- 172.21.15.98:0/513212727 shutdown_connections 2023-03-03T07:57:52.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.412+0000 7f184dd9d700 1 -- 172.21.15.98:0/513212727 wait complete. 2023-03-03T07:57:52.442 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:57:15.545608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:57:16.283131+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:57:52.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:57:52.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.693+0000 7feff35ff700 1 Processor -- start 2023-03-03T07:57:52.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.694+0000 7feff35ff700 1 -- start start 2023-03-03T07:57:52.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.695+0000 7feff35ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fefec0b26d0 0x7fefec0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:52.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.695+0000 7feff35ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec0cd7d0 con 0x7fefec0b5140 2023-03-03T07:57:52.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.695+0000 7feff35ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec0b90e0 con 0x7fefec0b8d10 2023-03-03T07:57:52.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.695+0000 7feff35ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefec0b55c0 con 0x7fefec0b26d0 2023-03-03T07:57:52.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.696+0000 7feff1b9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fefec0b5140 0x7fefec0d13a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44252/0 (socket says 172.21.15.98:44252) 2023-03-03T07:57:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.696+0000 7feff1b9c700 1 -- 172.21.15.98:0/894982233 learned_addr learned my addr 172.21.15.98:0/894982233 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.696+0000 7feff0b9a700 1 --2- 172.21.15.98:0/894982233 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fefec0b26d0 0x7fefec0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.696+0000 7fefe3fff700 1 -- 172.21.15.98:0/894982233 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015714136 0 0) 0x7fefec0cd7d0 con 0x7fefec0b5140 2023-03-03T07:57:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.696+0000 7fefe3fff700 1 -- 172.21.15.98:0/894982233 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd80035b0 con 0x7fefec0b5140 2023-03-03T07:57:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.696+0000 7fefe3fff700 1 -- 172.21.15.98:0/894982233 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46806838 0 0) 0x7fefec0b90e0 con 0x7fefec0b8d10 2023-03-03T07:57:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.696+0000 7fefe3fff700 1 -- 172.21.15.98:0/894982233 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefec0cd7d0 con 0x7fefec0b8d10 2023-03-03T07:57:52.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.696+0000 7feff0b9a700 1 -- 172.21.15.98:0/894982233 >> v1:172.21.15.100:6789/0 conn(0x7fefec0b8d10 legacy=0x7fefec0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:52.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.696+0000 7feff0b9a700 1 -- 172.21.15.98:0/894982233 >> v1:172.21.15.98:6789/0 conn(0x7fefec0b5140 legacy=0x7fefec0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:52.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.697+0000 7feff0b9a700 1 -- 172.21.15.98:0/894982233 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec0cda20 con 0x7fefec0b26d0 2023-03-03T07:57:52.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.697+0000 7feff0b9a700 1 --2- 172.21.15.98:0/894982233 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fefec0b26d0 0x7fefec0b4c00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fefdc009870 tx=0x7fefdc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec65ce4aee01c74d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:52.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.697+0000 7fefe3fff700 1 -- 172.21.15.98:0/894982233 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc011830 con 0x7fefec0b26d0 2023-03-03T07:57:52.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.697+0000 7fefe3fff700 1 -- 172.21.15.98:0/894982233 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefdc011990 con 0x7fefec0b26d0 2023-03-03T07:57:52.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.697+0000 7fefe3fff700 1 -- 172.21.15.98:0/894982233 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc011d20 con 0x7fefec0b26d0 2023-03-03T07:57:52.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.698+0000 7feff35ff700 1 -- 172.21.15.98:0/894982233 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fefec0b26d0 msgr2=0x7fefec0b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:52.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.698+0000 7feff35ff700 1 --2- 172.21.15.98:0/894982233 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fefec0b26d0 0x7fefec0b4c00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fefdc009870 tx=0x7fefdc009b80 comp rx=0 tx=0).stop 2023-03-03T07:57:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.698+0000 7feff35ff700 1 -- 172.21.15.98:0/894982233 shutdown_connections 2023-03-03T07:57:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.698+0000 7feff35ff700 1 --2- 172.21.15.98:0/894982233 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fefec0b26d0 0x7fefec0b4c00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.698+0000 7feff35ff700 1 -- 172.21.15.98:0/894982233 >> 172.21.15.98:0/894982233 conn(0x7fefec073e70 msgr2=0x7fefec06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.698+0000 7feff35ff700 1 -- 172.21.15.98:0/894982233 shutdown_connections 2023-03-03T07:57:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.698+0000 7feff35ff700 1 -- 172.21.15.98:0/894982233 wait complete. 2023-03-03T07:57:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.700+0000 7feff35ff700 1 Processor -- start 2023-03-03T07:57:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.700+0000 7feff35ff700 1 -- start start 2023-03-03T07:57:52.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.701+0000 7feff35ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fefec0b26d0 0x7fefec15f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:52.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.701+0000 7feff35ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefec0b5140 0x7fefec0ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:52.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.701+0000 7feff35ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fefec0b8d10 0x7fefec0b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.701+0000 7feff35ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec0cbcf0 con 0x7fefec0b5140 2023-03-03T07:57:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.701+0000 7feff35ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec0cbfd0 con 0x7fefec0b26d0 2023-03-03T07:57:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.701+0000 7feff35ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefec0cc2b0 con 0x7fefec0b8d10 2023-03-03T07:57:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.702+0000 7feff0b9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefec0b5140 0x7fefec0ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.702+0000 7feff0b9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefec0b5140 0x7fefec0ccc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42568/0 (socket says 172.21.15.98:42568) 2023-03-03T07:57:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.702+0000 7feff0b9a700 1 -- 172.21.15.98:0/3703007850 learned_addr learned my addr 172.21.15.98:0/3703007850 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.702+0000 7feff1b9c700 1 --2- 172.21.15.98:0/3703007850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fefec0b8d10 0x7fefec0b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:52.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.702+0000 7feff139b700 1 --2- 172.21.15.98:0/3703007850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fefec0b26d0 0x7fefec15f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:52.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.702+0000 7feff0b9a700 1 -- 172.21.15.98:0/3703007850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fefec0b8d10 msgr2=0x7fefec0b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:52.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.702+0000 7feff0b9a700 1 --2- 172.21.15.98:0/3703007850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fefec0b8d10 0x7fefec0b0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.702+0000 7feff0b9a700 1 -- 172.21.15.98:0/3703007850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fefec0b26d0 msgr2=0x7fefec15f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:52.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.702+0000 7feff0b9a700 1 --2- 172.21.15.98:0/3703007850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fefec0b26d0 0x7fefec15f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.702+0000 7feff0b9a700 1 -- 172.21.15.98:0/3703007850 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefdc0094a0 con 0x7fefec0b5140 2023-03-03T07:57:52.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.702+0000 7feff139b700 1 --2- 172.21.15.98:0/3703007850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fefec0b26d0 0x7fefec15f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:52.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.702+0000 7feff0b9a700 1 --2- 172.21.15.98:0/3703007850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefec0b5140 0x7fefec0ccc60 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fefdc014040 tx=0x7fefdc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:52.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.703+0000 7fefe1ffb700 1 -- 172.21.15.98:0/3703007850 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc01a760 con 0x7fefec0b5140 2023-03-03T07:57:52.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.703+0000 7fefe1ffb700 1 -- 172.21.15.98:0/3703007850 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefdc01a8c0 con 0x7fefec0b5140 2023-03-03T07:57:52.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.703+0000 7fefe1ffb700 1 -- 172.21.15.98:0/3703007850 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc01ac30 con 0x7fefec0b5140 2023-03-03T07:57:52.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.703+0000 7feff35ff700 1 -- 172.21.15.98:0/3703007850 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefec0d0e80 con 0x7fefec0b5140 2023-03-03T07:57:52.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.703+0000 7feff35ff700 1 -- 172.21.15.98:0/3703007850 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefec0cdd00 con 0x7fefec0b5140 2023-03-03T07:57:52.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.704+0000 7feff35ff700 1 -- 172.21.15.98:0/3703007850 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefec0cdfe0 con 0x7fefec0b5140 2023-03-03T07:57:52.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.705+0000 7fefe1ffb700 1 -- 172.21.15.98:0/3703007850 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fefdc00f050 con 0x7fefec0b5140 2023-03-03T07:57:52.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.706+0000 7fefe1ffb700 1 --2- 172.21.15.98:0/3703007850 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fefd00683c0 0x7fefd006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:52.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.707+0000 7fefe1ffb700 1 -- 172.21.15.98:0/3703007850 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fefdc086c40 con 0x7fefec0b5140 2023-03-03T07:57:52.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.707+0000 7feff139b700 1 --2- 172.21.15.98:0/3703007850 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fefd00683c0 0x7fefd006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:52.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.707+0000 7feff139b700 1 --2- 172.21.15.98:0/3703007850 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fefd00683c0 0x7fefd006a870 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fefe80011e0 tx=0x7fefe8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:52.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.711+0000 7fefe1ffb700 1 -- 172.21.15.98:0/3703007850 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fefdc053370 con 0x7fefec0b5140 2023-03-03T07:57:52.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.906+0000 7feff35ff700 1 -- 172.21.15.98:0/3703007850 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fefec0c6c70 con 0x7fefec0b5140 2023-03-03T07:57:52.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.906+0000 7fefe1ffb700 1 -- 172.21.15.98:0/3703007850 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7fefec0c6c70 con 0x7fefec0b5140 2023-03-03T07:57:52.909 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:57:52.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.908+0000 7feff35ff700 1 -- 172.21.15.98:0/3703007850 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fefd00683c0 msgr2=0x7fefd006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:52.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.909+0000 7feff35ff700 1 --2- 172.21.15.98:0/3703007850 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fefd00683c0 0x7fefd006a870 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fefe80011e0 tx=0x7fefe8009040 comp rx=0 tx=0).stop 2023-03-03T07:57:52.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.909+0000 7feff35ff700 1 -- 172.21.15.98:0/3703007850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefec0b5140 msgr2=0x7fefec0ccc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:52.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.909+0000 7feff35ff700 1 --2- 172.21.15.98:0/3703007850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefec0b5140 0x7fefec0ccc60 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fefdc014040 tx=0x7fefdc004750 comp rx=0 tx=0).stop 2023-03-03T07:57:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.909+0000 7feff35ff700 1 -- 172.21.15.98:0/3703007850 shutdown_connections 2023-03-03T07:57:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.909+0000 7feff35ff700 1 --2- 172.21.15.98:0/3703007850 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fefd00683c0 0x7fefd006a870 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.909+0000 7feff35ff700 1 --2- 172.21.15.98:0/3703007850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fefec0b8d10 0x7fefec0b0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.909+0000 7feff35ff700 1 --2- 172.21.15.98:0/3703007850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fefec0b26d0 0x7fefec15f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.909+0000 7feff35ff700 1 --2- 172.21.15.98:0/3703007850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefec0b5140 0x7fefec0ccc60 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.909+0000 7feff35ff700 1 -- 172.21.15.98:0/3703007850 >> 172.21.15.98:0/3703007850 conn(0x7fefec073e70 msgr2=0x7fefec15b750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.909+0000 7feff35ff700 1 -- 172.21.15.98:0/3703007850 shutdown_connections 2023-03-03T07:57:52.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:52.909+0000 7feff35ff700 1 -- 172.21.15.98:0/3703007850 wait complete. 2023-03-03T07:57:52.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:57:53.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.185+0000 7fb04759d700 1 Processor -- start 2023-03-03T07:57:53.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.186+0000 7fb04759d700 1 -- start start 2023-03-03T07:57:53.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.187+0000 7fb04759d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb0401609b0 0x7fb040160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:53.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.187+0000 7fb04759d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0400cf570 0x7fb04015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:53.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.187+0000 7fb04759d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0401612c0 con 0x7fb04015db30 2023-03-03T07:57:53.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.187+0000 7fb04759d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0400cebb0 con 0x7fb0401609b0 2023-03-03T07:57:53.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.187+0000 7fb04759d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0400cd780 con 0x7fb0400cf570 2023-03-03T07:57:53.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.188+0000 7fb04659b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb0401609b0 0x7fb040160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:53.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.188+0000 7fb045d9a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0400cf570 0x7fb04015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:53.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.188+0000 7fb04659b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb0401609b0 0x7fb040160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:32828/0 (socket says 172.21.15.98:32828) 2023-03-03T07:57:53.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.188+0000 7fb045d9a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0400cf570 0x7fb04015d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:49758/0 (socket says 172.21.15.98:49758) 2023-03-03T07:57:53.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.188+0000 7fb04659b700 1 -- 172.21.15.98:0/958536439 learned_addr learned my addr 172.21.15.98:0/958536439 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:53.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.188+0000 7fb04659b700 1 -- 172.21.15.98:0/958536439 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0400cf570 msgr2=0x7fb04015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:53.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.188+0000 7fb04659b700 1 --2- 172.21.15.98:0/958536439 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0400cf570 0x7fb04015d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.188+0000 7fb04659b700 1 -- 172.21.15.98:0/958536439 >> v1:172.21.15.100:6789/0 conn(0x7fb04015db30 legacy=0x7fb0400b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:53.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.188+0000 7fb04659b700 1 -- 172.21.15.98:0/958536439 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0400cda60 con 0x7fb0401609b0 2023-03-03T07:57:53.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.188+0000 7fb045599700 1 -- 172.21.15.98:0/958536439 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544734302 0 0) 0x7fb0401612c0 con 0x7fb04015db30 2023-03-03T07:57:53.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.188+0000 7fb045d9a700 1 --2- 172.21.15.98:0/958536439 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0400cf570 0x7fb04015d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:57:53.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.189+0000 7fb04659b700 1 --2- 172.21.15.98:0/958536439 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0401609b0 0x7fb040160d80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb03c0098f0 tx=0x7fb03c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cec1209b97866404 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:53.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.189+0000 7fb045599700 1 -- 172.21.15.98:0/958536439 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c011830 con 0x7fb0401609b0 2023-03-03T07:57:53.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.189+0000 7fb045599700 1 -- 172.21.15.98:0/958536439 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03c011990 con 0x7fb0401609b0 2023-03-03T07:57:53.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.189+0000 7fb045599700 1 -- 172.21.15.98:0/958536439 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c011d20 con 0x7fb0401609b0 2023-03-03T07:57:53.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.190+0000 7fb04759d700 1 -- 172.21.15.98:0/958536439 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0401609b0 msgr2=0x7fb040160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:53.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.190+0000 7fb04759d700 1 --2- 172.21.15.98:0/958536439 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0401609b0 0x7fb040160d80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb03c0098f0 tx=0x7fb03c009c00 comp rx=0 tx=0).stop 2023-03-03T07:57:53.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.190+0000 7fb04759d700 1 -- 172.21.15.98:0/958536439 shutdown_connections 2023-03-03T07:57:53.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.190+0000 7fb04759d700 1 --2- 172.21.15.98:0/958536439 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0401609b0 0x7fb040160d80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.190+0000 7fb04759d700 1 --2- 172.21.15.98:0/958536439 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0400cf570 0x7fb04015d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.190+0000 7fb04759d700 1 -- 172.21.15.98:0/958536439 >> 172.21.15.98:0/958536439 conn(0x7fb04006b880 msgr2=0x7fb0400cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:53.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.190+0000 7fb04759d700 1 -- 172.21.15.98:0/958536439 shutdown_connections 2023-03-03T07:57:53.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.190+0000 7fb04759d700 1 -- 172.21.15.98:0/958536439 wait complete. 2023-03-03T07:57:53.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.192+0000 7fb04759d700 1 Processor -- start 2023-03-03T07:57:53.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.192+0000 7fb04759d700 1 -- start start 2023-03-03T07:57:53.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.193+0000 7fb04759d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0400cf570 0x7fb0400cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:53.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.193+0000 7fb04759d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb04015db30 0x7fb0400c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:53.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.193+0000 7fb04759d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0401609b0 0x7fb040152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:53.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.193+0000 7fb04759d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0400cc520 con 0x7fb0401609b0 2023-03-03T07:57:53.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.193+0000 7fb04759d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0400c71f0 con 0x7fb04015db30 2023-03-03T07:57:53.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.193+0000 7fb04759d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0400c74d0 con 0x7fb0400cf570 2023-03-03T07:57:53.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.194+0000 7fb046d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0401609b0 0x7fb040152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:53.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.194+0000 7fb045d9a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb04015db30 0x7fb0400c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:53.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.194+0000 7fb04659b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0400cf570 0x7fb0400cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:53.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.194+0000 7fb045d9a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb04015db30 0x7fb0400c8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:32842/0 (socket says 172.21.15.98:32842) 2023-03-03T07:57:53.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.194+0000 7fb045d9a700 1 -- 172.21.15.98:0/628445809 learned_addr learned my addr 172.21.15.98:0/628445809 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:53.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.194+0000 7fb04659b700 1 -- 172.21.15.98:0/628445809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb04015db30 msgr2=0x7fb0400c8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:53.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.194+0000 7fb04659b700 1 --2- 172.21.15.98:0/628445809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb04015db30 0x7fb0400c8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.194+0000 7fb04659b700 1 -- 172.21.15.98:0/628445809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0401609b0 msgr2=0x7fb040152110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:53.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.194+0000 7fb04659b700 1 --2- 172.21.15.98:0/628445809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0401609b0 0x7fb040152110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.194+0000 7fb04659b700 1 -- 172.21.15.98:0/628445809 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0300094a0 con 0x7fb0400cf570 2023-03-03T07:57:53.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.195+0000 7fb04659b700 1 --2- 172.21.15.98:0/628445809 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0400cf570 0x7fb0400cbfe0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb03c016000 tx=0x7fb03c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:53.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.195+0000 7fb036ffd700 1 -- 172.21.15.98:0/628445809 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c004510 con 0x7fb0400cf570 2023-03-03T07:57:53.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.195+0000 7fb036ffd700 1 -- 172.21.15.98:0/628445809 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03c01e070 con 0x7fb0400cf570 2023-03-03T07:57:53.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.195+0000 7fb036ffd700 1 -- 172.21.15.98:0/628445809 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c0038c0 con 0x7fb0400cf570 2023-03-03T07:57:53.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.195+0000 7fb04759d700 1 -- 172.21.15.98:0/628445809 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb03c009550 con 0x7fb0400cf570 2023-03-03T07:57:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.195+0000 7fb04759d700 1 -- 172.21.15.98:0/628445809 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04015cdf0 con 0x7fb0400cf570 2023-03-03T07:57:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.196+0000 7fb04759d700 1 -- 172.21.15.98:0/628445809 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0400cdd40 con 0x7fb0400cf570 2023-03-03T07:57:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.197+0000 7fb036ffd700 1 -- 172.21.15.98:0/628445809 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fb03c00f050 con 0x7fb0400cf570 2023-03-03T07:57:53.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.199+0000 7fb036ffd700 1 --2- 172.21.15.98:0/628445809 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb0240683c0 0x7fb02406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:53.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.199+0000 7fb036ffd700 1 -- 172.21.15.98:0/628445809 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fb03c086c30 con 0x7fb0400cf570 2023-03-03T07:57:53.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.199+0000 7fb045d9a700 1 --2- 172.21.15.98:0/628445809 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb0240683c0 0x7fb02406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:53.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.200+0000 7fb045d9a700 1 --2- 172.21.15.98:0/628445809 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb0240683c0 0x7fb02406a870 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb0400b4110 tx=0x7fb030009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:53.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.204+0000 7fb036ffd700 1 -- 172.21.15.98:0/628445809 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb03c0532b0 con 0x7fb0400cf570 2023-03-03T07:57:53.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.399+0000 7fb04759d700 1 -- 172.21.15.98:0/628445809 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb04015d0d0 con 0x7fb0400cf570 2023-03-03T07:57:53.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.400+0000 7fb036ffd700 1 -- 172.21.15.98:0/628445809 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v40) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb03c0200a0 con 0x7fb0400cf570 2023-03-03T07:57:53.403 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:57:53.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.403+0000 7fb04759d700 1 -- 172.21.15.98:0/628445809 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb0240683c0 msgr2=0x7fb02406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:53.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.403+0000 7fb04759d700 1 --2- 172.21.15.98:0/628445809 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb0240683c0 0x7fb02406a870 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb0400b4110 tx=0x7fb030009210 comp rx=0 tx=0).stop 2023-03-03T07:57:53.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.403+0000 7fb04759d700 1 -- 172.21.15.98:0/628445809 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0400cf570 msgr2=0x7fb0400cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:53.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.403+0000 7fb04759d700 1 --2- 172.21.15.98:0/628445809 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0400cf570 0x7fb0400cbfe0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb03c016000 tx=0x7fb03c004750 comp rx=0 tx=0).stop 2023-03-03T07:57:53.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.403+0000 7fb04759d700 1 -- 172.21.15.98:0/628445809 shutdown_connections 2023-03-03T07:57:53.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.403+0000 7fb04759d700 1 --2- 172.21.15.98:0/628445809 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb0240683c0 0x7fb02406a870 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.403+0000 7fb04759d700 1 --2- 172.21.15.98:0/628445809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0401609b0 0x7fb040152110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.403+0000 7fb04759d700 1 --2- 172.21.15.98:0/628445809 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0400cf570 0x7fb0400cbfe0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.403+0000 7fb04759d700 1 --2- 172.21.15.98:0/628445809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb04015db30 0x7fb0400c8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.403+0000 7fb04759d700 1 -- 172.21.15.98:0/628445809 >> 172.21.15.98:0/628445809 conn(0x7fb04006b880 msgr2=0x7fb0400b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:53.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.403+0000 7fb04759d700 1 -- 172.21.15.98:0/628445809 shutdown_connections 2023-03-03T07:57:53.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.403+0000 7fb04759d700 1 -- 172.21.15.98:0/628445809 wait complete. 2023-03-03T07:57:53.433 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:57:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.679+0000 7f94da9ff700 1 Processor -- start 2023-03-03T07:57:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.680+0000 7f94da9ff700 1 -- start start 2023-03-03T07:57:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.681+0000 7f94da9ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f94d415dd20 0x7f94d40b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.681+0000 7f94da9ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f94d40cd5c0 0x7f94d41599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.681+0000 7f94da9ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d415e170 con 0x7f94d40b0800 2023-03-03T07:57:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.681+0000 7f94da9ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d40b1110 con 0x7f94d40cd5c0 2023-03-03T07:57:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.681+0000 7f94da9ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d40c9d50 con 0x7f94d415dd20 2023-03-03T07:57:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.682+0000 7f94d3fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f94d40b0800 0x7f94d40c93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44254/0 (socket says 172.21.15.98:44254) 2023-03-03T07:57:53.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.682+0000 7f94d3fff700 1 -- 172.21.15.98:0/3361176485 learned_addr learned my addr 172.21.15.98:0/3361176485 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:53.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.682+0000 7f94d8f9c700 1 --2- 172.21.15.98:0/3361176485 >> v2:172.21.15.98:3300/0 conn(0x7f94d40cd5c0 0x7f94d41599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:53.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.682+0000 7f94d37fe700 1 --2- 172.21.15.98:0/3361176485 >> v2:172.21.15.100:3300/0 conn(0x7f94d415dd20 0x7f94d40b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:53.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.682+0000 7f94d2ffd700 1 -- 172.21.15.98:0/3361176485 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467912363 0 0) 0x7f94d415e170 con 0x7f94d40b0800 2023-03-03T07:57:53.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.682+0000 7f94d2ffd700 1 -- 172.21.15.98:0/3361176485 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94bc0035b0 con 0x7f94d40b0800 2023-03-03T07:57:53.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.682+0000 7f94d8f9c700 1 -- 172.21.15.98:0/3361176485 >> v2:172.21.15.100:3300/0 conn(0x7f94d415dd20 msgr2=0x7f94d40b0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:53.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.682+0000 7f94d8f9c700 1 --2- 172.21.15.98:0/3361176485 >> v2:172.21.15.100:3300/0 conn(0x7f94d415dd20 0x7f94d40b0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.682+0000 7f94d8f9c700 1 -- 172.21.15.98:0/3361176485 >> v1:172.21.15.98:6789/0 conn(0x7f94d40b0800 legacy=0x7f94d40c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:53.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.682+0000 7f94d8f9c700 1 -- 172.21.15.98:0/3361176485 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d40ca030 con 0x7f94d40cd5c0 2023-03-03T07:57:53.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.683+0000 7f94d8f9c700 1 --2- 172.21.15.98:0/3361176485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40cd5c0 0x7f94d41599a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f94d40ced50 tx=0x7f94cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0259dd62aa6e3bb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:53.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.683+0000 7f94d2ffd700 1 -- 172.21.15.98:0/3361176485 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc00e070 con 0x7f94d40cd5c0 2023-03-03T07:57:53.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.683+0000 7f94d2ffd700 1 -- 172.21.15.98:0/3361176485 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94cc00bb20 con 0x7f94d40cd5c0 2023-03-03T07:57:53.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.683+0000 7f94d2ffd700 1 -- 172.21.15.98:0/3361176485 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc012430 con 0x7f94d40cd5c0 2023-03-03T07:57:53.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.683+0000 7f94da9ff700 1 -- 172.21.15.98:0/3361176485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40cd5c0 msgr2=0x7f94d41599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:53.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.683+0000 7f94da9ff700 1 --2- 172.21.15.98:0/3361176485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40cd5c0 0x7f94d41599a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f94d40ced50 tx=0x7f94cc00b330 comp rx=0 tx=0).stop 2023-03-03T07:57:53.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.684+0000 7f94da9ff700 1 -- 172.21.15.98:0/3361176485 shutdown_connections 2023-03-03T07:57:53.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.684+0000 7f94da9ff700 1 --2- 172.21.15.98:0/3361176485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40cd5c0 0x7f94d41599a0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.684+0000 7f94da9ff700 1 --2- 172.21.15.98:0/3361176485 >> v2:172.21.15.100:3300/0 conn(0x7f94d415dd20 0x7f94d40b0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.684+0000 7f94da9ff700 1 -- 172.21.15.98:0/3361176485 >> 172.21.15.98:0/3361176485 conn(0x7f94d40cb800 msgr2=0x7f94d40cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:53.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.684+0000 7f94da9ff700 1 -- 172.21.15.98:0/3361176485 shutdown_connections 2023-03-03T07:57:53.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.684+0000 7f94da9ff700 1 -- 172.21.15.98:0/3361176485 wait complete. 2023-03-03T07:57:53.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.686+0000 7f94da9ff700 1 Processor -- start 2023-03-03T07:57:53.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.686+0000 7f94da9ff700 1 -- start start 2023-03-03T07:57:53.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.687+0000 7f94da9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94d40b0800 0x7f94d40adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:53.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.687+0000 7f94da9ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40cd5c0 0x7f94d41418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:53.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.687+0000 7f94da9ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94d415dd20 0x7f94d4152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:53.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.687+0000 7f94da9ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d40ada50 con 0x7f94d40cd5c0 2023-03-03T07:57:53.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.687+0000 7f94da9ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d40ae420 con 0x7f94d415dd20 2023-03-03T07:57:53.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.687+0000 7f94da9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d40ad120 con 0x7f94d40b0800 2023-03-03T07:57:53.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.687+0000 7f94d37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40cd5c0 0x7f94d41418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:53.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.687+0000 7f94d37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40cd5c0 0x7f94d41418c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42600/0 (socket says 172.21.15.98:42600) 2023-03-03T07:57:53.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.687+0000 7f94d37fe700 1 -- 172.21.15.98:0/3416937948 learned_addr learned my addr 172.21.15.98:0/3416937948 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:53.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.687+0000 7f94d3fff700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94d40b0800 0x7f94d40adee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:53.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.688+0000 7f94d8f9c700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94d415dd20 0x7f94d4152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:53.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.688+0000 7f94d37fe700 1 -- 172.21.15.98:0/3416937948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94d40b0800 msgr2=0x7f94d40adee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:53.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.688+0000 7f94d37fe700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94d40b0800 0x7f94d40adee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.688+0000 7f94d37fe700 1 -- 172.21.15.98:0/3416937948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94d415dd20 msgr2=0x7f94d4152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:53.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.688+0000 7f94d37fe700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94d415dd20 0x7f94d4152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.688+0000 7f94d37fe700 1 -- 172.21.15.98:0/3416937948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc009a70 con 0x7f94d40cd5c0 2023-03-03T07:57:53.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.688+0000 7f94d3fff700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94d40b0800 0x7f94d40adee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:53.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.688+0000 7f94d8f9c700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94d415dd20 0x7f94d4152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:53.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.688+0000 7f94d37fe700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40cd5c0 0x7f94d41418c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f94c800e9b0 tx=0x7f94c800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:53.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.688+0000 7f94d0ff9700 1 -- 172.21.15.98:0/3416937948 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8009710 con 0x7f94d40cd5c0 2023-03-03T07:57:53.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.688+0000 7f94d0ff9700 1 -- 172.21.15.98:0/3416937948 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c800c850 con 0x7f94d40cd5c0 2023-03-03T07:57:53.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.688+0000 7f94d0ff9700 1 -- 172.21.15.98:0/3416937948 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c800cbc0 con 0x7f94d40cd5c0 2023-03-03T07:57:53.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.688+0000 7f94da9ff700 1 -- 172.21.15.98:0/3416937948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d40cd0a0 con 0x7f94d40cd5c0 2023-03-03T07:57:53.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.689+0000 7f94da9ff700 1 -- 172.21.15.98:0/3416937948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d40ccdc0 con 0x7f94d40cd5c0 2023-03-03T07:57:53.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.690+0000 7f94da9ff700 1 -- 172.21.15.98:0/3416937948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d40ca310 con 0x7f94d40cd5c0 2023-03-03T07:57:53.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.691+0000 7f94d0ff9700 1 -- 172.21.15.98:0/3416937948 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f94c8019030 con 0x7f94d40cd5c0 2023-03-03T07:57:53.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.693+0000 7f94d0ff9700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f94c00683c0 0x7f94c006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:53.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.693+0000 7f94d0ff9700 1 -- 172.21.15.98:0/3416937948 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f94c8085f00 con 0x7f94d40cd5c0 2023-03-03T07:57:53.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.693+0000 7f94d3fff700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f94c00683c0 0x7f94c006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:53.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.694+0000 7f94d3fff700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f94c00683c0 0x7f94c006a870 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f94c40011e0 tx=0x7f94c4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:53.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.697+0000 7f94d0ff9700 1 -- 172.21.15.98:0/3416937948 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f94c8052580 con 0x7f94d40cd5c0 2023-03-03T07:57:53.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.888+0000 7f94da9ff700 1 -- 172.21.15.98:0/3416937948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f94d404e6e0 con 0x7f94d40cd5c0 2023-03-03T07:57:53.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.889+0000 7f94d0ff9700 1 -- 172.21.15.98:0/3416937948 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v40) v1 ==== 104+0+11 (secure 0 0 0) 0x7f94c8021020 con 0x7f94d40cd5c0 2023-03-03T07:57:53.891 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:57:53.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.891+0000 7f94da9ff700 1 -- 172.21.15.98:0/3416937948 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f94c00683c0 msgr2=0x7f94c006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:53.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.891+0000 7f94da9ff700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f94c00683c0 0x7f94c006a870 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f94c40011e0 tx=0x7f94c4009040 comp rx=0 tx=0).stop 2023-03-03T07:57:53.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.891+0000 7f94da9ff700 1 -- 172.21.15.98:0/3416937948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40cd5c0 msgr2=0x7f94d41418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:53.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.891+0000 7f94da9ff700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40cd5c0 0x7f94d41418c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f94c800e9b0 tx=0x7f94c800ed70 comp rx=0 tx=0).stop 2023-03-03T07:57:53.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.891+0000 7f94da9ff700 1 -- 172.21.15.98:0/3416937948 shutdown_connections 2023-03-03T07:57:53.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.891+0000 7f94da9ff700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f94c00683c0 0x7f94c006a870 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.891+0000 7f94da9ff700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94d415dd20 0x7f94d4152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.891+0000 7f94da9ff700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94d40b0800 0x7f94d40adee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.891+0000 7f94da9ff700 1 --2- 172.21.15.98:0/3416937948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40cd5c0 0x7f94d41418c0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:53.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.891+0000 7f94da9ff700 1 -- 172.21.15.98:0/3416937948 >> 172.21.15.98:0/3416937948 conn(0x7f94d40cb800 msgr2=0x7f94d40ced80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:53.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.892+0000 7f94da9ff700 1 -- 172.21.15.98:0/3416937948 shutdown_connections 2023-03-03T07:57:53.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:53.892+0000 7f94da9ff700 1 -- 172.21.15.98:0/3416937948 wait complete. 2023-03-03T07:57:53.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:57:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.170+0000 7f200e5ff700 1 Processor -- start 2023-03-03T07:57:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.170+0000 7f200e5ff700 1 -- start start 2023-03-03T07:57:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.171+0000 7f200e5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f200815ca70 0x7f20080cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.172+0000 7f200e5ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f200806b4b0 0x7f20080cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.172+0000 7f200e5ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20080cf980 con 0x7f20080d00a0 2023-03-03T07:57:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.172+0000 7f200e5ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20080cd5b0 con 0x7f200806b4b0 2023-03-03T07:57:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.172+0000 7f200e5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20080d0510 con 0x7f200815ca70 2023-03-03T07:57:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.172+0000 7f200cb9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f20080d00a0 0x7f20080b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44268/0 (socket says 172.21.15.98:44268) 2023-03-03T07:57:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.172+0000 7f200cb9c700 1 -- 172.21.15.98:0/3187273328 learned_addr learned my addr 172.21.15.98:0/3187273328 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.172+0000 7f20077fe700 1 --2- 172.21.15.98:0/3187273328 >> v2:172.21.15.100:3300/0 conn(0x7f200806b4b0 0x7f20080cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.172+0000 7f2007fff700 1 --2- 172.21.15.98:0/3187273328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f200815ca70 0x7f20080cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.173+0000 7f2006ffd700 1 -- 172.21.15.98:0/3187273328 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234152626 0 0) 0x7f20080cf980 con 0x7f20080d00a0 2023-03-03T07:57:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.173+0000 7f2006ffd700 1 -- 172.21.15.98:0/3187273328 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ff00035b0 con 0x7f20080d00a0 2023-03-03T07:57:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.173+0000 7f20077fe700 1 -- 172.21.15.98:0/3187273328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f200815ca70 msgr2=0x7f20080cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.173+0000 7f20077fe700 1 --2- 172.21.15.98:0/3187273328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f200815ca70 0x7f20080cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.173+0000 7f20077fe700 1 -- 172.21.15.98:0/3187273328 >> v1:172.21.15.98:6789/0 conn(0x7f20080d00a0 legacy=0x7f20080b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.173+0000 7f20077fe700 1 -- 172.21.15.98:0/3187273328 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20080d2fd0 con 0x7f200806b4b0 2023-03-03T07:57:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.173+0000 7f20077fe700 1 --2- 172.21.15.98:0/3187273328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200806b4b0 0x7f20080cfb60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009840 tx=0x7f1ffc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a53cb949d668524 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.173+0000 7f2006ffd700 1 -- 172.21.15.98:0/3187273328 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc011830 con 0x7f200806b4b0 2023-03-03T07:57:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.173+0000 7f2006ffd700 1 -- 172.21.15.98:0/3187273328 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ffc011990 con 0x7f200806b4b0 2023-03-03T07:57:54.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.173+0000 7f2006ffd700 1 -- 172.21.15.98:0/3187273328 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc011d20 con 0x7f200806b4b0 2023-03-03T07:57:54.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.174+0000 7f200e5ff700 1 -- 172.21.15.98:0/3187273328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200806b4b0 msgr2=0x7f20080cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:54.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.174+0000 7f200e5ff700 1 --2- 172.21.15.98:0/3187273328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200806b4b0 0x7f20080cfb60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009840 tx=0x7f1ffc009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:54.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.174+0000 7f200e5ff700 1 -- 172.21.15.98:0/3187273328 shutdown_connections 2023-03-03T07:57:54.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.174+0000 7f200e5ff700 1 --2- 172.21.15.98:0/3187273328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f200815ca70 0x7f20080cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.174+0000 7f200e5ff700 1 --2- 172.21.15.98:0/3187273328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200806b4b0 0x7f20080cfb60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.174+0000 7f200e5ff700 1 -- 172.21.15.98:0/3187273328 >> 172.21.15.98:0/3187273328 conn(0x7f2008073e70 msgr2=0x7f200806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:54.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.174+0000 7f200e5ff700 1 -- 172.21.15.98:0/3187273328 shutdown_connections 2023-03-03T07:57:54.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.174+0000 7f200e5ff700 1 -- 172.21.15.98:0/3187273328 wait complete. 2023-03-03T07:57:54.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.176+0000 7f200e5ff700 1 Processor -- start 2023-03-03T07:57:54.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.176+0000 7f200e5ff700 1 -- start start 2023-03-03T07:57:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.177+0000 7f200e5ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f200806b4b0 0x7f20080b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.178+0000 7f200e5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20080d00a0 0x7f20081f1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.178+0000 7f200e5ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200815ca70 0x7f20081f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.178+0000 7f200e5ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20081f2390 con 0x7f200806b4b0 2023-03-03T07:57:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.178+0000 7f200e5ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20081f5a90 con 0x7f200815ca70 2023-03-03T07:57:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.178+0000 7f200e5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20081f6330 con 0x7f20080d00a0 2023-03-03T07:57:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.178+0000 7f2007fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f200806b4b0 0x7f20080b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.178+0000 7f2007fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f200806b4b0 0x7f20080b5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42612/0 (socket says 172.21.15.98:42612) 2023-03-03T07:57:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.178+0000 7f2007fff700 1 -- 172.21.15.98:0/3565380894 learned_addr learned my addr 172.21.15.98:0/3565380894 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.178+0000 7f200cb9c700 1 --2- 172.21.15.98:0/3565380894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200815ca70 0x7f20081f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.178+0000 7f20077fe700 1 --2- 172.21.15.98:0/3565380894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20080d00a0 0x7f20081f1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.178+0000 7f2007fff700 1 -- 172.21.15.98:0/3565380894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20080d00a0 msgr2=0x7f20081f1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.178+0000 7f2007fff700 1 --2- 172.21.15.98:0/3565380894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20080d00a0 0x7f20081f1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.178+0000 7f2007fff700 1 -- 172.21.15.98:0/3565380894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200815ca70 msgr2=0x7f20081f1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.178+0000 7f2007fff700 1 --2- 172.21.15.98:0/3565380894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200815ca70 0x7f20081f1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.178+0000 7f2007fff700 1 -- 172.21.15.98:0/3565380894 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ffc0094a0 con 0x7f200806b4b0 2023-03-03T07:57:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.179+0000 7f2007fff700 1 --2- 172.21.15.98:0/3565380894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f200806b4b0 0x7f20080b5020 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800e870 tx=0x7f1ff800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.179+0000 7f2004ff9700 1 -- 172.21.15.98:0/3565380894 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8009590 con 0x7f200806b4b0 2023-03-03T07:57:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.179+0000 7f2004ff9700 1 -- 172.21.15.98:0/3565380894 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff8013680 con 0x7f200806b4b0 2023-03-03T07:57:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.179+0000 7f2004ff9700 1 -- 172.21.15.98:0/3565380894 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff80139f0 con 0x7f200806b4b0 2023-03-03T07:57:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.179+0000 7f200e5ff700 1 -- 172.21.15.98:0/3565380894 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20080cd900 con 0x7f200806b4b0 2023-03-03T07:57:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.179+0000 7f200e5ff700 1 -- 172.21.15.98:0/3565380894 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20080d1800 con 0x7f200806b4b0 2023-03-03T07:57:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.180+0000 7f200e5ff700 1 -- 172.21.15.98:0/3565380894 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20080cdbe0 con 0x7f200806b4b0 2023-03-03T07:57:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.181+0000 7f2004ff9700 1 -- 172.21.15.98:0/3565380894 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f1ff8019030 con 0x7f200806b4b0 2023-03-03T07:57:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.183+0000 7f2004ff9700 1 --2- 172.21.15.98:0/3565380894 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1ff40683c0 0x7f1ff406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.183+0000 7f2004ff9700 1 -- 172.21.15.98:0/3565380894 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f1ff8085c10 con 0x7f200806b4b0 2023-03-03T07:57:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.183+0000 7f20077fe700 1 --2- 172.21.15.98:0/3565380894 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1ff40683c0 0x7f1ff406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.184+0000 7f20077fe700 1 --2- 172.21.15.98:0/3565380894 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1ff40683c0 0x7f1ff406a870 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009840 tx=0x7f1ffc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.186+0000 7f2004ff9700 1 -- 172.21.15.98:0/3565380894 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1ff8052230 con 0x7f200806b4b0 2023-03-03T07:57:54.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.384+0000 7f200e5ff700 1 -- 172.21.15.98:0/3565380894 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f20081f6610 con 0x7f200806b4b0 2023-03-03T07:57:54.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.384+0000 7f2004ff9700 1 -- 172.21.15.98:0/3565380894 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v40) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1ff8055850 con 0x7f200806b4b0 2023-03-03T07:57:54.387 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:57:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.387+0000 7f200e5ff700 1 -- 172.21.15.98:0/3565380894 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1ff40683c0 msgr2=0x7f1ff406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.387+0000 7f200e5ff700 1 --2- 172.21.15.98:0/3565380894 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1ff40683c0 0x7f1ff406a870 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009840 tx=0x7f1ffc008030 comp rx=0 tx=0).stop 2023-03-03T07:57:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.387+0000 7f200e5ff700 1 -- 172.21.15.98:0/3565380894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f200806b4b0 msgr2=0x7f20080b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.387+0000 7f200e5ff700 1 --2- 172.21.15.98:0/3565380894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f200806b4b0 0x7f20080b5020 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800e870 tx=0x7f1ff800eb80 comp rx=0 tx=0).stop 2023-03-03T07:57:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.387+0000 7f200e5ff700 1 -- 172.21.15.98:0/3565380894 shutdown_connections 2023-03-03T07:57:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.387+0000 7f200e5ff700 1 --2- 172.21.15.98:0/3565380894 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1ff40683c0 0x7f1ff406a870 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.387+0000 7f200e5ff700 1 --2- 172.21.15.98:0/3565380894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f200806b4b0 0x7f20080b5020 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.387+0000 7f200e5ff700 1 --2- 172.21.15.98:0/3565380894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200815ca70 0x7f20081f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.387+0000 7f200e5ff700 1 --2- 172.21.15.98:0/3565380894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20080d00a0 0x7f20081f1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.387+0000 7f200e5ff700 1 -- 172.21.15.98:0/3565380894 >> 172.21.15.98:0/3565380894 conn(0x7f2008073e70 msgr2=0x7f20080b6490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:54.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.387+0000 7f200e5ff700 1 -- 172.21.15.98:0/3565380894 shutdown_connections 2023-03-03T07:57:54.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.387+0000 7f200e5ff700 1 -- 172.21.15.98:0/3565380894 wait complete. 2023-03-03T07:57:54.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:57:54.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.663+0000 7fc98bfff700 1 Processor -- start 2023-03-03T07:57:54.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.664+0000 7fc98bfff700 1 -- start start 2023-03-03T07:57:54.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.665+0000 7fc98bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc98c160f80 0x7fc98c0d1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:54.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.665+0000 7fc98bfff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98c0d10f0 con 0x7fc98c146360 2023-03-03T07:57:54.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.665+0000 7fc98bfff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98c0cdf20 con 0x7fc98c0b51c0 2023-03-03T07:57:54.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.665+0000 7fc98bfff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98c0cd680 con 0x7fc98c160f80 2023-03-03T07:57:54.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.665+0000 7fc98affd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc98c146360 0x7fc98c0b4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44282/0 (socket says 172.21.15.98:44282) 2023-03-03T07:57:54.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.665+0000 7fc98affd700 1 -- 172.21.15.98:0/1577433706 learned_addr learned my addr 172.21.15.98:0/1577433706 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:54.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.666+0000 7fc98b7fe700 1 --2- 172.21.15.98:0/1577433706 >> v2:172.21.15.98:3300/0 conn(0x7fc98c160f80 0x7fc98c0d1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:54.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.666+0000 7fc989ffb700 1 -- 172.21.15.98:0/1577433706 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739345536 0 0) 0x7fc98c0d10f0 con 0x7fc98c146360 2023-03-03T07:57:54.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.666+0000 7fc989ffb700 1 -- 172.21.15.98:0/1577433706 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9740035b0 con 0x7fc98c146360 2023-03-03T07:57:54.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.666+0000 7fc98b7fe700 1 -- 172.21.15.98:0/1577433706 >> v1:172.21.15.100:6789/0 conn(0x7fc98c0b51c0 legacy=0x7fc98c0d1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:54.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.666+0000 7fc98b7fe700 1 -- 172.21.15.98:0/1577433706 >> v1:172.21.15.98:6789/0 conn(0x7fc98c146360 legacy=0x7fc98c0b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:54.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.666+0000 7fc98b7fe700 1 -- 172.21.15.98:0/1577433706 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98c0cd960 con 0x7fc98c160f80 2023-03-03T07:57:54.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.666+0000 7fc98b7fe700 1 --2- 172.21.15.98:0/1577433706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc98c160f80 0x7fc98c0d1c90 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc984009d90 tx=0x7fc98400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb58cab6976a11db server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:54.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.667+0000 7fc989ffb700 1 -- 172.21.15.98:0/1577433706 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98400e070 con 0x7fc98c160f80 2023-03-03T07:57:54.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.667+0000 7fc989ffb700 1 -- 172.21.15.98:0/1577433706 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc98400bb20 con 0x7fc98c160f80 2023-03-03T07:57:54.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.667+0000 7fc989ffb700 1 -- 172.21.15.98:0/1577433706 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc984012430 con 0x7fc98c160f80 2023-03-03T07:57:54.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.667+0000 7fc98bfff700 1 -- 172.21.15.98:0/1577433706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc98c160f80 msgr2=0x7fc98c0d1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:54.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.667+0000 7fc98bfff700 1 --2- 172.21.15.98:0/1577433706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc98c160f80 0x7fc98c0d1c90 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc984009d90 tx=0x7fc98400b330 comp rx=0 tx=0).stop 2023-03-03T07:57:54.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.667+0000 7fc98bfff700 1 -- 172.21.15.98:0/1577433706 shutdown_connections 2023-03-03T07:57:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.667+0000 7fc98bfff700 1 --2- 172.21.15.98:0/1577433706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc98c160f80 0x7fc98c0d1c90 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.667+0000 7fc98bfff700 1 -- 172.21.15.98:0/1577433706 >> 172.21.15.98:0/1577433706 conn(0x7fc98c06b8e0 msgr2=0x7fc98c0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.667+0000 7fc98bfff700 1 -- 172.21.15.98:0/1577433706 shutdown_connections 2023-03-03T07:57:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.668+0000 7fc98bfff700 1 -- 172.21.15.98:0/1577433706 wait complete. 2023-03-03T07:57:54.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.669+0000 7fc98bfff700 1 Processor -- start 2023-03-03T07:57:54.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.670+0000 7fc98bfff700 1 -- start start 2023-03-03T07:57:54.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98bfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc98c0b51c0 0x7fc98c153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:54.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc98c146360 0x7fc98c0c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:54.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98bfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc98c160f80 0x7fc98c0c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:54.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98bfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c0ccea0 con 0x7fc98c146360 2023-03-03T07:57:54.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98bfff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c0cafa0 con 0x7fc98c0b51c0 2023-03-03T07:57:54.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98bfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c0c6810 con 0x7fc98c160f80 2023-03-03T07:57:54.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc98c146360 0x7fc98c0c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:54.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc98c146360 0x7fc98c0c8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42634/0 (socket says 172.21.15.98:42634) 2023-03-03T07:57:54.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98a7fc700 1 -- 172.21.15.98:0/1949550074 learned_addr learned my addr 172.21.15.98:0/1949550074 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:54.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98affd700 1 --2- 172.21.15.98:0/1949550074 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc98c0b51c0 0x7fc98c153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:54.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98b7fe700 1 --2- 172.21.15.98:0/1949550074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc98c160f80 0x7fc98c0c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:54.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98a7fc700 1 -- 172.21.15.98:0/1949550074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc98c160f80 msgr2=0x7fc98c0c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:54.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98a7fc700 1 --2- 172.21.15.98:0/1949550074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc98c160f80 0x7fc98c0c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98a7fc700 1 -- 172.21.15.98:0/1949550074 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc98c0b51c0 msgr2=0x7fc98c153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:54.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98a7fc700 1 --2- 172.21.15.98:0/1949550074 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc98c0b51c0 0x7fc98c153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.671+0000 7fc98a7fc700 1 -- 172.21.15.98:0/1949550074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc984009a70 con 0x7fc98c146360 2023-03-03T07:57:54.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.672+0000 7fc98a7fc700 1 --2- 172.21.15.98:0/1949550074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc98c146360 0x7fc98c0c8d10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc98000c420 tx=0x7fc98000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:54.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.672+0000 7fc9737fe700 1 -- 172.21.15.98:0/1949550074 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9800052d0 con 0x7fc98c146360 2023-03-03T07:57:54.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.672+0000 7fc9737fe700 1 -- 172.21.15.98:0/1949550074 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc980005430 con 0x7fc98c146360 2023-03-03T07:57:54.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.672+0000 7fc98bfff700 1 -- 172.21.15.98:0/1949550074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc98c0d0d80 con 0x7fc98c146360 2023-03-03T07:57:54.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.672+0000 7fc9737fe700 1 -- 172.21.15.98:0/1949550074 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9800038c0 con 0x7fc98c146360 2023-03-03T07:57:54.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.672+0000 7fc98bfff700 1 -- 172.21.15.98:0/1949550074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98c0cdc40 con 0x7fc98c146360 2023-03-03T07:57:54.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.673+0000 7fc98bfff700 1 -- 172.21.15.98:0/1949550074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98c161380 con 0x7fc98c146360 2023-03-03T07:57:54.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.675+0000 7fc9737fe700 1 -- 172.21.15.98:0/1949550074 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fc980019030 con 0x7fc98c146360 2023-03-03T07:57:54.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.676+0000 7fc9737fe700 1 --2- 172.21.15.98:0/1949550074 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc978068390 0x7fc97806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:54.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.676+0000 7fc9737fe700 1 -- 172.21.15.98:0/1949550074 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fc980085180 con 0x7fc98c146360 2023-03-03T07:57:54.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.676+0000 7fc98affd700 1 --2- 172.21.15.98:0/1949550074 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc978068390 0x7fc97806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:54.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.677+0000 7fc98affd700 1 --2- 172.21.15.98:0/1949550074 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc978068390 0x7fc97806a840 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0011e0 tx=0x7fc97c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.680+0000 7fc9737fe700 1 -- 172.21.15.98:0/1949550074 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc980051800 con 0x7fc98c146360 2023-03-03T07:57:54.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.916+0000 7fc98bfff700 1 -- 172.21.15.98:0/1949550074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc98c0ccfe0 con 0x7fc98c146360 2023-03-03T07:57:54.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.916+0000 7fc9737fe700 1 -- 172.21.15.98:0/1949550074 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4654 (secure 0 0 0) 0x7fc98c0ccfe0 con 0x7fc98c146360 2023-03-03T07:57:54.919 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:54.919 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3849166495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3849166495},{"type":"v1","addr":"172.21.15.100:6834","nonce":3849166495}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/4285491249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":4285491249},{"type":"v1","addr":"172.21.15.100:6835","nonce":4285491249}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":5208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3888598890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3888598890},{"type":"v1","addr":"172.21.15.98:6836","nonce":3888598890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:57:17.291546+0000","modified":"2023-03-03T07:57:20.741013+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5227},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5227":{"gid":5227,"name":"c","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.98:6837/1163251191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1163251191},{"type":"v1","addr":"172.21.15.98:6837","nonce":1163251191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T07:57:54.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.919+0000 7fc98bfff700 1 -- 172.21.15.98:0/1949550074 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc978068390 msgr2=0x7fc97806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:54.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.919+0000 7fc98bfff700 1 --2- 172.21.15.98:0/1949550074 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc978068390 0x7fc97806a840 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0011e0 tx=0x7fc97c009040 comp rx=0 tx=0).stop 2023-03-03T07:57:54.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.919+0000 7fc98bfff700 1 -- 172.21.15.98:0/1949550074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc98c146360 msgr2=0x7fc98c0c8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:54.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.919+0000 7fc98bfff700 1 --2- 172.21.15.98:0/1949550074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc98c146360 0x7fc98c0c8d10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc98000c420 tx=0x7fc98000c7e0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.919+0000 7fc98bfff700 1 -- 172.21.15.98:0/1949550074 shutdown_connections 2023-03-03T07:57:54.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.919+0000 7fc98bfff700 1 --2- 172.21.15.98:0/1949550074 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc978068390 0x7fc97806a840 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.919+0000 7fc98bfff700 1 --2- 172.21.15.98:0/1949550074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc98c160f80 0x7fc98c0c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.919+0000 7fc98bfff700 1 --2- 172.21.15.98:0/1949550074 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc98c0b51c0 0x7fc98c153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.919+0000 7fc98bfff700 1 --2- 172.21.15.98:0/1949550074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc98c146360 0x7fc98c0c8d10 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.919+0000 7fc98bfff700 1 -- 172.21.15.98:0/1949550074 >> 172.21.15.98:0/1949550074 conn(0x7fc98c06b8e0 msgr2=0x7fc98c0d22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.919+0000 7fc98bfff700 1 -- 172.21.15.98:0/1949550074 shutdown_connections 2023-03-03T07:57:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:54.919+0000 7fc98bfff700 1 -- 172.21.15.98:0/1949550074 wait complete. 2023-03-03T07:57:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 15 2023-03-03T07:57:54.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:57:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.190+0000 7f090a7ff700 1 Processor -- start 2023-03-03T07:57:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.191+0000 7f090a7ff700 1 -- start start 2023-03-03T07:57:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.192+0000 7f090a7ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f090415cc00 0x7f09040cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.192+0000 7f090a7ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09040cde10 con 0x7f09040d2c60 2023-03-03T07:57:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.192+0000 7f090a7ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09040d1a00 con 0x7f09040d27c0 2023-03-03T07:57:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.192+0000 7f090a7ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09040ce040 con 0x7f090415cc00 2023-03-03T07:57:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.192+0000 7f0903fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f090415cc00 0x7f09040cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.192+0000 7f0908d9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f09040d2c60 0x7f09040cf4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58938/0 (socket says 172.21.15.98:58938) 2023-03-03T07:57:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.192+0000 7f0908d9c700 1 -- 172.21.15.98:0/275251600 learned_addr learned my addr 172.21.15.98:0/275251600 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.193+0000 7f0903fff700 1 -- 172.21.15.98:0/275251600 >> v1:172.21.15.100:6789/0 conn(0x7f09040d27c0 legacy=0x7f09040b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.193+0000 7f0903fff700 1 -- 172.21.15.98:0/275251600 >> v1:172.21.15.98:6789/0 conn(0x7f09040d2c60 legacy=0x7f09040cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.193+0000 7f0903fff700 1 -- 172.21.15.98:0/275251600 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09040d1ce0 con 0x7f090415cc00 2023-03-03T07:57:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.193+0000 7f0903fff700 1 --2- 172.21.15.98:0/275251600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090415cc00 0x7f09040cd8d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f08f4009840 tx=0x7f08f4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2359981b10bc705d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.193+0000 7f0902ffd700 1 -- 172.21.15.98:0/275251600 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4011830 con 0x7f090415cc00 2023-03-03T07:57:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.193+0000 7f0902ffd700 1 -- 172.21.15.98:0/275251600 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f4011990 con 0x7f090415cc00 2023-03-03T07:57:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.194+0000 7f0902ffd700 1 -- 172.21.15.98:0/275251600 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4011d20 con 0x7f090415cc00 2023-03-03T07:57:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.194+0000 7f090a7ff700 1 -- 172.21.15.98:0/275251600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090415cc00 msgr2=0x7f09040cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.194+0000 7f090a7ff700 1 --2- 172.21.15.98:0/275251600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090415cc00 0x7f09040cd8d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f08f4009840 tx=0x7f08f4009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.194+0000 7f090a7ff700 1 -- 172.21.15.98:0/275251600 shutdown_connections 2023-03-03T07:57:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.194+0000 7f090a7ff700 1 --2- 172.21.15.98:0/275251600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090415cc00 0x7f09040cd8d0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.194+0000 7f090a7ff700 1 -- 172.21.15.98:0/275251600 >> 172.21.15.98:0/275251600 conn(0x7f090406aee0 msgr2=0x7f090406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.194+0000 7f090a7ff700 1 -- 172.21.15.98:0/275251600 shutdown_connections 2023-03-03T07:57:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.194+0000 7f090a7ff700 1 -- 172.21.15.98:0/275251600 wait complete. 2023-03-03T07:57:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.196+0000 7f090a7ff700 1 Processor -- start 2023-03-03T07:57:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.197+0000 7f090a7ff700 1 -- start start 2023-03-03T07:57:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.198+0000 7f090a7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09040d27c0 0x7f09041504f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.198+0000 7f090a7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09040d2c60 0x7f09040b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.198+0000 7f090a7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f090415cc00 0x7f090415a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.198+0000 7f090a7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0904150a30 con 0x7f09040d2c60 2023-03-03T07:57:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.198+0000 7f090a7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09040cb6a0 con 0x7f09040d27c0 2023-03-03T07:57:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.198+0000 7f090a7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09040cb980 con 0x7f090415cc00 2023-03-03T07:57:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.198+0000 7f09037fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09040d2c60 0x7f09040b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.198+0000 7f09037fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09040d2c60 0x7f09040b4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48622/0 (socket says 172.21.15.98:48622) 2023-03-03T07:57:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.198+0000 7f09037fe700 1 -- 172.21.15.98:0/219581296 learned_addr learned my addr 172.21.15.98:0/219581296 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.198+0000 7f0903fff700 1 --2- 172.21.15.98:0/219581296 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09040d27c0 0x7f09041504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.198+0000 7f0908d9c700 1 --2- 172.21.15.98:0/219581296 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f090415cc00 0x7f090415a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.198+0000 7f09037fe700 1 -- 172.21.15.98:0/219581296 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f090415cc00 msgr2=0x7f090415a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.198+0000 7f09037fe700 1 --2- 172.21.15.98:0/219581296 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f090415cc00 0x7f090415a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.199+0000 7f09037fe700 1 -- 172.21.15.98:0/219581296 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09040d27c0 msgr2=0x7f09041504f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.199+0000 7f09037fe700 1 --2- 172.21.15.98:0/219581296 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09040d27c0 0x7f09041504f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.199+0000 7f09037fe700 1 -- 172.21.15.98:0/219581296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f40094a0 con 0x7f09040d2c60 2023-03-03T07:57:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.199+0000 7f09037fe700 1 --2- 172.21.15.98:0/219581296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09040d2c60 0x7f09040b4de0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f08f800c450 tx=0x7f08f800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.199+0000 7f0900ff9700 1 -- 172.21.15.98:0/219581296 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f80052d0 con 0x7f09040d2c60 2023-03-03T07:57:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.199+0000 7f0900ff9700 1 -- 172.21.15.98:0/219581296 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f8005430 con 0x7f09040d2c60 2023-03-03T07:57:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.199+0000 7f0900ff9700 1 -- 172.21.15.98:0/219581296 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f80038c0 con 0x7f09040d2c60 2023-03-03T07:57:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.199+0000 7f090a7ff700 1 -- 172.21.15.98:0/219581296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09040d22a0 con 0x7f09040d2c60 2023-03-03T07:57:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.199+0000 7f090a7ff700 1 -- 172.21.15.98:0/219581296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09040d1fc0 con 0x7f09040d2c60 2023-03-03T07:57:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.200+0000 7f090a7ff700 1 -- 172.21.15.98:0/219581296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09040b67e0 con 0x7f09040d2c60 2023-03-03T07:57:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.202+0000 7f0900ff9700 1 -- 172.21.15.98:0/219581296 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f08f8019030 con 0x7f09040d2c60 2023-03-03T07:57:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.204+0000 7f0900ff9700 1 --2- 172.21.15.98:0/219581296 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f08f00683c0 0x7f08f006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.204+0000 7f0900ff9700 1 -- 172.21.15.98:0/219581296 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f08f8085110 con 0x7f09040d2c60 2023-03-03T07:57:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.204+0000 7f0903fff700 1 --2- 172.21.15.98:0/219581296 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f08f00683c0 0x7f08f006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.205+0000 7f0903fff700 1 --2- 172.21.15.98:0/219581296 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f08f00683c0 0x7f08f006a870 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f08f40090b0 tx=0x7f08f4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.208+0000 7f0900ff9700 1 -- 172.21.15.98:0/219581296 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f08f8051790 con 0x7f09040d2c60 2023-03-03T07:57:55.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.404+0000 7f090a7ff700 1 -- 172.21.15.98:0/219581296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09040729b0 con 0x7f09040d2c60 2023-03-03T07:57:55.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.405+0000 7f0900ff9700 1 -- 172.21.15.98:0/219581296 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+17972 (secure 0 0 0) 0x7f08f8017090 con 0x7f09040d2c60 2023-03-03T07:57:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:55.408 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":40,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:57:28.259781+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:57:15.545608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-03-03T07:57:16.283131+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.31","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:57:55.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.408+0000 7f090a7ff700 1 -- 172.21.15.98:0/219581296 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f08f00683c0 msgr2=0x7f08f006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:55.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.408+0000 7f090a7ff700 1 --2- 172.21.15.98:0/219581296 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f08f00683c0 0x7f08f006a870 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f08f40090b0 tx=0x7f08f4008030 comp rx=0 tx=0).stop 2023-03-03T07:57:55.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.408+0000 7f090a7ff700 1 -- 172.21.15.98:0/219581296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09040d2c60 msgr2=0x7f09040b4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:55.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.408+0000 7f090a7ff700 1 --2- 172.21.15.98:0/219581296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09040d2c60 0x7f09040b4de0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f08f800c450 tx=0x7f08f800c760 comp rx=0 tx=0).stop 2023-03-03T07:57:55.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.408+0000 7f090a7ff700 1 -- 172.21.15.98:0/219581296 shutdown_connections 2023-03-03T07:57:55.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.408+0000 7f090a7ff700 1 --2- 172.21.15.98:0/219581296 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f08f00683c0 0x7f08f006a870 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.408+0000 7f090a7ff700 1 --2- 172.21.15.98:0/219581296 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f090415cc00 0x7f090415a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.408+0000 7f090a7ff700 1 --2- 172.21.15.98:0/219581296 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09040d27c0 0x7f09041504f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.408+0000 7f090a7ff700 1 --2- 172.21.15.98:0/219581296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09040d2c60 0x7f09040b4de0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.408+0000 7f090a7ff700 1 -- 172.21.15.98:0/219581296 >> 172.21.15.98:0/219581296 conn(0x7f090406aee0 msgr2=0x7f09040b7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:55.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.408+0000 7f090a7ff700 1 -- 172.21.15.98:0/219581296 shutdown_connections 2023-03-03T07:57:55.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.408+0000 7f090a7ff700 1 -- 172.21.15.98:0/219581296 wait complete. 2023-03-03T07:57:55.437 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-03-03T07:57:55.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:57:55.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.683+0000 7f8b5d19d700 1 Processor -- start 2023-03-03T07:57:55.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.684+0000 7f8b5d19d700 1 -- start start 2023-03-03T07:57:55.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.686+0000 7f8b5d19d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8b580d0010 0x7f8b580d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:55.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.686+0000 7f8b5d19d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8b580b4a00 0x7f8b580d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:55.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.686+0000 7f8b5d19d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b580b0c50 con 0x7f8b580d1480 2023-03-03T07:57:55.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.686+0000 7f8b5d19d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b580d18a0 con 0x7f8b580d0010 2023-03-03T07:57:55.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.686+0000 7f8b5d19d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b580b0040 con 0x7f8b580b4a00 2023-03-03T07:57:55.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.686+0000 7f8b57fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8b580d0010 0x7f8b580d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:55.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.686+0000 7f8b57fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8b580d0010 0x7f8b580d03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48628/0 (socket says 172.21.15.98:48628) 2023-03-03T07:57:55.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.686+0000 7f8b57fff700 1 -- 172.21.15.98:0/3123151299 learned_addr learned my addr 172.21.15.98:0/3123151299 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:55.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.687+0000 7f8b57fff700 1 -- 172.21.15.98:0/3123151299 >> v2:172.21.15.100:3300/0 conn(0x7f8b580b4a00 msgr2=0x7f8b580d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:55.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.687+0000 7f8b57fff700 1 --2- 172.21.15.98:0/3123151299 >> v2:172.21.15.100:3300/0 conn(0x7f8b580b4a00 0x7f8b580d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.687+0000 7f8b57fff700 1 -- 172.21.15.98:0/3123151299 >> v1:172.21.15.100:6789/0 conn(0x7f8b580d1480 legacy=0x7f8b580b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:55.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.687+0000 7f8b57fff700 1 -- 172.21.15.98:0/3123151299 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b580b0320 con 0x7f8b580d0010 2023-03-03T07:57:55.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.687+0000 7f8b57fff700 1 --2- 172.21.15.98:0/3123151299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b580d0010 0x7f8b580d03e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8b5808c300 tx=0x7f8b40009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96f7de807aad80d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:55.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.687+0000 7f8b577fe700 1 -- 172.21.15.98:0/3123151299 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40005e30 con 0x7f8b580d0010 2023-03-03T07:57:55.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.687+0000 7f8b577fe700 1 -- 172.21.15.98:0/3123151299 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b40006030 con 0x7f8b580d0010 2023-03-03T07:57:55.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.687+0000 7f8b577fe700 1 -- 172.21.15.98:0/3123151299 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b400063e0 con 0x7f8b580d0010 2023-03-03T07:57:55.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.688+0000 7f8b5d19d700 1 -- 172.21.15.98:0/3123151299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b580d0010 msgr2=0x7f8b580d03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:55.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.688+0000 7f8b5d19d700 1 --2- 172.21.15.98:0/3123151299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b580d0010 0x7f8b580d03e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8b5808c300 tx=0x7f8b40009bd0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.688+0000 7f8b5d19d700 1 -- 172.21.15.98:0/3123151299 shutdown_connections 2023-03-03T07:57:55.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.688+0000 7f8b5d19d700 1 --2- 172.21.15.98:0/3123151299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b580d0010 0x7f8b580d03e0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.688+0000 7f8b5d19d700 1 --2- 172.21.15.98:0/3123151299 >> v2:172.21.15.100:3300/0 conn(0x7f8b580b4a00 0x7f8b580d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.688+0000 7f8b5d19d700 1 -- 172.21.15.98:0/3123151299 >> 172.21.15.98:0/3123151299 conn(0x7f8b58073b00 msgr2=0x7f8b580d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:55.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.688+0000 7f8b5d19d700 1 -- 172.21.15.98:0/3123151299 shutdown_connections 2023-03-03T07:57:55.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.688+0000 7f8b5d19d700 1 -- 172.21.15.98:0/3123151299 wait complete. 2023-03-03T07:57:55.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.691+0000 7f8b5d19d700 1 Processor -- start 2023-03-03T07:57:55.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.691+0000 7f8b5d19d700 1 -- start start 2023-03-03T07:57:55.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.693+0000 7f8b5d19d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b580b4a00 0x7f8b58148620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:55.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.693+0000 7f8b5d19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b580d0010 0x7f8b580c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:55.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.693+0000 7f8b5d19d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b580d1480 0x7f8b580c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:55.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.693+0000 7f8b5d19d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b580c8880 con 0x7f8b580b4a00 2023-03-03T07:57:55.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.693+0000 7f8b5d19d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b580c7870 con 0x7f8b580d1480 2023-03-03T07:57:55.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.693+0000 7f8b5d19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b580c7ac0 con 0x7f8b580d0010 2023-03-03T07:57:55.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.693+0000 7f8b57fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b580b4a00 0x7f8b58148620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:55.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.693+0000 7f8b57fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b580b4a00 0x7f8b58148620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48644/0 (socket says 172.21.15.98:48644) 2023-03-03T07:57:55.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.693+0000 7f8b57fff700 1 -- 172.21.15.98:0/23836269 learned_addr learned my addr 172.21.15.98:0/23836269 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:55.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.693+0000 7f8b4ffff700 1 --2- 172.21.15.98:0/23836269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b580d0010 0x7f8b580c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:55.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.693+0000 7f8b5c99c700 1 --2- 172.21.15.98:0/23836269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b580d1480 0x7f8b580c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:55.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.694+0000 7f8b57fff700 1 -- 172.21.15.98:0/23836269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b580d0010 msgr2=0x7f8b580c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:55.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.694+0000 7f8b57fff700 1 --2- 172.21.15.98:0/23836269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b580d0010 0x7f8b580c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.694+0000 7f8b57fff700 1 -- 172.21.15.98:0/23836269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b580d1480 msgr2=0x7f8b580c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:55.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.694+0000 7f8b57fff700 1 --2- 172.21.15.98:0/23836269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b580d1480 0x7f8b580c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.694+0000 7f8b57fff700 1 -- 172.21.15.98:0/23836269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b40009500 con 0x7f8b580b4a00 2023-03-03T07:57:55.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.694+0000 7f8b4ffff700 1 --2- 172.21.15.98:0/23836269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b580d0010 0x7f8b580c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:55.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.694+0000 7f8b57fff700 1 --2- 172.21.15.98:0/23836269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b580b4a00 0x7f8b58148620 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f8b40006000 tx=0x7f8b40004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:55.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.694+0000 7f8b557fa700 1 -- 172.21.15.98:0/23836269 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40025480 con 0x7f8b580b4a00 2023-03-03T07:57:55.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.694+0000 7f8b557fa700 1 -- 172.21.15.98:0/23836269 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b40020070 con 0x7f8b580b4a00 2023-03-03T07:57:55.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.694+0000 7f8b557fa700 1 -- 172.21.15.98:0/23836269 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40025820 con 0x7f8b580b4a00 2023-03-03T07:57:55.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.694+0000 7f8b5d19d700 1 -- 172.21.15.98:0/23836269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b580b08e0 con 0x7f8b580b4a00 2023-03-03T07:57:55.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.695+0000 7f8b5d19d700 1 -- 172.21.15.98:0/23836269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b580b0600 con 0x7f8b580b4a00 2023-03-03T07:57:55.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.696+0000 7f8b5d19d700 1 -- 172.21.15.98:0/23836269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b580d0b30 con 0x7f8b580b4a00 2023-03-03T07:57:55.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.697+0000 7f8b557fa700 1 -- 172.21.15.98:0/23836269 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f8b4000f050 con 0x7f8b580b4a00 2023-03-03T07:57:55.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.698+0000 7f8b557fa700 1 --2- 172.21.15.98:0/23836269 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8b380683c0 0x7f8b3806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:55.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.698+0000 7f8b557fa700 1 -- 172.21.15.98:0/23836269 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f8b40087820 con 0x7f8b580b4a00 2023-03-03T07:57:55.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.698+0000 7f8b4ffff700 1 --2- 172.21.15.98:0/23836269 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8b380683c0 0x7f8b3806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:55.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.699+0000 7f8b4ffff700 1 --2- 172.21.15.98:0/23836269 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8b380683c0 0x7f8b3806a870 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8b5815c290 tx=0x7f8b48008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:55.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.702+0000 7f8b557fa700 1 -- 172.21.15.98:0/23836269 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b40053f50 con 0x7f8b580b4a00 2023-03-03T07:57:55.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.938+0000 7f8b5d19d700 1 -- 172.21.15.98:0/23836269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b580729b0 con 0x7f8b580b4a00 2023-03-03T07:57:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.939+0000 7f8b557fa700 1 -- 172.21.15.98:0/23836269 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f8b40057570 con 0x7f8b580b4a00 2023-03-03T07:57:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:55.942 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3849166495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3849166495},{"type":"v1","addr":"172.21.15.100:6834","nonce":3849166495}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/4285491249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":4285491249},{"type":"v1","addr":"172.21.15.100:6835","nonce":4285491249}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":5208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3888598890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3888598890},{"type":"v1","addr":"172.21.15.98:6836","nonce":3888598890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:57:17.291546+0000","modified":"2023-03-03T07:57:20.741013+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5227},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5227":{"gid":5227,"name":"c","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.98:6837/1163251191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1163251191},{"type":"v1","addr":"172.21.15.98:6837","nonce":1163251191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T07:57:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.941+0000 7f8b5d19d700 1 -- 172.21.15.98:0/23836269 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8b380683c0 msgr2=0x7f8b3806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.941+0000 7f8b5d19d700 1 --2- 172.21.15.98:0/23836269 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8b380683c0 0x7f8b3806a870 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8b5815c290 tx=0x7f8b48008040 comp rx=0 tx=0).stop 2023-03-03T07:57:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.941+0000 7f8b5d19d700 1 -- 172.21.15.98:0/23836269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b580b4a00 msgr2=0x7f8b58148620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.941+0000 7f8b5d19d700 1 --2- 172.21.15.98:0/23836269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b580b4a00 0x7f8b58148620 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f8b40006000 tx=0x7f8b40004750 comp rx=0 tx=0).stop 2023-03-03T07:57:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.941+0000 7f8b5d19d700 1 -- 172.21.15.98:0/23836269 shutdown_connections 2023-03-03T07:57:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.941+0000 7f8b5d19d700 1 --2- 172.21.15.98:0/23836269 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8b380683c0 0x7f8b3806a870 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.941+0000 7f8b5d19d700 1 --2- 172.21.15.98:0/23836269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b580b4a00 0x7f8b58148620 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.941+0000 7f8b5d19d700 1 --2- 172.21.15.98:0/23836269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b580d1480 0x7f8b580c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.941+0000 7f8b5d19d700 1 --2- 172.21.15.98:0/23836269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b580d0010 0x7f8b580c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:55.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.941+0000 7f8b5d19d700 1 -- 172.21.15.98:0/23836269 >> 172.21.15.98:0/23836269 conn(0x7f8b58073b00 msgr2=0x7f8b5806c700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:55.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.941+0000 7f8b5d19d700 1 -- 172.21.15.98:0/23836269 shutdown_connections 2023-03-03T07:57:55.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:55.941+0000 7f8b5d19d700 1 -- 172.21.15.98:0/23836269 wait complete. 2023-03-03T07:57:55.945 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 15 2023-03-03T07:57:55.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:57:56.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.213+0000 7f6fd899d700 1 Processor -- start 2023-03-03T07:57:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.213+0000 7f6fd899d700 1 -- start start 2023-03-03T07:57:56.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.215+0000 7f6fd899d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fd40b51c0 0x7f6fd4160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:56.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.215+0000 7f6fd899d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd41614c0 con 0x7f6fd40d1580 2023-03-03T07:57:56.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.215+0000 7f6fd899d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd40d10f0 con 0x7f6fd4146360 2023-03-03T07:57:56.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.215+0000 7f6fd899d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40cdf20 con 0x7f6fd40b51c0 2023-03-03T07:57:56.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.215+0000 7f6fd3fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6fd40d1580 0x7f6fd40d19e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58948/0 (socket says 172.21.15.98:58948) 2023-03-03T07:57:56.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.215+0000 7f6fd3fff700 1 -- 172.21.15.98:0/2407269359 learned_addr learned my addr 172.21.15.98:0/2407269359 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.215+0000 7f6fd2ffd700 1 --2- 172.21.15.98:0/2407269359 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fd40b51c0 0x7f6fd4160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.216+0000 7f6fd27fc700 1 -- 172.21.15.98:0/2407269359 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158609661 0 0) 0x7f6fcc001490 con 0x7f6fd40d1580 2023-03-03T07:57:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.216+0000 7f6fd27fc700 1 -- 172.21.15.98:0/2407269359 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fbc0035b0 con 0x7f6fd40d1580 2023-03-03T07:57:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.216+0000 7f6fd27fc700 1 -- 172.21.15.98:0/2407269359 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903044593 0 0) 0x7f6fd40d10f0 con 0x7f6fd4146360 2023-03-03T07:57:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.216+0000 7f6fd27fc700 1 -- 172.21.15.98:0/2407269359 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fcc001490 con 0x7f6fd4146360 2023-03-03T07:57:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.216+0000 7f6fd2ffd700 1 -- 172.21.15.98:0/2407269359 >> v1:172.21.15.100:6789/0 conn(0x7f6fd4146360 legacy=0x7f6fd40b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.216+0000 7f6fd27fc700 1 -- 172.21.15.98:0/2407269359 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 573088633 0 0) 0x7f6fbc0035b0 con 0x7f6fd40d1580 2023-03-03T07:57:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.216+0000 7f6fd2ffd700 1 -- 172.21.15.98:0/2407269359 >> v1:172.21.15.98:6789/0 conn(0x7f6fd40d1580 legacy=0x7f6fd40d19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.216+0000 7f6fd2ffd700 1 -- 172.21.15.98:0/2407269359 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd40cd680 con 0x7f6fd40b51c0 2023-03-03T07:57:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.216+0000 7f6fd2ffd700 1 --2- 172.21.15.98:0/2407269359 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fd40b51c0 0x7f6fd4160f80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009870 tx=0x7f6fc8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9142e265b731c262 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.217+0000 7f6fd27fc700 1 -- 172.21.15.98:0/2407269359 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc001b80 con 0x7f6fd40b51c0 2023-03-03T07:57:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.217+0000 7f6fd27fc700 1 -- 172.21.15.98:0/2407269359 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc8004030 con 0x7f6fd40b51c0 2023-03-03T07:57:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.217+0000 7f6fd27fc700 1 -- 172.21.15.98:0/2407269359 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc80043c0 con 0x7f6fd40b51c0 2023-03-03T07:57:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.217+0000 7f6fd899d700 1 -- 172.21.15.98:0/2407269359 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fd40b51c0 msgr2=0x7f6fd4160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.217+0000 7f6fd899d700 1 --2- 172.21.15.98:0/2407269359 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fd40b51c0 0x7f6fd4160f80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009870 tx=0x7f6fc8009b80 comp rx=0 tx=0).stop 2023-03-03T07:57:56.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.217+0000 7f6fd899d700 1 -- 172.21.15.98:0/2407269359 shutdown_connections 2023-03-03T07:57:56.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.217+0000 7f6fd899d700 1 --2- 172.21.15.98:0/2407269359 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fd40b51c0 0x7f6fd4160f80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.217+0000 7f6fd899d700 1 -- 172.21.15.98:0/2407269359 >> 172.21.15.98:0/2407269359 conn(0x7f6fd406b8e0 msgr2=0x7f6fd40cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:56.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.217+0000 7f6fd899d700 1 -- 172.21.15.98:0/2407269359 shutdown_connections 2023-03-03T07:57:56.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.217+0000 7f6fd899d700 1 -- 172.21.15.98:0/2407269359 wait complete. 2023-03-03T07:57:56.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.219+0000 7f6fd899d700 1 Processor -- start 2023-03-03T07:57:56.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.219+0000 7f6fd899d700 1 -- start start 2023-03-03T07:57:56.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.220+0000 7f6fd899d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fd40b51c0 0x7f6fd4154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:56.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.220+0000 7f6fd899d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fd40d1580 0x7f6fd40c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:56.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.220+0000 7f6fd899d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fd4146360 0x7f6fd40b8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:56.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.220+0000 7f6fd899d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40c9a70 con 0x7f6fd40b51c0 2023-03-03T07:57:56.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.220+0000 7f6fd899d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40c95e0 con 0x7f6fd4146360 2023-03-03T07:57:56.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.220+0000 7f6fd899d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40b3a70 con 0x7f6fd40d1580 2023-03-03T07:57:56.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.221+0000 7f6fd37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fd40b51c0 0x7f6fd4154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:56.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.221+0000 7f6fd37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fd40b51c0 0x7f6fd4154510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48654/0 (socket says 172.21.15.98:48654) 2023-03-03T07:57:56.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.221+0000 7f6fd37fe700 1 -- 172.21.15.98:0/2852263847 learned_addr learned my addr 172.21.15.98:0/2852263847 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:56.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.221+0000 7f6fd3fff700 1 --2- 172.21.15.98:0/2852263847 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fd4146360 0x7f6fd40b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:56.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.221+0000 7f6fd2ffd700 1 --2- 172.21.15.98:0/2852263847 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fd40d1580 0x7f6fd40c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.221+0000 7f6fd37fe700 1 -- 172.21.15.98:0/2852263847 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fd40d1580 msgr2=0x7f6fd40c6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.221+0000 7f6fd37fe700 1 --2- 172.21.15.98:0/2852263847 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fd40d1580 0x7f6fd40c6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.221+0000 7f6fd37fe700 1 -- 172.21.15.98:0/2852263847 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fd4146360 msgr2=0x7f6fd40b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.221+0000 7f6fd37fe700 1 --2- 172.21.15.98:0/2852263847 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fd4146360 0x7f6fd40b8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.221+0000 7f6fd37fe700 1 -- 172.21.15.98:0/2852263847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc80094a0 con 0x7f6fd40b51c0 2023-03-03T07:57:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.222+0000 7f6fd37fe700 1 --2- 172.21.15.98:0/2852263847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fd40b51c0 0x7f6fd4154510 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400c520 tx=0x7f6fc400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:56.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.222+0000 7f6fbbfff700 1 -- 172.21.15.98:0/2852263847 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc40052d0 con 0x7f6fd40b51c0 2023-03-03T07:57:56.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.222+0000 7f6fbbfff700 1 -- 172.21.15.98:0/2852263847 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc4005430 con 0x7f6fd40b51c0 2023-03-03T07:57:56.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.222+0000 7f6fbbfff700 1 -- 172.21.15.98:0/2852263847 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc40038c0 con 0x7f6fd40b51c0 2023-03-03T07:57:56.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.222+0000 7f6fd899d700 1 -- 172.21.15.98:0/2852263847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd40cdc40 con 0x7f6fd40b51c0 2023-03-03T07:57:56.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.222+0000 7f6fd899d700 1 -- 172.21.15.98:0/2852263847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd40cd960 con 0x7f6fd40b51c0 2023-03-03T07:57:56.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.223+0000 7f6fd899d700 1 -- 172.21.15.98:0/2852263847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd40d0d80 con 0x7f6fd40b51c0 2023-03-03T07:57:56.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.224+0000 7f6fbbfff700 1 -- 172.21.15.98:0/2852263847 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f6fc4019030 con 0x7f6fd40b51c0 2023-03-03T07:57:56.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.226+0000 7f6fbbfff700 1 --2- 172.21.15.98:0/2852263847 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fc00683c0 0x7f6fc006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:56.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.226+0000 7f6fbbfff700 1 -- 172.21.15.98:0/2852263847 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f6fc40851e0 con 0x7f6fd40b51c0 2023-03-03T07:57:56.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.226+0000 7f6fd2ffd700 1 --2- 172.21.15.98:0/2852263847 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fc00683c0 0x7f6fc006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:56.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.227+0000 7f6fd2ffd700 1 --2- 172.21.15.98:0/2852263847 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fc00683c0 0x7f6fc006a870 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40b55b0 tx=0x7f6fc8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:56.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.229+0000 7f6fbbfff700 1 -- 172.21.15.98:0/2852263847 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6fc4051860 con 0x7f6fd40b51c0 2023-03-03T07:57:56.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.471+0000 7f6fd899d700 1 -- 172.21.15.98:0/2852263847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fd4068300 con 0x7f6fd40b51c0 2023-03-03T07:57:56.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.472+0000 7f6fbbfff700 1 -- 172.21.15.98:0/2852263847 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f6fc4054e80 con 0x7f6fd40b51c0 2023-03-03T07:57:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:56.474 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3849166495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3849166495},{"type":"v1","addr":"172.21.15.100:6834","nonce":3849166495}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/4285491249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":4285491249},{"type":"v1","addr":"172.21.15.100:6835","nonce":4285491249}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":5208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3888598890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3888598890},{"type":"v1","addr":"172.21.15.98:6836","nonce":3888598890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:57:17.291546+0000","modified":"2023-03-03T07:57:20.741013+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5227},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5227":{"gid":5227,"name":"c","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.98:6837/1163251191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1163251191},{"type":"v1","addr":"172.21.15.98:6837","nonce":1163251191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T07:57:56.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.474+0000 7f6fd899d700 1 -- 172.21.15.98:0/2852263847 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fc00683c0 msgr2=0x7f6fc006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:56.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.474+0000 7f6fd899d700 1 --2- 172.21.15.98:0/2852263847 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fc00683c0 0x7f6fc006a870 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40b55b0 tx=0x7f6fc8008030 comp rx=0 tx=0).stop 2023-03-03T07:57:56.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.474+0000 7f6fd899d700 1 -- 172.21.15.98:0/2852263847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fd40b51c0 msgr2=0x7f6fd4154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:56.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.474+0000 7f6fd899d700 1 --2- 172.21.15.98:0/2852263847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fd40b51c0 0x7f6fd4154510 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400c520 tx=0x7f6fc400c830 comp rx=0 tx=0).stop 2023-03-03T07:57:56.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.474+0000 7f6fd899d700 1 -- 172.21.15.98:0/2852263847 shutdown_connections 2023-03-03T07:57:56.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.474+0000 7f6fd899d700 1 --2- 172.21.15.98:0/2852263847 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fc00683c0 0x7f6fc006a870 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.474+0000 7f6fd899d700 1 --2- 172.21.15.98:0/2852263847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fd40b51c0 0x7f6fd4154510 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.474+0000 7f6fd899d700 1 --2- 172.21.15.98:0/2852263847 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fd4146360 0x7f6fd40b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.474+0000 7f6fd899d700 1 --2- 172.21.15.98:0/2852263847 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fd40d1580 0x7f6fd40c6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.474+0000 7f6fd899d700 1 -- 172.21.15.98:0/2852263847 >> 172.21.15.98:0/2852263847 conn(0x7f6fd406b8e0 msgr2=0x7f6fd415c190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.474+0000 7f6fd899d700 1 -- 172.21.15.98:0/2852263847 shutdown_connections 2023-03-03T07:57:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.475+0000 7f6fd899d700 1 -- 172.21.15.98:0/2852263847 wait complete. 2023-03-03T07:57:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 15 2023-03-03T07:57:56.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:57:56.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.750+0000 7ff5b15ff700 1 Processor -- start 2023-03-03T07:57:56.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.750+0000 7ff5b15ff700 1 -- start start 2023-03-03T07:57:56.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.751+0000 7ff5b15ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff5ac15cc00 0x7ff5ac0cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.751+0000 7ff5b15ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff5ac0d2c60 0x7ff5ac0cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.751+0000 7ff5b15ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ac0cde10 con 0x7ff5ac0d27c0 2023-03-03T07:57:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.751+0000 7ff5b15ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5ac0ce040 con 0x7ff5ac15cc00 2023-03-03T07:57:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.751+0000 7ff5b15ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5ac0d30c0 con 0x7ff5ac0d2c60 2023-03-03T07:57:56.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.752+0000 7ff5aaffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff5ac15cc00 0x7ff5ac0cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:56.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.752+0000 7ff5aaffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff5ac15cc00 0x7ff5ac0cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48670/0 (socket says 172.21.15.98:48670) 2023-03-03T07:57:56.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.752+0000 7ff5aaffd700 1 -- 172.21.15.98:0/3625364721 learned_addr learned my addr 172.21.15.98:0/3625364721 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:56.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.752+0000 7ff5ab7fe700 1 --2- 172.21.15.98:0/3625364721 >> v2:172.21.15.100:3300/0 conn(0x7ff5ac0d2c60 0x7ff5ac0cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:56.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.753+0000 7ff5aaffd700 1 -- 172.21.15.98:0/3625364721 >> v2:172.21.15.100:3300/0 conn(0x7ff5ac0d2c60 msgr2=0x7ff5ac0cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:56.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.753+0000 7ff5aaffd700 1 --2- 172.21.15.98:0/3625364721 >> v2:172.21.15.100:3300/0 conn(0x7ff5ac0d2c60 0x7ff5ac0cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.753+0000 7ff5aaffd700 1 -- 172.21.15.98:0/3625364721 >> v1:172.21.15.100:6789/0 conn(0x7ff5ac0d27c0 legacy=0x7ff5ac0b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:56.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.753+0000 7ff5aaffd700 1 -- 172.21.15.98:0/3625364721 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac0d1a00 con 0x7ff5ac15cc00 2023-03-03T07:57:56.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.753+0000 7ff5aaffd700 1 --2- 172.21.15.98:0/3625364721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac15cc00 0x7ff5ac0cd8d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0d4360 tx=0x7ff59c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea25d2ef09e30866 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:56.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.753+0000 7ff5a9ffb700 1 -- 172.21.15.98:0/3625364721 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c011830 con 0x7ff5ac15cc00 2023-03-03T07:57:56.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.753+0000 7ff5a9ffb700 1 -- 172.21.15.98:0/3625364721 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59c011990 con 0x7ff5ac15cc00 2023-03-03T07:57:56.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.754+0000 7ff5a9ffb700 1 -- 172.21.15.98:0/3625364721 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c011d20 con 0x7ff5ac15cc00 2023-03-03T07:57:56.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.754+0000 7ff5b15ff700 1 -- 172.21.15.98:0/3625364721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac15cc00 msgr2=0x7ff5ac0cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:56.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.754+0000 7ff5b15ff700 1 --2- 172.21.15.98:0/3625364721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac15cc00 0x7ff5ac0cd8d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0d4360 tx=0x7ff59c009bd0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.754+0000 7ff5b15ff700 1 -- 172.21.15.98:0/3625364721 shutdown_connections 2023-03-03T07:57:56.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.754+0000 7ff5b15ff700 1 --2- 172.21.15.98:0/3625364721 >> v2:172.21.15.100:3300/0 conn(0x7ff5ac0d2c60 0x7ff5ac0cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.754+0000 7ff5b15ff700 1 --2- 172.21.15.98:0/3625364721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac15cc00 0x7ff5ac0cd8d0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.754+0000 7ff5b15ff700 1 -- 172.21.15.98:0/3625364721 >> 172.21.15.98:0/3625364721 conn(0x7ff5ac06aee0 msgr2=0x7ff5ac06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:56.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.754+0000 7ff5b15ff700 1 -- 172.21.15.98:0/3625364721 shutdown_connections 2023-03-03T07:57:56.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.754+0000 7ff5b15ff700 1 -- 172.21.15.98:0/3625364721 wait complete. 2023-03-03T07:57:56.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.757+0000 7ff5b15ff700 1 Processor -- start 2023-03-03T07:57:56.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.757+0000 7ff5b15ff700 1 -- start start 2023-03-03T07:57:56.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.759+0000 7ff5b15ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac0d27c0 0x7ff5ac1f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:56.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.759+0000 7ff5b15ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5ac0d2c60 0x7ff5ac0cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:56.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.759+0000 7ff5b15ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ac15cc00 0x7ff5ac0c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:56.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.759+0000 7ff5b15ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac1f2b80 con 0x7ff5ac0d27c0 2023-03-03T07:57:56.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.759+0000 7ff5b15ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac0b6e60 con 0x7ff5ac0d2c60 2023-03-03T07:57:56.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.759+0000 7ff5b15ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac0b7640 con 0x7ff5ac15cc00 2023-03-03T07:57:56.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.759+0000 7ff5aaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac0d27c0 0x7ff5ac1f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:56.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.759+0000 7ff5aaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac0d27c0 0x7ff5ac1f2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48682/0 (socket says 172.21.15.98:48682) 2023-03-03T07:57:56.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.759+0000 7ff5aaffd700 1 -- 172.21.15.98:0/1906193844 learned_addr learned my addr 172.21.15.98:0/1906193844 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:56.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.759+0000 7ff5aa7fc700 1 --2- 172.21.15.98:0/1906193844 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5ac0d2c60 0x7ff5ac0cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:56.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.759+0000 7ff5ab7fe700 1 --2- 172.21.15.98:0/1906193844 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ac15cc00 0x7ff5ac0c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:56.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.760+0000 7ff5aaffd700 1 -- 172.21.15.98:0/1906193844 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ac15cc00 msgr2=0x7ff5ac0c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.760+0000 7ff5aaffd700 1 --2- 172.21.15.98:0/1906193844 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ac15cc00 0x7ff5ac0c8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.760+0000 7ff5aaffd700 1 -- 172.21.15.98:0/1906193844 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5ac0d2c60 msgr2=0x7ff5ac0cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.760+0000 7ff5aaffd700 1 --2- 172.21.15.98:0/1906193844 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5ac0d2c60 0x7ff5ac0cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.760+0000 7ff5aaffd700 1 -- 172.21.15.98:0/1906193844 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c009550 con 0x7ff5ac0d27c0 2023-03-03T07:57:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.760+0000 7ff5aaffd700 1 --2- 172.21.15.98:0/1906193844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac0d27c0 0x7ff5ac1f2640 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff59c009c10 tx=0x7ff59c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.760+0000 7ff5937fe700 1 -- 172.21.15.98:0/1906193844 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c004510 con 0x7ff5ac0d27c0 2023-03-03T07:57:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.760+0000 7ff5937fe700 1 -- 172.21.15.98:0/1906193844 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59c01e070 con 0x7ff5ac0d27c0 2023-03-03T07:57:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.760+0000 7ff5b15ff700 1 -- 172.21.15.98:0/1906193844 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5ac0d1fc0 con 0x7ff5ac0d27c0 2023-03-03T07:57:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.760+0000 7ff5937fe700 1 -- 172.21.15.98:0/1906193844 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c0038e0 con 0x7ff5ac0d27c0 2023-03-03T07:57:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.761+0000 7ff5b15ff700 1 -- 172.21.15.98:0/1906193844 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5ac0d1ce0 con 0x7ff5ac0d27c0 2023-03-03T07:57:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.762+0000 7ff5b15ff700 1 -- 172.21.15.98:0/1906193844 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5ac0d22a0 con 0x7ff5ac0d27c0 2023-03-03T07:57:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.763+0000 7ff5937fe700 1 -- 172.21.15.98:0/1906193844 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7ff59c00f050 con 0x7ff5ac0d27c0 2023-03-03T07:57:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.764+0000 7ff5937fe700 1 --2- 172.21.15.98:0/1906193844 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff5980683c0 0x7ff59806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.764+0000 7ff5937fe700 1 -- 172.21.15.98:0/1906193844 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7ff59c086d00 con 0x7ff5ac0d27c0 2023-03-03T07:57:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.764+0000 7ff5aa7fc700 1 --2- 172.21.15.98:0/1906193844 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff5980683c0 0x7ff59806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:56.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.765+0000 7ff5aa7fc700 1 --2- 172.21.15.98:0/1906193844 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff5980683c0 0x7ff59806a870 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0007fd0 tx=0x7ff5a0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:56.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.768+0000 7ff5937fe700 1 -- 172.21.15.98:0/1906193844 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff59c053380 con 0x7ff5ac0d27c0 2023-03-03T07:57:56.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.963+0000 7ff5b15ff700 1 -- 172.21.15.98:0/1906193844 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5ac0729b0 con 0x7ff5ac0d27c0 2023-03-03T07:57:56.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.965+0000 7ff5937fe700 1 -- 172.21.15.98:0/1906193844 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+17972 (secure 0 0 0) 0x7ff59c020090 con 0x7ff5ac0d27c0 2023-03-03T07:57:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:57:56.968 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":40,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:57:28.259781+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:57:15.545608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-03-03T07:57:16.283131+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.31","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:57:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.967+0000 7ff5b15ff700 1 -- 172.21.15.98:0/1906193844 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff5980683c0 msgr2=0x7ff59806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.967+0000 7ff5b15ff700 1 --2- 172.21.15.98:0/1906193844 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff5980683c0 0x7ff59806a870 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0007fd0 tx=0x7ff5a0009040 comp rx=0 tx=0).stop 2023-03-03T07:57:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.967+0000 7ff5b15ff700 1 -- 172.21.15.98:0/1906193844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac0d27c0 msgr2=0x7ff5ac1f2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.967+0000 7ff5b15ff700 1 --2- 172.21.15.98:0/1906193844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac0d27c0 0x7ff5ac1f2640 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff59c009c10 tx=0x7ff59c004750 comp rx=0 tx=0).stop 2023-03-03T07:57:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.967+0000 7ff5b15ff700 1 -- 172.21.15.98:0/1906193844 shutdown_connections 2023-03-03T07:57:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.967+0000 7ff5b15ff700 1 --2- 172.21.15.98:0/1906193844 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff5980683c0 0x7ff59806a870 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.967+0000 7ff5b15ff700 1 --2- 172.21.15.98:0/1906193844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac0d27c0 0x7ff5ac1f2640 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.967+0000 7ff5b15ff700 1 --2- 172.21.15.98:0/1906193844 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ac15cc00 0x7ff5ac0c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.967+0000 7ff5b15ff700 1 --2- 172.21.15.98:0/1906193844 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5ac0d2c60 0x7ff5ac0cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.967+0000 7ff5b15ff700 1 -- 172.21.15.98:0/1906193844 >> 172.21.15.98:0/1906193844 conn(0x7ff5ac06aee0 msgr2=0x7ff5ac159810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.968+0000 7ff5b15ff700 1 -- 172.21.15.98:0/1906193844 shutdown_connections 2023-03-03T07:57:56.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:56.968+0000 7ff5b15ff700 1 -- 172.21.15.98:0/1906193844 wait complete. 2023-03-03T07:57:56.997 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-03-03T07:57:57.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.240+0000 7fa2af59e700 1 Processor -- start 2023-03-03T07:57:57.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.240+0000 7fa2af59e700 1 -- start start 2023-03-03T07:57:57.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.242+0000 7fa2af59e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa2b00cf570 0x7fa2b015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:57.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.242+0000 7fa2af59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa2b015db30 0x7fa2b00b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:57.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.242+0000 7fa2af59e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b00cd780 con 0x7fa2b01609b0 2023-03-03T07:57:57.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.242+0000 7fa2af59e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b00cebb0 con 0x7fa2b015db30 2023-03-03T07:57:57.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.242+0000 7fa2af59e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b00cda60 con 0x7fa2b00cf570 2023-03-03T07:57:57.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.242+0000 7fa2aed9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa2b015db30 0x7fa2b00b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:57.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.242+0000 7fa2ae59c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa2b01609b0 0x7fa2b0160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58954/0 (socket says 172.21.15.98:58954) 2023-03-03T07:57:57.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.242+0000 7fa2ae59c700 1 -- 172.21.15.98:0/3057192589 learned_addr learned my addr 172.21.15.98:0/3057192589 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:57.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.243+0000 7fa2a7fff700 1 --2- 172.21.15.98:0/3057192589 >> v2:172.21.15.100:3300/0 conn(0x7fa2b00cf570 0x7fa2b015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:57.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.243+0000 7fa2aed9d700 1 -- 172.21.15.98:0/3057192589 >> v2:172.21.15.100:3300/0 conn(0x7fa2b00cf570 msgr2=0x7fa2b015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:57.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.243+0000 7fa2aed9d700 1 --2- 172.21.15.98:0/3057192589 >> v2:172.21.15.100:3300/0 conn(0x7fa2b00cf570 0x7fa2b015d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:57.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.243+0000 7fa2aed9d700 1 -- 172.21.15.98:0/3057192589 >> v1:172.21.15.98:6789/0 conn(0x7fa2b01609b0 legacy=0x7fa2b0160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:57.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.243+0000 7fa2aed9d700 1 -- 172.21.15.98:0/3057192589 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b00cdd40 con 0x7fa2b015db30 2023-03-03T07:57:57.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.243+0000 7fa2add9b700 1 -- 172.21.15.98:0/3057192589 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808873093 0 0) 0x7fa2b00cd780 con 0x7fa2b01609b0 2023-03-03T07:57:57.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.243+0000 7fa2aed9d700 1 --2- 172.21.15.98:0/3057192589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2b015db30 0x7fa2b00b4aa0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00cf0c0 tx=0x7fa2a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1026b8607e0675 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:57.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.244+0000 7fa2add9b700 1 -- 172.21.15.98:0/3057192589 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a800e070 con 0x7fa2b015db30 2023-03-03T07:57:57.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.244+0000 7fa2add9b700 1 -- 172.21.15.98:0/3057192589 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a800bb20 con 0x7fa2b015db30 2023-03-03T07:57:57.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.244+0000 7fa2add9b700 1 -- 172.21.15.98:0/3057192589 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8012430 con 0x7fa2b015db30 2023-03-03T07:57:57.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.244+0000 7fa2af59e700 1 -- 172.21.15.98:0/3057192589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2b015db30 msgr2=0x7fa2b00b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:57.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.244+0000 7fa2af59e700 1 --2- 172.21.15.98:0/3057192589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2b015db30 0x7fa2b00b4aa0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00cf0c0 tx=0x7fa2a800b330 comp rx=0 tx=0).stop 2023-03-03T07:57:57.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.244+0000 7fa2af59e700 1 -- 172.21.15.98:0/3057192589 shutdown_connections 2023-03-03T07:57:57.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.244+0000 7fa2af59e700 1 --2- 172.21.15.98:0/3057192589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2b015db30 0x7fa2b00b4aa0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:57.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.244+0000 7fa2af59e700 1 --2- 172.21.15.98:0/3057192589 >> v2:172.21.15.100:3300/0 conn(0x7fa2b00cf570 0x7fa2b015d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:57.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.244+0000 7fa2af59e700 1 -- 172.21.15.98:0/3057192589 >> 172.21.15.98:0/3057192589 conn(0x7fa2b006b880 msgr2=0x7fa2b00cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:57.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.244+0000 7fa2af59e700 1 -- 172.21.15.98:0/3057192589 shutdown_connections 2023-03-03T07:57:57.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.244+0000 7fa2af59e700 1 -- 172.21.15.98:0/3057192589 wait complete. 2023-03-03T07:57:57.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.246+0000 7fa2af59e700 1 Processor -- start 2023-03-03T07:57:57.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.247+0000 7fa2af59e700 1 -- start start 2023-03-03T07:57:57.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.247+0000 7fa2af59e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa2b00cf570 0x7fa2b0151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:57.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.247+0000 7fa2af59e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2b015db30 0x7fa2b00b7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:57.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.248+0000 7fa2af59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2b01609b0 0x7fa2b00b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:57.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.248+0000 7fa2af59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b00c7900 con 0x7fa2b01609b0 2023-03-03T07:57:57.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.248+0000 7fa2af59e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b00c7be0 con 0x7fa2b00cf570 2023-03-03T07:57:57.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.248+0000 7fa2af59e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b00c7ec0 con 0x7fa2b015db30 2023-03-03T07:57:57.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.248+0000 7fa2aed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2b01609b0 0x7fa2b00b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:57.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.248+0000 7fa2aed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2b01609b0 0x7fa2b00b3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48700/0 (socket says 172.21.15.98:48700) 2023-03-03T07:57:57.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.248+0000 7fa2aed9d700 1 -- 172.21.15.98:0/1433573277 learned_addr learned my addr 172.21.15.98:0/1433573277 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:57.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.248+0000 7fa2a7fff700 1 --2- 172.21.15.98:0/1433573277 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2b015db30 0x7fa2b00b7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:57.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.248+0000 7fa2ae59c700 1 --2- 172.21.15.98:0/1433573277 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa2b00cf570 0x7fa2b0151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:57.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.248+0000 7fa2aed9d700 1 -- 172.21.15.98:0/1433573277 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2b015db30 msgr2=0x7fa2b00b7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:57.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.248+0000 7fa2aed9d700 1 --2- 172.21.15.98:0/1433573277 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2b015db30 0x7fa2b00b7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:57.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.248+0000 7fa2aed9d700 1 -- 172.21.15.98:0/1433573277 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa2b00cf570 msgr2=0x7fa2b0151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:57.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.248+0000 7fa2aed9d700 1 --2- 172.21.15.98:0/1433573277 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa2b00cf570 0x7fa2b0151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.248+0000 7fa2aed9d700 1 -- 172.21.15.98:0/1433573277 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a8009a70 con 0x7fa2b01609b0 2023-03-03T07:57:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.249+0000 7fa2aed9d700 1 --2- 172.21.15.98:0/1433573277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2b01609b0 0x7fa2b00b3f10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8013000 tx=0x7fa2a8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.249+0000 7fa2a6ffd700 1 -- 172.21.15.98:0/1433573277 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8012430 con 0x7fa2b01609b0 2023-03-03T07:57:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.249+0000 7fa2a6ffd700 1 -- 172.21.15.98:0/1433573277 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a8020070 con 0x7fa2b01609b0 2023-03-03T07:57:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.249+0000 7fa2a6ffd700 1 -- 172.21.15.98:0/1433573277 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8004280 con 0x7fa2b01609b0 2023-03-03T07:57:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.249+0000 7fa2af59e700 1 -- 172.21.15.98:0/1433573277 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b015d0d0 con 0x7fa2b01609b0 2023-03-03T07:57:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.249+0000 7fa2af59e700 1 -- 172.21.15.98:0/1433573277 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b015cdf0 con 0x7fa2b01609b0 2023-03-03T07:57:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.250+0000 7fa2af59e700 1 -- 172.21.15.98:0/1433573277 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b00b4fe0 con 0x7fa2b01609b0 2023-03-03T07:57:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.251+0000 7fa2a6ffd700 1 -- 172.21.15.98:0/1433573277 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fa2a802e070 con 0x7fa2b01609b0 2023-03-03T07:57:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.253+0000 7fa2a6ffd700 1 --2- 172.21.15.98:0/1433573277 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa2900683c0 0x7fa29006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.253+0000 7fa2a6ffd700 1 -- 172.21.15.98:0/1433573277 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fa2a8022070 con 0x7fa2b01609b0 2023-03-03T07:57:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.253+0000 7fa2ae59c700 1 --2- 172.21.15.98:0/1433573277 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa2900683c0 0x7fa29006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.254+0000 7fa2ae59c700 1 --2- 172.21.15.98:0/1433573277 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa2900683c0 0x7fa29006a870 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0000f60 tx=0x7fa2a0007e70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:57.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.257+0000 7fa2a6ffd700 1 -- 172.21.15.98:0/1433573277 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa2a8054170 con 0x7fa2b01609b0 2023-03-03T07:57:57.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:57.502+0000 7fa2af59e700 1 -- 172.21.15.98:0/1433573277 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa2b0068290 con 0x7fa2b01609b0 2023-03-03T07:57:58.467 INFO:tasks.ceph.mon.a.smithi098.stderr:2023-03-03T07:57:58.465+0000 7fb71abf9700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-03-03T07:57:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.469+0000 7fa2a6ffd700 1 -- 172.21.15.98:0/1433573277 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa2a8057790 con 0x7fa2b01609b0 2023-03-03T07:57:58.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.473+0000 7fa2af59e700 1 -- 172.21.15.98:0/1433573277 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa2900683c0 msgr2=0x7fa29006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:58.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.473+0000 7fa2af59e700 1 --2- 172.21.15.98:0/1433573277 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa2900683c0 0x7fa29006a870 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0000f60 tx=0x7fa2a0007e70 comp rx=0 tx=0).stop 2023-03-03T07:57:58.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.473+0000 7fa2af59e700 1 -- 172.21.15.98:0/1433573277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2b01609b0 msgr2=0x7fa2b00b3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:58.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.473+0000 7fa2af59e700 1 --2- 172.21.15.98:0/1433573277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2b01609b0 0x7fa2b00b3f10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8013000 tx=0x7fa2a8004ad0 comp rx=0 tx=0).stop 2023-03-03T07:57:58.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.473+0000 7fa2af59e700 1 -- 172.21.15.98:0/1433573277 shutdown_connections 2023-03-03T07:57:58.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.473+0000 7fa2af59e700 1 --2- 172.21.15.98:0/1433573277 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa2900683c0 0x7fa29006a870 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:58.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.473+0000 7fa2af59e700 1 --2- 172.21.15.98:0/1433573277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2b01609b0 0x7fa2b00b3f10 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:58.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.473+0000 7fa2af59e700 1 --2- 172.21.15.98:0/1433573277 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa2b00cf570 0x7fa2b0151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:58.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.473+0000 7fa2af59e700 1 --2- 172.21.15.98:0/1433573277 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2b015db30 0x7fa2b00b7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:58.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.473+0000 7fa2af59e700 1 -- 172.21.15.98:0/1433573277 >> 172.21.15.98:0/1433573277 conn(0x7fa2b006b880 msgr2=0x7fa2b015b880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:58.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.473+0000 7fa2af59e700 1 -- 172.21.15.98:0/1433573277 shutdown_connections 2023-03-03T07:57:58.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.473+0000 7fa2af59e700 1 -- 172.21.15.98:0/1433573277 wait complete. 2023-03-03T07:57:58.477 INFO:teuthology.orchestra.run.smithi098.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-03-03T07:57:58.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-03-03T07:57:58.573 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-03-03T07:57:58.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.759+0000 7fd00efff700 1 Processor -- start 2023-03-03T07:57:58.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.759+0000 7fd00efff700 1 -- start start 2023-03-03T07:57:58.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.760+0000 7fd00efff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd0080d0270 0x7fd0080cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:58.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.760+0000 7fd00efff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd00806b4b0 0x7fd0080d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:58.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.760+0000 7fd00efff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0080cf980 con 0x7fd00815ca70 2023-03-03T07:57:58.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.760+0000 7fd00efff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0080cd5b0 con 0x7fd0080d0270 2023-03-03T07:57:58.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.760+0000 7fd00efff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0080d1800 con 0x7fd00806b4b0 2023-03-03T07:57:58.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.761+0000 7fd00d59c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd00806b4b0 0x7fd0080d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:58.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.761+0000 7fd007fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd0080d0270 0x7fd0080cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:58.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.761+0000 7fd00cd9b700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fd00815ca70 0x7fd0080cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:45184/0 (socket says 172.21.15.98:45184) 2023-03-03T07:57:58.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.761+0000 7fd00d59c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd00806b4b0 0x7fd0080d1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:36916/0 (socket says 172.21.15.98:36916) 2023-03-03T07:57:58.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.761+0000 7fd00cd9b700 1 -- 172.21.15.98:0/2034762666 learned_addr learned my addr 172.21.15.98:0/2034762666 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:58.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.761+0000 7fd00d59c700 1 -- 172.21.15.98:0/2034762666 >> v2:172.21.15.100:3300/0 conn(0x7fd0080d0270 msgr2=0x7fd0080cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:58.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.761+0000 7fd00d59c700 1 --2- 172.21.15.98:0/2034762666 >> v2:172.21.15.100:3300/0 conn(0x7fd0080d0270 0x7fd0080cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:58.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.761+0000 7fd00d59c700 1 -- 172.21.15.98:0/2034762666 >> v1:172.21.15.100:6789/0 conn(0x7fd00815ca70 legacy=0x7fd0080cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:58.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.761+0000 7fd00d59c700 1 -- 172.21.15.98:0/2034762666 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0080cd900 con 0x7fd00806b4b0 2023-03-03T07:57:58.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.762+0000 7fd00d59c700 1 --2- 172.21.15.98:0/2034762666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd00806b4b0 0x7fd0080d1fc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd000009d90 tx=0x7fd00000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=965b5f98b8575300 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:58.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.762+0000 7fd0077fe700 1 -- 172.21.15.98:0/2034762666 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00000e070 con 0x7fd00806b4b0 2023-03-03T07:57:58.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.762+0000 7fd0077fe700 1 -- 172.21.15.98:0/2034762666 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00000bb20 con 0x7fd00806b4b0 2023-03-03T07:57:58.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.762+0000 7fd0077fe700 1 -- 172.21.15.98:0/2034762666 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd000012430 con 0x7fd00806b4b0 2023-03-03T07:57:58.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.763+0000 7fd00efff700 1 -- 172.21.15.98:0/2034762666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd00806b4b0 msgr2=0x7fd0080d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:58.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.763+0000 7fd00efff700 1 --2- 172.21.15.98:0/2034762666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd00806b4b0 0x7fd0080d1fc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd000009d90 tx=0x7fd00000b330 comp rx=0 tx=0).stop 2023-03-03T07:57:58.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.763+0000 7fd00efff700 1 -- 172.21.15.98:0/2034762666 shutdown_connections 2023-03-03T07:57:58.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.763+0000 7fd00efff700 1 --2- 172.21.15.98:0/2034762666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd00806b4b0 0x7fd0080d1fc0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:58.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.763+0000 7fd00efff700 1 --2- 172.21.15.98:0/2034762666 >> v2:172.21.15.100:3300/0 conn(0x7fd0080d0270 0x7fd0080cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:58.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.763+0000 7fd00efff700 1 -- 172.21.15.98:0/2034762666 >> 172.21.15.98:0/2034762666 conn(0x7fd008073e70 msgr2=0x7fd00806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:58.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.763+0000 7fd00efff700 1 -- 172.21.15.98:0/2034762666 shutdown_connections 2023-03-03T07:57:58.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.763+0000 7fd00efff700 1 -- 172.21.15.98:0/2034762666 wait complete. 2023-03-03T07:57:58.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.765+0000 7fd00efff700 1 Processor -- start 2023-03-03T07:57:58.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.765+0000 7fd00efff700 1 -- start start 2023-03-03T07:57:58.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.766+0000 7fd00efff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd00806b4b0 0x7fd0080cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:58.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.766+0000 7fd00efff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0080d0270 0x7fd0081f2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:58.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.766+0000 7fd00efff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd00815ca70 0x7fd0081f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:58.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.766+0000 7fd00efff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0081f2d80 con 0x7fd0080d0270 2023-03-03T07:57:58.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.766+0000 7fd00efff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0081f3030 con 0x7fd00815ca70 2023-03-03T07:57:58.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.766+0000 7fd00efff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0080c92b0 con 0x7fd00806b4b0 2023-03-03T07:57:58.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.766+0000 7fd007fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0080d0270 0x7fd0081f2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:58.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.766+0000 7fd007fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0080d0270 0x7fd0081f2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48716/0 (socket says 172.21.15.98:48716) 2023-03-03T07:57:58.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.766+0000 7fd007fff700 1 -- 172.21.15.98:0/3489381172 learned_addr learned my addr 172.21.15.98:0/3489381172 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:58.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.766+0000 7fd00d59c700 1 --2- 172.21.15.98:0/3489381172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd00815ca70 0x7fd0081f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:58.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.766+0000 7fd00cd9b700 1 --2- 172.21.15.98:0/3489381172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd00806b4b0 0x7fd0080cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:58.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.767+0000 7fd007fff700 1 -- 172.21.15.98:0/3489381172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd00806b4b0 msgr2=0x7fd0080cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:58.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.767+0000 7fd007fff700 1 --2- 172.21.15.98:0/3489381172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd00806b4b0 0x7fd0080cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:58.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.767+0000 7fd007fff700 1 -- 172.21.15.98:0/3489381172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd00815ca70 msgr2=0x7fd0081f2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:58.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.767+0000 7fd007fff700 1 --2- 172.21.15.98:0/3489381172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd00815ca70 0x7fd0081f2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:58.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.767+0000 7fd007fff700 1 -- 172.21.15.98:0/3489381172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd000009a70 con 0x7fd0080d0270 2023-03-03T07:57:58.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.767+0000 7fd007fff700 1 --2- 172.21.15.98:0/3489381172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0080d0270 0x7fd0081f2300 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcff400e9b0 tx=0x7fcff400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:58.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.768+0000 7fd0057fa700 1 -- 172.21.15.98:0/3489381172 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff400ee50 con 0x7fd0080d0270 2023-03-03T07:57:58.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.768+0000 7fd0057fa700 1 -- 172.21.15.98:0/3489381172 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff400c7f0 con 0x7fd0080d0270 2023-03-03T07:57:58.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.768+0000 7fd0057fa700 1 -- 172.21.15.98:0/3489381172 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff400cb60 con 0x7fd0080d0270 2023-03-03T07:57:58.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.768+0000 7fd00efff700 1 -- 172.21.15.98:0/3489381172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0080d1aa0 con 0x7fd0080d0270 2023-03-03T07:57:58.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.768+0000 7fd00efff700 1 -- 172.21.15.98:0/3489381172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0080cdbe0 con 0x7fd0080d0270 2023-03-03T07:57:58.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.769+0000 7fd00efff700 1 -- 172.21.15.98:0/3489381172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd00815ce40 con 0x7fd0080d0270 2023-03-03T07:57:58.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.770+0000 7fd0057fa700 1 -- 172.21.15.98:0/3489381172 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fcff4019030 con 0x7fd0080d0270 2023-03-03T07:57:58.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.771+0000 7fd0057fa700 1 --2- 172.21.15.98:0/3489381172 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcfec0683c0 0x7fcfec06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:58.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.771+0000 7fd0057fa700 1 -- 172.21.15.98:0/3489381172 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6177+0+0 (secure 0 0 0) 0x7fcff4085d50 con 0x7fd0080d0270 2023-03-03T07:57:58.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.771+0000 7fd00cd9b700 1 --2- 172.21.15.98:0/3489381172 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcfec0683c0 0x7fcfec06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:58.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.772+0000 7fd00cd9b700 1 --2- 172.21.15.98:0/3489381172 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcfec0683c0 0x7fcfec06a870 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcffc005910 tx=0x7fcffc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:58.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:58.776+0000 7fd0057fa700 1 -- 172.21.15.98:0/3489381172 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcff4052370 con 0x7fd0080d0270 2023-03-03T07:57:59.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.009+0000 7fd00efff700 1 -- 172.21.15.98:0/3489381172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd00804e6e0 con 0x7fd0080d0270 2023-03-03T07:57:59.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.578+0000 7fd0057fa700 1 -- 172.21.15.98:0/3489381172 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcff4055990 con 0x7fd0080d0270 2023-03-03T07:57:59.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.581+0000 7fd00efff700 1 -- 172.21.15.98:0/3489381172 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcfec0683c0 msgr2=0x7fcfec06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:59.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.581+0000 7fd00efff700 1 --2- 172.21.15.98:0/3489381172 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcfec0683c0 0x7fcfec06a870 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcffc005910 tx=0x7fcffc009040 comp rx=0 tx=0).stop 2023-03-03T07:57:59.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.581+0000 7fd00efff700 1 -- 172.21.15.98:0/3489381172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0080d0270 msgr2=0x7fd0081f2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:59.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.581+0000 7fd00efff700 1 --2- 172.21.15.98:0/3489381172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0080d0270 0x7fd0081f2300 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcff400e9b0 tx=0x7fcff400ecc0 comp rx=0 tx=0).stop 2023-03-03T07:57:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.581+0000 7fd00efff700 1 -- 172.21.15.98:0/3489381172 shutdown_connections 2023-03-03T07:57:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.581+0000 7fd00efff700 1 --2- 172.21.15.98:0/3489381172 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcfec0683c0 0x7fcfec06a870 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.581+0000 7fd00efff700 1 --2- 172.21.15.98:0/3489381172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd00815ca70 0x7fd0081f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.581+0000 7fd00efff700 1 --2- 172.21.15.98:0/3489381172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd00806b4b0 0x7fd0080cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.581+0000 7fd00efff700 1 --2- 172.21.15.98:0/3489381172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0080d0270 0x7fd0081f2300 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.581+0000 7fd00efff700 1 -- 172.21.15.98:0/3489381172 >> 172.21.15.98:0/3489381172 conn(0x7fd008073e70 msgr2=0x7fd0080d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.581+0000 7fd00efff700 1 -- 172.21.15.98:0/3489381172 shutdown_connections 2023-03-03T07:57:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.582+0000 7fd00efff700 1 -- 172.21.15.98:0/3489381172 wait complete. 2023-03-03T07:57:59.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-03-03T07:57:59.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.860+0000 7fc043fff700 1 Processor -- start 2023-03-03T07:57:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.860+0000 7fc043fff700 1 -- start start 2023-03-03T07:57:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.862+0000 7fc043fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc0440b3550 0x7fc0440cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.862+0000 7fc043fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc044073ed0 0x7fc0440cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.862+0000 7fc043fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0440b3c80 con 0x7fc044160780 2023-03-03T07:57:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.862+0000 7fc043fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0440b4370 con 0x7fc0440b3550 2023-03-03T07:57:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.862+0000 7fc043fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0440b3ed0 con 0x7fc044073ed0 2023-03-03T07:57:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.862+0000 7fc0437fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc044160780 0x7fc044160c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58966/0 (socket says 172.21.15.98:58966) 2023-03-03T07:57:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.862+0000 7fc0437fe700 1 -- 172.21.15.98:0/3661801242 learned_addr learned my addr 172.21.15.98:0/3661801242 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:57:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.862+0000 7fc0427fc700 1 --2- 172.21.15.98:0/3661801242 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc044073ed0 0x7fc0440cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.862+0000 7fc042ffd700 1 --2- 172.21.15.98:0/3661801242 >> v2:172.21.15.100:3300/0 conn(0x7fc0440b3550 0x7fc0440cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.862+0000 7fc041ffb700 1 -- 172.21.15.98:0/3661801242 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473270029 0 0) 0x7fc0440b3c80 con 0x7fc044160780 2023-03-03T07:57:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.863+0000 7fc041ffb700 1 -- 172.21.15.98:0/3661801242 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc02c0035b0 con 0x7fc044160780 2023-03-03T07:57:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.863+0000 7fc042ffd700 1 -- 172.21.15.98:0/3661801242 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc044073ed0 msgr2=0x7fc0440cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.863+0000 7fc042ffd700 1 --2- 172.21.15.98:0/3661801242 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc044073ed0 0x7fc0440cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.863+0000 7fc042ffd700 1 -- 172.21.15.98:0/3661801242 >> v1:172.21.15.98:6789/0 conn(0x7fc044160780 legacy=0x7fc044160c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.863+0000 7fc042ffd700 1 -- 172.21.15.98:0/3661801242 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0440cd640 con 0x7fc0440b3550 2023-03-03T07:57:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.863+0000 7fc042ffd700 1 --2- 172.21.15.98:0/3661801242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc0440b3550 0x7fc0440cfa00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc034009840 tx=0x7fc034009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9f0230ca15c94ad server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.864+0000 7fc041ffb700 1 -- 172.21.15.98:0/3661801242 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034011830 con 0x7fc0440b3550 2023-03-03T07:57:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.864+0000 7fc041ffb700 1 -- 172.21.15.98:0/3661801242 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc034011990 con 0x7fc0440b3550 2023-03-03T07:57:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.864+0000 7fc041ffb700 1 -- 172.21.15.98:0/3661801242 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034011d20 con 0x7fc0440b3550 2023-03-03T07:57:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.864+0000 7fc043fff700 1 -- 172.21.15.98:0/3661801242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc0440b3550 msgr2=0x7fc0440cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.864+0000 7fc043fff700 1 --2- 172.21.15.98:0/3661801242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc0440b3550 0x7fc0440cfa00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc034009840 tx=0x7fc034009b50 comp rx=0 tx=0).stop 2023-03-03T07:57:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.864+0000 7fc043fff700 1 -- 172.21.15.98:0/3661801242 shutdown_connections 2023-03-03T07:57:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.864+0000 7fc043fff700 1 --2- 172.21.15.98:0/3661801242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc0440b3550 0x7fc0440cfa00 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.864+0000 7fc043fff700 1 --2- 172.21.15.98:0/3661801242 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc044073ed0 0x7fc0440cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.864+0000 7fc043fff700 1 -- 172.21.15.98:0/3661801242 >> 172.21.15.98:0/3661801242 conn(0x7fc044073b00 msgr2=0x7fc0440d2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.865+0000 7fc043fff700 1 -- 172.21.15.98:0/3661801242 shutdown_connections 2023-03-03T07:57:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.865+0000 7fc043fff700 1 -- 172.21.15.98:0/3661801242 wait complete. 2023-03-03T07:57:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.866+0000 7fc043fff700 1 Processor -- start 2023-03-03T07:57:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.867+0000 7fc043fff700 1 -- start start 2023-03-03T07:57:59.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.868+0000 7fc043fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc044073ed0 0x7fc044150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:59.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.868+0000 7fc043fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0440b3550 0x7fc04415f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:59.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.868+0000 7fc043fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc044160780 0x7fc04415f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:59.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.868+0000 7fc043fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc044150aa0 con 0x7fc044073ed0 2023-03-03T07:57:59.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.868+0000 7fc043fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0440c9a40 con 0x7fc044160780 2023-03-03T07:57:59.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.868+0000 7fc043fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0440c8700 con 0x7fc0440b3550 2023-03-03T07:57:59.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.868+0000 7fc042ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc044073ed0 0x7fc044150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:59.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.868+0000 7fc042ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc044073ed0 0x7fc044150560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48718/0 (socket says 172.21.15.98:48718) 2023-03-03T07:57:59.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.868+0000 7fc042ffd700 1 -- 172.21.15.98:0/4217495316 learned_addr learned my addr 172.21.15.98:0/4217495316 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:57:59.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.868+0000 7fc0427fc700 1 --2- 172.21.15.98:0/4217495316 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0440b3550 0x7fc04415f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:59.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.868+0000 7fc0437fe700 1 --2- 172.21.15.98:0/4217495316 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc044160780 0x7fc04415f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:59.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.869+0000 7fc042ffd700 1 -- 172.21.15.98:0/4217495316 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0440b3550 msgr2=0x7fc04415f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:59.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.869+0000 7fc042ffd700 1 --2- 172.21.15.98:0/4217495316 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0440b3550 0x7fc04415f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:59.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.869+0000 7fc042ffd700 1 -- 172.21.15.98:0/4217495316 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc044160780 msgr2=0x7fc04415f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:59.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.869+0000 7fc042ffd700 1 --2- 172.21.15.98:0/4217495316 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc044160780 0x7fc04415f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:59.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.869+0000 7fc042ffd700 1 -- 172.21.15.98:0/4217495316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0340094a0 con 0x7fc044073ed0 2023-03-03T07:57:59.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.869+0000 7fc042ffd700 1 --2- 172.21.15.98:0/4217495316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc044073ed0 0x7fc044150560 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc034000c00 tx=0x7fc034004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:59.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.869+0000 7fc02b7fe700 1 -- 172.21.15.98:0/4217495316 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0340036a0 con 0x7fc044073ed0 2023-03-03T07:57:59.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.869+0000 7fc02b7fe700 1 -- 172.21.15.98:0/4217495316 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03401f070 con 0x7fc044073ed0 2023-03-03T07:57:59.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.869+0000 7fc02b7fe700 1 -- 172.21.15.98:0/4217495316 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034003a60 con 0x7fc044073ed0 2023-03-03T07:57:59.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.869+0000 7fc043fff700 1 -- 172.21.15.98:0/4217495316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0440d3d60 con 0x7fc044073ed0 2023-03-03T07:57:59.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.870+0000 7fc043fff700 1 -- 172.21.15.98:0/4217495316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0440cd920 con 0x7fc044073ed0 2023-03-03T07:57:59.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.871+0000 7fc043fff700 1 -- 172.21.15.98:0/4217495316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0440d3ef0 con 0x7fc044073ed0 2023-03-03T07:57:59.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.872+0000 7fc02b7fe700 1 -- 172.21.15.98:0/4217495316 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fc03400f050 con 0x7fc044073ed0 2023-03-03T07:57:59.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.873+0000 7fc02b7fe700 1 --2- 172.21.15.98:0/4217495316 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc0300683c0 0x7fc03006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:59.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.873+0000 7fc02b7fe700 1 -- 172.21.15.98:0/4217495316 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6177+0+0 (secure 0 0 0) 0x7fc034021070 con 0x7fc044073ed0 2023-03-03T07:57:59.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.873+0000 7fc0427fc700 1 --2- 172.21.15.98:0/4217495316 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc0300683c0 0x7fc03006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:59.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.874+0000 7fc0427fc700 1 --2- 172.21.15.98:0/4217495316 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc0300683c0 0x7fc03006a870 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc04406b6f0 tx=0x7fc038009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:59.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:57:59.877+0000 7fc02b7fe700 1 -- 172.21.15.98:0/4217495316 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc0440d3ef0 con 0x7fc044073ed0 2023-03-03T07:58:00.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.071+0000 7fc043fff700 1 -- 172.21.15.98:0/4217495316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc0440d3ef0 con 0x7fc044073ed0 2023-03-03T07:58:00.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.596+0000 7fc02b7fe700 1 -- 172.21.15.98:0/4217495316 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v42) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc0440d3ef0 con 0x7fc044073ed0 2023-03-03T07:58:00.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.599+0000 7fc043fff700 1 -- 172.21.15.98:0/4217495316 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc0300683c0 msgr2=0x7fc03006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:00.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.599+0000 7fc043fff700 1 --2- 172.21.15.98:0/4217495316 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc0300683c0 0x7fc03006a870 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc04406b6f0 tx=0x7fc038009210 comp rx=0 tx=0).stop 2023-03-03T07:58:00.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.599+0000 7fc043fff700 1 -- 172.21.15.98:0/4217495316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc044073ed0 msgr2=0x7fc044150560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:00.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.599+0000 7fc043fff700 1 --2- 172.21.15.98:0/4217495316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc044073ed0 0x7fc044150560 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc034000c00 tx=0x7fc034004750 comp rx=0 tx=0).stop 2023-03-03T07:58:00.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.600+0000 7fc043fff700 1 -- 172.21.15.98:0/4217495316 shutdown_connections 2023-03-03T07:58:00.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.600+0000 7fc043fff700 1 --2- 172.21.15.98:0/4217495316 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc0300683c0 0x7fc03006a870 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:00.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.600+0000 7fc043fff700 1 --2- 172.21.15.98:0/4217495316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc044073ed0 0x7fc044150560 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:00.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.600+0000 7fc043fff700 1 --2- 172.21.15.98:0/4217495316 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc044160780 0x7fc04415f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:00.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.600+0000 7fc043fff700 1 --2- 172.21.15.98:0/4217495316 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0440b3550 0x7fc04415f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:00.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.600+0000 7fc043fff700 1 -- 172.21.15.98:0/4217495316 >> 172.21.15.98:0/4217495316 conn(0x7fc044073b00 msgr2=0x7fc0440b7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:00.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.600+0000 7fc043fff700 1 -- 172.21.15.98:0/4217495316 shutdown_connections 2023-03-03T07:58:00.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.600+0000 7fc043fff700 1 -- 172.21.15.98:0/4217495316 wait complete. 2023-03-03T07:58:00.604 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' removed 2023-03-03T07:58:00.631 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-03-03T07:58:00.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.880+0000 7f157959d700 1 Processor -- start 2023-03-03T07:58:00.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.881+0000 7f157959d700 1 -- start start 2023-03-03T07:58:00.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.882+0000 7f157959d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15740d0110 0x7f15740cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:00.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.882+0000 7f157959d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1574073ed0 0x7f15740ce380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:00.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.882+0000 7f157959d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15740b3c80 con 0x7f15740b3550 2023-03-03T07:58:00.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.882+0000 7f157959d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15740b4370 con 0x7f15740d0110 2023-03-03T07:58:00.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.882+0000 7f157959d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15740b3ed0 con 0x7f1574073ed0 2023-03-03T07:58:00.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.883+0000 7f156bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15740d0110 0x7f15740cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:00.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.883+0000 7f156bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15740d0110 0x7f15740cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48726/0 (socket says 172.21.15.98:48726) 2023-03-03T07:58:00.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.883+0000 7f156bfff700 1 -- 172.21.15.98:0/483832407 learned_addr learned my addr 172.21.15.98:0/483832407 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:00.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.883+0000 7f1578d9c700 1 --2- 172.21.15.98:0/483832407 >> v2:172.21.15.100:3300/0 conn(0x7f1574073ed0 0x7f15740ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:00.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.883+0000 7f156bfff700 1 -- 172.21.15.98:0/483832407 >> v2:172.21.15.100:3300/0 conn(0x7f1574073ed0 msgr2=0x7f15740ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:00.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.883+0000 7f156bfff700 1 --2- 172.21.15.98:0/483832407 >> v2:172.21.15.100:3300/0 conn(0x7f1574073ed0 0x7f15740ce380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:00.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.883+0000 7f156bfff700 1 -- 172.21.15.98:0/483832407 >> v1:172.21.15.100:6789/0 conn(0x7f15740b3550 legacy=0x7f15740cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:00.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.883+0000 7f156bfff700 1 -- 172.21.15.98:0/483832407 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15740cd640 con 0x7f15740d0110 2023-03-03T07:58:00.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.884+0000 7f156bfff700 1 --2- 172.21.15.98:0/483832407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15740d0110 0x7f15740cde40 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f157408be20 tx=0x7f155c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c34c545287180d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:00.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.884+0000 7f15737fe700 1 -- 172.21.15.98:0/483832407 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c00b870 con 0x7f15740d0110 2023-03-03T07:58:00.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.884+0000 7f15737fe700 1 -- 172.21.15.98:0/483832407 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155c00b9d0 con 0x7f15740d0110 2023-03-03T07:58:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.884+0000 7f15737fe700 1 -- 172.21.15.98:0/483832407 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c00bd60 con 0x7f15740d0110 2023-03-03T07:58:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.884+0000 7f157959d700 1 -- 172.21.15.98:0/483832407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15740d0110 msgr2=0x7f15740cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.884+0000 7f157959d700 1 --2- 172.21.15.98:0/483832407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15740d0110 0x7f15740cde40 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f157408be20 tx=0x7f155c009bd0 comp rx=0 tx=0).stop 2023-03-03T07:58:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.885+0000 7f157959d700 1 -- 172.21.15.98:0/483832407 shutdown_connections 2023-03-03T07:58:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.885+0000 7f157959d700 1 --2- 172.21.15.98:0/483832407 >> v2:172.21.15.100:3300/0 conn(0x7f1574073ed0 0x7f15740ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.885+0000 7f157959d700 1 --2- 172.21.15.98:0/483832407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15740d0110 0x7f15740cde40 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.885+0000 7f157959d700 1 -- 172.21.15.98:0/483832407 >> 172.21.15.98:0/483832407 conn(0x7f1574073b00 msgr2=0x7f15740d2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:00.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.885+0000 7f157959d700 1 -- 172.21.15.98:0/483832407 shutdown_connections 2023-03-03T07:58:00.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.885+0000 7f157959d700 1 -- 172.21.15.98:0/483832407 wait complete. 2023-03-03T07:58:00.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.887+0000 7f157959d700 1 Processor -- start 2023-03-03T07:58:00.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.887+0000 7f157959d700 1 -- start start 2023-03-03T07:58:00.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.888+0000 7f157959d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1574073ed0 0x7f15741e3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:00.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.888+0000 7f157959d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15740b3550 0x7f15740c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:00.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.888+0000 7f157959d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15740d0110 0x7f15741f5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:00.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.888+0000 7f157959d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15741f26f0 con 0x7f15740d0110 2023-03-03T07:58:00.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.888+0000 7f157959d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15741f29d0 con 0x7f15740b3550 2023-03-03T07:58:00.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.888+0000 7f157959d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15741f2cb0 con 0x7f1574073ed0 2023-03-03T07:58:00.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.888+0000 7f1578d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15740d0110 0x7f15741f5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:00.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.888+0000 7f1578d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15740d0110 0x7f15741f5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48728/0 (socket says 172.21.15.98:48728) 2023-03-03T07:58:00.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.888+0000 7f1578d9c700 1 -- 172.21.15.98:0/2824162293 learned_addr learned my addr 172.21.15.98:0/2824162293 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:00.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.889+0000 7f1573fff700 1 --2- 172.21.15.98:0/2824162293 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1574073ed0 0x7f15741e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:00.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.889+0000 7f156bfff700 1 --2- 172.21.15.98:0/2824162293 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15740b3550 0x7f15740c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:00.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.889+0000 7f1578d9c700 1 -- 172.21.15.98:0/2824162293 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1574073ed0 msgr2=0x7f15741e3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:00.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.889+0000 7f1578d9c700 1 --2- 172.21.15.98:0/2824162293 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1574073ed0 0x7f15741e3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:00.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.889+0000 7f1578d9c700 1 -- 172.21.15.98:0/2824162293 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15740b3550 msgr2=0x7f15740c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:00.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.889+0000 7f1578d9c700 1 --2- 172.21.15.98:0/2824162293 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15740b3550 0x7f15740c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:00.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.889+0000 7f1578d9c700 1 -- 172.21.15.98:0/2824162293 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c009550 con 0x7f15740d0110 2023-03-03T07:58:00.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.889+0000 7f1578d9c700 1 --2- 172.21.15.98:0/2824162293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15740d0110 0x7f15741f5c50 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f156c007e40 tx=0x7f156c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:00.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.889+0000 7f15717fa700 1 -- 172.21.15.98:0/2824162293 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c00eea0 con 0x7f15740d0110 2023-03-03T07:58:00.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.889+0000 7f15717fa700 1 -- 172.21.15.98:0/2824162293 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156c014420 con 0x7f15740d0110 2023-03-03T07:58:00.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.889+0000 7f157959d700 1 -- 172.21.15.98:0/2824162293 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15740d3d60 con 0x7f15740d0110 2023-03-03T07:58:00.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.889+0000 7f15717fa700 1 -- 172.21.15.98:0/2824162293 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c0147b0 con 0x7f15740d0110 2023-03-03T07:58:00.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.890+0000 7f157959d700 1 -- 172.21.15.98:0/2824162293 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15740cd920 con 0x7f15740d0110 2023-03-03T07:58:00.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.891+0000 7f157959d700 1 -- 172.21.15.98:0/2824162293 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15740d3ef0 con 0x7f15740d0110 2023-03-03T07:58:00.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.892+0000 7f15717fa700 1 -- 172.21.15.98:0/2824162293 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f156c012050 con 0x7f15740d0110 2023-03-03T07:58:00.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.893+0000 7f15717fa700 1 --2- 172.21.15.98:0/2824162293 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15540683c0 0x7f155406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:00.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.893+0000 7f15717fa700 1 -- 172.21.15.98:0/2824162293 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5739+0+0 (secure 0 0 0) 0x7f156c086430 con 0x7f15740d0110 2023-03-03T07:58:00.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.893+0000 7f1573fff700 1 --2- 172.21.15.98:0/2824162293 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15540683c0 0x7f155406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:00.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.894+0000 7f1573fff700 1 --2- 172.21.15.98:0/2824162293 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15540683c0 0x7f155406a870 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1564000f60 tx=0x7f1564009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:00.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:00.897+0000 7f15717fa700 1 -- 172.21.15.98:0/2824162293 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f15740d3ef0 con 0x7f15740d0110 2023-03-03T07:58:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.091+0000 7f157959d700 1 -- 172.21.15.98:0/2824162293 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f15740d3ef0 con 0x7f15740d0110 2023-03-03T07:58:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.601+0000 7f15717fa700 1 -- 172.21.15.98:0/2824162293 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v43) v1 ==== 169+0+0 (secure 0 0 0) 0x7f15740d3ef0 con 0x7f15740d0110 2023-03-03T07:58:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.605+0000 7f157959d700 1 -- 172.21.15.98:0/2824162293 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15540683c0 msgr2=0x7f155406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.605+0000 7f157959d700 1 --2- 172.21.15.98:0/2824162293 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15540683c0 0x7f155406a870 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1564000f60 tx=0x7f1564009040 comp rx=0 tx=0).stop 2023-03-03T07:58:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.605+0000 7f157959d700 1 -- 172.21.15.98:0/2824162293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15740d0110 msgr2=0x7f15741f5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.605+0000 7f157959d700 1 --2- 172.21.15.98:0/2824162293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15740d0110 0x7f15741f5c50 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f156c007e40 tx=0x7f156c00e3c0 comp rx=0 tx=0).stop 2023-03-03T07:58:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.605+0000 7f157959d700 1 -- 172.21.15.98:0/2824162293 shutdown_connections 2023-03-03T07:58:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.605+0000 7f157959d700 1 --2- 172.21.15.98:0/2824162293 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15540683c0 0x7f155406a870 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.605+0000 7f157959d700 1 --2- 172.21.15.98:0/2824162293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15740d0110 0x7f15741f5c50 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.605+0000 7f157959d700 1 --2- 172.21.15.98:0/2824162293 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1574073ed0 0x7f15741e3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.605+0000 7f157959d700 1 --2- 172.21.15.98:0/2824162293 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15740b3550 0x7f15740c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.605+0000 7f157959d700 1 -- 172.21.15.98:0/2824162293 >> 172.21.15.98:0/2824162293 conn(0x7f1574073b00 msgr2=0x7f157406be10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.606+0000 7f157959d700 1 -- 172.21.15.98:0/2824162293 shutdown_connections 2023-03-03T07:58:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.606+0000 7f157959d700 1 -- 172.21.15.98:0/2824162293 wait complete. 2023-03-03T07:58:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' removed 2023-03-03T07:58:01.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-03-03T07:58:01.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.895+0000 7fe32df9d700 1 Processor -- start 2023-03-03T07:58:01.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.895+0000 7fe32df9d700 1 -- start start 2023-03-03T07:58:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.896+0000 7fe32df9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe3281609b0 0x7fe328160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.896+0000 7fe32df9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3281612c0 con 0x7fe3280b4aa0 2023-03-03T07:58:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.896+0000 7fe32df9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32815dd00 con 0x7fe3280cf570 2023-03-03T07:58:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.896+0000 7fe32df9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3280cebb0 con 0x7fe3281609b0 2023-03-03T07:58:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.897+0000 7fe32cf9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe3281609b0 0x7fe328160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.897+0000 7fe32d79c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe3280b4aa0 0x7fe3280d0bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58980/0 (socket says 172.21.15.98:58980) 2023-03-03T07:58:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.897+0000 7fe32cf9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe3281609b0 0x7fe328160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48732/0 (socket says 172.21.15.98:48732) 2023-03-03T07:58:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.897+0000 7fe32d79c700 1 -- 172.21.15.98:0/742197145 learned_addr learned my addr 172.21.15.98:0/742197145 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.897+0000 7fe32cf9b700 1 -- 172.21.15.98:0/742197145 >> v1:172.21.15.100:6789/0 conn(0x7fe3280cf570 legacy=0x7fe32815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.897+0000 7fe32cf9b700 1 -- 172.21.15.98:0/742197145 >> v1:172.21.15.98:6789/0 conn(0x7fe3280b4aa0 legacy=0x7fe3280d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.897+0000 7fe32cf9b700 1 -- 172.21.15.98:0/742197145 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3280cd780 con 0x7fe3281609b0 2023-03-03T07:58:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.897+0000 7fe3277fe700 1 -- 172.21.15.98:0/742197145 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807294588 0 0) 0x7fe3281612c0 con 0x7fe3280b4aa0 2023-03-03T07:58:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.898+0000 7fe32cf9b700 1 --2- 172.21.15.98:0/742197145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3281609b0 0x7fe328160d80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe31c009840 tx=0x7fe31c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172c39b454506519 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.898+0000 7fe3277fe700 1 -- 172.21.15.98:0/742197145 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c011830 con 0x7fe3281609b0 2023-03-03T07:58:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.898+0000 7fe3277fe700 1 -- 172.21.15.98:0/742197145 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31c011990 con 0x7fe3281609b0 2023-03-03T07:58:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.898+0000 7fe3277fe700 1 -- 172.21.15.98:0/742197145 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c011d20 con 0x7fe3281609b0 2023-03-03T07:58:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.898+0000 7fe32df9d700 1 -- 172.21.15.98:0/742197145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3281609b0 msgr2=0x7fe328160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.898+0000 7fe32df9d700 1 --2- 172.21.15.98:0/742197145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3281609b0 0x7fe328160d80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe31c009840 tx=0x7fe31c009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.898+0000 7fe32df9d700 1 -- 172.21.15.98:0/742197145 shutdown_connections 2023-03-03T07:58:01.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.898+0000 7fe32df9d700 1 --2- 172.21.15.98:0/742197145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3281609b0 0x7fe328160d80 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:01.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.898+0000 7fe32df9d700 1 -- 172.21.15.98:0/742197145 >> 172.21.15.98:0/742197145 conn(0x7fe32806b880 msgr2=0x7fe3280cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:01.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.899+0000 7fe32df9d700 1 -- 172.21.15.98:0/742197145 shutdown_connections 2023-03-03T07:58:01.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.899+0000 7fe32df9d700 1 -- 172.21.15.98:0/742197145 wait complete. 2023-03-03T07:58:01.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.901+0000 7fe32df9d700 1 Processor -- start 2023-03-03T07:58:01.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.901+0000 7fe32df9d700 1 -- start start 2023-03-03T07:58:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.902+0000 7fe32df9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3280b4aa0 0x7fe328151520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.902+0000 7fe32df9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3280cf570 0x7fe3280c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.902+0000 7fe32df9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3281609b0 0x7fe3280b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.902+0000 7fe32df9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3281f6850 con 0x7fe3280b4aa0 2023-03-03T07:58:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.902+0000 7fe32df9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe328151a60 con 0x7fe3280cf570 2023-03-03T07:58:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.902+0000 7fe32df9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3281f5cd0 con 0x7fe3281609b0 2023-03-03T07:58:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.902+0000 7fe32cf9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3280b4aa0 0x7fe328151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.902+0000 7fe32cf9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3280b4aa0 0x7fe328151520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48748/0 (socket says 172.21.15.98:48748) 2023-03-03T07:58:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.902+0000 7fe32cf9b700 1 -- 172.21.15.98:0/2843065578 learned_addr learned my addr 172.21.15.98:0/2843065578 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.903+0000 7fe327fff700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3280cf570 0x7fe3280c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.903+0000 7fe32d79c700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3281609b0 0x7fe3280b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.903+0000 7fe32cf9b700 1 -- 172.21.15.98:0/2843065578 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3281609b0 msgr2=0x7fe3280b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.903+0000 7fe32cf9b700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3281609b0 0x7fe3280b3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.903+0000 7fe32cf9b700 1 -- 172.21.15.98:0/2843065578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3280cf570 msgr2=0x7fe3280c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.903+0000 7fe32cf9b700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3280cf570 0x7fe3280c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.903+0000 7fe32cf9b700 1 -- 172.21.15.98:0/2843065578 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c0094a0 con 0x7fe3280b4aa0 2023-03-03T07:58:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.903+0000 7fe327fff700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3280cf570 0x7fe3280c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.903+0000 7fe32d79c700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3281609b0 0x7fe3280b3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.903+0000 7fe32cf9b700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3280b4aa0 0x7fe328151520 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe31c014040 tx=0x7fe31c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.903+0000 7fe3257fa700 1 -- 172.21.15.98:0/2843065578 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c01a940 con 0x7fe3280b4aa0 2023-03-03T07:58:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.903+0000 7fe3257fa700 1 -- 172.21.15.98:0/2843065578 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31c01aaa0 con 0x7fe3280b4aa0 2023-03-03T07:58:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.903+0000 7fe3257fa700 1 -- 172.21.15.98:0/2843065578 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c01ae10 con 0x7fe3280b4aa0 2023-03-03T07:58:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.903+0000 7fe32df9d700 1 -- 172.21.15.98:0/2843065578 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3280cdd40 con 0x7fe3280b4aa0 2023-03-03T07:58:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.904+0000 7fe32df9d700 1 -- 172.21.15.98:0/2843065578 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3280cda60 con 0x7fe3280b4aa0 2023-03-03T07:58:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.905+0000 7fe32df9d700 1 -- 172.21.15.98:0/2843065578 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32815cdf0 con 0x7fe3280b4aa0 2023-03-03T07:58:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.906+0000 7fe3257fa700 1 -- 172.21.15.98:0/2843065578 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fe31c00f050 con 0x7fe3280b4aa0 2023-03-03T07:58:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.907+0000 7fe3257fa700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe30c0683c0 0x7fe30c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.907+0000 7fe3257fa700 1 -- 172.21.15.98:0/2843065578 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fe31c086b50 con 0x7fe3280b4aa0 2023-03-03T07:58:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.907+0000 7fe327fff700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe30c0683c0 0x7fe30c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.908+0000 7fe327fff700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe30c0683c0 0x7fe30c06a870 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe328150ef0 tx=0x7fe314009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:01.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:01.911+0000 7fe3257fa700 1 -- 172.21.15.98:0/2843065578 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe31c0535c0 con 0x7fe3280b4aa0 2023-03-03T07:58:02.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:58:02.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:58:02.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.111+0000 7fe32df9d700 1 -- 172.21.15.98:0/2843065578 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe3280ce360 con 0x7fe3280b4aa0 2023-03-03T07:58:02.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.115+0000 7fe3257fa700 1 -- 172.21.15.98:0/2843065578 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe31c029020 con 0x7fe3280b4aa0 2023-03-03T07:58:02.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.117+0000 7fe32df9d700 1 -- 172.21.15.98:0/2843065578 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe30c0683c0 msgr2=0x7fe30c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:02.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.117+0000 7fe32df9d700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe30c0683c0 0x7fe30c06a870 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe328150ef0 tx=0x7fe314009040 comp rx=0 tx=0).stop 2023-03-03T07:58:02.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.117+0000 7fe32df9d700 1 -- 172.21.15.98:0/2843065578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3280b4aa0 msgr2=0x7fe328151520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:02.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.117+0000 7fe32df9d700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3280b4aa0 0x7fe328151520 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe31c014040 tx=0x7fe31c004750 comp rx=0 tx=0).stop 2023-03-03T07:58:02.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.117+0000 7fe32df9d700 1 -- 172.21.15.98:0/2843065578 shutdown_connections 2023-03-03T07:58:02.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.117+0000 7fe32df9d700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe30c0683c0 0x7fe30c06a870 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:02.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.118+0000 7fe32df9d700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3280b4aa0 0x7fe328151520 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:02.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.118+0000 7fe32df9d700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3281609b0 0x7fe3280b3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:02.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.118+0000 7fe32df9d700 1 --2- 172.21.15.98:0/2843065578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3280cf570 0x7fe3280c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:02.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.118+0000 7fe32df9d700 1 -- 172.21.15.98:0/2843065578 >> 172.21.15.98:0/2843065578 conn(0x7fe32806b880 msgr2=0x7fe3280b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:02.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.118+0000 7fe32df9d700 1 -- 172.21.15.98:0/2843065578 shutdown_connections 2023-03-03T07:58:02.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.118+0000 7fe32df9d700 1 -- 172.21.15.98:0/2843065578 wait complete. 2023-03-03T07:58:02.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade' 2023-03-03T07:58:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.391+0000 7fe6d39ff700 1 Processor -- start 2023-03-03T07:58:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.392+0000 7fe6d39ff700 1 -- start start 2023-03-03T07:58:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.393+0000 7fe6d39ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe6cc0b87c0 0x7fe6cc0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.393+0000 7fe6d39ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6cc0b5410 0x7fe6cc0d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.393+0000 7fe6d39ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6cc0d0ef0 con 0x7fe6cc0b8b90 2023-03-03T07:58:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.393+0000 7fe6d39ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6cc0d0600 con 0x7fe6cc0b5410 2023-03-03T07:58:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.393+0000 7fe6d39ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6cc0cd970 con 0x7fe6cc0b87c0 2023-03-03T07:58:02.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.394+0000 7fe6d1f9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6cc0b5410 0x7fe6cc0d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:02.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.394+0000 7fe6d1f9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6cc0b5410 0x7fe6cc0d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48758/0 (socket says 172.21.15.98:48758) 2023-03-03T07:58:02.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.394+0000 7fe6d1f9c700 1 -- 172.21.15.98:0/3466772930 learned_addr learned my addr 172.21.15.98:0/3466772930 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:02.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.394+0000 7fe6d179b700 1 --2- 172.21.15.98:0/3466772930 >> v2:172.21.15.100:3300/0 conn(0x7fe6cc0b87c0 0x7fe6cc0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:02.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.394+0000 7fe6d1f9c700 1 -- 172.21.15.98:0/3466772930 >> v2:172.21.15.100:3300/0 conn(0x7fe6cc0b87c0 msgr2=0x7fe6cc0cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:02.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.394+0000 7fe6d1f9c700 1 --2- 172.21.15.98:0/3466772930 >> v2:172.21.15.100:3300/0 conn(0x7fe6cc0b87c0 0x7fe6cc0cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:02.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.394+0000 7fe6d1f9c700 1 -- 172.21.15.98:0/3466772930 >> v1:172.21.15.100:6789/0 conn(0x7fe6cc0b8b90 legacy=0x7fe6cc0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:02.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.394+0000 7fe6d1f9c700 1 -- 172.21.15.98:0/3466772930 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc0cdc50 con 0x7fe6cc0b5410 2023-03-03T07:58:02.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.394+0000 7fe6d179b700 1 --2- 172.21.15.98:0/3466772930 >> v2:172.21.15.100:3300/0 conn(0x7fe6cc0b87c0 0x7fe6cc0cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:02.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.394+0000 7fe6d1f9c700 1 --2- 172.21.15.98:0/3466772930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0b5410 0x7fe6cc0d13d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc073840 tx=0x7fe6c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50347d620b58ca7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:02.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.395+0000 7fe6c3fff700 1 -- 172.21.15.98:0/3466772930 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c400e070 con 0x7fe6cc0b5410 2023-03-03T07:58:02.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.395+0000 7fe6c3fff700 1 -- 172.21.15.98:0/3466772930 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c400b9c0 con 0x7fe6cc0b5410 2023-03-03T07:58:02.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.395+0000 7fe6c3fff700 1 -- 172.21.15.98:0/3466772930 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c400bd70 con 0x7fe6cc0b5410 2023-03-03T07:58:02.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.395+0000 7fe6d39ff700 1 -- 172.21.15.98:0/3466772930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0b5410 msgr2=0x7fe6cc0d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:02.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.395+0000 7fe6d39ff700 1 --2- 172.21.15.98:0/3466772930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0b5410 0x7fe6cc0d13d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc073840 tx=0x7fe6c400b330 comp rx=0 tx=0).stop 2023-03-03T07:58:02.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.395+0000 7fe6d39ff700 1 -- 172.21.15.98:0/3466772930 shutdown_connections 2023-03-03T07:58:02.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.395+0000 7fe6d39ff700 1 --2- 172.21.15.98:0/3466772930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0b5410 0x7fe6cc0d13d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:02.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.395+0000 7fe6d39ff700 1 --2- 172.21.15.98:0/3466772930 >> v2:172.21.15.100:3300/0 conn(0x7fe6cc0b87c0 0x7fe6cc0cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:02.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.395+0000 7fe6d39ff700 1 -- 172.21.15.98:0/3466772930 >> 172.21.15.98:0/3466772930 conn(0x7fe6cc06aee0 msgr2=0x7fe6cc06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:02.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.396+0000 7fe6d39ff700 1 -- 172.21.15.98:0/3466772930 shutdown_connections 2023-03-03T07:58:02.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.396+0000 7fe6d39ff700 1 -- 172.21.15.98:0/3466772930 wait complete. 2023-03-03T07:58:02.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.398+0000 7fe6d39ff700 1 Processor -- start 2023-03-03T07:58:02.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.399+0000 7fe6d39ff700 1 -- start start 2023-03-03T07:58:02.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.400+0000 7fe6d39ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6cc0b5410 0x7fe6cc1f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:02.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.400+0000 7fe6d39ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0b87c0 0x7fe6cc0b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:02.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.400+0000 7fe6d39ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6cc0b8b90 0x7fe6cc1f5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:02.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.400+0000 7fe6d39ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc1e5f30 con 0x7fe6cc0b87c0 2023-03-03T07:58:02.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.400+0000 7fe6d39ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc1f34a0 con 0x7fe6cc0b5410 2023-03-03T07:58:02.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.400+0000 7fe6d39ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc1f5e50 con 0x7fe6cc0b8b90 2023-03-03T07:58:02.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.400+0000 7fe6d0f9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0b87c0 0x7fe6cc0b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:02.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.400+0000 7fe6d179b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6cc0b5410 0x7fe6cc1f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:02.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.400+0000 7fe6d0f9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0b87c0 0x7fe6cc0b1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48772/0 (socket says 172.21.15.98:48772) 2023-03-03T07:58:02.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.400+0000 7fe6d0f9a700 1 -- 172.21.15.98:0/1698556271 learned_addr learned my addr 172.21.15.98:0/1698556271 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:02.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.400+0000 7fe6d1f9c700 1 --2- 172.21.15.98:0/1698556271 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6cc0b8b90 0x7fe6cc1f5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:02.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.401+0000 7fe6d0f9a700 1 -- 172.21.15.98:0/1698556271 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6cc0b8b90 msgr2=0x7fe6cc1f5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:02.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.401+0000 7fe6d0f9a700 1 --2- 172.21.15.98:0/1698556271 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6cc0b8b90 0x7fe6cc1f5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:02.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.401+0000 7fe6d0f9a700 1 -- 172.21.15.98:0/1698556271 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6cc0b5410 msgr2=0x7fe6cc1f2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:02.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.401+0000 7fe6d0f9a700 1 --2- 172.21.15.98:0/1698556271 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6cc0b5410 0x7fe6cc1f2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:02.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.401+0000 7fe6d0f9a700 1 -- 172.21.15.98:0/1698556271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c4009b30 con 0x7fe6cc0b87c0 2023-03-03T07:58:02.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.401+0000 7fe6d0f9a700 1 --2- 172.21.15.98:0/1698556271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0b87c0 0x7fe6cc0b1b80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc00c450 tx=0x7fe6bc00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:02.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.401+0000 7fe6c1ffb700 1 -- 172.21.15.98:0/1698556271 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0052d0 con 0x7fe6cc0b87c0 2023-03-03T07:58:02.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.402+0000 7fe6c1ffb700 1 -- 172.21.15.98:0/1698556271 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6bc005430 con 0x7fe6cc0b87c0 2023-03-03T07:58:02.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.402+0000 7fe6c1ffb700 1 -- 172.21.15.98:0/1698556271 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0038c0 con 0x7fe6cc0b87c0 2023-03-03T07:58:02.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.402+0000 7fe6d39ff700 1 -- 172.21.15.98:0/1698556271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6cc0cdf30 con 0x7fe6cc0b87c0 2023-03-03T07:58:02.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.402+0000 7fe6d39ff700 1 -- 172.21.15.98:0/1698556271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6cc073a80 con 0x7fe6cc0b87c0 2023-03-03T07:58:02.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.403+0000 7fe6d39ff700 1 -- 172.21.15.98:0/1698556271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6cc0ce0c0 con 0x7fe6cc0b87c0 2023-03-03T07:58:02.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.404+0000 7fe6c1ffb700 1 -- 172.21.15.98:0/1698556271 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fe6bc019030 con 0x7fe6cc0b87c0 2023-03-03T07:58:02.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.405+0000 7fe6c1ffb700 1 --2- 172.21.15.98:0/1698556271 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6b00683c0 0x7fe6b006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:02.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.405+0000 7fe6c1ffb700 1 -- 172.21.15.98:0/1698556271 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fe6bc084dc0 con 0x7fe6cc0b87c0 2023-03-03T07:58:02.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.406+0000 7fe6d179b700 1 --2- 172.21.15.98:0/1698556271 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6b00683c0 0x7fe6b006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:02.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.406+0000 7fe6d179b700 1 --2- 172.21.15.98:0/1698556271 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6b00683c0 0x7fe6b006a870 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc1e7ce0 tx=0x7fe6c8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:02.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.409+0000 7fe6c1ffb700 1 -- 172.21.15.98:0/1698556271 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe6bc051700 con 0x7fe6cc0b87c0 2023-03-03T07:58:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:02.681+0000 7fe6d39ff700 1 -- 172.21.15.98:0/1698556271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]} v 0) v1 -- 0x7fe6cc0729b0 con 0x7fe6cc0b87c0 2023-03-03T07:58:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.605+0000 7fe6c1ffb700 1 -- 172.21.15.98:0/1698556271 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]}]=0 v152) v1 ==== 158+0+0 (secure 0 0 0) 0x7fe6bc0170e0 con 0x7fe6cc0b87c0 2023-03-03T07:58:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.609+0000 7fe6d39ff700 1 -- 172.21.15.98:0/1698556271 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6b00683c0 msgr2=0x7fe6b006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.609+0000 7fe6d39ff700 1 --2- 172.21.15.98:0/1698556271 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6b00683c0 0x7fe6b006a870 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc1e7ce0 tx=0x7fe6c8008040 comp rx=0 tx=0).stop 2023-03-03T07:58:03.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.609+0000 7fe6d39ff700 1 -- 172.21.15.98:0/1698556271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0b87c0 msgr2=0x7fe6cc0b1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:03.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.609+0000 7fe6d39ff700 1 --2- 172.21.15.98:0/1698556271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0b87c0 0x7fe6cc0b1b80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc00c450 tx=0x7fe6bc00c760 comp rx=0 tx=0).stop 2023-03-03T07:58:03.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.609+0000 7fe6d39ff700 1 -- 172.21.15.98:0/1698556271 shutdown_connections 2023-03-03T07:58:03.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.609+0000 7fe6d39ff700 1 --2- 172.21.15.98:0/1698556271 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6b00683c0 0x7fe6b006a870 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:03.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.609+0000 7fe6d39ff700 1 --2- 172.21.15.98:0/1698556271 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6cc0b8b90 0x7fe6cc1f5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:03.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.609+0000 7fe6d39ff700 1 --2- 172.21.15.98:0/1698556271 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6cc0b5410 0x7fe6cc1f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:03.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.609+0000 7fe6d39ff700 1 --2- 172.21.15.98:0/1698556271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0b87c0 0x7fe6cc0b1b80 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:03.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.609+0000 7fe6d39ff700 1 -- 172.21.15.98:0/1698556271 >> 172.21.15.98:0/1698556271 conn(0x7fe6cc06aee0 msgr2=0x7fe6cc0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:03.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.609+0000 7fe6d39ff700 1 -- 172.21.15.98:0/1698556271 shutdown_connections 2023-03-03T07:58:03.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.609+0000 7fe6d39ff700 1 -- 172.21.15.98:0/1698556271 wait complete. 2023-03-03T07:58:03.641 INFO:tasks.cephfs_test_runner:test_binary_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) ... ok 2023-03-03T07:58:03.642 INFO:tasks.cephfs_test_runner:Starting test: test_connection_expiration (tasks.cephfs.test_volumes.TestMisc) 2023-03-03T07:58:03.642 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration' 2023-03-03T07:58:03.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.888+0000 7f46975ff700 1 Processor -- start 2023-03-03T07:58:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.888+0000 7f46975ff700 1 -- start start 2023-03-03T07:58:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.889+0000 7f46975ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f46900cd8d0 0x7f46900b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.889+0000 7f46975ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f46900b6610 0x7f46900cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.889+0000 7f46975ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46900cdd70 con 0x7f469015cc00 2023-03-03T07:58:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.889+0000 7f46975ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46900ce040 con 0x7f46900cd8d0 2023-03-03T07:58:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.889+0000 7f46975ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4690073be0 con 0x7f46900b6610 2023-03-03T07:58:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.890+0000 7f4694b9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f46900cd8d0 0x7f46900b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.890+0000 7f469539b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f469015cc00 0x7f46900d27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58990/0 (socket says 172.21.15.98:58990) 2023-03-03T07:58:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.890+0000 7f469539b700 1 -- 172.21.15.98:0/1324085730 learned_addr learned my addr 172.21.15.98:0/1324085730 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.890+0000 7f4695b9c700 1 --2- 172.21.15.98:0/1324085730 >> v2:172.21.15.100:3300/0 conn(0x7f46900b6610 0x7f46900cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.890+0000 7f4687fff700 1 -- 172.21.15.98:0/1324085730 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058639296 0 0) 0x7f46900cdd70 con 0x7f469015cc00 2023-03-03T07:58:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.890+0000 7f4694b9a700 1 -- 172.21.15.98:0/1324085730 >> v2:172.21.15.100:3300/0 conn(0x7f46900b6610 msgr2=0x7f46900cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.890+0000 7f4694b9a700 1 --2- 172.21.15.98:0/1324085730 >> v2:172.21.15.100:3300/0 conn(0x7f46900b6610 0x7f46900cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.890+0000 7f4694b9a700 1 -- 172.21.15.98:0/1324085730 >> v1:172.21.15.98:6789/0 conn(0x7f469015cc00 legacy=0x7f46900d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:03.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.890+0000 7f4694b9a700 1 -- 172.21.15.98:0/1324085730 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46900d1a00 con 0x7f46900cd8d0 2023-03-03T07:58:03.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.890+0000 7f4695b9c700 1 --2- 172.21.15.98:0/1324085730 >> v2:172.21.15.100:3300/0 conn(0x7f46900b6610 0x7f46900cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:03.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.891+0000 7f4694b9a700 1 --2- 172.21.15.98:0/1324085730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46900cd8d0 0x7f46900b60d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f46900d4360 tx=0x7f4680009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d758908291e353 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:03.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.891+0000 7f4687fff700 1 -- 172.21.15.98:0/1324085730 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680011830 con 0x7f46900cd8d0 2023-03-03T07:58:03.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.891+0000 7f4687fff700 1 -- 172.21.15.98:0/1324085730 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4680011990 con 0x7f46900cd8d0 2023-03-03T07:58:03.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.891+0000 7f4687fff700 1 -- 172.21.15.98:0/1324085730 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680011d20 con 0x7f46900cd8d0 2023-03-03T07:58:03.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.891+0000 7f46975ff700 1 -- 172.21.15.98:0/1324085730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46900cd8d0 msgr2=0x7f46900b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:03.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.891+0000 7f46975ff700 1 --2- 172.21.15.98:0/1324085730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46900cd8d0 0x7f46900b60d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f46900d4360 tx=0x7f4680009b20 comp rx=0 tx=0).stop 2023-03-03T07:58:03.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.892+0000 7f46975ff700 1 -- 172.21.15.98:0/1324085730 shutdown_connections 2023-03-03T07:58:03.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.892+0000 7f46975ff700 1 --2- 172.21.15.98:0/1324085730 >> v2:172.21.15.100:3300/0 conn(0x7f46900b6610 0x7f46900cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:03.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.892+0000 7f46975ff700 1 --2- 172.21.15.98:0/1324085730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46900cd8d0 0x7f46900b60d0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:03.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.892+0000 7f46975ff700 1 -- 172.21.15.98:0/1324085730 >> 172.21.15.98:0/1324085730 conn(0x7f469006aee0 msgr2=0x7f469006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:03.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.892+0000 7f46975ff700 1 -- 172.21.15.98:0/1324085730 shutdown_connections 2023-03-03T07:58:03.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.892+0000 7f46975ff700 1 -- 172.21.15.98:0/1324085730 wait complete. 2023-03-03T07:58:03.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.894+0000 7f46975ff700 1 Processor -- start 2023-03-03T07:58:03.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.894+0000 7f46975ff700 1 -- start start 2023-03-03T07:58:03.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.895+0000 7f46975ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46900b6610 0x7f469014ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:03.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.895+0000 7f46975ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46900cd8d0 0x7f46900c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:03.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.895+0000 7f46975ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f469015cc00 0x7f46900c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:03.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.895+0000 7f46975ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46900c9ee0 con 0x7f46900b6610 2023-03-03T07:58:03.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.895+0000 7f46975ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46900c9640 con 0x7f46900cd8d0 2023-03-03T07:58:03.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.895+0000 7f46975ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46900c9920 con 0x7f469015cc00 2023-03-03T07:58:03.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.895+0000 7f469539b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46900b6610 0x7f469014ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:03.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.895+0000 7f469539b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46900b6610 0x7f469014ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48788/0 (socket says 172.21.15.98:48788) 2023-03-03T07:58:03.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.895+0000 7f469539b700 1 -- 172.21.15.98:0/732847101 learned_addr learned my addr 172.21.15.98:0/732847101 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:03.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.895+0000 7f4695b9c700 1 --2- 172.21.15.98:0/732847101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f469015cc00 0x7f46900c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:03.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.895+0000 7f4694b9a700 1 --2- 172.21.15.98:0/732847101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46900cd8d0 0x7f46900c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:03.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.895+0000 7f469539b700 1 -- 172.21.15.98:0/732847101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f469015cc00 msgr2=0x7f46900c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:03.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.895+0000 7f469539b700 1 --2- 172.21.15.98:0/732847101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f469015cc00 0x7f46900c72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:03.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.896+0000 7f469539b700 1 -- 172.21.15.98:0/732847101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46900cd8d0 msgr2=0x7f46900c6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:03.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.896+0000 7f469539b700 1 --2- 172.21.15.98:0/732847101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46900cd8d0 0x7f46900c6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:03.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.896+0000 7f469539b700 1 -- 172.21.15.98:0/732847101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46800094a0 con 0x7f46900b6610 2023-03-03T07:58:03.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.896+0000 7f4695b9c700 1 --2- 172.21.15.98:0/732847101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f469015cc00 0x7f46900c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:03.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.896+0000 7f469539b700 1 --2- 172.21.15.98:0/732847101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46900b6610 0x7f469014ea60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f468c0011e0 tx=0x7f468c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:03.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.896+0000 7f4685ffb700 1 -- 172.21.15.98:0/732847101 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c004c80 con 0x7f46900b6610 2023-03-03T07:58:03.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.896+0000 7f4685ffb700 1 -- 172.21.15.98:0/732847101 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468c004de0 con 0x7f46900b6610 2023-03-03T07:58:03.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.896+0000 7f46975ff700 1 -- 172.21.15.98:0/732847101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46900d1fc0 con 0x7f46900b6610 2023-03-03T07:58:03.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.896+0000 7f4685ffb700 1 -- 172.21.15.98:0/732847101 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c007310 con 0x7f46900b6610 2023-03-03T07:58:03.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.896+0000 7f46975ff700 1 -- 172.21.15.98:0/732847101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46900d1ce0 con 0x7f46900b6610 2023-03-03T07:58:03.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.897+0000 7f46975ff700 1 -- 172.21.15.98:0/732847101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46900d22a0 con 0x7f46900b6610 2023-03-03T07:58:03.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.899+0000 7f4685ffb700 1 -- 172.21.15.98:0/732847101 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f468c01a020 con 0x7f46900b6610 2023-03-03T07:58:03.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.900+0000 7f4685ffb700 1 --2- 172.21.15.98:0/732847101 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46740683c0 0x7f467406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:03.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.900+0000 7f4685ffb700 1 -- 172.21.15.98:0/732847101 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f468c0856a0 con 0x7f46900b6610 2023-03-03T07:58:03.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.900+0000 7f4694b9a700 1 --2- 172.21.15.98:0/732847101 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46740683c0 0x7f467406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:03.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.901+0000 7f4694b9a700 1 --2- 172.21.15.98:0/732847101 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46740683c0 0x7f467406a870 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f46800090b0 tx=0x7f468001c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:03.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:03.905+0000 7f4685ffb700 1 -- 172.21.15.98:0/732847101 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f468c052130 con 0x7f46900b6610 2023-03-03T07:58:04.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.172+0000 7f46975ff700 1 -- 172.21.15.98:0/732847101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]} v 0) v1 -- 0x7f46900729b0 con 0x7f46900b6610 2023-03-03T07:58:04.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.610+0000 7f4685ffb700 1 -- 172.21.15.98:0/732847101 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]}]=0 v153) v1 ==== 143+0+0 (secure 0 0 0) 0x7f468c055750 con 0x7f46900b6610 2023-03-03T07:58:04.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.613+0000 7f46975ff700 1 -- 172.21.15.98:0/732847101 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46740683c0 msgr2=0x7f467406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:04.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.613+0000 7f46975ff700 1 --2- 172.21.15.98:0/732847101 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46740683c0 0x7f467406a870 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f46800090b0 tx=0x7f468001c040 comp rx=0 tx=0).stop 2023-03-03T07:58:04.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.613+0000 7f46975ff700 1 -- 172.21.15.98:0/732847101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46900b6610 msgr2=0x7f469014ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:04.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.613+0000 7f46975ff700 1 --2- 172.21.15.98:0/732847101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46900b6610 0x7f469014ea60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f468c0011e0 tx=0x7f468c00ca30 comp rx=0 tx=0).stop 2023-03-03T07:58:04.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.613+0000 7f46975ff700 1 -- 172.21.15.98:0/732847101 shutdown_connections 2023-03-03T07:58:04.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.613+0000 7f46975ff700 1 --2- 172.21.15.98:0/732847101 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46740683c0 0x7f467406a870 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:04.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.613+0000 7f46975ff700 1 --2- 172.21.15.98:0/732847101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46900b6610 0x7f469014ea60 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.613+0000 7f46975ff700 1 --2- 172.21.15.98:0/732847101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f469015cc00 0x7f46900c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.613+0000 7f46975ff700 1 --2- 172.21.15.98:0/732847101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46900cd8d0 0x7f46900c6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.613+0000 7f46975ff700 1 -- 172.21.15.98:0/732847101 >> 172.21.15.98:0/732847101 conn(0x7f469006aee0 msgr2=0x7f4690159880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.614+0000 7f46975ff700 1 -- 172.21.15.98:0/732847101 shutdown_connections 2023-03-03T07:58:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.614+0000 7f46975ff700 1 -- 172.21.15.98:0/732847101 wait complete. 2023-03-03T07:58:04.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-03-03T07:58:04.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.890+0000 7f924bdff700 1 Processor -- start 2023-03-03T07:58:04.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.890+0000 7f924bdff700 1 -- start start 2023-03-03T07:58:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.891+0000 7f924bdff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f92440b8d10 0x7f92440cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.891+0000 7f924bdff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f92440b26d0 0x7f92440b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.892+0000 7f924bdff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92440cd7d0 con 0x7f92440b5140 2023-03-03T07:58:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.892+0000 7f924bdff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92440b55c0 con 0x7f92440b8d10 2023-03-03T07:58:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.892+0000 7f924bdff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92440b90e0 con 0x7f92440b26d0 2023-03-03T07:58:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.892+0000 7f9249b9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f92440b8d10 0x7f92440cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.892+0000 7f9249b9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f92440b8d10 0x7f92440cffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48800/0 (socket says 172.21.15.98:48800) 2023-03-03T07:58:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.892+0000 7f9249b9b700 1 -- 172.21.15.98:0/1353365089 learned_addr learned my addr 172.21.15.98:0/1353365089 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.892+0000 7f9249b9b700 1 -- 172.21.15.98:0/1353365089 >> v2:172.21.15.100:3300/0 conn(0x7f92440b26d0 msgr2=0x7f92440b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.892+0000 7f924939a700 1 --2- 172.21.15.98:0/1353365089 >> v2:172.21.15.100:3300/0 conn(0x7f92440b26d0 0x7f92440b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.892+0000 7f9249b9b700 1 --2- 172.21.15.98:0/1353365089 >> v2:172.21.15.100:3300/0 conn(0x7f92440b26d0 0x7f92440b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.892+0000 7f9249b9b700 1 -- 172.21.15.98:0/1353365089 >> v1:172.21.15.100:6789/0 conn(0x7f92440b5140 legacy=0x7f9244160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.893+0000 7f9249b9b700 1 -- 172.21.15.98:0/1353365089 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92440cda20 con 0x7f92440b8d10 2023-03-03T07:58:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.893+0000 7f924939a700 1 --2- 172.21.15.98:0/1353365089 >> v2:172.21.15.100:3300/0 conn(0x7f92440b26d0 0x7f92440b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.893+0000 7f9249b9b700 1 --2- 172.21.15.98:0/1353365089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92440b8d10 0x7f92440cffc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f924408bfa0 tx=0x7f9240009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76dc62cba244737d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.893+0000 7f9248b99700 1 -- 172.21.15.98:0/1353365089 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924000b870 con 0x7f92440b8d10 2023-03-03T07:58:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.893+0000 7f9248b99700 1 -- 172.21.15.98:0/1353365089 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924000b9d0 con 0x7f92440b8d10 2023-03-03T07:58:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.893+0000 7f9248b99700 1 -- 172.21.15.98:0/1353365089 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924000bd60 con 0x7f92440b8d10 2023-03-03T07:58:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.894+0000 7f924bdff700 1 -- 172.21.15.98:0/1353365089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92440b8d10 msgr2=0x7f92440cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.894+0000 7f924bdff700 1 --2- 172.21.15.98:0/1353365089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92440b8d10 0x7f92440cffc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f924408bfa0 tx=0x7f9240009bd0 comp rx=0 tx=0).stop 2023-03-03T07:58:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.894+0000 7f924bdff700 1 -- 172.21.15.98:0/1353365089 shutdown_connections 2023-03-03T07:58:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.894+0000 7f924bdff700 1 --2- 172.21.15.98:0/1353365089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92440b8d10 0x7f92440cffc0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.894+0000 7f924bdff700 1 --2- 172.21.15.98:0/1353365089 >> v2:172.21.15.100:3300/0 conn(0x7f92440b26d0 0x7f92440b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.894+0000 7f924bdff700 1 -- 172.21.15.98:0/1353365089 >> 172.21.15.98:0/1353365089 conn(0x7f9244073e70 msgr2=0x7f924406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.894+0000 7f924bdff700 1 -- 172.21.15.98:0/1353365089 shutdown_connections 2023-03-03T07:58:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.894+0000 7f924bdff700 1 -- 172.21.15.98:0/1353365089 wait complete. 2023-03-03T07:58:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.896+0000 7f924bdff700 1 Processor -- start 2023-03-03T07:58:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.896+0000 7f924bdff700 1 -- start start 2023-03-03T07:58:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.897+0000 7f924bdff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92440b26d0 0x7f92440cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.897+0000 7f924bdff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92440b5140 0x7f92440c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.897+0000 7f924bdff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92440b8d10 0x7f92440cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.897+0000 7f924bdff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924415ff90 con 0x7f92440b26d0 2023-03-03T07:58:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.897+0000 7f924bdff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924415d430 con 0x7f92440b5140 2023-03-03T07:58:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.897+0000 7f924bdff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924415dc90 con 0x7f92440b8d10 2023-03-03T07:58:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.898+0000 7f9249b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92440b26d0 0x7f92440cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.898+0000 7f9249b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92440b26d0 0x7f92440cd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48808/0 (socket says 172.21.15.98:48808) 2023-03-03T07:58:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.898+0000 7f9249b9b700 1 -- 172.21.15.98:0/689142375 learned_addr learned my addr 172.21.15.98:0/689142375 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:04.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.898+0000 7f924939a700 1 --2- 172.21.15.98:0/689142375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92440b5140 0x7f92440c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:04.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.898+0000 7f924a39c700 1 --2- 172.21.15.98:0/689142375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92440b8d10 0x7f92440cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.898+0000 7f9249b9b700 1 -- 172.21.15.98:0/689142375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92440b8d10 msgr2=0x7f92440cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.898+0000 7f9249b9b700 1 --2- 172.21.15.98:0/689142375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92440b8d10 0x7f92440cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.898+0000 7f9249b9b700 1 -- 172.21.15.98:0/689142375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92440b5140 msgr2=0x7f92440c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.898+0000 7f9249b9b700 1 --2- 172.21.15.98:0/689142375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92440b5140 0x7f92440c8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:04.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.898+0000 7f9249b9b700 1 -- 172.21.15.98:0/689142375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9240009550 con 0x7f92440b26d0 2023-03-03T07:58:04.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.898+0000 7f924a39c700 1 --2- 172.21.15.98:0/689142375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92440b8d10 0x7f92440cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.898+0000 7f9249b9b700 1 --2- 172.21.15.98:0/689142375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92440b26d0 0x7f92440cd050 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f9240009130 tx=0x7f9240004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.899+0000 7f923a7fc700 1 -- 172.21.15.98:0/689142375 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9240004520 con 0x7f92440b26d0 2023-03-03T07:58:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.899+0000 7f923a7fc700 1 -- 172.21.15.98:0/689142375 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924001e070 con 0x7f92440b26d0 2023-03-03T07:58:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.899+0000 7f923a7fc700 1 -- 172.21.15.98:0/689142375 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92400038c0 con 0x7f92440b26d0 2023-03-03T07:58:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.899+0000 7f924bdff700 1 -- 172.21.15.98:0/689142375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92440d0e80 con 0x7f92440b26d0 2023-03-03T07:58:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.899+0000 7f924bdff700 1 -- 172.21.15.98:0/689142375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92440cdd00 con 0x7f92440b26d0 2023-03-03T07:58:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.900+0000 7f924bdff700 1 -- 172.21.15.98:0/689142375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92440cdfe0 con 0x7f92440b26d0 2023-03-03T07:58:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.901+0000 7f923a7fc700 1 -- 172.21.15.98:0/689142375 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f924000f050 con 0x7f92440b26d0 2023-03-03T07:58:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.902+0000 7f923a7fc700 1 --2- 172.21.15.98:0/689142375 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f92280683c0 0x7f922806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.902+0000 7f923a7fc700 1 -- 172.21.15.98:0/689142375 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f9240020070 con 0x7f92440b26d0 2023-03-03T07:58:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.903+0000 7f924939a700 1 --2- 172.21.15.98:0/689142375 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f92280683c0 0x7f922806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.903+0000 7f924939a700 1 --2- 172.21.15.98:0/689142375 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f92280683c0 0x7f922806a870 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9234006fd0 tx=0x7f9234008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:04.906+0000 7f923a7fc700 1 -- 172.21.15.98:0/689142375 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9240053cd0 con 0x7f92440b26d0 2023-03-03T07:58:05.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.100+0000 7f924bdff700 1 -- 172.21.15.98:0/689142375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f92440cf470 con 0x7f92440b26d0 2023-03-03T07:58:05.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.104+0000 7f923a7fc700 1 -- 172.21.15.98:0/689142375 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f92400572f0 con 0x7f92440b26d0 2023-03-03T07:58:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.107+0000 7f924bdff700 1 -- 172.21.15.98:0/689142375 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f92280683c0 msgr2=0x7f922806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.107+0000 7f924bdff700 1 --2- 172.21.15.98:0/689142375 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f92280683c0 0x7f922806a870 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9234006fd0 tx=0x7f9234008040 comp rx=0 tx=0).stop 2023-03-03T07:58:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.107+0000 7f924bdff700 1 -- 172.21.15.98:0/689142375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92440b26d0 msgr2=0x7f92440cd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.107+0000 7f924bdff700 1 --2- 172.21.15.98:0/689142375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92440b26d0 0x7f92440cd050 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f9240009130 tx=0x7f9240004750 comp rx=0 tx=0).stop 2023-03-03T07:58:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.107+0000 7f924bdff700 1 -- 172.21.15.98:0/689142375 shutdown_connections 2023-03-03T07:58:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.107+0000 7f924bdff700 1 --2- 172.21.15.98:0/689142375 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f92280683c0 0x7f922806a870 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:05.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.107+0000 7f924bdff700 1 --2- 172.21.15.98:0/689142375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92440b26d0 0x7f92440cd050 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:05.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.107+0000 7f924bdff700 1 --2- 172.21.15.98:0/689142375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92440b8d10 0x7f92440cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:05.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.107+0000 7f924bdff700 1 --2- 172.21.15.98:0/689142375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92440b5140 0x7f92440c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:05.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.107+0000 7f924bdff700 1 -- 172.21.15.98:0/689142375 >> 172.21.15.98:0/689142375 conn(0x7f9244073e70 msgr2=0x7f92440b2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:05.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.107+0000 7f924bdff700 1 -- 172.21.15.98:0/689142375 shutdown_connections 2023-03-03T07:58:05.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.107+0000 7f924bdff700 1 -- 172.21.15.98:0/689142375 wait complete. 2023-03-03T07:58:05.137 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-03-03T07:58:05.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T07:58:05.266 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T07:58:05.400 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-03-03T07:58:05.401 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:58:05.401 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:58:05.469 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T07:58:05.469 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:58:05.470 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:58:05.538 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T07:58:05.539 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:58:05.539 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:58:05.608 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T07:58:05.609 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:58:05.609 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:58:05.679 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T07:58:05.680 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-03-03T07:58:05.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:58:05.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.922+0000 7f83089fe700 1 Processor -- start 2023-03-03T07:58:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.922+0000 7f83089fe700 1 -- start start 2023-03-03T07:58:05.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.924+0000 7f83089fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83040b51c0 0x7f8304160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:05.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.924+0000 7f83089fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83040d1580 0x7f83040d19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:05.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.924+0000 7f83089fe700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83041614c0 con 0x7f8304146360 2023-03-03T07:58:05.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.924+0000 7f83089fe700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83040d10f0 con 0x7f83040b51c0 2023-03-03T07:58:05.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.924+0000 7f83089fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83040cdf20 con 0x7f83040d1580 2023-03-03T07:58:05.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.924+0000 7f82fbfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83040b51c0 0x7f8304160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:05.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.924+0000 7f82fbfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83040b51c0 0x7f8304160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54288/0 (socket says 172.21.15.98:54288) 2023-03-03T07:58:05.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.924+0000 7f82fbfff700 1 -- 172.21.15.98:0/1928704196 learned_addr learned my addr 172.21.15.98:0/1928704196 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:05.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.924+0000 7f8302d9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8304146360 0x7f83040b4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57324/0 (socket says 172.21.15.98:57324) 2023-03-03T07:58:05.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.924+0000 7f830359e700 1 --2- 172.21.15.98:0/1928704196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83040d1580 0x7f83040d19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:05.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.925+0000 7f82fbfff700 1 -- 172.21.15.98:0/1928704196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83040d1580 msgr2=0x7f83040d19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:05.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.925+0000 7f82fbfff700 1 --2- 172.21.15.98:0/1928704196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83040d1580 0x7f83040d19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:05.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.925+0000 7f82fbfff700 1 -- 172.21.15.98:0/1928704196 >> v1:172.21.15.98:6789/0 conn(0x7f8304146360 legacy=0x7f83040b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:05.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.925+0000 7f82fbfff700 1 -- 172.21.15.98:0/1928704196 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83040cd680 con 0x7f83040b51c0 2023-03-03T07:58:05.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.925+0000 7f830259c700 1 -- 172.21.15.98:0/1928704196 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770407221 0 0) 0x7f82f4001110 con 0x7f8304146360 2023-03-03T07:58:05.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.925+0000 7f82fbfff700 1 --2- 172.21.15.98:0/1928704196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83040b51c0 0x7f8304160f80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0098f0 tx=0x7f82ec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9774bf1d858dbded server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:05.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.925+0000 7f830259c700 1 -- 172.21.15.98:0/1928704196 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec011830 con 0x7f83040b51c0 2023-03-03T07:58:05.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.925+0000 7f830259c700 1 -- 172.21.15.98:0/1928704196 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec011990 con 0x7f83040b51c0 2023-03-03T07:58:05.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.925+0000 7f830259c700 1 -- 172.21.15.98:0/1928704196 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec011d20 con 0x7f83040b51c0 2023-03-03T07:58:05.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.926+0000 7f83089fe700 1 -- 172.21.15.98:0/1928704196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83040b51c0 msgr2=0x7f8304160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:05.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.926+0000 7f83089fe700 1 --2- 172.21.15.98:0/1928704196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83040b51c0 0x7f8304160f80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0098f0 tx=0x7f82ec009c00 comp rx=0 tx=0).stop 2023-03-03T07:58:05.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.926+0000 7f83089fe700 1 -- 172.21.15.98:0/1928704196 shutdown_connections 2023-03-03T07:58:05.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.926+0000 7f83089fe700 1 --2- 172.21.15.98:0/1928704196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83040d1580 0x7f83040d19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:05.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.926+0000 7f83089fe700 1 --2- 172.21.15.98:0/1928704196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83040b51c0 0x7f8304160f80 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:05.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.926+0000 7f83089fe700 1 -- 172.21.15.98:0/1928704196 >> 172.21.15.98:0/1928704196 conn(0x7f830406b8e0 msgr2=0x7f83040cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:05.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.926+0000 7f83089fe700 1 -- 172.21.15.98:0/1928704196 shutdown_connections 2023-03-03T07:58:05.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.926+0000 7f83089fe700 1 -- 172.21.15.98:0/1928704196 wait complete. 2023-03-03T07:58:05.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.928+0000 7f83089fe700 1 Processor -- start 2023-03-03T07:58:05.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.929+0000 7f83089fe700 1 -- start start 2023-03-03T07:58:05.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.929+0000 7f83089fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83040b51c0 0x7f830415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:05.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f83089fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83040d1580 0x7f83041f3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:05.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f83089fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8304146360 0x7f830415f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:05.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f83089fe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83041f3560 con 0x7f83040d1580 2023-03-03T07:58:05.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f83089fe700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830415e880 con 0x7f83040b51c0 2023-03-03T07:58:05.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f83089fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830415eb60 con 0x7f8304146360 2023-03-03T07:58:05.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f82fbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83040d1580 0x7f83041f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:05.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f8302d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83040b51c0 0x7f830415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:05.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f82fbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83040d1580 0x7f83041f3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54294/0 (socket says 172.21.15.98:54294) 2023-03-03T07:58:05.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f82fbfff700 1 -- 172.21.15.98:0/1319893558 learned_addr learned my addr 172.21.15.98:0/1319893558 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:05.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f830359e700 1 --2- 172.21.15.98:0/1319893558 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8304146360 0x7f830415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:05.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f8302d9d700 1 -- 172.21.15.98:0/1319893558 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8304146360 msgr2=0x7f830415f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:05.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f8302d9d700 1 --2- 172.21.15.98:0/1319893558 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8304146360 0x7f830415f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:05.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f8302d9d700 1 -- 172.21.15.98:0/1319893558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83040d1580 msgr2=0x7f83041f3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:05.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f8302d9d700 1 --2- 172.21.15.98:0/1319893558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83040d1580 0x7f83041f3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:05.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f8302d9d700 1 -- 172.21.15.98:0/1319893558 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec009550 con 0x7f83040b51c0 2023-03-03T07:58:05.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.930+0000 7f82fbfff700 1 --2- 172.21.15.98:0/1319893558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83040d1580 0x7f83041f3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:58:05.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.931+0000 7f8302d9d700 1 --2- 172.21.15.98:0/1319893558 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83040b51c0 0x7f830415a2e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f82f400c780 tx=0x7f82f400ca90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:05.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.931+0000 7f82fb7fe700 1 -- 172.21.15.98:0/1319893558 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4004c80 con 0x7f83040b51c0 2023-03-03T07:58:05.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.931+0000 7f82fb7fe700 1 -- 172.21.15.98:0/1319893558 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f4004e00 con 0x7f83040b51c0 2023-03-03T07:58:05.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.931+0000 7f82fb7fe700 1 -- 172.21.15.98:0/1319893558 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4007310 con 0x7f83040b51c0 2023-03-03T07:58:05.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.931+0000 7f83089fe700 1 -- 172.21.15.98:0/1319893558 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83040cdc40 con 0x7f83040b51c0 2023-03-03T07:58:05.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.931+0000 7f83089fe700 1 -- 172.21.15.98:0/1319893558 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83040cd960 con 0x7f83040b51c0 2023-03-03T07:58:05.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.933+0000 7f83089fe700 1 -- 172.21.15.98:0/1319893558 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83040d0d80 con 0x7f83040b51c0 2023-03-03T07:58:05.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.934+0000 7f82fb7fe700 1 -- 172.21.15.98:0/1319893558 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f82f401a020 con 0x7f83040b51c0 2023-03-03T07:58:05.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.935+0000 7f82fb7fe700 1 --2- 172.21.15.98:0/1319893558 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f82e40683c0 0x7f82e406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:05.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.935+0000 7f82fb7fe700 1 -- 172.21.15.98:0/1319893558 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f82f4085410 con 0x7f83040b51c0 2023-03-03T07:58:05.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.935+0000 7f82fbfff700 1 --2- 172.21.15.98:0/1319893558 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f82e40683c0 0x7f82e406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:05.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.936+0000 7f82fbfff700 1 --2- 172.21.15.98:0/1319893558 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f82e40683c0 0x7f82e406a870 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f82ec000c00 tx=0x7f82ec008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:05.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:05.940+0000 7f82fb7fe700 1 -- 172.21.15.98:0/1319893558 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f82f4051dd0 con 0x7f83040b51c0 2023-03-03T07:58:06.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.177+0000 7f83089fe700 1 -- 172.21.15.98:0/1319893558 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83041614c0 con 0x7f83040b51c0 2023-03-03T07:58:06.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.178+0000 7f82fb7fe700 1 -- 172.21.15.98:0/1319893558 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f83041614c0 con 0x7f83040b51c0 2023-03-03T07:58:06.180 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:06.180 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3849166495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3849166495},{"type":"v1","addr":"172.21.15.100:6834","nonce":3849166495}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/4285491249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":4285491249},{"type":"v1","addr":"172.21.15.100:6835","nonce":4285491249}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":5208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3888598890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3888598890},{"type":"v1","addr":"172.21.15.98:6836","nonce":3888598890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":5667,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/4278778172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":4278778172},{"type":"v1","addr":"172.21.15.98:6837","nonce":4278778172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[]} 2023-03-03T07:58:06.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.180+0000 7f83089fe700 1 -- 172.21.15.98:0/1319893558 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f82e40683c0 msgr2=0x7f82e406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:06.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.180+0000 7f83089fe700 1 --2- 172.21.15.98:0/1319893558 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f82e40683c0 0x7f82e406a870 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f82ec000c00 tx=0x7f82ec008030 comp rx=0 tx=0).stop 2023-03-03T07:58:06.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.180+0000 7f83089fe700 1 -- 172.21.15.98:0/1319893558 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83040b51c0 msgr2=0x7f830415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:06.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.180+0000 7f83089fe700 1 --2- 172.21.15.98:0/1319893558 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83040b51c0 0x7f830415a2e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f82f400c780 tx=0x7f82f400ca90 comp rx=0 tx=0).stop 2023-03-03T07:58:06.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.180+0000 7f83089fe700 1 -- 172.21.15.98:0/1319893558 shutdown_connections 2023-03-03T07:58:06.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.180+0000 7f83089fe700 1 --2- 172.21.15.98:0/1319893558 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f82e40683c0 0x7f82e406a870 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:06.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.180+0000 7f83089fe700 1 --2- 172.21.15.98:0/1319893558 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8304146360 0x7f830415f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:06.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.180+0000 7f83089fe700 1 --2- 172.21.15.98:0/1319893558 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83040b51c0 0x7f830415a2e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:06.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.180+0000 7f83089fe700 1 --2- 172.21.15.98:0/1319893558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83040d1580 0x7f83041f3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:06.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.180+0000 7f83089fe700 1 -- 172.21.15.98:0/1319893558 >> 172.21.15.98:0/1319893558 conn(0x7f830406b8e0 msgr2=0x7f83040731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:06.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.181+0000 7f83089fe700 1 -- 172.21.15.98:0/1319893558 shutdown_connections 2023-03-03T07:58:06.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.181+0000 7f83089fe700 1 -- 172.21.15.98:0/1319893558 wait complete. 2023-03-03T07:58:06.185 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 17 2023-03-03T07:58:06.210 INFO:tasks.ceph.mds.a:Restarting daemon 2023-03-03T07:58:06.211 INFO:tasks.ceph.mds.a:Stopping old one... 2023-03-03T07:58:06.211 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-03-03T07:58:06.211 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T07:58:06.211 INFO:tasks.ceph.mds.c:Restarting daemon 2023-03-03T07:58:06.211 INFO:tasks.ceph.mds.c:Stopping old one... 2023-03-03T07:58:06.212 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-03-03T07:58:06.212 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T07:58:06.212 INFO:tasks.ceph.mds.b:Restarting daemon 2023-03-03T07:58:06.212 INFO:tasks.ceph.mds.b:Stopping old one... 2023-03-03T07:58:06.213 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-03-03T07:58:06.213 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T07:58:06.213 INFO:tasks.ceph.mds.d:Restarting daemon 2023-03-03T07:58:06.213 INFO:tasks.ceph.mds.d:Stopping old one... 2023-03-03T07:58:06.213 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-03-03T07:58:06.214 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T07:58:06.232 INFO:tasks.ceph.mds.a:Stopped 2023-03-03T07:58:06.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-03-03T07:58:06.234 INFO:tasks.ceph.mds.c:Stopped 2023-03-03T07:58:06.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-03-03T07:58:06.237 INFO:tasks.ceph.mds.b:Stopped 2023-03-03T07:58:06.237 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-03-03T07:58:06.238 INFO:tasks.ceph.mds.d:Stopped 2023-03-03T07:58:06.238 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-03-03T07:58:06.279 INFO:tasks.ceph.mds.a:Started 2023-03-03T07:58:06.279 DEBUG:teuthology.parallel:result is None 2023-03-03T07:58:06.282 INFO:tasks.ceph.mds.c:Started 2023-03-03T07:58:06.282 DEBUG:teuthology.parallel:result is None 2023-03-03T07:58:06.283 INFO:tasks.ceph.mds.b:Started 2023-03-03T07:58:06.284 DEBUG:teuthology.parallel:result is None 2023-03-03T07:58:06.287 INFO:tasks.ceph.mds.d:Started 2023-03-03T07:58:06.288 DEBUG:teuthology.parallel:result is None 2023-03-03T07:58:06.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-03-03T07:58:06.476 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2023-03-03T07:58:06.476 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2023-03-03T07:58:06.480 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-03-03T07:58:06.482 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2023-03-03T07:58:06.485 INFO:tasks.ceph.mds.c.smithi098.stderr:2023-03-03T07:58:06.483+0000 7fa4e8ffd700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6835/0: (98) Address already in use 2023-03-03T07:58:06.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.550+0000 7f8cb759e700 1 Processor -- start 2023-03-03T07:58:06.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.550+0000 7f8cb759e700 1 -- start start 2023-03-03T07:58:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.552+0000 7f8cb759e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb81609b0 0x7f8cb8160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.552+0000 7f8cb759e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8cb80cf570 0x7f8cb815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.552+0000 7f8cb759e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb81612c0 con 0x7f8cb815db30 2023-03-03T07:58:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.552+0000 7f8cb759e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb80cebb0 con 0x7f8cb80cf570 2023-03-03T07:58:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.552+0000 7f8cb759e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80cd780 con 0x7f8cb81609b0 2023-03-03T07:58:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.552+0000 7f8cb5d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8cb80cf570 0x7f8cb815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.552+0000 7f8cb6d9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8cb815db30 0x7f8cb80b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57336/0 (socket says 172.21.15.98:57336) 2023-03-03T07:58:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.552+0000 7f8cb5d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8cb80cf570 0x7f8cb815d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54310/0 (socket says 172.21.15.98:54310) 2023-03-03T07:58:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.552+0000 7f8cb6d9d700 1 -- 172.21.15.98:0/3331480904 learned_addr learned my addr 172.21.15.98:0/3331480904 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.552+0000 7f8cb659c700 1 --2- 172.21.15.98:0/3331480904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb81609b0 0x7f8cb8160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.553+0000 7f8cb5d9b700 1 -- 172.21.15.98:0/3331480904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb81609b0 msgr2=0x7f8cb8160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.553+0000 7f8cb5d9b700 1 --2- 172.21.15.98:0/3331480904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb81609b0 0x7f8cb8160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.553+0000 7f8cb5d9b700 1 -- 172.21.15.98:0/3331480904 >> v1:172.21.15.98:6789/0 conn(0x7f8cb815db30 legacy=0x7f8cb80b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.553+0000 7f8cb5d9b700 1 -- 172.21.15.98:0/3331480904 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80cda60 con 0x7f8cb80cf570 2023-03-03T07:58:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.553+0000 7f8cb5d9b700 1 --2- 172.21.15.98:0/3331480904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80cf570 0x7f8cb815d5f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8cac009840 tx=0x7f8cac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73128a43c050837 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.553+0000 7f8cb559a700 1 -- 172.21.15.98:0/3331480904 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac011830 con 0x7f8cb80cf570 2023-03-03T07:58:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.553+0000 7f8cb559a700 1 -- 172.21.15.98:0/3331480904 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cac011990 con 0x7f8cb80cf570 2023-03-03T07:58:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.553+0000 7f8cb559a700 1 -- 172.21.15.98:0/3331480904 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac011d20 con 0x7f8cb80cf570 2023-03-03T07:58:06.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.554+0000 7f8cb759e700 1 -- 172.21.15.98:0/3331480904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80cf570 msgr2=0x7f8cb815d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:06.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.554+0000 7f8cb759e700 1 --2- 172.21.15.98:0/3331480904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80cf570 0x7f8cb815d5f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8cac009840 tx=0x7f8cac009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:06.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.554+0000 7f8cb759e700 1 -- 172.21.15.98:0/3331480904 shutdown_connections 2023-03-03T07:58:06.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.554+0000 7f8cb759e700 1 --2- 172.21.15.98:0/3331480904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb81609b0 0x7f8cb8160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:06.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.554+0000 7f8cb759e700 1 --2- 172.21.15.98:0/3331480904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80cf570 0x7f8cb815d5f0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:06.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.554+0000 7f8cb759e700 1 -- 172.21.15.98:0/3331480904 >> 172.21.15.98:0/3331480904 conn(0x7f8cb806b880 msgr2=0x7f8cb80cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:06.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.554+0000 7f8cb759e700 1 -- 172.21.15.98:0/3331480904 shutdown_connections 2023-03-03T07:58:06.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.554+0000 7f8cb759e700 1 -- 172.21.15.98:0/3331480904 wait complete. 2023-03-03T07:58:06.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.556+0000 7f8cb759e700 1 Processor -- start 2023-03-03T07:58:06.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.556+0000 7f8cb759e700 1 -- start start 2023-03-03T07:58:06.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.557+0000 7f8cb759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80cf570 0x7f8cb81485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:06.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.557+0000 7f8cb759e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb815db30 0x7f8cb815bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:06.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.558+0000 7f8cb759e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb81609b0 0x7f8cb815c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.558+0000 7f8cb759e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb8148ae0 con 0x7f8cb80cf570 2023-03-03T07:58:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.558+0000 7f8cb759e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80b62d0 con 0x7f8cb81609b0 2023-03-03T07:58:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.558+0000 7f8cb759e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80c6cf0 con 0x7f8cb815db30 2023-03-03T07:58:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.558+0000 7f8cb659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80cf570 0x7f8cb81485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.558+0000 7f8cb659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80cf570 0x7f8cb81485a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54314/0 (socket says 172.21.15.98:54314) 2023-03-03T07:58:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.558+0000 7f8cb659c700 1 -- 172.21.15.98:0/3787807691 learned_addr learned my addr 172.21.15.98:0/3787807691 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.558+0000 7f8cb5d9b700 1 --2- 172.21.15.98:0/3787807691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb815db30 0x7f8cb815bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.558+0000 7f8cb6d9d700 1 --2- 172.21.15.98:0/3787807691 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb81609b0 0x7f8cb815c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.558+0000 7f8cb659c700 1 -- 172.21.15.98:0/3787807691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb815db30 msgr2=0x7f8cb815bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.558+0000 7f8cb659c700 1 --2- 172.21.15.98:0/3787807691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb815db30 0x7f8cb815bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.558+0000 7f8cb659c700 1 -- 172.21.15.98:0/3787807691 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb81609b0 msgr2=0x7f8cb815c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.558+0000 7f8cb659c700 1 --2- 172.21.15.98:0/3787807691 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb81609b0 0x7f8cb815c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.558+0000 7f8cb659c700 1 -- 172.21.15.98:0/3787807691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac0094a0 con 0x7f8cb80cf570 2023-03-03T07:58:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.558+0000 7f8cb6d9d700 1 --2- 172.21.15.98:0/3787807691 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb81609b0 0x7f8cb815c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.559+0000 7f8cb659c700 1 --2- 172.21.15.98:0/3787807691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80cf570 0x7f8cb81485a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800e870 tx=0x7f8ca800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.559+0000 7f8ca6ffd700 1 -- 172.21.15.98:0/3787807691 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca8009590 con 0x7f8cb80cf570 2023-03-03T07:58:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.559+0000 7f8ca6ffd700 1 -- 172.21.15.98:0/3787807691 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca8013680 con 0x7f8cb80cf570 2023-03-03T07:58:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.559+0000 7f8ca6ffd700 1 -- 172.21.15.98:0/3787807691 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca80139f0 con 0x7f8cb80cf570 2023-03-03T07:58:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.559+0000 7f8cb759e700 1 -- 172.21.15.98:0/3787807691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb815cdf0 con 0x7f8cb80cf570 2023-03-03T07:58:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.559+0000 7f8cb759e700 1 -- 172.21.15.98:0/3787807691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb80cdd40 con 0x7f8cb80cf570 2023-03-03T07:58:06.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.561+0000 7f8cb759e700 1 -- 172.21.15.98:0/3787807691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb815d0d0 con 0x7f8cb80cf570 2023-03-03T07:58:06.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.562+0000 7f8ca6ffd700 1 -- 172.21.15.98:0/3787807691 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f8ca8023070 con 0x7f8cb80cf570 2023-03-03T07:58:06.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.563+0000 7f8ca6ffd700 1 --2- 172.21.15.98:0/3787807691 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c980683c0 0x7f8c9806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:06.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.563+0000 7f8ca6ffd700 1 -- 172.21.15.98:0/3787807691 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f8ca8085cf0 con 0x7f8cb80cf570 2023-03-03T07:58:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.563+0000 7f8cb5d9b700 1 --2- 172.21.15.98:0/3787807691 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c980683c0 0x7f8c9806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.564+0000 7f8cb5d9b700 1 --2- 172.21.15.98:0/3787807691 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c980683c0 0x7f8c9806a870 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0090b0 tx=0x7f8cac008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:06.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.567+0000 7f8ca6ffd700 1 -- 172.21.15.98:0/3787807691 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8ca8052650 con 0x7f8cb80cf570 2023-03-03T07:58:06.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:06.783+0000 7f8cb759e700 1 -- 172.21.15.98:0/3787807691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8cb8068290 con 0x7f8cb80cf570 2023-03-03T07:58:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.629+0000 7f8ca6ffd700 1 -- 172.21.15.98:0/3787807691 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v48) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8ca8055c70 con 0x7f8cb80cf570 2023-03-03T07:58:07.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.632+0000 7f8cb759e700 1 -- 172.21.15.98:0/3787807691 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c980683c0 msgr2=0x7f8c9806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:07.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.632+0000 7f8cb759e700 1 --2- 172.21.15.98:0/3787807691 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c980683c0 0x7f8c9806a870 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0090b0 tx=0x7f8cac008030 comp rx=0 tx=0).stop 2023-03-03T07:58:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.632+0000 7f8cb759e700 1 -- 172.21.15.98:0/3787807691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80cf570 msgr2=0x7f8cb81485a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.632+0000 7f8cb759e700 1 --2- 172.21.15.98:0/3787807691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80cf570 0x7f8cb81485a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800e870 tx=0x7f8ca800eb80 comp rx=0 tx=0).stop 2023-03-03T07:58:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.633+0000 7f8cb759e700 1 -- 172.21.15.98:0/3787807691 shutdown_connections 2023-03-03T07:58:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.633+0000 7f8cb759e700 1 --2- 172.21.15.98:0/3787807691 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c980683c0 0x7f8c9806a870 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.633+0000 7f8cb759e700 1 --2- 172.21.15.98:0/3787807691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80cf570 0x7f8cb81485a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.633+0000 7f8cb759e700 1 --2- 172.21.15.98:0/3787807691 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb81609b0 0x7f8cb815c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.633+0000 7f8cb759e700 1 --2- 172.21.15.98:0/3787807691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb815db30 0x7f8cb815bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.633+0000 7f8cb759e700 1 -- 172.21.15.98:0/3787807691 >> 172.21.15.98:0/3787807691 conn(0x7f8cb806b880 msgr2=0x7f8cb80d0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.633+0000 7f8cb759e700 1 -- 172.21.15.98:0/3787807691 shutdown_connections 2023-03-03T07:58:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.633+0000 7f8cb759e700 1 -- 172.21.15.98:0/3787807691 wait complete. 2023-03-03T07:58:07.637 INFO:teuthology.orchestra.run.smithi098.stderr: removed all blocklist entries 2023-03-03T07:58:07.666 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-03-03T07:58:07.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.911+0000 7f3ba0d9d700 1 Processor -- start 2023-03-03T07:58:07.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.912+0000 7f3ba0d9d700 1 -- start start 2023-03-03T07:58:07.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.913+0000 7f3ba0d9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3b9c0b51c0 0x7f3b9c160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:07.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.913+0000 7f3ba0d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3b9c0d1580 0x7f3b9c0d1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:07.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.913+0000 7f3ba0d9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9c1614c0 con 0x7f3b9c146360 2023-03-03T07:58:07.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.913+0000 7f3ba0d9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9c0d10f0 con 0x7f3b9c0d1580 2023-03-03T07:58:07.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.913+0000 7f3ba0d9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9c0cdf20 con 0x7f3b9c0b51c0 2023-03-03T07:58:07.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.913+0000 7f3b9b7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3b9c146360 0x7f3b9c0b4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57342/0 (socket says 172.21.15.98:57342) 2023-03-03T07:58:07.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.913+0000 7f3b9b7fe700 1 -- 172.21.15.98:0/3358324240 learned_addr learned my addr 172.21.15.98:0/3358324240 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:07.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.913+0000 7f3b9bfff700 1 --2- 172.21.15.98:0/3358324240 >> v2:172.21.15.98:3300/0 conn(0x7f3b9c0d1580 0x7f3b9c0d1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:07.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.914+0000 7f3b93fff700 1 --2- 172.21.15.98:0/3358324240 >> v2:172.21.15.100:3300/0 conn(0x7f3b9c0b51c0 0x7f3b9c160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:07.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.914+0000 7f3b9bfff700 1 -- 172.21.15.98:0/3358324240 >> v2:172.21.15.100:3300/0 conn(0x7f3b9c0b51c0 msgr2=0x7f3b9c160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:07.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.914+0000 7f3b9bfff700 1 --2- 172.21.15.98:0/3358324240 >> v2:172.21.15.100:3300/0 conn(0x7f3b9c0b51c0 0x7f3b9c160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:07.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.914+0000 7f3b9bfff700 1 -- 172.21.15.98:0/3358324240 >> v1:172.21.15.98:6789/0 conn(0x7f3b9c146360 legacy=0x7f3b9c0b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:07.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.914+0000 7f3b9bfff700 1 -- 172.21.15.98:0/3358324240 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c0cd680 con 0x7f3b9c0d1580 2023-03-03T07:58:07.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.914+0000 7f3b9bfff700 1 --2- 172.21.15.98:0/3358324240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c0d1580 0x7f3b9c0d1980 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c06bd20 tx=0x7f3b9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abad7921e524271e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:07.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.915+0000 7f3b9affd700 1 -- 172.21.15.98:0/3358324240 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9400e070 con 0x7f3b9c0d1580 2023-03-03T07:58:07.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.915+0000 7f3b9affd700 1 -- 172.21.15.98:0/3358324240 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9400bb20 con 0x7f3b9c0d1580 2023-03-03T07:58:07.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.915+0000 7f3b9affd700 1 -- 172.21.15.98:0/3358324240 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94012430 con 0x7f3b9c0d1580 2023-03-03T07:58:07.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.915+0000 7f3ba0d9d700 1 -- 172.21.15.98:0/3358324240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c0d1580 msgr2=0x7f3b9c0d1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:07.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.915+0000 7f3ba0d9d700 1 --2- 172.21.15.98:0/3358324240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c0d1580 0x7f3b9c0d1980 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c06bd20 tx=0x7f3b9400b330 comp rx=0 tx=0).stop 2023-03-03T07:58:07.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.915+0000 7f3ba0d9d700 1 -- 172.21.15.98:0/3358324240 shutdown_connections 2023-03-03T07:58:07.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.915+0000 7f3ba0d9d700 1 --2- 172.21.15.98:0/3358324240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c0d1580 0x7f3b9c0d1980 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.915+0000 7f3ba0d9d700 1 --2- 172.21.15.98:0/3358324240 >> v2:172.21.15.100:3300/0 conn(0x7f3b9c0b51c0 0x7f3b9c160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.915+0000 7f3ba0d9d700 1 -- 172.21.15.98:0/3358324240 >> 172.21.15.98:0/3358324240 conn(0x7f3b9c06b8e0 msgr2=0x7f3b9c0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.915+0000 7f3ba0d9d700 1 -- 172.21.15.98:0/3358324240 shutdown_connections 2023-03-03T07:58:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.915+0000 7f3ba0d9d700 1 -- 172.21.15.98:0/3358324240 wait complete. 2023-03-03T07:58:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.917+0000 7f3ba0d9d700 1 Processor -- start 2023-03-03T07:58:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.917+0000 7f3ba0d9d700 1 -- start start 2023-03-03T07:58:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.918+0000 7f3ba0d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b9c0b51c0 0x7f3b9c1f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.918+0000 7f3ba0d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b9c0d1580 0x7f3b9c14d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.918+0000 7f3ba0d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c146360 0x7f3b9c0a99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.918+0000 7f3ba0d9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c15eb40 con 0x7f3b9c146360 2023-03-03T07:58:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.918+0000 7f3ba0d9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0a9f20 con 0x7f3b9c0d1580 2023-03-03T07:58:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.918+0000 7f3ba0d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0aa180 con 0x7f3b9c0b51c0 2023-03-03T07:58:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.919+0000 7f3b9bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c146360 0x7f3b9c0a99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.919+0000 7f3b9bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c146360 0x7f3b9c0a99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54336/0 (socket says 172.21.15.98:54336) 2023-03-03T07:58:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.919+0000 7f3b9bfff700 1 -- 172.21.15.98:0/2959594985 learned_addr learned my addr 172.21.15.98:0/2959594985 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.919+0000 7f3b93fff700 1 --2- 172.21.15.98:0/2959594985 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b9c0d1580 0x7f3b9c14d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.919+0000 7f3b9b7fe700 1 --2- 172.21.15.98:0/2959594985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b9c0b51c0 0x7f3b9c1f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.919+0000 7f3b9bfff700 1 -- 172.21.15.98:0/2959594985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b9c0b51c0 msgr2=0x7f3b9c1f1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.919+0000 7f3b9bfff700 1 --2- 172.21.15.98:0/2959594985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b9c0b51c0 0x7f3b9c1f1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.919+0000 7f3b9bfff700 1 -- 172.21.15.98:0/2959594985 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b9c0d1580 msgr2=0x7f3b9c14d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.919+0000 7f3b9bfff700 1 --2- 172.21.15.98:0/2959594985 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b9c0d1580 0x7f3b9c14d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.919+0000 7f3b9bfff700 1 -- 172.21.15.98:0/2959594985 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b94009a70 con 0x7f3b9c146360 2023-03-03T07:58:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.920+0000 7f3b9bfff700 1 --2- 172.21.15.98:0/2959594985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c146360 0x7f3b9c0a99e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3b94014040 tx=0x7f3b94004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.920+0000 7f3b98ff9700 1 -- 172.21.15.98:0/2959594985 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94012430 con 0x7f3b9c146360 2023-03-03T07:58:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.920+0000 7f3b98ff9700 1 -- 172.21.15.98:0/2959594985 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b94020070 con 0x7f3b9c146360 2023-03-03T07:58:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.920+0000 7f3b98ff9700 1 -- 172.21.15.98:0/2959594985 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b940098f0 con 0x7f3b9c146360 2023-03-03T07:58:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.920+0000 7f3ba0d9d700 1 -- 172.21.15.98:0/2959594985 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9c0cdc40 con 0x7f3b9c146360 2023-03-03T07:58:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.920+0000 7f3ba0d9d700 1 -- 172.21.15.98:0/2959594985 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9c0cd960 con 0x7f3b9c146360 2023-03-03T07:58:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.921+0000 7f3ba0d9d700 1 -- 172.21.15.98:0/2959594985 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9c0d0d80 con 0x7f3b9c146360 2023-03-03T07:58:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.922+0000 7f3b98ff9700 1 -- 172.21.15.98:0/2959594985 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f3b9402e070 con 0x7f3b9c146360 2023-03-03T07:58:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.923+0000 7f3b98ff9700 1 --2- 172.21.15.98:0/2959594985 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b7c0683c0 0x7f3b7c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.924+0000 7f3b98ff9700 1 -- 172.21.15.98:0/2959594985 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3b94022070 con 0x7f3b9c146360 2023-03-03T07:58:07.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.924+0000 7f3b9b7fe700 1 --2- 172.21.15.98:0/2959594985 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b7c0683c0 0x7f3b7c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:07.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.925+0000 7f3b9b7fe700 1 --2- 172.21.15.98:0/2959594985 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b7c0683c0 0x7f3b7c06a870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c005910 tx=0x7f3b8c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:07.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:07.928+0000 7f3b98ff9700 1 -- 172.21.15.98:0/2959594985 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3b940540b0 con 0x7f3b9c146360 2023-03-03T07:58:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.198+0000 7f3ba0d9d700 1 -- 172.21.15.98:0/2959594985 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3b9c15ec80 con 0x7f3b9c146360 2023-03-03T07:58:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.199+0000 7f3b98ff9700 1 -- 172.21.15.98:0/2959594985 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v9) v1 ==== 80+0+5298 (secure 0 0 0) 0x7f3b9c15ec80 con 0x7f3b9c146360 2023-03-03T07:58:08.202 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:08.203 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T07:58:08.203 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_dump": [ 2023-03-03T07:58:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.a", 2023-03-03T07:58:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkqFdgHxAAuwiGkaNnr5NplmhniDYAEQ==", 2023-03-03T07:58:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:58:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T07:58:08.206 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T07:58:08.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:58:08.206 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.207 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.b", 2023-03-03T07:58:08.207 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFk0x8kBhAA5JDo/wmI4650J9tUnvXVzw==", 2023-03-03T07:58:08.208 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.208 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:58:08.208 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T07:58:08.208 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T07:58:08.209 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:58:08.209 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.210 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.c", 2023-03-03T07:58:08.210 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkKoRwMBAAfkWGDUsqNgqfkscl0aBX1A==", 2023-03-03T07:58:08.210 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.210 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:58:08.210 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T07:58:08.211 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T07:58:08.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:58:08.211 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.211 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.d", 2023-03-03T07:58:08.211 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFkK9SKFxAAFSCozGfyRwrv46QPitRxTQ==", 2023-03-03T07:58:08.212 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.212 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:58:08.212 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T07:58:08.212 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T07:58:08.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:58:08.212 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.213 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.0", 2023-03-03T07:58:08.213 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDBpwFkTFzTERAAGHrFKuBlXIEfaYD7vs3ZHA==", 2023-03-03T07:58:08.213 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.213 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:58:08.214 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:58:08.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:58:08.214 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.214 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.1", 2023-03-03T07:58:08.214 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDEpwFkVWObLBAAW4acTvNbmscIe18NXVgRYg==", 2023-03-03T07:58:08.215 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.215 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:58:08.215 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:58:08.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:58:08.215 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.216 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.2", 2023-03-03T07:58:08.216 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDIpwFkK8lgChAAHVD5btBivUpcapSgDp2E0w==", 2023-03-03T07:58:08.216 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.216 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:58:08.217 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:58:08.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:58:08.217 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.217 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.3", 2023-03-03T07:58:08.217 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDLpwFksrb8IxAAN+CP1O3DJ1zDy+qyGNor2Q==", 2023-03-03T07:58:08.218 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.218 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:58:08.218 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:58:08.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:58:08.218 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.219 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.4", 2023-03-03T07:58:08.219 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDSpwFkkWGHIxAA4wejA0w05JPIUghEAVG/Rg==", 2023-03-03T07:58:08.219 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.219 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:58:08.219 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:58:08.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:58:08.220 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.220 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.5", 2023-03-03T07:58:08.220 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDWpwFknHmGAhAA6VM16luaykZfJEzr+yDZZA==", 2023-03-03T07:58:08.221 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.221 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:58:08.221 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:58:08.221 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:58:08.221 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.222 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.6", 2023-03-03T07:58:08.222 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDZpwFkcitNDRAAOz04YGTzx3Phenp0n8JeEA==", 2023-03-03T07:58:08.222 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.222 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:58:08.222 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:58:08.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:58:08.223 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.223 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.7", 2023-03-03T07:58:08.223 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDcpwFkrjmIFxAAbWeB2+8VQgyTplW+fk47OA==", 2023-03-03T07:58:08.223 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.224 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T07:58:08.224 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T07:58:08.224 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:58:08.224 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.224 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.0", 2023-03-03T07:58:08.224 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFk02lsJhAA0cH/evMnWdZN6DdBGCLjjg==", 2023-03-03T07:58:08.225 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.225 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:58:08.225 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow r", 2023-03-03T07:58:08.225 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rw pool=cephfs_data" 2023-03-03T07:58:08.225 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.226 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.1", 2023-03-03T07:58:08.226 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC9pwFkRqXrABAArm/01O5MtSTQHWQWKqNQEg==", 2023-03-03T07:58:08.226 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.226 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:58:08.226 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T07:58:08.227 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T07:58:08.227 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T07:58:08.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.228 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.2", 2023-03-03T07:58:08.228 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFkyKp3MRAA8Eh41TZNsO6/8VhAWCpNMQ==", 2023-03-03T07:58:08.229 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.229 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:58:08.229 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T07:58:08.230 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T07:58:08.230 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T07:58:08.230 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.230 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.3", 2023-03-03T07:58:08.231 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC9pwFk14bSCxAAy2nXc9G50W4lH5GtH84hpQ==", 2023-03-03T07:58:08.231 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.231 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T07:58:08.231 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T07:58:08.231 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T07:58:08.231 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T07:58:08.231 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.232 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.admin", 2023-03-03T07:58:08.232 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC6pwFkokmiEBAAFwi8rhPmk4Zyk95p760HDg==", 2023-03-03T07:58:08.232 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.232 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T07:58:08.233 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T07:58:08.233 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T07:58:08.233 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:58:08.234 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.235 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mgr.x", 2023-03-03T07:58:08.235 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkvdMQFBAAMi+x7psn5CDpd//d6jN2VA==", 2023-03-03T07:58:08.236 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.236 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T07:58:08.237 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile mgr", 2023-03-03T07:58:08.237 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:58:08.237 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:08.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:08.239 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mgr.y", 2023-03-03T07:58:08.239 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkfYO5BxAAFW/elbdoqVXBkUIKrAvYsg==", 2023-03-03T07:58:08.240 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T07:58:08.240 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T07:58:08.240 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile mgr", 2023-03-03T07:58:08.241 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T07:58:08.241 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.241 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:08.242 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T07:58:08.242 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T07:58:08.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.201+0000 7f3ba0d9d700 1 -- 172.21.15.98:0/2959594985 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b7c0683c0 msgr2=0x7f3b7c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:08.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.201+0000 7f3ba0d9d700 1 --2- 172.21.15.98:0/2959594985 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b7c0683c0 0x7f3b7c06a870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c005910 tx=0x7f3b8c009040 comp rx=0 tx=0).stop 2023-03-03T07:58:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.201+0000 7f3ba0d9d700 1 -- 172.21.15.98:0/2959594985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c146360 msgr2=0x7f3b9c0a99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.201+0000 7f3ba0d9d700 1 --2- 172.21.15.98:0/2959594985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c146360 0x7f3b9c0a99e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3b94014040 tx=0x7f3b94004ad0 comp rx=0 tx=0).stop 2023-03-03T07:58:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.202+0000 7f3ba0d9d700 1 -- 172.21.15.98:0/2959594985 shutdown_connections 2023-03-03T07:58:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.202+0000 7f3ba0d9d700 1 --2- 172.21.15.98:0/2959594985 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b7c0683c0 0x7f3b7c06a870 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.202+0000 7f3ba0d9d700 1 --2- 172.21.15.98:0/2959594985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c146360 0x7f3b9c0a99e0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.202+0000 7f3ba0d9d700 1 --2- 172.21.15.98:0/2959594985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b9c0b51c0 0x7f3b9c1f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.202+0000 7f3ba0d9d700 1 --2- 172.21.15.98:0/2959594985 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b9c0d1580 0x7f3b9c14d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.202+0000 7f3ba0d9d700 1 -- 172.21.15.98:0/2959594985 >> 172.21.15.98:0/2959594985 conn(0x7f3b9c06b8e0 msgr2=0x7f3b9c0b3590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.202+0000 7f3ba0d9d700 1 -- 172.21.15.98:0/2959594985 shutdown_connections 2023-03-03T07:58:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.202+0000 7f3ba0d9d700 1 -- 172.21.15.98:0/2959594985 wait complete. 2023-03-03T07:58:08.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:58:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.495+0000 7efc569ff700 1 Processor -- start 2023-03-03T07:58:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.495+0000 7efc569ff700 1 -- start start 2023-03-03T07:58:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.496+0000 7efc569ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc500b87c0 0x7efc500cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.496+0000 7efc569ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc500d0ef0 con 0x7efc500b5410 2023-03-03T07:58:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.496+0000 7efc569ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc500d0600 con 0x7efc500b8b90 2023-03-03T07:58:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.496+0000 7efc569ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc500cd970 con 0x7efc500b87c0 2023-03-03T07:58:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.497+0000 7efc4ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc500b87c0 0x7efc500cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.497+0000 7efc54f9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efc500b5410 0x7efc500d13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57348/0 (socket says 172.21.15.98:57348) 2023-03-03T07:58:08.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.497+0000 7efc54f9c700 1 -- 172.21.15.98:0/896538771 learned_addr learned my addr 172.21.15.98:0/896538771 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.497+0000 7efc4ffff700 1 -- 172.21.15.98:0/896538771 >> v1:172.21.15.100:6789/0 conn(0x7efc500b8b90 legacy=0x7efc500b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.497+0000 7efc4ffff700 1 -- 172.21.15.98:0/896538771 >> v1:172.21.15.98:6789/0 conn(0x7efc500b5410 legacy=0x7efc500d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.497+0000 7efc4ffff700 1 -- 172.21.15.98:0/896538771 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc500cdc50 con 0x7efc500b87c0 2023-03-03T07:58:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.497+0000 7efc4f7fe700 1 -- 172.21.15.98:0/896538771 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428319464 0 0) 0x7efc500d0ef0 con 0x7efc500b5410 2023-03-03T07:58:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.498+0000 7efc4ffff700 1 --2- 172.21.15.98:0/896538771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc500b87c0 0x7efc500cffe0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7efc40009840 tx=0x7efc40009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b7840eab9fbc53b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.498+0000 7efc4f7fe700 1 -- 172.21.15.98:0/896538771 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc40011830 con 0x7efc500b87c0 2023-03-03T07:58:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.498+0000 7efc4f7fe700 1 -- 172.21.15.98:0/896538771 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc40011990 con 0x7efc500b87c0 2023-03-03T07:58:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.498+0000 7efc4f7fe700 1 -- 172.21.15.98:0/896538771 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc40011d20 con 0x7efc500b87c0 2023-03-03T07:58:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.498+0000 7efc569ff700 1 -- 172.21.15.98:0/896538771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc500b87c0 msgr2=0x7efc500cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.498+0000 7efc569ff700 1 --2- 172.21.15.98:0/896538771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc500b87c0 0x7efc500cffe0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7efc40009840 tx=0x7efc40009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.499+0000 7efc569ff700 1 -- 172.21.15.98:0/896538771 shutdown_connections 2023-03-03T07:58:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.499+0000 7efc569ff700 1 --2- 172.21.15.98:0/896538771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc500b87c0 0x7efc500cffe0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.499+0000 7efc569ff700 1 -- 172.21.15.98:0/896538771 >> 172.21.15.98:0/896538771 conn(0x7efc5006aee0 msgr2=0x7efc5006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.499+0000 7efc569ff700 1 -- 172.21.15.98:0/896538771 shutdown_connections 2023-03-03T07:58:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.499+0000 7efc569ff700 1 -- 172.21.15.98:0/896538771 wait complete. 2023-03-03T07:58:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.501+0000 7efc569ff700 1 Processor -- start 2023-03-03T07:58:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.501+0000 7efc569ff700 1 -- start start 2023-03-03T07:58:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.502+0000 7efc569ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc500b5410 0x7efc5015f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.502+0000 7efc569ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc500b87c0 0x7efc500cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.502+0000 7efc569ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc500b8b90 0x7efc5013e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.502+0000 7efc569ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc500c7590 con 0x7efc500b87c0 2023-03-03T07:58:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.502+0000 7efc569ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc50152b90 con 0x7efc500b8b90 2023-03-03T07:58:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.502+0000 7efc569ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5015ec40 con 0x7efc500b5410 2023-03-03T07:58:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.502+0000 7efc47fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc500b87c0 0x7efc500cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.502+0000 7efc47fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc500b87c0 0x7efc500cc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54354/0 (socket says 172.21.15.98:54354) 2023-03-03T07:58:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.502+0000 7efc47fff700 1 -- 172.21.15.98:0/785422585 learned_addr learned my addr 172.21.15.98:0/785422585 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.503+0000 7efc4ffff700 1 --2- 172.21.15.98:0/785422585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc500b5410 0x7efc5015f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.503+0000 7efc54f9c700 1 --2- 172.21.15.98:0/785422585 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc500b8b90 0x7efc5013e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.503+0000 7efc47fff700 1 -- 172.21.15.98:0/785422585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc500b5410 msgr2=0x7efc5015f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.503+0000 7efc47fff700 1 --2- 172.21.15.98:0/785422585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc500b5410 0x7efc5015f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.503+0000 7efc47fff700 1 -- 172.21.15.98:0/785422585 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc500b8b90 msgr2=0x7efc5013e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.503+0000 7efc47fff700 1 --2- 172.21.15.98:0/785422585 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc500b8b90 0x7efc5013e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.503+0000 7efc47fff700 1 -- 172.21.15.98:0/785422585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc400094a0 con 0x7efc500b87c0 2023-03-03T07:58:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.503+0000 7efc47fff700 1 --2- 172.21.15.98:0/785422585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc500b87c0 0x7efc500cc9b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7efc3800c4b0 tx=0x7efc3800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.503+0000 7efc4d7fa700 1 -- 172.21.15.98:0/785422585 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc380052d0 con 0x7efc500b87c0 2023-03-03T07:58:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.503+0000 7efc4d7fa700 1 -- 172.21.15.98:0/785422585 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc38005430 con 0x7efc500b87c0 2023-03-03T07:58:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.503+0000 7efc4d7fa700 1 -- 172.21.15.98:0/785422585 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc380038c0 con 0x7efc500b87c0 2023-03-03T07:58:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.503+0000 7efc569ff700 1 -- 172.21.15.98:0/785422585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc500cdf30 con 0x7efc500b87c0 2023-03-03T07:58:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.504+0000 7efc569ff700 1 -- 172.21.15.98:0/785422585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc50073a80 con 0x7efc500b87c0 2023-03-03T07:58:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.505+0000 7efc569ff700 1 -- 172.21.15.98:0/785422585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc500ce0c0 con 0x7efc500b87c0 2023-03-03T07:58:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.506+0000 7efc4d7fa700 1 -- 172.21.15.98:0/785422585 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7efc38019030 con 0x7efc500b87c0 2023-03-03T07:58:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.507+0000 7efc4d7fa700 1 --2- 172.21.15.98:0/785422585 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc300683c0 0x7efc3006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.507+0000 7efc4d7fa700 1 -- 172.21.15.98:0/785422585 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efc38084e90 con 0x7efc500b87c0 2023-03-03T07:58:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.508+0000 7efc4ffff700 1 --2- 172.21.15.98:0/785422585 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc300683c0 0x7efc3006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.508+0000 7efc4ffff700 1 --2- 172.21.15.98:0/785422585 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc300683c0 0x7efc3006a870 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efc40000c00 tx=0x7efc4001c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:08.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.511+0000 7efc4d7fa700 1 -- 172.21.15.98:0/785422585 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efc38051880 con 0x7efc500b87c0 2023-03-03T07:58:08.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.709+0000 7efc569ff700 1 -- 172.21.15.98:0/785422585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc500729b0 con 0x7efc500b87c0 2023-03-03T07:58:08.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.710+0000 7efc4d7fa700 1 -- 172.21.15.98:0/785422585 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+14267 (secure 0 0 0) 0x7efc38054ea0 con 0x7efc500b87c0 2023-03-03T07:58:08.713 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:08.713 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":48,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:58:07.621839+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:58:08.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.713+0000 7efc569ff700 1 -- 172.21.15.98:0/785422585 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc300683c0 msgr2=0x7efc3006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:08.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.713+0000 7efc569ff700 1 --2- 172.21.15.98:0/785422585 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc300683c0 0x7efc3006a870 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efc40000c00 tx=0x7efc4001c040 comp rx=0 tx=0).stop 2023-03-03T07:58:08.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.713+0000 7efc569ff700 1 -- 172.21.15.98:0/785422585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc500b87c0 msgr2=0x7efc500cc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:08.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.713+0000 7efc569ff700 1 --2- 172.21.15.98:0/785422585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc500b87c0 0x7efc500cc9b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7efc3800c4b0 tx=0x7efc3800c870 comp rx=0 tx=0).stop 2023-03-03T07:58:08.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.713+0000 7efc569ff700 1 -- 172.21.15.98:0/785422585 shutdown_connections 2023-03-03T07:58:08.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.713+0000 7efc569ff700 1 --2- 172.21.15.98:0/785422585 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc300683c0 0x7efc3006a870 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:08.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.713+0000 7efc569ff700 1 --2- 172.21.15.98:0/785422585 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc500b8b90 0x7efc5013e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:08.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.713+0000 7efc569ff700 1 --2- 172.21.15.98:0/785422585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc500b5410 0x7efc5015f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:08.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.713+0000 7efc569ff700 1 --2- 172.21.15.98:0/785422585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc500b87c0 0x7efc500cc9b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:08.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.713+0000 7efc569ff700 1 -- 172.21.15.98:0/785422585 >> 172.21.15.98:0/785422585 conn(0x7efc5006aee0 msgr2=0x7efc500b1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:08.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.713+0000 7efc569ff700 1 -- 172.21.15.98:0/785422585 shutdown_connections 2023-03-03T07:58:08.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.713+0000 7efc569ff700 1 -- 172.21.15.98:0/785422585 wait complete. 2023-03-03T07:58:08.742 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:58:08.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:58:08.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.990+0000 7fe65dfff700 1 Processor -- start 2023-03-03T07:58:08.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.990+0000 7fe65dfff700 1 -- start start 2023-03-03T07:58:08.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.991+0000 7fe65dfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe65815ca70 0x7fe6580cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:08.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.991+0000 7fe65dfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6580d0270 0x7fe6580d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:08.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.991+0000 7fe65dfff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6580cf980 con 0x7fe65806b4b0 2023-03-03T07:58:08.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.991+0000 7fe65dfff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6580cd5b0 con 0x7fe65815ca70 2023-03-03T07:58:08.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.991+0000 7fe65dfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6580d1800 con 0x7fe6580d0270 2023-03-03T07:58:08.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.991+0000 7fe6577fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe65815ca70 0x7fe6580cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:08.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.992+0000 7fe64ffff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe65806b4b0 0x7fe6580cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57360/0 (socket says 172.21.15.98:57360) 2023-03-03T07:58:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.992+0000 7fe64ffff700 1 -- 172.21.15.98:0/2042291424 learned_addr learned my addr 172.21.15.98:0/2042291424 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.992+0000 7fe6577fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe65815ca70 0x7fe6580cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54366/0 (socket says 172.21.15.98:54366) 2023-03-03T07:58:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.992+0000 7fe657fff700 1 --2- 172.21.15.98:0/2042291424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6580d0270 0x7fe6580d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.992+0000 7fe6577fe700 1 -- 172.21.15.98:0/2042291424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6580d0270 msgr2=0x7fe6580d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.992+0000 7fe6577fe700 1 --2- 172.21.15.98:0/2042291424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6580d0270 0x7fe6580d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.992+0000 7fe6577fe700 1 -- 172.21.15.98:0/2042291424 >> v1:172.21.15.98:6789/0 conn(0x7fe65806b4b0 legacy=0x7fe6580cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:08.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.992+0000 7fe6577fe700 1 -- 172.21.15.98:0/2042291424 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6580cd900 con 0x7fe65815ca70 2023-03-03T07:58:08.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.992+0000 7fe656ffd700 1 -- 172.21.15.98:0/2042291424 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535910500 0 0) 0x7fe6580cf980 con 0x7fe65806b4b0 2023-03-03T07:58:08.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.992+0000 7fe6577fe700 1 --2- 172.21.15.98:0/2042291424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65815ca70 0x7fe6580cf440 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe648009840 tx=0x7fe648009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f94aaf3d476013b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:08.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.993+0000 7fe656ffd700 1 -- 172.21.15.98:0/2042291424 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe648011830 con 0x7fe65815ca70 2023-03-03T07:58:08.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.993+0000 7fe656ffd700 1 -- 172.21.15.98:0/2042291424 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe648011990 con 0x7fe65815ca70 2023-03-03T07:58:08.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.993+0000 7fe656ffd700 1 -- 172.21.15.98:0/2042291424 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe648011d20 con 0x7fe65815ca70 2023-03-03T07:58:08.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.993+0000 7fe65dfff700 1 -- 172.21.15.98:0/2042291424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65815ca70 msgr2=0x7fe6580cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:08.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.993+0000 7fe65dfff700 1 --2- 172.21.15.98:0/2042291424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65815ca70 0x7fe6580cf440 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe648009840 tx=0x7fe648009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:08.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.993+0000 7fe65dfff700 1 -- 172.21.15.98:0/2042291424 shutdown_connections 2023-03-03T07:58:08.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.993+0000 7fe65dfff700 1 --2- 172.21.15.98:0/2042291424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65815ca70 0x7fe6580cf440 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:08.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.993+0000 7fe65dfff700 1 --2- 172.21.15.98:0/2042291424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6580d0270 0x7fe6580d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:08.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.993+0000 7fe65dfff700 1 -- 172.21.15.98:0/2042291424 >> 172.21.15.98:0/2042291424 conn(0x7fe658073e70 msgr2=0x7fe65806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:08.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.993+0000 7fe65dfff700 1 -- 172.21.15.98:0/2042291424 shutdown_connections 2023-03-03T07:58:08.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.994+0000 7fe65dfff700 1 -- 172.21.15.98:0/2042291424 wait complete. 2023-03-03T07:58:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.996+0000 7fe65dfff700 1 Processor -- start 2023-03-03T07:58:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.997+0000 7fe65dfff700 1 -- start start 2023-03-03T07:58:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.998+0000 7fe65dfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe65806b4b0 0x7fe65814b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.998+0000 7fe65dfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6580d0270 0x7fe6580caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.998+0000 7fe65dfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65815ca70 0x7fe65815f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.998+0000 7fe65dfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6580caab0 con 0x7fe65815ca70 2023-03-03T07:58:09.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.998+0000 7fe65dfff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6581f3440 con 0x7fe65806b4b0 2023-03-03T07:58:09.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.998+0000 7fe65dfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6581f3160 con 0x7fe6580d0270 2023-03-03T07:58:09.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.998+0000 7fe657fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65815ca70 0x7fe65815f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.998+0000 7fe64ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6580d0270 0x7fe6580caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.998+0000 7fe6577fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe65806b4b0 0x7fe65814b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.998+0000 7fe657fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65815ca70 0x7fe65815f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54368/0 (socket says 172.21.15.98:54368) 2023-03-03T07:58:09.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.998+0000 7fe657fff700 1 -- 172.21.15.98:0/2439572944 learned_addr learned my addr 172.21.15.98:0/2439572944 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:09.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.999+0000 7fe657fff700 1 -- 172.21.15.98:0/2439572944 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6580d0270 msgr2=0x7fe6580caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:09.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.999+0000 7fe657fff700 1 --2- 172.21.15.98:0/2439572944 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6580d0270 0x7fe6580caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.999+0000 7fe657fff700 1 -- 172.21.15.98:0/2439572944 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe65806b4b0 msgr2=0x7fe65814b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:09.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.999+0000 7fe657fff700 1 --2- 172.21.15.98:0/2439572944 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe65806b4b0 0x7fe65814b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.999+0000 7fe657fff700 1 -- 172.21.15.98:0/2439572944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6480094a0 con 0x7fe65815ca70 2023-03-03T07:58:09.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.999+0000 7fe657fff700 1 --2- 172.21.15.98:0/2439572944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65815ca70 0x7fe65815f3e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe650007e60 tx=0x7fe65000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.999+0000 7fe654ff9700 1 -- 172.21.15.98:0/2439572944 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65000ed30 con 0x7fe65815ca70 2023-03-03T07:58:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.999+0000 7fe654ff9700 1 -- 172.21.15.98:0/2439572944 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe650014420 con 0x7fe65815ca70 2023-03-03T07:58:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.999+0000 7fe654ff9700 1 -- 172.21.15.98:0/2439572944 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6500147d0 con 0x7fe65815ca70 2023-03-03T07:58:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.999+0000 7fe65dfff700 1 -- 172.21.15.98:0/2439572944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6580d1aa0 con 0x7fe65815ca70 2023-03-03T07:58:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:08.999+0000 7fe65dfff700 1 -- 172.21.15.98:0/2439572944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6580cdbe0 con 0x7fe65815ca70 2023-03-03T07:58:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.001+0000 7fe65dfff700 1 -- 172.21.15.98:0/2439572944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65815ce40 con 0x7fe65815ca70 2023-03-03T07:58:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.002+0000 7fe654ff9700 1 -- 172.21.15.98:0/2439572944 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fe650012050 con 0x7fe65815ca70 2023-03-03T07:58:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.003+0000 7fe654ff9700 1 --2- 172.21.15.98:0/2439572944 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe638068390 0x7fe63806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.003+0000 7fe654ff9700 1 -- 172.21.15.98:0/2439572944 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe650085ff0 con 0x7fe65815ca70 2023-03-03T07:58:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.003+0000 7fe6577fe700 1 --2- 172.21.15.98:0/2439572944 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe638068390 0x7fe63806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.004+0000 7fe6577fe700 1 --2- 172.21.15.98:0/2439572944 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe638068390 0x7fe63806a840 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe6480090b0 tx=0x7fe648008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.007+0000 7fe654ff9700 1 -- 172.21.15.98:0/2439572944 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe650052a20 con 0x7fe65815ca70 2023-03-03T07:58:09.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.197+0000 7fe65dfff700 1 -- 172.21.15.98:0/2439572944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe6580ca250 con 0x7fe65815ca70 2023-03-03T07:58:09.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.197+0000 7fe654ff9700 1 -- 172.21.15.98:0/2439572944 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe6580ca250 con 0x7fe65815ca70 2023-03-03T07:58:09.200 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:58:09.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.201+0000 7fe65dfff700 1 -- 172.21.15.98:0/2439572944 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe638068390 msgr2=0x7fe63806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.201+0000 7fe65dfff700 1 --2- 172.21.15.98:0/2439572944 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe638068390 0x7fe63806a840 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe6480090b0 tx=0x7fe648008030 comp rx=0 tx=0).stop 2023-03-03T07:58:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.201+0000 7fe65dfff700 1 -- 172.21.15.98:0/2439572944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65815ca70 msgr2=0x7fe65815f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.201+0000 7fe65dfff700 1 --2- 172.21.15.98:0/2439572944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65815ca70 0x7fe65815f3e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe650007e60 tx=0x7fe65000e3c0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.201+0000 7fe65dfff700 1 -- 172.21.15.98:0/2439572944 shutdown_connections 2023-03-03T07:58:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.201+0000 7fe65dfff700 1 --2- 172.21.15.98:0/2439572944 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe638068390 0x7fe63806a840 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.201+0000 7fe65dfff700 1 --2- 172.21.15.98:0/2439572944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65815ca70 0x7fe65815f3e0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.201+0000 7fe65dfff700 1 --2- 172.21.15.98:0/2439572944 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe65806b4b0 0x7fe65814b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.201+0000 7fe65dfff700 1 --2- 172.21.15.98:0/2439572944 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6580d0270 0x7fe6580caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.201+0000 7fe65dfff700 1 -- 172.21.15.98:0/2439572944 >> 172.21.15.98:0/2439572944 conn(0x7fe658073e70 msgr2=0x7fe6580d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:09.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.201+0000 7fe65dfff700 1 -- 172.21.15.98:0/2439572944 shutdown_connections 2023-03-03T07:58:09.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.201+0000 7fe65dfff700 1 -- 172.21.15.98:0/2439572944 wait complete. 2023-03-03T07:58:09.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:58:09.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.473+0000 7f81ed3ff700 1 Processor -- start 2023-03-03T07:58:09.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.473+0000 7f81ed3ff700 1 -- start start 2023-03-03T07:58:09.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.475+0000 7f81ed3ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f81e815ca70 0x7f81e80cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.475+0000 7f81ed3ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f81e806b4b0 0x7f81e80cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.475+0000 7f81ed3ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e80cf980 con 0x7f81e80d00a0 2023-03-03T07:58:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.475+0000 7f81ed3ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81e80cd5b0 con 0x7f81e806b4b0 2023-03-03T07:58:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.475+0000 7f81ed3ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81e80d0540 con 0x7f81e815ca70 2023-03-03T07:58:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.475+0000 7f81e67fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f81e806b4b0 0x7f81e80cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.475+0000 7f81e77fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f81e80d00a0 0x7f81e80b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57374/0 (socket says 172.21.15.98:57374) 2023-03-03T07:58:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.475+0000 7f81e77fe700 1 -- 172.21.15.98:0/4186486402 learned_addr learned my addr 172.21.15.98:0/4186486402 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.475+0000 7f81e6ffd700 1 --2- 172.21.15.98:0/4186486402 >> v2:172.21.15.100:3300/0 conn(0x7f81e815ca70 0x7f81e80cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.476+0000 7f81e67fc700 1 -- 172.21.15.98:0/4186486402 >> v2:172.21.15.100:3300/0 conn(0x7f81e815ca70 msgr2=0x7f81e80cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:09.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.476+0000 7f81e67fc700 1 --2- 172.21.15.98:0/4186486402 >> v2:172.21.15.100:3300/0 conn(0x7f81e815ca70 0x7f81e80cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.476+0000 7f81e67fc700 1 -- 172.21.15.98:0/4186486402 >> v1:172.21.15.98:6789/0 conn(0x7f81e80d00a0 legacy=0x7f81e80b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.476+0000 7f81e67fc700 1 -- 172.21.15.98:0/4186486402 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e80d2fd0 con 0x7f81e806b4b0 2023-03-03T07:58:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.476+0000 7f81e5ffb700 1 -- 172.21.15.98:0/4186486402 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480930154 0 0) 0x7f81e80cf980 con 0x7f81e80d00a0 2023-03-03T07:58:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.476+0000 7f81e67fc700 1 --2- 172.21.15.98:0/4186486402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e806b4b0 0x7f81e80cfb60 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f81e806c2c0 tx=0x7f81dc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b44341a6860bf3b9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.476+0000 7f81e5ffb700 1 -- 172.21.15.98:0/4186486402 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc011830 con 0x7f81e806b4b0 2023-03-03T07:58:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.476+0000 7f81e5ffb700 1 -- 172.21.15.98:0/4186486402 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81dc011990 con 0x7f81e806b4b0 2023-03-03T07:58:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.476+0000 7f81e5ffb700 1 -- 172.21.15.98:0/4186486402 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc011d20 con 0x7f81e806b4b0 2023-03-03T07:58:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.477+0000 7f81ed3ff700 1 -- 172.21.15.98:0/4186486402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e806b4b0 msgr2=0x7f81e80cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.477+0000 7f81ed3ff700 1 --2- 172.21.15.98:0/4186486402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e806b4b0 0x7f81e80cfb60 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f81e806c2c0 tx=0x7f81dc009b20 comp rx=0 tx=0).stop 2023-03-03T07:58:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.477+0000 7f81ed3ff700 1 -- 172.21.15.98:0/4186486402 shutdown_connections 2023-03-03T07:58:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.477+0000 7f81ed3ff700 1 --2- 172.21.15.98:0/4186486402 >> v2:172.21.15.100:3300/0 conn(0x7f81e815ca70 0x7f81e80cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.477+0000 7f81ed3ff700 1 --2- 172.21.15.98:0/4186486402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e806b4b0 0x7f81e80cfb60 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.477+0000 7f81ed3ff700 1 -- 172.21.15.98:0/4186486402 >> 172.21.15.98:0/4186486402 conn(0x7f81e8073e70 msgr2=0x7f81e806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.477+0000 7f81ed3ff700 1 -- 172.21.15.98:0/4186486402 shutdown_connections 2023-03-03T07:58:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.477+0000 7f81ed3ff700 1 -- 172.21.15.98:0/4186486402 wait complete. 2023-03-03T07:58:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.479+0000 7f81ed3ff700 1 Processor -- start 2023-03-03T07:58:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.479+0000 7f81ed3ff700 1 -- start start 2023-03-03T07:58:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.480+0000 7f81ed3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81e806b4b0 0x7f81e815edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.480+0000 7f81ed3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e80d00a0 0x7f81e8145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.480+0000 7f81ed3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81e815ca70 0x7f81e81464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.480+0000 7f81ed3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e80b4c70 con 0x7f81e80d00a0 2023-03-03T07:58:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.480+0000 7f81ed3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e80b46b0 con 0x7f81e806b4b0 2023-03-03T07:58:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.481+0000 7f81ed3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e80b4990 con 0x7f81e815ca70 2023-03-03T07:58:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.481+0000 7f81e67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e80d00a0 0x7f81e8145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.481+0000 7f81e67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e80d00a0 0x7f81e8145fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54394/0 (socket says 172.21.15.98:54394) 2023-03-03T07:58:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.481+0000 7f81e67fc700 1 -- 172.21.15.98:0/1295990964 learned_addr learned my addr 172.21.15.98:0/1295990964 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.481+0000 7f81e6ffd700 1 --2- 172.21.15.98:0/1295990964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81e806b4b0 0x7f81e815edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.481+0000 7f81e77fe700 1 --2- 172.21.15.98:0/1295990964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81e815ca70 0x7f81e81464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.481+0000 7f81e67fc700 1 -- 172.21.15.98:0/1295990964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81e815ca70 msgr2=0x7f81e81464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.481+0000 7f81e67fc700 1 --2- 172.21.15.98:0/1295990964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81e815ca70 0x7f81e81464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.481+0000 7f81e67fc700 1 -- 172.21.15.98:0/1295990964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81e806b4b0 msgr2=0x7f81e815edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.481+0000 7f81e67fc700 1 --2- 172.21.15.98:0/1295990964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81e806b4b0 0x7f81e815edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.481+0000 7f81e67fc700 1 -- 172.21.15.98:0/1295990964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc0094a0 con 0x7f81e80d00a0 2023-03-03T07:58:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.482+0000 7f81e67fc700 1 --2- 172.21.15.98:0/1295990964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e80d00a0 0x7f81e8145fb0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f81dc013000 tx=0x7f81dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.482+0000 7f81cf7fe700 1 -- 172.21.15.98:0/1295990964 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc0036a0 con 0x7f81e80d00a0 2023-03-03T07:58:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.482+0000 7f81cf7fe700 1 -- 172.21.15.98:0/1295990964 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81dc01f070 con 0x7f81e80d00a0 2023-03-03T07:58:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.482+0000 7f81cf7fe700 1 -- 172.21.15.98:0/1295990964 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc003a20 con 0x7f81e80d00a0 2023-03-03T07:58:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.482+0000 7f81ed3ff700 1 -- 172.21.15.98:0/1295990964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e80cd900 con 0x7f81e80d00a0 2023-03-03T07:58:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.482+0000 7f81ed3ff700 1 -- 172.21.15.98:0/1295990964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e80d1800 con 0x7f81e80d00a0 2023-03-03T07:58:09.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.483+0000 7f81ed3ff700 1 -- 172.21.15.98:0/1295990964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e80cdbe0 con 0x7f81e80d00a0 2023-03-03T07:58:09.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.484+0000 7f81cf7fe700 1 -- 172.21.15.98:0/1295990964 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f81dc00f050 con 0x7f81e80d00a0 2023-03-03T07:58:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.486+0000 7f81cf7fe700 1 --2- 172.21.15.98:0/1295990964 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f81d40683c0 0x7f81d406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.486+0000 7f81cf7fe700 1 -- 172.21.15.98:0/1295990964 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f81dc021070 con 0x7f81e80d00a0 2023-03-03T07:58:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.486+0000 7f81e6ffd700 1 --2- 172.21.15.98:0/1295990964 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f81d40683c0 0x7f81d406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.487+0000 7f81e6ffd700 1 --2- 172.21.15.98:0/1295990964 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f81d40683c0 0x7f81d406a870 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f81d8009b40 tx=0x7f81d8009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:09.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.491+0000 7f81cf7fe700 1 -- 172.21.15.98:0/1295990964 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f81dc053e20 con 0x7f81e80d00a0 2023-03-03T07:58:09.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.686+0000 7f81ed3ff700 1 -- 172.21.15.98:0/1295990964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f81e80ca560 con 0x7f81e80d00a0 2023-03-03T07:58:09.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.686+0000 7f81cf7fe700 1 -- 172.21.15.98:0/1295990964 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v48) v1 ==== 92+0+10 (secure 0 0 0) 0x7f81e80ca560 con 0x7f81e80d00a0 2023-03-03T07:58:09.690 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:58:09.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.690+0000 7f81ed3ff700 1 -- 172.21.15.98:0/1295990964 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f81d40683c0 msgr2=0x7f81d406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:09.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.690+0000 7f81ed3ff700 1 --2- 172.21.15.98:0/1295990964 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f81d40683c0 0x7f81d406a870 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f81d8009b40 tx=0x7f81d8009210 comp rx=0 tx=0).stop 2023-03-03T07:58:09.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.690+0000 7f81ed3ff700 1 -- 172.21.15.98:0/1295990964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e80d00a0 msgr2=0x7f81e8145fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:09.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.690+0000 7f81ed3ff700 1 --2- 172.21.15.98:0/1295990964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e80d00a0 0x7f81e8145fb0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f81dc013000 tx=0x7f81dc004750 comp rx=0 tx=0).stop 2023-03-03T07:58:09.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.690+0000 7f81ed3ff700 1 -- 172.21.15.98:0/1295990964 shutdown_connections 2023-03-03T07:58:09.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.690+0000 7f81ed3ff700 1 --2- 172.21.15.98:0/1295990964 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f81d40683c0 0x7f81d406a870 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.690+0000 7f81ed3ff700 1 --2- 172.21.15.98:0/1295990964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81e815ca70 0x7f81e81464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.690+0000 7f81ed3ff700 1 --2- 172.21.15.98:0/1295990964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81e806b4b0 0x7f81e815edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.690+0000 7f81ed3ff700 1 --2- 172.21.15.98:0/1295990964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e80d00a0 0x7f81e8145fb0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.690+0000 7f81ed3ff700 1 -- 172.21.15.98:0/1295990964 >> 172.21.15.98:0/1295990964 conn(0x7f81e8073e70 msgr2=0x7f81e80b8100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:09.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.690+0000 7f81ed3ff700 1 -- 172.21.15.98:0/1295990964 shutdown_connections 2023-03-03T07:58:09.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.690+0000 7f81ed3ff700 1 -- 172.21.15.98:0/1295990964 wait complete. 2023-03-03T07:58:09.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-03-03T07:58:09.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.966+0000 7f4c27bff700 1 Processor -- start 2023-03-03T07:58:09.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.966+0000 7f4c27bff700 1 -- start start 2023-03-03T07:58:09.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.967+0000 7f4c27bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c2015cc00 0x7f4c200cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.967+0000 7f4c27bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c200d27c0 0x7f4c200d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.967+0000 7f4c27bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c200cde10 con 0x7f4c200b60d0 2023-03-03T07:58:09.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.967+0000 7f4c27bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c200ce040 con 0x7f4c200d27c0 2023-03-03T07:58:09.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.967+0000 7f4c27bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c20073be0 con 0x7f4c2015cc00 2023-03-03T07:58:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.968+0000 7f4c2519a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c200d27c0 0x7f4c200d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.968+0000 7f4c2519a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c200d27c0 0x7f4c200d2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54396/0 (socket says 172.21.15.98:54396) 2023-03-03T07:58:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.968+0000 7f4c2519a700 1 -- 172.21.15.98:0/890548632 learned_addr learned my addr 172.21.15.98:0/890548632 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.968+0000 7f4c2599b700 1 --2- 172.21.15.98:0/890548632 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c2015cc00 0x7f4c200cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.968+0000 7f4c2519a700 1 -- 172.21.15.98:0/890548632 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c2015cc00 msgr2=0x7f4c200cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.968+0000 7f4c2519a700 1 --2- 172.21.15.98:0/890548632 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c2015cc00 0x7f4c200cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.968+0000 7f4c2519a700 1 -- 172.21.15.98:0/890548632 >> v1:172.21.15.100:6789/0 conn(0x7f4c200b60d0 legacy=0x7f4c200b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.968+0000 7f4c2519a700 1 -- 172.21.15.98:0/890548632 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c200d1a00 con 0x7f4c200d27c0 2023-03-03T07:58:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.968+0000 7f4c2519a700 1 --2- 172.21.15.98:0/890548632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c200d27c0 0x7f4c200d2c30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4c100098f0 tx=0x7f4c10009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d6b9519ce8a93f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.969+0000 7f4c24999700 1 -- 172.21.15.98:0/890548632 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1000b8a0 con 0x7f4c200d27c0 2023-03-03T07:58:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.969+0000 7f4c24999700 1 -- 172.21.15.98:0/890548632 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1000ba00 con 0x7f4c200d27c0 2023-03-03T07:58:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.969+0000 7f4c24999700 1 -- 172.21.15.98:0/890548632 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1000bd90 con 0x7f4c200d27c0 2023-03-03T07:58:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.969+0000 7f4c27bff700 1 -- 172.21.15.98:0/890548632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c200d27c0 msgr2=0x7f4c200d2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.969+0000 7f4c27bff700 1 --2- 172.21.15.98:0/890548632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c200d27c0 0x7f4c200d2c30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4c100098f0 tx=0x7f4c10009c00 comp rx=0 tx=0).stop 2023-03-03T07:58:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.969+0000 7f4c27bff700 1 -- 172.21.15.98:0/890548632 shutdown_connections 2023-03-03T07:58:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.969+0000 7f4c27bff700 1 --2- 172.21.15.98:0/890548632 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c2015cc00 0x7f4c200cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.969+0000 7f4c27bff700 1 --2- 172.21.15.98:0/890548632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c200d27c0 0x7f4c200d2c30 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.969+0000 7f4c27bff700 1 -- 172.21.15.98:0/890548632 >> 172.21.15.98:0/890548632 conn(0x7f4c2006aee0 msgr2=0x7f4c2006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.970+0000 7f4c27bff700 1 -- 172.21.15.98:0/890548632 shutdown_connections 2023-03-03T07:58:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.970+0000 7f4c27bff700 1 -- 172.21.15.98:0/890548632 wait complete. 2023-03-03T07:58:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.971+0000 7f4c27bff700 1 Processor -- start 2023-03-03T07:58:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.972+0000 7f4c27bff700 1 -- start start 2023-03-03T07:58:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.973+0000 7f4c27bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c200b60d0 0x7f4c200b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.973+0000 7f4c27bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c200d27c0 0x7f4c2014f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.973+0000 7f4c27bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c2015cc00 0x7f4c200ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.973+0000 7f4c27bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c20156d00 con 0x7f4c2015cc00 2023-03-03T07:58:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.973+0000 7f4c27bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c200ca890 con 0x7f4c200d27c0 2023-03-03T07:58:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.973+0000 7f4c27bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2015d170 con 0x7f4c200b60d0 2023-03-03T07:58:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.973+0000 7f4c2619c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c2015cc00 0x7f4c200ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.973+0000 7f4c2599b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c200b60d0 0x7f4c200b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.973+0000 7f4c2619c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c2015cc00 0x7f4c200ca350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54402/0 (socket says 172.21.15.98:54402) 2023-03-03T07:58:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.973+0000 7f4c2619c700 1 -- 172.21.15.98:0/397793321 learned_addr learned my addr 172.21.15.98:0/397793321 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.973+0000 7f4c2519a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c200d27c0 0x7f4c2014f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.974+0000 7f4c2619c700 1 -- 172.21.15.98:0/397793321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c200b60d0 msgr2=0x7f4c200b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.974+0000 7f4c2619c700 1 --2- 172.21.15.98:0/397793321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c200b60d0 0x7f4c200b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.974+0000 7f4c2619c700 1 -- 172.21.15.98:0/397793321 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c200d27c0 msgr2=0x7f4c2014f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.974+0000 7f4c2619c700 1 --2- 172.21.15.98:0/397793321 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c200d27c0 0x7f4c2014f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.974+0000 7f4c2619c700 1 -- 172.21.15.98:0/397793321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c10009550 con 0x7f4c2015cc00 2023-03-03T07:58:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.974+0000 7f4c2519a700 1 --2- 172.21.15.98:0/397793321 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c200d27c0 0x7f4c2014f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:58:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.974+0000 7f4c2619c700 1 --2- 172.21.15.98:0/397793321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c2015cc00 0x7f4c200ca350 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800cb50 tx=0x7f4c1800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.974+0000 7f4c167fc700 1 -- 172.21.15.98:0/397793321 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c18003c00 con 0x7f4c2015cc00 2023-03-03T07:58:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.974+0000 7f4c167fc700 1 -- 172.21.15.98:0/397793321 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c18003d60 con 0x7f4c2015cc00 2023-03-03T07:58:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.974+0000 7f4c167fc700 1 -- 172.21.15.98:0/397793321 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1800aa20 con 0x7f4c2015cc00 2023-03-03T07:58:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.974+0000 7f4c27bff700 1 -- 172.21.15.98:0/397793321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c200d1fc0 con 0x7f4c2015cc00 2023-03-03T07:58:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.974+0000 7f4c27bff700 1 -- 172.21.15.98:0/397793321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c200d1ce0 con 0x7f4c2015cc00 2023-03-03T07:58:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.976+0000 7f4c27bff700 1 -- 172.21.15.98:0/397793321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c200d22a0 con 0x7f4c2015cc00 2023-03-03T07:58:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.977+0000 7f4c167fc700 1 -- 172.21.15.98:0/397793321 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f4c18010050 con 0x7f4c2015cc00 2023-03-03T07:58:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.978+0000 7f4c167fc700 1 --2- 172.21.15.98:0/397793321 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4c040683c0 0x7f4c0406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.978+0000 7f4c167fc700 1 -- 172.21.15.98:0/397793321 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4c18085920 con 0x7f4c2015cc00 2023-03-03T07:58:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.978+0000 7f4c2599b700 1 --2- 172.21.15.98:0/397793321 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4c040683c0 0x7f4c0406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.979+0000 7f4c2599b700 1 --2- 172.21.15.98:0/397793321 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4c040683c0 0x7f4c0406a870 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4c200cc360 tx=0x7f4c1c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:09.982+0000 7f4c167fc700 1 -- 172.21.15.98:0/397793321 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4c18052400 con 0x7f4c2015cc00 2023-03-03T07:58:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.175+0000 7f4c27bff700 1 -- 172.21.15.98:0/397793321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4c200729b0 con 0x7f4c2015cc00 2023-03-03T07:58:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.176+0000 7f4c167fc700 1 -- 172.21.15.98:0/397793321 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v48) v1 ==== 84+0+126 (secure 0 0 0) 0x7f4c18055a20 con 0x7f4c2015cc00 2023-03-03T07:58:10.177 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:10.177 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T07:58:10.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:10.178 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2023-03-03T07:58:10.178 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2023-03-03T07:58:10.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:10.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:10.178 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 2, 2023-03-03T07:58:10.178 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": "rbd" 2023-03-03T07:58:10.179 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:10.179 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T07:58:10.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.178+0000 7f4c27bff700 1 -- 172.21.15.98:0/397793321 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4c040683c0 msgr2=0x7f4c0406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:10.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.178+0000 7f4c27bff700 1 --2- 172.21.15.98:0/397793321 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4c040683c0 0x7f4c0406a870 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4c200cc360 tx=0x7f4c1c008040 comp rx=0 tx=0).stop 2023-03-03T07:58:10.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.178+0000 7f4c27bff700 1 -- 172.21.15.98:0/397793321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c2015cc00 msgr2=0x7f4c200ca350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:10.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.178+0000 7f4c27bff700 1 --2- 172.21.15.98:0/397793321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c2015cc00 0x7f4c200ca350 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800cb50 tx=0x7f4c1800ce60 comp rx=0 tx=0).stop 2023-03-03T07:58:10.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.178+0000 7f4c27bff700 1 -- 172.21.15.98:0/397793321 shutdown_connections 2023-03-03T07:58:10.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.178+0000 7f4c27bff700 1 --2- 172.21.15.98:0/397793321 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4c040683c0 0x7f4c0406a870 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:10.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.178+0000 7f4c27bff700 1 --2- 172.21.15.98:0/397793321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c2015cc00 0x7f4c200ca350 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:10.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.178+0000 7f4c27bff700 1 --2- 172.21.15.98:0/397793321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c200b60d0 0x7f4c200b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:10.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.178+0000 7f4c27bff700 1 --2- 172.21.15.98:0/397793321 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c200d27c0 0x7f4c2014f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:10.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.178+0000 7f4c27bff700 1 -- 172.21.15.98:0/397793321 >> 172.21.15.98:0/397793321 conn(0x7f4c2006aee0 msgr2=0x7f4c200b6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:10.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.178+0000 7f4c27bff700 1 -- 172.21.15.98:0/397793321 shutdown_connections 2023-03-03T07:58:10.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.178+0000 7f4c27bff700 1 -- 172.21.15.98:0/397793321 wait complete. 2023-03-03T07:58:10.206 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-03-03T07:58:10.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-03-03T07:58:10.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.455+0000 7fb3c359e700 1 Processor -- start 2023-03-03T07:58:10.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.455+0000 7fb3c359e700 1 -- start start 2023-03-03T07:58:10.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.456+0000 7fb3c359e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb3c40b3550 0x7fb3c40cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:10.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.456+0000 7fb3c359e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3c4073ed0 0x7fb3c40cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:10.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.456+0000 7fb3c359e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3c40b3c80 con 0x7fb3c4160780 2023-03-03T07:58:10.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.456+0000 7fb3c359e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c40b4370 con 0x7fb3c4073ed0 2023-03-03T07:58:10.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.456+0000 7fb3c359e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c40b3ed0 con 0x7fb3c40b3550 2023-03-03T07:58:10.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.456+0000 7fb3bbfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3c4073ed0 0x7fb3c40cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:10.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.457+0000 7fb3c2d9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb3c4160780 0x7fb3c4160ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57386/0 (socket says 172.21.15.98:57386) 2023-03-03T07:58:10.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.457+0000 7fb3c2d9d700 1 -- 172.21.15.98:0/2487093122 learned_addr learned my addr 172.21.15.98:0/2487093122 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:10.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.457+0000 7fb3c259c700 1 --2- 172.21.15.98:0/2487093122 >> v2:172.21.15.100:3300/0 conn(0x7fb3c40b3550 0x7fb3c40cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:10.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.457+0000 7fb3bbfff700 1 -- 172.21.15.98:0/2487093122 >> v2:172.21.15.100:3300/0 conn(0x7fb3c40b3550 msgr2=0x7fb3c40cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:10.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.457+0000 7fb3bbfff700 1 --2- 172.21.15.98:0/2487093122 >> v2:172.21.15.100:3300/0 conn(0x7fb3c40b3550 0x7fb3c40cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:10.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.457+0000 7fb3bbfff700 1 -- 172.21.15.98:0/2487093122 >> v1:172.21.15.98:6789/0 conn(0x7fb3c4160780 legacy=0x7fb3c4160ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:10.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.457+0000 7fb3bbfff700 1 -- 172.21.15.98:0/2487093122 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c40cd640 con 0x7fb3c4073ed0 2023-03-03T07:58:10.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.457+0000 7fb3bbfff700 1 --2- 172.21.15.98:0/2487093122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c4073ed0 0x7fb3c40cff40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb3c406a6e0 tx=0x7fb3ac009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a205a3cdb891226 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:10.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.458+0000 7fb3c1d9b700 1 -- 172.21.15.98:0/2487093122 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac011830 con 0x7fb3c4073ed0 2023-03-03T07:58:10.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.458+0000 7fb3c1d9b700 1 -- 172.21.15.98:0/2487093122 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ac011990 con 0x7fb3c4073ed0 2023-03-03T07:58:10.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.458+0000 7fb3c1d9b700 1 -- 172.21.15.98:0/2487093122 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac011d20 con 0x7fb3c4073ed0 2023-03-03T07:58:10.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.458+0000 7fb3c359e700 1 -- 172.21.15.98:0/2487093122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c4073ed0 msgr2=0x7fb3c40cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:10.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.458+0000 7fb3c359e700 1 --2- 172.21.15.98:0/2487093122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c4073ed0 0x7fb3c40cff40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb3c406a6e0 tx=0x7fb3ac009b20 comp rx=0 tx=0).stop 2023-03-03T07:58:10.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.459+0000 7fb3c359e700 1 -- 172.21.15.98:0/2487093122 shutdown_connections 2023-03-03T07:58:10.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.459+0000 7fb3c359e700 1 --2- 172.21.15.98:0/2487093122 >> v2:172.21.15.100:3300/0 conn(0x7fb3c40b3550 0x7fb3c40cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:10.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.459+0000 7fb3c359e700 1 --2- 172.21.15.98:0/2487093122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c4073ed0 0x7fb3c40cff40 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:10.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.459+0000 7fb3c359e700 1 -- 172.21.15.98:0/2487093122 >> 172.21.15.98:0/2487093122 conn(0x7fb3c4073b00 msgr2=0x7fb3c40d2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:10.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.459+0000 7fb3c359e700 1 -- 172.21.15.98:0/2487093122 shutdown_connections 2023-03-03T07:58:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.459+0000 7fb3c359e700 1 -- 172.21.15.98:0/2487093122 wait complete. 2023-03-03T07:58:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.461+0000 7fb3c359e700 1 Processor -- start 2023-03-03T07:58:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.461+0000 7fb3c359e700 1 -- start start 2023-03-03T07:58:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.462+0000 7fb3c359e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb3c4073ed0 0x7fb3c414d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.462+0000 7fb3c359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40b3550 0x7fb3c40cc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.462+0000 7fb3c359e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb3c4160780 0x7fb3c41f2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.462+0000 7fb3c359e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c414cfa0 con 0x7fb3c40b3550 2023-03-03T07:58:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.462+0000 7fb3c359e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c415c510 con 0x7fb3c4073ed0 2023-03-03T07:58:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.462+0000 7fb3c359e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c41f3340 con 0x7fb3c4160780 2023-03-03T07:58:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.462+0000 7fb3bbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40b3550 0x7fb3c40cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.462+0000 7fb3bbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40b3550 0x7fb3c40cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54420/0 (socket says 172.21.15.98:54420) 2023-03-03T07:58:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.462+0000 7fb3bbfff700 1 -- 172.21.15.98:0/572627460 learned_addr learned my addr 172.21.15.98:0/572627460 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.462+0000 7fb3c259c700 1 --2- 172.21.15.98:0/572627460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb3c4073ed0 0x7fb3c414d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.462+0000 7fb3c2d9d700 1 --2- 172.21.15.98:0/572627460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb3c4160780 0x7fb3c41f2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.463+0000 7fb3bbfff700 1 -- 172.21.15.98:0/572627460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb3c4160780 msgr2=0x7fb3c41f2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.463+0000 7fb3bbfff700 1 --2- 172.21.15.98:0/572627460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb3c4160780 0x7fb3c41f2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.463+0000 7fb3bbfff700 1 -- 172.21.15.98:0/572627460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb3c4073ed0 msgr2=0x7fb3c414d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.463+0000 7fb3bbfff700 1 --2- 172.21.15.98:0/572627460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb3c4073ed0 0x7fb3c414d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.463+0000 7fb3bbfff700 1 -- 172.21.15.98:0/572627460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ac0094a0 con 0x7fb3c40b3550 2023-03-03T07:58:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.463+0000 7fb3bbfff700 1 --2- 172.21.15.98:0/572627460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40b3550 0x7fb3c40cc500 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac000c00 tx=0x7fb3ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.463+0000 7fb3baffd700 1 -- 172.21.15.98:0/572627460 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac0036a0 con 0x7fb3c40b3550 2023-03-03T07:58:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.463+0000 7fb3baffd700 1 -- 172.21.15.98:0/572627460 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ac01f070 con 0x7fb3c40b3550 2023-03-03T07:58:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.463+0000 7fb3baffd700 1 -- 172.21.15.98:0/572627460 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac003a60 con 0x7fb3c40b3550 2023-03-03T07:58:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.463+0000 7fb3c359e700 1 -- 172.21.15.98:0/572627460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c40d3d60 con 0x7fb3c40b3550 2023-03-03T07:58:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.464+0000 7fb3c359e700 1 -- 172.21.15.98:0/572627460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c40cd920 con 0x7fb3c40b3550 2023-03-03T07:58:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.465+0000 7fb3c359e700 1 -- 172.21.15.98:0/572627460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c40d3ef0 con 0x7fb3c40b3550 2023-03-03T07:58:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.466+0000 7fb3baffd700 1 -- 172.21.15.98:0/572627460 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fb3ac00f050 con 0x7fb3c40b3550 2023-03-03T07:58:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.467+0000 7fb3baffd700 1 --2- 172.21.15.98:0/572627460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb3a40683c0 0x7fb3a406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.467+0000 7fb3baffd700 1 -- 172.21.15.98:0/572627460 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb3ac021070 con 0x7fb3c40b3550 2023-03-03T07:58:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.467+0000 7fb3c259c700 1 --2- 172.21.15.98:0/572627460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb3a40683c0 0x7fb3a406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.468+0000 7fb3c259c700 1 --2- 172.21.15.98:0/572627460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb3a40683c0 0x7fb3a406a870 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4006b20 tx=0x7fb3b4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.471+0000 7fb3baffd700 1 -- 172.21.15.98:0/572627460 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb3c40d3ef0 con 0x7fb3c40b3550 2023-03-03T07:58:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:10.666+0000 7fb3c359e700 1 -- 172.21.15.98:0/572627460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb3c40d3ef0 con 0x7fb3c40b3550 2023-03-03T07:58:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.652+0000 7fb3baffd700 1 -- 172.21.15.98:0/572627460 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v49) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb3c40d3ef0 con 0x7fb3c40b3550 2023-03-03T07:58:11.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.660+0000 7fb3c359e700 1 -- 172.21.15.98:0/572627460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb3a40683c0 msgr2=0x7fb3a406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:11.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.660+0000 7fb3c359e700 1 --2- 172.21.15.98:0/572627460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb3a40683c0 0x7fb3a406a870 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4006b20 tx=0x7fb3b4008040 comp rx=0 tx=0).stop 2023-03-03T07:58:11.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.660+0000 7fb3c359e700 1 -- 172.21.15.98:0/572627460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40b3550 msgr2=0x7fb3c40cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:11.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.660+0000 7fb3c359e700 1 --2- 172.21.15.98:0/572627460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40b3550 0x7fb3c40cc500 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac000c00 tx=0x7fb3ac004750 comp rx=0 tx=0).stop 2023-03-03T07:58:11.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.660+0000 7fb3c359e700 1 -- 172.21.15.98:0/572627460 shutdown_connections 2023-03-03T07:58:11.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.660+0000 7fb3c359e700 1 --2- 172.21.15.98:0/572627460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb3a40683c0 0x7fb3a406a870 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:11.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.661+0000 7fb3c359e700 1 --2- 172.21.15.98:0/572627460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb3c4160780 0x7fb3c41f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:11.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.661+0000 7fb3c359e700 1 --2- 172.21.15.98:0/572627460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb3c4073ed0 0x7fb3c414d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:11.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.661+0000 7fb3c359e700 1 --2- 172.21.15.98:0/572627460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c40b3550 0x7fb3c40cc500 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:11.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.661+0000 7fb3c359e700 1 -- 172.21.15.98:0/572627460 >> 172.21.15.98:0/572627460 conn(0x7fb3c4073b00 msgr2=0x7fb3c406c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:11.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.661+0000 7fb3c359e700 1 -- 172.21.15.98:0/572627460 shutdown_connections 2023-03-03T07:58:11.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.662+0000 7fb3c359e700 1 -- 172.21.15.98:0/572627460 wait complete. 2023-03-03T07:58:11.664 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2023-03-03T07:58:11.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-03-03T07:58:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.948+0000 7f1a5759e700 1 Processor -- start 2023-03-03T07:58:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.949+0000 7f1a5759e700 1 -- start start 2023-03-03T07:58:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.950+0000 7f1a5759e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1a581609b0 0x7f1a58160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.950+0000 7f1a5759e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a581612c0 con 0x7f1a580b4aa0 2023-03-03T07:58:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.950+0000 7f1a5759e700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5815dd00 con 0x7f1a580cf570 2023-03-03T07:58:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.950+0000 7f1a5759e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a580cebb0 con 0x7f1a581609b0 2023-03-03T07:58:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.951+0000 7f1a56d9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1a580b4aa0 0x7f1a580d0bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57398/0 (socket says 172.21.15.98:57398) 2023-03-03T07:58:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.951+0000 7f1a56d9d700 1 -- 172.21.15.98:0/3912959997 learned_addr learned my addr 172.21.15.98:0/3912959997 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.951+0000 7f1a5659c700 1 --2- 172.21.15.98:0/3912959997 >> v2:172.21.15.100:3300/0 conn(0x7f1a581609b0 0x7f1a58160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.951+0000 7f1a5559a700 1 -- 172.21.15.98:0/3912959997 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518370150 0 0) 0x7f1a581612c0 con 0x7f1a580b4aa0 2023-03-03T07:58:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.951+0000 7f1a5559a700 1 -- 172.21.15.98:0/3912959997 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a400035b0 con 0x7f1a580b4aa0 2023-03-03T07:58:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.951+0000 7f1a5559a700 1 -- 172.21.15.98:0/3912959997 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962560463 0 0) 0x7f1a5815dd00 con 0x7f1a580cf570 2023-03-03T07:58:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.951+0000 7f1a5559a700 1 -- 172.21.15.98:0/3912959997 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a581612c0 con 0x7f1a580cf570 2023-03-03T07:58:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.951+0000 7f1a5659c700 1 -- 172.21.15.98:0/3912959997 >> v1:172.21.15.100:6789/0 conn(0x7f1a580cf570 legacy=0x7f1a5815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.951+0000 7f1a5659c700 1 -- 172.21.15.98:0/3912959997 >> v1:172.21.15.98:6789/0 conn(0x7f1a580b4aa0 legacy=0x7f1a580d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.951+0000 7f1a5659c700 1 -- 172.21.15.98:0/3912959997 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a580cd780 con 0x7f1a581609b0 2023-03-03T07:58:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.952+0000 7f1a5659c700 1 --2- 172.21.15.98:0/3912959997 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a581609b0 0x7f1a58160d80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009840 tx=0x7f1a48009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76d1b157265ecbeb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.952+0000 7f1a5559a700 1 -- 172.21.15.98:0/3912959997 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48011830 con 0x7f1a581609b0 2023-03-03T07:58:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.952+0000 7f1a5559a700 1 -- 172.21.15.98:0/3912959997 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a48011990 con 0x7f1a581609b0 2023-03-03T07:58:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.952+0000 7f1a5559a700 1 -- 172.21.15.98:0/3912959997 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48011d20 con 0x7f1a581609b0 2023-03-03T07:58:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.952+0000 7f1a5759e700 1 -- 172.21.15.98:0/3912959997 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a581609b0 msgr2=0x7f1a58160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.953+0000 7f1a5759e700 1 --2- 172.21.15.98:0/3912959997 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a581609b0 0x7f1a58160d80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009840 tx=0x7f1a48009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.953+0000 7f1a5759e700 1 -- 172.21.15.98:0/3912959997 shutdown_connections 2023-03-03T07:58:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.953+0000 7f1a5759e700 1 --2- 172.21.15.98:0/3912959997 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a581609b0 0x7f1a58160d80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.953+0000 7f1a5759e700 1 -- 172.21.15.98:0/3912959997 >> 172.21.15.98:0/3912959997 conn(0x7f1a5806b880 msgr2=0x7f1a580cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.953+0000 7f1a5759e700 1 -- 172.21.15.98:0/3912959997 shutdown_connections 2023-03-03T07:58:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.953+0000 7f1a5759e700 1 -- 172.21.15.98:0/3912959997 wait complete. 2023-03-03T07:58:11.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.955+0000 7f1a5759e700 1 Processor -- start 2023-03-03T07:58:11.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.955+0000 7f1a5759e700 1 -- start start 2023-03-03T07:58:11.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.956+0000 7f1a5759e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a580b4aa0 0x7f1a58151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:11.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.956+0000 7f1a5759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a580cf570 0x7f1a581e8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:11.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.956+0000 7f1a5759e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a581609b0 0x7f1a580b66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:11.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.956+0000 7f1a5759e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a581e8fd0 con 0x7f1a580cf570 2023-03-03T07:58:11.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.956+0000 7f1a5759e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a581e8a10 con 0x7f1a580b4aa0 2023-03-03T07:58:11.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.956+0000 7f1a5759e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a581e8cf0 con 0x7f1a581609b0 2023-03-03T07:58:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.956+0000 7f1a55d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a580cf570 0x7f1a581e8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.956+0000 7f1a55d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a580cf570 0x7f1a581e8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54436/0 (socket says 172.21.15.98:54436) 2023-03-03T07:58:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.956+0000 7f1a55d9b700 1 -- 172.21.15.98:0/2677997930 learned_addr learned my addr 172.21.15.98:0/2677997930 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.957+0000 7f1a5659c700 1 --2- 172.21.15.98:0/2677997930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a580b4aa0 0x7f1a58151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.957+0000 7f1a56d9d700 1 --2- 172.21.15.98:0/2677997930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a581609b0 0x7f1a580b66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.957+0000 7f1a55d9b700 1 -- 172.21.15.98:0/2677997930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a581609b0 msgr2=0x7f1a580b66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.957+0000 7f1a55d9b700 1 --2- 172.21.15.98:0/2677997930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a581609b0 0x7f1a580b66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.957+0000 7f1a55d9b700 1 -- 172.21.15.98:0/2677997930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a580b4aa0 msgr2=0x7f1a58151da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.957+0000 7f1a55d9b700 1 --2- 172.21.15.98:0/2677997930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a580b4aa0 0x7f1a58151da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.957+0000 7f1a55d9b700 1 -- 172.21.15.98:0/2677997930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a480094a0 con 0x7f1a580cf570 2023-03-03T07:58:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.957+0000 7f1a55d9b700 1 --2- 172.21.15.98:0/2677997930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a580cf570 0x7f1a581e8030 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00c520 tx=0x7f1a4c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.957+0000 7f1a46ffd700 1 -- 172.21.15.98:0/2677997930 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c0052d0 con 0x7f1a580cf570 2023-03-03T07:58:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.957+0000 7f1a46ffd700 1 -- 172.21.15.98:0/2677997930 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c005430 con 0x7f1a580cf570 2023-03-03T07:58:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.957+0000 7f1a46ffd700 1 -- 172.21.15.98:0/2677997930 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c0038a0 con 0x7f1a580cf570 2023-03-03T07:58:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.958+0000 7f1a5759e700 1 -- 172.21.15.98:0/2677997930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a580cdd40 con 0x7f1a580cf570 2023-03-03T07:58:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.958+0000 7f1a5759e700 1 -- 172.21.15.98:0/2677997930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a580cda60 con 0x7f1a580cf570 2023-03-03T07:58:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.959+0000 7f1a5759e700 1 -- 172.21.15.98:0/2677997930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5815cdf0 con 0x7f1a580cf570 2023-03-03T07:58:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.960+0000 7f1a46ffd700 1 -- 172.21.15.98:0/2677997930 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f1a4c019030 con 0x7f1a580cf570 2023-03-03T07:58:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.961+0000 7f1a46ffd700 1 --2- 172.21.15.98:0/2677997930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1a380683c0 0x7f1a3806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.962+0000 7f1a46ffd700 1 -- 172.21.15.98:0/2677997930 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5580+0+0 (secure 0 0 0) 0x7f1a4c0854a0 con 0x7f1a580cf570 2023-03-03T07:58:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.962+0000 7f1a5659c700 1 --2- 172.21.15.98:0/2677997930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1a380683c0 0x7f1a3806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.962+0000 7f1a5659c700 1 --2- 172.21.15.98:0/2677997930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1a380683c0 0x7f1a3806a870 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009fd0 tx=0x7f1a4801c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:11.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:11.965+0000 7f1a46ffd700 1 -- 172.21.15.98:0/2677997930 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1a4c051d40 con 0x7f1a580cf570 2023-03-03T07:58:12.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.161+0000 7f1a5759e700 1 -- 172.21.15.98:0/2677997930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1a58068290 con 0x7f1a580cf570 2023-03-03T07:58:12.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.659+0000 7f1a46ffd700 1 -- 172.21.15.98:0/2677997930 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v50) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1a4c055360 con 0x7f1a580cf570 2023-03-03T07:58:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.665+0000 7f1a5759e700 1 -- 172.21.15.98:0/2677997930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1a380683c0 msgr2=0x7f1a3806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.665+0000 7f1a5759e700 1 --2- 172.21.15.98:0/2677997930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1a380683c0 0x7f1a3806a870 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009fd0 tx=0x7f1a4801c040 comp rx=0 tx=0).stop 2023-03-03T07:58:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.665+0000 7f1a5759e700 1 -- 172.21.15.98:0/2677997930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a580cf570 msgr2=0x7f1a581e8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.665+0000 7f1a5759e700 1 --2- 172.21.15.98:0/2677997930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a580cf570 0x7f1a581e8030 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00c520 tx=0x7f1a4c00c830 comp rx=0 tx=0).stop 2023-03-03T07:58:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.670+0000 7f1a5759e700 1 -- 172.21.15.98:0/2677997930 shutdown_connections 2023-03-03T07:58:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.670+0000 7f1a5759e700 1 --2- 172.21.15.98:0/2677997930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1a380683c0 0x7f1a3806a870 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.670+0000 7f1a5759e700 1 --2- 172.21.15.98:0/2677997930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a581609b0 0x7f1a580b66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.670+0000 7f1a5759e700 1 --2- 172.21.15.98:0/2677997930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a580b4aa0 0x7f1a58151da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.670+0000 7f1a5759e700 1 --2- 172.21.15.98:0/2677997930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a580cf570 0x7f1a581e8030 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.670+0000 7f1a5759e700 1 -- 172.21.15.98:0/2677997930 >> 172.21.15.98:0/2677997930 conn(0x7f1a5806b880 msgr2=0x7f1a580b3690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.675+0000 7f1a5759e700 1 -- 172.21.15.98:0/2677997930 shutdown_connections 2023-03-03T07:58:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.676+0000 7f1a5759e700 1 -- 172.21.15.98:0/2677997930 wait complete. 2023-03-03T07:58:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2023-03-03T07:58:12.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-03-03T07:58:12.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.960+0000 7f6b62bff700 1 Processor -- start 2023-03-03T07:58:12.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.961+0000 7f6b62bff700 1 -- start start 2023-03-03T07:58:12.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.962+0000 7f6b62bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b5c15ca70 0x7f6b5c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:12.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.962+0000 7f6b62bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6b5c06b4b0 0x7f6b5c0cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:12.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.962+0000 7f6b62bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5c0cf980 con 0x7f6b5c0d00a0 2023-03-03T07:58:12.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.962+0000 7f6b62bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b5c0cd5b0 con 0x7f6b5c06b4b0 2023-03-03T07:58:12.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.962+0000 7f6b62bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c0d0510 con 0x7f6b5c15ca70 2023-03-03T07:58:12.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.963+0000 7f6b5bfff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6b5c06b4b0 0x7f6b5c0cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:12.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.963+0000 7f6b6099b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b5c15ca70 0x7f6b5c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:12.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.963+0000 7f6b6119c700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f6b5c0d00a0 0x7f6b5c0b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:34568/0 (socket says 172.21.15.98:34568) 2023-03-03T07:58:12.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.963+0000 7f6b6119c700 1 -- 172.21.15.98:0/1648534252 learned_addr learned my addr 172.21.15.98:0/1648534252 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:12.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.964+0000 7f6b5b7fe700 1 -- 172.21.15.98:0/1648534252 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822432314 0 0) 0x7f6b5c0cf980 con 0x7f6b5c0d00a0 2023-03-03T07:58:12.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.964+0000 7f6b6099b700 1 -- 172.21.15.98:0/1648534252 >> v2:172.21.15.100:3300/0 conn(0x7f6b5c06b4b0 msgr2=0x7f6b5c0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:12.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.964+0000 7f6b6099b700 1 --2- 172.21.15.98:0/1648534252 >> v2:172.21.15.100:3300/0 conn(0x7f6b5c06b4b0 0x7f6b5c0cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:12.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.964+0000 7f6b6099b700 1 -- 172.21.15.98:0/1648534252 >> v1:172.21.15.100:6789/0 conn(0x7f6b5c0d00a0 legacy=0x7f6b5c0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:12.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.964+0000 7f6b6099b700 1 -- 172.21.15.98:0/1648534252 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c0d2fd0 con 0x7f6b5c15ca70 2023-03-03T07:58:12.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.964+0000 7f6b6099b700 1 --2- 172.21.15.98:0/1648534252 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b5c15ca70 0x7f6b5c0cf440 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6b50009840 tx=0x7f6b50009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6cf71ea7f44b5fe server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:12.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.965+0000 7f6b5b7fe700 1 -- 172.21.15.98:0/1648534252 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b50011830 con 0x7f6b5c15ca70 2023-03-03T07:58:12.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.965+0000 7f6b5b7fe700 1 -- 172.21.15.98:0/1648534252 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b50011990 con 0x7f6b5c15ca70 2023-03-03T07:58:12.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.965+0000 7f6b5b7fe700 1 -- 172.21.15.98:0/1648534252 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b50011d20 con 0x7f6b5c15ca70 2023-03-03T07:58:12.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.965+0000 7f6b62bff700 1 -- 172.21.15.98:0/1648534252 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b5c15ca70 msgr2=0x7f6b5c0cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:12.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.965+0000 7f6b62bff700 1 --2- 172.21.15.98:0/1648534252 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b5c15ca70 0x7f6b5c0cf440 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6b50009840 tx=0x7f6b50009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:12.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.965+0000 7f6b62bff700 1 -- 172.21.15.98:0/1648534252 shutdown_connections 2023-03-03T07:58:12.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.965+0000 7f6b62bff700 1 --2- 172.21.15.98:0/1648534252 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b5c15ca70 0x7f6b5c0cf440 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:12.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.965+0000 7f6b62bff700 1 --2- 172.21.15.98:0/1648534252 >> v2:172.21.15.100:3300/0 conn(0x7f6b5c06b4b0 0x7f6b5c0cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:12.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.965+0000 7f6b62bff700 1 -- 172.21.15.98:0/1648534252 >> 172.21.15.98:0/1648534252 conn(0x7f6b5c073e70 msgr2=0x7f6b5c06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:12.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.965+0000 7f6b62bff700 1 -- 172.21.15.98:0/1648534252 shutdown_connections 2023-03-03T07:58:12.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.965+0000 7f6b62bff700 1 -- 172.21.15.98:0/1648534252 wait complete. 2023-03-03T07:58:12.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.967+0000 7f6b62bff700 1 Processor -- start 2023-03-03T07:58:12.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.967+0000 7f6b62bff700 1 -- start start 2023-03-03T07:58:12.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.968+0000 7f6b62bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b5c06b4b0 0x7f6b5c1f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:12.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.968+0000 7f6b62bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b5c0d00a0 0x7f6b5c157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:12.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.968+0000 7f6b62bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c15ca70 0x7f6b5c0ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:12.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.969+0000 7f6b62bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c1580e0 con 0x7f6b5c15ca70 2023-03-03T07:58:12.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.969+0000 7f6b62bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c1f66b0 con 0x7f6b5c06b4b0 2023-03-03T07:58:12.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.969+0000 7f6b62bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c157e80 con 0x7f6b5c0d00a0 2023-03-03T07:58:12.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.969+0000 7f6b6119c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c15ca70 0x7f6b5c0ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:12.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.969+0000 7f6b6119c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c15ca70 0x7f6b5c0ccbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54438/0 (socket says 172.21.15.98:54438) 2023-03-03T07:58:12.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.969+0000 7f6b6119c700 1 -- 172.21.15.98:0/2787279742 learned_addr learned my addr 172.21.15.98:0/2787279742 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:12.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.969+0000 7f6b5bfff700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b5c0d00a0 0x7f6b5c157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:12.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.969+0000 7f6b6099b700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b5c06b4b0 0x7f6b5c1f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:12.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.969+0000 7f6b6119c700 1 -- 172.21.15.98:0/2787279742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b5c0d00a0 msgr2=0x7f6b5c157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:12.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.969+0000 7f6b6119c700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b5c0d00a0 0x7f6b5c157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:12.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.969+0000 7f6b6119c700 1 -- 172.21.15.98:0/2787279742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b5c06b4b0 msgr2=0x7f6b5c1f2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:12.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.969+0000 7f6b6119c700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b5c06b4b0 0x7f6b5c1f2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:12.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.969+0000 7f6b6119c700 1 -- 172.21.15.98:0/2787279742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b500094a0 con 0x7f6b5c15ca70 2023-03-03T07:58:12.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.970+0000 7f6b5bfff700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b5c0d00a0 0x7f6b5c157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:58:12.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.970+0000 7f6b6099b700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b5c06b4b0 0x7f6b5c1f2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:58:12.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.970+0000 7f6b6119c700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c15ca70 0x7f6b5c0ccbc0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6b54002230 tx=0x7f6b5400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:12.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.970+0000 7f6b597fa700 1 -- 172.21.15.98:0/2787279742 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5400aac0 con 0x7f6b5c15ca70 2023-03-03T07:58:12.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.970+0000 7f6b597fa700 1 -- 172.21.15.98:0/2787279742 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5400ac40 con 0x7f6b5c15ca70 2023-03-03T07:58:12.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.970+0000 7f6b597fa700 1 -- 172.21.15.98:0/2787279742 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b540070d0 con 0x7f6b5c15ca70 2023-03-03T07:58:12.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.970+0000 7f6b62bff700 1 -- 172.21.15.98:0/2787279742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b5c0cd900 con 0x7f6b5c15ca70 2023-03-03T07:58:12.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.970+0000 7f6b62bff700 1 -- 172.21.15.98:0/2787279742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b5c0d1800 con 0x7f6b5c15ca70 2023-03-03T07:58:12.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.972+0000 7f6b62bff700 1 -- 172.21.15.98:0/2787279742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5c0cdbe0 con 0x7f6b5c15ca70 2023-03-03T07:58:12.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.973+0000 7f6b597fa700 1 -- 172.21.15.98:0/2787279742 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f6b54010070 con 0x7f6b5c15ca70 2023-03-03T07:58:12.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.974+0000 7f6b597fa700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b400683c0 0x7f6b4006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:12.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.974+0000 7f6b597fa700 1 -- 172.21.15.98:0/2787279742 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f6b54085750 con 0x7f6b5c15ca70 2023-03-03T07:58:12.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.974+0000 7f6b6099b700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b400683c0 0x7f6b4006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:12.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.975+0000 7f6b6099b700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b400683c0 0x7f6b4006a870 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6b500090b0 tx=0x7f6b5001c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:12.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:12.979+0000 7f6b597fa700 1 -- 172.21.15.98:0/2787279742 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6b54051dd0 con 0x7f6b5c15ca70 2023-03-03T07:58:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.212+0000 7f6b62bff700 1 -- 172.21.15.98:0/2787279742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6b5c04e6e0 con 0x7f6b5c15ca70 2023-03-03T07:58:13.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.219+0000 7f6b597fa700 1 -- 172.21.15.98:0/2787279742 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v19) v1 ==== 172+0+0 (secure 0 0 0) 0x7f6b540553f0 con 0x7f6b5c15ca70 2023-03-03T07:58:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.224+0000 7f6b62bff700 1 -- 172.21.15.98:0/2787279742 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b400683c0 msgr2=0x7f6b4006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.224+0000 7f6b62bff700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b400683c0 0x7f6b4006a870 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6b500090b0 tx=0x7f6b5001c040 comp rx=0 tx=0).stop 2023-03-03T07:58:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.224+0000 7f6b62bff700 1 -- 172.21.15.98:0/2787279742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c15ca70 msgr2=0x7f6b5c0ccbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.224+0000 7f6b62bff700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c15ca70 0x7f6b5c0ccbc0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6b54002230 tx=0x7f6b5400cf50 comp rx=0 tx=0).stop 2023-03-03T07:58:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.225+0000 7f6b62bff700 1 -- 172.21.15.98:0/2787279742 shutdown_connections 2023-03-03T07:58:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.225+0000 7f6b62bff700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b400683c0 0x7f6b4006a870 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.225+0000 7f6b62bff700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c15ca70 0x7f6b5c0ccbc0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.225+0000 7f6b62bff700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b5c06b4b0 0x7f6b5c1f2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.225+0000 7f6b62bff700 1 --2- 172.21.15.98:0/2787279742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b5c0d00a0 0x7f6b5c157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.225+0000 7f6b62bff700 1 -- 172.21.15.98:0/2787279742 >> 172.21.15.98:0/2787279742 conn(0x7f6b5c073e70 msgr2=0x7f6b5c15a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.228+0000 7f6b62bff700 1 -- 172.21.15.98:0/2787279742 shutdown_connections 2023-03-03T07:58:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.233+0000 7f6b62bff700 1 -- 172.21.15.98:0/2787279742 wait complete. 2023-03-03T07:58:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 7 and data pool 8 2023-03-03T07:58:13.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:58:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.522+0000 7f9cf89fe700 1 Processor -- start 2023-03-03T07:58:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.522+0000 7f9cf89fe700 1 -- start start 2023-03-03T07:58:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.523+0000 7f9cf89fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cf40ce4a0 0x7f9cf415a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.523+0000 7f9cf89fe700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf40cd000 con 0x7f9cf41595d0 2023-03-03T07:58:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.524+0000 7f9cf89fe700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf415a5f0 con 0x7f9cf406c7c0 2023-03-03T07:58:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.524+0000 7f9cf89fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf406cd80 con 0x7f9cf40ce4a0 2023-03-03T07:58:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.524+0000 7f9cf2d9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9cf41595d0 0x7f9cf40c9d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57402/0 (socket says 172.21.15.98:57402) 2023-03-03T07:58:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.524+0000 7f9cf2d9d700 1 -- 172.21.15.98:0/428891699 learned_addr learned my addr 172.21.15.98:0/428891699 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:13.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.524+0000 7f9cf359e700 1 --2- 172.21.15.98:0/428891699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cf40ce4a0 0x7f9cf415a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:13.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.524+0000 7f9cf1d9b700 1 -- 172.21.15.98:0/428891699 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579520323 0 0) 0x7f9cf40cd000 con 0x7f9cf41595d0 2023-03-03T07:58:13.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.524+0000 7f9cf1d9b700 1 -- 172.21.15.98:0/428891699 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cdc0035b0 con 0x7f9cf41595d0 2023-03-03T07:58:13.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.524+0000 7f9cf1d9b700 1 -- 172.21.15.98:0/428891699 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115984997 0 0) 0x7f9cf415a5f0 con 0x7f9cf406c7c0 2023-03-03T07:58:13.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.524+0000 7f9cf1d9b700 1 -- 172.21.15.98:0/428891699 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cf40cd000 con 0x7f9cf406c7c0 2023-03-03T07:58:13.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.525+0000 7f9cf359e700 1 -- 172.21.15.98:0/428891699 >> v1:172.21.15.100:6789/0 conn(0x7f9cf406c7c0 legacy=0x7f9cf41599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:13.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.525+0000 7f9cf359e700 1 -- 172.21.15.98:0/428891699 >> v1:172.21.15.98:6789/0 conn(0x7f9cf41595d0 legacy=0x7f9cf40c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.525+0000 7f9cf359e700 1 -- 172.21.15.98:0/428891699 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf406d060 con 0x7f9cf40ce4a0 2023-03-03T07:58:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.525+0000 7f9cf359e700 1 --2- 172.21.15.98:0/428891699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cf40ce4a0 0x7f9cf415a0b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9cec009870 tx=0x7f9cec009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=744a124584c4981e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.525+0000 7f9cf1d9b700 1 -- 172.21.15.98:0/428891699 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec011830 con 0x7f9cf40ce4a0 2023-03-03T07:58:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.525+0000 7f9cf1d9b700 1 -- 172.21.15.98:0/428891699 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cec011990 con 0x7f9cf40ce4a0 2023-03-03T07:58:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.525+0000 7f9cf1d9b700 1 -- 172.21.15.98:0/428891699 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec011d20 con 0x7f9cf40ce4a0 2023-03-03T07:58:13.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.526+0000 7f9cf89fe700 1 -- 172.21.15.98:0/428891699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cf40ce4a0 msgr2=0x7f9cf415a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:13.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.526+0000 7f9cf89fe700 1 --2- 172.21.15.98:0/428891699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cf40ce4a0 0x7f9cf415a0b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9cec009870 tx=0x7f9cec009b80 comp rx=0 tx=0).stop 2023-03-03T07:58:13.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.526+0000 7f9cf89fe700 1 -- 172.21.15.98:0/428891699 shutdown_connections 2023-03-03T07:58:13.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.526+0000 7f9cf89fe700 1 --2- 172.21.15.98:0/428891699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cf40ce4a0 0x7f9cf415a0b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:13.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.526+0000 7f9cf89fe700 1 -- 172.21.15.98:0/428891699 >> 172.21.15.98:0/428891699 conn(0x7f9cf40cf950 msgr2=0x7f9cf40cb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:13.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.526+0000 7f9cf89fe700 1 -- 172.21.15.98:0/428891699 shutdown_connections 2023-03-03T07:58:13.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.526+0000 7f9cf89fe700 1 -- 172.21.15.98:0/428891699 wait complete. 2023-03-03T07:58:13.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.528+0000 7f9cf89fe700 1 Processor -- start 2023-03-03T07:58:13.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.528+0000 7f9cf89fe700 1 -- start start 2023-03-03T07:58:13.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.529+0000 7f9cf89fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9cf406c7c0 0x7f9cf41f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:13.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.529+0000 7f9cf89fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cf40ce4a0 0x7f9cf41f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:13.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.529+0000 7f9cf89fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cf41595d0 0x7f9cf40c2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:13.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.530+0000 7f9cf89fe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf41f3160 con 0x7f9cf41595d0 2023-03-03T07:58:13.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.530+0000 7f9cf89fe700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf41f2e80 con 0x7f9cf406c7c0 2023-03-03T07:58:13.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.530+0000 7f9cf89fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf415bf20 con 0x7f9cf40ce4a0 2023-03-03T07:58:13.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.530+0000 7f9cf359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cf41595d0 0x7f9cf40c2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:13.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.530+0000 7f9cf359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cf41595d0 0x7f9cf40c2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54442/0 (socket says 172.21.15.98:54442) 2023-03-03T07:58:13.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.530+0000 7f9cf359e700 1 -- 172.21.15.98:0/1260145601 learned_addr learned my addr 172.21.15.98:0/1260145601 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:13.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.530+0000 7f9cf2d9d700 1 --2- 172.21.15.98:0/1260145601 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9cf406c7c0 0x7f9cf41f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:13.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.530+0000 7f9cf259c700 1 --2- 172.21.15.98:0/1260145601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cf40ce4a0 0x7f9cf41f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:13.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.530+0000 7f9cf359e700 1 -- 172.21.15.98:0/1260145601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cf40ce4a0 msgr2=0x7f9cf41f4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:13.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.530+0000 7f9cf359e700 1 --2- 172.21.15.98:0/1260145601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cf40ce4a0 0x7f9cf41f4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:13.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.530+0000 7f9cf359e700 1 -- 172.21.15.98:0/1260145601 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9cf406c7c0 msgr2=0x7f9cf41f32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:13.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.530+0000 7f9cf359e700 1 --2- 172.21.15.98:0/1260145601 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9cf406c7c0 0x7f9cf41f32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:13.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.530+0000 7f9cf359e700 1 -- 172.21.15.98:0/1260145601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec0094a0 con 0x7f9cf41595d0 2023-03-03T07:58:13.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.530+0000 7f9cf259c700 1 --2- 172.21.15.98:0/1260145601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cf40ce4a0 0x7f9cf41f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:13.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.531+0000 7f9cf359e700 1 --2- 172.21.15.98:0/1260145601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cf41595d0 0x7f9cf40c2780 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9cec0097c0 tx=0x7f9cec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:13.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.531+0000 7f9cdb7fe700 1 -- 172.21.15.98:0/1260145601 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec01a6e0 con 0x7f9cf41595d0 2023-03-03T07:58:13.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.531+0000 7f9cdb7fe700 1 -- 172.21.15.98:0/1260145601 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cec01a840 con 0x7f9cf41595d0 2023-03-03T07:58:13.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.531+0000 7f9cf89fe700 1 -- 172.21.15.98:0/1260145601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf406d620 con 0x7f9cf41595d0 2023-03-03T07:58:13.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.531+0000 7f9cdb7fe700 1 -- 172.21.15.98:0/1260145601 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec01abf0 con 0x7f9cf41595d0 2023-03-03T07:58:13.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.531+0000 7f9cf89fe700 1 -- 172.21.15.98:0/1260145601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf406d340 con 0x7f9cf41595d0 2023-03-03T07:58:13.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.533+0000 7f9cf89fe700 1 -- 172.21.15.98:0/1260145601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf40b07f0 con 0x7f9cf41595d0 2023-03-03T07:58:13.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.534+0000 7f9cdb7fe700 1 -- 172.21.15.98:0/1260145601 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f9cec02b030 con 0x7f9cf41595d0 2023-03-03T07:58:13.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.535+0000 7f9cdb7fe700 1 --2- 172.21.15.98:0/1260145601 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ce00683c0 0x7f9ce006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:13.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.535+0000 7f9cdb7fe700 1 -- 172.21.15.98:0/1260145601 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9cec087080 con 0x7f9cf41595d0 2023-03-03T07:58:13.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.535+0000 7f9cf2d9d700 1 --2- 172.21.15.98:0/1260145601 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ce00683c0 0x7f9ce006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:13.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.536+0000 7f9cf2d9d700 1 --2- 172.21.15.98:0/1260145601 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ce00683c0 0x7f9ce006a870 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40012e0 tx=0x7f9ce4001a30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:13.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.539+0000 7f9cdb7fe700 1 -- 172.21.15.98:0/1260145601 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9cec0b85e0 con 0x7f9cf41595d0 2023-03-03T07:58:13.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.735+0000 7f9cf89fe700 1 -- 172.21.15.98:0/1260145601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cf415bd80 con 0x7f9cf41595d0 2023-03-03T07:58:13.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.736+0000 7f9cdb7fe700 1 -- 172.21.15.98:0/1260145601 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+17992 (secure 0 0 0) 0x7f9cf415bd80 con 0x7f9cf41595d0 2023-03-03T07:58:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:13.738 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":51,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:58:13.213569+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:58:10.728830+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-03-03T07:58:12.224649+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:58:13.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.739+0000 7f9cf89fe700 1 -- 172.21.15.98:0/1260145601 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ce00683c0 msgr2=0x7f9ce006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:13.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.739+0000 7f9cf89fe700 1 --2- 172.21.15.98:0/1260145601 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ce00683c0 0x7f9ce006a870 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40012e0 tx=0x7f9ce4001a30 comp rx=0 tx=0).stop 2023-03-03T07:58:13.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.739+0000 7f9cf89fe700 1 -- 172.21.15.98:0/1260145601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cf41595d0 msgr2=0x7f9cf40c2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:13.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.739+0000 7f9cf89fe700 1 --2- 172.21.15.98:0/1260145601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cf41595d0 0x7f9cf40c2780 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9cec0097c0 tx=0x7f9cec004750 comp rx=0 tx=0).stop 2023-03-03T07:58:13.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.739+0000 7f9cf89fe700 1 -- 172.21.15.98:0/1260145601 shutdown_connections 2023-03-03T07:58:13.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.739+0000 7f9cf89fe700 1 --2- 172.21.15.98:0/1260145601 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9ce00683c0 0x7f9ce006a870 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:13.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.739+0000 7f9cf89fe700 1 --2- 172.21.15.98:0/1260145601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cf41595d0 0x7f9cf40c2780 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:13.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.739+0000 7f9cf89fe700 1 --2- 172.21.15.98:0/1260145601 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9cf406c7c0 0x7f9cf41f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:13.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.739+0000 7f9cf89fe700 1 --2- 172.21.15.98:0/1260145601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cf40ce4a0 0x7f9cf41f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:13.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.739+0000 7f9cf89fe700 1 -- 172.21.15.98:0/1260145601 >> 172.21.15.98:0/1260145601 conn(0x7f9cf40cf950 msgr2=0x7f9cf40705a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:13.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.740+0000 7f9cf89fe700 1 -- 172.21.15.98:0/1260145601 shutdown_connections 2023-03-03T07:58:13.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:13.740+0000 7f9cf89fe700 1 -- 172.21.15.98:0/1260145601 wait complete. 2023-03-03T07:58:13.769 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:58:14.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.014+0000 7f1c547ff700 1 Processor -- start 2023-03-03T07:58:14.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.014+0000 7f1c547ff700 1 -- start start 2023-03-03T07:58:14.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.015+0000 7f1c547ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1c4c0b87c0 0x7f1c4c0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.015+0000 7f1c547ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c0d0ef0 con 0x7f1c4c0b8b90 2023-03-03T07:58:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.015+0000 7f1c547ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c0d0600 con 0x7f1c4c0b5410 2023-03-03T07:58:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.015+0000 7f1c547ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c4c0cd970 con 0x7f1c4c0b87c0 2023-03-03T07:58:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.016+0000 7f1c5259b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1c4c0b87c0 0x7f1c4c0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.016+0000 7f1c51d9a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1c4c0b8b90 0x7f1c4c0b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57412/0 (socket says 172.21.15.98:57412) 2023-03-03T07:58:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.016+0000 7f1c51d9a700 1 -- 172.21.15.98:0/1178029285 learned_addr learned my addr 172.21.15.98:0/1178029285 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.016+0000 7f1c5259b700 1 -- 172.21.15.98:0/1178029285 >> v1:172.21.15.100:6789/0 conn(0x7f1c4c0b5410 legacy=0x7f1c4c0d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.016+0000 7f1c5259b700 1 -- 172.21.15.98:0/1178029285 >> v1:172.21.15.98:6789/0 conn(0x7f1c4c0b8b90 legacy=0x7f1c4c0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.016+0000 7f1c51599700 1 -- 172.21.15.98:0/1178029285 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335160571 0 0) 0x7f1c4c0d0ef0 con 0x7f1c4c0b8b90 2023-03-03T07:58:14.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.016+0000 7f1c5259b700 1 -- 172.21.15.98:0/1178029285 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4c0cdc50 con 0x7f1c4c0b87c0 2023-03-03T07:58:14.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.017+0000 7f1c5259b700 1 --2- 172.21.15.98:0/1178029285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c0b87c0 0x7f1c4c0cffe0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1c48009840 tx=0x7f1c48009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=960cbdb2e4c3d433 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:14.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.017+0000 7f1c51599700 1 -- 172.21.15.98:0/1178029285 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48011830 con 0x7f1c4c0b87c0 2023-03-03T07:58:14.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.017+0000 7f1c51599700 1 -- 172.21.15.98:0/1178029285 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c48011990 con 0x7f1c4c0b87c0 2023-03-03T07:58:14.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.017+0000 7f1c51599700 1 -- 172.21.15.98:0/1178029285 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48011d20 con 0x7f1c4c0b87c0 2023-03-03T07:58:14.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.018+0000 7f1c547ff700 1 -- 172.21.15.98:0/1178029285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c0b87c0 msgr2=0x7f1c4c0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:14.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.018+0000 7f1c547ff700 1 --2- 172.21.15.98:0/1178029285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c0b87c0 0x7f1c4c0cffe0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1c48009840 tx=0x7f1c48009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:14.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.018+0000 7f1c547ff700 1 -- 172.21.15.98:0/1178029285 shutdown_connections 2023-03-03T07:58:14.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.018+0000 7f1c547ff700 1 --2- 172.21.15.98:0/1178029285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c0b87c0 0x7f1c4c0cffe0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:14.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.018+0000 7f1c547ff700 1 -- 172.21.15.98:0/1178029285 >> 172.21.15.98:0/1178029285 conn(0x7f1c4c06aee0 msgr2=0x7f1c4c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:14.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.018+0000 7f1c547ff700 1 -- 172.21.15.98:0/1178029285 shutdown_connections 2023-03-03T07:58:14.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.018+0000 7f1c547ff700 1 -- 172.21.15.98:0/1178029285 wait complete. 2023-03-03T07:58:14.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.020+0000 7f1c547ff700 1 Processor -- start 2023-03-03T07:58:14.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.020+0000 7f1c547ff700 1 -- start start 2023-03-03T07:58:14.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.021+0000 7f1c547ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1c4c0b5410 0x7f1c4c156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:14.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.021+0000 7f1c547ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1c4c0b87c0 0x7f1c4c1f5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:14.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.021+0000 7f1c547ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c0b8b90 0x7f1c4c1f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:14.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.021+0000 7f1c547ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c159c50 con 0x7f1c4c0b8b90 2023-03-03T07:58:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.021+0000 7f1c547ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c15ecb0 con 0x7f1c4c0b87c0 2023-03-03T07:58:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.021+0000 7f1c547ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c159970 con 0x7f1c4c0b5410 2023-03-03T07:58:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.021+0000 7f1c52d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c0b8b90 0x7f1c4c1f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.021+0000 7f1c52d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c0b8b90 0x7f1c4c1f5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54446/0 (socket says 172.21.15.98:54446) 2023-03-03T07:58:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.021+0000 7f1c52d9c700 1 -- 172.21.15.98:0/1866453444 learned_addr learned my addr 172.21.15.98:0/1866453444 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.022+0000 7f1c5259b700 1 --2- 172.21.15.98:0/1866453444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1c4c0b5410 0x7f1c4c156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.022+0000 7f1c51d9a700 1 --2- 172.21.15.98:0/1866453444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1c4c0b87c0 0x7f1c4c1f5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:14.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.022+0000 7f1c52d9c700 1 -- 172.21.15.98:0/1866453444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1c4c0b5410 msgr2=0x7f1c4c156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:14.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.022+0000 7f1c52d9c700 1 --2- 172.21.15.98:0/1866453444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1c4c0b5410 0x7f1c4c156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:14.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.022+0000 7f1c52d9c700 1 -- 172.21.15.98:0/1866453444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1c4c0b87c0 msgr2=0x7f1c4c1f5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:14.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.022+0000 7f1c52d9c700 1 --2- 172.21.15.98:0/1866453444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1c4c0b87c0 0x7f1c4c1f5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:14.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.022+0000 7f1c52d9c700 1 -- 172.21.15.98:0/1866453444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c480094a0 con 0x7f1c4c0b8b90 2023-03-03T07:58:14.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.022+0000 7f1c51d9a700 1 --2- 172.21.15.98:0/1866453444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1c4c0b87c0 0x7f1c4c1f5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:14.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.022+0000 7f1c52d9c700 1 --2- 172.21.15.98:0/1866453444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c0b8b90 0x7f1c4c1f5b70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400cb50 tx=0x7f1c4400cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:14.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.022+0000 7f1c42ffd700 1 -- 172.21.15.98:0/1866453444 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c44003c00 con 0x7f1c4c0b8b90 2023-03-03T07:58:14.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.022+0000 7f1c42ffd700 1 -- 172.21.15.98:0/1866453444 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c44003d60 con 0x7f1c4c0b8b90 2023-03-03T07:58:14.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.022+0000 7f1c42ffd700 1 -- 172.21.15.98:0/1866453444 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4400ab70 con 0x7f1c4c0b8b90 2023-03-03T07:58:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.022+0000 7f1c547ff700 1 -- 172.21.15.98:0/1866453444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4c0cdf30 con 0x7f1c4c0b8b90 2023-03-03T07:58:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.023+0000 7f1c547ff700 1 -- 172.21.15.98:0/1866453444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4c073a80 con 0x7f1c4c0b8b90 2023-03-03T07:58:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.024+0000 7f1c547ff700 1 -- 172.21.15.98:0/1866453444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c4c0ce0c0 con 0x7f1c4c0b8b90 2023-03-03T07:58:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.025+0000 7f1c42ffd700 1 -- 172.21.15.98:0/1866453444 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f1c44010050 con 0x7f1c4c0b8b90 2023-03-03T07:58:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.026+0000 7f1c42ffd700 1 --2- 172.21.15.98:0/1866453444 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1c300683c0 0x7f1c3006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.026+0000 7f1c42ffd700 1 -- 172.21.15.98:0/1866453444 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1c44085da0 con 0x7f1c4c0b8b90 2023-03-03T07:58:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.027+0000 7f1c5259b700 1 --2- 172.21.15.98:0/1866453444 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1c300683c0 0x7f1c3006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.027+0000 7f1c5259b700 1 --2- 172.21.15.98:0/1866453444 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1c300683c0 0x7f1c3006a870 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1c48000c00 tx=0x7f1c4801c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:14.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.030+0000 7f1c42ffd700 1 -- 172.21.15.98:0/1866453444 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1c44052440 con 0x7f1c4c0b8b90 2023-03-03T07:58:14.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.228+0000 7f1c547ff700 1 -- 172.21.15.98:0/1866453444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c4c0729b0 con 0x7f1c4c0b8b90 2023-03-03T07:58:14.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.229+0000 7f1c42ffd700 1 -- 172.21.15.98:0/1866453444 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+17979 (secure 0 0 0) 0x7f1c44055a60 con 0x7f1c4c0b8b90 2023-03-03T07:58:14.231 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:14.231 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":52,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:58:14.216922+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:58:10.728830+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-03-03T07:58:12.224649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:58:14.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.232+0000 7f1c547ff700 1 -- 172.21.15.98:0/1866453444 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1c300683c0 msgr2=0x7f1c3006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:14.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.232+0000 7f1c547ff700 1 --2- 172.21.15.98:0/1866453444 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1c300683c0 0x7f1c3006a870 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1c48000c00 tx=0x7f1c4801c040 comp rx=0 tx=0).stop 2023-03-03T07:58:14.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.232+0000 7f1c547ff700 1 -- 172.21.15.98:0/1866453444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c0b8b90 msgr2=0x7f1c4c1f5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:14.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.232+0000 7f1c547ff700 1 --2- 172.21.15.98:0/1866453444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c0b8b90 0x7f1c4c1f5b70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400cb50 tx=0x7f1c4400cf10 comp rx=0 tx=0).stop 2023-03-03T07:58:14.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.232+0000 7f1c547ff700 1 -- 172.21.15.98:0/1866453444 shutdown_connections 2023-03-03T07:58:14.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.232+0000 7f1c547ff700 1 --2- 172.21.15.98:0/1866453444 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1c300683c0 0x7f1c3006a870 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:14.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.232+0000 7f1c547ff700 1 --2- 172.21.15.98:0/1866453444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c0b8b90 0x7f1c4c1f5b70 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:14.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.232+0000 7f1c547ff700 1 --2- 172.21.15.98:0/1866453444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1c4c0b5410 0x7f1c4c156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:14.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.232+0000 7f1c547ff700 1 --2- 172.21.15.98:0/1866453444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1c4c0b87c0 0x7f1c4c1f5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:14.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.232+0000 7f1c547ff700 1 -- 172.21.15.98:0/1866453444 >> 172.21.15.98:0/1866453444 conn(0x7f1c4c06aee0 msgr2=0x7f1c4c0b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:14.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.232+0000 7f1c547ff700 1 -- 172.21.15.98:0/1866453444 shutdown_connections 2023-03-03T07:58:14.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.232+0000 7f1c547ff700 1 -- 172.21.15.98:0/1866453444 wait complete. 2023-03-03T07:58:14.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-03-03T07:58:14.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.511+0000 7fe49719d700 1 Processor -- start 2023-03-03T07:58:14.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.512+0000 7fe49719d700 1 -- start start 2023-03-03T07:58:14.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.513+0000 7fe49719d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4901609b0 0x7fe490160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:14.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.513+0000 7fe49719d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe4900cf570 0x7fe49015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:14.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.513+0000 7fe49719d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4901612c0 con 0x7fe49015db30 2023-03-03T07:58:14.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.513+0000 7fe49719d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4900cebb0 con 0x7fe4900cf570 2023-03-03T07:58:14.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.513+0000 7fe49719d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900cd780 con 0x7fe4901609b0 2023-03-03T07:58:14.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.514+0000 7fe49699c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe49015db30 0x7fe4900b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57418/0 (socket says 172.21.15.98:57418) 2023-03-03T07:58:14.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.514+0000 7fe49699c700 1 -- 172.21.15.98:0/378817 learned_addr learned my addr 172.21.15.98:0/378817 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:14.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.514+0000 7fe49619b700 1 --2- 172.21.15.98:0/378817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4901609b0 0x7fe490160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:14.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.514+0000 7fe49599a700 1 --2- 172.21.15.98:0/378817 >> v2:172.21.15.100:3300/0 conn(0x7fe4900cf570 0x7fe49015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:14.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.514+0000 7fe495199700 1 -- 172.21.15.98:0/378817 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926350706 0 0) 0x7fe4901612c0 con 0x7fe49015db30 2023-03-03T07:58:14.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.514+0000 7fe495199700 1 -- 172.21.15.98:0/378817 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe47c0035b0 con 0x7fe49015db30 2023-03-03T07:58:14.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.514+0000 7fe49599a700 1 -- 172.21.15.98:0/378817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4901609b0 msgr2=0x7fe490160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:14.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.514+0000 7fe49599a700 1 --2- 172.21.15.98:0/378817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4901609b0 0x7fe490160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:14.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.514+0000 7fe49599a700 1 -- 172.21.15.98:0/378817 >> v1:172.21.15.98:6789/0 conn(0x7fe49015db30 legacy=0x7fe4900b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:14.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.514+0000 7fe49599a700 1 -- 172.21.15.98:0/378817 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4900cda60 con 0x7fe4900cf570 2023-03-03T07:58:14.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.515+0000 7fe49619b700 1 --2- 172.21.15.98:0/378817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4901609b0 0x7fe490160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:58:14.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.515+0000 7fe49599a700 1 --2- 172.21.15.98:0/378817 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4900cf570 0x7fe49015d5f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe480009840 tx=0x7fe480009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b5fffa7499463b7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:14.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.515+0000 7fe495199700 1 -- 172.21.15.98:0/378817 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe480011830 con 0x7fe4900cf570 2023-03-03T07:58:14.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.515+0000 7fe495199700 1 -- 172.21.15.98:0/378817 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe480011990 con 0x7fe4900cf570 2023-03-03T07:58:14.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.515+0000 7fe495199700 1 -- 172.21.15.98:0/378817 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe480011d20 con 0x7fe4900cf570 2023-03-03T07:58:14.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.516+0000 7fe49719d700 1 -- 172.21.15.98:0/378817 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4900cf570 msgr2=0x7fe49015d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:14.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.516+0000 7fe49719d700 1 --2- 172.21.15.98:0/378817 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4900cf570 0x7fe49015d5f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe480009840 tx=0x7fe480009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:14.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.516+0000 7fe49719d700 1 -- 172.21.15.98:0/378817 shutdown_connections 2023-03-03T07:58:14.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.516+0000 7fe49719d700 1 --2- 172.21.15.98:0/378817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4901609b0 0x7fe490160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:14.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.516+0000 7fe49719d700 1 --2- 172.21.15.98:0/378817 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4900cf570 0x7fe49015d5f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:14.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.516+0000 7fe49719d700 1 -- 172.21.15.98:0/378817 >> 172.21.15.98:0/378817 conn(0x7fe49006b880 msgr2=0x7fe4900cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:14.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.516+0000 7fe49719d700 1 -- 172.21.15.98:0/378817 shutdown_connections 2023-03-03T07:58:14.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.516+0000 7fe49719d700 1 -- 172.21.15.98:0/378817 wait complete. 2023-03-03T07:58:14.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.518+0000 7fe49719d700 1 Processor -- start 2023-03-03T07:58:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.518+0000 7fe49719d700 1 -- start start 2023-03-03T07:58:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.519+0000 7fe49719d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4900cf570 0x7fe4900c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.519+0000 7fe49719d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe49015db30 0x7fe4900ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.519+0000 7fe49719d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4901609b0 0x7fe4900b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.519+0000 7fe49719d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900cb5c0 con 0x7fe4900cf570 2023-03-03T07:58:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.519+0000 7fe49719d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900ca9b0 con 0x7fe4901609b0 2023-03-03T07:58:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.520+0000 7fe49719d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900cac90 con 0x7fe49015db30 2023-03-03T07:58:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.520+0000 7fe49619b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4900cf570 0x7fe4900c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.520+0000 7fe49619b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4900cf570 0x7fe4900c9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54448/0 (socket says 172.21.15.98:54448) 2023-03-03T07:58:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.520+0000 7fe49619b700 1 -- 172.21.15.98:0/1208740050 learned_addr learned my addr 172.21.15.98:0/1208740050 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.520+0000 7fe49599a700 1 --2- 172.21.15.98:0/1208740050 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe49015db30 0x7fe4900ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.520+0000 7fe49699c700 1 --2- 172.21.15.98:0/1208740050 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4901609b0 0x7fe4900b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.520+0000 7fe49619b700 1 -- 172.21.15.98:0/1208740050 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe49015db30 msgr2=0x7fe4900ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.520+0000 7fe49619b700 1 --2- 172.21.15.98:0/1208740050 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe49015db30 0x7fe4900ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.520+0000 7fe49619b700 1 -- 172.21.15.98:0/1208740050 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4901609b0 msgr2=0x7fe4900b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.520+0000 7fe49619b700 1 --2- 172.21.15.98:0/1208740050 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4901609b0 0x7fe4900b3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.520+0000 7fe49619b700 1 -- 172.21.15.98:0/1208740050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c0094a0 con 0x7fe4900cf570 2023-03-03T07:58:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.521+0000 7fe49619b700 1 --2- 172.21.15.98:0/1208740050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4900cf570 0x7fe4900c9350 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe48c00ebe0 tx=0x7fe48c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.521+0000 7fe486ffd700 1 -- 172.21.15.98:0/1208740050 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c00c850 con 0x7fe4900cf570 2023-03-03T07:58:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.521+0000 7fe486ffd700 1 -- 172.21.15.98:0/1208740050 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48c00c9b0 con 0x7fe4900cf570 2023-03-03T07:58:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.521+0000 7fe486ffd700 1 -- 172.21.15.98:0/1208740050 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c00cd40 con 0x7fe4900cf570 2023-03-03T07:58:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.521+0000 7fe49719d700 1 -- 172.21.15.98:0/1208740050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4800094a0 con 0x7fe4900cf570 2023-03-03T07:58:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.521+0000 7fe49719d700 1 -- 172.21.15.98:0/1208740050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49015cdf0 con 0x7fe4900cf570 2023-03-03T07:58:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.522+0000 7fe49719d700 1 -- 172.21.15.98:0/1208740050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4900cdd40 con 0x7fe4900cf570 2023-03-03T07:58:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.524+0000 7fe486ffd700 1 -- 172.21.15.98:0/1208740050 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fe48c023070 con 0x7fe4900cf570 2023-03-03T07:58:14.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.525+0000 7fe486ffd700 1 --2- 172.21.15.98:0/1208740050 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe4740683c0 0x7fe47406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:14.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.525+0000 7fe486ffd700 1 -- 172.21.15.98:0/1208740050 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe48c0861a0 con 0x7fe4900cf570 2023-03-03T07:58:14.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.525+0000 7fe49599a700 1 --2- 172.21.15.98:0/1208740050 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe4740683c0 0x7fe47406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:14.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.526+0000 7fe49599a700 1 --2- 172.21.15.98:0/1208740050 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe4740683c0 0x7fe47406a870 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe4800090b0 tx=0x7fe480008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:14.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.529+0000 7fe486ffd700 1 -- 172.21.15.98:0/1208740050 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe48c052840 con 0x7fe4900cf570 2023-03-03T07:58:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:14.772+0000 7fe49719d700 1 -- 172.21.15.98:0/1208740050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe4900cb8a0 con 0x7fe4900cf570 2023-03-03T07:58:15.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.668+0000 7fe486ffd700 1 -- 172.21.15.98:0/1208740050 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v22) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe48c00cea0 con 0x7fe4900cf570 2023-03-03T07:58:15.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.672+0000 7fe49719d700 1 -- 172.21.15.98:0/1208740050 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe4740683c0 msgr2=0x7fe47406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:15.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.672+0000 7fe49719d700 1 --2- 172.21.15.98:0/1208740050 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe4740683c0 0x7fe47406a870 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe4800090b0 tx=0x7fe480008030 comp rx=0 tx=0).stop 2023-03-03T07:58:15.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.672+0000 7fe49719d700 1 -- 172.21.15.98:0/1208740050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4900cf570 msgr2=0x7fe4900c9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:15.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.672+0000 7fe49719d700 1 --2- 172.21.15.98:0/1208740050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4900cf570 0x7fe4900c9350 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe48c00ebe0 tx=0x7fe48c00eef0 comp rx=0 tx=0).stop 2023-03-03T07:58:15.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.672+0000 7fe49719d700 1 -- 172.21.15.98:0/1208740050 shutdown_connections 2023-03-03T07:58:15.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.672+0000 7fe49719d700 1 --2- 172.21.15.98:0/1208740050 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe4740683c0 0x7fe47406a870 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:15.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.672+0000 7fe49719d700 1 --2- 172.21.15.98:0/1208740050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4900cf570 0x7fe4900c9350 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:15.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.672+0000 7fe49719d700 1 --2- 172.21.15.98:0/1208740050 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4901609b0 0x7fe4900b3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:15.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.672+0000 7fe49719d700 1 --2- 172.21.15.98:0/1208740050 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe49015db30 0x7fe4900ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:15.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.672+0000 7fe49719d700 1 -- 172.21.15.98:0/1208740050 >> 172.21.15.98:0/1208740050 conn(0x7fe49006b880 msgr2=0x7fe49015e340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:15.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.672+0000 7fe49719d700 1 -- 172.21.15.98:0/1208740050 shutdown_connections 2023-03-03T07:58:15.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.673+0000 7fe49719d700 1 -- 172.21.15.98:0/1208740050 wait complete. 2023-03-03T07:58:15.705 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-03-03T07:58:15.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-03-03T07:58:15.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.959+0000 7f48be1ff700 1 Processor -- start 2023-03-03T07:58:15.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.959+0000 7f48be1ff700 1 -- start start 2023-03-03T07:58:15.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.960+0000 7f48be1ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f48b815ca70 0x7f48b80cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:15.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.960+0000 7f48be1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48b80d0270 0x7f48b80d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:15.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.961+0000 7f48be1ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b80cf980 con 0x7f48b806b4b0 2023-03-03T07:58:15.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.961+0000 7f48be1ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b80cd5b0 con 0x7f48b815ca70 2023-03-03T07:58:15.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.961+0000 7f48be1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b80d1800 con 0x7f48b80d0270 2023-03-03T07:58:15.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.961+0000 7f48b77fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f48b815ca70 0x7f48b80cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:15.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.961+0000 7f48b6ffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f48b806b4b0 0x7f48b80cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56754/0 (socket says 172.21.15.98:56754) 2023-03-03T07:58:15.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.961+0000 7f48b77fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f48b815ca70 0x7f48b80cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38060/0 (socket says 172.21.15.98:38060) 2023-03-03T07:58:15.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.961+0000 7f48b6ffd700 1 -- 172.21.15.98:0/953534086 learned_addr learned my addr 172.21.15.98:0/953534086 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:15.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.961+0000 7f48b7fff700 1 --2- 172.21.15.98:0/953534086 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48b80d0270 0x7f48b80d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:15.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.961+0000 7f48b77fe700 1 -- 172.21.15.98:0/953534086 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48b80d0270 msgr2=0x7f48b80d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:15.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.961+0000 7f48b77fe700 1 --2- 172.21.15.98:0/953534086 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48b80d0270 0x7f48b80d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:15.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.962+0000 7f48b77fe700 1 -- 172.21.15.98:0/953534086 >> v1:172.21.15.98:6789/0 conn(0x7f48b806b4b0 legacy=0x7f48b80cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:15.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.962+0000 7f48b77fe700 1 -- 172.21.15.98:0/953534086 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b80cd900 con 0x7f48b815ca70 2023-03-03T07:58:15.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.962+0000 7f48b67fc700 1 -- 172.21.15.98:0/953534086 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261062439 0 0) 0x7f48b80cf980 con 0x7f48b806b4b0 2023-03-03T07:58:15.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.962+0000 7f48b77fe700 1 --2- 172.21.15.98:0/953534086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b815ca70 0x7f48b80cf440 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009840 tx=0x7f48a8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda49086078a2f0b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:15.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.962+0000 7f48b67fc700 1 -- 172.21.15.98:0/953534086 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a8011830 con 0x7f48b815ca70 2023-03-03T07:58:15.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.962+0000 7f48b67fc700 1 -- 172.21.15.98:0/953534086 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a8011990 con 0x7f48b815ca70 2023-03-03T07:58:15.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.962+0000 7f48b67fc700 1 -- 172.21.15.98:0/953534086 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a8011d20 con 0x7f48b815ca70 2023-03-03T07:58:15.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.963+0000 7f48be1ff700 1 -- 172.21.15.98:0/953534086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b815ca70 msgr2=0x7f48b80cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:15.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.963+0000 7f48be1ff700 1 --2- 172.21.15.98:0/953534086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b815ca70 0x7f48b80cf440 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009840 tx=0x7f48a8009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:15.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.963+0000 7f48be1ff700 1 -- 172.21.15.98:0/953534086 shutdown_connections 2023-03-03T07:58:15.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.963+0000 7f48be1ff700 1 --2- 172.21.15.98:0/953534086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b815ca70 0x7f48b80cf440 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:15.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.963+0000 7f48be1ff700 1 --2- 172.21.15.98:0/953534086 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48b80d0270 0x7f48b80d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:15.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.963+0000 7f48be1ff700 1 -- 172.21.15.98:0/953534086 >> 172.21.15.98:0/953534086 conn(0x7f48b8073e70 msgr2=0x7f48b806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:15.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.963+0000 7f48be1ff700 1 -- 172.21.15.98:0/953534086 shutdown_connections 2023-03-03T07:58:15.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.963+0000 7f48be1ff700 1 -- 172.21.15.98:0/953534086 wait complete. 2023-03-03T07:58:15.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.965+0000 7f48be1ff700 1 Processor -- start 2023-03-03T07:58:15.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.965+0000 7f48be1ff700 1 -- start start 2023-03-03T07:58:15.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.966+0000 7f48be1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48b806b4b0 0x7f48b814b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:15.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.966+0000 7f48be1ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b80d0270 0x7f48b80caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:15.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.966+0000 7f48be1ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48b815ca70 0x7f48b815f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:15.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.966+0000 7f48be1ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b80caab0 con 0x7f48b80d0270 2023-03-03T07:58:15.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.966+0000 7f48be1ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b81f3440 con 0x7f48b815ca70 2023-03-03T07:58:15.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.966+0000 7f48be1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b81f3160 con 0x7f48b806b4b0 2023-03-03T07:58:15.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.966+0000 7f48b6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b80d0270 0x7f48b80caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:15.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.966+0000 7f48b6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b80d0270 0x7f48b80caed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38070/0 (socket says 172.21.15.98:38070) 2023-03-03T07:58:15.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.966+0000 7f48b6ffd700 1 -- 172.21.15.98:0/3741656527 learned_addr learned my addr 172.21.15.98:0/3741656527 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:15.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.966+0000 7f48b77fe700 1 --2- 172.21.15.98:0/3741656527 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48b806b4b0 0x7f48b814b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:15.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.967+0000 7f48b7fff700 1 --2- 172.21.15.98:0/3741656527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48b815ca70 0x7f48b815f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:15.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.967+0000 7f48b6ffd700 1 -- 172.21.15.98:0/3741656527 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48b806b4b0 msgr2=0x7f48b814b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:15.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.967+0000 7f48b6ffd700 1 --2- 172.21.15.98:0/3741656527 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48b806b4b0 0x7f48b814b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:15.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.967+0000 7f48b6ffd700 1 -- 172.21.15.98:0/3741656527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48b815ca70 msgr2=0x7f48b815f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:15.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.967+0000 7f48b6ffd700 1 --2- 172.21.15.98:0/3741656527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48b815ca70 0x7f48b815f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:15.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.967+0000 7f48b6ffd700 1 -- 172.21.15.98:0/3741656527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a80094a0 con 0x7f48b80d0270 2023-03-03T07:58:15.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.967+0000 7f48b6ffd700 1 --2- 172.21.15.98:0/3741656527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b80d0270 0x7f48b80caed0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00c520 tx=0x7f48ac00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:15.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.967+0000 7f489ffff700 1 -- 172.21.15.98:0/3741656527 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac004c80 con 0x7f48b80d0270 2023-03-03T07:58:15.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.967+0000 7f489ffff700 1 -- 172.21.15.98:0/3741656527 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ac004de0 con 0x7f48b80d0270 2023-03-03T07:58:15.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.968+0000 7f489ffff700 1 -- 172.21.15.98:0/3741656527 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac0072d0 con 0x7f48b80d0270 2023-03-03T07:58:15.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.968+0000 7f48be1ff700 1 -- 172.21.15.98:0/3741656527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b80d1aa0 con 0x7f48b80d0270 2023-03-03T07:58:15.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.968+0000 7f48be1ff700 1 -- 172.21.15.98:0/3741656527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b80cdbe0 con 0x7f48b80d0270 2023-03-03T07:58:15.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.969+0000 7f48be1ff700 1 -- 172.21.15.98:0/3741656527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b815ce40 con 0x7f48b80d0270 2023-03-03T07:58:15.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.970+0000 7f489ffff700 1 -- 172.21.15.98:0/3741656527 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f48ac01a020 con 0x7f48b80d0270 2023-03-03T07:58:15.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.971+0000 7f489ffff700 1 --2- 172.21.15.98:0/3741656527 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48a4068390 0x7f48a406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:15.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.971+0000 7f489ffff700 1 -- 172.21.15.98:0/3741656527 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f48ac085940 con 0x7f48b80d0270 2023-03-03T07:58:15.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.971+0000 7f48b77fe700 1 --2- 172.21.15.98:0/3741656527 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48a4068390 0x7f48a406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:15.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.972+0000 7f48b77fe700 1 --2- 172.21.15.98:0/3741656527 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48a4068390 0x7f48a406a840 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f48a80090b0 tx=0x7f48a8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:15.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:15.975+0000 7f489ffff700 1 -- 172.21.15.98:0/3741656527 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f48ac051fe0 con 0x7f48b80d0270 2023-03-03T07:58:16.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.213+0000 7f48be1ff700 1 -- 172.21.15.98:0/3741656527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f48b80cabf0 con 0x7f48b80d0270 2023-03-03T07:58:16.633 INFO:tasks.ceph.mds.a.smithi098.stderr:2023-03-03T07:58:16.631+0000 7ff317ff1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T07:58:16.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.675+0000 7f489ffff700 1 -- 172.21.15.98:0/3741656527 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v23) v1 ==== 122+0+0 (secure 0 0 0) 0x7f48b80cabf0 con 0x7f48b80d0270 2023-03-03T07:58:16.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.679+0000 7f48be1ff700 1 -- 172.21.15.98:0/3741656527 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48a4068390 msgr2=0x7f48a406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:16.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.679+0000 7f48be1ff700 1 --2- 172.21.15.98:0/3741656527 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48a4068390 0x7f48a406a840 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f48a80090b0 tx=0x7f48a8008030 comp rx=0 tx=0).stop 2023-03-03T07:58:16.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.679+0000 7f48be1ff700 1 -- 172.21.15.98:0/3741656527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b80d0270 msgr2=0x7f48b80caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:16.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.679+0000 7f48be1ff700 1 --2- 172.21.15.98:0/3741656527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b80d0270 0x7f48b80caed0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00c520 tx=0x7f48ac00c830 comp rx=0 tx=0).stop 2023-03-03T07:58:16.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.679+0000 7f48be1ff700 1 -- 172.21.15.98:0/3741656527 shutdown_connections 2023-03-03T07:58:16.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.679+0000 7f48be1ff700 1 --2- 172.21.15.98:0/3741656527 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f48a4068390 0x7f48a406a840 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:16.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.679+0000 7f48be1ff700 1 --2- 172.21.15.98:0/3741656527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48b815ca70 0x7f48b815f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:16.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.679+0000 7f48be1ff700 1 --2- 172.21.15.98:0/3741656527 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48b806b4b0 0x7f48b814b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:16.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.679+0000 7f48be1ff700 1 --2- 172.21.15.98:0/3741656527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b80d0270 0x7f48b80caed0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:16.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.679+0000 7f48be1ff700 1 -- 172.21.15.98:0/3741656527 >> 172.21.15.98:0/3741656527 conn(0x7f48b8073e70 msgr2=0x7f48b80d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:16.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.679+0000 7f48be1ff700 1 -- 172.21.15.98:0/3741656527 shutdown_connections 2023-03-03T07:58:16.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.679+0000 7f48be1ff700 1 -- 172.21.15.98:0/3741656527 wait complete. 2023-03-03T07:58:16.709 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-03-03T07:58:16.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:58:16.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.953+0000 7f05d43ff700 1 Processor -- start 2023-03-03T07:58:16.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.953+0000 7f05d43ff700 1 -- start start 2023-03-03T07:58:16.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.954+0000 7f05d43ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f05cc0b87c0 0x7f05cc0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:16.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.954+0000 7f05d43ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05cc0b5410 0x7f05cc0d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:16.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.954+0000 7f05d43ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05cc0d0ef0 con 0x7f05cc0b8b90 2023-03-03T07:58:16.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.954+0000 7f05d43ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05cc0d0600 con 0x7f05cc0b87c0 2023-03-03T07:58:16.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.954+0000 7f05d43ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc0cd970 con 0x7f05cc0b5410 2023-03-03T07:58:16.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.954+0000 7f05d199a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f05cc0b8b90 0x7f05cc0b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56770/0 (socket says 172.21.15.98:56770) 2023-03-03T07:58:16.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.955+0000 7f05d199a700 1 -- 172.21.15.98:0/1945612108 learned_addr learned my addr 172.21.15.98:0/1945612108 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:16.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.955+0000 7f05d219b700 1 --2- 172.21.15.98:0/1945612108 >> v2:172.21.15.100:3300/0 conn(0x7f05cc0b87c0 0x7f05cc0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:16.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.955+0000 7f05d299c700 1 --2- 172.21.15.98:0/1945612108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05cc0b5410 0x7f05cc0d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:16.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.955+0000 7f05d1199700 1 -- 172.21.15.98:0/1945612108 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240166414 0 0) 0x7f05cc0d0ef0 con 0x7f05cc0b8b90 2023-03-03T07:58:16.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.955+0000 7f05d1199700 1 -- 172.21.15.98:0/1945612108 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05b80035b0 con 0x7f05cc0b8b90 2023-03-03T07:58:16.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.955+0000 7f05d1199700 1 -- 172.21.15.98:0/1945612108 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2459059063 0 0) 0x7f05b80035b0 con 0x7f05cc0b8b90 2023-03-03T07:58:16.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.955+0000 7f05d219b700 1 -- 172.21.15.98:0/1945612108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05cc0b5410 msgr2=0x7f05cc0d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:16.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.955+0000 7f05d219b700 1 --2- 172.21.15.98:0/1945612108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05cc0b5410 0x7f05cc0d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:16.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.955+0000 7f05d219b700 1 -- 172.21.15.98:0/1945612108 >> v1:172.21.15.98:6789/0 conn(0x7f05cc0b8b90 legacy=0x7f05cc0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:16.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.955+0000 7f05d219b700 1 -- 172.21.15.98:0/1945612108 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05cc0cdc50 con 0x7f05cc0b87c0 2023-03-03T07:58:16.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.956+0000 7f05d219b700 1 --2- 172.21.15.98:0/1945612108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b87c0 0x7f05cc0cffe0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f05c8009840 tx=0x7f05c8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eff489cc7f3742e2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:16.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.956+0000 7f05d1199700 1 -- 172.21.15.98:0/1945612108 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05bc001670 con 0x7f05cc0b87c0 2023-03-03T07:58:16.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.956+0000 7f05d1199700 1 -- 172.21.15.98:0/1945612108 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c8004030 con 0x7f05cc0b87c0 2023-03-03T07:58:16.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.956+0000 7f05d1199700 1 -- 172.21.15.98:0/1945612108 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c80043c0 con 0x7f05cc0b87c0 2023-03-03T07:58:16.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.957+0000 7f05d43ff700 1 -- 172.21.15.98:0/1945612108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b87c0 msgr2=0x7f05cc0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:16.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.957+0000 7f05d43ff700 1 --2- 172.21.15.98:0/1945612108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b87c0 0x7f05cc0cffe0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f05c8009840 tx=0x7f05c8009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:16.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.957+0000 7f05d43ff700 1 -- 172.21.15.98:0/1945612108 shutdown_connections 2023-03-03T07:58:16.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.957+0000 7f05d43ff700 1 --2- 172.21.15.98:0/1945612108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b87c0 0x7f05cc0cffe0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:16.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.957+0000 7f05d43ff700 1 --2- 172.21.15.98:0/1945612108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05cc0b5410 0x7f05cc0d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:16.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.957+0000 7f05d43ff700 1 -- 172.21.15.98:0/1945612108 >> 172.21.15.98:0/1945612108 conn(0x7f05cc06aee0 msgr2=0x7f05cc06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:16.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.957+0000 7f05d43ff700 1 -- 172.21.15.98:0/1945612108 shutdown_connections 2023-03-03T07:58:16.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.957+0000 7f05d43ff700 1 -- 172.21.15.98:0/1945612108 wait complete. 2023-03-03T07:58:16.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.959+0000 7f05d43ff700 1 Processor -- start 2023-03-03T07:58:16.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.959+0000 7f05d43ff700 1 -- start start 2023-03-03T07:58:16.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.960+0000 7f05d43ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05cc0b5410 0x7f05cc15e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:16.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.960+0000 7f05d43ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05cc0b87c0 0x7f05cc15bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:16.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.960+0000 7f05d43ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b8b90 0x7f05cc15c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:16.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.960+0000 7f05d43ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc0ab900 con 0x7f05cc0b87c0 2023-03-03T07:58:16.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.960+0000 7f05d43ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc15c690 con 0x7f05cc0b8b90 2023-03-03T07:58:16.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.960+0000 7f05d43ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc15cb80 con 0x7f05cc0b5410 2023-03-03T07:58:16.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.961+0000 7f05d199a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05cc0b87c0 0x7f05cc15bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:16.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.961+0000 7f05d199a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05cc0b87c0 0x7f05cc15bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38072/0 (socket says 172.21.15.98:38072) 2023-03-03T07:58:16.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.961+0000 7f05d199a700 1 -- 172.21.15.98:0/854307547 learned_addr learned my addr 172.21.15.98:0/854307547 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:16.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.961+0000 7f05d219b700 1 --2- 172.21.15.98:0/854307547 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05cc0b5410 0x7f05cc15e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:16.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.961+0000 7f05d299c700 1 --2- 172.21.15.98:0/854307547 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b8b90 0x7f05cc15c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:16.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.961+0000 7f05d199a700 1 -- 172.21.15.98:0/854307547 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05cc0b5410 msgr2=0x7f05cc15e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:16.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.961+0000 7f05d199a700 1 --2- 172.21.15.98:0/854307547 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05cc0b5410 0x7f05cc15e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:16.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.961+0000 7f05d199a700 1 -- 172.21.15.98:0/854307547 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b8b90 msgr2=0x7f05cc15c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:16.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.961+0000 7f05d199a700 1 --2- 172.21.15.98:0/854307547 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b8b90 0x7f05cc15c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:16.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.961+0000 7f05d199a700 1 -- 172.21.15.98:0/854307547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c80094a0 con 0x7f05cc0b87c0 2023-03-03T07:58:16.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.962+0000 7f05d199a700 1 --2- 172.21.15.98:0/854307547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05cc0b87c0 0x7f05cc15bc10 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f05bc00c7b0 tx=0x7f05bc00cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:16.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.962+0000 7f05c2ffd700 1 -- 172.21.15.98:0/854307547 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05bc004c80 con 0x7f05cc0b87c0 2023-03-03T07:58:16.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.962+0000 7f05c2ffd700 1 -- 172.21.15.98:0/854307547 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05bc004de0 con 0x7f05cc0b87c0 2023-03-03T07:58:16.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.962+0000 7f05c2ffd700 1 -- 172.21.15.98:0/854307547 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05bc007310 con 0x7f05cc0b87c0 2023-03-03T07:58:16.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.962+0000 7f05d43ff700 1 -- 172.21.15.98:0/854307547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05cc0cdf30 con 0x7f05cc0b87c0 2023-03-03T07:58:16.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.962+0000 7f05d43ff700 1 -- 172.21.15.98:0/854307547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05cc073a80 con 0x7f05cc0b87c0 2023-03-03T07:58:16.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.963+0000 7f05d43ff700 1 -- 172.21.15.98:0/854307547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05cc0ce0c0 con 0x7f05cc0b87c0 2023-03-03T07:58:16.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.964+0000 7f05c2ffd700 1 -- 172.21.15.98:0/854307547 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f05bc01a020 con 0x7f05cc0b87c0 2023-03-03T07:58:16.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.966+0000 7f05c2ffd700 1 --2- 172.21.15.98:0/854307547 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f05b00683c0 0x7f05b006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:16.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.966+0000 7f05c2ffd700 1 -- 172.21.15.98:0/854307547 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f05bc0859a0 con 0x7f05cc0b87c0 2023-03-03T07:58:16.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.966+0000 7f05d219b700 1 --2- 172.21.15.98:0/854307547 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f05b00683c0 0x7f05b006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:16.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.967+0000 7f05d219b700 1 --2- 172.21.15.98:0/854307547 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f05b00683c0 0x7f05b006a870 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f05c80090b0 tx=0x7f05c8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:16.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:16.969+0000 7f05c2ffd700 1 -- 172.21.15.98:0/854307547 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f05bc052040 con 0x7f05cc0b87c0 2023-03-03T07:58:17.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.205+0000 7f05d43ff700 1 -- 172.21.15.98:0/854307547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05cc0cccd0 con 0x7f05cc0b87c0 2023-03-03T07:58:17.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.205+0000 7f05c2ffd700 1 -- 172.21.15.98:0/854307547 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f05bc007d70 con 0x7f05cc0b87c0 2023-03-03T07:58:17.207 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:17.207 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:58:17.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.208+0000 7f05d43ff700 1 -- 172.21.15.98:0/854307547 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f05b00683c0 msgr2=0x7f05b006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:17.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.208+0000 7f05d43ff700 1 --2- 172.21.15.98:0/854307547 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f05b00683c0 0x7f05b006a870 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f05c80090b0 tx=0x7f05c8008030 comp rx=0 tx=0).stop 2023-03-03T07:58:17.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.208+0000 7f05d43ff700 1 -- 172.21.15.98:0/854307547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05cc0b87c0 msgr2=0x7f05cc15bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:17.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.208+0000 7f05d43ff700 1 --2- 172.21.15.98:0/854307547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05cc0b87c0 0x7f05cc15bc10 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f05bc00c7b0 tx=0x7f05bc00cac0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.208+0000 7f05d43ff700 1 -- 172.21.15.98:0/854307547 shutdown_connections 2023-03-03T07:58:17.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.208+0000 7f05d43ff700 1 --2- 172.21.15.98:0/854307547 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f05b00683c0 0x7f05b006a870 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.208+0000 7f05d43ff700 1 --2- 172.21.15.98:0/854307547 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b8b90 0x7f05cc15c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.208+0000 7f05d43ff700 1 --2- 172.21.15.98:0/854307547 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05cc0b5410 0x7f05cc15e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.208+0000 7f05d43ff700 1 --2- 172.21.15.98:0/854307547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05cc0b87c0 0x7f05cc15bc10 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.208+0000 7f05d43ff700 1 -- 172.21.15.98:0/854307547 >> 172.21.15.98:0/854307547 conn(0x7f05cc06aee0 msgr2=0x7f05cc0b3460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:17.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.208+0000 7f05d43ff700 1 -- 172.21.15.98:0/854307547 shutdown_connections 2023-03-03T07:58:17.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.208+0000 7f05d43ff700 1 -- 172.21.15.98:0/854307547 wait complete. 2023-03-03T07:58:17.211 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2023-03-03T07:58:17.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:58:17.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.481+0000 7f80af7ff700 1 Processor -- start 2023-03-03T07:58:17.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.481+0000 7f80af7ff700 1 -- start start 2023-03-03T07:58:17.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.482+0000 7f80af7ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80a815cc00 0x7f80a80cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:17.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.483+0000 7f80af7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80a80d2c60 0x7f80a80cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:17.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.483+0000 7f80af7ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a80cde10 con 0x7f80a80d27c0 2023-03-03T07:58:17.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.483+0000 7f80af7ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a80ce040 con 0x7f80a815cc00 2023-03-03T07:58:17.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.483+0000 7f80af7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a8073be0 con 0x7f80a80d2c60 2023-03-03T07:58:17.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.483+0000 7f80ad59b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80a815cc00 0x7f80a80cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:17.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.483+0000 7f80ad59b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80a815cc00 0x7f80a80cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38080/0 (socket says 172.21.15.98:38080) 2023-03-03T07:58:17.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.483+0000 7f80ad59b700 1 -- 172.21.15.98:0/3587285429 learned_addr learned my addr 172.21.15.98:0/3587285429 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:17.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.483+0000 7f80acd9a700 1 --1- 172.21.15.98:0/3587285429 >> v1:172.21.15.98:6789/0 conn(0x7f80a80d27c0 0x7f80a80b60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56776/0 (socket says 172.21.15.98:56776) 2023-03-03T07:58:17.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.483+0000 7f80add9c700 1 --2- 172.21.15.98:0/3587285429 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80a80d2c60 0x7f80a80cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:17.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.483+0000 7f80ad59b700 1 -- 172.21.15.98:0/3587285429 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80a80d2c60 msgr2=0x7f80a80cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:17.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.483+0000 7f80ad59b700 1 --2- 172.21.15.98:0/3587285429 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80a80d2c60 0x7f80a80cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.483+0000 7f80ad59b700 1 -- 172.21.15.98:0/3587285429 >> v1:172.21.15.98:6789/0 conn(0x7f80a80d27c0 legacy=0x7f80a80b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:17.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.484+0000 7f80ad59b700 1 -- 172.21.15.98:0/3587285429 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a80d1a00 con 0x7f80a815cc00 2023-03-03T07:58:17.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.484+0000 7f809ffff700 1 -- 172.21.15.98:0/3587285429 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744833192 0 0) 0x7f80a80cde10 con 0x7f80a80d27c0 2023-03-03T07:58:17.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.484+0000 7f80ad59b700 1 --2- 172.21.15.98:0/3587285429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a815cc00 0x7f80a80cd8d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f80a40098f0 tx=0x7f80a4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7bd06a5770df9b6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:17.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.484+0000 7f809ffff700 1 -- 172.21.15.98:0/3587285429 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a4011830 con 0x7f80a815cc00 2023-03-03T07:58:17.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.484+0000 7f809ffff700 1 -- 172.21.15.98:0/3587285429 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a4011990 con 0x7f80a815cc00 2023-03-03T07:58:17.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.484+0000 7f809ffff700 1 -- 172.21.15.98:0/3587285429 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a4011d20 con 0x7f80a815cc00 2023-03-03T07:58:17.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.485+0000 7f80af7ff700 1 -- 172.21.15.98:0/3587285429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a815cc00 msgr2=0x7f80a80cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:17.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.485+0000 7f80af7ff700 1 --2- 172.21.15.98:0/3587285429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a815cc00 0x7f80a80cd8d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f80a40098f0 tx=0x7f80a4009c00 comp rx=0 tx=0).stop 2023-03-03T07:58:17.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.485+0000 7f80af7ff700 1 -- 172.21.15.98:0/3587285429 shutdown_connections 2023-03-03T07:58:17.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.485+0000 7f80af7ff700 1 --2- 172.21.15.98:0/3587285429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a815cc00 0x7f80a80cd8d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.485+0000 7f80af7ff700 1 --2- 172.21.15.98:0/3587285429 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80a80d2c60 0x7f80a80cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.485+0000 7f80af7ff700 1 -- 172.21.15.98:0/3587285429 >> 172.21.15.98:0/3587285429 conn(0x7f80a806aee0 msgr2=0x7f80a806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:17.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.485+0000 7f80af7ff700 1 -- 172.21.15.98:0/3587285429 shutdown_connections 2023-03-03T07:58:17.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.485+0000 7f80af7ff700 1 -- 172.21.15.98:0/3587285429 wait complete. 2023-03-03T07:58:17.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.487+0000 7f80af7ff700 1 Processor -- start 2023-03-03T07:58:17.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.487+0000 7f80af7ff700 1 -- start start 2023-03-03T07:58:17.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.488+0000 7f80af7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a80d27c0 0x7f80a80cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:17.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.488+0000 7f80af7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80a80d2c60 0x7f80a814f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:17.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.488+0000 7f80af7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80a815cc00 0x7f80a80cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:17.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.488+0000 7f80af7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a80b6250 con 0x7f80a80d27c0 2023-03-03T07:58:17.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.488+0000 7f80af7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a80b6530 con 0x7f80a80d2c60 2023-03-03T07:58:17.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.488+0000 7f80af7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a80b6810 con 0x7f80a815cc00 2023-03-03T07:58:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.489+0000 7f80ad59b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a80d27c0 0x7f80a80cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.489+0000 7f80ad59b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a80d27c0 0x7f80a80cb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38094/0 (socket says 172.21.15.98:38094) 2023-03-03T07:58:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.489+0000 7f80ad59b700 1 -- 172.21.15.98:0/253997987 learned_addr learned my addr 172.21.15.98:0/253997987 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.489+0000 7f80acd9a700 1 --2- 172.21.15.98:0/253997987 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80a80d2c60 0x7f80a814f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.489+0000 7f80add9c700 1 --2- 172.21.15.98:0/253997987 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80a815cc00 0x7f80a80cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.489+0000 7f80ad59b700 1 -- 172.21.15.98:0/253997987 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80a815cc00 msgr2=0x7f80a80cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.489+0000 7f80ad59b700 1 --2- 172.21.15.98:0/253997987 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80a815cc00 0x7f80a80cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.489+0000 7f80ad59b700 1 -- 172.21.15.98:0/253997987 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80a80d2c60 msgr2=0x7f80a814f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:17.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.489+0000 7f80ad59b700 1 --2- 172.21.15.98:0/253997987 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80a80d2c60 0x7f80a814f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.489+0000 7f80ad59b700 1 -- 172.21.15.98:0/253997987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a4009550 con 0x7f80a80d27c0 2023-03-03T07:58:17.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.489+0000 7f80ad59b700 1 --2- 172.21.15.98:0/253997987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a80d27c0 0x7f80a80cb5d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f80a4009c40 tx=0x7f80a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:17.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.490+0000 7f809dffb700 1 -- 172.21.15.98:0/253997987 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a4004510 con 0x7f80a80d27c0 2023-03-03T07:58:17.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.490+0000 7f809dffb700 1 -- 172.21.15.98:0/253997987 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a401e070 con 0x7f80a80d27c0 2023-03-03T07:58:17.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.490+0000 7f80af7ff700 1 -- 172.21.15.98:0/253997987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a80d1fc0 con 0x7f80a80d27c0 2023-03-03T07:58:17.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.490+0000 7f809dffb700 1 -- 172.21.15.98:0/253997987 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a40038c0 con 0x7f80a80d27c0 2023-03-03T07:58:17.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.490+0000 7f80af7ff700 1 -- 172.21.15.98:0/253997987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a80d1ce0 con 0x7f80a80d27c0 2023-03-03T07:58:17.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.491+0000 7f80af7ff700 1 -- 172.21.15.98:0/253997987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a80d22a0 con 0x7f80a80d27c0 2023-03-03T07:58:17.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.492+0000 7f809dffb700 1 -- 172.21.15.98:0/253997987 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f80a400f050 con 0x7f80a80d27c0 2023-03-03T07:58:17.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.494+0000 7f809dffb700 1 --2- 172.21.15.98:0/253997987 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f808c0683c0 0x7f808c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:17.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.494+0000 7f809dffb700 1 -- 172.21.15.98:0/253997987 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f80a4086ca0 con 0x7f80a80d27c0 2023-03-03T07:58:17.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.494+0000 7f80acd9a700 1 --2- 172.21.15.98:0/253997987 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f808c0683c0 0x7f808c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:17.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.495+0000 7f80acd9a700 1 --2- 172.21.15.98:0/253997987 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f808c0683c0 0x7f808c06a870 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f80a815f210 tx=0x7f8098007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:17.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.498+0000 7f809dffb700 1 -- 172.21.15.98:0/253997987 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f80a408b070 con 0x7f80a80d27c0 2023-03-03T07:58:17.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.693+0000 7f80af7ff700 1 -- 172.21.15.98:0/253997987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80a80729b0 con 0x7f80a80d27c0 2023-03-03T07:58:17.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.694+0000 7f809dffb700 1 -- 172.21.15.98:0/253997987 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+17979 (secure 0 0 0) 0x7f80a4020090 con 0x7f80a80d27c0 2023-03-03T07:58:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:17.697 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":52,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:58:14.216922+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:58:10.728830+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-03-03T07:58:12.224649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:58:17.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.697+0000 7f80af7ff700 1 -- 172.21.15.98:0/253997987 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f808c0683c0 msgr2=0x7f808c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:17.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.697+0000 7f80af7ff700 1 --2- 172.21.15.98:0/253997987 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f808c0683c0 0x7f808c06a870 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f80a815f210 tx=0x7f8098007de0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.697+0000 7f80af7ff700 1 -- 172.21.15.98:0/253997987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a80d27c0 msgr2=0x7f80a80cb5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:17.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.697+0000 7f80af7ff700 1 --2- 172.21.15.98:0/253997987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a80d27c0 0x7f80a80cb5d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f80a4009c40 tx=0x7f80a4004750 comp rx=0 tx=0).stop 2023-03-03T07:58:17.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.697+0000 7f80af7ff700 1 -- 172.21.15.98:0/253997987 shutdown_connections 2023-03-03T07:58:17.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.697+0000 7f80af7ff700 1 --2- 172.21.15.98:0/253997987 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f808c0683c0 0x7f808c06a870 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.697+0000 7f80af7ff700 1 --2- 172.21.15.98:0/253997987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a80d27c0 0x7f80a80cb5d0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.697+0000 7f80af7ff700 1 --2- 172.21.15.98:0/253997987 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80a815cc00 0x7f80a80cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.697+0000 7f80af7ff700 1 --2- 172.21.15.98:0/253997987 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80a80d2c60 0x7f80a814f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.697+0000 7f80af7ff700 1 -- 172.21.15.98:0/253997987 >> 172.21.15.98:0/253997987 conn(0x7f80a806aee0 msgr2=0x7f80a80b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.697+0000 7f80af7ff700 1 -- 172.21.15.98:0/253997987 shutdown_connections 2023-03-03T07:58:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.697+0000 7f80af7ff700 1 -- 172.21.15.98:0/253997987 wait complete. 2023-03-03T07:58:17.726 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-03-03T07:58:17.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T07:58:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.970+0000 7efc4a19d700 1 Processor -- start 2023-03-03T07:58:17.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.970+0000 7efc4a19d700 1 -- start start 2023-03-03T07:58:17.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.972+0000 7efc4a19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc440b51c0 0x7efc44160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:17.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.972+0000 7efc4a19d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efc440d1580 0x7efc440d19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:17.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.972+0000 7efc4a19d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc441614c0 con 0x7efc44146360 2023-03-03T07:58:17.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.972+0000 7efc4a19d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc440d10f0 con 0x7efc440d1580 2023-03-03T07:58:17.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.972+0000 7efc4a19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc440cdf20 con 0x7efc440b51c0 2023-03-03T07:58:17.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.972+0000 7efc4919b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efc44146360 0x7efc440b4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56778/0 (socket says 172.21.15.98:56778) 2023-03-03T07:58:17.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.972+0000 7efc4919b700 1 -- 172.21.15.98:0/4256573756 learned_addr learned my addr 172.21.15.98:0/4256573756 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:17.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.972+0000 7efc4899a700 1 --2- 172.21.15.98:0/4256573756 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc440b51c0 0x7efc44160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:17.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.972+0000 7efc4999c700 1 --2- 172.21.15.98:0/4256573756 >> v2:172.21.15.100:3300/0 conn(0x7efc440d1580 0x7efc440d19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.973+0000 7efc3bfff700 1 -- 172.21.15.98:0/4256573756 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484050425 0 0) 0x7efc40001110 con 0x7efc44146360 2023-03-03T07:58:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.973+0000 7efc3bfff700 1 -- 172.21.15.98:0/4256573756 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc300035b0 con 0x7efc44146360 2023-03-03T07:58:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.973+0000 7efc4899a700 1 -- 172.21.15.98:0/4256573756 >> v2:172.21.15.100:3300/0 conn(0x7efc440d1580 msgr2=0x7efc440d19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.973+0000 7efc4899a700 1 --2- 172.21.15.98:0/4256573756 >> v2:172.21.15.100:3300/0 conn(0x7efc440d1580 0x7efc440d19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.973+0000 7efc4899a700 1 -- 172.21.15.98:0/4256573756 >> v1:172.21.15.98:6789/0 conn(0x7efc44146360 legacy=0x7efc440b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.973+0000 7efc4899a700 1 -- 172.21.15.98:0/4256573756 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc440cd680 con 0x7efc440b51c0 2023-03-03T07:58:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.973+0000 7efc4899a700 1 --2- 172.21.15.98:0/4256573756 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc440b51c0 0x7efc44160f80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efc34009840 tx=0x7efc34009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd3ee0b6e12d6935 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:17.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.973+0000 7efc3bfff700 1 -- 172.21.15.98:0/4256573756 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34011830 con 0x7efc440b51c0 2023-03-03T07:58:17.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.974+0000 7efc3bfff700 1 -- 172.21.15.98:0/4256573756 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc34011990 con 0x7efc440b51c0 2023-03-03T07:58:17.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.974+0000 7efc3bfff700 1 -- 172.21.15.98:0/4256573756 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34011d20 con 0x7efc440b51c0 2023-03-03T07:58:17.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.974+0000 7efc4a19d700 1 -- 172.21.15.98:0/4256573756 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc440b51c0 msgr2=0x7efc44160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:17.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.974+0000 7efc4a19d700 1 --2- 172.21.15.98:0/4256573756 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc440b51c0 0x7efc44160f80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efc34009840 tx=0x7efc34009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:17.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.974+0000 7efc4a19d700 1 -- 172.21.15.98:0/4256573756 shutdown_connections 2023-03-03T07:58:17.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.974+0000 7efc4a19d700 1 --2- 172.21.15.98:0/4256573756 >> v2:172.21.15.100:3300/0 conn(0x7efc440d1580 0x7efc440d19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.974+0000 7efc4a19d700 1 --2- 172.21.15.98:0/4256573756 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc440b51c0 0x7efc44160f80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.974+0000 7efc4a19d700 1 -- 172.21.15.98:0/4256573756 >> 172.21.15.98:0/4256573756 conn(0x7efc4406b8e0 msgr2=0x7efc440cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:17.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.974+0000 7efc4a19d700 1 -- 172.21.15.98:0/4256573756 shutdown_connections 2023-03-03T07:58:17.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.974+0000 7efc4a19d700 1 -- 172.21.15.98:0/4256573756 wait complete. 2023-03-03T07:58:17.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.977+0000 7efc4a19d700 1 Processor -- start 2023-03-03T07:58:17.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.977+0000 7efc4a19d700 1 -- start start 2023-03-03T07:58:17.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.979+0000 7efc4a19d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440b51c0 0x7efc4415e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:17.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.979+0000 7efc4a19d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc440d1580 0x7efc4415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:17.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.979+0000 7efc4a19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc44146360 0x7efc44149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:17.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.979+0000 7efc4a19d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4415fc00 con 0x7efc440b51c0 2023-03-03T07:58:17.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.979+0000 7efc4a19d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4415eff0 con 0x7efc440d1580 2023-03-03T07:58:17.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.979+0000 7efc4a19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc4415f2d0 con 0x7efc44146360 2023-03-03T07:58:17.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.979+0000 7efc4919b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440b51c0 0x7efc4415e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:17.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.979+0000 7efc4919b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440b51c0 0x7efc4415e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38110/0 (socket says 172.21.15.98:38110) 2023-03-03T07:58:17.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.979+0000 7efc4919b700 1 -- 172.21.15.98:0/2276473318 learned_addr learned my addr 172.21.15.98:0/2276473318 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:17.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.979+0000 7efc4899a700 1 --2- 172.21.15.98:0/2276473318 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc440d1580 0x7efc4415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:17.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.979+0000 7efc4999c700 1 --2- 172.21.15.98:0/2276473318 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc44146360 0x7efc44149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:17.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.979+0000 7efc4919b700 1 -- 172.21.15.98:0/2276473318 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc44146360 msgr2=0x7efc44149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:17.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.979+0000 7efc4919b700 1 --2- 172.21.15.98:0/2276473318 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc44146360 0x7efc44149f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.979+0000 7efc4919b700 1 -- 172.21.15.98:0/2276473318 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc440d1580 msgr2=0x7efc4415cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.980+0000 7efc4919b700 1 --2- 172.21.15.98:0/2276473318 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc440d1580 0x7efc4415cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.980+0000 7efc4919b700 1 -- 172.21.15.98:0/2276473318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc340094a0 con 0x7efc440b51c0 2023-03-03T07:58:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.980+0000 7efc4919b700 1 --2- 172.21.15.98:0/2276473318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440b51c0 0x7efc4415e910 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7efc40007fd0 tx=0x7efc4000ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.980+0000 7efc39ffb700 1 -- 172.21.15.98:0/2276473318 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc40005d70 con 0x7efc440b51c0 2023-03-03T07:58:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.980+0000 7efc39ffb700 1 -- 172.21.15.98:0/2276473318 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc400070d0 con 0x7efc440b51c0 2023-03-03T07:58:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.980+0000 7efc4a19d700 1 -- 172.21.15.98:0/2276473318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc440cdc40 con 0x7efc440b51c0 2023-03-03T07:58:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.980+0000 7efc39ffb700 1 -- 172.21.15.98:0/2276473318 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc40007460 con 0x7efc440b51c0 2023-03-03T07:58:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.980+0000 7efc4a19d700 1 -- 172.21.15.98:0/2276473318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc440cd960 con 0x7efc440b51c0 2023-03-03T07:58:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.982+0000 7efc4a19d700 1 -- 172.21.15.98:0/2276473318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc440d0d80 con 0x7efc440b51c0 2023-03-03T07:58:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.983+0000 7efc39ffb700 1 -- 172.21.15.98:0/2276473318 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7efc40019030 con 0x7efc440b51c0 2023-03-03T07:58:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.984+0000 7efc39ffb700 1 --2- 172.21.15.98:0/2276473318 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc280683c0 0x7efc2806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.984+0000 7efc39ffb700 1 -- 172.21.15.98:0/2276473318 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6062+0+0 (secure 0 0 0) 0x7efc40085260 con 0x7efc440b51c0 2023-03-03T07:58:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.984+0000 7efc4899a700 1 --2- 172.21.15.98:0/2276473318 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc280683c0 0x7efc2806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.985+0000 7efc4899a700 1 --2- 172.21.15.98:0/2276473318 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc280683c0 0x7efc2806a870 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7efc440b55b0 tx=0x7efc3401c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:17.988+0000 7efc39ffb700 1 -- 172.21.15.98:0/2276473318 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efc40051880 con 0x7efc440b51c0 2023-03-03T07:58:18.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.177+0000 7efc4a19d700 1 -- 172.21.15.98:0/2276473318 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efc441614c0 con 0x7efc280683c0 2023-03-03T07:58:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.188+0000 7efc39ffb700 1 -- 172.21.15.98:0/2276473318 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+299938 (secure 0 0 0) 0x7efc441614c0 con 0x7efc280683c0 2023-03-03T07:58:18.193 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:18.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.193+0000 7efc4a19d700 1 -- 172.21.15.98:0/2276473318 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc280683c0 msgr2=0x7efc2806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:18.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.193+0000 7efc4a19d700 1 --2- 172.21.15.98:0/2276473318 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc280683c0 0x7efc2806a870 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7efc440b55b0 tx=0x7efc3401c040 comp rx=0 tx=0).stop 2023-03-03T07:58:18.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.193+0000 7efc4a19d700 1 -- 172.21.15.98:0/2276473318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440b51c0 msgr2=0x7efc4415e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:18.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.193+0000 7efc4a19d700 1 --2- 172.21.15.98:0/2276473318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440b51c0 0x7efc4415e910 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7efc40007fd0 tx=0x7efc4000ca90 comp rx=0 tx=0).stop 2023-03-03T07:58:18.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.193+0000 7efc4a19d700 1 -- 172.21.15.98:0/2276473318 shutdown_connections 2023-03-03T07:58:18.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.193+0000 7efc4a19d700 1 --2- 172.21.15.98:0/2276473318 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efc280683c0 0x7efc2806a870 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:18.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.193+0000 7efc4a19d700 1 --2- 172.21.15.98:0/2276473318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440b51c0 0x7efc4415e910 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:18.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.193+0000 7efc4a19d700 1 --2- 172.21.15.98:0/2276473318 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc44146360 0x7efc44149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:18.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.193+0000 7efc4a19d700 1 --2- 172.21.15.98:0/2276473318 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc440d1580 0x7efc4415cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:18.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.193+0000 7efc4a19d700 1 -- 172.21.15.98:0/2276473318 >> 172.21.15.98:0/2276473318 conn(0x7efc4406b8e0 msgr2=0x7efc4415b330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:18.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.193+0000 7efc4a19d700 1 -- 172.21.15.98:0/2276473318 shutdown_connections 2023-03-03T07:58:18.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.193+0000 7efc4a19d700 1 -- 172.21.15.98:0/2276473318 wait complete. 2023-03-03T07:58:18.197 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T07:58:18.223 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2023-03-03T07:58:16.208984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":73088,"kb_used_data":5632,"kb_used_omap":0,"kb_used_meta":67392,"kb_avail":754901632,"statfs":{"total":773094113280,"available":773019271168,"internally_reserved":0,"allocated":5767168,"data_stored":3131688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":69009408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.557726"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.657525+0000","last_change":"2023-03-03T07:58:11.657525+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.657525+0000","last_undegraded":"2023-03-03T07:58:11.657525+0000","last_fullsized":"2023-03-03T07:58:11.657525+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:45:11.942416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.660309+0000","last_change":"2023-03-03T07:58:11.660309+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.660309+0000","last_undegraded":"2023-03-03T07:58:11.660309+0000","last_fullsized":"2023-03-03T07:58:11.660309+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:48.066944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.659463+0000","last_change":"2023-03-03T07:58:11.659463+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.659463+0000","last_undegraded":"2023-03-03T07:58:11.659463+0000","last_fullsized":"2023-03-03T07:58:11.659463+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:20.568387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.660692+0000","last_change":"2023-03-03T07:58:11.660692+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.660692+0000","last_undegraded":"2023-03-03T07:58:11.660692+0000","last_fullsized":"2023-03-03T07:58:11.660692+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:05:05.629228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.654886+0000","last_change":"2023-03-03T07:58:11.654886+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.654886+0000","last_undegraded":"2023-03-03T07:58:11.654886+0000","last_fullsized":"2023-03-03T07:58:11.654886+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:29:58.100844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.654139+0000","last_change":"2023-03-03T07:58:11.654139+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.654139+0000","last_undegraded":"2023-03-03T07:58:11.654139+0000","last_fullsized":"2023-03-03T07:58:11.654139+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:29:58.100844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.656860+0000","last_change":"2023-03-03T07:58:11.656860+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.656860+0000","last_undegraded":"2023-03-03T07:58:11.656860+0000","last_fullsized":"2023-03-03T07:58:11.656860+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:51:36.043610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.654785+0000","last_change":"2023-03-03T07:58:11.654785+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.654785+0000","last_undegraded":"2023-03-03T07:58:11.654785+0000","last_fullsized":"2023-03-03T07:58:11.654785+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:16:02.075425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.657247+0000","last_change":"2023-03-03T07:58:11.657247+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.657247+0000","last_undegraded":"2023-03-03T07:58:11.657247+0000","last_fullsized":"2023-03-03T07:58:11.657247+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:05:05.629228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.654134+0000","last_change":"2023-03-03T07:58:11.654134+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.654134+0000","last_undegraded":"2023-03-03T07:58:11.654134+0000","last_fullsized":"2023-03-03T07:58:11.654134+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:46:18.685833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.657879+0000","last_change":"2023-03-03T07:58:11.657879+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.657879+0000","last_undegraded":"2023-03-03T07:58:11.657879+0000","last_fullsized":"2023-03-03T07:58:11.657879+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:25.389749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.659366+0000","last_change":"2023-03-03T07:58:11.659366+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.659366+0000","last_undegraded":"2023-03-03T07:58:11.659366+0000","last_fullsized":"2023-03-03T07:58:11.659366+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:27:33.499514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.654202+0000","last_change":"2023-03-03T07:58:11.654202+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.654202+0000","last_undegraded":"2023-03-03T07:58:11.654202+0000","last_fullsized":"2023-03-03T07:58:11.654202+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:58.525319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.656935+0000","last_change":"2023-03-03T07:58:11.656935+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.656935+0000","last_undegraded":"2023-03-03T07:58:11.656935+0000","last_fullsized":"2023-03-03T07:58:11.656935+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:15:45.710069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.657147+0000","last_change":"2023-03-03T07:58:11.657147+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.657147+0000","last_undegraded":"2023-03-03T07:58:11.657147+0000","last_fullsized":"2023-03-03T07:58:11.657147+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:40:17.732316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.657850+0000","last_change":"2023-03-03T07:58:11.657850+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.657850+0000","last_undegraded":"2023-03-03T07:58:11.657850+0000","last_fullsized":"2023-03-03T07:58:11.657850+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:46:18.685833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.654332+0000","last_change":"2023-03-03T07:58:11.654332+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.654332+0000","last_undegraded":"2023-03-03T07:58:11.654332+0000","last_fullsized":"2023-03-03T07:58:11.654332+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:46:18.685833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.658923+0000","last_change":"2023-03-03T07:58:11.658923+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.658923+0000","last_undegraded":"2023-03-03T07:58:11.658923+0000","last_fullsized":"2023-03-03T07:58:11.658923+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:34.055432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.657400+0000","last_change":"2023-03-03T07:58:11.657400+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.657400+0000","last_undegraded":"2023-03-03T07:58:11.657400+0000","last_fullsized":"2023-03-03T07:58:11.657400+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:19:51.509265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.656924+0000","last_change":"2023-03-03T07:58:11.656924+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.656924+0000","last_undegraded":"2023-03-03T07:58:11.656924+0000","last_fullsized":"2023-03-03T07:58:11.656924+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:40:45.616669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.659188+0000","last_change":"2023-03-03T07:58:11.659188+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.659188+0000","last_undegraded":"2023-03-03T07:58:11.659188+0000","last_fullsized":"2023-03-03T07:58:11.659188+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:04:23.831200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.658852+0000","last_change":"2023-03-03T07:58:11.658852+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.658852+0000","last_undegraded":"2023-03-03T07:58:11.658852+0000","last_fullsized":"2023-03-03T07:58:11.658852+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:39:43.086368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.654206+0000","last_change":"2023-03-03T07:58:11.654206+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.654206+0000","last_undegraded":"2023-03-03T07:58:11.654206+0000","last_fullsized":"2023-03-03T07:58:11.654206+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:27:33.499514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.658549+0000","last_change":"2023-03-03T07:58:11.658549+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.658549+0000","last_undegraded":"2023-03-03T07:58:11.658549+0000","last_fullsized":"2023-03-03T07:58:11.658549+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:47:14.909630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.654134+0000","last_change":"2023-03-03T07:58:11.654134+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.654134+0000","last_undegraded":"2023-03-03T07:58:11.654134+0000","last_fullsized":"2023-03-03T07:58:11.654134+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:27:33.499514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-03-03T07:58:07.635522+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T07:58:07.635522+0000","last_peered":"2023-03-03T07:58:07.635522+0000","last_clean":"2023-03-03T07:58:07.635522+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T07:58:07.635522+0000","last_undegraded":"2023-03-03T07:58:07.635522+0000","last_fullsized":"2023-03-03T07:58:07.635522+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.657091+0000","last_change":"2023-03-03T07:58:11.657091+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.657091+0000","last_undegraded":"2023-03-03T07:58:11.657091+0000","last_fullsized":"2023-03-03T07:58:11.657091+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:11.596647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-03-03T07:58:08.150445+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T07:58:08.150445+0000","last_peered":"2023-03-03T07:58:08.150445+0000","last_clean":"2023-03-03T07:58:08.150445+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T07:58:08.150445+0000","last_undegraded":"2023-03-03T07:58:08.150445+0000","last_fullsized":"2023-03-03T07:58:08.150445+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.654606+0000","last_change":"2023-03-03T07:58:11.654606+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.654606+0000","last_undegraded":"2023-03-03T07:58:11.654606+0000","last_fullsized":"2023-03-03T07:58:11.654606+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:35:13.361074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":33,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-03-03T07:58:08.150009+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T07:58:08.150009+0000","last_peered":"2023-03-03T07:58:08.150009+0000","last_clean":"2023-03-03T07:58:08.150009+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T07:58:08.150009+0000","last_undegraded":"2023-03-03T07:58:08.150009+0000","last_fullsized":"2023-03-03T07:58:08.150009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.654123+0000","last_change":"2023-03-03T07:58:11.654123+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.654123+0000","last_undegraded":"2023-03-03T07:58:11.654123+0000","last_fullsized":"2023-03-03T07:58:11.654123+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:28.227840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-03-03T07:58:08.150413+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T07:58:08.150413+0000","last_peered":"2023-03-03T07:58:08.150413+0000","last_clean":"2023-03-03T07:58:08.150413+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T07:58:08.150413+0000","last_undegraded":"2023-03-03T07:58:08.150413+0000","last_fullsized":"2023-03-03T07:58:08.150413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":188,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-03-03T07:58:08.150204+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T07:58:08.150204+0000","last_peered":"2023-03-03T07:58:08.150204+0000","last_clean":"2023-03-03T07:58:08.150204+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T07:58:08.150204+0000","last_undegraded":"2023-03-03T07:58:08.150204+0000","last_fullsized":"2023-03-03T07:58:08.150204+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.657393+0000","last_change":"2023-03-03T07:58:11.657393+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.657393+0000","last_undegraded":"2023-03-03T07:58:11.657393+0000","last_fullsized":"2023-03-03T07:58:11.657393+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:03:44.817102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":35,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-03-03T07:58:08.150203+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T07:58:08.150203+0000","last_peered":"2023-03-03T07:58:08.150203+0000","last_clean":"2023-03-03T07:58:08.150203+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T07:58:08.150203+0000","last_undegraded":"2023-03-03T07:58:08.150203+0000","last_fullsized":"2023-03-03T07:58:08.150203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.660670+0000","last_change":"2023-03-03T07:58:11.660670+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.660670+0000","last_undegraded":"2023-03-03T07:58:11.660670+0000","last_fullsized":"2023-03-03T07:58:11.660670+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:52:22.004977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":35,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-03-03T07:58:08.150223+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T07:58:08.150223+0000","last_peered":"2023-03-03T07:58:08.150223+0000","last_clean":"2023-03-03T07:58:08.150223+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T07:58:08.150223+0000","last_undegraded":"2023-03-03T07:58:08.150223+0000","last_fullsized":"2023-03-03T07:58:08.150223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.657923+0000","last_change":"2023-03-03T07:58:11.657923+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.657923+0000","last_undegraded":"2023-03-03T07:58:11.657923+0000","last_fullsized":"2023-03-03T07:58:11.657923+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:58.525319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-03-03T07:58:07.635543+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T07:58:07.635543+0000","last_peered":"2023-03-03T07:58:07.635543+0000","last_clean":"2023-03-03T07:58:07.635543+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T07:58:07.635543+0000","last_undegraded":"2023-03-03T07:58:07.635543+0000","last_fullsized":"2023-03-03T07:58:07.635543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T07:58:12.651095+0000","last_change":"2023-03-03T07:58:12.651095+0000","last_active":"2023-03-03T07:58:12.651095+0000","last_peered":"2023-03-03T07:58:12.651095+0000","last_clean":"2023-03-03T07:58:12.651095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:12.651095+0000","last_undegraded":"2023-03-03T07:58:12.651095+0000","last_fullsized":"2023-03-03T07:58:12.651095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2023-03-03T07:58:11.658193+0000","last_change":"2023-03-03T07:58:11.658193+0000","last_active":"2023-03-03T07:58:11.643076+0000","last_peered":"2023-03-03T07:58:11.643076+0000","last_clean":"2023-03-03T07:58:11.643076+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T07:58:11.658193+0000","last_undegraded":"2023-03-03T07:58:11.658193+0000","last_fullsized":"2023-03-03T07:58:11.658193+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:29:58.100844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-03-03T07:58:08.150946+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T07:58:08.150946+0000","last_peered":"2023-03-03T07:58:08.150946+0000","last_clean":"2023-03-03T07:58:08.150946+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T07:58:08.150946+0000","last_undegraded":"2023-03-03T07:58:08.150946+0000","last_fullsized":"2023-03-03T07:58:08.150946+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10112,"kb_used_data":1144,"kb_used_omap":0,"kb_used_meta":8960,"kb_avail":94361728,"statfs":{"total":96636764160,"available":96626409472,"internally_reserved":0,"allocated":1171456,"data_stored":836289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9175040},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.70299999999999996,"15min":0.71499999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88200000000000001,"5min":1.546,"15min":1.546},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.72899999999999998,"15min":0.73499999999999999},"min":{"1min":0.54900000000000004,"5min":0.5,"15min":0.5},"max":{"1min":0.89400000000000002,"5min":1.702,"15min":1.702},"last":0.80300000000000005}]},{"osd":1,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":1.073,"15min":1.145},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.873,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.77300000000000002,"15min":0.78000000000000003},"min":{"1min":0.63700000000000001,"5min":0.504,"15min":0.504},"max":{"1min":0.92500000000000004,"5min":1.944,"15min":1.944},"last":0.90300000000000002}]},{"osd":2,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.98599999999999999,"5min":1.252,"15min":1.252},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.79200000000000004,"15min":0.80100000000000005},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.016,"5min":1.329,"15min":1.329},"last":0.85199999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.71099999999999997,"15min":0.71199999999999997},"min":{"1min":0.53500000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.92500000000000004,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.78600000000000003,"15min":0.80000000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0640000000000001,"5min":1.294,"15min":1.294},"last":0.91600000000000004}]},{"osd":4,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.66700000000000004,"15min":0.66300000000000003},"min":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92700000000000005,"5min":1.077,"15min":1.077},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.63400000000000001,"15min":0.623},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.624}]},{"osd":5,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.79500000000000004,"15min":0.80900000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0409999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.82899999999999996,"15min":0.82799999999999996},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.0940000000000001,"5min":1.349,"15min":1.349},"last":0.872}]},{"osd":6,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.83699999999999997,"15min":0.83599999999999997},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.048,"5min":1.204,"15min":1.204},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.81399999999999995,"15min":0.82199999999999995},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0169999999999999,"5min":1.194,"15min":1.194},"last":0.88500000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8696,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94363144,"statfs":{"total":96636764160,"available":96627859456,"internally_reserved":0,"allocated":573440,"data_stored":244902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.68700000000000006,"15min":0.68200000000000005},"min":{"1min":0.42899999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.69399999999999995,"15min":0.69599999999999995},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96799999999999997,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.50700000000000001}]},{"osd":1,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.69899999999999995,"15min":0.68200000000000005},"min":{"1min":0.47699999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":3.323,"5min":3.323,"15min":3.323},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.046,"5min":1.135,"15min":1.135},"last":0.58599999999999997}]},{"osd":2,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.73999999999999999,"15min":0.72799999999999998},"min":{"1min":0.57399999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.54800000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.60599999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.82299999999999995,"15min":0.79300000000000004},"min":{"1min":0.57099999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":4.8959999999999999,"5min":4.8959999999999999,"15min":4.8959999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.77500000000000002,"15min":0.77200000000000002},"min":{"1min":0.61099999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.2270000000000001,"5min":1.343,"15min":1.343},"last":0.73699999999999999}]},{"osd":4,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.92900000000000005,"15min":0.95499999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.087,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.93899999999999995,"15min":0.96799999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92100000000000004,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.63300000000000001}]},{"osd":5,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.78300000000000003,"15min":0.79500000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.86299999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.83299999999999996,"15min":0.83499999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.1279999999999999,"5min":1.429,"15min":1.429},"last":0.70899999999999996}]},{"osd":7,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.85399999999999998,"15min":0.85099999999999998},"min":{"1min":0.67000000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.093,"5min":1.452,"15min":1.452},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.877,"15min":0.88900000000000001},"min":{"1min":0.59099999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.135,"5min":1.579,"15min":1.579},"last":0.79300000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8816,"kb_used_data":552,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363024,"statfs":{"total":96636764160,"available":96627736576,"internally_reserved":0,"allocated":565248,"data_stored":239751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.505,"15min":0.51500000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.59399999999999997,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.371},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.63700000000000001,"15min":0.66600000000000004},"min":{"1min":0.36499999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.60799999999999998,"5min":3.972,"15min":3.972},"last":0.505}]},{"osd":2,"last update":"Fri Mar 3 07:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.95899999999999996,"15min":1.0369999999999999},"min":{"1min":0.34899999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.621,"5min":11.897,"15min":11.897},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":1.0409999999999999,"15min":1.119},"min":{"1min":0.495,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.69999999999999996,"5min":11.804,"15min":11.804},"last":0.59399999999999997}]},{"osd":3,"last update":"Fri Mar 3 07:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":1.032,"15min":1.109},"min":{"1min":0.46300000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.70899999999999996,"5min":11.843999999999999,"15min":11.843999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.72699999999999998,"15min":0.76300000000000001},"min":{"1min":0.40100000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.64100000000000001,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.55400000000000005}]},{"osd":4,"last update":"Fri Mar 3 07:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.86399999999999999,"15min":0.878},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1459999999999999,"5min":1.613,"15min":1.613},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.86099999999999999,"15min":0.86899999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.123,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.94599999999999995}]},{"osd":5,"last update":"Fri Mar 3 07:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.92500000000000004,"15min":0.95099999999999996},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":0.94599999999999995,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.873,"15min":0.89100000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0960000000000001,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.95599999999999996}]},{"osd":6,"last update":"Fri Mar 3 07:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.996,"15min":1.0269999999999999},"min":{"1min":0.64300000000000002,"5min":0.628,"15min":0.628},"max":{"1min":1.028,"5min":3.907,"15min":3.907},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.94199999999999995,"15min":0.95999999999999996},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.109,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.93100000000000005}]},{"osd":7,"last update":"Fri Mar 3 07:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.88100000000000001,"15min":0.89500000000000002},"min":{"1min":0.624,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.0069999999999999,"5min":1.419,"15min":1.419},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.76300000000000001,"5min":1.0049999999999999,"15min":1.0449999999999999},"min":{"1min":0.623,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":0.95699999999999996,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":0.83999999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9852,"kb_used_data":1140,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94361988,"statfs":{"total":96636764160,"available":96626675712,"internally_reserved":0,"allocated":1167360,"data_stored":835270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46700000000000003,"15min":0.47499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.60499999999999998,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.51100000000000001,"15min":0.51900000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.61899999999999999,"5min":0.871,"15min":0.871},"last":0.52500000000000002}]},{"osd":2,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.41899999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67300000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.40400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69599999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.58199999999999996}]},{"osd":3,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.68000000000000005,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.63800000000000001,"15min":0.64800000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78500000000000003,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.51000000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.84999999999999998,"15min":0.85399999999999998},"min":{"1min":0.64600000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.99299999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.58199999999999996,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.022,"5min":1.605,"15min":1.605},"last":0.80800000000000005}]},{"osd":5,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82899999999999996,"15min":0.82999999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.081,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.82999999999999996,"15min":0.82799999999999996},"min":{"1min":0.68799999999999994,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.0880000000000001,"5min":1.286,"15min":1.286},"last":0.97799999999999998}]},{"osd":6,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.88900000000000001,"15min":0.89700000000000002},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.0069999999999999,"5min":1.623,"15min":1.623},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.87,"15min":0.877},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.127,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.83999999999999997}]},{"osd":7,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.88600000000000001,"15min":0.89600000000000002},"min":{"1min":0.57599999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.0569999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.86699999999999999,"15min":0.871},"min":{"1min":0.67700000000000005,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.071,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.83199999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8684,"kb_used_data":548,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94363156,"statfs":{"total":96636764160,"available":96627871744,"internally_reserved":0,"allocated":561152,"data_stored":238732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:57:31 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.87,"15min":0.94399999999999995},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.59799999999999998,"5min":5.7889999999999997,"15min":5.7889999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.72899999999999998,"15min":0.77200000000000002},"min":{"1min":0.374,"5min":0.372,"15min":0.372},"max":{"1min":0.63500000000000001,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":0.41999999999999998}]},{"osd":1,"last update":"Fri Mar 3 07:57:31 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.753,"15min":0.79400000000000004},"min":{"1min":0.34499999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.61099999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.002,"15min":1.0780000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.66100000000000003,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.55000000000000004}]},{"osd":3,"last update":"Fri Mar 3 07:57:31 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.2130000000000001,"15min":1.3300000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70199999999999996,"5min":12.961,"15min":12.961},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":1.028,"15min":1.1120000000000001},"min":{"1min":0.438,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.58899999999999997,"5min":6.1449999999999996,"15min":6.1449999999999996},"last":0.68700000000000006}]},{"osd":4,"last update":"Fri Mar 3 07:57:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.2769999999999999,"15min":1.353},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":0.93100000000000005,"5min":6.484,"15min":6.484},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.2529999999999999,"15min":1.3200000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":0.98499999999999999,"5min":6.5510000000000002,"15min":6.5510000000000002},"last":0.71599999999999997}]},{"osd":5,"last update":"Fri Mar 3 07:57:31 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.1799999999999999,"15min":1.2490000000000001},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.97199999999999998,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.81699999999999995,"5min":1.238,"15min":1.3080000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.92900000000000005,"5min":6.6369999999999996,"15min":6.6369999999999996},"last":0.57599999999999996}]},{"osd":6,"last update":"Fri Mar 3 07:57:31 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":1.236,"15min":1.304},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.0009999999999999,"5min":6.625,"15min":6.625},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.83199999999999996,"5min":1.264,"15min":1.3360000000000001},"min":{"1min":0.72699999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":0.997,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.72899999999999998}]},{"osd":7,"last update":"Fri Mar 3 07:57:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":1.226,"15min":1.302},"min":{"1min":0.64700000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.89900000000000002,"5min":6.7009999999999996,"15min":6.7009999999999996},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.21,"15min":1.2769999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":0.995,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.621}]}]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9084,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":8512,"kb_avail":94362756,"statfs":{"total":96636764160,"available":96627462144,"internally_reserved":0,"allocated":577536,"data_stored":245921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8716288},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.83399999999999996,"15min":0.90300000000000002},"min":{"1min":0.30599999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.626,"5min":9.5820000000000007,"15min":9.5820000000000007},"last":0.379},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.90400000000000003,"15min":0.97399999999999998},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.63500000000000001,"5min":9.7840000000000007,"15min":9.7840000000000007},"last":0.436}]},{"osd":1,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.33900000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.61399999999999999,"5min":1,"15min":1},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.98699999999999999,"15min":1.0669999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":9.702,"15min":9.702},"last":0.51800000000000002}]},{"osd":2,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.93300000000000005,"15min":0.999},"min":{"1min":0.38,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68300000000000005,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0189999999999999,"15min":1.095},"min":{"1min":0.42099999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.68000000000000005,"5min":9.734,"15min":9.734},"last":0.70699999999999996}]},{"osd":4,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":1.171,"15min":1.2350000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.052,"5min":9.8629999999999995,"15min":9.8629999999999995},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.871,"15min":0.85899999999999999},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":3.1070000000000002,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":0.69199999999999995}]},{"osd":5,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.60899999999999999,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1060000000000001,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.80300000000000005,"5min":1.2270000000000001,"15min":1.298},"min":{"1min":0.60199999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.95899999999999996,"5min":9.6600000000000001,"15min":9.6600000000000001},"last":0.79500000000000004}]},{"osd":6,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":1.2430000000000001,"15min":1.3129999999999999},"min":{"1min":0.60299999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":0.99299999999999999,"5min":9.8190000000000008,"15min":9.8190000000000008},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.872,"15min":0.88200000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0589999999999999,"5min":1.609,"15min":1.609},"last":0.71699999999999997}]},{"osd":7,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":1.278,"15min":1.3560000000000001},"min":{"1min":0.63100000000000001,"5min":0.62,"15min":0.62},"max":{"1min":0.95099999999999996,"5min":9.625,"15min":9.625},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.872,"5min":1.2529999999999999,"15min":1.3169999999999999},"min":{"1min":0.66900000000000004,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.026,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":0.85699999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8956,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362884,"statfs":{"total":96636764160,"available":96627593216,"internally_reserved":0,"allocated":577536,"data_stored":245921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.98599999999999999,"15min":1.028},"min":{"1min":0.48599999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.054,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66300000000000003,"15min":0.65900000000000003},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.80600000000000005}]},{"osd":1,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.119,"15min":1.1719999999999999},"min":{"1min":0.55900000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.153,"5min":9.5709999999999997,"15min":9.5709999999999997},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.91700000000000004,"15min":0.95299999999999996},"min":{"1min":0.55300000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.014,"5min":5.266,"15min":5.266},"last":1.153}]},{"osd":2,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":1.075,"15min":1.1299999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90700000000000003,"5min":8.4529999999999994,"15min":8.4529999999999994},"last":0.878},{"interface":"front","average":{"1min":0.76400000000000001,"5min":1.004,"15min":1.044},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0880000000000001,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.92300000000000004}]},{"osd":3,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73299999999999998,"15min":0.73399999999999999},"min":{"1min":0.53100000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.80400000000000005,"15min":0.80700000000000005},"min":{"1min":0.63700000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.069,"5min":1.389,"15min":1.389},"last":0.96699999999999997}]},{"osd":5,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.746,"15min":0.751},"min":{"1min":0.53500000000000003,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.86299999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.72799999999999998,"15min":0.72199999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.76000000000000001}]},{"osd":6,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.61299999999999999,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":1.026},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.59299999999999997,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0409999999999999,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":1.0880000000000001}]},{"osd":7,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.82799999999999996,"15min":0.83199999999999996},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.119,"5min":1.679,"15min":1.679},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.92800000000000005,"15min":0.94399999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1080000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":1.0009999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8888,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94362952,"statfs":{"total":96636764160,"available":96627662848,"internally_reserved":0,"allocated":573440,"data_stored":244902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:57:33 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.39700000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.83899999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.73499999999999999,"15min":0.73699999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.877,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.71499999999999997}]},{"osd":1,"last update":"Fri Mar 3 07:57:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90800000000000003,"5min":0.996,"15min":0.996},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.63,"15min":0.623},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.79600000000000004,"5min":0.879,"15min":0.879},"last":0.69999999999999996}]},{"osd":2,"last update":"Fri Mar 3 07:57:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67500000000000004,"15min":0.67700000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.80000000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.71399999999999997,"15min":0.71199999999999997},"min":{"1min":0.58399999999999996,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.85899999999999999,"5min":1.321,"15min":1.321},"last":0.83299999999999996}]},{"osd":3,"last update":"Fri Mar 3 07:57:33 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.68899999999999995,"15min":0.68600000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.85999999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.80400000000000005}]},{"osd":4,"last update":"Fri Mar 3 07:57:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.66500000000000004,"15min":0.66300000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90800000000000003,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.78600000000000003,"15min":0.78100000000000003},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.749}]},{"osd":6,"last update":"Fri Mar 3 07:57:33 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.79400000000000004,"15min":0.80800000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.98399999999999999,"5min":1.071,"15min":1.071},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.72099999999999997,"15min":0.72399999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94599999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.75700000000000001}]},{"osd":7,"last update":"Fri Mar 3 07:57:33 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.75900000000000001,"15min":0.75800000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.95899999999999996,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.83199999999999996,"15min":0.83799999999999997},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.94299999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.73399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T07:58:18.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T07:58:18.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.509+0000 7fa6667ff700 1 Processor -- start 2023-03-03T07:58:18.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.509+0000 7fa6667ff700 1 -- start start 2023-03-03T07:58:18.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.511+0000 7fa6667ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6600b87c0 0x7fa6600cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:18.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.511+0000 7fa6667ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa6600b8b90 0x7fa6600b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:18.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.511+0000 7fa6667ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6600d0ef0 con 0x7fa6600b5240 2023-03-03T07:58:18.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.511+0000 7fa6667ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6600d0600 con 0x7fa6600b8b90 2023-03-03T07:58:18.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.511+0000 7fa6667ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6600cd970 con 0x7fa6600b87c0 2023-03-03T07:58:18.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.511+0000 7fa657fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa6600b8b90 0x7fa6600b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:18.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.511+0000 7fa664d9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa6600b5240 0x7fa6601605b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56780/0 (socket says 172.21.15.98:56780) 2023-03-03T07:58:18.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.511+0000 7fa664d9c700 1 -- 172.21.15.98:0/1356950663 learned_addr learned my addr 172.21.15.98:0/1356950663 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:18.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.511+0000 7fa65ffff700 1 --2- 172.21.15.98:0/1356950663 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6600b87c0 0x7fa6600cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:18.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.512+0000 7fa657fff700 1 -- 172.21.15.98:0/1356950663 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6600b87c0 msgr2=0x7fa6600cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:18.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.512+0000 7fa657fff700 1 --2- 172.21.15.98:0/1356950663 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6600b87c0 0x7fa6600cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:18.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.512+0000 7fa657fff700 1 -- 172.21.15.98:0/1356950663 >> v1:172.21.15.98:6789/0 conn(0x7fa6600b5240 legacy=0x7fa6601605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:18.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.512+0000 7fa65f7fe700 1 -- 172.21.15.98:0/1356950663 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760444359 0 0) 0x7fa6600d0ef0 con 0x7fa6600b5240 2023-03-03T07:58:18.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.512+0000 7fa657fff700 1 -- 172.21.15.98:0/1356950663 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6600cdc50 con 0x7fa6600b8b90 2023-03-03T07:58:18.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.512+0000 7fa657fff700 1 --2- 172.21.15.98:0/1356950663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6600b8b90 0x7fa6600b4d00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa648009840 tx=0x7fa648009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82292c2eb184e1ff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:18.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.512+0000 7fa65f7fe700 1 -- 172.21.15.98:0/1356950663 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa648011830 con 0x7fa6600b8b90 2023-03-03T07:58:18.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.512+0000 7fa65f7fe700 1 -- 172.21.15.98:0/1356950663 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa648011990 con 0x7fa6600b8b90 2023-03-03T07:58:18.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.512+0000 7fa65f7fe700 1 -- 172.21.15.98:0/1356950663 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa648011d20 con 0x7fa6600b8b90 2023-03-03T07:58:18.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.513+0000 7fa6667ff700 1 -- 172.21.15.98:0/1356950663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6600b8b90 msgr2=0x7fa6600b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:18.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.513+0000 7fa6667ff700 1 --2- 172.21.15.98:0/1356950663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6600b8b90 0x7fa6600b4d00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa648009840 tx=0x7fa648009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:18.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.513+0000 7fa6667ff700 1 -- 172.21.15.98:0/1356950663 shutdown_connections 2023-03-03T07:58:18.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.513+0000 7fa6667ff700 1 --2- 172.21.15.98:0/1356950663 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6600b87c0 0x7fa6600cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:18.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.513+0000 7fa6667ff700 1 --2- 172.21.15.98:0/1356950663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6600b8b90 0x7fa6600b4d00 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:18.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.513+0000 7fa6667ff700 1 -- 172.21.15.98:0/1356950663 >> 172.21.15.98:0/1356950663 conn(0x7fa66006aee0 msgr2=0x7fa66006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:18.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.513+0000 7fa6667ff700 1 -- 172.21.15.98:0/1356950663 shutdown_connections 2023-03-03T07:58:18.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.513+0000 7fa6667ff700 1 -- 172.21.15.98:0/1356950663 wait complete. 2023-03-03T07:58:18.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.516+0000 7fa6667ff700 1 Processor -- start 2023-03-03T07:58:18.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.516+0000 7fa6667ff700 1 -- start start 2023-03-03T07:58:18.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.518+0000 7fa6667ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6600b5240 0x7fa6601f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:18.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.518+0000 7fa6667ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6600b87c0 0x7fa66015eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:18.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.518+0000 7fa6667ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6600b8b90 0x7fa6600abdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:18.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.518+0000 7fa6667ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6600aad40 con 0x7fa6600b87c0 2023-03-03T07:58:18.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.518+0000 7fa6667ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6600ab160 con 0x7fa6600b8b90 2023-03-03T07:58:18.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.518+0000 7fa6667ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6601f2d20 con 0x7fa6600b5240 2023-03-03T07:58:18.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.518+0000 7fa657fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6600b87c0 0x7fa66015eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:18.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.518+0000 7fa657fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6600b87c0 0x7fa66015eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38138/0 (socket says 172.21.15.98:38138) 2023-03-03T07:58:18.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.518+0000 7fa657fff700 1 -- 172.21.15.98:0/2514020063 learned_addr learned my addr 172.21.15.98:0/2514020063 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:18.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.518+0000 7fa65ffff700 1 --2- 172.21.15.98:0/2514020063 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6600b5240 0x7fa6601f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:18.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.519+0000 7fa664d9c700 1 --2- 172.21.15.98:0/2514020063 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6600b8b90 0x7fa6600abdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:18.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.519+0000 7fa657fff700 1 -- 172.21.15.98:0/2514020063 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6600b5240 msgr2=0x7fa6601f27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:18.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.519+0000 7fa657fff700 1 --2- 172.21.15.98:0/2514020063 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6600b5240 0x7fa6601f27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:18.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.519+0000 7fa657fff700 1 -- 172.21.15.98:0/2514020063 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6600b8b90 msgr2=0x7fa6600abdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:18.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.519+0000 7fa657fff700 1 --2- 172.21.15.98:0/2514020063 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6600b8b90 0x7fa6600abdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:18.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.519+0000 7fa657fff700 1 -- 172.21.15.98:0/2514020063 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6480094a0 con 0x7fa6600b87c0 2023-03-03T07:58:18.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.519+0000 7fa664d9c700 1 --2- 172.21.15.98:0/2514020063 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6600b8b90 0x7fa6600abdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:18.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.519+0000 7fa657fff700 1 --2- 172.21.15.98:0/2514020063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6600b87c0 0x7fa66015eaf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa648009fd0 tx=0x7fa648004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:18.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.519+0000 7fa65d7fa700 1 -- 172.21.15.98:0/2514020063 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6480036a0 con 0x7fa6600b87c0 2023-03-03T07:58:18.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.519+0000 7fa65d7fa700 1 -- 172.21.15.98:0/2514020063 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64801e070 con 0x7fa6600b87c0 2023-03-03T07:58:18.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.519+0000 7fa6667ff700 1 -- 172.21.15.98:0/2514020063 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6600cdf30 con 0x7fa6600b87c0 2023-03-03T07:58:18.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.519+0000 7fa65d7fa700 1 -- 172.21.15.98:0/2514020063 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa648003a60 con 0x7fa6600b87c0 2023-03-03T07:58:18.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.520+0000 7fa6667ff700 1 -- 172.21.15.98:0/2514020063 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa660073a80 con 0x7fa6600b87c0 2023-03-03T07:58:18.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.521+0000 7fa6667ff700 1 -- 172.21.15.98:0/2514020063 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6600ce0c0 con 0x7fa6600b87c0 2023-03-03T07:58:18.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.522+0000 7fa65d7fa700 1 -- 172.21.15.98:0/2514020063 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fa64800f050 con 0x7fa6600b87c0 2023-03-03T07:58:18.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.523+0000 7fa65d7fa700 1 --2- 172.21.15.98:0/2514020063 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6400683c0 0x7fa64006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:18.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.523+0000 7fa65d7fa700 1 -- 172.21.15.98:0/2514020063 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa648086f20 con 0x7fa6600b87c0 2023-03-03T07:58:18.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.523+0000 7fa65ffff700 1 --2- 172.21.15.98:0/2514020063 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6400683c0 0x7fa64006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:18.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.524+0000 7fa65ffff700 1 --2- 172.21.15.98:0/2514020063 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6400683c0 0x7fa64006a870 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa6600b6ef0 tx=0x7fa650008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:18.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.527+0000 7fa65d7fa700 1 -- 172.21.15.98:0/2514020063 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa6480535c0 con 0x7fa6600b87c0 2023-03-03T07:58:18.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.723+0000 7fa6667ff700 1 -- 172.21.15.98:0/2514020063 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa6600729b0 con 0x7fa6400683c0 2023-03-03T07:58:18.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.732+0000 7fa65d7fa700 1 -- 172.21.15.98:0/2514020063 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310619 (secure 0 0 0) 0x7fa6600729b0 con 0x7fa6400683c0 2023-03-03T07:58:18.736 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:18.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.737+0000 7fa6667ff700 1 -- 172.21.15.98:0/2514020063 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6400683c0 msgr2=0x7fa64006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:18.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.737+0000 7fa6667ff700 1 --2- 172.21.15.98:0/2514020063 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6400683c0 0x7fa64006a870 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa6600b6ef0 tx=0x7fa650008040 comp rx=0 tx=0).stop 2023-03-03T07:58:18.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.737+0000 7fa6667ff700 1 -- 172.21.15.98:0/2514020063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6600b87c0 msgr2=0x7fa66015eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:18.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.737+0000 7fa6667ff700 1 --2- 172.21.15.98:0/2514020063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6600b87c0 0x7fa66015eaf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa648009fd0 tx=0x7fa648004750 comp rx=0 tx=0).stop 2023-03-03T07:58:18.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.737+0000 7fa6667ff700 1 -- 172.21.15.98:0/2514020063 shutdown_connections 2023-03-03T07:58:18.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.737+0000 7fa6667ff700 1 --2- 172.21.15.98:0/2514020063 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa6400683c0 0x7fa64006a870 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:18.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.737+0000 7fa6667ff700 1 --2- 172.21.15.98:0/2514020063 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6600b8b90 0x7fa6600abdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:18.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.737+0000 7fa6667ff700 1 --2- 172.21.15.98:0/2514020063 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6600b5240 0x7fa6601f27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:18.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.737+0000 7fa6667ff700 1 --2- 172.21.15.98:0/2514020063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6600b87c0 0x7fa66015eaf0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:18.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.737+0000 7fa6667ff700 1 -- 172.21.15.98:0/2514020063 >> 172.21.15.98:0/2514020063 conn(0x7fa66006aee0 msgr2=0x7fa6600b2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:18.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.737+0000 7fa6667ff700 1 -- 172.21.15.98:0/2514020063 shutdown_connections 2023-03-03T07:58:18.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:18.737+0000 7fa6667ff700 1 -- 172.21.15.98:0/2514020063 wait complete. 2023-03-03T07:58:18.741 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T07:58:18.766 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2023-03-03T07:58:18.209346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":74668,"kb_used_data":5996,"kb_used_omap":0,"kb_used_meta":68608,"kb_avail":754900052,"statfs":{"total":773094113280,"available":773017653248,"internally_reserved":0,"allocated":6139904,"data_stored":3305307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":70254592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001050"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.247490+0000","last_change":"2023-03-03T07:58:13.239556+0000","last_active":"2023-03-03T07:58:13.247490+0000","last_peered":"2023-03-03T07:58:13.247490+0000","last_clean":"2023-03-03T07:58:13.247490+0000","last_became_active":"2023-03-03T07:58:13.239295+0000","last_became_peered":"2023-03-03T07:58:13.239295+0000","last_unstale":"2023-03-03T07:58:13.247490+0000","last_undegraded":"2023-03-03T07:58:13.247490+0000","last_fullsized":"2023-03-03T07:58:13.247490+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:59.062415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:16.499170+0000","last_change":"2023-03-03T07:58:13.239269+0000","last_active":"2023-03-03T07:58:16.499170+0000","last_peered":"2023-03-03T07:58:16.499170+0000","last_clean":"2023-03-03T07:58:16.499170+0000","last_became_active":"2023-03-03T07:58:13.238368+0000","last_became_peered":"2023-03-03T07:58:13.238368+0000","last_unstale":"2023-03-03T07:58:16.499170+0000","last_undegraded":"2023-03-03T07:58:16.499170+0000","last_fullsized":"2023-03-03T07:58:16.499170+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:26:44.393707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888592+0000","last_change":"2023-03-03T07:58:13.243441+0000","last_active":"2023-03-03T07:58:14.888592+0000","last_peered":"2023-03-03T07:58:14.888592+0000","last_clean":"2023-03-03T07:58:14.888592+0000","last_became_active":"2023-03-03T07:58:13.243142+0000","last_became_peered":"2023-03-03T07:58:13.243142+0000","last_unstale":"2023-03-03T07:58:14.888592+0000","last_undegraded":"2023-03-03T07:58:14.888592+0000","last_fullsized":"2023-03-03T07:58:14.888592+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:19:15.226804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:16.499055+0000","last_change":"2023-03-03T07:58:13.249595+0000","last_active":"2023-03-03T07:58:16.499055+0000","last_peered":"2023-03-03T07:58:16.499055+0000","last_clean":"2023-03-03T07:58:16.499055+0000","last_became_active":"2023-03-03T07:58:13.249245+0000","last_became_peered":"2023-03-03T07:58:13.249245+0000","last_unstale":"2023-03-03T07:58:16.499055+0000","last_undegraded":"2023-03-03T07:58:16.499055+0000","last_fullsized":"2023-03-03T07:58:16.499055+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:27:34.507533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.226720+0000","last_change":"2023-03-03T07:58:13.231800+0000","last_active":"2023-03-03T07:58:14.226720+0000","last_peered":"2023-03-03T07:58:14.226720+0000","last_clean":"2023-03-03T07:58:14.226720+0000","last_became_active":"2023-03-03T07:58:13.231523+0000","last_became_peered":"2023-03-03T07:58:13.231523+0000","last_unstale":"2023-03-03T07:58:14.226720+0000","last_undegraded":"2023-03-03T07:58:14.226720+0000","last_fullsized":"2023-03-03T07:58:14.226720+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:05:48.495144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.256772+0000","last_change":"2023-03-03T07:58:13.248611+0000","last_active":"2023-03-03T07:58:13.256772+0000","last_peered":"2023-03-03T07:58:13.256772+0000","last_clean":"2023-03-03T07:58:13.256772+0000","last_became_active":"2023-03-03T07:58:13.248353+0000","last_became_peered":"2023-03-03T07:58:13.248353+0000","last_unstale":"2023-03-03T07:58:13.256772+0000","last_undegraded":"2023-03-03T07:58:13.256772+0000","last_fullsized":"2023-03-03T07:58:13.256772+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:27:34.507533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888316+0000","last_change":"2023-03-03T07:58:13.243626+0000","last_active":"2023-03-03T07:58:14.888316+0000","last_peered":"2023-03-03T07:58:14.888316+0000","last_clean":"2023-03-03T07:58:14.888316+0000","last_became_active":"2023-03-03T07:58:13.243127+0000","last_became_peered":"2023-03-03T07:58:13.243127+0000","last_unstale":"2023-03-03T07:58:14.888316+0000","last_undegraded":"2023-03-03T07:58:14.888316+0000","last_fullsized":"2023-03-03T07:58:14.888316+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:29:19.584961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:15.883414+0000","last_change":"2023-03-03T07:58:13.241758+0000","last_active":"2023-03-03T07:58:15.883414+0000","last_peered":"2023-03-03T07:58:15.883414+0000","last_clean":"2023-03-03T07:58:15.883414+0000","last_became_active":"2023-03-03T07:58:13.241447+0000","last_became_peered":"2023-03-03T07:58:13.241447+0000","last_unstale":"2023-03-03T07:58:15.883414+0000","last_undegraded":"2023-03-03T07:58:15.883414+0000","last_fullsized":"2023-03-03T07:58:15.883414+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:19:15.226804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:15.883815+0000","last_change":"2023-03-03T07:58:13.241734+0000","last_active":"2023-03-03T07:58:15.883815+0000","last_peered":"2023-03-03T07:58:15.883815+0000","last_clean":"2023-03-03T07:58:15.883815+0000","last_became_active":"2023-03-03T07:58:13.241487+0000","last_became_peered":"2023-03-03T07:58:13.241487+0000","last_unstale":"2023-03-03T07:58:15.883815+0000","last_undegraded":"2023-03-03T07:58:15.883815+0000","last_fullsized":"2023-03-03T07:58:15.883815+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:23:20.497430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.247451+0000","last_change":"2023-03-03T07:58:13.235679+0000","last_active":"2023-03-03T07:58:13.247451+0000","last_peered":"2023-03-03T07:58:13.247451+0000","last_clean":"2023-03-03T07:58:13.247451+0000","last_became_active":"2023-03-03T07:58:13.235355+0000","last_became_peered":"2023-03-03T07:58:13.235355+0000","last_unstale":"2023-03-03T07:58:13.247451+0000","last_undegraded":"2023-03-03T07:58:13.247451+0000","last_fullsized":"2023-03-03T07:58:13.247451+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:29.235859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.605685+0000","last_change":"2023-03-03T07:58:13.233428+0000","last_active":"2023-03-03T07:58:14.605685+0000","last_peered":"2023-03-03T07:58:14.605685+0000","last_clean":"2023-03-03T07:58:14.605685+0000","last_became_active":"2023-03-03T07:58:13.233147+0000","last_became_peered":"2023-03-03T07:58:13.233147+0000","last_unstale":"2023-03-03T07:58:14.605685+0000","last_undegraded":"2023-03-03T07:58:14.605685+0000","last_fullsized":"2023-03-03T07:58:14.605685+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:12.604666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:15.883799+0000","last_change":"2023-03-03T07:58:13.247530+0000","last_active":"2023-03-03T07:58:15.883799+0000","last_peered":"2023-03-03T07:58:15.883799+0000","last_clean":"2023-03-03T07:58:15.883799+0000","last_became_active":"2023-03-03T07:58:13.247243+0000","last_became_peered":"2023-03-03T07:58:13.247243+0000","last_unstale":"2023-03-03T07:58:15.883799+0000","last_undegraded":"2023-03-03T07:58:15.883799+0000","last_fullsized":"2023-03-03T07:58:15.883799+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:01.457996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.247514+0000","last_change":"2023-03-03T07:58:13.237944+0000","last_active":"2023-03-03T07:58:13.247514+0000","last_peered":"2023-03-03T07:58:13.247514+0000","last_clean":"2023-03-03T07:58:13.247514+0000","last_became_active":"2023-03-03T07:58:13.237696+0000","last_became_peered":"2023-03-03T07:58:13.237696+0000","last_unstale":"2023-03-03T07:58:13.247514+0000","last_undegraded":"2023-03-03T07:58:13.247514+0000","last_fullsized":"2023-03-03T07:58:13.247514+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:29:59.108863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888420+0000","last_change":"2023-03-03T07:58:13.244670+0000","last_active":"2023-03-03T07:58:14.888420+0000","last_peered":"2023-03-03T07:58:14.888420+0000","last_clean":"2023-03-03T07:58:14.888420+0000","last_became_active":"2023-03-03T07:58:13.244361+0000","last_became_peered":"2023-03-03T07:58:13.244361+0000","last_unstale":"2023-03-03T07:58:14.888420+0000","last_undegraded":"2023-03-03T07:58:14.888420+0000","last_fullsized":"2023-03-03T07:58:14.888420+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:09:23.965446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:16.498945+0000","last_change":"2023-03-03T07:58:13.236982+0000","last_active":"2023-03-03T07:58:16.498945+0000","last_peered":"2023-03-03T07:58:16.498945+0000","last_clean":"2023-03-03T07:58:16.498945+0000","last_became_active":"2023-03-03T07:58:13.236682+0000","last_became_peered":"2023-03-03T07:58:13.236682+0000","last_unstale":"2023-03-03T07:58:16.498945+0000","last_undegraded":"2023-03-03T07:58:16.498945+0000","last_fullsized":"2023-03-03T07:58:16.498945+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:05:48.495144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.247898+0000","last_change":"2023-03-03T07:58:13.235549+0000","last_active":"2023-03-03T07:58:13.247898+0000","last_peered":"2023-03-03T07:58:13.247898+0000","last_clean":"2023-03-03T07:58:13.247898+0000","last_became_active":"2023-03-03T07:58:13.235288+0000","last_became_peered":"2023-03-03T07:58:13.235288+0000","last_unstale":"2023-03-03T07:58:13.247898+0000","last_undegraded":"2023-03-03T07:58:13.247898+0000","last_fullsized":"2023-03-03T07:58:13.247898+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:45:08.574507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888916+0000","last_change":"2023-03-03T07:58:13.240729+0000","last_active":"2023-03-03T07:58:14.888916+0000","last_peered":"2023-03-03T07:58:14.888916+0000","last_clean":"2023-03-03T07:58:14.888916+0000","last_became_active":"2023-03-03T07:58:13.240272+0000","last_became_peered":"2023-03-03T07:58:13.240272+0000","last_unstale":"2023-03-03T07:58:14.888916+0000","last_undegraded":"2023-03-03T07:58:14.888916+0000","last_fullsized":"2023-03-03T07:58:14.888916+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:20:12.215189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:16.499211+0000","last_change":"2023-03-03T07:58:13.239433+0000","last_active":"2023-03-03T07:58:16.499211+0000","last_peered":"2023-03-03T07:58:16.499211+0000","last_clean":"2023-03-03T07:58:16.499211+0000","last_became_active":"2023-03-03T07:58:13.238721+0000","last_became_peered":"2023-03-03T07:58:13.238721+0000","last_unstale":"2023-03-03T07:58:16.499211+0000","last_undegraded":"2023-03-03T07:58:16.499211+0000","last_fullsized":"2023-03-03T07:58:16.499211+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:45:08.574507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888620+0000","last_change":"2023-03-03T07:58:13.239419+0000","last_active":"2023-03-03T07:58:14.888620+0000","last_peered":"2023-03-03T07:58:14.888620+0000","last_clean":"2023-03-03T07:58:14.888620+0000","last_became_active":"2023-03-03T07:58:13.239113+0000","last_became_peered":"2023-03-03T07:58:13.239113+0000","last_unstale":"2023-03-03T07:58:14.888620+0000","last_undegraded":"2023-03-03T07:58:14.888620+0000","last_fullsized":"2023-03-03T07:58:14.888620+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:55.630425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.562573+0000","last_change":"2023-03-03T07:58:13.247011+0000","last_active":"2023-03-03T07:58:14.562573+0000","last_peered":"2023-03-03T07:58:14.562573+0000","last_clean":"2023-03-03T07:58:14.562573+0000","last_became_active":"2023-03-03T07:58:13.246666+0000","last_became_peered":"2023-03-03T07:58:13.246666+0000","last_unstale":"2023-03-03T07:58:14.562573+0000","last_undegraded":"2023-03-03T07:58:14.562573+0000","last_fullsized":"2023-03-03T07:58:14.562573+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:03:45.825121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.562373+0000","last_change":"2023-03-03T07:58:13.245288+0000","last_active":"2023-03-03T07:58:14.562373+0000","last_peered":"2023-03-03T07:58:14.562373+0000","last_clean":"2023-03-03T07:58:14.562373+0000","last_became_active":"2023-03-03T07:58:13.244981+0000","last_became_peered":"2023-03-03T07:58:13.244981+0000","last_unstale":"2023-03-03T07:58:14.562373+0000","last_undegraded":"2023-03-03T07:58:14.562373+0000","last_fullsized":"2023-03-03T07:58:14.562373+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:51:14.168162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:15.883466+0000","last_change":"2023-03-03T07:58:13.254162+0000","last_active":"2023-03-03T07:58:15.883466+0000","last_peered":"2023-03-03T07:58:15.883466+0000","last_clean":"2023-03-03T07:58:15.883466+0000","last_became_active":"2023-03-03T07:58:13.253884+0000","last_became_peered":"2023-03-03T07:58:13.253884+0000","last_unstale":"2023-03-03T07:58:15.883466+0000","last_undegraded":"2023-03-03T07:58:15.883466+0000","last_fullsized":"2023-03-03T07:58:15.883466+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:55.630425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.605471+0000","last_change":"2023-03-03T07:58:13.231763+0000","last_active":"2023-03-03T07:58:14.605471+0000","last_peered":"2023-03-03T07:58:14.605471+0000","last_clean":"2023-03-03T07:58:14.605471+0000","last_became_active":"2023-03-03T07:58:13.231462+0000","last_became_peered":"2023-03-03T07:58:13.231462+0000","last_unstale":"2023-03-03T07:58:14.605471+0000","last_undegraded":"2023-03-03T07:58:14.605471+0000","last_fullsized":"2023-03-03T07:58:14.605471+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:51:37.051629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888178+0000","last_change":"2023-03-03T07:58:13.244231+0000","last_active":"2023-03-03T07:58:14.888178+0000","last_peered":"2023-03-03T07:58:14.888178+0000","last_clean":"2023-03-03T07:58:14.888178+0000","last_became_active":"2023-03-03T07:58:13.243900+0000","last_became_peered":"2023-03-03T07:58:13.243900+0000","last_unstale":"2023-03-03T07:58:14.888178+0000","last_undegraded":"2023-03-03T07:58:14.888178+0000","last_fullsized":"2023-03-03T07:58:14.888178+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:03.146905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888897+0000","last_change":"2023-03-03T07:58:13.253526+0000","last_active":"2023-03-03T07:58:14.888897+0000","last_peered":"2023-03-03T07:58:14.888897+0000","last_clean":"2023-03-03T07:58:14.888897+0000","last_became_active":"2023-03-03T07:58:13.252653+0000","last_became_peered":"2023-03-03T07:58:13.252653+0000","last_unstale":"2023-03-03T07:58:14.888897+0000","last_undegraded":"2023-03-03T07:58:14.888897+0000","last_fullsized":"2023-03-03T07:58:14.888897+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:29:17.972536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:15.883835+0000","last_change":"2023-03-03T07:58:13.242335+0000","last_active":"2023-03-03T07:58:15.883835+0000","last_peered":"2023-03-03T07:58:15.883835+0000","last_clean":"2023-03-03T07:58:15.883835+0000","last_became_active":"2023-03-03T07:58:13.242008+0000","last_became_peered":"2023-03-03T07:58:13.242008+0000","last_unstale":"2023-03-03T07:58:15.883835+0000","last_undegraded":"2023-03-03T07:58:15.883835+0000","last_fullsized":"2023-03-03T07:58:15.883835+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:03.146905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.256316+0000","last_change":"2023-03-03T07:58:13.247619+0000","last_active":"2023-03-03T07:58:13.256316+0000","last_peered":"2023-03-03T07:58:13.256316+0000","last_clean":"2023-03-03T07:58:13.256316+0000","last_became_active":"2023-03-03T07:58:13.243318+0000","last_became_peered":"2023-03-03T07:58:13.243318+0000","last_unstale":"2023-03-03T07:58:13.256316+0000","last_undegraded":"2023-03-03T07:58:13.256316+0000","last_fullsized":"2023-03-03T07:58:13.256316+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:30:38.928605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.562786+0000","last_change":"2023-03-03T07:58:13.245802+0000","last_active":"2023-03-03T07:58:14.562786+0000","last_peered":"2023-03-03T07:58:14.562786+0000","last_clean":"2023-03-03T07:58:14.562786+0000","last_became_active":"2023-03-03T07:58:13.245534+0000","last_became_peered":"2023-03-03T07:58:13.245534+0000","last_unstale":"2023-03-03T07:58:14.562786+0000","last_undegraded":"2023-03-03T07:58:14.562786+0000","last_fullsized":"2023-03-03T07:58:14.562786+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:49.074963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888393+0000","last_change":"2023-03-03T07:58:13.253337+0000","last_active":"2023-03-03T07:58:14.888393+0000","last_peered":"2023-03-03T07:58:14.888393+0000","last_clean":"2023-03-03T07:58:14.888393+0000","last_became_active":"2023-03-03T07:58:13.253008+0000","last_became_peered":"2023-03-03T07:58:13.253008+0000","last_unstale":"2023-03-03T07:58:14.888393+0000","last_undegraded":"2023-03-03T07:58:14.888393+0000","last_fullsized":"2023-03-03T07:58:14.888393+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:01.457996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:16.499301+0000","last_change":"2023-03-03T07:58:13.243993+0000","last_active":"2023-03-03T07:58:16.499301+0000","last_peered":"2023-03-03T07:58:16.499301+0000","last_clean":"2023-03-03T07:58:16.499301+0000","last_became_active":"2023-03-03T07:58:13.243688+0000","last_became_peered":"2023-03-03T07:58:13.243688+0000","last_unstale":"2023-03-03T07:58:16.499301+0000","last_undegraded":"2023-03-03T07:58:16.499301+0000","last_fullsized":"2023-03-03T07:58:16.499301+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:59.062415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888241+0000","last_change":"2023-03-03T07:58:13.243425+0000","last_active":"2023-03-03T07:58:14.888241+0000","last_peered":"2023-03-03T07:58:14.888241+0000","last_clean":"2023-03-03T07:58:14.888241+0000","last_became_active":"2023-03-03T07:58:13.243098+0000","last_became_peered":"2023-03-03T07:58:13.243098+0000","last_unstale":"2023-03-03T07:58:14.888241+0000","last_undegraded":"2023-03-03T07:58:14.888241+0000","last_fullsized":"2023-03-03T07:58:14.888241+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:54:33.331634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.225305+0000","last_change":"2023-03-03T07:58:13.244279+0000","last_active":"2023-03-03T07:58:14.225305+0000","last_peered":"2023-03-03T07:58:14.225305+0000","last_clean":"2023-03-03T07:58:14.225305+0000","last_became_active":"2023-03-03T07:58:13.243817+0000","last_became_peered":"2023-03-03T07:58:13.243817+0000","last_unstale":"2023-03-03T07:58:14.225305+0000","last_undegraded":"2023-03-03T07:58:14.225305+0000","last_fullsized":"2023-03-03T07:58:14.225305+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:59.062415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:16.498796+0000","last_change":"2023-03-03T07:58:13.237961+0000","last_active":"2023-03-03T07:58:16.498796+0000","last_peered":"2023-03-03T07:58:16.498796+0000","last_clean":"2023-03-03T07:58:16.498796+0000","last_became_active":"2023-03-03T07:58:13.237661+0000","last_became_peered":"2023-03-03T07:58:13.237661+0000","last_unstale":"2023-03-03T07:58:16.498796+0000","last_undegraded":"2023-03-03T07:58:16.498796+0000","last_fullsized":"2023-03-03T07:58:16.498796+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:30:38.928605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.250660+0000","last_change":"2023-03-03T07:58:12.683519+0000","last_active":"2023-03-03T07:58:13.250660+0000","last_peered":"2023-03-03T07:58:13.250660+0000","last_clean":"2023-03-03T07:58:13.250660+0000","last_became_active":"2023-03-03T07:58:12.683223+0000","last_became_peered":"2023-03-03T07:58:12.683223+0000","last_unstale":"2023-03-03T07:58:13.250660+0000","last_undegraded":"2023-03-03T07:58:13.250660+0000","last_fullsized":"2023-03-03T07:58:13.250660+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:45:11.942416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.225434+0000","last_change":"2023-03-03T07:58:13.241877+0000","last_active":"2023-03-03T07:58:14.225434+0000","last_peered":"2023-03-03T07:58:14.225434+0000","last_clean":"2023-03-03T07:58:14.225434+0000","last_became_active":"2023-03-03T07:58:13.241029+0000","last_became_peered":"2023-03-03T07:58:13.241029+0000","last_unstale":"2023-03-03T07:58:14.225434+0000","last_undegraded":"2023-03-03T07:58:14.225434+0000","last_fullsized":"2023-03-03T07:58:14.225434+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:42.548121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.227950+0000","last_change":"2023-03-03T07:58:12.674627+0000","last_active":"2023-03-03T07:58:13.227950+0000","last_peered":"2023-03-03T07:58:13.227950+0000","last_clean":"2023-03-03T07:58:13.227950+0000","last_became_active":"2023-03-03T07:58:12.674388+0000","last_became_peered":"2023-03-03T07:58:12.674388+0000","last_unstale":"2023-03-03T07:58:13.227950+0000","last_undegraded":"2023-03-03T07:58:13.227950+0000","last_fullsized":"2023-03-03T07:58:13.227950+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:48.066944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.562449+0000","last_change":"2023-03-03T07:58:13.248291+0000","last_active":"2023-03-03T07:58:14.562449+0000","last_peered":"2023-03-03T07:58:14.562449+0000","last_clean":"2023-03-03T07:58:14.562449+0000","last_became_active":"2023-03-03T07:58:13.248008+0000","last_became_peered":"2023-03-03T07:58:13.248008+0000","last_unstale":"2023-03-03T07:58:14.562449+0000","last_undegraded":"2023-03-03T07:58:14.562449+0000","last_fullsized":"2023-03-03T07:58:14.562449+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:19:40.118527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.242265+0000","last_change":"2023-03-03T07:58:12.681176+0000","last_active":"2023-03-03T07:58:13.242265+0000","last_peered":"2023-03-03T07:58:13.242265+0000","last_clean":"2023-03-03T07:58:13.242265+0000","last_became_active":"2023-03-03T07:58:12.680918+0000","last_became_peered":"2023-03-03T07:58:12.680918+0000","last_unstale":"2023-03-03T07:58:13.242265+0000","last_undegraded":"2023-03-03T07:58:13.242265+0000","last_fullsized":"2023-03-03T07:58:13.242265+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:20.568387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:15.883662+0000","last_change":"2023-03-03T07:58:13.238148+0000","last_active":"2023-03-03T07:58:15.883662+0000","last_peered":"2023-03-03T07:58:15.883662+0000","last_clean":"2023-03-03T07:58:15.883662+0000","last_became_active":"2023-03-03T07:58:13.237971+0000","last_became_peered":"2023-03-03T07:58:13.237971+0000","last_unstale":"2023-03-03T07:58:15.883662+0000","last_undegraded":"2023-03-03T07:58:15.883662+0000","last_fullsized":"2023-03-03T07:58:15.883662+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:42:29.972879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.248395+0000","last_change":"2023-03-03T07:58:12.675518+0000","last_active":"2023-03-03T07:58:13.248395+0000","last_peered":"2023-03-03T07:58:13.248395+0000","last_clean":"2023-03-03T07:58:13.248395+0000","last_became_active":"2023-03-03T07:58:12.675247+0000","last_became_peered":"2023-03-03T07:58:12.675247+0000","last_unstale":"2023-03-03T07:58:13.248395+0000","last_undegraded":"2023-03-03T07:58:13.248395+0000","last_fullsized":"2023-03-03T07:58:13.248395+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:05:05.629228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.226788+0000","last_change":"2023-03-03T07:58:13.243056+0000","last_active":"2023-03-03T07:58:14.226788+0000","last_peered":"2023-03-03T07:58:14.226788+0000","last_clean":"2023-03-03T07:58:14.226788+0000","last_became_active":"2023-03-03T07:58:13.242726+0000","last_became_peered":"2023-03-03T07:58:13.242726+0000","last_unstale":"2023-03-03T07:58:14.226788+0000","last_undegraded":"2023-03-03T07:58:14.226788+0000","last_fullsized":"2023-03-03T07:58:14.226788+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:19:19.530521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.242835+0000","last_change":"2023-03-03T07:58:12.682040+0000","last_active":"2023-03-03T07:58:13.242835+0000","last_peered":"2023-03-03T07:58:13.242835+0000","last_clean":"2023-03-03T07:58:13.242835+0000","last_became_active":"2023-03-03T07:58:12.681390+0000","last_became_peered":"2023-03-03T07:58:12.681390+0000","last_unstale":"2023-03-03T07:58:13.242835+0000","last_undegraded":"2023-03-03T07:58:13.242835+0000","last_fullsized":"2023-03-03T07:58:13.242835+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:29:58.100844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:16.499237+0000","last_change":"2023-03-03T07:58:13.239152+0000","last_active":"2023-03-03T07:58:16.499237+0000","last_peered":"2023-03-03T07:58:16.499237+0000","last_clean":"2023-03-03T07:58:16.499237+0000","last_became_active":"2023-03-03T07:58:13.238877+0000","last_became_peered":"2023-03-03T07:58:13.238877+0000","last_unstale":"2023-03-03T07:58:16.499237+0000","last_undegraded":"2023-03-03T07:58:16.499237+0000","last_fullsized":"2023-03-03T07:58:16.499237+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:39:44.094387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.249405+0000","last_change":"2023-03-03T07:58:12.681665+0000","last_active":"2023-03-03T07:58:13.249405+0000","last_peered":"2023-03-03T07:58:13.249405+0000","last_clean":"2023-03-03T07:58:13.249405+0000","last_became_active":"2023-03-03T07:58:12.681345+0000","last_became_peered":"2023-03-03T07:58:12.681345+0000","last_unstale":"2023-03-03T07:58:13.249405+0000","last_undegraded":"2023-03-03T07:58:13.249405+0000","last_fullsized":"2023-03-03T07:58:13.249405+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:29:58.100844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:15.883930+0000","last_change":"2023-03-03T07:58:13.240569+0000","last_active":"2023-03-03T07:58:15.883930+0000","last_peered":"2023-03-03T07:58:15.883930+0000","last_clean":"2023-03-03T07:58:15.883930+0000","last_became_active":"2023-03-03T07:58:13.240189+0000","last_became_peered":"2023-03-03T07:58:13.240189+0000","last_unstale":"2023-03-03T07:58:15.883930+0000","last_undegraded":"2023-03-03T07:58:15.883930+0000","last_fullsized":"2023-03-03T07:58:15.883930+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:20:07.471844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.232909+0000","last_change":"2023-03-03T07:58:12.680416+0000","last_active":"2023-03-03T07:58:13.232909+0000","last_peered":"2023-03-03T07:58:13.232909+0000","last_clean":"2023-03-03T07:58:13.232909+0000","last_became_active":"2023-03-03T07:58:12.680097+0000","last_became_peered":"2023-03-03T07:58:12.680097+0000","last_unstale":"2023-03-03T07:58:13.232909+0000","last_undegraded":"2023-03-03T07:58:13.232909+0000","last_fullsized":"2023-03-03T07:58:13.232909+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:51:36.043610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:16.499396+0000","last_change":"2023-03-03T07:58:13.242874+0000","last_active":"2023-03-03T07:58:16.499396+0000","last_peered":"2023-03-03T07:58:16.499396+0000","last_clean":"2023-03-03T07:58:16.499396+0000","last_became_active":"2023-03-03T07:58:13.242574+0000","last_became_peered":"2023-03-03T07:58:13.242574+0000","last_unstale":"2023-03-03T07:58:16.499396+0000","last_undegraded":"2023-03-03T07:58:16.499396+0000","last_fullsized":"2023-03-03T07:58:16.499396+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:27:42.548121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.228372+0000","last_change":"2023-03-03T07:58:12.680048+0000","last_active":"2023-03-03T07:58:13.228372+0000","last_peered":"2023-03-03T07:58:13.228372+0000","last_clean":"2023-03-03T07:58:13.228372+0000","last_became_active":"2023-03-03T07:58:12.679716+0000","last_became_peered":"2023-03-03T07:58:12.679716+0000","last_unstale":"2023-03-03T07:58:13.228372+0000","last_undegraded":"2023-03-03T07:58:13.228372+0000","last_fullsized":"2023-03-03T07:58:13.228372+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:16:02.075425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.226451+0000","last_change":"2023-03-03T07:58:13.243568+0000","last_active":"2023-03-03T07:58:14.226451+0000","last_peered":"2023-03-03T07:58:14.226451+0000","last_clean":"2023-03-03T07:58:14.226451+0000","last_became_active":"2023-03-03T07:58:13.243214+0000","last_became_peered":"2023-03-03T07:58:13.243214+0000","last_unstale":"2023-03-03T07:58:14.226451+0000","last_undegraded":"2023-03-03T07:58:14.226451+0000","last_fullsized":"2023-03-03T07:58:14.226451+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:57.901148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.227394+0000","last_change":"2023-03-03T07:58:12.683712+0000","last_active":"2023-03-03T07:58:13.227394+0000","last_peered":"2023-03-03T07:58:13.227394+0000","last_clean":"2023-03-03T07:58:13.227394+0000","last_became_active":"2023-03-03T07:58:12.683452+0000","last_became_peered":"2023-03-03T07:58:12.683452+0000","last_unstale":"2023-03-03T07:58:13.227394+0000","last_undegraded":"2023-03-03T07:58:13.227394+0000","last_fullsized":"2023-03-03T07:58:13.227394+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:05:05.629228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.225305+0000","last_change":"2023-03-03T07:58:13.230470+0000","last_active":"2023-03-03T07:58:14.225305+0000","last_peered":"2023-03-03T07:58:14.225305+0000","last_clean":"2023-03-03T07:58:14.225305+0000","last_became_active":"2023-03-03T07:58:13.230163+0000","last_became_peered":"2023-03-03T07:58:13.230163+0000","last_unstale":"2023-03-03T07:58:14.225305+0000","last_undegraded":"2023-03-03T07:58:14.225305+0000","last_fullsized":"2023-03-03T07:58:14.225305+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:45:08.574507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.221548+0000","last_change":"2023-03-03T07:58:12.682845+0000","last_active":"2023-03-03T07:58:13.221548+0000","last_peered":"2023-03-03T07:58:13.221548+0000","last_clean":"2023-03-03T07:58:13.221548+0000","last_became_active":"2023-03-03T07:58:12.682560+0000","last_became_peered":"2023-03-03T07:58:12.682560+0000","last_unstale":"2023-03-03T07:58:13.221548+0000","last_undegraded":"2023-03-03T07:58:13.221548+0000","last_fullsized":"2023-03-03T07:58:13.221548+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:46:18.685833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888654+0000","last_change":"2023-03-03T07:58:13.254269+0000","last_active":"2023-03-03T07:58:14.888654+0000","last_peered":"2023-03-03T07:58:14.888654+0000","last_clean":"2023-03-03T07:58:14.888654+0000","last_became_active":"2023-03-03T07:58:13.253732+0000","last_became_peered":"2023-03-03T07:58:13.253732+0000","last_unstale":"2023-03-03T07:58:14.888654+0000","last_undegraded":"2023-03-03T07:58:14.888654+0000","last_fullsized":"2023-03-03T07:58:14.888654+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:13.508806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.256629+0000","last_change":"2023-03-03T07:58:12.677632+0000","last_active":"2023-03-03T07:58:13.256629+0000","last_peered":"2023-03-03T07:58:13.256629+0000","last_clean":"2023-03-03T07:58:13.256629+0000","last_became_active":"2023-03-03T07:58:12.677295+0000","last_became_peered":"2023-03-03T07:58:12.677295+0000","last_unstale":"2023-03-03T07:58:13.256629+0000","last_undegraded":"2023-03-03T07:58:13.256629+0000","last_fullsized":"2023-03-03T07:58:13.256629+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:25.389749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.226893+0000","last_change":"2023-03-03T07:58:13.227239+0000","last_active":"2023-03-03T07:58:14.226893+0000","last_peered":"2023-03-03T07:58:14.226893+0000","last_clean":"2023-03-03T07:58:14.226893+0000","last_became_active":"2023-03-03T07:58:13.226966+0000","last_became_peered":"2023-03-03T07:58:13.226966+0000","last_unstale":"2023-03-03T07:58:14.226893+0000","last_undegraded":"2023-03-03T07:58:14.226893+0000","last_fullsized":"2023-03-03T07:58:14.226893+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:01:46.348197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":17,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.259671+0000","last_change":"2023-03-03T07:58:12.668231+0000","last_active":"2023-03-03T07:58:13.259671+0000","last_peered":"2023-03-03T07:58:13.259671+0000","last_clean":"2023-03-03T07:58:13.259671+0000","last_became_active":"2023-03-03T07:58:12.667976+0000","last_became_peered":"2023-03-03T07:58:12.667976+0000","last_unstale":"2023-03-03T07:58:13.259671+0000","last_undegraded":"2023-03-03T07:58:13.259671+0000","last_fullsized":"2023-03-03T07:58:13.259671+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:27:33.499514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.225454+0000","last_change":"2023-03-03T07:58:13.241521+0000","last_active":"2023-03-03T07:58:14.225454+0000","last_peered":"2023-03-03T07:58:14.225454+0000","last_clean":"2023-03-03T07:58:14.225454+0000","last_became_active":"2023-03-03T07:58:13.241211+0000","last_became_peered":"2023-03-03T07:58:13.241211+0000","last_unstale":"2023-03-03T07:58:14.225454+0000","last_undegraded":"2023-03-03T07:58:14.225454+0000","last_fullsized":"2023-03-03T07:58:14.225454+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:26:44.393707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.250694+0000","last_change":"2023-03-03T07:58:12.682128+0000","last_active":"2023-03-03T07:58:13.250694+0000","last_peered":"2023-03-03T07:58:13.250694+0000","last_clean":"2023-03-03T07:58:13.250694+0000","last_became_active":"2023-03-03T07:58:12.681756+0000","last_became_peered":"2023-03-03T07:58:12.681756+0000","last_unstale":"2023-03-03T07:58:13.250694+0000","last_undegraded":"2023-03-03T07:58:13.250694+0000","last_fullsized":"2023-03-03T07:58:13.250694+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:58.525319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888698+0000","last_change":"2023-03-03T07:58:13.253785+0000","last_active":"2023-03-03T07:58:14.888698+0000","last_peered":"2023-03-03T07:58:14.888698+0000","last_clean":"2023-03-03T07:58:14.888698+0000","last_became_active":"2023-03-03T07:58:13.253521+0000","last_became_peered":"2023-03-03T07:58:13.253521+0000","last_unstale":"2023-03-03T07:58:14.888698+0000","last_undegraded":"2023-03-03T07:58:14.888698+0000","last_fullsized":"2023-03-03T07:58:14.888698+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:51:58.672958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.232300+0000","last_change":"2023-03-03T07:58:12.684174+0000","last_active":"2023-03-03T07:58:13.232300+0000","last_peered":"2023-03-03T07:58:13.232300+0000","last_clean":"2023-03-03T07:58:13.232300+0000","last_became_active":"2023-03-03T07:58:12.683850+0000","last_became_peered":"2023-03-03T07:58:12.683850+0000","last_unstale":"2023-03-03T07:58:13.232300+0000","last_undegraded":"2023-03-03T07:58:13.232300+0000","last_fullsized":"2023-03-03T07:58:13.232300+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:15:45.710069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:15.883984+0000","last_change":"2023-03-03T07:58:13.237659+0000","last_active":"2023-03-03T07:58:15.883984+0000","last_peered":"2023-03-03T07:58:15.883984+0000","last_clean":"2023-03-03T07:58:15.883984+0000","last_became_active":"2023-03-03T07:58:13.237401+0000","last_became_peered":"2023-03-03T07:58:13.237401+0000","last_unstale":"2023-03-03T07:58:15.883984+0000","last_undegraded":"2023-03-03T07:58:15.883984+0000","last_fullsized":"2023-03-03T07:58:15.883984+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:29:19.584961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.226944+0000","last_change":"2023-03-03T07:58:12.681325+0000","last_active":"2023-03-03T07:58:13.226944+0000","last_peered":"2023-03-03T07:58:13.226944+0000","last_clean":"2023-03-03T07:58:13.226944+0000","last_became_active":"2023-03-03T07:58:12.681019+0000","last_became_peered":"2023-03-03T07:58:12.681019+0000","last_unstale":"2023-03-03T07:58:13.226944+0000","last_undegraded":"2023-03-03T07:58:13.226944+0000","last_fullsized":"2023-03-03T07:58:13.226944+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:40:17.732316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888853+0000","last_change":"2023-03-03T07:58:13.240879+0000","last_active":"2023-03-03T07:58:14.888853+0000","last_peered":"2023-03-03T07:58:14.888853+0000","last_clean":"2023-03-03T07:58:14.888853+0000","last_became_active":"2023-03-03T07:58:13.240575+0000","last_became_peered":"2023-03-03T07:58:13.240575+0000","last_unstale":"2023-03-03T07:58:14.888853+0000","last_undegraded":"2023-03-03T07:58:14.888853+0000","last_fullsized":"2023-03-03T07:58:14.888853+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:41:10.929779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":19,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.258544+0000","last_change":"2023-03-03T07:58:12.667132+0000","last_active":"2023-03-03T07:58:13.258544+0000","last_peered":"2023-03-03T07:58:13.258544+0000","last_clean":"2023-03-03T07:58:13.258544+0000","last_became_active":"2023-03-03T07:58:12.666852+0000","last_became_peered":"2023-03-03T07:58:12.666852+0000","last_unstale":"2023-03-03T07:58:13.258544+0000","last_undegraded":"2023-03-03T07:58:13.258544+0000","last_fullsized":"2023-03-03T07:58:13.258544+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:46:18.685833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:15.883437+0000","last_change":"2023-03-03T07:58:13.238917+0000","last_active":"2023-03-03T07:58:15.883437+0000","last_peered":"2023-03-03T07:58:15.883437+0000","last_clean":"2023-03-03T07:58:15.883437+0000","last_became_active":"2023-03-03T07:58:13.238507+0000","last_became_peered":"2023-03-03T07:58:13.238507+0000","last_unstale":"2023-03-03T07:58:15.883437+0000","last_undegraded":"2023-03-03T07:58:15.883437+0000","last_fullsized":"2023-03-03T07:58:15.883437+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:38:01.382624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.252297+0000","last_change":"2023-03-03T07:58:12.683638+0000","last_active":"2023-03-03T07:58:13.252297+0000","last_peered":"2023-03-03T07:58:13.252297+0000","last_clean":"2023-03-03T07:58:13.252297+0000","last_became_active":"2023-03-03T07:58:12.683397+0000","last_became_peered":"2023-03-03T07:58:12.683397+0000","last_unstale":"2023-03-03T07:58:13.252297+0000","last_undegraded":"2023-03-03T07:58:13.252297+0000","last_fullsized":"2023-03-03T07:58:13.252297+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:46:18.685833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.255657+0000","last_change":"2023-03-03T07:58:13.242958+0000","last_active":"2023-03-03T07:58:13.255657+0000","last_peered":"2023-03-03T07:58:13.255657+0000","last_clean":"2023-03-03T07:58:13.255657+0000","last_became_active":"2023-03-03T07:58:13.242639+0000","last_became_peered":"2023-03-03T07:58:13.242639+0000","last_unstale":"2023-03-03T07:58:13.255657+0000","last_undegraded":"2023-03-03T07:58:13.255657+0000","last_fullsized":"2023-03-03T07:58:13.255657+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:26:44.393707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.246874+0000","last_change":"2023-03-03T07:58:12.683831+0000","last_active":"2023-03-03T07:58:13.246874+0000","last_peered":"2023-03-03T07:58:13.246874+0000","last_clean":"2023-03-03T07:58:13.246874+0000","last_became_active":"2023-03-03T07:58:12.683550+0000","last_became_peered":"2023-03-03T07:58:12.683550+0000","last_unstale":"2023-03-03T07:58:13.246874+0000","last_undegraded":"2023-03-03T07:58:13.246874+0000","last_fullsized":"2023-03-03T07:58:13.246874+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:34.055432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.226879+0000","last_change":"2023-03-03T07:58:13.245447+0000","last_active":"2023-03-03T07:58:14.226879+0000","last_peered":"2023-03-03T07:58:14.226879+0000","last_clean":"2023-03-03T07:58:14.226879+0000","last_became_active":"2023-03-03T07:58:13.245108+0000","last_became_peered":"2023-03-03T07:58:13.245108+0000","last_unstale":"2023-03-03T07:58:14.226879+0000","last_undegraded":"2023-03-03T07:58:14.226879+0000","last_fullsized":"2023-03-03T07:58:14.226879+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:12:15.693661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.257599+0000","last_change":"2023-03-03T07:58:12.682998+0000","last_active":"2023-03-03T07:58:13.257599+0000","last_peered":"2023-03-03T07:58:13.257599+0000","last_clean":"2023-03-03T07:58:13.257599+0000","last_became_active":"2023-03-03T07:58:12.682605+0000","last_became_peered":"2023-03-03T07:58:12.682605+0000","last_unstale":"2023-03-03T07:58:13.257599+0000","last_undegraded":"2023-03-03T07:58:13.257599+0000","last_fullsized":"2023-03-03T07:58:13.257599+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:19:51.509265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.247494+0000","last_change":"2023-03-03T07:58:13.236823+0000","last_active":"2023-03-03T07:58:13.247494+0000","last_peered":"2023-03-03T07:58:13.247494+0000","last_clean":"2023-03-03T07:58:13.247494+0000","last_became_active":"2023-03-03T07:58:13.236543+0000","last_became_peered":"2023-03-03T07:58:13.236543+0000","last_unstale":"2023-03-03T07:58:13.247494+0000","last_undegraded":"2023-03-03T07:58:13.247494+0000","last_fullsized":"2023-03-03T07:58:13.247494+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:46:19.693852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.232167+0000","last_change":"2023-03-03T07:58:12.681255+0000","last_active":"2023-03-03T07:58:13.232167+0000","last_peered":"2023-03-03T07:58:13.232167+0000","last_clean":"2023-03-03T07:58:13.232167+0000","last_became_active":"2023-03-03T07:58:12.680897+0000","last_became_peered":"2023-03-03T07:58:12.680897+0000","last_unstale":"2023-03-03T07:58:13.232167+0000","last_undegraded":"2023-03-03T07:58:13.232167+0000","last_fullsized":"2023-03-03T07:58:13.232167+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:40:45.616669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.225368+0000","last_change":"2023-03-03T07:58:13.228383+0000","last_active":"2023-03-03T07:58:14.225368+0000","last_peered":"2023-03-03T07:58:14.225368+0000","last_clean":"2023-03-03T07:58:14.225368+0000","last_became_active":"2023-03-03T07:58:13.228084+0000","last_became_peered":"2023-03-03T07:58:13.228084+0000","last_unstale":"2023-03-03T07:58:14.225368+0000","last_undegraded":"2023-03-03T07:58:14.225368+0000","last_fullsized":"2023-03-03T07:58:14.225368+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:30:38.928605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.236931+0000","last_change":"2023-03-03T07:58:12.673025+0000","last_active":"2023-03-03T07:58:13.236931+0000","last_peered":"2023-03-03T07:58:13.236931+0000","last_clean":"2023-03-03T07:58:13.236931+0000","last_became_active":"2023-03-03T07:58:12.672771+0000","last_became_peered":"2023-03-03T07:58:12.672771+0000","last_unstale":"2023-03-03T07:58:13.236931+0000","last_undegraded":"2023-03-03T07:58:13.236931+0000","last_fullsized":"2023-03-03T07:58:13.236931+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:04:23.831200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.562517+0000","last_change":"2023-03-03T07:58:13.247097+0000","last_active":"2023-03-03T07:58:14.562517+0000","last_peered":"2023-03-03T07:58:14.562517+0000","last_clean":"2023-03-03T07:58:14.562517+0000","last_became_active":"2023-03-03T07:58:13.246759+0000","last_became_peered":"2023-03-03T07:58:13.246759+0000","last_unstale":"2023-03-03T07:58:14.562517+0000","last_undegraded":"2023-03-03T07:58:14.562517+0000","last_fullsized":"2023-03-03T07:58:14.562517+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:40:18.740335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.223415+0000","last_change":"2023-03-03T07:58:12.667685+0000","last_active":"2023-03-03T07:58:13.223415+0000","last_peered":"2023-03-03T07:58:13.223415+0000","last_clean":"2023-03-03T07:58:13.223415+0000","last_became_active":"2023-03-03T07:58:12.667444+0000","last_became_peered":"2023-03-03T07:58:12.667444+0000","last_unstale":"2023-03-03T07:58:13.223415+0000","last_undegraded":"2023-03-03T07:58:13.223415+0000","last_fullsized":"2023-03-03T07:58:13.223415+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:39:43.086368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888773+0000","last_change":"2023-03-03T07:58:13.252741+0000","last_active":"2023-03-03T07:58:14.888773+0000","last_peered":"2023-03-03T07:58:14.888773+0000","last_clean":"2023-03-03T07:58:14.888773+0000","last_became_active":"2023-03-03T07:58:13.252466+0000","last_became_peered":"2023-03-03T07:58:13.252466+0000","last_unstale":"2023-03-03T07:58:14.888773+0000","last_undegraded":"2023-03-03T07:58:14.888773+0000","last_fullsized":"2023-03-03T07:58:14.888773+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:23:20.497430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.229232+0000","last_change":"2023-03-03T07:58:12.685530+0000","last_active":"2023-03-03T07:58:13.229232+0000","last_peered":"2023-03-03T07:58:13.229232+0000","last_clean":"2023-03-03T07:58:13.229232+0000","last_became_active":"2023-03-03T07:58:12.685047+0000","last_became_peered":"2023-03-03T07:58:12.685047+0000","last_unstale":"2023-03-03T07:58:13.229232+0000","last_undegraded":"2023-03-03T07:58:13.229232+0000","last_fullsized":"2023-03-03T07:58:13.229232+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:27:33.499514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.255675+0000","last_change":"2023-03-03T07:58:13.243137+0000","last_active":"2023-03-03T07:58:13.255675+0000","last_peered":"2023-03-03T07:58:13.255675+0000","last_clean":"2023-03-03T07:58:13.255675+0000","last_became_active":"2023-03-03T07:58:13.242879+0000","last_became_peered":"2023-03-03T07:58:13.242879+0000","last_unstale":"2023-03-03T07:58:13.255675+0000","last_undegraded":"2023-03-03T07:58:13.255675+0000","last_fullsized":"2023-03-03T07:58:13.255675+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:59.533338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.219090+0000","last_change":"2023-03-03T07:58:12.674448+0000","last_active":"2023-03-03T07:58:13.219090+0000","last_peered":"2023-03-03T07:58:13.219090+0000","last_clean":"2023-03-03T07:58:13.219090+0000","last_became_active":"2023-03-03T07:58:12.674210+0000","last_became_peered":"2023-03-03T07:58:12.674210+0000","last_unstale":"2023-03-03T07:58:13.219090+0000","last_undegraded":"2023-03-03T07:58:13.219090+0000","last_fullsized":"2023-03-03T07:58:13.219090+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:47:14.909630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888622+0000","last_change":"2023-03-03T07:58:13.244878+0000","last_active":"2023-03-03T07:58:14.888622+0000","last_peered":"2023-03-03T07:58:14.888622+0000","last_clean":"2023-03-03T07:58:14.888622+0000","last_became_active":"2023-03-03T07:58:13.244520+0000","last_became_peered":"2023-03-03T07:58:13.244520+0000","last_unstale":"2023-03-03T07:58:14.888622+0000","last_undegraded":"2023-03-03T07:58:14.888622+0000","last_fullsized":"2023-03-03T07:58:14.888622+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:20:07.471844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.221774+0000","last_change":"2023-03-03T07:58:12.678482+0000","last_active":"2023-03-03T07:58:13.221774+0000","last_peered":"2023-03-03T07:58:13.221774+0000","last_clean":"2023-03-03T07:58:13.221774+0000","last_became_active":"2023-03-03T07:58:12.678190+0000","last_became_peered":"2023-03-03T07:58:12.678190+0000","last_unstale":"2023-03-03T07:58:13.221774+0000","last_undegraded":"2023-03-03T07:58:13.221774+0000","last_fullsized":"2023-03-03T07:58:13.221774+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:27:33.499514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":36,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-03-03T07:58:12.662374+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T07:58:12.662374+0000","last_peered":"2023-03-03T07:58:12.662374+0000","last_clean":"2023-03-03T07:58:12.662374+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T07:58:12.662374+0000","last_undegraded":"2023-03-03T07:58:12.662374+0000","last_fullsized":"2023-03-03T07:58:12.662374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.225442+0000","last_change":"2023-03-03T07:58:13.244098+0000","last_active":"2023-03-03T07:58:14.225442+0000","last_peered":"2023-03-03T07:58:14.225442+0000","last_clean":"2023-03-03T07:58:14.225442+0000","last_became_active":"2023-03-03T07:58:13.243480+0000","last_became_peered":"2023-03-03T07:58:13.243480+0000","last_unstale":"2023-03-03T07:58:14.225442+0000","last_undegraded":"2023-03-03T07:58:14.225442+0000","last_fullsized":"2023-03-03T07:58:14.225442+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:39:44.094387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.259303+0000","last_change":"2023-03-03T07:58:12.683584+0000","last_active":"2023-03-03T07:58:13.259303+0000","last_peered":"2023-03-03T07:58:13.259303+0000","last_clean":"2023-03-03T07:58:13.259303+0000","last_became_active":"2023-03-03T07:58:12.683091+0000","last_became_peered":"2023-03-03T07:58:12.683091+0000","last_unstale":"2023-03-03T07:58:13.259303+0000","last_undegraded":"2023-03-03T07:58:13.259303+0000","last_fullsized":"2023-03-03T07:58:13.259303+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:37:11.596647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":37,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.223575+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T07:58:13.223575+0000","last_peered":"2023-03-03T07:58:13.223575+0000","last_clean":"2023-03-03T07:58:13.223575+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T07:58:13.223575+0000","last_undegraded":"2023-03-03T07:58:13.223575+0000","last_fullsized":"2023-03-03T07:58:13.223575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:15.883704+0000","last_change":"2023-03-03T07:58:13.246962+0000","last_active":"2023-03-03T07:58:15.883704+0000","last_peered":"2023-03-03T07:58:15.883704+0000","last_clean":"2023-03-03T07:58:15.883704+0000","last_became_active":"2023-03-03T07:58:13.246672+0000","last_became_peered":"2023-03-03T07:58:13.246672+0000","last_unstale":"2023-03-03T07:58:15.883704+0000","last_undegraded":"2023-03-03T07:58:15.883704+0000","last_fullsized":"2023-03-03T07:58:15.883704+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:54:33.331634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.230235+0000","last_change":"2023-03-03T07:58:12.676662+0000","last_active":"2023-03-03T07:58:13.230235+0000","last_peered":"2023-03-03T07:58:13.230235+0000","last_clean":"2023-03-03T07:58:13.230235+0000","last_became_active":"2023-03-03T07:58:12.676352+0000","last_became_peered":"2023-03-03T07:58:12.676352+0000","last_unstale":"2023-03-03T07:58:13.230235+0000","last_undegraded":"2023-03-03T07:58:13.230235+0000","last_fullsized":"2023-03-03T07:58:13.230235+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:35:13.361074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":36,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.219201+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T07:58:13.219201+0000","last_peered":"2023-03-03T07:58:13.219201+0000","last_clean":"2023-03-03T07:58:13.219201+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T07:58:13.219201+0000","last_undegraded":"2023-03-03T07:58:13.219201+0000","last_fullsized":"2023-03-03T07:58:13.219201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.562377+0000","last_change":"2023-03-03T07:58:13.237773+0000","last_active":"2023-03-03T07:58:14.562377+0000","last_peered":"2023-03-03T07:58:14.562377+0000","last_clean":"2023-03-03T07:58:14.562377+0000","last_became_active":"2023-03-03T07:58:13.237486+0000","last_became_peered":"2023-03-03T07:58:13.237486+0000","last_unstale":"2023-03-03T07:58:14.562377+0000","last_undegraded":"2023-03-03T07:58:14.562377+0000","last_fullsized":"2023-03-03T07:58:14.562377+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:57:03.944467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"52'1","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888798+0000","last_change":"2023-03-03T07:58:12.679407+0000","last_active":"2023-03-03T07:58:14.888798+0000","last_peered":"2023-03-03T07:58:14.888798+0000","last_clean":"2023-03-03T07:58:14.888798+0000","last_became_active":"2023-03-03T07:58:12.679060+0000","last_became_peered":"2023-03-03T07:58:12.679060+0000","last_unstale":"2023-03-03T07:58:14.888798+0000","last_undegraded":"2023-03-03T07:58:14.888798+0000","last_fullsized":"2023-03-03T07:58:14.888798+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:28.227840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":37,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.224399+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T07:58:13.224399+0000","last_peered":"2023-03-03T07:58:13.224399+0000","last_clean":"2023-03-03T07:58:13.224399+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T07:58:13.224399+0000","last_undegraded":"2023-03-03T07:58:13.224399+0000","last_fullsized":"2023-03-03T07:58:13.224399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":191,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.225751+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T07:58:13.225751+0000","last_peered":"2023-03-03T07:58:13.225751+0000","last_clean":"2023-03-03T07:58:13.225751+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T07:58:13.225751+0000","last_undegraded":"2023-03-03T07:58:13.225751+0000","last_fullsized":"2023-03-03T07:58:13.225751+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.256134+0000","last_change":"2023-03-03T07:58:13.243833+0000","last_active":"2023-03-03T07:58:13.256134+0000","last_peered":"2023-03-03T07:58:13.256134+0000","last_clean":"2023-03-03T07:58:13.256134+0000","last_became_active":"2023-03-03T07:58:13.243561+0000","last_became_peered":"2023-03-03T07:58:13.243561+0000","last_unstale":"2023-03-03T07:58:13.256134+0000","last_undegraded":"2023-03-03T07:58:13.256134+0000","last_fullsized":"2023-03-03T07:58:13.256134+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:39:44.094387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.228124+0000","last_change":"2023-03-03T07:58:12.678634+0000","last_active":"2023-03-03T07:58:13.228124+0000","last_peered":"2023-03-03T07:58:13.228124+0000","last_clean":"2023-03-03T07:58:13.228124+0000","last_became_active":"2023-03-03T07:58:12.678365+0000","last_became_peered":"2023-03-03T07:58:12.678365+0000","last_unstale":"2023-03-03T07:58:13.228124+0000","last_undegraded":"2023-03-03T07:58:13.228124+0000","last_fullsized":"2023-03-03T07:58:13.228124+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:03:44.817102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":38,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.225764+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T07:58:13.225764+0000","last_peered":"2023-03-03T07:58:13.225764+0000","last_clean":"2023-03-03T07:58:13.225764+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T07:58:13.225764+0000","last_undegraded":"2023-03-03T07:58:13.225764+0000","last_fullsized":"2023-03-03T07:58:13.225764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:15.883844+0000","last_change":"2023-03-03T07:58:13.240499+0000","last_active":"2023-03-03T07:58:15.883844+0000","last_peered":"2023-03-03T07:58:15.883844+0000","last_clean":"2023-03-03T07:58:15.883844+0000","last_became_active":"2023-03-03T07:58:13.240221+0000","last_became_peered":"2023-03-03T07:58:13.240221+0000","last_unstale":"2023-03-03T07:58:15.883844+0000","last_undegraded":"2023-03-03T07:58:15.883844+0000","last_fullsized":"2023-03-03T07:58:15.883844+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:39:10.089272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.224190+0000","last_change":"2023-03-03T07:58:12.677009+0000","last_active":"2023-03-03T07:58:13.224190+0000","last_peered":"2023-03-03T07:58:13.224190+0000","last_clean":"2023-03-03T07:58:13.224190+0000","last_became_active":"2023-03-03T07:58:12.676714+0000","last_became_peered":"2023-03-03T07:58:12.676714+0000","last_unstale":"2023-03-03T07:58:13.224190+0000","last_undegraded":"2023-03-03T07:58:13.224190+0000","last_fullsized":"2023-03-03T07:58:13.224190+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:52:22.004977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":38,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.228103+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T07:58:13.228103+0000","last_peered":"2023-03-03T07:58:13.228103+0000","last_clean":"2023-03-03T07:58:13.228103+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T07:58:13.228103+0000","last_undegraded":"2023-03-03T07:58:13.228103+0000","last_fullsized":"2023-03-03T07:58:13.228103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.888698+0000","last_change":"2023-03-03T07:58:13.253140+0000","last_active":"2023-03-03T07:58:14.888698+0000","last_peered":"2023-03-03T07:58:14.888698+0000","last_clean":"2023-03-03T07:58:14.888698+0000","last_became_active":"2023-03-03T07:58:13.252705+0000","last_became_peered":"2023-03-03T07:58:13.252705+0000","last_unstale":"2023-03-03T07:58:14.888698+0000","last_undegraded":"2023-03-03T07:58:14.888698+0000","last_fullsized":"2023-03-03T07:58:14.888698+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:02.109028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.224588+0000","last_change":"2023-03-03T07:58:12.674191+0000","last_active":"2023-03-03T07:58:13.224588+0000","last_peered":"2023-03-03T07:58:13.224588+0000","last_clean":"2023-03-03T07:58:13.224588+0000","last_became_active":"2023-03-03T07:58:12.673933+0000","last_became_peered":"2023-03-03T07:58:12.673933+0000","last_unstale":"2023-03-03T07:58:13.224588+0000","last_undegraded":"2023-03-03T07:58:13.224588+0000","last_fullsized":"2023-03-03T07:58:13.224588+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:58.525319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":36,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-03-03T07:58:12.663876+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T07:58:12.663876+0000","last_peered":"2023-03-03T07:58:12.663876+0000","last_clean":"2023-03-03T07:58:12.663876+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T07:58:12.663876+0000","last_undegraded":"2023-03-03T07:58:12.663876+0000","last_fullsized":"2023-03-03T07:58:12.663876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-03-03T07:58:14.225674+0000","last_change":"2023-03-03T07:58:13.241119+0000","last_active":"2023-03-03T07:58:14.225674+0000","last_peered":"2023-03-03T07:58:14.225674+0000","last_clean":"2023-03-03T07:58:14.225674+0000","last_became_active":"2023-03-03T07:58:13.240240+0000","last_became_peered":"2023-03-03T07:58:13.240240+0000","last_unstale":"2023-03-03T07:58:14.225674+0000","last_undegraded":"2023-03-03T07:58:14.225674+0000","last_fullsized":"2023-03-03T07:58:14.225674+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:12.651095+0000","last_clean_scrub_stamp":"2023-03-03T07:58:12.651095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:05:48.495144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.223500+0000","last_change":"2023-03-03T07:58:12.670132+0000","last_active":"2023-03-03T07:58:13.223500+0000","last_peered":"2023-03-03T07:58:13.223500+0000","last_clean":"2023-03-03T07:58:13.223500+0000","last_became_active":"2023-03-03T07:58:12.669843+0000","last_became_peered":"2023-03-03T07:58:12.669843+0000","last_unstale":"2023-03-03T07:58:13.223500+0000","last_undegraded":"2023-03-03T07:58:13.223500+0000","last_fullsized":"2023-03-03T07:58:13.223500+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:58:11.643076+0000","last_clean_scrub_stamp":"2023-03-03T07:58:11.643076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:29:58.100844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":37,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-03-03T07:58:13.233213+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T07:58:13.233213+0000","last_peered":"2023-03-03T07:58:13.233213+0000","last_clean":"2023-03-03T07:58:13.233213+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T07:58:13.233213+0000","last_undegraded":"2023-03-03T07:58:13.233213+0000","last_fullsized":"2023-03-03T07:58:13.233213+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10348,"kb_used_data":1188,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94361492,"statfs":{"total":96636764160,"available":96626167808,"internally_reserved":0,"allocated":1216512,"data_stored":856822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.70299999999999996,"15min":0.71499999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88200000000000001,"5min":1.546,"15min":1.546},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.72899999999999998,"15min":0.73499999999999999},"min":{"1min":0.54900000000000004,"5min":0.5,"15min":0.5},"max":{"1min":0.89400000000000002,"5min":1.702,"15min":1.702},"last":0.79400000000000004}]},{"osd":1,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":1.073,"15min":1.145},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.873,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.77300000000000002,"15min":0.78000000000000003},"min":{"1min":0.63700000000000001,"5min":0.504,"15min":0.504},"max":{"1min":0.92500000000000004,"5min":1.944,"15min":1.944},"last":0.63700000000000001}]},{"osd":2,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.98599999999999999,"5min":1.252,"15min":1.252},"last":0.753},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.79200000000000004,"15min":0.80100000000000005},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.016,"5min":1.329,"15min":1.329},"last":0.66300000000000003}]},{"osd":3,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.71099999999999997,"15min":0.71199999999999997},"min":{"1min":0.53500000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.92500000000000004,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.78600000000000003,"15min":0.80000000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0640000000000001,"5min":1.294,"15min":1.294},"last":0.55500000000000005}]},{"osd":4,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.66700000000000004,"15min":0.66300000000000003},"min":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92700000000000005,"5min":1.077,"15min":1.077},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.63400000000000001,"15min":0.623},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.69199999999999995}]},{"osd":5,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.79500000000000004,"15min":0.80900000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0409999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.82899999999999996,"15min":0.82799999999999996},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.0940000000000001,"5min":1.349,"15min":1.349},"last":0.876}]},{"osd":6,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.83699999999999997,"15min":0.83599999999999997},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.048,"5min":1.204,"15min":1.204},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.81399999999999995,"15min":0.82199999999999995},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0169999999999999,"5min":1.194,"15min":1.194},"last":0.80700000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8868,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362972,"statfs":{"total":96636764160,"available":96627683328,"internally_reserved":0,"allocated":618496,"data_stored":265947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.68700000000000006,"15min":0.68200000000000005},"min":{"1min":0.42899999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.69399999999999995,"15min":0.69599999999999995},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96799999999999997,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.57299999999999995}]},{"osd":1,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.69899999999999995,"15min":0.68200000000000005},"min":{"1min":0.47699999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":3.323,"5min":3.323,"15min":3.323},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.046,"5min":1.135,"15min":1.135},"last":0.86699999999999999}]},{"osd":2,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.73999999999999999,"15min":0.72799999999999998},"min":{"1min":0.57399999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.54800000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.73599999999999999}]},{"osd":3,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.82299999999999995,"15min":0.79300000000000004},"min":{"1min":0.57099999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":4.8959999999999999,"5min":4.8959999999999999,"15min":4.8959999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.77500000000000002,"15min":0.77200000000000002},"min":{"1min":0.61099999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.2270000000000001,"5min":1.343,"15min":1.343},"last":0.71799999999999997}]},{"osd":4,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.92900000000000005,"15min":0.95499999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.087,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.93899999999999995,"15min":0.96799999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92100000000000004,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.76000000000000001}]},{"osd":5,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.78300000000000003,"15min":0.79500000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.86299999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.83299999999999996,"15min":0.83499999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.1279999999999999,"5min":1.429,"15min":1.429},"last":1.02}]},{"osd":7,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.85399999999999998,"15min":0.85099999999999998},"min":{"1min":0.67000000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.093,"5min":1.452,"15min":1.452},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.877,"15min":0.88900000000000001},"min":{"1min":0.59099999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.135,"5min":1.579,"15min":1.579},"last":0.877}]}]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8988,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362852,"statfs":{"total":96636764160,"available":96627560448,"internally_reserved":0,"allocated":610304,"data_stored":259837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.505,"15min":0.51500000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.59399999999999997,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.63700000000000001,"15min":0.66600000000000004},"min":{"1min":0.36499999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.60799999999999998,"5min":3.972,"15min":3.972},"last":0.52900000000000003}]},{"osd":2,"last update":"Fri Mar 3 07:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.95899999999999996,"15min":1.0369999999999999},"min":{"1min":0.34899999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.621,"5min":11.897,"15min":11.897},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":1.0409999999999999,"15min":1.119},"min":{"1min":0.495,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.69999999999999996,"5min":11.804,"15min":11.804},"last":0.57999999999999996}]},{"osd":3,"last update":"Fri Mar 3 07:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":1.032,"15min":1.109},"min":{"1min":0.46300000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.70899999999999996,"5min":11.843999999999999,"15min":11.843999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.72699999999999998,"15min":0.76300000000000001},"min":{"1min":0.40100000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.64100000000000001,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.5}]},{"osd":4,"last update":"Fri Mar 3 07:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.86399999999999999,"15min":0.878},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1459999999999999,"5min":1.613,"15min":1.613},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.86099999999999999,"15min":0.86899999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.123,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.91800000000000004}]},{"osd":5,"last update":"Fri Mar 3 07:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.92500000000000004,"15min":0.95099999999999996},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":0.94599999999999995,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.873,"15min":0.89100000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0960000000000001,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.75900000000000001}]},{"osd":6,"last update":"Fri Mar 3 07:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.996,"15min":1.0269999999999999},"min":{"1min":0.64300000000000002,"5min":0.628,"15min":0.628},"max":{"1min":1.028,"5min":3.907,"15min":3.907},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.94199999999999995,"15min":0.95999999999999996},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.109,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.90800000000000003}]},{"osd":7,"last update":"Fri Mar 3 07:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.88100000000000001,"15min":0.89500000000000002},"min":{"1min":0.624,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.0069999999999999,"5min":1.419,"15min":1.419},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.76300000000000001,"5min":1.0049999999999999,"15min":1.0449999999999999},"min":{"1min":0.623,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":0.95699999999999996,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":0.97499999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10088,"kb_used_data":1184,"kb_used_omap":0,"kb_used_meta":8896,"kb_avail":94361752,"statfs":{"total":96636764160,"available":96626434048,"internally_reserved":0,"allocated":1212416,"data_stored":855735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9109504},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46700000000000003,"15min":0.47499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.60499999999999998,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.51100000000000001,"15min":0.51900000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.61899999999999999,"5min":0.871,"15min":0.871},"last":0.46899999999999997}]},{"osd":2,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.41899999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67300000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.40400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69599999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.58999999999999997}]},{"osd":3,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.68000000000000005,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.63800000000000001,"15min":0.64800000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78500000000000003,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.51100000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.84999999999999998,"15min":0.85399999999999998},"min":{"1min":0.64600000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.99299999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.58199999999999996,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.022,"5min":1.605,"15min":1.605},"last":0.78200000000000003}]},{"osd":5,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82899999999999996,"15min":0.82999999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.081,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.82999999999999996,"15min":0.82799999999999996},"min":{"1min":0.68799999999999994,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.0880000000000001,"5min":1.286,"15min":1.286},"last":0.82399999999999995}]},{"osd":6,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.88900000000000001,"15min":0.89700000000000002},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.0069999999999999,"5min":1.623,"15min":1.623},"last":0.754},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.87,"15min":0.877},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.127,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.76800000000000002}]},{"osd":7,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.88600000000000001,"15min":0.89600000000000002},"min":{"1min":0.57599999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.0569999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.86699999999999999,"15min":0.871},"min":{"1min":0.67700000000000005,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.071,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.73399999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8936,"kb_used_data":608,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94362904,"statfs":{"total":96636764160,"available":96627613696,"internally_reserved":0,"allocated":622592,"data_stored":266859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:57:31 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.87,"15min":0.94399999999999995},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.59799999999999998,"5min":5.7889999999999997,"15min":5.7889999999999997},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.72899999999999998,"15min":0.77200000000000002},"min":{"1min":0.374,"5min":0.372,"15min":0.372},"max":{"1min":0.63500000000000001,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":0.36799999999999999}]},{"osd":1,"last update":"Fri Mar 3 07:57:31 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.753,"15min":0.79400000000000004},"min":{"1min":0.34499999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.61099999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.002,"15min":1.0780000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.66100000000000003,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.49099999999999999}]},{"osd":3,"last update":"Fri Mar 3 07:57:31 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.2130000000000001,"15min":1.3300000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70199999999999996,"5min":12.961,"15min":12.961},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":1.028,"15min":1.1120000000000001},"min":{"1min":0.438,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.58899999999999997,"5min":6.1449999999999996,"15min":6.1449999999999996},"last":0.45200000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:57:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.2769999999999999,"15min":1.353},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":0.93100000000000005,"5min":6.484,"15min":6.484},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.2529999999999999,"15min":1.3200000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":0.98499999999999999,"5min":6.5510000000000002,"15min":6.5510000000000002},"last":1.0029999999999999}]},{"osd":5,"last update":"Fri Mar 3 07:57:31 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.1799999999999999,"15min":1.2490000000000001},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.97199999999999998,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.81699999999999995,"5min":1.238,"15min":1.3080000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.92900000000000005,"5min":6.6369999999999996,"15min":6.6369999999999996},"last":1.069}]},{"osd":6,"last update":"Fri Mar 3 07:57:31 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":1.236,"15min":1.304},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.0009999999999999,"5min":6.625,"15min":6.625},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":1.264,"15min":1.3360000000000001},"min":{"1min":0.72699999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":0.997,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.84799999999999998}]},{"osd":7,"last update":"Fri Mar 3 07:57:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":1.226,"15min":1.302},"min":{"1min":0.64700000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.89900000000000002,"5min":6.7009999999999996,"15min":6.7009999999999996},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.21,"15min":1.2769999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":0.995,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.82199999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9264,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":8640,"kb_avail":94362576,"statfs":{"total":96636764160,"available":96627277824,"internally_reserved":0,"allocated":630784,"data_stored":267900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8847360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.83399999999999996,"15min":0.90300000000000002},"min":{"1min":0.30599999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.626,"5min":9.5820000000000007,"15min":9.5820000000000007},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.90400000000000003,"15min":0.97399999999999998},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.63500000000000001,"5min":9.7840000000000007,"15min":9.7840000000000007},"last":0.70099999999999996}]},{"osd":1,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.33900000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.61399999999999999,"5min":1,"15min":1},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.98699999999999999,"15min":1.0669999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":9.702,"15min":9.702},"last":0.52000000000000002}]},{"osd":2,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.93300000000000005,"15min":0.999},"min":{"1min":0.38,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68300000000000005,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0189999999999999,"15min":1.095},"min":{"1min":0.42099999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.68000000000000005,"5min":9.734,"15min":9.734},"last":0.65500000000000003}]},{"osd":4,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":1.171,"15min":1.2350000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.052,"5min":9.8629999999999995,"15min":9.8629999999999995},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.871,"15min":0.85899999999999999},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":3.1070000000000002,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":0.96899999999999997}]},{"osd":5,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.60899999999999999,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1060000000000001,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.80300000000000005,"5min":1.2270000000000001,"15min":1.298},"min":{"1min":0.60199999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.95899999999999996,"5min":9.6600000000000001,"15min":9.6600000000000001},"last":1.02}]},{"osd":6,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":1.2430000000000001,"15min":1.3129999999999999},"min":{"1min":0.60299999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":0.99299999999999999,"5min":9.8190000000000008,"15min":9.8190000000000008},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.872,"15min":0.88200000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0589999999999999,"5min":1.609,"15min":1.609},"last":0.80300000000000005}]},{"osd":7,"last update":"Fri Mar 3 07:57:34 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":1.278,"15min":1.3560000000000001},"min":{"1min":0.63100000000000001,"5min":0.62,"15min":0.62},"max":{"1min":0.95099999999999996,"5min":9.625,"15min":9.625},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.872,"5min":1.2529999999999999,"15min":1.3169999999999999},"min":{"1min":0.66900000000000004,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.026,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":0.88900000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9124,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":8512,"kb_avail":94362716,"statfs":{"total":96636764160,"available":96627421184,"internally_reserved":0,"allocated":618496,"data_stored":266896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8716288},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.98599999999999999,"15min":1.028},"min":{"1min":0.48599999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.054,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66300000000000003,"15min":0.65900000000000003},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.66800000000000004}]},{"osd":1,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.119,"15min":1.1719999999999999},"min":{"1min":0.55900000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.153,"5min":9.5709999999999997,"15min":9.5709999999999997},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.91700000000000004,"15min":0.95299999999999996},"min":{"1min":0.55300000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.014,"5min":5.266,"15min":5.266},"last":0.68600000000000005}]},{"osd":2,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":1.075,"15min":1.1299999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90700000000000003,"5min":8.4529999999999994,"15min":8.4529999999999994},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.76400000000000001,"5min":1.004,"15min":1.044},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0880000000000001,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.73199999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73299999999999998,"15min":0.73399999999999999},"min":{"1min":0.53100000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.80400000000000005,"15min":0.80700000000000005},"min":{"1min":0.63700000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.069,"5min":1.389,"15min":1.389},"last":0.747}]},{"osd":5,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.746,"15min":0.751},"min":{"1min":0.53500000000000003,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.86299999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.72799999999999998,"15min":0.72199999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.69999999999999996}]},{"osd":6,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.61299999999999999,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.59299999999999997,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0409999999999999,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.63500000000000001}]},{"osd":7,"last update":"Fri Mar 3 07:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.82799999999999996,"15min":0.83199999999999996},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.119,"5min":1.679,"15min":1.679},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.92800000000000005,"15min":0.94399999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1080000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.77800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9052,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362788,"statfs":{"total":96636764160,"available":96627494912,"internally_reserved":0,"allocated":610304,"data_stored":265311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:57:33 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.39700000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.83899999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.73499999999999999,"15min":0.73699999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.877,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":2.528}]},{"osd":1,"last update":"Fri Mar 3 07:57:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90800000000000003,"5min":0.996,"15min":0.996},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.63,"15min":0.623},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.79600000000000004,"5min":0.879,"15min":0.879},"last":0.76800000000000002}]},{"osd":2,"last update":"Fri Mar 3 07:57:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67500000000000004,"15min":0.67700000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.80000000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.71399999999999997,"15min":0.71199999999999997},"min":{"1min":0.58399999999999996,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.85899999999999999,"5min":1.321,"15min":1.321},"last":0.79500000000000004}]},{"osd":3,"last update":"Fri Mar 3 07:57:33 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.68899999999999995,"15min":0.68600000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.85999999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.85299999999999998}]},{"osd":4,"last update":"Fri Mar 3 07:57:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.66500000000000004,"15min":0.66300000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90800000000000003,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.78600000000000003,"15min":0.78100000000000003},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.876}]},{"osd":6,"last update":"Fri Mar 3 07:57:33 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.79400000000000004,"15min":0.80800000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.98399999999999999,"5min":1.071,"15min":1.071},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.72099999999999997,"15min":0.72399999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94599999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.83299999999999996}]},{"osd":7,"last update":"Fri Mar 3 07:57:33 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.75900000000000001,"15min":0.75800000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.95899999999999996,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.83199999999999996,"15min":0.83799999999999997},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.94299999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.93200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T07:58:18.773 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-03-03T07:58:18.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2023-03-03T07:58:19.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.055+0000 7f98b17ff700 1 Processor -- start 2023-03-03T07:58:19.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.055+0000 7f98b17ff700 1 -- start start 2023-03-03T07:58:19.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.057+0000 7f98b17ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ac0b8d10 0x7f98ac0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:19.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.057+0000 7f98b17ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f98ac0b5310 0x7f98ac0d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:19.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.057+0000 7f98b17ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac0cd7d0 con 0x7f98ac0b26d0 2023-03-03T07:58:19.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.057+0000 7f98b17ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ac0b90e0 con 0x7f98ac0b5310 2023-03-03T07:58:19.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.057+0000 7f98b17ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac0cda20 con 0x7f98ac0b8d10 2023-03-03T07:58:19.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.057+0000 7f98a3fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f98ac0b26d0 0x7f98ac0b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56786/0 (socket says 172.21.15.98:56786) 2023-03-03T07:58:19.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.057+0000 7f98a3fff700 1 -- 172.21.15.98:0/2262911664 learned_addr learned my addr 172.21.15.98:0/2262911664 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:19.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.057+0000 7f98aaffd700 1 --2- 172.21.15.98:0/2262911664 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ac0b8d10 0x7f98ac0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:19.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.057+0000 7f98ab7fe700 1 --2- 172.21.15.98:0/2262911664 >> v2:172.21.15.100:3300/0 conn(0x7f98ac0b5310 0x7f98ac0d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:19.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.058+0000 7f98aa7fc700 1 -- 172.21.15.98:0/2262911664 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226621112 0 0) 0x7f98ac0cd7d0 con 0x7f98ac0b26d0 2023-03-03T07:58:19.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.058+0000 7f98aa7fc700 1 -- 172.21.15.98:0/2262911664 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98980035b0 con 0x7f98ac0b26d0 2023-03-03T07:58:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.058+0000 7f98aaffd700 1 -- 172.21.15.98:0/2262911664 >> v2:172.21.15.100:3300/0 conn(0x7f98ac0b5310 msgr2=0x7f98ac0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.058+0000 7f98aaffd700 1 --2- 172.21.15.98:0/2262911664 >> v2:172.21.15.100:3300/0 conn(0x7f98ac0b5310 0x7f98ac0d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.058+0000 7f98aaffd700 1 -- 172.21.15.98:0/2262911664 >> v1:172.21.15.98:6789/0 conn(0x7f98ac0b26d0 legacy=0x7f98ac0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.058+0000 7f98aaffd700 1 -- 172.21.15.98:0/2262911664 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac0cdd00 con 0x7f98ac0b8d10 2023-03-03T07:58:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.058+0000 7f98aaffd700 1 --2- 172.21.15.98:0/2262911664 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ac0b8d10 0x7f98ac0cffc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f989c009840 tx=0x7f989c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a35dd439bcbb79e2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:19.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.058+0000 7f98aa7fc700 1 -- 172.21.15.98:0/2262911664 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c011830 con 0x7f98ac0b8d10 2023-03-03T07:58:19.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.059+0000 7f98aa7fc700 1 -- 172.21.15.98:0/2262911664 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c011990 con 0x7f98ac0b8d10 2023-03-03T07:58:19.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.059+0000 7f98aa7fc700 1 -- 172.21.15.98:0/2262911664 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c011d20 con 0x7f98ac0b8d10 2023-03-03T07:58:19.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.059+0000 7f98b17ff700 1 -- 172.21.15.98:0/2262911664 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ac0b8d10 msgr2=0x7f98ac0cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:19.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.059+0000 7f98b17ff700 1 --2- 172.21.15.98:0/2262911664 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ac0b8d10 0x7f98ac0cffc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f989c009840 tx=0x7f989c009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:19.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.059+0000 7f98b17ff700 1 -- 172.21.15.98:0/2262911664 shutdown_connections 2023-03-03T07:58:19.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.059+0000 7f98b17ff700 1 --2- 172.21.15.98:0/2262911664 >> v2:172.21.15.100:3300/0 conn(0x7f98ac0b5310 0x7f98ac0d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.059+0000 7f98b17ff700 1 --2- 172.21.15.98:0/2262911664 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ac0b8d10 0x7f98ac0cffc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.059+0000 7f98b17ff700 1 -- 172.21.15.98:0/2262911664 >> 172.21.15.98:0/2262911664 conn(0x7f98ac073e70 msgr2=0x7f98ac06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:19.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.059+0000 7f98b17ff700 1 -- 172.21.15.98:0/2262911664 shutdown_connections 2023-03-03T07:58:19.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.059+0000 7f98b17ff700 1 -- 172.21.15.98:0/2262911664 wait complete. 2023-03-03T07:58:19.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.061+0000 7f98b17ff700 1 Processor -- start 2023-03-03T07:58:19.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.061+0000 7f98b17ff700 1 -- start start 2023-03-03T07:58:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.062+0000 7f98b17ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ac0b26d0 0x7f98ac153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.062+0000 7f98b17ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ac0b5310 0x7f98ac0cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.062+0000 7f98b17ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98ac0b8d10 0x7f98ac151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.062+0000 7f98b17ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac15ed80 con 0x7f98ac0b26d0 2023-03-03T07:58:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.062+0000 7f98b17ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac15e450 con 0x7f98ac0b8d10 2023-03-03T07:58:19.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.062+0000 7f98b17ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac15e730 con 0x7f98ac0b5310 2023-03-03T07:58:19.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.063+0000 7f98aaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ac0b26d0 0x7f98ac153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:19.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.063+0000 7f98aaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ac0b26d0 0x7f98ac153660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38154/0 (socket says 172.21.15.98:38154) 2023-03-03T07:58:19.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.063+0000 7f98aaffd700 1 -- 172.21.15.98:0/3048493809 learned_addr learned my addr 172.21.15.98:0/3048493809 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:19.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.063+0000 7f98ab7fe700 1 --2- 172.21.15.98:0/3048493809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98ac0b8d10 0x7f98ac151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:19.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.063+0000 7f98a3fff700 1 --2- 172.21.15.98:0/3048493809 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ac0b5310 0x7f98ac0cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:19.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.063+0000 7f98aaffd700 1 -- 172.21.15.98:0/3048493809 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ac0b5310 msgr2=0x7f98ac0cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:19.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.063+0000 7f98aaffd700 1 --2- 172.21.15.98:0/3048493809 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ac0b5310 0x7f98ac0cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.063+0000 7f98aaffd700 1 -- 172.21.15.98:0/3048493809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98ac0b8d10 msgr2=0x7f98ac151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:19.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.063+0000 7f98aaffd700 1 --2- 172.21.15.98:0/3048493809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98ac0b8d10 0x7f98ac151d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.063+0000 7f98aaffd700 1 -- 172.21.15.98:0/3048493809 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f989c0094a0 con 0x7f98ac0b26d0 2023-03-03T07:58:19.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.064+0000 7f98aaffd700 1 --2- 172.21.15.98:0/3048493809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ac0b26d0 0x7f98ac153660 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f989c0090b0 tx=0x7f989c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:19.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.064+0000 7f98a37fe700 1 -- 172.21.15.98:0/3048493809 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c0036a0 con 0x7f98ac0b26d0 2023-03-03T07:58:19.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.064+0000 7f98a37fe700 1 -- 172.21.15.98:0/3048493809 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c01f070 con 0x7f98ac0b26d0 2023-03-03T07:58:19.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.064+0000 7f98a37fe700 1 -- 172.21.15.98:0/3048493809 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c003a40 con 0x7f98ac0b26d0 2023-03-03T07:58:19.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.064+0000 7f98b17ff700 1 -- 172.21.15.98:0/3048493809 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ac0cdfe0 con 0x7f98ac0b26d0 2023-03-03T07:58:19.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.064+0000 7f98b17ff700 1 -- 172.21.15.98:0/3048493809 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ac0d0e80 con 0x7f98ac0b26d0 2023-03-03T07:58:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.065+0000 7f98b17ff700 1 -- 172.21.15.98:0/3048493809 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ac0d0500 con 0x7f98ac0b26d0 2023-03-03T07:58:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.066+0000 7f98a37fe700 1 -- 172.21.15.98:0/3048493809 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f989c00f050 con 0x7f98ac0b26d0 2023-03-03T07:58:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.068+0000 7f98a37fe700 1 --2- 172.21.15.98:0/3048493809 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f988c0683c0 0x7f988c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.068+0000 7f98a37fe700 1 -- 172.21.15.98:0/3048493809 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f989c021070 con 0x7f98ac0b26d0 2023-03-03T07:58:19.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.068+0000 7f98a3fff700 1 --2- 172.21.15.98:0/3048493809 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f988c0683c0 0x7f988c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:19.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.069+0000 7f98a3fff700 1 --2- 172.21.15.98:0/3048493809 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f988c0683c0 0x7f988c06a870 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9894007fd0 tx=0x7f9894007dc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.072+0000 7f98a37fe700 1 -- 172.21.15.98:0/3048493809 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f989c053cd0 con 0x7f98ac0b26d0 2023-03-03T07:58:19.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.341+0000 7f98b17ff700 1 -- 172.21.15.98:0/3048493809 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7f98ac04e6e0 con 0x7f98ac0b26d0 2023-03-03T07:58:19.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.346+0000 7f98a37fe700 1 -- 172.21.15.98:0/3048493809 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v10) v1 ==== 184+0+0 (secure 0 0 0) 0x7f989c0572f0 con 0x7f98ac0b26d0 2023-03-03T07:58:19.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.349+0000 7f98b17ff700 1 -- 172.21.15.98:0/3048493809 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f988c0683c0 msgr2=0x7f988c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:19.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.349+0000 7f98b17ff700 1 --2- 172.21.15.98:0/3048493809 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f988c0683c0 0x7f988c06a870 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9894007fd0 tx=0x7f9894007dc0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.349+0000 7f98b17ff700 1 -- 172.21.15.98:0/3048493809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ac0b26d0 msgr2=0x7f98ac153660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:19.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.349+0000 7f98b17ff700 1 --2- 172.21.15.98:0/3048493809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ac0b26d0 0x7f98ac153660 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f989c0090b0 tx=0x7f989c004750 comp rx=0 tx=0).stop 2023-03-03T07:58:19.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.349+0000 7f98b17ff700 1 -- 172.21.15.98:0/3048493809 shutdown_connections 2023-03-03T07:58:19.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.349+0000 7f98b17ff700 1 --2- 172.21.15.98:0/3048493809 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f988c0683c0 0x7f988c06a870 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.349+0000 7f98b17ff700 1 --2- 172.21.15.98:0/3048493809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ac0b26d0 0x7f98ac153660 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.349+0000 7f98b17ff700 1 --2- 172.21.15.98:0/3048493809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98ac0b8d10 0x7f98ac151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.349+0000 7f98b17ff700 1 --2- 172.21.15.98:0/3048493809 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ac0b5310 0x7f98ac0cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.349+0000 7f98b17ff700 1 -- 172.21.15.98:0/3048493809 >> 172.21.15.98:0/3048493809 conn(0x7f98ac073e70 msgr2=0x7f98ac0b8030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:19.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.349+0000 7f98b17ff700 1 -- 172.21.15.98:0/3048493809 shutdown_connections 2023-03-03T07:58:19.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.349+0000 7f98b17ff700 1 -- 172.21.15.98:0/3048493809 wait complete. 2023-03-03T07:58:19.352 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.0 2023-03-03T07:58:19.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:58:19.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.620+0000 7fe38259d700 1 Processor -- start 2023-03-03T07:58:19.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.621+0000 7fe38259d700 1 -- start start 2023-03-03T07:58:19.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.622+0000 7fe38259d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe37c0d1650 0x7fe37c0d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:19.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.622+0000 7fe38259d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe37c0b4a00 0x7fe37c0b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:19.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.622+0000 7fe38259d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe37c0b0c50 con 0x7fe37c0d0010 2023-03-03T07:58:19.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.622+0000 7fe38259d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe37c0b0040 con 0x7fe37c0d1650 2023-03-03T07:58:19.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.622+0000 7fe38259d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe37c0b0320 con 0x7fe37c0b4a00 2023-03-03T07:58:19.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.622+0000 7fe380d9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe37c0d1650 0x7fe37c0d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:19.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.622+0000 7fe380d9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe37c0d1650 0x7fe37c0d03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38160/0 (socket says 172.21.15.98:38160) 2023-03-03T07:58:19.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.622+0000 7fe380d9a700 1 -- 172.21.15.98:0/1254306785 learned_addr learned my addr 172.21.15.98:0/1254306785 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:19.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.623+0000 7fe381d9c700 1 --2- 172.21.15.98:0/1254306785 >> v2:172.21.15.100:3300/0 conn(0x7fe37c0b4a00 0x7fe37c0b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:19.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.623+0000 7fe380d9a700 1 -- 172.21.15.98:0/1254306785 >> v2:172.21.15.100:3300/0 conn(0x7fe37c0b4a00 msgr2=0x7fe37c0b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:19.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.623+0000 7fe380d9a700 1 --2- 172.21.15.98:0/1254306785 >> v2:172.21.15.100:3300/0 conn(0x7fe37c0b4a00 0x7fe37c0b2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.623+0000 7fe380d9a700 1 -- 172.21.15.98:0/1254306785 >> v1:172.21.15.100:6789/0 conn(0x7fe37c0d0010 legacy=0x7fe37c0d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.623+0000 7fe380d9a700 1 -- 172.21.15.98:0/1254306785 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe37c0b0600 con 0x7fe37c0d1650 2023-03-03T07:58:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.623+0000 7fe380d9a700 1 --2- 172.21.15.98:0/1254306785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe37c0d1650 0x7fe37c0d03e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0c6e40 tx=0x7fe36c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=499341899f1ba050 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.623+0000 7fe373fff700 1 -- 172.21.15.98:0/1254306785 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36c011830 con 0x7fe37c0d1650 2023-03-03T07:58:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.624+0000 7fe373fff700 1 -- 172.21.15.98:0/1254306785 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36c011990 con 0x7fe37c0d1650 2023-03-03T07:58:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.624+0000 7fe373fff700 1 -- 172.21.15.98:0/1254306785 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36c011d20 con 0x7fe37c0d1650 2023-03-03T07:58:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.624+0000 7fe38259d700 1 -- 172.21.15.98:0/1254306785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe37c0d1650 msgr2=0x7fe37c0d03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.624+0000 7fe38259d700 1 --2- 172.21.15.98:0/1254306785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe37c0d1650 0x7fe37c0d03e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0c6e40 tx=0x7fe36c009bd0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.624+0000 7fe38259d700 1 -- 172.21.15.98:0/1254306785 shutdown_connections 2023-03-03T07:58:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.624+0000 7fe38259d700 1 --2- 172.21.15.98:0/1254306785 >> v2:172.21.15.100:3300/0 conn(0x7fe37c0b4a00 0x7fe37c0b2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.624+0000 7fe38259d700 1 --2- 172.21.15.98:0/1254306785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe37c0d1650 0x7fe37c0d03e0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.624+0000 7fe38259d700 1 -- 172.21.15.98:0/1254306785 >> 172.21.15.98:0/1254306785 conn(0x7fe37c073b00 msgr2=0x7fe37c0d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.624+0000 7fe38259d700 1 -- 172.21.15.98:0/1254306785 shutdown_connections 2023-03-03T07:58:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.624+0000 7fe38259d700 1 -- 172.21.15.98:0/1254306785 wait complete. 2023-03-03T07:58:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.626+0000 7fe38259d700 1 Processor -- start 2023-03-03T07:58:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.627+0000 7fe38259d700 1 -- start start 2023-03-03T07:58:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.628+0000 7fe38259d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe37c0b4a00 0x7fe37c158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.628+0000 7fe38259d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe37c0d0010 0x7fe37c0b8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.628+0000 7fe38259d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe37c0d1650 0x7fe37c1e5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.628+0000 7fe38259d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37c0b61a0 con 0x7fe37c0d0010 2023-03-03T07:58:19.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.628+0000 7fe38259d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37c0b63f0 con 0x7fe37c0d1650 2023-03-03T07:58:19.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.628+0000 7fe38259d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe37c0b66d0 con 0x7fe37c0b4a00 2023-03-03T07:58:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.628+0000 7fe380d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe37c0d0010 0x7fe37c0b8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.628+0000 7fe380d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe37c0d0010 0x7fe37c0b8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38176/0 (socket says 172.21.15.98:38176) 2023-03-03T07:58:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.628+0000 7fe380d9a700 1 -- 172.21.15.98:0/2409700032 learned_addr learned my addr 172.21.15.98:0/2409700032 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.628+0000 7fe381d9c700 1 --2- 172.21.15.98:0/2409700032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe37c0d1650 0x7fe37c1e5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.628+0000 7fe38159b700 1 --2- 172.21.15.98:0/2409700032 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe37c0b4a00 0x7fe37c158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:19.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.629+0000 7fe380d9a700 1 -- 172.21.15.98:0/2409700032 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe37c0b4a00 msgr2=0x7fe37c158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:19.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.629+0000 7fe380d9a700 1 --2- 172.21.15.98:0/2409700032 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe37c0b4a00 0x7fe37c158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.629+0000 7fe380d9a700 1 -- 172.21.15.98:0/2409700032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe37c0d1650 msgr2=0x7fe37c1e5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:19.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.629+0000 7fe380d9a700 1 --2- 172.21.15.98:0/2409700032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe37c0d1650 0x7fe37c1e5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.629+0000 7fe380d9a700 1 -- 172.21.15.98:0/2409700032 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c009550 con 0x7fe37c0d0010 2023-03-03T07:58:19.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.629+0000 7fe380d9a700 1 --2- 172.21.15.98:0/2409700032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe37c0d0010 0x7fe37c0b8e00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe36c009c10 tx=0x7fe36c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:19.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.629+0000 7fe371ffb700 1 -- 172.21.15.98:0/2409700032 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36c004510 con 0x7fe37c0d0010 2023-03-03T07:58:19.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.629+0000 7fe371ffb700 1 -- 172.21.15.98:0/2409700032 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36c01e070 con 0x7fe37c0d0010 2023-03-03T07:58:19.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.629+0000 7fe38259d700 1 -- 172.21.15.98:0/2409700032 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe37c0d0b30 con 0x7fe37c0d0010 2023-03-03T07:58:19.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.629+0000 7fe371ffb700 1 -- 172.21.15.98:0/2409700032 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36c0038e0 con 0x7fe37c0d0010 2023-03-03T07:58:19.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.629+0000 7fe38259d700 1 -- 172.21.15.98:0/2409700032 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe37c0b08e0 con 0x7fe37c0d0010 2023-03-03T07:58:19.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.630+0000 7fe38259d700 1 -- 172.21.15.98:0/2409700032 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe37c0d0cc0 con 0x7fe37c0d0010 2023-03-03T07:58:19.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.632+0000 7fe371ffb700 1 -- 172.21.15.98:0/2409700032 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fe36c00f050 con 0x7fe37c0d0010 2023-03-03T07:58:19.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.633+0000 7fe371ffb700 1 --2- 172.21.15.98:0/2409700032 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3600683c0 0x7fe36006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:19.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.633+0000 7fe371ffb700 1 -- 172.21.15.98:0/2409700032 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe36c086c50 con 0x7fe37c0d0010 2023-03-03T07:58:19.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.633+0000 7fe38159b700 1 --2- 172.21.15.98:0/2409700032 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3600683c0 0x7fe36006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:19.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.634+0000 7fe38159b700 1 --2- 172.21.15.98:0/2409700032 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3600683c0 0x7fe36006a870 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0b18d0 tx=0x7fe378009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:19.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.637+0000 7fe371ffb700 1 -- 172.21.15.98:0/2409700032 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe36c0532f0 con 0x7fe37c0d0010 2023-03-03T07:58:19.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.874+0000 7fe38259d700 1 -- 172.21.15.98:0/2409700032 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe37c0729b0 con 0x7fe37c0d0010 2023-03-03T07:58:19.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.875+0000 7fe371ffb700 1 -- 172.21.15.98:0/2409700032 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4654 (secure 0 0 0) 0x7fe36c020090 con 0x7fe37c0d0010 2023-03-03T07:58:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:19.877 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:58:19.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.877+0000 7fe38259d700 1 -- 172.21.15.98:0/2409700032 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3600683c0 msgr2=0x7fe36006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:19.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.877+0000 7fe38259d700 1 --2- 172.21.15.98:0/2409700032 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3600683c0 0x7fe36006a870 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0b18d0 tx=0x7fe378009040 comp rx=0 tx=0).stop 2023-03-03T07:58:19.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.877+0000 7fe38259d700 1 -- 172.21.15.98:0/2409700032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe37c0d0010 msgr2=0x7fe37c0b8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:19.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.877+0000 7fe38259d700 1 --2- 172.21.15.98:0/2409700032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe37c0d0010 0x7fe37c0b8e00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe36c009c10 tx=0x7fe36c004750 comp rx=0 tx=0).stop 2023-03-03T07:58:19.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.877+0000 7fe38259d700 1 -- 172.21.15.98:0/2409700032 shutdown_connections 2023-03-03T07:58:19.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.877+0000 7fe38259d700 1 --2- 172.21.15.98:0/2409700032 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe3600683c0 0x7fe36006a870 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.877+0000 7fe38259d700 1 --2- 172.21.15.98:0/2409700032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe37c0d1650 0x7fe37c1e5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.877+0000 7fe38259d700 1 --2- 172.21.15.98:0/2409700032 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe37c0b4a00 0x7fe37c158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.877+0000 7fe38259d700 1 --2- 172.21.15.98:0/2409700032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe37c0d0010 0x7fe37c0b8e00 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:19.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.877+0000 7fe38259d700 1 -- 172.21.15.98:0/2409700032 >> 172.21.15.98:0/2409700032 conn(0x7fe37c073b00 msgr2=0x7fe37c0b3340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:19.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.878+0000 7fe38259d700 1 -- 172.21.15.98:0/2409700032 shutdown_connections 2023-03-03T07:58:19.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:19.878+0000 7fe38259d700 1 -- 172.21.15.98:0/2409700032 wait complete. 2023-03-03T07:58:19.881 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2023-03-03T07:58:20.308 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-03-03T07:58:13.214065+0000', 'modified': '2023-03-03T07:58:16.671988+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5745}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5745': {'gid': 5745, 'name': 'a', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.98:6837/1111074388', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 1111074388}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 1111074388}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-03-03T07:58:20.309 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-03-03T07:58:20.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-03-03T07:58:20.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.554+0000 7f16e759e700 1 Processor -- start 2023-03-03T07:58:20.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.554+0000 7f16e759e700 1 -- start start 2023-03-03T07:58:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.556+0000 7f16e759e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16e81609b0 0x7f16e8160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.556+0000 7f16e759e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16e81612c0 con 0x7f16e80cf570 2023-03-03T07:58:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.556+0000 7f16e759e700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16e815dd00 con 0x7f16e80b4aa0 2023-03-03T07:58:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.556+0000 7f16e759e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16e80cebb0 con 0x7f16e81609b0 2023-03-03T07:58:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.556+0000 7f16e5d9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f16e80cf570 0x7f16e815d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56790/0 (socket says 172.21.15.98:56790) 2023-03-03T07:58:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.556+0000 7f16e5d9b700 1 -- 172.21.15.98:0/3991644602 learned_addr learned my addr 172.21.15.98:0/3991644602 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.556+0000 7f16e659c700 1 --2- 172.21.15.98:0/3991644602 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16e81609b0 0x7f16e8160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.557+0000 7f16e559a700 1 -- 172.21.15.98:0/3991644602 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499865197 0 0) 0x7f16e81612c0 con 0x7f16e80cf570 2023-03-03T07:58:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.557+0000 7f16e559a700 1 -- 172.21.15.98:0/3991644602 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16d00035b0 con 0x7f16e80cf570 2023-03-03T07:58:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.557+0000 7f16e559a700 1 -- 172.21.15.98:0/3991644602 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798846314 0 0) 0x7f16e815dd00 con 0x7f16e80b4aa0 2023-03-03T07:58:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.557+0000 7f16e559a700 1 -- 172.21.15.98:0/3991644602 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16e81612c0 con 0x7f16e80b4aa0 2023-03-03T07:58:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.557+0000 7f16e659c700 1 -- 172.21.15.98:0/3991644602 >> v1:172.21.15.100:6789/0 conn(0x7f16e80b4aa0 legacy=0x7f16e80d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.557+0000 7f16e659c700 1 -- 172.21.15.98:0/3991644602 >> v1:172.21.15.98:6789/0 conn(0x7f16e80cf570 legacy=0x7f16e815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:20.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.557+0000 7f16e659c700 1 -- 172.21.15.98:0/3991644602 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e80cd780 con 0x7f16e81609b0 2023-03-03T07:58:20.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.557+0000 7f16e659c700 1 --2- 172.21.15.98:0/3991644602 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16e81609b0 0x7f16e8160d80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f16d8009870 tx=0x7f16d8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fe4e7c49ae353 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:20.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.558+0000 7f16e559a700 1 -- 172.21.15.98:0/3991644602 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d8011830 con 0x7f16e81609b0 2023-03-03T07:58:20.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.558+0000 7f16e559a700 1 -- 172.21.15.98:0/3991644602 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d8011990 con 0x7f16e81609b0 2023-03-03T07:58:20.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.558+0000 7f16e559a700 1 -- 172.21.15.98:0/3991644602 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d8011d00 con 0x7f16e81609b0 2023-03-03T07:58:20.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.558+0000 7f16e759e700 1 -- 172.21.15.98:0/3991644602 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16e81609b0 msgr2=0x7f16e8160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:20.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.558+0000 7f16e759e700 1 --2- 172.21.15.98:0/3991644602 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16e81609b0 0x7f16e8160d80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f16d8009870 tx=0x7f16d8009b80 comp rx=0 tx=0).stop 2023-03-03T07:58:20.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.558+0000 7f16e759e700 1 -- 172.21.15.98:0/3991644602 shutdown_connections 2023-03-03T07:58:20.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.558+0000 7f16e759e700 1 --2- 172.21.15.98:0/3991644602 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16e81609b0 0x7f16e8160d80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.558+0000 7f16e759e700 1 -- 172.21.15.98:0/3991644602 >> 172.21.15.98:0/3991644602 conn(0x7f16e806b880 msgr2=0x7f16e80cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:20.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.559+0000 7f16e759e700 1 -- 172.21.15.98:0/3991644602 shutdown_connections 2023-03-03T07:58:20.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.559+0000 7f16e759e700 1 -- 172.21.15.98:0/3991644602 wait complete. 2023-03-03T07:58:20.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.561+0000 7f16e759e700 1 Processor -- start 2023-03-03T07:58:20.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.561+0000 7f16e759e700 1 -- start start 2023-03-03T07:58:20.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.562+0000 7f16e759e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16e80b4aa0 0x7f16e80b69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.562+0000 7f16e759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16e80cf570 0x7f16e80c7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.562+0000 7f16e759e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16e81609b0 0x7f16e80cc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.562+0000 7f16e759e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e80b6f10 con 0x7f16e80cf570 2023-03-03T07:58:20.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.562+0000 7f16e759e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e80c7c90 con 0x7f16e81609b0 2023-03-03T07:58:20.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.562+0000 7f16e759e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16e80c7f70 con 0x7f16e80b4aa0 2023-03-03T07:58:20.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.562+0000 7f16e5d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16e80cf570 0x7f16e80c7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.562+0000 7f16e659c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16e80b4aa0 0x7f16e80b69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.562+0000 7f16e5d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16e80cf570 0x7f16e80c7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38184/0 (socket says 172.21.15.98:38184) 2023-03-03T07:58:20.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.562+0000 7f16e5d9b700 1 -- 172.21.15.98:0/1749043656 learned_addr learned my addr 172.21.15.98:0/1749043656 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:20.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.562+0000 7f16e6d9d700 1 --2- 172.21.15.98:0/1749043656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16e81609b0 0x7f16e80cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.563+0000 7f16e5d9b700 1 -- 172.21.15.98:0/1749043656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16e80b4aa0 msgr2=0x7f16e80b69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:20.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.563+0000 7f16e5d9b700 1 --2- 172.21.15.98:0/1749043656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16e80b4aa0 0x7f16e80b69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.563+0000 7f16e5d9b700 1 -- 172.21.15.98:0/1749043656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16e81609b0 msgr2=0x7f16e80cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:20.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.563+0000 7f16e5d9b700 1 --2- 172.21.15.98:0/1749043656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16e81609b0 0x7f16e80cc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.563+0000 7f16e5d9b700 1 -- 172.21.15.98:0/1749043656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d80094a0 con 0x7f16e80cf570 2023-03-03T07:58:20.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.563+0000 7f16e6d9d700 1 --2- 172.21.15.98:0/1749043656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16e81609b0 0x7f16e80cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:20.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.563+0000 7f16e5d9b700 1 --2- 172.21.15.98:0/1749043656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16e80cf570 0x7f16e80c7360 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f16dc00c700 tx=0x7f16dc00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:20.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.563+0000 7f16d6ffd700 1 -- 172.21.15.98:0/1749043656 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16dc005d70 con 0x7f16e80cf570 2023-03-03T07:58:20.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.563+0000 7f16d6ffd700 1 -- 172.21.15.98:0/1749043656 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16dc0070d0 con 0x7f16e80cf570 2023-03-03T07:58:20.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.563+0000 7f16d6ffd700 1 -- 172.21.15.98:0/1749043656 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16dc007460 con 0x7f16e80cf570 2023-03-03T07:58:20.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.563+0000 7f16e759e700 1 -- 172.21.15.98:0/1749043656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e80cdd40 con 0x7f16e80cf570 2023-03-03T07:58:20.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.563+0000 7f16e759e700 1 -- 172.21.15.98:0/1749043656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e80cda60 con 0x7f16e80cf570 2023-03-03T07:58:20.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.566+0000 7f16d6ffd700 1 -- 172.21.15.98:0/1749043656 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f16dc019030 con 0x7f16e80cf570 2023-03-03T07:58:20.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.567+0000 7f16d6ffd700 1 --2- 172.21.15.98:0/1749043656 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16c80683c0 0x7f16c806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.568+0000 7f16d6ffd700 1 -- 172.21.15.98:0/1749043656 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f16dc085290 con 0x7f16e80cf570 2023-03-03T07:58:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.568+0000 7f16e659c700 1 --2- 172.21.15.98:0/1749043656 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16c80683c0 0x7f16c806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.569+0000 7f16e659c700 1 --2- 172.21.15.98:0/1749043656 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16c80683c0 0x7f16c806a870 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f16d80090b0 tx=0x7f16d8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.570+0000 7f1700958b80 1 Processor -- start 2023-03-03T07:58:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.570+0000 7f1700958b80 1 -- start start 2023-03-03T07:58:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f1700958b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e9709a6ad0 0x55e9709a6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f1700958b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e970988ff0 0x55e9709cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f1700958b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e9709875c0 0x55e970987a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f1700958b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9708fc4f0 con 0x55e970988ff0 2023-03-03T07:58:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f1700958b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e970871b60 con 0x55e9709875c0 2023-03-03T07:58:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f1700958b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e9708e0ad0 con 0x55e9709a6ad0 2023-03-03T07:58:20.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f16e5d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e970988ff0 0x55e9709cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f16e5d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e970988ff0 0x55e9709cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38190/0 (socket says 172.21.15.98:38190) 2023-03-03T07:58:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f16e5d9b700 1 -- 172.21.15.98:0/1957175155 learned_addr learned my addr 172.21.15.98:0/1957175155 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f16e659c700 1 --2- 172.21.15.98:0/1957175155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e9709a6ad0 0x55e9709a6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f16e6d9d700 1 --2- 172.21.15.98:0/1957175155 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e9709875c0 0x55e970987a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f16e5d9b700 1 -- 172.21.15.98:0/1957175155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e9709a6ad0 msgr2=0x55e9709a6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f16e5d9b700 1 --2- 172.21.15.98:0/1957175155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e9709a6ad0 0x55e9709a6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f16e5d9b700 1 -- 172.21.15.98:0/1957175155 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e9709875c0 msgr2=0x55e970987a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f16e5d9b700 1 --2- 172.21.15.98:0/1957175155 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e9709875c0 0x55e970987a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.571+0000 7f16e5d9b700 1 -- 172.21.15.98:0/1957175155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e9706ce540 con 0x55e970988ff0 2023-03-03T07:58:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.572+0000 7f16e5d9b700 1 --2- 172.21.15.98:0/1957175155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e970988ff0 0x55e9709cd7c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f16dc0552f0 tx=0x7f16dc055260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f851754d7a77d4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.572+0000 7f16c7fff700 1 -- 172.21.15.98:0/1957175155 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16dc059360 con 0x55e970988ff0 2023-03-03T07:58:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.572+0000 7f16c7fff700 1 -- 172.21.15.98:0/1957175155 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16dc0850d0 con 0x55e970988ff0 2023-03-03T07:58:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.572+0000 7f16c7fff700 1 -- 172.21.15.98:0/1957175155 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16dc0077c0 con 0x55e970988ff0 2023-03-03T07:58:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.573+0000 7f1700958b80 1 -- 172.21.15.98:0/1957175155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e970988ff0 msgr2=0x55e9709cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.573+0000 7f1700958b80 1 --2- 172.21.15.98:0/1957175155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e970988ff0 0x55e9709cd7c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f16dc0552f0 tx=0x7f16dc055260 comp rx=0 tx=0).stop 2023-03-03T07:58:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.573+0000 7f1700958b80 1 -- 172.21.15.98:0/1957175155 shutdown_connections 2023-03-03T07:58:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.573+0000 7f1700958b80 1 --2- 172.21.15.98:0/1957175155 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e9709875c0 0x55e970987a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.573+0000 7f1700958b80 1 --2- 172.21.15.98:0/1957175155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e9709a6ad0 0x55e9709a6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.573+0000 7f1700958b80 1 --2- 172.21.15.98:0/1957175155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e970988ff0 0x55e9709cd7c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.573+0000 7f1700958b80 1 -- 172.21.15.98:0/1957175155 >> 172.21.15.98:0/1957175155 conn(0x55e970984e10 msgr2=0x55e9709857a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.573+0000 7f1700958b80 1 -- 172.21.15.98:0/1957175155 shutdown_connections 2023-03-03T07:58:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.573+0000 7f1700958b80 1 -- 172.21.15.98:0/1957175155 wait complete. 2023-03-03T07:58:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.574+0000 7f1700958b80 1 -- start start 2023-03-03T07:58:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.574+0000 7f1700958b80 1 Processor -- start 2023-03-03T07:58:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.575+0000 7f1700958b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e9709875c0 0x55e9709d38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.575+0000 7f1700958b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e970988ff0 0x55e9709d1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.575+0000 7f1700958b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e9709a6ad0 0x55e9709d1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.575+0000 7f1700958b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9708d9e40 con 0x55e970988ff0 2023-03-03T07:58:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.575+0000 7f1700958b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9706cf140 con 0x55e9709875c0 2023-03-03T07:58:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.575+0000 7f1700958b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e9709d2560 con 0x55e9709a6ad0 2023-03-03T07:58:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.575+0000 7f1700958b80 10 client.0 _ll_register_callbacks cb 0x55e9708781c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T07:58:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.576+0000 7f16e5d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e970988ff0 0x55e9709d1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.576+0000 7f16e659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e9709875c0 0x55e9709d38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.576+0000 7f16e5d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e970988ff0 0x55e9709d1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38202/0 (socket says 172.21.15.98:38202) 2023-03-03T07:58:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.576+0000 7f16e5d9b700 1 -- 172.21.15.98:0/1852811057 learned_addr learned my addr 172.21.15.98:0/1852811057 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.576+0000 7f16e6d9d700 1 --2- 172.21.15.98:0/1852811057 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e9709a6ad0 0x55e9709d1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.576+0000 7f16e5d9b700 1 -- 172.21.15.98:0/1852811057 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e9709a6ad0 msgr2=0x55e9709d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.576+0000 7f16e5d9b700 1 --2- 172.21.15.98:0/1852811057 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e9709a6ad0 0x55e9709d1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.576+0000 7f16e5d9b700 1 -- 172.21.15.98:0/1852811057 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e9709875c0 msgr2=0x55e9709d38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.576+0000 7f16e5d9b700 1 --2- 172.21.15.98:0/1852811057 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e9709875c0 0x55e9709d38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.576+0000 7f16e5d9b700 1 -- 172.21.15.98:0/1852811057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e80cda60 con 0x55e970988ff0 2023-03-03T07:58:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.576+0000 7f16e5d9b700 1 --2- 172.21.15.98:0/1852811057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e970988ff0 0x55e9709d1220 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f16dc051af0 tx=0x7f16dc007ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d977bad2a139698d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.576+0000 7f16c57fa700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-03-03T07:58:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.577+0000 7f16c57fa700 1 -- 172.21.15.98:0/1852811057 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16dc083120 con 0x55e970988ff0 2023-03-03T07:58:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.577+0000 7f16c57fa700 1 -- 172.21.15.98:0/1852811057 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(52..52 src has 1..52) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f16dc083280 con 0x55e970988ff0 2023-03-03T07:58:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.577+0000 7f1700958b80 1 -- 172.21.15.98:0/1852811057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e970979ea0 con 0x55e970988ff0 2023-03-03T07:58:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.577+0000 7f16c57fa700 1 -- 172.21.15.98:0/1852811057 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16dc083440 con 0x55e970988ff0 2023-03-03T07:58:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.577+0000 7f1700958b80 10 client.5898 fetch_fsmap learned FSMap version 23 2023-03-03T07:58:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.577+0000 7f1700958b80 1 -- 172.21.15.98:0/1852811057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x55e970708b10 con 0x55e970988ff0 2023-03-03T07:58:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.577+0000 7f16c57fa700 1 -- 172.21.15.98:0/1852811057 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e970979ea0 con 0x55e970988ff0 2023-03-03T07:58:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.577+0000 7f16c57fa700 1 -- 172.21.15.98:0/1852811057 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 23) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f16dc064050 con 0x55e970988ff0 2023-03-03T07:58:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.577+0000 7f1700958b80 10 client.5898 fetch_fsmap finished waiting for FSMap version 23 2023-03-03T07:58:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.577+0000 7f1700958b80 10 client.5898 resolve_mds: resolved name 'a' to daemon mds.a 2023-03-03T07:58:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.577+0000 7f1700958b80 20 client.5898 populate_metadata read hostname 'smithi098' 2023-03-03T07:58:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.577+0000 7f1700958b80 1 --2- 172.21.15.98:0/1852811057 >> [v2:172.21.15.98:6835/1111074388,v1:172.21.15.98:6837/1111074388] conn(0x55e9709a5380 0x55e9709e96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.577+0000 7f1700958b80 4 client.5898 mds_command: new command op to 5745 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-03T07:58:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.577+0000 7f1700958b80 1 -- 172.21.15.98:0/1852811057 --> [v2:172.21.15.98:6835/1111074388,v1:172.21.15.98:6837/1111074388] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e9708d3020 con 0x55e9709a5380 2023-03-03T07:58:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.578+0000 7f16e659c700 1 --2- 172.21.15.98:0/1852811057 >> [v2:172.21.15.98:6835/1111074388,v1:172.21.15.98:6837/1111074388] conn(0x55e9709a5380 0x55e9709e96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.578+0000 7f16e659c700 1 --2- 172.21.15.98:0/1852811057 >> [v2:172.21.15.98:6835/1111074388,v1:172.21.15.98:6837/1111074388] conn(0x55e9709a5380 0x55e9709e96d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69e22aa55734778c server_cookie=ef37432d6d5d7b2a in_seq=0 out_seq=0 2023-03-03T07:58:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.578+0000 7f16c57fa700 10 client.5898 ms_handle_connect on v2:172.21.15.98:6835/1111074388 2023-03-03T07:58:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.579+0000 7f16c57fa700 1 -- 172.21.15.98:0/1852811057 <== mds.0 v2:172.21.15.98:6835/1111074388 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55e9708d3020 con 0x55e9709a5380 2023-03-03T07:58:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.579+0000 7f16c57fa700 10 client.5898 handle_command_reply: tid=0 2023-03-03T07:58:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.579+0000 7f1700958b80 1 client.5898 shutdown 2023-03-03T07:58:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.579+0000 7f1700958b80 1 -- 172.21.15.98:0/1852811057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e970988ff0 msgr2=0x55e9709d1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.579+0000 7f1700958b80 1 --2- 172.21.15.98:0/1852811057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e970988ff0 0x55e9709d1220 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f16dc051af0 tx=0x7f16dc007ba0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.580+0000 7f1700958b80 1 -- 172.21.15.98:0/1852811057 shutdown_connections 2023-03-03T07:58:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.580+0000 7f1700958b80 1 --2- 172.21.15.98:0/1852811057 >> [v2:172.21.15.98:6835/1111074388,v1:172.21.15.98:6837/1111074388] conn(0x55e9709a5380 0x55e9709e96d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.580+0000 7f1700958b80 1 --2- 172.21.15.98:0/1852811057 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e9709a6ad0 0x55e9709d1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.580+0000 7f1700958b80 1 --2- 172.21.15.98:0/1852811057 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e9709875c0 0x55e9709d38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.580+0000 7f1700958b80 1 --2- 172.21.15.98:0/1852811057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e970988ff0 0x55e9709d1220 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.580+0000 7f1700958b80 1 -- 172.21.15.98:0/1852811057 >> 172.21.15.98:0/1852811057 conn(0x55e970984e10 msgr2=0x55e970985f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.580+0000 7f16c57fa700 0 client.5898 ms_handle_reset on v2:172.21.15.98:6835/1111074388 2023-03-03T07:58:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.580+0000 7f1700958b80 1 -- 172.21.15.98:0/1852811057 shutdown_connections 2023-03-03T07:58:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.580+0000 7f1700958b80 1 -- 172.21.15.98:0/1852811057 wait complete. 2023-03-03T07:58:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.580+0000 7f1700958b80 20 client.5898 trim_cache size 0 max 16384 2023-03-03T07:58:20.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.598+0000 7f1700958b80 1 Processor -- start 2023-03-03T07:58:20.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.598+0000 7f1700958b80 1 -- start start 2023-03-03T07:58:20.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.599+0000 7f1700958b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e9709875c0 0x55e9709808f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.599+0000 7f1700958b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e970988ff0 0x55e970984060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.599+0000 7f1700958b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9709a6ad0 0x55e9709d2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.599+0000 7f1700958b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9709cd0b0 con 0x55e9709a6ad0 2023-03-03T07:58:20.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.599+0000 7f1700958b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9709b4bc0 con 0x55e9709875c0 2023-03-03T07:58:20.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.599+0000 7f1700958b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e9709827b0 con 0x55e970988ff0 2023-03-03T07:58:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.600+0000 7f16e6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9709a6ad0 0x55e9709d2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.600+0000 7f16e6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9709a6ad0 0x55e9709d2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38216/0 (socket says 172.21.15.98:38216) 2023-03-03T07:58:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.600+0000 7f16e6d9d700 1 -- 172.21.15.98:0/3058422820 learned_addr learned my addr 172.21.15.98:0/3058422820 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.600+0000 7f16e659c700 1 --2- 172.21.15.98:0/3058422820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e9709875c0 0x55e9709808f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.600+0000 7f16e5d9b700 1 --2- 172.21.15.98:0/3058422820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e970988ff0 0x55e970984060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.600+0000 7f16e6d9d700 1 -- 172.21.15.98:0/3058422820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e970988ff0 msgr2=0x55e970984060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.600+0000 7f16e6d9d700 1 --2- 172.21.15.98:0/3058422820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e970988ff0 0x55e970984060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.600+0000 7f16e6d9d700 1 -- 172.21.15.98:0/3058422820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e9709875c0 msgr2=0x55e9709808f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.600+0000 7f16e6d9d700 1 --2- 172.21.15.98:0/3058422820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e9709875c0 0x55e9709808f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.600+0000 7f16e6d9d700 1 -- 172.21.15.98:0/3058422820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e9709b4d80 con 0x55e9709a6ad0 2023-03-03T07:58:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.600+0000 7f16e6d9d700 1 --2- 172.21.15.98:0/3058422820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9709a6ad0 0x55e9709d2d90 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f16e0001af0 tx=0x7f16e0001e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28ab23e754ff3d6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.600+0000 7f16d4ff9700 1 -- 172.21.15.98:0/3058422820 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e0005210 con 0x55e9709a6ad0 2023-03-03T07:58:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.601+0000 7f16d4ff9700 1 -- 172.21.15.98:0/3058422820 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e0005370 con 0x55e9709a6ad0 2023-03-03T07:58:20.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.601+0000 7f16d4ff9700 1 -- 172.21.15.98:0/3058422820 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e0005720 con 0x55e9709a6ad0 2023-03-03T07:58:20.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.601+0000 7f1700958b80 1 -- 172.21.15.98:0/3058422820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9709a6ad0 msgr2=0x55e9709d2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:20.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.601+0000 7f1700958b80 1 --2- 172.21.15.98:0/3058422820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9709a6ad0 0x55e9709d2d90 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f16e0001af0 tx=0x7f16e0001e00 comp rx=0 tx=0).stop 2023-03-03T07:58:20.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.601+0000 7f1700958b80 1 -- 172.21.15.98:0/3058422820 shutdown_connections 2023-03-03T07:58:20.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.601+0000 7f1700958b80 1 --2- 172.21.15.98:0/3058422820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9709a6ad0 0x55e9709d2d90 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.601+0000 7f1700958b80 1 --2- 172.21.15.98:0/3058422820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e9709875c0 0x55e9709808f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.601+0000 7f1700958b80 1 --2- 172.21.15.98:0/3058422820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e970988ff0 0x55e970984060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.601+0000 7f1700958b80 1 -- 172.21.15.98:0/3058422820 >> 172.21.15.98:0/3058422820 conn(0x55e970984e10 msgr2=0x55e97097de00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:20.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.601+0000 7f1700958b80 1 -- 172.21.15.98:0/3058422820 shutdown_connections 2023-03-03T07:58:20.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.601+0000 7f1700958b80 1 -- 172.21.15.98:0/3058422820 wait complete. 2023-03-03T07:58:20.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.602+0000 7f1700958b80 1 -- start start 2023-03-03T07:58:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.602+0000 7f1700958b80 1 Processor -- start 2023-03-03T07:58:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.603+0000 7f1700958b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9709875c0 0x55e970a34cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.603+0000 7f1700958b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e970988ff0 0x55e9709ed6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.603+0000 7f1700958b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e9709a6ad0 0x55e9709edc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.603+0000 7f1700958b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e970a358a0 con 0x55e9709875c0 2023-03-03T07:58:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.603+0000 7f1700958b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e970987a20 con 0x55e970988ff0 2023-03-03T07:58:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.603+0000 7f1700958b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e970a35210 con 0x55e9709a6ad0 2023-03-03T07:58:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.603+0000 7f1700958b80 10 client.0 _ll_register_callbacks cb 0x55e9708781c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T07:58:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.604+0000 7f16e659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9709875c0 0x55e970a34cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.604+0000 7f16e659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9709875c0 0x55e970a34cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38228/0 (socket says 172.21.15.98:38228) 2023-03-03T07:58:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.604+0000 7f16e659c700 1 -- 172.21.15.98:0/3873054342 learned_addr learned my addr 172.21.15.98:0/3873054342 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.604+0000 7f16e6d9d700 1 --2- 172.21.15.98:0/3873054342 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e9709a6ad0 0x55e9709edc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.604+0000 7f16e5d9b700 1 --2- 172.21.15.98:0/3873054342 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e970988ff0 0x55e9709ed6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.604+0000 7f16e659c700 1 -- 172.21.15.98:0/3873054342 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e9709a6ad0 msgr2=0x55e9709edc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.604+0000 7f16e659c700 1 --2- 172.21.15.98:0/3873054342 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e9709a6ad0 0x55e9709edc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.604+0000 7f16e659c700 1 -- 172.21.15.98:0/3873054342 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e970988ff0 msgr2=0x55e9709ed6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.604+0000 7f16e659c700 1 --2- 172.21.15.98:0/3873054342 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e970988ff0 0x55e9709ed6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.604+0000 7f16e659c700 1 -- 172.21.15.98:0/3873054342 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e000f040 con 0x55e9709875c0 2023-03-03T07:58:20.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.605+0000 7f16e659c700 1 --2- 172.21.15.98:0/3873054342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9709875c0 0x55e970a34cd0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f16d8011cd0 tx=0x7f16d80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59cc2cf1f00636a9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:20.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.605+0000 7f16c57fa700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-03-03T07:58:20.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.605+0000 7f16c57fa700 1 -- 172.21.15.98:0/3873054342 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d8020070 con 0x55e9709875c0 2023-03-03T07:58:20.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.605+0000 7f16c57fa700 1 -- 172.21.15.98:0/3873054342 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(52..52 src has 1..52) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f16d803fc40 con 0x55e9709875c0 2023-03-03T07:58:20.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.605+0000 7f1700958b80 1 -- 172.21.15.98:0/3873054342 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f16dc0850d0 con 0x55e9709875c0 2023-03-03T07:58:20.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.605+0000 7f16c57fa700 1 -- 172.21.15.98:0/3873054342 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16d803fe00 con 0x55e9709875c0 2023-03-03T07:58:20.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.605+0000 7f1700958b80 10 client.5904 fetch_fsmap learned FSMap version 23 2023-03-03T07:58:20.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.605+0000 7f1700958b80 1 -- 172.21.15.98:0/3873054342 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x55e970887a60 con 0x55e9709875c0 2023-03-03T07:58:20.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.605+0000 7f16c57fa700 1 -- 172.21.15.98:0/3873054342 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16dc0850d0 con 0x55e9709875c0 2023-03-03T07:58:20.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.606+0000 7f16c57fa700 1 -- 172.21.15.98:0/3873054342 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 23) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f16d8035a40 con 0x55e9709875c0 2023-03-03T07:58:20.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.606+0000 7f1700958b80 10 client.5904 fetch_fsmap finished waiting for FSMap version 23 2023-03-03T07:58:20.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.606+0000 7f1700958b80 10 client.5904 resolve_mds: resolved name 'a' to daemon mds.a 2023-03-03T07:58:20.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.606+0000 7f1700958b80 20 client.5904 populate_metadata read hostname 'smithi098' 2023-03-03T07:58:20.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.606+0000 7f1700958b80 1 --2- 172.21.15.98:0/3873054342 >> [v2:172.21.15.98:6835/1111074388,v1:172.21.15.98:6837/1111074388] conn(0x55e9709e5760 0x55e9709e5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.606+0000 7f1700958b80 4 client.5904 mds_command: new command op to 5745 tid=0[{"prefix": "status"}] 2023-03-03T07:58:20.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.606+0000 7f1700958b80 1 -- 172.21.15.98:0/3873054342 --> [v2:172.21.15.98:6835/1111074388,v1:172.21.15.98:6837/1111074388] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e970729ca0 con 0x55e9709e5760 2023-03-03T07:58:20.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.606+0000 7f16e5d9b700 1 --2- 172.21.15.98:0/3873054342 >> [v2:172.21.15.98:6835/1111074388,v1:172.21.15.98:6837/1111074388] conn(0x55e9709e5760 0x55e9709e5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.606+0000 7f16c57fa700 10 client.5904 ms_handle_connect on v2:172.21.15.98:6835/1111074388 2023-03-03T07:58:20.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.606+0000 7f16e5d9b700 1 --2- 172.21.15.98:0/3873054342 >> [v2:172.21.15.98:6835/1111074388,v1:172.21.15.98:6837/1111074388] conn(0x55e9709e5760 0x55e9709e5b30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b53beeba0ea0d08 server_cookie=634edab28eb63d35 in_seq=0 out_seq=0 2023-03-03T07:58:20.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.607+0000 7f16c57fa700 1 -- 172.21.15.98:0/3873054342 <== mds.0 v2:172.21.15.98:6835/1111074388 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55e970729ca0 con 0x55e9709e5760 2023-03-03T07:58:20.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.607+0000 7f16c57fa700 10 client.5904 handle_command_reply: tid=0 2023-03-03T07:58:20.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.607+0000 7f1700958b80 1 client.5904 shutdown 2023-03-03T07:58:20.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.607+0000 7f1700958b80 1 -- 172.21.15.98:0/3873054342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9709875c0 msgr2=0x55e970a34cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:20.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.607+0000 7f1700958b80 1 --2- 172.21.15.98:0/3873054342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9709875c0 0x55e970a34cd0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f16d8011cd0 tx=0x7f16d80047c0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.607+0000 7f1700958b80 1 -- 172.21.15.98:0/3873054342 shutdown_connections 2023-03-03T07:58:20.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.607+0000 7f1700958b80 1 --2- 172.21.15.98:0/3873054342 >> [v2:172.21.15.98:6835/1111074388,v1:172.21.15.98:6837/1111074388] conn(0x55e9709e5760 0x55e9709e5b30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.607+0000 7f1700958b80 1 --2- 172.21.15.98:0/3873054342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9709875c0 0x55e970a34cd0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.607+0000 7f1700958b80 1 --2- 172.21.15.98:0/3873054342 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e9709a6ad0 0x55e9709edc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.607+0000 7f1700958b80 1 --2- 172.21.15.98:0/3873054342 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e970988ff0 0x55e9709ed6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.607+0000 7f16c57fa700 0 client.5904 ms_handle_reset on v2:172.21.15.98:6835/1111074388 2023-03-03T07:58:20.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.607+0000 7f1700958b80 1 -- 172.21.15.98:0/3873054342 >> 172.21.15.98:0/3873054342 conn(0x55e970984e10 msgr2=0x55e9709d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:20.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.607+0000 7f1700958b80 1 -- 172.21.15.98:0/3873054342 shutdown_connections 2023-03-03T07:58:20.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.607+0000 7f1700958b80 1 -- 172.21.15.98:0/3873054342 wait complete. 2023-03-03T07:58:20.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.608+0000 7f1700958b80 20 client.5904 trim_cache size 0 max 16384 2023-03-03T07:58:20.616 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T07:58:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "fbbd322f-236a-44f8-96c5-be6963846c9b", 2023-03-03T07:58:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2023-03-03T07:58:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 5745, 2023-03-03T07:58:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-03-03T07:58:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-03-03T07:58:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-03-03T07:58:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 7.3906081419999996, 2023-03-03T07:58:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 23, 2023-03-03T07:58:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 52, 2023-03-03T07:58:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 52, 2023-03-03T07:58:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 14.118708868000001 2023-03-03T07:58:20.617 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T07:58:20.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.608+0000 7f16e759e700 1 -- 172.21.15.98:0/1749043656 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16c80683c0 msgr2=0x7f16c806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:20.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.608+0000 7f16e759e700 1 --2- 172.21.15.98:0/1749043656 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16c80683c0 0x7f16c806a870 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f16d80090b0 tx=0x7f16d8008030 comp rx=0 tx=0).stop 2023-03-03T07:58:20.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.608+0000 7f16e759e700 1 -- 172.21.15.98:0/1749043656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16e80cf570 msgr2=0x7f16e80c7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:20.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.608+0000 7f16e759e700 1 --2- 172.21.15.98:0/1749043656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16e80cf570 0x7f16e80c7360 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f16dc00c700 tx=0x7f16dc00ca10 comp rx=0 tx=0).stop 2023-03-03T07:58:20.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.608+0000 7f16e759e700 1 -- 172.21.15.98:0/1749043656 shutdown_connections 2023-03-03T07:58:20.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.608+0000 7f16e759e700 1 --2- 172.21.15.98:0/1749043656 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f16c80683c0 0x7f16c806a870 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.608+0000 7f16e759e700 1 --2- 172.21.15.98:0/1749043656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16e81609b0 0x7f16e80cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.608+0000 7f16e759e700 1 --2- 172.21.15.98:0/1749043656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16e80b4aa0 0x7f16e80b69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.608+0000 7f16e759e700 1 --2- 172.21.15.98:0/1749043656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16e80cf570 0x7f16e80c7360 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.608+0000 7f16e759e700 1 -- 172.21.15.98:0/1749043656 >> 172.21.15.98:0/1749043656 conn(0x7f16e806b880 msgr2=0x7f16e80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.609+0000 7f16e759e700 1 -- 172.21.15.98:0/1749043656 shutdown_connections 2023-03-03T07:58:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.609+0000 7f16e759e700 1 -- 172.21.15.98:0/1749043656 wait complete. 2023-03-03T07:58:20.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T07:58:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.883+0000 7f967f5ff700 1 Processor -- start 2023-03-03T07:58:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.883+0000 7f967f5ff700 1 -- start start 2023-03-03T07:58:20.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.884+0000 7f967f5ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f96780cd8d0 0x7f96780b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.884+0000 7f967f5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96780b6610 0x7f96780cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.885+0000 7f967f5ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96780cdd70 con 0x7f967815cc00 2023-03-03T07:58:20.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.885+0000 7f967f5ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96780ce040 con 0x7f96780cd8d0 2023-03-03T07:58:20.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.885+0000 7f967f5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9678073be0 con 0x7f96780b6610 2023-03-03T07:58:20.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.885+0000 7f967cb9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f96780cd8d0 0x7f96780b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.885+0000 7f967cb9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f96780cd8d0 0x7f96780b60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38240/0 (socket says 172.21.15.98:38240) 2023-03-03T07:58:20.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.885+0000 7f967cb9a700 1 -- 172.21.15.98:0/2048032665 learned_addr learned my addr 172.21.15.98:0/2048032665 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:20.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.885+0000 7f967db9c700 1 --2- 172.21.15.98:0/2048032665 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96780b6610 0x7f96780cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.885+0000 7f967cb9a700 1 -- 172.21.15.98:0/2048032665 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96780b6610 msgr2=0x7f96780cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:20.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.885+0000 7f967cb9a700 1 --2- 172.21.15.98:0/2048032665 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96780b6610 0x7f96780cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.885+0000 7f967cb9a700 1 -- 172.21.15.98:0/2048032665 >> v1:172.21.15.100:6789/0 conn(0x7f967815cc00 legacy=0x7f96780d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:20.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.885+0000 7f967cb9a700 1 -- 172.21.15.98:0/2048032665 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96780d1a00 con 0x7f96780cd8d0 2023-03-03T07:58:20.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.886+0000 7f967cb9a700 1 --2- 172.21.15.98:0/2048032665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96780cd8d0 0x7f96780b60d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f96680098f0 tx=0x7f9668009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8fcb3f80928e6b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:20.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.886+0000 7f966ffff700 1 -- 172.21.15.98:0/2048032665 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966800b8a0 con 0x7f96780cd8d0 2023-03-03T07:58:20.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.886+0000 7f966ffff700 1 -- 172.21.15.98:0/2048032665 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966800ba00 con 0x7f96780cd8d0 2023-03-03T07:58:20.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.886+0000 7f966ffff700 1 -- 172.21.15.98:0/2048032665 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966800bd90 con 0x7f96780cd8d0 2023-03-03T07:58:20.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.887+0000 7f967f5ff700 1 -- 172.21.15.98:0/2048032665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96780cd8d0 msgr2=0x7f96780b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:20.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.887+0000 7f967f5ff700 1 --2- 172.21.15.98:0/2048032665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96780cd8d0 0x7f96780b60d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f96680098f0 tx=0x7f9668009c00 comp rx=0 tx=0).stop 2023-03-03T07:58:20.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.887+0000 7f967f5ff700 1 -- 172.21.15.98:0/2048032665 shutdown_connections 2023-03-03T07:58:20.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.887+0000 7f967f5ff700 1 --2- 172.21.15.98:0/2048032665 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96780b6610 0x7f96780cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.887+0000 7f967f5ff700 1 --2- 172.21.15.98:0/2048032665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96780cd8d0 0x7f96780b60d0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.887+0000 7f967f5ff700 1 -- 172.21.15.98:0/2048032665 >> 172.21.15.98:0/2048032665 conn(0x7f967806aee0 msgr2=0x7f967806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:20.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.887+0000 7f967f5ff700 1 -- 172.21.15.98:0/2048032665 shutdown_connections 2023-03-03T07:58:20.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.887+0000 7f967f5ff700 1 -- 172.21.15.98:0/2048032665 wait complete. 2023-03-03T07:58:20.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.889+0000 7f967f5ff700 1 Processor -- start 2023-03-03T07:58:20.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.889+0000 7f967f5ff700 1 -- start start 2023-03-03T07:58:20.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.890+0000 7f967f5ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96780b6610 0x7f96780c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.890+0000 7f967f5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96780cd8d0 0x7f967813b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.890+0000 7f967f5ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f967815cc00 0x7f96780b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.890+0000 7f967f5ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96780cc300 con 0x7f96780b6610 2023-03-03T07:58:20.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.890+0000 7f967f5ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967815ddc0 con 0x7f967815cc00 2023-03-03T07:58:20.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.890+0000 7f967f5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967815d820 con 0x7f96780cd8d0 2023-03-03T07:58:20.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.890+0000 7f967d39b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96780b6610 0x7f96780c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.890+0000 7f967d39b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96780b6610 0x7f96780c9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38246/0 (socket says 172.21.15.98:38246) 2023-03-03T07:58:20.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.890+0000 7f967d39b700 1 -- 172.21.15.98:0/1343394908 learned_addr learned my addr 172.21.15.98:0/1343394908 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:20.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.891+0000 7f967cb9a700 1 --2- 172.21.15.98:0/1343394908 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96780cd8d0 0x7f967813b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.891+0000 7f967db9c700 1 --2- 172.21.15.98:0/1343394908 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f967815cc00 0x7f96780b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.891+0000 7f967d39b700 1 -- 172.21.15.98:0/1343394908 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96780cd8d0 msgr2=0x7f967813b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:20.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.891+0000 7f967d39b700 1 --2- 172.21.15.98:0/1343394908 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96780cd8d0 0x7f967813b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.891+0000 7f967d39b700 1 -- 172.21.15.98:0/1343394908 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f967815cc00 msgr2=0x7f96780b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:20.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.891+0000 7f967d39b700 1 --2- 172.21.15.98:0/1343394908 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f967815cc00 0x7f96780b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:20.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.891+0000 7f967d39b700 1 -- 172.21.15.98:0/1343394908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9668009550 con 0x7f96780b6610 2023-03-03T07:58:20.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.891+0000 7f967d39b700 1 --2- 172.21.15.98:0/1343394908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96780b6610 0x7f96780c9230 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f967400c510 tx=0x7f967400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:20.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.892+0000 7f966dffb700 1 -- 172.21.15.98:0/1343394908 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96740052d0 con 0x7f96780b6610 2023-03-03T07:58:20.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.892+0000 7f966dffb700 1 -- 172.21.15.98:0/1343394908 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9674005430 con 0x7f96780b6610 2023-03-03T07:58:20.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.892+0000 7f966dffb700 1 -- 172.21.15.98:0/1343394908 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96740038c0 con 0x7f96780b6610 2023-03-03T07:58:20.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.892+0000 7f967f5ff700 1 -- 172.21.15.98:0/1343394908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96780d1fc0 con 0x7f96780b6610 2023-03-03T07:58:20.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.892+0000 7f967f5ff700 1 -- 172.21.15.98:0/1343394908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96780d1ce0 con 0x7f96780b6610 2023-03-03T07:58:20.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.893+0000 7f967f5ff700 1 -- 172.21.15.98:0/1343394908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96780d22a0 con 0x7f96780b6610 2023-03-03T07:58:20.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.894+0000 7f966dffb700 1 -- 172.21.15.98:0/1343394908 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f9674019030 con 0x7f96780b6610 2023-03-03T07:58:20.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.895+0000 7f966dffb700 1 --2- 172.21.15.98:0/1343394908 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f965c0683c0 0x7f965c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:20.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.895+0000 7f966dffb700 1 -- 172.21.15.98:0/1343394908 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9674085130 con 0x7f96780b6610 2023-03-03T07:58:20.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.896+0000 7f967cb9a700 1 --2- 172.21.15.98:0/1343394908 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f965c0683c0 0x7f965c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:20.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.896+0000 7f967cb9a700 1 --2- 172.21.15.98:0/1343394908 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f965c0683c0 0x7f965c06a870 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9668009c40 tx=0x7f9668008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:20.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:20.899+0000 7f966dffb700 1 -- 172.21.15.98:0/1343394908 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f96740517d0 con 0x7f96780b6610 2023-03-03T07:58:21.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:21.136+0000 7f967f5ff700 1 -- 172.21.15.98:0/1343394908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f96780729b0 con 0x7f96780b6610 2023-03-03T07:58:21.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:21.136+0000 7f966dffb700 1 -- 172.21.15.98:0/1343394908 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9674054df0 con 0x7f96780b6610 2023-03-03T07:58:21.138 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T07:58:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:21.139+0000 7f967f5ff700 1 -- 172.21.15.98:0/1343394908 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f965c0683c0 msgr2=0x7f965c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:21.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:21.139+0000 7f967f5ff700 1 --2- 172.21.15.98:0/1343394908 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f965c0683c0 0x7f965c06a870 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9668009c40 tx=0x7f9668008030 comp rx=0 tx=0).stop 2023-03-03T07:58:21.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:21.139+0000 7f967f5ff700 1 -- 172.21.15.98:0/1343394908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96780b6610 msgr2=0x7f96780c9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:21.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:21.139+0000 7f967f5ff700 1 --2- 172.21.15.98:0/1343394908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96780b6610 0x7f96780c9230 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f967400c510 tx=0x7f967400c820 comp rx=0 tx=0).stop 2023-03-03T07:58:21.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:21.139+0000 7f967f5ff700 1 -- 172.21.15.98:0/1343394908 shutdown_connections 2023-03-03T07:58:21.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:21.139+0000 7f967f5ff700 1 --2- 172.21.15.98:0/1343394908 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f965c0683c0 0x7f965c06a870 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:21.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:21.139+0000 7f967f5ff700 1 --2- 172.21.15.98:0/1343394908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96780b6610 0x7f96780c9230 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:21.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:21.139+0000 7f967f5ff700 1 --2- 172.21.15.98:0/1343394908 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f967815cc00 0x7f96780b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:21.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:21.139+0000 7f967f5ff700 1 --2- 172.21.15.98:0/1343394908 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96780cd8d0 0x7f967813b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:21.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:21.139+0000 7f967f5ff700 1 -- 172.21.15.98:0/1343394908 >> 172.21.15.98:0/1343394908 conn(0x7f967806aee0 msgr2=0x7f9678159910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:21.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:21.139+0000 7f967f5ff700 1 -- 172.21.15.98:0/1343394908 shutdown_connections 2023-03-03T07:58:21.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:21.139+0000 7f967f5ff700 1 -- 172.21.15.98:0/1343394908 wait complete. 2023-03-03T07:58:21.167 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T07:58:21.168 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T07:58:21.168 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2023-03-03T07:58:21.168 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-03-03T07:58:21.169 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-03-03T07:58:21.169 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T07:58:21.169 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-03-03T07:58:21.169 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T07:58:21.170 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T07:58:21.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:21.170 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2023-03-03T07:58:21.240 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T07:58:21.241 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T07:58:21.241 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T07:58:21.242 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:58:21.242 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-03-03T07:58:21.242 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:58:21.242 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:58:21.243 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:58:21.243 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2023-03-03T07:58:21.243 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:58:21.244 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:58:21.244 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2023-03-03T07:58:21.244 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:58:21.244 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2023-03-03T07:58:21.245 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T07:58:21.245 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2023-03-03T07:58:21.245 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T07:58:21.246 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42055sec preferred_lft 42055sec 2023-03-03T07:58:21.246 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2023-03-03T07:58:21.246 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:58:21.246 INFO:teuthology.orchestra.run.smithi098.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T07:58:21.247 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 46:3f:9e:55:37:22 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:58:21.247 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-03-03T07:58:21.247 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:58:21.248 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::1857:c2ff:fe19:346f/64 scope link 2023-03-03T07:58:21.248 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:58:21.248 INFO:teuthology.orchestra.run.smithi098.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T07:58:21.248 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 46:3f:9e:55:37:22 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-03-03T07:58:21.249 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::443f:9eff:fe55:3722/64 scope link 2023-03-03T07:58:21.249 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:58:21.249 INFO:teuthology.orchestra.run.smithi098.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T07:58:21.250 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether d6:19:97:a6:7c:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-03-03T07:58:21.250 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::d419:97ff:fea6:7c83/64 scope link 2023-03-03T07:58:21.250 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:58:21.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:21.253 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T07:58:21.350 INFO:teuthology.orchestra.run.smithi098.stdout:1 2023-03-03T07:58:21.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:21.355 DEBUG:teuthology.orchestra.run.smithi098:> route 2023-03-03T07:58:21.427 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2023-03-03T07:58:21.427 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T07:58:21.427 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T07:58:21.427 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T07:58:21.428 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T07:58:21.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:21.430 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-03-03T07:58:21.430 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-03-03T07:58:21.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T07:58:21.431 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T07:58:21.431 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T07:58:21.431 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-03-03T07:58:21.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:21.673 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-03-03T07:58:21.742 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-03-03T07:58:21.742 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-03-03T07:58:21.743 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T07:58:21.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:21.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2023-03-03T07:58:21.845 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-03-03T07:58:21.846 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:58:21.847 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-03-03T07:58:21.915 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-03-03T07:58:21.916 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:58:21.916 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-03-03T07:58:21.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T07:58:22.087 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:58:22.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:58:22.186 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:58:22.190 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:58:22.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:22.191 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T07:58:22.260 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-03-03T07:58:22.260 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/) 2023-03-03T07:58:22.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T07:58:22.371 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:58:22.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:58:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:58:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:58:22.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:22.479 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T07:58:22.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:2023-03-03T07:58:22.487+0000 7f5c61db95c0 -1 init, newargv = 0x55ad8d3a39d0 newargc=15 2023-03-03T07:58:22.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[129732]: starting ceph client 2023-03-03T07:58:22.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[129732]: starting fuse 2023-03-03T07:58:22.558 INFO:teuthology.orchestra.run.smithi098.stdout:42 2023-03-03T07:58:22.559 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2023-03-03T07:58:22.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T07:58:22.560 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2023-03-03T07:58:22.560 DEBUG:teuthology.orchestra.run.smithi098:> import re 2023-03-03T07:58:22.561 DEBUG:teuthology.orchestra.run.smithi098:> import os 2023-03-03T07:58:22.561 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2023-03-03T07:58:22.561 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:58:22.561 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2023-03-03T07:58:22.562 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-03-03T07:58:22.562 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2023-03-03T07:58:22.562 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-03-03T07:58:22.563 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:58:22.563 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2023-03-03T07:58:22.563 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2023-03-03T07:58:22.564 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2023-03-03T07:58:22.564 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2023-03-03T07:58:22.564 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:58:22.564 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2023-03-03T07:58:22.565 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T07:58:22.565 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T07:58:22.565 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T07:58:22.566 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2023-03-03T07:58:22.566 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2023-03-03T07:58:22.566 DEBUG:teuthology.orchestra.run.smithi098:> return f 2023-03-03T07:58:22.567 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T07:58:22.567 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T07:58:22.567 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.0")) 2023-03-03T07:58:22.567 DEBUG:teuthology.orchestra.run.smithi098:> ' 2023-03-03T07:58:22.784 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.0.129732.asok 2023-03-03T07:58:22.962 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.129732.asok 2023-03-03T07:58:22.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:22.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.129732.asok status 2023-03-03T07:58:23.232 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T07:58:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2023-03-03T07:58:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T07:58:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T07:58:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "0", 2023-03-03T07:58:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2023-03-03T07:58:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-03-03T07:58:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "129732", 2023-03-03T07:58:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2023-03-03T07:58:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2023-03-03T07:58:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2023-03-03T07:58:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 5916, 2023-03-03T07:58:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2023-03-03T07:58:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2023-03-03T07:58:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2023-03-03T07:58:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5916 2023-03-03T07:58:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T07:58:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T07:58:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-03-03T07:58:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 107869130 2023-03-03T07:58:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T07:58:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T07:58:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-03-03T07:58:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 107869130 2023-03-03T07:58:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T07:58:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.5916 192.168.0.1:0/107869130", 2023-03-03T07:58:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/107869130", 2023-03-03T07:58:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2023-03-03T07:58:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 23, 2023-03-03T07:58:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 52, 2023-03-03T07:58:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2023-03-03T07:58:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2023-03-03T07:58:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2023-03-03T07:58:23.237 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T07:58:23.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:23.253 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2023-03-03T07:58:23.254 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.0 2023-03-03T07:58:23.323 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2023-03-03T07:58:23.323 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-03-03T07:58:23.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:23.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-03-03T07:58:23.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T07:58:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.672+0000 7f3bd919d700 1 Processor -- start 2023-03-03T07:58:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.672+0000 7f3bd919d700 1 -- start start 2023-03-03T07:58:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.674+0000 7f3bd919d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3bd4146360 0x7f3bd4160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.674+0000 7f3bd919d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3bd40b4ab0 0x7f3bd40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.674+0000 7f3bd919d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3bd40731e0 0x7f3bd4073640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.674+0000 7f3bd919d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd41614c0 con 0x7f3bd40b4ab0 2023-03-03T07:58:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.674+0000 7f3bd919d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd40b5440 con 0x7f3bd4146360 2023-03-03T07:58:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.674+0000 7f3bd919d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd40d10f0 con 0x7f3bd40731e0 2023-03-03T07:58:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.674+0000 7f3bd37fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3bd40b4ab0 0x7f3bd40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.674+0000 7f3bd37fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3bd40b4ab0 0x7f3bd40b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38262/0 (socket says 172.21.15.98:38262) 2023-03-03T07:58:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.674+0000 7f3bd37fe700 1 -- 172.21.15.98:0/11853596 learned_addr learned my addr 172.21.15.98:0/11853596 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.674+0000 7f3bd3fff700 1 --2- 172.21.15.98:0/11853596 >> v2:172.21.15.100:3300/0 conn(0x7f3bd4146360 0x7f3bd4160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.674+0000 7f3bd899c700 1 --2- 172.21.15.98:0/11853596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3bd40731e0 0x7f3bd4073640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.675+0000 7f3bd37fe700 1 -- 172.21.15.98:0/11853596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3bd40731e0 msgr2=0x7f3bd4073640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.675+0000 7f3bd37fe700 1 --2- 172.21.15.98:0/11853596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3bd40731e0 0x7f3bd4073640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.675+0000 7f3bd37fe700 1 -- 172.21.15.98:0/11853596 >> v2:172.21.15.100:3300/0 conn(0x7f3bd4146360 msgr2=0x7f3bd4160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.675+0000 7f3bd37fe700 1 --2- 172.21.15.98:0/11853596 >> v2:172.21.15.100:3300/0 conn(0x7f3bd4146360 0x7f3bd4160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.675+0000 7f3bd37fe700 1 -- 172.21.15.98:0/11853596 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd40cdf20 con 0x7f3bd40b4ab0 2023-03-03T07:58:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.675+0000 7f3bd37fe700 1 --2- 172.21.15.98:0/11853596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd40b4ab0 0x7f3bd40b4f00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40743f0 tx=0x7f3bc8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f656582764b816 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.675+0000 7f3bd2ffd700 1 -- 172.21.15.98:0/11853596 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc8011830 con 0x7f3bd40b4ab0 2023-03-03T07:58:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.675+0000 7f3bd2ffd700 1 -- 172.21.15.98:0/11853596 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc8011990 con 0x7f3bd40b4ab0 2023-03-03T07:58:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.675+0000 7f3bd2ffd700 1 -- 172.21.15.98:0/11853596 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc8011d20 con 0x7f3bd40b4ab0 2023-03-03T07:58:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.676+0000 7f3bd919d700 1 -- 172.21.15.98:0/11853596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd40b4ab0 msgr2=0x7f3bd40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.676+0000 7f3bd919d700 1 --2- 172.21.15.98:0/11853596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd40b4ab0 0x7f3bd40b4f00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40743f0 tx=0x7f3bc8009bd0 comp rx=0 tx=0).stop 2023-03-03T07:58:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.676+0000 7f3bd919d700 1 -- 172.21.15.98:0/11853596 shutdown_connections 2023-03-03T07:58:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.676+0000 7f3bd919d700 1 --2- 172.21.15.98:0/11853596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3bd40731e0 0x7f3bd4073640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.676+0000 7f3bd919d700 1 --2- 172.21.15.98:0/11853596 >> v2:172.21.15.100:3300/0 conn(0x7f3bd4146360 0x7f3bd4160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.676+0000 7f3bd919d700 1 --2- 172.21.15.98:0/11853596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd40b4ab0 0x7f3bd40b4f00 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.676+0000 7f3bd919d700 1 -- 172.21.15.98:0/11853596 >> 172.21.15.98:0/11853596 conn(0x7f3bd406b8e0 msgr2=0x7f3bd40cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.676+0000 7f3bd919d700 1 -- 172.21.15.98:0/11853596 shutdown_connections 2023-03-03T07:58:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.676+0000 7f3bd919d700 1 -- 172.21.15.98:0/11853596 wait complete. 2023-03-03T07:58:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.678+0000 7f3bd919d700 1 Processor -- start 2023-03-03T07:58:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.678+0000 7f3bd919d700 1 -- start start 2023-03-03T07:58:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.679+0000 7f3bd919d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd40731e0 0x7f3bd40c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.679+0000 7f3bd919d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3bd40b4ab0 0x7f3bd4156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.679+0000 7f3bd919d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3bd4146360 0x7f3bd40b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.679+0000 7f3bd919d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd40cb150 con 0x7f3bd40731e0 2023-03-03T07:58:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.679+0000 7f3bd919d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd40cb3a0 con 0x7f3bd4146360 2023-03-03T07:58:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.679+0000 7f3bd919d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd40cb680 con 0x7f3bd40b4ab0 2023-03-03T07:58:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.680+0000 7f3bd3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd40731e0 0x7f3bd40c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.680+0000 7f3bd3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd40731e0 0x7f3bd40c6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38278/0 (socket says 172.21.15.98:38278) 2023-03-03T07:58:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.680+0000 7f3bd3fff700 1 -- 172.21.15.98:0/863031950 learned_addr learned my addr 172.21.15.98:0/863031950 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.680+0000 7f3bd899c700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3bd4146360 0x7f3bd40b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.680+0000 7f3bd37fe700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3bd40b4ab0 0x7f3bd4156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.680+0000 7f3bd3fff700 1 -- 172.21.15.98:0/863031950 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3bd40b4ab0 msgr2=0x7f3bd4156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.680+0000 7f3bd3fff700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3bd40b4ab0 0x7f3bd4156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.680+0000 7f3bd3fff700 1 -- 172.21.15.98:0/863031950 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3bd4146360 msgr2=0x7f3bd40b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.680+0000 7f3bd3fff700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3bd4146360 0x7f3bd40b31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.680+0000 7f3bd3fff700 1 -- 172.21.15.98:0/863031950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc8009550 con 0x7f3bd40731e0 2023-03-03T07:58:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.680+0000 7f3bd899c700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3bd4146360 0x7f3bd40b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.680+0000 7f3bd37fe700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3bd40b4ab0 0x7f3bd4156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.681+0000 7f3bd3fff700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd40731e0 0x7f3bd40c6b60 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400efc0 tx=0x7f3bc400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.681+0000 7f3bd0ff9700 1 -- 172.21.15.98:0/863031950 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4009b80 con 0x7f3bd40731e0 2023-03-03T07:58:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.681+0000 7f3bd0ff9700 1 -- 172.21.15.98:0/863031950 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc400f040 con 0x7f3bd40731e0 2023-03-03T07:58:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.681+0000 7f3bd919d700 1 -- 172.21.15.98:0/863031950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd40cd960 con 0x7f3bd40731e0 2023-03-03T07:58:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.681+0000 7f3bd0ff9700 1 -- 172.21.15.98:0/863031950 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4014480 con 0x7f3bd40731e0 2023-03-03T07:58:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.681+0000 7f3bd919d700 1 -- 172.21.15.98:0/863031950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd40cd680 con 0x7f3bd40731e0 2023-03-03T07:58:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.682+0000 7f3bd919d700 1 -- 172.21.15.98:0/863031950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd40cdc40 con 0x7f3bd40731e0 2023-03-03T07:58:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.683+0000 7f3bd0ff9700 1 -- 172.21.15.98:0/863031950 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f3bc40145e0 con 0x7f3bd40731e0 2023-03-03T07:58:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.685+0000 7f3bd0ff9700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3bc00683c0 0x7f3bc006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.685+0000 7f3bd0ff9700 1 -- 172.21.15.98:0/863031950 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3bc4086880 con 0x7f3bd40731e0 2023-03-03T07:58:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.685+0000 7f3bd37fe700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3bc00683c0 0x7f3bc006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:23.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.686+0000 7f3bd37fe700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3bc00683c0 0x7f3bc006a870 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40c5b20 tx=0x7f3bc8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:23.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.689+0000 7f3bd0ff9700 1 -- 172.21.15.98:0/863031950 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3bc4052f20 con 0x7f3bd40731e0 2023-03-03T07:58:23.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.928+0000 7f3bd919d700 1 -- 172.21.15.98:0/863031950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3bd40cb840 con 0x7f3bd40731e0 2023-03-03T07:58:23.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.929+0000 7f3bd0ff9700 1 -- 172.21.15.98:0/863031950 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3bc4056540 con 0x7f3bd40731e0 2023-03-03T07:58:23.930 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T07:58:23.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.931+0000 7f3bd919d700 1 -- 172.21.15.98:0/863031950 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3bc00683c0 msgr2=0x7f3bc006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:23.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.931+0000 7f3bd919d700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3bc00683c0 0x7f3bc006a870 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40c5b20 tx=0x7f3bc8008030 comp rx=0 tx=0).stop 2023-03-03T07:58:23.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.931+0000 7f3bd919d700 1 -- 172.21.15.98:0/863031950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd40731e0 msgr2=0x7f3bd40c6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:23.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.931+0000 7f3bd919d700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd40731e0 0x7f3bd40c6b60 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400efc0 tx=0x7f3bc400c3a0 comp rx=0 tx=0).stop 2023-03-03T07:58:23.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.931+0000 7f3bd919d700 1 -- 172.21.15.98:0/863031950 shutdown_connections 2023-03-03T07:58:23.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.931+0000 7f3bd919d700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3bc00683c0 0x7f3bc006a870 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:23.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.931+0000 7f3bd919d700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd40731e0 0x7f3bd40c6b60 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:23.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.931+0000 7f3bd919d700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3bd4146360 0x7f3bd40b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:23.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.931+0000 7f3bd919d700 1 --2- 172.21.15.98:0/863031950 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3bd40b4ab0 0x7f3bd4156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:23.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.931+0000 7f3bd919d700 1 -- 172.21.15.98:0/863031950 >> 172.21.15.98:0/863031950 conn(0x7f3bd406b8e0 msgr2=0x7f3bd40b2280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:23.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.932+0000 7f3bd919d700 1 -- 172.21.15.98:0/863031950 shutdown_connections 2023-03-03T07:58:23.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:23.932+0000 7f3bd919d700 1 -- 172.21.15.98:0/863031950 wait complete. 2023-03-03T07:58:23.960 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T07:58:23.960 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T07:58:23.961 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2023-03-03T07:58:23.961 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-03-03T07:58:23.962 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-03-03T07:58:23.962 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T07:58:23.962 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-03-03T07:58:23.963 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T07:58:23.963 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-03-03T07:58:23.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:23.964 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2023-03-03T07:58:24.037 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T07:58:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T07:58:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T07:58:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:58:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2023-03-03T07:58:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:58:24.038 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:58:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:58:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2023-03-03T07:58:24.038 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:58:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:58:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2023-03-03T07:58:24.039 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T07:58:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:58:24.039 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T07:58:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:58:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T07:58:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42056sec preferred_lft 42056sec 2023-03-03T07:58:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2023-03-03T07:58:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:58:24.040 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T07:58:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ee:78:63:4a:ea:32 brd ff:ff:ff:ff:ff:ff 2023-03-03T07:58:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-03-03T07:58:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:58:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::9406:dcff:fe40:a4a4/64 scope link 2023-03-03T07:58:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:58:24.041 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T07:58:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:5e:bd:c1:84:4d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-03-03T07:58:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f45e:bdff:fec1:844d/64 scope link 2023-03-03T07:58:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:58:24.041 INFO:teuthology.orchestra.run.smithi100.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T07:58:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ee:78:63:4a:ea:32 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-03-03T07:58:24.042 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec78:63ff:fe4a:ea32/64 scope link 2023-03-03T07:58:24.042 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T07:58:24.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:24.044 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T07:58:24.141 INFO:teuthology.orchestra.run.smithi100.stdout:1 2023-03-03T07:58:24.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:24.147 DEBUG:teuthology.orchestra.run.smithi100:> route 2023-03-03T07:58:24.219 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2023-03-03T07:58:24.220 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T07:58:24.220 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T07:58:24.220 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T07:58:24.220 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T07:58:24.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:24.222 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2023-03-03T07:58:24.222 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2023-03-03T07:58:24.223 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T07:58:24.223 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T07:58:24.224 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T07:58:24.224 DEBUG:teuthology.orchestra.run.smithi100:> ') 2023-03-03T07:58:24.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:24.465 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2023-03-03T07:58:24.533 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-03-03T07:58:24.533 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-03-03T07:58:24.535 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-03-03T07:58:24.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:24.536 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2023-03-03T07:58:24.637 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-03-03T07:58:24.638 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:58:24.638 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-03-03T07:58:24.706 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-03-03T07:58:24.707 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T07:58:24.708 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-03-03T07:58:24.775 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T07:58:24.877 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:58:24.877 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:58:24.975 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:58:24.979 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:58:24.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:24.981 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T07:58:25.048 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-03-03T07:58:25.048 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_mountpoint=/) 2023-03-03T07:58:25.053 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T07:58:25.157 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T07:58:25.157 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T07:58:25.259 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T07:58:25.263 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T07:58:25.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:25.265 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T07:58:25.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2023-03-03T07:58:25.276+0000 7f91210dd5c0 -1 init, newargv = 0x5633d37eeb00 newargc=15 2023-03-03T07:58:25.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[117851]: starting ceph client 2023-03-03T07:58:25.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[117851]: starting fuse 2023-03-03T07:58:25.337 INFO:teuthology.orchestra.run.smithi100.stdout:42 2023-03-03T07:58:25.337 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2023-03-03T07:58:25.338 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T07:58:25.338 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2023-03-03T07:58:25.339 DEBUG:teuthology.orchestra.run.smithi100:> import re 2023-03-03T07:58:25.339 DEBUG:teuthology.orchestra.run.smithi100:> import os 2023-03-03T07:58:25.339 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2023-03-03T07:58:25.339 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:58:25.340 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2023-03-03T07:58:25.340 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-03-03T07:58:25.340 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2023-03-03T07:58:25.341 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-03-03T07:58:25.341 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:58:25.341 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2023-03-03T07:58:25.342 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2023-03-03T07:58:25.342 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2023-03-03T07:58:25.342 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2023-03-03T07:58:25.343 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:58:25.343 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2023-03-03T07:58:25.343 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T07:58:25.344 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T07:58:25.344 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T07:58:25.344 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2023-03-03T07:58:25.345 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2023-03-03T07:58:25.345 DEBUG:teuthology.orchestra.run.smithi100:> return f 2023-03-03T07:58:25.345 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T07:58:25.346 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T07:58:25.346 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.1")) 2023-03-03T07:58:25.346 DEBUG:teuthology.orchestra.run.smithi100:> ' 2023-03-03T07:58:25.562 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.1.117851.asok 2023-03-03T07:58:25.739 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.117851.asok 2023-03-03T07:58:25.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:25.740 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.117851.asok status 2023-03-03T07:58:26.008 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2023-03-03T07:58:26.009 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2023-03-03T07:58:26.009 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T07:58:26.009 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T07:58:26.010 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "1", 2023-03-03T07:58:26.010 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2023-03-03T07:58:26.010 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-03-03T07:58:26.010 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "117851", 2023-03-03T07:58:26.011 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2023-03-03T07:58:26.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:58:26.011 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2023-03-03T07:58:26.011 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2023-03-03T07:58:26.012 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 5884, 2023-03-03T07:58:26.012 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2023-03-03T07:58:26.012 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2023-03-03T07:58:26.012 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2023-03-03T07:58:26.013 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 5884 2023-03-03T07:58:26.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:58:26.013 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T07:58:26.013 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T07:58:26.014 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2023-03-03T07:58:26.014 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1237849258 2023-03-03T07:58:26.014 INFO:teuthology.orchestra.run.smithi100.stdout: } 2023-03-03T07:58:26.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:58:26.015 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T07:58:26.015 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T07:58:26.015 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2023-03-03T07:58:26.015 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1237849258 2023-03-03T07:58:26.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T07:58:26.016 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.5884 192.168.0.1:0/1237849258", 2023-03-03T07:58:26.016 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.1:0/1237849258", 2023-03-03T07:58:26.017 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2023-03-03T07:58:26.017 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 23, 2023-03-03T07:58:26.017 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 52, 2023-03-03T07:58:26.017 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2023-03-03T07:58:26.018 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2023-03-03T07:58:26.018 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2023-03-03T07:58:26.018 INFO:teuthology.orchestra.run.smithi100.stdout:} 2023-03-03T07:58:26.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:26.030 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2023-03-03T07:58:26.031 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2023-03-03T07:58:26.100 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2023-03-03T07:58:26.101 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-03-03T07:58:26.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:26.102 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-03-03T07:58:26.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-03-03T07:58:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.464+0000 7f83bcfff700 1 Processor -- start 2023-03-03T07:58:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.464+0000 7f83bcfff700 1 -- start start 2023-03-03T07:58:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.466+0000 7f83bcfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b80cd770 0x7f83b80b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.466+0000 7f83bcfff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f83b815bf70 0x7f83b815c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.466+0000 7f83bcfff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b80d0ac0 con 0x7f83b80b8880 2023-03-03T07:58:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.466+0000 7f83bcfff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b80cdbe0 con 0x7f83b815bf70 2023-03-03T07:58:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.466+0000 7f83bcfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b80cff70 con 0x7f83b80cd770 2023-03-03T07:58:26.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.466+0000 7f83b6d9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f83b815bf70 0x7f83b815c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:26.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.466+0000 7f83b5d9b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b80cd770 0x7f83b80b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:26.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.466+0000 7f83b6d9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f83b815bf70 0x7f83b815c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:47926/0 (socket says 172.21.15.98:47926) 2023-03-03T07:58:26.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.466+0000 7f83b6d9d700 1 -- 172.21.15.98:0/3283447529 learned_addr learned my addr 172.21.15.98:0/3283447529 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:26.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.467+0000 7f83b559a700 1 -- 172.21.15.98:0/3283447529 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349191619 0 0) 0x7f83b80d0ac0 con 0x7f83b80b8880 2023-03-03T07:58:26.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.467+0000 7f83b559a700 1 -- 172.21.15.98:0/3283447529 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83a00035b0 con 0x7f83b80b8880 2023-03-03T07:58:26.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.467+0000 7f83b6d9d700 1 -- 172.21.15.98:0/3283447529 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b80cd770 msgr2=0x7f83b80b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:26.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.467+0000 7f83b6d9d700 1 --2- 172.21.15.98:0/3283447529 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b80cd770 0x7f83b80b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:26.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.467+0000 7f83b6d9d700 1 -- 172.21.15.98:0/3283447529 >> v1:172.21.15.100:6789/0 conn(0x7f83b80b8880 legacy=0x7f83b80b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:26.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.467+0000 7f83b6d9d700 1 -- 172.21.15.98:0/3283447529 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b80d0250 con 0x7f83b815bf70 2023-03-03T07:58:26.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.467+0000 7f83b6d9d700 1 --2- 172.21.15.98:0/3283447529 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83b815bf70 0x7f83b815c3c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f83b0009e50 tx=0x7f83b000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fddf8a2846547fc1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:26.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.467+0000 7f83b559a700 1 -- 172.21.15.98:0/3283447529 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b000e070 con 0x7f83b815bf70 2023-03-03T07:58:26.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.467+0000 7f83b559a700 1 -- 172.21.15.98:0/3283447529 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b000b910 con 0x7f83b815bf70 2023-03-03T07:58:26.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.467+0000 7f83b559a700 1 -- 172.21.15.98:0/3283447529 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b000bcc0 con 0x7f83b815bf70 2023-03-03T07:58:26.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.468+0000 7f83bcfff700 1 -- 172.21.15.98:0/3283447529 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83b815bf70 msgr2=0x7f83b815c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:26.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.468+0000 7f83bcfff700 1 --2- 172.21.15.98:0/3283447529 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83b815bf70 0x7f83b815c3c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f83b0009e50 tx=0x7f83b000b330 comp rx=0 tx=0).stop 2023-03-03T07:58:26.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.468+0000 7f83bcfff700 1 -- 172.21.15.98:0/3283447529 shutdown_connections 2023-03-03T07:58:26.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.468+0000 7f83bcfff700 1 --2- 172.21.15.98:0/3283447529 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83b815bf70 0x7f83b815c3c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:26.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.468+0000 7f83bcfff700 1 --2- 172.21.15.98:0/3283447529 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b80cd770 0x7f83b80b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:26.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.468+0000 7f83bcfff700 1 -- 172.21.15.98:0/3283447529 >> 172.21.15.98:0/3283447529 conn(0x7f83b806ad80 msgr2=0x7f83b806b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:26.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.468+0000 7f83bcfff700 1 -- 172.21.15.98:0/3283447529 shutdown_connections 2023-03-03T07:58:26.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.468+0000 7f83bcfff700 1 -- 172.21.15.98:0/3283447529 wait complete. 2023-03-03T07:58:26.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.470+0000 7f83bcfff700 1 Processor -- start 2023-03-03T07:58:26.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.470+0000 7f83bcfff700 1 -- start start 2023-03-03T07:58:26.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.471+0000 7f83bcfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b80b8880 0x7f83b80c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:26.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.471+0000 7f83bcfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83b80cd770 0x7f83b81f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:26.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.471+0000 7f83bcfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83b815bf70 0x7f83b8159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:26.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.471+0000 7f83bcfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b80c9710 con 0x7f83b80cd770 2023-03-03T07:58:26.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.471+0000 7f83bcfff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b815c4e0 con 0x7f83b815bf70 2023-03-03T07:58:26.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.471+0000 7f83bcfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b815c7c0 con 0x7f83b80b8880 2023-03-03T07:58:26.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.472+0000 7f83b5d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83b80cd770 0x7f83b81f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:26.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.472+0000 7f83b5d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83b80cd770 0x7f83b81f2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51312/0 (socket says 172.21.15.98:51312) 2023-03-03T07:58:26.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.472+0000 7f83b5d9b700 1 -- 172.21.15.98:0/2904909229 learned_addr learned my addr 172.21.15.98:0/2904909229 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:26.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.472+0000 7f83b6d9d700 1 --2- 172.21.15.98:0/2904909229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83b815bf70 0x7f83b8159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:26.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.472+0000 7f83b659c700 1 --2- 172.21.15.98:0/2904909229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b80b8880 0x7f83b80c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:26.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.472+0000 7f83b5d9b700 1 -- 172.21.15.98:0/2904909229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b80b8880 msgr2=0x7f83b80c91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:26.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.472+0000 7f83b5d9b700 1 --2- 172.21.15.98:0/2904909229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b80b8880 0x7f83b80c91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.472+0000 7f83b5d9b700 1 -- 172.21.15.98:0/2904909229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83b815bf70 msgr2=0x7f83b8159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.472+0000 7f83b5d9b700 1 --2- 172.21.15.98:0/2904909229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83b815bf70 0x7f83b8159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.472+0000 7f83b5d9b700 1 -- 172.21.15.98:0/2904909229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b0009b30 con 0x7f83b80cd770 2023-03-03T07:58:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.473+0000 7f83b5d9b700 1 --2- 172.21.15.98:0/2904909229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83b80cd770 0x7f83b81f2760 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00e9b0 tx=0x7f83ac00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.473+0000 7f83a6ffd700 1 -- 172.21.15.98:0/2904909229 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac009710 con 0x7f83b80cd770 2023-03-03T07:58:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.473+0000 7f83a6ffd700 1 -- 172.21.15.98:0/2904909229 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac00ee50 con 0x7f83b80cd770 2023-03-03T07:58:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.473+0000 7f83a6ffd700 1 -- 172.21.15.98:0/2904909229 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac00ca50 con 0x7f83b80cd770 2023-03-03T07:58:26.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.473+0000 7f83bcfff700 1 -- 172.21.15.98:0/2904909229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b80d0cf0 con 0x7f83b80cd770 2023-03-03T07:58:26.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.473+0000 7f83bcfff700 1 -- 172.21.15.98:0/2904909229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b80d0530 con 0x7f83b80cd770 2023-03-03T07:58:26.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.474+0000 7f83bcfff700 1 -- 172.21.15.98:0/2904909229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b80b8c50 con 0x7f83b80cd770 2023-03-03T07:58:26.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.475+0000 7f83a6ffd700 1 -- 172.21.15.98:0/2904909229 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f83ac019030 con 0x7f83b80cd770 2023-03-03T07:58:26.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.476+0000 7f83a6ffd700 1 --2- 172.21.15.98:0/2904909229 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f83980683c0 0x7f839806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:26.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.477+0000 7f83a6ffd700 1 -- 172.21.15.98:0/2904909229 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f83ac085dd0 con 0x7f83b80cd770 2023-03-03T07:58:26.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.477+0000 7f83b659c700 1 --2- 172.21.15.98:0/2904909229 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f83980683c0 0x7f839806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:26.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.477+0000 7f83b659c700 1 --2- 172.21.15.98:0/2904909229 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f83980683c0 0x7f839806a870 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f83a80011e0 tx=0x7f83a8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:26.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.480+0000 7f83a6ffd700 1 -- 172.21.15.98:0/2904909229 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f83ac052410 con 0x7f83b80cd770 2023-03-03T07:58:26.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:26.720+0000 7f83bcfff700 1 -- 172.21.15.98:0/2904909229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f83b804e6e0 con 0x7f83b80cd770 2023-03-03T07:58:27.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:27.724+0000 7f83a6ffd700 1 -- 172.21.15.98:0/2904909229 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v24) v1 ==== 136+0+0 (secure 0 0 0) 0x7f83ac055a30 con 0x7f83b80cd770 2023-03-03T07:58:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:27.728+0000 7f83bcfff700 1 -- 172.21.15.98:0/2904909229 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f83980683c0 msgr2=0x7f839806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:27.728+0000 7f83bcfff700 1 --2- 172.21.15.98:0/2904909229 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f83980683c0 0x7f839806a870 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f83a80011e0 tx=0x7f83a8009040 comp rx=0 tx=0).stop 2023-03-03T07:58:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:27.728+0000 7f83bcfff700 1 -- 172.21.15.98:0/2904909229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83b80cd770 msgr2=0x7f83b81f2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:27.728+0000 7f83bcfff700 1 --2- 172.21.15.98:0/2904909229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83b80cd770 0x7f83b81f2760 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00e9b0 tx=0x7f83ac00ecc0 comp rx=0 tx=0).stop 2023-03-03T07:58:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:27.728+0000 7f83bcfff700 1 -- 172.21.15.98:0/2904909229 shutdown_connections 2023-03-03T07:58:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:27.728+0000 7f83bcfff700 1 --2- 172.21.15.98:0/2904909229 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f83980683c0 0x7f839806a870 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:27.728+0000 7f83bcfff700 1 --2- 172.21.15.98:0/2904909229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83b815bf70 0x7f83b8159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:27.728+0000 7f83bcfff700 1 --2- 172.21.15.98:0/2904909229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b80b8880 0x7f83b80c91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:27.728+0000 7f83bcfff700 1 --2- 172.21.15.98:0/2904909229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83b80cd770 0x7f83b81f2760 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:27.728+0000 7f83bcfff700 1 -- 172.21.15.98:0/2904909229 >> 172.21.15.98:0/2904909229 conn(0x7f83b806ad80 msgr2=0x7f83b80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:27.728+0000 7f83bcfff700 1 -- 172.21.15.98:0/2904909229 shutdown_connections 2023-03-03T07:58:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:27.728+0000 7f83bcfff700 1 -- 172.21.15.98:0/2904909229 wait complete. 2023-03-03T07:58:27.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-03-03T07:58:28.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.007+0000 7fcc99fff700 1 Processor -- start 2023-03-03T07:58:28.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.008+0000 7fcc99fff700 1 -- start start 2023-03-03T07:58:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.009+0000 7fcc99fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fcc94169ab0 0x7fcc9406d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.009+0000 7fcc99fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcc940ca910 0x7fcc9416dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.010+0000 7fcc99fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc9416e180 0x7fcc940ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.010+0000 7fcc99fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc9416d7b0 con 0x7fcc940ca910 2023-03-03T07:58:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.010+0000 7fcc99fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc9416d480 con 0x7fcc94169ab0 2023-03-03T07:58:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.010+0000 7fcc99fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc940cae80 con 0x7fcc9416e180 2023-03-03T07:58:28.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.010+0000 7fcc92ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcc940ca910 0x7fcc9416dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.010+0000 7fcc92ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcc940ca910 0x7fcc9416dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51324/0 (socket says 172.21.15.98:51324) 2023-03-03T07:58:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.010+0000 7fcc92ffd700 1 -- 172.21.15.98:0/2992340348 learned_addr learned my addr 172.21.15.98:0/2992340348 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.010+0000 7fcc937fe700 1 --2- 172.21.15.98:0/2992340348 >> v2:172.21.15.100:3300/0 conn(0x7fcc94169ab0 0x7fcc9406d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.010+0000 7fcc93fff700 1 --2- 172.21.15.98:0/2992340348 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc9416e180 0x7fcc940ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.011+0000 7fcc92ffd700 1 -- 172.21.15.98:0/2992340348 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc9416e180 msgr2=0x7fcc940ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.011+0000 7fcc92ffd700 1 --2- 172.21.15.98:0/2992340348 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc9416e180 0x7fcc940ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.011+0000 7fcc92ffd700 1 -- 172.21.15.98:0/2992340348 >> v2:172.21.15.100:3300/0 conn(0x7fcc94169ab0 msgr2=0x7fcc9406d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.011+0000 7fcc92ffd700 1 --2- 172.21.15.98:0/2992340348 >> v2:172.21.15.100:3300/0 conn(0x7fcc94169ab0 0x7fcc9406d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.011+0000 7fcc92ffd700 1 -- 172.21.15.98:0/2992340348 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc940cb160 con 0x7fcc940ca910 2023-03-03T07:58:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.011+0000 7fcc92ffd700 1 --2- 172.21.15.98:0/2992340348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc940ca910 0x7fcc9416dc40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fcc9406fb30 tx=0x7fcc88009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98569080d4fa3018 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.011+0000 7fcc927fc700 1 -- 172.21.15.98:0/2992340348 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc88011830 con 0x7fcc940ca910 2023-03-03T07:58:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.011+0000 7fcc927fc700 1 -- 172.21.15.98:0/2992340348 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc88011990 con 0x7fcc940ca910 2023-03-03T07:58:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.011+0000 7fcc927fc700 1 -- 172.21.15.98:0/2992340348 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc88011d20 con 0x7fcc940ca910 2023-03-03T07:58:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.012+0000 7fcc99fff700 1 -- 172.21.15.98:0/2992340348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc940ca910 msgr2=0x7fcc9416dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.012+0000 7fcc99fff700 1 --2- 172.21.15.98:0/2992340348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc940ca910 0x7fcc9416dc40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fcc9406fb30 tx=0x7fcc88009bd0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.012+0000 7fcc99fff700 1 -- 172.21.15.98:0/2992340348 shutdown_connections 2023-03-03T07:58:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.012+0000 7fcc99fff700 1 --2- 172.21.15.98:0/2992340348 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc9416e180 0x7fcc940ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.012+0000 7fcc99fff700 1 --2- 172.21.15.98:0/2992340348 >> v2:172.21.15.100:3300/0 conn(0x7fcc94169ab0 0x7fcc9406d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.012+0000 7fcc99fff700 1 --2- 172.21.15.98:0/2992340348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc940ca910 0x7fcc9416dc40 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.012+0000 7fcc99fff700 1 -- 172.21.15.98:0/2992340348 >> 172.21.15.98:0/2992340348 conn(0x7fcc940cb800 msgr2=0x7fcc940cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.012+0000 7fcc99fff700 1 -- 172.21.15.98:0/2992340348 shutdown_connections 2023-03-03T07:58:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.012+0000 7fcc99fff700 1 -- 172.21.15.98:0/2992340348 wait complete. 2023-03-03T07:58:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.014+0000 7fcc99fff700 1 Processor -- start 2023-03-03T07:58:28.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.014+0000 7fcc99fff700 1 -- start start 2023-03-03T07:58:28.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.015+0000 7fcc99fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc940ca910 0x7fcc941feef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:28.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.015+0000 7fcc99fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc94169ab0 0x7fcc941681a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:28.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.015+0000 7fcc99fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9416e180 0x7fcc9416a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.015+0000 7fcc99fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc941686e0 con 0x7fcc9416e180 2023-03-03T07:58:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.015+0000 7fcc99fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc940b37a0 con 0x7fcc940ca910 2023-03-03T07:58:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.015+0000 7fcc99fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc940b28b0 con 0x7fcc94169ab0 2023-03-03T07:58:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.016+0000 7fcc93fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9416e180 0x7fcc9416a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.016+0000 7fcc93fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9416e180 0x7fcc9416a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51332/0 (socket says 172.21.15.98:51332) 2023-03-03T07:58:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.016+0000 7fcc93fff700 1 -- 172.21.15.98:0/5642210 learned_addr learned my addr 172.21.15.98:0/5642210 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.016+0000 7fcc92ffd700 1 --2- 172.21.15.98:0/5642210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc94169ab0 0x7fcc941681a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.016+0000 7fcc937fe700 1 --2- 172.21.15.98:0/5642210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc940ca910 0x7fcc941feef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.016+0000 7fcc93fff700 1 -- 172.21.15.98:0/5642210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc94169ab0 msgr2=0x7fcc941681a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.016+0000 7fcc93fff700 1 --2- 172.21.15.98:0/5642210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc94169ab0 0x7fcc941681a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.016+0000 7fcc93fff700 1 -- 172.21.15.98:0/5642210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc940ca910 msgr2=0x7fcc941feef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.016+0000 7fcc93fff700 1 --2- 172.21.15.98:0/5642210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc940ca910 0x7fcc941feef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.016+0000 7fcc93fff700 1 -- 172.21.15.98:0/5642210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc88009550 con 0x7fcc9416e180 2023-03-03T07:58:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.016+0000 7fcc937fe700 1 --2- 172.21.15.98:0/5642210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc940ca910 0x7fcc941feef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.017+0000 7fcc93fff700 1 --2- 172.21.15.98:0/5642210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9416e180 0x7fcc9416a7b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00e7d0 tx=0x7fcc8c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.017+0000 7fcc7bfff700 1 -- 172.21.15.98:0/5642210 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c0095b0 con 0x7fcc9416e180 2023-03-03T07:58:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.017+0000 7fcc7bfff700 1 -- 172.21.15.98:0/5642210 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8c013680 con 0x7fcc9416e180 2023-03-03T07:58:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.017+0000 7fcc7bfff700 1 -- 172.21.15.98:0/5642210 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c013a10 con 0x7fcc9416e180 2023-03-03T07:58:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.017+0000 7fcc99fff700 1 -- 172.21.15.98:0/5642210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc9406cb50 con 0x7fcc9416e180 2023-03-03T07:58:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.017+0000 7fcc99fff700 1 -- 172.21.15.98:0/5642210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc940cb440 con 0x7fcc9416e180 2023-03-03T07:58:28.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.018+0000 7fcc99fff700 1 -- 172.21.15.98:0/5642210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9406cce0 con 0x7fcc9416e180 2023-03-03T07:58:28.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.019+0000 7fcc7bfff700 1 -- 172.21.15.98:0/5642210 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fcc8c019030 con 0x7fcc9416e180 2023-03-03T07:58:28.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.021+0000 7fcc7bfff700 1 --2- 172.21.15.98:0/5642210 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcc800683c0 0x7fcc8006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:28.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.021+0000 7fcc7bfff700 1 -- 172.21.15.98:0/5642210 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fcc8c085d50 con 0x7fcc9416e180 2023-03-03T07:58:28.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.021+0000 7fcc937fe700 1 --2- 172.21.15.98:0/5642210 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcc800683c0 0x7fcc8006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:28.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.022+0000 7fcc937fe700 1 --2- 172.21.15.98:0/5642210 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcc800683c0 0x7fcc8006a870 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcc84007520 tx=0x7fcc8400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:28.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.025+0000 7fcc7bfff700 1 -- 172.21.15.98:0/5642210 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcc8c0523b0 con 0x7fcc9416e180 2023-03-03T07:58:28.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.244+0000 7fcc99fff700 1 -- 172.21.15.98:0/5642210 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc940b3930 con 0x7fcc800683c0 2023-03-03T07:58:28.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.245+0000 7fcc7bfff700 1 -- 172.21.15.98:0/5642210 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fcc940b3930 con 0x7fcc800683c0 2023-03-03T07:58:28.247 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T07:58:28.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T07:58:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephfs" 2023-03-03T07:58:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T07:58:28.248 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T07:58:28.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.248+0000 7fcc99fff700 1 -- 172.21.15.98:0/5642210 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcc800683c0 msgr2=0x7fcc8006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:28.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.248+0000 7fcc99fff700 1 --2- 172.21.15.98:0/5642210 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcc800683c0 0x7fcc8006a870 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcc84007520 tx=0x7fcc8400b040 comp rx=0 tx=0).stop 2023-03-03T07:58:28.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.248+0000 7fcc99fff700 1 -- 172.21.15.98:0/5642210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9416e180 msgr2=0x7fcc9416a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:28.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.248+0000 7fcc99fff700 1 --2- 172.21.15.98:0/5642210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9416e180 0x7fcc9416a7b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00e7d0 tx=0x7fcc8c00eb90 comp rx=0 tx=0).stop 2023-03-03T07:58:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.249+0000 7fcc99fff700 1 -- 172.21.15.98:0/5642210 shutdown_connections 2023-03-03T07:58:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.249+0000 7fcc99fff700 1 --2- 172.21.15.98:0/5642210 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcc800683c0 0x7fcc8006a870 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.249+0000 7fcc99fff700 1 --2- 172.21.15.98:0/5642210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9416e180 0x7fcc9416a7b0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.249+0000 7fcc99fff700 1 --2- 172.21.15.98:0/5642210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc940ca910 0x7fcc941feef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.249+0000 7fcc99fff700 1 --2- 172.21.15.98:0/5642210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc94169ab0 0x7fcc941681a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.249+0000 7fcc99fff700 1 -- 172.21.15.98:0/5642210 >> 172.21.15.98:0/5642210 conn(0x7fcc940cb800 msgr2=0x7fcc940b0740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.249+0000 7fcc99fff700 1 -- 172.21.15.98:0/5642210 shutdown_connections 2023-03-03T07:58:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.249+0000 7fcc99fff700 1 -- 172.21.15.98:0/5642210 wait complete. 2023-03-03T07:58:28.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-03-03T07:58:28.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.525+0000 7fe7a7bff700 1 Processor -- start 2023-03-03T07:58:28.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.525+0000 7fe7a7bff700 1 -- start start 2023-03-03T07:58:28.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.526+0000 7fe7a7bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe7a00d0270 0x7fe7a00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:28.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.526+0000 7fe7a7bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a00cf980 con 0x7fe7a015ca70 2023-03-03T07:58:28.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.527+0000 7fe7a7bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a00d1800 con 0x7fe7a006b4b0 2023-03-03T07:58:28.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.527+0000 7fe7a7bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a00cd5b0 con 0x7fe7a00d0270 2023-03-03T07:58:28.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.527+0000 7fe7a599b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe7a015ca70 0x7fe7a00cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56402/0 (socket says 172.21.15.98:56402) 2023-03-03T07:58:28.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.527+0000 7fe7a599b700 1 -- 172.21.15.98:0/1678960111 learned_addr learned my addr 172.21.15.98:0/1678960111 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:28.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.527+0000 7fe7a519a700 1 --2- 172.21.15.98:0/1678960111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe7a00d0270 0x7fe7a00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:28.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.527+0000 7fe7a4999700 1 -- 172.21.15.98:0/1678960111 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506071135 0 0) 0x7fe7a00cf980 con 0x7fe7a015ca70 2023-03-03T07:58:28.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.527+0000 7fe7a4999700 1 -- 172.21.15.98:0/1678960111 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe78c0035b0 con 0x7fe7a015ca70 2023-03-03T07:58:28.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.528+0000 7fe7a4999700 1 -- 172.21.15.98:0/1678960111 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795483365 0 0) 0x7fe7a00d1800 con 0x7fe7a006b4b0 2023-03-03T07:58:28.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.528+0000 7fe7a4999700 1 -- 172.21.15.98:0/1678960111 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7a00cf980 con 0x7fe7a006b4b0 2023-03-03T07:58:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.528+0000 7fe7a519a700 1 -- 172.21.15.98:0/1678960111 >> v1:172.21.15.100:6789/0 conn(0x7fe7a006b4b0 legacy=0x7fe7a00d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.528+0000 7fe7a519a700 1 -- 172.21.15.98:0/1678960111 >> v1:172.21.15.98:6789/0 conn(0x7fe7a015ca70 legacy=0x7fe7a00cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.528+0000 7fe7a519a700 1 -- 172.21.15.98:0/1678960111 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a00cd900 con 0x7fe7a00d0270 2023-03-03T07:58:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.528+0000 7fe7a519a700 1 --2- 172.21.15.98:0/1678960111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe7a00d0270 0x7fe7a00cf440 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe790009870 tx=0x7fe790009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f77166850e418165 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.528+0000 7fe7a4999700 1 -- 172.21.15.98:0/1678960111 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe790011830 con 0x7fe7a00d0270 2023-03-03T07:58:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.528+0000 7fe7a4999700 1 -- 172.21.15.98:0/1678960111 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe790011990 con 0x7fe7a00d0270 2023-03-03T07:58:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.528+0000 7fe7a4999700 1 -- 172.21.15.98:0/1678960111 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe790011d20 con 0x7fe7a00d0270 2023-03-03T07:58:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.529+0000 7fe7a7bff700 1 -- 172.21.15.98:0/1678960111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe7a00d0270 msgr2=0x7fe7a00cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.529+0000 7fe7a7bff700 1 --2- 172.21.15.98:0/1678960111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe7a00d0270 0x7fe7a00cf440 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe790009870 tx=0x7fe790009b80 comp rx=0 tx=0).stop 2023-03-03T07:58:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.529+0000 7fe7a7bff700 1 -- 172.21.15.98:0/1678960111 shutdown_connections 2023-03-03T07:58:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.529+0000 7fe7a7bff700 1 --2- 172.21.15.98:0/1678960111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe7a00d0270 0x7fe7a00cf440 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.529+0000 7fe7a7bff700 1 -- 172.21.15.98:0/1678960111 >> 172.21.15.98:0/1678960111 conn(0x7fe7a0073e70 msgr2=0x7fe7a006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:28.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.529+0000 7fe7a7bff700 1 -- 172.21.15.98:0/1678960111 shutdown_connections 2023-03-03T07:58:28.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.529+0000 7fe7a7bff700 1 -- 172.21.15.98:0/1678960111 wait complete. 2023-03-03T07:58:28.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.531+0000 7fe7a7bff700 1 Processor -- start 2023-03-03T07:58:28.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.531+0000 7fe7a7bff700 1 -- start start 2023-03-03T07:58:28.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.532+0000 7fe7a7bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a006b4b0 0x7fe7a00b3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:28.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.532+0000 7fe7a7bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe7a00d0270 0x7fe7a01518b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:28.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.532+0000 7fe7a7bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe7a015ca70 0x7fe7a00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:28.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.532+0000 7fe7a7bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a00b4480 con 0x7fe7a006b4b0 2023-03-03T07:58:28.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.532+0000 7fe7a7bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a00cbf00 con 0x7fe7a00d0270 2023-03-03T07:58:28.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.532+0000 7fe7a7bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a00b3ec0 con 0x7fe7a015ca70 2023-03-03T07:58:28.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.533+0000 7fe7a599b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a006b4b0 0x7fe7a00b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:28.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.533+0000 7fe7a599b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a006b4b0 0x7fe7a00b3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51340/0 (socket says 172.21.15.98:51340) 2023-03-03T07:58:28.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.533+0000 7fe7a599b700 1 -- 172.21.15.98:0/2391337088 learned_addr learned my addr 172.21.15.98:0/2391337088 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:28.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.533+0000 7fe7a599b700 1 -- 172.21.15.98:0/2391337088 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe7a015ca70 msgr2=0x7fe7a00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:28.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.533+0000 7fe7a599b700 1 --2- 172.21.15.98:0/2391337088 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe7a015ca70 0x7fe7a00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.533+0000 7fe7a599b700 1 -- 172.21.15.98:0/2391337088 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe7a00d0270 msgr2=0x7fe7a01518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:28.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.533+0000 7fe7a599b700 1 --2- 172.21.15.98:0/2391337088 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe7a00d0270 0x7fe7a01518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.533+0000 7fe7a599b700 1 -- 172.21.15.98:0/2391337088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7900094a0 con 0x7fe7a006b4b0 2023-03-03T07:58:28.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.533+0000 7fe7a599b700 1 --2- 172.21.15.98:0/2391337088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a006b4b0 0x7fe7a00b3830 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00c730 tx=0x7fe79c00caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:28.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.534+0000 7fe7967fc700 1 -- 172.21.15.98:0/2391337088 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c005d70 con 0x7fe7a006b4b0 2023-03-03T07:58:28.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.534+0000 7fe7967fc700 1 -- 172.21.15.98:0/2391337088 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79c0070d0 con 0x7fe7a006b4b0 2023-03-03T07:58:28.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.534+0000 7fe7967fc700 1 -- 172.21.15.98:0/2391337088 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c007460 con 0x7fe7a006b4b0 2023-03-03T07:58:28.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.534+0000 7fe7a7bff700 1 -- 172.21.15.98:0/2391337088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a00d1aa0 con 0x7fe7a006b4b0 2023-03-03T07:58:28.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.534+0000 7fe7a7bff700 1 -- 172.21.15.98:0/2391337088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a00cdbe0 con 0x7fe7a006b4b0 2023-03-03T07:58:28.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.535+0000 7fe7a7bff700 1 -- 172.21.15.98:0/2391337088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a015ce40 con 0x7fe7a006b4b0 2023-03-03T07:58:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.536+0000 7fe7967fc700 1 -- 172.21.15.98:0/2391337088 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fe79c019030 con 0x7fe7a006b4b0 2023-03-03T07:58:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.537+0000 7fe7967fc700 1 --2- 172.21.15.98:0/2391337088 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe7840683c0 0x7fe78406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.537+0000 7fe7967fc700 1 -- 172.21.15.98:0/2391337088 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fe79c085240 con 0x7fe7a006b4b0 2023-03-03T07:58:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.538+0000 7fe7a519a700 1 --2- 172.21.15.98:0/2391337088 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe7840683c0 0x7fe78406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.538+0000 7fe7a519a700 1 --2- 172.21.15.98:0/2391337088 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe7840683c0 0x7fe78406a870 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe790009870 tx=0x7fe790010000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:28.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.541+0000 7fe7967fc700 1 -- 172.21.15.98:0/2391337088 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe79c051990 con 0x7fe7a006b4b0 2023-03-03T07:58:28.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.747+0000 7fe7a7bff700 1 -- 172.21.15.98:0/2391337088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe7a004e6e0 con 0x7fe7a006b4b0 2023-03-03T07:58:28.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.748+0000 7fe7967fc700 1 -- 172.21.15.98:0/2391337088 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe79c054fb0 con 0x7fe7a006b4b0 2023-03-03T07:58:28.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.751+0000 7fe7a7bff700 1 -- 172.21.15.98:0/2391337088 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe7840683c0 msgr2=0x7fe78406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:28.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.751+0000 7fe7a7bff700 1 --2- 172.21.15.98:0/2391337088 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe7840683c0 0x7fe78406a870 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe790009870 tx=0x7fe790010000 comp rx=0 tx=0).stop 2023-03-03T07:58:28.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.751+0000 7fe7a7bff700 1 -- 172.21.15.98:0/2391337088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a006b4b0 msgr2=0x7fe7a00b3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:28.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.751+0000 7fe7a7bff700 1 --2- 172.21.15.98:0/2391337088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a006b4b0 0x7fe7a00b3830 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00c730 tx=0x7fe79c00caf0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.751+0000 7fe7a7bff700 1 -- 172.21.15.98:0/2391337088 shutdown_connections 2023-03-03T07:58:28.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.751+0000 7fe7a7bff700 1 --2- 172.21.15.98:0/2391337088 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe7840683c0 0x7fe78406a870 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.751+0000 7fe7a7bff700 1 --2- 172.21.15.98:0/2391337088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a006b4b0 0x7fe7a00b3830 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.751+0000 7fe7a7bff700 1 --2- 172.21.15.98:0/2391337088 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe7a015ca70 0x7fe7a00cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.751+0000 7fe7a7bff700 1 --2- 172.21.15.98:0/2391337088 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe7a00d0270 0x7fe7a01518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:28.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.751+0000 7fe7a7bff700 1 -- 172.21.15.98:0/2391337088 >> 172.21.15.98:0/2391337088 conn(0x7fe7a0073e70 msgr2=0x7fe7a00b6810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:28.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.751+0000 7fe7a7bff700 1 -- 172.21.15.98:0/2391337088 shutdown_connections 2023-03-03T07:58:28.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:28.751+0000 7fe7a7bff700 1 -- 172.21.15.98:0/2391337088 wait complete. 2023-03-03T07:58:28.779 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:58:28.779 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:58:28.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:58:29.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.091+0000 7f41b359d700 1 Processor -- start 2023-03-03T07:58:29.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.091+0000 7f41b359d700 1 -- start start 2023-03-03T07:58:29.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.093+0000 7f41b359d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41ac146360 0x7f41ac160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:29.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.093+0000 7f41b359d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f41ac0b4ab0 0x7f41ac0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:29.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.093+0000 7f41b359d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ac1614c0 con 0x7f41ac0731e0 2023-03-03T07:58:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.093+0000 7f41b359d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41ac0b5460 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.093+0000 7f41b359d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac0d10f0 con 0x7f41ac146360 2023-03-03T07:58:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.093+0000 7f41b1d9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f41ac0b4ab0 0x7f41ac0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.093+0000 7f41b1d9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f41ac0b4ab0 0x7f41ac0b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51346/0 (socket says 172.21.15.98:51346) 2023-03-03T07:58:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.093+0000 7f41b1d9a700 1 -- 172.21.15.98:0/3337203506 learned_addr learned my addr 172.21.15.98:0/3337203506 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.094+0000 7f41b259b700 1 --2- 172.21.15.98:0/3337203506 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41ac146360 0x7f41ac160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.094+0000 7f41b1d9a700 1 -- 172.21.15.98:0/3337203506 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41ac146360 msgr2=0x7f41ac160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.094+0000 7f41b1d9a700 1 --2- 172.21.15.98:0/3337203506 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41ac146360 0x7f41ac160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.094+0000 7f41b1d9a700 1 -- 172.21.15.98:0/3337203506 >> v1:172.21.15.100:6789/0 conn(0x7f41ac0731e0 legacy=0x7f41ac073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.094+0000 7f41b1d9a700 1 -- 172.21.15.98:0/3337203506 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac0cdf20 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.094+0000 7f41b259b700 1 --2- 172.21.15.98:0/3337203506 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41ac146360 0x7f41ac160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.094+0000 7f41b1d9a700 1 --2- 172.21.15.98:0/3337203506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ac0b4ab0 0x7f41ac0b4f20 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f419c0098f0 tx=0x7f419c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f01167ee4d8ae2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.094+0000 7f41b1599700 1 -- 172.21.15.98:0/3337203506 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c004230 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.094+0000 7f41b1599700 1 -- 172.21.15.98:0/3337203506 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419c004390 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.094+0000 7f41b1599700 1 -- 172.21.15.98:0/3337203506 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c0036a0 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.095+0000 7f41b359d700 1 -- 172.21.15.98:0/3337203506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ac0b4ab0 msgr2=0x7f41ac0b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.095+0000 7f41b359d700 1 --2- 172.21.15.98:0/3337203506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ac0b4ab0 0x7f41ac0b4f20 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f419c0098f0 tx=0x7f419c009c00 comp rx=0 tx=0).stop 2023-03-03T07:58:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.095+0000 7f41b359d700 1 -- 172.21.15.98:0/3337203506 shutdown_connections 2023-03-03T07:58:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.095+0000 7f41b359d700 1 --2- 172.21.15.98:0/3337203506 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41ac146360 0x7f41ac160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.095+0000 7f41b359d700 1 --2- 172.21.15.98:0/3337203506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ac0b4ab0 0x7f41ac0b4f20 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.095+0000 7f41b359d700 1 -- 172.21.15.98:0/3337203506 >> 172.21.15.98:0/3337203506 conn(0x7f41ac06b8e0 msgr2=0x7f41ac0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.095+0000 7f41b359d700 1 -- 172.21.15.98:0/3337203506 shutdown_connections 2023-03-03T07:58:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.095+0000 7f41b359d700 1 -- 172.21.15.98:0/3337203506 wait complete. 2023-03-03T07:58:29.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.097+0000 7f41b359d700 1 Processor -- start 2023-03-03T07:58:29.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.097+0000 7f41b359d700 1 -- start start 2023-03-03T07:58:29.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.098+0000 7f41b359d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41ac0731e0 0x7f41ac15cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:29.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.098+0000 7f41b359d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ac0b4ab0 0x7f41ac0c6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:29.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.098+0000 7f41b359d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41ac146360 0x7f41ac0caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:29.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.098+0000 7f41b359d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac0c7dc0 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.099+0000 7f41b359d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac0c80a0 con 0x7f41ac0731e0 2023-03-03T07:58:29.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.099+0000 7f41b359d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac0c8380 con 0x7f41ac146360 2023-03-03T07:58:29.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.099+0000 7f41b1d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ac0b4ab0 0x7f41ac0c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:29.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.099+0000 7f41b1d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ac0b4ab0 0x7f41ac0c6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51352/0 (socket says 172.21.15.98:51352) 2023-03-03T07:58:29.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.099+0000 7f41b1d9a700 1 -- 172.21.15.98:0/3968319075 learned_addr learned my addr 172.21.15.98:0/3968319075 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:29.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.099+0000 7f41b259b700 1 --2- 172.21.15.98:0/3968319075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41ac0731e0 0x7f41ac15cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:29.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.099+0000 7f41b2d9c700 1 --2- 172.21.15.98:0/3968319075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41ac146360 0x7f41ac0caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:29.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.099+0000 7f41b1d9a700 1 -- 172.21.15.98:0/3968319075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41ac146360 msgr2=0x7f41ac0caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:29.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.099+0000 7f41b1d9a700 1 --2- 172.21.15.98:0/3968319075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41ac146360 0x7f41ac0caaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.099+0000 7f41b1d9a700 1 -- 172.21.15.98:0/3968319075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41ac0731e0 msgr2=0x7f41ac15cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:29.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.099+0000 7f41b1d9a700 1 --2- 172.21.15.98:0/3968319075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41ac0731e0 0x7f41ac15cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.099+0000 7f41b1d9a700 1 -- 172.21.15.98:0/3968319075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c009550 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.099+0000 7f41b259b700 1 --2- 172.21.15.98:0/3968319075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41ac0731e0 0x7f41ac15cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:29.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.100+0000 7f41b1d9a700 1 --2- 172.21.15.98:0/3968319075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ac0b4ab0 0x7f41ac0c6040 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f419c009130 tx=0x7f419c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:29.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.100+0000 7f41a2ffd700 1 -- 172.21.15.98:0/3968319075 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c00b860 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.100+0000 7f41a2ffd700 1 -- 172.21.15.98:0/3968319075 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419c00b9c0 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.100+0000 7f41a2ffd700 1 -- 172.21.15.98:0/3968319075 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c00bd70 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.100+0000 7f41b359d700 1 -- 172.21.15.98:0/3968319075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ac0cd960 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.100+0000 7f41b359d700 1 -- 172.21.15.98:0/3968319075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ac0cd680 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.101+0000 7f41b359d700 1 -- 172.21.15.98:0/3968319075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41ac0cdc40 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.102+0000 7f41a2ffd700 1 -- 172.21.15.98:0/3968319075 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f419c017020 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.103+0000 7f41a2ffd700 1 --2- 172.21.15.98:0/3968319075 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f41900683c0 0x7f419006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.104+0000 7f41a2ffd700 1 -- 172.21.15.98:0/3968319075 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f419c087630 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.104+0000 7f41b259b700 1 --2- 172.21.15.98:0/3968319075 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f41900683c0 0x7f419006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.104+0000 7f41b259b700 1 --2- 172.21.15.98:0/3968319075 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f41900683c0 0x7f419006a870 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f41a8006fd0 tx=0x7f41a8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:29.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.107+0000 7f41a2ffd700 1 -- 172.21.15.98:0/3968319075 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f419c053c90 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.312+0000 7f41b359d700 1 -- 172.21.15.98:0/3968319075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41ac1614c0 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.314+0000 7f41a2ffd700 1 -- 172.21.15.98:0/3968319075 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+18018 (secure 0 0 0) 0x7f41ac1614c0 con 0x7f41ac0b4ab0 2023-03-03T07:58:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:29.316 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":54,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:58:28.723281+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:58:10.728830+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-03-03T07:58:12.224649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.6","mappings":[{"from":0,"to":1}]},{"pgid":"8.17","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:58:29.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.316+0000 7f41b359d700 1 -- 172.21.15.98:0/3968319075 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f41900683c0 msgr2=0x7f419006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:29.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.316+0000 7f41b359d700 1 --2- 172.21.15.98:0/3968319075 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f41900683c0 0x7f419006a870 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f41a8006fd0 tx=0x7f41a8008040 comp rx=0 tx=0).stop 2023-03-03T07:58:29.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.316+0000 7f41b359d700 1 -- 172.21.15.98:0/3968319075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ac0b4ab0 msgr2=0x7f41ac0c6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:29.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.316+0000 7f41b359d700 1 --2- 172.21.15.98:0/3968319075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ac0b4ab0 0x7f41ac0c6040 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f419c009130 tx=0x7f419c004750 comp rx=0 tx=0).stop 2023-03-03T07:58:29.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.317+0000 7f41b359d700 1 -- 172.21.15.98:0/3968319075 shutdown_connections 2023-03-03T07:58:29.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.317+0000 7f41b359d700 1 --2- 172.21.15.98:0/3968319075 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f41900683c0 0x7f419006a870 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.317+0000 7f41b359d700 1 --2- 172.21.15.98:0/3968319075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41ac146360 0x7f41ac0caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.317+0000 7f41b359d700 1 --2- 172.21.15.98:0/3968319075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41ac0731e0 0x7f41ac15cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.317+0000 7f41b359d700 1 --2- 172.21.15.98:0/3968319075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ac0b4ab0 0x7f41ac0c6040 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.317+0000 7f41b359d700 1 -- 172.21.15.98:0/3968319075 >> 172.21.15.98:0/3968319075 conn(0x7f41ac06b8e0 msgr2=0x7f41ac0b33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:29.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.317+0000 7f41b359d700 1 -- 172.21.15.98:0/3968319075 shutdown_connections 2023-03-03T07:58:29.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.317+0000 7f41b359d700 1 -- 172.21.15.98:0/3968319075 wait complete. 2023-03-03T07:58:29.345 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:58:10.728830+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:58:12.224649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:58:29.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:58:29.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.594+0000 7f32aed9d700 1 Processor -- start 2023-03-03T07:58:29.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.594+0000 7f32aed9d700 1 -- start start 2023-03-03T07:58:29.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.595+0000 7f32aed9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f32a80b5e70 0x7f32a80b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:29.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.595+0000 7f32aed9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f32a80cf570 0x7f32a80b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:29.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.595+0000 7f32aed9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a815ce00 con 0x7f32a80b8910 2023-03-03T07:58:29.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.596+0000 7f32aed9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a80d0bf0 con 0x7f32a80cf570 2023-03-03T07:58:29.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.596+0000 7f32aed9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a80b8ce0 con 0x7f32a80b5e70 2023-03-03T07:58:29.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.596+0000 7f32ae59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f32a80cf570 0x7f32a80b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:29.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.596+0000 7f32ae59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f32a80cf570 0x7f32a80b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51362/0 (socket says 172.21.15.98:51362) 2023-03-03T07:58:29.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.596+0000 7f32ae59c700 1 -- 172.21.15.98:0/51843174 learned_addr learned my addr 172.21.15.98:0/51843174 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:29.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.596+0000 7f32ad59a700 1 --2- 172.21.15.98:0/51843174 >> v2:172.21.15.100:3300/0 conn(0x7f32a80b5e70 0x7f32a80b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:29.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.596+0000 7f32ae59c700 1 -- 172.21.15.98:0/51843174 >> v2:172.21.15.100:3300/0 conn(0x7f32a80b5e70 msgr2=0x7f32a80b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:29.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.596+0000 7f32ae59c700 1 --2- 172.21.15.98:0/51843174 >> v2:172.21.15.100:3300/0 conn(0x7f32a80b5e70 0x7f32a80b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.596+0000 7f32ae59c700 1 -- 172.21.15.98:0/51843174 >> v1:172.21.15.100:6789/0 conn(0x7f32a80b8910 legacy=0x7f32a81604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:29.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.596+0000 7f32ae59c700 1 -- 172.21.15.98:0/51843174 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a8160cf0 con 0x7f32a80cf570 2023-03-03T07:58:29.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.597+0000 7f32ae59c700 1 --2- 172.21.15.98:0/51843174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a80cf570 0x7f32a80b4fe0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f32a8074810 tx=0x7f32a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b44453a712aa8af server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:29.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.597+0000 7f32acd99700 1 -- 172.21.15.98:0/51843174 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a000e070 con 0x7f32a80cf570 2023-03-03T07:58:29.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.597+0000 7f32acd99700 1 -- 172.21.15.98:0/51843174 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a000b9c0 con 0x7f32a80cf570 2023-03-03T07:58:29.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.597+0000 7f32acd99700 1 -- 172.21.15.98:0/51843174 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a000bd70 con 0x7f32a80cf570 2023-03-03T07:58:29.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.597+0000 7f32aed9d700 1 -- 172.21.15.98:0/51843174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a80cf570 msgr2=0x7f32a80b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:29.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.597+0000 7f32aed9d700 1 --2- 172.21.15.98:0/51843174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a80cf570 0x7f32a80b4fe0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f32a8074810 tx=0x7f32a000b330 comp rx=0 tx=0).stop 2023-03-03T07:58:29.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.598+0000 7f32aed9d700 1 -- 172.21.15.98:0/51843174 shutdown_connections 2023-03-03T07:58:29.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.598+0000 7f32aed9d700 1 --2- 172.21.15.98:0/51843174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a80cf570 0x7f32a80b4fe0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.598+0000 7f32aed9d700 1 --2- 172.21.15.98:0/51843174 >> v2:172.21.15.100:3300/0 conn(0x7f32a80b5e70 0x7f32a80b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.598+0000 7f32aed9d700 1 -- 172.21.15.98:0/51843174 >> 172.21.15.98:0/51843174 conn(0x7f32a806b880 msgr2=0x7f32a80cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:29.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.598+0000 7f32aed9d700 1 -- 172.21.15.98:0/51843174 shutdown_connections 2023-03-03T07:58:29.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.598+0000 7f32aed9d700 1 -- 172.21.15.98:0/51843174 wait complete. 2023-03-03T07:58:29.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.600+0000 7f32aed9d700 1 Processor -- start 2023-03-03T07:58:29.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.601+0000 7f32aed9d700 1 -- start start 2023-03-03T07:58:29.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.602+0000 7f32aed9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f32a80b5e70 0x7f32a815b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:29.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.602+0000 7f32aed9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f32a80b8910 0x7f32a80cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.602+0000 7f32aed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a80cf570 0x7f32a80cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.602+0000 7f32aed9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a80b3b00 con 0x7f32a80cf570 2023-03-03T07:58:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.602+0000 7f32aed9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a80b27f0 con 0x7f32a80b8910 2023-03-03T07:58:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.602+0000 7f32aed9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a81e7340 con 0x7f32a80b5e70 2023-03-03T07:58:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.603+0000 7f32ae59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a80cf570 0x7f32a80cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.603+0000 7f32ae59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a80cf570 0x7f32a80cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51370/0 (socket says 172.21.15.98:51370) 2023-03-03T07:58:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.603+0000 7f32ae59c700 1 -- 172.21.15.98:0/1417878736 learned_addr learned my addr 172.21.15.98:0/1417878736 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:29.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.603+0000 7f32ad59a700 1 --2- 172.21.15.98:0/1417878736 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f32a80b8910 0x7f32a80cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:29.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.603+0000 7f32add9b700 1 --2- 172.21.15.98:0/1417878736 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f32a80b5e70 0x7f32a815b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:29.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.603+0000 7f32ae59c700 1 -- 172.21.15.98:0/1417878736 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f32a80b5e70 msgr2=0x7f32a815b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:29.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.603+0000 7f32ae59c700 1 --2- 172.21.15.98:0/1417878736 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f32a80b5e70 0x7f32a815b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.603+0000 7f32ae59c700 1 -- 172.21.15.98:0/1417878736 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f32a80b8910 msgr2=0x7f32a80cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:29.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.603+0000 7f32ae59c700 1 --2- 172.21.15.98:0/1417878736 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f32a80b8910 0x7f32a80cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.603+0000 7f32ae59c700 1 -- 172.21.15.98:0/1417878736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a0009b30 con 0x7f32a80cf570 2023-03-03T07:58:29.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.603+0000 7f32add9b700 1 --2- 172.21.15.98:0/1417878736 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f32a80b5e70 0x7f32a815b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:29.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.603+0000 7f32ae59c700 1 --2- 172.21.15.98:0/1417878736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a80cf570 0x7f32a80cc800 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f32a000b370 tx=0x7f32a0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:29.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.604+0000 7f329e7fc700 1 -- 172.21.15.98:0/1417878736 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a00099b0 con 0x7f32a80cf570 2023-03-03T07:58:29.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.604+0000 7f329e7fc700 1 -- 172.21.15.98:0/1417878736 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a00092e0 con 0x7f32a80cf570 2023-03-03T07:58:29.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.604+0000 7f32aed9d700 1 -- 172.21.15.98:0/1417878736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a81612b0 con 0x7f32a80cf570 2023-03-03T07:58:29.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.604+0000 7f329e7fc700 1 -- 172.21.15.98:0/1417878736 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a001c640 con 0x7f32a80cf570 2023-03-03T07:58:29.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.604+0000 7f32aed9d700 1 -- 172.21.15.98:0/1417878736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a8160fd0 con 0x7f32a80cf570 2023-03-03T07:58:29.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.605+0000 7f32aed9d700 1 -- 172.21.15.98:0/1417878736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a80d0eb0 con 0x7f32a80cf570 2023-03-03T07:58:29.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.606+0000 7f329e7fc700 1 -- 172.21.15.98:0/1417878736 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f32a0022030 con 0x7f32a80cf570 2023-03-03T07:58:29.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.607+0000 7f329e7fc700 1 --2- 172.21.15.98:0/1417878736 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f328c0683c0 0x7f328c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:29.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.607+0000 7f329e7fc700 1 -- 172.21.15.98:0/1417878736 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f32a00877f0 con 0x7f32a80cf570 2023-03-03T07:58:29.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.608+0000 7f32add9b700 1 --2- 172.21.15.98:0/1417878736 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f328c0683c0 0x7f328c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:29.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.608+0000 7f32add9b700 1 --2- 172.21.15.98:0/1417878736 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f328c0683c0 0x7f328c06a870 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f32a4000f60 tx=0x7f32a4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:29.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.612+0000 7f329e7fc700 1 -- 172.21.15.98:0/1417878736 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f32a0053e50 con 0x7f32a80cf570 2023-03-03T07:58:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.808+0000 7f32aed9d700 1 -- 172.21.15.98:0/1417878736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f32a80b3cb0 con 0x7f32a80cf570 2023-03-03T07:58:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.808+0000 7f329e7fc700 1 -- 172.21.15.98:0/1417878736 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f32a80b3cb0 con 0x7f32a80cf570 2023-03-03T07:58:29.810 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:58:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.811+0000 7f32aed9d700 1 -- 172.21.15.98:0/1417878736 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f328c0683c0 msgr2=0x7f328c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.811+0000 7f32aed9d700 1 --2- 172.21.15.98:0/1417878736 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f328c0683c0 0x7f328c06a870 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f32a4000f60 tx=0x7f32a4009040 comp rx=0 tx=0).stop 2023-03-03T07:58:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.811+0000 7f32aed9d700 1 -- 172.21.15.98:0/1417878736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a80cf570 msgr2=0x7f32a80cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.811+0000 7f32aed9d700 1 --2- 172.21.15.98:0/1417878736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a80cf570 0x7f32a80cc800 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f32a000b370 tx=0x7f32a0004ad0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.811+0000 7f32aed9d700 1 -- 172.21.15.98:0/1417878736 shutdown_connections 2023-03-03T07:58:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.811+0000 7f32aed9d700 1 --2- 172.21.15.98:0/1417878736 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f328c0683c0 0x7f328c06a870 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.811+0000 7f32aed9d700 1 --2- 172.21.15.98:0/1417878736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a80cf570 0x7f32a80cc800 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.811+0000 7f32aed9d700 1 --2- 172.21.15.98:0/1417878736 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f32a80b5e70 0x7f32a815b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.811+0000 7f32aed9d700 1 --2- 172.21.15.98:0/1417878736 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f32a80b8910 0x7f32a80cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.811+0000 7f32aed9d700 1 -- 172.21.15.98:0/1417878736 >> 172.21.15.98:0/1417878736 conn(0x7f32a806b880 msgr2=0x7f32a80b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.811+0000 7f32aed9d700 1 -- 172.21.15.98:0/1417878736 shutdown_connections 2023-03-03T07:58:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:29.811+0000 7f32aed9d700 1 -- 172.21.15.98:0/1417878736 wait complete. 2023-03-03T07:58:29.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:58:30.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.088+0000 7fc4bb5ff700 1 Processor -- start 2023-03-03T07:58:30.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.088+0000 7fc4bb5ff700 1 -- start start 2023-03-03T07:58:30.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.089+0000 7fc4bb5ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc4b40b5310 0x7fc4b4167850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:30.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.089+0000 7fc4bb5ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b40cd7d0 con 0x7fc4b40b8d10 2023-03-03T07:58:30.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.089+0000 7fc4bb5ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b40b90e0 con 0x7fc4b40b26d0 2023-03-03T07:58:30.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.089+0000 7fc4bb5ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b40cda20 con 0x7fc4b40b5310 2023-03-03T07:58:30.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.090+0000 7fc4b939b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc4b40b8d10 0x7fc4b40cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56406/0 (socket says 172.21.15.98:56406) 2023-03-03T07:58:30.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.090+0000 7fc4b939b700 1 -- 172.21.15.98:0/2298120066 learned_addr learned my addr 172.21.15.98:0/2298120066 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:30.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.090+0000 7fc4b9b9c700 1 --2- 172.21.15.98:0/2298120066 >> v2:172.21.15.98:3300/0 conn(0x7fc4b40b5310 0x7fc4b4167850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:30.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.090+0000 7fc4abfff700 1 -- 172.21.15.98:0/2298120066 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203211358 0 0) 0x7fc4b40cd7d0 con 0x7fc4b40b8d10 2023-03-03T07:58:30.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.090+0000 7fc4abfff700 1 -- 172.21.15.98:0/2298120066 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a00035b0 con 0x7fc4b40b8d10 2023-03-03T07:58:30.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.090+0000 7fc4b9b9c700 1 -- 172.21.15.98:0/2298120066 >> v1:172.21.15.100:6789/0 conn(0x7fc4b40b26d0 legacy=0x7fc4b40b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:30.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.090+0000 7fc4b9b9c700 1 -- 172.21.15.98:0/2298120066 >> v1:172.21.15.98:6789/0 conn(0x7fc4b40b8d10 legacy=0x7fc4b40cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:30.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.090+0000 7fc4b9b9c700 1 -- 172.21.15.98:0/2298120066 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b40cdd00 con 0x7fc4b40b5310 2023-03-03T07:58:30.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.091+0000 7fc4b9b9c700 1 --2- 172.21.15.98:0/2298120066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4b40b5310 0x7fc4b4167850 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009d90 tx=0x7fc4ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87de3125f966536 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:30.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.091+0000 7fc4abfff700 1 -- 172.21.15.98:0/2298120066 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac00e070 con 0x7fc4b40b5310 2023-03-03T07:58:30.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.091+0000 7fc4abfff700 1 -- 172.21.15.98:0/2298120066 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ac00bb20 con 0x7fc4b40b5310 2023-03-03T07:58:30.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.091+0000 7fc4abfff700 1 -- 172.21.15.98:0/2298120066 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac012430 con 0x7fc4b40b5310 2023-03-03T07:58:30.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.091+0000 7fc4bb5ff700 1 -- 172.21.15.98:0/2298120066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4b40b5310 msgr2=0x7fc4b4167850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:30.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.091+0000 7fc4bb5ff700 1 --2- 172.21.15.98:0/2298120066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4b40b5310 0x7fc4b4167850 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009d90 tx=0x7fc4ac00b330 comp rx=0 tx=0).stop 2023-03-03T07:58:30.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.092+0000 7fc4bb5ff700 1 -- 172.21.15.98:0/2298120066 shutdown_connections 2023-03-03T07:58:30.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.092+0000 7fc4bb5ff700 1 --2- 172.21.15.98:0/2298120066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4b40b5310 0x7fc4b4167850 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.092+0000 7fc4bb5ff700 1 -- 172.21.15.98:0/2298120066 >> 172.21.15.98:0/2298120066 conn(0x7fc4b4073e70 msgr2=0x7fc4b406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:30.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.092+0000 7fc4bb5ff700 1 -- 172.21.15.98:0/2298120066 shutdown_connections 2023-03-03T07:58:30.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.092+0000 7fc4bb5ff700 1 -- 172.21.15.98:0/2298120066 wait complete. 2023-03-03T07:58:30.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.094+0000 7fc4bb5ff700 1 Processor -- start 2023-03-03T07:58:30.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.094+0000 7fc4bb5ff700 1 -- start start 2023-03-03T07:58:30.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.095+0000 7fc4bb5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4b40b26d0 0x7fc4b40cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:30.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.095+0000 7fc4bb5ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4b40b5310 0x7fc4b41568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:30.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.095+0000 7fc4bb5ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4b40b8d10 0x7fc4b40b3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:30.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.095+0000 7fc4bb5ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b41f3420 con 0x7fc4b40b5310 2023-03-03T07:58:30.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.095+0000 7fc4bb5ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b41f27a0 con 0x7fc4b40b8d10 2023-03-03T07:58:30.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.095+0000 7fc4bb5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b41f2a80 con 0x7fc4b40b26d0 2023-03-03T07:58:30.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.095+0000 7fc4b8b9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4b40b5310 0x7fc4b41568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:30.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.096+0000 7fc4b8b9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4b40b5310 0x7fc4b41568f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51382/0 (socket says 172.21.15.98:51382) 2023-03-03T07:58:30.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.096+0000 7fc4b8b9a700 1 -- 172.21.15.98:0/2865444380 learned_addr learned my addr 172.21.15.98:0/2865444380 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:30.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.096+0000 7fc4b9b9c700 1 --2- 172.21.15.98:0/2865444380 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4b40b8d10 0x7fc4b40b3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:30.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.096+0000 7fc4b939b700 1 --2- 172.21.15.98:0/2865444380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4b40b26d0 0x7fc4b40cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:30.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.096+0000 7fc4b8b9a700 1 -- 172.21.15.98:0/2865444380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4b40b26d0 msgr2=0x7fc4b40cd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:30.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.096+0000 7fc4b8b9a700 1 --2- 172.21.15.98:0/2865444380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4b40b26d0 0x7fc4b40cd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.096+0000 7fc4b8b9a700 1 -- 172.21.15.98:0/2865444380 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4b40b8d10 msgr2=0x7fc4b40b3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:30.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.096+0000 7fc4b8b9a700 1 --2- 172.21.15.98:0/2865444380 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4b40b8d10 0x7fc4b40b3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.096+0000 7fc4b8b9a700 1 -- 172.21.15.98:0/2865444380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac009a70 con 0x7fc4b40b5310 2023-03-03T07:58:30.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.096+0000 7fc4b939b700 1 --2- 172.21.15.98:0/2865444380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4b40b26d0 0x7fc4b40cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:30.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.096+0000 7fc4b8b9a700 1 --2- 172.21.15.98:0/2865444380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4b40b5310 0x7fc4b41568f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400c450 tx=0x7fc4a400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:30.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.097+0000 7fc4a9ffb700 1 -- 172.21.15.98:0/2865444380 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40052d0 con 0x7fc4b40b5310 2023-03-03T07:58:30.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.097+0000 7fc4a9ffb700 1 -- 172.21.15.98:0/2865444380 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a4005430 con 0x7fc4b40b5310 2023-03-03T07:58:30.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.097+0000 7fc4bb5ff700 1 -- 172.21.15.98:0/2865444380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b40cdfe0 con 0x7fc4b40b5310 2023-03-03T07:58:30.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.097+0000 7fc4a9ffb700 1 -- 172.21.15.98:0/2865444380 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40038c0 con 0x7fc4b40b5310 2023-03-03T07:58:30.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.097+0000 7fc4bb5ff700 1 -- 172.21.15.98:0/2865444380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b40d0e80 con 0x7fc4b40b5310 2023-03-03T07:58:30.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.098+0000 7fc4bb5ff700 1 -- 172.21.15.98:0/2865444380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b40b5fd0 con 0x7fc4b40b5310 2023-03-03T07:58:30.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.099+0000 7fc4a9ffb700 1 -- 172.21.15.98:0/2865444380 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fc4a4019030 con 0x7fc4b40b5310 2023-03-03T07:58:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.101+0000 7fc4a9ffb700 1 --2- 172.21.15.98:0/2865444380 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc4980683c0 0x7fc49806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:30.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.101+0000 7fc4a9ffb700 1 -- 172.21.15.98:0/2865444380 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fc4a4085010 con 0x7fc4b40b5310 2023-03-03T07:58:30.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.102+0000 7fc4b939b700 1 --2- 172.21.15.98:0/2865444380 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc4980683c0 0x7fc49806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:30.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.102+0000 7fc4b939b700 1 --2- 172.21.15.98:0/2865444380 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc4980683c0 0x7fc49806a870 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc4b00011e0 tx=0x7fc4b0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.105+0000 7fc4a9ffb700 1 -- 172.21.15.98:0/2865444380 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc4a4051760 con 0x7fc4b40b5310 2023-03-03T07:58:30.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.303+0000 7fc4bb5ff700 1 -- 172.21.15.98:0/2865444380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc4b4158cf0 con 0x7fc4b40b5310 2023-03-03T07:58:30.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.303+0000 7fc4a9ffb700 1 -- 172.21.15.98:0/2865444380 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v54) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc4b4158cf0 con 0x7fc4b40b5310 2023-03-03T07:58:30.305 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:58:30.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.306+0000 7fc4bb5ff700 1 -- 172.21.15.98:0/2865444380 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc4980683c0 msgr2=0x7fc49806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:30.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.306+0000 7fc4bb5ff700 1 --2- 172.21.15.98:0/2865444380 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc4980683c0 0x7fc49806a870 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc4b00011e0 tx=0x7fc4b0009040 comp rx=0 tx=0).stop 2023-03-03T07:58:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.306+0000 7fc4bb5ff700 1 -- 172.21.15.98:0/2865444380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4b40b5310 msgr2=0x7fc4b41568f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.306+0000 7fc4bb5ff700 1 --2- 172.21.15.98:0/2865444380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4b40b5310 0x7fc4b41568f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400c450 tx=0x7fc4a400c810 comp rx=0 tx=0).stop 2023-03-03T07:58:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.306+0000 7fc4bb5ff700 1 -- 172.21.15.98:0/2865444380 shutdown_connections 2023-03-03T07:58:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.306+0000 7fc4bb5ff700 1 --2- 172.21.15.98:0/2865444380 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc4980683c0 0x7fc49806a870 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.306+0000 7fc4bb5ff700 1 --2- 172.21.15.98:0/2865444380 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4b40b8d10 0x7fc4b40b3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.306+0000 7fc4bb5ff700 1 --2- 172.21.15.98:0/2865444380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4b40b26d0 0x7fc4b40cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.306+0000 7fc4bb5ff700 1 --2- 172.21.15.98:0/2865444380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4b40b5310 0x7fc4b41568f0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.306+0000 7fc4bb5ff700 1 -- 172.21.15.98:0/2865444380 >> 172.21.15.98:0/2865444380 conn(0x7fc4b4073e70 msgr2=0x7fc4b415c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.306+0000 7fc4bb5ff700 1 -- 172.21.15.98:0/2865444380 shutdown_connections 2023-03-03T07:58:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.306+0000 7fc4bb5ff700 1 -- 172.21.15.98:0/2865444380 wait complete. 2023-03-03T07:58:30.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:58:30.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.582+0000 7febaf59e700 1 Processor -- start 2023-03-03T07:58:30.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.583+0000 7febaf59e700 1 -- start start 2023-03-03T07:58:30.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.584+0000 7febaf59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7febb01609b0 0x7febb0160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:30.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.584+0000 7febaf59e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7febb00cf570 0x7febb015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:30.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.584+0000 7febaf59e700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb01612c0 con 0x7febb015db30 2023-03-03T07:58:30.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.584+0000 7febaf59e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb00cebb0 con 0x7febb01609b0 2023-03-03T07:58:30.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.584+0000 7febaf59e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb00cd780 con 0x7febb00cf570 2023-03-03T07:58:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.584+0000 7febae59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7febb01609b0 0x7febb0160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.584+0000 7febae59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7febb01609b0 0x7febb0160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51398/0 (socket says 172.21.15.98:51398) 2023-03-03T07:58:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.584+0000 7febae59c700 1 -- 172.21.15.98:0/2862484770 learned_addr learned my addr 172.21.15.98:0/2862484770 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.585+0000 7febadd9b700 1 --2- 172.21.15.98:0/2862484770 >> v2:172.21.15.100:3300/0 conn(0x7febb00cf570 0x7febb015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.585+0000 7febae59c700 1 -- 172.21.15.98:0/2862484770 >> v2:172.21.15.100:3300/0 conn(0x7febb00cf570 msgr2=0x7febb015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.585+0000 7febae59c700 1 --2- 172.21.15.98:0/2862484770 >> v2:172.21.15.100:3300/0 conn(0x7febb00cf570 0x7febb015d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.585+0000 7febae59c700 1 -- 172.21.15.98:0/2862484770 >> v1:172.21.15.100:6789/0 conn(0x7febb015db30 legacy=0x7febb00b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.585+0000 7febae59c700 1 -- 172.21.15.98:0/2862484770 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb00cda60 con 0x7febb01609b0 2023-03-03T07:58:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.585+0000 7febae59c700 1 --2- 172.21.15.98:0/2862484770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febb01609b0 0x7febb0160d80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7febb00cf0c0 tx=0x7feba0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93262d23125c80a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.586+0000 7febad59a700 1 -- 172.21.15.98:0/2862484770 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba0011830 con 0x7febb01609b0 2023-03-03T07:58:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.586+0000 7febad59a700 1 -- 172.21.15.98:0/2862484770 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba0011990 con 0x7febb01609b0 2023-03-03T07:58:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.586+0000 7febad59a700 1 -- 172.21.15.98:0/2862484770 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba0011d20 con 0x7febb01609b0 2023-03-03T07:58:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.586+0000 7febaf59e700 1 -- 172.21.15.98:0/2862484770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febb01609b0 msgr2=0x7febb0160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.586+0000 7febaf59e700 1 --2- 172.21.15.98:0/2862484770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febb01609b0 0x7febb0160d80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7febb00cf0c0 tx=0x7feba0009bd0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.586+0000 7febaf59e700 1 -- 172.21.15.98:0/2862484770 shutdown_connections 2023-03-03T07:58:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.586+0000 7febaf59e700 1 --2- 172.21.15.98:0/2862484770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febb01609b0 0x7febb0160d80 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.586+0000 7febaf59e700 1 --2- 172.21.15.98:0/2862484770 >> v2:172.21.15.100:3300/0 conn(0x7febb00cf570 0x7febb015d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.586+0000 7febaf59e700 1 -- 172.21.15.98:0/2862484770 >> 172.21.15.98:0/2862484770 conn(0x7febb006b880 msgr2=0x7febb00cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.587+0000 7febaf59e700 1 -- 172.21.15.98:0/2862484770 shutdown_connections 2023-03-03T07:58:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.587+0000 7febaf59e700 1 -- 172.21.15.98:0/2862484770 wait complete. 2023-03-03T07:58:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.589+0000 7febaf59e700 1 Processor -- start 2023-03-03T07:58:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.589+0000 7febaf59e700 1 -- start start 2023-03-03T07:58:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.590+0000 7febaf59e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febb00cf570 0x7febb01485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.590+0000 7febaf59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febb015db30 0x7febb00b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.590+0000 7febaf59e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febb01609b0 0x7febb00b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.590+0000 7febaf59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb00b8dc0 con 0x7febb015db30 2023-03-03T07:58:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.590+0000 7febaf59e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb00c7340 con 0x7febb01609b0 2023-03-03T07:58:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.590+0000 7febaf59e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb00b8ae0 con 0x7febb00cf570 2023-03-03T07:58:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.590+0000 7febadd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febb015db30 0x7febb00b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.590+0000 7febadd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febb015db30 0x7febb00b4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51404/0 (socket says 172.21.15.98:51404) 2023-03-03T07:58:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.590+0000 7febadd9b700 1 -- 172.21.15.98:0/1390301569 learned_addr learned my addr 172.21.15.98:0/1390301569 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.590+0000 7febae59c700 1 --2- 172.21.15.98:0/1390301569 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febb00cf570 0x7febb01485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.590+0000 7febaed9d700 1 --2- 172.21.15.98:0/1390301569 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febb01609b0 0x7febb00b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.591+0000 7febadd9b700 1 -- 172.21.15.98:0/1390301569 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febb00cf570 msgr2=0x7febb01485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.591+0000 7febadd9b700 1 --2- 172.21.15.98:0/1390301569 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febb00cf570 0x7febb01485a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.591+0000 7febadd9b700 1 -- 172.21.15.98:0/1390301569 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febb01609b0 msgr2=0x7febb00b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.591+0000 7febadd9b700 1 --2- 172.21.15.98:0/1390301569 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febb01609b0 0x7febb00b4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.591+0000 7febadd9b700 1 -- 172.21.15.98:0/1390301569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba0009550 con 0x7febb015db30 2023-03-03T07:58:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.591+0000 7febadd9b700 1 --2- 172.21.15.98:0/1390301569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febb015db30 0x7febb00b4a90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7feba400e9d0 tx=0x7feba400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.591+0000 7feb9effd700 1 -- 172.21.15.98:0/1390301569 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba4009730 con 0x7febb015db30 2023-03-03T07:58:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.591+0000 7feb9effd700 1 -- 172.21.15.98:0/1390301569 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba400ee70 con 0x7febb015db30 2023-03-03T07:58:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.591+0000 7feb9effd700 1 -- 172.21.15.98:0/1390301569 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba400ca70 con 0x7febb015db30 2023-03-03T07:58:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.591+0000 7febaf59e700 1 -- 172.21.15.98:0/1390301569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb015cdf0 con 0x7febb015db30 2023-03-03T07:58:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.592+0000 7febaf59e700 1 -- 172.21.15.98:0/1390301569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb00cdd40 con 0x7febb015db30 2023-03-03T07:58:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.593+0000 7febaf59e700 1 -- 172.21.15.98:0/1390301569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb015d0d0 con 0x7febb015db30 2023-03-03T07:58:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.594+0000 7feb9effd700 1 -- 172.21.15.98:0/1390301569 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7feba4023070 con 0x7febb015db30 2023-03-03T07:58:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.595+0000 7feb9effd700 1 --2- 172.21.15.98:0/1390301569 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feb900683c0 0x7feb9006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.595+0000 7feb9effd700 1 -- 172.21.15.98:0/1390301569 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7feba4086130 con 0x7febb015db30 2023-03-03T07:58:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.595+0000 7febae59c700 1 --2- 172.21.15.98:0/1390301569 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feb900683c0 0x7feb9006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.596+0000 7febae59c700 1 --2- 172.21.15.98:0/1390301569 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feb900683c0 0x7feb9006a870 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7feba0000c00 tx=0x7feba0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.599+0000 7feb9effd700 1 -- 172.21.15.98:0/1390301569 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feba4052790 con 0x7febb015db30 2023-03-03T07:58:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.796+0000 7febaf59e700 1 -- 172.21.15.98:0/1390301569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7febb00c6b50 con 0x7febb015db30 2023-03-03T07:58:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.796+0000 7feb9effd700 1 -- 172.21.15.98:0/1390301569 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7febb00c6b50 con 0x7febb015db30 2023-03-03T07:58:30.798 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:58:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.799+0000 7febaf59e700 1 -- 172.21.15.98:0/1390301569 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feb900683c0 msgr2=0x7feb9006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.799+0000 7febaf59e700 1 --2- 172.21.15.98:0/1390301569 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feb900683c0 0x7feb9006a870 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7feba0000c00 tx=0x7feba0008030 comp rx=0 tx=0).stop 2023-03-03T07:58:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.799+0000 7febaf59e700 1 -- 172.21.15.98:0/1390301569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febb015db30 msgr2=0x7febb00b4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.799+0000 7febaf59e700 1 --2- 172.21.15.98:0/1390301569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febb015db30 0x7febb00b4a90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7feba400e9d0 tx=0x7feba400ece0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.799+0000 7febaf59e700 1 -- 172.21.15.98:0/1390301569 shutdown_connections 2023-03-03T07:58:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.799+0000 7febaf59e700 1 --2- 172.21.15.98:0/1390301569 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feb900683c0 0x7feb9006a870 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.799+0000 7febaf59e700 1 --2- 172.21.15.98:0/1390301569 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febb01609b0 0x7febb00b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.799+0000 7febaf59e700 1 --2- 172.21.15.98:0/1390301569 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febb00cf570 0x7febb01485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.799+0000 7febaf59e700 1 --2- 172.21.15.98:0/1390301569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febb015db30 0x7febb00b4a90 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.799+0000 7febaf59e700 1 -- 172.21.15.98:0/1390301569 >> 172.21.15.98:0/1390301569 conn(0x7febb006b880 msgr2=0x7febb015bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.799+0000 7febaf59e700 1 -- 172.21.15.98:0/1390301569 shutdown_connections 2023-03-03T07:58:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:30.799+0000 7febaf59e700 1 -- 172.21.15.98:0/1390301569 wait complete. 2023-03-03T07:58:30.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:58:31.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.073+0000 7f1cb19ff700 1 Processor -- start 2023-03-03T07:58:31.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.074+0000 7f1cb19ff700 1 -- start start 2023-03-03T07:58:31.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.075+0000 7f1cb19ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cac15ca70 0x7f1cac0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:31.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.075+0000 7f1cb19ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1cac06b4b0 0x7f1cac0cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:31.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.075+0000 7f1cb19ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1cac0d00a0 0x7f1cac0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:31.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.075+0000 7f1cb19ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cac0cd5b0 con 0x7f1cac0d00a0 2023-03-03T07:58:31.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.075+0000 7f1cb19ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cac0cf980 con 0x7f1cac06b4b0 2023-03-03T07:58:31.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.075+0000 7f1cb19ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0d0510 con 0x7f1cac15ca70 2023-03-03T07:58:31.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.076+0000 7f1cab7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1cac0d00a0 0x7f1cac0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:31.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.076+0000 7f1cab7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1cac0d00a0 0x7f1cac0b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51416/0 (socket says 172.21.15.98:51416) 2023-03-03T07:58:31.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.076+0000 7f1cab7fe700 1 -- 172.21.15.98:0/2928972354 learned_addr learned my addr 172.21.15.98:0/2928972354 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:31.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.076+0000 7f1caaffd700 1 --2- 172.21.15.98:0/2928972354 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cac15ca70 0x7f1cac0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:31.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.076+0000 7f1caa7fc700 1 --2- 172.21.15.98:0/2928972354 >> v2:172.21.15.100:3300/0 conn(0x7f1cac06b4b0 0x7f1cac0cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:31.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.076+0000 7f1cab7fe700 1 -- 172.21.15.98:0/2928972354 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cac15ca70 msgr2=0x7f1cac0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:31.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.076+0000 7f1cab7fe700 1 --2- 172.21.15.98:0/2928972354 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cac15ca70 0x7f1cac0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.076+0000 7f1cab7fe700 1 -- 172.21.15.98:0/2928972354 >> v2:172.21.15.100:3300/0 conn(0x7f1cac06b4b0 msgr2=0x7f1cac0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:31.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.076+0000 7f1cab7fe700 1 --2- 172.21.15.98:0/2928972354 >> v2:172.21.15.100:3300/0 conn(0x7f1cac06b4b0 0x7f1cac0cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.076+0000 7f1cab7fe700 1 -- 172.21.15.98:0/2928972354 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac0d1800 con 0x7f1cac0d00a0 2023-03-03T07:58:31.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.076+0000 7f1cab7fe700 1 --2- 172.21.15.98:0/2928972354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac0d00a0 0x7f1cac0b4c00 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0b5d40 tx=0x7f1ca400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1225cb24993292 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:31.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.077+0000 7f1ca9ffb700 1 -- 172.21.15.98:0/2928972354 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca400e070 con 0x7f1cac0d00a0 2023-03-03T07:58:31.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.077+0000 7f1ca9ffb700 1 -- 172.21.15.98:0/2928972354 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca400b8d0 con 0x7f1cac0d00a0 2023-03-03T07:58:31.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.077+0000 7f1ca9ffb700 1 -- 172.21.15.98:0/2928972354 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca400bc80 con 0x7f1cac0d00a0 2023-03-03T07:58:31.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.077+0000 7f1cb19ff700 1 -- 172.21.15.98:0/2928972354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac0d00a0 msgr2=0x7f1cac0b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:31.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.077+0000 7f1cb19ff700 1 --2- 172.21.15.98:0/2928972354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac0d00a0 0x7f1cac0b4c00 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0b5d40 tx=0x7f1ca400b330 comp rx=0 tx=0).stop 2023-03-03T07:58:31.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.077+0000 7f1cb19ff700 1 -- 172.21.15.98:0/2928972354 shutdown_connections 2023-03-03T07:58:31.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.077+0000 7f1cb19ff700 1 --2- 172.21.15.98:0/2928972354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac0d00a0 0x7f1cac0b4c00 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.077+0000 7f1cb19ff700 1 --2- 172.21.15.98:0/2928972354 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cac15ca70 0x7f1cac0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.077+0000 7f1cb19ff700 1 --2- 172.21.15.98:0/2928972354 >> v2:172.21.15.100:3300/0 conn(0x7f1cac06b4b0 0x7f1cac0cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.077+0000 7f1cb19ff700 1 -- 172.21.15.98:0/2928972354 >> 172.21.15.98:0/2928972354 conn(0x7f1cac073e70 msgr2=0x7f1cac06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:31.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.077+0000 7f1cb19ff700 1 -- 172.21.15.98:0/2928972354 shutdown_connections 2023-03-03T07:58:31.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.078+0000 7f1cb19ff700 1 -- 172.21.15.98:0/2928972354 wait complete. 2023-03-03T07:58:31.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.080+0000 7f1cb19ff700 1 Processor -- start 2023-03-03T07:58:31.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.080+0000 7f1cb19ff700 1 -- start start 2023-03-03T07:58:31.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.081+0000 7f1cb19ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cac06b4b0 0x7f1cac155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:31.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.081+0000 7f1cb19ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac0d00a0 0x7f1cac0b6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:31.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.081+0000 7f1cb19ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cac15ca70 0x7f1cac156f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:31.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.081+0000 7f1cb19ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0b2f20 con 0x7f1cac0d00a0 2023-03-03T07:58:31.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.081+0000 7f1cb19ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0b74c0 con 0x7f1cac06b4b0 2023-03-03T07:58:31.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.081+0000 7f1cb19ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0b77a0 con 0x7f1cac15ca70 2023-03-03T07:58:31.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.081+0000 7f1caa7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac0d00a0 0x7f1cac0b6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:31.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.081+0000 7f1caa7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac0d00a0 0x7f1cac0b6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51432/0 (socket says 172.21.15.98:51432) 2023-03-03T07:58:31.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.081+0000 7f1caa7fc700 1 -- 172.21.15.98:0/4102628120 learned_addr learned my addr 172.21.15.98:0/4102628120 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.082+0000 7f1caaffd700 1 --2- 172.21.15.98:0/4102628120 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cac06b4b0 0x7f1cac155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.082+0000 7f1cab7fe700 1 --2- 172.21.15.98:0/4102628120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cac15ca70 0x7f1cac156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.082+0000 7f1caa7fc700 1 -- 172.21.15.98:0/4102628120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cac15ca70 msgr2=0x7f1cac156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.082+0000 7f1caa7fc700 1 --2- 172.21.15.98:0/4102628120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cac15ca70 0x7f1cac156f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.082+0000 7f1caa7fc700 1 -- 172.21.15.98:0/4102628120 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cac06b4b0 msgr2=0x7f1cac155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.082+0000 7f1caa7fc700 1 --2- 172.21.15.98:0/4102628120 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cac06b4b0 0x7f1cac155740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.082+0000 7f1caa7fc700 1 -- 172.21.15.98:0/4102628120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca4009b30 con 0x7f1cac0d00a0 2023-03-03T07:58:31.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.082+0000 7f1caaffd700 1 --2- 172.21.15.98:0/4102628120 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cac06b4b0 0x7f1cac155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:31.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.082+0000 7f1caa7fc700 1 --2- 172.21.15.98:0/4102628120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac0d00a0 0x7f1cac0b6ae0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000e980 tx=0x7f1ca000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:31.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.082+0000 7f1c93fff700 1 -- 172.21.15.98:0/4102628120 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0009710 con 0x7f1cac0d00a0 2023-03-03T07:58:31.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.083+0000 7f1c93fff700 1 -- 172.21.15.98:0/4102628120 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca000c850 con 0x7f1cac0d00a0 2023-03-03T07:58:31.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.083+0000 7f1c93fff700 1 -- 172.21.15.98:0/4102628120 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca000cbe0 con 0x7f1cac0d00a0 2023-03-03T07:58:31.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.083+0000 7f1cb19ff700 1 -- 172.21.15.98:0/4102628120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cac0cdbe0 con 0x7f1cac0d00a0 2023-03-03T07:58:31.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.083+0000 7f1cb19ff700 1 -- 172.21.15.98:0/4102628120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cac0cd900 con 0x7f1cac0d00a0 2023-03-03T07:58:31.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.084+0000 7f1cb19ff700 1 -- 172.21.15.98:0/4102628120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cac0d1aa0 con 0x7f1cac0d00a0 2023-03-03T07:58:31.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.085+0000 7f1c93fff700 1 -- 172.21.15.98:0/4102628120 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f1ca0019030 con 0x7f1cac0d00a0 2023-03-03T07:58:31.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.087+0000 7f1c93fff700 1 --2- 172.21.15.98:0/4102628120 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1c980683c0 0x7f1c9806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:31.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.087+0000 7f1c93fff700 1 -- 172.21.15.98:0/4102628120 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f1ca0085dd0 con 0x7f1cac0d00a0 2023-03-03T07:58:31.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.087+0000 7f1caaffd700 1 --2- 172.21.15.98:0/4102628120 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1c980683c0 0x7f1c9806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:31.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.088+0000 7f1caaffd700 1 --2- 172.21.15.98:0/4102628120 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1c980683c0 0x7f1c9806a870 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1cac157f00 tx=0x7f1c9c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:31.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.090+0000 7f1c93fff700 1 -- 172.21.15.98:0/4102628120 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1ca00525e0 con 0x7f1cac0d00a0 2023-03-03T07:58:31.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.286+0000 7f1cb19ff700 1 -- 172.21.15.98:0/4102628120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1cac04e6e0 con 0x7f1cac0d00a0 2023-03-03T07:58:31.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.286+0000 7f1c93fff700 1 -- 172.21.15.98:0/4102628120 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1ca00170a0 con 0x7f1cac0d00a0 2023-03-03T07:58:31.288 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:58:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.289+0000 7f1cb19ff700 1 -- 172.21.15.98:0/4102628120 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1c980683c0 msgr2=0x7f1c9806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.289+0000 7f1cb19ff700 1 --2- 172.21.15.98:0/4102628120 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1c980683c0 0x7f1c9806a870 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1cac157f00 tx=0x7f1c9c008040 comp rx=0 tx=0).stop 2023-03-03T07:58:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.289+0000 7f1cb19ff700 1 -- 172.21.15.98:0/4102628120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac0d00a0 msgr2=0x7f1cac0b6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.289+0000 7f1cb19ff700 1 --2- 172.21.15.98:0/4102628120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac0d00a0 0x7f1cac0b6ae0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000e980 tx=0x7f1ca000ed40 comp rx=0 tx=0).stop 2023-03-03T07:58:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.289+0000 7f1cb19ff700 1 -- 172.21.15.98:0/4102628120 shutdown_connections 2023-03-03T07:58:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.289+0000 7f1cb19ff700 1 --2- 172.21.15.98:0/4102628120 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1c980683c0 0x7f1c9806a870 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.289+0000 7f1cb19ff700 1 --2- 172.21.15.98:0/4102628120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cac15ca70 0x7f1cac156f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.289+0000 7f1cb19ff700 1 --2- 172.21.15.98:0/4102628120 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cac06b4b0 0x7f1cac155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.289+0000 7f1cb19ff700 1 --2- 172.21.15.98:0/4102628120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac0d00a0 0x7f1cac0b6ae0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.289+0000 7f1cb19ff700 1 -- 172.21.15.98:0/4102628120 >> 172.21.15.98:0/4102628120 conn(0x7f1cac073e70 msgr2=0x7f1cac0b7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.289+0000 7f1cb19ff700 1 -- 172.21.15.98:0/4102628120 shutdown_connections 2023-03-03T07:58:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.289+0000 7f1cb19ff700 1 -- 172.21.15.98:0/4102628120 wait complete. 2023-03-03T07:58:31.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T07:58:31.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.566+0000 7fd33099d700 1 Processor -- start 2023-03-03T07:58:31.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.567+0000 7fd33099d700 1 -- start start 2023-03-03T07:58:31.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.568+0000 7fd33099d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd32c0cf570 0x7fd32c15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:31.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.568+0000 7fd33099d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd32c15db30 0x7fd32c0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:31.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.568+0000 7fd33099d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c0cd780 con 0x7fd32c1609b0 2023-03-03T07:58:31.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.568+0000 7fd33099d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd32c0cebb0 con 0x7fd32c0cf570 2023-03-03T07:58:31.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.568+0000 7fd33099d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd32c0cda60 con 0x7fd32c15db30 2023-03-03T07:58:31.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.568+0000 7fd32affd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd32c0cf570 0x7fd32c15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:31.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.568+0000 7fd32affd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd32c0cf570 0x7fd32c15d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51444/0 (socket says 172.21.15.98:51444) 2023-03-03T07:58:31.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.568+0000 7fd32affd700 1 -- 172.21.15.98:0/161484908 learned_addr learned my addr 172.21.15.98:0/161484908 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:31.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.569+0000 7fd32bfff700 1 --2- 172.21.15.98:0/161484908 >> v2:172.21.15.100:3300/0 conn(0x7fd32c15db30 0x7fd32c0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:31.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.569+0000 7fd32affd700 1 -- 172.21.15.98:0/161484908 >> v2:172.21.15.100:3300/0 conn(0x7fd32c15db30 msgr2=0x7fd32c0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:31.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.569+0000 7fd32affd700 1 --2- 172.21.15.98:0/161484908 >> v2:172.21.15.100:3300/0 conn(0x7fd32c15db30 0x7fd32c0b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.569+0000 7fd32affd700 1 -- 172.21.15.98:0/161484908 >> v1:172.21.15.100:6789/0 conn(0x7fd32c1609b0 legacy=0x7fd32c160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:31.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.569+0000 7fd32affd700 1 -- 172.21.15.98:0/161484908 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c0cdd40 con 0x7fd32c0cf570 2023-03-03T07:58:31.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.569+0000 7fd32bfff700 1 --2- 172.21.15.98:0/161484908 >> v2:172.21.15.100:3300/0 conn(0x7fd32c15db30 0x7fd32c0b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:31.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.569+0000 7fd32affd700 1 --2- 172.21.15.98:0/161484908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd32c0cf570 0x7fd32c15d5f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0ca080 tx=0x7fd320009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c062f7d1416eeb3e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:31.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.570+0000 7fd32a7fc700 1 -- 172.21.15.98:0/161484908 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd320004230 con 0x7fd32c0cf570 2023-03-03T07:58:31.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.570+0000 7fd32a7fc700 1 -- 172.21.15.98:0/161484908 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd320004390 con 0x7fd32c0cf570 2023-03-03T07:58:31.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.570+0000 7fd32a7fc700 1 -- 172.21.15.98:0/161484908 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3200036a0 con 0x7fd32c0cf570 2023-03-03T07:58:31.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.570+0000 7fd33099d700 1 -- 172.21.15.98:0/161484908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd32c0cf570 msgr2=0x7fd32c15d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:31.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.570+0000 7fd33099d700 1 --2- 172.21.15.98:0/161484908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd32c0cf570 0x7fd32c15d5f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0ca080 tx=0x7fd320009bd0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.570+0000 7fd33099d700 1 -- 172.21.15.98:0/161484908 shutdown_connections 2023-03-03T07:58:31.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.570+0000 7fd33099d700 1 --2- 172.21.15.98:0/161484908 >> v2:172.21.15.100:3300/0 conn(0x7fd32c15db30 0x7fd32c0b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.570+0000 7fd33099d700 1 --2- 172.21.15.98:0/161484908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd32c0cf570 0x7fd32c15d5f0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.570+0000 7fd33099d700 1 -- 172.21.15.98:0/161484908 >> 172.21.15.98:0/161484908 conn(0x7fd32c06b880 msgr2=0x7fd32c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:31.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.570+0000 7fd33099d700 1 -- 172.21.15.98:0/161484908 shutdown_connections 2023-03-03T07:58:31.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.570+0000 7fd33099d700 1 -- 172.21.15.98:0/161484908 wait complete. 2023-03-03T07:58:31.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.572+0000 7fd33099d700 1 Processor -- start 2023-03-03T07:58:31.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.573+0000 7fd33099d700 1 -- start start 2023-03-03T07:58:31.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.574+0000 7fd33099d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd32c0cf570 0x7fd32c152b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:31.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.574+0000 7fd33099d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd32c15db30 0x7fd32c0cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:31.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.574+0000 7fd33099d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd32c1609b0 0x7fd32c0cb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:31.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.574+0000 7fd33099d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c0b34e0 con 0x7fd32c15db30 2023-03-03T07:58:31.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.574+0000 7fd33099d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c0cbe80 con 0x7fd32c1609b0 2023-03-03T07:58:31.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.574+0000 7fd33099d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c0b4c70 con 0x7fd32c0cf570 2023-03-03T07:58:31.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.574+0000 7fd32affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd32c15db30 0x7fd32c0cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:31.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.574+0000 7fd32affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd32c15db30 0x7fd32c0cb310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51454/0 (socket says 172.21.15.98:51454) 2023-03-03T07:58:31.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.574+0000 7fd32affd700 1 -- 172.21.15.98:0/3066101257 learned_addr learned my addr 172.21.15.98:0/3066101257 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:31.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.574+0000 7fd32b7fe700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd32c0cf570 0x7fd32c152b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:31.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.574+0000 7fd32bfff700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd32c1609b0 0x7fd32c0cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:31.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.574+0000 7fd32affd700 1 -- 172.21.15.98:0/3066101257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd32c0cf570 msgr2=0x7fd32c152b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:31.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.575+0000 7fd32affd700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd32c0cf570 0x7fd32c152b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.575+0000 7fd32affd700 1 -- 172.21.15.98:0/3066101257 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd32c1609b0 msgr2=0x7fd32c0cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:31.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.575+0000 7fd32affd700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd32c1609b0 0x7fd32c0cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.575+0000 7fd32affd700 1 -- 172.21.15.98:0/3066101257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd320009550 con 0x7fd32c15db30 2023-03-03T07:58:31.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.575+0000 7fd32b7fe700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd32c0cf570 0x7fd32c152b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:31.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.575+0000 7fd32bfff700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd32c1609b0 0x7fd32c0cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:31.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.575+0000 7fd32affd700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd32c15db30 0x7fd32c0cb310 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fd320009130 tx=0x7fd320004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:31.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.575+0000 7fd313fff700 1 -- 172.21.15.98:0/3066101257 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd320003dc0 con 0x7fd32c15db30 2023-03-03T07:58:31.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.575+0000 7fd313fff700 1 -- 172.21.15.98:0/3066101257 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32001f070 con 0x7fd32c15db30 2023-03-03T07:58:31.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.575+0000 7fd313fff700 1 -- 172.21.15.98:0/3066101257 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd320011850 con 0x7fd32c15db30 2023-03-03T07:58:31.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.575+0000 7fd33099d700 1 -- 172.21.15.98:0/3066101257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32c15d0d0 con 0x7fd32c15db30 2023-03-03T07:58:31.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.575+0000 7fd33099d700 1 -- 172.21.15.98:0/3066101257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32c15cdf0 con 0x7fd32c15db30 2023-03-03T07:58:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.576+0000 7fd33099d700 1 -- 172.21.15.98:0/3066101257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32c0b4fe0 con 0x7fd32c15db30 2023-03-03T07:58:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.578+0000 7fd313fff700 1 -- 172.21.15.98:0/3066101257 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fd320017020 con 0x7fd32c15db30 2023-03-03T07:58:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.579+0000 7fd313fff700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3180683c0 0x7fd31806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.579+0000 7fd313fff700 1 -- 172.21.15.98:0/3066101257 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fd3200877b0 con 0x7fd32c15db30 2023-03-03T07:58:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.579+0000 7fd32b7fe700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3180683c0 0x7fd31806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.580+0000 7fd32b7fe700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3180683c0 0x7fd31806a870 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0b44b0 tx=0x7fd31c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:31.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.584+0000 7fd313fff700 1 -- 172.21.15.98:0/3066101257 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd320053e10 con 0x7fd32c15db30 2023-03-03T07:58:31.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.797+0000 7fd33099d700 1 -- 172.21.15.98:0/3066101257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd32c068290 con 0x7fd32c15db30 2023-03-03T07:58:31.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.798+0000 7fd313fff700 1 -- 172.21.15.98:0/3066101257 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v54) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd320057430 con 0x7fd32c15db30 2023-03-03T07:58:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.801+0000 7fd33099d700 1 -- 172.21.15.98:0/3066101257 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3180683c0 msgr2=0x7fd31806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.801+0000 7fd33099d700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3180683c0 0x7fd31806a870 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0b44b0 tx=0x7fd31c008040 comp rx=0 tx=0).stop 2023-03-03T07:58:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.801+0000 7fd33099d700 1 -- 172.21.15.98:0/3066101257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd32c15db30 msgr2=0x7fd32c0cb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.801+0000 7fd33099d700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd32c15db30 0x7fd32c0cb310 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fd320009130 tx=0x7fd320004750 comp rx=0 tx=0).stop 2023-03-03T07:58:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.801+0000 7fd33099d700 1 -- 172.21.15.98:0/3066101257 shutdown_connections 2023-03-03T07:58:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.801+0000 7fd33099d700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3180683c0 0x7fd31806a870 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.801+0000 7fd33099d700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd32c1609b0 0x7fd32c0cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.801+0000 7fd33099d700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd32c0cf570 0x7fd32c152b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.801+0000 7fd33099d700 1 --2- 172.21.15.98:0/3066101257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd32c15db30 0x7fd32c0cb310 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.801+0000 7fd33099d700 1 -- 172.21.15.98:0/3066101257 >> 172.21.15.98:0/3066101257 conn(0x7fd32c06b880 msgr2=0x7fd32c0d0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.801+0000 7fd33099d700 1 -- 172.21.15.98:0/3066101257 shutdown_connections 2023-03-03T07:58:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:31.801+0000 7fd33099d700 1 -- 172.21.15.98:0/3066101257 wait complete. 2023-03-03T07:58:31.805 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T07:58:31.829 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T07:58:31.830 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:58:31.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:58:32.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.143+0000 7f68ee7ff700 1 Processor -- start 2023-03-03T07:58:32.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.144+0000 7f68ee7ff700 1 -- start start 2023-03-03T07:58:32.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.145+0000 7f68ee7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e80b8b90 0x7f68e80cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:32.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.145+0000 7f68ee7ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f68e80b5410 0x7f68e80d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:32.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.145+0000 7f68ee7ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e80d0ef0 con 0x7f68e80b87c0 2023-03-03T07:58:32.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.145+0000 7f68ee7ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e80d0600 con 0x7f68e80b5410 2023-03-03T07:58:32.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.145+0000 7f68ee7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e80cd970 con 0x7f68e80b8b90 2023-03-03T07:58:32.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.145+0000 7f68ecd9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f68e80b5410 0x7f68e80d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:32.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.145+0000 7f68ecd9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f68e80b5410 0x7f68e80d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51470/0 (socket says 172.21.15.98:51470) 2023-03-03T07:58:32.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.145+0000 7f68ecd9c700 1 -- 172.21.15.98:0/4214547637 learned_addr learned my addr 172.21.15.98:0/4214547637 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:32.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.146+0000 7f68e77fe700 1 --2- 172.21.15.98:0/4214547637 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e80b8b90 0x7f68e80cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:32.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.146+0000 7f68ecd9c700 1 -- 172.21.15.98:0/4214547637 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e80b8b90 msgr2=0x7f68e80cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:32.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.146+0000 7f68ecd9c700 1 --2- 172.21.15.98:0/4214547637 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e80b8b90 0x7f68e80cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.146+0000 7f68ecd9c700 1 -- 172.21.15.98:0/4214547637 >> v1:172.21.15.100:6789/0 conn(0x7f68e80b87c0 legacy=0x7f68e80b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:32.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.146+0000 7f68ecd9c700 1 -- 172.21.15.98:0/4214547637 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e80cdc50 con 0x7f68e80b5410 2023-03-03T07:58:32.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.146+0000 7f68e77fe700 1 --2- 172.21.15.98:0/4214547637 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e80b8b90 0x7f68e80cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:32.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.146+0000 7f68ecd9c700 1 --2- 172.21.15.98:0/4214547637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e80b5410 0x7f68e80d13d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009e50 tx=0x7f68e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a660a6b98d5066c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:32.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.146+0000 7f68e6ffd700 1 -- 172.21.15.98:0/4214547637 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e000e070 con 0x7f68e80b5410 2023-03-03T07:58:32.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.147+0000 7f68e6ffd700 1 -- 172.21.15.98:0/4214547637 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e000b9c0 con 0x7f68e80b5410 2023-03-03T07:58:32.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.147+0000 7f68e6ffd700 1 -- 172.21.15.98:0/4214547637 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e000bd70 con 0x7f68e80b5410 2023-03-03T07:58:32.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.147+0000 7f68ee7ff700 1 -- 172.21.15.98:0/4214547637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e80b5410 msgr2=0x7f68e80d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:32.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.147+0000 7f68ee7ff700 1 --2- 172.21.15.98:0/4214547637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e80b5410 0x7f68e80d13d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009e50 tx=0x7f68e000b330 comp rx=0 tx=0).stop 2023-03-03T07:58:32.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.147+0000 7f68ee7ff700 1 -- 172.21.15.98:0/4214547637 shutdown_connections 2023-03-03T07:58:32.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.147+0000 7f68ee7ff700 1 --2- 172.21.15.98:0/4214547637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e80b5410 0x7f68e80d13d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.147+0000 7f68ee7ff700 1 --2- 172.21.15.98:0/4214547637 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e80b8b90 0x7f68e80cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.147+0000 7f68ee7ff700 1 -- 172.21.15.98:0/4214547637 >> 172.21.15.98:0/4214547637 conn(0x7f68e806aee0 msgr2=0x7f68e806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:32.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.147+0000 7f68ee7ff700 1 -- 172.21.15.98:0/4214547637 shutdown_connections 2023-03-03T07:58:32.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.147+0000 7f68ee7ff700 1 -- 172.21.15.98:0/4214547637 wait complete. 2023-03-03T07:58:32.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.150+0000 7f68ee7ff700 1 Processor -- start 2023-03-03T07:58:32.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.150+0000 7f68ee7ff700 1 -- start start 2023-03-03T07:58:32.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.152+0000 7f68ee7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e80b5410 0x7f68e815c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:32.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.152+0000 7f68ee7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68e80b87c0 0x7f68e80ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:32.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.152+0000 7f68ee7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e80b8b90 0x7f68e80c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:32.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.152+0000 7f68ee7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e80cb830 con 0x7f68e80b5410 2023-03-03T07:58:32.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.152+0000 7f68ee7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e80cacc0 con 0x7f68e80b87c0 2023-03-03T07:58:32.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.152+0000 7f68ee7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e80cafa0 con 0x7f68e80b8b90 2023-03-03T07:58:32.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.152+0000 7f68e7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e80b5410 0x7f68e815c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:32.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.152+0000 7f68e7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e80b5410 0x7f68e815c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51474/0 (socket says 172.21.15.98:51474) 2023-03-03T07:58:32.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.152+0000 7f68e7fff700 1 -- 172.21.15.98:0/3972935800 learned_addr learned my addr 172.21.15.98:0/3972935800 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:32.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.153+0000 7f68e7fff700 1 -- 172.21.15.98:0/3972935800 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e80b8b90 msgr2=0x7f68e80c83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:32.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.153+0000 7f68e7fff700 1 --2- 172.21.15.98:0/3972935800 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e80b8b90 0x7f68e80c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.153+0000 7f68e77fe700 1 --2- 172.21.15.98:0/3972935800 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68e80b87c0 0x7f68e80ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:32.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.153+0000 7f68e7fff700 1 -- 172.21.15.98:0/3972935800 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68e80b87c0 msgr2=0x7f68e80ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:32.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.153+0000 7f68e7fff700 1 --2- 172.21.15.98:0/3972935800 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68e80b87c0 0x7f68e80ca350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.153+0000 7f68e7fff700 1 -- 172.21.15.98:0/3972935800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e0009b30 con 0x7f68e80b5410 2023-03-03T07:58:32.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.153+0000 7f68e7fff700 1 --2- 172.21.15.98:0/3972935800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e80b5410 0x7f68e815c9b0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f68d800c4e0 tx=0x7f68d800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:32.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.153+0000 7f68e4ff9700 1 -- 172.21.15.98:0/3972935800 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d80052d0 con 0x7f68e80b5410 2023-03-03T07:58:32.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.153+0000 7f68e4ff9700 1 -- 172.21.15.98:0/3972935800 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d8005430 con 0x7f68e80b5410 2023-03-03T07:58:32.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.153+0000 7f68e4ff9700 1 -- 172.21.15.98:0/3972935800 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d80038c0 con 0x7f68e80b5410 2023-03-03T07:58:32.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.153+0000 7f68ee7ff700 1 -- 172.21.15.98:0/3972935800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e80cdf30 con 0x7f68e80b5410 2023-03-03T07:58:32.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.153+0000 7f68ee7ff700 1 -- 172.21.15.98:0/3972935800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e8073a80 con 0x7f68e80b5410 2023-03-03T07:58:32.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.154+0000 7f68ee7ff700 1 -- 172.21.15.98:0/3972935800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e80ce0c0 con 0x7f68e80b5410 2023-03-03T07:58:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.156+0000 7f68e4ff9700 1 -- 172.21.15.98:0/3972935800 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f68d8019030 con 0x7f68e80b5410 2023-03-03T07:58:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.157+0000 7f68e4ff9700 1 --2- 172.21.15.98:0/3972935800 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f68d40683c0 0x7f68d406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.157+0000 7f68e4ff9700 1 -- 172.21.15.98:0/3972935800 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f68d8085270 con 0x7f68e80b5410 2023-03-03T07:58:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.158+0000 7f68e77fe700 1 --2- 172.21.15.98:0/3972935800 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f68d40683c0 0x7f68d406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.158+0000 7f68e77fe700 1 --2- 172.21.15.98:0/3972935800 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f68d40683c0 0x7f68d406a870 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f68e815cef0 tx=0x7f68dc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:32.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.161+0000 7f68e4ff9700 1 -- 172.21.15.98:0/3972935800 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f68d80518d0 con 0x7f68e80b5410 2023-03-03T07:58:32.175 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:58:32.181 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:58:32.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.367+0000 7f68ee7ff700 1 -- 172.21.15.98:0/3972935800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68e80729b0 con 0x7f68e80b5410 2023-03-03T07:58:32.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.369+0000 7f68e4ff9700 1 -- 172.21.15.98:0/3972935800 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+18018 (secure 0 0 0) 0x7f68d8054ef0 con 0x7f68e80b5410 2023-03-03T07:58:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:32.371 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":54,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:58:28.723281+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:58:10.728830+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-03-03T07:58:12.224649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.6","mappings":[{"from":0,"to":1}]},{"pgid":"8.17","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:58:32.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.371+0000 7f68ee7ff700 1 -- 172.21.15.98:0/3972935800 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f68d40683c0 msgr2=0x7f68d406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:32.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.371+0000 7f68ee7ff700 1 --2- 172.21.15.98:0/3972935800 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f68d40683c0 0x7f68d406a870 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f68e815cef0 tx=0x7f68dc008040 comp rx=0 tx=0).stop 2023-03-03T07:58:32.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.371+0000 7f68ee7ff700 1 -- 172.21.15.98:0/3972935800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e80b5410 msgr2=0x7f68e815c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:32.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.372+0000 7f68ee7ff700 1 --2- 172.21.15.98:0/3972935800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e80b5410 0x7f68e815c9b0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f68d800c4e0 tx=0x7f68d800c8a0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.372+0000 7f68ee7ff700 1 -- 172.21.15.98:0/3972935800 shutdown_connections 2023-03-03T07:58:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.372+0000 7f68ee7ff700 1 --2- 172.21.15.98:0/3972935800 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f68d40683c0 0x7f68d406a870 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.372+0000 7f68ee7ff700 1 --2- 172.21.15.98:0/3972935800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e80b5410 0x7f68e815c9b0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.372+0000 7f68ee7ff700 1 --2- 172.21.15.98:0/3972935800 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e80b8b90 0x7f68e80c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.372+0000 7f68ee7ff700 1 --2- 172.21.15.98:0/3972935800 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68e80b87c0 0x7f68e80ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.372+0000 7f68ee7ff700 1 -- 172.21.15.98:0/3972935800 >> 172.21.15.98:0/3972935800 conn(0x7f68e806aee0 msgr2=0x7f68e815f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.372+0000 7f68ee7ff700 1 -- 172.21.15.98:0/3972935800 shutdown_connections 2023-03-03T07:58:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.372+0000 7f68ee7ff700 1 -- 172.21.15.98:0/3972935800 wait complete. 2023-03-03T07:58:32.400 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:58:10.728830+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:58:12.224649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:58:32.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:58:32.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.651+0000 7f02109ff700 1 Processor -- start 2023-03-03T07:58:32.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.651+0000 7f02109ff700 1 -- start start 2023-03-03T07:58:32.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.652+0000 7f02109ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f020c15dd20 0x7f020c0b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:32.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.652+0000 7f02109ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020c15e170 con 0x7f020c0b0800 2023-03-03T07:58:32.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.652+0000 7f02109ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020c0b1110 con 0x7f020c0cd5c0 2023-03-03T07:58:32.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.652+0000 7f02109ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f020c0c9d50 con 0x7f020c15dd20 2023-03-03T07:58:32.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.653+0000 7f020a59c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f020c0b0800 0x7f020c0c93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56412/0 (socket says 172.21.15.98:56412) 2023-03-03T07:58:32.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.653+0000 7f020a59c700 1 -- 172.21.15.98:0/3520832708 learned_addr learned my addr 172.21.15.98:0/3520832708 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:32.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.653+0000 7f0209d9b700 1 --2- 172.21.15.98:0/3520832708 >> v2:172.21.15.100:3300/0 conn(0x7f020c15dd20 0x7f020c0b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:32.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.653+0000 7f020959a700 1 -- 172.21.15.98:0/3520832708 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615898430 0 0) 0x7f020c15e170 con 0x7f020c0b0800 2023-03-03T07:58:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.653+0000 7f020959a700 1 -- 172.21.15.98:0/3520832708 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f40035b0 con 0x7f020c0b0800 2023-03-03T07:58:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.653+0000 7f0209d9b700 1 -- 172.21.15.98:0/3520832708 >> v1:172.21.15.100:6789/0 conn(0x7f020c0cd5c0 legacy=0x7f020c1599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.653+0000 7f020959a700 1 -- 172.21.15.98:0/3520832708 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234448065 0 0) 0x7f020c0b1110 con 0x7f020c0cd5c0 2023-03-03T07:58:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.653+0000 7f0209d9b700 1 -- 172.21.15.98:0/3520832708 >> v1:172.21.15.98:6789/0 conn(0x7f020c0b0800 legacy=0x7f020c0c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.653+0000 7f0209d9b700 1 -- 172.21.15.98:0/3520832708 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020c0ca030 con 0x7f020c15dd20 2023-03-03T07:58:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.654+0000 7f0209d9b700 1 --2- 172.21.15.98:0/3520832708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f020c15dd20 0x7f020c0b0bd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0200009840 tx=0x7f0200009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8f9e4db585c4b79 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.654+0000 7f020959a700 1 -- 172.21.15.98:0/3520832708 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0200011830 con 0x7f020c15dd20 2023-03-03T07:58:32.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.654+0000 7f020959a700 1 -- 172.21.15.98:0/3520832708 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0200011990 con 0x7f020c15dd20 2023-03-03T07:58:32.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.654+0000 7f020959a700 1 -- 172.21.15.98:0/3520832708 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0200011d20 con 0x7f020c15dd20 2023-03-03T07:58:32.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.654+0000 7f02109ff700 1 -- 172.21.15.98:0/3520832708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f020c15dd20 msgr2=0x7f020c0b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.654+0000 7f02109ff700 1 --2- 172.21.15.98:0/3520832708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f020c15dd20 0x7f020c0b0bd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0200009840 tx=0x7f0200009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.655+0000 7f02109ff700 1 -- 172.21.15.98:0/3520832708 shutdown_connections 2023-03-03T07:58:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.655+0000 7f02109ff700 1 --2- 172.21.15.98:0/3520832708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f020c15dd20 0x7f020c0b0bd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.655+0000 7f02109ff700 1 -- 172.21.15.98:0/3520832708 >> 172.21.15.98:0/3520832708 conn(0x7f020c0cb800 msgr2=0x7f020c0cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.655+0000 7f02109ff700 1 -- 172.21.15.98:0/3520832708 shutdown_connections 2023-03-03T07:58:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.655+0000 7f02109ff700 1 -- 172.21.15.98:0/3520832708 wait complete. 2023-03-03T07:58:32.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.657+0000 7f02109ff700 1 Processor -- start 2023-03-03T07:58:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.657+0000 7f02109ff700 1 -- start start 2023-03-03T07:58:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.658+0000 7f02109ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f020c0b0800 0x7f020c0c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.658+0000 7f02109ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f020c0cd5c0 0x7f020c14d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.658+0000 7f02109ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f020c15dd20 0x7f020c152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.658+0000 7f02109ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c1f66f0 con 0x7f020c0cd5c0 2023-03-03T07:58:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.658+0000 7f02109ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c1f69d0 con 0x7f020c15dd20 2023-03-03T07:58:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.658+0000 7f02109ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020c1f6cb0 con 0x7f020c0b0800 2023-03-03T07:58:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.658+0000 7f0209d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f020c0cd5c0 0x7f020c14d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.658+0000 7f0209d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f020c0cd5c0 0x7f020c14d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51480/0 (socket says 172.21.15.98:51480) 2023-03-03T07:58:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.658+0000 7f0209d9b700 1 -- 172.21.15.98:0/1504806344 learned_addr learned my addr 172.21.15.98:0/1504806344 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.659+0000 7f020ad9d700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f020c15dd20 0x7f020c152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.659+0000 7f020a59c700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f020c0b0800 0x7f020c0c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.659+0000 7f0209d9b700 1 -- 172.21.15.98:0/1504806344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f020c0b0800 msgr2=0x7f020c0c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.659+0000 7f0209d9b700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f020c0b0800 0x7f020c0c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.659+0000 7f0209d9b700 1 -- 172.21.15.98:0/1504806344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f020c15dd20 msgr2=0x7f020c152b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.659+0000 7f0209d9b700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f020c15dd20 0x7f020c152b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.659+0000 7f0209d9b700 1 -- 172.21.15.98:0/1504806344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02000094a0 con 0x7f020c0cd5c0 2023-03-03T07:58:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.659+0000 7f020ad9d700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f020c15dd20 0x7f020c152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.659+0000 7f020a59c700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f020c0b0800 0x7f020c0c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.659+0000 7f0209d9b700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f020c0cd5c0 0x7f020c14d020 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0200014040 tx=0x7f0200004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.659+0000 7f01faffd700 1 -- 172.21.15.98:0/1504806344 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020001a910 con 0x7f020c0cd5c0 2023-03-03T07:58:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.659+0000 7f01faffd700 1 -- 172.21.15.98:0/1504806344 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020001aa70 con 0x7f020c0cd5c0 2023-03-03T07:58:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.659+0000 7f02109ff700 1 -- 172.21.15.98:0/1504806344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f020c0cd0a0 con 0x7f020c0cd5c0 2023-03-03T07:58:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.659+0000 7f01faffd700 1 -- 172.21.15.98:0/1504806344 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020001ade0 con 0x7f020c0cd5c0 2023-03-03T07:58:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.660+0000 7f02109ff700 1 -- 172.21.15.98:0/1504806344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f020c0ccdc0 con 0x7f020c0cd5c0 2023-03-03T07:58:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.661+0000 7f02109ff700 1 -- 172.21.15.98:0/1504806344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f020c0ca310 con 0x7f020c0cd5c0 2023-03-03T07:58:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.662+0000 7f01faffd700 1 -- 172.21.15.98:0/1504806344 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f020002b030 con 0x7f020c0cd5c0 2023-03-03T07:58:32.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.664+0000 7f01faffd700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f01ec0683c0 0x7f01ec06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:32.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.664+0000 7f01faffd700 1 -- 172.21.15.98:0/1504806344 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f0200087200 con 0x7f020c0cd5c0 2023-03-03T07:58:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.664+0000 7f020a59c700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f01ec0683c0 0x7f01ec06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.665+0000 7f020a59c700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f01ec0683c0 0x7f01ec06a870 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f01fc005910 tx=0x7f01fc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.669+0000 7f01faffd700 1 -- 172.21.15.98:0/1504806344 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0200053860 con 0x7f020c0cd5c0 2023-03-03T07:58:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.864+0000 7f02109ff700 1 -- 172.21.15.98:0/1504806344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f020c157020 con 0x7f020c0cd5c0 2023-03-03T07:58:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.865+0000 7f01faffd700 1 -- 172.21.15.98:0/1504806344 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0200029020 con 0x7f020c0cd5c0 2023-03-03T07:58:32.867 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:58:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.867+0000 7f02109ff700 1 -- 172.21.15.98:0/1504806344 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f01ec0683c0 msgr2=0x7f01ec06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.867+0000 7f02109ff700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f01ec0683c0 0x7f01ec06a870 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f01fc005910 tx=0x7f01fc009040 comp rx=0 tx=0).stop 2023-03-03T07:58:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.867+0000 7f02109ff700 1 -- 172.21.15.98:0/1504806344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f020c0cd5c0 msgr2=0x7f020c14d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.867+0000 7f02109ff700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f020c0cd5c0 0x7f020c14d020 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0200014040 tx=0x7f0200004750 comp rx=0 tx=0).stop 2023-03-03T07:58:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.867+0000 7f02109ff700 1 -- 172.21.15.98:0/1504806344 shutdown_connections 2023-03-03T07:58:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.867+0000 7f02109ff700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f01ec0683c0 0x7f01ec06a870 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.867+0000 7f02109ff700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f020c15dd20 0x7f020c152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.867+0000 7f02109ff700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f020c0b0800 0x7f020c0c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.867+0000 7f02109ff700 1 --2- 172.21.15.98:0/1504806344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f020c0cd5c0 0x7f020c14d020 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.867+0000 7f02109ff700 1 -- 172.21.15.98:0/1504806344 >> 172.21.15.98:0/1504806344 conn(0x7f020c0cb800 msgr2=0x7f020c0af840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.868+0000 7f02109ff700 1 -- 172.21.15.98:0/1504806344 shutdown_connections 2023-03-03T07:58:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:32.868+0000 7f02109ff700 1 -- 172.21.15.98:0/1504806344 wait complete. 2023-03-03T07:58:32.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:58:33.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.143+0000 7f52f95ff700 1 Processor -- start 2023-03-03T07:58:33.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.143+0000 7f52f95ff700 1 -- start start 2023-03-03T07:58:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.145+0000 7f52f95ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f52f415ca70 0x7f52f40cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.145+0000 7f52f95ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f52f406b4b0 0x7f52f40cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.145+0000 7f52f95ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f40cf980 con 0x7f52f40d00a0 2023-03-03T07:58:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.145+0000 7f52f95ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f40cd5b0 con 0x7f52f406b4b0 2023-03-03T07:58:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.145+0000 7f52f95ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f40d0510 con 0x7f52f415ca70 2023-03-03T07:58:33.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.145+0000 7f52f27fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f52f406b4b0 0x7f52f40cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:33.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.145+0000 7f52f27fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f52f406b4b0 0x7f52f40cfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51492/0 (socket says 172.21.15.98:51492) 2023-03-03T07:58:33.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.145+0000 7f52f27fc700 1 -- 172.21.15.98:0/1082707981 learned_addr learned my addr 172.21.15.98:0/1082707981 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:33.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.145+0000 7f52f2ffd700 1 --2- 172.21.15.98:0/1082707981 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f52f415ca70 0x7f52f40cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:33.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.146+0000 7f52f27fc700 1 -- 172.21.15.98:0/1082707981 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f52f415ca70 msgr2=0x7f52f40cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:33.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.146+0000 7f52f27fc700 1 --2- 172.21.15.98:0/1082707981 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f52f415ca70 0x7f52f40cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.146+0000 7f52f27fc700 1 -- 172.21.15.98:0/1082707981 >> v1:172.21.15.100:6789/0 conn(0x7f52f40d00a0 legacy=0x7f52f40b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:33.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.146+0000 7f52f27fc700 1 -- 172.21.15.98:0/1082707981 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f40d2fd0 con 0x7f52f406b4b0 2023-03-03T07:58:33.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.146+0000 7f52f27fc700 1 --2- 172.21.15.98:0/1082707981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52f406b4b0 0x7f52f40cfb60 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f52e80098f0 tx=0x7f52e8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc26d9184468547 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:33.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.146+0000 7f52f1ffb700 1 -- 172.21.15.98:0/1082707981 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e800b8a0 con 0x7f52f406b4b0 2023-03-03T07:58:33.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.146+0000 7f52f1ffb700 1 -- 172.21.15.98:0/1082707981 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e800ba00 con 0x7f52f406b4b0 2023-03-03T07:58:33.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.146+0000 7f52f1ffb700 1 -- 172.21.15.98:0/1082707981 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e800bd90 con 0x7f52f406b4b0 2023-03-03T07:58:33.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.147+0000 7f52f95ff700 1 -- 172.21.15.98:0/1082707981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52f406b4b0 msgr2=0x7f52f40cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:33.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.147+0000 7f52f95ff700 1 --2- 172.21.15.98:0/1082707981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52f406b4b0 0x7f52f40cfb60 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f52e80098f0 tx=0x7f52e8009c00 comp rx=0 tx=0).stop 2023-03-03T07:58:33.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.147+0000 7f52f95ff700 1 -- 172.21.15.98:0/1082707981 shutdown_connections 2023-03-03T07:58:33.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.147+0000 7f52f95ff700 1 --2- 172.21.15.98:0/1082707981 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f52f415ca70 0x7f52f40cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.147+0000 7f52f95ff700 1 --2- 172.21.15.98:0/1082707981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52f406b4b0 0x7f52f40cfb60 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.147+0000 7f52f95ff700 1 -- 172.21.15.98:0/1082707981 >> 172.21.15.98:0/1082707981 conn(0x7f52f4073e70 msgr2=0x7f52f406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.147+0000 7f52f95ff700 1 -- 172.21.15.98:0/1082707981 shutdown_connections 2023-03-03T07:58:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.147+0000 7f52f95ff700 1 -- 172.21.15.98:0/1082707981 wait complete. 2023-03-03T07:58:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.149+0000 7f52f95ff700 1 Processor -- start 2023-03-03T07:58:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.149+0000 7f52f95ff700 1 -- start start 2023-03-03T07:58:33.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.150+0000 7f52f95ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f52f406b4b0 0x7f52f40b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:33.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.150+0000 7f52f95ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52f40d00a0 0x7f52f4154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:33.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.150+0000 7f52f95ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f52f415ca70 0x7f52f4154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:33.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.150+0000 7f52f95ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f41f63d0 con 0x7f52f40d00a0 2023-03-03T07:58:33.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.150+0000 7f52f95ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f41f5a30 con 0x7f52f406b4b0 2023-03-03T07:58:33.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.150+0000 7f52f95ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f41f5d10 con 0x7f52f415ca70 2023-03-03T07:58:33.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.150+0000 7f52f27fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52f40d00a0 0x7f52f4154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:33.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.150+0000 7f52f27fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52f40d00a0 0x7f52f4154220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51500/0 (socket says 172.21.15.98:51500) 2023-03-03T07:58:33.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.150+0000 7f52f27fc700 1 -- 172.21.15.98:0/532929006 learned_addr learned my addr 172.21.15.98:0/532929006 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:33.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.151+0000 7f52f2ffd700 1 --2- 172.21.15.98:0/532929006 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f52f406b4b0 0x7f52f40b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:33.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.151+0000 7f52f37fe700 1 --2- 172.21.15.98:0/532929006 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f52f415ca70 0x7f52f4154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:33.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.151+0000 7f52f27fc700 1 -- 172.21.15.98:0/532929006 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f52f415ca70 msgr2=0x7f52f4154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:33.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.151+0000 7f52f27fc700 1 --2- 172.21.15.98:0/532929006 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f52f415ca70 0x7f52f4154760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.151+0000 7f52f27fc700 1 -- 172.21.15.98:0/532929006 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f52f406b4b0 msgr2=0x7f52f40b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.151+0000 7f52f27fc700 1 --2- 172.21.15.98:0/532929006 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f52f406b4b0 0x7f52f40b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.151+0000 7f52f27fc700 1 -- 172.21.15.98:0/532929006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e8009550 con 0x7f52f40d00a0 2023-03-03T07:58:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.151+0000 7f52f27fc700 1 --2- 172.21.15.98:0/532929006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52f40d00a0 0x7f52f4154220 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f52e8000c00 tx=0x7f52e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.151+0000 7f52db7fe700 1 -- 172.21.15.98:0/532929006 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e8004380 con 0x7f52f40d00a0 2023-03-03T07:58:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.152+0000 7f52db7fe700 1 -- 172.21.15.98:0/532929006 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e801f070 con 0x7f52f40d00a0 2023-03-03T07:58:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.152+0000 7f52db7fe700 1 -- 172.21.15.98:0/532929006 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e80036a0 con 0x7f52f40d00a0 2023-03-03T07:58:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.152+0000 7f52f95ff700 1 -- 172.21.15.98:0/532929006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f40cd900 con 0x7f52f40d00a0 2023-03-03T07:58:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.152+0000 7f52f95ff700 1 -- 172.21.15.98:0/532929006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f40d1800 con 0x7f52f40d00a0 2023-03-03T07:58:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.153+0000 7f52f95ff700 1 -- 172.21.15.98:0/532929006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f40cdbe0 con 0x7f52f40d00a0 2023-03-03T07:58:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.154+0000 7f52db7fe700 1 -- 172.21.15.98:0/532929006 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f52e802a020 con 0x7f52f40d00a0 2023-03-03T07:58:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.156+0000 7f52db7fe700 1 --2- 172.21.15.98:0/532929006 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f52e00683c0 0x7f52e006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.156+0000 7f52db7fe700 1 -- 172.21.15.98:0/532929006 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f52e80873b0 con 0x7f52f40d00a0 2023-03-03T07:58:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.157+0000 7f52f2ffd700 1 --2- 172.21.15.98:0/532929006 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f52e00683c0 0x7f52e006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.157+0000 7f52f2ffd700 1 --2- 172.21.15.98:0/532929006 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f52e00683c0 0x7f52e006a870 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f52e4006b20 tx=0x7f52e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:33.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.161+0000 7f52db7fe700 1 -- 172.21.15.98:0/532929006 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f52e80539b0 con 0x7f52f40d00a0 2023-03-03T07:58:33.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.354+0000 7f52f95ff700 1 -- 172.21.15.98:0/532929006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f52f404e6e0 con 0x7f52f40d00a0 2023-03-03T07:58:33.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.355+0000 7f52db7fe700 1 -- 172.21.15.98:0/532929006 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v54) v1 ==== 92+0+10 (secure 0 0 0) 0x7f52e8056fd0 con 0x7f52f40d00a0 2023-03-03T07:58:33.356 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:58:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.357+0000 7f52f95ff700 1 -- 172.21.15.98:0/532929006 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f52e00683c0 msgr2=0x7f52e006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.357+0000 7f52f95ff700 1 --2- 172.21.15.98:0/532929006 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f52e00683c0 0x7f52e006a870 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f52e4006b20 tx=0x7f52e4008040 comp rx=0 tx=0).stop 2023-03-03T07:58:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.357+0000 7f52f95ff700 1 -- 172.21.15.98:0/532929006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52f40d00a0 msgr2=0x7f52f4154220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.357+0000 7f52f95ff700 1 --2- 172.21.15.98:0/532929006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52f40d00a0 0x7f52f4154220 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f52e8000c00 tx=0x7f52e8004750 comp rx=0 tx=0).stop 2023-03-03T07:58:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.357+0000 7f52f95ff700 1 -- 172.21.15.98:0/532929006 shutdown_connections 2023-03-03T07:58:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.357+0000 7f52f95ff700 1 --2- 172.21.15.98:0/532929006 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f52e00683c0 0x7f52e006a870 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.357+0000 7f52f95ff700 1 --2- 172.21.15.98:0/532929006 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f52f415ca70 0x7f52f4154760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.357+0000 7f52f95ff700 1 --2- 172.21.15.98:0/532929006 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f52f406b4b0 0x7f52f40b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.357+0000 7f52f95ff700 1 --2- 172.21.15.98:0/532929006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52f40d00a0 0x7f52f4154220 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.357+0000 7f52f95ff700 1 -- 172.21.15.98:0/532929006 >> 172.21.15.98:0/532929006 conn(0x7f52f4073e70 msgr2=0x7f52f415be30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.358+0000 7f52f95ff700 1 -- 172.21.15.98:0/532929006 shutdown_connections 2023-03-03T07:58:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.358+0000 7f52f95ff700 1 -- 172.21.15.98:0/532929006 wait complete. 2023-03-03T07:58:33.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:58:33.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.635+0000 7f376999d700 1 Processor -- start 2023-03-03T07:58:33.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.636+0000 7f376999d700 1 -- start start 2023-03-03T07:58:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.637+0000 7f376999d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f37641609b0 0x7f3764160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.637+0000 7f376999d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f37640cf570 0x7f376415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.637+0000 7f376999d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37641612c0 con 0x7f376415db30 2023-03-03T07:58:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.637+0000 7f376999d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37640cebb0 con 0x7f37641609b0 2023-03-03T07:58:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.637+0000 7f376999d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37640cd780 con 0x7f37640cf570 2023-03-03T07:58:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.637+0000 7f376899b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f37641609b0 0x7f3764160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.637+0000 7f376899b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f37641609b0 0x7f3764160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51504/0 (socket says 172.21.15.98:51504) 2023-03-03T07:58:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.637+0000 7f376899b700 1 -- 172.21.15.98:0/2836187635 learned_addr learned my addr 172.21.15.98:0/2836187635 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.638+0000 7f3763fff700 1 --2- 172.21.15.98:0/2836187635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f37640cf570 0x7f376415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.638+0000 7f376899b700 1 -- 172.21.15.98:0/2836187635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f37640cf570 msgr2=0x7f376415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.638+0000 7f376899b700 1 --2- 172.21.15.98:0/2836187635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f37640cf570 0x7f376415d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.638+0000 7f376899b700 1 -- 172.21.15.98:0/2836187635 >> v1:172.21.15.100:6789/0 conn(0x7f376415db30 legacy=0x7f37640b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.638+0000 7f376899b700 1 -- 172.21.15.98:0/2836187635 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37640cda60 con 0x7f37641609b0 2023-03-03T07:58:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.638+0000 7f376899b700 1 --2- 172.21.15.98:0/2836187635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37641609b0 0x7f3764160d80 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f37580098f0 tx=0x7f3758009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ee7c4f608c9741c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.639+0000 7f37637fe700 1 -- 172.21.15.98:0/2836187635 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3758011830 con 0x7f37641609b0 2023-03-03T07:58:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.639+0000 7f37637fe700 1 -- 172.21.15.98:0/2836187635 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3758011990 con 0x7f37641609b0 2023-03-03T07:58:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.639+0000 7f37637fe700 1 -- 172.21.15.98:0/2836187635 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3758011d20 con 0x7f37641609b0 2023-03-03T07:58:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.639+0000 7f376999d700 1 -- 172.21.15.98:0/2836187635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37641609b0 msgr2=0x7f3764160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.639+0000 7f376999d700 1 --2- 172.21.15.98:0/2836187635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37641609b0 0x7f3764160d80 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f37580098f0 tx=0x7f3758009c00 comp rx=0 tx=0).stop 2023-03-03T07:58:33.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.639+0000 7f376999d700 1 -- 172.21.15.98:0/2836187635 shutdown_connections 2023-03-03T07:58:33.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.639+0000 7f376999d700 1 --2- 172.21.15.98:0/2836187635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37641609b0 0x7f3764160d80 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.639+0000 7f376999d700 1 --2- 172.21.15.98:0/2836187635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f37640cf570 0x7f376415d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.639+0000 7f376999d700 1 -- 172.21.15.98:0/2836187635 >> 172.21.15.98:0/2836187635 conn(0x7f376406b880 msgr2=0x7f37640cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:33.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.639+0000 7f376999d700 1 -- 172.21.15.98:0/2836187635 shutdown_connections 2023-03-03T07:58:33.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.639+0000 7f376999d700 1 -- 172.21.15.98:0/2836187635 wait complete. 2023-03-03T07:58:33.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.641+0000 7f376999d700 1 Processor -- start 2023-03-03T07:58:33.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.642+0000 7f376999d700 1 -- start start 2023-03-03T07:58:33.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.642+0000 7f376999d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37640cf570 0x7f37641f1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:33.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.642+0000 7f376999d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f376415db30 0x7f37640cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:33.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.642+0000 7f376999d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f37641609b0 0x7f37640b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:33.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.643+0000 7f376999d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37640b6c70 con 0x7f37640cf570 2023-03-03T07:58:33.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.643+0000 7f376999d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37640b5ff0 con 0x7f37641609b0 2023-03-03T07:58:33.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.643+0000 7f376999d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37640b62d0 con 0x7f376415db30 2023-03-03T07:58:33.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.643+0000 7f376899b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37640cf570 0x7f37641f1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:33.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.643+0000 7f376899b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37640cf570 0x7f37641f1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51506/0 (socket says 172.21.15.98:51506) 2023-03-03T07:58:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.643+0000 7f376899b700 1 -- 172.21.15.98:0/1561914268 learned_addr learned my addr 172.21.15.98:0/1561914268 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.643+0000 7f3763fff700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f376415db30 0x7f37640cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.643+0000 7f376919c700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f37641609b0 0x7f37640b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.643+0000 7f376899b700 1 -- 172.21.15.98:0/1561914268 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f376415db30 msgr2=0x7f37640cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.643+0000 7f376899b700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f376415db30 0x7f37640cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.643+0000 7f376899b700 1 -- 172.21.15.98:0/1561914268 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f37641609b0 msgr2=0x7f37640b7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.643+0000 7f376899b700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f37641609b0 0x7f37640b7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.643+0000 7f376899b700 1 -- 172.21.15.98:0/1561914268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3758009550 con 0x7f37640cf570 2023-03-03T07:58:33.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.643+0000 7f3763fff700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f376415db30 0x7f37640cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:33.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.643+0000 7f376919c700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f37641609b0 0x7f37640b7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:33.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.644+0000 7f376899b700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37640cf570 0x7f37641f1a40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f3758014040 tx=0x7f3758004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:33.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.644+0000 7f37617fa700 1 -- 172.21.15.98:0/1561914268 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37580036a0 con 0x7f37640cf570 2023-03-03T07:58:33.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.644+0000 7f37617fa700 1 -- 172.21.15.98:0/1561914268 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375801e070 con 0x7f37640cf570 2023-03-03T07:58:33.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.644+0000 7f37617fa700 1 -- 172.21.15.98:0/1561914268 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3758003a40 con 0x7f37640cf570 2023-03-03T07:58:33.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.644+0000 7f376999d700 1 -- 172.21.15.98:0/1561914268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f376415cdf0 con 0x7f37640cf570 2023-03-03T07:58:33.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.644+0000 7f376999d700 1 -- 172.21.15.98:0/1561914268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37640cdd40 con 0x7f37640cf570 2023-03-03T07:58:33.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.645+0000 7f376999d700 1 -- 172.21.15.98:0/1561914268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f376415d0d0 con 0x7f37640cf570 2023-03-03T07:58:33.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.646+0000 7f37617fa700 1 -- 172.21.15.98:0/1561914268 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f375800f050 con 0x7f37640cf570 2023-03-03T07:58:33.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.647+0000 7f37617fa700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37480683c0 0x7f374806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:33.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.647+0000 7f37617fa700 1 -- 172.21.15.98:0/1561914268 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f3758086e10 con 0x7f37640cf570 2023-03-03T07:58:33.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.648+0000 7f3763fff700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37480683c0 0x7f374806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:33.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.648+0000 7f3763fff700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37480683c0 0x7f374806a870 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f37640cbae0 tx=0x7f3750008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:33.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.651+0000 7f37617fa700 1 -- 172.21.15.98:0/1561914268 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f375808a030 con 0x7f37640cf570 2023-03-03T07:58:33.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.850+0000 7f376999d700 1 -- 172.21.15.98:0/1561914268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f37640b6db0 con 0x7f37640cf570 2023-03-03T07:58:33.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.850+0000 7f37617fa700 1 -- 172.21.15.98:0/1561914268 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3758015020 con 0x7f37640cf570 2023-03-03T07:58:33.852 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:58:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.853+0000 7f376999d700 1 -- 172.21.15.98:0/1561914268 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37480683c0 msgr2=0x7f374806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.853+0000 7f376999d700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37480683c0 0x7f374806a870 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f37640cbae0 tx=0x7f3750008040 comp rx=0 tx=0).stop 2023-03-03T07:58:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.853+0000 7f376999d700 1 -- 172.21.15.98:0/1561914268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37640cf570 msgr2=0x7f37641f1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.853+0000 7f376999d700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37640cf570 0x7f37641f1a40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f3758014040 tx=0x7f3758004750 comp rx=0 tx=0).stop 2023-03-03T07:58:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.853+0000 7f376999d700 1 -- 172.21.15.98:0/1561914268 shutdown_connections 2023-03-03T07:58:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.853+0000 7f376999d700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37480683c0 0x7f374806a870 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.853+0000 7f376999d700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37640cf570 0x7f37641f1a40 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.853+0000 7f376999d700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f37641609b0 0x7f37640b7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.853+0000 7f376999d700 1 --2- 172.21.15.98:0/1561914268 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f376415db30 0x7f37640cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.853+0000 7f376999d700 1 -- 172.21.15.98:0/1561914268 >> 172.21.15.98:0/1561914268 conn(0x7f376406b880 msgr2=0x7f37640d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.853+0000 7f376999d700 1 -- 172.21.15.98:0/1561914268 shutdown_connections 2023-03-03T07:58:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:33.853+0000 7f376999d700 1 -- 172.21.15.98:0/1561914268 wait complete. 2023-03-03T07:58:33.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:58:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.131+0000 7f6eabdff700 1 Processor -- start 2023-03-03T07:58:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.131+0000 7f6eabdff700 1 -- start start 2023-03-03T07:58:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.132+0000 7f6eabdff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6ea415ca70 0x7f6ea40cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.132+0000 7f6eabdff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea40cf980 con 0x7f6ea406b4b0 2023-03-03T07:58:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.132+0000 7f6eabdff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea40d1800 con 0x7f6ea40d0270 2023-03-03T07:58:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.132+0000 7f6eabdff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea40cd5b0 con 0x7f6ea415ca70 2023-03-03T07:58:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.133+0000 7f6ea939a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6ea406b4b0 0x7f6ea40cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56422/0 (socket says 172.21.15.98:56422) 2023-03-03T07:58:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.133+0000 7f6ea939a700 1 -- 172.21.15.98:0/2877241112 learned_addr learned my addr 172.21.15.98:0/2877241112 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.133+0000 7f6ea9b9b700 1 --2- 172.21.15.98:0/2877241112 >> v2:172.21.15.100:3300/0 conn(0x7f6ea415ca70 0x7f6ea40cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.133+0000 7f6ea8b99700 1 -- 172.21.15.98:0/2877241112 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064097435 0 0) 0x7f6ea40cf980 con 0x7f6ea406b4b0 2023-03-03T07:58:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.133+0000 7f6ea8b99700 1 -- 172.21.15.98:0/2877241112 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e900035b0 con 0x7f6ea406b4b0 2023-03-03T07:58:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.134+0000 7f6ea8b99700 1 -- 172.21.15.98:0/2877241112 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 439349793 0 0) 0x7f6e900035b0 con 0x7f6ea406b4b0 2023-03-03T07:58:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.134+0000 7f6ea9b9b700 1 -- 172.21.15.98:0/2877241112 >> v1:172.21.15.100:6789/0 conn(0x7f6ea40d0270 legacy=0x7f6ea40d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.134+0000 7f6ea9b9b700 1 -- 172.21.15.98:0/2877241112 >> v1:172.21.15.98:6789/0 conn(0x7f6ea406b4b0 legacy=0x7f6ea40cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.134+0000 7f6ea9b9b700 1 -- 172.21.15.98:0/2877241112 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea40cd900 con 0x7f6ea415ca70 2023-03-03T07:58:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.134+0000 7f6ea9b9b700 1 --2- 172.21.15.98:0/2877241112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ea415ca70 0x7f6ea40cf440 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009840 tx=0x7f6ea0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e778cc39b987491 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.134+0000 7f6ea8b99700 1 -- 172.21.15.98:0/2877241112 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e94001670 con 0x7f6ea415ca70 2023-03-03T07:58:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.134+0000 7f6ea8b99700 1 -- 172.21.15.98:0/2877241112 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea0004030 con 0x7f6ea415ca70 2023-03-03T07:58:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.134+0000 7f6ea8b99700 1 -- 172.21.15.98:0/2877241112 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea00043c0 con 0x7f6ea415ca70 2023-03-03T07:58:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.135+0000 7f6eabdff700 1 -- 172.21.15.98:0/2877241112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ea415ca70 msgr2=0x7f6ea40cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.135+0000 7f6eabdff700 1 --2- 172.21.15.98:0/2877241112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ea415ca70 0x7f6ea40cf440 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009840 tx=0x7f6ea0009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.135+0000 7f6eabdff700 1 -- 172.21.15.98:0/2877241112 shutdown_connections 2023-03-03T07:58:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.135+0000 7f6eabdff700 1 --2- 172.21.15.98:0/2877241112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ea415ca70 0x7f6ea40cf440 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.135+0000 7f6eabdff700 1 -- 172.21.15.98:0/2877241112 >> 172.21.15.98:0/2877241112 conn(0x7f6ea4073e70 msgr2=0x7f6ea406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.135+0000 7f6eabdff700 1 -- 172.21.15.98:0/2877241112 shutdown_connections 2023-03-03T07:58:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.135+0000 7f6eabdff700 1 -- 172.21.15.98:0/2877241112 wait complete. 2023-03-03T07:58:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.137+0000 7f6eabdff700 1 Processor -- start 2023-03-03T07:58:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.137+0000 7f6eabdff700 1 -- start start 2023-03-03T07:58:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.138+0000 7f6eabdff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea406b4b0 0x7f6ea415fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.138+0000 7f6eabdff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ea40d0270 0x7f6ea40d2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.138+0000 7f6eabdff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6ea415ca70 0x7f6ea40d3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.138+0000 7f6eabdff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea415ffb0 con 0x7f6ea406b4b0 2023-03-03T07:58:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.138+0000 7f6eabdff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea41592d0 con 0x7f6ea40d0270 2023-03-03T07:58:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.138+0000 7f6eabdff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea4158750 con 0x7f6ea415ca70 2023-03-03T07:58:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.139+0000 7f6ea9b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea406b4b0 0x7f6ea415fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.139+0000 7f6ea9b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea406b4b0 0x7f6ea415fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51522/0 (socket says 172.21.15.98:51522) 2023-03-03T07:58:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.139+0000 7f6ea9b9b700 1 -- 172.21.15.98:0/4100625321 learned_addr learned my addr 172.21.15.98:0/4100625321 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.139+0000 7f6ea9b9b700 1 -- 172.21.15.98:0/4100625321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6ea415ca70 msgr2=0x7f6ea40d3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.139+0000 7f6ea9b9b700 1 --2- 172.21.15.98:0/4100625321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6ea415ca70 0x7f6ea40d3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.139+0000 7f6ea9b9b700 1 -- 172.21.15.98:0/4100625321 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ea40d0270 msgr2=0x7f6ea40d2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.139+0000 7f6ea9b9b700 1 --2- 172.21.15.98:0/4100625321 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ea40d0270 0x7f6ea40d2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.139+0000 7f6ea9b9b700 1 -- 172.21.15.98:0/4100625321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea00094a0 con 0x7f6ea406b4b0 2023-03-03T07:58:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.139+0000 7f6ea9b9b700 1 --2- 172.21.15.98:0/4100625321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea406b4b0 0x7f6ea415fa70 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6ea000a010 tx=0x7f6ea0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.139+0000 7f6e9a7fc700 1 -- 172.21.15.98:0/4100625321 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0003e30 con 0x7f6ea406b4b0 2023-03-03T07:58:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.140+0000 7f6e9a7fc700 1 -- 172.21.15.98:0/4100625321 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea001f070 con 0x7f6ea406b4b0 2023-03-03T07:58:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.140+0000 7f6eabdff700 1 -- 172.21.15.98:0/4100625321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea40d1aa0 con 0x7f6ea406b4b0 2023-03-03T07:58:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.140+0000 7f6e9a7fc700 1 -- 172.21.15.98:0/4100625321 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0011850 con 0x7f6ea406b4b0 2023-03-03T07:58:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.140+0000 7f6eabdff700 1 -- 172.21.15.98:0/4100625321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea40cdbe0 con 0x7f6ea406b4b0 2023-03-03T07:58:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.141+0000 7f6eabdff700 1 -- 172.21.15.98:0/4100625321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea415ce40 con 0x7f6ea406b4b0 2023-03-03T07:58:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.142+0000 7f6e9a7fc700 1 -- 172.21.15.98:0/4100625321 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f6ea002a020 con 0x7f6ea406b4b0 2023-03-03T07:58:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.144+0000 7f6e9a7fc700 1 --2- 172.21.15.98:0/4100625321 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6e880683c0 0x7f6e8806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.144+0000 7f6e9a7fc700 1 -- 172.21.15.98:0/4100625321 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f6ea0087500 con 0x7f6ea406b4b0 2023-03-03T07:58:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.144+0000 7f6ea939a700 1 --2- 172.21.15.98:0/4100625321 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6e880683c0 0x7f6e8806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:34.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.145+0000 7f6ea939a700 1 --2- 172.21.15.98:0/4100625321 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6e880683c0 0x7f6e8806a870 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6e94001eb0 tx=0x7f6e94001a20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.148+0000 7f6e9a7fc700 1 -- 172.21.15.98:0/4100625321 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6ea0053b60 con 0x7f6ea406b4b0 2023-03-03T07:58:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.342+0000 7f6eabdff700 1 -- 172.21.15.98:0/4100625321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6ea404e6e0 con 0x7f6ea406b4b0 2023-03-03T07:58:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.343+0000 7f6e9a7fc700 1 -- 172.21.15.98:0/4100625321 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6ea0057180 con 0x7f6ea406b4b0 2023-03-03T07:58:34.344 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:58:34.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.346+0000 7f6eabdff700 1 -- 172.21.15.98:0/4100625321 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6e880683c0 msgr2=0x7f6e8806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:34.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.346+0000 7f6eabdff700 1 --2- 172.21.15.98:0/4100625321 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6e880683c0 0x7f6e8806a870 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6e94001eb0 tx=0x7f6e94001a20 comp rx=0 tx=0).stop 2023-03-03T07:58:34.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.346+0000 7f6eabdff700 1 -- 172.21.15.98:0/4100625321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea406b4b0 msgr2=0x7f6ea415fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:34.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.346+0000 7f6eabdff700 1 --2- 172.21.15.98:0/4100625321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea406b4b0 0x7f6ea415fa70 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6ea000a010 tx=0x7f6ea0004750 comp rx=0 tx=0).stop 2023-03-03T07:58:34.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.346+0000 7f6eabdff700 1 -- 172.21.15.98:0/4100625321 shutdown_connections 2023-03-03T07:58:34.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.346+0000 7f6eabdff700 1 --2- 172.21.15.98:0/4100625321 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6e880683c0 0x7f6e8806a870 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:34.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.346+0000 7f6eabdff700 1 --2- 172.21.15.98:0/4100625321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea406b4b0 0x7f6ea415fa70 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:34.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.346+0000 7f6eabdff700 1 --2- 172.21.15.98:0/4100625321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6ea415ca70 0x7f6ea40d3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:34.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.346+0000 7f6eabdff700 1 --2- 172.21.15.98:0/4100625321 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ea40d0270 0x7f6ea40d2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:34.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.346+0000 7f6eabdff700 1 -- 172.21.15.98:0/4100625321 >> 172.21.15.98:0/4100625321 conn(0x7f6ea4073e70 msgr2=0x7f6ea406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:34.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.346+0000 7f6eabdff700 1 -- 172.21.15.98:0/4100625321 shutdown_connections 2023-03-03T07:58:34.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.346+0000 7f6eabdff700 1 -- 172.21.15.98:0/4100625321 wait complete. 2023-03-03T07:58:34.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T07:58:34.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.619+0000 7faeb219d700 1 Processor -- start 2023-03-03T07:58:34.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.619+0000 7faeb219d700 1 -- start start 2023-03-03T07:58:34.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.620+0000 7faeb219d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7faeac0cf570 0x7faeac15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:34.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.620+0000 7faeb219d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac0cd780 con 0x7faeac1609b0 2023-03-03T07:58:34.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.620+0000 7faeb219d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac0cda60 con 0x7faeac15db30 2023-03-03T07:58:34.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.620+0000 7faeb219d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeac0cebb0 con 0x7faeac0cf570 2023-03-03T07:58:34.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.621+0000 7faeb119b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7faeac1609b0 0x7faeac160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56426/0 (socket says 172.21.15.98:56426) 2023-03-03T07:58:34.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.621+0000 7faeb119b700 1 -- 172.21.15.98:0/2340310609 learned_addr learned my addr 172.21.15.98:0/2340310609 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:34.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.621+0000 7faeb099a700 1 --2- 172.21.15.98:0/2340310609 >> v2:172.21.15.100:3300/0 conn(0x7faeac0cf570 0x7faeac15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:34.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.621+0000 7faea3fff700 1 -- 172.21.15.98:0/2340310609 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401024862 0 0) 0x7faeac0cd780 con 0x7faeac1609b0 2023-03-03T07:58:34.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.621+0000 7faea3fff700 1 -- 172.21.15.98:0/2340310609 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae980035b0 con 0x7faeac1609b0 2023-03-03T07:58:34.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.621+0000 7faeb099a700 1 -- 172.21.15.98:0/2340310609 >> v1:172.21.15.100:6789/0 conn(0x7faeac15db30 legacy=0x7faeac0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:34.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.621+0000 7faea3fff700 1 -- 172.21.15.98:0/2340310609 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494133525 0 0) 0x7faeac0cda60 con 0x7faeac15db30 2023-03-03T07:58:34.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.621+0000 7faeb099a700 1 -- 172.21.15.98:0/2340310609 >> v1:172.21.15.98:6789/0 conn(0x7faeac1609b0 legacy=0x7faeac160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:34.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.622+0000 7faeb099a700 1 -- 172.21.15.98:0/2340310609 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeac0cdd40 con 0x7faeac0cf570 2023-03-03T07:58:34.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.622+0000 7faeb099a700 1 --2- 172.21.15.98:0/2340310609 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faeac0cf570 0x7faeac15d5f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009840 tx=0x7fae9c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad38f9df3a581655 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:34.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.622+0000 7faea3fff700 1 -- 172.21.15.98:0/2340310609 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c011830 con 0x7faeac0cf570 2023-03-03T07:58:34.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.622+0000 7faea3fff700 1 -- 172.21.15.98:0/2340310609 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9c011990 con 0x7faeac0cf570 2023-03-03T07:58:34.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.622+0000 7faea3fff700 1 -- 172.21.15.98:0/2340310609 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c011d20 con 0x7faeac0cf570 2023-03-03T07:58:34.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.623+0000 7faeb219d700 1 -- 172.21.15.98:0/2340310609 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faeac0cf570 msgr2=0x7faeac15d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:34.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.623+0000 7faeb219d700 1 --2- 172.21.15.98:0/2340310609 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faeac0cf570 0x7faeac15d5f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009840 tx=0x7fae9c009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:34.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.623+0000 7faeb219d700 1 -- 172.21.15.98:0/2340310609 shutdown_connections 2023-03-03T07:58:34.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.623+0000 7faeb219d700 1 --2- 172.21.15.98:0/2340310609 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faeac0cf570 0x7faeac15d5f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:34.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.623+0000 7faeb219d700 1 -- 172.21.15.98:0/2340310609 >> 172.21.15.98:0/2340310609 conn(0x7faeac06b880 msgr2=0x7faeac0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:34.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.623+0000 7faeb219d700 1 -- 172.21.15.98:0/2340310609 shutdown_connections 2023-03-03T07:58:34.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.623+0000 7faeb219d700 1 -- 172.21.15.98:0/2340310609 wait complete. 2023-03-03T07:58:34.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.625+0000 7faeb219d700 1 Processor -- start 2023-03-03T07:58:34.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.625+0000 7faeb219d700 1 -- start start 2023-03-03T07:58:34.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.626+0000 7faeb219d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faeac0cf570 0x7faeac14e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:34.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.626+0000 7faeb219d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faeac15db30 0x7faeac0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:34.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.626+0000 7faeb219d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeac1609b0 0x7faeac0b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:34.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.626+0000 7faeb219d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeac0d1840 con 0x7faeac1609b0 2023-03-03T07:58:34.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.626+0000 7faeb219d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeac0d0bc0 con 0x7faeac15db30 2023-03-03T07:58:34.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.626+0000 7faeb219d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeac0d0ea0 con 0x7faeac0cf570 2023-03-03T07:58:34.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.627+0000 7faeb199c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeac1609b0 0x7faeac0b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:34.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.627+0000 7faeb199c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeac1609b0 0x7faeac0b3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51538/0 (socket says 172.21.15.98:51538) 2023-03-03T07:58:34.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.627+0000 7faeb199c700 1 -- 172.21.15.98:0/416281574 learned_addr learned my addr 172.21.15.98:0/416281574 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:34.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.627+0000 7faeb099a700 1 --2- 172.21.15.98:0/416281574 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faeac15db30 0x7faeac0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:34.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.627+0000 7faeb119b700 1 --2- 172.21.15.98:0/416281574 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faeac0cf570 0x7faeac14e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.627+0000 7faeb199c700 1 -- 172.21.15.98:0/416281574 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faeac0cf570 msgr2=0x7faeac14e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.627+0000 7faeb199c700 1 --2- 172.21.15.98:0/416281574 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faeac0cf570 0x7faeac14e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.627+0000 7faeb199c700 1 -- 172.21.15.98:0/416281574 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faeac15db30 msgr2=0x7faeac0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:34.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.627+0000 7faeb199c700 1 --2- 172.21.15.98:0/416281574 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faeac15db30 0x7faeac0d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:34.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.627+0000 7faeb199c700 1 -- 172.21.15.98:0/416281574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c0094a0 con 0x7faeac1609b0 2023-03-03T07:58:34.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.627+0000 7faeb119b700 1 --2- 172.21.15.98:0/416281574 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faeac0cf570 0x7faeac14e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:34.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.628+0000 7faeb199c700 1 --2- 172.21.15.98:0/416281574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeac1609b0 0x7faeac0b3ad0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7faea400c890 tx=0x7faea400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:34.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.628+0000 7faea1ffb700 1 -- 172.21.15.98:0/416281574 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea4003e30 con 0x7faeac1609b0 2023-03-03T07:58:34.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.628+0000 7faea1ffb700 1 -- 172.21.15.98:0/416281574 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea4005070 con 0x7faeac1609b0 2023-03-03T07:58:34.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.628+0000 7faea1ffb700 1 -- 172.21.15.98:0/416281574 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea4005420 con 0x7faeac1609b0 2023-03-03T07:58:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.628+0000 7faeb219d700 1 -- 172.21.15.98:0/416281574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeac15d0d0 con 0x7faeac1609b0 2023-03-03T07:58:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.628+0000 7faeb219d700 1 -- 172.21.15.98:0/416281574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeac15cdf0 con 0x7faeac1609b0 2023-03-03T07:58:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.629+0000 7faeb219d700 1 -- 172.21.15.98:0/416281574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeac0b51b0 con 0x7faeac1609b0 2023-03-03T07:58:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.631+0000 7faea1ffb700 1 -- 172.21.15.98:0/416281574 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7faea4019030 con 0x7faeac1609b0 2023-03-03T07:58:34.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.632+0000 7faea1ffb700 1 --2- 172.21.15.98:0/416281574 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae900683c0 0x7fae9006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:34.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.633+0000 7faea1ffb700 1 -- 172.21.15.98:0/416281574 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7faea4085510 con 0x7faeac1609b0 2023-03-03T07:58:34.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.633+0000 7faeb119b700 1 --2- 172.21.15.98:0/416281574 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae900683c0 0x7fae9006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:34.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.633+0000 7faeb119b700 1 --2- 172.21.15.98:0/416281574 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae900683c0 0x7fae9006a870 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7faea8005910 tx=0x7faea8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.636+0000 7faea1ffb700 1 -- 172.21.15.98:0/416281574 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faea4051ae0 con 0x7faeac1609b0 2023-03-03T07:58:34.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.848+0000 7faeb219d700 1 -- 172.21.15.98:0/416281574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7faeac0d1980 con 0x7faeac1609b0 2023-03-03T07:58:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.849+0000 7faea1ffb700 1 -- 172.21.15.98:0/416281574 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v54) v1 ==== 80+0+0 (secure 0 0 0) 0x7faeac0d1980 con 0x7faeac1609b0 2023-03-03T07:58:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.851+0000 7faeb219d700 1 -- 172.21.15.98:0/416281574 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae900683c0 msgr2=0x7fae9006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.851+0000 7faeb219d700 1 --2- 172.21.15.98:0/416281574 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae900683c0 0x7fae9006a870 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7faea8005910 tx=0x7faea8009040 comp rx=0 tx=0).stop 2023-03-03T07:58:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.851+0000 7faeb219d700 1 -- 172.21.15.98:0/416281574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeac1609b0 msgr2=0x7faeac0b3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.851+0000 7faeb219d700 1 --2- 172.21.15.98:0/416281574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeac1609b0 0x7faeac0b3ad0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7faea400c890 tx=0x7faea400cba0 comp rx=0 tx=0).stop 2023-03-03T07:58:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.851+0000 7faeb219d700 1 -- 172.21.15.98:0/416281574 shutdown_connections 2023-03-03T07:58:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.851+0000 7faeb219d700 1 --2- 172.21.15.98:0/416281574 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae900683c0 0x7fae9006a870 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.851+0000 7faeb219d700 1 --2- 172.21.15.98:0/416281574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeac1609b0 0x7faeac0b3ad0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.851+0000 7faeb219d700 1 --2- 172.21.15.98:0/416281574 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faeac0cf570 0x7faeac14e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.851+0000 7faeb219d700 1 --2- 172.21.15.98:0/416281574 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faeac15db30 0x7faeac0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.851+0000 7faeb219d700 1 -- 172.21.15.98:0/416281574 >> 172.21.15.98:0/416281574 conn(0x7faeac06b880 msgr2=0x7faeac14f630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.852+0000 7faeb219d700 1 -- 172.21.15.98:0/416281574 shutdown_connections 2023-03-03T07:58:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:34.852+0000 7faeb219d700 1 -- 172.21.15.98:0/416281574 wait complete. 2023-03-03T07:58:34.856 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T07:58:34.880 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2023-03-03T07:58:34.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:34.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-03-03T07:58:34.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[129732]: fuse finished with error 0 and tester_r 0 2023-03-03T07:58:34.982 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T07:58:35.083 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T07:58:35.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:35.084 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-03-03T07:58:35.153 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:58:35.154 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:58:35.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:58:35.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.471+0000 7f0a743ff700 1 Processor -- start 2023-03-03T07:58:35.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.471+0000 7f0a743ff700 1 -- start start 2023-03-03T07:58:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.472+0000 7f0a743ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0a6c15cc00 0x7f0a6c0cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.472+0000 7f0a743ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0a6c0d27c0 0x7f0a6c0d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.472+0000 7f0a743ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0a6c0b60d0 0x7f0a6c0b6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.472+0000 7f0a743ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6c0ce040 con 0x7f0a6c0d27c0 2023-03-03T07:58:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.472+0000 7f0a743ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6c073be0 con 0x7f0a6c0b60d0 2023-03-03T07:58:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.472+0000 7f0a743ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0cde10 con 0x7f0a6c15cc00 2023-03-03T07:58:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.473+0000 7f0a7199a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0a6c0d27c0 0x7f0a6c0d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.473+0000 7f0a7199a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0a6c0d27c0 0x7f0a6c0d2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51412/0 (socket says 172.21.15.98:51412) 2023-03-03T07:58:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.473+0000 7f0a7199a700 1 -- 172.21.15.98:0/1234096027 learned_addr learned my addr 172.21.15.98:0/1234096027 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.473+0000 7f0a7299c700 1 --2- 172.21.15.98:0/1234096027 >> v2:172.21.15.100:3300/0 conn(0x7f0a6c0b60d0 0x7f0a6c0b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.473+0000 7f0a7219b700 1 --2- 172.21.15.98:0/1234096027 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0a6c15cc00 0x7f0a6c0cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.473+0000 7f0a7199a700 1 -- 172.21.15.98:0/1234096027 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0a6c15cc00 msgr2=0x7f0a6c0cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.473+0000 7f0a7199a700 1 --2- 172.21.15.98:0/1234096027 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0a6c15cc00 0x7f0a6c0cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.473+0000 7f0a7199a700 1 -- 172.21.15.98:0/1234096027 >> v2:172.21.15.100:3300/0 conn(0x7f0a6c0b60d0 msgr2=0x7f0a6c0b6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.473+0000 7f0a7199a700 1 --2- 172.21.15.98:0/1234096027 >> v2:172.21.15.100:3300/0 conn(0x7f0a6c0b60d0 0x7f0a6c0b6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.473+0000 7f0a7199a700 1 -- 172.21.15.98:0/1234096027 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c0d1a00 con 0x7f0a6c0d27c0 2023-03-03T07:58:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.474+0000 7f0a7199a700 1 --2- 172.21.15.98:0/1234096027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6c0d27c0 0x7f0a6c0d2c30 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c06bf10 tx=0x7f0a5c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93f49d6d7146ed76 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.474+0000 7f0a71199700 1 -- 172.21.15.98:0/1234096027 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c011830 con 0x7f0a6c0d27c0 2023-03-03T07:58:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.474+0000 7f0a71199700 1 -- 172.21.15.98:0/1234096027 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c011990 con 0x7f0a6c0d27c0 2023-03-03T07:58:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.474+0000 7f0a71199700 1 -- 172.21.15.98:0/1234096027 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c011d20 con 0x7f0a6c0d27c0 2023-03-03T07:58:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.475+0000 7f0a743ff700 1 -- 172.21.15.98:0/1234096027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6c0d27c0 msgr2=0x7f0a6c0d2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.475+0000 7f0a743ff700 1 --2- 172.21.15.98:0/1234096027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6c0d27c0 0x7f0a6c0d2c30 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c06bf10 tx=0x7f0a5c009bd0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.475+0000 7f0a743ff700 1 -- 172.21.15.98:0/1234096027 shutdown_connections 2023-03-03T07:58:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.475+0000 7f0a743ff700 1 --2- 172.21.15.98:0/1234096027 >> v2:172.21.15.100:3300/0 conn(0x7f0a6c0b60d0 0x7f0a6c0b6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.475+0000 7f0a743ff700 1 --2- 172.21.15.98:0/1234096027 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0a6c15cc00 0x7f0a6c0cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.475+0000 7f0a743ff700 1 --2- 172.21.15.98:0/1234096027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6c0d27c0 0x7f0a6c0d2c30 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.475+0000 7f0a743ff700 1 -- 172.21.15.98:0/1234096027 >> 172.21.15.98:0/1234096027 conn(0x7f0a6c06aee0 msgr2=0x7f0a6c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:35.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.475+0000 7f0a743ff700 1 -- 172.21.15.98:0/1234096027 shutdown_connections 2023-03-03T07:58:35.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.475+0000 7f0a743ff700 1 -- 172.21.15.98:0/1234096027 wait complete. 2023-03-03T07:58:35.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.477+0000 7f0a743ff700 1 Processor -- start 2023-03-03T07:58:35.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.477+0000 7f0a743ff700 1 -- start start 2023-03-03T07:58:35.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.478+0000 7f0a743ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6c0b60d0 0x7f0a6c0cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:35.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.478+0000 7f0a743ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0a6c0d27c0 0x7f0a6c0b5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:35.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.478+0000 7f0a743ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a6c15cc00 0x7f0a6c0c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:35.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.478+0000 7f0a743ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c150020 con 0x7f0a6c0b60d0 2023-03-03T07:58:35.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.478+0000 7f0a743ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c15feb0 con 0x7f0a6c15cc00 2023-03-03T07:58:35.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.478+0000 7f0a743ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0b41b0 con 0x7f0a6c0d27c0 2023-03-03T07:58:35.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.478+0000 7f0a7219b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6c0b60d0 0x7f0a6c0cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:35.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.479+0000 7f0a7219b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6c0b60d0 0x7f0a6c0cd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51420/0 (socket says 172.21.15.98:51420) 2023-03-03T07:58:35.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.479+0000 7f0a7219b700 1 -- 172.21.15.98:0/3317630745 learned_addr learned my addr 172.21.15.98:0/3317630745 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:35.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.479+0000 7f0a7219b700 1 -- 172.21.15.98:0/3317630745 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0a6c0d27c0 msgr2=0x7f0a6c0b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:35.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.479+0000 7f0a7219b700 1 --2- 172.21.15.98:0/3317630745 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0a6c0d27c0 0x7f0a6c0b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.479+0000 7f0a7219b700 1 -- 172.21.15.98:0/3317630745 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a6c15cc00 msgr2=0x7f0a6c0c8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:35.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.479+0000 7f0a7219b700 1 --2- 172.21.15.98:0/3317630745 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a6c15cc00 0x7f0a6c0c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.479+0000 7f0a7219b700 1 -- 172.21.15.98:0/3317630745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c009550 con 0x7f0a6c0b60d0 2023-03-03T07:58:35.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.479+0000 7f0a7219b700 1 --2- 172.21.15.98:0/3317630745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6c0b60d0 0x7f0a6c0cd000 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800efc0 tx=0x7f0a6800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:35.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.479+0000 7f0a62ffd700 1 -- 172.21.15.98:0/3317630745 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6800cdc0 con 0x7f0a6c0b60d0 2023-03-03T07:58:35.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.479+0000 7f0a62ffd700 1 -- 172.21.15.98:0/3317630745 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6800f040 con 0x7f0a6c0b60d0 2023-03-03T07:58:35.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.479+0000 7f0a743ff700 1 -- 172.21.15.98:0/3317630745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6c0d1fc0 con 0x7f0a6c0b60d0 2023-03-03T07:58:35.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.479+0000 7f0a62ffd700 1 -- 172.21.15.98:0/3317630745 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a680146a0 con 0x7f0a6c0b60d0 2023-03-03T07:58:35.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.480+0000 7f0a743ff700 1 -- 172.21.15.98:0/3317630745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6c0d1ce0 con 0x7f0a6c0b60d0 2023-03-03T07:58:35.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.481+0000 7f0a743ff700 1 -- 172.21.15.98:0/3317630745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6c0d22a0 con 0x7f0a6c0b60d0 2023-03-03T07:58:35.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.482+0000 7f0a62ffd700 1 -- 172.21.15.98:0/3317630745 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f0a68014800 con 0x7f0a6c0b60d0 2023-03-03T07:58:35.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.483+0000 7f0a62ffd700 1 --2- 172.21.15.98:0/3317630745 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0a500683c0 0x7f0a5006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:35.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.483+0000 7f0a62ffd700 1 -- 172.21.15.98:0/3317630745 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f0a68086a00 con 0x7f0a6c0b60d0 2023-03-03T07:58:35.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.484+0000 7f0a7199a700 1 --2- 172.21.15.98:0/3317630745 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0a500683c0 0x7f0a5006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:35.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.484+0000 7f0a7199a700 1 --2- 172.21.15.98:0/3317630745 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0a500683c0 0x7f0a5006a870 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0c9b50 tx=0x7f0a5c010000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:35.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.487+0000 7f0a62ffd700 1 -- 172.21.15.98:0/3317630745 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0a68053060 con 0x7f0a6c0b60d0 2023-03-03T07:58:35.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.682+0000 7f0a743ff700 1 -- 172.21.15.98:0/3317630745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a6c0729b0 con 0x7f0a6c0b60d0 2023-03-03T07:58:35.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.684+0000 7f0a62ffd700 1 -- 172.21.15.98:0/3317630745 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+18018 (secure 0 0 0) 0x7f0a68056680 con 0x7f0a6c0b60d0 2023-03-03T07:58:35.685 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:35.686 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":54,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:58:28.723281+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:58:10.728830+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-03-03T07:58:12.224649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.6","mappings":[{"from":0,"to":1}]},{"pgid":"8.17","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:58:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.686+0000 7f0a743ff700 1 -- 172.21.15.98:0/3317630745 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0a500683c0 msgr2=0x7f0a5006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.686+0000 7f0a743ff700 1 --2- 172.21.15.98:0/3317630745 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0a500683c0 0x7f0a5006a870 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0c9b50 tx=0x7f0a5c010000 comp rx=0 tx=0).stop 2023-03-03T07:58:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.686+0000 7f0a743ff700 1 -- 172.21.15.98:0/3317630745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6c0b60d0 msgr2=0x7f0a6c0cd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.686+0000 7f0a743ff700 1 --2- 172.21.15.98:0/3317630745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6c0b60d0 0x7f0a6c0cd000 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800efc0 tx=0x7f0a6800c3a0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.686+0000 7f0a743ff700 1 -- 172.21.15.98:0/3317630745 shutdown_connections 2023-03-03T07:58:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.686+0000 7f0a743ff700 1 --2- 172.21.15.98:0/3317630745 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0a500683c0 0x7f0a5006a870 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.686+0000 7f0a743ff700 1 --2- 172.21.15.98:0/3317630745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6c0b60d0 0x7f0a6c0cd000 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.686+0000 7f0a743ff700 1 --2- 172.21.15.98:0/3317630745 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a6c15cc00 0x7f0a6c0c8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.686+0000 7f0a743ff700 1 --2- 172.21.15.98:0/3317630745 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0a6c0d27c0 0x7f0a6c0b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.686+0000 7f0a743ff700 1 -- 172.21.15.98:0/3317630745 >> 172.21.15.98:0/3317630745 conn(0x7f0a6c06aee0 msgr2=0x7f0a6c0b6550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.687+0000 7f0a743ff700 1 -- 172.21.15.98:0/3317630745 shutdown_connections 2023-03-03T07:58:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.687+0000 7f0a743ff700 1 -- 172.21.15.98:0/3317630745 wait complete. 2023-03-03T07:58:35.716 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:58:10.728830+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:58:12.224649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:58:35.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:58:35.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.963+0000 7f6470fff700 1 Processor -- start 2023-03-03T07:58:35.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.963+0000 7f6470fff700 1 -- start start 2023-03-03T07:58:35.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.965+0000 7f6470fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f646c06b4b0 0x7f646c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:35.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.965+0000 7f6470fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646c0cf980 con 0x7f646c15ca70 2023-03-03T07:58:35.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.965+0000 7f6470fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646c0d1800 con 0x7f646c0d0270 2023-03-03T07:58:35.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.965+0000 7f6470fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646c0cd5b0 con 0x7f646c06b4b0 2023-03-03T07:58:35.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.965+0000 7f646a59c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f646c15ca70 0x7f646c0cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48036/0 (socket says 172.21.15.98:48036) 2023-03-03T07:58:35.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.965+0000 7f646a59c700 1 -- 172.21.15.98:0/2776124494 learned_addr learned my addr 172.21.15.98:0/2776124494 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:35.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.965+0000 7f646ad9d700 1 --2- 172.21.15.98:0/2776124494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f646c06b4b0 0x7f646c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:35.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099878193 0 0) 0x7f646c0cf980 con 0x7f646c15ca70 2023-03-03T07:58:35.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64540035b0 con 0x7f646c15ca70 2023-03-03T07:58:35.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057032441 0 0) 0x7f646c0d1800 con 0x7f646c0d0270 2023-03-03T07:58:35.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f646c0cf980 con 0x7f646c0d0270 2023-03-03T07:58:35.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4273280943 0 0) 0x7f64540035b0 con 0x7f646c15ca70 2023-03-03T07:58:35.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f646c06b4b0 msgr2=0x7f646c0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:35.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 --2- 172.21.15.98:0/2776124494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f646c06b4b0 0x7f646c0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 >> v1:172.21.15.100:6789/0 conn(0x7f646c0d0270 legacy=0x7f646c0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c0cd900 con 0x7f646c15ca70 2023-03-03T07:58:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f645c001670 con 0x7f646c15ca70 2023-03-03T07:58:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 >> v1:172.21.15.98:6789/0 conn(0x7f646c15ca70 legacy=0x7f646c0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 --2- 172.21.15.98:0/2776124494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f646c06b4b0 0x7f6454003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 --2- 172.21.15.98:0/2776124494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6454004170 0x7f6454004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 --2- 172.21.15.98:0/2776124494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6454004d90 0x7f6454005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c0cd5b0 con 0x7f6454004170 2023-03-03T07:58:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645400bba0 con 0x7f6454004d90 2023-03-03T07:58:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645400bd60 con 0x7f646c06b4b0 2023-03-03T07:58:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.966+0000 7f6469d9b700 1 --2- 172.21.15.98:0/2776124494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6454004170 0x7f6454004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.967+0000 7f646a59c700 1 --2- 172.21.15.98:0/2776124494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f646c06b4b0 0x7f6454003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.967+0000 7f646ad9d700 1 --2- 172.21.15.98:0/2776124494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6454004d90 0x7f6454005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.967+0000 7f6469d9b700 1 -- 172.21.15.98:0/2776124494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f646c06b4b0 msgr2=0x7f6454003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.967+0000 7f6469d9b700 1 --2- 172.21.15.98:0/2776124494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f646c06b4b0 0x7f6454003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.967+0000 7f6469d9b700 1 -- 172.21.15.98:0/2776124494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6454004d90 msgr2=0x7f6454005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:35.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.967+0000 7f6469d9b700 1 --2- 172.21.15.98:0/2776124494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6454004d90 0x7f6454005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.967+0000 7f6469d9b700 1 -- 172.21.15.98:0/2776124494 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64540035b0 con 0x7f6454004170 2023-03-03T07:58:35.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.967+0000 7f6469d9b700 1 --2- 172.21.15.98:0/2776124494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6454004170 0x7f6454004850 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f646c0ceed0 tx=0x7f6454005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63544a78d87ade76 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:35.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.967+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6460001d80 con 0x7f6454004170 2023-03-03T07:58:35.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.968+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64600052d0 con 0x7f6454004170 2023-03-03T07:58:35.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.968+0000 7f646959a700 1 -- 172.21.15.98:0/2776124494 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6460001470 con 0x7f6454004170 2023-03-03T07:58:35.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.968+0000 7f6470fff700 1 -- 172.21.15.98:0/2776124494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6454004170 msgr2=0x7f6454004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:35.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.968+0000 7f6470fff700 1 --2- 172.21.15.98:0/2776124494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6454004170 0x7f6454004850 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f646c0ceed0 tx=0x7f6454005880 comp rx=0 tx=0).stop 2023-03-03T07:58:35.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.968+0000 7f646ad9d700 1 -- 172.21.15.98:0/2776124494 reap_dead start 2023-03-03T07:58:35.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.968+0000 7f6470fff700 1 -- 172.21.15.98:0/2776124494 shutdown_connections 2023-03-03T07:58:35.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.968+0000 7f6470fff700 1 -- 172.21.15.98:0/2776124494 >> 172.21.15.98:0/2776124494 conn(0x7f646c073e70 msgr2=0x7f646c06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:35.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.968+0000 7f6470fff700 1 -- 172.21.15.98:0/2776124494 shutdown_connections 2023-03-03T07:58:35.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.968+0000 7f6470fff700 1 -- 172.21.15.98:0/2776124494 wait complete. 2023-03-03T07:58:35.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.970+0000 7f6470fff700 1 Processor -- start 2023-03-03T07:58:35.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.971+0000 7f6470fff700 1 -- start start 2023-03-03T07:58:35.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.971+0000 7f6470fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6454004170 0x7f646c0ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:35.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.972+0000 7f6470fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f646c0b2be0 0x7f646c0b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:35.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.972+0000 7f6470fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f646c1450b0 0x7f646c145550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:35.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.972+0000 7f6470fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c145a90 con 0x7f646c1450b0 2023-03-03T07:58:35.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.972+0000 7f6470fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c0ca370 con 0x7f6454004170 2023-03-03T07:58:35.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.972+0000 7f6470fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646c0b3050 con 0x7f646c0b2be0 2023-03-03T07:58:35.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.972+0000 7f646ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f646c1450b0 0x7f646c145550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:35.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.972+0000 7f646ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f646c1450b0 0x7f646c145550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51434/0 (socket says 172.21.15.98:51434) 2023-03-03T07:58:35.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.972+0000 7f646ad9d700 1 -- 172.21.15.98:0/1497979388 learned_addr learned my addr 172.21.15.98:0/1497979388 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:35.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.972+0000 7f6469d9b700 1 --2- 172.21.15.98:0/1497979388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f646c0b2be0 0x7f646c0b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:35.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.972+0000 7f646a59c700 1 --2- 172.21.15.98:0/1497979388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6454004170 0x7f646c0ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:35.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.972+0000 7f646ad9d700 1 -- 172.21.15.98:0/1497979388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f646c0b2be0 msgr2=0x7f646c0b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:35.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.972+0000 7f646ad9d700 1 --2- 172.21.15.98:0/1497979388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f646c0b2be0 0x7f646c0b4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.972+0000 7f646ad9d700 1 -- 172.21.15.98:0/1497979388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6454004170 msgr2=0x7f646c0ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.972+0000 7f646ad9d700 1 --2- 172.21.15.98:0/1497979388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6454004170 0x7f646c0ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.972+0000 7f646ad9d700 1 -- 172.21.15.98:0/1497979388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c0d1aa0 con 0x7f646c1450b0 2023-03-03T07:58:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.973+0000 7f646ad9d700 1 --2- 172.21.15.98:0/1497979388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f646c1450b0 0x7f646c145550 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f646c0b92d0 tx=0x7f6464011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.973+0000 7f645affd700 1 -- 172.21.15.98:0/1497979388 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646400e040 con 0x7f646c1450b0 2023-03-03T07:58:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.973+0000 7f645affd700 1 -- 172.21.15.98:0/1497979388 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646400f040 con 0x7f646c1450b0 2023-03-03T07:58:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.973+0000 7f645affd700 1 -- 172.21.15.98:0/1497979388 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646400fc10 con 0x7f646c1450b0 2023-03-03T07:58:35.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.973+0000 7f6470fff700 1 -- 172.21.15.98:0/1497979388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646c0cdbe0 con 0x7f646c1450b0 2023-03-03T07:58:35.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.973+0000 7f6470fff700 1 -- 172.21.15.98:0/1497979388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646c15ce40 con 0x7f646c1450b0 2023-03-03T07:58:35.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.974+0000 7f6470fff700 1 -- 172.21.15.98:0/1497979388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f646c04e6e0 con 0x7f646c1450b0 2023-03-03T07:58:35.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.976+0000 7f645affd700 1 -- 172.21.15.98:0/1497979388 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f646400fd70 con 0x7f646c1450b0 2023-03-03T07:58:35.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.977+0000 7f645affd700 1 --2- 172.21.15.98:0/1497979388 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f644c0683c0 0x7f644c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:35.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.977+0000 7f645affd700 1 -- 172.21.15.98:0/1497979388 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f64640863c0 con 0x7f646c1450b0 2023-03-03T07:58:35.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.977+0000 7f646a59c700 1 --2- 172.21.15.98:0/1497979388 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f644c0683c0 0x7f644c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:35.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.978+0000 7f646a59c700 1 --2- 172.21.15.98:0/1497979388 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f644c0683c0 0x7f644c06a870 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f645c001eb0 tx=0x7f645c001000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:35.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:35.981+0000 7f645affd700 1 -- 172.21.15.98:0/1497979388 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6464052990 con 0x7f646c1450b0 2023-03-03T07:58:36.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.173+0000 7f6470fff700 1 -- 172.21.15.98:0/1497979388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f646c0ca4b0 con 0x7f646c1450b0 2023-03-03T07:58:36.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.174+0000 7f645affd700 1 -- 172.21.15.98:0/1497979388 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6464055fb0 con 0x7f646c1450b0 2023-03-03T07:58:36.176 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:58:36.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.176+0000 7f6470fff700 1 -- 172.21.15.98:0/1497979388 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f644c0683c0 msgr2=0x7f644c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:36.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.176+0000 7f6470fff700 1 --2- 172.21.15.98:0/1497979388 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f644c0683c0 0x7f644c06a870 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f645c001eb0 tx=0x7f645c001000 comp rx=0 tx=0).stop 2023-03-03T07:58:36.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.176+0000 7f6470fff700 1 -- 172.21.15.98:0/1497979388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f646c1450b0 msgr2=0x7f646c145550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:36.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.176+0000 7f6470fff700 1 --2- 172.21.15.98:0/1497979388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f646c1450b0 0x7f646c145550 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f646c0b92d0 tx=0x7f6464011ed0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.177+0000 7f6470fff700 1 -- 172.21.15.98:0/1497979388 shutdown_connections 2023-03-03T07:58:36.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.177+0000 7f6470fff700 1 --2- 172.21.15.98:0/1497979388 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f644c0683c0 0x7f644c06a870 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.177+0000 7f6470fff700 1 --2- 172.21.15.98:0/1497979388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f646c1450b0 0x7f646c145550 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.177+0000 7f6470fff700 1 --2- 172.21.15.98:0/1497979388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6454004170 0x7f646c0ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.177+0000 7f6470fff700 1 --2- 172.21.15.98:0/1497979388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f646c0b2be0 0x7f646c0b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.177+0000 7f6470fff700 1 -- 172.21.15.98:0/1497979388 >> 172.21.15.98:0/1497979388 conn(0x7f646c073e70 msgr2=0x7f646c0d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:36.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.177+0000 7f6470fff700 1 -- 172.21.15.98:0/1497979388 shutdown_connections 2023-03-03T07:58:36.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.177+0000 7f6470fff700 1 -- 172.21.15.98:0/1497979388 wait complete. 2023-03-03T07:58:36.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:58:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.450+0000 7fc68d39d700 1 Processor -- start 2023-03-03T07:58:36.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.450+0000 7fc68d39d700 1 -- start start 2023-03-03T07:58:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.452+0000 7fc68d39d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc6880cf570 0x7fc68815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.452+0000 7fc68d39d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc68815db30 0x7fc6880b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:36.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.452+0000 7fc68d39d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6880cd780 con 0x7fc6881609b0 2023-03-03T07:58:36.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.452+0000 7fc68d39d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6880cebb0 con 0x7fc6880cf570 2023-03-03T07:58:36.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.452+0000 7fc68d39d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6880cda60 con 0x7fc68815db30 2023-03-03T07:58:36.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.452+0000 7fc687fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc6881609b0 0x7fc688160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48046/0 (socket says 172.21.15.98:48046) 2023-03-03T07:58:36.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.452+0000 7fc687fff700 1 -- 172.21.15.98:0/276480136 learned_addr learned my addr 172.21.15.98:0/276480136 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:36.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.452+0000 7fc6877fe700 1 --2- 172.21.15.98:0/276480136 >> v2:172.21.15.98:3300/0 conn(0x7fc6880cf570 0x7fc68815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:36.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.452+0000 7fc68cb9c700 1 --2- 172.21.15.98:0/276480136 >> v2:172.21.15.100:3300/0 conn(0x7fc68815db30 0x7fc6880b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:36.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.453+0000 7fc6877fe700 1 -- 172.21.15.98:0/276480136 >> v2:172.21.15.100:3300/0 conn(0x7fc68815db30 msgr2=0x7fc6880b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:36.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.453+0000 7fc6877fe700 1 --2- 172.21.15.98:0/276480136 >> v2:172.21.15.100:3300/0 conn(0x7fc68815db30 0x7fc6880b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.453+0000 7fc6877fe700 1 -- 172.21.15.98:0/276480136 >> v1:172.21.15.98:6789/0 conn(0x7fc6881609b0 legacy=0x7fc688160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:36.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.453+0000 7fc6877fe700 1 -- 172.21.15.98:0/276480136 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6880cdd40 con 0x7fc6880cf570 2023-03-03T07:58:36.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.453+0000 7fc686ffd700 1 -- 172.21.15.98:0/276480136 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084746912 0 0) 0x7fc6880cd780 con 0x7fc6881609b0 2023-03-03T07:58:36.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.453+0000 7fc6877fe700 1 --2- 172.21.15.98:0/276480136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6880cf570 0x7fc68815d5f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc6880ca080 tx=0x7fc67c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c3b2913af2166d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:36.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.454+0000 7fc686ffd700 1 -- 172.21.15.98:0/276480136 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67c011830 con 0x7fc6880cf570 2023-03-03T07:58:36.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.454+0000 7fc686ffd700 1 -- 172.21.15.98:0/276480136 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67c011990 con 0x7fc6880cf570 2023-03-03T07:58:36.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.454+0000 7fc686ffd700 1 -- 172.21.15.98:0/276480136 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67c011d20 con 0x7fc6880cf570 2023-03-03T07:58:36.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.454+0000 7fc68d39d700 1 -- 172.21.15.98:0/276480136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6880cf570 msgr2=0x7fc68815d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:36.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.454+0000 7fc68d39d700 1 --2- 172.21.15.98:0/276480136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6880cf570 0x7fc68815d5f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc6880ca080 tx=0x7fc67c009b20 comp rx=0 tx=0).stop 2023-03-03T07:58:36.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.454+0000 7fc68d39d700 1 -- 172.21.15.98:0/276480136 shutdown_connections 2023-03-03T07:58:36.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.454+0000 7fc68d39d700 1 --2- 172.21.15.98:0/276480136 >> v2:172.21.15.100:3300/0 conn(0x7fc68815db30 0x7fc6880b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.454+0000 7fc68d39d700 1 --2- 172.21.15.98:0/276480136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6880cf570 0x7fc68815d5f0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.454+0000 7fc68d39d700 1 -- 172.21.15.98:0/276480136 >> 172.21.15.98:0/276480136 conn(0x7fc68806b880 msgr2=0x7fc6880cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:36.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.454+0000 7fc68d39d700 1 -- 172.21.15.98:0/276480136 shutdown_connections 2023-03-03T07:58:36.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.454+0000 7fc68d39d700 1 -- 172.21.15.98:0/276480136 wait complete. 2023-03-03T07:58:36.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.456+0000 7fc68d39d700 1 Processor -- start 2023-03-03T07:58:36.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.457+0000 7fc68d39d700 1 -- start start 2023-03-03T07:58:36.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.458+0000 7fc68d39d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6880cf570 0x7fc68814c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:36.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.458+0000 7fc68d39d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc68815db30 0x7fc6880b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:36.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.458+0000 7fc68d39d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6881609b0 0x7fc6880b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:36.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.458+0000 7fc68d39d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68814cdb0 con 0x7fc6881609b0 2023-03-03T07:58:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.458+0000 7fc68d39d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6880cbbd0 con 0x7fc6880cf570 2023-03-03T07:58:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.458+0000 7fc68d39d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6880cbeb0 con 0x7fc68815db30 2023-03-03T07:58:36.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.458+0000 7fc68cb9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6881609b0 0x7fc6880b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:36.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.458+0000 7fc68cb9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6881609b0 0x7fc6880b4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51448/0 (socket says 172.21.15.98:51448) 2023-03-03T07:58:36.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.458+0000 7fc68cb9c700 1 -- 172.21.15.98:0/3049292396 learned_addr learned my addr 172.21.15.98:0/3049292396 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:36.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.458+0000 7fc687fff700 1 --2- 172.21.15.98:0/3049292396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6880cf570 0x7fc68814c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:36.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.458+0000 7fc6877fe700 1 --2- 172.21.15.98:0/3049292396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc68815db30 0x7fc6880b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:36.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.458+0000 7fc68cb9c700 1 -- 172.21.15.98:0/3049292396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc68815db30 msgr2=0x7fc6880b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:36.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.459+0000 7fc68cb9c700 1 --2- 172.21.15.98:0/3049292396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc68815db30 0x7fc6880b3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.459+0000 7fc68cb9c700 1 -- 172.21.15.98:0/3049292396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6880cf570 msgr2=0x7fc68814c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:36.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.459+0000 7fc68cb9c700 1 --2- 172.21.15.98:0/3049292396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6880cf570 0x7fc68814c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.459+0000 7fc68cb9c700 1 -- 172.21.15.98:0/3049292396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67c0094a0 con 0x7fc6881609b0 2023-03-03T07:58:36.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.459+0000 7fc6877fe700 1 --2- 172.21.15.98:0/3049292396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc68815db30 0x7fc6880b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.459+0000 7fc68cb9c700 1 --2- 172.21.15.98:0/3049292396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6881609b0 0x7fc6880b4010 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc68000ef90 tx=0x7fc68000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.459+0000 7fc684ff9700 1 -- 172.21.15.98:0/3049292396 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68000cd40 con 0x7fc6881609b0 2023-03-03T07:58:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.459+0000 7fc684ff9700 1 -- 172.21.15.98:0/3049292396 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc68000f040 con 0x7fc6881609b0 2023-03-03T07:58:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.459+0000 7fc68d39d700 1 -- 172.21.15.98:0/3049292396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc68815d0d0 con 0x7fc6881609b0 2023-03-03T07:58:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.459+0000 7fc684ff9700 1 -- 172.21.15.98:0/3049292396 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68000cea0 con 0x7fc6881609b0 2023-03-03T07:58:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.459+0000 7fc68d39d700 1 -- 172.21.15.98:0/3049292396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc68815cdf0 con 0x7fc6881609b0 2023-03-03T07:58:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.460+0000 7fc68d39d700 1 -- 172.21.15.98:0/3049292396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6880ca0f0 con 0x7fc6881609b0 2023-03-03T07:58:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.462+0000 7fc684ff9700 1 -- 172.21.15.98:0/3049292396 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fc6800146c0 con 0x7fc6881609b0 2023-03-03T07:58:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.463+0000 7fc684ff9700 1 --2- 172.21.15.98:0/3049292396 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc6740683c0 0x7fc67406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.463+0000 7fc684ff9700 1 -- 172.21.15.98:0/3049292396 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fc680086cc0 con 0x7fc6881609b0 2023-03-03T07:58:36.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.463+0000 7fc687fff700 1 --2- 172.21.15.98:0/3049292396 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc6740683c0 0x7fc67406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:36.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.464+0000 7fc687fff700 1 --2- 172.21.15.98:0/3049292396 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc6740683c0 0x7fc67406a870 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc678000f60 tx=0x7fc678007e70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:36.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.467+0000 7fc684ff9700 1 -- 172.21.15.98:0/3049292396 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc6880ca0f0 con 0x7fc6881609b0 2023-03-03T07:58:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.662+0000 7fc68d39d700 1 -- 172.21.15.98:0/3049292396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc6880ca0f0 con 0x7fc6881609b0 2023-03-03T07:58:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.662+0000 7fc684ff9700 1 -- 172.21.15.98:0/3049292396 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v54) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc6880ca0f0 con 0x7fc6881609b0 2023-03-03T07:58:36.664 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:58:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.665+0000 7fc68d39d700 1 -- 172.21.15.98:0/3049292396 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc6740683c0 msgr2=0x7fc67406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.665+0000 7fc68d39d700 1 --2- 172.21.15.98:0/3049292396 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc6740683c0 0x7fc67406a870 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc678000f60 tx=0x7fc678007e70 comp rx=0 tx=0).stop 2023-03-03T07:58:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.665+0000 7fc68d39d700 1 -- 172.21.15.98:0/3049292396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6881609b0 msgr2=0x7fc6880b4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.665+0000 7fc68d39d700 1 --2- 172.21.15.98:0/3049292396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6881609b0 0x7fc6880b4010 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc68000ef90 tx=0x7fc68000c3a0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.665+0000 7fc68d39d700 1 -- 172.21.15.98:0/3049292396 shutdown_connections 2023-03-03T07:58:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.665+0000 7fc68d39d700 1 --2- 172.21.15.98:0/3049292396 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc6740683c0 0x7fc67406a870 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.665+0000 7fc68d39d700 1 --2- 172.21.15.98:0/3049292396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6881609b0 0x7fc6880b4010 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.665+0000 7fc68d39d700 1 --2- 172.21.15.98:0/3049292396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6880cf570 0x7fc68814c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.665+0000 7fc68d39d700 1 --2- 172.21.15.98:0/3049292396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc68815db30 0x7fc6880b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.665+0000 7fc68d39d700 1 -- 172.21.15.98:0/3049292396 >> 172.21.15.98:0/3049292396 conn(0x7fc68806b880 msgr2=0x7fc688152310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:36.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.665+0000 7fc68d39d700 1 -- 172.21.15.98:0/3049292396 shutdown_connections 2023-03-03T07:58:36.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.665+0000 7fc68d39d700 1 -- 172.21.15.98:0/3049292396 wait complete. 2023-03-03T07:58:36.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:58:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.942+0000 7f59c359e700 1 Processor -- start 2023-03-03T07:58:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.943+0000 7f59c359e700 1 -- start start 2023-03-03T07:58:36.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.944+0000 7f59c359e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f59c40b8910 0x7f59c41604a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:36.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.944+0000 7f59c359e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f59c40b5e70 0x7f59c40b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:36.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.944+0000 7f59c359e700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c415ce00 con 0x7f59c40cf570 2023-03-03T07:58:36.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.944+0000 7f59c359e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c40d0bf0 con 0x7f59c40b8910 2023-03-03T07:58:36.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.944+0000 7f59c359e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c40b8ce0 con 0x7f59c40b5e70 2023-03-03T07:58:36.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.944+0000 7f59c259c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f59c40b8910 0x7f59c41604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:36.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.944+0000 7f59c259c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f59c40b8910 0x7f59c41604a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51456/0 (socket says 172.21.15.98:51456) 2023-03-03T07:58:36.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.944+0000 7f59c259c700 1 -- 172.21.15.98:0/2959350484 learned_addr learned my addr 172.21.15.98:0/2959350484 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:36.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.945+0000 7f59c1d9b700 1 --2- 172.21.15.98:0/2959350484 >> v2:172.21.15.100:3300/0 conn(0x7f59c40b5e70 0x7f59c40b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:36.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.945+0000 7f59c259c700 1 -- 172.21.15.98:0/2959350484 >> v2:172.21.15.100:3300/0 conn(0x7f59c40b5e70 msgr2=0x7f59c40b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:36.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.945+0000 7f59c259c700 1 --2- 172.21.15.98:0/2959350484 >> v2:172.21.15.100:3300/0 conn(0x7f59c40b5e70 0x7f59c40b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.945+0000 7f59c259c700 1 -- 172.21.15.98:0/2959350484 >> v1:172.21.15.100:6789/0 conn(0x7f59c40cf570 legacy=0x7f59c40d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:36.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.945+0000 7f59c259c700 1 -- 172.21.15.98:0/2959350484 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c4160cf0 con 0x7f59c40b8910 2023-03-03T07:58:36.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.945+0000 7f59c259c700 1 --2- 172.21.15.98:0/2959350484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c40b8910 0x7f59c41604a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f59c4074810 tx=0x7f59b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e18f2c33793da4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:36.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.946+0000 7f59c159a700 1 -- 172.21.15.98:0/2959350484 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b4004230 con 0x7f59c40b8910 2023-03-03T07:58:36.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.946+0000 7f59c159a700 1 -- 172.21.15.98:0/2959350484 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b4004390 con 0x7f59c40b8910 2023-03-03T07:58:36.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.946+0000 7f59c159a700 1 -- 172.21.15.98:0/2959350484 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b40036a0 con 0x7f59c40b8910 2023-03-03T07:58:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.946+0000 7f59c359e700 1 -- 172.21.15.98:0/2959350484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c40b8910 msgr2=0x7f59c41604a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.946+0000 7f59c359e700 1 --2- 172.21.15.98:0/2959350484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c40b8910 0x7f59c41604a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f59c4074810 tx=0x7f59b4009bd0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.946+0000 7f59c359e700 1 -- 172.21.15.98:0/2959350484 shutdown_connections 2023-03-03T07:58:36.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.946+0000 7f59c359e700 1 --2- 172.21.15.98:0/2959350484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c40b8910 0x7f59c41604a0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.946+0000 7f59c359e700 1 --2- 172.21.15.98:0/2959350484 >> v2:172.21.15.100:3300/0 conn(0x7f59c40b5e70 0x7f59c40b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.946+0000 7f59c359e700 1 -- 172.21.15.98:0/2959350484 >> 172.21.15.98:0/2959350484 conn(0x7f59c406b880 msgr2=0x7f59c40cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:36.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.946+0000 7f59c359e700 1 -- 172.21.15.98:0/2959350484 shutdown_connections 2023-03-03T07:58:36.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.946+0000 7f59c359e700 1 -- 172.21.15.98:0/2959350484 wait complete. 2023-03-03T07:58:36.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.949+0000 7f59c359e700 1 Processor -- start 2023-03-03T07:58:36.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.950+0000 7f59c359e700 1 -- start start 2023-03-03T07:58:36.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.951+0000 7f59c359e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f59c40b5e70 0x7f59c40c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:36.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.951+0000 7f59c359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c40b8910 0x7f59c40cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:36.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.951+0000 7f59c359e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f59c40cf570 0x7f59c414ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:36.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.951+0000 7f59c359e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c40ca690 con 0x7f59c40b8910 2023-03-03T07:58:36.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.951+0000 7f59c359e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c40ca970 con 0x7f59c40cf570 2023-03-03T07:58:36.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.951+0000 7f59c359e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c40cac50 con 0x7f59c40b5e70 2023-03-03T07:58:36.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.951+0000 7f59c1d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c40b8910 0x7f59c40cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:36.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.951+0000 7f59c1d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c40b8910 0x7f59c40cb730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51464/0 (socket says 172.21.15.98:51464) 2023-03-03T07:58:36.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.951+0000 7f59c1d9b700 1 -- 172.21.15.98:0/3647377424 learned_addr learned my addr 172.21.15.98:0/3647377424 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:36.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.951+0000 7f59c259c700 1 --2- 172.21.15.98:0/3647377424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f59c40b5e70 0x7f59c40c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:36.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.951+0000 7f59c2d9d700 1 --2- 172.21.15.98:0/3647377424 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f59c40cf570 0x7f59c414ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:36.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.952+0000 7f59c1d9b700 1 -- 172.21.15.98:0/3647377424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f59c40b5e70 msgr2=0x7f59c40c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:36.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.952+0000 7f59c1d9b700 1 --2- 172.21.15.98:0/3647377424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f59c40b5e70 0x7f59c40c8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.952+0000 7f59c1d9b700 1 -- 172.21.15.98:0/3647377424 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f59c40cf570 msgr2=0x7f59c414ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:36.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.952+0000 7f59c1d9b700 1 --2- 172.21.15.98:0/3647377424 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f59c40cf570 0x7f59c414ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:36.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.952+0000 7f59c1d9b700 1 -- 172.21.15.98:0/3647377424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b4009550 con 0x7f59c40b8910 2023-03-03T07:58:36.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.952+0000 7f59c259c700 1 --2- 172.21.15.98:0/3647377424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f59c40b5e70 0x7f59c40c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:36.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.952+0000 7f59c1d9b700 1 --2- 172.21.15.98:0/3647377424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c40b8910 0x7f59c40cb730 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f59b800e810 tx=0x7f59b800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:36.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.952+0000 7f59b2ffd700 1 -- 172.21.15.98:0/3647377424 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8013610 con 0x7f59c40b8910 2023-03-03T07:58:36.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.953+0000 7f59b2ffd700 1 -- 172.21.15.98:0/3647377424 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b8013770 con 0x7f59c40b8910 2023-03-03T07:58:36.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.953+0000 7f59b2ffd700 1 -- 172.21.15.98:0/3647377424 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8013b20 con 0x7f59c40b8910 2023-03-03T07:58:36.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.953+0000 7f59c359e700 1 -- 172.21.15.98:0/3647377424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c41612b0 con 0x7f59c40b8910 2023-03-03T07:58:36.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.953+0000 7f59c359e700 1 -- 172.21.15.98:0/3647377424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c4160fd0 con 0x7f59c40b8910 2023-03-03T07:58:36.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.954+0000 7f59c359e700 1 -- 172.21.15.98:0/3647377424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c40d0eb0 con 0x7f59c40b8910 2023-03-03T07:58:36.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.955+0000 7f59b2ffd700 1 -- 172.21.15.98:0/3647377424 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f59b8023070 con 0x7f59c40b8910 2023-03-03T07:58:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.957+0000 7f59b2ffd700 1 --2- 172.21.15.98:0/3647377424 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f59a40683c0 0x7f59a406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.957+0000 7f59b2ffd700 1 -- 172.21.15.98:0/3647377424 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f59b8085b70 con 0x7f59c40b8910 2023-03-03T07:58:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.957+0000 7f59c259c700 1 --2- 172.21.15.98:0/3647377424 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f59a40683c0 0x7f59a406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:36.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.958+0000 7f59c259c700 1 --2- 172.21.15.98:0/3647377424 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f59a40683c0 0x7f59a406a870 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009130 tx=0x7f59b4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:36.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:36.961+0000 7f59b2ffd700 1 -- 172.21.15.98:0/3647377424 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f59b8052150 con 0x7f59c40b8910 2023-03-03T07:58:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.155+0000 7f59c359e700 1 -- 172.21.15.98:0/3647377424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f59c4068290 con 0x7f59c40b8910 2023-03-03T07:58:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.155+0000 7f59b2ffd700 1 -- 172.21.15.98:0/3647377424 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7f59b8055770 con 0x7f59c40b8910 2023-03-03T07:58:37.157 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:58:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.158+0000 7f59c359e700 1 -- 172.21.15.98:0/3647377424 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f59a40683c0 msgr2=0x7f59a406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.158+0000 7f59c359e700 1 --2- 172.21.15.98:0/3647377424 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f59a40683c0 0x7f59a406a870 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009130 tx=0x7f59b4008030 comp rx=0 tx=0).stop 2023-03-03T07:58:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.158+0000 7f59c359e700 1 -- 172.21.15.98:0/3647377424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c40b8910 msgr2=0x7f59c40cb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.158+0000 7f59c359e700 1 --2- 172.21.15.98:0/3647377424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c40b8910 0x7f59c40cb730 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f59b800e810 tx=0x7f59b800eb20 comp rx=0 tx=0).stop 2023-03-03T07:58:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.158+0000 7f59c359e700 1 -- 172.21.15.98:0/3647377424 shutdown_connections 2023-03-03T07:58:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.158+0000 7f59c359e700 1 --2- 172.21.15.98:0/3647377424 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f59a40683c0 0x7f59a406a870 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.158+0000 7f59c359e700 1 --2- 172.21.15.98:0/3647377424 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f59c40cf570 0x7f59c414ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.158+0000 7f59c359e700 1 --2- 172.21.15.98:0/3647377424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f59c40b5e70 0x7f59c40c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.158+0000 7f59c359e700 1 --2- 172.21.15.98:0/3647377424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59c40b8910 0x7f59c40cb730 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.158+0000 7f59c359e700 1 -- 172.21.15.98:0/3647377424 >> 172.21.15.98:0/3647377424 conn(0x7f59c406b880 msgr2=0x7f59c40d2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.158+0000 7f59c359e700 1 -- 172.21.15.98:0/3647377424 shutdown_connections 2023-03-03T07:58:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.158+0000 7f59c359e700 1 -- 172.21.15.98:0/3647377424 wait complete. 2023-03-03T07:58:37.186 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:58:37.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.436+0000 7fae2ef9d700 1 Processor -- start 2023-03-03T07:58:37.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.436+0000 7fae2ef9d700 1 -- start start 2023-03-03T07:58:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.438+0000 7fae2ef9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae281609b0 0x7fae28160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.438+0000 7fae2ef9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae280cf570 0x7fae2815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.438+0000 7fae2ef9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae281612c0 con 0x7fae2815db30 2023-03-03T07:58:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.438+0000 7fae2ef9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae280cebb0 con 0x7fae280cf570 2023-03-03T07:58:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.438+0000 7fae2ef9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae280cd780 con 0x7fae281609b0 2023-03-03T07:58:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.438+0000 7fae2d79a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae280cf570 0x7fae2815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.438+0000 7fae2d79a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae280cf570 0x7fae2815d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51476/0 (socket says 172.21.15.98:51476) 2023-03-03T07:58:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.438+0000 7fae2d79a700 1 -- 172.21.15.98:0/4010175124 learned_addr learned my addr 172.21.15.98:0/4010175124 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.438+0000 7fae2e79c700 1 --1- 172.21.15.98:0/4010175124 >> v1:172.21.15.98:6789/0 conn(0x7fae2815db30 0x7fae280b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48060/0 (socket says 172.21.15.98:48060) 2023-03-03T07:58:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.439+0000 7fae2df9b700 1 --2- 172.21.15.98:0/4010175124 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae281609b0 0x7fae28160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.439+0000 7fae2d79a700 1 -- 172.21.15.98:0/4010175124 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae281609b0 msgr2=0x7fae28160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.439+0000 7fae2d79a700 1 --2- 172.21.15.98:0/4010175124 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae281609b0 0x7fae28160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.439+0000 7fae2d79a700 1 -- 172.21.15.98:0/4010175124 >> v1:172.21.15.98:6789/0 conn(0x7fae2815db30 legacy=0x7fae280b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.439+0000 7fae2d79a700 1 -- 172.21.15.98:0/4010175124 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae280cda60 con 0x7fae280cf570 2023-03-03T07:58:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.439+0000 7fae2d79a700 1 --2- 172.21.15.98:0/4010175124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae280cf570 0x7fae2815d5f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fae180098f0 tx=0x7fae18009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7181e82c499aa76 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.439+0000 7fae2cf99700 1 -- 172.21.15.98:0/4010175124 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1800b8a0 con 0x7fae280cf570 2023-03-03T07:58:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.440+0000 7fae2cf99700 1 -- 172.21.15.98:0/4010175124 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1800ba00 con 0x7fae280cf570 2023-03-03T07:58:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.440+0000 7fae2cf99700 1 -- 172.21.15.98:0/4010175124 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1800bd90 con 0x7fae280cf570 2023-03-03T07:58:37.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.440+0000 7fae2ef9d700 1 -- 172.21.15.98:0/4010175124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae280cf570 msgr2=0x7fae2815d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.440+0000 7fae2ef9d700 1 --2- 172.21.15.98:0/4010175124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae280cf570 0x7fae2815d5f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fae180098f0 tx=0x7fae18009c00 comp rx=0 tx=0).stop 2023-03-03T07:58:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.440+0000 7fae2ef9d700 1 -- 172.21.15.98:0/4010175124 shutdown_connections 2023-03-03T07:58:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.440+0000 7fae2ef9d700 1 --2- 172.21.15.98:0/4010175124 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae281609b0 0x7fae28160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.440+0000 7fae2ef9d700 1 --2- 172.21.15.98:0/4010175124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae280cf570 0x7fae2815d5f0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.440+0000 7fae2ef9d700 1 -- 172.21.15.98:0/4010175124 >> 172.21.15.98:0/4010175124 conn(0x7fae2806b880 msgr2=0x7fae280cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.440+0000 7fae2ef9d700 1 -- 172.21.15.98:0/4010175124 shutdown_connections 2023-03-03T07:58:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.440+0000 7fae2ef9d700 1 -- 172.21.15.98:0/4010175124 wait complete. 2023-03-03T07:58:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.442+0000 7fae2ef9d700 1 Processor -- start 2023-03-03T07:58:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.442+0000 7fae2ef9d700 1 -- start start 2023-03-03T07:58:37.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.443+0000 7fae2ef9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae280cf570 0x7fae28148380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:37.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.443+0000 7fae2ef9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae2815db30 0x7fae280c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:37.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.444+0000 7fae2ef9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae281609b0 0x7fae280cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.444+0000 7fae2ef9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae280c79a0 con 0x7fae281609b0 2023-03-03T07:58:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.444+0000 7fae2ef9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae280c8600 con 0x7fae2815db30 2023-03-03T07:58:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.444+0000 7fae2ef9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae280c88e0 con 0x7fae280cf570 2023-03-03T07:58:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.444+0000 7fae2e79c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae281609b0 0x7fae280cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.444+0000 7fae2e79c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae281609b0 0x7fae280cbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51484/0 (socket says 172.21.15.98:51484) 2023-03-03T07:58:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.444+0000 7fae2e79c700 1 -- 172.21.15.98:0/2827954173 learned_addr learned my addr 172.21.15.98:0/2827954173 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.444+0000 7fae2df9b700 1 --2- 172.21.15.98:0/2827954173 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae280cf570 0x7fae28148380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.444+0000 7fae2d79a700 1 --2- 172.21.15.98:0/2827954173 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae2815db30 0x7fae280c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.444+0000 7fae2e79c700 1 -- 172.21.15.98:0/2827954173 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae280cf570 msgr2=0x7fae28148380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.444+0000 7fae2e79c700 1 --2- 172.21.15.98:0/2827954173 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae280cf570 0x7fae28148380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.444+0000 7fae2e79c700 1 -- 172.21.15.98:0/2827954173 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae2815db30 msgr2=0x7fae280c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.444+0000 7fae2e79c700 1 --2- 172.21.15.98:0/2827954173 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae2815db30 0x7fae280c93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.444+0000 7fae2e79c700 1 -- 172.21.15.98:0/2827954173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae18009550 con 0x7fae281609b0 2023-03-03T07:58:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.445+0000 7fae2e79c700 1 --2- 172.21.15.98:0/2827954173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae281609b0 0x7fae280cbb10 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fae2000cbd0 tx=0x7fae2000cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.445+0000 7fae1e7fc700 1 -- 172.21.15.98:0/2827954173 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2000aaa0 con 0x7fae281609b0 2023-03-03T07:58:37.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.445+0000 7fae1e7fc700 1 -- 172.21.15.98:0/2827954173 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2000ac00 con 0x7fae281609b0 2023-03-03T07:58:37.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.445+0000 7fae1e7fc700 1 -- 172.21.15.98:0/2827954173 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae20007070 con 0x7fae281609b0 2023-03-03T07:58:37.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.445+0000 7fae2ef9d700 1 -- 172.21.15.98:0/2827954173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2815cdf0 con 0x7fae281609b0 2023-03-03T07:58:37.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.445+0000 7fae2ef9d700 1 -- 172.21.15.98:0/2827954173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae280cdd40 con 0x7fae281609b0 2023-03-03T07:58:37.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.446+0000 7fae2ef9d700 1 -- 172.21.15.98:0/2827954173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae2815d0d0 con 0x7fae281609b0 2023-03-03T07:58:37.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.448+0000 7fae1e7fc700 1 -- 172.21.15.98:0/2827954173 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fae20019030 con 0x7fae281609b0 2023-03-03T07:58:37.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.449+0000 7fae1e7fc700 1 --2- 172.21.15.98:0/2827954173 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae0c0683c0 0x7fae0c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.449+0000 7fae1e7fc700 1 -- 172.21.15.98:0/2827954173 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fae20085f40 con 0x7fae281609b0 2023-03-03T07:58:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.449+0000 7fae2df9b700 1 --2- 172.21.15.98:0/2827954173 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae0c0683c0 0x7fae0c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.450+0000 7fae2df9b700 1 --2- 172.21.15.98:0/2827954173 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae0c0683c0 0x7fae0c06a870 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fae24006b20 tx=0x7fae24008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.451+0000 7fae1e7fc700 1 -- 172.21.15.98:0/2827954173 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fae20052670 con 0x7fae281609b0 2023-03-03T07:58:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.646+0000 7fae2ef9d700 1 -- 172.21.15.98:0/2827954173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fae28068290 con 0x7fae281609b0 2023-03-03T07:58:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.647+0000 7fae1e7fc700 1 -- 172.21.15.98:0/2827954173 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7fae20017020 con 0x7fae281609b0 2023-03-03T07:58:37.648 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:58:37.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.649+0000 7fae2ef9d700 1 -- 172.21.15.98:0/2827954173 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae0c0683c0 msgr2=0x7fae0c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:37.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.649+0000 7fae2ef9d700 1 --2- 172.21.15.98:0/2827954173 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae0c0683c0 0x7fae0c06a870 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fae24006b20 tx=0x7fae24008040 comp rx=0 tx=0).stop 2023-03-03T07:58:37.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.649+0000 7fae2ef9d700 1 -- 172.21.15.98:0/2827954173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae281609b0 msgr2=0x7fae280cbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:37.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.649+0000 7fae2ef9d700 1 --2- 172.21.15.98:0/2827954173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae281609b0 0x7fae280cbb10 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fae2000cbd0 tx=0x7fae2000cee0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.649+0000 7fae2ef9d700 1 -- 172.21.15.98:0/2827954173 shutdown_connections 2023-03-03T07:58:37.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.649+0000 7fae2ef9d700 1 --2- 172.21.15.98:0/2827954173 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae0c0683c0 0x7fae0c06a870 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.649+0000 7fae2ef9d700 1 --2- 172.21.15.98:0/2827954173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae281609b0 0x7fae280cbb10 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.649+0000 7fae2ef9d700 1 --2- 172.21.15.98:0/2827954173 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae280cf570 0x7fae28148380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.649+0000 7fae2ef9d700 1 --2- 172.21.15.98:0/2827954173 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae2815db30 0x7fae280c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.649+0000 7fae2ef9d700 1 -- 172.21.15.98:0/2827954173 >> 172.21.15.98:0/2827954173 conn(0x7fae2806b880 msgr2=0x7fae2815e580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:37.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.649+0000 7fae2ef9d700 1 -- 172.21.15.98:0/2827954173 shutdown_connections 2023-03-03T07:58:37.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.649+0000 7fae2ef9d700 1 -- 172.21.15.98:0/2827954173 wait complete. 2023-03-03T07:58:37.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T07:58:37.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.923+0000 7f3fd09ff700 1 Processor -- start 2023-03-03T07:58:37.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.924+0000 7f3fd09ff700 1 -- start start 2023-03-03T07:58:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.925+0000 7f3fd09ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3fcc0d0270 0x7f3fcc0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.925+0000 7f3fd09ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fcc0cf980 con 0x7f3fcc15ca70 2023-03-03T07:58:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.925+0000 7f3fd09ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fcc0d1800 con 0x7f3fcc06b4b0 2023-03-03T07:58:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.925+0000 7f3fd09ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0cd5b0 con 0x7f3fcc0d0270 2023-03-03T07:58:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.925+0000 7f3fca59c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3fcc15ca70 0x7f3fcc0cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48064/0 (socket says 172.21.15.98:48064) 2023-03-03T07:58:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.925+0000 7f3fca59c700 1 -- 172.21.15.98:0/1882119235 learned_addr learned my addr 172.21.15.98:0/1882119235 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.926+0000 7f3fc9d9b700 1 --2- 172.21.15.98:0/1882119235 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3fcc0d0270 0x7f3fcc0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.926+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429868174 0 0) 0x7f3fcc0cf980 con 0x7f3fcc15ca70 2023-03-03T07:58:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.926+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fb40035b0 con 0x7f3fcc15ca70 2023-03-03T07:58:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.926+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126225140 0 0) 0x7f3fcc0d1800 con 0x7f3fcc06b4b0 2023-03-03T07:58:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.926+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fcc0cf980 con 0x7f3fcc06b4b0 2023-03-03T07:58:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.926+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4003395622 0 0) 0x7f3fb40035b0 con 0x7f3fcc15ca70 2023-03-03T07:58:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.926+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3fcc0d0270 msgr2=0x7f3fcc0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.926+0000 7f3fc959a700 1 --2- 172.21.15.98:0/1882119235 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3fcc0d0270 0x7f3fcc0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.926+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 >> v1:172.21.15.100:6789/0 conn(0x7f3fcc06b4b0 legacy=0x7f3fcc0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.926+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc0cd900 con 0x7f3fcc15ca70 2023-03-03T07:58:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.926+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f3fbc001670 con 0x7f3fcc15ca70 2023-03-03T07:58:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.927+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 >> v1:172.21.15.98:6789/0 conn(0x7f3fcc15ca70 legacy=0x7f3fcc0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.927+0000 7f3fc959a700 1 --2- 172.21.15.98:0/1882119235 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3fb4003bc0 0x7f3fb4003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.927+0000 7f3fc959a700 1 --2- 172.21.15.98:0/1882119235 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3fb4004850 0x7f3fb4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:37.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.927+0000 7f3fc959a700 1 --2- 172.21.15.98:0/1882119235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fb4005200 0x7f3fb400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:37.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.927+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0cd5b0 con 0x7f3fb4005200 2023-03-03T07:58:37.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.927+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb400c0e0 con 0x7f3fb4004850 2023-03-03T07:58:37.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.927+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb400c270 con 0x7f3fb4003bc0 2023-03-03T07:58:37.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.927+0000 7f3fcad9d700 1 --2- 172.21.15.98:0/1882119235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fb4005200 0x7f3fb400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:37.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.927+0000 7f3fca59c700 1 --2- 172.21.15.98:0/1882119235 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3fb4003bc0 0x7f3fb4003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:37.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.927+0000 7f3fc9d9b700 1 --2- 172.21.15.98:0/1882119235 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3fb4004850 0x7f3fb4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:37.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.927+0000 7f3fcad9d700 1 -- 172.21.15.98:0/1882119235 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3fb4003bc0 msgr2=0x7f3fb4003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:37.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.927+0000 7f3fcad9d700 1 --2- 172.21.15.98:0/1882119235 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3fb4003bc0 0x7f3fb4003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.927+0000 7f3fcad9d700 1 -- 172.21.15.98:0/1882119235 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3fb4004850 msgr2=0x7f3fb4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:37.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.927+0000 7f3fcad9d700 1 --2- 172.21.15.98:0/1882119235 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3fb4004850 0x7f3fb4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.927+0000 7f3fcad9d700 1 -- 172.21.15.98:0/1882119235 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb40035b0 con 0x7f3fb4005200 2023-03-03T07:58:37.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.928+0000 7f3fcad9d700 1 --2- 172.21.15.98:0/1882119235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fb4005200 0x7f3fb400ba10 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400c4c0 tx=0x7f3fb4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6905ccc6d37878d9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:37.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.928+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4003a00 con 0x7f3fb4005200 2023-03-03T07:58:37.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.928+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc4003b60 con 0x7f3fb4005200 2023-03-03T07:58:37.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.928+0000 7f3fc959a700 1 -- 172.21.15.98:0/1882119235 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc40050d0 con 0x7f3fb4005200 2023-03-03T07:58:37.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.928+0000 7f3fd09ff700 1 -- 172.21.15.98:0/1882119235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fb4005200 msgr2=0x7f3fb400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:37.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.928+0000 7f3fd09ff700 1 --2- 172.21.15.98:0/1882119235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fb4005200 0x7f3fb400ba10 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400c4c0 tx=0x7f3fb4005880 comp rx=0 tx=0).stop 2023-03-03T07:58:37.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.929+0000 7f3fcad9d700 1 -- 172.21.15.98:0/1882119235 reap_dead start 2023-03-03T07:58:37.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.929+0000 7f3fd09ff700 1 -- 172.21.15.98:0/1882119235 shutdown_connections 2023-03-03T07:58:37.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.929+0000 7f3fd09ff700 1 -- 172.21.15.98:0/1882119235 >> 172.21.15.98:0/1882119235 conn(0x7f3fcc073e70 msgr2=0x7f3fcc06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:37.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.929+0000 7f3fd09ff700 1 -- 172.21.15.98:0/1882119235 shutdown_connections 2023-03-03T07:58:37.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.929+0000 7f3fd09ff700 1 -- 172.21.15.98:0/1882119235 wait complete. 2023-03-03T07:58:37.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.931+0000 7f3fd09ff700 1 Processor -- start 2023-03-03T07:58:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.931+0000 7f3fd09ff700 1 -- start start 2023-03-03T07:58:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.932+0000 7f3fd09ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3fb4005200 0x7f3fcc1593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.932+0000 7f3fd09ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fcc0a9e40 0x7f3fcc0aa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.932+0000 7f3fd09ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3fcc0b4050 0x7f3fcc0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:37.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.932+0000 7f3fd09ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0aa7f0 con 0x7f3fcc0a9e40 2023-03-03T07:58:37.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.932+0000 7f3fd09ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0b4a10 con 0x7f3fcc0b4050 2023-03-03T07:58:37.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.932+0000 7f3fd09ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0b58b0 con 0x7f3fb4005200 2023-03-03T07:58:37.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.932+0000 7f3fc9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fcc0a9e40 0x7f3fcc0aa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:37.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.932+0000 7f3fc9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fcc0a9e40 0x7f3fcc0aa2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51508/0 (socket says 172.21.15.98:51508) 2023-03-03T07:58:37.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.932+0000 7f3fc9d9b700 1 -- 172.21.15.98:0/3516643264 learned_addr learned my addr 172.21.15.98:0/3516643264 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:37.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.933+0000 7f3fca59c700 1 --2- 172.21.15.98:0/3516643264 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3fb4005200 0x7f3fcc1593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:37.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.933+0000 7f3fcad9d700 1 --2- 172.21.15.98:0/3516643264 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3fcc0b4050 0x7f3fcc0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:37.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.933+0000 7f3fc9d9b700 1 -- 172.21.15.98:0/3516643264 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3fb4005200 msgr2=0x7f3fcc1593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:37.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.933+0000 7f3fc9d9b700 1 --2- 172.21.15.98:0/3516643264 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3fb4005200 0x7f3fcc1593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.933+0000 7f3fc9d9b700 1 -- 172.21.15.98:0/3516643264 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3fcc0b4050 msgr2=0x7f3fcc0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:37.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.933+0000 7f3fc9d9b700 1 --2- 172.21.15.98:0/3516643264 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3fcc0b4050 0x7f3fcc0b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:37.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.933+0000 7f3fc9d9b700 1 -- 172.21.15.98:0/3516643264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc0d1aa0 con 0x7f3fcc0a9e40 2023-03-03T07:58:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.933+0000 7f3fc9d9b700 1 --2- 172.21.15.98:0/3516643264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fcc0a9e40 0x7f3fcc0aa2b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0011960 tx=0x7f3fc0011c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.933+0000 7f3fbaffd700 1 -- 172.21.15.98:0/3516643264 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0004300 con 0x7f3fcc0a9e40 2023-03-03T07:58:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.934+0000 7f3fbaffd700 1 -- 172.21.15.98:0/3516643264 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc0004460 con 0x7f3fcc0a9e40 2023-03-03T07:58:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.934+0000 7f3fbaffd700 1 -- 172.21.15.98:0/3516643264 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc00038e0 con 0x7f3fcc0a9e40 2023-03-03T07:58:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.934+0000 7f3fd09ff700 1 -- 172.21.15.98:0/3516643264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fcc0cdbe0 con 0x7f3fcc0a9e40 2023-03-03T07:58:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.934+0000 7f3fd09ff700 1 -- 172.21.15.98:0/3516643264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fcc15ce40 con 0x7f3fcc0a9e40 2023-03-03T07:58:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.935+0000 7f3fd09ff700 1 -- 172.21.15.98:0/3516643264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fcc159900 con 0x7f3fcc0a9e40 2023-03-03T07:58:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.936+0000 7f3fbaffd700 1 -- 172.21.15.98:0/3516643264 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f3fc0025070 con 0x7f3fcc0a9e40 2023-03-03T07:58:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.938+0000 7f3fbaffd700 1 --2- 172.21.15.98:0/3516643264 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3fac0683c0 0x7f3fac06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.938+0000 7f3fbaffd700 1 -- 172.21.15.98:0/3516643264 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f3fc0085fb0 con 0x7f3fcc0a9e40 2023-03-03T07:58:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.938+0000 7f3fca59c700 1 --2- 172.21.15.98:0/3516643264 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3fac0683c0 0x7f3fac06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.939+0000 7f3fca59c700 1 --2- 172.21.15.98:0/3516643264 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3fac0683c0 0x7f3fac06a870 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc15ac20 tx=0x7f3fbc001040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:37.943+0000 7f3fbaffd700 1 -- 172.21.15.98:0/3516643264 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3fcc159900 con 0x7f3fcc0a9e40 2023-03-03T07:58:38.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.149+0000 7f3fd09ff700 1 -- 172.21.15.98:0/3516643264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3fcc159900 con 0x7f3fcc0a9e40 2023-03-03T07:58:38.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.150+0000 7f3fbaffd700 1 -- 172.21.15.98:0/3516643264 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v54) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3fcc159900 con 0x7f3fcc0a9e40 2023-03-03T07:58:38.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.152+0000 7f3fd09ff700 1 -- 172.21.15.98:0/3516643264 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3fac0683c0 msgr2=0x7f3fac06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:38.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.152+0000 7f3fd09ff700 1 --2- 172.21.15.98:0/3516643264 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3fac0683c0 0x7f3fac06a870 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc15ac20 tx=0x7f3fbc001040 comp rx=0 tx=0).stop 2023-03-03T07:58:38.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.152+0000 7f3fd09ff700 1 -- 172.21.15.98:0/3516643264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fcc0a9e40 msgr2=0x7f3fcc0aa2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:38.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.152+0000 7f3fd09ff700 1 --2- 172.21.15.98:0/3516643264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fcc0a9e40 0x7f3fcc0aa2b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0011960 tx=0x7f3fc0011c70 comp rx=0 tx=0).stop 2023-03-03T07:58:38.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.153+0000 7f3fd09ff700 1 -- 172.21.15.98:0/3516643264 shutdown_connections 2023-03-03T07:58:38.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.153+0000 7f3fd09ff700 1 --2- 172.21.15.98:0/3516643264 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3fac0683c0 0x7f3fac06a870 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.153+0000 7f3fd09ff700 1 --2- 172.21.15.98:0/3516643264 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3fcc0b4050 0x7f3fcc0b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.153+0000 7f3fd09ff700 1 --2- 172.21.15.98:0/3516643264 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3fb4005200 0x7f3fcc1593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.153+0000 7f3fd09ff700 1 --2- 172.21.15.98:0/3516643264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fcc0a9e40 0x7f3fcc0aa2b0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.153+0000 7f3fd09ff700 1 -- 172.21.15.98:0/3516643264 >> 172.21.15.98:0/3516643264 conn(0x7f3fcc073e70 msgr2=0x7f3fcc0d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:38.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.153+0000 7f3fd09ff700 1 -- 172.21.15.98:0/3516643264 shutdown_connections 2023-03-03T07:58:38.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.153+0000 7f3fd09ff700 1 -- 172.21.15.98:0/3516643264 wait complete. 2023-03-03T07:58:38.156 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T07:58:38.181 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T07:58:38.181 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T07:58:38.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T07:58:38.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.498+0000 7f2beeb9d700 1 Processor -- start 2023-03-03T07:58:38.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.498+0000 7f2beeb9d700 1 -- start start 2023-03-03T07:58:38.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.500+0000 7f2beeb9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2be80b51c0 0x7f2be8160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:38.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.500+0000 7f2beeb9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2be80d1580 0x7f2be80d19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:38.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.500+0000 7f2beeb9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be81614c0 con 0x7f2be8146360 2023-03-03T07:58:38.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.500+0000 7f2beeb9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be80d10f0 con 0x7f2be80d1580 2023-03-03T07:58:38.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.500+0000 7f2beeb9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80cdf20 con 0x7f2be80b51c0 2023-03-03T07:58:38.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.501+0000 7f2bee39c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2be80d1580 0x7f2be80d19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:38.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.501+0000 7f2bee39c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2be80d1580 0x7f2be80d19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51516/0 (socket says 172.21.15.98:51516) 2023-03-03T07:58:38.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.501+0000 7f2bee39c700 1 -- 172.21.15.98:0/691985017 learned_addr learned my addr 172.21.15.98:0/691985017 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:38.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.501+0000 7f2bed39a700 1 --2- 172.21.15.98:0/691985017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2be80b51c0 0x7f2be8160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.501+0000 7f2bee39c700 1 -- 172.21.15.98:0/691985017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2be80b51c0 msgr2=0x7f2be8160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.501+0000 7f2bee39c700 1 --2- 172.21.15.98:0/691985017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2be80b51c0 0x7f2be8160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.501+0000 7f2bee39c700 1 -- 172.21.15.98:0/691985017 >> v1:172.21.15.100:6789/0 conn(0x7f2be8146360 legacy=0x7f2be80b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.501+0000 7f2bee39c700 1 -- 172.21.15.98:0/691985017 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80cd680 con 0x7f2be80d1580 2023-03-03T07:58:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.501+0000 7f2bee39c700 1 --2- 172.21.15.98:0/691985017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2be80d1580 0x7f2be80d19e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009e50 tx=0x7f2be000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1af9ee31b14c9a8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:38.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.502+0000 7f2becb99700 1 -- 172.21.15.98:0/691985017 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be000e070 con 0x7f2be80d1580 2023-03-03T07:58:38.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.502+0000 7f2becb99700 1 -- 172.21.15.98:0/691985017 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be000b910 con 0x7f2be80d1580 2023-03-03T07:58:38.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.502+0000 7f2becb99700 1 -- 172.21.15.98:0/691985017 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be000bcc0 con 0x7f2be80d1580 2023-03-03T07:58:38.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.502+0000 7f2beeb9d700 1 -- 172.21.15.98:0/691985017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2be80d1580 msgr2=0x7f2be80d19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:38.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.502+0000 7f2beeb9d700 1 --2- 172.21.15.98:0/691985017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2be80d1580 0x7f2be80d19e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009e50 tx=0x7f2be000b330 comp rx=0 tx=0).stop 2023-03-03T07:58:38.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.502+0000 7f2beeb9d700 1 -- 172.21.15.98:0/691985017 shutdown_connections 2023-03-03T07:58:38.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.502+0000 7f2beeb9d700 1 --2- 172.21.15.98:0/691985017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2be80d1580 0x7f2be80d19e0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.502+0000 7f2beeb9d700 1 --2- 172.21.15.98:0/691985017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2be80b51c0 0x7f2be8160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.502+0000 7f2beeb9d700 1 -- 172.21.15.98:0/691985017 >> 172.21.15.98:0/691985017 conn(0x7f2be806b8e0 msgr2=0x7f2be80cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:38.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.502+0000 7f2beeb9d700 1 -- 172.21.15.98:0/691985017 shutdown_connections 2023-03-03T07:58:38.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.503+0000 7f2beeb9d700 1 -- 172.21.15.98:0/691985017 wait complete. 2023-03-03T07:58:38.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.504+0000 7f2beeb9d700 1 Processor -- start 2023-03-03T07:58:38.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.505+0000 7f2beeb9d700 1 -- start start 2023-03-03T07:58:38.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2beeb9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2be80b51c0 0x7f2be80c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:38.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2beeb9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2be80d1580 0x7f2be80c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:38.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2beeb9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2be8146360 0x7f2be80ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:38.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2beeb9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80c66d0 con 0x7f2be80b51c0 2023-03-03T07:58:38.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2beeb9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80cd290 con 0x7f2be80d1580 2023-03-03T07:58:38.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2beeb9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80c75c0 con 0x7f2be8146360 2023-03-03T07:58:38.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2bedb9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2be80b51c0 0x7f2be80c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:38.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2bedb9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2be80b51c0 0x7f2be80c6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51524/0 (socket says 172.21.15.98:51524) 2023-03-03T07:58:38.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2bedb9b700 1 -- 172.21.15.98:0/813813915 learned_addr learned my addr 172.21.15.98:0/813813915 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:38.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2bed39a700 1 --2- 172.21.15.98:0/813813915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2be80d1580 0x7f2be80c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:38.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2bee39c700 1 --2- 172.21.15.98:0/813813915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2be8146360 0x7f2be80ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:38.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2bedb9b700 1 -- 172.21.15.98:0/813813915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2be8146360 msgr2=0x7f2be80ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:38.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2bedb9b700 1 --2- 172.21.15.98:0/813813915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2be8146360 0x7f2be80ccd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2bedb9b700 1 -- 172.21.15.98:0/813813915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2be80d1580 msgr2=0x7f2be80c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:38.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2bedb9b700 1 --2- 172.21.15.98:0/813813915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2be80d1580 0x7f2be80c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.506+0000 7f2bedb9b700 1 -- 172.21.15.98:0/813813915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be0009b30 con 0x7f2be80b51c0 2023-03-03T07:58:38.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.507+0000 7f2bee39c700 1 --2- 172.21.15.98:0/813813915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2be8146360 0x7f2be80ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:38.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.507+0000 7f2bedb9b700 1 --2- 172.21.15.98:0/813813915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2be80b51c0 0x7f2be80c6fe0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2be400c510 tx=0x7f2be400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:38.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.507+0000 7f2bde7fc700 1 -- 172.21.15.98:0/813813915 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be40052d0 con 0x7f2be80b51c0 2023-03-03T07:58:38.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.507+0000 7f2bde7fc700 1 -- 172.21.15.98:0/813813915 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be4005430 con 0x7f2be80b51c0 2023-03-03T07:58:38.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.507+0000 7f2bde7fc700 1 -- 172.21.15.98:0/813813915 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be40038c0 con 0x7f2be80b51c0 2023-03-03T07:58:38.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.507+0000 7f2beeb9d700 1 -- 172.21.15.98:0/813813915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be80cdc40 con 0x7f2be80b51c0 2023-03-03T07:58:38.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.507+0000 7f2beeb9d700 1 -- 172.21.15.98:0/813813915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be80cd960 con 0x7f2be80b51c0 2023-03-03T07:58:38.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.508+0000 7f2beeb9d700 1 -- 172.21.15.98:0/813813915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be80d0d80 con 0x7f2be80b51c0 2023-03-03T07:58:38.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.510+0000 7f2bde7fc700 1 -- 172.21.15.98:0/813813915 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f2be4019030 con 0x7f2be80b51c0 2023-03-03T07:58:38.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.511+0000 7f2bde7fc700 1 --2- 172.21.15.98:0/813813915 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2bcc0683c0 0x7f2bcc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:38.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.511+0000 7f2bde7fc700 1 -- 172.21.15.98:0/813813915 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f2be40850a0 con 0x7f2be80b51c0 2023-03-03T07:58:38.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.511+0000 7f2bed39a700 1 --2- 172.21.15.98:0/813813915 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2bcc0683c0 0x7f2bcc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:38.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.512+0000 7f2bed39a700 1 --2- 172.21.15.98:0/813813915 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2bcc0683c0 0x7f2bcc06a870 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2be80cc510 tx=0x7f2bd8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:38.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.515+0000 7f2bde7fc700 1 -- 172.21.15.98:0/813813915 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2be4051700 con 0x7f2be80b51c0 2023-03-03T07:58:38.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.708+0000 7f2beeb9d700 1 -- 172.21.15.98:0/813813915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2be81614c0 con 0x7f2be80b51c0 2023-03-03T07:58:38.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.710+0000 7f2bde7fc700 1 -- 172.21.15.98:0/813813915 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+18018 (secure 0 0 0) 0x7f2be81614c0 con 0x7f2be80b51c0 2023-03-03T07:58:38.711 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:38.711 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":54,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:58:28.723281+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:58:10.728830+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-03-03T07:58:12.224649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.6","mappings":[{"from":0,"to":1}]},{"pgid":"8.17","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T07:58:38.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.712+0000 7f2beeb9d700 1 -- 172.21.15.98:0/813813915 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2bcc0683c0 msgr2=0x7f2bcc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:38.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.712+0000 7f2beeb9d700 1 --2- 172.21.15.98:0/813813915 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2bcc0683c0 0x7f2bcc06a870 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2be80cc510 tx=0x7f2bd8008040 comp rx=0 tx=0).stop 2023-03-03T07:58:38.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.712+0000 7f2beeb9d700 1 -- 172.21.15.98:0/813813915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2be80b51c0 msgr2=0x7f2be80c6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:38.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.712+0000 7f2beeb9d700 1 --2- 172.21.15.98:0/813813915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2be80b51c0 0x7f2be80c6fe0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2be400c510 tx=0x7f2be400c8d0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.712+0000 7f2beeb9d700 1 -- 172.21.15.98:0/813813915 shutdown_connections 2023-03-03T07:58:38.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.712+0000 7f2beeb9d700 1 --2- 172.21.15.98:0/813813915 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2bcc0683c0 0x7f2bcc06a870 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.712+0000 7f2beeb9d700 1 --2- 172.21.15.98:0/813813915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2be80b51c0 0x7f2be80c6fe0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.712+0000 7f2beeb9d700 1 --2- 172.21.15.98:0/813813915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2be8146360 0x7f2be80ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.712+0000 7f2beeb9d700 1 --2- 172.21.15.98:0/813813915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2be80d1580 0x7f2be80c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.712+0000 7f2beeb9d700 1 -- 172.21.15.98:0/813813915 >> 172.21.15.98:0/813813915 conn(0x7f2be806b8e0 msgr2=0x7f2be80b21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:38.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.713+0000 7f2beeb9d700 1 -- 172.21.15.98:0/813813915 shutdown_connections 2023-03-03T07:58:38.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.713+0000 7f2beeb9d700 1 -- 172.21.15.98:0/813813915 wait complete. 2023-03-03T07:58:38.741 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:58:10.728830+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:58:12.224649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T07:58:38.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T07:58:38.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.990+0000 7f7f83fff700 1 Processor -- start 2023-03-03T07:58:38.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.991+0000 7f7f83fff700 1 -- start start 2023-03-03T07:58:38.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.992+0000 7f7f83fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7f841609b0 0x7f7f84160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:38.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.992+0000 7f7f83fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7f840cf570 0x7f7f8415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:38.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.992+0000 7f7f83fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f8415db30 0x7f7f840b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:38.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.992+0000 7f7f83fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f840cebb0 con 0x7f7f841609b0 2023-03-03T07:58:38.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.992+0000 7f7f83fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f841612c0 con 0x7f7f840cf570 2023-03-03T07:58:38.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.992+0000 7f7f83fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f840cd780 con 0x7f7f8415db30 2023-03-03T07:58:38.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.992+0000 7f7f82ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7f841609b0 0x7f7f84160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:38.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.993+0000 7f7f82ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7f841609b0 0x7f7f84160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51532/0 (socket says 172.21.15.98:51532) 2023-03-03T07:58:38.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.993+0000 7f7f82ffd700 1 -- 172.21.15.98:0/3257972549 learned_addr learned my addr 172.21.15.98:0/3257972549 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:38.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.993+0000 7f7f837fe700 1 --2- 172.21.15.98:0/3257972549 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f8415db30 0x7f7f840b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:38.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.993+0000 7f7f827fc700 1 --2- 172.21.15.98:0/3257972549 >> v2:172.21.15.100:3300/0 conn(0x7f7f840cf570 0x7f7f8415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:38.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.993+0000 7f7f82ffd700 1 -- 172.21.15.98:0/3257972549 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f8415db30 msgr2=0x7f7f840b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:38.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.993+0000 7f7f82ffd700 1 --2- 172.21.15.98:0/3257972549 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f8415db30 0x7f7f840b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.993+0000 7f7f82ffd700 1 -- 172.21.15.98:0/3257972549 >> v2:172.21.15.100:3300/0 conn(0x7f7f840cf570 msgr2=0x7f7f8415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:38.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.993+0000 7f7f82ffd700 1 --2- 172.21.15.98:0/3257972549 >> v2:172.21.15.100:3300/0 conn(0x7f7f840cf570 0x7f7f8415d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.993+0000 7f7f82ffd700 1 -- 172.21.15.98:0/3257972549 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f840cda60 con 0x7f7f841609b0 2023-03-03T07:58:38.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.993+0000 7f7f82ffd700 1 --2- 172.21.15.98:0/3257972549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f841609b0 0x7f7f84160d80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7f840cf0c0 tx=0x7f7f74009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35628fea5e235358 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:38.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.994+0000 7f7f81ffb700 1 -- 172.21.15.98:0/3257972549 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f74011830 con 0x7f7f841609b0 2023-03-03T07:58:38.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.994+0000 7f7f81ffb700 1 -- 172.21.15.98:0/3257972549 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f74011990 con 0x7f7f841609b0 2023-03-03T07:58:38.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.994+0000 7f7f81ffb700 1 -- 172.21.15.98:0/3257972549 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f74011d20 con 0x7f7f841609b0 2023-03-03T07:58:38.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.994+0000 7f7f83fff700 1 -- 172.21.15.98:0/3257972549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f841609b0 msgr2=0x7f7f84160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:38.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.994+0000 7f7f83fff700 1 --2- 172.21.15.98:0/3257972549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f841609b0 0x7f7f84160d80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7f840cf0c0 tx=0x7f7f74009bd0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.995+0000 7f7f83fff700 1 -- 172.21.15.98:0/3257972549 shutdown_connections 2023-03-03T07:58:38.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.995+0000 7f7f83fff700 1 --2- 172.21.15.98:0/3257972549 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f8415db30 0x7f7f840b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.995+0000 7f7f83fff700 1 --2- 172.21.15.98:0/3257972549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f841609b0 0x7f7f84160d80 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.995+0000 7f7f83fff700 1 --2- 172.21.15.98:0/3257972549 >> v2:172.21.15.100:3300/0 conn(0x7f7f840cf570 0x7f7f8415d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:38.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.995+0000 7f7f83fff700 1 -- 172.21.15.98:0/3257972549 >> 172.21.15.98:0/3257972549 conn(0x7f7f8406b880 msgr2=0x7f7f840cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:38.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.995+0000 7f7f83fff700 1 -- 172.21.15.98:0/3257972549 shutdown_connections 2023-03-03T07:58:38.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.995+0000 7f7f83fff700 1 -- 172.21.15.98:0/3257972549 wait complete. 2023-03-03T07:58:38.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.997+0000 7f7f83fff700 1 Processor -- start 2023-03-03T07:58:39.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.998+0000 7f7f83fff700 1 -- start start 2023-03-03T07:58:39.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.999+0000 7f7f83fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f840cf570 0x7f7f840d1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.999+0000 7f7f83fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f8415db30 0x7f7f840c8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.999+0000 7f7f83fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f841609b0 0x7f7f84151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.999+0000 7f7f83fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f840cada0 con 0x7f7f841609b0 2023-03-03T07:58:39.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.999+0000 7f7f83fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f840c7d40 con 0x7f7f840cf570 2023-03-03T07:58:39.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:38.999+0000 7f7f83fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f840c8020 con 0x7f7f8415db30 2023-03-03T07:58:39.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.000+0000 7f7f837fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f841609b0 0x7f7f84151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.000+0000 7f7f837fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f841609b0 0x7f7f84151070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51546/0 (socket says 172.21.15.98:51546) 2023-03-03T07:58:39.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.000+0000 7f7f837fe700 1 -- 172.21.15.98:0/1218261877 learned_addr learned my addr 172.21.15.98:0/1218261877 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:39.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.000+0000 7f7f82ffd700 1 --2- 172.21.15.98:0/1218261877 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f840cf570 0x7f7f840d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.000+0000 7f7f827fc700 1 --2- 172.21.15.98:0/1218261877 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f8415db30 0x7f7f840c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.000+0000 7f7f837fe700 1 -- 172.21.15.98:0/1218261877 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f8415db30 msgr2=0x7f7f840c8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.000+0000 7f7f837fe700 1 --2- 172.21.15.98:0/1218261877 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f8415db30 0x7f7f840c8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.000+0000 7f7f837fe700 1 -- 172.21.15.98:0/1218261877 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f840cf570 msgr2=0x7f7f840d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.000+0000 7f7f837fe700 1 --2- 172.21.15.98:0/1218261877 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f840cf570 0x7f7f840d1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.000+0000 7f7f837fe700 1 -- 172.21.15.98:0/1218261877 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f74009550 con 0x7f7f841609b0 2023-03-03T07:58:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.000+0000 7f7f837fe700 1 --2- 172.21.15.98:0/1218261877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f841609b0 0x7f7f84151070 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00efc0 tx=0x7f7f7c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.001+0000 7f7f6b7fe700 1 -- 172.21.15.98:0/1218261877 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c009ea0 con 0x7f7f841609b0 2023-03-03T07:58:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.001+0000 7f7f6b7fe700 1 -- 172.21.15.98:0/1218261877 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7c00f040 con 0x7f7f841609b0 2023-03-03T07:58:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.001+0000 7f7f6b7fe700 1 -- 172.21.15.98:0/1218261877 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c00ced0 con 0x7f7f841609b0 2023-03-03T07:58:39.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.001+0000 7f7f83fff700 1 -- 172.21.15.98:0/1218261877 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8415cdf0 con 0x7f7f841609b0 2023-03-03T07:58:39.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.001+0000 7f7f83fff700 1 -- 172.21.15.98:0/1218261877 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f840cdd40 con 0x7f7f841609b0 2023-03-03T07:58:39.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.002+0000 7f7f83fff700 1 -- 172.21.15.98:0/1218261877 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f8415d0d0 con 0x7f7f841609b0 2023-03-03T07:58:39.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.003+0000 7f7f6b7fe700 1 -- 172.21.15.98:0/1218261877 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f7f7c014480 con 0x7f7f841609b0 2023-03-03T07:58:39.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.004+0000 7f7f6b7fe700 1 --2- 172.21.15.98:0/1218261877 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7f700683f0 0x7f7f7006a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.004+0000 7f7f6b7fe700 1 -- 172.21.15.98:0/1218261877 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f7f7c086860 con 0x7f7f841609b0 2023-03-03T07:58:39.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.005+0000 7f7f82ffd700 1 --2- 172.21.15.98:0/1218261877 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7f700683f0 0x7f7f7006a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.005+0000 7f7f82ffd700 1 --2- 172.21.15.98:0/1218261877 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7f700683f0 0x7f7f7006a8a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7f74014040 tx=0x7f7f74008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:39.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.008+0000 7f7f6b7fe700 1 -- 172.21.15.98:0/1218261877 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7f7c052e60 con 0x7f7f841609b0 2023-03-03T07:58:39.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.201+0000 7f7f83fff700 1 -- 172.21.15.98:0/1218261877 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7f840c7ba0 con 0x7f7f841609b0 2023-03-03T07:58:39.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.202+0000 7f7f6b7fe700 1 -- 172.21.15.98:0/1218261877 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7f7c018020 con 0x7f7f841609b0 2023-03-03T07:58:39.203 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T07:58:39.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.204+0000 7f7f83fff700 1 -- 172.21.15.98:0/1218261877 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7f700683f0 msgr2=0x7f7f7006a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.204+0000 7f7f83fff700 1 --2- 172.21.15.98:0/1218261877 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7f700683f0 0x7f7f7006a8a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7f74014040 tx=0x7f7f74008030 comp rx=0 tx=0).stop 2023-03-03T07:58:39.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.204+0000 7f7f83fff700 1 -- 172.21.15.98:0/1218261877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f841609b0 msgr2=0x7f7f84151070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.204+0000 7f7f83fff700 1 --2- 172.21.15.98:0/1218261877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f841609b0 0x7f7f84151070 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00efc0 tx=0x7f7f7c00c3a0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.204+0000 7f7f83fff700 1 -- 172.21.15.98:0/1218261877 shutdown_connections 2023-03-03T07:58:39.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.204+0000 7f7f83fff700 1 --2- 172.21.15.98:0/1218261877 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7f700683f0 0x7f7f7006a8a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.204+0000 7f7f83fff700 1 --2- 172.21.15.98:0/1218261877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f841609b0 0x7f7f84151070 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.204+0000 7f7f83fff700 1 --2- 172.21.15.98:0/1218261877 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f840cf570 0x7f7f840d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.204+0000 7f7f83fff700 1 --2- 172.21.15.98:0/1218261877 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f8415db30 0x7f7f840c8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.204+0000 7f7f83fff700 1 -- 172.21.15.98:0/1218261877 >> 172.21.15.98:0/1218261877 conn(0x7f7f8406b880 msgr2=0x7f7f84151c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:39.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.204+0000 7f7f83fff700 1 -- 172.21.15.98:0/1218261877 shutdown_connections 2023-03-03T07:58:39.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.204+0000 7f7f83fff700 1 -- 172.21.15.98:0/1218261877 wait complete. 2023-03-03T07:58:39.232 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T07:58:39.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.478+0000 7fae649ff700 1 Processor -- start 2023-03-03T07:58:39.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.478+0000 7fae649ff700 1 -- start start 2023-03-03T07:58:39.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.479+0000 7fae649ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae600b5380 0x7fae6006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.480+0000 7fae649ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae600cd970 con 0x7fae6015cc00 2023-03-03T07:58:39.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.480+0000 7fae649ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae600cdc50 con 0x7fae6015c830 2023-03-03T07:58:39.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.480+0000 7fae649ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae600cdf30 con 0x7fae600b5380 2023-03-03T07:58:39.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.480+0000 7fae57fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fae6015cc00 0x7fae600b4c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48080/0 (socket says 172.21.15.98:48080) 2023-03-03T07:58:39.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.480+0000 7fae57fff700 1 -- 172.21.15.98:0/826725239 learned_addr learned my addr 172.21.15.98:0/826725239 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:39.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.480+0000 7fae5ed9d700 1 --2- 172.21.15.98:0/826725239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae600b5380 0x7fae6006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.480+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669309199 0 0) 0x7fae600cd970 con 0x7fae6015cc00 2023-03-03T07:58:39.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.480+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae4c0035b0 con 0x7fae6015cc00 2023-03-03T07:58:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008081405 0 0) 0x7fae600cdc50 con 0x7fae6015c830 2023-03-03T07:58:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae600cd970 con 0x7fae6015c830 2023-03-03T07:58:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1927724300 0 0) 0x7fae4c0035b0 con 0x7fae6015cc00 2023-03-03T07:58:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae600b5380 msgr2=0x7fae6006b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5dd9b700 1 --2- 172.21.15.98:0/826725239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae600b5380 0x7fae6006b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 >> v1:172.21.15.100:6789/0 conn(0x7fae6015c830 legacy=0x7fae600d0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae600d0e60 con 0x7fae6015cc00 2023-03-03T07:58:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7fae48001670 con 0x7fae6015cc00 2023-03-03T07:58:39.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5ed9d700 1 --2- 172.21.15.98:0/826725239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae600b5380 0x7fae6006b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:58:39.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 >> v1:172.21.15.98:6789/0 conn(0x7fae6015cc00 legacy=0x7fae600b4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5dd9b700 1 --2- 172.21.15.98:0/826725239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae4c003bc0 0x7fae4c003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5dd9b700 1 --2- 172.21.15.98:0/826725239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae600b5380 0x7fae4c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5dd9b700 1 --2- 172.21.15.98:0/826725239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae4c004d90 0x7fae4c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae600cdf30 con 0x7fae4c003bc0 2023-03-03T07:58:39.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c00bb40 con 0x7fae4c004d90 2023-03-03T07:58:39.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c00bd00 con 0x7fae600b5380 2023-03-03T07:58:39.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5e59c700 1 --2- 172.21.15.98:0/826725239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae4c003bc0 0x7fae4c003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae5ed9d700 1 --2- 172.21.15.98:0/826725239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae4c004d90 0x7fae4c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.481+0000 7fae57fff700 1 --2- 172.21.15.98:0/826725239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae600b5380 0x7fae4c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.482+0000 7fae5e59c700 1 -- 172.21.15.98:0/826725239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae600b5380 msgr2=0x7fae4c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:39.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.482+0000 7fae5e59c700 1 --2- 172.21.15.98:0/826725239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae600b5380 0x7fae4c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.482+0000 7fae5e59c700 1 -- 172.21.15.98:0/826725239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae4c004d90 msgr2=0x7fae4c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:39.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.482+0000 7fae5e59c700 1 --2- 172.21.15.98:0/826725239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae4c004d90 0x7fae4c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.482+0000 7fae5e59c700 1 -- 172.21.15.98:0/826725239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae58009a70 con 0x7fae4c003bc0 2023-03-03T07:58:39.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.482+0000 7fae5e59c700 1 --2- 172.21.15.98:0/826725239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae4c003bc0 0x7fae4c003fb0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fae50001940 tx=0x7fae4c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43f510d00cae16e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:39.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.482+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50001430 con 0x7fae4c003bc0 2023-03-03T07:58:39.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.482+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae50001d70 con 0x7fae4c003bc0 2023-03-03T07:58:39.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.482+0000 7fae5dd9b700 1 -- 172.21.15.98:0/826725239 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50005510 con 0x7fae4c003bc0 2023-03-03T07:58:39.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.483+0000 7fae649ff700 1 -- 172.21.15.98:0/826725239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae4c003bc0 msgr2=0x7fae4c003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.483+0000 7fae649ff700 1 --2- 172.21.15.98:0/826725239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae4c003bc0 0x7fae4c003fb0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fae50001940 tx=0x7fae4c005880 comp rx=0 tx=0).stop 2023-03-03T07:58:39.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.483+0000 7fae5ed9d700 1 -- 172.21.15.98:0/826725239 reap_dead start 2023-03-03T07:58:39.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.483+0000 7fae649ff700 1 -- 172.21.15.98:0/826725239 shutdown_connections 2023-03-03T07:58:39.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.483+0000 7fae649ff700 1 -- 172.21.15.98:0/826725239 >> 172.21.15.98:0/826725239 conn(0x7fae60073e70 msgr2=0x7fae6006ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:39.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.483+0000 7fae649ff700 1 -- 172.21.15.98:0/826725239 shutdown_connections 2023-03-03T07:58:39.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.483+0000 7fae649ff700 1 -- 172.21.15.98:0/826725239 wait complete. 2023-03-03T07:58:39.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.485+0000 7fae649ff700 1 Processor -- start 2023-03-03T07:58:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.485+0000 7fae649ff700 1 -- start start 2023-03-03T07:58:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.486+0000 7fae649ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae4c003bc0 0x7fae6015a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.486+0000 7fae649ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6014e9d0 0x7fae600c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.486+0000 7fae649ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae600c87d0 0x7fae600c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.486+0000 7fae649ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6015f940 con 0x7fae6014e9d0 2023-03-03T07:58:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.486+0000 7fae649ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae600c8c40 con 0x7fae600c87d0 2023-03-03T07:58:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.486+0000 7fae649ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6015ecc0 con 0x7fae4c003bc0 2023-03-03T07:58:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.487+0000 7fae57fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6014e9d0 0x7fae600c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.487+0000 7fae57fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6014e9d0 0x7fae600c8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51552/0 (socket says 172.21.15.98:51552) 2023-03-03T07:58:39.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.487+0000 7fae57fff700 1 -- 172.21.15.98:0/579181296 learned_addr learned my addr 172.21.15.98:0/579181296 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:39.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.487+0000 7fae5e59c700 1 --2- 172.21.15.98:0/579181296 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae4c003bc0 0x7fae6015a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.487+0000 7fae5ed9d700 1 --2- 172.21.15.98:0/579181296 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae600c87d0 0x7fae600c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.487+0000 7fae57fff700 1 -- 172.21.15.98:0/579181296 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae4c003bc0 msgr2=0x7fae6015a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.487+0000 7fae57fff700 1 --2- 172.21.15.98:0/579181296 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae4c003bc0 0x7fae6015a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.487+0000 7fae57fff700 1 -- 172.21.15.98:0/579181296 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae600c87d0 msgr2=0x7fae600c6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.487+0000 7fae57fff700 1 --2- 172.21.15.98:0/579181296 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae600c87d0 0x7fae600c6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.487+0000 7fae57fff700 1 -- 172.21.15.98:0/579181296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae600d2670 con 0x7fae6014e9d0 2023-03-03T07:58:39.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.487+0000 7fae5ed9d700 1 --2- 172.21.15.98:0/579181296 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae600c87d0 0x7fae600c6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:39.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.487+0000 7fae57fff700 1 --2- 172.21.15.98:0/579181296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6014e9d0 0x7fae600c8290 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fae480010e0 tx=0x7fae4800ea60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:39.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.488+0000 7fae56ffd700 1 -- 172.21.15.98:0/579181296 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae48010510 con 0x7fae6014e9d0 2023-03-03T07:58:39.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.488+0000 7fae56ffd700 1 -- 172.21.15.98:0/579181296 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae48010670 con 0x7fae6014e9d0 2023-03-03T07:58:39.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.488+0000 7fae56ffd700 1 -- 172.21.15.98:0/579181296 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae480109e0 con 0x7fae6014e9d0 2023-03-03T07:58:39.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.488+0000 7fae649ff700 1 -- 172.21.15.98:0/579181296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae600d1140 con 0x7fae6014e9d0 2023-03-03T07:58:39.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.488+0000 7fae649ff700 1 -- 172.21.15.98:0/579181296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae600d2800 con 0x7fae6014e9d0 2023-03-03T07:58:39.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.489+0000 7fae649ff700 1 -- 172.21.15.98:0/579181296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6015fa80 con 0x7fae6014e9d0 2023-03-03T07:58:39.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.490+0000 7fae56ffd700 1 -- 172.21.15.98:0/579181296 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fae4801b030 con 0x7fae6014e9d0 2023-03-03T07:58:39.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.492+0000 7fae56ffd700 1 --2- 172.21.15.98:0/579181296 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae400683c0 0x7fae4006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.492+0000 7fae56ffd700 1 -- 172.21.15.98:0/579181296 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fae48085480 con 0x7fae6014e9d0 2023-03-03T07:58:39.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.492+0000 7fae5e59c700 1 --2- 172.21.15.98:0/579181296 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae400683c0 0x7fae4006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.493+0000 7fae5e59c700 1 --2- 172.21.15.98:0/579181296 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae400683c0 0x7fae4006a870 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fae600cffc0 tx=0x7fae5000a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:39.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.496+0000 7fae56ffd700 1 -- 172.21.15.98:0/579181296 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fae4804d5e0 con 0x7fae6014e9d0 2023-03-03T07:58:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.690+0000 7fae649ff700 1 -- 172.21.15.98:0/579181296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fae600ca7f0 con 0x7fae6014e9d0 2023-03-03T07:58:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.691+0000 7fae56ffd700 1 -- 172.21.15.98:0/579181296 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v54) v1 ==== 92+0+10 (secure 0 0 0) 0x7fae480190a0 con 0x7fae6014e9d0 2023-03-03T07:58:39.692 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T07:58:39.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.693+0000 7fae649ff700 1 -- 172.21.15.98:0/579181296 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae400683c0 msgr2=0x7fae4006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.693+0000 7fae649ff700 1 --2- 172.21.15.98:0/579181296 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae400683c0 0x7fae4006a870 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fae600cffc0 tx=0x7fae5000a000 comp rx=0 tx=0).stop 2023-03-03T07:58:39.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.693+0000 7fae649ff700 1 -- 172.21.15.98:0/579181296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6014e9d0 msgr2=0x7fae600c8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.693+0000 7fae649ff700 1 --2- 172.21.15.98:0/579181296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6014e9d0 0x7fae600c8290 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fae480010e0 tx=0x7fae4800ea60 comp rx=0 tx=0).stop 2023-03-03T07:58:39.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.693+0000 7fae649ff700 1 -- 172.21.15.98:0/579181296 shutdown_connections 2023-03-03T07:58:39.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.693+0000 7fae649ff700 1 --2- 172.21.15.98:0/579181296 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fae400683c0 0x7fae4006a870 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.693+0000 7fae649ff700 1 --2- 172.21.15.98:0/579181296 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae600c87d0 0x7fae600c6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.693+0000 7fae649ff700 1 --2- 172.21.15.98:0/579181296 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae4c003bc0 0x7fae6015a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.693+0000 7fae649ff700 1 --2- 172.21.15.98:0/579181296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6014e9d0 0x7fae600c8290 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.693+0000 7fae649ff700 1 -- 172.21.15.98:0/579181296 >> 172.21.15.98:0/579181296 conn(0x7fae60073e70 msgr2=0x7fae6006ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:39.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.694+0000 7fae649ff700 1 -- 172.21.15.98:0/579181296 shutdown_connections 2023-03-03T07:58:39.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.694+0000 7fae649ff700 1 -- 172.21.15.98:0/579181296 wait complete. 2023-03-03T07:58:39.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T07:58:39.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.969+0000 7fce4ed9d700 1 Processor -- start 2023-03-03T07:58:39.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.969+0000 7fce4ed9d700 1 -- start start 2023-03-03T07:58:39.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.971+0000 7fce4ed9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fce480b4aa0 0x7fce480b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.971+0000 7fce4ed9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4815dd00 con 0x7fce480cf570 2023-03-03T07:58:39.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.971+0000 7fce4ed9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce480cd780 con 0x7fce481609b0 2023-03-03T07:58:39.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.971+0000 7fce4ed9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce480cebb0 con 0x7fce480b4aa0 2023-03-03T07:58:39.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.971+0000 7fce4d59a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fce480cf570 0x7fce4815d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48096/0 (socket says 172.21.15.98:48096) 2023-03-03T07:58:39.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.971+0000 7fce4d59a700 1 -- 172.21.15.98:0/1821022890 learned_addr learned my addr 172.21.15.98:0/1821022890 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:39.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.971+0000 7fce4e59c700 1 --2- 172.21.15.98:0/1821022890 >> v2:172.21.15.100:3300/0 conn(0x7fce480b4aa0 0x7fce480b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.972+0000 7fce4cd99700 1 -- 172.21.15.98:0/1821022890 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292081053 0 0) 0x7fce4815dd00 con 0x7fce480cf570 2023-03-03T07:58:39.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.972+0000 7fce4cd99700 1 -- 172.21.15.98:0/1821022890 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce340035b0 con 0x7fce480cf570 2023-03-03T07:58:39.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.972+0000 7fce4e59c700 1 -- 172.21.15.98:0/1821022890 >> v1:172.21.15.100:6789/0 conn(0x7fce481609b0 legacy=0x7fce48160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.972+0000 7fce4e59c700 1 -- 172.21.15.98:0/1821022890 >> v1:172.21.15.98:6789/0 conn(0x7fce480cf570 legacy=0x7fce4815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.972+0000 7fce4e59c700 1 -- 172.21.15.98:0/1821022890 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce480cda60 con 0x7fce480b4aa0 2023-03-03T07:58:39.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.972+0000 7fce4e59c700 1 --2- 172.21.15.98:0/1821022890 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce480b4aa0 0x7fce480b4f20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fce40009d90 tx=0x7fce4000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11d20878210c2e9e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:39.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.973+0000 7fce4cd99700 1 -- 172.21.15.98:0/1821022890 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4000e070 con 0x7fce480b4aa0 2023-03-03T07:58:39.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.973+0000 7fce4cd99700 1 -- 172.21.15.98:0/1821022890 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4000bb20 con 0x7fce480b4aa0 2023-03-03T07:58:39.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.973+0000 7fce4cd99700 1 -- 172.21.15.98:0/1821022890 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce40012430 con 0x7fce480b4aa0 2023-03-03T07:58:39.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.973+0000 7fce4ed9d700 1 -- 172.21.15.98:0/1821022890 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce480b4aa0 msgr2=0x7fce480b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.973+0000 7fce4ed9d700 1 --2- 172.21.15.98:0/1821022890 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce480b4aa0 0x7fce480b4f20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fce40009d90 tx=0x7fce4000b330 comp rx=0 tx=0).stop 2023-03-03T07:58:39.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.973+0000 7fce4ed9d700 1 -- 172.21.15.98:0/1821022890 shutdown_connections 2023-03-03T07:58:39.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.973+0000 7fce4ed9d700 1 --2- 172.21.15.98:0/1821022890 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce480b4aa0 0x7fce480b4f20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.973+0000 7fce4ed9d700 1 -- 172.21.15.98:0/1821022890 >> 172.21.15.98:0/1821022890 conn(0x7fce4806b880 msgr2=0x7fce480cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:39.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.973+0000 7fce4ed9d700 1 -- 172.21.15.98:0/1821022890 shutdown_connections 2023-03-03T07:58:39.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.974+0000 7fce4ed9d700 1 -- 172.21.15.98:0/1821022890 wait complete. 2023-03-03T07:58:39.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.976+0000 7fce4ed9d700 1 Processor -- start 2023-03-03T07:58:39.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.976+0000 7fce4ed9d700 1 -- start start 2023-03-03T07:58:39.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.977+0000 7fce4ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce480b4aa0 0x7fce48150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.977+0000 7fce4ed9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce480cf570 0x7fce481f2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.977+0000 7fce4ed9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce481609b0 0x7fce481f2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.977+0000 7fce4ed9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce480c7a10 con 0x7fce480b4aa0 2023-03-03T07:58:39.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.977+0000 7fce4ed9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce480c7cf0 con 0x7fce480cf570 2023-03-03T07:58:39.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.977+0000 7fce4ed9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce480c7fd0 con 0x7fce481609b0 2023-03-03T07:58:39.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.977+0000 7fce4dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce480b4aa0 0x7fce48150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.977+0000 7fce4dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce480b4aa0 0x7fce48150e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51560/0 (socket says 172.21.15.98:51560) 2023-03-03T07:58:39.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.977+0000 7fce4dd9b700 1 -- 172.21.15.98:0/405661209 learned_addr learned my addr 172.21.15.98:0/405661209 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:39.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.977+0000 7fce4d59a700 1 --2- 172.21.15.98:0/405661209 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce480cf570 0x7fce481f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.978+0000 7fce4e59c700 1 --2- 172.21.15.98:0/405661209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce481609b0 0x7fce481f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.978+0000 7fce4dd9b700 1 -- 172.21.15.98:0/405661209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce481609b0 msgr2=0x7fce481f2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.978+0000 7fce4dd9b700 1 --2- 172.21.15.98:0/405661209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce481609b0 0x7fce481f2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.978+0000 7fce4dd9b700 1 -- 172.21.15.98:0/405661209 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce480cf570 msgr2=0x7fce481f2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:39.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.978+0000 7fce4dd9b700 1 --2- 172.21.15.98:0/405661209 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce480cf570 0x7fce481f2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:39.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.978+0000 7fce4dd9b700 1 -- 172.21.15.98:0/405661209 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce40009a70 con 0x7fce480b4aa0 2023-03-03T07:58:39.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.978+0000 7fce4dd9b700 1 --2- 172.21.15.98:0/405661209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce480b4aa0 0x7fce48150e20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fce4400c450 tx=0x7fce4400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:39.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.978+0000 7fce3e7fc700 1 -- 172.21.15.98:0/405661209 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce440052d0 con 0x7fce480b4aa0 2023-03-03T07:58:39.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.978+0000 7fce3e7fc700 1 -- 172.21.15.98:0/405661209 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce44005430 con 0x7fce480b4aa0 2023-03-03T07:58:39.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.978+0000 7fce3e7fc700 1 -- 172.21.15.98:0/405661209 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce440038c0 con 0x7fce480b4aa0 2023-03-03T07:58:39.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.978+0000 7fce4ed9d700 1 -- 172.21.15.98:0/405661209 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce4815cdf0 con 0x7fce480b4aa0 2023-03-03T07:58:39.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.979+0000 7fce4ed9d700 1 -- 172.21.15.98:0/405661209 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce480cdd40 con 0x7fce480b4aa0 2023-03-03T07:58:39.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.980+0000 7fce4ed9d700 1 -- 172.21.15.98:0/405661209 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4815d0d0 con 0x7fce480b4aa0 2023-03-03T07:58:39.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.981+0000 7fce3e7fc700 1 -- 172.21.15.98:0/405661209 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fce44019030 con 0x7fce480b4aa0 2023-03-03T07:58:39.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.982+0000 7fce3e7fc700 1 --2- 172.21.15.98:0/405661209 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fce2c0683c0 0x7fce2c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:39.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.982+0000 7fce3e7fc700 1 -- 172.21.15.98:0/405661209 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fce44085130 con 0x7fce480b4aa0 2023-03-03T07:58:39.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.983+0000 7fce4d59a700 1 --2- 172.21.15.98:0/405661209 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fce2c0683c0 0x7fce2c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:39.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.983+0000 7fce4d59a700 1 --2- 172.21.15.98:0/405661209 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fce2c0683c0 0x7fce2c06a870 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fce38007fd0 tx=0x7fce38007dc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:39.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:39.986+0000 7fce3e7fc700 1 -- 172.21.15.98:0/405661209 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fce44051790 con 0x7fce480b4aa0 2023-03-03T07:58:40.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.178+0000 7fce4ed9d700 1 -- 172.21.15.98:0/405661209 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fce48068290 con 0x7fce480b4aa0 2023-03-03T07:58:40.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.178+0000 7fce3e7fc700 1 -- 172.21.15.98:0/405661209 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7fce44054db0 con 0x7fce480b4aa0 2023-03-03T07:58:40.180 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T07:58:40.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.181+0000 7fce4ed9d700 1 -- 172.21.15.98:0/405661209 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fce2c0683c0 msgr2=0x7fce2c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:40.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.181+0000 7fce4ed9d700 1 --2- 172.21.15.98:0/405661209 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fce2c0683c0 0x7fce2c06a870 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fce38007fd0 tx=0x7fce38007dc0 comp rx=0 tx=0).stop 2023-03-03T07:58:40.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.181+0000 7fce4ed9d700 1 -- 172.21.15.98:0/405661209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce480b4aa0 msgr2=0x7fce48150e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.181+0000 7fce4ed9d700 1 --2- 172.21.15.98:0/405661209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce480b4aa0 0x7fce48150e20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fce4400c450 tx=0x7fce4400c760 comp rx=0 tx=0).stop 2023-03-03T07:58:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.181+0000 7fce4ed9d700 1 -- 172.21.15.98:0/405661209 shutdown_connections 2023-03-03T07:58:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.181+0000 7fce4ed9d700 1 --2- 172.21.15.98:0/405661209 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fce2c0683c0 0x7fce2c06a870 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.181+0000 7fce4ed9d700 1 --2- 172.21.15.98:0/405661209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce480b4aa0 0x7fce48150e20 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.181+0000 7fce4ed9d700 1 --2- 172.21.15.98:0/405661209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce481609b0 0x7fce481f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.181+0000 7fce4ed9d700 1 --2- 172.21.15.98:0/405661209 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce480cf570 0x7fce481f2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.181+0000 7fce4ed9d700 1 -- 172.21.15.98:0/405661209 >> 172.21.15.98:0/405661209 conn(0x7fce4806b880 msgr2=0x7fce480b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:40.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.181+0000 7fce4ed9d700 1 -- 172.21.15.98:0/405661209 shutdown_connections 2023-03-03T07:58:40.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.181+0000 7fce4ed9d700 1 -- 172.21.15.98:0/405661209 wait complete. 2023-03-03T07:58:40.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T07:58:40.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.456+0000 7ff2ee5ff700 1 Processor -- start 2023-03-03T07:58:40.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.456+0000 7ff2ee5ff700 1 -- start start 2023-03-03T07:58:40.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.457+0000 7ff2ee5ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2e80b5310 0x7ff2e8167850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:40.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.457+0000 7ff2ee5ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e80cd7d0 con 0x7ff2e80b26d0 2023-03-03T07:58:40.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.457+0000 7ff2ee5ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e80b90e0 con 0x7ff2e80b8d10 2023-03-03T07:58:40.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.457+0000 7ff2ee5ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e80cda20 con 0x7ff2e80b5310 2023-03-03T07:58:40.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.458+0000 7ff2ecb9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2e80b5310 0x7ff2e8167850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:40.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.458+0000 7ff2dffff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff2e80b26d0 0x7ff2e80b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48108/0 (socket says 172.21.15.98:48108) 2023-03-03T07:58:40.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.458+0000 7ff2ecb9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2e80b5310 0x7ff2e8167850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51572/0 (socket says 172.21.15.98:51572) 2023-03-03T07:58:40.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.458+0000 7ff2dffff700 1 -- 172.21.15.98:0/1643619230 learned_addr learned my addr 172.21.15.98:0/1643619230 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:40.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.458+0000 7ff2ecb9c700 1 -- 172.21.15.98:0/1643619230 >> v1:172.21.15.100:6789/0 conn(0x7ff2e80b8d10 legacy=0x7ff2e80cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:40.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.458+0000 7ff2ecb9c700 1 -- 172.21.15.98:0/1643619230 >> v1:172.21.15.98:6789/0 conn(0x7ff2e80b26d0 legacy=0x7ff2e80b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.458+0000 7ff2ecb9c700 1 -- 172.21.15.98:0/1643619230 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e80cdd00 con 0x7ff2e80b5310 2023-03-03T07:58:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.458+0000 7ff2e77fe700 1 -- 172.21.15.98:0/1643619230 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321024822 0 0) 0x7ff2e80cd7d0 con 0x7ff2e80b26d0 2023-03-03T07:58:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.459+0000 7ff2ecb9c700 1 --2- 172.21.15.98:0/1643619230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e80b5310 0x7ff2e8167850 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0009840 tx=0x7ff2e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e8511164a3c6f7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.459+0000 7ff2e77fe700 1 -- 172.21.15.98:0/1643619230 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e0011830 con 0x7ff2e80b5310 2023-03-03T07:58:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.459+0000 7ff2e77fe700 1 -- 172.21.15.98:0/1643619230 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e0011990 con 0x7ff2e80b5310 2023-03-03T07:58:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.459+0000 7ff2e77fe700 1 -- 172.21.15.98:0/1643619230 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e0011d20 con 0x7ff2e80b5310 2023-03-03T07:58:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.459+0000 7ff2ee5ff700 1 -- 172.21.15.98:0/1643619230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e80b5310 msgr2=0x7ff2e8167850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:40.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.459+0000 7ff2ee5ff700 1 --2- 172.21.15.98:0/1643619230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e80b5310 0x7ff2e8167850 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0009840 tx=0x7ff2e0009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:40.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.460+0000 7ff2ee5ff700 1 -- 172.21.15.98:0/1643619230 shutdown_connections 2023-03-03T07:58:40.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.460+0000 7ff2ee5ff700 1 --2- 172.21.15.98:0/1643619230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e80b5310 0x7ff2e8167850 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:40.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.460+0000 7ff2ee5ff700 1 -- 172.21.15.98:0/1643619230 >> 172.21.15.98:0/1643619230 conn(0x7ff2e8073e70 msgr2=0x7ff2e806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:40.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.460+0000 7ff2ee5ff700 1 -- 172.21.15.98:0/1643619230 shutdown_connections 2023-03-03T07:58:40.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.460+0000 7ff2ee5ff700 1 -- 172.21.15.98:0/1643619230 wait complete. 2023-03-03T07:58:40.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.462+0000 7ff2ee5ff700 1 Processor -- start 2023-03-03T07:58:40.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.463+0000 7ff2ee5ff700 1 -- start start 2023-03-03T07:58:40.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.464+0000 7ff2ee5ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e80b26d0 0x7ff2e815cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:40.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.464+0000 7ff2ee5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2e80b5310 0x7ff2e81f3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:40.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.464+0000 7ff2ee5ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2e80b8d10 0x7ff2e80cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:40.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.464+0000 7ff2ee5ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e81f3a40 con 0x7ff2e80b26d0 2023-03-03T07:58:40.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.464+0000 7ff2ee5ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e8157580 con 0x7ff2e80b8d10 2023-03-03T07:58:40.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.464+0000 7ff2ee5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e815d600 con 0x7ff2e80b5310 2023-03-03T07:58:40.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.465+0000 7ff2e7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e80b26d0 0x7ff2e815cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:40.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.465+0000 7ff2e7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e80b26d0 0x7ff2e815cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51580/0 (socket says 172.21.15.98:51580) 2023-03-03T07:58:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.465+0000 7ff2e7fff700 1 -- 172.21.15.98:0/2778949511 learned_addr learned my addr 172.21.15.98:0/2778949511 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.465+0000 7ff2dffff700 1 --2- 172.21.15.98:0/2778949511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2e80b5310 0x7ff2e81f3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.465+0000 7ff2ecb9c700 1 --2- 172.21.15.98:0/2778949511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2e80b8d10 0x7ff2e80cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.465+0000 7ff2e7fff700 1 -- 172.21.15.98:0/2778949511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2e80b5310 msgr2=0x7ff2e81f3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.465+0000 7ff2e7fff700 1 --2- 172.21.15.98:0/2778949511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2e80b5310 0x7ff2e81f3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.465+0000 7ff2e7fff700 1 -- 172.21.15.98:0/2778949511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2e80b8d10 msgr2=0x7ff2e80cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.465+0000 7ff2e7fff700 1 --2- 172.21.15.98:0/2778949511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2e80b8d10 0x7ff2e80cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.465+0000 7ff2e7fff700 1 -- 172.21.15.98:0/2778949511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e00094a0 con 0x7ff2e80b26d0 2023-03-03T07:58:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.466+0000 7ff2e7fff700 1 --2- 172.21.15.98:0/2778949511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e80b26d0 0x7ff2e815cf60 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8001f00 tx=0x7ff2d800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.466+0000 7ff2e57fa700 1 -- 172.21.15.98:0/2778949511 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8003c00 con 0x7ff2e80b26d0 2023-03-03T07:58:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.466+0000 7ff2e57fa700 1 -- 172.21.15.98:0/2778949511 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d8003d60 con 0x7ff2e80b26d0 2023-03-03T07:58:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.466+0000 7ff2e57fa700 1 -- 172.21.15.98:0/2778949511 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d800ab70 con 0x7ff2e80b26d0 2023-03-03T07:58:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.466+0000 7ff2ee5ff700 1 -- 172.21.15.98:0/2778949511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e80cdfe0 con 0x7ff2e80b26d0 2023-03-03T07:58:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.466+0000 7ff2ee5ff700 1 -- 172.21.15.98:0/2778949511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e80d0e80 con 0x7ff2e80b26d0 2023-03-03T07:58:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.467+0000 7ff2ee5ff700 1 -- 172.21.15.98:0/2778949511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e80b5fd0 con 0x7ff2e80b26d0 2023-03-03T07:58:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.468+0000 7ff2e57fa700 1 -- 172.21.15.98:0/2778949511 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7ff2d8010050 con 0x7ff2e80b26d0 2023-03-03T07:58:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.470+0000 7ff2e57fa700 1 --2- 172.21.15.98:0/2778949511 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff2c80683c0 0x7ff2c806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.470+0000 7ff2e57fa700 1 -- 172.21.15.98:0/2778949511 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7ff2d8085da0 con 0x7ff2e80b26d0 2023-03-03T07:58:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.470+0000 7ff2dffff700 1 --2- 172.21.15.98:0/2778949511 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff2c80683c0 0x7ff2c806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.471+0000 7ff2dffff700 1 --2- 172.21.15.98:0/2778949511 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff2c80683c0 0x7ff2c806a870 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff2e80cd250 tx=0x7ff2d0004a50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:40.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.474+0000 7ff2e57fa700 1 -- 172.21.15.98:0/2778949511 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff2d8052400 con 0x7ff2e80b26d0 2023-03-03T07:58:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.666+0000 7ff2ee5ff700 1 -- 172.21.15.98:0/2778949511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff2e804e6e0 con 0x7ff2e80b26d0 2023-03-03T07:58:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.666+0000 7ff2e57fa700 1 -- 172.21.15.98:0/2778949511 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff2d8055a20 con 0x7ff2e80b26d0 2023-03-03T07:58:40.668 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T07:58:40.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.669+0000 7ff2ee5ff700 1 -- 172.21.15.98:0/2778949511 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff2c80683c0 msgr2=0x7ff2c806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:40.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.669+0000 7ff2ee5ff700 1 --2- 172.21.15.98:0/2778949511 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff2c80683c0 0x7ff2c806a870 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff2e80cd250 tx=0x7ff2d0004a50 comp rx=0 tx=0).stop 2023-03-03T07:58:40.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.669+0000 7ff2ee5ff700 1 -- 172.21.15.98:0/2778949511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e80b26d0 msgr2=0x7ff2e815cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:40.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.669+0000 7ff2ee5ff700 1 --2- 172.21.15.98:0/2778949511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e80b26d0 0x7ff2e815cf60 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8001f00 tx=0x7ff2d800a060 comp rx=0 tx=0).stop 2023-03-03T07:58:40.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.669+0000 7ff2ee5ff700 1 -- 172.21.15.98:0/2778949511 shutdown_connections 2023-03-03T07:58:40.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.669+0000 7ff2ee5ff700 1 --2- 172.21.15.98:0/2778949511 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff2c80683c0 0x7ff2c806a870 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:40.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.669+0000 7ff2ee5ff700 1 --2- 172.21.15.98:0/2778949511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e80b26d0 0x7ff2e815cf60 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:40.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.669+0000 7ff2ee5ff700 1 --2- 172.21.15.98:0/2778949511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2e80b8d10 0x7ff2e80cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.669+0000 7ff2ee5ff700 1 --2- 172.21.15.98:0/2778949511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2e80b5310 0x7ff2e81f3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.669+0000 7ff2ee5ff700 1 -- 172.21.15.98:0/2778949511 >> 172.21.15.98:0/2778949511 conn(0x7ff2e8073e70 msgr2=0x7ff2e80cfc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.669+0000 7ff2ee5ff700 1 -- 172.21.15.98:0/2778949511 shutdown_connections 2023-03-03T07:58:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.669+0000 7ff2ee5ff700 1 -- 172.21.15.98:0/2778949511 wait complete. 2023-03-03T07:58:40.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T07:58:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.944+0000 7ffa1e7ff700 1 Processor -- start 2023-03-03T07:58:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.944+0000 7ffa1e7ff700 1 -- start start 2023-03-03T07:58:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.945+0000 7ffa1e7ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffa180b5310 0x7ffa18167850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.945+0000 7ffa1e7ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa180cd7d0 con 0x7ffa180b26d0 2023-03-03T07:58:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.946+0000 7ffa1e7ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa180b90e0 con 0x7ffa180b8d10 2023-03-03T07:58:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.946+0000 7ffa1e7ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa180cda20 con 0x7ffa180b5310 2023-03-03T07:58:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.946+0000 7ffa177fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ffa180b26d0 0x7ffa180b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48114/0 (socket says 172.21.15.98:48114) 2023-03-03T07:58:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.946+0000 7ffa177fe700 1 -- 172.21.15.98:0/4289718624 learned_addr learned my addr 172.21.15.98:0/4289718624 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.946+0000 7ffa1cd9c700 1 --2- 172.21.15.98:0/4289718624 >> v2:172.21.15.98:3300/0 conn(0x7ffa180b5310 0x7ffa18167850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.946+0000 7ffa1cd9c700 1 -- 172.21.15.98:0/4289718624 >> v1:172.21.15.100:6789/0 conn(0x7ffa180b8d10 legacy=0x7ffa180cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.946+0000 7ffa16ffd700 1 -- 172.21.15.98:0/4289718624 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556619401 0 0) 0x7ffa180cd7d0 con 0x7ffa180b26d0 2023-03-03T07:58:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.946+0000 7ffa1cd9c700 1 -- 172.21.15.98:0/4289718624 >> v1:172.21.15.98:6789/0 conn(0x7ffa180b26d0 legacy=0x7ffa180b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.946+0000 7ffa1cd9c700 1 -- 172.21.15.98:0/4289718624 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa180cdd00 con 0x7ffa180b5310 2023-03-03T07:58:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.947+0000 7ffa1cd9c700 1 --2- 172.21.15.98:0/4289718624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa180b5310 0x7ffa18167850 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ffa10009d90 tx=0x7ffa1000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c79829d69038b5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.947+0000 7ffa16ffd700 1 -- 172.21.15.98:0/4289718624 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1000e070 con 0x7ffa180b5310 2023-03-03T07:58:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.947+0000 7ffa16ffd700 1 -- 172.21.15.98:0/4289718624 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1000bbd0 con 0x7ffa180b5310 2023-03-03T07:58:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.947+0000 7ffa16ffd700 1 -- 172.21.15.98:0/4289718624 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa10012430 con 0x7ffa180b5310 2023-03-03T07:58:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.947+0000 7ffa1e7ff700 1 -- 172.21.15.98:0/4289718624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa180b5310 msgr2=0x7ffa18167850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.947+0000 7ffa1e7ff700 1 --2- 172.21.15.98:0/4289718624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa180b5310 0x7ffa18167850 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ffa10009d90 tx=0x7ffa1000b330 comp rx=0 tx=0).stop 2023-03-03T07:58:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.948+0000 7ffa1e7ff700 1 -- 172.21.15.98:0/4289718624 shutdown_connections 2023-03-03T07:58:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.948+0000 7ffa1e7ff700 1 --2- 172.21.15.98:0/4289718624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa180b5310 0x7ffa18167850 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.948+0000 7ffa1e7ff700 1 -- 172.21.15.98:0/4289718624 >> 172.21.15.98:0/4289718624 conn(0x7ffa18073e70 msgr2=0x7ffa1806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.948+0000 7ffa1e7ff700 1 -- 172.21.15.98:0/4289718624 shutdown_connections 2023-03-03T07:58:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.948+0000 7ffa1e7ff700 1 -- 172.21.15.98:0/4289718624 wait complete. 2023-03-03T07:58:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.950+0000 7ffa1e7ff700 1 Processor -- start 2023-03-03T07:58:40.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.951+0000 7ffa1e7ff700 1 -- start start 2023-03-03T07:58:40.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.952+0000 7ffa1e7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa180b26d0 0x7ffa180c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:40.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.952+0000 7ffa1e7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa180b5310 0x7ffa181e5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.952+0000 7ffa1e7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa180b8d10 0x7ffa181e6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.952+0000 7ffa1e7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa181e5b00 con 0x7ffa180b8d10 2023-03-03T07:58:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.952+0000 7ffa1e7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa181e5de0 con 0x7ffa180b26d0 2023-03-03T07:58:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.952+0000 7ffa1e7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa181e60c0 con 0x7ffa180b5310 2023-03-03T07:58:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.953+0000 7ffa1cd9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa180b8d10 0x7ffa181e6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.953+0000 7ffa1cd9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa180b8d10 0x7ffa181e6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51610/0 (socket says 172.21.15.98:51610) 2023-03-03T07:58:40.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.953+0000 7ffa1cd9c700 1 -- 172.21.15.98:0/4227776870 learned_addr learned my addr 172.21.15.98:0/4227776870 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:40.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.953+0000 7ffa17fff700 1 --2- 172.21.15.98:0/4227776870 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa180b26d0 0x7ffa180c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:40.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.953+0000 7ffa177fe700 1 --2- 172.21.15.98:0/4227776870 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa180b5310 0x7ffa181e5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:40.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.953+0000 7ffa1cd9c700 1 -- 172.21.15.98:0/4227776870 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa180b5310 msgr2=0x7ffa181e5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:40.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.953+0000 7ffa1cd9c700 1 --2- 172.21.15.98:0/4227776870 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa180b5310 0x7ffa181e5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:40.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.953+0000 7ffa1cd9c700 1 -- 172.21.15.98:0/4227776870 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa180b26d0 msgr2=0x7ffa180c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:40.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.953+0000 7ffa1cd9c700 1 --2- 172.21.15.98:0/4227776870 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa180b26d0 0x7ffa180c8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:40.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.953+0000 7ffa1cd9c700 1 -- 172.21.15.98:0/4227776870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa10009a70 con 0x7ffa180b8d10 2023-03-03T07:58:40.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.953+0000 7ffa177fe700 1 --2- 172.21.15.98:0/4227776870 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa180b5310 0x7ffa181e5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:58:40.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.954+0000 7ffa1cd9c700 1 --2- 172.21.15.98:0/4227776870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa180b8d10 0x7ffa181e6c80 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ffa10014040 tx=0x7ffa10004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:40.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.954+0000 7ffa14ff9700 1 -- 172.21.15.98:0/4227776870 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1001b810 con 0x7ffa180b8d10 2023-03-03T07:58:40.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.954+0000 7ffa14ff9700 1 -- 172.21.15.98:0/4227776870 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa10020070 con 0x7ffa180b8d10 2023-03-03T07:58:40.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.954+0000 7ffa14ff9700 1 -- 172.21.15.98:0/4227776870 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1001bbb0 con 0x7ffa180b8d10 2023-03-03T07:58:40.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.954+0000 7ffa1e7ff700 1 -- 172.21.15.98:0/4227776870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa180cdfe0 con 0x7ffa180b8d10 2023-03-03T07:58:40.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.954+0000 7ffa1e7ff700 1 -- 172.21.15.98:0/4227776870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa180d0e80 con 0x7ffa180b8d10 2023-03-03T07:58:40.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.955+0000 7ffa1e7ff700 1 -- 172.21.15.98:0/4227776870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa180b5fd0 con 0x7ffa180b8d10 2023-03-03T07:58:40.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.956+0000 7ffa14ff9700 1 -- 172.21.15.98:0/4227776870 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7ffa1002e070 con 0x7ffa180b8d10 2023-03-03T07:58:40.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.957+0000 7ffa14ff9700 1 --2- 172.21.15.98:0/4227776870 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa040683c0 0x7ffa0406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:40.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.958+0000 7ffa14ff9700 1 -- 172.21.15.98:0/4227776870 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7ffa10022070 con 0x7ffa180b8d10 2023-03-03T07:58:40.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.958+0000 7ffa17fff700 1 --2- 172.21.15.98:0/4227776870 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa040683c0 0x7ffa0406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:40.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.959+0000 7ffa17fff700 1 --2- 172.21.15.98:0/4227776870 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa040683c0 0x7ffa0406a870 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ffa08007fd0 tx=0x7ffa08009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:40.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:40.961+0000 7ffa14ff9700 1 -- 172.21.15.98:0/4227776870 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffa10054650 con 0x7ffa180b8d10 2023-03-03T07:58:41.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:41.167+0000 7ffa1e7ff700 1 -- 172.21.15.98:0/4227776870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffa1804e6e0 con 0x7ffa180b8d10 2023-03-03T07:58:41.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:41.168+0000 7ffa14ff9700 1 -- 172.21.15.98:0/4227776870 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v54) v1 ==== 80+0+0 (secure 0 0 0) 0x7ffa1002c080 con 0x7ffa180b8d10 2023-03-03T07:58:41.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:41.170+0000 7ffa1e7ff700 1 -- 172.21.15.98:0/4227776870 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa040683c0 msgr2=0x7ffa0406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:41.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:41.170+0000 7ffa1e7ff700 1 --2- 172.21.15.98:0/4227776870 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa040683c0 0x7ffa0406a870 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ffa08007fd0 tx=0x7ffa08009040 comp rx=0 tx=0).stop 2023-03-03T07:58:41.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:41.170+0000 7ffa1e7ff700 1 -- 172.21.15.98:0/4227776870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa180b8d10 msgr2=0x7ffa181e6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:41.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:41.170+0000 7ffa1e7ff700 1 --2- 172.21.15.98:0/4227776870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa180b8d10 0x7ffa181e6c80 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ffa10014040 tx=0x7ffa10004ad0 comp rx=0 tx=0).stop 2023-03-03T07:58:41.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:41.170+0000 7ffa1e7ff700 1 -- 172.21.15.98:0/4227776870 shutdown_connections 2023-03-03T07:58:41.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:41.170+0000 7ffa1e7ff700 1 --2- 172.21.15.98:0/4227776870 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa040683c0 0x7ffa0406a870 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:41.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:41.170+0000 7ffa1e7ff700 1 --2- 172.21.15.98:0/4227776870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa180b8d10 0x7ffa181e6c80 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:41.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:41.170+0000 7ffa1e7ff700 1 --2- 172.21.15.98:0/4227776870 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa180b26d0 0x7ffa180c8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:41.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:41.170+0000 7ffa1e7ff700 1 --2- 172.21.15.98:0/4227776870 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa180b5310 0x7ffa181e5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:41.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:41.170+0000 7ffa1e7ff700 1 -- 172.21.15.98:0/4227776870 >> 172.21.15.98:0/4227776870 conn(0x7ffa18073e70 msgr2=0x7ffa180cfc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:41.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:41.171+0000 7ffa1e7ff700 1 -- 172.21.15.98:0/4227776870 shutdown_connections 2023-03-03T07:58:41.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:41.171+0000 7ffa1e7ff700 1 -- 172.21.15.98:0/4227776870 wait complete. 2023-03-03T07:58:41.174 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T07:58:41.199 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2023-03-03T07:58:41.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:41.200 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-03-03T07:58:41.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[117851]: fuse finished with error 0 and tester_r 0 2023-03-03T07:58:41.308 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T07:58:47.410 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T07:58:47.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T07:58:47.413 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-03-03T07:58:47.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:58:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.738+0000 7f4b6bdff700 1 Processor -- start 2023-03-03T07:58:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.738+0000 7f4b6bdff700 1 -- start start 2023-03-03T07:58:47.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.740+0000 7f4b6bdff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4b640b87c0 0x7f4b640cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:47.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.740+0000 7f4b6bdff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4b640b5410 0x7f4b640d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:47.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.740+0000 7f4b6bdff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b640d0ef0 con 0x7f4b640b8b90 2023-03-03T07:58:47.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.740+0000 7f4b6bdff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b640d0600 con 0x7f4b640b87c0 2023-03-03T07:58:47.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.740+0000 7f4b6bdff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b640cd970 con 0x7f4b640b5410 2023-03-03T07:58:47.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.740+0000 7f4b69b9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4b640b87c0 0x7f4b640cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:47.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.740+0000 7f4b69b9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4b640b87c0 0x7f4b640cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56070/0 (socket says 172.21.15.98:56070) 2023-03-03T07:58:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.740+0000 7f4b69b9b700 1 -- 172.21.15.98:0/648126701 learned_addr learned my addr 172.21.15.98:0/648126701 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.741+0000 7f4b6a39c700 1 --2- 172.21.15.98:0/648126701 >> v2:172.21.15.100:3300/0 conn(0x7f4b640b5410 0x7f4b640d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.741+0000 7f4b69b9b700 1 -- 172.21.15.98:0/648126701 >> v2:172.21.15.100:3300/0 conn(0x7f4b640b5410 msgr2=0x7f4b640d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.741+0000 7f4b69b9b700 1 --2- 172.21.15.98:0/648126701 >> v2:172.21.15.100:3300/0 conn(0x7f4b640b5410 0x7f4b640d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.741+0000 7f4b69b9b700 1 -- 172.21.15.98:0/648126701 >> v1:172.21.15.100:6789/0 conn(0x7f4b640b8b90 legacy=0x7f4b640b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.741+0000 7f4b69b9b700 1 -- 172.21.15.98:0/648126701 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b640cdc50 con 0x7f4b640b87c0 2023-03-03T07:58:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.741+0000 7f4b6a39c700 1 --2- 172.21.15.98:0/648126701 >> v2:172.21.15.100:3300/0 conn(0x7f4b640b5410 0x7f4b640d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.741+0000 7f4b69b9b700 1 --2- 172.21.15.98:0/648126701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b640b87c0 0x7f4b640cffe0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f4b640c6d50 tx=0x7f4b60009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ceb10ea859e046 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.741+0000 7f4b68b99700 1 -- 172.21.15.98:0/648126701 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b60004230 con 0x7f4b640b87c0 2023-03-03T07:58:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.741+0000 7f4b68b99700 1 -- 172.21.15.98:0/648126701 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b60004390 con 0x7f4b640b87c0 2023-03-03T07:58:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.742+0000 7f4b68b99700 1 -- 172.21.15.98:0/648126701 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b600036a0 con 0x7f4b640b87c0 2023-03-03T07:58:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.742+0000 7f4b6bdff700 1 -- 172.21.15.98:0/648126701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b640b87c0 msgr2=0x7f4b640cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.742+0000 7f4b6bdff700 1 --2- 172.21.15.98:0/648126701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b640b87c0 0x7f4b640cffe0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f4b640c6d50 tx=0x7f4b60009bd0 comp rx=0 tx=0).stop 2023-03-03T07:58:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.742+0000 7f4b6bdff700 1 -- 172.21.15.98:0/648126701 shutdown_connections 2023-03-03T07:58:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.742+0000 7f4b6bdff700 1 --2- 172.21.15.98:0/648126701 >> v2:172.21.15.100:3300/0 conn(0x7f4b640b5410 0x7f4b640d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.742+0000 7f4b6bdff700 1 --2- 172.21.15.98:0/648126701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b640b87c0 0x7f4b640cffe0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.742+0000 7f4b6bdff700 1 -- 172.21.15.98:0/648126701 >> 172.21.15.98:0/648126701 conn(0x7f4b6406aee0 msgr2=0x7f4b6406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.742+0000 7f4b6bdff700 1 -- 172.21.15.98:0/648126701 shutdown_connections 2023-03-03T07:58:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.742+0000 7f4b6bdff700 1 -- 172.21.15.98:0/648126701 wait complete. 2023-03-03T07:58:47.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.744+0000 7f4b6bdff700 1 Processor -- start 2023-03-03T07:58:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.744+0000 7f4b6bdff700 1 -- start start 2023-03-03T07:58:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.745+0000 7f4b6bdff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b640b5410 0x7f4b640cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.745+0000 7f4b6bdff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b640b87c0 0x7f4b641f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.745+0000 7f4b6bdff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b640b8b90 0x7f4b641f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.745+0000 7f4b6bdff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b640cb6d0 con 0x7f4b640b8b90 2023-03-03T07:58:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.745+0000 7f4b6bdff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b640cb9b0 con 0x7f4b640b87c0 2023-03-03T07:58:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.745+0000 7f4b6bdff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b640cbc90 con 0x7f4b640b5410 2023-03-03T07:58:47.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.746+0000 7f4b6a39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b640b8b90 0x7f4b641f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:47.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.746+0000 7f4b6a39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b640b8b90 0x7f4b641f2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56082/0 (socket says 172.21.15.98:56082) 2023-03-03T07:58:47.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.746+0000 7f4b6a39c700 1 -- 172.21.15.98:0/2622288679 learned_addr learned my addr 172.21.15.98:0/2622288679 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:47.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.746+0000 7f4b69b9b700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b640b5410 0x7f4b640cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:47.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.746+0000 7f4b6a39c700 1 -- 172.21.15.98:0/2622288679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b640b5410 msgr2=0x7f4b640cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:47.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.746+0000 7f4b6939a700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b640b87c0 0x7f4b641f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:47.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.746+0000 7f4b6a39c700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b640b5410 0x7f4b640cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:47.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.746+0000 7f4b6a39c700 1 -- 172.21.15.98:0/2622288679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b640b87c0 msgr2=0x7f4b641f2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:47.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.746+0000 7f4b6a39c700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b640b87c0 0x7f4b641f2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:47.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.746+0000 7f4b6a39c700 1 -- 172.21.15.98:0/2622288679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b60009550 con 0x7f4b640b8b90 2023-03-03T07:58:47.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.746+0000 7f4b69b9b700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b640b5410 0x7f4b640cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:47.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.746+0000 7f4b6939a700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b640b87c0 0x7f4b641f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:47.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.746+0000 7f4b6a39c700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b640b8b90 0x7f4b641f2580 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00cef0 tx=0x7f4b5c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:47.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.747+0000 7f4b5a7fc700 1 -- 172.21.15.98:0/2622288679 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c00ae50 con 0x7f4b640b8b90 2023-03-03T07:58:47.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.747+0000 7f4b5a7fc700 1 -- 172.21.15.98:0/2622288679 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5c015440 con 0x7f4b640b8b90 2023-03-03T07:58:47.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.747+0000 7f4b5a7fc700 1 -- 172.21.15.98:0/2622288679 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c0157f0 con 0x7f4b640b8b90 2023-03-03T07:58:47.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.747+0000 7f4b6bdff700 1 -- 172.21.15.98:0/2622288679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b640cdf30 con 0x7f4b640b8b90 2023-03-03T07:58:47.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.747+0000 7f4b6bdff700 1 -- 172.21.15.98:0/2622288679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b64073a80 con 0x7f4b640b8b90 2023-03-03T07:58:47.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.748+0000 7f4b6bdff700 1 -- 172.21.15.98:0/2622288679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b640ce0c0 con 0x7f4b640b8b90 2023-03-03T07:58:47.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.749+0000 7f4b5a7fc700 1 -- 172.21.15.98:0/2622288679 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f4b5c01c030 con 0x7f4b640b8b90 2023-03-03T07:58:47.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.750+0000 7f4b5a7fc700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4b480683c0 0x7f4b4806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:47.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.750+0000 7f4b5a7fc700 1 -- 172.21.15.98:0/2622288679 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f4b5c0863b0 con 0x7f4b640b8b90 2023-03-03T07:58:47.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.751+0000 7f4b69b9b700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4b480683c0 0x7f4b4806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:47.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.751+0000 7f4b69b9b700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4b480683c0 0x7f4b4806a870 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009c10 tx=0x7f4b60008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:47.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.755+0000 7f4b5a7fc700 1 -- 172.21.15.98:0/2622288679 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4b5c052a10 con 0x7f4b640b8b90 2023-03-03T07:58:47.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.989+0000 7f4b6bdff700 1 -- 172.21.15.98:0/2622288679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b640729b0 con 0x7f4b640b8b90 2023-03-03T07:58:47.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.990+0000 7f4b5a7fc700 1 -- 172.21.15.98:0/2622288679 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f4b5c056030 con 0x7f4b640b8b90 2023-03-03T07:58:47.992 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:47.992 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:58:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.992+0000 7f4b6bdff700 1 -- 172.21.15.98:0/2622288679 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4b480683c0 msgr2=0x7f4b4806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.992+0000 7f4b6bdff700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4b480683c0 0x7f4b4806a870 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009c10 tx=0x7f4b60008030 comp rx=0 tx=0).stop 2023-03-03T07:58:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.993+0000 7f4b6bdff700 1 -- 172.21.15.98:0/2622288679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b640b8b90 msgr2=0x7f4b641f2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.993+0000 7f4b6bdff700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b640b8b90 0x7f4b641f2580 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00cef0 tx=0x7f4b5c00a340 comp rx=0 tx=0).stop 2023-03-03T07:58:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.993+0000 7f4b6bdff700 1 -- 172.21.15.98:0/2622288679 shutdown_connections 2023-03-03T07:58:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.993+0000 7f4b6bdff700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4b480683c0 0x7f4b4806a870 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.993+0000 7f4b6bdff700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b640b8b90 0x7f4b641f2580 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.993+0000 7f4b6bdff700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b640b5410 0x7f4b640cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.993+0000 7f4b6bdff700 1 --2- 172.21.15.98:0/2622288679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b640b87c0 0x7f4b641f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.993+0000 7f4b6bdff700 1 -- 172.21.15.98:0/2622288679 >> 172.21.15.98:0/2622288679 conn(0x7f4b6406aee0 msgr2=0x7f4b640b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.993+0000 7f4b6bdff700 1 -- 172.21.15.98:0/2622288679 shutdown_connections 2023-03-03T07:58:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:47.993+0000 7f4b6bdff700 1 -- 172.21.15.98:0/2622288679 wait complete. 2023-03-03T07:58:47.996 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T07:58:48.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2023-03-03T07:58:48.295 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2023-03-03T07:58:48.296 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2023-03-03T07:58:48.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000394155 2023-03-03T07:58:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.547+0000 7fde87dff700 1 Processor -- start 2023-03-03T07:58:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.547+0000 7fde87dff700 1 -- start start 2023-03-03T07:58:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.549+0000 7fde87dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8015ca70 0x7fde800cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.549+0000 7fde87dff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fde800d0270 0x7fde800d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.549+0000 7fde87dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde800cf980 con 0x7fde8006b4b0 2023-03-03T07:58:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.549+0000 7fde87dff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde800cd5b0 con 0x7fde800d0270 2023-03-03T07:58:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.549+0000 7fde87dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde800d1800 con 0x7fde8015ca70 2023-03-03T07:58:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.549+0000 7fde8639c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fde800d0270 0x7fde800d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.549+0000 7fde8639c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fde800d0270 0x7fde800d1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56090/0 (socket says 172.21.15.98:56090) 2023-03-03T07:58:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.549+0000 7fde8639c700 1 -- 172.21.15.98:0/1796786603 learned_addr learned my addr 172.21.15.98:0/1796786603 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.550+0000 7fde85b9b700 1 --2- 172.21.15.98:0/1796786603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8015ca70 0x7fde800cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.550+0000 7fde8639c700 1 -- 172.21.15.98:0/1796786603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8015ca70 msgr2=0x7fde800cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.550+0000 7fde8639c700 1 --2- 172.21.15.98:0/1796786603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8015ca70 0x7fde800cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.550+0000 7fde8639c700 1 -- 172.21.15.98:0/1796786603 >> v1:172.21.15.100:6789/0 conn(0x7fde8006b4b0 legacy=0x7fde800cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.550+0000 7fde8639c700 1 -- 172.21.15.98:0/1796786603 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde800cd900 con 0x7fde800d0270 2023-03-03T07:58:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.550+0000 7fde8639c700 1 --2- 172.21.15.98:0/1796786603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde800d0270 0x7fde800d1fc0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fde78009e50 tx=0x7fde7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22bae79faec34ed server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.550+0000 7fde84b99700 1 -- 172.21.15.98:0/1796786603 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7800e070 con 0x7fde800d0270 2023-03-03T07:58:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.550+0000 7fde84b99700 1 -- 172.21.15.98:0/1796786603 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7800b9c0 con 0x7fde800d0270 2023-03-03T07:58:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.550+0000 7fde84b99700 1 -- 172.21.15.98:0/1796786603 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7800bd70 con 0x7fde800d0270 2023-03-03T07:58:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.551+0000 7fde87dff700 1 -- 172.21.15.98:0/1796786603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde800d0270 msgr2=0x7fde800d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.551+0000 7fde87dff700 1 --2- 172.21.15.98:0/1796786603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde800d0270 0x7fde800d1fc0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fde78009e50 tx=0x7fde7800b330 comp rx=0 tx=0).stop 2023-03-03T07:58:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.551+0000 7fde87dff700 1 -- 172.21.15.98:0/1796786603 shutdown_connections 2023-03-03T07:58:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.551+0000 7fde87dff700 1 --2- 172.21.15.98:0/1796786603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde800d0270 0x7fde800d1fc0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.551+0000 7fde87dff700 1 --2- 172.21.15.98:0/1796786603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8015ca70 0x7fde800cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.551+0000 7fde87dff700 1 -- 172.21.15.98:0/1796786603 >> 172.21.15.98:0/1796786603 conn(0x7fde80073e70 msgr2=0x7fde8006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.551+0000 7fde87dff700 1 -- 172.21.15.98:0/1796786603 shutdown_connections 2023-03-03T07:58:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.551+0000 7fde87dff700 1 -- 172.21.15.98:0/1796786603 wait complete. 2023-03-03T07:58:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.553+0000 7fde87dff700 1 Processor -- start 2023-03-03T07:58:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.553+0000 7fde87dff700 1 -- start start 2023-03-03T07:58:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.554+0000 7fde87dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8006b4b0 0x7fde8015fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.554+0000 7fde87dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde800d0270 0x7fde801556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.554+0000 7fde87dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde8015ca70 0x7fde801e52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.554+0000 7fde87dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800b4800 con 0x7fde800d0270 2023-03-03T07:58:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.554+0000 7fde87dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800b4aa0 con 0x7fde8015ca70 2023-03-03T07:58:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.554+0000 7fde87dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde800b4d80 con 0x7fde8006b4b0 2023-03-03T07:58:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.555+0000 7fde8539a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde800d0270 0x7fde801556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.555+0000 7fde8539a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde800d0270 0x7fde801556d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56102/0 (socket says 172.21.15.98:56102) 2023-03-03T07:58:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.555+0000 7fde8539a700 1 -- 172.21.15.98:0/798506733 learned_addr learned my addr 172.21.15.98:0/798506733 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.555+0000 7fde85b9b700 1 --2- 172.21.15.98:0/798506733 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8006b4b0 0x7fde8015fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.555+0000 7fde8639c700 1 --2- 172.21.15.98:0/798506733 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde8015ca70 0x7fde801e52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.555+0000 7fde8539a700 1 -- 172.21.15.98:0/798506733 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8006b4b0 msgr2=0x7fde8015fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.555+0000 7fde8539a700 1 --2- 172.21.15.98:0/798506733 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8006b4b0 0x7fde8015fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.555+0000 7fde8539a700 1 -- 172.21.15.98:0/798506733 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde8015ca70 msgr2=0x7fde801e52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.555+0000 7fde8539a700 1 --2- 172.21.15.98:0/798506733 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde8015ca70 0x7fde801e52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.555+0000 7fde8539a700 1 -- 172.21.15.98:0/798506733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde78009b30 con 0x7fde800d0270 2023-03-03T07:58:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.555+0000 7fde8639c700 1 --2- 172.21.15.98:0/798506733 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde8015ca70 0x7fde801e52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:58:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.555+0000 7fde8539a700 1 --2- 172.21.15.98:0/798506733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde800d0270 0x7fde801556d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fde7000c420 tx=0x7fde7000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.556+0000 7fde767fc700 1 -- 172.21.15.98:0/798506733 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde700052d0 con 0x7fde800d0270 2023-03-03T07:58:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.556+0000 7fde767fc700 1 -- 172.21.15.98:0/798506733 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde70005430 con 0x7fde800d0270 2023-03-03T07:58:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.556+0000 7fde767fc700 1 -- 172.21.15.98:0/798506733 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde700038c0 con 0x7fde800d0270 2023-03-03T07:58:48.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.556+0000 7fde87dff700 1 -- 172.21.15.98:0/798506733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde800d1aa0 con 0x7fde800d0270 2023-03-03T07:58:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.556+0000 7fde87dff700 1 -- 172.21.15.98:0/798506733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde800cdbe0 con 0x7fde800d0270 2023-03-03T07:58:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.557+0000 7fde87dff700 1 -- 172.21.15.98:0/798506733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8015ce40 con 0x7fde800d0270 2023-03-03T07:58:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.558+0000 7fde767fc700 1 -- 172.21.15.98:0/798506733 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fde70019030 con 0x7fde800d0270 2023-03-03T07:58:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.559+0000 7fde767fc700 1 --2- 172.21.15.98:0/798506733 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fde640683c0 0x7fde6406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.559+0000 7fde767fc700 1 -- 172.21.15.98:0/798506733 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fde700851a0 con 0x7fde800d0270 2023-03-03T07:58:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.560+0000 7fde85b9b700 1 --2- 172.21.15.98:0/798506733 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fde640683c0 0x7fde6406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.561+0000 7fde85b9b700 1 --2- 172.21.15.98:0/798506733 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fde640683c0 0x7fde6406a870 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009930 tx=0x7fde7c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.564+0000 7fde767fc700 1 -- 172.21.15.98:0/798506733 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fde70051800 con 0x7fde800d0270 2023-03-03T07:58:48.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.788+0000 7fde87dff700 1 -- 172.21.15.98:0/798506733 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000394155", "target": ["mon-mgr", ""]}) v1 -- 0x7fde800b4550 con 0x7fde640683c0 2023-03-03T07:58:48.796 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:58:48.795+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:58:48.796 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:58:48.795+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:58:48.796 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:58:48.795+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:58:48.796 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:58:48.795+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:58:48.797 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T07:58:48.795+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T07:58:48.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.829+0000 7fde767fc700 1 -- 172.21.15.98:0/798506733 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fde800b4550 con 0x7fde640683c0 2023-03-03T07:58:48.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.831+0000 7fde87dff700 1 -- 172.21.15.98:0/798506733 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fde640683c0 msgr2=0x7fde6406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.831+0000 7fde87dff700 1 --2- 172.21.15.98:0/798506733 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fde640683c0 0x7fde6406a870 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009930 tx=0x7fde7c008040 comp rx=0 tx=0).stop 2023-03-03T07:58:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.831+0000 7fde87dff700 1 -- 172.21.15.98:0/798506733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde800d0270 msgr2=0x7fde801556d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.831+0000 7fde87dff700 1 --2- 172.21.15.98:0/798506733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde800d0270 0x7fde801556d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fde7000c420 tx=0x7fde7000c7e0 comp rx=0 tx=0).stop 2023-03-03T07:58:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.831+0000 7fde87dff700 1 -- 172.21.15.98:0/798506733 shutdown_connections 2023-03-03T07:58:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.831+0000 7fde87dff700 1 --2- 172.21.15.98:0/798506733 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fde640683c0 0x7fde6406a870 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:48.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.831+0000 7fde87dff700 1 --2- 172.21.15.98:0/798506733 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde8015ca70 0x7fde801e52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:48.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.831+0000 7fde87dff700 1 --2- 172.21.15.98:0/798506733 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde8006b4b0 0x7fde8015fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:48.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.831+0000 7fde87dff700 1 --2- 172.21.15.98:0/798506733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde800d0270 0x7fde801556d0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:48.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.831+0000 7fde87dff700 1 -- 172.21.15.98:0/798506733 >> 172.21.15.98:0/798506733 conn(0x7fde80073e70 msgr2=0x7fde800d0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:48.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.831+0000 7fde87dff700 1 -- 172.21.15.98:0/798506733 shutdown_connections 2023-03-03T07:58:48.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:48.832+0000 7fde87dff700 1 -- 172.21.15.98:0/798506733 wait complete. 2023-03-03T07:58:48.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:58:49.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.105+0000 7f334759e700 1 Processor -- start 2023-03-03T07:58:49.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.106+0000 7f334759e700 1 -- start start 2023-03-03T07:58:49.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.107+0000 7f334759e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3348146360 0x7f3348160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:49.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.107+0000 7f334759e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33480b4ab0 0x7f33480b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:49.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.107+0000 7f334759e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33481614c0 con 0x7f33480731e0 2023-03-03T07:58:49.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.107+0000 7f334759e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33480b5440 con 0x7f3348146360 2023-03-03T07:58:49.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.107+0000 7f334759e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33480d10f0 con 0x7f33480b4ab0 2023-03-03T07:58:49.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.108+0000 7f3346d9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f33480731e0 0x7f3348073660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55266/0 (socket says 172.21.15.98:55266) 2023-03-03T07:58:49.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.108+0000 7f3346d9d700 1 -- 172.21.15.98:0/2695952889 learned_addr learned my addr 172.21.15.98:0/2695952889 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:49.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.108+0000 7f334659c700 1 --2- 172.21.15.98:0/2695952889 >> v2:172.21.15.100:3300/0 conn(0x7f3348146360 0x7f3348160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:49.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.108+0000 7f3345d9b700 1 --2- 172.21.15.98:0/2695952889 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33480b4ab0 0x7f33480b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:49.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.108+0000 7f334559a700 1 -- 172.21.15.98:0/2695952889 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992595537 0 0) 0x7f3340001490 con 0x7f33480731e0 2023-03-03T07:58:49.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.108+0000 7f334559a700 1 -- 172.21.15.98:0/2695952889 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33300035b0 con 0x7f33480731e0 2023-03-03T07:58:49.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.108+0000 7f3345d9b700 1 -- 172.21.15.98:0/2695952889 >> v2:172.21.15.100:3300/0 conn(0x7f3348146360 msgr2=0x7f3348160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:49.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.108+0000 7f3345d9b700 1 --2- 172.21.15.98:0/2695952889 >> v2:172.21.15.100:3300/0 conn(0x7f3348146360 0x7f3348160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:49.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.108+0000 7f3345d9b700 1 -- 172.21.15.98:0/2695952889 >> v1:172.21.15.98:6789/0 conn(0x7f33480731e0 legacy=0x7f3348073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:49.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.108+0000 7f3345d9b700 1 -- 172.21.15.98:0/2695952889 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33480cdf20 con 0x7f33480b4ab0 2023-03-03T07:58:49.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.108+0000 7f334659c700 1 --2- 172.21.15.98:0/2695952889 >> v2:172.21.15.100:3300/0 conn(0x7f3348146360 0x7f3348160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:58:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.109+0000 7f3345d9b700 1 --2- 172.21.15.98:0/2695952889 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33480b4ab0 0x7f33480b4f00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f333c009840 tx=0x7f333c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d48e840e5ed178a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.109+0000 7f334559a700 1 -- 172.21.15.98:0/2695952889 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c011830 con 0x7f33480b4ab0 2023-03-03T07:58:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.109+0000 7f334559a700 1 -- 172.21.15.98:0/2695952889 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c011990 con 0x7f33480b4ab0 2023-03-03T07:58:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.109+0000 7f334559a700 1 -- 172.21.15.98:0/2695952889 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c011d00 con 0x7f33480b4ab0 2023-03-03T07:58:49.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.110+0000 7f334759e700 1 -- 172.21.15.98:0/2695952889 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33480b4ab0 msgr2=0x7f33480b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:49.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.110+0000 7f334759e700 1 --2- 172.21.15.98:0/2695952889 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33480b4ab0 0x7f33480b4f00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f333c009840 tx=0x7f333c009b50 comp rx=0 tx=0).stop 2023-03-03T07:58:49.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.110+0000 7f334759e700 1 -- 172.21.15.98:0/2695952889 shutdown_connections 2023-03-03T07:58:49.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.110+0000 7f334759e700 1 --2- 172.21.15.98:0/2695952889 >> v2:172.21.15.100:3300/0 conn(0x7f3348146360 0x7f3348160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:49.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.110+0000 7f334759e700 1 --2- 172.21.15.98:0/2695952889 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33480b4ab0 0x7f33480b4f00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:49.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.110+0000 7f334759e700 1 -- 172.21.15.98:0/2695952889 >> 172.21.15.98:0/2695952889 conn(0x7f334806b8e0 msgr2=0x7f33480cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:49.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.110+0000 7f334759e700 1 -- 172.21.15.98:0/2695952889 shutdown_connections 2023-03-03T07:58:49.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.110+0000 7f334759e700 1 -- 172.21.15.98:0/2695952889 wait complete. 2023-03-03T07:58:49.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.112+0000 7f334759e700 1 Processor -- start 2023-03-03T07:58:49.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.112+0000 7f334759e700 1 -- start start 2023-03-03T07:58:49.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.113+0000 7f334759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33480731e0 0x7f33480cc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:49.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.113+0000 7f334759e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33480b4ab0 0x7f33480735b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:49.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.113+0000 7f334759e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3348146360 0x7f3348073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:49.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.113+0000 7f334759e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33480c7930 con 0x7f33480731e0 2023-03-03T07:58:49.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.113+0000 7f334759e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33480c7c10 con 0x7f3348146360 2023-03-03T07:58:49.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.113+0000 7f334759e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33480c7ef0 con 0x7f33480b4ab0 2023-03-03T07:58:49.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.113+0000 7f334659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33480731e0 0x7f33480cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:49.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.113+0000 7f334659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33480731e0 0x7f33480cc780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56124/0 (socket says 172.21.15.98:56124) 2023-03-03T07:58:49.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.113+0000 7f334659c700 1 -- 172.21.15.98:0/183277930 learned_addr learned my addr 172.21.15.98:0/183277930 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:49.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.114+0000 7f334659c700 1 -- 172.21.15.98:0/183277930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33480b4ab0 msgr2=0x7f33480735b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:49.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.114+0000 7f334659c700 1 --2- 172.21.15.98:0/183277930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33480b4ab0 0x7f33480735b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:49.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.114+0000 7f334659c700 1 -- 172.21.15.98:0/183277930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3348146360 msgr2=0x7f3348073af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:49.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.114+0000 7f334659c700 1 --2- 172.21.15.98:0/183277930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3348146360 0x7f3348073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:49.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.114+0000 7f334659c700 1 -- 172.21.15.98:0/183277930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33380094a0 con 0x7f33480731e0 2023-03-03T07:58:49.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.114+0000 7f334659c700 1 --2- 172.21.15.98:0/183277930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33480731e0 0x7f33480cc780 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f333800ebe0 tx=0x7f3338009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:49.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.114+0000 7f3336ffd700 1 -- 172.21.15.98:0/183277930 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333800c8b0 con 0x7f33480731e0 2023-03-03T07:58:49.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.114+0000 7f3336ffd700 1 -- 172.21.15.98:0/183277930 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333800ca10 con 0x7f33480731e0 2023-03-03T07:58:49.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.114+0000 7f3336ffd700 1 -- 172.21.15.98:0/183277930 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333800cda0 con 0x7f33480731e0 2023-03-03T07:58:49.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.114+0000 7f334759e700 1 -- 172.21.15.98:0/183277930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333c0094a0 con 0x7f33480731e0 2023-03-03T07:58:49.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.115+0000 7f334759e700 1 -- 172.21.15.98:0/183277930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33480cd960 con 0x7f33480731e0 2023-03-03T07:58:49.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.116+0000 7f334759e700 1 -- 172.21.15.98:0/183277930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33480cd680 con 0x7f33480731e0 2023-03-03T07:58:49.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.117+0000 7f3336ffd700 1 -- 172.21.15.98:0/183277930 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f3338019030 con 0x7f33480731e0 2023-03-03T07:58:49.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.118+0000 7f3336ffd700 1 --2- 172.21.15.98:0/183277930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33280683c0 0x7f332806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:49.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.118+0000 7f3336ffd700 1 -- 172.21.15.98:0/183277930 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f3338085ff0 con 0x7f33480731e0 2023-03-03T07:58:49.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.119+0000 7f3345d9b700 1 --2- 172.21.15.98:0/183277930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33280683c0 0x7f332806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:49.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.119+0000 7f3345d9b700 1 --2- 172.21.15.98:0/183277930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33280683c0 0x7f332806a870 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f33481552d0 tx=0x7f333c010000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:49.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.122+0000 7f3336ffd700 1 -- 172.21.15.98:0/183277930 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f33380525e0 con 0x7f33480731e0 2023-03-03T07:58:49.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.358+0000 7f334759e700 1 -- 172.21.15.98:0/183277930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33480cdc40 con 0x7f33480731e0 2023-03-03T07:58:49.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.359+0000 7f3336ffd700 1 -- 172.21.15.98:0/183277930 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f3338021090 con 0x7f33480731e0 2023-03-03T07:58:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:49.362 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:58:49.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.361+0000 7f334759e700 1 -- 172.21.15.98:0/183277930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33280683c0 msgr2=0x7f332806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:49.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.361+0000 7f334759e700 1 --2- 172.21.15.98:0/183277930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33280683c0 0x7f332806a870 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f33481552d0 tx=0x7f333c010000 comp rx=0 tx=0).stop 2023-03-03T07:58:49.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.361+0000 7f334759e700 1 -- 172.21.15.98:0/183277930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33480731e0 msgr2=0x7f33480cc780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:49.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.361+0000 7f334759e700 1 --2- 172.21.15.98:0/183277930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33480731e0 0x7f33480cc780 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f333800ebe0 tx=0x7f3338009e20 comp rx=0 tx=0).stop 2023-03-03T07:58:49.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.362+0000 7f334759e700 1 -- 172.21.15.98:0/183277930 shutdown_connections 2023-03-03T07:58:49.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.362+0000 7f334759e700 1 --2- 172.21.15.98:0/183277930 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33280683c0 0x7f332806a870 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:49.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.362+0000 7f334759e700 1 --2- 172.21.15.98:0/183277930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33480731e0 0x7f33480cc780 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:49.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.362+0000 7f334759e700 1 --2- 172.21.15.98:0/183277930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3348146360 0x7f3348073af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:49.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.362+0000 7f334759e700 1 --2- 172.21.15.98:0/183277930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33480b4ab0 0x7f33480735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:49.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.362+0000 7f334759e700 1 -- 172.21.15.98:0/183277930 >> 172.21.15.98:0/183277930 conn(0x7f334806b8e0 msgr2=0x7f33480b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:49.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.362+0000 7f334759e700 1 -- 172.21.15.98:0/183277930 shutdown_connections 2023-03-03T07:58:49.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.362+0000 7f334759e700 1 -- 172.21.15.98:0/183277930 wait complete. 2023-03-03T07:58:49.365 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T07:58:49.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2023-03-03T07:58:49.665 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":6120,"entity":{"name":{"type":"client","num":6120},"addr":{"type":"any","addr":"172.21.15.98:0","nonce":1023372945}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":14,"uptime":0.843989447,"requests_in_flight":2,"num_completed_requests":0,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.957740710126505,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.6120 172.21.15.98:0/1023372945","completed_requests":[],"prealloc_inos":[{"start":"0x10000000005","length":996}],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"y","hostname":"smithi098","pid":"108777","root":"/"}}] 2023-03-03T07:58:49.690 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "y", "hostname": "smithi098", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108777", "root": "/" }, "completed_requests": [], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.98:0", "nonce": 1023372945, "type": "any" }, "name": { "num": 6120, "type": "client" } }, "id": 6120, "inst": "client.6120 172.21.15.98:0/1023372945", "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 996, "start": "0x10000000005" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 14, "requests_in_flight": 2, "session_cache_liveness": { "halflife": 300, "value": 21.957740710126505 }, "state": "open", "uptime": 0.843989447 } ] 2023-03-03T07:58:49.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:58:49.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.939+0000 7f98f2b9d700 1 Processor -- start 2023-03-03T07:58:49.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.939+0000 7f98f2b9d700 1 -- start start 2023-03-03T07:58:49.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.940+0000 7f98f2b9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f98ec146360 0x7f98ec160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:49.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.940+0000 7f98f2b9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ec1614c0 con 0x7f98ec0b4ab0 2023-03-03T07:58:49.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.941+0000 7f98f2b9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ec0d10f0 con 0x7f98ec0d2590 2023-03-03T07:58:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.941+0000 7f98f2b9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ec0cdf20 con 0x7f98ec146360 2023-03-03T07:58:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.941+0000 7f98f1b9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f98ec146360 0x7f98ec160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.941+0000 7f98f1b9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f98ec146360 0x7f98ec160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56136/0 (socket says 172.21.15.98:56136) 2023-03-03T07:58:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.941+0000 7f98f1b9b700 1 -- 172.21.15.98:0/1942835602 learned_addr learned my addr 172.21.15.98:0/1942835602 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.941+0000 7f98f139a700 1 --1- 172.21.15.98:0/1942835602 >> v1:172.21.15.98:6789/0 conn(0x7f98ec0b4ab0 0x7f98ec0731e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55276/0 (socket says 172.21.15.98:55276) 2023-03-03T07:58:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.941+0000 7f98f1b9b700 1 -- 172.21.15.98:0/1942835602 >> v1:172.21.15.100:6789/0 conn(0x7f98ec0d2590 legacy=0x7f98ec0738f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.941+0000 7f98f1b9b700 1 -- 172.21.15.98:0/1942835602 >> v1:172.21.15.98:6789/0 conn(0x7f98ec0b4ab0 legacy=0x7f98ec0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:49.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.941+0000 7f98f1b9b700 1 -- 172.21.15.98:0/1942835602 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ec074000 con 0x7f98ec146360 2023-03-03T07:58:49.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.941+0000 7f98f0b99700 1 -- 172.21.15.98:0/1942835602 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689986386 0 0) 0x7f98dc001110 con 0x7f98ec0b4ab0 2023-03-03T07:58:49.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.942+0000 7f98f1b9b700 1 --2- 172.21.15.98:0/1942835602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec146360 0x7f98ec160f80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f98e80098f0 tx=0x7f98e8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d686baa67478942b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:49.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.942+0000 7f98f0b99700 1 -- 172.21.15.98:0/1942835602 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e800b8a0 con 0x7f98ec146360 2023-03-03T07:58:49.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.942+0000 7f98f0b99700 1 -- 172.21.15.98:0/1942835602 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e800ba00 con 0x7f98ec146360 2023-03-03T07:58:49.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.942+0000 7f98f0b99700 1 -- 172.21.15.98:0/1942835602 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e800bd90 con 0x7f98ec146360 2023-03-03T07:58:49.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.942+0000 7f98f2b9d700 1 -- 172.21.15.98:0/1942835602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec146360 msgr2=0x7f98ec160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:49.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.942+0000 7f98f2b9d700 1 --2- 172.21.15.98:0/1942835602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec146360 0x7f98ec160f80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f98e80098f0 tx=0x7f98e8009c00 comp rx=0 tx=0).stop 2023-03-03T07:58:49.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.943+0000 7f98f2b9d700 1 -- 172.21.15.98:0/1942835602 shutdown_connections 2023-03-03T07:58:49.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.943+0000 7f98f2b9d700 1 --2- 172.21.15.98:0/1942835602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec146360 0x7f98ec160f80 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:49.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.943+0000 7f98f2b9d700 1 -- 172.21.15.98:0/1942835602 >> 172.21.15.98:0/1942835602 conn(0x7f98ec06b8e0 msgr2=0x7f98ec0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:49.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.943+0000 7f98f2b9d700 1 -- 172.21.15.98:0/1942835602 shutdown_connections 2023-03-03T07:58:49.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.943+0000 7f98f2b9d700 1 -- 172.21.15.98:0/1942835602 wait complete. 2023-03-03T07:58:49.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.945+0000 7f98f2b9d700 1 Processor -- start 2023-03-03T07:58:49.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.945+0000 7f98f2b9d700 1 -- start start 2023-03-03T07:58:49.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.946+0000 7f98f2b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ec0b4ab0 0x7f98ec156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:49.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.946+0000 7f98f2b9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec0d2590 0x7f98ec0c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:49.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.946+0000 7f98f2b9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98ec146360 0x7f98ec0c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:49.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.946+0000 7f98f2b9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec0c6870 con 0x7f98ec0d2590 2023-03-03T07:58:49.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.946+0000 7f98f2b9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec1f2450 con 0x7f98ec146360 2023-03-03T07:58:49.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.946+0000 7f98f2b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec0c76b0 con 0x7f98ec0b4ab0 2023-03-03T07:58:49.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.946+0000 7f98f139a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec0d2590 0x7f98ec0c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.946+0000 7f98f139a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec0d2590 0x7f98ec0c6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56152/0 (socket says 172.21.15.98:56152) 2023-03-03T07:58:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.946+0000 7f98f139a700 1 -- 172.21.15.98:0/3075282816 learned_addr learned my addr 172.21.15.98:0/3075282816 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.946+0000 7f98f1b9b700 1 --2- 172.21.15.98:0/3075282816 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ec0b4ab0 0x7f98ec156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.946+0000 7f98f239c700 1 --2- 172.21.15.98:0/3075282816 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98ec146360 0x7f98ec0c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.947+0000 7f98f139a700 1 -- 172.21.15.98:0/3075282816 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ec0b4ab0 msgr2=0x7f98ec156e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.947+0000 7f98f139a700 1 --2- 172.21.15.98:0/3075282816 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ec0b4ab0 0x7f98ec156e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.947+0000 7f98f139a700 1 -- 172.21.15.98:0/3075282816 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98ec146360 msgr2=0x7f98ec0c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.947+0000 7f98f139a700 1 --2- 172.21.15.98:0/3075282816 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98ec146360 0x7f98ec0c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.947+0000 7f98f139a700 1 -- 172.21.15.98:0/3075282816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e8009550 con 0x7f98ec0d2590 2023-03-03T07:58:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.947+0000 7f98f139a700 1 --2- 172.21.15.98:0/3075282816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec0d2590 0x7f98ec0c6b40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00c520 tx=0x7f98dc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.947+0000 7f98e27fc700 1 -- 172.21.15.98:0/3075282816 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc004c80 con 0x7f98ec0d2590 2023-03-03T07:58:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.947+0000 7f98e27fc700 1 -- 172.21.15.98:0/3075282816 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98dc004de0 con 0x7f98ec0d2590 2023-03-03T07:58:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.947+0000 7f98f2b9d700 1 -- 172.21.15.98:0/3075282816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ec0cd960 con 0x7f98ec0d2590 2023-03-03T07:58:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.947+0000 7f98e27fc700 1 -- 172.21.15.98:0/3075282816 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc007310 con 0x7f98ec0d2590 2023-03-03T07:58:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.947+0000 7f98f2b9d700 1 -- 172.21.15.98:0/3075282816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ec0cd680 con 0x7f98ec0d2590 2023-03-03T07:58:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.949+0000 7f98f2b9d700 1 -- 172.21.15.98:0/3075282816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ec0cdc40 con 0x7f98ec0d2590 2023-03-03T07:58:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.950+0000 7f98e27fc700 1 -- 172.21.15.98:0/3075282816 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f98dc01a020 con 0x7f98ec0d2590 2023-03-03T07:58:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.951+0000 7f98e27fc700 1 --2- 172.21.15.98:0/3075282816 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f98d00683c0 0x7f98d006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:49.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.951+0000 7f98e27fc700 1 -- 172.21.15.98:0/3075282816 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f98dc085770 con 0x7f98ec0d2590 2023-03-03T07:58:49.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.952+0000 7f98f1b9b700 1 --2- 172.21.15.98:0/3075282816 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f98d00683c0 0x7f98d006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:49.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.952+0000 7f98f1b9b700 1 --2- 172.21.15.98:0/3075282816 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f98d00683c0 0x7f98d006a870 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f98e8000c00 tx=0x7f98e8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:49.955+0000 7f98e27fc700 1 -- 172.21.15.98:0/3075282816 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f98dc051dd0 con 0x7f98ec0d2590 2023-03-03T07:58:50.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:50.189+0000 7f98f2b9d700 1 -- 172.21.15.98:0/3075282816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98ec1614c0 con 0x7f98ec0d2590 2023-03-03T07:58:50.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:50.190+0000 7f98e27fc700 1 -- 172.21.15.98:0/3075282816 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f98ec1614c0 con 0x7f98ec0d2590 2023-03-03T07:58:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:50.192 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:58:50.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:50.193+0000 7f98f2b9d700 1 -- 172.21.15.98:0/3075282816 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f98d00683c0 msgr2=0x7f98d006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:50.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:50.193+0000 7f98f2b9d700 1 --2- 172.21.15.98:0/3075282816 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f98d00683c0 0x7f98d006a870 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f98e8000c00 tx=0x7f98e8008030 comp rx=0 tx=0).stop 2023-03-03T07:58:50.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:50.193+0000 7f98f2b9d700 1 -- 172.21.15.98:0/3075282816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec0d2590 msgr2=0x7f98ec0c6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:50.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:50.193+0000 7f98f2b9d700 1 --2- 172.21.15.98:0/3075282816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec0d2590 0x7f98ec0c6b40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00c520 tx=0x7f98dc00c830 comp rx=0 tx=0).stop 2023-03-03T07:58:50.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:50.193+0000 7f98f2b9d700 1 -- 172.21.15.98:0/3075282816 shutdown_connections 2023-03-03T07:58:50.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:50.193+0000 7f98f2b9d700 1 --2- 172.21.15.98:0/3075282816 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f98d00683c0 0x7f98d006a870 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:50.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:50.193+0000 7f98f2b9d700 1 --2- 172.21.15.98:0/3075282816 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98ec146360 0x7f98ec0c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:50.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:50.193+0000 7f98f2b9d700 1 --2- 172.21.15.98:0/3075282816 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98ec0b4ab0 0x7f98ec156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:50.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:50.193+0000 7f98f2b9d700 1 --2- 172.21.15.98:0/3075282816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec0d2590 0x7f98ec0c6b40 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:50.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:50.193+0000 7f98f2b9d700 1 -- 172.21.15.98:0/3075282816 >> 172.21.15.98:0/3075282816 conn(0x7f98ec06b8e0 msgr2=0x7f98ec15ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:50.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:50.193+0000 7f98f2b9d700 1 -- 172.21.15.98:0/3075282816 shutdown_connections 2023-03-03T07:58:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:50.193+0000 7f98f2b9d700 1 -- 172.21.15.98:0/3075282816 wait complete. 2023-03-03T07:58:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T07:58:50.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2023-03-03T07:58:50.495 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":6120,"entity":{"name":{"type":"client","num":6120},"addr":{"type":"any","addr":"172.21.15.98:0","nonce":1023372945}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":14,"uptime":1.674979056,"requests_in_flight":2,"num_completed_requests":0,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.915622433047101,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.6120 172.21.15.98:0/1023372945","completed_requests":[],"prealloc_inos":[{"start":"0x10000000005","length":996}],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"y","hostname":"smithi098","pid":"108777","root":"/"}}] 2023-03-03T07:58:50.496 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "y", "hostname": "smithi098", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108777", "root": "/" }, "completed_requests": [], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.98:0", "nonce": 1023372945, "type": "any" }, "name": { "num": 6120, "type": "client" } }, "id": 6120, "inst": "client.6120 172.21.15.98:0/1023372945", "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 996, "start": "0x10000000005" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 14, "requests_in_flight": 2, "session_cache_liveness": { "halflife": 300, "value": 21.9156224330471 }, "state": "open", "uptime": 1.674979056 } ] 2023-03-03T07:58:50.496 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2023-03-03T07:58:55.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:58:55.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.748+0000 7fb743fff700 1 Processor -- start 2023-03-03T07:58:55.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.748+0000 7fb743fff700 1 -- start start 2023-03-03T07:58:55.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.749+0000 7fb743fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7440b51c0 0x7fb744160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:55.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.749+0000 7fb743fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb7440d1580 0x7fb7440d19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.749+0000 7fb743fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7441614c0 con 0x7fb744146360 2023-03-03T07:58:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.749+0000 7fb743fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7440d10f0 con 0x7fb7440d1580 2023-03-03T07:58:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.749+0000 7fb743fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7440cdf20 con 0x7fb7440b51c0 2023-03-03T07:58:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.750+0000 7fb742ffd700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fb744146360 0x7fb7440b4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:51136/0 (socket says 172.21.15.98:51136) 2023-03-03T07:58:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.750+0000 7fb742ffd700 1 -- 172.21.15.98:0/1994582156 learned_addr learned my addr 172.21.15.98:0/1994582156 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:58:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.750+0000 7fb7427fc700 1 --2- 172.21.15.98:0/1994582156 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7440b51c0 0x7fb744160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:55.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.750+0000 7fb7437fe700 1 --2- 172.21.15.98:0/1994582156 >> v2:172.21.15.100:3300/0 conn(0x7fb7440d1580 0x7fb7440d19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.751+0000 7fb741ffb700 1 -- 172.21.15.98:0/1994582156 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069709057 0 0) 0x7fb734001110 con 0x7fb744146360 2023-03-03T07:58:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.751+0000 7fb741ffb700 1 -- 172.21.15.98:0/1994582156 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb72c0035b0 con 0x7fb744146360 2023-03-03T07:58:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.751+0000 7fb7437fe700 1 -- 172.21.15.98:0/1994582156 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7440b51c0 msgr2=0x7fb744160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:58:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.751+0000 7fb7437fe700 1 --2- 172.21.15.98:0/1994582156 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7440b51c0 0x7fb744160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.751+0000 7fb7437fe700 1 -- 172.21.15.98:0/1994582156 >> v1:172.21.15.100:6789/0 conn(0x7fb744146360 legacy=0x7fb7440b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.751+0000 7fb7437fe700 1 -- 172.21.15.98:0/1994582156 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7440cd680 con 0x7fb7440d1580 2023-03-03T07:58:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.751+0000 7fb7427fc700 1 --2- 172.21.15.98:0/1994582156 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7440b51c0 0x7fb744160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:58:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.751+0000 7fb7437fe700 1 --2- 172.21.15.98:0/1994582156 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7440d1580 0x7fb7440d19e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009d90 tx=0x7fb73c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e70f1a3184deb3c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.751+0000 7fb741ffb700 1 -- 172.21.15.98:0/1994582156 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c00e070 con 0x7fb7440d1580 2023-03-03T07:58:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.752+0000 7fb741ffb700 1 -- 172.21.15.98:0/1994582156 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c00bb20 con 0x7fb7440d1580 2023-03-03T07:58:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.752+0000 7fb741ffb700 1 -- 172.21.15.98:0/1994582156 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c012430 con 0x7fb7440d1580 2023-03-03T07:58:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.752+0000 7fb743fff700 1 -- 172.21.15.98:0/1994582156 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7440d1580 msgr2=0x7fb7440d19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.752+0000 7fb743fff700 1 --2- 172.21.15.98:0/1994582156 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7440d1580 0x7fb7440d19e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009d90 tx=0x7fb73c00b330 comp rx=0 tx=0).stop 2023-03-03T07:58:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.752+0000 7fb743fff700 1 -- 172.21.15.98:0/1994582156 shutdown_connections 2023-03-03T07:58:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.752+0000 7fb743fff700 1 --2- 172.21.15.98:0/1994582156 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7440d1580 0x7fb7440d19e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.752+0000 7fb743fff700 1 --2- 172.21.15.98:0/1994582156 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7440b51c0 0x7fb744160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.752+0000 7fb743fff700 1 -- 172.21.15.98:0/1994582156 >> 172.21.15.98:0/1994582156 conn(0x7fb74406b8e0 msgr2=0x7fb7440cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.752+0000 7fb743fff700 1 -- 172.21.15.98:0/1994582156 shutdown_connections 2023-03-03T07:58:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.752+0000 7fb743fff700 1 -- 172.21.15.98:0/1994582156 wait complete. 2023-03-03T07:58:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.754+0000 7fb743fff700 1 Processor -- start 2023-03-03T07:58:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.754+0000 7fb743fff700 1 -- start start 2023-03-03T07:58:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.755+0000 7fb743fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7440b51c0 0x7fb7440c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.755+0000 7fb743fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7440d1580 0x7fb7440b85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.756+0000 7fb743fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb744146360 0x7fb7440c8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.756+0000 7fb743fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb744154d70 con 0x7fb744146360 2023-03-03T07:58:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.756+0000 7fb743fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7440b8c60 con 0x7fb7440b51c0 2023-03-03T07:58:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.756+0000 7fb743fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74415d800 con 0x7fb7440d1580 2023-03-03T07:58:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.756+0000 7fb7437fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb744146360 0x7fb7440c8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.756+0000 7fb7427fc700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7440d1580 0x7fb7440b85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.756+0000 7fb742ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7440b51c0 0x7fb7440c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.756+0000 7fb7437fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb744146360 0x7fb7440c8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40382/0 (socket says 172.21.15.98:40382) 2023-03-03T07:58:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.756+0000 7fb7437fe700 1 -- 172.21.15.98:0/633001292 learned_addr learned my addr 172.21.15.98:0/633001292 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:58:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.756+0000 7fb7437fe700 1 -- 172.21.15.98:0/633001292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7440d1580 msgr2=0x7fb7440b85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.756+0000 7fb7437fe700 1 --2- 172.21.15.98:0/633001292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7440d1580 0x7fb7440b85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.756+0000 7fb7437fe700 1 -- 172.21.15.98:0/633001292 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7440b51c0 msgr2=0x7fb7440c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.756+0000 7fb7437fe700 1 --2- 172.21.15.98:0/633001292 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7440b51c0 0x7fb7440c5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.757+0000 7fb7437fe700 1 -- 172.21.15.98:0/633001292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7380094a0 con 0x7fb744146360 2023-03-03T07:58:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.757+0000 7fb7437fe700 1 --2- 172.21.15.98:0/633001292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb744146360 0x7fb7440c8bc0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb73c014040 tx=0x7fb73c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.757+0000 7fb72bfff700 1 -- 172.21.15.98:0/633001292 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c0129f0 con 0x7fb744146360 2023-03-03T07:58:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.757+0000 7fb72bfff700 1 -- 172.21.15.98:0/633001292 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c020070 con 0x7fb744146360 2023-03-03T07:58:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.757+0000 7fb72bfff700 1 -- 172.21.15.98:0/633001292 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c012db0 con 0x7fb744146360 2023-03-03T07:58:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.757+0000 7fb743fff700 1 -- 172.21.15.98:0/633001292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73c009a70 con 0x7fb744146360 2023-03-03T07:58:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.758+0000 7fb743fff700 1 -- 172.21.15.98:0/633001292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7440cdc40 con 0x7fb744146360 2023-03-03T07:58:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.758+0000 7fb743fff700 1 -- 172.21.15.98:0/633001292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7440cd960 con 0x7fb744146360 2023-03-03T07:58:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.760+0000 7fb72bfff700 1 -- 172.21.15.98:0/633001292 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 86575+0+0 (secure 0 0 0) 0x7fb73c02e070 con 0x7fb744146360 2023-03-03T07:58:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.761+0000 7fb72bfff700 1 --2- 172.21.15.98:0/633001292 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb730068410 0x7fb73006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.761+0000 7fb72bfff700 1 -- 172.21.15.98:0/633001292 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fb73c022070 con 0x7fb744146360 2023-03-03T07:58:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.761+0000 7fb742ffd700 1 --2- 172.21.15.98:0/633001292 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb730068410 0x7fb73006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.762+0000 7fb742ffd700 1 --2- 172.21.15.98:0/633001292 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb730068410 0x7fb73006a8c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb734007fd0 tx=0x7fb734007c40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:55.765+0000 7fb72bfff700 1 -- 172.21.15.98:0/633001292 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb73c0540b0 con 0x7fb744146360 2023-03-03T07:58:56.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:56.004+0000 7fb743fff700 1 -- 172.21.15.98:0/633001292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7440d0d80 con 0x7fb744146360 2023-03-03T07:58:56.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:56.004+0000 7fb72bfff700 1 -- 172.21.15.98:0/633001292 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7fb73c02c090 con 0x7fb744146360 2023-03-03T07:58:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:58:56.006 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:58:56.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:56.007+0000 7fb743fff700 1 -- 172.21.15.98:0/633001292 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb730068410 msgr2=0x7fb73006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:56.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:56.007+0000 7fb743fff700 1 --2- 172.21.15.98:0/633001292 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb730068410 0x7fb73006a8c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb734007fd0 tx=0x7fb734007c40 comp rx=0 tx=0).stop 2023-03-03T07:58:56.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:56.007+0000 7fb743fff700 1 -- 172.21.15.98:0/633001292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb744146360 msgr2=0x7fb7440c8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:56.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:56.007+0000 7fb743fff700 1 --2- 172.21.15.98:0/633001292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb744146360 0x7fb7440c8bc0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb73c014040 tx=0x7fb73c004ad0 comp rx=0 tx=0).stop 2023-03-03T07:58:56.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:56.007+0000 7fb743fff700 1 -- 172.21.15.98:0/633001292 shutdown_connections 2023-03-03T07:58:56.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:56.007+0000 7fb743fff700 1 --2- 172.21.15.98:0/633001292 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb730068410 0x7fb73006a8c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:56.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:56.007+0000 7fb743fff700 1 --2- 172.21.15.98:0/633001292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb744146360 0x7fb7440c8bc0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:56.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:56.007+0000 7fb743fff700 1 --2- 172.21.15.98:0/633001292 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7440b51c0 0x7fb7440c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:56.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:56.007+0000 7fb743fff700 1 --2- 172.21.15.98:0/633001292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7440d1580 0x7fb7440b85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:56.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:56.007+0000 7fb743fff700 1 -- 172.21.15.98:0/633001292 >> 172.21.15.98:0/633001292 conn(0x7fb74406b8e0 msgr2=0x7fb7440b21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:56.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:56.007+0000 7fb743fff700 1 -- 172.21.15.98:0/633001292 shutdown_connections 2023-03-03T07:58:56.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:58:56.007+0000 7fb743fff700 1 -- 172.21.15.98:0/633001292 wait complete. 2023-03-03T07:58:56.010 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T07:58:56.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2023-03-03T07:58:56.311 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":6120,"entity":{"name":{"type":"client","num":6120},"addr":{"type":"any","addr":"172.21.15.98:0","nonce":1023372945}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":13,"uptime":7.4909063339999999,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.623097894148913,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.6120 172.21.15.98:0/1023372945","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"y","hostname":"smithi098","pid":"108777","root":"/"}}] 2023-03-03T07:58:56.312 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "y", "hostname": "smithi098", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108777", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.98:0", "nonce": 1023372945, "type": "any" }, "name": { "num": 6120, "type": "client" } }, "id": 6120, "inst": "client.6120 172.21.15.98:0/1023372945", "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 13, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.623097894148913 }, "state": "open", "uptime": 7.490906334 } ] 2023-03-03T07:58:56.312 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2023-03-03T07:59:01.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:59:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.564+0000 7f28043ff700 1 Processor -- start 2023-03-03T07:59:01.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.564+0000 7f28043ff700 1 -- start start 2023-03-03T07:59:01.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.565+0000 7f28043ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27fc0b87c0 0x7f27fc0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:01.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.566+0000 7f28043ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f27fc0b8b90 0x7f27fc0b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:01.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.566+0000 7f28043ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc0d0ef0 con 0x7f27fc0b5240 2023-03-03T07:59:01.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.566+0000 7f28043ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27fc0d0600 con 0x7f27fc0b87c0 2023-03-03T07:59:01.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.566+0000 7f28043ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27fc0cd970 con 0x7f27fc0b8b90 2023-03-03T07:59:01.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.566+0000 7f280219b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27fc0b87c0 0x7f27fc0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:01.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.566+0000 7f280219b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27fc0b87c0 0x7f27fc0cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40384/0 (socket says 172.21.15.98:40384) 2023-03-03T07:59:01.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.566+0000 7f280219b700 1 -- 172.21.15.98:0/2561936468 learned_addr learned my addr 172.21.15.98:0/2561936468 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:59:01.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.566+0000 7f280199a700 1 --2- 172.21.15.98:0/2561936468 >> v2:172.21.15.100:3300/0 conn(0x7f27fc0b8b90 0x7f27fc0b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:01.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.566+0000 7f280219b700 1 -- 172.21.15.98:0/2561936468 >> v2:172.21.15.100:3300/0 conn(0x7f27fc0b8b90 msgr2=0x7f27fc0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:01.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.566+0000 7f280219b700 1 --2- 172.21.15.98:0/2561936468 >> v2:172.21.15.100:3300/0 conn(0x7f27fc0b8b90 0x7f27fc0b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:01.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.567+0000 7f280219b700 1 -- 172.21.15.98:0/2561936468 >> v1:172.21.15.100:6789/0 conn(0x7f27fc0b5240 legacy=0x7f27fc1605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:01.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.567+0000 7f280219b700 1 -- 172.21.15.98:0/2561936468 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc0cdc50 con 0x7f27fc0b87c0 2023-03-03T07:59:01.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.567+0000 7f280219b700 1 --2- 172.21.15.98:0/2561936468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc0b87c0 0x7f27fc0cffe0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f27fc08c240 tx=0x7f27f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=977fdf3aba341f00 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:01.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.567+0000 7f2801199700 1 -- 172.21.15.98:0/2561936468 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f800b870 con 0x7f27fc0b87c0 2023-03-03T07:59:01.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.567+0000 7f2801199700 1 -- 172.21.15.98:0/2561936468 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f800b9d0 con 0x7f27fc0b87c0 2023-03-03T07:59:01.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.567+0000 7f2801199700 1 -- 172.21.15.98:0/2561936468 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f800bd60 con 0x7f27fc0b87c0 2023-03-03T07:59:01.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.568+0000 7f28043ff700 1 -- 172.21.15.98:0/2561936468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc0b87c0 msgr2=0x7f27fc0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:01.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.568+0000 7f28043ff700 1 --2- 172.21.15.98:0/2561936468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc0b87c0 0x7f27fc0cffe0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f27fc08c240 tx=0x7f27f8009bd0 comp rx=0 tx=0).stop 2023-03-03T07:59:01.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.568+0000 7f28043ff700 1 -- 172.21.15.98:0/2561936468 shutdown_connections 2023-03-03T07:59:01.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.568+0000 7f28043ff700 1 --2- 172.21.15.98:0/2561936468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc0b87c0 0x7f27fc0cffe0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:01.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.568+0000 7f28043ff700 1 --2- 172.21.15.98:0/2561936468 >> v2:172.21.15.100:3300/0 conn(0x7f27fc0b8b90 0x7f27fc0b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:01.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.568+0000 7f28043ff700 1 -- 172.21.15.98:0/2561936468 >> 172.21.15.98:0/2561936468 conn(0x7f27fc06aee0 msgr2=0x7f27fc06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:01.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.568+0000 7f28043ff700 1 -- 172.21.15.98:0/2561936468 shutdown_connections 2023-03-03T07:59:01.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.568+0000 7f28043ff700 1 -- 172.21.15.98:0/2561936468 wait complete. 2023-03-03T07:59:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.570+0000 7f28043ff700 1 Processor -- start 2023-03-03T07:59:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.570+0000 7f28043ff700 1 -- start start 2023-03-03T07:59:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.571+0000 7f28043ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc0b5240 0x7f27fc15b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.571+0000 7f28043ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27fc0b87c0 0x7f27fc1e66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.571+0000 7f28043ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27fc0b8b90 0x7f27fc0cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.571+0000 7f28043ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc0ca860 con 0x7f27fc0b5240 2023-03-03T07:59:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.571+0000 7f28043ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc15cc90 con 0x7f27fc0b8b90 2023-03-03T07:59:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.571+0000 7f28043ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc15cf70 con 0x7f27fc0b87c0 2023-03-03T07:59:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.571+0000 7f280219b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc0b5240 0x7f27fc15b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.571+0000 7f280219b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc0b5240 0x7f27fc15b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40392/0 (socket says 172.21.15.98:40392) 2023-03-03T07:59:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.571+0000 7f280219b700 1 -- 172.21.15.98:0/74489537 learned_addr learned my addr 172.21.15.98:0/74489537 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:59:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.572+0000 7f280199a700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27fc0b87c0 0x7f27fc1e66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.572+0000 7f280299c700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27fc0b8b90 0x7f27fc0cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.572+0000 7f280219b700 1 -- 172.21.15.98:0/74489537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27fc0b87c0 msgr2=0x7f27fc1e66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.572+0000 7f280219b700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27fc0b87c0 0x7f27fc1e66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.572+0000 7f280219b700 1 -- 172.21.15.98:0/74489537 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27fc0b8b90 msgr2=0x7f27fc0cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.572+0000 7f280219b700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27fc0b8b90 0x7f27fc0cc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.572+0000 7f280219b700 1 -- 172.21.15.98:0/74489537 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f8009550 con 0x7f27fc0b5240 2023-03-03T07:59:01.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.572+0000 7f280199a700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27fc0b87c0 0x7f27fc1e66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:59:01.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.572+0000 7f280299c700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27fc0b8b90 0x7f27fc0cc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:59:01.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.572+0000 7f280219b700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc0b5240 0x7f27fc15b830 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f27f8000c00 tx=0x7f27f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.573+0000 7f27f2ffd700 1 -- 172.21.15.98:0/74489537 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f8004540 con 0x7f27fc0b5240 2023-03-03T07:59:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.573+0000 7f27f2ffd700 1 -- 172.21.15.98:0/74489537 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f80036a0 con 0x7f27fc0b5240 2023-03-03T07:59:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.573+0000 7f27f2ffd700 1 -- 172.21.15.98:0/74489537 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f8003a30 con 0x7f27fc0b5240 2023-03-03T07:59:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.573+0000 7f28043ff700 1 -- 172.21.15.98:0/74489537 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27fc0cdf30 con 0x7f27fc0b5240 2023-03-03T07:59:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.573+0000 7f28043ff700 1 -- 172.21.15.98:0/74489537 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27fc073a80 con 0x7f27fc0b5240 2023-03-03T07:59:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.574+0000 7f28043ff700 1 -- 172.21.15.98:0/74489537 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27fc0ce0c0 con 0x7f27fc0b5240 2023-03-03T07:59:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.575+0000 7f27f2ffd700 1 -- 172.21.15.98:0/74489537 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 86575+0+0 (secure 0 0 0) 0x7f27f800f050 con 0x7f27fc0b5240 2023-03-03T07:59:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.576+0000 7f27f2ffd700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f27e0068410 0x7f27e006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.576+0000 7f27f2ffd700 1 -- 172.21.15.98:0/74489537 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f27f8087850 con 0x7f27fc0b5240 2023-03-03T07:59:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.577+0000 7f280199a700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f27e0068410 0x7f27e006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.577+0000 7f280199a700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f27e0068410 0x7f27e006a8c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f27fc15ff70 tx=0x7f27ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.581+0000 7f27f2ffd700 1 -- 172.21.15.98:0/74489537 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f27f8053e60 con 0x7f27fc0b5240 2023-03-03T07:59:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.817+0000 7f28043ff700 1 -- 172.21.15.98:0/74489537 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27fc0729b0 con 0x7f27fc0b5240 2023-03-03T07:59:01.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.817+0000 7f27f2ffd700 1 -- 172.21.15.98:0/74489537 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f27f8057480 con 0x7f27fc0b5240 2023-03-03T07:59:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:59:01.819 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:59:01.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.820+0000 7f28043ff700 1 -- 172.21.15.98:0/74489537 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f27e0068410 msgr2=0x7f27e006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:01.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.820+0000 7f28043ff700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f27e0068410 0x7f27e006a8c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f27fc15ff70 tx=0x7f27ec008040 comp rx=0 tx=0).stop 2023-03-03T07:59:01.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.820+0000 7f28043ff700 1 -- 172.21.15.98:0/74489537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc0b5240 msgr2=0x7f27fc15b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:01.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.820+0000 7f28043ff700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc0b5240 0x7f27fc15b830 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f27f8000c00 tx=0x7f27f8004750 comp rx=0 tx=0).stop 2023-03-03T07:59:01.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.820+0000 7f28043ff700 1 -- 172.21.15.98:0/74489537 shutdown_connections 2023-03-03T07:59:01.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.820+0000 7f28043ff700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f27e0068410 0x7f27e006a8c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:01.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.820+0000 7f28043ff700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc0b5240 0x7f27fc15b830 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:01.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.820+0000 7f28043ff700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27fc0b8b90 0x7f27fc0cc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:01.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.820+0000 7f28043ff700 1 --2- 172.21.15.98:0/74489537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27fc0b87c0 0x7f27fc1e66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:01.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.820+0000 7f28043ff700 1 -- 172.21.15.98:0/74489537 >> 172.21.15.98:0/74489537 conn(0x7f27fc06aee0 msgr2=0x7f27fc0b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:01.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.820+0000 7f28043ff700 1 -- 172.21.15.98:0/74489537 shutdown_connections 2023-03-03T07:59:01.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:01.820+0000 7f28043ff700 1 -- 172.21.15.98:0/74489537 wait complete. 2023-03-03T07:59:01.823 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T07:59:01.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2023-03-03T07:59:02.123 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":6120,"entity":{"name":{"type":"client","num":6120},"addr":{"type":"any","addr":"172.21.15.98:0","nonce":1023372945}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":12,"uptime":13.301833674999999,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.334724367401478,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.6120 172.21.15.98:0/1023372945","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"y","hostname":"smithi098","pid":"108777","root":"/"}}] 2023-03-03T07:59:02.123 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "y", "hostname": "smithi098", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108777", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.98:0", "nonce": 1023372945, "type": "any" }, "name": { "num": 6120, "type": "client" } }, "id": 6120, "inst": "client.6120 172.21.15.98:0/1023372945", "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 12, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.334724367401478 }, "state": "open", "uptime": 13.301833675 } ] 2023-03-03T07:59:02.124 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2023-03-03T07:59:02.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:59:02.324 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:59:07.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:59:07.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.377+0000 7f7875bff700 1 Processor -- start 2023-03-03T07:59:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.377+0000 7f7875bff700 1 -- start start 2023-03-03T07:59:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.379+0000 7f7875bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f78700b8b90 0x7f78700cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.379+0000 7f7875bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78700d0ef0 con 0x7f78700b5410 2023-03-03T07:59:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.379+0000 7f7875bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78700d0600 con 0x7f78700b87c0 2023-03-03T07:59:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.379+0000 7f7875bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78700cd970 con 0x7f78700b8b90 2023-03-03T07:59:07.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.379+0000 7f786ffff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f78700b5410 0x7f78700d13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54298/0 (socket says 172.21.15.98:54298) 2023-03-03T07:59:07.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.379+0000 7f786ffff700 1 -- 172.21.15.98:0/3537732813 learned_addr learned my addr 172.21.15.98:0/3537732813 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:59:07.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.379+0000 7f786effd700 1 --2- 172.21.15.98:0/3537732813 >> v2:172.21.15.100:3300/0 conn(0x7f78700b8b90 0x7f78700cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:07.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.379+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472936465 0 0) 0x7f78700d0ef0 con 0x7f78700b5410 2023-03-03T07:59:07.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78580035b0 con 0x7f78700b5410 2023-03-03T07:59:07.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396285006 0 0) 0x7f78700d0600 con 0x7f78700b87c0 2023-03-03T07:59:07.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78700d0ef0 con 0x7f78700b87c0 2023-03-03T07:59:07.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1823906957 0 0) 0x7f78580035b0 con 0x7f78700b5410 2023-03-03T07:59:07.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 >> v2:172.21.15.100:3300/0 conn(0x7f78700b8b90 msgr2=0x7f78700cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:07.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 --2- 172.21.15.98:0/3537732813 >> v2:172.21.15.100:3300/0 conn(0x7f78700b8b90 0x7f78700cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:07.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 >> v1:172.21.15.100:6789/0 conn(0x7f78700b87c0 legacy=0x7f78700b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:07.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78700cdc50 con 0x7f78700b5410 2023-03-03T07:59:07.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f78680019f0 con 0x7f78700b5410 2023-03-03T07:59:07.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 >> v1:172.21.15.98:6789/0 conn(0x7f78700b5410 legacy=0x7f78700d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:07.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 --2- 172.21.15.98:0/3537732813 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7858003b90 0x7f7858003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:07.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 --2- 172.21.15.98:0/3537732813 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7858004850 0x7f7858004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:07.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 --2- 172.21.15.98:0/3537732813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7858005200 0x7f785800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:07.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78700cd970 con 0x7f7858005200 2023-03-03T07:59:07.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785800c060 con 0x7f7858003b90 2023-03-03T07:59:07.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785800c220 con 0x7f7858004850 2023-03-03T07:59:07.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786ffff700 1 --2- 172.21.15.98:0/3537732813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7858005200 0x7f785800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:07.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.380+0000 7f786f7fe700 1 --2- 172.21.15.98:0/3537732813 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7858003b90 0x7f7858003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:07.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.381+0000 7f786effd700 1 --2- 172.21.15.98:0/3537732813 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7858004850 0x7f7858004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:07.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.381+0000 7f786ffff700 1 -- 172.21.15.98:0/3537732813 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7858004850 msgr2=0x7f7858004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:07.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.381+0000 7f786ffff700 1 --2- 172.21.15.98:0/3537732813 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7858004850 0x7f7858004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:07.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.381+0000 7f786ffff700 1 -- 172.21.15.98:0/3537732813 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7858003b90 msgr2=0x7f7858003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:07.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.381+0000 7f786ffff700 1 --2- 172.21.15.98:0/3537732813 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7858003b90 0x7f7858003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:07.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.381+0000 7f786ffff700 1 -- 172.21.15.98:0/3537732813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78580035b0 con 0x7f7858005200 2023-03-03T07:59:07.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.381+0000 7f786effd700 1 --2- 172.21.15.98:0/3537732813 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7858004850 0x7f7858004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:59:07.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.381+0000 7f786ffff700 1 -- 172.21.15.98:0/3537732813 reap_dead start 2023-03-03T07:59:07.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.381+0000 7f786ffff700 1 --2- 172.21.15.98:0/3537732813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7858005200 0x7f785800ba10 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f786800a7a0 tx=0x7f7858005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd31d57db0c3e63 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:07.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.381+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7868012600 con 0x7f7858005200 2023-03-03T07:59:07.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.382+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7868012780 con 0x7f7858005200 2023-03-03T07:59:07.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.382+0000 7f786e7fc700 1 -- 172.21.15.98:0/3537732813 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7868012b30 con 0x7f7858005200 2023-03-03T07:59:07.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.382+0000 7f7875bff700 1 -- 172.21.15.98:0/3537732813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7858005200 msgr2=0x7f785800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:07.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.382+0000 7f7875bff700 1 --2- 172.21.15.98:0/3537732813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7858005200 0x7f785800ba10 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f786800a7a0 tx=0x7f7858005880 comp rx=0 tx=0).stop 2023-03-03T07:59:07.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.382+0000 7f7875bff700 1 -- 172.21.15.98:0/3537732813 shutdown_connections 2023-03-03T07:59:07.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.382+0000 7f7875bff700 1 --2- 172.21.15.98:0/3537732813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7858005200 0x7f785800ba10 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:07.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.382+0000 7f7875bff700 1 -- 172.21.15.98:0/3537732813 >> 172.21.15.98:0/3537732813 conn(0x7f787006aee0 msgr2=0x7f787006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:07.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.382+0000 7f7875bff700 1 -- 172.21.15.98:0/3537732813 shutdown_connections 2023-03-03T07:59:07.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.382+0000 7f7875bff700 1 -- 172.21.15.98:0/3537732813 wait complete. 2023-03-03T07:59:07.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.384+0000 7f7875bff700 1 Processor -- start 2023-03-03T07:59:07.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.385+0000 7f7875bff700 1 -- start start 2023-03-03T07:59:07.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.386+0000 7f7875bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7858005200 0x7f7870145ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:07.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.386+0000 7f7875bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78700c99a0 0x7f78701585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:07.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.386+0000 7f7875bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7870156640 0x7f7870156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:07.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.386+0000 7f7875bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7870156ff0 con 0x7f78700c99a0 2023-03-03T07:59:07.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.386+0000 7f7875bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7870158b30 con 0x7f7870156640 2023-03-03T07:59:07.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.386+0000 7f7875bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7870159970 con 0x7f7858005200 2023-03-03T07:59:07.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.386+0000 7f786effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78700c99a0 0x7f78701585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:07.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.386+0000 7f786effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78700c99a0 0x7f78701585f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52910/0 (socket says 172.21.15.98:52910) 2023-03-03T07:59:07.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.386+0000 7f786effd700 1 -- 172.21.15.98:0/1492818301 learned_addr learned my addr 172.21.15.98:0/1492818301 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:59:07.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.386+0000 7f786f7fe700 1 --2- 172.21.15.98:0/1492818301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7858005200 0x7f7870145ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:07.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.386+0000 7f786ffff700 1 --2- 172.21.15.98:0/1492818301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7870156640 0x7f7870156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.386+0000 7f786effd700 1 -- 172.21.15.98:0/1492818301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7858005200 msgr2=0x7f7870145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.386+0000 7f786effd700 1 --2- 172.21.15.98:0/1492818301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7858005200 0x7f7870145ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.387+0000 7f786effd700 1 -- 172.21.15.98:0/1492818301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7870156640 msgr2=0x7f7870156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.387+0000 7f786effd700 1 --2- 172.21.15.98:0/1492818301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7870156640 0x7f7870156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.387+0000 7f786effd700 1 -- 172.21.15.98:0/1492818301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78700cdf30 con 0x7f78700c99a0 2023-03-03T07:59:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.387+0000 7f786ffff700 1 --2- 172.21.15.98:0/1492818301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7870156640 0x7f7870156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:59:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.387+0000 7f786effd700 1 --2- 172.21.15.98:0/1492818301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78700c99a0 0x7f78701585f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f78640108c0 tx=0x7f7864010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:07.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.387+0000 7f7857fff700 1 -- 172.21.15.98:0/1492818301 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7864004020 con 0x7f78700c99a0 2023-03-03T07:59:07.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.387+0000 7f7857fff700 1 -- 172.21.15.98:0/1492818301 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7864004180 con 0x7f78700c99a0 2023-03-03T07:59:07.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.387+0000 7f7857fff700 1 -- 172.21.15.98:0/1492818301 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7864004510 con 0x7f78700c99a0 2023-03-03T07:59:07.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.387+0000 7f7875bff700 1 -- 172.21.15.98:0/1492818301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7870073a80 con 0x7f78700c99a0 2023-03-03T07:59:07.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.387+0000 7f7875bff700 1 -- 172.21.15.98:0/1492818301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78700ce0c0 con 0x7f78700c99a0 2023-03-03T07:59:07.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.388+0000 7f7875bff700 1 -- 172.21.15.98:0/1492818301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78700729b0 con 0x7f78700c99a0 2023-03-03T07:59:07.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.390+0000 7f7857fff700 1 -- 172.21.15.98:0/1492818301 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 86575+0+0 (secure 0 0 0) 0x7f78640036a0 con 0x7f78700c99a0 2023-03-03T07:59:07.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.392+0000 7f7857fff700 1 --2- 172.21.15.98:0/1492818301 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f785c068440 0x7f785c06a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:07.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.392+0000 7f7857fff700 1 -- 172.21.15.98:0/1492818301 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f7864085f30 con 0x7f78700c99a0 2023-03-03T07:59:07.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.392+0000 7f786f7fe700 1 --2- 172.21.15.98:0/1492818301 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f785c068440 0x7f785c06a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:07.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.393+0000 7f786f7fe700 1 --2- 172.21.15.98:0/1492818301 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f785c068440 0x7f785c06a8f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7860000c00 tx=0x7f7860009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:07.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.396+0000 7f7857fff700 1 -- 172.21.15.98:0/1492818301 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7864052540 con 0x7f78700c99a0 2023-03-03T07:59:07.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.632+0000 7f7875bff700 1 -- 172.21.15.98:0/1492818301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f787015e100 con 0x7f78700c99a0 2023-03-03T07:59:07.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.633+0000 7f7857fff700 1 -- 172.21.15.98:0/1492818301 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f787015e100 con 0x7f78700c99a0 2023-03-03T07:59:07.635 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:59:07.635 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:59:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.635+0000 7f7875bff700 1 -- 172.21.15.98:0/1492818301 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f785c068440 msgr2=0x7f785c06a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.635+0000 7f7875bff700 1 --2- 172.21.15.98:0/1492818301 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f785c068440 0x7f785c06a8f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7860000c00 tx=0x7f7860009040 comp rx=0 tx=0).stop 2023-03-03T07:59:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.635+0000 7f7875bff700 1 -- 172.21.15.98:0/1492818301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78700c99a0 msgr2=0x7f78701585f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.635+0000 7f7875bff700 1 --2- 172.21.15.98:0/1492818301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78700c99a0 0x7f78701585f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f78640108c0 tx=0x7f7864010c80 comp rx=0 tx=0).stop 2023-03-03T07:59:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.635+0000 7f7875bff700 1 -- 172.21.15.98:0/1492818301 shutdown_connections 2023-03-03T07:59:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.635+0000 7f7875bff700 1 --2- 172.21.15.98:0/1492818301 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f785c068440 0x7f785c06a8f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.635+0000 7f7875bff700 1 --2- 172.21.15.98:0/1492818301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7870156640 0x7f7870156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:07.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.635+0000 7f7875bff700 1 --2- 172.21.15.98:0/1492818301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7858005200 0x7f7870145ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:07.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.635+0000 7f7875bff700 1 --2- 172.21.15.98:0/1492818301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78700c99a0 0x7f78701585f0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:07.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.635+0000 7f7875bff700 1 -- 172.21.15.98:0/1492818301 >> 172.21.15.98:0/1492818301 conn(0x7f787006aee0 msgr2=0x7f78700b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:07.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.636+0000 7f7875bff700 1 -- 172.21.15.98:0/1492818301 shutdown_connections 2023-03-03T07:59:07.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:07.636+0000 7f7875bff700 1 -- 172.21.15.98:0/1492818301 wait complete. 2023-03-03T07:59:07.639 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T07:59:07.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2023-03-03T07:59:07.936 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":6120,"entity":{"name":{"type":"client","num":6120},"addr":{"type":"any","addr":"172.21.15.98:0","nonce":1023372945}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":12,"uptime":19.114760988,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.050099424297049,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.6120 172.21.15.98:0/1023372945","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"y","hostname":"smithi098","pid":"108777","root":"/"}}] 2023-03-03T07:59:07.937 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "y", "hostname": "smithi098", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108777", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.98:0", "nonce": 1023372945, "type": "any" }, "name": { "num": 6120, "type": "client" } }, "id": 6120, "inst": "client.6120 172.21.15.98:0/1023372945", "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 12, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.05009942429705 }, "state": "open", "uptime": 19.114760988 } ] 2023-03-03T07:59:07.937 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2023-03-03T07:59:12.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:59:13.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.187+0000 7f6945bff700 1 Processor -- start 2023-03-03T07:59:13.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.187+0000 7f6945bff700 1 -- start start 2023-03-03T07:59:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.188+0000 7f6945bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f69400b8b90 0x7f69400cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.188+0000 7f6945bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69400d0ef0 con 0x7f69400b5410 2023-03-03T07:59:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.189+0000 7f6945bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69400d0600 con 0x7f69400b87c0 2023-03-03T07:59:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.189+0000 7f6945bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69400cd970 con 0x7f69400b8b90 2023-03-03T07:59:13.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.189+0000 7f693ffff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f69400b5410 0x7f69400d13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54302/0 (socket says 172.21.15.98:54302) 2023-03-03T07:59:13.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.189+0000 7f693ffff700 1 -- 172.21.15.98:0/2796271214 learned_addr learned my addr 172.21.15.98:0/2796271214 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:59:13.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.189+0000 7f693effd700 1 --2- 172.21.15.98:0/2796271214 >> v2:172.21.15.100:3300/0 conn(0x7f69400b8b90 0x7f69400cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:13.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.189+0000 7f693e7fc700 1 -- 172.21.15.98:0/2796271214 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649419004 0 0) 0x7f69400d0600 con 0x7f69400b87c0 2023-03-03T07:59:13.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.189+0000 7f693e7fc700 1 -- 172.21.15.98:0/2796271214 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69280035b0 con 0x7f69400b87c0 2023-03-03T07:59:13.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.190+0000 7f693effd700 1 -- 172.21.15.98:0/2796271214 >> v1:172.21.15.100:6789/0 conn(0x7f69400b87c0 legacy=0x7f69400b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:13.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.190+0000 7f693effd700 1 -- 172.21.15.98:0/2796271214 >> v1:172.21.15.98:6789/0 conn(0x7f69400b5410 legacy=0x7f69400d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:13.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.190+0000 7f693e7fc700 1 -- 172.21.15.98:0/2796271214 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321094972 0 0) 0x7f69400d0ef0 con 0x7f69400b5410 2023-03-03T07:59:13.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.190+0000 7f693effd700 1 -- 172.21.15.98:0/2796271214 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69400cdc50 con 0x7f69400b8b90 2023-03-03T07:59:13.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.190+0000 7f693effd700 1 --2- 172.21.15.98:0/2796271214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69400b8b90 0x7f69400cffe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6934009840 tx=0x7f6934009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=311404b56e32f29b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:13.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.190+0000 7f693e7fc700 1 -- 172.21.15.98:0/2796271214 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6934011830 con 0x7f69400b8b90 2023-03-03T07:59:13.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.190+0000 7f693e7fc700 1 -- 172.21.15.98:0/2796271214 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6934011990 con 0x7f69400b8b90 2023-03-03T07:59:13.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.190+0000 7f693e7fc700 1 -- 172.21.15.98:0/2796271214 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6934011d20 con 0x7f69400b8b90 2023-03-03T07:59:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.191+0000 7f6945bff700 1 -- 172.21.15.98:0/2796271214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69400b8b90 msgr2=0x7f69400cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.191+0000 7f6945bff700 1 --2- 172.21.15.98:0/2796271214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69400b8b90 0x7f69400cffe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6934009840 tx=0x7f6934009b50 comp rx=0 tx=0).stop 2023-03-03T07:59:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.191+0000 7f6945bff700 1 -- 172.21.15.98:0/2796271214 shutdown_connections 2023-03-03T07:59:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.191+0000 7f6945bff700 1 --2- 172.21.15.98:0/2796271214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69400b8b90 0x7f69400cffe0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.191+0000 7f6945bff700 1 -- 172.21.15.98:0/2796271214 >> 172.21.15.98:0/2796271214 conn(0x7f694006aee0 msgr2=0x7f694006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.191+0000 7f6945bff700 1 -- 172.21.15.98:0/2796271214 shutdown_connections 2023-03-03T07:59:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.191+0000 7f6945bff700 1 -- 172.21.15.98:0/2796271214 wait complete. 2023-03-03T07:59:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.193+0000 7f6945bff700 1 Processor -- start 2023-03-03T07:59:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.193+0000 7f6945bff700 1 -- start start 2023-03-03T07:59:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.194+0000 7f6945bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69400b5410 0x7f69401f6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.194+0000 7f6945bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69400b87c0 0x7f694015f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.194+0000 7f6945bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69400b8b90 0x7f694015f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.194+0000 7f6945bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69401f6880 con 0x7f69400b87c0 2023-03-03T07:59:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.194+0000 7f6945bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69400ca7a0 con 0x7f69400b8b90 2023-03-03T07:59:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.194+0000 7f6945bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69400caa80 con 0x7f69400b5410 2023-03-03T07:59:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.194+0000 7f693effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69400b87c0 0x7f694015f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.195+0000 7f693effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69400b87c0 0x7f694015f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52912/0 (socket says 172.21.15.98:52912) 2023-03-03T07:59:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.195+0000 7f693effd700 1 -- 172.21.15.98:0/2764902737 learned_addr learned my addr 172.21.15.98:0/2764902737 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:59:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.195+0000 7f693f7fe700 1 --2- 172.21.15.98:0/2764902737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69400b5410 0x7f69401f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.195+0000 7f693ffff700 1 --2- 172.21.15.98:0/2764902737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69400b8b90 0x7f694015f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.195+0000 7f693effd700 1 -- 172.21.15.98:0/2764902737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69400b5410 msgr2=0x7f69401f6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.195+0000 7f693effd700 1 --2- 172.21.15.98:0/2764902737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69400b5410 0x7f69401f6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.195+0000 7f693effd700 1 -- 172.21.15.98:0/2764902737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69400b8b90 msgr2=0x7f694015f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.195+0000 7f693effd700 1 --2- 172.21.15.98:0/2764902737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69400b8b90 0x7f694015f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.195+0000 7f693effd700 1 -- 172.21.15.98:0/2764902737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69340094a0 con 0x7f69400b87c0 2023-03-03T07:59:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.195+0000 7f693ffff700 1 --2- 172.21.15.98:0/2764902737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69400b8b90 0x7f694015f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:59:13.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.195+0000 7f693effd700 1 --2- 172.21.15.98:0/2764902737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69400b87c0 0x7f694015f280 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f69340090b0 tx=0x7f6934004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:13.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.196+0000 7f6927fff700 1 -- 172.21.15.98:0/2764902737 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693401a910 con 0x7f69400b87c0 2023-03-03T07:59:13.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.196+0000 7f6927fff700 1 -- 172.21.15.98:0/2764902737 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693401aa70 con 0x7f69400b87c0 2023-03-03T07:59:13.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.196+0000 7f6927fff700 1 -- 172.21.15.98:0/2764902737 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693401ade0 con 0x7f69400b87c0 2023-03-03T07:59:13.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.196+0000 7f6945bff700 1 -- 172.21.15.98:0/2764902737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69400cdf30 con 0x7f69400b87c0 2023-03-03T07:59:13.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.196+0000 7f6945bff700 1 -- 172.21.15.98:0/2764902737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6940073a80 con 0x7f69400b87c0 2023-03-03T07:59:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.197+0000 7f6945bff700 1 -- 172.21.15.98:0/2764902737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69400ce0c0 con 0x7f69400b87c0 2023-03-03T07:59:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.198+0000 7f6927fff700 1 -- 172.21.15.98:0/2764902737 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 86575+0+0 (secure 0 0 0) 0x7f693400f050 con 0x7f69400b87c0 2023-03-03T07:59:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.200+0000 7f6927fff700 1 --2- 172.21.15.98:0/2764902737 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f692c068410 0x7f692c06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.200+0000 7f6927fff700 1 -- 172.21.15.98:0/2764902737 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f6934086dd0 con 0x7f69400b87c0 2023-03-03T07:59:13.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.200+0000 7f693f7fe700 1 --2- 172.21.15.98:0/2764902737 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f692c068410 0x7f692c06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:13.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.201+0000 7f693f7fe700 1 --2- 172.21.15.98:0/2764902737 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f692c068410 0x7f692c06a8c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f69300011e0 tx=0x7f6930009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.204+0000 7f6927fff700 1 -- 172.21.15.98:0/2764902737 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6934053490 con 0x7f69400b87c0 2023-03-03T07:59:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.441+0000 7f6945bff700 1 -- 172.21.15.98:0/2764902737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69400729b0 con 0x7f69400b87c0 2023-03-03T07:59:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.441+0000 7f6927fff700 1 -- 172.21.15.98:0/2764902737 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f6934029090 con 0x7f69400b87c0 2023-03-03T07:59:13.444 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:59:13.444 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:59:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.444+0000 7f6945bff700 1 -- 172.21.15.98:0/2764902737 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f692c068410 msgr2=0x7f692c06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.444+0000 7f6945bff700 1 --2- 172.21.15.98:0/2764902737 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f692c068410 0x7f692c06a8c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f69300011e0 tx=0x7f6930009040 comp rx=0 tx=0).stop 2023-03-03T07:59:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.444+0000 7f6945bff700 1 -- 172.21.15.98:0/2764902737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69400b87c0 msgr2=0x7f694015f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.444+0000 7f6945bff700 1 --2- 172.21.15.98:0/2764902737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69400b87c0 0x7f694015f280 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f69340090b0 tx=0x7f6934004750 comp rx=0 tx=0).stop 2023-03-03T07:59:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.444+0000 7f6945bff700 1 -- 172.21.15.98:0/2764902737 shutdown_connections 2023-03-03T07:59:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.444+0000 7f6945bff700 1 --2- 172.21.15.98:0/2764902737 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f692c068410 0x7f692c06a8c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.444+0000 7f6945bff700 1 --2- 172.21.15.98:0/2764902737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69400b8b90 0x7f694015f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.444+0000 7f6945bff700 1 --2- 172.21.15.98:0/2764902737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69400b5410 0x7f69401f6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.444+0000 7f6945bff700 1 --2- 172.21.15.98:0/2764902737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69400b87c0 0x7f694015f280 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.444+0000 7f6945bff700 1 -- 172.21.15.98:0/2764902737 >> 172.21.15.98:0/2764902737 conn(0x7f694006aee0 msgr2=0x7f69400b1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.444+0000 7f6945bff700 1 -- 172.21.15.98:0/2764902737 shutdown_connections 2023-03-03T07:59:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:13.444+0000 7f6945bff700 1 -- 172.21.15.98:0/2764902737 wait complete. 2023-03-03T07:59:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T07:59:13.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2023-03-03T07:59:13.748 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":6120,"entity":{"name":{"type":"client","num":6120},"addr":{"type":"any","addr":"172.21.15.98:0","nonce":1023372945}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":11,"uptime":24.926688315,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.769319627607782,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.6120 172.21.15.98:0/1023372945","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"y","hostname":"smithi098","pid":"108777","root":"/"}}] 2023-03-03T07:59:13.749 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "y", "hostname": "smithi098", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108777", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.98:0", "nonce": 1023372945, "type": "any" }, "name": { "num": 6120, "type": "client" } }, "id": 6120, "inst": "client.6120 172.21.15.98:0/1023372945", "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 11, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.769319627607782 }, "state": "open", "uptime": 24.926688315 } ] 2023-03-03T07:59:13.749 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2023-03-03T07:59:18.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:59:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.000+0000 7ff5e1fff700 1 Processor -- start 2023-03-03T07:59:19.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.000+0000 7ff5e1fff700 1 -- start start 2023-03-03T07:59:19.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.001+0000 7ff5e1fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff5dc0b87c0 0x7ff5dc0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:19.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.001+0000 7ff5e1fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5dc0b8b90 0x7ff5dc0b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:19.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.002+0000 7ff5e1fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5dc0d0ef0 con 0x7ff5dc0b5240 2023-03-03T07:59:19.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.002+0000 7ff5e1fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5dc0d0600 con 0x7ff5dc0b87c0 2023-03-03T07:59:19.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.002+0000 7ff5e1fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0cd970 con 0x7ff5dc0b8b90 2023-03-03T07:59:19.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.002+0000 7ff5dbfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff5dc0b5240 0x7ff5dc1605b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33284/0 (socket says 172.21.15.98:33284) 2023-03-03T07:59:19.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.002+0000 7ff5dbfff700 1 -- 172.21.15.98:0/622143284 learned_addr learned my addr 172.21.15.98:0/622143284 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:59:19.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.002+0000 7ff5daffd700 1 --2- 172.21.15.98:0/622143284 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5dc0b8b90 0x7ff5dc0b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:19.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.002+0000 7ff5db7fe700 1 --2- 172.21.15.98:0/622143284 >> v2:172.21.15.100:3300/0 conn(0x7ff5dc0b87c0 0x7ff5dc0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:19.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.002+0000 7ff5da7fc700 1 -- 172.21.15.98:0/622143284 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037052372 0 0) 0x7ff5dc0d0ef0 con 0x7ff5dc0b5240 2023-03-03T07:59:19.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.002+0000 7ff5da7fc700 1 -- 172.21.15.98:0/622143284 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c40035b0 con 0x7ff5dc0b5240 2023-03-03T07:59:19.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.003+0000 7ff5db7fe700 1 -- 172.21.15.98:0/622143284 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5dc0b8b90 msgr2=0x7ff5dc0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:19.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.003+0000 7ff5db7fe700 1 --2- 172.21.15.98:0/622143284 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5dc0b8b90 0x7ff5dc0b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:19.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.003+0000 7ff5db7fe700 1 -- 172.21.15.98:0/622143284 >> v1:172.21.15.98:6789/0 conn(0x7ff5dc0b5240 legacy=0x7ff5dc1605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:19.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.003+0000 7ff5db7fe700 1 -- 172.21.15.98:0/622143284 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc0cdc50 con 0x7ff5dc0b87c0 2023-03-03T07:59:19.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.003+0000 7ff5daffd700 1 --2- 172.21.15.98:0/622143284 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5dc0b8b90 0x7ff5dc0b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:59:19.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.003+0000 7ff5db7fe700 1 --2- 172.21.15.98:0/622143284 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5dc0b87c0 0x7ff5dc0cffe0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009840 tx=0x7ff5cc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21332c1bc85d77cd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:19.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.003+0000 7ff5da7fc700 1 -- 172.21.15.98:0/622143284 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc011830 con 0x7ff5dc0b87c0 2023-03-03T07:59:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.003+0000 7ff5da7fc700 1 -- 172.21.15.98:0/622143284 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5cc011990 con 0x7ff5dc0b87c0 2023-03-03T07:59:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.003+0000 7ff5da7fc700 1 -- 172.21.15.98:0/622143284 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc011d20 con 0x7ff5dc0b87c0 2023-03-03T07:59:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.004+0000 7ff5e1fff700 1 -- 172.21.15.98:0/622143284 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5dc0b87c0 msgr2=0x7ff5dc0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.004+0000 7ff5e1fff700 1 --2- 172.21.15.98:0/622143284 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5dc0b87c0 0x7ff5dc0cffe0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009840 tx=0x7ff5cc009b50 comp rx=0 tx=0).stop 2023-03-03T07:59:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.004+0000 7ff5e1fff700 1 -- 172.21.15.98:0/622143284 shutdown_connections 2023-03-03T07:59:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.004+0000 7ff5e1fff700 1 --2- 172.21.15.98:0/622143284 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5dc0b87c0 0x7ff5dc0cffe0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.004+0000 7ff5e1fff700 1 --2- 172.21.15.98:0/622143284 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5dc0b8b90 0x7ff5dc0b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.004+0000 7ff5e1fff700 1 -- 172.21.15.98:0/622143284 >> 172.21.15.98:0/622143284 conn(0x7ff5dc06aee0 msgr2=0x7ff5dc06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.004+0000 7ff5e1fff700 1 -- 172.21.15.98:0/622143284 shutdown_connections 2023-03-03T07:59:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.004+0000 7ff5e1fff700 1 -- 172.21.15.98:0/622143284 wait complete. 2023-03-03T07:59:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.006+0000 7ff5e1fff700 1 Processor -- start 2023-03-03T07:59:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.006+0000 7ff5e1fff700 1 -- start start 2023-03-03T07:59:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.007+0000 7ff5e1fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5dc0b5240 0x7ff5dc15b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.007+0000 7ff5e1fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5dc0b87c0 0x7ff5dc1f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.007+0000 7ff5e1fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5dc0b8b90 0x7ff5dc1f54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.007+0000 7ff5e1fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0c8820 con 0x7ff5dc0b8b90 2023-03-03T07:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.007+0000 7ff5e1fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0c7f30 con 0x7ff5dc0b87c0 2023-03-03T07:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.007+0000 7ff5e1fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0c81d0 con 0x7ff5dc0b5240 2023-03-03T07:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.008+0000 7ff5dbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5dc0b8b90 0x7ff5dc1f54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.008+0000 7ff5dbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5dc0b8b90 0x7ff5dc1f54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37632/0 (socket says 172.21.15.98:37632) 2023-03-03T07:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.008+0000 7ff5dbfff700 1 -- 172.21.15.98:0/3726800835 learned_addr learned my addr 172.21.15.98:0/3726800835 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.008+0000 7ff5db7fe700 1 --2- 172.21.15.98:0/3726800835 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5dc0b5240 0x7ff5dc15b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.008+0000 7ff5daffd700 1 --2- 172.21.15.98:0/3726800835 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5dc0b87c0 0x7ff5dc1f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.008+0000 7ff5dbfff700 1 -- 172.21.15.98:0/3726800835 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5dc0b5240 msgr2=0x7ff5dc15b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.008+0000 7ff5dbfff700 1 --2- 172.21.15.98:0/3726800835 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5dc0b5240 0x7ff5dc15b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.008+0000 7ff5dbfff700 1 -- 172.21.15.98:0/3726800835 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5dc0b87c0 msgr2=0x7ff5dc1f2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.008+0000 7ff5dbfff700 1 --2- 172.21.15.98:0/3726800835 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5dc0b87c0 0x7ff5dc1f2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.008+0000 7ff5dbfff700 1 -- 172.21.15.98:0/3726800835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d00094a0 con 0x7ff5dc0b8b90 2023-03-03T07:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.009+0000 7ff5dbfff700 1 --2- 172.21.15.98:0/3726800835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5dc0b8b90 0x7ff5dc1f54a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400cb40 tx=0x7ff5d400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.009+0000 7ff5c3fff700 1 -- 172.21.15.98:0/3726800835 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d400a9e0 con 0x7ff5dc0b8b90 2023-03-03T07:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.009+0000 7ff5c3fff700 1 -- 172.21.15.98:0/3726800835 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d400ab40 con 0x7ff5dc0b8b90 2023-03-03T07:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.009+0000 7ff5e1fff700 1 -- 172.21.15.98:0/3726800835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5cc0094a0 con 0x7ff5dc0b8b90 2023-03-03T07:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.009+0000 7ff5c3fff700 1 -- 172.21.15.98:0/3726800835 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4012400 con 0x7ff5dc0b8b90 2023-03-03T07:59:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.009+0000 7ff5e1fff700 1 -- 172.21.15.98:0/3726800835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5dc0cdf30 con 0x7ff5dc0b8b90 2023-03-03T07:59:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.010+0000 7ff5e1fff700 1 -- 172.21.15.98:0/3726800835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5dc073a80 con 0x7ff5dc0b8b90 2023-03-03T07:59:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.011+0000 7ff5c3fff700 1 -- 172.21.15.98:0/3726800835 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 86575+0+0 (secure 0 0 0) 0x7ff5d4019030 con 0x7ff5dc0b8b90 2023-03-03T07:59:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.013+0000 7ff5c3fff700 1 --2- 172.21.15.98:0/3726800835 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff5c8068410 0x7ff5c806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.013+0000 7ff5c3fff700 1 -- 172.21.15.98:0/3726800835 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7ff5d4085970 con 0x7ff5dc0b8b90 2023-03-03T07:59:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.013+0000 7ff5db7fe700 1 --2- 172.21.15.98:0/3726800835 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff5c8068410 0x7ff5c806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.014+0000 7ff5db7fe700 1 --2- 172.21.15.98:0/3726800835 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff5c8068410 0x7ff5c806a8c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc000c00 tx=0x7ff5cc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:19.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.017+0000 7ff5c3fff700 1 -- 172.21.15.98:0/3726800835 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff5d404b5e0 con 0x7ff5dc0b8b90 2023-03-03T07:59:19.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.253+0000 7ff5e1fff700 1 -- 172.21.15.98:0/3726800835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5dc0729b0 con 0x7ff5dc0b8b90 2023-03-03T07:59:19.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.254+0000 7ff5c3fff700 1 -- 172.21.15.98:0/3726800835 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7ff5d4052030 con 0x7ff5dc0b8b90 2023-03-03T07:59:19.256 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:59:19.256 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:59:19.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.256+0000 7ff5e1fff700 1 -- 172.21.15.98:0/3726800835 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff5c8068410 msgr2=0x7ff5c806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:19.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.256+0000 7ff5e1fff700 1 --2- 172.21.15.98:0/3726800835 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff5c8068410 0x7ff5c806a8c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc000c00 tx=0x7ff5cc008030 comp rx=0 tx=0).stop 2023-03-03T07:59:19.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.256+0000 7ff5e1fff700 1 -- 172.21.15.98:0/3726800835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5dc0b8b90 msgr2=0x7ff5dc1f54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:19.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.256+0000 7ff5e1fff700 1 --2- 172.21.15.98:0/3726800835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5dc0b8b90 0x7ff5dc1f54a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400cb40 tx=0x7ff5d400ce50 comp rx=0 tx=0).stop 2023-03-03T07:59:19.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.257+0000 7ff5e1fff700 1 -- 172.21.15.98:0/3726800835 shutdown_connections 2023-03-03T07:59:19.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.257+0000 7ff5e1fff700 1 --2- 172.21.15.98:0/3726800835 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff5c8068410 0x7ff5c806a8c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:19.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.257+0000 7ff5e1fff700 1 --2- 172.21.15.98:0/3726800835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5dc0b8b90 0x7ff5dc1f54a0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:19.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.257+0000 7ff5e1fff700 1 --2- 172.21.15.98:0/3726800835 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5dc0b5240 0x7ff5dc15b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:19.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.257+0000 7ff5e1fff700 1 --2- 172.21.15.98:0/3726800835 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5dc0b87c0 0x7ff5dc1f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:19.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.257+0000 7ff5e1fff700 1 -- 172.21.15.98:0/3726800835 >> 172.21.15.98:0/3726800835 conn(0x7ff5dc06aee0 msgr2=0x7ff5dc0b72d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:19.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.257+0000 7ff5e1fff700 1 -- 172.21.15.98:0/3726800835 shutdown_connections 2023-03-03T07:59:19.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:19.257+0000 7ff5e1fff700 1 -- 172.21.15.98:0/3726800835 wait complete. 2023-03-03T07:59:19.261 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T07:59:19.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2023-03-03T07:59:19.561 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":6120,"entity":{"name":{"type":"client","num":6120},"addr":{"type":"any","addr":"172.21.15.98:0","nonce":1023372945}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":10,"uptime":30.740615615999999,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.492190359845512,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.6120 172.21.15.98:0/1023372945","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"y","hostname":"smithi098","pid":"108777","root":"/"}}] 2023-03-03T07:59:19.562 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "y", "hostname": "smithi098", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108777", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.98:0", "nonce": 1023372945, "type": "any" }, "name": { "num": 6120, "type": "client" } }, "id": 6120, "inst": "client.6120 172.21.15.98:0/1023372945", "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 10, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.492190359845512 }, "state": "open", "uptime": 30.740615616 } ] 2023-03-03T07:59:19.563 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2023-03-03T07:59:24.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:59:24.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.817+0000 7fed1659d700 1 Processor -- start 2023-03-03T07:59:24.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.817+0000 7fed1659d700 1 -- start start 2023-03-03T07:59:24.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.819+0000 7fed1659d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed100d0010 0x7fed100d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:24.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.819+0000 7fed1659d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed100b0c50 con 0x7fed100b4a00 2023-03-03T07:59:24.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.819+0000 7fed1659d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed100b0040 con 0x7fed100d1650 2023-03-03T07:59:24.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.819+0000 7fed1659d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed100b0320 con 0x7fed100d0010 2023-03-03T07:59:24.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.819+0000 7fed14d9a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fed100b4a00 0x7fed100d0f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33292/0 (socket says 172.21.15.98:33292) 2023-03-03T07:59:24.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.819+0000 7fed14d9a700 1 -- 172.21.15.98:0/2080105190 learned_addr learned my addr 172.21.15.98:0/2080105190 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:59:24.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed1559b700 1 --2- 172.21.15.98:0/2080105190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed100d0010 0x7fed100d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:24.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319062907 0 0) 0x7fed100b0c50 con 0x7fed100b4a00 2023-03-03T07:59:24.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecfc0035b0 con 0x7fed100b4a00 2023-03-03T07:59:24.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2357320897 0 0) 0x7fed100b0040 con 0x7fed100d1650 2023-03-03T07:59:24.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed100b0c50 con 0x7fed100d1650 2023-03-03T07:59:24.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 75413166 0 0) 0x7fecfc0035b0 con 0x7fed100b4a00 2023-03-03T07:59:24.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed100d0010 msgr2=0x7fed100d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:24.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 --2- 172.21.15.98:0/2080105190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed100d0010 0x7fed100d03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:24.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 >> v1:172.21.15.100:6789/0 conn(0x7fed100d1650 legacy=0x7fed100b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:24.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed100b0600 con 0x7fed100b4a00 2023-03-03T07:59:24.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7fed00001670 con 0x7fed100b4a00 2023-03-03T07:59:24.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 >> v1:172.21.15.98:6789/0 conn(0x7fed100b4a00 legacy=0x7fed100d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:24.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 --2- 172.21.15.98:0/2080105190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc003bc0 0x7fecfc003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:24.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 --2- 172.21.15.98:0/2080105190 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fecfc004850 0x7fecfc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:24.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 --2- 172.21.15.98:0/2080105190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fecfc005200 0x7fecfc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:24.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed100b0320 con 0x7fecfc003bc0 2023-03-03T07:59:24.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc00bfd0 con 0x7fecfc004850 2023-03-03T07:59:24.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc00c190 con 0x7fecfc005200 2023-03-03T07:59:24.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.820+0000 7fed1559b700 1 --2- 172.21.15.98:0/2080105190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed100d0010 0x7fed100d03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:59:24.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.821+0000 7fed14d9a700 1 --2- 172.21.15.98:0/2080105190 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fecfc004850 0x7fecfc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:24.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.821+0000 7fed1559b700 1 --2- 172.21.15.98:0/2080105190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc003bc0 0x7fecfc003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:24.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.821+0000 7fed15d9c700 1 --2- 172.21.15.98:0/2080105190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fecfc005200 0x7fecfc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:24.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.821+0000 7fed1559b700 1 -- 172.21.15.98:0/2080105190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fecfc005200 msgr2=0x7fecfc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:24.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.821+0000 7fed1559b700 1 --2- 172.21.15.98:0/2080105190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fecfc005200 0x7fecfc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:24.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.821+0000 7fed1559b700 1 -- 172.21.15.98:0/2080105190 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fecfc004850 msgr2=0x7fecfc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:24.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.821+0000 7fed1559b700 1 --2- 172.21.15.98:0/2080105190 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fecfc004850 0x7fecfc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:24.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.821+0000 7fed1559b700 1 -- 172.21.15.98:0/2080105190 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc0035b0 con 0x7fecfc003bc0 2023-03-03T07:59:24.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.821+0000 7fed1559b700 1 --2- 172.21.15.98:0/2080105190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc003bc0 0x7fecfc003fb0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fed1008bc60 tx=0x7fecfc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596032708f9f884b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:24.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.822+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c0096b0 con 0x7fecfc003bc0 2023-03-03T07:59:24.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.822+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0c004030 con 0x7fecfc003bc0 2023-03-03T07:59:24.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.822+0000 7fed07fff700 1 -- 172.21.15.98:0/2080105190 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c0043e0 con 0x7fecfc003bc0 2023-03-03T07:59:24.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.822+0000 7fed1659d700 1 -- 172.21.15.98:0/2080105190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc003bc0 msgr2=0x7fecfc003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:24.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.822+0000 7fed1659d700 1 --2- 172.21.15.98:0/2080105190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc003bc0 0x7fecfc003fb0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fed1008bc60 tx=0x7fecfc005880 comp rx=0 tx=0).stop 2023-03-03T07:59:24.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.822+0000 7fed15d9c700 1 -- 172.21.15.98:0/2080105190 reap_dead start 2023-03-03T07:59:24.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.822+0000 7fed1659d700 1 -- 172.21.15.98:0/2080105190 shutdown_connections 2023-03-03T07:59:24.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.822+0000 7fed1659d700 1 -- 172.21.15.98:0/2080105190 >> 172.21.15.98:0/2080105190 conn(0x7fed10073b00 msgr2=0x7fed100d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:24.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.822+0000 7fed1659d700 1 -- 172.21.15.98:0/2080105190 shutdown_connections 2023-03-03T07:59:24.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.822+0000 7fed1659d700 1 -- 172.21.15.98:0/2080105190 wait complete. 2023-03-03T07:59:24.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.825+0000 7fed1659d700 1 Processor -- start 2023-03-03T07:59:24.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.825+0000 7fed1659d700 1 -- start start 2023-03-03T07:59:24.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed1659d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc003bc0 0x7fed1015b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:24.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed1659d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1015c680 0x7fed10152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:24.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed1659d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed101e4370 0x7fed101e47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:24.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed1659d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed101e4d30 con 0x7fecfc003bc0 2023-03-03T07:59:24.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed1659d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed100c7920 con 0x7fed101e4370 2023-03-03T07:59:24.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed1659d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1015b470 con 0x7fed1015c680 2023-03-03T07:59:24.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed1559b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc003bc0 0x7fed1015b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:24.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed1559b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc003bc0 0x7fed1015b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37650/0 (socket says 172.21.15.98:37650) 2023-03-03T07:59:24.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed1559b700 1 -- 172.21.15.98:0/1671386884 learned_addr learned my addr 172.21.15.98:0/1671386884 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:59:24.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed1559b700 1 -- 172.21.15.98:0/1671386884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1015c680 msgr2=0x7fed10152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:24.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed14d9a700 1 --2- 172.21.15.98:0/1671386884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1015c680 0x7fed10152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:24.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed1559b700 1 --2- 172.21.15.98:0/1671386884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1015c680 0x7fed10152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:24.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed1559b700 1 -- 172.21.15.98:0/1671386884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed101e4370 msgr2=0x7fed101e47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:24.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed1559b700 1 --2- 172.21.15.98:0/1671386884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed101e4370 0x7fed101e47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:24.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed1559b700 1 -- 172.21.15.98:0/1671386884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed100d0b30 con 0x7fecfc003bc0 2023-03-03T07:59:24.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.826+0000 7fed14d9a700 1 --2- 172.21.15.98:0/1671386884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1015c680 0x7fed10152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:59:24.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.827+0000 7fed1559b700 1 --2- 172.21.15.98:0/1671386884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc003bc0 0x7fed1015b6f0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fed0c0095e0 tx=0x7fed0c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:24.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.827+0000 7fed05ffb700 1 -- 172.21.15.98:0/1671386884 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c0045b0 con 0x7fecfc003bc0 2023-03-03T07:59:24.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.827+0000 7fed05ffb700 1 -- 172.21.15.98:0/1671386884 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0c00e8d0 con 0x7fecfc003bc0 2023-03-03T07:59:24.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.827+0000 7fed05ffb700 1 -- 172.21.15.98:0/1671386884 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c00ec60 con 0x7fecfc003bc0 2023-03-03T07:59:24.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.827+0000 7fed1659d700 1 -- 172.21.15.98:0/1671386884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed100b08e0 con 0x7fecfc003bc0 2023-03-03T07:59:24.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.827+0000 7fed1659d700 1 -- 172.21.15.98:0/1671386884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed100d0cc0 con 0x7fecfc003bc0 2023-03-03T07:59:24.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.829+0000 7fed1659d700 1 -- 172.21.15.98:0/1671386884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed100729b0 con 0x7fecfc003bc0 2023-03-03T07:59:24.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.830+0000 7fed05ffb700 1 -- 172.21.15.98:0/1671386884 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 86575+0+0 (secure 0 0 0) 0x7fed0c013470 con 0x7fecfc003bc0 2023-03-03T07:59:24.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.831+0000 7fed05ffb700 1 --2- 172.21.15.98:0/1671386884 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fecf4068410 0x7fecf406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:24.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.831+0000 7fed05ffb700 1 -- 172.21.15.98:0/1671386884 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fed0c087af0 con 0x7fecfc003bc0 2023-03-03T07:59:24.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.831+0000 7fed14d9a700 1 --2- 172.21.15.98:0/1671386884 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fecf4068410 0x7fecf406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:24.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.832+0000 7fed14d9a700 1 --2- 172.21.15.98:0/1671386884 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fecf4068410 0x7fecf406a8c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fed1015ca50 tx=0x7fed00009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:24.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:24.836+0000 7fed05ffb700 1 -- 172.21.15.98:0/1671386884 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fed0c0540a0 con 0x7fecfc003bc0 2023-03-03T07:59:25.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:25.071+0000 7fed1659d700 1 -- 172.21.15.98:0/1671386884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed10068300 con 0x7fecfc003bc0 2023-03-03T07:59:25.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:25.072+0000 7fed05ffb700 1 -- 172.21.15.98:0/1671386884 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7fed0c02b090 con 0x7fecfc003bc0 2023-03-03T07:59:25.075 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:59:25.075 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:59:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:25.074+0000 7fed1659d700 1 -- 172.21.15.98:0/1671386884 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fecf4068410 msgr2=0x7fecf406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:25.074+0000 7fed1659d700 1 --2- 172.21.15.98:0/1671386884 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fecf4068410 0x7fecf406a8c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fed1015ca50 tx=0x7fed00009040 comp rx=0 tx=0).stop 2023-03-03T07:59:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:25.074+0000 7fed1659d700 1 -- 172.21.15.98:0/1671386884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc003bc0 msgr2=0x7fed1015b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:25.074+0000 7fed1659d700 1 --2- 172.21.15.98:0/1671386884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc003bc0 0x7fed1015b6f0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fed0c0095e0 tx=0x7fed0c0047c0 comp rx=0 tx=0).stop 2023-03-03T07:59:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:25.074+0000 7fed1659d700 1 -- 172.21.15.98:0/1671386884 shutdown_connections 2023-03-03T07:59:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:25.074+0000 7fed1659d700 1 --2- 172.21.15.98:0/1671386884 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fecf4068410 0x7fecf406a8c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:25.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:25.074+0000 7fed1659d700 1 --2- 172.21.15.98:0/1671386884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc003bc0 0x7fed1015b6f0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:25.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:25.074+0000 7fed1659d700 1 --2- 172.21.15.98:0/1671386884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed101e4370 0x7fed101e47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:25.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:25.074+0000 7fed1659d700 1 --2- 172.21.15.98:0/1671386884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1015c680 0x7fed10152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:25.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:25.074+0000 7fed1659d700 1 -- 172.21.15.98:0/1671386884 >> 172.21.15.98:0/1671386884 conn(0x7fed10073b00 msgr2=0x7fed100b0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:25.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:25.075+0000 7fed1659d700 1 -- 172.21.15.98:0/1671386884 shutdown_connections 2023-03-03T07:59:25.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:25.075+0000 7fed1659d700 1 -- 172.21.15.98:0/1671386884 wait complete. 2023-03-03T07:59:25.079 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T07:59:25.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2023-03-03T07:59:25.384 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":6120,"entity":{"name":{"type":"client","num":6120},"addr":{"type":"any","addr":"172.21.15.98:0","nonce":1023372945}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":9,"uptime":36.561543211999997,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.218431866521353,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.6120 172.21.15.98:0/1023372945","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"y","hostname":"smithi098","pid":"108777","root":"/"}}] 2023-03-03T07:59:25.385 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "y", "hostname": "smithi098", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108777", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.98:0", "nonce": 1023372945, "type": "any" }, "name": { "num": 6120, "type": "client" } }, "id": 6120, "inst": "client.6120 172.21.15.98:0/1023372945", "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 9, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.218431866521353 }, "state": "open", "uptime": 36.561543212 } ] 2023-03-03T07:59:25.386 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2023-03-03T07:59:30.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:59:30.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.640+0000 7f60dddff700 1 Processor -- start 2023-03-03T07:59:30.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.640+0000 7f60dddff700 1 -- start start 2023-03-03T07:59:30.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.641+0000 7f60dddff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f60d80b8b90 0x7f60d80cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:30.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.641+0000 7f60dddff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d80b5410 0x7f60d80d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:30.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.641+0000 7f60dddff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d80d0ef0 con 0x7f60d80b87c0 2023-03-03T07:59:30.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.641+0000 7f60dddff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d80d0600 con 0x7f60d80b8b90 2023-03-03T07:59:30.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.641+0000 7f60dddff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d80cd970 con 0x7f60d80b5410 2023-03-03T07:59:30.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.642+0000 7f60d77fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f60d80b87c0 0x7f60d80b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55064/0 (socket says 172.21.15.98:55064) 2023-03-03T07:59:30.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.642+0000 7f60d77fe700 1 -- 172.21.15.98:0/3884251160 learned_addr learned my addr 172.21.15.98:0/3884251160 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:59:30.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.642+0000 7f60d7fff700 1 --2- 172.21.15.98:0/3884251160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d80b5410 0x7f60d80d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:30.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.642+0000 7f60d6ffd700 1 --2- 172.21.15.98:0/3884251160 >> v2:172.21.15.100:3300/0 conn(0x7f60d80b8b90 0x7f60d80cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:30.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.642+0000 7f60d67fc700 1 -- 172.21.15.98:0/3884251160 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628998282 0 0) 0x7f60d80d0ef0 con 0x7f60d80b87c0 2023-03-03T07:59:30.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.642+0000 7f60d67fc700 1 -- 172.21.15.98:0/3884251160 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c00035b0 con 0x7f60d80b87c0 2023-03-03T07:59:30.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.643+0000 7f60d7fff700 1 -- 172.21.15.98:0/3884251160 >> v2:172.21.15.100:3300/0 conn(0x7f60d80b8b90 msgr2=0x7f60d80cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:30.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.643+0000 7f60d7fff700 1 --2- 172.21.15.98:0/3884251160 >> v2:172.21.15.100:3300/0 conn(0x7f60d80b8b90 0x7f60d80cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:30.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.643+0000 7f60d7fff700 1 -- 172.21.15.98:0/3884251160 >> v1:172.21.15.98:6789/0 conn(0x7f60d80b87c0 legacy=0x7f60d80b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:30.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.643+0000 7f60d7fff700 1 -- 172.21.15.98:0/3884251160 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d80cdc50 con 0x7f60d80b5410 2023-03-03T07:59:30.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.643+0000 7f60d7fff700 1 --2- 172.21.15.98:0/3884251160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d80b5410 0x7f60d80d13d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009840 tx=0x7f60d0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6ccc7dba179e0a5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:30.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.643+0000 7f60d67fc700 1 -- 172.21.15.98:0/3884251160 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d0011830 con 0x7f60d80b5410 2023-03-03T07:59:30.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.643+0000 7f60d67fc700 1 -- 172.21.15.98:0/3884251160 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d0011990 con 0x7f60d80b5410 2023-03-03T07:59:30.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.643+0000 7f60d67fc700 1 -- 172.21.15.98:0/3884251160 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d0011d20 con 0x7f60d80b5410 2023-03-03T07:59:30.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.644+0000 7f60dddff700 1 -- 172.21.15.98:0/3884251160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d80b5410 msgr2=0x7f60d80d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:30.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.644+0000 7f60dddff700 1 --2- 172.21.15.98:0/3884251160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d80b5410 0x7f60d80d13d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009840 tx=0x7f60d0009b50 comp rx=0 tx=0).stop 2023-03-03T07:59:30.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.644+0000 7f60dddff700 1 -- 172.21.15.98:0/3884251160 shutdown_connections 2023-03-03T07:59:30.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.644+0000 7f60dddff700 1 --2- 172.21.15.98:0/3884251160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d80b5410 0x7f60d80d13d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:30.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.644+0000 7f60dddff700 1 --2- 172.21.15.98:0/3884251160 >> v2:172.21.15.100:3300/0 conn(0x7f60d80b8b90 0x7f60d80cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:30.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.644+0000 7f60dddff700 1 -- 172.21.15.98:0/3884251160 >> 172.21.15.98:0/3884251160 conn(0x7f60d806aee0 msgr2=0x7f60d806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:30.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.644+0000 7f60dddff700 1 -- 172.21.15.98:0/3884251160 shutdown_connections 2023-03-03T07:59:30.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.644+0000 7f60dddff700 1 -- 172.21.15.98:0/3884251160 wait complete. 2023-03-03T07:59:30.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.646+0000 7f60dddff700 1 Processor -- start 2023-03-03T07:59:30.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.646+0000 7f60dddff700 1 -- start start 2023-03-03T07:59:30.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60dddff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d80b5410 0x7f60d814f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:30.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60dddff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60d80b87c0 0x7f60d80a9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:30.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60dddff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d80b8b90 0x7f60d80aa1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:30.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60dddff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d80c8af0 con 0x7f60d80b5410 2023-03-03T07:59:30.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60dddff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d80c7c20 con 0x7f60d80b87c0 2023-03-03T07:59:30.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60dddff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d80c7e70 con 0x7f60d80b8b90 2023-03-03T07:59:30.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60d77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d80b5410 0x7f60d814f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:30.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60d77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d80b5410 0x7f60d814f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45864/0 (socket says 172.21.15.98:45864) 2023-03-03T07:59:30.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60d77fe700 1 -- 172.21.15.98:0/4212801774 learned_addr learned my addr 172.21.15.98:0/4212801774 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:59:30.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60d6ffd700 1 --2- 172.21.15.98:0/4212801774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60d80b87c0 0x7f60d80a9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:30.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60d7fff700 1 --2- 172.21.15.98:0/4212801774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d80b8b90 0x7f60d80aa1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:30.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60d77fe700 1 -- 172.21.15.98:0/4212801774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d80b8b90 msgr2=0x7f60d80aa1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:30.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60d77fe700 1 --2- 172.21.15.98:0/4212801774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d80b8b90 0x7f60d80aa1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:30.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60d77fe700 1 -- 172.21.15.98:0/4212801774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60d80b87c0 msgr2=0x7f60d80a9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:30.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60d77fe700 1 --2- 172.21.15.98:0/4212801774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60d80b87c0 0x7f60d80a9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:30.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.648+0000 7f60d77fe700 1 -- 172.21.15.98:0/4212801774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d00094a0 con 0x7f60d80b5410 2023-03-03T07:59:30.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.649+0000 7f60d77fe700 1 --2- 172.21.15.98:0/4212801774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d80b5410 0x7f60d814f870 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f60c800cbc0 tx=0x7f60c800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:30.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.649+0000 7f60bffff700 1 -- 172.21.15.98:0/4212801774 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c800aaa0 con 0x7f60d80b5410 2023-03-03T07:59:30.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.649+0000 7f60bffff700 1 -- 172.21.15.98:0/4212801774 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c800ac00 con 0x7f60d80b5410 2023-03-03T07:59:30.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.649+0000 7f60bffff700 1 -- 172.21.15.98:0/4212801774 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c8012400 con 0x7f60d80b5410 2023-03-03T07:59:30.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.649+0000 7f60dddff700 1 -- 172.21.15.98:0/4212801774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d80cdf30 con 0x7f60d80b5410 2023-03-03T07:59:30.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.649+0000 7f60dddff700 1 -- 172.21.15.98:0/4212801774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d8073a80 con 0x7f60d80b5410 2023-03-03T07:59:30.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.651+0000 7f60dddff700 1 -- 172.21.15.98:0/4212801774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d80ce0c0 con 0x7f60d80b5410 2023-03-03T07:59:30.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.652+0000 7f60bffff700 1 -- 172.21.15.98:0/4212801774 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 86575+0+0 (secure 0 0 0) 0x7f60c8019030 con 0x7f60d80b5410 2023-03-03T07:59:30.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.653+0000 7f60bffff700 1 --2- 172.21.15.98:0/4212801774 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f60c4068410 0x7f60c406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:30.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.653+0000 7f60bffff700 1 -- 172.21.15.98:0/4212801774 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f60c8085b40 con 0x7f60d80b5410 2023-03-03T07:59:30.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.653+0000 7f60d6ffd700 1 --2- 172.21.15.98:0/4212801774 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f60c4068410 0x7f60c406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:30.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.654+0000 7f60d6ffd700 1 --2- 172.21.15.98:0/4212801774 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f60c4068410 0x7f60c406a8c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f60d815cd80 tx=0x7f60cc009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:30.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.657+0000 7f60bffff700 1 -- 172.21.15.98:0/4212801774 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f60c8052150 con 0x7f60d80b5410 2023-03-03T07:59:30.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.898+0000 7f60dddff700 1 -- 172.21.15.98:0/4212801774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60d80729b0 con 0x7f60d80b5410 2023-03-03T07:59:30.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.898+0000 7f60bffff700 1 -- 172.21.15.98:0/4212801774 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f60c8055770 con 0x7f60d80b5410 2023-03-03T07:59:30.900 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:59:30.900 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:59:30.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.901+0000 7f60dddff700 1 -- 172.21.15.98:0/4212801774 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f60c4068410 msgr2=0x7f60c406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:30.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.901+0000 7f60dddff700 1 --2- 172.21.15.98:0/4212801774 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f60c4068410 0x7f60c406a8c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f60d815cd80 tx=0x7f60cc009210 comp rx=0 tx=0).stop 2023-03-03T07:59:30.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.901+0000 7f60dddff700 1 -- 172.21.15.98:0/4212801774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d80b5410 msgr2=0x7f60d814f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:30.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.901+0000 7f60dddff700 1 --2- 172.21.15.98:0/4212801774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d80b5410 0x7f60d814f870 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f60c800cbc0 tx=0x7f60c800ced0 comp rx=0 tx=0).stop 2023-03-03T07:59:30.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.901+0000 7f60dddff700 1 -- 172.21.15.98:0/4212801774 shutdown_connections 2023-03-03T07:59:30.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.901+0000 7f60dddff700 1 --2- 172.21.15.98:0/4212801774 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f60c4068410 0x7f60c406a8c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:30.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.901+0000 7f60dddff700 1 --2- 172.21.15.98:0/4212801774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d80b5410 0x7f60d814f870 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:30.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.901+0000 7f60dddff700 1 --2- 172.21.15.98:0/4212801774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d80b8b90 0x7f60d80aa1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:30.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.901+0000 7f60dddff700 1 --2- 172.21.15.98:0/4212801774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60d80b87c0 0x7f60d80a9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:30.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.901+0000 7f60dddff700 1 -- 172.21.15.98:0/4212801774 >> 172.21.15.98:0/4212801774 conn(0x7f60d806aee0 msgr2=0x7f60d80b3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:30.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.901+0000 7f60dddff700 1 -- 172.21.15.98:0/4212801774 shutdown_connections 2023-03-03T07:59:30.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:30.901+0000 7f60dddff700 1 -- 172.21.15.98:0/4212801774 wait complete. 2023-03-03T07:59:30.904 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T07:59:30.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2023-03-03T07:59:31.204 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":6120,"entity":{"name":{"type":"client","num":6120},"addr":{"type":"any","addr":"172.21.15.98:0","nonce":1023372945}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":9,"uptime":42.382471418000002,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.948330529177582,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.6120 172.21.15.98:0/1023372945","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"y","hostname":"smithi098","pid":"108777","root":"/"}}] 2023-03-03T07:59:31.204 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "y", "hostname": "smithi098", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108777", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.98:0", "nonce": 1023372945, "type": "any" }, "name": { "num": 6120, "type": "client" } }, "id": 6120, "inst": "client.6120 172.21.15.98:0/1023372945", "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 9, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.948330529177582 }, "state": "open", "uptime": 42.382471418 } ] 2023-03-03T07:59:31.205 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2023-03-03T07:59:32.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:59:32.463 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:59:36.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:59:36.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.457+0000 7f9a88bff700 1 Processor -- start 2023-03-03T07:59:36.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.457+0000 7f9a88bff700 1 -- start start 2023-03-03T07:59:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.459+0000 7f9a88bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9a840cfb60 0x7f9a840cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.459+0000 7f9a88bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a840b6370 0x7f9a840b67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.459+0000 7f9a88bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a840d0510 con 0x7f9a8415ca40 2023-03-03T07:59:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.459+0000 7f9a88bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a840cdee0 con 0x7f9a840cfb60 2023-03-03T07:59:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.459+0000 7f9a88bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a840d1890 con 0x7f9a840b6370 2023-03-03T07:59:36.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.459+0000 7f9a8259c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9a8415ca40 0x7f9a840cd7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33712/0 (socket says 172.21.15.98:33712) 2023-03-03T07:59:36.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.459+0000 7f9a8259c700 1 -- 172.21.15.98:0/3483830943 learned_addr learned my addr 172.21.15.98:0/3483830943 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:59:36.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.459+0000 7f9a82d9d700 1 --2- 172.21.15.98:0/3483830943 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a840b6370 0x7f9a840b67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:36.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.459+0000 7f9a81d9b700 1 --2- 172.21.15.98:0/3483830943 >> v2:172.21.15.100:3300/0 conn(0x7f9a840cfb60 0x7f9a840cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:36.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.459+0000 7f9a8159a700 1 -- 172.21.15.98:0/3483830943 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665334256 0 0) 0x7f9a840d0510 con 0x7f9a8415ca40 2023-03-03T07:59:36.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.460+0000 7f9a8159a700 1 -- 172.21.15.98:0/3483830943 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a6c0035b0 con 0x7f9a8415ca40 2023-03-03T07:59:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.460+0000 7f9a8159a700 1 -- 172.21.15.98:0/3483830943 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 797769963 0 0) 0x7f9a6c0035b0 con 0x7f9a8415ca40 2023-03-03T07:59:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.460+0000 7f9a81d9b700 1 -- 172.21.15.98:0/3483830943 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a840b6370 msgr2=0x7f9a840b67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.460+0000 7f9a81d9b700 1 --2- 172.21.15.98:0/3483830943 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a840b6370 0x7f9a840b67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.460+0000 7f9a81d9b700 1 -- 172.21.15.98:0/3483830943 >> v1:172.21.15.98:6789/0 conn(0x7f9a8415ca40 legacy=0x7f9a840cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.460+0000 7f9a81d9b700 1 -- 172.21.15.98:0/3483830943 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a840d1b70 con 0x7f9a840cfb60 2023-03-03T07:59:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.460+0000 7f9a81d9b700 1 --2- 172.21.15.98:0/3483830943 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a840cfb60 0x7f9a840cffd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009840 tx=0x7f9a78009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fd1d5f97328e5b0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.461+0000 7f9a8159a700 1 -- 172.21.15.98:0/3483830943 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a740019f0 con 0x7f9a840cfb60 2023-03-03T07:59:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.461+0000 7f9a8159a700 1 -- 172.21.15.98:0/3483830943 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a78004030 con 0x7f9a840cfb60 2023-03-03T07:59:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.461+0000 7f9a8159a700 1 -- 172.21.15.98:0/3483830943 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a780043c0 con 0x7f9a840cfb60 2023-03-03T07:59:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.461+0000 7f9a88bff700 1 -- 172.21.15.98:0/3483830943 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a840cfb60 msgr2=0x7f9a840cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.461+0000 7f9a88bff700 1 --2- 172.21.15.98:0/3483830943 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a840cfb60 0x7f9a840cffd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009840 tx=0x7f9a78009b50 comp rx=0 tx=0).stop 2023-03-03T07:59:36.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.461+0000 7f9a88bff700 1 -- 172.21.15.98:0/3483830943 shutdown_connections 2023-03-03T07:59:36.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.461+0000 7f9a88bff700 1 --2- 172.21.15.98:0/3483830943 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a840b6370 0x7f9a840b67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:36.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.461+0000 7f9a88bff700 1 --2- 172.21.15.98:0/3483830943 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a840cfb60 0x7f9a840cffd0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:36.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.461+0000 7f9a88bff700 1 -- 172.21.15.98:0/3483830943 >> 172.21.15.98:0/3483830943 conn(0x7f9a84073e70 msgr2=0x7f9a8406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:36.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.461+0000 7f9a88bff700 1 -- 172.21.15.98:0/3483830943 shutdown_connections 2023-03-03T07:59:36.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.461+0000 7f9a88bff700 1 -- 172.21.15.98:0/3483830943 wait complete. 2023-03-03T07:59:36.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.463+0000 7f9a88bff700 1 Processor -- start 2023-03-03T07:59:36.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.464+0000 7f9a88bff700 1 -- start start 2023-03-03T07:59:36.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.465+0000 7f9a88bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a840b6370 0x7f9a84149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:36.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.465+0000 7f9a88bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a840cfb60 0x7f9a84150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:36.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.465+0000 7f9a88bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a8415ca40 0x7f9a8414e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:36.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.465+0000 7f9a88bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8414f0d0 con 0x7f9a840b6370 2023-03-03T07:59:36.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.465+0000 7f9a88bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8414f3b0 con 0x7f9a8415ca40 2023-03-03T07:59:36.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.465+0000 7f9a88bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8414f690 con 0x7f9a840cfb60 2023-03-03T07:59:36.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.465+0000 7f9a8259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a840b6370 0x7f9a84149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:36.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.465+0000 7f9a8259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a840b6370 0x7f9a84149ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41912/0 (socket says 172.21.15.98:41912) 2023-03-03T07:59:36.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.465+0000 7f9a8259c700 1 -- 172.21.15.98:0/1914965869 learned_addr learned my addr 172.21.15.98:0/1914965869 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:59:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.465+0000 7f9a81d9b700 1 --2- 172.21.15.98:0/1914965869 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a840cfb60 0x7f9a84150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.465+0000 7f9a82d9d700 1 --2- 172.21.15.98:0/1914965869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a8415ca40 0x7f9a8414e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.465+0000 7f9a8259c700 1 -- 172.21.15.98:0/1914965869 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a840cfb60 msgr2=0x7f9a84150170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.465+0000 7f9a8259c700 1 --2- 172.21.15.98:0/1914965869 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a840cfb60 0x7f9a84150170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.466+0000 7f9a8259c700 1 -- 172.21.15.98:0/1914965869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a8415ca40 msgr2=0x7f9a8414e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.466+0000 7f9a8259c700 1 --2- 172.21.15.98:0/1914965869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a8415ca40 0x7f9a8414e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.466+0000 7f9a8259c700 1 -- 172.21.15.98:0/1914965869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a780094a0 con 0x7f9a840b6370 2023-03-03T07:59:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.466+0000 7f9a81d9b700 1 --2- 172.21.15.98:0/1914965869 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a840cfb60 0x7f9a84150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:59:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.466+0000 7f9a8259c700 1 --2- 172.21.15.98:0/1914965869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a840b6370 0x7f9a84149ca0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f9a740018e0 tx=0x7f9a7400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.466+0000 7f9a72ffd700 1 -- 172.21.15.98:0/1914965869 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7400ada0 con 0x7f9a840b6370 2023-03-03T07:59:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.466+0000 7f9a72ffd700 1 -- 172.21.15.98:0/1914965869 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a74001a30 con 0x7f9a840b6370 2023-03-03T07:59:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.466+0000 7f9a72ffd700 1 -- 172.21.15.98:0/1914965869 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a740072f0 con 0x7f9a840b6370 2023-03-03T07:59:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.466+0000 7f9a88bff700 1 -- 172.21.15.98:0/1914965869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a840d2130 con 0x7f9a840b6370 2023-03-03T07:59:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.466+0000 7f9a88bff700 1 -- 172.21.15.98:0/1914965869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a840d1e50 con 0x7f9a840b6370 2023-03-03T07:59:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.467+0000 7f9a88bff700 1 -- 172.21.15.98:0/1914965869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8406b700 con 0x7f9a840b6370 2023-03-03T07:59:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.469+0000 7f9a72ffd700 1 -- 172.21.15.98:0/1914965869 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 86575+0+0 (secure 0 0 0) 0x7f9a74010070 con 0x7f9a840b6370 2023-03-03T07:59:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.470+0000 7f9a72ffd700 1 --2- 172.21.15.98:0/1914965869 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9a64068410 0x7f9a6406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.470+0000 7f9a72ffd700 1 -- 172.21.15.98:0/1914965869 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f9a74085af0 con 0x7f9a840b6370 2023-03-03T07:59:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.470+0000 7f9a81d9b700 1 --2- 172.21.15.98:0/1914965869 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9a64068410 0x7f9a6406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.471+0000 7f9a81d9b700 1 --2- 172.21.15.98:0/1914965869 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9a64068410 0x7f9a6406a8c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800a010 tx=0x7f9a78013040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:36.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.475+0000 7f9a72ffd700 1 -- 172.21.15.98:0/1914965869 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9a74052100 con 0x7f9a840b6370 2023-03-03T07:59:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.711+0000 7f9a88bff700 1 -- 172.21.15.98:0/1914965869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a8404e6e0 con 0x7f9a840b6370 2023-03-03T07:59:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.712+0000 7f9a72ffd700 1 -- 172.21.15.98:0/1914965869 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f9a74018020 con 0x7f9a840b6370 2023-03-03T07:59:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:59:36.714 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:59:36.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.714+0000 7f9a88bff700 1 -- 172.21.15.98:0/1914965869 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9a64068410 msgr2=0x7f9a6406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:36.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.714+0000 7f9a88bff700 1 --2- 172.21.15.98:0/1914965869 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9a64068410 0x7f9a6406a8c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800a010 tx=0x7f9a78013040 comp rx=0 tx=0).stop 2023-03-03T07:59:36.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.715+0000 7f9a88bff700 1 -- 172.21.15.98:0/1914965869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a840b6370 msgr2=0x7f9a84149ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:36.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.715+0000 7f9a88bff700 1 --2- 172.21.15.98:0/1914965869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a840b6370 0x7f9a84149ca0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f9a740018e0 tx=0x7f9a7400a340 comp rx=0 tx=0).stop 2023-03-03T07:59:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.715+0000 7f9a88bff700 1 -- 172.21.15.98:0/1914965869 shutdown_connections 2023-03-03T07:59:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.715+0000 7f9a88bff700 1 --2- 172.21.15.98:0/1914965869 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9a64068410 0x7f9a6406a8c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.715+0000 7f9a88bff700 1 --2- 172.21.15.98:0/1914965869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a840b6370 0x7f9a84149ca0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.715+0000 7f9a88bff700 1 --2- 172.21.15.98:0/1914965869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a8415ca40 0x7f9a8414e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.715+0000 7f9a88bff700 1 --2- 172.21.15.98:0/1914965869 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a840cfb60 0x7f9a84150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.715+0000 7f9a88bff700 1 -- 172.21.15.98:0/1914965869 >> 172.21.15.98:0/1914965869 conn(0x7f9a84073e70 msgr2=0x7f9a8406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.715+0000 7f9a88bff700 1 -- 172.21.15.98:0/1914965869 shutdown_connections 2023-03-03T07:59:36.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:36.715+0000 7f9a88bff700 1 -- 172.21.15.98:0/1914965869 wait complete. 2023-03-03T07:59:36.718 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T07:59:36.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2023-03-03T07:59:37.018 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":6120,"entity":{"name":{"type":"client","num":6120},"addr":{"type":"any","addr":"172.21.15.98:0","nonce":1023372945}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":8,"uptime":48.197399699000002,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.682110366449606,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.6120 172.21.15.98:0/1023372945","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"y","hostname":"smithi098","pid":"108777","root":"/"}}] 2023-03-03T07:59:37.019 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "y", "hostname": "smithi098", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108777", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.98:0", "nonce": 1023372945, "type": "any" }, "name": { "num": 6120, "type": "client" } }, "id": 6120, "inst": "client.6120 172.21.15.98:0/1023372945", "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 8, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.682110366449606 }, "state": "open", "uptime": 48.197399699 } ] 2023-03-03T07:59:37.020 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2023-03-03T07:59:42.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:59:42.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.274+0000 7fef4af9d700 1 Processor -- start 2023-03-03T07:59:42.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.274+0000 7fef4af9d700 1 -- start start 2023-03-03T07:59:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.275+0000 7fef4af9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fef44146360 0x7fef44160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.275+0000 7fef4af9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fef440d2590 0x7fef440b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:42.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.275+0000 7fef4af9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef441614c0 con 0x7fef440b4ab0 2023-03-03T07:59:42.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.275+0000 7fef4af9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef440b5440 con 0x7fef440d2590 2023-03-03T07:59:42.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.275+0000 7fef4af9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef440d10f0 con 0x7fef44146360 2023-03-03T07:59:42.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.276+0000 7fef4979a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fef440b4ab0 0x7fef440731e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33714/0 (socket says 172.21.15.98:33714) 2023-03-03T07:59:42.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.276+0000 7fef4979a700 1 -- 172.21.15.98:0/239159452 learned_addr learned my addr 172.21.15.98:0/239159452 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:59:42.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.276+0000 7fef4a79c700 1 --2- 172.21.15.98:0/239159452 >> v2:172.21.15.98:3300/0 conn(0x7fef440d2590 0x7fef440b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:42.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.276+0000 7fef49f9b700 1 --2- 172.21.15.98:0/239159452 >> v2:172.21.15.100:3300/0 conn(0x7fef44146360 0x7fef44160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:42.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.276+0000 7fef4a79c700 1 -- 172.21.15.98:0/239159452 >> v2:172.21.15.100:3300/0 conn(0x7fef44146360 msgr2=0x7fef44160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:42.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.276+0000 7fef4a79c700 1 --2- 172.21.15.98:0/239159452 >> v2:172.21.15.100:3300/0 conn(0x7fef44146360 0x7fef44160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:42.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.276+0000 7fef4a79c700 1 -- 172.21.15.98:0/239159452 >> v1:172.21.15.98:6789/0 conn(0x7fef440b4ab0 legacy=0x7fef440731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:42.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.276+0000 7fef4a79c700 1 -- 172.21.15.98:0/239159452 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef440cdf20 con 0x7fef440d2590 2023-03-03T07:59:42.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.276+0000 7fef49f9b700 1 --2- 172.21.15.98:0/239159452 >> v2:172.21.15.100:3300/0 conn(0x7fef44146360 0x7fef44160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:59:42.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.277+0000 7fef4a79c700 1 --2- 172.21.15.98:0/239159452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef440d2590 0x7fef440b4f00 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fef440743f0 tx=0x7fef3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5fc7bf76e883ad server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:42.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.277+0000 7fef48f99700 1 -- 172.21.15.98:0/239159452 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c00e070 con 0x7fef440d2590 2023-03-03T07:59:42.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.277+0000 7fef48f99700 1 -- 172.21.15.98:0/239159452 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3c00bbd0 con 0x7fef440d2590 2023-03-03T07:59:42.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.277+0000 7fef48f99700 1 -- 172.21.15.98:0/239159452 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c012430 con 0x7fef440d2590 2023-03-03T07:59:42.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.277+0000 7fef4af9d700 1 -- 172.21.15.98:0/239159452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef440d2590 msgr2=0x7fef440b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:42.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.277+0000 7fef4af9d700 1 --2- 172.21.15.98:0/239159452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef440d2590 0x7fef440b4f00 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fef440743f0 tx=0x7fef3c00b330 comp rx=0 tx=0).stop 2023-03-03T07:59:42.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.278+0000 7fef4af9d700 1 -- 172.21.15.98:0/239159452 shutdown_connections 2023-03-03T07:59:42.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.278+0000 7fef4af9d700 1 --2- 172.21.15.98:0/239159452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef440d2590 0x7fef440b4f00 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:42.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.278+0000 7fef4af9d700 1 --2- 172.21.15.98:0/239159452 >> v2:172.21.15.100:3300/0 conn(0x7fef44146360 0x7fef44160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:42.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.278+0000 7fef4af9d700 1 -- 172.21.15.98:0/239159452 >> 172.21.15.98:0/239159452 conn(0x7fef4406b8e0 msgr2=0x7fef440cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:42.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.278+0000 7fef4af9d700 1 -- 172.21.15.98:0/239159452 shutdown_connections 2023-03-03T07:59:42.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.278+0000 7fef4af9d700 1 -- 172.21.15.98:0/239159452 wait complete. 2023-03-03T07:59:42.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.280+0000 7fef4af9d700 1 Processor -- start 2023-03-03T07:59:42.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.280+0000 7fef4af9d700 1 -- start start 2023-03-03T07:59:42.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.281+0000 7fef4af9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef440b4ab0 0x7fef440ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:42.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.281+0000 7fef4af9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fef440d2590 0x7fef440cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:42.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.281+0000 7fef4af9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef44146360 0x7fef440c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:42.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.281+0000 7fef4af9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef440cb380 con 0x7fef44146360 2023-03-03T07:59:42.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.281+0000 7fef4af9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef440ca770 con 0x7fef440b4ab0 2023-03-03T07:59:42.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.281+0000 7fef4af9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef440caa50 con 0x7fef440d2590 2023-03-03T07:59:42.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.282+0000 7fef4a79c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef44146360 0x7fef440c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:42.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.282+0000 7fef4a79c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef44146360 0x7fef440c8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41928/0 (socket says 172.21.15.98:41928) 2023-03-03T07:59:42.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.282+0000 7fef4a79c700 1 -- 172.21.15.98:0/2589807469 learned_addr learned my addr 172.21.15.98:0/2589807469 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:59:42.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.282+0000 7fef49f9b700 1 --2- 172.21.15.98:0/2589807469 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef440b4ab0 0x7fef440ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:42.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.282+0000 7fef4979a700 1 --2- 172.21.15.98:0/2589807469 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fef440d2590 0x7fef440cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:42.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.282+0000 7fef4a79c700 1 -- 172.21.15.98:0/2589807469 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fef440d2590 msgr2=0x7fef440cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:42.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.282+0000 7fef4a79c700 1 --2- 172.21.15.98:0/2589807469 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fef440d2590 0x7fef440cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:42.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.282+0000 7fef4a79c700 1 -- 172.21.15.98:0/2589807469 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef440b4ab0 msgr2=0x7fef440ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:42.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.282+0000 7fef4a79c700 1 --2- 172.21.15.98:0/2589807469 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef440b4ab0 0x7fef440ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:42.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.282+0000 7fef4a79c700 1 -- 172.21.15.98:0/2589807469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c009a70 con 0x7fef44146360 2023-03-03T07:59:42.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.282+0000 7fef4a79c700 1 --2- 172.21.15.98:0/2589807469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef44146360 0x7fef440c8d70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fef3c014040 tx=0x7fef3c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:42.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.283+0000 7fef3a7fc700 1 -- 172.21.15.98:0/2589807469 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c012430 con 0x7fef44146360 2023-03-03T07:59:42.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.283+0000 7fef3a7fc700 1 -- 172.21.15.98:0/2589807469 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3c020070 con 0x7fef44146360 2023-03-03T07:59:42.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.283+0000 7fef4af9d700 1 -- 172.21.15.98:0/2589807469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef440cd960 con 0x7fef44146360 2023-03-03T07:59:42.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.283+0000 7fef3a7fc700 1 -- 172.21.15.98:0/2589807469 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c0098f0 con 0x7fef44146360 2023-03-03T07:59:42.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.283+0000 7fef4af9d700 1 -- 172.21.15.98:0/2589807469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef440cd680 con 0x7fef44146360 2023-03-03T07:59:42.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.284+0000 7fef4af9d700 1 -- 172.21.15.98:0/2589807469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef440cdc40 con 0x7fef44146360 2023-03-03T07:59:42.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.285+0000 7fef3a7fc700 1 -- 172.21.15.98:0/2589807469 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 86575+0+0 (secure 0 0 0) 0x7fef3c02e070 con 0x7fef44146360 2023-03-03T07:59:42.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.287+0000 7fef3a7fc700 1 --2- 172.21.15.98:0/2589807469 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fef28068410 0x7fef2806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:42.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.287+0000 7fef3a7fc700 1 -- 172.21.15.98:0/2589807469 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fef3c022070 con 0x7fef44146360 2023-03-03T07:59:42.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.287+0000 7fef49f9b700 1 --2- 172.21.15.98:0/2589807469 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fef28068410 0x7fef2806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:42.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.288+0000 7fef49f9b700 1 --2- 172.21.15.98:0/2589807469 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fef28068410 0x7fef2806a8c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fef40006f10 tx=0x7fef40008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:42.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.292+0000 7fef3a7fc700 1 -- 172.21.15.98:0/2589807469 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fef3c054230 con 0x7fef44146360 2023-03-03T07:59:42.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.527+0000 7fef4af9d700 1 -- 172.21.15.98:0/2589807469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef4406a910 con 0x7fef44146360 2023-03-03T07:59:42.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.527+0000 7fef3a7fc700 1 -- 172.21.15.98:0/2589807469 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7fef3c057850 con 0x7fef44146360 2023-03-03T07:59:42.529 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:59:42.529 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:59:42.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.530+0000 7fef4af9d700 1 -- 172.21.15.98:0/2589807469 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fef28068410 msgr2=0x7fef2806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:42.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.530+0000 7fef4af9d700 1 --2- 172.21.15.98:0/2589807469 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fef28068410 0x7fef2806a8c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fef40006f10 tx=0x7fef40008040 comp rx=0 tx=0).stop 2023-03-03T07:59:42.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.530+0000 7fef4af9d700 1 -- 172.21.15.98:0/2589807469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef44146360 msgr2=0x7fef440c8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:42.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.530+0000 7fef4af9d700 1 --2- 172.21.15.98:0/2589807469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef44146360 0x7fef440c8d70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fef3c014040 tx=0x7fef3c004ad0 comp rx=0 tx=0).stop 2023-03-03T07:59:42.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.530+0000 7fef4af9d700 1 -- 172.21.15.98:0/2589807469 shutdown_connections 2023-03-03T07:59:42.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.530+0000 7fef4af9d700 1 --2- 172.21.15.98:0/2589807469 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fef28068410 0x7fef2806a8c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:42.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.530+0000 7fef4af9d700 1 --2- 172.21.15.98:0/2589807469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef44146360 0x7fef440c8d70 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:42.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.530+0000 7fef4af9d700 1 --2- 172.21.15.98:0/2589807469 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fef440b4ab0 0x7fef440ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:42.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.530+0000 7fef4af9d700 1 --2- 172.21.15.98:0/2589807469 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fef440d2590 0x7fef440cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:42.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.530+0000 7fef4af9d700 1 -- 172.21.15.98:0/2589807469 >> 172.21.15.98:0/2589807469 conn(0x7fef4406b8e0 msgr2=0x7fef440d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:42.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.530+0000 7fef4af9d700 1 -- 172.21.15.98:0/2589807469 shutdown_connections 2023-03-03T07:59:42.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:42.530+0000 7fef4af9d700 1 -- 172.21.15.98:0/2589807469 wait complete. 2023-03-03T07:59:42.533 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T07:59:42.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2023-03-03T07:59:42.833 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":6120,"entity":{"name":{"type":"client","num":6120},"addr":{"type":"any","addr":"172.21.15.98:0","nonce":1023372945}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":8,"uptime":54.013327969999999,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.419398173195102,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.6120 172.21.15.98:0/1023372945","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"y","hostname":"smithi098","pid":"108777","root":"/"}}] 2023-03-03T07:59:42.834 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "y", "hostname": "smithi098", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108777", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.98:0", "nonce": 1023372945, "type": "any" }, "name": { "num": 6120, "type": "client" } }, "id": 6120, "inst": "client.6120 172.21.15.98:0/1023372945", "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 8, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.419398173195102 }, "state": "open", "uptime": 54.01332797 } ] 2023-03-03T07:59:42.834 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2023-03-03T07:59:47.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:59:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.086+0000 7f3cb7fff700 1 Processor -- start 2023-03-03T07:59:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.087+0000 7f3cb7fff700 1 -- start start 2023-03-03T07:59:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.088+0000 7f3cb7fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3cb80b51c0 0x7f3cb8160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.088+0000 7f3cb7fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3cb80d1580 0x7f3cb80d1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.088+0000 7f3cb7fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb81614c0 con 0x7f3cb8146360 2023-03-03T07:59:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.088+0000 7f3cb7fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb80d10f0 con 0x7f3cb80b51c0 2023-03-03T07:59:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.088+0000 7f3cb7fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb80cdf20 con 0x7f3cb80d1580 2023-03-03T07:59:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.088+0000 7f3cb67fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3cb80b51c0 0x7f3cb8160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:48.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.089+0000 7f3cb67fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3cb80b51c0 0x7f3cb8160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60600/0 (socket says 172.21.15.98:60600) 2023-03-03T07:59:48.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.089+0000 7f3cb67fc700 1 -- 172.21.15.98:0/250407553 learned_addr learned my addr 172.21.15.98:0/250407553 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:59:48.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.089+0000 7f3cb77fe700 1 --2- 172.21.15.98:0/250407553 >> v2:172.21.15.100:3300/0 conn(0x7f3cb80d1580 0x7f3cb80d1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:48.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.089+0000 7f3cb67fc700 1 -- 172.21.15.98:0/250407553 >> v2:172.21.15.100:3300/0 conn(0x7f3cb80d1580 msgr2=0x7f3cb80d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:48.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.089+0000 7f3cb67fc700 1 --2- 172.21.15.98:0/250407553 >> v2:172.21.15.100:3300/0 conn(0x7f3cb80d1580 0x7f3cb80d1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:48.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.089+0000 7f3cb67fc700 1 -- 172.21.15.98:0/250407553 >> v1:172.21.15.100:6789/0 conn(0x7f3cb8146360 legacy=0x7f3cb80b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:48.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.089+0000 7f3cb67fc700 1 -- 172.21.15.98:0/250407553 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb80cd680 con 0x7f3cb80b51c0 2023-03-03T07:59:48.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.089+0000 7f3cb77fe700 1 --2- 172.21.15.98:0/250407553 >> v2:172.21.15.100:3300/0 conn(0x7f3cb80d1580 0x7f3cb80d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T07:59:48.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.089+0000 7f3cb67fc700 1 --2- 172.21.15.98:0/250407553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb80b51c0 0x7f3cb8160f80 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3cb80d2fe0 tx=0x7f3cac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9851dc2bed4643 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:48.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.090+0000 7f3cb5ffb700 1 -- 172.21.15.98:0/250407553 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cac004230 con 0x7f3cb80b51c0 2023-03-03T07:59:48.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.090+0000 7f3cb5ffb700 1 -- 172.21.15.98:0/250407553 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cac004390 con 0x7f3cb80b51c0 2023-03-03T07:59:48.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.090+0000 7f3cb5ffb700 1 -- 172.21.15.98:0/250407553 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cac0036a0 con 0x7f3cb80b51c0 2023-03-03T07:59:48.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.090+0000 7f3cb7fff700 1 -- 172.21.15.98:0/250407553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb80b51c0 msgr2=0x7f3cb8160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:48.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.090+0000 7f3cb7fff700 1 --2- 172.21.15.98:0/250407553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb80b51c0 0x7f3cb8160f80 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3cb80d2fe0 tx=0x7f3cac009bd0 comp rx=0 tx=0).stop 2023-03-03T07:59:48.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.090+0000 7f3cb7fff700 1 -- 172.21.15.98:0/250407553 shutdown_connections 2023-03-03T07:59:48.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.090+0000 7f3cb7fff700 1 --2- 172.21.15.98:0/250407553 >> v2:172.21.15.100:3300/0 conn(0x7f3cb80d1580 0x7f3cb80d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:48.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.090+0000 7f3cb7fff700 1 --2- 172.21.15.98:0/250407553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb80b51c0 0x7f3cb8160f80 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:48.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.090+0000 7f3cb7fff700 1 -- 172.21.15.98:0/250407553 >> 172.21.15.98:0/250407553 conn(0x7f3cb806b8e0 msgr2=0x7f3cb80cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:48.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.090+0000 7f3cb7fff700 1 -- 172.21.15.98:0/250407553 shutdown_connections 2023-03-03T07:59:48.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.090+0000 7f3cb7fff700 1 -- 172.21.15.98:0/250407553 wait complete. 2023-03-03T07:59:48.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.092+0000 7f3cb7fff700 1 Processor -- start 2023-03-03T07:59:48.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.093+0000 7f3cb7fff700 1 -- start start 2023-03-03T07:59:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb7fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3cb80b51c0 0x7f3cb80c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb7fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3cb80d1580 0x7f3cb80c7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb8146360 0x7f3cb80c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb7fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80c7080 con 0x7f3cb8146360 2023-03-03T07:59:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb7fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80cad10 con 0x7f3cb80d1580 2023-03-03T07:59:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb7fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80ca100 con 0x7f3cb80b51c0 2023-03-03T07:59:48.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb8146360 0x7f3cb80c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:48.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb8146360 0x7f3cb80c7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60612/0 (socket says 172.21.15.98:60612) 2023-03-03T07:59:48.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb77fe700 1 -- 172.21.15.98:0/447925669 learned_addr learned my addr 172.21.15.98:0/447925669 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:59:48.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb6ffd700 1 --2- 172.21.15.98:0/447925669 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3cb80b51c0 0x7f3cb80c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:48.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb67fc700 1 --2- 172.21.15.98:0/447925669 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3cb80d1580 0x7f3cb80c7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:48.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb77fe700 1 -- 172.21.15.98:0/447925669 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3cb80b51c0 msgr2=0x7f3cb80c5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:48.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb77fe700 1 --2- 172.21.15.98:0/447925669 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3cb80b51c0 0x7f3cb80c5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:48.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb77fe700 1 -- 172.21.15.98:0/447925669 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3cb80d1580 msgr2=0x7f3cb80c7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:48.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb77fe700 1 --2- 172.21.15.98:0/447925669 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3cb80d1580 0x7f3cb80c7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:48.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.094+0000 7f3cb77fe700 1 -- 172.21.15.98:0/447925669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac009550 con 0x7f3cb8146360 2023-03-03T07:59:48.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.095+0000 7f3cb77fe700 1 --2- 172.21.15.98:0/447925669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb8146360 0x7f3cb80c7a70 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000c7f0 tx=0x7f3cb000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:48.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.095+0000 7f3c9ffff700 1 -- 172.21.15.98:0/447925669 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0011800 con 0x7f3cb8146360 2023-03-03T07:59:48.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.095+0000 7f3c9ffff700 1 -- 172.21.15.98:0/447925669 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb0011960 con 0x7f3cb8146360 2023-03-03T07:59:48.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.095+0000 7f3c9ffff700 1 -- 172.21.15.98:0/447925669 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0011d10 con 0x7f3cb8146360 2023-03-03T07:59:48.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.095+0000 7f3cb7fff700 1 -- 172.21.15.98:0/447925669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb80cdc40 con 0x7f3cb8146360 2023-03-03T07:59:48.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.095+0000 7f3cb7fff700 1 -- 172.21.15.98:0/447925669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb80cd960 con 0x7f3cb8146360 2023-03-03T07:59:48.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.096+0000 7f3cb7fff700 1 -- 172.21.15.98:0/447925669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb80d0d80 con 0x7f3cb8146360 2023-03-03T07:59:48.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.098+0000 7f3c9ffff700 1 -- 172.21.15.98:0/447925669 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 86575+0+0 (secure 0 0 0) 0x7f3cb0021070 con 0x7f3cb8146360 2023-03-03T07:59:48.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.100+0000 7f3c9ffff700 1 --2- 172.21.15.98:0/447925669 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3ca4068410 0x7f3ca406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:48.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.100+0000 7f3c9ffff700 1 -- 172.21.15.98:0/447925669 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f3cb0085d00 con 0x7f3cb8146360 2023-03-03T07:59:48.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.100+0000 7f3cb6ffd700 1 --2- 172.21.15.98:0/447925669 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3ca4068410 0x7f3ca406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:48.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.101+0000 7f3cb6ffd700 1 --2- 172.21.15.98:0/447925669 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3ca4068410 0x7f3ca406a8c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3ca800f300 tx=0x7f3ca80042c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:48.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.104+0000 7f3c9ffff700 1 -- 172.21.15.98:0/447925669 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3cb00522b0 con 0x7f3cb8146360 2023-03-03T07:59:48.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.339+0000 7f3cb7fff700 1 -- 172.21.15.98:0/447925669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cb8068300 con 0x7f3cb8146360 2023-03-03T07:59:48.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.339+0000 7f3c9ffff700 1 -- 172.21.15.98:0/447925669 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f3cb00558d0 con 0x7f3cb8146360 2023-03-03T07:59:48.341 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:59:48.341 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:59:48.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.342+0000 7f3cb7fff700 1 -- 172.21.15.98:0/447925669 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3ca4068410 msgr2=0x7f3ca406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:48.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.342+0000 7f3cb7fff700 1 --2- 172.21.15.98:0/447925669 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3ca4068410 0x7f3ca406a8c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3ca800f300 tx=0x7f3ca80042c0 comp rx=0 tx=0).stop 2023-03-03T07:59:48.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.342+0000 7f3cb7fff700 1 -- 172.21.15.98:0/447925669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb8146360 msgr2=0x7f3cb80c7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:48.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.342+0000 7f3cb7fff700 1 --2- 172.21.15.98:0/447925669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb8146360 0x7f3cb80c7a70 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000c7f0 tx=0x7f3cb000cb00 comp rx=0 tx=0).stop 2023-03-03T07:59:48.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.342+0000 7f3cb7fff700 1 -- 172.21.15.98:0/447925669 shutdown_connections 2023-03-03T07:59:48.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.342+0000 7f3cb7fff700 1 --2- 172.21.15.98:0/447925669 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3ca4068410 0x7f3ca406a8c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:48.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.342+0000 7f3cb7fff700 1 --2- 172.21.15.98:0/447925669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb8146360 0x7f3cb80c7a70 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:48.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.342+0000 7f3cb7fff700 1 --2- 172.21.15.98:0/447925669 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3cb80b51c0 0x7f3cb80c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:48.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.342+0000 7f3cb7fff700 1 --2- 172.21.15.98:0/447925669 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3cb80d1580 0x7f3cb80c7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:48.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.342+0000 7f3cb7fff700 1 -- 172.21.15.98:0/447925669 >> 172.21.15.98:0/447925669 conn(0x7f3cb806b8e0 msgr2=0x7f3cb80b8160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.342+0000 7f3cb7fff700 1 -- 172.21.15.98:0/447925669 shutdown_connections 2023-03-03T07:59:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:48.342+0000 7f3cb7fff700 1 -- 172.21.15.98:0/447925669 wait complete. 2023-03-03T07:59:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T07:59:48.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2023-03-03T07:59:48.646 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":6120,"entity":{"name":{"type":"client","num":6120},"addr":{"type":"any","addr":"172.21.15.98:0","nonce":1023372945}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":7,"uptime":59.826256278000002,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.160325407841082,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.6120 172.21.15.98:0/1023372945","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"y","hostname":"smithi098","pid":"108777","root":"/"}}] 2023-03-03T07:59:48.647 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "y", "hostname": "smithi098", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108777", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.98:0", "nonce": 1023372945, "type": "any" }, "name": { "num": 6120, "type": "client" } }, "id": 6120, "inst": "client.6120 172.21.15.98:0/1023372945", "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 7, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.160325407841082 }, "state": "open", "uptime": 59.826256278 } ] 2023-03-03T07:59:48.648 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2023-03-03T07:59:53.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:59:53.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.902+0000 7fab9a7ff700 1 Processor -- start 2023-03-03T07:59:53.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.902+0000 7fab9a7ff700 1 -- start start 2023-03-03T07:59:53.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.903+0000 7fab9a7ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab940b87c0 0x7fab940cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:53.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.903+0000 7fab9a7ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fab940b8b90 0x7fab940b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:53.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.903+0000 7fab9a7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab940b5240 0x7fab940b5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:53.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.903+0000 7fab9a7ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab940b5bb0 con 0x7fab940b87c0 2023-03-03T07:59:53.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.903+0000 7fab9a7ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab940d0ef0 con 0x7fab940b8b90 2023-03-03T07:59:53.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.903+0000 7fab9a7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab940d0600 con 0x7fab940b5240 2023-03-03T07:59:53.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.904+0000 7fab93fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab940b87c0 0x7fab940cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:53.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.904+0000 7fab93fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab940b87c0 0x7fab940cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60628/0 (socket says 172.21.15.98:60628) 2023-03-03T07:59:53.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.904+0000 7fab93fff700 1 -- 172.21.15.98:0/3703037593 learned_addr learned my addr 172.21.15.98:0/3703037593 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:59:53.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.904+0000 7fab937fe700 1 --2- 172.21.15.98:0/3703037593 >> v2:172.21.15.100:3300/0 conn(0x7fab940b8b90 0x7fab940b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:53.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.904+0000 7fab98d9c700 1 --2- 172.21.15.98:0/3703037593 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab940b5240 0x7fab940b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:53.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.904+0000 7fab93fff700 1 -- 172.21.15.98:0/3703037593 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab940b5240 msgr2=0x7fab940b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:53.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.904+0000 7fab93fff700 1 --2- 172.21.15.98:0/3703037593 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab940b5240 0x7fab940b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:53.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.904+0000 7fab93fff700 1 -- 172.21.15.98:0/3703037593 >> v2:172.21.15.100:3300/0 conn(0x7fab940b8b90 msgr2=0x7fab940b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:53.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.904+0000 7fab93fff700 1 --2- 172.21.15.98:0/3703037593 >> v2:172.21.15.100:3300/0 conn(0x7fab940b8b90 0x7fab940b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:53.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.904+0000 7fab93fff700 1 -- 172.21.15.98:0/3703037593 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab940cd970 con 0x7fab940b87c0 2023-03-03T07:59:53.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.905+0000 7fab93fff700 1 --2- 172.21.15.98:0/3703037593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab940b87c0 0x7fab940cffe0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fab9408c240 tx=0x7fab84009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812b400ff84ac304 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:53.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.905+0000 7fab92ffd700 1 -- 172.21.15.98:0/3703037593 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8400b870 con 0x7fab940b87c0 2023-03-03T07:59:53.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.905+0000 7fab92ffd700 1 -- 172.21.15.98:0/3703037593 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8400b9d0 con 0x7fab940b87c0 2023-03-03T07:59:53.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.905+0000 7fab92ffd700 1 -- 172.21.15.98:0/3703037593 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8400bd60 con 0x7fab940b87c0 2023-03-03T07:59:53.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.905+0000 7fab9a7ff700 1 -- 172.21.15.98:0/3703037593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab940b87c0 msgr2=0x7fab940cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:53.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.906+0000 7fab9a7ff700 1 --2- 172.21.15.98:0/3703037593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab940b87c0 0x7fab940cffe0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fab9408c240 tx=0x7fab84009bd0 comp rx=0 tx=0).stop 2023-03-03T07:59:53.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.906+0000 7fab9a7ff700 1 -- 172.21.15.98:0/3703037593 shutdown_connections 2023-03-03T07:59:53.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.906+0000 7fab9a7ff700 1 --2- 172.21.15.98:0/3703037593 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab940b5240 0x7fab940b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:53.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.906+0000 7fab9a7ff700 1 --2- 172.21.15.98:0/3703037593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab940b87c0 0x7fab940cffe0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:53.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.906+0000 7fab9a7ff700 1 --2- 172.21.15.98:0/3703037593 >> v2:172.21.15.100:3300/0 conn(0x7fab940b8b90 0x7fab940b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:53.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.906+0000 7fab9a7ff700 1 -- 172.21.15.98:0/3703037593 >> 172.21.15.98:0/3703037593 conn(0x7fab9406aee0 msgr2=0x7fab9406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:53.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.906+0000 7fab9a7ff700 1 -- 172.21.15.98:0/3703037593 shutdown_connections 2023-03-03T07:59:53.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.906+0000 7fab9a7ff700 1 -- 172.21.15.98:0/3703037593 wait complete. 2023-03-03T07:59:53.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.908+0000 7fab9a7ff700 1 Processor -- start 2023-03-03T07:59:53.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.908+0000 7fab9a7ff700 1 -- start start 2023-03-03T07:59:53.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.909+0000 7fab9a7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab940b5240 0x7fab941e7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:53.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.909+0000 7fab9a7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fab940b87c0 0x7fab940c8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:53.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.909+0000 7fab9a7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab940b8b90 0x7fab940c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:53.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.909+0000 7fab9a7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab940c95f0 con 0x7fab940b5240 2023-03-03T07:59:53.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.909+0000 7fab9a7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab94148150 con 0x7fab940b87c0 2023-03-03T07:59:53.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.909+0000 7fab9a7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab94148430 con 0x7fab940b8b90 2023-03-03T07:59:53.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.909+0000 7fab93fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab940b5240 0x7fab941e7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:53.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.909+0000 7fab93fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab940b5240 0x7fab941e7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60636/0 (socket says 172.21.15.98:60636) 2023-03-03T07:59:53.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.909+0000 7fab93fff700 1 -- 172.21.15.98:0/2105845676 learned_addr learned my addr 172.21.15.98:0/2105845676 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:59:53.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.910+0000 7fab98d9c700 1 --2- 172.21.15.98:0/2105845676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab940b8b90 0x7fab940c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:53.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.910+0000 7fab937fe700 1 --2- 172.21.15.98:0/2105845676 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fab940b87c0 0x7fab940c8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:53.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.910+0000 7fab93fff700 1 -- 172.21.15.98:0/2105845676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab940b8b90 msgr2=0x7fab940c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:53.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.910+0000 7fab93fff700 1 --2- 172.21.15.98:0/2105845676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab940b8b90 0x7fab940c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:53.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.910+0000 7fab93fff700 1 -- 172.21.15.98:0/2105845676 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fab940b87c0 msgr2=0x7fab940c8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:53.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.910+0000 7fab93fff700 1 --2- 172.21.15.98:0/2105845676 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fab940b87c0 0x7fab940c8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:53.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.910+0000 7fab93fff700 1 -- 172.21.15.98:0/2105845676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab84009550 con 0x7fab940b5240 2023-03-03T07:59:53.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.910+0000 7fab93fff700 1 --2- 172.21.15.98:0/2105845676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab940b5240 0x7fab941e7ae0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fab84000c00 tx=0x7fab84004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:53.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.910+0000 7fab90ff9700 1 -- 172.21.15.98:0/2105845676 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84004540 con 0x7fab940b5240 2023-03-03T07:59:53.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.910+0000 7fab90ff9700 1 -- 172.21.15.98:0/2105845676 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab840036a0 con 0x7fab940b5240 2023-03-03T07:59:53.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.910+0000 7fab90ff9700 1 -- 172.21.15.98:0/2105845676 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84003a50 con 0x7fab940b5240 2023-03-03T07:59:53.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.910+0000 7fab9a7ff700 1 -- 172.21.15.98:0/2105845676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab94073a80 con 0x7fab940b5240 2023-03-03T07:59:53.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.911+0000 7fab9a7ff700 1 -- 172.21.15.98:0/2105845676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab940cdc50 con 0x7fab940b5240 2023-03-03T07:59:53.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.912+0000 7fab9a7ff700 1 -- 172.21.15.98:0/2105845676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab940cdf30 con 0x7fab940b5240 2023-03-03T07:59:53.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.913+0000 7fab90ff9700 1 -- 172.21.15.98:0/2105845676 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 86575+0+0 (secure 0 0 0) 0x7fab8400f050 con 0x7fab940b5240 2023-03-03T07:59:53.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.914+0000 7fab90ff9700 1 --2- 172.21.15.98:0/2105845676 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fab80068410 0x7fab8006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.914+0000 7fab90ff9700 1 -- 172.21.15.98:0/2105845676 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fab84087880 con 0x7fab940b5240 2023-03-03T07:59:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.915+0000 7fab937fe700 1 --2- 172.21.15.98:0/2105845676 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fab80068410 0x7fab8006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.915+0000 7fab937fe700 1 --2- 172.21.15.98:0/2105845676 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fab80068410 0x7fab8006a8c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fab940cca60 tx=0x7fab88008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:53.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:53.919+0000 7fab90ff9700 1 -- 172.21.15.98:0/2105845676 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fab84053e90 con 0x7fab940b5240 2023-03-03T07:59:54.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:54.151+0000 7fab9a7ff700 1 -- 172.21.15.98:0/2105845676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab940729b0 con 0x7fab940b5240 2023-03-03T07:59:54.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:54.151+0000 7fab90ff9700 1 -- 172.21.15.98:0/2105845676 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7fab840574b0 con 0x7fab940b5240 2023-03-03T07:59:54.153 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:59:54.153 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:59:54.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:54.154+0000 7fab9a7ff700 1 -- 172.21.15.98:0/2105845676 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fab80068410 msgr2=0x7fab8006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:54.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:54.154+0000 7fab9a7ff700 1 --2- 172.21.15.98:0/2105845676 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fab80068410 0x7fab8006a8c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fab940cca60 tx=0x7fab88008040 comp rx=0 tx=0).stop 2023-03-03T07:59:54.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:54.154+0000 7fab9a7ff700 1 -- 172.21.15.98:0/2105845676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab940b5240 msgr2=0x7fab941e7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:54.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:54.154+0000 7fab9a7ff700 1 --2- 172.21.15.98:0/2105845676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab940b5240 0x7fab941e7ae0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fab84000c00 tx=0x7fab84004750 comp rx=0 tx=0).stop 2023-03-03T07:59:54.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:54.154+0000 7fab9a7ff700 1 -- 172.21.15.98:0/2105845676 shutdown_connections 2023-03-03T07:59:54.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:54.154+0000 7fab9a7ff700 1 --2- 172.21.15.98:0/2105845676 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fab80068410 0x7fab8006a8c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:54.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:54.154+0000 7fab9a7ff700 1 --2- 172.21.15.98:0/2105845676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab940b5240 0x7fab941e7ae0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:54.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:54.154+0000 7fab9a7ff700 1 --2- 172.21.15.98:0/2105845676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab940b8b90 0x7fab940c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:54.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:54.154+0000 7fab9a7ff700 1 --2- 172.21.15.98:0/2105845676 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fab940b87c0 0x7fab940c8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:54.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:54.154+0000 7fab9a7ff700 1 -- 172.21.15.98:0/2105845676 >> 172.21.15.98:0/2105845676 conn(0x7fab9406aee0 msgr2=0x7fab9415f080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:54.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:54.154+0000 7fab9a7ff700 1 -- 172.21.15.98:0/2105845676 shutdown_connections 2023-03-03T07:59:54.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:54.154+0000 7fab9a7ff700 1 -- 172.21.15.98:0/2105845676 wait complete. 2023-03-03T07:59:54.157 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T07:59:54.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2023-03-03T07:59:54.455 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":6120,"entity":{"name":{"type":"client","num":6120},"addr":{"type":"any","addr":"172.21.15.98:0","nonce":1023372945}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":7,"uptime":65.635184631000001,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.904883628680714,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.6120 172.21.15.98:0/1023372945","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"y","hostname":"smithi098","pid":"108777","root":"/"}}] 2023-03-03T07:59:54.456 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "y", "hostname": "smithi098", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108777", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.98:0", "nonce": 1023372945, "type": "any" }, "name": { "num": 6120, "type": "client" } }, "id": 6120, "inst": "client.6120 172.21.15.98:0/1023372945", "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 7, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.904883628680714 }, "state": "open", "uptime": 65.635184631 } ] 2023-03-03T07:59:54.456 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2023-03-03T07:59:59.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:59:59.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.709+0000 7f6a46d9d700 1 Processor -- start 2023-03-03T07:59:59.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.710+0000 7f6a46d9d700 1 -- start start 2023-03-03T07:59:59.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.711+0000 7f6a46d9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6a400d0010 0x7f6a400d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:59.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.711+0000 7f6a46d9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a400b0c50 con 0x7f6a400d1650 2023-03-03T07:59:59.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.711+0000 7f6a46d9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a400b0040 con 0x7f6a400b4a00 2023-03-03T07:59:59.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.711+0000 7f6a46d9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a400b0320 con 0x7f6a400d0010 2023-03-03T07:59:59.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.711+0000 7f6a4659c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6a400d1650 0x7f6a400b2ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34220/0 (socket says 172.21.15.98:34220) 2023-03-03T07:59:59.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.711+0000 7f6a4659c700 1 -- 172.21.15.98:0/80246095 learned_addr learned my addr 172.21.15.98:0/80246095 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T07:59:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.712+0000 7f6a45d9b700 1 --2- 172.21.15.98:0/80246095 >> v2:172.21.15.100:3300/0 conn(0x7f6a400d0010 0x7f6a400d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.712+0000 7f6a44d99700 1 -- 172.21.15.98:0/80246095 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195686835 0 0) 0x7f6a400b0c50 con 0x7f6a400d1650 2023-03-03T07:59:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.712+0000 7f6a44d99700 1 -- 172.21.15.98:0/80246095 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a2c0035b0 con 0x7f6a400d1650 2023-03-03T07:59:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.712+0000 7f6a44d99700 1 -- 172.21.15.98:0/80246095 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839879495 0 0) 0x7f6a400b0040 con 0x7f6a400b4a00 2023-03-03T07:59:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.712+0000 7f6a44d99700 1 -- 172.21.15.98:0/80246095 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a400b0c50 con 0x7f6a400b4a00 2023-03-03T07:59:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.712+0000 7f6a45d9b700 1 -- 172.21.15.98:0/80246095 >> v1:172.21.15.100:6789/0 conn(0x7f6a400b4a00 legacy=0x7f6a400d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.712+0000 7f6a45d9b700 1 -- 172.21.15.98:0/80246095 >> v1:172.21.15.98:6789/0 conn(0x7f6a400d1650 legacy=0x7f6a400b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.712+0000 7f6a44d99700 1 -- 172.21.15.98:0/80246095 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3036504663 0 0) 0x7f6a2c0035b0 con 0x7f6a400d1650 2023-03-03T07:59:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.712+0000 7f6a45d9b700 1 -- 172.21.15.98:0/80246095 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a400b0600 con 0x7f6a400d0010 2023-03-03T07:59:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.713+0000 7f6a45d9b700 1 --2- 172.21.15.98:0/80246095 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a400d0010 0x7f6a400d03e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009820 tx=0x7f6a3c009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=782a530864bdae0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.713+0000 7f6a44d99700 1 -- 172.21.15.98:0/80246095 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a380019f0 con 0x7f6a400d0010 2023-03-03T07:59:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.713+0000 7f6a44d99700 1 -- 172.21.15.98:0/80246095 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3c005c30 con 0x7f6a400d0010 2023-03-03T07:59:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.713+0000 7f6a44d99700 1 -- 172.21.15.98:0/80246095 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c004030 con 0x7f6a400d0010 2023-03-03T07:59:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.713+0000 7f6a46d9d700 1 -- 172.21.15.98:0/80246095 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a400d0010 msgr2=0x7f6a400d03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.713+0000 7f6a46d9d700 1 --2- 172.21.15.98:0/80246095 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a400d0010 0x7f6a400d03e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009820 tx=0x7f6a3c009b30 comp rx=0 tx=0).stop 2023-03-03T07:59:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.714+0000 7f6a46d9d700 1 -- 172.21.15.98:0/80246095 shutdown_connections 2023-03-03T07:59:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.714+0000 7f6a46d9d700 1 --2- 172.21.15.98:0/80246095 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a400d0010 0x7f6a400d03e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.714+0000 7f6a46d9d700 1 -- 172.21.15.98:0/80246095 >> 172.21.15.98:0/80246095 conn(0x7f6a40073b00 msgr2=0x7f6a400d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.714+0000 7f6a46d9d700 1 -- 172.21.15.98:0/80246095 shutdown_connections 2023-03-03T07:59:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.714+0000 7f6a46d9d700 1 -- 172.21.15.98:0/80246095 wait complete. 2023-03-03T07:59:59.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.716+0000 7f6a46d9d700 1 Processor -- start 2023-03-03T07:59:59.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.716+0000 7f6a46d9d700 1 -- start start 2023-03-03T07:59:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.717+0000 7f6a46d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a400b4a00 0x7f6a401e6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.717+0000 7f6a46d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a400d0010 0x7f6a400c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.717+0000 7f6a46d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a400d1650 0x7f6a4009f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.717+0000 7f6a46d9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400a0c00 con 0x7f6a400d0010 2023-03-03T07:59:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.717+0000 7f6a46d9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400ca160 con 0x7f6a400b4a00 2023-03-03T07:59:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.717+0000 7f6a46d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a401f2230 con 0x7f6a400d1650 2023-03-03T07:59:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.717+0000 7f6a4559a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a400d0010 0x7f6a400c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.717+0000 7f6a4559a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a400d0010 0x7f6a400c9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44586/0 (socket says 172.21.15.98:44586) 2023-03-03T07:59:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.717+0000 7f6a45d9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a400b4a00 0x7f6a401e6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.717+0000 7f6a4559a700 1 -- 172.21.15.98:0/3579684189 learned_addr learned my addr 172.21.15.98:0/3579684189 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T07:59:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.718+0000 7f6a4659c700 1 --2- 172.21.15.98:0/3579684189 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a400d1650 0x7f6a4009f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.718+0000 7f6a4559a700 1 -- 172.21.15.98:0/3579684189 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a400d1650 msgr2=0x7f6a4009f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.718+0000 7f6a4559a700 1 --2- 172.21.15.98:0/3579684189 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a400d1650 0x7f6a4009f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.718+0000 7f6a4559a700 1 -- 172.21.15.98:0/3579684189 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a400b4a00 msgr2=0x7f6a401e6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.718+0000 7f6a4559a700 1 --2- 172.21.15.98:0/3579684189 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a400b4a00 0x7f6a401e6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:59.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.718+0000 7f6a4559a700 1 -- 172.21.15.98:0/3579684189 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3c009430 con 0x7f6a400d0010 2023-03-03T07:59:59.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.718+0000 7f6a4559a700 1 --2- 172.21.15.98:0/3579684189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a400d0010 0x7f6a400c9c20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000c490 tx=0x7f6a3000c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:59.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.719+0000 7f6a367fc700 1 -- 172.21.15.98:0/3579684189 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3000cea0 con 0x7f6a400d0010 2023-03-03T07:59:59.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.719+0000 7f6a367fc700 1 -- 172.21.15.98:0/3579684189 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a300052f0 con 0x7f6a400d0010 2023-03-03T07:59:59.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.719+0000 7f6a46d9d700 1 -- 172.21.15.98:0/3579684189 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a400d0b30 con 0x7f6a400d0010 2023-03-03T07:59:59.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.719+0000 7f6a367fc700 1 -- 172.21.15.98:0/3579684189 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30003680 con 0x7f6a400d0010 2023-03-03T07:59:59.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.719+0000 7f6a46d9d700 1 -- 172.21.15.98:0/3579684189 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a400b08e0 con 0x7f6a400d0010 2023-03-03T07:59:59.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.720+0000 7f6a46d9d700 1 -- 172.21.15.98:0/3579684189 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a400d0cc0 con 0x7f6a400d0010 2023-03-03T07:59:59.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.721+0000 7f6a367fc700 1 -- 172.21.15.98:0/3579684189 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f6a30019030 con 0x7f6a400d0010 2023-03-03T07:59:59.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.722+0000 7f6a367fc700 1 --2- 172.21.15.98:0/3579684189 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6a240683c0 0x7f6a2406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:59.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.722+0000 7f6a367fc700 1 -- 172.21.15.98:0/3579684189 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f6a30085170 con 0x7f6a400d0010 2023-03-03T07:59:59.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.723+0000 7f6a45d9b700 1 --2- 172.21.15.98:0/3579684189 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6a240683c0 0x7f6a2406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:59.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.723+0000 7f6a45d9b700 1 --2- 172.21.15.98:0/3579684189 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6a240683c0 0x7f6a2406a870 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009fd0 tx=0x7f6a3c008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:59.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.726+0000 7f6a367fc700 1 -- 172.21.15.98:0/3579684189 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6a300517d0 con 0x7f6a400d0010 2023-03-03T07:59:59.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.960+0000 7f6a46d9d700 1 -- 172.21.15.98:0/3579684189 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a400729b0 con 0x7f6a400d0010 2023-03-03T07:59:59.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.961+0000 7f6a367fc700 1 -- 172.21.15.98:0/3579684189 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f6a30054df0 con 0x7f6a400d0010 2023-03-03T07:59:59.962 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T07:59:59.963 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T07:59:59.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.963+0000 7f6a46d9d700 1 -- 172.21.15.98:0/3579684189 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6a240683c0 msgr2=0x7f6a2406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:59.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.963+0000 7f6a46d9d700 1 --2- 172.21.15.98:0/3579684189 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6a240683c0 0x7f6a2406a870 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009fd0 tx=0x7f6a3c008070 comp rx=0 tx=0).stop 2023-03-03T07:59:59.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.963+0000 7f6a46d9d700 1 -- 172.21.15.98:0/3579684189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a400d0010 msgr2=0x7f6a400c9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:59.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.963+0000 7f6a46d9d700 1 --2- 172.21.15.98:0/3579684189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a400d0010 0x7f6a400c9c20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000c490 tx=0x7f6a3000c7a0 comp rx=0 tx=0).stop 2023-03-03T07:59:59.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.964+0000 7f6a46d9d700 1 -- 172.21.15.98:0/3579684189 shutdown_connections 2023-03-03T07:59:59.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.964+0000 7f6a46d9d700 1 --2- 172.21.15.98:0/3579684189 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6a240683c0 0x7f6a2406a870 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:59.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.964+0000 7f6a46d9d700 1 --2- 172.21.15.98:0/3579684189 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a400d1650 0x7f6a4009f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:59.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.964+0000 7f6a46d9d700 1 --2- 172.21.15.98:0/3579684189 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a400b4a00 0x7f6a401e6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:59.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.964+0000 7f6a46d9d700 1 --2- 172.21.15.98:0/3579684189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a400d0010 0x7f6a400c9c20 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:59.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.964+0000 7f6a46d9d700 1 -- 172.21.15.98:0/3579684189 >> 172.21.15.98:0/3579684189 conn(0x7f6a40073b00 msgr2=0x7f6a400b8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:59.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.964+0000 7f6a46d9d700 1 -- 172.21.15.98:0/3579684189 shutdown_connections 2023-03-03T07:59:59.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T07:59:59.964+0000 7f6a46d9d700 1 -- 172.21.15.98:0/3579684189 wait complete. 2023-03-03T07:59:59.967 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T07:59:59.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2023-03-03T08:00:00.270 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2023-03-03T08:00:00.271 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2023-03-03T08:00:00.271 DEBUG:tasks.ceph_test_case:wait_until_true: success in 60s and 0 retries 2023-03-03T08:00:00.272 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-03-03T08:00:00.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:00:00.401 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:00:00.557 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-03-03T08:00:00.558 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:00:00.558 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:00:00.627 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T08:00:00.627 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T08:00:00.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:00.628 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-03-03T08:00:00.695 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-03-03T08:00:00.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:00:00.697 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:00:00.697 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:00:00.766 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T08:00:00.767 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T08:00:00.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:00.768 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-03-03T08:00:00.834 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-03-03T08:00:00.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:00:00.836 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:00:00.837 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:00:00.904 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T08:00:00.905 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T08:00:00.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:00.906 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-03-03T08:00:00.973 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-03-03T08:00:00.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:00:00.975 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:00:00.975 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:00:01.043 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T08:00:01.044 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T08:00:01.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:01.044 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-03-03T08:00:01.112 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-03-03T08:00:01.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:00:01.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:00:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.359+0000 7f49fd7ff700 1 Processor -- start 2023-03-03T08:00:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.359+0000 7f49fd7ff700 1 -- start start 2023-03-03T08:00:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.360+0000 7f49fd7ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f49f8159010 0x7f49f80c9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.360+0000 7f49fd7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f8070ff0 0x7f49f80c9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.360+0000 7f49fd7ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f81597e0 con 0x7f49f80cc110 2023-03-03T08:00:01.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.360+0000 7f49fd7ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f8159a80 con 0x7f49f8159010 2023-03-03T08:00:01.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.360+0000 7f49fd7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f8159d60 con 0x7f49f8070ff0 2023-03-03T08:00:01.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.360+0000 7f49f6ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f49f8159010 0x7f49f80c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:01.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.361+0000 7f49f6ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f49f8159010 0x7f49f80c9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44592/0 (socket says 172.21.15.98:44592) 2023-03-03T08:00:01.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.361+0000 7f49effff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f49f80cc110 0x7f49f815cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34232/0 (socket says 172.21.15.98:34232) 2023-03-03T08:00:01.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.361+0000 7f49f6ffd700 1 -- 172.21.15.98:0/3282174477 learned_addr learned my addr 172.21.15.98:0/3282174477 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.361+0000 7f49f77fe700 1 --2- 172.21.15.98:0/3282174477 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f8070ff0 0x7f49f80c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.361+0000 7f49f6ffd700 1 -- 172.21.15.98:0/3282174477 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f8070ff0 msgr2=0x7f49f80c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.361+0000 7f49f6ffd700 1 --2- 172.21.15.98:0/3282174477 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f8070ff0 0x7f49f80c9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.361+0000 7f49f6ffd700 1 -- 172.21.15.98:0/3282174477 >> v1:172.21.15.98:6789/0 conn(0x7f49f80cc110 legacy=0x7f49f815cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.361+0000 7f49f6ffd700 1 -- 172.21.15.98:0/3282174477 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f815a040 con 0x7f49f8159010 2023-03-03T08:00:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.361+0000 7f49f67fc700 1 -- 172.21.15.98:0/3282174477 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816103411 0 0) 0x7f49f81597e0 con 0x7f49f80cc110 2023-03-03T08:00:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.362+0000 7f49f6ffd700 1 --2- 172.21.15.98:0/3282174477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49f8159010 0x7f49f80c9450 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f49e800bb90 tx=0x7f49e800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3835cdf5c82a08e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.362+0000 7f49f67fc700 1 -- 172.21.15.98:0/3282174477 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e800dd30 con 0x7f49f8159010 2023-03-03T08:00:01.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.362+0000 7f49f67fc700 1 -- 172.21.15.98:0/3282174477 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e800de90 con 0x7f49f8159010 2023-03-03T08:00:01.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.362+0000 7f49f67fc700 1 -- 172.21.15.98:0/3282174477 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8010690 con 0x7f49f8159010 2023-03-03T08:00:01.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.362+0000 7f49fd7ff700 1 -- 172.21.15.98:0/3282174477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49f8159010 msgr2=0x7f49f80c9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:01.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.362+0000 7f49fd7ff700 1 --2- 172.21.15.98:0/3282174477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49f8159010 0x7f49f80c9450 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f49e800bb90 tx=0x7f49e800bea0 comp rx=0 tx=0).stop 2023-03-03T08:00:01.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.362+0000 7f49fd7ff700 1 -- 172.21.15.98:0/3282174477 shutdown_connections 2023-03-03T08:00:01.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.362+0000 7f49fd7ff700 1 --2- 172.21.15.98:0/3282174477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49f8159010 0x7f49f80c9450 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:01.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.362+0000 7f49fd7ff700 1 --2- 172.21.15.98:0/3282174477 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f8070ff0 0x7f49f80c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:01.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.362+0000 7f49fd7ff700 1 -- 172.21.15.98:0/3282174477 >> 172.21.15.98:0/3282174477 conn(0x7f49f80b09f0 msgr2=0x7f49f80b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:01.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.363+0000 7f49fd7ff700 1 -- 172.21.15.98:0/3282174477 shutdown_connections 2023-03-03T08:00:01.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.363+0000 7f49fd7ff700 1 -- 172.21.15.98:0/3282174477 wait complete. 2023-03-03T08:00:01.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.365+0000 7f49fd7ff700 1 Processor -- start 2023-03-03T08:00:01.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.365+0000 7f49fd7ff700 1 -- start start 2023-03-03T08:00:01.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.366+0000 7f49fd7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f8070ff0 0x7f49f8151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:01.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.366+0000 7f49fd7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49f80cc110 0x7f49f8152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:01.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.366+0000 7f49fd7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49f8159010 0x7f49f8152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:01.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.366+0000 7f49fd7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f80b28e0 con 0x7f49f8159010 2023-03-03T08:00:01.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.366+0000 7f49fd7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f8152170 con 0x7f49f80cc110 2023-03-03T08:00:01.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.366+0000 7f49fd7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f80b1cf0 con 0x7f49f8070ff0 2023-03-03T08:00:01.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.366+0000 7f49f77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49f8159010 0x7f49f8152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:01.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.366+0000 7f49f6ffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f8070ff0 0x7f49f8151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:01.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.366+0000 7f49effff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49f80cc110 0x7f49f8152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:01.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.366+0000 7f49f6ffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f8070ff0 0x7f49f8151c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:41214/0 (socket says 172.21.15.98:41214) 2023-03-03T08:00:01.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.366+0000 7f49f6ffd700 1 -- 172.21.15.98:0/2467581332 learned_addr learned my addr 172.21.15.98:0/2467581332 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:01.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.367+0000 7f49f6ffd700 1 -- 172.21.15.98:0/2467581332 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49f80cc110 msgr2=0x7f49f8152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:01.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.367+0000 7f49f6ffd700 1 --2- 172.21.15.98:0/2467581332 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49f80cc110 0x7f49f8152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:01.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.367+0000 7f49f6ffd700 1 -- 172.21.15.98:0/2467581332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49f8159010 msgr2=0x7f49f8152fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:01.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.367+0000 7f49f6ffd700 1 --2- 172.21.15.98:0/2467581332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49f8159010 0x7f49f8152fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:01.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.367+0000 7f49f6ffd700 1 -- 172.21.15.98:0/2467581332 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e800b7f0 con 0x7f49f8070ff0 2023-03-03T08:00:01.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.367+0000 7f49effff700 1 --2- 172.21.15.98:0/2467581332 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49f80cc110 0x7f49f8152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:00:01.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.367+0000 7f49f77fe700 1 --2- 172.21.15.98:0/2467581332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49f8159010 0x7f49f8152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:00:01.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.367+0000 7f49f6ffd700 1 --2- 172.21.15.98:0/2467581332 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f8070ff0 0x7f49f8151c30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f49f80cff00 tx=0x7f49e8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:01.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.368+0000 7f49ef7fe700 1 -- 172.21.15.98:0/2467581332 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e801f070 con 0x7f49f8070ff0 2023-03-03T08:00:01.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.368+0000 7f49ef7fe700 1 -- 172.21.15.98:0/2467581332 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e8003970 con 0x7f49f8070ff0 2023-03-03T08:00:01.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.368+0000 7f49ef7fe700 1 -- 172.21.15.98:0/2467581332 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8003d20 con 0x7f49f8070ff0 2023-03-03T08:00:01.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.368+0000 7f49fd7ff700 1 -- 172.21.15.98:0/2467581332 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f80cc740 con 0x7f49f8070ff0 2023-03-03T08:00:01.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.368+0000 7f49fd7ff700 1 -- 172.21.15.98:0/2467581332 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f815a320 con 0x7f49f8070ff0 2023-03-03T08:00:01.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.369+0000 7f49fd7ff700 1 -- 172.21.15.98:0/2467581332 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f81593e0 con 0x7f49f8070ff0 2023-03-03T08:00:01.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.370+0000 7f49ef7fe700 1 -- 172.21.15.98:0/2467581332 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f49e8019400 con 0x7f49f8070ff0 2023-03-03T08:00:01.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.371+0000 7f49ef7fe700 1 --2- 172.21.15.98:0/2467581332 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49d80683c0 0x7f49d806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:01.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.371+0000 7f49ef7fe700 1 -- 172.21.15.98:0/2467581332 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f49e8089310 con 0x7f49f8070ff0 2023-03-03T08:00:01.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.372+0000 7f49effff700 1 --2- 172.21.15.98:0/2467581332 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49d80683c0 0x7f49d806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:01.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.372+0000 7f49effff700 1 --2- 172.21.15.98:0/2467581332 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49d80683c0 0x7f49d806a870 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f49f80c9110 tx=0x7f49e0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:01.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.377+0000 7f49ef7fe700 1 -- 172.21.15.98:0/2467581332 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f49e8055910 con 0x7f49f8070ff0 2023-03-03T08:00:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.613+0000 7f49fd7ff700 1 -- 172.21.15.98:0/2467581332 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49f80b2a20 con 0x7f49f8070ff0 2023-03-03T08:00:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.614+0000 7f49ef7fe700 1 -- 172.21.15.98:0/2467581332 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f49e8058f30 con 0x7f49f8070ff0 2023-03-03T08:00:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:01.616 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T08:00:01.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.617+0000 7f49fd7ff700 1 -- 172.21.15.98:0/2467581332 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49d80683c0 msgr2=0x7f49d806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:01.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.617+0000 7f49fd7ff700 1 --2- 172.21.15.98:0/2467581332 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49d80683c0 0x7f49d806a870 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f49f80c9110 tx=0x7f49e0009040 comp rx=0 tx=0).stop 2023-03-03T08:00:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.617+0000 7f49fd7ff700 1 -- 172.21.15.98:0/2467581332 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f8070ff0 msgr2=0x7f49f8151c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.617+0000 7f49fd7ff700 1 --2- 172.21.15.98:0/2467581332 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f8070ff0 0x7f49f8151c30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f49f80cff00 tx=0x7f49e8004750 comp rx=0 tx=0).stop 2023-03-03T08:00:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.617+0000 7f49fd7ff700 1 -- 172.21.15.98:0/2467581332 shutdown_connections 2023-03-03T08:00:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.617+0000 7f49fd7ff700 1 --2- 172.21.15.98:0/2467581332 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f49d80683c0 0x7f49d806a870 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.617+0000 7f49fd7ff700 1 --2- 172.21.15.98:0/2467581332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49f8159010 0x7f49f8152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.617+0000 7f49fd7ff700 1 --2- 172.21.15.98:0/2467581332 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f8070ff0 0x7f49f8151c30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.617+0000 7f49fd7ff700 1 --2- 172.21.15.98:0/2467581332 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49f80cc110 0x7f49f8152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.617+0000 7f49fd7ff700 1 -- 172.21.15.98:0/2467581332 >> 172.21.15.98:0/2467581332 conn(0x7f49f80b09f0 msgr2=0x7f49f8071470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.617+0000 7f49fd7ff700 1 -- 172.21.15.98:0/2467581332 shutdown_connections 2023-03-03T08:00:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.617+0000 7f49fd7ff700 1 -- 172.21.15.98:0/2467581332 wait complete. 2023-03-03T08:00:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T08:00:01.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:00:01.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.890+0000 7f0687dff700 1 Processor -- start 2023-03-03T08:00:01.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.890+0000 7f0687dff700 1 -- start start 2023-03-03T08:00:01.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.891+0000 7f0687dff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f06800b87c0 0x7f06800cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:01.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.891+0000 7f0687dff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f06800b5410 0x7f06800d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:01.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.892+0000 7f0687dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06800d0ef0 con 0x7f06800b8b90 2023-03-03T08:00:01.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.892+0000 7f0687dff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06800d0600 con 0x7f06800b5410 2023-03-03T08:00:01.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.892+0000 7f0687dff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06800cd970 con 0x7f06800b87c0 2023-03-03T08:00:01.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.892+0000 7f068639c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f06800b5410 0x7f06800d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:01.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.892+0000 7f0685b9b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f06800b87c0 0x7f06800cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:01.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.892+0000 7f068539a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f06800b8b90 0x7f06800b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34236/0 (socket says 172.21.15.98:34236) 2023-03-03T08:00:01.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.892+0000 7f068539a700 1 -- 172.21.15.98:0/951118815 learned_addr learned my addr 172.21.15.98:0/951118815 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:01.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.892+0000 7f0685b9b700 1 -- 172.21.15.98:0/951118815 >> v2:172.21.15.98:3300/0 conn(0x7f06800b5410 msgr2=0x7f06800d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:01.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.892+0000 7f0685b9b700 1 --2- 172.21.15.98:0/951118815 >> v2:172.21.15.98:3300/0 conn(0x7f06800b5410 0x7f06800d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:01.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.892+0000 7f0685b9b700 1 -- 172.21.15.98:0/951118815 >> v1:172.21.15.98:6789/0 conn(0x7f06800b8b90 legacy=0x7f06800b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:01.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.892+0000 7f0685b9b700 1 -- 172.21.15.98:0/951118815 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06800cdc50 con 0x7f06800b87c0 2023-03-03T08:00:01.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.893+0000 7f0685b9b700 1 --2- 172.21.15.98:0/951118815 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06800b87c0 0x7f06800cffe0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0680073840 tx=0x7f067c009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e92ff8d4f436ade6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:01.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.893+0000 7f0684b99700 1 -- 172.21.15.98:0/951118815 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c011830 con 0x7f06800b87c0 2023-03-03T08:00:01.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.893+0000 7f0684b99700 1 -- 172.21.15.98:0/951118815 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067c011990 con 0x7f06800b87c0 2023-03-03T08:00:01.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.893+0000 7f0684b99700 1 -- 172.21.15.98:0/951118815 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c011d20 con 0x7f06800b87c0 2023-03-03T08:00:01.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.894+0000 7f0687dff700 1 -- 172.21.15.98:0/951118815 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06800b87c0 msgr2=0x7f06800cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:01.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.894+0000 7f0687dff700 1 --2- 172.21.15.98:0/951118815 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06800b87c0 0x7f06800cffe0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0680073840 tx=0x7f067c009b20 comp rx=0 tx=0).stop 2023-03-03T08:00:01.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.894+0000 7f0687dff700 1 -- 172.21.15.98:0/951118815 shutdown_connections 2023-03-03T08:00:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.894+0000 7f0687dff700 1 --2- 172.21.15.98:0/951118815 >> v2:172.21.15.98:3300/0 conn(0x7f06800b5410 0x7f06800d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.894+0000 7f0687dff700 1 --2- 172.21.15.98:0/951118815 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06800b87c0 0x7f06800cffe0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.894+0000 7f0687dff700 1 -- 172.21.15.98:0/951118815 >> 172.21.15.98:0/951118815 conn(0x7f068006aee0 msgr2=0x7f068006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.894+0000 7f0687dff700 1 -- 172.21.15.98:0/951118815 shutdown_connections 2023-03-03T08:00:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.894+0000 7f0687dff700 1 -- 172.21.15.98:0/951118815 wait complete. 2023-03-03T08:00:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.896+0000 7f0687dff700 1 Processor -- start 2023-03-03T08:00:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.896+0000 7f0687dff700 1 -- start start 2023-03-03T08:00:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.897+0000 7f0687dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06800b5410 0x7f068014b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.897+0000 7f0687dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06800b87c0 0x7f06800c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.897+0000 7f0687dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06800b8b90 0x7f06800cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.897+0000 7f0687dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06801e66f0 con 0x7f06800b87c0 2023-03-03T08:00:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.897+0000 7f0687dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06801e5a70 con 0x7f06800b8b90 2023-03-03T08:00:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.897+0000 7f0687dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06801e5d50 con 0x7f06800b5410 2023-03-03T08:00:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.897+0000 7f068539a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06800b87c0 0x7f06800c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.898+0000 7f068539a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06800b87c0 0x7f06800c9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44606/0 (socket says 172.21.15.98:44606) 2023-03-03T08:00:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.898+0000 7f068539a700 1 -- 172.21.15.98:0/782963839 learned_addr learned my addr 172.21.15.98:0/782963839 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.898+0000 7f0685b9b700 1 --2- 172.21.15.98:0/782963839 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06800b5410 0x7f068014b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.898+0000 7f068639c700 1 --2- 172.21.15.98:0/782963839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06800b8b90 0x7f06800cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.898+0000 7f068539a700 1 -- 172.21.15.98:0/782963839 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06800b5410 msgr2=0x7f068014b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.898+0000 7f068539a700 1 --2- 172.21.15.98:0/782963839 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06800b5410 0x7f068014b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.898+0000 7f068539a700 1 -- 172.21.15.98:0/782963839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06800b8b90 msgr2=0x7f06800cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.898+0000 7f068539a700 1 --2- 172.21.15.98:0/782963839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06800b8b90 0x7f06800cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.898+0000 7f068539a700 1 -- 172.21.15.98:0/782963839 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c0094a0 con 0x7f06800b87c0 2023-03-03T08:00:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.898+0000 7f068539a700 1 --2- 172.21.15.98:0/782963839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06800b87c0 0x7f06800c9680 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f067000c450 tx=0x7f067000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:01.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.899+0000 7f06767fc700 1 -- 172.21.15.98:0/782963839 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670005d70 con 0x7f06800b87c0 2023-03-03T08:00:01.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.899+0000 7f06767fc700 1 -- 172.21.15.98:0/782963839 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067000ce60 con 0x7f06800b87c0 2023-03-03T08:00:01.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.899+0000 7f06767fc700 1 -- 172.21.15.98:0/782963839 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06700072d0 con 0x7f06800b87c0 2023-03-03T08:00:01.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.899+0000 7f0687dff700 1 -- 172.21.15.98:0/782963839 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06800cdf30 con 0x7f06800b87c0 2023-03-03T08:00:01.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.899+0000 7f0687dff700 1 -- 172.21.15.98:0/782963839 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0680073a80 con 0x7f06800b87c0 2023-03-03T08:00:01.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.900+0000 7f0687dff700 1 -- 172.21.15.98:0/782963839 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06800ce0c0 con 0x7f06800b87c0 2023-03-03T08:00:01.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.901+0000 7f06767fc700 1 -- 172.21.15.98:0/782963839 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f0670019030 con 0x7f06800b87c0 2023-03-03T08:00:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.902+0000 7f06767fc700 1 --2- 172.21.15.98:0/782963839 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f06640683c0 0x7f066406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.903+0000 7f06767fc700 1 -- 172.21.15.98:0/782963839 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f0670084f30 con 0x7f06800b87c0 2023-03-03T08:00:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.903+0000 7f0685b9b700 1 --2- 172.21.15.98:0/782963839 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f06640683c0 0x7f066406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.904+0000 7f0685b9b700 1 --2- 172.21.15.98:0/782963839 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f06640683c0 0x7f066406a870 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f067c0090b0 tx=0x7f067c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:01.906+0000 7f06767fc700 1 -- 172.21.15.98:0/782963839 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f067004b5e0 con 0x7f06800b87c0 2023-03-03T08:00:02.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.099+0000 7f0687dff700 1 -- 172.21.15.98:0/782963839 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06800729b0 con 0x7f06800b87c0 2023-03-03T08:00:02.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.101+0000 7f06767fc700 1 -- 172.21.15.98:0/782963839 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+18018 (secure 0 0 0) 0x7f0670017090 con 0x7f06800b87c0 2023-03-03T08:00:02.102 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:02.103 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":54,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:58:28.723281+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:58:10.728830+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-03-03T07:58:12.224649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.6","mappings":[{"from":0,"to":1}]},{"pgid":"8.17","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:00:02.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.104+0000 7f0687dff700 1 -- 172.21.15.98:0/782963839 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f06640683c0 msgr2=0x7f066406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:02.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.104+0000 7f0687dff700 1 --2- 172.21.15.98:0/782963839 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f06640683c0 0x7f066406a870 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f067c0090b0 tx=0x7f067c008030 comp rx=0 tx=0).stop 2023-03-03T08:00:02.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.104+0000 7f0687dff700 1 -- 172.21.15.98:0/782963839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06800b87c0 msgr2=0x7f06800c9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:02.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.104+0000 7f0687dff700 1 --2- 172.21.15.98:0/782963839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06800b87c0 0x7f06800c9680 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f067000c450 tx=0x7f067000c760 comp rx=0 tx=0).stop 2023-03-03T08:00:02.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.104+0000 7f0687dff700 1 -- 172.21.15.98:0/782963839 shutdown_connections 2023-03-03T08:00:02.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.104+0000 7f0687dff700 1 --2- 172.21.15.98:0/782963839 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f06640683c0 0x7f066406a870 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.104+0000 7f0687dff700 1 --2- 172.21.15.98:0/782963839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06800b8b90 0x7f06800cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.104+0000 7f0687dff700 1 --2- 172.21.15.98:0/782963839 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06800b5410 0x7f068014b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.104+0000 7f0687dff700 1 --2- 172.21.15.98:0/782963839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06800b87c0 0x7f06800c9680 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.104+0000 7f0687dff700 1 -- 172.21.15.98:0/782963839 >> 172.21.15.98:0/782963839 conn(0x7f068006aee0 msgr2=0x7f06800b36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:02.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.104+0000 7f0687dff700 1 -- 172.21.15.98:0/782963839 shutdown_connections 2023-03-03T08:00:02.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.104+0000 7f0687dff700 1 -- 172.21.15.98:0/782963839 wait complete. 2023-03-03T08:00:02.132 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T07:58:10.728830+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T07:58:12.224649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:00:02.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:00:02.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.383+0000 7fb1389ff700 1 Processor -- start 2023-03-03T08:00:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.383+0000 7fb1389ff700 1 -- start start 2023-03-03T08:00:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.384+0000 7fb1389ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb13415ca70 0x7fb1340cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.384+0000 7fb1389ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb13406b4b0 0x7fb1340cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.384+0000 7fb1389ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb1340d00a0 0x7fb1340b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.384+0000 7fb1389ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1340cd5b0 con 0x7fb13415ca70 2023-03-03T08:00:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.385+0000 7fb1389ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1340cf980 con 0x7fb1340d00a0 2023-03-03T08:00:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.385+0000 7fb1389ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1340d0510 con 0x7fb13406b4b0 2023-03-03T08:00:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.385+0000 7fb13259c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb13415ca70 0x7fb1340cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.385+0000 7fb13259c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb13415ca70 0x7fb1340cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44616/0 (socket says 172.21.15.98:44616) 2023-03-03T08:00:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.385+0000 7fb13259c700 1 -- 172.21.15.98:0/312843673 learned_addr learned my addr 172.21.15.98:0/312843673 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.385+0000 7fb131d9b700 1 --2- 172.21.15.98:0/312843673 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb13406b4b0 0x7fb1340cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.385+0000 7fb132d9d700 1 --2- 172.21.15.98:0/312843673 >> v2:172.21.15.100:3300/0 conn(0x7fb1340d00a0 0x7fb1340b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.385+0000 7fb13259c700 1 -- 172.21.15.98:0/312843673 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb13406b4b0 msgr2=0x7fb1340cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.385+0000 7fb13259c700 1 --2- 172.21.15.98:0/312843673 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb13406b4b0 0x7fb1340cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.385+0000 7fb13259c700 1 -- 172.21.15.98:0/312843673 >> v2:172.21.15.100:3300/0 conn(0x7fb1340d00a0 msgr2=0x7fb1340b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.385+0000 7fb13259c700 1 --2- 172.21.15.98:0/312843673 >> v2:172.21.15.100:3300/0 conn(0x7fb1340d00a0 0x7fb1340b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.385+0000 7fb13259c700 1 -- 172.21.15.98:0/312843673 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1340d1800 con 0x7fb13415ca70 2023-03-03T08:00:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.385+0000 7fb131d9b700 1 --2- 172.21.15.98:0/312843673 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb13406b4b0 0x7fb1340cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.385+0000 7fb132d9d700 1 --2- 172.21.15.98:0/312843673 >> v2:172.21.15.100:3300/0 conn(0x7fb1340d00a0 0x7fb1340b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.386+0000 7fb13259c700 1 --2- 172.21.15.98:0/312843673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13415ca70 0x7fb1340cf440 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fb13408b220 tx=0x7fb124009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f8a7f4cdd5502c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.386+0000 7fb13159a700 1 -- 172.21.15.98:0/312843673 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12400b860 con 0x7fb13415ca70 2023-03-03T08:00:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.386+0000 7fb13159a700 1 -- 172.21.15.98:0/312843673 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12400b9c0 con 0x7fb13415ca70 2023-03-03T08:00:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.386+0000 7fb13159a700 1 -- 172.21.15.98:0/312843673 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12400bd50 con 0x7fb13415ca70 2023-03-03T08:00:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.386+0000 7fb1389ff700 1 -- 172.21.15.98:0/312843673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13415ca70 msgr2=0x7fb1340cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.386+0000 7fb1389ff700 1 --2- 172.21.15.98:0/312843673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13415ca70 0x7fb1340cf440 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fb13408b220 tx=0x7fb124009c00 comp rx=0 tx=0).stop 2023-03-03T08:00:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.387+0000 7fb1389ff700 1 -- 172.21.15.98:0/312843673 shutdown_connections 2023-03-03T08:00:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.387+0000 7fb1389ff700 1 --2- 172.21.15.98:0/312843673 >> v2:172.21.15.100:3300/0 conn(0x7fb1340d00a0 0x7fb1340b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.387+0000 7fb1389ff700 1 --2- 172.21.15.98:0/312843673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13415ca70 0x7fb1340cf440 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.387+0000 7fb1389ff700 1 --2- 172.21.15.98:0/312843673 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb13406b4b0 0x7fb1340cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.387+0000 7fb1389ff700 1 -- 172.21.15.98:0/312843673 >> 172.21.15.98:0/312843673 conn(0x7fb134073e70 msgr2=0x7fb13406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.387+0000 7fb1389ff700 1 -- 172.21.15.98:0/312843673 shutdown_connections 2023-03-03T08:00:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.387+0000 7fb1389ff700 1 -- 172.21.15.98:0/312843673 wait complete. 2023-03-03T08:00:02.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.389+0000 7fb1389ff700 1 Processor -- start 2023-03-03T08:00:02.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.389+0000 7fb1389ff700 1 -- start start 2023-03-03T08:00:02.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.390+0000 7fb1389ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb13406b4b0 0x7fb1340b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.390+0000 7fb1389ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1340d00a0 0x7fb1341582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.390+0000 7fb1389ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb13415ca70 0x7fb134158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.390+0000 7fb1389ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb134158d50 con 0x7fb1340d00a0 2023-03-03T08:00:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.390+0000 7fb1389ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1340caf40 con 0x7fb13415ca70 2023-03-03T08:00:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.390+0000 7fb1389ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1340cb220 con 0x7fb13406b4b0 2023-03-03T08:00:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.390+0000 7fb131d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1340d00a0 0x7fb1341582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.390+0000 7fb131d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1340d00a0 0x7fb1341582d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44624/0 (socket says 172.21.15.98:44624) 2023-03-03T08:00:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.391+0000 7fb131d9b700 1 -- 172.21.15.98:0/3341089446 learned_addr learned my addr 172.21.15.98:0/3341089446 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.391+0000 7fb13259c700 1 --2- 172.21.15.98:0/3341089446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb13406b4b0 0x7fb1340b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.391+0000 7fb132d9d700 1 --2- 172.21.15.98:0/3341089446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb13415ca70 0x7fb134158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.391+0000 7fb131d9b700 1 -- 172.21.15.98:0/3341089446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb13406b4b0 msgr2=0x7fb1340b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.391+0000 7fb131d9b700 1 --2- 172.21.15.98:0/3341089446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb13406b4b0 0x7fb1340b4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.391+0000 7fb131d9b700 1 -- 172.21.15.98:0/3341089446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb13415ca70 msgr2=0x7fb134158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.391+0000 7fb131d9b700 1 --2- 172.21.15.98:0/3341089446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb13415ca70 0x7fb134158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.391+0000 7fb131d9b700 1 -- 172.21.15.98:0/3341089446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb124009550 con 0x7fb1340d00a0 2023-03-03T08:00:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.391+0000 7fb132d9d700 1 --2- 172.21.15.98:0/3341089446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb13415ca70 0x7fb134158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.391+0000 7fb131d9b700 1 --2- 172.21.15.98:0/3341089446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1340d00a0 0x7fb1341582d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fb12800d800 tx=0x7fb12800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.392+0000 7fb122ffd700 1 -- 172.21.15.98:0/3341089446 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128014620 con 0x7fb1340d00a0 2023-03-03T08:00:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.392+0000 7fb122ffd700 1 -- 172.21.15.98:0/3341089446 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb128014780 con 0x7fb1340d00a0 2023-03-03T08:00:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.392+0000 7fb122ffd700 1 -- 172.21.15.98:0/3341089446 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128014af0 con 0x7fb1340d00a0 2023-03-03T08:00:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.392+0000 7fb1389ff700 1 -- 172.21.15.98:0/3341089446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1340cdbe0 con 0x7fb1340d00a0 2023-03-03T08:00:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.392+0000 7fb1389ff700 1 -- 172.21.15.98:0/3341089446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1340cd900 con 0x7fb1340d00a0 2023-03-03T08:00:02.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.393+0000 7fb1389ff700 1 -- 172.21.15.98:0/3341089446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1340d1aa0 con 0x7fb1340d00a0 2023-03-03T08:00:02.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.394+0000 7fb122ffd700 1 -- 172.21.15.98:0/3341089446 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fb128014c50 con 0x7fb1340d00a0 2023-03-03T08:00:02.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.396+0000 7fb122ffd700 1 --2- 172.21.15.98:0/3341089446 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb1140683c0 0x7fb11406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:02.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.396+0000 7fb122ffd700 1 -- 172.21.15.98:0/3341089446 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fb128086300 con 0x7fb1340d00a0 2023-03-03T08:00:02.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.396+0000 7fb13259c700 1 --2- 172.21.15.98:0/3341089446 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb1140683c0 0x7fb11406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:02.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.397+0000 7fb13259c700 1 --2- 172.21.15.98:0/3341089446 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb1140683c0 0x7fb11406a870 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb124000c00 tx=0x7fb124008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:02.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.400+0000 7fb122ffd700 1 -- 172.21.15.98:0/3341089446 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb128052960 con 0x7fb1340d00a0 2023-03-03T08:00:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.593+0000 7fb1389ff700 1 -- 172.21.15.98:0/3341089446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb13404e6e0 con 0x7fb1340d00a0 2023-03-03T08:00:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.593+0000 7fb122ffd700 1 -- 172.21.15.98:0/3341089446 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb128055f80 con 0x7fb1340d00a0 2023-03-03T08:00:02.595 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:00:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.596+0000 7fb1389ff700 1 -- 172.21.15.98:0/3341089446 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb1140683c0 msgr2=0x7fb11406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.596+0000 7fb1389ff700 1 --2- 172.21.15.98:0/3341089446 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb1140683c0 0x7fb11406a870 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb124000c00 tx=0x7fb124008030 comp rx=0 tx=0).stop 2023-03-03T08:00:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.596+0000 7fb1389ff700 1 -- 172.21.15.98:0/3341089446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1340d00a0 msgr2=0x7fb1341582d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.596+0000 7fb1389ff700 1 --2- 172.21.15.98:0/3341089446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1340d00a0 0x7fb1341582d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fb12800d800 tx=0x7fb12800db10 comp rx=0 tx=0).stop 2023-03-03T08:00:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.596+0000 7fb1389ff700 1 -- 172.21.15.98:0/3341089446 shutdown_connections 2023-03-03T08:00:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.596+0000 7fb1389ff700 1 --2- 172.21.15.98:0/3341089446 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb1140683c0 0x7fb11406a870 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.596+0000 7fb1389ff700 1 --2- 172.21.15.98:0/3341089446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb13415ca70 0x7fb134158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.596+0000 7fb1389ff700 1 --2- 172.21.15.98:0/3341089446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb13406b4b0 0x7fb1340b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.596+0000 7fb1389ff700 1 --2- 172.21.15.98:0/3341089446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1340d00a0 0x7fb1341582d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.596+0000 7fb1389ff700 1 -- 172.21.15.98:0/3341089446 >> 172.21.15.98:0/3341089446 conn(0x7fb134073e70 msgr2=0x7fb1340d2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.596+0000 7fb1389ff700 1 -- 172.21.15.98:0/3341089446 shutdown_connections 2023-03-03T08:00:02.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.596+0000 7fb1389ff700 1 -- 172.21.15.98:0/3341089446 wait complete. 2023-03-03T08:00:02.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:00:02.608 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:00:02.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:00:02.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.911+0000 7f058f59e700 1 Processor -- start 2023-03-03T08:00:02.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.912+0000 7f058f59e700 1 -- start start 2023-03-03T08:00:02.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.913+0000 7f058f59e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f05901609b0 0x7f0590160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:02.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.913+0000 7f058f59e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900cf570 0x7f059015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:02.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.913+0000 7f058f59e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05901612c0 con 0x7f059015db30 2023-03-03T08:00:02.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.913+0000 7f058f59e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05900cebb0 con 0x7f05901609b0 2023-03-03T08:00:02.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.913+0000 7f058f59e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05900cd780 con 0x7f05900cf570 2023-03-03T08:00:02.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.913+0000 7f058ed9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f059015db30 0x7f05900b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34250/0 (socket says 172.21.15.98:34250) 2023-03-03T08:00:02.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.913+0000 7f058ed9d700 1 -- 172.21.15.98:0/1183065257 learned_addr learned my addr 172.21.15.98:0/1183065257 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:02.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.914+0000 7f0587fff700 1 --2- 172.21.15.98:0/1183065257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900cf570 0x7f059015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:02.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.914+0000 7f058e59c700 1 --2- 172.21.15.98:0/1183065257 >> v2:172.21.15.100:3300/0 conn(0x7f05901609b0 0x7f0590160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:02.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.914+0000 7f058dd9b700 1 -- 172.21.15.98:0/1183065257 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717566857 0 0) 0x7f05901612c0 con 0x7f059015db30 2023-03-03T08:00:02.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.914+0000 7f058dd9b700 1 -- 172.21.15.98:0/1183065257 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f057c0035b0 con 0x7f059015db30 2023-03-03T08:00:02.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.914+0000 7f0587fff700 1 -- 172.21.15.98:0/1183065257 >> v2:172.21.15.100:3300/0 conn(0x7f05901609b0 msgr2=0x7f0590160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:02.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.914+0000 7f0587fff700 1 --2- 172.21.15.98:0/1183065257 >> v2:172.21.15.100:3300/0 conn(0x7f05901609b0 0x7f0590160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.914+0000 7f0587fff700 1 -- 172.21.15.98:0/1183065257 >> v1:172.21.15.98:6789/0 conn(0x7f059015db30 legacy=0x7f05900b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:02.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.914+0000 7f0587fff700 1 -- 172.21.15.98:0/1183065257 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05900cda60 con 0x7f05900cf570 2023-03-03T08:00:02.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.915+0000 7f0587fff700 1 --2- 172.21.15.98:0/1183065257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900cf570 0x7f059015d5f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0578009840 tx=0x7f0578009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a94caba9ac041658 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:02.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.915+0000 7f058dd9b700 1 -- 172.21.15.98:0/1183065257 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0578011830 con 0x7f05900cf570 2023-03-03T08:00:02.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.915+0000 7f058dd9b700 1 -- 172.21.15.98:0/1183065257 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0578011990 con 0x7f05900cf570 2023-03-03T08:00:02.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.915+0000 7f058dd9b700 1 -- 172.21.15.98:0/1183065257 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0578011d20 con 0x7f05900cf570 2023-03-03T08:00:02.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.915+0000 7f058f59e700 1 -- 172.21.15.98:0/1183065257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900cf570 msgr2=0x7f059015d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:02.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.915+0000 7f058f59e700 1 --2- 172.21.15.98:0/1183065257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900cf570 0x7f059015d5f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0578009840 tx=0x7f0578009b50 comp rx=0 tx=0).stop 2023-03-03T08:00:02.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.916+0000 7f058f59e700 1 -- 172.21.15.98:0/1183065257 shutdown_connections 2023-03-03T08:00:02.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.916+0000 7f058f59e700 1 --2- 172.21.15.98:0/1183065257 >> v2:172.21.15.100:3300/0 conn(0x7f05901609b0 0x7f0590160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.916+0000 7f058f59e700 1 --2- 172.21.15.98:0/1183065257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05900cf570 0x7f059015d5f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.916+0000 7f058f59e700 1 -- 172.21.15.98:0/1183065257 >> 172.21.15.98:0/1183065257 conn(0x7f059006b880 msgr2=0x7f05900cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:02.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.916+0000 7f058f59e700 1 -- 172.21.15.98:0/1183065257 shutdown_connections 2023-03-03T08:00:02.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.916+0000 7f058f59e700 1 -- 172.21.15.98:0/1183065257 wait complete. 2023-03-03T08:00:02.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.918+0000 7f058f59e700 1 Processor -- start 2023-03-03T08:00:02.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.918+0000 7f058f59e700 1 -- start start 2023-03-03T08:00:02.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.919+0000 7f058f59e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05900cf570 0x7f05900c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:02.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.919+0000 7f058f59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f059015db30 0x7f05900b5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:02.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.919+0000 7f058f59e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05901609b0 0x7f05900b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:02.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.919+0000 7f058f59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05900b8c80 con 0x7f059015db30 2023-03-03T08:00:02.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.919+0000 7f058f59e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05900c8120 con 0x7f05900cf570 2023-03-03T08:00:02.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.919+0000 7f058f59e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05901f2570 con 0x7f05901609b0 2023-03-03T08:00:02.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.919+0000 7f0587fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f059015db30 0x7f05900b5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:02.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.919+0000 7f0587fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f059015db30 0x7f05900b5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44632/0 (socket says 172.21.15.98:44632) 2023-03-03T08:00:02.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.919+0000 7f0587fff700 1 -- 172.21.15.98:0/915727449 learned_addr learned my addr 172.21.15.98:0/915727449 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:02.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.919+0000 7f058ed9d700 1 --2- 172.21.15.98:0/915727449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05901609b0 0x7f05900b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:02.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.919+0000 7f058e59c700 1 --2- 172.21.15.98:0/915727449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05900cf570 0x7f05900c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:02.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.920+0000 7f0587fff700 1 -- 172.21.15.98:0/915727449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05901609b0 msgr2=0x7f05900b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:02.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.920+0000 7f0587fff700 1 --2- 172.21.15.98:0/915727449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05901609b0 0x7f05900b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.920+0000 7f0587fff700 1 -- 172.21.15.98:0/915727449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05900cf570 msgr2=0x7f05900c7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:02.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.920+0000 7f0587fff700 1 --2- 172.21.15.98:0/915727449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05900cf570 0x7f05900c7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:02.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.920+0000 7f0587fff700 1 -- 172.21.15.98:0/915727449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05780094a0 con 0x7f059015db30 2023-03-03T08:00:02.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.920+0000 7f0587fff700 1 --2- 172.21.15.98:0/915727449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f059015db30 0x7f05900b5e50 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0578009790 tx=0x7f0578004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:02.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.920+0000 7f0586ffd700 1 -- 172.21.15.98:0/915727449 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05780036a0 con 0x7f059015db30 2023-03-03T08:00:02.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.920+0000 7f0586ffd700 1 -- 172.21.15.98:0/915727449 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057801f070 con 0x7f059015db30 2023-03-03T08:00:02.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.920+0000 7f0586ffd700 1 -- 172.21.15.98:0/915727449 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0578003a20 con 0x7f059015db30 2023-03-03T08:00:02.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.920+0000 7f058f59e700 1 -- 172.21.15.98:0/915727449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059015cdf0 con 0x7f059015db30 2023-03-03T08:00:02.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.921+0000 7f058f59e700 1 -- 172.21.15.98:0/915727449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05900cdd40 con 0x7f059015db30 2023-03-03T08:00:02.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.922+0000 7f058f59e700 1 -- 172.21.15.98:0/915727449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059015d0d0 con 0x7f059015db30 2023-03-03T08:00:02.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.923+0000 7f0586ffd700 1 -- 172.21.15.98:0/915727449 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f057800f050 con 0x7f059015db30 2023-03-03T08:00:02.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.925+0000 7f0586ffd700 1 --2- 172.21.15.98:0/915727449 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0570068390 0x7f057006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:02.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.925+0000 7f0586ffd700 1 -- 172.21.15.98:0/915727449 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f0578021070 con 0x7f059015db30 2023-03-03T08:00:02.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.925+0000 7f058e59c700 1 --2- 172.21.15.98:0/915727449 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0570068390 0x7f057006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:02.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.925+0000 7f058e59c700 1 --2- 172.21.15.98:0/915727449 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0570068390 0x7f057006a840 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0580009ad0 tx=0x7f0580008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:02.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:02.929+0000 7f0586ffd700 1 -- 172.21.15.98:0/915727449 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0578053dc0 con 0x7f059015db30 2023-03-03T08:00:03.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.133+0000 7f058f59e700 1 -- 172.21.15.98:0/915727449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f05901f5810 con 0x7f059015db30 2023-03-03T08:00:03.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.133+0000 7f0586ffd700 1 -- 172.21.15.98:0/915727449 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v54) v1 ==== 92+0+10 (secure 0 0 0) 0x7f057802b020 con 0x7f059015db30 2023-03-03T08:00:03.135 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:00:03.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.136+0000 7f058f59e700 1 -- 172.21.15.98:0/915727449 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0570068390 msgr2=0x7f057006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:03.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.136+0000 7f058f59e700 1 --2- 172.21.15.98:0/915727449 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0570068390 0x7f057006a840 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0580009ad0 tx=0x7f0580008040 comp rx=0 tx=0).stop 2023-03-03T08:00:03.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.136+0000 7f058f59e700 1 -- 172.21.15.98:0/915727449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f059015db30 msgr2=0x7f05900b5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:03.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.136+0000 7f058f59e700 1 --2- 172.21.15.98:0/915727449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f059015db30 0x7f05900b5e50 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0578009790 tx=0x7f0578004750 comp rx=0 tx=0).stop 2023-03-03T08:00:03.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.136+0000 7f058f59e700 1 -- 172.21.15.98:0/915727449 shutdown_connections 2023-03-03T08:00:03.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.136+0000 7f058f59e700 1 --2- 172.21.15.98:0/915727449 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0570068390 0x7f057006a840 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.136+0000 7f058f59e700 1 --2- 172.21.15.98:0/915727449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05901609b0 0x7f05900b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.136+0000 7f058f59e700 1 --2- 172.21.15.98:0/915727449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05900cf570 0x7f05900c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.136+0000 7f058f59e700 1 --2- 172.21.15.98:0/915727449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f059015db30 0x7f05900b5e50 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.136+0000 7f058f59e700 1 -- 172.21.15.98:0/915727449 >> 172.21.15.98:0/915727449 conn(0x7f059006b880 msgr2=0x7f05900b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:03.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.136+0000 7f058f59e700 1 -- 172.21.15.98:0/915727449 shutdown_connections 2023-03-03T08:00:03.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.136+0000 7f058f59e700 1 -- 172.21.15.98:0/915727449 wait complete. 2023-03-03T08:00:03.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T08:00:03.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.412+0000 7f403df9d700 1 Processor -- start 2023-03-03T08:00:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.412+0000 7f403df9d700 1 -- start start 2023-03-03T08:00:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.414+0000 7f403df9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f403815c910 0x7f403815cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.414+0000 7f403df9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4038073f10 0x7f40380cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.414+0000 7f403df9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40380ce150 con 0x7f403815dde0 2023-03-03T08:00:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.414+0000 7f403df9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40380cde70 con 0x7f403815c910 2023-03-03T08:00:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.414+0000 7f403df9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f403815d300 con 0x7f4038073f10 2023-03-03T08:00:03.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.414+0000 7f403cf9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f403815c910 0x7f403815cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:03.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.414+0000 7f403cf9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f403815c910 0x7f403815cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44650/0 (socket says 172.21.15.98:44650) 2023-03-03T08:00:03.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.414+0000 7f403cf9b700 1 -- 172.21.15.98:0/1967233391 learned_addr learned my addr 172.21.15.98:0/1967233391 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:03.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.414+0000 7f4037fff700 1 --2- 172.21.15.98:0/1967233391 >> v2:172.21.15.100:3300/0 conn(0x7f4038073f10 0x7f40380cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:03.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.415+0000 7f403cf9b700 1 -- 172.21.15.98:0/1967233391 >> v2:172.21.15.100:3300/0 conn(0x7f4038073f10 msgr2=0x7f40380cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:03.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.415+0000 7f403cf9b700 1 --2- 172.21.15.98:0/1967233391 >> v2:172.21.15.100:3300/0 conn(0x7f4038073f10 0x7f40380cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.415+0000 7f403cf9b700 1 -- 172.21.15.98:0/1967233391 >> v1:172.21.15.100:6789/0 conn(0x7f403815dde0 legacy=0x7f40380cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:03.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.415+0000 7f403cf9b700 1 -- 172.21.15.98:0/1967233391 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403815d5e0 con 0x7f403815c910 2023-03-03T08:00:03.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.415+0000 7f403cf9b700 1 --2- 172.21.15.98:0/1967233391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f403815c910 0x7f403815cce0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f40380b5ad0 tx=0x7f402c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a1812af3c8aa24 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.415+0000 7f40377fe700 1 -- 172.21.15.98:0/1967233391 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c011830 con 0x7f403815c910 2023-03-03T08:00:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.415+0000 7f40377fe700 1 -- 172.21.15.98:0/1967233391 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402c011990 con 0x7f403815c910 2023-03-03T08:00:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.415+0000 7f40377fe700 1 -- 172.21.15.98:0/1967233391 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c011d20 con 0x7f403815c910 2023-03-03T08:00:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.416+0000 7f403df9d700 1 -- 172.21.15.98:0/1967233391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f403815c910 msgr2=0x7f403815cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.416+0000 7f403df9d700 1 --2- 172.21.15.98:0/1967233391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f403815c910 0x7f403815cce0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f40380b5ad0 tx=0x7f402c009bd0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.416+0000 7f403df9d700 1 -- 172.21.15.98:0/1967233391 shutdown_connections 2023-03-03T08:00:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.416+0000 7f403df9d700 1 --2- 172.21.15.98:0/1967233391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f403815c910 0x7f403815cce0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.416+0000 7f403df9d700 1 --2- 172.21.15.98:0/1967233391 >> v2:172.21.15.100:3300/0 conn(0x7f4038073f10 0x7f40380cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.416+0000 7f403df9d700 1 -- 172.21.15.98:0/1967233391 >> 172.21.15.98:0/1967233391 conn(0x7f403806ac00 msgr2=0x7f403806afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.416+0000 7f403df9d700 1 -- 172.21.15.98:0/1967233391 shutdown_connections 2023-03-03T08:00:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.416+0000 7f403df9d700 1 -- 172.21.15.98:0/1967233391 wait complete. 2023-03-03T08:00:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.418+0000 7f403df9d700 1 Processor -- start 2023-03-03T08:00:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.418+0000 7f403df9d700 1 -- start start 2023-03-03T08:00:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.419+0000 7f403df9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4038073f10 0x7f403814c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.419+0000 7f403df9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f403815c910 0x7f4038146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.419+0000 7f403df9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f403815dde0 0x7f403815e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.419+0000 7f403df9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40380cc070 con 0x7f403815dde0 2023-03-03T08:00:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.419+0000 7f403df9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40380cb780 con 0x7f403815c910 2023-03-03T08:00:03.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.419+0000 7f403df9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4038149200 con 0x7f4038073f10 2023-03-03T08:00:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.420+0000 7f403d79c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f403815dde0 0x7f403815e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.420+0000 7f403d79c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f403815dde0 0x7f403815e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44668/0 (socket says 172.21.15.98:44668) 2023-03-03T08:00:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.420+0000 7f403d79c700 1 -- 172.21.15.98:0/2874778512 learned_addr learned my addr 172.21.15.98:0/2874778512 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.420+0000 7f403cf9b700 1 --2- 172.21.15.98:0/2874778512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4038073f10 0x7f403814c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.420+0000 7f4037fff700 1 --2- 172.21.15.98:0/2874778512 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f403815c910 0x7f4038146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.420+0000 7f403d79c700 1 -- 172.21.15.98:0/2874778512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4038073f10 msgr2=0x7f403814c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:03.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.421+0000 7f403d79c700 1 --2- 172.21.15.98:0/2874778512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4038073f10 0x7f403814c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.421+0000 7f403d79c700 1 -- 172.21.15.98:0/2874778512 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f403815c910 msgr2=0x7f4038146210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:03.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.421+0000 7f403d79c700 1 --2- 172.21.15.98:0/2874778512 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f403815c910 0x7f4038146210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.421+0000 7f403d79c700 1 -- 172.21.15.98:0/2874778512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402c009550 con 0x7f403815dde0 2023-03-03T08:00:03.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.421+0000 7f403d79c700 1 --2- 172.21.15.98:0/2874778512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f403815dde0 0x7f403815e780 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f403000cb50 tx=0x7f403000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:03.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.421+0000 7f40357fa700 1 -- 172.21.15.98:0/2874778512 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030003c00 con 0x7f403815dde0 2023-03-03T08:00:03.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.421+0000 7f40357fa700 1 -- 172.21.15.98:0/2874778512 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4030003d60 con 0x7f403815dde0 2023-03-03T08:00:03.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.421+0000 7f403df9d700 1 -- 172.21.15.98:0/2874778512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40380cdad0 con 0x7f403815dde0 2023-03-03T08:00:03.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.421+0000 7f40357fa700 1 -- 172.21.15.98:0/2874778512 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403000aa20 con 0x7f403815dde0 2023-03-03T08:00:03.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.421+0000 7f403df9d700 1 -- 172.21.15.98:0/2874778512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403815d8c0 con 0x7f403815dde0 2023-03-03T08:00:03.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.423+0000 7f403df9d700 1 -- 172.21.15.98:0/2874778512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40380d0020 con 0x7f403815dde0 2023-03-03T08:00:03.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.424+0000 7f40357fa700 1 -- 172.21.15.98:0/2874778512 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f4030010050 con 0x7f403815dde0 2023-03-03T08:00:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.425+0000 7f40357fa700 1 --2- 172.21.15.98:0/2874778512 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f401c0683c0 0x7f401c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.425+0000 7f40357fa700 1 -- 172.21.15.98:0/2874778512 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f4030085dc0 con 0x7f403815dde0 2023-03-03T08:00:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.425+0000 7f403cf9b700 1 --2- 172.21.15.98:0/2874778512 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f401c0683c0 0x7f401c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.426+0000 7f403cf9b700 1 --2- 172.21.15.98:0/2874778512 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f401c0683c0 0x7f401c06a870 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f402c009c10 tx=0x7f402c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:03.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.429+0000 7f40357fa700 1 -- 172.21.15.98:0/2874778512 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4030052420 con 0x7f403815dde0 2023-03-03T08:00:03.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.621+0000 7f403df9d700 1 -- 172.21.15.98:0/2874778512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f40380cc1b0 con 0x7f403815dde0 2023-03-03T08:00:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.621+0000 7f40357fa700 1 -- 172.21.15.98:0/2874778512 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7f40380cc1b0 con 0x7f403815dde0 2023-03-03T08:00:03.623 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T08:00:03.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.624+0000 7f403df9d700 1 -- 172.21.15.98:0/2874778512 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f401c0683c0 msgr2=0x7f401c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:03.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.624+0000 7f403df9d700 1 --2- 172.21.15.98:0/2874778512 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f401c0683c0 0x7f401c06a870 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f402c009c10 tx=0x7f402c008030 comp rx=0 tx=0).stop 2023-03-03T08:00:03.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.624+0000 7f403df9d700 1 -- 172.21.15.98:0/2874778512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f403815dde0 msgr2=0x7f403815e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:03.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.624+0000 7f403df9d700 1 --2- 172.21.15.98:0/2874778512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f403815dde0 0x7f403815e780 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f403000cb50 tx=0x7f403000ce60 comp rx=0 tx=0).stop 2023-03-03T08:00:03.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.624+0000 7f403df9d700 1 -- 172.21.15.98:0/2874778512 shutdown_connections 2023-03-03T08:00:03.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.624+0000 7f403df9d700 1 --2- 172.21.15.98:0/2874778512 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f401c0683c0 0x7f401c06a870 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.624+0000 7f403df9d700 1 --2- 172.21.15.98:0/2874778512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f403815dde0 0x7f403815e780 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.624+0000 7f403df9d700 1 --2- 172.21.15.98:0/2874778512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4038073f10 0x7f403814c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.624+0000 7f403df9d700 1 --2- 172.21.15.98:0/2874778512 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f403815c910 0x7f4038146210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.624+0000 7f403df9d700 1 -- 172.21.15.98:0/2874778512 >> 172.21.15.98:0/2874778512 conn(0x7f403806ac00 msgr2=0x7f40380731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:03.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.624+0000 7f403df9d700 1 -- 172.21.15.98:0/2874778512 shutdown_connections 2023-03-03T08:00:03.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.624+0000 7f403df9d700 1 -- 172.21.15.98:0/2874778512 wait complete. 2023-03-03T08:00:03.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T08:00:03.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.900+0000 7fe3257ff700 1 Processor -- start 2023-03-03T08:00:03.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.900+0000 7fe3257ff700 1 -- start start 2023-03-03T08:00:03.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.902+0000 7fe3257ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe32015ca70 0x7fe3200cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:03.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.902+0000 7fe3257ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe32006b4b0 0x7fe3200cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:03.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.902+0000 7fe3257ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3200cf980 con 0x7fe3200d00a0 2023-03-03T08:00:03.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.902+0000 7fe3257ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3200cd5b0 con 0x7fe32015ca70 2023-03-03T08:00:03.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.902+0000 7fe3257ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3200d0540 con 0x7fe32006b4b0 2023-03-03T08:00:03.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.902+0000 7fe31effd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe32015ca70 0x7fe3200cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:03.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.902+0000 7fe31f7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe3200d00a0 0x7fe3200b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34252/0 (socket says 172.21.15.98:34252) 2023-03-03T08:00:03.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.902+0000 7fe31f7fe700 1 -- 172.21.15.98:0/614175563 learned_addr learned my addr 172.21.15.98:0/614175563 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:03.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.902+0000 7fe31e7fc700 1 --2- 172.21.15.98:0/614175563 >> v2:172.21.15.100:3300/0 conn(0x7fe32006b4b0 0x7fe3200cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:03.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.903+0000 7fe31effd700 1 -- 172.21.15.98:0/614175563 >> v2:172.21.15.100:3300/0 conn(0x7fe32006b4b0 msgr2=0x7fe3200cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:03.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.903+0000 7fe31effd700 1 --2- 172.21.15.98:0/614175563 >> v2:172.21.15.100:3300/0 conn(0x7fe32006b4b0 0x7fe3200cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.903+0000 7fe31effd700 1 -- 172.21.15.98:0/614175563 >> v1:172.21.15.98:6789/0 conn(0x7fe3200d00a0 legacy=0x7fe3200b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:03.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.903+0000 7fe31effd700 1 -- 172.21.15.98:0/614175563 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3200d2fd0 con 0x7fe32015ca70 2023-03-03T08:00:03.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.903+0000 7fe31e7fc700 1 --2- 172.21.15.98:0/614175563 >> v2:172.21.15.100:3300/0 conn(0x7fe32006b4b0 0x7fe3200cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:03.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.903+0000 7fe31dffb700 1 -- 172.21.15.98:0/614175563 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699151292 0 0) 0x7fe3200cf980 con 0x7fe3200d00a0 2023-03-03T08:00:03.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.903+0000 7fe31effd700 1 --2- 172.21.15.98:0/614175563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe32015ca70 0x7fe3200cf440 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe320160560 tx=0x7fe310009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9735493179dde374 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:03.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.903+0000 7fe31dffb700 1 -- 172.21.15.98:0/614175563 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe310011830 con 0x7fe32015ca70 2023-03-03T08:00:03.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.903+0000 7fe31dffb700 1 -- 172.21.15.98:0/614175563 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe310011990 con 0x7fe32015ca70 2023-03-03T08:00:03.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.903+0000 7fe31dffb700 1 -- 172.21.15.98:0/614175563 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe310011d20 con 0x7fe32015ca70 2023-03-03T08:00:03.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.904+0000 7fe3257ff700 1 -- 172.21.15.98:0/614175563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe32015ca70 msgr2=0x7fe3200cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:03.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.904+0000 7fe3257ff700 1 --2- 172.21.15.98:0/614175563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe32015ca70 0x7fe3200cf440 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe320160560 tx=0x7fe310009b20 comp rx=0 tx=0).stop 2023-03-03T08:00:03.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.904+0000 7fe3257ff700 1 -- 172.21.15.98:0/614175563 shutdown_connections 2023-03-03T08:00:03.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.904+0000 7fe3257ff700 1 --2- 172.21.15.98:0/614175563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe32015ca70 0x7fe3200cf440 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.904+0000 7fe3257ff700 1 --2- 172.21.15.98:0/614175563 >> v2:172.21.15.100:3300/0 conn(0x7fe32006b4b0 0x7fe3200cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.904+0000 7fe3257ff700 1 -- 172.21.15.98:0/614175563 >> 172.21.15.98:0/614175563 conn(0x7fe320073e70 msgr2=0x7fe32006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:03.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.904+0000 7fe3257ff700 1 -- 172.21.15.98:0/614175563 shutdown_connections 2023-03-03T08:00:03.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.904+0000 7fe3257ff700 1 -- 172.21.15.98:0/614175563 wait complete. 2023-03-03T08:00:03.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.906+0000 7fe3257ff700 1 Processor -- start 2023-03-03T08:00:03.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.906+0000 7fe3257ff700 1 -- start start 2023-03-03T08:00:03.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.907+0000 7fe3257ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe32006b4b0 0x7fe3200cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:03.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.907+0000 7fe3257ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3200d00a0 0x7fe32015ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:03.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.907+0000 7fe3257ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe32015ca70 0x7fe3201e6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:03.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.907+0000 7fe3257ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3200ccf80 con 0x7fe3200d00a0 2023-03-03T08:00:03.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.907+0000 7fe3257ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32015d9d0 con 0x7fe32006b4b0 2023-03-03T08:00:03.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.907+0000 7fe3257ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32015dcb0 con 0x7fe32015ca70 2023-03-03T08:00:03.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.908+0000 7fe31e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3200d00a0 0x7fe32015ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:03.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.908+0000 7fe31e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3200d00a0 0x7fe32015ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44678/0 (socket says 172.21.15.98:44678) 2023-03-03T08:00:03.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.908+0000 7fe31e7fc700 1 -- 172.21.15.98:0/3503187918 learned_addr learned my addr 172.21.15.98:0/3503187918 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:03.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.908+0000 7fe31effd700 1 --2- 172.21.15.98:0/3503187918 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe32006b4b0 0x7fe3200cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.908+0000 7fe31f7fe700 1 --2- 172.21.15.98:0/3503187918 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe32015ca70 0x7fe3201e6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.908+0000 7fe31e7fc700 1 -- 172.21.15.98:0/3503187918 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe32015ca70 msgr2=0x7fe3201e6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.908+0000 7fe31e7fc700 1 --2- 172.21.15.98:0/3503187918 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe32015ca70 0x7fe3201e6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.908+0000 7fe31e7fc700 1 -- 172.21.15.98:0/3503187918 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe32006b4b0 msgr2=0x7fe3200cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.908+0000 7fe31e7fc700 1 --2- 172.21.15.98:0/3503187918 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe32006b4b0 0x7fe3200cca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.908+0000 7fe31e7fc700 1 -- 172.21.15.98:0/3503187918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3100094a0 con 0x7fe3200d00a0 2023-03-03T08:00:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.909+0000 7fe31e7fc700 1 --2- 172.21.15.98:0/3503187918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3200d00a0 0x7fe32015ea70 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fe314009fd0 tx=0x7fe31400eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.909+0000 7fe3077fe700 1 -- 172.21.15.98:0/3503187918 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314013610 con 0x7fe3200d00a0 2023-03-03T08:00:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.909+0000 7fe3077fe700 1 -- 172.21.15.98:0/3503187918 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe314013770 con 0x7fe3200d00a0 2023-03-03T08:00:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.909+0000 7fe3077fe700 1 -- 172.21.15.98:0/3503187918 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314013b20 con 0x7fe3200d00a0 2023-03-03T08:00:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.909+0000 7fe3257ff700 1 -- 172.21.15.98:0/3503187918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3200cd900 con 0x7fe3200d00a0 2023-03-03T08:00:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.909+0000 7fe3257ff700 1 -- 172.21.15.98:0/3503187918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3200d1800 con 0x7fe3200d00a0 2023-03-03T08:00:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.910+0000 7fe3257ff700 1 -- 172.21.15.98:0/3503187918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3200cdbe0 con 0x7fe3200d00a0 2023-03-03T08:00:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.912+0000 7fe3077fe700 1 -- 172.21.15.98:0/3503187918 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fe314023070 con 0x7fe3200d00a0 2023-03-03T08:00:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.913+0000 7fe3077fe700 1 --2- 172.21.15.98:0/3503187918 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe30c0683c0 0x7fe30c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.913+0000 7fe3077fe700 1 -- 172.21.15.98:0/3503187918 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fe314085ba0 con 0x7fe3200d00a0 2023-03-03T08:00:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.913+0000 7fe31effd700 1 --2- 172.21.15.98:0/3503187918 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe30c0683c0 0x7fe30c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.914+0000 7fe31effd700 1 --2- 172.21.15.98:0/3503187918 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe30c0683c0 0x7fe30c06a870 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe3100090b0 tx=0x7fe31001c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:03.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:03.916+0000 7fe3077fe700 1 -- 172.21.15.98:0/3503187918 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe314052180 con 0x7fe3200d00a0 2023-03-03T08:00:04.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.108+0000 7fe3257ff700 1 -- 172.21.15.98:0/3503187918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe3200b8620 con 0x7fe3200d00a0 2023-03-03T08:00:04.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.108+0000 7fe3077fe700 1 -- 172.21.15.98:0/3503187918 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe31400f040 con 0x7fe3200d00a0 2023-03-03T08:00:04.110 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T08:00:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.111+0000 7fe3257ff700 1 -- 172.21.15.98:0/3503187918 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe30c0683c0 msgr2=0x7fe30c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.111+0000 7fe3257ff700 1 --2- 172.21.15.98:0/3503187918 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe30c0683c0 0x7fe30c06a870 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe3100090b0 tx=0x7fe31001c040 comp rx=0 tx=0).stop 2023-03-03T08:00:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.111+0000 7fe3257ff700 1 -- 172.21.15.98:0/3503187918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3200d00a0 msgr2=0x7fe32015ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.111+0000 7fe3257ff700 1 --2- 172.21.15.98:0/3503187918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3200d00a0 0x7fe32015ea70 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fe314009fd0 tx=0x7fe31400eac0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.111+0000 7fe3257ff700 1 -- 172.21.15.98:0/3503187918 shutdown_connections 2023-03-03T08:00:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.111+0000 7fe3257ff700 1 --2- 172.21.15.98:0/3503187918 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe30c0683c0 0x7fe30c06a870 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.111+0000 7fe3257ff700 1 --2- 172.21.15.98:0/3503187918 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe32015ca70 0x7fe3201e6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.111+0000 7fe3257ff700 1 --2- 172.21.15.98:0/3503187918 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe32006b4b0 0x7fe3200cca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.111+0000 7fe3257ff700 1 --2- 172.21.15.98:0/3503187918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3200d00a0 0x7fe32015ea70 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.111+0000 7fe3257ff700 1 -- 172.21.15.98:0/3503187918 >> 172.21.15.98:0/3503187918 conn(0x7fe320073e70 msgr2=0x7fe32015ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:04.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.111+0000 7fe3257ff700 1 -- 172.21.15.98:0/3503187918 shutdown_connections 2023-03-03T08:00:04.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.111+0000 7fe3257ff700 1 -- 172.21.15.98:0/3503187918 wait complete. 2023-03-03T08:00:04.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:00:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.385+0000 7f0f6bfff700 1 Processor -- start 2023-03-03T08:00:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.386+0000 7f0f6bfff700 1 -- start start 2023-03-03T08:00:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.387+0000 7f0f6bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0f6c146360 0x7f0f6c160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.387+0000 7f0f6bfff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0f6c0b4ab0 0x7f0f6c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.387+0000 7f0f6bfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f6c0731e0 0x7f0f6c073640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.387+0000 7f0f6bfff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f6c1614c0 con 0x7f0f6c146360 2023-03-03T08:00:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.387+0000 7f0f6bfff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f6c0b5440 con 0x7f0f6c0b4ab0 2023-03-03T08:00:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.387+0000 7f0f6bfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c0d10f0 con 0x7f0f6c0731e0 2023-03-03T08:00:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.387+0000 7f0f6affd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0f6c146360 0x7f0f6c160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.388+0000 7f0f6affd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0f6c146360 0x7f0f6c160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44680/0 (socket says 172.21.15.98:44680) 2023-03-03T08:00:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.388+0000 7f0f6affd700 1 -- 172.21.15.98:0/3952819400 learned_addr learned my addr 172.21.15.98:0/3952819400 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.388+0000 7f0f63fff700 1 --2- 172.21.15.98:0/3952819400 >> v2:172.21.15.100:3300/0 conn(0x7f0f6c0b4ab0 0x7f0f6c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.388+0000 7f0f6b7fe700 1 --2- 172.21.15.98:0/3952819400 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f6c0731e0 0x7f0f6c073640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.388+0000 7f0f6affd700 1 -- 172.21.15.98:0/3952819400 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f6c0731e0 msgr2=0x7f0f6c073640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.388+0000 7f0f6affd700 1 --2- 172.21.15.98:0/3952819400 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f6c0731e0 0x7f0f6c073640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.388+0000 7f0f6affd700 1 -- 172.21.15.98:0/3952819400 >> v2:172.21.15.100:3300/0 conn(0x7f0f6c0b4ab0 msgr2=0x7f0f6c0b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.388+0000 7f0f6affd700 1 --2- 172.21.15.98:0/3952819400 >> v2:172.21.15.100:3300/0 conn(0x7f0f6c0b4ab0 0x7f0f6c0b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.388+0000 7f0f6affd700 1 -- 172.21.15.98:0/3952819400 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6c0cdf20 con 0x7f0f6c146360 2023-03-03T08:00:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.388+0000 7f0f6affd700 1 --2- 172.21.15.98:0/3952819400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c146360 0x7f0f6c160f80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c06bd20 tx=0x7f0f5c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a3415d61a5cf7a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.389+0000 7f0f6a7fc700 1 -- 172.21.15.98:0/3952819400 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c011830 con 0x7f0f6c146360 2023-03-03T08:00:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.389+0000 7f0f6a7fc700 1 -- 172.21.15.98:0/3952819400 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f5c011990 con 0x7f0f6c146360 2023-03-03T08:00:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.389+0000 7f0f6a7fc700 1 -- 172.21.15.98:0/3952819400 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c011d20 con 0x7f0f6c146360 2023-03-03T08:00:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.389+0000 7f0f6bfff700 1 -- 172.21.15.98:0/3952819400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c146360 msgr2=0x7f0f6c160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.389+0000 7f0f6bfff700 1 --2- 172.21.15.98:0/3952819400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c146360 0x7f0f6c160f80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c06bd20 tx=0x7f0f5c009bd0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.389+0000 7f0f6bfff700 1 -- 172.21.15.98:0/3952819400 shutdown_connections 2023-03-03T08:00:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.389+0000 7f0f6bfff700 1 --2- 172.21.15.98:0/3952819400 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f6c0731e0 0x7f0f6c073640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.389+0000 7f0f6bfff700 1 --2- 172.21.15.98:0/3952819400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c146360 0x7f0f6c160f80 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.389+0000 7f0f6bfff700 1 --2- 172.21.15.98:0/3952819400 >> v2:172.21.15.100:3300/0 conn(0x7f0f6c0b4ab0 0x7f0f6c0b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.389+0000 7f0f6bfff700 1 -- 172.21.15.98:0/3952819400 >> 172.21.15.98:0/3952819400 conn(0x7f0f6c06b8e0 msgr2=0x7f0f6c0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.390+0000 7f0f6bfff700 1 -- 172.21.15.98:0/3952819400 shutdown_connections 2023-03-03T08:00:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.390+0000 7f0f6bfff700 1 -- 172.21.15.98:0/3952819400 wait complete. 2023-03-03T08:00:04.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.392+0000 7f0f6bfff700 1 Processor -- start 2023-03-03T08:00:04.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.392+0000 7f0f6bfff700 1 -- start start 2023-03-03T08:00:04.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.393+0000 7f0f6bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c0731e0 0x7f0f6c0c57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.393+0000 7f0f6bfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f6c0b4ab0 0x7f0f6c155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.393+0000 7f0f6bfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f6c146360 0x7f0f6c0b7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.393+0000 7f0f6bfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c0c5cf0 con 0x7f0f6c0731e0 2023-03-03T08:00:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.393+0000 7f0f6bfff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c0c7bb0 con 0x7f0f6c0b4ab0 2023-03-03T08:00:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.393+0000 7f0f6bfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c0c7e00 con 0x7f0f6c146360 2023-03-03T08:00:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.393+0000 7f0f6affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c0731e0 0x7f0f6c0c57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.393+0000 7f0f6affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c0731e0 0x7f0f6c0c57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44686/0 (socket says 172.21.15.98:44686) 2023-03-03T08:00:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.393+0000 7f0f6affd700 1 -- 172.21.15.98:0/276867733 learned_addr learned my addr 172.21.15.98:0/276867733 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.394+0000 7f0f6b7fe700 1 --2- 172.21.15.98:0/276867733 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f6c146360 0x7f0f6c0b7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.394+0000 7f0f63fff700 1 --2- 172.21.15.98:0/276867733 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f6c0b4ab0 0x7f0f6c155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.394+0000 7f0f6affd700 1 -- 172.21.15.98:0/276867733 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f6c146360 msgr2=0x7f0f6c0b7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.394+0000 7f0f6affd700 1 --2- 172.21.15.98:0/276867733 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f6c146360 0x7f0f6c0b7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.394+0000 7f0f6affd700 1 -- 172.21.15.98:0/276867733 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f6c0b4ab0 msgr2=0x7f0f6c155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.394+0000 7f0f6affd700 1 --2- 172.21.15.98:0/276867733 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f6c0b4ab0 0x7f0f6c155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.394+0000 7f0f6affd700 1 -- 172.21.15.98:0/276867733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c009550 con 0x7f0f6c0731e0 2023-03-03T08:00:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.394+0000 7f0f6affd700 1 --2- 172.21.15.98:0/276867733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c0731e0 0x7f0f6c0c57b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c009c10 tx=0x7f0f5c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.394+0000 7f0f637fe700 1 -- 172.21.15.98:0/276867733 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c004510 con 0x7f0f6c0731e0 2023-03-03T08:00:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.394+0000 7f0f637fe700 1 -- 172.21.15.98:0/276867733 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f5c01e070 con 0x7f0f6c0731e0 2023-03-03T08:00:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.394+0000 7f0f6bfff700 1 -- 172.21.15.98:0/276867733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f6c0cd960 con 0x7f0f6c0731e0 2023-03-03T08:00:04.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.394+0000 7f0f637fe700 1 -- 172.21.15.98:0/276867733 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c0038e0 con 0x7f0f6c0731e0 2023-03-03T08:00:04.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.395+0000 7f0f6bfff700 1 -- 172.21.15.98:0/276867733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f6c0cd680 con 0x7f0f6c0731e0 2023-03-03T08:00:04.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.396+0000 7f0f6bfff700 1 -- 172.21.15.98:0/276867733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f6c0cdc40 con 0x7f0f6c0731e0 2023-03-03T08:00:04.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.397+0000 7f0f637fe700 1 -- 172.21.15.98:0/276867733 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f0f5c00f050 con 0x7f0f6c0731e0 2023-03-03T08:00:04.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.398+0000 7f0f637fe700 1 --2- 172.21.15.98:0/276867733 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0f4c0683c0 0x7f0f4c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:04.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.398+0000 7f0f637fe700 1 -- 172.21.15.98:0/276867733 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f0f5c086d70 con 0x7f0f6c0731e0 2023-03-03T08:00:04.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.398+0000 7f0f63fff700 1 --2- 172.21.15.98:0/276867733 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0f4c0683c0 0x7f0f4c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:04.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.399+0000 7f0f63fff700 1 --2- 172.21.15.98:0/276867733 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0f4c0683c0 0x7f0f4c06a870 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0d25c0 tx=0x7f0f54008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:04.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.402+0000 7f0f637fe700 1 -- 172.21.15.98:0/276867733 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0f5c0533d0 con 0x7f0f6c0731e0 2023-03-03T08:00:04.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.640+0000 7f0f6bfff700 1 -- 172.21.15.98:0/276867733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f6c0d3e30 con 0x7f0f6c0731e0 2023-03-03T08:00:04.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.641+0000 7f0f637fe700 1 -- 172.21.15.98:0/276867733 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f0f5c015090 con 0x7f0f6c0731e0 2023-03-03T08:00:04.643 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:04.643 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T08:00:04.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.643+0000 7f0f6bfff700 1 -- 172.21.15.98:0/276867733 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0f4c0683c0 msgr2=0x7f0f4c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:04.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.643+0000 7f0f6bfff700 1 --2- 172.21.15.98:0/276867733 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0f4c0683c0 0x7f0f4c06a870 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0d25c0 tx=0x7f0f54008040 comp rx=0 tx=0).stop 2023-03-03T08:00:04.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.643+0000 7f0f6bfff700 1 -- 172.21.15.98:0/276867733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c0731e0 msgr2=0x7f0f6c0c57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:04.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.643+0000 7f0f6bfff700 1 --2- 172.21.15.98:0/276867733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c0731e0 0x7f0f6c0c57b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c009c10 tx=0x7f0f5c004750 comp rx=0 tx=0).stop 2023-03-03T08:00:04.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.644+0000 7f0f6bfff700 1 -- 172.21.15.98:0/276867733 shutdown_connections 2023-03-03T08:00:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.644+0000 7f0f6bfff700 1 --2- 172.21.15.98:0/276867733 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0f4c0683c0 0x7f0f4c06a870 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.644+0000 7f0f6bfff700 1 --2- 172.21.15.98:0/276867733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f6c0731e0 0x7f0f6c0c57b0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.644+0000 7f0f6bfff700 1 --2- 172.21.15.98:0/276867733 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f6c146360 0x7f0f6c0b7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.644+0000 7f0f6bfff700 1 --2- 172.21.15.98:0/276867733 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f6c0b4ab0 0x7f0f6c155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.644+0000 7f0f6bfff700 1 -- 172.21.15.98:0/276867733 >> 172.21.15.98:0/276867733 conn(0x7f0f6c06b8e0 msgr2=0x7f0f6c0d1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.644+0000 7f0f6bfff700 1 -- 172.21.15.98:0/276867733 shutdown_connections 2023-03-03T08:00:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.644+0000 7f0f6bfff700 1 -- 172.21.15.98:0/276867733 wait complete. 2023-03-03T08:00:04.647 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T08:00:04.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:00:04.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.918+0000 7f78b21ff700 1 Processor -- start 2023-03-03T08:00:04.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.918+0000 7f78b21ff700 1 -- start start 2023-03-03T08:00:04.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.919+0000 7f78b21ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78ac0b87c0 0x7f78ac0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:04.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.919+0000 7f78b21ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f78ac0b8b90 0x7f78ac0b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:04.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.919+0000 7f78b21ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f78ac0b5240 0x7f78ac0b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:04.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.919+0000 7f78b21ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78ac0d0ef0 con 0x7f78ac0b8b90 2023-03-03T08:00:04.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.919+0000 7f78b21ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78ac0d0600 con 0x7f78ac0b5240 2023-03-03T08:00:04.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.919+0000 7f78b21ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac0cd970 con 0x7f78ac0b87c0 2023-03-03T08:00:04.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.920+0000 7f78aaffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f78ac0b8b90 0x7f78ac0b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:04.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.920+0000 7f78aaffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f78ac0b8b90 0x7f78ac0b4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44694/0 (socket says 172.21.15.98:44694) 2023-03-03T08:00:04.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.920+0000 7f78aaffd700 1 -- 172.21.15.98:0/4153071503 learned_addr learned my addr 172.21.15.98:0/4153071503 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:04.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.920+0000 7f78abfff700 1 --2- 172.21.15.98:0/4153071503 >> v2:172.21.15.100:3300/0 conn(0x7f78ac0b5240 0x7f78ac0b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:04.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.920+0000 7f78ab7fe700 1 --2- 172.21.15.98:0/4153071503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78ac0b87c0 0x7f78ac0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:04.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.920+0000 7f78aaffd700 1 -- 172.21.15.98:0/4153071503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78ac0b87c0 msgr2=0x7f78ac0cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:04.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.920+0000 7f78aaffd700 1 --2- 172.21.15.98:0/4153071503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78ac0b87c0 0x7f78ac0cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.920+0000 7f78aaffd700 1 -- 172.21.15.98:0/4153071503 >> v2:172.21.15.100:3300/0 conn(0x7f78ac0b5240 msgr2=0x7f78ac0b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:04.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.920+0000 7f78aaffd700 1 --2- 172.21.15.98:0/4153071503 >> v2:172.21.15.100:3300/0 conn(0x7f78ac0b5240 0x7f78ac0b56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.920+0000 7f78aaffd700 1 -- 172.21.15.98:0/4153071503 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac0cdc50 con 0x7f78ac0b8b90 2023-03-03T08:00:04.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.921+0000 7f78aaffd700 1 --2- 172.21.15.98:0/4153071503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac0b8b90 0x7f78ac0b4d00 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0d31a0 tx=0x7f78a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f275853639306b7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:04.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.921+0000 7f78aa7fc700 1 -- 172.21.15.98:0/4153071503 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a000b870 con 0x7f78ac0b8b90 2023-03-03T08:00:04.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.921+0000 7f78aa7fc700 1 -- 172.21.15.98:0/4153071503 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a000b9d0 con 0x7f78ac0b8b90 2023-03-03T08:00:04.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.921+0000 7f78aa7fc700 1 -- 172.21.15.98:0/4153071503 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a000bd60 con 0x7f78ac0b8b90 2023-03-03T08:00:04.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.921+0000 7f78b21ff700 1 -- 172.21.15.98:0/4153071503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac0b8b90 msgr2=0x7f78ac0b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:04.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.921+0000 7f78b21ff700 1 --2- 172.21.15.98:0/4153071503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac0b8b90 0x7f78ac0b4d00 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0d31a0 tx=0x7f78a0009bd0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.922+0000 7f78b21ff700 1 -- 172.21.15.98:0/4153071503 shutdown_connections 2023-03-03T08:00:04.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.922+0000 7f78b21ff700 1 --2- 172.21.15.98:0/4153071503 >> v2:172.21.15.100:3300/0 conn(0x7f78ac0b5240 0x7f78ac0b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.922+0000 7f78b21ff700 1 --2- 172.21.15.98:0/4153071503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78ac0b87c0 0x7f78ac0cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.922+0000 7f78b21ff700 1 --2- 172.21.15.98:0/4153071503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac0b8b90 0x7f78ac0b4d00 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.922+0000 7f78b21ff700 1 -- 172.21.15.98:0/4153071503 >> 172.21.15.98:0/4153071503 conn(0x7f78ac06aee0 msgr2=0x7f78ac06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:04.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.922+0000 7f78b21ff700 1 -- 172.21.15.98:0/4153071503 shutdown_connections 2023-03-03T08:00:04.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.922+0000 7f78b21ff700 1 -- 172.21.15.98:0/4153071503 wait complete. 2023-03-03T08:00:04.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.924+0000 7f78b21ff700 1 Processor -- start 2023-03-03T08:00:04.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.924+0000 7f78b21ff700 1 -- start start 2023-03-03T08:00:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.925+0000 7f78b21ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac0b5240 0x7f78ac1e5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.925+0000 7f78b21ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78ac0b87c0 0x7f78ac15c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.925+0000 7f78b21ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78ac0b8b90 0x7f78ac15b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.925+0000 7f78b21ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac1e5a10 con 0x7f78ac0b5240 2023-03-03T08:00:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.925+0000 7f78b21ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac0c7860 con 0x7f78ac0b8b90 2023-03-03T08:00:04.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.925+0000 7f78b21ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac0c7b40 con 0x7f78ac0b87c0 2023-03-03T08:00:04.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.925+0000 7f78ab7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac0b5240 0x7f78ac1e5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:04.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.926+0000 7f78ab7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac0b5240 0x7f78ac1e5000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44704/0 (socket says 172.21.15.98:44704) 2023-03-03T08:00:04.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.926+0000 7f78ab7fe700 1 -- 172.21.15.98:0/120392310 learned_addr learned my addr 172.21.15.98:0/120392310 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:04.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.926+0000 7f78aaffd700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78ac0b87c0 0x7f78ac15c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:04.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.926+0000 7f78abfff700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78ac0b8b90 0x7f78ac15b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:04.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.926+0000 7f78ab7fe700 1 -- 172.21.15.98:0/120392310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78ac0b87c0 msgr2=0x7f78ac15c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:04.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.926+0000 7f78ab7fe700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78ac0b87c0 0x7f78ac15c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.926+0000 7f78ab7fe700 1 -- 172.21.15.98:0/120392310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78ac0b8b90 msgr2=0x7f78ac15b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:04.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.926+0000 7f78ab7fe700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78ac0b8b90 0x7f78ac15b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:04.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.926+0000 7f78ab7fe700 1 -- 172.21.15.98:0/120392310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a0009550 con 0x7f78ac0b5240 2023-03-03T08:00:04.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.926+0000 7f78abfff700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78ac0b8b90 0x7f78ac15b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:04.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.926+0000 7f78aaffd700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78ac0b87c0 0x7f78ac15c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:04.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.926+0000 7f78ab7fe700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac0b5240 0x7f78ac1e5000 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f789c00e810 tx=0x7f789c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:04.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.926+0000 7f7893fff700 1 -- 172.21.15.98:0/120392310 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789c013610 con 0x7f78ac0b5240 2023-03-03T08:00:04.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.927+0000 7f7893fff700 1 -- 172.21.15.98:0/120392310 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789c013770 con 0x7f78ac0b5240 2023-03-03T08:00:04.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.927+0000 7f7893fff700 1 -- 172.21.15.98:0/120392310 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789c013b20 con 0x7f78ac0b5240 2023-03-03T08:00:04.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.927+0000 7f78b21ff700 1 -- 172.21.15.98:0/120392310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ac0cdf30 con 0x7f78ac0b5240 2023-03-03T08:00:04.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.927+0000 7f78b21ff700 1 -- 172.21.15.98:0/120392310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78ac073a80 con 0x7f78ac0b5240 2023-03-03T08:00:04.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.928+0000 7f78b21ff700 1 -- 172.21.15.98:0/120392310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78ac0ce0c0 con 0x7f78ac0b5240 2023-03-03T08:00:04.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.929+0000 7f7893fff700 1 -- 172.21.15.98:0/120392310 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f789c023070 con 0x7f78ac0b5240 2023-03-03T08:00:04.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.930+0000 7f7893fff700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f78980683c0 0x7f789806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:04.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.930+0000 7f7893fff700 1 -- 172.21.15.98:0/120392310 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f789c085c20 con 0x7f78ac0b5240 2023-03-03T08:00:04.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.931+0000 7f78aaffd700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f78980683c0 0x7f789806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:04.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.931+0000 7f78aaffd700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f78980683c0 0x7f789806a870 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0b7f80 tx=0x7f78a0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:04.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:04.935+0000 7f7893fff700 1 -- 172.21.15.98:0/120392310 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f789c052200 con 0x7f78ac0b5240 2023-03-03T08:00:05.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.130+0000 7f78b21ff700 1 -- 172.21.15.98:0/120392310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78ac0729b0 con 0x7f78ac0b5240 2023-03-03T08:00:05.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.131+0000 7f7893fff700 1 -- 172.21.15.98:0/120392310 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+18018 (secure 0 0 0) 0x7f789c055820 con 0x7f78ac0b5240 2023-03-03T08:00:05.136 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:05.136 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":54,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:58:28.723281+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:58:10.728830+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-03-03T07:58:12.224649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.6","mappings":[{"from":0,"to":1}]},{"pgid":"8.17","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:00:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.134+0000 7f78b21ff700 1 -- 172.21.15.98:0/120392310 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f78980683c0 msgr2=0x7f789806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.134+0000 7f78b21ff700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f78980683c0 0x7f789806a870 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0b7f80 tx=0x7f78a0008030 comp rx=0 tx=0).stop 2023-03-03T08:00:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.134+0000 7f78b21ff700 1 -- 172.21.15.98:0/120392310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac0b5240 msgr2=0x7f78ac1e5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.134+0000 7f78b21ff700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac0b5240 0x7f78ac1e5000 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f789c00e810 tx=0x7f789c00ebd0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.134+0000 7f78b21ff700 1 -- 172.21.15.98:0/120392310 shutdown_connections 2023-03-03T08:00:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.134+0000 7f78b21ff700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f78980683c0 0x7f789806a870 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.134+0000 7f78b21ff700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac0b5240 0x7f78ac1e5000 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.134+0000 7f78b21ff700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78ac0b8b90 0x7f78ac15b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.134+0000 7f78b21ff700 1 --2- 172.21.15.98:0/120392310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78ac0b87c0 0x7f78ac15c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.134+0000 7f78b21ff700 1 -- 172.21.15.98:0/120392310 >> 172.21.15.98:0/120392310 conn(0x7f78ac06aee0 msgr2=0x7f78ac15f940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:05.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.134+0000 7f78b21ff700 1 -- 172.21.15.98:0/120392310 shutdown_connections 2023-03-03T08:00:05.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.134+0000 7f78b21ff700 1 -- 172.21.15.98:0/120392310 wait complete. 2023-03-03T08:00:05.163 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-03-03T08:00:05.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:00:05.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.410+0000 7fd42ffff700 1 Processor -- start 2023-03-03T08:00:05.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.411+0000 7fd42ffff700 1 -- start start 2023-03-03T08:00:05.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.412+0000 7fd42ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd430146360 0x7fd430160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:05.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.412+0000 7fd42ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4300b4ab0 0x7fd4300b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:05.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.412+0000 7fd42ffff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4301614c0 con 0x7fd4300731e0 2023-03-03T08:00:05.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.412+0000 7fd42ffff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4300b5440 con 0x7fd430146360 2023-03-03T08:00:05.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.412+0000 7fd42ffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4300d10f0 con 0x7fd4300b4ab0 2023-03-03T08:00:05.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.412+0000 7fd42effd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd430146360 0x7fd430160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:05.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.412+0000 7fd42effd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd430146360 0x7fd430160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50976/0 (socket says 172.21.15.98:50976) 2023-03-03T08:00:05.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.412+0000 7fd42f7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd4300731e0 0x7fd430073660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46518/0 (socket says 172.21.15.98:46518) 2023-03-03T08:00:05.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.412+0000 7fd42effd700 1 -- 172.21.15.98:0/3638154588 learned_addr learned my addr 172.21.15.98:0/3638154588 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:05.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.413+0000 7fd42e7fc700 1 --2- 172.21.15.98:0/3638154588 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4300b4ab0 0x7fd4300b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:05.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.413+0000 7fd42effd700 1 -- 172.21.15.98:0/3638154588 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4300b4ab0 msgr2=0x7fd4300b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:05.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.413+0000 7fd42effd700 1 --2- 172.21.15.98:0/3638154588 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4300b4ab0 0x7fd4300b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.413+0000 7fd42effd700 1 -- 172.21.15.98:0/3638154588 >> v1:172.21.15.98:6789/0 conn(0x7fd4300731e0 legacy=0x7fd430073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:05.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.413+0000 7fd42effd700 1 -- 172.21.15.98:0/3638154588 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4300cdf20 con 0x7fd430146360 2023-03-03T08:00:05.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.413+0000 7fd42effd700 1 --2- 172.21.15.98:0/3638154588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd430146360 0x7fd430160f80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd4200098f0 tx=0x7fd420009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9c12dfd6c9dba4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:05.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.413+0000 7fd42dffb700 1 -- 172.21.15.98:0/3638154588 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42000b8a0 con 0x7fd430146360 2023-03-03T08:00:05.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.414+0000 7fd42dffb700 1 -- 172.21.15.98:0/3638154588 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42000ba00 con 0x7fd430146360 2023-03-03T08:00:05.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.414+0000 7fd42dffb700 1 -- 172.21.15.98:0/3638154588 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42000bd90 con 0x7fd430146360 2023-03-03T08:00:05.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.414+0000 7fd42ffff700 1 -- 172.21.15.98:0/3638154588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd430146360 msgr2=0x7fd430160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:05.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.414+0000 7fd42ffff700 1 --2- 172.21.15.98:0/3638154588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd430146360 0x7fd430160f80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd4200098f0 tx=0x7fd420009c00 comp rx=0 tx=0).stop 2023-03-03T08:00:05.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.414+0000 7fd42ffff700 1 -- 172.21.15.98:0/3638154588 shutdown_connections 2023-03-03T08:00:05.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.414+0000 7fd42ffff700 1 --2- 172.21.15.98:0/3638154588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd430146360 0x7fd430160f80 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.414+0000 7fd42ffff700 1 --2- 172.21.15.98:0/3638154588 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4300b4ab0 0x7fd4300b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.414+0000 7fd42ffff700 1 -- 172.21.15.98:0/3638154588 >> 172.21.15.98:0/3638154588 conn(0x7fd43006b8e0 msgr2=0x7fd4300cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:05.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.414+0000 7fd42ffff700 1 -- 172.21.15.98:0/3638154588 shutdown_connections 2023-03-03T08:00:05.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.414+0000 7fd42ffff700 1 -- 172.21.15.98:0/3638154588 wait complete. 2023-03-03T08:00:05.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.416+0000 7fd42ffff700 1 Processor -- start 2023-03-03T08:00:05.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.417+0000 7fd42ffff700 1 -- start start 2023-03-03T08:00:05.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.418+0000 7fd42ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4300731e0 0x7fd4300cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:05.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.418+0000 7fd42ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4300b4ab0 0x7fd4300c9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:05.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.418+0000 7fd42ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd430146360 0x7fd4300735b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:05.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.418+0000 7fd42ffff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4300ccb00 con 0x7fd4300b4ab0 2023-03-03T08:00:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.418+0000 7fd42ffff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43015f8d0 con 0x7fd430146360 2023-03-03T08:00:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.418+0000 7fd42ffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd43015fbb0 con 0x7fd4300731e0 2023-03-03T08:00:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.418+0000 7fd42e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4300b4ab0 0x7fd4300c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.418+0000 7fd42e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4300b4ab0 0x7fd4300c9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50988/0 (socket says 172.21.15.98:50988) 2023-03-03T08:00:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.418+0000 7fd42e7fc700 1 -- 172.21.15.98:0/702224005 learned_addr learned my addr 172.21.15.98:0/702224005 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.418+0000 7fd42effd700 1 --2- 172.21.15.98:0/702224005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4300731e0 0x7fd4300cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.418+0000 7fd42f7fe700 1 --2- 172.21.15.98:0/702224005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd430146360 0x7fd4300735b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.419+0000 7fd42e7fc700 1 -- 172.21.15.98:0/702224005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4300731e0 msgr2=0x7fd4300cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.419+0000 7fd42e7fc700 1 --2- 172.21.15.98:0/702224005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4300731e0 0x7fd4300cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.419+0000 7fd42e7fc700 1 -- 172.21.15.98:0/702224005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd430146360 msgr2=0x7fd4300735b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.419+0000 7fd42e7fc700 1 --2- 172.21.15.98:0/702224005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd430146360 0x7fd4300735b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.419+0000 7fd42e7fc700 1 -- 172.21.15.98:0/702224005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd420009550 con 0x7fd4300b4ab0 2023-03-03T08:00:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.419+0000 7fd42e7fc700 1 --2- 172.21.15.98:0/702224005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4300b4ab0 0x7fd4300c9710 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd42400d800 tx=0x7fd42400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.419+0000 7fd4177fe700 1 -- 172.21.15.98:0/702224005 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4240125f0 con 0x7fd4300b4ab0 2023-03-03T08:00:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.419+0000 7fd4177fe700 1 -- 172.21.15.98:0/702224005 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd424012750 con 0x7fd4300b4ab0 2023-03-03T08:00:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.419+0000 7fd42ffff700 1 -- 172.21.15.98:0/702224005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4300cd960 con 0x7fd4300b4ab0 2023-03-03T08:00:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.419+0000 7fd4177fe700 1 -- 172.21.15.98:0/702224005 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd424012b00 con 0x7fd4300b4ab0 2023-03-03T08:00:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.419+0000 7fd42ffff700 1 -- 172.21.15.98:0/702224005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4300cd680 con 0x7fd4300b4ab0 2023-03-03T08:00:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.420+0000 7fd42ffff700 1 -- 172.21.15.98:0/702224005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4300cdc40 con 0x7fd4300b4ab0 2023-03-03T08:00:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.422+0000 7fd4177fe700 1 -- 172.21.15.98:0/702224005 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fd424012c60 con 0x7fd4300b4ab0 2023-03-03T08:00:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.423+0000 7fd4177fe700 1 --2- 172.21.15.98:0/702224005 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd41c0683c0 0x7fd41c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.423+0000 7fd4177fe700 1 -- 172.21.15.98:0/702224005 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fd424086320 con 0x7fd4300b4ab0 2023-03-03T08:00:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.423+0000 7fd42effd700 1 --2- 172.21.15.98:0/702224005 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd41c0683c0 0x7fd41c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.424+0000 7fd42effd700 1 --2- 172.21.15.98:0/702224005 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd41c0683c0 0x7fd41c06a870 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd420014040 tx=0x7fd420008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.427+0000 7fd4177fe700 1 -- 172.21.15.98:0/702224005 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd424052980 con 0x7fd4300b4ab0 2023-03-03T08:00:05.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.668+0000 7fd42ffff700 1 -- 172.21.15.98:0/702224005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd430068300 con 0x7fd4300b4ab0 2023-03-03T08:00:05.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.668+0000 7fd4177fe700 1 -- 172.21.15.98:0/702224005 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7fd424055fa0 con 0x7fd4300b4ab0 2023-03-03T08:00:05.670 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:05.670 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T08:00:05.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.671+0000 7fd42ffff700 1 -- 172.21.15.98:0/702224005 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd41c0683c0 msgr2=0x7fd41c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:05.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.671+0000 7fd42ffff700 1 --2- 172.21.15.98:0/702224005 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd41c0683c0 0x7fd41c06a870 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd420014040 tx=0x7fd420008030 comp rx=0 tx=0).stop 2023-03-03T08:00:05.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.671+0000 7fd42ffff700 1 -- 172.21.15.98:0/702224005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4300b4ab0 msgr2=0x7fd4300c9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:05.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.671+0000 7fd42ffff700 1 --2- 172.21.15.98:0/702224005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4300b4ab0 0x7fd4300c9710 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd42400d800 tx=0x7fd42400db10 comp rx=0 tx=0).stop 2023-03-03T08:00:05.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.671+0000 7fd42ffff700 1 -- 172.21.15.98:0/702224005 shutdown_connections 2023-03-03T08:00:05.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.671+0000 7fd42ffff700 1 --2- 172.21.15.98:0/702224005 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd41c0683c0 0x7fd41c06a870 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.671+0000 7fd42ffff700 1 --2- 172.21.15.98:0/702224005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd430146360 0x7fd4300735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.671+0000 7fd42ffff700 1 --2- 172.21.15.98:0/702224005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4300731e0 0x7fd4300cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.671+0000 7fd42ffff700 1 --2- 172.21.15.98:0/702224005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4300b4ab0 0x7fd4300c9710 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.671+0000 7fd42ffff700 1 -- 172.21.15.98:0/702224005 >> 172.21.15.98:0/702224005 conn(0x7fd43006b8e0 msgr2=0x7fd4300d2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:05.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.671+0000 7fd42ffff700 1 -- 172.21.15.98:0/702224005 shutdown_connections 2023-03-03T08:00:05.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.671+0000 7fd42ffff700 1 -- 172.21.15.98:0/702224005 wait complete. 2023-03-03T08:00:05.674 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T08:00:05.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:00:05.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.944+0000 7f692539d700 1 Processor -- start 2023-03-03T08:00:05.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.944+0000 7f692539d700 1 -- start start 2023-03-03T08:00:05.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.946+0000 7f692539d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6920146360 0x7f6920160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:05.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.946+0000 7f692539d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69201614c0 con 0x7f69200d2590 2023-03-03T08:00:05.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.946+0000 7f692539d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69200d10f0 con 0x7f69200b4ab0 2023-03-03T08:00:05.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.946+0000 7f692539d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69200cdf20 con 0x7f6920146360 2023-03-03T08:00:05.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.946+0000 7f691ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6920146360 0x7f6920160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:05.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.946+0000 7f6924b9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f69200d2590 0x7f69200738f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46530/0 (socket says 172.21.15.98:46530) 2023-03-03T08:00:05.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.946+0000 7f6924b9c700 1 -- 172.21.15.98:0/1809886068 learned_addr learned my addr 172.21.15.98:0/1809886068 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:05.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.947+0000 7f691ffff700 1 -- 172.21.15.98:0/1809886068 >> v1:172.21.15.100:6789/0 conn(0x7f69200b4ab0 legacy=0x7f69200731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:05.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.947+0000 7f691ffff700 1 -- 172.21.15.98:0/1809886068 >> v1:172.21.15.98:6789/0 conn(0x7f69200d2590 legacy=0x7f69200738f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:05.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.947+0000 7f691ffff700 1 -- 172.21.15.98:0/1809886068 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6920074000 con 0x7f6920146360 2023-03-03T08:00:05.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.947+0000 7f691ffff700 1 --2- 172.21.15.98:0/1809886068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6920146360 0x7f6920160f80 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f6910009840 tx=0x7f6910009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93031bff963c38b2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:05.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.947+0000 7f691effd700 1 -- 172.21.15.98:0/1809886068 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910011830 con 0x7f6920146360 2023-03-03T08:00:05.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.947+0000 7f691effd700 1 -- 172.21.15.98:0/1809886068 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6910011990 con 0x7f6920146360 2023-03-03T08:00:05.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.947+0000 7f691effd700 1 -- 172.21.15.98:0/1809886068 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910011d20 con 0x7f6920146360 2023-03-03T08:00:05.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.948+0000 7f692539d700 1 -- 172.21.15.98:0/1809886068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6920146360 msgr2=0x7f6920160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:05.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.948+0000 7f692539d700 1 --2- 172.21.15.98:0/1809886068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6920146360 0x7f6920160f80 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f6910009840 tx=0x7f6910009b50 comp rx=0 tx=0).stop 2023-03-03T08:00:05.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.948+0000 7f692539d700 1 -- 172.21.15.98:0/1809886068 shutdown_connections 2023-03-03T08:00:05.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.948+0000 7f692539d700 1 --2- 172.21.15.98:0/1809886068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6920146360 0x7f6920160f80 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.948+0000 7f692539d700 1 -- 172.21.15.98:0/1809886068 >> 172.21.15.98:0/1809886068 conn(0x7f692006b8e0 msgr2=0x7f69200cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.948+0000 7f692539d700 1 -- 172.21.15.98:0/1809886068 shutdown_connections 2023-03-03T08:00:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.948+0000 7f692539d700 1 -- 172.21.15.98:0/1809886068 wait complete. 2023-03-03T08:00:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.950+0000 7f692539d700 1 Processor -- start 2023-03-03T08:00:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.950+0000 7f692539d700 1 -- start start 2023-03-03T08:00:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.951+0000 7f692539d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69200b4ab0 0x7f69200c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.951+0000 7f692539d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69200d2590 0x7f6920152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.951+0000 7f692539d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6920146360 0x7f6920156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.951+0000 7f692539d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69201576a0 con 0x7f6920146360 2023-03-03T08:00:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.951+0000 7f692539d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69200c64b0 con 0x7f69200b4ab0 2023-03-03T08:00:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.951+0000 7f692539d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69200c6ce0 con 0x7f69200d2590 2023-03-03T08:00:05.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.951+0000 7f6924b9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6920146360 0x7f6920156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:05.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.952+0000 7f6924b9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6920146360 0x7f6920156b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51000/0 (socket says 172.21.15.98:51000) 2023-03-03T08:00:05.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.952+0000 7f6924b9c700 1 -- 172.21.15.98:0/4228325102 learned_addr learned my addr 172.21.15.98:0/4228325102 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:05.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.952+0000 7f691f7fe700 1 --2- 172.21.15.98:0/4228325102 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69200d2590 0x7f6920152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:05.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.952+0000 7f691ffff700 1 --2- 172.21.15.98:0/4228325102 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69200b4ab0 0x7f69200c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:05.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.952+0000 7f6924b9c700 1 -- 172.21.15.98:0/4228325102 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69200d2590 msgr2=0x7f6920152030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:05.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.952+0000 7f6924b9c700 1 --2- 172.21.15.98:0/4228325102 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69200d2590 0x7f6920152030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.952+0000 7f6924b9c700 1 -- 172.21.15.98:0/4228325102 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69200b4ab0 msgr2=0x7f69200c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:05.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.952+0000 7f6924b9c700 1 --2- 172.21.15.98:0/4228325102 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69200b4ab0 0x7f69200c5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:05.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.952+0000 7f6924b9c700 1 -- 172.21.15.98:0/4228325102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69100094a0 con 0x7f6920146360 2023-03-03T08:00:05.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.952+0000 7f691ffff700 1 --2- 172.21.15.98:0/4228325102 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69200b4ab0 0x7f69200c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:05.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.952+0000 7f6924b9c700 1 --2- 172.21.15.98:0/4228325102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6920146360 0x7f6920156b00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f691800ca80 tx=0x7f691800ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:05.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.952+0000 7f691cff9700 1 -- 172.21.15.98:0/4228325102 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691800aaa0 con 0x7f6920146360 2023-03-03T08:00:05.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.953+0000 7f691cff9700 1 -- 172.21.15.98:0/4228325102 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691800ac00 con 0x7f6920146360 2023-03-03T08:00:05.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.953+0000 7f692539d700 1 -- 172.21.15.98:0/4228325102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69200cd960 con 0x7f6920146360 2023-03-03T08:00:05.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.953+0000 7f691cff9700 1 -- 172.21.15.98:0/4228325102 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918012400 con 0x7f6920146360 2023-03-03T08:00:05.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.953+0000 7f692539d700 1 -- 172.21.15.98:0/4228325102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69200cd680 con 0x7f6920146360 2023-03-03T08:00:05.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.954+0000 7f692539d700 1 -- 172.21.15.98:0/4228325102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69200cdc40 con 0x7f6920146360 2023-03-03T08:00:05.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.955+0000 7f691cff9700 1 -- 172.21.15.98:0/4228325102 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f6918019030 con 0x7f6920146360 2023-03-03T08:00:05.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.956+0000 7f691cff9700 1 --2- 172.21.15.98:0/4228325102 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f690c0683f0 0x7f690c06a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:05.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.956+0000 7f691cff9700 1 -- 172.21.15.98:0/4228325102 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f69180857b0 con 0x7f6920146360 2023-03-03T08:00:05.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.956+0000 7f691ffff700 1 --2- 172.21.15.98:0/4228325102 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f690c0683f0 0x7f690c06a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:05.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.957+0000 7f691ffff700 1 --2- 172.21.15.98:0/4228325102 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f690c0683f0 0x7f690c06a8a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f69100090b0 tx=0x7f6910008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:05.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:05.960+0000 7f691cff9700 1 -- 172.21.15.98:0/4228325102 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f691804b5e0 con 0x7f6920146360 2023-03-03T08:00:06.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.200+0000 7f692539d700 1 -- 172.21.15.98:0/4228325102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69201614c0 con 0x7f6920146360 2023-03-03T08:00:06.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.200+0000 7f691cff9700 1 -- 172.21.15.98:0/4228325102 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f69201614c0 con 0x7f6920146360 2023-03-03T08:00:06.202 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:06.202 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T07:58:13.214065+0000","modified":"2023-03-03T07:58:16.671988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5745},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5745":{"gid":5745,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/1111074388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1111074388},{"type":"v1","addr":"172.21.15.98:6837","nonce":1111074388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-03-03T08:00:06.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.203+0000 7f692539d700 1 -- 172.21.15.98:0/4228325102 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f690c0683f0 msgr2=0x7f690c06a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:06.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.203+0000 7f692539d700 1 --2- 172.21.15.98:0/4228325102 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f690c0683f0 0x7f690c06a8a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f69100090b0 tx=0x7f6910008030 comp rx=0 tx=0).stop 2023-03-03T08:00:06.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.203+0000 7f692539d700 1 -- 172.21.15.98:0/4228325102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6920146360 msgr2=0x7f6920156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:06.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.203+0000 7f692539d700 1 --2- 172.21.15.98:0/4228325102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6920146360 0x7f6920156b00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f691800ca80 tx=0x7f691800ce40 comp rx=0 tx=0).stop 2023-03-03T08:00:06.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.203+0000 7f692539d700 1 -- 172.21.15.98:0/4228325102 shutdown_connections 2023-03-03T08:00:06.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.203+0000 7f692539d700 1 --2- 172.21.15.98:0/4228325102 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f690c0683f0 0x7f690c06a8a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.203+0000 7f692539d700 1 --2- 172.21.15.98:0/4228325102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6920146360 0x7f6920156b00 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.203+0000 7f692539d700 1 --2- 172.21.15.98:0/4228325102 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69200b4ab0 0x7f69200c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.203+0000 7f692539d700 1 --2- 172.21.15.98:0/4228325102 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69200d2590 0x7f6920152030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.203+0000 7f692539d700 1 -- 172.21.15.98:0/4228325102 >> 172.21.15.98:0/4228325102 conn(0x7f692006b8e0 msgr2=0x7f69200b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:06.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.203+0000 7f692539d700 1 -- 172.21.15.98:0/4228325102 shutdown_connections 2023-03-03T08:00:06.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.203+0000 7f692539d700 1 -- 172.21.15.98:0/4228325102 wait complete. 2023-03-03T08:00:06.206 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2023-03-03T08:00:06.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:00:06.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.476+0000 7fc4b09ff700 1 Processor -- start 2023-03-03T08:00:06.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.476+0000 7fc4b09ff700 1 -- start start 2023-03-03T08:00:06.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.477+0000 7fc4b09ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc4ac0b8b90 0x7fc4ac0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:06.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.477+0000 7fc4b09ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4ac0b5410 0x7fc4ac0d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:06.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.477+0000 7fc4b09ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ac0d0ef0 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.477+0000 7fc4b09ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ac0d0600 con 0x7fc4ac0b8b90 2023-03-03T08:00:06.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.477+0000 7fc4b09ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac0cd970 con 0x7fc4ac0b5410 2023-03-03T08:00:06.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.478+0000 7fc4aa59c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc4ac0b87c0 0x7fc4ac0b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46536/0 (socket says 172.21.15.98:46536) 2023-03-03T08:00:06.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.478+0000 7fc4aa59c700 1 -- 172.21.15.98:0/2254513887 learned_addr learned my addr 172.21.15.98:0/2254513887 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.478+0000 7fc4aad9d700 1 --2- 172.21.15.98:0/2254513887 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4ac0b5410 0x7fc4ac0d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.478+0000 7fc4a9d9b700 1 --2- 172.21.15.98:0/2254513887 >> v2:172.21.15.100:3300/0 conn(0x7fc4ac0b8b90 0x7fc4ac0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.478+0000 7fc4a959a700 1 -- 172.21.15.98:0/2254513887 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911936787 0 0) 0x7fc4ac0d0ef0 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.478+0000 7fc4a959a700 1 -- 172.21.15.98:0/2254513887 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4940035b0 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.478+0000 7fc4aad9d700 1 -- 172.21.15.98:0/2254513887 >> v2:172.21.15.100:3300/0 conn(0x7fc4ac0b8b90 msgr2=0x7fc4ac0cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.478+0000 7fc4aad9d700 1 --2- 172.21.15.98:0/2254513887 >> v2:172.21.15.100:3300/0 conn(0x7fc4ac0b8b90 0x7fc4ac0cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.478+0000 7fc4aad9d700 1 -- 172.21.15.98:0/2254513887 >> v1:172.21.15.98:6789/0 conn(0x7fc4ac0b87c0 legacy=0x7fc4ac0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.478+0000 7fc4aad9d700 1 -- 172.21.15.98:0/2254513887 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac0cdc50 con 0x7fc4ac0b5410 2023-03-03T08:00:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.479+0000 7fc4aad9d700 1 --2- 172.21.15.98:0/2254513887 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4ac0b5410 0x7fc4ac0d13d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4009d90 tx=0x7fc4a400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edac7b63f70f28 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.479+0000 7fc4a959a700 1 -- 172.21.15.98:0/2254513887 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a400e070 con 0x7fc4ac0b5410 2023-03-03T08:00:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.479+0000 7fc4a959a700 1 -- 172.21.15.98:0/2254513887 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a400bb20 con 0x7fc4ac0b5410 2023-03-03T08:00:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.479+0000 7fc4a959a700 1 -- 172.21.15.98:0/2254513887 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4012430 con 0x7fc4ac0b5410 2023-03-03T08:00:06.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.480+0000 7fc4b09ff700 1 -- 172.21.15.98:0/2254513887 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4ac0b5410 msgr2=0x7fc4ac0d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:06.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.480+0000 7fc4b09ff700 1 --2- 172.21.15.98:0/2254513887 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4ac0b5410 0x7fc4ac0d13d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4009d90 tx=0x7fc4a400b330 comp rx=0 tx=0).stop 2023-03-03T08:00:06.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.480+0000 7fc4b09ff700 1 -- 172.21.15.98:0/2254513887 shutdown_connections 2023-03-03T08:00:06.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.480+0000 7fc4b09ff700 1 --2- 172.21.15.98:0/2254513887 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4ac0b5410 0x7fc4ac0d13d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.480+0000 7fc4b09ff700 1 --2- 172.21.15.98:0/2254513887 >> v2:172.21.15.100:3300/0 conn(0x7fc4ac0b8b90 0x7fc4ac0cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.480+0000 7fc4b09ff700 1 -- 172.21.15.98:0/2254513887 >> 172.21.15.98:0/2254513887 conn(0x7fc4ac06aee0 msgr2=0x7fc4ac06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:06.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.480+0000 7fc4b09ff700 1 -- 172.21.15.98:0/2254513887 shutdown_connections 2023-03-03T08:00:06.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.480+0000 7fc4b09ff700 1 -- 172.21.15.98:0/2254513887 wait complete. 2023-03-03T08:00:06.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.482+0000 7fc4b09ff700 1 Processor -- start 2023-03-03T08:00:06.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.482+0000 7fc4b09ff700 1 -- start start 2023-03-03T08:00:06.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.483+0000 7fc4b09ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4ac0b5410 0x7fc4ac1f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.483+0000 7fc4b09ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4ac0b87c0 0x7fc4ac14c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.483+0000 7fc4b09ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4ac0b8b90 0x7fc4ac0a9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.483+0000 7fc4b09ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac1f3820 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.483+0000 7fc4b09ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac14df20 con 0x7fc4ac0b5410 2023-03-03T08:00:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.483+0000 7fc4b09ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac14cea0 con 0x7fc4ac0b8b90 2023-03-03T08:00:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.484+0000 7fc4a9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4ac0b87c0 0x7fc4ac14c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.484+0000 7fc4a9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4ac0b87c0 0x7fc4ac14c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51012/0 (socket says 172.21.15.98:51012) 2023-03-03T08:00:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.484+0000 7fc4a9d9b700 1 -- 172.21.15.98:0/1482997412 learned_addr learned my addr 172.21.15.98:0/1482997412 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.484+0000 7fc4aad9d700 1 --2- 172.21.15.98:0/1482997412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4ac0b8b90 0x7fc4ac0a9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.484+0000 7fc4aa59c700 1 --2- 172.21.15.98:0/1482997412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4ac0b5410 0x7fc4ac1f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.484+0000 7fc4a9d9b700 1 -- 172.21.15.98:0/1482997412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4ac0b8b90 msgr2=0x7fc4ac0a9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.484+0000 7fc4a9d9b700 1 --2- 172.21.15.98:0/1482997412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4ac0b8b90 0x7fc4ac0a9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.484+0000 7fc4a9d9b700 1 -- 172.21.15.98:0/1482997412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4ac0b5410 msgr2=0x7fc4ac1f32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.484+0000 7fc4a9d9b700 1 --2- 172.21.15.98:0/1482997412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4ac0b5410 0x7fc4ac1f32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.484+0000 7fc4a9d9b700 1 -- 172.21.15.98:0/1482997412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a4009a70 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.484+0000 7fc4a9d9b700 1 --2- 172.21.15.98:0/1482997412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4ac0b87c0 0x7fc4ac14c960 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000e840 tx=0x7fc4a000eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:06.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.485+0000 7fc49affd700 1 -- 172.21.15.98:0/1482997412 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0009590 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.485+0000 7fc49affd700 1 -- 172.21.15.98:0/1482997412 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a0013680 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.485+0000 7fc49affd700 1 -- 172.21.15.98:0/1482997412 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0013a30 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.485+0000 7fc4b09ff700 1 -- 172.21.15.98:0/1482997412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ac0cdf30 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.485+0000 7fc4b09ff700 1 -- 172.21.15.98:0/1482997412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ac073a80 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.486+0000 7fc4b09ff700 1 -- 172.21.15.98:0/1482997412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ac0ce0c0 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.487+0000 7fc49affd700 1 -- 172.21.15.98:0/1482997412 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fc4a0019030 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.489+0000 7fc49affd700 1 --2- 172.21.15.98:0/1482997412 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc48c0683c0 0x7fc48c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.489+0000 7fc49affd700 1 -- 172.21.15.98:0/1482997412 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fc4a0085c90 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.489+0000 7fc4aa59c700 1 --2- 172.21.15.98:0/1482997412 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc48c0683c0 0x7fc48c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.490+0000 7fc4aa59c700 1 --2- 172.21.15.98:0/1482997412 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc48c0683c0 0x7fc48c06a870 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc49c0011e0 tx=0x7fc49c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:06.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.493+0000 7fc49affd700 1 -- 172.21.15.98:0/1482997412 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc4a00522f0 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.690+0000 7fc4b09ff700 1 -- 172.21.15.98:0/1482997412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4ac0729b0 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.692+0000 7fc49affd700 1 -- 172.21.15.98:0/1482997412 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+18018 (secure 0 0 0) 0x7fc4a0055910 con 0x7fc4ac0b87c0 2023-03-03T08:00:06.693 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:06.693 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":54,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T07:58:28.723281+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-03-03T07:58:10.728830+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-03-03T07:58:12.224649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.6","mappings":[{"from":0,"to":1}]},{"pgid":"8.17","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:00:06.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.694+0000 7fc4b09ff700 1 -- 172.21.15.98:0/1482997412 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc48c0683c0 msgr2=0x7fc48c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:06.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.694+0000 7fc4b09ff700 1 --2- 172.21.15.98:0/1482997412 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc48c0683c0 0x7fc48c06a870 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc49c0011e0 tx=0x7fc49c009040 comp rx=0 tx=0).stop 2023-03-03T08:00:06.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.694+0000 7fc4b09ff700 1 -- 172.21.15.98:0/1482997412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4ac0b87c0 msgr2=0x7fc4ac14c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:06.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.694+0000 7fc4b09ff700 1 --2- 172.21.15.98:0/1482997412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4ac0b87c0 0x7fc4ac14c960 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000e840 tx=0x7fc4a000eb50 comp rx=0 tx=0).stop 2023-03-03T08:00:06.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.694+0000 7fc4b09ff700 1 -- 172.21.15.98:0/1482997412 shutdown_connections 2023-03-03T08:00:06.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.694+0000 7fc4b09ff700 1 --2- 172.21.15.98:0/1482997412 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc48c0683c0 0x7fc48c06a870 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.694+0000 7fc4b09ff700 1 --2- 172.21.15.98:0/1482997412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4ac0b8b90 0x7fc4ac0a9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.694+0000 7fc4b09ff700 1 --2- 172.21.15.98:0/1482997412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4ac0b5410 0x7fc4ac1f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.694+0000 7fc4b09ff700 1 --2- 172.21.15.98:0/1482997412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4ac0b87c0 0x7fc4ac14c960 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.694+0000 7fc4b09ff700 1 -- 172.21.15.98:0/1482997412 >> 172.21.15.98:0/1482997412 conn(0x7fc4ac06aee0 msgr2=0x7fc4ac0b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:06.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.695+0000 7fc4b09ff700 1 -- 172.21.15.98:0/1482997412 shutdown_connections 2023-03-03T08:00:06.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.695+0000 7fc4b09ff700 1 -- 172.21.15.98:0/1482997412 wait complete. 2023-03-03T08:00:06.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-03-03T08:00:06.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.969+0000 7f4089d9d700 1 Processor -- start 2023-03-03T08:00:06.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.969+0000 7f4089d9d700 1 -- start start 2023-03-03T08:00:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.971+0000 7f4089d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f40840cf570 0x7f408415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.971+0000 7f4089d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f408415db30 0x7f40840b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.971+0000 7f4089d9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40840cd780 con 0x7f40841609b0 2023-03-03T08:00:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.971+0000 7f4089d9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40840cebb0 con 0x7f40840cf570 2023-03-03T08:00:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.971+0000 7f4089d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40840cda60 con 0x7f408415db30 2023-03-03T08:00:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.971+0000 7f4083fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f40840cf570 0x7f408415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.971+0000 7f4088d9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f40841609b0 0x7f4084160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46542/0 (socket says 172.21.15.98:46542) 2023-03-03T08:00:06.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.971+0000 7f4088d9b700 1 -- 172.21.15.98:0/1534440174 learned_addr learned my addr 172.21.15.98:0/1534440174 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.972+0000 7f408959c700 1 --2- 172.21.15.98:0/1534440174 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f408415db30 0x7f40840b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.972+0000 7f4083fff700 1 -- 172.21.15.98:0/1534440174 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f408415db30 msgr2=0x7f40840b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.972+0000 7f4083fff700 1 --2- 172.21.15.98:0/1534440174 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f408415db30 0x7f40840b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.972+0000 7f4083fff700 1 -- 172.21.15.98:0/1534440174 >> v1:172.21.15.98:6789/0 conn(0x7f40841609b0 legacy=0x7f4084160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.972+0000 7f4083fff700 1 -- 172.21.15.98:0/1534440174 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40840cdd40 con 0x7f40840cf570 2023-03-03T08:00:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.972+0000 7f40837fe700 1 -- 172.21.15.98:0/1534440174 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241820401 0 0) 0x7f40840cd780 con 0x7f40841609b0 2023-03-03T08:00:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.972+0000 7f4083fff700 1 --2- 172.21.15.98:0/1534440174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40840cf570 0x7f408415d5f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f4070009840 tx=0x7f4070009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac1f2e3e35afd5aa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.972+0000 7f40837fe700 1 -- 172.21.15.98:0/1534440174 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4070011830 con 0x7f40840cf570 2023-03-03T08:00:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.972+0000 7f40837fe700 1 -- 172.21.15.98:0/1534440174 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4070011990 con 0x7f40840cf570 2023-03-03T08:00:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.972+0000 7f40837fe700 1 -- 172.21.15.98:0/1534440174 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4070011d20 con 0x7f40840cf570 2023-03-03T08:00:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.973+0000 7f4089d9d700 1 -- 172.21.15.98:0/1534440174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40840cf570 msgr2=0x7f408415d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.973+0000 7f4089d9d700 1 --2- 172.21.15.98:0/1534440174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40840cf570 0x7f408415d5f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f4070009840 tx=0x7f4070009b50 comp rx=0 tx=0).stop 2023-03-03T08:00:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.973+0000 7f4089d9d700 1 -- 172.21.15.98:0/1534440174 shutdown_connections 2023-03-03T08:00:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.973+0000 7f4089d9d700 1 --2- 172.21.15.98:0/1534440174 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f408415db30 0x7f40840b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.973+0000 7f4089d9d700 1 --2- 172.21.15.98:0/1534440174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40840cf570 0x7f408415d5f0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.973+0000 7f4089d9d700 1 -- 172.21.15.98:0/1534440174 >> 172.21.15.98:0/1534440174 conn(0x7f408406b880 msgr2=0x7f40840cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.973+0000 7f4089d9d700 1 -- 172.21.15.98:0/1534440174 shutdown_connections 2023-03-03T08:00:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.973+0000 7f4089d9d700 1 -- 172.21.15.98:0/1534440174 wait complete. 2023-03-03T08:00:06.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.975+0000 7f4089d9d700 1 Processor -- start 2023-03-03T08:00:06.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.975+0000 7f4089d9d700 1 -- start start 2023-03-03T08:00:06.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.976+0000 7f4089d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40840cf570 0x7f40840c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:06.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.976+0000 7f4089d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f408415db30 0x7f4084151310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:06.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.977+0000 7f4089d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40841609b0 0x7f4084151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.977+0000 7f4089d9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40840b6c50 con 0x7f40840cf570 2023-03-03T08:00:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.977+0000 7f4089d9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40840c99a0 con 0x7f40841609b0 2023-03-03T08:00:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.977+0000 7f4089d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40840c93f0 con 0x7f408415db30 2023-03-03T08:00:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.977+0000 7f4088d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40840cf570 0x7f40840c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.977+0000 7f4088d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40840cf570 0x7f40840c8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51040/0 (socket says 172.21.15.98:51040) 2023-03-03T08:00:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.977+0000 7f4088d9b700 1 -- 172.21.15.98:0/2747973390 learned_addr learned my addr 172.21.15.98:0/2747973390 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.977+0000 7f408959c700 1 --2- 172.21.15.98:0/2747973390 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40841609b0 0x7f4084151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.977+0000 7f4083fff700 1 --2- 172.21.15.98:0/2747973390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f408415db30 0x7f4084151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.977+0000 7f4088d9b700 1 -- 172.21.15.98:0/2747973390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f408415db30 msgr2=0x7f4084151310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.977+0000 7f4088d9b700 1 --2- 172.21.15.98:0/2747973390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f408415db30 0x7f4084151310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.977+0000 7f4088d9b700 1 -- 172.21.15.98:0/2747973390 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40841609b0 msgr2=0x7f4084151850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.977+0000 7f4088d9b700 1 --2- 172.21.15.98:0/2747973390 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40841609b0 0x7f4084151850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.977+0000 7f4088d9b700 1 -- 172.21.15.98:0/2747973390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40700094a0 con 0x7f40840cf570 2023-03-03T08:00:06.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.978+0000 7f4088d9b700 1 --2- 172.21.15.98:0/2747973390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40840cf570 0x7f40840c8eb0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f407800c550 tx=0x7f407800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:06.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.978+0000 7f40817fa700 1 -- 172.21.15.98:0/2747973390 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4078004c80 con 0x7f40840cf570 2023-03-03T08:00:06.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.978+0000 7f40817fa700 1 -- 172.21.15.98:0/2747973390 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4078004e00 con 0x7f40840cf570 2023-03-03T08:00:06.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.978+0000 7f4089d9d700 1 -- 172.21.15.98:0/2747973390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408415d0d0 con 0x7f40840cf570 2023-03-03T08:00:06.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.978+0000 7f40817fa700 1 -- 172.21.15.98:0/2747973390 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4078007310 con 0x7f40840cf570 2023-03-03T08:00:06.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.978+0000 7f4089d9d700 1 -- 172.21.15.98:0/2747973390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408415cdf0 con 0x7f40840cf570 2023-03-03T08:00:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.979+0000 7f4089d9d700 1 -- 172.21.15.98:0/2747973390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40840b4fe0 con 0x7f40840cf570 2023-03-03T08:00:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.980+0000 7f40817fa700 1 -- 172.21.15.98:0/2747973390 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f407801a020 con 0x7f40840cf570 2023-03-03T08:00:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.982+0000 7f40817fa700 1 --2- 172.21.15.98:0/2747973390 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f40680683c0 0x7f406806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.982+0000 7f40817fa700 1 -- 172.21.15.98:0/2747973390 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f4078085a20 con 0x7f40840cf570 2023-03-03T08:00:06.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.982+0000 7f4083fff700 1 --2- 172.21.15.98:0/2747973390 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f40680683c0 0x7f406806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:06.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.983+0000 7f4083fff700 1 --2- 172.21.15.98:0/2747973390 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f40680683c0 0x7f406806a870 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f40700090b0 tx=0x7f4070008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:06.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:06.985+0000 7f40817fa700 1 -- 172.21.15.98:0/2747973390 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4078052270 con 0x7f40840cf570 2023-03-03T08:00:07.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.220+0000 7f4089d9d700 1 -- 172.21.15.98:0/2747973390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4084068290 con 0x7f40840cf570 2023-03-03T08:00:07.577 INFO:tasks.ceph.mon.a.smithi098.stderr:2023-03-03T08:00:07.576+0000 7fb71abf9700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-03-03T08:00:07.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.583+0000 7f40817fa700 1 -- 172.21.15.98:0/2747973390 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4078055890 con 0x7f40840cf570 2023-03-03T08:00:07.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.587+0000 7f4089d9d700 1 -- 172.21.15.98:0/2747973390 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f40680683c0 msgr2=0x7f406806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:07.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.587+0000 7f4089d9d700 1 --2- 172.21.15.98:0/2747973390 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f40680683c0 0x7f406806a870 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f40700090b0 tx=0x7f4070008030 comp rx=0 tx=0).stop 2023-03-03T08:00:07.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.587+0000 7f4089d9d700 1 -- 172.21.15.98:0/2747973390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40840cf570 msgr2=0x7f40840c8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:07.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.587+0000 7f4089d9d700 1 --2- 172.21.15.98:0/2747973390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40840cf570 0x7f40840c8eb0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f407800c550 tx=0x7f407800c910 comp rx=0 tx=0).stop 2023-03-03T08:00:07.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.587+0000 7f4089d9d700 1 -- 172.21.15.98:0/2747973390 shutdown_connections 2023-03-03T08:00:07.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.587+0000 7f4089d9d700 1 --2- 172.21.15.98:0/2747973390 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f40680683c0 0x7f406806a870 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:07.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.587+0000 7f4089d9d700 1 --2- 172.21.15.98:0/2747973390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40840cf570 0x7f40840c8eb0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.587+0000 7f4089d9d700 1 --2- 172.21.15.98:0/2747973390 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40841609b0 0x7f4084151850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.587+0000 7f4089d9d700 1 --2- 172.21.15.98:0/2747973390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f408415db30 0x7f4084151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.587+0000 7f4089d9d700 1 -- 172.21.15.98:0/2747973390 >> 172.21.15.98:0/2747973390 conn(0x7f408406b880 msgr2=0x7f408415e000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.587+0000 7f4089d9d700 1 -- 172.21.15.98:0/2747973390 shutdown_connections 2023-03-03T08:00:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.587+0000 7f4089d9d700 1 -- 172.21.15.98:0/2747973390 wait complete. 2023-03-03T08:00:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-03-03T08:00:07.611 INFO:tasks.ceph.mds.a.smithi098.stderr: -9158> 2023-03-03T07:58:16.631+0000 7ff317ff1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T08:00:07.631 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-03-03T08:00:07.696 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2023-03-03T08:00:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.881+0000 7f8cb759d700 1 Processor -- start 2023-03-03T08:00:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.882+0000 7f8cb759d700 1 -- start start 2023-03-03T08:00:07.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.883+0000 7f8cb759d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8cb01609b0 0x7f8cb0160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.883+0000 7f8cb759d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8cb00b4aa0 0x7f8cb00b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.883+0000 7f8cb759d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb01612c0 con 0x7f8cb00cf570 2023-03-03T08:00:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.883+0000 7f8cb759d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb00cebb0 con 0x7f8cb00b4aa0 2023-03-03T08:00:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.883+0000 7f8cb759d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb00b5460 con 0x7f8cb01609b0 2023-03-03T08:00:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.883+0000 7f8cb6d9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8cb00b4aa0 0x7f8cb00b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.883+0000 7f8cb5d9a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8cb00cf570 0x7f8cb015d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46572/0 (socket says 172.21.15.98:46572) 2023-03-03T08:00:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.883+0000 7f8cb5d9a700 1 -- 172.21.15.98:0/4177834101 learned_addr learned my addr 172.21.15.98:0/4177834101 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.884+0000 7f8cb659b700 1 --2- 172.21.15.98:0/4177834101 >> v2:172.21.15.100:3300/0 conn(0x7f8cb01609b0 0x7f8cb0160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.884+0000 7f8cb5599700 1 -- 172.21.15.98:0/4177834101 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973897166 0 0) 0x7f8cb01612c0 con 0x7f8cb00cf570 2023-03-03T08:00:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.884+0000 7f8cb6d9c700 1 -- 172.21.15.98:0/4177834101 >> v2:172.21.15.100:3300/0 conn(0x7f8cb01609b0 msgr2=0x7f8cb0160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.884+0000 7f8cb6d9c700 1 --2- 172.21.15.98:0/4177834101 >> v2:172.21.15.100:3300/0 conn(0x7f8cb01609b0 0x7f8cb0160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.884+0000 7f8cb6d9c700 1 -- 172.21.15.98:0/4177834101 >> v1:172.21.15.98:6789/0 conn(0x7f8cb00cf570 legacy=0x7f8cb015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:07.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.884+0000 7f8cb6d9c700 1 -- 172.21.15.98:0/4177834101 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb015dd00 con 0x7f8cb00b4aa0 2023-03-03T08:00:07.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.884+0000 7f8cb6d9c700 1 --2- 172.21.15.98:0/4177834101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb00b4aa0 0x7f8cb00b4f20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8cb00742b0 tx=0x7f8ca800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae051dd26bc0160a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:07.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.885+0000 7f8cb5599700 1 -- 172.21.15.98:0/4177834101 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca800e070 con 0x7f8cb00b4aa0 2023-03-03T08:00:07.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.885+0000 7f8cb5599700 1 -- 172.21.15.98:0/4177834101 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca800bb20 con 0x7f8cb00b4aa0 2023-03-03T08:00:07.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.885+0000 7f8cb5599700 1 -- 172.21.15.98:0/4177834101 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca8012430 con 0x7f8cb00b4aa0 2023-03-03T08:00:07.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.885+0000 7f8cb759d700 1 -- 172.21.15.98:0/4177834101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb00b4aa0 msgr2=0x7f8cb00b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:07.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.885+0000 7f8cb759d700 1 --2- 172.21.15.98:0/4177834101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb00b4aa0 0x7f8cb00b4f20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8cb00742b0 tx=0x7f8ca800b330 comp rx=0 tx=0).stop 2023-03-03T08:00:07.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.885+0000 7f8cb759d700 1 -- 172.21.15.98:0/4177834101 shutdown_connections 2023-03-03T08:00:07.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.885+0000 7f8cb759d700 1 --2- 172.21.15.98:0/4177834101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb00b4aa0 0x7f8cb00b4f20 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:07.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.885+0000 7f8cb759d700 1 --2- 172.21.15.98:0/4177834101 >> v2:172.21.15.100:3300/0 conn(0x7f8cb01609b0 0x7f8cb0160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:07.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.885+0000 7f8cb759d700 1 -- 172.21.15.98:0/4177834101 >> 172.21.15.98:0/4177834101 conn(0x7f8cb006b880 msgr2=0x7f8cb00cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:07.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.886+0000 7f8cb759d700 1 -- 172.21.15.98:0/4177834101 shutdown_connections 2023-03-03T08:00:07.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.886+0000 7f8cb759d700 1 -- 172.21.15.98:0/4177834101 wait complete. 2023-03-03T08:00:07.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.887+0000 7f8cb759d700 1 Processor -- start 2023-03-03T08:00:07.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.888+0000 7f8cb759d700 1 -- start start 2023-03-03T08:00:07.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.889+0000 7f8cb759d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb00b4aa0 0x7f8cb0148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:07.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.889+0000 7f8cb759d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb00cf570 0x7f8cb01f5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:07.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.889+0000 7f8cb759d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb01609b0 0x7f8cb00d16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:07.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.889+0000 7f8cb759d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb00b7090 con 0x7f8cb00cf570 2023-03-03T08:00:07.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.889+0000 7f8cb759d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb00b7610 con 0x7f8cb01609b0 2023-03-03T08:00:07.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.889+0000 7f8cb759d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb01f63d0 con 0x7f8cb00b4aa0 2023-03-03T08:00:07.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.889+0000 7f8cb5d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb00cf570 0x7f8cb01f5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:07.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.889+0000 7f8cb5d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb00cf570 0x7f8cb01f5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51058/0 (socket says 172.21.15.98:51058) 2023-03-03T08:00:07.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.889+0000 7f8cb5d9a700 1 -- 172.21.15.98:0/1399823142 learned_addr learned my addr 172.21.15.98:0/1399823142 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:07.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.889+0000 7f8cb659b700 1 --2- 172.21.15.98:0/1399823142 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb00b4aa0 0x7f8cb0148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:07.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.889+0000 7f8cb6d9c700 1 --2- 172.21.15.98:0/1399823142 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb01609b0 0x7f8cb00d16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:07.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.889+0000 7f8cb5d9a700 1 -- 172.21.15.98:0/1399823142 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb00b4aa0 msgr2=0x7f8cb0148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:07.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.890+0000 7f8cb5d9a700 1 --2- 172.21.15.98:0/1399823142 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb00b4aa0 0x7f8cb0148eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:07.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.890+0000 7f8cb5d9a700 1 -- 172.21.15.98:0/1399823142 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb01609b0 msgr2=0x7f8cb00d16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:07.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.890+0000 7f8cb5d9a700 1 --2- 172.21.15.98:0/1399823142 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb01609b0 0x7f8cb00d16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:07.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.890+0000 7f8cb5d9a700 1 -- 172.21.15.98:0/1399823142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca8009a70 con 0x7f8cb00cf570 2023-03-03T08:00:07.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.890+0000 7f8cb659b700 1 --2- 172.21.15.98:0/1399823142 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb00b4aa0 0x7f8cb0148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:07.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.890+0000 7f8cb5d9a700 1 --2- 172.21.15.98:0/1399823142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb00cf570 0x7f8cb01f5910 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f8ca00011e0 tx=0x7f8ca000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:07.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.890+0000 7f8ca6ffd700 1 -- 172.21.15.98:0/1399823142 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0004c80 con 0x7f8cb00cf570 2023-03-03T08:00:07.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.890+0000 7f8ca6ffd700 1 -- 172.21.15.98:0/1399823142 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca0004de0 con 0x7f8cb00cf570 2023-03-03T08:00:07.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.890+0000 7f8ca6ffd700 1 -- 172.21.15.98:0/1399823142 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0007310 con 0x7f8cb00cf570 2023-03-03T08:00:07.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.890+0000 7f8cb759d700 1 -- 172.21.15.98:0/1399823142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb00cda60 con 0x7f8cb00cf570 2023-03-03T08:00:07.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.890+0000 7f8cb759d700 1 -- 172.21.15.98:0/1399823142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb00cd780 con 0x7f8cb00cf570 2023-03-03T08:00:07.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.891+0000 7f8cb759d700 1 -- 172.21.15.98:0/1399823142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb00cdd40 con 0x7f8cb00cf570 2023-03-03T08:00:07.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.893+0000 7f8ca6ffd700 1 -- 172.21.15.98:0/1399823142 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f8ca001a020 con 0x7f8cb00cf570 2023-03-03T08:00:07.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.894+0000 7f8ca6ffd700 1 --2- 172.21.15.98:0/1399823142 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c940683c0 0x7f8c9406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:07.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.894+0000 7f8ca6ffd700 1 -- 172.21.15.98:0/1399823142 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6206+0+0 (secure 0 0 0) 0x7f8ca0085b10 con 0x7f8cb00cf570 2023-03-03T08:00:07.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.894+0000 7f8cb659b700 1 --2- 172.21.15.98:0/1399823142 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c940683c0 0x7f8c9406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:07.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.895+0000 7f8cb659b700 1 --2- 172.21.15.98:0/1399823142 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c940683c0 0x7f8c9406a870 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8cac009b90 tx=0x7f8cac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:07.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:07.898+0000 7f8ca6ffd700 1 -- 172.21.15.98:0/1399823142 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8ca0052120 con 0x7f8cb00cf570 2023-03-03T08:00:08.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.135+0000 7f8cb759d700 1 -- 172.21.15.98:0/1399823142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8cb0068290 con 0x7f8cb00cf570 2023-03-03T08:00:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.585+0000 7f8ca6ffd700 1 -- 172.21.15.98:0/1399823142 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8ca0055740 con 0x7f8cb00cf570 2023-03-03T08:00:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.588+0000 7f8cb759d700 1 -- 172.21.15.98:0/1399823142 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c940683c0 msgr2=0x7f8c9406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.588+0000 7f8cb759d700 1 --2- 172.21.15.98:0/1399823142 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c940683c0 0x7f8c9406a870 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8cac009b90 tx=0x7f8cac008040 comp rx=0 tx=0).stop 2023-03-03T08:00:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.588+0000 7f8cb759d700 1 -- 172.21.15.98:0/1399823142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb00cf570 msgr2=0x7f8cb01f5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.588+0000 7f8cb759d700 1 --2- 172.21.15.98:0/1399823142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb00cf570 0x7f8cb01f5910 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f8ca00011e0 tx=0x7f8ca000ca30 comp rx=0 tx=0).stop 2023-03-03T08:00:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.588+0000 7f8cb759d700 1 -- 172.21.15.98:0/1399823142 shutdown_connections 2023-03-03T08:00:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.588+0000 7f8cb759d700 1 --2- 172.21.15.98:0/1399823142 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8c940683c0 0x7f8c9406a870 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.588+0000 7f8cb759d700 1 --2- 172.21.15.98:0/1399823142 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb01609b0 0x7f8cb00d16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.588+0000 7f8cb759d700 1 --2- 172.21.15.98:0/1399823142 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb00b4aa0 0x7f8cb0148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.588+0000 7f8cb759d700 1 --2- 172.21.15.98:0/1399823142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb00cf570 0x7f8cb01f5910 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.588+0000 7f8cb759d700 1 -- 172.21.15.98:0/1399823142 >> 172.21.15.98:0/1399823142 conn(0x7f8cb006b880 msgr2=0x7f8cb01530b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.589+0000 7f8cb759d700 1 -- 172.21.15.98:0/1399823142 shutdown_connections 2023-03-03T08:00:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.589+0000 7f8cb759d700 1 -- 172.21.15.98:0/1399823142 wait complete. 2023-03-03T08:00:08.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-03-03T08:00:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.868+0000 7f2f823ff700 1 Processor -- start 2023-03-03T08:00:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.868+0000 7f2f823ff700 1 -- start start 2023-03-03T08:00:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.869+0000 7f2f823ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f7c0b8880 0x7f2f7c0cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.869+0000 7f2f823ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f7c0b4ba0 0x7f2f7c0b5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.869+0000 7f2f823ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f7c0d0ac0 con 0x7f2f7c0b5550 2023-03-03T08:00:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.870+0000 7f2f823ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7c0b59a0 con 0x7f2f7c0b8880 2023-03-03T08:00:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.870+0000 7f2f823ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c0cff70 con 0x7f2f7c0b4ba0 2023-03-03T08:00:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.870+0000 7f2f7bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f7c0b8880 0x7f2f7c0cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.870+0000 7f2f7bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f7c0b8880 0x7f2f7c0cd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51068/0 (socket says 172.21.15.98:51068) 2023-03-03T08:00:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.870+0000 7f2f7bfff700 1 -- 172.21.15.98:0/4111311903 learned_addr learned my addr 172.21.15.98:0/4111311903 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.870+0000 7f2f8099c700 1 --1- 172.21.15.98:0/4111311903 >> v1:172.21.15.98:6789/0 conn(0x7f2f7c0b5550 0x7f2f7c1604c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46574/0 (socket says 172.21.15.98:46574) 2023-03-03T08:00:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.870+0000 7f2f7b7fe700 1 --2- 172.21.15.98:0/4111311903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f7c0b4ba0 0x7f2f7c0b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.870+0000 7f2f7bfff700 1 -- 172.21.15.98:0/4111311903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f7c0b4ba0 msgr2=0x7f2f7c0b5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.870+0000 7f2f7bfff700 1 --2- 172.21.15.98:0/4111311903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f7c0b4ba0 0x7f2f7c0b5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.870+0000 7f2f7bfff700 1 -- 172.21.15.98:0/4111311903 >> v1:172.21.15.98:6789/0 conn(0x7f2f7c0b5550 legacy=0x7f2f7c1604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.870+0000 7f2f7bfff700 1 -- 172.21.15.98:0/4111311903 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c0d0250 con 0x7f2f7c0b8880 2023-03-03T08:00:08.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.871+0000 7f2f7bfff700 1 --2- 172.21.15.98:0/4111311903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f7c0b8880 0x7f2f7c0cd770 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0098f0 tx=0x7f2f6c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b48b182760372d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:08.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.871+0000 7f2f7affd700 1 -- 172.21.15.98:0/4111311903 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c00b8a0 con 0x7f2f7c0b8880 2023-03-03T08:00:08.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.871+0000 7f2f7affd700 1 -- 172.21.15.98:0/4111311903 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6c00ba00 con 0x7f2f7c0b8880 2023-03-03T08:00:08.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.871+0000 7f2f7affd700 1 -- 172.21.15.98:0/4111311903 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c00bd90 con 0x7f2f7c0b8880 2023-03-03T08:00:08.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.871+0000 7f2f823ff700 1 -- 172.21.15.98:0/4111311903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f7c0b8880 msgr2=0x7f2f7c0cd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:08.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.871+0000 7f2f823ff700 1 --2- 172.21.15.98:0/4111311903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f7c0b8880 0x7f2f7c0cd770 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0098f0 tx=0x7f2f6c009c00 comp rx=0 tx=0).stop 2023-03-03T08:00:08.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.872+0000 7f2f823ff700 1 -- 172.21.15.98:0/4111311903 shutdown_connections 2023-03-03T08:00:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.872+0000 7f2f823ff700 1 --2- 172.21.15.98:0/4111311903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f7c0b8880 0x7f2f7c0cd770 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.872+0000 7f2f823ff700 1 --2- 172.21.15.98:0/4111311903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f7c0b4ba0 0x7f2f7c0b5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.872+0000 7f2f823ff700 1 -- 172.21.15.98:0/4111311903 >> 172.21.15.98:0/4111311903 conn(0x7f2f7c06ad80 msgr2=0x7f2f7c06b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.872+0000 7f2f823ff700 1 -- 172.21.15.98:0/4111311903 shutdown_connections 2023-03-03T08:00:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.872+0000 7f2f823ff700 1 -- 172.21.15.98:0/4111311903 wait complete. 2023-03-03T08:00:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.874+0000 7f2f823ff700 1 Processor -- start 2023-03-03T08:00:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.874+0000 7f2f823ff700 1 -- start start 2023-03-03T08:00:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.875+0000 7f2f823ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f7c0b4ba0 0x7f2f7c0c9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.875+0000 7f2f823ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f7c0b5550 0x7f2f7c158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.875+0000 7f2f823ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f7c0b8880 0x7f2f7c0d13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.875+0000 7f2f823ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c158010 con 0x7f2f7c0b4ba0 2023-03-03T08:00:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.875+0000 7f2f823ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c157390 con 0x7f2f7c0b5550 2023-03-03T08:00:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.875+0000 7f2f823ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c157670 con 0x7f2f7c0b8880 2023-03-03T08:00:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.875+0000 7f2f7bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f7c0b4ba0 0x7f2f7c0c9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.875+0000 7f2f7bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f7c0b4ba0 0x7f2f7c0c9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51070/0 (socket says 172.21.15.98:51070) 2023-03-03T08:00:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.875+0000 7f2f7bfff700 1 -- 172.21.15.98:0/141191938 learned_addr learned my addr 172.21.15.98:0/141191938 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.875+0000 7f2f7bfff700 1 -- 172.21.15.98:0/141191938 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f7c0b8880 msgr2=0x7f2f7c0d13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.875+0000 7f2f7bfff700 1 --2- 172.21.15.98:0/141191938 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f7c0b8880 0x7f2f7c0d13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.875+0000 7f2f7bfff700 1 -- 172.21.15.98:0/141191938 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f7c0b5550 msgr2=0x7f2f7c158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.875+0000 7f2f7bfff700 1 --2- 172.21.15.98:0/141191938 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f7c0b5550 0x7f2f7c158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.875+0000 7f2f7bfff700 1 -- 172.21.15.98:0/141191938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c009550 con 0x7f2f7c0b4ba0 2023-03-03T08:00:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.876+0000 7f2f7bfff700 1 --2- 172.21.15.98:0/141191938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f7c0b4ba0 0x7f2f7c0c9020 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009c40 tx=0x7f2f6c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.876+0000 7f2f78ff9700 1 -- 172.21.15.98:0/141191938 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c0036a0 con 0x7f2f7c0b4ba0 2023-03-03T08:00:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.876+0000 7f2f78ff9700 1 -- 172.21.15.98:0/141191938 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6c01e070 con 0x7f2f7c0b4ba0 2023-03-03T08:00:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.876+0000 7f2f78ff9700 1 -- 172.21.15.98:0/141191938 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c003a40 con 0x7f2f7c0b4ba0 2023-03-03T08:00:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.876+0000 7f2f823ff700 1 -- 172.21.15.98:0/141191938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7c0d0cf0 con 0x7f2f7c0b4ba0 2023-03-03T08:00:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.876+0000 7f2f823ff700 1 -- 172.21.15.98:0/141191938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7c0d0530 con 0x7f2f7c0b4ba0 2023-03-03T08:00:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.878+0000 7f2f823ff700 1 -- 172.21.15.98:0/141191938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f7c0b8c50 con 0x7f2f7c0b4ba0 2023-03-03T08:00:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.878+0000 7f2f78ff9700 1 -- 172.21.15.98:0/141191938 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f2f6c00f050 con 0x7f2f7c0b4ba0 2023-03-03T08:00:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.879+0000 7f2f78ff9700 1 --2- 172.21.15.98:0/141191938 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2f680683c0 0x7f2f6806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.880+0000 7f2f78ff9700 1 -- 172.21.15.98:0/141191938 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6206+0+0 (secure 0 0 0) 0x7f2f6c020070 con 0x7f2f7c0b4ba0 2023-03-03T08:00:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.880+0000 7f2f7b7fe700 1 --2- 172.21.15.98:0/141191938 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2f680683c0 0x7f2f6806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.881+0000 7f2f7b7fe700 1 --2- 172.21.15.98:0/141191938 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2f680683c0 0x7f2f6806a870 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c159b30 tx=0x7f2f70008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:08.883+0000 7f2f78ff9700 1 -- 172.21.15.98:0/141191938 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f6c053a70 con 0x7f2f7c0b4ba0 2023-03-03T08:00:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:09.076+0000 7f2f823ff700 1 -- 172.21.15.98:0/141191938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2f7c158280 con 0x7f2f7c0b4ba0 2023-03-03T08:00:09.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:09.712+0000 7f2f78ff9700 1 -- 172.21.15.98:0/141191938 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v56) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2f6c057090 con 0x7f2f7c0b4ba0 2023-03-03T08:00:09.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:09.716+0000 7f2f823ff700 1 -- 172.21.15.98:0/141191938 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2f680683c0 msgr2=0x7f2f6806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:09.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:09.716+0000 7f2f823ff700 1 --2- 172.21.15.98:0/141191938 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2f680683c0 0x7f2f6806a870 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c159b30 tx=0x7f2f70008040 comp rx=0 tx=0).stop 2023-03-03T08:00:09.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:09.716+0000 7f2f823ff700 1 -- 172.21.15.98:0/141191938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f7c0b4ba0 msgr2=0x7f2f7c0c9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:09.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:09.716+0000 7f2f823ff700 1 --2- 172.21.15.98:0/141191938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f7c0b4ba0 0x7f2f7c0c9020 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009c40 tx=0x7f2f6c004750 comp rx=0 tx=0).stop 2023-03-03T08:00:09.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:09.716+0000 7f2f823ff700 1 -- 172.21.15.98:0/141191938 shutdown_connections 2023-03-03T08:00:09.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:09.716+0000 7f2f823ff700 1 --2- 172.21.15.98:0/141191938 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2f680683c0 0x7f2f6806a870 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:09.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:09.716+0000 7f2f823ff700 1 --2- 172.21.15.98:0/141191938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f7c0b4ba0 0x7f2f7c0c9020 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:09.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:09.716+0000 7f2f823ff700 1 --2- 172.21.15.98:0/141191938 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f7c0b8880 0x7f2f7c0d13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:09.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:09.716+0000 7f2f823ff700 1 --2- 172.21.15.98:0/141191938 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f7c0b5550 0x7f2f7c158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:09.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:09.716+0000 7f2f823ff700 1 -- 172.21.15.98:0/141191938 >> 172.21.15.98:0/141191938 conn(0x7f2f7c06ad80 msgr2=0x7f2f7c0b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:09.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:09.716+0000 7f2f823ff700 1 -- 172.21.15.98:0/141191938 shutdown_connections 2023-03-03T08:00:09.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:09.717+0000 7f2f823ff700 1 -- 172.21.15.98:0/141191938 wait complete. 2023-03-03T08:00:09.720 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' removed 2023-03-03T08:00:09.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-03-03T08:00:10.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.000+0000 7fd3989ff700 1 Processor -- start 2023-03-03T08:00:10.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.000+0000 7fd3989ff700 1 -- start start 2023-03-03T08:00:10.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.002+0000 7fd3989ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3940b8880 0x7fd3940cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:10.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.002+0000 7fd3989ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3940b4ba0 0x7fd3940b5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:10.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.002+0000 7fd3989ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd3940b5550 0x7fd3941604c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.002+0000 7fd3989ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3940d0ac0 con 0x7fd3940b4ba0 2023-03-03T08:00:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.002+0000 7fd3989ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3940b59a0 con 0x7fd3940b5550 2023-03-03T08:00:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.002+0000 7fd3989ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3940cff70 con 0x7fd3940b8880 2023-03-03T08:00:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.002+0000 7fd391d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3940b4ba0 0x7fd3940b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.002+0000 7fd391d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3940b4ba0 0x7fd3940b5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51086/0 (socket says 172.21.15.98:51086) 2023-03-03T08:00:10.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.002+0000 7fd391d9b700 1 -- 172.21.15.98:0/2712735239 learned_addr learned my addr 172.21.15.98:0/2712735239 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:10.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.002+0000 7fd392d9d700 1 --2- 172.21.15.98:0/2712735239 >> v2:172.21.15.100:3300/0 conn(0x7fd3940b5550 0x7fd3941604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:10.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.002+0000 7fd39259c700 1 --2- 172.21.15.98:0/2712735239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3940b8880 0x7fd3940cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:10.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.003+0000 7fd391d9b700 1 -- 172.21.15.98:0/2712735239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3940b8880 msgr2=0x7fd3940cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:10.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.003+0000 7fd391d9b700 1 --2- 172.21.15.98:0/2712735239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3940b8880 0x7fd3940cd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:10.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.003+0000 7fd391d9b700 1 -- 172.21.15.98:0/2712735239 >> v2:172.21.15.100:3300/0 conn(0x7fd3940b5550 msgr2=0x7fd3941604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:10.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.003+0000 7fd391d9b700 1 --2- 172.21.15.98:0/2712735239 >> v2:172.21.15.100:3300/0 conn(0x7fd3940b5550 0x7fd3941604c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:10.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.003+0000 7fd391d9b700 1 -- 172.21.15.98:0/2712735239 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3940d0250 con 0x7fd3940b4ba0 2023-03-03T08:00:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.003+0000 7fd391d9b700 1 --2- 172.21.15.98:0/2712735239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3940b4ba0 0x7fd3940b5010 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd39408c3f0 tx=0x7fd388009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a69c9cde4d8daa43 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.003+0000 7fd39159a700 1 -- 172.21.15.98:0/2712735239 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd388011830 con 0x7fd3940b4ba0 2023-03-03T08:00:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.003+0000 7fd39159a700 1 -- 172.21.15.98:0/2712735239 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd388011990 con 0x7fd3940b4ba0 2023-03-03T08:00:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.003+0000 7fd39159a700 1 -- 172.21.15.98:0/2712735239 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd388011d20 con 0x7fd3940b4ba0 2023-03-03T08:00:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.004+0000 7fd3989ff700 1 -- 172.21.15.98:0/2712735239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3940b4ba0 msgr2=0x7fd3940b5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.004+0000 7fd3989ff700 1 --2- 172.21.15.98:0/2712735239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3940b4ba0 0x7fd3940b5010 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd39408c3f0 tx=0x7fd388009bd0 comp rx=0 tx=0).stop 2023-03-03T08:00:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.004+0000 7fd3989ff700 1 -- 172.21.15.98:0/2712735239 shutdown_connections 2023-03-03T08:00:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.004+0000 7fd3989ff700 1 --2- 172.21.15.98:0/2712735239 >> v2:172.21.15.100:3300/0 conn(0x7fd3940b5550 0x7fd3941604c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.004+0000 7fd3989ff700 1 --2- 172.21.15.98:0/2712735239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3940b8880 0x7fd3940cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.004+0000 7fd3989ff700 1 --2- 172.21.15.98:0/2712735239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3940b4ba0 0x7fd3940b5010 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.004+0000 7fd3989ff700 1 -- 172.21.15.98:0/2712735239 >> 172.21.15.98:0/2712735239 conn(0x7fd39406ad80 msgr2=0x7fd39406b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.004+0000 7fd3989ff700 1 -- 172.21.15.98:0/2712735239 shutdown_connections 2023-03-03T08:00:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.004+0000 7fd3989ff700 1 -- 172.21.15.98:0/2712735239 wait complete. 2023-03-03T08:00:10.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.006+0000 7fd3989ff700 1 Processor -- start 2023-03-03T08:00:10.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.006+0000 7fd3989ff700 1 -- start start 2023-03-03T08:00:10.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.007+0000 7fd3989ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3940b4ba0 0x7fd3941507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:10.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.007+0000 7fd3989ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3940b5550 0x7fd3940aa440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:10.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.007+0000 7fd3989ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3940b8880 0x7fd3941f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:10.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.007+0000 7fd3989ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3940cbf00 con 0x7fd3940b4ba0 2023-03-03T08:00:10.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.007+0000 7fd3989ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3941f2eb0 con 0x7fd3940b8880 2023-03-03T08:00:10.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.007+0000 7fd3989ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3940a9d90 con 0x7fd3940b5550 2023-03-03T08:00:10.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.008+0000 7fd39259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3940b4ba0 0x7fd3941507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:10.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.008+0000 7fd39259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3940b4ba0 0x7fd3941507a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51094/0 (socket says 172.21.15.98:51094) 2023-03-03T08:00:10.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.008+0000 7fd39259c700 1 -- 172.21.15.98:0/59269475 learned_addr learned my addr 172.21.15.98:0/59269475 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:10.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.008+0000 7fd391d9b700 1 --2- 172.21.15.98:0/59269475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3940b5550 0x7fd3940aa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:10.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.008+0000 7fd392d9d700 1 --2- 172.21.15.98:0/59269475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3940b8880 0x7fd3941f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:10.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.008+0000 7fd39259c700 1 -- 172.21.15.98:0/59269475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3940b5550 msgr2=0x7fd3940aa440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:10.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.008+0000 7fd39259c700 1 --2- 172.21.15.98:0/59269475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3940b5550 0x7fd3940aa440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:10.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.008+0000 7fd39259c700 1 -- 172.21.15.98:0/59269475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3940b8880 msgr2=0x7fd3941f65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:10.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.008+0000 7fd39259c700 1 --2- 172.21.15.98:0/59269475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3940b8880 0x7fd3941f65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:10.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.008+0000 7fd39259c700 1 -- 172.21.15.98:0/59269475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd388009550 con 0x7fd3940b4ba0 2023-03-03T08:00:10.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.009+0000 7fd39259c700 1 --2- 172.21.15.98:0/59269475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3940b4ba0 0x7fd3941507a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd38400e870 tx=0x7fd38400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:10.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.009+0000 7fd382ffd700 1 -- 172.21.15.98:0/59269475 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd384009590 con 0x7fd3940b4ba0 2023-03-03T08:00:10.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.009+0000 7fd382ffd700 1 -- 172.21.15.98:0/59269475 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd384013680 con 0x7fd3940b4ba0 2023-03-03T08:00:10.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.009+0000 7fd382ffd700 1 -- 172.21.15.98:0/59269475 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3840139f0 con 0x7fd3940b4ba0 2023-03-03T08:00:10.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.009+0000 7fd3989ff700 1 -- 172.21.15.98:0/59269475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3940d0cf0 con 0x7fd3940b4ba0 2023-03-03T08:00:10.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.009+0000 7fd3989ff700 1 -- 172.21.15.98:0/59269475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3940d0530 con 0x7fd3940b4ba0 2023-03-03T08:00:10.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.010+0000 7fd3989ff700 1 -- 172.21.15.98:0/59269475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3940b8c50 con 0x7fd3940b4ba0 2023-03-03T08:00:10.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.011+0000 7fd382ffd700 1 -- 172.21.15.98:0/59269475 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fd384023070 con 0x7fd3940b4ba0 2023-03-03T08:00:10.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.013+0000 7fd382ffd700 1 --2- 172.21.15.98:0/59269475 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3740683c0 0x7fd37406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:10.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.013+0000 7fd382ffd700 1 -- 172.21.15.98:0/59269475 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5768+0+0 (secure 0 0 0) 0x7fd384085da0 con 0x7fd3940b4ba0 2023-03-03T08:00:10.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.013+0000 7fd391d9b700 1 --2- 172.21.15.98:0/59269475 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3740683c0 0x7fd37406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:10.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.014+0000 7fd391d9b700 1 --2- 172.21.15.98:0/59269475 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3740683c0 0x7fd37406a870 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd388014040 tx=0x7fd388008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:10.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.016+0000 7fd382ffd700 1 -- 172.21.15.98:0/59269475 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd384052500 con 0x7fd3940b4ba0 2023-03-03T08:00:10.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.211+0000 7fd3989ff700 1 -- 172.21.15.98:0/59269475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd3940cc1e0 con 0x7fd3940b4ba0 2023-03-03T08:00:10.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.721+0000 7fd382ffd700 1 -- 172.21.15.98:0/59269475 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v57) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd384055b20 con 0x7fd3940b4ba0 2023-03-03T08:00:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.724+0000 7fd3989ff700 1 -- 172.21.15.98:0/59269475 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3740683c0 msgr2=0x7fd37406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.724+0000 7fd3989ff700 1 --2- 172.21.15.98:0/59269475 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3740683c0 0x7fd37406a870 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd388014040 tx=0x7fd388008030 comp rx=0 tx=0).stop 2023-03-03T08:00:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.725+0000 7fd3989ff700 1 -- 172.21.15.98:0/59269475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3940b4ba0 msgr2=0x7fd3941507a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.725+0000 7fd3989ff700 1 --2- 172.21.15.98:0/59269475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3940b4ba0 0x7fd3941507a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd38400e870 tx=0x7fd38400eb80 comp rx=0 tx=0).stop 2023-03-03T08:00:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.725+0000 7fd3989ff700 1 -- 172.21.15.98:0/59269475 shutdown_connections 2023-03-03T08:00:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.725+0000 7fd3989ff700 1 --2- 172.21.15.98:0/59269475 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3740683c0 0x7fd37406a870 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.725+0000 7fd3989ff700 1 --2- 172.21.15.98:0/59269475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3940b4ba0 0x7fd3941507a0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.725+0000 7fd3989ff700 1 --2- 172.21.15.98:0/59269475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3940b8880 0x7fd3941f65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.725+0000 7fd3989ff700 1 --2- 172.21.15.98:0/59269475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3940b5550 0x7fd3940aa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.725+0000 7fd3989ff700 1 -- 172.21.15.98:0/59269475 >> 172.21.15.98:0/59269475 conn(0x7fd39406ad80 msgr2=0x7fd3940b33a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:10.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.725+0000 7fd3989ff700 1 -- 172.21.15.98:0/59269475 shutdown_connections 2023-03-03T08:00:10.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:10.725+0000 7fd3989ff700 1 -- 172.21.15.98:0/59269475 wait complete. 2023-03-03T08:00:10.728 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' removed 2023-03-03T08:00:10.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-03-03T08:00:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.005+0000 7fba56f9d700 1 Processor -- start 2023-03-03T08:00:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.006+0000 7fba56f9d700 1 -- start start 2023-03-03T08:00:11.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.007+0000 7fba56f9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fba501609b0 0x7fba50160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:11.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.007+0000 7fba56f9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fba500b4aa0 0x7fba500b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:11.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.007+0000 7fba56f9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba501612c0 con 0x7fba500cf570 2023-03-03T08:00:11.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.007+0000 7fba56f9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba500cebb0 con 0x7fba500b4aa0 2023-03-03T08:00:11.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.007+0000 7fba56f9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba500b5460 con 0x7fba501609b0 2023-03-03T08:00:11.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.007+0000 7fba5679c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fba500b4aa0 0x7fba500b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:11.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.007+0000 7fba5679c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fba500b4aa0 0x7fba500b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51096/0 (socket says 172.21.15.98:51096) 2023-03-03T08:00:11.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.007+0000 7fba5679c700 1 -- 172.21.15.98:0/3048331585 learned_addr learned my addr 172.21.15.98:0/3048331585 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:11.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.008+0000 7fba55f9b700 1 --2- 172.21.15.98:0/3048331585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fba501609b0 0x7fba50160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:11.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.008+0000 7fba5679c700 1 -- 172.21.15.98:0/3048331585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fba501609b0 msgr2=0x7fba50160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:11.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.008+0000 7fba5679c700 1 --2- 172.21.15.98:0/3048331585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fba501609b0 0x7fba50160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:11.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.008+0000 7fba5679c700 1 -- 172.21.15.98:0/3048331585 >> v1:172.21.15.98:6789/0 conn(0x7fba500cf570 legacy=0x7fba5015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.008+0000 7fba5679c700 1 -- 172.21.15.98:0/3048331585 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5015dd00 con 0x7fba500b4aa0 2023-03-03T08:00:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.008+0000 7fba5679c700 1 --2- 172.21.15.98:0/3048331585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba500b4aa0 0x7fba500b4f20 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fba48009e50 tx=0x7fba4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785d10a9ea6f92d9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.008+0000 7fba54f99700 1 -- 172.21.15.98:0/3048331585 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4800e070 con 0x7fba500b4aa0 2023-03-03T08:00:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.009+0000 7fba54f99700 1 -- 172.21.15.98:0/3048331585 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4800b9c0 con 0x7fba500b4aa0 2023-03-03T08:00:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.009+0000 7fba54f99700 1 -- 172.21.15.98:0/3048331585 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4800bd70 con 0x7fba500b4aa0 2023-03-03T08:00:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.009+0000 7fba56f9d700 1 -- 172.21.15.98:0/3048331585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba500b4aa0 msgr2=0x7fba500b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.009+0000 7fba56f9d700 1 --2- 172.21.15.98:0/3048331585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba500b4aa0 0x7fba500b4f20 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fba48009e50 tx=0x7fba4800b330 comp rx=0 tx=0).stop 2023-03-03T08:00:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.009+0000 7fba56f9d700 1 -- 172.21.15.98:0/3048331585 shutdown_connections 2023-03-03T08:00:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.009+0000 7fba56f9d700 1 --2- 172.21.15.98:0/3048331585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba500b4aa0 0x7fba500b4f20 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.009+0000 7fba56f9d700 1 --2- 172.21.15.98:0/3048331585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fba501609b0 0x7fba50160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:11.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.009+0000 7fba56f9d700 1 -- 172.21.15.98:0/3048331585 >> 172.21.15.98:0/3048331585 conn(0x7fba5006b880 msgr2=0x7fba500cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:11.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.009+0000 7fba56f9d700 1 -- 172.21.15.98:0/3048331585 shutdown_connections 2023-03-03T08:00:11.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.009+0000 7fba56f9d700 1 -- 172.21.15.98:0/3048331585 wait complete. 2023-03-03T08:00:11.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.011+0000 7fba56f9d700 1 Processor -- start 2023-03-03T08:00:11.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.011+0000 7fba56f9d700 1 -- start start 2023-03-03T08:00:11.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.012+0000 7fba56f9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba500b4aa0 0x7fba501f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:11.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.012+0000 7fba56f9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fba500cf570 0x7fba500cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:11.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.012+0000 7fba56f9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba501609b0 0x7fba50151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:11.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.013+0000 7fba56f9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba501f2480 con 0x7fba501609b0 2023-03-03T08:00:11.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.013+0000 7fba56f9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba500cd010 con 0x7fba500b4aa0 2023-03-03T08:00:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.013+0000 7fba56f9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba500cca50 con 0x7fba500cf570 2023-03-03T08:00:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.013+0000 7fba5679c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba501609b0 0x7fba50151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.013+0000 7fba5679c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba501609b0 0x7fba50151fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51108/0 (socket says 172.21.15.98:51108) 2023-03-03T08:00:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.013+0000 7fba5679c700 1 -- 172.21.15.98:0/1900884014 learned_addr learned my addr 172.21.15.98:0/1900884014 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.013+0000 7fba5579a700 1 --2- 172.21.15.98:0/1900884014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fba500cf570 0x7fba500cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.013+0000 7fba55f9b700 1 --2- 172.21.15.98:0/1900884014 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba500b4aa0 0x7fba501f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.013+0000 7fba5679c700 1 -- 172.21.15.98:0/1900884014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fba500cf570 msgr2=0x7fba500cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.013+0000 7fba5679c700 1 --2- 172.21.15.98:0/1900884014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fba500cf570 0x7fba500cbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.013+0000 7fba5679c700 1 -- 172.21.15.98:0/1900884014 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba500b4aa0 msgr2=0x7fba501f1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.013+0000 7fba5679c700 1 --2- 172.21.15.98:0/1900884014 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba500b4aa0 0x7fba501f1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.013+0000 7fba5679c700 1 -- 172.21.15.98:0/1900884014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba48009b30 con 0x7fba501609b0 2023-03-03T08:00:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.013+0000 7fba55f9b700 1 --2- 172.21.15.98:0/1900884014 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba500b4aa0 0x7fba501f1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.014+0000 7fba5679c700 1 --2- 172.21.15.98:0/1900884014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba501609b0 0x7fba50151fa0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fba50074380 tx=0x7fba48004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.014+0000 7fba467fc700 1 -- 172.21.15.98:0/1900884014 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba480048b0 con 0x7fba501609b0 2023-03-03T08:00:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.014+0000 7fba467fc700 1 -- 172.21.15.98:0/1900884014 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba480092e0 con 0x7fba501609b0 2023-03-03T08:00:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.014+0000 7fba56f9d700 1 -- 172.21.15.98:0/1900884014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba500cda60 con 0x7fba501609b0 2023-03-03T08:00:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.014+0000 7fba467fc700 1 -- 172.21.15.98:0/1900884014 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4801c680 con 0x7fba501609b0 2023-03-03T08:00:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.014+0000 7fba56f9d700 1 -- 172.21.15.98:0/1900884014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba500cd780 con 0x7fba501609b0 2023-03-03T08:00:11.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.016+0000 7fba56f9d700 1 -- 172.21.15.98:0/1900884014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba500cdd40 con 0x7fba501609b0 2023-03-03T08:00:11.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.017+0000 7fba467fc700 1 -- 172.21.15.98:0/1900884014 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fba48022030 con 0x7fba501609b0 2023-03-03T08:00:11.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.018+0000 7fba467fc700 1 --2- 172.21.15.98:0/1900884014 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fba340683c0 0x7fba3406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:11.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.019+0000 7fba467fc700 1 -- 172.21.15.98:0/1900884014 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fba480872c0 con 0x7fba501609b0 2023-03-03T08:00:11.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.019+0000 7fba55f9b700 1 --2- 172.21.15.98:0/1900884014 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fba340683c0 0x7fba3406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:11.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.020+0000 7fba55f9b700 1 --2- 172.21.15.98:0/1900884014 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fba340683c0 0x7fba3406a870 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fba4c007a50 tx=0x7fba4c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:11.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.023+0000 7fba467fc700 1 -- 172.21.15.98:0/1900884014 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fba48053d50 con 0x7fba501609b0 2023-03-03T08:00:11.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.216+0000 7fba56f9d700 1 -- 172.21.15.98:0/1900884014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fba50068290 con 0x7fba501609b0 2023-03-03T08:00:11.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.221+0000 7fba467fc700 1 -- 172.21.15.98:0/1900884014 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7fba48057370 con 0x7fba501609b0 2023-03-03T08:00:11.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.223+0000 7fba56f9d700 1 -- 172.21.15.98:0/1900884014 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fba340683c0 msgr2=0x7fba3406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:11.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.223+0000 7fba56f9d700 1 --2- 172.21.15.98:0/1900884014 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fba340683c0 0x7fba3406a870 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fba4c007a50 tx=0x7fba4c008040 comp rx=0 tx=0).stop 2023-03-03T08:00:11.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.223+0000 7fba56f9d700 1 -- 172.21.15.98:0/1900884014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba501609b0 msgr2=0x7fba50151fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:11.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.223+0000 7fba56f9d700 1 --2- 172.21.15.98:0/1900884014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba501609b0 0x7fba50151fa0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fba50074380 tx=0x7fba48004ad0 comp rx=0 tx=0).stop 2023-03-03T08:00:11.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.224+0000 7fba56f9d700 1 -- 172.21.15.98:0/1900884014 shutdown_connections 2023-03-03T08:00:11.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.224+0000 7fba56f9d700 1 --2- 172.21.15.98:0/1900884014 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fba340683c0 0x7fba3406a870 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:11.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.224+0000 7fba56f9d700 1 --2- 172.21.15.98:0/1900884014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba501609b0 0x7fba50151fa0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:11.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.224+0000 7fba56f9d700 1 --2- 172.21.15.98:0/1900884014 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba500b4aa0 0x7fba501f1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:11.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.224+0000 7fba56f9d700 1 --2- 172.21.15.98:0/1900884014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fba500cf570 0x7fba500cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:11.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.224+0000 7fba56f9d700 1 -- 172.21.15.98:0/1900884014 >> 172.21.15.98:0/1900884014 conn(0x7fba5006b880 msgr2=0x7fba500d23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:11.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.224+0000 7fba56f9d700 1 -- 172.21.15.98:0/1900884014 shutdown_connections 2023-03-03T08:00:11.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.224+0000 7fba56f9d700 1 -- 172.21.15.98:0/1900884014 wait complete. 2023-03-03T08:00:11.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration' 2023-03-03T08:00:11.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.499+0000 7f7f64f9d700 1 Processor -- start 2023-03-03T08:00:11.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.500+0000 7f7f64f9d700 1 -- start start 2023-03-03T08:00:11.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.501+0000 7f7f64f9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f60146360 0x7f7f60160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:11.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.501+0000 7f7f64f9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7f600d2590 0x7f7f600b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:11.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.501+0000 7f7f64f9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f601614c0 con 0x7f7f600b4ab0 2023-03-03T08:00:11.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.502+0000 7f7f64f9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f600b5460 con 0x7f7f600d2590 2023-03-03T08:00:11.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.502+0000 7f7f64f9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f600d10f0 con 0x7f7f60146360 2023-03-03T08:00:11.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.502+0000 7f7f5effd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7f600b4ab0 0x7f7f600731e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46590/0 (socket says 172.21.15.98:46590) 2023-03-03T08:00:11.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.502+0000 7f7f5effd700 1 -- 172.21.15.98:0/700271070 learned_addr learned my addr 172.21.15.98:0/700271070 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.502+0000 7f7f5ffff700 1 --2- 172.21.15.98:0/700271070 >> v2:172.21.15.98:3300/0 conn(0x7f7f600d2590 0x7f7f600b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.502+0000 7f7f5f7fe700 1 --2- 172.21.15.98:0/700271070 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f60146360 0x7f7f60160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.502+0000 7f7f5e7fc700 1 -- 172.21.15.98:0/700271070 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358871738 0 0) 0x7f7f54001110 con 0x7f7f600b4ab0 2023-03-03T08:00:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.502+0000 7f7f5e7fc700 1 -- 172.21.15.98:0/700271070 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f480035b0 con 0x7f7f600b4ab0 2023-03-03T08:00:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.502+0000 7f7f5ffff700 1 -- 172.21.15.98:0/700271070 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f60146360 msgr2=0x7f7f60160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.502+0000 7f7f5ffff700 1 --2- 172.21.15.98:0/700271070 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f60146360 0x7f7f60160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.502+0000 7f7f5ffff700 1 -- 172.21.15.98:0/700271070 >> v1:172.21.15.98:6789/0 conn(0x7f7f600b4ab0 legacy=0x7f7f600731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.502+0000 7f7f5ffff700 1 -- 172.21.15.98:0/700271070 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f600cdf20 con 0x7f7f600d2590 2023-03-03T08:00:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.503+0000 7f7f5ffff700 1 --2- 172.21.15.98:0/700271070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f600d2590 0x7f7f600b4f20 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7f58009d90 tx=0x7f7f5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d0368f021d5ba7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.503+0000 7f7f5e7fc700 1 -- 172.21.15.98:0/700271070 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5800e070 con 0x7f7f600d2590 2023-03-03T08:00:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.503+0000 7f7f5e7fc700 1 -- 172.21.15.98:0/700271070 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5800bb20 con 0x7f7f600d2590 2023-03-03T08:00:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.503+0000 7f7f5e7fc700 1 -- 172.21.15.98:0/700271070 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f58012430 con 0x7f7f600d2590 2023-03-03T08:00:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.504+0000 7f7f64f9d700 1 -- 172.21.15.98:0/700271070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f600d2590 msgr2=0x7f7f600b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.504+0000 7f7f64f9d700 1 --2- 172.21.15.98:0/700271070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f600d2590 0x7f7f600b4f20 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7f58009d90 tx=0x7f7f5800b330 comp rx=0 tx=0).stop 2023-03-03T08:00:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.504+0000 7f7f64f9d700 1 -- 172.21.15.98:0/700271070 shutdown_connections 2023-03-03T08:00:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.504+0000 7f7f64f9d700 1 --2- 172.21.15.98:0/700271070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f600d2590 0x7f7f600b4f20 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.504+0000 7f7f64f9d700 1 --2- 172.21.15.98:0/700271070 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f60146360 0x7f7f60160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.504+0000 7f7f64f9d700 1 -- 172.21.15.98:0/700271070 >> 172.21.15.98:0/700271070 conn(0x7f7f6006b8e0 msgr2=0x7f7f600cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.504+0000 7f7f64f9d700 1 -- 172.21.15.98:0/700271070 shutdown_connections 2023-03-03T08:00:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.504+0000 7f7f64f9d700 1 -- 172.21.15.98:0/700271070 wait complete. 2023-03-03T08:00:11.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.506+0000 7f7f64f9d700 1 Processor -- start 2023-03-03T08:00:11.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.506+0000 7f7f64f9d700 1 -- start start 2023-03-03T08:00:11.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.507+0000 7f7f64f9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f600b4ab0 0x7f7f600c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:11.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.507+0000 7f7f64f9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f600d2590 0x7f7f600c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:11.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.507+0000 7f7f64f9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f60146360 0x7f7f601f1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:11.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.507+0000 7f7f64f9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f600c8290 con 0x7f7f600d2590 2023-03-03T08:00:11.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.507+0000 7f7f64f9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f600c9130 con 0x7f7f600b4ab0 2023-03-03T08:00:11.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.507+0000 7f7f64f9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f600c7610 con 0x7f7f60146360 2023-03-03T08:00:11.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.507+0000 7f7f5effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f600d2590 0x7f7f600c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:11.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.507+0000 7f7f5effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f600d2590 0x7f7f600c8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51118/0 (socket says 172.21.15.98:51118) 2023-03-03T08:00:11.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.507+0000 7f7f5effd700 1 -- 172.21.15.98:0/2061584359 learned_addr learned my addr 172.21.15.98:0/2061584359 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:11.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.508+0000 7f7f5f7fe700 1 --2- 172.21.15.98:0/2061584359 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f600b4ab0 0x7f7f600c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:11.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.508+0000 7f7f5ffff700 1 --2- 172.21.15.98:0/2061584359 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f60146360 0x7f7f601f1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:11.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.508+0000 7f7f5effd700 1 -- 172.21.15.98:0/2061584359 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f60146360 msgr2=0x7f7f601f1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:11.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.508+0000 7f7f5effd700 1 --2- 172.21.15.98:0/2061584359 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f60146360 0x7f7f601f1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:11.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.508+0000 7f7f5effd700 1 -- 172.21.15.98:0/2061584359 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f600b4ab0 msgr2=0x7f7f600c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:11.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.508+0000 7f7f5effd700 1 --2- 172.21.15.98:0/2061584359 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f600b4ab0 0x7f7f600c86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:11.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.508+0000 7f7f5effd700 1 -- 172.21.15.98:0/2061584359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f58009a70 con 0x7f7f600d2590 2023-03-03T08:00:11.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.508+0000 7f7f5effd700 1 --2- 172.21.15.98:0/2061584359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f600d2590 0x7f7f600c8bf0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f7f54007fd0 tx=0x7f7f5400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:11.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.508+0000 7f7f47fff700 1 -- 172.21.15.98:0/2061584359 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f54005d70 con 0x7f7f600d2590 2023-03-03T08:00:11.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.508+0000 7f7f47fff700 1 -- 172.21.15.98:0/2061584359 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f540070d0 con 0x7f7f600d2590 2023-03-03T08:00:11.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.508+0000 7f7f47fff700 1 -- 172.21.15.98:0/2061584359 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f54007460 con 0x7f7f600d2590 2023-03-03T08:00:11.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.508+0000 7f7f64f9d700 1 -- 172.21.15.98:0/2061584359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f600cd960 con 0x7f7f600d2590 2023-03-03T08:00:11.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.509+0000 7f7f64f9d700 1 -- 172.21.15.98:0/2061584359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f600cd680 con 0x7f7f600d2590 2023-03-03T08:00:11.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.510+0000 7f7f64f9d700 1 -- 172.21.15.98:0/2061584359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f600cdc40 con 0x7f7f600d2590 2023-03-03T08:00:11.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.511+0000 7f7f47fff700 1 -- 172.21.15.98:0/2061584359 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f7f54019030 con 0x7f7f600d2590 2023-03-03T08:00:11.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.513+0000 7f7f47fff700 1 --2- 172.21.15.98:0/2061584359 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7f4c0683c0 0x7f7f4c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.513+0000 7f7f47fff700 1 -- 172.21.15.98:0/2061584359 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f7f54084e70 con 0x7f7f600d2590 2023-03-03T08:00:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.513+0000 7f7f5f7fe700 1 --2- 172.21.15.98:0/2061584359 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7f4c0683c0 0x7f7f4c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.514+0000 7f7f5f7fe700 1 --2- 172.21.15.98:0/2061584359 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7f4c0683c0 0x7f7f4c06a870 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7f5000b9b0 tx=0x7f7f50008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:11.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.517+0000 7f7f47fff700 1 -- 172.21.15.98:0/2061584359 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7f540517a0 con 0x7f7f600d2590 2023-03-03T08:00:11.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:11.784+0000 7f7f64f9d700 1 -- 172.21.15.98:0/2061584359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]} v 0) v1 -- 0x7f7f601614c0 con 0x7f7f600d2590 2023-03-03T08:00:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:12.729+0000 7f7f47fff700 1 -- 172.21.15.98:0/2061584359 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]}]=0 v247) v1 ==== 140+0+0 (secure 0 0 0) 0x7f7f601614c0 con 0x7f7f600d2590 2023-03-03T08:00:12.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:12.732+0000 7f7f64f9d700 1 -- 172.21.15.98:0/2061584359 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7f4c0683c0 msgr2=0x7f7f4c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:12.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:12.732+0000 7f7f64f9d700 1 --2- 172.21.15.98:0/2061584359 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7f4c0683c0 0x7f7f4c06a870 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7f5000b9b0 tx=0x7f7f50008040 comp rx=0 tx=0).stop 2023-03-03T08:00:12.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:12.732+0000 7f7f64f9d700 1 -- 172.21.15.98:0/2061584359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f600d2590 msgr2=0x7f7f600c8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:12.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:12.732+0000 7f7f64f9d700 1 --2- 172.21.15.98:0/2061584359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f600d2590 0x7f7f600c8bf0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f7f54007fd0 tx=0x7f7f5400c860 comp rx=0 tx=0).stop 2023-03-03T08:00:12.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:12.733+0000 7f7f64f9d700 1 -- 172.21.15.98:0/2061584359 shutdown_connections 2023-03-03T08:00:12.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:12.733+0000 7f7f64f9d700 1 --2- 172.21.15.98:0/2061584359 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7f4c0683c0 0x7f7f4c06a870 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:12.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:12.733+0000 7f7f64f9d700 1 --2- 172.21.15.98:0/2061584359 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f60146360 0x7f7f601f1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:12.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:12.733+0000 7f7f64f9d700 1 --2- 172.21.15.98:0/2061584359 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f600b4ab0 0x7f7f600c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:12.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:12.733+0000 7f7f64f9d700 1 --2- 172.21.15.98:0/2061584359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f600d2590 0x7f7f600c8bf0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:12.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:12.733+0000 7f7f64f9d700 1 -- 172.21.15.98:0/2061584359 >> 172.21.15.98:0/2061584359 conn(0x7f7f6006b8e0 msgr2=0x7f7f600b3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:12.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:12.733+0000 7f7f64f9d700 1 -- 172.21.15.98:0/2061584359 shutdown_connections 2023-03-03T08:00:12.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:12.733+0000 7f7f64f9d700 1 -- 172.21.15.98:0/2061584359 wait complete. 2023-03-03T08:00:12.763 INFO:tasks.cephfs_test_runner:test_connection_expiration (tasks.cephfs.test_volumes.TestMisc) ... ok 2023-03-03T08:00:12.763 INFO:tasks.cephfs_test_runner:Starting test: test_malicious_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) 2023-03-03T08:00:12.764 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade' 2023-03-03T08:00:13.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.008+0000 7f3ca59ff700 1 Processor -- start 2023-03-03T08:00:13.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.008+0000 7f3ca59ff700 1 -- start start 2023-03-03T08:00:13.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.010+0000 7f3ca59ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ca00b8b90 0x7f3ca00cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:13.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.010+0000 7f3ca59ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca00d0ef0 con 0x7f3ca00b87c0 2023-03-03T08:00:13.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.010+0000 7f3ca59ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca00d0600 con 0x7f3ca00b5410 2023-03-03T08:00:13.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.010+0000 7f3ca59ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca00cd970 con 0x7f3ca00b8b90 2023-03-03T08:00:13.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.010+0000 7f3c9effd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3ca00b87c0 0x7f3ca00b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46596/0 (socket says 172.21.15.98:46596) 2023-03-03T08:00:13.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.010+0000 7f3c9effd700 1 -- 172.21.15.98:0/3877299406 learned_addr learned my addr 172.21.15.98:0/3877299406 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:13.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.010+0000 7f3c9e7fc700 1 --2- 172.21.15.98:0/3877299406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ca00b8b90 0x7f3ca00cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:13.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.010+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25492367 0 0) 0x7f3ca00d0ef0 con 0x7f3ca00b87c0 2023-03-03T08:00:13.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c880035b0 con 0x7f3ca00b87c0 2023-03-03T08:00:13.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125163205 0 0) 0x7f3ca00d0600 con 0x7f3ca00b5410 2023-03-03T08:00:13.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ca00d0ef0 con 0x7f3ca00b5410 2023-03-03T08:00:13.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1442713890 0 0) 0x7f3c880035b0 con 0x7f3ca00b87c0 2023-03-03T08:00:13.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ca00b8b90 msgr2=0x7f3ca00cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:13.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 --2- 172.21.15.98:0/3877299406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ca00b8b90 0x7f3ca00cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:13.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 >> v1:172.21.15.100:6789/0 conn(0x7f3ca00b5410 legacy=0x7f3ca00d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:13.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca00cdc50 con 0x7f3ca00b87c0 2023-03-03T08:00:13.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f3c90001670 con 0x7f3ca00b87c0 2023-03-03T08:00:13.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 >> v1:172.21.15.98:6789/0 conn(0x7f3ca00b87c0 legacy=0x7f3ca00b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:13.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 --2- 172.21.15.98:0/3877299406 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c88003bc0 0x7f3c88003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:13.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 --2- 172.21.15.98:0/3877299406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ca00b8b90 0x7f3c88004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:13.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 --2- 172.21.15.98:0/3877299406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c88004d90 0x7f3c88005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:13.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca00cd970 con 0x7f3c88004d90 2023-03-03T08:00:13.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8800bac0 con 0x7f3c88003bc0 2023-03-03T08:00:13.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8800bc80 con 0x7f3ca00b8b90 2023-03-03T08:00:13.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.011+0000 7f3c9f7fe700 1 --2- 172.21.15.98:0/3877299406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c88004d90 0x7f3c88005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:13.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.012+0000 7f3c9e7fc700 1 --2- 172.21.15.98:0/3877299406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ca00b8b90 0x7f3c88004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:13.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.012+0000 7f3c9effd700 1 --2- 172.21.15.98:0/3877299406 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c88003bc0 0x7f3c88003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:13.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.012+0000 7f3c9e7fc700 1 -- 172.21.15.98:0/3877299406 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c88003bc0 msgr2=0x7f3c88003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:13.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.012+0000 7f3c9e7fc700 1 --2- 172.21.15.98:0/3877299406 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c88003bc0 0x7f3c88003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:13.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.012+0000 7f3c9e7fc700 1 -- 172.21.15.98:0/3877299406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c88004d90 msgr2=0x7f3c88005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:13.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.012+0000 7f3c9e7fc700 1 --2- 172.21.15.98:0/3877299406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c88004d90 0x7f3c88005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:13.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.012+0000 7f3c9e7fc700 1 -- 172.21.15.98:0/3877299406 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c880035b0 con 0x7f3ca00b8b90 2023-03-03T08:00:13.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.012+0000 7f3c9f7fe700 1 --2- 172.21.15.98:0/3877299406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c88004d90 0x7f3c88005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:00:13.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.012+0000 7f3c9e7fc700 1 --2- 172.21.15.98:0/3877299406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ca00b8b90 0x7f3c88004850 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009b60 tx=0x7f3c88005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32db985c927217b4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:13.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.013+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9400e6c0 con 0x7f3ca00b8b90 2023-03-03T08:00:13.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.013+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9400e820 con 0x7f3ca00b8b90 2023-03-03T08:00:13.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.013+0000 7f3c9dffb700 1 -- 172.21.15.98:0/3877299406 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9400eb90 con 0x7f3ca00b8b90 2023-03-03T08:00:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.013+0000 7f3ca59ff700 1 -- 172.21.15.98:0/3877299406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ca00b8b90 msgr2=0x7f3c88004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.013+0000 7f3ca59ff700 1 --2- 172.21.15.98:0/3877299406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ca00b8b90 0x7f3c88004850 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009b60 tx=0x7f3c88005880 comp rx=0 tx=0).stop 2023-03-03T08:00:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.013+0000 7f3c9f7fe700 1 -- 172.21.15.98:0/3877299406 reap_dead start 2023-03-03T08:00:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.013+0000 7f3ca59ff700 1 -- 172.21.15.98:0/3877299406 shutdown_connections 2023-03-03T08:00:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.013+0000 7f3ca59ff700 1 -- 172.21.15.98:0/3877299406 >> 172.21.15.98:0/3877299406 conn(0x7f3ca006aee0 msgr2=0x7f3ca006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.013+0000 7f3ca59ff700 1 -- 172.21.15.98:0/3877299406 shutdown_connections 2023-03-03T08:00:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.014+0000 7f3ca59ff700 1 -- 172.21.15.98:0/3877299406 wait complete. 2023-03-03T08:00:13.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.016+0000 7f3ca59ff700 1 Processor -- start 2023-03-03T08:00:13.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.016+0000 7f3ca59ff700 1 -- start start 2023-03-03T08:00:13.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.017+0000 7f3ca59ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ca00b8b90 0x7f3ca015f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:13.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.017+0000 7f3ca59ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ca00ca350 0x7f3ca00ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:13.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.017+0000 7f3ca59ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ca01435d0 0x7f3ca0143a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:13.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.017+0000 7f3ca59ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca00cad00 con 0x7f3ca00b8b90 2023-03-03T08:00:13.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.017+0000 7f3ca59ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca015e100 con 0x7f3ca01435d0 2023-03-03T08:00:13.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.017+0000 7f3ca59ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca01f2390 con 0x7f3ca00ca350 2023-03-03T08:00:13.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.017+0000 7f3c9effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ca00b8b90 0x7f3ca015f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:13.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.017+0000 7f3c9effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ca00b8b90 0x7f3ca015f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51132/0 (socket says 172.21.15.98:51132) 2023-03-03T08:00:13.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.017+0000 7f3c9effd700 1 -- 172.21.15.98:0/4020944888 learned_addr learned my addr 172.21.15.98:0/4020944888 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:13.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.018+0000 7f3c9e7fc700 1 --2- 172.21.15.98:0/4020944888 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ca00ca350 0x7f3ca00ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:13.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.018+0000 7f3c9f7fe700 1 --2- 172.21.15.98:0/4020944888 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ca01435d0 0x7f3ca0143a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:13.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.018+0000 7f3c9effd700 1 -- 172.21.15.98:0/4020944888 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ca00ca350 msgr2=0x7f3ca00ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:13.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.018+0000 7f3c9effd700 1 --2- 172.21.15.98:0/4020944888 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ca00ca350 0x7f3ca00ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:13.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.018+0000 7f3c9effd700 1 -- 172.21.15.98:0/4020944888 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ca01435d0 msgr2=0x7f3ca0143a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:13.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.018+0000 7f3c9effd700 1 --2- 172.21.15.98:0/4020944888 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ca01435d0 0x7f3ca0143a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:13.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.018+0000 7f3c9effd700 1 -- 172.21.15.98:0/4020944888 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca00cdf30 con 0x7f3ca00b8b90 2023-03-03T08:00:13.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.018+0000 7f3c9effd700 1 --2- 172.21.15.98:0/4020944888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ca00b8b90 0x7f3ca015f3f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f3c900072c0 tx=0x7f3c900075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:13.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.018+0000 7f3c87fff700 1 -- 172.21.15.98:0/4020944888 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9000f6c0 con 0x7f3ca00b8b90 2023-03-03T08:00:13.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.018+0000 7f3c87fff700 1 -- 172.21.15.98:0/4020944888 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9000f820 con 0x7f3ca00b8b90 2023-03-03T08:00:13.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.019+0000 7f3ca59ff700 1 -- 172.21.15.98:0/4020944888 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca0073a80 con 0x7f3ca00b8b90 2023-03-03T08:00:13.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.019+0000 7f3c87fff700 1 -- 172.21.15.98:0/4020944888 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9000fbd0 con 0x7f3ca00b8b90 2023-03-03T08:00:13.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.019+0000 7f3ca59ff700 1 -- 172.21.15.98:0/4020944888 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca00ce0c0 con 0x7f3ca00b8b90 2023-03-03T08:00:13.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.020+0000 7f3ca59ff700 1 -- 172.21.15.98:0/4020944888 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca00729b0 con 0x7f3ca00b8b90 2023-03-03T08:00:13.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.021+0000 7f3c87fff700 1 -- 172.21.15.98:0/4020944888 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f3c9000fd30 con 0x7f3ca00b8b90 2023-03-03T08:00:13.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.022+0000 7f3c87fff700 1 --2- 172.21.15.98:0/4020944888 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3c8c0683c0 0x7f3c8c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:13.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.022+0000 7f3c87fff700 1 -- 172.21.15.98:0/4020944888 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f3c90085c40 con 0x7f3ca00b8b90 2023-03-03T08:00:13.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.023+0000 7f3c9e7fc700 1 --2- 172.21.15.98:0/4020944888 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3c8c0683c0 0x7f3c8c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.023+0000 7f3c9e7fc700 1 --2- 172.21.15.98:0/4020944888 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3c8c0683c0 0x7f3c8c06a870 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3c940090b0 tx=0x7f3c94008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.027+0000 7f3c87fff700 1 -- 172.21.15.98:0/4020944888 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3c90052600 con 0x7f3ca00b8b90 2023-03-03T08:00:13.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.296+0000 7f3ca59ff700 1 -- 172.21.15.98:0/4020944888 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]} v 0) v1 -- 0x7f3ca004e6e0 con 0x7f3ca00b8b90 2023-03-03T08:00:13.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.734+0000 7f3c87fff700 1 -- 172.21.15.98:0/4020944888 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]}]=0 v248) v1 ==== 164+0+0 (secure 0 0 0) 0x7f3c90055c20 con 0x7f3ca00b8b90 2023-03-03T08:00:13.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.737+0000 7f3ca59ff700 1 -- 172.21.15.98:0/4020944888 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3c8c0683c0 msgr2=0x7f3c8c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:13.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.737+0000 7f3ca59ff700 1 --2- 172.21.15.98:0/4020944888 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3c8c0683c0 0x7f3c8c06a870 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3c940090b0 tx=0x7f3c94008030 comp rx=0 tx=0).stop 2023-03-03T08:00:13.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.737+0000 7f3ca59ff700 1 -- 172.21.15.98:0/4020944888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ca00b8b90 msgr2=0x7f3ca015f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:13.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.737+0000 7f3ca59ff700 1 --2- 172.21.15.98:0/4020944888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ca00b8b90 0x7f3ca015f3f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f3c900072c0 tx=0x7f3c900075d0 comp rx=0 tx=0).stop 2023-03-03T08:00:13.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.737+0000 7f3ca59ff700 1 -- 172.21.15.98:0/4020944888 shutdown_connections 2023-03-03T08:00:13.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.737+0000 7f3ca59ff700 1 --2- 172.21.15.98:0/4020944888 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3c8c0683c0 0x7f3c8c06a870 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:13.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.737+0000 7f3ca59ff700 1 --2- 172.21.15.98:0/4020944888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ca00b8b90 0x7f3ca015f3f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:13.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.737+0000 7f3ca59ff700 1 --2- 172.21.15.98:0/4020944888 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ca01435d0 0x7f3ca0143a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:13.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.737+0000 7f3ca59ff700 1 --2- 172.21.15.98:0/4020944888 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ca00ca350 0x7f3ca00ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:13.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.737+0000 7f3ca59ff700 1 -- 172.21.15.98:0/4020944888 >> 172.21.15.98:0/4020944888 conn(0x7f3ca006aee0 msgr2=0x7f3ca00b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:13.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.737+0000 7f3ca59ff700 1 -- 172.21.15.98:0/4020944888 shutdown_connections 2023-03-03T08:00:13.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:13.737+0000 7f3ca59ff700 1 -- 172.21.15.98:0/4020944888 wait complete. 2023-03-03T08:00:13.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-03-03T08:00:14.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.015+0000 7f5e833ff700 1 Processor -- start 2023-03-03T08:00:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.015+0000 7f5e833ff700 1 -- start start 2023-03-03T08:00:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.017+0000 7f5e833ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5e7c0b26d0 0x7f5e7c0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.017+0000 7f5e833ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e7c0cd7d0 con 0x7f5e7c0b8d10 2023-03-03T08:00:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.017+0000 7f5e833ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e7c0b90e0 con 0x7f5e7c0b5140 2023-03-03T08:00:14.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.017+0000 7f5e833ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e7c0b55c0 con 0x7f5e7c0b26d0 2023-03-03T08:00:14.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.017+0000 7f5e8119b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5e7c0b8d10 0x7f5e7c0cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46612/0 (socket says 172.21.15.98:46612) 2023-03-03T08:00:14.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.017+0000 7f5e8119b700 1 -- 172.21.15.98:0/1955545864 learned_addr learned my addr 172.21.15.98:0/1955545864 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:14.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.017+0000 7f5e8099a700 1 --2- 172.21.15.98:0/1955545864 >> v2:172.21.15.100:3300/0 conn(0x7f5e7c0b26d0 0x7f5e7c0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:14.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.017+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886732460 0 0) 0x7f5e7c0cd7d0 con 0x7f5e7c0b8d10 2023-03-03T08:00:14.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e680035b0 con 0x7f5e7c0b8d10 2023-03-03T08:00:14.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395391062 0 0) 0x7f5e7c0b90e0 con 0x7f5e7c0b5140 2023-03-03T08:00:14.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e7c0cd7d0 con 0x7f5e7c0b5140 2023-03-03T08:00:14.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3019808806 0 0) 0x7f5e680035b0 con 0x7f5e7c0b8d10 2023-03-03T08:00:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 >> v2:172.21.15.100:3300/0 conn(0x7f5e7c0b26d0 msgr2=0x7f5e7c0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 --2- 172.21.15.98:0/1955545864 >> v2:172.21.15.100:3300/0 conn(0x7f5e7c0b26d0 0x7f5e7c0b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 >> v1:172.21.15.100:6789/0 conn(0x7f5e7c0b5140 legacy=0x7f5e7c0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7c0cda20 con 0x7f5e7c0b8d10 2023-03-03T08:00:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f5e78001670 con 0x7f5e7c0b8d10 2023-03-03T08:00:14.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 >> v1:172.21.15.98:6789/0 conn(0x7f5e7c0b8d10 legacy=0x7f5e7c0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:14.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 --2- 172.21.15.98:0/1955545864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e68003be0 0x7f5e68003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:14.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 --2- 172.21.15.98:0/1955545864 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e68004850 0x7f5e68004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:14.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 --2- 172.21.15.98:0/1955545864 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e68005250 0x7f5e6800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:14.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7c0b55c0 con 0x7f5e68003be0 2023-03-03T08:00:14.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6800c160 con 0x7f5e68005250 2023-03-03T08:00:14.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.018+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6800c320 con 0x7f5e68004850 2023-03-03T08:00:14.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.019+0000 7f5e8119b700 1 --2- 172.21.15.98:0/1955545864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e68003be0 0x7f5e68003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:14.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.019+0000 7f5e8099a700 1 --2- 172.21.15.98:0/1955545864 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e68004850 0x7f5e68004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:14.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.019+0000 7f5e8199c700 1 --2- 172.21.15.98:0/1955545864 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e68005250 0x7f5e6800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.019+0000 7f5e8119b700 1 -- 172.21.15.98:0/1955545864 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e68004850 msgr2=0x7f5e68004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.019+0000 7f5e8119b700 1 --2- 172.21.15.98:0/1955545864 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e68004850 0x7f5e68004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.019+0000 7f5e8119b700 1 -- 172.21.15.98:0/1955545864 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e68005250 msgr2=0x7f5e6800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.019+0000 7f5e8119b700 1 --2- 172.21.15.98:0/1955545864 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e68005250 0x7f5e6800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.019+0000 7f5e8119b700 1 -- 172.21.15.98:0/1955545864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e680035b0 con 0x7f5e68003be0 2023-03-03T08:00:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.019+0000 7f5e8199c700 1 -- 172.21.15.98:0/1955545864 reap_dead start 2023-03-03T08:00:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.019+0000 7f5e8119b700 1 --2- 172.21.15.98:0/1955545864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e68003be0 0x7f5e68003fd0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5e78001eb0 tx=0x7f5e68005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9368e2fd4ad44e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.020+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7800cc70 con 0x7f5e68003be0 2023-03-03T08:00:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.020+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7800cdf0 con 0x7f5e68003be0 2023-03-03T08:00:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.020+0000 7f5e73fff700 1 -- 172.21.15.98:0/1955545864 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e78005ba0 con 0x7f5e68003be0 2023-03-03T08:00:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.020+0000 7f5e833ff700 1 -- 172.21.15.98:0/1955545864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e68003be0 msgr2=0x7f5e68003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:14.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.020+0000 7f5e833ff700 1 --2- 172.21.15.98:0/1955545864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e68003be0 0x7f5e68003fd0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5e78001eb0 tx=0x7f5e68005880 comp rx=0 tx=0).stop 2023-03-03T08:00:14.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.020+0000 7f5e833ff700 1 -- 172.21.15.98:0/1955545864 shutdown_connections 2023-03-03T08:00:14.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.020+0000 7f5e833ff700 1 --2- 172.21.15.98:0/1955545864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e68003be0 0x7f5e68003fd0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:14.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.020+0000 7f5e833ff700 1 -- 172.21.15.98:0/1955545864 >> 172.21.15.98:0/1955545864 conn(0x7f5e7c073e70 msgr2=0x7f5e7c06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:14.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.020+0000 7f5e833ff700 1 -- 172.21.15.98:0/1955545864 shutdown_connections 2023-03-03T08:00:14.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.020+0000 7f5e833ff700 1 -- 172.21.15.98:0/1955545864 wait complete. 2023-03-03T08:00:14.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.022+0000 7f5e833ff700 1 Processor -- start 2023-03-03T08:00:14.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.023+0000 7f5e833ff700 1 -- start start 2023-03-03T08:00:14.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e833ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e68003be0 0x7f5e7c152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:14.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e833ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e7c0c6c90 0x7f5e7c0cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:14.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e833ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e7c151590 0x7f5e7c159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:14.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e833ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7c152070 con 0x7f5e7c0c6c90 2023-03-03T08:00:14.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e833ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7c153e70 con 0x7f5e68003be0 2023-03-03T08:00:14.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e833ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7c153260 con 0x7f5e7c151590 2023-03-03T08:00:14.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e8099a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e7c0c6c90 0x7f5e7c0cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:14.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e8099a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e7c0c6c90 0x7f5e7c0cbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51148/0 (socket says 172.21.15.98:51148) 2023-03-03T08:00:14.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e8099a700 1 -- 172.21.15.98:0/3727245573 learned_addr learned my addr 172.21.15.98:0/3727245573 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:14.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e8119b700 1 --2- 172.21.15.98:0/3727245573 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e68003be0 0x7f5e7c152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:14.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e8199c700 1 --2- 172.21.15.98:0/3727245573 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e7c151590 0x7f5e7c159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:14.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e8099a700 1 -- 172.21.15.98:0/3727245573 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e7c151590 msgr2=0x7f5e7c159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:14.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e8099a700 1 --2- 172.21.15.98:0/3727245573 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e7c151590 0x7f5e7c159a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:14.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e8099a700 1 -- 172.21.15.98:0/3727245573 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e68003be0 msgr2=0x7f5e7c152ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:14.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e8099a700 1 --2- 172.21.15.98:0/3727245573 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e68003be0 0x7f5e7c152ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:14.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.024+0000 7f5e8099a700 1 -- 172.21.15.98:0/3727245573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7c0d0e80 con 0x7f5e7c0c6c90 2023-03-03T08:00:14.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.025+0000 7f5e8099a700 1 --2- 172.21.15.98:0/3727245573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e7c0c6c90 0x7f5e7c0cbc80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c0119d0 tx=0x7f5e6c011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:14.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.025+0000 7f5e71ffb700 1 -- 172.21.15.98:0/3727245573 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c011ec0 con 0x7f5e7c0c6c90 2023-03-03T08:00:14.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.025+0000 7f5e71ffb700 1 -- 172.21.15.98:0/3727245573 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6c004240 con 0x7f5e7c0c6c90 2023-03-03T08:00:14.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.025+0000 7f5e71ffb700 1 -- 172.21.15.98:0/3727245573 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c00f730 con 0x7f5e7c0c6c90 2023-03-03T08:00:14.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.025+0000 7f5e833ff700 1 -- 172.21.15.98:0/3727245573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e7c0cdd00 con 0x7f5e7c0c6c90 2023-03-03T08:00:14.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.025+0000 7f5e833ff700 1 -- 172.21.15.98:0/3727245573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7c0cdfe0 con 0x7f5e7c0c6c90 2023-03-03T08:00:14.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.026+0000 7f5e833ff700 1 -- 172.21.15.98:0/3727245573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e7c04e6e0 con 0x7f5e7c0c6c90 2023-03-03T08:00:14.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.028+0000 7f5e71ffb700 1 -- 172.21.15.98:0/3727245573 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f5e6c025070 con 0x7f5e7c0c6c90 2023-03-03T08:00:14.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.029+0000 7f5e71ffb700 1 --2- 172.21.15.98:0/3727245573 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5e600683c0 0x7f5e6006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:14.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.029+0000 7f5e71ffb700 1 -- 172.21.15.98:0/3727245573 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f5e6c085ea0 con 0x7f5e7c0c6c90 2023-03-03T08:00:14.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.029+0000 7f5e8119b700 1 --2- 172.21.15.98:0/3727245573 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5e600683c0 0x7f5e6006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:14.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.030+0000 7f5e8119b700 1 --2- 172.21.15.98:0/3727245573 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5e600683c0 0x7f5e6006a870 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5e78000c00 tx=0x7f5e78007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:14.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.033+0000 7f5e71ffb700 1 -- 172.21.15.98:0/3727245573 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5e6c052930 con 0x7f5e7c0c6c90 2023-03-03T08:00:14.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.224+0000 7f5e833ff700 1 -- 172.21.15.98:0/3727245573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5e7c06b7a0 con 0x7f5e7c0c6c90 2023-03-03T08:00:14.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.230+0000 7f5e71ffb700 1 -- 172.21.15.98:0/3727245573 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5e6c0190c0 con 0x7f5e7c0c6c90 2023-03-03T08:00:14.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.233+0000 7f5e833ff700 1 -- 172.21.15.98:0/3727245573 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5e600683c0 msgr2=0x7f5e6006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:14.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.233+0000 7f5e833ff700 1 --2- 172.21.15.98:0/3727245573 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5e600683c0 0x7f5e6006a870 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5e78000c00 tx=0x7f5e78007000 comp rx=0 tx=0).stop 2023-03-03T08:00:14.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.233+0000 7f5e833ff700 1 -- 172.21.15.98:0/3727245573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e7c0c6c90 msgr2=0x7f5e7c0cbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:14.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.233+0000 7f5e833ff700 1 --2- 172.21.15.98:0/3727245573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e7c0c6c90 0x7f5e7c0cbc80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c0119d0 tx=0x7f5e6c011ce0 comp rx=0 tx=0).stop 2023-03-03T08:00:14.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.233+0000 7f5e833ff700 1 -- 172.21.15.98:0/3727245573 shutdown_connections 2023-03-03T08:00:14.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.233+0000 7f5e833ff700 1 --2- 172.21.15.98:0/3727245573 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5e600683c0 0x7f5e6006a870 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:14.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.233+0000 7f5e833ff700 1 --2- 172.21.15.98:0/3727245573 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e7c151590 0x7f5e7c159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:14.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.233+0000 7f5e833ff700 1 --2- 172.21.15.98:0/3727245573 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e68003be0 0x7f5e7c152ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:14.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.233+0000 7f5e833ff700 1 --2- 172.21.15.98:0/3727245573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e7c0c6c90 0x7f5e7c0cbc80 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:14.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.233+0000 7f5e833ff700 1 -- 172.21.15.98:0/3727245573 >> 172.21.15.98:0/3727245573 conn(0x7f5e7c073e70 msgr2=0x7f5e7c15ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:14.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.233+0000 7f5e833ff700 1 -- 172.21.15.98:0/3727245573 shutdown_connections 2023-03-03T08:00:14.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:14.233+0000 7f5e833ff700 1 -- 172.21.15.98:0/3727245573 wait complete. 2023-03-03T08:00:14.262 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-03-03T08:00:14.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:00:14.394 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:00:14.529 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-03-03T08:00:14.529 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:00:14.530 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:00:14.599 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T08:00:14.600 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:00:14.601 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:00:14.669 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T08:00:14.670 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:00:14.670 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:00:14.740 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T08:00:14.740 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:00:14.742 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:00:14.810 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T08:00:14.810 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-03-03T08:00:14.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:00:15.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.056+0000 7f3820bff700 1 Processor -- start 2023-03-03T08:00:15.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.056+0000 7f3820bff700 1 -- start start 2023-03-03T08:00:15.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.057+0000 7f3820bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f381c0b87c0 0x7f381c0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:15.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.057+0000 7f3820bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f381c0b8b90 0x7f381c0b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:15.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.057+0000 7f3820bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f381c0b5240 0x7f381c0b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:15.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.057+0000 7f3820bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f381c0d0ef0 con 0x7f381c0b8b90 2023-03-03T08:00:15.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.057+0000 7f3820bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f381c0d0600 con 0x7f381c0b5240 2023-03-03T08:00:15.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.057+0000 7f3820bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381c0cd970 con 0x7f381c0b87c0 2023-03-03T08:00:15.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.058+0000 7f3813fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f381c0b8b90 0x7f381c0b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:15.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.058+0000 7f3813fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f381c0b8b90 0x7f381c0b4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57944/0 (socket says 172.21.15.98:57944) 2023-03-03T08:00:15.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.058+0000 7f3813fff700 1 -- 172.21.15.98:0/1035118328 learned_addr learned my addr 172.21.15.98:0/1035118328 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:15.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.058+0000 7f381ad9d700 1 --2- 172.21.15.98:0/1035118328 >> v2:172.21.15.100:3300/0 conn(0x7f381c0b5240 0x7f381c0b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:15.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.058+0000 7f381a59c700 1 --2- 172.21.15.98:0/1035118328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f381c0b87c0 0x7f381c0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:15.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.058+0000 7f3813fff700 1 -- 172.21.15.98:0/1035118328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f381c0b87c0 msgr2=0x7f381c0cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:15.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.058+0000 7f3813fff700 1 --2- 172.21.15.98:0/1035118328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f381c0b87c0 0x7f381c0cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.058+0000 7f3813fff700 1 -- 172.21.15.98:0/1035118328 >> v2:172.21.15.100:3300/0 conn(0x7f381c0b5240 msgr2=0x7f381c0b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:15.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.058+0000 7f3813fff700 1 --2- 172.21.15.98:0/1035118328 >> v2:172.21.15.100:3300/0 conn(0x7f381c0b5240 0x7f381c0b56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.058+0000 7f3813fff700 1 -- 172.21.15.98:0/1035118328 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c0cdc50 con 0x7f381c0b8b90 2023-03-03T08:00:15.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.058+0000 7f381ad9d700 1 --2- 172.21.15.98:0/1035118328 >> v2:172.21.15.100:3300/0 conn(0x7f381c0b5240 0x7f381c0b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:15.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.058+0000 7f381a59c700 1 --2- 172.21.15.98:0/1035118328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f381c0b87c0 0x7f381c0cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:15.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.058+0000 7f3813fff700 1 --2- 172.21.15.98:0/1035118328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f381c0b8b90 0x7f381c0b4d00 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f381c0d31a0 tx=0x7f3804009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acfdb88156c45892 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:15.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.059+0000 7f3819d9b700 1 -- 172.21.15.98:0/1035118328 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380400b870 con 0x7f381c0b8b90 2023-03-03T08:00:15.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.059+0000 7f3819d9b700 1 -- 172.21.15.98:0/1035118328 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380400b9d0 con 0x7f381c0b8b90 2023-03-03T08:00:15.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.059+0000 7f3819d9b700 1 -- 172.21.15.98:0/1035118328 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380400bd60 con 0x7f381c0b8b90 2023-03-03T08:00:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.059+0000 7f3820bff700 1 -- 172.21.15.98:0/1035118328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f381c0b8b90 msgr2=0x7f381c0b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.059+0000 7f3820bff700 1 --2- 172.21.15.98:0/1035118328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f381c0b8b90 0x7f381c0b4d00 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f381c0d31a0 tx=0x7f3804009bd0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.059+0000 7f3820bff700 1 -- 172.21.15.98:0/1035118328 shutdown_connections 2023-03-03T08:00:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.059+0000 7f3820bff700 1 --2- 172.21.15.98:0/1035118328 >> v2:172.21.15.100:3300/0 conn(0x7f381c0b5240 0x7f381c0b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.059+0000 7f3820bff700 1 --2- 172.21.15.98:0/1035118328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f381c0b87c0 0x7f381c0cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.059+0000 7f3820bff700 1 --2- 172.21.15.98:0/1035118328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f381c0b8b90 0x7f381c0b4d00 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.059+0000 7f3820bff700 1 -- 172.21.15.98:0/1035118328 >> 172.21.15.98:0/1035118328 conn(0x7f381c06aee0 msgr2=0x7f381c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.060+0000 7f3820bff700 1 -- 172.21.15.98:0/1035118328 shutdown_connections 2023-03-03T08:00:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.060+0000 7f3820bff700 1 -- 172.21.15.98:0/1035118328 wait complete. 2023-03-03T08:00:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.062+0000 7f3820bff700 1 Processor -- start 2023-03-03T08:00:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.062+0000 7f3820bff700 1 -- start start 2023-03-03T08:00:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.063+0000 7f3820bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f381c0b5240 0x7f381c0cd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.063+0000 7f3820bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f381c0b87c0 0x7f381c15d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.063+0000 7f3820bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f381c0b8b90 0x7f381c0cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.063+0000 7f3820bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c0ccd70 con 0x7f381c0b8b90 2023-03-03T08:00:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.063+0000 7f3820bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c0cd740 con 0x7f381c0b87c0 2023-03-03T08:00:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.063+0000 7f3820bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381c0cc160 con 0x7f381c0b5240 2023-03-03T08:00:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.063+0000 7f381ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f381c0b8b90 0x7f381c0cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.063+0000 7f3813fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f381c0b87c0 0x7f381c15d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.063+0000 7f381ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f381c0b8b90 0x7f381c0cb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57958/0 (socket says 172.21.15.98:57958) 2023-03-03T08:00:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.063+0000 7f381ad9d700 1 -- 172.21.15.98:0/4113622817 learned_addr learned my addr 172.21.15.98:0/4113622817 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.063+0000 7f381a59c700 1 --2- 172.21.15.98:0/4113622817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f381c0b5240 0x7f381c0cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.064+0000 7f381ad9d700 1 -- 172.21.15.98:0/4113622817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f381c0b5240 msgr2=0x7f381c0cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.064+0000 7f381ad9d700 1 --2- 172.21.15.98:0/4113622817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f381c0b5240 0x7f381c0cd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.064+0000 7f381ad9d700 1 -- 172.21.15.98:0/4113622817 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f381c0b87c0 msgr2=0x7f381c15d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.064+0000 7f381ad9d700 1 --2- 172.21.15.98:0/4113622817 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f381c0b87c0 0x7f381c15d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.064+0000 7f381ad9d700 1 -- 172.21.15.98:0/4113622817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3804009550 con 0x7f381c0b8b90 2023-03-03T08:00:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.064+0000 7f3813fff700 1 --2- 172.21.15.98:0/4113622817 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f381c0b87c0 0x7f381c15d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:00:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.064+0000 7f381ad9d700 1 --2- 172.21.15.98:0/4113622817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f381c0b8b90 0x7f381c0cb7a0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f381400cef0 tx=0x7f381400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.064+0000 7f3812ffd700 1 -- 172.21.15.98:0/4113622817 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381400ac80 con 0x7f381c0b8b90 2023-03-03T08:00:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.064+0000 7f3812ffd700 1 -- 172.21.15.98:0/4113622817 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381400ade0 con 0x7f381c0b8b90 2023-03-03T08:00:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.064+0000 7f3820bff700 1 -- 172.21.15.98:0/4113622817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381c0cdf30 con 0x7f381c0b8b90 2023-03-03T08:00:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.064+0000 7f3812ffd700 1 -- 172.21.15.98:0/4113622817 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3814015680 con 0x7f381c0b8b90 2023-03-03T08:00:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.064+0000 7f3820bff700 1 -- 172.21.15.98:0/4113622817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f381c073a80 con 0x7f381c0b8b90 2023-03-03T08:00:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.065+0000 7f3820bff700 1 -- 172.21.15.98:0/4113622817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f381c0ce0c0 con 0x7f381c0b8b90 2023-03-03T08:00:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.067+0000 7f3812ffd700 1 -- 172.21.15.98:0/4113622817 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f381401c030 con 0x7f381c0b8b90 2023-03-03T08:00:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.068+0000 7f3812ffd700 1 --2- 172.21.15.98:0/4113622817 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37fc0683c0 0x7f37fc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.068+0000 7f3812ffd700 1 -- 172.21.15.98:0/4113622817 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f3814085e60 con 0x7f381c0b8b90 2023-03-03T08:00:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.068+0000 7f381a59c700 1 --2- 172.21.15.98:0/4113622817 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37fc0683c0 0x7f37fc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.069+0000 7f381a59c700 1 --2- 172.21.15.98:0/4113622817 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37fc0683c0 0x7f37fc06a870 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f380c006fd0 tx=0x7f380c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:15.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.072+0000 7f3812ffd700 1 -- 172.21.15.98:0/4113622817 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3814052820 con 0x7f381c0b8b90 2023-03-03T08:00:15.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.307+0000 7f3820bff700 1 -- 172.21.15.98:0/4113622817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f381c0729b0 con 0x7f381c0b8b90 2023-03-03T08:00:15.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.308+0000 7f3812ffd700 1 -- 172.21.15.98:0/4113622817 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f381401a090 con 0x7f381c0b8b90 2023-03-03T08:00:15.309 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:15.310 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/3952787875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3952787875},{"type":"v1","addr":"172.21.15.100:6834","nonce":3952787875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1241071793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1241071793},{"type":"v1","addr":"172.21.15.98:6836","nonce":1241071793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5739,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3712572493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":3712572493},{"type":"v1","addr":"172.21.15.100:6835","nonce":3712572493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":6303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1616842068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1616842068},{"type":"v1","addr":"172.21.15.98:6837","nonce":1616842068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[]} 2023-03-03T08:00:15.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.310+0000 7f3820bff700 1 -- 172.21.15.98:0/4113622817 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37fc0683c0 msgr2=0x7f37fc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:15.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.310+0000 7f3820bff700 1 --2- 172.21.15.98:0/4113622817 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37fc0683c0 0x7f37fc06a870 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f380c006fd0 tx=0x7f380c008040 comp rx=0 tx=0).stop 2023-03-03T08:00:15.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.310+0000 7f3820bff700 1 -- 172.21.15.98:0/4113622817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f381c0b8b90 msgr2=0x7f381c0cb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:15.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.310+0000 7f3820bff700 1 --2- 172.21.15.98:0/4113622817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f381c0b8b90 0x7f381c0cb7a0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f381400cef0 tx=0x7f381400cf20 comp rx=0 tx=0).stop 2023-03-03T08:00:15.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.310+0000 7f3820bff700 1 -- 172.21.15.98:0/4113622817 shutdown_connections 2023-03-03T08:00:15.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.310+0000 7f3820bff700 1 --2- 172.21.15.98:0/4113622817 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f37fc0683c0 0x7f37fc06a870 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.310+0000 7f3820bff700 1 --2- 172.21.15.98:0/4113622817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f381c0b8b90 0x7f381c0cb7a0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.310+0000 7f3820bff700 1 --2- 172.21.15.98:0/4113622817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f381c0b5240 0x7f381c0cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.310+0000 7f3820bff700 1 --2- 172.21.15.98:0/4113622817 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f381c0b87c0 0x7f381c15d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.310+0000 7f3820bff700 1 -- 172.21.15.98:0/4113622817 >> 172.21.15.98:0/4113622817 conn(0x7f381c06aee0 msgr2=0x7f381c0d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:15.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.311+0000 7f3820bff700 1 -- 172.21.15.98:0/4113622817 shutdown_connections 2023-03-03T08:00:15.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.311+0000 7f3820bff700 1 -- 172.21.15.98:0/4113622817 wait complete. 2023-03-03T08:00:15.314 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 26 2023-03-03T08:00:15.339 INFO:tasks.ceph.mds.a:Restarting daemon 2023-03-03T08:00:15.340 INFO:tasks.ceph.mds.a:Stopping old one... 2023-03-03T08:00:15.340 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-03-03T08:00:15.340 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:00:15.340 INFO:tasks.ceph.mds.c:Restarting daemon 2023-03-03T08:00:15.340 INFO:tasks.ceph.mds.c:Stopping old one... 2023-03-03T08:00:15.341 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-03-03T08:00:15.341 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:00:15.341 INFO:tasks.ceph.mds.b:Restarting daemon 2023-03-03T08:00:15.341 INFO:tasks.ceph.mds.b:Stopping old one... 2023-03-03T08:00:15.342 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-03-03T08:00:15.342 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:00:15.342 INFO:tasks.ceph.mds.d:Restarting daemon 2023-03-03T08:00:15.342 INFO:tasks.ceph.mds.d:Stopping old one... 2023-03-03T08:00:15.342 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-03-03T08:00:15.343 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:00:15.365 INFO:tasks.ceph.mds.a:Stopped 2023-03-03T08:00:15.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-03-03T08:00:15.366 INFO:tasks.ceph.mds.c:Stopped 2023-03-03T08:00:15.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-03-03T08:00:15.370 INFO:tasks.ceph.mds.b:Stopped 2023-03-03T08:00:15.370 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-03-03T08:00:15.371 INFO:tasks.ceph.mds.d:Stopped 2023-03-03T08:00:15.371 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-03-03T08:00:15.411 INFO:tasks.ceph.mds.a:Started 2023-03-03T08:00:15.411 DEBUG:teuthology.parallel:result is None 2023-03-03T08:00:15.415 INFO:tasks.ceph.mds.c:Started 2023-03-03T08:00:15.415 DEBUG:teuthology.parallel:result is None 2023-03-03T08:00:15.416 INFO:tasks.ceph.mds.b:Started 2023-03-03T08:00:15.416 DEBUG:teuthology.parallel:result is None 2023-03-03T08:00:15.420 INFO:tasks.ceph.mds.d:Started 2023-03-03T08:00:15.420 DEBUG:teuthology.parallel:result is None 2023-03-03T08:00:15.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-03-03T08:00:15.609 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2023-03-03T08:00:15.610 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2023-03-03T08:00:15.616 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-03-03T08:00:15.616 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2023-03-03T08:00:15.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.686+0000 7f630abff700 1 Processor -- start 2023-03-03T08:00:15.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.686+0000 7f630abff700 1 -- start start 2023-03-03T08:00:15.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.688+0000 7f630abff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63040d0270 0x7f63040cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:15.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.688+0000 7f630abff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f630406b4b0 0x7f63040d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:15.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.688+0000 7f630abff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63040cf980 con 0x7f630415ca70 2023-03-03T08:00:15.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.688+0000 7f630abff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63040cd5b0 con 0x7f630406b4b0 2023-03-03T08:00:15.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.688+0000 7f630abff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63040d1800 con 0x7f63040d0270 2023-03-03T08:00:15.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.688+0000 7f6303fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63040d0270 0x7f63040cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:15.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.688+0000 7f630919c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f630406b4b0 0x7f63040d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:15.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.688+0000 7f630899b700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f630415ca70 0x7f63040cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:54786/0 (socket says 172.21.15.98:54786) 2023-03-03T08:00:15.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.688+0000 7f630899b700 1 -- 172.21.15.98:0/867684385 learned_addr learned my addr 172.21.15.98:0/867684385 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:15.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.689+0000 7f630919c700 1 -- 172.21.15.98:0/867684385 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63040d0270 msgr2=0x7f63040cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:15.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.689+0000 7f630919c700 1 --2- 172.21.15.98:0/867684385 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63040d0270 0x7f63040cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.689+0000 7f630919c700 1 -- 172.21.15.98:0/867684385 >> v1:172.21.15.100:6789/0 conn(0x7f630415ca70 legacy=0x7f63040cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.689+0000 7f630919c700 1 -- 172.21.15.98:0/867684385 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63040cd900 con 0x7f630406b4b0 2023-03-03T08:00:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.689+0000 7f63037fe700 1 -- 172.21.15.98:0/867684385 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051817095 0 0) 0x7f63040cf980 con 0x7f630415ca70 2023-03-03T08:00:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.689+0000 7f630919c700 1 --2- 172.21.15.98:0/867684385 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f630406b4b0 0x7f63040d1fc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009d90 tx=0x7f62fc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=befdd1b517b15011 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.690+0000 7f63037fe700 1 -- 172.21.15.98:0/867684385 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc00e070 con 0x7f630406b4b0 2023-03-03T08:00:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.690+0000 7f63037fe700 1 -- 172.21.15.98:0/867684385 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc00bb20 con 0x7f630406b4b0 2023-03-03T08:00:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.690+0000 7f63037fe700 1 -- 172.21.15.98:0/867684385 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc012430 con 0x7f630406b4b0 2023-03-03T08:00:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.690+0000 7f630abff700 1 -- 172.21.15.98:0/867684385 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f630406b4b0 msgr2=0x7f63040d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.690+0000 7f630abff700 1 --2- 172.21.15.98:0/867684385 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f630406b4b0 0x7f63040d1fc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009d90 tx=0x7f62fc00b330 comp rx=0 tx=0).stop 2023-03-03T08:00:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.690+0000 7f630abff700 1 -- 172.21.15.98:0/867684385 shutdown_connections 2023-03-03T08:00:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.690+0000 7f630abff700 1 --2- 172.21.15.98:0/867684385 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f630406b4b0 0x7f63040d1fc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.690+0000 7f630abff700 1 --2- 172.21.15.98:0/867684385 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63040d0270 0x7f63040cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.690+0000 7f630abff700 1 -- 172.21.15.98:0/867684385 >> 172.21.15.98:0/867684385 conn(0x7f6304073e70 msgr2=0x7f630406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.690+0000 7f630abff700 1 -- 172.21.15.98:0/867684385 shutdown_connections 2023-03-03T08:00:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.690+0000 7f630abff700 1 -- 172.21.15.98:0/867684385 wait complete. 2023-03-03T08:00:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.692+0000 7f630abff700 1 Processor -- start 2023-03-03T08:00:15.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.693+0000 7f630abff700 1 -- start start 2023-03-03T08:00:15.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.693+0000 7f630abff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f630406b4b0 0x7f630414e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:15.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.694+0000 7f630abff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63040d0270 0x7f63040b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:15.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.694+0000 7f630abff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f630415ca70 0x7f63040b6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:15.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.694+0000 7f630abff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6304157ae0 con 0x7f630415ca70 2023-03-03T08:00:15.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.694+0000 7f630abff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63041f6420 con 0x7f63040d0270 2023-03-03T08:00:15.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.694+0000 7f630abff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63040b59a0 con 0x7f630406b4b0 2023-03-03T08:00:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.694+0000 7f630919c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f630415ca70 0x7f63040b6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.694+0000 7f630899b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f630406b4b0 0x7f630414e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.694+0000 7f630919c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f630415ca70 0x7f63040b6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58012/0 (socket says 172.21.15.98:58012) 2023-03-03T08:00:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.694+0000 7f630919c700 1 -- 172.21.15.98:0/1006429107 learned_addr learned my addr 172.21.15.98:0/1006429107 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.694+0000 7f6303fff700 1 --2- 172.21.15.98:0/1006429107 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63040d0270 0x7f63040b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.694+0000 7f630919c700 1 -- 172.21.15.98:0/1006429107 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f630406b4b0 msgr2=0x7f630414e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.694+0000 7f630919c700 1 --2- 172.21.15.98:0/1006429107 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f630406b4b0 0x7f630414e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.694+0000 7f630919c700 1 -- 172.21.15.98:0/1006429107 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63040d0270 msgr2=0x7f63040b69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:15.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.694+0000 7f630919c700 1 --2- 172.21.15.98:0/1006429107 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63040d0270 0x7f63040b69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:15.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.694+0000 7f630919c700 1 -- 172.21.15.98:0/1006429107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc009a70 con 0x7f630415ca70 2023-03-03T08:00:15.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.695+0000 7f630919c700 1 --2- 172.21.15.98:0/1006429107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f630415ca70 0x7f63040b6f00 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f62fc014040 tx=0x7f62fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:15.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.695+0000 7f63017fa700 1 -- 172.21.15.98:0/1006429107 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc012430 con 0x7f630415ca70 2023-03-03T08:00:15.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.695+0000 7f63017fa700 1 -- 172.21.15.98:0/1006429107 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc020070 con 0x7f630415ca70 2023-03-03T08:00:15.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.695+0000 7f63017fa700 1 -- 172.21.15.98:0/1006429107 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc004280 con 0x7f630415ca70 2023-03-03T08:00:15.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.695+0000 7f630abff700 1 -- 172.21.15.98:0/1006429107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63040d1aa0 con 0x7f630415ca70 2023-03-03T08:00:15.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.695+0000 7f630abff700 1 -- 172.21.15.98:0/1006429107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63040cdbe0 con 0x7f630415ca70 2023-03-03T08:00:15.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.696+0000 7f630abff700 1 -- 172.21.15.98:0/1006429107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630415ce40 con 0x7f630415ca70 2023-03-03T08:00:15.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.697+0000 7f63017fa700 1 -- 172.21.15.98:0/1006429107 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f62fc02e070 con 0x7f630415ca70 2023-03-03T08:00:15.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.699+0000 7f63017fa700 1 --2- 172.21.15.98:0/1006429107 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f62e80683c0 0x7f62e806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:15.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.699+0000 7f63017fa700 1 -- 172.21.15.98:0/1006429107 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f62fc022070 con 0x7f630415ca70 2023-03-03T08:00:15.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.699+0000 7f630899b700 1 --2- 172.21.15.98:0/1006429107 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f62e80683c0 0x7f62e806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:15.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.700+0000 7f630899b700 1 --2- 172.21.15.98:0/1006429107 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f62e80683c0 0x7f62e806a870 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f62f8007b10 tx=0x7f62f8007a60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.703+0000 7f63017fa700 1 -- 172.21.15.98:0/1006429107 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f62fc054180 con 0x7f630415ca70 2023-03-03T08:00:15.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:15.915+0000 7f630abff700 1 -- 172.21.15.98:0/1006429107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f630404e6e0 con 0x7f630415ca70 2023-03-03T08:00:16.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:16.743+0000 7f63017fa700 1 -- 172.21.15.98:0/1006429107 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v62) v1 ==== 97+0+0 (secure 0 0 0) 0x7f62fc0577a0 con 0x7f630415ca70 2023-03-03T08:00:16.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:16.746+0000 7f630abff700 1 -- 172.21.15.98:0/1006429107 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f62e80683c0 msgr2=0x7f62e806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:16.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:16.746+0000 7f630abff700 1 --2- 172.21.15.98:0/1006429107 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f62e80683c0 0x7f62e806a870 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f62f8007b10 tx=0x7f62f8007a60 comp rx=0 tx=0).stop 2023-03-03T08:00:16.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:16.746+0000 7f630abff700 1 -- 172.21.15.98:0/1006429107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f630415ca70 msgr2=0x7f63040b6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:16.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:16.746+0000 7f630abff700 1 --2- 172.21.15.98:0/1006429107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f630415ca70 0x7f63040b6f00 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f62fc014040 tx=0x7f62fc004ad0 comp rx=0 tx=0).stop 2023-03-03T08:00:16.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:16.746+0000 7f630abff700 1 -- 172.21.15.98:0/1006429107 shutdown_connections 2023-03-03T08:00:16.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:16.746+0000 7f630abff700 1 --2- 172.21.15.98:0/1006429107 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f62e80683c0 0x7f62e806a870 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:16.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:16.746+0000 7f630abff700 1 --2- 172.21.15.98:0/1006429107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f630415ca70 0x7f63040b6f00 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:16.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:16.746+0000 7f630abff700 1 --2- 172.21.15.98:0/1006429107 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f630406b4b0 0x7f630414e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:16.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:16.746+0000 7f630abff700 1 --2- 172.21.15.98:0/1006429107 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63040d0270 0x7f63040b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:16.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:16.746+0000 7f630abff700 1 -- 172.21.15.98:0/1006429107 >> 172.21.15.98:0/1006429107 conn(0x7f6304073e70 msgr2=0x7f630415f110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:16.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:16.746+0000 7f630abff700 1 -- 172.21.15.98:0/1006429107 shutdown_connections 2023-03-03T08:00:16.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:16.746+0000 7f630abff700 1 -- 172.21.15.98:0/1006429107 wait complete. 2023-03-03T08:00:16.751 INFO:teuthology.orchestra.run.smithi098.stderr: removed all blocklist entries 2023-03-03T08:00:16.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-03-03T08:00:17.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.022+0000 7fa3bffff700 1 Processor -- start 2023-03-03T08:00:17.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.023+0000 7fa3bffff700 1 -- start start 2023-03-03T08:00:17.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.024+0000 7fa3bffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3b80713e0 0x7fa3b80717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.024+0000 7fa3bffff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b8071cf0 con 0x7fa3b815e490 2023-03-03T08:00:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.024+0000 7fa3bffff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b815b760 con 0x7fa3b806c7f0 2023-03-03T08:00:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.024+0000 7fa3bffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b815ba40 con 0x7fa3b80713e0 2023-03-03T08:00:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.025+0000 7fa3bd59a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa3b815e490 0x7fa3b815ad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60464/0 (socket says 172.21.15.98:60464) 2023-03-03T08:00:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.025+0000 7fa3bd59a700 1 -- 172.21.15.98:0/3015953718 learned_addr learned my addr 172.21.15.98:0/3015953718 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.025+0000 7fa3bdd9b700 1 --2- 172.21.15.98:0/3015953718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3b80713e0 0x7fa3b80717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:17.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.025+0000 7fa3bcd99700 1 -- 172.21.15.98:0/3015953718 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973245377 0 0) 0x7fa3b8071cf0 con 0x7fa3b815e490 2023-03-03T08:00:17.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.025+0000 7fa3bcd99700 1 -- 172.21.15.98:0/3015953718 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3a40035b0 con 0x7fa3b815e490 2023-03-03T08:00:17.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.025+0000 7fa3bcd99700 1 -- 172.21.15.98:0/3015953718 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626760176 0 0) 0x7fa3b815b760 con 0x7fa3b806c7f0 2023-03-03T08:00:17.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.025+0000 7fa3bcd99700 1 -- 172.21.15.98:0/3015953718 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b8071cf0 con 0x7fa3b806c7f0 2023-03-03T08:00:17.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.026+0000 7fa3bdd9b700 1 -- 172.21.15.98:0/3015953718 >> v1:172.21.15.100:6789/0 conn(0x7fa3b806c7f0 legacy=0x7fa3b80ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:17.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.026+0000 7fa3bdd9b700 1 -- 172.21.15.98:0/3015953718 >> v1:172.21.15.98:6789/0 conn(0x7fa3b815e490 legacy=0x7fa3b815ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.026+0000 7fa3bdd9b700 1 -- 172.21.15.98:0/3015953718 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b815bd20 con 0x7fa3b80713e0 2023-03-03T08:00:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.026+0000 7fa3bdd9b700 1 --2- 172.21.15.98:0/3015953718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3b80713e0 0x7fa3b80717b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009870 tx=0x7fa3b4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f616952530dd1be7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.026+0000 7fa3bcd99700 1 -- 172.21.15.98:0/3015953718 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b4011830 con 0x7fa3b80713e0 2023-03-03T08:00:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.027+0000 7fa3bcd99700 1 -- 172.21.15.98:0/3015953718 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b4011990 con 0x7fa3b80713e0 2023-03-03T08:00:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.027+0000 7fa3bcd99700 1 -- 172.21.15.98:0/3015953718 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b4011d00 con 0x7fa3b80713e0 2023-03-03T08:00:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.027+0000 7fa3bffff700 1 -- 172.21.15.98:0/3015953718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3b80713e0 msgr2=0x7fa3b80717b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.027+0000 7fa3bffff700 1 --2- 172.21.15.98:0/3015953718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3b80713e0 0x7fa3b80717b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009870 tx=0x7fa3b4009b80 comp rx=0 tx=0).stop 2023-03-03T08:00:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.027+0000 7fa3bffff700 1 -- 172.21.15.98:0/3015953718 shutdown_connections 2023-03-03T08:00:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.027+0000 7fa3bffff700 1 --2- 172.21.15.98:0/3015953718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3b80713e0 0x7fa3b80717b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.027+0000 7fa3bffff700 1 -- 172.21.15.98:0/3015953718 >> 172.21.15.98:0/3015953718 conn(0x7fa3b80b09f0 msgr2=0x7fa3b80b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.027+0000 7fa3bffff700 1 -- 172.21.15.98:0/3015953718 shutdown_connections 2023-03-03T08:00:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.027+0000 7fa3bffff700 1 -- 172.21.15.98:0/3015953718 wait complete. 2023-03-03T08:00:17.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.029+0000 7fa3bffff700 1 Processor -- start 2023-03-03T08:00:17.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.030+0000 7fa3bffff700 1 -- start start 2023-03-03T08:00:17.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.030+0000 7fa3bffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b806c7f0 0x7fa3b8151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:17.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3bffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3b80713e0 0x7fa3b80c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:17.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3bffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3b815e490 0x7fa3b8156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:17.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3bffff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b80b3960 con 0x7fa3b806c7f0 2023-03-03T08:00:17.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3bffff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b81525a0 con 0x7fa3b815e490 2023-03-03T08:00:17.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3bffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b8151ab0 con 0x7fa3b80713e0 2023-03-03T08:00:17.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3bdd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b806c7f0 0x7fa3b8151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:17.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3bdd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b806c7f0 0x7fa3b8151570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58022/0 (socket says 172.21.15.98:58022) 2023-03-03T08:00:17.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3bdd9b700 1 -- 172.21.15.98:0/1080269838 learned_addr learned my addr 172.21.15.98:0/1080269838 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3bd59a700 1 --2- 172.21.15.98:0/1080269838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3b80713e0 0x7fa3b80c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3be59c700 1 --2- 172.21.15.98:0/1080269838 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3b815e490 0x7fa3b8156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3bdd9b700 1 -- 172.21.15.98:0/1080269838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3b80713e0 msgr2=0x7fa3b80c71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3bdd9b700 1 --2- 172.21.15.98:0/1080269838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3b80713e0 0x7fa3b80c71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3bdd9b700 1 -- 172.21.15.98:0/1080269838 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3b815e490 msgr2=0x7fa3b8156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3bdd9b700 1 --2- 172.21.15.98:0/1080269838 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3b815e490 0x7fa3b8156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3bdd9b700 1 -- 172.21.15.98:0/1080269838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b40094a0 con 0x7fa3b806c7f0 2023-03-03T08:00:17.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.031+0000 7fa3be59c700 1 --2- 172.21.15.98:0/1080269838 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3b815e490 0x7fa3b8156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:17.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.032+0000 7fa3bdd9b700 1 --2- 172.21.15.98:0/1080269838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b806c7f0 0x7fa3b8151570 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009bc0 tx=0x7fa3b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:17.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.032+0000 7fa3ae7fc700 1 -- 172.21.15.98:0/1080269838 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b401a860 con 0x7fa3b806c7f0 2023-03-03T08:00:17.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.032+0000 7fa3ae7fc700 1 -- 172.21.15.98:0/1080269838 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b401a9c0 con 0x7fa3b806c7f0 2023-03-03T08:00:17.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.032+0000 7fa3ae7fc700 1 -- 172.21.15.98:0/1080269838 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b401ad30 con 0x7fa3b806c7f0 2023-03-03T08:00:17.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.032+0000 7fa3bffff700 1 -- 172.21.15.98:0/1080269838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b806c2d0 con 0x7fa3b806c7f0 2023-03-03T08:00:17.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.032+0000 7fa3bffff700 1 -- 172.21.15.98:0/1080269838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b806bff0 con 0x7fa3b806c7f0 2023-03-03T08:00:17.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.033+0000 7fa3bffff700 1 -- 172.21.15.98:0/1080269838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b815b470 con 0x7fa3b806c7f0 2023-03-03T08:00:17.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.034+0000 7fa3ae7fc700 1 -- 172.21.15.98:0/1080269838 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fa3b400f050 con 0x7fa3b806c7f0 2023-03-03T08:00:17.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.036+0000 7fa3ae7fc700 1 --2- 172.21.15.98:0/1080269838 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa39c0683c0 0x7fa39c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:17.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.036+0000 7fa3ae7fc700 1 -- 172.21.15.98:0/1080269838 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa3b4086970 con 0x7fa3b806c7f0 2023-03-03T08:00:17.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.036+0000 7fa3bd59a700 1 --2- 172.21.15.98:0/1080269838 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa39c0683c0 0x7fa39c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:17.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.037+0000 7fa3bd59a700 1 --2- 172.21.15.98:0/1080269838 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa39c0683c0 0x7fa39c06a870 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80c5ff0 tx=0x7fa3a8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:17.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.040+0000 7fa3ae7fc700 1 -- 172.21.15.98:0/1080269838 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa3b4053410 con 0x7fa3b806c7f0 2023-03-03T08:00:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.305+0000 7fa3bffff700 1 -- 172.21.15.98:0/1080269838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa3b804e6e0 con 0x7fa3b806c7f0 2023-03-03T08:00:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.306+0000 7fa3ae7fc700 1 -- 172.21.15.98:0/1080269838 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v10) v1 ==== 80+0+5298 (secure 0 0 0) 0x7fa3b4029090 con 0x7fa3b806c7f0 2023-03-03T08:00:17.308 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:17.308 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:00:17.308 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_dump": [ 2023-03-03T08:00:17.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.308 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.a", 2023-03-03T08:00:17.308 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkqFdgHxAAuwiGkaNnr5NplmhniDYAEQ==", 2023-03-03T08:00:17.308 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:00:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:00:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:00:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:00:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.b", 2023-03-03T08:00:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFk0x8kBhAA5JDo/wmI4650J9tUnvXVzw==", 2023-03-03T08:00:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:00:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:00:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:00:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:00:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.c", 2023-03-03T08:00:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkKoRwMBAAfkWGDUsqNgqfkscl0aBX1A==", 2023-03-03T08:00:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:00:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:00:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:00:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:00:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.d", 2023-03-03T08:00:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFkK9SKFxAAFSCozGfyRwrv46QPitRxTQ==", 2023-03-03T08:00:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:00:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:00:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:00:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:00:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.0", 2023-03-03T08:00:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDBpwFkTFzTERAAGHrFKuBlXIEfaYD7vs3ZHA==", 2023-03-03T08:00:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:00:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:00:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:00:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.1", 2023-03-03T08:00:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDEpwFkVWObLBAAW4acTvNbmscIe18NXVgRYg==", 2023-03-03T08:00:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:00:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:00:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:00:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.2", 2023-03-03T08:00:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDIpwFkK8lgChAAHVD5btBivUpcapSgDp2E0w==", 2023-03-03T08:00:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:00:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:00:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:00:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.3", 2023-03-03T08:00:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDLpwFksrb8IxAAN+CP1O3DJ1zDy+qyGNor2Q==", 2023-03-03T08:00:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:00:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:00:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:00:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.4", 2023-03-03T08:00:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDSpwFkkWGHIxAA4wejA0w05JPIUghEAVG/Rg==", 2023-03-03T08:00:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:00:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:00:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:00:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.5", 2023-03-03T08:00:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDWpwFknHmGAhAA6VM16luaykZfJEzr+yDZZA==", 2023-03-03T08:00:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:00:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:00:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:00:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.6", 2023-03-03T08:00:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDZpwFkcitNDRAAOz04YGTzx3Phenp0n8JeEA==", 2023-03-03T08:00:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:00:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:00:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:00:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.7", 2023-03-03T08:00:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDcpwFkrjmIFxAAbWeB2+8VQgyTplW+fk47OA==", 2023-03-03T08:00:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:00:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:00:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:00:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.0", 2023-03-03T08:00:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFk02lsJhAA0cH/evMnWdZN6DdBGCLjjg==", 2023-03-03T08:00:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:00:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow r", 2023-03-03T08:00:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rw pool=cephfs_data" 2023-03-03T08:00:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.1", 2023-03-03T08:00:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC9pwFkRqXrABAArm/01O5MtSTQHWQWKqNQEg==", 2023-03-03T08:00:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:00:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T08:00:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T08:00:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T08:00:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.2", 2023-03-03T08:00:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFkyKp3MRAA8Eh41TZNsO6/8VhAWCpNMQ==", 2023-03-03T08:00:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:00:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T08:00:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T08:00:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T08:00:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.3", 2023-03-03T08:00:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC9pwFk14bSCxAAy2nXc9G50W4lH5GtH84hpQ==", 2023-03-03T08:00:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:00:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T08:00:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T08:00:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T08:00:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.admin", 2023-03-03T08:00:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC6pwFkokmiEBAAFwi8rhPmk4Zyk95p760HDg==", 2023-03-03T08:00:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T08:00:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:00:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:00:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:00:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mgr.x", 2023-03-03T08:00:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkvdMQFBAAMi+x7psn5CDpd//d6jN2VA==", 2023-03-03T08:00:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T08:00:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile mgr", 2023-03-03T08:00:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:00:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mgr.y", 2023-03-03T08:00:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkfYO5BxAAFW/elbdoqVXBkUIKrAvYsg==", 2023-03-03T08:00:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T08:00:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile mgr", 2023-03-03T08:00:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:00:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:00:17.333 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:00:17.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.309+0000 7fa3bffff700 1 -- 172.21.15.98:0/1080269838 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa39c0683c0 msgr2=0x7fa39c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:17.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.309+0000 7fa3bffff700 1 --2- 172.21.15.98:0/1080269838 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa39c0683c0 0x7fa39c06a870 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80c5ff0 tx=0x7fa3a8009040 comp rx=0 tx=0).stop 2023-03-03T08:00:17.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.309+0000 7fa3bffff700 1 -- 172.21.15.98:0/1080269838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b806c7f0 msgr2=0x7fa3b8151570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:17.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.309+0000 7fa3bffff700 1 --2- 172.21.15.98:0/1080269838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b806c7f0 0x7fa3b8151570 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009bc0 tx=0x7fa3b4004750 comp rx=0 tx=0).stop 2023-03-03T08:00:17.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.309+0000 7fa3bffff700 1 -- 172.21.15.98:0/1080269838 shutdown_connections 2023-03-03T08:00:17.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.309+0000 7fa3bffff700 1 --2- 172.21.15.98:0/1080269838 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa39c0683c0 0x7fa39c06a870 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:17.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.309+0000 7fa3bffff700 1 --2- 172.21.15.98:0/1080269838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b806c7f0 0x7fa3b8151570 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:17.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.309+0000 7fa3bffff700 1 --2- 172.21.15.98:0/1080269838 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3b815e490 0x7fa3b8156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:17.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.309+0000 7fa3bffff700 1 --2- 172.21.15.98:0/1080269838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3b80713e0 0x7fa3b80c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:17.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.309+0000 7fa3bffff700 1 -- 172.21.15.98:0/1080269838 >> 172.21.15.98:0/1080269838 conn(0x7fa3b80b09f0 msgr2=0x7fa3b80acd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:17.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.309+0000 7fa3bffff700 1 -- 172.21.15.98:0/1080269838 shutdown_connections 2023-03-03T08:00:17.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.309+0000 7fa3bffff700 1 -- 172.21.15.98:0/1080269838 wait complete. 2023-03-03T08:00:17.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:00:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.585+0000 7f7c265ff700 1 Processor -- start 2023-03-03T08:00:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.586+0000 7f7c265ff700 1 -- start start 2023-03-03T08:00:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.587+0000 7f7c265ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7c200b87c0 0x7f7c200cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.587+0000 7f7c265ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c200d0ef0 con 0x7f7c200b5410 2023-03-03T08:00:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.587+0000 7f7c265ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c200d0600 con 0x7f7c200b8b90 2023-03-03T08:00:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.587+0000 7f7c265ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c200cd970 con 0x7f7c200b87c0 2023-03-03T08:00:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.587+0000 7f7c1ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7c200b87c0 0x7f7c200cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.588+0000 7f7c24b9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7c200b5410 0x7f7c200d13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60468/0 (socket says 172.21.15.98:60468) 2023-03-03T08:00:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.588+0000 7f7c24b9c700 1 -- 172.21.15.98:0/3640325176 learned_addr learned my addr 172.21.15.98:0/3640325176 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:17.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.588+0000 7f7c1ffff700 1 -- 172.21.15.98:0/3640325176 >> v1:172.21.15.100:6789/0 conn(0x7f7c200b8b90 legacy=0x7f7c200b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.588+0000 7f7c1ffff700 1 -- 172.21.15.98:0/3640325176 >> v1:172.21.15.98:6789/0 conn(0x7f7c200b5410 legacy=0x7f7c200d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.588+0000 7f7c1ffff700 1 -- 172.21.15.98:0/3640325176 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c200cdc50 con 0x7f7c200b87c0 2023-03-03T08:00:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.588+0000 7f7c1ffff700 1 --2- 172.21.15.98:0/3640325176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c200b87c0 0x7f7c200cffe0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f7c10009840 tx=0x7f7c10009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530195bb4e159671 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.589+0000 7f7c1f7fe700 1 -- 172.21.15.98:0/3640325176 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10011830 con 0x7f7c200b87c0 2023-03-03T08:00:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.589+0000 7f7c1f7fe700 1 -- 172.21.15.98:0/3640325176 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c10011990 con 0x7f7c200b87c0 2023-03-03T08:00:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.589+0000 7f7c1f7fe700 1 -- 172.21.15.98:0/3640325176 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10011d20 con 0x7f7c200b87c0 2023-03-03T08:00:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.589+0000 7f7c265ff700 1 -- 172.21.15.98:0/3640325176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c200b87c0 msgr2=0x7f7c200cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:17.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.589+0000 7f7c265ff700 1 --2- 172.21.15.98:0/3640325176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c200b87c0 0x7f7c200cffe0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f7c10009840 tx=0x7f7c10009b50 comp rx=0 tx=0).stop 2023-03-03T08:00:17.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.589+0000 7f7c265ff700 1 -- 172.21.15.98:0/3640325176 shutdown_connections 2023-03-03T08:00:17.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.589+0000 7f7c265ff700 1 --2- 172.21.15.98:0/3640325176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c200b87c0 0x7f7c200cffe0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:17.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.589+0000 7f7c265ff700 1 -- 172.21.15.98:0/3640325176 >> 172.21.15.98:0/3640325176 conn(0x7f7c2006aee0 msgr2=0x7f7c2006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:17.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.589+0000 7f7c265ff700 1 -- 172.21.15.98:0/3640325176 shutdown_connections 2023-03-03T08:00:17.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.589+0000 7f7c265ff700 1 -- 172.21.15.98:0/3640325176 wait complete. 2023-03-03T08:00:17.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.591+0000 7f7c265ff700 1 Processor -- start 2023-03-03T08:00:17.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.592+0000 7f7c265ff700 1 -- start start 2023-03-03T08:00:17.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.593+0000 7f7c265ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c200b5410 0x7f7c20157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:17.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.593+0000 7f7c265ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c200b87c0 0x7f7c20156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:17.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.593+0000 7f7c265ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c200b8b90 0x7f7c200cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:17.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.593+0000 7f7c265ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c201567b0 con 0x7f7c200b5410 2023-03-03T08:00:17.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.593+0000 7f7c265ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c200c6f20 con 0x7f7c200b8b90 2023-03-03T08:00:17.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.593+0000 7f7c265ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c200c7200 con 0x7f7c200b87c0 2023-03-03T08:00:17.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.593+0000 7f7c1ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c200b5410 0x7f7c20157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:17.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.593+0000 7f7c1ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c200b5410 0x7f7c20157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58052/0 (socket says 172.21.15.98:58052) 2023-03-03T08:00:17.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.593+0000 7f7c1ffff700 1 -- 172.21.15.98:0/2250858798 learned_addr learned my addr 172.21.15.98:0/2250858798 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:17.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.593+0000 7f7c24b9c700 1 --2- 172.21.15.98:0/2250858798 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c200b8b90 0x7f7c200cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:17.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.593+0000 7f7c17fff700 1 --2- 172.21.15.98:0/2250858798 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c200b87c0 0x7f7c20156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:17.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.594+0000 7f7c1ffff700 1 -- 172.21.15.98:0/2250858798 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c200b87c0 msgr2=0x7f7c20156270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:17.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.594+0000 7f7c1ffff700 1 --2- 172.21.15.98:0/2250858798 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c200b87c0 0x7f7c20156270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:17.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.594+0000 7f7c1ffff700 1 -- 172.21.15.98:0/2250858798 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c200b8b90 msgr2=0x7f7c200cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:17.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.594+0000 7f7c1ffff700 1 --2- 172.21.15.98:0/2250858798 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c200b8b90 0x7f7c200cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:17.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.594+0000 7f7c1ffff700 1 -- 172.21.15.98:0/2250858798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c100094a0 con 0x7f7c200b5410 2023-03-03T08:00:17.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.594+0000 7f7c1ffff700 1 --2- 172.21.15.98:0/2250858798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c200b5410 0x7f7c20157510 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7c100090b0 tx=0x7f7c10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:17.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.594+0000 7f7c1d7fa700 1 -- 172.21.15.98:0/2250858798 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c100038a0 con 0x7f7c200b5410 2023-03-03T08:00:17.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.594+0000 7f7c1d7fa700 1 -- 172.21.15.98:0/2250858798 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1001f070 con 0x7f7c200b5410 2023-03-03T08:00:17.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.594+0000 7f7c1d7fa700 1 -- 172.21.15.98:0/2250858798 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10003c60 con 0x7f7c200b5410 2023-03-03T08:00:17.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.594+0000 7f7c265ff700 1 -- 172.21.15.98:0/2250858798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c200cdf30 con 0x7f7c200b5410 2023-03-03T08:00:17.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.595+0000 7f7c265ff700 1 -- 172.21.15.98:0/2250858798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c20073a80 con 0x7f7c200b5410 2023-03-03T08:00:17.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.596+0000 7f7c265ff700 1 -- 172.21.15.98:0/2250858798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c200ce0c0 con 0x7f7c200b5410 2023-03-03T08:00:17.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.597+0000 7f7c1d7fa700 1 -- 172.21.15.98:0/2250858798 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f7c1000f050 con 0x7f7c200b5410 2023-03-03T08:00:17.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.598+0000 7f7c1d7fa700 1 --2- 172.21.15.98:0/2250858798 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7c000683c0 0x7f7c0006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:17.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.598+0000 7f7c1d7fa700 1 -- 172.21.15.98:0/2250858798 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7c10021070 con 0x7f7c200b5410 2023-03-03T08:00:17.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.598+0000 7f7c17fff700 1 --2- 172.21.15.98:0/2250858798 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7c000683c0 0x7f7c0006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:17.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.599+0000 7f7c17fff700 1 --2- 172.21.15.98:0/2250858798 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7c000683c0 0x7f7c0006a870 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7c2015fc40 tx=0x7f7c08009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:17.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.602+0000 7f7c1d7fa700 1 -- 172.21.15.98:0/2250858798 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7c100540e0 con 0x7f7c200b5410 2023-03-03T08:00:17.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.797+0000 7f7c265ff700 1 -- 172.21.15.98:0/2250858798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c200729b0 con 0x7f7c200b5410 2023-03-03T08:00:17.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.798+0000 7f7c1d7fa700 1 -- 172.21.15.98:0/2250858798 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+14267 (secure 0 0 0) 0x7f7c1002b090 con 0x7f7c200b5410 2023-03-03T08:00:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:17.800 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":62,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:00:16.735236+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:00:17.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.800+0000 7f7c265ff700 1 -- 172.21.15.98:0/2250858798 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7c000683c0 msgr2=0x7f7c0006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:17.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.800+0000 7f7c265ff700 1 --2- 172.21.15.98:0/2250858798 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7c000683c0 0x7f7c0006a870 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7c2015fc40 tx=0x7f7c08009040 comp rx=0 tx=0).stop 2023-03-03T08:00:17.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.800+0000 7f7c265ff700 1 -- 172.21.15.98:0/2250858798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c200b5410 msgr2=0x7f7c20157510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:17.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.800+0000 7f7c265ff700 1 --2- 172.21.15.98:0/2250858798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c200b5410 0x7f7c20157510 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7c100090b0 tx=0x7f7c10004750 comp rx=0 tx=0).stop 2023-03-03T08:00:17.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.801+0000 7f7c265ff700 1 -- 172.21.15.98:0/2250858798 shutdown_connections 2023-03-03T08:00:17.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.801+0000 7f7c265ff700 1 --2- 172.21.15.98:0/2250858798 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7c000683c0 0x7f7c0006a870 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:17.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.801+0000 7f7c265ff700 1 --2- 172.21.15.98:0/2250858798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c200b5410 0x7f7c20157510 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:17.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.801+0000 7f7c265ff700 1 --2- 172.21.15.98:0/2250858798 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c200b8b90 0x7f7c200cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:17.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.801+0000 7f7c265ff700 1 --2- 172.21.15.98:0/2250858798 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c200b87c0 0x7f7c20156270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:17.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.801+0000 7f7c265ff700 1 -- 172.21.15.98:0/2250858798 >> 172.21.15.98:0/2250858798 conn(0x7f7c2006aee0 msgr2=0x7f7c2015c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:17.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.801+0000 7f7c265ff700 1 -- 172.21.15.98:0/2250858798 shutdown_connections 2023-03-03T08:00:17.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:17.801+0000 7f7c265ff700 1 -- 172.21.15.98:0/2250858798 wait complete. 2023-03-03T08:00:17.830 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:00:17.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:00:18.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.079+0000 7f6b9c7ff700 1 Processor -- start 2023-03-03T08:00:18.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.079+0000 7f6b9c7ff700 1 -- start start 2023-03-03T08:00:18.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.081+0000 7f6b9c7ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6b940b26d0 0x7f6b940b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:18.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.081+0000 7f6b9c7ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6b940b5140 0x7f6b940d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:18.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.081+0000 7f6b9c7ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b940cd7d0 con 0x7f6b940b8d10 2023-03-03T08:00:18.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.081+0000 7f6b9c7ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b940b55c0 con 0x7f6b940b5140 2023-03-03T08:00:18.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.081+0000 7f6b9c7ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b940b90e0 con 0x7f6b940b26d0 2023-03-03T08:00:18.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.081+0000 7f6b9ad9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6b940b5140 0x7f6b940d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:18.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.081+0000 7f6b9ad9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6b940b5140 0x7f6b940d13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58054/0 (socket says 172.21.15.98:58054) 2023-03-03T08:00:18.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.081+0000 7f6b9ad9c700 1 -- 172.21.15.98:0/1842169961 learned_addr learned my addr 172.21.15.98:0/1842169961 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:18.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.081+0000 7f6b99d9a700 1 --2- 172.21.15.98:0/1842169961 >> v2:172.21.15.100:3300/0 conn(0x7f6b940b26d0 0x7f6b940b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:18.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.082+0000 7f6b9ad9c700 1 -- 172.21.15.98:0/1842169961 >> v2:172.21.15.100:3300/0 conn(0x7f6b940b26d0 msgr2=0x7f6b940b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:18.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.082+0000 7f6b9ad9c700 1 --2- 172.21.15.98:0/1842169961 >> v2:172.21.15.100:3300/0 conn(0x7f6b940b26d0 0x7f6b940b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.082+0000 7f6b9ad9c700 1 -- 172.21.15.98:0/1842169961 >> v1:172.21.15.100:6789/0 conn(0x7f6b940b8d10 legacy=0x7f6b940cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:18.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.082+0000 7f6b9ad9c700 1 -- 172.21.15.98:0/1842169961 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b940cda20 con 0x7f6b940b5140 2023-03-03T08:00:18.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.082+0000 7f6b99d9a700 1 --2- 172.21.15.98:0/1842169961 >> v2:172.21.15.100:3300/0 conn(0x7f6b940b26d0 0x7f6b940b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:18.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.082+0000 7f6b9ad9c700 1 --2- 172.21.15.98:0/1842169961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b940b5140 0x7f6b940d13a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6b9408bfa0 tx=0x7f6b8c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7878a7cc923bf6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:18.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.082+0000 7f6b99599700 1 -- 172.21.15.98:0/1842169961 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c00e070 con 0x7f6b940b5140 2023-03-03T08:00:18.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.082+0000 7f6b99599700 1 -- 172.21.15.98:0/1842169961 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c00b9c0 con 0x7f6b940b5140 2023-03-03T08:00:18.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.082+0000 7f6b99599700 1 -- 172.21.15.98:0/1842169961 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c00bd70 con 0x7f6b940b5140 2023-03-03T08:00:18.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.083+0000 7f6b9c7ff700 1 -- 172.21.15.98:0/1842169961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b940b5140 msgr2=0x7f6b940d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:18.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.083+0000 7f6b9c7ff700 1 --2- 172.21.15.98:0/1842169961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b940b5140 0x7f6b940d13a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6b9408bfa0 tx=0x7f6b8c00b330 comp rx=0 tx=0).stop 2023-03-03T08:00:18.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.083+0000 7f6b9c7ff700 1 -- 172.21.15.98:0/1842169961 shutdown_connections 2023-03-03T08:00:18.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.083+0000 7f6b9c7ff700 1 --2- 172.21.15.98:0/1842169961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b940b5140 0x7f6b940d13a0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.083+0000 7f6b9c7ff700 1 --2- 172.21.15.98:0/1842169961 >> v2:172.21.15.100:3300/0 conn(0x7f6b940b26d0 0x7f6b940b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.083+0000 7f6b9c7ff700 1 -- 172.21.15.98:0/1842169961 >> 172.21.15.98:0/1842169961 conn(0x7f6b94073e70 msgr2=0x7f6b9406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:18.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.083+0000 7f6b9c7ff700 1 -- 172.21.15.98:0/1842169961 shutdown_connections 2023-03-03T08:00:18.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.083+0000 7f6b9c7ff700 1 -- 172.21.15.98:0/1842169961 wait complete. 2023-03-03T08:00:18.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.086+0000 7f6b9c7ff700 1 Processor -- start 2023-03-03T08:00:18.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.086+0000 7f6b9c7ff700 1 -- start start 2023-03-03T08:00:18.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.088+0000 7f6b9c7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b940b26d0 0x7f6b94150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:18.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.088+0000 7f6b9c7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b940b5140 0x7f6b940cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:18.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.088+0000 7f6b9c7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b940b8d10 0x7f6b9415f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:18.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.088+0000 7f6b9c7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b941f6750 con 0x7f6b940b5140 2023-03-03T08:00:18.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.088+0000 7f6b9c7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b940cc560 con 0x7f6b940b8d10 2023-03-03T08:00:18.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.088+0000 7f6b9c7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b94158920 con 0x7f6b940b26d0 2023-03-03T08:00:18.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.088+0000 7f6b99d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b940b5140 0x7f6b940cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:18.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.088+0000 7f6b99d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b940b5140 0x7f6b940cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58056/0 (socket says 172.21.15.98:58056) 2023-03-03T08:00:18.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.088+0000 7f6b99d9a700 1 -- 172.21.15.98:0/2927800980 learned_addr learned my addr 172.21.15.98:0/2927800980 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:18.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.088+0000 7f6b9ad9c700 1 --2- 172.21.15.98:0/2927800980 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b940b8d10 0x7f6b9415f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:18.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.088+0000 7f6b9a59b700 1 --2- 172.21.15.98:0/2927800980 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b940b26d0 0x7f6b94150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:18.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.089+0000 7f6b99d9a700 1 -- 172.21.15.98:0/2927800980 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b940b26d0 msgr2=0x7f6b94150f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:18.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.089+0000 7f6b99d9a700 1 --2- 172.21.15.98:0/2927800980 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b940b26d0 0x7f6b94150f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.089+0000 7f6b99d9a700 1 -- 172.21.15.98:0/2927800980 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b940b8d10 msgr2=0x7f6b9415f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:18.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.089+0000 7f6b99d9a700 1 --2- 172.21.15.98:0/2927800980 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b940b8d10 0x7f6b9415f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.089+0000 7f6b99d9a700 1 -- 172.21.15.98:0/2927800980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c009b30 con 0x7f6b940b5140 2023-03-03T08:00:18.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.089+0000 7f6b9a59b700 1 --2- 172.21.15.98:0/2927800980 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b940b26d0 0x7f6b94150f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:18.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.089+0000 7f6b99d9a700 1 --2- 172.21.15.98:0/2927800980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b940b5140 0x7f6b940cb980 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f6b8400e790 tx=0x7f6b8400eaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:18.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.089+0000 7f6b8affd700 1 -- 172.21.15.98:0/2927800980 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b84013610 con 0x7f6b940b5140 2023-03-03T08:00:18.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.089+0000 7f6b8affd700 1 -- 172.21.15.98:0/2927800980 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b84013770 con 0x7f6b940b5140 2023-03-03T08:00:18.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.089+0000 7f6b9c7ff700 1 -- 172.21.15.98:0/2927800980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b940d0e80 con 0x7f6b940b5140 2023-03-03T08:00:18.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.089+0000 7f6b8affd700 1 -- 172.21.15.98:0/2927800980 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b84013b20 con 0x7f6b940b5140 2023-03-03T08:00:18.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.089+0000 7f6b9c7ff700 1 -- 172.21.15.98:0/2927800980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b940cdd00 con 0x7f6b940b5140 2023-03-03T08:00:18.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.091+0000 7f6b9c7ff700 1 -- 172.21.15.98:0/2927800980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b940cdfe0 con 0x7f6b940b5140 2023-03-03T08:00:18.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.092+0000 7f6b8affd700 1 -- 172.21.15.98:0/2927800980 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f6b84023070 con 0x7f6b940b5140 2023-03-03T08:00:18.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.093+0000 7f6b8affd700 1 --2- 172.21.15.98:0/2927800980 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b780683c0 0x7f6b7806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:18.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.093+0000 7f6b8affd700 1 -- 172.21.15.98:0/2927800980 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6b84085680 con 0x7f6b940b5140 2023-03-03T08:00:18.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.093+0000 7f6b9a59b700 1 --2- 172.21.15.98:0/2927800980 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b780683c0 0x7f6b7806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:18.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.094+0000 7f6b9a59b700 1 --2- 172.21.15.98:0/2927800980 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b780683c0 0x7f6b7806a870 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6b940b4f80 tx=0x7f6b90009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:18.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.097+0000 7f6b8affd700 1 -- 172.21.15.98:0/2927800980 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6b84052070 con 0x7f6b940b5140 2023-03-03T08:00:18.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.288+0000 7f6b9c7ff700 1 -- 172.21.15.98:0/2927800980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6b941f6900 con 0x7f6b940b5140 2023-03-03T08:00:18.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.289+0000 7f6b8affd700 1 -- 172.21.15.98:0/2927800980 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6b941f6900 con 0x7f6b940b5140 2023-03-03T08:00:18.292 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:00:18.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.292+0000 7f6b9c7ff700 1 -- 172.21.15.98:0/2927800980 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b780683c0 msgr2=0x7f6b7806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:18.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.292+0000 7f6b9c7ff700 1 --2- 172.21.15.98:0/2927800980 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b780683c0 0x7f6b7806a870 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6b940b4f80 tx=0x7f6b90009040 comp rx=0 tx=0).stop 2023-03-03T08:00:18.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.292+0000 7f6b9c7ff700 1 -- 172.21.15.98:0/2927800980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b940b5140 msgr2=0x7f6b940cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:18.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.292+0000 7f6b9c7ff700 1 --2- 172.21.15.98:0/2927800980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b940b5140 0x7f6b940cb980 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f6b8400e790 tx=0x7f6b8400eaa0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.292+0000 7f6b9c7ff700 1 -- 172.21.15.98:0/2927800980 shutdown_connections 2023-03-03T08:00:18.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.292+0000 7f6b9c7ff700 1 --2- 172.21.15.98:0/2927800980 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b780683c0 0x7f6b7806a870 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.292+0000 7f6b9c7ff700 1 --2- 172.21.15.98:0/2927800980 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b940b8d10 0x7f6b9415f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.292+0000 7f6b9c7ff700 1 --2- 172.21.15.98:0/2927800980 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b940b26d0 0x7f6b94150f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.292+0000 7f6b9c7ff700 1 --2- 172.21.15.98:0/2927800980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b940b5140 0x7f6b940cb980 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.292+0000 7f6b9c7ff700 1 -- 172.21.15.98:0/2927800980 >> 172.21.15.98:0/2927800980 conn(0x7f6b94073e70 msgr2=0x7f6b940b8390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:18.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.292+0000 7f6b9c7ff700 1 -- 172.21.15.98:0/2927800980 shutdown_connections 2023-03-03T08:00:18.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.292+0000 7f6b9c7ff700 1 -- 172.21.15.98:0/2927800980 wait complete. 2023-03-03T08:00:18.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:00:18.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.567+0000 7fbf16bff700 1 Processor -- start 2023-03-03T08:00:18.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.567+0000 7fbf16bff700 1 -- start start 2023-03-03T08:00:18.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.568+0000 7fbf16bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fbf1015ca70 0x7fbf100cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:18.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.568+0000 7fbf16bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbf1006b4b0 0x7fbf100cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:18.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.569+0000 7fbf16bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf100cf980 con 0x7fbf100d00a0 2023-03-03T08:00:18.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.569+0000 7fbf16bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf100cd5b0 con 0x7fbf1006b4b0 2023-03-03T08:00:18.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.569+0000 7fbf16bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf100d0540 con 0x7fbf1015ca70 2023-03-03T08:00:18.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.569+0000 7fbf1519c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbf100d00a0 0x7fbf100b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60474/0 (socket says 172.21.15.98:60474) 2023-03-03T08:00:18.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.569+0000 7fbf1519c700 1 -- 172.21.15.98:0/938458413 learned_addr learned my addr 172.21.15.98:0/938458413 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:18.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.569+0000 7fbf0ffff700 1 --2- 172.21.15.98:0/938458413 >> v2:172.21.15.98:3300/0 conn(0x7fbf1006b4b0 0x7fbf100cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:18.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.569+0000 7fbf1499b700 1 --2- 172.21.15.98:0/938458413 >> v2:172.21.15.100:3300/0 conn(0x7fbf1015ca70 0x7fbf100cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:18.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.570+0000 7fbf0ffff700 1 -- 172.21.15.98:0/938458413 >> v2:172.21.15.100:3300/0 conn(0x7fbf1015ca70 msgr2=0x7fbf100cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:18.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.570+0000 7fbf0ffff700 1 --2- 172.21.15.98:0/938458413 >> v2:172.21.15.100:3300/0 conn(0x7fbf1015ca70 0x7fbf100cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.570+0000 7fbf0ffff700 1 -- 172.21.15.98:0/938458413 >> v1:172.21.15.98:6789/0 conn(0x7fbf100d00a0 legacy=0x7fbf100b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:18.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.570+0000 7fbf0ffff700 1 -- 172.21.15.98:0/938458413 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf100d2fd0 con 0x7fbf1006b4b0 2023-03-03T08:00:18.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.570+0000 7fbf1499b700 1 --2- 172.21.15.98:0/938458413 >> v2:172.21.15.100:3300/0 conn(0x7fbf1015ca70 0x7fbf100cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:00:18.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.570+0000 7fbf0f7fe700 1 -- 172.21.15.98:0/938458413 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357294786 0 0) 0x7fbf100cf980 con 0x7fbf100d00a0 2023-03-03T08:00:18.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.570+0000 7fbf0ffff700 1 --2- 172.21.15.98:0/938458413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf1006b4b0 0x7fbf100cfb60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbf1006c2c0 tx=0x7fbefc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=428ced1f5a14b2d2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:18.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.570+0000 7fbf0f7fe700 1 -- 172.21.15.98:0/938458413 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc011830 con 0x7fbf1006b4b0 2023-03-03T08:00:18.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.571+0000 7fbf0f7fe700 1 -- 172.21.15.98:0/938458413 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbefc011990 con 0x7fbf1006b4b0 2023-03-03T08:00:18.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.571+0000 7fbf0f7fe700 1 -- 172.21.15.98:0/938458413 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc011d20 con 0x7fbf1006b4b0 2023-03-03T08:00:18.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.571+0000 7fbf16bff700 1 -- 172.21.15.98:0/938458413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf1006b4b0 msgr2=0x7fbf100cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:18.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.571+0000 7fbf16bff700 1 --2- 172.21.15.98:0/938458413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf1006b4b0 0x7fbf100cfb60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbf1006c2c0 tx=0x7fbefc009b20 comp rx=0 tx=0).stop 2023-03-03T08:00:18.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.571+0000 7fbf16bff700 1 -- 172.21.15.98:0/938458413 shutdown_connections 2023-03-03T08:00:18.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.571+0000 7fbf16bff700 1 --2- 172.21.15.98:0/938458413 >> v2:172.21.15.100:3300/0 conn(0x7fbf1015ca70 0x7fbf100cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.571+0000 7fbf16bff700 1 --2- 172.21.15.98:0/938458413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf1006b4b0 0x7fbf100cfb60 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.571+0000 7fbf16bff700 1 -- 172.21.15.98:0/938458413 >> 172.21.15.98:0/938458413 conn(0x7fbf10073e70 msgr2=0x7fbf1006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:18.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.571+0000 7fbf16bff700 1 -- 172.21.15.98:0/938458413 shutdown_connections 2023-03-03T08:00:18.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.571+0000 7fbf16bff700 1 -- 172.21.15.98:0/938458413 wait complete. 2023-03-03T08:00:18.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.573+0000 7fbf16bff700 1 Processor -- start 2023-03-03T08:00:18.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.574+0000 7fbf16bff700 1 -- start start 2023-03-03T08:00:18.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf16bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf1006b4b0 0x7fbf1015edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:18.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf16bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf100d00a0 0x7fbf10145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:18.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf16bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf1015ca70 0x7fbf101464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:18.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf16bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100b4c70 con 0x7fbf100d00a0 2023-03-03T08:00:18.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf16bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100b46b0 con 0x7fbf1006b4b0 2023-03-03T08:00:18.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf16bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100b4990 con 0x7fbf1015ca70 2023-03-03T08:00:18.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf0ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf100d00a0 0x7fbf10145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:18.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf0ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf100d00a0 0x7fbf10145fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58080/0 (socket says 172.21.15.98:58080) 2023-03-03T08:00:18.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf0ffff700 1 -- 172.21.15.98:0/4268923968 learned_addr learned my addr 172.21.15.98:0/4268923968 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:18.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf1499b700 1 --2- 172.21.15.98:0/4268923968 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf1006b4b0 0x7fbf1015edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:18.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf1519c700 1 --2- 172.21.15.98:0/4268923968 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf1015ca70 0x7fbf101464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:18.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf0ffff700 1 -- 172.21.15.98:0/4268923968 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf1015ca70 msgr2=0x7fbf101464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:18.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf0ffff700 1 --2- 172.21.15.98:0/4268923968 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf1015ca70 0x7fbf101464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf0ffff700 1 -- 172.21.15.98:0/4268923968 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf1006b4b0 msgr2=0x7fbf1015edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:18.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf0ffff700 1 --2- 172.21.15.98:0/4268923968 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf1006b4b0 0x7fbf1015edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.575+0000 7fbf0ffff700 1 -- 172.21.15.98:0/4268923968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf040094a0 con 0x7fbf100d00a0 2023-03-03T08:00:18.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.576+0000 7fbf0ffff700 1 --2- 172.21.15.98:0/4268923968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf100d00a0 0x7fbf10145fb0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fbefc013000 tx=0x7fbefc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:18.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.576+0000 7fbf0d7fa700 1 -- 172.21.15.98:0/4268923968 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc0036a0 con 0x7fbf100d00a0 2023-03-03T08:00:18.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.576+0000 7fbf0d7fa700 1 -- 172.21.15.98:0/4268923968 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbefc01f070 con 0x7fbf100d00a0 2023-03-03T08:00:18.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.576+0000 7fbf0d7fa700 1 -- 172.21.15.98:0/4268923968 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc003a20 con 0x7fbf100d00a0 2023-03-03T08:00:18.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.576+0000 7fbf16bff700 1 -- 172.21.15.98:0/4268923968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbefc0094a0 con 0x7fbf100d00a0 2023-03-03T08:00:18.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.576+0000 7fbf16bff700 1 -- 172.21.15.98:0/4268923968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf100cd900 con 0x7fbf100d00a0 2023-03-03T08:00:18.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.577+0000 7fbf16bff700 1 -- 172.21.15.98:0/4268923968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf100d1800 con 0x7fbf100d00a0 2023-03-03T08:00:18.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.578+0000 7fbf0d7fa700 1 -- 172.21.15.98:0/4268923968 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fbefc00f050 con 0x7fbf100d00a0 2023-03-03T08:00:18.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.580+0000 7fbf0d7fa700 1 --2- 172.21.15.98:0/4268923968 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbef40683c0 0x7fbef406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:18.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.580+0000 7fbf0d7fa700 1 -- 172.21.15.98:0/4268923968 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbefc021070 con 0x7fbf100d00a0 2023-03-03T08:00:18.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.580+0000 7fbf1499b700 1 --2- 172.21.15.98:0/4268923968 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbef40683c0 0x7fbef406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:18.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.581+0000 7fbf1499b700 1 --2- 172.21.15.98:0/4268923968 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbef40683c0 0x7fbef406a870 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009cd0 tx=0x7fbf04009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:18.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.584+0000 7fbf0d7fa700 1 -- 172.21.15.98:0/4268923968 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbefc053df0 con 0x7fbf100d00a0 2023-03-03T08:00:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.778+0000 7fbf16bff700 1 -- 172.21.15.98:0/4268923968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fbf100cdbe0 con 0x7fbf100d00a0 2023-03-03T08:00:18.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.778+0000 7fbf0d7fa700 1 -- 172.21.15.98:0/4268923968 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v62) v1 ==== 92+0+10 (secure 0 0 0) 0x7fbefc057410 con 0x7fbf100d00a0 2023-03-03T08:00:18.780 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:00:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.781+0000 7fbf16bff700 1 -- 172.21.15.98:0/4268923968 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbef40683c0 msgr2=0x7fbef406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.781+0000 7fbf16bff700 1 --2- 172.21.15.98:0/4268923968 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbef40683c0 0x7fbef406a870 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009cd0 tx=0x7fbf04009210 comp rx=0 tx=0).stop 2023-03-03T08:00:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.781+0000 7fbf16bff700 1 -- 172.21.15.98:0/4268923968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf100d00a0 msgr2=0x7fbf10145fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.781+0000 7fbf16bff700 1 --2- 172.21.15.98:0/4268923968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf100d00a0 0x7fbf10145fb0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fbefc013000 tx=0x7fbefc004750 comp rx=0 tx=0).stop 2023-03-03T08:00:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.781+0000 7fbf16bff700 1 -- 172.21.15.98:0/4268923968 shutdown_connections 2023-03-03T08:00:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.781+0000 7fbf16bff700 1 --2- 172.21.15.98:0/4268923968 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fbef40683c0 0x7fbef406a870 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.781+0000 7fbf16bff700 1 --2- 172.21.15.98:0/4268923968 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf1015ca70 0x7fbf101464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.781+0000 7fbf16bff700 1 --2- 172.21.15.98:0/4268923968 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf1006b4b0 0x7fbf1015edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.781+0000 7fbf16bff700 1 --2- 172.21.15.98:0/4268923968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf100d00a0 0x7fbf10145fb0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.781+0000 7fbf16bff700 1 -- 172.21.15.98:0/4268923968 >> 172.21.15.98:0/4268923968 conn(0x7fbf10073e70 msgr2=0x7fbf100b8100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.781+0000 7fbf16bff700 1 -- 172.21.15.98:0/4268923968 shutdown_connections 2023-03-03T08:00:18.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:18.781+0000 7fbf16bff700 1 -- 172.21.15.98:0/4268923968 wait complete. 2023-03-03T08:00:18.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-03-03T08:00:19.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.054+0000 7f18a73ff700 1 Processor -- start 2023-03-03T08:00:19.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.055+0000 7f18a73ff700 1 -- start start 2023-03-03T08:00:19.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.056+0000 7f18a73ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f18a00b87c0 0x7f18a00cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:19.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.056+0000 7f18a73ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18a00b5410 0x7f18a00d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:19.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.056+0000 7f18a73ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a00d0ef0 con 0x7f18a00b8b90 2023-03-03T08:00:19.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.056+0000 7f18a73ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a00d0600 con 0x7f18a00b87c0 2023-03-03T08:00:19.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.056+0000 7f18a73ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a00cd970 con 0x7f18a00b5410 2023-03-03T08:00:19.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.057+0000 7f18a519b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f18a00b87c0 0x7f18a00cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:19.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.057+0000 7f18a599c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18a00b5410 0x7f18a00d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:19.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.057+0000 7f18a599c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18a00b5410 0x7f18a00d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:42790/0 (socket says 172.21.15.98:42790) 2023-03-03T08:00:19.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.057+0000 7f18a599c700 1 -- 172.21.15.98:0/422140310 learned_addr learned my addr 172.21.15.98:0/422140310 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:19.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.057+0000 7f18a519b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f18a00b87c0 0x7f18a00cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:43386/0 (socket says 172.21.15.98:43386) 2023-03-03T08:00:19.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.057+0000 7f18a499a700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f18a00b8b90 0x7f18a00b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:54824/0 (socket says 172.21.15.98:54824) 2023-03-03T08:00:19.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.057+0000 7f18a599c700 1 -- 172.21.15.98:0/422140310 >> v2:172.21.15.100:3300/0 conn(0x7f18a00b87c0 msgr2=0x7f18a00cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:19.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.057+0000 7f18a599c700 1 --2- 172.21.15.98:0/422140310 >> v2:172.21.15.100:3300/0 conn(0x7f18a00b87c0 0x7f18a00cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.057+0000 7f1897fff700 1 -- 172.21.15.98:0/422140310 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883768672 0 0) 0x7f18a00d0ef0 con 0x7f18a00b8b90 2023-03-03T08:00:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.057+0000 7f18a599c700 1 -- 172.21.15.98:0/422140310 >> v1:172.21.15.100:6789/0 conn(0x7f18a00b8b90 legacy=0x7f18a00b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.057+0000 7f18a599c700 1 -- 172.21.15.98:0/422140310 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a00cdc50 con 0x7f18a00b5410 2023-03-03T08:00:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.058+0000 7f18a599c700 1 --2- 172.21.15.98:0/422140310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18a00b5410 0x7f18a00d13d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1898009e50 tx=0x7f189800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=828215c3cabfa66c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.058+0000 7f1897fff700 1 -- 172.21.15.98:0/422140310 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189800e070 con 0x7f18a00b5410 2023-03-03T08:00:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.058+0000 7f1897fff700 1 -- 172.21.15.98:0/422140310 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189800b910 con 0x7f18a00b5410 2023-03-03T08:00:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.058+0000 7f1897fff700 1 -- 172.21.15.98:0/422140310 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189800bcc0 con 0x7f18a00b5410 2023-03-03T08:00:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.059+0000 7f18a73ff700 1 -- 172.21.15.98:0/422140310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18a00b5410 msgr2=0x7f18a00d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.059+0000 7f18a73ff700 1 --2- 172.21.15.98:0/422140310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18a00b5410 0x7f18a00d13d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1898009e50 tx=0x7f189800b330 comp rx=0 tx=0).stop 2023-03-03T08:00:19.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.059+0000 7f18a73ff700 1 -- 172.21.15.98:0/422140310 shutdown_connections 2023-03-03T08:00:19.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.059+0000 7f18a73ff700 1 --2- 172.21.15.98:0/422140310 >> v2:172.21.15.100:3300/0 conn(0x7f18a00b87c0 0x7f18a00cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:19.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.059+0000 7f18a73ff700 1 --2- 172.21.15.98:0/422140310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18a00b5410 0x7f18a00d13d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:19.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.059+0000 7f18a73ff700 1 -- 172.21.15.98:0/422140310 >> 172.21.15.98:0/422140310 conn(0x7f18a006aee0 msgr2=0x7f18a006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:19.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.059+0000 7f18a73ff700 1 -- 172.21.15.98:0/422140310 shutdown_connections 2023-03-03T08:00:19.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.059+0000 7f18a73ff700 1 -- 172.21.15.98:0/422140310 wait complete. 2023-03-03T08:00:19.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.061+0000 7f18a73ff700 1 Processor -- start 2023-03-03T08:00:19.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.061+0000 7f18a73ff700 1 -- start start 2023-03-03T08:00:19.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.062+0000 7f18a73ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18a00b5410 0x7f18a0151790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:19.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.062+0000 7f18a73ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18a00b87c0 0x7f18a015e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:19.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.062+0000 7f18a73ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a00b8b90 0x7f18a015eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:19.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.062+0000 7f18a73ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a00c7ef0 con 0x7f18a00b8b90 2023-03-03T08:00:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.062+0000 7f18a73ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a00c72e0 con 0x7f18a00b87c0 2023-03-03T08:00:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.062+0000 7f18a73ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a00c75c0 con 0x7f18a00b5410 2023-03-03T08:00:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.062+0000 7f18a599c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a00b8b90 0x7f18a015eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.063+0000 7f18a599c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a00b8b90 0x7f18a015eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58088/0 (socket says 172.21.15.98:58088) 2023-03-03T08:00:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.063+0000 7f18a599c700 1 -- 172.21.15.98:0/270995306 learned_addr learned my addr 172.21.15.98:0/270995306 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:19.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.063+0000 7f18a519b700 1 --2- 172.21.15.98:0/270995306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18a00b5410 0x7f18a0151790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:19.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.063+0000 7f18a499a700 1 --2- 172.21.15.98:0/270995306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18a00b87c0 0x7f18a015e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:19.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.063+0000 7f18a599c700 1 -- 172.21.15.98:0/270995306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18a00b5410 msgr2=0x7f18a0151790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:19.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.063+0000 7f18a599c700 1 --2- 172.21.15.98:0/270995306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18a00b5410 0x7f18a0151790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:19.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.063+0000 7f18a599c700 1 -- 172.21.15.98:0/270995306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18a00b87c0 msgr2=0x7f18a015e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:19.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.063+0000 7f18a599c700 1 --2- 172.21.15.98:0/270995306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18a00b87c0 0x7f18a015e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:19.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.063+0000 7f18a599c700 1 -- 172.21.15.98:0/270995306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1898009b30 con 0x7f18a00b8b90 2023-03-03T08:00:19.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.063+0000 7f18a599c700 1 --2- 172.21.15.98:0/270995306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a00b8b90 0x7f18a015eaa0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f189800b680 tx=0x7f1898004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.064+0000 7f1895ffb700 1 -- 172.21.15.98:0/270995306 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18980099b0 con 0x7f18a00b8b90 2023-03-03T08:00:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.064+0000 7f1895ffb700 1 -- 172.21.15.98:0/270995306 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18980092e0 con 0x7f18a00b8b90 2023-03-03T08:00:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.064+0000 7f1895ffb700 1 -- 172.21.15.98:0/270995306 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189801c660 con 0x7f18a00b8b90 2023-03-03T08:00:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.064+0000 7f18a73ff700 1 -- 172.21.15.98:0/270995306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a00cdf30 con 0x7f18a00b8b90 2023-03-03T08:00:19.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.064+0000 7f18a73ff700 1 -- 172.21.15.98:0/270995306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a0073a80 con 0x7f18a00b8b90 2023-03-03T08:00:19.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.065+0000 7f18a73ff700 1 -- 172.21.15.98:0/270995306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a00ce0c0 con 0x7f18a00b8b90 2023-03-03T08:00:19.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.066+0000 7f1895ffb700 1 -- 172.21.15.98:0/270995306 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f1898022030 con 0x7f18a00b8b90 2023-03-03T08:00:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.067+0000 7f1895ffb700 1 --2- 172.21.15.98:0/270995306 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f18840683c0 0x7f188406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.067+0000 7f1895ffb700 1 -- 172.21.15.98:0/270995306 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1898087450 con 0x7f18a00b8b90 2023-03-03T08:00:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.068+0000 7f18a519b700 1 --2- 172.21.15.98:0/270995306 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f18840683c0 0x7f188406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.069+0000 7f18a519b700 1 --2- 172.21.15.98:0/270995306 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f18840683c0 0x7f188406a870 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f189c009b70 tx=0x7f189c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.072+0000 7f1895ffb700 1 -- 172.21.15.98:0/270995306 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1898053e40 con 0x7f18a00b8b90 2023-03-03T08:00:19.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.268+0000 7f18a73ff700 1 -- 172.21.15.98:0/270995306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f18a00729b0 con 0x7f18a00b8b90 2023-03-03T08:00:19.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.268+0000 7f1895ffb700 1 -- 172.21.15.98:0/270995306 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v62) v1 ==== 84+0+126 (secure 0 0 0) 0x7f1898057460 con 0x7f18a00b8b90 2023-03-03T08:00:19.270 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:19.270 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T08:00:19.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:19.271 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2023-03-03T08:00:19.271 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2023-03-03T08:00:19.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:19.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:19.271 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 2, 2023-03-03T08:00:19.271 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": "rbd" 2023-03-03T08:00:19.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:19.272 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T08:00:19.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.271+0000 7f18a73ff700 1 -- 172.21.15.98:0/270995306 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f18840683c0 msgr2=0x7f188406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:19.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.271+0000 7f18a73ff700 1 --2- 172.21.15.98:0/270995306 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f18840683c0 0x7f188406a870 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f189c009b70 tx=0x7f189c008040 comp rx=0 tx=0).stop 2023-03-03T08:00:19.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.271+0000 7f18a73ff700 1 -- 172.21.15.98:0/270995306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a00b8b90 msgr2=0x7f18a015eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:19.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.271+0000 7f18a73ff700 1 --2- 172.21.15.98:0/270995306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a00b8b90 0x7f18a015eaa0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f189800b680 tx=0x7f1898004ad0 comp rx=0 tx=0).stop 2023-03-03T08:00:19.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.271+0000 7f18a73ff700 1 -- 172.21.15.98:0/270995306 shutdown_connections 2023-03-03T08:00:19.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.271+0000 7f18a73ff700 1 --2- 172.21.15.98:0/270995306 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f18840683c0 0x7f188406a870 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:19.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.271+0000 7f18a73ff700 1 --2- 172.21.15.98:0/270995306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a00b8b90 0x7f18a015eaa0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:19.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.271+0000 7f18a73ff700 1 --2- 172.21.15.98:0/270995306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18a00b5410 0x7f18a0151790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:19.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.271+0000 7f18a73ff700 1 --2- 172.21.15.98:0/270995306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18a00b87c0 0x7f18a015e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:19.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.271+0000 7f18a73ff700 1 -- 172.21.15.98:0/270995306 >> 172.21.15.98:0/270995306 conn(0x7f18a006aee0 msgr2=0x7f18a00d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:19.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.271+0000 7f18a73ff700 1 -- 172.21.15.98:0/270995306 shutdown_connections 2023-03-03T08:00:19.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.271+0000 7f18a73ff700 1 -- 172.21.15.98:0/270995306 wait complete. 2023-03-03T08:00:19.299 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-03-03T08:00:19.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-03-03T08:00:19.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.547+0000 7f3f3a79d700 1 Processor -- start 2023-03-03T08:00:19.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.548+0000 7f3f3a79d700 1 -- start start 2023-03-03T08:00:19.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.549+0000 7f3f3a79d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3f340d0110 0x7f3f340cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:19.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.549+0000 7f3f3a79d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f340b3c80 con 0x7f3f340b3550 2023-03-03T08:00:19.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.549+0000 7f3f3a79d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f340b4370 con 0x7f3f34073ed0 2023-03-03T08:00:19.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.549+0000 7f3f3a79d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f340b3ed0 con 0x7f3f340d0110 2023-03-03T08:00:19.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.549+0000 7f3f38f9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3f340d0110 0x7f3f340cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:19.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.549+0000 7f3f3979b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3f340b3550 0x7f3f340cfa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60476/0 (socket says 172.21.15.98:60476) 2023-03-03T08:00:19.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.549+0000 7f3f3979b700 1 -- 172.21.15.98:0/2746862459 learned_addr learned my addr 172.21.15.98:0/2746862459 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:19.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.550+0000 7f3f38f9a700 1 -- 172.21.15.98:0/2746862459 >> v1:172.21.15.100:6789/0 conn(0x7f3f34073ed0 legacy=0x7f3f340ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:19.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.550+0000 7f3f38f9a700 1 -- 172.21.15.98:0/2746862459 >> v1:172.21.15.98:6789/0 conn(0x7f3f340b3550 legacy=0x7f3f340cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:19.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.550+0000 7f3f38f9a700 1 -- 172.21.15.98:0/2746862459 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f340cd640 con 0x7f3f340d0110 2023-03-03T08:00:19.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.550+0000 7f3f2bfff700 1 -- 172.21.15.98:0/2746862459 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535502890 0 0) 0x7f3f340b3c80 con 0x7f3f340b3550 2023-03-03T08:00:19.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.550+0000 7f3f38f9a700 1 --2- 172.21.15.98:0/2746862459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f340d0110 0x7f3f340cde40 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009840 tx=0x7f3f24009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e8b47791a0c10d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:19.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.550+0000 7f3f2bfff700 1 -- 172.21.15.98:0/2746862459 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f24004230 con 0x7f3f340d0110 2023-03-03T08:00:19.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.551+0000 7f3f2bfff700 1 -- 172.21.15.98:0/2746862459 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f24004390 con 0x7f3f340d0110 2023-03-03T08:00:19.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.551+0000 7f3f2bfff700 1 -- 172.21.15.98:0/2746862459 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f240036a0 con 0x7f3f340d0110 2023-03-03T08:00:19.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.551+0000 7f3f3a79d700 1 -- 172.21.15.98:0/2746862459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f340d0110 msgr2=0x7f3f340cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:19.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.551+0000 7f3f3a79d700 1 --2- 172.21.15.98:0/2746862459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f340d0110 0x7f3f340cde40 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009840 tx=0x7f3f24009b50 comp rx=0 tx=0).stop 2023-03-03T08:00:19.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.551+0000 7f3f3a79d700 1 -- 172.21.15.98:0/2746862459 shutdown_connections 2023-03-03T08:00:19.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.551+0000 7f3f3a79d700 1 --2- 172.21.15.98:0/2746862459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f340d0110 0x7f3f340cde40 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:19.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.551+0000 7f3f3a79d700 1 -- 172.21.15.98:0/2746862459 >> 172.21.15.98:0/2746862459 conn(0x7f3f34073b00 msgr2=0x7f3f340d2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:19.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.551+0000 7f3f3a79d700 1 -- 172.21.15.98:0/2746862459 shutdown_connections 2023-03-03T08:00:19.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.551+0000 7f3f3a79d700 1 -- 172.21.15.98:0/2746862459 wait complete. 2023-03-03T08:00:19.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.553+0000 7f3f3a79d700 1 Processor -- start 2023-03-03T08:00:19.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.554+0000 7f3f3a79d700 1 -- start start 2023-03-03T08:00:19.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.554+0000 7f3f3a79d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f34073ed0 0x7f3f340b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:19.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.555+0000 7f3f3a79d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3f340b3550 0x7f3f34157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:19.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.555+0000 7f3f3a79d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f340d0110 0x7f3f340cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:19.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.555+0000 7f3f3a79d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f340b2000 con 0x7f3f340d0110 2023-03-03T08:00:19.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.555+0000 7f3f3a79d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f34158b30 con 0x7f3f340b3550 2023-03-03T08:00:19.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.555+0000 7f3f3a79d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f340b13f0 con 0x7f3f34073ed0 2023-03-03T08:00:19.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.555+0000 7f3f39f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f340d0110 0x7f3f340cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:19.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.555+0000 7f3f39f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f340d0110 0x7f3f340cbd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58110/0 (socket says 172.21.15.98:58110) 2023-03-03T08:00:19.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.555+0000 7f3f39f9c700 1 -- 172.21.15.98:0/196108821 learned_addr learned my addr 172.21.15.98:0/196108821 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:19.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.555+0000 7f3f38f9a700 1 --2- 172.21.15.98:0/196108821 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3f340b3550 0x7f3f34157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:19.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.555+0000 7f3f3979b700 1 --2- 172.21.15.98:0/196108821 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f34073ed0 0x7f3f340b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:19.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.555+0000 7f3f39f9c700 1 -- 172.21.15.98:0/196108821 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f34073ed0 msgr2=0x7f3f340b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:19.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.555+0000 7f3f39f9c700 1 --2- 172.21.15.98:0/196108821 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f34073ed0 0x7f3f340b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:19.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.555+0000 7f3f39f9c700 1 -- 172.21.15.98:0/196108821 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3f340b3550 msgr2=0x7f3f34157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:19.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.555+0000 7f3f39f9c700 1 --2- 172.21.15.98:0/196108821 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3f340b3550 0x7f3f34157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:19.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.555+0000 7f3f39f9c700 1 -- 172.21.15.98:0/196108821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f240094a0 con 0x7f3f340d0110 2023-03-03T08:00:19.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.556+0000 7f3f39f9c700 1 --2- 172.21.15.98:0/196108821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f340d0110 0x7f3f340cbd10 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00cc10 tx=0x7f3f2c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:19.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.556+0000 7f3f29ffb700 1 -- 172.21.15.98:0/196108821 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c00ab10 con 0x7f3f340d0110 2023-03-03T08:00:19.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.556+0000 7f3f29ffb700 1 -- 172.21.15.98:0/196108821 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2c00ac70 con 0x7f3f340d0110 2023-03-03T08:00:19.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.556+0000 7f3f3a79d700 1 -- 172.21.15.98:0/196108821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f340d3d60 con 0x7f3f340d0110 2023-03-03T08:00:19.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.556+0000 7f3f29ffb700 1 -- 172.21.15.98:0/196108821 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c012420 con 0x7f3f340d0110 2023-03-03T08:00:19.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.556+0000 7f3f3a79d700 1 -- 172.21.15.98:0/196108821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f340cd920 con 0x7f3f340d0110 2023-03-03T08:00:19.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.557+0000 7f3f3a79d700 1 -- 172.21.15.98:0/196108821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f340d3ef0 con 0x7f3f340d0110 2023-03-03T08:00:19.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.558+0000 7f3f29ffb700 1 -- 172.21.15.98:0/196108821 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f3f2c010050 con 0x7f3f340d0110 2023-03-03T08:00:19.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.560+0000 7f3f29ffb700 1 --2- 172.21.15.98:0/196108821 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3f180683c0 0x7f3f1806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:19.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.560+0000 7f3f29ffb700 1 -- 172.21.15.98:0/196108821 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3f2c085e60 con 0x7f3f340d0110 2023-03-03T08:00:19.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.560+0000 7f3f3979b700 1 --2- 172.21.15.98:0/196108821 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3f180683c0 0x7f3f1806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:19.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.561+0000 7f3f3979b700 1 --2- 172.21.15.98:0/196108821 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3f180683c0 0x7f3f1806a870 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3f30000f60 tx=0x7f3f30007be0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:19.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.563+0000 7f3f29ffb700 1 -- 172.21.15.98:0/196108821 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3f340d3ef0 con 0x7f3f340d0110 2023-03-03T08:00:19.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:19.760+0000 7f3f3a79d700 1 -- 172.21.15.98:0/196108821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3f340d3ef0 con 0x7f3f340d0110 2023-03-03T08:00:20.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:20.759+0000 7f3f29ffb700 1 -- 172.21.15.98:0/196108821 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v63) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3f340d3ef0 con 0x7f3f340d0110 2023-03-03T08:00:20.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:20.764+0000 7f3f3a79d700 1 -- 172.21.15.98:0/196108821 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3f180683c0 msgr2=0x7f3f1806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:20.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:20.764+0000 7f3f3a79d700 1 --2- 172.21.15.98:0/196108821 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3f180683c0 0x7f3f1806a870 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3f30000f60 tx=0x7f3f30007be0 comp rx=0 tx=0).stop 2023-03-03T08:00:20.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:20.764+0000 7f3f3a79d700 1 -- 172.21.15.98:0/196108821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f340d0110 msgr2=0x7f3f340cbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:20.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:20.764+0000 7f3f3a79d700 1 --2- 172.21.15.98:0/196108821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f340d0110 0x7f3f340cbd10 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00cc10 tx=0x7f3f2c00cf20 comp rx=0 tx=0).stop 2023-03-03T08:00:20.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:20.765+0000 7f3f3a79d700 1 -- 172.21.15.98:0/196108821 shutdown_connections 2023-03-03T08:00:20.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:20.765+0000 7f3f3a79d700 1 --2- 172.21.15.98:0/196108821 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3f180683c0 0x7f3f1806a870 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:20.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:20.765+0000 7f3f3a79d700 1 --2- 172.21.15.98:0/196108821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f340d0110 0x7f3f340cbd10 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:20.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:20.765+0000 7f3f3a79d700 1 --2- 172.21.15.98:0/196108821 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f34073ed0 0x7f3f340b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:20.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:20.765+0000 7f3f3a79d700 1 --2- 172.21.15.98:0/196108821 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3f340b3550 0x7f3f34157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:20.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:20.765+0000 7f3f3a79d700 1 -- 172.21.15.98:0/196108821 >> 172.21.15.98:0/196108821 conn(0x7f3f34073b00 msgr2=0x7f3f3415b370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:20.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:20.766+0000 7f3f3a79d700 1 -- 172.21.15.98:0/196108821 shutdown_connections 2023-03-03T08:00:20.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:20.767+0000 7f3f3a79d700 1 -- 172.21.15.98:0/196108821 wait complete. 2023-03-03T08:00:20.771 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2023-03-03T08:00:20.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-03-03T08:00:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.052+0000 7fd80cfff700 1 Processor -- start 2023-03-03T08:00:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.052+0000 7fd80cfff700 1 -- start start 2023-03-03T08:00:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.053+0000 7fd80cfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd8080b26d0 0x7fd8080b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.053+0000 7fd80cfff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd8080b5140 0x7fd8080d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.054+0000 7fd80cfff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8080cd7d0 con 0x7fd8080b8d10 2023-03-03T08:00:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.054+0000 7fd80cfff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8080b55c0 con 0x7fd8080b26d0 2023-03-03T08:00:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.054+0000 7fd80cfff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8080b90e0 con 0x7fd8080b5140 2023-03-03T08:00:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.054+0000 7fd805d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd8080b26d0 0x7fd8080b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.054+0000 7fd805d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd8080b26d0 0x7fd8080b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58118/0 (socket says 172.21.15.98:58118) 2023-03-03T08:00:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.054+0000 7fd805d9b700 1 -- 172.21.15.98:0/2013284250 learned_addr learned my addr 172.21.15.98:0/2013284250 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.054+0000 7fd806d9d700 1 --2- 172.21.15.98:0/2013284250 >> v2:172.21.15.100:3300/0 conn(0x7fd8080b5140 0x7fd8080d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:21.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.054+0000 7fd805d9b700 1 -- 172.21.15.98:0/2013284250 >> v2:172.21.15.100:3300/0 conn(0x7fd8080b5140 msgr2=0x7fd8080d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:21.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.054+0000 7fd805d9b700 1 --2- 172.21.15.98:0/2013284250 >> v2:172.21.15.100:3300/0 conn(0x7fd8080b5140 0x7fd8080d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:21.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.054+0000 7fd805d9b700 1 -- 172.21.15.98:0/2013284250 >> v1:172.21.15.100:6789/0 conn(0x7fd8080b8d10 legacy=0x7fd8080cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:21.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.054+0000 7fd805d9b700 1 -- 172.21.15.98:0/2013284250 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8080cda20 con 0x7fd8080b26d0 2023-03-03T08:00:21.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.055+0000 7fd805d9b700 1 --2- 172.21.15.98:0/2013284250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8080b26d0 0x7fd8080b4c00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd8080ca1e0 tx=0x7fd7fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac5cbc454759323 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:21.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.055+0000 7fd80559a700 1 -- 172.21.15.98:0/2013284250 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc00b870 con 0x7fd8080b26d0 2023-03-03T08:00:21.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.055+0000 7fd80559a700 1 -- 172.21.15.98:0/2013284250 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7fc00b9d0 con 0x7fd8080b26d0 2023-03-03T08:00:21.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.055+0000 7fd80559a700 1 -- 172.21.15.98:0/2013284250 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc00bd60 con 0x7fd8080b26d0 2023-03-03T08:00:21.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.055+0000 7fd80cfff700 1 -- 172.21.15.98:0/2013284250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8080b26d0 msgr2=0x7fd8080b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.055+0000 7fd80cfff700 1 --2- 172.21.15.98:0/2013284250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8080b26d0 0x7fd8080b4c00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd8080ca1e0 tx=0x7fd7fc009bd0 comp rx=0 tx=0).stop 2023-03-03T08:00:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.056+0000 7fd80cfff700 1 -- 172.21.15.98:0/2013284250 shutdown_connections 2023-03-03T08:00:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.056+0000 7fd80cfff700 1 --2- 172.21.15.98:0/2013284250 >> v2:172.21.15.100:3300/0 conn(0x7fd8080b5140 0x7fd8080d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.056+0000 7fd80cfff700 1 --2- 172.21.15.98:0/2013284250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8080b26d0 0x7fd8080b4c00 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.056+0000 7fd80cfff700 1 -- 172.21.15.98:0/2013284250 >> 172.21.15.98:0/2013284250 conn(0x7fd808073e70 msgr2=0x7fd80806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.056+0000 7fd80cfff700 1 -- 172.21.15.98:0/2013284250 shutdown_connections 2023-03-03T08:00:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.056+0000 7fd80cfff700 1 -- 172.21.15.98:0/2013284250 wait complete. 2023-03-03T08:00:21.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.058+0000 7fd80cfff700 1 Processor -- start 2023-03-03T08:00:21.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.058+0000 7fd80cfff700 1 -- start start 2023-03-03T08:00:21.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.059+0000 7fd80cfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8080b26d0 0x7fd808152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:21.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.059+0000 7fd80cfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8080b5140 0x7fd808154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:21.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.059+0000 7fd80cfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8080b8d10 0x7fd808156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:21.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.059+0000 7fd80cfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd808155930 con 0x7fd8080b5140 2023-03-03T08:00:21.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.059+0000 7fd80cfff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd808152c50 con 0x7fd8080b8d10 2023-03-03T08:00:21.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.059+0000 7fd80cfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd808154cb0 con 0x7fd8080b26d0 2023-03-03T08:00:21.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.059+0000 7fd805d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8080b5140 0x7fd808154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:21.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.059+0000 7fd805d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8080b5140 0x7fd808154260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58132/0 (socket says 172.21.15.98:58132) 2023-03-03T08:00:21.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.059+0000 7fd805d9b700 1 -- 172.21.15.98:0/4037752837 learned_addr learned my addr 172.21.15.98:0/4037752837 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:21.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.059+0000 7fd806d9d700 1 --2- 172.21.15.98:0/4037752837 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8080b8d10 0x7fd808156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:21.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.059+0000 7fd80659c700 1 --2- 172.21.15.98:0/4037752837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8080b26d0 0x7fd808152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:21.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.060+0000 7fd805d9b700 1 -- 172.21.15.98:0/4037752837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8080b26d0 msgr2=0x7fd808152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:21.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.060+0000 7fd805d9b700 1 --2- 172.21.15.98:0/4037752837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8080b26d0 0x7fd808152710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:21.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.060+0000 7fd805d9b700 1 -- 172.21.15.98:0/4037752837 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8080b8d10 msgr2=0x7fd808156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:21.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.060+0000 7fd805d9b700 1 --2- 172.21.15.98:0/4037752837 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8080b8d10 0x7fd808156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:21.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.060+0000 7fd805d9b700 1 -- 172.21.15.98:0/4037752837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7fc009550 con 0x7fd8080b5140 2023-03-03T08:00:21.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.060+0000 7fd80659c700 1 --2- 172.21.15.98:0/4037752837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8080b26d0 0x7fd808152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:21.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.060+0000 7fd805d9b700 1 --2- 172.21.15.98:0/4037752837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8080b5140 0x7fd808154260 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc000c00 tx=0x7fd7fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:21.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.060+0000 7fd7f6ffd700 1 -- 172.21.15.98:0/4037752837 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc004310 con 0x7fd8080b5140 2023-03-03T08:00:21.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.060+0000 7fd7f6ffd700 1 -- 172.21.15.98:0/4037752837 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7fc01e070 con 0x7fd8080b5140 2023-03-03T08:00:21.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.060+0000 7fd7f6ffd700 1 -- 172.21.15.98:0/4037752837 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc0036a0 con 0x7fd8080b5140 2023-03-03T08:00:21.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.060+0000 7fd80cfff700 1 -- 172.21.15.98:0/4037752837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8080d0e80 con 0x7fd8080b5140 2023-03-03T08:00:21.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.061+0000 7fd80cfff700 1 -- 172.21.15.98:0/4037752837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8080cdd00 con 0x7fd8080b5140 2023-03-03T08:00:21.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.062+0000 7fd80cfff700 1 -- 172.21.15.98:0/4037752837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8080cdfe0 con 0x7fd8080b5140 2023-03-03T08:00:21.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.063+0000 7fd7f6ffd700 1 -- 172.21.15.98:0/4037752837 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fd7fc029020 con 0x7fd8080b5140 2023-03-03T08:00:21.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.064+0000 7fd7f6ffd700 1 --2- 172.21.15.98:0/4037752837 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd7e80683c0 0x7fd7e806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:21.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.064+0000 7fd7f6ffd700 1 -- 172.21.15.98:0/4037752837 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5580+0+0 (secure 0 0 0) 0x7fd7fc087310 con 0x7fd8080b5140 2023-03-03T08:00:21.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.065+0000 7fd80659c700 1 --2- 172.21.15.98:0/4037752837 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd7e80683c0 0x7fd7e806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:21.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.065+0000 7fd80659c700 1 --2- 172.21.15.98:0/4037752837 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd7e80683c0 0x7fd7e806a870 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd808155e80 tx=0x7fd7f8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:21.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.067+0000 7fd7f6ffd700 1 -- 172.21.15.98:0/4037752837 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd7fc053bb0 con 0x7fd8080b5140 2023-03-03T08:00:21.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.261+0000 7fd80cfff700 1 -- 172.21.15.98:0/4037752837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd808155a70 con 0x7fd8080b5140 2023-03-03T08:00:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.769+0000 7fd7f6ffd700 1 -- 172.21.15.98:0/4037752837 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v64) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd808155a70 con 0x7fd8080b5140 2023-03-03T08:00:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.778+0000 7fd80cfff700 1 -- 172.21.15.98:0/4037752837 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd7e80683c0 msgr2=0x7fd7e806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.778+0000 7fd80cfff700 1 --2- 172.21.15.98:0/4037752837 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd7e80683c0 0x7fd7e806a870 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd808155e80 tx=0x7fd7f8009040 comp rx=0 tx=0).stop 2023-03-03T08:00:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.778+0000 7fd80cfff700 1 -- 172.21.15.98:0/4037752837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8080b5140 msgr2=0x7fd808154260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.778+0000 7fd80cfff700 1 --2- 172.21.15.98:0/4037752837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8080b5140 0x7fd808154260 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc000c00 tx=0x7fd7fc004750 comp rx=0 tx=0).stop 2023-03-03T08:00:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.779+0000 7fd80cfff700 1 -- 172.21.15.98:0/4037752837 shutdown_connections 2023-03-03T08:00:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.779+0000 7fd80cfff700 1 --2- 172.21.15.98:0/4037752837 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd7e80683c0 0x7fd7e806a870 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.779+0000 7fd80cfff700 1 --2- 172.21.15.98:0/4037752837 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8080b8d10 0x7fd808156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.779+0000 7fd80cfff700 1 --2- 172.21.15.98:0/4037752837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8080b26d0 0x7fd808152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.779+0000 7fd80cfff700 1 --2- 172.21.15.98:0/4037752837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8080b5140 0x7fd808154260 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.779+0000 7fd80cfff700 1 -- 172.21.15.98:0/4037752837 >> 172.21.15.98:0/4037752837 conn(0x7fd808073e70 msgr2=0x7fd8080b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:21.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.783+0000 7fd80cfff700 1 -- 172.21.15.98:0/4037752837 shutdown_connections 2023-03-03T08:00:21.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:21.783+0000 7fd80cfff700 1 -- 172.21.15.98:0/4037752837 wait complete. 2023-03-03T08:00:21.789 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2023-03-03T08:00:21.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-03-03T08:00:22.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.069+0000 7fa5205ff700 1 Processor -- start 2023-03-03T08:00:22.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.069+0000 7fa5205ff700 1 -- start start 2023-03-03T08:00:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.070+0000 7fa5205ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa5180d0270 0x7fa5180cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.070+0000 7fa5205ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa51806b4b0 0x7fa5180d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:22.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.070+0000 7fa5205ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5180cf980 con 0x7fa51815ca70 2023-03-03T08:00:22.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.070+0000 7fa5205ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5180cd5b0 con 0x7fa5180d0270 2023-03-03T08:00:22.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.070+0000 7fa5205ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5180d1800 con 0x7fa51806b4b0 2023-03-03T08:00:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.071+0000 7fa51db9a700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa5180d0270 0x7fa5180cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.071+0000 7fa51eb9c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa51806b4b0 0x7fa5180d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.071+0000 7fa51eb9c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa51806b4b0 0x7fa5180d1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:42828/0 (socket says 172.21.15.98:42828) 2023-03-03T08:00:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.071+0000 7fa51db9a700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa5180d0270 0x7fa5180cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:43424/0 (socket says 172.21.15.98:43424) 2023-03-03T08:00:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.071+0000 7fa51eb9c700 1 -- 172.21.15.98:0/2297391103 learned_addr learned my addr 172.21.15.98:0/2297391103 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.071+0000 7fa51db9a700 1 -- 172.21.15.98:0/2297391103 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa51806b4b0 msgr2=0x7fa5180d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.071+0000 7fa51db9a700 1 --2- 172.21.15.98:0/2297391103 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa51806b4b0 0x7fa5180d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.071+0000 7fa51db9a700 1 -- 172.21.15.98:0/2297391103 >> v1:172.21.15.100:6789/0 conn(0x7fa51815ca70 legacy=0x7fa5180cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.071+0000 7fa51db9a700 1 -- 172.21.15.98:0/2297391103 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5180cd900 con 0x7fa5180d0270 2023-03-03T08:00:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.071+0000 7fa51d399700 1 -- 172.21.15.98:0/2297391103 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637720817 0 0) 0x7fa5180cf980 con 0x7fa51815ca70 2023-03-03T08:00:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.072+0000 7fa51db9a700 1 --2- 172.21.15.98:0/2297391103 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5180d0270 0x7fa5180cf440 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa508009840 tx=0x7fa508009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a63e438cea936579 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.072+0000 7fa51d399700 1 -- 172.21.15.98:0/2297391103 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa508011830 con 0x7fa5180d0270 2023-03-03T08:00:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.072+0000 7fa51d399700 1 -- 172.21.15.98:0/2297391103 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa508011990 con 0x7fa5180d0270 2023-03-03T08:00:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.072+0000 7fa51d399700 1 -- 172.21.15.98:0/2297391103 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa508011d20 con 0x7fa5180d0270 2023-03-03T08:00:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.073+0000 7fa5205ff700 1 -- 172.21.15.98:0/2297391103 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5180d0270 msgr2=0x7fa5180cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.073+0000 7fa5205ff700 1 --2- 172.21.15.98:0/2297391103 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5180d0270 0x7fa5180cf440 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa508009840 tx=0x7fa508009b50 comp rx=0 tx=0).stop 2023-03-03T08:00:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.073+0000 7fa5205ff700 1 -- 172.21.15.98:0/2297391103 shutdown_connections 2023-03-03T08:00:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.073+0000 7fa5205ff700 1 --2- 172.21.15.98:0/2297391103 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa51806b4b0 0x7fa5180d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.073+0000 7fa5205ff700 1 --2- 172.21.15.98:0/2297391103 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5180d0270 0x7fa5180cf440 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.073+0000 7fa5205ff700 1 -- 172.21.15.98:0/2297391103 >> 172.21.15.98:0/2297391103 conn(0x7fa518073e70 msgr2=0x7fa51806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:22.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.073+0000 7fa5205ff700 1 -- 172.21.15.98:0/2297391103 shutdown_connections 2023-03-03T08:00:22.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.073+0000 7fa5205ff700 1 -- 172.21.15.98:0/2297391103 wait complete. 2023-03-03T08:00:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.075+0000 7fa5205ff700 1 Processor -- start 2023-03-03T08:00:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.075+0000 7fa5205ff700 1 -- start start 2023-03-03T08:00:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.076+0000 7fa5205ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa51806b4b0 0x7fa5180ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.076+0000 7fa5205ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5180d0270 0x7fa51815f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.076+0000 7fa5205ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa51815ca70 0x7fa518152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.076+0000 7fa5205ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5181f2340 con 0x7fa51806b4b0 2023-03-03T08:00:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.076+0000 7fa5205ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa518152880 con 0x7fa5180d0270 2023-03-03T08:00:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.076+0000 7fa5205ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5180ca930 con 0x7fa51815ca70 2023-03-03T08:00:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.077+0000 7fa51e39b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa51806b4b0 0x7fa5180ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.077+0000 7fa51e39b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa51806b4b0 0x7fa5180ca250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58144/0 (socket says 172.21.15.98:58144) 2023-03-03T08:00:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.077+0000 7fa51e39b700 1 -- 172.21.15.98:0/598416135 learned_addr learned my addr 172.21.15.98:0/598416135 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.077+0000 7fa51db9a700 1 --2- 172.21.15.98:0/598416135 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5180d0270 0x7fa51815f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.077+0000 7fa51eb9c700 1 --2- 172.21.15.98:0/598416135 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa51815ca70 0x7fa518152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.077+0000 7fa51e39b700 1 -- 172.21.15.98:0/598416135 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa51815ca70 msgr2=0x7fa518152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.077+0000 7fa51e39b700 1 --2- 172.21.15.98:0/598416135 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa51815ca70 0x7fa518152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.077+0000 7fa51e39b700 1 -- 172.21.15.98:0/598416135 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5180d0270 msgr2=0x7fa51815f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.077+0000 7fa51e39b700 1 --2- 172.21.15.98:0/598416135 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5180d0270 0x7fa51815f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.077+0000 7fa51e39b700 1 -- 172.21.15.98:0/598416135 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5080094a0 con 0x7fa51806b4b0 2023-03-03T08:00:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.077+0000 7fa51e39b700 1 --2- 172.21.15.98:0/598416135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa51806b4b0 0x7fa5180ca250 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa51400c580 tx=0x7fa51400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.078+0000 7fa50effd700 1 -- 172.21.15.98:0/598416135 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5140054e0 con 0x7fa51806b4b0 2023-03-03T08:00:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.078+0000 7fa50effd700 1 -- 172.21.15.98:0/598416135 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa514003680 con 0x7fa51806b4b0 2023-03-03T08:00:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.078+0000 7fa50effd700 1 -- 172.21.15.98:0/598416135 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa514003a30 con 0x7fa51806b4b0 2023-03-03T08:00:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.078+0000 7fa5205ff700 1 -- 172.21.15.98:0/598416135 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5180d1aa0 con 0x7fa51806b4b0 2023-03-03T08:00:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.078+0000 7fa5205ff700 1 -- 172.21.15.98:0/598416135 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5180cdbe0 con 0x7fa51806b4b0 2023-03-03T08:00:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.079+0000 7fa5205ff700 1 -- 172.21.15.98:0/598416135 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51815ce40 con 0x7fa51806b4b0 2023-03-03T08:00:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.081+0000 7fa50effd700 1 -- 172.21.15.98:0/598416135 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fa514019030 con 0x7fa51806b4b0 2023-03-03T08:00:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.082+0000 7fa50effd700 1 --2- 172.21.15.98:0/598416135 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa4fc0683c0 0x7fa4fc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.082+0000 7fa50effd700 1 -- 172.21.15.98:0/598416135 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5962+0+0 (secure 0 0 0) 0x7fa514085470 con 0x7fa51806b4b0 2023-03-03T08:00:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.082+0000 7fa51db9a700 1 --2- 172.21.15.98:0/598416135 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa4fc0683c0 0x7fa4fc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.083+0000 7fa51db9a700 1 --2- 172.21.15.98:0/598416135 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa4fc0683c0 0x7fa4fc06a870 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa508009840 tx=0x7fa50801c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:22.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.085+0000 7fa50effd700 1 -- 172.21.15.98:0/598416135 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa514051bf0 con 0x7fa51806b4b0 2023-03-03T08:00:22.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.321+0000 7fa5205ff700 1 -- 172.21.15.98:0/598416135 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa51804e6e0 con 0x7fa51806b4b0 2023-03-03T08:00:22.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.328+0000 7fa50effd700 1 -- 172.21.15.98:0/598416135 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v28) v1 ==== 173+0+0 (secure 0 0 0) 0x7fa5140170c0 con 0x7fa51806b4b0 2023-03-03T08:00:22.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.335+0000 7fa5205ff700 1 -- 172.21.15.98:0/598416135 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa4fc0683c0 msgr2=0x7fa4fc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:22.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.335+0000 7fa5205ff700 1 --2- 172.21.15.98:0/598416135 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa4fc0683c0 0x7fa4fc06a870 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa508009840 tx=0x7fa50801c040 comp rx=0 tx=0).stop 2023-03-03T08:00:22.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.335+0000 7fa5205ff700 1 -- 172.21.15.98:0/598416135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa51806b4b0 msgr2=0x7fa5180ca250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:22.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.335+0000 7fa5205ff700 1 --2- 172.21.15.98:0/598416135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa51806b4b0 0x7fa5180ca250 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa51400c580 tx=0x7fa51400c890 comp rx=0 tx=0).stop 2023-03-03T08:00:22.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.335+0000 7fa5205ff700 1 -- 172.21.15.98:0/598416135 shutdown_connections 2023-03-03T08:00:22.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.335+0000 7fa5205ff700 1 --2- 172.21.15.98:0/598416135 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa4fc0683c0 0x7fa4fc06a870 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.335+0000 7fa5205ff700 1 --2- 172.21.15.98:0/598416135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa51806b4b0 0x7fa5180ca250 secure :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa51400c580 tx=0x7fa51400c890 comp rx=0 tx=0).stop 2023-03-03T08:00:22.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.335+0000 7fa5205ff700 1 --2- 172.21.15.98:0/598416135 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa51815ca70 0x7fa518152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.335+0000 7fa5205ff700 1 --2- 172.21.15.98:0/598416135 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5180d0270 0x7fa51815f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.335+0000 7fa5205ff700 1 -- 172.21.15.98:0/598416135 >> 172.21.15.98:0/598416135 conn(0x7fa518073e70 msgr2=0x7fa5180b6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:22.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.336+0000 7fa5205ff700 1 -- 172.21.15.98:0/598416135 shutdown_connections 2023-03-03T08:00:22.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.336+0000 7fa5205ff700 1 -- 172.21.15.98:0/598416135 wait complete. 2023-03-03T08:00:22.347 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 9 and data pool 10 2023-03-03T08:00:22.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:00:22.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.625+0000 7f30affff700 1 Processor -- start 2023-03-03T08:00:22.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.626+0000 7f30affff700 1 -- start start 2023-03-03T08:00:22.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.627+0000 7f30affff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f30b0146360 0x7f30b0160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:22.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.627+0000 7f30affff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30b00b4ab0 0x7f30b00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:22.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.627+0000 7f30affff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b01614c0 con 0x7f30b00731e0 2023-03-03T08:00:22.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.627+0000 7f30affff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30b00b5440 con 0x7f30b0146360 2023-03-03T08:00:22.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.627+0000 7f30affff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b00d10f0 con 0x7f30b00b4ab0 2023-03-03T08:00:22.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.627+0000 7f30aeffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f30b0146360 0x7f30b0160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:22.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.628+0000 7f30aeffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f30b0146360 0x7f30b0160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58146/0 (socket says 172.21.15.98:58146) 2023-03-03T08:00:22.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.628+0000 7f30aeffd700 1 -- 172.21.15.98:0/2337135464 learned_addr learned my addr 172.21.15.98:0/2337135464 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:22.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.628+0000 7f30a7fff700 1 --2- 172.21.15.98:0/2337135464 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30b00b4ab0 0x7f30b00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:22.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.628+0000 7f30aeffd700 1 -- 172.21.15.98:0/2337135464 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30b00b4ab0 msgr2=0x7f30b00b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:22.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.628+0000 7f30aeffd700 1 --2- 172.21.15.98:0/2337135464 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30b00b4ab0 0x7f30b00b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.628+0000 7f30aeffd700 1 -- 172.21.15.98:0/2337135464 >> v1:172.21.15.100:6789/0 conn(0x7f30b00731e0 legacy=0x7f30b0073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:22.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.628+0000 7f30aeffd700 1 -- 172.21.15.98:0/2337135464 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b00cdf20 con 0x7f30b0146360 2023-03-03T08:00:22.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.628+0000 7f30aeffd700 1 --2- 172.21.15.98:0/2337135464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30b0146360 0x7f30b0160f80 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f30a00098f0 tx=0x7f30a0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e31ae65a69afc1c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:22.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.629+0000 7f30ae7fc700 1 -- 172.21.15.98:0/2337135464 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a000b8a0 con 0x7f30b0146360 2023-03-03T08:00:22.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.629+0000 7f30ae7fc700 1 -- 172.21.15.98:0/2337135464 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a000ba00 con 0x7f30b0146360 2023-03-03T08:00:22.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.629+0000 7f30ae7fc700 1 -- 172.21.15.98:0/2337135464 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a000bd90 con 0x7f30b0146360 2023-03-03T08:00:22.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.629+0000 7f30affff700 1 -- 172.21.15.98:0/2337135464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30b0146360 msgr2=0x7f30b0160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:22.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.629+0000 7f30affff700 1 --2- 172.21.15.98:0/2337135464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30b0146360 0x7f30b0160f80 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f30a00098f0 tx=0x7f30a0009c00 comp rx=0 tx=0).stop 2023-03-03T08:00:22.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.629+0000 7f30affff700 1 -- 172.21.15.98:0/2337135464 shutdown_connections 2023-03-03T08:00:22.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.629+0000 7f30affff700 1 --2- 172.21.15.98:0/2337135464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30b0146360 0x7f30b0160f80 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.629+0000 7f30affff700 1 --2- 172.21.15.98:0/2337135464 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30b00b4ab0 0x7f30b00b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.629+0000 7f30affff700 1 -- 172.21.15.98:0/2337135464 >> 172.21.15.98:0/2337135464 conn(0x7f30b006b8e0 msgr2=0x7f30b00cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:22.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.629+0000 7f30affff700 1 -- 172.21.15.98:0/2337135464 shutdown_connections 2023-03-03T08:00:22.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.629+0000 7f30affff700 1 -- 172.21.15.98:0/2337135464 wait complete. 2023-03-03T08:00:22.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.631+0000 7f30affff700 1 Processor -- start 2023-03-03T08:00:22.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.632+0000 7f30affff700 1 -- start start 2023-03-03T08:00:22.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30affff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30b00731e0 0x7f30b01f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:22.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30affff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30b00b4ab0 0x7f30b0151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:22.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30affff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30b0146360 0x7f30b015ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:22.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30affff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b01f5c10 con 0x7f30b00731e0 2023-03-03T08:00:22.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30affff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b015df90 con 0x7f30b00b4ab0 2023-03-03T08:00:22.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30affff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b015e230 con 0x7f30b0146360 2023-03-03T08:00:22.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30aeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30b00731e0 0x7f30b01f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:22.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30aeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30b00731e0 0x7f30b01f56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58152/0 (socket says 172.21.15.98:58152) 2023-03-03T08:00:22.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30aeffd700 1 -- 172.21.15.98:0/1622480915 learned_addr learned my addr 172.21.15.98:0/1622480915 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:22.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30a7fff700 1 --2- 172.21.15.98:0/1622480915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30b00b4ab0 0x7f30b0151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:22.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30af7fe700 1 --2- 172.21.15.98:0/1622480915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30b0146360 0x7f30b015ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:22.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30aeffd700 1 -- 172.21.15.98:0/1622480915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30b0146360 msgr2=0x7f30b015ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:22.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30aeffd700 1 --2- 172.21.15.98:0/1622480915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30b0146360 0x7f30b015ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30aeffd700 1 -- 172.21.15.98:0/1622480915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30b00b4ab0 msgr2=0x7f30b0151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:22.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30aeffd700 1 --2- 172.21.15.98:0/1622480915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30b00b4ab0 0x7f30b0151b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.633+0000 7f30aeffd700 1 -- 172.21.15.98:0/1622480915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a0009550 con 0x7f30b00731e0 2023-03-03T08:00:22.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.634+0000 7f30a7fff700 1 --2- 172.21.15.98:0/1622480915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30b00b4ab0 0x7f30b0151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:22.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.634+0000 7f30aeffd700 1 --2- 172.21.15.98:0/1622480915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30b00731e0 0x7f30b01f56d0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f30a0009c40 tx=0x7f30a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:22.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.634+0000 7f30a77fe700 1 -- 172.21.15.98:0/1622480915 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a00036a0 con 0x7f30b00731e0 2023-03-03T08:00:22.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.634+0000 7f30a77fe700 1 -- 172.21.15.98:0/1622480915 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a001e070 con 0x7f30b00731e0 2023-03-03T08:00:22.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.634+0000 7f30a77fe700 1 -- 172.21.15.98:0/1622480915 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a0003a40 con 0x7f30b00731e0 2023-03-03T08:00:22.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.634+0000 7f30affff700 1 -- 172.21.15.98:0/1622480915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b00cd960 con 0x7f30b00731e0 2023-03-03T08:00:22.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.634+0000 7f30affff700 1 -- 172.21.15.98:0/1622480915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b00cd680 con 0x7f30b00731e0 2023-03-03T08:00:22.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.636+0000 7f30affff700 1 -- 172.21.15.98:0/1622480915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30b00cdc40 con 0x7f30b00731e0 2023-03-03T08:00:22.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.637+0000 7f30a77fe700 1 -- 172.21.15.98:0/1622480915 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f30a000f050 con 0x7f30b00731e0 2023-03-03T08:00:22.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.638+0000 7f30a77fe700 1 --2- 172.21.15.98:0/1622480915 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f30900683c0 0x7f309006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:22.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.638+0000 7f30a77fe700 1 -- 172.21.15.98:0/1622480915 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f30a0020070 con 0x7f30b00731e0 2023-03-03T08:00:22.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.638+0000 7f30a7fff700 1 --2- 172.21.15.98:0/1622480915 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f30900683c0 0x7f309006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:22.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.639+0000 7f30a7fff700 1 --2- 172.21.15.98:0/1622480915 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f30900683c0 0x7f309006a870 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f30b00b2080 tx=0x7f3098008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:22.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.642+0000 7f30a77fe700 1 -- 172.21.15.98:0/1622480915 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f30a0053aa0 con 0x7f30b00731e0 2023-03-03T08:00:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.840+0000 7f30affff700 1 -- 172.21.15.98:0/1622480915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30b006bfc0 con 0x7f30b00731e0 2023-03-03T08:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.841+0000 7f30a77fe700 1 -- 172.21.15.98:0/1622480915 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+17934 (secure 0 0 0) 0x7f30a00570c0 con 0x7f30b00731e0 2023-03-03T08:00:22.843 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:22.843 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":65,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:00:22.323041+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:00:19.823120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-03-03T08:00:21.324346+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.844+0000 7f30affff700 1 -- 172.21.15.98:0/1622480915 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f30900683c0 msgr2=0x7f309006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.844+0000 7f30affff700 1 --2- 172.21.15.98:0/1622480915 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f30900683c0 0x7f309006a870 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f30b00b2080 tx=0x7f3098008040 comp rx=0 tx=0).stop 2023-03-03T08:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.844+0000 7f30affff700 1 -- 172.21.15.98:0/1622480915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30b00731e0 msgr2=0x7f30b01f56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.844+0000 7f30affff700 1 --2- 172.21.15.98:0/1622480915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30b00731e0 0x7f30b01f56d0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f30a0009c40 tx=0x7f30a0004750 comp rx=0 tx=0).stop 2023-03-03T08:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.844+0000 7f30affff700 1 -- 172.21.15.98:0/1622480915 shutdown_connections 2023-03-03T08:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.844+0000 7f30affff700 1 --2- 172.21.15.98:0/1622480915 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f30900683c0 0x7f309006a870 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.844+0000 7f30affff700 1 --2- 172.21.15.98:0/1622480915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30b00731e0 0x7f30b01f56d0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.844+0000 7f30affff700 1 --2- 172.21.15.98:0/1622480915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30b0146360 0x7f30b015ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.844+0000 7f30affff700 1 --2- 172.21.15.98:0/1622480915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30b00b4ab0 0x7f30b0151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.844+0000 7f30affff700 1 -- 172.21.15.98:0/1622480915 >> 172.21.15.98:0/1622480915 conn(0x7f30b006b8e0 msgr2=0x7f30b015aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.844+0000 7f30affff700 1 -- 172.21.15.98:0/1622480915 shutdown_connections 2023-03-03T08:00:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:22.844+0000 7f30affff700 1 -- 172.21.15.98:0/1622480915 wait complete. 2023-03-03T08:00:22.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:00:23.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.116+0000 7fe7117ff700 1 Processor -- start 2023-03-03T08:00:23.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.117+0000 7fe7117ff700 1 -- start start 2023-03-03T08:00:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.118+0000 7fe7117ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe70c0b8b90 0x7fe70c0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.118+0000 7fe7117ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe70c0b5410 0x7fe70c0d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.118+0000 7fe7117ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70c0d0ef0 con 0x7fe70c0b87c0 2023-03-03T08:00:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.118+0000 7fe7117ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe70c0d0600 con 0x7fe70c0b5410 2023-03-03T08:00:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.118+0000 7fe7117ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c0cd970 con 0x7fe70c0b8b90 2023-03-03T08:00:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.119+0000 7fe70b7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe70c0b5410 0x7fe70c0d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.119+0000 7fe70b7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe70c0b5410 0x7fe70c0d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58154/0 (socket says 172.21.15.98:58154) 2023-03-03T08:00:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.119+0000 7fe70b7fe700 1 -- 172.21.15.98:0/494244788 learned_addr learned my addr 172.21.15.98:0/494244788 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.119+0000 7fe70a7fc700 1 --2- 172.21.15.98:0/494244788 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe70c0b8b90 0x7fe70c0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.119+0000 7fe70b7fe700 1 -- 172.21.15.98:0/494244788 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe70c0b8b90 msgr2=0x7fe70c0cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.119+0000 7fe70b7fe700 1 --2- 172.21.15.98:0/494244788 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe70c0b8b90 0x7fe70c0cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.119+0000 7fe70b7fe700 1 -- 172.21.15.98:0/494244788 >> v1:172.21.15.100:6789/0 conn(0x7fe70c0b87c0 legacy=0x7fe70c0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.119+0000 7fe70b7fe700 1 -- 172.21.15.98:0/494244788 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c0cdc50 con 0x7fe70c0b5410 2023-03-03T08:00:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.120+0000 7fe70b7fe700 1 --2- 172.21.15.98:0/494244788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b5410 0x7fe70c0d13d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe704009e50 tx=0x7fe70400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3f155f682fdff5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.120+0000 7fe709ffb700 1 -- 172.21.15.98:0/494244788 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70400e070 con 0x7fe70c0b5410 2023-03-03T08:00:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.120+0000 7fe709ffb700 1 -- 172.21.15.98:0/494244788 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70400b910 con 0x7fe70c0b5410 2023-03-03T08:00:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.120+0000 7fe709ffb700 1 -- 172.21.15.98:0/494244788 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70400bcc0 con 0x7fe70c0b5410 2023-03-03T08:00:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.120+0000 7fe7117ff700 1 -- 172.21.15.98:0/494244788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b5410 msgr2=0x7fe70c0d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.120+0000 7fe7117ff700 1 --2- 172.21.15.98:0/494244788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b5410 0x7fe70c0d13d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe704009e50 tx=0x7fe70400b330 comp rx=0 tx=0).stop 2023-03-03T08:00:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.121+0000 7fe7117ff700 1 -- 172.21.15.98:0/494244788 shutdown_connections 2023-03-03T08:00:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.121+0000 7fe7117ff700 1 --2- 172.21.15.98:0/494244788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b5410 0x7fe70c0d13d0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.121+0000 7fe7117ff700 1 --2- 172.21.15.98:0/494244788 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe70c0b8b90 0x7fe70c0cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.121+0000 7fe7117ff700 1 -- 172.21.15.98:0/494244788 >> 172.21.15.98:0/494244788 conn(0x7fe70c06aee0 msgr2=0x7fe70c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.121+0000 7fe7117ff700 1 -- 172.21.15.98:0/494244788 shutdown_connections 2023-03-03T08:00:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.121+0000 7fe7117ff700 1 -- 172.21.15.98:0/494244788 wait complete. 2023-03-03T08:00:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.123+0000 7fe7117ff700 1 Processor -- start 2023-03-03T08:00:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.123+0000 7fe7117ff700 1 -- start start 2023-03-03T08:00:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.124+0000 7fe7117ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b5410 0x7fe70c14f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.124+0000 7fe7117ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe70c0b87c0 0x7fe70c1e5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:23.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.124+0000 7fe7117ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe70c0b8b90 0x7fe70c1e5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:23.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.124+0000 7fe7117ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c14f7d0 con 0x7fe70c0b5410 2023-03-03T08:00:23.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.124+0000 7fe7117ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c1f3330 con 0x7fe70c0b8b90 2023-03-03T08:00:23.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.124+0000 7fe7117ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c1f35b0 con 0x7fe70c0b87c0 2023-03-03T08:00:23.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.124+0000 7fe70affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b5410 0x7fe70c14f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:23.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.124+0000 7fe70affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b5410 0x7fe70c14f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58170/0 (socket says 172.21.15.98:58170) 2023-03-03T08:00:23.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.124+0000 7fe70affd700 1 -- 172.21.15.98:0/1249252811 learned_addr learned my addr 172.21.15.98:0/1249252811 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:23.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.124+0000 7fe70a7fc700 1 --2- 172.21.15.98:0/1249252811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe70c0b87c0 0x7fe70c1e5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:23.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.124+0000 7fe70b7fe700 1 --2- 172.21.15.98:0/1249252811 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe70c0b8b90 0x7fe70c1e5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:23.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.125+0000 7fe70affd700 1 -- 172.21.15.98:0/1249252811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe70c0b87c0 msgr2=0x7fe70c1e5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:23.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.125+0000 7fe70affd700 1 --2- 172.21.15.98:0/1249252811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe70c0b87c0 0x7fe70c1e5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:23.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.125+0000 7fe70affd700 1 -- 172.21.15.98:0/1249252811 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe70c0b8b90 msgr2=0x7fe70c1e5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:23.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.125+0000 7fe70affd700 1 --2- 172.21.15.98:0/1249252811 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe70c0b8b90 0x7fe70c1e5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:23.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.125+0000 7fe70affd700 1 -- 172.21.15.98:0/1249252811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe704009b30 con 0x7fe70c0b5410 2023-03-03T08:00:23.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.125+0000 7fe70affd700 1 --2- 172.21.15.98:0/1249252811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b5410 0x7fe70c14f290 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00c4e0 tx=0x7fe6fc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:23.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.125+0000 7fe6f37fe700 1 -- 172.21.15.98:0/1249252811 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc0052d0 con 0x7fe70c0b5410 2023-03-03T08:00:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.125+0000 7fe6f37fe700 1 -- 172.21.15.98:0/1249252811 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6fc005430 con 0x7fe70c0b5410 2023-03-03T08:00:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.125+0000 7fe6f37fe700 1 -- 172.21.15.98:0/1249252811 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc0038c0 con 0x7fe70c0b5410 2023-03-03T08:00:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.125+0000 7fe7117ff700 1 -- 172.21.15.98:0/1249252811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe70c0cdf30 con 0x7fe70c0b5410 2023-03-03T08:00:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.125+0000 7fe7117ff700 1 -- 172.21.15.98:0/1249252811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe70c073a80 con 0x7fe70c0b5410 2023-03-03T08:00:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.127+0000 7fe7117ff700 1 -- 172.21.15.98:0/1249252811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe70c0ce0c0 con 0x7fe70c0b5410 2023-03-03T08:00:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.128+0000 7fe6f37fe700 1 -- 172.21.15.98:0/1249252811 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fe6fc019030 con 0x7fe70c0b5410 2023-03-03T08:00:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.129+0000 7fe6f37fe700 1 --2- 172.21.15.98:0/1249252811 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6f80683f0 0x7fe6f806a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.129+0000 7fe6f37fe700 1 -- 172.21.15.98:0/1249252811 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe6fc085280 con 0x7fe70c0b5410 2023-03-03T08:00:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.129+0000 7fe70a7fc700 1 --2- 172.21.15.98:0/1249252811 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6f80683f0 0x7fe6f806a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.130+0000 7fe70a7fc700 1 --2- 172.21.15.98:0/1249252811 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6f80683f0 0x7fe6f806a8a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe70c067680 tx=0x7fe700008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.133+0000 7fe6f37fe700 1 -- 172.21.15.98:0/1249252811 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe6fc051920 con 0x7fe70c0b5410 2023-03-03T08:00:23.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.329+0000 7fe7117ff700 1 -- 172.21.15.98:0/1249252811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe70c0729b0 con 0x7fe70c0b5410 2023-03-03T08:00:23.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.334+0000 7fe6f37fe700 1 -- 172.21.15.98:0/1249252811 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+17921 (secure 0 0 0) 0x7fe6fc054f40 con 0x7fe70c0b5410 2023-03-03T08:00:23.335 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:23.336 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":66,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:00:23.326015+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:00:19.823120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-03-03T08:00:21.324346+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:00:23.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.337+0000 7fe7117ff700 1 -- 172.21.15.98:0/1249252811 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6f80683f0 msgr2=0x7fe6f806a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:23.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.337+0000 7fe7117ff700 1 --2- 172.21.15.98:0/1249252811 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6f80683f0 0x7fe6f806a8a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe70c067680 tx=0x7fe700008040 comp rx=0 tx=0).stop 2023-03-03T08:00:23.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.337+0000 7fe7117ff700 1 -- 172.21.15.98:0/1249252811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b5410 msgr2=0x7fe70c14f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:23.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.337+0000 7fe7117ff700 1 --2- 172.21.15.98:0/1249252811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b5410 0x7fe70c14f290 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00c4e0 tx=0x7fe6fc00c8a0 comp rx=0 tx=0).stop 2023-03-03T08:00:23.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.337+0000 7fe7117ff700 1 -- 172.21.15.98:0/1249252811 shutdown_connections 2023-03-03T08:00:23.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.337+0000 7fe7117ff700 1 --2- 172.21.15.98:0/1249252811 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe6f80683f0 0x7fe6f806a8a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:23.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.337+0000 7fe7117ff700 1 --2- 172.21.15.98:0/1249252811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe70c0b5410 0x7fe70c14f290 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:23.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.337+0000 7fe7117ff700 1 --2- 172.21.15.98:0/1249252811 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe70c0b8b90 0x7fe70c1e5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:23.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.337+0000 7fe7117ff700 1 --2- 172.21.15.98:0/1249252811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe70c0b87c0 0x7fe70c1e5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:23.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.337+0000 7fe7117ff700 1 -- 172.21.15.98:0/1249252811 >> 172.21.15.98:0/1249252811 conn(0x7fe70c06aee0 msgr2=0x7fe70c0b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:23.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.337+0000 7fe7117ff700 1 -- 172.21.15.98:0/1249252811 shutdown_connections 2023-03-03T08:00:23.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.337+0000 7fe7117ff700 1 -- 172.21.15.98:0/1249252811 wait complete. 2023-03-03T08:00:23.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-03-03T08:00:23.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.616+0000 7f5b9359d700 1 Processor -- start 2023-03-03T08:00:23.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.616+0000 7f5b9359d700 1 -- start start 2023-03-03T08:00:23.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.617+0000 7f5b9359d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b8c15c7c0 0x7f5b8c06dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:23.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.617+0000 7f5b9359d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5b8c1603b0 0x7f5b8c06e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:23.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.617+0000 7f5b9359d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8c0cb210 con 0x7f5b8c0cac50 2023-03-03T08:00:23.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.617+0000 7f5b9359d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b8c0cb4f0 con 0x7f5b8c1603b0 2023-03-03T08:00:23.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.617+0000 7f5b9359d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c0cb7d0 con 0x7f5b8c15c7c0 2023-03-03T08:00:23.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.618+0000 7f5b91d9a700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5b8c1603b0 0x7f5b8c06e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:23.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.618+0000 7f5b91d9a700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5b8c1603b0 0x7f5b8c06e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:43454/0 (socket says 172.21.15.98:43454) 2023-03-03T08:00:23.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.618+0000 7f5b92d9c700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f5b8c0cac50 0x7f5b8c06c550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:54880/0 (socket says 172.21.15.98:54880) 2023-03-03T08:00:23.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.618+0000 7f5b91d9a700 1 -- 172.21.15.98:0/2904308867 learned_addr learned my addr 172.21.15.98:0/2904308867 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:23.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.618+0000 7f5b9259b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b8c15c7c0 0x7f5b8c06dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:23.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.619+0000 7f5b91599700 1 -- 172.21.15.98:0/2904308867 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737327504 0 0) 0x7f5b8c0cb210 con 0x7f5b8c0cac50 2023-03-03T08:00:23.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.619+0000 7f5b9259b700 1 -- 172.21.15.98:0/2904308867 >> v2:172.21.15.100:3300/0 conn(0x7f5b8c1603b0 msgr2=0x7f5b8c06e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:23.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.619+0000 7f5b9259b700 1 --2- 172.21.15.98:0/2904308867 >> v2:172.21.15.100:3300/0 conn(0x7f5b8c1603b0 0x7f5b8c06e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:23.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.619+0000 7f5b9259b700 1 -- 172.21.15.98:0/2904308867 >> v1:172.21.15.100:6789/0 conn(0x7f5b8c0cac50 legacy=0x7f5b8c06c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:23.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.619+0000 7f5b9259b700 1 -- 172.21.15.98:0/2904308867 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b8c0cbab0 con 0x7f5b8c15c7c0 2023-03-03T08:00:23.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.619+0000 7f5b91d9a700 1 --2- 172.21.15.98:0/2904308867 >> v2:172.21.15.100:3300/0 conn(0x7f5b8c1603b0 0x7f5b8c06e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:00:23.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.619+0000 7f5b9259b700 1 --2- 172.21.15.98:0/2904308867 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b8c15c7c0 0x7f5b8c06dd40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5b88009820 tx=0x7f5b88009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5d8e5776e2238f2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:23.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.620+0000 7f5b91599700 1 -- 172.21.15.98:0/2904308867 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b88005e30 con 0x7f5b8c15c7c0 2023-03-03T08:00:23.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.620+0000 7f5b91599700 1 -- 172.21.15.98:0/2904308867 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b88006030 con 0x7f5b8c15c7c0 2023-03-03T08:00:23.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.620+0000 7f5b91599700 1 -- 172.21.15.98:0/2904308867 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b880063c0 con 0x7f5b8c15c7c0 2023-03-03T08:00:23.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.620+0000 7f5b9359d700 1 -- 172.21.15.98:0/2904308867 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b8c15c7c0 msgr2=0x7f5b8c06dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:23.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.620+0000 7f5b9359d700 1 --2- 172.21.15.98:0/2904308867 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b8c15c7c0 0x7f5b8c06dd40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5b88009820 tx=0x7f5b88009b30 comp rx=0 tx=0).stop 2023-03-03T08:00:23.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.620+0000 7f5b9359d700 1 -- 172.21.15.98:0/2904308867 shutdown_connections 2023-03-03T08:00:23.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.620+0000 7f5b9359d700 1 --2- 172.21.15.98:0/2904308867 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b8c15c7c0 0x7f5b8c06dd40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:23.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.620+0000 7f5b9359d700 1 --2- 172.21.15.98:0/2904308867 >> v2:172.21.15.100:3300/0 conn(0x7f5b8c1603b0 0x7f5b8c06e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:23.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.620+0000 7f5b9359d700 1 -- 172.21.15.98:0/2904308867 >> 172.21.15.98:0/2904308867 conn(0x7f5b8c0ccae0 msgr2=0x7f5b8c0cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:23.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.620+0000 7f5b9359d700 1 -- 172.21.15.98:0/2904308867 shutdown_connections 2023-03-03T08:00:23.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.620+0000 7f5b9359d700 1 -- 172.21.15.98:0/2904308867 wait complete. 2023-03-03T08:00:23.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.622+0000 7f5b9359d700 1 Processor -- start 2023-03-03T08:00:23.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.622+0000 7f5b9359d700 1 -- start start 2023-03-03T08:00:23.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.623+0000 7f5b9359d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b8c0cac50 0x7f5b8c0b1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:23.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.623+0000 7f5b9359d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b8c15c7c0 0x7f5b8c15ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:23.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.623+0000 7f5b9359d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b8c1603b0 0x7f5b8c15f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:23.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.623+0000 7f5b9359d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c0b1970 con 0x7f5b8c1603b0 2023-03-03T08:00:23.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.623+0000 7f5b9359d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c15f6c0 con 0x7f5b8c0cac50 2023-03-03T08:00:23.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.623+0000 7f5b9359d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c0c31e0 con 0x7f5b8c15c7c0 2023-03-03T08:00:23.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.624+0000 7f5b92d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b8c1603b0 0x7f5b8c15f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:23.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.624+0000 7f5b92d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b8c1603b0 0x7f5b8c15f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58186/0 (socket says 172.21.15.98:58186) 2023-03-03T08:00:23.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.624+0000 7f5b92d9c700 1 -- 172.21.15.98:0/3537708060 learned_addr learned my addr 172.21.15.98:0/3537708060 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:23.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.624+0000 7f5b9259b700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b8c0cac50 0x7f5b8c0b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:23.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.624+0000 7f5b91d9a700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b8c15c7c0 0x7f5b8c15ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:23.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.624+0000 7f5b92d9c700 1 -- 172.21.15.98:0/3537708060 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b8c15c7c0 msgr2=0x7f5b8c15ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:23.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.624+0000 7f5b92d9c700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b8c15c7c0 0x7f5b8c15ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:23.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.624+0000 7f5b92d9c700 1 -- 172.21.15.98:0/3537708060 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b8c0cac50 msgr2=0x7f5b8c0b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:23.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.624+0000 7f5b92d9c700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b8c0cac50 0x7f5b8c0b1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:23.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.624+0000 7f5b92d9c700 1 -- 172.21.15.98:0/3537708060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c009550 con 0x7f5b8c1603b0 2023-03-03T08:00:23.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.624+0000 7f5b9259b700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b8c0cac50 0x7f5b8c0b1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:23.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.624+0000 7f5b91d9a700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b8c15c7c0 0x7f5b8c15ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:23.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.625+0000 7f5b92d9c700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b8c1603b0 0x7f5b8c15f180 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400cb60 tx=0x7f5b8400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:23.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.625+0000 7f5b82ffd700 1 -- 172.21.15.98:0/3537708060 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8400ab50 con 0x7f5b8c1603b0 2023-03-03T08:00:23.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.625+0000 7f5b82ffd700 1 -- 172.21.15.98:0/3537708060 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8400acb0 con 0x7f5b8c1603b0 2023-03-03T08:00:23.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.625+0000 7f5b82ffd700 1 -- 172.21.15.98:0/3537708060 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b84007310 con 0x7f5b8c1603b0 2023-03-03T08:00:23.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.625+0000 7f5b9359d700 1 -- 172.21.15.98:0/3537708060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b88009430 con 0x7f5b8c1603b0 2023-03-03T08:00:23.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.625+0000 7f5b9359d700 1 -- 172.21.15.98:0/3537708060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8c15cd20 con 0x7f5b8c1603b0 2023-03-03T08:00:23.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.626+0000 7f5b9359d700 1 -- 172.21.15.98:0/3537708060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8c15cb90 con 0x7f5b8c1603b0 2023-03-03T08:00:23.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.627+0000 7f5b82ffd700 1 -- 172.21.15.98:0/3537708060 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f5b84010070 con 0x7f5b8c1603b0 2023-03-03T08:00:23.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.629+0000 7f5b82ffd700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5b700683f0 0x7f5b7006a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:23.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.629+0000 7f5b82ffd700 1 -- 172.21.15.98:0/3537708060 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5b84085770 con 0x7f5b8c1603b0 2023-03-03T08:00:23.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.629+0000 7f5b9259b700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5b700683f0 0x7f5b7006a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:23.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.630+0000 7f5b9259b700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5b700683f0 0x7f5b7006a8a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5b88009b70 tx=0x7f5b88008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:23.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.632+0000 7f5b82ffd700 1 -- 172.21.15.98:0/3537708060 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5b84051e10 con 0x7f5b8c1603b0 2023-03-03T08:00:23.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:23.873+0000 7f5b9359d700 1 -- 172.21.15.98:0/3537708060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5b8c073620 con 0x7f5b8c1603b0 2023-03-03T08:00:24.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:24.789+0000 7f5b82ffd700 1 -- 172.21.15.98:0/3537708060 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v30) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5b84055430 con 0x7f5b8c1603b0 2023-03-03T08:00:24.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:24.792+0000 7f5b9359d700 1 -- 172.21.15.98:0/3537708060 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5b700683f0 msgr2=0x7f5b7006a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:24.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:24.792+0000 7f5b9359d700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5b700683f0 0x7f5b7006a8a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5b88009b70 tx=0x7f5b88008070 comp rx=0 tx=0).stop 2023-03-03T08:00:24.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:24.792+0000 7f5b9359d700 1 -- 172.21.15.98:0/3537708060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b8c1603b0 msgr2=0x7f5b8c15f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:24.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:24.792+0000 7f5b9359d700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b8c1603b0 0x7f5b8c15f180 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400cb60 tx=0x7f5b8400cf20 comp rx=0 tx=0).stop 2023-03-03T08:00:24.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:24.792+0000 7f5b9359d700 1 -- 172.21.15.98:0/3537708060 shutdown_connections 2023-03-03T08:00:24.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:24.792+0000 7f5b9359d700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5b700683f0 0x7f5b7006a8a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:24.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:24.792+0000 7f5b9359d700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b8c1603b0 0x7f5b8c15f180 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:24.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:24.792+0000 7f5b9359d700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b8c0cac50 0x7f5b8c0b1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:24.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:24.792+0000 7f5b9359d700 1 --2- 172.21.15.98:0/3537708060 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b8c15c7c0 0x7f5b8c15ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:24.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:24.792+0000 7f5b9359d700 1 -- 172.21.15.98:0/3537708060 >> 172.21.15.98:0/3537708060 conn(0x7f5b8c0ccae0 msgr2=0x7f5b8c15b660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:24.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:24.792+0000 7f5b9359d700 1 -- 172.21.15.98:0/3537708060 shutdown_connections 2023-03-03T08:00:24.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:24.792+0000 7f5b9359d700 1 -- 172.21.15.98:0/3537708060 wait complete. 2023-03-03T08:00:24.824 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-03-03T08:00:24.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-03-03T08:00:25.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.075+0000 7fc373fff700 1 Processor -- start 2023-03-03T08:00:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.075+0000 7fc373fff700 1 -- start start 2023-03-03T08:00:25.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.076+0000 7fc373fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc3741609b0 0x7fc374160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:25.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.076+0000 7fc373fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3741612c0 con 0x7fc3740b4aa0 2023-03-03T08:00:25.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.076+0000 7fc373fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37415dd00 con 0x7fc3740cf570 2023-03-03T08:00:25.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.076+0000 7fc373fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3740cebb0 con 0x7fc3741609b0 2023-03-03T08:00:25.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.077+0000 7fc3737fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc3740b4aa0 0x7fc3740d0bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44852/0 (socket says 172.21.15.98:44852) 2023-03-03T08:00:25.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.077+0000 7fc3737fe700 1 -- 172.21.15.98:0/3825948457 learned_addr learned my addr 172.21.15.98:0/3825948457 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:25.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.077+0000 7fc372ffd700 1 --2- 172.21.15.98:0/3825948457 >> v2:172.21.15.100:3300/0 conn(0x7fc3741609b0 0x7fc374160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:25.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.077+0000 7fc3727fc700 1 -- 172.21.15.98:0/3825948457 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522070289 0 0) 0x7fc3741612c0 con 0x7fc3740b4aa0 2023-03-03T08:00:25.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.077+0000 7fc3727fc700 1 -- 172.21.15.98:0/3825948457 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3600035b0 con 0x7fc3740b4aa0 2023-03-03T08:00:25.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc3727fc700 1 -- 172.21.15.98:0/3825948457 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2490578013 0 0) 0x7fc3600035b0 con 0x7fc3740b4aa0 2023-03-03T08:00:25.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc3727fc700 1 -- 172.21.15.98:0/3825948457 >> v2:172.21.15.100:3300/0 conn(0x7fc3741609b0 msgr2=0x7fc374160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc3727fc700 1 --2- 172.21.15.98:0/3825948457 >> v2:172.21.15.100:3300/0 conn(0x7fc3741609b0 0x7fc374160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc3727fc700 1 -- 172.21.15.98:0/3825948457 >> v1:172.21.15.100:6789/0 conn(0x7fc3740cf570 legacy=0x7fc37415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc3727fc700 1 -- 172.21.15.98:0/3825948457 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3740cd780 con 0x7fc3740b4aa0 2023-03-03T08:00:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc3727fc700 1 -- 172.21.15.98:0/3825948457 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7fc36c001670 con 0x7fc3740b4aa0 2023-03-03T08:00:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc3727fc700 1 -- 172.21.15.98:0/3825948457 >> v1:172.21.15.98:6789/0 conn(0x7fc3740b4aa0 legacy=0x7fc3740d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc3727fc700 1 --2- 172.21.15.98:0/3825948457 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc360003a50 0x7fc360003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc3727fc700 1 --2- 172.21.15.98:0/3825948457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc360004850 0x7fc360004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:25.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc3727fc700 1 --2- 172.21.15.98:0/3825948457 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc360005200 0x7fc36000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:25.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc3727fc700 1 -- 172.21.15.98:0/3825948457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36000c0e0 con 0x7fc360004850 2023-03-03T08:00:25.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc3727fc700 1 -- 172.21.15.98:0/3825948457 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36000c220 con 0x7fc360005200 2023-03-03T08:00:25.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc3727fc700 1 -- 172.21.15.98:0/3825948457 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36000c3e0 con 0x7fc360003a50 2023-03-03T08:00:25.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc36a7fc700 1 --2- 172.21.15.98:0/3825948457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc360004850 0x7fc360004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:25.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc3737fe700 1 --2- 172.21.15.98:0/3825948457 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc360005200 0x7fc36000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:25.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.078+0000 7fc372ffd700 1 --2- 172.21.15.98:0/3825948457 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc360003a50 0x7fc360003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:25.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.079+0000 7fc36a7fc700 1 -- 172.21.15.98:0/3825948457 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc360003a50 msgr2=0x7fc360003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:25.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.079+0000 7fc36a7fc700 1 --2- 172.21.15.98:0/3825948457 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc360003a50 0x7fc360003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:25.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.079+0000 7fc36a7fc700 1 -- 172.21.15.98:0/3825948457 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc360005200 msgr2=0x7fc36000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:25.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.079+0000 7fc36a7fc700 1 --2- 172.21.15.98:0/3825948457 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc360005200 0x7fc36000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:25.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.079+0000 7fc36a7fc700 1 -- 172.21.15.98:0/3825948457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3600035b0 con 0x7fc360004850 2023-03-03T08:00:25.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.079+0000 7fc3737fe700 1 -- 172.21.15.98:0/3825948457 reap_dead start 2023-03-03T08:00:25.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.079+0000 7fc372ffd700 1 --2- 172.21.15.98:0/3825948457 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc360003a50 0x7fc360003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:25.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.079+0000 7fc36a7fc700 1 --2- 172.21.15.98:0/3825948457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc360004850 0x7fc360004cc0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc35c007fd0 tx=0x7fc360005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45fcb9110dc2804d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:25.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.079+0000 7fc3727fc700 1 -- 172.21.15.98:0/3825948457 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c0052d0 con 0x7fc360004850 2023-03-03T08:00:25.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.079+0000 7fc3727fc700 1 -- 172.21.15.98:0/3825948457 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35c005450 con 0x7fc360004850 2023-03-03T08:00:25.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.079+0000 7fc3727fc700 1 -- 172.21.15.98:0/3825948457 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c0038c0 con 0x7fc360004850 2023-03-03T08:00:25.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.080+0000 7fc373fff700 1 -- 172.21.15.98:0/3825948457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc360004850 msgr2=0x7fc360004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:25.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.080+0000 7fc373fff700 1 --2- 172.21.15.98:0/3825948457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc360004850 0x7fc360004cc0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc35c007fd0 tx=0x7fc360005880 comp rx=0 tx=0).stop 2023-03-03T08:00:25.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.080+0000 7fc373fff700 1 -- 172.21.15.98:0/3825948457 shutdown_connections 2023-03-03T08:00:25.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.080+0000 7fc373fff700 1 --2- 172.21.15.98:0/3825948457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc360004850 0x7fc360004cc0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:25.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.080+0000 7fc373fff700 1 -- 172.21.15.98:0/3825948457 >> 172.21.15.98:0/3825948457 conn(0x7fc37406b880 msgr2=0x7fc3740cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:25.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.080+0000 7fc373fff700 1 -- 172.21.15.98:0/3825948457 shutdown_connections 2023-03-03T08:00:25.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.080+0000 7fc373fff700 1 -- 172.21.15.98:0/3825948457 wait complete. 2023-03-03T08:00:25.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.082+0000 7fc373fff700 1 Processor -- start 2023-03-03T08:00:25.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.082+0000 7fc373fff700 1 -- start start 2023-03-03T08:00:25.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.083+0000 7fc373fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc360004850 0x7fc3741e5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:25.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.083+0000 7fc373fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc374150f30 0x7fc3741e66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:25.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.083+0000 7fc373fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3741f2d40 0x7fc3741f31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:25.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.083+0000 7fc373fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3741f2360 con 0x7fc3741f2d40 2023-03-03T08:00:25.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.083+0000 7fc373fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3741e6c10 con 0x7fc374150f30 2023-03-03T08:00:25.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.083+0000 7fc373fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3741e73e0 con 0x7fc360004850 2023-03-03T08:00:25.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.084+0000 7fc3737fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3741f2d40 0x7fc3741f31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:25.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.084+0000 7fc3737fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3741f2d40 0x7fc3741f31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44906/0 (socket says 172.21.15.98:44906) 2023-03-03T08:00:25.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.084+0000 7fc3737fe700 1 -- 172.21.15.98:0/2141088458 learned_addr learned my addr 172.21.15.98:0/2141088458 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:25.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.084+0000 7fc36a7fc700 1 --2- 172.21.15.98:0/2141088458 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc374150f30 0x7fc3741e66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:25.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.084+0000 7fc372ffd700 1 --2- 172.21.15.98:0/2141088458 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc360004850 0x7fc3741e5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:25.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.084+0000 7fc3737fe700 1 -- 172.21.15.98:0/2141088458 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc360004850 msgr2=0x7fc3741e5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:25.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.084+0000 7fc3737fe700 1 --2- 172.21.15.98:0/2141088458 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc360004850 0x7fc3741e5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:25.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.084+0000 7fc3737fe700 1 -- 172.21.15.98:0/2141088458 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc374150f30 msgr2=0x7fc3741e66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:25.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.084+0000 7fc3737fe700 1 --2- 172.21.15.98:0/2141088458 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc374150f30 0x7fc3741e66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:25.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.084+0000 7fc3737fe700 1 -- 172.21.15.98:0/2141088458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3740cdd40 con 0x7fc3741f2d40 2023-03-03T08:00:25.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.084+0000 7fc372ffd700 1 --2- 172.21.15.98:0/2141088458 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc360004850 0x7fc3741e5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:25.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.084+0000 7fc3737fe700 1 --2- 172.21.15.98:0/2141088458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3741f2d40 0x7fc3741f31b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc3740b5780 tx=0x7fc36c007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:25.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.085+0000 7fc36bfff700 1 -- 172.21.15.98:0/2141088458 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c012560 con 0x7fc3741f2d40 2023-03-03T08:00:25.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.085+0000 7fc36bfff700 1 -- 172.21.15.98:0/2141088458 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c0126c0 con 0x7fc3741f2d40 2023-03-03T08:00:25.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.085+0000 7fc36bfff700 1 -- 172.21.15.98:0/2141088458 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c012a70 con 0x7fc3741f2d40 2023-03-03T08:00:25.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.085+0000 7fc373fff700 1 -- 172.21.15.98:0/2141088458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3740cda60 con 0x7fc3741f2d40 2023-03-03T08:00:25.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.085+0000 7fc373fff700 1 -- 172.21.15.98:0/2141088458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37415cdf0 con 0x7fc3741f2d40 2023-03-03T08:00:25.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.086+0000 7fc373fff700 1 -- 172.21.15.98:0/2141088458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc374068290 con 0x7fc3741f2d40 2023-03-03T08:00:25.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.087+0000 7fc36bfff700 1 -- 172.21.15.98:0/2141088458 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fc36c012bd0 con 0x7fc3741f2d40 2023-03-03T08:00:25.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.089+0000 7fc36bfff700 1 --2- 172.21.15.98:0/2141088458 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3540683c0 0x7fc35406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:25.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.090+0000 7fc36bfff700 1 -- 172.21.15.98:0/2141088458 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc36c085ee0 con 0x7fc3741f2d40 2023-03-03T08:00:25.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.090+0000 7fc372ffd700 1 --2- 172.21.15.98:0/2141088458 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3540683c0 0x7fc35406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:25.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.090+0000 7fc372ffd700 1 --2- 172.21.15.98:0/2141088458 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3540683c0 0x7fc35406a870 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc374063f00 tx=0x7fc36400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:25.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.093+0000 7fc36bfff700 1 -- 172.21.15.98:0/2141088458 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc36c0524f0 con 0x7fc3741f2d40 2023-03-03T08:00:25.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.330+0000 7fc373fff700 1 -- 172.21.15.98:0/2141088458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc3740d0430 con 0x7fc3741f2d40 2023-03-03T08:00:25.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.791+0000 7fc36bfff700 1 -- 172.21.15.98:0/2141088458 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v31) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc36c012e60 con 0x7fc3741f2d40 2023-03-03T08:00:25.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.794+0000 7fc373fff700 1 -- 172.21.15.98:0/2141088458 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3540683c0 msgr2=0x7fc35406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:25.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.794+0000 7fc373fff700 1 --2- 172.21.15.98:0/2141088458 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3540683c0 0x7fc35406a870 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc374063f00 tx=0x7fc36400b040 comp rx=0 tx=0).stop 2023-03-03T08:00:25.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.794+0000 7fc373fff700 1 -- 172.21.15.98:0/2141088458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3741f2d40 msgr2=0x7fc3741f31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:25.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.794+0000 7fc373fff700 1 --2- 172.21.15.98:0/2141088458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3741f2d40 0x7fc3741f31b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc3740b5780 tx=0x7fc36c007340 comp rx=0 tx=0).stop 2023-03-03T08:00:25.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.794+0000 7fc373fff700 1 -- 172.21.15.98:0/2141088458 shutdown_connections 2023-03-03T08:00:25.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.794+0000 7fc373fff700 1 --2- 172.21.15.98:0/2141088458 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3540683c0 0x7fc35406a870 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:25.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.794+0000 7fc373fff700 1 --2- 172.21.15.98:0/2141088458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3741f2d40 0x7fc3741f31b0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:25.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.794+0000 7fc373fff700 1 --2- 172.21.15.98:0/2141088458 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc360004850 0x7fc3741e5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:25.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.794+0000 7fc373fff700 1 --2- 172.21.15.98:0/2141088458 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc374150f30 0x7fc3741e66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:25.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.794+0000 7fc373fff700 1 -- 172.21.15.98:0/2141088458 >> 172.21.15.98:0/2141088458 conn(0x7fc37406b880 msgr2=0x7fc3740b4280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:25.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.795+0000 7fc373fff700 1 -- 172.21.15.98:0/2141088458 shutdown_connections 2023-03-03T08:00:25.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:25.795+0000 7fc373fff700 1 -- 172.21.15.98:0/2141088458 wait complete. 2023-03-03T08:00:25.824 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-03-03T08:00:25.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:00:26.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.069+0000 7fda161ff700 1 Processor -- start 2023-03-03T08:00:26.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.070+0000 7fda161ff700 1 -- start start 2023-03-03T08:00:26.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.071+0000 7fda161ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fda100b87c0 0x7fda100cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:26.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.071+0000 7fda161ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fda100b5410 0x7fda100d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:26.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.071+0000 7fda161ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda100d0ef0 con 0x7fda100b8b90 2023-03-03T08:00:26.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.071+0000 7fda161ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda100d0600 con 0x7fda100b5410 2023-03-03T08:00:26.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.071+0000 7fda161ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda100cd970 con 0x7fda100b87c0 2023-03-03T08:00:26.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.071+0000 7fda0ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fda100b5410 0x7fda100d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:26.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.071+0000 7fda0ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fda100b5410 0x7fda100d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44910/0 (socket says 172.21.15.98:44910) 2023-03-03T08:00:26.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.071+0000 7fda0ffff700 1 -- 172.21.15.98:0/2223155226 learned_addr learned my addr 172.21.15.98:0/2223155226 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:26.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.072+0000 7fda0f7fe700 1 --2- 172.21.15.98:0/2223155226 >> v2:172.21.15.100:3300/0 conn(0x7fda100b87c0 0x7fda100cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:26.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.072+0000 7fda0ffff700 1 -- 172.21.15.98:0/2223155226 >> v2:172.21.15.100:3300/0 conn(0x7fda100b87c0 msgr2=0x7fda100cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.072+0000 7fda0ffff700 1 --2- 172.21.15.98:0/2223155226 >> v2:172.21.15.100:3300/0 conn(0x7fda100b87c0 0x7fda100cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.072+0000 7fda0ffff700 1 -- 172.21.15.98:0/2223155226 >> v1:172.21.15.100:6789/0 conn(0x7fda100b8b90 legacy=0x7fda100b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.072+0000 7fda0ffff700 1 -- 172.21.15.98:0/2223155226 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda100cdc50 con 0x7fda100b5410 2023-03-03T08:00:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.072+0000 7fda0ffff700 1 --2- 172.21.15.98:0/2223155226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda100b5410 0x7fda100d13d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fda10073840 tx=0x7fda0800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8474146bb040455 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.072+0000 7fda0e7fc700 1 -- 172.21.15.98:0/2223155226 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0800e070 con 0x7fda100b5410 2023-03-03T08:00:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.073+0000 7fda0e7fc700 1 -- 172.21.15.98:0/2223155226 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda08003a20 con 0x7fda100b5410 2023-03-03T08:00:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.073+0000 7fda0e7fc700 1 -- 172.21.15.98:0/2223155226 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08003dd0 con 0x7fda100b5410 2023-03-03T08:00:26.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.073+0000 7fda161ff700 1 -- 172.21.15.98:0/2223155226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda100b5410 msgr2=0x7fda100d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:26.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.073+0000 7fda161ff700 1 --2- 172.21.15.98:0/2223155226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda100b5410 0x7fda100d13d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fda10073840 tx=0x7fda0800b330 comp rx=0 tx=0).stop 2023-03-03T08:00:26.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.073+0000 7fda161ff700 1 -- 172.21.15.98:0/2223155226 shutdown_connections 2023-03-03T08:00:26.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.073+0000 7fda161ff700 1 --2- 172.21.15.98:0/2223155226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda100b5410 0x7fda100d13d0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.073+0000 7fda161ff700 1 --2- 172.21.15.98:0/2223155226 >> v2:172.21.15.100:3300/0 conn(0x7fda100b87c0 0x7fda100cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.073+0000 7fda161ff700 1 -- 172.21.15.98:0/2223155226 >> 172.21.15.98:0/2223155226 conn(0x7fda1006aee0 msgr2=0x7fda1006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:26.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.073+0000 7fda161ff700 1 -- 172.21.15.98:0/2223155226 shutdown_connections 2023-03-03T08:00:26.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.073+0000 7fda161ff700 1 -- 172.21.15.98:0/2223155226 wait complete. 2023-03-03T08:00:26.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.075+0000 7fda161ff700 1 Processor -- start 2023-03-03T08:00:26.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.076+0000 7fda161ff700 1 -- start start 2023-03-03T08:00:26.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.076+0000 7fda161ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda100b5410 0x7fda101e5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:26.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.077+0000 7fda161ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda100b87c0 0x7fda101e61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:26.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.077+0000 7fda161ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda100b8b90 0x7fda100cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:26.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.077+0000 7fda161ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda101e7610 con 0x7fda100b87c0 2023-03-03T08:00:26.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.077+0000 7fda161ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda101e6a00 con 0x7fda100b5410 2023-03-03T08:00:26.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.077+0000 7fda161ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda101e6ce0 con 0x7fda100b8b90 2023-03-03T08:00:26.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.077+0000 7fda0effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda100b87c0 0x7fda101e61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:26.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.077+0000 7fda0effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda100b87c0 0x7fda101e61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44918/0 (socket says 172.21.15.98:44918) 2023-03-03T08:00:26.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.077+0000 7fda0effd700 1 -- 172.21.15.98:0/2931478513 learned_addr learned my addr 172.21.15.98:0/2931478513 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:26.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.077+0000 7fda0f7fe700 1 --2- 172.21.15.98:0/2931478513 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda100b5410 0x7fda101e5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:26.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.077+0000 7fda0ffff700 1 --2- 172.21.15.98:0/2931478513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda100b8b90 0x7fda100cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:26.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.077+0000 7fda0effd700 1 -- 172.21.15.98:0/2931478513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda100b8b90 msgr2=0x7fda100cb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:26.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.078+0000 7fda0effd700 1 --2- 172.21.15.98:0/2931478513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda100b8b90 0x7fda100cb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.078+0000 7fda0effd700 1 -- 172.21.15.98:0/2931478513 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda100b5410 msgr2=0x7fda101e5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:26.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.078+0000 7fda0effd700 1 --2- 172.21.15.98:0/2931478513 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda100b5410 0x7fda101e5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.078+0000 7fda0effd700 1 -- 172.21.15.98:0/2931478513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda08009b30 con 0x7fda100b87c0 2023-03-03T08:00:26.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.078+0000 7fda0effd700 1 --2- 172.21.15.98:0/2931478513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda100b87c0 0x7fda101e61a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fda0400b450 tx=0x7fda0400b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:26.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.078+0000 7fd9f7fff700 1 -- 172.21.15.98:0/2931478513 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda04011830 con 0x7fda100b87c0 2023-03-03T08:00:26.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.078+0000 7fd9f7fff700 1 -- 172.21.15.98:0/2931478513 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda04011990 con 0x7fda100b87c0 2023-03-03T08:00:26.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.078+0000 7fd9f7fff700 1 -- 172.21.15.98:0/2931478513 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda04011d40 con 0x7fda100b87c0 2023-03-03T08:00:26.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.078+0000 7fda161ff700 1 -- 172.21.15.98:0/2931478513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda100cdf30 con 0x7fda100b87c0 2023-03-03T08:00:26.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.078+0000 7fda161ff700 1 -- 172.21.15.98:0/2931478513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda10073a80 con 0x7fda100b87c0 2023-03-03T08:00:26.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.079+0000 7fda161ff700 1 -- 172.21.15.98:0/2931478513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda100ce0c0 con 0x7fda100b87c0 2023-03-03T08:00:26.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.081+0000 7fd9f7fff700 1 -- 172.21.15.98:0/2931478513 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fda04021070 con 0x7fda100b87c0 2023-03-03T08:00:26.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.082+0000 7fd9f7fff700 1 --2- 172.21.15.98:0/2931478513 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd9fc0683c0 0x7fd9fc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:26.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.082+0000 7fd9f7fff700 1 -- 172.21.15.98:0/2931478513 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fda04085c20 con 0x7fda100b87c0 2023-03-03T08:00:26.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.082+0000 7fda0f7fe700 1 --2- 172.21.15.98:0/2931478513 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd9fc0683c0 0x7fd9fc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:26.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.083+0000 7fda0f7fe700 1 --2- 172.21.15.98:0/2931478513 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd9fc0683c0 0x7fd9fc06a870 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fda00006b20 tx=0x7fda00008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:26.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.086+0000 7fd9f7fff700 1 -- 172.21.15.98:0/2931478513 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda040522c0 con 0x7fda100b87c0 2023-03-03T08:00:26.243 INFO:tasks.ceph.mds.c.smithi098.stderr:2023-03-03T08:00:26.242+0000 7fac4e1f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T08:00:26.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.327+0000 7fda161ff700 1 -- 172.21.15.98:0/2931478513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda100729b0 con 0x7fda100b87c0 2023-03-03T08:00:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.328+0000 7fd9f7fff700 1 -- 172.21.15.98:0/2931478513 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4649 (secure 0 0 0) 0x7fda0400be60 con 0x7fda100b87c0 2023-03-03T08:00:26.330 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:26.330 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5702,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/976526047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":976526047},{"type":"v1","addr":"172.21.15.100:6834","nonce":976526047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3372584689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3372584689},{"type":"v1","addr":"172.21.15.100:6835","nonce":3372584689}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/505711827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":505711827},{"type":"v1","addr":"172.21.15.98:6837","nonce":505711827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:00:22.323396+0000","modified":"2023-03-03T08:00:25.787755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6387":{"gid":6387,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6836/3248147431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3248147431},{"type":"v1","addr":"172.21.15.98:6836","nonce":3248147431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-03-03T08:00:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.331+0000 7fda161ff700 1 -- 172.21.15.98:0/2931478513 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd9fc0683c0 msgr2=0x7fd9fc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.331+0000 7fda161ff700 1 --2- 172.21.15.98:0/2931478513 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd9fc0683c0 0x7fd9fc06a870 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fda00006b20 tx=0x7fda00008040 comp rx=0 tx=0).stop 2023-03-03T08:00:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.331+0000 7fda161ff700 1 -- 172.21.15.98:0/2931478513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda100b87c0 msgr2=0x7fda101e61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.331+0000 7fda161ff700 1 --2- 172.21.15.98:0/2931478513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda100b87c0 0x7fda101e61a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fda0400b450 tx=0x7fda0400b760 comp rx=0 tx=0).stop 2023-03-03T08:00:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.331+0000 7fda161ff700 1 -- 172.21.15.98:0/2931478513 shutdown_connections 2023-03-03T08:00:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.331+0000 7fda161ff700 1 --2- 172.21.15.98:0/2931478513 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd9fc0683c0 0x7fd9fc06a870 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.331+0000 7fda161ff700 1 --2- 172.21.15.98:0/2931478513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda100b8b90 0x7fda100cb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.331+0000 7fda161ff700 1 --2- 172.21.15.98:0/2931478513 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda100b5410 0x7fda101e5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.331+0000 7fda161ff700 1 --2- 172.21.15.98:0/2931478513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda100b87c0 0x7fda101e61a0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.331+0000 7fda161ff700 1 -- 172.21.15.98:0/2931478513 >> 172.21.15.98:0/2931478513 conn(0x7fda1006aee0 msgr2=0x7fda100b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:26.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.331+0000 7fda161ff700 1 -- 172.21.15.98:0/2931478513 shutdown_connections 2023-03-03T08:00:26.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.331+0000 7fda161ff700 1 -- 172.21.15.98:0/2931478513 wait complete. 2023-03-03T08:00:26.334 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 31 2023-03-03T08:00:26.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:00:26.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.612+0000 7f32035ff700 1 Processor -- start 2023-03-03T08:00:26.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.612+0000 7f32035ff700 1 -- start start 2023-03-03T08:00:26.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.613+0000 7f32035ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f31fc0b8b90 0x7f31fc0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:26.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.613+0000 7f32035ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31fc0b5410 0x7f31fc0d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:26.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.613+0000 7f32035ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31fc0d0ef0 con 0x7f31fc0b87c0 2023-03-03T08:00:26.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.613+0000 7f32035ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31fc0d0600 con 0x7f31fc0b8b90 2023-03-03T08:00:26.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.613+0000 7f32035ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc0cd970 con 0x7f31fc0b5410 2023-03-03T08:00:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.614+0000 7f3200b9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f31fc0b8b90 0x7f31fc0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.614+0000 7f3200b9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f31fc0b8b90 0x7f31fc0cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44934/0 (socket says 172.21.15.98:44934) 2023-03-03T08:00:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.614+0000 7f3200b9a700 1 -- 172.21.15.98:0/3083895789 learned_addr learned my addr 172.21.15.98:0/3083895789 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.614+0000 7f3201b9c700 1 --2- 172.21.15.98:0/3083895789 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31fc0b5410 0x7f31fc0d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.614+0000 7f3200b9a700 1 -- 172.21.15.98:0/3083895789 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31fc0b5410 msgr2=0x7f31fc0d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.614+0000 7f3200b9a700 1 --2- 172.21.15.98:0/3083895789 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31fc0b5410 0x7f31fc0d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.614+0000 7f3200b9a700 1 -- 172.21.15.98:0/3083895789 >> v1:172.21.15.100:6789/0 conn(0x7f31fc0b87c0 legacy=0x7f31fc0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.614+0000 7f3200b9a700 1 -- 172.21.15.98:0/3083895789 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31fc0cdc50 con 0x7f31fc0b8b90 2023-03-03T08:00:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.614+0000 7f31f3fff700 1 -- 172.21.15.98:0/3083895789 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222991591 0 0) 0x7f31fc0d0ef0 con 0x7f31fc0b87c0 2023-03-03T08:00:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.614+0000 7f3200b9a700 1 --2- 172.21.15.98:0/3083895789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31fc0b8b90 0x7f31fc0cffe0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0098f0 tx=0x7f31ec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40853db6d54a7df server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.615+0000 7f31f3fff700 1 -- 172.21.15.98:0/3083895789 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec011830 con 0x7f31fc0b8b90 2023-03-03T08:00:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.615+0000 7f31f3fff700 1 -- 172.21.15.98:0/3083895789 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec011990 con 0x7f31fc0b8b90 2023-03-03T08:00:26.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.615+0000 7f31f3fff700 1 -- 172.21.15.98:0/3083895789 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec011d20 con 0x7f31fc0b8b90 2023-03-03T08:00:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.615+0000 7f32035ff700 1 -- 172.21.15.98:0/3083895789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31fc0b8b90 msgr2=0x7f31fc0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.615+0000 7f32035ff700 1 --2- 172.21.15.98:0/3083895789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31fc0b8b90 0x7f31fc0cffe0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0098f0 tx=0x7f31ec009c00 comp rx=0 tx=0).stop 2023-03-03T08:00:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.615+0000 7f32035ff700 1 -- 172.21.15.98:0/3083895789 shutdown_connections 2023-03-03T08:00:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.615+0000 7f32035ff700 1 --2- 172.21.15.98:0/3083895789 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31fc0b5410 0x7f31fc0d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.615+0000 7f32035ff700 1 --2- 172.21.15.98:0/3083895789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31fc0b8b90 0x7f31fc0cffe0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.615+0000 7f32035ff700 1 -- 172.21.15.98:0/3083895789 >> 172.21.15.98:0/3083895789 conn(0x7f31fc06aee0 msgr2=0x7f31fc06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.615+0000 7f32035ff700 1 -- 172.21.15.98:0/3083895789 shutdown_connections 2023-03-03T08:00:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.616+0000 7f32035ff700 1 -- 172.21.15.98:0/3083895789 wait complete. 2023-03-03T08:00:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.617+0000 7f32035ff700 1 Processor -- start 2023-03-03T08:00:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.618+0000 7f32035ff700 1 -- start start 2023-03-03T08:00:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f32035ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31fc0b5410 0x7f31fc15abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f32035ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31fc0b87c0 0x7f31fc0cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f32035ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31fc0b8b90 0x7f31fc0aa840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:26.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f32035ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc15b0f0 con 0x7f31fc0b5410 2023-03-03T08:00:26.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f32035ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc0ca7f0 con 0x7f31fc0b8b90 2023-03-03T08:00:26.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f32035ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc0caad0 con 0x7f31fc0b87c0 2023-03-03T08:00:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f320139b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31fc0b5410 0x7f31fc15abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f3200b9a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31fc0b87c0 0x7f31fc0cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f3200b9a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31fc0b87c0 0x7f31fc0cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:32890/0 (socket says 172.21.15.98:32890) 2023-03-03T08:00:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f3200b9a700 1 -- 172.21.15.98:0/1995380343 learned_addr learned my addr 172.21.15.98:0/1995380343 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f3201b9c700 1 --2- 172.21.15.98:0/1995380343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31fc0b8b90 0x7f31fc0aa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f320139b700 1 -- 172.21.15.98:0/1995380343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31fc0b87c0 msgr2=0x7f31fc0cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f320139b700 1 --2- 172.21.15.98:0/1995380343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31fc0b87c0 0x7f31fc0cb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f320139b700 1 -- 172.21.15.98:0/1995380343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31fc0b8b90 msgr2=0x7f31fc0aa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f320139b700 1 --2- 172.21.15.98:0/1995380343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31fc0b8b90 0x7f31fc0aa840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.619+0000 7f320139b700 1 -- 172.21.15.98:0/1995380343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec009550 con 0x7f31fc0b5410 2023-03-03T08:00:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.620+0000 7f3200b9a700 1 --2- 172.21.15.98:0/1995380343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31fc0b87c0 0x7f31fc0cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:00:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.620+0000 7f320139b700 1 --2- 172.21.15.98:0/1995380343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31fc0b5410 0x7f31fc15abb0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f31f800c520 tx=0x7f31f800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:26.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.620+0000 7f31f1ffb700 1 -- 172.21.15.98:0/1995380343 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f80054e0 con 0x7f31fc0b5410 2023-03-03T08:00:26.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.620+0000 7f31f1ffb700 1 -- 172.21.15.98:0/1995380343 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f8003680 con 0x7f31fc0b5410 2023-03-03T08:00:26.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.620+0000 7f31f1ffb700 1 -- 172.21.15.98:0/1995380343 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f8003a30 con 0x7f31fc0b5410 2023-03-03T08:00:26.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.620+0000 7f32035ff700 1 -- 172.21.15.98:0/1995380343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31fc0cdf30 con 0x7f31fc0b5410 2023-03-03T08:00:26.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.620+0000 7f32035ff700 1 -- 172.21.15.98:0/1995380343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31fc073a80 con 0x7f31fc0b5410 2023-03-03T08:00:26.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.621+0000 7f32035ff700 1 -- 172.21.15.98:0/1995380343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31fc0ce0c0 con 0x7f31fc0b5410 2023-03-03T08:00:26.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.623+0000 7f31f1ffb700 1 -- 172.21.15.98:0/1995380343 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f31f8019030 con 0x7f31fc0b5410 2023-03-03T08:00:26.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.624+0000 7f31f1ffb700 1 --2- 172.21.15.98:0/1995380343 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f31e00683c0 0x7f31e006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:26.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.624+0000 7f31f1ffb700 1 -- 172.21.15.98:0/1995380343 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f31f8084fe0 con 0x7f31fc0b5410 2023-03-03T08:00:26.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.624+0000 7f3200b9a700 1 --2- 172.21.15.98:0/1995380343 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f31e00683c0 0x7f31e006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:26.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.625+0000 7f3200b9a700 1 --2- 172.21.15.98:0/1995380343 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f31e00683c0 0x7f31e006a870 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f31ec000c00 tx=0x7f31ec008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:26.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.628+0000 7f31f1ffb700 1 -- 172.21.15.98:0/1995380343 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f31f804b5e0 con 0x7f31fc0b5410 2023-03-03T08:00:26.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.823+0000 7f32035ff700 1 -- 172.21.15.98:0/1995380343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31fc0729b0 con 0x7f31fc0b5410 2023-03-03T08:00:26.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.824+0000 7f31f1ffb700 1 -- 172.21.15.98:0/1995380343 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+17921 (secure 0 0 0) 0x7f31f8017020 con 0x7f31fc0b5410 2023-03-03T08:00:26.826 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:26.826 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":66,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:00:23.326015+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:00:19.823120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-03-03T08:00:21.324346+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:00:26.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.827+0000 7f32035ff700 1 -- 172.21.15.98:0/1995380343 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f31e00683c0 msgr2=0x7f31e006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:26.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.827+0000 7f32035ff700 1 --2- 172.21.15.98:0/1995380343 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f31e00683c0 0x7f31e006a870 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f31ec000c00 tx=0x7f31ec008030 comp rx=0 tx=0).stop 2023-03-03T08:00:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.827+0000 7f32035ff700 1 -- 172.21.15.98:0/1995380343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31fc0b5410 msgr2=0x7f31fc15abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.827+0000 7f32035ff700 1 --2- 172.21.15.98:0/1995380343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31fc0b5410 0x7f31fc15abb0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f31f800c520 tx=0x7f31f800c830 comp rx=0 tx=0).stop 2023-03-03T08:00:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.827+0000 7f32035ff700 1 -- 172.21.15.98:0/1995380343 shutdown_connections 2023-03-03T08:00:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.827+0000 7f32035ff700 1 --2- 172.21.15.98:0/1995380343 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f31e00683c0 0x7f31e006a870 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.827+0000 7f32035ff700 1 --2- 172.21.15.98:0/1995380343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31fc0b5410 0x7f31fc15abb0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.827+0000 7f32035ff700 1 --2- 172.21.15.98:0/1995380343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31fc0b8b90 0x7f31fc0aa840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.827+0000 7f32035ff700 1 --2- 172.21.15.98:0/1995380343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31fc0b87c0 0x7f31fc0cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.827+0000 7f32035ff700 1 -- 172.21.15.98:0/1995380343 >> 172.21.15.98:0/1995380343 conn(0x7f31fc06aee0 msgr2=0x7f31fc0b7820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.827+0000 7f32035ff700 1 -- 172.21.15.98:0/1995380343 shutdown_connections 2023-03-03T08:00:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:26.827+0000 7f32035ff700 1 -- 172.21.15.98:0/1995380343 wait complete. 2023-03-03T08:00:26.856 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-03-03T08:00:26.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:00:27.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.102+0000 7f3b43bff700 1 Processor -- start 2023-03-03T08:00:27.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.102+0000 7f3b43bff700 1 -- start start 2023-03-03T08:00:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.103+0000 7f3b43bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3b3c0d23e0 0x7f3b3c0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.104+0000 7f3b43bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c0b5b20 con 0x7f3b3c0b87c0 2023-03-03T08:00:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.104+0000 7f3b43bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c0d0ef0 con 0x7f3b3c0b8b90 2023-03-03T08:00:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.104+0000 7f3b43bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b3c0d0600 con 0x7f3b3c0d23e0 2023-03-03T08:00:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.104+0000 7f3b4199b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3b3c0b87c0 0x7f3b3c0b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44860/0 (socket says 172.21.15.98:44860) 2023-03-03T08:00:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.104+0000 7f3b4199b700 1 -- 172.21.15.98:0/1937207583 learned_addr learned my addr 172.21.15.98:0/1937207583 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.104+0000 7f3b4219c700 1 --2- 172.21.15.98:0/1937207583 >> v2:172.21.15.98:3300/0 conn(0x7f3b3c0d23e0 0x7f3b3c0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.104+0000 7f3b40999700 1 -- 172.21.15.98:0/1937207583 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468295984 0 0) 0x7f3b3c0b5b20 con 0x7f3b3c0b87c0 2023-03-03T08:00:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.105+0000 7f3b4219c700 1 -- 172.21.15.98:0/1937207583 >> v1:172.21.15.100:6789/0 conn(0x7f3b3c0b8b90 legacy=0x7f3b3c0b5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.105+0000 7f3b4219c700 1 -- 172.21.15.98:0/1937207583 >> v1:172.21.15.98:6789/0 conn(0x7f3b3c0b87c0 legacy=0x7f3b3c0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.105+0000 7f3b4219c700 1 -- 172.21.15.98:0/1937207583 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3c0cd970 con 0x7f3b3c0d23e0 2023-03-03T08:00:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.105+0000 7f3b4219c700 1 --2- 172.21.15.98:0/1937207583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c0d23e0 0x7f3b3c0cffe0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f3b34009d90 tx=0x7f3b3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e39f1c383779fba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.105+0000 7f3b40999700 1 -- 172.21.15.98:0/1937207583 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3400e070 con 0x7f3b3c0d23e0 2023-03-03T08:00:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.105+0000 7f3b40999700 1 -- 172.21.15.98:0/1937207583 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3400bbd0 con 0x7f3b3c0d23e0 2023-03-03T08:00:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.105+0000 7f3b40999700 1 -- 172.21.15.98:0/1937207583 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b34012430 con 0x7f3b3c0d23e0 2023-03-03T08:00:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.106+0000 7f3b43bff700 1 -- 172.21.15.98:0/1937207583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c0d23e0 msgr2=0x7f3b3c0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.106+0000 7f3b43bff700 1 --2- 172.21.15.98:0/1937207583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c0d23e0 0x7f3b3c0cffe0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f3b34009d90 tx=0x7f3b3400b330 comp rx=0 tx=0).stop 2023-03-03T08:00:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.106+0000 7f3b43bff700 1 -- 172.21.15.98:0/1937207583 shutdown_connections 2023-03-03T08:00:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.106+0000 7f3b43bff700 1 --2- 172.21.15.98:0/1937207583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c0d23e0 0x7f3b3c0cffe0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.106+0000 7f3b43bff700 1 -- 172.21.15.98:0/1937207583 >> 172.21.15.98:0/1937207583 conn(0x7f3b3c06aee0 msgr2=0x7f3b3c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.106+0000 7f3b43bff700 1 -- 172.21.15.98:0/1937207583 shutdown_connections 2023-03-03T08:00:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.106+0000 7f3b43bff700 1 -- 172.21.15.98:0/1937207583 wait complete. 2023-03-03T08:00:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.108+0000 7f3b43bff700 1 Processor -- start 2023-03-03T08:00:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.108+0000 7f3b43bff700 1 -- start start 2023-03-03T08:00:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.109+0000 7f3b43bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c0b87c0 0x7f3b3c150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.109+0000 7f3b43bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b3c0b8b90 0x7f3b3c15aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.109+0000 7f3b43bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b3c0d23e0 0x7f3b3c15af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.109+0000 7f3b43bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c0cb410 con 0x7f3b3c0b87c0 2023-03-03T08:00:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.109+0000 7f3b43bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c0cc3c0 con 0x7f3b3c0d23e0 2023-03-03T08:00:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.109+0000 7f3b43bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c0cb130 con 0x7f3b3c0b8b90 2023-03-03T08:00:27.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.110+0000 7f3b4199b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c0b87c0 0x7f3b3c150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:27.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.110+0000 7f3b4199b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c0b87c0 0x7f3b3c150c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44950/0 (socket says 172.21.15.98:44950) 2023-03-03T08:00:27.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.110+0000 7f3b4199b700 1 -- 172.21.15.98:0/1782570749 learned_addr learned my addr 172.21.15.98:0/1782570749 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:27.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.110+0000 7f3b4119a700 1 --2- 172.21.15.98:0/1782570749 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b3c0b8b90 0x7f3b3c15aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:27.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.110+0000 7f3b4219c700 1 --2- 172.21.15.98:0/1782570749 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b3c0d23e0 0x7f3b3c15af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:27.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.110+0000 7f3b4199b700 1 -- 172.21.15.98:0/1782570749 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b3c0b8b90 msgr2=0x7f3b3c15aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:27.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.110+0000 7f3b4199b700 1 --2- 172.21.15.98:0/1782570749 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b3c0b8b90 0x7f3b3c15aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.110+0000 7f3b4199b700 1 -- 172.21.15.98:0/1782570749 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b3c0d23e0 msgr2=0x7f3b3c15af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:27.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.110+0000 7f3b4199b700 1 --2- 172.21.15.98:0/1782570749 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b3c0d23e0 0x7f3b3c15af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.110+0000 7f3b4199b700 1 -- 172.21.15.98:0/1782570749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b34009a70 con 0x7f3b3c0b87c0 2023-03-03T08:00:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.110+0000 7f3b4219c700 1 --2- 172.21.15.98:0/1782570749 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b3c0d23e0 0x7f3b3c15af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.110+0000 7f3b4199b700 1 --2- 172.21.15.98:0/1782570749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c0b87c0 0x7f3b3c150c40 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f3b3800c520 tx=0x7f3b3800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.111+0000 7f3b327fc700 1 -- 172.21.15.98:0/1782570749 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38004c80 con 0x7f3b3c0b87c0 2023-03-03T08:00:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.111+0000 7f3b327fc700 1 -- 172.21.15.98:0/1782570749 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b38004e00 con 0x7f3b3c0b87c0 2023-03-03T08:00:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.111+0000 7f3b327fc700 1 -- 172.21.15.98:0/1782570749 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38007310 con 0x7f3b3c0b87c0 2023-03-03T08:00:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.111+0000 7f3b43bff700 1 -- 172.21.15.98:0/1782570749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3c073a80 con 0x7f3b3c0b87c0 2023-03-03T08:00:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.111+0000 7f3b43bff700 1 -- 172.21.15.98:0/1782570749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3c0cdc50 con 0x7f3b3c0b87c0 2023-03-03T08:00:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.112+0000 7f3b43bff700 1 -- 172.21.15.98:0/1782570749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3c0cdf30 con 0x7f3b3c0b87c0 2023-03-03T08:00:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.113+0000 7f3b327fc700 1 -- 172.21.15.98:0/1782570749 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f3b3801a020 con 0x7f3b3c0b87c0 2023-03-03T08:00:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.114+0000 7f3b327fc700 1 --2- 172.21.15.98:0/1782570749 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b200683c0 0x7f3b2006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.114+0000 7f3b327fc700 1 -- 172.21.15.98:0/1782570749 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3b38085ad0 con 0x7f3b3c0b87c0 2023-03-03T08:00:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.115+0000 7f3b4119a700 1 --2- 172.21.15.98:0/1782570749 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b200683c0 0x7f3b2006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.115+0000 7f3b4119a700 1 --2- 172.21.15.98:0/1782570749 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b200683c0 0x7f3b2006a870 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c0b1230 tx=0x7f3b2c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:27.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.118+0000 7f3b327fc700 1 -- 172.21.15.98:0/1782570749 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3b38052170 con 0x7f3b3c0b87c0 2023-03-03T08:00:27.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.309+0000 7f3b43bff700 1 -- 172.21.15.98:0/1782570749 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b3c0729b0 con 0x7f3b200683c0 2023-03-03T08:00:27.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.319+0000 7f3b327fc700 1 -- 172.21.15.98:0/1782570749 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+307590 (secure 0 0 0) 0x7f3b3c0729b0 con 0x7f3b200683c0 2023-03-03T08:00:27.325 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:27.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.324+0000 7f3b43bff700 1 -- 172.21.15.98:0/1782570749 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b200683c0 msgr2=0x7f3b2006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:27.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.324+0000 7f3b43bff700 1 --2- 172.21.15.98:0/1782570749 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b200683c0 0x7f3b2006a870 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c0b1230 tx=0x7f3b2c009040 comp rx=0 tx=0).stop 2023-03-03T08:00:27.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.324+0000 7f3b43bff700 1 -- 172.21.15.98:0/1782570749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c0b87c0 msgr2=0x7f3b3c150c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:27.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.324+0000 7f3b43bff700 1 --2- 172.21.15.98:0/1782570749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c0b87c0 0x7f3b3c150c40 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f3b3800c520 tx=0x7f3b3800c8e0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.324+0000 7f3b43bff700 1 -- 172.21.15.98:0/1782570749 shutdown_connections 2023-03-03T08:00:27.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.324+0000 7f3b43bff700 1 --2- 172.21.15.98:0/1782570749 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3b200683c0 0x7f3b2006a870 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.324+0000 7f3b43bff700 1 --2- 172.21.15.98:0/1782570749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c0b87c0 0x7f3b3c150c40 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.324+0000 7f3b43bff700 1 --2- 172.21.15.98:0/1782570749 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b3c0d23e0 0x7f3b3c15af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.324+0000 7f3b43bff700 1 --2- 172.21.15.98:0/1782570749 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b3c0b8b90 0x7f3b3c15aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.324+0000 7f3b43bff700 1 -- 172.21.15.98:0/1782570749 >> 172.21.15.98:0/1782570749 conn(0x7f3b3c06aee0 msgr2=0x7f3b3c0b1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:27.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.324+0000 7f3b43bff700 1 -- 172.21.15.98:0/1782570749 shutdown_connections 2023-03-03T08:00:27.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.325+0000 7f3b43bff700 1 -- 172.21.15.98:0/1782570749 wait complete. 2023-03-03T08:00:27.328 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T08:00:27.354 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2023-03-03T08:00:26.241434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":77884,"kb_used_data":7100,"kb_used_omap":0,"kb_used_meta":70720,"kb_avail":754896836,"statfs":{"total":773094113280,"available":773014360064,"internally_reserved":0,"allocated":7270400,"data_stored":3850200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":72417280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.482613"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773441+0000","last_change":"2023-03-03T08:00:21.773441+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773441+0000","last_undegraded":"2023-03-03T08:00:21.773441+0000","last_fullsized":"2023-03-03T08:00:21.773441+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:22:36.090135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772633+0000","last_change":"2023-03-03T08:00:21.772633+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772633+0000","last_undegraded":"2023-03-03T08:00:21.772633+0000","last_fullsized":"2023-03-03T08:00:21.772633+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:45:30.839878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.774283+0000","last_change":"2023-03-03T08:00:21.774283+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.774283+0000","last_undegraded":"2023-03-03T08:00:21.774283+0000","last_fullsized":"2023-03-03T08:00:21.774283+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:57:44.236333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773450+0000","last_change":"2023-03-03T08:00:21.773450+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773450+0000","last_undegraded":"2023-03-03T08:00:21.773450+0000","last_fullsized":"2023-03-03T08:00:21.773450+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:39.487708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.779689+0000","last_change":"2023-03-03T08:00:21.779689+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.779689+0000","last_undegraded":"2023-03-03T08:00:21.779689+0000","last_fullsized":"2023-03-03T08:00:21.779689+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:16:26.224161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773944+0000","last_change":"2023-03-03T08:00:21.773944+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773944+0000","last_undegraded":"2023-03-03T08:00:21.773944+0000","last_fullsized":"2023-03-03T08:00:21.773944+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:34.298493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.781067+0000","last_change":"2023-03-03T08:00:21.781067+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.781067+0000","last_undegraded":"2023-03-03T08:00:21.781067+0000","last_fullsized":"2023-03-03T08:00:21.781067+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:00:54.378703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.776209+0000","last_change":"2023-03-03T08:00:21.776209+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.776209+0000","last_undegraded":"2023-03-03T08:00:21.776209+0000","last_fullsized":"2023-03-03T08:00:21.776209+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:40.317030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.774137+0000","last_change":"2023-03-03T08:00:21.774137+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.774137+0000","last_undegraded":"2023-03-03T08:00:21.774137+0000","last_fullsized":"2023-03-03T08:00:21.774137+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:23:14.650660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773818+0000","last_change":"2023-03-03T08:00:21.773818+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773818+0000","last_undegraded":"2023-03-03T08:00:21.773818+0000","last_fullsized":"2023-03-03T08:00:21.773818+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:53:16.100003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.775282+0000","last_change":"2023-03-03T08:00:21.775282+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.775282+0000","last_undegraded":"2023-03-03T08:00:21.775282+0000","last_fullsized":"2023-03-03T08:00:21.775282+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:19:01.442408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.777018+0000","last_change":"2023-03-03T08:00:21.777018+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.777018+0000","last_undegraded":"2023-03-03T08:00:21.777018+0000","last_fullsized":"2023-03-03T08:00:21.777018+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:44:04.177495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.775781+0000","last_change":"2023-03-03T08:00:21.775781+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.775781+0000","last_undegraded":"2023-03-03T08:00:21.775781+0000","last_fullsized":"2023-03-03T08:00:21.775781+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:20.721006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773239+0000","last_change":"2023-03-03T08:00:21.773239+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773239+0000","last_undegraded":"2023-03-03T08:00:21.773239+0000","last_fullsized":"2023-03-03T08:00:21.773239+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:20.721006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772936+0000","last_change":"2023-03-03T08:00:21.772936+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772936+0000","last_undegraded":"2023-03-03T08:00:21.772936+0000","last_fullsized":"2023-03-03T08:00:21.772936+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:27.867451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.782056+0000","last_change":"2023-03-03T08:00:21.782056+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.782056+0000","last_undegraded":"2023-03-03T08:00:21.782056+0000","last_fullsized":"2023-03-03T08:00:21.782056+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:36:10.563980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773016+0000","last_change":"2023-03-03T08:00:21.773016+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773016+0000","last_undegraded":"2023-03-03T08:00:21.773016+0000","last_fullsized":"2023-03-03T08:00:21.773016+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:03.239405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.774024+0000","last_change":"2023-03-03T08:00:21.774024+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.774024+0000","last_undegraded":"2023-03-03T08:00:21.774024+0000","last_fullsized":"2023-03-03T08:00:21.774024+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:44:04.177495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.781646+0000","last_change":"2023-03-03T08:00:21.781646+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.781646+0000","last_undegraded":"2023-03-03T08:00:21.781646+0000","last_fullsized":"2023-03-03T08:00:21.781646+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:35:40.983679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.777573+0000","last_change":"2023-03-03T08:00:21.777573+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.777573+0000","last_undegraded":"2023-03-03T08:00:21.777573+0000","last_fullsized":"2023-03-03T08:00:21.777573+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:00:39.446569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.775218+0000","last_change":"2023-03-03T08:00:21.775218+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.775218+0000","last_undegraded":"2023-03-03T08:00:21.775218+0000","last_fullsized":"2023-03-03T08:00:21.775218+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:19:01.442408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.777020+0000","last_change":"2023-03-03T08:00:21.777020+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.777020+0000","last_undegraded":"2023-03-03T08:00:21.777020+0000","last_fullsized":"2023-03-03T08:00:21.777020+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:52:14.400121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.779432+0000","last_change":"2023-03-03T08:00:21.779432+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.779432+0000","last_undegraded":"2023-03-03T08:00:21.779432+0000","last_fullsized":"2023-03-03T08:00:21.779432+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:51:43.370212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773447+0000","last_change":"2023-03-03T08:00:21.773447+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773447+0000","last_undegraded":"2023-03-03T08:00:21.773447+0000","last_fullsized":"2023-03-03T08:00:21.773447+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:06.529333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.780224+0000","last_change":"2023-03-03T08:00:21.780224+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.780224+0000","last_undegraded":"2023-03-03T08:00:21.780224+0000","last_fullsized":"2023-03-03T08:00:21.780224+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:08:44.952506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.776014+0000","last_change":"2023-03-03T08:00:21.776014+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.776014+0000","last_undegraded":"2023-03-03T08:00:21.776014+0000","last_fullsized":"2023-03-03T08:00:21.776014+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:06:54.326996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.774820+0000","last_change":"2023-03-03T08:00:21.774820+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.774820+0000","last_undegraded":"2023-03-03T08:00:21.774820+0000","last_fullsized":"2023-03-03T08:00:21.774820+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:06.529333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773725+0000","last_change":"2023-03-03T08:00:21.773725+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773725+0000","last_undegraded":"2023-03-03T08:00:21.773725+0000","last_fullsized":"2023-03-03T08:00:21.773725+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:53:16.425693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773667+0000","last_change":"2023-03-03T08:00:21.773667+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773667+0000","last_undegraded":"2023-03-03T08:00:21.773667+0000","last_fullsized":"2023-03-03T08:00:21.773667+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:43.689368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.779528+0000","last_change":"2023-03-03T08:00:21.779528+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.779528+0000","last_undegraded":"2023-03-03T08:00:21.779528+0000","last_fullsized":"2023-03-03T08:00:21.779528+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:06.529333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773370+0000","last_change":"2023-03-03T08:00:21.773370+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773370+0000","last_undegraded":"2023-03-03T08:00:21.773370+0000","last_fullsized":"2023-03-03T08:00:21.773370+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:40.317030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773284+0000","last_change":"2023-03-03T08:00:21.773284+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773284+0000","last_undegraded":"2023-03-03T08:00:21.773284+0000","last_fullsized":"2023-03-03T08:00:21.773284+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:28:15.779229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.778004+0000","last_change":"2023-03-03T08:00:21.778004+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.778004+0000","last_undegraded":"2023-03-03T08:00:21.778004+0000","last_fullsized":"2023-03-03T08:00:21.778004+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:25:55.351299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.793214+0000","last_change":"2023-03-03T08:00:21.791637+0000","last_active":"2023-03-03T08:00:21.793214+0000","last_peered":"2023-03-03T08:00:21.793214+0000","last_clean":"2023-03-03T08:00:21.793214+0000","last_became_active":"2023-03-03T08:00:21.791370+0000","last_became_peered":"2023-03-03T08:00:21.791370+0000","last_unstale":"2023-03-03T08:00:21.793214+0000","last_undegraded":"2023-03-03T08:00:21.793214+0000","last_fullsized":"2023-03-03T08:00:21.793214+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:53:37.291520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.774368+0000","last_change":"2023-03-03T08:00:21.774368+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.774368+0000","last_undegraded":"2023-03-03T08:00:21.774368+0000","last_fullsized":"2023-03-03T08:00:21.774368+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:27.867451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.793060+0000","last_change":"2023-03-03T08:00:21.792187+0000","last_active":"2023-03-03T08:00:21.793060+0000","last_peered":"2023-03-03T08:00:21.793060+0000","last_clean":"2023-03-03T08:00:21.793060+0000","last_became_active":"2023-03-03T08:00:21.791813+0000","last_became_peered":"2023-03-03T08:00:21.791813+0000","last_unstale":"2023-03-03T08:00:21.793060+0000","last_undegraded":"2023-03-03T08:00:21.793060+0000","last_fullsized":"2023-03-03T08:00:21.793060+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:26.856771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772978+0000","last_change":"2023-03-03T08:00:21.772978+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772978+0000","last_undegraded":"2023-03-03T08:00:21.772978+0000","last_fullsized":"2023-03-03T08:00:21.772978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:00:05.665712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.789310+0000","last_change":"2023-03-03T08:00:21.787824+0000","last_active":"2023-03-03T08:00:21.789310+0000","last_peered":"2023-03-03T08:00:21.789310+0000","last_clean":"2023-03-03T08:00:21.789310+0000","last_became_active":"2023-03-03T08:00:21.787512+0000","last_became_peered":"2023-03-03T08:00:21.787512+0000","last_unstale":"2023-03-03T08:00:21.789310+0000","last_undegraded":"2023-03-03T08:00:21.789310+0000","last_fullsized":"2023-03-03T08:00:21.789310+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:42.678687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772873+0000","last_change":"2023-03-03T08:00:21.772873+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772873+0000","last_undegraded":"2023-03-03T08:00:21.772873+0000","last_fullsized":"2023-03-03T08:00:21.772873+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:04.223183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.786139+0000","last_change":"2023-03-03T08:00:21.784586+0000","last_active":"2023-03-03T08:00:21.786139+0000","last_peered":"2023-03-03T08:00:21.786139+0000","last_clean":"2023-03-03T08:00:21.786139+0000","last_became_active":"2023-03-03T08:00:21.784312+0000","last_became_peered":"2023-03-03T08:00:21.784312+0000","last_unstale":"2023-03-03T08:00:21.786139+0000","last_undegraded":"2023-03-03T08:00:21.786139+0000","last_fullsized":"2023-03-03T08:00:21.786139+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:53:15.089322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.780959+0000","last_change":"2023-03-03T08:00:21.780959+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.780959+0000","last_undegraded":"2023-03-03T08:00:21.780959+0000","last_fullsized":"2023-03-03T08:00:21.780959+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:43:23.319076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.781768+0000","last_change":"2023-03-03T08:00:21.778523+0000","last_active":"2023-03-03T08:00:21.781768+0000","last_peered":"2023-03-03T08:00:21.781768+0000","last_clean":"2023-03-03T08:00:21.781768+0000","last_became_active":"2023-03-03T08:00:21.778260+0000","last_became_peered":"2023-03-03T08:00:21.778260+0000","last_unstale":"2023-03-03T08:00:21.781768+0000","last_undegraded":"2023-03-03T08:00:21.781768+0000","last_fullsized":"2023-03-03T08:00:21.781768+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:26.856771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772265+0000","last_change":"2023-03-03T08:00:21.772265+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772265+0000","last_undegraded":"2023-03-03T08:00:21.772265+0000","last_fullsized":"2023-03-03T08:00:21.772265+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:09.676363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.790687+0000","last_change":"2023-03-03T08:00:21.787110+0000","last_active":"2023-03-03T08:00:21.790687+0000","last_peered":"2023-03-03T08:00:21.790687+0000","last_clean":"2023-03-03T08:00:21.790687+0000","last_became_active":"2023-03-03T08:00:21.786811+0000","last_became_peered":"2023-03-03T08:00:21.786811+0000","last_unstale":"2023-03-03T08:00:21.790687+0000","last_undegraded":"2023-03-03T08:00:21.790687+0000","last_fullsized":"2023-03-03T08:00:21.790687+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:57:43.225652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.783126+0000","last_change":"2023-03-03T08:00:21.783126+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.783126+0000","last_undegraded":"2023-03-03T08:00:21.783126+0000","last_fullsized":"2023-03-03T08:00:21.783126+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:35:58.114402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.787677+0000","last_change":"2023-03-03T08:00:21.783560+0000","last_active":"2023-03-03T08:00:21.787677+0000","last_peered":"2023-03-03T08:00:21.787677+0000","last_clean":"2023-03-03T08:00:21.787677+0000","last_became_active":"2023-03-03T08:00:21.783190+0000","last_became_peered":"2023-03-03T08:00:21.783190+0000","last_unstale":"2023-03-03T08:00:21.787677+0000","last_undegraded":"2023-03-03T08:00:21.787677+0000","last_fullsized":"2023-03-03T08:00:21.787677+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:39.306350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.777148+0000","last_change":"2023-03-03T08:00:21.777148+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.777148+0000","last_undegraded":"2023-03-03T08:00:21.777148+0000","last_fullsized":"2023-03-03T08:00:21.777148+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:52.425703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794928+0000","last_change":"2023-03-03T08:00:21.792376+0000","last_active":"2023-03-03T08:00:21.794928+0000","last_peered":"2023-03-03T08:00:21.794928+0000","last_clean":"2023-03-03T08:00:21.794928+0000","last_became_active":"2023-03-03T08:00:21.792126+0000","last_became_peered":"2023-03-03T08:00:21.792126+0000","last_unstale":"2023-03-03T08:00:21.794928+0000","last_undegraded":"2023-03-03T08:00:21.794928+0000","last_fullsized":"2023-03-03T08:00:21.794928+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:44.522714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773311+0000","last_change":"2023-03-03T08:00:21.773311+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773311+0000","last_undegraded":"2023-03-03T08:00:21.773311+0000","last_fullsized":"2023-03-03T08:00:21.773311+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:55:49.478050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.795011+0000","last_change":"2023-03-03T08:00:21.790545+0000","last_active":"2023-03-03T08:00:21.795011+0000","last_peered":"2023-03-03T08:00:21.795011+0000","last_clean":"2023-03-03T08:00:21.795011+0000","last_became_active":"2023-03-03T08:00:21.790052+0000","last_became_peered":"2023-03-03T08:00:21.790052+0000","last_unstale":"2023-03-03T08:00:21.795011+0000","last_undegraded":"2023-03-03T08:00:21.795011+0000","last_fullsized":"2023-03-03T08:00:21.795011+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:51:33.987978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.777009+0000","last_change":"2023-03-03T08:00:21.777009+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.777009+0000","last_undegraded":"2023-03-03T08:00:21.777009+0000","last_fullsized":"2023-03-03T08:00:21.777009+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:34.298493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.785663+0000","last_change":"2023-03-03T08:00:21.783602+0000","last_active":"2023-03-03T08:00:21.785663+0000","last_peered":"2023-03-03T08:00:21.785663+0000","last_clean":"2023-03-03T08:00:21.785663+0000","last_became_active":"2023-03-03T08:00:21.783297+0000","last_became_peered":"2023-03-03T08:00:21.783297+0000","last_unstale":"2023-03-03T08:00:21.785663+0000","last_undegraded":"2023-03-03T08:00:21.785663+0000","last_fullsized":"2023-03-03T08:00:21.785663+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:08:45.794437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.780801+0000","last_change":"2023-03-03T08:00:21.780801+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.780801+0000","last_undegraded":"2023-03-03T08:00:21.780801+0000","last_fullsized":"2023-03-03T08:00:21.780801+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:12:21.211704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.795043+0000","last_change":"2023-03-03T08:00:21.790207+0000","last_active":"2023-03-03T08:00:21.795043+0000","last_peered":"2023-03-03T08:00:21.795043+0000","last_clean":"2023-03-03T08:00:21.795043+0000","last_became_active":"2023-03-03T08:00:21.789535+0000","last_became_peered":"2023-03-03T08:00:21.789535+0000","last_unstale":"2023-03-03T08:00:21.795043+0000","last_undegraded":"2023-03-03T08:00:21.795043+0000","last_fullsized":"2023-03-03T08:00:21.795043+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:02.228725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.774137+0000","last_change":"2023-03-03T08:00:21.774137+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.774137+0000","last_undegraded":"2023-03-03T08:00:21.774137+0000","last_fullsized":"2023-03-03T08:00:21.774137+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:20.734366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.791765+0000","last_change":"2023-03-03T08:00:21.789108+0000","last_active":"2023-03-03T08:00:21.791765+0000","last_peered":"2023-03-03T08:00:21.791765+0000","last_clean":"2023-03-03T08:00:21.791765+0000","last_became_active":"2023-03-03T08:00:21.788850+0000","last_became_peered":"2023-03-03T08:00:21.788850+0000","last_unstale":"2023-03-03T08:00:21.791765+0000","last_undegraded":"2023-03-03T08:00:21.791765+0000","last_fullsized":"2023-03-03T08:00:21.791765+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:16:16.126986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772211+0000","last_change":"2023-03-03T08:00:21.772211+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772211+0000","last_undegraded":"2023-03-03T08:00:21.772211+0000","last_fullsized":"2023-03-03T08:00:21.772211+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:06:54.326996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794883+0000","last_change":"2023-03-03T08:00:21.788681+0000","last_active":"2023-03-03T08:00:21.794883+0000","last_peered":"2023-03-03T08:00:21.794883+0000","last_clean":"2023-03-03T08:00:21.794883+0000","last_became_active":"2023-03-03T08:00:21.788087+0000","last_became_peered":"2023-03-03T08:00:21.788087+0000","last_unstale":"2023-03-03T08:00:21.794883+0000","last_undegraded":"2023-03-03T08:00:21.794883+0000","last_fullsized":"2023-03-03T08:00:21.794883+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:08:45.794437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.779051+0000","last_change":"2023-03-03T08:00:21.779051+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.779051+0000","last_undegraded":"2023-03-03T08:00:21.779051+0000","last_fullsized":"2023-03-03T08:00:21.779051+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:20.734366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794772+0000","last_change":"2023-03-03T08:00:21.787442+0000","last_active":"2023-03-03T08:00:21.794772+0000","last_peered":"2023-03-03T08:00:21.794772+0000","last_clean":"2023-03-03T08:00:21.794772+0000","last_became_active":"2023-03-03T08:00:21.786782+0000","last_became_peered":"2023-03-03T08:00:21.786782+0000","last_unstale":"2023-03-03T08:00:21.794772+0000","last_undegraded":"2023-03-03T08:00:21.794772+0000","last_fullsized":"2023-03-03T08:00:21.794772+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:55:48.684553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.782556+0000","last_change":"2023-03-03T08:00:21.782556+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.782556+0000","last_undegraded":"2023-03-03T08:00:21.782556+0000","last_fullsized":"2023-03-03T08:00:21.782556+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:16:17.137666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.793146+0000","last_change":"2023-03-03T08:00:21.787586+0000","last_active":"2023-03-03T08:00:21.793146+0000","last_peered":"2023-03-03T08:00:21.793146+0000","last_clean":"2023-03-03T08:00:21.793146+0000","last_became_active":"2023-03-03T08:00:21.787272+0000","last_became_peered":"2023-03-03T08:00:21.787272+0000","last_unstale":"2023-03-03T08:00:21.793146+0000","last_undegraded":"2023-03-03T08:00:21.793146+0000","last_fullsized":"2023-03-03T08:00:21.793146+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:12.459278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772821+0000","last_change":"2023-03-03T08:00:21.772821+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772821+0000","last_undegraded":"2023-03-03T08:00:21.772821+0000","last_fullsized":"2023-03-03T08:00:21.772821+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:36:19.801740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794728+0000","last_change":"2023-03-03T08:00:21.791488+0000","last_active":"2023-03-03T08:00:21.794728+0000","last_peered":"2023-03-03T08:00:21.794728+0000","last_clean":"2023-03-03T08:00:21.794728+0000","last_became_active":"2023-03-03T08:00:21.791122+0000","last_became_peered":"2023-03-03T08:00:21.791122+0000","last_unstale":"2023-03-03T08:00:21.794728+0000","last_undegraded":"2023-03-03T08:00:21.794728+0000","last_fullsized":"2023-03-03T08:00:21.794728+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:36:55.429526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.776073+0000","last_change":"2023-03-03T08:00:21.776073+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.776073+0000","last_undegraded":"2023-03-03T08:00:21.776073+0000","last_fullsized":"2023-03-03T08:00:21.776073+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:52.425703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794987+0000","last_change":"2023-03-03T08:00:21.792101+0000","last_active":"2023-03-03T08:00:21.794987+0000","last_peered":"2023-03-03T08:00:21.794987+0000","last_clean":"2023-03-03T08:00:21.794987+0000","last_became_active":"2023-03-03T08:00:21.791481+0000","last_became_peered":"2023-03-03T08:00:21.791481+0000","last_unstale":"2023-03-03T08:00:21.794987+0000","last_undegraded":"2023-03-03T08:00:21.794987+0000","last_fullsized":"2023-03-03T08:00:21.794987+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:12:20.201024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.781035+0000","last_change":"2023-03-03T08:00:21.781035+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.781035+0000","last_undegraded":"2023-03-03T08:00:21.781035+0000","last_fullsized":"2023-03-03T08:00:21.781035+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:23:00.594155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794616+0000","last_change":"2023-03-03T08:00:21.786969+0000","last_active":"2023-03-03T08:00:21.794616+0000","last_peered":"2023-03-03T08:00:21.794616+0000","last_clean":"2023-03-03T08:00:21.794616+0000","last_became_active":"2023-03-03T08:00:21.786545+0000","last_became_peered":"2023-03-03T08:00:21.786545+0000","last_unstale":"2023-03-03T08:00:21.794616+0000","last_undegraded":"2023-03-03T08:00:21.794616+0000","last_fullsized":"2023-03-03T08:00:21.794616+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:16:25.213480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.775504+0000","last_change":"2023-03-03T08:00:21.775504+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.775504+0000","last_undegraded":"2023-03-03T08:00:21.775504+0000","last_fullsized":"2023-03-03T08:00:21.775504+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:11:16.603939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.790610+0000","last_change":"2023-03-03T08:00:21.788125+0000","last_active":"2023-03-03T08:00:21.790610+0000","last_peered":"2023-03-03T08:00:21.790610+0000","last_clean":"2023-03-03T08:00:21.790610+0000","last_became_active":"2023-03-03T08:00:21.787862+0000","last_became_peered":"2023-03-03T08:00:21.787862+0000","last_unstale":"2023-03-03T08:00:21.790610+0000","last_undegraded":"2023-03-03T08:00:21.790610+0000","last_fullsized":"2023-03-03T08:00:21.790610+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:03.212503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.780684+0000","last_change":"2023-03-03T08:00:21.780684+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.780684+0000","last_undegraded":"2023-03-03T08:00:21.780684+0000","last_fullsized":"2023-03-03T08:00:21.780684+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:49:26.845615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.791731+0000","last_change":"2023-03-03T08:00:21.789082+0000","last_active":"2023-03-03T08:00:21.791731+0000","last_peered":"2023-03-03T08:00:21.791731+0000","last_clean":"2023-03-03T08:00:21.791731+0000","last_became_active":"2023-03-03T08:00:21.788827+0000","last_became_peered":"2023-03-03T08:00:21.788827+0000","last_unstale":"2023-03-03T08:00:21.791731+0000","last_undegraded":"2023-03-03T08:00:21.791731+0000","last_fullsized":"2023-03-03T08:00:21.791731+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:58.454988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.778794+0000","last_change":"2023-03-03T08:00:21.778794+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.778794+0000","last_undegraded":"2023-03-03T08:00:21.778794+0000","last_fullsized":"2023-03-03T08:00:21.778794+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:53:38.302201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794946+0000","last_change":"2023-03-03T08:00:21.788909+0000","last_active":"2023-03-03T08:00:21.794946+0000","last_peered":"2023-03-03T08:00:21.794946+0000","last_clean":"2023-03-03T08:00:21.794946+0000","last_became_active":"2023-03-03T08:00:21.788395+0000","last_became_peered":"2023-03-03T08:00:21.788395+0000","last_unstale":"2023-03-03T08:00:21.794946+0000","last_undegraded":"2023-03-03T08:00:21.794946+0000","last_fullsized":"2023-03-03T08:00:21.794946+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:17:29.803501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772230+0000","last_change":"2023-03-03T08:00:21.772230+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772230+0000","last_undegraded":"2023-03-03T08:00:21.772230+0000","last_fullsized":"2023-03-03T08:00:21.772230+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:08:44.952506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.787003+0000","last_change":"2023-03-03T08:00:21.782978+0000","last_active":"2023-03-03T08:00:21.787003+0000","last_peered":"2023-03-03T08:00:21.787003+0000","last_clean":"2023-03-03T08:00:21.787003+0000","last_became_active":"2023-03-03T08:00:21.782679+0000","last_became_peered":"2023-03-03T08:00:21.782679+0000","last_unstale":"2023-03-03T08:00:21.787003+0000","last_undegraded":"2023-03-03T08:00:21.787003+0000","last_fullsized":"2023-03-03T08:00:21.787003+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:19:00.431727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.774168+0000","last_change":"2023-03-03T08:00:21.774168+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.774168+0000","last_undegraded":"2023-03-03T08:00:21.774168+0000","last_fullsized":"2023-03-03T08:00:21.774168+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:51:34.998659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.787659+0000","last_change":"2023-03-03T08:00:21.783463+0000","last_active":"2023-03-03T08:00:21.787659+0000","last_peered":"2023-03-03T08:00:21.787659+0000","last_clean":"2023-03-03T08:00:21.787659+0000","last_became_active":"2023-03-03T08:00:21.782809+0000","last_became_peered":"2023-03-03T08:00:21.782809+0000","last_unstale":"2023-03-03T08:00:21.787659+0000","last_undegraded":"2023-03-03T08:00:21.787659+0000","last_fullsized":"2023-03-03T08:00:21.787659+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:44:03.166814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.781502+0000","last_change":"2023-03-03T08:00:21.781502+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.781502+0000","last_undegraded":"2023-03-03T08:00:21.781502+0000","last_fullsized":"2023-03-03T08:00:21.781502+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:22:43.504725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.786556+0000","last_change":"2023-03-03T08:00:21.784441+0000","last_active":"2023-03-03T08:00:21.786556+0000","last_peered":"2023-03-03T08:00:21.786556+0000","last_clean":"2023-03-03T08:00:21.786556+0000","last_became_active":"2023-03-03T08:00:21.784193+0000","last_became_peered":"2023-03-03T08:00:21.784193+0000","last_unstale":"2023-03-03T08:00:21.786556+0000","last_undegraded":"2023-03-03T08:00:21.786556+0000","last_fullsized":"2023-03-03T08:00:21.786556+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:00:08.298430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772932+0000","last_change":"2023-03-03T08:00:21.772932+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772932+0000","last_undegraded":"2023-03-03T08:00:21.772932+0000","last_fullsized":"2023-03-03T08:00:21.772932+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:40.317030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.795248+0000","last_change":"2023-03-03T08:00:21.792370+0000","last_active":"2023-03-03T08:00:21.795248+0000","last_peered":"2023-03-03T08:00:21.795248+0000","last_clean":"2023-03-03T08:00:21.795248+0000","last_became_active":"2023-03-03T08:00:21.791836+0000","last_became_peered":"2023-03-03T08:00:21.791836+0000","last_unstale":"2023-03-03T08:00:21.795248+0000","last_undegraded":"2023-03-03T08:00:21.795248+0000","last_fullsized":"2023-03-03T08:00:21.795248+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:42:51.458743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":47,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.766482+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T08:00:20.766482+0000","last_peered":"2023-03-03T08:00:20.766482+0000","last_clean":"2023-03-03T08:00:20.766482+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T08:00:20.766482+0000","last_undegraded":"2023-03-03T08:00:20.766482+0000","last_fullsized":"2023-03-03T08:00:20.766482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.782097+0000","last_change":"2023-03-03T08:00:21.782097+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.782097+0000","last_undegraded":"2023-03-03T08:00:21.782097+0000","last_fullsized":"2023-03-03T08:00:21.782097+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:59.465669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.793733+0000","last_change":"2023-03-03T08:00:21.792272+0000","last_active":"2023-03-03T08:00:21.793733+0000","last_peered":"2023-03-03T08:00:21.793733+0000","last_clean":"2023-03-03T08:00:21.793733+0000","last_became_active":"2023-03-03T08:00:21.791961+0000","last_became_peered":"2023-03-03T08:00:21.791961+0000","last_unstale":"2023-03-03T08:00:21.793733+0000","last_undegraded":"2023-03-03T08:00:21.793733+0000","last_fullsized":"2023-03-03T08:00:21.793733+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:57:43.225652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":49,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.773696+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T08:00:20.773696+0000","last_peered":"2023-03-03T08:00:20.773696+0000","last_clean":"2023-03-03T08:00:20.773696+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T08:00:20.773696+0000","last_undegraded":"2023-03-03T08:00:20.773696+0000","last_fullsized":"2023-03-03T08:00:20.773696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.776650+0000","last_change":"2023-03-03T08:00:21.776650+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.776650+0000","last_undegraded":"2023-03-03T08:00:21.776650+0000","last_fullsized":"2023-03-03T08:00:21.776650+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:39.487708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794791+0000","last_change":"2023-03-03T08:00:21.787607+0000","last_active":"2023-03-03T08:00:21.794791+0000","last_peered":"2023-03-03T08:00:21.794791+0000","last_clean":"2023-03-03T08:00:21.794791+0000","last_became_active":"2023-03-03T08:00:21.787106+0000","last_became_peered":"2023-03-03T08:00:21.787106+0000","last_unstale":"2023-03-03T08:00:21.794791+0000","last_undegraded":"2023-03-03T08:00:21.794791+0000","last_fullsized":"2023-03-03T08:00:21.794791+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:30:19.298647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":46,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.770256+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T08:00:20.770256+0000","last_peered":"2023-03-03T08:00:20.770256+0000","last_clean":"2023-03-03T08:00:20.770256+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T08:00:20.770256+0000","last_undegraded":"2023-03-03T08:00:20.770256+0000","last_fullsized":"2023-03-03T08:00:20.770256+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.779198+0000","last_change":"2023-03-03T08:00:21.779198+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.779198+0000","last_undegraded":"2023-03-03T08:00:21.779198+0000","last_fullsized":"2023-03-03T08:00:21.779198+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:00:09.309111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.779222+0000","last_change":"2023-03-03T08:00:21.775813+0000","last_active":"2023-03-03T08:00:21.779222+0000","last_peered":"2023-03-03T08:00:21.779222+0000","last_clean":"2023-03-03T08:00:21.779222+0000","last_became_active":"2023-03-03T08:00:21.775553+0000","last_became_peered":"2023-03-03T08:00:21.775553+0000","last_unstale":"2023-03-03T08:00:21.779222+0000","last_undegraded":"2023-03-03T08:00:21.779222+0000","last_fullsized":"2023-03-03T08:00:21.779222+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:19.710326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":49,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.773574+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T08:00:20.773574+0000","last_peered":"2023-03-03T08:00:20.773574+0000","last_clean":"2023-03-03T08:00:20.773574+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T08:00:20.773574+0000","last_undegraded":"2023-03-03T08:00:20.773574+0000","last_fullsized":"2023-03-03T08:00:20.773574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":203,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.761017+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T08:00:20.761017+0000","last_peered":"2023-03-03T08:00:20.761017+0000","last_clean":"2023-03-03T08:00:20.761017+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T08:00:20.761017+0000","last_undegraded":"2023-03-03T08:00:20.761017+0000","last_fullsized":"2023-03-03T08:00:20.761017+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772831+0000","last_change":"2023-03-03T08:00:21.772831+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772831+0000","last_undegraded":"2023-03-03T08:00:21.772831+0000","last_fullsized":"2023-03-03T08:00:21.772831+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:27.364263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.792517+0000","last_change":"2023-03-03T08:00:21.790111+0000","last_active":"2023-03-03T08:00:21.792517+0000","last_peered":"2023-03-03T08:00:21.792517+0000","last_clean":"2023-03-03T08:00:21.792517+0000","last_became_active":"2023-03-03T08:00:21.789831+0000","last_became_peered":"2023-03-03T08:00:21.789831+0000","last_unstale":"2023-03-03T08:00:21.792517+0000","last_undegraded":"2023-03-03T08:00:21.792517+0000","last_fullsized":"2023-03-03T08:00:21.792517+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:19.710326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":50,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.761012+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T08:00:20.761012+0000","last_peered":"2023-03-03T08:00:20.761012+0000","last_clean":"2023-03-03T08:00:20.761012+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T08:00:20.761012+0000","last_undegraded":"2023-03-03T08:00:20.761012+0000","last_fullsized":"2023-03-03T08:00:20.761012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.781191+0000","last_change":"2023-03-03T08:00:21.781191+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.781191+0000","last_undegraded":"2023-03-03T08:00:21.781191+0000","last_fullsized":"2023-03-03T08:00:21.781191+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:30:20.309327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.792980+0000","last_change":"2023-03-03T08:00:21.787378+0000","last_active":"2023-03-03T08:00:21.792980+0000","last_peered":"2023-03-03T08:00:21.792980+0000","last_clean":"2023-03-03T08:00:21.792980+0000","last_became_active":"2023-03-03T08:00:21.787102+0000","last_became_peered":"2023-03-03T08:00:21.787102+0000","last_unstale":"2023-03-03T08:00:21.792980+0000","last_undegraded":"2023-03-03T08:00:21.792980+0000","last_fullsized":"2023-03-03T08:00:21.792980+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:45:29.829197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":50,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.761012+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T08:00:20.761012+0000","last_peered":"2023-03-03T08:00:20.761012+0000","last_clean":"2023-03-03T08:00:20.761012+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T08:00:20.761012+0000","last_undegraded":"2023-03-03T08:00:20.761012+0000","last_fullsized":"2023-03-03T08:00:20.761012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772836+0000","last_change":"2023-03-03T08:00:21.772836+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772836+0000","last_undegraded":"2023-03-03T08:00:21.772836+0000","last_fullsized":"2023-03-03T08:00:21.772836+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:20.721006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.795022+0000","last_change":"2023-03-03T08:00:21.789735+0000","last_active":"2023-03-03T08:00:21.795022+0000","last_peered":"2023-03-03T08:00:21.795022+0000","last_clean":"2023-03-03T08:00:21.795022+0000","last_became_active":"2023-03-03T08:00:21.789201+0000","last_became_peered":"2023-03-03T08:00:21.789201+0000","last_unstale":"2023-03-03T08:00:21.795022+0000","last_undegraded":"2023-03-03T08:00:21.795022+0000","last_fullsized":"2023-03-03T08:00:21.795022+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:08.665683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":47,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.765221+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T08:00:20.765221+0000","last_peered":"2023-03-03T08:00:20.765221+0000","last_clean":"2023-03-03T08:00:20.765221+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T08:00:20.765221+0000","last_undegraded":"2023-03-03T08:00:20.765221+0000","last_fullsized":"2023-03-03T08:00:20.765221+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.780220+0000","last_change":"2023-03-03T08:00:21.780220+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.780220+0000","last_undegraded":"2023-03-03T08:00:21.780220+0000","last_fullsized":"2023-03-03T08:00:21.780220+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:17:07.748684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.781897+0000","last_change":"2023-03-03T08:00:21.778956+0000","last_active":"2023-03-03T08:00:21.781897+0000","last_peered":"2023-03-03T08:00:21.781897+0000","last_clean":"2023-03-03T08:00:21.781897+0000","last_became_active":"2023-03-03T08:00:21.778666+0000","last_became_peered":"2023-03-03T08:00:21.778666+0000","last_unstale":"2023-03-03T08:00:21.781897+0000","last_undegraded":"2023-03-03T08:00:21.781897+0000","last_fullsized":"2023-03-03T08:00:21.781897+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:05.518653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.761734+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T08:00:20.761734+0000","last_peered":"2023-03-03T08:00:20.761734+0000","last_clean":"2023-03-03T08:00:20.761734+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T08:00:20.761734+0000","last_undegraded":"2023-03-03T08:00:20.761734+0000","last_fullsized":"2023-03-03T08:00:20.761734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574909,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10748,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":9408,"kb_avail":94361092,"statfs":{"total":96636764160,"available":96625758208,"internally_reserved":0,"allocated":1363968,"data_stored":928698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9633792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.67500000000000004,"15min":0.70499999999999996},"min":{"1min":0.437,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82499999999999996,"5min":1.546,"15min":1.546},"last":0.376},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71199999999999997,"15min":0.72999999999999998},"min":{"1min":0.50600000000000001,"5min":0.5,"15min":0.5},"max":{"1min":0.96499999999999997,"5min":1.702,"15min":1.702},"last":0.751}]},{"osd":1,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.89000000000000001,"15min":1.0840000000000001},"min":{"1min":0.49199999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.111,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.434},{"interface":"front","average":{"1min":0.755,"5min":0.74099999999999999,"15min":0.76900000000000002},"min":{"1min":0.51600000000000001,"5min":0.504,"15min":0.504},"max":{"1min":1.1890000000000001,"5min":1.944,"15min":1.944},"last":0.70299999999999996}]},{"osd":2,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.71899999999999997,"15min":0.71399999999999997},"min":{"1min":0.63300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.224,"5min":1.252,"15min":1.252},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76800000000000002,"15min":0.79300000000000004},"min":{"1min":0.54600000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2490000000000001,"5min":1.329,"15min":1.329},"last":0.55800000000000005}]},{"osd":3,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.71999999999999997,"15min":0.71499999999999997},"min":{"1min":0.59799999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.754,"15min":0.79000000000000004},"min":{"1min":0.55000000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.93400000000000005,"5min":1.294,"15min":1.294},"last":0.51000000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.48399999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0049999999999999,"5min":1.077,"15min":1.077},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.66700000000000004,"15min":0.63400000000000001},"min":{"1min":0.51700000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.86899999999999999}]},{"osd":5,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.77100000000000002,"15min":0.80100000000000005},"min":{"1min":0.41699999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.036,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81699999999999995,"15min":0.82399999999999995},"min":{"1min":0.66400000000000003,"5min":0.496,"15min":0.496},"max":{"1min":1.2190000000000001,"5min":1.349,"15min":1.349},"last":0.80300000000000005}]},{"osd":6,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.64100000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0700000000000001,"5min":1.204,"15min":1.204},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.81100000000000005,"15min":0.82099999999999995},"min":{"1min":0.627,"5min":0.499,"15min":0.499},"max":{"1min":1.135,"5min":1.194,"15min":1.194},"last":0.95499999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574910,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9268,"kb_used_data":748,"kb_used_omap":0,"kb_used_meta":8512,"kb_avail":94362572,"statfs":{"total":96636764160,"available":96627273728,"internally_reserved":0,"allocated":765952,"data_stored":337311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8716288},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.70799999999999996,"15min":0.68899999999999995},"min":{"1min":0.57199999999999995,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.018,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68999999999999995,"15min":0.69499999999999995},"min":{"1min":0.52500000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.89700000000000002,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":1.0549999999999999}]},{"osd":1,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70999999999999996,"15min":0.68600000000000005},"min":{"1min":0.52700000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.97099999999999997,"5min":3.323,"15min":3.323},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.73499999999999999,"15min":0.73099999999999998},"min":{"1min":0.55400000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97199999999999998,"5min":1.135,"15min":1.135},"last":0.504}]},{"osd":2,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.753,"15min":0.73199999999999998},"min":{"1min":0.59499999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.98899999999999999,"5min":1.54,"15min":1.54},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.749,"15min":0.75900000000000001},"min":{"1min":0.54400000000000004,"5min":0.495,"15min":0.495},"max":{"1min":1.0089999999999999,"5min":1.698,"15min":1.698},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81299999999999994,"15min":0.78900000000000003},"min":{"1min":0.57499999999999996,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.3089999999999999,"5min":4.8959999999999999,"15min":4.8959999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77200000000000002,"15min":0.77100000000000002},"min":{"1min":0.60899999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.95799999999999996,"5min":1.343,"15min":1.343},"last":1.0800000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.79900000000000004,"15min":0.91100000000000003},"min":{"1min":0.439,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.871,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.80700000000000005,"15min":0.92400000000000004},"min":{"1min":0.53000000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.85099999999999998,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":1.012}]},{"osd":5,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.72599999999999998,"15min":0.77600000000000002},"min":{"1min":0.47599999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.95399999999999996,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.80300000000000005,"15min":0.82499999999999996},"min":{"1min":0.499,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91500000000000004,"5min":1.429,"15min":1.429},"last":1.202}]},{"osd":7,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.81899999999999995,"15min":0.83899999999999997},"min":{"1min":0.56699999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.0029999999999999,"5min":1.452,"15min":1.452},"last":1.107},{"interface":"front","average":{"1min":0.753,"5min":0.82099999999999995,"15min":0.87},"min":{"1min":0.59399999999999997,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.008,"5min":1.579,"15min":1.579},"last":0.76600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9324,"kb_used_data":740,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362516,"statfs":{"total":96636764160,"available":96627216384,"internally_reserved":0,"allocated":757760,"data_stored":331710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.49099999999999999,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.63100000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58099999999999996,"15min":0.64700000000000002},"min":{"1min":0.36799999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.90200000000000002,"5min":3.972,"15min":3.972},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.74099999999999999,"15min":0.96399999999999997},"min":{"1min":0.371,"5min":0.313,"15min":0.313},"max":{"1min":0.78900000000000003,"5min":11.897,"15min":11.897},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.82299999999999995,"15min":1.0469999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.85199999999999998,"5min":11.804,"15min":11.804},"last":0.73699999999999999}]},{"osd":3,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.81399999999999995,"15min":1.036},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.89500000000000002,"5min":11.843999999999999,"15min":11.843999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64300000000000002,"15min":0.73499999999999999},"min":{"1min":0.47899999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.96399999999999997,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.54500000000000004}]},{"osd":4,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81799999999999995,"15min":0.86299999999999999},"min":{"1min":0.55000000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0960000000000001,"5min":1.613,"15min":1.613},"last":0.745},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.86099999999999999},"min":{"1min":0.65000000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.0800000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.76200000000000001}]},{"osd":5,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.84699999999999998,"15min":0.92500000000000004},"min":{"1min":0.63900000000000001,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.997,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81999999999999995,"15min":0.873},"min":{"1min":0.57899999999999996,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98899999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.66100000000000003}]},{"osd":6,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.91300000000000003,"15min":0.999},"min":{"1min":0.68400000000000005,"5min":0.628,"15min":0.628},"max":{"1min":1.173,"5min":3.907,"15min":3.907},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.875,"5min":0.88400000000000001,"15min":0.93999999999999995},"min":{"1min":0.67300000000000004,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1819999999999999,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.70999999999999996}]},{"osd":7,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84499999999999997,"15min":0.88300000000000001},"min":{"1min":0.70399999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.026,"5min":1.419,"15min":1.419},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.871,"5min":0.90600000000000003,"15min":1.012},"min":{"1min":0.70399999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1659999999999999,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":0.67700000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574909,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10476,"kb_used_data":1316,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94361364,"statfs":{"total":96636764160,"available":96626036736,"internally_reserved":0,"allocated":1347584,"data_stored":921059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45800000000000002,"15min":0.47199999999999998},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65000000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.38},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.50600000000000001,"15min":0.51700000000000002},"min":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81999999999999995,"5min":0.871,"15min":0.871},"last":0.54300000000000004}]},{"osd":2,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53000000000000003,"15min":0.55000000000000004},"min":{"1min":0.38400000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77400000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.44,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73399999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.503}]},{"osd":3,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70399999999999996,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.46600000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.80800000000000005,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.53100000000000003}]},{"osd":4,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82299999999999995,"15min":0.84499999999999997},"min":{"1min":0.64700000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79800000000000004,"15min":0.81999999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0740000000000001,"5min":1.605,"15min":1.605},"last":0.96599999999999997}]},{"osd":5,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.80400000000000005,"15min":0.82199999999999995},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0980000000000001,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.871},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.82099999999999995,"15min":0.82499999999999996},"min":{"1min":0.61299999999999999,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.2070000000000001,"5min":1.286,"15min":1.286},"last":0.90400000000000003}]},{"osd":6,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.85399999999999998,"15min":0.88500000000000001},"min":{"1min":0.65300000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2470000000000001,"5min":1.623,"15min":1.623},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.86399999999999999},"min":{"1min":0.66300000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.137,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":1.0029999999999999}]},{"osd":7,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84499999999999997,"15min":0.88300000000000001},"min":{"1min":0.59399999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.083,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.83999999999999997,"15min":0.86199999999999999},"min":{"1min":0.58199999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.1719999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.93500000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574910,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9384,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":8640,"kb_avail":94362456,"statfs":{"total":96636764160,"available":96627154944,"internally_reserved":0,"allocated":753664,"data_stored":330691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8847360},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.64800000000000002,"15min":0.87},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.61699999999999999,"5min":5.7889999999999997,"15min":5.7889999999999997},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.60899999999999999,"15min":0.73199999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67400000000000004,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":0.40600000000000003}]},{"osd":1,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.629,"15min":0.753},"min":{"1min":0.36299999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77700000000000002,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.38},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.79300000000000004,"15min":1.0089999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.77100000000000002,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.60799999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.88900000000000001,"15min":1.222},"min":{"1min":0.41399999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.69799999999999995,"5min":12.961,"15min":12.961},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.80000000000000004,"15min":1.036},"min":{"1min":0.42699999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.74099999999999999,"5min":6.1449999999999996,"15min":6.1449999999999996},"last":0.56299999999999994}]},{"osd":4,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.0469999999999999,"15min":1.2769999999999999},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1639999999999999,"5min":6.484,"15min":6.484},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.0580000000000001,"15min":1.2549999999999999},"min":{"1min":0.63600000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.1759999999999999,"5min":6.5510000000000002,"15min":6.5510000000000002},"last":0.67400000000000004}]},{"osd":5,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.97399999999999998,"15min":1.1799999999999999},"min":{"1min":0.61499999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.034,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.84699999999999998,"5min":1.0269999999999999,"15min":1.238},"min":{"1min":0.60999999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1499999999999999,"5min":6.6369999999999996,"15min":6.6369999999999996},"last":0.63600000000000001}]},{"osd":6,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.0429999999999999,"15min":1.24},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.1240000000000001,"5min":6.625,"15min":6.625},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.88200000000000001,"5min":1.0549999999999999,"15min":1.2669999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.099,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.69899999999999995}]},{"osd":7,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":1.0129999999999999,"15min":1.2310000000000001},"min":{"1min":0.69899999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.087,"5min":6.7009999999999996,"15min":6.7009999999999996},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.0209999999999999,"15min":1.214},"min":{"1min":0.66500000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.081,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.621}]}]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9772,"kb_used_data":740,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94362068,"statfs":{"total":96636764160,"available":96626757632,"internally_reserved":0,"allocated":757760,"data_stored":331710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.63,"15min":0.83499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.56599999999999995,"5min":9.5820000000000007,"15min":9.5820000000000007},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.70099999999999996,"15min":0.90600000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83599999999999997,"5min":9.7840000000000007,"15min":9.7840000000000007},"last":0.42999999999999999}]},{"osd":1,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.501,"15min":0.52300000000000002},"min":{"1min":0.34999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78900000000000003,"5min":1,"15min":1},"last":0.371},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.75600000000000001,"15min":0.98999999999999999},"min":{"1min":0.41099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70499999999999996,"5min":9.702,"15min":9.702},"last":0.55600000000000005}]},{"osd":2,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.746,"15min":0.93600000000000005},"min":{"1min":0.436,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88300000000000001,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.80500000000000005,"15min":1.0229999999999999},"min":{"1min":0.498,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.86499999999999999,"5min":9.734,"15min":9.734},"last":0.65300000000000002}]},{"osd":4,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.96099999999999997,"15min":1.165},"min":{"1min":0.54400000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1819999999999999,"5min":9.8629999999999995,"15min":9.8629999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.83999999999999997,"15min":0.84799999999999998},"min":{"1min":0.629,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.097,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":0.69099999999999995}]},{"osd":5,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82099999999999995,"15min":0.83299999999999996},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.1899999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.999,"15min":1.222},"min":{"1min":0.628,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0860000000000001,"5min":9.6600000000000001,"15min":9.6600000000000001},"last":0.72399999999999998}]},{"osd":6,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.0129999999999999,"15min":1.2370000000000001},"min":{"1min":0.65700000000000003,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0600000000000001,"5min":9.8190000000000008,"15min":9.8190000000000008},"last":0.629},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82799999999999996,"15min":0.86699999999999999},"min":{"1min":0.629,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.2609999999999999,"5min":1.609,"15min":1.609},"last":0.73999999999999999}]},{"osd":7,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":1.0369999999999999,"15min":1.2749999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.204,"5min":9.625,"15min":9.625},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":1.034,"15min":1.244},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.119,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":0.70899999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9452,"kb_used_data":740,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362388,"statfs":{"total":96636764160,"available":96627085312,"internally_reserved":0,"allocated":757760,"data_stored":331710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.85199999999999998,"15min":0.98299999999999998},"min":{"1min":0.433,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0049999999999999,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.68400000000000005,"15min":0.66600000000000004},"min":{"1min":0.56100000000000005,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.95399999999999996,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.63700000000000001}]},{"osd":1,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.92700000000000005,"15min":1.1080000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94199999999999995,"5min":9.5709999999999997,"15min":9.5709999999999997},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.83999999999999997,"15min":0.92700000000000005},"min":{"1min":0.58599999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0349999999999999,"5min":5.266,"15min":5.266},"last":0.72299999999999998}]},{"osd":2,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.91600000000000004,"15min":1.077},"min":{"1min":0.60299999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.016,"5min":8.4529999999999994,"15min":8.4529999999999994},"last":0.749},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.88,"15min":1.002},"min":{"1min":0.53200000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.018,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.66000000000000003}]},{"osd":3,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.72899999999999998,"15min":0.73299999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.79000000000000004,"15min":0.80200000000000005},"min":{"1min":0.58399999999999996,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.98499999999999999,"5min":1.389,"15min":1.389},"last":0.80500000000000005}]},{"osd":5,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.71799999999999997,"15min":0.74199999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86799999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.72299999999999998,"15min":0.72099999999999997},"min":{"1min":0.47599999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.998,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.70299999999999996}]},{"osd":6,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.77300000000000002,"15min":0.78100000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.94599999999999995,"5min":1.032,"15min":1.032},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.501,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0700000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.871}]},{"osd":7,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.79700000000000004,"15min":0.82199999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0609999999999999,"5min":1.679,"15min":1.679},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.88,"15min":0.92800000000000005},"min":{"1min":0.64300000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0069999999999999,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.85199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574909,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9460,"kb_used_data":748,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362380,"statfs":{"total":96636764160,"available":96627077120,"internally_reserved":0,"allocated":765952,"data_stored":337311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.65700000000000003,"15min":0.64100000000000001},"min":{"1min":0.42899999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.435},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.75700000000000001,"15min":0.74399999999999999},"min":{"1min":0.60599999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96999999999999997,"5min":2.528,"15min":2.528},"last":1.038}]},{"osd":1,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.73699999999999999,"15min":0.745},"min":{"1min":0.66400000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.67200000000000004,"15min":0.63700000000000001},"min":{"1min":0.53900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.81899999999999995}]},{"osd":2,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69199999999999995,"15min":0.68300000000000005},"min":{"1min":0.52700000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.95399999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.72499999999999998,"15min":0.71599999999999997},"min":{"1min":0.59299999999999997,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.89200000000000002,"5min":1.321,"15min":1.321},"last":0.74299999999999999}]},{"osd":3,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.71599999999999997,"15min":0.69499999999999995},"min":{"1min":0.55600000000000005,"5min":0.496,"15min":0.496},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.873},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.745,"15min":0.73499999999999999},"min":{"1min":0.629,"5min":0.501,"15min":0.501},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.82899999999999996}]},{"osd":4,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.51400000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.79000000000000004,"15min":0.78200000000000003},"min":{"1min":0.59899999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":1.079}]},{"osd":6,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.79700000000000004},"min":{"1min":0.53000000000000003,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.94499999999999995,"5min":1.071,"15min":1.071},"last":1.012},{"interface":"front","average":{"1min":0.754,"5min":0.72199999999999998,"15min":0.72399999999999998},"min":{"1min":0.58999999999999997,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97099999999999997,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.65400000000000003}]},{"osd":7,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.75600000000000001,"15min":0.75700000000000001},"min":{"1min":0.55700000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.026,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81599999999999995,"15min":0.83299999999999996},"min":{"1min":0.67700000000000005,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.90400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:00:27.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:00:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.638+0000 7ff1e01ff700 1 Processor -- start 2023-03-03T08:00:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.638+0000 7ff1e01ff700 1 -- start start 2023-03-03T08:00:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.639+0000 7ff1e01ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff1d80b87c0 0x7ff1d80cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.639+0000 7ff1e01ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff1d80b8b90 0x7ff1d80b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.639+0000 7ff1e01ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d80d0ef0 con 0x7ff1d80b5240 2023-03-03T08:00:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.639+0000 7ff1e01ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1d80d0600 con 0x7ff1d80b87c0 2023-03-03T08:00:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.639+0000 7ff1e01ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1d80cd970 con 0x7ff1d80b8b90 2023-03-03T08:00:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.639+0000 7ff1ddf9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff1d80b87c0 0x7ff1d80cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.639+0000 7ff1ddf9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff1d80b87c0 0x7ff1d80cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44964/0 (socket says 172.21.15.98:44964) 2023-03-03T08:00:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.639+0000 7ff1ddf9b700 1 -- 172.21.15.98:0/3245278324 learned_addr learned my addr 172.21.15.98:0/3245278324 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.640+0000 7ff1ddf9b700 1 -- 172.21.15.98:0/3245278324 >> v2:172.21.15.100:3300/0 conn(0x7ff1d80b8b90 msgr2=0x7ff1d80b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.640+0000 7ff1ddf9b700 1 --2- 172.21.15.98:0/3245278324 >> v2:172.21.15.100:3300/0 conn(0x7ff1d80b8b90 0x7ff1d80b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.640+0000 7ff1ddf9b700 1 -- 172.21.15.98:0/3245278324 >> v1:172.21.15.100:6789/0 conn(0x7ff1d80b5240 legacy=0x7ff1d81605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.640+0000 7ff1ddf9b700 1 -- 172.21.15.98:0/3245278324 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d80cdc50 con 0x7ff1d80b87c0 2023-03-03T08:00:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.640+0000 7ff1ddf9b700 1 --2- 172.21.15.98:0/3245278324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d80b87c0 0x7ff1d80cffe0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7ff1d808c240 tx=0x7ff1d4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a467719676addd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.640+0000 7ff1dcf99700 1 -- 172.21.15.98:0/3245278324 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4004230 con 0x7ff1d80b87c0 2023-03-03T08:00:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.641+0000 7ff1dcf99700 1 -- 172.21.15.98:0/3245278324 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d4004390 con 0x7ff1d80b87c0 2023-03-03T08:00:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.641+0000 7ff1dcf99700 1 -- 172.21.15.98:0/3245278324 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d40036a0 con 0x7ff1d80b87c0 2023-03-03T08:00:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.641+0000 7ff1e01ff700 1 -- 172.21.15.98:0/3245278324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d80b87c0 msgr2=0x7ff1d80cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.641+0000 7ff1e01ff700 1 --2- 172.21.15.98:0/3245278324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d80b87c0 0x7ff1d80cffe0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7ff1d808c240 tx=0x7ff1d4009bd0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.641+0000 7ff1e01ff700 1 -- 172.21.15.98:0/3245278324 shutdown_connections 2023-03-03T08:00:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.641+0000 7ff1e01ff700 1 --2- 172.21.15.98:0/3245278324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d80b87c0 0x7ff1d80cffe0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.641+0000 7ff1e01ff700 1 --2- 172.21.15.98:0/3245278324 >> v2:172.21.15.100:3300/0 conn(0x7ff1d80b8b90 0x7ff1d80b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.641+0000 7ff1e01ff700 1 -- 172.21.15.98:0/3245278324 >> 172.21.15.98:0/3245278324 conn(0x7ff1d806aee0 msgr2=0x7ff1d806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:27.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.641+0000 7ff1e01ff700 1 -- 172.21.15.98:0/3245278324 shutdown_connections 2023-03-03T08:00:27.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.641+0000 7ff1e01ff700 1 -- 172.21.15.98:0/3245278324 wait complete. 2023-03-03T08:00:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.643+0000 7ff1e01ff700 1 Processor -- start 2023-03-03T08:00:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.643+0000 7ff1e01ff700 1 -- start start 2023-03-03T08:00:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.644+0000 7ff1e01ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d80b5240 0x7ff1d815ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.644+0000 7ff1e01ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1d80b87c0 0x7ff1d80ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.644+0000 7ff1e01ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1d80b8b90 0x7ff1d81e7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.644+0000 7ff1e01ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d815b040 con 0x7ff1d80b5240 2023-03-03T08:00:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.644+0000 7ff1e01ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d81e6cf0 con 0x7ff1d80b87c0 2023-03-03T08:00:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.644+0000 7ff1e01ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d81e6fd0 con 0x7ff1d80b8b90 2023-03-03T08:00:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.645+0000 7ff1ddf9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d80b5240 0x7ff1d815ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.645+0000 7ff1ddf9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d80b5240 0x7ff1d815ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44966/0 (socket says 172.21.15.98:44966) 2023-03-03T08:00:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.645+0000 7ff1ddf9b700 1 -- 172.21.15.98:0/2806586956 learned_addr learned my addr 172.21.15.98:0/2806586956 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:27.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.645+0000 7ff1ddf9b700 1 -- 172.21.15.98:0/2806586956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1d80b8b90 msgr2=0x7ff1d81e7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:27.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.645+0000 7ff1ddf9b700 1 --2- 172.21.15.98:0/2806586956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1d80b8b90 0x7ff1d81e7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.645+0000 7ff1ddf9b700 1 -- 172.21.15.98:0/2806586956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1d80b87c0 msgr2=0x7ff1d80ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:27.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.645+0000 7ff1ddf9b700 1 --2- 172.21.15.98:0/2806586956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1d80b87c0 0x7ff1d80ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.645+0000 7ff1ddf9b700 1 -- 172.21.15.98:0/2806586956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d4009550 con 0x7ff1d80b5240 2023-03-03T08:00:27.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.645+0000 7ff1ddf9b700 1 --2- 172.21.15.98:0/2806586956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d80b5240 0x7ff1d815ab00 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80d0a60 tx=0x7ff1d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:27.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.646+0000 7ff1ce7fc700 1 -- 172.21.15.98:0/2806586956 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4003dc0 con 0x7ff1d80b5240 2023-03-03T08:00:27.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.646+0000 7ff1ce7fc700 1 -- 172.21.15.98:0/2806586956 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d401f070 con 0x7ff1d80b5240 2023-03-03T08:00:27.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.646+0000 7ff1ce7fc700 1 -- 172.21.15.98:0/2806586956 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4011870 con 0x7ff1d80b5240 2023-03-03T08:00:27.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.646+0000 7ff1e01ff700 1 -- 172.21.15.98:0/2806586956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d80cdf30 con 0x7ff1d80b5240 2023-03-03T08:00:27.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.646+0000 7ff1e01ff700 1 -- 172.21.15.98:0/2806586956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d8073a80 con 0x7ff1d80b5240 2023-03-03T08:00:27.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.647+0000 7ff1e01ff700 1 -- 172.21.15.98:0/2806586956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d80ce0c0 con 0x7ff1d80b5240 2023-03-03T08:00:27.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.648+0000 7ff1ce7fc700 1 -- 172.21.15.98:0/2806586956 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7ff1d4017020 con 0x7ff1d80b5240 2023-03-03T08:00:27.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.650+0000 7ff1ce7fc700 1 --2- 172.21.15.98:0/2806586956 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff1bc0683c0 0x7ff1bc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:27.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.650+0000 7ff1ce7fc700 1 -- 172.21.15.98:0/2806586956 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff1d40877a0 con 0x7ff1d80b5240 2023-03-03T08:00:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.651+0000 7ff1dd79a700 1 --2- 172.21.15.98:0/2806586956 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff1bc0683c0 0x7ff1bc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.651+0000 7ff1dd79a700 1 --2- 172.21.15.98:0/2806586956 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff1bc0683c0 0x7ff1bc06a870 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff1d81e8dd0 tx=0x7ff1c8006c80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:27.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.655+0000 7ff1ce7fc700 1 -- 172.21.15.98:0/2806586956 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff1d4053de0 con 0x7ff1d80b5240 2023-03-03T08:00:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.846+0000 7ff1e01ff700 1 -- 172.21.15.98:0/2806586956 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff1d80729b0 con 0x7ff1bc0683c0 2023-03-03T08:00:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.855+0000 7ff1ce7fc700 1 -- 172.21.15.98:0/2806586956 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+307590 (secure 0 0 0) 0x7ff1d80729b0 con 0x7ff1bc0683c0 2023-03-03T08:00:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:27.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.860+0000 7ff1e01ff700 1 -- 172.21.15.98:0/2806586956 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff1bc0683c0 msgr2=0x7ff1bc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:27.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.860+0000 7ff1e01ff700 1 --2- 172.21.15.98:0/2806586956 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff1bc0683c0 0x7ff1bc06a870 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff1d81e8dd0 tx=0x7ff1c8006c80 comp rx=0 tx=0).stop 2023-03-03T08:00:27.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.860+0000 7ff1e01ff700 1 -- 172.21.15.98:0/2806586956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d80b5240 msgr2=0x7ff1d815ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:27.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.860+0000 7ff1e01ff700 1 --2- 172.21.15.98:0/2806586956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d80b5240 0x7ff1d815ab00 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80d0a60 tx=0x7ff1d4004750 comp rx=0 tx=0).stop 2023-03-03T08:00:27.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.860+0000 7ff1e01ff700 1 -- 172.21.15.98:0/2806586956 shutdown_connections 2023-03-03T08:00:27.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.860+0000 7ff1e01ff700 1 --2- 172.21.15.98:0/2806586956 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff1bc0683c0 0x7ff1bc06a870 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.860+0000 7ff1e01ff700 1 --2- 172.21.15.98:0/2806586956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d80b5240 0x7ff1d815ab00 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.860+0000 7ff1e01ff700 1 --2- 172.21.15.98:0/2806586956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1d80b8b90 0x7ff1d81e7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.860+0000 7ff1e01ff700 1 --2- 172.21.15.98:0/2806586956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1d80b87c0 0x7ff1d80ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:27.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.860+0000 7ff1e01ff700 1 -- 172.21.15.98:0/2806586956 >> 172.21.15.98:0/2806586956 conn(0x7ff1d806aee0 msgr2=0x7ff1d80b7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:27.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.860+0000 7ff1e01ff700 1 -- 172.21.15.98:0/2806586956 shutdown_connections 2023-03-03T08:00:27.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:27.860+0000 7ff1e01ff700 1 -- 172.21.15.98:0/2806586956 wait complete. 2023-03-03T08:00:27.863 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T08:00:27.889 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2023-03-03T08:00:26.241434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":77884,"kb_used_data":7100,"kb_used_omap":0,"kb_used_meta":70720,"kb_avail":754896836,"statfs":{"total":773094113280,"available":773014360064,"internally_reserved":0,"allocated":7270400,"data_stored":3850200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":72417280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.482613"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773441+0000","last_change":"2023-03-03T08:00:21.773441+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773441+0000","last_undegraded":"2023-03-03T08:00:21.773441+0000","last_fullsized":"2023-03-03T08:00:21.773441+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:22:36.090135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772633+0000","last_change":"2023-03-03T08:00:21.772633+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772633+0000","last_undegraded":"2023-03-03T08:00:21.772633+0000","last_fullsized":"2023-03-03T08:00:21.772633+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:45:30.839878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.774283+0000","last_change":"2023-03-03T08:00:21.774283+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.774283+0000","last_undegraded":"2023-03-03T08:00:21.774283+0000","last_fullsized":"2023-03-03T08:00:21.774283+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:57:44.236333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773450+0000","last_change":"2023-03-03T08:00:21.773450+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773450+0000","last_undegraded":"2023-03-03T08:00:21.773450+0000","last_fullsized":"2023-03-03T08:00:21.773450+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:39.487708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.779689+0000","last_change":"2023-03-03T08:00:21.779689+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.779689+0000","last_undegraded":"2023-03-03T08:00:21.779689+0000","last_fullsized":"2023-03-03T08:00:21.779689+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:16:26.224161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773944+0000","last_change":"2023-03-03T08:00:21.773944+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773944+0000","last_undegraded":"2023-03-03T08:00:21.773944+0000","last_fullsized":"2023-03-03T08:00:21.773944+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:34.298493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.781067+0000","last_change":"2023-03-03T08:00:21.781067+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.781067+0000","last_undegraded":"2023-03-03T08:00:21.781067+0000","last_fullsized":"2023-03-03T08:00:21.781067+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:00:54.378703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.776209+0000","last_change":"2023-03-03T08:00:21.776209+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.776209+0000","last_undegraded":"2023-03-03T08:00:21.776209+0000","last_fullsized":"2023-03-03T08:00:21.776209+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:40.317030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.774137+0000","last_change":"2023-03-03T08:00:21.774137+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.774137+0000","last_undegraded":"2023-03-03T08:00:21.774137+0000","last_fullsized":"2023-03-03T08:00:21.774137+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:23:14.650660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773818+0000","last_change":"2023-03-03T08:00:21.773818+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773818+0000","last_undegraded":"2023-03-03T08:00:21.773818+0000","last_fullsized":"2023-03-03T08:00:21.773818+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:53:16.100003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.775282+0000","last_change":"2023-03-03T08:00:21.775282+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.775282+0000","last_undegraded":"2023-03-03T08:00:21.775282+0000","last_fullsized":"2023-03-03T08:00:21.775282+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:19:01.442408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.777018+0000","last_change":"2023-03-03T08:00:21.777018+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.777018+0000","last_undegraded":"2023-03-03T08:00:21.777018+0000","last_fullsized":"2023-03-03T08:00:21.777018+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:44:04.177495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.775781+0000","last_change":"2023-03-03T08:00:21.775781+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.775781+0000","last_undegraded":"2023-03-03T08:00:21.775781+0000","last_fullsized":"2023-03-03T08:00:21.775781+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:20.721006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773239+0000","last_change":"2023-03-03T08:00:21.773239+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773239+0000","last_undegraded":"2023-03-03T08:00:21.773239+0000","last_fullsized":"2023-03-03T08:00:21.773239+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:20.721006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772936+0000","last_change":"2023-03-03T08:00:21.772936+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772936+0000","last_undegraded":"2023-03-03T08:00:21.772936+0000","last_fullsized":"2023-03-03T08:00:21.772936+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:27.867451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.782056+0000","last_change":"2023-03-03T08:00:21.782056+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.782056+0000","last_undegraded":"2023-03-03T08:00:21.782056+0000","last_fullsized":"2023-03-03T08:00:21.782056+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:36:10.563980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773016+0000","last_change":"2023-03-03T08:00:21.773016+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773016+0000","last_undegraded":"2023-03-03T08:00:21.773016+0000","last_fullsized":"2023-03-03T08:00:21.773016+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:03.239405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.774024+0000","last_change":"2023-03-03T08:00:21.774024+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.774024+0000","last_undegraded":"2023-03-03T08:00:21.774024+0000","last_fullsized":"2023-03-03T08:00:21.774024+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:44:04.177495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.781646+0000","last_change":"2023-03-03T08:00:21.781646+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.781646+0000","last_undegraded":"2023-03-03T08:00:21.781646+0000","last_fullsized":"2023-03-03T08:00:21.781646+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:35:40.983679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.777573+0000","last_change":"2023-03-03T08:00:21.777573+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.777573+0000","last_undegraded":"2023-03-03T08:00:21.777573+0000","last_fullsized":"2023-03-03T08:00:21.777573+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:00:39.446569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.775218+0000","last_change":"2023-03-03T08:00:21.775218+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.775218+0000","last_undegraded":"2023-03-03T08:00:21.775218+0000","last_fullsized":"2023-03-03T08:00:21.775218+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:19:01.442408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.777020+0000","last_change":"2023-03-03T08:00:21.777020+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.777020+0000","last_undegraded":"2023-03-03T08:00:21.777020+0000","last_fullsized":"2023-03-03T08:00:21.777020+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:52:14.400121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.779432+0000","last_change":"2023-03-03T08:00:21.779432+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.779432+0000","last_undegraded":"2023-03-03T08:00:21.779432+0000","last_fullsized":"2023-03-03T08:00:21.779432+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:51:43.370212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773447+0000","last_change":"2023-03-03T08:00:21.773447+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773447+0000","last_undegraded":"2023-03-03T08:00:21.773447+0000","last_fullsized":"2023-03-03T08:00:21.773447+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:06.529333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.780224+0000","last_change":"2023-03-03T08:00:21.780224+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.780224+0000","last_undegraded":"2023-03-03T08:00:21.780224+0000","last_fullsized":"2023-03-03T08:00:21.780224+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:08:44.952506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.776014+0000","last_change":"2023-03-03T08:00:21.776014+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.776014+0000","last_undegraded":"2023-03-03T08:00:21.776014+0000","last_fullsized":"2023-03-03T08:00:21.776014+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:06:54.326996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.774820+0000","last_change":"2023-03-03T08:00:21.774820+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.774820+0000","last_undegraded":"2023-03-03T08:00:21.774820+0000","last_fullsized":"2023-03-03T08:00:21.774820+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:06.529333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773725+0000","last_change":"2023-03-03T08:00:21.773725+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773725+0000","last_undegraded":"2023-03-03T08:00:21.773725+0000","last_fullsized":"2023-03-03T08:00:21.773725+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:53:16.425693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773667+0000","last_change":"2023-03-03T08:00:21.773667+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773667+0000","last_undegraded":"2023-03-03T08:00:21.773667+0000","last_fullsized":"2023-03-03T08:00:21.773667+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:43.689368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.779528+0000","last_change":"2023-03-03T08:00:21.779528+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.779528+0000","last_undegraded":"2023-03-03T08:00:21.779528+0000","last_fullsized":"2023-03-03T08:00:21.779528+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:06.529333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773370+0000","last_change":"2023-03-03T08:00:21.773370+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773370+0000","last_undegraded":"2023-03-03T08:00:21.773370+0000","last_fullsized":"2023-03-03T08:00:21.773370+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:40.317030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773284+0000","last_change":"2023-03-03T08:00:21.773284+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773284+0000","last_undegraded":"2023-03-03T08:00:21.773284+0000","last_fullsized":"2023-03-03T08:00:21.773284+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:28:15.779229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.778004+0000","last_change":"2023-03-03T08:00:21.778004+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.778004+0000","last_undegraded":"2023-03-03T08:00:21.778004+0000","last_fullsized":"2023-03-03T08:00:21.778004+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:25:55.351299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.793214+0000","last_change":"2023-03-03T08:00:21.791637+0000","last_active":"2023-03-03T08:00:21.793214+0000","last_peered":"2023-03-03T08:00:21.793214+0000","last_clean":"2023-03-03T08:00:21.793214+0000","last_became_active":"2023-03-03T08:00:21.791370+0000","last_became_peered":"2023-03-03T08:00:21.791370+0000","last_unstale":"2023-03-03T08:00:21.793214+0000","last_undegraded":"2023-03-03T08:00:21.793214+0000","last_fullsized":"2023-03-03T08:00:21.793214+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:53:37.291520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.774368+0000","last_change":"2023-03-03T08:00:21.774368+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.774368+0000","last_undegraded":"2023-03-03T08:00:21.774368+0000","last_fullsized":"2023-03-03T08:00:21.774368+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:27.867451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.793060+0000","last_change":"2023-03-03T08:00:21.792187+0000","last_active":"2023-03-03T08:00:21.793060+0000","last_peered":"2023-03-03T08:00:21.793060+0000","last_clean":"2023-03-03T08:00:21.793060+0000","last_became_active":"2023-03-03T08:00:21.791813+0000","last_became_peered":"2023-03-03T08:00:21.791813+0000","last_unstale":"2023-03-03T08:00:21.793060+0000","last_undegraded":"2023-03-03T08:00:21.793060+0000","last_fullsized":"2023-03-03T08:00:21.793060+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:26.856771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772978+0000","last_change":"2023-03-03T08:00:21.772978+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772978+0000","last_undegraded":"2023-03-03T08:00:21.772978+0000","last_fullsized":"2023-03-03T08:00:21.772978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:00:05.665712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.789310+0000","last_change":"2023-03-03T08:00:21.787824+0000","last_active":"2023-03-03T08:00:21.789310+0000","last_peered":"2023-03-03T08:00:21.789310+0000","last_clean":"2023-03-03T08:00:21.789310+0000","last_became_active":"2023-03-03T08:00:21.787512+0000","last_became_peered":"2023-03-03T08:00:21.787512+0000","last_unstale":"2023-03-03T08:00:21.789310+0000","last_undegraded":"2023-03-03T08:00:21.789310+0000","last_fullsized":"2023-03-03T08:00:21.789310+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:42.678687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772873+0000","last_change":"2023-03-03T08:00:21.772873+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772873+0000","last_undegraded":"2023-03-03T08:00:21.772873+0000","last_fullsized":"2023-03-03T08:00:21.772873+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:04.223183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.786139+0000","last_change":"2023-03-03T08:00:21.784586+0000","last_active":"2023-03-03T08:00:21.786139+0000","last_peered":"2023-03-03T08:00:21.786139+0000","last_clean":"2023-03-03T08:00:21.786139+0000","last_became_active":"2023-03-03T08:00:21.784312+0000","last_became_peered":"2023-03-03T08:00:21.784312+0000","last_unstale":"2023-03-03T08:00:21.786139+0000","last_undegraded":"2023-03-03T08:00:21.786139+0000","last_fullsized":"2023-03-03T08:00:21.786139+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:53:15.089322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.780959+0000","last_change":"2023-03-03T08:00:21.780959+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.780959+0000","last_undegraded":"2023-03-03T08:00:21.780959+0000","last_fullsized":"2023-03-03T08:00:21.780959+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:43:23.319076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.781768+0000","last_change":"2023-03-03T08:00:21.778523+0000","last_active":"2023-03-03T08:00:21.781768+0000","last_peered":"2023-03-03T08:00:21.781768+0000","last_clean":"2023-03-03T08:00:21.781768+0000","last_became_active":"2023-03-03T08:00:21.778260+0000","last_became_peered":"2023-03-03T08:00:21.778260+0000","last_unstale":"2023-03-03T08:00:21.781768+0000","last_undegraded":"2023-03-03T08:00:21.781768+0000","last_fullsized":"2023-03-03T08:00:21.781768+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:26.856771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772265+0000","last_change":"2023-03-03T08:00:21.772265+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772265+0000","last_undegraded":"2023-03-03T08:00:21.772265+0000","last_fullsized":"2023-03-03T08:00:21.772265+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:09.676363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.790687+0000","last_change":"2023-03-03T08:00:21.787110+0000","last_active":"2023-03-03T08:00:21.790687+0000","last_peered":"2023-03-03T08:00:21.790687+0000","last_clean":"2023-03-03T08:00:21.790687+0000","last_became_active":"2023-03-03T08:00:21.786811+0000","last_became_peered":"2023-03-03T08:00:21.786811+0000","last_unstale":"2023-03-03T08:00:21.790687+0000","last_undegraded":"2023-03-03T08:00:21.790687+0000","last_fullsized":"2023-03-03T08:00:21.790687+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:57:43.225652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.783126+0000","last_change":"2023-03-03T08:00:21.783126+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.783126+0000","last_undegraded":"2023-03-03T08:00:21.783126+0000","last_fullsized":"2023-03-03T08:00:21.783126+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:35:58.114402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.787677+0000","last_change":"2023-03-03T08:00:21.783560+0000","last_active":"2023-03-03T08:00:21.787677+0000","last_peered":"2023-03-03T08:00:21.787677+0000","last_clean":"2023-03-03T08:00:21.787677+0000","last_became_active":"2023-03-03T08:00:21.783190+0000","last_became_peered":"2023-03-03T08:00:21.783190+0000","last_unstale":"2023-03-03T08:00:21.787677+0000","last_undegraded":"2023-03-03T08:00:21.787677+0000","last_fullsized":"2023-03-03T08:00:21.787677+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:39.306350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.777148+0000","last_change":"2023-03-03T08:00:21.777148+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.777148+0000","last_undegraded":"2023-03-03T08:00:21.777148+0000","last_fullsized":"2023-03-03T08:00:21.777148+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:52.425703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794928+0000","last_change":"2023-03-03T08:00:21.792376+0000","last_active":"2023-03-03T08:00:21.794928+0000","last_peered":"2023-03-03T08:00:21.794928+0000","last_clean":"2023-03-03T08:00:21.794928+0000","last_became_active":"2023-03-03T08:00:21.792126+0000","last_became_peered":"2023-03-03T08:00:21.792126+0000","last_unstale":"2023-03-03T08:00:21.794928+0000","last_undegraded":"2023-03-03T08:00:21.794928+0000","last_fullsized":"2023-03-03T08:00:21.794928+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:44.522714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.773311+0000","last_change":"2023-03-03T08:00:21.773311+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.773311+0000","last_undegraded":"2023-03-03T08:00:21.773311+0000","last_fullsized":"2023-03-03T08:00:21.773311+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:55:49.478050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.795011+0000","last_change":"2023-03-03T08:00:21.790545+0000","last_active":"2023-03-03T08:00:21.795011+0000","last_peered":"2023-03-03T08:00:21.795011+0000","last_clean":"2023-03-03T08:00:21.795011+0000","last_became_active":"2023-03-03T08:00:21.790052+0000","last_became_peered":"2023-03-03T08:00:21.790052+0000","last_unstale":"2023-03-03T08:00:21.795011+0000","last_undegraded":"2023-03-03T08:00:21.795011+0000","last_fullsized":"2023-03-03T08:00:21.795011+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:51:33.987978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.777009+0000","last_change":"2023-03-03T08:00:21.777009+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.777009+0000","last_undegraded":"2023-03-03T08:00:21.777009+0000","last_fullsized":"2023-03-03T08:00:21.777009+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:34.298493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.785663+0000","last_change":"2023-03-03T08:00:21.783602+0000","last_active":"2023-03-03T08:00:21.785663+0000","last_peered":"2023-03-03T08:00:21.785663+0000","last_clean":"2023-03-03T08:00:21.785663+0000","last_became_active":"2023-03-03T08:00:21.783297+0000","last_became_peered":"2023-03-03T08:00:21.783297+0000","last_unstale":"2023-03-03T08:00:21.785663+0000","last_undegraded":"2023-03-03T08:00:21.785663+0000","last_fullsized":"2023-03-03T08:00:21.785663+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:08:45.794437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.780801+0000","last_change":"2023-03-03T08:00:21.780801+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.780801+0000","last_undegraded":"2023-03-03T08:00:21.780801+0000","last_fullsized":"2023-03-03T08:00:21.780801+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:12:21.211704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.795043+0000","last_change":"2023-03-03T08:00:21.790207+0000","last_active":"2023-03-03T08:00:21.795043+0000","last_peered":"2023-03-03T08:00:21.795043+0000","last_clean":"2023-03-03T08:00:21.795043+0000","last_became_active":"2023-03-03T08:00:21.789535+0000","last_became_peered":"2023-03-03T08:00:21.789535+0000","last_unstale":"2023-03-03T08:00:21.795043+0000","last_undegraded":"2023-03-03T08:00:21.795043+0000","last_fullsized":"2023-03-03T08:00:21.795043+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:02.228725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.774137+0000","last_change":"2023-03-03T08:00:21.774137+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.774137+0000","last_undegraded":"2023-03-03T08:00:21.774137+0000","last_fullsized":"2023-03-03T08:00:21.774137+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:20.734366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.791765+0000","last_change":"2023-03-03T08:00:21.789108+0000","last_active":"2023-03-03T08:00:21.791765+0000","last_peered":"2023-03-03T08:00:21.791765+0000","last_clean":"2023-03-03T08:00:21.791765+0000","last_became_active":"2023-03-03T08:00:21.788850+0000","last_became_peered":"2023-03-03T08:00:21.788850+0000","last_unstale":"2023-03-03T08:00:21.791765+0000","last_undegraded":"2023-03-03T08:00:21.791765+0000","last_fullsized":"2023-03-03T08:00:21.791765+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:16:16.126986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772211+0000","last_change":"2023-03-03T08:00:21.772211+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772211+0000","last_undegraded":"2023-03-03T08:00:21.772211+0000","last_fullsized":"2023-03-03T08:00:21.772211+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:06:54.326996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794883+0000","last_change":"2023-03-03T08:00:21.788681+0000","last_active":"2023-03-03T08:00:21.794883+0000","last_peered":"2023-03-03T08:00:21.794883+0000","last_clean":"2023-03-03T08:00:21.794883+0000","last_became_active":"2023-03-03T08:00:21.788087+0000","last_became_peered":"2023-03-03T08:00:21.788087+0000","last_unstale":"2023-03-03T08:00:21.794883+0000","last_undegraded":"2023-03-03T08:00:21.794883+0000","last_fullsized":"2023-03-03T08:00:21.794883+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:08:45.794437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.779051+0000","last_change":"2023-03-03T08:00:21.779051+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.779051+0000","last_undegraded":"2023-03-03T08:00:21.779051+0000","last_fullsized":"2023-03-03T08:00:21.779051+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:20.734366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794772+0000","last_change":"2023-03-03T08:00:21.787442+0000","last_active":"2023-03-03T08:00:21.794772+0000","last_peered":"2023-03-03T08:00:21.794772+0000","last_clean":"2023-03-03T08:00:21.794772+0000","last_became_active":"2023-03-03T08:00:21.786782+0000","last_became_peered":"2023-03-03T08:00:21.786782+0000","last_unstale":"2023-03-03T08:00:21.794772+0000","last_undegraded":"2023-03-03T08:00:21.794772+0000","last_fullsized":"2023-03-03T08:00:21.794772+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:55:48.684553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.782556+0000","last_change":"2023-03-03T08:00:21.782556+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.782556+0000","last_undegraded":"2023-03-03T08:00:21.782556+0000","last_fullsized":"2023-03-03T08:00:21.782556+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:16:17.137666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.793146+0000","last_change":"2023-03-03T08:00:21.787586+0000","last_active":"2023-03-03T08:00:21.793146+0000","last_peered":"2023-03-03T08:00:21.793146+0000","last_clean":"2023-03-03T08:00:21.793146+0000","last_became_active":"2023-03-03T08:00:21.787272+0000","last_became_peered":"2023-03-03T08:00:21.787272+0000","last_unstale":"2023-03-03T08:00:21.793146+0000","last_undegraded":"2023-03-03T08:00:21.793146+0000","last_fullsized":"2023-03-03T08:00:21.793146+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:12.459278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772821+0000","last_change":"2023-03-03T08:00:21.772821+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772821+0000","last_undegraded":"2023-03-03T08:00:21.772821+0000","last_fullsized":"2023-03-03T08:00:21.772821+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:36:19.801740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794728+0000","last_change":"2023-03-03T08:00:21.791488+0000","last_active":"2023-03-03T08:00:21.794728+0000","last_peered":"2023-03-03T08:00:21.794728+0000","last_clean":"2023-03-03T08:00:21.794728+0000","last_became_active":"2023-03-03T08:00:21.791122+0000","last_became_peered":"2023-03-03T08:00:21.791122+0000","last_unstale":"2023-03-03T08:00:21.794728+0000","last_undegraded":"2023-03-03T08:00:21.794728+0000","last_fullsized":"2023-03-03T08:00:21.794728+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:36:55.429526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.776073+0000","last_change":"2023-03-03T08:00:21.776073+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.776073+0000","last_undegraded":"2023-03-03T08:00:21.776073+0000","last_fullsized":"2023-03-03T08:00:21.776073+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:52.425703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794987+0000","last_change":"2023-03-03T08:00:21.792101+0000","last_active":"2023-03-03T08:00:21.794987+0000","last_peered":"2023-03-03T08:00:21.794987+0000","last_clean":"2023-03-03T08:00:21.794987+0000","last_became_active":"2023-03-03T08:00:21.791481+0000","last_became_peered":"2023-03-03T08:00:21.791481+0000","last_unstale":"2023-03-03T08:00:21.794987+0000","last_undegraded":"2023-03-03T08:00:21.794987+0000","last_fullsized":"2023-03-03T08:00:21.794987+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:12:20.201024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.781035+0000","last_change":"2023-03-03T08:00:21.781035+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.781035+0000","last_undegraded":"2023-03-03T08:00:21.781035+0000","last_fullsized":"2023-03-03T08:00:21.781035+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:23:00.594155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794616+0000","last_change":"2023-03-03T08:00:21.786969+0000","last_active":"2023-03-03T08:00:21.794616+0000","last_peered":"2023-03-03T08:00:21.794616+0000","last_clean":"2023-03-03T08:00:21.794616+0000","last_became_active":"2023-03-03T08:00:21.786545+0000","last_became_peered":"2023-03-03T08:00:21.786545+0000","last_unstale":"2023-03-03T08:00:21.794616+0000","last_undegraded":"2023-03-03T08:00:21.794616+0000","last_fullsized":"2023-03-03T08:00:21.794616+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:16:25.213480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.775504+0000","last_change":"2023-03-03T08:00:21.775504+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.775504+0000","last_undegraded":"2023-03-03T08:00:21.775504+0000","last_fullsized":"2023-03-03T08:00:21.775504+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:11:16.603939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.790610+0000","last_change":"2023-03-03T08:00:21.788125+0000","last_active":"2023-03-03T08:00:21.790610+0000","last_peered":"2023-03-03T08:00:21.790610+0000","last_clean":"2023-03-03T08:00:21.790610+0000","last_became_active":"2023-03-03T08:00:21.787862+0000","last_became_peered":"2023-03-03T08:00:21.787862+0000","last_unstale":"2023-03-03T08:00:21.790610+0000","last_undegraded":"2023-03-03T08:00:21.790610+0000","last_fullsized":"2023-03-03T08:00:21.790610+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:03.212503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.780684+0000","last_change":"2023-03-03T08:00:21.780684+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.780684+0000","last_undegraded":"2023-03-03T08:00:21.780684+0000","last_fullsized":"2023-03-03T08:00:21.780684+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:49:26.845615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.791731+0000","last_change":"2023-03-03T08:00:21.789082+0000","last_active":"2023-03-03T08:00:21.791731+0000","last_peered":"2023-03-03T08:00:21.791731+0000","last_clean":"2023-03-03T08:00:21.791731+0000","last_became_active":"2023-03-03T08:00:21.788827+0000","last_became_peered":"2023-03-03T08:00:21.788827+0000","last_unstale":"2023-03-03T08:00:21.791731+0000","last_undegraded":"2023-03-03T08:00:21.791731+0000","last_fullsized":"2023-03-03T08:00:21.791731+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:58.454988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.778794+0000","last_change":"2023-03-03T08:00:21.778794+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.778794+0000","last_undegraded":"2023-03-03T08:00:21.778794+0000","last_fullsized":"2023-03-03T08:00:21.778794+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:53:38.302201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794946+0000","last_change":"2023-03-03T08:00:21.788909+0000","last_active":"2023-03-03T08:00:21.794946+0000","last_peered":"2023-03-03T08:00:21.794946+0000","last_clean":"2023-03-03T08:00:21.794946+0000","last_became_active":"2023-03-03T08:00:21.788395+0000","last_became_peered":"2023-03-03T08:00:21.788395+0000","last_unstale":"2023-03-03T08:00:21.794946+0000","last_undegraded":"2023-03-03T08:00:21.794946+0000","last_fullsized":"2023-03-03T08:00:21.794946+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:17:29.803501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772230+0000","last_change":"2023-03-03T08:00:21.772230+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772230+0000","last_undegraded":"2023-03-03T08:00:21.772230+0000","last_fullsized":"2023-03-03T08:00:21.772230+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:08:44.952506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.787003+0000","last_change":"2023-03-03T08:00:21.782978+0000","last_active":"2023-03-03T08:00:21.787003+0000","last_peered":"2023-03-03T08:00:21.787003+0000","last_clean":"2023-03-03T08:00:21.787003+0000","last_became_active":"2023-03-03T08:00:21.782679+0000","last_became_peered":"2023-03-03T08:00:21.782679+0000","last_unstale":"2023-03-03T08:00:21.787003+0000","last_undegraded":"2023-03-03T08:00:21.787003+0000","last_fullsized":"2023-03-03T08:00:21.787003+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:19:00.431727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.774168+0000","last_change":"2023-03-03T08:00:21.774168+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.774168+0000","last_undegraded":"2023-03-03T08:00:21.774168+0000","last_fullsized":"2023-03-03T08:00:21.774168+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:51:34.998659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.787659+0000","last_change":"2023-03-03T08:00:21.783463+0000","last_active":"2023-03-03T08:00:21.787659+0000","last_peered":"2023-03-03T08:00:21.787659+0000","last_clean":"2023-03-03T08:00:21.787659+0000","last_became_active":"2023-03-03T08:00:21.782809+0000","last_became_peered":"2023-03-03T08:00:21.782809+0000","last_unstale":"2023-03-03T08:00:21.787659+0000","last_undegraded":"2023-03-03T08:00:21.787659+0000","last_fullsized":"2023-03-03T08:00:21.787659+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:44:03.166814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.781502+0000","last_change":"2023-03-03T08:00:21.781502+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.781502+0000","last_undegraded":"2023-03-03T08:00:21.781502+0000","last_fullsized":"2023-03-03T08:00:21.781502+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:22:43.504725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.786556+0000","last_change":"2023-03-03T08:00:21.784441+0000","last_active":"2023-03-03T08:00:21.786556+0000","last_peered":"2023-03-03T08:00:21.786556+0000","last_clean":"2023-03-03T08:00:21.786556+0000","last_became_active":"2023-03-03T08:00:21.784193+0000","last_became_peered":"2023-03-03T08:00:21.784193+0000","last_unstale":"2023-03-03T08:00:21.786556+0000","last_undegraded":"2023-03-03T08:00:21.786556+0000","last_fullsized":"2023-03-03T08:00:21.786556+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:00:08.298430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772932+0000","last_change":"2023-03-03T08:00:21.772932+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772932+0000","last_undegraded":"2023-03-03T08:00:21.772932+0000","last_fullsized":"2023-03-03T08:00:21.772932+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:40.317030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.795248+0000","last_change":"2023-03-03T08:00:21.792370+0000","last_active":"2023-03-03T08:00:21.795248+0000","last_peered":"2023-03-03T08:00:21.795248+0000","last_clean":"2023-03-03T08:00:21.795248+0000","last_became_active":"2023-03-03T08:00:21.791836+0000","last_became_peered":"2023-03-03T08:00:21.791836+0000","last_unstale":"2023-03-03T08:00:21.795248+0000","last_undegraded":"2023-03-03T08:00:21.795248+0000","last_fullsized":"2023-03-03T08:00:21.795248+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:42:51.458743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":47,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.766482+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T08:00:20.766482+0000","last_peered":"2023-03-03T08:00:20.766482+0000","last_clean":"2023-03-03T08:00:20.766482+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T08:00:20.766482+0000","last_undegraded":"2023-03-03T08:00:20.766482+0000","last_fullsized":"2023-03-03T08:00:20.766482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.782097+0000","last_change":"2023-03-03T08:00:21.782097+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.782097+0000","last_undegraded":"2023-03-03T08:00:21.782097+0000","last_fullsized":"2023-03-03T08:00:21.782097+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:59.465669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.793733+0000","last_change":"2023-03-03T08:00:21.792272+0000","last_active":"2023-03-03T08:00:21.793733+0000","last_peered":"2023-03-03T08:00:21.793733+0000","last_clean":"2023-03-03T08:00:21.793733+0000","last_became_active":"2023-03-03T08:00:21.791961+0000","last_became_peered":"2023-03-03T08:00:21.791961+0000","last_unstale":"2023-03-03T08:00:21.793733+0000","last_undegraded":"2023-03-03T08:00:21.793733+0000","last_fullsized":"2023-03-03T08:00:21.793733+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:57:43.225652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":49,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.773696+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T08:00:20.773696+0000","last_peered":"2023-03-03T08:00:20.773696+0000","last_clean":"2023-03-03T08:00:20.773696+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T08:00:20.773696+0000","last_undegraded":"2023-03-03T08:00:20.773696+0000","last_fullsized":"2023-03-03T08:00:20.773696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.776650+0000","last_change":"2023-03-03T08:00:21.776650+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.776650+0000","last_undegraded":"2023-03-03T08:00:21.776650+0000","last_fullsized":"2023-03-03T08:00:21.776650+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:39.487708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.794791+0000","last_change":"2023-03-03T08:00:21.787607+0000","last_active":"2023-03-03T08:00:21.794791+0000","last_peered":"2023-03-03T08:00:21.794791+0000","last_clean":"2023-03-03T08:00:21.794791+0000","last_became_active":"2023-03-03T08:00:21.787106+0000","last_became_peered":"2023-03-03T08:00:21.787106+0000","last_unstale":"2023-03-03T08:00:21.794791+0000","last_undegraded":"2023-03-03T08:00:21.794791+0000","last_fullsized":"2023-03-03T08:00:21.794791+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:30:19.298647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":46,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.770256+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T08:00:20.770256+0000","last_peered":"2023-03-03T08:00:20.770256+0000","last_clean":"2023-03-03T08:00:20.770256+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T08:00:20.770256+0000","last_undegraded":"2023-03-03T08:00:20.770256+0000","last_fullsized":"2023-03-03T08:00:20.770256+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.779198+0000","last_change":"2023-03-03T08:00:21.779198+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.779198+0000","last_undegraded":"2023-03-03T08:00:21.779198+0000","last_fullsized":"2023-03-03T08:00:21.779198+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:00:09.309111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.779222+0000","last_change":"2023-03-03T08:00:21.775813+0000","last_active":"2023-03-03T08:00:21.779222+0000","last_peered":"2023-03-03T08:00:21.779222+0000","last_clean":"2023-03-03T08:00:21.779222+0000","last_became_active":"2023-03-03T08:00:21.775553+0000","last_became_peered":"2023-03-03T08:00:21.775553+0000","last_unstale":"2023-03-03T08:00:21.779222+0000","last_undegraded":"2023-03-03T08:00:21.779222+0000","last_fullsized":"2023-03-03T08:00:21.779222+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:19.710326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":49,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.773574+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T08:00:20.773574+0000","last_peered":"2023-03-03T08:00:20.773574+0000","last_clean":"2023-03-03T08:00:20.773574+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T08:00:20.773574+0000","last_undegraded":"2023-03-03T08:00:20.773574+0000","last_fullsized":"2023-03-03T08:00:20.773574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":203,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.761017+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T08:00:20.761017+0000","last_peered":"2023-03-03T08:00:20.761017+0000","last_clean":"2023-03-03T08:00:20.761017+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T08:00:20.761017+0000","last_undegraded":"2023-03-03T08:00:20.761017+0000","last_fullsized":"2023-03-03T08:00:20.761017+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772831+0000","last_change":"2023-03-03T08:00:21.772831+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772831+0000","last_undegraded":"2023-03-03T08:00:21.772831+0000","last_fullsized":"2023-03-03T08:00:21.772831+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:27.364263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.792517+0000","last_change":"2023-03-03T08:00:21.790111+0000","last_active":"2023-03-03T08:00:21.792517+0000","last_peered":"2023-03-03T08:00:21.792517+0000","last_clean":"2023-03-03T08:00:21.792517+0000","last_became_active":"2023-03-03T08:00:21.789831+0000","last_became_peered":"2023-03-03T08:00:21.789831+0000","last_unstale":"2023-03-03T08:00:21.792517+0000","last_undegraded":"2023-03-03T08:00:21.792517+0000","last_fullsized":"2023-03-03T08:00:21.792517+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:19.710326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":50,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.761012+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T08:00:20.761012+0000","last_peered":"2023-03-03T08:00:20.761012+0000","last_clean":"2023-03-03T08:00:20.761012+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T08:00:20.761012+0000","last_undegraded":"2023-03-03T08:00:20.761012+0000","last_fullsized":"2023-03-03T08:00:20.761012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.781191+0000","last_change":"2023-03-03T08:00:21.781191+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.781191+0000","last_undegraded":"2023-03-03T08:00:21.781191+0000","last_fullsized":"2023-03-03T08:00:21.781191+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:30:20.309327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.792980+0000","last_change":"2023-03-03T08:00:21.787378+0000","last_active":"2023-03-03T08:00:21.792980+0000","last_peered":"2023-03-03T08:00:21.792980+0000","last_clean":"2023-03-03T08:00:21.792980+0000","last_became_active":"2023-03-03T08:00:21.787102+0000","last_became_peered":"2023-03-03T08:00:21.787102+0000","last_unstale":"2023-03-03T08:00:21.792980+0000","last_undegraded":"2023-03-03T08:00:21.792980+0000","last_fullsized":"2023-03-03T08:00:21.792980+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:45:29.829197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":50,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.761012+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T08:00:20.761012+0000","last_peered":"2023-03-03T08:00:20.761012+0000","last_clean":"2023-03-03T08:00:20.761012+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T08:00:20.761012+0000","last_undegraded":"2023-03-03T08:00:20.761012+0000","last_fullsized":"2023-03-03T08:00:20.761012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.772836+0000","last_change":"2023-03-03T08:00:21.772836+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.772836+0000","last_undegraded":"2023-03-03T08:00:21.772836+0000","last_fullsized":"2023-03-03T08:00:21.772836+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:20.721006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.795022+0000","last_change":"2023-03-03T08:00:21.789735+0000","last_active":"2023-03-03T08:00:21.795022+0000","last_peered":"2023-03-03T08:00:21.795022+0000","last_clean":"2023-03-03T08:00:21.795022+0000","last_became_active":"2023-03-03T08:00:21.789201+0000","last_became_peered":"2023-03-03T08:00:21.789201+0000","last_unstale":"2023-03-03T08:00:21.795022+0000","last_undegraded":"2023-03-03T08:00:21.795022+0000","last_fullsized":"2023-03-03T08:00:21.795022+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:08.665683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":47,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.765221+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T08:00:20.765221+0000","last_peered":"2023-03-03T08:00:20.765221+0000","last_clean":"2023-03-03T08:00:20.765221+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T08:00:20.765221+0000","last_undegraded":"2023-03-03T08:00:20.765221+0000","last_fullsized":"2023-03-03T08:00:20.765221+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2023-03-03T08:00:21.780220+0000","last_change":"2023-03-03T08:00:21.780220+0000","last_active":"2023-03-03T08:00:21.758670+0000","last_peered":"2023-03-03T08:00:21.758670+0000","last_clean":"2023-03-03T08:00:21.758670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T08:00:21.780220+0000","last_undegraded":"2023-03-03T08:00:21.780220+0000","last_fullsized":"2023-03-03T08:00:21.780220+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:17:07.748684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-03-03T08:00:21.781897+0000","last_change":"2023-03-03T08:00:21.778956+0000","last_active":"2023-03-03T08:00:21.781897+0000","last_peered":"2023-03-03T08:00:21.781897+0000","last_clean":"2023-03-03T08:00:21.781897+0000","last_became_active":"2023-03-03T08:00:21.778666+0000","last_became_peered":"2023-03-03T08:00:21.778666+0000","last_unstale":"2023-03-03T08:00:21.781897+0000","last_undegraded":"2023-03-03T08:00:21.781897+0000","last_fullsized":"2023-03-03T08:00:21.781897+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:05.518653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-03-03T08:00:20.761734+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T08:00:20.761734+0000","last_peered":"2023-03-03T08:00:20.761734+0000","last_clean":"2023-03-03T08:00:20.761734+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T08:00:20.761734+0000","last_undegraded":"2023-03-03T08:00:20.761734+0000","last_fullsized":"2023-03-03T08:00:20.761734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574909,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10748,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":9408,"kb_avail":94361092,"statfs":{"total":96636764160,"available":96625758208,"internally_reserved":0,"allocated":1363968,"data_stored":928698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9633792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.67500000000000004,"15min":0.70499999999999996},"min":{"1min":0.437,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82499999999999996,"5min":1.546,"15min":1.546},"last":0.376},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71199999999999997,"15min":0.72999999999999998},"min":{"1min":0.50600000000000001,"5min":0.5,"15min":0.5},"max":{"1min":0.96499999999999997,"5min":1.702,"15min":1.702},"last":0.751}]},{"osd":1,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.89000000000000001,"15min":1.0840000000000001},"min":{"1min":0.49199999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.111,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.434},{"interface":"front","average":{"1min":0.755,"5min":0.74099999999999999,"15min":0.76900000000000002},"min":{"1min":0.51600000000000001,"5min":0.504,"15min":0.504},"max":{"1min":1.1890000000000001,"5min":1.944,"15min":1.944},"last":0.70299999999999996}]},{"osd":2,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.71899999999999997,"15min":0.71399999999999997},"min":{"1min":0.63300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.224,"5min":1.252,"15min":1.252},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76800000000000002,"15min":0.79300000000000004},"min":{"1min":0.54600000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2490000000000001,"5min":1.329,"15min":1.329},"last":0.55800000000000005}]},{"osd":3,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.71999999999999997,"15min":0.71499999999999997},"min":{"1min":0.59799999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.754,"15min":0.79000000000000004},"min":{"1min":0.55000000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.93400000000000005,"5min":1.294,"15min":1.294},"last":0.51000000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.48399999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0049999999999999,"5min":1.077,"15min":1.077},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.66700000000000004,"15min":0.63400000000000001},"min":{"1min":0.51700000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.86899999999999999}]},{"osd":5,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.77100000000000002,"15min":0.80100000000000005},"min":{"1min":0.41699999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.036,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81699999999999995,"15min":0.82399999999999995},"min":{"1min":0.66400000000000003,"5min":0.496,"15min":0.496},"max":{"1min":1.2190000000000001,"5min":1.349,"15min":1.349},"last":0.80300000000000005}]},{"osd":6,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.64100000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0700000000000001,"5min":1.204,"15min":1.204},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.81100000000000005,"15min":0.82099999999999995},"min":{"1min":0.627,"5min":0.499,"15min":0.499},"max":{"1min":1.135,"5min":1.194,"15min":1.194},"last":0.95499999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574910,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9268,"kb_used_data":748,"kb_used_omap":0,"kb_used_meta":8512,"kb_avail":94362572,"statfs":{"total":96636764160,"available":96627273728,"internally_reserved":0,"allocated":765952,"data_stored":337311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8716288},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.70799999999999996,"15min":0.68899999999999995},"min":{"1min":0.57199999999999995,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.018,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68999999999999995,"15min":0.69499999999999995},"min":{"1min":0.52500000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.89700000000000002,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":1.0549999999999999}]},{"osd":1,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70999999999999996,"15min":0.68600000000000005},"min":{"1min":0.52700000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.97099999999999997,"5min":3.323,"15min":3.323},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.73499999999999999,"15min":0.73099999999999998},"min":{"1min":0.55400000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97199999999999998,"5min":1.135,"15min":1.135},"last":0.504}]},{"osd":2,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.753,"15min":0.73199999999999998},"min":{"1min":0.59499999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.98899999999999999,"5min":1.54,"15min":1.54},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.749,"15min":0.75900000000000001},"min":{"1min":0.54400000000000004,"5min":0.495,"15min":0.495},"max":{"1min":1.0089999999999999,"5min":1.698,"15min":1.698},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81299999999999994,"15min":0.78900000000000003},"min":{"1min":0.57499999999999996,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.3089999999999999,"5min":4.8959999999999999,"15min":4.8959999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77200000000000002,"15min":0.77100000000000002},"min":{"1min":0.60899999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.95799999999999996,"5min":1.343,"15min":1.343},"last":1.0800000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.79900000000000004,"15min":0.91100000000000003},"min":{"1min":0.439,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.871,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.80700000000000005,"15min":0.92400000000000004},"min":{"1min":0.53000000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.85099999999999998,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":1.012}]},{"osd":5,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.72599999999999998,"15min":0.77600000000000002},"min":{"1min":0.47599999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.95399999999999996,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.80300000000000005,"15min":0.82499999999999996},"min":{"1min":0.499,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91500000000000004,"5min":1.429,"15min":1.429},"last":1.202}]},{"osd":7,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.81899999999999995,"15min":0.83899999999999997},"min":{"1min":0.56699999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.0029999999999999,"5min":1.452,"15min":1.452},"last":1.107},{"interface":"front","average":{"1min":0.753,"5min":0.82099999999999995,"15min":0.87},"min":{"1min":0.59399999999999997,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.008,"5min":1.579,"15min":1.579},"last":0.76600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9324,"kb_used_data":740,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362516,"statfs":{"total":96636764160,"available":96627216384,"internally_reserved":0,"allocated":757760,"data_stored":331710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.49099999999999999,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.63100000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58099999999999996,"15min":0.64700000000000002},"min":{"1min":0.36799999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.90200000000000002,"5min":3.972,"15min":3.972},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.74099999999999999,"15min":0.96399999999999997},"min":{"1min":0.371,"5min":0.313,"15min":0.313},"max":{"1min":0.78900000000000003,"5min":11.897,"15min":11.897},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.82299999999999995,"15min":1.0469999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.85199999999999998,"5min":11.804,"15min":11.804},"last":0.73699999999999999}]},{"osd":3,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.81399999999999995,"15min":1.036},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.89500000000000002,"5min":11.843999999999999,"15min":11.843999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64300000000000002,"15min":0.73499999999999999},"min":{"1min":0.47899999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.96399999999999997,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.54500000000000004}]},{"osd":4,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81799999999999995,"15min":0.86299999999999999},"min":{"1min":0.55000000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0960000000000001,"5min":1.613,"15min":1.613},"last":0.745},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.86099999999999999},"min":{"1min":0.65000000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.0800000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.76200000000000001}]},{"osd":5,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.84699999999999998,"15min":0.92500000000000004},"min":{"1min":0.63900000000000001,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.997,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81999999999999995,"15min":0.873},"min":{"1min":0.57899999999999996,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98899999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.66100000000000003}]},{"osd":6,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.91300000000000003,"15min":0.999},"min":{"1min":0.68400000000000005,"5min":0.628,"15min":0.628},"max":{"1min":1.173,"5min":3.907,"15min":3.907},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.875,"5min":0.88400000000000001,"15min":0.93999999999999995},"min":{"1min":0.67300000000000004,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1819999999999999,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.70999999999999996}]},{"osd":7,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84499999999999997,"15min":0.88300000000000001},"min":{"1min":0.70399999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.026,"5min":1.419,"15min":1.419},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.871,"5min":0.90600000000000003,"15min":1.012},"min":{"1min":0.70399999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1659999999999999,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":0.67700000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574909,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10476,"kb_used_data":1316,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94361364,"statfs":{"total":96636764160,"available":96626036736,"internally_reserved":0,"allocated":1347584,"data_stored":921059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45800000000000002,"15min":0.47199999999999998},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65000000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.38},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.50600000000000001,"15min":0.51700000000000002},"min":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81999999999999995,"5min":0.871,"15min":0.871},"last":0.54300000000000004}]},{"osd":2,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53000000000000003,"15min":0.55000000000000004},"min":{"1min":0.38400000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77400000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.44,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73399999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.503}]},{"osd":3,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70399999999999996,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.46600000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.80800000000000005,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.53100000000000003}]},{"osd":4,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82299999999999995,"15min":0.84499999999999997},"min":{"1min":0.64700000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79800000000000004,"15min":0.81999999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0740000000000001,"5min":1.605,"15min":1.605},"last":0.96599999999999997}]},{"osd":5,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.80400000000000005,"15min":0.82199999999999995},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0980000000000001,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.871},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.82099999999999995,"15min":0.82499999999999996},"min":{"1min":0.61299999999999999,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.2070000000000001,"5min":1.286,"15min":1.286},"last":0.90400000000000003}]},{"osd":6,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.85399999999999998,"15min":0.88500000000000001},"min":{"1min":0.65300000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2470000000000001,"5min":1.623,"15min":1.623},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.86399999999999999},"min":{"1min":0.66300000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.137,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":1.0029999999999999}]},{"osd":7,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84499999999999997,"15min":0.88300000000000001},"min":{"1min":0.59399999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.083,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.83999999999999997,"15min":0.86199999999999999},"min":{"1min":0.58199999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.1719999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.93500000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574910,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9384,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":8640,"kb_avail":94362456,"statfs":{"total":96636764160,"available":96627154944,"internally_reserved":0,"allocated":753664,"data_stored":330691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8847360},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.64800000000000002,"15min":0.87},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.61699999999999999,"5min":5.7889999999999997,"15min":5.7889999999999997},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.60899999999999999,"15min":0.73199999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67400000000000004,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":0.40600000000000003}]},{"osd":1,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.629,"15min":0.753},"min":{"1min":0.36299999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77700000000000002,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.38},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.79300000000000004,"15min":1.0089999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.77100000000000002,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.60799999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.88900000000000001,"15min":1.222},"min":{"1min":0.41399999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.69799999999999995,"5min":12.961,"15min":12.961},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.80000000000000004,"15min":1.036},"min":{"1min":0.42699999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.74099999999999999,"5min":6.1449999999999996,"15min":6.1449999999999996},"last":0.56299999999999994}]},{"osd":4,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.0469999999999999,"15min":1.2769999999999999},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1639999999999999,"5min":6.484,"15min":6.484},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.0580000000000001,"15min":1.2549999999999999},"min":{"1min":0.63600000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.1759999999999999,"5min":6.5510000000000002,"15min":6.5510000000000002},"last":0.67400000000000004}]},{"osd":5,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.97399999999999998,"15min":1.1799999999999999},"min":{"1min":0.61499999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.034,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.84699999999999998,"5min":1.0269999999999999,"15min":1.238},"min":{"1min":0.60999999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1499999999999999,"5min":6.6369999999999996,"15min":6.6369999999999996},"last":0.63600000000000001}]},{"osd":6,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.0429999999999999,"15min":1.24},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.1240000000000001,"5min":6.625,"15min":6.625},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.88200000000000001,"5min":1.0549999999999999,"15min":1.2669999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.099,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.69899999999999995}]},{"osd":7,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":1.0129999999999999,"15min":1.2310000000000001},"min":{"1min":0.69899999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.087,"5min":6.7009999999999996,"15min":6.7009999999999996},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.0209999999999999,"15min":1.214},"min":{"1min":0.66500000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.081,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.621}]}]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9772,"kb_used_data":740,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94362068,"statfs":{"total":96636764160,"available":96626757632,"internally_reserved":0,"allocated":757760,"data_stored":331710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.63,"15min":0.83499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.56599999999999995,"5min":9.5820000000000007,"15min":9.5820000000000007},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.70099999999999996,"15min":0.90600000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83599999999999997,"5min":9.7840000000000007,"15min":9.7840000000000007},"last":0.42999999999999999}]},{"osd":1,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.501,"15min":0.52300000000000002},"min":{"1min":0.34999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78900000000000003,"5min":1,"15min":1},"last":0.371},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.75600000000000001,"15min":0.98999999999999999},"min":{"1min":0.41099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70499999999999996,"5min":9.702,"15min":9.702},"last":0.55600000000000005}]},{"osd":2,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.746,"15min":0.93600000000000005},"min":{"1min":0.436,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88300000000000001,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.80500000000000005,"15min":1.0229999999999999},"min":{"1min":0.498,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.86499999999999999,"5min":9.734,"15min":9.734},"last":0.65300000000000002}]},{"osd":4,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.96099999999999997,"15min":1.165},"min":{"1min":0.54400000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1819999999999999,"5min":9.8629999999999995,"15min":9.8629999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.83999999999999997,"15min":0.84799999999999998},"min":{"1min":0.629,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.097,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":0.69099999999999995}]},{"osd":5,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82099999999999995,"15min":0.83299999999999996},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.1899999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.999,"15min":1.222},"min":{"1min":0.628,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0860000000000001,"5min":9.6600000000000001,"15min":9.6600000000000001},"last":0.72399999999999998}]},{"osd":6,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.0129999999999999,"15min":1.2370000000000001},"min":{"1min":0.65700000000000003,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0600000000000001,"5min":9.8190000000000008,"15min":9.8190000000000008},"last":0.629},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82799999999999996,"15min":0.86699999999999999},"min":{"1min":0.629,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.2609999999999999,"5min":1.609,"15min":1.609},"last":0.73999999999999999}]},{"osd":7,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":1.0369999999999999,"15min":1.2749999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.204,"5min":9.625,"15min":9.625},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":1.034,"15min":1.244},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.119,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":0.70899999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9452,"kb_used_data":740,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362388,"statfs":{"total":96636764160,"available":96627085312,"internally_reserved":0,"allocated":757760,"data_stored":331710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.85199999999999998,"15min":0.98299999999999998},"min":{"1min":0.433,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0049999999999999,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.68400000000000005,"15min":0.66600000000000004},"min":{"1min":0.56100000000000005,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.95399999999999996,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.63700000000000001}]},{"osd":1,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.92700000000000005,"15min":1.1080000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94199999999999995,"5min":9.5709999999999997,"15min":9.5709999999999997},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.83999999999999997,"15min":0.92700000000000005},"min":{"1min":0.58599999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0349999999999999,"5min":5.266,"15min":5.266},"last":0.72299999999999998}]},{"osd":2,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.91600000000000004,"15min":1.077},"min":{"1min":0.60299999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.016,"5min":8.4529999999999994,"15min":8.4529999999999994},"last":0.749},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.88,"15min":1.002},"min":{"1min":0.53200000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.018,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.66000000000000003}]},{"osd":3,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.72899999999999998,"15min":0.73299999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.79000000000000004,"15min":0.80200000000000005},"min":{"1min":0.58399999999999996,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.98499999999999999,"5min":1.389,"15min":1.389},"last":0.80500000000000005}]},{"osd":5,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.71799999999999997,"15min":0.74199999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86799999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.72299999999999998,"15min":0.72099999999999997},"min":{"1min":0.47599999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.998,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.70299999999999996}]},{"osd":6,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.77300000000000002,"15min":0.78100000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.94599999999999995,"5min":1.032,"15min":1.032},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.501,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0700000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.871}]},{"osd":7,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.79700000000000004,"15min":0.82199999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0609999999999999,"5min":1.679,"15min":1.679},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.88,"15min":0.92800000000000005},"min":{"1min":0.64300000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0069999999999999,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.85199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574909,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9460,"kb_used_data":748,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362380,"statfs":{"total":96636764160,"available":96627077120,"internally_reserved":0,"allocated":765952,"data_stored":337311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.65700000000000003,"15min":0.64100000000000001},"min":{"1min":0.42899999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.435},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.75700000000000001,"15min":0.74399999999999999},"min":{"1min":0.60599999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96999999999999997,"5min":2.528,"15min":2.528},"last":1.038}]},{"osd":1,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.73699999999999999,"15min":0.745},"min":{"1min":0.66400000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.67200000000000004,"15min":0.63700000000000001},"min":{"1min":0.53900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.81899999999999995}]},{"osd":2,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69199999999999995,"15min":0.68300000000000005},"min":{"1min":0.52700000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.95399999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.72499999999999998,"15min":0.71599999999999997},"min":{"1min":0.59299999999999997,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.89200000000000002,"5min":1.321,"15min":1.321},"last":0.74299999999999999}]},{"osd":3,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.71599999999999997,"15min":0.69499999999999995},"min":{"1min":0.55600000000000005,"5min":0.496,"15min":0.496},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.873},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.745,"15min":0.73499999999999999},"min":{"1min":0.629,"5min":0.501,"15min":0.501},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.82899999999999996}]},{"osd":4,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.51400000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.79000000000000004,"15min":0.78200000000000003},"min":{"1min":0.59899999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":1.079}]},{"osd":6,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.79700000000000004},"min":{"1min":0.53000000000000003,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.94499999999999995,"5min":1.071,"15min":1.071},"last":1.012},{"interface":"front","average":{"1min":0.754,"5min":0.72199999999999998,"15min":0.72399999999999998},"min":{"1min":0.58999999999999997,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97099999999999997,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.65400000000000003}]},{"osd":7,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.75600000000000001,"15min":0.75700000000000001},"min":{"1min":0.55700000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.026,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81599999999999995,"15min":0.83299999999999996},"min":{"1min":0.67700000000000005,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.90400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:00:27.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:00:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.172+0000 7f44cf59e700 1 Processor -- start 2023-03-03T08:00:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.173+0000 7f44cf59e700 1 -- start start 2023-03-03T08:00:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.174+0000 7f44cf59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f44d00d1650 0x7f44d00d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.174+0000 7f44cf59e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f44d00b4a00 0x7f44d00b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.174+0000 7f44cf59e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d00b0c50 con 0x7f44d00d0010 2023-03-03T08:00:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.174+0000 7f44cf59e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d00b0040 con 0x7f44d00d1650 2023-03-03T08:00:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.174+0000 7f44cf59e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d00b0320 con 0x7f44d00b4a00 2023-03-03T08:00:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.174+0000 7f44cdd9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f44d00d1650 0x7f44d00d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.174+0000 7f44ce59c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f44d00d0010 0x7f44d00d0f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44874/0 (socket says 172.21.15.98:44874) 2023-03-03T08:00:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.174+0000 7f44ce59c700 1 -- 172.21.15.98:0/2463202588 learned_addr learned my addr 172.21.15.98:0/2463202588 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:28.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.175+0000 7f44ced9d700 1 --2- 172.21.15.98:0/2463202588 >> v2:172.21.15.100:3300/0 conn(0x7f44d00b4a00 0x7f44d00b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:28.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.175+0000 7f44cdd9b700 1 -- 172.21.15.98:0/2463202588 >> v2:172.21.15.100:3300/0 conn(0x7f44d00b4a00 msgr2=0x7f44d00b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:28.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.175+0000 7f44cdd9b700 1 --2- 172.21.15.98:0/2463202588 >> v2:172.21.15.100:3300/0 conn(0x7f44d00b4a00 0x7f44d00b2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.175+0000 7f44cdd9b700 1 -- 172.21.15.98:0/2463202588 >> v1:172.21.15.98:6789/0 conn(0x7f44d00d0010 legacy=0x7f44d00d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:28.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.175+0000 7f44cdd9b700 1 -- 172.21.15.98:0/2463202588 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d00b0600 con 0x7f44d00d1650 2023-03-03T08:00:28.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.175+0000 7f44cd59a700 1 -- 172.21.15.98:0/2463202588 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433362363 0 0) 0x7f44d00b0c50 con 0x7f44d00d0010 2023-03-03T08:00:28.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.175+0000 7f44cdd9b700 1 --2- 172.21.15.98:0/2463202588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d00d1650 0x7f44d00d03e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f44d00c6e40 tx=0x7f44c4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99ef43afef09c3d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.176+0000 7f44cd59a700 1 -- 172.21.15.98:0/2463202588 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c4011830 con 0x7f44d00d1650 2023-03-03T08:00:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.176+0000 7f44cd59a700 1 -- 172.21.15.98:0/2463202588 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c4011990 con 0x7f44d00d1650 2023-03-03T08:00:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.176+0000 7f44cd59a700 1 -- 172.21.15.98:0/2463202588 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c4011d20 con 0x7f44d00d1650 2023-03-03T08:00:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.176+0000 7f44cf59e700 1 -- 172.21.15.98:0/2463202588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d00d1650 msgr2=0x7f44d00d03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.176+0000 7f44cf59e700 1 --2- 172.21.15.98:0/2463202588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d00d1650 0x7f44d00d03e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f44d00c6e40 tx=0x7f44c4009b20 comp rx=0 tx=0).stop 2023-03-03T08:00:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.176+0000 7f44cf59e700 1 -- 172.21.15.98:0/2463202588 shutdown_connections 2023-03-03T08:00:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.176+0000 7f44cf59e700 1 --2- 172.21.15.98:0/2463202588 >> v2:172.21.15.100:3300/0 conn(0x7f44d00b4a00 0x7f44d00b2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.176+0000 7f44cf59e700 1 --2- 172.21.15.98:0/2463202588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d00d1650 0x7f44d00d03e0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.176+0000 7f44cf59e700 1 -- 172.21.15.98:0/2463202588 >> 172.21.15.98:0/2463202588 conn(0x7f44d0073b00 msgr2=0x7f44d00d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.176+0000 7f44cf59e700 1 -- 172.21.15.98:0/2463202588 shutdown_connections 2023-03-03T08:00:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.176+0000 7f44cf59e700 1 -- 172.21.15.98:0/2463202588 wait complete. 2023-03-03T08:00:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.179+0000 7f44cf59e700 1 Processor -- start 2023-03-03T08:00:28.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.179+0000 7f44cf59e700 1 -- start start 2023-03-03T08:00:28.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.180+0000 7f44cf59e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44d00b4a00 0x7f44d015aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:28.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.180+0000 7f44cf59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d00d0010 0x7f44d01e4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:28.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.180+0000 7f44cf59e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44d00d1650 0x7f44d00b6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:28.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.180+0000 7f44cf59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d00b9270 con 0x7f44d00d0010 2023-03-03T08:00:28.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.180+0000 7f44cf59e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d01e4e70 con 0x7f44d00b4a00 2023-03-03T08:00:28.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.180+0000 7f44cf59e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d01e6140 con 0x7f44d00d1650 2023-03-03T08:00:28.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.180+0000 7f44cdd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d00d0010 0x7f44d01e4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:28.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.180+0000 7f44cdd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d00d0010 0x7f44d01e4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44984/0 (socket says 172.21.15.98:44984) 2023-03-03T08:00:28.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.180+0000 7f44cdd9b700 1 -- 172.21.15.98:0/1773222652 learned_addr learned my addr 172.21.15.98:0/1773222652 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:28.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.181+0000 7f44ced9d700 1 --2- 172.21.15.98:0/1773222652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44d00d1650 0x7f44d00b6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:28.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.181+0000 7f44ce59c700 1 --2- 172.21.15.98:0/1773222652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44d00b4a00 0x7f44d015aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:28.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.181+0000 7f44cdd9b700 1 -- 172.21.15.98:0/1773222652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44d00d1650 msgr2=0x7f44d00b6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:28.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.181+0000 7f44cdd9b700 1 --2- 172.21.15.98:0/1773222652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44d00d1650 0x7f44d00b6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.181+0000 7f44cdd9b700 1 -- 172.21.15.98:0/1773222652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44d00b4a00 msgr2=0x7f44d015aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:28.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.181+0000 7f44cdd9b700 1 --2- 172.21.15.98:0/1773222652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44d00b4a00 0x7f44d015aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.181+0000 7f44cdd9b700 1 -- 172.21.15.98:0/1773222652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c40094a0 con 0x7f44d00d0010 2023-03-03T08:00:28.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.181+0000 7f44cdd9b700 1 --2- 172.21.15.98:0/1773222652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d00d0010 0x7f44d01e4930 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f44c4014040 tx=0x7f44c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:28.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.181+0000 7f44beffd700 1 -- 172.21.15.98:0/1773222652 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c40038a0 con 0x7f44d00d0010 2023-03-03T08:00:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.181+0000 7f44beffd700 1 -- 172.21.15.98:0/1773222652 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c401e070 con 0x7f44d00d0010 2023-03-03T08:00:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.181+0000 7f44cf59e700 1 -- 172.21.15.98:0/1773222652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d00d0b30 con 0x7f44d00d0010 2023-03-03T08:00:28.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.182+0000 7f44beffd700 1 -- 172.21.15.98:0/1773222652 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c4003c20 con 0x7f44d00d0010 2023-03-03T08:00:28.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.182+0000 7f44cf59e700 1 -- 172.21.15.98:0/1773222652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d00b08e0 con 0x7f44d00d0010 2023-03-03T08:00:28.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.183+0000 7f44cf59e700 1 -- 172.21.15.98:0/1773222652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d00d0cc0 con 0x7f44d00d0010 2023-03-03T08:00:28.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.184+0000 7f44beffd700 1 -- 172.21.15.98:0/1773222652 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f44c400f050 con 0x7f44d00d0010 2023-03-03T08:00:28.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.185+0000 7f44beffd700 1 --2- 172.21.15.98:0/1773222652 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f44b00683c0 0x7f44b006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:28.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.185+0000 7f44beffd700 1 -- 172.21.15.98:0/1773222652 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f44c4086f10 con 0x7f44d00d0010 2023-03-03T08:00:28.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.186+0000 7f44ce59c700 1 --2- 172.21.15.98:0/1773222652 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f44b00683c0 0x7f44b006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:28.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.186+0000 7f44ce59c700 1 --2- 172.21.15.98:0/1773222652 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f44b00683c0 0x7f44b006a870 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f44c0007fd0 tx=0x7f44c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:28.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.189+0000 7f44beffd700 1 -- 172.21.15.98:0/1773222652 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f44c40535b0 con 0x7f44d00d0010 2023-03-03T08:00:28.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.380+0000 7f44cf59e700 1 -- 172.21.15.98:0/1773222652 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f44d00729b0 con 0x7f44b00683c0 2023-03-03T08:00:28.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.389+0000 7f44beffd700 1 -- 172.21.15.98:0/1773222652 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310729 (secure 0 0 0) 0x7f44d00729b0 con 0x7f44b00683c0 2023-03-03T08:00:28.394 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:28.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.394+0000 7f44cf59e700 1 -- 172.21.15.98:0/1773222652 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f44b00683c0 msgr2=0x7f44b006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:28.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.394+0000 7f44cf59e700 1 --2- 172.21.15.98:0/1773222652 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f44b00683c0 0x7f44b006a870 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f44c0007fd0 tx=0x7f44c0009040 comp rx=0 tx=0).stop 2023-03-03T08:00:28.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.394+0000 7f44cf59e700 1 -- 172.21.15.98:0/1773222652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d00d0010 msgr2=0x7f44d01e4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:28.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.394+0000 7f44cf59e700 1 --2- 172.21.15.98:0/1773222652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d00d0010 0x7f44d01e4930 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f44c4014040 tx=0x7f44c4004750 comp rx=0 tx=0).stop 2023-03-03T08:00:28.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.394+0000 7f44cf59e700 1 -- 172.21.15.98:0/1773222652 shutdown_connections 2023-03-03T08:00:28.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.394+0000 7f44cf59e700 1 --2- 172.21.15.98:0/1773222652 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f44b00683c0 0x7f44b006a870 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.394+0000 7f44cf59e700 1 --2- 172.21.15.98:0/1773222652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44d00d1650 0x7f44d00b6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.394+0000 7f44cf59e700 1 --2- 172.21.15.98:0/1773222652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44d00b4a00 0x7f44d015aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.394+0000 7f44cf59e700 1 --2- 172.21.15.98:0/1773222652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d00d0010 0x7f44d01e4930 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.394+0000 7f44cf59e700 1 -- 172.21.15.98:0/1773222652 >> 172.21.15.98:0/1773222652 conn(0x7f44d0073b00 msgr2=0x7f44d006b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:28.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.394+0000 7f44cf59e700 1 -- 172.21.15.98:0/1773222652 shutdown_connections 2023-03-03T08:00:28.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.394+0000 7f44cf59e700 1 -- 172.21.15.98:0/1773222652 wait complete. 2023-03-03T08:00:28.398 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T08:00:28.424 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2023-03-03T08:00:28.241842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":79352,"kb_used_data":7416,"kb_used_omap":0,"kb_used_meta":71872,"kb_avail":754895368,"statfs":{"total":773094113280,"available":773012856832,"internally_reserved":0,"allocated":7593984,"data_stored":3998208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":73596928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.474170"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356978+0000","last_change":"2023-03-03T08:00:22.351816+0000","last_active":"2023-03-03T08:00:22.356978+0000","last_peered":"2023-03-03T08:00:22.356978+0000","last_clean":"2023-03-03T08:00:22.356978+0000","last_became_active":"2023-03-03T08:00:22.351560+0000","last_became_peered":"2023-03-03T08:00:22.351560+0000","last_unstale":"2023-03-03T08:00:22.356978+0000","last_undegraded":"2023-03-03T08:00:22.356978+0000","last_fullsized":"2023-03-03T08:00:22.356978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:22:36.090135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.355977+0000","last_change":"2023-03-03T08:00:22.349135+0000","last_active":"2023-03-03T08:00:22.355977+0000","last_peered":"2023-03-03T08:00:22.355977+0000","last_clean":"2023-03-03T08:00:22.355977+0000","last_became_active":"2023-03-03T08:00:22.348832+0000","last_became_peered":"2023-03-03T08:00:22.348832+0000","last_unstale":"2023-03-03T08:00:22.355977+0000","last_undegraded":"2023-03-03T08:00:22.355977+0000","last_fullsized":"2023-03-03T08:00:22.355977+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:45:30.839878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.343799+0000","last_change":"2023-03-03T08:00:22.341359+0000","last_active":"2023-03-03T08:00:22.343799+0000","last_peered":"2023-03-03T08:00:22.343799+0000","last_clean":"2023-03-03T08:00:22.343799+0000","last_became_active":"2023-03-03T08:00:22.341054+0000","last_became_peered":"2023-03-03T08:00:22.341054+0000","last_unstale":"2023-03-03T08:00:22.343799+0000","last_undegraded":"2023-03-03T08:00:22.343799+0000","last_fullsized":"2023-03-03T08:00:22.343799+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:57:44.236333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368309+0000","last_change":"2023-03-03T08:00:22.356433+0000","last_active":"2023-03-03T08:00:22.368309+0000","last_peered":"2023-03-03T08:00:22.368309+0000","last_clean":"2023-03-03T08:00:22.368309+0000","last_became_active":"2023-03-03T08:00:22.356154+0000","last_became_peered":"2023-03-03T08:00:22.356154+0000","last_unstale":"2023-03-03T08:00:22.368309+0000","last_undegraded":"2023-03-03T08:00:22.368309+0000","last_fullsized":"2023-03-03T08:00:22.368309+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:39.487708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.366920+0000","last_change":"2023-03-03T08:00:22.365431+0000","last_active":"2023-03-03T08:00:22.366920+0000","last_peered":"2023-03-03T08:00:22.366920+0000","last_clean":"2023-03-03T08:00:22.366920+0000","last_became_active":"2023-03-03T08:00:22.364882+0000","last_became_peered":"2023-03-03T08:00:22.364882+0000","last_unstale":"2023-03-03T08:00:22.366920+0000","last_undegraded":"2023-03-03T08:00:22.366920+0000","last_fullsized":"2023-03-03T08:00:22.366920+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:16:26.224161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.347929+0000","last_change":"2023-03-03T08:00:22.344899+0000","last_active":"2023-03-03T08:00:22.347929+0000","last_peered":"2023-03-03T08:00:22.347929+0000","last_clean":"2023-03-03T08:00:22.347929+0000","last_became_active":"2023-03-03T08:00:22.344558+0000","last_became_peered":"2023-03-03T08:00:22.344558+0000","last_unstale":"2023-03-03T08:00:22.347929+0000","last_undegraded":"2023-03-03T08:00:22.347929+0000","last_fullsized":"2023-03-03T08:00:22.347929+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:34.298493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.362506+0000","last_change":"2023-03-03T08:00:22.360864+0000","last_active":"2023-03-03T08:00:22.362506+0000","last_peered":"2023-03-03T08:00:22.362506+0000","last_clean":"2023-03-03T08:00:22.362506+0000","last_became_active":"2023-03-03T08:00:22.360550+0000","last_became_peered":"2023-03-03T08:00:22.360550+0000","last_unstale":"2023-03-03T08:00:22.362506+0000","last_undegraded":"2023-03-03T08:00:22.362506+0000","last_fullsized":"2023-03-03T08:00:22.362506+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:00:54.378703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.349837+0000","last_change":"2023-03-03T08:00:22.346751+0000","last_active":"2023-03-03T08:00:22.349837+0000","last_peered":"2023-03-03T08:00:22.349837+0000","last_clean":"2023-03-03T08:00:22.349837+0000","last_became_active":"2023-03-03T08:00:22.346388+0000","last_became_peered":"2023-03-03T08:00:22.346388+0000","last_unstale":"2023-03-03T08:00:22.349837+0000","last_undegraded":"2023-03-03T08:00:22.349837+0000","last_fullsized":"2023-03-03T08:00:22.349837+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:40.317030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356366+0000","last_change":"2023-03-03T08:00:22.347590+0000","last_active":"2023-03-03T08:00:22.356366+0000","last_peered":"2023-03-03T08:00:22.356366+0000","last_clean":"2023-03-03T08:00:22.356366+0000","last_became_active":"2023-03-03T08:00:22.347256+0000","last_became_peered":"2023-03-03T08:00:22.347256+0000","last_unstale":"2023-03-03T08:00:22.356366+0000","last_undegraded":"2023-03-03T08:00:22.356366+0000","last_fullsized":"2023-03-03T08:00:22.356366+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:23:14.650660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.359341+0000","last_change":"2023-03-03T08:00:22.352944+0000","last_active":"2023-03-03T08:00:22.359341+0000","last_peered":"2023-03-03T08:00:22.359341+0000","last_clean":"2023-03-03T08:00:22.359341+0000","last_became_active":"2023-03-03T08:00:22.352547+0000","last_became_peered":"2023-03-03T08:00:22.352547+0000","last_unstale":"2023-03-03T08:00:22.359341+0000","last_undegraded":"2023-03-03T08:00:22.359341+0000","last_fullsized":"2023-03-03T08:00:22.359341+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:53:16.100003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368088+0000","last_change":"2023-03-03T08:00:22.358102+0000","last_active":"2023-03-03T08:00:22.368088+0000","last_peered":"2023-03-03T08:00:22.368088+0000","last_clean":"2023-03-03T08:00:22.368088+0000","last_became_active":"2023-03-03T08:00:22.357726+0000","last_became_peered":"2023-03-03T08:00:22.357726+0000","last_unstale":"2023-03-03T08:00:22.368088+0000","last_undegraded":"2023-03-03T08:00:22.368088+0000","last_fullsized":"2023-03-03T08:00:22.368088+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:19:01.442408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.349788+0000","last_change":"2023-03-03T08:00:22.346586+0000","last_active":"2023-03-03T08:00:22.349788+0000","last_peered":"2023-03-03T08:00:22.349788+0000","last_clean":"2023-03-03T08:00:22.349788+0000","last_became_active":"2023-03-03T08:00:22.346252+0000","last_became_peered":"2023-03-03T08:00:22.346252+0000","last_unstale":"2023-03-03T08:00:22.349788+0000","last_undegraded":"2023-03-03T08:00:22.349788+0000","last_fullsized":"2023-03-03T08:00:22.349788+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:44:04.177495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.352213+0000","last_change":"2023-03-03T08:00:22.347364+0000","last_active":"2023-03-03T08:00:22.352213+0000","last_peered":"2023-03-03T08:00:22.352213+0000","last_clean":"2023-03-03T08:00:22.352213+0000","last_became_active":"2023-03-03T08:00:22.347036+0000","last_became_peered":"2023-03-03T08:00:22.347036+0000","last_unstale":"2023-03-03T08:00:22.352213+0000","last_undegraded":"2023-03-03T08:00:22.352213+0000","last_fullsized":"2023-03-03T08:00:22.352213+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:20.721006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.352093+0000","last_change":"2023-03-03T08:00:22.342081+0000","last_active":"2023-03-03T08:00:22.352093+0000","last_peered":"2023-03-03T08:00:22.352093+0000","last_clean":"2023-03-03T08:00:22.352093+0000","last_became_active":"2023-03-03T08:00:22.341783+0000","last_became_peered":"2023-03-03T08:00:22.341783+0000","last_unstale":"2023-03-03T08:00:22.352093+0000","last_undegraded":"2023-03-03T08:00:22.352093+0000","last_fullsized":"2023-03-03T08:00:22.352093+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:20.721006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.361028+0000","last_change":"2023-03-03T08:00:22.354119+0000","last_active":"2023-03-03T08:00:22.361028+0000","last_peered":"2023-03-03T08:00:22.361028+0000","last_clean":"2023-03-03T08:00:22.361028+0000","last_became_active":"2023-03-03T08:00:22.353824+0000","last_became_peered":"2023-03-03T08:00:22.353824+0000","last_unstale":"2023-03-03T08:00:22.361028+0000","last_undegraded":"2023-03-03T08:00:22.361028+0000","last_fullsized":"2023-03-03T08:00:22.361028+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:27.867451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.354242+0000","last_change":"2023-03-03T08:00:22.350728+0000","last_active":"2023-03-03T08:00:22.354242+0000","last_peered":"2023-03-03T08:00:22.354242+0000","last_clean":"2023-03-03T08:00:22.354242+0000","last_became_active":"2023-03-03T08:00:22.350468+0000","last_became_peered":"2023-03-03T08:00:22.350468+0000","last_unstale":"2023-03-03T08:00:22.354242+0000","last_undegraded":"2023-03-03T08:00:22.354242+0000","last_fullsized":"2023-03-03T08:00:22.354242+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:36:10.563980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.361036+0000","last_change":"2023-03-03T08:00:22.356264+0000","last_active":"2023-03-03T08:00:22.361036+0000","last_peered":"2023-03-03T08:00:22.361036+0000","last_clean":"2023-03-03T08:00:22.361036+0000","last_became_active":"2023-03-03T08:00:22.355666+0000","last_became_peered":"2023-03-03T08:00:22.355666+0000","last_unstale":"2023-03-03T08:00:22.361036+0000","last_undegraded":"2023-03-03T08:00:22.361036+0000","last_fullsized":"2023-03-03T08:00:22.361036+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:03.239405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.367982+0000","last_change":"2023-03-03T08:00:22.357934+0000","last_active":"2023-03-03T08:00:22.367982+0000","last_peered":"2023-03-03T08:00:22.367982+0000","last_clean":"2023-03-03T08:00:22.367982+0000","last_became_active":"2023-03-03T08:00:22.357644+0000","last_became_peered":"2023-03-03T08:00:22.357644+0000","last_unstale":"2023-03-03T08:00:22.367982+0000","last_undegraded":"2023-03-03T08:00:22.367982+0000","last_fullsized":"2023-03-03T08:00:22.367982+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:44:04.177495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.365907+0000","last_change":"2023-03-03T08:00:22.363152+0000","last_active":"2023-03-03T08:00:22.365907+0000","last_peered":"2023-03-03T08:00:22.365907+0000","last_clean":"2023-03-03T08:00:22.365907+0000","last_became_active":"2023-03-03T08:00:22.362838+0000","last_became_peered":"2023-03-03T08:00:22.362838+0000","last_unstale":"2023-03-03T08:00:22.365907+0000","last_undegraded":"2023-03-03T08:00:22.365907+0000","last_fullsized":"2023-03-03T08:00:22.365907+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:35:40.983679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.370386+0000","last_change":"2023-03-03T08:00:22.365139+0000","last_active":"2023-03-03T08:00:22.370386+0000","last_peered":"2023-03-03T08:00:22.370386+0000","last_clean":"2023-03-03T08:00:22.370386+0000","last_became_active":"2023-03-03T08:00:22.364818+0000","last_became_peered":"2023-03-03T08:00:22.364818+0000","last_unstale":"2023-03-03T08:00:22.370386+0000","last_undegraded":"2023-03-03T08:00:22.370386+0000","last_fullsized":"2023-03-03T08:00:22.370386+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:00:39.446569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360521+0000","last_change":"2023-03-03T08:00:22.352070+0000","last_active":"2023-03-03T08:00:22.360521+0000","last_peered":"2023-03-03T08:00:22.360521+0000","last_clean":"2023-03-03T08:00:22.360521+0000","last_became_active":"2023-03-03T08:00:22.351637+0000","last_became_peered":"2023-03-03T08:00:22.351637+0000","last_unstale":"2023-03-03T08:00:22.360521+0000","last_undegraded":"2023-03-03T08:00:22.360521+0000","last_fullsized":"2023-03-03T08:00:22.360521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:19:01.442408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.352202+0000","last_change":"2023-03-03T08:00:22.348419+0000","last_active":"2023-03-03T08:00:22.352202+0000","last_peered":"2023-03-03T08:00:22.352202+0000","last_clean":"2023-03-03T08:00:22.352202+0000","last_became_active":"2023-03-03T08:00:22.348168+0000","last_became_peered":"2023-03-03T08:00:22.348168+0000","last_unstale":"2023-03-03T08:00:22.352202+0000","last_undegraded":"2023-03-03T08:00:22.352202+0000","last_fullsized":"2023-03-03T08:00:22.352202+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:52:14.400121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.346307+0000","last_change":"2023-03-03T08:00:22.343763+0000","last_active":"2023-03-03T08:00:22.346307+0000","last_peered":"2023-03-03T08:00:22.346307+0000","last_clean":"2023-03-03T08:00:22.346307+0000","last_became_active":"2023-03-03T08:00:22.343407+0000","last_became_peered":"2023-03-03T08:00:22.343407+0000","last_unstale":"2023-03-03T08:00:22.346307+0000","last_undegraded":"2023-03-03T08:00:22.346307+0000","last_fullsized":"2023-03-03T08:00:22.346307+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:51:43.370212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.343549+0000","last_change":"2023-03-03T08:00:22.340970+0000","last_active":"2023-03-03T08:00:22.343549+0000","last_peered":"2023-03-03T08:00:22.343549+0000","last_clean":"2023-03-03T08:00:22.343549+0000","last_became_active":"2023-03-03T08:00:22.340635+0000","last_became_peered":"2023-03-03T08:00:22.340635+0000","last_unstale":"2023-03-03T08:00:22.343549+0000","last_undegraded":"2023-03-03T08:00:22.343549+0000","last_fullsized":"2023-03-03T08:00:22.343549+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:06.529333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.361178+0000","last_change":"2023-03-03T08:00:22.355393+0000","last_active":"2023-03-03T08:00:22.361178+0000","last_peered":"2023-03-03T08:00:22.361178+0000","last_clean":"2023-03-03T08:00:22.361178+0000","last_became_active":"2023-03-03T08:00:22.354452+0000","last_became_peered":"2023-03-03T08:00:22.354452+0000","last_unstale":"2023-03-03T08:00:22.361178+0000","last_undegraded":"2023-03-03T08:00:22.361178+0000","last_fullsized":"2023-03-03T08:00:22.361178+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:08:44.952506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.359810+0000","last_change":"2023-03-03T08:00:22.350155+0000","last_active":"2023-03-03T08:00:22.359810+0000","last_peered":"2023-03-03T08:00:22.359810+0000","last_clean":"2023-03-03T08:00:22.359810+0000","last_became_active":"2023-03-03T08:00:22.349712+0000","last_became_peered":"2023-03-03T08:00:22.349712+0000","last_unstale":"2023-03-03T08:00:22.359810+0000","last_undegraded":"2023-03-03T08:00:22.359810+0000","last_fullsized":"2023-03-03T08:00:22.359810+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:06:54.326996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.362401+0000","last_change":"2023-03-03T08:00:22.352054+0000","last_active":"2023-03-03T08:00:22.362401+0000","last_peered":"2023-03-03T08:00:22.362401+0000","last_clean":"2023-03-03T08:00:22.362401+0000","last_became_active":"2023-03-03T08:00:22.351777+0000","last_became_peered":"2023-03-03T08:00:22.351777+0000","last_unstale":"2023-03-03T08:00:22.362401+0000","last_undegraded":"2023-03-03T08:00:22.362401+0000","last_fullsized":"2023-03-03T08:00:22.362401+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:06.529333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360855+0000","last_change":"2023-03-03T08:00:22.358047+0000","last_active":"2023-03-03T08:00:22.360855+0000","last_peered":"2023-03-03T08:00:22.360855+0000","last_clean":"2023-03-03T08:00:22.360855+0000","last_became_active":"2023-03-03T08:00:22.357730+0000","last_became_peered":"2023-03-03T08:00:22.357730+0000","last_unstale":"2023-03-03T08:00:22.360855+0000","last_undegraded":"2023-03-03T08:00:22.360855+0000","last_fullsized":"2023-03-03T08:00:22.360855+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:53:16.425693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356826+0000","last_change":"2023-03-03T08:00:22.347292+0000","last_active":"2023-03-03T08:00:22.356826+0000","last_peered":"2023-03-03T08:00:22.356826+0000","last_clean":"2023-03-03T08:00:22.356826+0000","last_became_active":"2023-03-03T08:00:22.346956+0000","last_became_peered":"2023-03-03T08:00:22.346956+0000","last_unstale":"2023-03-03T08:00:22.356826+0000","last_undegraded":"2023-03-03T08:00:22.356826+0000","last_fullsized":"2023-03-03T08:00:22.356826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:43.689368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.350844+0000","last_change":"2023-03-03T08:00:22.346466+0000","last_active":"2023-03-03T08:00:22.350844+0000","last_peered":"2023-03-03T08:00:22.350844+0000","last_clean":"2023-03-03T08:00:22.350844+0000","last_became_active":"2023-03-03T08:00:22.346127+0000","last_became_peered":"2023-03-03T08:00:22.346127+0000","last_unstale":"2023-03-03T08:00:22.350844+0000","last_undegraded":"2023-03-03T08:00:22.350844+0000","last_fullsized":"2023-03-03T08:00:22.350844+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:06.529333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368067+0000","last_change":"2023-03-03T08:00:22.355948+0000","last_active":"2023-03-03T08:00:22.368067+0000","last_peered":"2023-03-03T08:00:22.368067+0000","last_clean":"2023-03-03T08:00:22.368067+0000","last_became_active":"2023-03-03T08:00:22.355644+0000","last_became_peered":"2023-03-03T08:00:22.355644+0000","last_unstale":"2023-03-03T08:00:22.368067+0000","last_undegraded":"2023-03-03T08:00:22.368067+0000","last_fullsized":"2023-03-03T08:00:22.368067+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:40.317030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.361441+0000","last_change":"2023-03-03T08:00:22.352801+0000","last_active":"2023-03-03T08:00:22.361441+0000","last_peered":"2023-03-03T08:00:22.361441+0000","last_clean":"2023-03-03T08:00:22.361441+0000","last_became_active":"2023-03-03T08:00:22.352383+0000","last_became_peered":"2023-03-03T08:00:22.352383+0000","last_unstale":"2023-03-03T08:00:22.361441+0000","last_undegraded":"2023-03-03T08:00:22.361441+0000","last_fullsized":"2023-03-03T08:00:22.361441+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:28:15.779229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.364255+0000","last_change":"2023-03-03T08:00:22.357842+0000","last_active":"2023-03-03T08:00:22.364255+0000","last_peered":"2023-03-03T08:00:22.364255+0000","last_clean":"2023-03-03T08:00:22.364255+0000","last_became_active":"2023-03-03T08:00:22.357562+0000","last_became_peered":"2023-03-03T08:00:22.357562+0000","last_unstale":"2023-03-03T08:00:22.364255+0000","last_undegraded":"2023-03-03T08:00:22.364255+0000","last_fullsized":"2023-03-03T08:00:22.364255+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:25:55.351299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"65'4","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360098+0000","last_change":"2023-03-03T08:00:21.791637+0000","last_active":"2023-03-03T08:00:22.360098+0000","last_peered":"2023-03-03T08:00:22.360098+0000","last_clean":"2023-03-03T08:00:22.360098+0000","last_became_active":"2023-03-03T08:00:21.791370+0000","last_became_peered":"2023-03-03T08:00:21.791370+0000","last_unstale":"2023-03-03T08:00:22.360098+0000","last_undegraded":"2023-03-03T08:00:22.360098+0000","last_fullsized":"2023-03-03T08:00:22.360098+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:53:37.291520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368899+0000","last_change":"2023-03-03T08:00:22.363157+0000","last_active":"2023-03-03T08:00:22.368899+0000","last_peered":"2023-03-03T08:00:22.368899+0000","last_clean":"2023-03-03T08:00:22.368899+0000","last_became_active":"2023-03-03T08:00:22.362830+0000","last_became_peered":"2023-03-03T08:00:22.362830+0000","last_unstale":"2023-03-03T08:00:22.368899+0000","last_undegraded":"2023-03-03T08:00:22.368899+0000","last_fullsized":"2023-03-03T08:00:22.368899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:27.867451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"65'1","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.359464+0000","last_change":"2023-03-03T08:00:21.792187+0000","last_active":"2023-03-03T08:00:22.359464+0000","last_peered":"2023-03-03T08:00:22.359464+0000","last_clean":"2023-03-03T08:00:22.359464+0000","last_became_active":"2023-03-03T08:00:21.791813+0000","last_became_peered":"2023-03-03T08:00:21.791813+0000","last_unstale":"2023-03-03T08:00:22.359464+0000","last_undegraded":"2023-03-03T08:00:22.359464+0000","last_fullsized":"2023-03-03T08:00:22.359464+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:26.856771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.353039+0000","last_change":"2023-03-03T08:00:22.351663+0000","last_active":"2023-03-03T08:00:22.353039+0000","last_peered":"2023-03-03T08:00:22.353039+0000","last_clean":"2023-03-03T08:00:22.353039+0000","last_became_active":"2023-03-03T08:00:22.351260+0000","last_became_peered":"2023-03-03T08:00:22.351260+0000","last_unstale":"2023-03-03T08:00:22.353039+0000","last_undegraded":"2023-03-03T08:00:22.353039+0000","last_fullsized":"2023-03-03T08:00:22.353039+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:00:05.665712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.358144+0000","last_change":"2023-03-03T08:00:21.787824+0000","last_active":"2023-03-03T08:00:22.358144+0000","last_peered":"2023-03-03T08:00:22.358144+0000","last_clean":"2023-03-03T08:00:22.358144+0000","last_became_active":"2023-03-03T08:00:21.787512+0000","last_became_peered":"2023-03-03T08:00:21.787512+0000","last_unstale":"2023-03-03T08:00:22.358144+0000","last_undegraded":"2023-03-03T08:00:22.358144+0000","last_fullsized":"2023-03-03T08:00:22.358144+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:42.678687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.359901+0000","last_change":"2023-03-03T08:00:22.354134+0000","last_active":"2023-03-03T08:00:22.359901+0000","last_peered":"2023-03-03T08:00:22.359901+0000","last_clean":"2023-03-03T08:00:22.359901+0000","last_became_active":"2023-03-03T08:00:22.353810+0000","last_became_peered":"2023-03-03T08:00:22.353810+0000","last_unstale":"2023-03-03T08:00:22.359901+0000","last_undegraded":"2023-03-03T08:00:22.359901+0000","last_fullsized":"2023-03-03T08:00:22.359901+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:04.223183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.354232+0000","last_change":"2023-03-03T08:00:21.784586+0000","last_active":"2023-03-03T08:00:22.354232+0000","last_peered":"2023-03-03T08:00:22.354232+0000","last_clean":"2023-03-03T08:00:22.354232+0000","last_became_active":"2023-03-03T08:00:21.784312+0000","last_became_peered":"2023-03-03T08:00:21.784312+0000","last_unstale":"2023-03-03T08:00:22.354232+0000","last_undegraded":"2023-03-03T08:00:22.354232+0000","last_fullsized":"2023-03-03T08:00:22.354232+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:53:15.089322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360628+0000","last_change":"2023-03-03T08:00:22.356583+0000","last_active":"2023-03-03T08:00:22.360628+0000","last_peered":"2023-03-03T08:00:22.360628+0000","last_clean":"2023-03-03T08:00:22.360628+0000","last_became_active":"2023-03-03T08:00:22.356348+0000","last_became_peered":"2023-03-03T08:00:22.356348+0000","last_unstale":"2023-03-03T08:00:22.360628+0000","last_undegraded":"2023-03-03T08:00:22.360628+0000","last_fullsized":"2023-03-03T08:00:22.360628+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:43:23.319076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"65'1","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.348088+0000","last_change":"2023-03-03T08:00:21.778523+0000","last_active":"2023-03-03T08:00:22.348088+0000","last_peered":"2023-03-03T08:00:22.348088+0000","last_clean":"2023-03-03T08:00:22.348088+0000","last_became_active":"2023-03-03T08:00:21.778260+0000","last_became_peered":"2023-03-03T08:00:21.778260+0000","last_unstale":"2023-03-03T08:00:22.348088+0000","last_undegraded":"2023-03-03T08:00:22.348088+0000","last_fullsized":"2023-03-03T08:00:22.348088+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:26.856771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356934+0000","last_change":"2023-03-03T08:00:22.350090+0000","last_active":"2023-03-03T08:00:22.356934+0000","last_peered":"2023-03-03T08:00:22.356934+0000","last_clean":"2023-03-03T08:00:22.356934+0000","last_became_active":"2023-03-03T08:00:22.349808+0000","last_became_peered":"2023-03-03T08:00:22.349808+0000","last_unstale":"2023-03-03T08:00:22.356934+0000","last_undegraded":"2023-03-03T08:00:22.356934+0000","last_fullsized":"2023-03-03T08:00:22.356934+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:09.676363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"65'1","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.348542+0000","last_change":"2023-03-03T08:00:21.787110+0000","last_active":"2023-03-03T08:00:22.348542+0000","last_peered":"2023-03-03T08:00:22.348542+0000","last_clean":"2023-03-03T08:00:22.348542+0000","last_became_active":"2023-03-03T08:00:21.786811+0000","last_became_peered":"2023-03-03T08:00:21.786811+0000","last_unstale":"2023-03-03T08:00:22.348542+0000","last_undegraded":"2023-03-03T08:00:22.348542+0000","last_fullsized":"2023-03-03T08:00:22.348542+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:57:43.225652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.366830+0000","last_change":"2023-03-03T08:00:22.365220+0000","last_active":"2023-03-03T08:00:22.366830+0000","last_peered":"2023-03-03T08:00:22.366830+0000","last_clean":"2023-03-03T08:00:22.366830+0000","last_became_active":"2023-03-03T08:00:22.364613+0000","last_became_peered":"2023-03-03T08:00:22.364613+0000","last_unstale":"2023-03-03T08:00:22.366830+0000","last_undegraded":"2023-03-03T08:00:22.366830+0000","last_fullsized":"2023-03-03T08:00:22.366830+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:35:58.114402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.332133+0000","last_change":"2023-03-03T08:00:21.783560+0000","last_active":"2023-03-03T08:00:22.332133+0000","last_peered":"2023-03-03T08:00:22.332133+0000","last_clean":"2023-03-03T08:00:22.332133+0000","last_became_active":"2023-03-03T08:00:21.783190+0000","last_became_peered":"2023-03-03T08:00:21.783190+0000","last_unstale":"2023-03-03T08:00:22.332133+0000","last_undegraded":"2023-03-03T08:00:22.332133+0000","last_fullsized":"2023-03-03T08:00:22.332133+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:39.306350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.352314+0000","last_change":"2023-03-03T08:00:22.345314+0000","last_active":"2023-03-03T08:00:22.352314+0000","last_peered":"2023-03-03T08:00:22.352314+0000","last_clean":"2023-03-03T08:00:22.352314+0000","last_became_active":"2023-03-03T08:00:22.345077+0000","last_became_peered":"2023-03-03T08:00:22.345077+0000","last_unstale":"2023-03-03T08:00:22.352314+0000","last_undegraded":"2023-03-03T08:00:22.352314+0000","last_fullsized":"2023-03-03T08:00:22.352314+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:52.425703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.344936+0000","last_change":"2023-03-03T08:00:21.792376+0000","last_active":"2023-03-03T08:00:22.344936+0000","last_peered":"2023-03-03T08:00:22.344936+0000","last_clean":"2023-03-03T08:00:22.344936+0000","last_became_active":"2023-03-03T08:00:21.792126+0000","last_became_peered":"2023-03-03T08:00:21.792126+0000","last_unstale":"2023-03-03T08:00:22.344936+0000","last_undegraded":"2023-03-03T08:00:22.344936+0000","last_fullsized":"2023-03-03T08:00:22.344936+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:44.522714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356052+0000","last_change":"2023-03-03T08:00:22.351274+0000","last_active":"2023-03-03T08:00:22.356052+0000","last_peered":"2023-03-03T08:00:22.356052+0000","last_clean":"2023-03-03T08:00:22.356052+0000","last_became_active":"2023-03-03T08:00:22.351025+0000","last_became_peered":"2023-03-03T08:00:22.351025+0000","last_unstale":"2023-03-03T08:00:22.356052+0000","last_undegraded":"2023-03-03T08:00:22.356052+0000","last_fullsized":"2023-03-03T08:00:22.356052+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:55:49.478050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.331929+0000","last_change":"2023-03-03T08:00:21.790545+0000","last_active":"2023-03-03T08:00:22.331929+0000","last_peered":"2023-03-03T08:00:22.331929+0000","last_clean":"2023-03-03T08:00:22.331929+0000","last_became_active":"2023-03-03T08:00:21.790052+0000","last_became_peered":"2023-03-03T08:00:21.790052+0000","last_unstale":"2023-03-03T08:00:22.331929+0000","last_undegraded":"2023-03-03T08:00:22.331929+0000","last_fullsized":"2023-03-03T08:00:22.331929+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:51:33.987978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.358127+0000","last_change":"2023-03-03T08:00:22.353413+0000","last_active":"2023-03-03T08:00:22.358127+0000","last_peered":"2023-03-03T08:00:22.358127+0000","last_clean":"2023-03-03T08:00:22.358127+0000","last_became_active":"2023-03-03T08:00:22.353114+0000","last_became_peered":"2023-03-03T08:00:22.353114+0000","last_unstale":"2023-03-03T08:00:22.358127+0000","last_undegraded":"2023-03-03T08:00:22.358127+0000","last_fullsized":"2023-03-03T08:00:22.358127+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:34.298493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.342606+0000","last_change":"2023-03-03T08:00:21.783602+0000","last_active":"2023-03-03T08:00:22.342606+0000","last_peered":"2023-03-03T08:00:22.342606+0000","last_clean":"2023-03-03T08:00:22.342606+0000","last_became_active":"2023-03-03T08:00:21.783297+0000","last_became_peered":"2023-03-03T08:00:21.783297+0000","last_unstale":"2023-03-03T08:00:22.342606+0000","last_undegraded":"2023-03-03T08:00:22.342606+0000","last_fullsized":"2023-03-03T08:00:22.342606+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:08:45.794437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.369192+0000","last_change":"2023-03-03T08:00:22.367931+0000","last_active":"2023-03-03T08:00:22.369192+0000","last_peered":"2023-03-03T08:00:22.369192+0000","last_clean":"2023-03-03T08:00:22.369192+0000","last_became_active":"2023-03-03T08:00:22.367374+0000","last_became_peered":"2023-03-03T08:00:22.367374+0000","last_unstale":"2023-03-03T08:00:22.369192+0000","last_undegraded":"2023-03-03T08:00:22.369192+0000","last_fullsized":"2023-03-03T08:00:22.369192+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:12:21.211704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"65'3","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368407+0000","last_change":"2023-03-03T08:00:21.790207+0000","last_active":"2023-03-03T08:00:22.368407+0000","last_peered":"2023-03-03T08:00:22.368407+0000","last_clean":"2023-03-03T08:00:22.368407+0000","last_became_active":"2023-03-03T08:00:21.789535+0000","last_became_peered":"2023-03-03T08:00:21.789535+0000","last_unstale":"2023-03-03T08:00:22.368407+0000","last_undegraded":"2023-03-03T08:00:22.368407+0000","last_fullsized":"2023-03-03T08:00:22.368407+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:02.228725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.347507+0000","last_change":"2023-03-03T08:00:22.343895+0000","last_active":"2023-03-03T08:00:22.347507+0000","last_peered":"2023-03-03T08:00:22.347507+0000","last_clean":"2023-03-03T08:00:22.347507+0000","last_became_active":"2023-03-03T08:00:22.343577+0000","last_became_peered":"2023-03-03T08:00:22.343577+0000","last_unstale":"2023-03-03T08:00:22.347507+0000","last_undegraded":"2023-03-03T08:00:22.347507+0000","last_fullsized":"2023-03-03T08:00:22.347507+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:20.734366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"65'6","reported_seq":17,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.370487+0000","last_change":"2023-03-03T08:00:21.789108+0000","last_active":"2023-03-03T08:00:22.370487+0000","last_peered":"2023-03-03T08:00:22.370487+0000","last_clean":"2023-03-03T08:00:22.370487+0000","last_became_active":"2023-03-03T08:00:21.788850+0000","last_became_peered":"2023-03-03T08:00:21.788850+0000","last_unstale":"2023-03-03T08:00:22.370487+0000","last_undegraded":"2023-03-03T08:00:22.370487+0000","last_fullsized":"2023-03-03T08:00:22.370487+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:16:16.126986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368203+0000","last_change":"2023-03-03T08:00:22.358537+0000","last_active":"2023-03-03T08:00:22.368203+0000","last_peered":"2023-03-03T08:00:22.368203+0000","last_clean":"2023-03-03T08:00:22.368203+0000","last_became_active":"2023-03-03T08:00:22.358277+0000","last_became_peered":"2023-03-03T08:00:22.358277+0000","last_unstale":"2023-03-03T08:00:22.368203+0000","last_undegraded":"2023-03-03T08:00:22.368203+0000","last_fullsized":"2023-03-03T08:00:22.368203+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:06:54.326996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.366491+0000","last_change":"2023-03-03T08:00:21.788681+0000","last_active":"2023-03-03T08:00:22.366491+0000","last_peered":"2023-03-03T08:00:22.366491+0000","last_clean":"2023-03-03T08:00:22.366491+0000","last_became_active":"2023-03-03T08:00:21.788087+0000","last_became_peered":"2023-03-03T08:00:21.788087+0000","last_unstale":"2023-03-03T08:00:22.366491+0000","last_undegraded":"2023-03-03T08:00:22.366491+0000","last_fullsized":"2023-03-03T08:00:22.366491+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:08:45.794437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.350067+0000","last_change":"2023-03-03T08:00:22.346539+0000","last_active":"2023-03-03T08:00:22.350067+0000","last_peered":"2023-03-03T08:00:22.350067+0000","last_clean":"2023-03-03T08:00:22.350067+0000","last_became_active":"2023-03-03T08:00:22.346149+0000","last_became_peered":"2023-03-03T08:00:22.346149+0000","last_unstale":"2023-03-03T08:00:22.350067+0000","last_undegraded":"2023-03-03T08:00:22.350067+0000","last_fullsized":"2023-03-03T08:00:22.350067+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:20.734366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.334121+0000","last_change":"2023-03-03T08:00:21.787442+0000","last_active":"2023-03-03T08:00:22.334121+0000","last_peered":"2023-03-03T08:00:22.334121+0000","last_clean":"2023-03-03T08:00:22.334121+0000","last_became_active":"2023-03-03T08:00:21.786782+0000","last_became_peered":"2023-03-03T08:00:21.786782+0000","last_unstale":"2023-03-03T08:00:22.334121+0000","last_undegraded":"2023-03-03T08:00:22.334121+0000","last_fullsized":"2023-03-03T08:00:22.334121+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:55:48.684553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.365840+0000","last_change":"2023-03-03T08:00:22.363185+0000","last_active":"2023-03-03T08:00:22.365840+0000","last_peered":"2023-03-03T08:00:22.365840+0000","last_clean":"2023-03-03T08:00:22.365840+0000","last_became_active":"2023-03-03T08:00:22.362886+0000","last_became_peered":"2023-03-03T08:00:22.362886+0000","last_unstale":"2023-03-03T08:00:22.365840+0000","last_undegraded":"2023-03-03T08:00:22.365840+0000","last_fullsized":"2023-03-03T08:00:22.365840+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:16:17.137666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.331162+0000","last_change":"2023-03-03T08:00:21.787586+0000","last_active":"2023-03-03T08:00:22.331162+0000","last_peered":"2023-03-03T08:00:22.331162+0000","last_clean":"2023-03-03T08:00:22.331162+0000","last_became_active":"2023-03-03T08:00:21.787272+0000","last_became_peered":"2023-03-03T08:00:21.787272+0000","last_unstale":"2023-03-03T08:00:22.331162+0000","last_undegraded":"2023-03-03T08:00:22.331162+0000","last_fullsized":"2023-03-03T08:00:22.331162+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:12.459278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.355207+0000","last_change":"2023-03-03T08:00:22.349923+0000","last_active":"2023-03-03T08:00:22.355207+0000","last_peered":"2023-03-03T08:00:22.355207+0000","last_clean":"2023-03-03T08:00:22.355207+0000","last_became_active":"2023-03-03T08:00:22.349552+0000","last_became_peered":"2023-03-03T08:00:22.349552+0000","last_unstale":"2023-03-03T08:00:22.355207+0000","last_undegraded":"2023-03-03T08:00:22.355207+0000","last_fullsized":"2023-03-03T08:00:22.355207+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:36:19.801740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"65'8","reported_seq":19,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.364523+0000","last_change":"2023-03-03T08:00:21.791488+0000","last_active":"2023-03-03T08:00:22.364523+0000","last_peered":"2023-03-03T08:00:22.364523+0000","last_clean":"2023-03-03T08:00:22.364523+0000","last_became_active":"2023-03-03T08:00:21.791122+0000","last_became_peered":"2023-03-03T08:00:21.791122+0000","last_unstale":"2023-03-03T08:00:22.364523+0000","last_undegraded":"2023-03-03T08:00:22.364523+0000","last_fullsized":"2023-03-03T08:00:22.364523+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:36:55.429526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.353072+0000","last_change":"2023-03-03T08:00:22.348199+0000","last_active":"2023-03-03T08:00:22.353072+0000","last_peered":"2023-03-03T08:00:22.353072+0000","last_clean":"2023-03-03T08:00:22.353072+0000","last_became_active":"2023-03-03T08:00:22.347841+0000","last_became_peered":"2023-03-03T08:00:22.347841+0000","last_unstale":"2023-03-03T08:00:22.353072+0000","last_undegraded":"2023-03-03T08:00:22.353072+0000","last_fullsized":"2023-03-03T08:00:22.353072+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:52.425703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.361110+0000","last_change":"2023-03-03T08:00:21.792101+0000","last_active":"2023-03-03T08:00:22.361110+0000","last_peered":"2023-03-03T08:00:22.361110+0000","last_clean":"2023-03-03T08:00:22.361110+0000","last_became_active":"2023-03-03T08:00:21.791481+0000","last_became_peered":"2023-03-03T08:00:21.791481+0000","last_unstale":"2023-03-03T08:00:22.361110+0000","last_undegraded":"2023-03-03T08:00:22.361110+0000","last_fullsized":"2023-03-03T08:00:22.361110+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:12:20.201024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360729+0000","last_change":"2023-03-03T08:00:22.358310+0000","last_active":"2023-03-03T08:00:22.360729+0000","last_peered":"2023-03-03T08:00:22.360729+0000","last_clean":"2023-03-03T08:00:22.360729+0000","last_became_active":"2023-03-03T08:00:22.358000+0000","last_became_peered":"2023-03-03T08:00:22.358000+0000","last_unstale":"2023-03-03T08:00:22.360729+0000","last_undegraded":"2023-03-03T08:00:22.360729+0000","last_fullsized":"2023-03-03T08:00:22.360729+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:23:00.594155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"66'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-03-03T08:00:23.335263+0000","last_change":"2023-03-03T08:00:21.786969+0000","last_active":"2023-03-03T08:00:23.335263+0000","last_peered":"2023-03-03T08:00:23.335263+0000","last_clean":"2023-03-03T08:00:23.335263+0000","last_became_active":"2023-03-03T08:00:21.786545+0000","last_became_peered":"2023-03-03T08:00:21.786545+0000","last_unstale":"2023-03-03T08:00:23.335263+0000","last_undegraded":"2023-03-03T08:00:23.335263+0000","last_fullsized":"2023-03-03T08:00:23.335263+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:16:25.213480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.352179+0000","last_change":"2023-03-03T08:00:22.348136+0000","last_active":"2023-03-03T08:00:22.352179+0000","last_peered":"2023-03-03T08:00:22.352179+0000","last_clean":"2023-03-03T08:00:22.352179+0000","last_became_active":"2023-03-03T08:00:22.347818+0000","last_became_peered":"2023-03-03T08:00:22.347818+0000","last_unstale":"2023-03-03T08:00:22.352179+0000","last_undegraded":"2023-03-03T08:00:22.352179+0000","last_fullsized":"2023-03-03T08:00:22.352179+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:11:16.603939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.332356+0000","last_change":"2023-03-03T08:00:21.788125+0000","last_active":"2023-03-03T08:00:22.332356+0000","last_peered":"2023-03-03T08:00:22.332356+0000","last_clean":"2023-03-03T08:00:22.332356+0000","last_became_active":"2023-03-03T08:00:21.787862+0000","last_became_peered":"2023-03-03T08:00:21.787862+0000","last_unstale":"2023-03-03T08:00:22.332356+0000","last_undegraded":"2023-03-03T08:00:22.332356+0000","last_fullsized":"2023-03-03T08:00:22.332356+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:03.212503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.364339+0000","last_change":"2023-03-03T08:00:22.356604+0000","last_active":"2023-03-03T08:00:22.364339+0000","last_peered":"2023-03-03T08:00:22.364339+0000","last_clean":"2023-03-03T08:00:22.364339+0000","last_became_active":"2023-03-03T08:00:22.356343+0000","last_became_peered":"2023-03-03T08:00:22.356343+0000","last_unstale":"2023-03-03T08:00:22.364339+0000","last_undegraded":"2023-03-03T08:00:22.364339+0000","last_fullsized":"2023-03-03T08:00:22.364339+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:49:26.845615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.336881+0000","last_change":"2023-03-03T08:00:21.789082+0000","last_active":"2023-03-03T08:00:22.336881+0000","last_peered":"2023-03-03T08:00:22.336881+0000","last_clean":"2023-03-03T08:00:22.336881+0000","last_became_active":"2023-03-03T08:00:21.788827+0000","last_became_peered":"2023-03-03T08:00:21.788827+0000","last_unstale":"2023-03-03T08:00:22.336881+0000","last_undegraded":"2023-03-03T08:00:22.336881+0000","last_fullsized":"2023-03-03T08:00:22.336881+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:58.454988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.364373+0000","last_change":"2023-03-03T08:00:22.357855+0000","last_active":"2023-03-03T08:00:22.364373+0000","last_peered":"2023-03-03T08:00:22.364373+0000","last_clean":"2023-03-03T08:00:22.364373+0000","last_became_active":"2023-03-03T08:00:22.357512+0000","last_became_peered":"2023-03-03T08:00:22.357512+0000","last_unstale":"2023-03-03T08:00:22.364373+0000","last_undegraded":"2023-03-03T08:00:22.364373+0000","last_fullsized":"2023-03-03T08:00:22.364373+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:53:38.302201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.331929+0000","last_change":"2023-03-03T08:00:21.788909+0000","last_active":"2023-03-03T08:00:22.331929+0000","last_peered":"2023-03-03T08:00:22.331929+0000","last_clean":"2023-03-03T08:00:22.331929+0000","last_became_active":"2023-03-03T08:00:21.788395+0000","last_became_peered":"2023-03-03T08:00:21.788395+0000","last_unstale":"2023-03-03T08:00:22.331929+0000","last_undegraded":"2023-03-03T08:00:22.331929+0000","last_fullsized":"2023-03-03T08:00:22.331929+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:17:29.803501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368951+0000","last_change":"2023-03-03T08:00:22.363754+0000","last_active":"2023-03-03T08:00:22.368951+0000","last_peered":"2023-03-03T08:00:22.368951+0000","last_clean":"2023-03-03T08:00:22.368951+0000","last_became_active":"2023-03-03T08:00:22.363463+0000","last_became_peered":"2023-03-03T08:00:22.363463+0000","last_unstale":"2023-03-03T08:00:22.368951+0000","last_undegraded":"2023-03-03T08:00:22.368951+0000","last_fullsized":"2023-03-03T08:00:22.368951+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:08:44.952506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.332741+0000","last_change":"2023-03-03T08:00:21.782978+0000","last_active":"2023-03-03T08:00:22.332741+0000","last_peered":"2023-03-03T08:00:22.332741+0000","last_clean":"2023-03-03T08:00:22.332741+0000","last_became_active":"2023-03-03T08:00:21.782679+0000","last_became_peered":"2023-03-03T08:00:21.782679+0000","last_unstale":"2023-03-03T08:00:22.332741+0000","last_undegraded":"2023-03-03T08:00:22.332741+0000","last_fullsized":"2023-03-03T08:00:22.332741+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:19:00.431727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.354447+0000","last_change":"2023-03-03T08:00:22.348976+0000","last_active":"2023-03-03T08:00:22.354447+0000","last_peered":"2023-03-03T08:00:22.354447+0000","last_clean":"2023-03-03T08:00:22.354447+0000","last_became_active":"2023-03-03T08:00:22.348701+0000","last_became_peered":"2023-03-03T08:00:22.348701+0000","last_unstale":"2023-03-03T08:00:22.354447+0000","last_undegraded":"2023-03-03T08:00:22.354447+0000","last_fullsized":"2023-03-03T08:00:22.354447+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:51:34.998659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.332229+0000","last_change":"2023-03-03T08:00:21.783463+0000","last_active":"2023-03-03T08:00:22.332229+0000","last_peered":"2023-03-03T08:00:22.332229+0000","last_clean":"2023-03-03T08:00:22.332229+0000","last_became_active":"2023-03-03T08:00:21.782809+0000","last_became_peered":"2023-03-03T08:00:21.782809+0000","last_unstale":"2023-03-03T08:00:22.332229+0000","last_undegraded":"2023-03-03T08:00:22.332229+0000","last_fullsized":"2023-03-03T08:00:22.332229+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:44:03.166814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360566+0000","last_change":"2023-03-03T08:00:22.355471+0000","last_active":"2023-03-03T08:00:22.360566+0000","last_peered":"2023-03-03T08:00:22.360566+0000","last_clean":"2023-03-03T08:00:22.360566+0000","last_became_active":"2023-03-03T08:00:22.355175+0000","last_became_peered":"2023-03-03T08:00:22.355175+0000","last_unstale":"2023-03-03T08:00:22.360566+0000","last_undegraded":"2023-03-03T08:00:22.360566+0000","last_fullsized":"2023-03-03T08:00:22.360566+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:22:43.504725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.334926+0000","last_change":"2023-03-03T08:00:21.784441+0000","last_active":"2023-03-03T08:00:22.334926+0000","last_peered":"2023-03-03T08:00:22.334926+0000","last_clean":"2023-03-03T08:00:22.334926+0000","last_became_active":"2023-03-03T08:00:21.784193+0000","last_became_peered":"2023-03-03T08:00:21.784193+0000","last_unstale":"2023-03-03T08:00:22.334926+0000","last_undegraded":"2023-03-03T08:00:22.334926+0000","last_fullsized":"2023-03-03T08:00:22.334926+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:00:08.298430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356321+0000","last_change":"2023-03-03T08:00:22.346849+0000","last_active":"2023-03-03T08:00:22.356321+0000","last_peered":"2023-03-03T08:00:22.356321+0000","last_clean":"2023-03-03T08:00:22.356321+0000","last_became_active":"2023-03-03T08:00:22.346557+0000","last_became_peered":"2023-03-03T08:00:22.346557+0000","last_unstale":"2023-03-03T08:00:22.356321+0000","last_undegraded":"2023-03-03T08:00:22.356321+0000","last_fullsized":"2023-03-03T08:00:22.356321+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:40.317030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"65'1","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360262+0000","last_change":"2023-03-03T08:00:21.792370+0000","last_active":"2023-03-03T08:00:22.360262+0000","last_peered":"2023-03-03T08:00:22.360262+0000","last_clean":"2023-03-03T08:00:22.360262+0000","last_became_active":"2023-03-03T08:00:21.791836+0000","last_became_peered":"2023-03-03T08:00:21.791836+0000","last_unstale":"2023-03-03T08:00:22.360262+0000","last_undegraded":"2023-03-03T08:00:22.360262+0000","last_fullsized":"2023-03-03T08:00:22.360262+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:42:51.458743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.336780+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T08:00:22.336780+0000","last_peered":"2023-03-03T08:00:22.336780+0000","last_clean":"2023-03-03T08:00:22.336780+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T08:00:22.336780+0000","last_undegraded":"2023-03-03T08:00:22.336780+0000","last_fullsized":"2023-03-03T08:00:22.336780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.366849+0000","last_change":"2023-03-03T08:00:22.365438+0000","last_active":"2023-03-03T08:00:22.366849+0000","last_peered":"2023-03-03T08:00:22.366849+0000","last_clean":"2023-03-03T08:00:22.366849+0000","last_became_active":"2023-03-03T08:00:22.364935+0000","last_became_peered":"2023-03-03T08:00:22.364935+0000","last_unstale":"2023-03-03T08:00:22.366849+0000","last_undegraded":"2023-03-03T08:00:22.366849+0000","last_fullsized":"2023-03-03T08:00:22.366849+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:59.465669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.331666+0000","last_change":"2023-03-03T08:00:21.792272+0000","last_active":"2023-03-03T08:00:22.331666+0000","last_peered":"2023-03-03T08:00:22.331666+0000","last_clean":"2023-03-03T08:00:22.331666+0000","last_became_active":"2023-03-03T08:00:21.791961+0000","last_became_peered":"2023-03-03T08:00:21.791961+0000","last_unstale":"2023-03-03T08:00:22.331666+0000","last_undegraded":"2023-03-03T08:00:22.331666+0000","last_fullsized":"2023-03-03T08:00:22.331666+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:57:43.225652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":51,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.334339+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T08:00:22.334339+0000","last_peered":"2023-03-03T08:00:22.334339+0000","last_clean":"2023-03-03T08:00:22.334339+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T08:00:22.334339+0000","last_undegraded":"2023-03-03T08:00:22.334339+0000","last_fullsized":"2023-03-03T08:00:22.334339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360169+0000","last_change":"2023-03-03T08:00:22.354435+0000","last_active":"2023-03-03T08:00:22.360169+0000","last_peered":"2023-03-03T08:00:22.360169+0000","last_clean":"2023-03-03T08:00:22.360169+0000","last_became_active":"2023-03-03T08:00:22.354144+0000","last_became_peered":"2023-03-03T08:00:22.354144+0000","last_unstale":"2023-03-03T08:00:22.360169+0000","last_undegraded":"2023-03-03T08:00:22.360169+0000","last_fullsized":"2023-03-03T08:00:22.360169+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:39.487708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.334241+0000","last_change":"2023-03-03T08:00:21.787607+0000","last_active":"2023-03-03T08:00:22.334241+0000","last_peered":"2023-03-03T08:00:22.334241+0000","last_clean":"2023-03-03T08:00:22.334241+0000","last_became_active":"2023-03-03T08:00:21.787106+0000","last_became_peered":"2023-03-03T08:00:21.787106+0000","last_unstale":"2023-03-03T08:00:22.334241+0000","last_undegraded":"2023-03-03T08:00:22.334241+0000","last_fullsized":"2023-03-03T08:00:22.334241+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:30:19.298647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.343813+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T08:00:22.343813+0000","last_peered":"2023-03-03T08:00:22.343813+0000","last_clean":"2023-03-03T08:00:22.343813+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T08:00:22.343813+0000","last_undegraded":"2023-03-03T08:00:22.343813+0000","last_fullsized":"2023-03-03T08:00:22.343813+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.366899+0000","last_change":"2023-03-03T08:00:22.365234+0000","last_active":"2023-03-03T08:00:22.366899+0000","last_peered":"2023-03-03T08:00:22.366899+0000","last_clean":"2023-03-03T08:00:22.366899+0000","last_became_active":"2023-03-03T08:00:22.364559+0000","last_became_peered":"2023-03-03T08:00:22.364559+0000","last_unstale":"2023-03-03T08:00:22.366899+0000","last_undegraded":"2023-03-03T08:00:22.366899+0000","last_fullsized":"2023-03-03T08:00:22.366899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:00:09.309111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.331737+0000","last_change":"2023-03-03T08:00:21.775813+0000","last_active":"2023-03-03T08:00:22.331737+0000","last_peered":"2023-03-03T08:00:22.331737+0000","last_clean":"2023-03-03T08:00:22.331737+0000","last_became_active":"2023-03-03T08:00:21.775553+0000","last_became_peered":"2023-03-03T08:00:21.775553+0000","last_unstale":"2023-03-03T08:00:22.331737+0000","last_undegraded":"2023-03-03T08:00:22.331737+0000","last_fullsized":"2023-03-03T08:00:22.331737+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:19.710326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.335082+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T08:00:22.335082+0000","last_peered":"2023-03-03T08:00:22.335082+0000","last_clean":"2023-03-03T08:00:22.335082+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T08:00:22.335082+0000","last_undegraded":"2023-03-03T08:00:22.335082+0000","last_fullsized":"2023-03-03T08:00:22.335082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":205,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.330930+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T08:00:22.330930+0000","last_peered":"2023-03-03T08:00:22.330930+0000","last_clean":"2023-03-03T08:00:22.330930+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T08:00:22.330930+0000","last_undegraded":"2023-03-03T08:00:22.330930+0000","last_fullsized":"2023-03-03T08:00:22.330930+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356401+0000","last_change":"2023-03-03T08:00:22.352632+0000","last_active":"2023-03-03T08:00:22.356401+0000","last_peered":"2023-03-03T08:00:22.356401+0000","last_clean":"2023-03-03T08:00:22.356401+0000","last_became_active":"2023-03-03T08:00:22.351927+0000","last_became_peered":"2023-03-03T08:00:22.351927+0000","last_unstale":"2023-03-03T08:00:22.356401+0000","last_undegraded":"2023-03-03T08:00:22.356401+0000","last_fullsized":"2023-03-03T08:00:22.356401+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:27.364263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.344601+0000","last_change":"2023-03-03T08:00:21.790111+0000","last_active":"2023-03-03T08:00:22.344601+0000","last_peered":"2023-03-03T08:00:22.344601+0000","last_clean":"2023-03-03T08:00:22.344601+0000","last_became_active":"2023-03-03T08:00:21.789831+0000","last_became_peered":"2023-03-03T08:00:21.789831+0000","last_unstale":"2023-03-03T08:00:22.344601+0000","last_undegraded":"2023-03-03T08:00:22.344601+0000","last_fullsized":"2023-03-03T08:00:22.344601+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:19.710326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":52,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.330895+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T08:00:22.330895+0000","last_peered":"2023-03-03T08:00:22.330895+0000","last_clean":"2023-03-03T08:00:22.330895+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T08:00:22.330895+0000","last_undegraded":"2023-03-03T08:00:22.330895+0000","last_fullsized":"2023-03-03T08:00:22.330895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.365925+0000","last_change":"2023-03-03T08:00:22.363947+0000","last_active":"2023-03-03T08:00:22.365925+0000","last_peered":"2023-03-03T08:00:22.365925+0000","last_clean":"2023-03-03T08:00:22.365925+0000","last_became_active":"2023-03-03T08:00:22.363477+0000","last_became_peered":"2023-03-03T08:00:22.363477+0000","last_unstale":"2023-03-03T08:00:22.365925+0000","last_undegraded":"2023-03-03T08:00:22.365925+0000","last_fullsized":"2023-03-03T08:00:22.365925+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:30:20.309327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.369342+0000","last_change":"2023-03-03T08:00:21.787378+0000","last_active":"2023-03-03T08:00:22.369342+0000","last_peered":"2023-03-03T08:00:22.369342+0000","last_clean":"2023-03-03T08:00:22.369342+0000","last_became_active":"2023-03-03T08:00:21.787102+0000","last_became_peered":"2023-03-03T08:00:21.787102+0000","last_unstale":"2023-03-03T08:00:22.369342+0000","last_undegraded":"2023-03-03T08:00:22.369342+0000","last_fullsized":"2023-03-03T08:00:22.369342+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:45:29.829197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":52,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.332485+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T08:00:22.332485+0000","last_peered":"2023-03-03T08:00:22.332485+0000","last_clean":"2023-03-03T08:00:22.332485+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T08:00:22.332485+0000","last_undegraded":"2023-03-03T08:00:22.332485+0000","last_fullsized":"2023-03-03T08:00:22.332485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.369032+0000","last_change":"2023-03-03T08:00:22.366608+0000","last_active":"2023-03-03T08:00:22.369032+0000","last_peered":"2023-03-03T08:00:22.369032+0000","last_clean":"2023-03-03T08:00:22.369032+0000","last_became_active":"2023-03-03T08:00:22.366325+0000","last_became_peered":"2023-03-03T08:00:22.366325+0000","last_unstale":"2023-03-03T08:00:22.369032+0000","last_undegraded":"2023-03-03T08:00:22.369032+0000","last_fullsized":"2023-03-03T08:00:22.369032+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:20.721006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"65'1","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368284+0000","last_change":"2023-03-03T08:00:21.789735+0000","last_active":"2023-03-03T08:00:22.368284+0000","last_peered":"2023-03-03T08:00:22.368284+0000","last_clean":"2023-03-03T08:00:22.368284+0000","last_became_active":"2023-03-03T08:00:21.789201+0000","last_became_peered":"2023-03-03T08:00:21.789201+0000","last_unstale":"2023-03-03T08:00:22.368284+0000","last_undegraded":"2023-03-03T08:00:22.368284+0000","last_fullsized":"2023-03-03T08:00:22.368284+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:08.665683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.337975+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T08:00:22.337975+0000","last_peered":"2023-03-03T08:00:22.337975+0000","last_clean":"2023-03-03T08:00:22.337975+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T08:00:22.337975+0000","last_undegraded":"2023-03-03T08:00:22.337975+0000","last_fullsized":"2023-03-03T08:00:22.337975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.364313+0000","last_change":"2023-03-03T08:00:22.356451+0000","last_active":"2023-03-03T08:00:22.364313+0000","last_peered":"2023-03-03T08:00:22.364313+0000","last_clean":"2023-03-03T08:00:22.364313+0000","last_became_active":"2023-03-03T08:00:22.356191+0000","last_became_peered":"2023-03-03T08:00:22.356191+0000","last_unstale":"2023-03-03T08:00:22.364313+0000","last_undegraded":"2023-03-03T08:00:22.364313+0000","last_fullsized":"2023-03-03T08:00:22.364313+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:17:07.748684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.333217+0000","last_change":"2023-03-03T08:00:21.778956+0000","last_active":"2023-03-03T08:00:22.333217+0000","last_peered":"2023-03-03T08:00:22.333217+0000","last_clean":"2023-03-03T08:00:22.333217+0000","last_became_active":"2023-03-03T08:00:21.778666+0000","last_became_peered":"2023-03-03T08:00:21.778666+0000","last_unstale":"2023-03-03T08:00:22.333217+0000","last_undegraded":"2023-03-03T08:00:22.333217+0000","last_fullsized":"2023-03-03T08:00:22.333217+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:05.518653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.349803+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T08:00:22.349803+0000","last_peered":"2023-03-03T08:00:22.349803+0000","last_clean":"2023-03-03T08:00:22.349803+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T08:00:22.349803+0000","last_undegraded":"2023-03-03T08:00:22.349803+0000","last_fullsized":"2023-03-03T08:00:22.349803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574910,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10980,"kb_used_data":1372,"kb_used_omap":0,"kb_used_meta":9600,"kb_avail":94360860,"statfs":{"total":96636764160,"available":96625520640,"internally_reserved":0,"allocated":1404928,"data_stored":943239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9830400},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.67500000000000004,"15min":0.70499999999999996},"min":{"1min":0.437,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82499999999999996,"5min":1.546,"15min":1.546},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71199999999999997,"15min":0.72999999999999998},"min":{"1min":0.50600000000000001,"5min":0.5,"15min":0.5},"max":{"1min":0.96499999999999997,"5min":1.702,"15min":1.702},"last":0.81499999999999995}]},{"osd":1,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.89000000000000001,"15min":1.0840000000000001},"min":{"1min":0.49199999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.111,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.755,"5min":0.74099999999999999,"15min":0.76900000000000002},"min":{"1min":0.51600000000000001,"5min":0.504,"15min":0.504},"max":{"1min":1.1890000000000001,"5min":1.944,"15min":1.944},"last":0.66900000000000004}]},{"osd":2,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.71899999999999997,"15min":0.71399999999999997},"min":{"1min":0.63300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.224,"5min":1.252,"15min":1.252},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76800000000000002,"15min":0.79300000000000004},"min":{"1min":0.54600000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2490000000000001,"5min":1.329,"15min":1.329},"last":0.76400000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.71999999999999997,"15min":0.71499999999999997},"min":{"1min":0.59799999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.754,"15min":0.79000000000000004},"min":{"1min":0.55000000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.93400000000000005,"5min":1.294,"15min":1.294},"last":0.79800000000000004}]},{"osd":4,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.48399999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0049999999999999,"5min":1.077,"15min":1.077},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.66700000000000004,"15min":0.63400000000000001},"min":{"1min":0.51700000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.77500000000000002}]},{"osd":5,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.77100000000000002,"15min":0.80100000000000005},"min":{"1min":0.41699999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.036,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81699999999999995,"15min":0.82399999999999995},"min":{"1min":0.66400000000000003,"5min":0.496,"15min":0.496},"max":{"1min":1.2190000000000001,"5min":1.349,"15min":1.349},"last":0.79000000000000004}]},{"osd":6,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.64100000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0700000000000001,"5min":1.204,"15min":1.204},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.81100000000000005,"15min":0.82099999999999995},"min":{"1min":0.627,"5min":0.499,"15min":0.499},"max":{"1min":1.135,"5min":1.194,"15min":1.194},"last":0.70799999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9424,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":8640,"kb_avail":94362416,"statfs":{"total":96636764160,"available":96627113984,"internally_reserved":0,"allocated":794624,"data_stored":351126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8847360},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.70799999999999996,"15min":0.68899999999999995},"min":{"1min":0.57199999999999995,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.018,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68999999999999995,"15min":0.69499999999999995},"min":{"1min":0.52500000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.89700000000000002,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.745}]},{"osd":1,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70999999999999996,"15min":0.68600000000000005},"min":{"1min":0.52700000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.97099999999999997,"5min":3.323,"15min":3.323},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.73499999999999999,"15min":0.73099999999999998},"min":{"1min":0.55400000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97199999999999998,"5min":1.135,"15min":1.135},"last":0.63700000000000001}]},{"osd":2,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.753,"15min":0.73199999999999998},"min":{"1min":0.59499999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.98899999999999999,"5min":1.54,"15min":1.54},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.749,"15min":0.75900000000000001},"min":{"1min":0.54400000000000004,"5min":0.495,"15min":0.495},"max":{"1min":1.0089999999999999,"5min":1.698,"15min":1.698},"last":0.71799999999999997}]},{"osd":3,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81299999999999994,"15min":0.78900000000000003},"min":{"1min":0.57499999999999996,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.3089999999999999,"5min":4.8959999999999999,"15min":4.8959999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77200000000000002,"15min":0.77100000000000002},"min":{"1min":0.60899999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.95799999999999996,"5min":1.343,"15min":1.343},"last":0.78600000000000003}]},{"osd":4,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.79900000000000004,"15min":0.91100000000000003},"min":{"1min":0.439,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.871,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.80700000000000005,"15min":0.92400000000000004},"min":{"1min":0.53000000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.85099999999999998,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.88900000000000001}]},{"osd":5,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.72599999999999998,"15min":0.77600000000000002},"min":{"1min":0.47599999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.95399999999999996,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.80300000000000005,"15min":0.82499999999999996},"min":{"1min":0.499,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91500000000000004,"5min":1.429,"15min":1.429},"last":0.69499999999999995}]},{"osd":7,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.81899999999999995,"15min":0.83899999999999997},"min":{"1min":0.56699999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.0029999999999999,"5min":1.452,"15min":1.452},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.82099999999999995,"15min":0.87},"min":{"1min":0.59399999999999997,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.008,"5min":1.579,"15min":1.579},"last":0.81299999999999994}]}]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9580,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":8768,"kb_avail":94362260,"statfs":{"total":96636764160,"available":96626954240,"internally_reserved":0,"allocated":823296,"data_stored":354407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8978432},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.49099999999999999,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.63100000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58099999999999996,"15min":0.64700000000000002},"min":{"1min":0.36799999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.90200000000000002,"5min":3.972,"15min":3.972},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.74099999999999999,"15min":0.96399999999999997},"min":{"1min":0.371,"5min":0.313,"15min":0.313},"max":{"1min":0.78900000000000003,"5min":11.897,"15min":11.897},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.82299999999999995,"15min":1.0469999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.85199999999999998,"5min":11.804,"15min":11.804},"last":0.73699999999999999}]},{"osd":3,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.81399999999999995,"15min":1.036},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.89500000000000002,"5min":11.843999999999999,"15min":11.843999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64300000000000002,"15min":0.73499999999999999},"min":{"1min":0.47899999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.96399999999999997,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.54500000000000004}]},{"osd":4,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81799999999999995,"15min":0.86299999999999999},"min":{"1min":0.55000000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0960000000000001,"5min":1.613,"15min":1.613},"last":0.745},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.86099999999999999},"min":{"1min":0.65000000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.0800000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.76200000000000001}]},{"osd":5,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.84699999999999998,"15min":0.92500000000000004},"min":{"1min":0.63900000000000001,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.997,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81999999999999995,"15min":0.873},"min":{"1min":0.57899999999999996,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98899999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.66100000000000003}]},{"osd":6,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.91300000000000003,"15min":0.999},"min":{"1min":0.68400000000000005,"5min":0.628,"15min":0.628},"max":{"1min":1.173,"5min":3.907,"15min":3.907},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.875,"5min":0.88400000000000001,"15min":0.93999999999999995},"min":{"1min":0.67300000000000004,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1819999999999999,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.70999999999999996}]},{"osd":7,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84499999999999997,"15min":0.88300000000000001},"min":{"1min":0.70399999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.026,"5min":1.419,"15min":1.419},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.871,"5min":0.90600000000000003,"15min":1.012},"min":{"1min":0.70399999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1659999999999999,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":0.67700000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10640,"kb_used_data":1352,"kb_used_omap":0,"kb_used_meta":9280,"kb_avail":94361200,"statfs":{"total":96636764160,"available":96625868800,"internally_reserved":0,"allocated":1384448,"data_stored":941460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9502720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45800000000000002,"15min":0.47199999999999998},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65000000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.50600000000000001,"15min":0.51700000000000002},"min":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81999999999999995,"5min":0.871,"15min":0.871},"last":0.54300000000000004}]},{"osd":2,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53000000000000003,"15min":0.55000000000000004},"min":{"1min":0.38400000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77400000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.44,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73399999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.46800000000000003}]},{"osd":3,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70399999999999996,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.46600000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.80800000000000005,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.50700000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82299999999999995,"15min":0.84499999999999997},"min":{"1min":0.64700000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79800000000000004,"15min":0.81999999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0740000000000001,"5min":1.605,"15min":1.605},"last":0.83499999999999996}]},{"osd":5,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.80400000000000005,"15min":0.82199999999999995},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0980000000000001,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.82099999999999995,"15min":0.82499999999999996},"min":{"1min":0.61299999999999999,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.2070000000000001,"5min":1.286,"15min":1.286},"last":0.749}]},{"osd":6,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.85399999999999998,"15min":0.88500000000000001},"min":{"1min":0.65300000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2470000000000001,"5min":1.623,"15min":1.623},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.86399999999999999},"min":{"1min":0.66300000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.137,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.97099999999999997}]},{"osd":7,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84499999999999997,"15min":0.88300000000000001},"min":{"1min":0.59399999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.083,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.83999999999999997,"15min":0.86199999999999999},"min":{"1min":0.58199999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.1719999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.78900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9484,"kb_used_data":772,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362356,"statfs":{"total":96636764160,"available":96627052544,"internally_reserved":0,"allocated":790528,"data_stored":351092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.64800000000000002,"15min":0.87},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.61699999999999999,"5min":5.7889999999999997,"15min":5.7889999999999997},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.60899999999999999,"15min":0.73199999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67400000000000004,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":0.36499999999999999}]},{"osd":1,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.629,"15min":0.753},"min":{"1min":0.36299999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77700000000000002,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.79300000000000004,"15min":1.0089999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.77100000000000002,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.51500000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.88900000000000001,"15min":1.222},"min":{"1min":0.41399999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.69799999999999995,"5min":12.961,"15min":12.961},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.80000000000000004,"15min":1.036},"min":{"1min":0.42699999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.74099999999999999,"5min":6.1449999999999996,"15min":6.1449999999999996},"last":0.48299999999999998}]},{"osd":4,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.0469999999999999,"15min":1.2769999999999999},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1639999999999999,"5min":6.484,"15min":6.484},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.0580000000000001,"15min":1.2549999999999999},"min":{"1min":0.63600000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.1759999999999999,"5min":6.5510000000000002,"15min":6.5510000000000002},"last":0.68000000000000005}]},{"osd":5,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.97399999999999998,"15min":1.1799999999999999},"min":{"1min":0.61499999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.034,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.84699999999999998,"5min":1.0269999999999999,"15min":1.238},"min":{"1min":0.60999999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1499999999999999,"5min":6.6369999999999996,"15min":6.6369999999999996},"last":0.77100000000000002}]},{"osd":6,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.0429999999999999,"15min":1.24},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.1240000000000001,"5min":6.625,"15min":6.625},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.88200000000000001,"5min":1.0549999999999999,"15min":1.2669999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.099,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.69099999999999995}]},{"osd":7,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":1.0129999999999999,"15min":1.2310000000000001},"min":{"1min":0.69899999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.087,"5min":6.7009999999999996,"15min":6.7009999999999996},"last":0.623},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.0209999999999999,"15min":1.214},"min":{"1min":0.66500000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.081,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.64100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574909,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9936,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94361904,"statfs":{"total":96636764160,"available":96626589696,"internally_reserved":0,"allocated":794624,"data_stored":352111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.63,"15min":0.83499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.56599999999999995,"5min":9.5820000000000007,"15min":9.5820000000000007},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.70099999999999996,"15min":0.90600000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83599999999999997,"5min":9.7840000000000007,"15min":9.7840000000000007},"last":0.38200000000000001}]},{"osd":1,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.501,"15min":0.52300000000000002},"min":{"1min":0.34999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78900000000000003,"5min":1,"15min":1},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.75600000000000001,"15min":0.98999999999999999},"min":{"1min":0.41099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70499999999999996,"5min":9.702,"15min":9.702},"last":0.54200000000000004}]},{"osd":2,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.746,"15min":0.93600000000000005},"min":{"1min":0.436,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88300000000000001,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.623},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.80500000000000005,"15min":1.0229999999999999},"min":{"1min":0.498,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.86499999999999999,"5min":9.734,"15min":9.734},"last":0.60699999999999998}]},{"osd":4,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.96099999999999997,"15min":1.165},"min":{"1min":0.54400000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1819999999999999,"5min":9.8629999999999995,"15min":9.8629999999999995},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.83999999999999997,"15min":0.84799999999999998},"min":{"1min":0.629,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.097,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":0.745}]},{"osd":5,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82099999999999995,"15min":0.83299999999999996},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.1899999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.999,"15min":1.222},"min":{"1min":0.628,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0860000000000001,"5min":9.6600000000000001,"15min":9.6600000000000001},"last":0.77000000000000002}]},{"osd":6,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.0129999999999999,"15min":1.2370000000000001},"min":{"1min":0.65700000000000003,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0600000000000001,"5min":9.8190000000000008,"15min":9.8190000000000008},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82799999999999996,"15min":0.86699999999999999},"min":{"1min":0.629,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.2609999999999999,"5min":1.609,"15min":1.609},"last":0.78000000000000003}]},{"osd":7,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":1.0369999999999999,"15min":1.2749999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.204,"5min":9.625,"15min":9.625},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.81399999999999995,"5min":1.034,"15min":1.244},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.119,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":0.86699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574909,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9692,"kb_used_data":788,"kb_used_omap":0,"kb_used_meta":8896,"kb_avail":94362148,"statfs":{"total":96636764160,"available":96626839552,"internally_reserved":0,"allocated":806912,"data_stored":353659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9109504},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.85199999999999998,"15min":0.98299999999999998},"min":{"1min":0.433,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0049999999999999,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.68400000000000005,"15min":0.66600000000000004},"min":{"1min":0.56100000000000005,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.95399999999999996,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.64900000000000002}]},{"osd":1,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.92700000000000005,"15min":1.1080000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94199999999999995,"5min":9.5709999999999997,"15min":9.5709999999999997},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.83999999999999997,"15min":0.92700000000000005},"min":{"1min":0.58599999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0349999999999999,"5min":5.266,"15min":5.266},"last":0.73999999999999999}]},{"osd":2,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.91600000000000004,"15min":1.077},"min":{"1min":0.60299999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.016,"5min":8.4529999999999994,"15min":8.4529999999999994},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.88,"15min":1.002},"min":{"1min":0.53200000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.018,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.85799999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.72899999999999998,"15min":0.73299999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.79000000000000004,"15min":0.80200000000000005},"min":{"1min":0.58399999999999996,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.98499999999999999,"5min":1.389,"15min":1.389},"last":0.86899999999999999}]},{"osd":5,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.71799999999999997,"15min":0.74199999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86799999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.72299999999999998,"15min":0.72099999999999997},"min":{"1min":0.47599999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.998,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.58699999999999997}]},{"osd":6,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.77300000000000002,"15min":0.78100000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.94599999999999995,"5min":1.032,"15min":1.032},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.501,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0700000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.81899999999999995}]},{"osd":7,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.79700000000000004,"15min":0.82199999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0609999999999999,"5min":1.679,"15min":1.679},"last":0.75},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.88,"15min":0.92800000000000005},"min":{"1min":0.64300000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0069999999999999,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.90500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9616,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362224,"statfs":{"total":96636764160,"available":96626917376,"internally_reserved":0,"allocated":794624,"data_stored":351114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.65700000000000003,"15min":0.64100000000000001},"min":{"1min":0.42899999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.75700000000000001,"15min":0.74399999999999999},"min":{"1min":0.60599999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96999999999999997,"5min":2.528,"15min":2.528},"last":0.57499999999999996}]},{"osd":1,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.73699999999999999,"15min":0.745},"min":{"1min":0.66400000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.752},{"interface":"front","average":{"1min":0.753,"5min":0.67200000000000004,"15min":0.63700000000000001},"min":{"1min":0.53900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.5}]},{"osd":2,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69199999999999995,"15min":0.68300000000000005},"min":{"1min":0.52700000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.95399999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.72499999999999998,"15min":0.71599999999999997},"min":{"1min":0.59299999999999997,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.89200000000000002,"5min":1.321,"15min":1.321},"last":0.63800000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.71599999999999997,"15min":0.69499999999999995},"min":{"1min":0.55600000000000005,"5min":0.496,"15min":0.496},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.745,"15min":0.73499999999999999},"min":{"1min":0.629,"5min":0.501,"15min":0.501},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.80800000000000005}]},{"osd":4,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.51400000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.79000000000000004,"15min":0.78200000000000003},"min":{"1min":0.59899999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.82999999999999996}]},{"osd":6,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.79700000000000004},"min":{"1min":0.53000000000000003,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.94499999999999995,"5min":1.071,"15min":1.071},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.72199999999999998,"15min":0.72399999999999998},"min":{"1min":0.58999999999999997,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97099999999999997,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.77800000000000002}]},{"osd":7,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.75600000000000001,"15min":0.75700000000000001},"min":{"1min":0.55700000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.026,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81599999999999995,"15min":0.83299999999999996},"min":{"1min":0.67700000000000005,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.90500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:00:28.437 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-03-03T08:00:28.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.706+0000 7f3ee5bff700 1 Processor -- start 2023-03-03T08:00:28.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.706+0000 7f3ee5bff700 1 -- start start 2023-03-03T08:00:28.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.707+0000 7f3ee5bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3ee00b87c0 0x7f3ee00cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.707+0000 7f3ee5bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ee00b8b90 0x7f3ee00b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.707+0000 7f3ee5bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee00d0ef0 con 0x7f3ee00b5240 2023-03-03T08:00:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.707+0000 7f3ee5bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee00d0600 con 0x7f3ee00b87c0 2023-03-03T08:00:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.707+0000 7f3ee5bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee00cd970 con 0x7f3ee00b8b90 2023-03-03T08:00:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.708+0000 7f3edf7fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3ee00b87c0 0x7f3ee00cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.708+0000 7f3edeffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ee00b8b90 0x7f3ee00b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.708+0000 7f3edffff700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f3ee00b5240 0x7f3ee01605b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:44042/0 (socket says 172.21.15.98:44042) 2023-03-03T08:00:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.708+0000 7f3edffff700 1 -- 172.21.15.98:0/3353493942 learned_addr learned my addr 172.21.15.98:0/3353493942 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.709+0000 7f3ede7fc700 1 -- 172.21.15.98:0/3353493942 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520546332 0 0) 0x7f3ee00d0ef0 con 0x7f3ee00b5240 2023-03-03T08:00:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.709+0000 7f3edf7fe700 1 -- 172.21.15.98:0/3353493942 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ee00b8b90 msgr2=0x7f3ee00b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.709+0000 7f3edf7fe700 1 --2- 172.21.15.98:0/3353493942 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ee00b8b90 0x7f3ee00b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.709+0000 7f3edf7fe700 1 -- 172.21.15.98:0/3353493942 >> v1:172.21.15.100:6789/0 conn(0x7f3ee00b5240 legacy=0x7f3ee01605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.709+0000 7f3edf7fe700 1 -- 172.21.15.98:0/3353493942 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee00cdc50 con 0x7f3ee00b87c0 2023-03-03T08:00:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.709+0000 7f3edeffd700 1 --2- 172.21.15.98:0/3353493942 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ee00b8b90 0x7f3ee00b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:00:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.709+0000 7f3edf7fe700 1 --2- 172.21.15.98:0/3353493942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ee00b87c0 0x7f3ee00cffe0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009840 tx=0x7f3ed0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e026cd6ea66e22d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.709+0000 7f3ede7fc700 1 -- 172.21.15.98:0/3353493942 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0011830 con 0x7f3ee00b87c0 2023-03-03T08:00:28.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.709+0000 7f3ede7fc700 1 -- 172.21.15.98:0/3353493942 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed0011990 con 0x7f3ee00b87c0 2023-03-03T08:00:28.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.709+0000 7f3ede7fc700 1 -- 172.21.15.98:0/3353493942 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0011d20 con 0x7f3ee00b87c0 2023-03-03T08:00:28.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.710+0000 7f3ee5bff700 1 -- 172.21.15.98:0/3353493942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ee00b87c0 msgr2=0x7f3ee00cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:28.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.710+0000 7f3ee5bff700 1 --2- 172.21.15.98:0/3353493942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ee00b87c0 0x7f3ee00cffe0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009840 tx=0x7f3ed0009b50 comp rx=0 tx=0).stop 2023-03-03T08:00:28.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.710+0000 7f3ee5bff700 1 -- 172.21.15.98:0/3353493942 shutdown_connections 2023-03-03T08:00:28.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.710+0000 7f3ee5bff700 1 --2- 172.21.15.98:0/3353493942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ee00b87c0 0x7f3ee00cffe0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.710+0000 7f3ee5bff700 1 --2- 172.21.15.98:0/3353493942 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ee00b8b90 0x7f3ee00b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.710+0000 7f3ee5bff700 1 -- 172.21.15.98:0/3353493942 >> 172.21.15.98:0/3353493942 conn(0x7f3ee006aee0 msgr2=0x7f3ee006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:28.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.710+0000 7f3ee5bff700 1 -- 172.21.15.98:0/3353493942 shutdown_connections 2023-03-03T08:00:28.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.710+0000 7f3ee5bff700 1 -- 172.21.15.98:0/3353493942 wait complete. 2023-03-03T08:00:28.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.712+0000 7f3ee5bff700 1 Processor -- start 2023-03-03T08:00:28.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.712+0000 7f3ee5bff700 1 -- start start 2023-03-03T08:00:28.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.713+0000 7f3ee5bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ee00b5240 0x7f3ee015b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:28.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.713+0000 7f3ee5bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ee00b87c0 0x7f3ee015bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:28.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.713+0000 7f3ee5bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ee00b8b90 0x7f3ee015abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:28.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.713+0000 7f3ee5bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee015b0f0 con 0x7f3ee00b5240 2023-03-03T08:00:28.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.713+0000 7f3ee5bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee015f360 con 0x7f3ee00b87c0 2023-03-03T08:00:28.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.713+0000 7f3ee5bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee015bb00 con 0x7f3ee00b8b90 2023-03-03T08:00:28.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.714+0000 7f3edf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ee00b5240 0x7f3ee015b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:28.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.714+0000 7f3edf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ee00b5240 0x7f3ee015b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44988/0 (socket says 172.21.15.98:44988) 2023-03-03T08:00:28.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.714+0000 7f3edf7fe700 1 -- 172.21.15.98:0/4226137973 learned_addr learned my addr 172.21.15.98:0/4226137973 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:28.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.714+0000 7f3edeffd700 1 --2- 172.21.15.98:0/4226137973 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ee00b87c0 0x7f3ee015bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:28.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.714+0000 7f3edffff700 1 --2- 172.21.15.98:0/4226137973 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ee00b8b90 0x7f3ee015abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:28.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.714+0000 7f3edf7fe700 1 -- 172.21.15.98:0/4226137973 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ee00b8b90 msgr2=0x7f3ee015abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:28.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.714+0000 7f3edf7fe700 1 --2- 172.21.15.98:0/4226137973 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ee00b8b90 0x7f3ee015abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.714+0000 7f3edf7fe700 1 -- 172.21.15.98:0/4226137973 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ee00b87c0 msgr2=0x7f3ee015bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:28.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.714+0000 7f3edf7fe700 1 --2- 172.21.15.98:0/4226137973 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ee00b87c0 0x7f3ee015bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.714+0000 7f3edf7fe700 1 -- 172.21.15.98:0/4226137973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed40094a0 con 0x7f3ee00b5240 2023-03-03T08:00:28.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.714+0000 7f3edf7fe700 1 --2- 172.21.15.98:0/4226137973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ee00b5240 0x7f3ee015b2b0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009fd0 tx=0x7f3ed0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:28.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.715+0000 7f3ec7fff700 1 -- 172.21.15.98:0/4226137973 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed00038a0 con 0x7f3ee00b5240 2023-03-03T08:00:28.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.715+0000 7f3ec7fff700 1 -- 172.21.15.98:0/4226137973 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed001f070 con 0x7f3ee00b5240 2023-03-03T08:00:28.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.715+0000 7f3ec7fff700 1 -- 172.21.15.98:0/4226137973 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0003c40 con 0x7f3ee00b5240 2023-03-03T08:00:28.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.715+0000 7f3ee5bff700 1 -- 172.21.15.98:0/4226137973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed00094a0 con 0x7f3ee00b5240 2023-03-03T08:00:28.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.715+0000 7f3ee5bff700 1 -- 172.21.15.98:0/4226137973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee00cdf30 con 0x7f3ee00b5240 2023-03-03T08:00:28.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.716+0000 7f3ee5bff700 1 -- 172.21.15.98:0/4226137973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee00ce0c0 con 0x7f3ee00b5240 2023-03-03T08:00:28.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.717+0000 7f3ec7fff700 1 -- 172.21.15.98:0/4226137973 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f3ed000f050 con 0x7f3ee00b5240 2023-03-03T08:00:28.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.718+0000 7f3ec7fff700 1 --2- 172.21.15.98:0/4226137973 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3ecc0683c0 0x7f3ecc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:28.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.718+0000 7f3ec7fff700 1 -- 172.21.15.98:0/4226137973 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3ed0021070 con 0x7f3ee00b5240 2023-03-03T08:00:28.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.719+0000 7f3edeffd700 1 --2- 172.21.15.98:0/4226137973 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3ecc0683c0 0x7f3ecc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:28.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.719+0000 7f3edeffd700 1 --2- 172.21.15.98:0/4226137973 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3ecc0683c0 0x7f3ecc06a870 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3ee00d0350 tx=0x7f3ed4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:28.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.722+0000 7f3ec7fff700 1 -- 172.21.15.98:0/4226137973 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3ed0053f20 con 0x7f3ee00b5240 2023-03-03T08:00:28.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.990+0000 7f3ee5bff700 1 -- 172.21.15.98:0/4226137973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f3ee00729b0 con 0x7f3ee00b5240 2023-03-03T08:00:28.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.990+0000 7f3ec7fff700 1 -- 172.21.15.98:0/4226137973 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1013 (secure 0 0 0) 0x7f3ed0015410 con 0x7f3ee00b5240 2023-03-03T08:00:28.992 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:28.992 INFO:teuthology.orchestra.run.smithi098.stdout:{"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":311,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":66,"num_osds":8,"num_up_osds":8,"osd_up_since":1677830131,"num_in_osds":8,"osd_in_since":1677830121,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":105}],"num_pgs":105,"num_pools":4,"num_objects":24,"data_bytes":592698,"bytes_used":81256448,"bytes_avail":773012856832,"bytes_total":773094113280,"write_bytes_sec":2214,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":31,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"c","status":"up:active","gid":6387}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2023-03-03T07:59:18.224736+0000","services":{}},"progress_events":{}} 2023-03-03T08:00:28.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.992+0000 7f3ee5bff700 1 -- 172.21.15.98:0/4226137973 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3ecc0683c0 msgr2=0x7f3ecc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:28.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.992+0000 7f3ee5bff700 1 --2- 172.21.15.98:0/4226137973 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3ecc0683c0 0x7f3ecc06a870 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3ee00d0350 tx=0x7f3ed4008040 comp rx=0 tx=0).stop 2023-03-03T08:00:28.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.992+0000 7f3ee5bff700 1 -- 172.21.15.98:0/4226137973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ee00b5240 msgr2=0x7f3ee015b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:28.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.993+0000 7f3ee5bff700 1 --2- 172.21.15.98:0/4226137973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ee00b5240 0x7f3ee015b2b0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009fd0 tx=0x7f3ed0004750 comp rx=0 tx=0).stop 2023-03-03T08:00:28.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.993+0000 7f3ee5bff700 1 -- 172.21.15.98:0/4226137973 shutdown_connections 2023-03-03T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.993+0000 7f3ee5bff700 1 --2- 172.21.15.98:0/4226137973 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3ecc0683c0 0x7f3ecc06a870 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.993+0000 7f3ee5bff700 1 --2- 172.21.15.98:0/4226137973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ee00b5240 0x7f3ee015b2b0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.993+0000 7f3ee5bff700 1 --2- 172.21.15.98:0/4226137973 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ee00b8b90 0x7f3ee015abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.993+0000 7f3ee5bff700 1 --2- 172.21.15.98:0/4226137973 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ee00b87c0 0x7f3ee015bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.993+0000 7f3ee5bff700 1 -- 172.21.15.98:0/4226137973 >> 172.21.15.98:0/4226137973 conn(0x7f3ee006aee0 msgr2=0x7f3ee00b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.993+0000 7f3ee5bff700 1 -- 172.21.15.98:0/4226137973 shutdown_connections 2023-03-03T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:28.993+0000 7f3ee5bff700 1 -- 172.21.15.98:0/4226137973 wait complete. 2023-03-03T08:00:29.021 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-03-03T08:00:29.021 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:00:29.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.266+0000 7f5baf9ff700 1 Processor -- start 2023-03-03T08:00:29.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.267+0000 7f5baf9ff700 1 -- start start 2023-03-03T08:00:29.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.268+0000 7f5baf9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ba80b8b90 0x7f5ba80cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:29.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.268+0000 7f5baf9ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba80d0ef0 con 0x7f5ba80b87c0 2023-03-03T08:00:29.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.268+0000 7f5baf9ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba80d0600 con 0x7f5ba80b5410 2023-03-03T08:00:29.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.268+0000 7f5baf9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba80cd970 con 0x7f5ba80b8b90 2023-03-03T08:00:29.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.269+0000 7f5bad79b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5ba80b87c0 0x7f5ba80b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44876/0 (socket says 172.21.15.98:44876) 2023-03-03T08:00:29.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.269+0000 7f5bad79b700 1 -- 172.21.15.98:0/2688724369 learned_addr learned my addr 172.21.15.98:0/2688724369 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:29.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.269+0000 7f5bacf9a700 1 --2- 172.21.15.98:0/2688724369 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ba80b8b90 0x7f5ba80cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.269+0000 7f5b9ffff700 1 -- 172.21.15.98:0/2688724369 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885590264 0 0) 0x7f5ba80d0ef0 con 0x7f5ba80b87c0 2023-03-03T08:00:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.269+0000 7f5b9ffff700 1 -- 172.21.15.98:0/2688724369 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b940035b0 con 0x7f5ba80b87c0 2023-03-03T08:00:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.269+0000 7f5b9ffff700 1 -- 172.21.15.98:0/2688724369 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312194614 0 0) 0x7f5ba80d0600 con 0x7f5ba80b5410 2023-03-03T08:00:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.269+0000 7f5b9ffff700 1 -- 172.21.15.98:0/2688724369 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba80d0ef0 con 0x7f5ba80b5410 2023-03-03T08:00:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.269+0000 7f5b9ffff700 1 -- 172.21.15.98:0/2688724369 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 860329481 0 0) 0x7f5b940035b0 con 0x7f5ba80b87c0 2023-03-03T08:00:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.269+0000 7f5bacf9a700 1 -- 172.21.15.98:0/2688724369 >> v1:172.21.15.100:6789/0 conn(0x7f5ba80b5410 legacy=0x7f5ba80d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.269+0000 7f5bacf9a700 1 -- 172.21.15.98:0/2688724369 >> v1:172.21.15.98:6789/0 conn(0x7f5ba80b87c0 legacy=0x7f5ba80b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.270+0000 7f5bacf9a700 1 -- 172.21.15.98:0/2688724369 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba80cdc50 con 0x7f5ba80b8b90 2023-03-03T08:00:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.270+0000 7f5bacf9a700 1 --2- 172.21.15.98:0/2688724369 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ba80b8b90 0x7f5ba80cffe0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5b98009870 tx=0x7f5b98009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9944117aa2a16750 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.270+0000 7f5b9ffff700 1 -- 172.21.15.98:0/2688724369 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba4001670 con 0x7f5ba80b8b90 2023-03-03T08:00:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.270+0000 7f5b9ffff700 1 -- 172.21.15.98:0/2688724369 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b98004030 con 0x7f5ba80b8b90 2023-03-03T08:00:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.270+0000 7f5b9ffff700 1 -- 172.21.15.98:0/2688724369 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b980043c0 con 0x7f5ba80b8b90 2023-03-03T08:00:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.271+0000 7f5baf9ff700 1 -- 172.21.15.98:0/2688724369 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ba80b8b90 msgr2=0x7f5ba80cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.271+0000 7f5baf9ff700 1 --2- 172.21.15.98:0/2688724369 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ba80b8b90 0x7f5ba80cffe0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5b98009870 tx=0x7f5b98009b80 comp rx=0 tx=0).stop 2023-03-03T08:00:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.271+0000 7f5baf9ff700 1 -- 172.21.15.98:0/2688724369 shutdown_connections 2023-03-03T08:00:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.271+0000 7f5baf9ff700 1 --2- 172.21.15.98:0/2688724369 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ba80b8b90 0x7f5ba80cffe0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.271+0000 7f5baf9ff700 1 -- 172.21.15.98:0/2688724369 >> 172.21.15.98:0/2688724369 conn(0x7f5ba806aee0 msgr2=0x7f5ba806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.271+0000 7f5baf9ff700 1 -- 172.21.15.98:0/2688724369 shutdown_connections 2023-03-03T08:00:29.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.271+0000 7f5baf9ff700 1 -- 172.21.15.98:0/2688724369 wait complete. 2023-03-03T08:00:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.273+0000 7f5baf9ff700 1 Processor -- start 2023-03-03T08:00:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.273+0000 7f5baf9ff700 1 -- start start 2023-03-03T08:00:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.274+0000 7f5baf9ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ba80b5410 0x7f5ba80ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.274+0000 7f5baf9ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ba80b87c0 0x7f5ba80cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.274+0000 7f5baf9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ba80b8b90 0x7f5ba81f53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.274+0000 7f5baf9ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba80cba40 con 0x7f5ba80b87c0 2023-03-03T08:00:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.274+0000 7f5baf9ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba80cb5b0 con 0x7f5ba80b5410 2023-03-03T08:00:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.274+0000 7f5baf9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba80ca9a0 con 0x7f5ba80b8b90 2023-03-03T08:00:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.274+0000 7f5bacf9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ba80b87c0 0x7f5ba80cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.274+0000 7f5bacf9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ba80b87c0 0x7f5ba80cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45004/0 (socket says 172.21.15.98:45004) 2023-03-03T08:00:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.274+0000 7f5bacf9a700 1 -- 172.21.15.98:0/3957050350 learned_addr learned my addr 172.21.15.98:0/3957050350 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.275+0000 7f5bacf9a700 1 -- 172.21.15.98:0/3957050350 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ba80b8b90 msgr2=0x7f5ba81f53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.275+0000 7f5bacf9a700 1 --2- 172.21.15.98:0/3957050350 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ba80b8b90 0x7f5ba81f53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.275+0000 7f5bacf9a700 1 -- 172.21.15.98:0/3957050350 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ba80b5410 msgr2=0x7f5ba80ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.275+0000 7f5bacf9a700 1 --2- 172.21.15.98:0/3957050350 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ba80b5410 0x7f5ba80ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.275+0000 7f5bacf9a700 1 -- 172.21.15.98:0/3957050350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b980094a0 con 0x7f5ba80b87c0 2023-03-03T08:00:29.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.275+0000 7f5bacf9a700 1 --2- 172.21.15.98:0/3957050350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ba80b87c0 0x7f5ba80cd160 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f5b98000c00 tx=0x7f5b98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:29.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.275+0000 7f5b9dffb700 1 -- 172.21.15.98:0/3957050350 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b98011630 con 0x7f5ba80b87c0 2023-03-03T08:00:29.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.275+0000 7f5b9dffb700 1 -- 172.21.15.98:0/3957050350 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9801f070 con 0x7f5ba80b87c0 2023-03-03T08:00:29.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.275+0000 7f5b9dffb700 1 -- 172.21.15.98:0/3957050350 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b980119f0 con 0x7f5ba80b87c0 2023-03-03T08:00:29.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.275+0000 7f5baf9ff700 1 -- 172.21.15.98:0/3957050350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba80cdf30 con 0x7f5ba80b87c0 2023-03-03T08:00:29.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.276+0000 7f5baf9ff700 1 -- 172.21.15.98:0/3957050350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba8073a80 con 0x7f5ba80b87c0 2023-03-03T08:00:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.277+0000 7f5baf9ff700 1 -- 172.21.15.98:0/3957050350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba80ce0c0 con 0x7f5ba80b87c0 2023-03-03T08:00:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.278+0000 7f5b9dffb700 1 -- 172.21.15.98:0/3957050350 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f5b98017020 con 0x7f5ba80b87c0 2023-03-03T08:00:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.279+0000 7f5b9dffb700 1 --2- 172.21.15.98:0/3957050350 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5b8c0683c0 0x7f5b8c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.279+0000 7f5b9dffb700 1 -- 172.21.15.98:0/3957050350 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5b98021070 con 0x7f5ba80b87c0 2023-03-03T08:00:29.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.279+0000 7f5bad79b700 1 --2- 172.21.15.98:0/3957050350 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5b8c0683c0 0x7f5b8c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:29.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.280+0000 7f5bad79b700 1 --2- 172.21.15.98:0/3957050350 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5b8c0683c0 0x7f5b8c06a870 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4001380 tx=0x7f5ba4001130 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:29.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.283+0000 7f5b9dffb700 1 -- 172.21.15.98:0/3957050350 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5b98053ec0 con 0x7f5ba80b87c0 2023-03-03T08:00:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.472+0000 7f5baf9ff700 1 -- 172.21.15.98:0/3957050350 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5ba80729b0 con 0x7f5b8c0683c0 2023-03-03T08:00:29.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.481+0000 7f5b9dffb700 1 -- 172.21.15.98:0/3957050350 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310729 (secure 0 0 0) 0x7f5ba80729b0 con 0x7f5b8c0683c0 2023-03-03T08:00:29.486 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:29.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.486+0000 7f5baf9ff700 1 -- 172.21.15.98:0/3957050350 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5b8c0683c0 msgr2=0x7f5b8c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:29.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.486+0000 7f5baf9ff700 1 --2- 172.21.15.98:0/3957050350 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5b8c0683c0 0x7f5b8c06a870 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4001380 tx=0x7f5ba4001130 comp rx=0 tx=0).stop 2023-03-03T08:00:29.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.486+0000 7f5baf9ff700 1 -- 172.21.15.98:0/3957050350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ba80b87c0 msgr2=0x7f5ba80cd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:29.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.486+0000 7f5baf9ff700 1 --2- 172.21.15.98:0/3957050350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ba80b87c0 0x7f5ba80cd160 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f5b98000c00 tx=0x7f5b98004750 comp rx=0 tx=0).stop 2023-03-03T08:00:29.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.486+0000 7f5baf9ff700 1 -- 172.21.15.98:0/3957050350 shutdown_connections 2023-03-03T08:00:29.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.486+0000 7f5baf9ff700 1 --2- 172.21.15.98:0/3957050350 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5b8c0683c0 0x7f5b8c06a870 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:29.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.486+0000 7f5baf9ff700 1 --2- 172.21.15.98:0/3957050350 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ba80b8b90 0x7f5ba81f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:29.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.486+0000 7f5baf9ff700 1 --2- 172.21.15.98:0/3957050350 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ba80b5410 0x7f5ba80ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:29.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.486+0000 7f5baf9ff700 1 --2- 172.21.15.98:0/3957050350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ba80b87c0 0x7f5ba80cd160 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:29.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.486+0000 7f5baf9ff700 1 -- 172.21.15.98:0/3957050350 >> 172.21.15.98:0/3957050350 conn(0x7f5ba806aee0 msgr2=0x7f5ba80d22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:29.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.486+0000 7f5baf9ff700 1 -- 172.21.15.98:0/3957050350 shutdown_connections 2023-03-03T08:00:29.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:29.486+0000 7f5baf9ff700 1 -- 172.21.15.98:0/3957050350 wait complete. 2023-03-03T08:00:29.490 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T08:00:29.515 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2023-03-03T08:00:28.241842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":79352,"kb_used_data":7416,"kb_used_omap":0,"kb_used_meta":71872,"kb_avail":754895368,"statfs":{"total":773094113280,"available":773012856832,"internally_reserved":0,"allocated":7593984,"data_stored":3998208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":73596928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.474170"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356978+0000","last_change":"2023-03-03T08:00:22.351816+0000","last_active":"2023-03-03T08:00:22.356978+0000","last_peered":"2023-03-03T08:00:22.356978+0000","last_clean":"2023-03-03T08:00:22.356978+0000","last_became_active":"2023-03-03T08:00:22.351560+0000","last_became_peered":"2023-03-03T08:00:22.351560+0000","last_unstale":"2023-03-03T08:00:22.356978+0000","last_undegraded":"2023-03-03T08:00:22.356978+0000","last_fullsized":"2023-03-03T08:00:22.356978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:22:36.090135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.355977+0000","last_change":"2023-03-03T08:00:22.349135+0000","last_active":"2023-03-03T08:00:22.355977+0000","last_peered":"2023-03-03T08:00:22.355977+0000","last_clean":"2023-03-03T08:00:22.355977+0000","last_became_active":"2023-03-03T08:00:22.348832+0000","last_became_peered":"2023-03-03T08:00:22.348832+0000","last_unstale":"2023-03-03T08:00:22.355977+0000","last_undegraded":"2023-03-03T08:00:22.355977+0000","last_fullsized":"2023-03-03T08:00:22.355977+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:45:30.839878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.343799+0000","last_change":"2023-03-03T08:00:22.341359+0000","last_active":"2023-03-03T08:00:22.343799+0000","last_peered":"2023-03-03T08:00:22.343799+0000","last_clean":"2023-03-03T08:00:22.343799+0000","last_became_active":"2023-03-03T08:00:22.341054+0000","last_became_peered":"2023-03-03T08:00:22.341054+0000","last_unstale":"2023-03-03T08:00:22.343799+0000","last_undegraded":"2023-03-03T08:00:22.343799+0000","last_fullsized":"2023-03-03T08:00:22.343799+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:57:44.236333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368309+0000","last_change":"2023-03-03T08:00:22.356433+0000","last_active":"2023-03-03T08:00:22.368309+0000","last_peered":"2023-03-03T08:00:22.368309+0000","last_clean":"2023-03-03T08:00:22.368309+0000","last_became_active":"2023-03-03T08:00:22.356154+0000","last_became_peered":"2023-03-03T08:00:22.356154+0000","last_unstale":"2023-03-03T08:00:22.368309+0000","last_undegraded":"2023-03-03T08:00:22.368309+0000","last_fullsized":"2023-03-03T08:00:22.368309+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:39.487708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.366920+0000","last_change":"2023-03-03T08:00:22.365431+0000","last_active":"2023-03-03T08:00:22.366920+0000","last_peered":"2023-03-03T08:00:22.366920+0000","last_clean":"2023-03-03T08:00:22.366920+0000","last_became_active":"2023-03-03T08:00:22.364882+0000","last_became_peered":"2023-03-03T08:00:22.364882+0000","last_unstale":"2023-03-03T08:00:22.366920+0000","last_undegraded":"2023-03-03T08:00:22.366920+0000","last_fullsized":"2023-03-03T08:00:22.366920+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:16:26.224161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.347929+0000","last_change":"2023-03-03T08:00:22.344899+0000","last_active":"2023-03-03T08:00:22.347929+0000","last_peered":"2023-03-03T08:00:22.347929+0000","last_clean":"2023-03-03T08:00:22.347929+0000","last_became_active":"2023-03-03T08:00:22.344558+0000","last_became_peered":"2023-03-03T08:00:22.344558+0000","last_unstale":"2023-03-03T08:00:22.347929+0000","last_undegraded":"2023-03-03T08:00:22.347929+0000","last_fullsized":"2023-03-03T08:00:22.347929+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:34.298493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.362506+0000","last_change":"2023-03-03T08:00:22.360864+0000","last_active":"2023-03-03T08:00:22.362506+0000","last_peered":"2023-03-03T08:00:22.362506+0000","last_clean":"2023-03-03T08:00:22.362506+0000","last_became_active":"2023-03-03T08:00:22.360550+0000","last_became_peered":"2023-03-03T08:00:22.360550+0000","last_unstale":"2023-03-03T08:00:22.362506+0000","last_undegraded":"2023-03-03T08:00:22.362506+0000","last_fullsized":"2023-03-03T08:00:22.362506+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:00:54.378703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.349837+0000","last_change":"2023-03-03T08:00:22.346751+0000","last_active":"2023-03-03T08:00:22.349837+0000","last_peered":"2023-03-03T08:00:22.349837+0000","last_clean":"2023-03-03T08:00:22.349837+0000","last_became_active":"2023-03-03T08:00:22.346388+0000","last_became_peered":"2023-03-03T08:00:22.346388+0000","last_unstale":"2023-03-03T08:00:22.349837+0000","last_undegraded":"2023-03-03T08:00:22.349837+0000","last_fullsized":"2023-03-03T08:00:22.349837+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:40.317030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356366+0000","last_change":"2023-03-03T08:00:22.347590+0000","last_active":"2023-03-03T08:00:22.356366+0000","last_peered":"2023-03-03T08:00:22.356366+0000","last_clean":"2023-03-03T08:00:22.356366+0000","last_became_active":"2023-03-03T08:00:22.347256+0000","last_became_peered":"2023-03-03T08:00:22.347256+0000","last_unstale":"2023-03-03T08:00:22.356366+0000","last_undegraded":"2023-03-03T08:00:22.356366+0000","last_fullsized":"2023-03-03T08:00:22.356366+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:23:14.650660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.359341+0000","last_change":"2023-03-03T08:00:22.352944+0000","last_active":"2023-03-03T08:00:22.359341+0000","last_peered":"2023-03-03T08:00:22.359341+0000","last_clean":"2023-03-03T08:00:22.359341+0000","last_became_active":"2023-03-03T08:00:22.352547+0000","last_became_peered":"2023-03-03T08:00:22.352547+0000","last_unstale":"2023-03-03T08:00:22.359341+0000","last_undegraded":"2023-03-03T08:00:22.359341+0000","last_fullsized":"2023-03-03T08:00:22.359341+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:53:16.100003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368088+0000","last_change":"2023-03-03T08:00:22.358102+0000","last_active":"2023-03-03T08:00:22.368088+0000","last_peered":"2023-03-03T08:00:22.368088+0000","last_clean":"2023-03-03T08:00:22.368088+0000","last_became_active":"2023-03-03T08:00:22.357726+0000","last_became_peered":"2023-03-03T08:00:22.357726+0000","last_unstale":"2023-03-03T08:00:22.368088+0000","last_undegraded":"2023-03-03T08:00:22.368088+0000","last_fullsized":"2023-03-03T08:00:22.368088+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:19:01.442408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.349788+0000","last_change":"2023-03-03T08:00:22.346586+0000","last_active":"2023-03-03T08:00:22.349788+0000","last_peered":"2023-03-03T08:00:22.349788+0000","last_clean":"2023-03-03T08:00:22.349788+0000","last_became_active":"2023-03-03T08:00:22.346252+0000","last_became_peered":"2023-03-03T08:00:22.346252+0000","last_unstale":"2023-03-03T08:00:22.349788+0000","last_undegraded":"2023-03-03T08:00:22.349788+0000","last_fullsized":"2023-03-03T08:00:22.349788+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:44:04.177495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.352213+0000","last_change":"2023-03-03T08:00:22.347364+0000","last_active":"2023-03-03T08:00:22.352213+0000","last_peered":"2023-03-03T08:00:22.352213+0000","last_clean":"2023-03-03T08:00:22.352213+0000","last_became_active":"2023-03-03T08:00:22.347036+0000","last_became_peered":"2023-03-03T08:00:22.347036+0000","last_unstale":"2023-03-03T08:00:22.352213+0000","last_undegraded":"2023-03-03T08:00:22.352213+0000","last_fullsized":"2023-03-03T08:00:22.352213+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:20.721006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.352093+0000","last_change":"2023-03-03T08:00:22.342081+0000","last_active":"2023-03-03T08:00:22.352093+0000","last_peered":"2023-03-03T08:00:22.352093+0000","last_clean":"2023-03-03T08:00:22.352093+0000","last_became_active":"2023-03-03T08:00:22.341783+0000","last_became_peered":"2023-03-03T08:00:22.341783+0000","last_unstale":"2023-03-03T08:00:22.352093+0000","last_undegraded":"2023-03-03T08:00:22.352093+0000","last_fullsized":"2023-03-03T08:00:22.352093+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:20.721006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.361028+0000","last_change":"2023-03-03T08:00:22.354119+0000","last_active":"2023-03-03T08:00:22.361028+0000","last_peered":"2023-03-03T08:00:22.361028+0000","last_clean":"2023-03-03T08:00:22.361028+0000","last_became_active":"2023-03-03T08:00:22.353824+0000","last_became_peered":"2023-03-03T08:00:22.353824+0000","last_unstale":"2023-03-03T08:00:22.361028+0000","last_undegraded":"2023-03-03T08:00:22.361028+0000","last_fullsized":"2023-03-03T08:00:22.361028+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:27.867451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.354242+0000","last_change":"2023-03-03T08:00:22.350728+0000","last_active":"2023-03-03T08:00:22.354242+0000","last_peered":"2023-03-03T08:00:22.354242+0000","last_clean":"2023-03-03T08:00:22.354242+0000","last_became_active":"2023-03-03T08:00:22.350468+0000","last_became_peered":"2023-03-03T08:00:22.350468+0000","last_unstale":"2023-03-03T08:00:22.354242+0000","last_undegraded":"2023-03-03T08:00:22.354242+0000","last_fullsized":"2023-03-03T08:00:22.354242+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:36:10.563980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.361036+0000","last_change":"2023-03-03T08:00:22.356264+0000","last_active":"2023-03-03T08:00:22.361036+0000","last_peered":"2023-03-03T08:00:22.361036+0000","last_clean":"2023-03-03T08:00:22.361036+0000","last_became_active":"2023-03-03T08:00:22.355666+0000","last_became_peered":"2023-03-03T08:00:22.355666+0000","last_unstale":"2023-03-03T08:00:22.361036+0000","last_undegraded":"2023-03-03T08:00:22.361036+0000","last_fullsized":"2023-03-03T08:00:22.361036+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:03.239405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.367982+0000","last_change":"2023-03-03T08:00:22.357934+0000","last_active":"2023-03-03T08:00:22.367982+0000","last_peered":"2023-03-03T08:00:22.367982+0000","last_clean":"2023-03-03T08:00:22.367982+0000","last_became_active":"2023-03-03T08:00:22.357644+0000","last_became_peered":"2023-03-03T08:00:22.357644+0000","last_unstale":"2023-03-03T08:00:22.367982+0000","last_undegraded":"2023-03-03T08:00:22.367982+0000","last_fullsized":"2023-03-03T08:00:22.367982+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:44:04.177495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.365907+0000","last_change":"2023-03-03T08:00:22.363152+0000","last_active":"2023-03-03T08:00:22.365907+0000","last_peered":"2023-03-03T08:00:22.365907+0000","last_clean":"2023-03-03T08:00:22.365907+0000","last_became_active":"2023-03-03T08:00:22.362838+0000","last_became_peered":"2023-03-03T08:00:22.362838+0000","last_unstale":"2023-03-03T08:00:22.365907+0000","last_undegraded":"2023-03-03T08:00:22.365907+0000","last_fullsized":"2023-03-03T08:00:22.365907+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:35:40.983679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.370386+0000","last_change":"2023-03-03T08:00:22.365139+0000","last_active":"2023-03-03T08:00:22.370386+0000","last_peered":"2023-03-03T08:00:22.370386+0000","last_clean":"2023-03-03T08:00:22.370386+0000","last_became_active":"2023-03-03T08:00:22.364818+0000","last_became_peered":"2023-03-03T08:00:22.364818+0000","last_unstale":"2023-03-03T08:00:22.370386+0000","last_undegraded":"2023-03-03T08:00:22.370386+0000","last_fullsized":"2023-03-03T08:00:22.370386+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:00:39.446569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360521+0000","last_change":"2023-03-03T08:00:22.352070+0000","last_active":"2023-03-03T08:00:22.360521+0000","last_peered":"2023-03-03T08:00:22.360521+0000","last_clean":"2023-03-03T08:00:22.360521+0000","last_became_active":"2023-03-03T08:00:22.351637+0000","last_became_peered":"2023-03-03T08:00:22.351637+0000","last_unstale":"2023-03-03T08:00:22.360521+0000","last_undegraded":"2023-03-03T08:00:22.360521+0000","last_fullsized":"2023-03-03T08:00:22.360521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:19:01.442408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.352202+0000","last_change":"2023-03-03T08:00:22.348419+0000","last_active":"2023-03-03T08:00:22.352202+0000","last_peered":"2023-03-03T08:00:22.352202+0000","last_clean":"2023-03-03T08:00:22.352202+0000","last_became_active":"2023-03-03T08:00:22.348168+0000","last_became_peered":"2023-03-03T08:00:22.348168+0000","last_unstale":"2023-03-03T08:00:22.352202+0000","last_undegraded":"2023-03-03T08:00:22.352202+0000","last_fullsized":"2023-03-03T08:00:22.352202+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:52:14.400121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.346307+0000","last_change":"2023-03-03T08:00:22.343763+0000","last_active":"2023-03-03T08:00:22.346307+0000","last_peered":"2023-03-03T08:00:22.346307+0000","last_clean":"2023-03-03T08:00:22.346307+0000","last_became_active":"2023-03-03T08:00:22.343407+0000","last_became_peered":"2023-03-03T08:00:22.343407+0000","last_unstale":"2023-03-03T08:00:22.346307+0000","last_undegraded":"2023-03-03T08:00:22.346307+0000","last_fullsized":"2023-03-03T08:00:22.346307+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:51:43.370212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.343549+0000","last_change":"2023-03-03T08:00:22.340970+0000","last_active":"2023-03-03T08:00:22.343549+0000","last_peered":"2023-03-03T08:00:22.343549+0000","last_clean":"2023-03-03T08:00:22.343549+0000","last_became_active":"2023-03-03T08:00:22.340635+0000","last_became_peered":"2023-03-03T08:00:22.340635+0000","last_unstale":"2023-03-03T08:00:22.343549+0000","last_undegraded":"2023-03-03T08:00:22.343549+0000","last_fullsized":"2023-03-03T08:00:22.343549+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:06.529333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.361178+0000","last_change":"2023-03-03T08:00:22.355393+0000","last_active":"2023-03-03T08:00:22.361178+0000","last_peered":"2023-03-03T08:00:22.361178+0000","last_clean":"2023-03-03T08:00:22.361178+0000","last_became_active":"2023-03-03T08:00:22.354452+0000","last_became_peered":"2023-03-03T08:00:22.354452+0000","last_unstale":"2023-03-03T08:00:22.361178+0000","last_undegraded":"2023-03-03T08:00:22.361178+0000","last_fullsized":"2023-03-03T08:00:22.361178+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:08:44.952506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.359810+0000","last_change":"2023-03-03T08:00:22.350155+0000","last_active":"2023-03-03T08:00:22.359810+0000","last_peered":"2023-03-03T08:00:22.359810+0000","last_clean":"2023-03-03T08:00:22.359810+0000","last_became_active":"2023-03-03T08:00:22.349712+0000","last_became_peered":"2023-03-03T08:00:22.349712+0000","last_unstale":"2023-03-03T08:00:22.359810+0000","last_undegraded":"2023-03-03T08:00:22.359810+0000","last_fullsized":"2023-03-03T08:00:22.359810+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:06:54.326996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.362401+0000","last_change":"2023-03-03T08:00:22.352054+0000","last_active":"2023-03-03T08:00:22.362401+0000","last_peered":"2023-03-03T08:00:22.362401+0000","last_clean":"2023-03-03T08:00:22.362401+0000","last_became_active":"2023-03-03T08:00:22.351777+0000","last_became_peered":"2023-03-03T08:00:22.351777+0000","last_unstale":"2023-03-03T08:00:22.362401+0000","last_undegraded":"2023-03-03T08:00:22.362401+0000","last_fullsized":"2023-03-03T08:00:22.362401+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:06.529333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360855+0000","last_change":"2023-03-03T08:00:22.358047+0000","last_active":"2023-03-03T08:00:22.360855+0000","last_peered":"2023-03-03T08:00:22.360855+0000","last_clean":"2023-03-03T08:00:22.360855+0000","last_became_active":"2023-03-03T08:00:22.357730+0000","last_became_peered":"2023-03-03T08:00:22.357730+0000","last_unstale":"2023-03-03T08:00:22.360855+0000","last_undegraded":"2023-03-03T08:00:22.360855+0000","last_fullsized":"2023-03-03T08:00:22.360855+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:53:16.425693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356826+0000","last_change":"2023-03-03T08:00:22.347292+0000","last_active":"2023-03-03T08:00:22.356826+0000","last_peered":"2023-03-03T08:00:22.356826+0000","last_clean":"2023-03-03T08:00:22.356826+0000","last_became_active":"2023-03-03T08:00:22.346956+0000","last_became_peered":"2023-03-03T08:00:22.346956+0000","last_unstale":"2023-03-03T08:00:22.356826+0000","last_undegraded":"2023-03-03T08:00:22.356826+0000","last_fullsized":"2023-03-03T08:00:22.356826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:43.689368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.350844+0000","last_change":"2023-03-03T08:00:22.346466+0000","last_active":"2023-03-03T08:00:22.350844+0000","last_peered":"2023-03-03T08:00:22.350844+0000","last_clean":"2023-03-03T08:00:22.350844+0000","last_became_active":"2023-03-03T08:00:22.346127+0000","last_became_peered":"2023-03-03T08:00:22.346127+0000","last_unstale":"2023-03-03T08:00:22.350844+0000","last_undegraded":"2023-03-03T08:00:22.350844+0000","last_fullsized":"2023-03-03T08:00:22.350844+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:06.529333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368067+0000","last_change":"2023-03-03T08:00:22.355948+0000","last_active":"2023-03-03T08:00:22.368067+0000","last_peered":"2023-03-03T08:00:22.368067+0000","last_clean":"2023-03-03T08:00:22.368067+0000","last_became_active":"2023-03-03T08:00:22.355644+0000","last_became_peered":"2023-03-03T08:00:22.355644+0000","last_unstale":"2023-03-03T08:00:22.368067+0000","last_undegraded":"2023-03-03T08:00:22.368067+0000","last_fullsized":"2023-03-03T08:00:22.368067+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:40.317030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.361441+0000","last_change":"2023-03-03T08:00:22.352801+0000","last_active":"2023-03-03T08:00:22.361441+0000","last_peered":"2023-03-03T08:00:22.361441+0000","last_clean":"2023-03-03T08:00:22.361441+0000","last_became_active":"2023-03-03T08:00:22.352383+0000","last_became_peered":"2023-03-03T08:00:22.352383+0000","last_unstale":"2023-03-03T08:00:22.361441+0000","last_undegraded":"2023-03-03T08:00:22.361441+0000","last_fullsized":"2023-03-03T08:00:22.361441+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:28:15.779229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.364255+0000","last_change":"2023-03-03T08:00:22.357842+0000","last_active":"2023-03-03T08:00:22.364255+0000","last_peered":"2023-03-03T08:00:22.364255+0000","last_clean":"2023-03-03T08:00:22.364255+0000","last_became_active":"2023-03-03T08:00:22.357562+0000","last_became_peered":"2023-03-03T08:00:22.357562+0000","last_unstale":"2023-03-03T08:00:22.364255+0000","last_undegraded":"2023-03-03T08:00:22.364255+0000","last_fullsized":"2023-03-03T08:00:22.364255+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:25:55.351299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"65'4","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360098+0000","last_change":"2023-03-03T08:00:21.791637+0000","last_active":"2023-03-03T08:00:22.360098+0000","last_peered":"2023-03-03T08:00:22.360098+0000","last_clean":"2023-03-03T08:00:22.360098+0000","last_became_active":"2023-03-03T08:00:21.791370+0000","last_became_peered":"2023-03-03T08:00:21.791370+0000","last_unstale":"2023-03-03T08:00:22.360098+0000","last_undegraded":"2023-03-03T08:00:22.360098+0000","last_fullsized":"2023-03-03T08:00:22.360098+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:53:37.291520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368899+0000","last_change":"2023-03-03T08:00:22.363157+0000","last_active":"2023-03-03T08:00:22.368899+0000","last_peered":"2023-03-03T08:00:22.368899+0000","last_clean":"2023-03-03T08:00:22.368899+0000","last_became_active":"2023-03-03T08:00:22.362830+0000","last_became_peered":"2023-03-03T08:00:22.362830+0000","last_unstale":"2023-03-03T08:00:22.368899+0000","last_undegraded":"2023-03-03T08:00:22.368899+0000","last_fullsized":"2023-03-03T08:00:22.368899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:27.867451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"65'1","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.359464+0000","last_change":"2023-03-03T08:00:21.792187+0000","last_active":"2023-03-03T08:00:22.359464+0000","last_peered":"2023-03-03T08:00:22.359464+0000","last_clean":"2023-03-03T08:00:22.359464+0000","last_became_active":"2023-03-03T08:00:21.791813+0000","last_became_peered":"2023-03-03T08:00:21.791813+0000","last_unstale":"2023-03-03T08:00:22.359464+0000","last_undegraded":"2023-03-03T08:00:22.359464+0000","last_fullsized":"2023-03-03T08:00:22.359464+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:26.856771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.353039+0000","last_change":"2023-03-03T08:00:22.351663+0000","last_active":"2023-03-03T08:00:22.353039+0000","last_peered":"2023-03-03T08:00:22.353039+0000","last_clean":"2023-03-03T08:00:22.353039+0000","last_became_active":"2023-03-03T08:00:22.351260+0000","last_became_peered":"2023-03-03T08:00:22.351260+0000","last_unstale":"2023-03-03T08:00:22.353039+0000","last_undegraded":"2023-03-03T08:00:22.353039+0000","last_fullsized":"2023-03-03T08:00:22.353039+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:00:05.665712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.358144+0000","last_change":"2023-03-03T08:00:21.787824+0000","last_active":"2023-03-03T08:00:22.358144+0000","last_peered":"2023-03-03T08:00:22.358144+0000","last_clean":"2023-03-03T08:00:22.358144+0000","last_became_active":"2023-03-03T08:00:21.787512+0000","last_became_peered":"2023-03-03T08:00:21.787512+0000","last_unstale":"2023-03-03T08:00:22.358144+0000","last_undegraded":"2023-03-03T08:00:22.358144+0000","last_fullsized":"2023-03-03T08:00:22.358144+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:42.678687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.359901+0000","last_change":"2023-03-03T08:00:22.354134+0000","last_active":"2023-03-03T08:00:22.359901+0000","last_peered":"2023-03-03T08:00:22.359901+0000","last_clean":"2023-03-03T08:00:22.359901+0000","last_became_active":"2023-03-03T08:00:22.353810+0000","last_became_peered":"2023-03-03T08:00:22.353810+0000","last_unstale":"2023-03-03T08:00:22.359901+0000","last_undegraded":"2023-03-03T08:00:22.359901+0000","last_fullsized":"2023-03-03T08:00:22.359901+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:04.223183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.354232+0000","last_change":"2023-03-03T08:00:21.784586+0000","last_active":"2023-03-03T08:00:22.354232+0000","last_peered":"2023-03-03T08:00:22.354232+0000","last_clean":"2023-03-03T08:00:22.354232+0000","last_became_active":"2023-03-03T08:00:21.784312+0000","last_became_peered":"2023-03-03T08:00:21.784312+0000","last_unstale":"2023-03-03T08:00:22.354232+0000","last_undegraded":"2023-03-03T08:00:22.354232+0000","last_fullsized":"2023-03-03T08:00:22.354232+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:53:15.089322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360628+0000","last_change":"2023-03-03T08:00:22.356583+0000","last_active":"2023-03-03T08:00:22.360628+0000","last_peered":"2023-03-03T08:00:22.360628+0000","last_clean":"2023-03-03T08:00:22.360628+0000","last_became_active":"2023-03-03T08:00:22.356348+0000","last_became_peered":"2023-03-03T08:00:22.356348+0000","last_unstale":"2023-03-03T08:00:22.360628+0000","last_undegraded":"2023-03-03T08:00:22.360628+0000","last_fullsized":"2023-03-03T08:00:22.360628+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:43:23.319076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"65'1","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.348088+0000","last_change":"2023-03-03T08:00:21.778523+0000","last_active":"2023-03-03T08:00:22.348088+0000","last_peered":"2023-03-03T08:00:22.348088+0000","last_clean":"2023-03-03T08:00:22.348088+0000","last_became_active":"2023-03-03T08:00:21.778260+0000","last_became_peered":"2023-03-03T08:00:21.778260+0000","last_unstale":"2023-03-03T08:00:22.348088+0000","last_undegraded":"2023-03-03T08:00:22.348088+0000","last_fullsized":"2023-03-03T08:00:22.348088+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:26.856771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356934+0000","last_change":"2023-03-03T08:00:22.350090+0000","last_active":"2023-03-03T08:00:22.356934+0000","last_peered":"2023-03-03T08:00:22.356934+0000","last_clean":"2023-03-03T08:00:22.356934+0000","last_became_active":"2023-03-03T08:00:22.349808+0000","last_became_peered":"2023-03-03T08:00:22.349808+0000","last_unstale":"2023-03-03T08:00:22.356934+0000","last_undegraded":"2023-03-03T08:00:22.356934+0000","last_fullsized":"2023-03-03T08:00:22.356934+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:09.676363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"65'1","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.348542+0000","last_change":"2023-03-03T08:00:21.787110+0000","last_active":"2023-03-03T08:00:22.348542+0000","last_peered":"2023-03-03T08:00:22.348542+0000","last_clean":"2023-03-03T08:00:22.348542+0000","last_became_active":"2023-03-03T08:00:21.786811+0000","last_became_peered":"2023-03-03T08:00:21.786811+0000","last_unstale":"2023-03-03T08:00:22.348542+0000","last_undegraded":"2023-03-03T08:00:22.348542+0000","last_fullsized":"2023-03-03T08:00:22.348542+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:57:43.225652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.366830+0000","last_change":"2023-03-03T08:00:22.365220+0000","last_active":"2023-03-03T08:00:22.366830+0000","last_peered":"2023-03-03T08:00:22.366830+0000","last_clean":"2023-03-03T08:00:22.366830+0000","last_became_active":"2023-03-03T08:00:22.364613+0000","last_became_peered":"2023-03-03T08:00:22.364613+0000","last_unstale":"2023-03-03T08:00:22.366830+0000","last_undegraded":"2023-03-03T08:00:22.366830+0000","last_fullsized":"2023-03-03T08:00:22.366830+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:35:58.114402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.332133+0000","last_change":"2023-03-03T08:00:21.783560+0000","last_active":"2023-03-03T08:00:22.332133+0000","last_peered":"2023-03-03T08:00:22.332133+0000","last_clean":"2023-03-03T08:00:22.332133+0000","last_became_active":"2023-03-03T08:00:21.783190+0000","last_became_peered":"2023-03-03T08:00:21.783190+0000","last_unstale":"2023-03-03T08:00:22.332133+0000","last_undegraded":"2023-03-03T08:00:22.332133+0000","last_fullsized":"2023-03-03T08:00:22.332133+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:39.306350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.352314+0000","last_change":"2023-03-03T08:00:22.345314+0000","last_active":"2023-03-03T08:00:22.352314+0000","last_peered":"2023-03-03T08:00:22.352314+0000","last_clean":"2023-03-03T08:00:22.352314+0000","last_became_active":"2023-03-03T08:00:22.345077+0000","last_became_peered":"2023-03-03T08:00:22.345077+0000","last_unstale":"2023-03-03T08:00:22.352314+0000","last_undegraded":"2023-03-03T08:00:22.352314+0000","last_fullsized":"2023-03-03T08:00:22.352314+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:52.425703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.344936+0000","last_change":"2023-03-03T08:00:21.792376+0000","last_active":"2023-03-03T08:00:22.344936+0000","last_peered":"2023-03-03T08:00:22.344936+0000","last_clean":"2023-03-03T08:00:22.344936+0000","last_became_active":"2023-03-03T08:00:21.792126+0000","last_became_peered":"2023-03-03T08:00:21.792126+0000","last_unstale":"2023-03-03T08:00:22.344936+0000","last_undegraded":"2023-03-03T08:00:22.344936+0000","last_fullsized":"2023-03-03T08:00:22.344936+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:44.522714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356052+0000","last_change":"2023-03-03T08:00:22.351274+0000","last_active":"2023-03-03T08:00:22.356052+0000","last_peered":"2023-03-03T08:00:22.356052+0000","last_clean":"2023-03-03T08:00:22.356052+0000","last_became_active":"2023-03-03T08:00:22.351025+0000","last_became_peered":"2023-03-03T08:00:22.351025+0000","last_unstale":"2023-03-03T08:00:22.356052+0000","last_undegraded":"2023-03-03T08:00:22.356052+0000","last_fullsized":"2023-03-03T08:00:22.356052+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:55:49.478050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.331929+0000","last_change":"2023-03-03T08:00:21.790545+0000","last_active":"2023-03-03T08:00:22.331929+0000","last_peered":"2023-03-03T08:00:22.331929+0000","last_clean":"2023-03-03T08:00:22.331929+0000","last_became_active":"2023-03-03T08:00:21.790052+0000","last_became_peered":"2023-03-03T08:00:21.790052+0000","last_unstale":"2023-03-03T08:00:22.331929+0000","last_undegraded":"2023-03-03T08:00:22.331929+0000","last_fullsized":"2023-03-03T08:00:22.331929+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:51:33.987978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.358127+0000","last_change":"2023-03-03T08:00:22.353413+0000","last_active":"2023-03-03T08:00:22.358127+0000","last_peered":"2023-03-03T08:00:22.358127+0000","last_clean":"2023-03-03T08:00:22.358127+0000","last_became_active":"2023-03-03T08:00:22.353114+0000","last_became_peered":"2023-03-03T08:00:22.353114+0000","last_unstale":"2023-03-03T08:00:22.358127+0000","last_undegraded":"2023-03-03T08:00:22.358127+0000","last_fullsized":"2023-03-03T08:00:22.358127+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:34.298493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.342606+0000","last_change":"2023-03-03T08:00:21.783602+0000","last_active":"2023-03-03T08:00:22.342606+0000","last_peered":"2023-03-03T08:00:22.342606+0000","last_clean":"2023-03-03T08:00:22.342606+0000","last_became_active":"2023-03-03T08:00:21.783297+0000","last_became_peered":"2023-03-03T08:00:21.783297+0000","last_unstale":"2023-03-03T08:00:22.342606+0000","last_undegraded":"2023-03-03T08:00:22.342606+0000","last_fullsized":"2023-03-03T08:00:22.342606+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:08:45.794437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.369192+0000","last_change":"2023-03-03T08:00:22.367931+0000","last_active":"2023-03-03T08:00:22.369192+0000","last_peered":"2023-03-03T08:00:22.369192+0000","last_clean":"2023-03-03T08:00:22.369192+0000","last_became_active":"2023-03-03T08:00:22.367374+0000","last_became_peered":"2023-03-03T08:00:22.367374+0000","last_unstale":"2023-03-03T08:00:22.369192+0000","last_undegraded":"2023-03-03T08:00:22.369192+0000","last_fullsized":"2023-03-03T08:00:22.369192+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:12:21.211704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"65'3","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368407+0000","last_change":"2023-03-03T08:00:21.790207+0000","last_active":"2023-03-03T08:00:22.368407+0000","last_peered":"2023-03-03T08:00:22.368407+0000","last_clean":"2023-03-03T08:00:22.368407+0000","last_became_active":"2023-03-03T08:00:21.789535+0000","last_became_peered":"2023-03-03T08:00:21.789535+0000","last_unstale":"2023-03-03T08:00:22.368407+0000","last_undegraded":"2023-03-03T08:00:22.368407+0000","last_fullsized":"2023-03-03T08:00:22.368407+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:02.228725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.347507+0000","last_change":"2023-03-03T08:00:22.343895+0000","last_active":"2023-03-03T08:00:22.347507+0000","last_peered":"2023-03-03T08:00:22.347507+0000","last_clean":"2023-03-03T08:00:22.347507+0000","last_became_active":"2023-03-03T08:00:22.343577+0000","last_became_peered":"2023-03-03T08:00:22.343577+0000","last_unstale":"2023-03-03T08:00:22.347507+0000","last_undegraded":"2023-03-03T08:00:22.347507+0000","last_fullsized":"2023-03-03T08:00:22.347507+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:20.734366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"65'6","reported_seq":17,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.370487+0000","last_change":"2023-03-03T08:00:21.789108+0000","last_active":"2023-03-03T08:00:22.370487+0000","last_peered":"2023-03-03T08:00:22.370487+0000","last_clean":"2023-03-03T08:00:22.370487+0000","last_became_active":"2023-03-03T08:00:21.788850+0000","last_became_peered":"2023-03-03T08:00:21.788850+0000","last_unstale":"2023-03-03T08:00:22.370487+0000","last_undegraded":"2023-03-03T08:00:22.370487+0000","last_fullsized":"2023-03-03T08:00:22.370487+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:16:16.126986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368203+0000","last_change":"2023-03-03T08:00:22.358537+0000","last_active":"2023-03-03T08:00:22.368203+0000","last_peered":"2023-03-03T08:00:22.368203+0000","last_clean":"2023-03-03T08:00:22.368203+0000","last_became_active":"2023-03-03T08:00:22.358277+0000","last_became_peered":"2023-03-03T08:00:22.358277+0000","last_unstale":"2023-03-03T08:00:22.368203+0000","last_undegraded":"2023-03-03T08:00:22.368203+0000","last_fullsized":"2023-03-03T08:00:22.368203+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:06:54.326996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.366491+0000","last_change":"2023-03-03T08:00:21.788681+0000","last_active":"2023-03-03T08:00:22.366491+0000","last_peered":"2023-03-03T08:00:22.366491+0000","last_clean":"2023-03-03T08:00:22.366491+0000","last_became_active":"2023-03-03T08:00:21.788087+0000","last_became_peered":"2023-03-03T08:00:21.788087+0000","last_unstale":"2023-03-03T08:00:22.366491+0000","last_undegraded":"2023-03-03T08:00:22.366491+0000","last_fullsized":"2023-03-03T08:00:22.366491+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:08:45.794437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.350067+0000","last_change":"2023-03-03T08:00:22.346539+0000","last_active":"2023-03-03T08:00:22.350067+0000","last_peered":"2023-03-03T08:00:22.350067+0000","last_clean":"2023-03-03T08:00:22.350067+0000","last_became_active":"2023-03-03T08:00:22.346149+0000","last_became_peered":"2023-03-03T08:00:22.346149+0000","last_unstale":"2023-03-03T08:00:22.350067+0000","last_undegraded":"2023-03-03T08:00:22.350067+0000","last_fullsized":"2023-03-03T08:00:22.350067+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:20.734366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.334121+0000","last_change":"2023-03-03T08:00:21.787442+0000","last_active":"2023-03-03T08:00:22.334121+0000","last_peered":"2023-03-03T08:00:22.334121+0000","last_clean":"2023-03-03T08:00:22.334121+0000","last_became_active":"2023-03-03T08:00:21.786782+0000","last_became_peered":"2023-03-03T08:00:21.786782+0000","last_unstale":"2023-03-03T08:00:22.334121+0000","last_undegraded":"2023-03-03T08:00:22.334121+0000","last_fullsized":"2023-03-03T08:00:22.334121+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:55:48.684553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.365840+0000","last_change":"2023-03-03T08:00:22.363185+0000","last_active":"2023-03-03T08:00:22.365840+0000","last_peered":"2023-03-03T08:00:22.365840+0000","last_clean":"2023-03-03T08:00:22.365840+0000","last_became_active":"2023-03-03T08:00:22.362886+0000","last_became_peered":"2023-03-03T08:00:22.362886+0000","last_unstale":"2023-03-03T08:00:22.365840+0000","last_undegraded":"2023-03-03T08:00:22.365840+0000","last_fullsized":"2023-03-03T08:00:22.365840+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:16:17.137666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.331162+0000","last_change":"2023-03-03T08:00:21.787586+0000","last_active":"2023-03-03T08:00:22.331162+0000","last_peered":"2023-03-03T08:00:22.331162+0000","last_clean":"2023-03-03T08:00:22.331162+0000","last_became_active":"2023-03-03T08:00:21.787272+0000","last_became_peered":"2023-03-03T08:00:21.787272+0000","last_unstale":"2023-03-03T08:00:22.331162+0000","last_undegraded":"2023-03-03T08:00:22.331162+0000","last_fullsized":"2023-03-03T08:00:22.331162+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:12.459278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.355207+0000","last_change":"2023-03-03T08:00:22.349923+0000","last_active":"2023-03-03T08:00:22.355207+0000","last_peered":"2023-03-03T08:00:22.355207+0000","last_clean":"2023-03-03T08:00:22.355207+0000","last_became_active":"2023-03-03T08:00:22.349552+0000","last_became_peered":"2023-03-03T08:00:22.349552+0000","last_unstale":"2023-03-03T08:00:22.355207+0000","last_undegraded":"2023-03-03T08:00:22.355207+0000","last_fullsized":"2023-03-03T08:00:22.355207+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:36:19.801740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"65'8","reported_seq":19,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.364523+0000","last_change":"2023-03-03T08:00:21.791488+0000","last_active":"2023-03-03T08:00:22.364523+0000","last_peered":"2023-03-03T08:00:22.364523+0000","last_clean":"2023-03-03T08:00:22.364523+0000","last_became_active":"2023-03-03T08:00:21.791122+0000","last_became_peered":"2023-03-03T08:00:21.791122+0000","last_unstale":"2023-03-03T08:00:22.364523+0000","last_undegraded":"2023-03-03T08:00:22.364523+0000","last_fullsized":"2023-03-03T08:00:22.364523+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:36:55.429526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.353072+0000","last_change":"2023-03-03T08:00:22.348199+0000","last_active":"2023-03-03T08:00:22.353072+0000","last_peered":"2023-03-03T08:00:22.353072+0000","last_clean":"2023-03-03T08:00:22.353072+0000","last_became_active":"2023-03-03T08:00:22.347841+0000","last_became_peered":"2023-03-03T08:00:22.347841+0000","last_unstale":"2023-03-03T08:00:22.353072+0000","last_undegraded":"2023-03-03T08:00:22.353072+0000","last_fullsized":"2023-03-03T08:00:22.353072+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:52.425703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.361110+0000","last_change":"2023-03-03T08:00:21.792101+0000","last_active":"2023-03-03T08:00:22.361110+0000","last_peered":"2023-03-03T08:00:22.361110+0000","last_clean":"2023-03-03T08:00:22.361110+0000","last_became_active":"2023-03-03T08:00:21.791481+0000","last_became_peered":"2023-03-03T08:00:21.791481+0000","last_unstale":"2023-03-03T08:00:22.361110+0000","last_undegraded":"2023-03-03T08:00:22.361110+0000","last_fullsized":"2023-03-03T08:00:22.361110+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:12:20.201024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360729+0000","last_change":"2023-03-03T08:00:22.358310+0000","last_active":"2023-03-03T08:00:22.360729+0000","last_peered":"2023-03-03T08:00:22.360729+0000","last_clean":"2023-03-03T08:00:22.360729+0000","last_became_active":"2023-03-03T08:00:22.358000+0000","last_became_peered":"2023-03-03T08:00:22.358000+0000","last_unstale":"2023-03-03T08:00:22.360729+0000","last_undegraded":"2023-03-03T08:00:22.360729+0000","last_fullsized":"2023-03-03T08:00:22.360729+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:23:00.594155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"66'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-03-03T08:00:23.335263+0000","last_change":"2023-03-03T08:00:21.786969+0000","last_active":"2023-03-03T08:00:23.335263+0000","last_peered":"2023-03-03T08:00:23.335263+0000","last_clean":"2023-03-03T08:00:23.335263+0000","last_became_active":"2023-03-03T08:00:21.786545+0000","last_became_peered":"2023-03-03T08:00:21.786545+0000","last_unstale":"2023-03-03T08:00:23.335263+0000","last_undegraded":"2023-03-03T08:00:23.335263+0000","last_fullsized":"2023-03-03T08:00:23.335263+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:16:25.213480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.352179+0000","last_change":"2023-03-03T08:00:22.348136+0000","last_active":"2023-03-03T08:00:22.352179+0000","last_peered":"2023-03-03T08:00:22.352179+0000","last_clean":"2023-03-03T08:00:22.352179+0000","last_became_active":"2023-03-03T08:00:22.347818+0000","last_became_peered":"2023-03-03T08:00:22.347818+0000","last_unstale":"2023-03-03T08:00:22.352179+0000","last_undegraded":"2023-03-03T08:00:22.352179+0000","last_fullsized":"2023-03-03T08:00:22.352179+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:11:16.603939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.332356+0000","last_change":"2023-03-03T08:00:21.788125+0000","last_active":"2023-03-03T08:00:22.332356+0000","last_peered":"2023-03-03T08:00:22.332356+0000","last_clean":"2023-03-03T08:00:22.332356+0000","last_became_active":"2023-03-03T08:00:21.787862+0000","last_became_peered":"2023-03-03T08:00:21.787862+0000","last_unstale":"2023-03-03T08:00:22.332356+0000","last_undegraded":"2023-03-03T08:00:22.332356+0000","last_fullsized":"2023-03-03T08:00:22.332356+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:03.212503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.364339+0000","last_change":"2023-03-03T08:00:22.356604+0000","last_active":"2023-03-03T08:00:22.364339+0000","last_peered":"2023-03-03T08:00:22.364339+0000","last_clean":"2023-03-03T08:00:22.364339+0000","last_became_active":"2023-03-03T08:00:22.356343+0000","last_became_peered":"2023-03-03T08:00:22.356343+0000","last_unstale":"2023-03-03T08:00:22.364339+0000","last_undegraded":"2023-03-03T08:00:22.364339+0000","last_fullsized":"2023-03-03T08:00:22.364339+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:49:26.845615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.336881+0000","last_change":"2023-03-03T08:00:21.789082+0000","last_active":"2023-03-03T08:00:22.336881+0000","last_peered":"2023-03-03T08:00:22.336881+0000","last_clean":"2023-03-03T08:00:22.336881+0000","last_became_active":"2023-03-03T08:00:21.788827+0000","last_became_peered":"2023-03-03T08:00:21.788827+0000","last_unstale":"2023-03-03T08:00:22.336881+0000","last_undegraded":"2023-03-03T08:00:22.336881+0000","last_fullsized":"2023-03-03T08:00:22.336881+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:58.454988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.364373+0000","last_change":"2023-03-03T08:00:22.357855+0000","last_active":"2023-03-03T08:00:22.364373+0000","last_peered":"2023-03-03T08:00:22.364373+0000","last_clean":"2023-03-03T08:00:22.364373+0000","last_became_active":"2023-03-03T08:00:22.357512+0000","last_became_peered":"2023-03-03T08:00:22.357512+0000","last_unstale":"2023-03-03T08:00:22.364373+0000","last_undegraded":"2023-03-03T08:00:22.364373+0000","last_fullsized":"2023-03-03T08:00:22.364373+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:53:38.302201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.331929+0000","last_change":"2023-03-03T08:00:21.788909+0000","last_active":"2023-03-03T08:00:22.331929+0000","last_peered":"2023-03-03T08:00:22.331929+0000","last_clean":"2023-03-03T08:00:22.331929+0000","last_became_active":"2023-03-03T08:00:21.788395+0000","last_became_peered":"2023-03-03T08:00:21.788395+0000","last_unstale":"2023-03-03T08:00:22.331929+0000","last_undegraded":"2023-03-03T08:00:22.331929+0000","last_fullsized":"2023-03-03T08:00:22.331929+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:17:29.803501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368951+0000","last_change":"2023-03-03T08:00:22.363754+0000","last_active":"2023-03-03T08:00:22.368951+0000","last_peered":"2023-03-03T08:00:22.368951+0000","last_clean":"2023-03-03T08:00:22.368951+0000","last_became_active":"2023-03-03T08:00:22.363463+0000","last_became_peered":"2023-03-03T08:00:22.363463+0000","last_unstale":"2023-03-03T08:00:22.368951+0000","last_undegraded":"2023-03-03T08:00:22.368951+0000","last_fullsized":"2023-03-03T08:00:22.368951+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:08:44.952506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.332741+0000","last_change":"2023-03-03T08:00:21.782978+0000","last_active":"2023-03-03T08:00:22.332741+0000","last_peered":"2023-03-03T08:00:22.332741+0000","last_clean":"2023-03-03T08:00:22.332741+0000","last_became_active":"2023-03-03T08:00:21.782679+0000","last_became_peered":"2023-03-03T08:00:21.782679+0000","last_unstale":"2023-03-03T08:00:22.332741+0000","last_undegraded":"2023-03-03T08:00:22.332741+0000","last_fullsized":"2023-03-03T08:00:22.332741+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:19:00.431727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.354447+0000","last_change":"2023-03-03T08:00:22.348976+0000","last_active":"2023-03-03T08:00:22.354447+0000","last_peered":"2023-03-03T08:00:22.354447+0000","last_clean":"2023-03-03T08:00:22.354447+0000","last_became_active":"2023-03-03T08:00:22.348701+0000","last_became_peered":"2023-03-03T08:00:22.348701+0000","last_unstale":"2023-03-03T08:00:22.354447+0000","last_undegraded":"2023-03-03T08:00:22.354447+0000","last_fullsized":"2023-03-03T08:00:22.354447+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:51:34.998659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.332229+0000","last_change":"2023-03-03T08:00:21.783463+0000","last_active":"2023-03-03T08:00:22.332229+0000","last_peered":"2023-03-03T08:00:22.332229+0000","last_clean":"2023-03-03T08:00:22.332229+0000","last_became_active":"2023-03-03T08:00:21.782809+0000","last_became_peered":"2023-03-03T08:00:21.782809+0000","last_unstale":"2023-03-03T08:00:22.332229+0000","last_undegraded":"2023-03-03T08:00:22.332229+0000","last_fullsized":"2023-03-03T08:00:22.332229+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:44:03.166814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360566+0000","last_change":"2023-03-03T08:00:22.355471+0000","last_active":"2023-03-03T08:00:22.360566+0000","last_peered":"2023-03-03T08:00:22.360566+0000","last_clean":"2023-03-03T08:00:22.360566+0000","last_became_active":"2023-03-03T08:00:22.355175+0000","last_became_peered":"2023-03-03T08:00:22.355175+0000","last_unstale":"2023-03-03T08:00:22.360566+0000","last_undegraded":"2023-03-03T08:00:22.360566+0000","last_fullsized":"2023-03-03T08:00:22.360566+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:22:43.504725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.334926+0000","last_change":"2023-03-03T08:00:21.784441+0000","last_active":"2023-03-03T08:00:22.334926+0000","last_peered":"2023-03-03T08:00:22.334926+0000","last_clean":"2023-03-03T08:00:22.334926+0000","last_became_active":"2023-03-03T08:00:21.784193+0000","last_became_peered":"2023-03-03T08:00:21.784193+0000","last_unstale":"2023-03-03T08:00:22.334926+0000","last_undegraded":"2023-03-03T08:00:22.334926+0000","last_fullsized":"2023-03-03T08:00:22.334926+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:00:08.298430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356321+0000","last_change":"2023-03-03T08:00:22.346849+0000","last_active":"2023-03-03T08:00:22.356321+0000","last_peered":"2023-03-03T08:00:22.356321+0000","last_clean":"2023-03-03T08:00:22.356321+0000","last_became_active":"2023-03-03T08:00:22.346557+0000","last_became_peered":"2023-03-03T08:00:22.346557+0000","last_unstale":"2023-03-03T08:00:22.356321+0000","last_undegraded":"2023-03-03T08:00:22.356321+0000","last_fullsized":"2023-03-03T08:00:22.356321+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:40.317030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"65'1","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360262+0000","last_change":"2023-03-03T08:00:21.792370+0000","last_active":"2023-03-03T08:00:22.360262+0000","last_peered":"2023-03-03T08:00:22.360262+0000","last_clean":"2023-03-03T08:00:22.360262+0000","last_became_active":"2023-03-03T08:00:21.791836+0000","last_became_peered":"2023-03-03T08:00:21.791836+0000","last_unstale":"2023-03-03T08:00:22.360262+0000","last_undegraded":"2023-03-03T08:00:22.360262+0000","last_fullsized":"2023-03-03T08:00:22.360262+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:42:51.458743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.336780+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T08:00:22.336780+0000","last_peered":"2023-03-03T08:00:22.336780+0000","last_clean":"2023-03-03T08:00:22.336780+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T08:00:22.336780+0000","last_undegraded":"2023-03-03T08:00:22.336780+0000","last_fullsized":"2023-03-03T08:00:22.336780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.366849+0000","last_change":"2023-03-03T08:00:22.365438+0000","last_active":"2023-03-03T08:00:22.366849+0000","last_peered":"2023-03-03T08:00:22.366849+0000","last_clean":"2023-03-03T08:00:22.366849+0000","last_became_active":"2023-03-03T08:00:22.364935+0000","last_became_peered":"2023-03-03T08:00:22.364935+0000","last_unstale":"2023-03-03T08:00:22.366849+0000","last_undegraded":"2023-03-03T08:00:22.366849+0000","last_fullsized":"2023-03-03T08:00:22.366849+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:59.465669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.331666+0000","last_change":"2023-03-03T08:00:21.792272+0000","last_active":"2023-03-03T08:00:22.331666+0000","last_peered":"2023-03-03T08:00:22.331666+0000","last_clean":"2023-03-03T08:00:22.331666+0000","last_became_active":"2023-03-03T08:00:21.791961+0000","last_became_peered":"2023-03-03T08:00:21.791961+0000","last_unstale":"2023-03-03T08:00:22.331666+0000","last_undegraded":"2023-03-03T08:00:22.331666+0000","last_fullsized":"2023-03-03T08:00:22.331666+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:57:43.225652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":51,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.334339+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T08:00:22.334339+0000","last_peered":"2023-03-03T08:00:22.334339+0000","last_clean":"2023-03-03T08:00:22.334339+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T08:00:22.334339+0000","last_undegraded":"2023-03-03T08:00:22.334339+0000","last_fullsized":"2023-03-03T08:00:22.334339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360169+0000","last_change":"2023-03-03T08:00:22.354435+0000","last_active":"2023-03-03T08:00:22.360169+0000","last_peered":"2023-03-03T08:00:22.360169+0000","last_clean":"2023-03-03T08:00:22.360169+0000","last_became_active":"2023-03-03T08:00:22.354144+0000","last_became_peered":"2023-03-03T08:00:22.354144+0000","last_unstale":"2023-03-03T08:00:22.360169+0000","last_undegraded":"2023-03-03T08:00:22.360169+0000","last_fullsized":"2023-03-03T08:00:22.360169+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:39.487708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.334241+0000","last_change":"2023-03-03T08:00:21.787607+0000","last_active":"2023-03-03T08:00:22.334241+0000","last_peered":"2023-03-03T08:00:22.334241+0000","last_clean":"2023-03-03T08:00:22.334241+0000","last_became_active":"2023-03-03T08:00:21.787106+0000","last_became_peered":"2023-03-03T08:00:21.787106+0000","last_unstale":"2023-03-03T08:00:22.334241+0000","last_undegraded":"2023-03-03T08:00:22.334241+0000","last_fullsized":"2023-03-03T08:00:22.334241+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:30:19.298647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.343813+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T08:00:22.343813+0000","last_peered":"2023-03-03T08:00:22.343813+0000","last_clean":"2023-03-03T08:00:22.343813+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T08:00:22.343813+0000","last_undegraded":"2023-03-03T08:00:22.343813+0000","last_fullsized":"2023-03-03T08:00:22.343813+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.366899+0000","last_change":"2023-03-03T08:00:22.365234+0000","last_active":"2023-03-03T08:00:22.366899+0000","last_peered":"2023-03-03T08:00:22.366899+0000","last_clean":"2023-03-03T08:00:22.366899+0000","last_became_active":"2023-03-03T08:00:22.364559+0000","last_became_peered":"2023-03-03T08:00:22.364559+0000","last_unstale":"2023-03-03T08:00:22.366899+0000","last_undegraded":"2023-03-03T08:00:22.366899+0000","last_fullsized":"2023-03-03T08:00:22.366899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:00:09.309111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.331737+0000","last_change":"2023-03-03T08:00:21.775813+0000","last_active":"2023-03-03T08:00:22.331737+0000","last_peered":"2023-03-03T08:00:22.331737+0000","last_clean":"2023-03-03T08:00:22.331737+0000","last_became_active":"2023-03-03T08:00:21.775553+0000","last_became_peered":"2023-03-03T08:00:21.775553+0000","last_unstale":"2023-03-03T08:00:22.331737+0000","last_undegraded":"2023-03-03T08:00:22.331737+0000","last_fullsized":"2023-03-03T08:00:22.331737+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:19.710326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.335082+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T08:00:22.335082+0000","last_peered":"2023-03-03T08:00:22.335082+0000","last_clean":"2023-03-03T08:00:22.335082+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T08:00:22.335082+0000","last_undegraded":"2023-03-03T08:00:22.335082+0000","last_fullsized":"2023-03-03T08:00:22.335082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":205,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.330930+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T08:00:22.330930+0000","last_peered":"2023-03-03T08:00:22.330930+0000","last_clean":"2023-03-03T08:00:22.330930+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T08:00:22.330930+0000","last_undegraded":"2023-03-03T08:00:22.330930+0000","last_fullsized":"2023-03-03T08:00:22.330930+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356401+0000","last_change":"2023-03-03T08:00:22.352632+0000","last_active":"2023-03-03T08:00:22.356401+0000","last_peered":"2023-03-03T08:00:22.356401+0000","last_clean":"2023-03-03T08:00:22.356401+0000","last_became_active":"2023-03-03T08:00:22.351927+0000","last_became_peered":"2023-03-03T08:00:22.351927+0000","last_unstale":"2023-03-03T08:00:22.356401+0000","last_undegraded":"2023-03-03T08:00:22.356401+0000","last_fullsized":"2023-03-03T08:00:22.356401+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:27.364263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.344601+0000","last_change":"2023-03-03T08:00:21.790111+0000","last_active":"2023-03-03T08:00:22.344601+0000","last_peered":"2023-03-03T08:00:22.344601+0000","last_clean":"2023-03-03T08:00:22.344601+0000","last_became_active":"2023-03-03T08:00:21.789831+0000","last_became_peered":"2023-03-03T08:00:21.789831+0000","last_unstale":"2023-03-03T08:00:22.344601+0000","last_undegraded":"2023-03-03T08:00:22.344601+0000","last_fullsized":"2023-03-03T08:00:22.344601+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:19.710326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":52,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.330895+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T08:00:22.330895+0000","last_peered":"2023-03-03T08:00:22.330895+0000","last_clean":"2023-03-03T08:00:22.330895+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T08:00:22.330895+0000","last_undegraded":"2023-03-03T08:00:22.330895+0000","last_fullsized":"2023-03-03T08:00:22.330895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.365925+0000","last_change":"2023-03-03T08:00:22.363947+0000","last_active":"2023-03-03T08:00:22.365925+0000","last_peered":"2023-03-03T08:00:22.365925+0000","last_clean":"2023-03-03T08:00:22.365925+0000","last_became_active":"2023-03-03T08:00:22.363477+0000","last_became_peered":"2023-03-03T08:00:22.363477+0000","last_unstale":"2023-03-03T08:00:22.365925+0000","last_undegraded":"2023-03-03T08:00:22.365925+0000","last_fullsized":"2023-03-03T08:00:22.365925+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:30:20.309327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.369342+0000","last_change":"2023-03-03T08:00:21.787378+0000","last_active":"2023-03-03T08:00:22.369342+0000","last_peered":"2023-03-03T08:00:22.369342+0000","last_clean":"2023-03-03T08:00:22.369342+0000","last_became_active":"2023-03-03T08:00:21.787102+0000","last_became_peered":"2023-03-03T08:00:21.787102+0000","last_unstale":"2023-03-03T08:00:22.369342+0000","last_undegraded":"2023-03-03T08:00:22.369342+0000","last_fullsized":"2023-03-03T08:00:22.369342+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:45:29.829197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":52,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.332485+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T08:00:22.332485+0000","last_peered":"2023-03-03T08:00:22.332485+0000","last_clean":"2023-03-03T08:00:22.332485+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T08:00:22.332485+0000","last_undegraded":"2023-03-03T08:00:22.332485+0000","last_fullsized":"2023-03-03T08:00:22.332485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.369032+0000","last_change":"2023-03-03T08:00:22.366608+0000","last_active":"2023-03-03T08:00:22.369032+0000","last_peered":"2023-03-03T08:00:22.369032+0000","last_clean":"2023-03-03T08:00:22.369032+0000","last_became_active":"2023-03-03T08:00:22.366325+0000","last_became_peered":"2023-03-03T08:00:22.366325+0000","last_unstale":"2023-03-03T08:00:22.369032+0000","last_undegraded":"2023-03-03T08:00:22.369032+0000","last_fullsized":"2023-03-03T08:00:22.369032+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:20.721006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"65'1","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368284+0000","last_change":"2023-03-03T08:00:21.789735+0000","last_active":"2023-03-03T08:00:22.368284+0000","last_peered":"2023-03-03T08:00:22.368284+0000","last_clean":"2023-03-03T08:00:22.368284+0000","last_became_active":"2023-03-03T08:00:21.789201+0000","last_became_peered":"2023-03-03T08:00:21.789201+0000","last_unstale":"2023-03-03T08:00:22.368284+0000","last_undegraded":"2023-03-03T08:00:22.368284+0000","last_fullsized":"2023-03-03T08:00:22.368284+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:08.665683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.337975+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T08:00:22.337975+0000","last_peered":"2023-03-03T08:00:22.337975+0000","last_clean":"2023-03-03T08:00:22.337975+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T08:00:22.337975+0000","last_undegraded":"2023-03-03T08:00:22.337975+0000","last_fullsized":"2023-03-03T08:00:22.337975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.364313+0000","last_change":"2023-03-03T08:00:22.356451+0000","last_active":"2023-03-03T08:00:22.364313+0000","last_peered":"2023-03-03T08:00:22.364313+0000","last_clean":"2023-03-03T08:00:22.364313+0000","last_became_active":"2023-03-03T08:00:22.356191+0000","last_became_peered":"2023-03-03T08:00:22.356191+0000","last_unstale":"2023-03-03T08:00:22.364313+0000","last_undegraded":"2023-03-03T08:00:22.364313+0000","last_fullsized":"2023-03-03T08:00:22.364313+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:17:07.748684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.333217+0000","last_change":"2023-03-03T08:00:21.778956+0000","last_active":"2023-03-03T08:00:22.333217+0000","last_peered":"2023-03-03T08:00:22.333217+0000","last_clean":"2023-03-03T08:00:22.333217+0000","last_became_active":"2023-03-03T08:00:21.778666+0000","last_became_peered":"2023-03-03T08:00:21.778666+0000","last_unstale":"2023-03-03T08:00:22.333217+0000","last_undegraded":"2023-03-03T08:00:22.333217+0000","last_fullsized":"2023-03-03T08:00:22.333217+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:05.518653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.349803+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T08:00:22.349803+0000","last_peered":"2023-03-03T08:00:22.349803+0000","last_clean":"2023-03-03T08:00:22.349803+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T08:00:22.349803+0000","last_undegraded":"2023-03-03T08:00:22.349803+0000","last_fullsized":"2023-03-03T08:00:22.349803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574910,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10980,"kb_used_data":1372,"kb_used_omap":0,"kb_used_meta":9600,"kb_avail":94360860,"statfs":{"total":96636764160,"available":96625520640,"internally_reserved":0,"allocated":1404928,"data_stored":943239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9830400},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.67500000000000004,"15min":0.70499999999999996},"min":{"1min":0.437,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82499999999999996,"5min":1.546,"15min":1.546},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71199999999999997,"15min":0.72999999999999998},"min":{"1min":0.50600000000000001,"5min":0.5,"15min":0.5},"max":{"1min":0.96499999999999997,"5min":1.702,"15min":1.702},"last":0.81499999999999995}]},{"osd":1,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.89000000000000001,"15min":1.0840000000000001},"min":{"1min":0.49199999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.111,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.755,"5min":0.74099999999999999,"15min":0.76900000000000002},"min":{"1min":0.51600000000000001,"5min":0.504,"15min":0.504},"max":{"1min":1.1890000000000001,"5min":1.944,"15min":1.944},"last":0.66900000000000004}]},{"osd":2,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.71899999999999997,"15min":0.71399999999999997},"min":{"1min":0.63300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.224,"5min":1.252,"15min":1.252},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76800000000000002,"15min":0.79300000000000004},"min":{"1min":0.54600000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2490000000000001,"5min":1.329,"15min":1.329},"last":0.76400000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.71999999999999997,"15min":0.71499999999999997},"min":{"1min":0.59799999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.754,"15min":0.79000000000000004},"min":{"1min":0.55000000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.93400000000000005,"5min":1.294,"15min":1.294},"last":0.79800000000000004}]},{"osd":4,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.48399999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0049999999999999,"5min":1.077,"15min":1.077},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.66700000000000004,"15min":0.63400000000000001},"min":{"1min":0.51700000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.77500000000000002}]},{"osd":5,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.77100000000000002,"15min":0.80100000000000005},"min":{"1min":0.41699999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.036,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81699999999999995,"15min":0.82399999999999995},"min":{"1min":0.66400000000000003,"5min":0.496,"15min":0.496},"max":{"1min":1.2190000000000001,"5min":1.349,"15min":1.349},"last":0.79000000000000004}]},{"osd":6,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.64100000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0700000000000001,"5min":1.204,"15min":1.204},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.81100000000000005,"15min":0.82099999999999995},"min":{"1min":0.627,"5min":0.499,"15min":0.499},"max":{"1min":1.135,"5min":1.194,"15min":1.194},"last":0.70799999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9424,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":8640,"kb_avail":94362416,"statfs":{"total":96636764160,"available":96627113984,"internally_reserved":0,"allocated":794624,"data_stored":351126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8847360},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.70799999999999996,"15min":0.68899999999999995},"min":{"1min":0.57199999999999995,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.018,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68999999999999995,"15min":0.69499999999999995},"min":{"1min":0.52500000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.89700000000000002,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.745}]},{"osd":1,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70999999999999996,"15min":0.68600000000000005},"min":{"1min":0.52700000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.97099999999999997,"5min":3.323,"15min":3.323},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.73499999999999999,"15min":0.73099999999999998},"min":{"1min":0.55400000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97199999999999998,"5min":1.135,"15min":1.135},"last":0.63700000000000001}]},{"osd":2,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.753,"15min":0.73199999999999998},"min":{"1min":0.59499999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.98899999999999999,"5min":1.54,"15min":1.54},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.749,"15min":0.75900000000000001},"min":{"1min":0.54400000000000004,"5min":0.495,"15min":0.495},"max":{"1min":1.0089999999999999,"5min":1.698,"15min":1.698},"last":0.71799999999999997}]},{"osd":3,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81299999999999994,"15min":0.78900000000000003},"min":{"1min":0.57499999999999996,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.3089999999999999,"5min":4.8959999999999999,"15min":4.8959999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77200000000000002,"15min":0.77100000000000002},"min":{"1min":0.60899999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.95799999999999996,"5min":1.343,"15min":1.343},"last":0.78600000000000003}]},{"osd":4,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.79900000000000004,"15min":0.91100000000000003},"min":{"1min":0.439,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.871,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.80700000000000005,"15min":0.92400000000000004},"min":{"1min":0.53000000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.85099999999999998,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.88900000000000001}]},{"osd":5,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.72599999999999998,"15min":0.77600000000000002},"min":{"1min":0.47599999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.95399999999999996,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.80300000000000005,"15min":0.82499999999999996},"min":{"1min":0.499,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91500000000000004,"5min":1.429,"15min":1.429},"last":0.69499999999999995}]},{"osd":7,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.81899999999999995,"15min":0.83899999999999997},"min":{"1min":0.56699999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.0029999999999999,"5min":1.452,"15min":1.452},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.82099999999999995,"15min":0.87},"min":{"1min":0.59399999999999997,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.008,"5min":1.579,"15min":1.579},"last":0.81299999999999994}]}]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9580,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":8768,"kb_avail":94362260,"statfs":{"total":96636764160,"available":96626954240,"internally_reserved":0,"allocated":823296,"data_stored":354407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8978432},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.49099999999999999,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.63100000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58099999999999996,"15min":0.64700000000000002},"min":{"1min":0.36799999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.90200000000000002,"5min":3.972,"15min":3.972},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.74099999999999999,"15min":0.96399999999999997},"min":{"1min":0.371,"5min":0.313,"15min":0.313},"max":{"1min":0.78900000000000003,"5min":11.897,"15min":11.897},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.82299999999999995,"15min":1.0469999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.85199999999999998,"5min":11.804,"15min":11.804},"last":0.73699999999999999}]},{"osd":3,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.81399999999999995,"15min":1.036},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.89500000000000002,"5min":11.843999999999999,"15min":11.843999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64300000000000002,"15min":0.73499999999999999},"min":{"1min":0.47899999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.96399999999999997,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.54500000000000004}]},{"osd":4,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81799999999999995,"15min":0.86299999999999999},"min":{"1min":0.55000000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0960000000000001,"5min":1.613,"15min":1.613},"last":0.745},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.86099999999999999},"min":{"1min":0.65000000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.0800000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.76200000000000001}]},{"osd":5,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.84699999999999998,"15min":0.92500000000000004},"min":{"1min":0.63900000000000001,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.997,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81999999999999995,"15min":0.873},"min":{"1min":0.57899999999999996,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98899999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.66100000000000003}]},{"osd":6,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.91300000000000003,"15min":0.999},"min":{"1min":0.68400000000000005,"5min":0.628,"15min":0.628},"max":{"1min":1.173,"5min":3.907,"15min":3.907},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.875,"5min":0.88400000000000001,"15min":0.93999999999999995},"min":{"1min":0.67300000000000004,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1819999999999999,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.70999999999999996}]},{"osd":7,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84499999999999997,"15min":0.88300000000000001},"min":{"1min":0.70399999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.026,"5min":1.419,"15min":1.419},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.871,"5min":0.90600000000000003,"15min":1.012},"min":{"1min":0.70399999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1659999999999999,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":0.67700000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10640,"kb_used_data":1352,"kb_used_omap":0,"kb_used_meta":9280,"kb_avail":94361200,"statfs":{"total":96636764160,"available":96625868800,"internally_reserved":0,"allocated":1384448,"data_stored":941460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9502720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45800000000000002,"15min":0.47199999999999998},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65000000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.50600000000000001,"15min":0.51700000000000002},"min":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81999999999999995,"5min":0.871,"15min":0.871},"last":0.54300000000000004}]},{"osd":2,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53000000000000003,"15min":0.55000000000000004},"min":{"1min":0.38400000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77400000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.44,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73399999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.46800000000000003}]},{"osd":3,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70399999999999996,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.46600000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.80800000000000005,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.50700000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82299999999999995,"15min":0.84499999999999997},"min":{"1min":0.64700000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79800000000000004,"15min":0.81999999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0740000000000001,"5min":1.605,"15min":1.605},"last":0.83499999999999996}]},{"osd":5,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.80400000000000005,"15min":0.82199999999999995},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0980000000000001,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.82099999999999995,"15min":0.82499999999999996},"min":{"1min":0.61299999999999999,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.2070000000000001,"5min":1.286,"15min":1.286},"last":0.749}]},{"osd":6,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.85399999999999998,"15min":0.88500000000000001},"min":{"1min":0.65300000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2470000000000001,"5min":1.623,"15min":1.623},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.86399999999999999},"min":{"1min":0.66300000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.137,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.97099999999999997}]},{"osd":7,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84499999999999997,"15min":0.88300000000000001},"min":{"1min":0.59399999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.083,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.83999999999999997,"15min":0.86199999999999999},"min":{"1min":0.58199999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.1719999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.78900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9484,"kb_used_data":772,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362356,"statfs":{"total":96636764160,"available":96627052544,"internally_reserved":0,"allocated":790528,"data_stored":351092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.64800000000000002,"15min":0.87},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.61699999999999999,"5min":5.7889999999999997,"15min":5.7889999999999997},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.60899999999999999,"15min":0.73199999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67400000000000004,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":0.36499999999999999}]},{"osd":1,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.629,"15min":0.753},"min":{"1min":0.36299999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77700000000000002,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.79300000000000004,"15min":1.0089999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.77100000000000002,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.51500000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.88900000000000001,"15min":1.222},"min":{"1min":0.41399999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.69799999999999995,"5min":12.961,"15min":12.961},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.80000000000000004,"15min":1.036},"min":{"1min":0.42699999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.74099999999999999,"5min":6.1449999999999996,"15min":6.1449999999999996},"last":0.48299999999999998}]},{"osd":4,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.0469999999999999,"15min":1.2769999999999999},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1639999999999999,"5min":6.484,"15min":6.484},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.0580000000000001,"15min":1.2549999999999999},"min":{"1min":0.63600000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.1759999999999999,"5min":6.5510000000000002,"15min":6.5510000000000002},"last":0.68000000000000005}]},{"osd":5,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.97399999999999998,"15min":1.1799999999999999},"min":{"1min":0.61499999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.034,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.84699999999999998,"5min":1.0269999999999999,"15min":1.238},"min":{"1min":0.60999999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1499999999999999,"5min":6.6369999999999996,"15min":6.6369999999999996},"last":0.77100000000000002}]},{"osd":6,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.0429999999999999,"15min":1.24},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.1240000000000001,"5min":6.625,"15min":6.625},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.88200000000000001,"5min":1.0549999999999999,"15min":1.2669999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.099,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.69099999999999995}]},{"osd":7,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":1.0129999999999999,"15min":1.2310000000000001},"min":{"1min":0.69899999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.087,"5min":6.7009999999999996,"15min":6.7009999999999996},"last":0.623},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.0209999999999999,"15min":1.214},"min":{"1min":0.66500000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.081,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.64100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574909,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9936,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94361904,"statfs":{"total":96636764160,"available":96626589696,"internally_reserved":0,"allocated":794624,"data_stored":352111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.63,"15min":0.83499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.56599999999999995,"5min":9.5820000000000007,"15min":9.5820000000000007},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.70099999999999996,"15min":0.90600000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83599999999999997,"5min":9.7840000000000007,"15min":9.7840000000000007},"last":0.38200000000000001}]},{"osd":1,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.501,"15min":0.52300000000000002},"min":{"1min":0.34999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78900000000000003,"5min":1,"15min":1},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.75600000000000001,"15min":0.98999999999999999},"min":{"1min":0.41099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70499999999999996,"5min":9.702,"15min":9.702},"last":0.54200000000000004}]},{"osd":2,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.746,"15min":0.93600000000000005},"min":{"1min":0.436,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88300000000000001,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.623},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.80500000000000005,"15min":1.0229999999999999},"min":{"1min":0.498,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.86499999999999999,"5min":9.734,"15min":9.734},"last":0.60699999999999998}]},{"osd":4,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.96099999999999997,"15min":1.165},"min":{"1min":0.54400000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1819999999999999,"5min":9.8629999999999995,"15min":9.8629999999999995},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.83999999999999997,"15min":0.84799999999999998},"min":{"1min":0.629,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.097,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":0.745}]},{"osd":5,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82099999999999995,"15min":0.83299999999999996},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.1899999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.999,"15min":1.222},"min":{"1min":0.628,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0860000000000001,"5min":9.6600000000000001,"15min":9.6600000000000001},"last":0.77000000000000002}]},{"osd":6,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.0129999999999999,"15min":1.2370000000000001},"min":{"1min":0.65700000000000003,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0600000000000001,"5min":9.8190000000000008,"15min":9.8190000000000008},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82799999999999996,"15min":0.86699999999999999},"min":{"1min":0.629,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.2609999999999999,"5min":1.609,"15min":1.609},"last":0.78000000000000003}]},{"osd":7,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":1.0369999999999999,"15min":1.2749999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.204,"5min":9.625,"15min":9.625},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.81399999999999995,"5min":1.034,"15min":1.244},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.119,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":0.86699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574909,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9692,"kb_used_data":788,"kb_used_omap":0,"kb_used_meta":8896,"kb_avail":94362148,"statfs":{"total":96636764160,"available":96626839552,"internally_reserved":0,"allocated":806912,"data_stored":353659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9109504},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.85199999999999998,"15min":0.98299999999999998},"min":{"1min":0.433,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0049999999999999,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.68400000000000005,"15min":0.66600000000000004},"min":{"1min":0.56100000000000005,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.95399999999999996,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.64900000000000002}]},{"osd":1,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.92700000000000005,"15min":1.1080000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94199999999999995,"5min":9.5709999999999997,"15min":9.5709999999999997},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.83999999999999997,"15min":0.92700000000000005},"min":{"1min":0.58599999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0349999999999999,"5min":5.266,"15min":5.266},"last":0.73999999999999999}]},{"osd":2,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.91600000000000004,"15min":1.077},"min":{"1min":0.60299999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.016,"5min":8.4529999999999994,"15min":8.4529999999999994},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.88,"15min":1.002},"min":{"1min":0.53200000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.018,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.85799999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.72899999999999998,"15min":0.73299999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.79000000000000004,"15min":0.80200000000000005},"min":{"1min":0.58399999999999996,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.98499999999999999,"5min":1.389,"15min":1.389},"last":0.86899999999999999}]},{"osd":5,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.71799999999999997,"15min":0.74199999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86799999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.72299999999999998,"15min":0.72099999999999997},"min":{"1min":0.47599999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.998,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.58699999999999997}]},{"osd":6,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.77300000000000002,"15min":0.78100000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.94599999999999995,"5min":1.032,"15min":1.032},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.501,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0700000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.81899999999999995}]},{"osd":7,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.79700000000000004,"15min":0.82199999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0609999999999999,"5min":1.679,"15min":1.679},"last":0.75},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.88,"15min":0.92800000000000005},"min":{"1min":0.64300000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0069999999999999,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.90500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9616,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362224,"statfs":{"total":96636764160,"available":96626917376,"internally_reserved":0,"allocated":794624,"data_stored":351114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.65700000000000003,"15min":0.64100000000000001},"min":{"1min":0.42899999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.75700000000000001,"15min":0.74399999999999999},"min":{"1min":0.60599999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96999999999999997,"5min":2.528,"15min":2.528},"last":0.57499999999999996}]},{"osd":1,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.73699999999999999,"15min":0.745},"min":{"1min":0.66400000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.752},{"interface":"front","average":{"1min":0.753,"5min":0.67200000000000004,"15min":0.63700000000000001},"min":{"1min":0.53900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.5}]},{"osd":2,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69199999999999995,"15min":0.68300000000000005},"min":{"1min":0.52700000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.95399999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.72499999999999998,"15min":0.71599999999999997},"min":{"1min":0.59299999999999997,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.89200000000000002,"5min":1.321,"15min":1.321},"last":0.63800000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.71599999999999997,"15min":0.69499999999999995},"min":{"1min":0.55600000000000005,"5min":0.496,"15min":0.496},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.745,"15min":0.73499999999999999},"min":{"1min":0.629,"5min":0.501,"15min":0.501},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.80800000000000005}]},{"osd":4,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.51400000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.79000000000000004,"15min":0.78200000000000003},"min":{"1min":0.59899999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.82999999999999996}]},{"osd":6,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.79700000000000004},"min":{"1min":0.53000000000000003,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.94499999999999995,"5min":1.071,"15min":1.071},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.72199999999999998,"15min":0.72399999999999998},"min":{"1min":0.58999999999999997,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97099999999999997,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.77800000000000002}]},{"osd":7,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.75600000000000001,"15min":0.75700000000000001},"min":{"1min":0.55700000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.026,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81599999999999995,"15min":0.83299999999999996},"min":{"1min":0.67700000000000005,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.90500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:00:32.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:00:32.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.772+0000 7f99b59ff700 1 Processor -- start 2023-03-03T08:00:32.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.772+0000 7f99b59ff700 1 -- start start 2023-03-03T08:00:32.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.774+0000 7f99b59ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f99b015cc00 0x7f99b00cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:32.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.774+0000 7f99b59ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b00cde10 con 0x7f99b00d27c0 2023-03-03T08:00:32.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.774+0000 7f99b59ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b00d1a00 con 0x7f99b00d2c60 2023-03-03T08:00:32.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.774+0000 7f99b59ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b00ce040 con 0x7f99b015cc00 2023-03-03T08:00:32.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.775+0000 7f99ae7fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f99b00d27c0 0x7f99b00b60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44890/0 (socket says 172.21.15.98:44890) 2023-03-03T08:00:32.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.775+0000 7f99ae7fc700 1 -- 172.21.15.98:0/3122285354 learned_addr learned my addr 172.21.15.98:0/3122285354 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:32.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.775+0000 7f99aeffd700 1 --2- 172.21.15.98:0/3122285354 >> v2:172.21.15.98:3300/0 conn(0x7f99b015cc00 0x7f99b00cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:32.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.775+0000 7f99adffb700 1 -- 172.21.15.98:0/3122285354 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167254383 0 0) 0x7f99b00cde10 con 0x7f99b00d27c0 2023-03-03T08:00:32.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.775+0000 7f99adffb700 1 -- 172.21.15.98:0/3122285354 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99980035b0 con 0x7f99b00d27c0 2023-03-03T08:00:32.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.775+0000 7f99aeffd700 1 -- 172.21.15.98:0/3122285354 >> v1:172.21.15.100:6789/0 conn(0x7f99b00d2c60 legacy=0x7f99b00cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:32.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.775+0000 7f99adffb700 1 -- 172.21.15.98:0/3122285354 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596267268 0 0) 0x7f99b00d1a00 con 0x7f99b00d2c60 2023-03-03T08:00:32.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.775+0000 7f99aeffd700 1 -- 172.21.15.98:0/3122285354 >> v1:172.21.15.98:6789/0 conn(0x7f99b00d27c0 legacy=0x7f99b00b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:32.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.775+0000 7f99aeffd700 1 -- 172.21.15.98:0/3122285354 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b00d1ce0 con 0x7f99b015cc00 2023-03-03T08:00:32.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.776+0000 7f99aeffd700 1 --2- 172.21.15.98:0/3122285354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99b015cc00 0x7f99b00cd8d0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f99a0009840 tx=0x7f99a0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=751dadcb8a851fb3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:32.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.776+0000 7f99adffb700 1 -- 172.21.15.98:0/3122285354 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0011830 con 0x7f99b015cc00 2023-03-03T08:00:32.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.776+0000 7f99adffb700 1 -- 172.21.15.98:0/3122285354 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a0011990 con 0x7f99b015cc00 2023-03-03T08:00:32.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.776+0000 7f99adffb700 1 -- 172.21.15.98:0/3122285354 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0011d20 con 0x7f99b015cc00 2023-03-03T08:00:32.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.776+0000 7f99b59ff700 1 -- 172.21.15.98:0/3122285354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99b015cc00 msgr2=0x7f99b00cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:32.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.776+0000 7f99b59ff700 1 --2- 172.21.15.98:0/3122285354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99b015cc00 0x7f99b00cd8d0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f99a0009840 tx=0x7f99a0009b50 comp rx=0 tx=0).stop 2023-03-03T08:00:32.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.777+0000 7f99b59ff700 1 -- 172.21.15.98:0/3122285354 shutdown_connections 2023-03-03T08:00:32.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.777+0000 7f99b59ff700 1 --2- 172.21.15.98:0/3122285354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99b015cc00 0x7f99b00cd8d0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:32.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.777+0000 7f99b59ff700 1 -- 172.21.15.98:0/3122285354 >> 172.21.15.98:0/3122285354 conn(0x7f99b006aee0 msgr2=0x7f99b006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:32.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.777+0000 7f99b59ff700 1 -- 172.21.15.98:0/3122285354 shutdown_connections 2023-03-03T08:00:32.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.777+0000 7f99b59ff700 1 -- 172.21.15.98:0/3122285354 wait complete. 2023-03-03T08:00:32.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.779+0000 7f99b59ff700 1 Processor -- start 2023-03-03T08:00:32.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.779+0000 7f99b59ff700 1 -- start start 2023-03-03T08:00:32.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.780+0000 7f99b59ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99b00d27c0 0x7f99b015fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:32.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.780+0000 7f99b59ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99b00d2c60 0x7f99b015cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:32.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.780+0000 7f99b59ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99b015cc00 0x7f99b015d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:32.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.780+0000 7f99b59ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b015ee40 con 0x7f99b00d2c60 2023-03-03T08:00:32.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.780+0000 7f99b59ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b01431f0 con 0x7f99b00d27c0 2023-03-03T08:00:32.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.780+0000 7f99b59ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b0142f10 con 0x7f99b015cc00 2023-03-03T08:00:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.780+0000 7f99ae7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99b00d2c60 0x7f99b015cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.780+0000 7f99ae7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99b00d2c60 0x7f99b015cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45026/0 (socket says 172.21.15.98:45026) 2023-03-03T08:00:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.780+0000 7f99ae7fc700 1 -- 172.21.15.98:0/3922893238 learned_addr learned my addr 172.21.15.98:0/3922893238 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.781+0000 7f99ae7fc700 1 -- 172.21.15.98:0/3922893238 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99b015cc00 msgr2=0x7f99b015d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.781+0000 7f99ae7fc700 1 --2- 172.21.15.98:0/3922893238 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99b015cc00 0x7f99b015d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.781+0000 7f99ae7fc700 1 -- 172.21.15.98:0/3922893238 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99b00d27c0 msgr2=0x7f99b015fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.781+0000 7f99ae7fc700 1 --2- 172.21.15.98:0/3922893238 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99b00d27c0 0x7f99b015fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.781+0000 7f99ae7fc700 1 -- 172.21.15.98:0/3922893238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a00094a0 con 0x7f99b00d2c60 2023-03-03T08:00:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.781+0000 7f99ae7fc700 1 --2- 172.21.15.98:0/3922893238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99b00d2c60 0x7f99b015cfd0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f99a400c450 tx=0x7f99a400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.781+0000 7f9997fff700 1 -- 172.21.15.98:0/3922893238 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4005d70 con 0x7f99b00d2c60 2023-03-03T08:00:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.781+0000 7f9997fff700 1 -- 172.21.15.98:0/3922893238 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a40070d0 con 0x7f99b00d2c60 2023-03-03T08:00:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.781+0000 7f99b59ff700 1 -- 172.21.15.98:0/3922893238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b00d22a0 con 0x7f99b00d2c60 2023-03-03T08:00:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.781+0000 7f9997fff700 1 -- 172.21.15.98:0/3922893238 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4007440 con 0x7f99b00d2c60 2023-03-03T08:00:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.781+0000 7f99b59ff700 1 -- 172.21.15.98:0/3922893238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b00d1fc0 con 0x7f99b00d2c60 2023-03-03T08:00:32.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.782+0000 7f99b59ff700 1 -- 172.21.15.98:0/3922893238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b00b67e0 con 0x7f99b00d2c60 2023-03-03T08:00:32.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.784+0000 7f9997fff700 1 -- 172.21.15.98:0/3922893238 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f99a4019030 con 0x7f99b00d2c60 2023-03-03T08:00:32.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.785+0000 7f9997fff700 1 --2- 172.21.15.98:0/3922893238 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f999c0683c0 0x7f999c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:32.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.785+0000 7f9997fff700 1 -- 172.21.15.98:0/3922893238 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f99a40850e0 con 0x7f99b00d2c60 2023-03-03T08:00:32.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.785+0000 7f99aeffd700 1 --2- 172.21.15.98:0/3922893238 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f999c0683c0 0x7f999c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:32.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.786+0000 7f99aeffd700 1 --2- 172.21.15.98:0/3922893238 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f999c0683c0 0x7f999c06a870 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f99a0009b90 tx=0x7f99a0010000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:32.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.789+0000 7f9997fff700 1 -- 172.21.15.98:0/3922893238 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f99a404b5e0 con 0x7f99b00d2c60 2023-03-03T08:00:32.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:00:32.838 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:00:32.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.987+0000 7f99b59ff700 1 -- 172.21.15.98:0/3922893238 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f99b00729b0 con 0x7f999c0683c0 2023-03-03T08:00:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:32.996+0000 7f9997fff700 1 -- 172.21.15.98:0/3922893238 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310729 (secure 0 0 0) 0x7f99b00729b0 con 0x7f999c0683c0 2023-03-03T08:00:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:33.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.002+0000 7f99b59ff700 1 -- 172.21.15.98:0/3922893238 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f999c0683c0 msgr2=0x7f999c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:33.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.002+0000 7f99b59ff700 1 --2- 172.21.15.98:0/3922893238 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f999c0683c0 0x7f999c06a870 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f99a0009b90 tx=0x7f99a0010000 comp rx=0 tx=0).stop 2023-03-03T08:00:33.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.002+0000 7f99b59ff700 1 -- 172.21.15.98:0/3922893238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99b00d2c60 msgr2=0x7f99b015cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:33.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.002+0000 7f99b59ff700 1 --2- 172.21.15.98:0/3922893238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99b00d2c60 0x7f99b015cfd0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f99a400c450 tx=0x7f99a400c810 comp rx=0 tx=0).stop 2023-03-03T08:00:33.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.002+0000 7f99b59ff700 1 -- 172.21.15.98:0/3922893238 shutdown_connections 2023-03-03T08:00:33.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.002+0000 7f99b59ff700 1 --2- 172.21.15.98:0/3922893238 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f999c0683c0 0x7f999c06a870 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.002+0000 7f99b59ff700 1 --2- 172.21.15.98:0/3922893238 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99b015cc00 0x7f99b015d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.002+0000 7f99b59ff700 1 --2- 172.21.15.98:0/3922893238 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99b00d27c0 0x7f99b015fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.002+0000 7f99b59ff700 1 --2- 172.21.15.98:0/3922893238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99b00d2c60 0x7f99b015cfd0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.002+0000 7f99b59ff700 1 -- 172.21.15.98:0/3922893238 >> 172.21.15.98:0/3922893238 conn(0x7f99b006aee0 msgr2=0x7f99b00b6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.002+0000 7f99b59ff700 1 -- 172.21.15.98:0/3922893238 shutdown_connections 2023-03-03T08:00:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.002+0000 7f99b59ff700 1 -- 172.21.15.98:0/3922893238 wait complete. 2023-03-03T08:00:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T08:00:33.032 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2023-03-03T08:00:32.242870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":79352,"kb_used_data":7416,"kb_used_omap":0,"kb_used_meta":71872,"kb_avail":754895368,"statfs":{"total":773094113280,"available":773012856832,"internally_reserved":0,"allocated":7593984,"data_stored":3998208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":73596928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.916153"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356978+0000","last_change":"2023-03-03T08:00:22.351816+0000","last_active":"2023-03-03T08:00:22.356978+0000","last_peered":"2023-03-03T08:00:22.356978+0000","last_clean":"2023-03-03T08:00:22.356978+0000","last_became_active":"2023-03-03T08:00:22.351560+0000","last_became_peered":"2023-03-03T08:00:22.351560+0000","last_unstale":"2023-03-03T08:00:22.356978+0000","last_undegraded":"2023-03-03T08:00:22.356978+0000","last_fullsized":"2023-03-03T08:00:22.356978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:22:36.090135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.355977+0000","last_change":"2023-03-03T08:00:22.349135+0000","last_active":"2023-03-03T08:00:22.355977+0000","last_peered":"2023-03-03T08:00:22.355977+0000","last_clean":"2023-03-03T08:00:22.355977+0000","last_became_active":"2023-03-03T08:00:22.348832+0000","last_became_peered":"2023-03-03T08:00:22.348832+0000","last_unstale":"2023-03-03T08:00:22.355977+0000","last_undegraded":"2023-03-03T08:00:22.355977+0000","last_fullsized":"2023-03-03T08:00:22.355977+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:45:30.839878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.343799+0000","last_change":"2023-03-03T08:00:22.341359+0000","last_active":"2023-03-03T08:00:22.343799+0000","last_peered":"2023-03-03T08:00:22.343799+0000","last_clean":"2023-03-03T08:00:22.343799+0000","last_became_active":"2023-03-03T08:00:22.341054+0000","last_became_peered":"2023-03-03T08:00:22.341054+0000","last_unstale":"2023-03-03T08:00:22.343799+0000","last_undegraded":"2023-03-03T08:00:22.343799+0000","last_fullsized":"2023-03-03T08:00:22.343799+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:57:44.236333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368309+0000","last_change":"2023-03-03T08:00:22.356433+0000","last_active":"2023-03-03T08:00:22.368309+0000","last_peered":"2023-03-03T08:00:22.368309+0000","last_clean":"2023-03-03T08:00:22.368309+0000","last_became_active":"2023-03-03T08:00:22.356154+0000","last_became_peered":"2023-03-03T08:00:22.356154+0000","last_unstale":"2023-03-03T08:00:22.368309+0000","last_undegraded":"2023-03-03T08:00:22.368309+0000","last_fullsized":"2023-03-03T08:00:22.368309+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:39.487708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.366920+0000","last_change":"2023-03-03T08:00:22.365431+0000","last_active":"2023-03-03T08:00:22.366920+0000","last_peered":"2023-03-03T08:00:22.366920+0000","last_clean":"2023-03-03T08:00:22.366920+0000","last_became_active":"2023-03-03T08:00:22.364882+0000","last_became_peered":"2023-03-03T08:00:22.364882+0000","last_unstale":"2023-03-03T08:00:22.366920+0000","last_undegraded":"2023-03-03T08:00:22.366920+0000","last_fullsized":"2023-03-03T08:00:22.366920+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:16:26.224161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.347929+0000","last_change":"2023-03-03T08:00:22.344899+0000","last_active":"2023-03-03T08:00:22.347929+0000","last_peered":"2023-03-03T08:00:22.347929+0000","last_clean":"2023-03-03T08:00:22.347929+0000","last_became_active":"2023-03-03T08:00:22.344558+0000","last_became_peered":"2023-03-03T08:00:22.344558+0000","last_unstale":"2023-03-03T08:00:22.347929+0000","last_undegraded":"2023-03-03T08:00:22.347929+0000","last_fullsized":"2023-03-03T08:00:22.347929+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:34.298493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.362506+0000","last_change":"2023-03-03T08:00:22.360864+0000","last_active":"2023-03-03T08:00:22.362506+0000","last_peered":"2023-03-03T08:00:22.362506+0000","last_clean":"2023-03-03T08:00:22.362506+0000","last_became_active":"2023-03-03T08:00:22.360550+0000","last_became_peered":"2023-03-03T08:00:22.360550+0000","last_unstale":"2023-03-03T08:00:22.362506+0000","last_undegraded":"2023-03-03T08:00:22.362506+0000","last_fullsized":"2023-03-03T08:00:22.362506+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:00:54.378703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.349837+0000","last_change":"2023-03-03T08:00:22.346751+0000","last_active":"2023-03-03T08:00:22.349837+0000","last_peered":"2023-03-03T08:00:22.349837+0000","last_clean":"2023-03-03T08:00:22.349837+0000","last_became_active":"2023-03-03T08:00:22.346388+0000","last_became_peered":"2023-03-03T08:00:22.346388+0000","last_unstale":"2023-03-03T08:00:22.349837+0000","last_undegraded":"2023-03-03T08:00:22.349837+0000","last_fullsized":"2023-03-03T08:00:22.349837+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:40.317030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356366+0000","last_change":"2023-03-03T08:00:22.347590+0000","last_active":"2023-03-03T08:00:22.356366+0000","last_peered":"2023-03-03T08:00:22.356366+0000","last_clean":"2023-03-03T08:00:22.356366+0000","last_became_active":"2023-03-03T08:00:22.347256+0000","last_became_peered":"2023-03-03T08:00:22.347256+0000","last_unstale":"2023-03-03T08:00:22.356366+0000","last_undegraded":"2023-03-03T08:00:22.356366+0000","last_fullsized":"2023-03-03T08:00:22.356366+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:23:14.650660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.359341+0000","last_change":"2023-03-03T08:00:22.352944+0000","last_active":"2023-03-03T08:00:22.359341+0000","last_peered":"2023-03-03T08:00:22.359341+0000","last_clean":"2023-03-03T08:00:22.359341+0000","last_became_active":"2023-03-03T08:00:22.352547+0000","last_became_peered":"2023-03-03T08:00:22.352547+0000","last_unstale":"2023-03-03T08:00:22.359341+0000","last_undegraded":"2023-03-03T08:00:22.359341+0000","last_fullsized":"2023-03-03T08:00:22.359341+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:53:16.100003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368088+0000","last_change":"2023-03-03T08:00:22.358102+0000","last_active":"2023-03-03T08:00:22.368088+0000","last_peered":"2023-03-03T08:00:22.368088+0000","last_clean":"2023-03-03T08:00:22.368088+0000","last_became_active":"2023-03-03T08:00:22.357726+0000","last_became_peered":"2023-03-03T08:00:22.357726+0000","last_unstale":"2023-03-03T08:00:22.368088+0000","last_undegraded":"2023-03-03T08:00:22.368088+0000","last_fullsized":"2023-03-03T08:00:22.368088+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:19:01.442408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.349788+0000","last_change":"2023-03-03T08:00:22.346586+0000","last_active":"2023-03-03T08:00:22.349788+0000","last_peered":"2023-03-03T08:00:22.349788+0000","last_clean":"2023-03-03T08:00:22.349788+0000","last_became_active":"2023-03-03T08:00:22.346252+0000","last_became_peered":"2023-03-03T08:00:22.346252+0000","last_unstale":"2023-03-03T08:00:22.349788+0000","last_undegraded":"2023-03-03T08:00:22.349788+0000","last_fullsized":"2023-03-03T08:00:22.349788+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:44:04.177495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.352213+0000","last_change":"2023-03-03T08:00:22.347364+0000","last_active":"2023-03-03T08:00:22.352213+0000","last_peered":"2023-03-03T08:00:22.352213+0000","last_clean":"2023-03-03T08:00:22.352213+0000","last_became_active":"2023-03-03T08:00:22.347036+0000","last_became_peered":"2023-03-03T08:00:22.347036+0000","last_unstale":"2023-03-03T08:00:22.352213+0000","last_undegraded":"2023-03-03T08:00:22.352213+0000","last_fullsized":"2023-03-03T08:00:22.352213+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:20.721006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.352093+0000","last_change":"2023-03-03T08:00:22.342081+0000","last_active":"2023-03-03T08:00:22.352093+0000","last_peered":"2023-03-03T08:00:22.352093+0000","last_clean":"2023-03-03T08:00:22.352093+0000","last_became_active":"2023-03-03T08:00:22.341783+0000","last_became_peered":"2023-03-03T08:00:22.341783+0000","last_unstale":"2023-03-03T08:00:22.352093+0000","last_undegraded":"2023-03-03T08:00:22.352093+0000","last_fullsized":"2023-03-03T08:00:22.352093+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:20.721006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.361028+0000","last_change":"2023-03-03T08:00:22.354119+0000","last_active":"2023-03-03T08:00:22.361028+0000","last_peered":"2023-03-03T08:00:22.361028+0000","last_clean":"2023-03-03T08:00:22.361028+0000","last_became_active":"2023-03-03T08:00:22.353824+0000","last_became_peered":"2023-03-03T08:00:22.353824+0000","last_unstale":"2023-03-03T08:00:22.361028+0000","last_undegraded":"2023-03-03T08:00:22.361028+0000","last_fullsized":"2023-03-03T08:00:22.361028+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:27.867451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.354242+0000","last_change":"2023-03-03T08:00:22.350728+0000","last_active":"2023-03-03T08:00:22.354242+0000","last_peered":"2023-03-03T08:00:22.354242+0000","last_clean":"2023-03-03T08:00:22.354242+0000","last_became_active":"2023-03-03T08:00:22.350468+0000","last_became_peered":"2023-03-03T08:00:22.350468+0000","last_unstale":"2023-03-03T08:00:22.354242+0000","last_undegraded":"2023-03-03T08:00:22.354242+0000","last_fullsized":"2023-03-03T08:00:22.354242+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:36:10.563980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.361036+0000","last_change":"2023-03-03T08:00:22.356264+0000","last_active":"2023-03-03T08:00:22.361036+0000","last_peered":"2023-03-03T08:00:22.361036+0000","last_clean":"2023-03-03T08:00:22.361036+0000","last_became_active":"2023-03-03T08:00:22.355666+0000","last_became_peered":"2023-03-03T08:00:22.355666+0000","last_unstale":"2023-03-03T08:00:22.361036+0000","last_undegraded":"2023-03-03T08:00:22.361036+0000","last_fullsized":"2023-03-03T08:00:22.361036+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:03.239405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.367982+0000","last_change":"2023-03-03T08:00:22.357934+0000","last_active":"2023-03-03T08:00:22.367982+0000","last_peered":"2023-03-03T08:00:22.367982+0000","last_clean":"2023-03-03T08:00:22.367982+0000","last_became_active":"2023-03-03T08:00:22.357644+0000","last_became_peered":"2023-03-03T08:00:22.357644+0000","last_unstale":"2023-03-03T08:00:22.367982+0000","last_undegraded":"2023-03-03T08:00:22.367982+0000","last_fullsized":"2023-03-03T08:00:22.367982+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:44:04.177495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.365907+0000","last_change":"2023-03-03T08:00:22.363152+0000","last_active":"2023-03-03T08:00:22.365907+0000","last_peered":"2023-03-03T08:00:22.365907+0000","last_clean":"2023-03-03T08:00:22.365907+0000","last_became_active":"2023-03-03T08:00:22.362838+0000","last_became_peered":"2023-03-03T08:00:22.362838+0000","last_unstale":"2023-03-03T08:00:22.365907+0000","last_undegraded":"2023-03-03T08:00:22.365907+0000","last_fullsized":"2023-03-03T08:00:22.365907+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:35:40.983679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.370386+0000","last_change":"2023-03-03T08:00:22.365139+0000","last_active":"2023-03-03T08:00:22.370386+0000","last_peered":"2023-03-03T08:00:22.370386+0000","last_clean":"2023-03-03T08:00:22.370386+0000","last_became_active":"2023-03-03T08:00:22.364818+0000","last_became_peered":"2023-03-03T08:00:22.364818+0000","last_unstale":"2023-03-03T08:00:22.370386+0000","last_undegraded":"2023-03-03T08:00:22.370386+0000","last_fullsized":"2023-03-03T08:00:22.370386+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:00:39.446569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360521+0000","last_change":"2023-03-03T08:00:22.352070+0000","last_active":"2023-03-03T08:00:22.360521+0000","last_peered":"2023-03-03T08:00:22.360521+0000","last_clean":"2023-03-03T08:00:22.360521+0000","last_became_active":"2023-03-03T08:00:22.351637+0000","last_became_peered":"2023-03-03T08:00:22.351637+0000","last_unstale":"2023-03-03T08:00:22.360521+0000","last_undegraded":"2023-03-03T08:00:22.360521+0000","last_fullsized":"2023-03-03T08:00:22.360521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:19:01.442408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.352202+0000","last_change":"2023-03-03T08:00:22.348419+0000","last_active":"2023-03-03T08:00:22.352202+0000","last_peered":"2023-03-03T08:00:22.352202+0000","last_clean":"2023-03-03T08:00:22.352202+0000","last_became_active":"2023-03-03T08:00:22.348168+0000","last_became_peered":"2023-03-03T08:00:22.348168+0000","last_unstale":"2023-03-03T08:00:22.352202+0000","last_undegraded":"2023-03-03T08:00:22.352202+0000","last_fullsized":"2023-03-03T08:00:22.352202+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:52:14.400121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.346307+0000","last_change":"2023-03-03T08:00:22.343763+0000","last_active":"2023-03-03T08:00:22.346307+0000","last_peered":"2023-03-03T08:00:22.346307+0000","last_clean":"2023-03-03T08:00:22.346307+0000","last_became_active":"2023-03-03T08:00:22.343407+0000","last_became_peered":"2023-03-03T08:00:22.343407+0000","last_unstale":"2023-03-03T08:00:22.346307+0000","last_undegraded":"2023-03-03T08:00:22.346307+0000","last_fullsized":"2023-03-03T08:00:22.346307+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:51:43.370212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.343549+0000","last_change":"2023-03-03T08:00:22.340970+0000","last_active":"2023-03-03T08:00:22.343549+0000","last_peered":"2023-03-03T08:00:22.343549+0000","last_clean":"2023-03-03T08:00:22.343549+0000","last_became_active":"2023-03-03T08:00:22.340635+0000","last_became_peered":"2023-03-03T08:00:22.340635+0000","last_unstale":"2023-03-03T08:00:22.343549+0000","last_undegraded":"2023-03-03T08:00:22.343549+0000","last_fullsized":"2023-03-03T08:00:22.343549+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:06.529333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.361178+0000","last_change":"2023-03-03T08:00:22.355393+0000","last_active":"2023-03-03T08:00:22.361178+0000","last_peered":"2023-03-03T08:00:22.361178+0000","last_clean":"2023-03-03T08:00:22.361178+0000","last_became_active":"2023-03-03T08:00:22.354452+0000","last_became_peered":"2023-03-03T08:00:22.354452+0000","last_unstale":"2023-03-03T08:00:22.361178+0000","last_undegraded":"2023-03-03T08:00:22.361178+0000","last_fullsized":"2023-03-03T08:00:22.361178+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:08:44.952506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.359810+0000","last_change":"2023-03-03T08:00:22.350155+0000","last_active":"2023-03-03T08:00:22.359810+0000","last_peered":"2023-03-03T08:00:22.359810+0000","last_clean":"2023-03-03T08:00:22.359810+0000","last_became_active":"2023-03-03T08:00:22.349712+0000","last_became_peered":"2023-03-03T08:00:22.349712+0000","last_unstale":"2023-03-03T08:00:22.359810+0000","last_undegraded":"2023-03-03T08:00:22.359810+0000","last_fullsized":"2023-03-03T08:00:22.359810+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:06:54.326996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.362401+0000","last_change":"2023-03-03T08:00:22.352054+0000","last_active":"2023-03-03T08:00:22.362401+0000","last_peered":"2023-03-03T08:00:22.362401+0000","last_clean":"2023-03-03T08:00:22.362401+0000","last_became_active":"2023-03-03T08:00:22.351777+0000","last_became_peered":"2023-03-03T08:00:22.351777+0000","last_unstale":"2023-03-03T08:00:22.362401+0000","last_undegraded":"2023-03-03T08:00:22.362401+0000","last_fullsized":"2023-03-03T08:00:22.362401+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:06.529333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360855+0000","last_change":"2023-03-03T08:00:22.358047+0000","last_active":"2023-03-03T08:00:22.360855+0000","last_peered":"2023-03-03T08:00:22.360855+0000","last_clean":"2023-03-03T08:00:22.360855+0000","last_became_active":"2023-03-03T08:00:22.357730+0000","last_became_peered":"2023-03-03T08:00:22.357730+0000","last_unstale":"2023-03-03T08:00:22.360855+0000","last_undegraded":"2023-03-03T08:00:22.360855+0000","last_fullsized":"2023-03-03T08:00:22.360855+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:53:16.425693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356826+0000","last_change":"2023-03-03T08:00:22.347292+0000","last_active":"2023-03-03T08:00:22.356826+0000","last_peered":"2023-03-03T08:00:22.356826+0000","last_clean":"2023-03-03T08:00:22.356826+0000","last_became_active":"2023-03-03T08:00:22.346956+0000","last_became_peered":"2023-03-03T08:00:22.346956+0000","last_unstale":"2023-03-03T08:00:22.356826+0000","last_undegraded":"2023-03-03T08:00:22.356826+0000","last_fullsized":"2023-03-03T08:00:22.356826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:43.689368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.350844+0000","last_change":"2023-03-03T08:00:22.346466+0000","last_active":"2023-03-03T08:00:22.350844+0000","last_peered":"2023-03-03T08:00:22.350844+0000","last_clean":"2023-03-03T08:00:22.350844+0000","last_became_active":"2023-03-03T08:00:22.346127+0000","last_became_peered":"2023-03-03T08:00:22.346127+0000","last_unstale":"2023-03-03T08:00:22.350844+0000","last_undegraded":"2023-03-03T08:00:22.350844+0000","last_fullsized":"2023-03-03T08:00:22.350844+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:06.529333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368067+0000","last_change":"2023-03-03T08:00:22.355948+0000","last_active":"2023-03-03T08:00:22.368067+0000","last_peered":"2023-03-03T08:00:22.368067+0000","last_clean":"2023-03-03T08:00:22.368067+0000","last_became_active":"2023-03-03T08:00:22.355644+0000","last_became_peered":"2023-03-03T08:00:22.355644+0000","last_unstale":"2023-03-03T08:00:22.368067+0000","last_undegraded":"2023-03-03T08:00:22.368067+0000","last_fullsized":"2023-03-03T08:00:22.368067+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:40.317030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.361441+0000","last_change":"2023-03-03T08:00:22.352801+0000","last_active":"2023-03-03T08:00:22.361441+0000","last_peered":"2023-03-03T08:00:22.361441+0000","last_clean":"2023-03-03T08:00:22.361441+0000","last_became_active":"2023-03-03T08:00:22.352383+0000","last_became_peered":"2023-03-03T08:00:22.352383+0000","last_unstale":"2023-03-03T08:00:22.361441+0000","last_undegraded":"2023-03-03T08:00:22.361441+0000","last_fullsized":"2023-03-03T08:00:22.361441+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:28:15.779229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.364255+0000","last_change":"2023-03-03T08:00:22.357842+0000","last_active":"2023-03-03T08:00:22.364255+0000","last_peered":"2023-03-03T08:00:22.364255+0000","last_clean":"2023-03-03T08:00:22.364255+0000","last_became_active":"2023-03-03T08:00:22.357562+0000","last_became_peered":"2023-03-03T08:00:22.357562+0000","last_unstale":"2023-03-03T08:00:22.364255+0000","last_undegraded":"2023-03-03T08:00:22.364255+0000","last_fullsized":"2023-03-03T08:00:22.364255+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:25:55.351299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"65'4","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360098+0000","last_change":"2023-03-03T08:00:21.791637+0000","last_active":"2023-03-03T08:00:22.360098+0000","last_peered":"2023-03-03T08:00:22.360098+0000","last_clean":"2023-03-03T08:00:22.360098+0000","last_became_active":"2023-03-03T08:00:21.791370+0000","last_became_peered":"2023-03-03T08:00:21.791370+0000","last_unstale":"2023-03-03T08:00:22.360098+0000","last_undegraded":"2023-03-03T08:00:22.360098+0000","last_fullsized":"2023-03-03T08:00:22.360098+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:53:37.291520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368899+0000","last_change":"2023-03-03T08:00:22.363157+0000","last_active":"2023-03-03T08:00:22.368899+0000","last_peered":"2023-03-03T08:00:22.368899+0000","last_clean":"2023-03-03T08:00:22.368899+0000","last_became_active":"2023-03-03T08:00:22.362830+0000","last_became_peered":"2023-03-03T08:00:22.362830+0000","last_unstale":"2023-03-03T08:00:22.368899+0000","last_undegraded":"2023-03-03T08:00:22.368899+0000","last_fullsized":"2023-03-03T08:00:22.368899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:27.867451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"65'1","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.359464+0000","last_change":"2023-03-03T08:00:21.792187+0000","last_active":"2023-03-03T08:00:22.359464+0000","last_peered":"2023-03-03T08:00:22.359464+0000","last_clean":"2023-03-03T08:00:22.359464+0000","last_became_active":"2023-03-03T08:00:21.791813+0000","last_became_peered":"2023-03-03T08:00:21.791813+0000","last_unstale":"2023-03-03T08:00:22.359464+0000","last_undegraded":"2023-03-03T08:00:22.359464+0000","last_fullsized":"2023-03-03T08:00:22.359464+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:26.856771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.353039+0000","last_change":"2023-03-03T08:00:22.351663+0000","last_active":"2023-03-03T08:00:22.353039+0000","last_peered":"2023-03-03T08:00:22.353039+0000","last_clean":"2023-03-03T08:00:22.353039+0000","last_became_active":"2023-03-03T08:00:22.351260+0000","last_became_peered":"2023-03-03T08:00:22.351260+0000","last_unstale":"2023-03-03T08:00:22.353039+0000","last_undegraded":"2023-03-03T08:00:22.353039+0000","last_fullsized":"2023-03-03T08:00:22.353039+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:00:05.665712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.358144+0000","last_change":"2023-03-03T08:00:21.787824+0000","last_active":"2023-03-03T08:00:22.358144+0000","last_peered":"2023-03-03T08:00:22.358144+0000","last_clean":"2023-03-03T08:00:22.358144+0000","last_became_active":"2023-03-03T08:00:21.787512+0000","last_became_peered":"2023-03-03T08:00:21.787512+0000","last_unstale":"2023-03-03T08:00:22.358144+0000","last_undegraded":"2023-03-03T08:00:22.358144+0000","last_fullsized":"2023-03-03T08:00:22.358144+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:42.678687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.359901+0000","last_change":"2023-03-03T08:00:22.354134+0000","last_active":"2023-03-03T08:00:22.359901+0000","last_peered":"2023-03-03T08:00:22.359901+0000","last_clean":"2023-03-03T08:00:22.359901+0000","last_became_active":"2023-03-03T08:00:22.353810+0000","last_became_peered":"2023-03-03T08:00:22.353810+0000","last_unstale":"2023-03-03T08:00:22.359901+0000","last_undegraded":"2023-03-03T08:00:22.359901+0000","last_fullsized":"2023-03-03T08:00:22.359901+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:04.223183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.354232+0000","last_change":"2023-03-03T08:00:21.784586+0000","last_active":"2023-03-03T08:00:22.354232+0000","last_peered":"2023-03-03T08:00:22.354232+0000","last_clean":"2023-03-03T08:00:22.354232+0000","last_became_active":"2023-03-03T08:00:21.784312+0000","last_became_peered":"2023-03-03T08:00:21.784312+0000","last_unstale":"2023-03-03T08:00:22.354232+0000","last_undegraded":"2023-03-03T08:00:22.354232+0000","last_fullsized":"2023-03-03T08:00:22.354232+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:53:15.089322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360628+0000","last_change":"2023-03-03T08:00:22.356583+0000","last_active":"2023-03-03T08:00:22.360628+0000","last_peered":"2023-03-03T08:00:22.360628+0000","last_clean":"2023-03-03T08:00:22.360628+0000","last_became_active":"2023-03-03T08:00:22.356348+0000","last_became_peered":"2023-03-03T08:00:22.356348+0000","last_unstale":"2023-03-03T08:00:22.360628+0000","last_undegraded":"2023-03-03T08:00:22.360628+0000","last_fullsized":"2023-03-03T08:00:22.360628+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:43:23.319076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"65'1","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.348088+0000","last_change":"2023-03-03T08:00:21.778523+0000","last_active":"2023-03-03T08:00:22.348088+0000","last_peered":"2023-03-03T08:00:22.348088+0000","last_clean":"2023-03-03T08:00:22.348088+0000","last_became_active":"2023-03-03T08:00:21.778260+0000","last_became_peered":"2023-03-03T08:00:21.778260+0000","last_unstale":"2023-03-03T08:00:22.348088+0000","last_undegraded":"2023-03-03T08:00:22.348088+0000","last_fullsized":"2023-03-03T08:00:22.348088+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:23:26.856771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356934+0000","last_change":"2023-03-03T08:00:22.350090+0000","last_active":"2023-03-03T08:00:22.356934+0000","last_peered":"2023-03-03T08:00:22.356934+0000","last_clean":"2023-03-03T08:00:22.356934+0000","last_became_active":"2023-03-03T08:00:22.349808+0000","last_became_peered":"2023-03-03T08:00:22.349808+0000","last_unstale":"2023-03-03T08:00:22.356934+0000","last_undegraded":"2023-03-03T08:00:22.356934+0000","last_fullsized":"2023-03-03T08:00:22.356934+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:09.676363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"65'1","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.348542+0000","last_change":"2023-03-03T08:00:21.787110+0000","last_active":"2023-03-03T08:00:22.348542+0000","last_peered":"2023-03-03T08:00:22.348542+0000","last_clean":"2023-03-03T08:00:22.348542+0000","last_became_active":"2023-03-03T08:00:21.786811+0000","last_became_peered":"2023-03-03T08:00:21.786811+0000","last_unstale":"2023-03-03T08:00:22.348542+0000","last_undegraded":"2023-03-03T08:00:22.348542+0000","last_fullsized":"2023-03-03T08:00:22.348542+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:57:43.225652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.366830+0000","last_change":"2023-03-03T08:00:22.365220+0000","last_active":"2023-03-03T08:00:22.366830+0000","last_peered":"2023-03-03T08:00:22.366830+0000","last_clean":"2023-03-03T08:00:22.366830+0000","last_became_active":"2023-03-03T08:00:22.364613+0000","last_became_peered":"2023-03-03T08:00:22.364613+0000","last_unstale":"2023-03-03T08:00:22.366830+0000","last_undegraded":"2023-03-03T08:00:22.366830+0000","last_fullsized":"2023-03-03T08:00:22.366830+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:35:58.114402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.332133+0000","last_change":"2023-03-03T08:00:21.783560+0000","last_active":"2023-03-03T08:00:22.332133+0000","last_peered":"2023-03-03T08:00:22.332133+0000","last_clean":"2023-03-03T08:00:22.332133+0000","last_became_active":"2023-03-03T08:00:21.783190+0000","last_became_peered":"2023-03-03T08:00:21.783190+0000","last_unstale":"2023-03-03T08:00:22.332133+0000","last_undegraded":"2023-03-03T08:00:22.332133+0000","last_fullsized":"2023-03-03T08:00:22.332133+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:39.306350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.352314+0000","last_change":"2023-03-03T08:00:22.345314+0000","last_active":"2023-03-03T08:00:22.352314+0000","last_peered":"2023-03-03T08:00:22.352314+0000","last_clean":"2023-03-03T08:00:22.352314+0000","last_became_active":"2023-03-03T08:00:22.345077+0000","last_became_peered":"2023-03-03T08:00:22.345077+0000","last_unstale":"2023-03-03T08:00:22.352314+0000","last_undegraded":"2023-03-03T08:00:22.352314+0000","last_fullsized":"2023-03-03T08:00:22.352314+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:52.425703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.344936+0000","last_change":"2023-03-03T08:00:21.792376+0000","last_active":"2023-03-03T08:00:22.344936+0000","last_peered":"2023-03-03T08:00:22.344936+0000","last_clean":"2023-03-03T08:00:22.344936+0000","last_became_active":"2023-03-03T08:00:21.792126+0000","last_became_peered":"2023-03-03T08:00:21.792126+0000","last_unstale":"2023-03-03T08:00:22.344936+0000","last_undegraded":"2023-03-03T08:00:22.344936+0000","last_fullsized":"2023-03-03T08:00:22.344936+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:44.522714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356052+0000","last_change":"2023-03-03T08:00:22.351274+0000","last_active":"2023-03-03T08:00:22.356052+0000","last_peered":"2023-03-03T08:00:22.356052+0000","last_clean":"2023-03-03T08:00:22.356052+0000","last_became_active":"2023-03-03T08:00:22.351025+0000","last_became_peered":"2023-03-03T08:00:22.351025+0000","last_unstale":"2023-03-03T08:00:22.356052+0000","last_undegraded":"2023-03-03T08:00:22.356052+0000","last_fullsized":"2023-03-03T08:00:22.356052+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:55:49.478050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.331929+0000","last_change":"2023-03-03T08:00:21.790545+0000","last_active":"2023-03-03T08:00:22.331929+0000","last_peered":"2023-03-03T08:00:22.331929+0000","last_clean":"2023-03-03T08:00:22.331929+0000","last_became_active":"2023-03-03T08:00:21.790052+0000","last_became_peered":"2023-03-03T08:00:21.790052+0000","last_unstale":"2023-03-03T08:00:22.331929+0000","last_undegraded":"2023-03-03T08:00:22.331929+0000","last_fullsized":"2023-03-03T08:00:22.331929+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:51:33.987978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.358127+0000","last_change":"2023-03-03T08:00:22.353413+0000","last_active":"2023-03-03T08:00:22.358127+0000","last_peered":"2023-03-03T08:00:22.358127+0000","last_clean":"2023-03-03T08:00:22.358127+0000","last_became_active":"2023-03-03T08:00:22.353114+0000","last_became_peered":"2023-03-03T08:00:22.353114+0000","last_unstale":"2023-03-03T08:00:22.358127+0000","last_undegraded":"2023-03-03T08:00:22.358127+0000","last_fullsized":"2023-03-03T08:00:22.358127+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:34.298493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.342606+0000","last_change":"2023-03-03T08:00:21.783602+0000","last_active":"2023-03-03T08:00:22.342606+0000","last_peered":"2023-03-03T08:00:22.342606+0000","last_clean":"2023-03-03T08:00:22.342606+0000","last_became_active":"2023-03-03T08:00:21.783297+0000","last_became_peered":"2023-03-03T08:00:21.783297+0000","last_unstale":"2023-03-03T08:00:22.342606+0000","last_undegraded":"2023-03-03T08:00:22.342606+0000","last_fullsized":"2023-03-03T08:00:22.342606+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:08:45.794437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.369192+0000","last_change":"2023-03-03T08:00:22.367931+0000","last_active":"2023-03-03T08:00:22.369192+0000","last_peered":"2023-03-03T08:00:22.369192+0000","last_clean":"2023-03-03T08:00:22.369192+0000","last_became_active":"2023-03-03T08:00:22.367374+0000","last_became_peered":"2023-03-03T08:00:22.367374+0000","last_unstale":"2023-03-03T08:00:22.369192+0000","last_undegraded":"2023-03-03T08:00:22.369192+0000","last_fullsized":"2023-03-03T08:00:22.369192+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:12:21.211704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"65'3","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368407+0000","last_change":"2023-03-03T08:00:21.790207+0000","last_active":"2023-03-03T08:00:22.368407+0000","last_peered":"2023-03-03T08:00:22.368407+0000","last_clean":"2023-03-03T08:00:22.368407+0000","last_became_active":"2023-03-03T08:00:21.789535+0000","last_became_peered":"2023-03-03T08:00:21.789535+0000","last_unstale":"2023-03-03T08:00:22.368407+0000","last_undegraded":"2023-03-03T08:00:22.368407+0000","last_fullsized":"2023-03-03T08:00:22.368407+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:02.228725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.347507+0000","last_change":"2023-03-03T08:00:22.343895+0000","last_active":"2023-03-03T08:00:22.347507+0000","last_peered":"2023-03-03T08:00:22.347507+0000","last_clean":"2023-03-03T08:00:22.347507+0000","last_became_active":"2023-03-03T08:00:22.343577+0000","last_became_peered":"2023-03-03T08:00:22.343577+0000","last_unstale":"2023-03-03T08:00:22.347507+0000","last_undegraded":"2023-03-03T08:00:22.347507+0000","last_fullsized":"2023-03-03T08:00:22.347507+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:20.734366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"65'6","reported_seq":17,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.370487+0000","last_change":"2023-03-03T08:00:21.789108+0000","last_active":"2023-03-03T08:00:22.370487+0000","last_peered":"2023-03-03T08:00:22.370487+0000","last_clean":"2023-03-03T08:00:22.370487+0000","last_became_active":"2023-03-03T08:00:21.788850+0000","last_became_peered":"2023-03-03T08:00:21.788850+0000","last_unstale":"2023-03-03T08:00:22.370487+0000","last_undegraded":"2023-03-03T08:00:22.370487+0000","last_fullsized":"2023-03-03T08:00:22.370487+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:16:16.126986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368203+0000","last_change":"2023-03-03T08:00:22.358537+0000","last_active":"2023-03-03T08:00:22.368203+0000","last_peered":"2023-03-03T08:00:22.368203+0000","last_clean":"2023-03-03T08:00:22.368203+0000","last_became_active":"2023-03-03T08:00:22.358277+0000","last_became_peered":"2023-03-03T08:00:22.358277+0000","last_unstale":"2023-03-03T08:00:22.368203+0000","last_undegraded":"2023-03-03T08:00:22.368203+0000","last_fullsized":"2023-03-03T08:00:22.368203+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:06:54.326996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.366491+0000","last_change":"2023-03-03T08:00:21.788681+0000","last_active":"2023-03-03T08:00:22.366491+0000","last_peered":"2023-03-03T08:00:22.366491+0000","last_clean":"2023-03-03T08:00:22.366491+0000","last_became_active":"2023-03-03T08:00:21.788087+0000","last_became_peered":"2023-03-03T08:00:21.788087+0000","last_unstale":"2023-03-03T08:00:22.366491+0000","last_undegraded":"2023-03-03T08:00:22.366491+0000","last_fullsized":"2023-03-03T08:00:22.366491+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:08:45.794437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.350067+0000","last_change":"2023-03-03T08:00:22.346539+0000","last_active":"2023-03-03T08:00:22.350067+0000","last_peered":"2023-03-03T08:00:22.350067+0000","last_clean":"2023-03-03T08:00:22.350067+0000","last_became_active":"2023-03-03T08:00:22.346149+0000","last_became_peered":"2023-03-03T08:00:22.346149+0000","last_unstale":"2023-03-03T08:00:22.350067+0000","last_undegraded":"2023-03-03T08:00:22.350067+0000","last_fullsized":"2023-03-03T08:00:22.350067+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:20.734366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.334121+0000","last_change":"2023-03-03T08:00:21.787442+0000","last_active":"2023-03-03T08:00:22.334121+0000","last_peered":"2023-03-03T08:00:22.334121+0000","last_clean":"2023-03-03T08:00:22.334121+0000","last_became_active":"2023-03-03T08:00:21.786782+0000","last_became_peered":"2023-03-03T08:00:21.786782+0000","last_unstale":"2023-03-03T08:00:22.334121+0000","last_undegraded":"2023-03-03T08:00:22.334121+0000","last_fullsized":"2023-03-03T08:00:22.334121+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:55:48.684553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.365840+0000","last_change":"2023-03-03T08:00:22.363185+0000","last_active":"2023-03-03T08:00:22.365840+0000","last_peered":"2023-03-03T08:00:22.365840+0000","last_clean":"2023-03-03T08:00:22.365840+0000","last_became_active":"2023-03-03T08:00:22.362886+0000","last_became_peered":"2023-03-03T08:00:22.362886+0000","last_unstale":"2023-03-03T08:00:22.365840+0000","last_undegraded":"2023-03-03T08:00:22.365840+0000","last_fullsized":"2023-03-03T08:00:22.365840+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:16:17.137666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.331162+0000","last_change":"2023-03-03T08:00:21.787586+0000","last_active":"2023-03-03T08:00:22.331162+0000","last_peered":"2023-03-03T08:00:22.331162+0000","last_clean":"2023-03-03T08:00:22.331162+0000","last_became_active":"2023-03-03T08:00:21.787272+0000","last_became_peered":"2023-03-03T08:00:21.787272+0000","last_unstale":"2023-03-03T08:00:22.331162+0000","last_undegraded":"2023-03-03T08:00:22.331162+0000","last_fullsized":"2023-03-03T08:00:22.331162+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:12.459278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.355207+0000","last_change":"2023-03-03T08:00:22.349923+0000","last_active":"2023-03-03T08:00:22.355207+0000","last_peered":"2023-03-03T08:00:22.355207+0000","last_clean":"2023-03-03T08:00:22.355207+0000","last_became_active":"2023-03-03T08:00:22.349552+0000","last_became_peered":"2023-03-03T08:00:22.349552+0000","last_unstale":"2023-03-03T08:00:22.355207+0000","last_undegraded":"2023-03-03T08:00:22.355207+0000","last_fullsized":"2023-03-03T08:00:22.355207+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:36:19.801740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"65'8","reported_seq":19,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.364523+0000","last_change":"2023-03-03T08:00:21.791488+0000","last_active":"2023-03-03T08:00:22.364523+0000","last_peered":"2023-03-03T08:00:22.364523+0000","last_clean":"2023-03-03T08:00:22.364523+0000","last_became_active":"2023-03-03T08:00:21.791122+0000","last_became_peered":"2023-03-03T08:00:21.791122+0000","last_unstale":"2023-03-03T08:00:22.364523+0000","last_undegraded":"2023-03-03T08:00:22.364523+0000","last_fullsized":"2023-03-03T08:00:22.364523+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:36:55.429526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.353072+0000","last_change":"2023-03-03T08:00:22.348199+0000","last_active":"2023-03-03T08:00:22.353072+0000","last_peered":"2023-03-03T08:00:22.353072+0000","last_clean":"2023-03-03T08:00:22.353072+0000","last_became_active":"2023-03-03T08:00:22.347841+0000","last_became_peered":"2023-03-03T08:00:22.347841+0000","last_unstale":"2023-03-03T08:00:22.353072+0000","last_undegraded":"2023-03-03T08:00:22.353072+0000","last_fullsized":"2023-03-03T08:00:22.353072+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:52.425703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.361110+0000","last_change":"2023-03-03T08:00:21.792101+0000","last_active":"2023-03-03T08:00:22.361110+0000","last_peered":"2023-03-03T08:00:22.361110+0000","last_clean":"2023-03-03T08:00:22.361110+0000","last_became_active":"2023-03-03T08:00:21.791481+0000","last_became_peered":"2023-03-03T08:00:21.791481+0000","last_unstale":"2023-03-03T08:00:22.361110+0000","last_undegraded":"2023-03-03T08:00:22.361110+0000","last_fullsized":"2023-03-03T08:00:22.361110+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:12:20.201024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360729+0000","last_change":"2023-03-03T08:00:22.358310+0000","last_active":"2023-03-03T08:00:22.360729+0000","last_peered":"2023-03-03T08:00:22.360729+0000","last_clean":"2023-03-03T08:00:22.360729+0000","last_became_active":"2023-03-03T08:00:22.358000+0000","last_became_peered":"2023-03-03T08:00:22.358000+0000","last_unstale":"2023-03-03T08:00:22.360729+0000","last_undegraded":"2023-03-03T08:00:22.360729+0000","last_fullsized":"2023-03-03T08:00:22.360729+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:23:00.594155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"66'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-03-03T08:00:23.335263+0000","last_change":"2023-03-03T08:00:21.786969+0000","last_active":"2023-03-03T08:00:23.335263+0000","last_peered":"2023-03-03T08:00:23.335263+0000","last_clean":"2023-03-03T08:00:23.335263+0000","last_became_active":"2023-03-03T08:00:21.786545+0000","last_became_peered":"2023-03-03T08:00:21.786545+0000","last_unstale":"2023-03-03T08:00:23.335263+0000","last_undegraded":"2023-03-03T08:00:23.335263+0000","last_fullsized":"2023-03-03T08:00:23.335263+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:16:25.213480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.352179+0000","last_change":"2023-03-03T08:00:22.348136+0000","last_active":"2023-03-03T08:00:22.352179+0000","last_peered":"2023-03-03T08:00:22.352179+0000","last_clean":"2023-03-03T08:00:22.352179+0000","last_became_active":"2023-03-03T08:00:22.347818+0000","last_became_peered":"2023-03-03T08:00:22.347818+0000","last_unstale":"2023-03-03T08:00:22.352179+0000","last_undegraded":"2023-03-03T08:00:22.352179+0000","last_fullsized":"2023-03-03T08:00:22.352179+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:11:16.603939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.332356+0000","last_change":"2023-03-03T08:00:21.788125+0000","last_active":"2023-03-03T08:00:22.332356+0000","last_peered":"2023-03-03T08:00:22.332356+0000","last_clean":"2023-03-03T08:00:22.332356+0000","last_became_active":"2023-03-03T08:00:21.787862+0000","last_became_peered":"2023-03-03T08:00:21.787862+0000","last_unstale":"2023-03-03T08:00:22.332356+0000","last_undegraded":"2023-03-03T08:00:22.332356+0000","last_fullsized":"2023-03-03T08:00:22.332356+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:03.212503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.364339+0000","last_change":"2023-03-03T08:00:22.356604+0000","last_active":"2023-03-03T08:00:22.364339+0000","last_peered":"2023-03-03T08:00:22.364339+0000","last_clean":"2023-03-03T08:00:22.364339+0000","last_became_active":"2023-03-03T08:00:22.356343+0000","last_became_peered":"2023-03-03T08:00:22.356343+0000","last_unstale":"2023-03-03T08:00:22.364339+0000","last_undegraded":"2023-03-03T08:00:22.364339+0000","last_fullsized":"2023-03-03T08:00:22.364339+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:49:26.845615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.336881+0000","last_change":"2023-03-03T08:00:21.789082+0000","last_active":"2023-03-03T08:00:22.336881+0000","last_peered":"2023-03-03T08:00:22.336881+0000","last_clean":"2023-03-03T08:00:22.336881+0000","last_became_active":"2023-03-03T08:00:21.788827+0000","last_became_peered":"2023-03-03T08:00:21.788827+0000","last_unstale":"2023-03-03T08:00:22.336881+0000","last_undegraded":"2023-03-03T08:00:22.336881+0000","last_fullsized":"2023-03-03T08:00:22.336881+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:58.454988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.364373+0000","last_change":"2023-03-03T08:00:22.357855+0000","last_active":"2023-03-03T08:00:22.364373+0000","last_peered":"2023-03-03T08:00:22.364373+0000","last_clean":"2023-03-03T08:00:22.364373+0000","last_became_active":"2023-03-03T08:00:22.357512+0000","last_became_peered":"2023-03-03T08:00:22.357512+0000","last_unstale":"2023-03-03T08:00:22.364373+0000","last_undegraded":"2023-03-03T08:00:22.364373+0000","last_fullsized":"2023-03-03T08:00:22.364373+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:53:38.302201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.331929+0000","last_change":"2023-03-03T08:00:21.788909+0000","last_active":"2023-03-03T08:00:22.331929+0000","last_peered":"2023-03-03T08:00:22.331929+0000","last_clean":"2023-03-03T08:00:22.331929+0000","last_became_active":"2023-03-03T08:00:21.788395+0000","last_became_peered":"2023-03-03T08:00:21.788395+0000","last_unstale":"2023-03-03T08:00:22.331929+0000","last_undegraded":"2023-03-03T08:00:22.331929+0000","last_fullsized":"2023-03-03T08:00:22.331929+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:17:29.803501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368951+0000","last_change":"2023-03-03T08:00:22.363754+0000","last_active":"2023-03-03T08:00:22.368951+0000","last_peered":"2023-03-03T08:00:22.368951+0000","last_clean":"2023-03-03T08:00:22.368951+0000","last_became_active":"2023-03-03T08:00:22.363463+0000","last_became_peered":"2023-03-03T08:00:22.363463+0000","last_unstale":"2023-03-03T08:00:22.368951+0000","last_undegraded":"2023-03-03T08:00:22.368951+0000","last_fullsized":"2023-03-03T08:00:22.368951+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:08:44.952506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.332741+0000","last_change":"2023-03-03T08:00:21.782978+0000","last_active":"2023-03-03T08:00:22.332741+0000","last_peered":"2023-03-03T08:00:22.332741+0000","last_clean":"2023-03-03T08:00:22.332741+0000","last_became_active":"2023-03-03T08:00:21.782679+0000","last_became_peered":"2023-03-03T08:00:21.782679+0000","last_unstale":"2023-03-03T08:00:22.332741+0000","last_undegraded":"2023-03-03T08:00:22.332741+0000","last_fullsized":"2023-03-03T08:00:22.332741+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:19:00.431727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.354447+0000","last_change":"2023-03-03T08:00:22.348976+0000","last_active":"2023-03-03T08:00:22.354447+0000","last_peered":"2023-03-03T08:00:22.354447+0000","last_clean":"2023-03-03T08:00:22.354447+0000","last_became_active":"2023-03-03T08:00:22.348701+0000","last_became_peered":"2023-03-03T08:00:22.348701+0000","last_unstale":"2023-03-03T08:00:22.354447+0000","last_undegraded":"2023-03-03T08:00:22.354447+0000","last_fullsized":"2023-03-03T08:00:22.354447+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:51:34.998659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.332229+0000","last_change":"2023-03-03T08:00:21.783463+0000","last_active":"2023-03-03T08:00:22.332229+0000","last_peered":"2023-03-03T08:00:22.332229+0000","last_clean":"2023-03-03T08:00:22.332229+0000","last_became_active":"2023-03-03T08:00:21.782809+0000","last_became_peered":"2023-03-03T08:00:21.782809+0000","last_unstale":"2023-03-03T08:00:22.332229+0000","last_undegraded":"2023-03-03T08:00:22.332229+0000","last_fullsized":"2023-03-03T08:00:22.332229+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:44:03.166814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360566+0000","last_change":"2023-03-03T08:00:22.355471+0000","last_active":"2023-03-03T08:00:22.360566+0000","last_peered":"2023-03-03T08:00:22.360566+0000","last_clean":"2023-03-03T08:00:22.360566+0000","last_became_active":"2023-03-03T08:00:22.355175+0000","last_became_peered":"2023-03-03T08:00:22.355175+0000","last_unstale":"2023-03-03T08:00:22.360566+0000","last_undegraded":"2023-03-03T08:00:22.360566+0000","last_fullsized":"2023-03-03T08:00:22.360566+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:22:43.504725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.334926+0000","last_change":"2023-03-03T08:00:21.784441+0000","last_active":"2023-03-03T08:00:22.334926+0000","last_peered":"2023-03-03T08:00:22.334926+0000","last_clean":"2023-03-03T08:00:22.334926+0000","last_became_active":"2023-03-03T08:00:21.784193+0000","last_became_peered":"2023-03-03T08:00:21.784193+0000","last_unstale":"2023-03-03T08:00:22.334926+0000","last_undegraded":"2023-03-03T08:00:22.334926+0000","last_fullsized":"2023-03-03T08:00:22.334926+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:00:08.298430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356321+0000","last_change":"2023-03-03T08:00:22.346849+0000","last_active":"2023-03-03T08:00:22.356321+0000","last_peered":"2023-03-03T08:00:22.356321+0000","last_clean":"2023-03-03T08:00:22.356321+0000","last_became_active":"2023-03-03T08:00:22.346557+0000","last_became_peered":"2023-03-03T08:00:22.346557+0000","last_unstale":"2023-03-03T08:00:22.356321+0000","last_undegraded":"2023-03-03T08:00:22.356321+0000","last_fullsized":"2023-03-03T08:00:22.356321+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:23:40.317030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"65'1","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360262+0000","last_change":"2023-03-03T08:00:21.792370+0000","last_active":"2023-03-03T08:00:22.360262+0000","last_peered":"2023-03-03T08:00:22.360262+0000","last_clean":"2023-03-03T08:00:22.360262+0000","last_became_active":"2023-03-03T08:00:21.791836+0000","last_became_peered":"2023-03-03T08:00:21.791836+0000","last_unstale":"2023-03-03T08:00:22.360262+0000","last_undegraded":"2023-03-03T08:00:22.360262+0000","last_fullsized":"2023-03-03T08:00:22.360262+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:42:51.458743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.336780+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T08:00:22.336780+0000","last_peered":"2023-03-03T08:00:22.336780+0000","last_clean":"2023-03-03T08:00:22.336780+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T08:00:22.336780+0000","last_undegraded":"2023-03-03T08:00:22.336780+0000","last_fullsized":"2023-03-03T08:00:22.336780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.366849+0000","last_change":"2023-03-03T08:00:22.365438+0000","last_active":"2023-03-03T08:00:22.366849+0000","last_peered":"2023-03-03T08:00:22.366849+0000","last_clean":"2023-03-03T08:00:22.366849+0000","last_became_active":"2023-03-03T08:00:22.364935+0000","last_became_peered":"2023-03-03T08:00:22.364935+0000","last_unstale":"2023-03-03T08:00:22.366849+0000","last_undegraded":"2023-03-03T08:00:22.366849+0000","last_fullsized":"2023-03-03T08:00:22.366849+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:59.465669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.331666+0000","last_change":"2023-03-03T08:00:21.792272+0000","last_active":"2023-03-03T08:00:22.331666+0000","last_peered":"2023-03-03T08:00:22.331666+0000","last_clean":"2023-03-03T08:00:22.331666+0000","last_became_active":"2023-03-03T08:00:21.791961+0000","last_became_peered":"2023-03-03T08:00:21.791961+0000","last_unstale":"2023-03-03T08:00:22.331666+0000","last_undegraded":"2023-03-03T08:00:22.331666+0000","last_fullsized":"2023-03-03T08:00:22.331666+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:57:43.225652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":51,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.334339+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T08:00:22.334339+0000","last_peered":"2023-03-03T08:00:22.334339+0000","last_clean":"2023-03-03T08:00:22.334339+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T08:00:22.334339+0000","last_undegraded":"2023-03-03T08:00:22.334339+0000","last_fullsized":"2023-03-03T08:00:22.334339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.360169+0000","last_change":"2023-03-03T08:00:22.354435+0000","last_active":"2023-03-03T08:00:22.360169+0000","last_peered":"2023-03-03T08:00:22.360169+0000","last_clean":"2023-03-03T08:00:22.360169+0000","last_became_active":"2023-03-03T08:00:22.354144+0000","last_became_peered":"2023-03-03T08:00:22.354144+0000","last_unstale":"2023-03-03T08:00:22.360169+0000","last_undegraded":"2023-03-03T08:00:22.360169+0000","last_fullsized":"2023-03-03T08:00:22.360169+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:39.487708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.334241+0000","last_change":"2023-03-03T08:00:21.787607+0000","last_active":"2023-03-03T08:00:22.334241+0000","last_peered":"2023-03-03T08:00:22.334241+0000","last_clean":"2023-03-03T08:00:22.334241+0000","last_became_active":"2023-03-03T08:00:21.787106+0000","last_became_peered":"2023-03-03T08:00:21.787106+0000","last_unstale":"2023-03-03T08:00:22.334241+0000","last_undegraded":"2023-03-03T08:00:22.334241+0000","last_fullsized":"2023-03-03T08:00:22.334241+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:30:19.298647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.343813+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T08:00:22.343813+0000","last_peered":"2023-03-03T08:00:22.343813+0000","last_clean":"2023-03-03T08:00:22.343813+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T08:00:22.343813+0000","last_undegraded":"2023-03-03T08:00:22.343813+0000","last_fullsized":"2023-03-03T08:00:22.343813+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.366899+0000","last_change":"2023-03-03T08:00:22.365234+0000","last_active":"2023-03-03T08:00:22.366899+0000","last_peered":"2023-03-03T08:00:22.366899+0000","last_clean":"2023-03-03T08:00:22.366899+0000","last_became_active":"2023-03-03T08:00:22.364559+0000","last_became_peered":"2023-03-03T08:00:22.364559+0000","last_unstale":"2023-03-03T08:00:22.366899+0000","last_undegraded":"2023-03-03T08:00:22.366899+0000","last_fullsized":"2023-03-03T08:00:22.366899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:00:09.309111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.331737+0000","last_change":"2023-03-03T08:00:21.775813+0000","last_active":"2023-03-03T08:00:22.331737+0000","last_peered":"2023-03-03T08:00:22.331737+0000","last_clean":"2023-03-03T08:00:22.331737+0000","last_became_active":"2023-03-03T08:00:21.775553+0000","last_became_peered":"2023-03-03T08:00:21.775553+0000","last_unstale":"2023-03-03T08:00:22.331737+0000","last_undegraded":"2023-03-03T08:00:22.331737+0000","last_fullsized":"2023-03-03T08:00:22.331737+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:19.710326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.335082+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T08:00:22.335082+0000","last_peered":"2023-03-03T08:00:22.335082+0000","last_clean":"2023-03-03T08:00:22.335082+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T08:00:22.335082+0000","last_undegraded":"2023-03-03T08:00:22.335082+0000","last_fullsized":"2023-03-03T08:00:22.335082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":205,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.330930+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T08:00:22.330930+0000","last_peered":"2023-03-03T08:00:22.330930+0000","last_clean":"2023-03-03T08:00:22.330930+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T08:00:22.330930+0000","last_undegraded":"2023-03-03T08:00:22.330930+0000","last_fullsized":"2023-03-03T08:00:22.330930+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.356401+0000","last_change":"2023-03-03T08:00:22.352632+0000","last_active":"2023-03-03T08:00:22.356401+0000","last_peered":"2023-03-03T08:00:22.356401+0000","last_clean":"2023-03-03T08:00:22.356401+0000","last_became_active":"2023-03-03T08:00:22.351927+0000","last_became_peered":"2023-03-03T08:00:22.351927+0000","last_unstale":"2023-03-03T08:00:22.356401+0000","last_undegraded":"2023-03-03T08:00:22.356401+0000","last_fullsized":"2023-03-03T08:00:22.356401+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:27.364263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.344601+0000","last_change":"2023-03-03T08:00:21.790111+0000","last_active":"2023-03-03T08:00:22.344601+0000","last_peered":"2023-03-03T08:00:22.344601+0000","last_clean":"2023-03-03T08:00:22.344601+0000","last_became_active":"2023-03-03T08:00:21.789831+0000","last_became_peered":"2023-03-03T08:00:21.789831+0000","last_unstale":"2023-03-03T08:00:22.344601+0000","last_undegraded":"2023-03-03T08:00:22.344601+0000","last_fullsized":"2023-03-03T08:00:22.344601+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:19.710326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":52,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.330895+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T08:00:22.330895+0000","last_peered":"2023-03-03T08:00:22.330895+0000","last_clean":"2023-03-03T08:00:22.330895+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T08:00:22.330895+0000","last_undegraded":"2023-03-03T08:00:22.330895+0000","last_fullsized":"2023-03-03T08:00:22.330895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.365925+0000","last_change":"2023-03-03T08:00:22.363947+0000","last_active":"2023-03-03T08:00:22.365925+0000","last_peered":"2023-03-03T08:00:22.365925+0000","last_clean":"2023-03-03T08:00:22.365925+0000","last_became_active":"2023-03-03T08:00:22.363477+0000","last_became_peered":"2023-03-03T08:00:22.363477+0000","last_unstale":"2023-03-03T08:00:22.365925+0000","last_undegraded":"2023-03-03T08:00:22.365925+0000","last_fullsized":"2023-03-03T08:00:22.365925+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:30:20.309327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"65'2","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.369342+0000","last_change":"2023-03-03T08:00:21.787378+0000","last_active":"2023-03-03T08:00:22.369342+0000","last_peered":"2023-03-03T08:00:22.369342+0000","last_clean":"2023-03-03T08:00:22.369342+0000","last_became_active":"2023-03-03T08:00:21.787102+0000","last_became_peered":"2023-03-03T08:00:21.787102+0000","last_unstale":"2023-03-03T08:00:22.369342+0000","last_undegraded":"2023-03-03T08:00:22.369342+0000","last_fullsized":"2023-03-03T08:00:22.369342+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:45:29.829197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":52,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.332485+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T08:00:22.332485+0000","last_peered":"2023-03-03T08:00:22.332485+0000","last_clean":"2023-03-03T08:00:22.332485+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T08:00:22.332485+0000","last_undegraded":"2023-03-03T08:00:22.332485+0000","last_fullsized":"2023-03-03T08:00:22.332485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.369032+0000","last_change":"2023-03-03T08:00:22.366608+0000","last_active":"2023-03-03T08:00:22.369032+0000","last_peered":"2023-03-03T08:00:22.369032+0000","last_clean":"2023-03-03T08:00:22.369032+0000","last_became_active":"2023-03-03T08:00:22.366325+0000","last_became_peered":"2023-03-03T08:00:22.366325+0000","last_unstale":"2023-03-03T08:00:22.369032+0000","last_undegraded":"2023-03-03T08:00:22.369032+0000","last_fullsized":"2023-03-03T08:00:22.369032+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:37:20.721006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"65'1","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.368284+0000","last_change":"2023-03-03T08:00:21.789735+0000","last_active":"2023-03-03T08:00:22.368284+0000","last_peered":"2023-03-03T08:00:22.368284+0000","last_clean":"2023-03-03T08:00:22.368284+0000","last_became_active":"2023-03-03T08:00:21.789201+0000","last_became_peered":"2023-03-03T08:00:21.789201+0000","last_unstale":"2023-03-03T08:00:22.368284+0000","last_undegraded":"2023-03-03T08:00:22.368284+0000","last_fullsized":"2023-03-03T08:00:22.368284+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:08.665683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.337975+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T08:00:22.337975+0000","last_peered":"2023-03-03T08:00:22.337975+0000","last_clean":"2023-03-03T08:00:22.337975+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T08:00:22.337975+0000","last_undegraded":"2023-03-03T08:00:22.337975+0000","last_fullsized":"2023-03-03T08:00:22.337975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.364313+0000","last_change":"2023-03-03T08:00:22.356451+0000","last_active":"2023-03-03T08:00:22.364313+0000","last_peered":"2023-03-03T08:00:22.364313+0000","last_clean":"2023-03-03T08:00:22.364313+0000","last_became_active":"2023-03-03T08:00:22.356191+0000","last_became_peered":"2023-03-03T08:00:22.356191+0000","last_unstale":"2023-03-03T08:00:22.364313+0000","last_undegraded":"2023-03-03T08:00:22.364313+0000","last_fullsized":"2023-03-03T08:00:22.364313+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:21.758670+0000","last_clean_scrub_stamp":"2023-03-03T08:00:21.758670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:17:07.748684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.333217+0000","last_change":"2023-03-03T08:00:21.778956+0000","last_active":"2023-03-03T08:00:22.333217+0000","last_peered":"2023-03-03T08:00:22.333217+0000","last_clean":"2023-03-03T08:00:22.333217+0000","last_became_active":"2023-03-03T08:00:21.778666+0000","last_became_peered":"2023-03-03T08:00:21.778666+0000","last_unstale":"2023-03-03T08:00:22.333217+0000","last_undegraded":"2023-03-03T08:00:22.333217+0000","last_fullsized":"2023-03-03T08:00:22.333217+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:00:20.747989+0000","last_clean_scrub_stamp":"2023-03-03T08:00:20.747989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:05.518653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-03-03T08:00:22.349803+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T08:00:22.349803+0000","last_peered":"2023-03-03T08:00:22.349803+0000","last_clean":"2023-03-03T08:00:22.349803+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T08:00:22.349803+0000","last_undegraded":"2023-03-03T08:00:22.349803+0000","last_fullsized":"2023-03-03T08:00:22.349803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574910,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10980,"kb_used_data":1372,"kb_used_omap":0,"kb_used_meta":9600,"kb_avail":94360860,"statfs":{"total":96636764160,"available":96625520640,"internally_reserved":0,"allocated":1404928,"data_stored":943239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9830400},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.67500000000000004,"15min":0.70499999999999996},"min":{"1min":0.437,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82499999999999996,"5min":1.546,"15min":1.546},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71199999999999997,"15min":0.72999999999999998},"min":{"1min":0.50600000000000001,"5min":0.5,"15min":0.5},"max":{"1min":0.96499999999999997,"5min":1.702,"15min":1.702},"last":0.81499999999999995}]},{"osd":1,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.89000000000000001,"15min":1.0840000000000001},"min":{"1min":0.49199999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.111,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.755,"5min":0.74099999999999999,"15min":0.76900000000000002},"min":{"1min":0.51600000000000001,"5min":0.504,"15min":0.504},"max":{"1min":1.1890000000000001,"5min":1.944,"15min":1.944},"last":0.66900000000000004}]},{"osd":2,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.71899999999999997,"15min":0.71399999999999997},"min":{"1min":0.63300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.224,"5min":1.252,"15min":1.252},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76800000000000002,"15min":0.79300000000000004},"min":{"1min":0.54600000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2490000000000001,"5min":1.329,"15min":1.329},"last":0.76400000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.71999999999999997,"15min":0.71499999999999997},"min":{"1min":0.59799999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.754,"15min":0.79000000000000004},"min":{"1min":0.55000000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.93400000000000005,"5min":1.294,"15min":1.294},"last":0.79800000000000004}]},{"osd":4,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.48399999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0049999999999999,"5min":1.077,"15min":1.077},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.66700000000000004,"15min":0.63400000000000001},"min":{"1min":0.51700000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.77500000000000002}]},{"osd":5,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.77100000000000002,"15min":0.80100000000000005},"min":{"1min":0.41699999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.036,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81699999999999995,"15min":0.82399999999999995},"min":{"1min":0.66400000000000003,"5min":0.496,"15min":0.496},"max":{"1min":1.2190000000000001,"5min":1.349,"15min":1.349},"last":0.79000000000000004}]},{"osd":6,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.64100000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0700000000000001,"5min":1.204,"15min":1.204},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.81100000000000005,"15min":0.82099999999999995},"min":{"1min":0.627,"5min":0.499,"15min":0.499},"max":{"1min":1.135,"5min":1.194,"15min":1.194},"last":0.70799999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9424,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":8640,"kb_avail":94362416,"statfs":{"total":96636764160,"available":96627113984,"internally_reserved":0,"allocated":794624,"data_stored":351126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8847360},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.70799999999999996,"15min":0.68899999999999995},"min":{"1min":0.57199999999999995,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.018,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68999999999999995,"15min":0.69499999999999995},"min":{"1min":0.52500000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.89700000000000002,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.745}]},{"osd":1,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70999999999999996,"15min":0.68600000000000005},"min":{"1min":0.52700000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.97099999999999997,"5min":3.323,"15min":3.323},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.73499999999999999,"15min":0.73099999999999998},"min":{"1min":0.55400000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97199999999999998,"5min":1.135,"15min":1.135},"last":0.63700000000000001}]},{"osd":2,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.753,"15min":0.73199999999999998},"min":{"1min":0.59499999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.98899999999999999,"5min":1.54,"15min":1.54},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.749,"15min":0.75900000000000001},"min":{"1min":0.54400000000000004,"5min":0.495,"15min":0.495},"max":{"1min":1.0089999999999999,"5min":1.698,"15min":1.698},"last":0.71799999999999997}]},{"osd":3,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81299999999999994,"15min":0.78900000000000003},"min":{"1min":0.57499999999999996,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.3089999999999999,"5min":4.8959999999999999,"15min":4.8959999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77200000000000002,"15min":0.77100000000000002},"min":{"1min":0.60899999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.95799999999999996,"5min":1.343,"15min":1.343},"last":0.78600000000000003}]},{"osd":4,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.79900000000000004,"15min":0.91100000000000003},"min":{"1min":0.439,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.871,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.80700000000000005,"15min":0.92400000000000004},"min":{"1min":0.53000000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.85099999999999998,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.88900000000000001}]},{"osd":5,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.72599999999999998,"15min":0.77600000000000002},"min":{"1min":0.47599999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.95399999999999996,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.80300000000000005,"15min":0.82499999999999996},"min":{"1min":0.499,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91500000000000004,"5min":1.429,"15min":1.429},"last":0.69499999999999995}]},{"osd":7,"last update":"Fri Mar 3 07:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.81899999999999995,"15min":0.83899999999999997},"min":{"1min":0.56699999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.0029999999999999,"5min":1.452,"15min":1.452},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.82099999999999995,"15min":0.87},"min":{"1min":0.59399999999999997,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.008,"5min":1.579,"15min":1.579},"last":0.81299999999999994}]}]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9580,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":8768,"kb_avail":94362260,"statfs":{"total":96636764160,"available":96626954240,"internally_reserved":0,"allocated":823296,"data_stored":354407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8978432},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.49099999999999999,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.63100000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58099999999999996,"15min":0.64700000000000002},"min":{"1min":0.36799999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.90200000000000002,"5min":3.972,"15min":3.972},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.74099999999999999,"15min":0.96399999999999997},"min":{"1min":0.371,"5min":0.313,"15min":0.313},"max":{"1min":0.78900000000000003,"5min":11.897,"15min":11.897},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.82299999999999995,"15min":1.0469999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.85199999999999998,"5min":11.804,"15min":11.804},"last":0.73699999999999999}]},{"osd":3,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.81399999999999995,"15min":1.036},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.89500000000000002,"5min":11.843999999999999,"15min":11.843999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64300000000000002,"15min":0.73499999999999999},"min":{"1min":0.47899999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.96399999999999997,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.54500000000000004}]},{"osd":4,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81799999999999995,"15min":0.86299999999999999},"min":{"1min":0.55000000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0960000000000001,"5min":1.613,"15min":1.613},"last":0.745},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.86099999999999999},"min":{"1min":0.65000000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.0800000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.76200000000000001}]},{"osd":5,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.84699999999999998,"15min":0.92500000000000004},"min":{"1min":0.63900000000000001,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.997,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81999999999999995,"15min":0.873},"min":{"1min":0.57899999999999996,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98899999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.66100000000000003}]},{"osd":6,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.91300000000000003,"15min":0.999},"min":{"1min":0.68400000000000005,"5min":0.628,"15min":0.628},"max":{"1min":1.173,"5min":3.907,"15min":3.907},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.875,"5min":0.88400000000000001,"15min":0.93999999999999995},"min":{"1min":0.67300000000000004,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1819999999999999,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.70999999999999996}]},{"osd":7,"last update":"Fri Mar 3 07:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84499999999999997,"15min":0.88300000000000001},"min":{"1min":0.70399999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.026,"5min":1.419,"15min":1.419},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.871,"5min":0.90600000000000003,"15min":1.012},"min":{"1min":0.70399999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1659999999999999,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":0.67700000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10640,"kb_used_data":1352,"kb_used_omap":0,"kb_used_meta":9280,"kb_avail":94361200,"statfs":{"total":96636764160,"available":96625868800,"internally_reserved":0,"allocated":1384448,"data_stored":941460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9502720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45800000000000002,"15min":0.47199999999999998},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65000000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.50600000000000001,"15min":0.51700000000000002},"min":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81999999999999995,"5min":0.871,"15min":0.871},"last":0.54300000000000004}]},{"osd":2,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53000000000000003,"15min":0.55000000000000004},"min":{"1min":0.38400000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77400000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.44,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73399999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.46800000000000003}]},{"osd":3,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70399999999999996,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.46600000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.80800000000000005,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.50700000000000001}]},{"osd":4,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82299999999999995,"15min":0.84499999999999997},"min":{"1min":0.64700000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79800000000000004,"15min":0.81999999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0740000000000001,"5min":1.605,"15min":1.605},"last":0.83499999999999996}]},{"osd":5,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.80400000000000005,"15min":0.82199999999999995},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0980000000000001,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.82099999999999995,"15min":0.82499999999999996},"min":{"1min":0.61299999999999999,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.2070000000000001,"5min":1.286,"15min":1.286},"last":0.749}]},{"osd":6,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.85399999999999998,"15min":0.88500000000000001},"min":{"1min":0.65300000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2470000000000001,"5min":1.623,"15min":1.623},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.86399999999999999},"min":{"1min":0.66300000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.137,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.97099999999999997}]},{"osd":7,"last update":"Fri Mar 3 07:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84499999999999997,"15min":0.88300000000000001},"min":{"1min":0.59399999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.083,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.83999999999999997,"15min":0.86199999999999999},"min":{"1min":0.58199999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.1719999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.78900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9484,"kb_used_data":772,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362356,"statfs":{"total":96636764160,"available":96627052544,"internally_reserved":0,"allocated":790528,"data_stored":351092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.64800000000000002,"15min":0.87},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.61699999999999999,"5min":5.7889999999999997,"15min":5.7889999999999997},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.60899999999999999,"15min":0.73199999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67400000000000004,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":0.36499999999999999}]},{"osd":1,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.629,"15min":0.753},"min":{"1min":0.36299999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77700000000000002,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.79300000000000004,"15min":1.0089999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.77100000000000002,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.51500000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.88900000000000001,"15min":1.222},"min":{"1min":0.41399999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.69799999999999995,"5min":12.961,"15min":12.961},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.80000000000000004,"15min":1.036},"min":{"1min":0.42699999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.74099999999999999,"5min":6.1449999999999996,"15min":6.1449999999999996},"last":0.48299999999999998}]},{"osd":4,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.0469999999999999,"15min":1.2769999999999999},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1639999999999999,"5min":6.484,"15min":6.484},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.0580000000000001,"15min":1.2549999999999999},"min":{"1min":0.63600000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.1759999999999999,"5min":6.5510000000000002,"15min":6.5510000000000002},"last":0.68000000000000005}]},{"osd":5,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.97399999999999998,"15min":1.1799999999999999},"min":{"1min":0.61499999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.034,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.84699999999999998,"5min":1.0269999999999999,"15min":1.238},"min":{"1min":0.60999999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1499999999999999,"5min":6.6369999999999996,"15min":6.6369999999999996},"last":0.77100000000000002}]},{"osd":6,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.0429999999999999,"15min":1.24},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.1240000000000001,"5min":6.625,"15min":6.625},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.88200000000000001,"5min":1.0549999999999999,"15min":1.2669999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.099,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.69099999999999995}]},{"osd":7,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":1.0129999999999999,"15min":1.2310000000000001},"min":{"1min":0.69899999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.087,"5min":6.7009999999999996,"15min":6.7009999999999996},"last":0.623},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.0209999999999999,"15min":1.214},"min":{"1min":0.66500000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.081,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.64100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574909,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9936,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94361904,"statfs":{"total":96636764160,"available":96626589696,"internally_reserved":0,"allocated":794624,"data_stored":352111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.63,"15min":0.83499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.56599999999999995,"5min":9.5820000000000007,"15min":9.5820000000000007},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.70099999999999996,"15min":0.90600000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83599999999999997,"5min":9.7840000000000007,"15min":9.7840000000000007},"last":0.38200000000000001}]},{"osd":1,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.501,"15min":0.52300000000000002},"min":{"1min":0.34999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78900000000000003,"5min":1,"15min":1},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.75600000000000001,"15min":0.98999999999999999},"min":{"1min":0.41099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70499999999999996,"5min":9.702,"15min":9.702},"last":0.54200000000000004}]},{"osd":2,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.746,"15min":0.93600000000000005},"min":{"1min":0.436,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88300000000000001,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.623},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.80500000000000005,"15min":1.0229999999999999},"min":{"1min":0.498,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.86499999999999999,"5min":9.734,"15min":9.734},"last":0.60699999999999998}]},{"osd":4,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.96099999999999997,"15min":1.165},"min":{"1min":0.54400000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1819999999999999,"5min":9.8629999999999995,"15min":9.8629999999999995},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.83999999999999997,"15min":0.84799999999999998},"min":{"1min":0.629,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.097,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":0.745}]},{"osd":5,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82099999999999995,"15min":0.83299999999999996},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.1899999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.999,"15min":1.222},"min":{"1min":0.628,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0860000000000001,"5min":9.6600000000000001,"15min":9.6600000000000001},"last":0.77000000000000002}]},{"osd":6,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.0129999999999999,"15min":1.2370000000000001},"min":{"1min":0.65700000000000003,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0600000000000001,"5min":9.8190000000000008,"15min":9.8190000000000008},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82799999999999996,"15min":0.86699999999999999},"min":{"1min":0.629,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.2609999999999999,"5min":1.609,"15min":1.609},"last":0.78000000000000003}]},{"osd":7,"last update":"Fri Mar 3 07:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":1.0369999999999999,"15min":1.2749999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.204,"5min":9.625,"15min":9.625},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.81399999999999995,"5min":1.034,"15min":1.244},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.119,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":0.86699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574909,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9692,"kb_used_data":788,"kb_used_omap":0,"kb_used_meta":8896,"kb_avail":94362148,"statfs":{"total":96636764160,"available":96626839552,"internally_reserved":0,"allocated":806912,"data_stored":353659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9109504},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.85199999999999998,"15min":0.98299999999999998},"min":{"1min":0.433,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0049999999999999,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.68400000000000005,"15min":0.66600000000000004},"min":{"1min":0.56100000000000005,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.95399999999999996,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.64900000000000002}]},{"osd":1,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.92700000000000005,"15min":1.1080000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94199999999999995,"5min":9.5709999999999997,"15min":9.5709999999999997},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.83999999999999997,"15min":0.92700000000000005},"min":{"1min":0.58599999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0349999999999999,"5min":5.266,"15min":5.266},"last":0.73999999999999999}]},{"osd":2,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.91600000000000004,"15min":1.077},"min":{"1min":0.60299999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.016,"5min":8.4529999999999994,"15min":8.4529999999999994},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.88,"15min":1.002},"min":{"1min":0.53200000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.018,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.85799999999999998}]},{"osd":3,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.72899999999999998,"15min":0.73299999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.79000000000000004,"15min":0.80200000000000005},"min":{"1min":0.58399999999999996,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.98499999999999999,"5min":1.389,"15min":1.389},"last":0.86899999999999999}]},{"osd":5,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.71799999999999997,"15min":0.74199999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86799999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.72299999999999998,"15min":0.72099999999999997},"min":{"1min":0.47599999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.998,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.58699999999999997}]},{"osd":6,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.77300000000000002,"15min":0.78100000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.94599999999999995,"5min":1.032,"15min":1.032},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.501,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0700000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.81899999999999995}]},{"osd":7,"last update":"Fri Mar 3 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.79700000000000004,"15min":0.82199999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0609999999999999,"5min":1.679,"15min":1.679},"last":0.75},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.88,"15min":0.92800000000000005},"min":{"1min":0.64300000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0069999999999999,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.90500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9616,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362224,"statfs":{"total":96636764160,"available":96626917376,"internally_reserved":0,"allocated":794624,"data_stored":351114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.65700000000000003,"15min":0.64100000000000001},"min":{"1min":0.42899999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.75700000000000001,"15min":0.74399999999999999},"min":{"1min":0.60599999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96999999999999997,"5min":2.528,"15min":2.528},"last":0.57499999999999996}]},{"osd":1,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.73699999999999999,"15min":0.745},"min":{"1min":0.66400000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.752},{"interface":"front","average":{"1min":0.753,"5min":0.67200000000000004,"15min":0.63700000000000001},"min":{"1min":0.53900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.5}]},{"osd":2,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69199999999999995,"15min":0.68300000000000005},"min":{"1min":0.52700000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.95399999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.72499999999999998,"15min":0.71599999999999997},"min":{"1min":0.59299999999999997,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.89200000000000002,"5min":1.321,"15min":1.321},"last":0.63800000000000001}]},{"osd":3,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.71599999999999997,"15min":0.69499999999999995},"min":{"1min":0.55600000000000005,"5min":0.496,"15min":0.496},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.745,"15min":0.73499999999999999},"min":{"1min":0.629,"5min":0.501,"15min":0.501},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.80800000000000005}]},{"osd":4,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.51400000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.79000000000000004,"15min":0.78200000000000003},"min":{"1min":0.59899999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.82999999999999996}]},{"osd":6,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.79700000000000004},"min":{"1min":0.53000000000000003,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.94499999999999995,"5min":1.071,"15min":1.071},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.72199999999999998,"15min":0.72399999999999998},"min":{"1min":0.58999999999999997,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97099999999999997,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.77800000000000002}]},{"osd":7,"last update":"Fri Mar 3 07:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.75600000000000001,"15min":0.75700000000000001},"min":{"1min":0.55700000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.026,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81599999999999995,"15min":0.83299999999999996},"min":{"1min":0.67700000000000005,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.90500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:00:33.039 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-03-03T08:00:33.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2023-03-03T08:00:33.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.323+0000 7feb0f7ff700 1 Processor -- start 2023-03-03T08:00:33.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.324+0000 7feb0f7ff700 1 -- start start 2023-03-03T08:00:33.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.325+0000 7feb0f7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb080d0270 0x7feb080cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:33.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.325+0000 7feb0f7ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb0806b4b0 0x7feb080d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:33.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.325+0000 7feb0f7ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb080cf980 con 0x7feb0815ca70 2023-03-03T08:00:33.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.325+0000 7feb0f7ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb080cd5b0 con 0x7feb0806b4b0 2023-03-03T08:00:33.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.325+0000 7feb0f7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb080d1800 con 0x7feb080d0270 2023-03-03T08:00:33.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.325+0000 7feb0d59b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7feb0815ca70 0x7feb080cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44892/0 (socket says 172.21.15.98:44892) 2023-03-03T08:00:33.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.325+0000 7feb0d59b700 1 -- 172.21.15.98:0/3139447300 learned_addr learned my addr 172.21.15.98:0/3139447300 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:33.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.326+0000 7feb0dd9c700 1 --2- 172.21.15.98:0/3139447300 >> v2:172.21.15.98:3300/0 conn(0x7feb0806b4b0 0x7feb080d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:33.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.326+0000 7feb0cd9a700 1 --2- 172.21.15.98:0/3139447300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb080d0270 0x7feb080cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:33.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.326+0000 7feafffff700 1 -- 172.21.15.98:0/3139447300 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674063608 0 0) 0x7feb080cf980 con 0x7feb0815ca70 2023-03-03T08:00:33.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.326+0000 7feafffff700 1 -- 172.21.15.98:0/3139447300 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf40035b0 con 0x7feb0815ca70 2023-03-03T08:00:33.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.326+0000 7feb0dd9c700 1 -- 172.21.15.98:0/3139447300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb080d0270 msgr2=0x7feb080cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:33.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.326+0000 7feb0dd9c700 1 --2- 172.21.15.98:0/3139447300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb080d0270 0x7feb080cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.326+0000 7feb0dd9c700 1 -- 172.21.15.98:0/3139447300 >> v1:172.21.15.98:6789/0 conn(0x7feb0815ca70 legacy=0x7feb080cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:33.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.326+0000 7feb0dd9c700 1 -- 172.21.15.98:0/3139447300 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb080cd900 con 0x7feb0806b4b0 2023-03-03T08:00:33.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.327+0000 7feb0dd9c700 1 --2- 172.21.15.98:0/3139447300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb0806b4b0 0x7feb080d1fc0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7feb00009d90 tx=0x7feb0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a0365d6422f20f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:33.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.327+0000 7feafffff700 1 -- 172.21.15.98:0/3139447300 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0000e070 con 0x7feb0806b4b0 2023-03-03T08:00:33.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.327+0000 7feafffff700 1 -- 172.21.15.98:0/3139447300 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0000bb20 con 0x7feb0806b4b0 2023-03-03T08:00:33.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.327+0000 7feafffff700 1 -- 172.21.15.98:0/3139447300 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb00012430 con 0x7feb0806b4b0 2023-03-03T08:00:33.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.327+0000 7feb0f7ff700 1 -- 172.21.15.98:0/3139447300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb0806b4b0 msgr2=0x7feb080d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:33.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.327+0000 7feb0f7ff700 1 --2- 172.21.15.98:0/3139447300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb0806b4b0 0x7feb080d1fc0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7feb00009d90 tx=0x7feb0000b330 comp rx=0 tx=0).stop 2023-03-03T08:00:33.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.328+0000 7feb0f7ff700 1 -- 172.21.15.98:0/3139447300 shutdown_connections 2023-03-03T08:00:33.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.328+0000 7feb0f7ff700 1 --2- 172.21.15.98:0/3139447300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb0806b4b0 0x7feb080d1fc0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.328+0000 7feb0f7ff700 1 --2- 172.21.15.98:0/3139447300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb080d0270 0x7feb080cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.328+0000 7feb0f7ff700 1 -- 172.21.15.98:0/3139447300 >> 172.21.15.98:0/3139447300 conn(0x7feb08073e70 msgr2=0x7feb0806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:33.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.328+0000 7feb0f7ff700 1 -- 172.21.15.98:0/3139447300 shutdown_connections 2023-03-03T08:00:33.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.328+0000 7feb0f7ff700 1 -- 172.21.15.98:0/3139447300 wait complete. 2023-03-03T08:00:33.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.330+0000 7feb0f7ff700 1 Processor -- start 2023-03-03T08:00:33.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.330+0000 7feb0f7ff700 1 -- start start 2023-03-03T08:00:33.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.331+0000 7feb0f7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb0806b4b0 0x7feb0814c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:33.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.331+0000 7feb0f7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb080d0270 0x7feb080cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:33.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.331+0000 7feb0f7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb0815ca70 0x7feb080cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:33.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.331+0000 7feb0f7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb080b3530 con 0x7feb0815ca70 2023-03-03T08:00:33.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.331+0000 7feb0f7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb080b2f70 con 0x7feb080d0270 2023-03-03T08:00:33.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.331+0000 7feb0f7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb080b3250 con 0x7feb0806b4b0 2023-03-03T08:00:33.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.331+0000 7feb0dd9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb0815ca70 0x7feb080cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:33.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.331+0000 7feb0d59b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb0806b4b0 0x7feb0814c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:33.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.331+0000 7feb0dd9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb0815ca70 0x7feb080cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45038/0 (socket says 172.21.15.98:45038) 2023-03-03T08:00:33.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.331+0000 7feb0dd9c700 1 -- 172.21.15.98:0/208725128 learned_addr learned my addr 172.21.15.98:0/208725128 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:33.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.331+0000 7feb0cd9a700 1 --2- 172.21.15.98:0/208725128 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb080d0270 0x7feb080cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:33.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.332+0000 7feb0dd9c700 1 -- 172.21.15.98:0/208725128 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb0806b4b0 msgr2=0x7feb0814c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:33.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.332+0000 7feb0dd9c700 1 --2- 172.21.15.98:0/208725128 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb0806b4b0 0x7feb0814c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.332+0000 7feb0dd9c700 1 -- 172.21.15.98:0/208725128 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb080d0270 msgr2=0x7feb080cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.332+0000 7feb0dd9c700 1 --2- 172.21.15.98:0/208725128 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb080d0270 0x7feb080cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.332+0000 7feb0dd9c700 1 -- 172.21.15.98:0/208725128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb00009a70 con 0x7feb0815ca70 2023-03-03T08:00:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.332+0000 7feb0dd9c700 1 --2- 172.21.15.98:0/208725128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb0815ca70 0x7feb080cb5e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7feb00014040 tx=0x7feb00004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.332+0000 7feafdffb700 1 -- 172.21.15.98:0/208725128 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb000129f0 con 0x7feb0815ca70 2023-03-03T08:00:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.332+0000 7feafdffb700 1 -- 172.21.15.98:0/208725128 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb00020070 con 0x7feb0815ca70 2023-03-03T08:00:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.332+0000 7feafdffb700 1 -- 172.21.15.98:0/208725128 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb00012db0 con 0x7feb0815ca70 2023-03-03T08:00:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.332+0000 7feb0f7ff700 1 -- 172.21.15.98:0/208725128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb080d1aa0 con 0x7feb0815ca70 2023-03-03T08:00:33.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.333+0000 7feb0f7ff700 1 -- 172.21.15.98:0/208725128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb080cdbe0 con 0x7feb0815ca70 2023-03-03T08:00:33.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.334+0000 7feb0f7ff700 1 -- 172.21.15.98:0/208725128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0815ce40 con 0x7feb0815ca70 2023-03-03T08:00:33.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.335+0000 7feafdffb700 1 -- 172.21.15.98:0/208725128 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7feb0002e070 con 0x7feb0815ca70 2023-03-03T08:00:33.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.337+0000 7feafdffb700 1 --2- 172.21.15.98:0/208725128 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feaec0683c0 0x7feaec06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:33.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.337+0000 7feafdffb700 1 -- 172.21.15.98:0/208725128 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7feb00022070 con 0x7feb0815ca70 2023-03-03T08:00:33.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.337+0000 7feb0d59b700 1 --2- 172.21.15.98:0/208725128 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feaec0683c0 0x7feaec06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:33.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.338+0000 7feb0d59b700 1 --2- 172.21.15.98:0/208725128 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feaec0683c0 0x7feaec06a870 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7feb04007fd0 tx=0x7feb04007dc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:33.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.341+0000 7feafdffb700 1 -- 172.21.15.98:0/208725128 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feb000540e0 con 0x7feb0815ca70 2023-03-03T08:00:33.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.611+0000 7feb0f7ff700 1 -- 172.21.15.98:0/208725128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7feb0804e6e0 con 0x7feb0815ca70 2023-03-03T08:00:33.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.615+0000 7feafdffb700 1 -- 172.21.15.98:0/208725128 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 184+0+0 (secure 0 0 0) 0x7feb0002c0e0 con 0x7feb0815ca70 2023-03-03T08:00:33.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.618+0000 7feb0f7ff700 1 -- 172.21.15.98:0/208725128 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feaec0683c0 msgr2=0x7feaec06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:33.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.618+0000 7feb0f7ff700 1 --2- 172.21.15.98:0/208725128 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feaec0683c0 0x7feaec06a870 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7feb04007fd0 tx=0x7feb04007dc0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.618+0000 7feb0f7ff700 1 -- 172.21.15.98:0/208725128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb0815ca70 msgr2=0x7feb080cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:33.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.618+0000 7feb0f7ff700 1 --2- 172.21.15.98:0/208725128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb0815ca70 0x7feb080cb5e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7feb00014040 tx=0x7feb00004ad0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.618+0000 7feb0f7ff700 1 -- 172.21.15.98:0/208725128 shutdown_connections 2023-03-03T08:00:33.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.618+0000 7feb0f7ff700 1 --2- 172.21.15.98:0/208725128 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7feaec0683c0 0x7feaec06a870 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.618+0000 7feb0f7ff700 1 --2- 172.21.15.98:0/208725128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb0815ca70 0x7feb080cb5e0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.618+0000 7feb0f7ff700 1 --2- 172.21.15.98:0/208725128 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb0806b4b0 0x7feb0814c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.618+0000 7feb0f7ff700 1 --2- 172.21.15.98:0/208725128 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb080d0270 0x7feb080cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.618+0000 7feb0f7ff700 1 -- 172.21.15.98:0/208725128 >> 172.21.15.98:0/208725128 conn(0x7feb08073e70 msgr2=0x7feb0815bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:33.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.618+0000 7feb0f7ff700 1 -- 172.21.15.98:0/208725128 shutdown_connections 2023-03-03T08:00:33.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.618+0000 7feb0f7ff700 1 -- 172.21.15.98:0/208725128 wait complete. 2023-03-03T08:00:33.622 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.0 2023-03-03T08:00:33.647 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:00:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.890+0000 7fc62ffff700 1 Processor -- start 2023-03-03T08:00:33.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.890+0000 7fc62ffff700 1 -- start start 2023-03-03T08:00:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.892+0000 7fc62ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc630146360 0x7fc630160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.892+0000 7fc62ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc6300d2590 0x7fc6300b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.892+0000 7fc62ffff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6301614c0 con 0x7fc6300b4ab0 2023-03-03T08:00:33.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.892+0000 7fc62ffff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6300b5460 con 0x7fc6300d2590 2023-03-03T08:00:33.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.892+0000 7fc62ffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300d10f0 con 0x7fc630146360 2023-03-03T08:00:33.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.892+0000 7fc62f7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc6300d2590 0x7fc6300b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:33.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.892+0000 7fc62f7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc6300d2590 0x7fc6300b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45052/0 (socket says 172.21.15.98:45052) 2023-03-03T08:00:33.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.892+0000 7fc62f7fe700 1 -- 172.21.15.98:0/995529463 learned_addr learned my addr 172.21.15.98:0/995529463 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:33.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.893+0000 7fc62effd700 1 --2- 172.21.15.98:0/995529463 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc630146360 0x7fc630160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:33.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.893+0000 7fc62f7fe700 1 -- 172.21.15.98:0/995529463 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc630146360 msgr2=0x7fc630160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:33.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.893+0000 7fc62f7fe700 1 --2- 172.21.15.98:0/995529463 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc630146360 0x7fc630160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.893+0000 7fc62f7fe700 1 -- 172.21.15.98:0/995529463 >> v1:172.21.15.100:6789/0 conn(0x7fc6300b4ab0 legacy=0x7fc6300731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:33.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.893+0000 7fc62f7fe700 1 -- 172.21.15.98:0/995529463 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6300cdf20 con 0x7fc6300d2590 2023-03-03T08:00:33.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.893+0000 7fc62effd700 1 --2- 172.21.15.98:0/995529463 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc630146360 0x7fc630160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:33.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.893+0000 7fc62f7fe700 1 --2- 172.21.15.98:0/995529463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6300d2590 0x7fc6300b4f20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fc628009e50 tx=0x7fc62800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207dc888c1fa7595 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:33.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.893+0000 7fc62dffb700 1 -- 172.21.15.98:0/995529463 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62800e070 con 0x7fc6300d2590 2023-03-03T08:00:33.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.893+0000 7fc62dffb700 1 -- 172.21.15.98:0/995529463 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc628003a20 con 0x7fc6300d2590 2023-03-03T08:00:33.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.893+0000 7fc62dffb700 1 -- 172.21.15.98:0/995529463 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc628003dd0 con 0x7fc6300d2590 2023-03-03T08:00:33.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.894+0000 7fc62ffff700 1 -- 172.21.15.98:0/995529463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6300d2590 msgr2=0x7fc6300b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:33.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.894+0000 7fc62ffff700 1 --2- 172.21.15.98:0/995529463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6300d2590 0x7fc6300b4f20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fc628009e50 tx=0x7fc62800b330 comp rx=0 tx=0).stop 2023-03-03T08:00:33.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.894+0000 7fc62ffff700 1 -- 172.21.15.98:0/995529463 shutdown_connections 2023-03-03T08:00:33.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.894+0000 7fc62ffff700 1 --2- 172.21.15.98:0/995529463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6300d2590 0x7fc6300b4f20 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.894+0000 7fc62ffff700 1 --2- 172.21.15.98:0/995529463 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc630146360 0x7fc630160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.894+0000 7fc62ffff700 1 -- 172.21.15.98:0/995529463 >> 172.21.15.98:0/995529463 conn(0x7fc63006b8e0 msgr2=0x7fc6300cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:33.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.894+0000 7fc62ffff700 1 -- 172.21.15.98:0/995529463 shutdown_connections 2023-03-03T08:00:33.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.894+0000 7fc62ffff700 1 -- 172.21.15.98:0/995529463 wait complete. 2023-03-03T08:00:33.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.896+0000 7fc62ffff700 1 Processor -- start 2023-03-03T08:00:33.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.896+0000 7fc62ffff700 1 -- start start 2023-03-03T08:00:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.897+0000 7fc62ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6300b4ab0 0x7fc6300c6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.897+0000 7fc62ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6300d2590 0x7fc6300c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.897+0000 7fc62ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc630146360 0x7fc6300c81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.897+0000 7fc62ffff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300cb110 con 0x7fc6300d2590 2023-03-03T08:00:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.897+0000 7fc62ffff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300c6660 con 0x7fc6300b4ab0 2023-03-03T08:00:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.898+0000 7fc62ffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300cc810 con 0x7fc630146360 2023-03-03T08:00:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.898+0000 7fc62e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6300d2590 0x7fc6300c7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.898+0000 7fc62e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6300d2590 0x7fc6300c7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45062/0 (socket says 172.21.15.98:45062) 2023-03-03T08:00:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.898+0000 7fc62e7fc700 1 -- 172.21.15.98:0/1765004402 learned_addr learned my addr 172.21.15.98:0/1765004402 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:33.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.898+0000 7fc62f7fe700 1 --2- 172.21.15.98:0/1765004402 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc630146360 0x7fc6300c81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:33.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.898+0000 7fc62effd700 1 --2- 172.21.15.98:0/1765004402 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6300b4ab0 0x7fc6300c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:33.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.898+0000 7fc62e7fc700 1 -- 172.21.15.98:0/1765004402 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc630146360 msgr2=0x7fc6300c81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:33.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.898+0000 7fc62e7fc700 1 --2- 172.21.15.98:0/1765004402 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc630146360 0x7fc6300c81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.898+0000 7fc62e7fc700 1 -- 172.21.15.98:0/1765004402 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6300b4ab0 msgr2=0x7fc6300c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:33.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.898+0000 7fc62e7fc700 1 --2- 172.21.15.98:0/1765004402 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6300b4ab0 0x7fc6300c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:33.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.898+0000 7fc62e7fc700 1 -- 172.21.15.98:0/1765004402 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc628009b30 con 0x7fc6300d2590 2023-03-03T08:00:33.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.898+0000 7fc62f7fe700 1 --2- 172.21.15.98:0/1765004402 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc630146360 0x7fc6300c81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:33.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.899+0000 7fc62e7fc700 1 --2- 172.21.15.98:0/1765004402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6300d2590 0x7fc6300c7c90 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc62400b4a0 tx=0x7fc62400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:33.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.899+0000 7fc617fff700 1 -- 172.21.15.98:0/1765004402 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc624011830 con 0x7fc6300d2590 2023-03-03T08:00:33.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.899+0000 7fc617fff700 1 -- 172.21.15.98:0/1765004402 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc624011990 con 0x7fc6300d2590 2023-03-03T08:00:33.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.899+0000 7fc617fff700 1 -- 172.21.15.98:0/1765004402 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc624011d40 con 0x7fc6300d2590 2023-03-03T08:00:33.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.899+0000 7fc62ffff700 1 -- 172.21.15.98:0/1765004402 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6300cd960 con 0x7fc6300d2590 2023-03-03T08:00:33.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.899+0000 7fc62ffff700 1 -- 172.21.15.98:0/1765004402 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6300cd680 con 0x7fc6300d2590 2023-03-03T08:00:33.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.900+0000 7fc62ffff700 1 -- 172.21.15.98:0/1765004402 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6300cdc40 con 0x7fc6300d2590 2023-03-03T08:00:33.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.901+0000 7fc617fff700 1 -- 172.21.15.98:0/1765004402 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fc62400f050 con 0x7fc6300d2590 2023-03-03T08:00:33.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.903+0000 7fc617fff700 1 --2- 172.21.15.98:0/1765004402 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc61c0683c0 0x7fc61c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:33.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.903+0000 7fc617fff700 1 -- 172.21.15.98:0/1765004402 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc624085ac0 con 0x7fc6300d2590 2023-03-03T08:00:33.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.903+0000 7fc62effd700 1 --2- 172.21.15.98:0/1765004402 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc61c0683c0 0x7fc61c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:33.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.904+0000 7fc62effd700 1 --2- 172.21.15.98:0/1765004402 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc61c0683c0 0x7fc61c06a870 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc620006fd0 tx=0x7fc620008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:33.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:33.906+0000 7fc617fff700 1 -- 172.21.15.98:0/1765004402 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc624052160 con 0x7fc6300d2590 2023-03-03T08:00:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.138+0000 7fc62ffff700 1 -- 172.21.15.98:0/1765004402 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6300ca0e0 con 0x7fc6300d2590 2023-03-03T08:00:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.139+0000 7fc617fff700 1 -- 172.21.15.98:0/1765004402 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4649 (secure 0 0 0) 0x7fc624015090 con 0x7fc6300d2590 2023-03-03T08:00:34.142 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:34.142 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5702,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/976526047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":976526047},{"type":"v1","addr":"172.21.15.100:6834","nonce":976526047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3372584689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3372584689},{"type":"v1","addr":"172.21.15.100:6835","nonce":3372584689}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/505711827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":505711827},{"type":"v1","addr":"172.21.15.98:6837","nonce":505711827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:00:22.323396+0000","modified":"2023-03-03T08:00:25.787755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6387":{"gid":6387,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6836/3248147431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3248147431},{"type":"v1","addr":"172.21.15.98:6836","nonce":3248147431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-03-03T08:00:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.142+0000 7fc62ffff700 1 -- 172.21.15.98:0/1765004402 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc61c0683c0 msgr2=0x7fc61c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.142+0000 7fc62ffff700 1 --2- 172.21.15.98:0/1765004402 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc61c0683c0 0x7fc61c06a870 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc620006fd0 tx=0x7fc620008040 comp rx=0 tx=0).stop 2023-03-03T08:00:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.142+0000 7fc62ffff700 1 -- 172.21.15.98:0/1765004402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6300d2590 msgr2=0x7fc6300c7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.142+0000 7fc62ffff700 1 --2- 172.21.15.98:0/1765004402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6300d2590 0x7fc6300c7c90 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc62400b4a0 tx=0x7fc62400b860 comp rx=0 tx=0).stop 2023-03-03T08:00:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.142+0000 7fc62ffff700 1 -- 172.21.15.98:0/1765004402 shutdown_connections 2023-03-03T08:00:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.142+0000 7fc62ffff700 1 --2- 172.21.15.98:0/1765004402 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc61c0683c0 0x7fc61c06a870 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.142+0000 7fc62ffff700 1 --2- 172.21.15.98:0/1765004402 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc630146360 0x7fc6300c81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.142+0000 7fc62ffff700 1 --2- 172.21.15.98:0/1765004402 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6300b4ab0 0x7fc6300c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.142+0000 7fc62ffff700 1 --2- 172.21.15.98:0/1765004402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6300d2590 0x7fc6300c7c90 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.142+0000 7fc62ffff700 1 -- 172.21.15.98:0/1765004402 >> 172.21.15.98:0/1765004402 conn(0x7fc63006b8e0 msgr2=0x7fc6300b2280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.142+0000 7fc62ffff700 1 -- 172.21.15.98:0/1765004402 shutdown_connections 2023-03-03T08:00:34.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.142+0000 7fc62ffff700 1 -- 172.21.15.98:0/1765004402 wait complete. 2023-03-03T08:00:34.146 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 31 2023-03-03T08:00:34.574 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 31, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-03-03T08:00:22.323396+0000', 'modified': '2023-03-03T08:00:25.787755+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6387}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6387': {'gid': 6387, 'name': 'c', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.98:6836/3248147431', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 3248147431}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 3248147431}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-03-03T08:00:34.574 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-03-03T08:00:34.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-03-03T08:00:34.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.821+0000 7f97ba9ff700 1 Processor -- start 2023-03-03T08:00:34.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.821+0000 7f97ba9ff700 1 -- start start 2023-03-03T08:00:34.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.822+0000 7f97ba9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97b40d0270 0x7f97b40cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.823+0000 7f97ba9ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b40cf980 con 0x7f97b406b4b0 2023-03-03T08:00:34.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.823+0000 7f97ba9ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b40d1800 con 0x7f97b415ca70 2023-03-03T08:00:34.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.823+0000 7f97ba9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b40cd5b0 con 0x7f97b40d0270 2023-03-03T08:00:34.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.823+0000 7f97b8f9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f97b406b4b0 0x7f97b40d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44894/0 (socket says 172.21.15.98:44894) 2023-03-03T08:00:34.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.823+0000 7f97b8f9c700 1 -- 172.21.15.98:0/1817880256 learned_addr learned my addr 172.21.15.98:0/1817880256 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:34.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.823+0000 7f97b37fe700 1 --2- 172.21.15.98:0/1817880256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97b40d0270 0x7f97b40cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.824+0000 7f97b2ffd700 1 -- 172.21.15.98:0/1817880256 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755681017 0 0) 0x7f97b40cf980 con 0x7f97b406b4b0 2023-03-03T08:00:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.824+0000 7f97b2ffd700 1 -- 172.21.15.98:0/1817880256 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f979c0035b0 con 0x7f97b406b4b0 2023-03-03T08:00:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.824+0000 7f97b2ffd700 1 -- 172.21.15.98:0/1817880256 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084818955 0 0) 0x7f97b40d1800 con 0x7f97b415ca70 2023-03-03T08:00:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.824+0000 7f97b2ffd700 1 -- 172.21.15.98:0/1817880256 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b40cf980 con 0x7f97b415ca70 2023-03-03T08:00:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.824+0000 7f97b37fe700 1 -- 172.21.15.98:0/1817880256 >> v1:172.21.15.100:6789/0 conn(0x7f97b415ca70 legacy=0x7f97b40cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.824+0000 7f97b37fe700 1 -- 172.21.15.98:0/1817880256 >> v1:172.21.15.98:6789/0 conn(0x7f97b406b4b0 legacy=0x7f97b40d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.824+0000 7f97b37fe700 1 -- 172.21.15.98:0/1817880256 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b40cd900 con 0x7f97b40d0270 2023-03-03T08:00:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.824+0000 7f97b37fe700 1 --2- 172.21.15.98:0/1817880256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97b40d0270 0x7f97b40cf440 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009870 tx=0x7f97a8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8a276a301af43b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.824+0000 7f97b2ffd700 1 -- 172.21.15.98:0/1817880256 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a8011830 con 0x7f97b40d0270 2023-03-03T08:00:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.824+0000 7f97b2ffd700 1 -- 172.21.15.98:0/1817880256 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a8011990 con 0x7f97b40d0270 2023-03-03T08:00:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.825+0000 7f97b2ffd700 1 -- 172.21.15.98:0/1817880256 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a8011d20 con 0x7f97b40d0270 2023-03-03T08:00:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.825+0000 7f97ba9ff700 1 -- 172.21.15.98:0/1817880256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97b40d0270 msgr2=0x7f97b40cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.825+0000 7f97ba9ff700 1 --2- 172.21.15.98:0/1817880256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97b40d0270 0x7f97b40cf440 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009870 tx=0x7f97a8009b80 comp rx=0 tx=0).stop 2023-03-03T08:00:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.825+0000 7f97ba9ff700 1 -- 172.21.15.98:0/1817880256 shutdown_connections 2023-03-03T08:00:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.825+0000 7f97ba9ff700 1 --2- 172.21.15.98:0/1817880256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97b40d0270 0x7f97b40cf440 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.825+0000 7f97ba9ff700 1 -- 172.21.15.98:0/1817880256 >> 172.21.15.98:0/1817880256 conn(0x7f97b4073e70 msgr2=0x7f97b406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.825+0000 7f97ba9ff700 1 -- 172.21.15.98:0/1817880256 shutdown_connections 2023-03-03T08:00:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.825+0000 7f97ba9ff700 1 -- 172.21.15.98:0/1817880256 wait complete. 2023-03-03T08:00:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.827+0000 7f97ba9ff700 1 Processor -- start 2023-03-03T08:00:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.827+0000 7f97ba9ff700 1 -- start start 2023-03-03T08:00:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.828+0000 7f97ba9ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97b406b4b0 0x7f97b41f63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.828+0000 7f97ba9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97b40d0270 0x7f97b4151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.828+0000 7f97ba9ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97b415ca70 0x7f97b40b3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.828+0000 7f97ba9ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b40cc320 con 0x7f97b415ca70 2023-03-03T08:00:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.828+0000 7f97ba9ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b4152c80 con 0x7f97b406b4b0 2023-03-03T08:00:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.828+0000 7f97ba9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b41f6910 con 0x7f97b40d0270 2023-03-03T08:00:34.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.829+0000 7f97b8f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97b415ca70 0x7f97b40b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.829+0000 7f97b8f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97b415ca70 0x7f97b40b3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45066/0 (socket says 172.21.15.98:45066) 2023-03-03T08:00:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.829+0000 7f97b8f9c700 1 -- 172.21.15.98:0/505111659 learned_addr learned my addr 172.21.15.98:0/505111659 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.829+0000 7f97b37fe700 1 --2- 172.21.15.98:0/505111659 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97b40d0270 0x7f97b4151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.829+0000 7f97b3fff700 1 --2- 172.21.15.98:0/505111659 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97b406b4b0 0x7f97b41f63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.829+0000 7f97b8f9c700 1 -- 172.21.15.98:0/505111659 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97b40d0270 msgr2=0x7f97b4151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.829+0000 7f97b8f9c700 1 --2- 172.21.15.98:0/505111659 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97b40d0270 0x7f97b4151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.829+0000 7f97b8f9c700 1 -- 172.21.15.98:0/505111659 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97b406b4b0 msgr2=0x7f97b41f63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.829+0000 7f97b8f9c700 1 --2- 172.21.15.98:0/505111659 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97b406b4b0 0x7f97b41f63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.829+0000 7f97b8f9c700 1 -- 172.21.15.98:0/505111659 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a80094a0 con 0x7f97b415ca70 2023-03-03T08:00:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.829+0000 7f97b3fff700 1 --2- 172.21.15.98:0/505111659 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97b406b4b0 0x7f97b41f63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.830+0000 7f97b8f9c700 1 --2- 172.21.15.98:0/505111659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97b415ca70 0x7f97b40b3070 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00cbd0 tx=0x7f97ac00cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.830+0000 7f97b0ff9700 1 -- 172.21.15.98:0/505111659 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac00aa60 con 0x7f97b415ca70 2023-03-03T08:00:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.830+0000 7f97b0ff9700 1 -- 172.21.15.98:0/505111659 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ac00abc0 con 0x7f97b415ca70 2023-03-03T08:00:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.830+0000 7f97b0ff9700 1 -- 172.21.15.98:0/505111659 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac012400 con 0x7f97b415ca70 2023-03-03T08:00:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.830+0000 7f97ba9ff700 1 -- 172.21.15.98:0/505111659 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b40d1aa0 con 0x7f97b415ca70 2023-03-03T08:00:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.830+0000 7f97ba9ff700 1 -- 172.21.15.98:0/505111659 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b40cdbe0 con 0x7f97b415ca70 2023-03-03T08:00:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.833+0000 7f97b0ff9700 1 -- 172.21.15.98:0/505111659 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f97ac019030 con 0x7f97b415ca70 2023-03-03T08:00:34.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.835+0000 7f97b0ff9700 1 --2- 172.21.15.98:0/505111659 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f97a00683c0 0x7f97a006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.835+0000 7f97b0ff9700 1 -- 172.21.15.98:0/505111659 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f97ac085ae0 con 0x7f97b415ca70 2023-03-03T08:00:34.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.835+0000 7f97b3fff700 1 --2- 172.21.15.98:0/505111659 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f97a00683c0 0x7f97a006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.836+0000 7f97b3fff700 1 --2- 172.21.15.98:0/505111659 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f97a00683c0 0x7f97a006a870 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f97a4007fd0 tx=0x7f97a4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:34.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.837+0000 7f97ce300b80 1 Processor -- start 2023-03-03T08:00:34.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.837+0000 7f97ce300b80 1 -- start start 2023-03-03T08:00:34.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.838+0000 7f97ce300b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a9a70 0x5598133a9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.838+0000 7f97ce300b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a8cd0 0x5598133f8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.838+0000 7f97ce300b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a4560 0x5598133f8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.838+0000 7f97ce300b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55981331f4f0 con 0x5598133a9a70 2023-03-03T08:00:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.838+0000 7f97ce300b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x559813294b60 con 0x5598133a4560 2023-03-03T08:00:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.838+0000 7f97ce300b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x559813303ad0 con 0x5598133a8cd0 2023-03-03T08:00:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.838+0000 7f97b3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a9a70 0x5598133a9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.838+0000 7f97b3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a9a70 0x5598133a9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45080/0 (socket says 172.21.15.98:45080) 2023-03-03T08:00:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.839+0000 7f97b3fff700 1 -- 172.21.15.98:0/3314758651 learned_addr learned my addr 172.21.15.98:0/3314758651 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.839+0000 7f97b37fe700 1 --2- 172.21.15.98:0/3314758651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a8cd0 0x5598133f8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.839+0000 7f97b8f9c700 1 --2- 172.21.15.98:0/3314758651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a4560 0x5598133f8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.839+0000 7f97b3fff700 1 -- 172.21.15.98:0/3314758651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a8cd0 msgr2=0x5598133f8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.839+0000 7f97b3fff700 1 --2- 172.21.15.98:0/3314758651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a8cd0 0x5598133f8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.839+0000 7f97b3fff700 1 -- 172.21.15.98:0/3314758651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a4560 msgr2=0x5598133f8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.839+0000 7f97b3fff700 1 --2- 172.21.15.98:0/3314758651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a4560 0x5598133f8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.839+0000 7f97b3fff700 1 -- 172.21.15.98:0/3314758651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598130f1540 con 0x5598133a9a70 2023-03-03T08:00:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.839+0000 7f97b3fff700 1 --2- 172.21.15.98:0/3314758651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a9a70 0x5598133a9e40 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f97a4005340 tx=0x7f97a4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7cb98b850045f25 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.840+0000 7f9799ffb700 1 -- 172.21.15.98:0/3314758651 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a4003dd0 con 0x5598133a9a70 2023-03-03T08:00:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.840+0000 7f9799ffb700 1 -- 172.21.15.98:0/3314758651 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a4021450 con 0x5598133a9a70 2023-03-03T08:00:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.840+0000 7f9799ffb700 1 -- 172.21.15.98:0/3314758651 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a4021800 con 0x5598133a9a70 2023-03-03T08:00:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.840+0000 7f97ce300b80 1 -- 172.21.15.98:0/3314758651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a9a70 msgr2=0x5598133a9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.840+0000 7f97ce300b80 1 --2- 172.21.15.98:0/3314758651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a9a70 0x5598133a9e40 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f97a4005340 tx=0x7f97a4011000 comp rx=0 tx=0).stop 2023-03-03T08:00:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.840+0000 7f97ce300b80 1 -- 172.21.15.98:0/3314758651 shutdown_connections 2023-03-03T08:00:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.840+0000 7f97ce300b80 1 --2- 172.21.15.98:0/3314758651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a9a70 0x5598133a9e40 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.840+0000 7f97ce300b80 1 --2- 172.21.15.98:0/3314758651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a4560 0x5598133f8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.840+0000 7f97ce300b80 1 --2- 172.21.15.98:0/3314758651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a8cd0 0x5598133f8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.840+0000 7f97ce300b80 1 -- 172.21.15.98:0/3314758651 >> 172.21.15.98:0/3314758651 conn(0x5598133a7c30 msgr2=0x5598133a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.840+0000 7f97ce300b80 1 -- 172.21.15.98:0/3314758651 shutdown_connections 2023-03-03T08:00:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.840+0000 7f97ce300b80 1 -- 172.21.15.98:0/3314758651 wait complete. 2023-03-03T08:00:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.841+0000 7f97ce300b80 1 -- start start 2023-03-03T08:00:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.842+0000 7f97ce300b80 1 Processor -- start 2023-03-03T08:00:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97ce300b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a4560 0x5598133d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97ce300b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a8cd0 0x5598133f0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97ce300b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a9a70 0x5598133d7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97ce300b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598132fce40 con 0x5598133a4560 2023-03-03T08:00:34.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97ce300b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598130f2140 con 0x5598133a8cd0 2023-03-03T08:00:34.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97ce300b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x5598133d5c60 con 0x5598133a9a70 2023-03-03T08:00:34.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97ce300b80 10 client.0 _ll_register_callbacks cb 0x55981329b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T08:00:34.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97b3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a4560 0x5598133d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97b3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a4560 0x5598133d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45086/0 (socket says 172.21.15.98:45086) 2023-03-03T08:00:34.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97b3fff700 1 -- 172.21.15.98:0/329743693 learned_addr learned my addr 172.21.15.98:0/329743693 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97b37fe700 1 --2- 172.21.15.98:0/329743693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a8cd0 0x5598133f0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97b8f9c700 1 --2- 172.21.15.98:0/329743693 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a9a70 0x5598133d7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97b3fff700 1 -- 172.21.15.98:0/329743693 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a9a70 msgr2=0x5598133d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97b3fff700 1 --2- 172.21.15.98:0/329743693 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a9a70 0x5598133d7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97b3fff700 1 -- 172.21.15.98:0/329743693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a8cd0 msgr2=0x5598133f0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.843+0000 7f97b3fff700 1 --2- 172.21.15.98:0/329743693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a8cd0 0x5598133f0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.844+0000 7f97b3fff700 1 -- 172.21.15.98:0/329743693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a400f040 con 0x5598133a4560 2023-03-03T08:00:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.844+0000 7f97b3fff700 1 --2- 172.21.15.98:0/329743693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a4560 0x5598133d0060 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f97a40016c0 tx=0x7f97a4000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d6eeecd4656833 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.844+0000 7f9792ffd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-03-03T08:00:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.844+0000 7f9792ffd700 1 -- 172.21.15.98:0/329743693 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a4023320 con 0x5598133a4560 2023-03-03T08:00:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.844+0000 7f9792ffd700 1 -- 172.21.15.98:0/329743693 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f97a4023660 con 0x5598133a4560 2023-03-03T08:00:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.844+0000 7f97ce300b80 1 -- 172.21.15.98:0/329743693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55981339cc50 con 0x5598133a4560 2023-03-03T08:00:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.844+0000 7f9792ffd700 1 -- 172.21.15.98:0/329743693 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97a4023480 con 0x5598133a4560 2023-03-03T08:00:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.844+0000 7f97ce300b80 10 client.6546 fetch_fsmap learned FSMap version 31 2023-03-03T08:00:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.844+0000 7f97ce300b80 1 -- 172.21.15.98:0/329743693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=2+}) v3 -- 0x55981312bb10 con 0x5598133a4560 2023-03-03T08:00:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.845+0000 7f9792ffd700 1 -- 172.21.15.98:0/329743693 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55981339cc50 con 0x5598133a4560 2023-03-03T08:00:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.845+0000 7f9792ffd700 1 -- 172.21.15.98:0/329743693 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f97a402b030 con 0x5598133a4560 2023-03-03T08:00:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.845+0000 7f97ce300b80 10 client.6546 fetch_fsmap finished waiting for FSMap version 31 2023-03-03T08:00:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.845+0000 7f97ce300b80 10 client.6546 resolve_mds: resolved name 'c' to daemon mds.c 2023-03-03T08:00:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.845+0000 7f97ce300b80 20 client.6546 populate_metadata read hostname 'smithi098' 2023-03-03T08:00:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.845+0000 7f97ce300b80 1 --2- 172.21.15.98:0/329743693 >> [v2:172.21.15.98:6834/3248147431,v1:172.21.15.98:6836/3248147431] conn(0x5598133f2270 0x5598133ff070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.845+0000 7f97ce300b80 4 client.6546 mds_command: new command op to 6387 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-03T08:00:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.845+0000 7f97ce300b80 1 -- 172.21.15.98:0/329743693 --> [v2:172.21.15.98:6834/3248147431,v1:172.21.15.98:6836/3248147431] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5598132f6020 con 0x5598133f2270 2023-03-03T08:00:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.845+0000 7f97b37fe700 1 --2- 172.21.15.98:0/329743693 >> [v2:172.21.15.98:6834/3248147431,v1:172.21.15.98:6836/3248147431] conn(0x5598133f2270 0x5598133ff070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.846+0000 7f9792ffd700 10 client.6546 ms_handle_connect on v2:172.21.15.98:6834/3248147431 2023-03-03T08:00:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.846+0000 7f97b37fe700 1 --2- 172.21.15.98:0/329743693 >> [v2:172.21.15.98:6834/3248147431,v1:172.21.15.98:6836/3248147431] conn(0x5598133f2270 0x5598133ff070 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa0ab6d9a2ace2fb server_cookie=72af18e04059d047 in_seq=0 out_seq=0 2023-03-03T08:00:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.847+0000 7f9792ffd700 1 -- 172.21.15.98:0/329743693 <== mds.0 v2:172.21.15.98:6834/3248147431 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x5598132f6020 con 0x5598133f2270 2023-03-03T08:00:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.847+0000 7f9792ffd700 10 client.6546 handle_command_reply: tid=0 2023-03-03T08:00:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.847+0000 7f97ce300b80 1 client.6546 shutdown 2023-03-03T08:00:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.847+0000 7f97ce300b80 1 -- 172.21.15.98:0/329743693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a4560 msgr2=0x5598133d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.847+0000 7f97ce300b80 1 --2- 172.21.15.98:0/329743693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a4560 0x5598133d0060 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f97a40016c0 tx=0x7f97a4000ed0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.847+0000 7f97ce300b80 1 -- 172.21.15.98:0/329743693 shutdown_connections 2023-03-03T08:00:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.847+0000 7f97ce300b80 1 --2- 172.21.15.98:0/329743693 >> [v2:172.21.15.98:6834/3248147431,v1:172.21.15.98:6836/3248147431] conn(0x5598133f2270 0x5598133ff070 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.847+0000 7f97ce300b80 1 --2- 172.21.15.98:0/329743693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a4560 0x5598133d0060 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.847+0000 7f97ce300b80 1 --2- 172.21.15.98:0/329743693 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a9a70 0x5598133d7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.847+0000 7f9792ffd700 0 client.6546 ms_handle_reset on v2:172.21.15.98:6834/3248147431 2023-03-03T08:00:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.847+0000 7f97ce300b80 1 --2- 172.21.15.98:0/329743693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a8cd0 0x5598133f0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.847+0000 7f97ce300b80 1 -- 172.21.15.98:0/329743693 >> 172.21.15.98:0/329743693 conn(0x5598133a7c30 msgr2=0x5598133d4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.847+0000 7f97ce300b80 1 -- 172.21.15.98:0/329743693 shutdown_connections 2023-03-03T08:00:34.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.847+0000 7f97ce300b80 1 -- 172.21.15.98:0/329743693 wait complete. 2023-03-03T08:00:34.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.848+0000 7f97ce300b80 20 client.6546 trim_cache size 0 max 16384 2023-03-03T08:00:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.866+0000 7f97ce300b80 1 Processor -- start 2023-03-03T08:00:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.866+0000 7f97ce300b80 1 -- start start 2023-03-03T08:00:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97ce300b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a4560 0x5598133d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97ce300b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a8cd0 0x5598133d1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97ce300b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a9a70 0x5598133cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97ce300b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598133d70d0 con 0x5598133a4560 2023-03-03T08:00:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97ce300b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598133e6190 con 0x5598133a9a70 2023-03-03T08:00:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97ce300b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x5598133d7ce0 con 0x5598133a8cd0 2023-03-03T08:00:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97b3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a4560 0x5598133d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97b3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a4560 0x5598133d17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45100/0 (socket says 172.21.15.98:45100) 2023-03-03T08:00:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97b3fff700 1 -- 172.21.15.98:0/3892465751 learned_addr learned my addr 172.21.15.98:0/3892465751 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97b8f9c700 1 --2- 172.21.15.98:0/3892465751 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a9a70 0x5598133cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97b37fe700 1 --2- 172.21.15.98:0/3892465751 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a8cd0 0x5598133d1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97b3fff700 1 -- 172.21.15.98:0/3892465751 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a8cd0 msgr2=0x5598133d1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97b3fff700 1 --2- 172.21.15.98:0/3892465751 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a8cd0 0x5598133d1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97b3fff700 1 -- 172.21.15.98:0/3892465751 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a9a70 msgr2=0x5598133cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97b3fff700 1 --2- 172.21.15.98:0/3892465751 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a9a70 0x5598133cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.867+0000 7f97b3fff700 1 -- 172.21.15.98:0/3892465751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598133a0e80 con 0x5598133a4560 2023-03-03T08:00:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.868+0000 7f97b3fff700 1 --2- 172.21.15.98:0/3892465751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a4560 0x5598133d17e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f97a4000f10 tx=0x7f97a4027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b723bbdb397868ff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.868+0000 7f979a7fc700 1 -- 172.21.15.98:0/3892465751 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a40238c0 con 0x5598133a4560 2023-03-03T08:00:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.868+0000 7f979a7fc700 1 -- 172.21.15.98:0/3892465751 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a4019040 con 0x5598133a4560 2023-03-03T08:00:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.868+0000 7f979a7fc700 1 -- 172.21.15.98:0/3892465751 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a4023560 con 0x5598133a4560 2023-03-03T08:00:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.868+0000 7f97ce300b80 1 -- 172.21.15.98:0/3892465751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a4560 msgr2=0x5598133d17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.868+0000 7f97ce300b80 1 --2- 172.21.15.98:0/3892465751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a4560 0x5598133d17e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f97a4000f10 tx=0x7f97a4027000 comp rx=0 tx=0).stop 2023-03-03T08:00:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.869+0000 7f97ce300b80 1 -- 172.21.15.98:0/3892465751 shutdown_connections 2023-03-03T08:00:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.869+0000 7f97ce300b80 1 --2- 172.21.15.98:0/3892465751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a4560 0x5598133d17e0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.869+0000 7f97ce300b80 1 --2- 172.21.15.98:0/3892465751 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a9a70 0x5598133cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.869+0000 7f97ce300b80 1 --2- 172.21.15.98:0/3892465751 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a8cd0 0x5598133d1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.869+0000 7f97ce300b80 1 -- 172.21.15.98:0/3892465751 >> 172.21.15.98:0/3892465751 conn(0x5598133a7c30 msgr2=0x5598133a32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:34.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.869+0000 7f97ce300b80 1 -- 172.21.15.98:0/3892465751 shutdown_connections 2023-03-03T08:00:34.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.869+0000 7f97ce300b80 1 -- 172.21.15.98:0/3892465751 wait complete. 2023-03-03T08:00:34.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.869+0000 7f97ce300b80 1 -- start start 2023-03-03T08:00:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.870+0000 7f97ce300b80 1 Processor -- start 2023-03-03T08:00:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97ce300b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a4560 0x55981343bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97ce300b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a8cd0 0x5598133a8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97ce300b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a9a70 0x5598133a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97ce300b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55981343c7f0 con 0x5598133a9a70 2023-03-03T08:00:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97ce300b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x559813456bd0 con 0x5598133a8cd0 2023-03-03T08:00:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97ce300b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x559813456eb0 con 0x5598133a4560 2023-03-03T08:00:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97ce300b80 10 client.0 _ll_register_callbacks cb 0x55981329b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T08:00:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97b8f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a9a70 0x5598133a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97b8f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a9a70 0x5598133a8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45102/0 (socket says 172.21.15.98:45102) 2023-03-03T08:00:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97b3fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a4560 0x55981343bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97b8f9c700 1 -- 172.21.15.98:0/1397305370 learned_addr learned my addr 172.21.15.98:0/1397305370 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97b37fe700 1 --2- 172.21.15.98:0/1397305370 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a8cd0 0x5598133a8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97b8f9c700 1 -- 172.21.15.98:0/1397305370 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a4560 msgr2=0x55981343bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97b8f9c700 1 --2- 172.21.15.98:0/1397305370 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a4560 0x55981343bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97b8f9c700 1 -- 172.21.15.98:0/1397305370 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a8cd0 msgr2=0x5598133a8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97b8f9c700 1 --2- 172.21.15.98:0/1397305370 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a8cd0 0x5598133a8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.871+0000 7f97b8f9c700 1 -- 172.21.15.98:0/1397305370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55981312bb10 con 0x5598133a9a70 2023-03-03T08:00:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.872+0000 7f97b8f9c700 1 --2- 172.21.15.98:0/1397305370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a9a70 0x5598133a8540 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00a340 tx=0x7f97ac055660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a42f90b24ccd9f0e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.872+0000 7f9792ffd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-03-03T08:00:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.872+0000 7f9792ffd700 1 -- 172.21.15.98:0/1397305370 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac051ea0 con 0x5598133a9a70 2023-03-03T08:00:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.872+0000 7f9792ffd700 1 -- 172.21.15.98:0/1397305370 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f97ac01dc50 con 0x5598133a9a70 2023-03-03T08:00:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.872+0000 7f97ce300b80 1 -- 172.21.15.98:0/1397305370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f97a4021450 con 0x5598133a9a70 2023-03-03T08:00:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.872+0000 7f9792ffd700 1 -- 172.21.15.98:0/1397305370 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97ac01de10 con 0x5598133a9a70 2023-03-03T08:00:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.872+0000 7f97ce300b80 10 client.6552 fetch_fsmap learned FSMap version 31 2023-03-03T08:00:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.872+0000 7f97ce300b80 1 -- 172.21.15.98:0/1397305370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=2+}) v3 -- 0x5598132aaa60 con 0x5598133a9a70 2023-03-03T08:00:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.872+0000 7f9792ffd700 1 -- 172.21.15.98:0/1397305370 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a4021450 con 0x5598133a9a70 2023-03-03T08:00:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.872+0000 7f9792ffd700 1 -- 172.21.15.98:0/1397305370 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f97ac054da0 con 0x5598133a9a70 2023-03-03T08:00:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.873+0000 7f97ce300b80 10 client.6552 fetch_fsmap finished waiting for FSMap version 31 2023-03-03T08:00:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.873+0000 7f97ce300b80 10 client.6552 resolve_mds: resolved name 'c' to daemon mds.c 2023-03-03T08:00:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.873+0000 7f97ce300b80 20 client.6552 populate_metadata read hostname 'smithi098' 2023-03-03T08:00:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.873+0000 7f97ce300b80 1 --2- 172.21.15.98:0/1397305370 >> [v2:172.21.15.98:6834/3248147431,v1:172.21.15.98:6836/3248147431] conn(0x5598133d0340 0x5598133d0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.873+0000 7f97ce300b80 4 client.6552 mds_command: new command op to 6387 tid=0[{"prefix": "status"}] 2023-03-03T08:00:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.873+0000 7f97ce300b80 1 -- 172.21.15.98:0/1397305370 --> [v2:172.21.15.98:6834/3248147431,v1:172.21.15.98:6836/3248147431] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55981314cca0 con 0x5598133d0340 2023-03-03T08:00:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.873+0000 7f97b3fff700 1 --2- 172.21.15.98:0/1397305370 >> [v2:172.21.15.98:6834/3248147431,v1:172.21.15.98:6836/3248147431] conn(0x5598133d0340 0x5598133d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f9792ffd700 10 client.6552 ms_handle_connect on v2:172.21.15.98:6834/3248147431 2023-03-03T08:00:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f97b3fff700 1 --2- 172.21.15.98:0/1397305370 >> [v2:172.21.15.98:6834/3248147431,v1:172.21.15.98:6836/3248147431] conn(0x5598133d0340 0x5598133d0710 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33db96885b856af server_cookie=dc11804211b48cf3 in_seq=0 out_seq=0 2023-03-03T08:00:34.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f9792ffd700 1 -- 172.21.15.98:0/1397305370 <== mds.0 v2:172.21.15.98:6834/3248147431 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55981314cca0 con 0x5598133d0340 2023-03-03T08:00:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f9792ffd700 10 client.6552 handle_command_reply: tid=0 2023-03-03T08:00:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f97ce300b80 1 client.6552 shutdown 2023-03-03T08:00:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f97ce300b80 1 -- 172.21.15.98:0/1397305370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a9a70 msgr2=0x5598133a8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f97ce300b80 1 --2- 172.21.15.98:0/1397305370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a9a70 0x5598133a8540 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00a340 tx=0x7f97ac055660 comp rx=0 tx=0).stop 2023-03-03T08:00:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f97ce300b80 1 -- 172.21.15.98:0/1397305370 shutdown_connections 2023-03-03T08:00:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f97ce300b80 1 --2- 172.21.15.98:0/1397305370 >> [v2:172.21.15.98:6834/3248147431,v1:172.21.15.98:6836/3248147431] conn(0x5598133d0340 0x5598133d0710 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f97ce300b80 1 --2- 172.21.15.98:0/1397305370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598133a9a70 0x5598133a8540 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f9792ffd700 0 client.6552 ms_handle_reset on v2:172.21.15.98:6834/3248147431 2023-03-03T08:00:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f97ce300b80 1 --2- 172.21.15.98:0/1397305370 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5598133a4560 0x55981343bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f97ce300b80 1 --2- 172.21.15.98:0/1397305370 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5598133a8cd0 0x5598133a8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f97ce300b80 1 -- 172.21.15.98:0/1397305370 >> 172.21.15.98:0/1397305370 conn(0x5598133a7c30 msgr2=0x5598133a32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f97ce300b80 1 -- 172.21.15.98:0/1397305370 shutdown_connections 2023-03-03T08:00:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.874+0000 7f97ce300b80 1 -- 172.21.15.98:0/1397305370 wait complete. 2023-03-03T08:00:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.875+0000 7f97ce300b80 20 client.6552 trim_cache size 0 max 16384 2023-03-03T08:00:34.884 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:00:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "fbbd322f-236a-44f8-96c5-be6963846c9b", 2023-03-03T08:00:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2023-03-03T08:00:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 6387, 2023-03-03T08:00:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-03-03T08:00:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-03-03T08:00:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-03-03T08:00:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 12.549115884000001, 2023-03-03T08:00:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 31, 2023-03-03T08:00:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 66, 2023-03-03T08:00:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 66, 2023-03-03T08:00:34.886 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 19.254837091999999 2023-03-03T08:00:34.886 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:00:34.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.875+0000 7f97ba9ff700 1 -- 172.21.15.98:0/505111659 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f97a00683c0 msgr2=0x7f97a006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:34.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.875+0000 7f97ba9ff700 1 --2- 172.21.15.98:0/505111659 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f97a00683c0 0x7f97a006a870 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f97a4007fd0 tx=0x7f97a4009040 comp rx=0 tx=0).stop 2023-03-03T08:00:34.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.875+0000 7f97ba9ff700 1 -- 172.21.15.98:0/505111659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97b415ca70 msgr2=0x7f97b40b3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:34.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.875+0000 7f97ba9ff700 1 --2- 172.21.15.98:0/505111659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97b415ca70 0x7f97b40b3070 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00cbd0 tx=0x7f97ac00cee0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.875+0000 7f97ba9ff700 1 -- 172.21.15.98:0/505111659 shutdown_connections 2023-03-03T08:00:34.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.875+0000 7f97ba9ff700 1 --2- 172.21.15.98:0/505111659 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f97a00683c0 0x7f97a006a870 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.875+0000 7f97ba9ff700 1 --2- 172.21.15.98:0/505111659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97b415ca70 0x7f97b40b3070 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.875+0000 7f97ba9ff700 1 --2- 172.21.15.98:0/505111659 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97b406b4b0 0x7f97b41f63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.875+0000 7f97ba9ff700 1 --2- 172.21.15.98:0/505111659 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97b40d0270 0x7f97b4151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:34.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.875+0000 7f97ba9ff700 1 -- 172.21.15.98:0/505111659 >> 172.21.15.98:0/505111659 conn(0x7f97b4073e70 msgr2=0x7f97b40b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:34.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.876+0000 7f97ba9ff700 1 -- 172.21.15.98:0/505111659 shutdown_connections 2023-03-03T08:00:34.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:34.876+0000 7f97ba9ff700 1 -- 172.21.15.98:0/505111659 wait complete. 2023-03-03T08:00:34.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T08:00:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.144+0000 7f45fa3ff700 1 Processor -- start 2023-03-03T08:00:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.145+0000 7f45fa3ff700 1 -- start start 2023-03-03T08:00:35.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.146+0000 7f45fa3ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f45f40b87c0 0x7f45f40cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:35.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.146+0000 7f45fa3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f45f40b8b90 0x7f45f40b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:35.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.146+0000 7f45fa3ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f40d0ef0 con 0x7f45f40b5240 2023-03-03T08:00:35.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.146+0000 7f45fa3ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f40d0600 con 0x7f45f40b87c0 2023-03-03T08:00:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.146+0000 7f45fa3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f40cd970 con 0x7f45f40b8b90 2023-03-03T08:00:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.146+0000 7f45f3fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f45f40b87c0 0x7f45f40cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.146+0000 7f45f899c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f45f40b5240 0x7f45f41605b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36588/0 (socket says 172.21.15.98:36588) 2023-03-03T08:00:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.146+0000 7f45f899c700 1 -- 172.21.15.98:0/1237196179 learned_addr learned my addr 172.21.15.98:0/1237196179 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.147+0000 7f45f37fe700 1 --2- 172.21.15.98:0/1237196179 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f45f40b8b90 0x7f45f40b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.147+0000 7f45f3fff700 1 -- 172.21.15.98:0/1237196179 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f45f40b8b90 msgr2=0x7f45f40b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.147+0000 7f45f3fff700 1 --2- 172.21.15.98:0/1237196179 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f45f40b8b90 0x7f45f40b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.147+0000 7f45f3fff700 1 -- 172.21.15.98:0/1237196179 >> v1:172.21.15.98:6789/0 conn(0x7f45f40b5240 legacy=0x7f45f41605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.147+0000 7f45f3fff700 1 -- 172.21.15.98:0/1237196179 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f40cdc50 con 0x7f45f40b87c0 2023-03-03T08:00:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.147+0000 7f45f2ffd700 1 -- 172.21.15.98:0/1237196179 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44612279 0 0) 0x7f45f40d0ef0 con 0x7f45f40b5240 2023-03-03T08:00:35.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.147+0000 7f45f3fff700 1 --2- 172.21.15.98:0/1237196179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40b87c0 0x7f45f40cffe0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f45e4009840 tx=0x7f45e4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3deabd1705e983 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:35.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.147+0000 7f45f2ffd700 1 -- 172.21.15.98:0/1237196179 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4011830 con 0x7f45f40b87c0 2023-03-03T08:00:35.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.148+0000 7f45f2ffd700 1 -- 172.21.15.98:0/1237196179 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e4011990 con 0x7f45f40b87c0 2023-03-03T08:00:35.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.148+0000 7f45f2ffd700 1 -- 172.21.15.98:0/1237196179 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4011d20 con 0x7f45f40b87c0 2023-03-03T08:00:35.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.148+0000 7f45fa3ff700 1 -- 172.21.15.98:0/1237196179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40b87c0 msgr2=0x7f45f40cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:35.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.148+0000 7f45fa3ff700 1 --2- 172.21.15.98:0/1237196179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40b87c0 0x7f45f40cffe0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f45e4009840 tx=0x7f45e4009b50 comp rx=0 tx=0).stop 2023-03-03T08:00:35.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.148+0000 7f45fa3ff700 1 -- 172.21.15.98:0/1237196179 shutdown_connections 2023-03-03T08:00:35.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.148+0000 7f45fa3ff700 1 --2- 172.21.15.98:0/1237196179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40b87c0 0x7f45f40cffe0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:35.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.148+0000 7f45fa3ff700 1 --2- 172.21.15.98:0/1237196179 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f45f40b8b90 0x7f45f40b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:35.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.148+0000 7f45fa3ff700 1 -- 172.21.15.98:0/1237196179 >> 172.21.15.98:0/1237196179 conn(0x7f45f406aee0 msgr2=0x7f45f406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:35.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.148+0000 7f45fa3ff700 1 -- 172.21.15.98:0/1237196179 shutdown_connections 2023-03-03T08:00:35.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.148+0000 7f45fa3ff700 1 -- 172.21.15.98:0/1237196179 wait complete. 2023-03-03T08:00:35.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.150+0000 7f45fa3ff700 1 Processor -- start 2023-03-03T08:00:35.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.150+0000 7f45fa3ff700 1 -- start start 2023-03-03T08:00:35.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.151+0000 7f45fa3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f45f40b5240 0x7f45f415b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:35.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.151+0000 7f45fa3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40b87c0 0x7f45f415bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:35.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.151+0000 7f45fa3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f45f40b8b90 0x7f45f415abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:35.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.152+0000 7f45fa3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f415b0f0 con 0x7f45f40b87c0 2023-03-03T08:00:35.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.152+0000 7f45fa3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f415f360 con 0x7f45f40b5240 2023-03-03T08:00:35.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.152+0000 7f45fa3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f415bb00 con 0x7f45f40b8b90 2023-03-03T08:00:35.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.152+0000 7f45f37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40b87c0 0x7f45f415bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.152+0000 7f45f37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40b87c0 0x7f45f415bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51474/0 (socket says 172.21.15.98:51474) 2023-03-03T08:00:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.152+0000 7f45f37fe700 1 -- 172.21.15.98:0/2448347104 learned_addr learned my addr 172.21.15.98:0/2448347104 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.152+0000 7f45f899c700 1 --2- 172.21.15.98:0/2448347104 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f45f40b8b90 0x7f45f415abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.152+0000 7f45f3fff700 1 --2- 172.21.15.98:0/2448347104 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f45f40b5240 0x7f45f415b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.152+0000 7f45f37fe700 1 -- 172.21.15.98:0/2448347104 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f45f40b8b90 msgr2=0x7f45f415abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.152+0000 7f45f37fe700 1 --2- 172.21.15.98:0/2448347104 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f45f40b8b90 0x7f45f415abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.152+0000 7f45f37fe700 1 -- 172.21.15.98:0/2448347104 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f45f40b5240 msgr2=0x7f45f415b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.152+0000 7f45f37fe700 1 --2- 172.21.15.98:0/2448347104 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f45f40b5240 0x7f45f415b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.152+0000 7f45f37fe700 1 -- 172.21.15.98:0/2448347104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e40094a0 con 0x7f45f40b87c0 2023-03-03T08:00:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.153+0000 7f45f37fe700 1 --2- 172.21.15.98:0/2448347104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40b87c0 0x7f45f415bfe0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f45e800d800 tx=0x7f45e800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.153+0000 7f45f0ff9700 1 -- 172.21.15.98:0/2448347104 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e80125f0 con 0x7f45f40b87c0 2023-03-03T08:00:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.153+0000 7f45f0ff9700 1 -- 172.21.15.98:0/2448347104 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e8012750 con 0x7f45f40b87c0 2023-03-03T08:00:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.153+0000 7f45f0ff9700 1 -- 172.21.15.98:0/2448347104 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e8012b00 con 0x7f45f40b87c0 2023-03-03T08:00:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.153+0000 7f45fa3ff700 1 -- 172.21.15.98:0/2448347104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f40cdf30 con 0x7f45f40b87c0 2023-03-03T08:00:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.153+0000 7f45fa3ff700 1 -- 172.21.15.98:0/2448347104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f4073a80 con 0x7f45f40b87c0 2023-03-03T08:00:35.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.154+0000 7f45fa3ff700 1 -- 172.21.15.98:0/2448347104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f40ce0c0 con 0x7f45f40b87c0 2023-03-03T08:00:35.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.155+0000 7f45f0ff9700 1 -- 172.21.15.98:0/2448347104 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f45e8012c60 con 0x7f45f40b87c0 2023-03-03T08:00:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.156+0000 7f45f0ff9700 1 --2- 172.21.15.98:0/2448347104 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f45e00683c0 0x7f45e006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.157+0000 7f45f0ff9700 1 -- 172.21.15.98:0/2448347104 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f45e8085ff0 con 0x7f45f40b87c0 2023-03-03T08:00:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.157+0000 7f45f3fff700 1 --2- 172.21.15.98:0/2448347104 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f45e00683c0 0x7f45e006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.158+0000 7f45f3fff700 1 --2- 172.21.15.98:0/2448347104 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f45e00683c0 0x7f45e006a870 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f45e40090b0 tx=0x7f45e4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.160+0000 7f45f0ff9700 1 -- 172.21.15.98:0/2448347104 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f45e80526d0 con 0x7f45f40b87c0 2023-03-03T08:00:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.398+0000 7f45fa3ff700 1 -- 172.21.15.98:0/2448347104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f45f40729b0 con 0x7f45f40b87c0 2023-03-03T08:00:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.398+0000 7f45f0ff9700 1 -- 172.21.15.98:0/2448347104 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v31) v1 ==== 53+0+73 (secure 0 0 0) 0x7f45e8055cf0 con 0x7f45f40b87c0 2023-03-03T08:00:35.401 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T08:00:35.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.401+0000 7f45fa3ff700 1 -- 172.21.15.98:0/2448347104 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f45e00683c0 msgr2=0x7f45e006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:35.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.401+0000 7f45fa3ff700 1 --2- 172.21.15.98:0/2448347104 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f45e00683c0 0x7f45e006a870 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f45e40090b0 tx=0x7f45e4008030 comp rx=0 tx=0).stop 2023-03-03T08:00:35.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.401+0000 7f45fa3ff700 1 -- 172.21.15.98:0/2448347104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40b87c0 msgr2=0x7f45f415bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:35.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.401+0000 7f45fa3ff700 1 --2- 172.21.15.98:0/2448347104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40b87c0 0x7f45f415bfe0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f45e800d800 tx=0x7f45e800db10 comp rx=0 tx=0).stop 2023-03-03T08:00:35.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.401+0000 7f45fa3ff700 1 -- 172.21.15.98:0/2448347104 shutdown_connections 2023-03-03T08:00:35.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.401+0000 7f45fa3ff700 1 --2- 172.21.15.98:0/2448347104 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f45e00683c0 0x7f45e006a870 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:35.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.401+0000 7f45fa3ff700 1 --2- 172.21.15.98:0/2448347104 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f45f40b8b90 0x7f45f415abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:35.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.401+0000 7f45fa3ff700 1 --2- 172.21.15.98:0/2448347104 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f45f40b5240 0x7f45f415b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:35.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.401+0000 7f45fa3ff700 1 --2- 172.21.15.98:0/2448347104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40b87c0 0x7f45f415bfe0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:35.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.401+0000 7f45fa3ff700 1 -- 172.21.15.98:0/2448347104 >> 172.21.15.98:0/2448347104 conn(0x7f45f406aee0 msgr2=0x7f45f40b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:35.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.401+0000 7f45fa3ff700 1 -- 172.21.15.98:0/2448347104 shutdown_connections 2023-03-03T08:00:35.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:35.401+0000 7f45fa3ff700 1 -- 172.21.15.98:0/2448347104 wait complete. 2023-03-03T08:00:35.429 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T08:00:35.430 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T08:00:35.430 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2023-03-03T08:00:35.431 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-03-03T08:00:35.431 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-03-03T08:00:35.431 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T08:00:35.431 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-03-03T08:00:35.432 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T08:00:35.432 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T08:00:35.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:35.433 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2023-03-03T08:00:35.501 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T08:00:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T08:00:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T08:00:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:00:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-03-03T08:00:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:00:35.502 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:00:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:00:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2023-03-03T08:00:35.503 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:00:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:00:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2023-03-03T08:00:35.503 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:00:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2023-03-03T08:00:35.504 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T08:00:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2023-03-03T08:00:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T08:00:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 41920sec preferred_lft 41920sec 2023-03-03T08:00:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2023-03-03T08:00:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:00:35.504 INFO:teuthology.orchestra.run.smithi098.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T08:00:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 46:3f:9e:55:37:22 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:00:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-03-03T08:00:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:00:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::1857:c2ff:fe19:346f/64 scope link 2023-03-03T08:00:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:00:35.505 INFO:teuthology.orchestra.run.smithi098.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:00:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 46:3f:9e:55:37:22 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-03-03T08:00:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::443f:9eff:fe55:3722/64 scope link 2023-03-03T08:00:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:00:35.506 INFO:teuthology.orchestra.run.smithi098.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:00:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether d6:19:97:a6:7c:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-03-03T08:00:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::d419:97ff:fea6:7c83/64 scope link 2023-03-03T08:00:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:00:35.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:35.507 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T08:00:35.604 INFO:teuthology.orchestra.run.smithi098.stdout:1 2023-03-03T08:00:35.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:35.609 DEBUG:teuthology.orchestra.run.smithi098:> route 2023-03-03T08:00:35.683 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2023-03-03T08:00:35.683 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T08:00:35.683 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T08:00:35.683 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T08:00:35.684 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T08:00:35.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:35.685 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-03-03T08:00:35.686 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-03-03T08:00:35.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T08:00:35.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T08:00:35.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T08:00:35.687 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-03-03T08:00:35.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:35.928 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-03-03T08:00:35.997 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-03-03T08:00:35.997 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-03-03T08:00:35.998 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T08:00:35.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:35.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2023-03-03T08:00:36.098 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-03-03T08:00:36.099 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:00:36.099 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-03-03T08:00:36.168 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-03-03T08:00:36.168 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:00:36.169 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-03-03T08:00:36.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T08:00:36.337 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:00:36.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:00:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:00:36.440 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:00:36.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:36.441 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T08:00:36.510 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-03-03T08:00:36.511 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/) 2023-03-03T08:00:36.516 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T08:00:36.619 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:00:36.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:00:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:00:36.729 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:00:36.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:36.730 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T08:00:36.744 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:2023-03-03T08:00:36.742+0000 7fc7b01865c0 -1 init, newargv = 0x5634f713a160 newargc=15 2023-03-03T08:00:36.745 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[135583]: starting ceph client 2023-03-03T08:00:36.754 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[135583]: starting fuse 2023-03-03T08:00:36.805 INFO:teuthology.orchestra.run.smithi098.stdout:42 2023-03-03T08:00:36.806 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2023-03-03T08:00:36.806 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T08:00:36.806 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2023-03-03T08:00:36.807 DEBUG:teuthology.orchestra.run.smithi098:> import re 2023-03-03T08:00:36.807 DEBUG:teuthology.orchestra.run.smithi098:> import os 2023-03-03T08:00:36.807 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2023-03-03T08:00:36.808 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:00:36.808 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2023-03-03T08:00:36.808 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-03-03T08:00:36.808 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2023-03-03T08:00:36.809 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-03-03T08:00:36.809 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:00:36.809 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2023-03-03T08:00:36.810 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2023-03-03T08:00:36.810 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2023-03-03T08:00:36.810 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2023-03-03T08:00:36.811 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:00:36.811 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2023-03-03T08:00:36.811 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T08:00:36.811 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T08:00:36.812 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T08:00:36.812 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2023-03-03T08:00:36.812 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2023-03-03T08:00:36.813 DEBUG:teuthology.orchestra.run.smithi098:> return f 2023-03-03T08:00:36.813 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T08:00:36.813 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:00:36.813 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.0")) 2023-03-03T08:00:36.814 DEBUG:teuthology.orchestra.run.smithi098:> ' 2023-03-03T08:00:37.031 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.0.135583.asok 2023-03-03T08:00:37.209 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.135583.asok 2023-03-03T08:00:37.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:37.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.135583.asok status 2023-03-03T08:00:37.479 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:00:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2023-03-03T08:00:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T08:00:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T08:00:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "0", 2023-03-03T08:00:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2023-03-03T08:00:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-03-03T08:00:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "135583", 2023-03-03T08:00:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2023-03-03T08:00:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2023-03-03T08:00:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2023-03-03T08:00:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 6567, 2023-03-03T08:00:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2023-03-03T08:00:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2023-03-03T08:00:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2023-03-03T08:00:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 6567 2023-03-03T08:00:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T08:00:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T08:00:37.483 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:00:37.483 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2360422128 2023-03-03T08:00:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T08:00:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T08:00:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:00:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2360422128 2023-03-03T08:00:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:00:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.6567 192.168.0.1:0/2360422128", 2023-03-03T08:00:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/2360422128", 2023-03-03T08:00:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2023-03-03T08:00:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 31, 2023-03-03T08:00:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 66, 2023-03-03T08:00:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2023-03-03T08:00:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2023-03-03T08:00:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2023-03-03T08:00:37.486 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:00:37.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:37.500 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2023-03-03T08:00:37.501 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.0 2023-03-03T08:00:37.570 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2023-03-03T08:00:37.571 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-03-03T08:00:37.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:37.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-03-03T08:00:37.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T08:00:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.920+0000 7fc6dbfff700 1 Processor -- start 2023-03-03T08:00:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.920+0000 7fc6dbfff700 1 -- start start 2023-03-03T08:00:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.922+0000 7fc6dbfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6dc160f80 0x7fc6dc0d1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.922+0000 7fc6dbfff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc0d10f0 con 0x7fc6dc146360 2023-03-03T08:00:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.922+0000 7fc6dbfff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc0cdf20 con 0x7fc6dc0b51c0 2023-03-03T08:00:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.922+0000 7fc6dbfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc0cd680 con 0x7fc6dc160f80 2023-03-03T08:00:37.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.922+0000 7fc6daffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc6dc146360 0x7fc6dc0b4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36598/0 (socket says 172.21.15.98:36598) 2023-03-03T08:00:37.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.922+0000 7fc6daffd700 1 -- 172.21.15.98:0/3539015872 learned_addr learned my addr 172.21.15.98:0/3539015872 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:37.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.923+0000 7fc6db7fe700 1 --2- 172.21.15.98:0/3539015872 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6dc160f80 0x7fc6dc0d1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:37.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.923+0000 7fc6d9ffb700 1 -- 172.21.15.98:0/3539015872 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124207576 0 0) 0x7fc6dc0d10f0 con 0x7fc6dc146360 2023-03-03T08:00:37.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.923+0000 7fc6d9ffb700 1 -- 172.21.15.98:0/3539015872 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c40035b0 con 0x7fc6dc146360 2023-03-03T08:00:37.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.923+0000 7fc6d9ffb700 1 -- 172.21.15.98:0/3539015872 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531771403 0 0) 0x7fc6dc0cdf20 con 0x7fc6dc0b51c0 2023-03-03T08:00:37.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.923+0000 7fc6d9ffb700 1 -- 172.21.15.98:0/3539015872 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6dc0d10f0 con 0x7fc6dc0b51c0 2023-03-03T08:00:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.923+0000 7fc6d9ffb700 1 -- 172.21.15.98:0/3539015872 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 959907130 0 0) 0x7fc6c40035b0 con 0x7fc6dc146360 2023-03-03T08:00:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.923+0000 7fc6db7fe700 1 -- 172.21.15.98:0/3539015872 >> v1:172.21.15.100:6789/0 conn(0x7fc6dc0b51c0 legacy=0x7fc6dc0d1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.923+0000 7fc6db7fe700 1 -- 172.21.15.98:0/3539015872 >> v1:172.21.15.98:6789/0 conn(0x7fc6dc146360 legacy=0x7fc6dc0b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.923+0000 7fc6db7fe700 1 -- 172.21.15.98:0/3539015872 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc0cd960 con 0x7fc6dc160f80 2023-03-03T08:00:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.923+0000 7fc6db7fe700 1 --2- 172.21.15.98:0/3539015872 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6dc160f80 0x7fc6dc0d1c90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009dc0 tx=0x7fc6d400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25456157c6de9352 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.924+0000 7fc6d9ffb700 1 -- 172.21.15.98:0/3539015872 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d400e070 con 0x7fc6dc160f80 2023-03-03T08:00:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.924+0000 7fc6d9ffb700 1 -- 172.21.15.98:0/3539015872 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d4003a20 con 0x7fc6dc160f80 2023-03-03T08:00:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.924+0000 7fc6d9ffb700 1 -- 172.21.15.98:0/3539015872 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc001670 con 0x7fc6dc160f80 2023-03-03T08:00:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.924+0000 7fc6dbfff700 1 -- 172.21.15.98:0/3539015872 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6dc160f80 msgr2=0x7fc6dc0d1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.924+0000 7fc6dbfff700 1 --2- 172.21.15.98:0/3539015872 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6dc160f80 0x7fc6dc0d1c90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009dc0 tx=0x7fc6d400b330 comp rx=0 tx=0).stop 2023-03-03T08:00:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.924+0000 7fc6dbfff700 1 -- 172.21.15.98:0/3539015872 shutdown_connections 2023-03-03T08:00:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.924+0000 7fc6dbfff700 1 --2- 172.21.15.98:0/3539015872 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6dc160f80 0x7fc6dc0d1c90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.924+0000 7fc6dbfff700 1 -- 172.21.15.98:0/3539015872 >> 172.21.15.98:0/3539015872 conn(0x7fc6dc06b8e0 msgr2=0x7fc6dc0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.925+0000 7fc6dbfff700 1 -- 172.21.15.98:0/3539015872 shutdown_connections 2023-03-03T08:00:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.925+0000 7fc6dbfff700 1 -- 172.21.15.98:0/3539015872 wait complete. 2023-03-03T08:00:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.927+0000 7fc6dbfff700 1 Processor -- start 2023-03-03T08:00:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.927+0000 7fc6dbfff700 1 -- start start 2023-03-03T08:00:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.928+0000 7fc6dbfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6dc0b51c0 0x7fc6dc0c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.928+0000 7fc6dbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6dc146360 0x7fc6dc0c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.928+0000 7fc6dbfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6dc160f80 0x7fc6dc14bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.928+0000 7fc6dbfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc0c6400 con 0x7fc6dc146360 2023-03-03T08:00:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.928+0000 7fc6dbfff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc160040 con 0x7fc6dc160f80 2023-03-03T08:00:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.928+0000 7fc6dbfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc15f4c0 con 0x7fc6dc0b51c0 2023-03-03T08:00:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.928+0000 7fc6da7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6dc146360 0x7fc6dc0c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.928+0000 7fc6da7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6dc146360 0x7fc6dc0c5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51490/0 (socket says 172.21.15.98:51490) 2023-03-03T08:00:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.928+0000 7fc6da7fc700 1 -- 172.21.15.98:0/907997899 learned_addr learned my addr 172.21.15.98:0/907997899 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:37.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.928+0000 7fc6db7fe700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6dc160f80 0x7fc6dc14bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:37.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.928+0000 7fc6daffd700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6dc0b51c0 0x7fc6dc0c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:37.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.929+0000 7fc6da7fc700 1 -- 172.21.15.98:0/907997899 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6dc0b51c0 msgr2=0x7fc6dc0c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:37.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.929+0000 7fc6da7fc700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6dc0b51c0 0x7fc6dc0c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:37.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.929+0000 7fc6da7fc700 1 -- 172.21.15.98:0/907997899 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6dc160f80 msgr2=0x7fc6dc14bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:37.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.929+0000 7fc6da7fc700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6dc160f80 0x7fc6dc14bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:37.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.929+0000 7fc6da7fc700 1 -- 172.21.15.98:0/907997899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d4009a70 con 0x7fc6dc146360 2023-03-03T08:00:37.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.929+0000 7fc6db7fe700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6dc160f80 0x7fc6dc14bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:37.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.929+0000 7fc6daffd700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6dc0b51c0 0x7fc6dc0c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:37.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.929+0000 7fc6da7fc700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6dc146360 0x7fc6dc0c5ec0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000c450 tx=0x7fc6d000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:37.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.929+0000 7fc6c3fff700 1 -- 172.21.15.98:0/907997899 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d00052d0 con 0x7fc6dc146360 2023-03-03T08:00:37.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.929+0000 7fc6c3fff700 1 -- 172.21.15.98:0/907997899 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d0005430 con 0x7fc6dc146360 2023-03-03T08:00:37.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.929+0000 7fc6dbfff700 1 -- 172.21.15.98:0/907997899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6dc0d0d80 con 0x7fc6dc146360 2023-03-03T08:00:37.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.929+0000 7fc6c3fff700 1 -- 172.21.15.98:0/907997899 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d00038c0 con 0x7fc6dc146360 2023-03-03T08:00:37.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.929+0000 7fc6dbfff700 1 -- 172.21.15.98:0/907997899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6dc0cdc40 con 0x7fc6dc146360 2023-03-03T08:00:37.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.931+0000 7fc6dbfff700 1 -- 172.21.15.98:0/907997899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6dc1613c0 con 0x7fc6dc146360 2023-03-03T08:00:37.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.932+0000 7fc6c3fff700 1 -- 172.21.15.98:0/907997899 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fc6d0019030 con 0x7fc6dc146360 2023-03-03T08:00:37.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.933+0000 7fc6c3fff700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc6c80683c0 0x7fc6c806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:37.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.933+0000 7fc6c3fff700 1 -- 172.21.15.98:0/907997899 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc6d0085260 con 0x7fc6dc146360 2023-03-03T08:00:37.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.933+0000 7fc6daffd700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc6c80683c0 0x7fc6c806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:37.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.934+0000 7fc6daffd700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc6c80683c0 0x7fc6c806a870 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc007fd0 tx=0x7fc6cc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:37.937+0000 7fc6c3fff700 1 -- 172.21.15.98:0/907997899 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc6d0051960 con 0x7fc6dc146360 2023-03-03T08:00:38.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:38.174+0000 7fc6dbfff700 1 -- 172.21.15.98:0/907997899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc6dc0c7030 con 0x7fc6dc146360 2023-03-03T08:00:38.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:38.174+0000 7fc6c3fff700 1 -- 172.21.15.98:0/907997899 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v31) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc6dc0c7030 con 0x7fc6dc146360 2023-03-03T08:00:38.177 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T08:00:38.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:38.177+0000 7fc6dbfff700 1 -- 172.21.15.98:0/907997899 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc6c80683c0 msgr2=0x7fc6c806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:38.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:38.177+0000 7fc6dbfff700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc6c80683c0 0x7fc6c806a870 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc007fd0 tx=0x7fc6cc009040 comp rx=0 tx=0).stop 2023-03-03T08:00:38.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:38.177+0000 7fc6dbfff700 1 -- 172.21.15.98:0/907997899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6dc146360 msgr2=0x7fc6dc0c5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:38.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:38.177+0000 7fc6dbfff700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6dc146360 0x7fc6dc0c5ec0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000c450 tx=0x7fc6d000c810 comp rx=0 tx=0).stop 2023-03-03T08:00:38.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:38.177+0000 7fc6dbfff700 1 -- 172.21.15.98:0/907997899 shutdown_connections 2023-03-03T08:00:38.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:38.177+0000 7fc6dbfff700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc6c80683c0 0x7fc6c806a870 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:38.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:38.177+0000 7fc6dbfff700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6dc160f80 0x7fc6dc14bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:38.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:38.177+0000 7fc6dbfff700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6dc0b51c0 0x7fc6dc0c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:38.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:38.177+0000 7fc6dbfff700 1 --2- 172.21.15.98:0/907997899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6dc146360 0x7fc6dc0c5ec0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:38.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:38.177+0000 7fc6dbfff700 1 -- 172.21.15.98:0/907997899 >> 172.21.15.98:0/907997899 conn(0x7fc6dc06b8e0 msgr2=0x7fc6dc15c450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:38.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:38.177+0000 7fc6dbfff700 1 -- 172.21.15.98:0/907997899 shutdown_connections 2023-03-03T08:00:38.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:38.177+0000 7fc6dbfff700 1 -- 172.21.15.98:0/907997899 wait complete. 2023-03-03T08:00:38.206 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T08:00:38.207 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T08:00:38.207 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2023-03-03T08:00:38.207 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-03-03T08:00:38.207 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-03-03T08:00:38.208 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T08:00:38.208 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-03-03T08:00:38.208 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T08:00:38.208 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-03-03T08:00:38.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:38.209 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2023-03-03T08:00:38.282 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T08:00:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T08:00:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T08:00:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:00:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2023-03-03T08:00:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:00:38.283 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:00:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:00:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2023-03-03T08:00:38.284 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:00:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:00:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2023-03-03T08:00:38.284 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:00:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:00:38.284 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T08:00:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:00:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T08:00:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 41922sec preferred_lft 41922sec 2023-03-03T08:00:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2023-03-03T08:00:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:00:38.285 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T08:00:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ee:78:63:4a:ea:32 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:00:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-03-03T08:00:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:00:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::9406:dcff:fe40:a4a4/64 scope link 2023-03-03T08:00:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:00:38.286 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:00:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:5e:bd:c1:84:4d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-03-03T08:00:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f45e:bdff:fec1:844d/64 scope link 2023-03-03T08:00:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:00:38.287 INFO:teuthology.orchestra.run.smithi100.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:00:38.287 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ee:78:63:4a:ea:32 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-03-03T08:00:38.287 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec78:63ff:fe4a:ea32/64 scope link 2023-03-03T08:00:38.287 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:00:38.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:38.288 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T08:00:38.389 INFO:teuthology.orchestra.run.smithi100.stdout:1 2023-03-03T08:00:38.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:38.395 DEBUG:teuthology.orchestra.run.smithi100:> route 2023-03-03T08:00:38.467 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2023-03-03T08:00:38.467 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T08:00:38.467 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T08:00:38.467 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T08:00:38.468 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T08:00:38.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:38.469 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2023-03-03T08:00:38.469 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2023-03-03T08:00:38.470 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T08:00:38.470 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T08:00:38.470 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T08:00:38.470 DEBUG:teuthology.orchestra.run.smithi100:> ') 2023-03-03T08:00:38.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:38.712 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2023-03-03T08:00:38.780 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-03-03T08:00:38.781 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-03-03T08:00:38.782 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-03-03T08:00:38.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:38.782 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2023-03-03T08:00:38.882 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-03-03T08:00:38.883 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:00:38.883 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-03-03T08:00:38.951 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-03-03T08:00:38.952 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:00:38.952 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-03-03T08:00:39.019 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T08:00:39.121 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:00:39.121 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:00:39.218 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:00:39.222 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:00:39.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:39.224 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T08:00:39.292 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-03-03T08:00:39.293 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_mountpoint=/) 2023-03-03T08:00:39.298 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T08:00:39.405 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:00:39.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:00:39.505 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:00:39.509 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:00:39.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:39.510 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T08:00:39.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2023-03-03T08:00:39.519+0000 7f9f7ad8d5c0 -1 init, newargv = 0x558b52dc4250 newargc=15 2023-03-03T08:00:39.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[118867]: starting ceph client 2023-03-03T08:00:39.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[118867]: starting fuse 2023-03-03T08:00:39.582 INFO:teuthology.orchestra.run.smithi100.stdout:42 2023-03-03T08:00:39.583 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2023-03-03T08:00:39.584 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T08:00:39.584 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2023-03-03T08:00:39.584 DEBUG:teuthology.orchestra.run.smithi100:> import re 2023-03-03T08:00:39.585 DEBUG:teuthology.orchestra.run.smithi100:> import os 2023-03-03T08:00:39.585 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2023-03-03T08:00:39.585 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:00:39.586 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2023-03-03T08:00:39.586 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-03-03T08:00:39.586 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2023-03-03T08:00:39.586 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-03-03T08:00:39.587 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:00:39.587 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2023-03-03T08:00:39.587 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2023-03-03T08:00:39.588 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2023-03-03T08:00:39.588 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2023-03-03T08:00:39.588 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:00:39.588 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2023-03-03T08:00:39.589 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T08:00:39.589 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T08:00:39.589 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T08:00:39.590 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2023-03-03T08:00:39.590 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2023-03-03T08:00:39.590 DEBUG:teuthology.orchestra.run.smithi100:> return f 2023-03-03T08:00:39.591 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T08:00:39.591 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:00:39.591 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.1")) 2023-03-03T08:00:39.591 DEBUG:teuthology.orchestra.run.smithi100:> ' 2023-03-03T08:00:39.810 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.1.118867.asok 2023-03-03T08:00:39.987 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.118867.asok 2023-03-03T08:00:39.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:39.988 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.118867.asok status 2023-03-03T08:00:40.256 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2023-03-03T08:00:40.257 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2023-03-03T08:00:40.257 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T08:00:40.257 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T08:00:40.257 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "1", 2023-03-03T08:00:40.257 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2023-03-03T08:00:40.258 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-03-03T08:00:40.258 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "118867", 2023-03-03T08:00:40.258 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2023-03-03T08:00:40.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:00:40.258 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2023-03-03T08:00:40.258 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2023-03-03T08:00:40.258 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 6445, 2023-03-03T08:00:40.258 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2023-03-03T08:00:40.259 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2023-03-03T08:00:40.259 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2023-03-03T08:00:40.259 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 6445 2023-03-03T08:00:40.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:00:40.259 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T08:00:40.259 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T08:00:40.259 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:00:40.259 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2596213384 2023-03-03T08:00:40.260 INFO:teuthology.orchestra.run.smithi100.stdout: } 2023-03-03T08:00:40.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:00:40.260 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T08:00:40.260 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T08:00:40.260 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:00:40.260 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2596213384 2023-03-03T08:00:40.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:00:40.261 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.6445 192.168.0.1:0/2596213384", 2023-03-03T08:00:40.261 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.1:0/2596213384", 2023-03-03T08:00:40.261 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2023-03-03T08:00:40.261 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 31, 2023-03-03T08:00:40.261 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 66, 2023-03-03T08:00:40.261 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2023-03-03T08:00:40.261 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2023-03-03T08:00:40.261 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2023-03-03T08:00:40.262 INFO:teuthology.orchestra.run.smithi100.stdout:} 2023-03-03T08:00:40.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:40.277 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2023-03-03T08:00:40.277 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2023-03-03T08:00:40.346 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2023-03-03T08:00:40.346 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-03-03T08:00:40.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:40.347 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-03-03T08:00:40.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-03-03T08:00:40.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.703+0000 7fc1481ff700 1 Processor -- start 2023-03-03T08:00:40.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.703+0000 7fc1481ff700 1 -- start start 2023-03-03T08:00:40.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.705+0000 7fc1481ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc1400cd770 0x7fc1400b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:40.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.705+0000 7fc1481ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc14015bf70 0x7fc14015c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:40.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.705+0000 7fc1481ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1400d0ac0 con 0x7fc1400b8880 2023-03-03T08:00:40.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.705+0000 7fc1481ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1400cdbe0 con 0x7fc14015bf70 2023-03-03T08:00:40.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.705+0000 7fc1481ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1400cff70 con 0x7fc1400cd770 2023-03-03T08:00:40.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.705+0000 7fc14679c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc14015bf70 0x7fc14015c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:40.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.705+0000 7fc145f9b700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fc1400b8880 0x7fc1400b4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:39432/0 (socket says 172.21.15.98:39432) 2023-03-03T08:00:40.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.705+0000 7fc145f9b700 1 -- 172.21.15.98:0/708033875 learned_addr learned my addr 172.21.15.98:0/708033875 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:40.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.705+0000 7fc14579a700 1 --2- 172.21.15.98:0/708033875 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc1400cd770 0x7fc1400b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:40.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.706+0000 7fc14679c700 1 -- 172.21.15.98:0/708033875 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc1400cd770 msgr2=0x7fc1400b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:40.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.706+0000 7fc14679c700 1 --2- 172.21.15.98:0/708033875 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc1400cd770 0x7fc1400b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:40.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.706+0000 7fc14679c700 1 -- 172.21.15.98:0/708033875 >> v1:172.21.15.100:6789/0 conn(0x7fc1400b8880 legacy=0x7fc1400b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:40.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.706+0000 7fc14679c700 1 -- 172.21.15.98:0/708033875 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1400d0250 con 0x7fc14015bf70 2023-03-03T08:00:40.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.706+0000 7fc144f99700 1 -- 172.21.15.98:0/708033875 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903503123 0 0) 0x7fc1400d0ac0 con 0x7fc1400b8880 2023-03-03T08:00:40.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.706+0000 7fc14679c700 1 --2- 172.21.15.98:0/708033875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc14015bf70 0x7fc14015c3c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc138009d90 tx=0x7fc13800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec2b4c6e6cd7b19 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:40.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.706+0000 7fc144f99700 1 -- 172.21.15.98:0/708033875 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13800e070 con 0x7fc14015bf70 2023-03-03T08:00:40.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.706+0000 7fc144f99700 1 -- 172.21.15.98:0/708033875 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13800bb20 con 0x7fc14015bf70 2023-03-03T08:00:40.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.706+0000 7fc144f99700 1 -- 172.21.15.98:0/708033875 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc138012430 con 0x7fc14015bf70 2023-03-03T08:00:40.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.707+0000 7fc1481ff700 1 -- 172.21.15.98:0/708033875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc14015bf70 msgr2=0x7fc14015c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:40.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.707+0000 7fc1481ff700 1 --2- 172.21.15.98:0/708033875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc14015bf70 0x7fc14015c3c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc138009d90 tx=0x7fc13800b330 comp rx=0 tx=0).stop 2023-03-03T08:00:40.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.707+0000 7fc1481ff700 1 -- 172.21.15.98:0/708033875 shutdown_connections 2023-03-03T08:00:40.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.707+0000 7fc1481ff700 1 --2- 172.21.15.98:0/708033875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc14015bf70 0x7fc14015c3c0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:40.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.707+0000 7fc1481ff700 1 --2- 172.21.15.98:0/708033875 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc1400cd770 0x7fc1400b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:40.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.707+0000 7fc1481ff700 1 -- 172.21.15.98:0/708033875 >> 172.21.15.98:0/708033875 conn(0x7fc14006ad80 msgr2=0x7fc14006b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:40.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.707+0000 7fc1481ff700 1 -- 172.21.15.98:0/708033875 shutdown_connections 2023-03-03T08:00:40.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.707+0000 7fc1481ff700 1 -- 172.21.15.98:0/708033875 wait complete. 2023-03-03T08:00:40.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.709+0000 7fc1481ff700 1 Processor -- start 2023-03-03T08:00:40.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.709+0000 7fc1481ff700 1 -- start start 2023-03-03T08:00:40.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.710+0000 7fc1481ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1400b8880 0x7fc140157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:40.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.710+0000 7fc1481ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1400cd770 0x7fc1401e9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:40.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.710+0000 7fc1481ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc14015bf70 0x7fc140144210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:40.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.710+0000 7fc1481ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1401421c0 con 0x7fc1400cd770 2023-03-03T08:00:40.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.710+0000 7fc1481ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc140159bb0 con 0x7fc1400b8880 2023-03-03T08:00:40.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.710+0000 7fc1481ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc140159e90 con 0x7fc14015bf70 2023-03-03T08:00:40.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.711+0000 7fc14579a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1400cd770 0x7fc1401e9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:40.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.711+0000 7fc14579a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1400cd770 0x7fc1401e9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51504/0 (socket says 172.21.15.98:51504) 2023-03-03T08:00:40.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.711+0000 7fc14579a700 1 -- 172.21.15.98:0/3197103991 learned_addr learned my addr 172.21.15.98:0/3197103991 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:40.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.711+0000 7fc14679c700 1 --2- 172.21.15.98:0/3197103991 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc14015bf70 0x7fc140144210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:40.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.711+0000 7fc145f9b700 1 --2- 172.21.15.98:0/3197103991 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1400b8880 0x7fc140157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:40.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.711+0000 7fc14579a700 1 -- 172.21.15.98:0/3197103991 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc14015bf70 msgr2=0x7fc140144210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:40.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.711+0000 7fc14579a700 1 --2- 172.21.15.98:0/3197103991 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc14015bf70 0x7fc140144210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:40.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.711+0000 7fc14579a700 1 -- 172.21.15.98:0/3197103991 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1400b8880 msgr2=0x7fc140157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:40.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.711+0000 7fc14579a700 1 --2- 172.21.15.98:0/3197103991 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1400b8880 0x7fc140157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:40.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.711+0000 7fc14579a700 1 -- 172.21.15.98:0/3197103991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc138009a70 con 0x7fc1400cd770 2023-03-03T08:00:40.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.711+0000 7fc145f9b700 1 --2- 172.21.15.98:0/3197103991 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1400b8880 0x7fc140157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:40.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.711+0000 7fc14579a700 1 --2- 172.21.15.98:0/3197103991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1400cd770 0x7fc1401e9180 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc13000e9b0 tx=0x7fc13000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:40.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.712+0000 7fc1367fc700 1 -- 172.21.15.98:0/3197103991 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc130009710 con 0x7fc1400cd770 2023-03-03T08:00:40.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.712+0000 7fc1367fc700 1 -- 172.21.15.98:0/3197103991 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13000c850 con 0x7fc1400cd770 2023-03-03T08:00:40.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.712+0000 7fc1367fc700 1 -- 172.21.15.98:0/3197103991 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13000cbc0 con 0x7fc1400cd770 2023-03-03T08:00:40.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.712+0000 7fc1481ff700 1 -- 172.21.15.98:0/3197103991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1400d0cf0 con 0x7fc1400cd770 2023-03-03T08:00:40.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.712+0000 7fc1481ff700 1 -- 172.21.15.98:0/3197103991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1400d0530 con 0x7fc1400cd770 2023-03-03T08:00:40.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.713+0000 7fc1481ff700 1 -- 172.21.15.98:0/3197103991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1400b8c50 con 0x7fc1400cd770 2023-03-03T08:00:40.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.714+0000 7fc1367fc700 1 -- 172.21.15.98:0/3197103991 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fc130019030 con 0x7fc1400cd770 2023-03-03T08:00:40.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.716+0000 7fc1367fc700 1 --2- 172.21.15.98:0/3197103991 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc1240683c0 0x7fc12406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:40.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.716+0000 7fc1367fc700 1 -- 172.21.15.98:0/3197103991 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc130085df0 con 0x7fc1400cd770 2023-03-03T08:00:40.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.716+0000 7fc145f9b700 1 --2- 172.21.15.98:0/3197103991 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc1240683c0 0x7fc12406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:40.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.717+0000 7fc145f9b700 1 --2- 172.21.15.98:0/3197103991 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc1240683c0 0x7fc12406a870 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc13c000f60 tx=0x7fc13c007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:40.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.719+0000 7fc1367fc700 1 -- 172.21.15.98:0/3197103991 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc130052490 con 0x7fc1400cd770 2023-03-03T08:00:40.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:40.958+0000 7fc1481ff700 1 -- 172.21.15.98:0/3197103991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc14004e6e0 con 0x7fc1400cd770 2023-03-03T08:00:41.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.639+0000 7fc1367fc700 1 -- 172.21.15.98:0/3197103991 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v32) v1 ==== 136+0+0 (secure 0 0 0) 0x7fc130055ab0 con 0x7fc1400cd770 2023-03-03T08:00:41.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.643+0000 7fc1481ff700 1 -- 172.21.15.98:0/3197103991 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc1240683c0 msgr2=0x7fc12406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:41.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.643+0000 7fc1481ff700 1 --2- 172.21.15.98:0/3197103991 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc1240683c0 0x7fc12406a870 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc13c000f60 tx=0x7fc13c007d30 comp rx=0 tx=0).stop 2023-03-03T08:00:41.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.643+0000 7fc1481ff700 1 -- 172.21.15.98:0/3197103991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1400cd770 msgr2=0x7fc1401e9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:41.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.643+0000 7fc1481ff700 1 --2- 172.21.15.98:0/3197103991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1400cd770 0x7fc1401e9180 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc13000e9b0 tx=0x7fc13000ecc0 comp rx=0 tx=0).stop 2023-03-03T08:00:41.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.643+0000 7fc1481ff700 1 -- 172.21.15.98:0/3197103991 shutdown_connections 2023-03-03T08:00:41.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.643+0000 7fc1481ff700 1 --2- 172.21.15.98:0/3197103991 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc1240683c0 0x7fc12406a870 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:41.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.643+0000 7fc1481ff700 1 --2- 172.21.15.98:0/3197103991 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc14015bf70 0x7fc140144210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:41.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.643+0000 7fc1481ff700 1 --2- 172.21.15.98:0/3197103991 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1400b8880 0x7fc140157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:41.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.643+0000 7fc1481ff700 1 --2- 172.21.15.98:0/3197103991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1400cd770 0x7fc1401e9180 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:41.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.643+0000 7fc1481ff700 1 -- 172.21.15.98:0/3197103991 >> 172.21.15.98:0/3197103991 conn(0x7fc14006ad80 msgr2=0x7fc1400b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:41.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.643+0000 7fc1481ff700 1 -- 172.21.15.98:0/3197103991 shutdown_connections 2023-03-03T08:00:41.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.643+0000 7fc1481ff700 1 -- 172.21.15.98:0/3197103991 wait complete. 2023-03-03T08:00:41.674 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-03-03T08:00:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.920+0000 7fd3e619d700 1 Processor -- start 2023-03-03T08:00:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.921+0000 7fd3e619d700 1 -- start start 2023-03-03T08:00:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.922+0000 7fd3e619d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd3e00b4aa0 0x7fd3e00b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.922+0000 7fd3e619d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e015dd00 con 0x7fd3e00cf570 2023-03-03T08:00:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.922+0000 7fd3e619d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e00cd780 con 0x7fd3e01609b0 2023-03-03T08:00:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.922+0000 7fd3e619d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e00cebb0 con 0x7fd3e00b4aa0 2023-03-03T08:00:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.922+0000 7fd3e499a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd3e00cf570 0x7fd3e015d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36612/0 (socket says 172.21.15.98:36612) 2023-03-03T08:00:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.923+0000 7fd3e499a700 1 -- 172.21.15.98:0/3582715729 learned_addr learned my addr 172.21.15.98:0/3582715729 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.923+0000 7fd3e599c700 1 --2- 172.21.15.98:0/3582715729 >> v2:172.21.15.100:3300/0 conn(0x7fd3e00b4aa0 0x7fd3e00b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.923+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403555856 0 0) 0x7fd3e015dd00 con 0x7fd3e00cf570 2023-03-03T08:00:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.923+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3cc0035b0 con 0x7fd3e00cf570 2023-03-03T08:00:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.923+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723438894 0 0) 0x7fd3e00cd780 con 0x7fd3e01609b0 2023-03-03T08:00:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.923+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3e015dd00 con 0x7fd3e01609b0 2023-03-03T08:00:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.923+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1162615234 0 0) 0x7fd3cc0035b0 con 0x7fd3e00cf570 2023-03-03T08:00:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.923+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 >> v2:172.21.15.100:3300/0 conn(0x7fd3e00b4aa0 msgr2=0x7fd3e00b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.923+0000 7fd3d7fff700 1 --2- 172.21.15.98:0/3582715729 >> v2:172.21.15.100:3300/0 conn(0x7fd3e00b4aa0 0x7fd3e00b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.923+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 >> v1:172.21.15.100:6789/0 conn(0x7fd3e01609b0 legacy=0x7fd3e0160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.923+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e00cda60 con 0x7fd3e00cf570 2023-03-03T08:00:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.923+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7fd3d0001670 con 0x7fd3e00cf570 2023-03-03T08:00:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 >> v1:172.21.15.98:6789/0 conn(0x7fd3e00cf570 legacy=0x7fd3e015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3d7fff700 1 --2- 172.21.15.98:0/3582715729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc003be0 0x7fd3cc003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3d7fff700 1 --2- 172.21.15.98:0/3582715729 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3cc004850 0x7fd3cc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3d7fff700 1 --2- 172.21.15.98:0/3582715729 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3cc005200 0x7fd3cc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc00c0e0 con 0x7fd3cc003be0 2023-03-03T08:00:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc00c220 con 0x7fd3cc005200 2023-03-03T08:00:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc00c3b0 con 0x7fd3cc004850 2023-03-03T08:00:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3e519b700 1 --2- 172.21.15.98:0/3582715729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc003be0 0x7fd3cc003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3e499a700 1 --2- 172.21.15.98:0/3582715729 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3cc004850 0x7fd3cc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3e599c700 1 --2- 172.21.15.98:0/3582715729 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3cc005200 0x7fd3cc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3e519b700 1 -- 172.21.15.98:0/3582715729 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3cc004850 msgr2=0x7fd3cc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3e519b700 1 --2- 172.21.15.98:0/3582715729 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3cc004850 0x7fd3cc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3e519b700 1 -- 172.21.15.98:0/3582715729 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3cc005200 msgr2=0x7fd3cc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3e519b700 1 --2- 172.21.15.98:0/3582715729 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3cc005200 0x7fd3cc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3e519b700 1 -- 172.21.15.98:0/3582715729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3cc0035b0 con 0x7fd3cc003be0 2023-03-03T08:00:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.924+0000 7fd3e599c700 1 -- 172.21.15.98:0/3582715729 reap_dead start 2023-03-03T08:00:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.925+0000 7fd3e499a700 1 --2- 172.21.15.98:0/3582715729 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3cc004850 0x7fd3cc004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.925+0000 7fd3e519b700 1 --2- 172.21.15.98:0/3582715729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc003be0 0x7fd3cc003fd0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc007fd0 tx=0x7fd3cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=956c3ab993584aef server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.925+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc001430 con 0x7fd3cc003be0 2023-03-03T08:00:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.925+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3dc0052d0 con 0x7fd3cc003be0 2023-03-03T08:00:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.925+0000 7fd3d7fff700 1 -- 172.21.15.98:0/3582715729 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc001d40 con 0x7fd3cc003be0 2023-03-03T08:00:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.926+0000 7fd3e619d700 1 -- 172.21.15.98:0/3582715729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc003be0 msgr2=0x7fd3cc003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.926+0000 7fd3e619d700 1 --2- 172.21.15.98:0/3582715729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc003be0 0x7fd3cc003fd0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc007fd0 tx=0x7fd3cc005880 comp rx=0 tx=0).stop 2023-03-03T08:00:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.926+0000 7fd3e619d700 1 -- 172.21.15.98:0/3582715729 shutdown_connections 2023-03-03T08:00:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.926+0000 7fd3e619d700 1 --2- 172.21.15.98:0/3582715729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc003be0 0x7fd3cc003fd0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.926+0000 7fd3e619d700 1 -- 172.21.15.98:0/3582715729 >> 172.21.15.98:0/3582715729 conn(0x7fd3e006b880 msgr2=0x7fd3e00cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.926+0000 7fd3e619d700 1 -- 172.21.15.98:0/3582715729 shutdown_connections 2023-03-03T08:00:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.926+0000 7fd3e619d700 1 -- 172.21.15.98:0/3582715729 wait complete. 2023-03-03T08:00:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.928+0000 7fd3e619d700 1 Processor -- start 2023-03-03T08:00:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.928+0000 7fd3e619d700 1 -- start start 2023-03-03T08:00:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.929+0000 7fd3e619d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc003be0 0x7fd3e00b89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.929+0000 7fd3e619d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3e014f2d0 0x7fd3e00c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.929+0000 7fd3e619d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3e00c9450 0x7fd3e00d1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.929+0000 7fd3e619d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e00c8a80 con 0x7fd3cc003be0 2023-03-03T08:00:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.929+0000 7fd3e619d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e00c7d50 con 0x7fd3e014f2d0 2023-03-03T08:00:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.929+0000 7fd3e619d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e00c8030 con 0x7fd3e00c9450 2023-03-03T08:00:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.929+0000 7fd3e519b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc003be0 0x7fd3e00b89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.930+0000 7fd3e519b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc003be0 0x7fd3e00b89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51510/0 (socket says 172.21.15.98:51510) 2023-03-03T08:00:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.930+0000 7fd3e519b700 1 -- 172.21.15.98:0/2503501734 learned_addr learned my addr 172.21.15.98:0/2503501734 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:41.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.930+0000 7fd3e599c700 1 --2- 172.21.15.98:0/2503501734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3e00c9450 0x7fd3e00d1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:41.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.930+0000 7fd3e499a700 1 --2- 172.21.15.98:0/2503501734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3e014f2d0 0x7fd3e00c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:41.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.930+0000 7fd3e519b700 1 -- 172.21.15.98:0/2503501734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3e00c9450 msgr2=0x7fd3e00d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:41.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.930+0000 7fd3e519b700 1 --2- 172.21.15.98:0/2503501734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3e00c9450 0x7fd3e00d1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:41.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.930+0000 7fd3e519b700 1 -- 172.21.15.98:0/2503501734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3e014f2d0 msgr2=0x7fd3e00c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:41.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.930+0000 7fd3e519b700 1 --2- 172.21.15.98:0/2503501734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3e014f2d0 0x7fd3e00c8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:41.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.930+0000 7fd3e519b700 1 -- 172.21.15.98:0/2503501734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e015cdf0 con 0x7fd3cc003be0 2023-03-03T08:00:41.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.930+0000 7fd3e519b700 1 --2- 172.21.15.98:0/2503501734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc003be0 0x7fd3e00b89a0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc0047a0 tx=0x7fd3dc0012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:41.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.931+0000 7fd3d5ffb700 1 -- 172.21.15.98:0/2503501734 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc001d40 con 0x7fd3cc003be0 2023-03-03T08:00:41.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.931+0000 7fd3d5ffb700 1 -- 172.21.15.98:0/2503501734 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3dc0013d0 con 0x7fd3cc003be0 2023-03-03T08:00:41.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.931+0000 7fd3d5ffb700 1 -- 172.21.15.98:0/2503501734 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc007040 con 0x7fd3cc003be0 2023-03-03T08:00:41.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.931+0000 7fd3e619d700 1 -- 172.21.15.98:0/2503501734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e00cdd40 con 0x7fd3cc003be0 2023-03-03T08:00:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.931+0000 7fd3e619d700 1 -- 172.21.15.98:0/2503501734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e015d0d0 con 0x7fd3cc003be0 2023-03-03T08:00:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.932+0000 7fd3e619d700 1 -- 172.21.15.98:0/2503501734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e0068290 con 0x7fd3cc003be0 2023-03-03T08:00:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.933+0000 7fd3d5ffb700 1 -- 172.21.15.98:0/2503501734 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fd3dc010050 con 0x7fd3cc003be0 2023-03-03T08:00:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.935+0000 7fd3d5ffb700 1 --2- 172.21.15.98:0/2503501734 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3c40683c0 0x7fd3c406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.935+0000 7fd3d5ffb700 1 -- 172.21.15.98:0/2503501734 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd3dc086510 con 0x7fd3cc003be0 2023-03-03T08:00:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.935+0000 7fd3e499a700 1 --2- 172.21.15.98:0/2503501734 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3c40683c0 0x7fd3c406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.936+0000 7fd3e499a700 1 --2- 172.21.15.98:0/2503501734 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3c40683c0 0x7fd3c406a870 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0001b60 tx=0x7fd3d0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:41.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:41.939+0000 7fd3d5ffb700 1 -- 172.21.15.98:0/2503501734 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd3dc052bb0 con 0x7fd3cc003be0 2023-03-03T08:00:42.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.156+0000 7fd3e619d700 1 -- 172.21.15.98:0/2503501734 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3e00c8bc0 con 0x7fd3c40683c0 2023-03-03T08:00:42.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.158+0000 7fd3d5ffb700 1 -- 172.21.15.98:0/2503501734 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fd3d0005e00 con 0x7fd3c40683c0 2023-03-03T08:00:42.160 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T08:00:42.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:42.160 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephfs" 2023-03-03T08:00:42.160 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:42.161 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T08:00:42.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.160+0000 7fd3e619d700 1 -- 172.21.15.98:0/2503501734 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3c40683c0 msgr2=0x7fd3c406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:42.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.160+0000 7fd3e619d700 1 --2- 172.21.15.98:0/2503501734 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3c40683c0 0x7fd3c406a870 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0001b60 tx=0x7fd3d0009040 comp rx=0 tx=0).stop 2023-03-03T08:00:42.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.160+0000 7fd3e619d700 1 -- 172.21.15.98:0/2503501734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc003be0 msgr2=0x7fd3e00b89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:42.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.160+0000 7fd3e619d700 1 --2- 172.21.15.98:0/2503501734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc003be0 0x7fd3e00b89a0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc0047a0 tx=0x7fd3dc0012d0 comp rx=0 tx=0).stop 2023-03-03T08:00:42.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.160+0000 7fd3e619d700 1 -- 172.21.15.98:0/2503501734 shutdown_connections 2023-03-03T08:00:42.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.160+0000 7fd3e619d700 1 --2- 172.21.15.98:0/2503501734 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd3c40683c0 0x7fd3c406a870 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:42.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.160+0000 7fd3e619d700 1 --2- 172.21.15.98:0/2503501734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc003be0 0x7fd3e00b89a0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:42.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.160+0000 7fd3e619d700 1 --2- 172.21.15.98:0/2503501734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3e00c9450 0x7fd3e00d1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:42.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.160+0000 7fd3e619d700 1 --2- 172.21.15.98:0/2503501734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd3e014f2d0 0x7fd3e00c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:42.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.160+0000 7fd3e619d700 1 -- 172.21.15.98:0/2503501734 >> 172.21.15.98:0/2503501734 conn(0x7fd3e006b880 msgr2=0x7fd3e00d0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:42.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.161+0000 7fd3e619d700 1 -- 172.21.15.98:0/2503501734 shutdown_connections 2023-03-03T08:00:42.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.161+0000 7fd3e619d700 1 -- 172.21.15.98:0/2503501734 wait complete. 2023-03-03T08:00:42.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-03-03T08:00:42.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.438+0000 7f45a719d700 1 Processor -- start 2023-03-03T08:00:42.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.439+0000 7f45a719d700 1 -- start start 2023-03-03T08:00:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.440+0000 7f45a719d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f45a00b8910 0x7f45a01604a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.440+0000 7f45a719d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f45a00b5e70 0x7f45a00b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.440+0000 7f45a719d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a015ce00 con 0x7f45a00cf570 2023-03-03T08:00:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.440+0000 7f45a719d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a00d0bf0 con 0x7f45a00b5e70 2023-03-03T08:00:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.440+0000 7f45a719d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a00b8ce0 con 0x7f45a00b8910 2023-03-03T08:00:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.441+0000 7f45a599a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f45a00b5e70 0x7f45a00b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.441+0000 7f45a599a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f45a00b5e70 0x7f45a00b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51516/0 (socket says 172.21.15.98:51516) 2023-03-03T08:00:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.441+0000 7f45a599a700 1 -- 172.21.15.98:0/656968242 learned_addr learned my addr 172.21.15.98:0/656968242 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.441+0000 7f45a619b700 1 --2- 172.21.15.98:0/656968242 >> v2:172.21.15.100:3300/0 conn(0x7f45a00b8910 0x7f45a01604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.441+0000 7f45a599a700 1 -- 172.21.15.98:0/656968242 >> v2:172.21.15.100:3300/0 conn(0x7f45a00b8910 msgr2=0x7f45a01604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.441+0000 7f45a599a700 1 --2- 172.21.15.98:0/656968242 >> v2:172.21.15.100:3300/0 conn(0x7f45a00b8910 0x7f45a01604a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.441+0000 7f45a599a700 1 -- 172.21.15.98:0/656968242 >> v1:172.21.15.100:6789/0 conn(0x7f45a00cf570 legacy=0x7f45a00d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.441+0000 7f45a599a700 1 -- 172.21.15.98:0/656968242 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a0160cf0 con 0x7f45a00b5e70 2023-03-03T08:00:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.442+0000 7f45a599a700 1 --2- 172.21.15.98:0/656968242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45a00b5e70 0x7f45a00b4aa0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f45a008bf50 tx=0x7f4590009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d82ddf53f68df884 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.442+0000 7f45a5199700 1 -- 172.21.15.98:0/656968242 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459000b870 con 0x7f45a00b5e70 2023-03-03T08:00:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.442+0000 7f45a5199700 1 -- 172.21.15.98:0/656968242 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459000b9d0 con 0x7f45a00b5e70 2023-03-03T08:00:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.442+0000 7f45a5199700 1 -- 172.21.15.98:0/656968242 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459000bd60 con 0x7f45a00b5e70 2023-03-03T08:00:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.442+0000 7f45a719d700 1 -- 172.21.15.98:0/656968242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45a00b5e70 msgr2=0x7f45a00b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.442+0000 7f45a719d700 1 --2- 172.21.15.98:0/656968242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45a00b5e70 0x7f45a00b4aa0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f45a008bf50 tx=0x7f4590009bd0 comp rx=0 tx=0).stop 2023-03-03T08:00:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.442+0000 7f45a719d700 1 -- 172.21.15.98:0/656968242 shutdown_connections 2023-03-03T08:00:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.443+0000 7f45a719d700 1 --2- 172.21.15.98:0/656968242 >> v2:172.21.15.100:3300/0 conn(0x7f45a00b8910 0x7f45a01604a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.443+0000 7f45a719d700 1 --2- 172.21.15.98:0/656968242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45a00b5e70 0x7f45a00b4aa0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.443+0000 7f45a719d700 1 -- 172.21.15.98:0/656968242 >> 172.21.15.98:0/656968242 conn(0x7f45a006b880 msgr2=0x7f45a00cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.443+0000 7f45a719d700 1 -- 172.21.15.98:0/656968242 shutdown_connections 2023-03-03T08:00:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.443+0000 7f45a719d700 1 -- 172.21.15.98:0/656968242 wait complete. 2023-03-03T08:00:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.445+0000 7f45a719d700 1 Processor -- start 2023-03-03T08:00:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.445+0000 7f45a719d700 1 -- start start 2023-03-03T08:00:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.446+0000 7f45a719d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f45a00b5e70 0x7f45a01e6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.446+0000 7f45a719d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f45a00b8910 0x7f45a014e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.446+0000 7f45a719d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45a00cf570 0x7f45a00cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.446+0000 7f45a719d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a01f2950 con 0x7f45a00cf570 2023-03-03T08:00:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.446+0000 7f45a719d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a01f2c10 con 0x7f45a00b5e70 2023-03-03T08:00:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.446+0000 7f45a719d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a01e7520 con 0x7f45a00b8910 2023-03-03T08:00:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.446+0000 7f45a699c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45a00cf570 0x7f45a00cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.446+0000 7f45a619b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f45a00b5e70 0x7f45a01e6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.446+0000 7f45a699c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45a00cf570 0x7f45a00cbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51518/0 (socket says 172.21.15.98:51518) 2023-03-03T08:00:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.446+0000 7f45a599a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f45a00b8910 0x7f45a014e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.446+0000 7f45a699c700 1 -- 172.21.15.98:0/3462304867 learned_addr learned my addr 172.21.15.98:0/3462304867 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.447+0000 7f45a699c700 1 -- 172.21.15.98:0/3462304867 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f45a00b8910 msgr2=0x7f45a014e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.447+0000 7f45a699c700 1 --2- 172.21.15.98:0/3462304867 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f45a00b8910 0x7f45a014e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.447+0000 7f45a699c700 1 -- 172.21.15.98:0/3462304867 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f45a00b5e70 msgr2=0x7f45a01e6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.447+0000 7f45a699c700 1 --2- 172.21.15.98:0/3462304867 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f45a00b5e70 0x7f45a01e6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.447+0000 7f45a699c700 1 -- 172.21.15.98:0/3462304867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4590009550 con 0x7f45a00cf570 2023-03-03T08:00:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.447+0000 7f45a699c700 1 --2- 172.21.15.98:0/3462304867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45a00cf570 0x7f45a00cbaa0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f459800cb50 tx=0x7f459800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.447+0000 7f4596ffd700 1 -- 172.21.15.98:0/3462304867 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4598003c00 con 0x7f45a00cf570 2023-03-03T08:00:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.447+0000 7f4596ffd700 1 -- 172.21.15.98:0/3462304867 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4598003d60 con 0x7f45a00cf570 2023-03-03T08:00:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.447+0000 7f45a719d700 1 -- 172.21.15.98:0/3462304867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a01612b0 con 0x7f45a00cf570 2023-03-03T08:00:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.447+0000 7f4596ffd700 1 -- 172.21.15.98:0/3462304867 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459800aa20 con 0x7f45a00cf570 2023-03-03T08:00:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.447+0000 7f45a719d700 1 -- 172.21.15.98:0/3462304867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a0160fd0 con 0x7f45a00cf570 2023-03-03T08:00:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.449+0000 7f45a719d700 1 -- 172.21.15.98:0/3462304867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a00d0eb0 con 0x7f45a00cf570 2023-03-03T08:00:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.450+0000 7f4596ffd700 1 -- 172.21.15.98:0/3462304867 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f4598010050 con 0x7f45a00cf570 2023-03-03T08:00:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.451+0000 7f4596ffd700 1 --2- 172.21.15.98:0/3462304867 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f45840683f0 0x7f458406a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.451+0000 7f4596ffd700 1 -- 172.21.15.98:0/3462304867 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4598085bc0 con 0x7f45a00cf570 2023-03-03T08:00:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.451+0000 7f45a619b700 1 --2- 172.21.15.98:0/3462304867 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f45840683f0 0x7f458406a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.452+0000 7f45a619b700 1 --2- 172.21.15.98:0/3462304867 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f45840683f0 0x7f458406a8a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f459c006b20 tx=0x7f459c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.455+0000 7f4596ffd700 1 -- 172.21.15.98:0/3462304867 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4598052260 con 0x7f45a00cf570 2023-03-03T08:00:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.652+0000 7f45a719d700 1 -- 172.21.15.98:0/3462304867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f45a015cde0 con 0x7f45a00cf570 2023-03-03T08:00:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.652+0000 7f4596ffd700 1 -- 172.21.15.98:0/3462304867 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4598055880 con 0x7f45a00cf570 2023-03-03T08:00:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.655+0000 7f45a719d700 1 -- 172.21.15.98:0/3462304867 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f45840683f0 msgr2=0x7f458406a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.655+0000 7f45a719d700 1 --2- 172.21.15.98:0/3462304867 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f45840683f0 0x7f458406a8a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f459c006b20 tx=0x7f459c008040 comp rx=0 tx=0).stop 2023-03-03T08:00:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.655+0000 7f45a719d700 1 -- 172.21.15.98:0/3462304867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45a00cf570 msgr2=0x7f45a00cbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.655+0000 7f45a719d700 1 --2- 172.21.15.98:0/3462304867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45a00cf570 0x7f45a00cbaa0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f459800cb50 tx=0x7f459800ce60 comp rx=0 tx=0).stop 2023-03-03T08:00:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.655+0000 7f45a719d700 1 -- 172.21.15.98:0/3462304867 shutdown_connections 2023-03-03T08:00:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.655+0000 7f45a719d700 1 --2- 172.21.15.98:0/3462304867 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f45840683f0 0x7f458406a8a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.655+0000 7f45a719d700 1 --2- 172.21.15.98:0/3462304867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45a00cf570 0x7f45a00cbaa0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.655+0000 7f45a719d700 1 --2- 172.21.15.98:0/3462304867 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f45a00b5e70 0x7f45a01e6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.655+0000 7f45a719d700 1 --2- 172.21.15.98:0/3462304867 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f45a00b8910 0x7f45a014e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.655+0000 7f45a719d700 1 -- 172.21.15.98:0/3462304867 >> 172.21.15.98:0/3462304867 conn(0x7f45a006b880 msgr2=0x7f45a015efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.655+0000 7f45a719d700 1 -- 172.21.15.98:0/3462304867 shutdown_connections 2023-03-03T08:00:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:42.655+0000 7f45a719d700 1 -- 172.21.15.98:0/3462304867 wait complete. 2023-03-03T08:00:42.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:42.686 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/_nogroup/subvolume_0000000001024199) 2023-03-03T08:00:42.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:42.801 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2023-03-03T08:00:42.873 INFO:teuthology.orchestra.run.smithi098.stdout:cephfs_data 2023-03-03T08:00:42.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:42.874 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/_nogroup/subvolume_0000000001024199) 2023-03-03T08:00:42.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000001024200 2023-03-03T08:00:43.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.228+0000 7ff2057ff700 1 Processor -- start 2023-03-03T08:00:43.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.228+0000 7ff2057ff700 1 -- start start 2023-03-03T08:00:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.230+0000 7ff2057ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff2000b8b30 0x7ff2000b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.230+0000 7ff2057ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2000b55d0 con 0x7ff2000b8760 2023-03-03T08:00:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.230+0000 7ff2057ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2000d1290 con 0x7ff2000b5170 2023-03-03T08:00:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.230+0000 7ff2057ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2000d07b0 con 0x7ff2000b8b30 2023-03-03T08:00:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.230+0000 7ff1feffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff2000b8760 0x7ff2000cd800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36624/0 (socket says 172.21.15.98:36624) 2023-03-03T08:00:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.230+0000 7ff1feffd700 1 -- 172.21.15.98:0/2180378263 learned_addr learned my addr 172.21.15.98:0/2180378263 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.230+0000 7ff1fe7fc700 1 --2- 172.21.15.98:0/2180378263 >> v2:172.21.15.100:3300/0 conn(0x7ff2000b8b30 0x7ff2000b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.231+0000 7ff1fdffb700 1 -- 172.21.15.98:0/2180378263 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663059620 0 0) 0x7ff2000b55d0 con 0x7ff2000b8760 2023-03-03T08:00:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.231+0000 7ff1fdffb700 1 -- 172.21.15.98:0/2180378263 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e80035b0 con 0x7ff2000b8760 2023-03-03T08:00:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.231+0000 7ff1fdffb700 1 -- 172.21.15.98:0/2180378263 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2017403995 0 0) 0x7ff2000d1290 con 0x7ff2000b5170 2023-03-03T08:00:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.231+0000 7ff1fdffb700 1 -- 172.21.15.98:0/2180378263 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2000b55d0 con 0x7ff2000b5170 2023-03-03T08:00:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.231+0000 7ff1fe7fc700 1 -- 172.21.15.98:0/2180378263 >> v1:172.21.15.100:6789/0 conn(0x7ff2000b5170 legacy=0x7ff2000d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.231+0000 7ff1fe7fc700 1 -- 172.21.15.98:0/2180378263 >> v1:172.21.15.98:6789/0 conn(0x7ff2000b8760 legacy=0x7ff2000cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.231+0000 7ff1fe7fc700 1 -- 172.21.15.98:0/2180378263 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2000b5a10 con 0x7ff2000b8b30 2023-03-03T08:00:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.231+0000 7ff1fe7fc700 1 --2- 172.21.15.98:0/2180378263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2000b8b30 0x7ff2000b4c30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009840 tx=0x7ff1f4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8b93b76cefc3222 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.232+0000 7ff1fdffb700 1 -- 172.21.15.98:0/2180378263 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4011830 con 0x7ff2000b8b30 2023-03-03T08:00:43.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.232+0000 7ff1fdffb700 1 -- 172.21.15.98:0/2180378263 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f4011990 con 0x7ff2000b8b30 2023-03-03T08:00:43.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.232+0000 7ff1fdffb700 1 -- 172.21.15.98:0/2180378263 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4011d20 con 0x7ff2000b8b30 2023-03-03T08:00:43.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.232+0000 7ff2057ff700 1 -- 172.21.15.98:0/2180378263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2000b8b30 msgr2=0x7ff2000b4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.232+0000 7ff2057ff700 1 --2- 172.21.15.98:0/2180378263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2000b8b30 0x7ff2000b4c30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009840 tx=0x7ff1f4009b50 comp rx=0 tx=0).stop 2023-03-03T08:00:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.232+0000 7ff2057ff700 1 -- 172.21.15.98:0/2180378263 shutdown_connections 2023-03-03T08:00:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.232+0000 7ff2057ff700 1 --2- 172.21.15.98:0/2180378263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2000b8b30 0x7ff2000b4c30 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.232+0000 7ff2057ff700 1 -- 172.21.15.98:0/2180378263 >> 172.21.15.98:0/2180378263 conn(0x7ff200073e70 msgr2=0x7ff20006adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.232+0000 7ff2057ff700 1 -- 172.21.15.98:0/2180378263 shutdown_connections 2023-03-03T08:00:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.232+0000 7ff2057ff700 1 -- 172.21.15.98:0/2180378263 wait complete. 2023-03-03T08:00:43.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.234+0000 7ff2057ff700 1 Processor -- start 2023-03-03T08:00:43.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.235+0000 7ff2057ff700 1 -- start start 2023-03-03T08:00:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff2057ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2000b5170 0x7ff200158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff2057ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2000b8760 0x7ff2001f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff2057ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2000b8b30 0x7ff2001f3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff2057ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2001f2e10 con 0x7ff2000b8760 2023-03-03T08:00:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff2057ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff200158a60 con 0x7ff2000b5170 2023-03-03T08:00:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff2057ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff200158d40 con 0x7ff2000b8b30 2023-03-03T08:00:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff1fe7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2000b8760 0x7ff2001f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff1fe7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2000b8760 0x7ff2001f28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51532/0 (socket says 172.21.15.98:51532) 2023-03-03T08:00:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff1fe7fc700 1 -- 172.21.15.98:0/3027847261 learned_addr learned my addr 172.21.15.98:0/3027847261 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff1feffd700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2000b5170 0x7ff200158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff1ff7fe700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2000b8b30 0x7ff2001f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff1fe7fc700 1 -- 172.21.15.98:0/3027847261 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2000b8b30 msgr2=0x7ff2001f3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff1fe7fc700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2000b8b30 0x7ff2001f3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff1fe7fc700 1 -- 172.21.15.98:0/3027847261 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2000b5170 msgr2=0x7ff200158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff1fe7fc700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2000b5170 0x7ff200158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff1fe7fc700 1 -- 172.21.15.98:0/3027847261 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f40094a0 con 0x7ff2000b8760 2023-03-03T08:00:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff1ff7fe700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2000b8b30 0x7ff2001f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.236+0000 7ff1feffd700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2000b5170 0x7ff200158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.237+0000 7ff1fe7fc700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2000b8760 0x7ff2001f28d0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009b90 tx=0x7ff1f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.237+0000 7ff1e77fe700 1 -- 172.21.15.98:0/3027847261 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f401a860 con 0x7ff2000b8760 2023-03-03T08:00:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.237+0000 7ff1e77fe700 1 -- 172.21.15.98:0/3027847261 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f401a9c0 con 0x7ff2000b8760 2023-03-03T08:00:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.237+0000 7ff1e77fe700 1 -- 172.21.15.98:0/3027847261 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f401ad70 con 0x7ff2000b8760 2023-03-03T08:00:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.237+0000 7ff2057ff700 1 -- 172.21.15.98:0/3027847261 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2000d01f0 con 0x7ff2000b8760 2023-03-03T08:00:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.237+0000 7ff2057ff700 1 -- 172.21.15.98:0/3027847261 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2000cff10 con 0x7ff2000b8760 2023-03-03T08:00:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.238+0000 7ff2057ff700 1 -- 172.21.15.98:0/3027847261 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2000d04d0 con 0x7ff2000b8760 2023-03-03T08:00:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.239+0000 7ff1e77fe700 1 -- 172.21.15.98:0/3027847261 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 86535+0+0 (secure 0 0 0) 0x7ff1f400f050 con 0x7ff2000b8760 2023-03-03T08:00:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.241+0000 7ff1e77fe700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff1ec0683c0 0x7ff1ec06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.241+0000 7ff1e77fe700 1 -- 172.21.15.98:0/3027847261 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff1f4086e50 con 0x7ff2000b8760 2023-03-03T08:00:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.241+0000 7ff1feffd700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff1ec0683c0 0x7ff1ec06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.242+0000 7ff1feffd700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff1ec0683c0 0x7ff1ec06a870 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0000f60 tx=0x7ff1f0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.245+0000 7ff1e77fe700 1 -- 172.21.15.98:0/3027847261 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff1f40534f0 con 0x7ff2000b8760 2023-03-03T08:00:43.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.472+0000 7ff2057ff700 1 -- 172.21.15.98:0/3027847261 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000001024200", "target": ["mon-mgr", ""]}) v1 -- 0x7ff200158f00 con 0x7ff1ec0683c0 2023-03-03T08:00:43.481 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:00:43.479+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:00:43.481 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:00:43.479+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:00:43.481 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:00:43.479+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:00:43.481 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:00:43.479+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:00:43.481 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:00:43.479+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:00:43.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.519+0000 7ff1e77fe700 1 -- 172.21.15.98:0/3027847261 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff1f00014f0 con 0x7ff1ec0683c0 2023-03-03T08:00:43.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.521+0000 7ff2057ff700 1 -- 172.21.15.98:0/3027847261 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff1ec0683c0 msgr2=0x7ff1ec06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:43.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.521+0000 7ff2057ff700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff1ec0683c0 0x7ff1ec06a870 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0000f60 tx=0x7ff1f0009040 comp rx=0 tx=0).stop 2023-03-03T08:00:43.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.521+0000 7ff2057ff700 1 -- 172.21.15.98:0/3027847261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2000b8760 msgr2=0x7ff2001f28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:43.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.521+0000 7ff2057ff700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2000b8760 0x7ff2001f28d0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009b90 tx=0x7ff1f4004750 comp rx=0 tx=0).stop 2023-03-03T08:00:43.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.521+0000 7ff2057ff700 1 -- 172.21.15.98:0/3027847261 shutdown_connections 2023-03-03T08:00:43.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.521+0000 7ff2057ff700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff1ec0683c0 0x7ff1ec06a870 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:43.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.521+0000 7ff2057ff700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2000b8b30 0x7ff2001f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:43.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.521+0000 7ff2057ff700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2000b5170 0x7ff200158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:43.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.521+0000 7ff2057ff700 1 --2- 172.21.15.98:0/3027847261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2000b8760 0x7ff2001f28d0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:43.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.521+0000 7ff2057ff700 1 -- 172.21.15.98:0/3027847261 >> 172.21.15.98:0/3027847261 conn(0x7ff200073e70 msgr2=0x7ff2000b7980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:43.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.522+0000 7ff2057ff700 1 -- 172.21.15.98:0/3027847261 shutdown_connections 2023-03-03T08:00:43.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:43.522+0000 7ff2057ff700 1 -- 172.21.15.98:0/3027847261 wait complete. 2023-03-03T08:00:43.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:43.551 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cp ./volumes/_nogroup/subvolume_0000000001024200/.meta ./volumes/_nogroup/subvolume_0000000001024199) 2023-03-03T08:00:45.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000001024199 2023-03-03T08:00:45.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.904+0000 7f08dbbff700 1 Processor -- start 2023-03-03T08:00:45.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.904+0000 7f08dbbff700 1 -- start start 2023-03-03T08:00:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.905+0000 7f08dbbff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08d415ca70 0x7f08d40cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.905+0000 7f08dbbff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f08d40d0270 0x7f08d40d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.905+0000 7f08dbbff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d40cf980 con 0x7f08d406b4b0 2023-03-03T08:00:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.906+0000 7f08dbbff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08d40cd5b0 con 0x7f08d415ca70 2023-03-03T08:00:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.906+0000 7f08dbbff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08d40d1800 con 0x7f08d40d0270 2023-03-03T08:00:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.906+0000 7f08d999b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08d415ca70 0x7f08d40cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.906+0000 7f08d999b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08d415ca70 0x7f08d40cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59658/0 (socket says 172.21.15.98:59658) 2023-03-03T08:00:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.906+0000 7f08d999b700 1 -- 172.21.15.98:0/3365927997 learned_addr learned my addr 172.21.15.98:0/3365927997 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.906+0000 7f08da19c700 1 --2- 172.21.15.98:0/3365927997 >> v2:172.21.15.100:3300/0 conn(0x7f08d40d0270 0x7f08d40d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.906+0000 7f08d999b700 1 -- 172.21.15.98:0/3365927997 >> v2:172.21.15.100:3300/0 conn(0x7f08d40d0270 msgr2=0x7f08d40d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.906+0000 7f08d999b700 1 --2- 172.21.15.98:0/3365927997 >> v2:172.21.15.100:3300/0 conn(0x7f08d40d0270 0x7f08d40d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.906+0000 7f08d999b700 1 -- 172.21.15.98:0/3365927997 >> v1:172.21.15.100:6789/0 conn(0x7f08d406b4b0 legacy=0x7f08d40cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.906+0000 7f08d999b700 1 -- 172.21.15.98:0/3365927997 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d40cd900 con 0x7f08d415ca70 2023-03-03T08:00:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.906+0000 7f08da19c700 1 --2- 172.21.15.98:0/3365927997 >> v2:172.21.15.100:3300/0 conn(0x7f08d40d0270 0x7f08d40d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.907+0000 7f08d999b700 1 --2- 172.21.15.98:0/3365927997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d415ca70 0x7f08d40cf440 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f08d40ca200 tx=0x7f08d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0656b3f633540fc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.907+0000 7f08d8999700 1 -- 172.21.15.98:0/3365927997 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d0004230 con 0x7f08d415ca70 2023-03-03T08:00:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.907+0000 7f08d8999700 1 -- 172.21.15.98:0/3365927997 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d0004390 con 0x7f08d415ca70 2023-03-03T08:00:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.907+0000 7f08d8999700 1 -- 172.21.15.98:0/3365927997 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d00036a0 con 0x7f08d415ca70 2023-03-03T08:00:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.907+0000 7f08dbbff700 1 -- 172.21.15.98:0/3365927997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d415ca70 msgr2=0x7f08d40cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.907+0000 7f08dbbff700 1 --2- 172.21.15.98:0/3365927997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d415ca70 0x7f08d40cf440 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f08d40ca200 tx=0x7f08d0009bd0 comp rx=0 tx=0).stop 2023-03-03T08:00:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.908+0000 7f08dbbff700 1 -- 172.21.15.98:0/3365927997 shutdown_connections 2023-03-03T08:00:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.908+0000 7f08dbbff700 1 --2- 172.21.15.98:0/3365927997 >> v2:172.21.15.100:3300/0 conn(0x7f08d40d0270 0x7f08d40d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.908+0000 7f08dbbff700 1 --2- 172.21.15.98:0/3365927997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d415ca70 0x7f08d40cf440 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.908+0000 7f08dbbff700 1 -- 172.21.15.98:0/3365927997 >> 172.21.15.98:0/3365927997 conn(0x7f08d4073e70 msgr2=0x7f08d406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:45.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.908+0000 7f08dbbff700 1 -- 172.21.15.98:0/3365927997 shutdown_connections 2023-03-03T08:00:45.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.908+0000 7f08dbbff700 1 -- 172.21.15.98:0/3365927997 wait complete. 2023-03-03T08:00:45.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.910+0000 7f08dbbff700 1 Processor -- start 2023-03-03T08:00:45.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.910+0000 7f08dbbff700 1 -- start start 2023-03-03T08:00:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.911+0000 7f08dbbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08d406b4b0 0x7f08d4153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.911+0000 7f08dbbff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08d40d0270 0x7f08d40ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.911+0000 7f08dbbff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d415ca70 0x7f08d40cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.911+0000 7f08dbbff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d4153cd0 con 0x7f08d415ca70 2023-03-03T08:00:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.911+0000 7f08dbbff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d4153ed0 con 0x7f08d40d0270 2023-03-03T08:00:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.911+0000 7f08dbbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d40cafd0 con 0x7f08d406b4b0 2023-03-03T08:00:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.911+0000 7f08da19c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d415ca70 0x7f08d40cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.911+0000 7f08da19c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d415ca70 0x7f08d40cc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59662/0 (socket says 172.21.15.98:59662) 2023-03-03T08:00:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.911+0000 7f08da19c700 1 -- 172.21.15.98:0/2988549630 learned_addr learned my addr 172.21.15.98:0/2988549630 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.911+0000 7f08d919a700 1 --2- 172.21.15.98:0/2988549630 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08d40d0270 0x7f08d40ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.911+0000 7f08d999b700 1 --2- 172.21.15.98:0/2988549630 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08d406b4b0 0x7f08d4153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.912+0000 7f08da19c700 1 -- 172.21.15.98:0/2988549630 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08d406b4b0 msgr2=0x7f08d4153790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.912+0000 7f08da19c700 1 --2- 172.21.15.98:0/2988549630 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08d406b4b0 0x7f08d4153790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.912+0000 7f08da19c700 1 -- 172.21.15.98:0/2988549630 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08d40d0270 msgr2=0x7f08d40ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:45.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.912+0000 7f08da19c700 1 --2- 172.21.15.98:0/2988549630 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08d40d0270 0x7f08d40ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:45.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.912+0000 7f08da19c700 1 -- 172.21.15.98:0/2988549630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d0009550 con 0x7f08d415ca70 2023-03-03T08:00:45.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.912+0000 7f08da19c700 1 --2- 172.21.15.98:0/2988549630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d415ca70 0x7f08d40cc8d0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f08cc00cd10 tx=0x7f08cc00a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:45.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.912+0000 7f08ca7fc700 1 -- 172.21.15.98:0/2988549630 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc00ce60 con 0x7f08d415ca70 2023-03-03T08:00:45.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.912+0000 7f08ca7fc700 1 -- 172.21.15.98:0/2988549630 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08cc003a20 con 0x7f08d415ca70 2023-03-03T08:00:45.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.912+0000 7f08dbbff700 1 -- 172.21.15.98:0/2988549630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d40d1aa0 con 0x7f08d415ca70 2023-03-03T08:00:45.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.912+0000 7f08ca7fc700 1 -- 172.21.15.98:0/2988549630 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc003db0 con 0x7f08d415ca70 2023-03-03T08:00:45.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.913+0000 7f08dbbff700 1 -- 172.21.15.98:0/2988549630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d40cdbe0 con 0x7f08d415ca70 2023-03-03T08:00:45.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.914+0000 7f08dbbff700 1 -- 172.21.15.98:0/2988549630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d415ce40 con 0x7f08d415ca70 2023-03-03T08:00:45.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.915+0000 7f08ca7fc700 1 -- 172.21.15.98:0/2988549630 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 10) v1 ==== 86575+0+0 (secure 0 0 0) 0x7f08cc01f070 con 0x7f08d415ca70 2023-03-03T08:00:45.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.916+0000 7f08ca7fc700 1 --2- 172.21.15.98:0/2988549630 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f08b8068410 0x7f08b806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:45.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.916+0000 7f08ca7fc700 1 -- 172.21.15.98:0/2988549630 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f08cc0863e0 con 0x7f08d415ca70 2023-03-03T08:00:45.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.917+0000 7f08d999b700 1 --2- 172.21.15.98:0/2988549630 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f08b8068410 0x7f08b806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:45.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.917+0000 7f08d999b700 1 --2- 172.21.15.98:0/2988549630 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f08b8068410 0x7f08b806a8c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009130 tx=0x7f08d0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:45.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:45.921+0000 7f08ca7fc700 1 -- 172.21.15.98:0/2988549630 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f08cc0529d0 con 0x7f08d415ca70 2023-03-03T08:00:46.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.143+0000 7f08dbbff700 1 -- 172.21.15.98:0/2988549630 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000001024199", "target": ["mon-mgr", ""]}) v1 -- 0x7f08d40cb490 con 0x7f08b8068410 2023-03-03T08:00:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.184+0000 7f08ca7fc700 1 -- 172.21.15.98:0/2988549630 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f08d40cb490 con 0x7f08b8068410 2023-03-03T08:00:46.187 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/_nogroup/subvolume_0000000001024199 2023-03-03T08:00:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.187+0000 7f08dbbff700 1 -- 172.21.15.98:0/2988549630 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f08b8068410 msgr2=0x7f08b806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.187+0000 7f08dbbff700 1 --2- 172.21.15.98:0/2988549630 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f08b8068410 0x7f08b806a8c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009130 tx=0x7f08d0008030 comp rx=0 tx=0).stop 2023-03-03T08:00:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.187+0000 7f08dbbff700 1 -- 172.21.15.98:0/2988549630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d415ca70 msgr2=0x7f08d40cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.187+0000 7f08dbbff700 1 --2- 172.21.15.98:0/2988549630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d415ca70 0x7f08d40cc8d0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f08cc00cd10 tx=0x7f08cc00a560 comp rx=0 tx=0).stop 2023-03-03T08:00:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.187+0000 7f08dbbff700 1 -- 172.21.15.98:0/2988549630 shutdown_connections 2023-03-03T08:00:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.187+0000 7f08dbbff700 1 --2- 172.21.15.98:0/2988549630 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f08b8068410 0x7f08b806a8c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.187+0000 7f08dbbff700 1 --2- 172.21.15.98:0/2988549630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08d415ca70 0x7f08d40cc8d0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.187+0000 7f08dbbff700 1 --2- 172.21.15.98:0/2988549630 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08d406b4b0 0x7f08d4153790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.187+0000 7f08dbbff700 1 --2- 172.21.15.98:0/2988549630 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08d40d0270 0x7f08d40ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.187+0000 7f08dbbff700 1 -- 172.21.15.98:0/2988549630 >> 172.21.15.98:0/2988549630 conn(0x7f08d4073e70 msgr2=0x7f08d40d2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.187+0000 7f08dbbff700 1 -- 172.21.15.98:0/2988549630 shutdown_connections 2023-03-03T08:00:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.187+0000 7f08dbbff700 1 -- 172.21.15.98:0/2988549630 wait complete. 2023-03-03T08:00:46.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000001024199 2023-03-03T08:00:46.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.467+0000 7fc26dbff700 1 Processor -- start 2023-03-03T08:00:46.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.467+0000 7fc26dbff700 1 -- start start 2023-03-03T08:00:46.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.469+0000 7fc26dbff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc26815ca70 0x7fc2680cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:46.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.469+0000 7fc26dbff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2680cf980 con 0x7fc26806b4b0 2023-03-03T08:00:46.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.469+0000 7fc26dbff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2680d1800 con 0x7fc2680d0270 2023-03-03T08:00:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.469+0000 7fc26dbff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2680cd5b0 con 0x7fc26815ca70 2023-03-03T08:00:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.469+0000 7fc266ffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc26806b4b0 0x7fc2680cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37670/0 (socket says 172.21.15.98:37670) 2023-03-03T08:00:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.469+0000 7fc266ffd700 1 -- 172.21.15.98:0/3939925422 learned_addr learned my addr 172.21.15.98:0/3939925422 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.470+0000 7fc2677fe700 1 --2- 172.21.15.98:0/3939925422 >> v2:172.21.15.100:3300/0 conn(0x7fc26815ca70 0x7fc2680cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.470+0000 7fc2667fc700 1 -- 172.21.15.98:0/3939925422 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134815602 0 0) 0x7fc2680cf980 con 0x7fc26806b4b0 2023-03-03T08:00:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.470+0000 7fc2667fc700 1 -- 172.21.15.98:0/3939925422 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2500035b0 con 0x7fc26806b4b0 2023-03-03T08:00:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.470+0000 7fc2667fc700 1 -- 172.21.15.98:0/3939925422 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219345374 0 0) 0x7fc2680d1800 con 0x7fc2680d0270 2023-03-03T08:00:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.470+0000 7fc2667fc700 1 -- 172.21.15.98:0/3939925422 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2680cf980 con 0x7fc2680d0270 2023-03-03T08:00:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.470+0000 7fc2667fc700 1 -- 172.21.15.98:0/3939925422 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1674792605 0 0) 0x7fc2500035b0 con 0x7fc26806b4b0 2023-03-03T08:00:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.470+0000 7fc2677fe700 1 -- 172.21.15.98:0/3939925422 >> v1:172.21.15.100:6789/0 conn(0x7fc2680d0270 legacy=0x7fc2680d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.470+0000 7fc2677fe700 1 -- 172.21.15.98:0/3939925422 >> v1:172.21.15.98:6789/0 conn(0x7fc26806b4b0 legacy=0x7fc2680cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.470+0000 7fc2677fe700 1 -- 172.21.15.98:0/3939925422 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2680cd900 con 0x7fc26815ca70 2023-03-03T08:00:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.471+0000 7fc2677fe700 1 --2- 172.21.15.98:0/3939925422 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc26815ca70 0x7fc2680cf440 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc258009840 tx=0x7fc258009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d91ef5041a574e5b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.471+0000 7fc2667fc700 1 -- 172.21.15.98:0/3939925422 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c001670 con 0x7fc26815ca70 2023-03-03T08:00:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.471+0000 7fc2667fc700 1 -- 172.21.15.98:0/3939925422 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc258004030 con 0x7fc26815ca70 2023-03-03T08:00:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.471+0000 7fc2667fc700 1 -- 172.21.15.98:0/3939925422 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2580043c0 con 0x7fc26815ca70 2023-03-03T08:00:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.471+0000 7fc26dbff700 1 -- 172.21.15.98:0/3939925422 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc26815ca70 msgr2=0x7fc2680cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.471+0000 7fc26dbff700 1 --2- 172.21.15.98:0/3939925422 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc26815ca70 0x7fc2680cf440 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc258009840 tx=0x7fc258009b50 comp rx=0 tx=0).stop 2023-03-03T08:00:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.472+0000 7fc26dbff700 1 -- 172.21.15.98:0/3939925422 shutdown_connections 2023-03-03T08:00:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.472+0000 7fc26dbff700 1 --2- 172.21.15.98:0/3939925422 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc26815ca70 0x7fc2680cf440 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.472+0000 7fc26dbff700 1 -- 172.21.15.98:0/3939925422 >> 172.21.15.98:0/3939925422 conn(0x7fc268073e70 msgr2=0x7fc26806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.472+0000 7fc26dbff700 1 -- 172.21.15.98:0/3939925422 shutdown_connections 2023-03-03T08:00:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.472+0000 7fc26dbff700 1 -- 172.21.15.98:0/3939925422 wait complete. 2023-03-03T08:00:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.474+0000 7fc26dbff700 1 Processor -- start 2023-03-03T08:00:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.475+0000 7fc26dbff700 1 -- start start 2023-03-03T08:00:46.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.476+0000 7fc26dbff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc26806b4b0 0x7fc26815e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:46.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.476+0000 7fc26dbff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2680d0270 0x7fc268156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:46.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.476+0000 7fc26dbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc26815ca70 0x7fc2681e62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:46.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.476+0000 7fc26dbff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2680b3e90 con 0x7fc26806b4b0 2023-03-03T08:00:46.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.476+0000 7fc26dbff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26815e280 con 0x7fc2680d0270 2023-03-03T08:00:46.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.476+0000 7fc26dbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc268156c90 con 0x7fc26815ca70 2023-03-03T08:00:46.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.477+0000 7fc2677fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc26806b4b0 0x7fc26815e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:46.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.477+0000 7fc2677fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc26806b4b0 0x7fc26815e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59676/0 (socket says 172.21.15.98:59676) 2023-03-03T08:00:46.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.477+0000 7fc2677fe700 1 -- 172.21.15.98:0/3919215853 learned_addr learned my addr 172.21.15.98:0/3919215853 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.477+0000 7fc266ffd700 1 --2- 172.21.15.98:0/3919215853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2680d0270 0x7fc268156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.477+0000 7fc267fff700 1 --2- 172.21.15.98:0/3919215853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc26815ca70 0x7fc2681e62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.477+0000 7fc2677fe700 1 -- 172.21.15.98:0/3919215853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc26815ca70 msgr2=0x7fc2681e62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.477+0000 7fc2677fe700 1 --2- 172.21.15.98:0/3919215853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc26815ca70 0x7fc2681e62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.477+0000 7fc2677fe700 1 -- 172.21.15.98:0/3919215853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2680d0270 msgr2=0x7fc268156750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.477+0000 7fc2677fe700 1 --2- 172.21.15.98:0/3919215853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2680d0270 0x7fc268156750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.477+0000 7fc2677fe700 1 -- 172.21.15.98:0/3919215853 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2580094a0 con 0x7fc26806b4b0 2023-03-03T08:00:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.477+0000 7fc267fff700 1 --2- 172.21.15.98:0/3919215853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc26815ca70 0x7fc2681e62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:46.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.477+0000 7fc2677fe700 1 --2- 172.21.15.98:0/3919215853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc26806b4b0 0x7fc26815e710 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc25800a010 tx=0x7fc258004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:46.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.478+0000 7fc24ffff700 1 -- 172.21.15.98:0/3919215853 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc258003d00 con 0x7fc26806b4b0 2023-03-03T08:00:46.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.478+0000 7fc24ffff700 1 -- 172.21.15.98:0/3919215853 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25801f070 con 0x7fc26806b4b0 2023-03-03T08:00:46.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.478+0000 7fc26dbff700 1 -- 172.21.15.98:0/3919215853 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2680d1aa0 con 0x7fc26806b4b0 2023-03-03T08:00:46.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.478+0000 7fc24ffff700 1 -- 172.21.15.98:0/3919215853 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc258011870 con 0x7fc26806b4b0 2023-03-03T08:00:46.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.478+0000 7fc26dbff700 1 -- 172.21.15.98:0/3919215853 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2680cdbe0 con 0x7fc26806b4b0 2023-03-03T08:00:46.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.479+0000 7fc26dbff700 1 -- 172.21.15.98:0/3919215853 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26815ce40 con 0x7fc26806b4b0 2023-03-03T08:00:46.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.480+0000 7fc24ffff700 1 -- 172.21.15.98:0/3919215853 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 10) v1 ==== 86575+0+0 (secure 0 0 0) 0x7fc25802a020 con 0x7fc26806b4b0 2023-03-03T08:00:46.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.481+0000 7fc24ffff700 1 --2- 172.21.15.98:0/3919215853 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc254068410 0x7fc25406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:46.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.482+0000 7fc24ffff700 1 -- 172.21.15.98:0/3919215853 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc258087320 con 0x7fc26806b4b0 2023-03-03T08:00:46.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.482+0000 7fc266ffd700 1 --2- 172.21.15.98:0/3919215853 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc254068410 0x7fc25406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:46.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.483+0000 7fc266ffd700 1 --2- 172.21.15.98:0/3919215853 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc254068410 0x7fc25406a8c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc25c001a80 tx=0x7fc25c001a10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:46.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.486+0000 7fc24ffff700 1 -- 172.21.15.98:0/3919215853 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc258053970 con 0x7fc26806b4b0 2023-03-03T08:00:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.713+0000 7fc26dbff700 1 -- 172.21.15.98:0/3919215853 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000001024199", "target": ["mon-mgr", ""]}) v1 -- 0x7fc26815e560 con 0x7fc254068410 2023-03-03T08:00:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.718+0000 7fc24ffff700 1 -- 172.21.15.98:0/3919215853 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7fc26815e560 con 0x7fc254068410 2023-03-03T08:00:46.721 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/_nogroup/subvolume_0000000001024199 2023-03-03T08:00:46.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.720+0000 7fc26dbff700 1 -- 172.21.15.98:0/3919215853 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc254068410 msgr2=0x7fc25406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:46.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.720+0000 7fc26dbff700 1 --2- 172.21.15.98:0/3919215853 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc254068410 0x7fc25406a8c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc25c001a80 tx=0x7fc25c001a10 comp rx=0 tx=0).stop 2023-03-03T08:00:46.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.720+0000 7fc26dbff700 1 -- 172.21.15.98:0/3919215853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc26806b4b0 msgr2=0x7fc26815e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:46.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.720+0000 7fc26dbff700 1 --2- 172.21.15.98:0/3919215853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc26806b4b0 0x7fc26815e710 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc25800a010 tx=0x7fc258004750 comp rx=0 tx=0).stop 2023-03-03T08:00:46.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.721+0000 7fc26dbff700 1 -- 172.21.15.98:0/3919215853 shutdown_connections 2023-03-03T08:00:46.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.721+0000 7fc26dbff700 1 --2- 172.21.15.98:0/3919215853 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc254068410 0x7fc25406a8c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:46.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.721+0000 7fc26dbff700 1 --2- 172.21.15.98:0/3919215853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc26806b4b0 0x7fc26815e710 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:46.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.721+0000 7fc26dbff700 1 --2- 172.21.15.98:0/3919215853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc26815ca70 0x7fc2681e62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.721+0000 7fc26dbff700 1 --2- 172.21.15.98:0/3919215853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2680d0270 0x7fc268156750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.721+0000 7fc26dbff700 1 -- 172.21.15.98:0/3919215853 >> 172.21.15.98:0/3919215853 conn(0x7fc268073e70 msgr2=0x7fc268158560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.721+0000 7fc26dbff700 1 -- 172.21.15.98:0/3919215853 shutdown_connections 2023-03-03T08:00:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:46.721+0000 7fc26dbff700 1 -- 172.21.15.98:0/3919215853 wait complete. 2023-03-03T08:00:46.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:46.751 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/9faf9fcf89fe0288f65ec8d2c7099476.meta) 2023-03-03T08:00:50.635 INFO:teuthology.orchestra.run.smithi098.stdout:[GLOBAL] 2023-03-03T08:00:50.636 INFO:teuthology.orchestra.run.smithi098.stdout:version = 1 2023-03-03T08:00:50.636 INFO:teuthology.orchestra.run.smithi098.stdout:type = subvolume 2023-03-03T08:00:50.636 INFO:teuthology.orchestra.run.smithi098.stdout:path = /volumes/_nogroup/subvolume_0000000001024199 2023-03-03T08:00:50.636 INFO:teuthology.orchestra.run.smithi098.stdout:state = complete 2023-03-03T08:00:50.636 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:50.642 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000001024199 alice 2023-03-03T08:00:50.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.897+0000 7ff51ab9d700 1 Processor -- start 2023-03-03T08:00:50.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.898+0000 7ff51ab9d700 1 -- start start 2023-03-03T08:00:50.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.899+0000 7ff51ab9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff5140d0110 0x7ff5140cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:50.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.899+0000 7ff51ab9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5140b3c80 con 0x7ff5140b3550 2023-03-03T08:00:50.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.899+0000 7ff51ab9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5140b4370 con 0x7ff514073ed0 2023-03-03T08:00:50.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.899+0000 7ff51ab9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5140b3ed0 con 0x7ff5140d0110 2023-03-03T08:00:50.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.899+0000 7ff519b9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff5140b3550 0x7ff5140cfa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37684/0 (socket says 172.21.15.98:37684) 2023-03-03T08:00:50.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.899+0000 7ff519b9b700 1 -- 172.21.15.98:0/3490293855 learned_addr learned my addr 172.21.15.98:0/3490293855 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:50.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.900+0000 7ff51939a700 1 --2- 172.21.15.98:0/3490293855 >> v2:172.21.15.100:3300/0 conn(0x7ff5140d0110 0x7ff5140cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:50.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.900+0000 7ff518b99700 1 -- 172.21.15.98:0/3490293855 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522199779 0 0) 0x7ff5140b3c80 con 0x7ff5140b3550 2023-03-03T08:00:50.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.900+0000 7ff518b99700 1 -- 172.21.15.98:0/3490293855 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5000035b0 con 0x7ff5140b3550 2023-03-03T08:00:50.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.900+0000 7ff518b99700 1 -- 172.21.15.98:0/3490293855 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3885236341 0 0) 0x7ff5000035b0 con 0x7ff5140b3550 2023-03-03T08:00:50.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.900+0000 7ff518b99700 1 -- 172.21.15.98:0/3490293855 >> v2:172.21.15.100:3300/0 conn(0x7ff5140d0110 msgr2=0x7ff5140cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:50.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.900+0000 7ff518b99700 1 --2- 172.21.15.98:0/3490293855 >> v2:172.21.15.100:3300/0 conn(0x7ff5140d0110 0x7ff5140cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:50.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.900+0000 7ff518b99700 1 -- 172.21.15.98:0/3490293855 >> v1:172.21.15.100:6789/0 conn(0x7ff514073ed0 legacy=0x7ff5140ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:50.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.900+0000 7ff518b99700 1 -- 172.21.15.98:0/3490293855 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5140cd640 con 0x7ff5140b3550 2023-03-03T08:00:50.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.900+0000 7ff518b99700 1 -- 172.21.15.98:0/3490293855 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7ff510001670 con 0x7ff5140b3550 2023-03-03T08:00:50.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.900+0000 7ff518b99700 1 -- 172.21.15.98:0/3490293855 >> v1:172.21.15.98:6789/0 conn(0x7ff5140b3550 legacy=0x7ff5140cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:50.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.900+0000 7ff518b99700 1 --2- 172.21.15.98:0/3490293855 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff500003a50 0x7ff500003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:50.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.901+0000 7ff518b99700 1 --2- 172.21.15.98:0/3490293855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff500004850 0x7ff500004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:50.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.901+0000 7ff518b99700 1 --2- 172.21.15.98:0/3490293855 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff500005250 0x7ff50000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:50.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.901+0000 7ff518b99700 1 -- 172.21.15.98:0/3490293855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5140b3ed0 con 0x7ff500004850 2023-03-03T08:00:50.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.901+0000 7ff518b99700 1 -- 172.21.15.98:0/3490293855 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50000c160 con 0x7ff500005250 2023-03-03T08:00:50.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.901+0000 7ff518b99700 1 -- 172.21.15.98:0/3490293855 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50000c320 con 0x7ff500003a50 2023-03-03T08:00:50.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.901+0000 7ff51939a700 1 --2- 172.21.15.98:0/3490293855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff500004850 0x7ff500004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:50.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.901+0000 7ff519b9b700 1 --2- 172.21.15.98:0/3490293855 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff500003a50 0x7ff500003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:50.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.901+0000 7ff51939a700 1 -- 172.21.15.98:0/3490293855 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff500003a50 msgr2=0x7ff500003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:50.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.901+0000 7ff51939a700 1 --2- 172.21.15.98:0/3490293855 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff500003a50 0x7ff500003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:50.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.901+0000 7ff51939a700 1 -- 172.21.15.98:0/3490293855 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff500005250 msgr2=0x7ff50000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:50.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.901+0000 7ff51a39c700 1 --2- 172.21.15.98:0/3490293855 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff500005250 0x7ff50000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:50.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.901+0000 7ff51939a700 1 --2- 172.21.15.98:0/3490293855 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff500005250 0x7ff50000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:50.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.901+0000 7ff51939a700 1 -- 172.21.15.98:0/3490293855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5000035b0 con 0x7ff500004850 2023-03-03T08:00:50.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.901+0000 7ff51a39c700 1 --2- 172.21.15.98:0/3490293855 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff500005250 0x7ff50000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:50.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.901+0000 7ff51a39c700 1 -- 172.21.15.98:0/3490293855 reap_dead start 2023-03-03T08:00:50.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.902+0000 7ff51939a700 1 --2- 172.21.15.98:0/3490293855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff500004850 0x7ff500004cc0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7ff504009b60 tx=0x7ff500005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd904ab9a20c288 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:50.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.902+0000 7ff518b99700 1 -- 172.21.15.98:0/3490293855 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50400e6c0 con 0x7ff500004850 2023-03-03T08:00:50.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.902+0000 7ff518b99700 1 -- 172.21.15.98:0/3490293855 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50400e820 con 0x7ff500004850 2023-03-03T08:00:50.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.902+0000 7ff518b99700 1 -- 172.21.15.98:0/3490293855 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50400eb90 con 0x7ff500004850 2023-03-03T08:00:50.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.902+0000 7ff51ab9d700 1 -- 172.21.15.98:0/3490293855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff500004850 msgr2=0x7ff500004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:50.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.902+0000 7ff51ab9d700 1 --2- 172.21.15.98:0/3490293855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff500004850 0x7ff500004cc0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7ff504009b60 tx=0x7ff500005880 comp rx=0 tx=0).stop 2023-03-03T08:00:50.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.903+0000 7ff51ab9d700 1 -- 172.21.15.98:0/3490293855 shutdown_connections 2023-03-03T08:00:50.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.903+0000 7ff51ab9d700 1 --2- 172.21.15.98:0/3490293855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff500004850 0x7ff500004cc0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:50.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.903+0000 7ff51ab9d700 1 -- 172.21.15.98:0/3490293855 >> 172.21.15.98:0/3490293855 conn(0x7ff514073b00 msgr2=0x7ff5140d2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:50.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.903+0000 7ff51ab9d700 1 -- 172.21.15.98:0/3490293855 shutdown_connections 2023-03-03T08:00:50.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.903+0000 7ff51ab9d700 1 -- 172.21.15.98:0/3490293855 wait complete. 2023-03-03T08:00:50.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.905+0000 7ff51ab9d700 1 Processor -- start 2023-03-03T08:00:50.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.905+0000 7ff51ab9d700 1 -- start start 2023-03-03T08:00:50.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.906+0000 7ff51ab9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff500004850 0x7ff5140cbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:50.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.906+0000 7ff51ab9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5140cc5f0 0x7ff514155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:50.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.906+0000 7ff51ab9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5140b79d0 0x7ff5140b7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:50.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.906+0000 7ff51ab9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5140b8380 con 0x7ff5140b79d0 2023-03-03T08:00:50.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.906+0000 7ff51ab9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5140b1f90 con 0x7ff5140cc5f0 2023-03-03T08:00:50.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.906+0000 7ff51ab9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5140b1380 con 0x7ff500004850 2023-03-03T08:00:50.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.906+0000 7ff51a39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5140b79d0 0x7ff5140b7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:50.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.906+0000 7ff51a39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5140b79d0 0x7ff5140b7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59696/0 (socket says 172.21.15.98:59696) 2023-03-03T08:00:50.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.906+0000 7ff51a39c700 1 -- 172.21.15.98:0/3807347701 learned_addr learned my addr 172.21.15.98:0/3807347701 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:50.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.907+0000 7ff51939a700 1 --2- 172.21.15.98:0/3807347701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5140cc5f0 0x7ff514155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:50.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.907+0000 7ff519b9b700 1 --2- 172.21.15.98:0/3807347701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff500004850 0x7ff5140cbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:50.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.907+0000 7ff51a39c700 1 -- 172.21.15.98:0/3807347701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff500004850 msgr2=0x7ff5140cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:50.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.907+0000 7ff51a39c700 1 --2- 172.21.15.98:0/3807347701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff500004850 0x7ff5140cbaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:50.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.907+0000 7ff51a39c700 1 -- 172.21.15.98:0/3807347701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5140cc5f0 msgr2=0x7ff514155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:50.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.907+0000 7ff51a39c700 1 --2- 172.21.15.98:0/3807347701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5140cc5f0 0x7ff514155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:50.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.907+0000 7ff51a39c700 1 -- 172.21.15.98:0/3807347701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5140d3d60 con 0x7ff5140b79d0 2023-03-03T08:00:50.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.907+0000 7ff51a39c700 1 --2- 172.21.15.98:0/3807347701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5140b79d0 0x7ff5140b7e40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ff5140d04e0 tx=0x7ff50c00de70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:50.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.907+0000 7ff50a7fc700 1 -- 172.21.15.98:0/3807347701 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c003e10 con 0x7ff5140b79d0 2023-03-03T08:00:50.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.907+0000 7ff50a7fc700 1 -- 172.21.15.98:0/3807347701 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50c005070 con 0x7ff5140b79d0 2023-03-03T08:00:50.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.907+0000 7ff51ab9d700 1 -- 172.21.15.98:0/3807347701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5140cd920 con 0x7ff5140b79d0 2023-03-03T08:00:50.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.907+0000 7ff50a7fc700 1 -- 172.21.15.98:0/3807347701 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c005420 con 0x7ff5140b79d0 2023-03-03T08:00:50.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.908+0000 7ff51ab9d700 1 -- 172.21.15.98:0/3807347701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5140b1940 con 0x7ff5140b79d0 2023-03-03T08:00:50.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.909+0000 7ff51ab9d700 1 -- 172.21.15.98:0/3807347701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5140d3ef0 con 0x7ff5140b79d0 2023-03-03T08:00:50.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.910+0000 7ff50a7fc700 1 -- 172.21.15.98:0/3807347701 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 10) v1 ==== 86575+0+0 (secure 0 0 0) 0x7ff50c01a030 con 0x7ff5140b79d0 2023-03-03T08:00:50.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.912+0000 7ff50a7fc700 1 --2- 172.21.15.98:0/3807347701 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4f8068410 0x7ff4f806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:50.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.912+0000 7ff50a7fc700 1 -- 172.21.15.98:0/3807347701 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff50c085860 con 0x7ff5140b79d0 2023-03-03T08:00:50.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.912+0000 7ff519b9b700 1 --2- 172.21.15.98:0/3807347701 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4f8068410 0x7ff4f806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:50.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.913+0000 7ff519b9b700 1 --2- 172.21.15.98:0/3807347701 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4f8068410 0x7ff4f806a8c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff510001eb0 tx=0x7ff510000ed0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:50.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:50.917+0000 7ff50a7fc700 1 -- 172.21.15.98:0/3807347701 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff5140d3ef0 con 0x7ff5140b79d0 2023-03-03T08:00:51.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.143+0000 7ff51ab9d700 1 -- 172.21.15.98:0/3807347701 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000001024199", "auth_id": "alice", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5140d3ef0 con 0x7ff4f8068410 2023-03-03T08:00:51.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.186+0000 7ff50a7fc700 1 -- 172.21.15.98:0/3807347701 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff5140d3ef0 con 0x7ff4f8068410 2023-03-03T08:00:51.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.188+0000 7ff51ab9d700 1 -- 172.21.15.98:0/3807347701 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4f8068410 msgr2=0x7ff4f806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:51.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.188+0000 7ff51ab9d700 1 --2- 172.21.15.98:0/3807347701 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4f8068410 0x7ff4f806a8c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff510001eb0 tx=0x7ff510000ed0 comp rx=0 tx=0).stop 2023-03-03T08:00:51.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.188+0000 7ff51ab9d700 1 -- 172.21.15.98:0/3807347701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5140b79d0 msgr2=0x7ff5140b7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:51.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.188+0000 7ff51ab9d700 1 --2- 172.21.15.98:0/3807347701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5140b79d0 0x7ff5140b7e40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ff5140d04e0 tx=0x7ff50c00de70 comp rx=0 tx=0).stop 2023-03-03T08:00:51.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.188+0000 7ff51ab9d700 1 -- 172.21.15.98:0/3807347701 shutdown_connections 2023-03-03T08:00:51.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.188+0000 7ff51ab9d700 1 --2- 172.21.15.98:0/3807347701 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff4f8068410 0x7ff4f806a8c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:51.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.188+0000 7ff51ab9d700 1 --2- 172.21.15.98:0/3807347701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5140b79d0 0x7ff5140b7e40 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:51.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.188+0000 7ff51ab9d700 1 --2- 172.21.15.98:0/3807347701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff500004850 0x7ff5140cbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:51.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.188+0000 7ff51ab9d700 1 --2- 172.21.15.98:0/3807347701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5140cc5f0 0x7ff514155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:51.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.188+0000 7ff51ab9d700 1 -- 172.21.15.98:0/3807347701 >> 172.21.15.98:0/3807347701 conn(0x7ff514073b00 msgr2=0x7ff51406c030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:51.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.189+0000 7ff51ab9d700 1 -- 172.21.15.98:0/3807347701 shutdown_connections 2023-03-03T08:00:51.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.189+0000 7ff51ab9d700 1 -- 172.21.15.98:0/3807347701 wait complete. 2023-03-03T08:00:51.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth get client.alice --format=json-pretty 2023-03-03T08:00:51.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.463+0000 7fd9049fe700 1 Processor -- start 2023-03-03T08:00:51.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.463+0000 7fd9049fe700 1 -- start start 2023-03-03T08:00:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.465+0000 7fd9049fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9000b51b0 0x7fd9000d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.465+0000 7fd9049fe700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90015ce00 con 0x7fd9000b5e70 2023-03-03T08:00:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.465+0000 7fd9049fe700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9000d0bf0 con 0x7fd9000b8910 2023-03-03T08:00:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.465+0000 7fd9049fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000b8ce0 con 0x7fd9000b51b0 2023-03-03T08:00:51.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.465+0000 7fd8fe59c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd9000b5e70 0x7fd9000b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37688/0 (socket says 172.21.15.98:37688) 2023-03-03T08:00:51.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.465+0000 7fd8fe59c700 1 -- 172.21.15.98:0/2130231127 learned_addr learned my addr 172.21.15.98:0/2130231127 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:51.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.465+0000 7fd8ff59e700 1 --2- 172.21.15.98:0/2130231127 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9000b51b0 0x7fd9000d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:51.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437606207 0 0) 0x7fd8f4001110 con 0x7fd9000b5e70 2023-03-03T08:00:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e80035b0 con 0x7fd9000b5e70 2023-03-03T08:00:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127743937 0 0) 0x7fd9000d0bf0 con 0x7fd9000b8910 2023-03-03T08:00:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8f4001110 con 0x7fd9000b8910 2023-03-03T08:00:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 67707908 0 0) 0x7fd8e80035b0 con 0x7fd9000b5e70 2023-03-03T08:00:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9000b51b0 msgr2=0x7fd9000d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 --2- 172.21.15.98:0/2130231127 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9000b51b0 0x7fd9000d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 >> v1:172.21.15.100:6789/0 conn(0x7fd9000b8910 legacy=0x7fd9001604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:51.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd900160cf0 con 0x7fd9000b5e70 2023-03-03T08:00:51.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7fd8f4001800 con 0x7fd9000b5e70 2023-03-03T08:00:51.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8ff59e700 1 --2- 172.21.15.98:0/2130231127 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9000b51b0 0x7fd9000d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:00:51.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 >> v1:172.21.15.98:6789/0 conn(0x7fd9000b5e70 legacy=0x7fd9000b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:51.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 --2- 172.21.15.98:0/2130231127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e8003bc0 0x7fd8e8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:51.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 --2- 172.21.15.98:0/2130231127 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8e8004850 0x7fd8e8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:51.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 --2- 172.21.15.98:0/2130231127 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9000b51b0 0x7fd8e8005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:51.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000b8ce0 con 0x7fd8e8003bc0 2023-03-03T08:00:51.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e800bba0 con 0x7fd8e8004850 2023-03-03T08:00:51.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.466+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e800bd30 con 0x7fd9000b51b0 2023-03-03T08:00:51.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.467+0000 7fd8fed9d700 1 --2- 172.21.15.98:0/2130231127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e8003bc0 0x7fd8e8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:51.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.467+0000 7fd8ff59e700 1 --2- 172.21.15.98:0/2130231127 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9000b51b0 0x7fd8e8005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:51.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.467+0000 7fd8fe59c700 1 --2- 172.21.15.98:0/2130231127 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8e8004850 0x7fd8e8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:51.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.467+0000 7fd8fed9d700 1 -- 172.21.15.98:0/2130231127 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9000b51b0 msgr2=0x7fd8e8005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:51.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.467+0000 7fd8fed9d700 1 --2- 172.21.15.98:0/2130231127 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9000b51b0 0x7fd8e8005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:51.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.467+0000 7fd8fed9d700 1 -- 172.21.15.98:0/2130231127 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8e8004850 msgr2=0x7fd8e8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:51.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.467+0000 7fd8fed9d700 1 --2- 172.21.15.98:0/2130231127 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8e8004850 0x7fd8e8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:51.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.467+0000 7fd8fed9d700 1 -- 172.21.15.98:0/2130231127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f8009a70 con 0x7fd8e8003bc0 2023-03-03T08:00:51.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.468+0000 7fd8fed9d700 1 --2- 172.21.15.98:0/2130231127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e8003bc0 0x7fd8e8003fb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0001940 tx=0x7fd8e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7548f0473a0bcbb8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:51.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.468+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0001430 con 0x7fd8e8003bc0 2023-03-03T08:00:51.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.468+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f0001d70 con 0x7fd8e8003bc0 2023-03-03T08:00:51.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.468+0000 7fd8fdd9b700 1 -- 172.21.15.98:0/2130231127 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0005510 con 0x7fd8e8003bc0 2023-03-03T08:00:51.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.468+0000 7fd9049fe700 1 -- 172.21.15.98:0/2130231127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e8003bc0 msgr2=0x7fd8e8003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:51.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.468+0000 7fd9049fe700 1 --2- 172.21.15.98:0/2130231127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e8003bc0 0x7fd8e8003fb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0001940 tx=0x7fd8e8005880 comp rx=0 tx=0).stop 2023-03-03T08:00:51.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.468+0000 7fd8ff59e700 1 -- 172.21.15.98:0/2130231127 reap_dead start 2023-03-03T08:00:51.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.468+0000 7fd9049fe700 1 -- 172.21.15.98:0/2130231127 shutdown_connections 2023-03-03T08:00:51.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.468+0000 7fd9049fe700 1 -- 172.21.15.98:0/2130231127 >> 172.21.15.98:0/2130231127 conn(0x7fd90006b880 msgr2=0x7fd9000cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:51.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.468+0000 7fd9049fe700 1 -- 172.21.15.98:0/2130231127 shutdown_connections 2023-03-03T08:00:51.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.468+0000 7fd9049fe700 1 -- 172.21.15.98:0/2130231127 wait complete. 2023-03-03T08:00:51.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.470+0000 7fd9049fe700 1 Processor -- start 2023-03-03T08:00:51.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.471+0000 7fd9049fe700 1 -- start start 2023-03-03T08:00:51.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.472+0000 7fd9049fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8e8003bc0 0x7fd900157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:51.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.472+0000 7fd9049fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9000ca850 0x7fd9000cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:51.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.472+0000 7fd9049fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9000b3650 0x7fd9000b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:51.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.472+0000 7fd9049fe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000b4030 con 0x7fd9000ca850 2023-03-03T08:00:51.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.472+0000 7fd9049fe700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90015cfa0 con 0x7fd8e8003bc0 2023-03-03T08:00:51.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.472+0000 7fd9049fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90015d280 con 0x7fd9000b3650 2023-03-03T08:00:51.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.472+0000 7fd8fe59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9000ca850 0x7fd9000cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:51.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.472+0000 7fd8fe59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9000ca850 0x7fd9000cacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59704/0 (socket says 172.21.15.98:59704) 2023-03-03T08:00:51.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.472+0000 7fd8fe59c700 1 -- 172.21.15.98:0/1717788889 learned_addr learned my addr 172.21.15.98:0/1717788889 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:51.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.472+0000 7fd8fed9d700 1 --2- 172.21.15.98:0/1717788889 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8e8003bc0 0x7fd900157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:51.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.472+0000 7fd8ff59e700 1 --2- 172.21.15.98:0/1717788889 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9000b3650 0x7fd9000b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:51.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.472+0000 7fd8fe59c700 1 -- 172.21.15.98:0/1717788889 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9000b3650 msgr2=0x7fd9000b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:51.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.473+0000 7fd8fe59c700 1 --2- 172.21.15.98:0/1717788889 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9000b3650 0x7fd9000b3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.473+0000 7fd8fe59c700 1 -- 172.21.15.98:0/1717788889 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8e8003bc0 msgr2=0x7fd900157260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.473+0000 7fd8fe59c700 1 --2- 172.21.15.98:0/1717788889 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8e8003bc0 0x7fd900157260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.473+0000 7fd8fe59c700 1 -- 172.21.15.98:0/1717788889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9001612b0 con 0x7fd9000ca850 2023-03-03T08:00:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.473+0000 7fd8ff59e700 1 --2- 172.21.15.98:0/1717788889 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9000b3650 0x7fd9000b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.473+0000 7fd8fe59c700 1 --2- 172.21.15.98:0/1717788889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9000ca850 0x7fd9000cacc0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4007310 tx=0x7fd8f40076d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.473+0000 7fd8e77fe700 1 -- 172.21.15.98:0/1717788889 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f400f870 con 0x7fd9000ca850 2023-03-03T08:00:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.473+0000 7fd8e77fe700 1 -- 172.21.15.98:0/1717788889 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f400f9d0 con 0x7fd9000ca850 2023-03-03T08:00:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.473+0000 7fd8e77fe700 1 -- 172.21.15.98:0/1717788889 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f400fd60 con 0x7fd9000ca850 2023-03-03T08:00:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.473+0000 7fd9049fe700 1 -- 172.21.15.98:0/1717788889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd900160fd0 con 0x7fd9000ca850 2023-03-03T08:00:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.473+0000 7fd9049fe700 1 -- 172.21.15.98:0/1717788889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9000d0eb0 con 0x7fd9000ca850 2023-03-03T08:00:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.475+0000 7fd9049fe700 1 -- 172.21.15.98:0/1717788889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd900068290 con 0x7fd9000ca850 2023-03-03T08:00:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.476+0000 7fd8e77fe700 1 -- 172.21.15.98:0/1717788889 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 10) v1 ==== 86575+0+0 (secure 0 0 0) 0x7fd8f401d460 con 0x7fd9000ca850 2023-03-03T08:00:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.477+0000 7fd8e77fe700 1 --2- 172.21.15.98:0/1717788889 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8ec068410 0x7fd8ec06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.477+0000 7fd8e77fe700 1 -- 172.21.15.98:0/1717788889 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd8f4085fe0 con 0x7fd9000ca850 2023-03-03T08:00:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.477+0000 7fd8fed9d700 1 --2- 172.21.15.98:0/1717788889 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8ec068410 0x7fd8ec06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.478+0000 7fd8fed9d700 1 --2- 172.21.15.98:0/1717788889 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8ec068410 0x7fd8ec06a8c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0015040 tx=0x7fd8f000a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:51.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.481+0000 7fd8e77fe700 1 -- 172.21.15.98:0/1717788889 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd8f40526a0 con 0x7fd9000ca850 2023-03-03T08:00:51.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.751+0000 7fd9049fe700 1 -- 172.21.15.98:0/1717788889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth get", "entity": "client.alice", "format": "json-pretty"} v 0) v1 -- 0x7fd90015ce00 con 0x7fd9000ca850 2023-03-03T08:00:51.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.752+0000 7fd8e77fe700 1 -- 172.21.15.98:0/1717788889 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth get", "entity": "client.alice", "format": "json-pretty"}]=0 v12) v1 ==== 107+0+295 (secure 0 0 0) 0x7fd90015ce00 con 0x7fd9000ca850 2023-03-03T08:00:51.754 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:51.755 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T08:00:51.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:00:51.756 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.alice", 2023-03-03T08:00:51.757 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQAzqQFkigHjCRAA+vJpqEsD7NuAQLrmRJE9Ag==", 2023-03-03T08:00:51.757 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:00:51.757 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow rw path=/volumes/_nogroup/subvolume_0000000001024199", 2023-03-03T08:00:51.758 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow r", 2023-03-03T08:00:51.758 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rw pool=cephfs_data" 2023-03-03T08:00:51.758 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:51.758 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:00:51.759 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T08:00:51.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.755+0000 7fd9049fe700 1 -- 172.21.15.98:0/1717788889 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8ec068410 msgr2=0x7fd8ec06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:51.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.755+0000 7fd9049fe700 1 --2- 172.21.15.98:0/1717788889 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8ec068410 0x7fd8ec06a8c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0015040 tx=0x7fd8f000a000 comp rx=0 tx=0).stop 2023-03-03T08:00:51.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.755+0000 7fd9049fe700 1 -- 172.21.15.98:0/1717788889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9000ca850 msgr2=0x7fd9000cacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:51.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.755+0000 7fd9049fe700 1 --2- 172.21.15.98:0/1717788889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9000ca850 0x7fd9000cacc0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4007310 tx=0x7fd8f40076d0 comp rx=0 tx=0).stop 2023-03-03T08:00:51.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.755+0000 7fd9049fe700 1 -- 172.21.15.98:0/1717788889 shutdown_connections 2023-03-03T08:00:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.755+0000 7fd9049fe700 1 --2- 172.21.15.98:0/1717788889 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8ec068410 0x7fd8ec06a8c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.755+0000 7fd9049fe700 1 --2- 172.21.15.98:0/1717788889 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9000b3650 0x7fd9000b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.755+0000 7fd9049fe700 1 --2- 172.21.15.98:0/1717788889 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8e8003bc0 0x7fd900157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.755+0000 7fd9049fe700 1 --2- 172.21.15.98:0/1717788889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9000ca850 0x7fd9000cacc0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.755+0000 7fd9049fe700 1 -- 172.21.15.98:0/1717788889 >> 172.21.15.98:0/1717788889 conn(0x7fd90006b880 msgr2=0x7fd9000737a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.755+0000 7fd9049fe700 1 -- 172.21.15.98:0/1717788889 shutdown_connections 2023-03-03T08:00:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:51.755+0000 7fd9049fe700 1 -- 172.21.15.98:0/1717788889 wait complete. 2023-03-03T08:00:51.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000001024199 2023-03-03T08:00:52.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.029+0000 7f240af9d700 1 Processor -- start 2023-03-03T08:00:52.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.030+0000 7f240af9d700 1 -- start start 2023-03-03T08:00:52.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.031+0000 7f240af9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f24040b4aa0 0x7f24040b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:52.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.031+0000 7f240af9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240415dd00 con 0x7f24040cf570 2023-03-03T08:00:52.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.031+0000 7f240af9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24040cd780 con 0x7f24041609b0 2023-03-03T08:00:52.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.031+0000 7f240af9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24040cebb0 con 0x7f24040b4aa0 2023-03-03T08:00:52.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.031+0000 7f240979a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f24040cf570 0x7f240415d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37700/0 (socket says 172.21.15.98:37700) 2023-03-03T08:00:52.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.031+0000 7f240979a700 1 -- 172.21.15.98:0/2394580866 learned_addr learned my addr 172.21.15.98:0/2394580866 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:52.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.031+0000 7f240a79c700 1 --2- 172.21.15.98:0/2394580866 >> v2:172.21.15.100:3300/0 conn(0x7f24040b4aa0 0x7f24040b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:52.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.032+0000 7f2408f99700 1 -- 172.21.15.98:0/2394580866 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356989047 0 0) 0x7f240415dd00 con 0x7f24040cf570 2023-03-03T08:00:52.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.032+0000 7f2408f99700 1 -- 172.21.15.98:0/2394580866 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23f00035b0 con 0x7f24040cf570 2023-03-03T08:00:52.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.032+0000 7f240a79c700 1 -- 172.21.15.98:0/2394580866 >> v1:172.21.15.100:6789/0 conn(0x7f24041609b0 legacy=0x7f2404160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:52.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.032+0000 7f240a79c700 1 -- 172.21.15.98:0/2394580866 >> v1:172.21.15.98:6789/0 conn(0x7f24040cf570 legacy=0x7f240415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:52.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.032+0000 7f240a79c700 1 -- 172.21.15.98:0/2394580866 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24040cda60 con 0x7f24040b4aa0 2023-03-03T08:00:52.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.032+0000 7f240a79c700 1 --2- 172.21.15.98:0/2394580866 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24040b4aa0 0x7f24040b4f20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009d90 tx=0x7f23fc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c937f441cc2e435 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:52.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.033+0000 7f2408f99700 1 -- 172.21.15.98:0/2394580866 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00e070 con 0x7f24040b4aa0 2023-03-03T08:00:52.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.033+0000 7f2408f99700 1 -- 172.21.15.98:0/2394580866 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc00bb20 con 0x7f24040b4aa0 2023-03-03T08:00:52.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.033+0000 7f2408f99700 1 -- 172.21.15.98:0/2394580866 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc012430 con 0x7f24040b4aa0 2023-03-03T08:00:52.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.033+0000 7f240af9d700 1 -- 172.21.15.98:0/2394580866 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24040b4aa0 msgr2=0x7f24040b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:52.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.033+0000 7f240af9d700 1 --2- 172.21.15.98:0/2394580866 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24040b4aa0 0x7f24040b4f20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009d90 tx=0x7f23fc00b330 comp rx=0 tx=0).stop 2023-03-03T08:00:52.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.033+0000 7f240af9d700 1 -- 172.21.15.98:0/2394580866 shutdown_connections 2023-03-03T08:00:52.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.033+0000 7f240af9d700 1 --2- 172.21.15.98:0/2394580866 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24040b4aa0 0x7f24040b4f20 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.033+0000 7f240af9d700 1 -- 172.21.15.98:0/2394580866 >> 172.21.15.98:0/2394580866 conn(0x7f240406b880 msgr2=0x7f24040cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:52.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.033+0000 7f240af9d700 1 -- 172.21.15.98:0/2394580866 shutdown_connections 2023-03-03T08:00:52.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.033+0000 7f240af9d700 1 -- 172.21.15.98:0/2394580866 wait complete. 2023-03-03T08:00:52.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.035+0000 7f240af9d700 1 Processor -- start 2023-03-03T08:00:52.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.035+0000 7f240af9d700 1 -- start start 2023-03-03T08:00:52.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.036+0000 7f240af9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24040b4aa0 0x7f2404150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:52.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.036+0000 7f240af9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24040cf570 0x7f24041f2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:52.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.037+0000 7f240af9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24041609b0 0x7f24041f2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:52.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.037+0000 7f240af9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24040c7a10 con 0x7f24040cf570 2023-03-03T08:00:52.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.037+0000 7f240af9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24040c7cf0 con 0x7f24041609b0 2023-03-03T08:00:52.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.037+0000 7f240af9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24040c7fd0 con 0x7f24040b4aa0 2023-03-03T08:00:52.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.037+0000 7f240979a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24040cf570 0x7f24041f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:52.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.037+0000 7f240979a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24040cf570 0x7f24041f2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59706/0 (socket says 172.21.15.98:59706) 2023-03-03T08:00:52.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.037+0000 7f240979a700 1 -- 172.21.15.98:0/948998491 learned_addr learned my addr 172.21.15.98:0/948998491 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:52.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.037+0000 7f2409f9b700 1 --2- 172.21.15.98:0/948998491 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24040b4aa0 0x7f2404150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:52.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.037+0000 7f240a79c700 1 --2- 172.21.15.98:0/948998491 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24041609b0 0x7f24041f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:52.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.037+0000 7f240979a700 1 -- 172.21.15.98:0/948998491 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24040b4aa0 msgr2=0x7f2404150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:52.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.037+0000 7f240979a700 1 --2- 172.21.15.98:0/948998491 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24040b4aa0 0x7f2404150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.037+0000 7f240979a700 1 -- 172.21.15.98:0/948998491 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24041609b0 msgr2=0x7f24041f2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:52.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.037+0000 7f240979a700 1 --2- 172.21.15.98:0/948998491 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24041609b0 0x7f24041f2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.037+0000 7f240979a700 1 -- 172.21.15.98:0/948998491 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc009a70 con 0x7f24040cf570 2023-03-03T08:00:52.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.038+0000 7f240979a700 1 --2- 172.21.15.98:0/948998491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24040cf570 0x7f24041f2320 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f23f400c700 tx=0x7f23f400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:52.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.038+0000 7f23fa7fc700 1 -- 172.21.15.98:0/948998491 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4005d70 con 0x7f24040cf570 2023-03-03T08:00:52.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.038+0000 7f23fa7fc700 1 -- 172.21.15.98:0/948998491 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f40070d0 con 0x7f24040cf570 2023-03-03T08:00:52.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.038+0000 7f240af9d700 1 -- 172.21.15.98:0/948998491 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240415cdf0 con 0x7f24040cf570 2023-03-03T08:00:52.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.038+0000 7f23fa7fc700 1 -- 172.21.15.98:0/948998491 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4007460 con 0x7f24040cf570 2023-03-03T08:00:52.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.038+0000 7f240af9d700 1 -- 172.21.15.98:0/948998491 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24040cdd40 con 0x7f24040cf570 2023-03-03T08:00:52.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.039+0000 7f240af9d700 1 -- 172.21.15.98:0/948998491 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240415d0d0 con 0x7f24040cf570 2023-03-03T08:00:52.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.040+0000 7f23fa7fc700 1 -- 172.21.15.98:0/948998491 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 10) v1 ==== 86575+0+0 (secure 0 0 0) 0x7f23f4019030 con 0x7f24040cf570 2023-03-03T08:00:52.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.042+0000 7f23fa7fc700 1 --2- 172.21.15.98:0/948998491 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f23e8068410 0x7f23e806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:52.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.042+0000 7f23fa7fc700 1 -- 172.21.15.98:0/948998491 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f23f4085160 con 0x7f24040cf570 2023-03-03T08:00:52.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.042+0000 7f2409f9b700 1 --2- 172.21.15.98:0/948998491 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f23e8068410 0x7f23e806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:52.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.043+0000 7f2409f9b700 1 --2- 172.21.15.98:0/948998491 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f23e8068410 0x7f23e806a8c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f24040d2a90 tx=0x7f2400009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:52.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.045+0000 7f23fa7fc700 1 -- 172.21.15.98:0/948998491 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f23f4051840 con 0x7f24040cf570 2023-03-03T08:00:52.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.270+0000 7f240af9d700 1 -- 172.21.15.98:0/948998491 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000001024199", "target": ["mon-mgr", ""]}) v1 -- 0x7f24040c7e60 con 0x7f23e8068410 2023-03-03T08:00:52.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.284+0000 7f23fa7fc700 1 -- 172.21.15.98:0/948998491 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f24040c7e60 con 0x7f23e8068410 2023-03-03T08:00:52.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.288+0000 7f240af9d700 1 -- 172.21.15.98:0/948998491 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f23e8068410 msgr2=0x7f23e806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:52.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.288+0000 7f240af9d700 1 --2- 172.21.15.98:0/948998491 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f23e8068410 0x7f23e806a8c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f24040d2a90 tx=0x7f2400009040 comp rx=0 tx=0).stop 2023-03-03T08:00:52.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.288+0000 7f240af9d700 1 -- 172.21.15.98:0/948998491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24040cf570 msgr2=0x7f24041f2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:52.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.288+0000 7f240af9d700 1 --2- 172.21.15.98:0/948998491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24040cf570 0x7f24041f2320 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f23f400c700 tx=0x7f23f400ca10 comp rx=0 tx=0).stop 2023-03-03T08:00:52.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.288+0000 7f240af9d700 1 -- 172.21.15.98:0/948998491 shutdown_connections 2023-03-03T08:00:52.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.288+0000 7f240af9d700 1 --2- 172.21.15.98:0/948998491 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f23e8068410 0x7f23e806a8c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.288+0000 7f240af9d700 1 --2- 172.21.15.98:0/948998491 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24041609b0 0x7f24041f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.288+0000 7f240af9d700 1 --2- 172.21.15.98:0/948998491 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24040b4aa0 0x7f2404150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.288+0000 7f240af9d700 1 --2- 172.21.15.98:0/948998491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24040cf570 0x7f24041f2320 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.288+0000 7f240af9d700 1 -- 172.21.15.98:0/948998491 >> 172.21.15.98:0/948998491 conn(0x7f240406b880 msgr2=0x7f24040b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:52.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.288+0000 7f240af9d700 1 -- 172.21.15.98:0/948998491 shutdown_connections 2023-03-03T08:00:52.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.288+0000 7f240af9d700 1 -- 172.21.15.98:0/948998491 wait complete. 2023-03-03T08:00:52.291 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:00:52.290+0000 7f810ec10700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:00:52.292 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:00:52.290+0000 7f810ec10700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:00:52.292 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:00:52.290+0000 7f810ec10700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:00:52.292 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:00:52.290+0000 7f810ec10700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:00:52.292 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:00:52.290+0000 7f810ec10700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:00:52.311 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:00:52.310+0000 7f810f411700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:00:52.311 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:00:52.310+0000 7f810f411700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:00:52.312 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:00:52.310+0000 7f810f411700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:00:52.312 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:00:52.310+0000 7f810f411700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:00:52.312 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:00:52.310+0000 7f810f411700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:00:52.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000001024200 2023-03-03T08:00:52.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.577+0000 7f653ad9d700 1 Processor -- start 2023-03-03T08:00:52.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.577+0000 7f653ad9d700 1 -- start start 2023-03-03T08:00:52.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.578+0000 7f653ad9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f65341609b0 0x7f6534160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:52.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.578+0000 7f653ad9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65340b4aa0 0x7f65340b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:52.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.578+0000 7f653ad9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65341612c0 con 0x7f65340cf570 2023-03-03T08:00:52.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.578+0000 7f653ad9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65340cebb0 con 0x7f65341609b0 2023-03-03T08:00:52.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.578+0000 7f653ad9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65340b5460 con 0x7f65340b4aa0 2023-03-03T08:00:52.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.579+0000 7f6539d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f65341609b0 0x7f6534160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:52.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.579+0000 7f653959a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f65340cf570 0x7f653415d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37708/0 (socket says 172.21.15.98:37708) 2023-03-03T08:00:52.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.579+0000 7f6539d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f65341609b0 0x7f6534160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59742/0 (socket says 172.21.15.98:59742) 2023-03-03T08:00:52.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.579+0000 7f653959a700 1 -- 172.21.15.98:0/492321091 learned_addr learned my addr 172.21.15.98:0/492321091 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:52.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.579+0000 7f653a59c700 1 --2- 172.21.15.98:0/492321091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65340b4aa0 0x7f65340b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:52.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.579+0000 7f6539d9b700 1 -- 172.21.15.98:0/492321091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65340b4aa0 msgr2=0x7f65340b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:52.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.579+0000 7f6539d9b700 1 --2- 172.21.15.98:0/492321091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65340b4aa0 0x7f65340b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.579+0000 7f6539d9b700 1 -- 172.21.15.98:0/492321091 >> v1:172.21.15.98:6789/0 conn(0x7f65340cf570 legacy=0x7f653415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:52.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.579+0000 7f6539d9b700 1 -- 172.21.15.98:0/492321091 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653415dd00 con 0x7f65341609b0 2023-03-03T08:00:52.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.579+0000 7f653a59c700 1 --2- 172.21.15.98:0/492321091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65340b4aa0 0x7f65340b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:52.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.579+0000 7f6539d9b700 1 --2- 172.21.15.98:0/492321091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65341609b0 0x7f6534160d80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f6530009840 tx=0x7f6530009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3afd52872e75f17a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:52.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.580+0000 7f6538d99700 1 -- 172.21.15.98:0/492321091 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530011830 con 0x7f65341609b0 2023-03-03T08:00:52.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.580+0000 7f6538d99700 1 -- 172.21.15.98:0/492321091 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6530011990 con 0x7f65341609b0 2023-03-03T08:00:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.580+0000 7f6538d99700 1 -- 172.21.15.98:0/492321091 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530011d20 con 0x7f65341609b0 2023-03-03T08:00:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.580+0000 7f653ad9d700 1 -- 172.21.15.98:0/492321091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65341609b0 msgr2=0x7f6534160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.580+0000 7f653ad9d700 1 --2- 172.21.15.98:0/492321091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65341609b0 0x7f6534160d80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f6530009840 tx=0x7f6530009b50 comp rx=0 tx=0).stop 2023-03-03T08:00:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.580+0000 7f653ad9d700 1 -- 172.21.15.98:0/492321091 shutdown_connections 2023-03-03T08:00:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.580+0000 7f653ad9d700 1 --2- 172.21.15.98:0/492321091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65341609b0 0x7f6534160d80 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.580+0000 7f653ad9d700 1 --2- 172.21.15.98:0/492321091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65340b4aa0 0x7f65340b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.580+0000 7f653ad9d700 1 -- 172.21.15.98:0/492321091 >> 172.21.15.98:0/492321091 conn(0x7f653406b880 msgr2=0x7f65340cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:52.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.581+0000 7f653ad9d700 1 -- 172.21.15.98:0/492321091 shutdown_connections 2023-03-03T08:00:52.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.581+0000 7f653ad9d700 1 -- 172.21.15.98:0/492321091 wait complete. 2023-03-03T08:00:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.583+0000 7f653ad9d700 1 Processor -- start 2023-03-03T08:00:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.583+0000 7f653ad9d700 1 -- start start 2023-03-03T08:00:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.584+0000 7f653ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65340b4aa0 0x7f65340b60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.584+0000 7f653ad9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65340cf570 0x7f65340b8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.584+0000 7f653ad9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f65341609b0 0x7f65340b8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:52.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.584+0000 7f653ad9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65340b8ea0 con 0x7f65340b4aa0 2023-03-03T08:00:52.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.584+0000 7f653ad9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653415ba20 con 0x7f65341609b0 2023-03-03T08:00:52.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.584+0000 7f653ad9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f653415bd00 con 0x7f65340cf570 2023-03-03T08:00:52.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.584+0000 7f6539d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65340b4aa0 0x7f65340b60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:52.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.584+0000 7f6539d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65340b4aa0 0x7f65340b60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59746/0 (socket says 172.21.15.98:59746) 2023-03-03T08:00:52.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.584+0000 7f6539d9b700 1 -- 172.21.15.98:0/2313386714 learned_addr learned my addr 172.21.15.98:0/2313386714 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:52.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.584+0000 7f6539d9b700 1 -- 172.21.15.98:0/2313386714 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65340cf570 msgr2=0x7f65340b8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:52.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.584+0000 7f6539d9b700 1 --2- 172.21.15.98:0/2313386714 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65340cf570 0x7f65340b8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.584+0000 7f6539d9b700 1 -- 172.21.15.98:0/2313386714 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f65341609b0 msgr2=0x7f65340b8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:52.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.584+0000 7f6539d9b700 1 --2- 172.21.15.98:0/2313386714 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f65341609b0 0x7f65340b8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.585+0000 7f6539d9b700 1 -- 172.21.15.98:0/2313386714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65300094a0 con 0x7f65340b4aa0 2023-03-03T08:00:52.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.585+0000 7f6539d9b700 1 --2- 172.21.15.98:0/2313386714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65340b4aa0 0x7f65340b60c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f6530000c00 tx=0x7f6530004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:52.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.585+0000 7f652a7fc700 1 -- 172.21.15.98:0/2313386714 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65300036a0 con 0x7f65340b4aa0 2023-03-03T08:00:52.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.585+0000 7f652a7fc700 1 -- 172.21.15.98:0/2313386714 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653001e070 con 0x7f65340b4aa0 2023-03-03T08:00:52.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.585+0000 7f652a7fc700 1 -- 172.21.15.98:0/2313386714 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530003a20 con 0x7f65340b4aa0 2023-03-03T08:00:52.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.585+0000 7f653ad9d700 1 -- 172.21.15.98:0/2313386714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65340cda60 con 0x7f65340b4aa0 2023-03-03T08:00:52.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.585+0000 7f653ad9d700 1 -- 172.21.15.98:0/2313386714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65340cd780 con 0x7f65340b4aa0 2023-03-03T08:00:52.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.587+0000 7f653ad9d700 1 -- 172.21.15.98:0/2313386714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65340cdd40 con 0x7f65340b4aa0 2023-03-03T08:00:52.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.588+0000 7f652a7fc700 1 -- 172.21.15.98:0/2313386714 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 10) v1 ==== 86575+0+0 (secure 0 0 0) 0x7f653000f050 con 0x7f65340b4aa0 2023-03-03T08:00:52.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.589+0000 7f652a7fc700 1 --2- 172.21.15.98:0/2313386714 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6518068410 0x7f651806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:52.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.589+0000 7f652a7fc700 1 -- 172.21.15.98:0/2313386714 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6530086f30 con 0x7f65340b4aa0 2023-03-03T08:00:52.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.589+0000 7f653959a700 1 --2- 172.21.15.98:0/2313386714 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6518068410 0x7f651806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:52.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.590+0000 7f653959a700 1 --2- 172.21.15.98:0/2313386714 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6518068410 0x7f651806a8c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6524001670 tx=0x7f6524007ad0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:52.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.594+0000 7f652a7fc700 1 -- 172.21.15.98:0/2313386714 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6530053580 con 0x7f65340b4aa0 2023-03-03T08:00:52.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.818+0000 7f653ad9d700 1 -- 172.21.15.98:0/2313386714 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000001024200", "target": ["mon-mgr", ""]}) v1 -- 0x7f6534068290 con 0x7f6518068410 2023-03-03T08:00:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.834+0000 7f652a7fc700 1 -- 172.21.15.98:0/2313386714 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6534068290 con 0x7f6518068410 2023-03-03T08:00:52.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.837+0000 7f653ad9d700 1 -- 172.21.15.98:0/2313386714 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6518068410 msgr2=0x7f651806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.837+0000 7f653ad9d700 1 --2- 172.21.15.98:0/2313386714 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6518068410 0x7f651806a8c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6524001670 tx=0x7f6524007ad0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.837+0000 7f653ad9d700 1 -- 172.21.15.98:0/2313386714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65340b4aa0 msgr2=0x7f65340b60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.837+0000 7f653ad9d700 1 --2- 172.21.15.98:0/2313386714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65340b4aa0 0x7f65340b60c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f6530000c00 tx=0x7f6530004750 comp rx=0 tx=0).stop 2023-03-03T08:00:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.837+0000 7f653ad9d700 1 -- 172.21.15.98:0/2313386714 shutdown_connections 2023-03-03T08:00:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.837+0000 7f653ad9d700 1 --2- 172.21.15.98:0/2313386714 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6518068410 0x7f651806a8c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.837+0000 7f653ad9d700 1 --2- 172.21.15.98:0/2313386714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65340b4aa0 0x7f65340b60c0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.837+0000 7f653ad9d700 1 --2- 172.21.15.98:0/2313386714 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f65341609b0 0x7f65340b8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.837+0000 7f653ad9d700 1 --2- 172.21.15.98:0/2313386714 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65340cf570 0x7f65340b8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.837+0000 7f653ad9d700 1 -- 172.21.15.98:0/2313386714 >> 172.21.15.98:0/2313386714 conn(0x7f653406b880 msgr2=0x7f653415a080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.837+0000 7f653ad9d700 1 -- 172.21.15.98:0/2313386714 shutdown_connections 2023-03-03T08:00:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:52.837+0000 7f653ad9d700 1 -- 172.21.15.98:0/2313386714 wait complete. 2023-03-03T08:00:52.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:52.870 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2023-03-03T08:00:53.009 INFO:teuthology.orchestra.run.smithi098.stdout:2 2023-03-03T08:00:53.011 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-03-03T08:00:53.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:00:53.142 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:00:53.272 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-03-03T08:00:53.272 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:00:53.273 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:00:53.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:00:53.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.585+0000 7f56cebff700 1 Processor -- start 2023-03-03T08:00:53.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.585+0000 7f56cebff700 1 -- start start 2023-03-03T08:00:53.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.586+0000 7f56cebff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c80b87c0 0x7f56c80cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:53.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.587+0000 7f56cebff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c80d0ef0 con 0x7f56c80b5410 2023-03-03T08:00:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.587+0000 7f56cebff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c80d0600 con 0x7f56c80b8b90 2023-03-03T08:00:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.587+0000 7f56cebff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c80cd970 con 0x7f56c80b87c0 2023-03-03T08:00:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.587+0000 7f56cd19c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f56c80b5410 0x7f56c80d13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37724/0 (socket says 172.21.15.98:37724) 2023-03-03T08:00:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.587+0000 7f56cd19c700 1 -- 172.21.15.98:0/1940088879 learned_addr learned my addr 172.21.15.98:0/1940088879 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.587+0000 7f56cc99b700 1 --2- 172.21.15.98:0/1940088879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c80b87c0 0x7f56c80cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.587+0000 7f56c77fe700 1 -- 172.21.15.98:0/1940088879 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540991741 0 0) 0x7f56c80d0ef0 con 0x7f56c80b5410 2023-03-03T08:00:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.587+0000 7f56c77fe700 1 -- 172.21.15.98:0/1940088879 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56b80035b0 con 0x7f56c80b5410 2023-03-03T08:00:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.587+0000 7f56c77fe700 1 -- 172.21.15.98:0/1940088879 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690895274 0 0) 0x7f56c80d0600 con 0x7f56c80b8b90 2023-03-03T08:00:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.587+0000 7f56c77fe700 1 -- 172.21.15.98:0/1940088879 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56c80d0ef0 con 0x7f56c80b8b90 2023-03-03T08:00:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.588+0000 7f56cc99b700 1 -- 172.21.15.98:0/1940088879 >> v1:172.21.15.100:6789/0 conn(0x7f56c80b8b90 legacy=0x7f56c80b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.588+0000 7f56cc99b700 1 -- 172.21.15.98:0/1940088879 >> v1:172.21.15.98:6789/0 conn(0x7f56c80b5410 legacy=0x7f56c80d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.588+0000 7f56cc99b700 1 -- 172.21.15.98:0/1940088879 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c80cdc50 con 0x7f56c80b87c0 2023-03-03T08:00:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.588+0000 7f56cc99b700 1 --2- 172.21.15.98:0/1940088879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c80b87c0 0x7f56c80cffe0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f56bc009870 tx=0x7f56bc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8142f70e21c9e42e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.588+0000 7f56c77fe700 1 -- 172.21.15.98:0/1940088879 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc004230 con 0x7f56c80b87c0 2023-03-03T08:00:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.588+0000 7f56c77fe700 1 -- 172.21.15.98:0/1940088879 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc004390 con 0x7f56c80b87c0 2023-03-03T08:00:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.588+0000 7f56c77fe700 1 -- 172.21.15.98:0/1940088879 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc0036a0 con 0x7f56c80b87c0 2023-03-03T08:00:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.589+0000 7f56cebff700 1 -- 172.21.15.98:0/1940088879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c80b87c0 msgr2=0x7f56c80cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.589+0000 7f56cebff700 1 --2- 172.21.15.98:0/1940088879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c80b87c0 0x7f56c80cffe0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f56bc009870 tx=0x7f56bc009b80 comp rx=0 tx=0).stop 2023-03-03T08:00:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.589+0000 7f56cebff700 1 -- 172.21.15.98:0/1940088879 shutdown_connections 2023-03-03T08:00:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.589+0000 7f56cebff700 1 --2- 172.21.15.98:0/1940088879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c80b87c0 0x7f56c80cffe0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.589+0000 7f56cebff700 1 -- 172.21.15.98:0/1940088879 >> 172.21.15.98:0/1940088879 conn(0x7f56c806aee0 msgr2=0x7f56c806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.589+0000 7f56cebff700 1 -- 172.21.15.98:0/1940088879 shutdown_connections 2023-03-03T08:00:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.589+0000 7f56cebff700 1 -- 172.21.15.98:0/1940088879 wait complete. 2023-03-03T08:00:53.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.591+0000 7f56cebff700 1 Processor -- start 2023-03-03T08:00:53.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.592+0000 7f56cebff700 1 -- start start 2023-03-03T08:00:53.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56cebff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56c80b5410 0x7f56c81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:53.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56cebff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56c80b87c0 0x7f56c815eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:53.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56cebff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c80b8b90 0x7f56c81f2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:53.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56cebff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c815fb00 con 0x7f56c80b5410 2023-03-03T08:00:53.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56cebff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c815f050 con 0x7f56c80b87c0 2023-03-03T08:00:53.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56cebff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c81575a0 con 0x7f56c80b8b90 2023-03-03T08:00:53.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56cc99b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56c80b5410 0x7f56c81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:53.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56cc99b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56c80b5410 0x7f56c81529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59758/0 (socket says 172.21.15.98:59758) 2023-03-03T08:00:53.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56cc99b700 1 -- 172.21.15.98:0/4230894772 learned_addr learned my addr 172.21.15.98:0/4230894772 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:53.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56cd19c700 1 --2- 172.21.15.98:0/4230894772 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c80b8b90 0x7f56c81f2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:53.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56c7fff700 1 --2- 172.21.15.98:0/4230894772 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56c80b87c0 0x7f56c815eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:53.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56cc99b700 1 -- 172.21.15.98:0/4230894772 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c80b8b90 msgr2=0x7f56c81f2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:53.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56cc99b700 1 --2- 172.21.15.98:0/4230894772 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c80b8b90 0x7f56c81f2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:53.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56cc99b700 1 -- 172.21.15.98:0/4230894772 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56c80b87c0 msgr2=0x7f56c815eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:53.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56cc99b700 1 --2- 172.21.15.98:0/4230894772 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56c80b87c0 0x7f56c815eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:53.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56cc99b700 1 -- 172.21.15.98:0/4230894772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc0094a0 con 0x7f56c80b5410 2023-03-03T08:00:53.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.593+0000 7f56c7fff700 1 --2- 172.21.15.98:0/4230894772 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56c80b87c0 0x7f56c815eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:53.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.594+0000 7f56cc99b700 1 --2- 172.21.15.98:0/4230894772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56c80b5410 0x7f56c81529d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0090b0 tx=0x7f56bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:53.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.594+0000 7f56c57fa700 1 -- 172.21.15.98:0/4230894772 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc003ea0 con 0x7f56c80b5410 2023-03-03T08:00:53.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.594+0000 7f56c57fa700 1 -- 172.21.15.98:0/4230894772 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc01e070 con 0x7f56c80b5410 2023-03-03T08:00:53.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.594+0000 7f56c57fa700 1 -- 172.21.15.98:0/4230894772 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc011870 con 0x7f56c80b5410 2023-03-03T08:00:53.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.594+0000 7f56cebff700 1 -- 172.21.15.98:0/4230894772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c80cdf30 con 0x7f56c80b5410 2023-03-03T08:00:53.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.594+0000 7f56cebff700 1 -- 172.21.15.98:0/4230894772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c8073a80 con 0x7f56c80b5410 2023-03-03T08:00:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.595+0000 7f56cebff700 1 -- 172.21.15.98:0/4230894772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c80ce0c0 con 0x7f56c80b5410 2023-03-03T08:00:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.597+0000 7f56c57fa700 1 -- 172.21.15.98:0/4230894772 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 10) v1 ==== 86575+0+0 (secure 0 0 0) 0x7f56bc017020 con 0x7f56c80b5410 2023-03-03T08:00:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.599+0000 7f56c57fa700 1 --2- 172.21.15.98:0/4230894772 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f56ac068410 0x7f56ac06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.599+0000 7f56c57fa700 1 -- 172.21.15.98:0/4230894772 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f56bc087070 con 0x7f56c80b5410 2023-03-03T08:00:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.599+0000 7f56c7fff700 1 --2- 172.21.15.98:0/4230894772 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f56ac068410 0x7f56ac06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.599+0000 7f56c7fff700 1 --2- 172.21.15.98:0/4230894772 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f56ac068410 0x7f56ac06a8c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f56c80b2980 tx=0x7f56b4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:53.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.603+0000 7f56c57fa700 1 -- 172.21.15.98:0/4230894772 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f56bc0b85e0 con 0x7f56c80b5410 2023-03-03T08:00:53.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.800+0000 7f56cebff700 1 -- 172.21.15.98:0/4230894772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56c80729b0 con 0x7f56c80b5410 2023-03-03T08:00:53.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.802+0000 7f56c57fa700 1 -- 172.21.15.98:0/4230894772 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+17921 (secure 0 0 0) 0x7f56bc020070 con 0x7f56c80b5410 2023-03-03T08:00:53.803 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:53.804 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":66,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:00:23.326015+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:00:19.823120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-03-03T08:00:21.324346+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:00:53.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.804+0000 7f56cebff700 1 -- 172.21.15.98:0/4230894772 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f56ac068410 msgr2=0x7f56ac06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:53.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.804+0000 7f56cebff700 1 --2- 172.21.15.98:0/4230894772 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f56ac068410 0x7f56ac06a8c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f56c80b2980 tx=0x7f56b4009040 comp rx=0 tx=0).stop 2023-03-03T08:00:53.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.804+0000 7f56cebff700 1 -- 172.21.15.98:0/4230894772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56c80b5410 msgr2=0x7f56c81529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:53.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.804+0000 7f56cebff700 1 --2- 172.21.15.98:0/4230894772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56c80b5410 0x7f56c81529d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0090b0 tx=0x7f56bc004750 comp rx=0 tx=0).stop 2023-03-03T08:00:53.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.804+0000 7f56cebff700 1 -- 172.21.15.98:0/4230894772 shutdown_connections 2023-03-03T08:00:53.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.804+0000 7f56cebff700 1 --2- 172.21.15.98:0/4230894772 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f56ac068410 0x7f56ac06a8c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:53.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.804+0000 7f56cebff700 1 --2- 172.21.15.98:0/4230894772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56c80b5410 0x7f56c81529d0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:53.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.804+0000 7f56cebff700 1 --2- 172.21.15.98:0/4230894772 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c80b8b90 0x7f56c81f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:53.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.804+0000 7f56cebff700 1 --2- 172.21.15.98:0/4230894772 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56c80b87c0 0x7f56c815eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:53.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.804+0000 7f56cebff700 1 -- 172.21.15.98:0/4230894772 >> 172.21.15.98:0/4230894772 conn(0x7f56c806aee0 msgr2=0x7f56c80b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:53.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.805+0000 7f56cebff700 1 -- 172.21.15.98:0/4230894772 shutdown_connections 2023-03-03T08:00:53.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:53.805+0000 7f56cebff700 1 -- 172.21.15.98:0/4230894772 wait complete. 2023-03-03T08:00:53.833 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T08:00:19.823120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T08:00:21.324346+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:00:53.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:00:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.082+0000 7fa87359e700 1 Processor -- start 2023-03-03T08:00:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.083+0000 7fa87359e700 1 -- start start 2023-03-03T08:00:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.084+0000 7fa87359e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa8741609b0 0x7fa874160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.084+0000 7fa87359e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa8740cf570 0x7fa87415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.084+0000 7fa87359e700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8741612c0 con 0x7fa87415db30 2023-03-03T08:00:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.084+0000 7fa87359e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8740cebb0 con 0x7fa8741609b0 2023-03-03T08:00:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.084+0000 7fa87359e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8740cd780 con 0x7fa8740cf570 2023-03-03T08:00:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.084+0000 7fa87259c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa8741609b0 0x7fa874160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.084+0000 7fa87259c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa8741609b0 0x7fa874160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59774/0 (socket says 172.21.15.98:59774) 2023-03-03T08:00:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.084+0000 7fa87259c700 1 -- 172.21.15.98:0/2102487822 learned_addr learned my addr 172.21.15.98:0/2102487822 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.085+0000 7fa86bfff700 1 --2- 172.21.15.98:0/2102487822 >> v2:172.21.15.100:3300/0 conn(0x7fa8740cf570 0x7fa87415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.085+0000 7fa87259c700 1 -- 172.21.15.98:0/2102487822 >> v2:172.21.15.100:3300/0 conn(0x7fa8740cf570 msgr2=0x7fa87415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.085+0000 7fa87259c700 1 --2- 172.21.15.98:0/2102487822 >> v2:172.21.15.100:3300/0 conn(0x7fa8740cf570 0x7fa87415d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.085+0000 7fa87259c700 1 -- 172.21.15.98:0/2102487822 >> v1:172.21.15.100:6789/0 conn(0x7fa87415db30 legacy=0x7fa8740b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.085+0000 7fa87259c700 1 -- 172.21.15.98:0/2102487822 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8740cda60 con 0x7fa8741609b0 2023-03-03T08:00:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.085+0000 7fa87259c700 1 --2- 172.21.15.98:0/2102487822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8741609b0 0x7fa874160d80 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa8740cf0c0 tx=0x7fa864009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a55e31ff00e5a19 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.086+0000 7fa871d9b700 1 -- 172.21.15.98:0/2102487822 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864011830 con 0x7fa8741609b0 2023-03-03T08:00:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.086+0000 7fa871d9b700 1 -- 172.21.15.98:0/2102487822 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa864011990 con 0x7fa8741609b0 2023-03-03T08:00:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.086+0000 7fa871d9b700 1 -- 172.21.15.98:0/2102487822 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864011d20 con 0x7fa8741609b0 2023-03-03T08:00:54.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.086+0000 7fa87359e700 1 -- 172.21.15.98:0/2102487822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8741609b0 msgr2=0x7fa874160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:54.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.086+0000 7fa87359e700 1 --2- 172.21.15.98:0/2102487822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8741609b0 0x7fa874160d80 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa8740cf0c0 tx=0x7fa864009bd0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.086+0000 7fa87359e700 1 -- 172.21.15.98:0/2102487822 shutdown_connections 2023-03-03T08:00:54.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.086+0000 7fa87359e700 1 --2- 172.21.15.98:0/2102487822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8741609b0 0x7fa874160d80 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.086+0000 7fa87359e700 1 --2- 172.21.15.98:0/2102487822 >> v2:172.21.15.100:3300/0 conn(0x7fa8740cf570 0x7fa87415d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.086+0000 7fa87359e700 1 -- 172.21.15.98:0/2102487822 >> 172.21.15.98:0/2102487822 conn(0x7fa87406b880 msgr2=0x7fa8740cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:54.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.086+0000 7fa87359e700 1 -- 172.21.15.98:0/2102487822 shutdown_connections 2023-03-03T08:00:54.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.087+0000 7fa87359e700 1 -- 172.21.15.98:0/2102487822 wait complete. 2023-03-03T08:00:54.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.088+0000 7fa87359e700 1 Processor -- start 2023-03-03T08:00:54.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.089+0000 7fa87359e700 1 -- start start 2023-03-03T08:00:54.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa87359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8740cf570 0x7fa8741485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:54.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa87359e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa87415db30 0x7fa8740b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:54.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa87359e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa8741609b0 0x7fa8740b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:54.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa87359e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8740b8dc0 con 0x7fa8740cf570 2023-03-03T08:00:54.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa87359e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8740c7340 con 0x7fa87415db30 2023-03-03T08:00:54.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa87359e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8740b8ae0 con 0x7fa8741609b0 2023-03-03T08:00:54.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa87259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8740cf570 0x7fa8741485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:54.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa87259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8740cf570 0x7fa8741485a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59778/0 (socket says 172.21.15.98:59778) 2023-03-03T08:00:54.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa87259c700 1 -- 172.21.15.98:0/1807719800 learned_addr learned my addr 172.21.15.98:0/1807719800 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:54.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa86bfff700 1 --2- 172.21.15.98:0/1807719800 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa87415db30 0x7fa8740b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:54.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa872d9d700 1 --2- 172.21.15.98:0/1807719800 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa8741609b0 0x7fa8740b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:54.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa87259c700 1 -- 172.21.15.98:0/1807719800 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa8741609b0 msgr2=0x7fa8740b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:54.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa87259c700 1 --2- 172.21.15.98:0/1807719800 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa8741609b0 0x7fa8740b4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa87259c700 1 -- 172.21.15.98:0/1807719800 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa87415db30 msgr2=0x7fa8740b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:54.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa87259c700 1 --2- 172.21.15.98:0/1807719800 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa87415db30 0x7fa8740b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.090+0000 7fa87259c700 1 -- 172.21.15.98:0/1807719800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa864009550 con 0x7fa8740cf570 2023-03-03T08:00:54.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.091+0000 7fa87259c700 1 --2- 172.21.15.98:0/1807719800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8740cf570 0x7fa8741485a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa864009130 tx=0x7fa864004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:54.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.091+0000 7fa86affd700 1 -- 172.21.15.98:0/1807719800 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864004510 con 0x7fa8740cf570 2023-03-03T08:00:54.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.091+0000 7fa86affd700 1 -- 172.21.15.98:0/1807719800 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86401e070 con 0x7fa8740cf570 2023-03-03T08:00:54.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.091+0000 7fa86affd700 1 -- 172.21.15.98:0/1807719800 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8640038a0 con 0x7fa8740cf570 2023-03-03T08:00:54.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.091+0000 7fa87359e700 1 -- 172.21.15.98:0/1807719800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87415cdf0 con 0x7fa8740cf570 2023-03-03T08:00:54.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.091+0000 7fa87359e700 1 -- 172.21.15.98:0/1807719800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8740cdd40 con 0x7fa8740cf570 2023-03-03T08:00:54.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.092+0000 7fa87359e700 1 -- 172.21.15.98:0/1807719800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87415d0d0 con 0x7fa8740cf570 2023-03-03T08:00:54.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.094+0000 7fa86affd700 1 -- 172.21.15.98:0/1807719800 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 10) v1 ==== 86575+0+0 (secure 0 0 0) 0x7fa86400f050 con 0x7fa8740cf570 2023-03-03T08:00:54.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.095+0000 7fa86affd700 1 --2- 172.21.15.98:0/1807719800 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa854068410 0x7fa85406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:54.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.095+0000 7fa86affd700 1 -- 172.21.15.98:0/1807719800 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa864086da0 con 0x7fa8740cf570 2023-03-03T08:00:54.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.095+0000 7fa86bfff700 1 --2- 172.21.15.98:0/1807719800 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa854068410 0x7fa85406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:54.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.096+0000 7fa86bfff700 1 --2- 172.21.15.98:0/1807719800 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa854068410 0x7fa85406a8c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa874160560 tx=0x7fa85c009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:54.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.099+0000 7fa86affd700 1 -- 172.21.15.98:0/1807719800 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa864053440 con 0x7fa8740cf570 2023-03-03T08:00:54.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.292+0000 7fa87359e700 1 -- 172.21.15.98:0/1807719800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa8740c6b50 con 0x7fa8740cf570 2023-03-03T08:00:54.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.293+0000 7fa86affd700 1 -- 172.21.15.98:0/1807719800 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa8740c6b50 con 0x7fa8740cf570 2023-03-03T08:00:54.294 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:00:54.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.296+0000 7fa87359e700 1 -- 172.21.15.98:0/1807719800 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa854068410 msgr2=0x7fa85406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:54.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.296+0000 7fa87359e700 1 --2- 172.21.15.98:0/1807719800 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa854068410 0x7fa85406a8c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa874160560 tx=0x7fa85c009210 comp rx=0 tx=0).stop 2023-03-03T08:00:54.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.296+0000 7fa87359e700 1 -- 172.21.15.98:0/1807719800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8740cf570 msgr2=0x7fa8741485a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:54.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.296+0000 7fa87359e700 1 --2- 172.21.15.98:0/1807719800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8740cf570 0x7fa8741485a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa864009130 tx=0x7fa864004750 comp rx=0 tx=0).stop 2023-03-03T08:00:54.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.296+0000 7fa87359e700 1 -- 172.21.15.98:0/1807719800 shutdown_connections 2023-03-03T08:00:54.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.296+0000 7fa87359e700 1 --2- 172.21.15.98:0/1807719800 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa854068410 0x7fa85406a8c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.296+0000 7fa87359e700 1 --2- 172.21.15.98:0/1807719800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8740cf570 0x7fa8741485a0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.296+0000 7fa87359e700 1 --2- 172.21.15.98:0/1807719800 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa8741609b0 0x7fa8740b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.296+0000 7fa87359e700 1 --2- 172.21.15.98:0/1807719800 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa87415db30 0x7fa8740b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.296+0000 7fa87359e700 1 -- 172.21.15.98:0/1807719800 >> 172.21.15.98:0/1807719800 conn(0x7fa87406b880 msgr2=0x7fa87415bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:54.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.296+0000 7fa87359e700 1 -- 172.21.15.98:0/1807719800 shutdown_connections 2023-03-03T08:00:54.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.296+0000 7fa87359e700 1 -- 172.21.15.98:0/1807719800 wait complete. 2023-03-03T08:00:54.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:00:54.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.575+0000 7f0bbaf9d700 1 Processor -- start 2023-03-03T08:00:54.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.576+0000 7f0bbaf9d700 1 -- start start 2023-03-03T08:00:54.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.577+0000 7f0bbaf9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bb41609b0 0x7f0bb4160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:54.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.577+0000 7f0bbaf9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0bb40cf570 0x7f0bb415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:54.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.577+0000 7f0bbaf9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb41612c0 con 0x7f0bb415db30 2023-03-03T08:00:54.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.577+0000 7f0bbaf9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb40cebb0 con 0x7f0bb40cf570 2023-03-03T08:00:54.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.577+0000 7f0bbaf9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb40cd780 con 0x7f0bb41609b0 2023-03-03T08:00:54.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.577+0000 7f0bb979a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0bb40cf570 0x7f0bb415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:54.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.577+0000 7f0bba79c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0bb415db30 0x7f0bb40b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37728/0 (socket says 172.21.15.98:37728) 2023-03-03T08:00:54.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.577+0000 7f0bba79c700 1 -- 172.21.15.98:0/122252117 learned_addr learned my addr 172.21.15.98:0/122252117 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:54.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.578+0000 7f0bb9f9b700 1 --2- 172.21.15.98:0/122252117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bb41609b0 0x7f0bb4160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:54.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.578+0000 7f0bb979a700 1 -- 172.21.15.98:0/122252117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bb41609b0 msgr2=0x7f0bb4160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:54.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.578+0000 7f0bb979a700 1 --2- 172.21.15.98:0/122252117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bb41609b0 0x7f0bb4160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.578+0000 7f0bb979a700 1 -- 172.21.15.98:0/122252117 >> v1:172.21.15.98:6789/0 conn(0x7f0bb415db30 legacy=0x7f0bb40b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:54.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.578+0000 7f0bb979a700 1 -- 172.21.15.98:0/122252117 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb40cda60 con 0x7f0bb40cf570 2023-03-03T08:00:54.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.578+0000 7f0bb8f99700 1 -- 172.21.15.98:0/122252117 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178604118 0 0) 0x7f0bb41612c0 con 0x7f0bb415db30 2023-03-03T08:00:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.578+0000 7f0bb979a700 1 --2- 172.21.15.98:0/122252117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bb40cf570 0x7f0bb415d5f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4009840 tx=0x7f0ba4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4387cb18022f48ce server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.578+0000 7f0bb8f99700 1 -- 172.21.15.98:0/122252117 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4011830 con 0x7f0bb40cf570 2023-03-03T08:00:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.579+0000 7f0bb8f99700 1 -- 172.21.15.98:0/122252117 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba4011990 con 0x7f0bb40cf570 2023-03-03T08:00:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.579+0000 7f0bb8f99700 1 -- 172.21.15.98:0/122252117 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4011d20 con 0x7f0bb40cf570 2023-03-03T08:00:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.579+0000 7f0bbaf9d700 1 -- 172.21.15.98:0/122252117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bb40cf570 msgr2=0x7f0bb415d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.579+0000 7f0bbaf9d700 1 --2- 172.21.15.98:0/122252117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bb40cf570 0x7f0bb415d5f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4009840 tx=0x7f0ba4009b50 comp rx=0 tx=0).stop 2023-03-03T08:00:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.579+0000 7f0bbaf9d700 1 -- 172.21.15.98:0/122252117 shutdown_connections 2023-03-03T08:00:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.579+0000 7f0bbaf9d700 1 --2- 172.21.15.98:0/122252117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bb41609b0 0x7f0bb4160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.579+0000 7f0bbaf9d700 1 --2- 172.21.15.98:0/122252117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bb40cf570 0x7f0bb415d5f0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.579+0000 7f0bbaf9d700 1 -- 172.21.15.98:0/122252117 >> 172.21.15.98:0/122252117 conn(0x7f0bb406b880 msgr2=0x7f0bb40cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.579+0000 7f0bbaf9d700 1 -- 172.21.15.98:0/122252117 shutdown_connections 2023-03-03T08:00:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.579+0000 7f0bbaf9d700 1 -- 172.21.15.98:0/122252117 wait complete. 2023-03-03T08:00:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.581+0000 7f0bbaf9d700 1 Processor -- start 2023-03-03T08:00:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.581+0000 7f0bbaf9d700 1 -- start start 2023-03-03T08:00:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.582+0000 7f0bbaf9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bb40cf570 0x7f0bb414c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.582+0000 7f0bbaf9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bb415db30 0x7f0bb415e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.582+0000 7f0bbaf9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bb41609b0 0x7f0bb415ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.582+0000 7f0bbaf9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb414cb20 con 0x7f0bb415db30 2023-03-03T08:00:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.582+0000 7f0bbaf9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb40c8760 con 0x7f0bb40cf570 2023-03-03T08:00:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.582+0000 7f0bbaf9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb40c8a40 con 0x7f0bb41609b0 2023-03-03T08:00:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.583+0000 7f0bb979a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bb415db30 0x7f0bb415e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.583+0000 7f0bb979a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bb415db30 0x7f0bb415e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59796/0 (socket says 172.21.15.98:59796) 2023-03-03T08:00:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.583+0000 7f0bb979a700 1 -- 172.21.15.98:0/975734969 learned_addr learned my addr 172.21.15.98:0/975734969 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.583+0000 7f0bba79c700 1 --2- 172.21.15.98:0/975734969 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bb41609b0 0x7f0bb415ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.583+0000 7f0bb9f9b700 1 --2- 172.21.15.98:0/975734969 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bb40cf570 0x7f0bb414c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.583+0000 7f0bb979a700 1 -- 172.21.15.98:0/975734969 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bb41609b0 msgr2=0x7f0bb415ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.583+0000 7f0bb979a700 1 --2- 172.21.15.98:0/975734969 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bb41609b0 0x7f0bb415ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.583+0000 7f0bb979a700 1 -- 172.21.15.98:0/975734969 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bb40cf570 msgr2=0x7f0bb414c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.583+0000 7f0bb979a700 1 --2- 172.21.15.98:0/975734969 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bb40cf570 0x7f0bb414c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.583+0000 7f0bb979a700 1 -- 172.21.15.98:0/975734969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba40094a0 con 0x7f0bb415db30 2023-03-03T08:00:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.583+0000 7f0bb9f9b700 1 --2- 172.21.15.98:0/975734969 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bb40cf570 0x7f0bb414c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.584+0000 7f0bb979a700 1 --2- 172.21.15.98:0/975734969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bb415db30 0x7f0bb415e510 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4009b90 tx=0x7f0ba4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.584+0000 7f0baa7fc700 1 -- 172.21.15.98:0/975734969 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba40036a0 con 0x7f0bb415db30 2023-03-03T08:00:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.584+0000 7f0baa7fc700 1 -- 172.21.15.98:0/975734969 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba401f070 con 0x7f0bb415db30 2023-03-03T08:00:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.584+0000 7f0baa7fc700 1 -- 172.21.15.98:0/975734969 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4003a20 con 0x7f0bb415db30 2023-03-03T08:00:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.584+0000 7f0bbaf9d700 1 -- 172.21.15.98:0/975734969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb415cdf0 con 0x7f0bb415db30 2023-03-03T08:00:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.584+0000 7f0bbaf9d700 1 -- 172.21.15.98:0/975734969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb40cdd40 con 0x7f0bb415db30 2023-03-03T08:00:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.585+0000 7f0bbaf9d700 1 -- 172.21.15.98:0/975734969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb415d0d0 con 0x7f0bb415db30 2023-03-03T08:00:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.586+0000 7f0baa7fc700 1 -- 172.21.15.98:0/975734969 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 10) v1 ==== 86575+0+0 (secure 0 0 0) 0x7f0ba400f050 con 0x7f0bb415db30 2023-03-03T08:00:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.587+0000 7f0baa7fc700 1 --2- 172.21.15.98:0/975734969 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0b98068410 0x7f0b9806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.587+0000 7f0baa7fc700 1 -- 172.21.15.98:0/975734969 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0ba4021070 con 0x7f0bb415db30 2023-03-03T08:00:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.588+0000 7f0bb9f9b700 1 --2- 172.21.15.98:0/975734969 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0b98068410 0x7f0b9806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.588+0000 7f0bb9f9b700 1 --2- 172.21.15.98:0/975734969 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0b98068410 0x7f0b9806a8c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0007a50 tx=0x7f0bb0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.591+0000 7f0baa7fc700 1 -- 172.21.15.98:0/975734969 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0ba4053f10 con 0x7f0bb415db30 2023-03-03T08:00:54.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.692+0000 7f0baa7fc700 1 -- 172.21.15.98:0/975734969 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f0ba40542e0 con 0x7f0bb415db30 2023-03-03T08:00:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.788+0000 7f0bbaf9d700 1 -- 172.21.15.98:0/975734969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f0bb4068290 con 0x7f0bb415db30 2023-03-03T08:00:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.788+0000 7f0baa7fc700 1 -- 172.21.15.98:0/975734969 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v66) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0ba4057940 con 0x7f0bb415db30 2023-03-03T08:00:54.790 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:00:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.791+0000 7f0bbaf9d700 1 -- 172.21.15.98:0/975734969 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0b98068410 msgr2=0x7f0b9806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.791+0000 7f0bbaf9d700 1 --2- 172.21.15.98:0/975734969 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0b98068410 0x7f0b9806a8c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0007a50 tx=0x7f0bb0008040 comp rx=0 tx=0).stop 2023-03-03T08:00:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.791+0000 7f0bbaf9d700 1 -- 172.21.15.98:0/975734969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bb415db30 msgr2=0x7f0bb415e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.791+0000 7f0bbaf9d700 1 --2- 172.21.15.98:0/975734969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bb415db30 0x7f0bb415e510 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4009b90 tx=0x7f0ba4004750 comp rx=0 tx=0).stop 2023-03-03T08:00:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.791+0000 7f0bbaf9d700 1 -- 172.21.15.98:0/975734969 shutdown_connections 2023-03-03T08:00:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.791+0000 7f0bbaf9d700 1 --2- 172.21.15.98:0/975734969 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0b98068410 0x7f0b9806a8c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.791+0000 7f0bbaf9d700 1 --2- 172.21.15.98:0/975734969 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bb41609b0 0x7f0bb415ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.791+0000 7f0bbaf9d700 1 --2- 172.21.15.98:0/975734969 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bb40cf570 0x7f0bb414c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.791+0000 7f0bbaf9d700 1 --2- 172.21.15.98:0/975734969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bb415db30 0x7f0bb415e510 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:54.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.791+0000 7f0bbaf9d700 1 -- 172.21.15.98:0/975734969 >> 172.21.15.98:0/975734969 conn(0x7f0bb406b880 msgr2=0x7f0bb40d1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:54.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.791+0000 7f0bbaf9d700 1 -- 172.21.15.98:0/975734969 shutdown_connections 2023-03-03T08:00:54.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:54.791+0000 7f0bbaf9d700 1 -- 172.21.15.98:0/975734969 wait complete. 2023-03-03T08:00:54.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T08:00:55.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.068+0000 7f50a53ff700 1 Processor -- start 2023-03-03T08:00:55.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.068+0000 7f50a53ff700 1 -- start start 2023-03-03T08:00:55.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.069+0000 7f50a53ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50a00b8d10 0x7f50a00cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:55.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.069+0000 7f50a53ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a00cd7d0 con 0x7f50a00b26d0 2023-03-03T08:00:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.069+0000 7f50a53ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a00b90e0 con 0x7f50a00b5310 2023-03-03T08:00:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.069+0000 7f50a53ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a00cda20 con 0x7f50a00b8d10 2023-03-03T08:00:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.070+0000 7f5097fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f50a00b26d0 0x7f50a00b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34058/0 (socket says 172.21.15.98:34058) 2023-03-03T08:00:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.070+0000 7f5097fff700 1 -- 172.21.15.98:0/4067412903 learned_addr learned my addr 172.21.15.98:0/4067412903 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.070+0000 7f509effd700 1 --2- 172.21.15.98:0/4067412903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50a00b8d10 0x7f50a00cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:55.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.070+0000 7f509e7fc700 1 -- 172.21.15.98:0/4067412903 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048950795 0 0) 0x7f50a00cd7d0 con 0x7f50a00b26d0 2023-03-03T08:00:55.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.070+0000 7f509e7fc700 1 -- 172.21.15.98:0/4067412903 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f508c0035b0 con 0x7f50a00b26d0 2023-03-03T08:00:55.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.070+0000 7f509e7fc700 1 -- 172.21.15.98:0/4067412903 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1851313681 0 0) 0x7f508c0035b0 con 0x7f50a00b26d0 2023-03-03T08:00:55.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.070+0000 7f509e7fc700 1 -- 172.21.15.98:0/4067412903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50a00b8d10 msgr2=0x7f50a00cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:55.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.070+0000 7f509e7fc700 1 --2- 172.21.15.98:0/4067412903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50a00b8d10 0x7f50a00cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.070+0000 7f509e7fc700 1 -- 172.21.15.98:0/4067412903 >> v1:172.21.15.100:6789/0 conn(0x7f50a00b5310 legacy=0x7f50a00d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:55.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.070+0000 7f509e7fc700 1 -- 172.21.15.98:0/4067412903 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a00cdd00 con 0x7f50a00b26d0 2023-03-03T08:00:55.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.071+0000 7f509e7fc700 1 -- 172.21.15.98:0/4067412903 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f5088001670 con 0x7f50a00b26d0 2023-03-03T08:00:55.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.071+0000 7f509e7fc700 1 -- 172.21.15.98:0/4067412903 >> v1:172.21.15.98:6789/0 conn(0x7f50a00b26d0 legacy=0x7f50a00b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:55.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.071+0000 7f509e7fc700 1 --2- 172.21.15.98:0/4067412903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50a00b8d10 0x7f508c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:55.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.071+0000 7f509e7fc700 1 --2- 172.21.15.98:0/4067412903 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f508c004000 0x7f508c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:55.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.071+0000 7f509e7fc700 1 --2- 172.21.15.98:0/4067412903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f508c004d90 0x7f508c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:55.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.071+0000 7f509e7fc700 1 -- 172.21.15.98:0/4067412903 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a00cda20 con 0x7f508c004d90 2023-03-03T08:00:55.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.071+0000 7f509e7fc700 1 -- 172.21.15.98:0/4067412903 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c00bbd0 con 0x7f508c004000 2023-03-03T08:00:55.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.071+0000 7f509e7fc700 1 -- 172.21.15.98:0/4067412903 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f508c00bd90 con 0x7f50a00b8d10 2023-03-03T08:00:55.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.071+0000 7f5097fff700 1 --2- 172.21.15.98:0/4067412903 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f508c004000 0x7f508c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:55.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.071+0000 7f509effd700 1 --2- 172.21.15.98:0/4067412903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50a00b8d10 0x7f508c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:55.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.071+0000 7f509f7fe700 1 --2- 172.21.15.98:0/4067412903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f508c004d90 0x7f508c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:55.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.072+0000 7f5097fff700 1 -- 172.21.15.98:0/4067412903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50a00b8d10 msgr2=0x7f508c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:55.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.072+0000 7f5097fff700 1 --2- 172.21.15.98:0/4067412903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50a00b8d10 0x7f508c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.072+0000 7f5097fff700 1 -- 172.21.15.98:0/4067412903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f508c004d90 msgr2=0x7f508c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:55.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.072+0000 7f5097fff700 1 --2- 172.21.15.98:0/4067412903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f508c004d90 0x7f508c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.072+0000 7f5097fff700 1 -- 172.21.15.98:0/4067412903 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c0035b0 con 0x7f508c004000 2023-03-03T08:00:55.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.072+0000 7f509effd700 1 --2- 172.21.15.98:0/4067412903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50a00b8d10 0x7f508c003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:00:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.072+0000 7f5097fff700 1 --2- 172.21.15.98:0/4067412903 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f508c004000 0x7f508c004850 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f508800c160 tx=0x7f508c005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d07c6bafef1d48b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.072+0000 7f509e7fc700 1 -- 172.21.15.98:0/4067412903 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088007b70 con 0x7f508c004000 2023-03-03T08:00:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.072+0000 7f509e7fc700 1 -- 172.21.15.98:0/4067412903 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508800cd20 con 0x7f508c004000 2023-03-03T08:00:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.072+0000 7f509e7fc700 1 -- 172.21.15.98:0/4067412903 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088005ba0 con 0x7f508c004000 2023-03-03T08:00:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.073+0000 7f50a53ff700 1 -- 172.21.15.98:0/4067412903 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f508c004000 msgr2=0x7f508c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.073+0000 7f50a53ff700 1 --2- 172.21.15.98:0/4067412903 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f508c004000 0x7f508c004850 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f508800c160 tx=0x7f508c005880 comp rx=0 tx=0).stop 2023-03-03T08:00:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.073+0000 7f509f7fe700 1 -- 172.21.15.98:0/4067412903 reap_dead start 2023-03-03T08:00:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.073+0000 7f50a53ff700 1 -- 172.21.15.98:0/4067412903 shutdown_connections 2023-03-03T08:00:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.073+0000 7f50a53ff700 1 -- 172.21.15.98:0/4067412903 >> 172.21.15.98:0/4067412903 conn(0x7f50a0073e70 msgr2=0x7f50a006ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.073+0000 7f50a53ff700 1 -- 172.21.15.98:0/4067412903 shutdown_connections 2023-03-03T08:00:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.073+0000 7f50a53ff700 1 -- 172.21.15.98:0/4067412903 wait complete. 2023-03-03T08:00:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.075+0000 7f50a53ff700 1 Processor -- start 2023-03-03T08:00:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.076+0000 7f50a53ff700 1 -- start start 2023-03-03T08:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f50a53ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f508c004000 0x7f50a0154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f50a53ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50a01f57c0 0x7f50a01f5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f50a53ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50a00ca250 0x7f50a00ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f50a53ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a00cac00 con 0x7f50a01f57c0 2023-03-03T08:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f50a53ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a01f6330 con 0x7f508c004000 2023-03-03T08:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f50a53ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a01f6610 con 0x7f50a00ca250 2023-03-03T08:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f5097fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50a01f57c0 0x7f50a01f5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f509effd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f508c004000 0x7f50a0154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f5097fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50a01f57c0 0x7f50a01f5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47992/0 (socket says 172.21.15.98:47992) 2023-03-03T08:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f5097fff700 1 -- 172.21.15.98:0/1728154117 learned_addr learned my addr 172.21.15.98:0/1728154117 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f509f7fe700 1 --2- 172.21.15.98:0/1728154117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50a00ca250 0x7f50a00ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f5097fff700 1 -- 172.21.15.98:0/1728154117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50a00ca250 msgr2=0x7f50a00ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f5097fff700 1 --2- 172.21.15.98:0/1728154117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50a00ca250 0x7f50a00ca6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f5097fff700 1 -- 172.21.15.98:0/1728154117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f508c004000 msgr2=0x7f50a0154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f5097fff700 1 --2- 172.21.15.98:0/1728154117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f508c004000 0x7f50a0154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.077+0000 7f5097fff700 1 -- 172.21.15.98:0/1728154117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a00cdfe0 con 0x7f50a01f57c0 2023-03-03T08:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.078+0000 7f5097fff700 1 --2- 172.21.15.98:0/1728154117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50a01f57c0 0x7f50a01f5c30 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f508800c750 tx=0x7f5088001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.078+0000 7f50977fe700 1 -- 172.21.15.98:0/1728154117 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088004c80 con 0x7f50a01f57c0 2023-03-03T08:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.078+0000 7f50977fe700 1 -- 172.21.15.98:0/1728154117 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5088004e20 con 0x7f50a01f57c0 2023-03-03T08:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.078+0000 7f50977fe700 1 -- 172.21.15.98:0/1728154117 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088007310 con 0x7f50a01f57c0 2023-03-03T08:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.078+0000 7f50a53ff700 1 -- 172.21.15.98:0/1728154117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a00d0e80 con 0x7f50a01f57c0 2023-03-03T08:00:55.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.078+0000 7f50a53ff700 1 -- 172.21.15.98:0/1728154117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a00b5fd0 con 0x7f50a01f57c0 2023-03-03T08:00:55.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.080+0000 7f50a53ff700 1 -- 172.21.15.98:0/1728154117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a004e6e0 con 0x7f50a01f57c0 2023-03-03T08:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.081+0000 7f50977fe700 1 -- 172.21.15.98:0/1728154117 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f5088010070 con 0x7f50a01f57c0 2023-03-03T08:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.082+0000 7f50977fe700 1 --2- 172.21.15.98:0/1728154117 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50800684a0 0x7f508006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.082+0000 7f50977fe700 1 -- 172.21.15.98:0/1728154117 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f50880864e0 con 0x7f50a01f57c0 2023-03-03T08:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.082+0000 7f509effd700 1 --2- 172.21.15.98:0/1728154117 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50800684a0 0x7f508006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.083+0000 7f509effd700 1 --2- 172.21.15.98:0/1728154117 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50800684a0 0x7f508006a950 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f50a00cb6b0 tx=0x7f5090009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:55.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.087+0000 7f50977fe700 1 -- 172.21.15.98:0/1728154117 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5088052ba0 con 0x7f50a01f57c0 2023-03-03T08:00:55.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.281+0000 7f50a53ff700 1 -- 172.21.15.98:0/1728154117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f50a00cf7f0 con 0x7f50a01f57c0 2023-03-03T08:00:55.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.282+0000 7f50977fe700 1 -- 172.21.15.98:0/1728154117 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v66) v1 ==== 104+0+11 (secure 0 0 0) 0x7f50880561c0 con 0x7f50a01f57c0 2023-03-03T08:00:55.283 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T08:00:55.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.284+0000 7f50a53ff700 1 -- 172.21.15.98:0/1728154117 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50800684a0 msgr2=0x7f508006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:55.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.284+0000 7f50a53ff700 1 --2- 172.21.15.98:0/1728154117 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50800684a0 0x7f508006a950 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f50a00cb6b0 tx=0x7f5090009210 comp rx=0 tx=0).stop 2023-03-03T08:00:55.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.284+0000 7f50a53ff700 1 -- 172.21.15.98:0/1728154117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50a01f57c0 msgr2=0x7f50a01f5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:55.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.284+0000 7f50a53ff700 1 --2- 172.21.15.98:0/1728154117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50a01f57c0 0x7f50a01f5c30 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f508800c750 tx=0x7f5088001930 comp rx=0 tx=0).stop 2023-03-03T08:00:55.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.285+0000 7f50a53ff700 1 -- 172.21.15.98:0/1728154117 shutdown_connections 2023-03-03T08:00:55.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.285+0000 7f50a53ff700 1 --2- 172.21.15.98:0/1728154117 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f50800684a0 0x7f508006a950 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.285+0000 7f50a53ff700 1 --2- 172.21.15.98:0/1728154117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f50a00ca250 0x7f50a00ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.285+0000 7f50a53ff700 1 --2- 172.21.15.98:0/1728154117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f508c004000 0x7f50a0154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.285+0000 7f50a53ff700 1 --2- 172.21.15.98:0/1728154117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50a01f57c0 0x7f50a01f5c30 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.285+0000 7f50a53ff700 1 -- 172.21.15.98:0/1728154117 >> 172.21.15.98:0/1728154117 conn(0x7f50a0073e70 msgr2=0x7f50a00cfff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:55.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.285+0000 7f50a53ff700 1 -- 172.21.15.98:0/1728154117 shutdown_connections 2023-03-03T08:00:55.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.285+0000 7f50a53ff700 1 -- 172.21.15.98:0/1728154117 wait complete. 2023-03-03T08:00:55.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T08:00:55.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.563+0000 7f8b7fbff700 1 Processor -- start 2023-03-03T08:00:55.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.563+0000 7f8b7fbff700 1 -- start start 2023-03-03T08:00:55.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.564+0000 7f8b7fbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b7806b4b0 0x7f8b780cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:55.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.564+0000 7f8b7fbff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b780cf980 con 0x7f8b780d0270 2023-03-03T08:00:55.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.565+0000 7f8b7fbff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b780d1800 con 0x7f8b7815ca70 2023-03-03T08:00:55.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.565+0000 7f8b7fbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b780cd5b0 con 0x7f8b7806b4b0 2023-03-03T08:00:55.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.565+0000 7f8b7d19a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8b780d0270 0x7f8b780d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34072/0 (socket says 172.21.15.98:34072) 2023-03-03T08:00:55.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.565+0000 7f8b7d19a700 1 -- 172.21.15.98:0/697061502 learned_addr learned my addr 172.21.15.98:0/697061502 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:55.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.565+0000 7f8b7e19c700 1 --2- 172.21.15.98:0/697061502 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b7806b4b0 0x7f8b780cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:55.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.565+0000 7f8b7c999700 1 -- 172.21.15.98:0/697061502 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839289318 0 0) 0x7f8b780cf980 con 0x7f8b780d0270 2023-03-03T08:00:55.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.565+0000 7f8b7c999700 1 -- 172.21.15.98:0/697061502 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b640035b0 con 0x7f8b780d0270 2023-03-03T08:00:55.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.566+0000 7f8b7e19c700 1 -- 172.21.15.98:0/697061502 >> v1:172.21.15.100:6789/0 conn(0x7f8b7815ca70 legacy=0x7f8b780cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:55.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.566+0000 7f8b7e19c700 1 -- 172.21.15.98:0/697061502 >> v1:172.21.15.98:6789/0 conn(0x7f8b780d0270 legacy=0x7f8b780d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:55.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.566+0000 7f8b7e19c700 1 -- 172.21.15.98:0/697061502 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b780cd900 con 0x7f8b7806b4b0 2023-03-03T08:00:55.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.566+0000 7f8b7e19c700 1 --2- 172.21.15.98:0/697061502 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b7806b4b0 0x7f8b780cf440 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009dc0 tx=0x7f8b7000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfefc1d36b80b7d4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:55.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.566+0000 7f8b7c999700 1 -- 172.21.15.98:0/697061502 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7000e070 con 0x7f8b7806b4b0 2023-03-03T08:00:55.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.566+0000 7f8b7c999700 1 -- 172.21.15.98:0/697061502 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7000bb20 con 0x7f8b7806b4b0 2023-03-03T08:00:55.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.566+0000 7f8b7c999700 1 -- 172.21.15.98:0/697061502 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b70012430 con 0x7f8b7806b4b0 2023-03-03T08:00:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.567+0000 7f8b7fbff700 1 -- 172.21.15.98:0/697061502 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b7806b4b0 msgr2=0x7f8b780cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.567+0000 7f8b7fbff700 1 --2- 172.21.15.98:0/697061502 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b7806b4b0 0x7f8b780cf440 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009dc0 tx=0x7f8b7000b330 comp rx=0 tx=0).stop 2023-03-03T08:00:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.567+0000 7f8b7fbff700 1 -- 172.21.15.98:0/697061502 shutdown_connections 2023-03-03T08:00:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.567+0000 7f8b7fbff700 1 --2- 172.21.15.98:0/697061502 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b7806b4b0 0x7f8b780cf440 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.567+0000 7f8b7fbff700 1 -- 172.21.15.98:0/697061502 >> 172.21.15.98:0/697061502 conn(0x7f8b78073e70 msgr2=0x7f8b7806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:55.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.567+0000 7f8b7fbff700 1 -- 172.21.15.98:0/697061502 shutdown_connections 2023-03-03T08:00:55.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.567+0000 7f8b7fbff700 1 -- 172.21.15.98:0/697061502 wait complete. 2023-03-03T08:00:55.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.569+0000 7f8b7fbff700 1 Processor -- start 2023-03-03T08:00:55.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.569+0000 7f8b7fbff700 1 -- start start 2023-03-03T08:00:55.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.570+0000 7f8b7fbff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b7806b4b0 0x7f8b78158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:55.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.570+0000 7f8b7fbff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b780d0270 0x7f8b781f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:55.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.570+0000 7f8b7fbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b7815ca70 0x7f8b7815d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:55.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.570+0000 7f8b7fbff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b780d3560 con 0x7f8b7806b4b0 2023-03-03T08:00:55.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.570+0000 7f8b7fbff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7815dbd0 con 0x7f8b780d0270 2023-03-03T08:00:55.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.570+0000 7f8b7fbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b781f24e0 con 0x7f8b7815ca70 2023-03-03T08:00:55.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.570+0000 7f8b7d99b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b7806b4b0 0x7f8b78158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:55.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.570+0000 7f8b7d99b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b7806b4b0 0x7f8b78158c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48004/0 (socket says 172.21.15.98:48004) 2023-03-03T08:00:55.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.571+0000 7f8b7d99b700 1 -- 172.21.15.98:0/3310663522 learned_addr learned my addr 172.21.15.98:0/3310663522 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:55.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.571+0000 7f8b7e19c700 1 --2- 172.21.15.98:0/3310663522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b7815ca70 0x7f8b7815d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:55.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.571+0000 7f8b7d19a700 1 --2- 172.21.15.98:0/3310663522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b780d0270 0x7f8b781f2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:55.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.571+0000 7f8b7d99b700 1 -- 172.21.15.98:0/3310663522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b7815ca70 msgr2=0x7f8b7815d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:55.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.571+0000 7f8b7d99b700 1 --2- 172.21.15.98:0/3310663522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b7815ca70 0x7f8b7815d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.571+0000 7f8b7d99b700 1 -- 172.21.15.98:0/3310663522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b780d0270 msgr2=0x7f8b781f2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:55.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.571+0000 7f8b7d99b700 1 --2- 172.21.15.98:0/3310663522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b780d0270 0x7f8b781f2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.571+0000 7f8b7d99b700 1 -- 172.21.15.98:0/3310663522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b70009a70 con 0x7f8b7806b4b0 2023-03-03T08:00:55.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.571+0000 7f8b7d19a700 1 --2- 172.21.15.98:0/3310663522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b780d0270 0x7f8b781f2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:55.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.571+0000 7f8b7d99b700 1 --2- 172.21.15.98:0/3310663522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b7806b4b0 0x7f8b78158c30 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8b7400c700 tx=0x7f8b7400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:55.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.572+0000 7f8b6e7fc700 1 -- 172.21.15.98:0/3310663522 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b740052d0 con 0x7f8b7806b4b0 2023-03-03T08:00:55.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.572+0000 7f8b6e7fc700 1 -- 172.21.15.98:0/3310663522 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b74005430 con 0x7f8b7806b4b0 2023-03-03T08:00:55.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.572+0000 7f8b6e7fc700 1 -- 172.21.15.98:0/3310663522 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74003880 con 0x7f8b7806b4b0 2023-03-03T08:00:55.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.572+0000 7f8b7fbff700 1 -- 172.21.15.98:0/3310663522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b780d1aa0 con 0x7f8b7806b4b0 2023-03-03T08:00:55.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.572+0000 7f8b7fbff700 1 -- 172.21.15.98:0/3310663522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b780cdbe0 con 0x7f8b7806b4b0 2023-03-03T08:00:55.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.573+0000 7f8b7fbff700 1 -- 172.21.15.98:0/3310663522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b7815ce40 con 0x7f8b7806b4b0 2023-03-03T08:00:55.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.574+0000 7f8b6e7fc700 1 -- 172.21.15.98:0/3310663522 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f8b74019030 con 0x7f8b7806b4b0 2023-03-03T08:00:55.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.575+0000 7f8b6e7fc700 1 --2- 172.21.15.98:0/3310663522 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8b5c0684a0 0x7f8b5c06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:55.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.576+0000 7f8b6e7fc700 1 -- 172.21.15.98:0/3310663522 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8b74085240 con 0x7f8b7806b4b0 2023-03-03T08:00:55.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.576+0000 7f8b7d19a700 1 --2- 172.21.15.98:0/3310663522 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8b5c0684a0 0x7f8b5c06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:55.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.576+0000 7f8b7d19a700 1 --2- 172.21.15.98:0/3310663522 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8b5c0684a0 0x7f8b5c06a950 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8b780ca1a0 tx=0x7f8b68009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:55.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.579+0000 7f8b6e7fc700 1 -- 172.21.15.98:0/3310663522 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b7404b5e0 con 0x7f8b7806b4b0 2023-03-03T08:00:55.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.777+0000 7f8b7fbff700 1 -- 172.21.15.98:0/3310663522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8b7804e6e0 con 0x7f8b7806b4b0 2023-03-03T08:00:55.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.777+0000 7f8b6e7fc700 1 -- 172.21.15.98:0/3310663522 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v66) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8b740518b0 con 0x7f8b7806b4b0 2023-03-03T08:00:55.779 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T08:00:55.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.780+0000 7f8b7fbff700 1 -- 172.21.15.98:0/3310663522 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8b5c0684a0 msgr2=0x7f8b5c06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:55.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.780+0000 7f8b7fbff700 1 --2- 172.21.15.98:0/3310663522 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8b5c0684a0 0x7f8b5c06a950 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8b780ca1a0 tx=0x7f8b68009040 comp rx=0 tx=0).stop 2023-03-03T08:00:55.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.780+0000 7f8b7fbff700 1 -- 172.21.15.98:0/3310663522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b7806b4b0 msgr2=0x7f8b78158c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:55.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.780+0000 7f8b7fbff700 1 --2- 172.21.15.98:0/3310663522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b7806b4b0 0x7f8b78158c30 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8b7400c700 tx=0x7f8b7400ca10 comp rx=0 tx=0).stop 2023-03-03T08:00:55.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.780+0000 7f8b7fbff700 1 -- 172.21.15.98:0/3310663522 shutdown_connections 2023-03-03T08:00:55.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.780+0000 7f8b7fbff700 1 --2- 172.21.15.98:0/3310663522 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f8b5c0684a0 0x7f8b5c06a950 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.780+0000 7f8b7fbff700 1 --2- 172.21.15.98:0/3310663522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b7806b4b0 0x7f8b78158c30 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.780+0000 7f8b7fbff700 1 --2- 172.21.15.98:0/3310663522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b7815ca70 0x7f8b7815d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.780+0000 7f8b7fbff700 1 --2- 172.21.15.98:0/3310663522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b780d0270 0x7f8b781f2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:55.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.780+0000 7f8b7fbff700 1 -- 172.21.15.98:0/3310663522 >> 172.21.15.98:0/3310663522 conn(0x7f8b78073e70 msgr2=0x7f8b780cf400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:55.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.780+0000 7f8b7fbff700 1 -- 172.21.15.98:0/3310663522 shutdown_connections 2023-03-03T08:00:55.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:55.780+0000 7f8b7fbff700 1 -- 172.21.15.98:0/3310663522 wait complete. 2023-03-03T08:00:55.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T08:00:56.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.054+0000 7fb1e99ff700 1 Processor -- start 2023-03-03T08:00:56.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.054+0000 7fb1e99ff700 1 -- start start 2023-03-03T08:00:56.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.055+0000 7fb1e99ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e40b26d0 0x7fb1e40b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:56.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.055+0000 7fb1e99ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb1e40b5140 0x7fb1e40d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:56.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.055+0000 7fb1e99ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e40cd7d0 con 0x7fb1e40b8d10 2023-03-03T08:00:56.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.055+0000 7fb1e99ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e40b55c0 con 0x7fb1e40b5140 2023-03-03T08:00:56.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.055+0000 7fb1e99ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e40b90e0 con 0x7fb1e40b26d0 2023-03-03T08:00:56.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.056+0000 7fb1e37fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb1e40b5140 0x7fb1e40d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:56.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.056+0000 7fb1e37fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb1e40b5140 0x7fb1e40d13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48008/0 (socket says 172.21.15.98:48008) 2023-03-03T08:00:56.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.056+0000 7fb1e37fe700 1 -- 172.21.15.98:0/1359395069 learned_addr learned my addr 172.21.15.98:0/1359395069 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.056+0000 7fb1e27fc700 1 --2- 172.21.15.98:0/1359395069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e40b26d0 0x7fb1e40b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.056+0000 7fb1e37fe700 1 -- 172.21.15.98:0/1359395069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e40b26d0 msgr2=0x7fb1e40b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.056+0000 7fb1e37fe700 1 --2- 172.21.15.98:0/1359395069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e40b26d0 0x7fb1e40b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.056+0000 7fb1e37fe700 1 -- 172.21.15.98:0/1359395069 >> v1:172.21.15.100:6789/0 conn(0x7fb1e40b8d10 legacy=0x7fb1e40cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.056+0000 7fb1e37fe700 1 -- 172.21.15.98:0/1359395069 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e40cda20 con 0x7fb1e40b5140 2023-03-03T08:00:56.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.057+0000 7fb1e37fe700 1 --2- 172.21.15.98:0/1359395069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e40b5140 0x7fb1e40d13a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc009e50 tx=0x7fb1dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b2879c2b0417ed server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:56.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.057+0000 7fb1e1ffb700 1 -- 172.21.15.98:0/1359395069 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc00e070 con 0x7fb1e40b5140 2023-03-03T08:00:56.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.057+0000 7fb1e1ffb700 1 -- 172.21.15.98:0/1359395069 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1dc00b910 con 0x7fb1e40b5140 2023-03-03T08:00:56.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.057+0000 7fb1e1ffb700 1 -- 172.21.15.98:0/1359395069 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc00bcc0 con 0x7fb1e40b5140 2023-03-03T08:00:56.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.057+0000 7fb1e99ff700 1 -- 172.21.15.98:0/1359395069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e40b5140 msgr2=0x7fb1e40d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:56.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.057+0000 7fb1e99ff700 1 --2- 172.21.15.98:0/1359395069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e40b5140 0x7fb1e40d13a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc009e50 tx=0x7fb1dc00b330 comp rx=0 tx=0).stop 2023-03-03T08:00:56.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.057+0000 7fb1e99ff700 1 -- 172.21.15.98:0/1359395069 shutdown_connections 2023-03-03T08:00:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.057+0000 7fb1e99ff700 1 --2- 172.21.15.98:0/1359395069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e40b5140 0x7fb1e40d13a0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.057+0000 7fb1e99ff700 1 --2- 172.21.15.98:0/1359395069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e40b26d0 0x7fb1e40b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.057+0000 7fb1e99ff700 1 -- 172.21.15.98:0/1359395069 >> 172.21.15.98:0/1359395069 conn(0x7fb1e4073e70 msgr2=0x7fb1e406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.058+0000 7fb1e99ff700 1 -- 172.21.15.98:0/1359395069 shutdown_connections 2023-03-03T08:00:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.058+0000 7fb1e99ff700 1 -- 172.21.15.98:0/1359395069 wait complete. 2023-03-03T08:00:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.060+0000 7fb1e99ff700 1 Processor -- start 2023-03-03T08:00:56.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.060+0000 7fb1e99ff700 1 -- start start 2023-03-03T08:00:56.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.061+0000 7fb1e99ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e40b26d0 0x7fb1e41f5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:56.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.061+0000 7fb1e99ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1e40b5140 0x7fb1e40b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:56.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.061+0000 7fb1e99ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e40b8d10 0x7fb1e40b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:56.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.061+0000 7fb1e99ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e415dde0 con 0x7fb1e40b8d10 2023-03-03T08:00:56.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.061+0000 7fb1e99ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e415e0c0 con 0x7fb1e40b5140 2023-03-03T08:00:56.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.061+0000 7fb1e99ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e415e3a0 con 0x7fb1e40b26d0 2023-03-03T08:00:56.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.061+0000 7fb1e37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e40b8d10 0x7fb1e40b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:56.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.062+0000 7fb1e37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e40b8d10 0x7fb1e40b48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48014/0 (socket says 172.21.15.98:48014) 2023-03-03T08:00:56.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.062+0000 7fb1e37fe700 1 -- 172.21.15.98:0/3844910304 learned_addr learned my addr 172.21.15.98:0/3844910304 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:56.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.062+0000 7fb1e2ffd700 1 --2- 172.21.15.98:0/3844910304 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e40b26d0 0x7fb1e41f5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:56.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.062+0000 7fb1e27fc700 1 --2- 172.21.15.98:0/3844910304 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1e40b5140 0x7fb1e40b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:56.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.062+0000 7fb1e37fe700 1 -- 172.21.15.98:0/3844910304 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e40b26d0 msgr2=0x7fb1e41f5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:56.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.062+0000 7fb1e37fe700 1 --2- 172.21.15.98:0/3844910304 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e40b26d0 0x7fb1e41f5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:56.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.062+0000 7fb1e37fe700 1 -- 172.21.15.98:0/3844910304 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1e40b5140 msgr2=0x7fb1e40b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:56.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.062+0000 7fb1e37fe700 1 --2- 172.21.15.98:0/3844910304 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1e40b5140 0x7fb1e40b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:56.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.062+0000 7fb1e37fe700 1 -- 172.21.15.98:0/3844910304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc009b30 con 0x7fb1e40b8d10 2023-03-03T08:00:56.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.062+0000 7fb1e37fe700 1 --2- 172.21.15.98:0/3844910304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e40b8d10 0x7fb1e40b48a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb1e408c2c0 tx=0x7fb1dc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:56.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.063+0000 7fb1cbfff700 1 -- 172.21.15.98:0/3844910304 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc004840 con 0x7fb1e40b8d10 2023-03-03T08:00:56.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.063+0000 7fb1cbfff700 1 -- 172.21.15.98:0/3844910304 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1dc0099b0 con 0x7fb1e40b8d10 2023-03-03T08:00:56.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.063+0000 7fb1cbfff700 1 -- 172.21.15.98:0/3844910304 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc01c680 con 0x7fb1e40b8d10 2023-03-03T08:00:56.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.063+0000 7fb1e99ff700 1 -- 172.21.15.98:0/3844910304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e40d0e80 con 0x7fb1e40b8d10 2023-03-03T08:00:56.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.063+0000 7fb1e99ff700 1 -- 172.21.15.98:0/3844910304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e40cdd00 con 0x7fb1e40b8d10 2023-03-03T08:00:56.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.064+0000 7fb1e99ff700 1 -- 172.21.15.98:0/3844910304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e40cdfe0 con 0x7fb1e40b8d10 2023-03-03T08:00:56.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.065+0000 7fb1cbfff700 1 -- 172.21.15.98:0/3844910304 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fb1dc02c070 con 0x7fb1e40b8d10 2023-03-03T08:00:56.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.066+0000 7fb1cbfff700 1 --2- 172.21.15.98:0/3844910304 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb1d00684f0 0x7fb1d006a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:56.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.067+0000 7fb1cbfff700 1 -- 172.21.15.98:0/3844910304 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb1dc0878d0 con 0x7fb1e40b8d10 2023-03-03T08:00:56.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.067+0000 7fb1e2ffd700 1 --2- 172.21.15.98:0/3844910304 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb1d00684f0 0x7fb1d006a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:56.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.068+0000 7fb1e2ffd700 1 --2- 172.21.15.98:0/3844910304 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb1d00684f0 0x7fb1d006a9a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4007fd0 tx=0x7fb1d4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:56.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.071+0000 7fb1cbfff700 1 -- 172.21.15.98:0/3844910304 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb1dc054010 con 0x7fb1e40b8d10 2023-03-03T08:00:56.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.280+0000 7fb1e99ff700 1 -- 172.21.15.98:0/3844910304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb1e404e6e0 con 0x7fb1e40b8d10 2023-03-03T08:00:56.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.281+0000 7fb1cbfff700 1 -- 172.21.15.98:0/3844910304 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v66) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb1dc057630 con 0x7fb1e40b8d10 2023-03-03T08:00:56.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.283+0000 7fb1e99ff700 1 -- 172.21.15.98:0/3844910304 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb1d00684f0 msgr2=0x7fb1d006a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:56.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.283+0000 7fb1e99ff700 1 --2- 172.21.15.98:0/3844910304 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb1d00684f0 0x7fb1d006a9a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4007fd0 tx=0x7fb1d4009040 comp rx=0 tx=0).stop 2023-03-03T08:00:56.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.283+0000 7fb1e99ff700 1 -- 172.21.15.98:0/3844910304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e40b8d10 msgr2=0x7fb1e40b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:56.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.283+0000 7fb1e99ff700 1 --2- 172.21.15.98:0/3844910304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e40b8d10 0x7fb1e40b48a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb1e408c2c0 tx=0x7fb1dc004ad0 comp rx=0 tx=0).stop 2023-03-03T08:00:56.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.283+0000 7fb1e99ff700 1 -- 172.21.15.98:0/3844910304 shutdown_connections 2023-03-03T08:00:56.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.284+0000 7fb1e99ff700 1 --2- 172.21.15.98:0/3844910304 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb1d00684f0 0x7fb1d006a9a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:56.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.284+0000 7fb1e99ff700 1 --2- 172.21.15.98:0/3844910304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e40b8d10 0x7fb1e40b48a0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:56.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.284+0000 7fb1e99ff700 1 --2- 172.21.15.98:0/3844910304 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e40b26d0 0x7fb1e41f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:56.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.284+0000 7fb1e99ff700 1 --2- 172.21.15.98:0/3844910304 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1e40b5140 0x7fb1e40b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:56.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.284+0000 7fb1e99ff700 1 -- 172.21.15.98:0/3844910304 >> 172.21.15.98:0/3844910304 conn(0x7fb1e4073e70 msgr2=0x7fb1e406ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:56.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.284+0000 7fb1e99ff700 1 -- 172.21.15.98:0/3844910304 shutdown_connections 2023-03-03T08:00:56.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.284+0000 7fb1e99ff700 1 -- 172.21.15.98:0/3844910304 wait complete. 2023-03-03T08:00:56.287 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T08:00:56.312 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2023-03-03T08:00:56.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:56.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-03-03T08:00:56.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[135583]: fuse finished with error 0 and tester_r 0 2023-03-03T08:00:56.417 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T08:00:56.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:56.418 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-03-03T08:00:56.552 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:00:56.553 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:00:56.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:00:56.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.869+0000 7f3d9dfff700 1 Processor -- start 2023-03-03T08:00:56.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.869+0000 7f3d9dfff700 1 -- start start 2023-03-03T08:00:56.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.870+0000 7f3d9dfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3d980b87c0 0x7f3d980cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:56.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.870+0000 7f3d9dfff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3d980b8b90 0x7f3d980b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:56.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.871+0000 7f3d9dfff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d980d0ef0 con 0x7f3d980b5240 2023-03-03T08:00:56.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.871+0000 7f3d9dfff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d980d0600 con 0x7f3d980b87c0 2023-03-03T08:00:56.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.871+0000 7f3d9dfff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d980cd970 con 0x7f3d980b8b90 2023-03-03T08:00:56.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.871+0000 7f3d977fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3d980b87c0 0x7f3d980cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:56.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.871+0000 7f3d97fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3d980b5240 0x7f3d981605b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34088/0 (socket says 172.21.15.98:34088) 2023-03-03T08:00:56.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.871+0000 7f3d97fff700 1 -- 172.21.15.98:0/2381515104 learned_addr learned my addr 172.21.15.98:0/2381515104 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:56.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.871+0000 7f3d8ffff700 1 --2- 172.21.15.98:0/2381515104 >> v2:172.21.15.100:3300/0 conn(0x7f3d980b8b90 0x7f3d980b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:56.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.871+0000 7f3d977fe700 1 -- 172.21.15.98:0/2381515104 >> v2:172.21.15.100:3300/0 conn(0x7f3d980b8b90 msgr2=0x7f3d980b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:56.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.871+0000 7f3d977fe700 1 --2- 172.21.15.98:0/2381515104 >> v2:172.21.15.100:3300/0 conn(0x7f3d980b8b90 0x7f3d980b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:56.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.871+0000 7f3d977fe700 1 -- 172.21.15.98:0/2381515104 >> v1:172.21.15.98:6789/0 conn(0x7f3d980b5240 legacy=0x7f3d981605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:56.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.871+0000 7f3d977fe700 1 -- 172.21.15.98:0/2381515104 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d980cdc50 con 0x7f3d980b87c0 2023-03-03T08:00:56.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.871+0000 7f3d8ffff700 1 --2- 172.21.15.98:0/2381515104 >> v2:172.21.15.100:3300/0 conn(0x7f3d980b8b90 0x7f3d980b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:56.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.871+0000 7f3d96ffd700 1 -- 172.21.15.98:0/2381515104 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773666540 0 0) 0x7f3d980d0ef0 con 0x7f3d980b5240 2023-03-03T08:00:56.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.872+0000 7f3d977fe700 1 --2- 172.21.15.98:0/2381515104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d980b87c0 0x7f3d980cffe0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f3d9808c240 tx=0x7f3d88009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba87ee977a6ec7d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:56.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.872+0000 7f3d96ffd700 1 -- 172.21.15.98:0/2381515104 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d88011830 con 0x7f3d980b87c0 2023-03-03T08:00:56.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.872+0000 7f3d96ffd700 1 -- 172.21.15.98:0/2381515104 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d88011990 con 0x7f3d980b87c0 2023-03-03T08:00:56.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.872+0000 7f3d96ffd700 1 -- 172.21.15.98:0/2381515104 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d88011d20 con 0x7f3d980b87c0 2023-03-03T08:00:56.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.873+0000 7f3d9dfff700 1 -- 172.21.15.98:0/2381515104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d980b87c0 msgr2=0x7f3d980cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:56.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.873+0000 7f3d9dfff700 1 --2- 172.21.15.98:0/2381515104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d980b87c0 0x7f3d980cffe0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f3d9808c240 tx=0x7f3d88009b20 comp rx=0 tx=0).stop 2023-03-03T08:00:56.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.873+0000 7f3d9dfff700 1 -- 172.21.15.98:0/2381515104 shutdown_connections 2023-03-03T08:00:56.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.873+0000 7f3d9dfff700 1 --2- 172.21.15.98:0/2381515104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d980b87c0 0x7f3d980cffe0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:56.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.873+0000 7f3d9dfff700 1 --2- 172.21.15.98:0/2381515104 >> v2:172.21.15.100:3300/0 conn(0x7f3d980b8b90 0x7f3d980b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:56.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.873+0000 7f3d9dfff700 1 -- 172.21.15.98:0/2381515104 >> 172.21.15.98:0/2381515104 conn(0x7f3d9806aee0 msgr2=0x7f3d9806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:56.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.873+0000 7f3d9dfff700 1 -- 172.21.15.98:0/2381515104 shutdown_connections 2023-03-03T08:00:56.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.873+0000 7f3d9dfff700 1 -- 172.21.15.98:0/2381515104 wait complete. 2023-03-03T08:00:56.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.875+0000 7f3d9dfff700 1 Processor -- start 2023-03-03T08:00:56.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.875+0000 7f3d9dfff700 1 -- start start 2023-03-03T08:00:56.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.876+0000 7f3d9dfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d980b5240 0x7f3d980a9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:56.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.876+0000 7f3d9dfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d980b87c0 0x7f3d981f5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:56.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.876+0000 7f3d9dfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d980b8b90 0x7f3d9815b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:56.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.876+0000 7f3d9dfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d980cb480 con 0x7f3d980b5240 2023-03-03T08:00:56.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.876+0000 7f3d9dfff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d980cb760 con 0x7f3d980b8b90 2023-03-03T08:00:56.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.876+0000 7f3d9dfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d980cba40 con 0x7f3d980b87c0 2023-03-03T08:00:56.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.876+0000 7f3d977fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d980b5240 0x7f3d980a9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:56.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.877+0000 7f3d977fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d980b5240 0x7f3d980a9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48030/0 (socket says 172.21.15.98:48030) 2023-03-03T08:00:56.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.877+0000 7f3d977fe700 1 -- 172.21.15.98:0/340589878 learned_addr learned my addr 172.21.15.98:0/340589878 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:56.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.877+0000 7f3d97fff700 1 --2- 172.21.15.98:0/340589878 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d980b8b90 0x7f3d9815b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:56.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.877+0000 7f3d8ffff700 1 --2- 172.21.15.98:0/340589878 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d980b87c0 0x7f3d981f5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:56.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.877+0000 7f3d977fe700 1 -- 172.21.15.98:0/340589878 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d980b87c0 msgr2=0x7f3d981f5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:56.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.877+0000 7f3d977fe700 1 --2- 172.21.15.98:0/340589878 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d980b87c0 0x7f3d981f5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:56.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.877+0000 7f3d977fe700 1 -- 172.21.15.98:0/340589878 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d980b8b90 msgr2=0x7f3d9815b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:56.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.877+0000 7f3d977fe700 1 --2- 172.21.15.98:0/340589878 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d980b8b90 0x7f3d9815b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:56.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.877+0000 7f3d977fe700 1 -- 172.21.15.98:0/340589878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d880094a0 con 0x7f3d980b5240 2023-03-03T08:00:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.877+0000 7f3d977fe700 1 --2- 172.21.15.98:0/340589878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d980b5240 0x7f3d980a9c70 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3d880090b0 tx=0x7f3d88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.878+0000 7f3d94ff9700 1 -- 172.21.15.98:0/340589878 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d880036a0 con 0x7f3d980b5240 2023-03-03T08:00:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.878+0000 7f3d94ff9700 1 -- 172.21.15.98:0/340589878 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8801f070 con 0x7f3d980b5240 2023-03-03T08:00:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.878+0000 7f3d9dfff700 1 -- 172.21.15.98:0/340589878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d980cdf30 con 0x7f3d980b5240 2023-03-03T08:00:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.878+0000 7f3d94ff9700 1 -- 172.21.15.98:0/340589878 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d88003a20 con 0x7f3d980b5240 2023-03-03T08:00:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.878+0000 7f3d9dfff700 1 -- 172.21.15.98:0/340589878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d98073a80 con 0x7f3d980b5240 2023-03-03T08:00:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.879+0000 7f3d9dfff700 1 -- 172.21.15.98:0/340589878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d980ce0c0 con 0x7f3d980b5240 2023-03-03T08:00:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.880+0000 7f3d94ff9700 1 -- 172.21.15.98:0/340589878 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f3d8800f050 con 0x7f3d980b5240 2023-03-03T08:00:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.882+0000 7f3d94ff9700 1 --2- 172.21.15.98:0/340589878 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3d780684a0 0x7f3d7806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.882+0000 7f3d94ff9700 1 -- 172.21.15.98:0/340589878 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3d88021070 con 0x7f3d980b5240 2023-03-03T08:00:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.882+0000 7f3d8ffff700 1 --2- 172.21.15.98:0/340589878 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3d780684a0 0x7f3d7806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.883+0000 7f3d8ffff700 1 --2- 172.21.15.98:0/340589878 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3d780684a0 0x7f3d7806a950 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3d980b8fd0 tx=0x7f3d80008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:56.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:56.886+0000 7f3d94ff9700 1 -- 172.21.15.98:0/340589878 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3d88053eb0 con 0x7f3d980b5240 2023-03-03T08:00:57.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.080+0000 7f3d9dfff700 1 -- 172.21.15.98:0/340589878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d980729b0 con 0x7f3d980b5240 2023-03-03T08:00:57.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.081+0000 7f3d94ff9700 1 -- 172.21.15.98:0/340589878 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+17921 (secure 0 0 0) 0x7f3d880574d0 con 0x7f3d980b5240 2023-03-03T08:00:57.083 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:00:57.083 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":66,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:00:23.326015+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:00:19.823120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-03-03T08:00:21.324346+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:00:57.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.084+0000 7f3d9dfff700 1 -- 172.21.15.98:0/340589878 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3d780684a0 msgr2=0x7f3d7806a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:57.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.084+0000 7f3d9dfff700 1 --2- 172.21.15.98:0/340589878 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3d780684a0 0x7f3d7806a950 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3d980b8fd0 tx=0x7f3d80008040 comp rx=0 tx=0).stop 2023-03-03T08:00:57.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.084+0000 7f3d9dfff700 1 -- 172.21.15.98:0/340589878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d980b5240 msgr2=0x7f3d980a9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:57.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.084+0000 7f3d9dfff700 1 --2- 172.21.15.98:0/340589878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d980b5240 0x7f3d980a9c70 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3d880090b0 tx=0x7f3d88004750 comp rx=0 tx=0).stop 2023-03-03T08:00:57.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.084+0000 7f3d9dfff700 1 -- 172.21.15.98:0/340589878 shutdown_connections 2023-03-03T08:00:57.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.084+0000 7f3d9dfff700 1 --2- 172.21.15.98:0/340589878 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3d780684a0 0x7f3d7806a950 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.084+0000 7f3d9dfff700 1 --2- 172.21.15.98:0/340589878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d980b5240 0x7f3d980a9c70 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.084+0000 7f3d9dfff700 1 --2- 172.21.15.98:0/340589878 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d980b8b90 0x7f3d9815b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.084+0000 7f3d9dfff700 1 --2- 172.21.15.98:0/340589878 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d980b87c0 0x7f3d981f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.084+0000 7f3d9dfff700 1 -- 172.21.15.98:0/340589878 >> 172.21.15.98:0/340589878 conn(0x7f3d9806aee0 msgr2=0x7f3d9815f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:57.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.084+0000 7f3d9dfff700 1 -- 172.21.15.98:0/340589878 shutdown_connections 2023-03-03T08:00:57.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.084+0000 7f3d9dfff700 1 -- 172.21.15.98:0/340589878 wait complete. 2023-03-03T08:00:57.113 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T08:00:19.823120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T08:00:21.324346+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:00:57.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:00:57.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.361+0000 7f21d85ff700 1 Processor -- start 2023-03-03T08:00:57.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.361+0000 7f21d85ff700 1 -- start start 2023-03-03T08:00:57.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.362+0000 7f21d85ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21d015ca70 0x7f21d00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:57.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.363+0000 7f21d85ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f21d006b4b0 0x7f21d00cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:57.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.363+0000 7f21d85ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d00cf980 con 0x7f21d00d00a0 2023-03-03T08:00:57.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.363+0000 7f21d85ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d00cd5b0 con 0x7f21d006b4b0 2023-03-03T08:00:57.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.363+0000 7f21d85ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d00d0510 con 0x7f21d015ca70 2023-03-03T08:00:57.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.363+0000 7f21d6b9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f21d00d00a0 0x7f21d00b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34090/0 (socket says 172.21.15.98:34090) 2023-03-03T08:00:57.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.363+0000 7f21d6b9c700 1 -- 172.21.15.98:0/2147323775 learned_addr learned my addr 172.21.15.98:0/2147323775 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:57.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.363+0000 7f21d5b9a700 1 --2- 172.21.15.98:0/2147323775 >> v2:172.21.15.100:3300/0 conn(0x7f21d006b4b0 0x7f21d00cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:57.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.363+0000 7f21d639b700 1 --2- 172.21.15.98:0/2147323775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21d015ca70 0x7f21d00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:57.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.364+0000 7f21d5399700 1 -- 172.21.15.98:0/2147323775 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273201608 0 0) 0x7f21d00cf980 con 0x7f21d00d00a0 2023-03-03T08:00:57.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.364+0000 7f21d5399700 1 -- 172.21.15.98:0/2147323775 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21bc0035b0 con 0x7f21d00d00a0 2023-03-03T08:00:57.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.364+0000 7f21d639b700 1 -- 172.21.15.98:0/2147323775 >> v2:172.21.15.100:3300/0 conn(0x7f21d006b4b0 msgr2=0x7f21d00cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:57.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.364+0000 7f21d639b700 1 --2- 172.21.15.98:0/2147323775 >> v2:172.21.15.100:3300/0 conn(0x7f21d006b4b0 0x7f21d00cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.364+0000 7f21d639b700 1 -- 172.21.15.98:0/2147323775 >> v1:172.21.15.98:6789/0 conn(0x7f21d00d00a0 legacy=0x7f21d00b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:57.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.364+0000 7f21d639b700 1 -- 172.21.15.98:0/2147323775 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d00d2fd0 con 0x7f21d015ca70 2023-03-03T08:00:57.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.364+0000 7f21d639b700 1 --2- 172.21.15.98:0/2147323775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21d015ca70 0x7f21d00cf440 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009840 tx=0x7f21cc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ee1db63608eacc6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:57.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.364+0000 7f21d5399700 1 -- 172.21.15.98:0/2147323775 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc011830 con 0x7f21d015ca70 2023-03-03T08:00:57.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.365+0000 7f21d5399700 1 -- 172.21.15.98:0/2147323775 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21cc011990 con 0x7f21d015ca70 2023-03-03T08:00:57.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.365+0000 7f21d5399700 1 -- 172.21.15.98:0/2147323775 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc011d20 con 0x7f21d015ca70 2023-03-03T08:00:57.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.365+0000 7f21d85ff700 1 -- 172.21.15.98:0/2147323775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21d015ca70 msgr2=0x7f21d00cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:57.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.365+0000 7f21d85ff700 1 --2- 172.21.15.98:0/2147323775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21d015ca70 0x7f21d00cf440 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009840 tx=0x7f21cc009b50 comp rx=0 tx=0).stop 2023-03-03T08:00:57.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.365+0000 7f21d85ff700 1 -- 172.21.15.98:0/2147323775 shutdown_connections 2023-03-03T08:00:57.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.365+0000 7f21d85ff700 1 --2- 172.21.15.98:0/2147323775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21d015ca70 0x7f21d00cf440 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.365+0000 7f21d85ff700 1 --2- 172.21.15.98:0/2147323775 >> v2:172.21.15.100:3300/0 conn(0x7f21d006b4b0 0x7f21d00cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.365+0000 7f21d85ff700 1 -- 172.21.15.98:0/2147323775 >> 172.21.15.98:0/2147323775 conn(0x7f21d0073e70 msgr2=0x7f21d006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:57.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.365+0000 7f21d85ff700 1 -- 172.21.15.98:0/2147323775 shutdown_connections 2023-03-03T08:00:57.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.365+0000 7f21d85ff700 1 -- 172.21.15.98:0/2147323775 wait complete. 2023-03-03T08:00:57.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.367+0000 7f21d85ff700 1 Processor -- start 2023-03-03T08:00:57.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.367+0000 7f21d85ff700 1 -- start start 2023-03-03T08:00:57.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.368+0000 7f21d85ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d006b4b0 0x7f21d0156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:57.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.368+0000 7f21d85ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21d00d00a0 0x7f21d015d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:57.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.368+0000 7f21d85ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21d015ca70 0x7f21d00b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:57.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.368+0000 7f21d85ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d015d560 con 0x7f21d006b4b0 2023-03-03T08:00:57.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.368+0000 7f21d85ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d00ca380 con 0x7f21d00d00a0 2023-03-03T08:00:57.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.368+0000 7f21d85ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d00ca620 con 0x7f21d015ca70 2023-03-03T08:00:57.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.369+0000 7f21d639b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d006b4b0 0x7f21d0156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:57.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.369+0000 7f21d639b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d006b4b0 0x7f21d0156950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48046/0 (socket says 172.21.15.98:48046) 2023-03-03T08:00:57.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.369+0000 7f21d639b700 1 -- 172.21.15.98:0/1286587673 learned_addr learned my addr 172.21.15.98:0/1286587673 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:57.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.369+0000 7f21d5b9a700 1 --2- 172.21.15.98:0/1286587673 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21d00d00a0 0x7f21d015d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:57.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.369+0000 7f21d6b9c700 1 --2- 172.21.15.98:0/1286587673 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21d015ca70 0x7f21d00b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:57.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.369+0000 7f21d639b700 1 -- 172.21.15.98:0/1286587673 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21d015ca70 msgr2=0x7f21d00b3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:57.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.369+0000 7f21d639b700 1 --2- 172.21.15.98:0/1286587673 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21d015ca70 0x7f21d00b3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.369+0000 7f21d639b700 1 -- 172.21.15.98:0/1286587673 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21d00d00a0 msgr2=0x7f21d015d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:57.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.369+0000 7f21d639b700 1 --2- 172.21.15.98:0/1286587673 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21d00d00a0 0x7f21d015d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.369+0000 7f21d639b700 1 -- 172.21.15.98:0/1286587673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc0094a0 con 0x7f21d006b4b0 2023-03-03T08:00:57.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.369+0000 7f21d5b9a700 1 --2- 172.21.15.98:0/1286587673 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21d00d00a0 0x7f21d015d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:57.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.369+0000 7f21d639b700 1 --2- 172.21.15.98:0/1286587673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d006b4b0 0x7f21d0156950 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009fd0 tx=0x7f21cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:57.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.370+0000 7f21c6ffd700 1 -- 172.21.15.98:0/1286587673 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc0036a0 con 0x7f21d006b4b0 2023-03-03T08:00:57.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.370+0000 7f21c6ffd700 1 -- 172.21.15.98:0/1286587673 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21cc01e070 con 0x7f21d006b4b0 2023-03-03T08:00:57.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.370+0000 7f21c6ffd700 1 -- 172.21.15.98:0/1286587673 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc003a60 con 0x7f21d006b4b0 2023-03-03T08:00:57.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.370+0000 7f21d85ff700 1 -- 172.21.15.98:0/1286587673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d00cd900 con 0x7f21d006b4b0 2023-03-03T08:00:57.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.370+0000 7f21d85ff700 1 -- 172.21.15.98:0/1286587673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d00d1800 con 0x7f21d006b4b0 2023-03-03T08:00:57.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.371+0000 7f21d85ff700 1 -- 172.21.15.98:0/1286587673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d00cdbe0 con 0x7f21d006b4b0 2023-03-03T08:00:57.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.372+0000 7f21c6ffd700 1 -- 172.21.15.98:0/1286587673 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f21cc00f050 con 0x7f21d006b4b0 2023-03-03T08:00:57.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.373+0000 7f21c6ffd700 1 --2- 172.21.15.98:0/1286587673 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f21b40684a0 0x7f21b406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:57.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.373+0000 7f21c6ffd700 1 -- 172.21.15.98:0/1286587673 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f21cc086e70 con 0x7f21d006b4b0 2023-03-03T08:00:57.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.374+0000 7f21d5b9a700 1 --2- 172.21.15.98:0/1286587673 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f21b40684a0 0x7f21b406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:57.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.374+0000 7f21d5b9a700 1 --2- 172.21.15.98:0/1286587673 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f21b40684a0 0x7f21b406a950 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f21d00c9b10 tx=0x7f21c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:57.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.378+0000 7f21c6ffd700 1 -- 172.21.15.98:0/1286587673 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f21cc053530 con 0x7f21d006b4b0 2023-03-03T08:00:57.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.573+0000 7f21d85ff700 1 -- 172.21.15.98:0/1286587673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f21d004e6e0 con 0x7f21d006b4b0 2023-03-03T08:00:57.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.574+0000 7f21c6ffd700 1 -- 172.21.15.98:0/1286587673 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7f21cc0150a0 con 0x7f21d006b4b0 2023-03-03T08:00:57.576 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:00:57.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.576+0000 7f21d85ff700 1 -- 172.21.15.98:0/1286587673 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f21b40684a0 msgr2=0x7f21b406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:57.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.576+0000 7f21d85ff700 1 --2- 172.21.15.98:0/1286587673 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f21b40684a0 0x7f21b406a950 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f21d00c9b10 tx=0x7f21c0008040 comp rx=0 tx=0).stop 2023-03-03T08:00:57.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.576+0000 7f21d85ff700 1 -- 172.21.15.98:0/1286587673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d006b4b0 msgr2=0x7f21d0156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:57.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.576+0000 7f21d85ff700 1 --2- 172.21.15.98:0/1286587673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d006b4b0 0x7f21d0156950 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009fd0 tx=0x7f21cc004750 comp rx=0 tx=0).stop 2023-03-03T08:00:57.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.576+0000 7f21d85ff700 1 -- 172.21.15.98:0/1286587673 shutdown_connections 2023-03-03T08:00:57.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.576+0000 7f21d85ff700 1 --2- 172.21.15.98:0/1286587673 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f21b40684a0 0x7f21b406a950 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.576+0000 7f21d85ff700 1 --2- 172.21.15.98:0/1286587673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d006b4b0 0x7f21d0156950 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.576+0000 7f21d85ff700 1 --2- 172.21.15.98:0/1286587673 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21d015ca70 0x7f21d00b3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.576+0000 7f21d85ff700 1 --2- 172.21.15.98:0/1286587673 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21d00d00a0 0x7f21d015d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.576+0000 7f21d85ff700 1 -- 172.21.15.98:0/1286587673 >> 172.21.15.98:0/1286587673 conn(0x7f21d0073e70 msgr2=0x7f21d006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:57.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.577+0000 7f21d85ff700 1 -- 172.21.15.98:0/1286587673 shutdown_connections 2023-03-03T08:00:57.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.577+0000 7f21d85ff700 1 -- 172.21.15.98:0/1286587673 wait complete. 2023-03-03T08:00:57.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:00:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.852+0000 7f3dda5ff700 1 Processor -- start 2023-03-03T08:00:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.852+0000 7f3dda5ff700 1 -- start start 2023-03-03T08:00:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.854+0000 7f3dda5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dd40d0270 0x7f3dd40cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.854+0000 7f3dda5ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd40cf980 con 0x7f3dd415ca70 2023-03-03T08:00:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.854+0000 7f3dda5ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd40d1800 con 0x7f3dd406b4b0 2023-03-03T08:00:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.854+0000 7f3dda5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd40cd5b0 con 0x7f3dd40d0270 2023-03-03T08:00:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.854+0000 7f3dd3fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3dd415ca70 0x7f3dd40cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34096/0 (socket says 172.21.15.98:34096) 2023-03-03T08:00:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.854+0000 7f3dd3fff700 1 -- 172.21.15.98:0/2417156417 learned_addr learned my addr 172.21.15.98:0/2417156417 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:57.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dcbfff700 1 --2- 172.21.15.98:0/2417156417 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dd40d0270 0x7f3dd40cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dd37fe700 1 -- 172.21.15.98:0/2417156417 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945240431 0 0) 0x7f3dd40cf980 con 0x7f3dd415ca70 2023-03-03T08:00:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dd37fe700 1 -- 172.21.15.98:0/2417156417 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dc00035b0 con 0x7f3dd415ca70 2023-03-03T08:00:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dd37fe700 1 -- 172.21.15.98:0/2417156417 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3050836516 0 0) 0x7f3dc00035b0 con 0x7f3dd415ca70 2023-03-03T08:00:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dd37fe700 1 -- 172.21.15.98:0/2417156417 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dd40d0270 msgr2=0x7f3dd40cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dd37fe700 1 --2- 172.21.15.98:0/2417156417 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dd40d0270 0x7f3dd40cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dd37fe700 1 -- 172.21.15.98:0/2417156417 >> v1:172.21.15.100:6789/0 conn(0x7f3dd406b4b0 legacy=0x7f3dd40d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dd37fe700 1 -- 172.21.15.98:0/2417156417 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd40cd900 con 0x7f3dd415ca70 2023-03-03T08:00:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dd37fe700 1 -- 172.21.15.98:0/2417156417 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f3dc4001670 con 0x7f3dd415ca70 2023-03-03T08:00:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dd37fe700 1 -- 172.21.15.98:0/2417156417 >> v1:172.21.15.98:6789/0 conn(0x7f3dd415ca70 legacy=0x7f3dd40cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dd37fe700 1 --2- 172.21.15.98:0/2417156417 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3dc0003a30 0x7f3dc0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dd37fe700 1 --2- 172.21.15.98:0/2417156417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00043d0 0x7f3dc0004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dd37fe700 1 --2- 172.21.15.98:0/2417156417 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dc0005130 0x7f3dc000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dd37fe700 1 -- 172.21.15.98:0/2417156417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd40cd5b0 con 0x7f3dc00043d0 2023-03-03T08:00:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dd37fe700 1 -- 172.21.15.98:0/2417156417 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc000c000 con 0x7f3dc0003a30 2023-03-03T08:00:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dd37fe700 1 -- 172.21.15.98:0/2417156417 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc000c1c0 con 0x7f3dc0005130 2023-03-03T08:00:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.855+0000 7f3dcbfff700 1 --2- 172.21.15.98:0/2417156417 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dd40d0270 0x7f3dd40cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:00:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.856+0000 7f3dcbfff700 1 --2- 172.21.15.98:0/2417156417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00043d0 0x7f3dc0004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.856+0000 7f3dd3fff700 1 --2- 172.21.15.98:0/2417156417 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3dc0003a30 0x7f3dc0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.856+0000 7f3dd8b9c700 1 --2- 172.21.15.98:0/2417156417 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dc0005130 0x7f3dc000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:57.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.856+0000 7f3dcbfff700 1 -- 172.21.15.98:0/2417156417 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dc0005130 msgr2=0x7f3dc000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:57.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.856+0000 7f3dcbfff700 1 --2- 172.21.15.98:0/2417156417 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dc0005130 0x7f3dc000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.856+0000 7f3dcbfff700 1 -- 172.21.15.98:0/2417156417 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3dc0003a30 msgr2=0x7f3dc0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:57.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.856+0000 7f3dcbfff700 1 --2- 172.21.15.98:0/2417156417 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3dc0003a30 0x7f3dc0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.856+0000 7f3dcbfff700 1 -- 172.21.15.98:0/2417156417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc00035b0 con 0x7f3dc00043d0 2023-03-03T08:00:57.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.856+0000 7f3dd3fff700 1 --2- 172.21.15.98:0/2417156417 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3dc0003a30 0x7f3dc0003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:57.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.856+0000 7f3dd8b9c700 1 --2- 172.21.15.98:0/2417156417 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dc0005130 0x7f3dc000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:57.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.857+0000 7f3dcbfff700 1 --2- 172.21.15.98:0/2417156417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00043d0 0x7f3dc0004890 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3dd40b2b30 tx=0x7f3dc0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9598055988b0bd5c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.857+0000 7f3dd37fe700 1 -- 172.21.15.98:0/2417156417 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc004030 con 0x7f3dc00043d0 2023-03-03T08:00:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.857+0000 7f3dd37fe700 1 -- 172.21.15.98:0/2417156417 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dbc004190 con 0x7f3dc00043d0 2023-03-03T08:00:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.857+0000 7f3dd37fe700 1 -- 172.21.15.98:0/2417156417 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc004520 con 0x7f3dc00043d0 2023-03-03T08:00:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.857+0000 7f3dda5ff700 1 -- 172.21.15.98:0/2417156417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00043d0 msgr2=0x7f3dc0004890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.857+0000 7f3dda5ff700 1 --2- 172.21.15.98:0/2417156417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00043d0 0x7f3dc0004890 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3dd40b2b30 tx=0x7f3dc0005880 comp rx=0 tx=0).stop 2023-03-03T08:00:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.857+0000 7f3dd8b9c700 1 -- 172.21.15.98:0/2417156417 reap_dead start 2023-03-03T08:00:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.857+0000 7f3dda5ff700 1 -- 172.21.15.98:0/2417156417 shutdown_connections 2023-03-03T08:00:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.857+0000 7f3dda5ff700 1 -- 172.21.15.98:0/2417156417 >> 172.21.15.98:0/2417156417 conn(0x7f3dd4073e70 msgr2=0x7f3dd406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.857+0000 7f3dda5ff700 1 -- 172.21.15.98:0/2417156417 shutdown_connections 2023-03-03T08:00:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.858+0000 7f3dda5ff700 1 -- 172.21.15.98:0/2417156417 wait complete. 2023-03-03T08:00:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.859+0000 7f3dda5ff700 1 Processor -- start 2023-03-03T08:00:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.860+0000 7f3dda5ff700 1 -- start start 2023-03-03T08:00:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.861+0000 7f3dda5ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00043d0 0x7f3dd40b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.861+0000 7f3dda5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dd414b530 0x7f3dd40cc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.861+0000 7f3dda5ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3dd40cc7a0 0x7f3dd415e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.861+0000 7f3dda5ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd40ccc10 con 0x7f3dc00043d0 2023-03-03T08:00:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.861+0000 7f3dda5ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd40b3bb0 con 0x7f3dd40cc7a0 2023-03-03T08:00:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.861+0000 7f3dda5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd40b4040 con 0x7f3dd414b530 2023-03-03T08:00:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.861+0000 7f3dd3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00043d0 0x7f3dd40b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.861+0000 7f3dd3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00043d0 0x7f3dd40b51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48070/0 (socket says 172.21.15.98:48070) 2023-03-03T08:00:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.861+0000 7f3dd3fff700 1 -- 172.21.15.98:0/3762104548 learned_addr learned my addr 172.21.15.98:0/3762104548 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.861+0000 7f3dd3fff700 1 -- 172.21.15.98:0/3762104548 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dd414b530 msgr2=0x7f3dd40cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.861+0000 7f3dcbfff700 1 --2- 172.21.15.98:0/3762104548 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dd414b530 0x7f3dd40cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.861+0000 7f3dd8b9c700 1 --2- 172.21.15.98:0/3762104548 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3dd40cc7a0 0x7f3dd415e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.861+0000 7f3dd3fff700 1 --2- 172.21.15.98:0/3762104548 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dd414b530 0x7f3dd40cc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.861+0000 7f3dd3fff700 1 -- 172.21.15.98:0/3762104548 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3dd40cc7a0 msgr2=0x7f3dd415e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.861+0000 7f3dd3fff700 1 --2- 172.21.15.98:0/3762104548 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3dd40cc7a0 0x7f3dd415e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.862+0000 7f3dd3fff700 1 -- 172.21.15.98:0/3762104548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd40d1aa0 con 0x7f3dc00043d0 2023-03-03T08:00:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.862+0000 7f3dcbfff700 1 --2- 172.21.15.98:0/3762104548 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dd414b530 0x7f3dd40cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:00:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.862+0000 7f3dd3fff700 1 --2- 172.21.15.98:0/3762104548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00043d0 0x7f3dd40b51f0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400db70 tx=0x7f3dc400df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.862+0000 7f3dd17fa700 1 -- 172.21.15.98:0/3762104548 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4004c80 con 0x7f3dc00043d0 2023-03-03T08:00:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.862+0000 7f3dd17fa700 1 -- 172.21.15.98:0/3762104548 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc4004de0 con 0x7f3dc00043d0 2023-03-03T08:00:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.862+0000 7f3dda5ff700 1 -- 172.21.15.98:0/3762104548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd40cdbe0 con 0x7f3dc00043d0 2023-03-03T08:00:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.862+0000 7f3dd17fa700 1 -- 172.21.15.98:0/3762104548 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4007310 con 0x7f3dc00043d0 2023-03-03T08:00:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.862+0000 7f3dda5ff700 1 -- 172.21.15.98:0/3762104548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd415ce40 con 0x7f3dc00043d0 2023-03-03T08:00:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.863+0000 7f3dda5ff700 1 -- 172.21.15.98:0/3762104548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd40d3c10 con 0x7f3dc00043d0 2023-03-03T08:00:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.865+0000 7f3dd17fa700 1 -- 172.21.15.98:0/3762104548 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f3dc401b030 con 0x7f3dc00043d0 2023-03-03T08:00:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.866+0000 7f3dd17fa700 1 --2- 172.21.15.98:0/3762104548 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3db40684a0 0x7f3db406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.867+0000 7f3dd17fa700 1 -- 172.21.15.98:0/3762104548 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3dc40859a0 con 0x7f3dc00043d0 2023-03-03T08:00:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.867+0000 7f3dcbfff700 1 --2- 172.21.15.98:0/3762104548 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3db40684a0 0x7f3db406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.867+0000 7f3dcbfff700 1 --2- 172.21.15.98:0/3762104548 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3db40684a0 0x7f3db406a950 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3dd40b2b30 tx=0x7f3dbc008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:57.871+0000 7f3dd17fa700 1 -- 172.21.15.98:0/3762104548 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3dc4051fd0 con 0x7f3dc00043d0 2023-03-03T08:00:58.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.061+0000 7f3dda5ff700 1 -- 172.21.15.98:0/3762104548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3dd404e6e0 con 0x7f3dc00043d0 2023-03-03T08:00:58.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.062+0000 7f3dd17fa700 1 -- 172.21.15.98:0/3762104548 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v66) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3dc4019070 con 0x7f3dc00043d0 2023-03-03T08:00:58.064 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:00:58.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.064+0000 7f3dda5ff700 1 -- 172.21.15.98:0/3762104548 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3db40684a0 msgr2=0x7f3db406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:58.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.064+0000 7f3dda5ff700 1 --2- 172.21.15.98:0/3762104548 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3db40684a0 0x7f3db406a950 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3dd40b2b30 tx=0x7f3dbc008070 comp rx=0 tx=0).stop 2023-03-03T08:00:58.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.064+0000 7f3dda5ff700 1 -- 172.21.15.98:0/3762104548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00043d0 msgr2=0x7f3dd40b51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:58.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.064+0000 7f3dda5ff700 1 --2- 172.21.15.98:0/3762104548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00043d0 0x7f3dd40b51f0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400db70 tx=0x7f3dc400df30 comp rx=0 tx=0).stop 2023-03-03T08:00:58.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.064+0000 7f3dda5ff700 1 -- 172.21.15.98:0/3762104548 shutdown_connections 2023-03-03T08:00:58.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.064+0000 7f3dda5ff700 1 --2- 172.21.15.98:0/3762104548 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3db40684a0 0x7f3db406a950 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.064+0000 7f3dda5ff700 1 --2- 172.21.15.98:0/3762104548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00043d0 0x7f3dd40b51f0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.064+0000 7f3dda5ff700 1 --2- 172.21.15.98:0/3762104548 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3dd40cc7a0 0x7f3dd415e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.064+0000 7f3dda5ff700 1 --2- 172.21.15.98:0/3762104548 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3dd414b530 0x7f3dd40cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.064+0000 7f3dda5ff700 1 -- 172.21.15.98:0/3762104548 >> 172.21.15.98:0/3762104548 conn(0x7f3dd4073e70 msgr2=0x7f3dd40d3500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:58.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.065+0000 7f3dda5ff700 1 -- 172.21.15.98:0/3762104548 shutdown_connections 2023-03-03T08:00:58.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.065+0000 7f3dda5ff700 1 -- 172.21.15.98:0/3762104548 wait complete. 2023-03-03T08:00:58.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T08:00:58.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.339+0000 7f5ca279d700 1 Processor -- start 2023-03-03T08:00:58.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.340+0000 7f5ca279d700 1 -- start start 2023-03-03T08:00:58.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.341+0000 7f5ca279d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c9c1609b0 0x7f5c9c160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:58.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.341+0000 7f5ca279d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9c1612c0 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.341+0000 7f5ca279d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9c15dd00 con 0x7f5c9c0cf570 2023-03-03T08:00:58.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.341+0000 7f5ca279d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c0cebb0 con 0x7f5c9c1609b0 2023-03-03T08:00:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.341+0000 7f5ca1f9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5c9c0b4aa0 0x7f5c9c0d0bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34104/0 (socket says 172.21.15.98:34104) 2023-03-03T08:00:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.341+0000 7f5ca1f9c700 1 -- 172.21.15.98:0/947090642 learned_addr learned my addr 172.21.15.98:0/947090642 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.342+0000 7f5ca179b700 1 --2- 172.21.15.98:0/947090642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c9c1609b0 0x7f5c9c160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.342+0000 7f5c93fff700 1 -- 172.21.15.98:0/947090642 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38712461 0 0) 0x7f5c9c1612c0 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.342+0000 7f5c93fff700 1 -- 172.21.15.98:0/947090642 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c880035b0 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.342+0000 7f5c93fff700 1 -- 172.21.15.98:0/947090642 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728767809 0 0) 0x7f5c9c15dd00 con 0x7f5c9c0cf570 2023-03-03T08:00:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.342+0000 7f5c93fff700 1 -- 172.21.15.98:0/947090642 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c9c1612c0 con 0x7f5c9c0cf570 2023-03-03T08:00:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.342+0000 7f5ca179b700 1 -- 172.21.15.98:0/947090642 >> v1:172.21.15.100:6789/0 conn(0x7f5c9c0cf570 legacy=0x7f5c9c15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.342+0000 7f5ca179b700 1 -- 172.21.15.98:0/947090642 >> v1:172.21.15.98:6789/0 conn(0x7f5c9c0b4aa0 legacy=0x7f5c9c0d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.342+0000 7f5ca179b700 1 -- 172.21.15.98:0/947090642 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9c0cd780 con 0x7f5c9c1609b0 2023-03-03T08:00:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.343+0000 7f5ca179b700 1 --2- 172.21.15.98:0/947090642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c9c1609b0 0x7f5c9c160d80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5c98009870 tx=0x7f5c98009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26c6f93087088498 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.343+0000 7f5c93fff700 1 -- 172.21.15.98:0/947090642 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c98011830 con 0x7f5c9c1609b0 2023-03-03T08:00:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.343+0000 7f5c93fff700 1 -- 172.21.15.98:0/947090642 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c98011990 con 0x7f5c9c1609b0 2023-03-03T08:00:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.343+0000 7f5c93fff700 1 -- 172.21.15.98:0/947090642 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c98011d20 con 0x7f5c9c1609b0 2023-03-03T08:00:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.344+0000 7f5ca279d700 1 -- 172.21.15.98:0/947090642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c9c1609b0 msgr2=0x7f5c9c160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.344+0000 7f5ca279d700 1 --2- 172.21.15.98:0/947090642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c9c1609b0 0x7f5c9c160d80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5c98009870 tx=0x7f5c98009b80 comp rx=0 tx=0).stop 2023-03-03T08:00:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.344+0000 7f5ca279d700 1 -- 172.21.15.98:0/947090642 shutdown_connections 2023-03-03T08:00:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.344+0000 7f5ca279d700 1 --2- 172.21.15.98:0/947090642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c9c1609b0 0x7f5c9c160d80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.344+0000 7f5ca279d700 1 -- 172.21.15.98:0/947090642 >> 172.21.15.98:0/947090642 conn(0x7f5c9c06b880 msgr2=0x7f5c9c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.344+0000 7f5ca279d700 1 -- 172.21.15.98:0/947090642 shutdown_connections 2023-03-03T08:00:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.344+0000 7f5ca279d700 1 -- 172.21.15.98:0/947090642 wait complete. 2023-03-03T08:00:58.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.346+0000 7f5ca279d700 1 Processor -- start 2023-03-03T08:00:58.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.346+0000 7f5ca279d700 1 -- start start 2023-03-03T08:00:58.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.347+0000 7f5ca279d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c9c0b4aa0 0x7f5c9c0c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:58.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.347+0000 7f5ca279d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c9c0cf570 0x7f5c9c1f5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:58.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.347+0000 7f5ca279d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5c9c1609b0 0x7f5c9c152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:58.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.347+0000 7f5ca279d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c1f6940 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.347+0000 7f5ca279d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c0c97a0 con 0x7f5c9c1609b0 2023-03-03T08:00:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.347+0000 7f5ca279d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c1f2360 con 0x7f5c9c0cf570 2023-03-03T08:00:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.347+0000 7f5ca179b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c9c0b4aa0 0x7f5c9c0c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.348+0000 7f5ca179b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c9c0b4aa0 0x7f5c9c0c9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48080/0 (socket says 172.21.15.98:48080) 2023-03-03T08:00:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.348+0000 7f5ca179b700 1 -- 172.21.15.98:0/2544983366 learned_addr learned my addr 172.21.15.98:0/2544983366 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.348+0000 7f5ca0f9a700 1 --2- 172.21.15.98:0/2544983366 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c9c0cf570 0x7f5c9c1f5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:58.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.348+0000 7f5ca1f9c700 1 --2- 172.21.15.98:0/2544983366 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5c9c1609b0 0x7f5c9c152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:58.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.348+0000 7f5ca179b700 1 -- 172.21.15.98:0/2544983366 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c9c0cf570 msgr2=0x7f5c9c1f5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:58.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.348+0000 7f5ca179b700 1 --2- 172.21.15.98:0/2544983366 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c9c0cf570 0x7f5c9c1f5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.348+0000 7f5ca179b700 1 -- 172.21.15.98:0/2544983366 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5c9c1609b0 msgr2=0x7f5c9c152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.348+0000 7f5ca179b700 1 --2- 172.21.15.98:0/2544983366 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5c9c1609b0 0x7f5c9c152640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.348+0000 7f5ca179b700 1 -- 172.21.15.98:0/2544983366 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c980094a0 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.348+0000 7f5ca179b700 1 --2- 172.21.15.98:0/2544983366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c9c0b4aa0 0x7f5c9c0c9260 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5c98009bc0 tx=0x7f5c98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.349+0000 7f5c91ffb700 1 -- 172.21.15.98:0/2544983366 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9801a6e0 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.349+0000 7f5c91ffb700 1 -- 172.21.15.98:0/2544983366 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9801a860 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.349+0000 7f5c91ffb700 1 -- 172.21.15.98:0/2544983366 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9801ac10 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.349+0000 7f5ca279d700 1 -- 172.21.15.98:0/2544983366 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c9c0cdd40 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.349+0000 7f5ca279d700 1 -- 172.21.15.98:0/2544983366 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c9c0cda60 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.350+0000 7f5ca279d700 1 -- 172.21.15.98:0/2544983366 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c9c15cdf0 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.351+0000 7f5c91ffb700 1 -- 172.21.15.98:0/2544983366 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f5c9800f050 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.352+0000 7f5c91ffb700 1 --2- 172.21.15.98:0/2544983366 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5c800684a0 0x7f5c8006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.352+0000 7f5c91ffb700 1 -- 172.21.15.98:0/2544983366 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5c98086ce0 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.353+0000 7f5ca0f9a700 1 --2- 172.21.15.98:0/2544983366 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5c800684a0 0x7f5c8006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.353+0000 7f5ca0f9a700 1 --2- 172.21.15.98:0/2544983366 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5c800684a0 0x7f5c8006a950 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c0011e0 tx=0x7f5c8c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.357+0000 7f5c91ffb700 1 -- 172.21.15.98:0/2544983366 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5c980533a0 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.552+0000 7f5ca279d700 1 -- 172.21.15.98:0/2544983366 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5c9c068290 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.552+0000 7f5c91ffb700 1 -- 172.21.15.98:0/2544983366 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v66) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5c98015020 con 0x7f5c9c0b4aa0 2023-03-03T08:00:58.554 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T08:00:58.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.555+0000 7f5ca279d700 1 -- 172.21.15.98:0/2544983366 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5c800684a0 msgr2=0x7f5c8006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:58.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.555+0000 7f5ca279d700 1 --2- 172.21.15.98:0/2544983366 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5c800684a0 0x7f5c8006a950 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c0011e0 tx=0x7f5c8c009040 comp rx=0 tx=0).stop 2023-03-03T08:00:58.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.555+0000 7f5ca279d700 1 -- 172.21.15.98:0/2544983366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c9c0b4aa0 msgr2=0x7f5c9c0c9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:58.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.555+0000 7f5ca279d700 1 --2- 172.21.15.98:0/2544983366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c9c0b4aa0 0x7f5c9c0c9260 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5c98009bc0 tx=0x7f5c98004750 comp rx=0 tx=0).stop 2023-03-03T08:00:58.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.555+0000 7f5ca279d700 1 -- 172.21.15.98:0/2544983366 shutdown_connections 2023-03-03T08:00:58.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.555+0000 7f5ca279d700 1 --2- 172.21.15.98:0/2544983366 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5c800684a0 0x7f5c8006a950 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.555+0000 7f5ca279d700 1 --2- 172.21.15.98:0/2544983366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c9c0b4aa0 0x7f5c9c0c9260 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.555+0000 7f5ca279d700 1 --2- 172.21.15.98:0/2544983366 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5c9c1609b0 0x7f5c9c152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.555+0000 7f5ca279d700 1 --2- 172.21.15.98:0/2544983366 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5c9c0cf570 0x7f5c9c1f5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.555+0000 7f5ca279d700 1 -- 172.21.15.98:0/2544983366 >> 172.21.15.98:0/2544983366 conn(0x7f5c9c06b880 msgr2=0x7f5c9c0b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:58.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.555+0000 7f5ca279d700 1 -- 172.21.15.98:0/2544983366 shutdown_connections 2023-03-03T08:00:58.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.555+0000 7f5ca279d700 1 -- 172.21.15.98:0/2544983366 wait complete. 2023-03-03T08:00:58.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T08:00:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.832+0000 7fc2c43ff700 1 Processor -- start 2023-03-03T08:00:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.833+0000 7fc2c43ff700 1 -- start start 2023-03-03T08:00:58.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.834+0000 7fc2c43ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2bc0b26d0 0x7fc2bc0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:58.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.834+0000 7fc2c43ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2bc0b5140 0x7fc2bc0d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:58.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.834+0000 7fc2c43ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2bc0cd7d0 con 0x7fc2bc0b8d10 2023-03-03T08:00:58.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.834+0000 7fc2c43ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2bc0b55c0 con 0x7fc2bc0b5140 2023-03-03T08:00:58.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.834+0000 7fc2c43ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc0b90e0 con 0x7fc2bc0b26d0 2023-03-03T08:00:58.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.834+0000 7fc2c219b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc2bc0b8d10 0x7fc2bc0cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34118/0 (socket says 172.21.15.98:34118) 2023-03-03T08:00:58.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.834+0000 7fc2c219b700 1 -- 172.21.15.98:0/3794727608 learned_addr learned my addr 172.21.15.98:0/3794727608 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:58.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.835+0000 7fc2c299c700 1 --2- 172.21.15.98:0/3794727608 >> v2:172.21.15.98:3300/0 conn(0x7fc2bc0b5140 0x7fc2bc0d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:58.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.835+0000 7fc2c199a700 1 --2- 172.21.15.98:0/3794727608 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2bc0b26d0 0x7fc2bc0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:58.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.835+0000 7fc2c1199700 1 -- 172.21.15.98:0/3794727608 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877700642 0 0) 0x7fc2bc0cd7d0 con 0x7fc2bc0b8d10 2023-03-03T08:00:58.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.835+0000 7fc2c1199700 1 -- 172.21.15.98:0/3794727608 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a80035b0 con 0x7fc2bc0b8d10 2023-03-03T08:00:58.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.835+0000 7fc2c299c700 1 -- 172.21.15.98:0/3794727608 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2bc0b26d0 msgr2=0x7fc2bc0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:58.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.835+0000 7fc2c299c700 1 --2- 172.21.15.98:0/3794727608 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2bc0b26d0 0x7fc2bc0b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.835+0000 7fc2c299c700 1 -- 172.21.15.98:0/3794727608 >> v1:172.21.15.98:6789/0 conn(0x7fc2bc0b8d10 legacy=0x7fc2bc0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:58.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.835+0000 7fc2c299c700 1 -- 172.21.15.98:0/3794727608 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc0cda20 con 0x7fc2bc0b5140 2023-03-03T08:00:58.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.835+0000 7fc2c299c700 1 --2- 172.21.15.98:0/3794727608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2bc0b5140 0x7fc2bc0d13a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009d90 tx=0x7fc2b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12edc997271b7ee server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:58.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.836+0000 7fc2c1199700 1 -- 172.21.15.98:0/3794727608 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b400e070 con 0x7fc2bc0b5140 2023-03-03T08:00:58.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.836+0000 7fc2c1199700 1 -- 172.21.15.98:0/3794727608 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b400bb20 con 0x7fc2bc0b5140 2023-03-03T08:00:58.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.836+0000 7fc2c1199700 1 -- 172.21.15.98:0/3794727608 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b4012430 con 0x7fc2bc0b5140 2023-03-03T08:00:58.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.836+0000 7fc2c43ff700 1 -- 172.21.15.98:0/3794727608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2bc0b5140 msgr2=0x7fc2bc0d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:58.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.836+0000 7fc2c43ff700 1 --2- 172.21.15.98:0/3794727608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2bc0b5140 0x7fc2bc0d13a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009d90 tx=0x7fc2b400b330 comp rx=0 tx=0).stop 2023-03-03T08:00:58.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.836+0000 7fc2c43ff700 1 -- 172.21.15.98:0/3794727608 shutdown_connections 2023-03-03T08:00:58.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.836+0000 7fc2c43ff700 1 --2- 172.21.15.98:0/3794727608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2bc0b5140 0x7fc2bc0d13a0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.836+0000 7fc2c43ff700 1 --2- 172.21.15.98:0/3794727608 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2bc0b26d0 0x7fc2bc0b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.836+0000 7fc2c43ff700 1 -- 172.21.15.98:0/3794727608 >> 172.21.15.98:0/3794727608 conn(0x7fc2bc073e70 msgr2=0x7fc2bc06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:58.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.837+0000 7fc2c43ff700 1 -- 172.21.15.98:0/3794727608 shutdown_connections 2023-03-03T08:00:58.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.837+0000 7fc2c43ff700 1 -- 172.21.15.98:0/3794727608 wait complete. 2023-03-03T08:00:58.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.839+0000 7fc2c43ff700 1 Processor -- start 2023-03-03T08:00:58.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.839+0000 7fc2c43ff700 1 -- start start 2023-03-03T08:00:58.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.840+0000 7fc2c43ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2bc0b26d0 0x7fc2bc1546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:58.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.840+0000 7fc2c43ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2bc0b5140 0x7fc2bc1f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:58.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.840+0000 7fc2c43ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2bc0b8d10 0x7fc2bc1f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:58.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.840+0000 7fc2c43ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc1f2fa0 con 0x7fc2bc0b26d0 2023-03-03T08:00:58.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.840+0000 7fc2c43ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc1f3280 con 0x7fc2bc0b5140 2023-03-03T08:00:58.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.840+0000 7fc2c43ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc1f3860 con 0x7fc2bc0b8d10 2023-03-03T08:00:58.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.840+0000 7fc2c219b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2bc0b26d0 0x7fc2bc1546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:58.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.840+0000 7fc2c219b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2bc0b26d0 0x7fc2bc1546a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48094/0 (socket says 172.21.15.98:48094) 2023-03-03T08:00:58.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.840+0000 7fc2c219b700 1 -- 172.21.15.98:0/755046272 learned_addr learned my addr 172.21.15.98:0/755046272 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:58.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.841+0000 7fc2c299c700 1 --2- 172.21.15.98:0/755046272 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2bc0b8d10 0x7fc2bc1f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:58.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.841+0000 7fc2c199a700 1 --2- 172.21.15.98:0/755046272 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2bc0b5140 0x7fc2bc1f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:58.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.841+0000 7fc2c219b700 1 -- 172.21.15.98:0/755046272 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2bc0b8d10 msgr2=0x7fc2bc1f29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:58.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.841+0000 7fc2c219b700 1 --2- 172.21.15.98:0/755046272 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2bc0b8d10 0x7fc2bc1f29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.841+0000 7fc2c219b700 1 -- 172.21.15.98:0/755046272 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2bc0b5140 msgr2=0x7fc2bc1f2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.841+0000 7fc2c219b700 1 --2- 172.21.15.98:0/755046272 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2bc0b5140 0x7fc2bc1f2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.841+0000 7fc2c219b700 1 -- 172.21.15.98:0/755046272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b4009a70 con 0x7fc2bc0b26d0 2023-03-03T08:00:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.841+0000 7fc2c199a700 1 --2- 172.21.15.98:0/755046272 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2bc0b5140 0x7fc2bc1f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.841+0000 7fc2c219b700 1 --2- 172.21.15.98:0/755046272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2bc0b26d0 0x7fc2bc1546a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8001eb0 tx=0x7fc2b800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.841+0000 7fc2b2ffd700 1 -- 172.21.15.98:0/755046272 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8005d70 con 0x7fc2bc0b26d0 2023-03-03T08:00:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.841+0000 7fc2b2ffd700 1 -- 172.21.15.98:0/755046272 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b80070d0 con 0x7fc2bc0b26d0 2023-03-03T08:00:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.841+0000 7fc2b2ffd700 1 -- 172.21.15.98:0/755046272 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8007460 con 0x7fc2bc0b26d0 2023-03-03T08:00:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.841+0000 7fc2c43ff700 1 -- 172.21.15.98:0/755046272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2bc0d0e80 con 0x7fc2bc0b26d0 2023-03-03T08:00:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.842+0000 7fc2c43ff700 1 -- 172.21.15.98:0/755046272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2bc0cdd00 con 0x7fc2bc0b26d0 2023-03-03T08:00:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.843+0000 7fc2c43ff700 1 -- 172.21.15.98:0/755046272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2bc0cdfe0 con 0x7fc2bc0b26d0 2023-03-03T08:00:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.844+0000 7fc2b2ffd700 1 -- 172.21.15.98:0/755046272 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fc2b8019030 con 0x7fc2bc0b26d0 2023-03-03T08:00:58.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.846+0000 7fc2b2ffd700 1 --2- 172.21.15.98:0/755046272 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc2a00684a0 0x7fc2a006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:58.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.846+0000 7fc2b2ffd700 1 -- 172.21.15.98:0/755046272 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc2b8085240 con 0x7fc2bc0b26d0 2023-03-03T08:00:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.846+0000 7fc2c199a700 1 --2- 172.21.15.98:0/755046272 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc2a00684a0 0x7fc2a006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.847+0000 7fc2c199a700 1 --2- 172.21.15.98:0/755046272 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc2a00684a0 0x7fc2a006a950 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0ce990 tx=0x7fc2ac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:58.850+0000 7fc2b2ffd700 1 -- 172.21.15.98:0/755046272 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc2b8051900 con 0x7fc2bc0b26d0 2023-03-03T08:00:59.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.041+0000 7fc2c43ff700 1 -- 172.21.15.98:0/755046272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc2bc04e6e0 con 0x7fc2bc0b26d0 2023-03-03T08:00:59.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.041+0000 7fc2b2ffd700 1 -- 172.21.15.98:0/755046272 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v66) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc2b8054f20 con 0x7fc2bc0b26d0 2023-03-03T08:00:59.043 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T08:00:59.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.044+0000 7fc2c43ff700 1 -- 172.21.15.98:0/755046272 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc2a00684a0 msgr2=0x7fc2a006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:59.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.044+0000 7fc2c43ff700 1 --2- 172.21.15.98:0/755046272 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc2a00684a0 0x7fc2a006a950 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0ce990 tx=0x7fc2ac008040 comp rx=0 tx=0).stop 2023-03-03T08:00:59.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.044+0000 7fc2c43ff700 1 -- 172.21.15.98:0/755046272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2bc0b26d0 msgr2=0x7fc2bc1546a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:59.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.044+0000 7fc2c43ff700 1 --2- 172.21.15.98:0/755046272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2bc0b26d0 0x7fc2bc1546a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8001eb0 tx=0x7fc2b800c7b0 comp rx=0 tx=0).stop 2023-03-03T08:00:59.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.044+0000 7fc2c43ff700 1 -- 172.21.15.98:0/755046272 shutdown_connections 2023-03-03T08:00:59.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.044+0000 7fc2c43ff700 1 --2- 172.21.15.98:0/755046272 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc2a00684a0 0x7fc2a006a950 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:59.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.044+0000 7fc2c43ff700 1 --2- 172.21.15.98:0/755046272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2bc0b26d0 0x7fc2bc1546a0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:59.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.044+0000 7fc2c43ff700 1 --2- 172.21.15.98:0/755046272 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2bc0b8d10 0x7fc2bc1f29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:59.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.044+0000 7fc2c43ff700 1 --2- 172.21.15.98:0/755046272 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2bc0b5140 0x7fc2bc1f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:59.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.044+0000 7fc2c43ff700 1 -- 172.21.15.98:0/755046272 >> 172.21.15.98:0/755046272 conn(0x7fc2bc073e70 msgr2=0x7fc2bc15b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:59.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.044+0000 7fc2c43ff700 1 -- 172.21.15.98:0/755046272 shutdown_connections 2023-03-03T08:00:59.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.044+0000 7fc2c43ff700 1 -- 172.21.15.98:0/755046272 wait complete. 2023-03-03T08:00:59.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T08:00:59.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.318+0000 7f3c42dff700 1 Processor -- start 2023-03-03T08:00:59.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.318+0000 7f3c42dff700 1 -- start start 2023-03-03T08:00:59.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.319+0000 7f3c42dff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3c3c15c830 0x7f3c3c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:59.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.320+0000 7f3c42dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c3c15cc00 0x7f3c3c0d0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:59.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.320+0000 7f3c42dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c3c0d0540 con 0x7f3c3c0b4c70 2023-03-03T08:00:59.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.320+0000 7f3c42dff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c3c0cd970 con 0x7f3c3c15c830 2023-03-03T08:00:59.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.320+0000 7f3c42dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c0cdc50 con 0x7f3c3c15cc00 2023-03-03T08:00:59.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.320+0000 7f3c40b9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3c3c15c830 0x7f3c3c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:59.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.320+0000 7f3c4139c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3c3c0b4c70 0x7f3c3c160600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34126/0 (socket says 172.21.15.98:34126) 2023-03-03T08:00:59.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.320+0000 7f3c4139c700 1 -- 172.21.15.98:0/2782172836 learned_addr learned my addr 172.21.15.98:0/2782172836 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:00:59.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.320+0000 7f3c3bfff700 1 --2- 172.21.15.98:0/2782172836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c3c15cc00 0x7f3c3c0d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:59.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.321+0000 7f3c40b9b700 1 -- 172.21.15.98:0/2782172836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c3c15cc00 msgr2=0x7f3c3c0d0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:00:59.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.321+0000 7f3c40b9b700 1 --2- 172.21.15.98:0/2782172836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c3c15cc00 0x7f3c3c0d0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:59.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.321+0000 7f3c40b9b700 1 -- 172.21.15.98:0/2782172836 >> v1:172.21.15.98:6789/0 conn(0x7f3c3c0b4c70 legacy=0x7f3c3c160600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:59.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.321+0000 7f3c40b9b700 1 -- 172.21.15.98:0/2782172836 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3c0cdf30 con 0x7f3c3c15c830 2023-03-03T08:00:59.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.321+0000 7f3c3b7fe700 1 -- 172.21.15.98:0/2782172836 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203354218 0 0) 0x7f3c3c0d0540 con 0x7f3c3c0b4c70 2023-03-03T08:00:59.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.321+0000 7f3c40b9b700 1 --2- 172.21.15.98:0/2782172836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3c15c830 0x7f3c3c06b9a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f3c30009840 tx=0x7f3c30009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec55ee0049a4908e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:59.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.321+0000 7f3c3b7fe700 1 -- 172.21.15.98:0/2782172836 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30011830 con 0x7f3c3c15c830 2023-03-03T08:00:59.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.321+0000 7f3c3b7fe700 1 -- 172.21.15.98:0/2782172836 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c30011990 con 0x7f3c3c15c830 2023-03-03T08:00:59.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.321+0000 7f3c3b7fe700 1 -- 172.21.15.98:0/2782172836 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30011d20 con 0x7f3c3c15c830 2023-03-03T08:00:59.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.322+0000 7f3c42dff700 1 -- 172.21.15.98:0/2782172836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3c15c830 msgr2=0x7f3c3c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:59.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.322+0000 7f3c42dff700 1 --2- 172.21.15.98:0/2782172836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3c15c830 0x7f3c3c06b9a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f3c30009840 tx=0x7f3c30009b50 comp rx=0 tx=0).stop 2023-03-03T08:00:59.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.322+0000 7f3c42dff700 1 -- 172.21.15.98:0/2782172836 shutdown_connections 2023-03-03T08:00:59.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.322+0000 7f3c42dff700 1 --2- 172.21.15.98:0/2782172836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3c15c830 0x7f3c3c06b9a0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:59.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.322+0000 7f3c42dff700 1 --2- 172.21.15.98:0/2782172836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c3c15cc00 0x7f3c3c0d0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:59.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.322+0000 7f3c42dff700 1 -- 172.21.15.98:0/2782172836 >> 172.21.15.98:0/2782172836 conn(0x7f3c3c073e70 msgr2=0x7f3c3c06ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:59.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.322+0000 7f3c42dff700 1 -- 172.21.15.98:0/2782172836 shutdown_connections 2023-03-03T08:00:59.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.322+0000 7f3c42dff700 1 -- 172.21.15.98:0/2782172836 wait complete. 2023-03-03T08:00:59.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.325+0000 7f3c42dff700 1 Processor -- start 2023-03-03T08:00:59.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.325+0000 7f3c42dff700 1 -- start start 2023-03-03T08:00:59.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.326+0000 7f3c42dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3c0b4c70 0x7f3c3c0b71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:59.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.326+0000 7f3c42dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c3c15c830 0x7f3c3c0c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:59.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.326+0000 7f3c42dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c3c15cc00 0x7f3c3c14a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:59.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.327+0000 7f3c42dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c0c8010 con 0x7f3c3c0b4c70 2023-03-03T08:00:59.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.327+0000 7f3c42dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c14ac60 con 0x7f3c3c15cc00 2023-03-03T08:00:59.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.327+0000 7f3c42dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c0c7400 con 0x7f3c3c15c830 2023-03-03T08:00:59.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.327+0000 7f3c40b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3c0b4c70 0x7f3c3c0b71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:59.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.327+0000 7f3c40b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3c0b4c70 0x7f3c3c0b71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48120/0 (socket says 172.21.15.98:48120) 2023-03-03T08:00:59.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.327+0000 7f3c40b9b700 1 -- 172.21.15.98:0/1892434259 learned_addr learned my addr 172.21.15.98:0/1892434259 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:00:59.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.327+0000 7f3c3bfff700 1 --2- 172.21.15.98:0/1892434259 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c3c15c830 0x7f3c3c0c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:59.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.327+0000 7f3c4139c700 1 --2- 172.21.15.98:0/1892434259 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c3c15cc00 0x7f3c3c14a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:59.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.327+0000 7f3c40b9b700 1 -- 172.21.15.98:0/1892434259 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c3c15c830 msgr2=0x7f3c3c0c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:59.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.327+0000 7f3c40b9b700 1 --2- 172.21.15.98:0/1892434259 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c3c15c830 0x7f3c3c0c6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:59.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.327+0000 7f3c40b9b700 1 -- 172.21.15.98:0/1892434259 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c3c15cc00 msgr2=0x7f3c3c14a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:59.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.327+0000 7f3c40b9b700 1 --2- 172.21.15.98:0/1892434259 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c3c15cc00 0x7f3c3c14a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:59.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.327+0000 7f3c40b9b700 1 -- 172.21.15.98:0/1892434259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c300094a0 con 0x7f3c3c0b4c70 2023-03-03T08:00:59.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.327+0000 7f3c3bfff700 1 --2- 172.21.15.98:0/1892434259 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c3c15c830 0x7f3c3c0c6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:00:59.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.327+0000 7f3c40b9b700 1 --2- 172.21.15.98:0/1892434259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3c0b4c70 0x7f3c3c0b71c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f3c30009fd0 tx=0x7f3c30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:59.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.328+0000 7f3c397fa700 1 -- 172.21.15.98:0/1892434259 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c300038a0 con 0x7f3c3c0b4c70 2023-03-03T08:00:59.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.328+0000 7f3c397fa700 1 -- 172.21.15.98:0/1892434259 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3001f070 con 0x7f3c3c0b4c70 2023-03-03T08:00:59.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.328+0000 7f3c397fa700 1 -- 172.21.15.98:0/1892434259 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30003c40 con 0x7f3c3c0b4c70 2023-03-03T08:00:59.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.328+0000 7f3c42dff700 1 -- 172.21.15.98:0/1892434259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c3c0d1140 con 0x7f3c3c0b4c70 2023-03-03T08:00:59.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.328+0000 7f3c42dff700 1 -- 172.21.15.98:0/1892434259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c3c0d0e60 con 0x7f3c3c0b4c70 2023-03-03T08:00:59.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.329+0000 7f3c42dff700 1 -- 172.21.15.98:0/1892434259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c3c0d2670 con 0x7f3c3c0b4c70 2023-03-03T08:00:59.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.330+0000 7f3c397fa700 1 -- 172.21.15.98:0/1892434259 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f3c3000f050 con 0x7f3c3c0b4c70 2023-03-03T08:00:59.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.332+0000 7f3c397fa700 1 --2- 172.21.15.98:0/1892434259 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3c200684a0 0x7f3c2006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:00:59.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.332+0000 7f3c397fa700 1 -- 172.21.15.98:0/1892434259 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3c30021070 con 0x7f3c3c0b4c70 2023-03-03T08:00:59.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.332+0000 7f3c3bfff700 1 --2- 172.21.15.98:0/1892434259 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3c200684a0 0x7f3c2006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:00:59.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.333+0000 7f3c3bfff700 1 --2- 172.21.15.98:0/1892434259 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3c200684a0 0x7f3c2006a950 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c0b5b80 tx=0x7f3c28008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:00:59.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.336+0000 7f3c397fa700 1 -- 172.21.15.98:0/1892434259 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3c30053f90 con 0x7f3c3c0b4c70 2023-03-03T08:00:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.544+0000 7f3c42dff700 1 -- 172.21.15.98:0/1892434259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3c3c04e6e0 con 0x7f3c3c0b4c70 2023-03-03T08:00:59.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.545+0000 7f3c397fa700 1 -- 172.21.15.98:0/1892434259 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v66) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3c30015080 con 0x7f3c3c0b4c70 2023-03-03T08:00:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.548+0000 7f3c42dff700 1 -- 172.21.15.98:0/1892434259 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3c200684a0 msgr2=0x7f3c2006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.548+0000 7f3c42dff700 1 --2- 172.21.15.98:0/1892434259 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3c200684a0 0x7f3c2006a950 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c0b5b80 tx=0x7f3c28008040 comp rx=0 tx=0).stop 2023-03-03T08:00:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.548+0000 7f3c42dff700 1 -- 172.21.15.98:0/1892434259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3c0b4c70 msgr2=0x7f3c3c0b71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:00:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.548+0000 7f3c42dff700 1 --2- 172.21.15.98:0/1892434259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3c0b4c70 0x7f3c3c0b71c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f3c30009fd0 tx=0x7f3c30004750 comp rx=0 tx=0).stop 2023-03-03T08:00:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.548+0000 7f3c42dff700 1 -- 172.21.15.98:0/1892434259 shutdown_connections 2023-03-03T08:00:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.548+0000 7f3c42dff700 1 --2- 172.21.15.98:0/1892434259 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3c200684a0 0x7f3c2006a950 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.548+0000 7f3c42dff700 1 --2- 172.21.15.98:0/1892434259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3c0b4c70 0x7f3c3c0b71c0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.548+0000 7f3c42dff700 1 --2- 172.21.15.98:0/1892434259 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c3c15cc00 0x7f3c3c14a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.548+0000 7f3c42dff700 1 --2- 172.21.15.98:0/1892434259 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c3c15c830 0x7f3c3c0c6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:00:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.548+0000 7f3c42dff700 1 -- 172.21.15.98:0/1892434259 >> 172.21.15.98:0/1892434259 conn(0x7f3c3c073e70 msgr2=0x7f3c3c15f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:00:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.548+0000 7f3c42dff700 1 -- 172.21.15.98:0/1892434259 shutdown_connections 2023-03-03T08:00:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:00:59.548+0000 7f3c42dff700 1 -- 172.21.15.98:0/1892434259 wait complete. 2023-03-03T08:00:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T08:00:59.577 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2023-03-03T08:00:59.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:59.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-03-03T08:00:59.680 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[118867]: fuse finished with error 0 and tester_r 0 2023-03-03T08:00:59.688 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T08:00:59.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:59.688 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-03-03T08:00:59.760 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:00:59.760 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:00:59.828 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T08:00:59.829 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T08:00:59.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:59.829 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-03-03T08:00:59.896 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-03-03T08:00:59.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:00:59.898 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:00:59.899 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:00:59.966 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T08:00:59.966 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T08:00:59.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:00:59.967 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-03-03T08:01:00.034 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-03-03T08:01:00.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:01:00.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:01:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.281+0000 7f05d0dff700 1 Processor -- start 2023-03-03T08:01:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.281+0000 7f05d0dff700 1 -- start start 2023-03-03T08:01:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.282+0000 7f05d0dff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f05cc0b87c0 0x7f05cc0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.282+0000 7f05d0dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05cc0d0ef0 con 0x7f05cc0b5410 2023-03-03T08:01:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.282+0000 7f05d0dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05cc0d0600 con 0x7f05cc0b8b90 2023-03-03T08:01:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.282+0000 7f05d0dff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05cc0cd970 con 0x7f05cc0b87c0 2023-03-03T08:01:00.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.283+0000 7f05cad9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f05cc0b5410 0x7f05cc0d13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34128/0 (socket says 172.21.15.98:34128) 2023-03-03T08:01:00.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.283+0000 7f05cad9d700 1 -- 172.21.15.98:0/2109447830 learned_addr learned my addr 172.21.15.98:0/2109447830 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:00.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.283+0000 7f05ca59c700 1 --2- 172.21.15.98:0/2109447830 >> v2:172.21.15.100:3300/0 conn(0x7f05cc0b87c0 0x7f05cc0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:00.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.283+0000 7f05c959a700 1 -- 172.21.15.98:0/2109447830 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487332224 0 0) 0x7f05cc0d0ef0 con 0x7f05cc0b5410 2023-03-03T08:01:00.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.283+0000 7f05c959a700 1 -- 172.21.15.98:0/2109447830 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05b40035b0 con 0x7f05cc0b5410 2023-03-03T08:01:00.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.284+0000 7f05ca59c700 1 -- 172.21.15.98:0/2109447830 >> v1:172.21.15.100:6789/0 conn(0x7f05cc0b8b90 legacy=0x7f05cc0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:00.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.284+0000 7f05c959a700 1 -- 172.21.15.98:0/2109447830 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2742776226 0 0) 0x7f05b40035b0 con 0x7f05cc0b5410 2023-03-03T08:01:00.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.284+0000 7f05ca59c700 1 -- 172.21.15.98:0/2109447830 >> v1:172.21.15.98:6789/0 conn(0x7f05cc0b5410 legacy=0x7f05cc0d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:00.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.284+0000 7f05ca59c700 1 -- 172.21.15.98:0/2109447830 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05cc0cdc50 con 0x7f05cc0b87c0 2023-03-03T08:01:00.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.284+0000 7f05ca59c700 1 --2- 172.21.15.98:0/2109447830 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b87c0 0x7f05cc0cffe0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009840 tx=0x7f05bc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2d7e8302e3d791b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:00.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.284+0000 7f05c959a700 1 -- 172.21.15.98:0/2109447830 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c40019f0 con 0x7f05cc0b87c0 2023-03-03T08:01:00.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.284+0000 7f05c959a700 1 -- 172.21.15.98:0/2109447830 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05bc004030 con 0x7f05cc0b87c0 2023-03-03T08:01:00.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.284+0000 7f05c959a700 1 -- 172.21.15.98:0/2109447830 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05bc0043c0 con 0x7f05cc0b87c0 2023-03-03T08:01:00.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.285+0000 7f05d0dff700 1 -- 172.21.15.98:0/2109447830 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b87c0 msgr2=0x7f05cc0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:00.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.285+0000 7f05d0dff700 1 --2- 172.21.15.98:0/2109447830 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b87c0 0x7f05cc0cffe0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009840 tx=0x7f05bc009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:00.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.285+0000 7f05d0dff700 1 -- 172.21.15.98:0/2109447830 shutdown_connections 2023-03-03T08:01:00.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.285+0000 7f05d0dff700 1 --2- 172.21.15.98:0/2109447830 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b87c0 0x7f05cc0cffe0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:00.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.285+0000 7f05d0dff700 1 -- 172.21.15.98:0/2109447830 >> 172.21.15.98:0/2109447830 conn(0x7f05cc06aee0 msgr2=0x7f05cc06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:00.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.285+0000 7f05d0dff700 1 -- 172.21.15.98:0/2109447830 shutdown_connections 2023-03-03T08:01:00.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.285+0000 7f05d0dff700 1 -- 172.21.15.98:0/2109447830 wait complete. 2023-03-03T08:01:00.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.287+0000 7f05d0dff700 1 Processor -- start 2023-03-03T08:01:00.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.287+0000 7f05d0dff700 1 -- start start 2023-03-03T08:01:00.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.288+0000 7f05d0dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05cc0b5410 0x7f05cc1f6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:00.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.288+0000 7f05d0dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b87c0 0x7f05cc1551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:00.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.288+0000 7f05d0dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05cc0b8b90 0x7f05cc1532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:00.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.288+0000 7f05d0dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc149350 con 0x7f05cc0b8b90 2023-03-03T08:01:00.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.288+0000 7f05d0dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc15e200 con 0x7f05cc0b87c0 2023-03-03T08:01:00.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.288+0000 7f05d0dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc155ca0 con 0x7f05cc0b5410 2023-03-03T08:01:00.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.289+0000 7f05cad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05cc0b8b90 0x7f05cc1532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:00.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.289+0000 7f05cad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05cc0b8b90 0x7f05cc1532f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48122/0 (socket says 172.21.15.98:48122) 2023-03-03T08:01:00.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.289+0000 7f05cad9d700 1 -- 172.21.15.98:0/1995238091 learned_addr learned my addr 172.21.15.98:0/1995238091 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:00.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.289+0000 7f05ca59c700 1 --2- 172.21.15.98:0/1995238091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05cc0b5410 0x7f05cc1f6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:00.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.289+0000 7f05c9d9b700 1 --2- 172.21.15.98:0/1995238091 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b87c0 0x7f05cc1551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:00.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.289+0000 7f05cad9d700 1 -- 172.21.15.98:0/1995238091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05cc0b5410 msgr2=0x7f05cc1f6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:00.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.289+0000 7f05cad9d700 1 --2- 172.21.15.98:0/1995238091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05cc0b5410 0x7f05cc1f6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.289+0000 7f05cad9d700 1 -- 172.21.15.98:0/1995238091 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b87c0 msgr2=0x7f05cc1551c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.289+0000 7f05cad9d700 1 --2- 172.21.15.98:0/1995238091 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b87c0 0x7f05cc1551c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.289+0000 7f05cad9d700 1 -- 172.21.15.98:0/1995238091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc0094a0 con 0x7f05cc0b8b90 2023-03-03T08:01:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.289+0000 7f05c9d9b700 1 --2- 172.21.15.98:0/1995238091 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b87c0 0x7f05cc1551c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.290+0000 7f05cad9d700 1 --2- 172.21.15.98:0/1995238091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05cc0b8b90 0x7f05cc1532f0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f05c400cbe0 tx=0x7f05c400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.290+0000 7f05baffd700 1 -- 172.21.15.98:0/1995238091 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c400ab10 con 0x7f05cc0b8b90 2023-03-03T08:01:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.290+0000 7f05baffd700 1 -- 172.21.15.98:0/1995238091 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c400ac90 con 0x7f05cc0b8b90 2023-03-03T08:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.290+0000 7f05baffd700 1 -- 172.21.15.98:0/1995238091 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c40070d0 con 0x7f05cc0b8b90 2023-03-03T08:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.290+0000 7f05d0dff700 1 -- 172.21.15.98:0/1995238091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05cc0cdf30 con 0x7f05cc0b8b90 2023-03-03T08:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.290+0000 7f05d0dff700 1 -- 172.21.15.98:0/1995238091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05cc073a80 con 0x7f05cc0b8b90 2023-03-03T08:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.291+0000 7f05d0dff700 1 -- 172.21.15.98:0/1995238091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05cc0ce0c0 con 0x7f05cc0b8b90 2023-03-03T08:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.293+0000 7f05baffd700 1 -- 172.21.15.98:0/1995238091 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f05c4010070 con 0x7f05cc0b8b90 2023-03-03T08:01:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.294+0000 7f05baffd700 1 --2- 172.21.15.98:0/1995238091 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f05ac0684a0 0x7f05ac06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.294+0000 7f05baffd700 1 -- 172.21.15.98:0/1995238091 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f05c40859b0 con 0x7f05cc0b8b90 2023-03-03T08:01:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.294+0000 7f05ca59c700 1 --2- 172.21.15.98:0/1995238091 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f05ac0684a0 0x7f05ac06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:00.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.295+0000 7f05ca59c700 1 --2- 172.21.15.98:0/1995238091 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f05ac0684a0 0x7f05ac06a950 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f05bc0090b0 tx=0x7f05bc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:00.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.298+0000 7f05baffd700 1 -- 172.21.15.98:0/1995238091 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f05c4051f70 con 0x7f05cc0b8b90 2023-03-03T08:01:00.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.534+0000 7f05d0dff700 1 -- 172.21.15.98:0/1995238091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05cc0729b0 con 0x7f05cc0b8b90 2023-03-03T08:01:00.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.534+0000 7f05baffd700 1 -- 172.21.15.98:0/1995238091 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4649 (secure 0 0 0) 0x7f05c4018090 con 0x7f05cc0b8b90 2023-03-03T08:01:00.536 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:00.536 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5702,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/976526047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":976526047},{"type":"v1","addr":"172.21.15.100:6834","nonce":976526047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3372584689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3372584689},{"type":"v1","addr":"172.21.15.100:6835","nonce":3372584689}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/505711827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":505711827},{"type":"v1","addr":"172.21.15.98:6837","nonce":505711827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:00:22.323396+0000","modified":"2023-03-03T08:00:25.787755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6387":{"gid":6387,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6836/3248147431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3248147431},{"type":"v1","addr":"172.21.15.98:6836","nonce":3248147431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-03-03T08:01:00.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.537+0000 7f05d0dff700 1 -- 172.21.15.98:0/1995238091 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f05ac0684a0 msgr2=0x7f05ac06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:00.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.537+0000 7f05d0dff700 1 --2- 172.21.15.98:0/1995238091 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f05ac0684a0 0x7f05ac06a950 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f05bc0090b0 tx=0x7f05bc008030 comp rx=0 tx=0).stop 2023-03-03T08:01:00.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.537+0000 7f05d0dff700 1 -- 172.21.15.98:0/1995238091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05cc0b8b90 msgr2=0x7f05cc1532f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:00.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.537+0000 7f05d0dff700 1 --2- 172.21.15.98:0/1995238091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05cc0b8b90 0x7f05cc1532f0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f05c400cbe0 tx=0x7f05c400cef0 comp rx=0 tx=0).stop 2023-03-03T08:01:00.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.537+0000 7f05d0dff700 1 -- 172.21.15.98:0/1995238091 shutdown_connections 2023-03-03T08:01:00.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.537+0000 7f05d0dff700 1 --2- 172.21.15.98:0/1995238091 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f05ac0684a0 0x7f05ac06a950 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:00.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.537+0000 7f05d0dff700 1 --2- 172.21.15.98:0/1995238091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05cc0b8b90 0x7f05cc1532f0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:00.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.537+0000 7f05d0dff700 1 --2- 172.21.15.98:0/1995238091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05cc0b5410 0x7f05cc1f6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:00.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.537+0000 7f05d0dff700 1 --2- 172.21.15.98:0/1995238091 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05cc0b87c0 0x7f05cc1551c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:00.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.537+0000 7f05d0dff700 1 -- 172.21.15.98:0/1995238091 >> 172.21.15.98:0/1995238091 conn(0x7f05cc06aee0 msgr2=0x7f05cc0b7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:00.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.537+0000 7f05d0dff700 1 -- 172.21.15.98:0/1995238091 shutdown_connections 2023-03-03T08:01:00.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.537+0000 7f05d0dff700 1 -- 172.21.15.98:0/1995238091 wait complete. 2023-03-03T08:01:00.540 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2023-03-03T08:01:00.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:01:00.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.813+0000 7f705b7ff700 1 Processor -- start 2023-03-03T08:01:00.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.813+0000 7f705b7ff700 1 -- start start 2023-03-03T08:01:00.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.815+0000 7f705b7ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f70540cd8d0 0x7f70540b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:00.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.815+0000 7f705b7ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70540b6610 0x7f70540cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:00.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.815+0000 7f705b7ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70540cdd70 con 0x7f705415cc00 2023-03-03T08:01:00.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.815+0000 7f705b7ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70540ce040 con 0x7f70540b6610 2023-03-03T08:01:00.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.815+0000 7f705b7ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7054073be0 con 0x7f70540cd8d0 2023-03-03T08:01:00.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.815+0000 7f7059d9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70540b6610 0x7f70540cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.815+0000 7f705959b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f705415cc00 0x7f70540d27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34136/0 (socket says 172.21.15.98:34136) 2023-03-03T08:01:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.815+0000 7f705959b700 1 -- 172.21.15.98:0/1621679873 learned_addr learned my addr 172.21.15.98:0/1621679873 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.815+0000 7f7058d9a700 1 --2- 172.21.15.98:0/1621679873 >> v2:172.21.15.100:3300/0 conn(0x7f70540cd8d0 0x7f70540b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.816+0000 7f704bfff700 1 -- 172.21.15.98:0/1621679873 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077968472 0 0) 0x7f70540cdd70 con 0x7f705415cc00 2023-03-03T08:01:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.816+0000 7f7059d9c700 1 -- 172.21.15.98:0/1621679873 >> v2:172.21.15.100:3300/0 conn(0x7f70540cd8d0 msgr2=0x7f70540b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.816+0000 7f7059d9c700 1 --2- 172.21.15.98:0/1621679873 >> v2:172.21.15.100:3300/0 conn(0x7f70540cd8d0 0x7f70540b60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.816+0000 7f7059d9c700 1 -- 172.21.15.98:0/1621679873 >> v1:172.21.15.98:6789/0 conn(0x7f705415cc00 legacy=0x7f70540d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.816+0000 7f7059d9c700 1 -- 172.21.15.98:0/1621679873 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70540d1a00 con 0x7f70540b6610 2023-03-03T08:01:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.816+0000 7f7059d9c700 1 --2- 172.21.15.98:0/1621679873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70540b6610 0x7f70540cf4d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f70540b5e70 tx=0x7f704c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc402499dd5ed25 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.816+0000 7f704bfff700 1 -- 172.21.15.98:0/1621679873 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c00e070 con 0x7f70540b6610 2023-03-03T08:01:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.816+0000 7f704bfff700 1 -- 172.21.15.98:0/1621679873 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704c00bb20 con 0x7f70540b6610 2023-03-03T08:01:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.816+0000 7f704bfff700 1 -- 172.21.15.98:0/1621679873 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c012430 con 0x7f70540b6610 2023-03-03T08:01:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.817+0000 7f705b7ff700 1 -- 172.21.15.98:0/1621679873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70540b6610 msgr2=0x7f70540cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.817+0000 7f705b7ff700 1 --2- 172.21.15.98:0/1621679873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70540b6610 0x7f70540cf4d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f70540b5e70 tx=0x7f704c00b330 comp rx=0 tx=0).stop 2023-03-03T08:01:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.817+0000 7f705b7ff700 1 -- 172.21.15.98:0/1621679873 shutdown_connections 2023-03-03T08:01:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.817+0000 7f705b7ff700 1 --2- 172.21.15.98:0/1621679873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70540b6610 0x7f70540cf4d0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.817+0000 7f705b7ff700 1 --2- 172.21.15.98:0/1621679873 >> v2:172.21.15.100:3300/0 conn(0x7f70540cd8d0 0x7f70540b60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.817+0000 7f705b7ff700 1 -- 172.21.15.98:0/1621679873 >> 172.21.15.98:0/1621679873 conn(0x7f705406aee0 msgr2=0x7f705406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.817+0000 7f705b7ff700 1 -- 172.21.15.98:0/1621679873 shutdown_connections 2023-03-03T08:01:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.817+0000 7f705b7ff700 1 -- 172.21.15.98:0/1621679873 wait complete. 2023-03-03T08:01:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.819+0000 7f705b7ff700 1 Processor -- start 2023-03-03T08:01:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.819+0000 7f705b7ff700 1 -- start start 2023-03-03T08:01:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.820+0000 7f705b7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70540b6610 0x7f70541f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.820+0000 7f705b7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70540cd8d0 0x7f70541f1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.820+0000 7f705b7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f705415cc00 0x7f705415ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.820+0000 7f705b7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705414ff70 con 0x7f70540b6610 2023-03-03T08:01:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.820+0000 7f705b7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70541f2ea0 con 0x7f705415cc00 2023-03-03T08:01:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.820+0000 7f705b7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705415db20 con 0x7f70540cd8d0 2023-03-03T08:01:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.821+0000 7f705959b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70540b6610 0x7f70541f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.821+0000 7f705959b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70540b6610 0x7f70541f2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48142/0 (socket says 172.21.15.98:48142) 2023-03-03T08:01:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.821+0000 7f705959b700 1 -- 172.21.15.98:0/241584138 learned_addr learned my addr 172.21.15.98:0/241584138 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.821+0000 7f7058d9a700 1 --2- 172.21.15.98:0/241584138 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70540cd8d0 0x7f70541f1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.821+0000 7f7059d9c700 1 --2- 172.21.15.98:0/241584138 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f705415cc00 0x7f705415ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.821+0000 7f705959b700 1 -- 172.21.15.98:0/241584138 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70540cd8d0 msgr2=0x7f70541f1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.821+0000 7f705959b700 1 --2- 172.21.15.98:0/241584138 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70540cd8d0 0x7f70541f1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.821+0000 7f705959b700 1 -- 172.21.15.98:0/241584138 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f705415cc00 msgr2=0x7f705415ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.821+0000 7f705959b700 1 --2- 172.21.15.98:0/241584138 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f705415cc00 0x7f705415ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.821+0000 7f705959b700 1 -- 172.21.15.98:0/241584138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704c009a70 con 0x7f70540b6610 2023-03-03T08:01:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.822+0000 7f705959b700 1 --2- 172.21.15.98:0/241584138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70540b6610 0x7f70541f2960 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f70500011e0 tx=0x7f705000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.822+0000 7f7049ffb700 1 -- 172.21.15.98:0/241584138 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7050004c80 con 0x7f70540b6610 2023-03-03T08:01:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.822+0000 7f7049ffb700 1 -- 172.21.15.98:0/241584138 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7050004e00 con 0x7f70540b6610 2023-03-03T08:01:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.822+0000 7f705b7ff700 1 -- 172.21.15.98:0/241584138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70540d1fc0 con 0x7f70540b6610 2023-03-03T08:01:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.822+0000 7f7049ffb700 1 -- 172.21.15.98:0/241584138 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7050007310 con 0x7f70540b6610 2023-03-03T08:01:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.822+0000 7f705b7ff700 1 -- 172.21.15.98:0/241584138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70540d1ce0 con 0x7f70540b6610 2023-03-03T08:01:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.823+0000 7f705b7ff700 1 -- 172.21.15.98:0/241584138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70540d22a0 con 0x7f70540b6610 2023-03-03T08:01:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.825+0000 7f7049ffb700 1 -- 172.21.15.98:0/241584138 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f705001a020 con 0x7f70540b6610 2023-03-03T08:01:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.826+0000 7f7049ffb700 1 --2- 172.21.15.98:0/241584138 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f70380684a0 0x7f703806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.826+0000 7f7049ffb700 1 -- 172.21.15.98:0/241584138 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7050085a20 con 0x7f70540b6610 2023-03-03T08:01:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.826+0000 7f7058d9a700 1 --2- 172.21.15.98:0/241584138 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f70380684a0 0x7f703806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.827+0000 7f7058d9a700 1 --2- 172.21.15.98:0/241584138 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f70380684a0 0x7f703806a950 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f70540ca1f0 tx=0x7f7044009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:00.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:00.829+0000 7f7049ffb700 1 -- 172.21.15.98:0/241584138 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f70500520d0 con 0x7f70540b6610 2023-03-03T08:01:01.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.022+0000 7f705b7ff700 1 -- 172.21.15.98:0/241584138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70540729b0 con 0x7f70540b6610 2023-03-03T08:01:01.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.024+0000 7f7049ffb700 1 -- 172.21.15.98:0/241584138 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+17921 (secure 0 0 0) 0x7f70500556f0 con 0x7f70540b6610 2023-03-03T08:01:01.026 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:01.026 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":66,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:00:23.326015+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:00:19.823120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-03-03T08:00:21.324346+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:01:01.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.026+0000 7f705b7ff700 1 -- 172.21.15.98:0/241584138 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f70380684a0 msgr2=0x7f703806a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:01.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.026+0000 7f705b7ff700 1 --2- 172.21.15.98:0/241584138 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f70380684a0 0x7f703806a950 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f70540ca1f0 tx=0x7f7044009210 comp rx=0 tx=0).stop 2023-03-03T08:01:01.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.026+0000 7f705b7ff700 1 -- 172.21.15.98:0/241584138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70540b6610 msgr2=0x7f70541f2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:01.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.026+0000 7f705b7ff700 1 --2- 172.21.15.98:0/241584138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70540b6610 0x7f70541f2960 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f70500011e0 tx=0x7f705000ca30 comp rx=0 tx=0).stop 2023-03-03T08:01:01.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.026+0000 7f705b7ff700 1 -- 172.21.15.98:0/241584138 shutdown_connections 2023-03-03T08:01:01.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.026+0000 7f705b7ff700 1 --2- 172.21.15.98:0/241584138 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f70380684a0 0x7f703806a950 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.026+0000 7f705b7ff700 1 --2- 172.21.15.98:0/241584138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70540b6610 0x7f70541f2960 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.026+0000 7f705b7ff700 1 --2- 172.21.15.98:0/241584138 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f705415cc00 0x7f705415ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.026+0000 7f705b7ff700 1 --2- 172.21.15.98:0/241584138 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70540cd8d0 0x7f70541f1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.026+0000 7f705b7ff700 1 -- 172.21.15.98:0/241584138 >> 172.21.15.98:0/241584138 conn(0x7f705406aee0 msgr2=0x7f70540b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:01.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.027+0000 7f705b7ff700 1 -- 172.21.15.98:0/241584138 shutdown_connections 2023-03-03T08:01:01.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.027+0000 7f705b7ff700 1 -- 172.21.15.98:0/241584138 wait complete. 2023-03-03T08:01:01.055 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T08:00:19.823120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T08:00:21.324346+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:01:01.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:01:01.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.303+0000 7f3b0e99d700 1 Processor -- start 2023-03-03T08:01:01.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.304+0000 7f3b0e99d700 1 -- start start 2023-03-03T08:01:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.305+0000 7f3b0e99d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b081609b0 0x7f3b08160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.305+0000 7f3b0e99d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3b080b4aa0 0x7f3b080b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.305+0000 7f3b0e99d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b081612c0 con 0x7f3b080cf570 2023-03-03T08:01:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.305+0000 7f3b0e99d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b080cebb0 con 0x7f3b080b4aa0 2023-03-03T08:01:01.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.305+0000 7f3b0e99d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b080b5460 con 0x7f3b081609b0 2023-03-03T08:01:01.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.305+0000 7f3b0e19c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3b080b4aa0 0x7f3b080b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:01.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.305+0000 7f3b0d19a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3b080cf570 0x7f3b0815d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34148/0 (socket says 172.21.15.98:34148) 2023-03-03T08:01:01.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.305+0000 7f3b0d19a700 1 -- 172.21.15.98:0/2269901096 learned_addr learned my addr 172.21.15.98:0/2269901096 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:01.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.306+0000 7f3b0d99b700 1 --2- 172.21.15.98:0/2269901096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b081609b0 0x7f3b08160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:01.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.306+0000 7f3b0e19c700 1 -- 172.21.15.98:0/2269901096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b081609b0 msgr2=0x7f3b08160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:01.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.306+0000 7f3b0e19c700 1 --2- 172.21.15.98:0/2269901096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b081609b0 0x7f3b08160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.306+0000 7f3b0e19c700 1 -- 172.21.15.98:0/2269901096 >> v1:172.21.15.98:6789/0 conn(0x7f3b080cf570 legacy=0x7f3b0815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:01.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.306+0000 7f3b0e19c700 1 -- 172.21.15.98:0/2269901096 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0815dd00 con 0x7f3b080b4aa0 2023-03-03T08:01:01.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.306+0000 7f3b0e19c700 1 --2- 172.21.15.98:0/2269901096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b080b4aa0 0x7f3b080b4f20 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009d90 tx=0x7f3b0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c40a37f3e5bed2d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:01.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.306+0000 7f3b0c999700 1 -- 172.21.15.98:0/2269901096 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0000e070 con 0x7f3b080b4aa0 2023-03-03T08:01:01.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.307+0000 7f3b0c999700 1 -- 172.21.15.98:0/2269901096 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0000bbd0 con 0x7f3b080b4aa0 2023-03-03T08:01:01.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.307+0000 7f3b0c999700 1 -- 172.21.15.98:0/2269901096 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00012430 con 0x7f3b080b4aa0 2023-03-03T08:01:01.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.307+0000 7f3b0e99d700 1 -- 172.21.15.98:0/2269901096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b080b4aa0 msgr2=0x7f3b080b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:01.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.307+0000 7f3b0e99d700 1 --2- 172.21.15.98:0/2269901096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b080b4aa0 0x7f3b080b4f20 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009d90 tx=0x7f3b0000b330 comp rx=0 tx=0).stop 2023-03-03T08:01:01.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.307+0000 7f3b0e99d700 1 -- 172.21.15.98:0/2269901096 shutdown_connections 2023-03-03T08:01:01.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.307+0000 7f3b0e99d700 1 --2- 172.21.15.98:0/2269901096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b080b4aa0 0x7f3b080b4f20 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.307+0000 7f3b0e99d700 1 --2- 172.21.15.98:0/2269901096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b081609b0 0x7f3b08160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.307+0000 7f3b0e99d700 1 -- 172.21.15.98:0/2269901096 >> 172.21.15.98:0/2269901096 conn(0x7f3b0806b880 msgr2=0x7f3b080cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:01.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.307+0000 7f3b0e99d700 1 -- 172.21.15.98:0/2269901096 shutdown_connections 2023-03-03T08:01:01.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.307+0000 7f3b0e99d700 1 -- 172.21.15.98:0/2269901096 wait complete. 2023-03-03T08:01:01.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.309+0000 7f3b0e99d700 1 Processor -- start 2023-03-03T08:01:01.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.309+0000 7f3b0e99d700 1 -- start start 2023-03-03T08:01:01.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.310+0000 7f3b0e99d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b080b4aa0 0x7f3b081f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:01.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.311+0000 7f3b0e99d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b080cf570 0x7f3b080cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:01.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.311+0000 7f3b0e99d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b081609b0 0x7f3b08151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:01.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.311+0000 7f3b0e99d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b081f2480 con 0x7f3b081609b0 2023-03-03T08:01:01.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.311+0000 7f3b0e99d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b080cd010 con 0x7f3b080cf570 2023-03-03T08:01:01.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.311+0000 7f3b0e99d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b080cca50 con 0x7f3b080b4aa0 2023-03-03T08:01:01.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.311+0000 7f3b0e19c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b081609b0 0x7f3b08151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:01.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.311+0000 7f3b0e19c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b081609b0 0x7f3b08151fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48158/0 (socket says 172.21.15.98:48158) 2023-03-03T08:01:01.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.311+0000 7f3b0e19c700 1 -- 172.21.15.98:0/1513335582 learned_addr learned my addr 172.21.15.98:0/1513335582 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:01.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.311+0000 7f3b0d99b700 1 --2- 172.21.15.98:0/1513335582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b080b4aa0 0x7f3b081f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:01.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.311+0000 7f3b0d19a700 1 --2- 172.21.15.98:0/1513335582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b080cf570 0x7f3b080cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:01.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.311+0000 7f3b0e19c700 1 -- 172.21.15.98:0/1513335582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b080b4aa0 msgr2=0x7f3b081f1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:01.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.311+0000 7f3b0e19c700 1 --2- 172.21.15.98:0/1513335582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b080b4aa0 0x7f3b081f1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.311+0000 7f3b0e19c700 1 -- 172.21.15.98:0/1513335582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b080cf570 msgr2=0x7f3b080cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:01.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.311+0000 7f3b0e19c700 1 --2- 172.21.15.98:0/1513335582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b080cf570 0x7f3b080cbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.311+0000 7f3b0e19c700 1 -- 172.21.15.98:0/1513335582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b00009a70 con 0x7f3b081609b0 2023-03-03T08:01:01.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.312+0000 7f3b0e19c700 1 --2- 172.21.15.98:0/1513335582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b081609b0 0x7f3b08151fa0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f3b00014040 tx=0x7f3b00004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:01.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.312+0000 7f3afe7fc700 1 -- 172.21.15.98:0/1513335582 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00012430 con 0x7f3b081609b0 2023-03-03T08:01:01.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.312+0000 7f3afe7fc700 1 -- 172.21.15.98:0/1513335582 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b00020070 con 0x7f3b081609b0 2023-03-03T08:01:01.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.312+0000 7f3b0e99d700 1 -- 172.21.15.98:0/1513335582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b080cda60 con 0x7f3b081609b0 2023-03-03T08:01:01.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.312+0000 7f3afe7fc700 1 -- 172.21.15.98:0/1513335582 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00004280 con 0x7f3b081609b0 2023-03-03T08:01:01.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.312+0000 7f3b0e99d700 1 -- 172.21.15.98:0/1513335582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b080cd780 con 0x7f3b081609b0 2023-03-03T08:01:01.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.313+0000 7f3b0e99d700 1 -- 172.21.15.98:0/1513335582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b080cdd40 con 0x7f3b081609b0 2023-03-03T08:01:01.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.315+0000 7f3afe7fc700 1 -- 172.21.15.98:0/1513335582 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f3b0002e070 con 0x7f3b081609b0 2023-03-03T08:01:01.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.317+0000 7f3afe7fc700 1 --2- 172.21.15.98:0/1513335582 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3aec0684a0 0x7f3aec06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:01.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.317+0000 7f3afe7fc700 1 -- 172.21.15.98:0/1513335582 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3b00022070 con 0x7f3b081609b0 2023-03-03T08:01:01.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.317+0000 7f3b0d99b700 1 --2- 172.21.15.98:0/1513335582 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3aec0684a0 0x7f3aec06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:01.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.318+0000 7f3b0d99b700 1 --2- 172.21.15.98:0/1513335582 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3aec0684a0 0x7f3aec06a950 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400b9b0 tx=0x7f3b04008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:01.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.321+0000 7f3afe7fc700 1 -- 172.21.15.98:0/1513335582 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3b000543a0 con 0x7f3b081609b0 2023-03-03T08:01:01.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.515+0000 7f3b0e99d700 1 -- 172.21.15.98:0/1513335582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3b08068290 con 0x7f3b081609b0 2023-03-03T08:01:01.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.515+0000 7f3afe7fc700 1 -- 172.21.15.98:0/1513335582 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3b00087e60 con 0x7f3b081609b0 2023-03-03T08:01:01.517 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:01:01.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.518+0000 7f3b0e99d700 1 -- 172.21.15.98:0/1513335582 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3aec0684a0 msgr2=0x7f3aec06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:01.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.518+0000 7f3b0e99d700 1 --2- 172.21.15.98:0/1513335582 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3aec0684a0 0x7f3aec06a950 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400b9b0 tx=0x7f3b04008040 comp rx=0 tx=0).stop 2023-03-03T08:01:01.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.518+0000 7f3b0e99d700 1 -- 172.21.15.98:0/1513335582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b081609b0 msgr2=0x7f3b08151fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:01.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.518+0000 7f3b0e99d700 1 --2- 172.21.15.98:0/1513335582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b081609b0 0x7f3b08151fa0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f3b00014040 tx=0x7f3b00004ad0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.518+0000 7f3b0e99d700 1 -- 172.21.15.98:0/1513335582 shutdown_connections 2023-03-03T08:01:01.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.518+0000 7f3b0e99d700 1 --2- 172.21.15.98:0/1513335582 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3aec0684a0 0x7f3aec06a950 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.518+0000 7f3b0e99d700 1 --2- 172.21.15.98:0/1513335582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b081609b0 0x7f3b08151fa0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.518+0000 7f3b0e99d700 1 --2- 172.21.15.98:0/1513335582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b080b4aa0 0x7f3b081f1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.518+0000 7f3b0e99d700 1 --2- 172.21.15.98:0/1513335582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b080cf570 0x7f3b080cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.518+0000 7f3b0e99d700 1 -- 172.21.15.98:0/1513335582 >> 172.21.15.98:0/1513335582 conn(0x7f3b0806b880 msgr2=0x7f3b080d23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:01.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.518+0000 7f3b0e99d700 1 -- 172.21.15.98:0/1513335582 shutdown_connections 2023-03-03T08:01:01.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.518+0000 7f3b0e99d700 1 -- 172.21.15.98:0/1513335582 wait complete. 2023-03-03T08:01:01.546 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:01:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.804+0000 7f58075ff700 1 Processor -- start 2023-03-03T08:01:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.804+0000 7f58075ff700 1 -- start start 2023-03-03T08:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.806+0000 7f58075ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f58000b0800 0x7f580015dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.806+0000 7f58075ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f58000c93d0 0x7f58000b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.806+0000 7f58075ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58000b1110 con 0x7f58000cd5c0 2023-03-03T08:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.806+0000 7f58075ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58000c9d50 con 0x7f58000c93d0 2023-03-03T08:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.806+0000 7f58075ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58000ca030 con 0x7f58000b0800 2023-03-03T08:01:01.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.806+0000 7f5804b9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f58000c93d0 0x7f58000b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:01.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.806+0000 7f5805b9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f58000cd5c0 0x7f580006c930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34160/0 (socket says 172.21.15.98:34160) 2023-03-03T08:01:01.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.806+0000 7f5805b9c700 1 -- 172.21.15.98:0/3424328627 learned_addr learned my addr 172.21.15.98:0/3424328627 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.806+0000 7f580539b700 1 --2- 172.21.15.98:0/3424328627 >> v2:172.21.15.100:3300/0 conn(0x7f58000b0800 0x7f580015dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.807+0000 7f5804b9a700 1 -- 172.21.15.98:0/3424328627 >> v2:172.21.15.100:3300/0 conn(0x7f58000b0800 msgr2=0x7f580015dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.807+0000 7f5804b9a700 1 --2- 172.21.15.98:0/3424328627 >> v2:172.21.15.100:3300/0 conn(0x7f58000b0800 0x7f580015dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.807+0000 7f5804b9a700 1 -- 172.21.15.98:0/3424328627 >> v1:172.21.15.98:6789/0 conn(0x7f58000cd5c0 legacy=0x7f580006c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.807+0000 7f5804b9a700 1 -- 172.21.15.98:0/3424328627 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58000ccdc0 con 0x7f58000c93d0 2023-03-03T08:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.807+0000 7f5804b9a700 1 --2- 172.21.15.98:0/3424328627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58000c93d0 0x7f58000b0bd0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f58000b1970 tx=0x7f57f0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a21c047d3267f69b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.807+0000 7f57f7fff700 1 -- 172.21.15.98:0/3424328627 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f0004230 con 0x7f58000c93d0 2023-03-03T08:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.807+0000 7f57f7fff700 1 -- 172.21.15.98:0/3424328627 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f0004390 con 0x7f58000c93d0 2023-03-03T08:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.807+0000 7f57f7fff700 1 -- 172.21.15.98:0/3424328627 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f00036a0 con 0x7f58000c93d0 2023-03-03T08:01:01.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.808+0000 7f58075ff700 1 -- 172.21.15.98:0/3424328627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58000c93d0 msgr2=0x7f58000b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:01.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.808+0000 7f58075ff700 1 --2- 172.21.15.98:0/3424328627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58000c93d0 0x7f58000b0bd0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f58000b1970 tx=0x7f57f0009b20 comp rx=0 tx=0).stop 2023-03-03T08:01:01.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.808+0000 7f58075ff700 1 -- 172.21.15.98:0/3424328627 shutdown_connections 2023-03-03T08:01:01.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.808+0000 7f58075ff700 1 --2- 172.21.15.98:0/3424328627 >> v2:172.21.15.100:3300/0 conn(0x7f58000b0800 0x7f580015dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.808+0000 7f58075ff700 1 --2- 172.21.15.98:0/3424328627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58000c93d0 0x7f58000b0bd0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.808+0000 7f58075ff700 1 -- 172.21.15.98:0/3424328627 >> 172.21.15.98:0/3424328627 conn(0x7f58000cb800 msgr2=0x7f58000cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:01.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.808+0000 7f58075ff700 1 -- 172.21.15.98:0/3424328627 shutdown_connections 2023-03-03T08:01:01.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.808+0000 7f58075ff700 1 -- 172.21.15.98:0/3424328627 wait complete. 2023-03-03T08:01:01.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.810+0000 7f58075ff700 1 Processor -- start 2023-03-03T08:01:01.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.810+0000 7f58075ff700 1 -- start start 2023-03-03T08:01:01.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.811+0000 7f58075ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58000b0800 0x7f5800152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:01.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.811+0000 7f58075ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58000c93d0 0x7f5800157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:01.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.812+0000 7f58075ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58000cd5c0 0x7f58000ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:01.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.812+0000 7f58075ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58000c28b0 con 0x7f58000b0800 2023-03-03T08:01:01.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.812+0000 7f58075ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58000c2ac0 con 0x7f58000c93d0 2023-03-03T08:01:01.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.812+0000 7f58075ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58000c2da0 con 0x7f58000cd5c0 2023-03-03T08:01:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.812+0000 7f580539b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58000b0800 0x7f5800152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.812+0000 7f580539b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58000b0800 0x7f5800152730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48168/0 (socket says 172.21.15.98:48168) 2023-03-03T08:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.812+0000 7f580539b700 1 -- 172.21.15.98:0/2509519336 learned_addr learned my addr 172.21.15.98:0/2509519336 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.812+0000 7f5804b9a700 1 --2- 172.21.15.98:0/2509519336 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58000c93d0 0x7f5800157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.812+0000 7f5805b9c700 1 --2- 172.21.15.98:0/2509519336 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58000cd5c0 0x7f58000ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.812+0000 7f580539b700 1 -- 172.21.15.98:0/2509519336 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58000cd5c0 msgr2=0x7f58000ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.812+0000 7f580539b700 1 --2- 172.21.15.98:0/2509519336 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58000cd5c0 0x7f58000ad920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.812+0000 7f580539b700 1 -- 172.21.15.98:0/2509519336 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58000c93d0 msgr2=0x7f5800157a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.812+0000 7f580539b700 1 --2- 172.21.15.98:0/2509519336 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58000c93d0 0x7f5800157a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.812+0000 7f580539b700 1 -- 172.21.15.98:0/2509519336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f00094a0 con 0x7f58000b0800 2023-03-03T08:01:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.812+0000 7f5804b9a700 1 --2- 172.21.15.98:0/2509519336 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58000c93d0 0x7f5800157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.813+0000 7f580539b700 1 --2- 172.21.15.98:0/2509519336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58000b0800 0x7f5800152730 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00e9b0 tx=0x7f57fc00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.813+0000 7f57f5ffb700 1 -- 172.21.15.98:0/2509519336 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc009710 con 0x7f58000b0800 2023-03-03T08:01:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.813+0000 7f57f5ffb700 1 -- 172.21.15.98:0/2509519336 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57fc00ee50 con 0x7f58000b0800 2023-03-03T08:01:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.813+0000 7f57f5ffb700 1 -- 172.21.15.98:0/2509519336 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc00ca50 con 0x7f58000b0800 2023-03-03T08:01:01.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.813+0000 7f58075ff700 1 -- 172.21.15.98:0/2509519336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58000c9820 con 0x7f58000b0800 2023-03-03T08:01:01.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.813+0000 7f58075ff700 1 -- 172.21.15.98:0/2509519336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58000cd0a0 con 0x7f58000b0800 2023-03-03T08:01:01.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.814+0000 7f58075ff700 1 -- 172.21.15.98:0/2509519336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58000ca310 con 0x7f58000b0800 2023-03-03T08:01:01.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.816+0000 7f57f5ffb700 1 -- 172.21.15.98:0/2509519336 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f57fc019030 con 0x7f58000b0800 2023-03-03T08:01:01.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.817+0000 7f57f5ffb700 1 --2- 172.21.15.98:0/2509519336 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f57e40684a0 0x7f57e406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:01.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.818+0000 7f57f5ffb700 1 -- 172.21.15.98:0/2509519336 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f57fc085da0 con 0x7f58000b0800 2023-03-03T08:01:01.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.818+0000 7f5804b9a700 1 --2- 172.21.15.98:0/2509519336 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f57e40684a0 0x7f57e406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:01.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.818+0000 7f5804b9a700 1 --2- 172.21.15.98:0/2509519336 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f57e40684a0 0x7f57e406a950 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f57f0014040 tx=0x7f57f0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:01.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:01.821+0000 7f57f5ffb700 1 -- 172.21.15.98:0/2509519336 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f57fc055e90 con 0x7f58000b0800 2023-03-03T08:01:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.015+0000 7f58075ff700 1 -- 172.21.15.98:0/2509519336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f58001583a0 con 0x7f58000b0800 2023-03-03T08:01:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.015+0000 7f57f5ffb700 1 -- 172.21.15.98:0/2509519336 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v66) v1 ==== 92+0+10 (secure 0 0 0) 0x7f57fc0170a0 con 0x7f58000b0800 2023-03-03T08:01:02.017 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:01:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.018+0000 7f58075ff700 1 -- 172.21.15.98:0/2509519336 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f57e40684a0 msgr2=0x7f57e406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.018+0000 7f58075ff700 1 --2- 172.21.15.98:0/2509519336 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f57e40684a0 0x7f57e406a950 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f57f0014040 tx=0x7f57f0008030 comp rx=0 tx=0).stop 2023-03-03T08:01:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.018+0000 7f58075ff700 1 -- 172.21.15.98:0/2509519336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58000b0800 msgr2=0x7f5800152730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.018+0000 7f58075ff700 1 --2- 172.21.15.98:0/2509519336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58000b0800 0x7f5800152730 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00e9b0 tx=0x7f57fc00ecc0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.018+0000 7f58075ff700 1 -- 172.21.15.98:0/2509519336 shutdown_connections 2023-03-03T08:01:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.018+0000 7f58075ff700 1 --2- 172.21.15.98:0/2509519336 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f57e40684a0 0x7f57e406a950 secure :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f57f0014040 tx=0x7f57f0008030 comp rx=0 tx=0).stop 2023-03-03T08:01:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.018+0000 7f58075ff700 1 --2- 172.21.15.98:0/2509519336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58000b0800 0x7f5800152730 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.018+0000 7f58075ff700 1 --2- 172.21.15.98:0/2509519336 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58000cd5c0 0x7f58000ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.018+0000 7f58075ff700 1 --2- 172.21.15.98:0/2509519336 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58000c93d0 0x7f5800157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.018+0000 7f58075ff700 1 -- 172.21.15.98:0/2509519336 >> 172.21.15.98:0/2509519336 conn(0x7f58000cb800 msgr2=0x7f58000b0050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.018+0000 7f58075ff700 1 -- 172.21.15.98:0/2509519336 shutdown_connections 2023-03-03T08:01:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.018+0000 7f58075ff700 1 -- 172.21.15.98:0/2509519336 wait complete. 2023-03-03T08:01:02.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T08:01:02.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.297+0000 7ffab69ff700 1 Processor -- start 2023-03-03T08:01:02.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.297+0000 7ffab69ff700 1 -- start start 2023-03-03T08:01:02.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.299+0000 7ffab69ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffab015dd20 0x7ffab00b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:02.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.299+0000 7ffab69ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ffab00cd5c0 0x7ffab01599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:02.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.299+0000 7ffab69ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab015e170 con 0x7ffab00b0800 2023-03-03T08:01:02.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.299+0000 7ffab69ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffab00b1110 con 0x7ffab015dd20 2023-03-03T08:01:02.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.299+0000 7ffab69ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffab00c9d50 con 0x7ffab00cd5c0 2023-03-03T08:01:02.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.299+0000 7ffaaf7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffab015dd20 0x7ffab00b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:02.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.299+0000 7ffaaf7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffab015dd20 0x7ffab00b0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48176/0 (socket says 172.21.15.98:48176) 2023-03-03T08:01:02.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.299+0000 7ffaaf7fe700 1 -- 172.21.15.98:0/3178654433 learned_addr learned my addr 172.21.15.98:0/3178654433 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:02.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.299+0000 7ffab4f9c700 1 --2- 172.21.15.98:0/3178654433 >> v2:172.21.15.100:3300/0 conn(0x7ffab00cd5c0 0x7ffab01599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:02.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.300+0000 7ffaaf7fe700 1 -- 172.21.15.98:0/3178654433 >> v2:172.21.15.100:3300/0 conn(0x7ffab00cd5c0 msgr2=0x7ffab01599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:02.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.300+0000 7ffaaf7fe700 1 --2- 172.21.15.98:0/3178654433 >> v2:172.21.15.100:3300/0 conn(0x7ffab00cd5c0 0x7ffab01599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.300+0000 7ffaaf7fe700 1 -- 172.21.15.98:0/3178654433 >> v1:172.21.15.100:6789/0 conn(0x7ffab00b0800 legacy=0x7ffab00c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:02.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.300+0000 7ffaaf7fe700 1 -- 172.21.15.98:0/3178654433 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab00ca030 con 0x7ffab015dd20 2023-03-03T08:01:02.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.300+0000 7ffaaf7fe700 1 --2- 172.21.15.98:0/3178654433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab015dd20 0x7ffab00b0bd0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7ffab006da80 tx=0x7ffaa4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff389f12bfc88c6c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:02.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.300+0000 7ffaaeffd700 1 -- 172.21.15.98:0/3178654433 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4011830 con 0x7ffab015dd20 2023-03-03T08:01:02.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.301+0000 7ffaaeffd700 1 -- 172.21.15.98:0/3178654433 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa4011990 con 0x7ffab015dd20 2023-03-03T08:01:02.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.301+0000 7ffaaeffd700 1 -- 172.21.15.98:0/3178654433 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4011d20 con 0x7ffab015dd20 2023-03-03T08:01:02.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.301+0000 7ffab69ff700 1 -- 172.21.15.98:0/3178654433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab015dd20 msgr2=0x7ffab00b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:02.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.301+0000 7ffab69ff700 1 --2- 172.21.15.98:0/3178654433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab015dd20 0x7ffab00b0bd0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7ffab006da80 tx=0x7ffaa4009bd0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.301+0000 7ffab69ff700 1 -- 172.21.15.98:0/3178654433 shutdown_connections 2023-03-03T08:01:02.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.301+0000 7ffab69ff700 1 --2- 172.21.15.98:0/3178654433 >> v2:172.21.15.100:3300/0 conn(0x7ffab00cd5c0 0x7ffab01599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.301+0000 7ffab69ff700 1 --2- 172.21.15.98:0/3178654433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab015dd20 0x7ffab00b0bd0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.301+0000 7ffab69ff700 1 -- 172.21.15.98:0/3178654433 >> 172.21.15.98:0/3178654433 conn(0x7ffab00cb800 msgr2=0x7ffab00cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.302+0000 7ffab69ff700 1 -- 172.21.15.98:0/3178654433 shutdown_connections 2023-03-03T08:01:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.302+0000 7ffab69ff700 1 -- 172.21.15.98:0/3178654433 wait complete. 2023-03-03T08:01:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.304+0000 7ffab69ff700 1 Processor -- start 2023-03-03T08:01:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.304+0000 7ffab69ff700 1 -- start start 2023-03-03T08:01:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.305+0000 7ffab69ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffab00b0800 0x7ffab01532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.305+0000 7ffab69ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffab00cd5c0 0x7ffab00c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.305+0000 7ffab69ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab015dd20 0x7ffab00c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.305+0000 7ffab69ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00c86f0 con 0x7ffab015dd20 2023-03-03T08:01:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.305+0000 7ffab69ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00b2940 con 0x7ffab00b0800 2023-03-03T08:01:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.305+0000 7ffab69ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab0153810 con 0x7ffab00cd5c0 2023-03-03T08:01:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.305+0000 7ffab4f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab015dd20 0x7ffab00c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.305+0000 7ffab4f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab015dd20 0x7ffab00c81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48188/0 (socket says 172.21.15.98:48188) 2023-03-03T08:01:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.305+0000 7ffab4f9c700 1 -- 172.21.15.98:0/1989368288 learned_addr learned my addr 172.21.15.98:0/1989368288 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.305+0000 7ffaaffff700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffab00b0800 0x7ffab01532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.305+0000 7ffaaf7fe700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffab00cd5c0 0x7ffab00c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.306+0000 7ffab4f9c700 1 -- 172.21.15.98:0/1989368288 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffab00cd5c0 msgr2=0x7ffab00c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.306+0000 7ffab4f9c700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffab00cd5c0 0x7ffab00c7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.306+0000 7ffab4f9c700 1 -- 172.21.15.98:0/1989368288 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffab00b0800 msgr2=0x7ffab01532d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.306+0000 7ffab4f9c700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffab00b0800 0x7ffab01532d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.306+0000 7ffab4f9c700 1 -- 172.21.15.98:0/1989368288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa4009550 con 0x7ffab015dd20 2023-03-03T08:01:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.306+0000 7ffaaf7fe700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffab00cd5c0 0x7ffab00c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.306+0000 7ffaaffff700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffab00b0800 0x7ffab01532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.306+0000 7ffab4f9c700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab015dd20 0x7ffab00c81b0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800efc0 tx=0x7ffaa800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.306+0000 7ffaacff9700 1 -- 172.21.15.98:0/1989368288 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa800cd40 con 0x7ffab015dd20 2023-03-03T08:01:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.306+0000 7ffaacff9700 1 -- 172.21.15.98:0/1989368288 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa800f040 con 0x7ffab015dd20 2023-03-03T08:01:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.306+0000 7ffab69ff700 1 -- 172.21.15.98:0/1989368288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab00cd0a0 con 0x7ffab015dd20 2023-03-03T08:01:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.306+0000 7ffaacff9700 1 -- 172.21.15.98:0/1989368288 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa80146c0 con 0x7ffab015dd20 2023-03-03T08:01:02.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.306+0000 7ffab69ff700 1 -- 172.21.15.98:0/1989368288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab00ccdc0 con 0x7ffab015dd20 2023-03-03T08:01:02.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.308+0000 7ffab69ff700 1 -- 172.21.15.98:0/1989368288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab00ca310 con 0x7ffab015dd20 2023-03-03T08:01:02.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.309+0000 7ffaacff9700 1 -- 172.21.15.98:0/1989368288 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7ffaa8014820 con 0x7ffab015dd20 2023-03-03T08:01:02.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.310+0000 7ffaacff9700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa9c0684a0 0x7ffa9c06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.310+0000 7ffaacff9700 1 -- 172.21.15.98:0/1989368288 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ffaa8086990 con 0x7ffab015dd20 2023-03-03T08:01:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.310+0000 7ffaaffff700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa9c0684a0 0x7ffa9c06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.311+0000 7ffaaffff700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa9c0684a0 0x7ffa9c06a950 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ffab00c9280 tx=0x7ffaa0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:02.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.314+0000 7ffaacff9700 1 -- 172.21.15.98:0/1989368288 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffaa8056a80 con 0x7ffab015dd20 2023-03-03T08:01:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.513+0000 7ffab69ff700 1 -- 172.21.15.98:0/1989368288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffab004e6e0 con 0x7ffab015dd20 2023-03-03T08:01:02.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.514+0000 7ffaacff9700 1 -- 172.21.15.98:0/1989368288 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v66) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffaa800cea0 con 0x7ffab015dd20 2023-03-03T08:01:02.515 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T08:01:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.517+0000 7ffab69ff700 1 -- 172.21.15.98:0/1989368288 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa9c0684a0 msgr2=0x7ffa9c06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.517+0000 7ffab69ff700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa9c0684a0 0x7ffa9c06a950 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ffab00c9280 tx=0x7ffaa0009040 comp rx=0 tx=0).stop 2023-03-03T08:01:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.517+0000 7ffab69ff700 1 -- 172.21.15.98:0/1989368288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab015dd20 msgr2=0x7ffab00c81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.517+0000 7ffab69ff700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab015dd20 0x7ffab00c81b0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800efc0 tx=0x7ffaa800c3a0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.517+0000 7ffab69ff700 1 -- 172.21.15.98:0/1989368288 shutdown_connections 2023-03-03T08:01:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.517+0000 7ffab69ff700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ffa9c0684a0 0x7ffa9c06a950 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.517+0000 7ffab69ff700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab015dd20 0x7ffab00c81b0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.517+0000 7ffab69ff700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffab00b0800 0x7ffab01532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.517+0000 7ffab69ff700 1 --2- 172.21.15.98:0/1989368288 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffab00cd5c0 0x7ffab00c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.517+0000 7ffab69ff700 1 -- 172.21.15.98:0/1989368288 >> 172.21.15.98:0/1989368288 conn(0x7ffab00cb800 msgr2=0x7ffab00af380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.517+0000 7ffab69ff700 1 -- 172.21.15.98:0/1989368288 shutdown_connections 2023-03-03T08:01:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.517+0000 7ffab69ff700 1 -- 172.21.15.98:0/1989368288 wait complete. 2023-03-03T08:01:02.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T08:01:02.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.795+0000 7f86dd3ff700 1 Processor -- start 2023-03-03T08:01:02.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.795+0000 7f86dd3ff700 1 -- start start 2023-03-03T08:01:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.797+0000 7f86dd3ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f86d815ca70 0x7f86d80cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.797+0000 7f86dd3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86d806b4b0 0x7f86d80cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.797+0000 7f86dd3ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d80cf980 con 0x7f86d80d00a0 2023-03-03T08:01:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.797+0000 7f86dd3ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d80cd5b0 con 0x7f86d815ca70 2023-03-03T08:01:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.797+0000 7f86dd3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d80d0510 con 0x7f86d806b4b0 2023-03-03T08:01:02.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.797+0000 7f86d77fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f86d80d00a0 0x7f86d80b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34166/0 (socket says 172.21.15.98:34166) 2023-03-03T08:01:02.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.797+0000 7f86d77fe700 1 -- 172.21.15.98:0/2972309141 learned_addr learned my addr 172.21.15.98:0/2972309141 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:02.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.797+0000 7f86d6ffd700 1 --2- 172.21.15.98:0/2972309141 >> v2:172.21.15.100:3300/0 conn(0x7f86d815ca70 0x7f86d80cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:02.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.797+0000 7f86d67fc700 1 --2- 172.21.15.98:0/2972309141 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86d806b4b0 0x7f86d80cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:02.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.798+0000 7f86d5ffb700 1 -- 172.21.15.98:0/2972309141 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990940281 0 0) 0x7f86d80cf980 con 0x7f86d80d00a0 2023-03-03T08:01:02.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.798+0000 7f86d5ffb700 1 -- 172.21.15.98:0/2972309141 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c00035b0 con 0x7f86d80d00a0 2023-03-03T08:01:02.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.798+0000 7f86d6ffd700 1 -- 172.21.15.98:0/2972309141 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86d806b4b0 msgr2=0x7f86d80cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:02.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.798+0000 7f86d6ffd700 1 --2- 172.21.15.98:0/2972309141 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86d806b4b0 0x7f86d80cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.798+0000 7f86d6ffd700 1 -- 172.21.15.98:0/2972309141 >> v1:172.21.15.98:6789/0 conn(0x7f86d80d00a0 legacy=0x7f86d80b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:02.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.798+0000 7f86d6ffd700 1 -- 172.21.15.98:0/2972309141 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d80d2fd0 con 0x7f86d815ca70 2023-03-03T08:01:02.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.798+0000 7f86d6ffd700 1 --2- 172.21.15.98:0/2972309141 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86d815ca70 0x7f86d80cf440 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009840 tx=0x7f86c8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbf19483f2d84eb9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:02.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.798+0000 7f86d5ffb700 1 -- 172.21.15.98:0/2972309141 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c8011830 con 0x7f86d815ca70 2023-03-03T08:01:02.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.799+0000 7f86d5ffb700 1 -- 172.21.15.98:0/2972309141 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c8011990 con 0x7f86d815ca70 2023-03-03T08:01:02.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.799+0000 7f86d5ffb700 1 -- 172.21.15.98:0/2972309141 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c8011d20 con 0x7f86d815ca70 2023-03-03T08:01:02.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.799+0000 7f86dd3ff700 1 -- 172.21.15.98:0/2972309141 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86d815ca70 msgr2=0x7f86d80cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:02.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.799+0000 7f86dd3ff700 1 --2- 172.21.15.98:0/2972309141 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86d815ca70 0x7f86d80cf440 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009840 tx=0x7f86c8009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:02.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.799+0000 7f86dd3ff700 1 -- 172.21.15.98:0/2972309141 shutdown_connections 2023-03-03T08:01:02.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.799+0000 7f86dd3ff700 1 --2- 172.21.15.98:0/2972309141 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86d815ca70 0x7f86d80cf440 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.799+0000 7f86dd3ff700 1 --2- 172.21.15.98:0/2972309141 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86d806b4b0 0x7f86d80cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.799+0000 7f86dd3ff700 1 -- 172.21.15.98:0/2972309141 >> 172.21.15.98:0/2972309141 conn(0x7f86d8073e70 msgr2=0x7f86d806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:02.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.799+0000 7f86dd3ff700 1 -- 172.21.15.98:0/2972309141 shutdown_connections 2023-03-03T08:01:02.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.799+0000 7f86dd3ff700 1 -- 172.21.15.98:0/2972309141 wait complete. 2023-03-03T08:01:02.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.801+0000 7f86dd3ff700 1 Processor -- start 2023-03-03T08:01:02.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.802+0000 7f86dd3ff700 1 -- start start 2023-03-03T08:01:02.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.803+0000 7f86dd3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d806b4b0 0x7f86d81f2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:02.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.803+0000 7f86dd3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86d80d00a0 0x7f86d815b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:02.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.803+0000 7f86dd3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86d815ca70 0x7f86d815f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:02.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.803+0000 7f86dd3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d80ca2f0 con 0x7f86d806b4b0 2023-03-03T08:01:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.803+0000 7f86dd3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d80ca5d0 con 0x7f86d80d00a0 2023-03-03T08:01:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.803+0000 7f86dd3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d80ca8b0 con 0x7f86d815ca70 2023-03-03T08:01:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.803+0000 7f86d6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d806b4b0 0x7f86d81f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.803+0000 7f86d6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d806b4b0 0x7f86d81f2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48190/0 (socket says 172.21.15.98:48190) 2023-03-03T08:01:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.803+0000 7f86d6ffd700 1 -- 172.21.15.98:0/3076908132 learned_addr learned my addr 172.21.15.98:0/3076908132 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:02.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.803+0000 7f86d67fc700 1 --2- 172.21.15.98:0/3076908132 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86d80d00a0 0x7f86d815b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:02.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.803+0000 7f86d77fe700 1 --2- 172.21.15.98:0/3076908132 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86d815ca70 0x7f86d815f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:02.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.804+0000 7f86d6ffd700 1 -- 172.21.15.98:0/3076908132 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86d815ca70 msgr2=0x7f86d815f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:02.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.804+0000 7f86d6ffd700 1 --2- 172.21.15.98:0/3076908132 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86d815ca70 0x7f86d815f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.804+0000 7f86d6ffd700 1 -- 172.21.15.98:0/3076908132 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86d80d00a0 msgr2=0x7f86d815b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:02.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.804+0000 7f86d6ffd700 1 --2- 172.21.15.98:0/3076908132 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86d80d00a0 0x7f86d815b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:02.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.804+0000 7f86d6ffd700 1 -- 172.21.15.98:0/3076908132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c80094a0 con 0x7f86d806b4b0 2023-03-03T08:01:02.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.804+0000 7f86d6ffd700 1 --2- 172.21.15.98:0/3076908132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d806b4b0 0x7f86d81f2900 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009fd0 tx=0x7f86c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:02.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.804+0000 7f86bf7fe700 1 -- 172.21.15.98:0/3076908132 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c8004590 con 0x7f86d806b4b0 2023-03-03T08:01:02.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.804+0000 7f86bf7fe700 1 -- 172.21.15.98:0/3076908132 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c801e070 con 0x7f86d806b4b0 2023-03-03T08:01:02.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.804+0000 7f86bf7fe700 1 -- 172.21.15.98:0/3076908132 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c80038e0 con 0x7f86d806b4b0 2023-03-03T08:01:02.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.804+0000 7f86dd3ff700 1 -- 172.21.15.98:0/3076908132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d80cd900 con 0x7f86d806b4b0 2023-03-03T08:01:02.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.805+0000 7f86dd3ff700 1 -- 172.21.15.98:0/3076908132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d80d1800 con 0x7f86d806b4b0 2023-03-03T08:01:02.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.806+0000 7f86dd3ff700 1 -- 172.21.15.98:0/3076908132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d80cdbe0 con 0x7f86d806b4b0 2023-03-03T08:01:02.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.807+0000 7f86bf7fe700 1 -- 172.21.15.98:0/3076908132 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f86c800f050 con 0x7f86d806b4b0 2023-03-03T08:01:02.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.808+0000 7f86bf7fe700 1 --2- 172.21.15.98:0/3076908132 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f86c40684a0 0x7f86c406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:02.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.808+0000 7f86bf7fe700 1 -- 172.21.15.98:0/3076908132 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f86c8086db0 con 0x7f86d806b4b0 2023-03-03T08:01:02.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.808+0000 7f86d67fc700 1 --2- 172.21.15.98:0/3076908132 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f86c40684a0 0x7f86c406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:02.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.809+0000 7f86d67fc700 1 --2- 172.21.15.98:0/3076908132 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f86c40684a0 0x7f86c406a950 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f86d81e7fe0 tx=0x7f86cc009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:02.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:02.812+0000 7f86bf7fe700 1 -- 172.21.15.98:0/3076908132 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f86c808b070 con 0x7f86d806b4b0 2023-03-03T08:01:03.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.008+0000 7f86dd3ff700 1 -- 172.21.15.98:0/3076908132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f86d804e6e0 con 0x7f86d806b4b0 2023-03-03T08:01:03.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.008+0000 7f86bf7fe700 1 -- 172.21.15.98:0/3076908132 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v66) v1 ==== 100+0+11 (secure 0 0 0) 0x7f86c8015020 con 0x7f86d806b4b0 2023-03-03T08:01:03.010 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T08:01:03.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.011+0000 7f86dd3ff700 1 -- 172.21.15.98:0/3076908132 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f86c40684a0 msgr2=0x7f86c406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:03.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.011+0000 7f86dd3ff700 1 --2- 172.21.15.98:0/3076908132 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f86c40684a0 0x7f86c406a950 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f86d81e7fe0 tx=0x7f86cc009210 comp rx=0 tx=0).stop 2023-03-03T08:01:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.011+0000 7f86dd3ff700 1 -- 172.21.15.98:0/3076908132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d806b4b0 msgr2=0x7f86d81f2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.011+0000 7f86dd3ff700 1 --2- 172.21.15.98:0/3076908132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d806b4b0 0x7f86d81f2900 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009fd0 tx=0x7f86c8004750 comp rx=0 tx=0).stop 2023-03-03T08:01:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.011+0000 7f86dd3ff700 1 -- 172.21.15.98:0/3076908132 shutdown_connections 2023-03-03T08:01:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.011+0000 7f86dd3ff700 1 --2- 172.21.15.98:0/3076908132 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f86c40684a0 0x7f86c406a950 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.011+0000 7f86dd3ff700 1 --2- 172.21.15.98:0/3076908132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d806b4b0 0x7f86d81f2900 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.011+0000 7f86dd3ff700 1 --2- 172.21.15.98:0/3076908132 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86d815ca70 0x7f86d815f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.011+0000 7f86dd3ff700 1 --2- 172.21.15.98:0/3076908132 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86d80d00a0 0x7f86d815b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.011+0000 7f86dd3ff700 1 -- 172.21.15.98:0/3076908132 >> 172.21.15.98:0/3076908132 conn(0x7f86d8073e70 msgr2=0x7f86d80d10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.011+0000 7f86dd3ff700 1 -- 172.21.15.98:0/3076908132 shutdown_connections 2023-03-03T08:01:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.011+0000 7f86dd3ff700 1 -- 172.21.15.98:0/3076908132 wait complete. 2023-03-03T08:01:03.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:01:03.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:01:03.091 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:01:03.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.290+0000 7f33bd7ff700 1 Processor -- start 2023-03-03T08:01:03.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.290+0000 7f33bd7ff700 1 -- start start 2023-03-03T08:01:03.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.291+0000 7f33bd7ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f33b80b87c0 0x7f33b80cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:03.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.291+0000 7f33bd7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33b80b5410 0x7f33b80d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:03.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.291+0000 7f33bd7ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b80d0ef0 con 0x7f33b80b8b90 2023-03-03T08:01:03.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.291+0000 7f33bd7ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b80d0600 con 0x7f33b80b87c0 2023-03-03T08:01:03.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.291+0000 7f33bd7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80cd970 con 0x7f33b80b5410 2023-03-03T08:01:03.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.292+0000 7f33b67fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f33b80b8b90 0x7f33b80b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34170/0 (socket says 172.21.15.98:34170) 2023-03-03T08:01:03.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.292+0000 7f33b67fc700 1 -- 172.21.15.98:0/2165720991 learned_addr learned my addr 172.21.15.98:0/2165720991 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:03.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.292+0000 7f33b6ffd700 1 --2- 172.21.15.98:0/2165720991 >> v2:172.21.15.98:3300/0 conn(0x7f33b80b87c0 0x7f33b80cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:03.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.292+0000 7f33b77fe700 1 --2- 172.21.15.98:0/2165720991 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33b80b5410 0x7f33b80d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:03.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.292+0000 7f33b5ffb700 1 -- 172.21.15.98:0/2165720991 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073366182 0 0) 0x7f33b80d0ef0 con 0x7f33b80b8b90 2023-03-03T08:01:03.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.292+0000 7f33b5ffb700 1 -- 172.21.15.98:0/2165720991 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a00035b0 con 0x7f33b80b8b90 2023-03-03T08:01:03.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.292+0000 7f33b6ffd700 1 -- 172.21.15.98:0/2165720991 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33b80b5410 msgr2=0x7f33b80d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:03.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.292+0000 7f33b6ffd700 1 --2- 172.21.15.98:0/2165720991 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33b80b5410 0x7f33b80d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.292+0000 7f33b6ffd700 1 -- 172.21.15.98:0/2165720991 >> v1:172.21.15.98:6789/0 conn(0x7f33b80b8b90 legacy=0x7f33b80b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:03.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.292+0000 7f33b6ffd700 1 -- 172.21.15.98:0/2165720991 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b80cdc50 con 0x7f33b80b87c0 2023-03-03T08:01:03.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.293+0000 7f33b6ffd700 1 --2- 172.21.15.98:0/2165720991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b80b87c0 0x7f33b80cffe0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f33a8009840 tx=0x7f33a8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c527ba16536eea server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:03.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.293+0000 7f33b5ffb700 1 -- 172.21.15.98:0/2165720991 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8011830 con 0x7f33b80b87c0 2023-03-03T08:01:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.293+0000 7f33b5ffb700 1 -- 172.21.15.98:0/2165720991 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a8011990 con 0x7f33b80b87c0 2023-03-03T08:01:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.293+0000 7f33b5ffb700 1 -- 172.21.15.98:0/2165720991 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8011d20 con 0x7f33b80b87c0 2023-03-03T08:01:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.293+0000 7f33bd7ff700 1 -- 172.21.15.98:0/2165720991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b80b87c0 msgr2=0x7f33b80cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.293+0000 7f33bd7ff700 1 --2- 172.21.15.98:0/2165720991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b80b87c0 0x7f33b80cffe0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f33a8009840 tx=0x7f33a8009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.294+0000 7f33bd7ff700 1 -- 172.21.15.98:0/2165720991 shutdown_connections 2023-03-03T08:01:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.294+0000 7f33bd7ff700 1 --2- 172.21.15.98:0/2165720991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b80b87c0 0x7f33b80cffe0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.294+0000 7f33bd7ff700 1 --2- 172.21.15.98:0/2165720991 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33b80b5410 0x7f33b80d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.294+0000 7f33bd7ff700 1 -- 172.21.15.98:0/2165720991 >> 172.21.15.98:0/2165720991 conn(0x7f33b806aee0 msgr2=0x7f33b806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:03.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.294+0000 7f33bd7ff700 1 -- 172.21.15.98:0/2165720991 shutdown_connections 2023-03-03T08:01:03.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.294+0000 7f33bd7ff700 1 -- 172.21.15.98:0/2165720991 wait complete. 2023-03-03T08:01:03.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.296+0000 7f33bd7ff700 1 Processor -- start 2023-03-03T08:01:03.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.296+0000 7f33bd7ff700 1 -- start start 2023-03-03T08:01:03.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.297+0000 7f33bd7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33b80b5410 0x7f33b815f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:03.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.297+0000 7f33bd7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33b80b87c0 0x7f33b81f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:03.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.297+0000 7f33bd7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b80b8b90 0x7f33b80aa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:03.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.297+0000 7f33bd7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b815d110 con 0x7f33b80b8b90 2023-03-03T08:01:03.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.297+0000 7f33bd7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b81f22a0 con 0x7f33b80b5410 2023-03-03T08:01:03.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.297+0000 7f33bd7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b81f2580 con 0x7f33b80b87c0 2023-03-03T08:01:03.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.297+0000 7f33b77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b80b8b90 0x7f33b80aa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:03.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.297+0000 7f33b77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b80b8b90 0x7f33b80aa9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48204/0 (socket says 172.21.15.98:48204) 2023-03-03T08:01:03.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.297+0000 7f33b77fe700 1 -- 172.21.15.98:0/676631148 learned_addr learned my addr 172.21.15.98:0/676631148 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:03.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.297+0000 7f33b6ffd700 1 --2- 172.21.15.98:0/676631148 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33b80b5410 0x7f33b815f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:03.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.298+0000 7f33b67fc700 1 --2- 172.21.15.98:0/676631148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33b80b87c0 0x7f33b81f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:03.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.298+0000 7f33b77fe700 1 -- 172.21.15.98:0/676631148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33b80b87c0 msgr2=0x7f33b81f1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:03.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.298+0000 7f33b77fe700 1 --2- 172.21.15.98:0/676631148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33b80b87c0 0x7f33b81f1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.298+0000 7f33b77fe700 1 -- 172.21.15.98:0/676631148 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33b80b5410 msgr2=0x7f33b815f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.298+0000 7f33b77fe700 1 --2- 172.21.15.98:0/676631148 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33b80b5410 0x7f33b815f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.298+0000 7f33b77fe700 1 -- 172.21.15.98:0/676631148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a80094a0 con 0x7f33b80b8b90 2023-03-03T08:01:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.298+0000 7f33b77fe700 1 --2- 172.21.15.98:0/676631148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b80b8b90 0x7f33b80aa9a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f33b000efc0 tx=0x7f33b000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.298+0000 7f339f7fe700 1 -- 172.21.15.98:0/676631148 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b0009ea0 con 0x7f33b80b8b90 2023-03-03T08:01:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.298+0000 7f339f7fe700 1 -- 172.21.15.98:0/676631148 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b000f040 con 0x7f33b80b8b90 2023-03-03T08:01:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.298+0000 7f339f7fe700 1 -- 172.21.15.98:0/676631148 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b000ced0 con 0x7f33b80b8b90 2023-03-03T08:01:03.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.298+0000 7f33bd7ff700 1 -- 172.21.15.98:0/676631148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b80cdf30 con 0x7f33b80b8b90 2023-03-03T08:01:03.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.299+0000 7f33bd7ff700 1 -- 172.21.15.98:0/676631148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b8073a80 con 0x7f33b80b8b90 2023-03-03T08:01:03.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.300+0000 7f33bd7ff700 1 -- 172.21.15.98:0/676631148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b80ce0c0 con 0x7f33b80b8b90 2023-03-03T08:01:03.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.301+0000 7f339f7fe700 1 -- 172.21.15.98:0/676631148 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f33b0014480 con 0x7f33b80b8b90 2023-03-03T08:01:03.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.302+0000 7f339f7fe700 1 --2- 172.21.15.98:0/676631148 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33a40684a0 0x7f33a406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:03.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.302+0000 7f339f7fe700 1 -- 172.21.15.98:0/676631148 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f33b00868a0 con 0x7f33b80b8b90 2023-03-03T08:01:03.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.303+0000 7f33b6ffd700 1 --2- 172.21.15.98:0/676631148 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33a40684a0 0x7f33a406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:03.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.304+0000 7f33b6ffd700 1 --2- 172.21.15.98:0/676631148 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33a40684a0 0x7f33a406a950 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f33a8000c00 tx=0x7f33a8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:03.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.306+0000 7f339f7fe700 1 -- 172.21.15.98:0/676631148 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f33b0052f00 con 0x7f33b80b8b90 2023-03-03T08:01:03.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.541+0000 7f33bd7ff700 1 -- 172.21.15.98:0/676631148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33b80729b0 con 0x7f33b80b8b90 2023-03-03T08:01:03.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.542+0000 7f339f7fe700 1 -- 172.21.15.98:0/676631148 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4649 (secure 0 0 0) 0x7f33b0056520 con 0x7f33b80b8b90 2023-03-03T08:01:03.543 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:03.544 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5702,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/976526047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":976526047},{"type":"v1","addr":"172.21.15.100:6834","nonce":976526047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3372584689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3372584689},{"type":"v1","addr":"172.21.15.100:6835","nonce":3372584689}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/505711827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":505711827},{"type":"v1","addr":"172.21.15.98:6837","nonce":505711827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:00:22.323396+0000","modified":"2023-03-03T08:00:25.787755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6387":{"gid":6387,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6836/3248147431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3248147431},{"type":"v1","addr":"172.21.15.98:6836","nonce":3248147431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-03-03T08:01:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.545+0000 7f33bd7ff700 1 -- 172.21.15.98:0/676631148 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33a40684a0 msgr2=0x7f33a406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.545+0000 7f33bd7ff700 1 --2- 172.21.15.98:0/676631148 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33a40684a0 0x7f33a406a950 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f33a8000c00 tx=0x7f33a8008030 comp rx=0 tx=0).stop 2023-03-03T08:01:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.545+0000 7f33bd7ff700 1 -- 172.21.15.98:0/676631148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b80b8b90 msgr2=0x7f33b80aa9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.545+0000 7f33bd7ff700 1 --2- 172.21.15.98:0/676631148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b80b8b90 0x7f33b80aa9a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f33b000efc0 tx=0x7f33b000c3a0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.545+0000 7f33bd7ff700 1 -- 172.21.15.98:0/676631148 shutdown_connections 2023-03-03T08:01:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.545+0000 7f33bd7ff700 1 --2- 172.21.15.98:0/676631148 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f33a40684a0 0x7f33a406a950 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.545+0000 7f33bd7ff700 1 --2- 172.21.15.98:0/676631148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33b80b8b90 0x7f33b80aa9a0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.545+0000 7f33bd7ff700 1 --2- 172.21.15.98:0/676631148 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33b80b5410 0x7f33b815f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.545+0000 7f33bd7ff700 1 --2- 172.21.15.98:0/676631148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33b80b87c0 0x7f33b81f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.545+0000 7f33bd7ff700 1 -- 172.21.15.98:0/676631148 >> 172.21.15.98:0/676631148 conn(0x7f33b806aee0 msgr2=0x7f33b815c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.545+0000 7f33bd7ff700 1 -- 172.21.15.98:0/676631148 shutdown_connections 2023-03-03T08:01:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.545+0000 7f33bd7ff700 1 -- 172.21.15.98:0/676631148 wait complete. 2023-03-03T08:01:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2023-03-03T08:01:03.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:01:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.818+0000 7f28c539d700 1 Processor -- start 2023-03-03T08:01:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.818+0000 7f28c539d700 1 -- start start 2023-03-03T08:01:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.819+0000 7f28c539d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28c0146360 0x7f28c0160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.819+0000 7f28c539d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c01614c0 con 0x7f28c00b4ab0 2023-03-03T08:01:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.819+0000 7f28c539d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c00d10f0 con 0x7f28c00d2590 2023-03-03T08:01:03.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.819+0000 7f28c539d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c00cdf20 con 0x7f28c0146360 2023-03-03T08:01:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.819+0000 7f28bf7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f28c00b4ab0 0x7f28c00731e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34180/0 (socket says 172.21.15.98:34180) 2023-03-03T08:01:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.820+0000 7f28bf7fe700 1 -- 172.21.15.98:0/3571029263 learned_addr learned my addr 172.21.15.98:0/3571029263 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.820+0000 7f28bffff700 1 --2- 172.21.15.98:0/3571029263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28c0146360 0x7f28c0160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.820+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950187220 0 0) 0x7f28b4001110 con 0x7f28c00b4ab0 2023-03-03T08:01:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.820+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28a80035b0 con 0x7f28c00b4ab0 2023-03-03T08:01:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.820+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783577404 0 0) 0x7f28c00d10f0 con 0x7f28c00d2590 2023-03-03T08:01:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.820+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28b4001110 con 0x7f28c00d2590 2023-03-03T08:01:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.820+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 861806540 0 0) 0x7f28a80035b0 con 0x7f28c00b4ab0 2023-03-03T08:01:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.820+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28c0146360 msgr2=0x7f28c0160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.820+0000 7f28beffd700 1 --2- 172.21.15.98:0/3571029263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28c0146360 0x7f28c0160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.820+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 >> v1:172.21.15.100:6789/0 conn(0x7f28c00d2590 legacy=0x7f28c00738f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.820+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c0074000 con 0x7f28c00b4ab0 2023-03-03T08:01:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f28b4001800 con 0x7f28c00b4ab0 2023-03-03T08:01:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28bffff700 1 --2- 172.21.15.98:0/3571029263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28c0146360 0x7f28c0160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 >> v1:172.21.15.98:6789/0 conn(0x7f28c00b4ab0 legacy=0x7f28c00731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28beffd700 1 --2- 172.21.15.98:0/3571029263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28c0146360 0x7f28a8003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28beffd700 1 --2- 172.21.15.98:0/3571029263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28a8004150 0x7f28a8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28beffd700 1 --2- 172.21.15.98:0/3571029263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a8004d90 0x7f28a8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c00cdf20 con 0x7f28a8004150 2023-03-03T08:01:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a800bba0 con 0x7f28a8004d90 2023-03-03T08:01:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a800bd60 con 0x7f28c0146360 2023-03-03T08:01:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28bf7fe700 1 --2- 172.21.15.98:0/3571029263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28a8004150 0x7f28a8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28bffff700 1 --2- 172.21.15.98:0/3571029263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28c0146360 0x7f28a8003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:03.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28c4b9c700 1 --2- 172.21.15.98:0/3571029263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a8004d90 0x7f28a8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28bf7fe700 1 -- 172.21.15.98:0/3571029263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28c0146360 msgr2=0x7f28a8003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28bf7fe700 1 --2- 172.21.15.98:0/3571029263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28c0146360 0x7f28a8003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28bf7fe700 1 -- 172.21.15.98:0/3571029263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a8004d90 msgr2=0x7f28a8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28bf7fe700 1 --2- 172.21.15.98:0/3571029263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a8004d90 0x7f28a8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.821+0000 7f28bf7fe700 1 -- 172.21.15.98:0/3571029263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b00094a0 con 0x7f28a8004150 2023-03-03T08:01:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.822+0000 7f28bf7fe700 1 --2- 172.21.15.98:0/3571029263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28a8004150 0x7f28a8004850 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f28b400c110 tx=0x7f28a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fbc71195a5a8924 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.822+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b400ccb0 con 0x7f28a8004150 2023-03-03T08:01:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.822+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b400ce10 con 0x7f28a8004150 2023-03-03T08:01:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.822+0000 7f28beffd700 1 -- 172.21.15.98:0/3571029263 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b4005ba0 con 0x7f28a8004150 2023-03-03T08:01:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.822+0000 7f28c539d700 1 -- 172.21.15.98:0/3571029263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28a8004150 msgr2=0x7f28a8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.822+0000 7f28c539d700 1 --2- 172.21.15.98:0/3571029263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28a8004150 0x7f28a8004850 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f28b400c110 tx=0x7f28a8005880 comp rx=0 tx=0).stop 2023-03-03T08:01:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.822+0000 7f28c4b9c700 1 -- 172.21.15.98:0/3571029263 reap_dead start 2023-03-03T08:01:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.823+0000 7f28c539d700 1 -- 172.21.15.98:0/3571029263 shutdown_connections 2023-03-03T08:01:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.823+0000 7f28c539d700 1 -- 172.21.15.98:0/3571029263 >> 172.21.15.98:0/3571029263 conn(0x7f28c006b8e0 msgr2=0x7f28c00cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.823+0000 7f28c539d700 1 -- 172.21.15.98:0/3571029263 shutdown_connections 2023-03-03T08:01:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.823+0000 7f28c539d700 1 -- 172.21.15.98:0/3571029263 wait complete. 2023-03-03T08:01:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.825+0000 7f28c539d700 1 Processor -- start 2023-03-03T08:01:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.825+0000 7f28c539d700 1 -- start start 2023-03-03T08:01:03.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.826+0000 7f28c539d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a8004150 0x7f28c00c7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:03.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.826+0000 7f28c539d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28c00c96f0 0x7f28c01f1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:03.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.826+0000 7f28c539d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28c01f21e0 0x7f28c01f2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:03.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.826+0000 7f28c539d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c00c8960 con 0x7f28c00c96f0 2023-03-03T08:01:03.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.826+0000 7f28c539d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c00c6380 con 0x7f28a8004150 2023-03-03T08:01:03.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.826+0000 7f28c539d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c00c7a80 con 0x7f28c01f21e0 2023-03-03T08:01:03.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.826+0000 7f28bf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28c00c96f0 0x7f28c01f1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:03.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.826+0000 7f28bf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28c00c96f0 0x7f28c01f1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48220/0 (socket says 172.21.15.98:48220) 2023-03-03T08:01:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.826+0000 7f28bf7fe700 1 -- 172.21.15.98:0/1224476037 learned_addr learned my addr 172.21.15.98:0/1224476037 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.827+0000 7f28bffff700 1 --2- 172.21.15.98:0/1224476037 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a8004150 0x7f28c00c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.827+0000 7f28c4b9c700 1 --2- 172.21.15.98:0/1224476037 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28c01f21e0 0x7f28c01f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.827+0000 7f28bf7fe700 1 -- 172.21.15.98:0/1224476037 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28c01f21e0 msgr2=0x7f28c01f2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.827+0000 7f28bf7fe700 1 --2- 172.21.15.98:0/1224476037 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28c01f21e0 0x7f28c01f2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.827+0000 7f28bf7fe700 1 -- 172.21.15.98:0/1224476037 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a8004150 msgr2=0x7f28c00c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.827+0000 7f28bf7fe700 1 --2- 172.21.15.98:0/1224476037 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a8004150 0x7f28c00c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.827+0000 7f28bf7fe700 1 -- 172.21.15.98:0/1224476037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c00cd960 con 0x7f28c00c96f0 2023-03-03T08:01:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.827+0000 7f28bf7fe700 1 --2- 172.21.15.98:0/1224476037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28c00c96f0 0x7f28c01f1ca0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f28b400c700 tx=0x7f28b4001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.828+0000 7f28bcff9700 1 -- 172.21.15.98:0/1224476037 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b4004c80 con 0x7f28c00c96f0 2023-03-03T08:01:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.828+0000 7f28bcff9700 1 -- 172.21.15.98:0/1224476037 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b4004de0 con 0x7f28c00c96f0 2023-03-03T08:01:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.828+0000 7f28bcff9700 1 -- 172.21.15.98:0/1224476037 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b40072d0 con 0x7f28c00c96f0 2023-03-03T08:01:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.828+0000 7f28c539d700 1 -- 172.21.15.98:0/1224476037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c00cd680 con 0x7f28c00c96f0 2023-03-03T08:01:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.828+0000 7f28c539d700 1 -- 172.21.15.98:0/1224476037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c00cdc40 con 0x7f28c00c96f0 2023-03-03T08:01:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.829+0000 7f28c539d700 1 -- 172.21.15.98:0/1224476037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c00c8aa0 con 0x7f28c00c96f0 2023-03-03T08:01:03.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.830+0000 7f28bcff9700 1 -- 172.21.15.98:0/1224476037 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f28b4010070 con 0x7f28c00c96f0 2023-03-03T08:01:03.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.831+0000 7f28bcff9700 1 --2- 172.21.15.98:0/1224476037 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f28ac0684a0 0x7f28ac06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:03.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.832+0000 7f28bcff9700 1 -- 172.21.15.98:0/1224476037 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f28b4086cb0 con 0x7f28c00c96f0 2023-03-03T08:01:03.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.832+0000 7f28bffff700 1 --2- 172.21.15.98:0/1224476037 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f28ac0684a0 0x7f28ac06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:03.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.832+0000 7f28bffff700 1 --2- 172.21.15.98:0/1224476037 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f28ac0684a0 0x7f28ac06a950 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f28b0009d00 tx=0x7f28b0009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:03.836+0000 7f28bcff9700 1 -- 172.21.15.98:0/1224476037 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f28c00c8aa0 con 0x7f28c00c96f0 2023-03-03T08:01:04.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.027+0000 7f28c539d700 1 -- 172.21.15.98:0/1224476037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28c01614c0 con 0x7f28c00c96f0 2023-03-03T08:01:04.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.028+0000 7f28bcff9700 1 -- 172.21.15.98:0/1224476037 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+17921 (secure 0 0 0) 0x7f28c01614c0 con 0x7f28c00c96f0 2023-03-03T08:01:04.030 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:04.030 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":66,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:00:23.326015+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:00:19.823120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-03-03T08:00:21.324346+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:01:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.031+0000 7f28c539d700 1 -- 172.21.15.98:0/1224476037 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f28ac0684a0 msgr2=0x7f28ac06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.031+0000 7f28c539d700 1 --2- 172.21.15.98:0/1224476037 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f28ac0684a0 0x7f28ac06a950 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f28b0009d00 tx=0x7f28b0009210 comp rx=0 tx=0).stop 2023-03-03T08:01:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.031+0000 7f28c539d700 1 -- 172.21.15.98:0/1224476037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28c00c96f0 msgr2=0x7f28c01f1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.031+0000 7f28c539d700 1 --2- 172.21.15.98:0/1224476037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28c00c96f0 0x7f28c01f1ca0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f28b400c700 tx=0x7f28b4001ac0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.031+0000 7f28c539d700 1 -- 172.21.15.98:0/1224476037 shutdown_connections 2023-03-03T08:01:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.031+0000 7f28c539d700 1 --2- 172.21.15.98:0/1224476037 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f28ac0684a0 0x7f28ac06a950 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.031+0000 7f28c539d700 1 --2- 172.21.15.98:0/1224476037 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28c01f21e0 0x7f28c01f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.031+0000 7f28c539d700 1 --2- 172.21.15.98:0/1224476037 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a8004150 0x7f28c00c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.031+0000 7f28c539d700 1 --2- 172.21.15.98:0/1224476037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28c00c96f0 0x7f28c01f1ca0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.031+0000 7f28c539d700 1 -- 172.21.15.98:0/1224476037 >> 172.21.15.98:0/1224476037 conn(0x7f28c006b8e0 msgr2=0x7f28c00b7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.031+0000 7f28c539d700 1 -- 172.21.15.98:0/1224476037 shutdown_connections 2023-03-03T08:01:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.031+0000 7f28c539d700 1 -- 172.21.15.98:0/1224476037 wait complete. 2023-03-03T08:01:04.060 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-03-03T08:01:04.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:01:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.304+0000 7f9b9abff700 1 Processor -- start 2023-03-03T08:01:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.304+0000 7f9b9abff700 1 -- start start 2023-03-03T08:01:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.305+0000 7f9b9abff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9b940b87c0 0x7f9b940cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.305+0000 7f9b9abff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b940b8b90 0x7f9b940b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.305+0000 7f9b9abff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b940d0ef0 con 0x7f9b940b5240 2023-03-03T08:01:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.305+0000 7f9b9abff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b940d0600 con 0x7f9b940b87c0 2023-03-03T08:01:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.305+0000 7f9b9abff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b940cd970 con 0x7f9b940b8b90 2023-03-03T08:01:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.306+0000 7f9b9919c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9b940b5240 0x7f9b941605b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34188/0 (socket says 172.21.15.98:34188) 2023-03-03T08:01:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.306+0000 7f9b9919c700 1 -- 172.21.15.98:0/2861450099 learned_addr learned my addr 172.21.15.98:0/2861450099 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:04.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.306+0000 7f9b93fff700 1 --2- 172.21.15.98:0/2861450099 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b940b8b90 0x7f9b940b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.306+0000 7f9b9899b700 1 --2- 172.21.15.98:0/2861450099 >> v2:172.21.15.100:3300/0 conn(0x7f9b940b87c0 0x7f9b940cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.306+0000 7f9b937fe700 1 -- 172.21.15.98:0/2861450099 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425005948 0 0) 0x7f9b940d0ef0 con 0x7f9b940b5240 2023-03-03T08:01:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.306+0000 7f9b937fe700 1 -- 172.21.15.98:0/2861450099 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b840035b0 con 0x7f9b940b5240 2023-03-03T08:01:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.306+0000 7f9b93fff700 1 -- 172.21.15.98:0/2861450099 >> v2:172.21.15.100:3300/0 conn(0x7f9b940b87c0 msgr2=0x7f9b940cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.306+0000 7f9b93fff700 1 --2- 172.21.15.98:0/2861450099 >> v2:172.21.15.100:3300/0 conn(0x7f9b940b87c0 0x7f9b940cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.306+0000 7f9b93fff700 1 -- 172.21.15.98:0/2861450099 >> v1:172.21.15.98:6789/0 conn(0x7f9b940b5240 legacy=0x7f9b941605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.306+0000 7f9b93fff700 1 -- 172.21.15.98:0/2861450099 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b940cdc50 con 0x7f9b940b8b90 2023-03-03T08:01:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.306+0000 7f9b9899b700 1 --2- 172.21.15.98:0/2861450099 >> v2:172.21.15.100:3300/0 conn(0x7f9b940b87c0 0x7f9b940cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.307+0000 7f9b93fff700 1 --2- 172.21.15.98:0/2861450099 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b940b8b90 0x7f9b940b4d00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009840 tx=0x7f9b80009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f64b76c5cf9b7602 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.307+0000 7f9b937fe700 1 -- 172.21.15.98:0/2861450099 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b80011830 con 0x7f9b940b8b90 2023-03-03T08:01:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.307+0000 7f9b937fe700 1 -- 172.21.15.98:0/2861450099 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b80011990 con 0x7f9b940b8b90 2023-03-03T08:01:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.307+0000 7f9b937fe700 1 -- 172.21.15.98:0/2861450099 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b80011d00 con 0x7f9b940b8b90 2023-03-03T08:01:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.308+0000 7f9b9abff700 1 -- 172.21.15.98:0/2861450099 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b940b8b90 msgr2=0x7f9b940b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.308+0000 7f9b9abff700 1 --2- 172.21.15.98:0/2861450099 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b940b8b90 0x7f9b940b4d00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009840 tx=0x7f9b80009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.308+0000 7f9b9abff700 1 -- 172.21.15.98:0/2861450099 shutdown_connections 2023-03-03T08:01:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.308+0000 7f9b9abff700 1 --2- 172.21.15.98:0/2861450099 >> v2:172.21.15.100:3300/0 conn(0x7f9b940b87c0 0x7f9b940cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.308+0000 7f9b9abff700 1 --2- 172.21.15.98:0/2861450099 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b940b8b90 0x7f9b940b4d00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.308+0000 7f9b9abff700 1 -- 172.21.15.98:0/2861450099 >> 172.21.15.98:0/2861450099 conn(0x7f9b9406aee0 msgr2=0x7f9b9406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.308+0000 7f9b9abff700 1 -- 172.21.15.98:0/2861450099 shutdown_connections 2023-03-03T08:01:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.308+0000 7f9b9abff700 1 -- 172.21.15.98:0/2861450099 wait complete. 2023-03-03T08:01:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.311+0000 7f9b9abff700 1 Processor -- start 2023-03-03T08:01:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.311+0000 7f9b9abff700 1 -- start start 2023-03-03T08:01:04.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.312+0000 7f9b9abff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b940b5240 0x7f9b941f1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:04.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.313+0000 7f9b9abff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b940b87c0 0x7f9b940c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:04.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.313+0000 7f9b9abff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9b940b8b90 0x7f9b940b7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:04.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.313+0000 7f9b9abff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b940c9890 con 0x7f9b940b87c0 2023-03-03T08:01:04.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.313+0000 7f9b9abff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b940c9aa0 con 0x7f9b940b8b90 2023-03-03T08:01:04.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.313+0000 7f9b9abff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b940c9d80 con 0x7f9b940b5240 2023-03-03T08:01:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.313+0000 7f9b93fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b940b87c0 0x7f9b940c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.313+0000 7f9b93fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b940b87c0 0x7f9b940c6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48224/0 (socket says 172.21.15.98:48224) 2023-03-03T08:01:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.313+0000 7f9b93fff700 1 -- 172.21.15.98:0/1429848079 learned_addr learned my addr 172.21.15.98:0/1429848079 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.313+0000 7f9b9899b700 1 --2- 172.21.15.98:0/1429848079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b940b5240 0x7f9b941f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.313+0000 7f9b9919c700 1 --2- 172.21.15.98:0/1429848079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9b940b8b90 0x7f9b940b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.313+0000 7f9b93fff700 1 -- 172.21.15.98:0/1429848079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b940b5240 msgr2=0x7f9b941f1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.313+0000 7f9b93fff700 1 --2- 172.21.15.98:0/1429848079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b940b5240 0x7f9b941f1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.313+0000 7f9b93fff700 1 -- 172.21.15.98:0/1429848079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9b940b8b90 msgr2=0x7f9b940b7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.313+0000 7f9b93fff700 1 --2- 172.21.15.98:0/1429848079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9b940b8b90 0x7f9b940b7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.313+0000 7f9b93fff700 1 -- 172.21.15.98:0/1429848079 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b880094a0 con 0x7f9b940b87c0 2023-03-03T08:01:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.314+0000 7f9b93fff700 1 --2- 172.21.15.98:0/1429848079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b940b87c0 0x7f9b940c6db0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009fd0 tx=0x7f9b80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.314+0000 7f9b917fa700 1 -- 172.21.15.98:0/1429848079 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b80004550 con 0x7f9b940b87c0 2023-03-03T08:01:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.314+0000 7f9b917fa700 1 -- 172.21.15.98:0/1429848079 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8001e070 con 0x7f9b940b87c0 2023-03-03T08:01:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.314+0000 7f9b917fa700 1 -- 172.21.15.98:0/1429848079 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b800038e0 con 0x7f9b940b87c0 2023-03-03T08:01:04.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.314+0000 7f9b9abff700 1 -- 172.21.15.98:0/1429848079 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b800094a0 con 0x7f9b940b87c0 2023-03-03T08:01:04.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.314+0000 7f9b9abff700 1 -- 172.21.15.98:0/1429848079 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b940cdf30 con 0x7f9b940b87c0 2023-03-03T08:01:04.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.316+0000 7f9b9abff700 1 -- 172.21.15.98:0/1429848079 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b940ce0c0 con 0x7f9b940b87c0 2023-03-03T08:01:04.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.317+0000 7f9b917fa700 1 -- 172.21.15.98:0/1429848079 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f9b8000f050 con 0x7f9b940b87c0 2023-03-03T08:01:04.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.318+0000 7f9b917fa700 1 --2- 172.21.15.98:0/1429848079 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9b780684a0 0x7f9b7806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:04.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.318+0000 7f9b917fa700 1 -- 172.21.15.98:0/1429848079 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9b80086d90 con 0x7f9b940b87c0 2023-03-03T08:01:04.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.318+0000 7f9b9899b700 1 --2- 172.21.15.98:0/1429848079 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9b780684a0 0x7f9b7806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:04.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.319+0000 7f9b9899b700 1 --2- 172.21.15.98:0/1429848079 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9b780684a0 0x7f9b7806a950 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009d70 tx=0x7f9b88009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:04.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.322+0000 7f9b917fa700 1 -- 172.21.15.98:0/1429848079 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9b800533d0 con 0x7f9b940b87c0 2023-03-03T08:01:04.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.555+0000 7f9b9abff700 1 -- 172.21.15.98:0/1429848079 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b940729b0 con 0x7f9b940b87c0 2023-03-03T08:01:04.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.555+0000 7f9b917fa700 1 -- 172.21.15.98:0/1429848079 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4649 (secure 0 0 0) 0x7f9b80015090 con 0x7f9b940b87c0 2023-03-03T08:01:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:04.557 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5702,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/976526047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":976526047},{"type":"v1","addr":"172.21.15.100:6834","nonce":976526047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3372584689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3372584689},{"type":"v1","addr":"172.21.15.100:6835","nonce":3372584689}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/505711827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":505711827},{"type":"v1","addr":"172.21.15.98:6837","nonce":505711827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:00:22.323396+0000","modified":"2023-03-03T08:00:25.787755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6387":{"gid":6387,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6836/3248147431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3248147431},{"type":"v1","addr":"172.21.15.98:6836","nonce":3248147431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-03-03T08:01:04.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.558+0000 7f9b9abff700 1 -- 172.21.15.98:0/1429848079 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9b780684a0 msgr2=0x7f9b7806a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:04.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.558+0000 7f9b9abff700 1 --2- 172.21.15.98:0/1429848079 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9b780684a0 0x7f9b7806a950 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009d70 tx=0x7f9b88009210 comp rx=0 tx=0).stop 2023-03-03T08:01:04.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.558+0000 7f9b9abff700 1 -- 172.21.15.98:0/1429848079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b940b87c0 msgr2=0x7f9b940c6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:04.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.558+0000 7f9b9abff700 1 --2- 172.21.15.98:0/1429848079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b940b87c0 0x7f9b940c6db0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009fd0 tx=0x7f9b80004750 comp rx=0 tx=0).stop 2023-03-03T08:01:04.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.558+0000 7f9b9abff700 1 -- 172.21.15.98:0/1429848079 shutdown_connections 2023-03-03T08:01:04.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.558+0000 7f9b9abff700 1 --2- 172.21.15.98:0/1429848079 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9b780684a0 0x7f9b7806a950 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.558+0000 7f9b9abff700 1 --2- 172.21.15.98:0/1429848079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9b940b8b90 0x7f9b940b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.558+0000 7f9b9abff700 1 --2- 172.21.15.98:0/1429848079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b940b5240 0x7f9b941f1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.558+0000 7f9b9abff700 1 --2- 172.21.15.98:0/1429848079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b940b87c0 0x7f9b940c6db0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.558+0000 7f9b9abff700 1 -- 172.21.15.98:0/1429848079 >> 172.21.15.98:0/1429848079 conn(0x7f9b9406aee0 msgr2=0x7f9b940b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:04.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.558+0000 7f9b9abff700 1 -- 172.21.15.98:0/1429848079 shutdown_connections 2023-03-03T08:01:04.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.558+0000 7f9b9abff700 1 -- 172.21.15.98:0/1429848079 wait complete. 2023-03-03T08:01:04.561 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2023-03-03T08:01:04.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:01:04.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.831+0000 7fa0aeb9d700 1 Processor -- start 2023-03-03T08:01:04.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.832+0000 7fa0aeb9d700 1 -- start start 2023-03-03T08:01:04.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.833+0000 7fa0aeb9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa0a80d1650 0x7fa0a80d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:04.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.833+0000 7fa0aeb9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa0a80b4a00 0x7fa0a80b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:04.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.833+0000 7fa0aeb9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a80b0c50 con 0x7fa0a80d0010 2023-03-03T08:01:04.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.833+0000 7fa0aeb9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a80b0040 con 0x7fa0a80d1650 2023-03-03T08:01:04.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.833+0000 7fa0aeb9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80b0320 con 0x7fa0a80b4a00 2023-03-03T08:01:04.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.833+0000 7fa0adb9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa0a80d0010 0x7fa0a80d0f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34202/0 (socket says 172.21.15.98:34202) 2023-03-03T08:01:04.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.833+0000 7fa0adb9b700 1 -- 172.21.15.98:0/953781564 learned_addr learned my addr 172.21.15.98:0/953781564 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:04.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.834+0000 7fa0ad39a700 1 --2- 172.21.15.98:0/953781564 >> v2:172.21.15.100:3300/0 conn(0x7fa0a80d1650 0x7fa0a80d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:04.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.834+0000 7fa0ae39c700 1 --2- 172.21.15.98:0/953781564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa0a80b4a00 0x7fa0a80b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:04.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.834+0000 7fa0acb99700 1 -- 172.21.15.98:0/953781564 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091631573 0 0) 0x7fa0a80b0c50 con 0x7fa0a80d0010 2023-03-03T08:01:04.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.834+0000 7fa0acb99700 1 -- 172.21.15.98:0/953781564 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0940035b0 con 0x7fa0a80d0010 2023-03-03T08:01:04.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.834+0000 7fa0acb99700 1 -- 172.21.15.98:0/953781564 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3368375117 0 0) 0x7fa0940035b0 con 0x7fa0a80d0010 2023-03-03T08:01:04.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.834+0000 7fa0acb99700 1 -- 172.21.15.98:0/953781564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa0a80b4a00 msgr2=0x7fa0a80b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:04.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.834+0000 7fa0acb99700 1 --2- 172.21.15.98:0/953781564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa0a80b4a00 0x7fa0a80b2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.834+0000 7fa0acb99700 1 -- 172.21.15.98:0/953781564 >> v2:172.21.15.100:3300/0 conn(0x7fa0a80d1650 msgr2=0x7fa0a80d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:04.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.834+0000 7fa0acb99700 1 --2- 172.21.15.98:0/953781564 >> v2:172.21.15.100:3300/0 conn(0x7fa0a80d1650 0x7fa0a80d03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.834+0000 7fa0acb99700 1 -- 172.21.15.98:0/953781564 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a80b0600 con 0x7fa0a80d0010 2023-03-03T08:01:04.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.834+0000 7fa0acb99700 1 -- 172.21.15.98:0/953781564 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7fa0a4005120 con 0x7fa0a80d0010 2023-03-03T08:01:04.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.834+0000 7fa0acb99700 1 -- 172.21.15.98:0/953781564 >> v1:172.21.15.98:6789/0 conn(0x7fa0a80d0010 legacy=0x7fa0a80d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:04.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.835+0000 7fa0acb99700 1 --2- 172.21.15.98:0/953781564 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa094003a00 0x7fa094003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:04.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.835+0000 7fa0acb99700 1 --2- 172.21.15.98:0/953781564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa094004430 0x7fa0940048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:04.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.835+0000 7fa0acb99700 1 --2- 172.21.15.98:0/953781564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa0a80b4a00 0x7fa094004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:04.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.835+0000 7fa0acb99700 1 -- 172.21.15.98:0/953781564 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80b0040 con 0x7fa094004430 2023-03-03T08:01:04.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.835+0000 7fa0acb99700 1 -- 172.21.15.98:0/953781564 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80b0320 con 0x7fa094003a00 2023-03-03T08:01:04.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.835+0000 7fa0acb99700 1 -- 172.21.15.98:0/953781564 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09400ba10 con 0x7fa0a80b4a00 2023-03-03T08:01:04.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.835+0000 7fa0ad39a700 1 --2- 172.21.15.98:0/953781564 >> v2:172.21.15.100:3300/0 conn(0x7fa0a80d1650 0x7fa0a80d03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:04.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.835+0000 7fa0ad39a700 1 --2- 172.21.15.98:0/953781564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa094004430 0x7fa0940048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:04.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.835+0000 7fa0adb9b700 1 --2- 172.21.15.98:0/953781564 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa094003a00 0x7fa094003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:04.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.835+0000 7fa0ae39c700 1 --2- 172.21.15.98:0/953781564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa0a80b4a00 0x7fa094004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:04.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.835+0000 7fa0ad39a700 1 -- 172.21.15.98:0/953781564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa0a80b4a00 msgr2=0x7fa094004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:04.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.836+0000 7fa0ad39a700 1 --2- 172.21.15.98:0/953781564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa0a80b4a00 0x7fa094004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.836+0000 7fa0ad39a700 1 -- 172.21.15.98:0/953781564 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa094003a00 msgr2=0x7fa094003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:04.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.836+0000 7fa0ad39a700 1 --2- 172.21.15.98:0/953781564 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa094003a00 0x7fa094003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.836+0000 7fa0ad39a700 1 -- 172.21.15.98:0/953781564 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0940035b0 con 0x7fa094004430 2023-03-03T08:01:04.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.836+0000 7fa0ad39a700 1 --2- 172.21.15.98:0/953781564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa094004430 0x7fa0940048f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fa0a808b880 tx=0x7fa094005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62563fa68f8246be server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:04.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.836+0000 7fa0acb99700 1 -- 172.21.15.98:0/953781564 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098004030 con 0x7fa094004430 2023-03-03T08:01:04.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.836+0000 7fa0acb99700 1 -- 172.21.15.98:0/953781564 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa098004190 con 0x7fa094004430 2023-03-03T08:01:04.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.836+0000 7fa0acb99700 1 -- 172.21.15.98:0/953781564 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098004540 con 0x7fa094004430 2023-03-03T08:01:04.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.837+0000 7fa0aeb9d700 1 -- 172.21.15.98:0/953781564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa094004430 msgr2=0x7fa0940048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:04.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.837+0000 7fa0aeb9d700 1 --2- 172.21.15.98:0/953781564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa094004430 0x7fa0940048f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fa0a808b880 tx=0x7fa094005880 comp rx=0 tx=0).stop 2023-03-03T08:01:04.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.837+0000 7fa0ae39c700 1 -- 172.21.15.98:0/953781564 reap_dead start 2023-03-03T08:01:04.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.837+0000 7fa0aeb9d700 1 -- 172.21.15.98:0/953781564 shutdown_connections 2023-03-03T08:01:04.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.837+0000 7fa0aeb9d700 1 -- 172.21.15.98:0/953781564 >> 172.21.15.98:0/953781564 conn(0x7fa0a8073b00 msgr2=0x7fa0a80d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:04.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.837+0000 7fa0aeb9d700 1 -- 172.21.15.98:0/953781564 shutdown_connections 2023-03-03T08:01:04.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.837+0000 7fa0aeb9d700 1 -- 172.21.15.98:0/953781564 wait complete. 2023-03-03T08:01:04.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.839+0000 7fa0aeb9d700 1 Processor -- start 2023-03-03T08:01:04.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.839+0000 7fa0aeb9d700 1 -- start start 2023-03-03T08:01:04.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.840+0000 7fa0aeb9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa094004430 0x7fa0a809dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:04.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.840+0000 7fa0aeb9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0a809fa40 0x7fa0a815a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:04.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.840+0000 7fa0aeb9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa0a815ae90 0x7fa0a81f1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:04.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.840+0000 7fa0aeb9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a809feb0 con 0x7fa0a809fa40 2023-03-03T08:01:04.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.840+0000 7fa0aeb9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80a0190 con 0x7fa0a815ae90 2023-03-03T08:01:04.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.840+0000 7fa0aeb9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80a0470 con 0x7fa094004430 2023-03-03T08:01:04.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.841+0000 7fa0ad39a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0a809fa40 0x7fa0a815a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:04.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.841+0000 7fa0ad39a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0a809fa40 0x7fa0a815a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48236/0 (socket says 172.21.15.98:48236) 2023-03-03T08:01:04.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.841+0000 7fa0ad39a700 1 -- 172.21.15.98:0/445454245 learned_addr learned my addr 172.21.15.98:0/445454245 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:04.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.841+0000 7fa0ae39c700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa0a815ae90 0x7fa0a81f1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:04.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.841+0000 7fa0adb9b700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa094004430 0x7fa0a809dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:04.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.841+0000 7fa0ad39a700 1 -- 172.21.15.98:0/445454245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa094004430 msgr2=0x7fa0a809dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:04.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.841+0000 7fa0ad39a700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa094004430 0x7fa0a809dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.841+0000 7fa0ad39a700 1 -- 172.21.15.98:0/445454245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa0a815ae90 msgr2=0x7fa0a81f1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:04.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.841+0000 7fa0ad39a700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa0a815ae90 0x7fa0a81f1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:04.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.841+0000 7fa0ad39a700 1 -- 172.21.15.98:0/445454245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a80d0b30 con 0x7fa0a809fa40 2023-03-03T08:01:04.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.841+0000 7fa0ae39c700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa0a815ae90 0x7fa0a81f1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:04.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.841+0000 7fa0adb9b700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa094004430 0x7fa0a809dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:04.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.841+0000 7fa0ad39a700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0a809fa40 0x7fa0a815a950 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fa0940056d0 tx=0x7fa0980047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:04.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.842+0000 7fa09e7fc700 1 -- 172.21.15.98:0/445454245 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098020070 con 0x7fa0a809fa40 2023-03-03T08:01:04.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.842+0000 7fa09e7fc700 1 -- 172.21.15.98:0/445454245 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0980038b0 con 0x7fa0a809fa40 2023-03-03T08:01:04.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.842+0000 7fa0aeb9d700 1 -- 172.21.15.98:0/445454245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a80b08e0 con 0x7fa0a809fa40 2023-03-03T08:01:04.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.842+0000 7fa09e7fc700 1 -- 172.21.15.98:0/445454245 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098004030 con 0x7fa0a809fa40 2023-03-03T08:01:04.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.842+0000 7fa0aeb9d700 1 -- 172.21.15.98:0/445454245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a80d0cc0 con 0x7fa0a809fa40 2023-03-03T08:01:04.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.843+0000 7fa0aeb9d700 1 -- 172.21.15.98:0/445454245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a80729b0 con 0x7fa0a809fa40 2023-03-03T08:01:04.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.844+0000 7fa09e7fc700 1 -- 172.21.15.98:0/445454245 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fa09802c030 con 0x7fa0a809fa40 2023-03-03T08:01:04.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.845+0000 7fa09e7fc700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa08c0684a0 0x7fa08c06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:04.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.845+0000 7fa09e7fc700 1 -- 172.21.15.98:0/445454245 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa098087360 con 0x7fa0a809fa40 2023-03-03T08:01:04.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.846+0000 7fa0adb9b700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa08c0684a0 0x7fa08c06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:04.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.846+0000 7fa0adb9b700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa08c0684a0 0x7fa08c06a950 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4005ae0 tx=0x7fa0a4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:04.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:04.849+0000 7fa09e7fc700 1 -- 172.21.15.98:0/445454245 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa098053a20 con 0x7fa0a809fa40 2023-03-03T08:01:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.085+0000 7fa0aeb9d700 1 -- 172.21.15.98:0/445454245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0a8074230 con 0x7fa0a809fa40 2023-03-03T08:01:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.085+0000 7fa09e7fc700 1 -- 172.21.15.98:0/445454245 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4649 (secure 0 0 0) 0x7fa098057040 con 0x7fa0a809fa40 2023-03-03T08:01:05.087 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:05.087 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5702,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/976526047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":976526047},{"type":"v1","addr":"172.21.15.100:6834","nonce":976526047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3372584689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3372584689},{"type":"v1","addr":"172.21.15.100:6835","nonce":3372584689}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/505711827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":505711827},{"type":"v1","addr":"172.21.15.98:6837","nonce":505711827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:00:22.323396+0000","modified":"2023-03-03T08:00:25.787755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6387":{"gid":6387,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6836/3248147431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3248147431},{"type":"v1","addr":"172.21.15.98:6836","nonce":3248147431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-03-03T08:01:05.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.088+0000 7fa0aeb9d700 1 -- 172.21.15.98:0/445454245 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa08c0684a0 msgr2=0x7fa08c06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:05.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.088+0000 7fa0aeb9d700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa08c0684a0 0x7fa08c06a950 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4005ae0 tx=0x7fa0a4009040 comp rx=0 tx=0).stop 2023-03-03T08:01:05.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.088+0000 7fa0aeb9d700 1 -- 172.21.15.98:0/445454245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0a809fa40 msgr2=0x7fa0a815a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:05.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.088+0000 7fa0aeb9d700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0a809fa40 0x7fa0a815a950 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fa0940056d0 tx=0x7fa0980047c0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.088+0000 7fa0aeb9d700 1 -- 172.21.15.98:0/445454245 shutdown_connections 2023-03-03T08:01:05.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.088+0000 7fa0aeb9d700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa08c0684a0 0x7fa08c06a950 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.088+0000 7fa0aeb9d700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa0a815ae90 0x7fa0a81f1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.088+0000 7fa0aeb9d700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa094004430 0x7fa0a809dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.088+0000 7fa0aeb9d700 1 --2- 172.21.15.98:0/445454245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0a809fa40 0x7fa0a815a950 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.088+0000 7fa0aeb9d700 1 -- 172.21.15.98:0/445454245 >> 172.21.15.98:0/445454245 conn(0x7fa0a8073b00 msgr2=0x7fa0a80d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:05.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.088+0000 7fa0aeb9d700 1 -- 172.21.15.98:0/445454245 shutdown_connections 2023-03-03T08:01:05.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.088+0000 7fa0aeb9d700 1 -- 172.21.15.98:0/445454245 wait complete. 2023-03-03T08:01:05.091 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2023-03-03T08:01:05.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:01:05.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.362+0000 7ff2aaf9d700 1 Processor -- start 2023-03-03T08:01:05.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.362+0000 7ff2aaf9d700 1 -- start start 2023-03-03T08:01:05.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.364+0000 7ff2aaf9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2a40b51c0 0x7ff2a4160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:05.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.364+0000 7ff2aaf9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2a40d1580 0x7ff2a40d19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:05.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.364+0000 7ff2aaf9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a41614c0 con 0x7ff2a4146360 2023-03-03T08:01:05.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.364+0000 7ff2aaf9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a40d10f0 con 0x7ff2a40d1580 2023-03-03T08:01:05.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.364+0000 7ff2aaf9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40cdf20 con 0x7ff2a40b51c0 2023-03-03T08:01:05.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.365+0000 7ff2aa79c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2a40d1580 0x7ff2a40d19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:05.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.365+0000 7ff2aa79c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2a40d1580 0x7ff2a40d19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32880/0 (socket says 172.21.15.98:32880) 2023-03-03T08:01:05.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.365+0000 7ff2aa79c700 1 -- 172.21.15.98:0/482008604 learned_addr learned my addr 172.21.15.98:0/482008604 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:05.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.365+0000 7ff2a979a700 1 --2- 172.21.15.98:0/482008604 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2a40b51c0 0x7ff2a4160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:05.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.365+0000 7ff2aa79c700 1 -- 172.21.15.98:0/482008604 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2a40b51c0 msgr2=0x7ff2a4160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:05.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.365+0000 7ff2aa79c700 1 --2- 172.21.15.98:0/482008604 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2a40b51c0 0x7ff2a4160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.365+0000 7ff2aa79c700 1 -- 172.21.15.98:0/482008604 >> v1:172.21.15.100:6789/0 conn(0x7ff2a4146360 legacy=0x7ff2a40b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:05.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.365+0000 7ff2aa79c700 1 -- 172.21.15.98:0/482008604 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a40cd680 con 0x7ff2a40d1580 2023-03-03T08:01:05.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.365+0000 7ff2aa79c700 1 --2- 172.21.15.98:0/482008604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a40d1580 0x7ff2a40d19e0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ff29c009e50 tx=0x7ff29c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=243efbf7b5252485 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:05.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.366+0000 7ff2a8f99700 1 -- 172.21.15.98:0/482008604 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29c00e070 con 0x7ff2a40d1580 2023-03-03T08:01:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.366+0000 7ff2a8f99700 1 -- 172.21.15.98:0/482008604 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29c00b910 con 0x7ff2a40d1580 2023-03-03T08:01:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.366+0000 7ff2a8f99700 1 -- 172.21.15.98:0/482008604 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29c00bcc0 con 0x7ff2a40d1580 2023-03-03T08:01:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.366+0000 7ff2aaf9d700 1 -- 172.21.15.98:0/482008604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a40d1580 msgr2=0x7ff2a40d19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.366+0000 7ff2aaf9d700 1 --2- 172.21.15.98:0/482008604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a40d1580 0x7ff2a40d19e0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ff29c009e50 tx=0x7ff29c00b330 comp rx=0 tx=0).stop 2023-03-03T08:01:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.366+0000 7ff2aaf9d700 1 -- 172.21.15.98:0/482008604 shutdown_connections 2023-03-03T08:01:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.366+0000 7ff2aaf9d700 1 --2- 172.21.15.98:0/482008604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a40d1580 0x7ff2a40d19e0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.366+0000 7ff2aaf9d700 1 --2- 172.21.15.98:0/482008604 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2a40b51c0 0x7ff2a4160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.366+0000 7ff2aaf9d700 1 -- 172.21.15.98:0/482008604 >> 172.21.15.98:0/482008604 conn(0x7ff2a406b8e0 msgr2=0x7ff2a40cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:05.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.366+0000 7ff2aaf9d700 1 -- 172.21.15.98:0/482008604 shutdown_connections 2023-03-03T08:01:05.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.366+0000 7ff2aaf9d700 1 -- 172.21.15.98:0/482008604 wait complete. 2023-03-03T08:01:05.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.368+0000 7ff2aaf9d700 1 Processor -- start 2023-03-03T08:01:05.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.369+0000 7ff2aaf9d700 1 -- start start 2023-03-03T08:01:05.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.369+0000 7ff2aaf9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2a40b51c0 0x7ff2a40c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:05.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.369+0000 7ff2aaf9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a40d1580 0x7ff2a415f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:05.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.369+0000 7ff2aaf9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2a4146360 0x7ff2a415f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:05.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.369+0000 7ff2aaf9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40c63d0 con 0x7ff2a40d1580 2023-03-03T08:01:05.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.369+0000 7ff2aaf9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40b2c10 con 0x7ff2a4146360 2023-03-03T08:01:05.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.369+0000 7ff2aaf9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a415faa0 con 0x7ff2a40b51c0 2023-03-03T08:01:05.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.370+0000 7ff2a979a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a40d1580 0x7ff2a415f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:05.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.370+0000 7ff2a979a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a40d1580 0x7ff2a415f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32890/0 (socket says 172.21.15.98:32890) 2023-03-03T08:01:05.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.370+0000 7ff2a979a700 1 -- 172.21.15.98:0/3754169872 learned_addr learned my addr 172.21.15.98:0/3754169872 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:05.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.370+0000 7ff2aa79c700 1 --2- 172.21.15.98:0/3754169872 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2a4146360 0x7ff2a415f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:05.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.370+0000 7ff2a9f9b700 1 --2- 172.21.15.98:0/3754169872 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2a40b51c0 0x7ff2a40c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:05.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.370+0000 7ff2a979a700 1 -- 172.21.15.98:0/3754169872 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2a40b51c0 msgr2=0x7ff2a40c5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:05.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.370+0000 7ff2a979a700 1 --2- 172.21.15.98:0/3754169872 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2a40b51c0 0x7ff2a40c5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.370+0000 7ff2a979a700 1 -- 172.21.15.98:0/3754169872 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2a4146360 msgr2=0x7ff2a415f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:05.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.370+0000 7ff2a979a700 1 --2- 172.21.15.98:0/3754169872 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2a4146360 0x7ff2a415f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.370+0000 7ff2a979a700 1 -- 172.21.15.98:0/3754169872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29c009b30 con 0x7ff2a40d1580 2023-03-03T08:01:05.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.370+0000 7ff2a9f9b700 1 --2- 172.21.15.98:0/3754169872 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2a40b51c0 0x7ff2a40c5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:05.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.371+0000 7ff2a979a700 1 --2- 172.21.15.98:0/3754169872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a40d1580 0x7ff2a415f020 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ff29400e870 tx=0x7ff29400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:05.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.371+0000 7ff29a7fc700 1 -- 172.21.15.98:0/3754169872 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294009590 con 0x7ff2a40d1580 2023-03-03T08:01:05.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.371+0000 7ff29a7fc700 1 -- 172.21.15.98:0/3754169872 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff294013680 con 0x7ff2a40d1580 2023-03-03T08:01:05.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.371+0000 7ff2aaf9d700 1 -- 172.21.15.98:0/3754169872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a40cdc40 con 0x7ff2a40d1580 2023-03-03T08:01:05.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.371+0000 7ff29a7fc700 1 -- 172.21.15.98:0/3754169872 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294013a30 con 0x7ff2a40d1580 2023-03-03T08:01:05.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.371+0000 7ff2aaf9d700 1 -- 172.21.15.98:0/3754169872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a40cd960 con 0x7ff2a40d1580 2023-03-03T08:01:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.372+0000 7ff2aaf9d700 1 -- 172.21.15.98:0/3754169872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a40d0d80 con 0x7ff2a40d1580 2023-03-03T08:01:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.374+0000 7ff29a7fc700 1 -- 172.21.15.98:0/3754169872 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7ff294019030 con 0x7ff2a40d1580 2023-03-03T08:01:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.375+0000 7ff29a7fc700 1 --2- 172.21.15.98:0/3754169872 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff2880684a0 0x7ff28806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.375+0000 7ff29a7fc700 1 -- 172.21.15.98:0/3754169872 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff294085cc0 con 0x7ff2a40d1580 2023-03-03T08:01:05.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.375+0000 7ff2a9f9b700 1 --2- 172.21.15.98:0/3754169872 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff2880684a0 0x7ff28806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:05.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.376+0000 7ff2a9f9b700 1 --2- 172.21.15.98:0/3754169872 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff2880684a0 0x7ff28806a950 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff2a40cf8e0 tx=0x7ff2a0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:05.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.379+0000 7ff29a7fc700 1 -- 172.21.15.98:0/3754169872 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff29408c080 con 0x7ff2a40d1580 2023-03-03T08:01:05.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.574+0000 7ff2aaf9d700 1 -- 172.21.15.98:0/3754169872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2a4068300 con 0x7ff2a40d1580 2023-03-03T08:01:05.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.575+0000 7ff29a7fc700 1 -- 172.21.15.98:0/3754169872 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+17921 (secure 0 0 0) 0x7ff294052380 con 0x7ff2a40d1580 2023-03-03T08:01:05.577 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:05.577 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":66,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:00:23.326015+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:00:19.823120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-03-03T08:00:21.324346+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:01:05.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.578+0000 7ff2aaf9d700 1 -- 172.21.15.98:0/3754169872 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff2880684a0 msgr2=0x7ff28806a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.578+0000 7ff2aaf9d700 1 --2- 172.21.15.98:0/3754169872 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff2880684a0 0x7ff28806a950 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff2a40cf8e0 tx=0x7ff2a0009040 comp rx=0 tx=0).stop 2023-03-03T08:01:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.578+0000 7ff2aaf9d700 1 -- 172.21.15.98:0/3754169872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a40d1580 msgr2=0x7ff2a415f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.578+0000 7ff2aaf9d700 1 --2- 172.21.15.98:0/3754169872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a40d1580 0x7ff2a415f020 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ff29400e870 tx=0x7ff29400eb80 comp rx=0 tx=0).stop 2023-03-03T08:01:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.578+0000 7ff2aaf9d700 1 -- 172.21.15.98:0/3754169872 shutdown_connections 2023-03-03T08:01:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.578+0000 7ff2aaf9d700 1 --2- 172.21.15.98:0/3754169872 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff2880684a0 0x7ff28806a950 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.578+0000 7ff2aaf9d700 1 --2- 172.21.15.98:0/3754169872 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2a4146360 0x7ff2a415f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.578+0000 7ff2aaf9d700 1 --2- 172.21.15.98:0/3754169872 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2a40b51c0 0x7ff2a40c5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.578+0000 7ff2aaf9d700 1 --2- 172.21.15.98:0/3754169872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a40d1580 0x7ff2a415f020 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.578+0000 7ff2aaf9d700 1 -- 172.21.15.98:0/3754169872 >> 172.21.15.98:0/3754169872 conn(0x7ff2a406b8e0 msgr2=0x7ff2a40731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:05.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.578+0000 7ff2aaf9d700 1 -- 172.21.15.98:0/3754169872 shutdown_connections 2023-03-03T08:01:05.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.579+0000 7ff2aaf9d700 1 -- 172.21.15.98:0/3754169872 wait complete. 2023-03-03T08:01:05.607 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-03-03T08:01:05.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.853+0000 7fc99b59e700 1 Processor -- start 2023-03-03T08:01:05.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.853+0000 7fc99b59e700 1 -- start start 2023-03-03T08:01:05.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.855+0000 7fc99b59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc99c1609b0 0x7fc99c160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:05.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.855+0000 7fc99b59e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc99c0cf570 0x7fc99c15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:05.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.855+0000 7fc99b59e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc99c15db30 0x7fc99c0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:05.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.855+0000 7fc99b59e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc99c0cebb0 con 0x7fc99c1609b0 2023-03-03T08:01:05.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.855+0000 7fc99b59e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc99c1612c0 con 0x7fc99c0cf570 2023-03-03T08:01:05.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.855+0000 7fc99b59e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c0cd780 con 0x7fc99c15db30 2023-03-03T08:01:05.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.855+0000 7fc99a59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc99c1609b0 0x7fc99c160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:05.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.855+0000 7fc99a59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc99c1609b0 0x7fc99c160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32892/0 (socket says 172.21.15.98:32892) 2023-03-03T08:01:05.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.855+0000 7fc99a59c700 1 -- 172.21.15.98:0/138715188 learned_addr learned my addr 172.21.15.98:0/138715188 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:05.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.856+0000 7fc999d9b700 1 --2- 172.21.15.98:0/138715188 >> v2:172.21.15.100:3300/0 conn(0x7fc99c0cf570 0x7fc99c15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:05.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.856+0000 7fc99ad9d700 1 --2- 172.21.15.98:0/138715188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc99c15db30 0x7fc99c0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:05.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.856+0000 7fc99a59c700 1 -- 172.21.15.98:0/138715188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc99c15db30 msgr2=0x7fc99c0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:05.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.856+0000 7fc99a59c700 1 --2- 172.21.15.98:0/138715188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc99c15db30 0x7fc99c0b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.856+0000 7fc99a59c700 1 -- 172.21.15.98:0/138715188 >> v2:172.21.15.100:3300/0 conn(0x7fc99c0cf570 msgr2=0x7fc99c15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:05.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.856+0000 7fc99a59c700 1 --2- 172.21.15.98:0/138715188 >> v2:172.21.15.100:3300/0 conn(0x7fc99c0cf570 0x7fc99c15d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.856+0000 7fc99a59c700 1 -- 172.21.15.98:0/138715188 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99c0cda60 con 0x7fc99c1609b0 2023-03-03T08:01:05.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.856+0000 7fc999d9b700 1 --2- 172.21.15.98:0/138715188 >> v2:172.21.15.100:3300/0 conn(0x7fc99c0cf570 0x7fc99c15d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:05.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.856+0000 7fc99a59c700 1 --2- 172.21.15.98:0/138715188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc99c1609b0 0x7fc99c160d80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0cf0c0 tx=0x7fc98c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c3d0c57471299b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:05.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.856+0000 7fc99959a700 1 -- 172.21.15.98:0/138715188 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98c00b870 con 0x7fc99c1609b0 2023-03-03T08:01:05.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.857+0000 7fc99959a700 1 -- 172.21.15.98:0/138715188 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc98c00b9d0 con 0x7fc99c1609b0 2023-03-03T08:01:05.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.857+0000 7fc99959a700 1 -- 172.21.15.98:0/138715188 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98c00bd60 con 0x7fc99c1609b0 2023-03-03T08:01:05.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.857+0000 7fc99b59e700 1 -- 172.21.15.98:0/138715188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc99c1609b0 msgr2=0x7fc99c160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:05.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.857+0000 7fc99b59e700 1 --2- 172.21.15.98:0/138715188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc99c1609b0 0x7fc99c160d80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0cf0c0 tx=0x7fc98c009bd0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.857+0000 7fc99b59e700 1 -- 172.21.15.98:0/138715188 shutdown_connections 2023-03-03T08:01:05.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.857+0000 7fc99b59e700 1 --2- 172.21.15.98:0/138715188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc99c15db30 0x7fc99c0b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.857+0000 7fc99b59e700 1 --2- 172.21.15.98:0/138715188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc99c1609b0 0x7fc99c160d80 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.857+0000 7fc99b59e700 1 --2- 172.21.15.98:0/138715188 >> v2:172.21.15.100:3300/0 conn(0x7fc99c0cf570 0x7fc99c15d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.857+0000 7fc99b59e700 1 -- 172.21.15.98:0/138715188 >> 172.21.15.98:0/138715188 conn(0x7fc99c06b880 msgr2=0x7fc99c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:05.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.857+0000 7fc99b59e700 1 -- 172.21.15.98:0/138715188 shutdown_connections 2023-03-03T08:01:05.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.857+0000 7fc99b59e700 1 -- 172.21.15.98:0/138715188 wait complete. 2023-03-03T08:01:05.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.859+0000 7fc99b59e700 1 Processor -- start 2023-03-03T08:01:05.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.860+0000 7fc99b59e700 1 -- start start 2023-03-03T08:01:05.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99b59e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc99c0cf570 0x7fc99c150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:05.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99b59e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc99c15db30 0x7fc99c0b5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:05.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99b59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc99c1609b0 0x7fc99c0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:05.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99b59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c13b520 con 0x7fc99c1609b0 2023-03-03T08:01:05.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99b59e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c1e6620 con 0x7fc99c0cf570 2023-03-03T08:01:05.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99b59e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c0d1f90 con 0x7fc99c15db30 2023-03-03T08:01:05.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc99c1609b0 0x7fc99c0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:05.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc99c1609b0 0x7fc99c0b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32896/0 (socket says 172.21.15.98:32896) 2023-03-03T08:01:05.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99ad9d700 1 -- 172.21.15.98:0/1596890531 learned_addr learned my addr 172.21.15.98:0/1596890531 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:05.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99a59c700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc99c0cf570 0x7fc99c150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc999d9b700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc99c15db30 0x7fc99c0b5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99ad9d700 1 -- 172.21.15.98:0/1596890531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc99c15db30 msgr2=0x7fc99c0b5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99ad9d700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc99c15db30 0x7fc99c0b5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99ad9d700 1 -- 172.21.15.98:0/1596890531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc99c0cf570 msgr2=0x7fc99c150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99ad9d700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc99c0cf570 0x7fc99c150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99ad9d700 1 -- 172.21.15.98:0/1596890531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98c009550 con 0x7fc99c1609b0 2023-03-03T08:01:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc99a59c700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc99c0cf570 0x7fc99c150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:05.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.861+0000 7fc999d9b700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc99c15db30 0x7fc99c0b5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:05.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.862+0000 7fc99ad9d700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc99c1609b0 0x7fc99c0b6390 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fc994007e70 tx=0x7fc99400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:05.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.862+0000 7fc98affd700 1 -- 172.21.15.98:0/1596890531 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99400ee50 con 0x7fc99c1609b0 2023-03-03T08:01:05.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.862+0000 7fc98affd700 1 -- 172.21.15.98:0/1596890531 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc994014420 con 0x7fc99c1609b0 2023-03-03T08:01:05.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.862+0000 7fc98affd700 1 -- 172.21.15.98:0/1596890531 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9940147d0 con 0x7fc99c1609b0 2023-03-03T08:01:05.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.862+0000 7fc99b59e700 1 -- 172.21.15.98:0/1596890531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99c15cdf0 con 0x7fc99c1609b0 2023-03-03T08:01:05.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.862+0000 7fc99b59e700 1 -- 172.21.15.98:0/1596890531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99c0cdd40 con 0x7fc99c1609b0 2023-03-03T08:01:05.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.863+0000 7fc99b59e700 1 -- 172.21.15.98:0/1596890531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc99c15d0d0 con 0x7fc99c1609b0 2023-03-03T08:01:05.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.865+0000 7fc98affd700 1 -- 172.21.15.98:0/1596890531 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fc994012050 con 0x7fc99c1609b0 2023-03-03T08:01:05.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.866+0000 7fc98affd700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc97c0684a0 0x7fc97c06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:05.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.866+0000 7fc98affd700 1 -- 172.21.15.98:0/1596890531 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc9940865b0 con 0x7fc99c1609b0 2023-03-03T08:01:05.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.866+0000 7fc99a59c700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc97c0684a0 0x7fc97c06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:05.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.867+0000 7fc99a59c700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc97c0684a0 0x7fc97c06a950 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc98c014040 tx=0x7fc98c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:05.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:05.870+0000 7fc98affd700 1 -- 172.21.15.98:0/1596890531 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc9940566a0 con 0x7fc99c1609b0 2023-03-03T08:01:06.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:06.106+0000 7fc99b59e700 1 -- 172.21.15.98:0/1596890531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc99c13b660 con 0x7fc99c1609b0 2023-03-03T08:01:06.725 INFO:tasks.ceph.mon.a.smithi098.stderr:2023-03-03T08:01:06.724+0000 7fb71abf9700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-03-03T08:01:06.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:06.733+0000 7fc98affd700 1 -- 172.21.15.98:0/1596890531 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v33) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc99c13b660 con 0x7fc99c1609b0 2023-03-03T08:01:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:06.736+0000 7fc99b59e700 1 -- 172.21.15.98:0/1596890531 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc97c0684a0 msgr2=0x7fc97c06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:06.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:06.736+0000 7fc99b59e700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc97c0684a0 0x7fc97c06a950 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc98c014040 tx=0x7fc98c008030 comp rx=0 tx=0).stop 2023-03-03T08:01:06.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:06.736+0000 7fc99b59e700 1 -- 172.21.15.98:0/1596890531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc99c1609b0 msgr2=0x7fc99c0b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:06.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:06.736+0000 7fc99b59e700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc99c1609b0 0x7fc99c0b6390 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fc994007e70 tx=0x7fc99400e3c0 comp rx=0 tx=0).stop 2023-03-03T08:01:06.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:06.736+0000 7fc99b59e700 1 -- 172.21.15.98:0/1596890531 shutdown_connections 2023-03-03T08:01:06.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:06.736+0000 7fc99b59e700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc97c0684a0 0x7fc97c06a950 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:06.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:06.736+0000 7fc99b59e700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc99c1609b0 0x7fc99c0b6390 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:06.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:06.736+0000 7fc99b59e700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc99c0cf570 0x7fc99c150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:06.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:06.736+0000 7fc99b59e700 1 --2- 172.21.15.98:0/1596890531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc99c15db30 0x7fc99c0b5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:06.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:06.736+0000 7fc99b59e700 1 -- 172.21.15.98:0/1596890531 >> 172.21.15.98:0/1596890531 conn(0x7fc99c06b880 msgr2=0x7fc99c15b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:06.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:06.737+0000 7fc99b59e700 1 -- 172.21.15.98:0/1596890531 shutdown_connections 2023-03-03T08:01:06.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:06.737+0000 7fc99b59e700 1 -- 172.21.15.98:0/1596890531 wait complete. 2023-03-03T08:01:06.740 INFO:teuthology.orchestra.run.smithi098.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-03-03T08:01:06.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-03-03T08:01:06.833 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-03-03T08:01:07.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.020+0000 7f7df03ff700 1 Processor -- start 2023-03-03T08:01:07.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.020+0000 7f7df03ff700 1 -- start start 2023-03-03T08:01:07.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.021+0000 7f7df03ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de80b26d0 0x7f7de80b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:07.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.021+0000 7f7df03ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7de80b5140 0x7f7de80d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:07.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.022+0000 7f7df03ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de80cd7d0 con 0x7f7de80b8d10 2023-03-03T08:01:07.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.022+0000 7f7df03ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de80b55c0 con 0x7f7de80b5140 2023-03-03T08:01:07.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.022+0000 7f7df03ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de80b90e0 con 0x7f7de80b26d0 2023-03-03T08:01:07.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.022+0000 7f7dee19b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7de80b8d10 0x7f7de80cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54608/0 (socket says 172.21.15.98:54608) 2023-03-03T08:01:07.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.022+0000 7f7dee19b700 1 -- 172.21.15.98:0/531965996 learned_addr learned my addr 172.21.15.98:0/531965996 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:07.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.022+0000 7f7ded99a700 1 --2- 172.21.15.98:0/531965996 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de80b26d0 0x7f7de80b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:07.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.022+0000 7f7dee99c700 1 --2- 172.21.15.98:0/531965996 >> v2:172.21.15.100:3300/0 conn(0x7f7de80b5140 0x7f7de80d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:07.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.022+0000 7f7ded199700 1 -- 172.21.15.98:0/531965996 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247450171 0 0) 0x7f7de80cd7d0 con 0x7f7de80b8d10 2023-03-03T08:01:07.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.023+0000 7f7ded199700 1 -- 172.21.15.98:0/531965996 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd40035b0 con 0x7f7de80b8d10 2023-03-03T08:01:07.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.023+0000 7f7ded99a700 1 -- 172.21.15.98:0/531965996 >> v2:172.21.15.100:3300/0 conn(0x7f7de80b5140 msgr2=0x7f7de80d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:07.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.023+0000 7f7ded99a700 1 --2- 172.21.15.98:0/531965996 >> v2:172.21.15.100:3300/0 conn(0x7f7de80b5140 0x7f7de80d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:07.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.023+0000 7f7ded99a700 1 -- 172.21.15.98:0/531965996 >> v1:172.21.15.98:6789/0 conn(0x7f7de80b8d10 legacy=0x7f7de80cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:07.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.023+0000 7f7ded99a700 1 -- 172.21.15.98:0/531965996 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de80cda20 con 0x7f7de80b26d0 2023-03-03T08:01:07.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.023+0000 7f7ded99a700 1 --2- 172.21.15.98:0/531965996 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de80b26d0 0x7f7de80b4c00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8009840 tx=0x7f7dd8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0aa642be2868d30 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:07.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.023+0000 7f7ded199700 1 -- 172.21.15.98:0/531965996 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8011830 con 0x7f7de80b26d0 2023-03-03T08:01:07.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.024+0000 7f7ded199700 1 -- 172.21.15.98:0/531965996 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd8011990 con 0x7f7de80b26d0 2023-03-03T08:01:07.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.024+0000 7f7ded199700 1 -- 172.21.15.98:0/531965996 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8011d20 con 0x7f7de80b26d0 2023-03-03T08:01:07.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.024+0000 7f7df03ff700 1 -- 172.21.15.98:0/531965996 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de80b26d0 msgr2=0x7f7de80b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:07.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.024+0000 7f7df03ff700 1 --2- 172.21.15.98:0/531965996 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de80b26d0 0x7f7de80b4c00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8009840 tx=0x7f7dd8009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:07.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.024+0000 7f7df03ff700 1 -- 172.21.15.98:0/531965996 shutdown_connections 2023-03-03T08:01:07.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.024+0000 7f7df03ff700 1 --2- 172.21.15.98:0/531965996 >> v2:172.21.15.100:3300/0 conn(0x7f7de80b5140 0x7f7de80d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:07.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.024+0000 7f7df03ff700 1 --2- 172.21.15.98:0/531965996 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de80b26d0 0x7f7de80b4c00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:07.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.024+0000 7f7df03ff700 1 -- 172.21.15.98:0/531965996 >> 172.21.15.98:0/531965996 conn(0x7f7de8073e70 msgr2=0x7f7de806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:07.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.024+0000 7f7df03ff700 1 -- 172.21.15.98:0/531965996 shutdown_connections 2023-03-03T08:01:07.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.024+0000 7f7df03ff700 1 -- 172.21.15.98:0/531965996 wait complete. 2023-03-03T08:01:07.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.027+0000 7f7df03ff700 1 Processor -- start 2023-03-03T08:01:07.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.028+0000 7f7df03ff700 1 -- start start 2023-03-03T08:01:07.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.029+0000 7f7df03ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de80b26d0 0x7f7de81f5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:07.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.029+0000 7f7df03ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de80b5140 0x7f7de815f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:07.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.029+0000 7f7df03ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7de80b8d10 0x7f7de8155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:07.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.029+0000 7f7df03ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de80ccbd0 con 0x7f7de80b26d0 2023-03-03T08:01:07.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.029+0000 7f7df03ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de81f5b40 con 0x7f7de80b8d10 2023-03-03T08:01:07.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.029+0000 7f7df03ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de80cbfc0 con 0x7f7de80b5140 2023-03-03T08:01:07.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.029+0000 7f7dee19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de80b26d0 0x7f7de81f5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:07.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.029+0000 7f7dee19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de80b26d0 0x7f7de81f5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32924/0 (socket says 172.21.15.98:32924) 2023-03-03T08:01:07.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.029+0000 7f7dee19b700 1 -- 172.21.15.98:0/992893878 learned_addr learned my addr 172.21.15.98:0/992893878 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:07.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.030+0000 7f7ded99a700 1 --2- 172.21.15.98:0/992893878 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de80b5140 0x7f7de815f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:07.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.030+0000 7f7dee99c700 1 --2- 172.21.15.98:0/992893878 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7de80b8d10 0x7f7de8155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:07.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.030+0000 7f7dee19b700 1 -- 172.21.15.98:0/992893878 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de80b5140 msgr2=0x7f7de815f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:07.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.030+0000 7f7dee19b700 1 --2- 172.21.15.98:0/992893878 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de80b5140 0x7f7de815f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:07.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.030+0000 7f7dee19b700 1 -- 172.21.15.98:0/992893878 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7de80b8d10 msgr2=0x7f7de8155470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:07.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.030+0000 7f7dee19b700 1 --2- 172.21.15.98:0/992893878 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7de80b8d10 0x7f7de8155470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:07.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.030+0000 7f7dee19b700 1 -- 172.21.15.98:0/992893878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd80094a0 con 0x7f7de80b26d0 2023-03-03T08:01:07.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.030+0000 7f7dee19b700 1 --2- 172.21.15.98:0/992893878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de80b26d0 0x7f7de81f5600 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7de4001eb0 tx=0x7f7de400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:07.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.030+0000 7f7ddeffd700 1 -- 172.21.15.98:0/992893878 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de4005d70 con 0x7f7de80b26d0 2023-03-03T08:01:07.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.030+0000 7f7ddeffd700 1 -- 172.21.15.98:0/992893878 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de40070d0 con 0x7f7de80b26d0 2023-03-03T08:01:07.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.030+0000 7f7df03ff700 1 -- 172.21.15.98:0/992893878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de80d0e80 con 0x7f7de80b26d0 2023-03-03T08:01:07.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.030+0000 7f7ddeffd700 1 -- 172.21.15.98:0/992893878 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de4007460 con 0x7f7de80b26d0 2023-03-03T08:01:07.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.031+0000 7f7df03ff700 1 -- 172.21.15.98:0/992893878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de80cdd00 con 0x7f7de80b26d0 2023-03-03T08:01:07.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.032+0000 7f7df03ff700 1 -- 172.21.15.98:0/992893878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de80cdfe0 con 0x7f7de80b26d0 2023-03-03T08:01:07.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.033+0000 7f7ddeffd700 1 -- 172.21.15.98:0/992893878 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f7de4019030 con 0x7f7de80b26d0 2023-03-03T08:01:07.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.034+0000 7f7ddeffd700 1 --2- 172.21.15.98:0/992893878 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7dcc0684a0 0x7f7dcc06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:07.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.034+0000 7f7ddeffd700 1 -- 172.21.15.98:0/992893878 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7de4085380 con 0x7f7de80b26d0 2023-03-03T08:01:07.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.035+0000 7f7ded99a700 1 --2- 172.21.15.98:0/992893878 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7dcc0684a0 0x7f7dcc06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:07.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.035+0000 7f7ded99a700 1 --2- 172.21.15.98:0/992893878 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7dcc0684a0 0x7f7dcc06a950 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8000c00 tx=0x7f7dd801c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:07.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.038+0000 7f7ddeffd700 1 -- 172.21.15.98:0/992893878 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7de40519e0 con 0x7f7de80b26d0 2023-03-03T08:01:07.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.274+0000 7f7df03ff700 1 -- 172.21.15.98:0/992893878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7de804e6e0 con 0x7f7de80b26d0 2023-03-03T08:01:07.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.735+0000 7f7ddeffd700 1 -- 172.21.15.98:0/992893878 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v34) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7de4055000 con 0x7f7de80b26d0 2023-03-03T08:01:07.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.739+0000 7f7df03ff700 1 -- 172.21.15.98:0/992893878 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7dcc0684a0 msgr2=0x7f7dcc06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:07.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.739+0000 7f7df03ff700 1 --2- 172.21.15.98:0/992893878 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7dcc0684a0 0x7f7dcc06a950 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8000c00 tx=0x7f7dd801c040 comp rx=0 tx=0).stop 2023-03-03T08:01:07.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.739+0000 7f7df03ff700 1 -- 172.21.15.98:0/992893878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de80b26d0 msgr2=0x7f7de81f5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:07.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.739+0000 7f7df03ff700 1 --2- 172.21.15.98:0/992893878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de80b26d0 0x7f7de81f5600 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7de4001eb0 tx=0x7f7de400c860 comp rx=0 tx=0).stop 2023-03-03T08:01:07.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.739+0000 7f7df03ff700 1 -- 172.21.15.98:0/992893878 shutdown_connections 2023-03-03T08:01:07.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.739+0000 7f7df03ff700 1 --2- 172.21.15.98:0/992893878 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7dcc0684a0 0x7f7dcc06a950 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:07.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.739+0000 7f7df03ff700 1 --2- 172.21.15.98:0/992893878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de80b26d0 0x7f7de81f5600 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:07.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.739+0000 7f7df03ff700 1 --2- 172.21.15.98:0/992893878 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7de80b8d10 0x7f7de8155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:07.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.739+0000 7f7df03ff700 1 --2- 172.21.15.98:0/992893878 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de80b5140 0x7f7de815f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:07.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.739+0000 7f7df03ff700 1 -- 172.21.15.98:0/992893878 >> 172.21.15.98:0/992893878 conn(0x7f7de8073e70 msgr2=0x7f7de80d13a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:07.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.739+0000 7f7df03ff700 1 -- 172.21.15.98:0/992893878 shutdown_connections 2023-03-03T08:01:07.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:07.739+0000 7f7df03ff700 1 -- 172.21.15.98:0/992893878 wait complete. 2023-03-03T08:01:07.769 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-03-03T08:01:08.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.018+0000 7f43537ff700 1 Processor -- start 2023-03-03T08:01:08.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.018+0000 7f43537ff700 1 -- start start 2023-03-03T08:01:08.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.020+0000 7f43537ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f434c0b8880 0x7f434c0cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:08.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.020+0000 7f43537ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b4ba0 0x7f434c0b5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.020+0000 7f43537ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f434c0d0ac0 con 0x7f434c0b5550 2023-03-03T08:01:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.020+0000 7f43537ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f434c0b59a0 con 0x7f434c0b8880 2023-03-03T08:01:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.020+0000 7f43537ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c0cff70 con 0x7f434c0b4ba0 2023-03-03T08:01:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.020+0000 7f4351d9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f434c0b5550 0x7f434c1604c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54616/0 (socket says 172.21.15.98:54616) 2023-03-03T08:01:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.020+0000 7f4351d9c700 1 -- 172.21.15.98:0/2508967870 learned_addr learned my addr 172.21.15.98:0/2508967870 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.020+0000 7f4350d9a700 1 --2- 172.21.15.98:0/2508967870 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b4ba0 0x7f434c0b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:08.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.020+0000 7f435159b700 1 --2- 172.21.15.98:0/2508967870 >> v2:172.21.15.100:3300/0 conn(0x7f434c0b8880 0x7f434c0cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.021+0000 7f4343fff700 1 -- 172.21.15.98:0/2508967870 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166008979 0 0) 0x7f434c0d0ac0 con 0x7f434c0b5550 2023-03-03T08:01:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.021+0000 7f4343fff700 1 -- 172.21.15.98:0/2508967870 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43380035b0 con 0x7f434c0b5550 2023-03-03T08:01:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.021+0000 7f435159b700 1 -- 172.21.15.98:0/2508967870 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b4ba0 msgr2=0x7f434c0b5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.021+0000 7f435159b700 1 --2- 172.21.15.98:0/2508967870 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b4ba0 0x7f434c0b5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.021+0000 7f435159b700 1 -- 172.21.15.98:0/2508967870 >> v1:172.21.15.98:6789/0 conn(0x7f434c0b5550 legacy=0x7f434c1604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:08.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.021+0000 7f435159b700 1 -- 172.21.15.98:0/2508967870 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434c0d0250 con 0x7f434c0b8880 2023-03-03T08:01:08.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.021+0000 7f4350d9a700 1 --2- 172.21.15.98:0/2508967870 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b4ba0 0x7f434c0b5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:08.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.021+0000 7f435159b700 1 --2- 172.21.15.98:0/2508967870 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434c0b8880 0x7f434c0cd770 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4348009840 tx=0x7f4348009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3180ab32a6cd0a2c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:08.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.022+0000 7f4343fff700 1 -- 172.21.15.98:0/2508967870 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348011830 con 0x7f434c0b8880 2023-03-03T08:01:08.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.022+0000 7f4343fff700 1 -- 172.21.15.98:0/2508967870 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4348011990 con 0x7f434c0b8880 2023-03-03T08:01:08.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.022+0000 7f4343fff700 1 -- 172.21.15.98:0/2508967870 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348011d20 con 0x7f434c0b8880 2023-03-03T08:01:08.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.022+0000 7f43537ff700 1 -- 172.21.15.98:0/2508967870 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434c0b8880 msgr2=0x7f434c0cd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:08.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.022+0000 7f43537ff700 1 --2- 172.21.15.98:0/2508967870 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434c0b8880 0x7f434c0cd770 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4348009840 tx=0x7f4348009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:08.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.022+0000 7f43537ff700 1 -- 172.21.15.98:0/2508967870 shutdown_connections 2023-03-03T08:01:08.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.022+0000 7f43537ff700 1 --2- 172.21.15.98:0/2508967870 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434c0b8880 0x7f434c0cd770 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:08.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.022+0000 7f43537ff700 1 --2- 172.21.15.98:0/2508967870 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b4ba0 0x7f434c0b5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:08.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.022+0000 7f43537ff700 1 -- 172.21.15.98:0/2508967870 >> 172.21.15.98:0/2508967870 conn(0x7f434c06ad80 msgr2=0x7f434c06b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:08.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.022+0000 7f43537ff700 1 -- 172.21.15.98:0/2508967870 shutdown_connections 2023-03-03T08:01:08.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.022+0000 7f43537ff700 1 -- 172.21.15.98:0/2508967870 wait complete. 2023-03-03T08:01:08.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.024+0000 7f43537ff700 1 Processor -- start 2023-03-03T08:01:08.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.025+0000 7f43537ff700 1 -- start start 2023-03-03T08:01:08.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.026+0000 7f43537ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f434c0b4ba0 0x7f434c159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:08.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.026+0000 7f43537ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434c0b5550 0x7f434c15b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:08.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.026+0000 7f43537ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b8880 0x7f434c155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:08.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.026+0000 7f43537ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c15bd40 con 0x7f434c0b4ba0 2023-03-03T08:01:08.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.026+0000 7f43537ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c15c390 con 0x7f434c0b5550 2023-03-03T08:01:08.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.026+0000 7f43537ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c15c040 con 0x7f434c0b8880 2023-03-03T08:01:08.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.026+0000 7f435159b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f434c0b4ba0 0x7f434c159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:08.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.026+0000 7f435159b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f434c0b4ba0 0x7f434c159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32930/0 (socket says 172.21.15.98:32930) 2023-03-03T08:01:08.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.026+0000 7f435159b700 1 -- 172.21.15.98:0/3267439921 learned_addr learned my addr 172.21.15.98:0/3267439921 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:08.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.026+0000 7f4350d9a700 1 --2- 172.21.15.98:0/3267439921 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434c0b5550 0x7f434c15b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:08.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.026+0000 7f4351d9c700 1 --2- 172.21.15.98:0/3267439921 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b8880 0x7f434c155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:08.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.026+0000 7f435159b700 1 -- 172.21.15.98:0/3267439921 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b8880 msgr2=0x7f434c155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:08.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.026+0000 7f435159b700 1 --2- 172.21.15.98:0/3267439921 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b8880 0x7f434c155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:08.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.027+0000 7f435159b700 1 -- 172.21.15.98:0/3267439921 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434c0b5550 msgr2=0x7f434c15b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:08.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.027+0000 7f435159b700 1 --2- 172.21.15.98:0/3267439921 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434c0b5550 0x7f434c15b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:08.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.027+0000 7f435159b700 1 -- 172.21.15.98:0/3267439921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c0094a0 con 0x7f434c0b4ba0 2023-03-03T08:01:08.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.027+0000 7f4350d9a700 1 --2- 172.21.15.98:0/3267439921 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434c0b5550 0x7f434c15b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:08.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.027+0000 7f435159b700 1 --2- 172.21.15.98:0/3267439921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f434c0b4ba0 0x7f434c159740 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f4348009fd0 tx=0x7f4348004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:08.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.027+0000 7f4341ffb700 1 -- 172.21.15.98:0/3267439921 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43480036a0 con 0x7f434c0b4ba0 2023-03-03T08:01:08.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.027+0000 7f4341ffb700 1 -- 172.21.15.98:0/3267439921 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434801e070 con 0x7f434c0b4ba0 2023-03-03T08:01:08.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.027+0000 7f4341ffb700 1 -- 172.21.15.98:0/3267439921 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348003a60 con 0x7f434c0b4ba0 2023-03-03T08:01:08.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.027+0000 7f43537ff700 1 -- 172.21.15.98:0/3267439921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43480094a0 con 0x7f434c0b4ba0 2023-03-03T08:01:08.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.027+0000 7f43537ff700 1 -- 172.21.15.98:0/3267439921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434c0d0cf0 con 0x7f434c0b4ba0 2023-03-03T08:01:08.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.029+0000 7f43537ff700 1 -- 172.21.15.98:0/3267439921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434c0d0530 con 0x7f434c0b4ba0 2023-03-03T08:01:08.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.030+0000 7f4341ffb700 1 -- 172.21.15.98:0/3267439921 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f434800f050 con 0x7f434c0b4ba0 2023-03-03T08:01:08.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.031+0000 7f4341ffb700 1 --2- 172.21.15.98:0/3267439921 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f43300684a0 0x7f433006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:08.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.031+0000 7f4341ffb700 1 -- 172.21.15.98:0/3267439921 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4348086ee0 con 0x7f434c0b4ba0 2023-03-03T08:01:08.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.031+0000 7f4350d9a700 1 --2- 172.21.15.98:0/3267439921 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f43300684a0 0x7f433006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:08.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.032+0000 7f4350d9a700 1 --2- 172.21.15.98:0/3267439921 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f43300684a0 0x7f433006a950 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f434c157150 tx=0x7f433c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:08.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.035+0000 7f4341ffb700 1 -- 172.21.15.98:0/3267439921 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4348053540 con 0x7f434c0b4ba0 2023-03-03T08:01:08.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.227+0000 7f43537ff700 1 -- 172.21.15.98:0/3267439921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f434c0b8c50 con 0x7f434c0b4ba0 2023-03-03T08:01:08.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.740+0000 7f4341ffb700 1 -- 172.21.15.98:0/3267439921 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v68) v1 ==== 181+0+0 (secure 0 0 0) 0x7f43480150d0 con 0x7f434c0b4ba0 2023-03-03T08:01:08.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.743+0000 7f43537ff700 1 -- 172.21.15.98:0/3267439921 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f43300684a0 msgr2=0x7f433006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:08.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.743+0000 7f43537ff700 1 --2- 172.21.15.98:0/3267439921 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f43300684a0 0x7f433006a950 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f434c157150 tx=0x7f433c008040 comp rx=0 tx=0).stop 2023-03-03T08:01:08.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.743+0000 7f43537ff700 1 -- 172.21.15.98:0/3267439921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f434c0b4ba0 msgr2=0x7f434c159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:08.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.743+0000 7f43537ff700 1 --2- 172.21.15.98:0/3267439921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f434c0b4ba0 0x7f434c159740 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f4348009fd0 tx=0x7f4348004750 comp rx=0 tx=0).stop 2023-03-03T08:01:08.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.743+0000 7f43537ff700 1 -- 172.21.15.98:0/3267439921 shutdown_connections 2023-03-03T08:01:08.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.743+0000 7f43537ff700 1 --2- 172.21.15.98:0/3267439921 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f43300684a0 0x7f433006a950 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:08.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.743+0000 7f43537ff700 1 --2- 172.21.15.98:0/3267439921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f434c0b4ba0 0x7f434c159740 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:08.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.743+0000 7f43537ff700 1 --2- 172.21.15.98:0/3267439921 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b8880 0x7f434c155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:08.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.743+0000 7f43537ff700 1 --2- 172.21.15.98:0/3267439921 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434c0b5550 0x7f434c15b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:08.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.743+0000 7f43537ff700 1 -- 172.21.15.98:0/3267439921 >> 172.21.15.98:0/3267439921 conn(0x7f434c06ad80 msgr2=0x7f434c0d2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:08.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.743+0000 7f43537ff700 1 -- 172.21.15.98:0/3267439921 shutdown_connections 2023-03-03T08:01:08.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:08.743+0000 7f43537ff700 1 -- 172.21.15.98:0/3267439921 wait complete. 2023-03-03T08:01:08.746 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' removed 2023-03-03T08:01:08.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-03-03T08:01:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.023+0000 7f6449bff700 1 Processor -- start 2023-03-03T08:01:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.023+0000 7f6449bff700 1 -- start start 2023-03-03T08:01:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.024+0000 7f6449bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f64440b8880 0x7f64440cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.024+0000 7f6449bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f64440b4ba0 0x7f64440b5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.024+0000 7f6449bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64440b5550 0x7f64441604c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.024+0000 7f6449bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64440d0ac0 con 0x7f64440b4ba0 2023-03-03T08:01:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.024+0000 7f6449bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64440b59d0 con 0x7f64440b8880 2023-03-03T08:01:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.024+0000 7f6449bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64440cff70 con 0x7f64440b5550 2023-03-03T08:01:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.025+0000 7f643bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f64440b4ba0 0x7f64440b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.025+0000 7f643bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f64440b4ba0 0x7f64440b5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32940/0 (socket says 172.21.15.98:32940) 2023-03-03T08:01:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.025+0000 7f643bfff700 1 -- 172.21.15.98:0/1095373182 learned_addr learned my addr 172.21.15.98:0/1095373182 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.025+0000 7f64437fe700 1 --2- 172.21.15.98:0/1095373182 >> v2:172.21.15.100:3300/0 conn(0x7f64440b8880 0x7f64440cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.025+0000 7f6443fff700 1 --2- 172.21.15.98:0/1095373182 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64440b5550 0x7f64441604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.025+0000 7f643bfff700 1 -- 172.21.15.98:0/1095373182 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64440b5550 msgr2=0x7f64441604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.025+0000 7f643bfff700 1 --2- 172.21.15.98:0/1095373182 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64440b5550 0x7f64441604c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.025+0000 7f643bfff700 1 -- 172.21.15.98:0/1095373182 >> v2:172.21.15.100:3300/0 conn(0x7f64440b8880 msgr2=0x7f64440cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.025+0000 7f643bfff700 1 --2- 172.21.15.98:0/1095373182 >> v2:172.21.15.100:3300/0 conn(0x7f64440b8880 0x7f64440cd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.026+0000 7f643bfff700 1 -- 172.21.15.98:0/1095373182 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64440d0250 con 0x7f64440b4ba0 2023-03-03T08:01:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.026+0000 7f6443fff700 1 --2- 172.21.15.98:0/1095373182 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64440b5550 0x7f64441604c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.026+0000 7f643bfff700 1 --2- 172.21.15.98:0/1095373182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64440b4ba0 0x7f64440b5010 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f64440b5cb0 tx=0x7f642c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39860cd0394f2413 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.026+0000 7f6442ffd700 1 -- 172.21.15.98:0/1095373182 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642c011830 con 0x7f64440b4ba0 2023-03-03T08:01:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.026+0000 7f6442ffd700 1 -- 172.21.15.98:0/1095373182 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642c011990 con 0x7f64440b4ba0 2023-03-03T08:01:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.026+0000 7f6442ffd700 1 -- 172.21.15.98:0/1095373182 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642c011d20 con 0x7f64440b4ba0 2023-03-03T08:01:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.027+0000 7f6449bff700 1 -- 172.21.15.98:0/1095373182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64440b4ba0 msgr2=0x7f64440b5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.027+0000 7f6449bff700 1 --2- 172.21.15.98:0/1095373182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64440b4ba0 0x7f64440b5010 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f64440b5cb0 tx=0x7f642c009bd0 comp rx=0 tx=0).stop 2023-03-03T08:01:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.027+0000 7f6449bff700 1 -- 172.21.15.98:0/1095373182 shutdown_connections 2023-03-03T08:01:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.027+0000 7f6449bff700 1 --2- 172.21.15.98:0/1095373182 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64440b5550 0x7f64441604c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.027+0000 7f6449bff700 1 --2- 172.21.15.98:0/1095373182 >> v2:172.21.15.100:3300/0 conn(0x7f64440b8880 0x7f64440cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.027+0000 7f6449bff700 1 --2- 172.21.15.98:0/1095373182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64440b4ba0 0x7f64440b5010 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.027+0000 7f6449bff700 1 -- 172.21.15.98:0/1095373182 >> 172.21.15.98:0/1095373182 conn(0x7f644406ad80 msgr2=0x7f644406b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.027+0000 7f6449bff700 1 -- 172.21.15.98:0/1095373182 shutdown_connections 2023-03-03T08:01:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.027+0000 7f6449bff700 1 -- 172.21.15.98:0/1095373182 wait complete. 2023-03-03T08:01:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.029+0000 7f6449bff700 1 Processor -- start 2023-03-03T08:01:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.029+0000 7f6449bff700 1 -- start start 2023-03-03T08:01:09.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.030+0000 7f6449bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64440b4ba0 0x7f6444150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:09.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.030+0000 7f6449bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f64440b5550 0x7f644414c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:09.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.031+0000 7f6449bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64440b8880 0x7f6444152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:09.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.031+0000 7f6449bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644414cd30 con 0x7f64440b8880 2023-03-03T08:01:09.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.031+0000 7f6449bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644414cfd0 con 0x7f64440b5550 2023-03-03T08:01:09.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.031+0000 7f6449bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644414d2b0 con 0x7f64440b4ba0 2023-03-03T08:01:09.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.031+0000 7f6443fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64440b8880 0x7f6444152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:09.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.031+0000 7f6443fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64440b8880 0x7f6444152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32956/0 (socket says 172.21.15.98:32956) 2023-03-03T08:01:09.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.031+0000 7f6443fff700 1 -- 172.21.15.98:0/614265326 learned_addr learned my addr 172.21.15.98:0/614265326 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:09.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.031+0000 7f643bfff700 1 --2- 172.21.15.98:0/614265326 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f64440b5550 0x7f644414c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.031+0000 7f64437fe700 1 --2- 172.21.15.98:0/614265326 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64440b4ba0 0x7f6444150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.031+0000 7f6443fff700 1 -- 172.21.15.98:0/614265326 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64440b4ba0 msgr2=0x7f6444150180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.031+0000 7f6443fff700 1 --2- 172.21.15.98:0/614265326 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64440b4ba0 0x7f6444150180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.031+0000 7f6443fff700 1 -- 172.21.15.98:0/614265326 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f64440b5550 msgr2=0x7f644414c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.031+0000 7f6443fff700 1 --2- 172.21.15.98:0/614265326 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f64440b5550 0x7f644414c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.031+0000 7f6443fff700 1 -- 172.21.15.98:0/614265326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c009550 con 0x7f64440b8880 2023-03-03T08:01:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.031+0000 7f64437fe700 1 --2- 172.21.15.98:0/614265326 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64440b4ba0 0x7f6444150180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.032+0000 7f6443fff700 1 --2- 172.21.15.98:0/614265326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64440b8880 0x7f6444152900 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f643c007e70 tx=0x7f643c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:09.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.032+0000 7f6440ff9700 1 -- 172.21.15.98:0/614265326 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c00ddd0 con 0x7f64440b8880 2023-03-03T08:01:09.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.032+0000 7f6440ff9700 1 -- 172.21.15.98:0/614265326 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c00f040 con 0x7f64440b8880 2023-03-03T08:01:09.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.032+0000 7f6449bff700 1 -- 172.21.15.98:0/614265326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64440d0cf0 con 0x7f64440b8880 2023-03-03T08:01:09.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.032+0000 7f6440ff9700 1 -- 172.21.15.98:0/614265326 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c003c60 con 0x7f64440b8880 2023-03-03T08:01:09.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.032+0000 7f6449bff700 1 -- 172.21.15.98:0/614265326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64440d0530 con 0x7f64440b8880 2023-03-03T08:01:09.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.033+0000 7f6449bff700 1 -- 172.21.15.98:0/614265326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64440b8c50 con 0x7f64440b8880 2023-03-03T08:01:09.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.035+0000 7f6440ff9700 1 -- 172.21.15.98:0/614265326 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f643c01c030 con 0x7f64440b8880 2023-03-03T08:01:09.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.036+0000 7f6440ff9700 1 --2- 172.21.15.98:0/614265326 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f64240684a0 0x7f642406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:09.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.036+0000 7f6440ff9700 1 -- 172.21.15.98:0/614265326 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5710+0+0 (secure 0 0 0) 0x7f643c085d00 con 0x7f64440b8880 2023-03-03T08:01:09.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.036+0000 7f64437fe700 1 --2- 172.21.15.98:0/614265326 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f64240684a0 0x7f642406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:09.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.037+0000 7f64437fe700 1 --2- 172.21.15.98:0/614265326 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f64240684a0 0x7f642406a950 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6434009bb0 tx=0x7f6434008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:09.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.040+0000 7f6440ff9700 1 -- 172.21.15.98:0/614265326 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f643c052520 con 0x7f64440b8880 2023-03-03T08:01:09.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.235+0000 7f6449bff700 1 -- 172.21.15.98:0/614265326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f64440ce4f0 con 0x7f64440b8880 2023-03-03T08:01:09.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.743+0000 7f6440ff9700 1 -- 172.21.15.98:0/614265326 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v69) v1 ==== 169+0+0 (secure 0 0 0) 0x7f643c01a0d0 con 0x7f64440b8880 2023-03-03T08:01:09.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.747+0000 7f6449bff700 1 -- 172.21.15.98:0/614265326 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f64240684a0 msgr2=0x7f642406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:09.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.747+0000 7f6449bff700 1 --2- 172.21.15.98:0/614265326 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f64240684a0 0x7f642406a950 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6434009bb0 tx=0x7f6434008040 comp rx=0 tx=0).stop 2023-03-03T08:01:09.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.747+0000 7f6449bff700 1 -- 172.21.15.98:0/614265326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64440b8880 msgr2=0x7f6444152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:09.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.747+0000 7f6449bff700 1 --2- 172.21.15.98:0/614265326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64440b8880 0x7f6444152900 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f643c007e70 tx=0x7f643c00d3b0 comp rx=0 tx=0).stop 2023-03-03T08:01:09.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.748+0000 7f6449bff700 1 -- 172.21.15.98:0/614265326 shutdown_connections 2023-03-03T08:01:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.748+0000 7f6449bff700 1 --2- 172.21.15.98:0/614265326 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f64240684a0 0x7f642406a950 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.748+0000 7f6449bff700 1 --2- 172.21.15.98:0/614265326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64440b8880 0x7f6444152900 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.748+0000 7f6449bff700 1 --2- 172.21.15.98:0/614265326 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64440b4ba0 0x7f6444150180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.748+0000 7f6449bff700 1 --2- 172.21.15.98:0/614265326 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f64440b5550 0x7f644414c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.748+0000 7f6449bff700 1 -- 172.21.15.98:0/614265326 >> 172.21.15.98:0/614265326 conn(0x7f644406ad80 msgr2=0x7f6444160dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.749+0000 7f6449bff700 1 -- 172.21.15.98:0/614265326 shutdown_connections 2023-03-03T08:01:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:09.749+0000 7f6449bff700 1 -- 172.21.15.98:0/614265326 wait complete. 2023-03-03T08:01:09.752 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' removed 2023-03-03T08:01:09.779 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-03-03T08:01:10.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.034+0000 7efd0dbff700 1 Processor -- start 2023-03-03T08:01:10.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.034+0000 7efd0dbff700 1 -- start start 2023-03-03T08:01:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.035+0000 7efd0dbff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efd0815ca70 0x7efd080cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.035+0000 7efd0dbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd080d0270 0x7efd080d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:10.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.035+0000 7efd0dbff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd080cf980 con 0x7efd0806b4b0 2023-03-03T08:01:10.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.035+0000 7efd0dbff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd080cd5b0 con 0x7efd0815ca70 2023-03-03T08:01:10.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.035+0000 7efd0dbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd080d1800 con 0x7efd080d0270 2023-03-03T08:01:10.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.036+0000 7efd077fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efd0815ca70 0x7efd080cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:10.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.036+0000 7efd07fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd080d0270 0x7efd080d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:10.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.036+0000 7efd07fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd080d0270 0x7efd080d1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:49516/0 (socket says 172.21.15.98:49516) 2023-03-03T08:01:10.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.036+0000 7efd077fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efd0815ca70 0x7efd080cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:49626/0 (socket says 172.21.15.98:49626) 2023-03-03T08:01:10.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.036+0000 7efd07fff700 1 -- 172.21.15.98:0/3271176801 learned_addr learned my addr 172.21.15.98:0/3271176801 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:10.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.037+0000 7efd07fff700 1 -- 172.21.15.98:0/3271176801 >> v2:172.21.15.100:3300/0 conn(0x7efd0815ca70 msgr2=0x7efd080cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:10.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.037+0000 7efd07fff700 1 --2- 172.21.15.98:0/3271176801 >> v2:172.21.15.100:3300/0 conn(0x7efd0815ca70 0x7efd080cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:10.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.037+0000 7efd07fff700 1 -- 172.21.15.98:0/3271176801 >> v1:172.21.15.100:6789/0 conn(0x7efd0806b4b0 legacy=0x7efd080cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:10.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.037+0000 7efd07fff700 1 -- 172.21.15.98:0/3271176801 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd080cd900 con 0x7efd080d0270 2023-03-03T08:01:10.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.037+0000 7efd07fff700 1 --2- 172.21.15.98:0/3271176801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd080d0270 0x7efd080d1fc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efd00009e50 tx=0x7efd0000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1990df080601ec30 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:10.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.037+0000 7efd067fc700 1 -- 172.21.15.98:0/3271176801 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0000e070 con 0x7efd080d0270 2023-03-03T08:01:10.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.037+0000 7efd067fc700 1 -- 172.21.15.98:0/3271176801 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd0000b910 con 0x7efd080d0270 2023-03-03T08:01:10.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.037+0000 7efd067fc700 1 -- 172.21.15.98:0/3271176801 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0000bcc0 con 0x7efd080d0270 2023-03-03T08:01:10.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.038+0000 7efd0dbff700 1 -- 172.21.15.98:0/3271176801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd080d0270 msgr2=0x7efd080d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:10.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.038+0000 7efd0dbff700 1 --2- 172.21.15.98:0/3271176801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd080d0270 0x7efd080d1fc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efd00009e50 tx=0x7efd0000b330 comp rx=0 tx=0).stop 2023-03-03T08:01:10.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.038+0000 7efd0dbff700 1 -- 172.21.15.98:0/3271176801 shutdown_connections 2023-03-03T08:01:10.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.038+0000 7efd0dbff700 1 --2- 172.21.15.98:0/3271176801 >> v2:172.21.15.100:3300/0 conn(0x7efd0815ca70 0x7efd080cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:10.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.038+0000 7efd0dbff700 1 --2- 172.21.15.98:0/3271176801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd080d0270 0x7efd080d1fc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:10.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.038+0000 7efd0dbff700 1 -- 172.21.15.98:0/3271176801 >> 172.21.15.98:0/3271176801 conn(0x7efd08073e70 msgr2=0x7efd0806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:10.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.038+0000 7efd0dbff700 1 -- 172.21.15.98:0/3271176801 shutdown_connections 2023-03-03T08:01:10.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.038+0000 7efd0dbff700 1 -- 172.21.15.98:0/3271176801 wait complete. 2023-03-03T08:01:10.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.040+0000 7efd0dbff700 1 Processor -- start 2023-03-03T08:01:10.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.040+0000 7efd0dbff700 1 -- start start 2023-03-03T08:01:10.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd0dbff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd0806b4b0 0x7efd081f64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:10.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd0dbff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd080d0270 0x7efd0815f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:10.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd0dbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd0815ca70 0x7efd080c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:10.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd0dbff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0815dd10 con 0x7efd0806b4b0 2023-03-03T08:01:10.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd0dbff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0815fe40 con 0x7efd080d0270 2023-03-03T08:01:10.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd0dbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd0815d100 con 0x7efd0815ca70 2023-03-03T08:01:10.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd077fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd0806b4b0 0x7efd081f64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:10.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd077fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd0806b4b0 0x7efd081f64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32958/0 (socket says 172.21.15.98:32958) 2023-03-03T08:01:10.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd077fe700 1 -- 172.21.15.98:0/3027322999 learned_addr learned my addr 172.21.15.98:0/3027322999 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:10.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd07fff700 1 --2- 172.21.15.98:0/3027322999 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd0815ca70 0x7efd080c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:10.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd06ffd700 1 --2- 172.21.15.98:0/3027322999 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd080d0270 0x7efd0815f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:10.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd077fe700 1 -- 172.21.15.98:0/3027322999 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd0815ca70 msgr2=0x7efd080c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:10.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd077fe700 1 --2- 172.21.15.98:0/3027322999 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd0815ca70 0x7efd080c8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:10.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd077fe700 1 -- 172.21.15.98:0/3027322999 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd080d0270 msgr2=0x7efd0815f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:10.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd077fe700 1 --2- 172.21.15.98:0/3027322999 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd080d0270 0x7efd0815f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:10.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.042+0000 7efd077fe700 1 -- 172.21.15.98:0/3027322999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd00009b30 con 0x7efd0806b4b0 2023-03-03T08:01:10.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.043+0000 7efd077fe700 1 --2- 172.21.15.98:0/3027322999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd0806b4b0 0x7efd081f64f0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7efcf800e9b0 tx=0x7efcf800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:10.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.043+0000 7efceffff700 1 -- 172.21.15.98:0/3027322999 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf8009710 con 0x7efd0806b4b0 2023-03-03T08:01:10.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.043+0000 7efceffff700 1 -- 172.21.15.98:0/3027322999 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf800ee50 con 0x7efd0806b4b0 2023-03-03T08:01:10.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.043+0000 7efceffff700 1 -- 172.21.15.98:0/3027322999 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf800ca50 con 0x7efd0806b4b0 2023-03-03T08:01:10.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.043+0000 7efd0dbff700 1 -- 172.21.15.98:0/3027322999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd080d1aa0 con 0x7efd0806b4b0 2023-03-03T08:01:10.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.043+0000 7efd0dbff700 1 -- 172.21.15.98:0/3027322999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd080cdbe0 con 0x7efd0806b4b0 2023-03-03T08:01:10.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.044+0000 7efd0dbff700 1 -- 172.21.15.98:0/3027322999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd0815ce40 con 0x7efd0806b4b0 2023-03-03T08:01:10.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.046+0000 7efceffff700 1 -- 172.21.15.98:0/3027322999 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7efcf8019030 con 0x7efd0806b4b0 2023-03-03T08:01:10.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.047+0000 7efceffff700 1 --2- 172.21.15.98:0/3027322999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efcf40684a0 0x7efcf406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:10.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.047+0000 7efceffff700 1 -- 172.21.15.98:0/3027322999 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5296+0+0 (secure 0 0 0) 0x7efcf8085b00 con 0x7efd0806b4b0 2023-03-03T08:01:10.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.047+0000 7efd06ffd700 1 --2- 172.21.15.98:0/3027322999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efcf40684a0 0x7efcf406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:10.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.048+0000 7efd06ffd700 1 --2- 172.21.15.98:0/3027322999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efcf40684a0 0x7efcf406a950 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7efd080666a0 tx=0x7efcfc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:10.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.051+0000 7efceffff700 1 -- 172.21.15.98:0/3027322999 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efcf8087e60 con 0x7efd0806b4b0 2023-03-03T08:01:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.253+0000 7efd0dbff700 1 -- 172.21.15.98:0/3027322999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7efd0815dff0 con 0x7efd0806b4b0 2023-03-03T08:01:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.257+0000 7efceffff700 1 -- 172.21.15.98:0/3027322999 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7efcf8055f10 con 0x7efd0806b4b0 2023-03-03T08:01:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.260+0000 7efd0dbff700 1 -- 172.21.15.98:0/3027322999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efcf40684a0 msgr2=0x7efcf406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.260+0000 7efd0dbff700 1 --2- 172.21.15.98:0/3027322999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efcf40684a0 0x7efcf406a950 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7efd080666a0 tx=0x7efcfc009040 comp rx=0 tx=0).stop 2023-03-03T08:01:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.260+0000 7efd0dbff700 1 -- 172.21.15.98:0/3027322999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd0806b4b0 msgr2=0x7efd081f64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.260+0000 7efd0dbff700 1 --2- 172.21.15.98:0/3027322999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd0806b4b0 0x7efd081f64f0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7efcf800e9b0 tx=0x7efcf800ecc0 comp rx=0 tx=0).stop 2023-03-03T08:01:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.260+0000 7efd0dbff700 1 -- 172.21.15.98:0/3027322999 shutdown_connections 2023-03-03T08:01:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.260+0000 7efd0dbff700 1 --2- 172.21.15.98:0/3027322999 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efcf40684a0 0x7efcf406a950 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.260+0000 7efd0dbff700 1 --2- 172.21.15.98:0/3027322999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd0806b4b0 0x7efd081f64f0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.260+0000 7efd0dbff700 1 --2- 172.21.15.98:0/3027322999 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd0815ca70 0x7efd080c8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.260+0000 7efd0dbff700 1 --2- 172.21.15.98:0/3027322999 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd080d0270 0x7efd0815f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:10.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.260+0000 7efd0dbff700 1 -- 172.21.15.98:0/3027322999 >> 172.21.15.98:0/3027322999 conn(0x7efd08073e70 msgr2=0x7efd080b6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:10.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.260+0000 7efd0dbff700 1 -- 172.21.15.98:0/3027322999 shutdown_connections 2023-03-03T08:01:10.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.260+0000 7efd0dbff700 1 -- 172.21.15.98:0/3027322999 wait complete. 2023-03-03T08:01:10.289 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade' 2023-03-03T08:01:10.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.533+0000 7f1f57fff700 1 Processor -- start 2023-03-03T08:01:10.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.533+0000 7f1f57fff700 1 -- start start 2023-03-03T08:01:10.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.535+0000 7f1f57fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1f500b8b90 0x7f1f500cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:10.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.535+0000 7f1f57fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f500b5410 0x7f1f500d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:10.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.535+0000 7f1f57fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f500d0ef0 con 0x7f1f500b87c0 2023-03-03T08:01:10.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.535+0000 7f1f57fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f500d0600 con 0x7f1f500b8b90 2023-03-03T08:01:10.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.535+0000 7f1f57fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f500cd970 con 0x7f1f500b5410 2023-03-03T08:01:10.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.535+0000 7f1f55d9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1f500b87c0 0x7f1f500b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54632/0 (socket says 172.21.15.98:54632) 2023-03-03T08:01:10.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.535+0000 7f1f55d9b700 1 -- 172.21.15.98:0/1180936601 learned_addr learned my addr 172.21.15.98:0/1180936601 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:10.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.535+0000 7f1f5659c700 1 --2- 172.21.15.98:0/1180936601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f500b5410 0x7f1f500d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:10.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.535+0000 7f1f5559a700 1 --2- 172.21.15.98:0/1180936601 >> v2:172.21.15.100:3300/0 conn(0x7f1f500b8b90 0x7f1f500cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:10.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.535+0000 7f1f54d99700 1 -- 172.21.15.98:0/1180936601 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335606306 0 0) 0x7f1f500d0ef0 con 0x7f1f500b87c0 2023-03-03T08:01:10.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.536+0000 7f1f54d99700 1 -- 172.21.15.98:0/1180936601 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f3c0035b0 con 0x7f1f500b87c0 2023-03-03T08:01:10.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.536+0000 7f1f5659c700 1 -- 172.21.15.98:0/1180936601 >> v2:172.21.15.100:3300/0 conn(0x7f1f500b8b90 msgr2=0x7f1f500cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:10.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.536+0000 7f1f5659c700 1 --2- 172.21.15.98:0/1180936601 >> v2:172.21.15.100:3300/0 conn(0x7f1f500b8b90 0x7f1f500cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:10.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.536+0000 7f1f5659c700 1 -- 172.21.15.98:0/1180936601 >> v1:172.21.15.98:6789/0 conn(0x7f1f500b87c0 legacy=0x7f1f500b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:10.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.536+0000 7f1f5659c700 1 -- 172.21.15.98:0/1180936601 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f500cdc50 con 0x7f1f500b5410 2023-03-03T08:01:10.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.536+0000 7f1f5659c700 1 --2- 172.21.15.98:0/1180936601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f500b5410 0x7f1f500d13d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1f48009d90 tx=0x7f1f4800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2637584f5e5cb9e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:10.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.536+0000 7f1f54d99700 1 -- 172.21.15.98:0/1180936601 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4800e070 con 0x7f1f500b5410 2023-03-03T08:01:10.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.536+0000 7f1f54d99700 1 -- 172.21.15.98:0/1180936601 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4800bb20 con 0x7f1f500b5410 2023-03-03T08:01:10.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.536+0000 7f1f54d99700 1 -- 172.21.15.98:0/1180936601 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f48012430 con 0x7f1f500b5410 2023-03-03T08:01:10.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.537+0000 7f1f57fff700 1 -- 172.21.15.98:0/1180936601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f500b5410 msgr2=0x7f1f500d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:10.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.537+0000 7f1f57fff700 1 --2- 172.21.15.98:0/1180936601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f500b5410 0x7f1f500d13d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1f48009d90 tx=0x7f1f4800b330 comp rx=0 tx=0).stop 2023-03-03T08:01:10.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.537+0000 7f1f57fff700 1 -- 172.21.15.98:0/1180936601 shutdown_connections 2023-03-03T08:01:10.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.537+0000 7f1f57fff700 1 --2- 172.21.15.98:0/1180936601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f500b5410 0x7f1f500d13d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:10.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.537+0000 7f1f57fff700 1 --2- 172.21.15.98:0/1180936601 >> v2:172.21.15.100:3300/0 conn(0x7f1f500b8b90 0x7f1f500cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:10.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.537+0000 7f1f57fff700 1 -- 172.21.15.98:0/1180936601 >> 172.21.15.98:0/1180936601 conn(0x7f1f5006aee0 msgr2=0x7f1f5006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:10.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.537+0000 7f1f57fff700 1 -- 172.21.15.98:0/1180936601 shutdown_connections 2023-03-03T08:01:10.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.537+0000 7f1f57fff700 1 -- 172.21.15.98:0/1180936601 wait complete. 2023-03-03T08:01:10.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.539+0000 7f1f57fff700 1 Processor -- start 2023-03-03T08:01:10.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.540+0000 7f1f57fff700 1 -- start start 2023-03-03T08:01:10.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f57fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f500b5410 0x7f1f5014f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:10.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f57fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f500b87c0 0x7f1f500a9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:10.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f57fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f500b8b90 0x7f1f500aa1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:10.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f57fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f500c8af0 con 0x7f1f500b87c0 2023-03-03T08:01:10.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f57fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f500c7c20 con 0x7f1f500b5410 2023-03-03T08:01:10.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f57fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f500c7e70 con 0x7f1f500b8b90 2023-03-03T08:01:10.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f5559a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f500b87c0 0x7f1f500a9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:10.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f5559a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f500b87c0 0x7f1f500a9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32972/0 (socket says 172.21.15.98:32972) 2023-03-03T08:01:10.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f5559a700 1 -- 172.21.15.98:0/718245618 learned_addr learned my addr 172.21.15.98:0/718245618 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:10.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f55d9b700 1 --2- 172.21.15.98:0/718245618 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f500b5410 0x7f1f5014f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:10.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f5659c700 1 --2- 172.21.15.98:0/718245618 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f500b8b90 0x7f1f500aa1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:10.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f5559a700 1 -- 172.21.15.98:0/718245618 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f500b8b90 msgr2=0x7f1f500aa1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:10.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f5559a700 1 --2- 172.21.15.98:0/718245618 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f500b8b90 0x7f1f500aa1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:10.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f5559a700 1 -- 172.21.15.98:0/718245618 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f500b5410 msgr2=0x7f1f5014f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:10.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f5559a700 1 --2- 172.21.15.98:0/718245618 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f500b5410 0x7f1f5014f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:10.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.541+0000 7f1f5559a700 1 -- 172.21.15.98:0/718245618 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f48009a70 con 0x7f1f500b87c0 2023-03-03T08:01:10.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.542+0000 7f1f5559a700 1 --2- 172.21.15.98:0/718245618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f500b87c0 0x7f1f500a9c70 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000e840 tx=0x7f1f4000eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:10.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.542+0000 7f1f467fc700 1 -- 172.21.15.98:0/718245618 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40009590 con 0x7f1f500b87c0 2023-03-03T08:01:10.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.542+0000 7f1f467fc700 1 -- 172.21.15.98:0/718245618 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f40013680 con 0x7f1f500b87c0 2023-03-03T08:01:10.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.542+0000 7f1f467fc700 1 -- 172.21.15.98:0/718245618 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40013a30 con 0x7f1f500b87c0 2023-03-03T08:01:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.542+0000 7f1f57fff700 1 -- 172.21.15.98:0/718245618 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f500cdf30 con 0x7f1f500b87c0 2023-03-03T08:01:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.542+0000 7f1f57fff700 1 -- 172.21.15.98:0/718245618 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f50073a80 con 0x7f1f500b87c0 2023-03-03T08:01:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.543+0000 7f1f57fff700 1 -- 172.21.15.98:0/718245618 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f500ce0c0 con 0x7f1f500b87c0 2023-03-03T08:01:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.544+0000 7f1f467fc700 1 -- 172.21.15.98:0/718245618 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f1f40019030 con 0x7f1f500b87c0 2023-03-03T08:01:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.546+0000 7f1f467fc700 1 --2- 172.21.15.98:0/718245618 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1f340684a0 0x7f1f3406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.546+0000 7f1f467fc700 1 -- 172.21.15.98:0/718245618 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f1f400858b0 con 0x7f1f500b87c0 2023-03-03T08:01:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.546+0000 7f1f55d9b700 1 --2- 172.21.15.98:0/718245618 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1f340684a0 0x7f1f3406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.547+0000 7f1f55d9b700 1 --2- 172.21.15.98:0/718245618 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1f340684a0 0x7f1f3406a950 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c007fd0 tx=0x7f1f4c007dc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:10.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.549+0000 7f1f467fc700 1 -- 172.21.15.98:0/718245618 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f40055cc0 con 0x7f1f500b87c0 2023-03-03T08:01:10.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:10.819+0000 7f1f57fff700 1 -- 172.21.15.98:0/718245618 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]} v 0) v1 -- 0x7f1f500729b0 con 0x7f1f500b87c0 2023-03-03T08:01:11.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:11.757+0000 7f1f467fc700 1 -- 172.21.15.98:0/718245618 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]}]=0 v299) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1f400558b0 con 0x7f1f500b87c0 2023-03-03T08:01:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:11.761+0000 7f1f57fff700 1 -- 172.21.15.98:0/718245618 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1f340684a0 msgr2=0x7f1f3406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:11.761+0000 7f1f57fff700 1 --2- 172.21.15.98:0/718245618 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1f340684a0 0x7f1f3406a950 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c007fd0 tx=0x7f1f4c007dc0 comp rx=0 tx=0).stop 2023-03-03T08:01:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:11.761+0000 7f1f57fff700 1 -- 172.21.15.98:0/718245618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f500b87c0 msgr2=0x7f1f500a9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:11.761+0000 7f1f57fff700 1 --2- 172.21.15.98:0/718245618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f500b87c0 0x7f1f500a9c70 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000e840 tx=0x7f1f4000eb50 comp rx=0 tx=0).stop 2023-03-03T08:01:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:11.761+0000 7f1f57fff700 1 -- 172.21.15.98:0/718245618 shutdown_connections 2023-03-03T08:01:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:11.761+0000 7f1f57fff700 1 --2- 172.21.15.98:0/718245618 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1f340684a0 0x7f1f3406a950 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:11.761+0000 7f1f57fff700 1 --2- 172.21.15.98:0/718245618 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f500b8b90 0x7f1f500aa1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:11.761+0000 7f1f57fff700 1 --2- 172.21.15.98:0/718245618 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f500b5410 0x7f1f5014f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:11.761+0000 7f1f57fff700 1 --2- 172.21.15.98:0/718245618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f500b87c0 0x7f1f500a9c70 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:11.761+0000 7f1f57fff700 1 -- 172.21.15.98:0/718245618 >> 172.21.15.98:0/718245618 conn(0x7f1f5006aee0 msgr2=0x7f1f500b3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:11.761+0000 7f1f57fff700 1 -- 172.21.15.98:0/718245618 shutdown_connections 2023-03-03T08:01:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:11.761+0000 7f1f57fff700 1 -- 172.21.15.98:0/718245618 wait complete. 2023-03-03T08:01:11.792 INFO:tasks.cephfs_test_runner:test_malicious_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) ... ok 2023-03-03T08:01:11.793 INFO:tasks.cephfs_test_runner:Starting test: test_mgr_eviction (tasks.cephfs.test_volumes.TestMisc) 2023-03-03T08:01:11.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction' 2023-03-03T08:01:12.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.039+0000 7f735559d700 1 Processor -- start 2023-03-03T08:01:12.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.040+0000 7f735559d700 1 -- start start 2023-03-03T08:01:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.041+0000 7f735559d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f73500d0010 0x7f73500d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.041+0000 7f735559d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f73500b4a00 0x7f73500d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.041+0000 7f735559d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73500d1480 0x7f73500b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.041+0000 7f735559d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73500d18e0 con 0x7f73500b4a00 2023-03-03T08:01:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.041+0000 7f735559d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73500b0c50 con 0x7f73500d0010 2023-03-03T08:01:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.041+0000 7f735559d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73500b0040 con 0x7f73500d1480 2023-03-03T08:01:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.041+0000 7f734f7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f73500b4a00 0x7f73500d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.041+0000 7f734f7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f73500b4a00 0x7f73500d0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32978/0 (socket says 172.21.15.98:32978) 2023-03-03T08:01:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.041+0000 7f734f7fe700 1 -- 172.21.15.98:0/3185307512 learned_addr learned my addr 172.21.15.98:0/3185307512 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.042+0000 7f7354d9c700 1 --2- 172.21.15.98:0/3185307512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73500d1480 0x7f73500b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.042+0000 7f734ffff700 1 --2- 172.21.15.98:0/3185307512 >> v2:172.21.15.100:3300/0 conn(0x7f73500d0010 0x7f73500d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:12.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.042+0000 7f734f7fe700 1 -- 172.21.15.98:0/3185307512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73500d1480 msgr2=0x7f73500b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:12.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.042+0000 7f734f7fe700 1 --2- 172.21.15.98:0/3185307512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73500d1480 0x7f73500b10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:12.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.042+0000 7f734f7fe700 1 -- 172.21.15.98:0/3185307512 >> v2:172.21.15.100:3300/0 conn(0x7f73500d0010 msgr2=0x7f73500d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:12.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.042+0000 7f734f7fe700 1 --2- 172.21.15.98:0/3185307512 >> v2:172.21.15.100:3300/0 conn(0x7f73500d0010 0x7f73500d03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:12.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.042+0000 7f734f7fe700 1 -- 172.21.15.98:0/3185307512 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73500b0320 con 0x7f73500b4a00 2023-03-03T08:01:12.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.042+0000 7f7354d9c700 1 --2- 172.21.15.98:0/3185307512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73500d1480 0x7f73500b10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:12.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.042+0000 7f734f7fe700 1 --2- 172.21.15.98:0/3185307512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73500b4a00 0x7f73500d0f40 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f735008c540 tx=0x7f7344009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=935eb37a0f39b1f9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:12.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.043+0000 7f734effd700 1 -- 172.21.15.98:0/3185307512 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734400b870 con 0x7f73500b4a00 2023-03-03T08:01:12.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.043+0000 7f734effd700 1 -- 172.21.15.98:0/3185307512 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734400b9d0 con 0x7f73500b4a00 2023-03-03T08:01:12.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.043+0000 7f734effd700 1 -- 172.21.15.98:0/3185307512 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734400bd60 con 0x7f73500b4a00 2023-03-03T08:01:12.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.043+0000 7f735559d700 1 -- 172.21.15.98:0/3185307512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73500b4a00 msgr2=0x7f73500d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:12.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.043+0000 7f735559d700 1 --2- 172.21.15.98:0/3185307512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73500b4a00 0x7f73500d0f40 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f735008c540 tx=0x7f7344009bd0 comp rx=0 tx=0).stop 2023-03-03T08:01:12.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.043+0000 7f735559d700 1 -- 172.21.15.98:0/3185307512 shutdown_connections 2023-03-03T08:01:12.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.043+0000 7f735559d700 1 --2- 172.21.15.98:0/3185307512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73500d1480 0x7f73500b10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:12.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.043+0000 7f735559d700 1 --2- 172.21.15.98:0/3185307512 >> v2:172.21.15.100:3300/0 conn(0x7f73500d0010 0x7f73500d03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:12.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.043+0000 7f735559d700 1 --2- 172.21.15.98:0/3185307512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73500b4a00 0x7f73500d0f40 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:12.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.043+0000 7f735559d700 1 -- 172.21.15.98:0/3185307512 >> 172.21.15.98:0/3185307512 conn(0x7f7350073b00 msgr2=0x7f73500d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:12.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.043+0000 7f735559d700 1 -- 172.21.15.98:0/3185307512 shutdown_connections 2023-03-03T08:01:12.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.043+0000 7f735559d700 1 -- 172.21.15.98:0/3185307512 wait complete. 2023-03-03T08:01:12.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.046+0000 7f735559d700 1 Processor -- start 2023-03-03T08:01:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.046+0000 7f735559d700 1 -- start start 2023-03-03T08:01:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.048+0000 7f735559d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73500b4a00 0x7f7350155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.048+0000 7f735559d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73500d0010 0x7f73501e6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.048+0000 7f735559d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73500d1480 0x7f7350156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.048+0000 7f735559d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73501e4fb0 con 0x7f73500b4a00 2023-03-03T08:01:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.048+0000 7f735559d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73501e4330 con 0x7f73500d1480 2023-03-03T08:01:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.048+0000 7f735559d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73501e4610 con 0x7f73500d0010 2023-03-03T08:01:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.048+0000 7f734ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73500b4a00 0x7f7350155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.048+0000 7f734ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73500b4a00 0x7f7350155b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32982/0 (socket says 172.21.15.98:32982) 2023-03-03T08:01:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.048+0000 7f734ffff700 1 -- 172.21.15.98:0/1300907825 learned_addr learned my addr 172.21.15.98:0/1300907825 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.048+0000 7f734f7fe700 1 --2- 172.21.15.98:0/1300907825 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73500d0010 0x7f73501e6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.048+0000 7f7354d9c700 1 --2- 172.21.15.98:0/1300907825 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73500d1480 0x7f7350156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.048+0000 7f734ffff700 1 -- 172.21.15.98:0/1300907825 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73500d0010 msgr2=0x7f73501e6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.048+0000 7f734ffff700 1 --2- 172.21.15.98:0/1300907825 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73500d0010 0x7f73501e6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.049+0000 7f734ffff700 1 -- 172.21.15.98:0/1300907825 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73500d1480 msgr2=0x7f7350156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.049+0000 7f734ffff700 1 --2- 172.21.15.98:0/1300907825 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73500d1480 0x7f7350156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.049+0000 7f734ffff700 1 -- 172.21.15.98:0/1300907825 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7344009550 con 0x7f73500b4a00 2023-03-03T08:01:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.049+0000 7f734ffff700 1 --2- 172.21.15.98:0/1300907825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73500b4a00 0x7f7350155b50 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f734000e7e0 tx=0x7f734000eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.049+0000 7f734cff9700 1 -- 172.21.15.98:0/1300907825 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7340013610 con 0x7f73500b4a00 2023-03-03T08:01:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.049+0000 7f734cff9700 1 -- 172.21.15.98:0/1300907825 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7340013770 con 0x7f73500b4a00 2023-03-03T08:01:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.049+0000 7f734cff9700 1 -- 172.21.15.98:0/1300907825 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7340013b00 con 0x7f73500b4a00 2023-03-03T08:01:12.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.049+0000 7f735559d700 1 -- 172.21.15.98:0/1300907825 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73500b08e0 con 0x7f73500b4a00 2023-03-03T08:01:12.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.050+0000 7f735559d700 1 -- 172.21.15.98:0/1300907825 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73500b0600 con 0x7f73500b4a00 2023-03-03T08:01:12.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.051+0000 7f735559d700 1 -- 172.21.15.98:0/1300907825 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73500d0b30 con 0x7f73500b4a00 2023-03-03T08:01:12.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.052+0000 7f734cff9700 1 -- 172.21.15.98:0/1300907825 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f7340019030 con 0x7f73500b4a00 2023-03-03T08:01:12.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.053+0000 7f734cff9700 1 --2- 172.21.15.98:0/1300907825 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f733c0684a0 0x7f733c06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:12.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.053+0000 7f734cff9700 1 -- 172.21.15.98:0/1300907825 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f7340085b00 con 0x7f73500b4a00 2023-03-03T08:01:12.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.053+0000 7f734f7fe700 1 --2- 172.21.15.98:0/1300907825 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f733c0684a0 0x7f733c06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:12.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.054+0000 7f734f7fe700 1 --2- 172.21.15.98:0/1300907825 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f733c0684a0 0x7f733c06a950 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f73500b4e30 tx=0x7f7344008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:12.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.057+0000 7f734cff9700 1 -- 172.21.15.98:0/1300907825 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7340052550 con 0x7f73500b4a00 2023-03-03T08:01:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.333+0000 7f735559d700 1 -- 172.21.15.98:0/1300907825 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]} v 0) v1 -- 0x7f73500729b0 con 0x7f73500b4a00 2023-03-03T08:01:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.762+0000 7f734cff9700 1 -- 172.21.15.98:0/1300907825 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]}]=0 v300) v1 ==== 134+0+0 (secure 0 0 0) 0x7f73400210c0 con 0x7f73500b4a00 2023-03-03T08:01:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.764+0000 7f735559d700 1 -- 172.21.15.98:0/1300907825 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f733c0684a0 msgr2=0x7f733c06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.764+0000 7f735559d700 1 --2- 172.21.15.98:0/1300907825 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f733c0684a0 0x7f733c06a950 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f73500b4e30 tx=0x7f7344008030 comp rx=0 tx=0).stop 2023-03-03T08:01:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.765+0000 7f735559d700 1 -- 172.21.15.98:0/1300907825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73500b4a00 msgr2=0x7f7350155b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.765+0000 7f735559d700 1 --2- 172.21.15.98:0/1300907825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73500b4a00 0x7f7350155b50 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f734000e7e0 tx=0x7f734000eba0 comp rx=0 tx=0).stop 2023-03-03T08:01:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.765+0000 7f735559d700 1 -- 172.21.15.98:0/1300907825 shutdown_connections 2023-03-03T08:01:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.765+0000 7f735559d700 1 --2- 172.21.15.98:0/1300907825 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f733c0684a0 0x7f733c06a950 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.765+0000 7f735559d700 1 --2- 172.21.15.98:0/1300907825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73500b4a00 0x7f7350155b50 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.765+0000 7f735559d700 1 --2- 172.21.15.98:0/1300907825 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73500d1480 0x7f7350156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.765+0000 7f735559d700 1 --2- 172.21.15.98:0/1300907825 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73500d0010 0x7f73501e6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.765+0000 7f735559d700 1 -- 172.21.15.98:0/1300907825 >> 172.21.15.98:0/1300907825 conn(0x7f7350073b00 msgr2=0x7f73500b1970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.765+0000 7f735559d700 1 -- 172.21.15.98:0/1300907825 shutdown_connections 2023-03-03T08:01:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:12.765+0000 7f735559d700 1 -- 172.21.15.98:0/1300907825 wait complete. 2023-03-03T08:01:12.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-03-03T08:01:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.040+0000 7fcc711ff700 1 Processor -- start 2023-03-03T08:01:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.041+0000 7fcc711ff700 1 -- start start 2023-03-03T08:01:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.042+0000 7fcc711ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc6c0d0270 0x7fcc6c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.042+0000 7fcc711ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcc6c06b4b0 0x7fcc6c0d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.042+0000 7fcc711ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6c0cf980 con 0x7fcc6c15ca70 2023-03-03T08:01:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.042+0000 7fcc711ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc6c0cd5b0 con 0x7fcc6c06b4b0 2023-03-03T08:01:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.042+0000 7fcc711ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c0d1800 con 0x7fcc6c0d0270 2023-03-03T08:01:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.043+0000 7fcc6b59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcc6c06b4b0 0x7fcc6c0d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.043+0000 7fcc6b59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcc6c06b4b0 0x7fcc6c0d1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32998/0 (socket says 172.21.15.98:32998) 2023-03-03T08:01:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.043+0000 7fcc6b59e700 1 -- 172.21.15.98:0/3912247317 learned_addr learned my addr 172.21.15.98:0/3912247317 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.043+0000 7fcc6a59c700 1 --2- 172.21.15.98:0/3912247317 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc6c0d0270 0x7fcc6c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.043+0000 7fcc6b59e700 1 -- 172.21.15.98:0/3912247317 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc6c0d0270 msgr2=0x7fcc6c0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.043+0000 7fcc6b59e700 1 --2- 172.21.15.98:0/3912247317 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc6c0d0270 0x7fcc6c0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.043+0000 7fcc6b59e700 1 -- 172.21.15.98:0/3912247317 >> v1:172.21.15.100:6789/0 conn(0x7fcc6c15ca70 legacy=0x7fcc6c0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.043+0000 7fcc6b59e700 1 -- 172.21.15.98:0/3912247317 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c0cd900 con 0x7fcc6c06b4b0 2023-03-03T08:01:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.043+0000 7fcc6b59e700 1 --2- 172.21.15.98:0/3912247317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c06b4b0 0x7fcc6c0d1fc0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fcc64009e50 tx=0x7fcc6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50359f6b20152ecd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.044+0000 7fcc69d9b700 1 -- 172.21.15.98:0/3912247317 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6400e070 con 0x7fcc6c06b4b0 2023-03-03T08:01:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.044+0000 7fcc69d9b700 1 -- 172.21.15.98:0/3912247317 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6400b910 con 0x7fcc6c06b4b0 2023-03-03T08:01:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.044+0000 7fcc69d9b700 1 -- 172.21.15.98:0/3912247317 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6400bcc0 con 0x7fcc6c06b4b0 2023-03-03T08:01:13.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.044+0000 7fcc711ff700 1 -- 172.21.15.98:0/3912247317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c06b4b0 msgr2=0x7fcc6c0d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:13.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.044+0000 7fcc711ff700 1 --2- 172.21.15.98:0/3912247317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c06b4b0 0x7fcc6c0d1fc0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fcc64009e50 tx=0x7fcc6400b330 comp rx=0 tx=0).stop 2023-03-03T08:01:13.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.044+0000 7fcc711ff700 1 -- 172.21.15.98:0/3912247317 shutdown_connections 2023-03-03T08:01:13.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.044+0000 7fcc711ff700 1 --2- 172.21.15.98:0/3912247317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c06b4b0 0x7fcc6c0d1fc0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:13.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.044+0000 7fcc711ff700 1 --2- 172.21.15.98:0/3912247317 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc6c0d0270 0x7fcc6c0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.044+0000 7fcc711ff700 1 -- 172.21.15.98:0/3912247317 >> 172.21.15.98:0/3912247317 conn(0x7fcc6c073e70 msgr2=0x7fcc6c06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.045+0000 7fcc711ff700 1 -- 172.21.15.98:0/3912247317 shutdown_connections 2023-03-03T08:01:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.045+0000 7fcc711ff700 1 -- 172.21.15.98:0/3912247317 wait complete. 2023-03-03T08:01:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.046+0000 7fcc711ff700 1 Processor -- start 2023-03-03T08:01:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.047+0000 7fcc711ff700 1 -- start start 2023-03-03T08:01:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.048+0000 7fcc711ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc6c06b4b0 0x7fcc6c0b7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.048+0000 7fcc711ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc6c0d0270 0x7fcc6c0ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.048+0000 7fcc711ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c15ca70 0x7fcc6c151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.048+0000 7fcc711ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c0cb5b0 con 0x7fcc6c15ca70 2023-03-03T08:01:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.048+0000 7fcc711ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c0c7eb0 con 0x7fcc6c06b4b0 2023-03-03T08:01:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.048+0000 7fcc711ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c0cad10 con 0x7fcc6c0d0270 2023-03-03T08:01:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.048+0000 7fcc6b59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c15ca70 0x7fcc6c151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.048+0000 7fcc6b59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c15ca70 0x7fcc6c151320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33004/0 (socket says 172.21.15.98:33004) 2023-03-03T08:01:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.048+0000 7fcc6b59e700 1 -- 172.21.15.98:0/336684110 learned_addr learned my addr 172.21.15.98:0/336684110 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.048+0000 7fcc6a59c700 1 --2- 172.21.15.98:0/336684110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc6c0d0270 0x7fcc6c0ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.048+0000 7fcc6ad9d700 1 --2- 172.21.15.98:0/336684110 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc6c06b4b0 0x7fcc6c0b7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.049+0000 7fcc6b59e700 1 -- 172.21.15.98:0/336684110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc6c0d0270 msgr2=0x7fcc6c0ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.049+0000 7fcc6b59e700 1 --2- 172.21.15.98:0/336684110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc6c0d0270 0x7fcc6c0ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.049+0000 7fcc6b59e700 1 -- 172.21.15.98:0/336684110 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc6c06b4b0 msgr2=0x7fcc6c0b7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.049+0000 7fcc6b59e700 1 --2- 172.21.15.98:0/336684110 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc6c06b4b0 0x7fcc6c0b7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.049+0000 7fcc6b59e700 1 -- 172.21.15.98:0/336684110 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc64009b30 con 0x7fcc6c15ca70 2023-03-03T08:01:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.049+0000 7fcc6b59e700 1 --2- 172.21.15.98:0/336684110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c15ca70 0x7fcc6c151320 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0b5d70 tx=0x7fcc64004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.049+0000 7fcc537fe700 1 -- 172.21.15.98:0/336684110 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc64004800 con 0x7fcc6c15ca70 2023-03-03T08:01:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.049+0000 7fcc537fe700 1 -- 172.21.15.98:0/336684110 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc64004960 con 0x7fcc6c15ca70 2023-03-03T08:01:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.049+0000 7fcc711ff700 1 -- 172.21.15.98:0/336684110 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc6c0d1aa0 con 0x7fcc6c15ca70 2023-03-03T08:01:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.049+0000 7fcc537fe700 1 -- 172.21.15.98:0/336684110 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6401b670 con 0x7fcc6c15ca70 2023-03-03T08:01:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.049+0000 7fcc711ff700 1 -- 172.21.15.98:0/336684110 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6c0cdbe0 con 0x7fcc6c15ca70 2023-03-03T08:01:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.051+0000 7fcc711ff700 1 -- 172.21.15.98:0/336684110 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc6c15ce40 con 0x7fcc6c15ca70 2023-03-03T08:01:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.052+0000 7fcc537fe700 1 -- 172.21.15.98:0/336684110 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fcc6402b070 con 0x7fcc6c15ca70 2023-03-03T08:01:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.054+0000 7fcc537fe700 1 --2- 172.21.15.98:0/336684110 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcc580684a0 0x7fcc5806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.054+0000 7fcc537fe700 1 -- 172.21.15.98:0/336684110 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fcc640872a0 con 0x7fcc6c15ca70 2023-03-03T08:01:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.054+0000 7fcc6ad9d700 1 --2- 172.21.15.98:0/336684110 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcc580684a0 0x7fcc5806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.055+0000 7fcc6ad9d700 1 --2- 172.21.15.98:0/336684110 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcc580684a0 0x7fcc5806a950 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c007fd0 tx=0x7fcc5c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:13.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.058+0000 7fcc537fe700 1 -- 172.21.15.98:0/336684110 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcc64053c80 con 0x7fcc6c15ca70 2023-03-03T08:01:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.249+0000 7fcc711ff700 1 -- 172.21.15.98:0/336684110 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcc6c04e6e0 con 0x7fcc6c15ca70 2023-03-03T08:01:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.254+0000 7fcc537fe700 1 -- 172.21.15.98:0/336684110 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcc640572a0 con 0x7fcc6c15ca70 2023-03-03T08:01:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.256+0000 7fcc711ff700 1 -- 172.21.15.98:0/336684110 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcc580684a0 msgr2=0x7fcc5806a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.256+0000 7fcc711ff700 1 --2- 172.21.15.98:0/336684110 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcc580684a0 0x7fcc5806a950 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c007fd0 tx=0x7fcc5c009040 comp rx=0 tx=0).stop 2023-03-03T08:01:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.256+0000 7fcc711ff700 1 -- 172.21.15.98:0/336684110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c15ca70 msgr2=0x7fcc6c151320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.256+0000 7fcc711ff700 1 --2- 172.21.15.98:0/336684110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c15ca70 0x7fcc6c151320 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0b5d70 tx=0x7fcc64004ad0 comp rx=0 tx=0).stop 2023-03-03T08:01:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.256+0000 7fcc711ff700 1 -- 172.21.15.98:0/336684110 shutdown_connections 2023-03-03T08:01:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.256+0000 7fcc711ff700 1 --2- 172.21.15.98:0/336684110 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fcc580684a0 0x7fcc5806a950 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.256+0000 7fcc711ff700 1 --2- 172.21.15.98:0/336684110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c15ca70 0x7fcc6c151320 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.256+0000 7fcc711ff700 1 --2- 172.21.15.98:0/336684110 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc6c06b4b0 0x7fcc6c0b7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.256+0000 7fcc711ff700 1 --2- 172.21.15.98:0/336684110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc6c0d0270 0x7fcc6c0ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.256+0000 7fcc711ff700 1 -- 172.21.15.98:0/336684110 >> 172.21.15.98:0/336684110 conn(0x7fcc6c073e70 msgr2=0x7fcc6c0b5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.257+0000 7fcc711ff700 1 -- 172.21.15.98:0/336684110 shutdown_connections 2023-03-03T08:01:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:13.257+0000 7fcc711ff700 1 -- 172.21.15.98:0/336684110 wait complete. 2023-03-03T08:01:13.285 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-03-03T08:01:13.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:01:13.418 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:01:13.565 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-03-03T08:01:13.565 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:01:13.566 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:01:13.633 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T08:01:13.634 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:01:13.634 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:01:13.703 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T08:01:13.703 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:01:13.704 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:01:13.772 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T08:01:13.773 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:01:13.773 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:01:13.840 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T08:01:13.841 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-03-03T08:01:13.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:01:14.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.085+0000 7f803dbff700 1 Processor -- start 2023-03-03T08:01:14.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.085+0000 7f803dbff700 1 -- start start 2023-03-03T08:01:14.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.086+0000 7f803dbff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f80380b87c0 0x7f80380cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:14.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.087+0000 7f803dbff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80380b5410 0x7f80380d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:14.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.087+0000 7f803dbff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80380d0ef0 con 0x7f80380b8b90 2023-03-03T08:01:14.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.087+0000 7f803dbff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80380d0600 con 0x7f80380b5410 2023-03-03T08:01:14.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.087+0000 7f803dbff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80380cd970 con 0x7f80380b87c0 2023-03-03T08:01:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.087+0000 7f8037fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80380b5410 0x7f80380d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.087+0000 7f8037fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80380b5410 0x7f80380d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33006/0 (socket says 172.21.15.98:33006) 2023-03-03T08:01:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.087+0000 7f8037fff700 1 -- 172.21.15.98:0/1590315771 learned_addr learned my addr 172.21.15.98:0/1590315771 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:14.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.087+0000 7f80377fe700 1 --2- 172.21.15.98:0/1590315771 >> v2:172.21.15.100:3300/0 conn(0x7f80380b87c0 0x7f80380cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:14.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.087+0000 7f8037fff700 1 -- 172.21.15.98:0/1590315771 >> v2:172.21.15.100:3300/0 conn(0x7f80380b87c0 msgr2=0x7f80380cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:14.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.087+0000 7f8037fff700 1 --2- 172.21.15.98:0/1590315771 >> v2:172.21.15.100:3300/0 conn(0x7f80380b87c0 0x7f80380cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:14.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.087+0000 7f8037fff700 1 -- 172.21.15.98:0/1590315771 >> v1:172.21.15.100:6789/0 conn(0x7f80380b8b90 legacy=0x7f80380b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:14.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.088+0000 7f8037fff700 1 -- 172.21.15.98:0/1590315771 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80380cdc50 con 0x7f80380b5410 2023-03-03T08:01:14.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.088+0000 7f8037fff700 1 --2- 172.21.15.98:0/1590315771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80380b5410 0x7f80380d13d0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f8038073840 tx=0x7f803000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c0b871c0190169 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:14.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.088+0000 7f80367fc700 1 -- 172.21.15.98:0/1590315771 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803000e070 con 0x7f80380b5410 2023-03-03T08:01:14.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.088+0000 7f80367fc700 1 -- 172.21.15.98:0/1590315771 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803000b9c0 con 0x7f80380b5410 2023-03-03T08:01:14.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.088+0000 7f80367fc700 1 -- 172.21.15.98:0/1590315771 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803000bd70 con 0x7f80380b5410 2023-03-03T08:01:14.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.089+0000 7f803dbff700 1 -- 172.21.15.98:0/1590315771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80380b5410 msgr2=0x7f80380d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.089+0000 7f803dbff700 1 --2- 172.21.15.98:0/1590315771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80380b5410 0x7f80380d13d0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f8038073840 tx=0x7f803000b330 comp rx=0 tx=0).stop 2023-03-03T08:01:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.089+0000 7f803dbff700 1 -- 172.21.15.98:0/1590315771 shutdown_connections 2023-03-03T08:01:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.089+0000 7f803dbff700 1 --2- 172.21.15.98:0/1590315771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80380b5410 0x7f80380d13d0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.089+0000 7f803dbff700 1 --2- 172.21.15.98:0/1590315771 >> v2:172.21.15.100:3300/0 conn(0x7f80380b87c0 0x7f80380cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.089+0000 7f803dbff700 1 -- 172.21.15.98:0/1590315771 >> 172.21.15.98:0/1590315771 conn(0x7f803806aee0 msgr2=0x7f803806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.089+0000 7f803dbff700 1 -- 172.21.15.98:0/1590315771 shutdown_connections 2023-03-03T08:01:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.089+0000 7f803dbff700 1 -- 172.21.15.98:0/1590315771 wait complete. 2023-03-03T08:01:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.091+0000 7f803dbff700 1 Processor -- start 2023-03-03T08:01:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.091+0000 7f803dbff700 1 -- start start 2023-03-03T08:01:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.092+0000 7f803dbff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80380b5410 0x7f80380b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.092+0000 7f803dbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80380b87c0 0x7f80380ab2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.092+0000 7f803dbff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80380b8b90 0x7f80381f5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.092+0000 7f803dbff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80380cbbc0 con 0x7f80380b5410 2023-03-03T08:01:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.092+0000 7f803dbff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80380abcf0 con 0x7f80380b8b90 2023-03-03T08:01:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.092+0000 7f803dbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80380cb570 con 0x7f80380b87c0 2023-03-03T08:01:14.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.092+0000 7f80377fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80380b5410 0x7f80380b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:14.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.093+0000 7f80377fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80380b5410 0x7f80380b19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33018/0 (socket says 172.21.15.98:33018) 2023-03-03T08:01:14.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.093+0000 7f80377fe700 1 -- 172.21.15.98:0/4208688216 learned_addr learned my addr 172.21.15.98:0/4208688216 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:14.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.093+0000 7f8037fff700 1 --2- 172.21.15.98:0/4208688216 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80380b8b90 0x7f80381f5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:14.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.093+0000 7f8036ffd700 1 --2- 172.21.15.98:0/4208688216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80380b87c0 0x7f80380ab2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:14.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.093+0000 7f80377fe700 1 -- 172.21.15.98:0/4208688216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80380b87c0 msgr2=0x7f80380ab2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:14.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.093+0000 7f80377fe700 1 --2- 172.21.15.98:0/4208688216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80380b87c0 0x7f80380ab2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:14.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.093+0000 7f80377fe700 1 -- 172.21.15.98:0/4208688216 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80380b8b90 msgr2=0x7f80381f5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:14.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.093+0000 7f80377fe700 1 --2- 172.21.15.98:0/4208688216 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80380b8b90 0x7f80381f5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.093+0000 7f80377fe700 1 -- 172.21.15.98:0/4208688216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8030009b30 con 0x7f80380b5410 2023-03-03T08:01:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.093+0000 7f80377fe700 1 --2- 172.21.15.98:0/4208688216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80380b5410 0x7f80380b19c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f802800e810 tx=0x7f802800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.093+0000 7f801ffff700 1 -- 172.21.15.98:0/4208688216 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802800c070 con 0x7f80380b5410 2023-03-03T08:01:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.093+0000 7f801ffff700 1 -- 172.21.15.98:0/4208688216 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8028013610 con 0x7f80380b5410 2023-03-03T08:01:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.093+0000 7f801ffff700 1 -- 172.21.15.98:0/4208688216 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80280139c0 con 0x7f80380b5410 2023-03-03T08:01:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.094+0000 7f803dbff700 1 -- 172.21.15.98:0/4208688216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80380cdf30 con 0x7f80380b5410 2023-03-03T08:01:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.094+0000 7f803dbff700 1 -- 172.21.15.98:0/4208688216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8038073a80 con 0x7f80380b5410 2023-03-03T08:01:14.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.095+0000 7f803dbff700 1 -- 172.21.15.98:0/4208688216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80380ce0c0 con 0x7f80380b5410 2023-03-03T08:01:14.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.096+0000 7f801ffff700 1 -- 172.21.15.98:0/4208688216 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f8028023070 con 0x7f80380b5410 2023-03-03T08:01:14.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.098+0000 7f801ffff700 1 --2- 172.21.15.98:0/4208688216 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f80240684a0 0x7f802406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:14.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.098+0000 7f801ffff700 1 -- 172.21.15.98:0/4208688216 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f8028085960 con 0x7f80380b5410 2023-03-03T08:01:14.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.098+0000 7f8036ffd700 1 --2- 172.21.15.98:0/4208688216 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f80240684a0 0x7f802406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:14.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.099+0000 7f8036ffd700 1 --2- 172.21.15.98:0/4208688216 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f80240684a0 0x7f802406a950 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f80380cd0a0 tx=0x7f802c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:14.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.102+0000 7f801ffff700 1 -- 172.21.15.98:0/4208688216 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f80280522c0 con 0x7f80380b5410 2023-03-03T08:01:14.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.337+0000 7f803dbff700 1 -- 172.21.15.98:0/4208688216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80380729b0 con 0x7f80380b5410 2023-03-03T08:01:14.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.337+0000 7f801ffff700 1 -- 172.21.15.98:0/4208688216 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+3526 (secure 0 0 0) 0x7f802800f040 con 0x7f80380b5410 2023-03-03T08:01:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:14.340 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":34,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5702,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/976526047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":976526047},{"type":"v1","addr":"172.21.15.100:6834","nonce":976526047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3372584689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3372584689},{"type":"v1","addr":"172.21.15.100:6835","nonce":3372584689}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/505711827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":505711827},{"type":"v1","addr":"172.21.15.98:6837","nonce":505711827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6831,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/746282543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":746282543},{"type":"v1","addr":"172.21.15.98:6836","nonce":746282543}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[]} 2023-03-03T08:01:14.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.340+0000 7f803dbff700 1 -- 172.21.15.98:0/4208688216 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f80240684a0 msgr2=0x7f802406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:14.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.340+0000 7f803dbff700 1 --2- 172.21.15.98:0/4208688216 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f80240684a0 0x7f802406a950 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f80380cd0a0 tx=0x7f802c009040 comp rx=0 tx=0).stop 2023-03-03T08:01:14.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.340+0000 7f803dbff700 1 -- 172.21.15.98:0/4208688216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80380b5410 msgr2=0x7f80380b19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:14.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.340+0000 7f803dbff700 1 --2- 172.21.15.98:0/4208688216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80380b5410 0x7f80380b19c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f802800e810 tx=0x7f802800eb20 comp rx=0 tx=0).stop 2023-03-03T08:01:14.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.340+0000 7f803dbff700 1 -- 172.21.15.98:0/4208688216 shutdown_connections 2023-03-03T08:01:14.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.340+0000 7f803dbff700 1 --2- 172.21.15.98:0/4208688216 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f80240684a0 0x7f802406a950 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:14.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.340+0000 7f803dbff700 1 --2- 172.21.15.98:0/4208688216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80380b5410 0x7f80380b19c0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:14.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.340+0000 7f803dbff700 1 --2- 172.21.15.98:0/4208688216 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80380b8b90 0x7f80381f5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:14.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.340+0000 7f803dbff700 1 --2- 172.21.15.98:0/4208688216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80380b87c0 0x7f80380ab2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:14.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.340+0000 7f803dbff700 1 -- 172.21.15.98:0/4208688216 >> 172.21.15.98:0/4208688216 conn(0x7f803806aee0 msgr2=0x7f80380d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:14.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.340+0000 7f803dbff700 1 -- 172.21.15.98:0/4208688216 shutdown_connections 2023-03-03T08:01:14.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.340+0000 7f803dbff700 1 -- 172.21.15.98:0/4208688216 wait complete. 2023-03-03T08:01:14.345 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 34 2023-03-03T08:01:14.369 INFO:tasks.ceph.mds.a:Restarting daemon 2023-03-03T08:01:14.369 INFO:tasks.ceph.mds.a:Stopping old one... 2023-03-03T08:01:14.370 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-03-03T08:01:14.370 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:01:14.370 INFO:tasks.ceph.mds.c:Restarting daemon 2023-03-03T08:01:14.370 INFO:tasks.ceph.mds.c:Stopping old one... 2023-03-03T08:01:14.370 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-03-03T08:01:14.371 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:01:14.371 INFO:tasks.ceph.mds.b:Restarting daemon 2023-03-03T08:01:14.371 INFO:tasks.ceph.mds.b:Stopping old one... 2023-03-03T08:01:14.371 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-03-03T08:01:14.372 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:01:14.372 INFO:tasks.ceph.mds.d:Restarting daemon 2023-03-03T08:01:14.372 INFO:tasks.ceph.mds.d:Stopping old one... 2023-03-03T08:01:14.372 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-03-03T08:01:14.373 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:01:14.391 INFO:tasks.ceph.mds.a:Stopped 2023-03-03T08:01:14.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-03-03T08:01:14.393 INFO:tasks.ceph.mds.c:Stopped 2023-03-03T08:01:14.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-03-03T08:01:14.403 INFO:tasks.ceph.mds.b:Stopped 2023-03-03T08:01:14.404 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-03-03T08:01:14.405 INFO:tasks.ceph.mds.d:Stopped 2023-03-03T08:01:14.405 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-03-03T08:01:14.438 INFO:tasks.ceph.mds.a:Started 2023-03-03T08:01:14.438 DEBUG:teuthology.parallel:result is None 2023-03-03T08:01:14.442 INFO:tasks.ceph.mds.c:Started 2023-03-03T08:01:14.442 DEBUG:teuthology.parallel:result is None 2023-03-03T08:01:14.449 INFO:tasks.ceph.mds.b:Started 2023-03-03T08:01:14.450 DEBUG:teuthology.parallel:result is None 2023-03-03T08:01:14.453 INFO:tasks.ceph.mds.d:Started 2023-03-03T08:01:14.454 DEBUG:teuthology.parallel:result is None 2023-03-03T08:01:14.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-03-03T08:01:14.637 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-03-03T08:01:14.638 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2023-03-03T08:01:14.644 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2023-03-03T08:01:14.644 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2023-03-03T08:01:14.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.721+0000 7fc307fff700 1 Processor -- start 2023-03-03T08:01:14.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.722+0000 7fc307fff700 1 -- start start 2023-03-03T08:01:14.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.723+0000 7fc307fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3080b8910 0x7fc3081604a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:14.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.723+0000 7fc307fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3080b5e70 0x7fc3080b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:14.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.723+0000 7fc307fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30815ce00 con 0x7fc3080cf570 2023-03-03T08:01:14.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.723+0000 7fc307fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3080d0bf0 con 0x7fc3080b8910 2023-03-03T08:01:14.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.723+0000 7fc307fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080b8ce0 con 0x7fc3080b5e70 2023-03-03T08:01:14.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.723+0000 7fc306ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3080b8910 0x7fc3081604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:14.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.723+0000 7fc306ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3080b8910 0x7fc3081604a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33064/0 (socket says 172.21.15.98:33064) 2023-03-03T08:01:14.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.723+0000 7fc306ffd700 1 -- 172.21.15.98:0/247606439 learned_addr learned my addr 172.21.15.98:0/247606439 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:14.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.724+0000 7fc2fffff700 1 --2- 172.21.15.98:0/247606439 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3080b5e70 0x7fc3080b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:14.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.724+0000 7fc306ffd700 1 -- 172.21.15.98:0/247606439 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3080b5e70 msgr2=0x7fc3080b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:14.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.724+0000 7fc306ffd700 1 --2- 172.21.15.98:0/247606439 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3080b5e70 0x7fc3080b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:14.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.724+0000 7fc306ffd700 1 -- 172.21.15.98:0/247606439 >> v1:172.21.15.100:6789/0 conn(0x7fc3080cf570 legacy=0x7fc3080d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:14.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.724+0000 7fc306ffd700 1 -- 172.21.15.98:0/247606439 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc308160cf0 con 0x7fc3080b8910 2023-03-03T08:01:14.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.724+0000 7fc306ffd700 1 --2- 172.21.15.98:0/247606439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3080b8910 0x7fc3081604a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fc2f80098f0 tx=0x7fc2f8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3721c113358f85fd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:14.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.724+0000 7fc3067fc700 1 -- 172.21.15.98:0/247606439 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8011830 con 0x7fc3080b8910 2023-03-03T08:01:14.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.725+0000 7fc3067fc700 1 -- 172.21.15.98:0/247606439 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f8011990 con 0x7fc3080b8910 2023-03-03T08:01:14.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.725+0000 7fc3067fc700 1 -- 172.21.15.98:0/247606439 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8011d20 con 0x7fc3080b8910 2023-03-03T08:01:14.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.725+0000 7fc307fff700 1 -- 172.21.15.98:0/247606439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3080b8910 msgr2=0x7fc3081604a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:14.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.725+0000 7fc307fff700 1 --2- 172.21.15.98:0/247606439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3080b8910 0x7fc3081604a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fc2f80098f0 tx=0x7fc2f8009c00 comp rx=0 tx=0).stop 2023-03-03T08:01:14.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.725+0000 7fc307fff700 1 -- 172.21.15.98:0/247606439 shutdown_connections 2023-03-03T08:01:14.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.725+0000 7fc307fff700 1 --2- 172.21.15.98:0/247606439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3080b8910 0x7fc3081604a0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:14.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.725+0000 7fc307fff700 1 --2- 172.21.15.98:0/247606439 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3080b5e70 0x7fc3080b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:14.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.725+0000 7fc307fff700 1 -- 172.21.15.98:0/247606439 >> 172.21.15.98:0/247606439 conn(0x7fc30806b880 msgr2=0x7fc3080cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:14.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.725+0000 7fc307fff700 1 -- 172.21.15.98:0/247606439 shutdown_connections 2023-03-03T08:01:14.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.725+0000 7fc307fff700 1 -- 172.21.15.98:0/247606439 wait complete. 2023-03-03T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.727+0000 7fc307fff700 1 Processor -- start 2023-03-03T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.727+0000 7fc307fff700 1 -- start start 2023-03-03T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.728+0000 7fc307fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3080b5e70 0x7fc308154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.728+0000 7fc307fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3080b8910 0x7fc3080cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.728+0000 7fc307fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3080cf570 0x7fc3080b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.728+0000 7fc307fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080cb1b0 con 0x7fc3080b5e70 2023-03-03T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.728+0000 7fc307fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080b31d0 con 0x7fc3080b8910 2023-03-03T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.728+0000 7fc307fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080b34b0 con 0x7fc3080cf570 2023-03-03T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.729+0000 7fc306ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3080b5e70 0x7fc308154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.729+0000 7fc3077fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3080cf570 0x7fc3080b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.729+0000 7fc306ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3080b5e70 0x7fc308154780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33070/0 (socket says 172.21.15.98:33070) 2023-03-03T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.729+0000 7fc306ffd700 1 -- 172.21.15.98:0/3354598426 learned_addr learned my addr 172.21.15.98:0/3354598426 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:14.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.729+0000 7fc2fffff700 1 --2- 172.21.15.98:0/3354598426 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3080b8910 0x7fc3080cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:14.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.729+0000 7fc306ffd700 1 -- 172.21.15.98:0/3354598426 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3080cf570 msgr2=0x7fc3080b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.729+0000 7fc306ffd700 1 --2- 172.21.15.98:0/3354598426 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3080cf570 0x7fc3080b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.729+0000 7fc306ffd700 1 -- 172.21.15.98:0/3354598426 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3080b8910 msgr2=0x7fc3080cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.729+0000 7fc306ffd700 1 --2- 172.21.15.98:0/3354598426 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3080b8910 0x7fc3080cac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.729+0000 7fc306ffd700 1 -- 172.21.15.98:0/3354598426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f8009550 con 0x7fc3080b5e70 2023-03-03T08:01:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.730+0000 7fc306ffd700 1 --2- 172.21.15.98:0/3354598426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3080b5e70 0x7fc308154780 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8000c00 tx=0x7fc2f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.730+0000 7fc2ff7fe700 1 -- 172.21.15.98:0/3354598426 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8004400 con 0x7fc3080b5e70 2023-03-03T08:01:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.730+0000 7fc2ff7fe700 1 -- 172.21.15.98:0/3354598426 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f801f070 con 0x7fc3080b5e70 2023-03-03T08:01:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.730+0000 7fc307fff700 1 -- 172.21.15.98:0/3354598426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3081612b0 con 0x7fc3080b5e70 2023-03-03T08:01:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.730+0000 7fc2ff7fe700 1 -- 172.21.15.98:0/3354598426 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f80038e0 con 0x7fc3080b5e70 2023-03-03T08:01:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.730+0000 7fc307fff700 1 -- 172.21.15.98:0/3354598426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc308160fd0 con 0x7fc3080b5e70 2023-03-03T08:01:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.731+0000 7fc307fff700 1 -- 172.21.15.98:0/3354598426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3080d0eb0 con 0x7fc3080b5e70 2023-03-03T08:01:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.733+0000 7fc2ff7fe700 1 -- 172.21.15.98:0/3354598426 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fc2f802a020 con 0x7fc3080b5e70 2023-03-03T08:01:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.734+0000 7fc2ff7fe700 1 --2- 172.21.15.98:0/3354598426 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc2e80684a0 0x7fc2e806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.734+0000 7fc2ff7fe700 1 -- 172.21.15.98:0/3354598426 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fc2f80870e0 con 0x7fc3080b5e70 2023-03-03T08:01:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.734+0000 7fc2fffff700 1 --2- 172.21.15.98:0/3354598426 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc2e80684a0 0x7fc2e806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.735+0000 7fc2fffff700 1 --2- 172.21.15.98:0/3354598426 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc2e80684a0 0x7fc2e806a950 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc308154cc0 tx=0x7fc2f0009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:14.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.738+0000 7fc2ff7fe700 1 -- 172.21.15.98:0/3354598426 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc2f8053a40 con 0x7fc3080b5e70 2023-03-03T08:01:14.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:14.954+0000 7fc307fff700 1 -- 172.21.15.98:0/3354598426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc308068290 con 0x7fc3080b5e70 2023-03-03T08:01:15.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:15.775+0000 7fc2ff7fe700 1 -- 172.21.15.98:0/3354598426 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v74) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc2f8057060 con 0x7fc3080b5e70 2023-03-03T08:01:15.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:15.779+0000 7fc307fff700 1 -- 172.21.15.98:0/3354598426 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc2e80684a0 msgr2=0x7fc2e806a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:15.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:15.779+0000 7fc307fff700 1 --2- 172.21.15.98:0/3354598426 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc2e80684a0 0x7fc2e806a950 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc308154cc0 tx=0x7fc2f0009210 comp rx=0 tx=0).stop 2023-03-03T08:01:15.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:15.779+0000 7fc307fff700 1 -- 172.21.15.98:0/3354598426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3080b5e70 msgr2=0x7fc308154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:15.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:15.779+0000 7fc307fff700 1 --2- 172.21.15.98:0/3354598426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3080b5e70 0x7fc308154780 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8000c00 tx=0x7fc2f8004750 comp rx=0 tx=0).stop 2023-03-03T08:01:15.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:15.779+0000 7fc307fff700 1 -- 172.21.15.98:0/3354598426 shutdown_connections 2023-03-03T08:01:15.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:15.779+0000 7fc307fff700 1 --2- 172.21.15.98:0/3354598426 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc2e80684a0 0x7fc2e806a950 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:15.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:15.779+0000 7fc307fff700 1 --2- 172.21.15.98:0/3354598426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3080b5e70 0x7fc308154780 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:15.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:15.779+0000 7fc307fff700 1 --2- 172.21.15.98:0/3354598426 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3080cf570 0x7fc3080b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:15.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:15.779+0000 7fc307fff700 1 --2- 172.21.15.98:0/3354598426 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3080b8910 0x7fc3080cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:15.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:15.779+0000 7fc307fff700 1 -- 172.21.15.98:0/3354598426 >> 172.21.15.98:0/3354598426 conn(0x7fc30806b880 msgr2=0x7fc3080737c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:15.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:15.779+0000 7fc307fff700 1 -- 172.21.15.98:0/3354598426 shutdown_connections 2023-03-03T08:01:15.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:15.780+0000 7fc307fff700 1 -- 172.21.15.98:0/3354598426 wait complete. 2023-03-03T08:01:15.784 INFO:teuthology.orchestra.run.smithi098.stderr: removed all blocklist entries 2023-03-03T08:01:15.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-03-03T08:01:16.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.057+0000 7f5727fff700 1 Processor -- start 2023-03-03T08:01:16.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.057+0000 7f5727fff700 1 -- start start 2023-03-03T08:01:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.059+0000 7f5727fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5728146360 0x7f5728160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.059+0000 7f5727fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57280b4ab0 0x7f57280b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.059+0000 7f5727fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57281614c0 con 0x7f57280731e0 2023-03-03T08:01:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.059+0000 7f5727fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57280b5440 con 0x7f5728146360 2023-03-03T08:01:16.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.059+0000 7f5727fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57280d10f0 con 0x7f57280b4ab0 2023-03-03T08:01:16.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.059+0000 7f57277fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f57280731e0 0x7f5728073660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52266/0 (socket says 172.21.15.98:52266) 2023-03-03T08:01:16.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.059+0000 7f57277fe700 1 -- 172.21.15.98:0/2430662328 learned_addr learned my addr 172.21.15.98:0/2430662328 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:16.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.059+0000 7f57267fc700 1 --2- 172.21.15.98:0/2430662328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57280b4ab0 0x7f57280b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:16.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.059+0000 7f5726ffd700 1 --2- 172.21.15.98:0/2430662328 >> v2:172.21.15.100:3300/0 conn(0x7f5728146360 0x7f5728160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:16.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.060+0000 7f5725ffb700 1 -- 172.21.15.98:0/2430662328 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868457556 0 0) 0x7f5720001490 con 0x7f57280731e0 2023-03-03T08:01:16.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.060+0000 7f5725ffb700 1 -- 172.21.15.98:0/2430662328 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57100035b0 con 0x7f57280731e0 2023-03-03T08:01:16.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.060+0000 7f57267fc700 1 -- 172.21.15.98:0/2430662328 >> v2:172.21.15.100:3300/0 conn(0x7f5728146360 msgr2=0x7f5728160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:16.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.060+0000 7f57267fc700 1 --2- 172.21.15.98:0/2430662328 >> v2:172.21.15.100:3300/0 conn(0x7f5728146360 0x7f5728160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.060+0000 7f57267fc700 1 -- 172.21.15.98:0/2430662328 >> v1:172.21.15.98:6789/0 conn(0x7f57280731e0 legacy=0x7f5728073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:16.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.060+0000 7f57267fc700 1 -- 172.21.15.98:0/2430662328 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57280cdf20 con 0x7f57280b4ab0 2023-03-03T08:01:16.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.060+0000 7f5726ffd700 1 --2- 172.21.15.98:0/2430662328 >> v2:172.21.15.100:3300/0 conn(0x7f5728146360 0x7f5728160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:16.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.061+0000 7f57267fc700 1 --2- 172.21.15.98:0/2430662328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57280b4ab0 0x7f57280b4f00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f571c009840 tx=0x7f571c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=479c19ec9464d3ea server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:16.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.061+0000 7f5725ffb700 1 -- 172.21.15.98:0/2430662328 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c011830 con 0x7f57280b4ab0 2023-03-03T08:01:16.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.061+0000 7f5725ffb700 1 -- 172.21.15.98:0/2430662328 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c011990 con 0x7f57280b4ab0 2023-03-03T08:01:16.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.061+0000 7f5725ffb700 1 -- 172.21.15.98:0/2430662328 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c011d20 con 0x7f57280b4ab0 2023-03-03T08:01:16.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.061+0000 7f5727fff700 1 -- 172.21.15.98:0/2430662328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57280b4ab0 msgr2=0x7f57280b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:16.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.061+0000 7f5727fff700 1 --2- 172.21.15.98:0/2430662328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57280b4ab0 0x7f57280b4f00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f571c009840 tx=0x7f571c009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:16.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.061+0000 7f5727fff700 1 -- 172.21.15.98:0/2430662328 shutdown_connections 2023-03-03T08:01:16.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.061+0000 7f5727fff700 1 --2- 172.21.15.98:0/2430662328 >> v2:172.21.15.100:3300/0 conn(0x7f5728146360 0x7f5728160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.061+0000 7f5727fff700 1 --2- 172.21.15.98:0/2430662328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57280b4ab0 0x7f57280b4f00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.062+0000 7f5727fff700 1 -- 172.21.15.98:0/2430662328 >> 172.21.15.98:0/2430662328 conn(0x7f572806b8e0 msgr2=0x7f57280cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:16.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.062+0000 7f5727fff700 1 -- 172.21.15.98:0/2430662328 shutdown_connections 2023-03-03T08:01:16.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.062+0000 7f5727fff700 1 -- 172.21.15.98:0/2430662328 wait complete. 2023-03-03T08:01:16.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.064+0000 7f5727fff700 1 Processor -- start 2023-03-03T08:01:16.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.064+0000 7f5727fff700 1 -- start start 2023-03-03T08:01:16.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.065+0000 7f5727fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57280731e0 0x7f57280c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:16.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.065+0000 7f5727fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57280b4ab0 0x7f57280b93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:16.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.065+0000 7f5727fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5728146360 0x7f57281f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:16.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.065+0000 7f5727fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57280b3990 con 0x7f57280731e0 2023-03-03T08:01:16.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.065+0000 7f5727fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57280c72c0 con 0x7f5728146360 2023-03-03T08:01:16.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.065+0000 7f5727fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57280b36b0 con 0x7f57280b4ab0 2023-03-03T08:01:16.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.065+0000 7f5726ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57280731e0 0x7f57280c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:16.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.065+0000 7f5726ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57280731e0 0x7f57280c6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60308/0 (socket says 172.21.15.98:60308) 2023-03-03T08:01:16.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.065+0000 7f5726ffd700 1 -- 172.21.15.98:0/4235031515 learned_addr learned my addr 172.21.15.98:0/4235031515 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:16.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.066+0000 7f5726ffd700 1 -- 172.21.15.98:0/4235031515 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57280b4ab0 msgr2=0x7f57280b93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:16.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.066+0000 7f5726ffd700 1 --2- 172.21.15.98:0/4235031515 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57280b4ab0 0x7f57280b93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.066+0000 7f5726ffd700 1 -- 172.21.15.98:0/4235031515 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5728146360 msgr2=0x7f57281f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:16.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.066+0000 7f5726ffd700 1 --2- 172.21.15.98:0/4235031515 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5728146360 0x7f57281f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.066+0000 7f5726ffd700 1 -- 172.21.15.98:0/4235031515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57180094a0 con 0x7f57280731e0 2023-03-03T08:01:16.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.066+0000 7f5726ffd700 1 --2- 172.21.15.98:0/4235031515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57280731e0 0x7f57280c6050 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f571800ec10 tx=0x7f5718009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:16.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.066+0000 7f570f7fe700 1 -- 172.21.15.98:0/4235031515 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571800c8e0 con 0x7f57280731e0 2023-03-03T08:01:16.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.066+0000 7f570f7fe700 1 -- 172.21.15.98:0/4235031515 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571800ca40 con 0x7f57280731e0 2023-03-03T08:01:16.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.066+0000 7f5727fff700 1 -- 172.21.15.98:0/4235031515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571c0094a0 con 0x7f57280731e0 2023-03-03T08:01:16.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.066+0000 7f570f7fe700 1 -- 172.21.15.98:0/4235031515 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571800cdd0 con 0x7f57280731e0 2023-03-03T08:01:16.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.067+0000 7f5727fff700 1 -- 172.21.15.98:0/4235031515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57280cd960 con 0x7f57280731e0 2023-03-03T08:01:16.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.068+0000 7f5727fff700 1 -- 172.21.15.98:0/4235031515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57280cd680 con 0x7f57280731e0 2023-03-03T08:01:16.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.069+0000 7f570f7fe700 1 -- 172.21.15.98:0/4235031515 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f5718019030 con 0x7f57280731e0 2023-03-03T08:01:16.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.070+0000 7f570f7fe700 1 --2- 172.21.15.98:0/4235031515 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f57140684a0 0x7f571406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:16.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.070+0000 7f570f7fe700 1 -- 172.21.15.98:0/4235031515 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5718085bb0 con 0x7f57280731e0 2023-03-03T08:01:16.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.070+0000 7f57267fc700 1 --2- 172.21.15.98:0/4235031515 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f57140684a0 0x7f571406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:16.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.071+0000 7f57267fc700 1 --2- 172.21.15.98:0/4235031515 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f57140684a0 0x7f571406a950 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f571c009b90 tx=0x7f571c010000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:16.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.074+0000 7f570f7fe700 1 -- 172.21.15.98:0/4235031515 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f57180525c0 con 0x7f57280731e0 2023-03-03T08:01:16.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.342+0000 7f5727fff700 1 -- 172.21.15.98:0/4235031515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f57280cdc40 con 0x7f57280731e0 2023-03-03T08:01:16.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.343+0000 7f570f7fe700 1 -- 172.21.15.98:0/4235031515 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5626 (secure 0 0 0) 0x7f5718055be0 con 0x7f57280731e0 2023-03-03T08:01:16.345 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:16.345 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:01:16.346 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_dump": [ 2023-03-03T08:01:16.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.346 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.a", 2023-03-03T08:01:16.346 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkqFdgHxAAuwiGkaNnr5NplmhniDYAEQ==", 2023-03-03T08:01:16.346 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.347 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:01:16.347 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:01:16.347 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:01:16.348 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:01:16.348 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.349 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.b", 2023-03-03T08:01:16.349 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFk0x8kBhAA5JDo/wmI4650J9tUnvXVzw==", 2023-03-03T08:01:16.349 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.349 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:01:16.350 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:01:16.350 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:01:16.350 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:01:16.350 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.351 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.c", 2023-03-03T08:01:16.352 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkKoRwMBAAfkWGDUsqNgqfkscl0aBX1A==", 2023-03-03T08:01:16.352 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:01:16.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:01:16.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:01:16.353 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:01:16.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.354 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.d", 2023-03-03T08:01:16.354 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFkK9SKFxAAFSCozGfyRwrv46QPitRxTQ==", 2023-03-03T08:01:16.354 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:01:16.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:01:16.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:01:16.355 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:01:16.355 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.356 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.0", 2023-03-03T08:01:16.356 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDBpwFkTFzTERAAGHrFKuBlXIEfaYD7vs3ZHA==", 2023-03-03T08:01:16.356 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:01:16.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:01:16.357 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:01:16.357 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.357 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.1", 2023-03-03T08:01:16.357 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDEpwFkVWObLBAAW4acTvNbmscIe18NXVgRYg==", 2023-03-03T08:01:16.357 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:01:16.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:01:16.358 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:01:16.358 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.358 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.2", 2023-03-03T08:01:16.358 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDIpwFkK8lgChAAHVD5btBivUpcapSgDp2E0w==", 2023-03-03T08:01:16.359 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:01:16.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:01:16.359 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:01:16.359 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.360 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.3", 2023-03-03T08:01:16.360 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDLpwFksrb8IxAAN+CP1O3DJ1zDy+qyGNor2Q==", 2023-03-03T08:01:16.360 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:01:16.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:01:16.360 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:01:16.360 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.361 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.4", 2023-03-03T08:01:16.361 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDSpwFkkWGHIxAA4wejA0w05JPIUghEAVG/Rg==", 2023-03-03T08:01:16.361 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:01:16.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:01:16.362 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:01:16.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.363 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.5", 2023-03-03T08:01:16.364 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDWpwFknHmGAhAA6VM16luaykZfJEzr+yDZZA==", 2023-03-03T08:01:16.364 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.364 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:01:16.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:01:16.365 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:01:16.365 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.366 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.6", 2023-03-03T08:01:16.366 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDZpwFkcitNDRAAOz04YGTzx3Phenp0n8JeEA==", 2023-03-03T08:01:16.366 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.366 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:01:16.366 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:01:16.367 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:01:16.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.368 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.7", 2023-03-03T08:01:16.368 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDcpwFkrjmIFxAAbWeB2+8VQgyTplW+fk47OA==", 2023-03-03T08:01:16.368 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.368 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:01:16.368 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:01:16.368 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:01:16.369 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.369 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.0", 2023-03-03T08:01:16.369 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFk02lsJhAA0cH/evMnWdZN6DdBGCLjjg==", 2023-03-03T08:01:16.370 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.370 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:01:16.370 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow r", 2023-03-03T08:01:16.370 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rw pool=cephfs_data" 2023-03-03T08:01:16.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.371 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.1", 2023-03-03T08:01:16.371 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC9pwFkRqXrABAArm/01O5MtSTQHWQWKqNQEg==", 2023-03-03T08:01:16.371 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:01:16.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T08:01:16.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T08:01:16.372 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T08:01:16.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.372 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.2", 2023-03-03T08:01:16.373 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFkyKp3MRAA8Eh41TZNsO6/8VhAWCpNMQ==", 2023-03-03T08:01:16.373 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.373 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:01:16.373 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T08:01:16.373 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T08:01:16.373 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T08:01:16.373 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.374 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.3", 2023-03-03T08:01:16.374 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC9pwFk14bSCxAAy2nXc9G50W4lH5GtH84hpQ==", 2023-03-03T08:01:16.374 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:01:16.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T08:01:16.375 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T08:01:16.375 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T08:01:16.375 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.375 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.admin", 2023-03-03T08:01:16.376 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC6pwFkokmiEBAAFwi8rhPmk4Zyk95p760HDg==", 2023-03-03T08:01:16.376 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T08:01:16.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:01:16.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:01:16.376 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:01:16.376 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.377 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.alice", 2023-03-03T08:01:16.377 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQAzqQFkigHjCRAA+vJpqEsD7NuAQLrmRJE9Ag==", 2023-03-03T08:01:16.377 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.377 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow rw path=/volumes/_nogroup/subvolume_0000000001024199", 2023-03-03T08:01:16.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow r", 2023-03-03T08:01:16.378 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rw pool=cephfs_data" 2023-03-03T08:01:16.378 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.378 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mgr.x", 2023-03-03T08:01:16.378 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkvdMQFBAAMi+x7psn5CDpd//d6jN2VA==", 2023-03-03T08:01:16.379 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T08:01:16.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile mgr", 2023-03-03T08:01:16.379 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:01:16.379 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:16.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:16.379 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mgr.y", 2023-03-03T08:01:16.380 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkfYO5BxAAFW/elbdoqVXBkUIKrAvYsg==", 2023-03-03T08:01:16.380 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:01:16.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T08:01:16.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile mgr", 2023-03-03T08:01:16.380 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:01:16.380 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:16.381 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:01:16.381 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:01:16.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.346+0000 7f5727fff700 1 -- 172.21.15.98:0/4235031515 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f57140684a0 msgr2=0x7f571406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:16.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.346+0000 7f5727fff700 1 --2- 172.21.15.98:0/4235031515 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f57140684a0 0x7f571406a950 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f571c009b90 tx=0x7f571c010000 comp rx=0 tx=0).stop 2023-03-03T08:01:16.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.346+0000 7f5727fff700 1 -- 172.21.15.98:0/4235031515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57280731e0 msgr2=0x7f57280c6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:16.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.346+0000 7f5727fff700 1 --2- 172.21.15.98:0/4235031515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57280731e0 0x7f57280c6050 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f571800ec10 tx=0x7f5718009e20 comp rx=0 tx=0).stop 2023-03-03T08:01:16.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.346+0000 7f5727fff700 1 -- 172.21.15.98:0/4235031515 shutdown_connections 2023-03-03T08:01:16.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.346+0000 7f5727fff700 1 --2- 172.21.15.98:0/4235031515 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f57140684a0 0x7f571406a950 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.346+0000 7f5727fff700 1 --2- 172.21.15.98:0/4235031515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57280731e0 0x7f57280c6050 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.346+0000 7f5727fff700 1 --2- 172.21.15.98:0/4235031515 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5728146360 0x7f57281f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.346+0000 7f5727fff700 1 --2- 172.21.15.98:0/4235031515 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57280b4ab0 0x7f57280b93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.346+0000 7f5727fff700 1 -- 172.21.15.98:0/4235031515 >> 172.21.15.98:0/4235031515 conn(0x7f572806b8e0 msgr2=0x7f572815c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:16.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.346+0000 7f5727fff700 1 -- 172.21.15.98:0/4235031515 shutdown_connections 2023-03-03T08:01:16.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.346+0000 7f5727fff700 1 -- 172.21.15.98:0/4235031515 wait complete. 2023-03-03T08:01:16.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.alice 2023-03-03T08:01:16.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.630+0000 7fad0d3ff700 1 Processor -- start 2023-03-03T08:01:16.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.630+0000 7fad0d3ff700 1 -- start start 2023-03-03T08:01:16.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.631+0000 7fad0d3ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad0815ca70 0x7fad080cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:16.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.632+0000 7fad0d3ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad080cf980 con 0x7fad0806b4b0 2023-03-03T08:01:16.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.632+0000 7fad0d3ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad080d1800 con 0x7fad080d0270 2023-03-03T08:01:16.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.632+0000 7fad0d3ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad080cd5b0 con 0x7fad0815ca70 2023-03-03T08:01:16.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.632+0000 7fad06ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad0815ca70 0x7fad080cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:16.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.632+0000 7fad067fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fad0806b4b0 0x7fad080cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52282/0 (socket says 172.21.15.98:52282) 2023-03-03T08:01:16.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.632+0000 7fad067fc700 1 -- 172.21.15.98:0/274102948 learned_addr learned my addr 172.21.15.98:0/274102948 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:16.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.632+0000 7fad06ffd700 1 -- 172.21.15.98:0/274102948 >> v1:172.21.15.100:6789/0 conn(0x7fad080d0270 legacy=0x7fad080d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:16.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.632+0000 7fad06ffd700 1 -- 172.21.15.98:0/274102948 >> v1:172.21.15.98:6789/0 conn(0x7fad0806b4b0 legacy=0x7fad080cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:16.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.632+0000 7fad06ffd700 1 -- 172.21.15.98:0/274102948 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad080cd900 con 0x7fad0815ca70 2023-03-03T08:01:16.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.633+0000 7fad06ffd700 1 --2- 172.21.15.98:0/274102948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0815ca70 0x7fad080cf440 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7facf8009840 tx=0x7facf8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f6e83b6038045a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:16.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.633+0000 7fad05ffb700 1 -- 172.21.15.98:0/274102948 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf8011830 con 0x7fad0815ca70 2023-03-03T08:01:16.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.633+0000 7fad05ffb700 1 -- 172.21.15.98:0/274102948 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf8011990 con 0x7fad0815ca70 2023-03-03T08:01:16.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.633+0000 7fad05ffb700 1 -- 172.21.15.98:0/274102948 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf8011d20 con 0x7fad0815ca70 2023-03-03T08:01:16.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.634+0000 7fad0d3ff700 1 -- 172.21.15.98:0/274102948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0815ca70 msgr2=0x7fad080cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:16.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.634+0000 7fad0d3ff700 1 --2- 172.21.15.98:0/274102948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0815ca70 0x7fad080cf440 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7facf8009840 tx=0x7facf8009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:16.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.634+0000 7fad0d3ff700 1 -- 172.21.15.98:0/274102948 shutdown_connections 2023-03-03T08:01:16.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.634+0000 7fad0d3ff700 1 --2- 172.21.15.98:0/274102948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0815ca70 0x7fad080cf440 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.634+0000 7fad0d3ff700 1 -- 172.21.15.98:0/274102948 >> 172.21.15.98:0/274102948 conn(0x7fad08073e70 msgr2=0x7fad0806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:16.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.634+0000 7fad0d3ff700 1 -- 172.21.15.98:0/274102948 shutdown_connections 2023-03-03T08:01:16.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.634+0000 7fad0d3ff700 1 -- 172.21.15.98:0/274102948 wait complete. 2023-03-03T08:01:16.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.636+0000 7fad0d3ff700 1 Processor -- start 2023-03-03T08:01:16.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.636+0000 7fad0d3ff700 1 -- start start 2023-03-03T08:01:16.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.637+0000 7fad0d3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0806b4b0 0x7fad080c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:16.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.637+0000 7fad0d3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad080d0270 0x7fad080c7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:16.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.637+0000 7fad0d3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0815ca70 0x7fad080c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:16.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.637+0000 7fad0d3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad080c8a60 con 0x7fad0815ca70 2023-03-03T08:01:16.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.637+0000 7fad0d3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad080c8d40 con 0x7fad080d0270 2023-03-03T08:01:16.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.637+0000 7fad0d3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad080c9020 con 0x7fad0806b4b0 2023-03-03T08:01:16.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.637+0000 7fad077fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0815ca70 0x7fad080c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:16.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.637+0000 7fad077fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0815ca70 0x7fad080c83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60332/0 (socket says 172.21.15.98:60332) 2023-03-03T08:01:16.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.637+0000 7fad077fe700 1 -- 172.21.15.98:0/1775835393 learned_addr learned my addr 172.21.15.98:0/1775835393 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:16.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.637+0000 7fad067fc700 1 --2- 172.21.15.98:0/1775835393 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad080d0270 0x7fad080c7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:16.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.637+0000 7fad06ffd700 1 --2- 172.21.15.98:0/1775835393 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0806b4b0 0x7fad080c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:16.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.638+0000 7fad077fe700 1 -- 172.21.15.98:0/1775835393 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0806b4b0 msgr2=0x7fad080c9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:16.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.638+0000 7fad077fe700 1 --2- 172.21.15.98:0/1775835393 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0806b4b0 0x7fad080c9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.638+0000 7fad077fe700 1 -- 172.21.15.98:0/1775835393 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad080d0270 msgr2=0x7fad080c7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:16.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.638+0000 7fad077fe700 1 --2- 172.21.15.98:0/1775835393 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad080d0270 0x7fad080c7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.638+0000 7fad077fe700 1 -- 172.21.15.98:0/1775835393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf80094a0 con 0x7fad0815ca70 2023-03-03T08:01:16.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.638+0000 7fad077fe700 1 --2- 172.21.15.98:0/1775835393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0815ca70 0x7fad080c83e0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fad0000cb90 tx=0x7fad0000cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:16.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.638+0000 7facef7fe700 1 -- 172.21.15.98:0/1775835393 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad00003c00 con 0x7fad0815ca70 2023-03-03T08:01:16.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.638+0000 7facef7fe700 1 -- 172.21.15.98:0/1775835393 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad00003d60 con 0x7fad0815ca70 2023-03-03T08:01:16.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.638+0000 7facef7fe700 1 -- 172.21.15.98:0/1775835393 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0000aae0 con 0x7fad0815ca70 2023-03-03T08:01:16.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.638+0000 7fad0d3ff700 1 -- 172.21.15.98:0/1775835393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad080d1aa0 con 0x7fad0815ca70 2023-03-03T08:01:16.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.639+0000 7fad0d3ff700 1 -- 172.21.15.98:0/1775835393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad080cdbe0 con 0x7fad0815ca70 2023-03-03T08:01:16.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.640+0000 7fad0d3ff700 1 -- 172.21.15.98:0/1775835393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad0815ce40 con 0x7fad0815ca70 2023-03-03T08:01:16.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.641+0000 7facef7fe700 1 -- 172.21.15.98:0/1775835393 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fad00010050 con 0x7fad0815ca70 2023-03-03T08:01:16.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.642+0000 7facef7fe700 1 --2- 172.21.15.98:0/1775835393 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7facf40684a0 0x7facf406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:16.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.642+0000 7facef7fe700 1 -- 172.21.15.98:0/1775835393 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fad00085a70 con 0x7fad0815ca70 2023-03-03T08:01:16.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.642+0000 7fad06ffd700 1 --2- 172.21.15.98:0/1775835393 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7facf40684a0 0x7facf406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:16.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.643+0000 7fad06ffd700 1 --2- 172.21.15.98:0/1775835393 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7facf40684a0 0x7facf406a950 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7facf8000c00 tx=0x7facf801c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:16.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.646+0000 7facef7fe700 1 -- 172.21.15.98:0/1775835393 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fad00052320 con 0x7fad0815ca70 2023-03-03T08:01:16.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.912+0000 7fad0d3ff700 1 -- 172.21.15.98:0/1775835393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.alice"} v 0) v1 -- 0x7fad0804e6e0 con 0x7fad0815ca70 2023-03-03T08:01:16.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.917+0000 7facef7fe700 1 -- 172.21.15.98:0/1775835393 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.alice"}]=0 v13) v1 ==== 82+0+0 (secure 0 0 0) 0x7fad00055940 con 0x7fad0815ca70 2023-03-03T08:01:16.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.919+0000 7fad0d3ff700 1 -- 172.21.15.98:0/1775835393 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7facf40684a0 msgr2=0x7facf406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:16.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.919+0000 7fad0d3ff700 1 --2- 172.21.15.98:0/1775835393 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7facf40684a0 0x7facf406a950 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7facf8000c00 tx=0x7facf801c040 comp rx=0 tx=0).stop 2023-03-03T08:01:16.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.919+0000 7fad0d3ff700 1 -- 172.21.15.98:0/1775835393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0815ca70 msgr2=0x7fad080c83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:16.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.919+0000 7fad0d3ff700 1 --2- 172.21.15.98:0/1775835393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0815ca70 0x7fad080c83e0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fad0000cb90 tx=0x7fad0000cea0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.919+0000 7fad0d3ff700 1 -- 172.21.15.98:0/1775835393 shutdown_connections 2023-03-03T08:01:16.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.919+0000 7fad0d3ff700 1 --2- 172.21.15.98:0/1775835393 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7facf40684a0 0x7facf406a950 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.919+0000 7fad0d3ff700 1 --2- 172.21.15.98:0/1775835393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0815ca70 0x7fad080c83e0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.919+0000 7fad0d3ff700 1 --2- 172.21.15.98:0/1775835393 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0806b4b0 0x7fad080c9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.919+0000 7fad0d3ff700 1 --2- 172.21.15.98:0/1775835393 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad080d0270 0x7fad080c7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:16.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.920+0000 7fad0d3ff700 1 -- 172.21.15.98:0/1775835393 >> 172.21.15.98:0/1775835393 conn(0x7fad08073e70 msgr2=0x7fad080b50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:16.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.920+0000 7fad0d3ff700 1 -- 172.21.15.98:0/1775835393 shutdown_connections 2023-03-03T08:01:16.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:16.920+0000 7fad0d3ff700 1 -- 172.21.15.98:0/1775835393 wait complete. 2023-03-03T08:01:16.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:01:17.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.193+0000 7f63923ff700 1 Processor -- start 2023-03-03T08:01:17.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.193+0000 7f63923ff700 1 -- start start 2023-03-03T08:01:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.195+0000 7f63923ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638c0b87c0 0x7f638c0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.195+0000 7f63923ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f638c0b8b90 0x7f638c0b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.195+0000 7f63923ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c0d0ef0 con 0x7f638c0b5240 2023-03-03T08:01:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.195+0000 7f63923ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f638c0d0600 con 0x7f638c0b8b90 2023-03-03T08:01:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.195+0000 7f63923ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638c0cd970 con 0x7f638c0b87c0 2023-03-03T08:01:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.195+0000 7f638b7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f638c0b8b90 0x7f638c0b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.195+0000 7f638b7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f638c0b8b90 0x7f638c0b4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60342/0 (socket says 172.21.15.98:60342) 2023-03-03T08:01:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.195+0000 7f638b7fe700 1 -- 172.21.15.98:0/3178541521 learned_addr learned my addr 172.21.15.98:0/3178541521 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.195+0000 7f638bfff700 1 --2- 172.21.15.98:0/3178541521 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638c0b87c0 0x7f638c0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.196+0000 7f638b7fe700 1 -- 172.21.15.98:0/3178541521 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638c0b87c0 msgr2=0x7f638c0cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.196+0000 7f638b7fe700 1 --2- 172.21.15.98:0/3178541521 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638c0b87c0 0x7f638c0cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.196+0000 7f638b7fe700 1 -- 172.21.15.98:0/3178541521 >> v1:172.21.15.100:6789/0 conn(0x7f638c0b5240 legacy=0x7f638c1605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.196+0000 7f638b7fe700 1 -- 172.21.15.98:0/3178541521 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c0cdc50 con 0x7f638c0b8b90 2023-03-03T08:01:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.196+0000 7f638b7fe700 1 --2- 172.21.15.98:0/3178541521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638c0b8b90 0x7f638c0b4d00 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f63800098f0 tx=0x7f6380009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766d88234855467c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.196+0000 7f638affd700 1 -- 172.21.15.98:0/3178541521 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638000b8a0 con 0x7f638c0b8b90 2023-03-03T08:01:17.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.196+0000 7f638affd700 1 -- 172.21.15.98:0/3178541521 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638000ba00 con 0x7f638c0b8b90 2023-03-03T08:01:17.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.196+0000 7f638affd700 1 -- 172.21.15.98:0/3178541521 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638000bd90 con 0x7f638c0b8b90 2023-03-03T08:01:17.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.197+0000 7f63923ff700 1 -- 172.21.15.98:0/3178541521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638c0b8b90 msgr2=0x7f638c0b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:17.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.197+0000 7f63923ff700 1 --2- 172.21.15.98:0/3178541521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638c0b8b90 0x7f638c0b4d00 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f63800098f0 tx=0x7f6380009c00 comp rx=0 tx=0).stop 2023-03-03T08:01:17.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.197+0000 7f63923ff700 1 -- 172.21.15.98:0/3178541521 shutdown_connections 2023-03-03T08:01:17.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.197+0000 7f63923ff700 1 --2- 172.21.15.98:0/3178541521 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638c0b87c0 0x7f638c0cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.197+0000 7f63923ff700 1 --2- 172.21.15.98:0/3178541521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638c0b8b90 0x7f638c0b4d00 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.197+0000 7f63923ff700 1 -- 172.21.15.98:0/3178541521 >> 172.21.15.98:0/3178541521 conn(0x7f638c06aee0 msgr2=0x7f638c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.197+0000 7f63923ff700 1 -- 172.21.15.98:0/3178541521 shutdown_connections 2023-03-03T08:01:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.197+0000 7f63923ff700 1 -- 172.21.15.98:0/3178541521 wait complete. 2023-03-03T08:01:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.199+0000 7f63923ff700 1 Processor -- start 2023-03-03T08:01:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.199+0000 7f63923ff700 1 -- start start 2023-03-03T08:01:17.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.200+0000 7f63923ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638c0b5240 0x7f638c0b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:17.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.200+0000 7f63923ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638c0b87c0 0x7f638c0cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:17.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.200+0000 7f63923ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f638c0b8b90 0x7f638c1e7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:17.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.200+0000 7f63923ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638c14b200 con 0x7f638c0b87c0 2023-03-03T08:01:17.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.200+0000 7f63923ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638c0b2610 con 0x7f638c0b8b90 2023-03-03T08:01:17.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.200+0000 7f63923ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638c0cccc0 con 0x7f638c0b5240 2023-03-03T08:01:17.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.201+0000 7f638b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638c0b87c0 0x7f638c0cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:17.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.201+0000 7f638b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638c0b87c0 0x7f638c0cbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60354/0 (socket says 172.21.15.98:60354) 2023-03-03T08:01:17.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.201+0000 7f638bfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638c0b5240 0x7f638c0b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:17.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.201+0000 7f638b7fe700 1 -- 172.21.15.98:0/3761137535 learned_addr learned my addr 172.21.15.98:0/3761137535 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:17.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.201+0000 7f639099c700 1 --2- 172.21.15.98:0/3761137535 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f638c0b8b90 0x7f638c1e7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:17.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.201+0000 7f638b7fe700 1 -- 172.21.15.98:0/3761137535 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638c0b5240 msgr2=0x7f638c0b20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:17.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.201+0000 7f638b7fe700 1 --2- 172.21.15.98:0/3761137535 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638c0b5240 0x7f638c0b20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.201+0000 7f638b7fe700 1 -- 172.21.15.98:0/3761137535 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f638c0b8b90 msgr2=0x7f638c1e7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:17.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.201+0000 7f638b7fe700 1 --2- 172.21.15.98:0/3761137535 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f638c0b8b90 0x7f638c1e7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.201+0000 7f638b7fe700 1 -- 172.21.15.98:0/3761137535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6380009550 con 0x7f638c0b87c0 2023-03-03T08:01:17.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.201+0000 7f638b7fe700 1 --2- 172.21.15.98:0/3761137535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638c0b87c0 0x7f638c0cbdb0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f6380009c40 tx=0x7f6380004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:17.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.202+0000 7f6388ff9700 1 -- 172.21.15.98:0/3761137535 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6380004510 con 0x7f638c0b87c0 2023-03-03T08:01:17.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.202+0000 7f6388ff9700 1 -- 172.21.15.98:0/3761137535 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638001e070 con 0x7f638c0b87c0 2023-03-03T08:01:17.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.202+0000 7f6388ff9700 1 -- 172.21.15.98:0/3761137535 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63800038e0 con 0x7f638c0b87c0 2023-03-03T08:01:17.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.202+0000 7f63923ff700 1 -- 172.21.15.98:0/3761137535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f638c0cdf30 con 0x7f638c0b87c0 2023-03-03T08:01:17.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.202+0000 7f63923ff700 1 -- 172.21.15.98:0/3761137535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f638c073a80 con 0x7f638c0b87c0 2023-03-03T08:01:17.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.203+0000 7f63923ff700 1 -- 172.21.15.98:0/3761137535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f638c0ce0c0 con 0x7f638c0b87c0 2023-03-03T08:01:17.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.204+0000 7f6388ff9700 1 -- 172.21.15.98:0/3761137535 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f638000f050 con 0x7f638c0b87c0 2023-03-03T08:01:17.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.206+0000 7f6388ff9700 1 --2- 172.21.15.98:0/3761137535 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63780684d0 0x7f637806a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:17.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.206+0000 7f6388ff9700 1 -- 172.21.15.98:0/3761137535 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6380020070 con 0x7f638c0b87c0 2023-03-03T08:01:17.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.206+0000 7f638bfff700 1 --2- 172.21.15.98:0/3761137535 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63780684d0 0x7f637806a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:17.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.207+0000 7f638bfff700 1 --2- 172.21.15.98:0/3761137535 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63780684d0 0x7f637806a980 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f637c0099d0 tx=0x7f637c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:17.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.210+0000 7f6388ff9700 1 -- 172.21.15.98:0/3761137535 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f63800539d0 con 0x7f638c0b87c0 2023-03-03T08:01:17.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.407+0000 7f63923ff700 1 -- 172.21.15.98:0/3761137535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f638c0729b0 con 0x7f638c0b87c0 2023-03-03T08:01:17.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.408+0000 7f6388ff9700 1 -- 172.21.15.98:0/3761137535 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+14268 (secure 0 0 0) 0x7f6380056ff0 con 0x7f638c0b87c0 2023-03-03T08:01:17.410 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:17.410 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":74,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:01:15.771969+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:01:17.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.411+0000 7f63923ff700 1 -- 172.21.15.98:0/3761137535 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63780684d0 msgr2=0x7f637806a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:17.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.411+0000 7f63923ff700 1 --2- 172.21.15.98:0/3761137535 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63780684d0 0x7f637806a980 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f637c0099d0 tx=0x7f637c008040 comp rx=0 tx=0).stop 2023-03-03T08:01:17.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.411+0000 7f63923ff700 1 -- 172.21.15.98:0/3761137535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638c0b87c0 msgr2=0x7f638c0cbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:17.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.411+0000 7f63923ff700 1 --2- 172.21.15.98:0/3761137535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638c0b87c0 0x7f638c0cbdb0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f6380009c40 tx=0x7f6380004750 comp rx=0 tx=0).stop 2023-03-03T08:01:17.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.411+0000 7f63923ff700 1 -- 172.21.15.98:0/3761137535 shutdown_connections 2023-03-03T08:01:17.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.411+0000 7f63923ff700 1 --2- 172.21.15.98:0/3761137535 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f63780684d0 0x7f637806a980 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.411+0000 7f63923ff700 1 --2- 172.21.15.98:0/3761137535 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f638c0b8b90 0x7f638c1e7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.411+0000 7f63923ff700 1 --2- 172.21.15.98:0/3761137535 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638c0b5240 0x7f638c0b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.411+0000 7f63923ff700 1 --2- 172.21.15.98:0/3761137535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638c0b87c0 0x7f638c0cbdb0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.411+0000 7f63923ff700 1 -- 172.21.15.98:0/3761137535 >> 172.21.15.98:0/3761137535 conn(0x7f638c06aee0 msgr2=0x7f638c0b36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:17.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.411+0000 7f63923ff700 1 -- 172.21.15.98:0/3761137535 shutdown_connections 2023-03-03T08:01:17.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.411+0000 7f63923ff700 1 -- 172.21.15.98:0/3761137535 wait complete. 2023-03-03T08:01:17.440 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:01:17.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:01:17.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.689+0000 7fdcc2b9d700 1 Processor -- start 2023-03-03T08:01:17.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.689+0000 7fdcc2b9d700 1 -- start start 2023-03-03T08:01:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.691+0000 7fdcc2b9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdcbc1609b0 0x7fdcbc160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.691+0000 7fdcc2b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcbc0b4aa0 0x7fdcbc0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.691+0000 7fdcc2b9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcbc1612c0 con 0x7fdcbc0cf570 2023-03-03T08:01:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.691+0000 7fdcc2b9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcbc0cebb0 con 0x7fdcbc1609b0 2023-03-03T08:01:17.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.691+0000 7fdcc2b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcbc0b5460 con 0x7fdcbc0b4aa0 2023-03-03T08:01:17.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.691+0000 7fdcc139a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdcbc0cf570 0x7fdcbc15d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52296/0 (socket says 172.21.15.98:52296) 2023-03-03T08:01:17.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.691+0000 7fdcc139a700 1 -- 172.21.15.98:0/4197778863 learned_addr learned my addr 172.21.15.98:0/4197778863 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:17.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.691+0000 7fdcc239c700 1 --2- 172.21.15.98:0/4197778863 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcbc0b4aa0 0x7fdcbc0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:17.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.691+0000 7fdcc1b9b700 1 --2- 172.21.15.98:0/4197778863 >> v2:172.21.15.100:3300/0 conn(0x7fdcbc1609b0 0x7fdcbc160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:17.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.692+0000 7fdcc0b99700 1 -- 172.21.15.98:0/4197778863 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970970571 0 0) 0x7fdcbc1612c0 con 0x7fdcbc0cf570 2023-03-03T08:01:17.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.692+0000 7fdcc0b99700 1 -- 172.21.15.98:0/4197778863 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca80035b0 con 0x7fdcbc0cf570 2023-03-03T08:01:17.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.692+0000 7fdcc1b9b700 1 -- 172.21.15.98:0/4197778863 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcbc0b4aa0 msgr2=0x7fdcbc0b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:17.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.692+0000 7fdcc1b9b700 1 --2- 172.21.15.98:0/4197778863 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcbc0b4aa0 0x7fdcbc0b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.692+0000 7fdcc1b9b700 1 -- 172.21.15.98:0/4197778863 >> v1:172.21.15.98:6789/0 conn(0x7fdcbc0cf570 legacy=0x7fdcbc15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:17.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.692+0000 7fdcc1b9b700 1 -- 172.21.15.98:0/4197778863 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcbc15dd00 con 0x7fdcbc1609b0 2023-03-03T08:01:17.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.692+0000 7fdcc239c700 1 --2- 172.21.15.98:0/4197778863 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcbc0b4aa0 0x7fdcbc0b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:17.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.692+0000 7fdcc1b9b700 1 --2- 172.21.15.98:0/4197778863 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdcbc1609b0 0x7fdcbc160d80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009840 tx=0x7fdcb8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2a2a3101c77bb2c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:17.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.693+0000 7fdcc0b99700 1 -- 172.21.15.98:0/4197778863 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8011830 con 0x7fdcbc1609b0 2023-03-03T08:01:17.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.693+0000 7fdcc0b99700 1 -- 172.21.15.98:0/4197778863 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb8011990 con 0x7fdcbc1609b0 2023-03-03T08:01:17.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.693+0000 7fdcc0b99700 1 -- 172.21.15.98:0/4197778863 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8011d00 con 0x7fdcbc1609b0 2023-03-03T08:01:17.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.693+0000 7fdcc2b9d700 1 -- 172.21.15.98:0/4197778863 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdcbc1609b0 msgr2=0x7fdcbc160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:17.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.693+0000 7fdcc2b9d700 1 --2- 172.21.15.98:0/4197778863 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdcbc1609b0 0x7fdcbc160d80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009840 tx=0x7fdcb8009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:17.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.693+0000 7fdcc2b9d700 1 -- 172.21.15.98:0/4197778863 shutdown_connections 2023-03-03T08:01:17.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.693+0000 7fdcc2b9d700 1 --2- 172.21.15.98:0/4197778863 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdcbc1609b0 0x7fdcbc160d80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.693+0000 7fdcc2b9d700 1 --2- 172.21.15.98:0/4197778863 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcbc0b4aa0 0x7fdcbc0b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.693+0000 7fdcc2b9d700 1 -- 172.21.15.98:0/4197778863 >> 172.21.15.98:0/4197778863 conn(0x7fdcbc06b880 msgr2=0x7fdcbc0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:17.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.693+0000 7fdcc2b9d700 1 -- 172.21.15.98:0/4197778863 shutdown_connections 2023-03-03T08:01:17.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.694+0000 7fdcc2b9d700 1 -- 172.21.15.98:0/4197778863 wait complete. 2023-03-03T08:01:17.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.696+0000 7fdcc2b9d700 1 Processor -- start 2023-03-03T08:01:17.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.697+0000 7fdcc2b9d700 1 -- start start 2023-03-03T08:01:17.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.698+0000 7fdcc2b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcbc0b4aa0 0x7fdcbc14ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:17.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.698+0000 7fdcc2b9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcbc0cf570 0x7fdcbc0c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:17.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.698+0000 7fdcc2b9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdcbc1609b0 0x7fdcbc152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:17.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.698+0000 7fdcc2b9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcbc0c7890 con 0x7fdcbc0cf570 2023-03-03T08:01:17.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.698+0000 7fdcc2b9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcbc0cca40 con 0x7fdcbc1609b0 2023-03-03T08:01:17.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.698+0000 7fdcc2b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcbc0cbe70 con 0x7fdcbc0b4aa0 2023-03-03T08:01:17.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.698+0000 7fdcc139a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcbc0cf570 0x7fdcbc0c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:17.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.698+0000 7fdcc139a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcbc0cf570 0x7fdcbc0c9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60368/0 (socket says 172.21.15.98:60368) 2023-03-03T08:01:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.698+0000 7fdcc139a700 1 -- 172.21.15.98:0/2128968023 learned_addr learned my addr 172.21.15.98:0/2128968023 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.699+0000 7fdcc239c700 1 --2- 172.21.15.98:0/2128968023 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdcbc1609b0 0x7fdcbc152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.699+0000 7fdcc1b9b700 1 --2- 172.21.15.98:0/2128968023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcbc0b4aa0 0x7fdcbc14ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.699+0000 7fdcc139a700 1 -- 172.21.15.98:0/2128968023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcbc0b4aa0 msgr2=0x7fdcbc14ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.699+0000 7fdcc139a700 1 --2- 172.21.15.98:0/2128968023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcbc0b4aa0 0x7fdcbc14ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.699+0000 7fdcc139a700 1 -- 172.21.15.98:0/2128968023 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdcbc1609b0 msgr2=0x7fdcbc152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.699+0000 7fdcc139a700 1 --2- 172.21.15.98:0/2128968023 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdcbc1609b0 0x7fdcbc152310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.699+0000 7fdcc139a700 1 -- 172.21.15.98:0/2128968023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb4009a70 con 0x7fdcbc0cf570 2023-03-03T08:01:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.699+0000 7fdcc139a700 1 --2- 172.21.15.98:0/2128968023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcbc0cf570 0x7fdcbc0c9320 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00c4d0 tx=0x7fdcac00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.699+0000 7fdcb27fc700 1 -- 172.21.15.98:0/2128968023 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac005d70 con 0x7fdcbc0cf570 2023-03-03T08:01:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.699+0000 7fdcb27fc700 1 -- 172.21.15.98:0/2128968023 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcac0070d0 con 0x7fdcbc0cf570 2023-03-03T08:01:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.699+0000 7fdcb27fc700 1 -- 172.21.15.98:0/2128968023 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac007440 con 0x7fdcbc0cf570 2023-03-03T08:01:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.699+0000 7fdcc2b9d700 1 -- 172.21.15.98:0/2128968023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb80094a0 con 0x7fdcbc0cf570 2023-03-03T08:01:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.700+0000 7fdcc2b9d700 1 -- 172.21.15.98:0/2128968023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcbc0cda60 con 0x7fdcbc0cf570 2023-03-03T08:01:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.701+0000 7fdcc2b9d700 1 -- 172.21.15.98:0/2128968023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcbc0cd780 con 0x7fdcbc0cf570 2023-03-03T08:01:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.702+0000 7fdcb27fc700 1 -- 172.21.15.98:0/2128968023 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fdcac019030 con 0x7fdcbc0cf570 2023-03-03T08:01:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.704+0000 7fdcb27fc700 1 --2- 172.21.15.98:0/2128968023 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdca00684a0 0x7fdca006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.704+0000 7fdcb27fc700 1 -- 172.21.15.98:0/2128968023 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdcac084e30 con 0x7fdcbc0cf570 2023-03-03T08:01:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.704+0000 7fdcc1b9b700 1 --2- 172.21.15.98:0/2128968023 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdca00684a0 0x7fdca006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.705+0000 7fdcc1b9b700 1 --2- 172.21.15.98:0/2128968023 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdca00684a0 0x7fdca006a950 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8000c00 tx=0x7fdcb801c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.709+0000 7fdcb27fc700 1 -- 172.21.15.98:0/2128968023 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdcac04b5e0 con 0x7fdcbc0cf570 2023-03-03T08:01:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.905+0000 7fdcc2b9d700 1 -- 172.21.15.98:0/2128968023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdcbc0cdd40 con 0x7fdcbc0cf570 2023-03-03T08:01:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.905+0000 7fdcb27fc700 1 -- 172.21.15.98:0/2128968023 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdcac017020 con 0x7fdcbc0cf570 2023-03-03T08:01:17.907 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:01:17.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.908+0000 7fdcc2b9d700 1 -- 172.21.15.98:0/2128968023 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdca00684a0 msgr2=0x7fdca006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:17.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.908+0000 7fdcc2b9d700 1 --2- 172.21.15.98:0/2128968023 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdca00684a0 0x7fdca006a950 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8000c00 tx=0x7fdcb801c040 comp rx=0 tx=0).stop 2023-03-03T08:01:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.908+0000 7fdcc2b9d700 1 -- 172.21.15.98:0/2128968023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcbc0cf570 msgr2=0x7fdcbc0c9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.908+0000 7fdcc2b9d700 1 --2- 172.21.15.98:0/2128968023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcbc0cf570 0x7fdcbc0c9320 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00c4d0 tx=0x7fdcac00c7e0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.908+0000 7fdcc2b9d700 1 -- 172.21.15.98:0/2128968023 shutdown_connections 2023-03-03T08:01:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.908+0000 7fdcc2b9d700 1 --2- 172.21.15.98:0/2128968023 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fdca00684a0 0x7fdca006a950 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.908+0000 7fdcc2b9d700 1 --2- 172.21.15.98:0/2128968023 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdcbc1609b0 0x7fdcbc152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.908+0000 7fdcc2b9d700 1 --2- 172.21.15.98:0/2128968023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdcbc0b4aa0 0x7fdcbc14ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.908+0000 7fdcc2b9d700 1 --2- 172.21.15.98:0/2128968023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcbc0cf570 0x7fdcbc0c9320 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:17.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.908+0000 7fdcc2b9d700 1 -- 172.21.15.98:0/2128968023 >> 172.21.15.98:0/2128968023 conn(0x7fdcbc06b880 msgr2=0x7fdcbc0b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:17.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.908+0000 7fdcc2b9d700 1 -- 172.21.15.98:0/2128968023 shutdown_connections 2023-03-03T08:01:17.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:17.908+0000 7fdcc2b9d700 1 -- 172.21.15.98:0/2128968023 wait complete. 2023-03-03T08:01:17.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:01:18.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.184+0000 7f700ab9d700 1 Processor -- start 2023-03-03T08:01:18.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.185+0000 7f700ab9d700 1 -- start start 2023-03-03T08:01:18.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.186+0000 7f700ab9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70040b8910 0x7f70041604a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:18.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.186+0000 7f700ab9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f700415ce00 con 0x7f70040b5e70 2023-03-03T08:01:18.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.186+0000 7f700ab9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70040d0bf0 con 0x7f70040b51b0 2023-03-03T08:01:18.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.186+0000 7f700ab9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70040b8ce0 con 0x7f70040b8910 2023-03-03T08:01:18.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.186+0000 7f700939a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f70040b5e70 0x7f70040b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52306/0 (socket says 172.21.15.98:52306) 2023-03-03T08:01:18.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.186+0000 7f700939a700 1 -- 172.21.15.98:0/4159789063 learned_addr learned my addr 172.21.15.98:0/4159789063 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:18.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.186+0000 7f7009b9b700 1 --2- 172.21.15.98:0/4159789063 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70040b8910 0x7f70041604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:18.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7008b99700 1 -- 172.21.15.98:0/4159789063 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889867464 0 0) 0x7f6ff4001110 con 0x7f70040b5e70 2023-03-03T08:01:18.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7008b99700 1 -- 172.21.15.98:0/4159789063 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff00035b0 con 0x7f70040b5e70 2023-03-03T08:01:18.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7008b99700 1 -- 172.21.15.98:0/4159789063 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3940564421 0 0) 0x7f6ff00035b0 con 0x7f70040b5e70 2023-03-03T08:01:18.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7008b99700 1 -- 172.21.15.98:0/4159789063 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70040b8910 msgr2=0x7f70041604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:18.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7008b99700 1 --2- 172.21.15.98:0/4159789063 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70040b8910 0x7f70041604a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7008b99700 1 -- 172.21.15.98:0/4159789063 >> v1:172.21.15.100:6789/0 conn(0x7f70040b51b0 legacy=0x7f70040d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:18.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7008b99700 1 -- 172.21.15.98:0/4159789063 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7004160cf0 con 0x7f70040b5e70 2023-03-03T08:01:18.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7008b99700 1 -- 172.21.15.98:0/4159789063 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f6ff4001800 con 0x7f70040b5e70 2023-03-03T08:01:18.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7009b9b700 1 --2- 172.21.15.98:0/4159789063 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70040b8910 0x7f70041604a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:18.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7008b99700 1 -- 172.21.15.98:0/4159789063 >> v1:172.21.15.98:6789/0 conn(0x7f70040b5e70 legacy=0x7f70040b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:18.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7008b99700 1 --2- 172.21.15.98:0/4159789063 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ff0003a30 0x7f6ff0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:18.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7008b99700 1 --2- 172.21.15.98:0/4159789063 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70040b8910 0x7f6ff0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:18.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7008b99700 1 --2- 172.21.15.98:0/4159789063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff0004d90 0x7f6ff0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:18.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7008b99700 1 -- 172.21.15.98:0/4159789063 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70040b8ce0 con 0x7f6ff0004d90 2023-03-03T08:01:18.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7008b99700 1 -- 172.21.15.98:0/4159789063 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff000bba0 con 0x7f6ff0003a30 2023-03-03T08:01:18.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.187+0000 7f7008b99700 1 -- 172.21.15.98:0/4159789063 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff000bd10 con 0x7f70040b8910 2023-03-03T08:01:18.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.188+0000 7f700a39c700 1 --2- 172.21.15.98:0/4159789063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff0004d90 0x7f6ff0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:18.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.188+0000 7f7009b9b700 1 --2- 172.21.15.98:0/4159789063 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ff0003a30 0x7f6ff0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:18.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.188+0000 7f700939a700 1 --2- 172.21.15.98:0/4159789063 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70040b8910 0x7f6ff0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:18.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.188+0000 7f700a39c700 1 -- 172.21.15.98:0/4159789063 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70040b8910 msgr2=0x7f6ff0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:18.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.188+0000 7f700a39c700 1 --2- 172.21.15.98:0/4159789063 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70040b8910 0x7f6ff0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.188+0000 7f700a39c700 1 -- 172.21.15.98:0/4159789063 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ff0003a30 msgr2=0x7f6ff0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:18.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.188+0000 7f700a39c700 1 --2- 172.21.15.98:0/4159789063 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ff0003a30 0x7f6ff0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.188+0000 7f700a39c700 1 -- 172.21.15.98:0/4159789063 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70000094a0 con 0x7f6ff0004d90 2023-03-03T08:01:18.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.189+0000 7f700a39c700 1 --2- 172.21.15.98:0/4159789063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff0004d90 0x7f6ff0005200 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f70040d0260 tx=0x7f6ff0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f2a283a14df554 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:18.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.189+0000 7f7008b99700 1 -- 172.21.15.98:0/4159789063 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc001e70 con 0x7f6ff0004d90 2023-03-03T08:01:18.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.189+0000 7f7008b99700 1 -- 172.21.15.98:0/4159789063 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ffc00cdf0 con 0x7f6ff0004d90 2023-03-03T08:01:18.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.189+0000 7f7008b99700 1 -- 172.21.15.98:0/4159789063 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc012660 con 0x7f6ff0004d90 2023-03-03T08:01:18.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.189+0000 7f700ab9d700 1 -- 172.21.15.98:0/4159789063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff0004d90 msgr2=0x7f6ff0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:18.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.189+0000 7f700ab9d700 1 --2- 172.21.15.98:0/4159789063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff0004d90 0x7f6ff0005200 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f70040d0260 tx=0x7f6ff0005880 comp rx=0 tx=0).stop 2023-03-03T08:01:18.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.189+0000 7f700a39c700 1 -- 172.21.15.98:0/4159789063 reap_dead start 2023-03-03T08:01:18.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.190+0000 7f700ab9d700 1 -- 172.21.15.98:0/4159789063 shutdown_connections 2023-03-03T08:01:18.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.190+0000 7f700ab9d700 1 -- 172.21.15.98:0/4159789063 >> 172.21.15.98:0/4159789063 conn(0x7f700406b880 msgr2=0x7f70040cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:18.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.190+0000 7f700ab9d700 1 -- 172.21.15.98:0/4159789063 shutdown_connections 2023-03-03T08:01:18.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.190+0000 7f700ab9d700 1 -- 172.21.15.98:0/4159789063 wait complete. 2023-03-03T08:01:18.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.192+0000 7f700ab9d700 1 Processor -- start 2023-03-03T08:01:18.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.193+0000 7f700ab9d700 1 -- start start 2023-03-03T08:01:18.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.194+0000 7f700ab9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6ff0004d90 0x7f70041f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:18.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.194+0000 7f700ab9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70041f1e10 0x7f70040b49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:18.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.194+0000 7f700ab9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70041f5550 0x7f70041f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:18.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.194+0000 7f700ab9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700414d250 con 0x7f70041f5550 2023-03-03T08:01:18.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.194+0000 7f700ab9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70041f60c0 con 0x7f70041f1e10 2023-03-03T08:01:18.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.194+0000 7f700ab9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70041f63a0 con 0x7f6ff0004d90 2023-03-03T08:01:18.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.195+0000 7f700a39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70041f5550 0x7f70041f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:18.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.195+0000 7f700a39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70041f5550 0x7f70041f59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60388/0 (socket says 172.21.15.98:60388) 2023-03-03T08:01:18.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.195+0000 7f700a39c700 1 -- 172.21.15.98:0/2535742256 learned_addr learned my addr 172.21.15.98:0/2535742256 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:18.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.195+0000 7f7009b9b700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6ff0004d90 0x7f70041f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:18.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.195+0000 7f700939a700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70041f1e10 0x7f70040b49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.195+0000 7f700a39c700 1 -- 172.21.15.98:0/2535742256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6ff0004d90 msgr2=0x7f70041f2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.195+0000 7f700a39c700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6ff0004d90 0x7f70041f2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.195+0000 7f700a39c700 1 -- 172.21.15.98:0/2535742256 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70041f1e10 msgr2=0x7f70040b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.195+0000 7f700a39c700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70041f1e10 0x7f70040b49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.195+0000 7f700a39c700 1 -- 172.21.15.98:0/2535742256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70041612b0 con 0x7f70041f5550 2023-03-03T08:01:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.195+0000 7f700939a700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70041f1e10 0x7f70040b49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.195+0000 7f7009b9b700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6ff0004d90 0x7f70041f2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.195+0000 7f700a39c700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70041f5550 0x7f70041f59d0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00057e0 tx=0x7f6ffc0015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.196+0000 7f6ffa7fc700 1 -- 172.21.15.98:0/2535742256 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc0050c0 con 0x7f70041f5550 2023-03-03T08:01:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.196+0000 7f6ffa7fc700 1 -- 172.21.15.98:0/2535742256 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ffc005220 con 0x7f70041f5550 2023-03-03T08:01:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.196+0000 7f6ffa7fc700 1 -- 172.21.15.98:0/2535742256 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc0055f0 con 0x7f70041f5550 2023-03-03T08:01:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.196+0000 7f700ab9d700 1 -- 172.21.15.98:0/2535742256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7004160fd0 con 0x7f70041f5550 2023-03-03T08:01:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.196+0000 7f700ab9d700 1 -- 172.21.15.98:0/2535742256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70040d0eb0 con 0x7f70041f5550 2023-03-03T08:01:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.197+0000 7f700ab9d700 1 -- 172.21.15.98:0/2535742256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70040743f0 con 0x7f70041f5550 2023-03-03T08:01:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.198+0000 7f6ffa7fc700 1 -- 172.21.15.98:0/2535742256 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f6ffc010050 con 0x7f70041f5550 2023-03-03T08:01:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.199+0000 7f6ffa7fc700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fe80684a0 0x7f6fe806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.199+0000 7f6ffa7fc700 1 -- 172.21.15.98:0/2535742256 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6ffc0870f0 con 0x7f70041f5550 2023-03-03T08:01:18.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.200+0000 7f7009b9b700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fe80684a0 0x7f6fe806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:18.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.200+0000 7f7009b9b700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fe80684a0 0x7f6fe806a950 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7000009df0 tx=0x7f7000008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:18.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.203+0000 7f6ffa7fc700 1 -- 172.21.15.98:0/2535742256 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f70040743f0 con 0x7f70041f5550 2023-03-03T08:01:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.400+0000 7f700ab9d700 1 -- 172.21.15.98:0/2535742256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f70040743f0 con 0x7f70041f5550 2023-03-03T08:01:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.401+0000 7f6ffa7fc700 1 -- 172.21.15.98:0/2535742256 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v74) v1 ==== 92+0+10 (secure 0 0 0) 0x7f70040743f0 con 0x7f70041f5550 2023-03-03T08:01:18.403 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:01:18.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.403+0000 7f700ab9d700 1 -- 172.21.15.98:0/2535742256 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fe80684a0 msgr2=0x7f6fe806a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.403+0000 7f700ab9d700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fe80684a0 0x7f6fe806a950 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7000009df0 tx=0x7f7000008040 comp rx=0 tx=0).stop 2023-03-03T08:01:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.403+0000 7f700ab9d700 1 -- 172.21.15.98:0/2535742256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70041f5550 msgr2=0x7f70041f59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.403+0000 7f700ab9d700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70041f5550 0x7f70041f59d0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00057e0 tx=0x7f6ffc0015b0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.403+0000 7f700ab9d700 1 -- 172.21.15.98:0/2535742256 shutdown_connections 2023-03-03T08:01:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.403+0000 7f700ab9d700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fe80684a0 0x7f6fe806a950 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.403+0000 7f700ab9d700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70041f5550 0x7f70041f59d0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.403+0000 7f700ab9d700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6ff0004d90 0x7f70041f2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.403+0000 7f700ab9d700 1 --2- 172.21.15.98:0/2535742256 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70041f1e10 0x7f70040b49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.403+0000 7f700ab9d700 1 -- 172.21.15.98:0/2535742256 >> 172.21.15.98:0/2535742256 conn(0x7f700406b880 msgr2=0x7f70040b6350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.403+0000 7f700ab9d700 1 -- 172.21.15.98:0/2535742256 shutdown_connections 2023-03-03T08:01:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.404+0000 7f700ab9d700 1 -- 172.21.15.98:0/2535742256 wait complete. 2023-03-03T08:01:18.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-03-03T08:01:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.676+0000 7fd7129ff700 1 Processor -- start 2023-03-03T08:01:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.676+0000 7fd7129ff700 1 -- start start 2023-03-03T08:01:18.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.677+0000 7fd7129ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd70c0d23e0 0x7fd70c0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:18.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.677+0000 7fd7129ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70c0b5b20 con 0x7fd70c0b8b90 2023-03-03T08:01:18.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.677+0000 7fd7129ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70c0d0ef0 con 0x7fd70c0b87c0 2023-03-03T08:01:18.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.677+0000 7fd7129ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd70c0d0600 con 0x7fd70c0d23e0 2023-03-03T08:01:18.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.678+0000 7fd70b7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd70c0b8b90 0x7fd70c0b5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52314/0 (socket says 172.21.15.98:52314) 2023-03-03T08:01:18.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.678+0000 7fd70b7fe700 1 -- 172.21.15.98:0/1691582341 learned_addr learned my addr 172.21.15.98:0/1691582341 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:18.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.678+0000 7fd710f9c700 1 --2- 172.21.15.98:0/1691582341 >> v2:172.21.15.100:3300/0 conn(0x7fd70c0d23e0 0x7fd70c0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:18.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.678+0000 7fd70affd700 1 -- 172.21.15.98:0/1691582341 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685575026 0 0) 0x7fd70c0b5b20 con 0x7fd70c0b8b90 2023-03-03T08:01:18.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.678+0000 7fd70affd700 1 -- 172.21.15.98:0/1691582341 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6f40035b0 con 0x7fd70c0b8b90 2023-03-03T08:01:18.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.678+0000 7fd70affd700 1 -- 172.21.15.98:0/1691582341 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089245640 0 0) 0x7fd70c0d0ef0 con 0x7fd70c0b87c0 2023-03-03T08:01:18.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.678+0000 7fd70affd700 1 -- 172.21.15.98:0/1691582341 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd70c0b5b20 con 0x7fd70c0b87c0 2023-03-03T08:01:18.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.678+0000 7fd710f9c700 1 -- 172.21.15.98:0/1691582341 >> v1:172.21.15.100:6789/0 conn(0x7fd70c0b87c0 legacy=0x7fd70c0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:18.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.678+0000 7fd710f9c700 1 -- 172.21.15.98:0/1691582341 >> v1:172.21.15.98:6789/0 conn(0x7fd70c0b8b90 legacy=0x7fd70c0b5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:18.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.678+0000 7fd710f9c700 1 -- 172.21.15.98:0/1691582341 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70c0cd970 con 0x7fd70c0d23e0 2023-03-03T08:01:18.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.679+0000 7fd710f9c700 1 --2- 172.21.15.98:0/1691582341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd70c0d23e0 0x7fd70c0cffe0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd704009d90 tx=0x7fd70400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5af431ed40249cc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:18.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.679+0000 7fd70affd700 1 -- 172.21.15.98:0/1691582341 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70400e070 con 0x7fd70c0d23e0 2023-03-03T08:01:18.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.679+0000 7fd70affd700 1 -- 172.21.15.98:0/1691582341 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70400bb20 con 0x7fd70c0d23e0 2023-03-03T08:01:18.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.679+0000 7fd70affd700 1 -- 172.21.15.98:0/1691582341 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd704012430 con 0x7fd70c0d23e0 2023-03-03T08:01:18.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.680+0000 7fd7129ff700 1 -- 172.21.15.98:0/1691582341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd70c0d23e0 msgr2=0x7fd70c0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:18.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.680+0000 7fd7129ff700 1 --2- 172.21.15.98:0/1691582341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd70c0d23e0 0x7fd70c0cffe0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd704009d90 tx=0x7fd70400b330 comp rx=0 tx=0).stop 2023-03-03T08:01:18.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.680+0000 7fd7129ff700 1 -- 172.21.15.98:0/1691582341 shutdown_connections 2023-03-03T08:01:18.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.680+0000 7fd7129ff700 1 --2- 172.21.15.98:0/1691582341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd70c0d23e0 0x7fd70c0cffe0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.680+0000 7fd7129ff700 1 -- 172.21.15.98:0/1691582341 >> 172.21.15.98:0/1691582341 conn(0x7fd70c06aee0 msgr2=0x7fd70c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:18.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.680+0000 7fd7129ff700 1 -- 172.21.15.98:0/1691582341 shutdown_connections 2023-03-03T08:01:18.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.680+0000 7fd7129ff700 1 -- 172.21.15.98:0/1691582341 wait complete. 2023-03-03T08:01:18.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.682+0000 7fd7129ff700 1 Processor -- start 2023-03-03T08:01:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.682+0000 7fd7129ff700 1 -- start start 2023-03-03T08:01:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.683+0000 7fd7129ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd70c0b87c0 0x7fd70c156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.683+0000 7fd7129ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd70c0b8b90 0x7fd70c1f5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.683+0000 7fd7129ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd70c0d23e0 0x7fd70c15e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.683+0000 7fd7129ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70c1f6480 con 0x7fd70c0d23e0 2023-03-03T08:01:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.683+0000 7fd7129ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70c1577f0 con 0x7fd70c0b8b90 2023-03-03T08:01:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.683+0000 7fd7129ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70c157230 con 0x7fd70c0b87c0 2023-03-03T08:01:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.683+0000 7fd710f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd70c0d23e0 0x7fd70c15e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.683+0000 7fd710f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd70c0d23e0 0x7fd70c15e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60390/0 (socket says 172.21.15.98:60390) 2023-03-03T08:01:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.683+0000 7fd710f9c700 1 -- 172.21.15.98:0/4225956583 learned_addr learned my addr 172.21.15.98:0/4225956583 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.684+0000 7fd70bfff700 1 --2- 172.21.15.98:0/4225956583 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd70c0b87c0 0x7fd70c156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.684+0000 7fd70b7fe700 1 --2- 172.21.15.98:0/4225956583 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd70c0b8b90 0x7fd70c1f5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.684+0000 7fd710f9c700 1 -- 172.21.15.98:0/4225956583 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd70c0b87c0 msgr2=0x7fd70c156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.684+0000 7fd710f9c700 1 --2- 172.21.15.98:0/4225956583 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd70c0b87c0 0x7fd70c156ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.684+0000 7fd710f9c700 1 -- 172.21.15.98:0/4225956583 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd70c0b8b90 msgr2=0x7fd70c1f5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.684+0000 7fd710f9c700 1 --2- 172.21.15.98:0/4225956583 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd70c0b8b90 0x7fd70c1f5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.684+0000 7fd710f9c700 1 -- 172.21.15.98:0/4225956583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd704009a70 con 0x7fd70c0d23e0 2023-03-03T08:01:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.684+0000 7fd70b7fe700 1 --2- 172.21.15.98:0/4225956583 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd70c0b8b90 0x7fd70c1f5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.684+0000 7fd710f9c700 1 --2- 172.21.15.98:0/4225956583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd70c0d23e0 0x7fd70c15e510 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0d0a60 tx=0x7fd704004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.685+0000 7fd708ff9700 1 -- 172.21.15.98:0/4225956583 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70401b810 con 0x7fd70c0d23e0 2023-03-03T08:01:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.685+0000 7fd708ff9700 1 -- 172.21.15.98:0/4225956583 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd704020070 con 0x7fd70c0d23e0 2023-03-03T08:01:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.685+0000 7fd708ff9700 1 -- 172.21.15.98:0/4225956583 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70401bbb0 con 0x7fd70c0d23e0 2023-03-03T08:01:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.685+0000 7fd7129ff700 1 -- 172.21.15.98:0/4225956583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70c073a80 con 0x7fd70c0d23e0 2023-03-03T08:01:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.685+0000 7fd7129ff700 1 -- 172.21.15.98:0/4225956583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd70c0cdc50 con 0x7fd70c0d23e0 2023-03-03T08:01:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.686+0000 7fd7129ff700 1 -- 172.21.15.98:0/4225956583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd70c0cdf30 con 0x7fd70c0d23e0 2023-03-03T08:01:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.687+0000 7fd708ff9700 1 -- 172.21.15.98:0/4225956583 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fd70402e070 con 0x7fd70c0d23e0 2023-03-03T08:01:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.688+0000 7fd708ff9700 1 --2- 172.21.15.98:0/4225956583 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd6f80684a0 0x7fd6f806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.688+0000 7fd708ff9700 1 -- 172.21.15.98:0/4225956583 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd704022070 con 0x7fd70c0d23e0 2023-03-03T08:01:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.689+0000 7fd70bfff700 1 --2- 172.21.15.98:0/4225956583 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd6f80684a0 0x7fd6f806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.689+0000 7fd70bfff700 1 --2- 172.21.15.98:0/4225956583 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd6f80684a0 0x7fd6f806a950 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc007fd0 tx=0x7fd6fc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:18.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.693+0000 7fd708ff9700 1 -- 172.21.15.98:0/4225956583 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd704054640 con 0x7fd70c0d23e0 2023-03-03T08:01:18.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.889+0000 7fd7129ff700 1 -- 172.21.15.98:0/4225956583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd70c0729b0 con 0x7fd70c0d23e0 2023-03-03T08:01:18.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.890+0000 7fd708ff9700 1 -- 172.21.15.98:0/4225956583 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v74) v1 ==== 84+0+126 (secure 0 0 0) 0x7fd70402c020 con 0x7fd70c0d23e0 2023-03-03T08:01:18.891 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:18.892 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T08:01:18.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:18.892 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2023-03-03T08:01:18.892 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2023-03-03T08:01:18.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:18.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:18.892 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 2, 2023-03-03T08:01:18.893 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": "rbd" 2023-03-03T08:01:18.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:18.893 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T08:01:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.892+0000 7fd7129ff700 1 -- 172.21.15.98:0/4225956583 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd6f80684a0 msgr2=0x7fd6f806a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.892+0000 7fd7129ff700 1 --2- 172.21.15.98:0/4225956583 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd6f80684a0 0x7fd6f806a950 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc007fd0 tx=0x7fd6fc009040 comp rx=0 tx=0).stop 2023-03-03T08:01:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.892+0000 7fd7129ff700 1 -- 172.21.15.98:0/4225956583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd70c0d23e0 msgr2=0x7fd70c15e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.892+0000 7fd7129ff700 1 --2- 172.21.15.98:0/4225956583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd70c0d23e0 0x7fd70c15e510 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0d0a60 tx=0x7fd704004ad0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.892+0000 7fd7129ff700 1 -- 172.21.15.98:0/4225956583 shutdown_connections 2023-03-03T08:01:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.892+0000 7fd7129ff700 1 --2- 172.21.15.98:0/4225956583 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd6f80684a0 0x7fd6f806a950 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.892+0000 7fd7129ff700 1 --2- 172.21.15.98:0/4225956583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd70c0d23e0 0x7fd70c15e510 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.892+0000 7fd7129ff700 1 --2- 172.21.15.98:0/4225956583 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd70c0b87c0 0x7fd70c156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.892+0000 7fd7129ff700 1 --2- 172.21.15.98:0/4225956583 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd70c0b8b90 0x7fd70c1f5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:18.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.892+0000 7fd7129ff700 1 -- 172.21.15.98:0/4225956583 >> 172.21.15.98:0/4225956583 conn(0x7fd70c06aee0 msgr2=0x7fd70c0b1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:18.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.892+0000 7fd7129ff700 1 -- 172.21.15.98:0/4225956583 shutdown_connections 2023-03-03T08:01:18.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:18.892+0000 7fd7129ff700 1 -- 172.21.15.98:0/4225956583 wait complete. 2023-03-03T08:01:18.920 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-03-03T08:01:18.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-03-03T08:01:19.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.166+0000 7f24d299d700 1 Processor -- start 2023-03-03T08:01:19.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.166+0000 7f24d299d700 1 -- start start 2023-03-03T08:01:19.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.168+0000 7f24d299d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24cc0b3550 0x7f24cc0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:19.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.168+0000 7f24d299d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24cc0b3c80 con 0x7f24cc0cee50 2023-03-03T08:01:19.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.168+0000 7f24d299d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24cc1615a0 con 0x7f24cc073ed0 2023-03-03T08:01:19.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.168+0000 7f24d299d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc0b4370 con 0x7f24cc0b3550 2023-03-03T08:01:19.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.168+0000 7f24d219c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f24cc0cee50 0x7f24cc160e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52318/0 (socket says 172.21.15.98:52318) 2023-03-03T08:01:19.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.168+0000 7f24d219c700 1 -- 172.21.15.98:0/2842626042 learned_addr learned my addr 172.21.15.98:0/2842626042 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:19.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.168+0000 7f24d199b700 1 --2- 172.21.15.98:0/2842626042 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24cc0b3550 0x7f24cc0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:19.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.169+0000 7f24d0999700 1 -- 172.21.15.98:0/2842626042 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420415093 0 0) 0x7f24cc0b3c80 con 0x7f24cc0cee50 2023-03-03T08:01:19.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.169+0000 7f24d0999700 1 -- 172.21.15.98:0/2842626042 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24b80035b0 con 0x7f24cc0cee50 2023-03-03T08:01:19.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.169+0000 7f24d199b700 1 -- 172.21.15.98:0/2842626042 >> v1:172.21.15.100:6789/0 conn(0x7f24cc073ed0 legacy=0x7f24cc160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:19.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.169+0000 7f24d199b700 1 -- 172.21.15.98:0/2842626042 >> v1:172.21.15.98:6789/0 conn(0x7f24cc0cee50 legacy=0x7f24cc160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:19.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.169+0000 7f24d199b700 1 -- 172.21.15.98:0/2842626042 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24cc0b3ed0 con 0x7f24cc0b3550 2023-03-03T08:01:19.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.169+0000 7f24d199b700 1 --2- 172.21.15.98:0/2842626042 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24cc0b3550 0x7f24cc0cfa00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f24c8009870 tx=0x7f24c8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55f61762c71bca64 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:19.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.169+0000 7f24d0999700 1 -- 172.21.15.98:0/2842626042 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c8011830 con 0x7f24cc0b3550 2023-03-03T08:01:19.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.170+0000 7f24d0999700 1 -- 172.21.15.98:0/2842626042 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c8011990 con 0x7f24cc0b3550 2023-03-03T08:01:19.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.170+0000 7f24d0999700 1 -- 172.21.15.98:0/2842626042 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c8011d20 con 0x7f24cc0b3550 2023-03-03T08:01:19.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.170+0000 7f24d299d700 1 -- 172.21.15.98:0/2842626042 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24cc0b3550 msgr2=0x7f24cc0cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:19.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.170+0000 7f24d299d700 1 --2- 172.21.15.98:0/2842626042 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24cc0b3550 0x7f24cc0cfa00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f24c8009870 tx=0x7f24c8009b80 comp rx=0 tx=0).stop 2023-03-03T08:01:19.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.170+0000 7f24d299d700 1 -- 172.21.15.98:0/2842626042 shutdown_connections 2023-03-03T08:01:19.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.170+0000 7f24d299d700 1 --2- 172.21.15.98:0/2842626042 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24cc0b3550 0x7f24cc0cfa00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:19.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.170+0000 7f24d299d700 1 -- 172.21.15.98:0/2842626042 >> 172.21.15.98:0/2842626042 conn(0x7f24cc073b00 msgr2=0x7f24cc0d2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:19.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.170+0000 7f24d299d700 1 -- 172.21.15.98:0/2842626042 shutdown_connections 2023-03-03T08:01:19.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.170+0000 7f24d299d700 1 -- 172.21.15.98:0/2842626042 wait complete. 2023-03-03T08:01:19.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.172+0000 7f24d299d700 1 Processor -- start 2023-03-03T08:01:19.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.173+0000 7f24d299d700 1 -- start start 2023-03-03T08:01:19.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.174+0000 7f24d299d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc073ed0 0x7f24cc0c76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:19.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.174+0000 7f24d299d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24cc0b3550 0x7f24cc0c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:19.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.174+0000 7f24d299d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24cc0cee50 0x7f24cc0b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:19.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.174+0000 7f24d299d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc0c95a0 con 0x7f24cc073ed0 2023-03-03T08:01:19.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.174+0000 7f24d299d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc0c9820 con 0x7f24cc0cee50 2023-03-03T08:01:19.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.174+0000 7f24d299d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc0afe20 con 0x7f24cc0b3550 2023-03-03T08:01:19.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.174+0000 7f24d199b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc073ed0 0x7f24cc0c76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:19.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.174+0000 7f24d199b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc073ed0 0x7f24cc0c76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60406/0 (socket says 172.21.15.98:60406) 2023-03-03T08:01:19.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.174+0000 7f24d199b700 1 -- 172.21.15.98:0/1142008833 learned_addr learned my addr 172.21.15.98:0/1142008833 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:19.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.174+0000 7f24d199b700 1 -- 172.21.15.98:0/1142008833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24cc0b3550 msgr2=0x7f24cc0c9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:19.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.174+0000 7f24d119a700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24cc0b3550 0x7f24cc0c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:19.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.174+0000 7f24d219c700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24cc0cee50 0x7f24cc0b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:19.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.175+0000 7f24d199b700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24cc0b3550 0x7f24cc0c9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:19.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.175+0000 7f24d199b700 1 -- 172.21.15.98:0/1142008833 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24cc0cee50 msgr2=0x7f24cc0b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:19.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.175+0000 7f24d199b700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24cc0cee50 0x7f24cc0b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:19.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.175+0000 7f24d199b700 1 -- 172.21.15.98:0/1142008833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c80094a0 con 0x7f24cc073ed0 2023-03-03T08:01:19.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.175+0000 7f24d119a700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24cc0b3550 0x7f24cc0c9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:19.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.175+0000 7f24d219c700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24cc0cee50 0x7f24cc0b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.175+0000 7f24d199b700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc073ed0 0x7f24cc0c76d0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f24c80090b0 tx=0x7f24c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.175+0000 7f24c27fc700 1 -- 172.21.15.98:0/1142008833 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c80038a0 con 0x7f24cc073ed0 2023-03-03T08:01:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.175+0000 7f24c27fc700 1 -- 172.21.15.98:0/1142008833 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c801f070 con 0x7f24cc073ed0 2023-03-03T08:01:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.175+0000 7f24c27fc700 1 -- 172.21.15.98:0/1142008833 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c8003c00 con 0x7f24cc073ed0 2023-03-03T08:01:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.175+0000 7f24d299d700 1 -- 172.21.15.98:0/1142008833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24cc0cd920 con 0x7f24cc073ed0 2023-03-03T08:01:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.176+0000 7f24d299d700 1 -- 172.21.15.98:0/1142008833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24cc0cd640 con 0x7f24cc073ed0 2023-03-03T08:01:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.177+0000 7f24d299d700 1 -- 172.21.15.98:0/1142008833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24cc0d3d60 con 0x7f24cc073ed0 2023-03-03T08:01:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.178+0000 7f24c27fc700 1 -- 172.21.15.98:0/1142008833 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f24c800f050 con 0x7f24cc073ed0 2023-03-03T08:01:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.179+0000 7f24c27fc700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f24b00684a0 0x7f24b006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.179+0000 7f24c27fc700 1 -- 172.21.15.98:0/1142008833 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f24c8021070 con 0x7f24cc073ed0 2023-03-03T08:01:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.179+0000 7f24d119a700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f24b00684a0 0x7f24b006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.180+0000 7f24d119a700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f24b00684a0 0x7f24b006a950 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f24cc0a0f30 tx=0x7f24bc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:19.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.183+0000 7f24c27fc700 1 -- 172.21.15.98:0/1142008833 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f24c8053f60 con 0x7f24cc073ed0 2023-03-03T08:01:19.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.376+0000 7f24d299d700 1 -- 172.21.15.98:0/1142008833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f24cc068290 con 0x7f24cc073ed0 2023-03-03T08:01:19.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.814+0000 7f24c27fc700 1 -- 172.21.15.98:0/1142008833 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v75) v1 ==== 138+0+0 (secure 0 0 0) 0x7f24c80150b0 con 0x7f24cc073ed0 2023-03-03T08:01:19.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.818+0000 7f24d299d700 1 -- 172.21.15.98:0/1142008833 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f24b00684a0 msgr2=0x7f24b006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.818+0000 7f24d299d700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f24b00684a0 0x7f24b006a950 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f24cc0a0f30 tx=0x7f24bc009040 comp rx=0 tx=0).stop 2023-03-03T08:01:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.818+0000 7f24d299d700 1 -- 172.21.15.98:0/1142008833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc073ed0 msgr2=0x7f24cc0c76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.818+0000 7f24d299d700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc073ed0 0x7f24cc0c76d0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f24c80090b0 tx=0x7f24c8004750 comp rx=0 tx=0).stop 2023-03-03T08:01:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.818+0000 7f24d299d700 1 -- 172.21.15.98:0/1142008833 shutdown_connections 2023-03-03T08:01:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.818+0000 7f24d299d700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f24b00684a0 0x7f24b006a950 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.818+0000 7f24d299d700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc073ed0 0x7f24cc0c76d0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.818+0000 7f24d299d700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24cc0cee50 0x7f24cc0b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.818+0000 7f24d299d700 1 --2- 172.21.15.98:0/1142008833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24cc0b3550 0x7f24cc0c9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.818+0000 7f24d299d700 1 -- 172.21.15.98:0/1142008833 >> 172.21.15.98:0/1142008833 conn(0x7f24cc073b00 msgr2=0x7f24cc15c080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.819+0000 7f24d299d700 1 -- 172.21.15.98:0/1142008833 shutdown_connections 2023-03-03T08:01:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:19.819+0000 7f24d299d700 1 -- 172.21.15.98:0/1142008833 wait complete. 2023-03-03T08:01:19.825 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2023-03-03T08:01:19.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-03-03T08:01:20.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.106+0000 7f0beffff700 1 Processor -- start 2023-03-03T08:01:20.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.106+0000 7f0beffff700 1 -- start start 2023-03-03T08:01:20.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.107+0000 7f0beffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0be815a890 0x7f0be8071b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:20.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.107+0000 7f0beffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0be80c93d0 0x7f0be80b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:20.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.107+0000 7f0beffff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be80c9940 con 0x7f0be80b0800 2023-03-03T08:01:20.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.107+0000 7f0beffff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0be80c9c20 con 0x7f0be815a890 2023-03-03T08:01:20.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.107+0000 7f0beffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be80c9f00 con 0x7f0be80c93d0 2023-03-03T08:01:20.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.108+0000 7f0bedd9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0be815a890 0x7f0be8071b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:20.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.108+0000 7f0bedd9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0be815a890 0x7f0be8071b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60412/0 (socket says 172.21.15.98:60412) 2023-03-03T08:01:20.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.108+0000 7f0bedd9b700 1 -- 172.21.15.98:0/61511463 learned_addr learned my addr 172.21.15.98:0/61511463 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:20.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.108+0000 7f0bed59a700 1 --2- 172.21.15.98:0/61511463 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0be80c93d0 0x7f0be80b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:20.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.108+0000 7f0bedd9b700 1 -- 172.21.15.98:0/61511463 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0be80c93d0 msgr2=0x7f0be80b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:20.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.108+0000 7f0bedd9b700 1 --2- 172.21.15.98:0/61511463 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0be80c93d0 0x7f0be80b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:20.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.108+0000 7f0bedd9b700 1 -- 172.21.15.98:0/61511463 >> v1:172.21.15.98:6789/0 conn(0x7f0be80b0800 legacy=0x7f0be80ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:20.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.108+0000 7f0bedd9b700 1 -- 172.21.15.98:0/61511463 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be80ca1e0 con 0x7f0be815a890 2023-03-03T08:01:20.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.108+0000 7f0becd99700 1 -- 172.21.15.98:0/61511463 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847643234 0 0) 0x7f0be80c9940 con 0x7f0be80b0800 2023-03-03T08:01:20.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.109+0000 7f0bedd9b700 1 --2- 172.21.15.98:0/61511463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be815a890 0x7f0be8071b40 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f0be40098f0 tx=0x7f0be4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d440c0e6fd7161fb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:20.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.109+0000 7f0becd99700 1 -- 172.21.15.98:0/61511463 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4011830 con 0x7f0be815a890 2023-03-03T08:01:20.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.109+0000 7f0becd99700 1 -- 172.21.15.98:0/61511463 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be4011990 con 0x7f0be815a890 2023-03-03T08:01:20.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.109+0000 7f0becd99700 1 -- 172.21.15.98:0/61511463 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4011d20 con 0x7f0be815a890 2023-03-03T08:01:20.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.109+0000 7f0beffff700 1 -- 172.21.15.98:0/61511463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be815a890 msgr2=0x7f0be8071b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:20.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.109+0000 7f0beffff700 1 --2- 172.21.15.98:0/61511463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be815a890 0x7f0be8071b40 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f0be40098f0 tx=0x7f0be4009c00 comp rx=0 tx=0).stop 2023-03-03T08:01:20.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.110+0000 7f0beffff700 1 -- 172.21.15.98:0/61511463 shutdown_connections 2023-03-03T08:01:20.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.110+0000 7f0beffff700 1 --2- 172.21.15.98:0/61511463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be815a890 0x7f0be8071b40 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:20.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.110+0000 7f0beffff700 1 --2- 172.21.15.98:0/61511463 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0be80c93d0 0x7f0be80b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:20.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.110+0000 7f0beffff700 1 -- 172.21.15.98:0/61511463 >> 172.21.15.98:0/61511463 conn(0x7f0be80cb800 msgr2=0x7f0be80cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:20.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.110+0000 7f0beffff700 1 -- 172.21.15.98:0/61511463 shutdown_connections 2023-03-03T08:01:20.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.110+0000 7f0beffff700 1 -- 172.21.15.98:0/61511463 wait complete. 2023-03-03T08:01:20.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.112+0000 7f0beffff700 1 Processor -- start 2023-03-03T08:01:20.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.112+0000 7f0beffff700 1 -- start start 2023-03-03T08:01:20.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.113+0000 7f0beffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0be80b0800 0x7f0be80b2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:20.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.113+0000 7f0beffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be80c93d0 0x7f0be80cde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:20.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.113+0000 7f0beffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be815a890 0x7f0be80ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:20.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.113+0000 7f0beffff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be80c6fd0 con 0x7f0be815a890 2023-03-03T08:01:20.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.113+0000 7f0beffff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be80c7200 con 0x7f0be80c93d0 2023-03-03T08:01:20.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.113+0000 7f0beffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be80c74e0 con 0x7f0be80b0800 2023-03-03T08:01:20.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.113+0000 7f0bee59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be815a890 0x7f0be80ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:20.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.113+0000 7f0bee59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be815a890 0x7f0be80ce3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60416/0 (socket says 172.21.15.98:60416) 2023-03-03T08:01:20.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.113+0000 7f0bee59c700 1 -- 172.21.15.98:0/2235049993 learned_addr learned my addr 172.21.15.98:0/2235049993 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:20.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.114+0000 7f0bed59a700 1 --2- 172.21.15.98:0/2235049993 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be80c93d0 0x7f0be80cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:20.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.114+0000 7f0bedd9b700 1 --2- 172.21.15.98:0/2235049993 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0be80b0800 0x7f0be80b2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:20.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.114+0000 7f0bee59c700 1 -- 172.21.15.98:0/2235049993 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0be80b0800 msgr2=0x7f0be80b2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:20.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.114+0000 7f0bee59c700 1 --2- 172.21.15.98:0/2235049993 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0be80b0800 0x7f0be80b2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:20.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.114+0000 7f0bee59c700 1 -- 172.21.15.98:0/2235049993 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be80c93d0 msgr2=0x7f0be80cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:20.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.114+0000 7f0bee59c700 1 --2- 172.21.15.98:0/2235049993 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be80c93d0 0x7f0be80cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:20.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.114+0000 7f0bee59c700 1 -- 172.21.15.98:0/2235049993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be4009550 con 0x7f0be815a890 2023-03-03T08:01:20.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.114+0000 7f0bee59c700 1 --2- 172.21.15.98:0/2235049993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be815a890 0x7f0be80ce3b0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f0be0001560 tx=0x7f0be000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:20.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.114+0000 7f0bde7fc700 1 -- 172.21.15.98:0/2235049993 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be0003e30 con 0x7f0be815a890 2023-03-03T08:01:20.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.114+0000 7f0bde7fc700 1 -- 172.21.15.98:0/2235049993 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be0005070 con 0x7f0be815a890 2023-03-03T08:01:20.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.114+0000 7f0beffff700 1 -- 172.21.15.98:0/2235049993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be8072080 con 0x7f0be815a890 2023-03-03T08:01:20.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.115+0000 7f0bde7fc700 1 -- 172.21.15.98:0/2235049993 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be0005420 con 0x7f0be815a890 2023-03-03T08:01:20.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.115+0000 7f0beffff700 1 -- 172.21.15.98:0/2235049993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be80ca4c0 con 0x7f0be815a890 2023-03-03T08:01:20.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.116+0000 7f0beffff700 1 -- 172.21.15.98:0/2235049993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be80b2110 con 0x7f0be815a890 2023-03-03T08:01:20.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.118+0000 7f0bde7fc700 1 -- 172.21.15.98:0/2235049993 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f0be0019030 con 0x7f0be815a890 2023-03-03T08:01:20.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.120+0000 7f0bde7fc700 1 --2- 172.21.15.98:0/2235049993 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0bcc0684a0 0x7f0bcc06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:20.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.120+0000 7f0bde7fc700 1 -- 172.21.15.98:0/2235049993 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5580+0+0 (secure 0 0 0) 0x7f0be00856d0 con 0x7f0be815a890 2023-03-03T08:01:20.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.120+0000 7f0bedd9b700 1 --2- 172.21.15.98:0/2235049993 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0bcc0684a0 0x7f0bcc06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:20.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.121+0000 7f0bedd9b700 1 --2- 172.21.15.98:0/2235049993 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0bcc0684a0 0x7f0bcc06a950 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0be4014040 tx=0x7f0be4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:20.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.123+0000 7f0bde7fc700 1 -- 172.21.15.98:0/2235049993 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0be0052100 con 0x7f0be815a890 2023-03-03T08:01:20.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.318+0000 7f0beffff700 1 -- 172.21.15.98:0/2235049993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0be804e6e0 con 0x7f0be815a890 2023-03-03T08:01:20.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.829+0000 7f0bde7fc700 1 -- 172.21.15.98:0/2235049993 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v76) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0be0083e60 con 0x7f0be815a890 2023-03-03T08:01:20.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.839+0000 7f0beffff700 1 -- 172.21.15.98:0/2235049993 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0bcc0684a0 msgr2=0x7f0bcc06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:20.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.839+0000 7f0beffff700 1 --2- 172.21.15.98:0/2235049993 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0bcc0684a0 0x7f0bcc06a950 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0be4014040 tx=0x7f0be4008030 comp rx=0 tx=0).stop 2023-03-03T08:01:20.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.839+0000 7f0beffff700 1 -- 172.21.15.98:0/2235049993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be815a890 msgr2=0x7f0be80ce3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:20.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.839+0000 7f0beffff700 1 --2- 172.21.15.98:0/2235049993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be815a890 0x7f0be80ce3b0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f0be0001560 tx=0x7f0be000cc00 comp rx=0 tx=0).stop 2023-03-03T08:01:20.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.840+0000 7f0beffff700 1 -- 172.21.15.98:0/2235049993 shutdown_connections 2023-03-03T08:01:20.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.840+0000 7f0beffff700 1 --2- 172.21.15.98:0/2235049993 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0bcc0684a0 0x7f0bcc06a950 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:20.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.840+0000 7f0beffff700 1 --2- 172.21.15.98:0/2235049993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be815a890 0x7f0be80ce3b0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:20.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.840+0000 7f0beffff700 1 --2- 172.21.15.98:0/2235049993 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0be80b0800 0x7f0be80b2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:20.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.840+0000 7f0beffff700 1 --2- 172.21.15.98:0/2235049993 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be80c93d0 0x7f0be80cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:20.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.840+0000 7f0beffff700 1 -- 172.21.15.98:0/2235049993 >> 172.21.15.98:0/2235049993 conn(0x7f0be80cb800 msgr2=0x7f0be80ca8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:20.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.841+0000 7f0beffff700 1 -- 172.21.15.98:0/2235049993 shutdown_connections 2023-03-03T08:01:20.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:20.843+0000 7f0beffff700 1 -- 172.21.15.98:0/2235049993 wait complete. 2023-03-03T08:01:20.851 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2023-03-03T08:01:20.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-03-03T08:01:21.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.124+0000 7f7008b9d700 1 Processor -- start 2023-03-03T08:01:21.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.124+0000 7f7008b9d700 1 -- start start 2023-03-03T08:01:21.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.125+0000 7f7008b9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70040cf570 0x7f700415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:21.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.125+0000 7f7008b9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70040cd780 con 0x7f700415db30 2023-03-03T08:01:21.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.125+0000 7f7008b9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70040cda60 con 0x7f70041609b0 2023-03-03T08:01:21.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.125+0000 7f7008b9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70040cebb0 con 0x7f70040cf570 2023-03-03T08:01:21.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.125+0000 7f6ffbfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70040cf570 0x7f700415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.126+0000 7f6ffbfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70040cf570 0x7f700415d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60430/0 (socket says 172.21.15.98:60430) 2023-03-03T08:01:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.126+0000 7f6ffbfff700 1 -- 172.21.15.98:0/930387347 learned_addr learned my addr 172.21.15.98:0/930387347 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.126+0000 7f7003fff700 1 --1- 172.21.15.98:0/930387347 >> v1:172.21.15.98:6789/0 conn(0x7f700415db30 0x7f70040b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52330/0 (socket says 172.21.15.98:52330) 2023-03-03T08:01:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.126+0000 7f6ffbfff700 1 -- 172.21.15.98:0/930387347 >> v1:172.21.15.100:6789/0 conn(0x7f70041609b0 legacy=0x7f7004160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.126+0000 7f6ffbfff700 1 -- 172.21.15.98:0/930387347 >> v1:172.21.15.98:6789/0 conn(0x7f700415db30 legacy=0x7f70040b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.126+0000 7f6ffbfff700 1 -- 172.21.15.98:0/930387347 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70040cdd40 con 0x7f70040cf570 2023-03-03T08:01:21.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.127+0000 7f6ffbfff700 1 --2- 172.21.15.98:0/930387347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70040cf570 0x7f700415d5f0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0098f0 tx=0x7f6fec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6523cfee8fc66d9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:21.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.127+0000 7f7002ffd700 1 -- 172.21.15.98:0/930387347 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec00b8a0 con 0x7f70040cf570 2023-03-03T08:01:21.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.127+0000 7f7002ffd700 1 -- 172.21.15.98:0/930387347 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec00ba00 con 0x7f70040cf570 2023-03-03T08:01:21.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.127+0000 7f7002ffd700 1 -- 172.21.15.98:0/930387347 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec00bd90 con 0x7f70040cf570 2023-03-03T08:01:21.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.127+0000 7f7008b9d700 1 -- 172.21.15.98:0/930387347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70040cf570 msgr2=0x7f700415d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:21.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.127+0000 7f7008b9d700 1 --2- 172.21.15.98:0/930387347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70040cf570 0x7f700415d5f0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0098f0 tx=0x7f6fec009c00 comp rx=0 tx=0).stop 2023-03-03T08:01:21.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.127+0000 7f7008b9d700 1 -- 172.21.15.98:0/930387347 shutdown_connections 2023-03-03T08:01:21.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.127+0000 7f7008b9d700 1 --2- 172.21.15.98:0/930387347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70040cf570 0x7f700415d5f0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.128+0000 7f7008b9d700 1 -- 172.21.15.98:0/930387347 >> 172.21.15.98:0/930387347 conn(0x7f700406b880 msgr2=0x7f70040cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:21.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.128+0000 7f7008b9d700 1 -- 172.21.15.98:0/930387347 shutdown_connections 2023-03-03T08:01:21.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.128+0000 7f7008b9d700 1 -- 172.21.15.98:0/930387347 wait complete. 2023-03-03T08:01:21.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.130+0000 7f7008b9d700 1 Processor -- start 2023-03-03T08:01:21.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.130+0000 7f7008b9d700 1 -- start start 2023-03-03T08:01:21.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.131+0000 7f7008b9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70040cf570 0x7f70040b83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:21.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.131+0000 7f7008b9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f700415db30 0x7f70040b8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:21.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.131+0000 7f7008b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70041609b0 0x7f70040c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:21.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.131+0000 7f7008b9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70040c6cf0 con 0x7f700415db30 2023-03-03T08:01:21.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.131+0000 7f7008b9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70040c6fd0 con 0x7f70040cf570 2023-03-03T08:01:21.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.131+0000 7f7008b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70040c72b0 con 0x7f70041609b0 2023-03-03T08:01:21.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.131+0000 7f6ffbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f700415db30 0x7f70040b8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:21.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.131+0000 7f6ffbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f700415db30 0x7f70040b8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60434/0 (socket says 172.21.15.98:60434) 2023-03-03T08:01:21.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.131+0000 7f6ffbfff700 1 -- 172.21.15.98:0/1111629180 learned_addr learned my addr 172.21.15.98:0/1111629180 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:21.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.131+0000 7f70037fe700 1 --2- 172.21.15.98:0/1111629180 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70040cf570 0x7f70040b83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:21.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.132+0000 7f7003fff700 1 --2- 172.21.15.98:0/1111629180 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70041609b0 0x7f70040c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:21.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.132+0000 7f6ffbfff700 1 -- 172.21.15.98:0/1111629180 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70041609b0 msgr2=0x7f70040c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:21.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.132+0000 7f6ffbfff700 1 --2- 172.21.15.98:0/1111629180 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70041609b0 0x7f70040c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.132+0000 7f6ffbfff700 1 -- 172.21.15.98:0/1111629180 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70040cf570 msgr2=0x7f70040b83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:21.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.132+0000 7f6ffbfff700 1 --2- 172.21.15.98:0/1111629180 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70040cf570 0x7f70040b83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.132+0000 7f6ffbfff700 1 -- 172.21.15.98:0/1111629180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec009550 con 0x7f700415db30 2023-03-03T08:01:21.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.132+0000 7f6ffbfff700 1 --2- 172.21.15.98:0/1111629180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f700415db30 0x7f70040b8910 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f6fec000c00 tx=0x7f6fec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:21.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.132+0000 7f7000ff9700 1 -- 172.21.15.98:0/1111629180 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec0045b0 con 0x7f700415db30 2023-03-03T08:01:21.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.132+0000 7f7000ff9700 1 -- 172.21.15.98:0/1111629180 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec01f070 con 0x7f700415db30 2023-03-03T08:01:21.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.132+0000 7f7000ff9700 1 -- 172.21.15.98:0/1111629180 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec0038a0 con 0x7f700415db30 2023-03-03T08:01:21.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.132+0000 7f7008b9d700 1 -- 172.21.15.98:0/1111629180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f700415d0d0 con 0x7f700415db30 2023-03-03T08:01:21.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.133+0000 7f7008b9d700 1 -- 172.21.15.98:0/1111629180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f700415cdf0 con 0x7f700415db30 2023-03-03T08:01:21.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.134+0000 7f7008b9d700 1 -- 172.21.15.98:0/1111629180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70040b51b0 con 0x7f700415db30 2023-03-03T08:01:21.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.135+0000 7f7000ff9700 1 -- 172.21.15.98:0/1111629180 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f6fec02a020 con 0x7f700415db30 2023-03-03T08:01:21.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.136+0000 7f7000ff9700 1 --2- 172.21.15.98:0/1111629180 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fe40684a0 0x7f6fe406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:21.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.136+0000 7f7000ff9700 1 -- 172.21.15.98:0/1111629180 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f6fec0875c0 con 0x7f700415db30 2023-03-03T08:01:21.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.136+0000 7f70037fe700 1 --2- 172.21.15.98:0/1111629180 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fe40684a0 0x7f6fe406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:21.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.137+0000 7f70037fe700 1 --2- 172.21.15.98:0/1111629180 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fe40684a0 0x7f6fe406a950 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f700415bee0 tx=0x7f6ff4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:21.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.141+0000 7f7000ff9700 1 -- 172.21.15.98:0/1111629180 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6fec053ce0 con 0x7f700415db30 2023-03-03T08:01:21.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.378+0000 7f7008b9d700 1 -- 172.21.15.98:0/1111629180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f70040d00b0 con 0x7f700415db30 2023-03-03T08:01:21.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.386+0000 7f7000ff9700 1 -- 172.21.15.98:0/1111629180 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v36) v1 ==== 174+0+0 (secure 0 0 0) 0x7f6fec057300 con 0x7f700415db30 2023-03-03T08:01:21.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.393+0000 7f7008b9d700 1 -- 172.21.15.98:0/1111629180 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fe40684a0 msgr2=0x7f6fe406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:21.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.393+0000 7f7008b9d700 1 --2- 172.21.15.98:0/1111629180 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fe40684a0 0x7f6fe406a950 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f700415bee0 tx=0x7f6ff4009040 comp rx=0 tx=0).stop 2023-03-03T08:01:21.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.393+0000 7f7008b9d700 1 -- 172.21.15.98:0/1111629180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f700415db30 msgr2=0x7f70040b8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:21.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.393+0000 7f7008b9d700 1 --2- 172.21.15.98:0/1111629180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f700415db30 0x7f70040b8910 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f6fec000c00 tx=0x7f6fec004750 comp rx=0 tx=0).stop 2023-03-03T08:01:21.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.396+0000 7f7008b9d700 1 -- 172.21.15.98:0/1111629180 shutdown_connections 2023-03-03T08:01:21.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.396+0000 7f7008b9d700 1 --2- 172.21.15.98:0/1111629180 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6fe40684a0 0x7f6fe406a950 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.396+0000 7f7008b9d700 1 --2- 172.21.15.98:0/1111629180 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70041609b0 0x7f70040c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.396+0000 7f7008b9d700 1 --2- 172.21.15.98:0/1111629180 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70040cf570 0x7f70040b83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.396+0000 7f7008b9d700 1 --2- 172.21.15.98:0/1111629180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f700415db30 0x7f70040b8910 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.396+0000 7f7008b9d700 1 -- 172.21.15.98:0/1111629180 >> 172.21.15.98:0/1111629180 conn(0x7f700406b880 msgr2=0x7f700415e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:21.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.402+0000 7f7008b9d700 1 -- 172.21.15.98:0/1111629180 shutdown_connections 2023-03-03T08:01:21.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.403+0000 7f7008b9d700 1 -- 172.21.15.98:0/1111629180 wait complete. 2023-03-03T08:01:21.417 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 11 and data pool 12 2023-03-03T08:01:21.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:01:21.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.689+0000 7fb9c5bff700 1 Processor -- start 2023-03-03T08:01:21.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.689+0000 7fb9c5bff700 1 -- start start 2023-03-03T08:01:21.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.690+0000 7fb9c5bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb9c015c8d0 0x7fb9c00b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:21.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.690+0000 7fb9c5bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb9c00d1270 0x7fb9c00d1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:21.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.690+0000 7fb9c5bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c00d0550 con 0x7fb9c0160ea0 2023-03-03T08:01:21.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.690+0000 7fb9c5bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c00d0e50 con 0x7fb9c00d1270 2023-03-03T08:01:21.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.690+0000 7fb9c5bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c01613a0 con 0x7fb9c015c8d0 2023-03-03T08:01:21.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.691+0000 7fb9b7fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb9c0160ea0 0x7fb9c00b51d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52336/0 (socket says 172.21.15.98:52336) 2023-03-03T08:01:21.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.691+0000 7fb9b7fff700 1 -- 172.21.15.98:0/2626006497 learned_addr learned my addr 172.21.15.98:0/2626006497 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:21.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.691+0000 7fb9bffff700 1 --2- 172.21.15.98:0/2626006497 >> v2:172.21.15.98:3300/0 conn(0x7fb9c00d1270 0x7fb9c00d1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:21.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.691+0000 7fb9bf7fe700 1 --2- 172.21.15.98:0/2626006497 >> v2:172.21.15.100:3300/0 conn(0x7fb9c015c8d0 0x7fb9c00b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:21.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.691+0000 7fb9bffff700 1 -- 172.21.15.98:0/2626006497 >> v2:172.21.15.100:3300/0 conn(0x7fb9c015c8d0 msgr2=0x7fb9c00b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:21.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.691+0000 7fb9bffff700 1 --2- 172.21.15.98:0/2626006497 >> v2:172.21.15.100:3300/0 conn(0x7fb9c015c8d0 0x7fb9c00b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.691+0000 7fb9bffff700 1 -- 172.21.15.98:0/2626006497 >> v1:172.21.15.98:6789/0 conn(0x7fb9c0160ea0 legacy=0x7fb9c00b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.691+0000 7fb9bffff700 1 -- 172.21.15.98:0/2626006497 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c00cd900 con 0x7fb9c00d1270 2023-03-03T08:01:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.691+0000 7fb9bf7fe700 1 --2- 172.21.15.98:0/2626006497 >> v2:172.21.15.100:3300/0 conn(0x7fb9c015c8d0 0x7fb9c00b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.691+0000 7fb9beffd700 1 -- 172.21.15.98:0/2626006497 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499840163 0 0) 0x7fb9c00d0550 con 0x7fb9c0160ea0 2023-03-03T08:01:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.692+0000 7fb9bffff700 1 --2- 172.21.15.98:0/2626006497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c00d1270 0x7fb9c00d1670 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb9c006d620 tx=0x7fb9b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3370594b5c2c2a63 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.692+0000 7fb9beffd700 1 -- 172.21.15.98:0/2626006497 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b800e070 con 0x7fb9c00d1270 2023-03-03T08:01:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.692+0000 7fb9beffd700 1 -- 172.21.15.98:0/2626006497 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b800bbd0 con 0x7fb9c00d1270 2023-03-03T08:01:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.692+0000 7fb9beffd700 1 -- 172.21.15.98:0/2626006497 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8012430 con 0x7fb9c00d1270 2023-03-03T08:01:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.692+0000 7fb9c5bff700 1 -- 172.21.15.98:0/2626006497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c00d1270 msgr2=0x7fb9c00d1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.692+0000 7fb9c5bff700 1 --2- 172.21.15.98:0/2626006497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c00d1270 0x7fb9c00d1670 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb9c006d620 tx=0x7fb9b800b330 comp rx=0 tx=0).stop 2023-03-03T08:01:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.693+0000 7fb9c5bff700 1 -- 172.21.15.98:0/2626006497 shutdown_connections 2023-03-03T08:01:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.693+0000 7fb9c5bff700 1 --2- 172.21.15.98:0/2626006497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c00d1270 0x7fb9c00d1670 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.693+0000 7fb9c5bff700 1 --2- 172.21.15.98:0/2626006497 >> v2:172.21.15.100:3300/0 conn(0x7fb9c015c8d0 0x7fb9c00b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.693+0000 7fb9c5bff700 1 -- 172.21.15.98:0/2626006497 >> 172.21.15.98:0/2626006497 conn(0x7fb9c0073e70 msgr2=0x7fb9c006ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.693+0000 7fb9c5bff700 1 -- 172.21.15.98:0/2626006497 shutdown_connections 2023-03-03T08:01:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.693+0000 7fb9c5bff700 1 -- 172.21.15.98:0/2626006497 wait complete. 2023-03-03T08:01:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.695+0000 7fb9c5bff700 1 Processor -- start 2023-03-03T08:01:21.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.695+0000 7fb9c5bff700 1 -- start start 2023-03-03T08:01:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.696+0000 7fb9c5bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9c00d1270 0x7fb9c014f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.696+0000 7fb9c5bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9c015c8d0 0x7fb9c015f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:21.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.696+0000 7fb9c5bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c0160ea0 0x7fb9c00b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:21.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.696+0000 7fb9c5bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c015e370 con 0x7fb9c0160ea0 2023-03-03T08:01:21.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.696+0000 7fb9c5bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c015e630 con 0x7fb9c00d1270 2023-03-03T08:01:21.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.696+0000 7fb9c5bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c015e910 con 0x7fb9c015c8d0 2023-03-03T08:01:21.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.696+0000 7fb9bffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c0160ea0 0x7fb9c00b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:21.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.696+0000 7fb9bffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c0160ea0 0x7fb9c00b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60448/0 (socket says 172.21.15.98:60448) 2023-03-03T08:01:21.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.697+0000 7fb9bffff700 1 -- 172.21.15.98:0/4214862520 learned_addr learned my addr 172.21.15.98:0/4214862520 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:21.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.697+0000 7fb9bf7fe700 1 --2- 172.21.15.98:0/4214862520 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9c00d1270 0x7fb9c014f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:21.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.697+0000 7fb9b7fff700 1 --2- 172.21.15.98:0/4214862520 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9c015c8d0 0x7fb9c015f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:21.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.697+0000 7fb9bffff700 1 -- 172.21.15.98:0/4214862520 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9c015c8d0 msgr2=0x7fb9c015f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:21.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.697+0000 7fb9bffff700 1 --2- 172.21.15.98:0/4214862520 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9c015c8d0 0x7fb9c015f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.697+0000 7fb9bffff700 1 -- 172.21.15.98:0/4214862520 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9c00d1270 msgr2=0x7fb9c014f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:21.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.697+0000 7fb9bffff700 1 --2- 172.21.15.98:0/4214862520 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9c00d1270 0x7fb9c014f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.697+0000 7fb9bffff700 1 -- 172.21.15.98:0/4214862520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b8009a70 con 0x7fb9c0160ea0 2023-03-03T08:01:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.697+0000 7fb9bffff700 1 --2- 172.21.15.98:0/4214862520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c0160ea0 0x7fb9c00b4970 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00b5e00 tx=0x7fb9b8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.698+0000 7fb9bcff9700 1 -- 172.21.15.98:0/4214862520 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8012430 con 0x7fb9c0160ea0 2023-03-03T08:01:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.698+0000 7fb9bcff9700 1 -- 172.21.15.98:0/4214862520 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b8020070 con 0x7fb9c0160ea0 2023-03-03T08:01:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.698+0000 7fb9bcff9700 1 -- 172.21.15.98:0/4214862520 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8004280 con 0x7fb9c0160ea0 2023-03-03T08:01:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.698+0000 7fb9c5bff700 1 -- 172.21.15.98:0/4214862520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c00cdec0 con 0x7fb9c0160ea0 2023-03-03T08:01:21.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.698+0000 7fb9c5bff700 1 -- 172.21.15.98:0/4214862520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c00cdbe0 con 0x7fb9c0160ea0 2023-03-03T08:01:21.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.699+0000 7fb9c5bff700 1 -- 172.21.15.98:0/4214862520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c015cca0 con 0x7fb9c0160ea0 2023-03-03T08:01:21.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.700+0000 7fb9bcff9700 1 -- 172.21.15.98:0/4214862520 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fb9b802e070 con 0x7fb9c0160ea0 2023-03-03T08:01:21.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.702+0000 7fb9bcff9700 1 --2- 172.21.15.98:0/4214862520 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb9a00684a0 0x7fb9a006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:21.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.702+0000 7fb9bcff9700 1 -- 172.21.15.98:0/4214862520 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb9b8022070 con 0x7fb9c0160ea0 2023-03-03T08:01:21.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.702+0000 7fb9bf7fe700 1 --2- 172.21.15.98:0/4214862520 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb9a00684a0 0x7fb9a006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:21.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.703+0000 7fb9bf7fe700 1 --2- 172.21.15.98:0/4214862520 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb9a00684a0 0x7fb9a006a950 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb9b00099d0 tx=0x7fb9b0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:21.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.706+0000 7fb9bcff9700 1 -- 172.21.15.98:0/4214862520 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb9b8054280 con 0x7fb9c0160ea0 2023-03-03T08:01:21.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.901+0000 7fb9c5bff700 1 -- 172.21.15.98:0/4214862520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9c004e6e0 con 0x7fb9c0160ea0 2023-03-03T08:01:21.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.902+0000 7fb9bcff9700 1 -- 172.21.15.98:0/4214862520 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+17995 (secure 0 0 0) 0x7fb9b80578a0 con 0x7fb9c0160ea0 2023-03-03T08:01:21.904 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:21.904 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":77,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:01:21.379287+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:01:19.439394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-03-03T08:01:20.381345+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:01:21.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.904+0000 7fb9c5bff700 1 -- 172.21.15.98:0/4214862520 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb9a00684a0 msgr2=0x7fb9a006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:21.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.904+0000 7fb9c5bff700 1 --2- 172.21.15.98:0/4214862520 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb9a00684a0 0x7fb9a006a950 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb9b00099d0 tx=0x7fb9b0008040 comp rx=0 tx=0).stop 2023-03-03T08:01:21.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.904+0000 7fb9c5bff700 1 -- 172.21.15.98:0/4214862520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c0160ea0 msgr2=0x7fb9c00b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:21.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.904+0000 7fb9c5bff700 1 --2- 172.21.15.98:0/4214862520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c0160ea0 0x7fb9c00b4970 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00b5e00 tx=0x7fb9b8004ad0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.905+0000 7fb9c5bff700 1 -- 172.21.15.98:0/4214862520 shutdown_connections 2023-03-03T08:01:21.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.905+0000 7fb9c5bff700 1 --2- 172.21.15.98:0/4214862520 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb9a00684a0 0x7fb9a006a950 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.905+0000 7fb9c5bff700 1 --2- 172.21.15.98:0/4214862520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c0160ea0 0x7fb9c00b4970 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.905+0000 7fb9c5bff700 1 --2- 172.21.15.98:0/4214862520 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9c00d1270 0x7fb9c014f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.905+0000 7fb9c5bff700 1 --2- 172.21.15.98:0/4214862520 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9c015c8d0 0x7fb9c015f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:21.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.905+0000 7fb9c5bff700 1 -- 172.21.15.98:0/4214862520 >> 172.21.15.98:0/4214862520 conn(0x7fb9c0073e70 msgr2=0x7fb9c015b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:21.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.905+0000 7fb9c5bff700 1 -- 172.21.15.98:0/4214862520 shutdown_connections 2023-03-03T08:01:21.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:21.905+0000 7fb9c5bff700 1 -- 172.21.15.98:0/4214862520 wait complete. 2023-03-03T08:01:21.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:01:22.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.179+0000 7ff0d9fff700 1 Processor -- start 2023-03-03T08:01:22.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.179+0000 7ff0d9fff700 1 -- start start 2023-03-03T08:01:22.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.180+0000 7ff0d9fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0d40b87c0 0x7ff0d40cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.180+0000 7ff0d9fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff0d40b5410 0x7ff0d40d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.180+0000 7ff0d9fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d40d0ef0 con 0x7ff0d40b8b90 2023-03-03T08:01:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.180+0000 7ff0d9fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d40d0600 con 0x7ff0d40b5410 2023-03-03T08:01:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.180+0000 7ff0d9fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40cd970 con 0x7ff0d40b87c0 2023-03-03T08:01:22.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.181+0000 7ff0d3fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff0d40b5410 0x7ff0d40d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:22.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.181+0000 7ff0d37fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0d40b87c0 0x7ff0d40cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:22.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.181+0000 7ff0cbfff700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ff0d40b8b90 0x7ff0d40b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:37702/0 (socket says 172.21.15.98:37702) 2023-03-03T08:01:22.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.181+0000 7ff0cbfff700 1 -- 172.21.15.98:0/1024977801 learned_addr learned my addr 172.21.15.98:0/1024977801 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:22.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.181+0000 7ff0d37fe700 1 -- 172.21.15.98:0/1024977801 >> v2:172.21.15.98:3300/0 conn(0x7ff0d40b5410 msgr2=0x7ff0d40d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:22.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.181+0000 7ff0d37fe700 1 --2- 172.21.15.98:0/1024977801 >> v2:172.21.15.98:3300/0 conn(0x7ff0d40b5410 0x7ff0d40d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:22.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.181+0000 7ff0d37fe700 1 -- 172.21.15.98:0/1024977801 >> v1:172.21.15.100:6789/0 conn(0x7ff0d40b8b90 legacy=0x7ff0d40b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:22.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.181+0000 7ff0d2ffd700 1 -- 172.21.15.98:0/1024977801 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963768588 0 0) 0x7ff0d40d0ef0 con 0x7ff0d40b8b90 2023-03-03T08:01:22.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.181+0000 7ff0d37fe700 1 -- 172.21.15.98:0/1024977801 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d40cdc50 con 0x7ff0d40b87c0 2023-03-03T08:01:22.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.181+0000 7ff0d3fff700 1 --2- 172.21.15.98:0/1024977801 >> v2:172.21.15.98:3300/0 conn(0x7ff0d40b5410 0x7ff0d40d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:22.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.182+0000 7ff0d37fe700 1 --2- 172.21.15.98:0/1024977801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0d40b87c0 0x7ff0d40cffe0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009840 tx=0x7ff0c4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7430763fe286aded server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:22.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.182+0000 7ff0d2ffd700 1 -- 172.21.15.98:0/1024977801 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4011830 con 0x7ff0d40b87c0 2023-03-03T08:01:22.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.182+0000 7ff0d2ffd700 1 -- 172.21.15.98:0/1024977801 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c4011990 con 0x7ff0d40b87c0 2023-03-03T08:01:22.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.182+0000 7ff0d2ffd700 1 -- 172.21.15.98:0/1024977801 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4011d20 con 0x7ff0d40b87c0 2023-03-03T08:01:22.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.182+0000 7ff0d9fff700 1 -- 172.21.15.98:0/1024977801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0d40b87c0 msgr2=0x7ff0d40cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:22.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.182+0000 7ff0d9fff700 1 --2- 172.21.15.98:0/1024977801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0d40b87c0 0x7ff0d40cffe0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009840 tx=0x7ff0c4009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:22.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.183+0000 7ff0d9fff700 1 -- 172.21.15.98:0/1024977801 shutdown_connections 2023-03-03T08:01:22.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.183+0000 7ff0d9fff700 1 --2- 172.21.15.98:0/1024977801 >> v2:172.21.15.98:3300/0 conn(0x7ff0d40b5410 0x7ff0d40d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:22.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.183+0000 7ff0d9fff700 1 --2- 172.21.15.98:0/1024977801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0d40b87c0 0x7ff0d40cffe0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:22.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.183+0000 7ff0d9fff700 1 -- 172.21.15.98:0/1024977801 >> 172.21.15.98:0/1024977801 conn(0x7ff0d406aee0 msgr2=0x7ff0d406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:22.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.183+0000 7ff0d9fff700 1 -- 172.21.15.98:0/1024977801 shutdown_connections 2023-03-03T08:01:22.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.183+0000 7ff0d9fff700 1 -- 172.21.15.98:0/1024977801 wait complete. 2023-03-03T08:01:22.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.185+0000 7ff0d9fff700 1 Processor -- start 2023-03-03T08:01:22.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.185+0000 7ff0d9fff700 1 -- start start 2023-03-03T08:01:22.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.186+0000 7ff0d9fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d40b5410 0x7ff0d40c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.186+0000 7ff0d9fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0d40b87c0 0x7ff0d40c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.186+0000 7ff0d9fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0d40b8b90 0x7ff0d415b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.186+0000 7ff0d9fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40c9950 con 0x7ff0d40b5410 2023-03-03T08:01:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.186+0000 7ff0d9fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d41f2140 con 0x7ff0d40b8b90 2023-03-03T08:01:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.186+0000 7ff0d9fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d415bb20 con 0x7ff0d40b87c0 2023-03-03T08:01:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.186+0000 7ff0d37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d40b5410 0x7ff0d40c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.186+0000 7ff0d37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d40b5410 0x7ff0d40c6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60460/0 (socket says 172.21.15.98:60460) 2023-03-03T08:01:22.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.186+0000 7ff0d37fe700 1 -- 172.21.15.98:0/72434017 learned_addr learned my addr 172.21.15.98:0/72434017 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:22.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.187+0000 7ff0cbfff700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0d40b87c0 0x7ff0d40c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:22.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.187+0000 7ff0d3fff700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0d40b8b90 0x7ff0d415b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:22.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.187+0000 7ff0d37fe700 1 -- 172.21.15.98:0/72434017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0d40b87c0 msgr2=0x7ff0d40c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:22.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.187+0000 7ff0d37fe700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0d40b87c0 0x7ff0d40c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:22.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.187+0000 7ff0d37fe700 1 -- 172.21.15.98:0/72434017 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0d40b8b90 msgr2=0x7ff0d415b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:22.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.187+0000 7ff0d37fe700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0d40b8b90 0x7ff0d415b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:22.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.187+0000 7ff0d37fe700 1 -- 172.21.15.98:0/72434017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0cc009a70 con 0x7ff0d40b5410 2023-03-03T08:01:22.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.187+0000 7ff0cbfff700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0d40b87c0 0x7ff0d40c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:22.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.187+0000 7ff0d3fff700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0d40b8b90 0x7ff0d415b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:22.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.187+0000 7ff0d37fe700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d40b5410 0x7ff0d40c6db0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009fd0 tx=0x7ff0c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.187+0000 7ff0d0ff9700 1 -- 172.21.15.98:0/72434017 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c40038a0 con 0x7ff0d40b5410 2023-03-03T08:01:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.187+0000 7ff0d0ff9700 1 -- 172.21.15.98:0/72434017 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c401f070 con 0x7ff0d40b5410 2023-03-03T08:01:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.188+0000 7ff0d9fff700 1 -- 172.21.15.98:0/72434017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c40094a0 con 0x7ff0d40b5410 2023-03-03T08:01:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.188+0000 7ff0d0ff9700 1 -- 172.21.15.98:0/72434017 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4003c40 con 0x7ff0d40b5410 2023-03-03T08:01:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.188+0000 7ff0d9fff700 1 -- 172.21.15.98:0/72434017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d40cdf30 con 0x7ff0d40b5410 2023-03-03T08:01:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.189+0000 7ff0d9fff700 1 -- 172.21.15.98:0/72434017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d40ce0c0 con 0x7ff0d40b5410 2023-03-03T08:01:22.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.190+0000 7ff0d0ff9700 1 -- 172.21.15.98:0/72434017 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7ff0c400f050 con 0x7ff0d40b5410 2023-03-03T08:01:22.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.191+0000 7ff0d0ff9700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff0b40684a0 0x7ff0b406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:22.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.191+0000 7ff0d0ff9700 1 -- 172.21.15.98:0/72434017 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff0c4021070 con 0x7ff0d40b5410 2023-03-03T08:01:22.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.192+0000 7ff0cbfff700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff0b40684a0 0x7ff0b406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:22.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.192+0000 7ff0cbfff700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff0b40684a0 0x7ff0b406a950 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff0d40cd390 tx=0x7ff0bc007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:22.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.195+0000 7ff0d0ff9700 1 -- 172.21.15.98:0/72434017 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff0c4053f10 con 0x7ff0d40b5410 2023-03-03T08:01:22.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.397+0000 7ff0d9fff700 1 -- 172.21.15.98:0/72434017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0d40729b0 con 0x7ff0d40b5410 2023-03-03T08:01:22.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.398+0000 7ff0d0ff9700 1 -- 172.21.15.98:0/72434017 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+17982 (secure 0 0 0) 0x7ff0c4057530 con 0x7ff0d40b5410 2023-03-03T08:01:22.400 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:22.400 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":78,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:01:22.383102+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:01:19.439394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-03-03T08:01:20.381345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:01:22.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.401+0000 7ff0d9fff700 1 -- 172.21.15.98:0/72434017 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff0b40684a0 msgr2=0x7ff0b406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.401+0000 7ff0d9fff700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff0b40684a0 0x7ff0b406a950 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff0d40cd390 tx=0x7ff0bc007d30 comp rx=0 tx=0).stop 2023-03-03T08:01:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.401+0000 7ff0d9fff700 1 -- 172.21.15.98:0/72434017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d40b5410 msgr2=0x7ff0d40c6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.401+0000 7ff0d9fff700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d40b5410 0x7ff0d40c6db0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009fd0 tx=0x7ff0c4004750 comp rx=0 tx=0).stop 2023-03-03T08:01:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.401+0000 7ff0d9fff700 1 -- 172.21.15.98:0/72434017 shutdown_connections 2023-03-03T08:01:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.401+0000 7ff0d9fff700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff0b40684a0 0x7ff0b406a950 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:22.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.401+0000 7ff0d9fff700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d40b5410 0x7ff0d40c6db0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:22.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.401+0000 7ff0d9fff700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0d40b8b90 0x7ff0d415b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:22.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.401+0000 7ff0d9fff700 1 --2- 172.21.15.98:0/72434017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0d40b87c0 0x7ff0d40c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:22.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.401+0000 7ff0d9fff700 1 -- 172.21.15.98:0/72434017 >> 172.21.15.98:0/72434017 conn(0x7ff0d406aee0 msgr2=0x7ff0d40b4150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:22.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.401+0000 7ff0d9fff700 1 -- 172.21.15.98:0/72434017 shutdown_connections 2023-03-03T08:01:22.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.401+0000 7ff0d9fff700 1 -- 172.21.15.98:0/72434017 wait complete. 2023-03-03T08:01:22.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-03-03T08:01:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.678+0000 7f79ad1ff700 1 Processor -- start 2023-03-03T08:01:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.678+0000 7f79ad1ff700 1 -- start start 2023-03-03T08:01:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.679+0000 7f79ad1ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f79a806b4b0 0x7f79a80cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.679+0000 7f79ad1ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a80cf980 con 0x7f79a815ca70 2023-03-03T08:01:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.680+0000 7f79ad1ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a80d1800 con 0x7f79a80d0270 2023-03-03T08:01:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.680+0000 7f79ad1ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79a80cd5b0 con 0x7f79a806b4b0 2023-03-03T08:01:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.680+0000 7f79a6d9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f79a815ca70 0x7f79a80cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52344/0 (socket says 172.21.15.98:52344) 2023-03-03T08:01:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.680+0000 7f79a6d9d700 1 -- 172.21.15.98:0/219269145 learned_addr learned my addr 172.21.15.98:0/219269145 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.680+0000 7f79a759e700 1 --2- 172.21.15.98:0/219269145 >> v2:172.21.15.100:3300/0 conn(0x7f79a806b4b0 0x7f79a80cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.680+0000 7f79a5d9b700 1 -- 172.21.15.98:0/219269145 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770383468 0 0) 0x7f79a80cf980 con 0x7f79a815ca70 2023-03-03T08:01:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.680+0000 7f79a5d9b700 1 -- 172.21.15.98:0/219269145 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79900035b0 con 0x7f79a815ca70 2023-03-03T08:01:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a5d9b700 1 -- 172.21.15.98:0/219269145 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 472975555 0 0) 0x7f79900035b0 con 0x7f79a815ca70 2023-03-03T08:01:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a5d9b700 1 -- 172.21.15.98:0/219269145 >> v2:172.21.15.100:3300/0 conn(0x7f79a806b4b0 msgr2=0x7f79a80cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a5d9b700 1 --2- 172.21.15.98:0/219269145 >> v2:172.21.15.100:3300/0 conn(0x7f79a806b4b0 0x7f79a80cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a5d9b700 1 -- 172.21.15.98:0/219269145 >> v1:172.21.15.100:6789/0 conn(0x7f79a80d0270 legacy=0x7f79a80d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a5d9b700 1 -- 172.21.15.98:0/219269145 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a80cd900 con 0x7f79a815ca70 2023-03-03T08:01:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a5d9b700 1 -- 172.21.15.98:0/219269145 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f7998001670 con 0x7f79a815ca70 2023-03-03T08:01:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a5d9b700 1 -- 172.21.15.98:0/219269145 >> v1:172.21.15.98:6789/0 conn(0x7f79a815ca70 legacy=0x7f79a80cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a5d9b700 1 --2- 172.21.15.98:0/219269145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7990003a50 0x7f7990003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a5d9b700 1 --2- 172.21.15.98:0/219269145 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7990004850 0x7f7990004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a5d9b700 1 --2- 172.21.15.98:0/219269145 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7990005250 0x7f799000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a5d9b700 1 -- 172.21.15.98:0/219269145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a80cd5b0 con 0x7f7990003a50 2023-03-03T08:01:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a5d9b700 1 -- 172.21.15.98:0/219269145 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799000c160 con 0x7f7990005250 2023-03-03T08:01:22.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a5d9b700 1 -- 172.21.15.98:0/219269145 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799000c320 con 0x7f7990004850 2023-03-03T08:01:22.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a6d9d700 1 --2- 172.21.15.98:0/219269145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7990003a50 0x7f7990003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:22.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a659c700 1 --2- 172.21.15.98:0/219269145 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7990004850 0x7f7990004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:22.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.681+0000 7f79a759e700 1 --2- 172.21.15.98:0/219269145 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7990005250 0x7f799000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:22.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.682+0000 7f79a6d9d700 1 -- 172.21.15.98:0/219269145 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7990004850 msgr2=0x7f7990004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:22.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.682+0000 7f79a6d9d700 1 --2- 172.21.15.98:0/219269145 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7990004850 0x7f7990004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:22.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.682+0000 7f79a6d9d700 1 -- 172.21.15.98:0/219269145 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7990005250 msgr2=0x7f799000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:22.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.682+0000 7f79a6d9d700 1 --2- 172.21.15.98:0/219269145 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7990005250 0x7f799000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:22.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.682+0000 7f79a6d9d700 1 -- 172.21.15.98:0/219269145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79900035b0 con 0x7f7990003a50 2023-03-03T08:01:22.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.682+0000 7f79a759e700 1 -- 172.21.15.98:0/219269145 reap_dead start 2023-03-03T08:01:22.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.682+0000 7f79a6d9d700 1 --2- 172.21.15.98:0/219269145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7990003a50 0x7f7990003e60 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f799800c160 tx=0x7f7990005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abfb24eeba81760a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:22.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.682+0000 7f79a5d9b700 1 -- 172.21.15.98:0/219269145 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7998007b70 con 0x7f7990003a50 2023-03-03T08:01:22.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.682+0000 7f79a5d9b700 1 -- 172.21.15.98:0/219269145 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f799800cd20 con 0x7f7990003a50 2023-03-03T08:01:22.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.682+0000 7f79a5d9b700 1 -- 172.21.15.98:0/219269145 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7998005ba0 con 0x7f7990003a50 2023-03-03T08:01:22.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.683+0000 7f79ad1ff700 1 -- 172.21.15.98:0/219269145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7990003a50 msgr2=0x7f7990003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:22.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.683+0000 7f79ad1ff700 1 --2- 172.21.15.98:0/219269145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7990003a50 0x7f7990003e60 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f799800c160 tx=0x7f7990005880 comp rx=0 tx=0).stop 2023-03-03T08:01:22.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.683+0000 7f79ad1ff700 1 -- 172.21.15.98:0/219269145 shutdown_connections 2023-03-03T08:01:22.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.683+0000 7f79ad1ff700 1 --2- 172.21.15.98:0/219269145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7990003a50 0x7f7990003e60 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:22.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.683+0000 7f79ad1ff700 1 -- 172.21.15.98:0/219269145 >> 172.21.15.98:0/219269145 conn(0x7f79a8073e70 msgr2=0x7f79a806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:22.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.683+0000 7f79ad1ff700 1 -- 172.21.15.98:0/219269145 shutdown_connections 2023-03-03T08:01:22.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.683+0000 7f79ad1ff700 1 -- 172.21.15.98:0/219269145 wait complete. 2023-03-03T08:01:22.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.685+0000 7f79ad1ff700 1 Processor -- start 2023-03-03T08:01:22.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.685+0000 7f79ad1ff700 1 -- start start 2023-03-03T08:01:22.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.686+0000 7f79ad1ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7990003a50 0x7f79a81f5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:22.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.686+0000 7f79ad1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f79a81580f0 0x7f79a8157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:22.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.686+0000 7f79ad1ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79a80cc9f0 0x7f79a80cce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:22.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.686+0000 7f79ad1ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a80cd3b0 con 0x7f7990003a50 2023-03-03T08:01:22.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.686+0000 7f79ad1ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a81e53a0 con 0x7f79a80cc9f0 2023-03-03T08:01:22.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.686+0000 7f79ad1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79a81f68b0 con 0x7f79a81580f0 2023-03-03T08:01:22.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.687+0000 7f79a6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7990003a50 0x7f79a81f5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:22.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.687+0000 7f79a6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7990003a50 0x7f79a81f5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60484/0 (socket says 172.21.15.98:60484) 2023-03-03T08:01:22.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.687+0000 7f79a6d9d700 1 -- 172.21.15.98:0/3481231258 learned_addr learned my addr 172.21.15.98:0/3481231258 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:22.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.687+0000 7f79a759e700 1 --2- 172.21.15.98:0/3481231258 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79a80cc9f0 0x7f79a80cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:22.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.687+0000 7f79a6d9d700 1 -- 172.21.15.98:0/3481231258 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f79a81580f0 msgr2=0x7f79a8157a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:22.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.687+0000 7f79a6d9d700 1 --2- 172.21.15.98:0/3481231258 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f79a81580f0 0x7f79a8157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:22.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.687+0000 7f79a6d9d700 1 -- 172.21.15.98:0/3481231258 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79a80cc9f0 msgr2=0x7f79a80cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:22.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.687+0000 7f79a6d9d700 1 --2- 172.21.15.98:0/3481231258 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79a80cc9f0 0x7f79a80cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:22.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.687+0000 7f79a6d9d700 1 -- 172.21.15.98:0/3481231258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a80d1aa0 con 0x7f7990003a50 2023-03-03T08:01:22.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.687+0000 7f79a6d9d700 1 --2- 172.21.15.98:0/3481231258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7990003a50 0x7f79a81f5930 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f799800c750 tx=0x7f7998001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:22.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.687+0000 7f798f7fe700 1 -- 172.21.15.98:0/3481231258 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7998004d40 con 0x7f7990003a50 2023-03-03T08:01:22.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.687+0000 7f798f7fe700 1 -- 172.21.15.98:0/3481231258 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79980070d0 con 0x7f7990003a50 2023-03-03T08:01:22.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.687+0000 7f79ad1ff700 1 -- 172.21.15.98:0/3481231258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a80cdbe0 con 0x7f7990003a50 2023-03-03T08:01:22.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.688+0000 7f798f7fe700 1 -- 172.21.15.98:0/3481231258 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7998004ea0 con 0x7f7990003a50 2023-03-03T08:01:22.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.688+0000 7f79ad1ff700 1 -- 172.21.15.98:0/3481231258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a815ce40 con 0x7f7990003a50 2023-03-03T08:01:22.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.689+0000 7f79ad1ff700 1 -- 172.21.15.98:0/3481231258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79a804e6e0 con 0x7f7990003a50 2023-03-03T08:01:22.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.690+0000 7f798f7fe700 1 -- 172.21.15.98:0/3481231258 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f7998010070 con 0x7f7990003a50 2023-03-03T08:01:22.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.692+0000 7f798f7fe700 1 --2- 172.21.15.98:0/3481231258 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f79940684a0 0x7f799406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:22.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.692+0000 7f798f7fe700 1 -- 172.21.15.98:0/3481231258 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7998086930 con 0x7f7990003a50 2023-03-03T08:01:22.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.692+0000 7f79a659c700 1 --2- 172.21.15.98:0/3481231258 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f79940684a0 0x7f799406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:22.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.693+0000 7f79a659c700 1 --2- 172.21.15.98:0/3481231258 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f79940684a0 0x7f799406a950 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f79a8064000 tx=0x7f799c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:22.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.697+0000 7f798f7fe700 1 -- 172.21.15.98:0/3481231258 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7998052ff0 con 0x7f7990003a50 2023-03-03T08:01:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:22.935+0000 7f79ad1ff700 1 -- 172.21.15.98:0/3481231258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f79a80d1540 con 0x7f7990003a50 2023-03-03T08:01:23.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:23.836+0000 7f798f7fe700 1 -- 172.21.15.98:0/3481231258 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v39) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7998056610 con 0x7f7990003a50 2023-03-03T08:01:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:23.839+0000 7f79ad1ff700 1 -- 172.21.15.98:0/3481231258 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f79940684a0 msgr2=0x7f799406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:23.839+0000 7f79ad1ff700 1 --2- 172.21.15.98:0/3481231258 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f79940684a0 0x7f799406a950 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f79a8064000 tx=0x7f799c009040 comp rx=0 tx=0).stop 2023-03-03T08:01:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:23.839+0000 7f79ad1ff700 1 -- 172.21.15.98:0/3481231258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7990003a50 msgr2=0x7f79a81f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:23.839+0000 7f79ad1ff700 1 --2- 172.21.15.98:0/3481231258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7990003a50 0x7f79a81f5930 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f799800c750 tx=0x7f7998001930 comp rx=0 tx=0).stop 2023-03-03T08:01:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:23.839+0000 7f79ad1ff700 1 -- 172.21.15.98:0/3481231258 shutdown_connections 2023-03-03T08:01:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:23.839+0000 7f79ad1ff700 1 --2- 172.21.15.98:0/3481231258 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f79940684a0 0x7f799406a950 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:23.839+0000 7f79ad1ff700 1 --2- 172.21.15.98:0/3481231258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7990003a50 0x7f79a81f5930 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:23.839+0000 7f79ad1ff700 1 --2- 172.21.15.98:0/3481231258 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79a80cc9f0 0x7f79a80cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:23.839+0000 7f79ad1ff700 1 --2- 172.21.15.98:0/3481231258 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f79a81580f0 0x7f79a8157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:23.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:23.839+0000 7f79ad1ff700 1 -- 172.21.15.98:0/3481231258 >> 172.21.15.98:0/3481231258 conn(0x7f79a8073e70 msgr2=0x7f79a80b80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:23.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:23.839+0000 7f79ad1ff700 1 -- 172.21.15.98:0/3481231258 shutdown_connections 2023-03-03T08:01:23.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:23.839+0000 7f79ad1ff700 1 -- 172.21.15.98:0/3481231258 wait complete. 2023-03-03T08:01:23.871 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-03-03T08:01:23.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-03-03T08:01:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.122+0000 7f078b59e700 1 Processor -- start 2023-03-03T08:01:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.122+0000 7f078b59e700 1 -- start start 2023-03-03T08:01:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.123+0000 7f078b59e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078c0b4aa0 0x7f078c0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.123+0000 7f078b59e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f078c15dd00 con 0x7f078c0cf570 2023-03-03T08:01:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.123+0000 7f078b59e700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f078c0cd780 con 0x7f078c1609b0 2023-03-03T08:01:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.123+0000 7f078b59e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078c0cebb0 con 0x7f078c0b4aa0 2023-03-03T08:01:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.124+0000 7f0783fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f078c0cf570 0x7f078c15d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52346/0 (socket says 172.21.15.98:52346) 2023-03-03T08:01:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.124+0000 7f0783fff700 1 -- 172.21.15.98:0/91819446 learned_addr learned my addr 172.21.15.98:0/91819446 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.124+0000 7f078ad9d700 1 --2- 172.21.15.98:0/91819446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078c0b4aa0 0x7f078c0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.124+0000 7f0789d9b700 1 -- 172.21.15.98:0/91819446 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791300823 0 0) 0x7f078c15dd00 con 0x7f078c0cf570 2023-03-03T08:01:24.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.124+0000 7f0789d9b700 1 -- 172.21.15.98:0/91819446 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07780035b0 con 0x7f078c0cf570 2023-03-03T08:01:24.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.124+0000 7f078ad9d700 1 -- 172.21.15.98:0/91819446 >> v1:172.21.15.100:6789/0 conn(0x7f078c1609b0 legacy=0x7f078c160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:24.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.124+0000 7f0789d9b700 1 -- 172.21.15.98:0/91819446 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328097878 0 0) 0x7f078c0cd780 con 0x7f078c1609b0 2023-03-03T08:01:24.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.124+0000 7f078ad9d700 1 -- 172.21.15.98:0/91819446 >> v1:172.21.15.98:6789/0 conn(0x7f078c0cf570 legacy=0x7f078c15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:24.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.124+0000 7f078ad9d700 1 -- 172.21.15.98:0/91819446 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c0cda60 con 0x7f078c0b4aa0 2023-03-03T08:01:24.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.125+0000 7f078ad9d700 1 --2- 172.21.15.98:0/91819446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078c0b4aa0 0x7f078c0b4f20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0784009dc0 tx=0x7f078400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfbb739458573849 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:24.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.125+0000 7f0789d9b700 1 -- 172.21.15.98:0/91819446 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078400e070 con 0x7f078c0b4aa0 2023-03-03T08:01:24.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.125+0000 7f0789d9b700 1 -- 172.21.15.98:0/91819446 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078400bb20 con 0x7f078c0b4aa0 2023-03-03T08:01:24.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.125+0000 7f0789d9b700 1 -- 172.21.15.98:0/91819446 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078400bed0 con 0x7f078c0b4aa0 2023-03-03T08:01:24.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.126+0000 7f078b59e700 1 -- 172.21.15.98:0/91819446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078c0b4aa0 msgr2=0x7f078c0b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:24.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.126+0000 7f078b59e700 1 --2- 172.21.15.98:0/91819446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078c0b4aa0 0x7f078c0b4f20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0784009dc0 tx=0x7f078400b330 comp rx=0 tx=0).stop 2023-03-03T08:01:24.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.126+0000 7f078b59e700 1 -- 172.21.15.98:0/91819446 shutdown_connections 2023-03-03T08:01:24.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.126+0000 7f078b59e700 1 --2- 172.21.15.98:0/91819446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078c0b4aa0 0x7f078c0b4f20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:24.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.126+0000 7f078b59e700 1 -- 172.21.15.98:0/91819446 >> 172.21.15.98:0/91819446 conn(0x7f078c06b880 msgr2=0x7f078c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:24.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.126+0000 7f078b59e700 1 -- 172.21.15.98:0/91819446 shutdown_connections 2023-03-03T08:01:24.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.126+0000 7f078b59e700 1 -- 172.21.15.98:0/91819446 wait complete. 2023-03-03T08:01:24.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.129+0000 7f078b59e700 1 Processor -- start 2023-03-03T08:01:24.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.129+0000 7f078b59e700 1 -- start start 2023-03-03T08:01:24.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.130+0000 7f078b59e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078c0b4aa0 0x7f078c14f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:24.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.130+0000 7f078b59e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f078c0cf570 0x7f078c0c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:24.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.131+0000 7f078b59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078c1609b0 0x7f078c0b7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:24.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.131+0000 7f078b59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078c14f9c0 con 0x7f078c1609b0 2023-03-03T08:01:24.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.131+0000 7f078b59e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078c0b74a0 con 0x7f078c0cf570 2023-03-03T08:01:24.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.131+0000 7f078b59e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078c0b6bb0 con 0x7f078c0b4aa0 2023-03-03T08:01:24.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.131+0000 7f078ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078c1609b0 0x7f078c0b7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:24.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.131+0000 7f078ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078c1609b0 0x7f078c0b7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60492/0 (socket says 172.21.15.98:60492) 2023-03-03T08:01:24.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.131+0000 7f078ad9d700 1 -- 172.21.15.98:0/114608228 learned_addr learned my addr 172.21.15.98:0/114608228 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:24.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.131+0000 7f078a59c700 1 --2- 172.21.15.98:0/114608228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078c0b4aa0 0x7f078c14f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:24.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.131+0000 7f0783fff700 1 --2- 172.21.15.98:0/114608228 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f078c0cf570 0x7f078c0c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:24.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.131+0000 7f078ad9d700 1 -- 172.21.15.98:0/114608228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078c0b4aa0 msgr2=0x7f078c14f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:24.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.131+0000 7f078ad9d700 1 --2- 172.21.15.98:0/114608228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078c0b4aa0 0x7f078c14f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:24.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.131+0000 7f078ad9d700 1 -- 172.21.15.98:0/114608228 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f078c0cf570 msgr2=0x7f078c0c88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:24.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.131+0000 7f078ad9d700 1 --2- 172.21.15.98:0/114608228 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f078c0cf570 0x7f078c0c88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:24.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.131+0000 7f078ad9d700 1 -- 172.21.15.98:0/114608228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0784009a70 con 0x7f078c1609b0 2023-03-03T08:01:24.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.132+0000 7f078ad9d700 1 --2- 172.21.15.98:0/114608228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078c1609b0 0x7f078c0b7930 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f0784009f30 tx=0x7f0784004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:24.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.132+0000 7f0782ffd700 1 -- 172.21.15.98:0/114608228 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078401b810 con 0x7f078c1609b0 2023-03-03T08:01:24.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.132+0000 7f0782ffd700 1 -- 172.21.15.98:0/114608228 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0784020070 con 0x7f078c1609b0 2023-03-03T08:01:24.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.132+0000 7f0782ffd700 1 -- 172.21.15.98:0/114608228 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078401bbb0 con 0x7f078c1609b0 2023-03-03T08:01:24.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.132+0000 7f078b59e700 1 -- 172.21.15.98:0/114608228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f078c15cdf0 con 0x7f078c1609b0 2023-03-03T08:01:24.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.132+0000 7f078b59e700 1 -- 172.21.15.98:0/114608228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f078c0cdd40 con 0x7f078c1609b0 2023-03-03T08:01:24.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.133+0000 7f078b59e700 1 -- 172.21.15.98:0/114608228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f078c0c91f0 con 0x7f078c1609b0 2023-03-03T08:01:24.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.134+0000 7f0782ffd700 1 -- 172.21.15.98:0/114608228 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f078401bd10 con 0x7f078c1609b0 2023-03-03T08:01:24.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.136+0000 7f0782ffd700 1 --2- 172.21.15.98:0/114608228 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f076c0684a0 0x7f076c06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:24.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.136+0000 7f0782ffd700 1 -- 172.21.15.98:0/114608228 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0784022070 con 0x7f078c1609b0 2023-03-03T08:01:24.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.136+0000 7f078a59c700 1 --2- 172.21.15.98:0/114608228 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f076c0684a0 0x7f076c06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:24.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.137+0000 7f078a59c700 1 --2- 172.21.15.98:0/114608228 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f076c0684a0 0x7f076c06a950 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f077c000f60 tx=0x7f077c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:24.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.140+0000 7f0782ffd700 1 -- 172.21.15.98:0/114608228 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0784054b60 con 0x7f078c1609b0 2023-03-03T08:01:24.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.376+0000 7f078b59e700 1 -- 172.21.15.98:0/114608228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f078c15d0d0 con 0x7f078c1609b0 2023-03-03T08:01:24.782 INFO:tasks.ceph.mds.d.smithi100.stderr:2023-03-03T08:01:24.781+0000 7efd8d9f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T08:01:24.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.838+0000 7f0782ffd700 1 -- 172.21.15.98:0/114608228 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0784058180 con 0x7f078c1609b0 2023-03-03T08:01:24.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.841+0000 7f078b59e700 1 -- 172.21.15.98:0/114608228 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f076c0684a0 msgr2=0x7f076c06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:24.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.841+0000 7f078b59e700 1 --2- 172.21.15.98:0/114608228 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f076c0684a0 0x7f076c06a950 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f077c000f60 tx=0x7f077c009040 comp rx=0 tx=0).stop 2023-03-03T08:01:24.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.841+0000 7f078b59e700 1 -- 172.21.15.98:0/114608228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078c1609b0 msgr2=0x7f078c0b7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:24.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.841+0000 7f078b59e700 1 --2- 172.21.15.98:0/114608228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078c1609b0 0x7f078c0b7930 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f0784009f30 tx=0x7f0784004ad0 comp rx=0 tx=0).stop 2023-03-03T08:01:24.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.841+0000 7f078b59e700 1 -- 172.21.15.98:0/114608228 shutdown_connections 2023-03-03T08:01:24.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.841+0000 7f078b59e700 1 --2- 172.21.15.98:0/114608228 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f076c0684a0 0x7f076c06a950 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:24.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.841+0000 7f078b59e700 1 --2- 172.21.15.98:0/114608228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078c1609b0 0x7f078c0b7930 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:24.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.841+0000 7f078b59e700 1 --2- 172.21.15.98:0/114608228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078c0b4aa0 0x7f078c14f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:24.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.841+0000 7f078b59e700 1 --2- 172.21.15.98:0/114608228 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f078c0cf570 0x7f078c0c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:24.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.841+0000 7f078b59e700 1 -- 172.21.15.98:0/114608228 >> 172.21.15.98:0/114608228 conn(0x7f078c06b880 msgr2=0x7f078c15c000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:24.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.841+0000 7f078b59e700 1 -- 172.21.15.98:0/114608228 shutdown_connections 2023-03-03T08:01:24.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:24.841+0000 7f078b59e700 1 -- 172.21.15.98:0/114608228 wait complete. 2023-03-03T08:01:24.869 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-03-03T08:01:24.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:01:25.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.112+0000 7f29aa1ff700 1 Processor -- start 2023-03-03T08:01:25.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.112+0000 7f29aa1ff700 1 -- start start 2023-03-03T08:01:25.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.113+0000 7f29aa1ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f29a40b8b90 0x7f29a40cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:25.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.113+0000 7f29aa1ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29a40b5410 0x7f29a40d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:25.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.113+0000 7f29aa1ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a40d0ef0 con 0x7f29a40b87c0 2023-03-03T08:01:25.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.113+0000 7f29aa1ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a40d0600 con 0x7f29a40b5410 2023-03-03T08:01:25.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.113+0000 7f29aa1ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a40cd970 con 0x7f29a40b8b90 2023-03-03T08:01:25.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.114+0000 7f29a3fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29a40b5410 0x7f29a40d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:25.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.114+0000 7f29a37fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f29a40b87c0 0x7f29a40b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54056/0 (socket says 172.21.15.98:54056) 2023-03-03T08:01:25.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.114+0000 7f29a37fe700 1 -- 172.21.15.98:0/3820041232 learned_addr learned my addr 172.21.15.98:0/3820041232 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:25.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.114+0000 7f29a2ffd700 1 --2- 172.21.15.98:0/3820041232 >> v2:172.21.15.100:3300/0 conn(0x7f29a40b8b90 0x7f29a40cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:25.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.114+0000 7f29a27fc700 1 -- 172.21.15.98:0/3820041232 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689387500 0 0) 0x7f29a40d0ef0 con 0x7f29a40b87c0 2023-03-03T08:01:25.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.114+0000 7f29a3fff700 1 -- 172.21.15.98:0/3820041232 >> v2:172.21.15.100:3300/0 conn(0x7f29a40b8b90 msgr2=0x7f29a40cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:25.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.114+0000 7f29a3fff700 1 --2- 172.21.15.98:0/3820041232 >> v2:172.21.15.100:3300/0 conn(0x7f29a40b8b90 0x7f29a40cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.114+0000 7f29a3fff700 1 -- 172.21.15.98:0/3820041232 >> v1:172.21.15.98:6789/0 conn(0x7f29a40b87c0 legacy=0x7f29a40b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:25.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.114+0000 7f29a3fff700 1 -- 172.21.15.98:0/3820041232 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a40cdc50 con 0x7f29a40b5410 2023-03-03T08:01:25.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.115+0000 7f29a3fff700 1 --2- 172.21.15.98:0/3820041232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29a40b5410 0x7f29a40d13d0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f29a408c240 tx=0x7f299c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab697ef688a4671 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:25.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.115+0000 7f29a27fc700 1 -- 172.21.15.98:0/3820041232 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c00e070 con 0x7f29a40b5410 2023-03-03T08:01:25.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.115+0000 7f29a27fc700 1 -- 172.21.15.98:0/3820041232 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299c00bb20 con 0x7f29a40b5410 2023-03-03T08:01:25.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.115+0000 7f29a27fc700 1 -- 172.21.15.98:0/3820041232 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c012430 con 0x7f29a40b5410 2023-03-03T08:01:25.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.115+0000 7f29aa1ff700 1 -- 172.21.15.98:0/3820041232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29a40b5410 msgr2=0x7f29a40d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:25.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.115+0000 7f29aa1ff700 1 --2- 172.21.15.98:0/3820041232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29a40b5410 0x7f29a40d13d0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f29a408c240 tx=0x7f299c00b330 comp rx=0 tx=0).stop 2023-03-03T08:01:25.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.116+0000 7f29aa1ff700 1 -- 172.21.15.98:0/3820041232 shutdown_connections 2023-03-03T08:01:25.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.116+0000 7f29aa1ff700 1 --2- 172.21.15.98:0/3820041232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29a40b5410 0x7f29a40d13d0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.116+0000 7f29aa1ff700 1 --2- 172.21.15.98:0/3820041232 >> v2:172.21.15.100:3300/0 conn(0x7f29a40b8b90 0x7f29a40cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.116+0000 7f29aa1ff700 1 -- 172.21.15.98:0/3820041232 >> 172.21.15.98:0/3820041232 conn(0x7f29a406aee0 msgr2=0x7f29a406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:25.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.116+0000 7f29aa1ff700 1 -- 172.21.15.98:0/3820041232 shutdown_connections 2023-03-03T08:01:25.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.116+0000 7f29aa1ff700 1 -- 172.21.15.98:0/3820041232 wait complete. 2023-03-03T08:01:25.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.118+0000 7f29aa1ff700 1 Processor -- start 2023-03-03T08:01:25.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.118+0000 7f29aa1ff700 1 -- start start 2023-03-03T08:01:25.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.119+0000 7f29aa1ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29a40b5410 0x7f29a40adf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:25.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.119+0000 7f29aa1ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29a40b87c0 0x7f29a41f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:25.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.119+0000 7f29aa1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29a40b8b90 0x7f29a40b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:25.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.119+0000 7f29aa1ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a40acfb0 con 0x7f29a40b5410 2023-03-03T08:01:25.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.119+0000 7f29aa1ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a40ad190 con 0x7f29a40b87c0 2023-03-03T08:01:25.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.119+0000 7f29aa1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a40ad470 con 0x7f29a40b8b90 2023-03-03T08:01:25.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.119+0000 7f29a37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29a40b5410 0x7f29a40adf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:25.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.119+0000 7f29a37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29a40b5410 0x7f29a40adf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60168/0 (socket says 172.21.15.98:60168) 2023-03-03T08:01:25.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.119+0000 7f29a37fe700 1 -- 172.21.15.98:0/1892470793 learned_addr learned my addr 172.21.15.98:0/1892470793 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:25.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.120+0000 7f29a37fe700 1 -- 172.21.15.98:0/1892470793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29a40b8b90 msgr2=0x7f29a40b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:25.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.120+0000 7f29a37fe700 1 --2- 172.21.15.98:0/1892470793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29a40b8b90 0x7f29a40b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.120+0000 7f29a37fe700 1 -- 172.21.15.98:0/1892470793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29a40b87c0 msgr2=0x7f29a41f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:25.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.120+0000 7f29a37fe700 1 --2- 172.21.15.98:0/1892470793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29a40b87c0 0x7f29a41f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.120+0000 7f29a37fe700 1 -- 172.21.15.98:0/1892470793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c009a70 con 0x7f29a40b5410 2023-03-03T08:01:25.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.120+0000 7f29a37fe700 1 --2- 172.21.15.98:0/1892470793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29a40b5410 0x7f29a40adf30 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f29940011e0 tx=0x7f299400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:25.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.120+0000 7f298bfff700 1 -- 172.21.15.98:0/1892470793 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994004c80 con 0x7f29a40b5410 2023-03-03T08:01:25.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.120+0000 7f298bfff700 1 -- 172.21.15.98:0/1892470793 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2994004e00 con 0x7f29a40b5410 2023-03-03T08:01:25.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.120+0000 7f298bfff700 1 -- 172.21.15.98:0/1892470793 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994007310 con 0x7f29a40b5410 2023-03-03T08:01:25.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.120+0000 7f29aa1ff700 1 -- 172.21.15.98:0/1892470793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a40cdf30 con 0x7f29a40b5410 2023-03-03T08:01:25.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.121+0000 7f29aa1ff700 1 -- 172.21.15.98:0/1892470793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a4073a80 con 0x7f29a40b5410 2023-03-03T08:01:25.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.122+0000 7f29aa1ff700 1 -- 172.21.15.98:0/1892470793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a40ce0c0 con 0x7f29a40b5410 2023-03-03T08:01:25.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.123+0000 7f298bfff700 1 -- 172.21.15.98:0/1892470793 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f299401a020 con 0x7f29a40b5410 2023-03-03T08:01:25.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.124+0000 7f298bfff700 1 --2- 172.21.15.98:0/1892470793 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f29900684a0 0x7f299006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:25.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.124+0000 7f298bfff700 1 -- 172.21.15.98:0/1892470793 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2994085ca0 con 0x7f29a40b5410 2023-03-03T08:01:25.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.125+0000 7f29a2ffd700 1 --2- 172.21.15.98:0/1892470793 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f29900684a0 0x7f299006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:25.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.125+0000 7f29a2ffd700 1 --2- 172.21.15.98:0/1892470793 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f29900684a0 0x7f299006a950 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2998006b20 tx=0x7f2998006ab0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:25.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.128+0000 7f298bfff700 1 -- 172.21.15.98:0/1892470793 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2994052260 con 0x7f29a40b5410 2023-03-03T08:01:25.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.364+0000 7f29aa1ff700 1 -- 172.21.15.98:0/1892470793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29a40729b0 con 0x7f29a40b5410 2023-03-03T08:01:25.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.364+0000 7f298bfff700 1 -- 172.21.15.98:0/1892470793 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f2994055880 con 0x7f29a40b5410 2023-03-03T08:01:25.366 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:25.367 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6050,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/318891906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":318891906},{"type":"v1","addr":"172.21.15.98:6836","nonce":318891906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1280578660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1280578660},{"type":"v1","addr":"172.21.15.98:6837","nonce":1280578660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6906,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1395511883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1395511883},{"type":"v1","addr":"172.21.15.100:6834","nonce":1395511883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:01:21.379777+0000","modified":"2023-03-03T08:01:24.835314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6909},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6909":{"gid":6909,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/341424177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":341424177},{"type":"v1","addr":"172.21.15.100:6835","nonce":341424177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-03-03T08:01:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.367+0000 7f29aa1ff700 1 -- 172.21.15.98:0/1892470793 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f29900684a0 msgr2=0x7f299006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.367+0000 7f29aa1ff700 1 --2- 172.21.15.98:0/1892470793 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f29900684a0 0x7f299006a950 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2998006b20 tx=0x7f2998006ab0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.367+0000 7f29aa1ff700 1 -- 172.21.15.98:0/1892470793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29a40b5410 msgr2=0x7f29a40adf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.367+0000 7f29aa1ff700 1 --2- 172.21.15.98:0/1892470793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29a40b5410 0x7f29a40adf30 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f29940011e0 tx=0x7f299400cae0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.367+0000 7f29aa1ff700 1 -- 172.21.15.98:0/1892470793 shutdown_connections 2023-03-03T08:01:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.367+0000 7f29aa1ff700 1 --2- 172.21.15.98:0/1892470793 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f29900684a0 0x7f299006a950 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.367+0000 7f29aa1ff700 1 --2- 172.21.15.98:0/1892470793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29a40b5410 0x7f29a40adf30 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.367+0000 7f29aa1ff700 1 --2- 172.21.15.98:0/1892470793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29a40b8b90 0x7f29a40b77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.367+0000 7f29aa1ff700 1 --2- 172.21.15.98:0/1892470793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29a40b87c0 0x7f29a41f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.367+0000 7f29aa1ff700 1 -- 172.21.15.98:0/1892470793 >> 172.21.15.98:0/1892470793 conn(0x7f29a406aee0 msgr2=0x7f29a40b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.367+0000 7f29aa1ff700 1 -- 172.21.15.98:0/1892470793 shutdown_connections 2023-03-03T08:01:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.367+0000 7f29aa1ff700 1 -- 172.21.15.98:0/1892470793 wait complete. 2023-03-03T08:01:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 40 2023-03-03T08:01:25.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:01:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.639+0000 7fc7c33ff700 1 Processor -- start 2023-03-03T08:01:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.639+0000 7fc7c33ff700 1 -- start start 2023-03-03T08:01:25.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.640+0000 7fc7c33ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc0b8b90 0x7fc7bc0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:25.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.641+0000 7fc7c33ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7bc0d0ef0 con 0x7fc7bc0b87c0 2023-03-03T08:01:25.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.641+0000 7fc7c33ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7bc0d0600 con 0x7fc7bc0b5410 2023-03-03T08:01:25.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.641+0000 7fc7c33ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0cd970 con 0x7fc7bc0b8b90 2023-03-03T08:01:25.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.641+0000 7fc7c119b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc7bc0b87c0 0x7fc7bc0b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54072/0 (socket says 172.21.15.98:54072) 2023-03-03T08:01:25.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.641+0000 7fc7c119b700 1 -- 172.21.15.98:0/4236297014 learned_addr learned my addr 172.21.15.98:0/4236297014 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:25.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.641+0000 7fc7c099a700 1 --2- 172.21.15.98:0/4236297014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc0b8b90 0x7fc7bc0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:25.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.641+0000 7fc7b3fff700 1 -- 172.21.15.98:0/4236297014 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290924191 0 0) 0x7fc7bc0d0ef0 con 0x7fc7bc0b87c0 2023-03-03T08:01:25.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.641+0000 7fc7b3fff700 1 -- 172.21.15.98:0/4236297014 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a80035b0 con 0x7fc7bc0b87c0 2023-03-03T08:01:25.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.642+0000 7fc7b3fff700 1 -- 172.21.15.98:0/4236297014 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 126048755 0 0) 0x7fc7a80035b0 con 0x7fc7bc0b87c0 2023-03-03T08:01:25.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.642+0000 7fc7c099a700 1 -- 172.21.15.98:0/4236297014 >> v1:172.21.15.100:6789/0 conn(0x7fc7bc0b5410 legacy=0x7fc7bc0d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:25.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.642+0000 7fc7c099a700 1 -- 172.21.15.98:0/4236297014 >> v1:172.21.15.98:6789/0 conn(0x7fc7bc0b87c0 legacy=0x7fc7bc0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:25.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.642+0000 7fc7c099a700 1 -- 172.21.15.98:0/4236297014 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc0cdc50 con 0x7fc7bc0b8b90 2023-03-03T08:01:25.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.642+0000 7fc7c099a700 1 --2- 172.21.15.98:0/4236297014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc0b8b90 0x7fc7bc0cffe0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009870 tx=0x7fc7ac009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=801abe0bb6fd20c5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:25.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.642+0000 7fc7b3fff700 1 -- 172.21.15.98:0/4236297014 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b8001670 con 0x7fc7bc0b8b90 2023-03-03T08:01:25.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.642+0000 7fc7b3fff700 1 -- 172.21.15.98:0/4236297014 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac004030 con 0x7fc7bc0b8b90 2023-03-03T08:01:25.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.642+0000 7fc7b3fff700 1 -- 172.21.15.98:0/4236297014 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0043c0 con 0x7fc7bc0b8b90 2023-03-03T08:01:25.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.643+0000 7fc7c33ff700 1 -- 172.21.15.98:0/4236297014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc0b8b90 msgr2=0x7fc7bc0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:25.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.643+0000 7fc7c33ff700 1 --2- 172.21.15.98:0/4236297014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc0b8b90 0x7fc7bc0cffe0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009870 tx=0x7fc7ac009b80 comp rx=0 tx=0).stop 2023-03-03T08:01:25.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.643+0000 7fc7c33ff700 1 -- 172.21.15.98:0/4236297014 shutdown_connections 2023-03-03T08:01:25.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.643+0000 7fc7c33ff700 1 --2- 172.21.15.98:0/4236297014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc0b8b90 0x7fc7bc0cffe0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.643+0000 7fc7c33ff700 1 -- 172.21.15.98:0/4236297014 >> 172.21.15.98:0/4236297014 conn(0x7fc7bc06aee0 msgr2=0x7fc7bc06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:25.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.643+0000 7fc7c33ff700 1 -- 172.21.15.98:0/4236297014 shutdown_connections 2023-03-03T08:01:25.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.643+0000 7fc7c33ff700 1 -- 172.21.15.98:0/4236297014 wait complete. 2023-03-03T08:01:25.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.645+0000 7fc7c33ff700 1 Processor -- start 2023-03-03T08:01:25.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.645+0000 7fc7c33ff700 1 -- start start 2023-03-03T08:01:25.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.646+0000 7fc7c33ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7bc0b5410 0x7fc7bc156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:25.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.646+0000 7fc7c33ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc0b87c0 0x7fc7bc0c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:25.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.646+0000 7fc7c33ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7bc0b8b90 0x7fc7bc157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:25.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.646+0000 7fc7c33ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0cc290 con 0x7fc7bc0b5410 2023-03-03T08:01:25.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.646+0000 7fc7c33ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc157640 con 0x7fc7bc0b8b90 2023-03-03T08:01:25.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.646+0000 7fc7c33ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0cb630 con 0x7fc7bc0b87c0 2023-03-03T08:01:25.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.647+0000 7fc7c119b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7bc0b5410 0x7fc7bc156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:25.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.647+0000 7fc7c119b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7bc0b5410 0x7fc7bc156250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60180/0 (socket says 172.21.15.98:60180) 2023-03-03T08:01:25.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.647+0000 7fc7c119b700 1 -- 172.21.15.98:0/284508536 learned_addr learned my addr 172.21.15.98:0/284508536 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:25.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.647+0000 7fc7c099a700 1 --2- 172.21.15.98:0/284508536 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc0b87c0 0x7fc7bc0c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:25.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.647+0000 7fc7c199c700 1 --2- 172.21.15.98:0/284508536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7bc0b8b90 0x7fc7bc157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:25.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.647+0000 7fc7c119b700 1 -- 172.21.15.98:0/284508536 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc0b87c0 msgr2=0x7fc7bc0c81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:25.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.647+0000 7fc7c119b700 1 --2- 172.21.15.98:0/284508536 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc0b87c0 0x7fc7bc0c81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.647+0000 7fc7c119b700 1 -- 172.21.15.98:0/284508536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7bc0b8b90 msgr2=0x7fc7bc157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:25.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.647+0000 7fc7c119b700 1 --2- 172.21.15.98:0/284508536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7bc0b8b90 0x7fc7bc157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.647+0000 7fc7c119b700 1 -- 172.21.15.98:0/284508536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac0094a0 con 0x7fc7bc0b5410 2023-03-03T08:01:25.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.647+0000 7fc7c119b700 1 --2- 172.21.15.98:0/284508536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7bc0b5410 0x7fc7bc156250 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fc7b800c7b0 tx=0x7fc7b800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:25.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.648+0000 7fc7b1ffb700 1 -- 172.21.15.98:0/284508536 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b8007560 con 0x7fc7bc0b5410 2023-03-03T08:01:25.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.648+0000 7fc7b1ffb700 1 -- 172.21.15.98:0/284508536 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b80076e0 con 0x7fc7bc0b5410 2023-03-03T08:01:25.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.648+0000 7fc7b1ffb700 1 -- 172.21.15.98:0/284508536 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b8004ab0 con 0x7fc7bc0b5410 2023-03-03T08:01:25.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.648+0000 7fc7c33ff700 1 -- 172.21.15.98:0/284508536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7bc0cdf30 con 0x7fc7bc0b5410 2023-03-03T08:01:25.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.648+0000 7fc7c33ff700 1 -- 172.21.15.98:0/284508536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7bc073a80 con 0x7fc7bc0b5410 2023-03-03T08:01:25.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.649+0000 7fc7c33ff700 1 -- 172.21.15.98:0/284508536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7bc0ce0c0 con 0x7fc7bc0b5410 2023-03-03T08:01:25.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.650+0000 7fc7b1ffb700 1 -- 172.21.15.98:0/284508536 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fc7b801a020 con 0x7fc7bc0b5410 2023-03-03T08:01:25.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.651+0000 7fc7b1ffb700 1 --2- 172.21.15.98:0/284508536 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc7a00684d0 0x7fc7a006a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:25.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.651+0000 7fc7b1ffb700 1 -- 172.21.15.98:0/284508536 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc7b8085e80 con 0x7fc7bc0b5410 2023-03-03T08:01:25.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.652+0000 7fc7c099a700 1 --2- 172.21.15.98:0/284508536 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc7a00684d0 0x7fc7a006a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:25.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.652+0000 7fc7c099a700 1 --2- 172.21.15.98:0/284508536 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc7a00684d0 0x7fc7a006a980 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac0097c0 tx=0x7fc7ac008000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:25.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.655+0000 7fc7b1ffb700 1 -- 172.21.15.98:0/284508536 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc7b8052440 con 0x7fc7bc0b5410 2023-03-03T08:01:25.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.850+0000 7fc7c33ff700 1 -- 172.21.15.98:0/284508536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7bc0729b0 con 0x7fc7bc0b5410 2023-03-03T08:01:25.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.851+0000 7fc7b1ffb700 1 -- 172.21.15.98:0/284508536 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+17982 (secure 0 0 0) 0x7fc7b8055a60 con 0x7fc7bc0b5410 2023-03-03T08:01:25.853 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:25.853 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":78,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:01:22.383102+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:01:19.439394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-03-03T08:01:20.381345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:01:25.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.854+0000 7fc7c33ff700 1 -- 172.21.15.98:0/284508536 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc7a00684d0 msgr2=0x7fc7a006a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:25.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.854+0000 7fc7c33ff700 1 --2- 172.21.15.98:0/284508536 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc7a00684d0 0x7fc7a006a980 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac0097c0 tx=0x7fc7ac008000 comp rx=0 tx=0).stop 2023-03-03T08:01:25.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.854+0000 7fc7c33ff700 1 -- 172.21.15.98:0/284508536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7bc0b5410 msgr2=0x7fc7bc156250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:25.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.854+0000 7fc7c33ff700 1 --2- 172.21.15.98:0/284508536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7bc0b5410 0x7fc7bc156250 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fc7b800c7b0 tx=0x7fc7b800cb70 comp rx=0 tx=0).stop 2023-03-03T08:01:25.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.854+0000 7fc7c33ff700 1 -- 172.21.15.98:0/284508536 shutdown_connections 2023-03-03T08:01:25.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.854+0000 7fc7c33ff700 1 --2- 172.21.15.98:0/284508536 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc7a00684d0 0x7fc7a006a980 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.854+0000 7fc7c33ff700 1 --2- 172.21.15.98:0/284508536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7bc0b5410 0x7fc7bc156250 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.854+0000 7fc7c33ff700 1 --2- 172.21.15.98:0/284508536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7bc0b8b90 0x7fc7bc157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.854+0000 7fc7c33ff700 1 --2- 172.21.15.98:0/284508536 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc0b87c0 0x7fc7bc0c81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:25.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.854+0000 7fc7c33ff700 1 -- 172.21.15.98:0/284508536 >> 172.21.15.98:0/284508536 conn(0x7fc7bc06aee0 msgr2=0x7fc7bc0b7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:25.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.854+0000 7fc7c33ff700 1 -- 172.21.15.98:0/284508536 shutdown_connections 2023-03-03T08:01:25.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:25.854+0000 7fc7c33ff700 1 -- 172.21.15.98:0/284508536 wait complete. 2023-03-03T08:01:25.882 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-03-03T08:01:25.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:01:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.128+0000 7f11de99d700 1 Processor -- start 2023-03-03T08:01:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.128+0000 7f11de99d700 1 -- start start 2023-03-03T08:01:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.130+0000 7f11de99d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f11d8146360 0x7f11d8160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.130+0000 7f11de99d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f11d80d2590 0x7f11d80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.130+0000 7f11de99d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d81614c0 con 0x7f11d80b4ab0 2023-03-03T08:01:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.130+0000 7f11de99d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d80b5440 con 0x7f11d8146360 2023-03-03T08:01:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.130+0000 7f11de99d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d80d10f0 con 0x7f11d80d2590 2023-03-03T08:01:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.130+0000 7f11dd99b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f11d8146360 0x7f11d8160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.130+0000 7f11dd99b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f11d8146360 0x7f11d8160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60182/0 (socket says 172.21.15.98:60182) 2023-03-03T08:01:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.130+0000 7f11dd19a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f11d80b4ab0 0x7f11d80731e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54086/0 (socket says 172.21.15.98:54086) 2023-03-03T08:01:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.130+0000 7f11dd99b700 1 -- 172.21.15.98:0/330964915 learned_addr learned my addr 172.21.15.98:0/330964915 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.130+0000 7f11de19c700 1 --2- 172.21.15.98:0/330964915 >> v2:172.21.15.100:3300/0 conn(0x7f11d80d2590 0x7f11d80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.131+0000 7f11dd99b700 1 -- 172.21.15.98:0/330964915 >> v2:172.21.15.100:3300/0 conn(0x7f11d80d2590 msgr2=0x7f11d80b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.131+0000 7f11dd99b700 1 --2- 172.21.15.98:0/330964915 >> v2:172.21.15.100:3300/0 conn(0x7f11d80d2590 0x7f11d80b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.131+0000 7f11dd99b700 1 -- 172.21.15.98:0/330964915 >> v1:172.21.15.98:6789/0 conn(0x7f11d80b4ab0 legacy=0x7f11d80731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.131+0000 7f11dd99b700 1 -- 172.21.15.98:0/330964915 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d80cdf20 con 0x7f11d8146360 2023-03-03T08:01:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.131+0000 7f11de19c700 1 --2- 172.21.15.98:0/330964915 >> v2:172.21.15.100:3300/0 conn(0x7f11d80d2590 0x7f11d80b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.131+0000 7f11dc999700 1 -- 172.21.15.98:0/330964915 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293792946 0 0) 0x7f11c8001110 con 0x7f11d80b4ab0 2023-03-03T08:01:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.131+0000 7f11dd99b700 1 --2- 172.21.15.98:0/330964915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d8146360 0x7f11d8160f80 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f11d80b5720 tx=0x7f11d4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8764fbff5026ab22 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.131+0000 7f11dc999700 1 -- 172.21.15.98:0/330964915 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d400b870 con 0x7f11d8146360 2023-03-03T08:01:26.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.131+0000 7f11dc999700 1 -- 172.21.15.98:0/330964915 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d400b9d0 con 0x7f11d8146360 2023-03-03T08:01:26.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.131+0000 7f11dc999700 1 -- 172.21.15.98:0/330964915 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d400bd60 con 0x7f11d8146360 2023-03-03T08:01:26.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.132+0000 7f11de99d700 1 -- 172.21.15.98:0/330964915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d8146360 msgr2=0x7f11d8160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:26.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.132+0000 7f11de99d700 1 --2- 172.21.15.98:0/330964915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d8146360 0x7f11d8160f80 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f11d80b5720 tx=0x7f11d4009bd0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.132+0000 7f11de99d700 1 -- 172.21.15.98:0/330964915 shutdown_connections 2023-03-03T08:01:26.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.132+0000 7f11de99d700 1 --2- 172.21.15.98:0/330964915 >> v2:172.21.15.100:3300/0 conn(0x7f11d80d2590 0x7f11d80b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.132+0000 7f11de99d700 1 --2- 172.21.15.98:0/330964915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d8146360 0x7f11d8160f80 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.132+0000 7f11de99d700 1 -- 172.21.15.98:0/330964915 >> 172.21.15.98:0/330964915 conn(0x7f11d806b8e0 msgr2=0x7f11d80cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:26.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.132+0000 7f11de99d700 1 -- 172.21.15.98:0/330964915 shutdown_connections 2023-03-03T08:01:26.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.132+0000 7f11de99d700 1 -- 172.21.15.98:0/330964915 wait complete. 2023-03-03T08:01:26.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.134+0000 7f11de99d700 1 Processor -- start 2023-03-03T08:01:26.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.134+0000 7f11de99d700 1 -- start start 2023-03-03T08:01:26.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.135+0000 7f11de99d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11d80b4ab0 0x7f11d80cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:26.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.135+0000 7f11de99d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11d80d2590 0x7f11d80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:26.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.135+0000 7f11de99d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d8146360 0x7f11d8151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:26.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.135+0000 7f11de99d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d80d0a80 con 0x7f11d8146360 2023-03-03T08:01:26.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.135+0000 7f11de99d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d80ca3c0 con 0x7f11d80d2590 2023-03-03T08:01:26.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.135+0000 7f11de99d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d80b3a00 con 0x7f11d80b4ab0 2023-03-03T08:01:26.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.136+0000 7f11de19c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d8146360 0x7f11d8151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:26.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.136+0000 7f11de19c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d8146360 0x7f11d8151a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60196/0 (socket says 172.21.15.98:60196) 2023-03-03T08:01:26.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.136+0000 7f11de19c700 1 -- 172.21.15.98:0/2680703804 learned_addr learned my addr 172.21.15.98:0/2680703804 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:26.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.136+0000 7f11dd19a700 1 --2- 172.21.15.98:0/2680703804 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11d80d2590 0x7f11d80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:26.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.136+0000 7f11dd99b700 1 --2- 172.21.15.98:0/2680703804 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11d80b4ab0 0x7f11d80cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:26.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.136+0000 7f11de19c700 1 -- 172.21.15.98:0/2680703804 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11d80b4ab0 msgr2=0x7f11d80cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:26.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.136+0000 7f11de19c700 1 --2- 172.21.15.98:0/2680703804 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11d80b4ab0 0x7f11d80cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.136+0000 7f11de19c700 1 -- 172.21.15.98:0/2680703804 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11d80d2590 msgr2=0x7f11d80cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:26.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.136+0000 7f11de19c700 1 --2- 172.21.15.98:0/2680703804 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11d80d2590 0x7f11d80cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.136+0000 7f11de19c700 1 -- 172.21.15.98:0/2680703804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d4009550 con 0x7f11d8146360 2023-03-03T08:01:26.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.137+0000 7f11de19c700 1 --2- 172.21.15.98:0/2680703804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d8146360 0x7f11d8151a90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f11d0007e70 tx=0x7f11d000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:26.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.137+0000 7f11ce7fc700 1 -- 172.21.15.98:0/2680703804 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d000dd20 con 0x7f11d8146360 2023-03-03T08:01:26.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.137+0000 7f11ce7fc700 1 -- 172.21.15.98:0/2680703804 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d000f040 con 0x7f11d8146360 2023-03-03T08:01:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.137+0000 7f11de99d700 1 -- 172.21.15.98:0/2680703804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d80cd960 con 0x7f11d8146360 2023-03-03T08:01:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.137+0000 7f11ce7fc700 1 -- 172.21.15.98:0/2680703804 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d000dea0 con 0x7f11d8146360 2023-03-03T08:01:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.137+0000 7f11de99d700 1 -- 172.21.15.98:0/2680703804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d80cd680 con 0x7f11d8146360 2023-03-03T08:01:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.138+0000 7f11de99d700 1 -- 172.21.15.98:0/2680703804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d80cdc40 con 0x7f11d8146360 2023-03-03T08:01:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.140+0000 7f11ce7fc700 1 -- 172.21.15.98:0/2680703804 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f11d001c030 con 0x7f11d8146360 2023-03-03T08:01:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.141+0000 7f11ce7fc700 1 --2- 172.21.15.98:0/2680703804 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f11bc0684a0 0x7f11bc06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.141+0000 7f11ce7fc700 1 -- 172.21.15.98:0/2680703804 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f11d0085d70 con 0x7f11d8146360 2023-03-03T08:01:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.141+0000 7f11dd99b700 1 --2- 172.21.15.98:0/2680703804 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f11bc0684a0 0x7f11bc06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.142+0000 7f11dd99b700 1 --2- 172.21.15.98:0/2680703804 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f11bc0684a0 0x7f11bc06a950 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009c10 tx=0x7f11d4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.145+0000 7f11ce7fc700 1 -- 172.21.15.98:0/2680703804 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f11d0052430 con 0x7f11d8146360 2023-03-03T08:01:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.336+0000 7f11de99d700 1 -- 172.21.15.98:0/2680703804 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11d8068300 con 0x7f11bc0684a0 2023-03-03T08:01:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.346+0000 7f11ce7fc700 1 -- 172.21.15.98:0/2680703804 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310950 (secure 0 0 0) 0x7f11d8068300 con 0x7f11bc0684a0 2023-03-03T08:01:26.351 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.351+0000 7f11de99d700 1 -- 172.21.15.98:0/2680703804 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f11bc0684a0 msgr2=0x7f11bc06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.351+0000 7f11de99d700 1 --2- 172.21.15.98:0/2680703804 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f11bc0684a0 0x7f11bc06a950 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009c10 tx=0x7f11d4008030 comp rx=0 tx=0).stop 2023-03-03T08:01:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.351+0000 7f11de99d700 1 -- 172.21.15.98:0/2680703804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d8146360 msgr2=0x7f11d8151a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.351+0000 7f11de99d700 1 --2- 172.21.15.98:0/2680703804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d8146360 0x7f11d8151a90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f11d0007e70 tx=0x7f11d000d3b0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.351+0000 7f11de99d700 1 -- 172.21.15.98:0/2680703804 shutdown_connections 2023-03-03T08:01:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.351+0000 7f11de99d700 1 --2- 172.21.15.98:0/2680703804 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f11bc0684a0 0x7f11bc06a950 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.351+0000 7f11de99d700 1 --2- 172.21.15.98:0/2680703804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d8146360 0x7f11d8151a90 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.351+0000 7f11de99d700 1 --2- 172.21.15.98:0/2680703804 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11d80b4ab0 0x7f11d80cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.351+0000 7f11de99d700 1 --2- 172.21.15.98:0/2680703804 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11d80d2590 0x7f11d80cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.351+0000 7f11de99d700 1 -- 172.21.15.98:0/2680703804 >> 172.21.15.98:0/2680703804 conn(0x7f11d806b8e0 msgr2=0x7f11d80b2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.351+0000 7f11de99d700 1 -- 172.21.15.98:0/2680703804 shutdown_connections 2023-03-03T08:01:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.351+0000 7f11de99d700 1 -- 172.21.15.98:0/2680703804 wait complete. 2023-03-03T08:01:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T08:01:26.380 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2023-03-03T08:01:26.254106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":139,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":87468,"kb_used_data":8492,"kb_used_omap":0,"kb_used_meta":78912,"kb_avail":754887252,"statfs":{"total":773094113280,"available":773004546048,"internally_reserved":0,"allocated":8695808,"data_stored":4511076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":80805888},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000983"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.419214+0000","last_change":"2023-03-03T08:01:21.416277+0000","last_active":"2023-03-03T08:01:21.419214+0000","last_peered":"2023-03-03T08:01:21.419214+0000","last_clean":"2023-03-03T08:01:21.419214+0000","last_became_active":"2023-03-03T08:01:21.415936+0000","last_became_peered":"2023-03-03T08:01:21.415936+0000","last_unstale":"2023-03-03T08:01:21.419214+0000","last_undegraded":"2023-03-03T08:01:21.419214+0000","last_fullsized":"2023-03-03T08:01:21.419214+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:26:14.276515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407560+0000","last_change":"2023-03-03T08:01:21.399865+0000","last_active":"2023-03-03T08:01:21.407560+0000","last_peered":"2023-03-03T08:01:21.407560+0000","last_clean":"2023-03-03T08:01:21.407560+0000","last_became_active":"2023-03-03T08:01:21.399529+0000","last_became_peered":"2023-03-03T08:01:21.399529+0000","last_unstale":"2023-03-03T08:01:21.407560+0000","last_undegraded":"2023-03-03T08:01:21.407560+0000","last_fullsized":"2023-03-03T08:01:21.407560+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:37:59.548033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407863+0000","last_change":"2023-03-03T08:01:21.402712+0000","last_active":"2023-03-03T08:01:21.407863+0000","last_peered":"2023-03-03T08:01:21.407863+0000","last_clean":"2023-03-03T08:01:21.407863+0000","last_became_active":"2023-03-03T08:01:21.402451+0000","last_became_peered":"2023-03-03T08:01:21.402451+0000","last_unstale":"2023-03-03T08:01:21.407863+0000","last_undegraded":"2023-03-03T08:01:21.407863+0000","last_fullsized":"2023-03-03T08:01:21.407863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:13:20.676253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421293+0000","last_change":"2023-03-03T08:01:21.417804+0000","last_active":"2023-03-03T08:01:21.421293+0000","last_peered":"2023-03-03T08:01:21.421293+0000","last_clean":"2023-03-03T08:01:21.421293+0000","last_became_active":"2023-03-03T08:01:21.417541+0000","last_became_peered":"2023-03-03T08:01:21.417541+0000","last_unstale":"2023-03-03T08:01:21.421293+0000","last_undegraded":"2023-03-03T08:01:21.421293+0000","last_fullsized":"2023-03-03T08:01:21.421293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:18.488635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.415089+0000","last_change":"2023-03-03T08:01:21.411867+0000","last_active":"2023-03-03T08:01:21.415089+0000","last_peered":"2023-03-03T08:01:21.415089+0000","last_clean":"2023-03-03T08:01:21.415089+0000","last_became_active":"2023-03-03T08:01:21.411572+0000","last_became_peered":"2023-03-03T08:01:21.411572+0000","last_unstale":"2023-03-03T08:01:21.415089+0000","last_undegraded":"2023-03-03T08:01:21.415089+0000","last_fullsized":"2023-03-03T08:01:21.415089+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:41:35.166694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.420555+0000","last_change":"2023-03-03T08:01:21.419143+0000","last_active":"2023-03-03T08:01:21.420555+0000","last_peered":"2023-03-03T08:01:21.420555+0000","last_clean":"2023-03-03T08:01:21.420555+0000","last_became_active":"2023-03-03T08:01:21.418890+0000","last_became_peered":"2023-03-03T08:01:21.418890+0000","last_unstale":"2023-03-03T08:01:21.420555+0000","last_undegraded":"2023-03-03T08:01:21.420555+0000","last_fullsized":"2023-03-03T08:01:21.420555+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:30:42.888519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418581+0000","last_change":"2023-03-03T08:01:21.412809+0000","last_active":"2023-03-03T08:01:21.418581+0000","last_peered":"2023-03-03T08:01:21.418581+0000","last_clean":"2023-03-03T08:01:21.418581+0000","last_became_active":"2023-03-03T08:01:21.412418+0000","last_became_peered":"2023-03-03T08:01:21.412418+0000","last_unstale":"2023-03-03T08:01:21.418581+0000","last_undegraded":"2023-03-03T08:01:21.418581+0000","last_fullsized":"2023-03-03T08:01:21.418581+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:30:09.829742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.415126+0000","last_change":"2023-03-03T08:01:21.412254+0000","last_active":"2023-03-03T08:01:21.415126+0000","last_peered":"2023-03-03T08:01:21.415126+0000","last_clean":"2023-03-03T08:01:21.415126+0000","last_became_active":"2023-03-03T08:01:21.411988+0000","last_became_peered":"2023-03-03T08:01:21.411988+0000","last_unstale":"2023-03-03T08:01:21.415126+0000","last_undegraded":"2023-03-03T08:01:21.415126+0000","last_fullsized":"2023-03-03T08:01:21.415126+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:03:06.355953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421694+0000","last_change":"2023-03-03T08:01:21.412728+0000","last_active":"2023-03-03T08:01:21.421694+0000","last_peered":"2023-03-03T08:01:21.421694+0000","last_clean":"2023-03-03T08:01:21.421694+0000","last_became_active":"2023-03-03T08:01:21.412449+0000","last_became_peered":"2023-03-03T08:01:21.412449+0000","last_unstale":"2023-03-03T08:01:21.421694+0000","last_undegraded":"2023-03-03T08:01:21.421694+0000","last_fullsized":"2023-03-03T08:01:21.421694+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:47:50.975464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417638+0000","last_change":"2023-03-03T08:01:21.410465+0000","last_active":"2023-03-03T08:01:21.417638+0000","last_peered":"2023-03-03T08:01:21.417638+0000","last_clean":"2023-03-03T08:01:21.417638+0000","last_became_active":"2023-03-03T08:01:21.410184+0000","last_became_peered":"2023-03-03T08:01:21.410184+0000","last_unstale":"2023-03-03T08:01:21.417638+0000","last_undegraded":"2023-03-03T08:01:21.417638+0000","last_fullsized":"2023-03-03T08:01:21.417638+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:49:08.176139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407410+0000","last_change":"2023-03-03T08:01:21.400759+0000","last_active":"2023-03-03T08:01:21.407410+0000","last_peered":"2023-03-03T08:01:21.407410+0000","last_clean":"2023-03-03T08:01:21.407410+0000","last_became_active":"2023-03-03T08:01:21.400216+0000","last_became_peered":"2023-03-03T08:01:21.400216+0000","last_unstale":"2023-03-03T08:01:21.407410+0000","last_undegraded":"2023-03-03T08:01:21.407410+0000","last_fullsized":"2023-03-03T08:01:21.407410+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:25:46.913975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.408582+0000","last_change":"2023-03-03T08:01:21.405052+0000","last_active":"2023-03-03T08:01:21.408582+0000","last_peered":"2023-03-03T08:01:21.408582+0000","last_clean":"2023-03-03T08:01:21.408582+0000","last_became_active":"2023-03-03T08:01:21.404593+0000","last_became_peered":"2023-03-03T08:01:21.404593+0000","last_unstale":"2023-03-03T08:01:21.408582+0000","last_undegraded":"2023-03-03T08:01:21.408582+0000","last_fullsized":"2023-03-03T08:01:21.408582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:05:28.891281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.410906+0000","last_change":"2023-03-03T08:01:21.408597+0000","last_active":"2023-03-03T08:01:21.410906+0000","last_peered":"2023-03-03T08:01:21.410906+0000","last_clean":"2023-03-03T08:01:21.410906+0000","last_became_active":"2023-03-03T08:01:21.408282+0000","last_became_peered":"2023-03-03T08:01:21.408282+0000","last_unstale":"2023-03-03T08:01:21.410906+0000","last_undegraded":"2023-03-03T08:01:21.410906+0000","last_fullsized":"2023-03-03T08:01:21.410906+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:43:21.042754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417575+0000","last_change":"2023-03-03T08:01:21.414742+0000","last_active":"2023-03-03T08:01:21.417575+0000","last_peered":"2023-03-03T08:01:21.417575+0000","last_clean":"2023-03-03T08:01:21.417575+0000","last_became_active":"2023-03-03T08:01:21.414475+0000","last_became_peered":"2023-03-03T08:01:21.414475+0000","last_unstale":"2023-03-03T08:01:21.417575+0000","last_undegraded":"2023-03-03T08:01:21.417575+0000","last_fullsized":"2023-03-03T08:01:21.417575+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:48:45.104859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.401138+0000","last_change":"2023-03-03T08:01:21.396417+0000","last_active":"2023-03-03T08:01:21.401138+0000","last_peered":"2023-03-03T08:01:21.401138+0000","last_clean":"2023-03-03T08:01:21.401138+0000","last_became_active":"2023-03-03T08:01:21.396067+0000","last_became_peered":"2023-03-03T08:01:21.396067+0000","last_unstale":"2023-03-03T08:01:21.401138+0000","last_undegraded":"2023-03-03T08:01:21.401138+0000","last_fullsized":"2023-03-03T08:01:21.401138+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:29:14.686210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.404001+0000","last_change":"2023-03-03T08:01:21.397779+0000","last_active":"2023-03-03T08:01:21.404001+0000","last_peered":"2023-03-03T08:01:21.404001+0000","last_clean":"2023-03-03T08:01:21.404001+0000","last_became_active":"2023-03-03T08:01:21.397486+0000","last_became_peered":"2023-03-03T08:01:21.397486+0000","last_unstale":"2023-03-03T08:01:21.404001+0000","last_undegraded":"2023-03-03T08:01:21.404001+0000","last_fullsized":"2023-03-03T08:01:21.404001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:23:21.871284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417373+0000","last_change":"2023-03-03T08:01:21.415221+0000","last_active":"2023-03-03T08:01:21.417373+0000","last_peered":"2023-03-03T08:01:21.417373+0000","last_clean":"2023-03-03T08:01:21.417373+0000","last_became_active":"2023-03-03T08:01:21.414862+0000","last_became_peered":"2023-03-03T08:01:21.414862+0000","last_unstale":"2023-03-03T08:01:21.417373+0000","last_undegraded":"2023-03-03T08:01:21.417373+0000","last_fullsized":"2023-03-03T08:01:21.417373+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:52:52.458592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.420223+0000","last_change":"2023-03-03T08:01:21.418852+0000","last_active":"2023-03-03T08:01:21.420223+0000","last_peered":"2023-03-03T08:01:21.420223+0000","last_clean":"2023-03-03T08:01:21.420223+0000","last_became_active":"2023-03-03T08:01:21.418555+0000","last_became_peered":"2023-03-03T08:01:21.418555+0000","last_unstale":"2023-03-03T08:01:21.420223+0000","last_undegraded":"2023-03-03T08:01:21.420223+0000","last_fullsized":"2023-03-03T08:01:21.420223+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:10:22.496044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.408255+0000","last_change":"2023-03-03T08:01:21.400801+0000","last_active":"2023-03-03T08:01:21.408255+0000","last_peered":"2023-03-03T08:01:21.408255+0000","last_clean":"2023-03-03T08:01:21.408255+0000","last_became_active":"2023-03-03T08:01:21.400549+0000","last_became_peered":"2023-03-03T08:01:21.400549+0000","last_unstale":"2023-03-03T08:01:21.408255+0000","last_undegraded":"2023-03-03T08:01:21.408255+0000","last_fullsized":"2023-03-03T08:01:21.408255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:58:06.304755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417365+0000","last_change":"2023-03-03T08:01:21.412605+0000","last_active":"2023-03-03T08:01:21.417365+0000","last_peered":"2023-03-03T08:01:21.417365+0000","last_clean":"2023-03-03T08:01:21.417365+0000","last_became_active":"2023-03-03T08:01:21.412358+0000","last_became_peered":"2023-03-03T08:01:21.412358+0000","last_unstale":"2023-03-03T08:01:21.417365+0000","last_undegraded":"2023-03-03T08:01:21.417365+0000","last_fullsized":"2023-03-03T08:01:21.417365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:59:33.772626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.424134+0000","last_change":"2023-03-03T08:01:21.418902+0000","last_active":"2023-03-03T08:01:21.424134+0000","last_peered":"2023-03-03T08:01:21.424134+0000","last_clean":"2023-03-03T08:01:21.424134+0000","last_became_active":"2023-03-03T08:01:21.418610+0000","last_became_peered":"2023-03-03T08:01:21.418610+0000","last_unstale":"2023-03-03T08:01:21.424134+0000","last_undegraded":"2023-03-03T08:01:21.424134+0000","last_fullsized":"2023-03-03T08:01:21.424134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:26:14.276515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.414417+0000","last_change":"2023-03-03T08:01:21.411987+0000","last_active":"2023-03-03T08:01:21.414417+0000","last_peered":"2023-03-03T08:01:21.414417+0000","last_clean":"2023-03-03T08:01:21.414417+0000","last_became_active":"2023-03-03T08:01:21.411634+0000","last_became_peered":"2023-03-03T08:01:21.411634+0000","last_unstale":"2023-03-03T08:01:21.414417+0000","last_undegraded":"2023-03-03T08:01:21.414417+0000","last_fullsized":"2023-03-03T08:01:21.414417+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:59:46.957695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421685+0000","last_change":"2023-03-03T08:01:21.415312+0000","last_active":"2023-03-03T08:01:21.421685+0000","last_peered":"2023-03-03T08:01:21.421685+0000","last_clean":"2023-03-03T08:01:21.421685+0000","last_became_active":"2023-03-03T08:01:21.412152+0000","last_became_peered":"2023-03-03T08:01:21.412152+0000","last_unstale":"2023-03-03T08:01:21.421685+0000","last_undegraded":"2023-03-03T08:01:21.421685+0000","last_fullsized":"2023-03-03T08:01:21.421685+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:05:28.891281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.402627+0000","last_change":"2023-03-03T08:01:21.395987+0000","last_active":"2023-03-03T08:01:21.402627+0000","last_peered":"2023-03-03T08:01:21.402627+0000","last_clean":"2023-03-03T08:01:21.402627+0000","last_became_active":"2023-03-03T08:01:21.395709+0000","last_became_peered":"2023-03-03T08:01:21.395709+0000","last_unstale":"2023-03-03T08:01:21.402627+0000","last_undegraded":"2023-03-03T08:01:21.402627+0000","last_fullsized":"2023-03-03T08:01:21.402627+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:16:23.956197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.400544+0000","last_change":"2023-03-03T08:01:21.395857+0000","last_active":"2023-03-03T08:01:21.400544+0000","last_peered":"2023-03-03T08:01:21.400544+0000","last_clean":"2023-03-03T08:01:21.400544+0000","last_became_active":"2023-03-03T08:01:21.395595+0000","last_became_peered":"2023-03-03T08:01:21.395595+0000","last_unstale":"2023-03-03T08:01:21.400544+0000","last_undegraded":"2023-03-03T08:01:21.400544+0000","last_fullsized":"2023-03-03T08:01:21.400544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:43.087204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.408859+0000","last_change":"2023-03-03T08:01:21.404683+0000","last_active":"2023-03-03T08:01:21.408859+0000","last_peered":"2023-03-03T08:01:21.408859+0000","last_clean":"2023-03-03T08:01:21.408859+0000","last_became_active":"2023-03-03T08:01:21.404412+0000","last_became_peered":"2023-03-03T08:01:21.404412+0000","last_unstale":"2023-03-03T08:01:21.408859+0000","last_undegraded":"2023-03-03T08:01:21.408859+0000","last_fullsized":"2023-03-03T08:01:21.408859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:43:37.184568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.405619+0000","last_change":"2023-03-03T08:01:21.400605+0000","last_active":"2023-03-03T08:01:21.405619+0000","last_peered":"2023-03-03T08:01:21.405619+0000","last_clean":"2023-03-03T08:01:21.405619+0000","last_became_active":"2023-03-03T08:01:21.400346+0000","last_became_peered":"2023-03-03T08:01:21.400346+0000","last_unstale":"2023-03-03T08:01:21.405619+0000","last_undegraded":"2023-03-03T08:01:21.405619+0000","last_fullsized":"2023-03-03T08:01:21.405619+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:11:17.517768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.411532+0000","last_change":"2023-03-03T08:01:21.407540+0000","last_active":"2023-03-03T08:01:21.411532+0000","last_peered":"2023-03-03T08:01:21.411532+0000","last_clean":"2023-03-03T08:01:21.411532+0000","last_became_active":"2023-03-03T08:01:21.407302+0000","last_became_peered":"2023-03-03T08:01:21.407302+0000","last_unstale":"2023-03-03T08:01:21.411532+0000","last_undegraded":"2023-03-03T08:01:21.411532+0000","last_fullsized":"2023-03-03T08:01:21.411532+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:20:46.479131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.415500+0000","last_change":"2023-03-03T08:01:21.410064+0000","last_active":"2023-03-03T08:01:21.415500+0000","last_peered":"2023-03-03T08:01:21.415500+0000","last_clean":"2023-03-03T08:01:21.415500+0000","last_became_active":"2023-03-03T08:01:21.409788+0000","last_became_peered":"2023-03-03T08:01:21.409788+0000","last_unstale":"2023-03-03T08:01:21.415500+0000","last_undegraded":"2023-03-03T08:01:21.415500+0000","last_fullsized":"2023-03-03T08:01:21.415500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:20:30.614418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.408343+0000","last_change":"2023-03-03T08:01:21.405782+0000","last_active":"2023-03-03T08:01:21.408343+0000","last_peered":"2023-03-03T08:01:21.408343+0000","last_clean":"2023-03-03T08:01:21.408343+0000","last_became_active":"2023-03-03T08:01:21.405330+0000","last_became_peered":"2023-03-03T08:01:21.405330+0000","last_unstale":"2023-03-03T08:01:21.408343+0000","last_undegraded":"2023-03-03T08:01:21.408343+0000","last_fullsized":"2023-03-03T08:01:21.408343+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:31.468080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.415082+0000","last_change":"2023-03-03T08:01:21.410532+0000","last_active":"2023-03-03T08:01:21.415082+0000","last_peered":"2023-03-03T08:01:21.415082+0000","last_clean":"2023-03-03T08:01:21.415082+0000","last_became_active":"2023-03-03T08:01:21.410275+0000","last_became_peered":"2023-03-03T08:01:21.410275+0000","last_unstale":"2023-03-03T08:01:21.415082+0000","last_undegraded":"2023-03-03T08:01:21.415082+0000","last_fullsized":"2023-03-03T08:01:21.415082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:30:09.829742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.416517+0000","last_change":"2023-03-03T08:01:21.412820+0000","last_active":"2023-03-03T08:01:21.416517+0000","last_peered":"2023-03-03T08:01:21.416517+0000","last_clean":"2023-03-03T08:01:21.416517+0000","last_became_active":"2023-03-03T08:01:21.412562+0000","last_became_peered":"2023-03-03T08:01:21.412562+0000","last_unstale":"2023-03-03T08:01:21.416517+0000","last_undegraded":"2023-03-03T08:01:21.416517+0000","last_fullsized":"2023-03-03T08:01:21.416517+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:25.264212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417044+0000","last_change":"2023-03-03T08:01:21.415783+0000","last_active":"2023-03-03T08:01:21.417044+0000","last_peered":"2023-03-03T08:01:21.417044+0000","last_clean":"2023-03-03T08:01:21.417044+0000","last_became_active":"2023-03-03T08:01:21.415460+0000","last_became_peered":"2023-03-03T08:01:21.415460+0000","last_unstale":"2023-03-03T08:01:21.417044+0000","last_undegraded":"2023-03-03T08:01:21.417044+0000","last_fullsized":"2023-03-03T08:01:21.417044+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:22.792169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"77'4","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418387+0000","last_change":"2023-03-03T08:01:20.846783+0000","last_active":"2023-03-03T08:01:21.418387+0000","last_peered":"2023-03-03T08:01:21.418387+0000","last_clean":"2023-03-03T08:01:21.418387+0000","last_became_active":"2023-03-03T08:01:20.846528+0000","last_became_peered":"2023-03-03T08:01:20.846528+0000","last_unstale":"2023-03-03T08:01:21.418387+0000","last_undegraded":"2023-03-03T08:01:21.418387+0000","last_fullsized":"2023-03-03T08:01:21.418387+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:36:30.843302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.408321+0000","last_change":"2023-03-03T08:01:21.401300+0000","last_active":"2023-03-03T08:01:21.408321+0000","last_peered":"2023-03-03T08:01:21.408321+0000","last_clean":"2023-03-03T08:01:21.408321+0000","last_became_active":"2023-03-03T08:01:21.400951+0000","last_became_peered":"2023-03-03T08:01:21.400951+0000","last_unstale":"2023-03-03T08:01:21.408321+0000","last_undegraded":"2023-03-03T08:01:21.408321+0000","last_fullsized":"2023-03-03T08:01:21.408321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:07:47.722052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"77'1","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.416193+0000","last_change":"2023-03-03T08:01:20.846342+0000","last_active":"2023-03-03T08:01:21.416193+0000","last_peered":"2023-03-03T08:01:21.416193+0000","last_clean":"2023-03-03T08:01:21.416193+0000","last_became_active":"2023-03-03T08:01:20.845994+0000","last_became_peered":"2023-03-03T08:01:20.845994+0000","last_unstale":"2023-03-03T08:01:21.416193+0000","last_undegraded":"2023-03-03T08:01:21.416193+0000","last_fullsized":"2023-03-03T08:01:21.416193+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:22:55.388447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421626+0000","last_change":"2023-03-03T08:01:21.413013+0000","last_active":"2023-03-03T08:01:21.421626+0000","last_peered":"2023-03-03T08:01:21.421626+0000","last_clean":"2023-03-03T08:01:21.421626+0000","last_became_active":"2023-03-03T08:01:21.412339+0000","last_became_peered":"2023-03-03T08:01:21.412339+0000","last_unstale":"2023-03-03T08:01:21.421626+0000","last_undegraded":"2023-03-03T08:01:21.421626+0000","last_fullsized":"2023-03-03T08:01:21.421626+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:15:49.926634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"77'2","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.411438+0000","last_change":"2023-03-03T08:01:20.849422+0000","last_active":"2023-03-03T08:01:21.411438+0000","last_peered":"2023-03-03T08:01:21.411438+0000","last_clean":"2023-03-03T08:01:21.411438+0000","last_became_active":"2023-03-03T08:01:20.849065+0000","last_became_peered":"2023-03-03T08:01:20.849065+0000","last_unstale":"2023-03-03T08:01:21.411438+0000","last_undegraded":"2023-03-03T08:01:21.411438+0000","last_fullsized":"2023-03-03T08:01:21.411438+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:17.475047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.424097+0000","last_change":"2023-03-03T08:01:21.418555+0000","last_active":"2023-03-03T08:01:21.424097+0000","last_peered":"2023-03-03T08:01:21.424097+0000","last_clean":"2023-03-03T08:01:21.424097+0000","last_became_active":"2023-03-03T08:01:21.418286+0000","last_became_peered":"2023-03-03T08:01:21.418286+0000","last_unstale":"2023-03-03T08:01:21.424097+0000","last_undegraded":"2023-03-03T08:01:21.424097+0000","last_fullsized":"2023-03-03T08:01:21.424097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:31.468080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"77'2","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418704+0000","last_change":"2023-03-03T08:01:20.841488+0000","last_active":"2023-03-03T08:01:21.418704+0000","last_peered":"2023-03-03T08:01:21.418704+0000","last_clean":"2023-03-03T08:01:21.418704+0000","last_became_active":"2023-03-03T08:01:20.841131+0000","last_became_peered":"2023-03-03T08:01:20.841131+0000","last_unstale":"2023-03-03T08:01:21.418704+0000","last_undegraded":"2023-03-03T08:01:21.418704+0000","last_fullsized":"2023-03-03T08:01:21.418704+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:27:31.570735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407343+0000","last_change":"2023-03-03T08:01:21.399954+0000","last_active":"2023-03-03T08:01:21.407343+0000","last_peered":"2023-03-03T08:01:21.407343+0000","last_clean":"2023-03-03T08:01:21.407343+0000","last_became_active":"2023-03-03T08:01:21.399094+0000","last_became_peered":"2023-03-03T08:01:21.399094+0000","last_unstale":"2023-03-03T08:01:21.407343+0000","last_undegraded":"2023-03-03T08:01:21.407343+0000","last_fullsized":"2023-03-03T08:01:21.407343+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:32:17.007774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"77'1","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418296+0000","last_change":"2023-03-03T08:01:20.845956+0000","last_active":"2023-03-03T08:01:21.418296+0000","last_peered":"2023-03-03T08:01:21.418296+0000","last_clean":"2023-03-03T08:01:21.418296+0000","last_became_active":"2023-03-03T08:01:20.845648+0000","last_became_peered":"2023-03-03T08:01:20.845648+0000","last_unstale":"2023-03-03T08:01:21.418296+0000","last_undegraded":"2023-03-03T08:01:21.418296+0000","last_fullsized":"2023-03-03T08:01:21.418296+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:36:37.038190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407390+0000","last_change":"2023-03-03T08:01:21.400571+0000","last_active":"2023-03-03T08:01:21.407390+0000","last_peered":"2023-03-03T08:01:21.407390+0000","last_clean":"2023-03-03T08:01:21.407390+0000","last_became_active":"2023-03-03T08:01:21.399841+0000","last_became_peered":"2023-03-03T08:01:21.399841+0000","last_unstale":"2023-03-03T08:01:21.407390+0000","last_undegraded":"2023-03-03T08:01:21.407390+0000","last_fullsized":"2023-03-03T08:01:21.407390+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:26:21.966048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"77'1","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.411711+0000","last_change":"2023-03-03T08:01:20.832412+0000","last_active":"2023-03-03T08:01:21.411711+0000","last_peered":"2023-03-03T08:01:21.411711+0000","last_clean":"2023-03-03T08:01:21.411711+0000","last_became_active":"2023-03-03T08:01:20.832103+0000","last_became_peered":"2023-03-03T08:01:20.832103+0000","last_unstale":"2023-03-03T08:01:21.411711+0000","last_undegraded":"2023-03-03T08:01:21.411711+0000","last_fullsized":"2023-03-03T08:01:21.411711+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:10:21.482456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.424064+0000","last_change":"2023-03-03T08:01:21.417997+0000","last_active":"2023-03-03T08:01:21.424064+0000","last_peered":"2023-03-03T08:01:21.424064+0000","last_clean":"2023-03-03T08:01:21.424064+0000","last_became_active":"2023-03-03T08:01:21.417554+0000","last_became_peered":"2023-03-03T08:01:21.417554+0000","last_unstale":"2023-03-03T08:01:21.424064+0000","last_undegraded":"2023-03-03T08:01:21.424064+0000","last_fullsized":"2023-03-03T08:01:21.424064+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:10:41.011913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.389525+0000","last_change":"2023-03-03T08:01:20.842935+0000","last_active":"2023-03-03T08:01:21.389525+0000","last_peered":"2023-03-03T08:01:21.389525+0000","last_clean":"2023-03-03T08:01:21.389525+0000","last_became_active":"2023-03-03T08:01:20.842585+0000","last_became_peered":"2023-03-03T08:01:20.842585+0000","last_unstale":"2023-03-03T08:01:21.389525+0000","last_undegraded":"2023-03-03T08:01:21.389525+0000","last_fullsized":"2023-03-03T08:01:21.389525+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:05:27.877693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418662+0000","last_change":"2023-03-03T08:01:21.413744+0000","last_active":"2023-03-03T08:01:21.418662+0000","last_peered":"2023-03-03T08:01:21.418662+0000","last_clean":"2023-03-03T08:01:21.418662+0000","last_became_active":"2023-03-03T08:01:21.412814+0000","last_became_peered":"2023-03-03T08:01:21.412814+0000","last_unstale":"2023-03-03T08:01:21.418662+0000","last_undegraded":"2023-03-03T08:01:21.418662+0000","last_fullsized":"2023-03-03T08:01:21.418662+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:06.319757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.392653+0000","last_change":"2023-03-03T08:01:20.848411+0000","last_active":"2023-03-03T08:01:21.392653+0000","last_peered":"2023-03-03T08:01:21.392653+0000","last_clean":"2023-03-03T08:01:21.392653+0000","last_became_active":"2023-03-03T08:01:20.848174+0000","last_became_peered":"2023-03-03T08:01:20.848174+0000","last_unstale":"2023-03-03T08:01:21.392653+0000","last_undegraded":"2023-03-03T08:01:21.392653+0000","last_fullsized":"2023-03-03T08:01:21.392653+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:59:31.880965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407340+0000","last_change":"2023-03-03T08:01:21.402743+0000","last_active":"2023-03-03T08:01:21.407340+0000","last_peered":"2023-03-03T08:01:21.407340+0000","last_clean":"2023-03-03T08:01:21.407340+0000","last_became_active":"2023-03-03T08:01:21.402462+0000","last_became_peered":"2023-03-03T08:01:21.402462+0000","last_unstale":"2023-03-03T08:01:21.407340+0000","last_undegraded":"2023-03-03T08:01:21.407340+0000","last_fullsized":"2023-03-03T08:01:21.407340+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:53:57.306250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.392931+0000","last_change":"2023-03-03T08:01:20.847275+0000","last_active":"2023-03-03T08:01:21.392931+0000","last_peered":"2023-03-03T08:01:21.392931+0000","last_clean":"2023-03-03T08:01:21.392931+0000","last_became_active":"2023-03-03T08:01:20.846644+0000","last_became_peered":"2023-03-03T08:01:20.846644+0000","last_unstale":"2023-03-03T08:01:21.392931+0000","last_undegraded":"2023-03-03T08:01:21.392931+0000","last_fullsized":"2023-03-03T08:01:21.392931+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:33:56.386284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421094+0000","last_change":"2023-03-03T08:01:21.415505+0000","last_active":"2023-03-03T08:01:21.421094+0000","last_peered":"2023-03-03T08:01:21.421094+0000","last_clean":"2023-03-03T08:01:21.421094+0000","last_became_active":"2023-03-03T08:01:21.415194+0000","last_became_peered":"2023-03-03T08:01:21.415194+0000","last_unstale":"2023-03-03T08:01:21.421094+0000","last_undegraded":"2023-03-03T08:01:21.421094+0000","last_fullsized":"2023-03-03T08:01:21.421094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:11.768864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.389902+0000","last_change":"2023-03-03T08:01:20.843547+0000","last_active":"2023-03-03T08:01:21.389902+0000","last_peered":"2023-03-03T08:01:21.389902+0000","last_clean":"2023-03-03T08:01:21.389902+0000","last_became_active":"2023-03-03T08:01:20.843206+0000","last_became_peered":"2023-03-03T08:01:20.843206+0000","last_unstale":"2023-03-03T08:01:21.389902+0000","last_undegraded":"2023-03-03T08:01:21.389902+0000","last_fullsized":"2023-03-03T08:01:21.389902+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:30:08.816154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.400189+0000","last_change":"2023-03-03T08:01:21.395725+0000","last_active":"2023-03-03T08:01:21.400189+0000","last_peered":"2023-03-03T08:01:21.400189+0000","last_clean":"2023-03-03T08:01:21.400189+0000","last_became_active":"2023-03-03T08:01:21.395462+0000","last_became_peered":"2023-03-03T08:01:21.395462+0000","last_unstale":"2023-03-03T08:01:21.400189+0000","last_undegraded":"2023-03-03T08:01:21.400189+0000","last_fullsized":"2023-03-03T08:01:21.400189+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:11:17.517768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"77'3","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417035+0000","last_change":"2023-03-03T08:01:20.849625+0000","last_active":"2023-03-03T08:01:21.417035+0000","last_peered":"2023-03-03T08:01:21.417035+0000","last_clean":"2023-03-03T08:01:21.417035+0000","last_became_active":"2023-03-03T08:01:20.849325+0000","last_became_peered":"2023-03-03T08:01:20.849325+0000","last_unstale":"2023-03-03T08:01:21.417035+0000","last_undegraded":"2023-03-03T08:01:21.417035+0000","last_fullsized":"2023-03-03T08:01:21.417035+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:09:04.999100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.412342+0000","last_change":"2023-03-03T08:01:21.404444+0000","last_active":"2023-03-03T08:01:21.412342+0000","last_peered":"2023-03-03T08:01:21.412342+0000","last_clean":"2023-03-03T08:01:21.412342+0000","last_became_active":"2023-03-03T08:01:21.404029+0000","last_became_peered":"2023-03-03T08:01:21.404029+0000","last_unstale":"2023-03-03T08:01:21.412342+0000","last_undegraded":"2023-03-03T08:01:21.412342+0000","last_fullsized":"2023-03-03T08:01:21.412342+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:06.319757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"77'6","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.424463+0000","last_change":"2023-03-03T08:01:20.844608+0000","last_active":"2023-03-03T08:01:21.424463+0000","last_peered":"2023-03-03T08:01:21.424463+0000","last_clean":"2023-03-03T08:01:21.424463+0000","last_became_active":"2023-03-03T08:01:20.844308+0000","last_became_peered":"2023-03-03T08:01:20.844308+0000","last_unstale":"2023-03-03T08:01:21.424463+0000","last_undegraded":"2023-03-03T08:01:21.424463+0000","last_fullsized":"2023-03-03T08:01:21.424463+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:13:43.554478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.405619+0000","last_change":"2023-03-03T08:01:21.397824+0000","last_active":"2023-03-03T08:01:21.405619+0000","last_peered":"2023-03-03T08:01:21.405619+0000","last_clean":"2023-03-03T08:01:21.405619+0000","last_became_active":"2023-03-03T08:01:21.397501+0000","last_became_peered":"2023-03-03T08:01:21.397501+0000","last_unstale":"2023-03-03T08:01:21.405619+0000","last_undegraded":"2023-03-03T08:01:21.405619+0000","last_fullsized":"2023-03-03T08:01:21.405619+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:15:49.926634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"77'2","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418890+0000","last_change":"2023-03-03T08:01:20.832929+0000","last_active":"2023-03-03T08:01:21.418890+0000","last_peered":"2023-03-03T08:01:21.418890+0000","last_clean":"2023-03-03T08:01:21.418890+0000","last_became_active":"2023-03-03T08:01:20.832582+0000","last_became_peered":"2023-03-03T08:01:20.832582+0000","last_unstale":"2023-03-03T08:01:21.418890+0000","last_undegraded":"2023-03-03T08:01:21.418890+0000","last_fullsized":"2023-03-03T08:01:21.418890+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:13:19.662666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421566+0000","last_change":"2023-03-03T08:01:21.418465+0000","last_active":"2023-03-03T08:01:21.421566+0000","last_peered":"2023-03-03T08:01:21.421566+0000","last_clean":"2023-03-03T08:01:21.421566+0000","last_became_active":"2023-03-03T08:01:21.418161+0000","last_became_peered":"2023-03-03T08:01:21.418161+0000","last_unstale":"2023-03-03T08:01:21.421566+0000","last_undegraded":"2023-03-03T08:01:21.421566+0000","last_fullsized":"2023-03-03T08:01:21.421566+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:39:44.111270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.392157+0000","last_change":"2023-03-03T08:01:20.847931+0000","last_active":"2023-03-03T08:01:21.392157+0000","last_peered":"2023-03-03T08:01:21.392157+0000","last_clean":"2023-03-03T08:01:21.392157+0000","last_became_active":"2023-03-03T08:01:20.847650+0000","last_became_peered":"2023-03-03T08:01:20.847650+0000","last_unstale":"2023-03-03T08:01:21.392157+0000","last_undegraded":"2023-03-03T08:01:21.392157+0000","last_fullsized":"2023-03-03T08:01:21.392157+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:10.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421602+0000","last_change":"2023-03-03T08:01:21.419818+0000","last_active":"2023-03-03T08:01:21.421602+0000","last_peered":"2023-03-03T08:01:21.421602+0000","last_clean":"2023-03-03T08:01:21.421602+0000","last_became_active":"2023-03-03T08:01:21.419562+0000","last_became_peered":"2023-03-03T08:01:21.419562+0000","last_unstale":"2023-03-03T08:01:21.421602+0000","last_undegraded":"2023-03-03T08:01:21.421602+0000","last_fullsized":"2023-03-03T08:01:21.421602+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:13:40.650825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.392832+0000","last_change":"2023-03-03T08:01:20.849416+0000","last_active":"2023-03-03T08:01:21.392832+0000","last_peered":"2023-03-03T08:01:21.392832+0000","last_clean":"2023-03-03T08:01:21.392832+0000","last_became_active":"2023-03-03T08:01:20.849109+0000","last_became_peered":"2023-03-03T08:01:20.849109+0000","last_unstale":"2023-03-03T08:01:21.392832+0000","last_undegraded":"2023-03-03T08:01:21.392832+0000","last_fullsized":"2023-03-03T08:01:21.392832+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:05:04.070505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418680+0000","last_change":"2023-03-03T08:01:21.413916+0000","last_active":"2023-03-03T08:01:21.418680+0000","last_peered":"2023-03-03T08:01:21.418680+0000","last_clean":"2023-03-03T08:01:21.418680+0000","last_became_active":"2023-03-03T08:01:21.413151+0000","last_became_peered":"2023-03-03T08:01:21.413151+0000","last_unstale":"2023-03-03T08:01:21.418680+0000","last_undegraded":"2023-03-03T08:01:21.418680+0000","last_fullsized":"2023-03-03T08:01:21.418680+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:13:20.676253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"77'8","reported_seq":18,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.416354+0000","last_change":"2023-03-03T08:01:20.842341+0000","last_active":"2023-03-03T08:01:21.416354+0000","last_peered":"2023-03-03T08:01:21.416354+0000","last_clean":"2023-03-03T08:01:21.416354+0000","last_became_active":"2023-03-03T08:01:20.842059+0000","last_became_peered":"2023-03-03T08:01:20.842059+0000","last_unstale":"2023-03-03T08:01:21.416354+0000","last_undegraded":"2023-03-03T08:01:21.416354+0000","last_fullsized":"2023-03-03T08:01:21.416354+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:17.475047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421714+0000","last_change":"2023-03-03T08:01:21.415591+0000","last_active":"2023-03-03T08:01:21.421714+0000","last_peered":"2023-03-03T08:01:21.421714+0000","last_clean":"2023-03-03T08:01:21.421714+0000","last_became_active":"2023-03-03T08:01:21.415298+0000","last_became_peered":"2023-03-03T08:01:21.415298+0000","last_unstale":"2023-03-03T08:01:21.421714+0000","last_undegraded":"2023-03-03T08:01:21.421714+0000","last_fullsized":"2023-03-03T08:01:21.421714+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:37.071959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"77'2","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.414910+0000","last_change":"2023-03-03T08:01:20.837524+0000","last_active":"2023-03-03T08:01:21.414910+0000","last_peered":"2023-03-03T08:01:21.414910+0000","last_clean":"2023-03-03T08:01:21.414910+0000","last_became_active":"2023-03-03T08:01:20.837178+0000","last_became_peered":"2023-03-03T08:01:20.837178+0000","last_unstale":"2023-03-03T08:01:21.414910+0000","last_undegraded":"2023-03-03T08:01:21.414910+0000","last_fullsized":"2023-03-03T08:01:21.414910+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:21.778582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417095+0000","last_change":"2023-03-03T08:01:21.410084+0000","last_active":"2023-03-03T08:01:21.417095+0000","last_peered":"2023-03-03T08:01:21.417095+0000","last_clean":"2023-03-03T08:01:21.417095+0000","last_became_active":"2023-03-03T08:01:21.409589+0000","last_became_peered":"2023-03-03T08:01:21.409589+0000","last_unstale":"2023-03-03T08:01:21.417095+0000","last_undegraded":"2023-03-03T08:01:21.417095+0000","last_fullsized":"2023-03-03T08:01:21.417095+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:05:11.213383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.390693+0000","last_change":"2023-03-03T08:01:20.845764+0000","last_active":"2023-03-03T08:01:21.390693+0000","last_peered":"2023-03-03T08:01:21.390693+0000","last_clean":"2023-03-03T08:01:21.390693+0000","last_became_active":"2023-03-03T08:01:20.845424+0000","last_became_peered":"2023-03-03T08:01:20.845424+0000","last_unstale":"2023-03-03T08:01:21.390693+0000","last_undegraded":"2023-03-03T08:01:21.390693+0000","last_fullsized":"2023-03-03T08:01:21.390693+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:15:48.913046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421603+0000","last_change":"2023-03-03T08:01:21.411993+0000","last_active":"2023-03-03T08:01:21.421603+0000","last_peered":"2023-03-03T08:01:21.421603+0000","last_clean":"2023-03-03T08:01:21.421603+0000","last_became_active":"2023-03-03T08:01:21.411708+0000","last_became_peered":"2023-03-03T08:01:21.411708+0000","last_unstale":"2023-03-03T08:01:21.421603+0000","last_undegraded":"2023-03-03T08:01:21.421603+0000","last_fullsized":"2023-03-03T08:01:21.421603+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:04:07.988256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.393368+0000","last_change":"2023-03-03T08:01:20.847450+0000","last_active":"2023-03-03T08:01:21.393368+0000","last_peered":"2023-03-03T08:01:21.393368+0000","last_clean":"2023-03-03T08:01:21.393368+0000","last_became_active":"2023-03-03T08:01:20.847172+0000","last_became_peered":"2023-03-03T08:01:20.847172+0000","last_unstale":"2023-03-03T08:01:21.393368+0000","last_undegraded":"2023-03-03T08:01:21.393368+0000","last_fullsized":"2023-03-03T08:01:21.393368+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:53.717180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.402568+0000","last_change":"2023-03-03T08:01:21.397210+0000","last_active":"2023-03-03T08:01:21.402568+0000","last_peered":"2023-03-03T08:01:21.402568+0000","last_clean":"2023-03-03T08:01:21.402568+0000","last_became_active":"2023-03-03T08:01:21.396965+0000","last_became_peered":"2023-03-03T08:01:21.396965+0000","last_unstale":"2023-03-03T08:01:21.402568+0000","last_undegraded":"2023-03-03T08:01:21.402568+0000","last_fullsized":"2023-03-03T08:01:21.402568+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:36:31.007402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.390638+0000","last_change":"2023-03-03T08:01:20.846744+0000","last_active":"2023-03-03T08:01:21.390638+0000","last_peered":"2023-03-03T08:01:21.390638+0000","last_clean":"2023-03-03T08:01:21.390638+0000","last_became_active":"2023-03-03T08:01:20.846186+0000","last_became_peered":"2023-03-03T08:01:20.846186+0000","last_unstale":"2023-03-03T08:01:21.390638+0000","last_undegraded":"2023-03-03T08:01:21.390638+0000","last_fullsized":"2023-03-03T08:01:21.390638+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:36:37.038190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.419258+0000","last_change":"2023-03-03T08:01:21.413103+0000","last_active":"2023-03-03T08:01:21.419258+0000","last_peered":"2023-03-03T08:01:21.419258+0000","last_clean":"2023-03-03T08:01:21.419258+0000","last_became_active":"2023-03-03T08:01:21.412609+0000","last_became_peered":"2023-03-03T08:01:21.412609+0000","last_unstale":"2023-03-03T08:01:21.419258+0000","last_undegraded":"2023-03-03T08:01:21.419258+0000","last_fullsized":"2023-03-03T08:01:21.419258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:24.051431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.396587+0000","last_change":"2023-03-03T08:01:20.844629+0000","last_active":"2023-03-03T08:01:21.396587+0000","last_peered":"2023-03-03T08:01:21.396587+0000","last_clean":"2023-03-03T08:01:21.396587+0000","last_became_active":"2023-03-03T08:01:20.844354+0000","last_became_peered":"2023-03-03T08:01:20.844354+0000","last_unstale":"2023-03-03T08:01:21.396587+0000","last_undegraded":"2023-03-03T08:01:21.396587+0000","last_fullsized":"2023-03-03T08:01:21.396587+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:05:58.352008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418573+0000","last_change":"2023-03-03T08:01:21.413165+0000","last_active":"2023-03-03T08:01:21.418573+0000","last_peered":"2023-03-03T08:01:21.418573+0000","last_clean":"2023-03-03T08:01:21.418573+0000","last_became_active":"2023-03-03T08:01:21.412837+0000","last_became_peered":"2023-03-03T08:01:21.412837+0000","last_unstale":"2023-03-03T08:01:21.418573+0000","last_undegraded":"2023-03-03T08:01:21.418573+0000","last_fullsized":"2023-03-03T08:01:21.418573+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:11:56.348547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.396702+0000","last_change":"2023-03-03T08:01:20.849541+0000","last_active":"2023-03-03T08:01:21.396702+0000","last_peered":"2023-03-03T08:01:21.396702+0000","last_clean":"2023-03-03T08:01:21.396702+0000","last_became_active":"2023-03-03T08:01:20.849292+0000","last_became_peered":"2023-03-03T08:01:20.849292+0000","last_unstale":"2023-03-03T08:01:21.396702+0000","last_undegraded":"2023-03-03T08:01:21.396702+0000","last_fullsized":"2023-03-03T08:01:21.396702+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:01:20.228416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.408090+0000","last_change":"2023-03-03T08:01:21.402044+0000","last_active":"2023-03-03T08:01:21.408090+0000","last_peered":"2023-03-03T08:01:21.408090+0000","last_clean":"2023-03-03T08:01:21.408090+0000","last_became_active":"2023-03-03T08:01:21.401730+0000","last_became_peered":"2023-03-03T08:01:21.401730+0000","last_unstale":"2023-03-03T08:01:21.408090+0000","last_undegraded":"2023-03-03T08:01:21.408090+0000","last_fullsized":"2023-03-03T08:01:21.408090+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:11:56.348547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.406440+0000","last_change":"2023-03-03T08:01:20.843316+0000","last_active":"2023-03-03T08:01:21.406440+0000","last_peered":"2023-03-03T08:01:21.406440+0000","last_clean":"2023-03-03T08:01:21.406440+0000","last_became_active":"2023-03-03T08:01:20.843051+0000","last_became_peered":"2023-03-03T08:01:20.843051+0000","last_unstale":"2023-03-03T08:01:21.406440+0000","last_undegraded":"2023-03-03T08:01:21.406440+0000","last_fullsized":"2023-03-03T08:01:21.406440+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:34.165624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.412961+0000","last_change":"2023-03-03T08:01:21.403427+0000","last_active":"2023-03-03T08:01:21.412961+0000","last_peered":"2023-03-03T08:01:21.412961+0000","last_clean":"2023-03-03T08:01:21.412961+0000","last_became_active":"2023-03-03T08:01:21.403183+0000","last_became_peered":"2023-03-03T08:01:21.403183+0000","last_unstale":"2023-03-03T08:01:21.412961+0000","last_undegraded":"2023-03-03T08:01:21.412961+0000","last_fullsized":"2023-03-03T08:01:21.412961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:36:38.051778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.384905+0000","last_change":"2023-03-03T08:01:20.843157+0000","last_active":"2023-03-03T08:01:21.384905+0000","last_peered":"2023-03-03T08:01:21.384905+0000","last_clean":"2023-03-03T08:01:21.384905+0000","last_became_active":"2023-03-03T08:01:20.842876+0000","last_became_peered":"2023-03-03T08:01:20.842876+0000","last_unstale":"2023-03-03T08:01:21.384905+0000","last_undegraded":"2023-03-03T08:01:21.384905+0000","last_fullsized":"2023-03-03T08:01:21.384905+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:20.845420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407810+0000","last_change":"2023-03-03T08:01:21.405676+0000","last_active":"2023-03-03T08:01:21.407810+0000","last_peered":"2023-03-03T08:01:21.407810+0000","last_clean":"2023-03-03T08:01:21.407810+0000","last_became_active":"2023-03-03T08:01:21.405416+0000","last_became_peered":"2023-03-03T08:01:21.405416+0000","last_unstale":"2023-03-03T08:01:21.407810+0000","last_undegraded":"2023-03-03T08:01:21.407810+0000","last_fullsized":"2023-03-03T08:01:21.407810+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:04:51.424269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"77'1","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.424258+0000","last_change":"2023-03-03T08:01:20.847881+0000","last_active":"2023-03-03T08:01:21.424258+0000","last_peered":"2023-03-03T08:01:21.424258+0000","last_clean":"2023-03-03T08:01:21.424258+0000","last_became_active":"2023-03-03T08:01:20.847612+0000","last_became_peered":"2023-03-03T08:01:20.847612+0000","last_unstale":"2023-03-03T08:01:21.424258+0000","last_undegraded":"2023-03-03T08:01:21.424258+0000","last_fullsized":"2023-03-03T08:01:21.424258+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:22.628846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.828322+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T08:01:20.828322+0000","last_peered":"2023-03-03T08:01:20.828322+0000","last_clean":"2023-03-03T08:01:20.828322+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T08:01:20.828322+0000","last_undegraded":"2023-03-03T08:01:20.828322+0000","last_fullsized":"2023-03-03T08:01:20.828322+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407362+0000","last_change":"2023-03-03T08:01:21.401771+0000","last_active":"2023-03-03T08:01:21.407362+0000","last_peered":"2023-03-03T08:01:21.407362+0000","last_clean":"2023-03-03T08:01:21.407362+0000","last_became_active":"2023-03-03T08:01:21.401462+0000","last_became_peered":"2023-03-03T08:01:21.401462+0000","last_unstale":"2023-03-03T08:01:21.407362+0000","last_undegraded":"2023-03-03T08:01:21.407362+0000","last_fullsized":"2023-03-03T08:01:21.407362+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:03:54.417729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"77'2","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.413402+0000","last_change":"2023-03-03T08:01:20.837345+0000","last_active":"2023-03-03T08:01:21.413402+0000","last_peered":"2023-03-03T08:01:21.413402+0000","last_clean":"2023-03-03T08:01:21.413402+0000","last_became_active":"2023-03-03T08:01:20.837091+0000","last_became_peered":"2023-03-03T08:01:20.837091+0000","last_unstale":"2023-03-03T08:01:21.413402+0000","last_undegraded":"2023-03-03T08:01:21.413402+0000","last_fullsized":"2023-03-03T08:01:21.413402+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:04:06.974669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.827538+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T08:01:20.827538+0000","last_peered":"2023-03-03T08:01:20.827538+0000","last_clean":"2023-03-03T08:01:20.827538+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T08:01:20.827538+0000","last_undegraded":"2023-03-03T08:01:20.827538+0000","last_fullsized":"2023-03-03T08:01:20.827538+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418696+0000","last_change":"2023-03-03T08:01:21.414074+0000","last_active":"2023-03-03T08:01:21.418696+0000","last_peered":"2023-03-03T08:01:21.418696+0000","last_clean":"2023-03-03T08:01:21.418696+0000","last_became_active":"2023-03-03T08:01:21.413426+0000","last_became_peered":"2023-03-03T08:01:21.413426+0000","last_unstale":"2023-03-03T08:01:21.418696+0000","last_undegraded":"2023-03-03T08:01:21.418696+0000","last_fullsized":"2023-03-03T08:01:21.418696+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:48:09.597186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.388393+0000","last_change":"2023-03-03T08:01:20.846487+0000","last_active":"2023-03-03T08:01:21.388393+0000","last_peered":"2023-03-03T08:01:21.388393+0000","last_clean":"2023-03-03T08:01:21.388393+0000","last_became_active":"2023-03-03T08:01:20.846189+0000","last_became_peered":"2023-03-03T08:01:20.846189+0000","last_unstale":"2023-03-03T08:01:21.388393+0000","last_undegraded":"2023-03-03T08:01:21.388393+0000","last_fullsized":"2023-03-03T08:01:21.388393+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:43:36.170981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.841001+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T08:01:20.841001+0000","last_peered":"2023-03-03T08:01:20.841001+0000","last_clean":"2023-03-03T08:01:20.841001+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T08:01:20.841001+0000","last_undegraded":"2023-03-03T08:01:20.841001+0000","last_fullsized":"2023-03-03T08:01:20.841001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.416553+0000","last_change":"2023-03-03T08:01:21.409108+0000","last_active":"2023-03-03T08:01:21.416553+0000","last_peered":"2023-03-03T08:01:21.416553+0000","last_clean":"2023-03-03T08:01:21.416553+0000","last_became_active":"2023-03-03T08:01:21.408790+0000","last_became_peered":"2023-03-03T08:01:21.408790+0000","last_unstale":"2023-03-03T08:01:21.416553+0000","last_undegraded":"2023-03-03T08:01:21.416553+0000","last_fullsized":"2023-03-03T08:01:21.416553+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:48:45.104859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.388765+0000","last_change":"2023-03-03T08:01:20.845887+0000","last_active":"2023-03-03T08:01:21.388765+0000","last_peered":"2023-03-03T08:01:21.388765+0000","last_clean":"2023-03-03T08:01:21.388765+0000","last_became_active":"2023-03-03T08:01:20.845611+0000","last_became_peered":"2023-03-03T08:01:20.845611+0000","last_unstale":"2023-03-03T08:01:21.388765+0000","last_undegraded":"2023-03-03T08:01:21.388765+0000","last_fullsized":"2023-03-03T08:01:21.388765+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:05:27.877693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.827980+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T08:01:20.827980+0000","last_peered":"2023-03-03T08:01:20.827980+0000","last_clean":"2023-03-03T08:01:20.827980+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T08:01:20.827980+0000","last_undegraded":"2023-03-03T08:01:20.827980+0000","last_fullsized":"2023-03-03T08:01:20.827980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":213,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.837947+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T08:01:20.837947+0000","last_peered":"2023-03-03T08:01:20.837947+0000","last_clean":"2023-03-03T08:01:20.837947+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T08:01:20.837947+0000","last_undegraded":"2023-03-03T08:01:20.837947+0000","last_fullsized":"2023-03-03T08:01:20.837947+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.419478+0000","last_change":"2023-03-03T08:01:21.418514+0000","last_active":"2023-03-03T08:01:21.419478+0000","last_peered":"2023-03-03T08:01:21.419478+0000","last_clean":"2023-03-03T08:01:21.419478+0000","last_became_active":"2023-03-03T08:01:21.418220+0000","last_became_peered":"2023-03-03T08:01:21.418220+0000","last_unstale":"2023-03-03T08:01:21.419478+0000","last_undegraded":"2023-03-03T08:01:21.419478+0000","last_fullsized":"2023-03-03T08:01:21.419478+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:32:30.480453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.389498+0000","last_change":"2023-03-03T08:01:20.846292+0000","last_active":"2023-03-03T08:01:21.389498+0000","last_peered":"2023-03-03T08:01:21.389498+0000","last_clean":"2023-03-03T08:01:21.389498+0000","last_became_active":"2023-03-03T08:01:20.845788+0000","last_became_peered":"2023-03-03T08:01:20.845788+0000","last_unstale":"2023-03-03T08:01:21.389498+0000","last_undegraded":"2023-03-03T08:01:21.389498+0000","last_fullsized":"2023-03-03T08:01:21.389498+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:15:48.913046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":60,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.837840+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T08:01:20.837840+0000","last_peered":"2023-03-03T08:01:20.837840+0000","last_clean":"2023-03-03T08:01:20.837840+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T08:01:20.837840+0000","last_undegraded":"2023-03-03T08:01:20.837840+0000","last_fullsized":"2023-03-03T08:01:20.837840+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.410857+0000","last_change":"2023-03-03T08:01:21.401304+0000","last_active":"2023-03-03T08:01:21.410857+0000","last_peered":"2023-03-03T08:01:21.410857+0000","last_clean":"2023-03-03T08:01:21.410857+0000","last_became_active":"2023-03-03T08:01:21.401062+0000","last_became_peered":"2023-03-03T08:01:21.401062+0000","last_unstale":"2023-03-03T08:01:21.410857+0000","last_undegraded":"2023-03-03T08:01:21.410857+0000","last_fullsized":"2023-03-03T08:01:21.410857+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:20:30.614418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.385865+0000","last_change":"2023-03-03T08:01:20.845118+0000","last_active":"2023-03-03T08:01:21.385865+0000","last_peered":"2023-03-03T08:01:21.385865+0000","last_clean":"2023-03-03T08:01:21.385865+0000","last_became_active":"2023-03-03T08:01:20.844872+0000","last_became_peered":"2023-03-03T08:01:20.844872+0000","last_unstale":"2023-03-03T08:01:21.385865+0000","last_undegraded":"2023-03-03T08:01:21.385865+0000","last_fullsized":"2023-03-03T08:01:21.385865+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:31.917263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":60,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.841571+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T08:01:20.841571+0000","last_peered":"2023-03-03T08:01:20.841571+0000","last_clean":"2023-03-03T08:01:20.841571+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T08:01:20.841571+0000","last_undegraded":"2023-03-03T08:01:20.841571+0000","last_fullsized":"2023-03-03T08:01:20.841571+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.410876+0000","last_change":"2023-03-03T08:01:21.405444+0000","last_active":"2023-03-03T08:01:21.410876+0000","last_peered":"2023-03-03T08:01:21.410876+0000","last_clean":"2023-03-03T08:01:21.410876+0000","last_became_active":"2023-03-03T08:01:21.405144+0000","last_became_peered":"2023-03-03T08:01:21.405144+0000","last_unstale":"2023-03-03T08:01:21.410876+0000","last_undegraded":"2023-03-03T08:01:21.410876+0000","last_fullsized":"2023-03-03T08:01:21.410876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:18.426888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"77'1","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.414071+0000","last_change":"2023-03-03T08:01:20.849805+0000","last_active":"2023-03-03T08:01:21.414071+0000","last_peered":"2023-03-03T08:01:21.414071+0000","last_clean":"2023-03-03T08:01:21.414071+0000","last_became_active":"2023-03-03T08:01:20.849516+0000","last_became_peered":"2023-03-03T08:01:20.849516+0000","last_unstale":"2023-03-03T08:01:21.414071+0000","last_undegraded":"2023-03-03T08:01:21.414071+0000","last_fullsized":"2023-03-03T08:01:21.414071+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:49:07.162552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.828651+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T08:01:20.828651+0000","last_peered":"2023-03-03T08:01:20.828651+0000","last_clean":"2023-03-03T08:01:20.828651+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T08:01:20.828651+0000","last_undegraded":"2023-03-03T08:01:20.828651+0000","last_fullsized":"2023-03-03T08:01:20.828651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417090+0000","last_change":"2023-03-03T08:01:21.414678+0000","last_active":"2023-03-03T08:01:21.417090+0000","last_peered":"2023-03-03T08:01:21.417090+0000","last_clean":"2023-03-03T08:01:21.417090+0000","last_became_active":"2023-03-03T08:01:21.414362+0000","last_became_peered":"2023-03-03T08:01:21.414362+0000","last_unstale":"2023-03-03T08:01:21.417090+0000","last_undegraded":"2023-03-03T08:01:21.417090+0000","last_fullsized":"2023-03-03T08:01:21.417090+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:04:35.331292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"77'2","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417373+0000","last_change":"2023-03-03T08:01:20.832495+0000","last_active":"2023-03-03T08:01:21.417373+0000","last_peered":"2023-03-03T08:01:21.417373+0000","last_clean":"2023-03-03T08:01:21.417373+0000","last_became_active":"2023-03-03T08:01:20.832210+0000","last_became_peered":"2023-03-03T08:01:20.832210+0000","last_unstale":"2023-03-03T08:01:21.417373+0000","last_undegraded":"2023-03-03T08:01:21.417373+0000","last_fullsized":"2023-03-03T08:01:21.417373+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:14:54.101606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.833675+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T08:01:20.833675+0000","last_peered":"2023-03-03T08:01:20.833675+0000","last_clean":"2023-03-03T08:01:20.833675+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T08:01:20.833675+0000","last_undegraded":"2023-03-03T08:01:20.833675+0000","last_fullsized":"2023-03-03T08:01:20.833675+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574921,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11428,"kb_used_data":1500,"kb_used_omap":0,"kb_used_meta":9920,"kb_avail":94360412,"statfs":{"total":96636764160,"available":96625061888,"internally_reserved":0,"allocated":1536000,"data_stored":1008599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10158080},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66000000000000003,"15min":0.69999999999999996},"min":{"1min":0.376,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.86799999999999999,"5min":1.546,"15min":1.546},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70299999999999996,"15min":0.72699999999999998},"min":{"1min":0.54400000000000004,"5min":0.5,"15min":0.5},"max":{"1min":0.86399999999999999,"5min":1.702,"15min":1.702},"last":0.503}]},{"osd":1,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.79700000000000004,"15min":1.0529999999999999},"min":{"1min":0.434,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.105,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.72399999999999998,"15min":0.76400000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.93200000000000005,"5min":1.944,"15min":1.944},"last":0.69699999999999995}]},{"osd":2,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72299999999999998,"15min":0.71499999999999997},"min":{"1min":0.56599999999999995,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.91300000000000003,"5min":1.252,"15min":1.252},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.747,"15min":0.78600000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.95699999999999996,"5min":1.329,"15min":1.329},"last":0.71299999999999997}]},{"osd":3,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71699999999999997,"15min":0.71399999999999997},"min":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.97699999999999998,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73499999999999999,"15min":0.78300000000000003},"min":{"1min":0.51000000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.014,"5min":1.294,"15min":1.294},"last":0.66500000000000004}]},{"osd":4,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63800000000000001,"15min":0.65400000000000003},"min":{"1min":0.436,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84399999999999997,"5min":1.077,"15min":1.077},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67400000000000004,"15min":0.63600000000000001},"min":{"1min":0.45100000000000001,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.90500000000000003,"5min":1.081,"15min":1.081},"last":0.59299999999999997}]},{"osd":5,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.73499999999999999,"15min":0.78900000000000003},"min":{"1min":0.40699999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95899999999999996,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80400000000000005,"15min":0.81999999999999995},"min":{"1min":0.56100000000000005,"5min":0.496,"15min":0.496},"max":{"1min":1.1539999999999999,"5min":1.349,"15min":1.349},"last":0.78000000000000003}]},{"osd":6,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.80600000000000005,"15min":0.82599999999999996},"min":{"1min":0.56000000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0580000000000001,"5min":1.204,"15min":1.204},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.80500000000000005,"15min":0.81899999999999995},"min":{"1min":0.53900000000000003,"5min":0.499,"15min":0.499},"max":{"1min":1.129,"5min":1.194,"15min":1.194},"last":0.68300000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574922,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9804,"kb_used_data":900,"kb_used_omap":0,"kb_used_meta":8896,"kb_avail":94362036,"statfs":{"total":96636764160,"available":96626724864,"internally_reserved":0,"allocated":921600,"data_stored":410062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9109504},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.72899999999999998,"15min":0.69599999999999995},"min":{"1min":0.53800000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.103,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.70599999999999996,"15min":0.69999999999999996},"min":{"1min":0.50600000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.42899999999999999}]},{"osd":1,"last update":"Fri Mar 3 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72299999999999998,"15min":0.68999999999999995},"min":{"1min":0.48799999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1240000000000001,"5min":3.323,"15min":3.323},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.745,"15min":0.73399999999999999},"min":{"1min":0.504,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.70899999999999996}]},{"osd":2,"last update":"Fri Mar 3 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76100000000000001,"15min":0.73499999999999999},"min":{"1min":0.52700000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1140000000000001,"5min":1.54,"15min":1.54},"last":0.5},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.52100000000000002,"5min":0.495,"15min":0.495},"max":{"1min":1.167,"5min":1.698,"15min":1.698},"last":0.73099999999999998}]},{"osd":3,"last update":"Fri Mar 3 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.81699999999999995,"15min":0.79100000000000004},"min":{"1min":0.58499999999999996,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.1559999999999999,"5min":4.8959999999999999,"15min":4.8959999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.79000000000000004,"15min":0.77700000000000002},"min":{"1min":0.61799999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.181,"5min":1.343,"15min":1.343},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Mar 3 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.751,"15min":0.89500000000000002},"min":{"1min":0.505,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.95699999999999996,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.753},{"interface":"front","average":{"1min":0.748,"5min":0.76000000000000001,"15min":0.90800000000000003},"min":{"1min":0.53700000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.012,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.86199999999999999}]},{"osd":5,"last update":"Fri Mar 3 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.77000000000000002},"min":{"1min":0.52600000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.03,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.80200000000000005,"15min":0.82499999999999996},"min":{"1min":0.61799999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.202,"5min":1.429,"15min":1.429},"last":0.78100000000000003}]},{"osd":7,"last update":"Fri Mar 3 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81299999999999994,"15min":0.83699999999999997},"min":{"1min":0.60399999999999998,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.107,"5min":1.452,"15min":1.452},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.79900000000000004,"15min":0.86299999999999999},"min":{"1min":0.60199999999999998,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0820000000000001,"5min":1.579,"15min":1.579},"last":0.84399999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10256,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":9344,"kb_avail":94361584,"statfs":{"total":96636764160,"available":96626262016,"internally_reserved":0,"allocated":925696,"data_stored":411081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9568256},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.47899999999999998,"15min":0.50700000000000001},"min":{"1min":0.31,"5min":0.307,"15min":0.307},"max":{"1min":0.65700000000000003,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.53700000000000003,"15min":0.63300000000000001},"min":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.63200000000000001,"5min":3.972,"15min":3.972},"last":0.51500000000000001}]},{"osd":2,"last update":"Fri Mar 3 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.63,"15min":0.92700000000000005},"min":{"1min":0.38400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.621,"5min":11.897,"15min":11.897},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.70799999999999996,"15min":1.008},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73699999999999999,"5min":11.804,"15min":11.804},"last":0.53200000000000003}]},{"osd":3,"last update":"Fri Mar 3 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.69999999999999996,"15min":0.998},"min":{"1min":0.443,"5min":0.371,"15min":0.371},"max":{"1min":0.76600000000000001,"5min":11.843999999999999,"15min":11.843999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59799999999999998,"15min":0.71999999999999997},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81100000000000005,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.56200000000000006}]},{"osd":4,"last update":"Fri Mar 3 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.79100000000000004,"15min":0.85399999999999998},"min":{"1min":0.51500000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.94299999999999995,"5min":1.613,"15min":1.613},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.85799999999999998},"min":{"1min":0.65500000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.98599999999999999,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.72099999999999997}]},{"osd":5,"last update":"Fri Mar 3 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.80200000000000005,"15min":0.91000000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.95099999999999996,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.78300000000000003,"15min":0.86099999999999999},"min":{"1min":0.55300000000000005,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.86699999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.77200000000000002}]},{"osd":6,"last update":"Fri Mar 3 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.86499999999999999,"15min":0.98299999999999998},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.012,"5min":3.907,"15min":3.907},"last":0.754},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.84599999999999997,"15min":0.92800000000000005},"min":{"1min":0.627,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":0.95199999999999996,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.74199999999999999}]},{"osd":7,"last update":"Fri Mar 3 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.82499999999999996,"15min":0.877},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.417,"5min":1.419,"15min":1.419},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.85399999999999998,"15min":0.995},"min":{"1min":0.58199999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.3560000000000001,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":0.65500000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574921,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11100,"kb_used_data":1492,"kb_used_omap":0,"kb_used_meta":9600,"kb_avail":94360740,"statfs":{"total":96636764160,"available":96625397760,"internally_reserved":0,"allocated":1527808,"data_stored":1007558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9830400},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 08:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.44600000000000001,"15min":0.46800000000000003},"min":{"1min":0.311,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.59899999999999998,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.505,"15min":0.51700000000000002},"min":{"1min":0.36499999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.61899999999999999,"5min":0.871,"15min":0.871},"last":0.47499999999999998}]},{"osd":2,"last update":"Fri Mar 3 08:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52200000000000002,"15min":0.54700000000000004},"min":{"1min":0.38700000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72799999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57099999999999995,"15min":0.58999999999999997},"min":{"1min":0.441,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80800000000000005,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.441}]},{"osd":3,"last update":"Fri Mar 3 08:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53100000000000003,"15min":0.54900000000000004},"min":{"1min":0.36099999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71199999999999997,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60399999999999998,"15min":0.63700000000000001},"min":{"1min":0.48999999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83499999999999996,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.52300000000000002}]},{"osd":4,"last update":"Fri Mar 3 08:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.80900000000000005,"15min":0.83999999999999997},"min":{"1min":0.65800000000000003,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.95099999999999996,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78800000000000003,"15min":0.81699999999999995},"min":{"1min":0.57799999999999996,"5min":0.505,"15min":0.505},"max":{"1min":1.028,"5min":1.605,"15min":1.605},"last":0.70299999999999996}]},{"osd":5,"last update":"Fri Mar 3 08:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.79500000000000004,"15min":0.81899999999999995},"min":{"1min":0.58799999999999997,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.81200000000000006,"15min":0.82199999999999995},"min":{"1min":0.66900000000000004,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1,"5min":1.286,"15min":1.286},"last":0.72999999999999998}]},{"osd":6,"last update":"Fri Mar 3 08:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.82999999999999996,"15min":0.877},"min":{"1min":0.61399999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1419999999999999,"5min":1.623,"15min":1.623},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81799999999999995,"15min":0.85999999999999999},"min":{"1min":0.59899999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0129999999999999,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.71999999999999997}]},{"osd":7,"last update":"Fri Mar 3 08:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82899999999999996,"15min":0.877},"min":{"1min":0.63,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1020000000000001,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.82699999999999996,"15min":0.85799999999999998},"min":{"1min":0.64700000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":0.98699999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.64700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574922,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9956,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94361884,"statfs":{"total":96636764160,"available":96626569216,"internally_reserved":0,"allocated":946176,"data_stored":417916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.53700000000000003,"15min":0.83299999999999996},"min":{"1min":0.34699999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.57699999999999996,"5min":5.7889999999999997,"15min":5.7889999999999997},"last":0.307},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.55100000000000005,"15min":0.71299999999999997},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78000000000000003,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":0.48399999999999999}]},{"osd":1,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.56699999999999995,"15min":0.73199999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68799999999999994,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.68600000000000005,"15min":0.97299999999999998},"min":{"1min":0.45900000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.74099999999999999,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.50700000000000001}]},{"osd":3,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.71999999999999997,"15min":1.1659999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.752,"5min":12.961,"15min":12.961},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.68000000000000005,"15min":0.996},"min":{"1min":0.41599999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.71999999999999997,"5min":6.1449999999999996,"15min":6.1449999999999996},"last":0.40600000000000003}]},{"osd":4,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.92900000000000005,"15min":1.2370000000000001},"min":{"1min":0.57199999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.468,"5min":6.484,"15min":6.484},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.94899999999999995,"15min":1.2190000000000001},"min":{"1min":0.65000000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":0.96599999999999997,"5min":6.5510000000000002,"15min":6.5510000000000002},"last":0.75800000000000001}]},{"osd":5,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.86099999999999999,"15min":1.143},"min":{"1min":0.57299999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.304,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.91400000000000003,"15min":1.2},"min":{"1min":0.58599999999999997,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.95899999999999996,"5min":6.6369999999999996,"15min":6.6369999999999996},"last":0.65100000000000002}]},{"osd":6,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.93999999999999995,"15min":1.206},"min":{"1min":0.65400000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.393,"5min":6.625,"15min":6.625},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.94599999999999995,"15min":1.23},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.347,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.78900000000000003}]},{"osd":7,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.89300000000000002,"15min":1.1910000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.91900000000000004,"5min":6.7009999999999996,"15min":6.7009999999999996},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.91300000000000003,"15min":1.1779999999999999},"min":{"1min":0.621,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":0.98599999999999999,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.66200000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574920,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14432,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":13504,"kb_avail":94357408,"statfs":{"total":96636764160,"available":96621985792,"internally_reserved":0,"allocated":942080,"data_stored":418775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13828096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.53500000000000003,"15min":0.80300000000000005},"min":{"1min":0.32400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.74199999999999999,"5min":9.5820000000000007,"15min":9.5820000000000007},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.60399999999999998,"15min":0.874},"min":{"1min":0.36499999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72399999999999998,"5min":9.7840000000000007,"15min":9.7840000000000007},"last":0.40799999999999997}]},{"osd":1,"last update":"Fri Mar 3 08:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.49399999999999999,"15min":0.52000000000000002},"min":{"1min":0.34399999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68400000000000005,"5min":1,"15min":1},"last":0.375},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.64800000000000002,"15min":0.95399999999999996},"min":{"1min":0.39500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77400000000000002,"5min":9.702,"15min":9.702},"last":0.58399999999999996}]},{"osd":2,"last update":"Fri Mar 3 08:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.65800000000000003,"15min":0.90700000000000003},"min":{"1min":0.46200000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81200000000000006,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.70699999999999996,"15min":0.99099999999999999},"min":{"1min":0.49399999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.93400000000000005,"5min":9.734,"15min":9.734},"last":0.54800000000000004}]},{"osd":4,"last update":"Fri Mar 3 08:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.86199999999999999,"15min":1.1319999999999999},"min":{"1min":0.59899999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.96699999999999997,"5min":9.8629999999999995,"15min":9.8629999999999995},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.81799999999999995,"15min":0.84099999999999997},"min":{"1min":0.55700000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.97099999999999997,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":0.71699999999999997}]},{"osd":5,"last update":"Fri Mar 3 08:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.80700000000000005,"15min":0.82799999999999996},"min":{"1min":0.59999999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.88400000000000001,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.88600000000000001,"15min":1.1839999999999999},"min":{"1min":0.58999999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.99099999999999999,"5min":9.6600000000000001,"15min":9.6600000000000001},"last":0.747}]},{"osd":6,"last update":"Fri Mar 3 08:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.89900000000000002,"15min":1.1990000000000001},"min":{"1min":0.629,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.89800000000000002,"5min":9.8190000000000008,"15min":9.8190000000000008},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80900000000000005,"15min":0.86099999999999999},"min":{"1min":0.60299999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.014,"5min":1.609,"15min":1.609},"last":0.65600000000000003}]},{"osd":7,"last update":"Fri Mar 3 08:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.91700000000000004,"15min":1.2350000000000001},"min":{"1min":0.65400000000000003,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.024,"5min":9.625,"15min":9.625},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.92600000000000005,"15min":1.208},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":0.97599999999999998,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":0.70699999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574920,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10480,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":9536,"kb_avail":94361360,"statfs":{"total":96636764160,"available":96626032640,"internally_reserved":0,"allocated":958464,"data_stored":419861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9764864},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.79500000000000004,"15min":0.96399999999999997},"min":{"1min":0.58499999999999996,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.96399999999999997,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.69799999999999995,"15min":0.67000000000000004},"min":{"1min":0.45900000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89300000000000002,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.58199999999999996}]},{"osd":1,"last update":"Fri Mar 3 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.83899999999999997,"15min":1.079},"min":{"1min":0.53500000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94399999999999995,"5min":9.5709999999999997,"15min":9.5709999999999997},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.79700000000000004,"15min":0.91300000000000003},"min":{"1min":0.56499999999999995,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.89000000000000001,"5min":5.266,"15min":5.266},"last":0.61299999999999999}]},{"osd":2,"last update":"Fri Mar 3 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.83799999999999997,"15min":1.0509999999999999},"min":{"1min":0.53700000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92700000000000005,"5min":8.4529999999999994,"15min":8.4529999999999994},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.82499999999999996,"15min":0.98399999999999999},"min":{"1min":0.59999999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.97899999999999998,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.73999999999999999}]},{"osd":3,"last update":"Fri Mar 3 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.73599999999999999,"15min":0.73499999999999999},"min":{"1min":0.54300000000000004,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.97899999999999998,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.78900000000000003,"15min":0.80200000000000005},"min":{"1min":0.64100000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.99299999999999999,"5min":1.389,"15min":1.389},"last":0.81999999999999995}]},{"osd":5,"last update":"Fri Mar 3 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.71399999999999997,"15min":0.73999999999999999},"min":{"1min":0.49299999999999999,"5min":0.434,"15min":0.434},"max":{"1min":0.98799999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.71599999999999997,"15min":0.71799999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91100000000000003,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.68899999999999995}]},{"osd":6,"last update":"Fri Mar 3 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76100000000000001,"15min":0.77700000000000002},"min":{"1min":0.54700000000000004,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0029999999999999,"5min":1.032,"15min":1.032},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.80000000000000004,"15min":0.81599999999999995},"min":{"1min":0.60399999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.96499999999999997,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.70799999999999996}]},{"osd":7,"last update":"Fri Mar 3 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.78400000000000003,"15min":0.81799999999999995},"min":{"1min":0.59199999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0429999999999999,"5min":1.679,"15min":1.679},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.85699999999999998,"15min":0.92000000000000004},"min":{"1min":0.64900000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.085,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.90800000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574921,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10012,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":9088,"kb_avail":94361828,"statfs":{"total":96636764160,"available":96626511872,"internally_reserved":0,"allocated":937984,"data_stored":417224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9306112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.64300000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.75900000000000001,"15min":0.745},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0580000000000001,"5min":2.528,"15min":2.528},"last":0.61599999999999999}]},{"osd":1,"last update":"Fri Mar 3 08:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73799999999999999,"15min":0.746},"min":{"1min":0.55000000000000004,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.97299999999999998,"5min":1.038,"15min":1.038},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68799999999999994,"15min":0.64200000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.89400000000000002,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.53800000000000003}]},{"osd":2,"last update":"Fri Mar 3 08:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.69199999999999995,"15min":0.68300000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72299999999999998,"15min":0.71499999999999997},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.997,"5min":1.321,"15min":1.321},"last":0.55400000000000005}]},{"osd":3,"last update":"Fri Mar 3 08:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.72099999999999997,"15min":0.69699999999999995},"min":{"1min":0.51400000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.754,"15min":0.73799999999999999},"min":{"1min":0.56999999999999995,"5min":0.501,"15min":0.501},"max":{"1min":0.92800000000000005,"5min":1.012,"15min":1.012},"last":0.59599999999999997}]},{"osd":4,"last update":"Fri Mar 3 08:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.67200000000000004,"15min":0.66500000000000004},"min":{"1min":0.47899999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90900000000000003,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79300000000000004,"15min":0.78300000000000003},"min":{"1min":0.66600000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.57599999999999996}]},{"osd":6,"last update":"Fri Mar 3 08:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.746,"15min":0.79200000000000004},"min":{"1min":0.54000000000000004,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.012,"5min":1.071,"15min":1.071},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.71699999999999997,"15min":0.72299999999999998},"min":{"1min":0.48999999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.93899999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.496}]},{"osd":7,"last update":"Fri Mar 3 08:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.58699999999999997,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.80200000000000005,"15min":0.82799999999999996},"min":{"1min":0.60299999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.90500000000000003,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.67000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:01:26.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:01:26.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.664+0000 7fa58ed9d700 1 Processor -- start 2023-03-03T08:01:26.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.665+0000 7fa58ed9d700 1 -- start start 2023-03-03T08:01:26.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.666+0000 7fa58ed9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa5880b51c0 0x7fa588160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:26.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.666+0000 7fa58ed9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa5880d1580 0x7fa5880d19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:26.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.666+0000 7fa58ed9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5881614c0 con 0x7fa588146360 2023-03-03T08:01:26.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.666+0000 7fa58ed9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5880d10f0 con 0x7fa5880b51c0 2023-03-03T08:01:26.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.666+0000 7fa58ed9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5880cdf20 con 0x7fa5880d1580 2023-03-03T08:01:26.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.666+0000 7fa58d59a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa5880b51c0 0x7fa588160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:26.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.666+0000 7fa58d59a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa5880b51c0 0x7fa588160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60208/0 (socket says 172.21.15.98:60208) 2023-03-03T08:01:26.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.666+0000 7fa58e59c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa5880d1580 0x7fa5880d19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:26.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.666+0000 7fa58dd9b700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fa588146360 0x7fa5880b4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:53358/0 (socket says 172.21.15.98:53358) 2023-03-03T08:01:26.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.666+0000 7fa58d59a700 1 -- 172.21.15.98:0/4210897199 learned_addr learned my addr 172.21.15.98:0/4210897199 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:26.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.667+0000 7fa58d59a700 1 -- 172.21.15.98:0/4210897199 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa5880d1580 msgr2=0x7fa5880d19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:26.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.667+0000 7fa58d59a700 1 --2- 172.21.15.98:0/4210897199 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa5880d1580 0x7fa5880d19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.667+0000 7fa58d59a700 1 -- 172.21.15.98:0/4210897199 >> v1:172.21.15.100:6789/0 conn(0x7fa588146360 legacy=0x7fa5880b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:26.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.667+0000 7fa58d59a700 1 -- 172.21.15.98:0/4210897199 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5880cd680 con 0x7fa5880b51c0 2023-03-03T08:01:26.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.667+0000 7fa58d59a700 1 --2- 172.21.15.98:0/4210897199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5880b51c0 0x7fa588160f80 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fa5780098f0 tx=0x7fa578009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b060b31a7afa79 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:26.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.668+0000 7fa58cd99700 1 -- 172.21.15.98:0/4210897199 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa578011830 con 0x7fa5880b51c0 2023-03-03T08:01:26.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.668+0000 7fa58cd99700 1 -- 172.21.15.98:0/4210897199 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa578011990 con 0x7fa5880b51c0 2023-03-03T08:01:26.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.668+0000 7fa58cd99700 1 -- 172.21.15.98:0/4210897199 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa578011d20 con 0x7fa5880b51c0 2023-03-03T08:01:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.668+0000 7fa58ed9d700 1 -- 172.21.15.98:0/4210897199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5880b51c0 msgr2=0x7fa588160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.668+0000 7fa58ed9d700 1 --2- 172.21.15.98:0/4210897199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5880b51c0 0x7fa588160f80 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fa5780098f0 tx=0x7fa578009c00 comp rx=0 tx=0).stop 2023-03-03T08:01:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.668+0000 7fa58ed9d700 1 -- 172.21.15.98:0/4210897199 shutdown_connections 2023-03-03T08:01:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.668+0000 7fa58ed9d700 1 --2- 172.21.15.98:0/4210897199 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa5880d1580 0x7fa5880d19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.668+0000 7fa58ed9d700 1 --2- 172.21.15.98:0/4210897199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5880b51c0 0x7fa588160f80 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.668+0000 7fa58ed9d700 1 -- 172.21.15.98:0/4210897199 >> 172.21.15.98:0/4210897199 conn(0x7fa58806b8e0 msgr2=0x7fa5880cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.668+0000 7fa58ed9d700 1 -- 172.21.15.98:0/4210897199 shutdown_connections 2023-03-03T08:01:26.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.668+0000 7fa58ed9d700 1 -- 172.21.15.98:0/4210897199 wait complete. 2023-03-03T08:01:26.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.670+0000 7fa58ed9d700 1 Processor -- start 2023-03-03T08:01:26.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.671+0000 7fa58ed9d700 1 -- start start 2023-03-03T08:01:26.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5880b51c0 0x7fa5880c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:26.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58ed9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5880d1580 0x7fa5880cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:26.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58ed9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa588146360 0x7fa5880cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:26.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58ed9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5880cd300 con 0x7fa5880b51c0 2023-03-03T08:01:26.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58ed9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5880cc780 con 0x7fa5880d1580 2023-03-03T08:01:26.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58ed9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5880cca60 con 0x7fa588146360 2023-03-03T08:01:26.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5880b51c0 0x7fa5880c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:26.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5880b51c0 0x7fa5880c5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60218/0 (socket says 172.21.15.98:60218) 2023-03-03T08:01:26.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58dd9b700 1 -- 172.21.15.98:0/3154437460 learned_addr learned my addr 172.21.15.98:0/3154437460 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:26.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58e59c700 1 --2- 172.21.15.98:0/3154437460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa588146360 0x7fa5880cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:26.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58d59a700 1 --2- 172.21.15.98:0/3154437460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5880d1580 0x7fa5880cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:26.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58dd9b700 1 -- 172.21.15.98:0/3154437460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa588146360 msgr2=0x7fa5880cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:26.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58dd9b700 1 --2- 172.21.15.98:0/3154437460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa588146360 0x7fa5880cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58dd9b700 1 -- 172.21.15.98:0/3154437460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5880d1580 msgr2=0x7fa5880cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:26.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58dd9b700 1 --2- 172.21.15.98:0/3154437460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5880d1580 0x7fa5880cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58dd9b700 1 -- 172.21.15.98:0/3154437460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa578009550 con 0x7fa5880b51c0 2023-03-03T08:01:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.672+0000 7fa58d59a700 1 --2- 172.21.15.98:0/3154437460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5880d1580 0x7fa5880cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.673+0000 7fa58dd9b700 1 --2- 172.21.15.98:0/3154437460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5880b51c0 0x7fa5880c5d00 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fa58400c520 tx=0x7fa58400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.673+0000 7fa57e7fc700 1 -- 172.21.15.98:0/3154437460 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa584005cb0 con 0x7fa5880b51c0 2023-03-03T08:01:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.673+0000 7fa57e7fc700 1 -- 172.21.15.98:0/3154437460 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa584005e10 con 0x7fa5880b51c0 2023-03-03T08:01:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.673+0000 7fa57e7fc700 1 -- 172.21.15.98:0/3154437460 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5840072d0 con 0x7fa5880b51c0 2023-03-03T08:01:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.673+0000 7fa58ed9d700 1 -- 172.21.15.98:0/3154437460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5880cdc40 con 0x7fa5880b51c0 2023-03-03T08:01:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.673+0000 7fa58ed9d700 1 -- 172.21.15.98:0/3154437460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5880cd960 con 0x7fa5880b51c0 2023-03-03T08:01:26.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.674+0000 7fa58ed9d700 1 -- 172.21.15.98:0/3154437460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5880d0d80 con 0x7fa5880b51c0 2023-03-03T08:01:26.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.676+0000 7fa57e7fc700 1 -- 172.21.15.98:0/3154437460 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fa584019030 con 0x7fa5880b51c0 2023-03-03T08:01:26.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.677+0000 7fa57e7fc700 1 --2- 172.21.15.98:0/3154437460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa56c0684a0 0x7fa56c06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:26.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.677+0000 7fa57e7fc700 1 -- 172.21.15.98:0/3154437460 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa584085120 con 0x7fa5880b51c0 2023-03-03T08:01:26.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.677+0000 7fa58d59a700 1 --2- 172.21.15.98:0/3154437460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa56c0684a0 0x7fa56c06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:26.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.678+0000 7fa58d59a700 1 --2- 172.21.15.98:0/3154437460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa56c0684a0 0x7fa56c06a950 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa5880b55b0 tx=0x7fa578008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:26.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.681+0000 7fa57e7fc700 1 -- 172.21.15.98:0/3154437460 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa58404b5e0 con 0x7fa5880b51c0 2023-03-03T08:01:26.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.871+0000 7fa58ed9d700 1 -- 172.21.15.98:0/3154437460 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa5881614c0 con 0x7fa56c0684a0 2023-03-03T08:01:26.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.880+0000 7fa57e7fc700 1 -- 172.21.15.98:0/3154437460 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310950 (secure 0 0 0) 0x7fa5881614c0 con 0x7fa56c0684a0 2023-03-03T08:01:26.885 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:26.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.886+0000 7fa58ed9d700 1 -- 172.21.15.98:0/3154437460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa56c0684a0 msgr2=0x7fa56c06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:26.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.886+0000 7fa58ed9d700 1 --2- 172.21.15.98:0/3154437460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa56c0684a0 0x7fa56c06a950 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa5880b55b0 tx=0x7fa578008030 comp rx=0 tx=0).stop 2023-03-03T08:01:26.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.886+0000 7fa58ed9d700 1 -- 172.21.15.98:0/3154437460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5880b51c0 msgr2=0x7fa5880c5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:26.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.886+0000 7fa58ed9d700 1 --2- 172.21.15.98:0/3154437460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5880b51c0 0x7fa5880c5d00 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fa58400c520 tx=0x7fa58400c8e0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.886+0000 7fa58ed9d700 1 -- 172.21.15.98:0/3154437460 shutdown_connections 2023-03-03T08:01:26.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.886+0000 7fa58ed9d700 1 --2- 172.21.15.98:0/3154437460 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa56c0684a0 0x7fa56c06a950 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.886+0000 7fa58ed9d700 1 --2- 172.21.15.98:0/3154437460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5880b51c0 0x7fa5880c5d00 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.886+0000 7fa58ed9d700 1 --2- 172.21.15.98:0/3154437460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa588146360 0x7fa5880cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.886+0000 7fa58ed9d700 1 --2- 172.21.15.98:0/3154437460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5880d1580 0x7fa5880cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:26.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.886+0000 7fa58ed9d700 1 -- 172.21.15.98:0/3154437460 >> 172.21.15.98:0/3154437460 conn(0x7fa58806b8e0 msgr2=0x7fa5880b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:26.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.886+0000 7fa58ed9d700 1 -- 172.21.15.98:0/3154437460 shutdown_connections 2023-03-03T08:01:26.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:26.886+0000 7fa58ed9d700 1 -- 172.21.15.98:0/3154437460 wait complete. 2023-03-03T08:01:26.889 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T08:01:26.914 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2023-03-03T08:01:26.254106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":139,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":87468,"kb_used_data":8492,"kb_used_omap":0,"kb_used_meta":78912,"kb_avail":754887252,"statfs":{"total":773094113280,"available":773004546048,"internally_reserved":0,"allocated":8695808,"data_stored":4511076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":80805888},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000983"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.419214+0000","last_change":"2023-03-03T08:01:21.416277+0000","last_active":"2023-03-03T08:01:21.419214+0000","last_peered":"2023-03-03T08:01:21.419214+0000","last_clean":"2023-03-03T08:01:21.419214+0000","last_became_active":"2023-03-03T08:01:21.415936+0000","last_became_peered":"2023-03-03T08:01:21.415936+0000","last_unstale":"2023-03-03T08:01:21.419214+0000","last_undegraded":"2023-03-03T08:01:21.419214+0000","last_fullsized":"2023-03-03T08:01:21.419214+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:26:14.276515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407560+0000","last_change":"2023-03-03T08:01:21.399865+0000","last_active":"2023-03-03T08:01:21.407560+0000","last_peered":"2023-03-03T08:01:21.407560+0000","last_clean":"2023-03-03T08:01:21.407560+0000","last_became_active":"2023-03-03T08:01:21.399529+0000","last_became_peered":"2023-03-03T08:01:21.399529+0000","last_unstale":"2023-03-03T08:01:21.407560+0000","last_undegraded":"2023-03-03T08:01:21.407560+0000","last_fullsized":"2023-03-03T08:01:21.407560+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:37:59.548033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407863+0000","last_change":"2023-03-03T08:01:21.402712+0000","last_active":"2023-03-03T08:01:21.407863+0000","last_peered":"2023-03-03T08:01:21.407863+0000","last_clean":"2023-03-03T08:01:21.407863+0000","last_became_active":"2023-03-03T08:01:21.402451+0000","last_became_peered":"2023-03-03T08:01:21.402451+0000","last_unstale":"2023-03-03T08:01:21.407863+0000","last_undegraded":"2023-03-03T08:01:21.407863+0000","last_fullsized":"2023-03-03T08:01:21.407863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:13:20.676253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421293+0000","last_change":"2023-03-03T08:01:21.417804+0000","last_active":"2023-03-03T08:01:21.421293+0000","last_peered":"2023-03-03T08:01:21.421293+0000","last_clean":"2023-03-03T08:01:21.421293+0000","last_became_active":"2023-03-03T08:01:21.417541+0000","last_became_peered":"2023-03-03T08:01:21.417541+0000","last_unstale":"2023-03-03T08:01:21.421293+0000","last_undegraded":"2023-03-03T08:01:21.421293+0000","last_fullsized":"2023-03-03T08:01:21.421293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:18.488635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.415089+0000","last_change":"2023-03-03T08:01:21.411867+0000","last_active":"2023-03-03T08:01:21.415089+0000","last_peered":"2023-03-03T08:01:21.415089+0000","last_clean":"2023-03-03T08:01:21.415089+0000","last_became_active":"2023-03-03T08:01:21.411572+0000","last_became_peered":"2023-03-03T08:01:21.411572+0000","last_unstale":"2023-03-03T08:01:21.415089+0000","last_undegraded":"2023-03-03T08:01:21.415089+0000","last_fullsized":"2023-03-03T08:01:21.415089+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:41:35.166694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.420555+0000","last_change":"2023-03-03T08:01:21.419143+0000","last_active":"2023-03-03T08:01:21.420555+0000","last_peered":"2023-03-03T08:01:21.420555+0000","last_clean":"2023-03-03T08:01:21.420555+0000","last_became_active":"2023-03-03T08:01:21.418890+0000","last_became_peered":"2023-03-03T08:01:21.418890+0000","last_unstale":"2023-03-03T08:01:21.420555+0000","last_undegraded":"2023-03-03T08:01:21.420555+0000","last_fullsized":"2023-03-03T08:01:21.420555+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:30:42.888519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418581+0000","last_change":"2023-03-03T08:01:21.412809+0000","last_active":"2023-03-03T08:01:21.418581+0000","last_peered":"2023-03-03T08:01:21.418581+0000","last_clean":"2023-03-03T08:01:21.418581+0000","last_became_active":"2023-03-03T08:01:21.412418+0000","last_became_peered":"2023-03-03T08:01:21.412418+0000","last_unstale":"2023-03-03T08:01:21.418581+0000","last_undegraded":"2023-03-03T08:01:21.418581+0000","last_fullsized":"2023-03-03T08:01:21.418581+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:30:09.829742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.415126+0000","last_change":"2023-03-03T08:01:21.412254+0000","last_active":"2023-03-03T08:01:21.415126+0000","last_peered":"2023-03-03T08:01:21.415126+0000","last_clean":"2023-03-03T08:01:21.415126+0000","last_became_active":"2023-03-03T08:01:21.411988+0000","last_became_peered":"2023-03-03T08:01:21.411988+0000","last_unstale":"2023-03-03T08:01:21.415126+0000","last_undegraded":"2023-03-03T08:01:21.415126+0000","last_fullsized":"2023-03-03T08:01:21.415126+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:03:06.355953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421694+0000","last_change":"2023-03-03T08:01:21.412728+0000","last_active":"2023-03-03T08:01:21.421694+0000","last_peered":"2023-03-03T08:01:21.421694+0000","last_clean":"2023-03-03T08:01:21.421694+0000","last_became_active":"2023-03-03T08:01:21.412449+0000","last_became_peered":"2023-03-03T08:01:21.412449+0000","last_unstale":"2023-03-03T08:01:21.421694+0000","last_undegraded":"2023-03-03T08:01:21.421694+0000","last_fullsized":"2023-03-03T08:01:21.421694+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:47:50.975464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417638+0000","last_change":"2023-03-03T08:01:21.410465+0000","last_active":"2023-03-03T08:01:21.417638+0000","last_peered":"2023-03-03T08:01:21.417638+0000","last_clean":"2023-03-03T08:01:21.417638+0000","last_became_active":"2023-03-03T08:01:21.410184+0000","last_became_peered":"2023-03-03T08:01:21.410184+0000","last_unstale":"2023-03-03T08:01:21.417638+0000","last_undegraded":"2023-03-03T08:01:21.417638+0000","last_fullsized":"2023-03-03T08:01:21.417638+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:49:08.176139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407410+0000","last_change":"2023-03-03T08:01:21.400759+0000","last_active":"2023-03-03T08:01:21.407410+0000","last_peered":"2023-03-03T08:01:21.407410+0000","last_clean":"2023-03-03T08:01:21.407410+0000","last_became_active":"2023-03-03T08:01:21.400216+0000","last_became_peered":"2023-03-03T08:01:21.400216+0000","last_unstale":"2023-03-03T08:01:21.407410+0000","last_undegraded":"2023-03-03T08:01:21.407410+0000","last_fullsized":"2023-03-03T08:01:21.407410+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:25:46.913975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.408582+0000","last_change":"2023-03-03T08:01:21.405052+0000","last_active":"2023-03-03T08:01:21.408582+0000","last_peered":"2023-03-03T08:01:21.408582+0000","last_clean":"2023-03-03T08:01:21.408582+0000","last_became_active":"2023-03-03T08:01:21.404593+0000","last_became_peered":"2023-03-03T08:01:21.404593+0000","last_unstale":"2023-03-03T08:01:21.408582+0000","last_undegraded":"2023-03-03T08:01:21.408582+0000","last_fullsized":"2023-03-03T08:01:21.408582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:05:28.891281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.410906+0000","last_change":"2023-03-03T08:01:21.408597+0000","last_active":"2023-03-03T08:01:21.410906+0000","last_peered":"2023-03-03T08:01:21.410906+0000","last_clean":"2023-03-03T08:01:21.410906+0000","last_became_active":"2023-03-03T08:01:21.408282+0000","last_became_peered":"2023-03-03T08:01:21.408282+0000","last_unstale":"2023-03-03T08:01:21.410906+0000","last_undegraded":"2023-03-03T08:01:21.410906+0000","last_fullsized":"2023-03-03T08:01:21.410906+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:43:21.042754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417575+0000","last_change":"2023-03-03T08:01:21.414742+0000","last_active":"2023-03-03T08:01:21.417575+0000","last_peered":"2023-03-03T08:01:21.417575+0000","last_clean":"2023-03-03T08:01:21.417575+0000","last_became_active":"2023-03-03T08:01:21.414475+0000","last_became_peered":"2023-03-03T08:01:21.414475+0000","last_unstale":"2023-03-03T08:01:21.417575+0000","last_undegraded":"2023-03-03T08:01:21.417575+0000","last_fullsized":"2023-03-03T08:01:21.417575+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:48:45.104859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.401138+0000","last_change":"2023-03-03T08:01:21.396417+0000","last_active":"2023-03-03T08:01:21.401138+0000","last_peered":"2023-03-03T08:01:21.401138+0000","last_clean":"2023-03-03T08:01:21.401138+0000","last_became_active":"2023-03-03T08:01:21.396067+0000","last_became_peered":"2023-03-03T08:01:21.396067+0000","last_unstale":"2023-03-03T08:01:21.401138+0000","last_undegraded":"2023-03-03T08:01:21.401138+0000","last_fullsized":"2023-03-03T08:01:21.401138+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:29:14.686210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.404001+0000","last_change":"2023-03-03T08:01:21.397779+0000","last_active":"2023-03-03T08:01:21.404001+0000","last_peered":"2023-03-03T08:01:21.404001+0000","last_clean":"2023-03-03T08:01:21.404001+0000","last_became_active":"2023-03-03T08:01:21.397486+0000","last_became_peered":"2023-03-03T08:01:21.397486+0000","last_unstale":"2023-03-03T08:01:21.404001+0000","last_undegraded":"2023-03-03T08:01:21.404001+0000","last_fullsized":"2023-03-03T08:01:21.404001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:23:21.871284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417373+0000","last_change":"2023-03-03T08:01:21.415221+0000","last_active":"2023-03-03T08:01:21.417373+0000","last_peered":"2023-03-03T08:01:21.417373+0000","last_clean":"2023-03-03T08:01:21.417373+0000","last_became_active":"2023-03-03T08:01:21.414862+0000","last_became_peered":"2023-03-03T08:01:21.414862+0000","last_unstale":"2023-03-03T08:01:21.417373+0000","last_undegraded":"2023-03-03T08:01:21.417373+0000","last_fullsized":"2023-03-03T08:01:21.417373+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:52:52.458592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.420223+0000","last_change":"2023-03-03T08:01:21.418852+0000","last_active":"2023-03-03T08:01:21.420223+0000","last_peered":"2023-03-03T08:01:21.420223+0000","last_clean":"2023-03-03T08:01:21.420223+0000","last_became_active":"2023-03-03T08:01:21.418555+0000","last_became_peered":"2023-03-03T08:01:21.418555+0000","last_unstale":"2023-03-03T08:01:21.420223+0000","last_undegraded":"2023-03-03T08:01:21.420223+0000","last_fullsized":"2023-03-03T08:01:21.420223+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:10:22.496044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.408255+0000","last_change":"2023-03-03T08:01:21.400801+0000","last_active":"2023-03-03T08:01:21.408255+0000","last_peered":"2023-03-03T08:01:21.408255+0000","last_clean":"2023-03-03T08:01:21.408255+0000","last_became_active":"2023-03-03T08:01:21.400549+0000","last_became_peered":"2023-03-03T08:01:21.400549+0000","last_unstale":"2023-03-03T08:01:21.408255+0000","last_undegraded":"2023-03-03T08:01:21.408255+0000","last_fullsized":"2023-03-03T08:01:21.408255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:58:06.304755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417365+0000","last_change":"2023-03-03T08:01:21.412605+0000","last_active":"2023-03-03T08:01:21.417365+0000","last_peered":"2023-03-03T08:01:21.417365+0000","last_clean":"2023-03-03T08:01:21.417365+0000","last_became_active":"2023-03-03T08:01:21.412358+0000","last_became_peered":"2023-03-03T08:01:21.412358+0000","last_unstale":"2023-03-03T08:01:21.417365+0000","last_undegraded":"2023-03-03T08:01:21.417365+0000","last_fullsized":"2023-03-03T08:01:21.417365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:59:33.772626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.424134+0000","last_change":"2023-03-03T08:01:21.418902+0000","last_active":"2023-03-03T08:01:21.424134+0000","last_peered":"2023-03-03T08:01:21.424134+0000","last_clean":"2023-03-03T08:01:21.424134+0000","last_became_active":"2023-03-03T08:01:21.418610+0000","last_became_peered":"2023-03-03T08:01:21.418610+0000","last_unstale":"2023-03-03T08:01:21.424134+0000","last_undegraded":"2023-03-03T08:01:21.424134+0000","last_fullsized":"2023-03-03T08:01:21.424134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:26:14.276515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.414417+0000","last_change":"2023-03-03T08:01:21.411987+0000","last_active":"2023-03-03T08:01:21.414417+0000","last_peered":"2023-03-03T08:01:21.414417+0000","last_clean":"2023-03-03T08:01:21.414417+0000","last_became_active":"2023-03-03T08:01:21.411634+0000","last_became_peered":"2023-03-03T08:01:21.411634+0000","last_unstale":"2023-03-03T08:01:21.414417+0000","last_undegraded":"2023-03-03T08:01:21.414417+0000","last_fullsized":"2023-03-03T08:01:21.414417+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:59:46.957695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421685+0000","last_change":"2023-03-03T08:01:21.415312+0000","last_active":"2023-03-03T08:01:21.421685+0000","last_peered":"2023-03-03T08:01:21.421685+0000","last_clean":"2023-03-03T08:01:21.421685+0000","last_became_active":"2023-03-03T08:01:21.412152+0000","last_became_peered":"2023-03-03T08:01:21.412152+0000","last_unstale":"2023-03-03T08:01:21.421685+0000","last_undegraded":"2023-03-03T08:01:21.421685+0000","last_fullsized":"2023-03-03T08:01:21.421685+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:05:28.891281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.402627+0000","last_change":"2023-03-03T08:01:21.395987+0000","last_active":"2023-03-03T08:01:21.402627+0000","last_peered":"2023-03-03T08:01:21.402627+0000","last_clean":"2023-03-03T08:01:21.402627+0000","last_became_active":"2023-03-03T08:01:21.395709+0000","last_became_peered":"2023-03-03T08:01:21.395709+0000","last_unstale":"2023-03-03T08:01:21.402627+0000","last_undegraded":"2023-03-03T08:01:21.402627+0000","last_fullsized":"2023-03-03T08:01:21.402627+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:16:23.956197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.400544+0000","last_change":"2023-03-03T08:01:21.395857+0000","last_active":"2023-03-03T08:01:21.400544+0000","last_peered":"2023-03-03T08:01:21.400544+0000","last_clean":"2023-03-03T08:01:21.400544+0000","last_became_active":"2023-03-03T08:01:21.395595+0000","last_became_peered":"2023-03-03T08:01:21.395595+0000","last_unstale":"2023-03-03T08:01:21.400544+0000","last_undegraded":"2023-03-03T08:01:21.400544+0000","last_fullsized":"2023-03-03T08:01:21.400544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:43.087204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.408859+0000","last_change":"2023-03-03T08:01:21.404683+0000","last_active":"2023-03-03T08:01:21.408859+0000","last_peered":"2023-03-03T08:01:21.408859+0000","last_clean":"2023-03-03T08:01:21.408859+0000","last_became_active":"2023-03-03T08:01:21.404412+0000","last_became_peered":"2023-03-03T08:01:21.404412+0000","last_unstale":"2023-03-03T08:01:21.408859+0000","last_undegraded":"2023-03-03T08:01:21.408859+0000","last_fullsized":"2023-03-03T08:01:21.408859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:43:37.184568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.405619+0000","last_change":"2023-03-03T08:01:21.400605+0000","last_active":"2023-03-03T08:01:21.405619+0000","last_peered":"2023-03-03T08:01:21.405619+0000","last_clean":"2023-03-03T08:01:21.405619+0000","last_became_active":"2023-03-03T08:01:21.400346+0000","last_became_peered":"2023-03-03T08:01:21.400346+0000","last_unstale":"2023-03-03T08:01:21.405619+0000","last_undegraded":"2023-03-03T08:01:21.405619+0000","last_fullsized":"2023-03-03T08:01:21.405619+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:11:17.517768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.411532+0000","last_change":"2023-03-03T08:01:21.407540+0000","last_active":"2023-03-03T08:01:21.411532+0000","last_peered":"2023-03-03T08:01:21.411532+0000","last_clean":"2023-03-03T08:01:21.411532+0000","last_became_active":"2023-03-03T08:01:21.407302+0000","last_became_peered":"2023-03-03T08:01:21.407302+0000","last_unstale":"2023-03-03T08:01:21.411532+0000","last_undegraded":"2023-03-03T08:01:21.411532+0000","last_fullsized":"2023-03-03T08:01:21.411532+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:20:46.479131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.415500+0000","last_change":"2023-03-03T08:01:21.410064+0000","last_active":"2023-03-03T08:01:21.415500+0000","last_peered":"2023-03-03T08:01:21.415500+0000","last_clean":"2023-03-03T08:01:21.415500+0000","last_became_active":"2023-03-03T08:01:21.409788+0000","last_became_peered":"2023-03-03T08:01:21.409788+0000","last_unstale":"2023-03-03T08:01:21.415500+0000","last_undegraded":"2023-03-03T08:01:21.415500+0000","last_fullsized":"2023-03-03T08:01:21.415500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:20:30.614418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.408343+0000","last_change":"2023-03-03T08:01:21.405782+0000","last_active":"2023-03-03T08:01:21.408343+0000","last_peered":"2023-03-03T08:01:21.408343+0000","last_clean":"2023-03-03T08:01:21.408343+0000","last_became_active":"2023-03-03T08:01:21.405330+0000","last_became_peered":"2023-03-03T08:01:21.405330+0000","last_unstale":"2023-03-03T08:01:21.408343+0000","last_undegraded":"2023-03-03T08:01:21.408343+0000","last_fullsized":"2023-03-03T08:01:21.408343+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:31.468080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.415082+0000","last_change":"2023-03-03T08:01:21.410532+0000","last_active":"2023-03-03T08:01:21.415082+0000","last_peered":"2023-03-03T08:01:21.415082+0000","last_clean":"2023-03-03T08:01:21.415082+0000","last_became_active":"2023-03-03T08:01:21.410275+0000","last_became_peered":"2023-03-03T08:01:21.410275+0000","last_unstale":"2023-03-03T08:01:21.415082+0000","last_undegraded":"2023-03-03T08:01:21.415082+0000","last_fullsized":"2023-03-03T08:01:21.415082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:30:09.829742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.416517+0000","last_change":"2023-03-03T08:01:21.412820+0000","last_active":"2023-03-03T08:01:21.416517+0000","last_peered":"2023-03-03T08:01:21.416517+0000","last_clean":"2023-03-03T08:01:21.416517+0000","last_became_active":"2023-03-03T08:01:21.412562+0000","last_became_peered":"2023-03-03T08:01:21.412562+0000","last_unstale":"2023-03-03T08:01:21.416517+0000","last_undegraded":"2023-03-03T08:01:21.416517+0000","last_fullsized":"2023-03-03T08:01:21.416517+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:25.264212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417044+0000","last_change":"2023-03-03T08:01:21.415783+0000","last_active":"2023-03-03T08:01:21.417044+0000","last_peered":"2023-03-03T08:01:21.417044+0000","last_clean":"2023-03-03T08:01:21.417044+0000","last_became_active":"2023-03-03T08:01:21.415460+0000","last_became_peered":"2023-03-03T08:01:21.415460+0000","last_unstale":"2023-03-03T08:01:21.417044+0000","last_undegraded":"2023-03-03T08:01:21.417044+0000","last_fullsized":"2023-03-03T08:01:21.417044+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:22.792169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"77'4","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418387+0000","last_change":"2023-03-03T08:01:20.846783+0000","last_active":"2023-03-03T08:01:21.418387+0000","last_peered":"2023-03-03T08:01:21.418387+0000","last_clean":"2023-03-03T08:01:21.418387+0000","last_became_active":"2023-03-03T08:01:20.846528+0000","last_became_peered":"2023-03-03T08:01:20.846528+0000","last_unstale":"2023-03-03T08:01:21.418387+0000","last_undegraded":"2023-03-03T08:01:21.418387+0000","last_fullsized":"2023-03-03T08:01:21.418387+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:36:30.843302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.408321+0000","last_change":"2023-03-03T08:01:21.401300+0000","last_active":"2023-03-03T08:01:21.408321+0000","last_peered":"2023-03-03T08:01:21.408321+0000","last_clean":"2023-03-03T08:01:21.408321+0000","last_became_active":"2023-03-03T08:01:21.400951+0000","last_became_peered":"2023-03-03T08:01:21.400951+0000","last_unstale":"2023-03-03T08:01:21.408321+0000","last_undegraded":"2023-03-03T08:01:21.408321+0000","last_fullsized":"2023-03-03T08:01:21.408321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:07:47.722052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"77'1","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.416193+0000","last_change":"2023-03-03T08:01:20.846342+0000","last_active":"2023-03-03T08:01:21.416193+0000","last_peered":"2023-03-03T08:01:21.416193+0000","last_clean":"2023-03-03T08:01:21.416193+0000","last_became_active":"2023-03-03T08:01:20.845994+0000","last_became_peered":"2023-03-03T08:01:20.845994+0000","last_unstale":"2023-03-03T08:01:21.416193+0000","last_undegraded":"2023-03-03T08:01:21.416193+0000","last_fullsized":"2023-03-03T08:01:21.416193+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:22:55.388447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421626+0000","last_change":"2023-03-03T08:01:21.413013+0000","last_active":"2023-03-03T08:01:21.421626+0000","last_peered":"2023-03-03T08:01:21.421626+0000","last_clean":"2023-03-03T08:01:21.421626+0000","last_became_active":"2023-03-03T08:01:21.412339+0000","last_became_peered":"2023-03-03T08:01:21.412339+0000","last_unstale":"2023-03-03T08:01:21.421626+0000","last_undegraded":"2023-03-03T08:01:21.421626+0000","last_fullsized":"2023-03-03T08:01:21.421626+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:15:49.926634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"77'2","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.411438+0000","last_change":"2023-03-03T08:01:20.849422+0000","last_active":"2023-03-03T08:01:21.411438+0000","last_peered":"2023-03-03T08:01:21.411438+0000","last_clean":"2023-03-03T08:01:21.411438+0000","last_became_active":"2023-03-03T08:01:20.849065+0000","last_became_peered":"2023-03-03T08:01:20.849065+0000","last_unstale":"2023-03-03T08:01:21.411438+0000","last_undegraded":"2023-03-03T08:01:21.411438+0000","last_fullsized":"2023-03-03T08:01:21.411438+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:17.475047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.424097+0000","last_change":"2023-03-03T08:01:21.418555+0000","last_active":"2023-03-03T08:01:21.424097+0000","last_peered":"2023-03-03T08:01:21.424097+0000","last_clean":"2023-03-03T08:01:21.424097+0000","last_became_active":"2023-03-03T08:01:21.418286+0000","last_became_peered":"2023-03-03T08:01:21.418286+0000","last_unstale":"2023-03-03T08:01:21.424097+0000","last_undegraded":"2023-03-03T08:01:21.424097+0000","last_fullsized":"2023-03-03T08:01:21.424097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:31.468080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"77'2","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418704+0000","last_change":"2023-03-03T08:01:20.841488+0000","last_active":"2023-03-03T08:01:21.418704+0000","last_peered":"2023-03-03T08:01:21.418704+0000","last_clean":"2023-03-03T08:01:21.418704+0000","last_became_active":"2023-03-03T08:01:20.841131+0000","last_became_peered":"2023-03-03T08:01:20.841131+0000","last_unstale":"2023-03-03T08:01:21.418704+0000","last_undegraded":"2023-03-03T08:01:21.418704+0000","last_fullsized":"2023-03-03T08:01:21.418704+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:27:31.570735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407343+0000","last_change":"2023-03-03T08:01:21.399954+0000","last_active":"2023-03-03T08:01:21.407343+0000","last_peered":"2023-03-03T08:01:21.407343+0000","last_clean":"2023-03-03T08:01:21.407343+0000","last_became_active":"2023-03-03T08:01:21.399094+0000","last_became_peered":"2023-03-03T08:01:21.399094+0000","last_unstale":"2023-03-03T08:01:21.407343+0000","last_undegraded":"2023-03-03T08:01:21.407343+0000","last_fullsized":"2023-03-03T08:01:21.407343+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:32:17.007774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"77'1","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418296+0000","last_change":"2023-03-03T08:01:20.845956+0000","last_active":"2023-03-03T08:01:21.418296+0000","last_peered":"2023-03-03T08:01:21.418296+0000","last_clean":"2023-03-03T08:01:21.418296+0000","last_became_active":"2023-03-03T08:01:20.845648+0000","last_became_peered":"2023-03-03T08:01:20.845648+0000","last_unstale":"2023-03-03T08:01:21.418296+0000","last_undegraded":"2023-03-03T08:01:21.418296+0000","last_fullsized":"2023-03-03T08:01:21.418296+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:36:37.038190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407390+0000","last_change":"2023-03-03T08:01:21.400571+0000","last_active":"2023-03-03T08:01:21.407390+0000","last_peered":"2023-03-03T08:01:21.407390+0000","last_clean":"2023-03-03T08:01:21.407390+0000","last_became_active":"2023-03-03T08:01:21.399841+0000","last_became_peered":"2023-03-03T08:01:21.399841+0000","last_unstale":"2023-03-03T08:01:21.407390+0000","last_undegraded":"2023-03-03T08:01:21.407390+0000","last_fullsized":"2023-03-03T08:01:21.407390+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:26:21.966048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"77'1","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.411711+0000","last_change":"2023-03-03T08:01:20.832412+0000","last_active":"2023-03-03T08:01:21.411711+0000","last_peered":"2023-03-03T08:01:21.411711+0000","last_clean":"2023-03-03T08:01:21.411711+0000","last_became_active":"2023-03-03T08:01:20.832103+0000","last_became_peered":"2023-03-03T08:01:20.832103+0000","last_unstale":"2023-03-03T08:01:21.411711+0000","last_undegraded":"2023-03-03T08:01:21.411711+0000","last_fullsized":"2023-03-03T08:01:21.411711+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:10:21.482456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.424064+0000","last_change":"2023-03-03T08:01:21.417997+0000","last_active":"2023-03-03T08:01:21.424064+0000","last_peered":"2023-03-03T08:01:21.424064+0000","last_clean":"2023-03-03T08:01:21.424064+0000","last_became_active":"2023-03-03T08:01:21.417554+0000","last_became_peered":"2023-03-03T08:01:21.417554+0000","last_unstale":"2023-03-03T08:01:21.424064+0000","last_undegraded":"2023-03-03T08:01:21.424064+0000","last_fullsized":"2023-03-03T08:01:21.424064+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:10:41.011913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.389525+0000","last_change":"2023-03-03T08:01:20.842935+0000","last_active":"2023-03-03T08:01:21.389525+0000","last_peered":"2023-03-03T08:01:21.389525+0000","last_clean":"2023-03-03T08:01:21.389525+0000","last_became_active":"2023-03-03T08:01:20.842585+0000","last_became_peered":"2023-03-03T08:01:20.842585+0000","last_unstale":"2023-03-03T08:01:21.389525+0000","last_undegraded":"2023-03-03T08:01:21.389525+0000","last_fullsized":"2023-03-03T08:01:21.389525+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:05:27.877693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418662+0000","last_change":"2023-03-03T08:01:21.413744+0000","last_active":"2023-03-03T08:01:21.418662+0000","last_peered":"2023-03-03T08:01:21.418662+0000","last_clean":"2023-03-03T08:01:21.418662+0000","last_became_active":"2023-03-03T08:01:21.412814+0000","last_became_peered":"2023-03-03T08:01:21.412814+0000","last_unstale":"2023-03-03T08:01:21.418662+0000","last_undegraded":"2023-03-03T08:01:21.418662+0000","last_fullsized":"2023-03-03T08:01:21.418662+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:06.319757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.392653+0000","last_change":"2023-03-03T08:01:20.848411+0000","last_active":"2023-03-03T08:01:21.392653+0000","last_peered":"2023-03-03T08:01:21.392653+0000","last_clean":"2023-03-03T08:01:21.392653+0000","last_became_active":"2023-03-03T08:01:20.848174+0000","last_became_peered":"2023-03-03T08:01:20.848174+0000","last_unstale":"2023-03-03T08:01:21.392653+0000","last_undegraded":"2023-03-03T08:01:21.392653+0000","last_fullsized":"2023-03-03T08:01:21.392653+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:59:31.880965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407340+0000","last_change":"2023-03-03T08:01:21.402743+0000","last_active":"2023-03-03T08:01:21.407340+0000","last_peered":"2023-03-03T08:01:21.407340+0000","last_clean":"2023-03-03T08:01:21.407340+0000","last_became_active":"2023-03-03T08:01:21.402462+0000","last_became_peered":"2023-03-03T08:01:21.402462+0000","last_unstale":"2023-03-03T08:01:21.407340+0000","last_undegraded":"2023-03-03T08:01:21.407340+0000","last_fullsized":"2023-03-03T08:01:21.407340+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:53:57.306250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.392931+0000","last_change":"2023-03-03T08:01:20.847275+0000","last_active":"2023-03-03T08:01:21.392931+0000","last_peered":"2023-03-03T08:01:21.392931+0000","last_clean":"2023-03-03T08:01:21.392931+0000","last_became_active":"2023-03-03T08:01:20.846644+0000","last_became_peered":"2023-03-03T08:01:20.846644+0000","last_unstale":"2023-03-03T08:01:21.392931+0000","last_undegraded":"2023-03-03T08:01:21.392931+0000","last_fullsized":"2023-03-03T08:01:21.392931+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:33:56.386284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421094+0000","last_change":"2023-03-03T08:01:21.415505+0000","last_active":"2023-03-03T08:01:21.421094+0000","last_peered":"2023-03-03T08:01:21.421094+0000","last_clean":"2023-03-03T08:01:21.421094+0000","last_became_active":"2023-03-03T08:01:21.415194+0000","last_became_peered":"2023-03-03T08:01:21.415194+0000","last_unstale":"2023-03-03T08:01:21.421094+0000","last_undegraded":"2023-03-03T08:01:21.421094+0000","last_fullsized":"2023-03-03T08:01:21.421094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:11.768864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.389902+0000","last_change":"2023-03-03T08:01:20.843547+0000","last_active":"2023-03-03T08:01:21.389902+0000","last_peered":"2023-03-03T08:01:21.389902+0000","last_clean":"2023-03-03T08:01:21.389902+0000","last_became_active":"2023-03-03T08:01:20.843206+0000","last_became_peered":"2023-03-03T08:01:20.843206+0000","last_unstale":"2023-03-03T08:01:21.389902+0000","last_undegraded":"2023-03-03T08:01:21.389902+0000","last_fullsized":"2023-03-03T08:01:21.389902+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:30:08.816154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.400189+0000","last_change":"2023-03-03T08:01:21.395725+0000","last_active":"2023-03-03T08:01:21.400189+0000","last_peered":"2023-03-03T08:01:21.400189+0000","last_clean":"2023-03-03T08:01:21.400189+0000","last_became_active":"2023-03-03T08:01:21.395462+0000","last_became_peered":"2023-03-03T08:01:21.395462+0000","last_unstale":"2023-03-03T08:01:21.400189+0000","last_undegraded":"2023-03-03T08:01:21.400189+0000","last_fullsized":"2023-03-03T08:01:21.400189+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:11:17.517768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"77'3","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417035+0000","last_change":"2023-03-03T08:01:20.849625+0000","last_active":"2023-03-03T08:01:21.417035+0000","last_peered":"2023-03-03T08:01:21.417035+0000","last_clean":"2023-03-03T08:01:21.417035+0000","last_became_active":"2023-03-03T08:01:20.849325+0000","last_became_peered":"2023-03-03T08:01:20.849325+0000","last_unstale":"2023-03-03T08:01:21.417035+0000","last_undegraded":"2023-03-03T08:01:21.417035+0000","last_fullsized":"2023-03-03T08:01:21.417035+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:09:04.999100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.412342+0000","last_change":"2023-03-03T08:01:21.404444+0000","last_active":"2023-03-03T08:01:21.412342+0000","last_peered":"2023-03-03T08:01:21.412342+0000","last_clean":"2023-03-03T08:01:21.412342+0000","last_became_active":"2023-03-03T08:01:21.404029+0000","last_became_peered":"2023-03-03T08:01:21.404029+0000","last_unstale":"2023-03-03T08:01:21.412342+0000","last_undegraded":"2023-03-03T08:01:21.412342+0000","last_fullsized":"2023-03-03T08:01:21.412342+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:33:06.319757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"77'6","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.424463+0000","last_change":"2023-03-03T08:01:20.844608+0000","last_active":"2023-03-03T08:01:21.424463+0000","last_peered":"2023-03-03T08:01:21.424463+0000","last_clean":"2023-03-03T08:01:21.424463+0000","last_became_active":"2023-03-03T08:01:20.844308+0000","last_became_peered":"2023-03-03T08:01:20.844308+0000","last_unstale":"2023-03-03T08:01:21.424463+0000","last_undegraded":"2023-03-03T08:01:21.424463+0000","last_fullsized":"2023-03-03T08:01:21.424463+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:13:43.554478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.405619+0000","last_change":"2023-03-03T08:01:21.397824+0000","last_active":"2023-03-03T08:01:21.405619+0000","last_peered":"2023-03-03T08:01:21.405619+0000","last_clean":"2023-03-03T08:01:21.405619+0000","last_became_active":"2023-03-03T08:01:21.397501+0000","last_became_peered":"2023-03-03T08:01:21.397501+0000","last_unstale":"2023-03-03T08:01:21.405619+0000","last_undegraded":"2023-03-03T08:01:21.405619+0000","last_fullsized":"2023-03-03T08:01:21.405619+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:15:49.926634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"77'2","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418890+0000","last_change":"2023-03-03T08:01:20.832929+0000","last_active":"2023-03-03T08:01:21.418890+0000","last_peered":"2023-03-03T08:01:21.418890+0000","last_clean":"2023-03-03T08:01:21.418890+0000","last_became_active":"2023-03-03T08:01:20.832582+0000","last_became_peered":"2023-03-03T08:01:20.832582+0000","last_unstale":"2023-03-03T08:01:21.418890+0000","last_undegraded":"2023-03-03T08:01:21.418890+0000","last_fullsized":"2023-03-03T08:01:21.418890+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:13:19.662666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421566+0000","last_change":"2023-03-03T08:01:21.418465+0000","last_active":"2023-03-03T08:01:21.421566+0000","last_peered":"2023-03-03T08:01:21.421566+0000","last_clean":"2023-03-03T08:01:21.421566+0000","last_became_active":"2023-03-03T08:01:21.418161+0000","last_became_peered":"2023-03-03T08:01:21.418161+0000","last_unstale":"2023-03-03T08:01:21.421566+0000","last_undegraded":"2023-03-03T08:01:21.421566+0000","last_fullsized":"2023-03-03T08:01:21.421566+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:39:44.111270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.392157+0000","last_change":"2023-03-03T08:01:20.847931+0000","last_active":"2023-03-03T08:01:21.392157+0000","last_peered":"2023-03-03T08:01:21.392157+0000","last_clean":"2023-03-03T08:01:21.392157+0000","last_became_active":"2023-03-03T08:01:20.847650+0000","last_became_peered":"2023-03-03T08:01:20.847650+0000","last_unstale":"2023-03-03T08:01:21.392157+0000","last_undegraded":"2023-03-03T08:01:21.392157+0000","last_fullsized":"2023-03-03T08:01:21.392157+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:10.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421602+0000","last_change":"2023-03-03T08:01:21.419818+0000","last_active":"2023-03-03T08:01:21.421602+0000","last_peered":"2023-03-03T08:01:21.421602+0000","last_clean":"2023-03-03T08:01:21.421602+0000","last_became_active":"2023-03-03T08:01:21.419562+0000","last_became_peered":"2023-03-03T08:01:21.419562+0000","last_unstale":"2023-03-03T08:01:21.421602+0000","last_undegraded":"2023-03-03T08:01:21.421602+0000","last_fullsized":"2023-03-03T08:01:21.421602+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:13:40.650825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.392832+0000","last_change":"2023-03-03T08:01:20.849416+0000","last_active":"2023-03-03T08:01:21.392832+0000","last_peered":"2023-03-03T08:01:21.392832+0000","last_clean":"2023-03-03T08:01:21.392832+0000","last_became_active":"2023-03-03T08:01:20.849109+0000","last_became_peered":"2023-03-03T08:01:20.849109+0000","last_unstale":"2023-03-03T08:01:21.392832+0000","last_undegraded":"2023-03-03T08:01:21.392832+0000","last_fullsized":"2023-03-03T08:01:21.392832+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:05:04.070505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418680+0000","last_change":"2023-03-03T08:01:21.413916+0000","last_active":"2023-03-03T08:01:21.418680+0000","last_peered":"2023-03-03T08:01:21.418680+0000","last_clean":"2023-03-03T08:01:21.418680+0000","last_became_active":"2023-03-03T08:01:21.413151+0000","last_became_peered":"2023-03-03T08:01:21.413151+0000","last_unstale":"2023-03-03T08:01:21.418680+0000","last_undegraded":"2023-03-03T08:01:21.418680+0000","last_fullsized":"2023-03-03T08:01:21.418680+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:13:20.676253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"77'8","reported_seq":18,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.416354+0000","last_change":"2023-03-03T08:01:20.842341+0000","last_active":"2023-03-03T08:01:21.416354+0000","last_peered":"2023-03-03T08:01:21.416354+0000","last_clean":"2023-03-03T08:01:21.416354+0000","last_became_active":"2023-03-03T08:01:20.842059+0000","last_became_peered":"2023-03-03T08:01:20.842059+0000","last_unstale":"2023-03-03T08:01:21.416354+0000","last_undegraded":"2023-03-03T08:01:21.416354+0000","last_fullsized":"2023-03-03T08:01:21.416354+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:17.475047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421714+0000","last_change":"2023-03-03T08:01:21.415591+0000","last_active":"2023-03-03T08:01:21.421714+0000","last_peered":"2023-03-03T08:01:21.421714+0000","last_clean":"2023-03-03T08:01:21.421714+0000","last_became_active":"2023-03-03T08:01:21.415298+0000","last_became_peered":"2023-03-03T08:01:21.415298+0000","last_unstale":"2023-03-03T08:01:21.421714+0000","last_undegraded":"2023-03-03T08:01:21.421714+0000","last_fullsized":"2023-03-03T08:01:21.421714+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:37.071959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"77'2","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.414910+0000","last_change":"2023-03-03T08:01:20.837524+0000","last_active":"2023-03-03T08:01:21.414910+0000","last_peered":"2023-03-03T08:01:21.414910+0000","last_clean":"2023-03-03T08:01:21.414910+0000","last_became_active":"2023-03-03T08:01:20.837178+0000","last_became_peered":"2023-03-03T08:01:20.837178+0000","last_unstale":"2023-03-03T08:01:21.414910+0000","last_undegraded":"2023-03-03T08:01:21.414910+0000","last_fullsized":"2023-03-03T08:01:21.414910+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:21.778582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417095+0000","last_change":"2023-03-03T08:01:21.410084+0000","last_active":"2023-03-03T08:01:21.417095+0000","last_peered":"2023-03-03T08:01:21.417095+0000","last_clean":"2023-03-03T08:01:21.417095+0000","last_became_active":"2023-03-03T08:01:21.409589+0000","last_became_peered":"2023-03-03T08:01:21.409589+0000","last_unstale":"2023-03-03T08:01:21.417095+0000","last_undegraded":"2023-03-03T08:01:21.417095+0000","last_fullsized":"2023-03-03T08:01:21.417095+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:05:11.213383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.390693+0000","last_change":"2023-03-03T08:01:20.845764+0000","last_active":"2023-03-03T08:01:21.390693+0000","last_peered":"2023-03-03T08:01:21.390693+0000","last_clean":"2023-03-03T08:01:21.390693+0000","last_became_active":"2023-03-03T08:01:20.845424+0000","last_became_peered":"2023-03-03T08:01:20.845424+0000","last_unstale":"2023-03-03T08:01:21.390693+0000","last_undegraded":"2023-03-03T08:01:21.390693+0000","last_fullsized":"2023-03-03T08:01:21.390693+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:15:48.913046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.421603+0000","last_change":"2023-03-03T08:01:21.411993+0000","last_active":"2023-03-03T08:01:21.421603+0000","last_peered":"2023-03-03T08:01:21.421603+0000","last_clean":"2023-03-03T08:01:21.421603+0000","last_became_active":"2023-03-03T08:01:21.411708+0000","last_became_peered":"2023-03-03T08:01:21.411708+0000","last_unstale":"2023-03-03T08:01:21.421603+0000","last_undegraded":"2023-03-03T08:01:21.421603+0000","last_fullsized":"2023-03-03T08:01:21.421603+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:04:07.988256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.393368+0000","last_change":"2023-03-03T08:01:20.847450+0000","last_active":"2023-03-03T08:01:21.393368+0000","last_peered":"2023-03-03T08:01:21.393368+0000","last_clean":"2023-03-03T08:01:21.393368+0000","last_became_active":"2023-03-03T08:01:20.847172+0000","last_became_peered":"2023-03-03T08:01:20.847172+0000","last_unstale":"2023-03-03T08:01:21.393368+0000","last_undegraded":"2023-03-03T08:01:21.393368+0000","last_fullsized":"2023-03-03T08:01:21.393368+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:45:53.717180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.402568+0000","last_change":"2023-03-03T08:01:21.397210+0000","last_active":"2023-03-03T08:01:21.402568+0000","last_peered":"2023-03-03T08:01:21.402568+0000","last_clean":"2023-03-03T08:01:21.402568+0000","last_became_active":"2023-03-03T08:01:21.396965+0000","last_became_peered":"2023-03-03T08:01:21.396965+0000","last_unstale":"2023-03-03T08:01:21.402568+0000","last_undegraded":"2023-03-03T08:01:21.402568+0000","last_fullsized":"2023-03-03T08:01:21.402568+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:36:31.007402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.390638+0000","last_change":"2023-03-03T08:01:20.846744+0000","last_active":"2023-03-03T08:01:21.390638+0000","last_peered":"2023-03-03T08:01:21.390638+0000","last_clean":"2023-03-03T08:01:21.390638+0000","last_became_active":"2023-03-03T08:01:20.846186+0000","last_became_peered":"2023-03-03T08:01:20.846186+0000","last_unstale":"2023-03-03T08:01:21.390638+0000","last_undegraded":"2023-03-03T08:01:21.390638+0000","last_fullsized":"2023-03-03T08:01:21.390638+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:36:37.038190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.419258+0000","last_change":"2023-03-03T08:01:21.413103+0000","last_active":"2023-03-03T08:01:21.419258+0000","last_peered":"2023-03-03T08:01:21.419258+0000","last_clean":"2023-03-03T08:01:21.419258+0000","last_became_active":"2023-03-03T08:01:21.412609+0000","last_became_peered":"2023-03-03T08:01:21.412609+0000","last_unstale":"2023-03-03T08:01:21.419258+0000","last_undegraded":"2023-03-03T08:01:21.419258+0000","last_fullsized":"2023-03-03T08:01:21.419258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:24.051431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.396587+0000","last_change":"2023-03-03T08:01:20.844629+0000","last_active":"2023-03-03T08:01:21.396587+0000","last_peered":"2023-03-03T08:01:21.396587+0000","last_clean":"2023-03-03T08:01:21.396587+0000","last_became_active":"2023-03-03T08:01:20.844354+0000","last_became_peered":"2023-03-03T08:01:20.844354+0000","last_unstale":"2023-03-03T08:01:21.396587+0000","last_undegraded":"2023-03-03T08:01:21.396587+0000","last_fullsized":"2023-03-03T08:01:21.396587+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:05:58.352008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418573+0000","last_change":"2023-03-03T08:01:21.413165+0000","last_active":"2023-03-03T08:01:21.418573+0000","last_peered":"2023-03-03T08:01:21.418573+0000","last_clean":"2023-03-03T08:01:21.418573+0000","last_became_active":"2023-03-03T08:01:21.412837+0000","last_became_peered":"2023-03-03T08:01:21.412837+0000","last_unstale":"2023-03-03T08:01:21.418573+0000","last_undegraded":"2023-03-03T08:01:21.418573+0000","last_fullsized":"2023-03-03T08:01:21.418573+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:11:56.348547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.396702+0000","last_change":"2023-03-03T08:01:20.849541+0000","last_active":"2023-03-03T08:01:21.396702+0000","last_peered":"2023-03-03T08:01:21.396702+0000","last_clean":"2023-03-03T08:01:21.396702+0000","last_became_active":"2023-03-03T08:01:20.849292+0000","last_became_peered":"2023-03-03T08:01:20.849292+0000","last_unstale":"2023-03-03T08:01:21.396702+0000","last_undegraded":"2023-03-03T08:01:21.396702+0000","last_fullsized":"2023-03-03T08:01:21.396702+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:01:20.228416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.408090+0000","last_change":"2023-03-03T08:01:21.402044+0000","last_active":"2023-03-03T08:01:21.408090+0000","last_peered":"2023-03-03T08:01:21.408090+0000","last_clean":"2023-03-03T08:01:21.408090+0000","last_became_active":"2023-03-03T08:01:21.401730+0000","last_became_peered":"2023-03-03T08:01:21.401730+0000","last_unstale":"2023-03-03T08:01:21.408090+0000","last_undegraded":"2023-03-03T08:01:21.408090+0000","last_fullsized":"2023-03-03T08:01:21.408090+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:11:56.348547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.406440+0000","last_change":"2023-03-03T08:01:20.843316+0000","last_active":"2023-03-03T08:01:21.406440+0000","last_peered":"2023-03-03T08:01:21.406440+0000","last_clean":"2023-03-03T08:01:21.406440+0000","last_became_active":"2023-03-03T08:01:20.843051+0000","last_became_peered":"2023-03-03T08:01:20.843051+0000","last_unstale":"2023-03-03T08:01:21.406440+0000","last_undegraded":"2023-03-03T08:01:21.406440+0000","last_fullsized":"2023-03-03T08:01:21.406440+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:34.165624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.412961+0000","last_change":"2023-03-03T08:01:21.403427+0000","last_active":"2023-03-03T08:01:21.412961+0000","last_peered":"2023-03-03T08:01:21.412961+0000","last_clean":"2023-03-03T08:01:21.412961+0000","last_became_active":"2023-03-03T08:01:21.403183+0000","last_became_peered":"2023-03-03T08:01:21.403183+0000","last_unstale":"2023-03-03T08:01:21.412961+0000","last_undegraded":"2023-03-03T08:01:21.412961+0000","last_fullsized":"2023-03-03T08:01:21.412961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:36:38.051778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.384905+0000","last_change":"2023-03-03T08:01:20.843157+0000","last_active":"2023-03-03T08:01:21.384905+0000","last_peered":"2023-03-03T08:01:21.384905+0000","last_clean":"2023-03-03T08:01:21.384905+0000","last_became_active":"2023-03-03T08:01:20.842876+0000","last_became_peered":"2023-03-03T08:01:20.842876+0000","last_unstale":"2023-03-03T08:01:21.384905+0000","last_undegraded":"2023-03-03T08:01:21.384905+0000","last_fullsized":"2023-03-03T08:01:21.384905+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:20.845420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407810+0000","last_change":"2023-03-03T08:01:21.405676+0000","last_active":"2023-03-03T08:01:21.407810+0000","last_peered":"2023-03-03T08:01:21.407810+0000","last_clean":"2023-03-03T08:01:21.407810+0000","last_became_active":"2023-03-03T08:01:21.405416+0000","last_became_peered":"2023-03-03T08:01:21.405416+0000","last_unstale":"2023-03-03T08:01:21.407810+0000","last_undegraded":"2023-03-03T08:01:21.407810+0000","last_fullsized":"2023-03-03T08:01:21.407810+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:04:51.424269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"77'1","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.424258+0000","last_change":"2023-03-03T08:01:20.847881+0000","last_active":"2023-03-03T08:01:21.424258+0000","last_peered":"2023-03-03T08:01:21.424258+0000","last_clean":"2023-03-03T08:01:21.424258+0000","last_became_active":"2023-03-03T08:01:20.847612+0000","last_became_peered":"2023-03-03T08:01:20.847612+0000","last_unstale":"2023-03-03T08:01:21.424258+0000","last_undegraded":"2023-03-03T08:01:21.424258+0000","last_fullsized":"2023-03-03T08:01:21.424258+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:22.628846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.828322+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T08:01:20.828322+0000","last_peered":"2023-03-03T08:01:20.828322+0000","last_clean":"2023-03-03T08:01:20.828322+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T08:01:20.828322+0000","last_undegraded":"2023-03-03T08:01:20.828322+0000","last_fullsized":"2023-03-03T08:01:20.828322+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.407362+0000","last_change":"2023-03-03T08:01:21.401771+0000","last_active":"2023-03-03T08:01:21.407362+0000","last_peered":"2023-03-03T08:01:21.407362+0000","last_clean":"2023-03-03T08:01:21.407362+0000","last_became_active":"2023-03-03T08:01:21.401462+0000","last_became_peered":"2023-03-03T08:01:21.401462+0000","last_unstale":"2023-03-03T08:01:21.407362+0000","last_undegraded":"2023-03-03T08:01:21.407362+0000","last_fullsized":"2023-03-03T08:01:21.407362+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:03:54.417729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"77'2","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.413402+0000","last_change":"2023-03-03T08:01:20.837345+0000","last_active":"2023-03-03T08:01:21.413402+0000","last_peered":"2023-03-03T08:01:21.413402+0000","last_clean":"2023-03-03T08:01:21.413402+0000","last_became_active":"2023-03-03T08:01:20.837091+0000","last_became_peered":"2023-03-03T08:01:20.837091+0000","last_unstale":"2023-03-03T08:01:21.413402+0000","last_undegraded":"2023-03-03T08:01:21.413402+0000","last_fullsized":"2023-03-03T08:01:21.413402+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:04:06.974669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.827538+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T08:01:20.827538+0000","last_peered":"2023-03-03T08:01:20.827538+0000","last_clean":"2023-03-03T08:01:20.827538+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T08:01:20.827538+0000","last_undegraded":"2023-03-03T08:01:20.827538+0000","last_fullsized":"2023-03-03T08:01:20.827538+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.418696+0000","last_change":"2023-03-03T08:01:21.414074+0000","last_active":"2023-03-03T08:01:21.418696+0000","last_peered":"2023-03-03T08:01:21.418696+0000","last_clean":"2023-03-03T08:01:21.418696+0000","last_became_active":"2023-03-03T08:01:21.413426+0000","last_became_peered":"2023-03-03T08:01:21.413426+0000","last_unstale":"2023-03-03T08:01:21.418696+0000","last_undegraded":"2023-03-03T08:01:21.418696+0000","last_fullsized":"2023-03-03T08:01:21.418696+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:48:09.597186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.388393+0000","last_change":"2023-03-03T08:01:20.846487+0000","last_active":"2023-03-03T08:01:21.388393+0000","last_peered":"2023-03-03T08:01:21.388393+0000","last_clean":"2023-03-03T08:01:21.388393+0000","last_became_active":"2023-03-03T08:01:20.846189+0000","last_became_peered":"2023-03-03T08:01:20.846189+0000","last_unstale":"2023-03-03T08:01:21.388393+0000","last_undegraded":"2023-03-03T08:01:21.388393+0000","last_fullsized":"2023-03-03T08:01:21.388393+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:43:36.170981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.841001+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T08:01:20.841001+0000","last_peered":"2023-03-03T08:01:20.841001+0000","last_clean":"2023-03-03T08:01:20.841001+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T08:01:20.841001+0000","last_undegraded":"2023-03-03T08:01:20.841001+0000","last_fullsized":"2023-03-03T08:01:20.841001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.416553+0000","last_change":"2023-03-03T08:01:21.409108+0000","last_active":"2023-03-03T08:01:21.416553+0000","last_peered":"2023-03-03T08:01:21.416553+0000","last_clean":"2023-03-03T08:01:21.416553+0000","last_became_active":"2023-03-03T08:01:21.408790+0000","last_became_peered":"2023-03-03T08:01:21.408790+0000","last_unstale":"2023-03-03T08:01:21.416553+0000","last_undegraded":"2023-03-03T08:01:21.416553+0000","last_fullsized":"2023-03-03T08:01:21.416553+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:48:45.104859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.388765+0000","last_change":"2023-03-03T08:01:20.845887+0000","last_active":"2023-03-03T08:01:21.388765+0000","last_peered":"2023-03-03T08:01:21.388765+0000","last_clean":"2023-03-03T08:01:21.388765+0000","last_became_active":"2023-03-03T08:01:20.845611+0000","last_became_peered":"2023-03-03T08:01:20.845611+0000","last_unstale":"2023-03-03T08:01:21.388765+0000","last_undegraded":"2023-03-03T08:01:21.388765+0000","last_fullsized":"2023-03-03T08:01:21.388765+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:05:27.877693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.827980+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T08:01:20.827980+0000","last_peered":"2023-03-03T08:01:20.827980+0000","last_clean":"2023-03-03T08:01:20.827980+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T08:01:20.827980+0000","last_undegraded":"2023-03-03T08:01:20.827980+0000","last_fullsized":"2023-03-03T08:01:20.827980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":213,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.837947+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T08:01:20.837947+0000","last_peered":"2023-03-03T08:01:20.837947+0000","last_clean":"2023-03-03T08:01:20.837947+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T08:01:20.837947+0000","last_undegraded":"2023-03-03T08:01:20.837947+0000","last_fullsized":"2023-03-03T08:01:20.837947+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.419478+0000","last_change":"2023-03-03T08:01:21.418514+0000","last_active":"2023-03-03T08:01:21.419478+0000","last_peered":"2023-03-03T08:01:21.419478+0000","last_clean":"2023-03-03T08:01:21.419478+0000","last_became_active":"2023-03-03T08:01:21.418220+0000","last_became_peered":"2023-03-03T08:01:21.418220+0000","last_unstale":"2023-03-03T08:01:21.419478+0000","last_undegraded":"2023-03-03T08:01:21.419478+0000","last_fullsized":"2023-03-03T08:01:21.419478+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:32:30.480453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.389498+0000","last_change":"2023-03-03T08:01:20.846292+0000","last_active":"2023-03-03T08:01:21.389498+0000","last_peered":"2023-03-03T08:01:21.389498+0000","last_clean":"2023-03-03T08:01:21.389498+0000","last_became_active":"2023-03-03T08:01:20.845788+0000","last_became_peered":"2023-03-03T08:01:20.845788+0000","last_unstale":"2023-03-03T08:01:21.389498+0000","last_undegraded":"2023-03-03T08:01:21.389498+0000","last_fullsized":"2023-03-03T08:01:21.389498+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:15:48.913046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":60,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.837840+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T08:01:20.837840+0000","last_peered":"2023-03-03T08:01:20.837840+0000","last_clean":"2023-03-03T08:01:20.837840+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T08:01:20.837840+0000","last_undegraded":"2023-03-03T08:01:20.837840+0000","last_fullsized":"2023-03-03T08:01:20.837840+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.410857+0000","last_change":"2023-03-03T08:01:21.401304+0000","last_active":"2023-03-03T08:01:21.410857+0000","last_peered":"2023-03-03T08:01:21.410857+0000","last_clean":"2023-03-03T08:01:21.410857+0000","last_became_active":"2023-03-03T08:01:21.401062+0000","last_became_peered":"2023-03-03T08:01:21.401062+0000","last_unstale":"2023-03-03T08:01:21.410857+0000","last_undegraded":"2023-03-03T08:01:21.410857+0000","last_fullsized":"2023-03-03T08:01:21.410857+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:20:30.614418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.385865+0000","last_change":"2023-03-03T08:01:20.845118+0000","last_active":"2023-03-03T08:01:21.385865+0000","last_peered":"2023-03-03T08:01:21.385865+0000","last_clean":"2023-03-03T08:01:21.385865+0000","last_became_active":"2023-03-03T08:01:20.844872+0000","last_became_peered":"2023-03-03T08:01:20.844872+0000","last_unstale":"2023-03-03T08:01:21.385865+0000","last_undegraded":"2023-03-03T08:01:21.385865+0000","last_fullsized":"2023-03-03T08:01:21.385865+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:31.917263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":60,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.841571+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T08:01:20.841571+0000","last_peered":"2023-03-03T08:01:20.841571+0000","last_clean":"2023-03-03T08:01:20.841571+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T08:01:20.841571+0000","last_undegraded":"2023-03-03T08:01:20.841571+0000","last_fullsized":"2023-03-03T08:01:20.841571+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.410876+0000","last_change":"2023-03-03T08:01:21.405444+0000","last_active":"2023-03-03T08:01:21.410876+0000","last_peered":"2023-03-03T08:01:21.410876+0000","last_clean":"2023-03-03T08:01:21.410876+0000","last_became_active":"2023-03-03T08:01:21.405144+0000","last_became_peered":"2023-03-03T08:01:21.405144+0000","last_unstale":"2023-03-03T08:01:21.410876+0000","last_undegraded":"2023-03-03T08:01:21.410876+0000","last_fullsized":"2023-03-03T08:01:21.410876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:18.426888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"77'1","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.414071+0000","last_change":"2023-03-03T08:01:20.849805+0000","last_active":"2023-03-03T08:01:21.414071+0000","last_peered":"2023-03-03T08:01:21.414071+0000","last_clean":"2023-03-03T08:01:21.414071+0000","last_became_active":"2023-03-03T08:01:20.849516+0000","last_became_peered":"2023-03-03T08:01:20.849516+0000","last_unstale":"2023-03-03T08:01:21.414071+0000","last_undegraded":"2023-03-03T08:01:21.414071+0000","last_fullsized":"2023-03-03T08:01:21.414071+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:49:07.162552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.828651+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T08:01:20.828651+0000","last_peered":"2023-03-03T08:01:20.828651+0000","last_clean":"2023-03-03T08:01:20.828651+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T08:01:20.828651+0000","last_undegraded":"2023-03-03T08:01:20.828651+0000","last_fullsized":"2023-03-03T08:01:20.828651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417090+0000","last_change":"2023-03-03T08:01:21.414678+0000","last_active":"2023-03-03T08:01:21.417090+0000","last_peered":"2023-03-03T08:01:21.417090+0000","last_clean":"2023-03-03T08:01:21.417090+0000","last_became_active":"2023-03-03T08:01:21.414362+0000","last_became_peered":"2023-03-03T08:01:21.414362+0000","last_unstale":"2023-03-03T08:01:21.417090+0000","last_undegraded":"2023-03-03T08:01:21.417090+0000","last_fullsized":"2023-03-03T08:01:21.417090+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:20.810198+0000","last_clean_scrub_stamp":"2023-03-03T08:01:20.810198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:04:35.331292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"77'2","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2023-03-03T08:01:21.417373+0000","last_change":"2023-03-03T08:01:20.832495+0000","last_active":"2023-03-03T08:01:21.417373+0000","last_peered":"2023-03-03T08:01:21.417373+0000","last_clean":"2023-03-03T08:01:21.417373+0000","last_became_active":"2023-03-03T08:01:20.832210+0000","last_became_peered":"2023-03-03T08:01:20.832210+0000","last_unstale":"2023-03-03T08:01:21.417373+0000","last_undegraded":"2023-03-03T08:01:21.417373+0000","last_fullsized":"2023-03-03T08:01:21.417373+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:01:19.796611+0000","last_clean_scrub_stamp":"2023-03-03T08:01:19.796611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:14:54.101606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-03-03T08:01:20.833675+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T08:01:20.833675+0000","last_peered":"2023-03-03T08:01:20.833675+0000","last_clean":"2023-03-03T08:01:20.833675+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T08:01:20.833675+0000","last_undegraded":"2023-03-03T08:01:20.833675+0000","last_fullsized":"2023-03-03T08:01:20.833675+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574921,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11428,"kb_used_data":1500,"kb_used_omap":0,"kb_used_meta":9920,"kb_avail":94360412,"statfs":{"total":96636764160,"available":96625061888,"internally_reserved":0,"allocated":1536000,"data_stored":1008599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10158080},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66000000000000003,"15min":0.69999999999999996},"min":{"1min":0.376,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.86799999999999999,"5min":1.546,"15min":1.546},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70299999999999996,"15min":0.72699999999999998},"min":{"1min":0.54400000000000004,"5min":0.5,"15min":0.5},"max":{"1min":0.86399999999999999,"5min":1.702,"15min":1.702},"last":0.503}]},{"osd":1,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.79700000000000004,"15min":1.0529999999999999},"min":{"1min":0.434,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.105,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.72399999999999998,"15min":0.76400000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.93200000000000005,"5min":1.944,"15min":1.944},"last":0.69699999999999995}]},{"osd":2,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72299999999999998,"15min":0.71499999999999997},"min":{"1min":0.56599999999999995,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.91300000000000003,"5min":1.252,"15min":1.252},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.747,"15min":0.78600000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.95699999999999996,"5min":1.329,"15min":1.329},"last":0.71299999999999997}]},{"osd":3,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71699999999999997,"15min":0.71399999999999997},"min":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.97699999999999998,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73499999999999999,"15min":0.78300000000000003},"min":{"1min":0.51000000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.014,"5min":1.294,"15min":1.294},"last":0.66500000000000004}]},{"osd":4,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63800000000000001,"15min":0.65400000000000003},"min":{"1min":0.436,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84399999999999997,"5min":1.077,"15min":1.077},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67400000000000004,"15min":0.63600000000000001},"min":{"1min":0.45100000000000001,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.90500000000000003,"5min":1.081,"15min":1.081},"last":0.59299999999999997}]},{"osd":5,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.73499999999999999,"15min":0.78900000000000003},"min":{"1min":0.40699999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95899999999999996,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80400000000000005,"15min":0.81999999999999995},"min":{"1min":0.56100000000000005,"5min":0.496,"15min":0.496},"max":{"1min":1.1539999999999999,"5min":1.349,"15min":1.349},"last":0.78000000000000003}]},{"osd":6,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.80600000000000005,"15min":0.82599999999999996},"min":{"1min":0.56000000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0580000000000001,"5min":1.204,"15min":1.204},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.80500000000000005,"15min":0.81899999999999995},"min":{"1min":0.53900000000000003,"5min":0.499,"15min":0.499},"max":{"1min":1.129,"5min":1.194,"15min":1.194},"last":0.68300000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574922,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9804,"kb_used_data":900,"kb_used_omap":0,"kb_used_meta":8896,"kb_avail":94362036,"statfs":{"total":96636764160,"available":96626724864,"internally_reserved":0,"allocated":921600,"data_stored":410062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9109504},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.72899999999999998,"15min":0.69599999999999995},"min":{"1min":0.53800000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.103,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.70599999999999996,"15min":0.69999999999999996},"min":{"1min":0.50600000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.42899999999999999}]},{"osd":1,"last update":"Fri Mar 3 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72299999999999998,"15min":0.68999999999999995},"min":{"1min":0.48799999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1240000000000001,"5min":3.323,"15min":3.323},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.745,"15min":0.73399999999999999},"min":{"1min":0.504,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.70899999999999996}]},{"osd":2,"last update":"Fri Mar 3 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76100000000000001,"15min":0.73499999999999999},"min":{"1min":0.52700000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1140000000000001,"5min":1.54,"15min":1.54},"last":0.5},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.52100000000000002,"5min":0.495,"15min":0.495},"max":{"1min":1.167,"5min":1.698,"15min":1.698},"last":0.73099999999999998}]},{"osd":3,"last update":"Fri Mar 3 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.81699999999999995,"15min":0.79100000000000004},"min":{"1min":0.58499999999999996,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.1559999999999999,"5min":4.8959999999999999,"15min":4.8959999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.79000000000000004,"15min":0.77700000000000002},"min":{"1min":0.61799999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.181,"5min":1.343,"15min":1.343},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Mar 3 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.751,"15min":0.89500000000000002},"min":{"1min":0.505,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.95699999999999996,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.753},{"interface":"front","average":{"1min":0.748,"5min":0.76000000000000001,"15min":0.90800000000000003},"min":{"1min":0.53700000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.012,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.86199999999999999}]},{"osd":5,"last update":"Fri Mar 3 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.77000000000000002},"min":{"1min":0.52600000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.03,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.80200000000000005,"15min":0.82499999999999996},"min":{"1min":0.61799999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.202,"5min":1.429,"15min":1.429},"last":0.78100000000000003}]},{"osd":7,"last update":"Fri Mar 3 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81299999999999994,"15min":0.83699999999999997},"min":{"1min":0.60399999999999998,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.107,"5min":1.452,"15min":1.452},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.79900000000000004,"15min":0.86299999999999999},"min":{"1min":0.60199999999999998,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0820000000000001,"5min":1.579,"15min":1.579},"last":0.84399999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10256,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":9344,"kb_avail":94361584,"statfs":{"total":96636764160,"available":96626262016,"internally_reserved":0,"allocated":925696,"data_stored":411081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9568256},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.47899999999999998,"15min":0.50700000000000001},"min":{"1min":0.31,"5min":0.307,"15min":0.307},"max":{"1min":0.65700000000000003,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.53700000000000003,"15min":0.63300000000000001},"min":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.63200000000000001,"5min":3.972,"15min":3.972},"last":0.51500000000000001}]},{"osd":2,"last update":"Fri Mar 3 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.63,"15min":0.92700000000000005},"min":{"1min":0.38400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.621,"5min":11.897,"15min":11.897},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.70799999999999996,"15min":1.008},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73699999999999999,"5min":11.804,"15min":11.804},"last":0.53200000000000003}]},{"osd":3,"last update":"Fri Mar 3 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.69999999999999996,"15min":0.998},"min":{"1min":0.443,"5min":0.371,"15min":0.371},"max":{"1min":0.76600000000000001,"5min":11.843999999999999,"15min":11.843999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59799999999999998,"15min":0.71999999999999997},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81100000000000005,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.56200000000000006}]},{"osd":4,"last update":"Fri Mar 3 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.79100000000000004,"15min":0.85399999999999998},"min":{"1min":0.51500000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.94299999999999995,"5min":1.613,"15min":1.613},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.85799999999999998},"min":{"1min":0.65500000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.98599999999999999,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.72099999999999997}]},{"osd":5,"last update":"Fri Mar 3 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.80200000000000005,"15min":0.91000000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.95099999999999996,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.78300000000000003,"15min":0.86099999999999999},"min":{"1min":0.55300000000000005,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.86699999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.77200000000000002}]},{"osd":6,"last update":"Fri Mar 3 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.86499999999999999,"15min":0.98299999999999998},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.012,"5min":3.907,"15min":3.907},"last":0.754},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.84599999999999997,"15min":0.92800000000000005},"min":{"1min":0.627,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":0.95199999999999996,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.74199999999999999}]},{"osd":7,"last update":"Fri Mar 3 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.82499999999999996,"15min":0.877},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.417,"5min":1.419,"15min":1.419},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.85399999999999998,"15min":0.995},"min":{"1min":0.58199999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.3560000000000001,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":0.65500000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574921,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11100,"kb_used_data":1492,"kb_used_omap":0,"kb_used_meta":9600,"kb_avail":94360740,"statfs":{"total":96636764160,"available":96625397760,"internally_reserved":0,"allocated":1527808,"data_stored":1007558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9830400},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 08:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.44600000000000001,"15min":0.46800000000000003},"min":{"1min":0.311,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.59899999999999998,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.505,"15min":0.51700000000000002},"min":{"1min":0.36499999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.61899999999999999,"5min":0.871,"15min":0.871},"last":0.47499999999999998}]},{"osd":2,"last update":"Fri Mar 3 08:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52200000000000002,"15min":0.54700000000000004},"min":{"1min":0.38700000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72799999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57099999999999995,"15min":0.58999999999999997},"min":{"1min":0.441,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80800000000000005,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.441}]},{"osd":3,"last update":"Fri Mar 3 08:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53100000000000003,"15min":0.54900000000000004},"min":{"1min":0.36099999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71199999999999997,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60399999999999998,"15min":0.63700000000000001},"min":{"1min":0.48999999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83499999999999996,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.52300000000000002}]},{"osd":4,"last update":"Fri Mar 3 08:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.80900000000000005,"15min":0.83999999999999997},"min":{"1min":0.65800000000000003,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.95099999999999996,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78800000000000003,"15min":0.81699999999999995},"min":{"1min":0.57799999999999996,"5min":0.505,"15min":0.505},"max":{"1min":1.028,"5min":1.605,"15min":1.605},"last":0.70299999999999996}]},{"osd":5,"last update":"Fri Mar 3 08:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.79500000000000004,"15min":0.81899999999999995},"min":{"1min":0.58799999999999997,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.81200000000000006,"15min":0.82199999999999995},"min":{"1min":0.66900000000000004,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1,"5min":1.286,"15min":1.286},"last":0.72999999999999998}]},{"osd":6,"last update":"Fri Mar 3 08:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.82999999999999996,"15min":0.877},"min":{"1min":0.61399999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1419999999999999,"5min":1.623,"15min":1.623},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81799999999999995,"15min":0.85999999999999999},"min":{"1min":0.59899999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0129999999999999,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.71999999999999997}]},{"osd":7,"last update":"Fri Mar 3 08:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82899999999999996,"15min":0.877},"min":{"1min":0.63,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1020000000000001,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.82699999999999996,"15min":0.85799999999999998},"min":{"1min":0.64700000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":0.98699999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.64700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574922,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9956,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94361884,"statfs":{"total":96636764160,"available":96626569216,"internally_reserved":0,"allocated":946176,"data_stored":417916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.53700000000000003,"15min":0.83299999999999996},"min":{"1min":0.34699999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.57699999999999996,"5min":5.7889999999999997,"15min":5.7889999999999997},"last":0.307},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.55100000000000005,"15min":0.71299999999999997},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78000000000000003,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":0.48399999999999999}]},{"osd":1,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.56699999999999995,"15min":0.73199999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68799999999999994,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.68600000000000005,"15min":0.97299999999999998},"min":{"1min":0.45900000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.74099999999999999,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.50700000000000001}]},{"osd":3,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.71999999999999997,"15min":1.1659999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.752,"5min":12.961,"15min":12.961},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.68000000000000005,"15min":0.996},"min":{"1min":0.41599999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.71999999999999997,"5min":6.1449999999999996,"15min":6.1449999999999996},"last":0.40600000000000003}]},{"osd":4,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.92900000000000005,"15min":1.2370000000000001},"min":{"1min":0.57199999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.468,"5min":6.484,"15min":6.484},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.94899999999999995,"15min":1.2190000000000001},"min":{"1min":0.65000000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":0.96599999999999997,"5min":6.5510000000000002,"15min":6.5510000000000002},"last":0.75800000000000001}]},{"osd":5,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.86099999999999999,"15min":1.143},"min":{"1min":0.57299999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.304,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.91400000000000003,"15min":1.2},"min":{"1min":0.58599999999999997,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.95899999999999996,"5min":6.6369999999999996,"15min":6.6369999999999996},"last":0.65100000000000002}]},{"osd":6,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.93999999999999995,"15min":1.206},"min":{"1min":0.65400000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.393,"5min":6.625,"15min":6.625},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.94599999999999995,"15min":1.23},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.347,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.78900000000000003}]},{"osd":7,"last update":"Fri Mar 3 08:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.89300000000000002,"15min":1.1910000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.91900000000000004,"5min":6.7009999999999996,"15min":6.7009999999999996},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.91300000000000003,"15min":1.1779999999999999},"min":{"1min":0.621,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":0.98599999999999999,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.66200000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574920,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14432,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":13504,"kb_avail":94357408,"statfs":{"total":96636764160,"available":96621985792,"internally_reserved":0,"allocated":942080,"data_stored":418775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13828096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.53500000000000003,"15min":0.80300000000000005},"min":{"1min":0.32400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.74199999999999999,"5min":9.5820000000000007,"15min":9.5820000000000007},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.60399999999999998,"15min":0.874},"min":{"1min":0.36499999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72399999999999998,"5min":9.7840000000000007,"15min":9.7840000000000007},"last":0.40799999999999997}]},{"osd":1,"last update":"Fri Mar 3 08:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.49399999999999999,"15min":0.52000000000000002},"min":{"1min":0.34399999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68400000000000005,"5min":1,"15min":1},"last":0.375},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.64800000000000002,"15min":0.95399999999999996},"min":{"1min":0.39500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77400000000000002,"5min":9.702,"15min":9.702},"last":0.58399999999999996}]},{"osd":2,"last update":"Fri Mar 3 08:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.65800000000000003,"15min":0.90700000000000003},"min":{"1min":0.46200000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81200000000000006,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.70699999999999996,"15min":0.99099999999999999},"min":{"1min":0.49399999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.93400000000000005,"5min":9.734,"15min":9.734},"last":0.54800000000000004}]},{"osd":4,"last update":"Fri Mar 3 08:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.86199999999999999,"15min":1.1319999999999999},"min":{"1min":0.59899999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.96699999999999997,"5min":9.8629999999999995,"15min":9.8629999999999995},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.81799999999999995,"15min":0.84099999999999997},"min":{"1min":0.55700000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.97099999999999997,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":0.71699999999999997}]},{"osd":5,"last update":"Fri Mar 3 08:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.80700000000000005,"15min":0.82799999999999996},"min":{"1min":0.59999999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.88400000000000001,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.88600000000000001,"15min":1.1839999999999999},"min":{"1min":0.58999999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.99099999999999999,"5min":9.6600000000000001,"15min":9.6600000000000001},"last":0.747}]},{"osd":6,"last update":"Fri Mar 3 08:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.89900000000000002,"15min":1.1990000000000001},"min":{"1min":0.629,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.89800000000000002,"5min":9.8190000000000008,"15min":9.8190000000000008},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80900000000000005,"15min":0.86099999999999999},"min":{"1min":0.60299999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.014,"5min":1.609,"15min":1.609},"last":0.65600000000000003}]},{"osd":7,"last update":"Fri Mar 3 08:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.91700000000000004,"15min":1.2350000000000001},"min":{"1min":0.65400000000000003,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.024,"5min":9.625,"15min":9.625},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.92600000000000005,"15min":1.208},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":0.97599999999999998,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":0.70699999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574920,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10480,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":9536,"kb_avail":94361360,"statfs":{"total":96636764160,"available":96626032640,"internally_reserved":0,"allocated":958464,"data_stored":419861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9764864},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.79500000000000004,"15min":0.96399999999999997},"min":{"1min":0.58499999999999996,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.96399999999999997,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.69799999999999995,"15min":0.67000000000000004},"min":{"1min":0.45900000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89300000000000002,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.58199999999999996}]},{"osd":1,"last update":"Fri Mar 3 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.83899999999999997,"15min":1.079},"min":{"1min":0.53500000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94399999999999995,"5min":9.5709999999999997,"15min":9.5709999999999997},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.79700000000000004,"15min":0.91300000000000003},"min":{"1min":0.56499999999999995,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.89000000000000001,"5min":5.266,"15min":5.266},"last":0.61299999999999999}]},{"osd":2,"last update":"Fri Mar 3 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.83799999999999997,"15min":1.0509999999999999},"min":{"1min":0.53700000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92700000000000005,"5min":8.4529999999999994,"15min":8.4529999999999994},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.82499999999999996,"15min":0.98399999999999999},"min":{"1min":0.59999999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.97899999999999998,"5min":6.6829999999999998,"15min":6.6829999999999998},"last":0.73999999999999999}]},{"osd":3,"last update":"Fri Mar 3 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.73599999999999999,"15min":0.73499999999999999},"min":{"1min":0.54300000000000004,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.97899999999999998,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.78900000000000003,"15min":0.80200000000000005},"min":{"1min":0.64100000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.99299999999999999,"5min":1.389,"15min":1.389},"last":0.81999999999999995}]},{"osd":5,"last update":"Fri Mar 3 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.71399999999999997,"15min":0.73999999999999999},"min":{"1min":0.49299999999999999,"5min":0.434,"15min":0.434},"max":{"1min":0.98799999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.71599999999999997,"15min":0.71799999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91100000000000003,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.68899999999999995}]},{"osd":6,"last update":"Fri Mar 3 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76100000000000001,"15min":0.77700000000000002},"min":{"1min":0.54700000000000004,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0029999999999999,"5min":1.032,"15min":1.032},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.80000000000000004,"15min":0.81599999999999995},"min":{"1min":0.60399999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.96499999999999997,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.70799999999999996}]},{"osd":7,"last update":"Fri Mar 3 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.78400000000000003,"15min":0.81799999999999995},"min":{"1min":0.59199999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0429999999999999,"5min":1.679,"15min":1.679},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.85699999999999998,"15min":0.92000000000000004},"min":{"1min":0.64900000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.085,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.90800000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574921,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10012,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":9088,"kb_avail":94361828,"statfs":{"total":96636764160,"available":96626511872,"internally_reserved":0,"allocated":937984,"data_stored":417224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9306112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.64300000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.75900000000000001,"15min":0.745},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0580000000000001,"5min":2.528,"15min":2.528},"last":0.61599999999999999}]},{"osd":1,"last update":"Fri Mar 3 08:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73799999999999999,"15min":0.746},"min":{"1min":0.55000000000000004,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.97299999999999998,"5min":1.038,"15min":1.038},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68799999999999994,"15min":0.64200000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.89400000000000002,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.53800000000000003}]},{"osd":2,"last update":"Fri Mar 3 08:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.69199999999999995,"15min":0.68300000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72299999999999998,"15min":0.71499999999999997},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.997,"5min":1.321,"15min":1.321},"last":0.55400000000000005}]},{"osd":3,"last update":"Fri Mar 3 08:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.72099999999999997,"15min":0.69699999999999995},"min":{"1min":0.51400000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.754,"15min":0.73799999999999999},"min":{"1min":0.56999999999999995,"5min":0.501,"15min":0.501},"max":{"1min":0.92800000000000005,"5min":1.012,"15min":1.012},"last":0.59599999999999997}]},{"osd":4,"last update":"Fri Mar 3 08:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.67200000000000004,"15min":0.66500000000000004},"min":{"1min":0.47899999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90900000000000003,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79300000000000004,"15min":0.78300000000000003},"min":{"1min":0.66600000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.57599999999999996}]},{"osd":6,"last update":"Fri Mar 3 08:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.746,"15min":0.79200000000000004},"min":{"1min":0.54000000000000004,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.012,"5min":1.071,"15min":1.071},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.71699999999999997,"15min":0.72299999999999998},"min":{"1min":0.48999999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.93899999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.496}]},{"osd":7,"last update":"Fri Mar 3 08:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.58699999999999997,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.80200000000000005,"15min":0.82799999999999996},"min":{"1min":0.60299999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.90500000000000003,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.67000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:01:26.922 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-03-03T08:01:26.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2023-03-03T08:01:27.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.205+0000 7efc039ff700 1 Processor -- start 2023-03-03T08:01:27.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.205+0000 7efc039ff700 1 -- start start 2023-03-03T08:01:27.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.206+0000 7efc039ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efbfc15ca70 0x7efbfc0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:27.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.206+0000 7efc039ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efbfc06b4b0 0x7efbfc0cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:27.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.207+0000 7efc039ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbfc0cf980 con 0x7efbfc0d00a0 2023-03-03T08:01:27.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.207+0000 7efc039ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbfc0cd5b0 con 0x7efbfc15ca70 2023-03-03T08:01:27.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.207+0000 7efc039ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbfc0d0540 con 0x7efbfc06b4b0 2023-03-03T08:01:27.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.207+0000 7efc0179b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efbfc15ca70 0x7efbfc0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:27.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.207+0000 7efc0179b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efbfc15ca70 0x7efbfc0cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60232/0 (socket says 172.21.15.98:60232) 2023-03-03T08:01:27.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.207+0000 7efc0179b700 1 -- 172.21.15.98:0/851567129 learned_addr learned my addr 172.21.15.98:0/851567129 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:27.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.207+0000 7efc00f9a700 1 --2- 172.21.15.98:0/851567129 >> v2:172.21.15.100:3300/0 conn(0x7efbfc06b4b0 0x7efbfc0cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:27.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.207+0000 7efc0179b700 1 -- 172.21.15.98:0/851567129 >> v2:172.21.15.100:3300/0 conn(0x7efbfc06b4b0 msgr2=0x7efbfc0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:27.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.207+0000 7efc0179b700 1 --2- 172.21.15.98:0/851567129 >> v2:172.21.15.100:3300/0 conn(0x7efbfc06b4b0 0x7efbfc0cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:27.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.207+0000 7efc0179b700 1 -- 172.21.15.98:0/851567129 >> v1:172.21.15.100:6789/0 conn(0x7efbfc0d00a0 legacy=0x7efbfc0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:27.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.208+0000 7efc0179b700 1 -- 172.21.15.98:0/851567129 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc0d2fd0 con 0x7efbfc15ca70 2023-03-03T08:01:27.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.208+0000 7efc0179b700 1 --2- 172.21.15.98:0/851567129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbfc15ca70 0x7efbfc0cf440 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7efbfc160560 tx=0x7efbf8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b746c0182ec2d9c5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:27.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.208+0000 7efbf3fff700 1 -- 172.21.15.98:0/851567129 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf800b870 con 0x7efbfc15ca70 2023-03-03T08:01:27.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.208+0000 7efbf3fff700 1 -- 172.21.15.98:0/851567129 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf800b9d0 con 0x7efbfc15ca70 2023-03-03T08:01:27.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.208+0000 7efbf3fff700 1 -- 172.21.15.98:0/851567129 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf800bd60 con 0x7efbfc15ca70 2023-03-03T08:01:27.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.209+0000 7efc039ff700 1 -- 172.21.15.98:0/851567129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbfc15ca70 msgr2=0x7efbfc0cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:27.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.209+0000 7efc039ff700 1 --2- 172.21.15.98:0/851567129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbfc15ca70 0x7efbfc0cf440 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7efbfc160560 tx=0x7efbf8009bd0 comp rx=0 tx=0).stop 2023-03-03T08:01:27.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.209+0000 7efc039ff700 1 -- 172.21.15.98:0/851567129 shutdown_connections 2023-03-03T08:01:27.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.209+0000 7efc039ff700 1 --2- 172.21.15.98:0/851567129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbfc15ca70 0x7efbfc0cf440 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:27.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.209+0000 7efc039ff700 1 --2- 172.21.15.98:0/851567129 >> v2:172.21.15.100:3300/0 conn(0x7efbfc06b4b0 0x7efbfc0cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:27.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.209+0000 7efc039ff700 1 -- 172.21.15.98:0/851567129 >> 172.21.15.98:0/851567129 conn(0x7efbfc073e70 msgr2=0x7efbfc06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:27.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.209+0000 7efc039ff700 1 -- 172.21.15.98:0/851567129 shutdown_connections 2023-03-03T08:01:27.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.209+0000 7efc039ff700 1 -- 172.21.15.98:0/851567129 wait complete. 2023-03-03T08:01:27.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.212+0000 7efc039ff700 1 Processor -- start 2023-03-03T08:01:27.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.212+0000 7efc039ff700 1 -- start start 2023-03-03T08:01:27.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.213+0000 7efc039ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbfc06b4b0 0x7efbfc1f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:27.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.213+0000 7efc039ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efbfc0d00a0 0x7efbfc0ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:27.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.214+0000 7efc039ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efbfc15ca70 0x7efbfc0b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:27.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.214+0000 7efc039ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc1f21c0 con 0x7efbfc06b4b0 2023-03-03T08:01:27.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.214+0000 7efc039ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0b3410 con 0x7efbfc0d00a0 2023-03-03T08:01:27.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.214+0000 7efc039ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0b36f0 con 0x7efbfc15ca70 2023-03-03T08:01:27.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.214+0000 7efc0179b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbfc06b4b0 0x7efbfc1f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:27.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.214+0000 7efc0179b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbfc06b4b0 0x7efbfc1f1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60234/0 (socket says 172.21.15.98:60234) 2023-03-03T08:01:27.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.214+0000 7efc0179b700 1 -- 172.21.15.98:0/741123443 learned_addr learned my addr 172.21.15.98:0/741123443 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:27.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.214+0000 7efc00f9a700 1 --2- 172.21.15.98:0/741123443 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efbfc0d00a0 0x7efbfc0ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:27.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.214+0000 7efc01f9c700 1 --2- 172.21.15.98:0/741123443 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efbfc15ca70 0x7efbfc0b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:27.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.214+0000 7efc0179b700 1 -- 172.21.15.98:0/741123443 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efbfc15ca70 msgr2=0x7efbfc0b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:27.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.214+0000 7efc0179b700 1 --2- 172.21.15.98:0/741123443 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efbfc15ca70 0x7efbfc0b44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:27.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.214+0000 7efc0179b700 1 -- 172.21.15.98:0/741123443 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efbfc0d00a0 msgr2=0x7efbfc0ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:27.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.214+0000 7efc0179b700 1 --2- 172.21.15.98:0/741123443 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efbfc0d00a0 0x7efbfc0ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:27.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.214+0000 7efc0179b700 1 -- 172.21.15.98:0/741123443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf8009550 con 0x7efbfc06b4b0 2023-03-03T08:01:27.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.214+0000 7efc01f9c700 1 --2- 172.21.15.98:0/741123443 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efbfc15ca70 0x7efbfc0b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:27.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.215+0000 7efc0179b700 1 --2- 172.21.15.98:0/741123443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbfc06b4b0 0x7efbfc1f1c80 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7efbf8000c00 tx=0x7efbf8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:27.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.215+0000 7efbf1ffb700 1 -- 172.21.15.98:0/741123443 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf80129c0 con 0x7efbfc06b4b0 2023-03-03T08:01:27.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.215+0000 7efbf1ffb700 1 -- 172.21.15.98:0/741123443 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf801e070 con 0x7efbfc06b4b0 2023-03-03T08:01:27.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.215+0000 7efbf1ffb700 1 -- 172.21.15.98:0/741123443 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf8012d20 con 0x7efbfc06b4b0 2023-03-03T08:01:27.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.215+0000 7efc039ff700 1 -- 172.21.15.98:0/741123443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbfc0cd900 con 0x7efbfc06b4b0 2023-03-03T08:01:27.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.215+0000 7efc039ff700 1 -- 172.21.15.98:0/741123443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbfc0d1800 con 0x7efbfc06b4b0 2023-03-03T08:01:27.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.216+0000 7efc039ff700 1 -- 172.21.15.98:0/741123443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbfc0cdbe0 con 0x7efbfc06b4b0 2023-03-03T08:01:27.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.217+0000 7efbf1ffb700 1 -- 172.21.15.98:0/741123443 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7efbf800f050 con 0x7efbfc06b4b0 2023-03-03T08:01:27.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.219+0000 7efbf1ffb700 1 --2- 172.21.15.98:0/741123443 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efbe00684a0 0x7efbe006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:27.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.219+0000 7efbf1ffb700 1 -- 172.21.15.98:0/741123443 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7efbf8087090 con 0x7efbfc06b4b0 2023-03-03T08:01:27.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.219+0000 7efc00f9a700 1 --2- 172.21.15.98:0/741123443 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efbe00684a0 0x7efbe006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:27.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.220+0000 7efc00f9a700 1 --2- 172.21.15.98:0/741123443 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efbe00684a0 0x7efbe006a950 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0ceed0 tx=0x7efbec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:27.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.223+0000 7efbf1ffb700 1 -- 172.21.15.98:0/741123443 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efbf8053750 con 0x7efbfc06b4b0 2023-03-03T08:01:27.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.491+0000 7efc039ff700 1 -- 172.21.15.98:0/741123443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7efbfc04e6e0 con 0x7efbfc06b4b0 2023-03-03T08:01:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.496+0000 7efbf1ffb700 1 -- 172.21.15.98:0/741123443 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 184+0+0 (secure 0 0 0) 0x7efbf8020020 con 0x7efbfc06b4b0 2023-03-03T08:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.498+0000 7efc039ff700 1 -- 172.21.15.98:0/741123443 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efbe00684a0 msgr2=0x7efbe006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.498+0000 7efc039ff700 1 --2- 172.21.15.98:0/741123443 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efbe00684a0 0x7efbe006a950 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0ceed0 tx=0x7efbec008040 comp rx=0 tx=0).stop 2023-03-03T08:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.498+0000 7efc039ff700 1 -- 172.21.15.98:0/741123443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbfc06b4b0 msgr2=0x7efbfc1f1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.498+0000 7efc039ff700 1 --2- 172.21.15.98:0/741123443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbfc06b4b0 0x7efbfc1f1c80 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7efbf8000c00 tx=0x7efbf8004750 comp rx=0 tx=0).stop 2023-03-03T08:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.498+0000 7efc039ff700 1 -- 172.21.15.98:0/741123443 shutdown_connections 2023-03-03T08:01:27.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.498+0000 7efc039ff700 1 --2- 172.21.15.98:0/741123443 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7efbe00684a0 0x7efbe006a950 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:27.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.498+0000 7efc039ff700 1 --2- 172.21.15.98:0/741123443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbfc06b4b0 0x7efbfc1f1c80 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:27.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.498+0000 7efc039ff700 1 --2- 172.21.15.98:0/741123443 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efbfc15ca70 0x7efbfc0b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:27.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.498+0000 7efc039ff700 1 --2- 172.21.15.98:0/741123443 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efbfc0d00a0 0x7efbfc0ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:27.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.498+0000 7efc039ff700 1 -- 172.21.15.98:0/741123443 >> 172.21.15.98:0/741123443 conn(0x7efbfc073e70 msgr2=0x7efbfc1e52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:27.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.499+0000 7efc039ff700 1 -- 172.21.15.98:0/741123443 shutdown_connections 2023-03-03T08:01:27.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.499+0000 7efc039ff700 1 -- 172.21.15.98:0/741123443 wait complete. 2023-03-03T08:01:27.502 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.0 2023-03-03T08:01:27.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:01:27.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.771+0000 7f2adcbff700 1 Processor -- start 2023-03-03T08:01:27.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.771+0000 7f2adcbff700 1 -- start start 2023-03-03T08:01:27.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.773+0000 7f2adcbff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2ad80cd8d0 0x7f2ad80b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:27.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.773+0000 7f2adcbff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad80cdd70 con 0x7f2ad80b6610 2023-03-03T08:01:27.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.773+0000 7f2adcbff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad80d1a00 con 0x7f2ad815cc00 2023-03-03T08:01:27.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.773+0000 7f2adcbff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ad80ce040 con 0x7f2ad80cd8d0 2023-03-03T08:01:27.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.773+0000 7f2ad6d9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2ad80b6610 0x7f2ad80cf4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54092/0 (socket says 172.21.15.98:54092) 2023-03-03T08:01:27.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.773+0000 7f2ad6d9d700 1 -- 172.21.15.98:0/2265749228 learned_addr learned my addr 172.21.15.98:0/2265749228 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:27.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.773+0000 7f2acffff700 1 --2- 172.21.15.98:0/2265749228 >> v2:172.21.15.100:3300/0 conn(0x7f2ad80cd8d0 0x7f2ad80b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:27.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.774+0000 7f2ad5d9b700 1 -- 172.21.15.98:0/2265749228 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607770537 0 0) 0x7f2ad80cdd70 con 0x7f2ad80b6610 2023-03-03T08:01:27.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.774+0000 7f2ad5d9b700 1 -- 172.21.15.98:0/2265749228 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac40035b0 con 0x7f2ad80b6610 2023-03-03T08:01:27.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.774+0000 7f2acffff700 1 -- 172.21.15.98:0/2265749228 >> v1:172.21.15.100:6789/0 conn(0x7f2ad815cc00 legacy=0x7f2ad80d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:27.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.774+0000 7f2ad5d9b700 1 -- 172.21.15.98:0/2265749228 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769295301 0 0) 0x7f2ad80d1a00 con 0x7f2ad815cc00 2023-03-03T08:01:27.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.774+0000 7f2acffff700 1 -- 172.21.15.98:0/2265749228 >> v1:172.21.15.98:6789/0 conn(0x7f2ad80b6610 legacy=0x7f2ad80cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:27.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.774+0000 7f2acffff700 1 -- 172.21.15.98:0/2265749228 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad80d1ce0 con 0x7f2ad80cd8d0 2023-03-03T08:01:27.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.774+0000 7f2acffff700 1 --2- 172.21.15.98:0/2265749228 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ad80cd8d0 0x7f2ad80b60d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009840 tx=0x7f2ac0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=470b77d967a640d8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:27.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.775+0000 7f2ad5d9b700 1 -- 172.21.15.98:0/2265749228 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0011830 con 0x7f2ad80cd8d0 2023-03-03T08:01:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.775+0000 7f2ad5d9b700 1 -- 172.21.15.98:0/2265749228 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac0011990 con 0x7f2ad80cd8d0 2023-03-03T08:01:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.775+0000 7f2ad5d9b700 1 -- 172.21.15.98:0/2265749228 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0011d20 con 0x7f2ad80cd8d0 2023-03-03T08:01:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.775+0000 7f2adcbff700 1 -- 172.21.15.98:0/2265749228 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ad80cd8d0 msgr2=0x7f2ad80b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.775+0000 7f2adcbff700 1 --2- 172.21.15.98:0/2265749228 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ad80cd8d0 0x7f2ad80b60d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009840 tx=0x7f2ac0009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.775+0000 7f2adcbff700 1 -- 172.21.15.98:0/2265749228 shutdown_connections 2023-03-03T08:01:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.775+0000 7f2adcbff700 1 --2- 172.21.15.98:0/2265749228 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ad80cd8d0 0x7f2ad80b60d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:27.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.775+0000 7f2adcbff700 1 -- 172.21.15.98:0/2265749228 >> 172.21.15.98:0/2265749228 conn(0x7f2ad806aee0 msgr2=0x7f2ad806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:27.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.775+0000 7f2adcbff700 1 -- 172.21.15.98:0/2265749228 shutdown_connections 2023-03-03T08:01:27.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.775+0000 7f2adcbff700 1 -- 172.21.15.98:0/2265749228 wait complete. 2023-03-03T08:01:27.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.777+0000 7f2adcbff700 1 Processor -- start 2023-03-03T08:01:27.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.777+0000 7f2adcbff700 1 -- start start 2023-03-03T08:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.778+0000 7f2adcbff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ad80b6610 0x7f2ad814f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.778+0000 7f2adcbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ad80cd8d0 0x7f2ad80ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.778+0000 7f2adcbff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ad815cc00 0x7f2ad80b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.778+0000 7f2adcbff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad80b47c0 con 0x7f2ad815cc00 2023-03-03T08:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.778+0000 7f2adcbff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad815f880 con 0x7f2ad80b6610 2023-03-03T08:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.778+0000 7f2adcbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad815fee0 con 0x7f2ad80cd8d0 2023-03-03T08:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.779+0000 7f2ad6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ad815cc00 0x7f2ad80b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.779+0000 7f2ad6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ad815cc00 0x7f2ad80b4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60244/0 (socket says 172.21.15.98:60244) 2023-03-03T08:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.779+0000 7f2ad6d9d700 1 -- 172.21.15.98:0/3711698657 learned_addr learned my addr 172.21.15.98:0/3711698657 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.779+0000 7f2ad659c700 1 --2- 172.21.15.98:0/3711698657 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ad80b6610 0x7f2ad814f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.779+0000 7f2acffff700 1 --2- 172.21.15.98:0/3711698657 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ad80cd8d0 0x7f2ad80ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.779+0000 7f2ad6d9d700 1 -- 172.21.15.98:0/3711698657 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ad80cd8d0 msgr2=0x7f2ad80ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.779+0000 7f2ad6d9d700 1 --2- 172.21.15.98:0/3711698657 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ad80cd8d0 0x7f2ad80ca350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.779+0000 7f2ad6d9d700 1 -- 172.21.15.98:0/3711698657 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ad80b6610 msgr2=0x7f2ad814f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.779+0000 7f2ad6d9d700 1 --2- 172.21.15.98:0/3711698657 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ad80b6610 0x7f2ad814f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.779+0000 7f2ad6d9d700 1 -- 172.21.15.98:0/3711698657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac00094a0 con 0x7f2ad815cc00 2023-03-03T08:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.780+0000 7f2ad6d9d700 1 --2- 172.21.15.98:0/3711698657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ad815cc00 0x7f2ad80b4be0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000caa0 tx=0x7f2ad000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.780+0000 7f2aceffd700 1 -- 172.21.15.98:0/3711698657 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad000a9e0 con 0x7f2ad815cc00 2023-03-03T08:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.780+0000 7f2aceffd700 1 -- 172.21.15.98:0/3711698657 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad000ab40 con 0x7f2ad815cc00 2023-03-03T08:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.780+0000 7f2aceffd700 1 -- 172.21.15.98:0/3711698657 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0012400 con 0x7f2ad815cc00 2023-03-03T08:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.780+0000 7f2adcbff700 1 -- 172.21.15.98:0/3711698657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad80d22a0 con 0x7f2ad815cc00 2023-03-03T08:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.780+0000 7f2adcbff700 1 -- 172.21.15.98:0/3711698657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad80d1fc0 con 0x7f2ad815cc00 2023-03-03T08:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.781+0000 7f2adcbff700 1 -- 172.21.15.98:0/3711698657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad80d2ed0 con 0x7f2ad815cc00 2023-03-03T08:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.782+0000 7f2aceffd700 1 -- 172.21.15.98:0/3711698657 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f2ad0019030 con 0x7f2ad815cc00 2023-03-03T08:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.784+0000 7f2aceffd700 1 --2- 172.21.15.98:0/3711698657 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2ab80684a0 0x7f2ab806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.784+0000 7f2aceffd700 1 -- 172.21.15.98:0/3711698657 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2ad00859d0 con 0x7f2ad815cc00 2023-03-03T08:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.784+0000 7f2ad659c700 1 --2- 172.21.15.98:0/3711698657 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2ab80684a0 0x7f2ab806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.785+0000 7f2ad659c700 1 --2- 172.21.15.98:0/3711698657 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2ab80684a0 0x7f2ab806a950 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8000f60 tx=0x7f2ac8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:27.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:27.788+0000 7f2aceffd700 1 -- 172.21.15.98:0/3711698657 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2ad0052150 con 0x7f2ad815cc00 2023-03-03T08:01:28.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.024+0000 7f2adcbff700 1 -- 172.21.15.98:0/3711698657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ad80729b0 con 0x7f2ad815cc00 2023-03-03T08:01:28.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.024+0000 7f2aceffd700 1 -- 172.21.15.98:0/3711698657 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f2ad0055770 con 0x7f2ad815cc00 2023-03-03T08:01:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:28.026 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6050,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/318891906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":318891906},{"type":"v1","addr":"172.21.15.98:6836","nonce":318891906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1280578660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1280578660},{"type":"v1","addr":"172.21.15.98:6837","nonce":1280578660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6906,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1395511883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1395511883},{"type":"v1","addr":"172.21.15.100:6834","nonce":1395511883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:01:21.379777+0000","modified":"2023-03-03T08:01:24.835314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6909},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6909":{"gid":6909,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/341424177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":341424177},{"type":"v1","addr":"172.21.15.100:6835","nonce":341424177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-03-03T08:01:28.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.027+0000 7f2adcbff700 1 -- 172.21.15.98:0/3711698657 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2ab80684a0 msgr2=0x7f2ab806a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:28.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.027+0000 7f2adcbff700 1 --2- 172.21.15.98:0/3711698657 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2ab80684a0 0x7f2ab806a950 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8000f60 tx=0x7f2ac8009040 comp rx=0 tx=0).stop 2023-03-03T08:01:28.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.027+0000 7f2adcbff700 1 -- 172.21.15.98:0/3711698657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ad815cc00 msgr2=0x7f2ad80b4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:28.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.027+0000 7f2adcbff700 1 --2- 172.21.15.98:0/3711698657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ad815cc00 0x7f2ad80b4be0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000caa0 tx=0x7f2ad000cdb0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.027+0000 7f2adcbff700 1 -- 172.21.15.98:0/3711698657 shutdown_connections 2023-03-03T08:01:28.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.027+0000 7f2adcbff700 1 --2- 172.21.15.98:0/3711698657 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2ab80684a0 0x7f2ab806a950 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.027+0000 7f2adcbff700 1 --2- 172.21.15.98:0/3711698657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ad815cc00 0x7f2ad80b4be0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.027+0000 7f2adcbff700 1 --2- 172.21.15.98:0/3711698657 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ad80b6610 0x7f2ad814f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.027+0000 7f2adcbff700 1 --2- 172.21.15.98:0/3711698657 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ad80cd8d0 0x7f2ad80ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.027+0000 7f2adcbff700 1 -- 172.21.15.98:0/3711698657 >> 172.21.15.98:0/3711698657 conn(0x7f2ad806aee0 msgr2=0x7f2ad80b6b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:28.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.027+0000 7f2adcbff700 1 -- 172.21.15.98:0/3711698657 shutdown_connections 2023-03-03T08:01:28.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.027+0000 7f2adcbff700 1 -- 172.21.15.98:0/3711698657 wait complete. 2023-03-03T08:01:28.030 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 40 2023-03-03T08:01:28.458 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-03-03T08:01:21.379777+0000', 'modified': '2023-03-03T08:01:24.835314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6909}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6909': {'gid': 6909, 'name': 'd', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.100:6835/341424177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6833', 'nonce': 341424177}, {'type': 'v1', 'addr': '172.21.15.100:6835', 'nonce': 341424177}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-03-03T08:01:28.459 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-03-03T08:01:28.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-03-03T08:01:28.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.706+0000 7f1bce79d700 1 Processor -- start 2023-03-03T08:01:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.706+0000 7f1bce79d700 1 -- start start 2023-03-03T08:01:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.708+0000 7f1bce79d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1bc81609b0 0x7f1bc8160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.708+0000 7f1bce79d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1bc80cf570 0x7f1bc815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.708+0000 7f1bce79d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc81612c0 con 0x7f1bc815db30 2023-03-03T08:01:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.708+0000 7f1bce79d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bc80cebb0 con 0x7f1bc81609b0 2023-03-03T08:01:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.708+0000 7f1bce79d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc80cd780 con 0x7f1bc80cf570 2023-03-03T08:01:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.708+0000 7f1bcd79b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1bc81609b0 0x7f1bc8160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.708+0000 7f1bccf9a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1bc80cf570 0x7f1bc815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.709+0000 7f1bcd79b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1bc81609b0 0x7f1bc8160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:48142/0 (socket says 172.21.15.98:48142) 2023-03-03T08:01:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.709+0000 7f1bcd79b700 1 -- 172.21.15.98:0/3326307100 learned_addr learned my addr 172.21.15.98:0/3326307100 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.709+0000 7f1bcdf9c700 1 --1- 172.21.15.98:0/3326307100 >> v1:172.21.15.100:6789/0 conn(0x7f1bc815db30 0x7f1bc80b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:53384/0 (socket says 172.21.15.98:53384) 2023-03-03T08:01:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.709+0000 7f1bccf9a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1bc80cf570 0x7f1bc815d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:56144/0 (socket says 172.21.15.98:56144) 2023-03-03T08:01:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.709+0000 7f1bbffff700 1 -- 172.21.15.98:0/3326307100 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871514859 0 0) 0x7f1bc81612c0 con 0x7f1bc815db30 2023-03-03T08:01:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.709+0000 7f1bbffff700 1 -- 172.21.15.98:0/3326307100 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb40035b0 con 0x7f1bc815db30 2023-03-03T08:01:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.709+0000 7f1bcd79b700 1 -- 172.21.15.98:0/3326307100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1bc80cf570 msgr2=0x7f1bc815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.709+0000 7f1bcd79b700 1 --2- 172.21.15.98:0/3326307100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1bc80cf570 0x7f1bc815d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.709+0000 7f1bcd79b700 1 -- 172.21.15.98:0/3326307100 >> v1:172.21.15.100:6789/0 conn(0x7f1bc815db30 legacy=0x7f1bc80b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:28.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.709+0000 7f1bcd79b700 1 -- 172.21.15.98:0/3326307100 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc80cda60 con 0x7f1bc81609b0 2023-03-03T08:01:28.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.709+0000 7f1bccf9a700 1 --2- 172.21.15.98:0/3326307100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1bc80cf570 0x7f1bc815d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:28.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.710+0000 7f1bcd79b700 1 --2- 172.21.15.98:0/3326307100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1bc81609b0 0x7f1bc8160d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1bc40098f0 tx=0x7f1bc4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4e73b979e598da3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:28.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.710+0000 7f1bbffff700 1 -- 172.21.15.98:0/3326307100 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc4011830 con 0x7f1bc81609b0 2023-03-03T08:01:28.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.710+0000 7f1bbffff700 1 -- 172.21.15.98:0/3326307100 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc4011990 con 0x7f1bc81609b0 2023-03-03T08:01:28.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.710+0000 7f1bbffff700 1 -- 172.21.15.98:0/3326307100 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc4011d20 con 0x7f1bc81609b0 2023-03-03T08:01:28.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.710+0000 7f1bce79d700 1 -- 172.21.15.98:0/3326307100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1bc81609b0 msgr2=0x7f1bc8160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:28.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.710+0000 7f1bce79d700 1 --2- 172.21.15.98:0/3326307100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1bc81609b0 0x7f1bc8160d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1bc40098f0 tx=0x7f1bc4009c00 comp rx=0 tx=0).stop 2023-03-03T08:01:28.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.711+0000 7f1bce79d700 1 -- 172.21.15.98:0/3326307100 shutdown_connections 2023-03-03T08:01:28.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.711+0000 7f1bce79d700 1 --2- 172.21.15.98:0/3326307100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1bc81609b0 0x7f1bc8160d80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.711+0000 7f1bce79d700 1 --2- 172.21.15.98:0/3326307100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1bc80cf570 0x7f1bc815d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.711+0000 7f1bce79d700 1 -- 172.21.15.98:0/3326307100 >> 172.21.15.98:0/3326307100 conn(0x7f1bc806b880 msgr2=0x7f1bc80cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:28.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.711+0000 7f1bce79d700 1 -- 172.21.15.98:0/3326307100 shutdown_connections 2023-03-03T08:01:28.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.711+0000 7f1bce79d700 1 -- 172.21.15.98:0/3326307100 wait complete. 2023-03-03T08:01:28.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.713+0000 7f1bce79d700 1 Processor -- start 2023-03-03T08:01:28.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.713+0000 7f1bce79d700 1 -- start start 2023-03-03T08:01:28.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.714+0000 7f1bce79d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1bc80cf570 0x7f1bc81f1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.714+0000 7f1bce79d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc815db30 0x7f1bc81f1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.714+0000 7f1bce79d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1bc81609b0 0x7f1bc80d16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.714+0000 7f1bce79d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc81f22d0 con 0x7f1bc815db30 2023-03-03T08:01:28.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.714+0000 7f1bce79d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc81f25b0 con 0x7f1bc81609b0 2023-03-03T08:01:28.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.714+0000 7f1bce79d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc81f2890 con 0x7f1bc80cf570 2023-03-03T08:01:28.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.714+0000 7f1bccf9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc815db30 0x7f1bc81f1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.714+0000 7f1bccf9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc815db30 0x7f1bc81f1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60256/0 (socket says 172.21.15.98:60256) 2023-03-03T08:01:28.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.714+0000 7f1bccf9a700 1 -- 172.21.15.98:0/1462536826 learned_addr learned my addr 172.21.15.98:0/1462536826 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:28.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.714+0000 7f1bcd79b700 1 --2- 172.21.15.98:0/1462536826 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1bc80cf570 0x7f1bc81f1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.714+0000 7f1bcdf9c700 1 --2- 172.21.15.98:0/1462536826 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1bc81609b0 0x7f1bc80d16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.715+0000 7f1bccf9a700 1 -- 172.21.15.98:0/1462536826 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1bc80cf570 msgr2=0x7f1bc81f1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:28.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.715+0000 7f1bccf9a700 1 --2- 172.21.15.98:0/1462536826 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1bc80cf570 0x7f1bc81f1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.715+0000 7f1bccf9a700 1 -- 172.21.15.98:0/1462536826 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1bc81609b0 msgr2=0x7f1bc80d16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:28.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.715+0000 7f1bccf9a700 1 --2- 172.21.15.98:0/1462536826 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1bc81609b0 0x7f1bc80d16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.715+0000 7f1bccf9a700 1 -- 172.21.15.98:0/1462536826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb80094a0 con 0x7f1bc815db30 2023-03-03T08:01:28.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.715+0000 7f1bccf9a700 1 --2- 172.21.15.98:0/1462536826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc815db30 0x7f1bc81f1c40 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f1bb800ebe0 tx=0x7f1bb800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:28.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.715+0000 7f1bbdffb700 1 -- 172.21.15.98:0/1462536826 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb800c850 con 0x7f1bc815db30 2023-03-03T08:01:28.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.715+0000 7f1bbdffb700 1 -- 172.21.15.98:0/1462536826 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb800c9b0 con 0x7f1bc815db30 2023-03-03T08:01:28.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.715+0000 7f1bce79d700 1 -- 172.21.15.98:0/1462536826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc4009550 con 0x7f1bc815db30 2023-03-03T08:01:28.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.716+0000 7f1bbdffb700 1 -- 172.21.15.98:0/1462536826 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb800cd40 con 0x7f1bc815db30 2023-03-03T08:01:28.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.716+0000 7f1bce79d700 1 -- 172.21.15.98:0/1462536826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc815cdf0 con 0x7f1bc815db30 2023-03-03T08:01:28.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.718+0000 7f1bbdffb700 1 -- 172.21.15.98:0/1462536826 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f1bb8019030 con 0x7f1bc815db30 2023-03-03T08:01:28.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.719+0000 7f1bbdffb700 1 --2- 172.21.15.98:0/1462536826 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1bac0684a0 0x7f1bac06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.720+0000 7f1bbdffb700 1 -- 172.21.15.98:0/1462536826 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1bb8085e50 con 0x7f1bc815db30 2023-03-03T08:01:28.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.720+0000 7f1bcd79b700 1 --2- 172.21.15.98:0/1462536826 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1bac0684a0 0x7f1bac06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.720+0000 7f1bcd79b700 1 --2- 172.21.15.98:0/1462536826 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1bac0684a0 0x7f1bac06a950 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009840 tx=0x7f1bc4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:28.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.722+0000 7f1be3234b80 1 Processor -- start 2023-03-03T08:01:28.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.722+0000 7f1be3234b80 1 -- start start 2023-03-03T08:01:28.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.723+0000 7f1be3234b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588277aead0 0x5588277aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.723+0000 7f1be3234b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x558827790ff0 0x5588277d57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.723+0000 7f1be3234b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55882778f5c0 0x55882778fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.723+0000 7f1be3234b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588277044f0 con 0x5588277aead0 2023-03-03T08:01:28.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.723+0000 7f1be3234b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x558827679b60 con 0x558827790ff0 2023-03-03T08:01:28.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.723+0000 7f1be3234b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588276e8ad0 con 0x55882778f5c0 2023-03-03T08:01:28.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.723+0000 7f1bcd79b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588277aead0 0x5588277aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.723+0000 7f1bcd79b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588277aead0 0x5588277aeea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60264/0 (socket says 172.21.15.98:60264) 2023-03-03T08:01:28.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.723+0000 7f1bcd79b700 1 -- 172.21.15.98:0/1104337108 learned_addr learned my addr 172.21.15.98:0/1104337108 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:28.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.723+0000 7f1bccf9a700 1 --2- 172.21.15.98:0/1104337108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x558827790ff0 0x5588277d57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.723+0000 7f1bcdf9c700 1 --2- 172.21.15.98:0/1104337108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55882778f5c0 0x55882778fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.724+0000 7f1bccf9a700 1 -- 172.21.15.98:0/1104337108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55882778f5c0 msgr2=0x55882778fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:28.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.724+0000 7f1bccf9a700 1 --2- 172.21.15.98:0/1104337108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55882778f5c0 0x55882778fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.724+0000 7f1bccf9a700 1 -- 172.21.15.98:0/1104337108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588277aead0 msgr2=0x5588277aeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:28.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.724+0000 7f1bccf9a700 1 --2- 172.21.15.98:0/1104337108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588277aead0 0x5588277aeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.724+0000 7f1bccf9a700 1 -- 172.21.15.98:0/1104337108 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588274d6540 con 0x558827790ff0 2023-03-03T08:01:28.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.724+0000 7f1bcd79b700 1 --2- 172.21.15.98:0/1104337108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588277aead0 0x5588277aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:28.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.724+0000 7f1bccf9a700 1 --2- 172.21.15.98:0/1104337108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x558827790ff0 0x5588277d57c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8055ed0 tx=0x7f1bb800cfa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d6777d03239160e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:28.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.724+0000 7f1bb2ffd700 1 -- 172.21.15.98:0/1104337108 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb806d070 con 0x558827790ff0 2023-03-03T08:01:28.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.724+0000 7f1bb2ffd700 1 -- 172.21.15.98:0/1104337108 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb80556c0 con 0x558827790ff0 2023-03-03T08:01:28.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.724+0000 7f1bb2ffd700 1 -- 172.21.15.98:0/1104337108 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb805b070 con 0x558827790ff0 2023-03-03T08:01:28.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.725+0000 7f1be3234b80 1 -- 172.21.15.98:0/1104337108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x558827790ff0 msgr2=0x5588277d57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:28.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.725+0000 7f1be3234b80 1 --2- 172.21.15.98:0/1104337108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x558827790ff0 0x5588277d57c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8055ed0 tx=0x7f1bb800cfa0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.725+0000 7f1be3234b80 1 -- 172.21.15.98:0/1104337108 shutdown_connections 2023-03-03T08:01:28.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.725+0000 7f1be3234b80 1 --2- 172.21.15.98:0/1104337108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588277aead0 0x5588277aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.725+0000 7f1be3234b80 1 --2- 172.21.15.98:0/1104337108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55882778f5c0 0x55882778fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.725+0000 7f1be3234b80 1 --2- 172.21.15.98:0/1104337108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x558827790ff0 0x5588277d57c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.725+0000 7f1be3234b80 1 -- 172.21.15.98:0/1104337108 >> 172.21.15.98:0/1104337108 conn(0x55882778ce10 msgr2=0x55882778d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:28.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.725+0000 7f1be3234b80 1 -- 172.21.15.98:0/1104337108 shutdown_connections 2023-03-03T08:01:28.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.725+0000 7f1be3234b80 1 -- 172.21.15.98:0/1104337108 wait complete. 2023-03-03T08:01:28.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.726+0000 7f1be3234b80 1 -- start start 2023-03-03T08:01:28.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.726+0000 7f1be3234b80 1 Processor -- start 2023-03-03T08:01:28.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.727+0000 7f1be3234b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55882778f5c0 0x5588277afe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.727+0000 7f1be3234b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x558827790ff0 0x5588277d7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.727+0000 7f1be3234b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5588277aead0 0x5588277d8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.727+0000 7f1be3234b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588276e1e40 con 0x55882778f5c0 2023-03-03T08:01:28.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.727+0000 7f1be3234b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588274d7140 con 0x558827790ff0 2023-03-03T08:01:28.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.727+0000 7f1be3234b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588277ad2c0 con 0x5588277aead0 2023-03-03T08:01:28.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.727+0000 7f1be3234b80 10 client.0 _ll_register_callbacks cb 0x5588276801c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T08:01:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.728+0000 7f1bcd79b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55882778f5c0 0x5588277afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.728+0000 7f1bcd79b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55882778f5c0 0x5588277afe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60268/0 (socket says 172.21.15.98:60268) 2023-03-03T08:01:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.728+0000 7f1bcd79b700 1 -- 172.21.15.98:0/1077657349 learned_addr learned my addr 172.21.15.98:0/1077657349 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.728+0000 7f1bcdf9c700 1 --2- 172.21.15.98:0/1077657349 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5588277aead0 0x5588277d8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.728+0000 7f1bccf9a700 1 --2- 172.21.15.98:0/1077657349 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x558827790ff0 0x5588277d7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.728+0000 7f1bcd79b700 1 -- 172.21.15.98:0/1077657349 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5588277aead0 msgr2=0x5588277d8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.728+0000 7f1bcd79b700 1 --2- 172.21.15.98:0/1077657349 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5588277aead0 0x5588277d8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.728+0000 7f1bcd79b700 1 -- 172.21.15.98:0/1077657349 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x558827790ff0 msgr2=0x5588277d7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:28.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.728+0000 7f1bcd79b700 1 --2- 172.21.15.98:0/1077657349 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x558827790ff0 0x5588277d7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.728+0000 7f1bcd79b700 1 -- 172.21.15.98:0/1077657349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc815cdf0 con 0x55882778f5c0 2023-03-03T08:01:28.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.728+0000 7f1bcd79b700 1 --2- 172.21.15.98:0/1077657349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55882778f5c0 0x5588277afe80 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009f50 tx=0x7f1bc40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6d5713ff2a934d2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:28.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.728+0000 7f1ba7fff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-03-03T08:01:28.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.729+0000 7f1ba7fff700 1 -- 172.21.15.98:0/1077657349 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc402b040 con 0x55882778f5c0 2023-03-03T08:01:28.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.729+0000 7f1ba7fff700 1 -- 172.21.15.98:0/1077657349 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1bc4003b00 con 0x55882778f5c0 2023-03-03T08:01:28.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.729+0000 7f1be3234b80 1 -- 172.21.15.98:0/1077657349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558827781ea0 con 0x55882778f5c0 2023-03-03T08:01:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.729+0000 7f1ba7fff700 1 -- 172.21.15.98:0/1077657349 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bc401f070 con 0x55882778f5c0 2023-03-03T08:01:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.729+0000 7f1be3234b80 10 client.7056 fetch_fsmap learned FSMap version 40 2023-03-03T08:01:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.729+0000 7f1be3234b80 1 -- 172.21.15.98:0/1077657349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x558827510b10 con 0x55882778f5c0 2023-03-03T08:01:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.729+0000 7f1ba7fff700 1 -- 172.21.15.98:0/1077657349 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558827781ea0 con 0x55882778f5c0 2023-03-03T08:01:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.729+0000 7f1ba7fff700 1 -- 172.21.15.98:0/1077657349 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 40) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f1bc402e070 con 0x55882778f5c0 2023-03-03T08:01:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.730+0000 7f1be3234b80 10 client.7056 fetch_fsmap finished waiting for FSMap version 40 2023-03-03T08:01:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.730+0000 7f1be3234b80 10 client.7056 resolve_mds: resolved name 'd' to daemon mds.d 2023-03-03T08:01:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.730+0000 7f1be3234b80 20 client.7056 populate_metadata read hostname 'smithi098' 2023-03-03T08:01:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.730+0000 7f1be3234b80 1 --2- 172.21.15.98:0/1077657349 >> [v2:172.21.15.100:6833/341424177,v1:172.21.15.100:6835/341424177] conn(0x5588277e7e90 0x5588277b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.730+0000 7f1be3234b80 4 client.7056 mds_command: new command op to 6909 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-03T08:01:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.730+0000 7f1be3234b80 1 -- 172.21.15.98:0/1077657349 --> [v2:172.21.15.100:6833/341424177,v1:172.21.15.100:6835/341424177] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5588276db020 con 0x5588277e7e90 2023-03-03T08:01:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.730+0000 7f1bccf9a700 1 --2- 172.21.15.98:0/1077657349 >> [v2:172.21.15.100:6833/341424177,v1:172.21.15.100:6835/341424177] conn(0x5588277e7e90 0x5588277b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.731+0000 7f1bccf9a700 1 --2- 172.21.15.98:0/1077657349 >> [v2:172.21.15.100:6833/341424177,v1:172.21.15.100:6835/341424177] conn(0x5588277e7e90 0x5588277b0fe0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=365d894a144b4874 server_cookie=344a0659fea69cf3 in_seq=0 out_seq=0 2023-03-03T08:01:28.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.731+0000 7f1ba7fff700 10 client.7056 ms_handle_connect on v2:172.21.15.100:6833/341424177 2023-03-03T08:01:28.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.732+0000 7f1ba7fff700 1 -- 172.21.15.98:0/1077657349 <== mds.0 v2:172.21.15.100:6833/341424177 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x5588276db020 con 0x5588277e7e90 2023-03-03T08:01:28.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.732+0000 7f1ba7fff700 10 client.7056 handle_command_reply: tid=0 2023-03-03T08:01:28.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.732+0000 7f1be3234b80 1 client.7056 shutdown 2023-03-03T08:01:28.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.732+0000 7f1be3234b80 1 -- 172.21.15.98:0/1077657349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55882778f5c0 msgr2=0x5588277afe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:28.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.732+0000 7f1be3234b80 1 --2- 172.21.15.98:0/1077657349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55882778f5c0 0x5588277afe80 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009f50 tx=0x7f1bc40047c0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.732+0000 7f1be3234b80 1 -- 172.21.15.98:0/1077657349 shutdown_connections 2023-03-03T08:01:28.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.732+0000 7f1be3234b80 1 --2- 172.21.15.98:0/1077657349 >> [v2:172.21.15.100:6833/341424177,v1:172.21.15.100:6835/341424177] conn(0x5588277e7e90 0x5588277b0fe0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.732+0000 7f1be3234b80 1 --2- 172.21.15.98:0/1077657349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55882778f5c0 0x5588277afe80 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.732+0000 7f1be3234b80 1 --2- 172.21.15.98:0/1077657349 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5588277aead0 0x5588277d8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.732+0000 7f1be3234b80 1 --2- 172.21.15.98:0/1077657349 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x558827790ff0 0x5588277d7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.732+0000 7f1ba7fff700 0 client.7056 ms_handle_reset on v2:172.21.15.100:6833/341424177 2023-03-03T08:01:28.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.732+0000 7f1be3234b80 1 -- 172.21.15.98:0/1077657349 >> 172.21.15.98:0/1077657349 conn(0x55882778ce10 msgr2=0x55882778fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:28.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.733+0000 7f1be3234b80 1 -- 172.21.15.98:0/1077657349 shutdown_connections 2023-03-03T08:01:28.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.733+0000 7f1be3234b80 1 -- 172.21.15.98:0/1077657349 wait complete. 2023-03-03T08:01:28.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.733+0000 7f1be3234b80 20 client.7056 trim_cache size 0 max 16384 2023-03-03T08:01:28.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.751+0000 7f1be3234b80 1 Processor -- start 2023-03-03T08:01:28.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.751+0000 7f1be3234b80 1 -- start start 2023-03-03T08:01:28.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1be3234b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55882778f5c0 0x5588277b0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1be3234b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558827790ff0 0x5588277b9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1be3234b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5588277aead0 0x5588277d7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1be3234b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x558827790900 con 0x558827790ff0 2023-03-03T08:01:28.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1be3234b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588277ce880 con 0x5588277aead0 2023-03-03T08:01:28.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1be3234b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55882778d1e0 con 0x55882778f5c0 2023-03-03T08:01:28.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1bccf9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558827790ff0 0x5588277b9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1bccf9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558827790ff0 0x5588277b9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60282/0 (socket says 172.21.15.98:60282) 2023-03-03T08:01:28.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1bccf9a700 1 -- 172.21.15.98:0/1350689391 learned_addr learned my addr 172.21.15.98:0/1350689391 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:28.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1bcd79b700 1 --2- 172.21.15.98:0/1350689391 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55882778f5c0 0x5588277b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1bcdf9c700 1 --2- 172.21.15.98:0/1350689391 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5588277aead0 0x5588277d7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1bccf9a700 1 -- 172.21.15.98:0/1350689391 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55882778f5c0 msgr2=0x5588277b0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:28.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1bccf9a700 1 --2- 172.21.15.98:0/1350689391 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55882778f5c0 0x5588277b0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1bccf9a700 1 -- 172.21.15.98:0/1350689391 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5588277aead0 msgr2=0x5588277d7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:28.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1bccf9a700 1 --2- 172.21.15.98:0/1350689391 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5588277aead0 0x5588277d7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.753+0000 7f1bccf9a700 1 -- 172.21.15.98:0/1350689391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588277ba330 con 0x558827790ff0 2023-03-03T08:01:28.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.754+0000 7f1bccf9a700 1 --2- 172.21.15.98:0/1350689391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558827790ff0 0x5588277b9710 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1bb805dd30 tx=0x7f1bb8056070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a67fee8998605a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:28.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.754+0000 7f1bb37fe700 1 -- 172.21.15.98:0/1350689391 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb806d040 con 0x558827790ff0 2023-03-03T08:01:28.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.754+0000 7f1bb37fe700 1 -- 172.21.15.98:0/1350689391 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb805ddd0 con 0x558827790ff0 2023-03-03T08:01:28.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.754+0000 7f1bb37fe700 1 -- 172.21.15.98:0/1350689391 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb806d1a0 con 0x558827790ff0 2023-03-03T08:01:28.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.754+0000 7f1be3234b80 1 -- 172.21.15.98:0/1350689391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558827790ff0 msgr2=0x5588277b9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:28.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.754+0000 7f1be3234b80 1 --2- 172.21.15.98:0/1350689391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558827790ff0 0x5588277b9710 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1bb805dd30 tx=0x7f1bb8056070 comp rx=0 tx=0).stop 2023-03-03T08:01:28.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.754+0000 7f1be3234b80 1 -- 172.21.15.98:0/1350689391 shutdown_connections 2023-03-03T08:01:28.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.754+0000 7f1be3234b80 1 --2- 172.21.15.98:0/1350689391 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5588277aead0 0x5588277d7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.754+0000 7f1be3234b80 1 --2- 172.21.15.98:0/1350689391 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55882778f5c0 0x5588277b0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.754+0000 7f1be3234b80 1 --2- 172.21.15.98:0/1350689391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558827790ff0 0x5588277b9710 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.754+0000 7f1be3234b80 1 -- 172.21.15.98:0/1350689391 >> 172.21.15.98:0/1350689391 conn(0x55882778ce10 msgr2=0x558827788510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:28.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.755+0000 7f1be3234b80 1 -- 172.21.15.98:0/1350689391 shutdown_connections 2023-03-03T08:01:28.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.755+0000 7f1be3234b80 1 -- 172.21.15.98:0/1350689391 wait complete. 2023-03-03T08:01:28.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.755+0000 7f1be3234b80 1 -- start start 2023-03-03T08:01:28.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.756+0000 7f1be3234b80 1 Processor -- start 2023-03-03T08:01:28.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.757+0000 7f1be3234b80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55882778f5c0 0x55882783c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.757+0000 7f1be3234b80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x558827790ff0 0x5588277af780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.757+0000 7f1be3234b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588277aead0 0x5588277b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.757+0000 7f1be3234b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55882783cb10 con 0x5588277aead0 2023-03-03T08:01:28.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.757+0000 7f1be3234b80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588277b0260 con 0x55882778f5c0 2023-03-03T08:01:28.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.757+0000 7f1be3234b80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55882783cf40 con 0x558827790ff0 2023-03-03T08:01:28.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.757+0000 7f1be3234b80 10 client.0 _ll_register_callbacks cb 0x5588276801c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T08:01:28.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.757+0000 7f1bcdf9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588277aead0 0x5588277b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.757+0000 7f1bcdf9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588277aead0 0x5588277b15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60294/0 (socket says 172.21.15.98:60294) 2023-03-03T08:01:28.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.757+0000 7f1bcdf9c700 1 -- 172.21.15.98:0/392411749 learned_addr learned my addr 172.21.15.98:0/392411749 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:28.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.757+0000 7f1bccf9a700 1 --2- 172.21.15.98:0/392411749 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x558827790ff0 0x5588277af780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.757+0000 7f1bcd79b700 1 --2- 172.21.15.98:0/392411749 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55882778f5c0 0x55882783c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.758+0000 7f1bcdf9c700 1 -- 172.21.15.98:0/392411749 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x558827790ff0 msgr2=0x5588277af780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:28.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.758+0000 7f1bcdf9c700 1 --2- 172.21.15.98:0/392411749 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x558827790ff0 0x5588277af780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.758+0000 7f1bcdf9c700 1 -- 172.21.15.98:0/392411749 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55882778f5c0 msgr2=0x55882783c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:28.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.758+0000 7f1bcdf9c700 1 --2- 172.21.15.98:0/392411749 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55882778f5c0 0x55882783c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.758+0000 7f1bcdf9c700 1 -- 172.21.15.98:0/392411749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588274d6540 con 0x5588277aead0 2023-03-03T08:01:28.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.758+0000 7f1bcdf9c700 1 --2- 172.21.15.98:0/392411749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588277aead0 0x5588277b15d0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0001560 tx=0x7f1bc0007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68cc5ce55e9dfe95 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:28.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.758+0000 7f1ba7fff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-03-03T08:01:28.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.758+0000 7f1ba7fff700 1 -- 172.21.15.98:0/392411749 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0015a00 con 0x5588277aead0 2023-03-03T08:01:28.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.758+0000 7f1ba7fff700 1 -- 172.21.15.98:0/392411749 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1bc0015b60 con 0x5588277aead0 2023-03-03T08:01:28.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.758+0000 7f1be3234b80 1 -- 172.21.15.98:0/392411749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1bb80556c0 con 0x5588277aead0 2023-03-03T08:01:28.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.759+0000 7f1ba7fff700 1 -- 172.21.15.98:0/392411749 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bc0015d20 con 0x5588277aead0 2023-03-03T08:01:28.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.759+0000 7f1be3234b80 10 client.7062 fetch_fsmap learned FSMap version 40 2023-03-03T08:01:28.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.759+0000 7f1be3234b80 1 -- 172.21.15.98:0/392411749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x55882768fa60 con 0x5588277aead0 2023-03-03T08:01:28.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.759+0000 7f1ba7fff700 1 -- 172.21.15.98:0/392411749 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb80556c0 con 0x5588277aead0 2023-03-03T08:01:28.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.759+0000 7f1ba7fff700 1 -- 172.21.15.98:0/392411749 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 40) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f1bc0012050 con 0x5588277aead0 2023-03-03T08:01:28.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.759+0000 7f1be3234b80 10 client.7062 fetch_fsmap finished waiting for FSMap version 40 2023-03-03T08:01:28.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.759+0000 7f1be3234b80 10 client.7062 resolve_mds: resolved name 'd' to daemon mds.d 2023-03-03T08:01:28.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.759+0000 7f1be3234b80 20 client.7062 populate_metadata read hostname 'smithi098' 2023-03-03T08:01:28.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.759+0000 7f1be3234b80 1 --2- 172.21.15.98:0/392411749 >> [v2:172.21.15.100:6833/341424177,v1:172.21.15.100:6835/341424177] conn(0x558827788700 0x558827788ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:28.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.759+0000 7f1be3234b80 4 client.7062 mds_command: new command op to 6909 tid=0[{"prefix": "status"}] 2023-03-03T08:01:28.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.759+0000 7f1be3234b80 1 -- 172.21.15.98:0/392411749 --> [v2:172.21.15.100:6833/341424177,v1:172.21.15.100:6835/341424177] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558827531ca0 con 0x558827788700 2023-03-03T08:01:28.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.759+0000 7f1bcd79b700 1 --2- 172.21.15.98:0/392411749 >> [v2:172.21.15.100:6833/341424177,v1:172.21.15.100:6835/341424177] conn(0x558827788700 0x558827788ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:28.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.760+0000 7f1bcd79b700 1 --2- 172.21.15.98:0/392411749 >> [v2:172.21.15.100:6833/341424177,v1:172.21.15.100:6835/341424177] conn(0x558827788700 0x558827788ad0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9554050c28e0fe2f server_cookie=219414577b374d89 in_seq=0 out_seq=0 2023-03-03T08:01:28.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.760+0000 7f1ba7fff700 10 client.7062 ms_handle_connect on v2:172.21.15.100:6833/341424177 2023-03-03T08:01:28.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.760+0000 7f1ba7fff700 1 -- 172.21.15.98:0/392411749 <== mds.0 v2:172.21.15.100:6833/341424177 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x558827531ca0 con 0x558827788700 2023-03-03T08:01:28.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.760+0000 7f1ba7fff700 10 client.7062 handle_command_reply: tid=0 2023-03-03T08:01:28.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.760+0000 7f1be3234b80 1 client.7062 shutdown 2023-03-03T08:01:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.760+0000 7f1be3234b80 1 -- 172.21.15.98:0/392411749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588277aead0 msgr2=0x5588277b15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.760+0000 7f1be3234b80 1 --2- 172.21.15.98:0/392411749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588277aead0 0x5588277b15d0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0001560 tx=0x7f1bc0007a50 comp rx=0 tx=0).stop 2023-03-03T08:01:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.761+0000 7f1be3234b80 1 -- 172.21.15.98:0/392411749 shutdown_connections 2023-03-03T08:01:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.761+0000 7f1be3234b80 1 --2- 172.21.15.98:0/392411749 >> [v2:172.21.15.100:6833/341424177,v1:172.21.15.100:6835/341424177] conn(0x558827788700 0x558827788ad0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.761+0000 7f1be3234b80 1 --2- 172.21.15.98:0/392411749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588277aead0 0x5588277b15d0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.761+0000 7f1be3234b80 1 --2- 172.21.15.98:0/392411749 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55882778f5c0 0x55882783c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.761+0000 7f1be3234b80 1 --2- 172.21.15.98:0/392411749 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x558827790ff0 0x5588277af780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.761+0000 7f1be3234b80 1 -- 172.21.15.98:0/392411749 >> 172.21.15.98:0/392411749 conn(0x55882778ce10 msgr2=0x5588277ce020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:28.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.761+0000 7f1ba7fff700 0 client.7062 ms_handle_reset on v2:172.21.15.100:6833/341424177 2023-03-03T08:01:28.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.761+0000 7f1be3234b80 1 -- 172.21.15.98:0/392411749 shutdown_connections 2023-03-03T08:01:28.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.761+0000 7f1be3234b80 1 -- 172.21.15.98:0/392411749 wait complete. 2023-03-03T08:01:28.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.761+0000 7f1be3234b80 20 client.7062 trim_cache size 0 max 16384 2023-03-03T08:01:28.769 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:01:28.770 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "fbbd322f-236a-44f8-96c5-be6963846c9b", 2023-03-03T08:01:28.770 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2023-03-03T08:01:28.770 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 6909, 2023-03-03T08:01:28.770 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-03-03T08:01:28.770 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-03-03T08:01:28.770 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-03-03T08:01:28.770 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 7.3780459949999999, 2023-03-03T08:01:28.771 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 40, 2023-03-03T08:01:28.771 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 78, 2023-03-03T08:01:28.771 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 78, 2023-03-03T08:01:28.771 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 14.108782621 2023-03-03T08:01:28.771 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:01:28.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.762+0000 7f1bce79d700 1 -- 172.21.15.98:0/1462536826 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1bac0684a0 msgr2=0x7f1bac06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:28.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.762+0000 7f1bce79d700 1 --2- 172.21.15.98:0/1462536826 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1bac0684a0 0x7f1bac06a950 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009840 tx=0x7f1bc4008030 comp rx=0 tx=0).stop 2023-03-03T08:01:28.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.762+0000 7f1bce79d700 1 -- 172.21.15.98:0/1462536826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc815db30 msgr2=0x7f1bc81f1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:28.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.762+0000 7f1bce79d700 1 --2- 172.21.15.98:0/1462536826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc815db30 0x7f1bc81f1c40 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f1bb800ebe0 tx=0x7f1bb800eef0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.762+0000 7f1bce79d700 1 -- 172.21.15.98:0/1462536826 shutdown_connections 2023-03-03T08:01:28.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.762+0000 7f1bce79d700 1 --2- 172.21.15.98:0/1462536826 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f1bac0684a0 0x7f1bac06a950 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.762+0000 7f1bce79d700 1 --2- 172.21.15.98:0/1462536826 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1bc81609b0 0x7f1bc80d16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.762+0000 7f1bce79d700 1 --2- 172.21.15.98:0/1462536826 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1bc80cf570 0x7f1bc81f1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.762+0000 7f1bce79d700 1 --2- 172.21.15.98:0/1462536826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc815db30 0x7f1bc81f1c40 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:28.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.762+0000 7f1bce79d700 1 -- 172.21.15.98:0/1462536826 >> 172.21.15.98:0/1462536826 conn(0x7f1bc806b880 msgr2=0x7f1bc80b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:28.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.762+0000 7f1bce79d700 1 -- 172.21.15.98:0/1462536826 shutdown_connections 2023-03-03T08:01:28.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:28.762+0000 7f1bce79d700 1 -- 172.21.15.98:0/1462536826 wait complete. 2023-03-03T08:01:28.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T08:01:29.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.036+0000 7f97841ff700 1 Processor -- start 2023-03-03T08:01:29.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.036+0000 7f97841ff700 1 -- start start 2023-03-03T08:01:29.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.037+0000 7f97841ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f977c0cd8d0 0x7f977c0b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:29.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.037+0000 7f97841ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c0cdd70 con 0x7f977c0b6610 2023-03-03T08:01:29.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.037+0000 7f97841ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c0d1a00 con 0x7f977c15cc00 2023-03-03T08:01:29.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.037+0000 7f97841ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f977c0ce040 con 0x7f977c0cd8d0 2023-03-03T08:01:29.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.038+0000 7f978179a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f977c0cd8d0 0x7f977c0b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:29.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.038+0000 7f978179a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f977c0cd8d0 0x7f977c0b60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60310/0 (socket says 172.21.15.98:60310) 2023-03-03T08:01:29.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.038+0000 7f978179a700 1 -- 172.21.15.98:0/630518594 learned_addr learned my addr 172.21.15.98:0/630518594 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:29.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.038+0000 7f978279c700 1 --1- 172.21.15.98:0/630518594 >> v1:172.21.15.98:6789/0 conn(0x7f977c0b6610 0x7f977c0cf4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54106/0 (socket says 172.21.15.98:54106) 2023-03-03T08:01:29.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.038+0000 7f978179a700 1 -- 172.21.15.98:0/630518594 >> v1:172.21.15.100:6789/0 conn(0x7f977c15cc00 legacy=0x7f977c0d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:29.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.038+0000 7f978179a700 1 -- 172.21.15.98:0/630518594 >> v1:172.21.15.98:6789/0 conn(0x7f977c0b6610 legacy=0x7f977c0cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:29.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.038+0000 7f978179a700 1 -- 172.21.15.98:0/630518594 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c0d1ce0 con 0x7f977c0cd8d0 2023-03-03T08:01:29.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.039+0000 7f978179a700 1 --2- 172.21.15.98:0/630518594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f977c0cd8d0 0x7f977c0b60d0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f976c0098f0 tx=0x7f976c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f6627a2d9ee828 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:29.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.039+0000 7f9780f99700 1 -- 172.21.15.98:0/630518594 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c004230 con 0x7f977c0cd8d0 2023-03-03T08:01:29.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.039+0000 7f9780f99700 1 -- 172.21.15.98:0/630518594 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976c004390 con 0x7f977c0cd8d0 2023-03-03T08:01:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.039+0000 7f9780f99700 1 -- 172.21.15.98:0/630518594 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c0036a0 con 0x7f977c0cd8d0 2023-03-03T08:01:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.039+0000 7f97841ff700 1 -- 172.21.15.98:0/630518594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f977c0cd8d0 msgr2=0x7f977c0b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.039+0000 7f97841ff700 1 --2- 172.21.15.98:0/630518594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f977c0cd8d0 0x7f977c0b60d0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f976c0098f0 tx=0x7f976c009c00 comp rx=0 tx=0).stop 2023-03-03T08:01:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.040+0000 7f97841ff700 1 -- 172.21.15.98:0/630518594 shutdown_connections 2023-03-03T08:01:29.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.040+0000 7f97841ff700 1 --2- 172.21.15.98:0/630518594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f977c0cd8d0 0x7f977c0b60d0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:29.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.040+0000 7f97841ff700 1 -- 172.21.15.98:0/630518594 >> 172.21.15.98:0/630518594 conn(0x7f977c06aee0 msgr2=0x7f977c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:29.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.040+0000 7f97841ff700 1 -- 172.21.15.98:0/630518594 shutdown_connections 2023-03-03T08:01:29.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.040+0000 7f97841ff700 1 -- 172.21.15.98:0/630518594 wait complete. 2023-03-03T08:01:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.042+0000 7f97841ff700 1 Processor -- start 2023-03-03T08:01:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.042+0000 7f97841ff700 1 -- start start 2023-03-03T08:01:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.043+0000 7f97841ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f977c0b6610 0x7f977c151440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.043+0000 7f97841ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f977c0cd8d0 0x7f977c1f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.043+0000 7f97841ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f977c15cc00 0x7f977c1f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.043+0000 7f97841ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c1f2ce0 con 0x7f977c0cd8d0 2023-03-03T08:01:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.043+0000 7f97841ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c151d90 con 0x7f977c0b6610 2023-03-03T08:01:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.043+0000 7f97841ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977c0c86b0 con 0x7f977c15cc00 2023-03-03T08:01:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.043+0000 7f978179a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f977c0cd8d0 0x7f977c1f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.043+0000 7f978179a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f977c0cd8d0 0x7f977c1f2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60314/0 (socket says 172.21.15.98:60314) 2023-03-03T08:01:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.043+0000 7f978179a700 1 -- 172.21.15.98:0/3417296603 learned_addr learned my addr 172.21.15.98:0/3417296603 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.043+0000 7f9781f9b700 1 --2- 172.21.15.98:0/3417296603 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f977c0b6610 0x7f977c151440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.044+0000 7f978179a700 1 -- 172.21.15.98:0/3417296603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f977c15cc00 msgr2=0x7f977c1f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.044+0000 7f978279c700 1 --2- 172.21.15.98:0/3417296603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f977c15cc00 0x7f977c1f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.044+0000 7f978179a700 1 --2- 172.21.15.98:0/3417296603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f977c15cc00 0x7f977c1f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.044+0000 7f978179a700 1 -- 172.21.15.98:0/3417296603 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f977c0b6610 msgr2=0x7f977c151440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.044+0000 7f978179a700 1 --2- 172.21.15.98:0/3417296603 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f977c0b6610 0x7f977c151440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.044+0000 7f978179a700 1 -- 172.21.15.98:0/3417296603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976c009550 con 0x7f977c0cd8d0 2023-03-03T08:01:29.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.044+0000 7f978279c700 1 --2- 172.21.15.98:0/3417296603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f977c15cc00 0x7f977c1f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:29.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.044+0000 7f978179a700 1 --2- 172.21.15.98:0/3417296603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f977c0cd8d0 0x7f977c1f2260 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f976c000c00 tx=0x7f976c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:29.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.044+0000 7f97727fc700 1 -- 172.21.15.98:0/3417296603 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c00b660 con 0x7f977c0cd8d0 2023-03-03T08:01:29.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.044+0000 7f97727fc700 1 -- 172.21.15.98:0/3417296603 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976c00b7e0 con 0x7f977c0cd8d0 2023-03-03T08:01:29.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.044+0000 7f97727fc700 1 -- 172.21.15.98:0/3417296603 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c00bb90 con 0x7f977c0cd8d0 2023-03-03T08:01:29.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.044+0000 7f97841ff700 1 -- 172.21.15.98:0/3417296603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c0d22a0 con 0x7f977c0cd8d0 2023-03-03T08:01:29.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.045+0000 7f97841ff700 1 -- 172.21.15.98:0/3417296603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977c0d1fc0 con 0x7f977c0cd8d0 2023-03-03T08:01:29.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.046+0000 7f97841ff700 1 -- 172.21.15.98:0/3417296603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977c0d2ed0 con 0x7f977c0cd8d0 2023-03-03T08:01:29.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.047+0000 7f97727fc700 1 -- 172.21.15.98:0/3417296603 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f976c017020 con 0x7f977c0cd8d0 2023-03-03T08:01:29.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.048+0000 7f97727fc700 1 --2- 172.21.15.98:0/3417296603 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f97600684a0 0x7f976006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:29.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.048+0000 7f97727fc700 1 -- 172.21.15.98:0/3417296603 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f976c087870 con 0x7f977c0cd8d0 2023-03-03T08:01:29.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.049+0000 7f9781f9b700 1 --2- 172.21.15.98:0/3417296603 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f97600684a0 0x7f976006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:29.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.049+0000 7f9781f9b700 1 --2- 172.21.15.98:0/3417296603 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f97600684a0 0x7f976006a950 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f977c15bf60 tx=0x7f9778009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:29.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.052+0000 7f97727fc700 1 -- 172.21.15.98:0/3417296603 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f976c053ed0 con 0x7f977c0cd8d0 2023-03-03T08:01:29.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.288+0000 7f97841ff700 1 -- 172.21.15.98:0/3417296603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f977c0729b0 con 0x7f977c0cd8d0 2023-03-03T08:01:29.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.289+0000 7f97727fc700 1 -- 172.21.15.98:0/3417296603 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v40) v1 ==== 53+0+73 (secure 0 0 0) 0x7f976c00bcf0 con 0x7f977c0cd8d0 2023-03-03T08:01:29.290 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T08:01:29.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.291+0000 7f97841ff700 1 -- 172.21.15.98:0/3417296603 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f97600684a0 msgr2=0x7f976006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.291+0000 7f97841ff700 1 --2- 172.21.15.98:0/3417296603 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f97600684a0 0x7f976006a950 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f977c15bf60 tx=0x7f9778009040 comp rx=0 tx=0).stop 2023-03-03T08:01:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.291+0000 7f97841ff700 1 -- 172.21.15.98:0/3417296603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f977c0cd8d0 msgr2=0x7f977c1f2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.291+0000 7f97841ff700 1 --2- 172.21.15.98:0/3417296603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f977c0cd8d0 0x7f977c1f2260 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f976c000c00 tx=0x7f976c004750 comp rx=0 tx=0).stop 2023-03-03T08:01:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.291+0000 7f97841ff700 1 -- 172.21.15.98:0/3417296603 shutdown_connections 2023-03-03T08:01:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.291+0000 7f97841ff700 1 --2- 172.21.15.98:0/3417296603 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f97600684a0 0x7f976006a950 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.291+0000 7f97841ff700 1 --2- 172.21.15.98:0/3417296603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f977c15cc00 0x7f977c1f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.291+0000 7f97841ff700 1 --2- 172.21.15.98:0/3417296603 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f977c0b6610 0x7f977c151440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.291+0000 7f97841ff700 1 --2- 172.21.15.98:0/3417296603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f977c0cd8d0 0x7f977c1f2260 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.291+0000 7f97841ff700 1 -- 172.21.15.98:0/3417296603 >> 172.21.15.98:0/3417296603 conn(0x7f977c06aee0 msgr2=0x7f977c0b2170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.291+0000 7f97841ff700 1 -- 172.21.15.98:0/3417296603 shutdown_connections 2023-03-03T08:01:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:29.291+0000 7f97841ff700 1 -- 172.21.15.98:0/3417296603 wait complete. 2023-03-03T08:01:29.320 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T08:01:29.320 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T08:01:29.320 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2023-03-03T08:01:29.321 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-03-03T08:01:29.321 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-03-03T08:01:29.321 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T08:01:29.322 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-03-03T08:01:29.322 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T08:01:29.322 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T08:01:29.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:29.323 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2023-03-03T08:01:29.391 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T08:01:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T08:01:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T08:01:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:01:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-03-03T08:01:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:01:29.392 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:01:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:01:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2023-03-03T08:01:29.392 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:01:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:01:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2023-03-03T08:01:29.393 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:01:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2023-03-03T08:01:29.393 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T08:01:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2023-03-03T08:01:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T08:01:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 41866sec preferred_lft 41866sec 2023-03-03T08:01:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2023-03-03T08:01:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:01:29.394 INFO:teuthology.orchestra.run.smithi098.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T08:01:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 46:3f:9e:55:37:22 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:01:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-03-03T08:01:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:01:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::1857:c2ff:fe19:346f/64 scope link 2023-03-03T08:01:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:01:29.395 INFO:teuthology.orchestra.run.smithi098.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:01:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 46:3f:9e:55:37:22 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-03-03T08:01:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::443f:9eff:fe55:3722/64 scope link 2023-03-03T08:01:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:01:29.395 INFO:teuthology.orchestra.run.smithi098.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:01:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether d6:19:97:a6:7c:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-03-03T08:01:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::d419:97ff:fea6:7c83/64 scope link 2023-03-03T08:01:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:01:29.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:29.397 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T08:01:29.495 INFO:teuthology.orchestra.run.smithi098.stdout:1 2023-03-03T08:01:29.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:29.501 DEBUG:teuthology.orchestra.run.smithi098:> route 2023-03-03T08:01:29.573 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2023-03-03T08:01:29.573 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T08:01:29.573 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T08:01:29.574 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T08:01:29.574 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T08:01:29.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:29.575 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-03-03T08:01:29.576 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-03-03T08:01:29.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T08:01:29.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T08:01:29.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T08:01:29.577 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-03-03T08:01:29.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:29.819 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-03-03T08:01:29.888 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-03-03T08:01:29.888 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-03-03T08:01:29.889 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T08:01:29.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:29.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2023-03-03T08:01:29.990 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-03-03T08:01:29.991 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:01:29.991 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-03-03T08:01:30.060 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-03-03T08:01:30.061 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:01:30.061 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-03-03T08:01:30.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T08:01:30.230 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:01:30.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:01:30.328 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:01:30.332 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:01:30.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:30.333 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T08:01:30.402 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-03-03T08:01:30.402 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/) 2023-03-03T08:01:30.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T08:01:30.512 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:01:30.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:01:30.619 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:01:30.623 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:01:30.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:30.625 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T08:01:30.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:2023-03-03T08:01:30.634+0000 7f17077635c0 -1 init, newargv = 0x557d4fcd7f30 newargc=15 2023-03-03T08:01:30.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[139873]: starting ceph client 2023-03-03T08:01:30.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[139873]: starting fuse 2023-03-03T08:01:30.700 INFO:teuthology.orchestra.run.smithi098.stdout:42 2023-03-03T08:01:30.701 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2023-03-03T08:01:30.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T08:01:30.702 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2023-03-03T08:01:30.702 DEBUG:teuthology.orchestra.run.smithi098:> import re 2023-03-03T08:01:30.702 DEBUG:teuthology.orchestra.run.smithi098:> import os 2023-03-03T08:01:30.703 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2023-03-03T08:01:30.703 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:01:30.703 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2023-03-03T08:01:30.704 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-03-03T08:01:30.704 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2023-03-03T08:01:30.704 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-03-03T08:01:30.705 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:01:30.705 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2023-03-03T08:01:30.705 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2023-03-03T08:01:30.706 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2023-03-03T08:01:30.706 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2023-03-03T08:01:30.706 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:01:30.707 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2023-03-03T08:01:30.707 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T08:01:30.707 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T08:01:30.707 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T08:01:30.708 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2023-03-03T08:01:30.708 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2023-03-03T08:01:30.708 DEBUG:teuthology.orchestra.run.smithi098:> return f 2023-03-03T08:01:30.709 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T08:01:30.709 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:01:30.709 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.0")) 2023-03-03T08:01:30.710 DEBUG:teuthology.orchestra.run.smithi098:> ' 2023-03-03T08:01:30.928 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.0.139873.asok 2023-03-03T08:01:31.106 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.139873.asok 2023-03-03T08:01:31.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:31.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.139873.asok status 2023-03-03T08:01:31.375 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:01:31.376 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2023-03-03T08:01:31.376 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T08:01:31.376 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T08:01:31.376 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "0", 2023-03-03T08:01:31.376 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2023-03-03T08:01:31.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-03-03T08:01:31.377 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "139873", 2023-03-03T08:01:31.377 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2023-03-03T08:01:31.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:31.377 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2023-03-03T08:01:31.377 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2023-03-03T08:01:31.377 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 7080, 2023-03-03T08:01:31.377 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2023-03-03T08:01:31.377 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2023-03-03T08:01:31.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2023-03-03T08:01:31.379 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 7080 2023-03-03T08:01:31.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:31.379 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T08:01:31.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T08:01:31.379 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:01:31.380 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2974927912 2023-03-03T08:01:31.380 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:31.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:31.380 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T08:01:31.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T08:01:31.380 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:01:31.380 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2974927912 2023-03-03T08:01:31.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:31.381 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.7080 192.168.0.1:0/2974927912", 2023-03-03T08:01:31.381 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/2974927912", 2023-03-03T08:01:31.381 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2023-03-03T08:01:31.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 40, 2023-03-03T08:01:31.381 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 78, 2023-03-03T08:01:31.381 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2023-03-03T08:01:31.382 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2023-03-03T08:01:31.382 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2023-03-03T08:01:31.382 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:01:31.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:31.396 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2023-03-03T08:01:31.396 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.0 2023-03-03T08:01:31.465 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2023-03-03T08:01:31.466 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-03-03T08:01:31.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:31.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-03-03T08:01:31.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T08:01:31.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.817+0000 7f46cb59e700 1 Processor -- start 2023-03-03T08:01:31.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.817+0000 7f46cb59e700 1 -- start start 2023-03-03T08:01:31.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.819+0000 7f46cb59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f46cc146360 0x7f46cc160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:31.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.819+0000 7f46cb59e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f46cc0b4ab0 0x7f46cc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:31.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.819+0000 7f46cb59e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46cc1614c0 con 0x7f46cc0731e0 2023-03-03T08:01:31.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.819+0000 7f46cb59e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46cc0b5440 con 0x7f46cc146360 2023-03-03T08:01:31.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.819+0000 7f46cb59e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46cc0d10f0 con 0x7f46cc0b4ab0 2023-03-03T08:01:31.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.819+0000 7f46ca59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f46cc146360 0x7f46cc160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:31.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.819+0000 7f46ca59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f46cc146360 0x7f46cc160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60324/0 (socket says 172.21.15.98:60324) 2023-03-03T08:01:31.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.819+0000 7f46ca59c700 1 -- 172.21.15.98:0/417991530 learned_addr learned my addr 172.21.15.98:0/417991530 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:31.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.819+0000 7f46cad9d700 1 --1- 172.21.15.98:0/417991530 >> v1:172.21.15.98:6789/0 conn(0x7f46cc0731e0 0x7f46cc073600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54112/0 (socket says 172.21.15.98:54112) 2023-03-03T08:01:31.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.819+0000 7f46c9d9b700 1 --2- 172.21.15.98:0/417991530 >> v2:172.21.15.100:3300/0 conn(0x7f46cc0b4ab0 0x7f46cc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:31.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.820+0000 7f46ca59c700 1 -- 172.21.15.98:0/417991530 >> v2:172.21.15.100:3300/0 conn(0x7f46cc0b4ab0 msgr2=0x7f46cc0b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:31.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.820+0000 7f46ca59c700 1 --2- 172.21.15.98:0/417991530 >> v2:172.21.15.100:3300/0 conn(0x7f46cc0b4ab0 0x7f46cc0b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:31.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.820+0000 7f46ca59c700 1 -- 172.21.15.98:0/417991530 >> v1:172.21.15.98:6789/0 conn(0x7f46cc0731e0 legacy=0x7f46cc073600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:31.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.820+0000 7f46ca59c700 1 -- 172.21.15.98:0/417991530 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc0cdf20 con 0x7f46cc146360 2023-03-03T08:01:31.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.820+0000 7f46ca59c700 1 --2- 172.21.15.98:0/417991530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46cc146360 0x7f46cc160f80 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f46cc06bd20 tx=0x7f46bc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616bede8e3f26ae1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:31.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.820+0000 7f46c959a700 1 -- 172.21.15.98:0/417991530 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc011830 con 0x7f46cc146360 2023-03-03T08:01:31.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.820+0000 7f46c959a700 1 -- 172.21.15.98:0/417991530 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46bc011990 con 0x7f46cc146360 2023-03-03T08:01:31.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.820+0000 7f46c959a700 1 -- 172.21.15.98:0/417991530 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc011d20 con 0x7f46cc146360 2023-03-03T08:01:31.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.821+0000 7f46cb59e700 1 -- 172.21.15.98:0/417991530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46cc146360 msgr2=0x7f46cc160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:31.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.821+0000 7f46cb59e700 1 --2- 172.21.15.98:0/417991530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46cc146360 0x7f46cc160f80 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f46cc06bd20 tx=0x7f46bc009bd0 comp rx=0 tx=0).stop 2023-03-03T08:01:31.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.821+0000 7f46cb59e700 1 -- 172.21.15.98:0/417991530 shutdown_connections 2023-03-03T08:01:31.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.821+0000 7f46cb59e700 1 --2- 172.21.15.98:0/417991530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46cc146360 0x7f46cc160f80 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:31.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.821+0000 7f46cb59e700 1 --2- 172.21.15.98:0/417991530 >> v2:172.21.15.100:3300/0 conn(0x7f46cc0b4ab0 0x7f46cc0b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:31.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.821+0000 7f46cb59e700 1 -- 172.21.15.98:0/417991530 >> 172.21.15.98:0/417991530 conn(0x7f46cc06b8e0 msgr2=0x7f46cc0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:31.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.821+0000 7f46cb59e700 1 -- 172.21.15.98:0/417991530 shutdown_connections 2023-03-03T08:01:31.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.821+0000 7f46cb59e700 1 -- 172.21.15.98:0/417991530 wait complete. 2023-03-03T08:01:31.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.823+0000 7f46cb59e700 1 Processor -- start 2023-03-03T08:01:31.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.823+0000 7f46cb59e700 1 -- start start 2023-03-03T08:01:31.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.824+0000 7f46cb59e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46cc0731e0 0x7f46cc0cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:31.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.824+0000 7f46cb59e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46cc0b4ab0 0x7f46cc1f3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:31.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.824+0000 7f46cb59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46cc146360 0x7f46cc1f3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:31.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.824+0000 7f46cb59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc1f2880 con 0x7f46cc146360 2023-03-03T08:01:31.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.824+0000 7f46cb59e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0ccf10 con 0x7f46cc0731e0 2023-03-03T08:01:31.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.824+0000 7f46cb59e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0cc950 con 0x7f46cc0b4ab0 2023-03-03T08:01:31.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.825+0000 7f46cad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46cc146360 0x7f46cc1f3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:31.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.825+0000 7f46cad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46cc146360 0x7f46cc1f3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60326/0 (socket says 172.21.15.98:60326) 2023-03-03T08:01:31.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.825+0000 7f46cad9d700 1 -- 172.21.15.98:0/1080576047 learned_addr learned my addr 172.21.15.98:0/1080576047 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:31.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.825+0000 7f46ca59c700 1 --2- 172.21.15.98:0/1080576047 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46cc0731e0 0x7f46cc0cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:31.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.825+0000 7f46c9d9b700 1 --2- 172.21.15.98:0/1080576047 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46cc0b4ab0 0x7f46cc1f3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:31.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.825+0000 7f46cad9d700 1 -- 172.21.15.98:0/1080576047 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46cc0b4ab0 msgr2=0x7f46cc1f3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:31.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.825+0000 7f46cad9d700 1 --2- 172.21.15.98:0/1080576047 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46cc0b4ab0 0x7f46cc1f3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:31.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.825+0000 7f46cad9d700 1 -- 172.21.15.98:0/1080576047 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46cc0731e0 msgr2=0x7f46cc0cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:31.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.825+0000 7f46cad9d700 1 --2- 172.21.15.98:0/1080576047 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46cc0731e0 0x7f46cc0cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:31.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.825+0000 7f46cad9d700 1 -- 172.21.15.98:0/1080576047 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc009550 con 0x7f46cc146360 2023-03-03T08:01:31.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.826+0000 7f46cad9d700 1 --2- 172.21.15.98:0/1080576047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46cc146360 0x7f46cc1f3de0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f46c400cbc0 tx=0x7f46c400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:31.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.826+0000 7f46baffd700 1 -- 172.21.15.98:0/1080576047 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c400a9e0 con 0x7f46cc146360 2023-03-03T08:01:31.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.826+0000 7f46baffd700 1 -- 172.21.15.98:0/1080576047 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c400ab40 con 0x7f46cc146360 2023-03-03T08:01:31.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.826+0000 7f46baffd700 1 -- 172.21.15.98:0/1080576047 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4007070 con 0x7f46cc146360 2023-03-03T08:01:31.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.826+0000 7f46cb59e700 1 -- 172.21.15.98:0/1080576047 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46cc0cd960 con 0x7f46cc146360 2023-03-03T08:01:31.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.826+0000 7f46cb59e700 1 -- 172.21.15.98:0/1080576047 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46cc0cd680 con 0x7f46cc146360 2023-03-03T08:01:31.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.827+0000 7f46cb59e700 1 -- 172.21.15.98:0/1080576047 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46cc0cdc40 con 0x7f46cc146360 2023-03-03T08:01:31.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.828+0000 7f46baffd700 1 -- 172.21.15.98:0/1080576047 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f46c4019030 con 0x7f46cc146360 2023-03-03T08:01:31.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.830+0000 7f46baffd700 1 --2- 172.21.15.98:0/1080576047 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46ac0684a0 0x7f46ac06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:31.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.830+0000 7f46baffd700 1 -- 172.21.15.98:0/1080576047 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f46c40858e0 con 0x7f46cc146360 2023-03-03T08:01:31.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.830+0000 7f46ca59c700 1 --2- 172.21.15.98:0/1080576047 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46ac0684a0 0x7f46ac06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:31.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.831+0000 7f46ca59c700 1 --2- 172.21.15.98:0/1080576047 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46ac0684a0 0x7f46ac06a950 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009c10 tx=0x7f46bc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:31.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:31.834+0000 7f46baffd700 1 -- 172.21.15.98:0/1080576047 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f46c404b5e0 con 0x7f46cc146360 2023-03-03T08:01:32.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:32.067+0000 7f46cb59e700 1 -- 172.21.15.98:0/1080576047 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f46cc1614c0 con 0x7f46cc146360 2023-03-03T08:01:32.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:32.068+0000 7f46baffd700 1 -- 172.21.15.98:0/1080576047 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v40) v1 ==== 53+0+73 (secure 0 0 0) 0x7f46cc1614c0 con 0x7f46cc146360 2023-03-03T08:01:32.070 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T08:01:32.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:32.070+0000 7f46cb59e700 1 -- 172.21.15.98:0/1080576047 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46ac0684a0 msgr2=0x7f46ac06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:32.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:32.070+0000 7f46cb59e700 1 --2- 172.21.15.98:0/1080576047 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46ac0684a0 0x7f46ac06a950 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009c10 tx=0x7f46bc008030 comp rx=0 tx=0).stop 2023-03-03T08:01:32.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:32.071+0000 7f46cb59e700 1 -- 172.21.15.98:0/1080576047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46cc146360 msgr2=0x7f46cc1f3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:32.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:32.071+0000 7f46cb59e700 1 --2- 172.21.15.98:0/1080576047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46cc146360 0x7f46cc1f3de0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f46c400cbc0 tx=0x7f46c400ced0 comp rx=0 tx=0).stop 2023-03-03T08:01:32.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:32.071+0000 7f46cb59e700 1 -- 172.21.15.98:0/1080576047 shutdown_connections 2023-03-03T08:01:32.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:32.071+0000 7f46cb59e700 1 --2- 172.21.15.98:0/1080576047 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f46ac0684a0 0x7f46ac06a950 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:32.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:32.071+0000 7f46cb59e700 1 --2- 172.21.15.98:0/1080576047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46cc146360 0x7f46cc1f3de0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:32.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:32.071+0000 7f46cb59e700 1 --2- 172.21.15.98:0/1080576047 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46cc0731e0 0x7f46cc0cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:32.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:32.071+0000 7f46cb59e700 1 --2- 172.21.15.98:0/1080576047 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46cc0b4ab0 0x7f46cc1f3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:32.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:32.071+0000 7f46cb59e700 1 -- 172.21.15.98:0/1080576047 >> 172.21.15.98:0/1080576047 conn(0x7f46cc06b8e0 msgr2=0x7f46cc073600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:32.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:32.071+0000 7f46cb59e700 1 -- 172.21.15.98:0/1080576047 shutdown_connections 2023-03-03T08:01:32.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:32.071+0000 7f46cb59e700 1 -- 172.21.15.98:0/1080576047 wait complete. 2023-03-03T08:01:32.099 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T08:01:32.100 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T08:01:32.100 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2023-03-03T08:01:32.101 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-03-03T08:01:32.101 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-03-03T08:01:32.101 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T08:01:32.101 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-03-03T08:01:32.102 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T08:01:32.102 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-03-03T08:01:32.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:32.103 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2023-03-03T08:01:32.178 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T08:01:32.179 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T08:01:32.179 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T08:01:32.179 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:01:32.179 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2023-03-03T08:01:32.179 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:01:32.179 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:01:32.180 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:01:32.180 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2023-03-03T08:01:32.180 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:01:32.180 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:01:32.180 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2023-03-03T08:01:32.180 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:01:32.180 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:01:32.181 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T08:01:32.181 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:01:32.181 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T08:01:32.181 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 41868sec preferred_lft 41868sec 2023-03-03T08:01:32.182 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2023-03-03T08:01:32.182 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:01:32.182 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T08:01:32.182 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ee:78:63:4a:ea:32 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:01:32.182 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-03-03T08:01:32.182 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:01:32.182 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::9406:dcff:fe40:a4a4/64 scope link 2023-03-03T08:01:32.182 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:01:32.183 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:01:32.183 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:5e:bd:c1:84:4d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-03-03T08:01:32.183 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f45e:bdff:fec1:844d/64 scope link 2023-03-03T08:01:32.183 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:01:32.183 INFO:teuthology.orchestra.run.smithi100.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:01:32.183 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ee:78:63:4a:ea:32 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-03-03T08:01:32.183 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec78:63ff:fe4a:ea32/64 scope link 2023-03-03T08:01:32.183 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:01:32.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:32.185 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T08:01:32.283 INFO:teuthology.orchestra.run.smithi100.stdout:1 2023-03-03T08:01:32.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:32.289 DEBUG:teuthology.orchestra.run.smithi100:> route 2023-03-03T08:01:32.361 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2023-03-03T08:01:32.361 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T08:01:32.361 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T08:01:32.361 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T08:01:32.362 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T08:01:32.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:32.363 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2023-03-03T08:01:32.363 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2023-03-03T08:01:32.364 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T08:01:32.364 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T08:01:32.364 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T08:01:32.365 DEBUG:teuthology.orchestra.run.smithi100:> ') 2023-03-03T08:01:32.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:32.607 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2023-03-03T08:01:32.675 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-03-03T08:01:32.675 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-03-03T08:01:32.676 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-03-03T08:01:32.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:32.677 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2023-03-03T08:01:32.779 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-03-03T08:01:32.780 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:01:32.780 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-03-03T08:01:32.849 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-03-03T08:01:32.849 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:01:32.850 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-03-03T08:01:32.918 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T08:01:33.020 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:01:33.021 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:01:33.117 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:01:33.121 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:01:33.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:33.123 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T08:01:33.191 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-03-03T08:01:33.192 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_mountpoint=/) 2023-03-03T08:01:33.198 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T08:01:33.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:01:33.236 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:01:33.305 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:01:33.306 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:01:33.408 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:01:33.413 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:01:33.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:33.416 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T08:01:33.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2023-03-03T08:01:33.422+0000 7f28d21685c0 -1 init, newargv = 0x55f0a56c5a50 newargc=15 2023-03-03T08:01:33.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[119758]: starting ceph client 2023-03-03T08:01:33.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[119758]: starting fuse 2023-03-03T08:01:33.494 INFO:teuthology.orchestra.run.smithi100.stdout:42 2023-03-03T08:01:33.495 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2023-03-03T08:01:33.495 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T08:01:33.496 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2023-03-03T08:01:33.496 DEBUG:teuthology.orchestra.run.smithi100:> import re 2023-03-03T08:01:33.496 DEBUG:teuthology.orchestra.run.smithi100:> import os 2023-03-03T08:01:33.497 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2023-03-03T08:01:33.497 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:01:33.497 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2023-03-03T08:01:33.498 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-03-03T08:01:33.498 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2023-03-03T08:01:33.498 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-03-03T08:01:33.498 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:01:33.499 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2023-03-03T08:01:33.499 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2023-03-03T08:01:33.499 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2023-03-03T08:01:33.500 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2023-03-03T08:01:33.500 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:01:33.500 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2023-03-03T08:01:33.500 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T08:01:33.501 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T08:01:33.501 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T08:01:33.501 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2023-03-03T08:01:33.502 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2023-03-03T08:01:33.502 DEBUG:teuthology.orchestra.run.smithi100:> return f 2023-03-03T08:01:33.502 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T08:01:33.503 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:01:33.503 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.1")) 2023-03-03T08:01:33.503 DEBUG:teuthology.orchestra.run.smithi100:> ' 2023-03-03T08:01:33.721 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.1.119758.asok 2023-03-03T08:01:33.899 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.119758.asok 2023-03-03T08:01:33.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:33.900 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.119758.asok status 2023-03-03T08:01:34.168 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2023-03-03T08:01:34.169 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2023-03-03T08:01:34.169 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T08:01:34.169 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T08:01:34.169 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "1", 2023-03-03T08:01:34.169 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2023-03-03T08:01:34.169 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-03-03T08:01:34.170 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "119758", 2023-03-03T08:01:34.170 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2023-03-03T08:01:34.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:01:34.170 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2023-03-03T08:01:34.170 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2023-03-03T08:01:34.170 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 6161, 2023-03-03T08:01:34.170 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2023-03-03T08:01:34.170 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2023-03-03T08:01:34.171 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2023-03-03T08:01:34.171 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 6161 2023-03-03T08:01:34.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:01:34.171 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T08:01:34.171 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T08:01:34.171 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:01:34.171 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2681533911 2023-03-03T08:01:34.171 INFO:teuthology.orchestra.run.smithi100.stdout: } 2023-03-03T08:01:34.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:01:34.172 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T08:01:34.172 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T08:01:34.172 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:01:34.172 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2681533911 2023-03-03T08:01:34.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:01:34.172 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.6161 192.168.0.1:0/2681533911", 2023-03-03T08:01:34.173 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.1:0/2681533911", 2023-03-03T08:01:34.173 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2023-03-03T08:01:34.173 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 40, 2023-03-03T08:01:34.173 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 78, 2023-03-03T08:01:34.173 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2023-03-03T08:01:34.173 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2023-03-03T08:01:34.173 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2023-03-03T08:01:34.173 INFO:teuthology.orchestra.run.smithi100.stdout:} 2023-03-03T08:01:34.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:34.190 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2023-03-03T08:01:34.190 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2023-03-03T08:01:34.259 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2023-03-03T08:01:34.259 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-03-03T08:01:34.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:34.260 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-03-03T08:01:34.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-03-03T08:01:34.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.620+0000 7fe8d41ff700 1 Processor -- start 2023-03-03T08:01:34.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.620+0000 7fe8d41ff700 1 -- start start 2023-03-03T08:01:34.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.621+0000 7fe8d41ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe8cc0b8880 0x7fe8cc0cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:34.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.621+0000 7fe8d41ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe8cc0b5720 0x7fe8cc15bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:34.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.621+0000 7fe8d41ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8cc0d0ac0 con 0x7fe8cc0b4ba0 2023-03-03T08:01:34.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.621+0000 7fe8d41ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8cc0cff70 con 0x7fe8cc0b8880 2023-03-03T08:01:34.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.621+0000 7fe8d41ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8cc0d0250 con 0x7fe8cc0b5720 2023-03-03T08:01:34.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.621+0000 7fe8d1f9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe8cc0b8880 0x7fe8cc0cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:34.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.622+0000 7fe8d1f9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe8cc0b8880 0x7fe8cc0cd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60334/0 (socket says 172.21.15.98:60334) 2023-03-03T08:01:34.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.622+0000 7fe8d1f9b700 1 -- 172.21.15.98:0/3585943387 learned_addr learned my addr 172.21.15.98:0/3585943387 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:34.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.622+0000 7fe8d279c700 1 --2- 172.21.15.98:0/3585943387 >> v2:172.21.15.100:3300/0 conn(0x7fe8cc0b5720 0x7fe8cc15bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:34.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.622+0000 7fe8d1f9b700 1 -- 172.21.15.98:0/3585943387 >> v2:172.21.15.100:3300/0 conn(0x7fe8cc0b5720 msgr2=0x7fe8cc15bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:34.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.622+0000 7fe8d1f9b700 1 --2- 172.21.15.98:0/3585943387 >> v2:172.21.15.100:3300/0 conn(0x7fe8cc0b5720 0x7fe8cc15bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:34.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.622+0000 7fe8d1f9b700 1 -- 172.21.15.98:0/3585943387 >> v1:172.21.15.100:6789/0 conn(0x7fe8cc0b4ba0 legacy=0x7fe8cc0b5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:34.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.622+0000 7fe8d1f9b700 1 -- 172.21.15.98:0/3585943387 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc0d0530 con 0x7fe8cc0b8880 2023-03-03T08:01:34.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.622+0000 7fe8d1f9b700 1 --2- 172.21.15.98:0/3585943387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8cc0b8880 0x7fe8cc0cd770 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc0cef50 tx=0x7fe8c8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=720b6d7fd057e43a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:34.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.623+0000 7fe8d0f99700 1 -- 172.21.15.98:0/3585943387 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8011830 con 0x7fe8cc0b8880 2023-03-03T08:01:34.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.623+0000 7fe8d0f99700 1 -- 172.21.15.98:0/3585943387 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c8011990 con 0x7fe8cc0b8880 2023-03-03T08:01:34.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.623+0000 7fe8d0f99700 1 -- 172.21.15.98:0/3585943387 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8011d20 con 0x7fe8cc0b8880 2023-03-03T08:01:34.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.623+0000 7fe8d41ff700 1 -- 172.21.15.98:0/3585943387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8cc0b8880 msgr2=0x7fe8cc0cd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:34.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.623+0000 7fe8d41ff700 1 --2- 172.21.15.98:0/3585943387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8cc0b8880 0x7fe8cc0cd770 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc0cef50 tx=0x7fe8c8009bd0 comp rx=0 tx=0).stop 2023-03-03T08:01:34.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.623+0000 7fe8d41ff700 1 -- 172.21.15.98:0/3585943387 shutdown_connections 2023-03-03T08:01:34.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.623+0000 7fe8d41ff700 1 --2- 172.21.15.98:0/3585943387 >> v2:172.21.15.100:3300/0 conn(0x7fe8cc0b5720 0x7fe8cc15bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:34.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.623+0000 7fe8d41ff700 1 --2- 172.21.15.98:0/3585943387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8cc0b8880 0x7fe8cc0cd770 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:34.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.623+0000 7fe8d41ff700 1 -- 172.21.15.98:0/3585943387 >> 172.21.15.98:0/3585943387 conn(0x7fe8cc06ad80 msgr2=0x7fe8cc06b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:34.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.623+0000 7fe8d41ff700 1 -- 172.21.15.98:0/3585943387 shutdown_connections 2023-03-03T08:01:34.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.624+0000 7fe8d41ff700 1 -- 172.21.15.98:0/3585943387 wait complete. 2023-03-03T08:01:34.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.625+0000 7fe8d41ff700 1 Processor -- start 2023-03-03T08:01:34.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.626+0000 7fe8d41ff700 1 -- start start 2023-03-03T08:01:34.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d41ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8cc0b4ba0 0x7fe8cc0c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:34.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d41ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8cc0b5720 0x7fe8cc14d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:34.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d41ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8cc0b8880 0x7fe8cc0cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d41ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc14aaf0 con 0x7fe8cc0b4ba0 2023-03-03T08:01:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d41ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc14d770 con 0x7fe8cc0b5720 2023-03-03T08:01:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d41ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc149ee0 con 0x7fe8cc0b8880 2023-03-03T08:01:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d1f9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8cc0b4ba0 0x7fe8cc0c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d1f9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8cc0b4ba0 0x7fe8cc0c9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60344/0 (socket says 172.21.15.98:60344) 2023-03-03T08:01:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d1f9b700 1 -- 172.21.15.98:0/610210518 learned_addr learned my addr 172.21.15.98:0/610210518 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:34.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d179a700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8cc0b5720 0x7fe8cc14d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:34.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d279c700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8cc0b8880 0x7fe8cc0cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:34.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d1f9b700 1 -- 172.21.15.98:0/610210518 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8cc0b8880 msgr2=0x7fe8cc0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d1f9b700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8cc0b8880 0x7fe8cc0cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d1f9b700 1 -- 172.21.15.98:0/610210518 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8cc0b5720 msgr2=0x7fe8cc14d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d1f9b700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8cc0b5720 0x7fe8cc14d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d1f9b700 1 -- 172.21.15.98:0/610210518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c8009550 con 0x7fe8cc0b4ba0 2023-03-03T08:01:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d279c700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8cc0b8880 0x7fe8cc0cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.627+0000 7fe8d179a700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8cc0b5720 0x7fe8cc14d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:34.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.628+0000 7fe8d1f9b700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8cc0b4ba0 0x7fe8cc0c9260 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8000c00 tx=0x7fe8c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:34.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.628+0000 7fe8c27fc700 1 -- 172.21.15.98:0/610210518 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8004570 con 0x7fe8cc0b4ba0 2023-03-03T08:01:34.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.628+0000 7fe8c27fc700 1 -- 172.21.15.98:0/610210518 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c801f070 con 0x7fe8cc0b4ba0 2023-03-03T08:01:34.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.628+0000 7fe8c27fc700 1 -- 172.21.15.98:0/610210518 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c80038e0 con 0x7fe8cc0b4ba0 2023-03-03T08:01:34.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.628+0000 7fe8d41ff700 1 -- 172.21.15.98:0/610210518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8cc0b8c50 con 0x7fe8cc0b4ba0 2023-03-03T08:01:34.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.628+0000 7fe8d41ff700 1 -- 172.21.15.98:0/610210518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8cc0d0cf0 con 0x7fe8cc0b4ba0 2023-03-03T08:01:34.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.629+0000 7fe8d41ff700 1 -- 172.21.15.98:0/610210518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8cc0cde70 con 0x7fe8cc0b4ba0 2023-03-03T08:01:34.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.630+0000 7fe8c27fc700 1 -- 172.21.15.98:0/610210518 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fe8c802a020 con 0x7fe8cc0b4ba0 2023-03-03T08:01:34.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.632+0000 7fe8c27fc700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8b00684a0 0x7fe8b006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:34.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.632+0000 7fe8c27fc700 1 -- 172.21.15.98:0/610210518 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe8c8087560 con 0x7fe8cc0b4ba0 2023-03-03T08:01:34.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.632+0000 7fe8d179a700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8b00684a0 0x7fe8b006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:34.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.633+0000 7fe8d179a700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8b00684a0 0x7fe8b006a950 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc0d2060 tx=0x7fe8bc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.636+0000 7fe8c27fc700 1 -- 172.21.15.98:0/610210518 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe8c8053c20 con 0x7fe8cc0b4ba0 2023-03-03T08:01:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:34.876+0000 7fe8d41ff700 1 -- 172.21.15.98:0/610210518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe8cc04e6e0 con 0x7fe8cc0b4ba0 2023-03-03T08:01:35.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.531+0000 7fe8c27fc700 1 -- 172.21.15.98:0/610210518 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v41) v1 ==== 136+0+0 (secure 0 0 0) 0x7fe8c8057240 con 0x7fe8cc0b4ba0 2023-03-03T08:01:35.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.534+0000 7fe8d41ff700 1 -- 172.21.15.98:0/610210518 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8b00684a0 msgr2=0x7fe8b006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:35.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.534+0000 7fe8d41ff700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8b00684a0 0x7fe8b006a950 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc0d2060 tx=0x7fe8bc009040 comp rx=0 tx=0).stop 2023-03-03T08:01:35.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.534+0000 7fe8d41ff700 1 -- 172.21.15.98:0/610210518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8cc0b4ba0 msgr2=0x7fe8cc0c9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:35.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.534+0000 7fe8d41ff700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8cc0b4ba0 0x7fe8cc0c9260 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8000c00 tx=0x7fe8c8004750 comp rx=0 tx=0).stop 2023-03-03T08:01:35.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.534+0000 7fe8d41ff700 1 -- 172.21.15.98:0/610210518 shutdown_connections 2023-03-03T08:01:35.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.534+0000 7fe8d41ff700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe8b00684a0 0x7fe8b006a950 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.534+0000 7fe8d41ff700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8cc0b4ba0 0x7fe8cc0c9260 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.534+0000 7fe8d41ff700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8cc0b8880 0x7fe8cc0cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.534+0000 7fe8d41ff700 1 --2- 172.21.15.98:0/610210518 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8cc0b5720 0x7fe8cc14d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.534+0000 7fe8d41ff700 1 -- 172.21.15.98:0/610210518 >> 172.21.15.98:0/610210518 conn(0x7fe8cc06ad80 msgr2=0x7fe8cc0d1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.534+0000 7fe8d41ff700 1 -- 172.21.15.98:0/610210518 shutdown_connections 2023-03-03T08:01:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.534+0000 7fe8d41ff700 1 -- 172.21.15.98:0/610210518 wait complete. 2023-03-03T08:01:35.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-03-03T08:01:35.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.813+0000 7fd9ed5ff700 1 Processor -- start 2023-03-03T08:01:35.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.814+0000 7fd9ed5ff700 1 -- start start 2023-03-03T08:01:35.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.815+0000 7fd9ed5ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd9e815ca70 0x7fd9e80cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.815+0000 7fd9ed5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9e80d0270 0x7fd9e80d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.815+0000 7fd9ed5ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e80cf980 con 0x7fd9e806b4b0 2023-03-03T08:01:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.815+0000 7fd9ed5ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e80cd5b0 con 0x7fd9e815ca70 2023-03-03T08:01:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.815+0000 7fd9ed5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e80d1800 con 0x7fd9e80d0270 2023-03-03T08:01:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.815+0000 7fd9e6ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd9e815ca70 0x7fd9e80cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.815+0000 7fd9e6ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd9e815ca70 0x7fd9e80cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49552/0 (socket says 172.21.15.98:49552) 2023-03-03T08:01:35.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.815+0000 7fd9e6ffd700 1 -- 172.21.15.98:0/1063840198 learned_addr learned my addr 172.21.15.98:0/1063840198 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:35.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.816+0000 7fd9e77fe700 1 --2- 172.21.15.98:0/1063840198 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9e80d0270 0x7fd9e80d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:35.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.816+0000 7fd9e6ffd700 1 -- 172.21.15.98:0/1063840198 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9e80d0270 msgr2=0x7fd9e80d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:35.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.816+0000 7fd9e6ffd700 1 --2- 172.21.15.98:0/1063840198 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9e80d0270 0x7fd9e80d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:35.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.816+0000 7fd9e6ffd700 1 -- 172.21.15.98:0/1063840198 >> v1:172.21.15.100:6789/0 conn(0x7fd9e806b4b0 legacy=0x7fd9e80cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:35.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.816+0000 7fd9e6ffd700 1 -- 172.21.15.98:0/1063840198 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e80cd900 con 0x7fd9e815ca70 2023-03-03T08:01:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.816+0000 7fd9e6ffd700 1 --2- 172.21.15.98:0/1063840198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9e815ca70 0x7fd9e80cf440 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fd9d80098f0 tx=0x7fd9d8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=745cc046193525d6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.817+0000 7fd9e5ffb700 1 -- 172.21.15.98:0/1063840198 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8011830 con 0x7fd9e815ca70 2023-03-03T08:01:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.817+0000 7fd9e5ffb700 1 -- 172.21.15.98:0/1063840198 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d8011990 con 0x7fd9e815ca70 2023-03-03T08:01:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.817+0000 7fd9e5ffb700 1 -- 172.21.15.98:0/1063840198 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8011d20 con 0x7fd9e815ca70 2023-03-03T08:01:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.817+0000 7fd9ed5ff700 1 -- 172.21.15.98:0/1063840198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9e815ca70 msgr2=0x7fd9e80cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.817+0000 7fd9ed5ff700 1 --2- 172.21.15.98:0/1063840198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9e815ca70 0x7fd9e80cf440 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fd9d80098f0 tx=0x7fd9d8009c00 comp rx=0 tx=0).stop 2023-03-03T08:01:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.817+0000 7fd9ed5ff700 1 -- 172.21.15.98:0/1063840198 shutdown_connections 2023-03-03T08:01:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.817+0000 7fd9ed5ff700 1 --2- 172.21.15.98:0/1063840198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9e815ca70 0x7fd9e80cf440 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.817+0000 7fd9ed5ff700 1 --2- 172.21.15.98:0/1063840198 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9e80d0270 0x7fd9e80d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.817+0000 7fd9ed5ff700 1 -- 172.21.15.98:0/1063840198 >> 172.21.15.98:0/1063840198 conn(0x7fd9e8073e70 msgr2=0x7fd9e806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.817+0000 7fd9ed5ff700 1 -- 172.21.15.98:0/1063840198 shutdown_connections 2023-03-03T08:01:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.817+0000 7fd9ed5ff700 1 -- 172.21.15.98:0/1063840198 wait complete. 2023-03-03T08:01:35.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.819+0000 7fd9ed5ff700 1 Processor -- start 2023-03-03T08:01:35.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.820+0000 7fd9ed5ff700 1 -- start start 2023-03-03T08:01:35.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.821+0000 7fd9ed5ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e806b4b0 0x7fd9e81f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:35.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.821+0000 7fd9ed5ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9e80d0270 0x7fd9e8153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:35.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.821+0000 7fd9ed5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9e815ca70 0x7fd9e815e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:35.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.821+0000 7fd9ed5ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e80ca250 con 0x7fd9e80d0270 2023-03-03T08:01:35.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.821+0000 7fd9ed5ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e80cadd0 con 0x7fd9e806b4b0 2023-03-03T08:01:35.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.821+0000 7fd9ed5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e81f3b00 con 0x7fd9e815ca70 2023-03-03T08:01:35.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.821+0000 7fd9e67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9e80d0270 0x7fd9e8153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:35.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.821+0000 7fd9e67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9e80d0270 0x7fd9e8153700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49558/0 (socket says 172.21.15.98:49558) 2023-03-03T08:01:35.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.821+0000 7fd9e67fc700 1 -- 172.21.15.98:0/2806765381 learned_addr learned my addr 172.21.15.98:0/2806765381 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:35.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.821+0000 7fd9e6ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e806b4b0 0x7fd9e81f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:35.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.821+0000 7fd9e77fe700 1 --2- 172.21.15.98:0/2806765381 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9e815ca70 0x7fd9e815e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:35.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.822+0000 7fd9e67fc700 1 -- 172.21.15.98:0/2806765381 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9e815ca70 msgr2=0x7fd9e815e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:35.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.822+0000 7fd9e67fc700 1 --2- 172.21.15.98:0/2806765381 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9e815ca70 0x7fd9e815e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:35.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.822+0000 7fd9e67fc700 1 -- 172.21.15.98:0/2806765381 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e806b4b0 msgr2=0x7fd9e81f35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:35.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.822+0000 7fd9e67fc700 1 --2- 172.21.15.98:0/2806765381 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e806b4b0 0x7fd9e81f35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:35.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.822+0000 7fd9e67fc700 1 -- 172.21.15.98:0/2806765381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d8009550 con 0x7fd9e80d0270 2023-03-03T08:01:35.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.822+0000 7fd9e67fc700 1 --2- 172.21.15.98:0/2806765381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9e80d0270 0x7fd9e8153700 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00c420 tx=0x7fd9dc00c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:35.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.822+0000 7fd9cf7fe700 1 -- 172.21.15.98:0/2806765381 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc0052d0 con 0x7fd9e80d0270 2023-03-03T08:01:35.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.822+0000 7fd9cf7fe700 1 -- 172.21.15.98:0/2806765381 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc005430 con 0x7fd9e80d0270 2023-03-03T08:01:35.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.822+0000 7fd9ed5ff700 1 -- 172.21.15.98:0/2806765381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e80d1aa0 con 0x7fd9e80d0270 2023-03-03T08:01:35.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.822+0000 7fd9cf7fe700 1 -- 172.21.15.98:0/2806765381 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc0038c0 con 0x7fd9e80d0270 2023-03-03T08:01:35.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.822+0000 7fd9ed5ff700 1 -- 172.21.15.98:0/2806765381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e80cdbe0 con 0x7fd9e80d0270 2023-03-03T08:01:35.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.824+0000 7fd9ed5ff700 1 -- 172.21.15.98:0/2806765381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e815ce40 con 0x7fd9e80d0270 2023-03-03T08:01:35.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.825+0000 7fd9cf7fe700 1 -- 172.21.15.98:0/2806765381 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fd9dc019030 con 0x7fd9e80d0270 2023-03-03T08:01:35.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.826+0000 7fd9cf7fe700 1 --2- 172.21.15.98:0/2806765381 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd9d40684a0 0x7fd9d406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:35.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.826+0000 7fd9cf7fe700 1 -- 172.21.15.98:0/2806765381 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd9dc0854b0 con 0x7fd9e80d0270 2023-03-03T08:01:35.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.827+0000 7fd9e6ffd700 1 --2- 172.21.15.98:0/2806765381 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd9d40684a0 0x7fd9d406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:35.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.827+0000 7fd9e6ffd700 1 --2- 172.21.15.98:0/2806765381 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd9d40684a0 0x7fd9d406a950 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009c40 tx=0x7fd9d8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:35.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:35.829+0000 7fd9cf7fe700 1 -- 172.21.15.98:0/2806765381 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd9dc051bf0 con 0x7fd9e80d0270 2023-03-03T08:01:36.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.046+0000 7fd9ed5ff700 1 -- 172.21.15.98:0/2806765381 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9e80caf40 con 0x7fd9d40684a0 2023-03-03T08:01:36.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.048+0000 7fd9cf7fe700 1 -- 172.21.15.98:0/2806765381 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fd9e80caf40 con 0x7fd9d40684a0 2023-03-03T08:01:36.049 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T08:01:36.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:36.049 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephfs" 2023-03-03T08:01:36.050 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:36.050 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T08:01:36.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.050+0000 7fd9ed5ff700 1 -- 172.21.15.98:0/2806765381 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd9d40684a0 msgr2=0x7fd9d406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:36.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.050+0000 7fd9ed5ff700 1 --2- 172.21.15.98:0/2806765381 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd9d40684a0 0x7fd9d406a950 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009c40 tx=0x7fd9d8008030 comp rx=0 tx=0).stop 2023-03-03T08:01:36.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.050+0000 7fd9ed5ff700 1 -- 172.21.15.98:0/2806765381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9e80d0270 msgr2=0x7fd9e8153700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:36.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.050+0000 7fd9ed5ff700 1 --2- 172.21.15.98:0/2806765381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9e80d0270 0x7fd9e8153700 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00c420 tx=0x7fd9dc00c730 comp rx=0 tx=0).stop 2023-03-03T08:01:36.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.050+0000 7fd9ed5ff700 1 -- 172.21.15.98:0/2806765381 shutdown_connections 2023-03-03T08:01:36.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.050+0000 7fd9ed5ff700 1 --2- 172.21.15.98:0/2806765381 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd9d40684a0 0x7fd9d406a950 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.050+0000 7fd9ed5ff700 1 --2- 172.21.15.98:0/2806765381 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9e815ca70 0x7fd9e815e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.050+0000 7fd9ed5ff700 1 --2- 172.21.15.98:0/2806765381 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e806b4b0 0x7fd9e81f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.050+0000 7fd9ed5ff700 1 --2- 172.21.15.98:0/2806765381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9e80d0270 0x7fd9e8153700 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.051+0000 7fd9ed5ff700 1 -- 172.21.15.98:0/2806765381 >> 172.21.15.98:0/2806765381 conn(0x7fd9e8073e70 msgr2=0x7fd9e80d2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:36.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.051+0000 7fd9ed5ff700 1 -- 172.21.15.98:0/2806765381 shutdown_connections 2023-03-03T08:01:36.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.051+0000 7fd9ed5ff700 1 -- 172.21.15.98:0/2806765381 wait complete. 2023-03-03T08:01:36.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-03-03T08:01:36.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.331+0000 7f6b293ff700 1 Processor -- start 2023-03-03T08:01:36.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.331+0000 7f6b293ff700 1 -- start start 2023-03-03T08:01:36.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.333+0000 7f6b293ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6b2415ca70 0x7f6b240cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:36.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.333+0000 7f6b293ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b2406b4b0 0x7f6b240cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:36.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.333+0000 7f6b293ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b240cf980 con 0x7f6b240d00a0 2023-03-03T08:01:36.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.333+0000 7f6b293ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b240cd5b0 con 0x7f6b2415ca70 2023-03-03T08:01:36.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.333+0000 7f6b293ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b240d0510 con 0x7f6b2406b4b0 2023-03-03T08:01:36.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.333+0000 7f6b237fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6b240d00a0 0x7f6b240b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36620/0 (socket says 172.21.15.98:36620) 2023-03-03T08:01:36.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.333+0000 7f6b237fe700 1 -- 172.21.15.98:0/736213581 learned_addr learned my addr 172.21.15.98:0/736213581 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:36.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.333+0000 7f6b227fc700 1 --2- 172.21.15.98:0/736213581 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b2406b4b0 0x7f6b240cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:36.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.333+0000 7f6b22ffd700 1 --2- 172.21.15.98:0/736213581 >> v2:172.21.15.100:3300/0 conn(0x7f6b2415ca70 0x7f6b240cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:36.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.334+0000 7f6b21ffb700 1 -- 172.21.15.98:0/736213581 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970157375 0 0) 0x7f6b240cf980 con 0x7f6b240d00a0 2023-03-03T08:01:36.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.334+0000 7f6b21ffb700 1 -- 172.21.15.98:0/736213581 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b0c0035b0 con 0x7f6b240d00a0 2023-03-03T08:01:36.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.334+0000 7f6b227fc700 1 -- 172.21.15.98:0/736213581 >> v2:172.21.15.100:3300/0 conn(0x7f6b2415ca70 msgr2=0x7f6b240cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.334+0000 7f6b227fc700 1 --2- 172.21.15.98:0/736213581 >> v2:172.21.15.100:3300/0 conn(0x7f6b2415ca70 0x7f6b240cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.334+0000 7f6b227fc700 1 -- 172.21.15.98:0/736213581 >> v1:172.21.15.98:6789/0 conn(0x7f6b240d00a0 legacy=0x7f6b240b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.334+0000 7f6b227fc700 1 -- 172.21.15.98:0/736213581 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b240d2fd0 con 0x7f6b2406b4b0 2023-03-03T08:01:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.334+0000 7f6b227fc700 1 --2- 172.21.15.98:0/736213581 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b2406b4b0 0x7f6b240cfb60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009840 tx=0x7f6b18009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bcb205ada473fa6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.335+0000 7f6b21ffb700 1 -- 172.21.15.98:0/736213581 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18011830 con 0x7f6b2406b4b0 2023-03-03T08:01:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.335+0000 7f6b21ffb700 1 -- 172.21.15.98:0/736213581 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b18011990 con 0x7f6b2406b4b0 2023-03-03T08:01:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.335+0000 7f6b21ffb700 1 -- 172.21.15.98:0/736213581 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18011d20 con 0x7f6b2406b4b0 2023-03-03T08:01:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.335+0000 7f6b293ff700 1 -- 172.21.15.98:0/736213581 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b2406b4b0 msgr2=0x7f6b240cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.335+0000 7f6b293ff700 1 --2- 172.21.15.98:0/736213581 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b2406b4b0 0x7f6b240cfb60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009840 tx=0x7f6b18009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.335+0000 7f6b293ff700 1 -- 172.21.15.98:0/736213581 shutdown_connections 2023-03-03T08:01:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.335+0000 7f6b293ff700 1 --2- 172.21.15.98:0/736213581 >> v2:172.21.15.100:3300/0 conn(0x7f6b2415ca70 0x7f6b240cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.335+0000 7f6b293ff700 1 --2- 172.21.15.98:0/736213581 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b2406b4b0 0x7f6b240cfb60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.335+0000 7f6b293ff700 1 -- 172.21.15.98:0/736213581 >> 172.21.15.98:0/736213581 conn(0x7f6b24073e70 msgr2=0x7f6b2406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.335+0000 7f6b293ff700 1 -- 172.21.15.98:0/736213581 shutdown_connections 2023-03-03T08:01:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.335+0000 7f6b293ff700 1 -- 172.21.15.98:0/736213581 wait complete. 2023-03-03T08:01:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.337+0000 7f6b293ff700 1 Processor -- start 2023-03-03T08:01:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.338+0000 7f6b293ff700 1 -- start start 2023-03-03T08:01:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b293ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b2406b4b0 0x7f6b240ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b293ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b240d00a0 0x7f6b24150410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b293ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b2415ca70 0x7f6b24153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b293ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b240cb4d0 con 0x7f6b240d00a0 2023-03-03T08:01:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b293ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b240cce30 con 0x7f6b2415ca70 2023-03-03T08:01:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b293ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b24150950 con 0x7f6b2406b4b0 2023-03-03T08:01:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b227fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b240d00a0 0x7f6b24150410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b227fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b240d00a0 0x7f6b24150410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49564/0 (socket says 172.21.15.98:49564) 2023-03-03T08:01:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b227fc700 1 -- 172.21.15.98:0/4079757529 learned_addr learned my addr 172.21.15.98:0/4079757529 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b22ffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b2406b4b0 0x7f6b240ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b237fe700 1 --2- 172.21.15.98:0/4079757529 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b2415ca70 0x7f6b24153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b227fc700 1 -- 172.21.15.98:0/4079757529 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b2406b4b0 msgr2=0x7f6b240ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b227fc700 1 --2- 172.21.15.98:0/4079757529 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b2406b4b0 0x7f6b240ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b227fc700 1 -- 172.21.15.98:0/4079757529 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b2415ca70 msgr2=0x7f6b24153a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b227fc700 1 --2- 172.21.15.98:0/4079757529 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b2415ca70 0x7f6b24153a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.339+0000 7f6b227fc700 1 -- 172.21.15.98:0/4079757529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b180094a0 con 0x7f6b240d00a0 2023-03-03T08:01:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.340+0000 7f6b227fc700 1 --2- 172.21.15.98:0/4079757529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b240d00a0 0x7f6b24150410 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6b180090b0 tx=0x7f6b18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.340+0000 7f6b0b7fe700 1 -- 172.21.15.98:0/4079757529 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b180036a0 con 0x7f6b240d00a0 2023-03-03T08:01:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.340+0000 7f6b0b7fe700 1 -- 172.21.15.98:0/4079757529 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1801f070 con 0x7f6b240d00a0 2023-03-03T08:01:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.340+0000 7f6b0b7fe700 1 -- 172.21.15.98:0/4079757529 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18003a60 con 0x7f6b240d00a0 2023-03-03T08:01:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.340+0000 7f6b293ff700 1 -- 172.21.15.98:0/4079757529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b240cd900 con 0x7f6b240d00a0 2023-03-03T08:01:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.340+0000 7f6b293ff700 1 -- 172.21.15.98:0/4079757529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b240d1800 con 0x7f6b240d00a0 2023-03-03T08:01:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.341+0000 7f6b293ff700 1 -- 172.21.15.98:0/4079757529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b240cdbe0 con 0x7f6b240d00a0 2023-03-03T08:01:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.342+0000 7f6b0b7fe700 1 -- 172.21.15.98:0/4079757529 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f6b1800f050 con 0x7f6b240d00a0 2023-03-03T08:01:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.344+0000 7f6b0b7fe700 1 --2- 172.21.15.98:0/4079757529 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b100684a0 0x7f6b1006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.344+0000 7f6b0b7fe700 1 -- 172.21.15.98:0/4079757529 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6b18021070 con 0x7f6b240d00a0 2023-03-03T08:01:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.344+0000 7f6b22ffd700 1 --2- 172.21.15.98:0/4079757529 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b100684a0 0x7f6b1006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.345+0000 7f6b22ffd700 1 --2- 172.21.15.98:0/4079757529 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b100684a0 0x7f6b1006a950 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6b14009b40 tx=0x7f6b14009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:36.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.348+0000 7f6b0b7fe700 1 -- 172.21.15.98:0/4079757529 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6b18053d40 con 0x7f6b240d00a0 2023-03-03T08:01:36.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.544+0000 7f6b293ff700 1 -- 172.21.15.98:0/4079757529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6b240cb610 con 0x7f6b240d00a0 2023-03-03T08:01:36.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.544+0000 7f6b0b7fe700 1 -- 172.21.15.98:0/4079757529 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6b240cb610 con 0x7f6b240d00a0 2023-03-03T08:01:36.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.547+0000 7f6b293ff700 1 -- 172.21.15.98:0/4079757529 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b100684a0 msgr2=0x7f6b1006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:36.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.547+0000 7f6b293ff700 1 --2- 172.21.15.98:0/4079757529 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b100684a0 0x7f6b1006a950 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6b14009b40 tx=0x7f6b14009210 comp rx=0 tx=0).stop 2023-03-03T08:01:36.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.547+0000 7f6b293ff700 1 -- 172.21.15.98:0/4079757529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b240d00a0 msgr2=0x7f6b24150410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:36.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.547+0000 7f6b293ff700 1 --2- 172.21.15.98:0/4079757529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b240d00a0 0x7f6b24150410 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6b180090b0 tx=0x7f6b18004750 comp rx=0 tx=0).stop 2023-03-03T08:01:36.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.547+0000 7f6b293ff700 1 -- 172.21.15.98:0/4079757529 shutdown_connections 2023-03-03T08:01:36.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.547+0000 7f6b293ff700 1 --2- 172.21.15.98:0/4079757529 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f6b100684a0 0x7f6b1006a950 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.547+0000 7f6b293ff700 1 --2- 172.21.15.98:0/4079757529 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b2415ca70 0x7f6b24153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.547+0000 7f6b293ff700 1 --2- 172.21.15.98:0/4079757529 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b2406b4b0 0x7f6b240ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.547+0000 7f6b293ff700 1 --2- 172.21.15.98:0/4079757529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b240d00a0 0x7f6b24150410 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.547+0000 7f6b293ff700 1 -- 172.21.15.98:0/4079757529 >> 172.21.15.98:0/4079757529 conn(0x7f6b24073e70 msgr2=0x7f6b240cf400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:36.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.547+0000 7f6b293ff700 1 -- 172.21.15.98:0/4079757529 shutdown_connections 2023-03-03T08:01:36.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.548+0000 7f6b293ff700 1 -- 172.21.15.98:0/4079757529 wait complete. 2023-03-03T08:01:36.576 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:01:36.577 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:01:36.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:01:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.892+0000 7f4bfcb9d700 1 Processor -- start 2023-03-03T08:01:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.893+0000 7f4bfcb9d700 1 -- start start 2023-03-03T08:01:36.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.894+0000 7f4bfcb9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4bf80d1650 0x7f4bf80d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:36.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.894+0000 7f4bfcb9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4bf80b4a00 0x7f4bf80b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:36.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.894+0000 7f4bfcb9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf80b0c50 con 0x7f4bf80d0010 2023-03-03T08:01:36.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.894+0000 7f4bfcb9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bf80b0040 con 0x7f4bf80b4a00 2023-03-03T08:01:36.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.894+0000 7f4bfcb9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bf80b0320 con 0x7f4bf80d1650 2023-03-03T08:01:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.894+0000 7f4bf77fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4bf80d0010 0x7f4bf80d0f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36624/0 (socket says 172.21.15.98:36624) 2023-03-03T08:01:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.894+0000 7f4bf77fe700 1 -- 172.21.15.98:0/3453201587 learned_addr learned my addr 172.21.15.98:0/3453201587 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.894+0000 7f4bf7fff700 1 --2- 172.21.15.98:0/3453201587 >> v2:172.21.15.98:3300/0 conn(0x7f4bf80b4a00 0x7f4bf80b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.895+0000 7f4bf6ffd700 1 --2- 172.21.15.98:0/3453201587 >> v2:172.21.15.100:3300/0 conn(0x7f4bf80d1650 0x7f4bf80d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.895+0000 7f4bf67fc700 1 -- 172.21.15.98:0/3453201587 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627255372 0 0) 0x7f4bf80b0c50 con 0x7f4bf80d0010 2023-03-03T08:01:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.895+0000 7f4bf67fc700 1 -- 172.21.15.98:0/3453201587 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be00035b0 con 0x7f4bf80d0010 2023-03-03T08:01:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.895+0000 7f4bf7fff700 1 -- 172.21.15.98:0/3453201587 >> v2:172.21.15.100:3300/0 conn(0x7f4bf80d1650 msgr2=0x7f4bf80d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.895+0000 7f4bf7fff700 1 --2- 172.21.15.98:0/3453201587 >> v2:172.21.15.100:3300/0 conn(0x7f4bf80d1650 0x7f4bf80d03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.895+0000 7f4bf7fff700 1 -- 172.21.15.98:0/3453201587 >> v1:172.21.15.98:6789/0 conn(0x7f4bf80d0010 legacy=0x7f4bf80d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.895+0000 7f4bf7fff700 1 -- 172.21.15.98:0/3453201587 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf80b0600 con 0x7f4bf80b4a00 2023-03-03T08:01:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.895+0000 7f4bf7fff700 1 --2- 172.21.15.98:0/3453201587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf80b4a00 0x7f4bf80b2ad0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f4bf808c300 tx=0x7f4bf000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff32142ae4cfad55 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.896+0000 7f4bf67fc700 1 -- 172.21.15.98:0/3453201587 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf000e070 con 0x7f4bf80b4a00 2023-03-03T08:01:36.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.896+0000 7f4bf67fc700 1 -- 172.21.15.98:0/3453201587 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf000bb20 con 0x7f4bf80b4a00 2023-03-03T08:01:36.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.896+0000 7f4bf67fc700 1 -- 172.21.15.98:0/3453201587 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0012430 con 0x7f4bf80b4a00 2023-03-03T08:01:36.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.896+0000 7f4bfcb9d700 1 -- 172.21.15.98:0/3453201587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf80b4a00 msgr2=0x7f4bf80b2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.896+0000 7f4bfcb9d700 1 --2- 172.21.15.98:0/3453201587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf80b4a00 0x7f4bf80b2ad0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f4bf808c300 tx=0x7f4bf000b330 comp rx=0 tx=0).stop 2023-03-03T08:01:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.896+0000 7f4bfcb9d700 1 -- 172.21.15.98:0/3453201587 shutdown_connections 2023-03-03T08:01:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.896+0000 7f4bfcb9d700 1 --2- 172.21.15.98:0/3453201587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf80b4a00 0x7f4bf80b2ad0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.896+0000 7f4bfcb9d700 1 --2- 172.21.15.98:0/3453201587 >> v2:172.21.15.100:3300/0 conn(0x7f4bf80d1650 0x7f4bf80d03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.896+0000 7f4bfcb9d700 1 -- 172.21.15.98:0/3453201587 >> 172.21.15.98:0/3453201587 conn(0x7f4bf8073b00 msgr2=0x7f4bf80d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.896+0000 7f4bfcb9d700 1 -- 172.21.15.98:0/3453201587 shutdown_connections 2023-03-03T08:01:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.896+0000 7f4bfcb9d700 1 -- 172.21.15.98:0/3453201587 wait complete. 2023-03-03T08:01:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.899+0000 7f4bfcb9d700 1 Processor -- start 2023-03-03T08:01:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.900+0000 7f4bfcb9d700 1 -- start start 2023-03-03T08:01:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.901+0000 7f4bfcb9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4bf80b4a00 0x7f4bf8153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.901+0000 7f4bfcb9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4bf80d0010 0x7f4bf80b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.901+0000 7f4bfcb9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf80d1650 0x7f4bf81e2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.901+0000 7f4bfcb9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf8153a50 con 0x7f4bf80d1650 2023-03-03T08:01:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.901+0000 7f4bfcb9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf80b2a10 con 0x7f4bf80b4a00 2023-03-03T08:01:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.901+0000 7f4bfcb9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf80b1c10 con 0x7f4bf80d0010 2023-03-03T08:01:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.901+0000 7f4bf7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf80d1650 0x7f4bf81e2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.901+0000 7f4bf7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf80d1650 0x7f4bf81e2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49584/0 (socket says 172.21.15.98:49584) 2023-03-03T08:01:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.901+0000 7f4bf7fff700 1 -- 172.21.15.98:0/3708948591 learned_addr learned my addr 172.21.15.98:0/3708948591 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.902+0000 7f4bf6ffd700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4bf80d0010 0x7f4bf80b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.902+0000 7f4bf77fe700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4bf80b4a00 0x7f4bf8153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.902+0000 7f4bf7fff700 1 -- 172.21.15.98:0/3708948591 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4bf80d0010 msgr2=0x7f4bf80b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.902+0000 7f4bf7fff700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4bf80d0010 0x7f4bf80b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.902+0000 7f4bf7fff700 1 -- 172.21.15.98:0/3708948591 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4bf80b4a00 msgr2=0x7f4bf8153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.902+0000 7f4bf7fff700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4bf80b4a00 0x7f4bf8153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.902+0000 7f4bf7fff700 1 -- 172.21.15.98:0/3708948591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf0009a70 con 0x7f4bf80d1650 2023-03-03T08:01:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.902+0000 7f4bf6ffd700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4bf80d0010 0x7f4bf80b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.902+0000 7f4bf77fe700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4bf80b4a00 0x7f4bf8153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.902+0000 7f4bf7fff700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf80d1650 0x7f4bf81e2680 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0014040 tx=0x7f4bf0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.903+0000 7f4bdffff700 1 -- 172.21.15.98:0/3708948591 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0012a70 con 0x7f4bf80d1650 2023-03-03T08:01:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.903+0000 7f4bdffff700 1 -- 172.21.15.98:0/3708948591 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf0020070 con 0x7f4bf80d1650 2023-03-03T08:01:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.903+0000 7f4bdffff700 1 -- 172.21.15.98:0/3708948591 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0012e30 con 0x7f4bf80d1650 2023-03-03T08:01:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.903+0000 7f4bfcb9d700 1 -- 172.21.15.98:0/3708948591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf80d0b30 con 0x7f4bf80d1650 2023-03-03T08:01:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.903+0000 7f4bfcb9d700 1 -- 172.21.15.98:0/3708948591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf80b08e0 con 0x7f4bf80d1650 2023-03-03T08:01:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.904+0000 7f4bfcb9d700 1 -- 172.21.15.98:0/3708948591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf80d0cc0 con 0x7f4bf80d1650 2023-03-03T08:01:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.905+0000 7f4bdffff700 1 -- 172.21.15.98:0/3708948591 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f4bf002e070 con 0x7f4bf80d1650 2023-03-03T08:01:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.906+0000 7f4bdffff700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4be40684a0 0x7f4be406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.906+0000 7f4bdffff700 1 -- 172.21.15.98:0/3708948591 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4bf0022070 con 0x7f4bf80d1650 2023-03-03T08:01:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.907+0000 7f4bf77fe700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4be40684a0 0x7f4be406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.907+0000 7f4bf77fe700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4be40684a0 0x7f4be406a950 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4be80010e0 tx=0x7f4be8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:36.910+0000 7f4bdffff700 1 -- 172.21.15.98:0/3708948591 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4bf0057ce0 con 0x7f4bf80d1650 2023-03-03T08:01:37.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.103+0000 7f4bfcb9d700 1 -- 172.21.15.98:0/3708948591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bf80729b0 con 0x7f4bf80d1650 2023-03-03T08:01:37.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.104+0000 7f4bdffff700 1 -- 172.21.15.98:0/3708948591 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+17982 (secure 0 0 0) 0x7f4bf002c090 con 0x7f4bf80d1650 2023-03-03T08:01:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:37.106 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":78,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:01:22.383102+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:01:19.439394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-03-03T08:01:20.381345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:01:37.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.107+0000 7f4bfcb9d700 1 -- 172.21.15.98:0/3708948591 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4be40684a0 msgr2=0x7f4be406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:37.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.107+0000 7f4bfcb9d700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4be40684a0 0x7f4be406a950 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4be80010e0 tx=0x7f4be8009040 comp rx=0 tx=0).stop 2023-03-03T08:01:37.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.107+0000 7f4bfcb9d700 1 -- 172.21.15.98:0/3708948591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf80d1650 msgr2=0x7f4bf81e2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:37.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.107+0000 7f4bfcb9d700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf80d1650 0x7f4bf81e2680 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0014040 tx=0x7f4bf0004ad0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.107+0000 7f4bfcb9d700 1 -- 172.21.15.98:0/3708948591 shutdown_connections 2023-03-03T08:01:37.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.107+0000 7f4bfcb9d700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4be40684a0 0x7f4be406a950 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.107+0000 7f4bfcb9d700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf80d1650 0x7f4bf81e2680 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.107+0000 7f4bfcb9d700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4bf80b4a00 0x7f4bf8153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.107+0000 7f4bfcb9d700 1 --2- 172.21.15.98:0/3708948591 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4bf80d0010 0x7f4bf80b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.107+0000 7f4bfcb9d700 1 -- 172.21.15.98:0/3708948591 >> 172.21.15.98:0/3708948591 conn(0x7f4bf8073b00 msgr2=0x7f4bf80b8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:37.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.107+0000 7f4bfcb9d700 1 -- 172.21.15.98:0/3708948591 shutdown_connections 2023-03-03T08:01:37.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.107+0000 7f4bfcb9d700 1 -- 172.21.15.98:0/3708948591 wait complete. 2023-03-03T08:01:37.135 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T08:01:19.439394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T08:01:20.381345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:01:37.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:01:37.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.388+0000 7f992359e700 1 Processor -- start 2023-03-03T08:01:37.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.389+0000 7f992359e700 1 -- start start 2023-03-03T08:01:37.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.390+0000 7f992359e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99240cf570 0x7f992415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:37.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.390+0000 7f992359e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99240cd780 con 0x7f992415db30 2023-03-03T08:01:37.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.390+0000 7f992359e700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99240cda60 con 0x7f99241609b0 2023-03-03T08:01:37.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.390+0000 7f992359e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99240cebb0 con 0x7f99240cf570 2023-03-03T08:01:37.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.391+0000 7f9922d9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f992415db30 0x7f99240b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36628/0 (socket says 172.21.15.98:36628) 2023-03-03T08:01:37.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.391+0000 7f9922d9d700 1 -- 172.21.15.98:0/3743724085 learned_addr learned my addr 172.21.15.98:0/3743724085 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:37.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.391+0000 7f9921d9b700 1 --2- 172.21.15.98:0/3743724085 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99240cf570 0x7f992415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:37.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.391+0000 7f992159a700 1 -- 172.21.15.98:0/3743724085 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574550668 0 0) 0x7f99240cd780 con 0x7f992415db30 2023-03-03T08:01:37.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.391+0000 7f992159a700 1 -- 172.21.15.98:0/3743724085 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f990c0035b0 con 0x7f992415db30 2023-03-03T08:01:37.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.391+0000 7f992159a700 1 -- 172.21.15.98:0/3743724085 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510337185 0 0) 0x7f99240cda60 con 0x7f99241609b0 2023-03-03T08:01:37.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.391+0000 7f992159a700 1 -- 172.21.15.98:0/3743724085 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99240cd780 con 0x7f99241609b0 2023-03-03T08:01:37.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.391+0000 7f9921d9b700 1 -- 172.21.15.98:0/3743724085 >> v1:172.21.15.100:6789/0 conn(0x7f99241609b0 legacy=0x7f9924160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:37.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.391+0000 7f9921d9b700 1 -- 172.21.15.98:0/3743724085 >> v1:172.21.15.98:6789/0 conn(0x7f992415db30 legacy=0x7f99240b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:37.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.392+0000 7f992159a700 1 -- 172.21.15.98:0/3743724085 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1164506155 0 0) 0x7f990c0035b0 con 0x7f992415db30 2023-03-03T08:01:37.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.392+0000 7f9921d9b700 1 -- 172.21.15.98:0/3743724085 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99240cdd40 con 0x7f99240cf570 2023-03-03T08:01:37.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.392+0000 7f9921d9b700 1 --2- 172.21.15.98:0/3743724085 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99240cf570 0x7f992415d5f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9918009870 tx=0x7f9918009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d87a716ebb7405a1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:37.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.392+0000 7f992159a700 1 -- 172.21.15.98:0/3743724085 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9918011830 con 0x7f99240cf570 2023-03-03T08:01:37.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.392+0000 7f992159a700 1 -- 172.21.15.98:0/3743724085 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9918011990 con 0x7f99240cf570 2023-03-03T08:01:37.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.392+0000 7f992159a700 1 -- 172.21.15.98:0/3743724085 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9918011d20 con 0x7f99240cf570 2023-03-03T08:01:37.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.393+0000 7f992359e700 1 -- 172.21.15.98:0/3743724085 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99240cf570 msgr2=0x7f992415d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:37.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.393+0000 7f992359e700 1 --2- 172.21.15.98:0/3743724085 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99240cf570 0x7f992415d5f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9918009870 tx=0x7f9918009b80 comp rx=0 tx=0).stop 2023-03-03T08:01:37.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.393+0000 7f992359e700 1 -- 172.21.15.98:0/3743724085 shutdown_connections 2023-03-03T08:01:37.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.393+0000 7f992359e700 1 --2- 172.21.15.98:0/3743724085 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99240cf570 0x7f992415d5f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.393+0000 7f992359e700 1 -- 172.21.15.98:0/3743724085 >> 172.21.15.98:0/3743724085 conn(0x7f992406b880 msgr2=0x7f99240cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:37.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.393+0000 7f992359e700 1 -- 172.21.15.98:0/3743724085 shutdown_connections 2023-03-03T08:01:37.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.393+0000 7f992359e700 1 -- 172.21.15.98:0/3743724085 wait complete. 2023-03-03T08:01:37.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.395+0000 7f992359e700 1 Processor -- start 2023-03-03T08:01:37.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.395+0000 7f992359e700 1 -- start start 2023-03-03T08:01:37.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f992359e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99240cf570 0x7f99240c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:37.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f992359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f992415db30 0x7f992415c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:37.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f992359e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99241609b0 0x7f99240cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:37.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f992359e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992415cc20 con 0x7f992415db30 2023-03-03T08:01:37.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f992359e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992415c940 con 0x7f99241609b0 2023-03-03T08:01:37.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f992359e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9924150100 con 0x7f99240cf570 2023-03-03T08:01:37.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f9921d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f992415db30 0x7f992415c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:37.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f992259c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99240cf570 0x7f99240c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:37.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f9921d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f992415db30 0x7f992415c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49600/0 (socket says 172.21.15.98:49600) 2023-03-03T08:01:37.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f9921d9b700 1 -- 172.21.15.98:0/2722541191 learned_addr learned my addr 172.21.15.98:0/2722541191 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:37.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f9922d9d700 1 --2- 172.21.15.98:0/2722541191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99241609b0 0x7f99240cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:37.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f9921d9b700 1 -- 172.21.15.98:0/2722541191 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99240cf570 msgr2=0x7f99240c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:37.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f9921d9b700 1 --2- 172.21.15.98:0/2722541191 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99240cf570 0x7f99240c5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f9921d9b700 1 -- 172.21.15.98:0/2722541191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99241609b0 msgr2=0x7f99240cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:37.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f9921d9b700 1 --2- 172.21.15.98:0/2722541191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99241609b0 0x7f99240cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.397+0000 7f9921d9b700 1 -- 172.21.15.98:0/2722541191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99180094a0 con 0x7f992415db30 2023-03-03T08:01:37.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.398+0000 7f9922d9d700 1 --2- 172.21.15.98:0/2722541191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99241609b0 0x7f99240cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:37.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.398+0000 7f9921d9b700 1 --2- 172.21.15.98:0/2722541191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f992415db30 0x7f992415c0f0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f99180090b0 tx=0x7f9918004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:37.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.398+0000 7f9912ffd700 1 -- 172.21.15.98:0/2722541191 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991801a760 con 0x7f992415db30 2023-03-03T08:01:37.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.398+0000 7f9912ffd700 1 -- 172.21.15.98:0/2722541191 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f991801a8c0 con 0x7f992415db30 2023-03-03T08:01:37.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.398+0000 7f9912ffd700 1 -- 172.21.15.98:0/2722541191 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991801ac30 con 0x7f992415db30 2023-03-03T08:01:37.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.398+0000 7f992359e700 1 -- 172.21.15.98:0/2722541191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992415d0d0 con 0x7f992415db30 2023-03-03T08:01:37.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.399+0000 7f992359e700 1 -- 172.21.15.98:0/2722541191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f992415cdf0 con 0x7f992415db30 2023-03-03T08:01:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.400+0000 7f992359e700 1 -- 172.21.15.98:0/2722541191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99240b51b0 con 0x7f992415db30 2023-03-03T08:01:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.401+0000 7f9912ffd700 1 -- 172.21.15.98:0/2722541191 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f991800f050 con 0x7f992415db30 2023-03-03T08:01:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.403+0000 7f9912ffd700 1 --2- 172.21.15.98:0/2722541191 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f99040684a0 0x7f990406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.403+0000 7f9912ffd700 1 -- 172.21.15.98:0/2722541191 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9918086c40 con 0x7f992415db30 2023-03-03T08:01:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.404+0000 7f992259c700 1 --2- 172.21.15.98:0/2722541191 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f99040684a0 0x7f990406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:37.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.404+0000 7f992259c700 1 --2- 172.21.15.98:0/2722541191 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f99040684a0 0x7f990406a950 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9914001220 tx=0x7f9914009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:37.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.408+0000 7f9912ffd700 1 -- 172.21.15.98:0/2722541191 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f99180534a0 con 0x7f992415db30 2023-03-03T08:01:37.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.604+0000 7f992359e700 1 -- 172.21.15.98:0/2722541191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9924068290 con 0x7f992415db30 2023-03-03T08:01:37.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.605+0000 7f9912ffd700 1 -- 172.21.15.98:0/2722541191 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9918029020 con 0x7f992415db30 2023-03-03T08:01:37.606 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:01:37.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.607+0000 7f992359e700 1 -- 172.21.15.98:0/2722541191 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f99040684a0 msgr2=0x7f990406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:37.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.607+0000 7f992359e700 1 --2- 172.21.15.98:0/2722541191 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f99040684a0 0x7f990406a950 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9914001220 tx=0x7f9914009040 comp rx=0 tx=0).stop 2023-03-03T08:01:37.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.607+0000 7f992359e700 1 -- 172.21.15.98:0/2722541191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f992415db30 msgr2=0x7f992415c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:37.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.607+0000 7f992359e700 1 --2- 172.21.15.98:0/2722541191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f992415db30 0x7f992415c0f0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f99180090b0 tx=0x7f9918004750 comp rx=0 tx=0).stop 2023-03-03T08:01:37.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.607+0000 7f992359e700 1 -- 172.21.15.98:0/2722541191 shutdown_connections 2023-03-03T08:01:37.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.607+0000 7f992359e700 1 --2- 172.21.15.98:0/2722541191 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f99040684a0 0x7f990406a950 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.607+0000 7f992359e700 1 --2- 172.21.15.98:0/2722541191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99241609b0 0x7f99240cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.607+0000 7f992359e700 1 --2- 172.21.15.98:0/2722541191 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99240cf570 0x7f99240c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.607+0000 7f992359e700 1 --2- 172.21.15.98:0/2722541191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f992415db30 0x7f992415c0f0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.607+0000 7f992359e700 1 -- 172.21.15.98:0/2722541191 >> 172.21.15.98:0/2722541191 conn(0x7f992406b880 msgr2=0x7f992414f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.607+0000 7f992359e700 1 -- 172.21.15.98:0/2722541191 shutdown_connections 2023-03-03T08:01:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.608+0000 7f992359e700 1 -- 172.21.15.98:0/2722541191 wait complete. 2023-03-03T08:01:37.642 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:01:37.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.890+0000 7fce595ff700 1 Processor -- start 2023-03-03T08:01:37.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.890+0000 7fce595ff700 1 -- start start 2023-03-03T08:01:37.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.891+0000 7fce595ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fce5415ca70 0x7fce540cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:37.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.891+0000 7fce595ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce540d0270 0x7fce540d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:37.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.892+0000 7fce595ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce540cf980 con 0x7fce5406b4b0 2023-03-03T08:01:37.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.892+0000 7fce595ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce540cd5b0 con 0x7fce5415ca70 2023-03-03T08:01:37.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.892+0000 7fce595ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce540d1800 con 0x7fce540d0270 2023-03-03T08:01:37.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.892+0000 7fce527fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fce5406b4b0 0x7fce540cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36636/0 (socket says 172.21.15.98:36636) 2023-03-03T08:01:37.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.892+0000 7fce527fc700 1 -- 172.21.15.98:0/49267858 learned_addr learned my addr 172.21.15.98:0/49267858 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:37.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.892+0000 7fce537fe700 1 --2- 172.21.15.98:0/49267858 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce540d0270 0x7fce540d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:37.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.892+0000 7fce52ffd700 1 --2- 172.21.15.98:0/49267858 >> v2:172.21.15.100:3300/0 conn(0x7fce5415ca70 0x7fce540cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:37.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.892+0000 7fce51ffb700 1 -- 172.21.15.98:0/49267858 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669146444 0 0) 0x7fce540cf980 con 0x7fce5406b4b0 2023-03-03T08:01:37.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.893+0000 7fce51ffb700 1 -- 172.21.15.98:0/49267858 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce3c0035b0 con 0x7fce5406b4b0 2023-03-03T08:01:37.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.893+0000 7fce537fe700 1 -- 172.21.15.98:0/49267858 >> v2:172.21.15.100:3300/0 conn(0x7fce5415ca70 msgr2=0x7fce540cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:37.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.893+0000 7fce537fe700 1 --2- 172.21.15.98:0/49267858 >> v2:172.21.15.100:3300/0 conn(0x7fce5415ca70 0x7fce540cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.893+0000 7fce537fe700 1 -- 172.21.15.98:0/49267858 >> v1:172.21.15.98:6789/0 conn(0x7fce5406b4b0 legacy=0x7fce540cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:37.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.893+0000 7fce537fe700 1 -- 172.21.15.98:0/49267858 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce540cd900 con 0x7fce540d0270 2023-03-03T08:01:37.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.893+0000 7fce537fe700 1 --2- 172.21.15.98:0/49267858 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce540d0270 0x7fce540d1fc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009d90 tx=0x7fce4c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c89840cfe552e75 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:37.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.893+0000 7fce51ffb700 1 -- 172.21.15.98:0/49267858 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c00e070 con 0x7fce540d0270 2023-03-03T08:01:37.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.893+0000 7fce51ffb700 1 -- 172.21.15.98:0/49267858 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4c00bb20 con 0x7fce540d0270 2023-03-03T08:01:37.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.894+0000 7fce51ffb700 1 -- 172.21.15.98:0/49267858 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c00be90 con 0x7fce540d0270 2023-03-03T08:01:37.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.894+0000 7fce595ff700 1 -- 172.21.15.98:0/49267858 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce540d0270 msgr2=0x7fce540d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:37.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.894+0000 7fce595ff700 1 --2- 172.21.15.98:0/49267858 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce540d0270 0x7fce540d1fc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009d90 tx=0x7fce4c00b330 comp rx=0 tx=0).stop 2023-03-03T08:01:37.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.894+0000 7fce595ff700 1 -- 172.21.15.98:0/49267858 shutdown_connections 2023-03-03T08:01:37.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.894+0000 7fce595ff700 1 --2- 172.21.15.98:0/49267858 >> v2:172.21.15.100:3300/0 conn(0x7fce5415ca70 0x7fce540cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.894+0000 7fce595ff700 1 --2- 172.21.15.98:0/49267858 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce540d0270 0x7fce540d1fc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.894+0000 7fce595ff700 1 -- 172.21.15.98:0/49267858 >> 172.21.15.98:0/49267858 conn(0x7fce54073e70 msgr2=0x7fce5406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:37.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.894+0000 7fce595ff700 1 -- 172.21.15.98:0/49267858 shutdown_connections 2023-03-03T08:01:37.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.894+0000 7fce595ff700 1 -- 172.21.15.98:0/49267858 wait complete. 2023-03-03T08:01:37.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.896+0000 7fce595ff700 1 Processor -- start 2023-03-03T08:01:37.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.896+0000 7fce595ff700 1 -- start start 2023-03-03T08:01:37.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.897+0000 7fce595ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5406b4b0 0x7fce5415d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:37.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.897+0000 7fce595ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce540d0270 0x7fce5415f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.897+0000 7fce595ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce5415ca70 0x7fce5415a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.897+0000 7fce595ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5415f650 con 0x7fce5406b4b0 2023-03-03T08:01:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.897+0000 7fce595ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce540cb160 con 0x7fce5415ca70 2023-03-03T08:01:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.897+0000 7fce595ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce5415d6d0 con 0x7fce540d0270 2023-03-03T08:01:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.898+0000 7fce52ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5406b4b0 0x7fce5415d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.898+0000 7fce52ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5406b4b0 0x7fce5415d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49610/0 (socket says 172.21.15.98:49610) 2023-03-03T08:01:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.898+0000 7fce52ffd700 1 -- 172.21.15.98:0/299519511 learned_addr learned my addr 172.21.15.98:0/299519511 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:37.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.898+0000 7fce527fc700 1 --2- 172.21.15.98:0/299519511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce540d0270 0x7fce5415f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:37.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.898+0000 7fce537fe700 1 --2- 172.21.15.98:0/299519511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce5415ca70 0x7fce5415a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:37.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.898+0000 7fce52ffd700 1 -- 172.21.15.98:0/299519511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce540d0270 msgr2=0x7fce5415f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:37.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.898+0000 7fce52ffd700 1 --2- 172.21.15.98:0/299519511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce540d0270 0x7fce5415f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.898+0000 7fce52ffd700 1 -- 172.21.15.98:0/299519511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce5415ca70 msgr2=0x7fce5415a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:37.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.898+0000 7fce52ffd700 1 --2- 172.21.15.98:0/299519511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce5415ca70 0x7fce5415a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.898+0000 7fce52ffd700 1 -- 172.21.15.98:0/299519511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4c009a70 con 0x7fce5406b4b0 2023-03-03T08:01:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.898+0000 7fce537fe700 1 --2- 172.21.15.98:0/299519511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce5415ca70 0x7fce5415a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.898+0000 7fce52ffd700 1 --2- 172.21.15.98:0/299519511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5406b4b0 0x7fce5415d060 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fce4400e9b0 tx=0x7fce4400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.899+0000 7fce3b7fe700 1 -- 172.21.15.98:0/299519511 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce44009710 con 0x7fce5406b4b0 2023-03-03T08:01:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.899+0000 7fce3b7fe700 1 -- 172.21.15.98:0/299519511 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4400c850 con 0x7fce5406b4b0 2023-03-03T08:01:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.899+0000 7fce595ff700 1 -- 172.21.15.98:0/299519511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce540d1aa0 con 0x7fce5406b4b0 2023-03-03T08:01:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.899+0000 7fce3b7fe700 1 -- 172.21.15.98:0/299519511 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4400cbc0 con 0x7fce5406b4b0 2023-03-03T08:01:37.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.899+0000 7fce595ff700 1 -- 172.21.15.98:0/299519511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce540cdbe0 con 0x7fce5406b4b0 2023-03-03T08:01:37.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.900+0000 7fce595ff700 1 -- 172.21.15.98:0/299519511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5415ce40 con 0x7fce5406b4b0 2023-03-03T08:01:37.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.902+0000 7fce3b7fe700 1 -- 172.21.15.98:0/299519511 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fce44019030 con 0x7fce5406b4b0 2023-03-03T08:01:37.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.903+0000 7fce3b7fe700 1 --2- 172.21.15.98:0/299519511 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fce400684a0 0x7fce4006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:37.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.903+0000 7fce3b7fe700 1 -- 172.21.15.98:0/299519511 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fce44085e40 con 0x7fce5406b4b0 2023-03-03T08:01:37.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.903+0000 7fce527fc700 1 --2- 172.21.15.98:0/299519511 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fce400684a0 0x7fce4006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:37.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.904+0000 7fce527fc700 1 --2- 172.21.15.98:0/299519511 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fce400684a0 0x7fce4006a950 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fce541f5620 tx=0x7fce48009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:37.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:37.907+0000 7fce3b7fe700 1 -- 172.21.15.98:0/299519511 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fce44052500 con 0x7fce5406b4b0 2023-03-03T08:01:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.099+0000 7fce595ff700 1 -- 172.21.15.98:0/299519511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fce5404e6e0 con 0x7fce5406b4b0 2023-03-03T08:01:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.099+0000 7fce3b7fe700 1 -- 172.21.15.98:0/299519511 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v78) v1 ==== 92+0+10 (secure 0 0 0) 0x7fce44021020 con 0x7fce5406b4b0 2023-03-03T08:01:38.101 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:01:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.102+0000 7fce595ff700 1 -- 172.21.15.98:0/299519511 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fce400684a0 msgr2=0x7fce4006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.102+0000 7fce595ff700 1 --2- 172.21.15.98:0/299519511 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fce400684a0 0x7fce4006a950 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fce541f5620 tx=0x7fce48009040 comp rx=0 tx=0).stop 2023-03-03T08:01:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.102+0000 7fce595ff700 1 -- 172.21.15.98:0/299519511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5406b4b0 msgr2=0x7fce5415d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.102+0000 7fce595ff700 1 --2- 172.21.15.98:0/299519511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5406b4b0 0x7fce5415d060 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fce4400e9b0 tx=0x7fce4400ecc0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.102+0000 7fce595ff700 1 -- 172.21.15.98:0/299519511 shutdown_connections 2023-03-03T08:01:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.102+0000 7fce595ff700 1 --2- 172.21.15.98:0/299519511 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fce400684a0 0x7fce4006a950 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.102+0000 7fce595ff700 1 --2- 172.21.15.98:0/299519511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5406b4b0 0x7fce5415d060 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.102+0000 7fce595ff700 1 --2- 172.21.15.98:0/299519511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce5415ca70 0x7fce5415a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.102+0000 7fce595ff700 1 --2- 172.21.15.98:0/299519511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce540d0270 0x7fce5415f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.102+0000 7fce595ff700 1 -- 172.21.15.98:0/299519511 >> 172.21.15.98:0/299519511 conn(0x7fce54073e70 msgr2=0x7fce541e52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.102+0000 7fce595ff700 1 -- 172.21.15.98:0/299519511 shutdown_connections 2023-03-03T08:01:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.102+0000 7fce595ff700 1 -- 172.21.15.98:0/299519511 wait complete. 2023-03-03T08:01:38.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T08:01:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.379+0000 7f23fdbff700 1 Processor -- start 2023-03-03T08:01:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.379+0000 7f23fdbff700 1 -- start start 2023-03-03T08:01:38.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.381+0000 7f23fdbff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f23f815ca70 0x7f23f80cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.381+0000 7f23fdbff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f23f80d0270 0x7f23f80d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.381+0000 7f23fdbff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f80cf980 con 0x7f23f806b4b0 2023-03-03T08:01:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.381+0000 7f23fdbff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f80cd5b0 con 0x7f23f80d0270 2023-03-03T08:01:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.381+0000 7f23fdbff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f80d1800 con 0x7f23f815ca70 2023-03-03T08:01:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.381+0000 7f23f7fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f23f80d0270 0x7f23f80d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.381+0000 7f23f77fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f23f815ca70 0x7f23f80cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.381+0000 7f23f77fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f23f815ca70 0x7f23f80cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:52218/0 (socket says 172.21.15.98:52218) 2023-03-03T08:01:38.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.381+0000 7f23f7fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f23f80d0270 0x7f23f80d1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49612/0 (socket says 172.21.15.98:49612) 2023-03-03T08:01:38.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.381+0000 7f23f77fe700 1 -- 172.21.15.98:0/2921407800 learned_addr learned my addr 172.21.15.98:0/2921407800 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:38.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.381+0000 7f23f6ffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f23f806b4b0 0x7f23f80cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36640/0 (socket says 172.21.15.98:36640) 2023-03-03T08:01:38.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.382+0000 7f23f7fff700 1 -- 172.21.15.98:0/2921407800 >> v2:172.21.15.100:3300/0 conn(0x7f23f815ca70 msgr2=0x7f23f80cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:38.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.382+0000 7f23f7fff700 1 --2- 172.21.15.98:0/2921407800 >> v2:172.21.15.100:3300/0 conn(0x7f23f815ca70 0x7f23f80cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.382+0000 7f23f7fff700 1 -- 172.21.15.98:0/2921407800 >> v1:172.21.15.98:6789/0 conn(0x7f23f806b4b0 legacy=0x7f23f80cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:38.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.382+0000 7f23f7fff700 1 -- 172.21.15.98:0/2921407800 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f80cd900 con 0x7f23f80d0270 2023-03-03T08:01:38.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.382+0000 7f23f67fc700 1 -- 172.21.15.98:0/2921407800 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371127451 0 0) 0x7f23f80cf980 con 0x7f23f806b4b0 2023-03-03T08:01:38.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.382+0000 7f23f7fff700 1 --2- 172.21.15.98:0/2921407800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f80d0270 0x7f23f80d1fc0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f23f806c2c0 tx=0x7f23f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3711a2527129df22 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:38.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.383+0000 7f23f67fc700 1 -- 172.21.15.98:0/2921407800 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f000e070 con 0x7f23f80d0270 2023-03-03T08:01:38.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.383+0000 7f23f67fc700 1 -- 172.21.15.98:0/2921407800 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f000bb20 con 0x7f23f80d0270 2023-03-03T08:01:38.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.383+0000 7f23f67fc700 1 -- 172.21.15.98:0/2921407800 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f0012430 con 0x7f23f80d0270 2023-03-03T08:01:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.383+0000 7f23fdbff700 1 -- 172.21.15.98:0/2921407800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f80d0270 msgr2=0x7f23f80d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.383+0000 7f23fdbff700 1 --2- 172.21.15.98:0/2921407800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f80d0270 0x7f23f80d1fc0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f23f806c2c0 tx=0x7f23f000b330 comp rx=0 tx=0).stop 2023-03-03T08:01:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.383+0000 7f23fdbff700 1 -- 172.21.15.98:0/2921407800 shutdown_connections 2023-03-03T08:01:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.383+0000 7f23fdbff700 1 --2- 172.21.15.98:0/2921407800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f80d0270 0x7f23f80d1fc0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.383+0000 7f23fdbff700 1 --2- 172.21.15.98:0/2921407800 >> v2:172.21.15.100:3300/0 conn(0x7f23f815ca70 0x7f23f80cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.383+0000 7f23fdbff700 1 -- 172.21.15.98:0/2921407800 >> 172.21.15.98:0/2921407800 conn(0x7f23f8073e70 msgr2=0x7f23f806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.383+0000 7f23fdbff700 1 -- 172.21.15.98:0/2921407800 shutdown_connections 2023-03-03T08:01:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.384+0000 7f23fdbff700 1 -- 172.21.15.98:0/2921407800 wait complete. 2023-03-03T08:01:38.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.386+0000 7f23fdbff700 1 Processor -- start 2023-03-03T08:01:38.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.386+0000 7f23fdbff700 1 -- start start 2023-03-03T08:01:38.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.387+0000 7f23fdbff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f806b4b0 0x7f23f814e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:38.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.387+0000 7f23fdbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23f80d0270 0x7f23f80cc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:38.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.387+0000 7f23fdbff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23f815ca70 0x7f23f8155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:38.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.387+0000 7f23fdbff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f80b5a40 con 0x7f23f806b4b0 2023-03-03T08:01:38.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.387+0000 7f23fdbff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f80ca380 con 0x7f23f815ca70 2023-03-03T08:01:38.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.387+0000 7f23fdbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f80cc9a0 con 0x7f23f80d0270 2023-03-03T08:01:38.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.387+0000 7f23f77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f806b4b0 0x7f23f814e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:38.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.387+0000 7f23f77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f806b4b0 0x7f23f814e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49626/0 (socket says 172.21.15.98:49626) 2023-03-03T08:01:38.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.387+0000 7f23f77fe700 1 -- 172.21.15.98:0/3635358975 learned_addr learned my addr 172.21.15.98:0/3635358975 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:38.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.388+0000 7f23f77fe700 1 -- 172.21.15.98:0/3635358975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23f80d0270 msgr2=0x7f23f80cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:38.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.388+0000 7f23f77fe700 1 --2- 172.21.15.98:0/3635358975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23f80d0270 0x7f23f80cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.388+0000 7f23f77fe700 1 -- 172.21.15.98:0/3635358975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23f815ca70 msgr2=0x7f23f8155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:38.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.388+0000 7f23f77fe700 1 --2- 172.21.15.98:0/3635358975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23f815ca70 0x7f23f8155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.388+0000 7f23f77fe700 1 -- 172.21.15.98:0/3635358975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f0009a70 con 0x7f23f806b4b0 2023-03-03T08:01:38.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.388+0000 7f23f77fe700 1 --2- 172.21.15.98:0/3635358975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f806b4b0 0x7f23f814e4b0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f23e800e910 tx=0x7f23e800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:38.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.388+0000 7f23dffff700 1 -- 172.21.15.98:0/3635358975 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e800c850 con 0x7f23f806b4b0 2023-03-03T08:01:38.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.388+0000 7f23dffff700 1 -- 172.21.15.98:0/3635358975 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e800c9b0 con 0x7f23f806b4b0 2023-03-03T08:01:38.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.388+0000 7f23dffff700 1 -- 172.21.15.98:0/3635358975 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e800cd60 con 0x7f23f806b4b0 2023-03-03T08:01:38.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.388+0000 7f23fdbff700 1 -- 172.21.15.98:0/3635358975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f80d1aa0 con 0x7f23f806b4b0 2023-03-03T08:01:38.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.389+0000 7f23fdbff700 1 -- 172.21.15.98:0/3635358975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f80cdbe0 con 0x7f23f806b4b0 2023-03-03T08:01:38.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.390+0000 7f23fdbff700 1 -- 172.21.15.98:0/3635358975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f815ce40 con 0x7f23f806b4b0 2023-03-03T08:01:38.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.391+0000 7f23dffff700 1 -- 172.21.15.98:0/3635358975 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f23e8019030 con 0x7f23f806b4b0 2023-03-03T08:01:38.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.392+0000 7f23dffff700 1 --2- 172.21.15.98:0/3635358975 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f23e40684a0 0x7f23e406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:38.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.393+0000 7f23dffff700 1 -- 172.21.15.98:0/3635358975 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f23e8085e90 con 0x7f23f806b4b0 2023-03-03T08:01:38.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.393+0000 7f23f6ffd700 1 --2- 172.21.15.98:0/3635358975 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f23e40684a0 0x7f23e406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:38.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.394+0000 7f23f6ffd700 1 --2- 172.21.15.98:0/3635358975 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f23e40684a0 0x7f23e406a950 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f23f81526b0 tx=0x7f23ec0079f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:38.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.397+0000 7f23dffff700 1 -- 172.21.15.98:0/3635358975 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f23e8052550 con 0x7f23f806b4b0 2023-03-03T08:01:38.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.590+0000 7f23fdbff700 1 -- 172.21.15.98:0/3635358975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f23f80ca620 con 0x7f23f806b4b0 2023-03-03T08:01:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.590+0000 7f23dffff700 1 -- 172.21.15.98:0/3635358975 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) v1 ==== 104+0+11 (secure 0 0 0) 0x7f23e8017020 con 0x7f23f806b4b0 2023-03-03T08:01:38.625 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T08:01:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.593+0000 7f23fdbff700 1 -- 172.21.15.98:0/3635358975 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f23e40684a0 msgr2=0x7f23e406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.593+0000 7f23fdbff700 1 --2- 172.21.15.98:0/3635358975 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f23e40684a0 0x7f23e406a950 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f23f81526b0 tx=0x7f23ec0079f0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.593+0000 7f23fdbff700 1 -- 172.21.15.98:0/3635358975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f806b4b0 msgr2=0x7f23f814e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.593+0000 7f23fdbff700 1 --2- 172.21.15.98:0/3635358975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f806b4b0 0x7f23f814e4b0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f23e800e910 tx=0x7f23e800ecd0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.593+0000 7f23fdbff700 1 -- 172.21.15.98:0/3635358975 shutdown_connections 2023-03-03T08:01:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.593+0000 7f23fdbff700 1 --2- 172.21.15.98:0/3635358975 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f23e40684a0 0x7f23e406a950 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.593+0000 7f23fdbff700 1 --2- 172.21.15.98:0/3635358975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f806b4b0 0x7f23f814e4b0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.593+0000 7f23fdbff700 1 --2- 172.21.15.98:0/3635358975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23f815ca70 0x7f23f8155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.593+0000 7f23fdbff700 1 --2- 172.21.15.98:0/3635358975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23f80d0270 0x7f23f80cc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.593+0000 7f23fdbff700 1 -- 172.21.15.98:0/3635358975 >> 172.21.15.98:0/3635358975 conn(0x7f23f8073e70 msgr2=0x7f23f80b8050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.593+0000 7f23fdbff700 1 -- 172.21.15.98:0/3635358975 shutdown_connections 2023-03-03T08:01:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.593+0000 7f23fdbff700 1 -- 172.21.15.98:0/3635358975 wait complete. 2023-03-03T08:01:38.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T08:01:38.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.878+0000 7f26e7fff700 1 Processor -- start 2023-03-03T08:01:38.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.878+0000 7f26e7fff700 1 -- start start 2023-03-03T08:01:38.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.880+0000 7f26e7fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e81609b0 0x7f26e8160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:38.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.880+0000 7f26e7fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f26e80cf570 0x7f26e815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:38.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.880+0000 7f26e7fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e81612c0 con 0x7f26e815db30 2023-03-03T08:01:38.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.880+0000 7f26e7fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e80cebb0 con 0x7f26e80cf570 2023-03-03T08:01:38.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.880+0000 7f26e7fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e80cd780 con 0x7f26e81609b0 2023-03-03T08:01:38.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.880+0000 7f26e67fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f26e80cf570 0x7f26e815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:38.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.880+0000 7f26e67fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f26e80cf570 0x7f26e815d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49628/0 (socket says 172.21.15.98:49628) 2023-03-03T08:01:38.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.880+0000 7f26e67fc700 1 -- 172.21.15.98:0/614157601 learned_addr learned my addr 172.21.15.98:0/614157601 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.880+0000 7f26e6ffd700 1 --2- 172.21.15.98:0/614157601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e81609b0 0x7f26e8160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.881+0000 7f26e67fc700 1 -- 172.21.15.98:0/614157601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e81609b0 msgr2=0x7f26e8160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.881+0000 7f26e67fc700 1 --2- 172.21.15.98:0/614157601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e81609b0 0x7f26e8160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.881+0000 7f26e67fc700 1 -- 172.21.15.98:0/614157601 >> v1:172.21.15.100:6789/0 conn(0x7f26e815db30 legacy=0x7f26e80b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.881+0000 7f26e67fc700 1 -- 172.21.15.98:0/614157601 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e80cda60 con 0x7f26e80cf570 2023-03-03T08:01:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.881+0000 7f26e67fc700 1 --2- 172.21.15.98:0/614157601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e80cf570 0x7f26e815d5f0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0098f0 tx=0x7f26dc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feccc96872468934 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.881+0000 7f26e5ffb700 1 -- 172.21.15.98:0/614157601 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc00b8a0 con 0x7f26e80cf570 2023-03-03T08:01:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.881+0000 7f26e5ffb700 1 -- 172.21.15.98:0/614157601 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc00ba00 con 0x7f26e80cf570 2023-03-03T08:01:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.881+0000 7f26e5ffb700 1 -- 172.21.15.98:0/614157601 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc00bd90 con 0x7f26e80cf570 2023-03-03T08:01:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.882+0000 7f26e7fff700 1 -- 172.21.15.98:0/614157601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e80cf570 msgr2=0x7f26e815d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:38.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.882+0000 7f26e7fff700 1 --2- 172.21.15.98:0/614157601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e80cf570 0x7f26e815d5f0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0098f0 tx=0x7f26dc009c00 comp rx=0 tx=0).stop 2023-03-03T08:01:38.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.882+0000 7f26e7fff700 1 -- 172.21.15.98:0/614157601 shutdown_connections 2023-03-03T08:01:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.882+0000 7f26e7fff700 1 --2- 172.21.15.98:0/614157601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e81609b0 0x7f26e8160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.882+0000 7f26e7fff700 1 --2- 172.21.15.98:0/614157601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e80cf570 0x7f26e815d5f0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.882+0000 7f26e7fff700 1 -- 172.21.15.98:0/614157601 >> 172.21.15.98:0/614157601 conn(0x7f26e806b880 msgr2=0x7f26e80cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:38.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.882+0000 7f26e7fff700 1 -- 172.21.15.98:0/614157601 shutdown_connections 2023-03-03T08:01:38.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.882+0000 7f26e7fff700 1 -- 172.21.15.98:0/614157601 wait complete. 2023-03-03T08:01:38.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.884+0000 7f26e7fff700 1 Processor -- start 2023-03-03T08:01:38.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.884+0000 7f26e7fff700 1 -- start start 2023-03-03T08:01:38.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.885+0000 7f26e7fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e80cf570 0x7f26e8150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:38.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.885+0000 7f26e7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e815db30 0x7f26e80b8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:38.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.885+0000 7f26e7fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26e81609b0 0x7f26e80b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:38.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.885+0000 7f26e7fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e80b2d60 con 0x7f26e815db30 2023-03-03T08:01:38.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.885+0000 7f26e7fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e80b3040 con 0x7f26e81609b0 2023-03-03T08:01:38.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.885+0000 7f26e7fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e80b3320 con 0x7f26e80cf570 2023-03-03T08:01:38.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.886+0000 7f26e67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e815db30 0x7f26e80b8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:38.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.886+0000 7f26e67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e815db30 0x7f26e80b8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49644/0 (socket says 172.21.15.98:49644) 2023-03-03T08:01:38.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.886+0000 7f26e67fc700 1 -- 172.21.15.98:0/3110633787 learned_addr learned my addr 172.21.15.98:0/3110633787 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:38.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.886+0000 7f26e6ffd700 1 --2- 172.21.15.98:0/3110633787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e80cf570 0x7f26e8150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:38.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.886+0000 7f26e77fe700 1 --2- 172.21.15.98:0/3110633787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26e81609b0 0x7f26e80b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:38.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.886+0000 7f26e67fc700 1 -- 172.21.15.98:0/3110633787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e80cf570 msgr2=0x7f26e8150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:38.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.886+0000 7f26e67fc700 1 --2- 172.21.15.98:0/3110633787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e80cf570 0x7f26e8150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.886+0000 7f26e67fc700 1 -- 172.21.15.98:0/3110633787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26e81609b0 msgr2=0x7f26e80b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:38.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.886+0000 7f26e67fc700 1 --2- 172.21.15.98:0/3110633787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26e81609b0 0x7f26e80b40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:38.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.886+0000 7f26e67fc700 1 -- 172.21.15.98:0/3110633787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc009550 con 0x7f26e815db30 2023-03-03T08:01:38.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.886+0000 7f26e67fc700 1 --2- 172.21.15.98:0/3110633787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e815db30 0x7f26e80b8780 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009c40 tx=0x7f26dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:38.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.887+0000 7f26cf7fe700 1 -- 172.21.15.98:0/3110633787 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc004400 con 0x7f26e815db30 2023-03-03T08:01:38.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.887+0000 7f26cf7fe700 1 -- 172.21.15.98:0/3110633787 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc01f070 con 0x7f26e815db30 2023-03-03T08:01:38.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.887+0000 7f26cf7fe700 1 -- 172.21.15.98:0/3110633787 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc0038e0 con 0x7f26e815db30 2023-03-03T08:01:38.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.887+0000 7f26e7fff700 1 -- 172.21.15.98:0/3110633787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e815cdf0 con 0x7f26e815db30 2023-03-03T08:01:38.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.887+0000 7f26e7fff700 1 -- 172.21.15.98:0/3110633787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e80cdd40 con 0x7f26e815db30 2023-03-03T08:01:38.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.888+0000 7f26e7fff700 1 -- 172.21.15.98:0/3110633787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e815d0d0 con 0x7f26e815db30 2023-03-03T08:01:38.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.890+0000 7f26cf7fe700 1 -- 172.21.15.98:0/3110633787 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f26dc02a020 con 0x7f26e815db30 2023-03-03T08:01:38.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.891+0000 7f26cf7fe700 1 --2- 172.21.15.98:0/3110633787 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f26d40684a0 0x7f26d406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:38.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.891+0000 7f26cf7fe700 1 -- 172.21.15.98:0/3110633787 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f26dc087ac0 con 0x7f26e815db30 2023-03-03T08:01:38.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.891+0000 7f26e6ffd700 1 --2- 172.21.15.98:0/3110633787 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f26d40684a0 0x7f26d406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:38.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.892+0000 7f26e6ffd700 1 --2- 172.21.15.98:0/3110633787 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f26d40684a0 0x7f26d406a950 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f26d800b9b0 tx=0x7f26d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:38.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:38.894+0000 7f26cf7fe700 1 -- 172.21.15.98:0/3110633787 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f26dc054310 con 0x7f26e815db30 2023-03-03T08:01:39.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.084+0000 7f26e7fff700 1 -- 172.21.15.98:0/3110633787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f26e8068290 con 0x7f26e815db30 2023-03-03T08:01:39.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.084+0000 7f26cf7fe700 1 -- 172.21.15.98:0/3110633787 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) v1 ==== 100+0+11 (secure 0 0 0) 0x7f26dc057930 con 0x7f26e815db30 2023-03-03T08:01:39.086 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T08:01:39.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.087+0000 7f26e7fff700 1 -- 172.21.15.98:0/3110633787 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f26d40684a0 msgr2=0x7f26d406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:39.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.087+0000 7f26e7fff700 1 --2- 172.21.15.98:0/3110633787 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f26d40684a0 0x7f26d406a950 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f26d800b9b0 tx=0x7f26d8008040 comp rx=0 tx=0).stop 2023-03-03T08:01:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.087+0000 7f26e7fff700 1 -- 172.21.15.98:0/3110633787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e815db30 msgr2=0x7f26e80b8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.087+0000 7f26e7fff700 1 --2- 172.21.15.98:0/3110633787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e815db30 0x7f26e80b8780 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009c40 tx=0x7f26dc004750 comp rx=0 tx=0).stop 2023-03-03T08:01:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.087+0000 7f26e7fff700 1 -- 172.21.15.98:0/3110633787 shutdown_connections 2023-03-03T08:01:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.087+0000 7f26e7fff700 1 --2- 172.21.15.98:0/3110633787 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f26d40684a0 0x7f26d406a950 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.087+0000 7f26e7fff700 1 --2- 172.21.15.98:0/3110633787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26e81609b0 0x7f26e80b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.087+0000 7f26e7fff700 1 --2- 172.21.15.98:0/3110633787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e80cf570 0x7f26e8150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.087+0000 7f26e7fff700 1 --2- 172.21.15.98:0/3110633787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e815db30 0x7f26e80b8780 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.087+0000 7f26e7fff700 1 -- 172.21.15.98:0/3110633787 >> 172.21.15.98:0/3110633787 conn(0x7f26e806b880 msgr2=0x7f26e815e580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:39.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.087+0000 7f26e7fff700 1 -- 172.21.15.98:0/3110633787 shutdown_connections 2023-03-03T08:01:39.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.087+0000 7f26e7fff700 1 -- 172.21.15.98:0/3110633787 wait complete. 2023-03-03T08:01:39.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T08:01:39.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.360+0000 7fe2e499d700 1 Processor -- start 2023-03-03T08:01:39.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.361+0000 7fe2e499d700 1 -- start start 2023-03-03T08:01:39.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.362+0000 7fe2e499d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe2e01609b0 0x7fe2e0160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:39.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.362+0000 7fe2e499d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e01612c0 con 0x7fe2e00cf570 2023-03-03T08:01:39.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.362+0000 7fe2e499d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e015dd00 con 0x7fe2e00b4aa0 2023-03-03T08:01:39.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.362+0000 7fe2e499d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2e00cebb0 con 0x7fe2e01609b0 2023-03-03T08:01:39.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.362+0000 7fe2df7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe2e01609b0 0x7fe2e0160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:39.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.362+0000 7fe2deffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe2e00cf570 0x7fe2e015d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36650/0 (socket says 172.21.15.98:36650) 2023-03-03T08:01:39.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.362+0000 7fe2deffd700 1 -- 172.21.15.98:0/1446934883 learned_addr learned my addr 172.21.15.98:0/1446934883 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:39.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.363+0000 7fe2df7fe700 1 -- 172.21.15.98:0/1446934883 >> v1:172.21.15.100:6789/0 conn(0x7fe2e00b4aa0 legacy=0x7fe2e00d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:39.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.363+0000 7fe2df7fe700 1 -- 172.21.15.98:0/1446934883 >> v1:172.21.15.98:6789/0 conn(0x7fe2e00cf570 legacy=0x7fe2e015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:39.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.363+0000 7fe2df7fe700 1 -- 172.21.15.98:0/1446934883 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e00cd780 con 0x7fe2e01609b0 2023-03-03T08:01:39.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.363+0000 7fe2de7fc700 1 -- 172.21.15.98:0/1446934883 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218019920 0 0) 0x7fe2e01612c0 con 0x7fe2e00cf570 2023-03-03T08:01:39.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.363+0000 7fe2df7fe700 1 --2- 172.21.15.98:0/1446934883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2e01609b0 0x7fe2e0160d80 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009840 tx=0x7fe2d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5b78b41f3919bf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:39.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.364+0000 7fe2de7fc700 1 -- 172.21.15.98:0/1446934883 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0011830 con 0x7fe2e01609b0 2023-03-03T08:01:39.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.364+0000 7fe2de7fc700 1 -- 172.21.15.98:0/1446934883 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d0011990 con 0x7fe2e01609b0 2023-03-03T08:01:39.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.364+0000 7fe2de7fc700 1 -- 172.21.15.98:0/1446934883 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0011d20 con 0x7fe2e01609b0 2023-03-03T08:01:39.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.364+0000 7fe2e499d700 1 -- 172.21.15.98:0/1446934883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2e01609b0 msgr2=0x7fe2e0160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:39.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.364+0000 7fe2e499d700 1 --2- 172.21.15.98:0/1446934883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2e01609b0 0x7fe2e0160d80 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009840 tx=0x7fe2d0009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:39.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.364+0000 7fe2e499d700 1 -- 172.21.15.98:0/1446934883 shutdown_connections 2023-03-03T08:01:39.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.364+0000 7fe2e499d700 1 --2- 172.21.15.98:0/1446934883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2e01609b0 0x7fe2e0160d80 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.364+0000 7fe2e499d700 1 -- 172.21.15.98:0/1446934883 >> 172.21.15.98:0/1446934883 conn(0x7fe2e006b880 msgr2=0x7fe2e00cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:39.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.364+0000 7fe2e499d700 1 -- 172.21.15.98:0/1446934883 shutdown_connections 2023-03-03T08:01:39.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.364+0000 7fe2e499d700 1 -- 172.21.15.98:0/1446934883 wait complete. 2023-03-03T08:01:39.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.367+0000 7fe2e499d700 1 Processor -- start 2023-03-03T08:01:39.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.368+0000 7fe2e499d700 1 -- start start 2023-03-03T08:01:39.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.369+0000 7fe2e499d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2e00b4aa0 0x7fe2e014c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:39.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.369+0000 7fe2e499d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2e00cf570 0x7fe2e00b7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:39.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.369+0000 7fe2e499d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2e01609b0 0x7fe2e00b7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:39.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.369+0000 7fe2e499d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e0150210 con 0x7fe2e00b4aa0 2023-03-03T08:01:39.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.369+0000 7fe2e499d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e0150460 con 0x7fe2e00cf570 2023-03-03T08:01:39.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.369+0000 7fe2e499d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e0150740 con 0x7fe2e01609b0 2023-03-03T08:01:39.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.369+0000 7fe2df7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2e00b4aa0 0x7fe2e014c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:39.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.369+0000 7fe2df7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2e00b4aa0 0x7fe2e014c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49670/0 (socket says 172.21.15.98:49670) 2023-03-03T08:01:39.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.369+0000 7fe2df7fe700 1 -- 172.21.15.98:0/1998911652 learned_addr learned my addr 172.21.15.98:0/1998911652 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:39.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.369+0000 7fe2deffd700 1 --2- 172.21.15.98:0/1998911652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2e00cf570 0x7fe2e00b7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:39.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.369+0000 7fe2dffff700 1 --2- 172.21.15.98:0/1998911652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2e01609b0 0x7fe2e00b7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:39.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.370+0000 7fe2df7fe700 1 -- 172.21.15.98:0/1998911652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2e01609b0 msgr2=0x7fe2e00b7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:39.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.370+0000 7fe2df7fe700 1 --2- 172.21.15.98:0/1998911652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2e01609b0 0x7fe2e00b7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.370+0000 7fe2df7fe700 1 -- 172.21.15.98:0/1998911652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2e00cf570 msgr2=0x7fe2e00b7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:39.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.370+0000 7fe2df7fe700 1 --2- 172.21.15.98:0/1998911652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2e00cf570 0x7fe2e00b7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.370+0000 7fe2df7fe700 1 -- 172.21.15.98:0/1998911652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d00094a0 con 0x7fe2e00b4aa0 2023-03-03T08:01:39.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.370+0000 7fe2deffd700 1 --2- 172.21.15.98:0/1998911652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2e00cf570 0x7fe2e00b7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:39.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.370+0000 7fe2df7fe700 1 --2- 172.21.15.98:0/1998911652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2e00b4aa0 0x7fe2e014c3e0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0014040 tx=0x7fe2d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:39.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.370+0000 7fe2c7fff700 1 -- 172.21.15.98:0/1998911652 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d001a890 con 0x7fe2e00b4aa0 2023-03-03T08:01:39.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.370+0000 7fe2c7fff700 1 -- 172.21.15.98:0/1998911652 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d001a9f0 con 0x7fe2e00b4aa0 2023-03-03T08:01:39.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.371+0000 7fe2c7fff700 1 -- 172.21.15.98:0/1998911652 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d001ad60 con 0x7fe2e00b4aa0 2023-03-03T08:01:39.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.371+0000 7fe2e499d700 1 -- 172.21.15.98:0/1998911652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e00cdd40 con 0x7fe2e00b4aa0 2023-03-03T08:01:39.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.371+0000 7fe2e499d700 1 -- 172.21.15.98:0/1998911652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e00cda60 con 0x7fe2e00b4aa0 2023-03-03T08:01:39.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.372+0000 7fe2e499d700 1 -- 172.21.15.98:0/1998911652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e015cdf0 con 0x7fe2e00b4aa0 2023-03-03T08:01:39.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.373+0000 7fe2c7fff700 1 -- 172.21.15.98:0/1998911652 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fe2d000f050 con 0x7fe2e00b4aa0 2023-03-03T08:01:39.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.374+0000 7fe2c7fff700 1 --2- 172.21.15.98:0/1998911652 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe2cc0684a0 0x7fe2cc06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:39.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.374+0000 7fe2c7fff700 1 -- 172.21.15.98:0/1998911652 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe2d0086ea0 con 0x7fe2e00b4aa0 2023-03-03T08:01:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.375+0000 7fe2deffd700 1 --2- 172.21.15.98:0/1998911652 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe2cc0684a0 0x7fe2cc06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.375+0000 7fe2deffd700 1 --2- 172.21.15.98:0/1998911652 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe2cc0684a0 0x7fe2cc06a950 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe2e01f5300 tx=0x7fe2d4007ce0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:39.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.379+0000 7fe2c7fff700 1 -- 172.21.15.98:0/1998911652 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe2d0053610 con 0x7fe2e00b4aa0 2023-03-03T08:01:39.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.586+0000 7fe2e499d700 1 -- 172.21.15.98:0/1998911652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe2e0068290 con 0x7fe2e00b4aa0 2023-03-03T08:01:39.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.586+0000 7fe2c7fff700 1 -- 172.21.15.98:0/1998911652 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v78) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe2d0029070 con 0x7fe2e00b4aa0 2023-03-03T08:01:39.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.589+0000 7fe2e499d700 1 -- 172.21.15.98:0/1998911652 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe2cc0684a0 msgr2=0x7fe2cc06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:39.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.589+0000 7fe2e499d700 1 --2- 172.21.15.98:0/1998911652 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe2cc0684a0 0x7fe2cc06a950 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe2e01f5300 tx=0x7fe2d4007ce0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.589+0000 7fe2e499d700 1 -- 172.21.15.98:0/1998911652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2e00b4aa0 msgr2=0x7fe2e014c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:39.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.589+0000 7fe2e499d700 1 --2- 172.21.15.98:0/1998911652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2e00b4aa0 0x7fe2e014c3e0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0014040 tx=0x7fe2d0004750 comp rx=0 tx=0).stop 2023-03-03T08:01:39.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.589+0000 7fe2e499d700 1 -- 172.21.15.98:0/1998911652 shutdown_connections 2023-03-03T08:01:39.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.589+0000 7fe2e499d700 1 --2- 172.21.15.98:0/1998911652 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fe2cc0684a0 0x7fe2cc06a950 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.589+0000 7fe2e499d700 1 --2- 172.21.15.98:0/1998911652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2e00b4aa0 0x7fe2e014c3e0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.589+0000 7fe2e499d700 1 --2- 172.21.15.98:0/1998911652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2e01609b0 0x7fe2e00b7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.589+0000 7fe2e499d700 1 --2- 172.21.15.98:0/1998911652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2e00cf570 0x7fe2e00b7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.589+0000 7fe2e499d700 1 -- 172.21.15.98:0/1998911652 >> 172.21.15.98:0/1998911652 conn(0x7fe2e006b880 msgr2=0x7fe2e00b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:39.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.589+0000 7fe2e499d700 1 -- 172.21.15.98:0/1998911652 shutdown_connections 2023-03-03T08:01:39.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.589+0000 7fe2e499d700 1 -- 172.21.15.98:0/1998911652 wait complete. 2023-03-03T08:01:39.593 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T08:01:39.618 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:01:39.618 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:01:39.687 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:01:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.931+0000 7f00a3fff700 1 Processor -- start 2023-03-03T08:01:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.932+0000 7f00a3fff700 1 -- start start 2023-03-03T08:01:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.933+0000 7f00a3fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f00a4146360 0x7f00a4160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.933+0000 7f00a3fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00a40b4ab0 0x7f00a40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.933+0000 7f00a3fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f00a40731e0 0x7f00a4073660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.933+0000 7f00a3fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a41614c0 con 0x7f00a40731e0 2023-03-03T08:01:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.933+0000 7f00a3fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a40b5440 con 0x7f00a4146360 2023-03-03T08:01:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.933+0000 7f00a3fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40d10f0 con 0x7f00a40b4ab0 2023-03-03T08:01:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.933+0000 7f00a37fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f00a40731e0 0x7f00a4073660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.934+0000 7f00a37fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f00a40731e0 0x7f00a4073660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49678/0 (socket says 172.21.15.98:49678) 2023-03-03T08:01:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.934+0000 7f00a37fe700 1 -- 172.21.15.98:0/1047511172 learned_addr learned my addr 172.21.15.98:0/1047511172 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.934+0000 7f009bfff700 1 --2- 172.21.15.98:0/1047511172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00a40b4ab0 0x7f00a40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.934+0000 7f00a2ffd700 1 --2- 172.21.15.98:0/1047511172 >> v2:172.21.15.100:3300/0 conn(0x7f00a4146360 0x7f00a4160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.934+0000 7f00a37fe700 1 -- 172.21.15.98:0/1047511172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00a40b4ab0 msgr2=0x7f00a40b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.934+0000 7f00a37fe700 1 --2- 172.21.15.98:0/1047511172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00a40b4ab0 0x7f00a40b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.934+0000 7f00a37fe700 1 -- 172.21.15.98:0/1047511172 >> v2:172.21.15.100:3300/0 conn(0x7f00a4146360 msgr2=0x7f00a4160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.934+0000 7f00a37fe700 1 --2- 172.21.15.98:0/1047511172 >> v2:172.21.15.100:3300/0 conn(0x7f00a4146360 0x7f00a4160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.934+0000 7f00a37fe700 1 -- 172.21.15.98:0/1047511172 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a40cdf20 con 0x7f00a40731e0 2023-03-03T08:01:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.934+0000 7f00a37fe700 1 --2- 172.21.15.98:0/1047511172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a40731e0 0x7f00a4073660 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f00a40743f0 tx=0x7f009c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2586e00ae1632a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.935+0000 7f00a27fc700 1 -- 172.21.15.98:0/1047511172 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c00e070 con 0x7f00a40731e0 2023-03-03T08:01:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.935+0000 7f00a27fc700 1 -- 172.21.15.98:0/1047511172 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009c00b9c0 con 0x7f00a40731e0 2023-03-03T08:01:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.935+0000 7f00a27fc700 1 -- 172.21.15.98:0/1047511172 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c00bd70 con 0x7f00a40731e0 2023-03-03T08:01:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.935+0000 7f00a3fff700 1 -- 172.21.15.98:0/1047511172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a40731e0 msgr2=0x7f00a4073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.935+0000 7f00a3fff700 1 --2- 172.21.15.98:0/1047511172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a40731e0 0x7f00a4073660 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f00a40743f0 tx=0x7f009c00b330 comp rx=0 tx=0).stop 2023-03-03T08:01:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.935+0000 7f00a3fff700 1 -- 172.21.15.98:0/1047511172 shutdown_connections 2023-03-03T08:01:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.935+0000 7f00a3fff700 1 --2- 172.21.15.98:0/1047511172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a40731e0 0x7f00a4073660 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.935+0000 7f00a3fff700 1 --2- 172.21.15.98:0/1047511172 >> v2:172.21.15.100:3300/0 conn(0x7f00a4146360 0x7f00a4160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.935+0000 7f00a3fff700 1 --2- 172.21.15.98:0/1047511172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00a40b4ab0 0x7f00a40b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.935+0000 7f00a3fff700 1 -- 172.21.15.98:0/1047511172 >> 172.21.15.98:0/1047511172 conn(0x7f00a406b8e0 msgr2=0x7f00a40cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.935+0000 7f00a3fff700 1 -- 172.21.15.98:0/1047511172 shutdown_connections 2023-03-03T08:01:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.935+0000 7f00a3fff700 1 -- 172.21.15.98:0/1047511172 wait complete. 2023-03-03T08:01:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.938+0000 7f00a3fff700 1 Processor -- start 2023-03-03T08:01:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.938+0000 7f00a3fff700 1 -- start start 2023-03-03T08:01:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.939+0000 7f00a3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a40731e0 0x7f00a41f54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.939+0000 7f00a3fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00a40b4ab0 0x7f00a415fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.939+0000 7f00a3fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00a4146360 0x7f00a41f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.939+0000 7f00a3fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40b8330 con 0x7f00a40731e0 2023-03-03T08:01:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.939+0000 7f00a3fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a41f4410 con 0x7f00a40b4ab0 2023-03-03T08:01:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.939+0000 7f00a3fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a41f46f0 con 0x7f00a4146360 2023-03-03T08:01:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.939+0000 7f00a2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a40731e0 0x7f00a41f54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.939+0000 7f00a2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a40731e0 0x7f00a41f54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49688/0 (socket says 172.21.15.98:49688) 2023-03-03T08:01:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.939+0000 7f00a2ffd700 1 -- 172.21.15.98:0/2337160003 learned_addr learned my addr 172.21.15.98:0/2337160003 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.939+0000 7f009bfff700 1 --2- 172.21.15.98:0/2337160003 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00a40b4ab0 0x7f00a415fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.939+0000 7f00a37fe700 1 --2- 172.21.15.98:0/2337160003 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00a4146360 0x7f00a41f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.940+0000 7f00a2ffd700 1 -- 172.21.15.98:0/2337160003 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00a4146360 msgr2=0x7f00a41f38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.940+0000 7f00a2ffd700 1 --2- 172.21.15.98:0/2337160003 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00a4146360 0x7f00a41f38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.940+0000 7f00a2ffd700 1 -- 172.21.15.98:0/2337160003 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00a40b4ab0 msgr2=0x7f00a415fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.940+0000 7f00a2ffd700 1 --2- 172.21.15.98:0/2337160003 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00a40b4ab0 0x7f00a415fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.940+0000 7f00a2ffd700 1 -- 172.21.15.98:0/2337160003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c009b30 con 0x7f00a40731e0 2023-03-03T08:01:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.940+0000 7f00a2ffd700 1 --2- 172.21.15.98:0/2337160003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a40731e0 0x7f00a41f54b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f009400d800 tx=0x7f009400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.940+0000 7f00a0ff9700 1 -- 172.21.15.98:0/2337160003 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00940125f0 con 0x7f00a40731e0 2023-03-03T08:01:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.940+0000 7f00a0ff9700 1 -- 172.21.15.98:0/2337160003 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0094012750 con 0x7f00a40731e0 2023-03-03T08:01:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.940+0000 7f00a0ff9700 1 -- 172.21.15.98:0/2337160003 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0094012b00 con 0x7f00a40731e0 2023-03-03T08:01:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.940+0000 7f00a3fff700 1 -- 172.21.15.98:0/2337160003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a40cd960 con 0x7f00a40731e0 2023-03-03T08:01:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.940+0000 7f00a3fff700 1 -- 172.21.15.98:0/2337160003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a40cd680 con 0x7f00a40731e0 2023-03-03T08:01:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.941+0000 7f00a3fff700 1 -- 172.21.15.98:0/2337160003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a40cdc40 con 0x7f00a40731e0 2023-03-03T08:01:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.943+0000 7f00a0ff9700 1 -- 172.21.15.98:0/2337160003 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f0094012c60 con 0x7f00a40731e0 2023-03-03T08:01:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.944+0000 7f00a0ff9700 1 --2- 172.21.15.98:0/2337160003 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f00840684a0 0x7f008406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.944+0000 7f00a0ff9700 1 -- 172.21.15.98:0/2337160003 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f00940862d0 con 0x7f00a40731e0 2023-03-03T08:01:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.944+0000 7f009bfff700 1 --2- 172.21.15.98:0/2337160003 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f00840684a0 0x7f008406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.945+0000 7f009bfff700 1 --2- 172.21.15.98:0/2337160003 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f00840684a0 0x7f008406a950 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f00a4073eb0 tx=0x7f008c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:39.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:39.948+0000 7f00a0ff9700 1 -- 172.21.15.98:0/2337160003 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0094052a80 con 0x7f00a40731e0 2023-03-03T08:01:40.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.144+0000 7f00a3fff700 1 -- 172.21.15.98:0/2337160003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00a40ce650 con 0x7f00a40731e0 2023-03-03T08:01:40.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.145+0000 7f00a0ff9700 1 -- 172.21.15.98:0/2337160003 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+17982 (secure 0 0 0) 0x7f00940560a0 con 0x7f00a40731e0 2023-03-03T08:01:40.148 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:40.148 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":78,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:01:22.383102+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:01:19.439394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-03-03T08:01:20.381345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:01:40.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.148+0000 7f00a3fff700 1 -- 172.21.15.98:0/2337160003 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f00840684a0 msgr2=0x7f008406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:40.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.148+0000 7f00a3fff700 1 --2- 172.21.15.98:0/2337160003 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f00840684a0 0x7f008406a950 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f00a4073eb0 tx=0x7f008c008040 comp rx=0 tx=0).stop 2023-03-03T08:01:40.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.148+0000 7f00a3fff700 1 -- 172.21.15.98:0/2337160003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a40731e0 msgr2=0x7f00a41f54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:40.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.148+0000 7f00a3fff700 1 --2- 172.21.15.98:0/2337160003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a40731e0 0x7f00a41f54b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f009400d800 tx=0x7f009400dbc0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.148+0000 7f00a3fff700 1 -- 172.21.15.98:0/2337160003 shutdown_connections 2023-03-03T08:01:40.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.148+0000 7f00a3fff700 1 --2- 172.21.15.98:0/2337160003 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f00840684a0 0x7f008406a950 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.148+0000 7f00a3fff700 1 --2- 172.21.15.98:0/2337160003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a40731e0 0x7f00a41f54b0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.148+0000 7f00a3fff700 1 --2- 172.21.15.98:0/2337160003 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00a4146360 0x7f00a41f38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.148+0000 7f00a3fff700 1 --2- 172.21.15.98:0/2337160003 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00a40b4ab0 0x7f00a415fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.148+0000 7f00a3fff700 1 -- 172.21.15.98:0/2337160003 >> 172.21.15.98:0/2337160003 conn(0x7f00a406b8e0 msgr2=0x7f00a40b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:40.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.148+0000 7f00a3fff700 1 -- 172.21.15.98:0/2337160003 shutdown_connections 2023-03-03T08:01:40.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.148+0000 7f00a3fff700 1 -- 172.21.15.98:0/2337160003 wait complete. 2023-03-03T08:01:40.177 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T08:01:19.439394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T08:01:20.381345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:01:40.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:01:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.426+0000 7fc38759e700 1 Processor -- start 2023-03-03T08:01:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.426+0000 7fc38759e700 1 -- start start 2023-03-03T08:01:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.428+0000 7fc38759e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc38815a6f0 0x7fc38806e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.428+0000 7fc38759e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3880cbe30 0x7fc3880cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.428+0000 7fc38759e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38806cff0 con 0x7fc38806a330 2023-03-03T08:01:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.428+0000 7fc38759e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc38815aac0 con 0x7fc38815a6f0 2023-03-03T08:01:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.428+0000 7fc38759e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38806d2d0 con 0x7fc3880cbe30 2023-03-03T08:01:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.429+0000 7fc38659c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc38815a6f0 0x7fc38806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.429+0000 7fc386d9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc38806a330 0x7fc3880c38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36654/0 (socket says 172.21.15.98:36654) 2023-03-03T08:01:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.429+0000 7fc386d9d700 1 -- 172.21.15.98:0/3235767307 learned_addr learned my addr 172.21.15.98:0/3235767307 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.429+0000 7fc385d9b700 1 --2- 172.21.15.98:0/3235767307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3880cbe30 0x7fc3880cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.429+0000 7fc385d9b700 1 -- 172.21.15.98:0/3235767307 >> v2:172.21.15.100:3300/0 conn(0x7fc38815a6f0 msgr2=0x7fc38806e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.429+0000 7fc38559a700 1 -- 172.21.15.98:0/3235767307 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332555980 0 0) 0x7fc38806cff0 con 0x7fc38806a330 2023-03-03T08:01:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.429+0000 7fc385d9b700 1 --2- 172.21.15.98:0/3235767307 >> v2:172.21.15.100:3300/0 conn(0x7fc38815a6f0 0x7fc38806e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.429+0000 7fc385d9b700 1 -- 172.21.15.98:0/3235767307 >> v1:172.21.15.98:6789/0 conn(0x7fc38806a330 legacy=0x7fc3880c38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.429+0000 7fc385d9b700 1 -- 172.21.15.98:0/3235767307 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38806d5b0 con 0x7fc3880cbe30 2023-03-03T08:01:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.429+0000 7fc385d9b700 1 --2- 172.21.15.98:0/3235767307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3880cbe30 0x7fc3880cc2a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009840 tx=0x7fc37c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d045da37f3017f4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.430+0000 7fc38559a700 1 -- 172.21.15.98:0/3235767307 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c011830 con 0x7fc3880cbe30 2023-03-03T08:01:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.430+0000 7fc38559a700 1 -- 172.21.15.98:0/3235767307 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37c011990 con 0x7fc3880cbe30 2023-03-03T08:01:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.430+0000 7fc38559a700 1 -- 172.21.15.98:0/3235767307 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c011d20 con 0x7fc3880cbe30 2023-03-03T08:01:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.430+0000 7fc38759e700 1 -- 172.21.15.98:0/3235767307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3880cbe30 msgr2=0x7fc3880cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.430+0000 7fc38759e700 1 --2- 172.21.15.98:0/3235767307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3880cbe30 0x7fc3880cc2a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009840 tx=0x7fc37c009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.430+0000 7fc38759e700 1 -- 172.21.15.98:0/3235767307 shutdown_connections 2023-03-03T08:01:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.431+0000 7fc38759e700 1 --2- 172.21.15.98:0/3235767307 >> v2:172.21.15.100:3300/0 conn(0x7fc38815a6f0 0x7fc38806e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.431+0000 7fc38759e700 1 --2- 172.21.15.98:0/3235767307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3880cbe30 0x7fc3880cc2a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.431+0000 7fc38759e700 1 -- 172.21.15.98:0/3235767307 >> 172.21.15.98:0/3235767307 conn(0x7fc3880cc7e0 msgr2=0x7fc3880ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.431+0000 7fc38759e700 1 -- 172.21.15.98:0/3235767307 shutdown_connections 2023-03-03T08:01:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.431+0000 7fc38759e700 1 -- 172.21.15.98:0/3235767307 wait complete. 2023-03-03T08:01:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.433+0000 7fc38759e700 1 Processor -- start 2023-03-03T08:01:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.433+0000 7fc38759e700 1 -- start start 2023-03-03T08:01:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.434+0000 7fc38759e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc38806a330 0x7fc388156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.434+0000 7fc38759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3880cbe30 0x7fc3881f13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.434+0000 7fc38759e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc38815a6f0 0x7fc388155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.434+0000 7fc38759e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3881f18f0 con 0x7fc3880cbe30 2023-03-03T08:01:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.434+0000 7fc38759e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3881f1b50 con 0x7fc38806a330 2023-03-03T08:01:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.434+0000 7fc38759e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3881f1e30 con 0x7fc38815a6f0 2023-03-03T08:01:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.434+0000 7fc385d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3880cbe30 0x7fc3881f13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.434+0000 7fc385d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3880cbe30 0x7fc3881f13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49700/0 (socket says 172.21.15.98:49700) 2023-03-03T08:01:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.434+0000 7fc385d9b700 1 -- 172.21.15.98:0/2667671739 learned_addr learned my addr 172.21.15.98:0/2667671739 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.434+0000 7fc38659c700 1 --2- 172.21.15.98:0/2667671739 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc38806a330 0x7fc388156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.434+0000 7fc386d9d700 1 --2- 172.21.15.98:0/2667671739 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc38815a6f0 0x7fc388155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.435+0000 7fc385d9b700 1 -- 172.21.15.98:0/2667671739 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc38815a6f0 msgr2=0x7fc388155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.435+0000 7fc385d9b700 1 --2- 172.21.15.98:0/2667671739 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc38815a6f0 0x7fc388155740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.435+0000 7fc385d9b700 1 -- 172.21.15.98:0/2667671739 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc38806a330 msgr2=0x7fc388156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.435+0000 7fc385d9b700 1 --2- 172.21.15.98:0/2667671739 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc38806a330 0x7fc388156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.435+0000 7fc385d9b700 1 -- 172.21.15.98:0/2667671739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c0094a0 con 0x7fc3880cbe30 2023-03-03T08:01:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.435+0000 7fc38659c700 1 --2- 172.21.15.98:0/2667671739 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc38806a330 0x7fc388156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.435+0000 7fc385d9b700 1 --2- 172.21.15.98:0/2667671739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3880cbe30 0x7fc3881f13b0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009fd0 tx=0x7fc37c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.435+0000 7fc376ffd700 1 -- 172.21.15.98:0/2667671739 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c0038a0 con 0x7fc3880cbe30 2023-03-03T08:01:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.435+0000 7fc376ffd700 1 -- 172.21.15.98:0/2667671739 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37c01f070 con 0x7fc3880cbe30 2023-03-03T08:01:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.435+0000 7fc376ffd700 1 -- 172.21.15.98:0/2667671739 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c003c40 con 0x7fc3880cbe30 2023-03-03T08:01:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.435+0000 7fc38759e700 1 -- 172.21.15.98:0/2667671739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38806db70 con 0x7fc3880cbe30 2023-03-03T08:01:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.436+0000 7fc38759e700 1 -- 172.21.15.98:0/2667671739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38806d890 con 0x7fc3880cbe30 2023-03-03T08:01:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.437+0000 7fc38759e700 1 -- 172.21.15.98:0/2667671739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc388071400 con 0x7fc3880cbe30 2023-03-03T08:01:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.438+0000 7fc376ffd700 1 -- 172.21.15.98:0/2667671739 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fc37c00f050 con 0x7fc3880cbe30 2023-03-03T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.439+0000 7fc376ffd700 1 --2- 172.21.15.98:0/2667671739 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3680684a0 0x7fc36806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.439+0000 7fc376ffd700 1 -- 172.21.15.98:0/2667671739 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc37c021070 con 0x7fc3880cbe30 2023-03-03T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.439+0000 7fc38659c700 1 --2- 172.21.15.98:0/2667671739 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3680684a0 0x7fc36806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.440+0000 7fc38659c700 1 --2- 172.21.15.98:0/2667671739 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3680684a0 0x7fc36806a950 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc378009b90 tx=0x7fc378008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.443+0000 7fc376ffd700 1 -- 172.21.15.98:0/2667671739 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc37c053f90 con 0x7fc3880cbe30 2023-03-03T08:01:40.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.636+0000 7fc38759e700 1 -- 172.21.15.98:0/2667671739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc388068290 con 0x7fc3880cbe30 2023-03-03T08:01:40.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.637+0000 7fc376ffd700 1 -- 172.21.15.98:0/2667671739 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc37c0150a0 con 0x7fc3880cbe30 2023-03-03T08:01:40.638 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:01:40.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.639+0000 7fc38759e700 1 -- 172.21.15.98:0/2667671739 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3680684a0 msgr2=0x7fc36806a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:40.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.639+0000 7fc38759e700 1 --2- 172.21.15.98:0/2667671739 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3680684a0 0x7fc36806a950 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc378009b90 tx=0x7fc378008040 comp rx=0 tx=0).stop 2023-03-03T08:01:40.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.640+0000 7fc38759e700 1 -- 172.21.15.98:0/2667671739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3880cbe30 msgr2=0x7fc3881f13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:40.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.640+0000 7fc38759e700 1 --2- 172.21.15.98:0/2667671739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3880cbe30 0x7fc3881f13b0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009fd0 tx=0x7fc37c004750 comp rx=0 tx=0).stop 2023-03-03T08:01:40.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.640+0000 7fc38759e700 1 -- 172.21.15.98:0/2667671739 shutdown_connections 2023-03-03T08:01:40.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.640+0000 7fc38759e700 1 --2- 172.21.15.98:0/2667671739 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc3680684a0 0x7fc36806a950 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.640+0000 7fc38759e700 1 --2- 172.21.15.98:0/2667671739 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc38815a6f0 0x7fc388155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.640+0000 7fc38759e700 1 --2- 172.21.15.98:0/2667671739 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc38806a330 0x7fc388156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.640+0000 7fc38759e700 1 --2- 172.21.15.98:0/2667671739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3880cbe30 0x7fc3881f13b0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.640+0000 7fc38759e700 1 -- 172.21.15.98:0/2667671739 >> 172.21.15.98:0/2667671739 conn(0x7fc3880cc7e0 msgr2=0x7fc3881574b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:40.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.640+0000 7fc38759e700 1 -- 172.21.15.98:0/2667671739 shutdown_connections 2023-03-03T08:01:40.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.640+0000 7fc38759e700 1 -- 172.21.15.98:0/2667671739 wait complete. 2023-03-03T08:01:40.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:01:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.917+0000 7f7af45ff700 1 Processor -- start 2023-03-03T08:01:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.917+0000 7f7af45ff700 1 -- start start 2023-03-03T08:01:40.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.918+0000 7f7af45ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7aec0d0270 0x7f7aec0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:40.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.918+0000 7f7af45ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7aec06b4b0 0x7f7aec0d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:40.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.919+0000 7f7af45ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aec0cf980 con 0x7f7aec15ca70 2023-03-03T08:01:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.919+0000 7f7af45ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aec0cd5b0 con 0x7f7aec0d0270 2023-03-03T08:01:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.919+0000 7f7af45ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec0d1800 con 0x7f7aec06b4b0 2023-03-03T08:01:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.919+0000 7f7af239b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7aec15ca70 0x7f7aec0cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36668/0 (socket says 172.21.15.98:36668) 2023-03-03T08:01:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.919+0000 7f7af239b700 1 -- 172.21.15.98:0/2032574034 learned_addr learned my addr 172.21.15.98:0/2032574034 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.919+0000 7f7af1b9a700 1 --2- 172.21.15.98:0/2032574034 >> v2:172.21.15.98:3300/0 conn(0x7f7aec0d0270 0x7f7aec0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.919+0000 7f7af2b9c700 1 --2- 172.21.15.98:0/2032574034 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7aec06b4b0 0x7f7aec0d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.919+0000 7f7af1399700 1 -- 172.21.15.98:0/2032574034 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178327304 0 0) 0x7f7aec0cf980 con 0x7f7aec15ca70 2023-03-03T08:01:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.919+0000 7f7af1399700 1 -- 172.21.15.98:0/2032574034 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ad80035b0 con 0x7f7aec15ca70 2023-03-03T08:01:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.920+0000 7f7af1b9a700 1 -- 172.21.15.98:0/2032574034 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7aec06b4b0 msgr2=0x7f7aec0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.920+0000 7f7af1b9a700 1 --2- 172.21.15.98:0/2032574034 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7aec06b4b0 0x7f7aec0d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.920+0000 7f7af1b9a700 1 -- 172.21.15.98:0/2032574034 >> v1:172.21.15.98:6789/0 conn(0x7f7aec15ca70 legacy=0x7f7aec0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.920+0000 7f7af1b9a700 1 -- 172.21.15.98:0/2032574034 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aec0cd900 con 0x7f7aec0d0270 2023-03-03T08:01:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.920+0000 7f7af1b9a700 1 --2- 172.21.15.98:0/2032574034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7aec0d0270 0x7f7aec0cf440 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009840 tx=0x7f7adc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e36475f8236e0bf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.920+0000 7f7af1399700 1 -- 172.21.15.98:0/2032574034 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc011830 con 0x7f7aec0d0270 2023-03-03T08:01:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.920+0000 7f7af1399700 1 -- 172.21.15.98:0/2032574034 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7adc011990 con 0x7f7aec0d0270 2023-03-03T08:01:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.920+0000 7f7af1399700 1 -- 172.21.15.98:0/2032574034 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc011d20 con 0x7f7aec0d0270 2023-03-03T08:01:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.921+0000 7f7af45ff700 1 -- 172.21.15.98:0/2032574034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7aec0d0270 msgr2=0x7f7aec0cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.921+0000 7f7af45ff700 1 --2- 172.21.15.98:0/2032574034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7aec0d0270 0x7f7aec0cf440 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009840 tx=0x7f7adc009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.921+0000 7f7af45ff700 1 -- 172.21.15.98:0/2032574034 shutdown_connections 2023-03-03T08:01:40.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.921+0000 7f7af45ff700 1 --2- 172.21.15.98:0/2032574034 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7aec06b4b0 0x7f7aec0d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.921+0000 7f7af45ff700 1 --2- 172.21.15.98:0/2032574034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7aec0d0270 0x7f7aec0cf440 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.921+0000 7f7af45ff700 1 -- 172.21.15.98:0/2032574034 >> 172.21.15.98:0/2032574034 conn(0x7f7aec073e70 msgr2=0x7f7aec06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:40.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.921+0000 7f7af45ff700 1 -- 172.21.15.98:0/2032574034 shutdown_connections 2023-03-03T08:01:40.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.921+0000 7f7af45ff700 1 -- 172.21.15.98:0/2032574034 wait complete. 2023-03-03T08:01:40.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.923+0000 7f7af45ff700 1 Processor -- start 2023-03-03T08:01:40.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.923+0000 7f7af45ff700 1 -- start start 2023-03-03T08:01:40.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.924+0000 7f7af45ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7aec06b4b0 0x7f7aec0c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:40.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.924+0000 7f7af45ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7aec0d0270 0x7f7aec1f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:40.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.924+0000 7f7af45ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7aec15ca70 0x7f7aec0c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:40.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.924+0000 7f7af45ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec1f68f0 con 0x7f7aec0d0270 2023-03-03T08:01:40.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.924+0000 7f7af45ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec0c9c10 con 0x7f7aec06b4b0 2023-03-03T08:01:40.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.924+0000 7f7af45ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec0c7940 con 0x7f7aec15ca70 2023-03-03T08:01:40.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.925+0000 7f7af1b9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7aec0d0270 0x7f7aec1f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:40.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.925+0000 7f7af1b9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7aec0d0270 0x7f7aec1f63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49718/0 (socket says 172.21.15.98:49718) 2023-03-03T08:01:40.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.925+0000 7f7af1b9a700 1 -- 172.21.15.98:0/574138152 learned_addr learned my addr 172.21.15.98:0/574138152 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:40.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.925+0000 7f7af239b700 1 --2- 172.21.15.98:0/574138152 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7aec06b4b0 0x7f7aec0c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:40.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.925+0000 7f7af2b9c700 1 --2- 172.21.15.98:0/574138152 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7aec15ca70 0x7f7aec0c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:40.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.925+0000 7f7af1b9a700 1 -- 172.21.15.98:0/574138152 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7aec15ca70 msgr2=0x7f7aec0c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:40.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.925+0000 7f7af1b9a700 1 --2- 172.21.15.98:0/574138152 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7aec15ca70 0x7f7aec0c96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.925+0000 7f7af1b9a700 1 -- 172.21.15.98:0/574138152 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7aec06b4b0 msgr2=0x7f7aec0c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:40.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.925+0000 7f7af1b9a700 1 --2- 172.21.15.98:0/574138152 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7aec06b4b0 0x7f7aec0c7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:40.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.925+0000 7f7af1b9a700 1 -- 172.21.15.98:0/574138152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc0094a0 con 0x7f7aec0d0270 2023-03-03T08:01:40.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.925+0000 7f7af1b9a700 1 --2- 172.21.15.98:0/574138152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7aec0d0270 0x7f7aec1f63b0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f7adc000c00 tx=0x7f7adc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.926+0000 7f7ae2ffd700 1 -- 172.21.15.98:0/574138152 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0036a0 con 0x7f7aec0d0270 2023-03-03T08:01:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.926+0000 7f7ae2ffd700 1 -- 172.21.15.98:0/574138152 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7adc01e070 con 0x7f7aec0d0270 2023-03-03T08:01:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.926+0000 7f7ae2ffd700 1 -- 172.21.15.98:0/574138152 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc003a60 con 0x7f7aec0d0270 2023-03-03T08:01:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.926+0000 7f7af45ff700 1 -- 172.21.15.98:0/574138152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aec0d1aa0 con 0x7f7aec0d0270 2023-03-03T08:01:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.926+0000 7f7af45ff700 1 -- 172.21.15.98:0/574138152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aec0cdbe0 con 0x7f7aec0d0270 2023-03-03T08:01:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.927+0000 7f7af45ff700 1 -- 172.21.15.98:0/574138152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aec15ce40 con 0x7f7aec0d0270 2023-03-03T08:01:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.928+0000 7f7ae2ffd700 1 -- 172.21.15.98:0/574138152 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f7adc00f050 con 0x7f7aec0d0270 2023-03-03T08:01:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.929+0000 7f7ae2ffd700 1 --2- 172.21.15.98:0/574138152 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7ad00684a0 0x7f7ad006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.929+0000 7f7ae2ffd700 1 -- 172.21.15.98:0/574138152 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7adc086e30 con 0x7f7aec0d0270 2023-03-03T08:01:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.930+0000 7f7af239b700 1 --2- 172.21.15.98:0/574138152 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7ad00684a0 0x7f7ad006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.930+0000 7f7af239b700 1 --2- 172.21.15.98:0/574138152 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7ad00684a0 0x7f7ad006a950 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80011e0 tx=0x7f7ae8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:40.933+0000 7f7ae2ffd700 1 -- 172.21.15.98:0/574138152 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7adc0534f0 con 0x7f7aec0d0270 2023-03-03T08:01:41.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.125+0000 7f7af45ff700 1 -- 172.21.15.98:0/574138152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7aec04e6e0 con 0x7f7aec0d0270 2023-03-03T08:01:41.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.126+0000 7f7ae2ffd700 1 -- 172.21.15.98:0/574138152 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v78) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7adc0150a0 con 0x7f7aec0d0270 2023-03-03T08:01:41.128 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:01:41.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.129+0000 7f7af45ff700 1 -- 172.21.15.98:0/574138152 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7ad00684a0 msgr2=0x7f7ad006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:41.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.129+0000 7f7af45ff700 1 --2- 172.21.15.98:0/574138152 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7ad00684a0 0x7f7ad006a950 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80011e0 tx=0x7f7ae8009040 comp rx=0 tx=0).stop 2023-03-03T08:01:41.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.129+0000 7f7af45ff700 1 -- 172.21.15.98:0/574138152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7aec0d0270 msgr2=0x7f7aec1f63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:41.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.129+0000 7f7af45ff700 1 --2- 172.21.15.98:0/574138152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7aec0d0270 0x7f7aec1f63b0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f7adc000c00 tx=0x7f7adc004750 comp rx=0 tx=0).stop 2023-03-03T08:01:41.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.129+0000 7f7af45ff700 1 -- 172.21.15.98:0/574138152 shutdown_connections 2023-03-03T08:01:41.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.129+0000 7f7af45ff700 1 --2- 172.21.15.98:0/574138152 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7ad00684a0 0x7f7ad006a950 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.129+0000 7f7af45ff700 1 --2- 172.21.15.98:0/574138152 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7aec15ca70 0x7f7aec0c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.129+0000 7f7af45ff700 1 --2- 172.21.15.98:0/574138152 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7aec06b4b0 0x7f7aec0c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.129+0000 7f7af45ff700 1 --2- 172.21.15.98:0/574138152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7aec0d0270 0x7f7aec1f63b0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.129+0000 7f7af45ff700 1 -- 172.21.15.98:0/574138152 >> 172.21.15.98:0/574138152 conn(0x7f7aec073e70 msgr2=0x7f7aec0b6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:41.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.129+0000 7f7af45ff700 1 -- 172.21.15.98:0/574138152 shutdown_connections 2023-03-03T08:01:41.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.129+0000 7f7af45ff700 1 -- 172.21.15.98:0/574138152 wait complete. 2023-03-03T08:01:41.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T08:01:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.404+0000 7f61b499d700 1 Processor -- start 2023-03-03T08:01:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.404+0000 7f61b499d700 1 -- start start 2023-03-03T08:01:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.406+0000 7f61b499d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f61b00cb3a0 0x7f61b015b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.406+0000 7f61b499d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f61b00b1ae0 0x7f61b00b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.406+0000 7f61b499d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b006bea0 con 0x7f61b00cafd0 2023-03-03T08:01:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.406+0000 7f61b499d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b015ba30 con 0x7f61b00b1ae0 2023-03-03T08:01:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.406+0000 7f61b499d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b00cea90 con 0x7f61b00cb3a0 2023-03-03T08:01:41.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.406+0000 7f61affff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f61b00b1ae0 0x7f61b00b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:41.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.407+0000 7f61affff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f61b00b1ae0 0x7f61b00b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49730/0 (socket says 172.21.15.98:49730) 2023-03-03T08:01:41.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.407+0000 7f61affff700 1 -- 172.21.15.98:0/3706918775 learned_addr learned my addr 172.21.15.98:0/3706918775 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:41.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.407+0000 7f61aeffd700 1 --2- 172.21.15.98:0/3706918775 >> v2:172.21.15.100:3300/0 conn(0x7f61b00cb3a0 0x7f61b015b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:41.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.407+0000 7f61affff700 1 -- 172.21.15.98:0/3706918775 >> v2:172.21.15.100:3300/0 conn(0x7f61b00cb3a0 msgr2=0x7f61b015b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:41.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.407+0000 7f61affff700 1 --2- 172.21.15.98:0/3706918775 >> v2:172.21.15.100:3300/0 conn(0x7f61b00cb3a0 0x7f61b015b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.407+0000 7f61affff700 1 -- 172.21.15.98:0/3706918775 >> v1:172.21.15.100:6789/0 conn(0x7f61b00cafd0 legacy=0x7f61b00ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:41.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.407+0000 7f61affff700 1 -- 172.21.15.98:0/3706918775 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b00b2230 con 0x7f61b00b1ae0 2023-03-03T08:01:41.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.407+0000 7f61affff700 1 --2- 172.21.15.98:0/3706918775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b00b1ae0 0x7f61b00b3150 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f61b006e700 tx=0x7f61a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7028f0d45f141638 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:41.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.408+0000 7f61ae7fc700 1 -- 172.21.15.98:0/3706918775 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a800e070 con 0x7f61b00b1ae0 2023-03-03T08:01:41.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.408+0000 7f61ae7fc700 1 -- 172.21.15.98:0/3706918775 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a800b8d0 con 0x7f61b00b1ae0 2023-03-03T08:01:41.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.408+0000 7f61ae7fc700 1 -- 172.21.15.98:0/3706918775 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a800bc80 con 0x7f61b00b1ae0 2023-03-03T08:01:41.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.408+0000 7f61b499d700 1 -- 172.21.15.98:0/3706918775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b00b1ae0 msgr2=0x7f61b00b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:41.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.408+0000 7f61b499d700 1 --2- 172.21.15.98:0/3706918775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b00b1ae0 0x7f61b00b3150 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f61b006e700 tx=0x7f61a800b330 comp rx=0 tx=0).stop 2023-03-03T08:01:41.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.408+0000 7f61b499d700 1 -- 172.21.15.98:0/3706918775 shutdown_connections 2023-03-03T08:01:41.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.409+0000 7f61b499d700 1 --2- 172.21.15.98:0/3706918775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b00b1ae0 0x7f61b00b3150 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.409+0000 7f61b499d700 1 --2- 172.21.15.98:0/3706918775 >> v2:172.21.15.100:3300/0 conn(0x7f61b00cb3a0 0x7f61b015b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.409+0000 7f61b499d700 1 -- 172.21.15.98:0/3706918775 >> 172.21.15.98:0/3706918775 conn(0x7f61b00733c0 msgr2=0x7f61b006a330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:41.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.409+0000 7f61b499d700 1 -- 172.21.15.98:0/3706918775 shutdown_connections 2023-03-03T08:01:41.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.409+0000 7f61b499d700 1 -- 172.21.15.98:0/3706918775 wait complete. 2023-03-03T08:01:41.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.411+0000 7f61b499d700 1 Processor -- start 2023-03-03T08:01:41.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.411+0000 7f61b499d700 1 -- start start 2023-03-03T08:01:41.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.412+0000 7f61b499d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f61b00b1ae0 0x7f61b014c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.412+0000 7f61b499d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f61b00cafd0 0x7f61b014bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.412+0000 7f61b499d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b00cb3a0 0x7f61b00b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.412+0000 7f61b499d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b00b0bc0 con 0x7f61b00cb3a0 2023-03-03T08:01:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.412+0000 7f61b499d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b014cdb0 con 0x7f61b00cafd0 2023-03-03T08:01:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.412+0000 7f61b499d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b01e4530 con 0x7f61b00b1ae0 2023-03-03T08:01:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.412+0000 7f61affff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b00cb3a0 0x7f61b00b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.412+0000 7f61affff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b00cb3a0 0x7f61b00b0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49744/0 (socket says 172.21.15.98:49744) 2023-03-03T08:01:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.412+0000 7f61affff700 1 -- 172.21.15.98:0/622448793 learned_addr learned my addr 172.21.15.98:0/622448793 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.412+0000 7f61aeffd700 1 --2- 172.21.15.98:0/622448793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f61b00cafd0 0x7f61b014bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.412+0000 7f61af7fe700 1 --2- 172.21.15.98:0/622448793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f61b00b1ae0 0x7f61b014c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.413+0000 7f61affff700 1 -- 172.21.15.98:0/622448793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f61b00b1ae0 msgr2=0x7f61b014c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.413+0000 7f61affff700 1 --2- 172.21.15.98:0/622448793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f61b00b1ae0 0x7f61b014c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.413+0000 7f61affff700 1 -- 172.21.15.98:0/622448793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f61b00cafd0 msgr2=0x7f61b014bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.413+0000 7f61affff700 1 --2- 172.21.15.98:0/622448793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f61b00cafd0 0x7f61b014bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.413+0000 7f61affff700 1 -- 172.21.15.98:0/622448793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a8009b30 con 0x7f61b00cb3a0 2023-03-03T08:01:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.413+0000 7f61affff700 1 --2- 172.21.15.98:0/622448793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b00cb3a0 0x7f61b00b0fe0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f61a800b680 tx=0x7f61a8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.413+0000 7f6197fff700 1 -- 172.21.15.98:0/622448793 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a8004830 con 0x7f61b00cb3a0 2023-03-03T08:01:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.413+0000 7f6197fff700 1 -- 172.21.15.98:0/622448793 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a80092e0 con 0x7f61b00cb3a0 2023-03-03T08:01:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.413+0000 7f61b499d700 1 -- 172.21.15.98:0/622448793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b00b27f0 con 0x7f61b00cb3a0 2023-03-03T08:01:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.413+0000 7f6197fff700 1 -- 172.21.15.98:0/622448793 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a801c680 con 0x7f61b00cb3a0 2023-03-03T08:01:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.414+0000 7f61b499d700 1 -- 172.21.15.98:0/622448793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b00b2510 con 0x7f61b00cb3a0 2023-03-03T08:01:41.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.415+0000 7f61b499d700 1 -- 172.21.15.98:0/622448793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b00b1f40 con 0x7f61b00cb3a0 2023-03-03T08:01:41.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.416+0000 7f6197fff700 1 -- 172.21.15.98:0/622448793 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f61a801c7e0 con 0x7f61b00cb3a0 2023-03-03T08:01:41.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.418+0000 7f6197fff700 1 --2- 172.21.15.98:0/622448793 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f619c0684a0 0x7f619c06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:41.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.418+0000 7f6197fff700 1 -- 172.21.15.98:0/622448793 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f61a8087380 con 0x7f61b00cb3a0 2023-03-03T08:01:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.418+0000 7f61af7fe700 1 --2- 172.21.15.98:0/622448793 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f619c0684a0 0x7f619c06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.419+0000 7f61af7fe700 1 --2- 172.21.15.98:0/622448793 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f619c0684a0 0x7f619c06a950 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f61a00018a0 tx=0x7f61a0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.422+0000 7f6197fff700 1 -- 172.21.15.98:0/622448793 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f61a8053a40 con 0x7f61b00cb3a0 2023-03-03T08:01:41.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.618+0000 7f61b499d700 1 -- 172.21.15.98:0/622448793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f61b0068290 con 0x7f61b00cb3a0 2023-03-03T08:01:41.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.618+0000 7f6197fff700 1 -- 172.21.15.98:0/622448793 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) v1 ==== 104+0+11 (secure 0 0 0) 0x7f61a8057060 con 0x7f61b00cb3a0 2023-03-03T08:01:41.620 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T08:01:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.621+0000 7f61b499d700 1 -- 172.21.15.98:0/622448793 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f619c0684a0 msgr2=0x7f619c06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.621+0000 7f61b499d700 1 --2- 172.21.15.98:0/622448793 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f619c0684a0 0x7f619c06a950 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f61a00018a0 tx=0x7f61a0009040 comp rx=0 tx=0).stop 2023-03-03T08:01:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.621+0000 7f61b499d700 1 -- 172.21.15.98:0/622448793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b00cb3a0 msgr2=0x7f61b00b0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.621+0000 7f61b499d700 1 --2- 172.21.15.98:0/622448793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b00cb3a0 0x7f61b00b0fe0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f61a800b680 tx=0x7f61a8004ad0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.621+0000 7f61b499d700 1 -- 172.21.15.98:0/622448793 shutdown_connections 2023-03-03T08:01:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.621+0000 7f61b499d700 1 --2- 172.21.15.98:0/622448793 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f619c0684a0 0x7f619c06a950 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.621+0000 7f61b499d700 1 --2- 172.21.15.98:0/622448793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b00cb3a0 0x7f61b00b0fe0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.621+0000 7f61b499d700 1 --2- 172.21.15.98:0/622448793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f61b00b1ae0 0x7f61b014c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.621+0000 7f61b499d700 1 --2- 172.21.15.98:0/622448793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f61b00cafd0 0x7f61b014bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.621+0000 7f61b499d700 1 -- 172.21.15.98:0/622448793 >> 172.21.15.98:0/622448793 conn(0x7f61b00733c0 msgr2=0x7f61b0153990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:41.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.621+0000 7f61b499d700 1 -- 172.21.15.98:0/622448793 shutdown_connections 2023-03-03T08:01:41.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.621+0000 7f61b499d700 1 -- 172.21.15.98:0/622448793 wait complete. 2023-03-03T08:01:41.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T08:01:41.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.897+0000 7fa5fc9ff700 1 Processor -- start 2023-03-03T08:01:41.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.897+0000 7fa5fc9ff700 1 -- start start 2023-03-03T08:01:41.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.899+0000 7fa5fc9ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa5f815ca70 0x7fa5f80cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:41.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.899+0000 7fa5fc9ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f80cf980 con 0x7fa5f80d0270 2023-03-03T08:01:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.899+0000 7fa5fc9ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f80d1800 con 0x7fa5f806b4b0 2023-03-03T08:01:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.899+0000 7fa5fc9ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f80cd5b0 con 0x7fa5f815ca70 2023-03-03T08:01:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.900+0000 7fa5f6d9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa5f80d0270 0x7fa5f80d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36670/0 (socket says 172.21.15.98:36670) 2023-03-03T08:01:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.900+0000 7fa5f6d9d700 1 -- 172.21.15.98:0/3463066096 learned_addr learned my addr 172.21.15.98:0/3463066096 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.900+0000 7fa5f659c700 1 --2- 172.21.15.98:0/3463066096 >> v2:172.21.15.100:3300/0 conn(0x7fa5f815ca70 0x7fa5f80cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.900+0000 7fa5f659c700 1 -- 172.21.15.98:0/3463066096 >> v1:172.21.15.100:6789/0 conn(0x7fa5f806b4b0 legacy=0x7fa5f80cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.900+0000 7fa5f659c700 1 -- 172.21.15.98:0/3463066096 >> v1:172.21.15.98:6789/0 conn(0x7fa5f80d0270 legacy=0x7fa5f80d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.900+0000 7fa5f659c700 1 -- 172.21.15.98:0/3463066096 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f80cd900 con 0x7fa5f815ca70 2023-03-03T08:01:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.900+0000 7fa5f559a700 1 -- 172.21.15.98:0/3463066096 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815738842 0 0) 0x7fa5f80cf980 con 0x7fa5f80d0270 2023-03-03T08:01:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.900+0000 7fa5f659c700 1 --2- 172.21.15.98:0/3463066096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5f815ca70 0x7fa5f80cf440 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8009840 tx=0x7fa5e8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea78be53904f8148 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:41.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.901+0000 7fa5f559a700 1 -- 172.21.15.98:0/3463066096 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8011830 con 0x7fa5f815ca70 2023-03-03T08:01:41.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.901+0000 7fa5f559a700 1 -- 172.21.15.98:0/3463066096 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e8011990 con 0x7fa5f815ca70 2023-03-03T08:01:41.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.901+0000 7fa5f559a700 1 -- 172.21.15.98:0/3463066096 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8011d20 con 0x7fa5f815ca70 2023-03-03T08:01:41.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.901+0000 7fa5fc9ff700 1 -- 172.21.15.98:0/3463066096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5f815ca70 msgr2=0x7fa5f80cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:41.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.901+0000 7fa5fc9ff700 1 --2- 172.21.15.98:0/3463066096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5f815ca70 0x7fa5f80cf440 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8009840 tx=0x7fa5e8009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:41.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.901+0000 7fa5fc9ff700 1 -- 172.21.15.98:0/3463066096 shutdown_connections 2023-03-03T08:01:41.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.901+0000 7fa5fc9ff700 1 --2- 172.21.15.98:0/3463066096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5f815ca70 0x7fa5f80cf440 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.902+0000 7fa5fc9ff700 1 -- 172.21.15.98:0/3463066096 >> 172.21.15.98:0/3463066096 conn(0x7fa5f8073e70 msgr2=0x7fa5f806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.902+0000 7fa5fc9ff700 1 -- 172.21.15.98:0/3463066096 shutdown_connections 2023-03-03T08:01:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.902+0000 7fa5fc9ff700 1 -- 172.21.15.98:0/3463066096 wait complete. 2023-03-03T08:01:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.904+0000 7fa5fc9ff700 1 Processor -- start 2023-03-03T08:01:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.904+0000 7fa5fc9ff700 1 -- start start 2023-03-03T08:01:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.905+0000 7fa5fc9ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5f806b4b0 0x7fa5f815a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.905+0000 7fa5fc9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa5f80d0270 0x7fa5f80b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.905+0000 7fa5fc9ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5f815ca70 0x7fa5f80b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.905+0000 7fa5fc9ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f815d090 con 0x7fa5f806b4b0 2023-03-03T08:01:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.905+0000 7fa5fc9ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f80ab480 con 0x7fa5f815ca70 2023-03-03T08:01:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.905+0000 7fa5fc9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f815ac90 con 0x7fa5f80d0270 2023-03-03T08:01:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.905+0000 7fa5f659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5f806b4b0 0x7fa5f815a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.905+0000 7fa5f659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5f806b4b0 0x7fa5f815a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49756/0 (socket says 172.21.15.98:49756) 2023-03-03T08:01:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.905+0000 7fa5f659c700 1 -- 172.21.15.98:0/3606759158 learned_addr learned my addr 172.21.15.98:0/3606759158 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.905+0000 7fa5f659c700 1 -- 172.21.15.98:0/3606759158 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa5f80d0270 msgr2=0x7fa5f80b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.906+0000 7fa5f5d9b700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa5f80d0270 0x7fa5f80b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.906+0000 7fa5f6d9d700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5f815ca70 0x7fa5f80b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.906+0000 7fa5f659c700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa5f80d0270 0x7fa5f80b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.906+0000 7fa5f659c700 1 -- 172.21.15.98:0/3606759158 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5f815ca70 msgr2=0x7fa5f80b3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.906+0000 7fa5f659c700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5f815ca70 0x7fa5f80b3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.906+0000 7fa5f659c700 1 -- 172.21.15.98:0/3606759158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e80094a0 con 0x7fa5f806b4b0 2023-03-03T08:01:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.906+0000 7fa5f5d9b700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa5f80d0270 0x7fa5f80b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.906+0000 7fa5f6d9d700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5f815ca70 0x7fa5f80b3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.906+0000 7fa5f659c700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5f806b4b0 0x7fa5f815a750 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fa5e80090b0 tx=0x7fa5e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.906+0000 7fa5e6ffd700 1 -- 172.21.15.98:0/3606759158 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e801a980 con 0x7fa5f806b4b0 2023-03-03T08:01:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.906+0000 7fa5e6ffd700 1 -- 172.21.15.98:0/3606759158 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e801aae0 con 0x7fa5f806b4b0 2023-03-03T08:01:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.906+0000 7fa5e6ffd700 1 -- 172.21.15.98:0/3606759158 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e801ae70 con 0x7fa5f806b4b0 2023-03-03T08:01:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.906+0000 7fa5fc9ff700 1 -- 172.21.15.98:0/3606759158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f80d1aa0 con 0x7fa5f806b4b0 2023-03-03T08:01:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.907+0000 7fa5fc9ff700 1 -- 172.21.15.98:0/3606759158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f80cdbe0 con 0x7fa5f806b4b0 2023-03-03T08:01:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.908+0000 7fa5fc9ff700 1 -- 172.21.15.98:0/3606759158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f815ce40 con 0x7fa5f806b4b0 2023-03-03T08:01:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.909+0000 7fa5e6ffd700 1 -- 172.21.15.98:0/3606759158 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fa5e800f050 con 0x7fa5f806b4b0 2023-03-03T08:01:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.910+0000 7fa5e6ffd700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa5d80684a0 0x7fa5d806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.910+0000 7fa5e6ffd700 1 -- 172.21.15.98:0/3606759158 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa5e8086f10 con 0x7fa5f806b4b0 2023-03-03T08:01:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.910+0000 7fa5f5d9b700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa5d80684a0 0x7fa5d806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.911+0000 7fa5f5d9b700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa5d80684a0 0x7fa5d806a950 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8160100 tx=0x7fa5ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:41.914+0000 7fa5e6ffd700 1 -- 172.21.15.98:0/3606759158 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa5e80536c0 con 0x7fa5f806b4b0 2023-03-03T08:01:42.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.105+0000 7fa5fc9ff700 1 -- 172.21.15.98:0/3606759158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa5f804e6e0 con 0x7fa5f806b4b0 2023-03-03T08:01:42.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.106+0000 7fa5e6ffd700 1 -- 172.21.15.98:0/3606759158 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa5e8029070 con 0x7fa5f806b4b0 2023-03-03T08:01:42.108 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T08:01:42.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.108+0000 7fa5fc9ff700 1 -- 172.21.15.98:0/3606759158 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa5d80684a0 msgr2=0x7fa5d806a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:42.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.109+0000 7fa5fc9ff700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa5d80684a0 0x7fa5d806a950 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8160100 tx=0x7fa5ec009040 comp rx=0 tx=0).stop 2023-03-03T08:01:42.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.109+0000 7fa5fc9ff700 1 -- 172.21.15.98:0/3606759158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5f806b4b0 msgr2=0x7fa5f815a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:42.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.109+0000 7fa5fc9ff700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5f806b4b0 0x7fa5f815a750 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fa5e80090b0 tx=0x7fa5e8004750 comp rx=0 tx=0).stop 2023-03-03T08:01:42.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.109+0000 7fa5fc9ff700 1 -- 172.21.15.98:0/3606759158 shutdown_connections 2023-03-03T08:01:42.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.109+0000 7fa5fc9ff700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fa5d80684a0 0x7fa5d806a950 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.109+0000 7fa5fc9ff700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5f806b4b0 0x7fa5f815a750 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.109+0000 7fa5fc9ff700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5f815ca70 0x7fa5f80b3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.109+0000 7fa5fc9ff700 1 --2- 172.21.15.98:0/3606759158 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa5f80d0270 0x7fa5f80b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.109+0000 7fa5fc9ff700 1 -- 172.21.15.98:0/3606759158 >> 172.21.15.98:0/3606759158 conn(0x7fa5f8073e70 msgr2=0x7fa5f80c9840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:42.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.109+0000 7fa5fc9ff700 1 -- 172.21.15.98:0/3606759158 shutdown_connections 2023-03-03T08:01:42.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.109+0000 7fa5fc9ff700 1 -- 172.21.15.98:0/3606759158 wait complete. 2023-03-03T08:01:42.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T08:01:42.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.385+0000 7f3701f9d700 1 Processor -- start 2023-03-03T08:01:42.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.386+0000 7f3701f9d700 1 -- start start 2023-03-03T08:01:42.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.387+0000 7f3701f9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f36fc1609b0 0x7f36fc160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:42.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.387+0000 7f3701f9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36fc0cf570 0x7f36fc15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:42.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.387+0000 7f3701f9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36fc15db30 0x7f36fc0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:42.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.387+0000 7f3701f9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36fc0cebb0 con 0x7f36fc0cf570 2023-03-03T08:01:42.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.387+0000 7f3701f9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36fc1612c0 con 0x7f36fc1609b0 2023-03-03T08:01:42.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.387+0000 7f3701f9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc0cd780 con 0x7f36fc15db30 2023-03-03T08:01:42.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.387+0000 7f36fbfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36fc0cf570 0x7f36fc15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:42.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.387+0000 7f36fbfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36fc0cf570 0x7f36fc15d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49760/0 (socket says 172.21.15.98:49760) 2023-03-03T08:01:42.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.387+0000 7f36fbfff700 1 -- 172.21.15.98:0/2239279884 learned_addr learned my addr 172.21.15.98:0/2239279884 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:42.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.388+0000 7f3700f9b700 1 --2- 172.21.15.98:0/2239279884 >> v2:172.21.15.100:3300/0 conn(0x7f36fc1609b0 0x7f36fc160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:42.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.388+0000 7f370179c700 1 --2- 172.21.15.98:0/2239279884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36fc15db30 0x7f36fc0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:42.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.388+0000 7f36fbfff700 1 -- 172.21.15.98:0/2239279884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36fc15db30 msgr2=0x7f36fc0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:42.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.388+0000 7f36fbfff700 1 --2- 172.21.15.98:0/2239279884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36fc15db30 0x7f36fc0b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.388+0000 7f36fbfff700 1 -- 172.21.15.98:0/2239279884 >> v2:172.21.15.100:3300/0 conn(0x7f36fc1609b0 msgr2=0x7f36fc160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:42.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.388+0000 7f36fbfff700 1 --2- 172.21.15.98:0/2239279884 >> v2:172.21.15.100:3300/0 conn(0x7f36fc1609b0 0x7f36fc160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.388+0000 7f36fbfff700 1 -- 172.21.15.98:0/2239279884 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36fc0cda60 con 0x7f36fc0cf570 2023-03-03T08:01:42.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.388+0000 7f370179c700 1 --2- 172.21.15.98:0/2239279884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36fc15db30 0x7f36fc0b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:42.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.388+0000 7f36fbfff700 1 --2- 172.21.15.98:0/2239279884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc0cf570 0x7f36fc15d5f0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f36fc0742b0 tx=0x7f36e8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f7472796e68166 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:42.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.388+0000 7f36fb7fe700 1 -- 172.21.15.98:0/2239279884 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e800b870 con 0x7f36fc0cf570 2023-03-03T08:01:42.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.389+0000 7f36fb7fe700 1 -- 172.21.15.98:0/2239279884 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e800b9d0 con 0x7f36fc0cf570 2023-03-03T08:01:42.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.389+0000 7f36fb7fe700 1 -- 172.21.15.98:0/2239279884 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e800bd60 con 0x7f36fc0cf570 2023-03-03T08:01:42.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.389+0000 7f3701f9d700 1 -- 172.21.15.98:0/2239279884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc0cf570 msgr2=0x7f36fc15d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:42.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.389+0000 7f3701f9d700 1 --2- 172.21.15.98:0/2239279884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc0cf570 0x7f36fc15d5f0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f36fc0742b0 tx=0x7f36e8009bd0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.389+0000 7f3701f9d700 1 -- 172.21.15.98:0/2239279884 shutdown_connections 2023-03-03T08:01:42.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.389+0000 7f3701f9d700 1 --2- 172.21.15.98:0/2239279884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36fc15db30 0x7f36fc0b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.389+0000 7f3701f9d700 1 --2- 172.21.15.98:0/2239279884 >> v2:172.21.15.100:3300/0 conn(0x7f36fc1609b0 0x7f36fc160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.389+0000 7f3701f9d700 1 --2- 172.21.15.98:0/2239279884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc0cf570 0x7f36fc15d5f0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.389+0000 7f3701f9d700 1 -- 172.21.15.98:0/2239279884 >> 172.21.15.98:0/2239279884 conn(0x7f36fc06b880 msgr2=0x7f36fc0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.389+0000 7f3701f9d700 1 -- 172.21.15.98:0/2239279884 shutdown_connections 2023-03-03T08:01:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.389+0000 7f3701f9d700 1 -- 172.21.15.98:0/2239279884 wait complete. 2023-03-03T08:01:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.391+0000 7f3701f9d700 1 Processor -- start 2023-03-03T08:01:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.391+0000 7f3701f9d700 1 -- start start 2023-03-03T08:01:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.392+0000 7f3701f9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36fc0cf570 0x7f36fc151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.392+0000 7f3701f9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36fc15db30 0x7f36fc0cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.392+0000 7f3701f9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc1609b0 0x7f36fc0b5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.392+0000 7f3701f9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc0b76f0 con 0x7f36fc1609b0 2023-03-03T08:01:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.392+0000 7f3701f9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc0b79d0 con 0x7f36fc0cf570 2023-03-03T08:01:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.393+0000 7f3701f9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc0b7cb0 con 0x7f36fc15db30 2023-03-03T08:01:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.393+0000 7f370179c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc1609b0 0x7f36fc0b5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.393+0000 7f370179c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc1609b0 0x7f36fc0b5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49768/0 (socket says 172.21.15.98:49768) 2023-03-03T08:01:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.393+0000 7f370179c700 1 -- 172.21.15.98:0/2432200103 learned_addr learned my addr 172.21.15.98:0/2432200103 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.393+0000 7f36fbfff700 1 --2- 172.21.15.98:0/2432200103 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36fc15db30 0x7f36fc0cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.393+0000 7f3700f9b700 1 --2- 172.21.15.98:0/2432200103 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36fc0cf570 0x7f36fc151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.393+0000 7f370179c700 1 -- 172.21.15.98:0/2432200103 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36fc15db30 msgr2=0x7f36fc0cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.393+0000 7f370179c700 1 --2- 172.21.15.98:0/2432200103 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36fc15db30 0x7f36fc0cc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.393+0000 7f370179c700 1 -- 172.21.15.98:0/2432200103 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36fc0cf570 msgr2=0x7f36fc151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.393+0000 7f370179c700 1 --2- 172.21.15.98:0/2432200103 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36fc0cf570 0x7f36fc151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.393+0000 7f370179c700 1 -- 172.21.15.98:0/2432200103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e8009550 con 0x7f36fc1609b0 2023-03-03T08:01:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.393+0000 7f3700f9b700 1 --2- 172.21.15.98:0/2432200103 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36fc0cf570 0x7f36fc151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.394+0000 7f370179c700 1 --2- 172.21.15.98:0/2432200103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc1609b0 0x7f36fc0b5e50 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f36f400cef0 tx=0x7f36f400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.394+0000 7f36f97fa700 1 -- 172.21.15.98:0/2432200103 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f400add0 con 0x7f36fc1609b0 2023-03-03T08:01:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.394+0000 7f36f97fa700 1 -- 172.21.15.98:0/2432200103 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f4015440 con 0x7f36fc1609b0 2023-03-03T08:01:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.394+0000 7f36f97fa700 1 -- 172.21.15.98:0/2432200103 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f40157f0 con 0x7f36fc1609b0 2023-03-03T08:01:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.394+0000 7f3701f9d700 1 -- 172.21.15.98:0/2432200103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36fc15cdf0 con 0x7f36fc1609b0 2023-03-03T08:01:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.394+0000 7f3701f9d700 1 -- 172.21.15.98:0/2432200103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36fc0cdd40 con 0x7f36fc1609b0 2023-03-03T08:01:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.395+0000 7f3701f9d700 1 -- 172.21.15.98:0/2432200103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36fc15d0d0 con 0x7f36fc1609b0 2023-03-03T08:01:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.397+0000 7f36f97fa700 1 -- 172.21.15.98:0/2432200103 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f36f401c030 con 0x7f36fc1609b0 2023-03-03T08:01:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.398+0000 7f36f97fa700 1 --2- 172.21.15.98:0/2432200103 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36e00684a0 0x7f36e006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.398+0000 7f36f97fa700 1 -- 172.21.15.98:0/2432200103 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f36f40862f0 con 0x7f36fc1609b0 2023-03-03T08:01:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.398+0000 7f3700f9b700 1 --2- 172.21.15.98:0/2432200103 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36e00684a0 0x7f36e006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:42.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.399+0000 7f3700f9b700 1 --2- 172.21.15.98:0/2432200103 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36e00684a0 0x7f36e006a950 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f36f0009820 tx=0x7f36f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:42.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.402+0000 7f36f97fa700 1 -- 172.21.15.98:0/2432200103 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f36f40529b0 con 0x7f36fc1609b0 2023-03-03T08:01:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.612+0000 7f3701f9d700 1 -- 172.21.15.98:0/2432200103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f36fc068290 con 0x7f36fc1609b0 2023-03-03T08:01:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.612+0000 7f36f97fa700 1 -- 172.21.15.98:0/2432200103 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v78) v1 ==== 80+0+0 (secure 0 0 0) 0x7f36f4055fd0 con 0x7f36fc1609b0 2023-03-03T08:01:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.615+0000 7f3701f9d700 1 -- 172.21.15.98:0/2432200103 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36e00684a0 msgr2=0x7f36e006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.615+0000 7f3701f9d700 1 --2- 172.21.15.98:0/2432200103 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36e00684a0 0x7f36e006a950 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f36f0009820 tx=0x7f36f0008040 comp rx=0 tx=0).stop 2023-03-03T08:01:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.615+0000 7f3701f9d700 1 -- 172.21.15.98:0/2432200103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc1609b0 msgr2=0x7f36fc0b5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.615+0000 7f3701f9d700 1 --2- 172.21.15.98:0/2432200103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc1609b0 0x7f36fc0b5e50 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f36f400cef0 tx=0x7f36f400a340 comp rx=0 tx=0).stop 2023-03-03T08:01:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.615+0000 7f3701f9d700 1 -- 172.21.15.98:0/2432200103 shutdown_connections 2023-03-03T08:01:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.615+0000 7f3701f9d700 1 --2- 172.21.15.98:0/2432200103 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f36e00684a0 0x7f36e006a950 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.615+0000 7f3701f9d700 1 --2- 172.21.15.98:0/2432200103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc1609b0 0x7f36fc0b5e50 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.615+0000 7f3701f9d700 1 --2- 172.21.15.98:0/2432200103 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36fc0cf570 0x7f36fc151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.615+0000 7f3701f9d700 1 --2- 172.21.15.98:0/2432200103 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36fc15db30 0x7f36fc0cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.615+0000 7f3701f9d700 1 -- 172.21.15.98:0/2432200103 >> 172.21.15.98:0/2432200103 conn(0x7f36fc06b880 msgr2=0x7f36fc0b4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.615+0000 7f3701f9d700 1 -- 172.21.15.98:0/2432200103 shutdown_connections 2023-03-03T08:01:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:42.615+0000 7f3701f9d700 1 -- 172.21.15.98:0/2432200103 wait complete. 2023-03-03T08:01:42.619 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T08:01:42.643 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2023-03-03T08:01:42.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:42.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-03-03T08:01:42.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[139873]: fuse finished with error 0 and tester_r 0 2023-03-03T08:01:42.746 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:01:42.829 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T08:01:42.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:42.830 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-03-03T08:01:42.900 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:01:42.900 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:01:42.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:01:43.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.215+0000 7f69109fe700 1 Processor -- start 2023-03-03T08:01:43.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.216+0000 7f69109fe700 1 -- start start 2023-03-03T08:01:43.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.217+0000 7f69109fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f690c0b51c0 0x7f690c160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:43.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.217+0000 7f69109fe700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f690c1614c0 con 0x7f690c146360 2023-03-03T08:01:43.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.217+0000 7f69109fe700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f690c0d10f0 con 0x7f690c0d1580 2023-03-03T08:01:43.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.217+0000 7f69109fe700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f690c0cdf20 con 0x7f690c0b51c0 2023-03-03T08:01:43.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.217+0000 7f690ad9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f690c146360 0x7f690c0b4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36680/0 (socket says 172.21.15.98:36680) 2023-03-03T08:01:43.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.217+0000 7f690ad9d700 1 -- 172.21.15.98:0/4204533218 learned_addr learned my addr 172.21.15.98:0/4204533218 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:43.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f690a59c700 1 --2- 172.21.15.98:0/4204533218 >> v2:172.21.15.100:3300/0 conn(0x7f690c0b51c0 0x7f690c160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:43.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f6909d9b700 1 -- 172.21.15.98:0/4204533218 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480897536 0 0) 0x7f68fc001110 con 0x7f690c146360 2023-03-03T08:01:43.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f6909d9b700 1 -- 172.21.15.98:0/4204533218 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68f40035b0 con 0x7f690c146360 2023-03-03T08:01:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f6909d9b700 1 -- 172.21.15.98:0/4204533218 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3657911343 0 0) 0x7f68f40035b0 con 0x7f690c146360 2023-03-03T08:01:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f6909d9b700 1 -- 172.21.15.98:0/4204533218 >> v2:172.21.15.100:3300/0 conn(0x7f690c0b51c0 msgr2=0x7f690c160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f6909d9b700 1 --2- 172.21.15.98:0/4204533218 >> v2:172.21.15.100:3300/0 conn(0x7f690c0b51c0 0x7f690c160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f6909d9b700 1 -- 172.21.15.98:0/4204533218 >> v1:172.21.15.100:6789/0 conn(0x7f690c0d1580 legacy=0x7f690c0d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f6909d9b700 1 -- 172.21.15.98:0/4204533218 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690c0cd680 con 0x7f690c146360 2023-03-03T08:01:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f6909d9b700 1 -- 172.21.15.98:0/4204533218 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f68fc001800 con 0x7f690c146360 2023-03-03T08:01:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f6909d9b700 1 -- 172.21.15.98:0/4204533218 >> v1:172.21.15.98:6789/0 conn(0x7f690c146360 legacy=0x7f690c0b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:43.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f6909d9b700 1 --2- 172.21.15.98:0/4204533218 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68f4003a00 0x7f68f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:43.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f6909d9b700 1 --2- 172.21.15.98:0/4204533218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f40043f0 0x7f68f40048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:43.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f6909d9b700 1 --2- 172.21.15.98:0/4204533218 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68f4005150 0x7f68f400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:43.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f6909d9b700 1 -- 172.21.15.98:0/4204533218 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0cdf20 con 0x7f68f40043f0 2023-03-03T08:01:43.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f6909d9b700 1 -- 172.21.15.98:0/4204533218 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f400c080 con 0x7f68f4003a00 2023-03-03T08:01:43.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f6909d9b700 1 -- 172.21.15.98:0/4204533218 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f400c240 con 0x7f68f4005150 2023-03-03T08:01:43.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.218+0000 7f690a59c700 1 --2- 172.21.15.98:0/4204533218 >> v2:172.21.15.100:3300/0 conn(0x7f690c0b51c0 0x7f690c160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:43.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.219+0000 7f690a59c700 1 --2- 172.21.15.98:0/4204533218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f40043f0 0x7f68f40048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:43.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.219+0000 7f690ad9d700 1 --2- 172.21.15.98:0/4204533218 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68f4003a00 0x7f68f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:43.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.219+0000 7f690b59e700 1 --2- 172.21.15.98:0/4204533218 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68f4005150 0x7f68f400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:43.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.219+0000 7f690a59c700 1 -- 172.21.15.98:0/4204533218 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68f4005150 msgr2=0x7f68f400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:43.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.219+0000 7f690a59c700 1 --2- 172.21.15.98:0/4204533218 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68f4005150 0x7f68f400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.219+0000 7f690a59c700 1 -- 172.21.15.98:0/4204533218 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68f4003a00 msgr2=0x7f68f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:43.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.219+0000 7f690a59c700 1 --2- 172.21.15.98:0/4204533218 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68f4003a00 0x7f68f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.219+0000 7f690a59c700 1 -- 172.21.15.98:0/4204533218 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f40035b0 con 0x7f68f40043f0 2023-03-03T08:01:43.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.219+0000 7f690b59e700 1 -- 172.21.15.98:0/4204533218 reap_dead start 2023-03-03T08:01:43.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.220+0000 7f690a59c700 1 --2- 172.21.15.98:0/4204533218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f40043f0 0x7f68f40048b0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f6900009ce0 tx=0x7f68f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2c5e4ab7150ae7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:43.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.220+0000 7f6909d9b700 1 -- 172.21.15.98:0/4204533218 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900004030 con 0x7f68f40043f0 2023-03-03T08:01:43.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.220+0000 7f6909d9b700 1 -- 172.21.15.98:0/4204533218 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6900004190 con 0x7f68f40043f0 2023-03-03T08:01:43.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.220+0000 7f6909d9b700 1 -- 172.21.15.98:0/4204533218 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900004520 con 0x7f68f40043f0 2023-03-03T08:01:43.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.220+0000 7f69109fe700 1 -- 172.21.15.98:0/4204533218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f40043f0 msgr2=0x7f68f40048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:43.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.220+0000 7f69109fe700 1 --2- 172.21.15.98:0/4204533218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f40043f0 0x7f68f40048b0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f6900009ce0 tx=0x7f68f4005880 comp rx=0 tx=0).stop 2023-03-03T08:01:43.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.221+0000 7f69109fe700 1 -- 172.21.15.98:0/4204533218 shutdown_connections 2023-03-03T08:01:43.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.221+0000 7f69109fe700 1 --2- 172.21.15.98:0/4204533218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f40043f0 0x7f68f40048b0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.221+0000 7f69109fe700 1 -- 172.21.15.98:0/4204533218 >> 172.21.15.98:0/4204533218 conn(0x7f690c06b8e0 msgr2=0x7f690c0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:43.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.221+0000 7f69109fe700 1 -- 172.21.15.98:0/4204533218 shutdown_connections 2023-03-03T08:01:43.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.221+0000 7f69109fe700 1 -- 172.21.15.98:0/4204533218 wait complete. 2023-03-03T08:01:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.223+0000 7f69109fe700 1 Processor -- start 2023-03-03T08:01:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.224+0000 7f69109fe700 1 -- start start 2023-03-03T08:01:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.225+0000 7f69109fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f40043f0 0x7f690c0b7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.225+0000 7f69109fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c0c6b60 0x7f690c0bb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:43.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.225+0000 7f69109fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f690c0c5b70 0x7f690c0bbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:43.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.225+0000 7f69109fe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0c7d40 con 0x7f68f40043f0 2023-03-03T08:01:43.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.225+0000 7f69109fe700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0c70e0 con 0x7f690c0c6b60 2023-03-03T08:01:43.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.225+0000 7f69109fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0c73a0 con 0x7f690c0c5b70 2023-03-03T08:01:43.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.225+0000 7f690ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f40043f0 0x7f690c0b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:43.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.226+0000 7f690ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f40043f0 0x7f690c0b7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49788/0 (socket says 172.21.15.98:49788) 2023-03-03T08:01:43.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.226+0000 7f690ad9d700 1 -- 172.21.15.98:0/1812881201 learned_addr learned my addr 172.21.15.98:0/1812881201 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:43.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.226+0000 7f690ad9d700 1 -- 172.21.15.98:0/1812881201 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f690c0c5b70 msgr2=0x7f690c0bbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:43.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.226+0000 7f690b59e700 1 --2- 172.21.15.98:0/1812881201 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f690c0c5b70 0x7f690c0bbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:43.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.226+0000 7f690ad9d700 1 --2- 172.21.15.98:0/1812881201 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f690c0c5b70 0x7f690c0bbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.226+0000 7f690ad9d700 1 -- 172.21.15.98:0/1812881201 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c0c6b60 msgr2=0x7f690c0bb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:43.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.226+0000 7f690a59c700 1 --2- 172.21.15.98:0/1812881201 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c0c6b60 0x7f690c0bb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:43.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.226+0000 7f690ad9d700 1 --2- 172.21.15.98:0/1812881201 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c0c6b60 0x7f690c0bb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.226+0000 7f690ad9d700 1 -- 172.21.15.98:0/1812881201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690c0cdc40 con 0x7f68f40043f0 2023-03-03T08:01:43.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.226+0000 7f690a59c700 1 --2- 172.21.15.98:0/1812881201 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c0c6b60 0x7f690c0bb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:43.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.226+0000 7f690ad9d700 1 --2- 172.21.15.98:0/1812881201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f40043f0 0x7f690c0b7140 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f68fc001e40 tx=0x7f68fc007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:43.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.226+0000 7f68f37fe700 1 -- 172.21.15.98:0/1812881201 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc00f640 con 0x7f68f40043f0 2023-03-03T08:01:43.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.227+0000 7f68f37fe700 1 -- 172.21.15.98:0/1812881201 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68fc00f7e0 con 0x7f68f40043f0 2023-03-03T08:01:43.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.227+0000 7f68f37fe700 1 -- 172.21.15.98:0/1812881201 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc00fb90 con 0x7f68f40043f0 2023-03-03T08:01:43.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.227+0000 7f69109fe700 1 -- 172.21.15.98:0/1812881201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f690c0cd960 con 0x7f68f40043f0 2023-03-03T08:01:43.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.227+0000 7f69109fe700 1 -- 172.21.15.98:0/1812881201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f690c0d0d80 con 0x7f68f40043f0 2023-03-03T08:01:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.228+0000 7f69109fe700 1 -- 172.21.15.98:0/1812881201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f690c074460 con 0x7f68f40043f0 2023-03-03T08:01:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.229+0000 7f68f37fe700 1 -- 172.21.15.98:0/1812881201 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f68fc00fcf0 con 0x7f68f40043f0 2023-03-03T08:01:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.230+0000 7f68f37fe700 1 --2- 172.21.15.98:0/1812881201 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f68f80684a0 0x7f68f806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.230+0000 7f68f37fe700 1 -- 172.21.15.98:0/1812881201 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f68fc085e20 con 0x7f68f40043f0 2023-03-03T08:01:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.231+0000 7f690a59c700 1 --2- 172.21.15.98:0/1812881201 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f68f80684a0 0x7f68f806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.231+0000 7f690a59c700 1 --2- 172.21.15.98:0/1812881201 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f68f80684a0 0x7f68f806a950 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f69000090b0 tx=0x7f6900008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.234+0000 7f68f37fe700 1 -- 172.21.15.98:0/1812881201 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f690c074460 con 0x7f68f40043f0 2023-03-03T08:01:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.429+0000 7f69109fe700 1 -- 172.21.15.98:0/1812881201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f690c1614c0 con 0x7f68f40043f0 2023-03-03T08:01:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.430+0000 7f68f37fe700 1 -- 172.21.15.98:0/1812881201 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+17982 (secure 0 0 0) 0x7f690c1614c0 con 0x7f68f40043f0 2023-03-03T08:01:43.432 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:43.432 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":78,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:01:22.383102+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:01:19.439394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-03-03T08:01:20.381345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:01:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.433+0000 7f69109fe700 1 -- 172.21.15.98:0/1812881201 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f68f80684a0 msgr2=0x7f68f806a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.433+0000 7f69109fe700 1 --2- 172.21.15.98:0/1812881201 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f68f80684a0 0x7f68f806a950 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f69000090b0 tx=0x7f6900008030 comp rx=0 tx=0).stop 2023-03-03T08:01:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.433+0000 7f69109fe700 1 -- 172.21.15.98:0/1812881201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f40043f0 msgr2=0x7f690c0b7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.433+0000 7f69109fe700 1 --2- 172.21.15.98:0/1812881201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f40043f0 0x7f690c0b7140 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f68fc001e40 tx=0x7f68fc007620 comp rx=0 tx=0).stop 2023-03-03T08:01:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.433+0000 7f69109fe700 1 -- 172.21.15.98:0/1812881201 shutdown_connections 2023-03-03T08:01:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.433+0000 7f69109fe700 1 --2- 172.21.15.98:0/1812881201 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f68f80684a0 0x7f68f806a950 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.433+0000 7f69109fe700 1 --2- 172.21.15.98:0/1812881201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f40043f0 0x7f690c0b7140 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.433+0000 7f69109fe700 1 --2- 172.21.15.98:0/1812881201 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f690c0c5b70 0x7f690c0bbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.433+0000 7f69109fe700 1 --2- 172.21.15.98:0/1812881201 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c0c6b60 0x7f690c0bb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.433+0000 7f69109fe700 1 -- 172.21.15.98:0/1812881201 >> 172.21.15.98:0/1812881201 conn(0x7f690c06b8e0 msgr2=0x7f690c0d2670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.433+0000 7f69109fe700 1 -- 172.21.15.98:0/1812881201 shutdown_connections 2023-03-03T08:01:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.434+0000 7f69109fe700 1 -- 172.21.15.98:0/1812881201 wait complete. 2023-03-03T08:01:43.462 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T08:01:19.439394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T08:01:20.381345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:01:43.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:01:43.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.711+0000 7ff098fff700 1 Processor -- start 2023-03-03T08:01:43.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.711+0000 7ff098fff700 1 -- start start 2023-03-03T08:01:43.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.713+0000 7ff098fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff0940b8d10 0x7ff0940cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:43.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.713+0000 7ff098fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0940b26d0 0x7ff0940b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:43.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.713+0000 7ff098fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0940cd7d0 con 0x7ff0940b5140 2023-03-03T08:01:43.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.713+0000 7ff098fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0940b55c0 con 0x7ff0940b8d10 2023-03-03T08:01:43.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.713+0000 7ff098fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0940b90e0 con 0x7ff0940b26d0 2023-03-03T08:01:43.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.713+0000 7ff092d9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff0940b5140 0x7ff094160520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36684/0 (socket says 172.21.15.98:36684) 2023-03-03T08:01:43.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.714+0000 7ff092d9d700 1 -- 172.21.15.98:0/1066138671 learned_addr learned my addr 172.21.15.98:0/1066138671 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:43.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.714+0000 7ff091d9b700 1 --2- 172.21.15.98:0/1066138671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0940b26d0 0x7ff0940b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:43.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.714+0000 7ff09259c700 1 --2- 172.21.15.98:0/1066138671 >> v2:172.21.15.100:3300/0 conn(0x7ff0940b8d10 0x7ff0940cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:43.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.714+0000 7ff09159a700 1 -- 172.21.15.98:0/1066138671 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412327283 0 0) 0x7ff0940cd7d0 con 0x7ff0940b5140 2023-03-03T08:01:43.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.714+0000 7ff09159a700 1 -- 172.21.15.98:0/1066138671 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff07c0035b0 con 0x7ff0940b5140 2023-03-03T08:01:43.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.714+0000 7ff09259c700 1 -- 172.21.15.98:0/1066138671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0940b26d0 msgr2=0x7ff0940b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:43.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.714+0000 7ff09259c700 1 --2- 172.21.15.98:0/1066138671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0940b26d0 0x7ff0940b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.714+0000 7ff09259c700 1 -- 172.21.15.98:0/1066138671 >> v1:172.21.15.98:6789/0 conn(0x7ff0940b5140 legacy=0x7ff094160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:43.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.714+0000 7ff09259c700 1 -- 172.21.15.98:0/1066138671 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0940cda20 con 0x7ff0940b8d10 2023-03-03T08:01:43.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.714+0000 7ff091d9b700 1 --2- 172.21.15.98:0/1066138671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0940b26d0 0x7ff0940b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:43.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.715+0000 7ff09259c700 1 --2- 172.21.15.98:0/1066138671 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0940b8d10 0x7ff0940cffc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff084009840 tx=0x7ff084009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba0de5f9e4b6344d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:43.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.715+0000 7ff09159a700 1 -- 172.21.15.98:0/1066138671 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084011830 con 0x7ff0940b8d10 2023-03-03T08:01:43.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.715+0000 7ff09159a700 1 -- 172.21.15.98:0/1066138671 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff084011990 con 0x7ff0940b8d10 2023-03-03T08:01:43.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.715+0000 7ff09159a700 1 -- 172.21.15.98:0/1066138671 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084011d20 con 0x7ff0940b8d10 2023-03-03T08:01:43.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.716+0000 7ff098fff700 1 -- 172.21.15.98:0/1066138671 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0940b8d10 msgr2=0x7ff0940cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:43.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.716+0000 7ff098fff700 1 --2- 172.21.15.98:0/1066138671 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0940b8d10 0x7ff0940cffc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff084009840 tx=0x7ff084009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:43.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.716+0000 7ff098fff700 1 -- 172.21.15.98:0/1066138671 shutdown_connections 2023-03-03T08:01:43.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.716+0000 7ff098fff700 1 --2- 172.21.15.98:0/1066138671 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0940b8d10 0x7ff0940cffc0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.716+0000 7ff098fff700 1 --2- 172.21.15.98:0/1066138671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0940b26d0 0x7ff0940b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.716+0000 7ff098fff700 1 -- 172.21.15.98:0/1066138671 >> 172.21.15.98:0/1066138671 conn(0x7ff094073e70 msgr2=0x7ff09406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:43.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.716+0000 7ff098fff700 1 -- 172.21.15.98:0/1066138671 shutdown_connections 2023-03-03T08:01:43.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.716+0000 7ff098fff700 1 -- 172.21.15.98:0/1066138671 wait complete. 2023-03-03T08:01:43.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.718+0000 7ff098fff700 1 Processor -- start 2023-03-03T08:01:43.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.718+0000 7ff098fff700 1 -- start start 2023-03-03T08:01:43.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff098fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0940b26d0 0x7ff0941f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:43.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff098fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0940b5140 0x7ff0941efa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:43.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff098fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0940b8d10 0x7ff0941f2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:43.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff098fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09415ee90 con 0x7ff0940b5140 2023-03-03T08:01:43.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff098fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09415e860 con 0x7ff0940b8d10 2023-03-03T08:01:43.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff098fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0941f30e0 con 0x7ff0940b26d0 2023-03-03T08:01:43.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff091d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0940b5140 0x7ff0941efa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:43.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff091d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0940b5140 0x7ff0941efa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49800/0 (socket says 172.21.15.98:49800) 2023-03-03T08:01:43.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff091d9b700 1 -- 172.21.15.98:0/3589432365 learned_addr learned my addr 172.21.15.98:0/3589432365 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:43.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff09259c700 1 --2- 172.21.15.98:0/3589432365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0940b26d0 0x7ff0941f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:43.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff092d9d700 1 --2- 172.21.15.98:0/3589432365 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0940b8d10 0x7ff0941f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:43.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff091d9b700 1 -- 172.21.15.98:0/3589432365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0940b26d0 msgr2=0x7ff0941f37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:43.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff091d9b700 1 --2- 172.21.15.98:0/3589432365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0940b26d0 0x7ff0941f37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff091d9b700 1 -- 172.21.15.98:0/3589432365 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0940b8d10 msgr2=0x7ff0941f2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:43.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff091d9b700 1 --2- 172.21.15.98:0/3589432365 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0940b8d10 0x7ff0941f2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.720+0000 7ff091d9b700 1 -- 172.21.15.98:0/3589432365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0880094a0 con 0x7ff0940b5140 2023-03-03T08:01:43.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.721+0000 7ff091d9b700 1 --2- 172.21.15.98:0/3589432365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0940b5140 0x7ff0941efa70 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff08800ebe0 tx=0x7ff088009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:43.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.721+0000 7ff082ffd700 1 -- 172.21.15.98:0/3589432365 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08800c8b0 con 0x7ff0940b5140 2023-03-03T08:01:43.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.721+0000 7ff082ffd700 1 -- 172.21.15.98:0/3589432365 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08800ca10 con 0x7ff0940b5140 2023-03-03T08:01:43.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.721+0000 7ff098fff700 1 -- 172.21.15.98:0/3589432365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0840094a0 con 0x7ff0940b5140 2023-03-03T08:01:43.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.721+0000 7ff082ffd700 1 -- 172.21.15.98:0/3589432365 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08800cda0 con 0x7ff0940b5140 2023-03-03T08:01:43.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.721+0000 7ff098fff700 1 -- 172.21.15.98:0/3589432365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0940d0e80 con 0x7ff0940b5140 2023-03-03T08:01:43.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.722+0000 7ff098fff700 1 -- 172.21.15.98:0/3589432365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0940cdd00 con 0x7ff0940b5140 2023-03-03T08:01:43.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.723+0000 7ff082ffd700 1 -- 172.21.15.98:0/3589432365 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7ff088019030 con 0x7ff0940b5140 2023-03-03T08:01:43.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.725+0000 7ff082ffd700 1 --2- 172.21.15.98:0/3589432365 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff0740684a0 0x7ff07406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:43.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.725+0000 7ff082ffd700 1 -- 172.21.15.98:0/3589432365 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff088085f00 con 0x7ff0940b5140 2023-03-03T08:01:43.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.725+0000 7ff09259c700 1 --2- 172.21.15.98:0/3589432365 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff0740684a0 0x7ff07406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:43.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.726+0000 7ff09259c700 1 --2- 172.21.15.98:0/3589432365 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff0740684a0 0x7ff07406a950 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff084000c00 tx=0x7ff084008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:43.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.729+0000 7ff082ffd700 1 -- 172.21.15.98:0/3589432365 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff0880525c0 con 0x7ff0940b5140 2023-03-03T08:01:43.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.920+0000 7ff098fff700 1 -- 172.21.15.98:0/3589432365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff0940cdfe0 con 0x7ff0940b5140 2023-03-03T08:01:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.921+0000 7ff082ffd700 1 -- 172.21.15.98:0/3589432365 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff088055be0 con 0x7ff0940b5140 2023-03-03T08:01:43.922 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:01:43.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.923+0000 7ff098fff700 1 -- 172.21.15.98:0/3589432365 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff0740684a0 msgr2=0x7ff07406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.923+0000 7ff098fff700 1 --2- 172.21.15.98:0/3589432365 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff0740684a0 0x7ff07406a950 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff084000c00 tx=0x7ff084008030 comp rx=0 tx=0).stop 2023-03-03T08:01:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.923+0000 7ff098fff700 1 -- 172.21.15.98:0/3589432365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0940b5140 msgr2=0x7ff0941efa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.923+0000 7ff098fff700 1 --2- 172.21.15.98:0/3589432365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0940b5140 0x7ff0941efa70 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff08800ebe0 tx=0x7ff088009e20 comp rx=0 tx=0).stop 2023-03-03T08:01:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.923+0000 7ff098fff700 1 -- 172.21.15.98:0/3589432365 shutdown_connections 2023-03-03T08:01:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.923+0000 7ff098fff700 1 --2- 172.21.15.98:0/3589432365 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7ff0740684a0 0x7ff07406a950 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.923+0000 7ff098fff700 1 --2- 172.21.15.98:0/3589432365 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0940b8d10 0x7ff0941f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.923+0000 7ff098fff700 1 --2- 172.21.15.98:0/3589432365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0940b26d0 0x7ff0941f37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.923+0000 7ff098fff700 1 --2- 172.21.15.98:0/3589432365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0940b5140 0x7ff0941efa70 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.923+0000 7ff098fff700 1 -- 172.21.15.98:0/3589432365 >> 172.21.15.98:0/3589432365 conn(0x7ff094073e70 msgr2=0x7ff0940b7670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.924+0000 7ff098fff700 1 -- 172.21.15.98:0/3589432365 shutdown_connections 2023-03-03T08:01:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:43.924+0000 7ff098fff700 1 -- 172.21.15.98:0/3589432365 wait complete. 2023-03-03T08:01:43.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:01:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.200+0000 7fd5847ff700 1 Processor -- start 2023-03-03T08:01:44.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.200+0000 7fd5847ff700 1 -- start start 2023-03-03T08:01:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.201+0000 7fd5847ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd57c0b8d10 0x7fd57c0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.202+0000 7fd5847ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd57c0b26d0 0x7fd57c0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.202+0000 7fd5847ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c0cd7d0 con 0x7fd57c0b5140 2023-03-03T08:01:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.202+0000 7fd5847ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57c0b55c0 con 0x7fd57c0b8d10 2023-03-03T08:01:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.202+0000 7fd5847ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57c0b90e0 con 0x7fd57c0b26d0 2023-03-03T08:01:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.202+0000 7fd58259b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd57c0b8d10 0x7fd57c0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.202+0000 7fd582d9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd57c0b5140 0x7fd57c160520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36692/0 (socket says 172.21.15.98:36692) 2023-03-03T08:01:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.202+0000 7fd58259b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd57c0b8d10 0x7fd57c0cffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49802/0 (socket says 172.21.15.98:49802) 2023-03-03T08:01:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.202+0000 7fd582d9c700 1 -- 172.21.15.98:0/2238159683 learned_addr learned my addr 172.21.15.98:0/2238159683 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.202+0000 7fd581d9a700 1 --2- 172.21.15.98:0/2238159683 >> v2:172.21.15.100:3300/0 conn(0x7fd57c0b26d0 0x7fd57c0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.202+0000 7fd58259b700 1 -- 172.21.15.98:0/2238159683 >> v2:172.21.15.100:3300/0 conn(0x7fd57c0b26d0 msgr2=0x7fd57c0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.202+0000 7fd58259b700 1 --2- 172.21.15.98:0/2238159683 >> v2:172.21.15.100:3300/0 conn(0x7fd57c0b26d0 0x7fd57c0b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.202+0000 7fd58259b700 1 -- 172.21.15.98:0/2238159683 >> v1:172.21.15.98:6789/0 conn(0x7fd57c0b5140 legacy=0x7fd57c160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.203+0000 7fd58259b700 1 -- 172.21.15.98:0/2238159683 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c0cda20 con 0x7fd57c0b8d10 2023-03-03T08:01:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.203+0000 7fd581599700 1 -- 172.21.15.98:0/2238159683 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023990384 0 0) 0x7fd57c0cd7d0 con 0x7fd57c0b5140 2023-03-03T08:01:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.203+0000 7fd58259b700 1 --2- 172.21.15.98:0/2238159683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd57c0b8d10 0x7fd57c0cffc0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd57c08bfa0 tx=0x7fd578009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551a5d19974453c3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.203+0000 7fd581599700 1 -- 172.21.15.98:0/2238159683 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578011830 con 0x7fd57c0b8d10 2023-03-03T08:01:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.203+0000 7fd581599700 1 -- 172.21.15.98:0/2238159683 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd578011990 con 0x7fd57c0b8d10 2023-03-03T08:01:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.203+0000 7fd581599700 1 -- 172.21.15.98:0/2238159683 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578011d20 con 0x7fd57c0b8d10 2023-03-03T08:01:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.204+0000 7fd5847ff700 1 -- 172.21.15.98:0/2238159683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd57c0b8d10 msgr2=0x7fd57c0cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.204+0000 7fd5847ff700 1 --2- 172.21.15.98:0/2238159683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd57c0b8d10 0x7fd57c0cffc0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd57c08bfa0 tx=0x7fd578009b20 comp rx=0 tx=0).stop 2023-03-03T08:01:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.204+0000 7fd5847ff700 1 -- 172.21.15.98:0/2238159683 shutdown_connections 2023-03-03T08:01:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.204+0000 7fd5847ff700 1 --2- 172.21.15.98:0/2238159683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd57c0b8d10 0x7fd57c0cffc0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.204+0000 7fd5847ff700 1 --2- 172.21.15.98:0/2238159683 >> v2:172.21.15.100:3300/0 conn(0x7fd57c0b26d0 0x7fd57c0b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.204+0000 7fd5847ff700 1 -- 172.21.15.98:0/2238159683 >> 172.21.15.98:0/2238159683 conn(0x7fd57c073e70 msgr2=0x7fd57c06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.204+0000 7fd5847ff700 1 -- 172.21.15.98:0/2238159683 shutdown_connections 2023-03-03T08:01:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.204+0000 7fd5847ff700 1 -- 172.21.15.98:0/2238159683 wait complete. 2023-03-03T08:01:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.206+0000 7fd5847ff700 1 Processor -- start 2023-03-03T08:01:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.206+0000 7fd5847ff700 1 -- start start 2023-03-03T08:01:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.207+0000 7fd5847ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd57c0b26d0 0x7fd57c15cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.207+0000 7fd5847ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd57c0b5140 0x7fd57c153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.207+0000 7fd5847ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd57c0b8d10 0x7fd57c15e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.207+0000 7fd5847ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c15f080 con 0x7fd57c0b26d0 2023-03-03T08:01:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.207+0000 7fd5847ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c15edc0 con 0x7fd57c0b8d10 2023-03-03T08:01:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.207+0000 7fd5847ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c15d680 con 0x7fd57c0b5140 2023-03-03T08:01:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.208+0000 7fd58259b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd57c0b26d0 0x7fd57c15cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.208+0000 7fd58259b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd57c0b26d0 0x7fd57c15cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49808/0 (socket says 172.21.15.98:49808) 2023-03-03T08:01:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.208+0000 7fd58259b700 1 -- 172.21.15.98:0/1854387565 learned_addr learned my addr 172.21.15.98:0/1854387565 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.208+0000 7fd58259b700 1 -- 172.21.15.98:0/1854387565 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd57c0b5140 msgr2=0x7fd57c153740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.208+0000 7fd58259b700 1 --2- 172.21.15.98:0/1854387565 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd57c0b5140 0x7fd57c153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.208+0000 7fd58259b700 1 -- 172.21.15.98:0/1854387565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd57c0b8d10 msgr2=0x7fd57c15e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.208+0000 7fd58259b700 1 --2- 172.21.15.98:0/1854387565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd57c0b8d10 0x7fd57c15e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.208+0000 7fd58259b700 1 -- 172.21.15.98:0/1854387565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5780094a0 con 0x7fd57c0b26d0 2023-03-03T08:01:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.208+0000 7fd58259b700 1 --2- 172.21.15.98:0/1854387565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd57c0b26d0 0x7fd57c15cff0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fd578009b60 tx=0x7fd578004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.209+0000 7fd572ffd700 1 -- 172.21.15.98:0/1854387565 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5780038a0 con 0x7fd57c0b26d0 2023-03-03T08:01:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.209+0000 7fd572ffd700 1 -- 172.21.15.98:0/1854387565 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57801e070 con 0x7fd57c0b26d0 2023-03-03T08:01:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.209+0000 7fd5847ff700 1 -- 172.21.15.98:0/1854387565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57c0d0e80 con 0x7fd57c0b26d0 2023-03-03T08:01:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.209+0000 7fd572ffd700 1 -- 172.21.15.98:0/1854387565 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578003c00 con 0x7fd57c0b26d0 2023-03-03T08:01:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.209+0000 7fd5847ff700 1 -- 172.21.15.98:0/1854387565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57c0cdd00 con 0x7fd57c0b26d0 2023-03-03T08:01:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.210+0000 7fd5847ff700 1 -- 172.21.15.98:0/1854387565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57c0cdfe0 con 0x7fd57c0b26d0 2023-03-03T08:01:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.211+0000 7fd572ffd700 1 -- 172.21.15.98:0/1854387565 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fd57800f050 con 0x7fd57c0b26d0 2023-03-03T08:01:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.213+0000 7fd572ffd700 1 --2- 172.21.15.98:0/1854387565 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5600684a0 0x7fd56006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.213+0000 7fd572ffd700 1 -- 172.21.15.98:0/1854387565 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd578086f40 con 0x7fd57c0b26d0 2023-03-03T08:01:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.213+0000 7fd581d9a700 1 --2- 172.21.15.98:0/1854387565 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5600684a0 0x7fd56006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.214+0000 7fd581d9a700 1 --2- 172.21.15.98:0/1854387565 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5600684a0 0x7fd56006a950 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd57c147a30 tx=0x7fd56c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:44.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.217+0000 7fd572ffd700 1 -- 172.21.15.98:0/1854387565 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd578053600 con 0x7fd57c0b26d0 2023-03-03T08:01:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.411+0000 7fd5847ff700 1 -- 172.21.15.98:0/1854387565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd57c06ce30 con 0x7fd57c0b26d0 2023-03-03T08:01:44.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.411+0000 7fd572ffd700 1 -- 172.21.15.98:0/1854387565 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v78) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd578020020 con 0x7fd57c0b26d0 2023-03-03T08:01:44.413 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:01:44.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.414+0000 7fd5847ff700 1 -- 172.21.15.98:0/1854387565 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5600684a0 msgr2=0x7fd56006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:44.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.414+0000 7fd5847ff700 1 --2- 172.21.15.98:0/1854387565 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5600684a0 0x7fd56006a950 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd57c147a30 tx=0x7fd56c008040 comp rx=0 tx=0).stop 2023-03-03T08:01:44.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.414+0000 7fd5847ff700 1 -- 172.21.15.98:0/1854387565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd57c0b26d0 msgr2=0x7fd57c15cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:44.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.414+0000 7fd5847ff700 1 --2- 172.21.15.98:0/1854387565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd57c0b26d0 0x7fd57c15cff0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fd578009b60 tx=0x7fd578004750 comp rx=0 tx=0).stop 2023-03-03T08:01:44.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.414+0000 7fd5847ff700 1 -- 172.21.15.98:0/1854387565 shutdown_connections 2023-03-03T08:01:44.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.414+0000 7fd5847ff700 1 --2- 172.21.15.98:0/1854387565 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd5600684a0 0x7fd56006a950 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.414+0000 7fd5847ff700 1 --2- 172.21.15.98:0/1854387565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd57c0b26d0 0x7fd57c15cff0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.414+0000 7fd5847ff700 1 --2- 172.21.15.98:0/1854387565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd57c0b8d10 0x7fd57c15e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.414+0000 7fd5847ff700 1 --2- 172.21.15.98:0/1854387565 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd57c0b5140 0x7fd57c153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.415+0000 7fd5847ff700 1 -- 172.21.15.98:0/1854387565 >> 172.21.15.98:0/1854387565 conn(0x7fd57c073e70 msgr2=0x7fd57c0b7220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:44.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.415+0000 7fd5847ff700 1 -- 172.21.15.98:0/1854387565 shutdown_connections 2023-03-03T08:01:44.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.415+0000 7fd5847ff700 1 -- 172.21.15.98:0/1854387565 wait complete. 2023-03-03T08:01:44.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T08:01:44.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.693+0000 7f4a4a5ff700 1 Processor -- start 2023-03-03T08:01:44.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.693+0000 7f4a4a5ff700 1 -- start start 2023-03-03T08:01:44.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.694+0000 7f4a4a5ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4a440b26d0 0x7f4a440b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:44.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.694+0000 7f4a4a5ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a440cd7d0 con 0x7f4a440b8d10 2023-03-03T08:01:44.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.694+0000 7f4a4a5ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a440b90e0 con 0x7f4a440b5140 2023-03-03T08:01:44.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.694+0000 7f4a4a5ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a440b55c0 con 0x7f4a440b26d0 2023-03-03T08:01:44.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.695+0000 7f4a43fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4a440b8d10 0x7f4a440cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36698/0 (socket says 172.21.15.98:36698) 2023-03-03T08:01:44.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.695+0000 7f4a43fff700 1 -- 172.21.15.98:0/1269973847 learned_addr learned my addr 172.21.15.98:0/1269973847 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:44.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.695+0000 7f4a437fe700 1 --2- 172.21.15.98:0/1269973847 >> v2:172.21.15.98:3300/0 conn(0x7f4a440b26d0 0x7f4a440b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:44.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.695+0000 7f4a42ffd700 1 -- 172.21.15.98:0/1269973847 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392529769 0 0) 0x7f4a440cd7d0 con 0x7f4a440b8d10 2023-03-03T08:01:44.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.695+0000 7f4a42ffd700 1 -- 172.21.15.98:0/1269973847 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a2c0035b0 con 0x7f4a440b8d10 2023-03-03T08:01:44.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.695+0000 7f4a437fe700 1 -- 172.21.15.98:0/1269973847 >> v1:172.21.15.100:6789/0 conn(0x7f4a440b5140 legacy=0x7f4a440d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:44.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.695+0000 7f4a437fe700 1 -- 172.21.15.98:0/1269973847 >> v1:172.21.15.98:6789/0 conn(0x7f4a440b8d10 legacy=0x7f4a440cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:44.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.695+0000 7f4a437fe700 1 -- 172.21.15.98:0/1269973847 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a440cda20 con 0x7f4a440b26d0 2023-03-03T08:01:44.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.696+0000 7f4a437fe700 1 --2- 172.21.15.98:0/1269973847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a440b26d0 0x7f4a440b4c00 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f4a38009840 tx=0x7f4a38009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e28aed6093bb4b0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:44.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.696+0000 7f4a42ffd700 1 -- 172.21.15.98:0/1269973847 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38011830 con 0x7f4a440b26d0 2023-03-03T08:01:44.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.696+0000 7f4a42ffd700 1 -- 172.21.15.98:0/1269973847 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a38011990 con 0x7f4a440b26d0 2023-03-03T08:01:44.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.696+0000 7f4a42ffd700 1 -- 172.21.15.98:0/1269973847 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38011d20 con 0x7f4a440b26d0 2023-03-03T08:01:44.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.696+0000 7f4a4a5ff700 1 -- 172.21.15.98:0/1269973847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a440b26d0 msgr2=0x7f4a440b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:44.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.696+0000 7f4a4a5ff700 1 --2- 172.21.15.98:0/1269973847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a440b26d0 0x7f4a440b4c00 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f4a38009840 tx=0x7f4a38009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:44.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.697+0000 7f4a4a5ff700 1 -- 172.21.15.98:0/1269973847 shutdown_connections 2023-03-03T08:01:44.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.697+0000 7f4a4a5ff700 1 --2- 172.21.15.98:0/1269973847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a440b26d0 0x7f4a440b4c00 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.697+0000 7f4a4a5ff700 1 -- 172.21.15.98:0/1269973847 >> 172.21.15.98:0/1269973847 conn(0x7f4a44073e70 msgr2=0x7f4a4406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:44.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.697+0000 7f4a4a5ff700 1 -- 172.21.15.98:0/1269973847 shutdown_connections 2023-03-03T08:01:44.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.697+0000 7f4a4a5ff700 1 -- 172.21.15.98:0/1269973847 wait complete. 2023-03-03T08:01:44.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.699+0000 7f4a4a5ff700 1 Processor -- start 2023-03-03T08:01:44.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.699+0000 7f4a4a5ff700 1 -- start start 2023-03-03T08:01:44.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.700+0000 7f4a4a5ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a440b26d0 0x7f4a440c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:44.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.700+0000 7f4a4a5ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a440b5140 0x7f4a44156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:44.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.700+0000 7f4a4a5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a440b8d10 0x7f4a44156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:44.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.700+0000 7f4a4a5ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a44152d50 con 0x7f4a440b26d0 2023-03-03T08:01:44.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.700+0000 7f4a4a5ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4415f6a0 con 0x7f4a440b5140 2023-03-03T08:01:44.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.700+0000 7f4a4a5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4415f980 con 0x7f4a440b8d10 2023-03-03T08:01:44.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.700+0000 7f4a43fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a440b26d0 0x7f4a440c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:44.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.700+0000 7f4a43fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a440b26d0 0x7f4a440c8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49830/0 (socket says 172.21.15.98:49830) 2023-03-03T08:01:44.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.700+0000 7f4a43fff700 1 -- 172.21.15.98:0/1234555849 learned_addr learned my addr 172.21.15.98:0/1234555849 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:44.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.701+0000 7f4a437fe700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a440b5140 0x7f4a44156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:44.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.701+0000 7f4a43fff700 1 -- 172.21.15.98:0/1234555849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a440b8d10 msgr2=0x7f4a44156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:44.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.701+0000 7f4a48b9c700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a440b8d10 0x7f4a44156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:44.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.701+0000 7f4a43fff700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a440b8d10 0x7f4a44156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.701+0000 7f4a43fff700 1 -- 172.21.15.98:0/1234555849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a440b5140 msgr2=0x7f4a44156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:44.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.701+0000 7f4a43fff700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a440b5140 0x7f4a44156670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.701+0000 7f4a43fff700 1 -- 172.21.15.98:0/1234555849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a380094a0 con 0x7f4a440b26d0 2023-03-03T08:01:44.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.701+0000 7f4a437fe700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a440b5140 0x7f4a44156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:44.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.701+0000 7f4a48b9c700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a440b8d10 0x7f4a44156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:44.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.701+0000 7f4a43fff700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a440b26d0 0x7f4a440c8f10 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f4a34001eb0 tx=0x7f4a3400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:44.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.701+0000 7f4a40ff9700 1 -- 172.21.15.98:0/1234555849 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a34005d70 con 0x7f4a440b26d0 2023-03-03T08:01:44.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.701+0000 7f4a40ff9700 1 -- 172.21.15.98:0/1234555849 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a340070d0 con 0x7f4a440b26d0 2023-03-03T08:01:44.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.701+0000 7f4a40ff9700 1 -- 172.21.15.98:0/1234555849 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a34007460 con 0x7f4a440b26d0 2023-03-03T08:01:44.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.701+0000 7f4a4a5ff700 1 -- 172.21.15.98:0/1234555849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a440d0e80 con 0x7f4a440b26d0 2023-03-03T08:01:44.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.702+0000 7f4a4a5ff700 1 -- 172.21.15.98:0/1234555849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a440cdd00 con 0x7f4a440b26d0 2023-03-03T08:01:44.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.703+0000 7f4a4a5ff700 1 -- 172.21.15.98:0/1234555849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a440cdfe0 con 0x7f4a440b26d0 2023-03-03T08:01:44.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.704+0000 7f4a40ff9700 1 -- 172.21.15.98:0/1234555849 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f4a34019030 con 0x7f4a440b26d0 2023-03-03T08:01:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.705+0000 7f4a40ff9700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4a300684a0 0x7f4a3006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.705+0000 7f4a40ff9700 1 -- 172.21.15.98:0/1234555849 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4a34085320 con 0x7f4a440b26d0 2023-03-03T08:01:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.705+0000 7f4a437fe700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4a300684a0 0x7f4a3006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.706+0000 7f4a437fe700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4a300684a0 0x7f4a3006a950 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4a44158690 tx=0x7f4a38008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.709+0000 7f4a40ff9700 1 -- 172.21.15.98:0/1234555849 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4a340519e0 con 0x7f4a440b26d0 2023-03-03T08:01:44.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.906+0000 7f4a4a5ff700 1 -- 172.21.15.98:0/1234555849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4a4404e6e0 con 0x7f4a440b26d0 2023-03-03T08:01:44.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.906+0000 7f4a40ff9700 1 -- 172.21.15.98:0/1234555849 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4a34055000 con 0x7f4a440b26d0 2023-03-03T08:01:44.908 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T08:01:44.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.909+0000 7f4a4a5ff700 1 -- 172.21.15.98:0/1234555849 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4a300684a0 msgr2=0x7f4a3006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:44.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.909+0000 7f4a4a5ff700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4a300684a0 0x7f4a3006a950 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4a44158690 tx=0x7f4a38008030 comp rx=0 tx=0).stop 2023-03-03T08:01:44.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.909+0000 7f4a4a5ff700 1 -- 172.21.15.98:0/1234555849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a440b26d0 msgr2=0x7f4a440c8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:44.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.909+0000 7f4a4a5ff700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a440b26d0 0x7f4a440c8f10 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f4a34001eb0 tx=0x7f4a3400c860 comp rx=0 tx=0).stop 2023-03-03T08:01:44.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.909+0000 7f4a4a5ff700 1 -- 172.21.15.98:0/1234555849 shutdown_connections 2023-03-03T08:01:44.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.909+0000 7f4a4a5ff700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f4a300684a0 0x7f4a3006a950 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.909+0000 7f4a4a5ff700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a440b26d0 0x7f4a440c8f10 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.909+0000 7f4a4a5ff700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4a440b8d10 0x7f4a44156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.909+0000 7f4a4a5ff700 1 --2- 172.21.15.98:0/1234555849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a440b5140 0x7f4a44156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:44.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.909+0000 7f4a4a5ff700 1 -- 172.21.15.98:0/1234555849 >> 172.21.15.98:0/1234555849 conn(0x7f4a44073e70 msgr2=0x7f4a440b31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:44.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.909+0000 7f4a4a5ff700 1 -- 172.21.15.98:0/1234555849 shutdown_connections 2023-03-03T08:01:44.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:44.909+0000 7f4a4a5ff700 1 -- 172.21.15.98:0/1234555849 wait complete. 2023-03-03T08:01:44.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T08:01:45.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.183+0000 7f3a657ff700 1 Processor -- start 2023-03-03T08:01:45.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.183+0000 7f3a657ff700 1 -- start start 2023-03-03T08:01:45.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.184+0000 7f3a657ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3a6015ca70 0x7f3a600cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:45.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.184+0000 7f3a657ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3a6006b4b0 0x7f3a600cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:45.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.185+0000 7f3a657ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a600cf980 con 0x7f3a600d00a0 2023-03-03T08:01:45.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.185+0000 7f3a657ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a600cd5b0 con 0x7f3a6015ca70 2023-03-03T08:01:45.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.185+0000 7f3a657ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a600d0540 con 0x7f3a6006b4b0 2023-03-03T08:01:45.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.185+0000 7f3a5effd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3a6015ca70 0x7f3a600cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:45.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.185+0000 7f3a5effd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3a6015ca70 0x7f3a600cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38578/0 (socket says 172.21.15.98:38578) 2023-03-03T08:01:45.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.185+0000 7f3a5effd700 1 -- 172.21.15.98:0/1643779589 learned_addr learned my addr 172.21.15.98:0/1643779589 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:45.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.185+0000 7f3a5e7fc700 1 --2- 172.21.15.98:0/1643779589 >> v2:172.21.15.100:3300/0 conn(0x7f3a6006b4b0 0x7f3a600cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.185+0000 7f3a5effd700 1 -- 172.21.15.98:0/1643779589 >> v2:172.21.15.100:3300/0 conn(0x7f3a6006b4b0 msgr2=0x7f3a600cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.185+0000 7f3a5effd700 1 --2- 172.21.15.98:0/1643779589 >> v2:172.21.15.100:3300/0 conn(0x7f3a6006b4b0 0x7f3a600cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.185+0000 7f3a5effd700 1 -- 172.21.15.98:0/1643779589 >> v1:172.21.15.100:6789/0 conn(0x7f3a600d00a0 legacy=0x7f3a600b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.185+0000 7f3a5effd700 1 -- 172.21.15.98:0/1643779589 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a600d2fd0 con 0x7f3a6015ca70 2023-03-03T08:01:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.186+0000 7f3a5effd700 1 --2- 172.21.15.98:0/1643779589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a6015ca70 0x7f3a600cf440 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f3a60160560 tx=0x7f3a50009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc12780d50c86a82 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.186+0000 7f3a5dffb700 1 -- 172.21.15.98:0/1643779589 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50004230 con 0x7f3a6015ca70 2023-03-03T08:01:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.186+0000 7f3a5dffb700 1 -- 172.21.15.98:0/1643779589 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a50004390 con 0x7f3a6015ca70 2023-03-03T08:01:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.186+0000 7f3a5dffb700 1 -- 172.21.15.98:0/1643779589 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a500036a0 con 0x7f3a6015ca70 2023-03-03T08:01:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.187+0000 7f3a657ff700 1 -- 172.21.15.98:0/1643779589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a6015ca70 msgr2=0x7f3a600cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.187+0000 7f3a657ff700 1 --2- 172.21.15.98:0/1643779589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a6015ca70 0x7f3a600cf440 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f3a60160560 tx=0x7f3a50009bd0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.187+0000 7f3a657ff700 1 -- 172.21.15.98:0/1643779589 shutdown_connections 2023-03-03T08:01:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.187+0000 7f3a657ff700 1 --2- 172.21.15.98:0/1643779589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a6015ca70 0x7f3a600cf440 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.187+0000 7f3a657ff700 1 --2- 172.21.15.98:0/1643779589 >> v2:172.21.15.100:3300/0 conn(0x7f3a6006b4b0 0x7f3a600cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.187+0000 7f3a657ff700 1 -- 172.21.15.98:0/1643779589 >> 172.21.15.98:0/1643779589 conn(0x7f3a60073e70 msgr2=0x7f3a6006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.187+0000 7f3a657ff700 1 -- 172.21.15.98:0/1643779589 shutdown_connections 2023-03-03T08:01:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.187+0000 7f3a657ff700 1 -- 172.21.15.98:0/1643779589 wait complete. 2023-03-03T08:01:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.189+0000 7f3a657ff700 1 Processor -- start 2023-03-03T08:01:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.189+0000 7f3a657ff700 1 -- start start 2023-03-03T08:01:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.190+0000 7f3a657ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a6006b4b0 0x7f3a60159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.190+0000 7f3a657ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a600d00a0 0x7f3a600cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.190+0000 7f3a657ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a6015ca70 0x7f3a600cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.190+0000 7f3a657ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a600cb2f0 con 0x7f3a6015ca70 2023-03-03T08:01:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.190+0000 7f3a657ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a600ca250 con 0x7f3a600d00a0 2023-03-03T08:01:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.190+0000 7f3a657ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a600ca530 con 0x7f3a6006b4b0 2023-03-03T08:01:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.191+0000 7f3a5f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a6015ca70 0x7f3a600cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.191+0000 7f3a5f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a6015ca70 0x7f3a600cc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38590/0 (socket says 172.21.15.98:38590) 2023-03-03T08:01:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.191+0000 7f3a5f7fe700 1 -- 172.21.15.98:0/972281072 learned_addr learned my addr 172.21.15.98:0/972281072 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.191+0000 7f3a5effd700 1 --2- 172.21.15.98:0/972281072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a6006b4b0 0x7f3a60159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.191+0000 7f3a5e7fc700 1 --2- 172.21.15.98:0/972281072 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a600d00a0 0x7f3a600cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.191+0000 7f3a5f7fe700 1 -- 172.21.15.98:0/972281072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a6006b4b0 msgr2=0x7f3a60159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.191+0000 7f3a5f7fe700 1 --2- 172.21.15.98:0/972281072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a6006b4b0 0x7f3a60159b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.191+0000 7f3a5f7fe700 1 -- 172.21.15.98:0/972281072 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a600d00a0 msgr2=0x7f3a600cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.191+0000 7f3a5f7fe700 1 --2- 172.21.15.98:0/972281072 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a600d00a0 0x7f3a600cbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.191+0000 7f3a5f7fe700 1 -- 172.21.15.98:0/972281072 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a50009550 con 0x7f3a6015ca70 2023-03-03T08:01:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.191+0000 7f3a5f7fe700 1 --2- 172.21.15.98:0/972281072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a6015ca70 0x7f3a600cc470 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f3a5800cc10 tx=0x7f3a5800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.192+0000 7f3a477fe700 1 -- 172.21.15.98:0/972281072 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5800abb0 con 0x7f3a6015ca70 2023-03-03T08:01:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.192+0000 7f3a477fe700 1 -- 172.21.15.98:0/972281072 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5800ad10 con 0x7f3a6015ca70 2023-03-03T08:01:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.192+0000 7f3a657ff700 1 -- 172.21.15.98:0/972281072 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a600cd900 con 0x7f3a6015ca70 2023-03-03T08:01:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.192+0000 7f3a477fe700 1 -- 172.21.15.98:0/972281072 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58012660 con 0x7f3a6015ca70 2023-03-03T08:01:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.192+0000 7f3a657ff700 1 -- 172.21.15.98:0/972281072 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a600d1800 con 0x7f3a6015ca70 2023-03-03T08:01:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.193+0000 7f3a657ff700 1 -- 172.21.15.98:0/972281072 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a600cdbe0 con 0x7f3a6015ca70 2023-03-03T08:01:45.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.194+0000 7f3a477fe700 1 -- 172.21.15.98:0/972281072 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f3a58022070 con 0x7f3a6015ca70 2023-03-03T08:01:45.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.195+0000 7f3a477fe700 1 --2- 172.21.15.98:0/972281072 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3a4c0684a0 0x7f3a4c06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:45.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.195+0000 7f3a477fe700 1 -- 172.21.15.98:0/972281072 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3a58086590 con 0x7f3a6015ca70 2023-03-03T08:01:45.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.196+0000 7f3a5effd700 1 --2- 172.21.15.98:0/972281072 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3a4c0684a0 0x7f3a4c06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.196+0000 7f3a5effd700 1 --2- 172.21.15.98:0/972281072 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3a4c0684a0 0x7f3a4c06a950 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009130 tx=0x7f3a50008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:45.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.200+0000 7f3a477fe700 1 -- 172.21.15.98:0/972281072 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3a58052c50 con 0x7f3a6015ca70 2023-03-03T08:01:45.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.396+0000 7f3a657ff700 1 -- 172.21.15.98:0/972281072 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3a6004e6e0 con 0x7f3a6015ca70 2023-03-03T08:01:45.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.396+0000 7f3a477fe700 1 -- 172.21.15.98:0/972281072 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3a580160a0 con 0x7f3a6015ca70 2023-03-03T08:01:45.398 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T08:01:45.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.399+0000 7f3a657ff700 1 -- 172.21.15.98:0/972281072 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3a4c0684a0 msgr2=0x7f3a4c06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:45.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.399+0000 7f3a657ff700 1 --2- 172.21.15.98:0/972281072 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3a4c0684a0 0x7f3a4c06a950 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009130 tx=0x7f3a50008030 comp rx=0 tx=0).stop 2023-03-03T08:01:45.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.399+0000 7f3a657ff700 1 -- 172.21.15.98:0/972281072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a6015ca70 msgr2=0x7f3a600cc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:45.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.399+0000 7f3a657ff700 1 --2- 172.21.15.98:0/972281072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a6015ca70 0x7f3a600cc470 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f3a5800cc10 tx=0x7f3a5800a060 comp rx=0 tx=0).stop 2023-03-03T08:01:45.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.399+0000 7f3a657ff700 1 -- 172.21.15.98:0/972281072 shutdown_connections 2023-03-03T08:01:45.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.399+0000 7f3a657ff700 1 --2- 172.21.15.98:0/972281072 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f3a4c0684a0 0x7f3a4c06a950 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.399+0000 7f3a657ff700 1 --2- 172.21.15.98:0/972281072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a6015ca70 0x7f3a600cc470 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.399+0000 7f3a657ff700 1 --2- 172.21.15.98:0/972281072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a6006b4b0 0x7f3a60159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.399+0000 7f3a657ff700 1 --2- 172.21.15.98:0/972281072 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a600d00a0 0x7f3a600cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.399+0000 7f3a657ff700 1 -- 172.21.15.98:0/972281072 >> 172.21.15.98:0/972281072 conn(0x7f3a60073e70 msgr2=0x7f3a601e52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.399+0000 7f3a657ff700 1 -- 172.21.15.98:0/972281072 shutdown_connections 2023-03-03T08:01:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.399+0000 7f3a657ff700 1 -- 172.21.15.98:0/972281072 wait complete. 2023-03-03T08:01:45.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T08:01:45.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.678+0000 7f0d717ff700 1 Processor -- start 2023-03-03T08:01:45.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.678+0000 7f0d717ff700 1 -- start start 2023-03-03T08:01:45.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.680+0000 7f0d717ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d6c15ca70 0x7f0d6c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:45.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.680+0000 7f0d717ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d6c06b4b0 0x7f0d6c0cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:45.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.680+0000 7f0d717ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0d6c0d00a0 0x7f0d6c0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:45.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.680+0000 7f0d717ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6c0cd5b0 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.680+0000 7f0d717ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6c0cf980 con 0x7f0d6c0d00a0 2023-03-03T08:01:45.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.680+0000 7f0d717ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c0d0510 con 0x7f0d6c15ca70 2023-03-03T08:01:45.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.680+0000 7f0d6a7fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d6c06b4b0 0x7f0d6c0cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:45.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.680+0000 7f0d6a7fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d6c06b4b0 0x7f0d6c0cfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38602/0 (socket says 172.21.15.98:38602) 2023-03-03T08:01:45.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.680+0000 7f0d6a7fc700 1 -- 172.21.15.98:0/1300807167 learned_addr learned my addr 172.21.15.98:0/1300807167 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:45.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.680+0000 7f0d6affd700 1 --2- 172.21.15.98:0/1300807167 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d6c15ca70 0x7f0d6c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:45.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.680+0000 7f0d6b7fe700 1 --2- 172.21.15.98:0/1300807167 >> v2:172.21.15.100:3300/0 conn(0x7f0d6c0d00a0 0x7f0d6c0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:45.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.680+0000 7f0d6a7fc700 1 -- 172.21.15.98:0/1300807167 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d6c15ca70 msgr2=0x7f0d6c0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:45.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.680+0000 7f0d6a7fc700 1 --2- 172.21.15.98:0/1300807167 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d6c15ca70 0x7f0d6c0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.680+0000 7f0d6a7fc700 1 -- 172.21.15.98:0/1300807167 >> v2:172.21.15.100:3300/0 conn(0x7f0d6c0d00a0 msgr2=0x7f0d6c0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:45.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.681+0000 7f0d6a7fc700 1 --2- 172.21.15.98:0/1300807167 >> v2:172.21.15.100:3300/0 conn(0x7f0d6c0d00a0 0x7f0d6c0b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.681+0000 7f0d6a7fc700 1 -- 172.21.15.98:0/1300807167 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c0d1800 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.681+0000 7f0d6a7fc700 1 --2- 172.21.15.98:0/1300807167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d6c06b4b0 0x7f0d6c0cfb60 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c08c450 tx=0x7f0d60009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5fa702521fddf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:45.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.681+0000 7f0d69ffb700 1 -- 172.21.15.98:0/1300807167 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6000b860 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.681+0000 7f0d69ffb700 1 -- 172.21.15.98:0/1300807167 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6000b9c0 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.681+0000 7f0d69ffb700 1 -- 172.21.15.98:0/1300807167 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6000bd50 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.682+0000 7f0d717ff700 1 -- 172.21.15.98:0/1300807167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d6c06b4b0 msgr2=0x7f0d6c0cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:45.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.682+0000 7f0d717ff700 1 --2- 172.21.15.98:0/1300807167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d6c06b4b0 0x7f0d6c0cfb60 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c08c450 tx=0x7f0d60009c00 comp rx=0 tx=0).stop 2023-03-03T08:01:45.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.682+0000 7f0d717ff700 1 -- 172.21.15.98:0/1300807167 shutdown_connections 2023-03-03T08:01:45.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.682+0000 7f0d717ff700 1 --2- 172.21.15.98:0/1300807167 >> v2:172.21.15.100:3300/0 conn(0x7f0d6c0d00a0 0x7f0d6c0b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.682+0000 7f0d717ff700 1 --2- 172.21.15.98:0/1300807167 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d6c15ca70 0x7f0d6c0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.682+0000 7f0d717ff700 1 --2- 172.21.15.98:0/1300807167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d6c06b4b0 0x7f0d6c0cfb60 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.682+0000 7f0d717ff700 1 -- 172.21.15.98:0/1300807167 >> 172.21.15.98:0/1300807167 conn(0x7f0d6c073e70 msgr2=0x7f0d6c06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:45.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.682+0000 7f0d717ff700 1 -- 172.21.15.98:0/1300807167 shutdown_connections 2023-03-03T08:01:45.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.682+0000 7f0d717ff700 1 -- 172.21.15.98:0/1300807167 wait complete. 2023-03-03T08:01:45.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.684+0000 7f0d717ff700 1 Processor -- start 2023-03-03T08:01:45.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.684+0000 7f0d717ff700 1 -- start start 2023-03-03T08:01:45.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.685+0000 7f0d717ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d6c06b4b0 0x7f0d6c15eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:45.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.685+0000 7f0d717ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d6c0d00a0 0x7f0d6c0cac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:45.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.685+0000 7f0d717ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d6c15ca70 0x7f0d6c15f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:45.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.685+0000 7f0d717ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c1530f0 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.685+0000 7f0d717ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c0cc310 con 0x7f0d6c0d00a0 2023-03-03T08:01:45.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.685+0000 7f0d717ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c159d60 con 0x7f0d6c15ca70 2023-03-03T08:01:45.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.686+0000 7f0d6affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d6c06b4b0 0x7f0d6c15eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:45.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.686+0000 7f0d6affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d6c06b4b0 0x7f0d6c15eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38612/0 (socket says 172.21.15.98:38612) 2023-03-03T08:01:45.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.686+0000 7f0d6affd700 1 -- 172.21.15.98:0/4248990054 learned_addr learned my addr 172.21.15.98:0/4248990054 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:45.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.686+0000 7f0d6a7fc700 1 --2- 172.21.15.98:0/4248990054 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d6c0d00a0 0x7f0d6c0cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:45.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.686+0000 7f0d6b7fe700 1 --2- 172.21.15.98:0/4248990054 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d6c15ca70 0x7f0d6c15f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:45.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.686+0000 7f0d6affd700 1 -- 172.21.15.98:0/4248990054 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d6c15ca70 msgr2=0x7f0d6c15f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:45.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.686+0000 7f0d6affd700 1 --2- 172.21.15.98:0/4248990054 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d6c15ca70 0x7f0d6c15f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.686+0000 7f0d6affd700 1 -- 172.21.15.98:0/4248990054 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d6c0d00a0 msgr2=0x7f0d6c0cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:45.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.686+0000 7f0d6affd700 1 --2- 172.21.15.98:0/4248990054 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d6c0d00a0 0x7f0d6c0cac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.686+0000 7f0d6affd700 1 -- 172.21.15.98:0/4248990054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d60009550 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.686+0000 7f0d6a7fc700 1 --2- 172.21.15.98:0/4248990054 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d6c0d00a0 0x7f0d6c0cac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:45.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.686+0000 7f0d6affd700 1 --2- 172.21.15.98:0/4248990054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d6c06b4b0 0x7f0d6c15eae0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c00d800 tx=0x7f0d5c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:45.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.687+0000 7f0d537fe700 1 -- 172.21.15.98:0/4248990054 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c0125f0 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.687+0000 7f0d537fe700 1 -- 172.21.15.98:0/4248990054 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5c012750 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.687+0000 7f0d717ff700 1 -- 172.21.15.98:0/4248990054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6c0cdbe0 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.687+0000 7f0d537fe700 1 -- 172.21.15.98:0/4248990054 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c012b00 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.687+0000 7f0d717ff700 1 -- 172.21.15.98:0/4248990054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6c0cd900 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.688+0000 7f0d717ff700 1 -- 172.21.15.98:0/4248990054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6c0d1aa0 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.689+0000 7f0d537fe700 1 -- 172.21.15.98:0/4248990054 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f0d5c012c60 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.690+0000 7f0d537fe700 1 --2- 172.21.15.98:0/4248990054 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0d580684a0 0x7f0d5806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:45.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.691+0000 7f0d537fe700 1 -- 172.21.15.98:0/4248990054 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0d5c086370 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.691+0000 7f0d6a7fc700 1 --2- 172.21.15.98:0/4248990054 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0d580684a0 0x7f0d5806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:45.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.692+0000 7f0d6a7fc700 1 --2- 172.21.15.98:0/4248990054 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0d580684a0 0x7f0d5806a950 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c08c450 tx=0x7f0d60008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:45.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.695+0000 7f0d537fe700 1 -- 172.21.15.98:0/4248990054 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0d5c052a30 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.906+0000 7f0d717ff700 1 -- 172.21.15.98:0/4248990054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0d6c04e6e0 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.906+0000 7f0d537fe700 1 -- 172.21.15.98:0/4248990054 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v78) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0d5c056050 con 0x7f0d6c06b4b0 2023-03-03T08:01:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.909+0000 7f0d717ff700 1 -- 172.21.15.98:0/4248990054 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0d580684a0 msgr2=0x7f0d5806a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.909+0000 7f0d717ff700 1 --2- 172.21.15.98:0/4248990054 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0d580684a0 0x7f0d5806a950 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c08c450 tx=0x7f0d60008030 comp rx=0 tx=0).stop 2023-03-03T08:01:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.909+0000 7f0d717ff700 1 -- 172.21.15.98:0/4248990054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d6c06b4b0 msgr2=0x7f0d6c15eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.909+0000 7f0d717ff700 1 --2- 172.21.15.98:0/4248990054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d6c06b4b0 0x7f0d6c15eae0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c00d800 tx=0x7f0d5c00dbc0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.909+0000 7f0d717ff700 1 -- 172.21.15.98:0/4248990054 shutdown_connections 2023-03-03T08:01:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.909+0000 7f0d717ff700 1 --2- 172.21.15.98:0/4248990054 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f0d580684a0 0x7f0d5806a950 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.909+0000 7f0d717ff700 1 --2- 172.21.15.98:0/4248990054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d6c06b4b0 0x7f0d6c15eae0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.909+0000 7f0d717ff700 1 --2- 172.21.15.98:0/4248990054 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d6c15ca70 0x7f0d6c15f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.909+0000 7f0d717ff700 1 --2- 172.21.15.98:0/4248990054 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d6c0d00a0 0x7f0d6c0cac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.909+0000 7f0d717ff700 1 -- 172.21.15.98:0/4248990054 >> 172.21.15.98:0/4248990054 conn(0x7f0d6c073e70 msgr2=0x7f0d6c0d2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.909+0000 7f0d717ff700 1 -- 172.21.15.98:0/4248990054 shutdown_connections 2023-03-03T08:01:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:45.909+0000 7f0d717ff700 1 -- 172.21.15.98:0/4248990054 wait complete. 2023-03-03T08:01:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T08:01:45.937 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:01:45.937 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:01:46.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:01:46.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.252+0000 7f42c03ff700 1 Processor -- start 2023-03-03T08:01:46.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.253+0000 7f42c03ff700 1 -- start start 2023-03-03T08:01:46.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.254+0000 7f42c03ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f42b80b87c0 0x7f42b80cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:46.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.254+0000 7f42c03ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b80d0ef0 con 0x7f42b80b8b90 2023-03-03T08:01:46.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.254+0000 7f42c03ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b80d0600 con 0x7f42b80b5410 2023-03-03T08:01:46.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.254+0000 7f42c03ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b80cd970 con 0x7f42b80b87c0 2023-03-03T08:01:46.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.254+0000 7f42be19b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f42b80b87c0 0x7f42b80cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:46.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.254+0000 7f42be19b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f42b80b87c0 0x7f42b80cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38618/0 (socket says 172.21.15.98:38618) 2023-03-03T08:01:46.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.254+0000 7f42be19b700 1 -- 172.21.15.98:0/3329420558 learned_addr learned my addr 172.21.15.98:0/3329420558 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:46.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.254+0000 7f42bd99a700 1 --1- 172.21.15.98:0/3329420558 >> v1:172.21.15.98:6789/0 conn(0x7f42b80b8b90 0x7f42b80b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33380/0 (socket says 172.21.15.98:33380) 2023-03-03T08:01:46.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.255+0000 7f42be19b700 1 -- 172.21.15.98:0/3329420558 >> v1:172.21.15.100:6789/0 conn(0x7f42b80b5410 legacy=0x7f42b80d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:46.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.255+0000 7f42be19b700 1 -- 172.21.15.98:0/3329420558 >> v1:172.21.15.98:6789/0 conn(0x7f42b80b8b90 legacy=0x7f42b80b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:46.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.255+0000 7f42be19b700 1 -- 172.21.15.98:0/3329420558 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b80cdc50 con 0x7f42b80b87c0 2023-03-03T08:01:46.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.255+0000 7f42bd199700 1 -- 172.21.15.98:0/3329420558 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796853127 0 0) 0x7f42b80d0ef0 con 0x7f42b80b8b90 2023-03-03T08:01:46.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.255+0000 7f42be19b700 1 --2- 172.21.15.98:0/3329420558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42b80b87c0 0x7f42b80cffe0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f42b40098f0 tx=0x7f42b4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d63553a4b53b08e9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:46.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.255+0000 7f42bd199700 1 -- 172.21.15.98:0/3329420558 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b400b8a0 con 0x7f42b80b87c0 2023-03-03T08:01:46.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.256+0000 7f42bd199700 1 -- 172.21.15.98:0/3329420558 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b400ba00 con 0x7f42b80b87c0 2023-03-03T08:01:46.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.256+0000 7f42bd199700 1 -- 172.21.15.98:0/3329420558 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b400bd90 con 0x7f42b80b87c0 2023-03-03T08:01:46.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.256+0000 7f42c03ff700 1 -- 172.21.15.98:0/3329420558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42b80b87c0 msgr2=0x7f42b80cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:46.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.256+0000 7f42c03ff700 1 --2- 172.21.15.98:0/3329420558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42b80b87c0 0x7f42b80cffe0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f42b40098f0 tx=0x7f42b4009c00 comp rx=0 tx=0).stop 2023-03-03T08:01:46.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.256+0000 7f42c03ff700 1 -- 172.21.15.98:0/3329420558 shutdown_connections 2023-03-03T08:01:46.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.256+0000 7f42c03ff700 1 --2- 172.21.15.98:0/3329420558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42b80b87c0 0x7f42b80cffe0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:46.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.256+0000 7f42c03ff700 1 -- 172.21.15.98:0/3329420558 >> 172.21.15.98:0/3329420558 conn(0x7f42b806aee0 msgr2=0x7f42b806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:46.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.256+0000 7f42c03ff700 1 -- 172.21.15.98:0/3329420558 shutdown_connections 2023-03-03T08:01:46.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.256+0000 7f42c03ff700 1 -- 172.21.15.98:0/3329420558 wait complete. 2023-03-03T08:01:46.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.259+0000 7f42c03ff700 1 Processor -- start 2023-03-03T08:01:46.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.259+0000 7f42c03ff700 1 -- start start 2023-03-03T08:01:46.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.260+0000 7f42c03ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42b80b5410 0x7f42b815f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:46.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.260+0000 7f42c03ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42b80b87c0 0x7f42b8152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:46.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.260+0000 7f42c03ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42b80b8b90 0x7f42b8153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:46.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.260+0000 7f42c03ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b8152ab0 con 0x7f42b80b5410 2023-03-03T08:01:46.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.260+0000 7f42c03ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b815f200 con 0x7f42b80b87c0 2023-03-03T08:01:46.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.260+0000 7f42c03ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b8154240 con 0x7f42b80b8b90 2023-03-03T08:01:46.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.260+0000 7f42be19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42b80b5410 0x7f42b815f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:46.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.260+0000 7f42be19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42b80b5410 0x7f42b815f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38622/0 (socket says 172.21.15.98:38622) 2023-03-03T08:01:46.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.260+0000 7f42be19b700 1 -- 172.21.15.98:0/4019837720 learned_addr learned my addr 172.21.15.98:0/4019837720 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:46.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.261+0000 7f42bd99a700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42b80b87c0 0x7f42b8152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:46.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.261+0000 7f42be19b700 1 -- 172.21.15.98:0/4019837720 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42b80b8b90 msgr2=0x7f42b8153d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:46.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.261+0000 7f42be99c700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42b80b8b90 0x7f42b8153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:46.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.261+0000 7f42be19b700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42b80b8b90 0x7f42b8153d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:46.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.261+0000 7f42be19b700 1 -- 172.21.15.98:0/4019837720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42b80b87c0 msgr2=0x7f42b8152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:46.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.261+0000 7f42be19b700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42b80b87c0 0x7f42b8152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:46.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.261+0000 7f42be19b700 1 -- 172.21.15.98:0/4019837720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b4009550 con 0x7f42b80b5410 2023-03-03T08:01:46.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.261+0000 7f42bd99a700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42b80b87c0 0x7f42b8152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:46.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.261+0000 7f42be99c700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42b80b8b90 0x7f42b8153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:46.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.261+0000 7f42be19b700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42b80b5410 0x7f42b815f690 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f42b4000c00 tx=0x7f42b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:46.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.261+0000 7f42aeffd700 1 -- 172.21.15.98:0/4019837720 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b4011a70 con 0x7f42b80b5410 2023-03-03T08:01:46.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.261+0000 7f42aeffd700 1 -- 172.21.15.98:0/4019837720 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b401f070 con 0x7f42b80b5410 2023-03-03T08:01:46.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.261+0000 7f42aeffd700 1 -- 172.21.15.98:0/4019837720 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b4011df0 con 0x7f42b80b5410 2023-03-03T08:01:46.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.261+0000 7f42c03ff700 1 -- 172.21.15.98:0/4019837720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b80cdf30 con 0x7f42b80b5410 2023-03-03T08:01:46.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.262+0000 7f42c03ff700 1 -- 172.21.15.98:0/4019837720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b8073a80 con 0x7f42b80b5410 2023-03-03T08:01:46.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.263+0000 7f42c03ff700 1 -- 172.21.15.98:0/4019837720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b80ce0c0 con 0x7f42b80b5410 2023-03-03T08:01:46.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.264+0000 7f42aeffd700 1 -- 172.21.15.98:0/4019837720 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f42b400f050 con 0x7f42b80b5410 2023-03-03T08:01:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.265+0000 7f42aeffd700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f429c0684a0 0x7f429c06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.265+0000 7f42aeffd700 1 -- 172.21.15.98:0/4019837720 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f42b4021070 con 0x7f42b80b5410 2023-03-03T08:01:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.265+0000 7f42bd99a700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f429c0684a0 0x7f429c06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.266+0000 7f42bd99a700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f429c0684a0 0x7f429c06a950 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f42b81600b0 tx=0x7f42a8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:46.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.269+0000 7f42aeffd700 1 -- 172.21.15.98:0/4019837720 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f42b40540c0 con 0x7f42b80b5410 2023-03-03T08:01:46.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.464+0000 7f42c03ff700 1 -- 172.21.15.98:0/4019837720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42b80729b0 con 0x7f42b80b5410 2023-03-03T08:01:46.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.465+0000 7f42aeffd700 1 -- 172.21.15.98:0/4019837720 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+17982 (secure 0 0 0) 0x7f42b40576e0 con 0x7f42b80b5410 2023-03-03T08:01:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:46.467 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":78,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:01:22.383102+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:01:19.439394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-03-03T08:01:20.381345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:01:46.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.468+0000 7f42c03ff700 1 -- 172.21.15.98:0/4019837720 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f429c0684a0 msgr2=0x7f429c06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:46.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.468+0000 7f42c03ff700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f429c0684a0 0x7f429c06a950 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f42b81600b0 tx=0x7f42a8009040 comp rx=0 tx=0).stop 2023-03-03T08:01:46.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.468+0000 7f42c03ff700 1 -- 172.21.15.98:0/4019837720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42b80b5410 msgr2=0x7f42b815f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:46.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.468+0000 7f42c03ff700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42b80b5410 0x7f42b815f690 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f42b4000c00 tx=0x7f42b4004750 comp rx=0 tx=0).stop 2023-03-03T08:01:46.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.468+0000 7f42c03ff700 1 -- 172.21.15.98:0/4019837720 shutdown_connections 2023-03-03T08:01:46.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.468+0000 7f42c03ff700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f429c0684a0 0x7f429c06a950 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:46.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.468+0000 7f42c03ff700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42b80b5410 0x7f42b815f690 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:46.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.468+0000 7f42c03ff700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42b80b8b90 0x7f42b8153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:46.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.468+0000 7f42c03ff700 1 --2- 172.21.15.98:0/4019837720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42b80b87c0 0x7f42b8152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:46.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.468+0000 7f42c03ff700 1 -- 172.21.15.98:0/4019837720 >> 172.21.15.98:0/4019837720 conn(0x7f42b806aee0 msgr2=0x7f42b80b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:46.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.468+0000 7f42c03ff700 1 -- 172.21.15.98:0/4019837720 shutdown_connections 2023-03-03T08:01:46.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.468+0000 7f42c03ff700 1 -- 172.21.15.98:0/4019837720 wait complete. 2023-03-03T08:01:46.498 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T08:01:19.439394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T08:01:20.381345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:01:46.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:01:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.750+0000 7f2df7fff700 1 Processor -- start 2023-03-03T08:01:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.751+0000 7f2df7fff700 1 -- start start 2023-03-03T08:01:46.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.752+0000 7f2df7fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2df80cf570 0x7f2df815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:46.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.752+0000 7f2df7fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df80cd780 con 0x7f2df81609b0 2023-03-03T08:01:46.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.752+0000 7f2df7fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df80cda60 con 0x7f2df815db30 2023-03-03T08:01:46.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.752+0000 7f2df7fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df80cebb0 con 0x7f2df80cf570 2023-03-03T08:01:46.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.752+0000 7f2df6ffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2df81609b0 0x7f2df8160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33396/0 (socket says 172.21.15.98:33396) 2023-03-03T08:01:46.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.752+0000 7f2df6ffd700 1 -- 172.21.15.98:0/948389607 learned_addr learned my addr 172.21.15.98:0/948389607 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:46.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.753+0000 7f2deffff700 1 --2- 172.21.15.98:0/948389607 >> v2:172.21.15.100:3300/0 conn(0x7f2df80cf570 0x7f2df815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:46.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.753+0000 7f2df67fc700 1 -- 172.21.15.98:0/948389607 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403964747 0 0) 0x7f2df80cd780 con 0x7f2df81609b0 2023-03-03T08:01:46.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.753+0000 7f2df67fc700 1 -- 172.21.15.98:0/948389607 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2de40035b0 con 0x7f2df81609b0 2023-03-03T08:01:46.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.753+0000 7f2df67fc700 1 -- 172.21.15.98:0/948389607 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078247110 0 0) 0x7f2df80cda60 con 0x7f2df815db30 2023-03-03T08:01:46.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.753+0000 7f2df67fc700 1 -- 172.21.15.98:0/948389607 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df80cd780 con 0x7f2df815db30 2023-03-03T08:01:46.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.753+0000 7f2df67fc700 1 -- 172.21.15.98:0/948389607 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4065634076 0 0) 0x7f2de40035b0 con 0x7f2df81609b0 2023-03-03T08:01:46.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.753+0000 7f2deffff700 1 -- 172.21.15.98:0/948389607 >> v1:172.21.15.100:6789/0 conn(0x7f2df815db30 legacy=0x7f2df80b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:46.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.753+0000 7f2deffff700 1 -- 172.21.15.98:0/948389607 >> v1:172.21.15.98:6789/0 conn(0x7f2df81609b0 legacy=0x7f2df8160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:46.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.753+0000 7f2deffff700 1 -- 172.21.15.98:0/948389607 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df80cdd40 con 0x7f2df80cf570 2023-03-03T08:01:46.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.754+0000 7f2deffff700 1 --2- 172.21.15.98:0/948389607 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2df80cf570 0x7f2df815d5f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2de0009840 tx=0x7f2de0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85101232aff76e21 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:46.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.754+0000 7f2df67fc700 1 -- 172.21.15.98:0/948389607 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8001670 con 0x7f2df80cf570 2023-03-03T08:01:46.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.754+0000 7f2df67fc700 1 -- 172.21.15.98:0/948389607 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de0004030 con 0x7f2df80cf570 2023-03-03T08:01:46.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.754+0000 7f2df67fc700 1 -- 172.21.15.98:0/948389607 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de00043c0 con 0x7f2df80cf570 2023-03-03T08:01:46.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.755+0000 7f2df7fff700 1 -- 172.21.15.98:0/948389607 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2df80cf570 msgr2=0x7f2df815d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:46.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.755+0000 7f2df7fff700 1 --2- 172.21.15.98:0/948389607 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2df80cf570 0x7f2df815d5f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2de0009840 tx=0x7f2de0009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:46.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.755+0000 7f2df7fff700 1 -- 172.21.15.98:0/948389607 shutdown_connections 2023-03-03T08:01:46.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.755+0000 7f2df7fff700 1 --2- 172.21.15.98:0/948389607 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2df80cf570 0x7f2df815d5f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:46.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.755+0000 7f2df7fff700 1 -- 172.21.15.98:0/948389607 >> 172.21.15.98:0/948389607 conn(0x7f2df806b880 msgr2=0x7f2df80cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:46.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.755+0000 7f2df7fff700 1 -- 172.21.15.98:0/948389607 shutdown_connections 2023-03-03T08:01:46.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.755+0000 7f2df7fff700 1 -- 172.21.15.98:0/948389607 wait complete. 2023-03-03T08:01:46.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.757+0000 7f2df7fff700 1 Processor -- start 2023-03-03T08:01:46.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.757+0000 7f2df7fff700 1 -- start start 2023-03-03T08:01:46.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.758+0000 7f2df7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df80cf570 0x7f2df80b8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:46.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.758+0000 7f2df7fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2df815db30 0x7f2df814f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:46.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.758+0000 7f2df7fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2df81609b0 0x7f2df80b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:46.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.758+0000 7f2df7fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df80b3b00 con 0x7f2df80cf570 2023-03-03T08:01:46.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.758+0000 7f2df7fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df814fb50 con 0x7f2df81609b0 2023-03-03T08:01:46.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.758+0000 7f2df7fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df80caa40 con 0x7f2df815db30 2023-03-03T08:01:46.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.758+0000 7f2df6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df80cf570 0x7f2df80b8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:46.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.759+0000 7f2df6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df80cf570 0x7f2df80b8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38628/0 (socket says 172.21.15.98:38628) 2023-03-03T08:01:46.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.759+0000 7f2df6ffd700 1 -- 172.21.15.98:0/3738311097 learned_addr learned my addr 172.21.15.98:0/3738311097 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:46.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.759+0000 7f2deffff700 1 --2- 172.21.15.98:0/3738311097 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2df815db30 0x7f2df814f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:46.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.759+0000 7f2df77fe700 1 --2- 172.21.15.98:0/3738311097 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2df81609b0 0x7f2df80b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:46.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.759+0000 7f2df6ffd700 1 -- 172.21.15.98:0/3738311097 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2df815db30 msgr2=0x7f2df814f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:46.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.759+0000 7f2df6ffd700 1 --2- 172.21.15.98:0/3738311097 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2df815db30 0x7f2df814f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:46.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.759+0000 7f2df6ffd700 1 -- 172.21.15.98:0/3738311097 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2df81609b0 msgr2=0x7f2df80b2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:46.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.759+0000 7f2df6ffd700 1 --2- 172.21.15.98:0/3738311097 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2df81609b0 0x7f2df80b2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:46.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.759+0000 7f2df6ffd700 1 -- 172.21.15.98:0/3738311097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de00094a0 con 0x7f2df80cf570 2023-03-03T08:01:46.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.759+0000 7f2df6ffd700 1 --2- 172.21.15.98:0/3738311097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df80cf570 0x7f2df80b8070 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f2de800c520 tx=0x7f2de800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:46.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.759+0000 7f2def7fe700 1 -- 172.21.15.98:0/3738311097 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8004c80 con 0x7f2df80cf570 2023-03-03T08:01:46.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.760+0000 7f2def7fe700 1 -- 172.21.15.98:0/3738311097 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de8004de0 con 0x7f2df80cf570 2023-03-03T08:01:46.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.760+0000 7f2df7fff700 1 -- 172.21.15.98:0/3738311097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df815d0d0 con 0x7f2df80cf570 2023-03-03T08:01:46.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.760+0000 7f2def7fe700 1 -- 172.21.15.98:0/3738311097 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8007310 con 0x7f2df80cf570 2023-03-03T08:01:46.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.760+0000 7f2df7fff700 1 -- 172.21.15.98:0/3738311097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df815cdf0 con 0x7f2df80cf570 2023-03-03T08:01:46.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.761+0000 7f2df7fff700 1 -- 172.21.15.98:0/3738311097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df80b51b0 con 0x7f2df80cf570 2023-03-03T08:01:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.762+0000 7f2def7fe700 1 -- 172.21.15.98:0/3738311097 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f2de801a020 con 0x7f2df80cf570 2023-03-03T08:01:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.763+0000 7f2def7fe700 1 --2- 172.21.15.98:0/3738311097 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2dd80684a0 0x7f2dd806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.763+0000 7f2def7fe700 1 -- 172.21.15.98:0/3738311097 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2de8085b80 con 0x7f2df80cf570 2023-03-03T08:01:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.764+0000 7f2deffff700 1 --2- 172.21.15.98:0/3738311097 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2dd80684a0 0x7f2dd806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:46.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.764+0000 7f2deffff700 1 --2- 172.21.15.98:0/3738311097 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2dd80684a0 0x7f2dd806a950 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2de00090b0 tx=0x7f2de0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.767+0000 7f2def7fe700 1 -- 172.21.15.98:0/3738311097 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2de8052230 con 0x7f2df80cf570 2023-03-03T08:01:46.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.962+0000 7f2df7fff700 1 -- 172.21.15.98:0/3738311097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2df80caec0 con 0x7f2df80cf570 2023-03-03T08:01:46.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.962+0000 7f2def7fe700 1 -- 172.21.15.98:0/3738311097 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2de8055850 con 0x7f2df80cf570 2023-03-03T08:01:46.964 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:01:46.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.965+0000 7f2df7fff700 1 -- 172.21.15.98:0/3738311097 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2dd80684a0 msgr2=0x7f2dd806a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:46.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.965+0000 7f2df7fff700 1 --2- 172.21.15.98:0/3738311097 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2dd80684a0 0x7f2dd806a950 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2de00090b0 tx=0x7f2de0008030 comp rx=0 tx=0).stop 2023-03-03T08:01:46.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.965+0000 7f2df7fff700 1 -- 172.21.15.98:0/3738311097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df80cf570 msgr2=0x7f2df80b8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.965+0000 7f2df7fff700 1 --2- 172.21.15.98:0/3738311097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df80cf570 0x7f2df80b8070 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f2de800c520 tx=0x7f2de800c8e0 comp rx=0 tx=0).stop 2023-03-03T08:01:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.965+0000 7f2df7fff700 1 -- 172.21.15.98:0/3738311097 shutdown_connections 2023-03-03T08:01:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.965+0000 7f2df7fff700 1 --2- 172.21.15.98:0/3738311097 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f2dd80684a0 0x7f2dd806a950 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.965+0000 7f2df7fff700 1 --2- 172.21.15.98:0/3738311097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df80cf570 0x7f2df80b8070 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.965+0000 7f2df7fff700 1 --2- 172.21.15.98:0/3738311097 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2df81609b0 0x7f2df80b2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.965+0000 7f2df7fff700 1 --2- 172.21.15.98:0/3738311097 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2df815db30 0x7f2df814f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:46.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.965+0000 7f2df7fff700 1 -- 172.21.15.98:0/3738311097 >> 172.21.15.98:0/3738311097 conn(0x7f2df806b880 msgr2=0x7f2df8152280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:46.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.965+0000 7f2df7fff700 1 -- 172.21.15.98:0/3738311097 shutdown_connections 2023-03-03T08:01:46.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:46.965+0000 7f2df7fff700 1 -- 172.21.15.98:0/3738311097 wait complete. 2023-03-03T08:01:46.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:01:47.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.242+0000 7fd872bff700 1 Processor -- start 2023-03-03T08:01:47.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.242+0000 7fd872bff700 1 -- start start 2023-03-03T08:01:47.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.243+0000 7fd872bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd86c15ca70 0x7fd86c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:47.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.244+0000 7fd872bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd86c0d0270 0x7fd86c0d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:47.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.244+0000 7fd872bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd86c0cf980 con 0x7fd86c06b4b0 2023-03-03T08:01:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.244+0000 7fd872bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86c0cd5b0 con 0x7fd86c15ca70 2023-03-03T08:01:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.244+0000 7fd872bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c0d1800 con 0x7fd86c0d0270 2023-03-03T08:01:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.244+0000 7fd87099b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd86c15ca70 0x7fd86c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.244+0000 7fd87099b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd86c15ca70 0x7fd86c0cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38638/0 (socket says 172.21.15.98:38638) 2023-03-03T08:01:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.244+0000 7fd87099b700 1 -- 172.21.15.98:0/879885534 learned_addr learned my addr 172.21.15.98:0/879885534 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.244+0000 7fd87119c700 1 --2- 172.21.15.98:0/879885534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd86c0d0270 0x7fd86c0d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:47.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.245+0000 7fd87099b700 1 -- 172.21.15.98:0/879885534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd86c0d0270 msgr2=0x7fd86c0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:47.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.245+0000 7fd87099b700 1 --2- 172.21.15.98:0/879885534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd86c0d0270 0x7fd86c0d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.245+0000 7fd87099b700 1 -- 172.21.15.98:0/879885534 >> v1:172.21.15.100:6789/0 conn(0x7fd86c06b4b0 legacy=0x7fd86c0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:47.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.245+0000 7fd87099b700 1 -- 172.21.15.98:0/879885534 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c0cd900 con 0x7fd86c15ca70 2023-03-03T08:01:47.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.245+0000 7fd87099b700 1 --2- 172.21.15.98:0/879885534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c15ca70 0x7fd86c0cf440 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fd8600098f0 tx=0x7fd860009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90fbbc76ba377e03 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:47.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.245+0000 7fd86b7fe700 1 -- 172.21.15.98:0/879885534 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860011830 con 0x7fd86c15ca70 2023-03-03T08:01:47.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.245+0000 7fd86b7fe700 1 -- 172.21.15.98:0/879885534 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd860011990 con 0x7fd86c15ca70 2023-03-03T08:01:47.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.245+0000 7fd86b7fe700 1 -- 172.21.15.98:0/879885534 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860011d20 con 0x7fd86c15ca70 2023-03-03T08:01:47.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.246+0000 7fd872bff700 1 -- 172.21.15.98:0/879885534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c15ca70 msgr2=0x7fd86c0cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:47.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.246+0000 7fd872bff700 1 --2- 172.21.15.98:0/879885534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c15ca70 0x7fd86c0cf440 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fd8600098f0 tx=0x7fd860009c00 comp rx=0 tx=0).stop 2023-03-03T08:01:47.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.246+0000 7fd872bff700 1 -- 172.21.15.98:0/879885534 shutdown_connections 2023-03-03T08:01:47.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.246+0000 7fd872bff700 1 --2- 172.21.15.98:0/879885534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c15ca70 0x7fd86c0cf440 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.246+0000 7fd872bff700 1 --2- 172.21.15.98:0/879885534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd86c0d0270 0x7fd86c0d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.246+0000 7fd872bff700 1 -- 172.21.15.98:0/879885534 >> 172.21.15.98:0/879885534 conn(0x7fd86c073e70 msgr2=0x7fd86c06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:47.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.246+0000 7fd872bff700 1 -- 172.21.15.98:0/879885534 shutdown_connections 2023-03-03T08:01:47.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.246+0000 7fd872bff700 1 -- 172.21.15.98:0/879885534 wait complete. 2023-03-03T08:01:47.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.248+0000 7fd872bff700 1 Processor -- start 2023-03-03T08:01:47.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.248+0000 7fd872bff700 1 -- start start 2023-03-03T08:01:47.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.249+0000 7fd872bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd86c06b4b0 0x7fd86c1e5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:47.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.249+0000 7fd872bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c0d0270 0x7fd86c0c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:47.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.249+0000 7fd872bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd86c15ca70 0x7fd86c0c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:47.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.249+0000 7fd872bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c0c6530 con 0x7fd86c0d0270 2023-03-03T08:01:47.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.249+0000 7fd872bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c0c87c0 con 0x7fd86c06b4b0 2023-03-03T08:01:47.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.249+0000 7fd872bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c0c7bb0 con 0x7fd86c15ca70 2023-03-03T08:01:47.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.249+0000 7fd86bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c0d0270 0x7fd86c0c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:47.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.249+0000 7fd86bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c0d0270 0x7fd86c0c5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38654/0 (socket says 172.21.15.98:38654) 2023-03-03T08:01:47.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.249+0000 7fd86bfff700 1 -- 172.21.15.98:0/3558838991 learned_addr learned my addr 172.21.15.98:0/3558838991 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:47.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.250+0000 7fd87099b700 1 --2- 172.21.15.98:0/3558838991 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd86c06b4b0 0x7fd86c1e5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:47.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.250+0000 7fd87119c700 1 --2- 172.21.15.98:0/3558838991 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd86c15ca70 0x7fd86c0c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:47.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.250+0000 7fd86bfff700 1 -- 172.21.15.98:0/3558838991 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd86c15ca70 msgr2=0x7fd86c0c5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:47.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.250+0000 7fd86bfff700 1 --2- 172.21.15.98:0/3558838991 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd86c15ca70 0x7fd86c0c5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.250+0000 7fd86bfff700 1 -- 172.21.15.98:0/3558838991 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd86c06b4b0 msgr2=0x7fd86c1e5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:47.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.250+0000 7fd86bfff700 1 --2- 172.21.15.98:0/3558838991 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd86c06b4b0 0x7fd86c1e5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.250+0000 7fd86bfff700 1 -- 172.21.15.98:0/3558838991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd860009550 con 0x7fd86c0d0270 2023-03-03T08:01:47.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.250+0000 7fd87119c700 1 --2- 172.21.15.98:0/3558838991 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd86c15ca70 0x7fd86c0c5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:47.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.250+0000 7fd86bfff700 1 --2- 172.21.15.98:0/3558838991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c0d0270 0x7fd86c0c5ab0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd85800c420 tx=0x7fd85800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:47.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.250+0000 7fd8697fa700 1 -- 172.21.15.98:0/3558838991 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8580052d0 con 0x7fd86c0d0270 2023-03-03T08:01:47.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.250+0000 7fd8697fa700 1 -- 172.21.15.98:0/3558838991 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd858005430 con 0x7fd86c0d0270 2023-03-03T08:01:47.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.250+0000 7fd8697fa700 1 -- 172.21.15.98:0/3558838991 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8580038c0 con 0x7fd86c0d0270 2023-03-03T08:01:47.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.250+0000 7fd872bff700 1 -- 172.21.15.98:0/3558838991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86c0d1aa0 con 0x7fd86c0d0270 2023-03-03T08:01:47.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.251+0000 7fd872bff700 1 -- 172.21.15.98:0/3558838991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86c0cdbe0 con 0x7fd86c0d0270 2023-03-03T08:01:47.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.252+0000 7fd872bff700 1 -- 172.21.15.98:0/3558838991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86c15ce40 con 0x7fd86c0d0270 2023-03-03T08:01:47.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.253+0000 7fd8697fa700 1 -- 172.21.15.98:0/3558838991 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fd858019030 con 0x7fd86c0d0270 2023-03-03T08:01:47.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.254+0000 7fd8697fa700 1 --2- 172.21.15.98:0/3558838991 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8500684a0 0x7fd85006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:47.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.254+0000 7fd8697fa700 1 -- 172.21.15.98:0/3558838991 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd8580851b0 con 0x7fd86c0d0270 2023-03-03T08:01:47.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.254+0000 7fd87099b700 1 --2- 172.21.15.98:0/3558838991 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8500684a0 0x7fd85006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:47.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.255+0000 7fd87099b700 1 --2- 172.21.15.98:0/3558838991 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8500684a0 0x7fd85006a950 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd860009c40 tx=0x7fd860008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:47.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.258+0000 7fd8697fa700 1 -- 172.21.15.98:0/3558838991 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd858051870 con 0x7fd86c0d0270 2023-03-03T08:01:47.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.449+0000 7fd872bff700 1 -- 172.21.15.98:0/3558838991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd86c04e6e0 con 0x7fd86c0d0270 2023-03-03T08:01:47.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.449+0000 7fd8697fa700 1 -- 172.21.15.98:0/3558838991 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v78) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd858054e90 con 0x7fd86c0d0270 2023-03-03T08:01:47.451 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:01:47.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.452+0000 7fd872bff700 1 -- 172.21.15.98:0/3558838991 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8500684a0 msgr2=0x7fd85006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.452+0000 7fd872bff700 1 --2- 172.21.15.98:0/3558838991 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8500684a0 0x7fd85006a950 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd860009c40 tx=0x7fd860008030 comp rx=0 tx=0).stop 2023-03-03T08:01:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.452+0000 7fd872bff700 1 -- 172.21.15.98:0/3558838991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c0d0270 msgr2=0x7fd86c0c5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.452+0000 7fd872bff700 1 --2- 172.21.15.98:0/3558838991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c0d0270 0x7fd86c0c5ab0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd85800c420 tx=0x7fd85800c7e0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.452+0000 7fd872bff700 1 -- 172.21.15.98:0/3558838991 shutdown_connections 2023-03-03T08:01:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.452+0000 7fd872bff700 1 --2- 172.21.15.98:0/3558838991 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fd8500684a0 0x7fd85006a950 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.452+0000 7fd872bff700 1 --2- 172.21.15.98:0/3558838991 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd86c15ca70 0x7fd86c0c5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.452+0000 7fd872bff700 1 --2- 172.21.15.98:0/3558838991 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd86c06b4b0 0x7fd86c1e5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.452+0000 7fd872bff700 1 --2- 172.21.15.98:0/3558838991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c0d0270 0x7fd86c0c5ab0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.452+0000 7fd872bff700 1 -- 172.21.15.98:0/3558838991 >> 172.21.15.98:0/3558838991 conn(0x7fd86c073e70 msgr2=0x7fd86c0b8100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:47.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.452+0000 7fd872bff700 1 -- 172.21.15.98:0/3558838991 shutdown_connections 2023-03-03T08:01:47.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.452+0000 7fd872bff700 1 -- 172.21.15.98:0/3558838991 wait complete. 2023-03-03T08:01:47.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T08:01:47.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.729+0000 7fb559dff700 1 Processor -- start 2023-03-03T08:01:47.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.729+0000 7fb559dff700 1 -- start start 2023-03-03T08:01:47.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.730+0000 7fb559dff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb55415ca70 0x7fb5540cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:47.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.730+0000 7fb559dff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5540d0270 0x7fb5540d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:47.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.730+0000 7fb559dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5540cf980 con 0x7fb55406b4b0 2023-03-03T08:01:47.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.730+0000 7fb559dff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5540cd5b0 con 0x7fb5540d0270 2023-03-03T08:01:47.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.730+0000 7fb559dff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5540d1800 con 0x7fb55415ca70 2023-03-03T08:01:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.731+0000 7fb552ffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb55406b4b0 0x7fb5540cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33402/0 (socket says 172.21.15.98:33402) 2023-03-03T08:01:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.731+0000 7fb552ffd700 1 -- 172.21.15.98:0/3250257555 learned_addr learned my addr 172.21.15.98:0/3250257555 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.731+0000 7fb553fff700 1 --2- 172.21.15.98:0/3250257555 >> v2:172.21.15.98:3300/0 conn(0x7fb5540d0270 0x7fb5540d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.731+0000 7fb5537fe700 1 --2- 172.21.15.98:0/3250257555 >> v2:172.21.15.100:3300/0 conn(0x7fb55415ca70 0x7fb5540cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.731+0000 7fb553fff700 1 -- 172.21.15.98:0/3250257555 >> v2:172.21.15.100:3300/0 conn(0x7fb55415ca70 msgr2=0x7fb5540cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.731+0000 7fb553fff700 1 --2- 172.21.15.98:0/3250257555 >> v2:172.21.15.100:3300/0 conn(0x7fb55415ca70 0x7fb5540cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.731+0000 7fb553fff700 1 -- 172.21.15.98:0/3250257555 >> v1:172.21.15.98:6789/0 conn(0x7fb55406b4b0 legacy=0x7fb5540cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.731+0000 7fb553fff700 1 -- 172.21.15.98:0/3250257555 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5540cd900 con 0x7fb5540d0270 2023-03-03T08:01:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.731+0000 7fb5527fc700 1 -- 172.21.15.98:0/3250257555 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614480259 0 0) 0x7fb5540cf980 con 0x7fb55406b4b0 2023-03-03T08:01:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.732+0000 7fb553fff700 1 --2- 172.21.15.98:0/3250257555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5540d0270 0x7fb5540d1fc0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fb55406c2c0 tx=0x7fb54c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5386ba3f16416f7c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.732+0000 7fb5527fc700 1 -- 172.21.15.98:0/3250257555 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c00e070 con 0x7fb5540d0270 2023-03-03T08:01:47.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.732+0000 7fb5527fc700 1 -- 172.21.15.98:0/3250257555 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54c00bb20 con 0x7fb5540d0270 2023-03-03T08:01:47.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.732+0000 7fb5527fc700 1 -- 172.21.15.98:0/3250257555 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c012430 con 0x7fb5540d0270 2023-03-03T08:01:47.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.732+0000 7fb559dff700 1 -- 172.21.15.98:0/3250257555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5540d0270 msgr2=0x7fb5540d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:47.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.732+0000 7fb559dff700 1 --2- 172.21.15.98:0/3250257555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5540d0270 0x7fb5540d1fc0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fb55406c2c0 tx=0x7fb54c00b330 comp rx=0 tx=0).stop 2023-03-03T08:01:47.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.732+0000 7fb559dff700 1 -- 172.21.15.98:0/3250257555 shutdown_connections 2023-03-03T08:01:47.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.732+0000 7fb559dff700 1 --2- 172.21.15.98:0/3250257555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5540d0270 0x7fb5540d1fc0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.732+0000 7fb559dff700 1 --2- 172.21.15.98:0/3250257555 >> v2:172.21.15.100:3300/0 conn(0x7fb55415ca70 0x7fb5540cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.732+0000 7fb559dff700 1 -- 172.21.15.98:0/3250257555 >> 172.21.15.98:0/3250257555 conn(0x7fb554073e70 msgr2=0x7fb55406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:47.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.733+0000 7fb559dff700 1 -- 172.21.15.98:0/3250257555 shutdown_connections 2023-03-03T08:01:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.733+0000 7fb559dff700 1 -- 172.21.15.98:0/3250257555 wait complete. 2023-03-03T08:01:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.735+0000 7fb559dff700 1 Processor -- start 2023-03-03T08:01:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.735+0000 7fb559dff700 1 -- start start 2023-03-03T08:01:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.736+0000 7fb559dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55406b4b0 0x7fb55414e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.736+0000 7fb559dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5540d0270 0x7fb5540cc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.736+0000 7fb559dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb55415ca70 0x7fb554155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.736+0000 7fb559dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540b5a40 con 0x7fb55406b4b0 2023-03-03T08:01:47.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.736+0000 7fb559dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540ca380 con 0x7fb55415ca70 2023-03-03T08:01:47.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.736+0000 7fb559dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540cc9a0 con 0x7fb5540d0270 2023-03-03T08:01:47.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.736+0000 7fb5537fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55406b4b0 0x7fb55414e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:47.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.736+0000 7fb5537fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55406b4b0 0x7fb55414e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38676/0 (socket says 172.21.15.98:38676) 2023-03-03T08:01:47.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.736+0000 7fb5537fe700 1 -- 172.21.15.98:0/86317348 learned_addr learned my addr 172.21.15.98:0/86317348 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:47.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.737+0000 7fb552ffd700 1 --2- 172.21.15.98:0/86317348 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5540d0270 0x7fb5540cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:47.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.737+0000 7fb553fff700 1 --2- 172.21.15.98:0/86317348 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb55415ca70 0x7fb554155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:47.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.737+0000 7fb5537fe700 1 -- 172.21.15.98:0/86317348 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5540d0270 msgr2=0x7fb5540cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:47.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.737+0000 7fb5537fe700 1 --2- 172.21.15.98:0/86317348 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5540d0270 0x7fb5540cc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.737+0000 7fb5537fe700 1 -- 172.21.15.98:0/86317348 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb55415ca70 msgr2=0x7fb554155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:47.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.737+0000 7fb5537fe700 1 --2- 172.21.15.98:0/86317348 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb55415ca70 0x7fb554155cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.737+0000 7fb5537fe700 1 -- 172.21.15.98:0/86317348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c009a70 con 0x7fb55406b4b0 2023-03-03T08:01:47.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.737+0000 7fb5537fe700 1 --2- 172.21.15.98:0/86317348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55406b4b0 0x7fb55414e4b0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fb54400e9b0 tx=0x7fb54400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:47.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.737+0000 7fb53bfff700 1 -- 172.21.15.98:0/86317348 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544009710 con 0x7fb55406b4b0 2023-03-03T08:01:47.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.738+0000 7fb53bfff700 1 -- 172.21.15.98:0/86317348 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54400c850 con 0x7fb55406b4b0 2023-03-03T08:01:47.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.738+0000 7fb559dff700 1 -- 172.21.15.98:0/86317348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5540d1aa0 con 0x7fb55406b4b0 2023-03-03T08:01:47.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.738+0000 7fb53bfff700 1 -- 172.21.15.98:0/86317348 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54400cbc0 con 0x7fb55406b4b0 2023-03-03T08:01:47.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.738+0000 7fb559dff700 1 -- 172.21.15.98:0/86317348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5540cdbe0 con 0x7fb55406b4b0 2023-03-03T08:01:47.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.739+0000 7fb559dff700 1 -- 172.21.15.98:0/86317348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55415ce40 con 0x7fb55406b4b0 2023-03-03T08:01:47.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.740+0000 7fb53bfff700 1 -- 172.21.15.98:0/86317348 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fb544019030 con 0x7fb55406b4b0 2023-03-03T08:01:47.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.741+0000 7fb53bfff700 1 --2- 172.21.15.98:0/86317348 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb5400684a0 0x7fb54006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:47.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.741+0000 7fb53bfff700 1 -- 172.21.15.98:0/86317348 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb544085f60 con 0x7fb55406b4b0 2023-03-03T08:01:47.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.742+0000 7fb552ffd700 1 --2- 172.21.15.98:0/86317348 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb5400684a0 0x7fb54006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.742+0000 7fb552ffd700 1 --2- 172.21.15.98:0/86317348 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb5400684a0 0x7fb54006a950 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb5541526b0 tx=0x7fb548007c90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.746+0000 7fb53bfff700 1 -- 172.21.15.98:0/86317348 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb544056050 con 0x7fb55406b4b0 2023-03-03T08:01:47.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.937+0000 7fb559dff700 1 -- 172.21.15.98:0/86317348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb5540ca620 con 0x7fb55406b4b0 2023-03-03T08:01:47.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.938+0000 7fb53bfff700 1 -- 172.21.15.98:0/86317348 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb544017020 con 0x7fb55406b4b0 2023-03-03T08:01:47.940 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T08:01:47.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.941+0000 7fb559dff700 1 -- 172.21.15.98:0/86317348 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb5400684a0 msgr2=0x7fb54006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:47.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.941+0000 7fb559dff700 1 --2- 172.21.15.98:0/86317348 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb5400684a0 0x7fb54006a950 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb5541526b0 tx=0x7fb548007c90 comp rx=0 tx=0).stop 2023-03-03T08:01:47.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.941+0000 7fb559dff700 1 -- 172.21.15.98:0/86317348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55406b4b0 msgr2=0x7fb55414e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.941+0000 7fb559dff700 1 --2- 172.21.15.98:0/86317348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55406b4b0 0x7fb55414e4b0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fb54400e9b0 tx=0x7fb54400ed70 comp rx=0 tx=0).stop 2023-03-03T08:01:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.941+0000 7fb559dff700 1 -- 172.21.15.98:0/86317348 shutdown_connections 2023-03-03T08:01:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.941+0000 7fb559dff700 1 --2- 172.21.15.98:0/86317348 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fb5400684a0 0x7fb54006a950 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.941+0000 7fb559dff700 1 --2- 172.21.15.98:0/86317348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55406b4b0 0x7fb55414e4b0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.941+0000 7fb559dff700 1 --2- 172.21.15.98:0/86317348 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb55415ca70 0x7fb554155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.941+0000 7fb559dff700 1 --2- 172.21.15.98:0/86317348 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5540d0270 0x7fb5540cc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.941+0000 7fb559dff700 1 -- 172.21.15.98:0/86317348 >> 172.21.15.98:0/86317348 conn(0x7fb554073e70 msgr2=0x7fb5540b8050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.941+0000 7fb559dff700 1 -- 172.21.15.98:0/86317348 shutdown_connections 2023-03-03T08:01:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:47.941+0000 7fb559dff700 1 -- 172.21.15.98:0/86317348 wait complete. 2023-03-03T08:01:47.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T08:01:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.215+0000 7fc5acd9d700 1 Processor -- start 2023-03-03T08:01:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.216+0000 7fc5acd9d700 1 -- start start 2023-03-03T08:01:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.217+0000 7fc5acd9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5a80b4aa0 0x7fc5a80b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.217+0000 7fc5acd9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a815dd00 con 0x7fc5a81609b0 2023-03-03T08:01:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.217+0000 7fc5acd9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a80cd780 con 0x7fc5a80cf570 2023-03-03T08:01:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.217+0000 7fc5acd9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5a80cebb0 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.217+0000 7fc5a77fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc5a81609b0 0x7fc5a8160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33414/0 (socket says 172.21.15.98:33414) 2023-03-03T08:01:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.217+0000 7fc5a77fe700 1 -- 172.21.15.98:0/3064841917 learned_addr learned my addr 172.21.15.98:0/3064841917 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.217+0000 7fc5a7fff700 1 --2- 172.21.15.98:0/3064841917 >> v2:172.21.15.98:3300/0 conn(0x7fc5a80b4aa0 0x7fc5a80b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.218+0000 7fc5a7fff700 1 -- 172.21.15.98:0/3064841917 >> v1:172.21.15.100:6789/0 conn(0x7fc5a80cf570 legacy=0x7fc5a815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.218+0000 7fc5a7fff700 1 -- 172.21.15.98:0/3064841917 >> v1:172.21.15.98:6789/0 conn(0x7fc5a81609b0 legacy=0x7fc5a8160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.218+0000 7fc5a7fff700 1 -- 172.21.15.98:0/3064841917 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a80cda60 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.218+0000 7fc5a7fff700 1 --2- 172.21.15.98:0/3064841917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a80b4aa0 0x7fc5a80b4f20 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0009d90 tx=0x7fc5a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f8998fae74e525 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.218+0000 7fc5a67fc700 1 -- 172.21.15.98:0/3064841917 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a000e070 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.218+0000 7fc5a67fc700 1 -- 172.21.15.98:0/3064841917 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a000bbd0 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.218+0000 7fc5a67fc700 1 -- 172.21.15.98:0/3064841917 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0012430 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.219+0000 7fc5acd9d700 1 -- 172.21.15.98:0/3064841917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a80b4aa0 msgr2=0x7fc5a80b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.219+0000 7fc5acd9d700 1 --2- 172.21.15.98:0/3064841917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a80b4aa0 0x7fc5a80b4f20 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0009d90 tx=0x7fc5a000b330 comp rx=0 tx=0).stop 2023-03-03T08:01:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.219+0000 7fc5acd9d700 1 -- 172.21.15.98:0/3064841917 shutdown_connections 2023-03-03T08:01:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.219+0000 7fc5acd9d700 1 --2- 172.21.15.98:0/3064841917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a80b4aa0 0x7fc5a80b4f20 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.219+0000 7fc5acd9d700 1 -- 172.21.15.98:0/3064841917 >> 172.21.15.98:0/3064841917 conn(0x7fc5a806b880 msgr2=0x7fc5a80cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.219+0000 7fc5acd9d700 1 -- 172.21.15.98:0/3064841917 shutdown_connections 2023-03-03T08:01:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.219+0000 7fc5acd9d700 1 -- 172.21.15.98:0/3064841917 wait complete. 2023-03-03T08:01:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.222+0000 7fc5acd9d700 1 Processor -- start 2023-03-03T08:01:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.222+0000 7fc5acd9d700 1 -- start start 2023-03-03T08:01:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5acd9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a80b4aa0 0x7fc5a80d3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5acd9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5a80cf570 0x7fc5a80cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5acd9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a81609b0 0x7fc5a80cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5acd9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a80cbcb0 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5acd9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a80d23d0 con 0x7fc5a80cf570 2023-03-03T08:01:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5acd9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a80cb310 con 0x7fc5a81609b0 2023-03-03T08:01:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a80b4aa0 0x7fc5a80d3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a80b4aa0 0x7fc5a80d3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38702/0 (socket says 172.21.15.98:38702) 2023-03-03T08:01:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5a77fe700 1 -- 172.21.15.98:0/4042403966 learned_addr learned my addr 172.21.15.98:0/4042403966 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5a6ffd700 1 --2- 172.21.15.98:0/4042403966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5a80cf570 0x7fc5a80cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5a7fff700 1 --2- 172.21.15.98:0/4042403966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a81609b0 0x7fc5a80cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5a77fe700 1 -- 172.21.15.98:0/4042403966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a81609b0 msgr2=0x7fc5a80cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5a77fe700 1 --2- 172.21.15.98:0/4042403966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a81609b0 0x7fc5a80cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5a77fe700 1 -- 172.21.15.98:0/4042403966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5a80cf570 msgr2=0x7fc5a80cc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5a77fe700 1 --2- 172.21.15.98:0/4042403966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5a80cf570 0x7fc5a80cc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5a77fe700 1 -- 172.21.15.98:0/4042403966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a0009a70 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.224+0000 7fc5a7fff700 1 --2- 172.21.15.98:0/4042403966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a81609b0 0x7fc5a80cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:01:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.225+0000 7fc5a77fe700 1 --2- 172.21.15.98:0/4042403966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a80b4aa0 0x7fc5a80d3090 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fc59800c480 tx=0x7fc59800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.225+0000 7fc58ffff700 1 -- 172.21.15.98:0/4042403966 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc598005d70 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.225+0000 7fc58ffff700 1 -- 172.21.15.98:0/4042403966 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5980070d0 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.225+0000 7fc5acd9d700 1 -- 172.21.15.98:0/4042403966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a815cdf0 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.225+0000 7fc58ffff700 1 -- 172.21.15.98:0/4042403966 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc598007440 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.225+0000 7fc5acd9d700 1 -- 172.21.15.98:0/4042403966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a80cdd40 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.226+0000 7fc5acd9d700 1 -- 172.21.15.98:0/4042403966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a815d0d0 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.227+0000 7fc58ffff700 1 -- 172.21.15.98:0/4042403966 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7fc598019030 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.229+0000 7fc58ffff700 1 --2- 172.21.15.98:0/4042403966 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc5940684a0 0x7fc59406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.229+0000 7fc58ffff700 1 -- 172.21.15.98:0/4042403966 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc598085340 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.229+0000 7fc5a6ffd700 1 --2- 172.21.15.98:0/4042403966 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc5940684a0 0x7fc59406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:48.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.230+0000 7fc5a6ffd700 1 --2- 172.21.15.98:0/4042403966 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc5940684a0 0x7fc59406a950 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc5a80b4140 tx=0x7fc59c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:48.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.232+0000 7fc58ffff700 1 -- 172.21.15.98:0/4042403966 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc59804b5e0 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.425+0000 7fc5acd9d700 1 -- 172.21.15.98:0/4042403966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc5a80cbdf0 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.426+0000 7fc58ffff700 1 -- 172.21.15.98:0/4042403966 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc5a80cbdf0 con 0x7fc5a80b4aa0 2023-03-03T08:01:48.427 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T08:01:48.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.428+0000 7fc5acd9d700 1 -- 172.21.15.98:0/4042403966 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc5940684a0 msgr2=0x7fc59406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:48.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.428+0000 7fc5acd9d700 1 --2- 172.21.15.98:0/4042403966 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc5940684a0 0x7fc59406a950 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc5a80b4140 tx=0x7fc59c009040 comp rx=0 tx=0).stop 2023-03-03T08:01:48.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.428+0000 7fc5acd9d700 1 -- 172.21.15.98:0/4042403966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a80b4aa0 msgr2=0x7fc5a80d3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:48.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.428+0000 7fc5acd9d700 1 --2- 172.21.15.98:0/4042403966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a80b4aa0 0x7fc5a80d3090 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fc59800c480 tx=0x7fc59800c840 comp rx=0 tx=0).stop 2023-03-03T08:01:48.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.428+0000 7fc5acd9d700 1 -- 172.21.15.98:0/4042403966 shutdown_connections 2023-03-03T08:01:48.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.428+0000 7fc5acd9d700 1 --2- 172.21.15.98:0/4042403966 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7fc5940684a0 0x7fc59406a950 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.428+0000 7fc5acd9d700 1 --2- 172.21.15.98:0/4042403966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a80b4aa0 0x7fc5a80d3090 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.428+0000 7fc5acd9d700 1 --2- 172.21.15.98:0/4042403966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a81609b0 0x7fc5a80cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.428+0000 7fc5acd9d700 1 --2- 172.21.15.98:0/4042403966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5a80cf570 0x7fc5a80cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.428+0000 7fc5acd9d700 1 -- 172.21.15.98:0/4042403966 >> 172.21.15.98:0/4042403966 conn(0x7fc5a806b880 msgr2=0x7fc5a8148d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:48.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.429+0000 7fc5acd9d700 1 -- 172.21.15.98:0/4042403966 shutdown_connections 2023-03-03T08:01:48.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.429+0000 7fc5acd9d700 1 -- 172.21.15.98:0/4042403966 wait complete. 2023-03-03T08:01:48.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T08:01:48.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.704+0000 7f15a71ff700 1 Processor -- start 2023-03-03T08:01:48.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.704+0000 7f15a71ff700 1 -- start start 2023-03-03T08:01:48.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.705+0000 7f15a71ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f15a015ca70 0x7f15a00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:48.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.705+0000 7f15a71ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a006b4b0 0x7f15a00cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:48.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.705+0000 7f15a71ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15a00d00a0 0x7f15a00b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:48.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.705+0000 7f15a71ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a00cd5b0 con 0x7f15a00d00a0 2023-03-03T08:01:48.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.705+0000 7f15a71ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a00cf980 con 0x7f15a015ca70 2023-03-03T08:01:48.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.705+0000 7f15a71ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a00d0510 con 0x7f15a006b4b0 2023-03-03T08:01:48.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.706+0000 7f15a579c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15a00d00a0 0x7f15a00b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:48.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.706+0000 7f15a579c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15a00d00a0 0x7f15a00b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38714/0 (socket says 172.21.15.98:38714) 2023-03-03T08:01:48.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.706+0000 7f15a579c700 1 -- 172.21.15.98:0/197875181 learned_addr learned my addr 172.21.15.98:0/197875181 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:48.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.706+0000 7f15a4f9b700 1 --2- 172.21.15.98:0/197875181 >> v2:172.21.15.100:3300/0 conn(0x7f15a015ca70 0x7f15a00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:48.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.706+0000 7f159ffff700 1 --2- 172.21.15.98:0/197875181 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a006b4b0 0x7f15a00cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:48.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.706+0000 7f15a579c700 1 -- 172.21.15.98:0/197875181 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a006b4b0 msgr2=0x7f15a00cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:48.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.706+0000 7f15a579c700 1 --2- 172.21.15.98:0/197875181 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a006b4b0 0x7f15a00cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.706+0000 7f15a579c700 1 -- 172.21.15.98:0/197875181 >> v2:172.21.15.100:3300/0 conn(0x7f15a015ca70 msgr2=0x7f15a00cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:48.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.706+0000 7f15a579c700 1 --2- 172.21.15.98:0/197875181 >> v2:172.21.15.100:3300/0 conn(0x7f15a015ca70 0x7f15a00cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.706+0000 7f15a579c700 1 -- 172.21.15.98:0/197875181 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a00d1800 con 0x7f15a00d00a0 2023-03-03T08:01:48.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.707+0000 7f15a579c700 1 --2- 172.21.15.98:0/197875181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a00d00a0 0x7f15a00b4c00 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f15a006c2c0 tx=0x7f159800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f45a5484db79ad7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:48.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.707+0000 7f159f7fe700 1 -- 172.21.15.98:0/197875181 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159800e070 con 0x7f15a00d00a0 2023-03-03T08:01:48.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.707+0000 7f159f7fe700 1 -- 172.21.15.98:0/197875181 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159800b8d0 con 0x7f15a00d00a0 2023-03-03T08:01:48.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.707+0000 7f159f7fe700 1 -- 172.21.15.98:0/197875181 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159800bc80 con 0x7f15a00d00a0 2023-03-03T08:01:48.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.708+0000 7f15a71ff700 1 -- 172.21.15.98:0/197875181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a00d00a0 msgr2=0x7f15a00b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:48.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.708+0000 7f15a71ff700 1 --2- 172.21.15.98:0/197875181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a00d00a0 0x7f15a00b4c00 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f15a006c2c0 tx=0x7f159800b330 comp rx=0 tx=0).stop 2023-03-03T08:01:48.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.708+0000 7f15a71ff700 1 -- 172.21.15.98:0/197875181 shutdown_connections 2023-03-03T08:01:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.708+0000 7f15a71ff700 1 --2- 172.21.15.98:0/197875181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a00d00a0 0x7f15a00b4c00 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.708+0000 7f15a71ff700 1 --2- 172.21.15.98:0/197875181 >> v2:172.21.15.100:3300/0 conn(0x7f15a015ca70 0x7f15a00cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.708+0000 7f15a71ff700 1 --2- 172.21.15.98:0/197875181 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a006b4b0 0x7f15a00cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.708+0000 7f15a71ff700 1 -- 172.21.15.98:0/197875181 >> 172.21.15.98:0/197875181 conn(0x7f15a0073e70 msgr2=0x7f15a006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.708+0000 7f15a71ff700 1 -- 172.21.15.98:0/197875181 shutdown_connections 2023-03-03T08:01:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.708+0000 7f15a71ff700 1 -- 172.21.15.98:0/197875181 wait complete. 2023-03-03T08:01:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.710+0000 7f15a71ff700 1 Processor -- start 2023-03-03T08:01:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.710+0000 7f15a71ff700 1 -- start start 2023-03-03T08:01:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.711+0000 7f15a71ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15a006b4b0 0x7f15a00b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.711+0000 7f15a71ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a00d00a0 0x7f15a0151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.711+0000 7f15a71ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a015ca70 0x7f15a015e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.711+0000 7f15a71ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a015f360 con 0x7f15a015ca70 2023-03-03T08:01:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.711+0000 7f15a71ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a015f640 con 0x7f15a006b4b0 2023-03-03T08:01:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.711+0000 7f15a71ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a015f920 con 0x7f15a00d00a0 2023-03-03T08:01:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.712+0000 7f15a579c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a015ca70 0x7f15a015e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.712+0000 7f15a579c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a015ca70 0x7f15a015e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38718/0 (socket says 172.21.15.98:38718) 2023-03-03T08:01:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.712+0000 7f15a579c700 1 -- 172.21.15.98:0/2824866589 learned_addr learned my addr 172.21.15.98:0/2824866589 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.712+0000 7f159ffff700 1 --2- 172.21.15.98:0/2824866589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a00d00a0 0x7f15a0151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.712+0000 7f15a4f9b700 1 --2- 172.21.15.98:0/2824866589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15a006b4b0 0x7f15a00b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.712+0000 7f15a579c700 1 -- 172.21.15.98:0/2824866589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a00d00a0 msgr2=0x7f15a0151ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.712+0000 7f15a579c700 1 --2- 172.21.15.98:0/2824866589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a00d00a0 0x7f15a0151ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.712+0000 7f15a579c700 1 -- 172.21.15.98:0/2824866589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15a006b4b0 msgr2=0x7f15a00b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.712+0000 7f15a579c700 1 --2- 172.21.15.98:0/2824866589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15a006b4b0 0x7f15a00b5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.712+0000 7f15a579c700 1 -- 172.21.15.98:0/2824866589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1598009b30 con 0x7f15a015ca70 2023-03-03T08:01:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.712+0000 7f15a579c700 1 --2- 172.21.15.98:0/2824866589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a015ca70 0x7f15a015e690 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f1598012000 tx=0x7f1598004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.713+0000 7f159d7fa700 1 -- 172.21.15.98:0/2824866589 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1598003e10 con 0x7f15a015ca70 2023-03-03T08:01:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.713+0000 7f159d7fa700 1 -- 172.21.15.98:0/2824866589 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1598004250 con 0x7f15a015ca70 2023-03-03T08:01:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.713+0000 7f159d7fa700 1 -- 172.21.15.98:0/2824866589 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1598004600 con 0x7f15a015ca70 2023-03-03T08:01:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.713+0000 7f15a71ff700 1 -- 172.21.15.98:0/2824866589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a00cdbe0 con 0x7f15a015ca70 2023-03-03T08:01:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.713+0000 7f15a71ff700 1 -- 172.21.15.98:0/2824866589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a00cd900 con 0x7f15a015ca70 2023-03-03T08:01:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.714+0000 7f15a71ff700 1 -- 172.21.15.98:0/2824866589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a00d1aa0 con 0x7f15a015ca70 2023-03-03T08:01:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.715+0000 7f159d7fa700 1 -- 172.21.15.98:0/2824866589 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f1598004760 con 0x7f15a015ca70 2023-03-03T08:01:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.716+0000 7f159d7fa700 1 --2- 172.21.15.98:0/2824866589 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15840684a0 0x7f158406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.716+0000 7f159d7fa700 1 -- 172.21.15.98:0/2824866589 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f15980874d0 con 0x7f15a015ca70 2023-03-03T08:01:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.717+0000 7f15a4f9b700 1 --2- 172.21.15.98:0/2824866589 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15840684a0 0x7f158406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.717+0000 7f15a4f9b700 1 --2- 172.21.15.98:0/2824866589 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15840684a0 0x7f158406a950 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1594009a50 tx=0x7f1594008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.720+0000 7f159d7fa700 1 -- 172.21.15.98:0/2824866589 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1598053b90 con 0x7f15a015ca70 2023-03-03T08:01:48.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.928+0000 7f15a71ff700 1 -- 172.21.15.98:0/2824866589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f15a015fe40 con 0x7f15a015ca70 2023-03-03T08:01:48.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.929+0000 7f159d7fa700 1 -- 172.21.15.98:0/2824866589 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v78) v1 ==== 80+0+0 (secure 0 0 0) 0x7f15a015fe40 con 0x7f15a015ca70 2023-03-03T08:01:48.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.931+0000 7f15a71ff700 1 -- 172.21.15.98:0/2824866589 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15840684a0 msgr2=0x7f158406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:48.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.931+0000 7f15a71ff700 1 --2- 172.21.15.98:0/2824866589 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15840684a0 0x7f158406a950 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1594009a50 tx=0x7f1594008040 comp rx=0 tx=0).stop 2023-03-03T08:01:48.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.931+0000 7f15a71ff700 1 -- 172.21.15.98:0/2824866589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a015ca70 msgr2=0x7f15a015e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:48.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.931+0000 7f15a71ff700 1 --2- 172.21.15.98:0/2824866589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a015ca70 0x7f15a015e690 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f1598012000 tx=0x7f1598004ad0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.932+0000 7f15a71ff700 1 -- 172.21.15.98:0/2824866589 shutdown_connections 2023-03-03T08:01:48.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.932+0000 7f15a71ff700 1 --2- 172.21.15.98:0/2824866589 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f15840684a0 0x7f158406a950 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.932+0000 7f15a71ff700 1 --2- 172.21.15.98:0/2824866589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a015ca70 0x7f15a015e690 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.932+0000 7f15a71ff700 1 --2- 172.21.15.98:0/2824866589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15a006b4b0 0x7f15a00b5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.932+0000 7f15a71ff700 1 --2- 172.21.15.98:0/2824866589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15a00d00a0 0x7f15a0151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:48.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.932+0000 7f15a71ff700 1 -- 172.21.15.98:0/2824866589 >> 172.21.15.98:0/2824866589 conn(0x7f15a0073e70 msgr2=0x7f15a00b63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:48.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.932+0000 7f15a71ff700 1 -- 172.21.15.98:0/2824866589 shutdown_connections 2023-03-03T08:01:48.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:48.932+0000 7f15a71ff700 1 -- 172.21.15.98:0/2824866589 wait complete. 2023-03-03T08:01:48.935 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-03-03T08:01:48.960 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2023-03-03T08:01:48.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:48.961 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-03-03T08:01:49.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[119758]: fuse finished with error 0 and tester_r 0 2023-03-03T08:01:49.070 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:01:55.172 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T08:01:55.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:01:55.173 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-03-03T08:01:55.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:01:55.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.498+0000 7f9d5abff700 1 Processor -- start 2023-03-03T08:01:55.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.498+0000 7f9d5abff700 1 -- start start 2023-03-03T08:01:55.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.500+0000 7f9d5abff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d540c68f0 0x7f9d540b2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:55.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.500+0000 7f9d5abff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9d5415d5b0 0x7f9d5415da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:55.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.500+0000 7f9d5abff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5415ce50 con 0x7f9d5406a330 2023-03-03T08:01:55.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.500+0000 7f9d5abff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d5415d120 con 0x7f9d5415d5b0 2023-03-03T08:01:55.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.500+0000 7f9d5abff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5406aaa0 con 0x7f9d540c68f0 2023-03-03T08:01:55.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.500+0000 7f9d5919c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9d5415d5b0 0x7f9d5415da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:55.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.500+0000 7f9d5919c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9d5415d5b0 0x7f9d5415da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47758/0 (socket says 172.21.15.98:47758) 2023-03-03T08:01:55.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.500+0000 7f9d5919c700 1 -- 172.21.15.98:0/1345177900 learned_addr learned my addr 172.21.15.98:0/1345177900 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:55.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.500+0000 7f9d5899b700 1 --2- 172.21.15.98:0/1345177900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d540c68f0 0x7f9d540b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:55.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.501+0000 7f9d5919c700 1 -- 172.21.15.98:0/1345177900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d540c68f0 msgr2=0x7f9d540b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.501+0000 7f9d5919c700 1 --2- 172.21.15.98:0/1345177900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d540c68f0 0x7f9d540b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.501+0000 7f9d5919c700 1 -- 172.21.15.98:0/1345177900 >> v1:172.21.15.100:6789/0 conn(0x7f9d5406a330 legacy=0x7f9d54160490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.501+0000 7f9d5919c700 1 -- 172.21.15.98:0/1345177900 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5406ad80 con 0x7f9d5415d5b0 2023-03-03T08:01:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.501+0000 7f9d5919c700 1 --2- 172.21.15.98:0/1345177900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d5415d5b0 0x7f9d5415da30 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c009e50 tx=0x7f9d4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4150e001c402ce87 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.501+0000 7f9d537fe700 1 -- 172.21.15.98:0/1345177900 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c00e070 con 0x7f9d5415d5b0 2023-03-03T08:01:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.501+0000 7f9d537fe700 1 -- 172.21.15.98:0/1345177900 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4c00b9c0 con 0x7f9d5415d5b0 2023-03-03T08:01:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.501+0000 7f9d537fe700 1 -- 172.21.15.98:0/1345177900 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c00bd70 con 0x7f9d5415d5b0 2023-03-03T08:01:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.502+0000 7f9d5abff700 1 -- 172.21.15.98:0/1345177900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d5415d5b0 msgr2=0x7f9d5415da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.502+0000 7f9d5abff700 1 --2- 172.21.15.98:0/1345177900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d5415d5b0 0x7f9d5415da30 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c009e50 tx=0x7f9d4c00b330 comp rx=0 tx=0).stop 2023-03-03T08:01:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.502+0000 7f9d5abff700 1 -- 172.21.15.98:0/1345177900 shutdown_connections 2023-03-03T08:01:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.502+0000 7f9d5abff700 1 --2- 172.21.15.98:0/1345177900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d5415d5b0 0x7f9d5415da30 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.502+0000 7f9d5abff700 1 --2- 172.21.15.98:0/1345177900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d540c68f0 0x7f9d540b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.502+0000 7f9d5abff700 1 -- 172.21.15.98:0/1345177900 >> 172.21.15.98:0/1345177900 conn(0x7f9d5406b440 msgr2=0x7f9d5406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.502+0000 7f9d5abff700 1 -- 172.21.15.98:0/1345177900 shutdown_connections 2023-03-03T08:01:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.502+0000 7f9d5abff700 1 -- 172.21.15.98:0/1345177900 wait complete. 2023-03-03T08:01:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.505+0000 7f9d5abff700 1 Processor -- start 2023-03-03T08:01:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.505+0000 7f9d5abff700 1 -- start start 2023-03-03T08:01:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d5abff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d5406a330 0x7f9d541f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d5abff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9d540c68f0 0x7f9d541f3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d5abff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d5415d5b0 0x7f9d541f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d5abff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d541f56f0 con 0x7f9d5415d5b0 2023-03-03T08:01:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d5abff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d541f29a0 con 0x7f9d540c68f0 2023-03-03T08:01:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d5abff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d541f2c80 con 0x7f9d5406a330 2023-03-03T08:01:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d5919c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d5415d5b0 0x7f9d541f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d5899b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d5406a330 0x7f9d541f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d53fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9d540c68f0 0x7f9d541f3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d5899b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d5406a330 0x7f9d541f2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:39044/0 (socket says 172.21.15.98:39044) 2023-03-03T08:01:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d5899b700 1 -- 172.21.15.98:0/488508814 learned_addr learned my addr 172.21.15.98:0/488508814 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d5919c700 1 -- 172.21.15.98:0/488508814 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d5406a330 msgr2=0x7f9d541f2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d5919c700 1 --2- 172.21.15.98:0/488508814 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d5406a330 0x7f9d541f2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d5919c700 1 -- 172.21.15.98:0/488508814 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9d540c68f0 msgr2=0x7f9d541f3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d5919c700 1 --2- 172.21.15.98:0/488508814 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9d540c68f0 0x7f9d541f3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.507+0000 7f9d5919c700 1 -- 172.21.15.98:0/488508814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c009b30 con 0x7f9d5415d5b0 2023-03-03T08:01:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.508+0000 7f9d5919c700 1 --2- 172.21.15.98:0/488508814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d5415d5b0 0x7f9d541f51b0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c015000 tx=0x7f9d4c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.508+0000 7f9d517fa700 1 -- 172.21.15.98:0/488508814 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c004840 con 0x7f9d5415d5b0 2023-03-03T08:01:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.508+0000 7f9d517fa700 1 -- 172.21.15.98:0/488508814 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4c0092e0 con 0x7f9d5415d5b0 2023-03-03T08:01:55.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.508+0000 7f9d517fa700 1 -- 172.21.15.98:0/488508814 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c01c660 con 0x7f9d5415d5b0 2023-03-03T08:01:55.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.508+0000 7f9d5abff700 1 -- 172.21.15.98:0/488508814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d5406a7a0 con 0x7f9d5415d5b0 2023-03-03T08:01:55.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.508+0000 7f9d5abff700 1 -- 172.21.15.98:0/488508814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d5406b060 con 0x7f9d5415d5b0 2023-03-03T08:01:55.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.509+0000 7f9d5abff700 1 -- 172.21.15.98:0/488508814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d540c6cc0 con 0x7f9d5415d5b0 2023-03-03T08:01:55.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.511+0000 7f9d517fa700 1 -- 172.21.15.98:0/488508814 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f9d4c022030 con 0x7f9d5415d5b0 2023-03-03T08:01:55.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.512+0000 7f9d517fa700 1 --2- 172.21.15.98:0/488508814 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9d380684a0 0x7f9d3806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:55.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.512+0000 7f9d517fa700 1 -- 172.21.15.98:0/488508814 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9d4c087620 con 0x7f9d5415d5b0 2023-03-03T08:01:55.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.512+0000 7f9d5899b700 1 --2- 172.21.15.98:0/488508814 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9d380684a0 0x7f9d3806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:55.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.513+0000 7f9d5899b700 1 --2- 172.21.15.98:0/488508814 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9d380684a0 0x7f9d3806a950 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9d540cc2b0 tx=0x7f9d48008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:55.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.516+0000 7f9d517fa700 1 -- 172.21.15.98:0/488508814 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9d4c053ce0 con 0x7f9d5415d5b0 2023-03-03T08:01:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.752+0000 7f9d5abff700 1 -- 172.21.15.98:0/488508814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d5404e6e0 con 0x7f9d5415d5b0 2023-03-03T08:01:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.752+0000 7f9d517fa700 1 -- 172.21.15.98:0/488508814 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f9d4c057300 con 0x7f9d5415d5b0 2023-03-03T08:01:55.755 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:55.755 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6050,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/318891906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":318891906},{"type":"v1","addr":"172.21.15.98:6836","nonce":318891906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1280578660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1280578660},{"type":"v1","addr":"172.21.15.98:6837","nonce":1280578660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6906,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1395511883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1395511883},{"type":"v1","addr":"172.21.15.100:6834","nonce":1395511883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:01:21.379777+0000","modified":"2023-03-03T08:01:24.835314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6909},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6909":{"gid":6909,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/341424177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":341424177},{"type":"v1","addr":"172.21.15.100:6835","nonce":341424177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-03-03T08:01:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.755+0000 7f9d5abff700 1 -- 172.21.15.98:0/488508814 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9d380684a0 msgr2=0x7f9d3806a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.755+0000 7f9d5abff700 1 --2- 172.21.15.98:0/488508814 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9d380684a0 0x7f9d3806a950 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9d540cc2b0 tx=0x7f9d48008040 comp rx=0 tx=0).stop 2023-03-03T08:01:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.755+0000 7f9d5abff700 1 -- 172.21.15.98:0/488508814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d5415d5b0 msgr2=0x7f9d541f51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.755+0000 7f9d5abff700 1 --2- 172.21.15.98:0/488508814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d5415d5b0 0x7f9d541f51b0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c015000 tx=0x7f9d4c004ad0 comp rx=0 tx=0).stop 2023-03-03T08:01:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.755+0000 7f9d5abff700 1 -- 172.21.15.98:0/488508814 shutdown_connections 2023-03-03T08:01:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.755+0000 7f9d5abff700 1 --2- 172.21.15.98:0/488508814 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f9d380684a0 0x7f9d3806a950 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.755+0000 7f9d5abff700 1 --2- 172.21.15.98:0/488508814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d5415d5b0 0x7f9d541f51b0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.755+0000 7f9d5abff700 1 --2- 172.21.15.98:0/488508814 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d5406a330 0x7f9d541f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.755+0000 7f9d5abff700 1 --2- 172.21.15.98:0/488508814 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9d540c68f0 0x7f9d541f3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.755+0000 7f9d5abff700 1 -- 172.21.15.98:0/488508814 >> 172.21.15.98:0/488508814 conn(0x7f9d5406b440 msgr2=0x7f9d540b13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.755+0000 7f9d5abff700 1 -- 172.21.15.98:0/488508814 shutdown_connections 2023-03-03T08:01:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:55.755+0000 7f9d5abff700 1 -- 172.21.15.98:0/488508814 wait complete. 2023-03-03T08:01:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2023-03-03T08:01:55.785 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2023-03-03T08:01:56.065 INFO:teuthology.orchestra.run.smithi100.stdout:[] 2023-03-03T08:01:56.066 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2023-03-03T08:01:56.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000757167 2023-03-03T08:01:56.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.312+0000 7f5a479ff700 1 Processor -- start 2023-03-03T08:01:56.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.313+0000 7f5a479ff700 1 -- start start 2023-03-03T08:01:56.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.314+0000 7f5a479ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5a400d0270 0x7f5a400cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:56.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.314+0000 7f5a479ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a4006b4b0 0x7f5a400d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:56.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.314+0000 7f5a479ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a400cf980 con 0x7f5a4015ca70 2023-03-03T08:01:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.314+0000 7f5a479ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a400cd5b0 con 0x7f5a400d0270 2023-03-03T08:01:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.314+0000 7f5a479ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400d1800 con 0x7f5a4006b4b0 2023-03-03T08:01:56.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.314+0000 7f5a4579b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5a4015ca70 0x7f5a400cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45612/0 (socket says 172.21.15.98:45612) 2023-03-03T08:01:56.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.314+0000 7f5a4579b700 1 -- 172.21.15.98:0/3743465392 learned_addr learned my addr 172.21.15.98:0/3743465392 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:56.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.315+0000 7f5a45f9c700 1 --2- 172.21.15.98:0/3743465392 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a4006b4b0 0x7f5a400d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:56.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.315+0000 7f5a44f9a700 1 --2- 172.21.15.98:0/3743465392 >> v2:172.21.15.100:3300/0 conn(0x7f5a400d0270 0x7f5a400cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:56.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.315+0000 7f5a37fff700 1 -- 172.21.15.98:0/3743465392 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001330859 0 0) 0x7f5a400cf980 con 0x7f5a4015ca70 2023-03-03T08:01:56.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.315+0000 7f5a37fff700 1 -- 172.21.15.98:0/3743465392 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a2c0035b0 con 0x7f5a4015ca70 2023-03-03T08:01:56.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.315+0000 7f5a45f9c700 1 -- 172.21.15.98:0/3743465392 >> v2:172.21.15.100:3300/0 conn(0x7f5a400d0270 msgr2=0x7f5a400cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:56.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.315+0000 7f5a45f9c700 1 --2- 172.21.15.98:0/3743465392 >> v2:172.21.15.100:3300/0 conn(0x7f5a400d0270 0x7f5a400cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:56.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.315+0000 7f5a37fff700 1 -- 172.21.15.98:0/3743465392 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4251269849 0 0) 0x7f5a2c0035b0 con 0x7f5a4015ca70 2023-03-03T08:01:56.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.315+0000 7f5a45f9c700 1 -- 172.21.15.98:0/3743465392 >> v1:172.21.15.98:6789/0 conn(0x7f5a4015ca70 legacy=0x7f5a400cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:56.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.315+0000 7f5a45f9c700 1 -- 172.21.15.98:0/3743465392 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a400cd900 con 0x7f5a4006b4b0 2023-03-03T08:01:56.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.315+0000 7f5a44f9a700 1 --2- 172.21.15.98:0/3743465392 >> v2:172.21.15.100:3300/0 conn(0x7f5a400d0270 0x7f5a400cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:56.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.316+0000 7f5a45f9c700 1 --2- 172.21.15.98:0/3743465392 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a4006b4b0 0x7f5a400d1fc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5a38009d90 tx=0x7f5a3800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26d889cc091be41d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:56.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.316+0000 7f5a37fff700 1 -- 172.21.15.98:0/3743465392 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3800e070 con 0x7f5a4006b4b0 2023-03-03T08:01:56.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.316+0000 7f5a37fff700 1 -- 172.21.15.98:0/3743465392 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a38003a20 con 0x7f5a4006b4b0 2023-03-03T08:01:56.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.316+0000 7f5a37fff700 1 -- 172.21.15.98:0/3743465392 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c001670 con 0x7f5a4006b4b0 2023-03-03T08:01:56.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.317+0000 7f5a479ff700 1 -- 172.21.15.98:0/3743465392 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a4006b4b0 msgr2=0x7f5a400d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:56.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.317+0000 7f5a479ff700 1 --2- 172.21.15.98:0/3743465392 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a4006b4b0 0x7f5a400d1fc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5a38009d90 tx=0x7f5a3800b330 comp rx=0 tx=0).stop 2023-03-03T08:01:56.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.317+0000 7f5a479ff700 1 -- 172.21.15.98:0/3743465392 shutdown_connections 2023-03-03T08:01:56.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.317+0000 7f5a479ff700 1 --2- 172.21.15.98:0/3743465392 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a4006b4b0 0x7f5a400d1fc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:56.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.317+0000 7f5a479ff700 1 --2- 172.21.15.98:0/3743465392 >> v2:172.21.15.100:3300/0 conn(0x7f5a400d0270 0x7f5a400cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:56.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.317+0000 7f5a479ff700 1 -- 172.21.15.98:0/3743465392 >> 172.21.15.98:0/3743465392 conn(0x7f5a40073e70 msgr2=0x7f5a4006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:56.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.317+0000 7f5a479ff700 1 -- 172.21.15.98:0/3743465392 shutdown_connections 2023-03-03T08:01:56.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.317+0000 7f5a479ff700 1 -- 172.21.15.98:0/3743465392 wait complete. 2023-03-03T08:01:56.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.319+0000 7f5a479ff700 1 Processor -- start 2023-03-03T08:01:56.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.319+0000 7f5a479ff700 1 -- start start 2023-03-03T08:01:56.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.320+0000 7f5a479ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a4006b4b0 0x7f5a401f1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:56.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.320+0000 7f5a479ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5a400d0270 0x7f5a400cc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:56.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.320+0000 7f5a479ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a4015ca70 0x7f5a400cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:56.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.320+0000 7f5a479ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400ca250 con 0x7f5a4006b4b0 2023-03-03T08:01:56.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.320+0000 7f5a479ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400ca4f0 con 0x7f5a400d0270 2023-03-03T08:01:56.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.320+0000 7f5a479ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400ca7d0 con 0x7f5a4015ca70 2023-03-03T08:01:56.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.320+0000 7f5a4579b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a4006b4b0 0x7f5a401f1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:56.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.320+0000 7f5a4579b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a4006b4b0 0x7f5a401f1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47768/0 (socket says 172.21.15.98:47768) 2023-03-03T08:01:56.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.320+0000 7f5a4579b700 1 -- 172.21.15.98:0/1199221884 learned_addr learned my addr 172.21.15.98:0/1199221884 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:56.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.321+0000 7f5a45f9c700 1 --2- 172.21.15.98:0/1199221884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a4015ca70 0x7f5a400cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:56.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.321+0000 7f5a44f9a700 1 --2- 172.21.15.98:0/1199221884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5a400d0270 0x7f5a400cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:56.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.321+0000 7f5a4579b700 1 -- 172.21.15.98:0/1199221884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a4015ca70 msgr2=0x7f5a400cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:56.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.321+0000 7f5a4579b700 1 --2- 172.21.15.98:0/1199221884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a4015ca70 0x7f5a400cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:56.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.321+0000 7f5a4579b700 1 -- 172.21.15.98:0/1199221884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5a400d0270 msgr2=0x7f5a400cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:56.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.321+0000 7f5a4579b700 1 --2- 172.21.15.98:0/1199221884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5a400d0270 0x7f5a400cc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:56.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.321+0000 7f5a4579b700 1 -- 172.21.15.98:0/1199221884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a300094a0 con 0x7f5a4006b4b0 2023-03-03T08:01:56.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.321+0000 7f5a45f9c700 1 --2- 172.21.15.98:0/1199221884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a4015ca70 0x7f5a400cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:56.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.321+0000 7f5a4579b700 1 --2- 172.21.15.98:0/1199221884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a4006b4b0 0x7f5a401f1700 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00c520 tx=0x7f5a3c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:56.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.322+0000 7f5a35ffb700 1 -- 172.21.15.98:0/1199221884 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c004c80 con 0x7f5a4006b4b0 2023-03-03T08:01:56.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.322+0000 7f5a35ffb700 1 -- 172.21.15.98:0/1199221884 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c004de0 con 0x7f5a4006b4b0 2023-03-03T08:01:56.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.322+0000 7f5a35ffb700 1 -- 172.21.15.98:0/1199221884 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c007310 con 0x7f5a4006b4b0 2023-03-03T08:01:56.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.322+0000 7f5a479ff700 1 -- 172.21.15.98:0/1199221884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a38009a70 con 0x7f5a4006b4b0 2023-03-03T08:01:56.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.322+0000 7f5a479ff700 1 -- 172.21.15.98:0/1199221884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a400d1aa0 con 0x7f5a4006b4b0 2023-03-03T08:01:56.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.323+0000 7f5a479ff700 1 -- 172.21.15.98:0/1199221884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a400cdbe0 con 0x7f5a4006b4b0 2023-03-03T08:01:56.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.324+0000 7f5a35ffb700 1 -- 172.21.15.98:0/1199221884 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f5a3c01a020 con 0x7f5a4006b4b0 2023-03-03T08:01:56.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.325+0000 7f5a35ffb700 1 --2- 172.21.15.98:0/1199221884 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5a240684a0 0x7f5a2406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:56.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.325+0000 7f5a35ffb700 1 -- 172.21.15.98:0/1199221884 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5a3c085bb0 con 0x7f5a4006b4b0 2023-03-03T08:01:56.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.325+0000 7f5a44f9a700 1 --2- 172.21.15.98:0/1199221884 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5a240684a0 0x7f5a2406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:56.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.326+0000 7f5a44f9a700 1 --2- 172.21.15.98:0/1199221884 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5a240684a0 0x7f5a2406a950 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5a400c6900 tx=0x7f5a30009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:56.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.329+0000 7f5a35ffb700 1 -- 172.21.15.98:0/1199221884 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5a3c052110 con 0x7f5a4006b4b0 2023-03-03T08:01:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.553+0000 7f5a479ff700 1 -- 172.21.15.98:0/1199221884 --> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000757167", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a4015ce40 con 0x7f5a240684a0 2023-03-03T08:01:56.561 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:01:56.559+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:01:56.561 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:01:56.559+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:01:56.561 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:01:56.559+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:01:56.561 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:01:56.559+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:01:56.561 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:01:56.559+0000 7f813dbf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:01:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.599+0000 7f5a35ffb700 1 -- 172.21.15.98:0/1199221884 <== mgr.4110 v2:172.21.15.98:6832/108777 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5a4015ce40 con 0x7f5a240684a0 2023-03-03T08:01:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.602+0000 7f5a479ff700 1 -- 172.21.15.98:0/1199221884 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5a240684a0 msgr2=0x7f5a2406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.602+0000 7f5a479ff700 1 --2- 172.21.15.98:0/1199221884 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5a240684a0 0x7f5a2406a950 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5a400c6900 tx=0x7f5a30009210 comp rx=0 tx=0).stop 2023-03-03T08:01:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.602+0000 7f5a479ff700 1 -- 172.21.15.98:0/1199221884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a4006b4b0 msgr2=0x7f5a401f1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.602+0000 7f5a479ff700 1 --2- 172.21.15.98:0/1199221884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a4006b4b0 0x7f5a401f1700 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00c520 tx=0x7f5a3c00c830 comp rx=0 tx=0).stop 2023-03-03T08:01:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.602+0000 7f5a479ff700 1 -- 172.21.15.98:0/1199221884 shutdown_connections 2023-03-03T08:01:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.602+0000 7f5a479ff700 1 --2- 172.21.15.98:0/1199221884 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f5a240684a0 0x7f5a2406a950 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.602+0000 7f5a479ff700 1 --2- 172.21.15.98:0/1199221884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a4006b4b0 0x7f5a401f1700 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.602+0000 7f5a479ff700 1 --2- 172.21.15.98:0/1199221884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a4015ca70 0x7f5a400cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.602+0000 7f5a479ff700 1 --2- 172.21.15.98:0/1199221884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5a400d0270 0x7f5a400cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.602+0000 7f5a479ff700 1 -- 172.21.15.98:0/1199221884 >> 172.21.15.98:0/1199221884 conn(0x7f5a40073e70 msgr2=0x7f5a400d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.603+0000 7f5a479ff700 1 -- 172.21.15.98:0/1199221884 shutdown_connections 2023-03-03T08:01:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.603+0000 7f5a479ff700 1 -- 172.21.15.98:0/1199221884 wait complete. 2023-03-03T08:01:56.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:01:56.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.876+0000 7f84f679d700 1 Processor -- start 2023-03-03T08:01:56.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.877+0000 7f84f679d700 1 -- start start 2023-03-03T08:01:56.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.878+0000 7f84f679d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84f00d0010 0x7f84f00d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:56.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.878+0000 7f84f679d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f84f00d1650 0x7f84f00b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:56.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.878+0000 7f84f679d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f00b0c50 con 0x7f84f00b4a00 2023-03-03T08:01:56.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.878+0000 7f84f679d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84f00b0040 con 0x7f84f00d1650 2023-03-03T08:01:56.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.878+0000 7f84f679d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f00b0320 con 0x7f84f00d0010 2023-03-03T08:01:56.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.878+0000 7f84f579b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84f00d0010 0x7f84f00d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:56.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.878+0000 7f84f4f9a700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f84f00b4a00 0x7f84f00d0f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:58434/0 (socket says 172.21.15.98:58434) 2023-03-03T08:01:56.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.879+0000 7f84f4f9a700 1 -- 172.21.15.98:0/820614555 learned_addr learned my addr 172.21.15.98:0/820614555 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:56.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.879+0000 7f84f5f9c700 1 --2- 172.21.15.98:0/820614555 >> v2:172.21.15.100:3300/0 conn(0x7f84f00d1650 0x7f84f00b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:56.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.879+0000 7f84f579b700 1 -- 172.21.15.98:0/820614555 >> v2:172.21.15.100:3300/0 conn(0x7f84f00d1650 msgr2=0x7f84f00b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:56.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.879+0000 7f84f579b700 1 --2- 172.21.15.98:0/820614555 >> v2:172.21.15.100:3300/0 conn(0x7f84f00d1650 0x7f84f00b2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:56.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.879+0000 7f84f579b700 1 -- 172.21.15.98:0/820614555 >> v1:172.21.15.100:6789/0 conn(0x7f84f00b4a00 legacy=0x7f84f00d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:56.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.879+0000 7f84e7fff700 1 -- 172.21.15.98:0/820614555 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511064787 0 0) 0x7f84f00b0c50 con 0x7f84f00b4a00 2023-03-03T08:01:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.879+0000 7f84f579b700 1 -- 172.21.15.98:0/820614555 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f00b0600 con 0x7f84f00d0010 2023-03-03T08:01:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.879+0000 7f84f579b700 1 --2- 172.21.15.98:0/820614555 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84f00d0010 0x7f84f00d03e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f84ec009820 tx=0x7f84ec009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7920648b96d23750 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.880+0000 7f84e7fff700 1 -- 172.21.15.98:0/820614555 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec005e30 con 0x7f84f00d0010 2023-03-03T08:01:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.880+0000 7f84e7fff700 1 -- 172.21.15.98:0/820614555 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ec006030 con 0x7f84f00d0010 2023-03-03T08:01:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.880+0000 7f84e7fff700 1 -- 172.21.15.98:0/820614555 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec0063c0 con 0x7f84f00d0010 2023-03-03T08:01:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.880+0000 7f84f679d700 1 -- 172.21.15.98:0/820614555 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84f00d0010 msgr2=0x7f84f00d03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.880+0000 7f84f679d700 1 --2- 172.21.15.98:0/820614555 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84f00d0010 0x7f84f00d03e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f84ec009820 tx=0x7f84ec009b30 comp rx=0 tx=0).stop 2023-03-03T08:01:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.880+0000 7f84f679d700 1 -- 172.21.15.98:0/820614555 shutdown_connections 2023-03-03T08:01:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.880+0000 7f84f679d700 1 --2- 172.21.15.98:0/820614555 >> v2:172.21.15.100:3300/0 conn(0x7f84f00d1650 0x7f84f00b2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.880+0000 7f84f679d700 1 --2- 172.21.15.98:0/820614555 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84f00d0010 0x7f84f00d03e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.880+0000 7f84f679d700 1 -- 172.21.15.98:0/820614555 >> 172.21.15.98:0/820614555 conn(0x7f84f0073b00 msgr2=0x7f84f00d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.881+0000 7f84f679d700 1 -- 172.21.15.98:0/820614555 shutdown_connections 2023-03-03T08:01:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.881+0000 7f84f679d700 1 -- 172.21.15.98:0/820614555 wait complete. 2023-03-03T08:01:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.883+0000 7f84f679d700 1 Processor -- start 2023-03-03T08:01:56.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.883+0000 7f84f679d700 1 -- start start 2023-03-03T08:01:56.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.884+0000 7f84f679d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84f00b4a00 0x7f84f00c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:56.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.884+0000 7f84f679d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f00d0010 0x7f84f01e5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:56.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.884+0000 7f84f679d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84f00d1650 0x7f84f00c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:56.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.884+0000 7f84f679d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f00c6ea0 con 0x7f84f00d0010 2023-03-03T08:01:56.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.884+0000 7f84f679d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f00c70f0 con 0x7f84f00b4a00 2023-03-03T08:01:56.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.884+0000 7f84f679d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f00c73d0 con 0x7f84f00d1650 2023-03-03T08:01:56.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.884+0000 7f84f579b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84f00b4a00 0x7f84f00c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:56.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.884+0000 7f84f579b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84f00b4a00 0x7f84f00c7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:50368/0 (socket says 172.21.15.98:50368) 2023-03-03T08:01:56.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.884+0000 7f84f579b700 1 -- 172.21.15.98:0/4068684768 learned_addr learned my addr 172.21.15.98:0/4068684768 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:56.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.884+0000 7f84f5f9c700 1 --2- 172.21.15.98:0/4068684768 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84f00d1650 0x7f84f00c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:56.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.884+0000 7f84f4f9a700 1 --2- 172.21.15.98:0/4068684768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f00d0010 0x7f84f01e5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:56.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.885+0000 7f84f579b700 1 -- 172.21.15.98:0/4068684768 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84f00d1650 msgr2=0x7f84f00c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:56.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.885+0000 7f84f579b700 1 --2- 172.21.15.98:0/4068684768 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84f00d1650 0x7f84f00c8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:56.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.885+0000 7f84f579b700 1 -- 172.21.15.98:0/4068684768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f00d0010 msgr2=0x7f84f01e5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:56.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.885+0000 7f84f579b700 1 --2- 172.21.15.98:0/4068684768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f00d0010 0x7f84f01e5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:56.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.885+0000 7f84f579b700 1 -- 172.21.15.98:0/4068684768 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ec009430 con 0x7f84f00b4a00 2023-03-03T08:01:56.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.885+0000 7f84f579b700 1 --2- 172.21.15.98:0/4068684768 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84f00b4a00 0x7f84f00c7eb0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f84ec000c00 tx=0x7f84ec004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:56.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.886+0000 7f84e5ffb700 1 -- 172.21.15.98:0/4068684768 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec0061a0 con 0x7f84f00b4a00 2023-03-03T08:01:56.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.886+0000 7f84f679d700 1 -- 172.21.15.98:0/4068684768 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f00d0b30 con 0x7f84f00b4a00 2023-03-03T08:01:56.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.886+0000 7f84e5ffb700 1 -- 172.21.15.98:0/4068684768 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ec020070 con 0x7f84f00b4a00 2023-03-03T08:01:56.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.886+0000 7f84e5ffb700 1 -- 172.21.15.98:0/4068684768 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec006540 con 0x7f84f00b4a00 2023-03-03T08:01:56.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.886+0000 7f84f679d700 1 -- 172.21.15.98:0/4068684768 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f00b08e0 con 0x7f84f00b4a00 2023-03-03T08:01:56.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.887+0000 7f84f679d700 1 -- 172.21.15.98:0/4068684768 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f00d0cc0 con 0x7f84f00b4a00 2023-03-03T08:01:56.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.888+0000 7f84e5ffb700 1 -- 172.21.15.98:0/4068684768 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f84ec02b020 con 0x7f84f00b4a00 2023-03-03T08:01:56.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.889+0000 7f84e5ffb700 1 --2- 172.21.15.98:0/4068684768 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f84d40684a0 0x7f84d406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:56.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.889+0000 7f84e5ffb700 1 -- 172.21.15.98:0/4068684768 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f84ec087570 con 0x7f84f00b4a00 2023-03-03T08:01:56.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.889+0000 7f84f4f9a700 1 --2- 172.21.15.98:0/4068684768 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f84d40684a0 0x7f84d406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:56.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.890+0000 7f84f4f9a700 1 --2- 172.21.15.98:0/4068684768 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f84d40684a0 0x7f84d406a950 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f84f00a12f0 tx=0x7f84e0007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:56.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:56.893+0000 7f84e5ffb700 1 -- 172.21.15.98:0/4068684768 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f84ec053b50 con 0x7f84f00b4a00 2023-03-03T08:01:57.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.128+0000 7f84f679d700 1 -- 172.21.15.98:0/4068684768 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84f00729b0 con 0x7f84f00b4a00 2023-03-03T08:01:57.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.129+0000 7f84e5ffb700 1 -- 172.21.15.98:0/4068684768 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f84ec057170 con 0x7f84f00b4a00 2023-03-03T08:01:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:57.131 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6050,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/318891906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":318891906},{"type":"v1","addr":"172.21.15.98:6836","nonce":318891906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1280578660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1280578660},{"type":"v1","addr":"172.21.15.98:6837","nonce":1280578660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6906,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1395511883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1395511883},{"type":"v1","addr":"172.21.15.100:6834","nonce":1395511883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:01:21.379777+0000","modified":"2023-03-03T08:01:24.835314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6909},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6909":{"gid":6909,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/341424177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":341424177},{"type":"v1","addr":"172.21.15.100:6835","nonce":341424177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-03-03T08:01:57.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.131+0000 7f84f679d700 1 -- 172.21.15.98:0/4068684768 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f84d40684a0 msgr2=0x7f84d406a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:57.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.131+0000 7f84f679d700 1 --2- 172.21.15.98:0/4068684768 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f84d40684a0 0x7f84d406a950 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f84f00a12f0 tx=0x7f84e0007de0 comp rx=0 tx=0).stop 2023-03-03T08:01:57.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.131+0000 7f84f679d700 1 -- 172.21.15.98:0/4068684768 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84f00b4a00 msgr2=0x7f84f00c7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:57.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.131+0000 7f84f679d700 1 --2- 172.21.15.98:0/4068684768 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84f00b4a00 0x7f84f00c7eb0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f84ec000c00 tx=0x7f84ec004750 comp rx=0 tx=0).stop 2023-03-03T08:01:57.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.131+0000 7f84f679d700 1 -- 172.21.15.98:0/4068684768 shutdown_connections 2023-03-03T08:01:57.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.131+0000 7f84f679d700 1 --2- 172.21.15.98:0/4068684768 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f84d40684a0 0x7f84d406a950 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:57.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.131+0000 7f84f679d700 1 --2- 172.21.15.98:0/4068684768 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84f00d1650 0x7f84f00c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:57.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.131+0000 7f84f679d700 1 --2- 172.21.15.98:0/4068684768 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84f00b4a00 0x7f84f00c7eb0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:57.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.131+0000 7f84f679d700 1 --2- 172.21.15.98:0/4068684768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f00d0010 0x7f84f01e5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:57.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.131+0000 7f84f679d700 1 -- 172.21.15.98:0/4068684768 >> 172.21.15.98:0/4068684768 conn(0x7f84f0073b00 msgr2=0x7f84f00b4250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:57.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.131+0000 7f84f679d700 1 -- 172.21.15.98:0/4068684768 shutdown_connections 2023-03-03T08:01:57.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.132+0000 7f84f679d700 1 -- 172.21.15.98:0/4068684768 wait complete. 2023-03-03T08:01:57.136 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2023-03-03T08:01:57.161 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2023-03-03T08:01:57.436 INFO:teuthology.orchestra.run.smithi100.stdout:[{"id":6266,"entity":{"name":{"type":"client","num":6266},"addr":{"type":"any","addr":"172.21.15.98:0","nonce":2644355772}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":14,"uptime":0.85097748500000003,"requests_in_flight":2,"num_completed_requests":0,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.957524538935555,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.6266 172.21.15.98:0/2644355772","completed_requests":[],"prealloc_inos":[{"start":"0x10000000005","length":996}],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"y","hostname":"smithi098","pid":"108777","root":"/"}}] 2023-03-03T08:01:57.437 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "y", "hostname": "smithi098", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108777", "root": "/" }, "completed_requests": [], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.98:0", "nonce": 2644355772, "type": "any" }, "name": { "num": 6266, "type": "client" } }, "id": 6266, "inst": "client.6266 172.21.15.98:0/2644355772", "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 996, "start": "0x10000000005" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 14, "requests_in_flight": 2, "session_cache_liveness": { "halflife": 300, "value": 21.957524538935555 }, "state": "open", "uptime": 0.850977485 } ] 2023-03-03T08:01:57.437 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2023-03-03T08:01:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.682+0000 7f7e849ff700 1 Processor -- start 2023-03-03T08:01:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.683+0000 7f7e849ff700 1 -- start start 2023-03-03T08:01:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.684+0000 7f7e849ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7e800b8b90 0x7f7e800cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.684+0000 7f7e849ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e800b5410 0x7f7e800d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.684+0000 7f7e849ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e800d0ef0 con 0x7f7e800b87c0 2023-03-03T08:01:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.684+0000 7f7e849ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e800d0600 con 0x7f7e800b8b90 2023-03-03T08:01:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.684+0000 7f7e849ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e800cd970 con 0x7f7e800b5410 2023-03-03T08:01:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.684+0000 7f7e7dd9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7e800b8b90 0x7f7e800cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.685+0000 7f7e7ed9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e800b5410 0x7f7e800d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.685+0000 7f7e7e59c700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f7e800b87c0 0x7f7e800b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:58440/0 (socket says 172.21.15.98:58440) 2023-03-03T08:01:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.685+0000 7f7e7e59c700 1 -- 172.21.15.98:0/2154228118 learned_addr learned my addr 172.21.15.98:0/2154228118 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:01:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.685+0000 7f7e7dd9b700 1 -- 172.21.15.98:0/2154228118 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e800b5410 msgr2=0x7f7e800d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.685+0000 7f7e7dd9b700 1 --2- 172.21.15.98:0/2154228118 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e800b5410 0x7f7e800d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.685+0000 7f7e7dd9b700 1 -- 172.21.15.98:0/2154228118 >> v1:172.21.15.100:6789/0 conn(0x7f7e800b87c0 legacy=0x7f7e800b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.685+0000 7f7e7dd9b700 1 -- 172.21.15.98:0/2154228118 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e800cdc50 con 0x7f7e800b8b90 2023-03-03T08:01:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.685+0000 7f7e7dd9b700 1 --2- 172.21.15.98:0/2154228118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e800b8b90 0x7f7e800cffe0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009840 tx=0x7f7e74009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451080ffe8c955cb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.686+0000 7f7e7d59a700 1 -- 172.21.15.98:0/2154228118 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e74011830 con 0x7f7e800b8b90 2023-03-03T08:01:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.686+0000 7f7e7d59a700 1 -- 172.21.15.98:0/2154228118 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e74011990 con 0x7f7e800b8b90 2023-03-03T08:01:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.686+0000 7f7e7d59a700 1 -- 172.21.15.98:0/2154228118 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e74011d20 con 0x7f7e800b8b90 2023-03-03T08:01:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.686+0000 7f7e849ff700 1 -- 172.21.15.98:0/2154228118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e800b8b90 msgr2=0x7f7e800cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.686+0000 7f7e849ff700 1 --2- 172.21.15.98:0/2154228118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e800b8b90 0x7f7e800cffe0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009840 tx=0x7f7e74009b50 comp rx=0 tx=0).stop 2023-03-03T08:01:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.686+0000 7f7e849ff700 1 -- 172.21.15.98:0/2154228118 shutdown_connections 2023-03-03T08:01:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.686+0000 7f7e849ff700 1 --2- 172.21.15.98:0/2154228118 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e800b5410 0x7f7e800d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.686+0000 7f7e849ff700 1 --2- 172.21.15.98:0/2154228118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e800b8b90 0x7f7e800cffe0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.686+0000 7f7e849ff700 1 -- 172.21.15.98:0/2154228118 >> 172.21.15.98:0/2154228118 conn(0x7f7e8006aee0 msgr2=0x7f7e8006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.686+0000 7f7e849ff700 1 -- 172.21.15.98:0/2154228118 shutdown_connections 2023-03-03T08:01:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.686+0000 7f7e849ff700 1 -- 172.21.15.98:0/2154228118 wait complete. 2023-03-03T08:01:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.688+0000 7f7e849ff700 1 Processor -- start 2023-03-03T08:01:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.689+0000 7f7e849ff700 1 -- start start 2023-03-03T08:01:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.689+0000 7f7e849ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e800b5410 0x7f7e8015e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.690+0000 7f7e849ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e800b87c0 0x7f7e80156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.690+0000 7f7e849ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e800b8b90 0x7f7e8015bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.690+0000 7f7e849ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8015e4f0 con 0x7f7e800b5410 2023-03-03T08:01:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.690+0000 7f7e849ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e800ca350 con 0x7f7e800b87c0 2023-03-03T08:01:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.690+0000 7f7e849ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e800ca630 con 0x7f7e800b8b90 2023-03-03T08:01:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.690+0000 7f7e7e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e800b5410 0x7f7e8015e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.690+0000 7f7e7e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e800b5410 0x7f7e8015e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47832/0 (socket says 172.21.15.98:47832) 2023-03-03T08:01:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.690+0000 7f7e7e59c700 1 -- 172.21.15.98:0/1394485955 learned_addr learned my addr 172.21.15.98:0/1394485955 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.690+0000 7f7e7ed9d700 1 --2- 172.21.15.98:0/1394485955 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e800b8b90 0x7f7e8015bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.690+0000 7f7e7dd9b700 1 --2- 172.21.15.98:0/1394485955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e800b87c0 0x7f7e80156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.690+0000 7f7e7e59c700 1 -- 172.21.15.98:0/1394485955 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e800b8b90 msgr2=0x7f7e8015bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.690+0000 7f7e7e59c700 1 --2- 172.21.15.98:0/1394485955 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e800b8b90 0x7f7e8015bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.690+0000 7f7e7e59c700 1 -- 172.21.15.98:0/1394485955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e800b87c0 msgr2=0x7f7e80156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.690+0000 7f7e7e59c700 1 --2- 172.21.15.98:0/1394485955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e800b87c0 0x7f7e80156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.690+0000 7f7e7e59c700 1 -- 172.21.15.98:0/1394485955 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e740094a0 con 0x7f7e800b5410 2023-03-03T08:01:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.691+0000 7f7e7e59c700 1 --2- 172.21.15.98:0/1394485955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e800b5410 0x7f7e8015e7d0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000c520 tx=0x7f7e7000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.691+0000 7f7e6effd700 1 -- 172.21.15.98:0/1394485955 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e70005d70 con 0x7f7e800b5410 2023-03-03T08:01:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.691+0000 7f7e6effd700 1 -- 172.21.15.98:0/1394485955 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e700070d0 con 0x7f7e800b5410 2023-03-03T08:01:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.691+0000 7f7e6effd700 1 -- 172.21.15.98:0/1394485955 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e70007440 con 0x7f7e800b5410 2023-03-03T08:01:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.691+0000 7f7e849ff700 1 -- 172.21.15.98:0/1394485955 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e800cdf30 con 0x7f7e800b5410 2023-03-03T08:01:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.691+0000 7f7e849ff700 1 -- 172.21.15.98:0/1394485955 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e80073a80 con 0x7f7e800b5410 2023-03-03T08:01:57.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.692+0000 7f7e849ff700 1 -- 172.21.15.98:0/1394485955 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e800ce0c0 con 0x7f7e800b5410 2023-03-03T08:01:57.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.693+0000 7f7e6effd700 1 -- 172.21.15.98:0/1394485955 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 86655+0+0 (secure 0 0 0) 0x7f7e70019030 con 0x7f7e800b5410 2023-03-03T08:01:57.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.695+0000 7f7e6effd700 1 --2- 172.21.15.98:0/1394485955 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7e600684a0 0x7f7e6006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:57.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.695+0000 7f7e6effd700 1 -- 172.21.15.98:0/1394485955 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7e70085190 con 0x7f7e800b5410 2023-03-03T08:01:57.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.695+0000 7f7e7dd9b700 1 --2- 172.21.15.98:0/1394485955 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7e600684a0 0x7f7e6006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:57.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.695+0000 7f7e7dd9b700 1 --2- 172.21.15.98:0/1394485955 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7e600684a0 0x7f7e6006a950 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7e800b8fd0 tx=0x7f7e7401c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.699+0000 7f7e6effd700 1 -- 172.21.15.98:0/1394485955 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7e7004b5e0 con 0x7f7e800b5410 2023-03-03T08:01:57.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.940+0000 7f7e849ff700 1 -- 172.21.15.98:0/1394485955 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f7e800729b0 con 0x7f7e800b5410 2023-03-03T08:01:57.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.948+0000 7f7e6effd700 1 -- 172.21.15.98:0/1394485955 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v11) v1 ==== 81+0+455804 (secure 0 0 0) 0x7f7e70051800 con 0x7f7e800b5410 2023-03-03T08:01:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:01:57.957 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:01:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 11, 2023-03-03T08:01:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: "active_gid": 4110, 2023-03-03T08:01:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: "active_name": "y", 2023-03-03T08:01:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: "active_addrs": { 2023-03-03T08:01:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-03-03T08:01:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-03-03T08:01:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6832", 2023-03-03T08:01:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 108777 2023-03-03T08:01:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T08:01:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6833", 2023-03-03T08:01:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 108777 2023-03-03T08:01:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:01:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: "active_addr": "172.21.15.98:6833/108777", 2023-03-03T08:01:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: "active_change": "2023-03-03T07:55:26.151884+0000", 2023-03-03T08:01:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: "active_mgr_features": 4540138322906710015, 2023-03-03T08:01:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: "available": true, 2023-03-03T08:01:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: "standbys": [ 2023-03-03T08:01:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 4107, 2023-03-03T08:01:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "x", 2023-03-03T08:01:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_features": 4540138322906710015, 2023-03-03T08:01:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2023-03-03T08:01:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2023-03-03T08:01:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2023-03-03T08:01:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2023-03-03T08:01:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:57.962 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.962 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.962 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2023-03-03T08:01:57.962 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.962 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.962 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.962 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2023-03-03T08:01:57.963 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:57.963 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.963 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.963 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:57.963 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:57.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:57.963 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.964 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.964 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:57.964 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.964 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.964 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:57.964 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:57.965 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:57.965 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:57.965 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:57.965 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:57.965 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:57.965 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:57.965 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:57.966 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:57.966 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.966 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.966 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:57.966 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:57.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:57.966 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.967 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.967 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:57.967 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.967 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.967 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.967 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:57.967 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:57.967 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.968 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.968 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:57.968 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:57.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:57.968 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.968 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.968 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:57.969 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.969 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.969 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:57.969 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:57.969 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:57.969 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:57.969 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:57.970 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:57.970 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:57.970 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:57.970 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:57.970 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:57.970 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.970 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.971 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:57.971 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:57.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:57.971 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.971 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.971 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:57.971 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.972 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.972 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.972 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:57.972 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:57.972 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.972 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.973 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2023-03-03T08:01:57.973 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2023-03-03T08:01:57.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:57.973 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.973 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.973 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:57.973 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.973 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.974 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.974 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2023-03-03T08:01:57.974 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:57.974 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.974 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.974 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2023-03-03T08:01:57.974 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2023-03-03T08:01:57.975 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:57.975 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.975 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.975 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2023-03-03T08:01:57.975 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.975 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.975 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.976 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2023-03-03T08:01:57.976 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:57.976 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.976 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.976 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2023-03-03T08:01:57.976 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2023-03-03T08:01:57.977 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:57.977 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.977 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.977 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:57.977 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.977 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.977 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.978 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2023-03-03T08:01:57.978 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:57.978 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.978 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.978 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2023-03-03T08:01:57.978 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2023-03-03T08:01:57.978 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:57.979 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.979 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.979 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:57.979 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.979 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.979 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.979 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2023-03-03T08:01:57.979 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:57.980 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.980 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.980 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2023-03-03T08:01:57.980 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2023-03-03T08:01:57.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:57.980 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.980 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.981 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2023-03-03T08:01:57.981 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.981 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.981 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.981 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2023-03-03T08:01:57.981 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:57.981 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.982 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.982 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2023-03-03T08:01:57.982 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2023-03-03T08:01:57.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:57.982 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.983 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.983 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:57.983 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.983 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.983 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.983 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2023-03-03T08:01:57.983 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:57.983 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.984 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.984 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2023-03-03T08:01:57.984 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2023-03-03T08:01:57.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:57.984 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.984 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.985 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:57.985 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.985 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.985 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.985 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2023-03-03T08:01:57.985 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:57.985 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.985 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.986 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2023-03-03T08:01:57.986 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2023-03-03T08:01:57.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:57.986 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.986 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.986 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:57.987 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.987 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.987 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.987 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2023-03-03T08:01:57.987 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:57.987 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.987 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.987 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:57.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:57.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:57.988 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2023-03-03T08:01:57.988 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:57.988 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:57.988 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:57.988 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2023-03-03T08:01:57.989 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2023-03-03T08:01:57.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:57.989 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.989 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.989 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:57.989 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.989 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.989 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.990 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2023-03-03T08:01:57.990 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:57.990 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.990 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.990 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2023-03-03T08:01:57.990 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2023-03-03T08:01:57.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:57.991 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.991 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.991 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2023-03-03T08:01:57.991 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.991 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.991 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.991 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2023-03-03T08:01:57.992 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-03-03T08:01:57.992 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.992 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.992 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2023-03-03T08:01:57.992 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2023-03-03T08:01:57.992 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2023-03-03T08:01:57.992 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.993 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.993 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2023-03-03T08:01:57.993 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2023-03-03T08:01:57.993 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2023-03-03T08:01:57.993 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.993 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-03-03T08:01:57.993 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-03-03T08:01:57.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.994 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.994 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2023-03-03T08:01:57.994 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2023-03-03T08:01:57.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2023-03-03T08:01:57.994 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.995 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.995 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2023-03-03T08:01:57.995 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2023-03-03T08:01:57.995 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2023-03-03T08:01:57.995 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.995 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-03-03T08:01:57.995 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:57.995 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.996 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.996 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2023-03-03T08:01:57.996 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2023-03-03T08:01:57.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:57.996 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.996 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.996 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2023-03-03T08:01:57.997 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:57.997 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:57.997 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.997 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-03-03T08:01:57.997 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-03-03T08:01:57.997 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.997 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:57.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:57.998 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2023-03-03T08:01:57.998 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2023-03-03T08:01:57.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:57.998 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:57.998 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:57.998 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2023-03-03T08:01:57.998 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2023-03-03T08:01:57.998 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2023-03-03T08:01:57.999 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:57.999 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2023-03-03T08:01:57.999 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-03-03T08:01:57.999 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:57.999 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.000 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2023-03-03T08:01:58.000 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2023-03-03T08:01:58.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.000 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.000 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.000 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2023-03-03T08:01:58.001 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.001 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.001 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.001 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2023-03-03T08:01:58.001 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-03-03T08:01:58.001 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.001 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.002 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2023-03-03T08:01:58.002 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2023-03-03T08:01:58.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2023-03-03T08:01:58.002 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.002 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.002 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2023-03-03T08:01:58.002 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2023-03-03T08:01:58.003 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2023-03-03T08:01:58.003 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.003 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-03-03T08:01:58.003 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-03-03T08:01:58.003 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.003 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.003 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.004 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.004 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.004 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.004 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.004 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.004 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.004 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.004 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.005 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.005 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.005 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.005 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.005 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.005 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.005 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.006 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.006 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.006 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.006 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.006 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.006 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.007 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.007 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.007 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.007 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.007 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.007 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.008 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.008 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.008 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.008 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.008 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.008 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.008 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.009 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.009 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.009 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.009 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.009 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.009 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.009 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.010 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.010 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.010 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.010 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.010 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.010 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.010 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.010 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.011 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.011 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.011 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.011 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.011 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.011 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.012 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.012 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.012 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.012 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.012 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.012 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.012 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.012 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.013 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.013 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2023-03-03T08:01:58.013 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2023-03-03T08:01:58.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:58.013 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.013 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.014 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2023-03-03T08:01:58.014 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.014 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.014 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.014 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-03-03T08:01:58.014 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.014 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.014 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.015 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2023-03-03T08:01:58.015 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2023-03-03T08:01:58.015 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.015 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.015 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.015 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2023-03-03T08:01:58.016 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.016 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.016 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.016 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2023-03-03T08:01:58.016 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2023-03-03T08:01:58.016 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap" 2023-03-03T08:01:58.016 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.016 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2023-03-03T08:01:58.017 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.017 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.017 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.017 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2023-03-03T08:01:58.017 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2023-03-03T08:01:58.017 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.018 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.018 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.018 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.018 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.018 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.018 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.018 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-03-03T08:01:58.019 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.019 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.019 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.019 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2023-03-03T08:01:58.019 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2023-03-03T08:01:58.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.019 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.020 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.020 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2023-03-03T08:01:58.020 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.020 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.020 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.020 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-03-03T08:01:58.020 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.020 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.021 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.021 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2023-03-03T08:01:58.021 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2023-03-03T08:01:58.021 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.021 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.021 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.022 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2023-03-03T08:01:58.022 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2023-03-03T08:01:58.022 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.022 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.022 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2023-03-03T08:01:58.022 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-03-03T08:01:58.022 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.023 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.023 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2023-03-03T08:01:58.023 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2023-03-03T08:01:58.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2023-03-03T08:01:58.023 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.023 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.024 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2023-03-03T08:01:58.024 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.024 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.024 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.024 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-03-03T08:01:58.024 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.024 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.025 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.025 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.025 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.025 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2023-03-03T08:01:58.025 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.025 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.026 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.026 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2023-03-03T08:01:58.026 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2023-03-03T08:01:58.026 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:58.026 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.026 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.026 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2023-03-03T08:01:58.027 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.027 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.027 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.027 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-03-03T08:01:58.027 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.027 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.027 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.028 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2023-03-03T08:01:58.028 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2023-03-03T08:01:58.028 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.028 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.028 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.028 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2023-03-03T08:01:58.028 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.029 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.029 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.029 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-03-03T08:01:58.029 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.029 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.029 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.029 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2023-03-03T08:01:58.030 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2023-03-03T08:01:58.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.030 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.030 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.030 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2023-03-03T08:01:58.030 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.030 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.031 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.031 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-03-03T08:01:58.031 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.031 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.031 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.031 INFO:teuthology.orchestra.run.smithi098.stdout: "alertmanager_web_password": { 2023-03-03T08:01:58.032 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alertmanager_web_password", 2023-03-03T08:01:58.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.032 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.032 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.032 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2023-03-03T08:01:58.032 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.032 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.033 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.033 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Alertmanager web password", 2023-03-03T08:01:58.033 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.033 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.033 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.033 INFO:teuthology.orchestra.run.smithi098.stdout: "alertmanager_web_user": { 2023-03-03T08:01:58.033 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alertmanager_web_user", 2023-03-03T08:01:58.034 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.034 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.034 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.034 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2023-03-03T08:01:58.034 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.035 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.035 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.035 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Alertmanager web user", 2023-03-03T08:01:58.035 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.035 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.035 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.036 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2023-03-03T08:01:58.036 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2023-03-03T08:01:58.036 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.036 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.036 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.036 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.037 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.037 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.037 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.037 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-03-03T08:01:58.037 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-03-03T08:01:58.037 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.037 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.038 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2023-03-03T08:01:58.038 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2023-03-03T08:01:58.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.038 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.038 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.038 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2023-03-03T08:01:58.039 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.039 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.039 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.039 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2023-03-03T08:01:58.039 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.039 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.039 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.040 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2023-03-03T08:01:58.040 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2023-03-03T08:01:58.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:58.040 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.040 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.040 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2023-03-03T08:01:58.040 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.040 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.041 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.041 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-03-03T08:01:58.041 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.041 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.041 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.041 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2023-03-03T08:01:58.042 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2023-03-03T08:01:58.042 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.042 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.042 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.042 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.042 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.042 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.042 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.043 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-03-03T08:01:58.043 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.043 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.043 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.043 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2023-03-03T08:01:58.043 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2023-03-03T08:01:58.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.044 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.044 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.044 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.044 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.044 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.044 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.045 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-03-03T08:01:58.045 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.045 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.045 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.045 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2023-03-03T08:01:58.045 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2023-03-03T08:01:58.045 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.046 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.046 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.046 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.046 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.046 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.046 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.046 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-03-03T08:01:58.047 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.047 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.047 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.047 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2023-03-03T08:01:58.047 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2023-03-03T08:01:58.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.047 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.048 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.048 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.23.0", 2023-03-03T08:01:58.048 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.048 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.048 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.048 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2023-03-03T08:01:58.048 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.049 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.049 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.049 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2023-03-03T08:01:58.049 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2023-03-03T08:01:58.049 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.049 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.049 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.050 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2023-03-03T08:01:58.050 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.050 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.050 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.050 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2023-03-03T08:01:58.050 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.050 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.050 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.051 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2023-03-03T08:01:58.051 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2023-03-03T08:01:58.051 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.051 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.051 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.051 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-03-03T08:01:58.051 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2023-03-03T08:01:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2023-03-03T08:01:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2023-03-03T08:01:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph-grafana:8.3.5", 2023-03-03T08:01:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2023-03-03T08:01:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2023-03-03T08:01:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2023-03-03T08:01:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-03-03T08:01:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2023-03-03T08:01:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2023-03-03T08:01:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2023-03-03T08:01:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-03-03T08:01:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2023-03-03T08:01:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2023-03-03T08:01:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2023-03-03T08:01:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-03-03T08:01:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2023-03-03T08:01:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2023-03-03T08:01:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2023-03-03T08:01:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-03-03T08:01:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2023-03-03T08:01:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2023-03-03T08:01:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2023-03-03T08:01:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.1.5", 2023-03-03T08:01:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2023-03-03T08:01:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2023-03-03T08:01:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2023-03-03T08:01:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-03-03T08:01:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2023-03-03T08:01:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2023-03-03T08:01:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2023-03-03T08:01:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.3.1", 2023-03-03T08:01:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2023-03-03T08:01:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2023-03-03T08:01:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2023-03-03T08:01:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.33.4", 2023-03-03T08:01:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2023-03-03T08:01:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2023-03-03T08:01:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2023-03-03T08:01:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-03-03T08:01:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2023-03-03T08:01:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2023-03-03T08:01:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2023-03-03T08:01:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-03-03T08:01:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2023-03-03T08:01:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2023-03-03T08:01:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2023-03-03T08:01:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2023-03-03T08:01:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2023-03-03T08:01:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2023-03-03T08:01:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2023-03-03T08:01:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-03-03T08:01:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.077 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.077 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.077 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2023-03-03T08:01:58.077 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2023-03-03T08:01:58.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.077 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.078 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.078 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2023-03-03T08:01:58.078 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.078 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.078 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.078 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-03-03T08:01:58.078 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.078 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.079 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.079 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2023-03-03T08:01:58.079 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2023-03-03T08:01:58.079 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.079 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.079 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.080 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2023-03-03T08:01:58.080 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.080 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.080 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.080 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2023-03-03T08:01:58.080 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.080 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.080 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.081 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2023-03-03T08:01:58.081 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2023-03-03T08:01:58.081 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.081 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.081 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.081 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.082 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.082 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.082 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.082 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2023-03-03T08:01:58.082 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.082 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.082 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.083 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2023-03-03T08:01:58.083 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2023-03-03T08:01:58.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.083 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.083 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.083 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2023-03-03T08:01:58.083 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.084 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.084 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.084 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2023-03-03T08:01:58.084 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.084 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.085 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.085 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2023-03-03T08:01:58.085 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2023-03-03T08:01:58.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.085 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.085 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.086 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2023-03-03T08:01:58.086 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.086 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.086 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.086 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2023-03-03T08:01:58.086 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.086 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.086 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.087 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.087 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.087 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.087 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.087 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.088 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.088 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.088 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.088 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.088 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.088 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.088 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.089 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.089 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.089 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.089 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.089 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.089 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.089 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.090 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2023-03-03T08:01:58.090 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2023-03-03T08:01:58.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.090 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.090 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.090 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.090 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.091 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.091 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.091 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-03-03T08:01:58.091 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.091 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.091 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.091 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.092 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.092 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.092 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.092 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.092 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.092 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.092 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.092 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.093 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-03-03T08:01:58.093 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.093 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.093 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.093 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.094 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.094 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.094 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.094 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.094 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.094 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.095 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.095 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.095 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.095 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.095 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.095 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.095 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.096 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.096 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.096 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.096 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.096 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.096 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.097 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.097 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.097 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.097 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.097 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.098 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.098 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.098 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.098 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.098 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.098 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2023-03-03T08:01:58.099 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-03-03T08:01:58.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.099 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.099 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.099 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.099 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.100 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.100 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.100 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-03-03T08:01:58.100 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.100 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.100 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-03-03T08:01:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-03-03T08:01:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2023-03-03T08:01:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-03-03T08:01:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2023-03-03T08:01:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2023-03-03T08:01:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2023-03-03T08:01:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2023-03-03T08:01:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2023-03-03T08:01:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2023-03-03T08:01:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2023-03-03T08:01:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-03-03T08:01:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2023-03-03T08:01:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2023-03-03T08:01:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2023-03-03T08:01:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2023-03-03T08:01:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2023-03-03T08:01:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2023-03-03T08:01:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2023-03-03T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2023-03-03T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2023-03-03T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2023-03-03T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2023-03-03T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-03-03T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-03-03T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_web_password": { 2023-03-03T08:01:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_web_password", 2023-03-03T08:01:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2023-03-03T08:01:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus web password", 2023-03-03T08:01:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_web_user": { 2023-03-03T08:01:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_web_user", 2023-03-03T08:01:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2023-03-03T08:01:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus web user", 2023-03-03T08:01:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2023-03-03T08:01:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2023-03-03T08:01:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-03-03T08:01:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2023-03-03T08:01:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2023-03-03T08:01:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-03-03T08:01:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2023-03-03T08:01:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2023-03-03T08:01:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-03-03T08:01:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2023-03-03T08:01:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2023-03-03T08:01:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-03-03T08:01:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2023-03-03T08:01:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2023-03-03T08:01:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-03-03T08:01:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2023-03-03T08:01:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2023-03-03T08:01:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2023-03-03T08:01:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2023-03-03T08:01:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2023-03-03T08:01:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2023-03-03T08:01:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-03-03T08:01:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2023-03-03T08:01:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2023-03-03T08:01:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-03-03T08:01:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2023-03-03T08:01:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2023-03-03T08:01:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-03-03T08:01:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2023-03-03T08:01:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2023-03-03T08:01:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2023-03-03T08:01:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2023-03-03T08:01:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2023-03-03T08:01:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-03-03T08:01:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2023-03-03T08:01:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2023-03-03T08:01:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-03-03T08:01:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2023-03-03T08:01:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2023-03-03T08:01:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2023-03-03T08:01:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2023-03-03T08:01:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2023-03-03T08:01:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2023-03-03T08:01:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2023-03-03T08:01:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2023-03-03T08:01:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2023-03-03T08:01:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2023-03-03T08:01:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-03-03T08:01:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-03-03T08:01:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2023-03-03T08:01:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2023-03-03T08:01:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2023-03-03T08:01:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-03-03T08:01:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-03-03T08:01:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2023-03-03T08:01:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2023-03-03T08:01:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-03-03T08:01:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-03-03T08:01:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2023-03-03T08:01:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2023-03-03T08:01:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-03-03T08:01:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-03-03T08:01:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-03-03T08:01:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-03-03T08:01:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-03-03T08:01:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-03-03T08:01:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2023-03-03T08:01:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-03-03T08:01:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2023-03-03T08:01:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-03-03T08:01:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2023-03-03T08:01:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-03-03T08:01:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-03-03T08:01:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-03-03T08:01:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2023-03-03T08:01:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2023-03-03T08:01:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2023-03-03T08:01:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-03-03T08:01:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-03-03T08:01:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2023-03-03T08:01:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2023-03-03T08:01:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2023-03-03T08:01:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2023-03-03T08:01:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2023-03-03T08:01:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-03-03T08:01:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-03-03T08:01:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-03-03T08:01:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-03-03T08:01:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-03-03T08:01:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-03-03T08:01:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2023-03-03T08:01:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-03-03T08:01:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2023-03-03T08:01:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2023-03-03T08:01:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-03-03T08:01:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-03-03T08:01:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-03-03T08:01:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-03-03T08:01:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-03-03T08:01:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-03-03T08:01:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-03-03T08:01:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-03-03T08:01:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-03-03T08:01:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-03-03T08:01:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-03-03T08:01:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-03-03T08:01:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-03-03T08:01:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-03-03T08:01:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-03-03T08:01:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-03-03T08:01:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2023-03-03T08:01:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2023-03-03T08:01:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-03-03T08:01:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-03-03T08:01:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-03-03T08:01:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-03-03T08:01:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-03-03T08:01:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2023-03-03T08:01:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-03-03T08:01:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-03-03T08:01:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2023-03-03T08:01:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2023-03-03T08:01:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-03-03T08:01:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2023-03-03T08:01:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2023-03-03T08:01:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2023-03-03T08:01:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-03-03T08:01:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-03-03T08:01:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2023-03-03T08:01:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2023-03-03T08:01:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2023-03-03T08:01:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2023-03-03T08:01:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2023-03-03T08:01:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-03-03T08:01:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-03-03T08:01:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2023-03-03T08:01:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-03-03T08:01:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-03-03T08:01:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2023-03-03T08:01:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-03-03T08:01:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-03-03T08:01:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2023-03-03T08:01:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2023-03-03T08:01:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2023-03-03T08:01:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2023-03-03T08:01:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2023-03-03T08:01:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.236 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.236 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.236 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.236 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.236 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.236 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.236 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.236 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.237 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.237 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2023-03-03T08:01:58.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2023-03-03T08:01:58.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.237 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.237 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.237 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.238 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.238 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.238 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.238 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2023-03-03T08:01:58.238 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.238 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.238 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.239 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2023-03-03T08:01:58.239 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2023-03-03T08:01:58.239 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.239 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.239 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.239 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2023-03-03T08:01:58.239 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.240 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.240 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.240 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.240 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.240 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.240 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.240 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2023-03-03T08:01:58.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2023-03-03T08:01:58.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.241 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.241 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.241 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.241 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.241 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.242 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.242 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.242 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.242 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.242 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.242 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.242 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.243 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.243 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.243 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.243 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.243 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.243 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.243 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.244 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.244 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.244 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.244 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.244 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.244 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.244 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.245 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.245 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.245 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.245 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.245 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.246 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.246 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.246 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.246 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.246 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.246 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.246 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.247 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.247 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.247 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.248 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.248 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.248 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.248 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.248 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.248 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.249 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.249 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.249 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.249 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.249 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.249 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.249 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.250 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.250 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.250 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.250 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.250 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.251 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.251 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.251 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.251 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.251 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.251 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.252 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.252 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.252 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.252 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.252 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2023-03-03T08:01:58.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2023-03-03T08:01:58.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.253 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.253 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.253 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.253 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.253 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.253 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.253 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2023-03-03T08:01:58.254 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.254 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.254 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.254 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2023-03-03T08:01:58.254 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2023-03-03T08:01:58.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.254 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.255 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.255 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.255 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.255 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.255 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.255 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.255 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.256 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.256 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.256 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2023-03-03T08:01:58.256 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2023-03-03T08:01:58.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.256 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.256 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.257 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2023-03-03T08:01:58.257 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.257 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.257 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.257 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.257 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.257 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.257 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.258 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2023-03-03T08:01:58.258 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2023-03-03T08:01:58.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.258 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.258 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.258 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2023-03-03T08:01:58.259 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.259 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.259 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.259 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.259 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.259 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.259 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.260 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2023-03-03T08:01:58.260 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2023-03-03T08:01:58.260 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.260 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.260 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.260 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.260 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.260 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.261 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.261 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.261 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.261 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.261 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.261 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2023-03-03T08:01:58.261 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2023-03-03T08:01:58.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.262 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.262 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2023-03-03T08:01:58.262 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.262 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.262 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.263 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.263 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.263 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.263 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.263 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2023-03-03T08:01:58.263 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2023-03-03T08:01:58.263 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.264 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.264 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.264 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2023-03-03T08:01:58.264 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.264 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.264 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.264 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.265 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2023-03-03T08:01:58.265 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.265 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.265 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.265 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.265 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.265 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2023-03-03T08:01:58.266 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2023-03-03T08:01:58.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.266 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.266 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.266 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2023-03-03T08:01:58.266 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2023-03-03T08:01:58.266 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2023-03-03T08:01:58.266 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.267 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.267 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.267 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.267 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.267 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2023-03-03T08:01:58.267 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2023-03-03T08:01:58.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.268 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.268 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.268 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.268 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.268 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.268 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.268 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.269 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.269 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.269 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.269 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.269 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.269 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2023-03-03T08:01:58.270 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.270 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.270 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.270 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2023-03-03T08:01:58.270 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2023-03-03T08:01:58.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.270 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.271 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.271 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.271 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.271 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.271 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.271 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2023-03-03T08:01:58.271 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.271 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.272 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.272 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.272 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.272 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.272 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.273 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.273 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.273 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.273 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.273 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.273 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.273 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.273 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.274 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.274 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.274 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.274 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.274 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.274 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.274 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.275 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.275 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.275 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.275 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.276 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.276 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.276 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.276 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.276 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.276 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.276 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.277 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.277 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.277 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.277 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.277 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.277 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.277 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.278 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.278 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.278 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.278 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.278 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.278 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.278 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.279 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.279 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.279 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.279 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.279 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.279 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.279 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.280 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.280 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.280 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.280 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.280 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.280 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.280 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.281 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.281 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.281 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.281 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2023-03-03T08:01:58.281 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2023-03-03T08:01:58.281 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.281 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.282 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.282 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2023-03-03T08:01:58.282 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.282 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.282 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.282 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-03-03T08:01:58.282 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.283 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.283 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.283 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2023-03-03T08:01:58.283 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2023-03-03T08:01:58.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.283 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.283 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.284 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2023-03-03T08:01:58.284 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.284 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.284 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.284 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2023-03-03T08:01:58.284 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.284 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.284 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.285 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2023-03-03T08:01:58.285 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2023-03-03T08:01:58.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.285 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.285 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.285 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2023-03-03T08:01:58.285 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.286 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.286 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.286 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2023-03-03T08:01:58.286 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.286 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.286 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.287 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2023-03-03T08:01:58.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2023-03-03T08:01:58.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.287 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.287 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.287 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2023-03-03T08:01:58.287 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.287 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.288 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.288 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2023-03-03T08:01:58.288 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.288 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.288 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.288 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2023-03-03T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2023-03-03T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.290 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-03-03T08:01:58.290 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.290 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.290 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.290 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2023-03-03T08:01:58.290 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2023-03-03T08:01:58.290 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.291 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.291 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.291 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2023-03-03T08:01:58.291 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.291 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.291 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.291 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2023-03-03T08:01:58.292 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.292 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.292 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.292 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2023-03-03T08:01:58.292 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2023-03-03T08:01:58.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.292 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.293 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.293 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2023-03-03T08:01:58.293 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.293 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.293 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.293 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-03-03T08:01:58.293 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.294 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.294 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.294 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.294 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.294 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2023-03-03T08:01:58.294 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.295 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.295 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.295 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.295 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.295 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.295 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.295 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.296 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.296 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.296 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.296 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.296 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.296 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.296 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.296 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.297 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.297 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.297 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.297 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.297 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.297 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.298 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.298 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.298 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.298 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.298 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.298 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.298 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.299 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.299 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.299 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.299 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.299 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.299 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.300 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.300 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.300 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.300 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.300 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.300 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.300 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.301 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.301 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.301 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.301 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.301 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.301 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.301 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.301 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2023-03-03T08:01:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2023-03-03T08:01:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2023-03-03T08:01:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2023-03-03T08:01:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2023-03-03T08:01:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2023-03-03T08:01:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2023-03-03T08:01:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2023-03-03T08:01:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2023-03-03T08:01:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2023-03-03T08:01:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2023-03-03T08:01:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2023-03-03T08:01:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2023-03-03T08:01:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2023-03-03T08:01:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2023-03-03T08:01:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-03-03T08:01:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2023-03-03T08:01:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2023-03-03T08:01:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2023-03-03T08:01:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-03-03T08:01:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2023-03-03T08:01:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2023-03-03T08:01:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2023-03-03T08:01:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2023-03-03T08:01:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2023-03-03T08:01:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2023-03-03T08:01:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2023-03-03T08:01:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-03-03T08:01:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.325 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.325 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.325 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.325 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.325 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.325 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.325 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.326 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.326 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.326 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.326 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.326 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.326 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.327 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.327 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.327 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.327 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.327 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.327 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2023-03-03T08:01:58.327 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2023-03-03T08:01:58.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.328 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.328 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.328 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.328 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.328 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.328 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.329 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2023-03-03T08:01:58.329 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.329 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.329 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.329 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2023-03-03T08:01:58.329 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2023-03-03T08:01:58.330 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.330 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.330 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.330 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2023-03-03T08:01:58.330 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.330 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.330 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.331 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2023-03-03T08:01:58.331 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.331 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.331 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.331 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2023-03-03T08:01:58.331 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2023-03-03T08:01:58.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.332 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.332 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.332 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2023-03-03T08:01:58.332 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.332 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.332 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.333 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-03-03T08:01:58.333 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.333 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.333 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.333 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2023-03-03T08:01:58.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2023-03-03T08:01:58.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.334 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.334 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.334 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2023-03-03T08:01:58.334 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2023-03-03T08:01:58.334 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2023-03-03T08:01:58.335 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.335 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-03-03T08:01:58.335 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.335 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.335 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.335 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2023-03-03T08:01:58.335 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2023-03-03T08:01:58.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.336 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.336 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.336 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.336 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.336 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.336 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.336 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2023-03-03T08:01:58.337 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.337 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.337 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.337 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2023-03-03T08:01:58.337 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2023-03-03T08:01:58.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.338 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.338 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.338 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2023-03-03T08:01:58.338 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.338 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.338 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.338 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-03-03T08:01:58.338 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.339 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.339 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.339 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.339 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.339 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2023-03-03T08:01:58.340 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.340 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.340 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.340 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.340 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.340 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.340 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.340 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.341 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.341 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.341 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.341 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.341 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.341 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.341 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.341 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.342 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.342 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.342 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.342 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.342 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.342 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.342 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.343 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.343 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.343 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.343 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.343 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.343 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.343 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.344 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.344 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.344 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.344 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.344 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.346 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.346 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.346 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.346 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.346 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.346 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.346 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.346 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.347 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.347 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.347 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.347 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.347 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.347 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.347 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.348 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.348 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.348 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.348 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.348 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.348 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.348 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.349 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.349 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.349 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.349 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.349 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.349 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.350 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2023-03-03T08:01:58.350 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.350 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.350 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.350 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.350 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.351 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.351 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.351 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.351 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.351 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.351 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.351 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.352 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.352 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.352 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.352 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.352 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.352 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.352 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.352 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.353 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.353 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.353 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.353 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.353 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.354 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.354 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.354 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.354 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.354 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.354 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.354 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.355 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.355 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.355 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.355 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.355 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.355 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.356 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.356 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.356 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.356 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.356 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.356 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.356 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.356 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.357 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.357 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.357 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.357 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.357 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.357 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.357 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.358 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.358 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.358 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.358 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.358 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.358 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.359 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.359 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.359 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.359 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.359 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.359 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.359 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.360 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2023-03-03T08:01:58.360 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.360 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.360 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.360 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2023-03-03T08:01:58.360 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2023-03-03T08:01:58.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.361 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.361 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.361 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2023-03-03T08:01:58.361 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.361 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.361 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.361 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2023-03-03T08:01:58.362 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-03-03T08:01:58.362 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.362 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.362 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.362 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.362 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.363 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.363 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.363 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.363 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.363 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.363 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.363 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.364 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.364 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.364 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.364 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.364 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.364 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.364 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.365 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.365 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.365 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.365 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.365 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.365 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.366 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.366 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.366 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.366 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.366 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.366 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.366 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.367 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.367 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.367 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.367 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.367 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.368 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.368 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.368 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.368 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.368 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.368 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.368 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.368 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.369 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.369 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.369 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.369 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.370 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.370 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.370 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.370 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.370 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.371 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.371 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.371 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.371 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.371 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.371 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.371 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.371 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.372 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2023-03-03T08:01:58.372 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2023-03-03T08:01:58.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.372 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.372 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.372 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.372 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.373 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.373 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.373 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2023-03-03T08:01:58.373 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-03-03T08:01:58.373 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.373 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2023-03-03T08:01:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2023-03-03T08:01:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2023-03-03T08:01:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.375 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.375 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2023-03-03T08:01:58.375 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.375 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.375 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.375 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2023-03-03T08:01:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2023-03-03T08:01:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2023-03-03T08:01:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2023-03-03T08:01:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2023-03-03T08:01:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2023-03-03T08:01:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2023-03-03T08:01:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2023-03-03T08:01:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2023-03-03T08:01:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2023-03-03T08:01:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2023-03-03T08:01:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-03-03T08:01:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2023-03-03T08:01:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2023-03-03T08:01:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.396 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.396 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.396 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.396 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.396 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.396 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.397 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.397 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.397 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.397 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.397 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.397 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.398 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.398 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.398 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.398 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.398 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.398 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.398 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.400 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.400 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.400 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.400 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.400 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.400 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.401 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.401 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.401 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.401 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.401 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.401 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.401 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.402 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.402 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.402 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.402 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2023-03-03T08:01:58.403 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.403 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.403 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.403 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.403 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.403 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.403 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.404 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.404 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.404 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.404 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.404 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.404 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.405 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.405 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.405 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.405 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.405 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.405 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.405 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.406 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.406 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.406 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.406 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.406 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.407 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.407 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.407 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.407 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.407 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.407 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.407 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.407 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.408 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.408 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.408 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.408 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.408 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.409 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.409 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.409 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.409 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.409 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.409 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.409 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.410 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.410 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.410 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.410 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.410 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.410 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.410 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.410 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.411 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.411 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.411 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.411 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.411 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.412 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.412 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.412 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.412 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.412 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.412 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.412 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.412 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2023-03-03T08:01:58.413 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.413 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.413 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.413 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2023-03-03T08:01:58.414 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2023-03-03T08:01:58.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.414 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.414 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.414 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.414 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.414 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.414 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.415 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-03-03T08:01:58.415 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.415 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.415 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.416 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.416 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.416 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.416 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.416 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.416 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.417 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.417 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.417 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.417 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.417 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.417 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.417 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.418 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.418 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.418 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.418 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.418 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.418 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.419 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.419 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.419 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.419 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.419 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.419 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.419 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.420 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.420 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.420 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.420 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.420 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.421 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.421 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.421 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.421 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.421 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.421 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.422 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.422 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.422 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.422 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.422 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.422 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.422 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.423 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.423 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.423 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.423 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.423 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.423 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.424 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.424 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.424 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.424 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.424 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.424 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.424 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.424 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.425 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2023-03-03T08:01:58.425 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2023-03-03T08:01:58.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.425 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.425 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.425 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.426 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.426 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.426 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.426 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2023-03-03T08:01:58.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2023-03-03T08:01:58.426 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2023-03-03T08:01:58.426 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.426 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2023-03-03T08:01:58.427 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.427 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.427 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.427 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.427 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.428 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2023-03-03T08:01:58.428 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.428 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.428 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.428 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.428 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.429 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.429 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.429 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.429 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.429 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.429 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.429 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.430 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.430 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.430 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.430 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.430 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.430 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.430 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.430 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.431 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.431 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.431 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.431 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.432 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.432 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.432 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.432 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.432 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.432 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.432 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.432 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.433 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.433 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.433 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.433 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.433 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.433 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.434 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.434 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.434 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.434 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.434 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.434 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.434 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.434 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.435 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.435 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.435 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.435 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.435 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.435 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.436 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.436 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.436 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.436 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.436 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.436 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.437 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.437 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.437 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.437 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.437 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.437 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.437 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2023-03-03T08:01:58.438 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.438 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.438 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.438 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.439 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.439 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.439 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.439 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.439 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.439 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.439 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.440 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.440 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.440 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.440 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.440 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.440 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.440 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.441 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.441 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.441 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.441 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.441 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.441 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.442 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.442 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.442 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.442 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.442 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.442 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.443 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.443 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.443 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.443 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.443 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.443 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.444 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.444 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.444 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.444 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.444 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.444 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.444 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.445 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.445 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.445 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.445 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.445 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.445 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.445 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.446 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.446 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.446 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.446 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.446 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.446 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.448 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2023-03-03T08:01:58.448 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.448 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.448 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.449 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.449 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.449 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.449 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.449 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.449 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.450 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.450 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.450 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.450 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.450 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.450 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.450 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.451 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.451 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.451 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.451 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.451 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.451 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.452 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.452 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.452 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.452 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.452 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.452 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.452 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.453 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.453 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.453 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.453 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.454 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.454 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.454 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.454 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.454 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.454 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.455 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.455 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.455 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.455 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.455 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.455 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.455 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.456 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.456 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.456 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.456 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.457 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.457 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.457 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.457 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.457 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.457 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.457 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.458 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.458 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.458 INFO:teuthology.orchestra.run.smithi098.stdout: "noautoscale": { 2023-03-03T08:01:58.458 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "noautoscale", 2023-03-03T08:01:58.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.459 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.459 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.459 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.459 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.459 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.459 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "global autoscale flag", 2023-03-03T08:01:58.460 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-03-03T08:01:58.460 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.460 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.460 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2023-03-03T08:01:58.460 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2023-03-03T08:01:58.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.460 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.461 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.461 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2023-03-03T08:01:58.461 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.461 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.461 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.461 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.461 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.462 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.462 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.462 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2023-03-03T08:01:58.462 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2023-03-03T08:01:58.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:58.462 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.462 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.463 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2023-03-03T08:01:58.463 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2023-03-03T08:01:58.463 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.463 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.463 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2023-03-03T08:01:58.463 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-03-03T08:01:58.463 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.464 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.464 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2023-03-03T08:01:58.464 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.465 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.465 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.465 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2023-03-03T08:01:58.465 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2023-03-03T08:01:58.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.465 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.465 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.466 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.466 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.466 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.466 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.466 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2023-03-03T08:01:58.466 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.467 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.467 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.467 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2023-03-03T08:01:58.467 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2023-03-03T08:01:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.469 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.469 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.469 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.469 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.469 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.469 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.470 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.470 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.470 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.470 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.470 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.470 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.470 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.471 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.471 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.471 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.471 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.471 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.471 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.471 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.471 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.472 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.472 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.472 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.472 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.473 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.473 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.473 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.473 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.473 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.473 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.473 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.473 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.474 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.474 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.474 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.474 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.474 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.475 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.475 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.475 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.475 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.475 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.475 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.475 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.475 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.476 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.476 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.476 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.476 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.476 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.476 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.476 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.477 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.477 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.477 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.477 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.477 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.477 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.478 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.478 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.478 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.478 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.478 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.478 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.478 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2023-03-03T08:01:58.479 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2023-03-03T08:01:58.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.479 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.479 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.479 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2023-03-03T08:01:58.479 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.479 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.480 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.480 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2023-03-03T08:01:58.480 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.480 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.480 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.480 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2023-03-03T08:01:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2023-03-03T08:01:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2023-03-03T08:01:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2023-03-03T08:01:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2023-03-03T08:01:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2023-03-03T08:01:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2023-03-03T08:01:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2023-03-03T08:01:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2023-03-03T08:01:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2023-03-03T08:01:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-03-03T08:01:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2023-03-03T08:01:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2023-03-03T08:01:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2023-03-03T08:01:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2023-03-03T08:01:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2023-03-03T08:01:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2023-03-03T08:01:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2023-03-03T08:01:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-03-03T08:01:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2023-03-03T08:01:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2023-03-03T08:01:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2023-03-03T08:01:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-03-03T08:01:58.504 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.504 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.504 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.504 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2023-03-03T08:01:58.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2023-03-03T08:01:58.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.505 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.505 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.505 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2023-03-03T08:01:58.505 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.505 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.505 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.505 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.505 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.506 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.506 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.506 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2023-03-03T08:01:58.506 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2023-03-03T08:01:58.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.507 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.507 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.507 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2023-03-03T08:01:58.507 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.507 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.507 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.507 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2023-03-03T08:01:58.508 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2023-03-03T08:01:58.508 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.508 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.508 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.508 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.508 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.509 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2023-03-03T08:01:58.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2023-03-03T08:01:58.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.509 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.509 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.509 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2023-03-03T08:01:58.509 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2023-03-03T08:01:58.510 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2023-03-03T08:01:58.510 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.510 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.510 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.510 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.510 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2023-03-03T08:01:58.511 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.511 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.511 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.512 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.512 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.512 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.512 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.512 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.512 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.512 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.513 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.513 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.513 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.513 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.513 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.513 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.513 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.513 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.514 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.514 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.514 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.514 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.514 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.514 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.515 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.515 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.515 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.515 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.515 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.515 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.516 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.516 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.516 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.516 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.516 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.517 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.517 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.517 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.517 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.517 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.517 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.518 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.518 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.518 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.518 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.518 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.518 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.518 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.519 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.519 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.519 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.519 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.519 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.520 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.520 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.520 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.520 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.520 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.520 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.520 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.521 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.521 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.521 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.521 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2023-03-03T08:01:58.521 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2023-03-03T08:01:58.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.522 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.522 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.522 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2023-03-03T08:01:58.522 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.522 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.522 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.522 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.522 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.523 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.523 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.523 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2023-03-03T08:01:58.523 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2023-03-03T08:01:58.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.523 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.523 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.524 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.524 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.524 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.524 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.524 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.524 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.524 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.525 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.525 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2023-03-03T08:01:58.525 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2023-03-03T08:01:58.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.525 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.525 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.525 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.526 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.526 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.526 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.526 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.526 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.526 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.526 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.527 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2023-03-03T08:01:58.527 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.527 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.527 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.528 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2023-03-03T08:01:58.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2023-03-03T08:01:58.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.528 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.528 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.528 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.528 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.528 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.529 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.529 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.529 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.529 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.529 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.529 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2023-03-03T08:01:58.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2023-03-03T08:01:58.530 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.530 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.530 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.530 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.530 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.530 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.530 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.531 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.531 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.531 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.531 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.531 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.531 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.532 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.532 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.532 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.532 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.532 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.533 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.533 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.533 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.533 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.533 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.533 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.533 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.533 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2023-03-03T08:01:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2023-03-03T08:01:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2023-03-03T08:01:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2023-03-03T08:01:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2023-03-03T08:01:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "drive_group_interval": { 2023-03-03T08:01:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "drive_group_interval", 2023-03-03T08:01:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300.0", 2023-03-03T08:01:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-03-03T08:01:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.554 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.554 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.554 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.554 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.554 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.554 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.554 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.555 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.555 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.555 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.555 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.555 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.556 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.556 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.556 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.556 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.556 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.556 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.556 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.557 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.557 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.557 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2023-03-03T08:01:58.557 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2023-03-03T08:01:58.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.557 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.558 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.558 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2023-03-03T08:01:58.558 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.558 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.558 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.558 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-03-03T08:01:58.559 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.559 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.559 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.559 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.559 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2023-03-03T08:01:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.561 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.561 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.561 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.561 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.561 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.561 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.561 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.562 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.562 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.562 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.562 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.562 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.562 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.562 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.562 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.563 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.563 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.563 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.564 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.564 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.564 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.564 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.564 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.564 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.564 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.565 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.565 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.565 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.565 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.565 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.565 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.566 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.566 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.566 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.566 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.566 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.566 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.566 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.566 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.567 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.567 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.567 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.567 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.567 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.567 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.567 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.568 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.568 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.568 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.568 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.569 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.569 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.569 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.569 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.569 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.569 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.569 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.569 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.570 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.570 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.570 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2023-03-03T08:01:58.570 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2023-03-03T08:01:58.570 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.571 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.571 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.571 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.571 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.571 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.571 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.571 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.572 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.572 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.572 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.572 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2023-03-03T08:01:58.572 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2023-03-03T08:01:58.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.572 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.573 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.573 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2023-03-03T08:01:58.573 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.573 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.573 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.573 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.573 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.574 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.574 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.574 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2023-03-03T08:01:58.574 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2023-03-03T08:01:58.574 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.574 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.574 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.575 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.575 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.575 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.575 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.575 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.575 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.575 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.576 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.576 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2023-03-03T08:01:58.576 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2023-03-03T08:01:58.576 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.576 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.576 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.577 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.577 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.577 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.577 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.577 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.577 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.577 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.578 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.578 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2023-03-03T08:01:58.578 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2023-03-03T08:01:58.578 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:58.578 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.578 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.579 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.579 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.579 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.579 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.579 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.579 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.579 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.580 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.580 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2023-03-03T08:01:58.580 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2023-03-03T08:01:58.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.580 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.580 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.580 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.581 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.581 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.581 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.581 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.581 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.581 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.581 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.582 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2023-03-03T08:01:58.582 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2023-03-03T08:01:58.582 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.582 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.582 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.582 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.583 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.583 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.583 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.583 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.583 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.583 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.583 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2023-03-03T08:01:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2023-03-03T08:01:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.585 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.585 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.585 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.585 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.585 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.585 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.585 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.586 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2023-03-03T08:01:58.586 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2023-03-03T08:01:58.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.586 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.586 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.586 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.587 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2023-03-03T08:01:58.587 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2023-03-03T08:01:58.587 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.587 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.587 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.587 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.588 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.588 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2023-03-03T08:01:58.588 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2023-03-03T08:01:58.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.588 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.588 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.589 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.589 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.589 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.589 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.589 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.589 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.590 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.590 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.590 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2023-03-03T08:01:58.590 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2023-03-03T08:01:58.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.590 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.591 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.591 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.591 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.591 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.591 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.591 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.591 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.591 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2023-03-03T08:01:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2023-03-03T08:01:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.594 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.594 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.594 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2023-03-03T08:01:58.594 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.594 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.595 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.595 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2023-03-03T08:01:58.595 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2023-03-03T08:01:58.595 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.595 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.595 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.595 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.596 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.596 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.596 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.596 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2023-03-03T08:01:58.596 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.596 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.596 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.597 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2023-03-03T08:01:58.597 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2023-03-03T08:01:58.597 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.597 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.597 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.597 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.598 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.598 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.598 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.598 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2023-03-03T08:01:58.598 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.598 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.598 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.599 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.599 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.599 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.599 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.599 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.599 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.599 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.600 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.600 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.600 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.600 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.600 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.600 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.600 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.600 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.601 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.601 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.601 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.601 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.601 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.601 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.602 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.602 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.602 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.602 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.602 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.602 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.602 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2023-03-03T08:01:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.615 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.615 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.615 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.615 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.615 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.615 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.615 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.616 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.616 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.616 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.616 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.616 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.616 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.617 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.617 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.617 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.617 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.617 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.617 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.618 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.618 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.618 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.618 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.618 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.618 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.618 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.619 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2023-03-03T08:01:58.619 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.619 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.619 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.619 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.619 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.620 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.620 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.620 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.620 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.620 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.620 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.620 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.621 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.621 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.621 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.621 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.621 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.621 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.621 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.621 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.622 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.622 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.622 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.622 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.622 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.623 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.623 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.623 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.623 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.623 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.623 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.623 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.623 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.624 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.624 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.624 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.624 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.624 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.625 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.625 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.625 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.625 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.625 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.625 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.625 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.625 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.626 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.626 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.626 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.626 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.626 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.626 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.627 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.627 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.627 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.627 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.627 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.627 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.627 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.628 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.628 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.628 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.628 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.628 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.628 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.628 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.629 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2023-03-03T08:01:58.629 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.629 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.629 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.629 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2023-03-03T08:01:58.629 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2023-03-03T08:01:58.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.630 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.630 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.630 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-03-03T08:01:58.630 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.630 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.630 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.631 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.631 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.631 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.631 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.631 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2023-03-03T08:01:58.631 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2023-03-03T08:01:58.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.632 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.632 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.632 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2023-03-03T08:01:58.632 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.632 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.632 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.633 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.633 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.633 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.633 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.633 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.633 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.634 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.634 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.634 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.634 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.634 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.634 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.634 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.635 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.635 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.635 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.635 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.635 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.635 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.635 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.635 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.636 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.636 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.636 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.636 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.636 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.636 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.636 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.637 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.637 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.637 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.637 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.637 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.637 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.637 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.638 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.638 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.638 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.638 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.639 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.639 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.639 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.639 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.639 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.639 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.639 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.640 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.640 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.640 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.640 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.640 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.640 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.640 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.640 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.641 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.641 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.641 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.641 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.641 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.641 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.642 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.642 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.642 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.642 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.642 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.642 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.642 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.643 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2023-03-03T08:01:58.643 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.643 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.643 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.643 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2023-03-03T08:01:58.644 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2023-03-03T08:01:58.644 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.644 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.644 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.644 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.644 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.644 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.644 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.645 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2023-03-03T08:01:58.645 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.645 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.645 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.645 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2023-03-03T08:01:58.645 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2023-03-03T08:01:58.646 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.646 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.646 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.646 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.646 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.646 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.646 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.646 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-03-03T08:01:58.647 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.647 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.647 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.647 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2023-03-03T08:01:58.647 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2023-03-03T08:01:58.647 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.647 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.648 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.648 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.648 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.648 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.648 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.648 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-03-03T08:01:58.648 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.648 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.649 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.649 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2023-03-03T08:01:58.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2023-03-03T08:01:58.649 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.649 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.649 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.650 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.650 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.650 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.650 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.650 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-03-03T08:01:58.650 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.650 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.650 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2023-03-03T08:01:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2023-03-03T08:01:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.652 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.652 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.652 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2023-03-03T08:01:58.652 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.652 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.652 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.652 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2023-03-03T08:01:58.653 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2023-03-03T08:01:58.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.653 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.653 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.653 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.653 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.653 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.654 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.654 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.654 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.654 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.654 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.654 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2023-03-03T08:01:58.654 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2023-03-03T08:01:58.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.655 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.655 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.655 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.655 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.655 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.655 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.655 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.656 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.656 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.656 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.656 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2023-03-03T08:01:58.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2023-03-03T08:01:58.656 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.657 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.657 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.657 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-03-03T08:01:58.657 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.657 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.657 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.657 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.658 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.658 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.658 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.658 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2023-03-03T08:01:58.658 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2023-03-03T08:01:58.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.658 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.659 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.659 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.659 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.659 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.659 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.659 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.659 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.659 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.660 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.660 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2023-03-03T08:01:58.660 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2023-03-03T08:01:58.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.660 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.660 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.660 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2023-03-03T08:01:58.661 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2023-03-03T08:01:58.661 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.661 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.661 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.661 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.661 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.661 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.662 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2023-03-03T08:01:58.662 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2023-03-03T08:01:58.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.662 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.662 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.662 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2023-03-03T08:01:58.662 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.664 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.664 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.664 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.664 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.665 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.665 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.665 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2023-03-03T08:01:58.665 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2023-03-03T08:01:58.665 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.665 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.666 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.666 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.666 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.666 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.666 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.666 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.666 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.666 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.667 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.667 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.667 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.667 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.667 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.667 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.667 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.668 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.668 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.668 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.668 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.668 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.668 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.668 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.669 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.669 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.669 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.669 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.669 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.669 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.669 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.670 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.670 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.670 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.670 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.670 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.670 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.671 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.671 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.671 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.671 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.671 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.671 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.671 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.672 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.672 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.672 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.672 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.672 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.673 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.673 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.673 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.673 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.673 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.673 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.673 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.674 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.674 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.674 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.674 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.674 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.674 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.674 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.675 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.675 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.675 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.675 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.675 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.676 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.676 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.676 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.676 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.676 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.676 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.676 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2023-03-03T08:01:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2023-03-03T08:01:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.678 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.678 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.678 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.678 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.678 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.678 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.678 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2023-03-03T08:01:58.679 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2023-03-03T08:01:58.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.679 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.679 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.679 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.679 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.679 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.680 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.680 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.680 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.680 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.680 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.681 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2023-03-03T08:01:58.681 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2023-03-03T08:01:58.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.681 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.681 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.681 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-03-03T08:01:58.681 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.682 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.682 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.682 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.682 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.682 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.682 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.683 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.683 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.683 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2023-03-03T08:01:58.683 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.683 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.683 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.684 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.684 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.684 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.684 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.684 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.684 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.685 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.685 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.685 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.685 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.685 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.685 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.685 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.686 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.686 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.686 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.686 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.686 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.686 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.687 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.687 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.687 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.687 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.687 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.687 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.687 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.688 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.688 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.688 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.688 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.688 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.688 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.689 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.689 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.689 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.689 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.689 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.689 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.689 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.690 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.690 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.690 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.690 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.690 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.690 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.690 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.691 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.691 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.691 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.691 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.691 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.692 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.692 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.692 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.692 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.692 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.692 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.692 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.693 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.693 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.693 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.693 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2023-03-03T08:01:58.694 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.694 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.694 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.694 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.694 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.694 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2023-03-03T08:01:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2023-03-03T08:01:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2023-03-03T08:01:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2023-03-03T08:01:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2023-03-03T08:01:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2023-03-03T08:01:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2023-03-03T08:01:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-03-03T08:01:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2023-03-03T08:01:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2023-03-03T08:01:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2023-03-03T08:01:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2023-03-03T08:01:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2023-03-03T08:01:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2023-03-03T08:01:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2023-03-03T08:01:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2023-03-03T08:01:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2023-03-03T08:01:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2023-03-03T08:01:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.716 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.716 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.716 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.716 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.716 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.716 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.716 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.716 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.717 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.717 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.717 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.717 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.717 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.718 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.718 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.718 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.718 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.718 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.718 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.718 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.719 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.719 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.719 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.719 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.719 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.719 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.720 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.720 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.720 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.720 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.720 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.720 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.721 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.721 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.721 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.721 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.721 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.721 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.722 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.722 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.722 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.722 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.722 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.722 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.722 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.723 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.723 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.723 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.723 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.723 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.724 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.724 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.724 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.724 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.724 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.724 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.725 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2023-03-03T08:01:58.725 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2023-03-03T08:01:58.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.725 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.725 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.725 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.725 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.725 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.726 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.726 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.726 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.726 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.726 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.726 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2023-03-03T08:01:58.727 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2023-03-03T08:01:58.727 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.727 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.727 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.727 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2023-03-03T08:01:58.727 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.727 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.728 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.728 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.728 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.728 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.728 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.729 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2023-03-03T08:01:58.729 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2023-03-03T08:01:58.729 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.729 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.729 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.729 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-03-03T08:01:58.729 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.730 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.730 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.730 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.730 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.730 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.730 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.730 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.731 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.731 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.731 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:01:58.731 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.731 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.731 INFO:teuthology.orchestra.run.smithi098.stdout: "modules": [ 2023-03-03T08:01:58.731 INFO:teuthology.orchestra.run.smithi098.stdout: "iostat", 2023-03-03T08:01:58.732 INFO:teuthology.orchestra.run.smithi098.stdout: "nfs", 2023-03-03T08:01:58.732 INFO:teuthology.orchestra.run.smithi098.stdout: "restful" 2023-03-03T08:01:58.732 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.732 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2023-03-03T08:01:58.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.732 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2023-03-03T08:01:58.732 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.733 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.733 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.733 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2023-03-03T08:01:58.733 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2023-03-03T08:01:58.733 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.733 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.734 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.734 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2023-03-03T08:01:58.734 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.734 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.734 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.734 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2023-03-03T08:01:58.734 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.735 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.735 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.735 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.735 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.735 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.735 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.735 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.736 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.736 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.736 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.736 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.736 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.736 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.736 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.737 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.737 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.737 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.737 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.737 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.737 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.737 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.737 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.738 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.738 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.738 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.738 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.738 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.738 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.739 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.739 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.739 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.739 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.739 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.739 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.740 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.740 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.740 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.740 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.740 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.740 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.740 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.740 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.741 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.741 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.741 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.741 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.741 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.741 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.741 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.742 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.742 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.742 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.742 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.742 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.742 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.742 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.743 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.743 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.743 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.743 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.743 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.743 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.743 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.744 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.744 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.744 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.744 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2023-03-03T08:01:58.744 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2023-03-03T08:01:58.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.745 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.745 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.745 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.745 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.745 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.745 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.745 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2023-03-03T08:01:58.745 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.746 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.746 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.746 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2023-03-03T08:01:58.746 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2023-03-03T08:01:58.746 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.746 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.747 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.747 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2023-03-03T08:01:58.747 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.747 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.747 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.747 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2023-03-03T08:01:58.747 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.747 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.748 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.748 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2023-03-03T08:01:58.748 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2023-03-03T08:01:58.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.748 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.748 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.748 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.749 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.749 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.749 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.749 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2023-03-03T08:01:58.749 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.749 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.749 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.750 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2023-03-03T08:01:58.750 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2023-03-03T08:01:58.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.750 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.750 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.750 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.751 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.751 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.751 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.751 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2023-03-03T08:01:58.751 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.751 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.751 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.752 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2023-03-03T08:01:58.752 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2023-03-03T08:01:58.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.752 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.752 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.752 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2023-03-03T08:01:58.752 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.752 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.753 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.753 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2023-03-03T08:01:58.753 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.753 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.753 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.753 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2023-03-03T08:01:58.754 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2023-03-03T08:01:58.754 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.754 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.754 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.754 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.754 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.754 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.754 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.755 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2023-03-03T08:01:58.755 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.755 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.755 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.755 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2023-03-03T08:01:58.755 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2023-03-03T08:01:58.755 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.756 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.756 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.756 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.756 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.756 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.756 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.756 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2023-03-03T08:01:58.757 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.757 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.757 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.757 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2023-03-03T08:01:58.757 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2023-03-03T08:01:58.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.758 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.758 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.758 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.758 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.758 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.758 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.758 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2023-03-03T08:01:58.758 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.759 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.759 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.759 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.759 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.759 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2023-03-03T08:01:58.760 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.760 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.760 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.760 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2023-03-03T08:01:58.760 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2023-03-03T08:01:58.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.760 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.760 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.761 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.761 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.761 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.761 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.761 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2023-03-03T08:01:58.761 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.761 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.761 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.762 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2023-03-03T08:01:58.762 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2023-03-03T08:01:58.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.762 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.762 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.762 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2023-03-03T08:01:58.762 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.763 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.763 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.763 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2023-03-03T08:01:58.763 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-03-03T08:01:58.763 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.763 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.763 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2023-03-03T08:01:58.764 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2023-03-03T08:01:58.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2023-03-03T08:01:58.764 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.764 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.764 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2023-03-03T08:01:58.764 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2023-03-03T08:01:58.764 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2023-03-03T08:01:58.765 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.765 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-03-03T08:01:58.765 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-03-03T08:01:58.765 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.765 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.765 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2023-03-03T08:01:58.765 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2023-03-03T08:01:58.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2023-03-03T08:01:58.766 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.766 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.766 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2023-03-03T08:01:58.766 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2023-03-03T08:01:58.766 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2023-03-03T08:01:58.766 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.767 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-03-03T08:01:58.767 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.767 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.767 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.767 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2023-03-03T08:01:58.767 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2023-03-03T08:01:58.767 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.768 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.768 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.768 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2023-03-03T08:01:58.768 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.768 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.768 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.768 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-03-03T08:01:58.769 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-03-03T08:01:58.769 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.769 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.769 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2023-03-03T08:01:58.769 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2023-03-03T08:01:58.769 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:58.770 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.770 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.770 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2023-03-03T08:01:58.770 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2023-03-03T08:01:58.771 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2023-03-03T08:01:58.771 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.771 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2023-03-03T08:01:58.771 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-03-03T08:01:58.771 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.771 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.772 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2023-03-03T08:01:58.772 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2023-03-03T08:01:58.772 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.772 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.773 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.773 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2023-03-03T08:01:58.773 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.773 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.773 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.773 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2023-03-03T08:01:58.774 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-03-03T08:01:58.774 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.774 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.774 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2023-03-03T08:01:58.775 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2023-03-03T08:01:58.775 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2023-03-03T08:01:58.775 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.775 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.775 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2023-03-03T08:01:58.775 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2023-03-03T08:01:58.775 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2023-03-03T08:01:58.776 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.776 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-03-03T08:01:58.776 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-03-03T08:01:58.776 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.776 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.776 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.777 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.777 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.777 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.777 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.777 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.777 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.777 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.777 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.778 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.778 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.778 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.778 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.778 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.778 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.778 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.778 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.779 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.779 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.779 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.779 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.779 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.779 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.780 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.780 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.780 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.780 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.780 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.780 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.780 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.780 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.781 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.781 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.781 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.781 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.781 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.781 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.782 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.782 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.782 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.782 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.782 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.782 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.782 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.783 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.783 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.783 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.783 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.783 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.783 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.783 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.784 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.784 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.784 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.784 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.784 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.784 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.785 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.785 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.785 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.785 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.785 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.785 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.785 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.785 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.786 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2023-03-03T08:01:58.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2023-03-03T08:01:58.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:58.786 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.786 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.786 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2023-03-03T08:01:58.787 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.787 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.787 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.787 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-03-03T08:01:58.787 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.787 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.787 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.788 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2023-03-03T08:01:58.788 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2023-03-03T08:01:58.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.788 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.788 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.788 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2023-03-03T08:01:58.788 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.789 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.789 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.789 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2023-03-03T08:01:58.789 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2023-03-03T08:01:58.789 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap" 2023-03-03T08:01:58.789 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.789 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2023-03-03T08:01:58.790 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.790 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.790 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.790 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2023-03-03T08:01:58.790 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2023-03-03T08:01:58.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.791 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.791 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.791 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.791 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.791 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.791 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.791 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-03-03T08:01:58.791 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.792 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.792 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.792 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2023-03-03T08:01:58.792 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2023-03-03T08:01:58.792 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.793 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.793 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.793 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2023-03-03T08:01:58.793 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.793 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.793 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.793 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-03-03T08:01:58.793 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.794 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.794 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.794 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2023-03-03T08:01:58.794 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2023-03-03T08:01:58.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.794 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.795 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.795 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2023-03-03T08:01:58.795 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2023-03-03T08:01:58.795 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.795 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.795 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2023-03-03T08:01:58.795 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-03-03T08:01:58.795 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.796 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.796 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2023-03-03T08:01:58.796 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2023-03-03T08:01:58.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2023-03-03T08:01:58.796 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.796 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.796 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2023-03-03T08:01:58.797 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.797 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.797 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.797 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-03-03T08:01:58.797 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.797 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.798 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.798 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2023-03-03T08:01:58.798 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.799 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.799 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.799 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2023-03-03T08:01:58.799 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2023-03-03T08:01:58.799 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:58.799 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.799 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.799 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2023-03-03T08:01:58.800 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.800 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.800 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.800 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-03-03T08:01:58.800 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.800 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.800 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.801 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2023-03-03T08:01:58.801 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2023-03-03T08:01:58.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.801 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.801 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.801 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2023-03-03T08:01:58.802 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.802 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.802 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.802 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-03-03T08:01:58.802 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.802 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.802 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.803 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2023-03-03T08:01:58.803 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2023-03-03T08:01:58.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.803 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.803 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.803 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2023-03-03T08:01:58.803 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.804 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.804 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.804 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-03-03T08:01:58.804 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.804 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.804 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.805 INFO:teuthology.orchestra.run.smithi098.stdout: "alertmanager_web_password": { 2023-03-03T08:01:58.805 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alertmanager_web_password", 2023-03-03T08:01:58.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.805 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.805 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.805 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2023-03-03T08:01:58.805 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.806 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.806 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.806 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Alertmanager web password", 2023-03-03T08:01:58.806 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.806 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.806 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.807 INFO:teuthology.orchestra.run.smithi098.stdout: "alertmanager_web_user": { 2023-03-03T08:01:58.807 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alertmanager_web_user", 2023-03-03T08:01:58.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.807 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.807 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.807 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2023-03-03T08:01:58.807 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.807 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.808 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.808 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Alertmanager web user", 2023-03-03T08:01:58.808 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.808 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.808 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.808 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2023-03-03T08:01:58.809 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2023-03-03T08:01:58.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.809 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.809 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.809 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.809 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.809 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.810 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.810 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-03-03T08:01:58.810 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-03-03T08:01:58.810 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.810 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.810 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2023-03-03T08:01:58.811 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2023-03-03T08:01:58.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.811 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.811 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.811 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2023-03-03T08:01:58.811 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.811 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.812 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.812 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2023-03-03T08:01:58.812 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.812 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.812 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.812 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2023-03-03T08:01:58.812 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2023-03-03T08:01:58.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:58.813 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.813 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.813 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2023-03-03T08:01:58.813 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.813 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.813 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.814 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-03-03T08:01:58.814 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.814 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.814 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.814 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2023-03-03T08:01:58.814 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2023-03-03T08:01:58.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.815 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.815 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.815 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.815 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.815 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.815 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.815 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-03-03T08:01:58.816 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.816 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.816 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.816 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2023-03-03T08:01:58.816 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2023-03-03T08:01:58.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.817 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.817 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.817 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.817 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.817 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.817 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.817 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-03-03T08:01:58.818 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.818 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.818 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.818 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2023-03-03T08:01:58.818 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2023-03-03T08:01:58.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.818 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-03-03T08:01:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2023-03-03T08:01:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2023-03-03T08:01:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.23.0", 2023-03-03T08:01:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2023-03-03T08:01:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2023-03-03T08:01:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2023-03-03T08:01:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2023-03-03T08:01:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2023-03-03T08:01:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2023-03-03T08:01:58.824 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2023-03-03T08:01:58.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.824 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.824 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.824 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-03-03T08:01:58.824 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.824 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2023-03-03T08:01:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2023-03-03T08:01:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2023-03-03T08:01:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph-grafana:8.3.5", 2023-03-03T08:01:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2023-03-03T08:01:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2023-03-03T08:01:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2023-03-03T08:01:58.828 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.828 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.828 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.828 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-03-03T08:01:58.828 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.828 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.828 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.828 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2023-03-03T08:01:58.829 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.829 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.829 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.829 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2023-03-03T08:01:58.829 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2023-03-03T08:01:58.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.830 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.830 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.830 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-03-03T08:01:58.830 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.830 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.830 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.830 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2023-03-03T08:01:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2023-03-03T08:01:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2023-03-03T08:01:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.832 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.832 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.832 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-03-03T08:01:58.832 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.832 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.832 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.833 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2023-03-03T08:01:58.833 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.833 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.833 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.833 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2023-03-03T08:01:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2023-03-03T08:01:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-03-03T08:01:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2023-03-03T08:01:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2023-03-03T08:01:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2023-03-03T08:01:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.1.5", 2023-03-03T08:01:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2023-03-03T08:01:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2023-03-03T08:01:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2023-03-03T08:01:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-03-03T08:01:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2023-03-03T08:01:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2023-03-03T08:01:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2023-03-03T08:01:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.3.1", 2023-03-03T08:01:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2023-03-03T08:01:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2023-03-03T08:01:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2023-03-03T08:01:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.33.4", 2023-03-03T08:01:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2023-03-03T08:01:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2023-03-03T08:01:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2023-03-03T08:01:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-03-03T08:01:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2023-03-03T08:01:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2023-03-03T08:01:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2023-03-03T08:01:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-03-03T08:01:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2023-03-03T08:01:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2023-03-03T08:01:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2023-03-03T08:01:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2023-03-03T08:01:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2023-03-03T08:01:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2023-03-03T08:01:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2023-03-03T08:01:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-03-03T08:01:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2023-03-03T08:01:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2023-03-03T08:01:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2023-03-03T08:01:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.854 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.854 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-03-03T08:01:58.854 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.854 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.854 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.854 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2023-03-03T08:01:58.855 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2023-03-03T08:01:58.855 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.855 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.855 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.855 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2023-03-03T08:01:58.855 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2023-03-03T08:01:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2023-03-03T08:01:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2023-03-03T08:01:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2023-03-03T08:01:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2023-03-03T08:01:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2023-03-03T08:01:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2023-03-03T08:01:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2023-03-03T08:01:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2023-03-03T08:01:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2023-03-03T08:01:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2023-03-03T08:01:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2023-03-03T08:01:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2023-03-03T08:01:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2023-03-03T08:01:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-03-03T08:01:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-03-03T08:01:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2023-03-03T08:01:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-03-03T08:01:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-03-03T08:01:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-03-03T08:01:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-03-03T08:01:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2023-03-03T08:01:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-03-03T08:01:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2023-03-03T08:01:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2023-03-03T08:01:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.881 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.881 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2023-03-03T08:01:58.881 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.881 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.881 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.881 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2023-03-03T08:01:58.882 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.882 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.882 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.882 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2023-03-03T08:01:58.882 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2023-03-03T08:01:58.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.883 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.883 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.883 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2023-03-03T08:01:58.883 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.883 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.883 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.883 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-03-03T08:01:58.884 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.884 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.884 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.884 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2023-03-03T08:01:58.884 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2023-03-03T08:01:58.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.885 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.885 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.885 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.885 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.885 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.885 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2023-03-03T08:01:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2023-03-03T08:01:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2023-03-03T08:01:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2023-03-03T08:01:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2023-03-03T08:01:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2023-03-03T08:01:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2023-03-03T08:01:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2023-03-03T08:01:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2023-03-03T08:01:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-03-03T08:01:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-03-03T08:01:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_web_password": { 2023-03-03T08:01:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_web_password", 2023-03-03T08:01:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2023-03-03T08:01:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus web password", 2023-03-03T08:01:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_web_user": { 2023-03-03T08:01:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_web_user", 2023-03-03T08:01:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2023-03-03T08:01:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus web user", 2023-03-03T08:01:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2023-03-03T08:01:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2023-03-03T08:01:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-03-03T08:01:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2023-03-03T08:01:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2023-03-03T08:01:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.899 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.899 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.899 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.899 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.899 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.899 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-03-03T08:01:58.899 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.899 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.900 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.900 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2023-03-03T08:01:58.900 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2023-03-03T08:01:58.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.900 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.900 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.901 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.901 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.901 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.901 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.901 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-03-03T08:01:58.901 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.901 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.902 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.902 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2023-03-03T08:01:58.902 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2023-03-03T08:01:58.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.902 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.902 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.903 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.903 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.903 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.903 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.903 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-03-03T08:01:58.903 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.903 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.904 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.904 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2023-03-03T08:01:58.904 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2023-03-03T08:01:58.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.904 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.904 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.905 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.905 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.905 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.905 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.905 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-03-03T08:01:58.905 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.905 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.905 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.906 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2023-03-03T08:01:58.906 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2023-03-03T08:01:58.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.906 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.906 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.906 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2023-03-03T08:01:58.907 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.907 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.907 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.907 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2023-03-03T08:01:58.907 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.907 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.907 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.908 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2023-03-03T08:01:58.908 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2023-03-03T08:01:58.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.908 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.908 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.908 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.908 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.908 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-03-03T08:01:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2023-03-03T08:01:58.909 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2023-03-03T08:01:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.910 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-03-03T08:01:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2023-03-03T08:01:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2023-03-03T08:01:58.911 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.912 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-03-03T08:01:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2023-03-03T08:01:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2023-03-03T08:01:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.913 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2023-03-03T08:01:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.914 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2023-03-03T08:01:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2023-03-03T08:01:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.915 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-03-03T08:01:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2023-03-03T08:01:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2023-03-03T08:01:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.917 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-03-03T08:01:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.919 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2023-03-03T08:01:58.919 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.919 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.919 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.919 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:58.919 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:58.920 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.920 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.920 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.920 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.920 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.920 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.920 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.921 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.921 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.921 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.921 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.921 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.921 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.921 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.922 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.922 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.922 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.922 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.922 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:58.922 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:58.923 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.923 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.923 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.923 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.923 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.923 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.923 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.923 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.924 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.924 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.924 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.924 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:58.924 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:58.924 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.925 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.925 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.925 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:58.925 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.925 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.925 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:58.925 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:58.925 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:58.926 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:58.926 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:58.926 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:58.926 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:58.926 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:58.926 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.926 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.927 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.927 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.927 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:58.927 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:58.927 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.927 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.927 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.928 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.928 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.928 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.928 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.928 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.928 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.928 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.928 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.929 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2023-03-03T08:01:58.929 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2023-03-03T08:01:58.929 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.929 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.929 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.929 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2023-03-03T08:01:58.930 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.930 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.930 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.930 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2023-03-03T08:01:58.930 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.930 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.930 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.931 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2023-03-03T08:01:58.931 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2023-03-03T08:01:58.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:58.931 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.931 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:58.931 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2023-03-03T08:01:58.931 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.932 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.932 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.932 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2023-03-03T08:01:58.932 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.932 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.932 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:58.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:58.933 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2023-03-03T08:01:58.933 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:58.933 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:58.933 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:58.933 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-03-03T08:01:58.934 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-03-03T08:01:58.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.934 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.934 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.934 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2023-03-03T08:01:58.934 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.934 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.934 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.935 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.935 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.935 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.935 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.935 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2023-03-03T08:01:58.935 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2023-03-03T08:01:58.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.936 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.936 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.936 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.936 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.936 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.936 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.937 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.937 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.937 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.937 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.937 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-03-03T08:01:58.937 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-03-03T08:01:58.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.938 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.938 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.938 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.938 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.938 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.938 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.938 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.939 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.939 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.939 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.939 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2023-03-03T08:01:58.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2023-03-03T08:01:58.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.939 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.940 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.940 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.940 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.940 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.940 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.940 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.940 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.941 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.941 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.941 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-03-03T08:01:58.941 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-03-03T08:01:58.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.941 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.942 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.942 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.942 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.942 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.942 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.942 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.942 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.943 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.943 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.943 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2023-03-03T08:01:58.943 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2023-03-03T08:01:58.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.943 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.944 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.944 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.944 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.944 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.944 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.944 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.944 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.945 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.945 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.945 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-03-03T08:01:58.945 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-03-03T08:01:58.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.945 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.945 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.946 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.946 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.946 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.946 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.946 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.946 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.946 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.946 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-03-03T08:01:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-03-03T08:01:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-03-03T08:01:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-03-03T08:01:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.950 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.950 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.950 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.950 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.950 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.950 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2023-03-03T08:01:58.951 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-03-03T08:01:58.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.951 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.951 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.951 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.951 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.951 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.951 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.952 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.952 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.952 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.952 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.952 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2023-03-03T08:01:58.952 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-03-03T08:01:58.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.953 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.953 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.953 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.953 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.953 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.953 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.953 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.954 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.954 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.954 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.954 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2023-03-03T08:01:58.954 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-03-03T08:01:58.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.955 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.955 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.955 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.955 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.955 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.955 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.955 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.956 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.956 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.956 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.956 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-03-03T08:01:58.956 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-03-03T08:01:58.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.956 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.957 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.957 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.957 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.957 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.957 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.957 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.957 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.957 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.958 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.958 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2023-03-03T08:01:58.958 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2023-03-03T08:01:58.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.958 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.958 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.959 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2023-03-03T08:01:58.959 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.959 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.959 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.959 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.959 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.959 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.959 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.960 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-03-03T08:01:58.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-03-03T08:01:58.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.960 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.960 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.961 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.961 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.961 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.961 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.961 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.961 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.961 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.962 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.962 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2023-03-03T08:01:58.962 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2023-03-03T08:01:58.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.962 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.962 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.962 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.963 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.963 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.963 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.963 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.963 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.963 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.963 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.964 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2023-03-03T08:01:58.964 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2023-03-03T08:01:58.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.964 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.964 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.964 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2023-03-03T08:01:58.964 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.964 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.965 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.965 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.965 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.965 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.965 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.965 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-03-03T08:01:58.966 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-03-03T08:01:58.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.966 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.966 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.966 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.966 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.966 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.966 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.967 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.967 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.967 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.967 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.967 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-03-03T08:01:58.967 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-03-03T08:01:58.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.968 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.968 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.968 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.968 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.968 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.968 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.968 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.969 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.969 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.969 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.969 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-03-03T08:01:58.969 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-03-03T08:01:58.969 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.970 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.970 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.970 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.970 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.970 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.970 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.970 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.970 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.971 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.971 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.971 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2023-03-03T08:01:58.971 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-03-03T08:01:58.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.972 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.972 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.972 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.972 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.972 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.972 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.972 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.973 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.973 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.973 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.973 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2023-03-03T08:01:58.973 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2023-03-03T08:01:58.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.974 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.974 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:58.974 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.974 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.974 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.974 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.975 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.975 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.975 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.975 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-03-03T08:01:58.975 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-03-03T08:01:58.975 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.976 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.976 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.976 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.976 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.976 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.976 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.976 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.976 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.977 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.977 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.977 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-03-03T08:01:58.977 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-03-03T08:01:58.977 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.977 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.977 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.978 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.978 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.978 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.978 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.978 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.978 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.978 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.978 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.979 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-03-03T08:01:58.979 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-03-03T08:01:58.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.979 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.979 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.979 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.980 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.980 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.980 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.980 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.980 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.980 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.980 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.981 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-03-03T08:01:58.981 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-03-03T08:01:58.981 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.981 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.981 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.981 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.981 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.981 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.982 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.982 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.982 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.982 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.982 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.982 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-03-03T08:01:58.983 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-03-03T08:01:58.983 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.983 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.983 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.983 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.983 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.983 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.983 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.984 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.984 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.984 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.984 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.984 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-03-03T08:01:58.984 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-03-03T08:01:58.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.985 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.985 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.985 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.985 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.985 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.985 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.985 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.986 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.986 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.986 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.986 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-03-03T08:01:58.986 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-03-03T08:01:58.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.986 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.987 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.987 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.987 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.987 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.987 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.987 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.987 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.987 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.988 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.988 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-03-03T08:01:58.988 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-03-03T08:01:58.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.988 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.988 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.989 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:58.989 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.989 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.989 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.989 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.989 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.989 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.990 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.990 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2023-03-03T08:01:58.990 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2023-03-03T08:01:58.990 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:58.990 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.990 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.990 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:58.991 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.991 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.991 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.991 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.991 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.991 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.991 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.992 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-03-03T08:01:58.992 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-03-03T08:01:58.992 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.992 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.992 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.992 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-03-03T08:01:58.992 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.993 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.993 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.993 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.993 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.993 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.993 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.993 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-03-03T08:01:58.994 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-03-03T08:01:58.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.994 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.994 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.994 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2023-03-03T08:01:58.994 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.994 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.994 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.995 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.995 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.995 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.995 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.995 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-03-03T08:01:58.995 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-03-03T08:01:58.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.996 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.996 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.996 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2023-03-03T08:01:58.996 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.996 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.996 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.996 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.996 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.997 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.997 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.997 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2023-03-03T08:01:58.997 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-03-03T08:01:58.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:58.997 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.998 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.998 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2023-03-03T08:01:58.998 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:58.998 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:58.998 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:58.998 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:58.998 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:58.998 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:58.999 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:58.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:58.999 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2023-03-03T08:01:58.999 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2023-03-03T08:01:58.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:58.999 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:58.999 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:58.999 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.000 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.000 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.000 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.000 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.000 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.000 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.000 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.001 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-03-03T08:01:59.001 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-03-03T08:01:59.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.001 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.001 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.001 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2023-03-03T08:01:59.002 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.002 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.002 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.002 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.002 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.002 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.003 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.003 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2023-03-03T08:01:59.003 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2023-03-03T08:01:59.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.003 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.003 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.003 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.004 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.004 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.004 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.004 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.004 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.004 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.004 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.005 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2023-03-03T08:01:59.005 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2023-03-03T08:01:59.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.005 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.005 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.005 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:59.005 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.006 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.006 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.006 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.006 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.006 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.006 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.007 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-03-03T08:01:59.007 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-03-03T08:01:59.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.007 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.007 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.007 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2023-03-03T08:01:59.007 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.008 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.008 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.008 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.008 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.008 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.008 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.009 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-03-03T08:01:59.009 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-03-03T08:01:59.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.009 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.009 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.009 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2023-03-03T08:01:59.009 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.010 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.010 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.010 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.010 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.010 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.010 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.010 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-03-03T08:01:59.011 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-03-03T08:01:59.011 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.011 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.011 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.011 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2023-03-03T08:01:59.011 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.011 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.012 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.012 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.012 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.012 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.012 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.012 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2023-03-03T08:01:59.013 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2023-03-03T08:01:59.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.013 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.013 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.013 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.013 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.014 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.014 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.014 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.014 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.014 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.014 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.014 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2023-03-03T08:01:59.015 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2023-03-03T08:01:59.015 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.015 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.015 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.015 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.015 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.015 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.016 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.016 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.016 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.016 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.016 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.016 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2023-03-03T08:01:59.017 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2023-03-03T08:01:59.017 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.017 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.017 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.017 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.017 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.017 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.018 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.018 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2023-03-03T08:01:59.018 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.018 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.018 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.018 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2023-03-03T08:01:59.018 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2023-03-03T08:01:59.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.019 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.019 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.019 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2023-03-03T08:01:59.019 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.019 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.019 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.020 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.020 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.020 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.020 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.020 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2023-03-03T08:01:59.020 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2023-03-03T08:01:59.021 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.021 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.021 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.021 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.021 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.021 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.021 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.022 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.022 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.022 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.022 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.022 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.022 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.023 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.023 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.023 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.023 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.023 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.023 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.023 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.024 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.024 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.024 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.024 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.024 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.024 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.024 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.025 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.025 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.025 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.025 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.025 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.026 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.026 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.026 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.026 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.026 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.026 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.026 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.027 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.027 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.027 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.027 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.027 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.027 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.028 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.028 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.028 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.028 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.028 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.028 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.028 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.029 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.029 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.029 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.029 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.029 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.029 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.029 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.029 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.030 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.030 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.030 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.030 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.031 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.031 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.031 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.031 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.031 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.031 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.031 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.031 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.032 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.032 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2023-03-03T08:01:59.032 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2023-03-03T08:01:59.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.032 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.032 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.033 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.033 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.033 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.033 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.033 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2023-03-03T08:01:59.033 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.034 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.034 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.034 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2023-03-03T08:01:59.034 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2023-03-03T08:01:59.034 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.034 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.034 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.035 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.035 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.035 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.035 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.035 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.035 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.035 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.036 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.036 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2023-03-03T08:01:59.036 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2023-03-03T08:01:59.036 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.036 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.036 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.037 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2023-03-03T08:01:59.037 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.037 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.037 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.037 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.037 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.037 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.037 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.038 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2023-03-03T08:01:59.038 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2023-03-03T08:01:59.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.038 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.038 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.039 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2023-03-03T08:01:59.039 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.039 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.039 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.039 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.039 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.039 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.040 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.040 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2023-03-03T08:01:59.040 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2023-03-03T08:01:59.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.040 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.040 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.041 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:59.041 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.041 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.041 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.041 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.041 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.041 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.042 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.042 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2023-03-03T08:01:59.042 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2023-03-03T08:01:59.042 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.042 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.042 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.042 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2023-03-03T08:01:59.043 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.043 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.043 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.043 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.043 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.043 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.044 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.044 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2023-03-03T08:01:59.044 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2023-03-03T08:01:59.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.044 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.044 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.044 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2023-03-03T08:01:59.045 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.045 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.045 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.045 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.045 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2023-03-03T08:01:59.045 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.046 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.046 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.046 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.046 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.046 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2023-03-03T08:01:59.115 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2023-03-03T08:01:59.116 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.116 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.116 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.116 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2023-03-03T08:01:59.116 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2023-03-03T08:01:59.117 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2023-03-03T08:01:59.117 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.117 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.117 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.117 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.117 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.118 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2023-03-03T08:01:59.118 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2023-03-03T08:01:59.118 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.118 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.118 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.118 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.118 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.118 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.119 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.119 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.119 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.119 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.119 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.120 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.120 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2023-03-03T08:01:59.120 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.120 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.121 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.121 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2023-03-03T08:01:59.121 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2023-03-03T08:01:59.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.122 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.122 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.122 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:59.122 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.122 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.122 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.123 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2023-03-03T08:01:59.123 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.123 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.123 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.124 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.124 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.124 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.124 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.124 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.124 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.125 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.125 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.125 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.125 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.125 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.125 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.125 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.125 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.126 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.126 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.126 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.126 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.126 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.126 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.127 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.127 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.127 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.127 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.127 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.127 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.127 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.128 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.128 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.128 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.128 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.128 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.129 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.129 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.129 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.129 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.129 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.129 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.129 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.130 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.130 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.130 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.130 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.131 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.131 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.131 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.131 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.132 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.132 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.132 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.133 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.133 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.133 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.133 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.133 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.133 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.134 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.134 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.134 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.134 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.134 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2023-03-03T08:01:59.134 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2023-03-03T08:01:59.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:59.135 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.135 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.135 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2023-03-03T08:01:59.135 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.135 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.135 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.135 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-03-03T08:01:59.136 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.136 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.136 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.136 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2023-03-03T08:01:59.136 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2023-03-03T08:01:59.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.136 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.137 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.137 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2023-03-03T08:01:59.137 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.137 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.137 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.137 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2023-03-03T08:01:59.138 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.138 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.138 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.138 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2023-03-03T08:01:59.138 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2023-03-03T08:01:59.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:59.139 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.139 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.139 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2023-03-03T08:01:59.139 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.139 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.139 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.139 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2023-03-03T08:01:59.140 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.140 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.140 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.140 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2023-03-03T08:01:59.141 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2023-03-03T08:01:59.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:59.141 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.141 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.141 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2023-03-03T08:01:59.141 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.141 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.142 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.143 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2023-03-03T08:01:59.144 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.144 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.144 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.145 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2023-03-03T08:01:59.145 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2023-03-03T08:01:59.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.145 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.146 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.146 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:59.146 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.146 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.146 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.147 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-03-03T08:01:59.147 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.147 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.147 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.147 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2023-03-03T08:01:59.147 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2023-03-03T08:01:59.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:59.148 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.148 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.148 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2023-03-03T08:01:59.148 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.148 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.148 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.148 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2023-03-03T08:01:59.149 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.149 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.149 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.149 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2023-03-03T08:01:59.150 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2023-03-03T08:01:59.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:59.150 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.150 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.150 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2023-03-03T08:01:59.151 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.151 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.151 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.151 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-03-03T08:01:59.151 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.151 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.151 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.152 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.152 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.152 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2023-03-03T08:01:59.152 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.152 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.152 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.153 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.153 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.153 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.153 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.153 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.153 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.154 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.154 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.154 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.154 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.155 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.155 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.155 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.155 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.156 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.156 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.156 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.156 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.157 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.157 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.157 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.158 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.158 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.158 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.159 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.159 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.159 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.160 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.160 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.160 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.160 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.161 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.161 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.161 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.162 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.162 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.162 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.163 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.163 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.163 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.163 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.164 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.164 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.164 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.164 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.165 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.165 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.165 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.165 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.165 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.166 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.166 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.166 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.168 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.168 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.168 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.168 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.168 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.169 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.169 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.169 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.169 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.169 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.169 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.170 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2023-03-03T08:01:59.170 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2023-03-03T08:01:59.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.170 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.170 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.170 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2023-03-03T08:01:59.170 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.171 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.171 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.171 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.171 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.171 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.171 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.171 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2023-03-03T08:01:59.172 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2023-03-03T08:01:59.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.172 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.172 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.172 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2023-03-03T08:01:59.172 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.172 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.172 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.173 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.173 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.173 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.173 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.173 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2023-03-03T08:01:59.173 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2023-03-03T08:01:59.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.174 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.174 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.174 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2023-03-03T08:01:59.174 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.174 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.174 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.174 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.175 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.175 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.175 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.176 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2023-03-03T08:01:59.176 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2023-03-03T08:01:59.176 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2023-03-03T08:01:59.176 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.176 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2023-03-03T08:01:59.176 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2023-03-03T08:01:59.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.176 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.177 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.177 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2023-03-03T08:01:59.177 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.177 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.177 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.177 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-03-03T08:01:59.178 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.178 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.178 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.179 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2023-03-03T08:01:59.179 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2023-03-03T08:01:59.179 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.179 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.179 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.179 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2023-03-03T08:01:59.179 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.180 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.180 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.180 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-03-03T08:01:59.180 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.180 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.180 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.180 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2023-03-03T08:01:59.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2023-03-03T08:01:59.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.181 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.181 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.181 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.181 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.181 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.182 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.182 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2023-03-03T08:01:59.182 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.182 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.182 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.182 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2023-03-03T08:01:59.182 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2023-03-03T08:01:59.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:59.183 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.183 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.183 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2023-03-03T08:01:59.183 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2023-03-03T08:01:59.183 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.183 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.183 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-03-03T08:01:59.184 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.184 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.184 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.184 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.184 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.184 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.185 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.185 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.185 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.185 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.185 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.185 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.185 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.185 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.186 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.186 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.186 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.186 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.186 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.186 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.186 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.186 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.187 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.187 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.187 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.187 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.187 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.187 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.188 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.188 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.188 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.188 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.188 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.189 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.189 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.189 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.189 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.189 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.189 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.190 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.190 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.190 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.190 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.190 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.190 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.190 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.190 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.191 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.191 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.191 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.191 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.191 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.191 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.191 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.192 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.192 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.192 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.192 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.192 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.192 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.192 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.193 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.193 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.193 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.193 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.193 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.193 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.193 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.194 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2023-03-03T08:01:59.194 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2023-03-03T08:01:59.194 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.194 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.194 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.194 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.194 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.194 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.195 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.195 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2023-03-03T08:01:59.195 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.195 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.195 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.195 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2023-03-03T08:01:59.195 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2023-03-03T08:01:59.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.196 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.196 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.196 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2023-03-03T08:01:59.196 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.196 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.196 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.196 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2023-03-03T08:01:59.197 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.197 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.197 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.197 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2023-03-03T08:01:59.197 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2023-03-03T08:01:59.197 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.198 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.198 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.198 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2023-03-03T08:01:59.198 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.198 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.198 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.198 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-03-03T08:01:59.198 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.199 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.199 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.199 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2023-03-03T08:01:59.199 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2023-03-03T08:01:59.199 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.199 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.200 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.200 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2023-03-03T08:01:59.200 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2023-03-03T08:01:59.200 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2023-03-03T08:01:59.200 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.201 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-03-03T08:01:59.201 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.201 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.201 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.201 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2023-03-03T08:01:59.201 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2023-03-03T08:01:59.202 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.202 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.202 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.202 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.202 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.202 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.202 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.202 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2023-03-03T08:01:59.203 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.203 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.203 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.203 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2023-03-03T08:01:59.203 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2023-03-03T08:01:59.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.203 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.204 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.204 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2023-03-03T08:01:59.204 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.204 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.204 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.204 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-03-03T08:01:59.204 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.204 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.205 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.205 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.205 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.205 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2023-03-03T08:01:59.205 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.205 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.206 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.206 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.206 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.206 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.206 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.206 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.206 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.207 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.207 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.207 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.207 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.207 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.207 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.207 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.207 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.208 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.208 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.208 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.208 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.208 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.208 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.208 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.209 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.209 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.209 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.209 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.209 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.209 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.209 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.209 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.213 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.213 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.213 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.213 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.214 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.214 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.214 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.214 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.214 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.214 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.214 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.215 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.215 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.215 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.215 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.215 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2023-03-03T08:01:59.215 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.216 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.216 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.216 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.216 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.216 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.216 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.216 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.217 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.217 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.217 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.217 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.217 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.217 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.217 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.217 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.218 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.218 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.218 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.218 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.218 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.218 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.218 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.219 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.219 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.219 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.219 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.219 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.219 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.219 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.220 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.220 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.220 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.220 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.220 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.220 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.221 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.221 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.221 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.221 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.221 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.221 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.221 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.222 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.222 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.222 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.222 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.222 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.222 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.222 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.222 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.223 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.223 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.223 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.223 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.223 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.223 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.224 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.224 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.224 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.224 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.224 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.224 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.224 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.224 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.225 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.225 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.225 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2023-03-03T08:01:59.225 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.225 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.225 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.226 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2023-03-03T08:01:59.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2023-03-03T08:01:59.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.226 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.226 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.226 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2023-03-03T08:01:59.226 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.226 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.227 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.227 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2023-03-03T08:01:59.227 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-03-03T08:01:59.227 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.227 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.227 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.228 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.228 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.228 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.228 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.228 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.228 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.228 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.229 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.229 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.229 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.229 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.229 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.229 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.229 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.230 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.230 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.230 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.230 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.231 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.231 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.231 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.231 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.231 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.232 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.232 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.232 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.232 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.232 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.232 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.232 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.233 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.233 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.233 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.233 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.233 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.233 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.234 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.234 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.234 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.234 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.234 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.234 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.234 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.234 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.235 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.235 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.235 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.235 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.235 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.235 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.235 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.236 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.236 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.236 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.236 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.236 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.236 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.236 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.237 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.237 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.237 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.237 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2023-03-03T08:01:59.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2023-03-03T08:01:59.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.238 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.238 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.238 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.238 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.238 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.238 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.238 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2023-03-03T08:01:59.238 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-03-03T08:01:59.239 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.239 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.239 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2023-03-03T08:01:59.239 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2023-03-03T08:01:59.239 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.239 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.239 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.240 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2023-03-03T08:01:59.240 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.240 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.240 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.240 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2023-03-03T08:01:59.240 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.240 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.241 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.241 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2023-03-03T08:01:59.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2023-03-03T08:01:59.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.241 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.241 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.241 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2023-03-03T08:01:59.242 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.242 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.242 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.242 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2023-03-03T08:01:59.242 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.242 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.242 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.243 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2023-03-03T08:01:59.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2023-03-03T08:01:59.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.243 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.243 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.243 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.243 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.244 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.244 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.244 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2023-03-03T08:01:59.244 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.244 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.244 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.244 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2023-03-03T08:01:59.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2023-03-03T08:01:59.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.245 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.245 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.245 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2023-03-03T08:01:59.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.245 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.245 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.246 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2023-03-03T08:01:59.246 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-03-03T08:01:59.246 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.246 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.246 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.246 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2023-03-03T08:01:59.247 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.247 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.247 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.247 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.248 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.248 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.248 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.248 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.248 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.248 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.248 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.248 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.249 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.249 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.249 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.249 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.249 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.249 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.249 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.249 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.250 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.250 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.250 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.251 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.251 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.251 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.251 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.251 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.251 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.251 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.252 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.252 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.252 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.252 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.252 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.253 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.253 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.253 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.253 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.253 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.253 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.253 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.254 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.254 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.254 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.254 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.254 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.254 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.254 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.254 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.255 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.255 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.255 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.255 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.255 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.255 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.256 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.256 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.256 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.256 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.256 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.256 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.257 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2023-03-03T08:01:59.257 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.257 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.257 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.257 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.257 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.258 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.258 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.258 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.258 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.258 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.258 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.258 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.259 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.259 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.259 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.259 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.259 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.259 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.259 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.259 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.260 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.260 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.260 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.260 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.260 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.260 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.260 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.261 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.261 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.261 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.261 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.261 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.261 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.261 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.262 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.262 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.262 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.262 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.262 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.263 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.263 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.263 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.263 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.263 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.263 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.263 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.264 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.264 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.264 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.264 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.264 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.264 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.264 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.265 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.265 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.265 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.265 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.265 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.265 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.266 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.266 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.266 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.266 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.266 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.267 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2023-03-03T08:01:59.267 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.267 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.267 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.267 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.267 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.268 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.268 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.268 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.268 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.268 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.269 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.269 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.269 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.269 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.269 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.270 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.270 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.270 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.270 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.270 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.270 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.270 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.271 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.271 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.271 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.271 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.271 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.271 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.271 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.272 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.272 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.272 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.272 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.272 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.272 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.272 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.273 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.273 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.273 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.273 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.273 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.273 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.273 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.274 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.274 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.274 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.274 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.274 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.274 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.274 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.275 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.275 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.275 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.275 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.276 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.276 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.276 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.276 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.276 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.276 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.276 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.277 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.277 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.277 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.277 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2023-03-03T08:01:59.278 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.278 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.278 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.278 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2023-03-03T08:01:59.278 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2023-03-03T08:01:59.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.278 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.279 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.279 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.279 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.279 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.279 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.279 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-03-03T08:01:59.279 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.279 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.280 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.280 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.280 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.280 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.280 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.281 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.281 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.281 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.281 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.281 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.281 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.281 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.281 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.282 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.282 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.282 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.282 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.282 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.282 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.282 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.283 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.283 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.283 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.283 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.283 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.283 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.283 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.284 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.284 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.284 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.284 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.284 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.284 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.285 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.285 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.285 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.285 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.285 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.285 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.285 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.286 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.286 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.286 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.286 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.286 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.286 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.286 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.286 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.287 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.287 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.287 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.287 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.287 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.288 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.288 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.288 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.288 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.288 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.288 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.288 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.288 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.289 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2023-03-03T08:01:59.289 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2023-03-03T08:01:59.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.289 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.289 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.289 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.289 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.290 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.290 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.290 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2023-03-03T08:01:59.290 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2023-03-03T08:01:59.290 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2023-03-03T08:01:59.290 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.290 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2023-03-03T08:01:59.290 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.291 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.291 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.291 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.291 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2023-03-03T08:01:59.292 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.292 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.292 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.292 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.292 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.292 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.292 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2023-03-03T08:01:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2023-03-03T08:01:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "noautoscale": { 2023-03-03T08:01:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "noautoscale", 2023-03-03T08:01:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "global autoscale flag", 2023-03-03T08:01:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-03-03T08:01:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2023-03-03T08:01:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2023-03-03T08:01:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2023-03-03T08:01:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2023-03-03T08:01:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2023-03-03T08:01:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2023-03-03T08:01:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2023-03-03T08:01:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2023-03-03T08:01:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-03-03T08:01:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2023-03-03T08:01:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2023-03-03T08:01:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2023-03-03T08:01:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2023-03-03T08:01:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2023-03-03T08:01:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2023-03-03T08:01:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2023-03-03T08:01:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2023-03-03T08:01:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2023-03-03T08:01:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2023-03-03T08:01:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2023-03-03T08:01:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2023-03-03T08:01:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2023-03-03T08:01:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2023-03-03T08:01:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2023-03-03T08:01:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2023-03-03T08:01:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2023-03-03T08:01:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2023-03-03T08:01:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2023-03-03T08:01:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2023-03-03T08:01:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-03-03T08:01:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2023-03-03T08:01:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2023-03-03T08:01:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2023-03-03T08:01:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2023-03-03T08:01:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2023-03-03T08:01:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2023-03-03T08:01:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2023-03-03T08:01:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-03-03T08:01:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2023-03-03T08:01:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2023-03-03T08:01:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2023-03-03T08:01:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-03-03T08:01:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2023-03-03T08:01:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2023-03-03T08:01:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2023-03-03T08:01:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2023-03-03T08:01:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2023-03-03T08:01:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2023-03-03T08:01:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2023-03-03T08:01:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2023-03-03T08:01:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2023-03-03T08:01:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2023-03-03T08:01:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2023-03-03T08:01:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2023-03-03T08:01:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2023-03-03T08:01:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2023-03-03T08:01:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2023-03-03T08:01:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2023-03-03T08:01:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2023-03-03T08:01:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2023-03-03T08:01:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2023-03-03T08:01:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.384 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.384 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.384 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.384 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.384 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.384 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.384 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2023-03-03T08:01:59.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2023-03-03T08:01:59.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.385 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.385 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.385 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.385 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.385 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.386 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.386 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.386 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.386 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.386 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.386 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.386 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2023-03-03T08:01:59.387 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.387 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.387 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.387 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2023-03-03T08:01:59.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2023-03-03T08:01:59.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.388 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.388 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.388 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:59.388 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.388 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.388 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.388 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.389 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.389 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.389 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.389 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2023-03-03T08:01:59.389 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2023-03-03T08:01:59.389 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.389 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.390 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.390 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.390 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.390 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.390 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.390 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.390 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.390 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.391 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.391 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.391 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.391 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.391 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.391 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.392 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.392 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.392 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.392 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.392 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.392 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.392 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.392 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.393 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.393 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.393 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.393 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.393 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.393 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.393 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.394 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.394 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.394 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.394 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.394 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.394 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.395 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.395 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.395 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.395 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.395 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.395 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.396 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.396 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.396 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.396 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.396 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.396 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.396 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.397 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.397 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.397 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.397 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.397 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.397 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.397 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.397 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.398 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.398 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.398 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.398 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.398 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.398 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.399 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.399 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.399 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.399 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.399 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.399 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.399 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.399 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.400 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2023-03-03T08:01:59.400 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2023-03-03T08:01:59.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.400 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.400 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.400 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.400 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.401 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.401 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.401 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.401 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.401 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.401 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.402 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2023-03-03T08:01:59.402 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2023-03-03T08:01:59.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.402 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.402 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.402 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.402 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.402 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.403 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.403 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.403 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.403 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.403 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2023-03-03T08:01:59.404 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.404 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.404 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.404 INFO:teuthology.orchestra.run.smithi098.stdout: "drive_group_interval": { 2023-03-03T08:01:59.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "drive_group_interval", 2023-03-03T08:01:59.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:59.405 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.405 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.405 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300.0", 2023-03-03T08:01:59.405 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.405 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.405 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.405 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-03-03T08:01:59.405 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.406 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.406 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.406 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.406 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.406 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.406 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.407 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.407 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.407 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.407 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.407 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.407 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.407 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.408 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.408 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.408 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.408 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.408 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.408 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.408 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.408 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.409 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.409 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.409 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.409 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.409 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.410 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.410 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.410 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.410 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.410 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.410 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.410 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.411 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.411 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.411 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.411 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.411 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.411 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.412 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.412 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.412 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.412 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.412 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.412 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.412 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.412 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.413 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.413 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.413 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.413 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.414 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.414 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.414 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.414 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.414 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.414 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.414 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.414 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.415 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.415 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2023-03-03T08:01:59.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2023-03-03T08:01:59.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.415 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.415 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.416 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2023-03-03T08:01:59.416 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.416 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.416 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.416 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-03-03T08:01:59.417 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.417 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.417 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.417 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.417 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2023-03-03T08:01:59.418 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.418 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.418 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.418 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.418 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.418 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.419 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.419 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.419 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.419 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.419 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.419 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.419 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.419 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.420 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.420 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.420 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.420 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.420 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.420 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.420 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.420 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.421 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.421 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.421 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.421 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.422 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.422 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.422 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.422 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.422 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.422 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.422 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.423 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.423 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.423 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.423 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.423 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.423 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.424 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.424 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.424 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.424 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.424 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.424 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.424 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.424 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.425 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.425 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.425 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.425 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.425 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.425 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.426 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.426 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.426 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.426 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.426 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.426 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.426 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.426 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.427 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.427 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.427 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2023-03-03T08:01:59.427 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2023-03-03T08:01:59.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.427 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.427 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.428 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.428 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.428 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.428 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.428 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.428 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.428 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.428 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.429 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2023-03-03T08:01:59.429 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2023-03-03T08:01:59.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.429 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.429 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.429 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2023-03-03T08:01:59.430 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.430 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.430 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.430 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.430 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.430 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.430 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2023-03-03T08:01:59.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2023-03-03T08:01:59.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.431 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.431 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.431 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.431 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.431 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.432 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.432 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.432 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.432 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.432 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.432 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2023-03-03T08:01:59.432 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2023-03-03T08:01:59.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.433 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.433 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.433 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.433 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.433 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.433 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.434 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.434 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.434 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.434 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.434 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2023-03-03T08:01:59.434 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2023-03-03T08:01:59.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2023-03-03T08:01:59.435 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.435 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.435 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.435 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.435 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.435 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.435 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.435 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.436 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.436 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.436 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2023-03-03T08:01:59.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2023-03-03T08:01:59.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.436 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.436 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.437 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.437 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.437 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.437 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.437 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.437 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.437 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.437 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2023-03-03T08:01:59.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2023-03-03T08:01:59.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.438 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.438 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.439 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.439 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.439 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.439 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.439 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.439 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.439 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.440 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2023-03-03T08:01:59.440 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2023-03-03T08:01:59.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.440 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.440 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.440 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:59.440 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.440 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.441 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.441 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.441 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.441 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.441 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2023-03-03T08:01:59.441 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2023-03-03T08:01:59.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.442 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.442 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.442 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2023-03-03T08:01:59.442 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2023-03-03T08:01:59.442 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.442 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.443 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.443 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.443 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.443 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2023-03-03T08:01:59.443 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2023-03-03T08:01:59.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.443 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.444 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.444 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.444 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.444 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.444 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.444 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.444 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.445 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.445 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2023-03-03T08:01:59.445 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2023-03-03T08:01:59.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.445 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.445 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.446 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.446 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.446 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.446 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.446 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.446 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.446 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.446 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.447 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2023-03-03T08:01:59.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2023-03-03T08:01:59.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.447 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.447 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.447 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.447 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.448 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.448 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.448 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.448 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.448 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.448 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2023-03-03T08:01:59.449 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.449 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.449 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.449 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2023-03-03T08:01:59.450 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2023-03-03T08:01:59.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.450 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.450 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.450 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.450 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.450 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.450 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.451 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2023-03-03T08:01:59.451 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.451 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.451 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.451 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2023-03-03T08:01:59.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2023-03-03T08:01:59.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.452 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.452 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.452 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.452 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.452 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.452 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.452 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2023-03-03T08:01:59.452 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.453 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.453 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.453 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.453 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.453 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.454 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.454 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.454 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.454 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.454 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.454 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.455 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.455 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.455 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.455 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.455 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.455 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.455 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.455 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.456 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.456 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.456 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.456 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.456 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.457 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.457 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.457 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.457 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.457 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.457 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.457 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.458 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.458 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.458 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.458 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.458 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.458 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.458 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.459 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.459 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.459 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.459 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.459 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.459 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.459 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.460 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.460 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.460 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.460 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.460 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.460 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.461 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.461 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.461 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.461 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.461 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.461 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.461 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.461 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.462 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.462 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.462 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2023-03-03T08:01:59.462 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.463 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.463 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.463 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.463 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.463 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.463 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.464 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.464 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.464 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.464 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.464 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.464 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.464 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.464 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.465 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.465 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.465 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.465 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.465 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.466 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.466 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.466 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.466 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.466 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.466 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.467 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.467 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.467 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.467 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.467 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.467 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.467 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.468 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.468 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.468 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.468 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.468 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.468 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.469 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.469 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.469 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.469 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.469 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.469 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.469 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.469 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.470 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.470 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.470 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.470 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.470 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.471 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.471 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.471 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.471 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.471 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.471 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.471 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.471 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2023-03-03T08:01:59.472 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.472 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.472 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.473 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.473 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.473 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.473 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.473 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.473 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.474 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.474 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.474 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.474 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.474 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.474 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.474 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.474 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.475 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.475 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.475 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.475 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.475 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.475 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.475 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.476 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.476 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.476 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.476 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.476 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.476 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.476 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.477 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.477 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.477 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.477 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.477 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.477 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.478 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.478 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.478 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.478 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.478 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.478 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.478 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.479 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.479 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.479 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.479 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.479 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.479 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.479 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.480 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.480 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.480 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.480 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.480 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.480 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.481 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.481 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.481 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.481 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.481 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.481 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.481 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.482 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2023-03-03T08:01:59.482 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.482 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.482 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.482 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2023-03-03T08:01:59.483 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2023-03-03T08:01:59.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.483 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.483 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.483 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-03-03T08:01:59.483 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.483 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.484 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.484 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.484 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.484 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.484 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.484 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2023-03-03T08:01:59.484 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2023-03-03T08:01:59.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:59.485 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.485 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.485 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2023-03-03T08:01:59.485 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.485 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.485 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.485 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.486 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.486 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.486 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.486 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.486 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.486 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.487 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.487 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.487 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.487 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.487 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.487 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.487 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.487 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.488 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.488 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.488 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.488 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.488 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.488 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.488 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.489 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.489 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.489 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.489 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.489 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.489 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.490 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.490 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.490 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.490 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.490 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.490 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.490 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.491 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.491 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.491 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.491 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.491 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.491 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.492 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.492 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.492 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.492 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.492 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.492 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.492 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.492 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.493 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.493 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.493 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.493 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.493 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.494 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.494 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.494 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.494 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.494 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.494 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.495 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.495 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.495 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2023-03-03T08:01:59.496 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.496 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.496 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.496 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2023-03-03T08:01:59.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2023-03-03T08:01:59.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.496 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.496 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.497 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:59.497 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.497 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.497 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.497 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2023-03-03T08:01:59.497 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.497 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.497 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.498 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2023-03-03T08:01:59.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2023-03-03T08:01:59.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.498 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.498 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.498 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:59.498 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.499 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.499 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.499 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-03-03T08:01:59.499 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.499 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.499 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.499 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2023-03-03T08:01:59.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2023-03-03T08:01:59.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.500 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.500 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.500 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2023-03-03T08:01:59.500 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.500 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.500 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.501 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-03-03T08:01:59.501 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.501 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.501 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.501 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2023-03-03T08:01:59.501 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2023-03-03T08:01:59.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.502 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.502 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.502 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.502 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.502 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.502 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-03-03T08:01:59.503 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.503 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.503 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.503 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2023-03-03T08:01:59.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2023-03-03T08:01:59.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.503 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.504 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.504 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.504 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.504 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.504 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2023-03-03T08:01:59.504 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.504 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.505 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.505 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2023-03-03T08:01:59.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2023-03-03T08:01:59.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.505 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.505 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.505 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.506 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.506 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.506 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.506 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.506 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.506 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.506 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.507 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2023-03-03T08:01:59.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2023-03-03T08:01:59.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.507 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.507 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.507 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.507 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.508 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.508 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.508 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.508 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.508 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.508 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.508 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2023-03-03T08:01:59.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2023-03-03T08:01:59.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.509 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.509 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.509 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-03-03T08:01:59.509 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.509 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.510 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.510 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.510 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.510 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.510 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.510 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2023-03-03T08:01:59.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2023-03-03T08:01:59.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.511 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.511 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.511 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.511 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.511 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.511 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.511 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.512 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.512 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.512 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.512 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2023-03-03T08:01:59.512 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2023-03-03T08:01:59.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.512 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.513 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.513 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2023-03-03T08:01:59.513 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2023-03-03T08:01:59.513 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.513 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.513 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.513 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.513 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.514 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.514 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2023-03-03T08:01:59.514 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2023-03-03T08:01:59.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.514 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.514 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.515 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2023-03-03T08:01:59.515 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.515 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.515 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.515 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.515 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.515 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.515 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.516 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2023-03-03T08:01:59.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2023-03-03T08:01:59.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.516 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.516 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.516 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.516 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.517 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.517 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.517 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.517 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.517 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.517 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.517 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.518 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.518 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.518 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.518 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.518 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.518 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.519 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.519 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.519 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.519 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.519 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.519 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.519 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.519 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.520 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.520 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.520 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.520 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.521 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.521 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.521 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.521 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.521 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.521 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.521 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.521 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.522 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.522 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.522 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.522 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.522 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.523 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.523 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.523 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.523 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.523 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.523 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.523 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.523 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.524 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.524 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.524 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.524 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.524 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.524 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.524 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.525 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.525 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.525 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.525 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.525 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.525 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.525 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.526 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.526 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.526 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.526 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.526 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.526 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2023-03-03T08:01:59.526 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2023-03-03T08:01:59.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.527 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.527 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.527 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.527 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.527 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.527 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.527 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.528 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.528 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.528 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.528 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2023-03-03T08:01:59.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2023-03-03T08:01:59.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.529 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.529 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.529 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.529 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.529 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.529 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.529 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.529 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.530 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.530 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.530 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2023-03-03T08:01:59.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2023-03-03T08:01:59.530 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.530 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.530 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.531 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-03-03T08:01:59.531 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.531 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.531 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.531 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.531 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.531 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.531 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.532 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2023-03-03T08:01:59.532 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.532 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.532 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.533 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.533 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.533 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.533 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.533 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.533 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.533 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.534 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.534 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.534 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.534 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.534 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.534 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.534 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.534 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.535 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.535 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.535 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.535 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.535 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.536 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.536 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.536 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.536 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.536 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.536 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.536 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.537 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.537 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.537 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.537 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.537 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.538 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.538 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.538 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.538 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.538 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.538 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.538 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.539 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.539 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.539 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.539 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.539 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.539 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.539 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.539 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.540 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.540 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.540 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.540 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.540 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.541 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.541 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.541 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.541 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.541 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.541 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.541 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.541 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.542 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.542 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2023-03-03T08:01:59.542 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.542 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.542 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.542 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.543 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.543 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.543 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.543 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.543 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.543 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.544 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.544 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.544 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.544 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.544 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.544 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.544 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.544 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.545 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.545 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.545 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.545 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.545 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.545 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.545 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.546 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.546 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.546 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.546 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.546 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.546 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.546 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.546 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.547 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.547 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.547 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.547 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.547 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.547 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.548 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.548 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.548 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.548 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.548 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.548 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.548 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.548 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.549 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.549 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.549 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.549 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.549 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.549 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.549 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.550 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.550 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.550 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.550 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.550 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.550 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.551 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.551 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.551 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.551 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.551 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.551 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.551 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2023-03-03T08:01:59.552 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2023-03-03T08:01:59.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.552 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.552 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.552 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2023-03-03T08:01:59.552 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.552 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.552 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.553 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2023-03-03T08:01:59.553 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.553 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.553 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.553 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2023-03-03T08:01:59.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2023-03-03T08:01:59.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.554 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.554 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.554 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2023-03-03T08:01:59.554 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.554 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.554 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.554 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-03-03T08:01:59.554 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.555 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.555 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.555 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.555 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.555 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2023-03-03T08:01:59.556 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2023-03-03T08:01:59.556 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2023-03-03T08:01:59.556 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2023-03-03T08:01:59.556 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2023-03-03T08:01:59.556 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2023-03-03T08:01:59.556 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2023-03-03T08:01:59.556 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.556 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.557 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2023-03-03T08:01:59.557 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.557 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.557 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.557 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.557 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.557 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.557 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2023-03-03T08:01:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2023-03-03T08:01:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.559 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.559 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.559 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.559 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.559 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.559 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.559 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2023-03-03T08:01:59.560 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2023-03-03T08:01:59.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2023-03-03T08:01:59.560 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.560 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.560 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2023-03-03T08:01:59.560 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.560 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.560 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.561 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.561 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.561 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.561 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.561 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2023-03-03T08:01:59.561 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2023-03-03T08:01:59.562 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.562 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.562 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.562 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.562 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.562 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.562 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.562 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.563 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.563 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.563 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.563 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.563 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.563 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.563 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.563 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.564 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.564 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.564 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2023-03-03T08:01:59.564 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2023-03-03T08:01:59.564 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.564 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.564 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.565 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.565 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.565 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.565 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.565 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.565 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.565 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.565 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.566 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2023-03-03T08:01:59.566 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2023-03-03T08:01:59.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.566 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.566 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.566 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2023-03-03T08:01:59.567 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.567 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.567 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2023-03-03T08:01:59.567 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2023-03-03T08:01:59.567 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2023-03-03T08:01:59.567 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2023-03-03T08:01:59.567 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2023-03-03T08:01:59.567 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2023-03-03T08:01:59.568 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2023-03-03T08:01:59.568 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.568 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.568 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.568 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.568 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.568 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2023-03-03T08:01:59.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2023-03-03T08:01:59.569 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2023-03-03T08:01:59.569 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.569 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2023-03-03T08:01:59.569 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2023-03-03T08:01:59.569 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.569 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.569 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.570 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.570 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.570 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.570 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.570 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2023-03-03T08:01:59.570 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2023-03-03T08:01:59.571 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.571 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.571 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.571 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2023-03-03T08:01:59.571 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.571 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.571 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.571 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.572 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.572 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.572 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.572 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2023-03-03T08:01:59.572 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2023-03-03T08:01:59.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2023-03-03T08:01:59.572 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.573 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.573 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2023-03-03T08:01:59.573 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.573 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.573 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.573 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.573 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.573 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.574 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.574 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2023-03-03T08:01:59.574 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2023-03-03T08:01:59.574 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2023-03-03T08:01:59.574 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2023-03-03T08:01:59.574 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2023-03-03T08:01:59.574 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-03-03T08:01:59.575 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2023-03-03T08:01:59.575 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2023-03-03T08:01:59.575 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2023-03-03T08:01:59.575 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2023-03-03T08:01:59.575 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2023-03-03T08:01:59.575 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2023-03-03T08:01:59.575 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2023-03-03T08:01:59.575 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.576 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.576 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.576 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.576 INFO:teuthology.orchestra.run.smithi098.stdout: "services": {}, 2023-03-03T08:01:59.576 INFO:teuthology.orchestra.run.smithi098.stdout: "always_on_modules": { 2023-03-03T08:01:59.576 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus": [ 2023-03-03T08:01:59.576 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2023-03-03T08:01:59.576 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2023-03-03T08:01:59.577 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2023-03-03T08:01:59.577 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2023-03-03T08:01:59.577 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2023-03-03T08:01:59.577 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2023-03-03T08:01:59.577 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2023-03-03T08:01:59.577 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2023-03-03T08:01:59.577 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2023-03-03T08:01:59.577 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2023-03-03T08:01:59.578 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.578 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific": [ 2023-03-03T08:01:59.578 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2023-03-03T08:01:59.578 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2023-03-03T08:01:59.578 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2023-03-03T08:01:59.578 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2023-03-03T08:01:59.578 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2023-03-03T08:01:59.578 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2023-03-03T08:01:59.579 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2023-03-03T08:01:59.579 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2023-03-03T08:01:59.579 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2023-03-03T08:01:59.579 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2023-03-03T08:01:59.579 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.579 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy": [ 2023-03-03T08:01:59.579 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2023-03-03T08:01:59.579 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2023-03-03T08:01:59.580 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2023-03-03T08:01:59.580 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2023-03-03T08:01:59.580 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2023-03-03T08:01:59.580 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2023-03-03T08:01:59.580 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2023-03-03T08:01:59.580 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2023-03-03T08:01:59.580 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2023-03-03T08:01:59.581 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2023-03-03T08:01:59.581 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:01:59.581 INFO:teuthology.orchestra.run.smithi098.stdout: "reef": [ 2023-03-03T08:01:59.581 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2023-03-03T08:01:59.581 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2023-03-03T08:01:59.581 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2023-03-03T08:01:59.581 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2023-03-03T08:01:59.581 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2023-03-03T08:01:59.582 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2023-03-03T08:01:59.582 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2023-03-03T08:01:59.582 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2023-03-03T08:01:59.582 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2023-03-03T08:01:59.582 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2023-03-03T08:01:59.582 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:01:59.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.582 INFO:teuthology.orchestra.run.smithi098.stdout: "last_failure_osd_epoch": 0, 2023-03-03T08:01:59.583 INFO:teuthology.orchestra.run.smithi098.stdout: "active_clients": [ 2023-03-03T08:01:59.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.583 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-03-03T08:01:59.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-03-03T08:01:59.583 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2023-03-03T08:01:59.583 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1647401399 2023-03-03T08:01:59.583 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.584 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:01:59.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.584 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-03-03T08:01:59.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-03-03T08:01:59.584 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2023-03-03T08:01:59.584 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 918927922 2023-03-03T08:01:59.585 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.585 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:01:59.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.585 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-03-03T08:01:59.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-03-03T08:01:59.585 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2023-03-03T08:01:59.586 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3559163180 2023-03-03T08:01:59.586 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.586 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:01:59.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.586 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-03-03T08:01:59.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-03-03T08:01:59.587 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2023-03-03T08:01:59.587 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1900692259 2023-03-03T08:01:59.587 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.587 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:01:59.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.587 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-03-03T08:01:59.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-03-03T08:01:59.588 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2023-03-03T08:01:59.588 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2323983002 2023-03-03T08:01:59.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.588 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:01:59.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.589 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-03-03T08:01:59.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-03-03T08:01:59.589 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2023-03-03T08:01:59.589 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2444435923 2023-03-03T08:01:59.589 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.589 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:01:59.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-03-03T08:01:59.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-03-03T08:01:59.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2023-03-03T08:01:59.590 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3513799141 2023-03-03T08:01:59.590 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.590 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:01:59.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-03-03T08:01:59.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-03-03T08:01:59.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2023-03-03T08:01:59.591 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3630329692 2023-03-03T08:01:59.591 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.592 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:01:59.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-03-03T08:01:59.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-03-03T08:01:59.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2023-03-03T08:01:59.592 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 4204485361 2023-03-03T08:01:59.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.593 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:01:59.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:01:59.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.593 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-03-03T08:01:59.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:01:59.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-03-03T08:01:59.593 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2023-03-03T08:01:59.594 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3107079680 2023-03-03T08:01:59.594 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.594 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:01:59.594 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:01:59.594 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:01:59.594 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:01:59.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.954+0000 7f7e849ff700 1 -- 172.21.15.98:0/1394485955 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7e600684a0 msgr2=0x7f7e6006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:59.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.954+0000 7f7e849ff700 1 --2- 172.21.15.98:0/1394485955 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7e600684a0 0x7f7e6006a950 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7e800b8fd0 tx=0x7f7e7401c040 comp rx=0 tx=0).stop 2023-03-03T08:01:59.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.954+0000 7f7e849ff700 1 -- 172.21.15.98:0/1394485955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e800b5410 msgr2=0x7f7e8015e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:59.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.954+0000 7f7e849ff700 1 --2- 172.21.15.98:0/1394485955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e800b5410 0x7f7e8015e7d0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000c520 tx=0x7f7e7000c830 comp rx=0 tx=0).stop 2023-03-03T08:01:59.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.954+0000 7f7e849ff700 1 -- 172.21.15.98:0/1394485955 shutdown_connections 2023-03-03T08:01:59.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.954+0000 7f7e849ff700 1 --2- 172.21.15.98:0/1394485955 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f7e600684a0 0x7f7e6006a950 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:59.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.954+0000 7f7e849ff700 1 --2- 172.21.15.98:0/1394485955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e800b5410 0x7f7e8015e7d0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:59.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.954+0000 7f7e849ff700 1 --2- 172.21.15.98:0/1394485955 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e800b8b90 0x7f7e8015bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:59.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.954+0000 7f7e849ff700 1 --2- 172.21.15.98:0/1394485955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e800b87c0 0x7f7e80156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:59.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.954+0000 7f7e849ff700 1 -- 172.21.15.98:0/1394485955 >> 172.21.15.98:0/1394485955 conn(0x7f7e8006aee0 msgr2=0x7f7e800b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:59.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.954+0000 7f7e849ff700 1 -- 172.21.15.98:0/1394485955 shutdown_connections 2023-03-03T08:01:59.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:57.954+0000 7f7e849ff700 1 -- 172.21.15.98:0/1394485955 wait complete. 2023-03-03T08:01:59.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr fail y 2023-03-03T08:01:59.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.853+0000 7f71185ff700 1 Processor -- start 2023-03-03T08:01:59.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.853+0000 7f71185ff700 1 -- start start 2023-03-03T08:01:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.854+0000 7f71185ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f711015ca70 0x7f71100cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.854+0000 7f71185ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71100d0270 0x7f71100d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.854+0000 7f71185ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71100cf980 con 0x7f711006b4b0 2023-03-03T08:01:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.854+0000 7f71185ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71100cd5b0 con 0x7f711015ca70 2023-03-03T08:01:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.854+0000 7f71185ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71100d1800 con 0x7f71100d0270 2023-03-03T08:01:59.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.855+0000 7f7116b9c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71100d0270 0x7f71100d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:59.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.855+0000 7f7116b9c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71100d0270 0x7f71100d1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:39130/0 (socket says 172.21.15.98:39130) 2023-03-03T08:01:59.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.855+0000 7f7116b9c700 1 -- 172.21.15.98:0/3697465175 learned_addr learned my addr 172.21.15.98:0/3697465175 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:59.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.855+0000 7f7115b9a700 1 --1- 172.21.15.98:0/3697465175 >> v1:172.21.15.100:6789/0 conn(0x7f711006b4b0 0x7f71100cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:58444/0 (socket says 172.21.15.98:58444) 2023-03-03T08:01:59.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.855+0000 7f711639b700 1 --2- 172.21.15.98:0/3697465175 >> v2:172.21.15.100:3300/0 conn(0x7f711015ca70 0x7f71100cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:59.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.855+0000 7f7116b9c700 1 -- 172.21.15.98:0/3697465175 >> v2:172.21.15.100:3300/0 conn(0x7f711015ca70 msgr2=0x7f71100cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.855+0000 7f7116b9c700 1 --2- 172.21.15.98:0/3697465175 >> v2:172.21.15.100:3300/0 conn(0x7f711015ca70 0x7f71100cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.855+0000 7f7116b9c700 1 -- 172.21.15.98:0/3697465175 >> v1:172.21.15.100:6789/0 conn(0x7f711006b4b0 legacy=0x7f71100cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.855+0000 7f7115399700 1 -- 172.21.15.98:0/3697465175 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507045620 0 0) 0x7f71100cf980 con 0x7f711006b4b0 2023-03-03T08:01:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.855+0000 7f7116b9c700 1 -- 172.21.15.98:0/3697465175 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71100cd900 con 0x7f71100d0270 2023-03-03T08:01:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.856+0000 7f7116b9c700 1 --2- 172.21.15.98:0/3697465175 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71100d0270 0x7f71100d1fc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7108009e50 tx=0x7f710800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86b667bf47d30fef server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:59.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.856+0000 7f7115399700 1 -- 172.21.15.98:0/3697465175 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710800e070 con 0x7f71100d0270 2023-03-03T08:01:59.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.856+0000 7f7115399700 1 -- 172.21.15.98:0/3697465175 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f710800b910 con 0x7f71100d0270 2023-03-03T08:01:59.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.856+0000 7f7115399700 1 -- 172.21.15.98:0/3697465175 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710800bcc0 con 0x7f71100d0270 2023-03-03T08:01:59.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.857+0000 7f71185ff700 1 -- 172.21.15.98:0/3697465175 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71100d0270 msgr2=0x7f71100d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:59.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.857+0000 7f71185ff700 1 --2- 172.21.15.98:0/3697465175 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71100d0270 0x7f71100d1fc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7108009e50 tx=0x7f710800b330 comp rx=0 tx=0).stop 2023-03-03T08:01:59.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.857+0000 7f71185ff700 1 -- 172.21.15.98:0/3697465175 shutdown_connections 2023-03-03T08:01:59.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.857+0000 7f71185ff700 1 --2- 172.21.15.98:0/3697465175 >> v2:172.21.15.100:3300/0 conn(0x7f711015ca70 0x7f71100cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:59.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.857+0000 7f71185ff700 1 --2- 172.21.15.98:0/3697465175 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f71100d0270 0x7f71100d1fc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:59.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.857+0000 7f71185ff700 1 -- 172.21.15.98:0/3697465175 >> 172.21.15.98:0/3697465175 conn(0x7f7110073e70 msgr2=0x7f711006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:59.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.857+0000 7f71185ff700 1 -- 172.21.15.98:0/3697465175 shutdown_connections 2023-03-03T08:01:59.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.857+0000 7f71185ff700 1 -- 172.21.15.98:0/3697465175 wait complete. 2023-03-03T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.859+0000 7f71185ff700 1 Processor -- start 2023-03-03T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.859+0000 7f71185ff700 1 -- start start 2023-03-03T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.860+0000 7f71185ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f711006b4b0 0x7f71100ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.860+0000 7f71185ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71100d0270 0x7f7110157ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.860+0000 7f71185ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f711015ca70 0x7f7110158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.860+0000 7f71185ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71100b51f0 con 0x7f711015ca70 2023-03-03T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.860+0000 7f71185ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f711014d440 con 0x7f71100d0270 2023-03-03T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.860+0000 7f71185ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71100b5420 con 0x7f711006b4b0 2023-03-03T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.860+0000 7f7116b9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f711015ca70 0x7f7110158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.860+0000 7f7116b9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f711015ca70 0x7f7110158000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47838/0 (socket says 172.21.15.98:47838) 2023-03-03T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.860+0000 7f7116b9c700 1 -- 172.21.15.98:0/3278584950 learned_addr learned my addr 172.21.15.98:0/3278584950 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.860+0000 7f711639b700 1 --2- 172.21.15.98:0/3278584950 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f711006b4b0 0x7f71100ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.860+0000 7f7115b9a700 1 --2- 172.21.15.98:0/3278584950 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71100d0270 0x7f7110157ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.861+0000 7f7116b9c700 1 -- 172.21.15.98:0/3278584950 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f711006b4b0 msgr2=0x7f71100ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.861+0000 7f7116b9c700 1 --2- 172.21.15.98:0/3278584950 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f711006b4b0 0x7f71100ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.861+0000 7f7116b9c700 1 -- 172.21.15.98:0/3278584950 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71100d0270 msgr2=0x7f7110157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.861+0000 7f7116b9c700 1 --2- 172.21.15.98:0/3278584950 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71100d0270 0x7f7110157ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.861+0000 7f7116b9c700 1 -- 172.21.15.98:0/3278584950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7108009b30 con 0x7f711015ca70 2023-03-03T08:01:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.861+0000 7f7115b9a700 1 --2- 172.21.15.98:0/3278584950 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71100d0270 0x7f7110157ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:01:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.861+0000 7f7116b9c700 1 --2- 172.21.15.98:0/3278584950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f711015ca70 0x7f7110158000 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f710800b680 tx=0x7f7108004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.861+0000 7f7106ffd700 1 -- 172.21.15.98:0/3278584950 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7108004960 con 0x7f711015ca70 2023-03-03T08:01:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.861+0000 7f7106ffd700 1 -- 172.21.15.98:0/3278584950 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71080099b0 con 0x7f711015ca70 2023-03-03T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.861+0000 7f71185ff700 1 -- 172.21.15.98:0/3278584950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71100d1aa0 con 0x7f711015ca70 2023-03-03T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.861+0000 7f7106ffd700 1 -- 172.21.15.98:0/3278584950 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710801b650 con 0x7f711015ca70 2023-03-03T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.862+0000 7f71185ff700 1 -- 172.21.15.98:0/3278584950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71100cdbe0 con 0x7f711015ca70 2023-03-03T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.863+0000 7f71185ff700 1 -- 172.21.15.98:0/3278584950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f711015ce40 con 0x7f711015ca70 2023-03-03T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.864+0000 7f7106ffd700 1 -- 172.21.15.98:0/3278584950 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) v1 ==== 86695+0+0 (secure 0 0 0) 0x7f710802b070 con 0x7f711015ca70 2023-03-03T08:01:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.865+0000 7f7106ffd700 1 --2- 172.21.15.98:0/3278584950 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f70f40684e0 0x7f70f406a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.865+0000 7f7106ffd700 1 -- 172.21.15.98:0/3278584950 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7108087880 con 0x7f711015ca70 2023-03-03T08:01:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.865+0000 7f711639b700 1 --2- 172.21.15.98:0/3278584950 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f70f40684e0 0x7f70f406a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.866+0000 7f711639b700 1 --2- 172.21.15.98:0/3278584950 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f70f40684e0 0x7f70f406a990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7110159690 tx=0x7f710c009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:59.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:01:59.869+0000 7f7106ffd700 1 -- 172.21.15.98:0/3278584950 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7108053e00 con 0x7f711015ca70 2023-03-03T08:02:00.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.115+0000 7f71185ff700 1 -- 172.21.15.98:0/3278584950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr fail", "who": "y"} v 0) v1 -- 0x7f711004e6e0 con 0x7f711015ca70 2023-03-03T08:02:00.127 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:00.125+0000 7f817e9f5700 -1 mgr handle_mgr_map I was active but no longer am 2023-03-03T08:02:00.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.126+0000 7f7106ffd700 1 -- 172.21.15.98:0/3278584950 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 13) v1 ==== 43190+0+0 (secure 0 0 0) 0x7f7108021030 con 0x7f711015ca70 2023-03-03T08:02:00.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.126+0000 7f7106ffd700 1 -- 172.21.15.98:0/3278584950 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f70f40684e0 msgr2=0x7f70f406a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:00.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.126+0000 7f7106ffd700 1 --2- 172.21.15.98:0/3278584950 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f70f40684e0 0x7f70f406a990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7110159690 tx=0x7f710c009210 comp rx=0 tx=0).stop 2023-03-03T08:02:00.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.128+0000 7f7106ffd700 1 -- 172.21.15.98:0/3278584950 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{"prefix": "mgr fail", "who": "y"}]=0 v13) v1 ==== 68+0+0 (secure 0 0 0) 0x7f7108087690 con 0x7f711015ca70 2023-03-03T08:02:00.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.132+0000 7f71185ff700 1 -- 172.21.15.98:0/3278584950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f711015ca70 msgr2=0x7f7110158000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:00.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.132+0000 7f71185ff700 1 --2- 172.21.15.98:0/3278584950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f711015ca70 0x7f7110158000 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f710800b680 tx=0x7f7108004ad0 comp rx=0 tx=0).stop 2023-03-03T08:02:00.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.132+0000 7f71185ff700 1 -- 172.21.15.98:0/3278584950 shutdown_connections 2023-03-03T08:02:00.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.132+0000 7f71185ff700 1 --2- 172.21.15.98:0/3278584950 >> [v2:172.21.15.98:6832/108777,v1:172.21.15.98:6833/108777] conn(0x7f70f40684e0 0x7f70f406a990 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:00.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.132+0000 7f71185ff700 1 --2- 172.21.15.98:0/3278584950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f711015ca70 0x7f7110158000 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:00.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.132+0000 7f71185ff700 1 --2- 172.21.15.98:0/3278584950 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f711006b4b0 0x7f71100ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:00.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.132+0000 7f71185ff700 1 --2- 172.21.15.98:0/3278584950 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71100d0270 0x7f7110157ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:00.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.132+0000 7f71185ff700 1 -- 172.21.15.98:0/3278584950 >> 172.21.15.98:0/3278584950 conn(0x7f7110073e70 msgr2=0x7f7110150830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:00.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.133+0000 7f71185ff700 1 -- 172.21.15.98:0/3278584950 shutdown_connections 2023-03-03T08:02:00.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.134+0000 7f71185ff700 1 -- 172.21.15.98:0/3278584950 wait complete. 2023-03-03T08:02:00.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:02:00.262 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:02:00.261+0000 7fcb843cf700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:02:00.263 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:02:00.261+0000 7fcb843cf700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:02:00.263 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:02:00.261+0000 7fcb843cf700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:02:00.263 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:02:00.261+0000 7fcb843cf700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:02:00.263 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:02:00.261+0000 7fcb843cf700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:02:00.263 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:02:00.261+0000 7fcb813c9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:02:00.264 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:02:00.261+0000 7fcb813c9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:02:00.264 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:02:00.261+0000 7fcb813c9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:02:00.264 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:02:00.261+0000 7fcb813c9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:02:00.264 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:02:00.261+0000 7fcb813c9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:02:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.456+0000 7f6a99b9d700 1 Processor -- start 2023-03-03T08:02:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.456+0000 7f6a99b9d700 1 -- start start 2023-03-03T08:02:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.458+0000 7f6a99b9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6a940d0010 0x7f6a940d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.458+0000 7f6a99b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a940b4a00 0x7f6a940d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.458+0000 7f6a99b9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a940b0c50 con 0x7f6a940d1480 2023-03-03T08:02:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.458+0000 7f6a99b9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a940d1900 con 0x7f6a940d0010 2023-03-03T08:02:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.458+0000 7f6a99b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940b0040 con 0x7f6a940b4a00 2023-03-03T08:02:00.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.458+0000 7f6a98b9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6a940d0010 0x7f6a940d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.458+0000 7f6a9939c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6a940d1480 0x7f6a940b10e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45620/0 (socket says 172.21.15.98:45620) 2023-03-03T08:02:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.458+0000 7f6a9939c700 1 -- 172.21.15.98:0/3249748100 learned_addr learned my addr 172.21.15.98:0/3249748100 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.458+0000 7f6a93fff700 1 --2- 172.21.15.98:0/3249748100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a940b4a00 0x7f6a940d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.459+0000 7f6a98b9b700 1 -- 172.21.15.98:0/3249748100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a940b4a00 msgr2=0x7f6a940d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.459+0000 7f6a98b9b700 1 --2- 172.21.15.98:0/3249748100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a940b4a00 0x7f6a940d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.459+0000 7f6a98b9b700 1 -- 172.21.15.98:0/3249748100 >> v1:172.21.15.98:6789/0 conn(0x7f6a940d1480 legacy=0x7f6a940b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.459+0000 7f6a98b9b700 1 -- 172.21.15.98:0/3249748100 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a940b0320 con 0x7f6a940d0010 2023-03-03T08:02:00.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.459+0000 7f6a98b9b700 1 --2- 172.21.15.98:0/3249748100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a940d0010 0x7f6a940d03e0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f6a88009820 tx=0x7f6a88009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf0d0f146880aa96 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:00.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.459+0000 7f6a937fe700 1 -- 172.21.15.98:0/3249748100 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a88005e30 con 0x7f6a940d0010 2023-03-03T08:02:00.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.459+0000 7f6a937fe700 1 -- 172.21.15.98:0/3249748100 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a88006030 con 0x7f6a940d0010 2023-03-03T08:02:00.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.459+0000 7f6a937fe700 1 -- 172.21.15.98:0/3249748100 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a880063c0 con 0x7f6a940d0010 2023-03-03T08:02:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.460+0000 7f6a99b9d700 1 -- 172.21.15.98:0/3249748100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a940d0010 msgr2=0x7f6a940d03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.460+0000 7f6a99b9d700 1 --2- 172.21.15.98:0/3249748100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a940d0010 0x7f6a940d03e0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f6a88009820 tx=0x7f6a88009b30 comp rx=0 tx=0).stop 2023-03-03T08:02:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.460+0000 7f6a99b9d700 1 -- 172.21.15.98:0/3249748100 shutdown_connections 2023-03-03T08:02:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.460+0000 7f6a99b9d700 1 --2- 172.21.15.98:0/3249748100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a940d0010 0x7f6a940d03e0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.460+0000 7f6a99b9d700 1 --2- 172.21.15.98:0/3249748100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a940b4a00 0x7f6a940d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.460+0000 7f6a99b9d700 1 -- 172.21.15.98:0/3249748100 >> 172.21.15.98:0/3249748100 conn(0x7f6a94073b00 msgr2=0x7f6a940d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.460+0000 7f6a99b9d700 1 -- 172.21.15.98:0/3249748100 shutdown_connections 2023-03-03T08:02:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.460+0000 7f6a99b9d700 1 -- 172.21.15.98:0/3249748100 wait complete. 2023-03-03T08:02:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.463+0000 7f6a99b9d700 1 Processor -- start 2023-03-03T08:02:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.464+0000 7f6a99b9d700 1 -- start start 2023-03-03T08:02:00.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.465+0000 7f6a99b9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a940b4a00 0x7f6a9409d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:00.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.465+0000 7f6a99b9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a940d0010 0x7f6a9415c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.465+0000 7f6a99b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a940d1480 0x7f6a940a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.465+0000 7f6a99b9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9415cb40 con 0x7f6a940b4a00 2023-03-03T08:02:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.465+0000 7f6a99b9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940a10a0 con 0x7f6a940d0010 2023-03-03T08:02:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.465+0000 7f6a99b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9409d700 con 0x7f6a940d1480 2023-03-03T08:02:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.465+0000 7f6a98b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a940b4a00 0x7f6a9409d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.465+0000 7f6a98b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a940b4a00 0x7f6a9409d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47884/0 (socket says 172.21.15.98:47884) 2023-03-03T08:02:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.465+0000 7f6a98b9b700 1 -- 172.21.15.98:0/1378655339 learned_addr learned my addr 172.21.15.98:0/1378655339 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.465+0000 7f6a9939c700 1 --2- 172.21.15.98:0/1378655339 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a940d1480 0x7f6a940a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.465+0000 7f6a93fff700 1 --2- 172.21.15.98:0/1378655339 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a940d0010 0x7f6a9415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.466+0000 7f6a98b9b700 1 -- 172.21.15.98:0/1378655339 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a940d1480 msgr2=0x7f6a940a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.466+0000 7f6a98b9b700 1 --2- 172.21.15.98:0/1378655339 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a940d1480 0x7f6a940a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.466+0000 7f6a98b9b700 1 -- 172.21.15.98:0/1378655339 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a940d0010 msgr2=0x7f6a9415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.466+0000 7f6a98b9b700 1 --2- 172.21.15.98:0/1378655339 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a940d0010 0x7f6a9415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.466+0000 7f6a98b9b700 1 -- 172.21.15.98:0/1378655339 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a88009430 con 0x7f6a940b4a00 2023-03-03T08:02:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.466+0000 7f6a9939c700 1 --2- 172.21.15.98:0/1378655339 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a940d1480 0x7f6a940a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.466+0000 7f6a98b9b700 1 --2- 172.21.15.98:0/1378655339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a940b4a00 0x7f6a9409d120 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f6a88000c00 tx=0x7f6a88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.466+0000 7f6a917fa700 1 -- 172.21.15.98:0/1378655339 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a880063a0 con 0x7f6a940b4a00 2023-03-03T08:02:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.466+0000 7f6a917fa700 1 -- 172.21.15.98:0/1378655339 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a88020070 con 0x7f6a940b4a00 2023-03-03T08:02:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.466+0000 7f6a917fa700 1 -- 172.21.15.98:0/1378655339 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a88006740 con 0x7f6a940b4a00 2023-03-03T08:02:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.466+0000 7f6a99b9d700 1 -- 172.21.15.98:0/1378655339 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a940b08e0 con 0x7f6a940b4a00 2023-03-03T08:02:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.467+0000 7f6a99b9d700 1 -- 172.21.15.98:0/1378655339 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a940b0600 con 0x7f6a940b4a00 2023-03-03T08:02:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.468+0000 7f6a917fa700 1 -- 172.21.15.98:0/1378655339 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 43190+0+0 (secure 0 0 0) 0x7f6a8802b020 con 0x7f6a940b4a00 2023-03-03T08:02:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.468+0000 7f6a99b9d700 1 -- 172.21.15.98:0/1378655339 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a940d0b30 con 0x7f6a940b4a00 2023-03-03T08:02:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.468+0000 7f6a917fa700 1 -- 172.21.15.98:0/1378655339 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6621+0+0 (secure 0 0 0) 0x7f6a8804a440 con 0x7f6a940b4a00 2023-03-03T08:02:00.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.474+0000 7f6a917fa700 1 -- 172.21.15.98:0/1378655339 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6a880247e0 con 0x7f6a940b4a00 2023-03-03T08:02:00.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.720+0000 7f6a99b9d700 1 -- 172.21.15.98:0/1378655339 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a940729b0 con 0x7f6a940b4a00 2023-03-03T08:02:00.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.721+0000 7f6a917fa700 1 -- 172.21.15.98:0/1378655339 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f6a8801b410 con 0x7f6a940b4a00 2023-03-03T08:02:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:00.724 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6050,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/318891906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":318891906},{"type":"v1","addr":"172.21.15.98:6836","nonce":318891906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1280578660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1280578660},{"type":"v1","addr":"172.21.15.98:6837","nonce":1280578660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6906,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1395511883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1395511883},{"type":"v1","addr":"172.21.15.100:6834","nonce":1395511883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:01:21.379777+0000","modified":"2023-03-03T08:01:24.835314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6909},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6909":{"gid":6909,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/341424177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":341424177},{"type":"v1","addr":"172.21.15.100:6835","nonce":341424177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-03-03T08:02:00.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.724+0000 7f6a99b9d700 1 -- 172.21.15.98:0/1378655339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a940b4a00 msgr2=0x7f6a9409d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:00.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.724+0000 7f6a99b9d700 1 --2- 172.21.15.98:0/1378655339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a940b4a00 0x7f6a9409d120 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f6a88000c00 tx=0x7f6a88004750 comp rx=0 tx=0).stop 2023-03-03T08:02:00.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.724+0000 7f6a99b9d700 1 -- 172.21.15.98:0/1378655339 shutdown_connections 2023-03-03T08:02:00.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.724+0000 7f6a99b9d700 1 --2- 172.21.15.98:0/1378655339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a940b4a00 0x7f6a9409d120 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.724+0000 7f6a99b9d700 1 --2- 172.21.15.98:0/1378655339 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a940d1480 0x7f6a940a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.724+0000 7f6a99b9d700 1 --2- 172.21.15.98:0/1378655339 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a940d0010 0x7f6a9415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.724+0000 7f6a99b9d700 1 -- 172.21.15.98:0/1378655339 >> 172.21.15.98:0/1378655339 conn(0x7f6a94073b00 msgr2=0x7f6a940b80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.724+0000 7f6a99b9d700 1 -- 172.21.15.98:0/1378655339 shutdown_connections 2023-03-03T08:02:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:00.724+0000 7f6a99b9d700 1 -- 172.21.15.98:0/1378655339 wait complete. 2023-03-03T08:02:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2023-03-03T08:02:00.755 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2023-03-03T08:02:01.032 INFO:teuthology.orchestra.run.smithi100.stdout:[{"id":7102,"entity":{"name":{"type":"client","num":7102},"addr":{"type":"any","addr":"172.21.15.100:0","nonce":3034043584}},"state":"open","num_leases":0,"num_caps":2,"request_load_avg":2,"uptime":0.74198036999999994,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":1.996590405308468,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[],"inst":"client.7102 172.21.15.100:0/3034043584","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"x","hostname":"smithi100","pid":"108673","root":"/"}},{"id":6320,"entity":{"name":{"type":"client","num":6320},"addr":{"type":"any","addr":"172.21.15.100:0","nonce":2007882199}},"state":"open","num_leases":0,"num_caps":2,"request_load_avg":2,"uptime":0.74198036999999994,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":1.996590405308468,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"delegated_inos":[],"inst":"client.6320 172.21.15.100:0/2007882199","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x000000000007ffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"472aaef4ec3b3a2c29f73ca67854785a910f1f9b","ceph_version":"ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)","entity_id":"x","hostname":"smithi100","pid":"108673","root":"/"}}] 2023-03-03T08:02:01.033 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "x", "hostname": "smithi100", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108673", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "172.21.15.100:0", "nonce": 3034043584, "type": "any" }, "name": { "num": 7102, "type": "client" } }, "id": 7102, "inst": "client.7102 172.21.15.100:0/3034043584", "num_caps": 2, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 2, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 1.996590405308468 }, "state": "open", "uptime": 0.74198037 }, { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", "client_features": { "feature_bits": "0x000000000007ffff" }, "entity_id": "x", "hostname": "smithi100", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "108673", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "172.21.15.100:0", "nonce": 2007882199, "type": "any" }, "name": { "num": 6320, "type": "client" } }, "id": 6320, "inst": "client.6320 172.21.15.100:0/2007882199", "num_caps": 2, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 2, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 1.996590405308468 }, "state": "open", "uptime": 0.74198037 } ] 2023-03-03T08:02:01.033 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2023-03-03T08:02:01.034 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-03-03T08:02:01.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:02:01.172 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:02:01.306 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-03-03T08:02:01.307 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:02:01.307 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:02:01.340 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:01.338+0000 7f8457ad6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-03T08:02:01.374 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T08:02:01.375 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T08:02:01.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:01.376 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-03-03T08:02:01.452 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:01.450+0000 7f8457ad6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-03T08:02:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-03-03T08:02:01.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:02:01.486 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:02:01.486 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:02:01.554 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T08:02:01.554 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T08:02:01.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:01.555 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-03-03T08:02:01.563 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:01.561+0000 7f8457ad6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-03T08:02:01.623 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-03-03T08:02:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:02:01.624 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:02:01.625 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:02:01.695 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T08:02:01.695 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T08:02:01.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:01.696 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-03-03T08:02:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-03-03T08:02:01.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:02:01.766 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:02:01.767 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:02:01.828 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:01.825+0000 7f8457ad6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-03T08:02:01.835 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T08:02:01.836 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T08:02:01.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:01.837 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-03-03T08:02:01.905 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-03-03T08:02:01.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:02:01.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:02:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.158+0000 7f08b6d9d700 1 Processor -- start 2023-03-03T08:02:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.158+0000 7f08b6d9d700 1 -- start start 2023-03-03T08:02:02.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.160+0000 7f08b6d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08b00d0010 0x7f08b00d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.160+0000 7f08b6d9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b00b0c50 con 0x7f08b00d1650 2023-03-03T08:02:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.160+0000 7f08b6d9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b00b0040 con 0x7f08b00b4a00 2023-03-03T08:02:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.160+0000 7f08b6d9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b00b0320 con 0x7f08b00d0010 2023-03-03T08:02:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.160+0000 7f08b5d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08b00d0010 0x7f08b00d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.160+0000 7f08b5d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08b00d0010 0x7f08b00d03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47890/0 (socket says 172.21.15.98:47890) 2023-03-03T08:02:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.160+0000 7f08b659c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f08b00d1650 0x7f08b00b2ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45634/0 (socket says 172.21.15.98:45634) 2023-03-03T08:02:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.160+0000 7f08b5d9b700 1 -- 172.21.15.98:0/1832454749 learned_addr learned my addr 172.21.15.98:0/1832454749 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.161+0000 7f08b5d9b700 1 -- 172.21.15.98:0/1832454749 >> v1:172.21.15.100:6789/0 conn(0x7f08b00b4a00 legacy=0x7f08b00d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.161+0000 7f08b5d9b700 1 -- 172.21.15.98:0/1832454749 >> v1:172.21.15.98:6789/0 conn(0x7f08b00d1650 legacy=0x7f08b00b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.161+0000 7f08b5d9b700 1 -- 172.21.15.98:0/1832454749 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b00b0600 con 0x7f08b00d0010 2023-03-03T08:02:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.161+0000 7f08b4d99700 1 -- 172.21.15.98:0/1832454749 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729980585 0 0) 0x7f08b00b0c50 con 0x7f08b00d1650 2023-03-03T08:02:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.161+0000 7f08b5d9b700 1 --2- 172.21.15.98:0/1832454749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08b00d0010 0x7f08b00d03e0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0098e0 tx=0x7f08ac009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed2c18901f7dd87 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.162+0000 7f08b4d99700 1 -- 172.21.15.98:0/1832454749 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac00b8a0 con 0x7f08b00d0010 2023-03-03T08:02:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.162+0000 7f08b4d99700 1 -- 172.21.15.98:0/1832454749 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac00ba00 con 0x7f08b00d0010 2023-03-03T08:02:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.162+0000 7f08b4d99700 1 -- 172.21.15.98:0/1832454749 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac00bdb0 con 0x7f08b00d0010 2023-03-03T08:02:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.162+0000 7f08b6d9d700 1 -- 172.21.15.98:0/1832454749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08b00d0010 msgr2=0x7f08b00d03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.162+0000 7f08b6d9d700 1 --2- 172.21.15.98:0/1832454749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08b00d0010 0x7f08b00d03e0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0098e0 tx=0x7f08ac009bf0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.162+0000 7f08b6d9d700 1 -- 172.21.15.98:0/1832454749 shutdown_connections 2023-03-03T08:02:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.162+0000 7f08b6d9d700 1 --2- 172.21.15.98:0/1832454749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08b00d0010 0x7f08b00d03e0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.162+0000 7f08b6d9d700 1 -- 172.21.15.98:0/1832454749 >> 172.21.15.98:0/1832454749 conn(0x7f08b0073b00 msgr2=0x7f08b00d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.163+0000 7f08b6d9d700 1 -- 172.21.15.98:0/1832454749 shutdown_connections 2023-03-03T08:02:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.163+0000 7f08b6d9d700 1 -- 172.21.15.98:0/1832454749 wait complete. 2023-03-03T08:02:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.165+0000 7f08b6d9d700 1 Processor -- start 2023-03-03T08:02:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.165+0000 7f08b6d9d700 1 -- start start 2023-03-03T08:02:02.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.166+0000 7f08b6d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08b00b4a00 0x7f08b00c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:02.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.166+0000 7f08b6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08b00d0010 0x7f08b009e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:02.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.166+0000 7f08b6d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08b00d1650 0x7f08b00b8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:02.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.166+0000 7f08b6d9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00c8e90 con 0x7f08b00d0010 2023-03-03T08:02:02.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.166+0000 7f08b6d9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00c9170 con 0x7f08b00d1650 2023-03-03T08:02:02.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.166+0000 7f08b6d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00c9450 con 0x7f08b00b4a00 2023-03-03T08:02:02.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.166+0000 7f08b559a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08b00d0010 0x7f08b009e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:02.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.166+0000 7f08b559a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08b00d0010 0x7f08b009e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47906/0 (socket says 172.21.15.98:47906) 2023-03-03T08:02:02.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.166+0000 7f08b559a700 1 -- 172.21.15.98:0/1582108084 learned_addr learned my addr 172.21.15.98:0/1582108084 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:02.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.167+0000 7f08b659c700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08b00d1650 0x7f08b00b8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:02.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.167+0000 7f08b5d9b700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08b00b4a00 0x7f08b00c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:02.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.167+0000 7f08b559a700 1 -- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08b00b4a00 msgr2=0x7f08b00c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:02.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.167+0000 7f08b559a700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08b00b4a00 0x7f08b00c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.167+0000 7f08b559a700 1 -- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08b00d1650 msgr2=0x7f08b00b8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:02.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.167+0000 7f08b559a700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08b00d1650 0x7f08b00b8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.167+0000 7f08b559a700 1 -- 172.21.15.98:0/1582108084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac0094f0 con 0x7f08b00d0010 2023-03-03T08:02:02.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.167+0000 7f08b5d9b700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08b00b4a00 0x7f08b00c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:02.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.167+0000 7f08b659c700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08b00d1650 0x7f08b00b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:02.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.167+0000 7f08b559a700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08b00d0010 0x7f08b009e420 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f08a000c500 tx=0x7f08a000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:02.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.167+0000 7f08a67fc700 1 -- 172.21.15.98:0/1582108084 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a00052d0 con 0x7f08b00d0010 2023-03-03T08:02:02.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.167+0000 7f08a67fc700 1 -- 172.21.15.98:0/1582108084 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a0005430 con 0x7f08b00d0010 2023-03-03T08:02:02.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.167+0000 7f08b6d9d700 1 -- 172.21.15.98:0/1582108084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b00d0b30 con 0x7f08b00d0010 2023-03-03T08:02:02.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.167+0000 7f08a67fc700 1 -- 172.21.15.98:0/1582108084 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a00038c0 con 0x7f08b00d0010 2023-03-03T08:02:02.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.168+0000 7f08b6d9d700 1 -- 172.21.15.98:0/1582108084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b00b08e0 con 0x7f08b00d0010 2023-03-03T08:02:02.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.169+0000 7f08b6d9d700 1 -- 172.21.15.98:0/1582108084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b00d0cc0 con 0x7f08b00d0010 2023-03-03T08:02:02.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.169+0000 7f08a67fc700 1 -- 172.21.15.98:0/1582108084 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) v1 ==== 43500+0+0 (secure 0 0 0) 0x7f08a0019030 con 0x7f08b00d0010 2023-03-03T08:02:02.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.170+0000 7f08a67fc700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f08940363f0 0x7f08940388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:02.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.170+0000 7f08a67fc700 1 -- 172.21.15.98:0/1582108084 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6621+0+0 (secure 0 0 0) 0x7f08a0048380 con 0x7f08b00d0010 2023-03-03T08:02:02.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.170+0000 7f08b5d9b700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f08940363f0 0x7f08940388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:02.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.171+0000 7f08b5d9b700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f08940363f0 0x7f08940388a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009130 tx=0x7f08ac008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.175+0000 7f08a67fc700 1 -- 172.21.15.98:0/1582108084 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f08a00196d0 con 0x7f08b00d0010 2023-03-03T08:02:02.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.419+0000 7f08b6d9d700 1 -- 172.21.15.98:0/1582108084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08b00729b0 con 0x7f08b00d0010 2023-03-03T08:02:02.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.420+0000 7f08a67fc700 1 -- 172.21.15.98:0/1582108084 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f08a0017020 con 0x7f08b00d0010 2023-03-03T08:02:02.423 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:02.423 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6050,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/318891906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":318891906},{"type":"v1","addr":"172.21.15.98:6836","nonce":318891906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1280578660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1280578660},{"type":"v1","addr":"172.21.15.98:6837","nonce":1280578660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6906,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1395511883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1395511883},{"type":"v1","addr":"172.21.15.100:6834","nonce":1395511883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:01:21.379777+0000","modified":"2023-03-03T08:01:24.835314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6909},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6909":{"gid":6909,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/341424177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":341424177},{"type":"v1","addr":"172.21.15.100:6835","nonce":341424177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-03-03T08:02:02.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.423+0000 7f08b6d9d700 1 -- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f08940363f0 msgr2=0x7f08940388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:02.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.423+0000 7f08b6d9d700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f08940363f0 0x7f08940388a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009130 tx=0x7f08ac008030 comp rx=0 tx=0).stop 2023-03-03T08:02:02.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.423+0000 7f08b6d9d700 1 -- 172.21.15.98:0/1582108084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08b00d0010 msgr2=0x7f08b009e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:02.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.423+0000 7f08b6d9d700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08b00d0010 0x7f08b009e420 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f08a000c500 tx=0x7f08a000c8c0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.423+0000 7f08b6d9d700 1 -- 172.21.15.98:0/1582108084 shutdown_connections 2023-03-03T08:02:02.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.423+0000 7f08b6d9d700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08b00d1650 0x7f08b00b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.423+0000 7f08b6d9d700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08b00b4a00 0x7f08b00c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.423+0000 7f08b6d9d700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f08940363f0 0x7f08940388a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.423+0000 7f08b6d9d700 1 --2- 172.21.15.98:0/1582108084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08b00d0010 0x7f08b009e420 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.423+0000 7f08b6d9d700 1 -- 172.21.15.98:0/1582108084 >> 172.21.15.98:0/1582108084 conn(0x7f08b0073b00 msgr2=0x7f08b00b1970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:02.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.423+0000 7f08b6d9d700 1 -- 172.21.15.98:0/1582108084 shutdown_connections 2023-03-03T08:02:02.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.423+0000 7f08b6d9d700 1 -- 172.21.15.98:0/1582108084 wait complete. 2023-03-03T08:02:02.428 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2023-03-03T08:02:02.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:02:02.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.715+0000 7f419359e700 1 Processor -- start 2023-03-03T08:02:02.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.715+0000 7f419359e700 1 -- start start 2023-03-03T08:02:02.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.718+0000 7f419359e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f41940b51c0 0x7f4194160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:02.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.718+0000 7f419359e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41940d1580 0x7f41940d19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:02.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.718+0000 7f419359e700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41941614c0 con 0x7f4194146360 2023-03-03T08:02:02.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.718+0000 7f419359e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41940d10f0 con 0x7f41940b51c0 2023-03-03T08:02:02.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.718+0000 7f419359e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41940cdf20 con 0x7f41940d1580 2023-03-03T08:02:02.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.718+0000 7f4192d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41940d1580 0x7f41940d19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:02.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.718+0000 7f4191d9b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f41940b51c0 0x7f4194160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:02.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.718+0000 7f419259c700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f4194146360 0x7f41940b4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:58454/0 (socket says 172.21.15.98:58454) 2023-03-03T08:02:02.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.718+0000 7f419259c700 1 -- 172.21.15.98:0/1363413234 learned_addr learned my addr 172.21.15.98:0/1363413234 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:02.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.719+0000 7f4192d9d700 1 -- 172.21.15.98:0/1363413234 >> v2:172.21.15.100:3300/0 conn(0x7f41940b51c0 msgr2=0x7f4194160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:02.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.719+0000 7f4192d9d700 1 --2- 172.21.15.98:0/1363413234 >> v2:172.21.15.100:3300/0 conn(0x7f41940b51c0 0x7f4194160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.719+0000 7f4192d9d700 1 -- 172.21.15.98:0/1363413234 >> v1:172.21.15.100:6789/0 conn(0x7f4194146360 legacy=0x7f41940b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:02.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.719+0000 7f419159a700 1 -- 172.21.15.98:0/1363413234 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828724355 0 0) 0x7f4184001110 con 0x7f4194146360 2023-03-03T08:02:02.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.719+0000 7f4192d9d700 1 -- 172.21.15.98:0/1363413234 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41940cd680 con 0x7f41940d1580 2023-03-03T08:02:02.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.720+0000 7f4192d9d700 1 --2- 172.21.15.98:0/1363413234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41940d1580 0x7f41940d19c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f418c009d90 tx=0x7f418c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ff1267022611070 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:02.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.720+0000 7f419159a700 1 -- 172.21.15.98:0/1363413234 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c00e070 con 0x7f41940d1580 2023-03-03T08:02:02.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.720+0000 7f419159a700 1 -- 172.21.15.98:0/1363413234 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418c00bb20 con 0x7f41940d1580 2023-03-03T08:02:02.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.720+0000 7f419159a700 1 -- 172.21.15.98:0/1363413234 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c012430 con 0x7f41940d1580 2023-03-03T08:02:02.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.721+0000 7f419359e700 1 -- 172.21.15.98:0/1363413234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41940d1580 msgr2=0x7f41940d19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:02.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.721+0000 7f419359e700 1 --2- 172.21.15.98:0/1363413234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41940d1580 0x7f41940d19c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f418c009d90 tx=0x7f418c00b330 comp rx=0 tx=0).stop 2023-03-03T08:02:02.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.721+0000 7f419359e700 1 -- 172.21.15.98:0/1363413234 shutdown_connections 2023-03-03T08:02:02.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.721+0000 7f419359e700 1 --2- 172.21.15.98:0/1363413234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41940d1580 0x7f41940d19c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.721+0000 7f419359e700 1 --2- 172.21.15.98:0/1363413234 >> v2:172.21.15.100:3300/0 conn(0x7f41940b51c0 0x7f4194160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.721+0000 7f419359e700 1 -- 172.21.15.98:0/1363413234 >> 172.21.15.98:0/1363413234 conn(0x7f419406b8e0 msgr2=0x7f41940cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:02.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.721+0000 7f419359e700 1 -- 172.21.15.98:0/1363413234 shutdown_connections 2023-03-03T08:02:02.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.721+0000 7f419359e700 1 -- 172.21.15.98:0/1363413234 wait complete. 2023-03-03T08:02:02.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.724+0000 7f419359e700 1 Processor -- start 2023-03-03T08:02:02.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.725+0000 7f419359e700 1 -- start start 2023-03-03T08:02:02.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.727+0000 7f419359e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41940b51c0 0x7f41941505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:02.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.727+0000 7f419359e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41940d1580 0x7f41940ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:02.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.727+0000 7f419359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4194146360 0x7f41940ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:02.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.727+0000 7f419359e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41941f6a60 con 0x7f4194146360 2023-03-03T08:02:02.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.727+0000 7f419359e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41940b78e0 con 0x7f41940d1580 2023-03-03T08:02:02.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.727+0000 7f419359e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41940b7bc0 con 0x7f41940b51c0 2023-03-03T08:02:02.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.727+0000 7f419259c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41940b51c0 0x7f41941505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:02.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.727+0000 7f419259c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41940b51c0 0x7f41941505b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:39214/0 (socket says 172.21.15.98:39214) 2023-03-03T08:02:02.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.727+0000 7f419259c700 1 -- 172.21.15.98:0/1443763948 learned_addr learned my addr 172.21.15.98:0/1443763948 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:02.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.727+0000 7f4191d9b700 1 --2- 172.21.15.98:0/1443763948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41940d1580 0x7f41940ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:02.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.728+0000 7f4192d9d700 1 --2- 172.21.15.98:0/1443763948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4194146360 0x7f41940ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:02.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.728+0000 7f419259c700 1 -- 172.21.15.98:0/1443763948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41940d1580 msgr2=0x7f41940ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:02.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.728+0000 7f419259c700 1 --2- 172.21.15.98:0/1443763948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41940d1580 0x7f41940ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.728+0000 7f419259c700 1 -- 172.21.15.98:0/1443763948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4194146360 msgr2=0x7f41940ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:02.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.728+0000 7f419259c700 1 --2- 172.21.15.98:0/1443763948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4194146360 0x7f41940ca640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.728+0000 7f419259c700 1 -- 172.21.15.98:0/1443763948 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c009a70 con 0x7f41940b51c0 2023-03-03T08:02:02.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.728+0000 7f4192d9d700 1 --2- 172.21.15.98:0/1443763948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4194146360 0x7f41940ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:02.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.728+0000 7f419259c700 1 --2- 172.21.15.98:0/1443763948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41940b51c0 0x7f41941505b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f418400c730 tx=0x7f418400caf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:02.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.729+0000 7f4182ffd700 1 -- 172.21.15.98:0/1443763948 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4184004c80 con 0x7f41940b51c0 2023-03-03T08:02:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.729+0000 7f4182ffd700 1 -- 172.21.15.98:0/1443763948 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4184004de0 con 0x7f41940b51c0 2023-03-03T08:02:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.729+0000 7f4182ffd700 1 -- 172.21.15.98:0/1443763948 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4184007310 con 0x7f41940b51c0 2023-03-03T08:02:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.729+0000 7f419359e700 1 -- 172.21.15.98:0/1443763948 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41940cdc40 con 0x7f41940b51c0 2023-03-03T08:02:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.729+0000 7f419359e700 1 -- 172.21.15.98:0/1443763948 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41940cd960 con 0x7f41940b51c0 2023-03-03T08:02:02.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.731+0000 7f419359e700 1 -- 172.21.15.98:0/1443763948 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41940d0d80 con 0x7f41940b51c0 2023-03-03T08:02:02.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.732+0000 7f4182ffd700 1 -- 172.21.15.98:0/1443763948 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 14) v1 ==== 43500+0+0 (secure 0 0 0) 0x7f418401a020 con 0x7f41940b51c0 2023-03-03T08:02:02.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.743+0000 7f4182ffd700 1 --2- 172.21.15.98:0/1443763948 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f41740363f0 0x7f41740388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:02.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.743+0000 7f4182ffd700 1 -- 172.21.15.98:0/1443763948 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6621+0+0 (secure 0 0 0) 0x7f4184048b60 con 0x7f41940b51c0 2023-03-03T08:02:02.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.743+0000 7f4191d9b700 1 --2- 172.21.15.98:0/1443763948 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f41740363f0 0x7f41740388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.749+0000 7f4182ffd700 1 -- 172.21.15.98:0/1443763948 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4184011aa0 con 0x7f41940b51c0 2023-03-03T08:02:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.749+0000 7f4191d9b700 1 --2- 172.21.15.98:0/1443763948 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f41740363f0 0x7f41740388a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f419415a980 tx=0x7f4188008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:02.844 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:02.841+0000 7f8457ad6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-03T08:02:02.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.958+0000 7f419359e700 1 -- 172.21.15.98:0/1443763948 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41941614c0 con 0x7f41940b51c0 2023-03-03T08:02:02.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.960+0000 7f4182ffd700 1 -- 172.21.15.98:0/1443763948 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18784 (secure 0 0 0) 0x7f418401b340 con 0x7f41940b51c0 2023-03-03T08:02:02.963 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:02.963 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":79,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:02:00.117924+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:01:19.439394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-03-03T08:01:20.381345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/4204485361":"2023-03-04T08:02:00.117879+0000","172.21.15.98:6833/108777":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/3513799141":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/2444435923":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/2644355772":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/1900692259":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/3107079680":"2023-03-04T08:02:00.117879+0000","172.21.15.98:6832/108777":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/3559163180":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/918927922":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/1647401399":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/3630329692":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/2323983002":"2023-03-04T08:02:00.117879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:02:02.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.963+0000 7f419359e700 1 -- 172.21.15.98:0/1443763948 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f41740363f0 msgr2=0x7f41740388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:02.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.963+0000 7f419359e700 1 --2- 172.21.15.98:0/1443763948 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f41740363f0 0x7f41740388a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f419415a980 tx=0x7f4188008040 comp rx=0 tx=0).stop 2023-03-03T08:02:02.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.963+0000 7f419359e700 1 -- 172.21.15.98:0/1443763948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41940b51c0 msgr2=0x7f41941505b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:02.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.963+0000 7f419359e700 1 --2- 172.21.15.98:0/1443763948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41940b51c0 0x7f41941505b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f418400c730 tx=0x7f418400caf0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.963+0000 7f419359e700 1 -- 172.21.15.98:0/1443763948 shutdown_connections 2023-03-03T08:02:02.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.963+0000 7f419359e700 1 --2- 172.21.15.98:0/1443763948 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f41740363f0 0x7f41740388a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.963+0000 7f419359e700 1 --2- 172.21.15.98:0/1443763948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4194146360 0x7f41940ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.963+0000 7f419359e700 1 --2- 172.21.15.98:0/1443763948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41940b51c0 0x7f41941505b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.963+0000 7f419359e700 1 --2- 172.21.15.98:0/1443763948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41940d1580 0x7f41940ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:02.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.963+0000 7f419359e700 1 -- 172.21.15.98:0/1443763948 >> 172.21.15.98:0/1443763948 conn(0x7f419406b8e0 msgr2=0x7f41940b44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:02.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.963+0000 7f419359e700 1 -- 172.21.15.98:0/1443763948 shutdown_connections 2023-03-03T08:02:02.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:02.963+0000 7f419359e700 1 -- 172.21.15.98:0/1443763948 wait complete. 2023-03-03T08:02:02.993 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T08:01:19.439394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T08:01:20.381345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:02:02.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:02:03.083 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:03.081+0000 7f8457ad6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-03T08:02:03.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.247+0000 7fe7f41ff700 1 Processor -- start 2023-03-03T08:02:03.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.248+0000 7fe7f41ff700 1 -- start start 2023-03-03T08:02:03.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.249+0000 7fe7f41ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe7ec15ca70 0x7fe7ec0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:03.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.249+0000 7fe7f41ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ec0cf980 con 0x7fe7ec06b4b0 2023-03-03T08:02:03.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.249+0000 7fe7f41ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ec0d1800 con 0x7fe7ec0d0270 2023-03-03T08:02:03.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.249+0000 7fe7f41ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7ec0cd5b0 con 0x7fe7ec15ca70 2023-03-03T08:02:03.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.249+0000 7fe7f1f9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe7ec15ca70 0x7fe7ec0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:03.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.249+0000 7fe7f179a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe7ec06b4b0 0x7fe7ec0cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45644/0 (socket says 172.21.15.98:45644) 2023-03-03T08:02:03.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.249+0000 7fe7f179a700 1 -- 172.21.15.98:0/464670005 learned_addr learned my addr 172.21.15.98:0/464670005 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:03.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.250+0000 7fe7f1f9b700 1 -- 172.21.15.98:0/464670005 >> v1:172.21.15.100:6789/0 conn(0x7fe7ec0d0270 legacy=0x7fe7ec0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:03.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.250+0000 7fe7f1f9b700 1 -- 172.21.15.98:0/464670005 >> v1:172.21.15.98:6789/0 conn(0x7fe7ec06b4b0 legacy=0x7fe7ec0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:03.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.250+0000 7fe7f1f9b700 1 -- 172.21.15.98:0/464670005 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ec0cd900 con 0x7fe7ec15ca70 2023-03-03T08:02:03.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.250+0000 7fe7f0f99700 1 -- 172.21.15.98:0/464670005 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527753410 0 0) 0x7fe7ec0cf980 con 0x7fe7ec06b4b0 2023-03-03T08:02:03.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.250+0000 7fe7f1f9b700 1 --2- 172.21.15.98:0/464670005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7ec15ca70 0x7fe7ec0cf440 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8009840 tx=0x7fe7e8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b59d2a25ec16ec6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:03.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.251+0000 7fe7f0f99700 1 -- 172.21.15.98:0/464670005 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8011830 con 0x7fe7ec15ca70 2023-03-03T08:02:03.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.251+0000 7fe7f0f99700 1 -- 172.21.15.98:0/464670005 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e8011990 con 0x7fe7ec15ca70 2023-03-03T08:02:03.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.251+0000 7fe7f0f99700 1 -- 172.21.15.98:0/464670005 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8011d20 con 0x7fe7ec15ca70 2023-03-03T08:02:03.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.251+0000 7fe7f41ff700 1 -- 172.21.15.98:0/464670005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7ec15ca70 msgr2=0x7fe7ec0cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:03.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.251+0000 7fe7f41ff700 1 --2- 172.21.15.98:0/464670005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7ec15ca70 0x7fe7ec0cf440 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8009840 tx=0x7fe7e8009b50 comp rx=0 tx=0).stop 2023-03-03T08:02:03.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.252+0000 7fe7f41ff700 1 -- 172.21.15.98:0/464670005 shutdown_connections 2023-03-03T08:02:03.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.252+0000 7fe7f41ff700 1 --2- 172.21.15.98:0/464670005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7ec15ca70 0x7fe7ec0cf440 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:03.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.252+0000 7fe7f41ff700 1 -- 172.21.15.98:0/464670005 >> 172.21.15.98:0/464670005 conn(0x7fe7ec073e70 msgr2=0x7fe7ec06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:03.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.252+0000 7fe7f41ff700 1 -- 172.21.15.98:0/464670005 shutdown_connections 2023-03-03T08:02:03.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.252+0000 7fe7f41ff700 1 -- 172.21.15.98:0/464670005 wait complete. 2023-03-03T08:02:03.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.254+0000 7fe7f41ff700 1 Processor -- start 2023-03-03T08:02:03.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.254+0000 7fe7f41ff700 1 -- start start 2023-03-03T08:02:03.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.255+0000 7fe7f41ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe7ec06b4b0 0x7fe7ec156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:03.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.255+0000 7fe7f41ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7ec0d0270 0x7fe7ec1580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:03.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.255+0000 7fe7f41ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe7ec15ca70 0x7fe7ec15d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:03.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.255+0000 7fe7f41ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec1e52a0 con 0x7fe7ec0d0270 2023-03-03T08:02:03.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.255+0000 7fe7f41ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec1e54f0 con 0x7fe7ec15ca70 2023-03-03T08:02:03.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.255+0000 7fe7f41ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec1e57d0 con 0x7fe7ec06b4b0 2023-03-03T08:02:03.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.255+0000 7fe7f179a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7ec0d0270 0x7fe7ec1580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:03.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.255+0000 7fe7f179a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7ec0d0270 0x7fe7ec1580d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47944/0 (socket says 172.21.15.98:47944) 2023-03-03T08:02:03.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.255+0000 7fe7f179a700 1 -- 172.21.15.98:0/1147008755 learned_addr learned my addr 172.21.15.98:0/1147008755 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:03.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.256+0000 7fe7f1f9b700 1 --2- 172.21.15.98:0/1147008755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe7ec06b4b0 0x7fe7ec156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:03.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.256+0000 7fe7f279c700 1 --2- 172.21.15.98:0/1147008755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe7ec15ca70 0x7fe7ec15d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.256+0000 7fe7f179a700 1 -- 172.21.15.98:0/1147008755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe7ec06b4b0 msgr2=0x7fe7ec156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.256+0000 7fe7f179a700 1 --2- 172.21.15.98:0/1147008755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe7ec06b4b0 0x7fe7ec156a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.256+0000 7fe7f179a700 1 -- 172.21.15.98:0/1147008755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe7ec15ca70 msgr2=0x7fe7ec15d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.256+0000 7fe7f179a700 1 --2- 172.21.15.98:0/1147008755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe7ec15ca70 0x7fe7ec15d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.256+0000 7fe7f179a700 1 -- 172.21.15.98:0/1147008755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e80094a0 con 0x7fe7ec0d0270 2023-03-03T08:02:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.256+0000 7fe7f279c700 1 --2- 172.21.15.98:0/1147008755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe7ec15ca70 0x7fe7ec15d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.256+0000 7fe7f179a700 1 --2- 172.21.15.98:0/1147008755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7ec0d0270 0x7fe7ec1580d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc001eb0 tx=0x7fe7dc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.256+0000 7fe7e27fc700 1 -- 172.21.15.98:0/1147008755 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc004c80 con 0x7fe7ec0d0270 2023-03-03T08:02:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.256+0000 7fe7e27fc700 1 -- 172.21.15.98:0/1147008755 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7dc004de0 con 0x7fe7ec0d0270 2023-03-03T08:02:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.256+0000 7fe7f41ff700 1 -- 172.21.15.98:0/1147008755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ec0d1aa0 con 0x7fe7ec0d0270 2023-03-03T08:02:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.257+0000 7fe7e27fc700 1 -- 172.21.15.98:0/1147008755 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc007310 con 0x7fe7ec0d0270 2023-03-03T08:02:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.257+0000 7fe7f41ff700 1 -- 172.21.15.98:0/1147008755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ec0cdbe0 con 0x7fe7ec0d0270 2023-03-03T08:02:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.258+0000 7fe7e27fc700 1 -- 172.21.15.98:0/1147008755 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) v1 ==== 43500+0+0 (secure 0 0 0) 0x7fe7dc01a020 con 0x7fe7ec0d0270 2023-03-03T08:02:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.258+0000 7fe7f41ff700 1 -- 172.21.15.98:0/1147008755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ec15ce40 con 0x7fe7ec0d0270 2023-03-03T08:02:03.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.259+0000 7fe7e27fc700 1 --2- 172.21.15.98:0/1147008755 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe7d00363f0 0x7fe7d00388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:03.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.259+0000 7fe7e27fc700 1 -- 172.21.15.98:0/1147008755 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6621+0+0 (secure 0 0 0) 0x7fe7dc048bf0 con 0x7fe7ec0d0270 2023-03-03T08:02:03.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.259+0000 7fe7f1f9b700 1 --2- 172.21.15.98:0/1147008755 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe7d00363f0 0x7fe7d00388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:03.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.260+0000 7fe7f1f9b700 1 --2- 172.21.15.98:0/1147008755 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe7d00363f0 0x7fe7d00388a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8014040 tx=0x7fe7e801c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.264+0000 7fe7e27fc700 1 -- 172.21.15.98:0/1147008755 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe7dc007470 con 0x7fe7ec0d0270 2023-03-03T08:02:03.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:02:03.382 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:02:03.391 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:03.389+0000 7f8457ad6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-03T08:02:03.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.465+0000 7fe7f41ff700 1 -- 172.21.15.98:0/1147008755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe7ec04e6e0 con 0x7fe7ec0d0270 2023-03-03T08:02:03.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.466+0000 7fe7e27fc700 1 -- 172.21.15.98:0/1147008755 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe7dc01b340 con 0x7fe7ec0d0270 2023-03-03T08:02:03.469 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:02:03.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.469+0000 7fe7f41ff700 1 -- 172.21.15.98:0/1147008755 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe7d00363f0 msgr2=0x7fe7d00388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:03.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.469+0000 7fe7f41ff700 1 --2- 172.21.15.98:0/1147008755 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe7d00363f0 0x7fe7d00388a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8014040 tx=0x7fe7e801c040 comp rx=0 tx=0).stop 2023-03-03T08:02:03.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.469+0000 7fe7f41ff700 1 -- 172.21.15.98:0/1147008755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7ec0d0270 msgr2=0x7fe7ec1580d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:03.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.469+0000 7fe7f41ff700 1 --2- 172.21.15.98:0/1147008755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7ec0d0270 0x7fe7ec1580d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc001eb0 tx=0x7fe7dc00c800 comp rx=0 tx=0).stop 2023-03-03T08:02:03.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.469+0000 7fe7f41ff700 1 -- 172.21.15.98:0/1147008755 shutdown_connections 2023-03-03T08:02:03.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.469+0000 7fe7f41ff700 1 --2- 172.21.15.98:0/1147008755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe7ec15ca70 0x7fe7ec15d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:03.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.469+0000 7fe7f41ff700 1 --2- 172.21.15.98:0/1147008755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe7ec06b4b0 0x7fe7ec156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:03.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.469+0000 7fe7f41ff700 1 --2- 172.21.15.98:0/1147008755 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe7d00363f0 0x7fe7d00388a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:03.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.469+0000 7fe7f41ff700 1 --2- 172.21.15.98:0/1147008755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7ec0d0270 0x7fe7ec1580d0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.469+0000 7fe7f41ff700 1 -- 172.21.15.98:0/1147008755 >> 172.21.15.98:0/1147008755 conn(0x7fe7ec073e70 msgr2=0x7fe7ec0b6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.469+0000 7fe7f41ff700 1 -- 172.21.15.98:0/1147008755 shutdown_connections 2023-03-03T08:02:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.469+0000 7fe7f41ff700 1 -- 172.21.15.98:0/1147008755 wait complete. 2023-03-03T08:02:03.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:02:03.515 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:03.513+0000 7f8457ad6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-03T08:02:03.755 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:03.753+0000 7f8457ad6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-03T08:02:03.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.764+0000 7f4762d9d700 1 Processor -- start 2023-03-03T08:02:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.765+0000 7f4762d9d700 1 -- start start 2023-03-03T08:02:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.766+0000 7f4762d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475c0b8910 0x7f475c1604a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.766+0000 7f4762d9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f475c15ce00 con 0x7f475c0b5e70 2023-03-03T08:02:03.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.766+0000 7f4762d9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f475c0d0bf0 con 0x7f475c0b51b0 2023-03-03T08:02:03.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.766+0000 7f4762d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f475c0b8ce0 con 0x7f475c0b8910 2023-03-03T08:02:03.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.766+0000 7f476159a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f475c0b5e70 0x7f475c0b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45646/0 (socket says 172.21.15.98:45646) 2023-03-03T08:02:03.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.767+0000 7f4761d9b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475c0b8910 0x7f475c1604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:03.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.767+0000 7f476159a700 1 -- 172.21.15.98:0/3351340303 learned_addr learned my addr 172.21.15.98:0/3351340303 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.767+0000 7f4761d9b700 1 -- 172.21.15.98:0/3351340303 >> v1:172.21.15.100:6789/0 conn(0x7f475c0b51b0 legacy=0x7f475c0d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.767+0000 7f4761d9b700 1 -- 172.21.15.98:0/3351340303 >> v1:172.21.15.98:6789/0 conn(0x7f475c0b5e70 legacy=0x7f475c0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.767+0000 7f4761d9b700 1 -- 172.21.15.98:0/3351340303 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475c160cf0 con 0x7f475c0b8910 2023-03-03T08:02:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.767+0000 7f4760d99700 1 -- 172.21.15.98:0/3351340303 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945045444 0 0) 0x7f474c001110 con 0x7f475c0b5e70 2023-03-03T08:02:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.767+0000 7f4761d9b700 1 --2- 172.21.15.98:0/3351340303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475c0b8910 0x7f475c1604a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4758009870 tx=0x7f4758009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfbe101e4f33a9e6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.768+0000 7f4760d99700 1 -- 172.21.15.98:0/3351340303 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4758011830 con 0x7f475c0b8910 2023-03-03T08:02:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.768+0000 7f4760d99700 1 -- 172.21.15.98:0/3351340303 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4758011990 con 0x7f475c0b8910 2023-03-03T08:02:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.768+0000 7f4760d99700 1 -- 172.21.15.98:0/3351340303 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4758011d20 con 0x7f475c0b8910 2023-03-03T08:02:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.768+0000 7f4762d9d700 1 -- 172.21.15.98:0/3351340303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475c0b8910 msgr2=0x7f475c1604a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.768+0000 7f4762d9d700 1 --2- 172.21.15.98:0/3351340303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475c0b8910 0x7f475c1604a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4758009870 tx=0x7f4758009b80 comp rx=0 tx=0).stop 2023-03-03T08:02:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.769+0000 7f4762d9d700 1 -- 172.21.15.98:0/3351340303 shutdown_connections 2023-03-03T08:02:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.769+0000 7f4762d9d700 1 --2- 172.21.15.98:0/3351340303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475c0b8910 0x7f475c1604a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.769+0000 7f4762d9d700 1 -- 172.21.15.98:0/3351340303 >> 172.21.15.98:0/3351340303 conn(0x7f475c06b880 msgr2=0x7f475c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.769+0000 7f4762d9d700 1 -- 172.21.15.98:0/3351340303 shutdown_connections 2023-03-03T08:02:03.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.769+0000 7f4762d9d700 1 -- 172.21.15.98:0/3351340303 wait complete. 2023-03-03T08:02:03.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.771+0000 7f4762d9d700 1 Processor -- start 2023-03-03T08:02:03.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.771+0000 7f4762d9d700 1 -- start start 2023-03-03T08:02:03.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.772+0000 7f4762d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475c0b51b0 0x7f475c15cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:03.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.772+0000 7f4762d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f475c0b5e70 0x7f475c0cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:03.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.772+0000 7f4762d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f475c0b8910 0x7f475c151510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:03.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.772+0000 7f4762d9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475c1f23a0 con 0x7f475c0b8910 2023-03-03T08:02:03.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.772+0000 7f4762d9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475c0ccc90 con 0x7f475c0b5e70 2023-03-03T08:02:03.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.772+0000 7f4762d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f475c15d630 con 0x7f475c0b51b0 2023-03-03T08:02:03.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.772+0000 7f476259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f475c0b8910 0x7f475c151510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:03.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.773+0000 7f476259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f475c0b8910 0x7f475c151510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47956/0 (socket says 172.21.15.98:47956) 2023-03-03T08:02:03.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.773+0000 7f476259c700 1 -- 172.21.15.98:0/4031819080 learned_addr learned my addr 172.21.15.98:0/4031819080 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:03.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.773+0000 7f476159a700 1 --2- 172.21.15.98:0/4031819080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f475c0b5e70 0x7f475c0cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:03.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.773+0000 7f4761d9b700 1 --2- 172.21.15.98:0/4031819080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475c0b51b0 0x7f475c15cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:03.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.773+0000 7f476259c700 1 -- 172.21.15.98:0/4031819080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475c0b51b0 msgr2=0x7f475c15cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:03.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.773+0000 7f476259c700 1 --2- 172.21.15.98:0/4031819080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475c0b51b0 0x7f475c15cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:03.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.773+0000 7f476259c700 1 -- 172.21.15.98:0/4031819080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f475c0b5e70 msgr2=0x7f475c0cafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:03.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.773+0000 7f476259c700 1 --2- 172.21.15.98:0/4031819080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f475c0b5e70 0x7f475c0cafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:03.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.773+0000 7f476259c700 1 -- 172.21.15.98:0/4031819080 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47580094a0 con 0x7f475c0b8910 2023-03-03T08:02:03.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.773+0000 7f476159a700 1 --2- 172.21.15.98:0/4031819080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f475c0b5e70 0x7f475c0cafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:03.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.773+0000 7f476259c700 1 --2- 172.21.15.98:0/4031819080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f475c0b8910 0x7f475c151510 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f475400cb90 tx=0x7f475400cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:03.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.774+0000 7f47527fc700 1 -- 172.21.15.98:0/4031819080 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4754003c00 con 0x7f475c0b8910 2023-03-03T08:02:03.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.774+0000 7f47527fc700 1 -- 172.21.15.98:0/4031819080 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4754003d60 con 0x7f475c0b8910 2023-03-03T08:02:03.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.774+0000 7f47527fc700 1 -- 172.21.15.98:0/4031819080 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475400aae0 con 0x7f475c0b8910 2023-03-03T08:02:03.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.774+0000 7f4762d9d700 1 -- 172.21.15.98:0/4031819080 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f475c1612b0 con 0x7f475c0b8910 2023-03-03T08:02:03.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.774+0000 7f4762d9d700 1 -- 172.21.15.98:0/4031819080 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f475c160fd0 con 0x7f475c0b8910 2023-03-03T08:02:03.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.775+0000 7f47527fc700 1 -- 172.21.15.98:0/4031819080 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) v1 ==== 43500+0+0 (secure 0 0 0) 0x7f4754010050 con 0x7f475c0b8910 2023-03-03T08:02:03.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.775+0000 7f4762d9d700 1 -- 172.21.15.98:0/4031819080 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f475c0d0eb0 con 0x7f475c0b8910 2023-03-03T08:02:03.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.776+0000 7f47527fc700 1 --2- 172.21.15.98:0/4031819080 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f47400363f0 0x7f47400388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:03.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.776+0000 7f47527fc700 1 -- 172.21.15.98:0/4031819080 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6621+0+0 (secure 0 0 0) 0x7f4754048e30 con 0x7f475c0b8910 2023-03-03T08:02:03.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.777+0000 7f4761d9b700 1 --2- 172.21.15.98:0/4031819080 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f47400363f0 0x7f47400388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:03.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.777+0000 7f4761d9b700 1 --2- 172.21.15.98:0/4031819080 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f47400363f0 0x7f47400388a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f47580090b0 tx=0x7f475801c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:03.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.785+0000 7f47527fc700 1 -- 172.21.15.98:0/4031819080 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f475404d070 con 0x7f475c0b8910 2023-03-03T08:02:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.985+0000 7f4762d9d700 1 -- 172.21.15.98:0/4031819080 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f475c068290 con 0x7f475c0b8910 2023-03-03T08:02:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.986+0000 7f47527fc700 1 -- 172.21.15.98:0/4031819080 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v79) v1 ==== 92+0+10 (secure 0 0 0) 0x7f47540194e0 con 0x7f475c0b8910 2023-03-03T08:02:03.989 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:02:03.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.989+0000 7f4762d9d700 1 -- 172.21.15.98:0/4031819080 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f47400363f0 msgr2=0x7f47400388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:03.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.989+0000 7f4762d9d700 1 --2- 172.21.15.98:0/4031819080 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f47400363f0 0x7f47400388a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f47580090b0 tx=0x7f475801c040 comp rx=0 tx=0).stop 2023-03-03T08:02:03.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.989+0000 7f4762d9d700 1 -- 172.21.15.98:0/4031819080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f475c0b8910 msgr2=0x7f475c151510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:03.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.989+0000 7f4762d9d700 1 --2- 172.21.15.98:0/4031819080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f475c0b8910 0x7f475c151510 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f475400cb90 tx=0x7f475400cea0 comp rx=0 tx=0).stop 2023-03-03T08:02:03.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.989+0000 7f4762d9d700 1 -- 172.21.15.98:0/4031819080 shutdown_connections 2023-03-03T08:02:03.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.989+0000 7f4762d9d700 1 --2- 172.21.15.98:0/4031819080 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f47400363f0 0x7f47400388a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:03.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.989+0000 7f4762d9d700 1 --2- 172.21.15.98:0/4031819080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f475c0b8910 0x7f475c151510 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:03.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.989+0000 7f4762d9d700 1 --2- 172.21.15.98:0/4031819080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475c0b51b0 0x7f475c15cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:03.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.989+0000 7f4762d9d700 1 --2- 172.21.15.98:0/4031819080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f475c0b5e70 0x7f475c0cafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:03.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.989+0000 7f4762d9d700 1 -- 172.21.15.98:0/4031819080 >> 172.21.15.98:0/4031819080 conn(0x7f475c06b880 msgr2=0x7f475c0b6350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:03.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.989+0000 7f4762d9d700 1 -- 172.21.15.98:0/4031819080 shutdown_connections 2023-03-03T08:02:03.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:03.989+0000 7f4762d9d700 1 -- 172.21.15.98:0/4031819080 wait complete. 2023-03-03T08:02:04.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T08:02:04.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.272+0000 7f01bc9fe700 1 Processor -- start 2023-03-03T08:02:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.272+0000 7f01bc9fe700 1 -- start start 2023-03-03T08:02:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.274+0000 7f01bc9fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f01b80b8910 0x7f01b81604a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.274+0000 7f01bc9fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f01b80b51b0 0x7f01b80d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.274+0000 7f01bc9fe700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b815ce00 con 0x7f01b80b5e70 2023-03-03T08:02:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.274+0000 7f01bc9fe700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b80d0bf0 con 0x7f01b80b51b0 2023-03-03T08:02:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.274+0000 7f01bc9fe700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b80b8ce0 con 0x7f01b80b8910 2023-03-03T08:02:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.274+0000 7f01b759e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f01b80b51b0 0x7f01b80d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.274+0000 7f01b759e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f01b80b51b0 0x7f01b80d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47972/0 (socket says 172.21.15.98:47972) 2023-03-03T08:02:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.274+0000 7f01b759e700 1 -- 172.21.15.98:0/3309236014 learned_addr learned my addr 172.21.15.98:0/3309236014 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.275+0000 7f01b6d9d700 1 --2- 172.21.15.98:0/3309236014 >> v2:172.21.15.100:3300/0 conn(0x7f01b80b8910 0x7f01b81604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.275+0000 7f01b759e700 1 -- 172.21.15.98:0/3309236014 >> v2:172.21.15.100:3300/0 conn(0x7f01b80b8910 msgr2=0x7f01b81604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.275+0000 7f01b759e700 1 --2- 172.21.15.98:0/3309236014 >> v2:172.21.15.100:3300/0 conn(0x7f01b80b8910 0x7f01b81604a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.275+0000 7f01b759e700 1 -- 172.21.15.98:0/3309236014 >> v1:172.21.15.100:6789/0 conn(0x7f01b80b5e70 legacy=0x7f01b80b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.275+0000 7f01b759e700 1 -- 172.21.15.98:0/3309236014 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b8160cf0 con 0x7f01b80b51b0 2023-03-03T08:02:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.275+0000 7f01b759e700 1 --2- 172.21.15.98:0/3309236014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b80b51b0 0x7f01b80d13d0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f01b808bf50 tx=0x7f01b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ef875ac7810c89 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.276+0000 7f01b659c700 1 -- 172.21.15.98:0/3309236014 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b000e070 con 0x7f01b80b51b0 2023-03-03T08:02:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.276+0000 7f01b659c700 1 -- 172.21.15.98:0/3309236014 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b000b9c0 con 0x7f01b80b51b0 2023-03-03T08:02:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.276+0000 7f01b659c700 1 -- 172.21.15.98:0/3309236014 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b000bd70 con 0x7f01b80b51b0 2023-03-03T08:02:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.276+0000 7f01bc9fe700 1 -- 172.21.15.98:0/3309236014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b80b51b0 msgr2=0x7f01b80d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.276+0000 7f01bc9fe700 1 --2- 172.21.15.98:0/3309236014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b80b51b0 0x7f01b80d13d0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f01b808bf50 tx=0x7f01b000b330 comp rx=0 tx=0).stop 2023-03-03T08:02:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.276+0000 7f01bc9fe700 1 -- 172.21.15.98:0/3309236014 shutdown_connections 2023-03-03T08:02:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.276+0000 7f01bc9fe700 1 --2- 172.21.15.98:0/3309236014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b80b51b0 0x7f01b80d13d0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.276+0000 7f01bc9fe700 1 --2- 172.21.15.98:0/3309236014 >> v2:172.21.15.100:3300/0 conn(0x7f01b80b8910 0x7f01b81604a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.276+0000 7f01bc9fe700 1 -- 172.21.15.98:0/3309236014 >> 172.21.15.98:0/3309236014 conn(0x7f01b806b880 msgr2=0x7f01b80cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.276+0000 7f01bc9fe700 1 -- 172.21.15.98:0/3309236014 shutdown_connections 2023-03-03T08:02:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.277+0000 7f01bc9fe700 1 -- 172.21.15.98:0/3309236014 wait complete. 2023-03-03T08:02:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.278+0000 7f01bc9fe700 1 Processor -- start 2023-03-03T08:02:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.279+0000 7f01bc9fe700 1 -- start start 2023-03-03T08:02:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.280+0000 7f01bc9fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01b80b51b0 0x7f01b80cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.280+0000 7f01bc9fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01b80b5e70 0x7f01b815f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.280+0000 7f01bc9fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b80b8910 0x7f01b80c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.280+0000 7f01bc9fe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b80ccfc0 con 0x7f01b80b8910 2023-03-03T08:02:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.280+0000 7f01bc9fe700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b815f2a0 con 0x7f01b80b51b0 2023-03-03T08:02:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.280+0000 7f01bc9fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b815fc70 con 0x7f01b80b5e70 2023-03-03T08:02:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.280+0000 7f01b759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b80b8910 0x7f01b80c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.280+0000 7f01b759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b80b8910 0x7f01b80c8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47980/0 (socket says 172.21.15.98:47980) 2023-03-03T08:02:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.280+0000 7f01b759e700 1 -- 172.21.15.98:0/619850184 learned_addr learned my addr 172.21.15.98:0/619850184 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.280+0000 7f01b6d9d700 1 --2- 172.21.15.98:0/619850184 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01b80b51b0 0x7f01b80cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.280+0000 7f01affff700 1 --2- 172.21.15.98:0/619850184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01b80b5e70 0x7f01b815f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.281+0000 7f01b759e700 1 -- 172.21.15.98:0/619850184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01b80b5e70 msgr2=0x7f01b815f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.281+0000 7f01b759e700 1 --2- 172.21.15.98:0/619850184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01b80b5e70 0x7f01b815f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.281+0000 7f01b759e700 1 -- 172.21.15.98:0/619850184 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01b80b51b0 msgr2=0x7f01b80cca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.281+0000 7f01b759e700 1 --2- 172.21.15.98:0/619850184 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01b80b51b0 0x7f01b80cca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.281+0000 7f01b759e700 1 -- 172.21.15.98:0/619850184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b0009b30 con 0x7f01b80b8910 2023-03-03T08:02:04.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.281+0000 7f01b6d9d700 1 --2- 172.21.15.98:0/619850184 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01b80b51b0 0x7f01b80cca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:04.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.281+0000 7f01b759e700 1 --2- 172.21.15.98:0/619850184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b80b8910 0x7f01b80c8eb0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f01b0012000 tx=0x7f01b0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:04.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.281+0000 7f01af7fe700 1 -- 172.21.15.98:0/619850184 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0004880 con 0x7f01b80b8910 2023-03-03T08:02:04.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.281+0000 7f01af7fe700 1 -- 172.21.15.98:0/619850184 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b00099b0 con 0x7f01b80b8910 2023-03-03T08:02:04.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.281+0000 7f01af7fe700 1 -- 172.21.15.98:0/619850184 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b001c680 con 0x7f01b80b8910 2023-03-03T08:02:04.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.281+0000 7f01bc9fe700 1 -- 172.21.15.98:0/619850184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b81612b0 con 0x7f01b80b8910 2023-03-03T08:02:04.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.282+0000 7f01bc9fe700 1 -- 172.21.15.98:0/619850184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b8160fd0 con 0x7f01b80b8910 2023-03-03T08:02:04.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.283+0000 7f01af7fe700 1 -- 172.21.15.98:0/619850184 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) v1 ==== 43500+0+0 (secure 0 0 0) 0x7f01b0022030 con 0x7f01b80b8910 2023-03-03T08:02:04.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.283+0000 7f01bc9fe700 1 -- 172.21.15.98:0/619850184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b80d0e90 con 0x7f01b80b8910 2023-03-03T08:02:04.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.283+0000 7f01af7fe700 1 --2- 172.21.15.98:0/619850184 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f01980363f0 0x7f01980388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:04.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.283+0000 7f01af7fe700 1 -- 172.21.15.98:0/619850184 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6621+0+0 (secure 0 0 0) 0x7f01b004a7c0 con 0x7f01b80b8910 2023-03-03T08:02:04.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.284+0000 7f01b6d9d700 1 --2- 172.21.15.98:0/619850184 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f01980363f0 0x7f01980388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:04.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.285+0000 7f01b6d9d700 1 --2- 172.21.15.98:0/619850184 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f01980363f0 0x7f01980388a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f01b8154ec0 tx=0x7f01a8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.289+0000 7f01af7fe700 1 -- 172.21.15.98:0/619850184 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f01b0023930 con 0x7f01b80b8910 2023-03-03T08:02:04.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.520+0000 7f01bc9fe700 1 -- 172.21.15.98:0/619850184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f01b815f430 con 0x7f01b80b8910 2023-03-03T08:02:04.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.521+0000 7f01af7fe700 1 -- 172.21.15.98:0/619850184 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) v1 ==== 104+0+11 (secure 0 0 0) 0x7f01b0023360 con 0x7f01b80b8910 2023-03-03T08:02:04.524 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T08:02:04.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.523+0000 7f01bc9fe700 1 -- 172.21.15.98:0/619850184 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f01980363f0 msgr2=0x7f01980388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:04.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.523+0000 7f01bc9fe700 1 --2- 172.21.15.98:0/619850184 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f01980363f0 0x7f01980388a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f01b8154ec0 tx=0x7f01a8008040 comp rx=0 tx=0).stop 2023-03-03T08:02:04.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.523+0000 7f01bc9fe700 1 -- 172.21.15.98:0/619850184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b80b8910 msgr2=0x7f01b80c8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:04.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.523+0000 7f01bc9fe700 1 --2- 172.21.15.98:0/619850184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b80b8910 0x7f01b80c8eb0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f01b0012000 tx=0x7f01b0004ad0 comp rx=0 tx=0).stop 2023-03-03T08:02:04.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.524+0000 7f01bc9fe700 1 -- 172.21.15.98:0/619850184 shutdown_connections 2023-03-03T08:02:04.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.524+0000 7f01bc9fe700 1 --2- 172.21.15.98:0/619850184 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f01980363f0 0x7f01980388a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:04.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.524+0000 7f01bc9fe700 1 --2- 172.21.15.98:0/619850184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b80b8910 0x7f01b80c8eb0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.524+0000 7f01bc9fe700 1 --2- 172.21.15.98:0/619850184 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01b80b51b0 0x7f01b80cca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.524+0000 7f01bc9fe700 1 --2- 172.21.15.98:0/619850184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01b80b5e70 0x7f01b815f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.524+0000 7f01bc9fe700 1 -- 172.21.15.98:0/619850184 >> 172.21.15.98:0/619850184 conn(0x7f01b806b880 msgr2=0x7f01b815c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.524+0000 7f01bc9fe700 1 -- 172.21.15.98:0/619850184 shutdown_connections 2023-03-03T08:02:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.524+0000 7f01bc9fe700 1 -- 172.21.15.98:0/619850184 wait complete. 2023-03-03T08:02:04.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T08:02:04.581 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:04.578+0000 7f8457ad6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-03T08:02:04.711 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:04.709+0000 7f8457ad6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-03T08:02:04.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.807+0000 7febe579d700 1 Processor -- start 2023-03-03T08:02:04.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.807+0000 7febe579d700 1 -- start start 2023-03-03T08:02:04.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.808+0000 7febe579d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7febe00b8910 0x7febe01604a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:04.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.808+0000 7febe579d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febe00b5e70 0x7febe00b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:04.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.808+0000 7febe579d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe015ce00 con 0x7febe00cf570 2023-03-03T08:02:04.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.808+0000 7febe579d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe00d0bf0 con 0x7febe00b8910 2023-03-03T08:02:04.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.808+0000 7febe579d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe00b8ce0 con 0x7febe00b5e70 2023-03-03T08:02:04.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.809+0000 7febdffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7febe00b8910 0x7febe01604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:04.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.809+0000 7febdffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7febe00b8910 0x7febe01604a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47986/0 (socket says 172.21.15.98:47986) 2023-03-03T08:02:04.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.809+0000 7febdffff700 1 -- 172.21.15.98:0/44925354 learned_addr learned my addr 172.21.15.98:0/44925354 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:04.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.809+0000 7febd7fff700 1 --2- 172.21.15.98:0/44925354 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febe00b5e70 0x7febe00b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:04.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.809+0000 7febdffff700 1 -- 172.21.15.98:0/44925354 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febe00b5e70 msgr2=0x7febe00b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:04.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.809+0000 7febdffff700 1 --2- 172.21.15.98:0/44925354 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febe00b5e70 0x7febe00b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:04.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.809+0000 7febdffff700 1 -- 172.21.15.98:0/44925354 >> v1:172.21.15.100:6789/0 conn(0x7febe00cf570 legacy=0x7febe00d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:04.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.809+0000 7febdffff700 1 -- 172.21.15.98:0/44925354 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe0160cf0 con 0x7febe00b8910 2023-03-03T08:02:04.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.810+0000 7febdffff700 1 --2- 172.21.15.98:0/44925354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe00b8910 0x7febe01604a0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7febd00098f0 tx=0x7febd0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9bbdb12d1914f8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:04.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.810+0000 7febdf7fe700 1 -- 172.21.15.98:0/44925354 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0011830 con 0x7febe00b8910 2023-03-03T08:02:04.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.810+0000 7febdf7fe700 1 -- 172.21.15.98:0/44925354 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd0011990 con 0x7febe00b8910 2023-03-03T08:02:04.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.810+0000 7febdf7fe700 1 -- 172.21.15.98:0/44925354 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0011d20 con 0x7febe00b8910 2023-03-03T08:02:04.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.810+0000 7febe579d700 1 -- 172.21.15.98:0/44925354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe00b8910 msgr2=0x7febe01604a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:04.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.810+0000 7febe579d700 1 --2- 172.21.15.98:0/44925354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe00b8910 0x7febe01604a0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7febd00098f0 tx=0x7febd0009c00 comp rx=0 tx=0).stop 2023-03-03T08:02:04.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.811+0000 7febe579d700 1 -- 172.21.15.98:0/44925354 shutdown_connections 2023-03-03T08:02:04.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.811+0000 7febe579d700 1 --2- 172.21.15.98:0/44925354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe00b8910 0x7febe01604a0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:04.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.811+0000 7febe579d700 1 --2- 172.21.15.98:0/44925354 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febe00b5e70 0x7febe00b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:04.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.811+0000 7febe579d700 1 -- 172.21.15.98:0/44925354 >> 172.21.15.98:0/44925354 conn(0x7febe006b880 msgr2=0x7febe00cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:04.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.811+0000 7febe579d700 1 -- 172.21.15.98:0/44925354 shutdown_connections 2023-03-03T08:02:04.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.811+0000 7febe579d700 1 -- 172.21.15.98:0/44925354 wait complete. 2023-03-03T08:02:04.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.813+0000 7febe579d700 1 Processor -- start 2023-03-03T08:02:04.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.813+0000 7febe579d700 1 -- start start 2023-03-03T08:02:04.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.814+0000 7febe579d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febe00b5e70 0x7febe01f18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:04.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.814+0000 7febe579d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe00b8910 0x7febe00b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.814+0000 7febe579d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febe00cf570 0x7febe00b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.814+0000 7febe579d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe00b4680 con 0x7febe00b8910 2023-03-03T08:02:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.814+0000 7febe579d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe00b3ab0 con 0x7febe00cf570 2023-03-03T08:02:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.814+0000 7febe579d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe00b3d50 con 0x7febe00b5e70 2023-03-03T08:02:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.814+0000 7febdffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febe00b5e70 0x7febe01f18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.814+0000 7febd7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe00b8910 0x7febe00b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.814+0000 7febdffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febe00b5e70 0x7febe01f18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:39260/0 (socket says 172.21.15.98:39260) 2023-03-03T08:02:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.815+0000 7febdffff700 1 -- 172.21.15.98:0/107941101 learned_addr learned my addr 172.21.15.98:0/107941101 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.814+0000 7febd7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe00b8910 0x7febe00b2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47994/0 (socket says 172.21.15.98:47994) 2023-03-03T08:02:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.815+0000 7febe4f9c700 1 --2- 172.21.15.98:0/107941101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febe00cf570 0x7febe00b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.815+0000 7febdffff700 1 -- 172.21.15.98:0/107941101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febe00cf570 msgr2=0x7febe00b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.815+0000 7febdffff700 1 --2- 172.21.15.98:0/107941101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febe00cf570 0x7febe00b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.815+0000 7febdffff700 1 -- 172.21.15.98:0/107941101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe00b8910 msgr2=0x7febe00b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.815+0000 7febdffff700 1 --2- 172.21.15.98:0/107941101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe00b8910 0x7febe00b2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:04.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.815+0000 7febdffff700 1 -- 172.21.15.98:0/107941101 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd0009550 con 0x7febe00b5e70 2023-03-03T08:02:04.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.815+0000 7febd7fff700 1 --2- 172.21.15.98:0/107941101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe00b8910 0x7febe00b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:04.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.815+0000 7febdffff700 1 --2- 172.21.15.98:0/107941101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febe00b5e70 0x7febe01f18f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7febd0000c00 tx=0x7febd0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:04.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.816+0000 7febdd7fa700 1 -- 172.21.15.98:0/107941101 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd00036a0 con 0x7febe00b5e70 2023-03-03T08:02:04.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.816+0000 7febe579d700 1 -- 172.21.15.98:0/107941101 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe01612b0 con 0x7febe00b5e70 2023-03-03T08:02:04.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.816+0000 7febdd7fa700 1 -- 172.21.15.98:0/107941101 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd0003800 con 0x7febe00b5e70 2023-03-03T08:02:04.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.816+0000 7febdd7fa700 1 -- 172.21.15.98:0/107941101 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0003bb0 con 0x7febe00b5e70 2023-03-03T08:02:04.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.816+0000 7febe579d700 1 -- 172.21.15.98:0/107941101 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe0160fd0 con 0x7febe00b5e70 2023-03-03T08:02:04.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.817+0000 7febdd7fa700 1 -- 172.21.15.98:0/107941101 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 14) v1 ==== 43500+0+0 (secure 0 0 0) 0x7febd000f050 con 0x7febe00b5e70 2023-03-03T08:02:04.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.818+0000 7febe579d700 1 -- 172.21.15.98:0/107941101 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe00d0eb0 con 0x7febe00b5e70 2023-03-03T08:02:04.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.818+0000 7febdd7fa700 1 --2- 172.21.15.98:0/107941101 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7febc00363f0 0x7febc00388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:04.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.818+0000 7febdd7fa700 1 -- 172.21.15.98:0/107941101 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6621+0+0 (secure 0 0 0) 0x7febd004a720 con 0x7febe00b5e70 2023-03-03T08:02:04.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.819+0000 7febd7fff700 1 --2- 172.21.15.98:0/107941101 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7febc00363f0 0x7febc00388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:04.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.820+0000 7febd7fff700 1 --2- 172.21.15.98:0/107941101 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7febc00363f0 0x7febc00388a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7febe00cd050 tx=0x7febc8009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:04.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:04.824+0000 7febdd7fa700 1 -- 172.21.15.98:0/107941101 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7febd002d3a0 con 0x7febe00b5e70 2023-03-03T08:02:04.969 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:04.967+0000 7f8457ad6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-03T08:02:05.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.023+0000 7febe579d700 1 -- 172.21.15.98:0/107941101 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7febe00b4960 con 0x7febe00b5e70 2023-03-03T08:02:05.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.024+0000 7febdd7fa700 1 -- 172.21.15.98:0/107941101 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v79) v1 ==== 100+0+11 (secure 0 0 0) 0x7febd0015020 con 0x7febe00b5e70 2023-03-03T08:02:05.026 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T08:02:05.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.026+0000 7febe579d700 1 -- 172.21.15.98:0/107941101 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7febc00363f0 msgr2=0x7febc00388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:05.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.026+0000 7febe579d700 1 --2- 172.21.15.98:0/107941101 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7febc00363f0 0x7febc00388a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7febe00cd050 tx=0x7febc8009210 comp rx=0 tx=0).stop 2023-03-03T08:02:05.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.026+0000 7febe579d700 1 -- 172.21.15.98:0/107941101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febe00b5e70 msgr2=0x7febe01f18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:05.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.026+0000 7febe579d700 1 --2- 172.21.15.98:0/107941101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febe00b5e70 0x7febe01f18f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7febd0000c00 tx=0x7febd0004750 comp rx=0 tx=0).stop 2023-03-03T08:02:05.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.027+0000 7febe579d700 1 -- 172.21.15.98:0/107941101 shutdown_connections 2023-03-03T08:02:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.027+0000 7febe579d700 1 --2- 172.21.15.98:0/107941101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febe00cf570 0x7febe00b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.027+0000 7febe579d700 1 --2- 172.21.15.98:0/107941101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febe00b5e70 0x7febe01f18f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.027+0000 7febe579d700 1 --2- 172.21.15.98:0/107941101 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7febc00363f0 0x7febc00388a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.027+0000 7febe579d700 1 --2- 172.21.15.98:0/107941101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe00b8910 0x7febe00b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.027+0000 7febe579d700 1 -- 172.21.15.98:0/107941101 >> 172.21.15.98:0/107941101 conn(0x7febe006b880 msgr2=0x7febe00731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.027+0000 7febe579d700 1 -- 172.21.15.98:0/107941101 shutdown_connections 2023-03-03T08:02:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.027+0000 7febe579d700 1 -- 172.21.15.98:0/107941101 wait complete. 2023-03-03T08:02:05.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:02:05.326 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:05.324+0000 7f8457ad6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-03T08:02:05.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.346+0000 7f91941ff700 1 Processor -- start 2023-03-03T08:02:05.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.346+0000 7f91941ff700 1 -- start start 2023-03-03T08:02:05.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.348+0000 7f91941ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f918c0b87c0 0x7f918c0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:05.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.348+0000 7f91941ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f918c0b5410 0x7f918c0d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:05.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.348+0000 7f91941ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f918c0d0ef0 con 0x7f918c0b8b90 2023-03-03T08:02:05.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.348+0000 7f91941ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f918c0d0600 con 0x7f918c0b5410 2023-03-03T08:02:05.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.348+0000 7f91941ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0cd970 con 0x7f918c0b87c0 2023-03-03T08:02:05.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.349+0000 7f919279c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f918c0b5410 0x7f918c0d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:05.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.349+0000 7f919279c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f918c0b5410 0x7f918c0d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35174/0 (socket says 172.21.15.98:35174) 2023-03-03T08:02:05.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.349+0000 7f919279c700 1 -- 172.21.15.98:0/652561043 learned_addr learned my addr 172.21.15.98:0/652561043 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:05.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.349+0000 7f9191f9b700 1 --2- 172.21.15.98:0/652561043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f918c0b87c0 0x7f918c0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:05.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.349+0000 7f919279c700 1 -- 172.21.15.98:0/652561043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f918c0b87c0 msgr2=0x7f918c0cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:05.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.349+0000 7f919279c700 1 --2- 172.21.15.98:0/652561043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f918c0b87c0 0x7f918c0cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.349+0000 7f919279c700 1 -- 172.21.15.98:0/652561043 >> v1:172.21.15.100:6789/0 conn(0x7f918c0b8b90 legacy=0x7f918c0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:05.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.349+0000 7f919279c700 1 -- 172.21.15.98:0/652561043 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c0cdc50 con 0x7f918c0b5410 2023-03-03T08:02:05.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.350+0000 7f919279c700 1 --2- 172.21.15.98:0/652561043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c0b5410 0x7f918c0d13d0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f9184009e50 tx=0x7f918400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0679985e1b9367 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:05.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.350+0000 7f9190f99700 1 -- 172.21.15.98:0/652561043 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918400e070 con 0x7f918c0b5410 2023-03-03T08:02:05.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.350+0000 7f9190f99700 1 -- 172.21.15.98:0/652561043 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9184003a20 con 0x7f918c0b5410 2023-03-03T08:02:05.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.350+0000 7f9190f99700 1 -- 172.21.15.98:0/652561043 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184003dd0 con 0x7f918c0b5410 2023-03-03T08:02:05.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.351+0000 7f91941ff700 1 -- 172.21.15.98:0/652561043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c0b5410 msgr2=0x7f918c0d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:05.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.351+0000 7f91941ff700 1 --2- 172.21.15.98:0/652561043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c0b5410 0x7f918c0d13d0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f9184009e50 tx=0x7f918400b330 comp rx=0 tx=0).stop 2023-03-03T08:02:05.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.351+0000 7f91941ff700 1 -- 172.21.15.98:0/652561043 shutdown_connections 2023-03-03T08:02:05.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.351+0000 7f91941ff700 1 --2- 172.21.15.98:0/652561043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c0b5410 0x7f918c0d13d0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.351+0000 7f91941ff700 1 --2- 172.21.15.98:0/652561043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f918c0b87c0 0x7f918c0cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.351+0000 7f91941ff700 1 -- 172.21.15.98:0/652561043 >> 172.21.15.98:0/652561043 conn(0x7f918c06aee0 msgr2=0x7f918c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:05.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.351+0000 7f91941ff700 1 -- 172.21.15.98:0/652561043 shutdown_connections 2023-03-03T08:02:05.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.351+0000 7f91941ff700 1 -- 172.21.15.98:0/652561043 wait complete. 2023-03-03T08:02:05.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.354+0000 7f91941ff700 1 Processor -- start 2023-03-03T08:02:05.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.354+0000 7f91941ff700 1 -- start start 2023-03-03T08:02:05.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.355+0000 7f91941ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f918c0b5410 0x7f918c15c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:05.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.355+0000 7f91941ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c0b87c0 0x7f918c1f2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:05.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.355+0000 7f91941ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f918c0b8b90 0x7f918c0aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:05.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.355+0000 7f91941ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c15d090 con 0x7f918c0b87c0 2023-03-03T08:02:05.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.355+0000 7f91941ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c15d2e0 con 0x7f918c0b8b90 2023-03-03T08:02:05.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.355+0000 7f91941ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918c15d5c0 con 0x7f918c0b5410 2023-03-03T08:02:05.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.356+0000 7f919179a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c0b87c0 0x7f918c1f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:05.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.356+0000 7f919179a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c0b87c0 0x7f918c1f2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35188/0 (socket says 172.21.15.98:35188) 2023-03-03T08:02:05.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.356+0000 7f919179a700 1 -- 172.21.15.98:0/1987088832 learned_addr learned my addr 172.21.15.98:0/1987088832 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:05.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.356+0000 7f9191f9b700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f918c0b5410 0x7f918c15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:05.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.356+0000 7f919279c700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f918c0b8b90 0x7f918c0aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:05.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.356+0000 7f919179a700 1 -- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f918c0b5410 msgr2=0x7f918c15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:05.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.356+0000 7f919179a700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f918c0b5410 0x7f918c15c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.356+0000 7f919179a700 1 -- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f918c0b8b90 msgr2=0x7f918c0aa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:05.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.356+0000 7f919179a700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f918c0b8b90 0x7f918c0aa4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.356+0000 7f919179a700 1 -- 172.21.15.98:0/1987088832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9184009b30 con 0x7f918c0b87c0 2023-03-03T08:02:05.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.356+0000 7f9191f9b700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f918c0b5410 0x7f918c15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:05.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.356+0000 7f919279c700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f918c0b8b90 0x7f918c0aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:05.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.357+0000 7f919179a700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c0b87c0 0x7f918c1f2210 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f917c00b4a0 tx=0x7f917c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:05.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.357+0000 7f91827fc700 1 -- 172.21.15.98:0/1987088832 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c011830 con 0x7f918c0b87c0 2023-03-03T08:02:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.357+0000 7f91827fc700 1 -- 172.21.15.98:0/1987088832 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917c011990 con 0x7f918c0b87c0 2023-03-03T08:02:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.357+0000 7f91827fc700 1 -- 172.21.15.98:0/1987088832 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c011d40 con 0x7f918c0b87c0 2023-03-03T08:02:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.357+0000 7f91941ff700 1 -- 172.21.15.98:0/1987088832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918c0cdf30 con 0x7f918c0b87c0 2023-03-03T08:02:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.357+0000 7f91941ff700 1 -- 172.21.15.98:0/1987088832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918c073a80 con 0x7f918c0b87c0 2023-03-03T08:02:05.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.358+0000 7f91827fc700 1 -- 172.21.15.98:0/1987088832 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) v1 ==== 43500+0+0 (secure 0 0 0) 0x7f917c00f050 con 0x7f918c0b87c0 2023-03-03T08:02:05.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.358+0000 7f91941ff700 1 -- 172.21.15.98:0/1987088832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918c0ce0c0 con 0x7f918c0b87c0 2023-03-03T08:02:05.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.359+0000 7f91827fc700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f91700363f0 0x7f91700388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:05.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.359+0000 7f91827fc700 1 -- 172.21.15.98:0/1987088832 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6621+0+0 (secure 0 0 0) 0x7f917c048c70 con 0x7f918c0b87c0 2023-03-03T08:02:05.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.359+0000 7f9191f9b700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f91700363f0 0x7f91700388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:05.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.360+0000 7f9191f9b700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f91700363f0 0x7f91700388a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f91880099f0 tx=0x7f9188008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:05.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.364+0000 7f91827fc700 1 -- 172.21.15.98:0/1987088832 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f917c0202b0 con 0x7f918c0b87c0 2023-03-03T08:02:05.585 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:05.583+0000 7f8457ad6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-03T08:02:05.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.607+0000 7f91941ff700 1 -- 172.21.15.98:0/1987088832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f918c0729b0 con 0x7f918c0b87c0 2023-03-03T08:02:05.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.608+0000 7f91827fc700 1 -- 172.21.15.98:0/1987088832 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f917c015070 con 0x7f918c0b87c0 2023-03-03T08:02:05.610 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:05.610 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6050,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/318891906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":318891906},{"type":"v1","addr":"172.21.15.98:6836","nonce":318891906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1280578660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1280578660},{"type":"v1","addr":"172.21.15.98:6837","nonce":1280578660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6906,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1395511883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1395511883},{"type":"v1","addr":"172.21.15.100:6834","nonce":1395511883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:01:21.379777+0000","modified":"2023-03-03T08:01:24.835314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6909},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6909":{"gid":6909,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/341424177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":341424177},{"type":"v1","addr":"172.21.15.100:6835","nonce":341424177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-03-03T08:02:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.610+0000 7f91941ff700 1 -- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f91700363f0 msgr2=0x7f91700388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.610+0000 7f91941ff700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f91700363f0 0x7f91700388a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f91880099f0 tx=0x7f9188008040 comp rx=0 tx=0).stop 2023-03-03T08:02:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.610+0000 7f91941ff700 1 -- 172.21.15.98:0/1987088832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c0b87c0 msgr2=0x7f918c1f2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.610+0000 7f91941ff700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c0b87c0 0x7f918c1f2210 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f917c00b4a0 tx=0x7f917c00b860 comp rx=0 tx=0).stop 2023-03-03T08:02:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.611+0000 7f91941ff700 1 -- 172.21.15.98:0/1987088832 shutdown_connections 2023-03-03T08:02:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.611+0000 7f91941ff700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f918c0b8b90 0x7f918c0aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.611+0000 7f91941ff700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f918c0b5410 0x7f918c15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.611+0000 7f91941ff700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f91700363f0 0x7f91700388a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.611+0000 7f91941ff700 1 --2- 172.21.15.98:0/1987088832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c0b87c0 0x7f918c1f2210 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.611+0000 7f91941ff700 1 -- 172.21.15.98:0/1987088832 >> 172.21.15.98:0/1987088832 conn(0x7f918c06aee0 msgr2=0x7f918c15f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.611+0000 7f91941ff700 1 -- 172.21.15.98:0/1987088832 shutdown_connections 2023-03-03T08:02:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.611+0000 7f91941ff700 1 -- 172.21.15.98:0/1987088832 wait complete. 2023-03-03T08:02:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2023-03-03T08:02:05.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:02:05.731 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:05.728+0000 7f8457ad6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-03T08:02:05.868 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:05.866+0000 7f8457ad6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-03T08:02:05.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.889+0000 7fb28719d700 1 Processor -- start 2023-03-03T08:02:05.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.890+0000 7fb28719d700 1 -- start start 2023-03-03T08:02:05.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.891+0000 7fb28719d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2800d0010 0x7fb2800d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:05.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.891+0000 7fb28719d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb2800b4a00 0x7fb2800d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:05.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.891+0000 7fb28719d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2800b0c50 con 0x7fb2800d1480 2023-03-03T08:02:05.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.891+0000 7fb28719d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2800d1900 con 0x7fb2800b4a00 2023-03-03T08:02:05.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.891+0000 7fb28719d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2800b0040 con 0x7fb2800d0010 2023-03-03T08:02:05.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.891+0000 7fb28699c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb2800d1480 0x7fb2800b10e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45500/0 (socket says 172.21.15.98:45500) 2023-03-03T08:02:05.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.891+0000 7fb28699c700 1 -- 172.21.15.98:0/495516420 learned_addr learned my addr 172.21.15.98:0/495516420 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:05.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.892+0000 7fb28599a700 1 --2- 172.21.15.98:0/495516420 >> v2:172.21.15.100:3300/0 conn(0x7fb2800b4a00 0x7fb2800d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.892+0000 7fb28619b700 1 --2- 172.21.15.98:0/495516420 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2800d0010 0x7fb2800d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.892+0000 7fb285199700 1 -- 172.21.15.98:0/495516420 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972987680 0 0) 0x7fb2800b0c50 con 0x7fb2800d1480 2023-03-03T08:02:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.892+0000 7fb285199700 1 -- 172.21.15.98:0/495516420 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb26c0035b0 con 0x7fb2800d1480 2023-03-03T08:02:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.892+0000 7fb28619b700 1 -- 172.21.15.98:0/495516420 >> v2:172.21.15.100:3300/0 conn(0x7fb2800b4a00 msgr2=0x7fb2800d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.892+0000 7fb28619b700 1 --2- 172.21.15.98:0/495516420 >> v2:172.21.15.100:3300/0 conn(0x7fb2800b4a00 0x7fb2800d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.892+0000 7fb28619b700 1 -- 172.21.15.98:0/495516420 >> v1:172.21.15.98:6789/0 conn(0x7fb2800d1480 legacy=0x7fb2800b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:05.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.892+0000 7fb28619b700 1 -- 172.21.15.98:0/495516420 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2800b0320 con 0x7fb2800d0010 2023-03-03T08:02:05.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.892+0000 7fb28599a700 1 --2- 172.21.15.98:0/495516420 >> v2:172.21.15.100:3300/0 conn(0x7fb2800b4a00 0x7fb2800d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:05.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.893+0000 7fb28619b700 1 --2- 172.21.15.98:0/495516420 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2800d0010 0x7fb2800d03e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb27c009820 tx=0x7fb27c009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33f913707af2e8a1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:05.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.893+0000 7fb285199700 1 -- 172.21.15.98:0/495516420 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c005e30 con 0x7fb2800d0010 2023-03-03T08:02:05.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.893+0000 7fb285199700 1 -- 172.21.15.98:0/495516420 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27c006030 con 0x7fb2800d0010 2023-03-03T08:02:05.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.893+0000 7fb285199700 1 -- 172.21.15.98:0/495516420 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c0063c0 con 0x7fb2800d0010 2023-03-03T08:02:05.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.893+0000 7fb28719d700 1 -- 172.21.15.98:0/495516420 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2800d0010 msgr2=0x7fb2800d03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:05.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.893+0000 7fb28719d700 1 --2- 172.21.15.98:0/495516420 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2800d0010 0x7fb2800d03e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb27c009820 tx=0x7fb27c009b30 comp rx=0 tx=0).stop 2023-03-03T08:02:05.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.894+0000 7fb28719d700 1 -- 172.21.15.98:0/495516420 shutdown_connections 2023-03-03T08:02:05.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.894+0000 7fb28719d700 1 --2- 172.21.15.98:0/495516420 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2800d0010 0x7fb2800d03e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.894+0000 7fb28719d700 1 --2- 172.21.15.98:0/495516420 >> v2:172.21.15.100:3300/0 conn(0x7fb2800b4a00 0x7fb2800d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.894+0000 7fb28719d700 1 -- 172.21.15.98:0/495516420 >> 172.21.15.98:0/495516420 conn(0x7fb280073b00 msgr2=0x7fb2800d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:05.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.894+0000 7fb28719d700 1 -- 172.21.15.98:0/495516420 shutdown_connections 2023-03-03T08:02:05.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.894+0000 7fb28719d700 1 -- 172.21.15.98:0/495516420 wait complete. 2023-03-03T08:02:05.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.897+0000 7fb28719d700 1 Processor -- start 2023-03-03T08:02:05.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.897+0000 7fb28719d700 1 -- start start 2023-03-03T08:02:05.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.898+0000 7fb28719d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2800b4a00 0x7fb2800c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:05.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.898+0000 7fb28719d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2800d0010 0x7fb28015a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:05.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.898+0000 7fb28719d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2800d1480 0x7fb28015a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:05.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.898+0000 7fb28719d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb280158220 con 0x7fb2800b4a00 2023-03-03T08:02:05.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.898+0000 7fb28719d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb280158500 con 0x7fb2800d0010 2023-03-03T08:02:05.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.898+0000 7fb28719d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2801587e0 con 0x7fb2800d1480 2023-03-03T08:02:05.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.899+0000 7fb28619b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2800b4a00 0x7fb2800c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:05.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.899+0000 7fb28619b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2800b4a00 0x7fb2800c79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35196/0 (socket says 172.21.15.98:35196) 2023-03-03T08:02:05.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.899+0000 7fb28619b700 1 -- 172.21.15.98:0/3687287052 learned_addr learned my addr 172.21.15.98:0/3687287052 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:05.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.899+0000 7fb28599a700 1 --2- 172.21.15.98:0/3687287052 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2800d0010 0x7fb28015a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:05.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.899+0000 7fb28699c700 1 --2- 172.21.15.98:0/3687287052 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2800d1480 0x7fb28015a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:05.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.899+0000 7fb28619b700 1 -- 172.21.15.98:0/3687287052 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2800d1480 msgr2=0x7fb28015a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:05.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.899+0000 7fb28619b700 1 --2- 172.21.15.98:0/3687287052 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2800d1480 0x7fb28015a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.899+0000 7fb28619b700 1 -- 172.21.15.98:0/3687287052 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2800d0010 msgr2=0x7fb28015a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:05.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.899+0000 7fb28619b700 1 --2- 172.21.15.98:0/3687287052 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2800d0010 0x7fb28015a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:05.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.899+0000 7fb28619b700 1 -- 172.21.15.98:0/3687287052 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2700094a0 con 0x7fb2800b4a00 2023-03-03T08:02:05.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.900+0000 7fb28619b700 1 --2- 172.21.15.98:0/3687287052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2800b4a00 0x7fb2800c79f0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fb27c000c00 tx=0x7fb27c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:05.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.900+0000 7fb276ffd700 1 -- 172.21.15.98:0/3687287052 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c0061a0 con 0x7fb2800b4a00 2023-03-03T08:02:05.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.900+0000 7fb276ffd700 1 -- 172.21.15.98:0/3687287052 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27c020070 con 0x7fb2800b4a00 2023-03-03T08:02:05.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.900+0000 7fb28719d700 1 -- 172.21.15.98:0/3687287052 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27c009430 con 0x7fb2800b4a00 2023-03-03T08:02:05.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.900+0000 7fb276ffd700 1 -- 172.21.15.98:0/3687287052 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c006560 con 0x7fb2800b4a00 2023-03-03T08:02:05.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.900+0000 7fb28719d700 1 -- 172.21.15.98:0/3687287052 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2800b08e0 con 0x7fb2800b4a00 2023-03-03T08:02:05.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.901+0000 7fb276ffd700 1 -- 172.21.15.98:0/3687287052 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) v1 ==== 43500+0+0 (secure 0 0 0) 0x7fb27c02b020 con 0x7fb2800b4a00 2023-03-03T08:02:05.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.901+0000 7fb28719d700 1 -- 172.21.15.98:0/3687287052 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2800b0600 con 0x7fb2800b4a00 2023-03-03T08:02:05.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.902+0000 7fb276ffd700 1 --2- 172.21.15.98:0/3687287052 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb2640363f0 0x7fb2640388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:05.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.902+0000 7fb276ffd700 1 -- 172.21.15.98:0/3687287052 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6621+0+0 (secure 0 0 0) 0x7fb27c04a5c0 con 0x7fb2800b4a00 2023-03-03T08:02:05.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.902+0000 7fb28599a700 1 --2- 172.21.15.98:0/3687287052 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb2640363f0 0x7fb2640388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:05.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.903+0000 7fb28599a700 1 --2- 172.21.15.98:0/3687287052 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb2640363f0 0x7fb2640388a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb2800b4280 tx=0x7fb270009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:05.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:05.907+0000 7fb276ffd700 1 -- 172.21.15.98:0/3687287052 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb27c019590 con 0x7fb2800b4a00 2023-03-03T08:02:06.014 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:06.011+0000 7f8457ad6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-03T08:02:06.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.110+0000 7fb28719d700 1 -- 172.21.15.98:0/3687287052 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2800729b0 con 0x7fb2800b4a00 2023-03-03T08:02:06.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.113+0000 7fb276ffd700 1 -- 172.21.15.98:0/3687287052 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18784 (secure 0 0 0) 0x7fb27c01b410 con 0x7fb2800b4a00 2023-03-03T08:02:06.115 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:06.116 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":79,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:02:00.117924+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:01:19.439394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-03-03T08:01:20.381345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/4204485361":"2023-03-04T08:02:00.117879+0000","172.21.15.98:6833/108777":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/3513799141":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/2444435923":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/2644355772":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/1900692259":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/3107079680":"2023-03-04T08:02:00.117879+0000","172.21.15.98:6832/108777":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/3559163180":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/918927922":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/1647401399":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/3630329692":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/2323983002":"2023-03-04T08:02:00.117879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:02:06.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.115+0000 7fb28719d700 1 -- 172.21.15.98:0/3687287052 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb2640363f0 msgr2=0x7fb2640388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:06.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.115+0000 7fb28719d700 1 --2- 172.21.15.98:0/3687287052 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb2640363f0 0x7fb2640388a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb2800b4280 tx=0x7fb270009210 comp rx=0 tx=0).stop 2023-03-03T08:02:06.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.116+0000 7fb28719d700 1 -- 172.21.15.98:0/3687287052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2800b4a00 msgr2=0x7fb2800c79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:06.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.116+0000 7fb28719d700 1 --2- 172.21.15.98:0/3687287052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2800b4a00 0x7fb2800c79f0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fb27c000c00 tx=0x7fb27c004750 comp rx=0 tx=0).stop 2023-03-03T08:02:06.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.116+0000 7fb28719d700 1 -- 172.21.15.98:0/3687287052 shutdown_connections 2023-03-03T08:02:06.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.116+0000 7fb28719d700 1 --2- 172.21.15.98:0/3687287052 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb2640363f0 0x7fb2640388a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.116+0000 7fb28719d700 1 --2- 172.21.15.98:0/3687287052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2800b4a00 0x7fb2800c79f0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.116+0000 7fb28719d700 1 --2- 172.21.15.98:0/3687287052 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2800d1480 0x7fb28015a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.116+0000 7fb28719d700 1 --2- 172.21.15.98:0/3687287052 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2800d0010 0x7fb28015a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.116+0000 7fb28719d700 1 -- 172.21.15.98:0/3687287052 >> 172.21.15.98:0/3687287052 conn(0x7fb280073b00 msgr2=0x7fb28006c730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:06.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.116+0000 7fb28719d700 1 -- 172.21.15.98:0/3687287052 shutdown_connections 2023-03-03T08:02:06.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.116+0000 7fb28719d700 1 -- 172.21.15.98:0/3687287052 wait complete. 2023-03-03T08:02:06.156 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-03-03T08:02:06.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:02:06.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.410+0000 7f64c359d700 1 Processor -- start 2023-03-03T08:02:06.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.411+0000 7f64c359d700 1 -- start start 2023-03-03T08:02:06.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.412+0000 7f64c359d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f64bc0d0010 0x7f64bc0d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:06.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.412+0000 7f64c359d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64bc0b4a00 0x7f64bc0d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:06.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.412+0000 7f64c359d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f64bc0d1480 0x7f64bc0b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:06.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.412+0000 7f64c359d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64bc0d1900 con 0x7f64bc0d0010 2023-03-03T08:02:06.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.412+0000 7f64c359d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64bc0b0c50 con 0x7f64bc0d1480 2023-03-03T08:02:06.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.412+0000 7f64c359d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc0b0040 con 0x7f64bc0b4a00 2023-03-03T08:02:06.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.412+0000 7f64c259b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f64bc0d0010 0x7f64bc0d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:06.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.412+0000 7f64c259b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f64bc0d0010 0x7f64bc0d03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35202/0 (socket says 172.21.15.98:35202) 2023-03-03T08:02:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.412+0000 7f64c259b700 1 -- 172.21.15.98:0/3593177216 learned_addr learned my addr 172.21.15.98:0/3593177216 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.413+0000 7f64c2d9c700 1 --2- 172.21.15.98:0/3593177216 >> v2:172.21.15.100:3300/0 conn(0x7f64bc0d1480 0x7f64bc0b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.413+0000 7f64c1d9a700 1 --2- 172.21.15.98:0/3593177216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64bc0b4a00 0x7f64bc0d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.413+0000 7f64c259b700 1 -- 172.21.15.98:0/3593177216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64bc0b4a00 msgr2=0x7f64bc0d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.413+0000 7f64c259b700 1 --2- 172.21.15.98:0/3593177216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64bc0b4a00 0x7f64bc0d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.413+0000 7f64c259b700 1 -- 172.21.15.98:0/3593177216 >> v2:172.21.15.100:3300/0 conn(0x7f64bc0d1480 msgr2=0x7f64bc0b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.413+0000 7f64c259b700 1 --2- 172.21.15.98:0/3593177216 >> v2:172.21.15.100:3300/0 conn(0x7f64bc0d1480 0x7f64bc0b10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.413+0000 7f64c259b700 1 -- 172.21.15.98:0/3593177216 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64bc0b0320 con 0x7f64bc0d0010 2023-03-03T08:02:06.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.413+0000 7f64c2d9c700 1 --2- 172.21.15.98:0/3593177216 >> v2:172.21.15.100:3300/0 conn(0x7f64bc0d1480 0x7f64bc0b10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:06.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.413+0000 7f64c259b700 1 --2- 172.21.15.98:0/3593177216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64bc0d0010 0x7f64bc0d03e0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f64bc08b860 tx=0x7f64b8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07750cdf0da3c48 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:06.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.414+0000 7f64c1599700 1 -- 172.21.15.98:0/3593177216 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b8005e30 con 0x7f64bc0d0010 2023-03-03T08:02:06.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.414+0000 7f64c1599700 1 -- 172.21.15.98:0/3593177216 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b8004030 con 0x7f64bc0d0010 2023-03-03T08:02:06.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.414+0000 7f64c1599700 1 -- 172.21.15.98:0/3593177216 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b80043e0 con 0x7f64bc0d0010 2023-03-03T08:02:06.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.414+0000 7f64c359d700 1 -- 172.21.15.98:0/3593177216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64bc0d0010 msgr2=0x7f64bc0d03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:06.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.414+0000 7f64c359d700 1 --2- 172.21.15.98:0/3593177216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64bc0d0010 0x7f64bc0d03e0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f64bc08b860 tx=0x7f64b8009bd0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.414+0000 7f64c359d700 1 -- 172.21.15.98:0/3593177216 shutdown_connections 2023-03-03T08:02:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.414+0000 7f64c359d700 1 --2- 172.21.15.98:0/3593177216 >> v2:172.21.15.100:3300/0 conn(0x7f64bc0d1480 0x7f64bc0b10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.414+0000 7f64c359d700 1 --2- 172.21.15.98:0/3593177216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64bc0d0010 0x7f64bc0d03e0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.414+0000 7f64c359d700 1 --2- 172.21.15.98:0/3593177216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64bc0b4a00 0x7f64bc0d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.414+0000 7f64c359d700 1 -- 172.21.15.98:0/3593177216 >> 172.21.15.98:0/3593177216 conn(0x7f64bc073b00 msgr2=0x7f64bc0d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.414+0000 7f64c359d700 1 -- 172.21.15.98:0/3593177216 shutdown_connections 2023-03-03T08:02:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.414+0000 7f64c359d700 1 -- 172.21.15.98:0/3593177216 wait complete. 2023-03-03T08:02:06.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.418+0000 7f64c359d700 1 Processor -- start 2023-03-03T08:02:06.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.418+0000 7f64c359d700 1 -- start start 2023-03-03T08:02:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.420+0000 7f64c359d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f64bc0b4a00 0x7f64bc0c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.420+0000 7f64c359d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64bc0d0010 0x7f64bc1e4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.420+0000 7f64c359d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64bc0d1480 0x7f64bc1e5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.420+0000 7f64c359d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc1e56a0 con 0x7f64bc0d0010 2023-03-03T08:02:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.420+0000 7f64c359d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc0c9e40 con 0x7f64bc0b4a00 2023-03-03T08:02:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.420+0000 7f64c359d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc1e4b20 con 0x7f64bc0d1480 2023-03-03T08:02:06.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.420+0000 7f64c1d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64bc0d0010 0x7f64bc1e4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:06.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.420+0000 7f64c259b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f64bc0b4a00 0x7f64bc0c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:06.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.420+0000 7f64c1d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64bc0d0010 0x7f64bc1e4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35218/0 (socket says 172.21.15.98:35218) 2023-03-03T08:02:06.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.420+0000 7f64c1d9a700 1 -- 172.21.15.98:0/3624681726 learned_addr learned my addr 172.21.15.98:0/3624681726 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:06.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.420+0000 7f64c2d9c700 1 --2- 172.21.15.98:0/3624681726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64bc0d1480 0x7f64bc1e5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:06.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.421+0000 7f64c1d9a700 1 -- 172.21.15.98:0/3624681726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64bc0d1480 msgr2=0x7f64bc1e5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:06.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.421+0000 7f64c1d9a700 1 --2- 172.21.15.98:0/3624681726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64bc0d1480 0x7f64bc1e5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.421+0000 7f64c1d9a700 1 -- 172.21.15.98:0/3624681726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f64bc0b4a00 msgr2=0x7f64bc0c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:06.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.421+0000 7f64c1d9a700 1 --2- 172.21.15.98:0/3624681726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f64bc0b4a00 0x7f64bc0c9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.421+0000 7f64c1d9a700 1 -- 172.21.15.98:0/3624681726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b8009500 con 0x7f64bc0d0010 2023-03-03T08:02:06.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.421+0000 7f64c1d9a700 1 --2- 172.21.15.98:0/3624681726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64bc0d0010 0x7f64bc1e4290 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00e900 tx=0x7f64ac00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:06.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.421+0000 7f64b2ffd700 1 -- 172.21.15.98:0/3624681726 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac013680 con 0x7f64bc0d0010 2023-03-03T08:02:06.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.421+0000 7f64b2ffd700 1 -- 172.21.15.98:0/3624681726 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ac0137e0 con 0x7f64bc0d0010 2023-03-03T08:02:06.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.421+0000 7f64c359d700 1 -- 172.21.15.98:0/3624681726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64bc0b08e0 con 0x7f64bc0d0010 2023-03-03T08:02:06.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.421+0000 7f64b2ffd700 1 -- 172.21.15.98:0/3624681726 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac013b70 con 0x7f64bc0d0010 2023-03-03T08:02:06.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.422+0000 7f64c359d700 1 -- 172.21.15.98:0/3624681726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64bc0b0600 con 0x7f64bc0d0010 2023-03-03T08:02:06.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.423+0000 7f64b2ffd700 1 -- 172.21.15.98:0/3624681726 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) v1 ==== 43500+0+0 (secure 0 0 0) 0x7f64ac019030 con 0x7f64bc0d0010 2023-03-03T08:02:06.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.423+0000 7f64c359d700 1 -- 172.21.15.98:0/3624681726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64bc0d0b30 con 0x7f64bc0d0010 2023-03-03T08:02:06.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.424+0000 7f64b2ffd700 1 --2- 172.21.15.98:0/3624681726 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f64a00363f0 0x7f64a00388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:06.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.424+0000 7f64b2ffd700 1 -- 172.21.15.98:0/3624681726 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6621+0+0 (secure 0 0 0) 0x7f64ac048e20 con 0x7f64bc0d0010 2023-03-03T08:02:06.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.424+0000 7f64c259b700 1 --2- 172.21.15.98:0/3624681726 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f64a00363f0 0x7f64a00388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:06.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.425+0000 7f64c259b700 1 --2- 172.21.15.98:0/3624681726 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f64a00363f0 0x7f64a00388a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f64b8000c00 tx=0x7f64b8008070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:06.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.429+0000 7f64b2ffd700 1 -- 172.21.15.98:0/3624681726 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f64ac01b9a0 con 0x7f64bc0d0010 2023-03-03T08:02:06.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.693+0000 7f64c359d700 1 -- 172.21.15.98:0/3624681726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64bc0729b0 con 0x7f64bc0d0010 2023-03-03T08:02:06.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.694+0000 7f64b2ffd700 1 -- 172.21.15.98:0/3624681726 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f64ac017020 con 0x7f64bc0d0010 2023-03-03T08:02:06.696 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:06.697 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6050,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/318891906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":318891906},{"type":"v1","addr":"172.21.15.98:6836","nonce":318891906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1280578660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1280578660},{"type":"v1","addr":"172.21.15.98:6837","nonce":1280578660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6906,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1395511883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1395511883},{"type":"v1","addr":"172.21.15.100:6834","nonce":1395511883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:01:21.379777+0000","modified":"2023-03-03T08:01:24.835314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6909},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6909":{"gid":6909,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/341424177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":341424177},{"type":"v1","addr":"172.21.15.100:6835","nonce":341424177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-03-03T08:02:06.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.697+0000 7f64c359d700 1 -- 172.21.15.98:0/3624681726 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f64a00363f0 msgr2=0x7f64a00388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:06.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.697+0000 7f64c359d700 1 --2- 172.21.15.98:0/3624681726 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f64a00363f0 0x7f64a00388a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f64b8000c00 tx=0x7f64b8008070 comp rx=0 tx=0).stop 2023-03-03T08:02:06.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.697+0000 7f64c359d700 1 -- 172.21.15.98:0/3624681726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64bc0d0010 msgr2=0x7f64bc1e4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:06.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.697+0000 7f64c359d700 1 --2- 172.21.15.98:0/3624681726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64bc0d0010 0x7f64bc1e4290 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00e900 tx=0x7f64ac00ec10 comp rx=0 tx=0).stop 2023-03-03T08:02:06.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.697+0000 7f64c359d700 1 -- 172.21.15.98:0/3624681726 shutdown_connections 2023-03-03T08:02:06.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.697+0000 7f64c359d700 1 --2- 172.21.15.98:0/3624681726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f64bc0d1480 0x7f64bc1e5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.697+0000 7f64c359d700 1 --2- 172.21.15.98:0/3624681726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f64bc0b4a00 0x7f64bc0c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.697+0000 7f64c359d700 1 --2- 172.21.15.98:0/3624681726 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f64a00363f0 0x7f64a00388a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.697+0000 7f64c359d700 1 --2- 172.21.15.98:0/3624681726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64bc0d0010 0x7f64bc1e4290 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.697+0000 7f64c359d700 1 -- 172.21.15.98:0/3624681726 >> 172.21.15.98:0/3624681726 conn(0x7f64bc073b00 msgr2=0x7f64bc06bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:06.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.697+0000 7f64c359d700 1 -- 172.21.15.98:0/3624681726 shutdown_connections 2023-03-03T08:02:06.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.697+0000 7f64c359d700 1 -- 172.21.15.98:0/3624681726 wait complete. 2023-03-03T08:02:06.701 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2023-03-03T08:02:06.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:02:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.981+0000 7f93ab19d700 1 Processor -- start 2023-03-03T08:02:06.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.982+0000 7f93ab19d700 1 -- start start 2023-03-03T08:02:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.983+0000 7f93ab19d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93a4146360 0x7f93a4160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.983+0000 7f93ab19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93a40b4ab0 0x7f93a40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.984+0000 7f93ab19d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a41614c0 con 0x7f93a40731e0 2023-03-03T08:02:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.984+0000 7f93ab19d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a40b5440 con 0x7f93a4146360 2023-03-03T08:02:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.984+0000 7f93ab19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a40d10f0 con 0x7f93a40b4ab0 2023-03-03T08:02:06.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.984+0000 7f93aa19b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93a4146360 0x7f93a4160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:06.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.984+0000 7f93aa19b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93a4146360 0x7f93a4160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35220/0 (socket says 172.21.15.98:35220) 2023-03-03T08:02:06.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.984+0000 7f93aa19b700 1 -- 172.21.15.98:0/57497807 learned_addr learned my addr 172.21.15.98:0/57497807 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:06.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.984+0000 7f93a999a700 1 --2- 172.21.15.98:0/57497807 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93a40b4ab0 0x7f93a40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:06.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.985+0000 7f93aa19b700 1 -- 172.21.15.98:0/57497807 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93a40b4ab0 msgr2=0x7f93a40b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:06.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.985+0000 7f93aa19b700 1 --2- 172.21.15.98:0/57497807 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93a40b4ab0 0x7f93a40b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.985+0000 7f93aa19b700 1 -- 172.21.15.98:0/57497807 >> v1:172.21.15.100:6789/0 conn(0x7f93a40731e0 legacy=0x7f93a4073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:06.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.985+0000 7f93aa19b700 1 -- 172.21.15.98:0/57497807 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a40cdf20 con 0x7f93a4146360 2023-03-03T08:02:06.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.985+0000 7f93a9199700 1 -- 172.21.15.98:0/57497807 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162155320 0 0) 0x7f939c001490 con 0x7f93a40731e0 2023-03-03T08:02:06.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.985+0000 7f93aa19b700 1 --2- 172.21.15.98:0/57497807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4146360 0x7f93a4160f80 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f93a00098f0 tx=0x7f93a0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381626250becb286 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:06.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.985+0000 7f93a9199700 1 -- 172.21.15.98:0/57497807 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a0011830 con 0x7f93a4146360 2023-03-03T08:02:06.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.985+0000 7f93a9199700 1 -- 172.21.15.98:0/57497807 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93a0011990 con 0x7f93a4146360 2023-03-03T08:02:06.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.985+0000 7f93a9199700 1 -- 172.21.15.98:0/57497807 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a0011d20 con 0x7f93a4146360 2023-03-03T08:02:06.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.986+0000 7f93ab19d700 1 -- 172.21.15.98:0/57497807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4146360 msgr2=0x7f93a4160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:06.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.986+0000 7f93ab19d700 1 --2- 172.21.15.98:0/57497807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4146360 0x7f93a4160f80 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f93a00098f0 tx=0x7f93a0009c00 comp rx=0 tx=0).stop 2023-03-03T08:02:06.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.986+0000 7f93ab19d700 1 -- 172.21.15.98:0/57497807 shutdown_connections 2023-03-03T08:02:06.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.986+0000 7f93ab19d700 1 --2- 172.21.15.98:0/57497807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4146360 0x7f93a4160f80 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.986+0000 7f93ab19d700 1 --2- 172.21.15.98:0/57497807 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93a40b4ab0 0x7f93a40b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.986+0000 7f93ab19d700 1 -- 172.21.15.98:0/57497807 >> 172.21.15.98:0/57497807 conn(0x7f93a406b8e0 msgr2=0x7f93a40cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:06.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.986+0000 7f93ab19d700 1 -- 172.21.15.98:0/57497807 shutdown_connections 2023-03-03T08:02:06.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.986+0000 7f93ab19d700 1 -- 172.21.15.98:0/57497807 wait complete. 2023-03-03T08:02:06.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.988+0000 7f93ab19d700 1 Processor -- start 2023-03-03T08:02:06.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.988+0000 7f93ab19d700 1 -- start start 2023-03-03T08:02:06.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.989+0000 7f93ab19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93a40731e0 0x7f93a40c8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:06.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.989+0000 7f93ab19d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a40b4ab0 0x7f93a41f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:06.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.989+0000 7f93ab19d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93a4146360 0x7f93a4159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:06.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.990+0000 7f93ab19d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a40bc1c0 con 0x7f93a40b4ab0 2023-03-03T08:02:06.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.990+0000 7f93ab19d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a41f5c10 con 0x7f93a4146360 2023-03-03T08:02:06.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.990+0000 7f93ab19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a40c8720 con 0x7f93a40731e0 2023-03-03T08:02:06.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.990+0000 7f93a999a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a40b4ab0 0x7f93a41f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:06.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.990+0000 7f93a999a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a40b4ab0 0x7f93a41f56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35222/0 (socket says 172.21.15.98:35222) 2023-03-03T08:02:06.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.990+0000 7f93aa99c700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93a4146360 0x7f93a4159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:06.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.990+0000 7f93a999a700 1 -- 172.21.15.98:0/2637405244 learned_addr learned my addr 172.21.15.98:0/2637405244 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:06.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.990+0000 7f93aa19b700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93a40731e0 0x7f93a40c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:06.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.991+0000 7f93aa99c700 1 -- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93a40731e0 msgr2=0x7f93a40c8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:06.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.991+0000 7f93aa99c700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93a40731e0 0x7f93a40c8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.991+0000 7f93aa99c700 1 -- 172.21.15.98:0/2637405244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a40b4ab0 msgr2=0x7f93a41f56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:06.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.991+0000 7f93aa99c700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a40b4ab0 0x7f93a41f56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:06.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.991+0000 7f93aa99c700 1 -- 172.21.15.98:0/2637405244 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a0009550 con 0x7f93a4146360 2023-03-03T08:02:06.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.991+0000 7f93aa19b700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93a40731e0 0x7f93a40c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:06.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.991+0000 7f93a999a700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a40b4ab0 0x7f93a41f56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:06.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.991+0000 7f93aa99c700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93a4146360 0x7f93a4159e90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f939c00c780 tx=0x7f939c00ca90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:06.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.991+0000 7f939affd700 1 -- 172.21.15.98:0/2637405244 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c003e30 con 0x7f93a4146360 2023-03-03T08:02:06.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.991+0000 7f939affd700 1 -- 172.21.15.98:0/2637405244 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939c0050d0 con 0x7f93a4146360 2023-03-03T08:02:06.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.991+0000 7f939affd700 1 -- 172.21.15.98:0/2637405244 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c005480 con 0x7f93a4146360 2023-03-03T08:02:06.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.991+0000 7f93ab19d700 1 -- 172.21.15.98:0/2637405244 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a40cd960 con 0x7f93a4146360 2023-03-03T08:02:06.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.992+0000 7f93ab19d700 1 -- 172.21.15.98:0/2637405244 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a40cd680 con 0x7f93a4146360 2023-03-03T08:02:06.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.993+0000 7f93ab19d700 1 -- 172.21.15.98:0/2637405244 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a40cdc40 con 0x7f93a4146360 2023-03-03T08:02:06.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.993+0000 7f939affd700 1 -- 172.21.15.98:0/2637405244 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 14) v1 ==== 43500+0+0 (secure 0 0 0) 0x7f939c019030 con 0x7f93a4146360 2023-03-03T08:02:06.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.994+0000 7f939affd700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f93880363f0 0x7f93880388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:06.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.994+0000 7f939affd700 1 -- 172.21.15.98:0/2637405244 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6621+0+0 (secure 0 0 0) 0x7f939c048330 con 0x7f93a4146360 2023-03-03T08:02:06.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.994+0000 7f93aa19b700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f93880363f0 0x7f93880388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:06.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.995+0000 7f93aa19b700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f93880363f0 0x7f93880388a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f93a0014040 tx=0x7f93a0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:07.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:06.999+0000 7f939affd700 1 -- 172.21.15.98:0/2637405244 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f939c019680 con 0x7f93a4146360 2023-03-03T08:02:07.135 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:07.133+0000 7f8457ad6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-03T08:02:07.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.243+0000 7f93ab19d700 1 -- 172.21.15.98:0/2637405244 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93a41614c0 con 0x7f93a4146360 2023-03-03T08:02:07.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.244+0000 7f939affd700 1 -- 172.21.15.98:0/2637405244 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f93a41614c0 con 0x7f93a4146360 2023-03-03T08:02:07.246 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:07.246 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6050,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/318891906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":318891906},{"type":"v1","addr":"172.21.15.98:6836","nonce":318891906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1280578660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1280578660},{"type":"v1","addr":"172.21.15.98:6837","nonce":1280578660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6906,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1395511883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1395511883},{"type":"v1","addr":"172.21.15.100:6834","nonce":1395511883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:01:21.379777+0000","modified":"2023-03-03T08:01:24.835314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6909},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6909":{"gid":6909,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/341424177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":341424177},{"type":"v1","addr":"172.21.15.100:6835","nonce":341424177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-03-03T08:02:07.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.247+0000 7f93ab19d700 1 -- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f93880363f0 msgr2=0x7f93880388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:07.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.247+0000 7f93ab19d700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f93880363f0 0x7f93880388a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f93a0014040 tx=0x7f93a0008030 comp rx=0 tx=0).stop 2023-03-03T08:02:07.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.247+0000 7f93ab19d700 1 -- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93a4146360 msgr2=0x7f93a4159e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:07.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.247+0000 7f93ab19d700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93a4146360 0x7f93a4159e90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f939c00c780 tx=0x7f939c00ca90 comp rx=0 tx=0).stop 2023-03-03T08:02:07.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.247+0000 7f93ab19d700 1 -- 172.21.15.98:0/2637405244 shutdown_connections 2023-03-03T08:02:07.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.247+0000 7f93ab19d700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93a4146360 0x7f93a4159e90 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:07.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.247+0000 7f93ab19d700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93a40731e0 0x7f93a40c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:07.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.247+0000 7f93ab19d700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f93880363f0 0x7f93880388a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:07.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.247+0000 7f93ab19d700 1 --2- 172.21.15.98:0/2637405244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a40b4ab0 0x7f93a41f56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:07.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.247+0000 7f93ab19d700 1 -- 172.21.15.98:0/2637405244 >> 172.21.15.98:0/2637405244 conn(0x7f93a406b8e0 msgr2=0x7f93a40b2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:07.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.247+0000 7f93ab19d700 1 -- 172.21.15.98:0/2637405244 shutdown_connections 2023-03-03T08:02:07.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.247+0000 7f93ab19d700 1 -- 172.21.15.98:0/2637405244 wait complete. 2023-03-03T08:02:07.251 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2023-03-03T08:02:07.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:02:07.283 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:07.281+0000 7f8457ad6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-03T08:02:07.435 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:07.433+0000 7f8457ad6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-03T08:02:07.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.524+0000 7f34ec9fe700 1 Processor -- start 2023-03-03T08:02:07.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.525+0000 7f34ec9fe700 1 -- start start 2023-03-03T08:02:07.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.526+0000 7f34ec9fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f34e80d0010 0x7f34e80d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:07.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.526+0000 7f34ec9fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34e80b4a00 0x7f34e80d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:07.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.526+0000 7f34ec9fe700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e80b0c50 con 0x7f34e80d1480 2023-03-03T08:02:07.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.526+0000 7f34ec9fe700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e80d1900 con 0x7f34e80d0010 2023-03-03T08:02:07.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.526+0000 7f34ec9fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e80b0040 con 0x7f34e80b4a00 2023-03-03T08:02:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.526+0000 7f34e6d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f34e80d0010 0x7f34e80d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.526+0000 7f34e6d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f34e80d0010 0x7f34e80d03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35224/0 (socket says 172.21.15.98:35224) 2023-03-03T08:02:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.526+0000 7f34e6d9d700 1 -- 172.21.15.98:0/1078280009 learned_addr learned my addr 172.21.15.98:0/1078280009 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.526+0000 7f34e759e700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f34e80d1480 0x7f34e80b10e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45510/0 (socket says 172.21.15.98:45510) 2023-03-03T08:02:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.527+0000 7f34e659c700 1 --2- 172.21.15.98:0/1078280009 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34e80b4a00 0x7f34e80d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.527+0000 7f34e6d9d700 1 -- 172.21.15.98:0/1078280009 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34e80b4a00 msgr2=0x7f34e80d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:07.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.527+0000 7f34e6d9d700 1 --2- 172.21.15.98:0/1078280009 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34e80b4a00 0x7f34e80d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:07.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.527+0000 7f34e6d9d700 1 -- 172.21.15.98:0/1078280009 >> v1:172.21.15.98:6789/0 conn(0x7f34e80d1480 legacy=0x7f34e80b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:07.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.527+0000 7f34e6d9d700 1 -- 172.21.15.98:0/1078280009 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e80b0320 con 0x7f34e80d0010 2023-03-03T08:02:07.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.527+0000 7f34e6d9d700 1 --2- 172.21.15.98:0/1078280009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34e80d0010 0x7f34e80d03e0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f34d8006fd0 tx=0x7f34d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d02450f2c35b25 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:07.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.528+0000 7f34e5d9b700 1 -- 172.21.15.98:0/1078280009 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8005e30 con 0x7f34e80d0010 2023-03-03T08:02:07.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.528+0000 7f34e5d9b700 1 -- 172.21.15.98:0/1078280009 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d8004030 con 0x7f34e80d0010 2023-03-03T08:02:07.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.528+0000 7f34e5d9b700 1 -- 172.21.15.98:0/1078280009 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d80043e0 con 0x7f34e80d0010 2023-03-03T08:02:07.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.528+0000 7f34ec9fe700 1 -- 172.21.15.98:0/1078280009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34e80d0010 msgr2=0x7f34e80d03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:07.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.528+0000 7f34ec9fe700 1 --2- 172.21.15.98:0/1078280009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34e80d0010 0x7f34e80d03e0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f34d8006fd0 tx=0x7f34d8009bd0 comp rx=0 tx=0).stop 2023-03-03T08:02:07.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.528+0000 7f34ec9fe700 1 -- 172.21.15.98:0/1078280009 shutdown_connections 2023-03-03T08:02:07.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.528+0000 7f34ec9fe700 1 --2- 172.21.15.98:0/1078280009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34e80d0010 0x7f34e80d03e0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:07.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.528+0000 7f34ec9fe700 1 --2- 172.21.15.98:0/1078280009 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34e80b4a00 0x7f34e80d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:07.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.528+0000 7f34ec9fe700 1 -- 172.21.15.98:0/1078280009 >> 172.21.15.98:0/1078280009 conn(0x7f34e8073b00 msgr2=0x7f34e80d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:07.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.528+0000 7f34ec9fe700 1 -- 172.21.15.98:0/1078280009 shutdown_connections 2023-03-03T08:02:07.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.528+0000 7f34ec9fe700 1 -- 172.21.15.98:0/1078280009 wait complete. 2023-03-03T08:02:07.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.531+0000 7f34ec9fe700 1 Processor -- start 2023-03-03T08:02:07.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.532+0000 7f34ec9fe700 1 -- start start 2023-03-03T08:02:07.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.533+0000 7f34ec9fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f34e80b4a00 0x7f34e815a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:07.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.533+0000 7f34ec9fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34e80d0010 0x7f34e806c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:07.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.533+0000 7f34ec9fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34e80d1480 0x7f34e806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:07.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.533+0000 7f34ec9fe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8157a60 con 0x7f34e80d0010 2023-03-03T08:02:07.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.533+0000 7f34ec9fe700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8157cb0 con 0x7f34e80b4a00 2023-03-03T08:02:07.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.533+0000 7f34ec9fe700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8157f90 con 0x7f34e80d1480 2023-03-03T08:02:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.533+0000 7f34e659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34e80d0010 0x7f34e806c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.533+0000 7f34e6d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f34e80b4a00 0x7f34e815a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.533+0000 7f34e659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34e80d0010 0x7f34e806c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35228/0 (socket says 172.21.15.98:35228) 2023-03-03T08:02:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.533+0000 7f34e6d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f34e80b4a00 0x7f34e815a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:39434/0 (socket says 172.21.15.98:39434) 2023-03-03T08:02:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.533+0000 7f34e659c700 1 -- 172.21.15.98:0/226984053 learned_addr learned my addr 172.21.15.98:0/226984053 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.533+0000 7f34e759e700 1 --2- 172.21.15.98:0/226984053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34e80d1480 0x7f34e806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:07.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.534+0000 7f34e6d9d700 1 -- 172.21.15.98:0/226984053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34e80d1480 msgr2=0x7f34e806c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:07.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.534+0000 7f34e6d9d700 1 --2- 172.21.15.98:0/226984053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34e80d1480 0x7f34e806c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:07.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.534+0000 7f34e6d9d700 1 -- 172.21.15.98:0/226984053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34e80d0010 msgr2=0x7f34e806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:07.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.534+0000 7f34e6d9d700 1 --2- 172.21.15.98:0/226984053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34e80d0010 0x7f34e806c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:07.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.534+0000 7f34e6d9d700 1 -- 172.21.15.98:0/226984053 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d8009500 con 0x7f34e80b4a00 2023-03-03T08:02:07.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.534+0000 7f34e659c700 1 --2- 172.21.15.98:0/226984053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34e80d0010 0x7f34e806c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:07.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.534+0000 7f34e759e700 1 --2- 172.21.15.98:0/226984053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34e80d1480 0x7f34e806c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:07.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.534+0000 7f34e6d9d700 1 --2- 172.21.15.98:0/226984053 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f34e80b4a00 0x7f34e815a7a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009fd0 tx=0x7f34d8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:07.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.537+0000 7f34cf7fe700 1 -- 172.21.15.98:0/226984053 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8003c70 con 0x7f34e80b4a00 2023-03-03T08:02:07.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.537+0000 7f34cf7fe700 1 -- 172.21.15.98:0/226984053 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d8003dd0 con 0x7f34e80b4a00 2023-03-03T08:02:07.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.537+0000 7f34cf7fe700 1 -- 172.21.15.98:0/226984053 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d801c620 con 0x7f34e80b4a00 2023-03-03T08:02:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.537+0000 7f34ec9fe700 1 -- 172.21.15.98:0/226984053 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e80b08e0 con 0x7f34e80b4a00 2023-03-03T08:02:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.537+0000 7f34ec9fe700 1 -- 172.21.15.98:0/226984053 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e80b0600 con 0x7f34e80b4a00 2023-03-03T08:02:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.538+0000 7f34ec9fe700 1 -- 172.21.15.98:0/226984053 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e80d0b30 con 0x7f34e80b4a00 2023-03-03T08:02:07.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.539+0000 7f34cf7fe700 1 -- 172.21.15.98:0/226984053 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 14) v1 ==== 43500+0+0 (secure 0 0 0) 0x7f34d802b070 con 0x7f34e80b4a00 2023-03-03T08:02:07.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.539+0000 7f34cf7fe700 1 --2- 172.21.15.98:0/226984053 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f34d4036440 0x7f34d40388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:07.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.539+0000 7f34cf7fe700 1 -- 172.21.15.98:0/226984053 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6621+0+0 (secure 0 0 0) 0x7f34d8049bf0 con 0x7f34e80b4a00 2023-03-03T08:02:07.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.540+0000 7f34e659c700 1 --2- 172.21.15.98:0/226984053 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f34d4036440 0x7f34d40388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:07.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.541+0000 7f34e659c700 1 --2- 172.21.15.98:0/226984053 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f34d4036440 0x7f34d40388f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f34e80b40e0 tx=0x7f34dc009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:07.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.545+0000 7f34cf7fe700 1 -- 172.21.15.98:0/226984053 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f34d8019330 con 0x7f34e80b4a00 2023-03-03T08:02:07.593 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:07.590+0000 7f8457ad6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-03T08:02:07.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.752+0000 7f34ec9fe700 1 -- 172.21.15.98:0/226984053 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34e80729b0 con 0x7f34e80b4a00 2023-03-03T08:02:07.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.753+0000 7f34cf7fe700 1 -- 172.21.15.98:0/226984053 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18784 (secure 0 0 0) 0x7f34d8016030 con 0x7f34e80b4a00 2023-03-03T08:02:07.757 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:07.757 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":79,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:02:00.117924+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:01:19.439394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-03-03T08:01:20.381345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/4204485361":"2023-03-04T08:02:00.117879+0000","172.21.15.98:6833/108777":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/3513799141":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/2444435923":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/2644355772":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/1900692259":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/3107079680":"2023-03-04T08:02:00.117879+0000","172.21.15.98:6832/108777":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/3559163180":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/918927922":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/1647401399":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/3630329692":"2023-03-04T08:02:00.117879+0000","172.21.15.98:0/2323983002":"2023-03-04T08:02:00.117879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:02:07.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.756+0000 7f34ec9fe700 1 -- 172.21.15.98:0/226984053 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f34d4036440 msgr2=0x7f34d40388f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:07.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.756+0000 7f34ec9fe700 1 --2- 172.21.15.98:0/226984053 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f34d4036440 0x7f34d40388f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f34e80b40e0 tx=0x7f34dc009210 comp rx=0 tx=0).stop 2023-03-03T08:02:07.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.756+0000 7f34ec9fe700 1 -- 172.21.15.98:0/226984053 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f34e80b4a00 msgr2=0x7f34e815a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:07.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.756+0000 7f34ec9fe700 1 --2- 172.21.15.98:0/226984053 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f34e80b4a00 0x7f34e815a7a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009fd0 tx=0x7f34d8004750 comp rx=0 tx=0).stop 2023-03-03T08:02:07.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.756+0000 7f34ec9fe700 1 -- 172.21.15.98:0/226984053 shutdown_connections 2023-03-03T08:02:07.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.756+0000 7f34ec9fe700 1 --2- 172.21.15.98:0/226984053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34e80d1480 0x7f34e806c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:07.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.756+0000 7f34ec9fe700 1 --2- 172.21.15.98:0/226984053 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f34e80b4a00 0x7f34e815a7a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:07.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.756+0000 7f34ec9fe700 1 --2- 172.21.15.98:0/226984053 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f34d4036440 0x7f34d40388f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:07.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.756+0000 7f34ec9fe700 1 --2- 172.21.15.98:0/226984053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34e80d0010 0x7f34e806c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:07.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.756+0000 7f34ec9fe700 1 -- 172.21.15.98:0/226984053 >> 172.21.15.98:0/226984053 conn(0x7f34e8073b00 msgr2=0x7f34e80b7000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:07.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.756+0000 7f34ec9fe700 1 -- 172.21.15.98:0/226984053 shutdown_connections 2023-03-03T08:02:07.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:07.757+0000 7f34ec9fe700 1 -- 172.21.15.98:0/226984053 wait complete. 2023-03-03T08:02:07.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-03-03T08:02:07.899 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:07.897+0000 7f8457ad6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-03T08:02:08.045 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:08.043+0000 7f8457ad6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-03T08:02:08.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.080+0000 7fa7c359e700 1 Processor -- start 2023-03-03T08:02:08.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.080+0000 7fa7c359e700 1 -- start start 2023-03-03T08:02:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.081+0000 7fa7c359e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7c40b5e70 0x7fa7c40b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.081+0000 7fa7c359e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa7c40cf570 0x7fa7c40b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.082+0000 7fa7c359e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c415ce00 con 0x7fa7c40b8910 2023-03-03T08:02:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.082+0000 7fa7c359e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7c40d0bf0 con 0x7fa7c40cf570 2023-03-03T08:02:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.082+0000 7fa7c359e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c40b8ce0 con 0x7fa7c40b5e70 2023-03-03T08:02:08.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.082+0000 7fa7c259c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa7c40b8910 0x7fa7c41604a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45512/0 (socket says 172.21.15.98:45512) 2023-03-03T08:02:08.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.082+0000 7fa7c259c700 1 -- 172.21.15.98:0/2465234579 learned_addr learned my addr 172.21.15.98:0/2465234579 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:08.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.082+0000 7fa7c2d9d700 1 --2- 172.21.15.98:0/2465234579 >> v2:172.21.15.100:3300/0 conn(0x7fa7c40cf570 0x7fa7c40b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:08.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.082+0000 7fa7c1d9b700 1 --2- 172.21.15.98:0/2465234579 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7c40b5e70 0x7fa7c40b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:08.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.083+0000 7fa7c159a700 1 -- 172.21.15.98:0/2465234579 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971590458 0 0) 0x7fa7b4001110 con 0x7fa7c40b8910 2023-03-03T08:02:08.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.083+0000 7fa7c159a700 1 -- 172.21.15.98:0/2465234579 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7ac0035b0 con 0x7fa7c40b8910 2023-03-03T08:02:08.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.083+0000 7fa7c1d9b700 1 -- 172.21.15.98:0/2465234579 >> v2:172.21.15.100:3300/0 conn(0x7fa7c40cf570 msgr2=0x7fa7c40b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:08.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.083+0000 7fa7c1d9b700 1 --2- 172.21.15.98:0/2465234579 >> v2:172.21.15.100:3300/0 conn(0x7fa7c40cf570 0x7fa7c40b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.083+0000 7fa7c1d9b700 1 -- 172.21.15.98:0/2465234579 >> v1:172.21.15.98:6789/0 conn(0x7fa7c40b8910 legacy=0x7fa7c41604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.083+0000 7fa7c1d9b700 1 -- 172.21.15.98:0/2465234579 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c4160cf0 con 0x7fa7c40b5e70 2023-03-03T08:02:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.083+0000 7fa7c2d9d700 1 --2- 172.21.15.98:0/2465234579 >> v2:172.21.15.100:3300/0 conn(0x7fa7c40cf570 0x7fa7c40b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.083+0000 7fa7c1d9b700 1 --2- 172.21.15.98:0/2465234579 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7c40b5e70 0x7fa7c40b4aa0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8009840 tx=0x7fa7b8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e02293f17bb6ceb4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.083+0000 7fa7c159a700 1 -- 172.21.15.98:0/2465234579 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b8011830 con 0x7fa7c40b5e70 2023-03-03T08:02:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.084+0000 7fa7c159a700 1 -- 172.21.15.98:0/2465234579 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b8011990 con 0x7fa7c40b5e70 2023-03-03T08:02:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.084+0000 7fa7c159a700 1 -- 172.21.15.98:0/2465234579 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b8011d20 con 0x7fa7c40b5e70 2023-03-03T08:02:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.084+0000 7fa7c359e700 1 -- 172.21.15.98:0/2465234579 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7c40b5e70 msgr2=0x7fa7c40b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.084+0000 7fa7c359e700 1 --2- 172.21.15.98:0/2465234579 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7c40b5e70 0x7fa7c40b4aa0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8009840 tx=0x7fa7b8009b50 comp rx=0 tx=0).stop 2023-03-03T08:02:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.084+0000 7fa7c359e700 1 -- 172.21.15.98:0/2465234579 shutdown_connections 2023-03-03T08:02:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.084+0000 7fa7c359e700 1 --2- 172.21.15.98:0/2465234579 >> v2:172.21.15.100:3300/0 conn(0x7fa7c40cf570 0x7fa7c40b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.084+0000 7fa7c359e700 1 --2- 172.21.15.98:0/2465234579 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7c40b5e70 0x7fa7c40b4aa0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.084+0000 7fa7c359e700 1 -- 172.21.15.98:0/2465234579 >> 172.21.15.98:0/2465234579 conn(0x7fa7c406b880 msgr2=0x7fa7c40cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.084+0000 7fa7c359e700 1 -- 172.21.15.98:0/2465234579 shutdown_connections 2023-03-03T08:02:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.084+0000 7fa7c359e700 1 -- 172.21.15.98:0/2465234579 wait complete. 2023-03-03T08:02:08.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.086+0000 7fa7c359e700 1 Processor -- start 2023-03-03T08:02:08.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.087+0000 7fa7c359e700 1 -- start start 2023-03-03T08:02:08.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.088+0000 7fa7c359e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7c40b5e70 0x7fa7c4153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:08.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.088+0000 7fa7c359e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7c40b8910 0x7fa7c40ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:08.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.088+0000 7fa7c359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c40cf570 0x7fa7c40ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:08.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.088+0000 7fa7c359e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c40cc080 con 0x7fa7c40cf570 2023-03-03T08:02:08.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.088+0000 7fa7c359e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c41e6900 con 0x7fa7c40b5e70 2023-03-03T08:02:08.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.088+0000 7fa7c359e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c41e6be0 con 0x7fa7c40b8910 2023-03-03T08:02:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.088+0000 7fa7c2d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c40cf570 0x7fa7c40ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.088+0000 7fa7c2d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c40cf570 0x7fa7c40ca630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35232/0 (socket says 172.21.15.98:35232) 2023-03-03T08:02:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.088+0000 7fa7c2d9d700 1 -- 172.21.15.98:0/133991896 learned_addr learned my addr 172.21.15.98:0/133991896 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.088+0000 7fa7c259c700 1 --2- 172.21.15.98:0/133991896 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7c40b5e70 0x7fa7c4153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.088+0000 7fa7c1d9b700 1 --2- 172.21.15.98:0/133991896 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7c40b8910 0x7fa7c40ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.089+0000 7fa7c2d9d700 1 -- 172.21.15.98:0/133991896 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7c40b8910 msgr2=0x7fa7c40ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.089+0000 7fa7c2d9d700 1 --2- 172.21.15.98:0/133991896 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7c40b8910 0x7fa7c40ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.089+0000 7fa7c2d9d700 1 -- 172.21.15.98:0/133991896 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7c40b5e70 msgr2=0x7fa7c4153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.089+0000 7fa7c2d9d700 1 --2- 172.21.15.98:0/133991896 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7c40b5e70 0x7fa7c4153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.089+0000 7fa7c2d9d700 1 -- 172.21.15.98:0/133991896 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7bc009a70 con 0x7fa7c40cf570 2023-03-03T08:02:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.089+0000 7fa7c2d9d700 1 --2- 172.21.15.98:0/133991896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c40cf570 0x7fa7c40ca630 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc00efc0 tx=0x7fa7bc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.089+0000 7fa7b2ffd700 1 -- 172.21.15.98:0/133991896 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc00ca90 con 0x7fa7c40cf570 2023-03-03T08:02:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.089+0000 7fa7b2ffd700 1 -- 172.21.15.98:0/133991896 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7bc00f040 con 0x7fa7c40cf570 2023-03-03T08:02:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.089+0000 7fa7b2ffd700 1 -- 172.21.15.98:0/133991896 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc00ce10 con 0x7fa7c40cf570 2023-03-03T08:02:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.089+0000 7fa7c359e700 1 -- 172.21.15.98:0/133991896 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b80094a0 con 0x7fa7c40cf570 2023-03-03T08:02:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.090+0000 7fa7c359e700 1 -- 172.21.15.98:0/133991896 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c41612b0 con 0x7fa7c40cf570 2023-03-03T08:02:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.091+0000 7fa7b2ffd700 1 -- 172.21.15.98:0/133991896 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) v1 ==== 43500+0+0 (secure 0 0 0) 0x7fa7bc014480 con 0x7fa7c40cf570 2023-03-03T08:02:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.091+0000 7fa7c359e700 1 -- 172.21.15.98:0/133991896 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c4160fd0 con 0x7fa7c40cf570 2023-03-03T08:02:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.091+0000 7fa7b2ffd700 1 --2- 172.21.15.98:0/133991896 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa7a40363f0 0x7fa7a40388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.092+0000 7fa7b2ffd700 1 -- 172.21.15.98:0/133991896 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6621+0+0 (secure 0 0 0) 0x7fa7bc0497a0 con 0x7fa7c40cf570 2023-03-03T08:02:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.092+0000 7fa7c259c700 1 --2- 172.21.15.98:0/133991896 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa7a40363f0 0x7fa7a40388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.093+0000 7fa7c259c700 1 --2- 172.21.15.98:0/133991896 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa7a40363f0 0x7fa7a40388a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4007fd0 tx=0x7fa7b4007c40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:08.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.097+0000 7fa7b2ffd700 1 -- 172.21.15.98:0/133991896 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa7bc012930 con 0x7fa7c40cf570 2023-03-03T08:02:08.345 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-03-03T08:02:08.343+0000 7f8457ad6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-03T08:02:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.344+0000 7fa7c359e700 1 -- 172.21.15.98:0/133991896 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa7c40d0eb0 con 0x7fa7c40cf570 2023-03-03T08:02:08.654 INFO:tasks.ceph.mon.a.smithi098.stderr:2023-03-03T08:02:08.651+0000 7fb71abf9700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-03-03T08:02:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.667+0000 7fa7b2ffd700 1 -- 172.21.15.98:0/133991896 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fa7bc01d410 con 0x7fa7c40cf570 2023-03-03T08:02:08.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.668+0000 7fa7b2ffd700 1 -- 172.21.15.98:0/133991896 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v42) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa7bc055b80 con 0x7fa7c40cf570 2023-03-03T08:02:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.671+0000 7fa7c359e700 1 -- 172.21.15.98:0/133991896 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa7a40363f0 msgr2=0x7fa7a40388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.671+0000 7fa7c359e700 1 --2- 172.21.15.98:0/133991896 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa7a40363f0 0x7fa7a40388a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4007fd0 tx=0x7fa7b4007c40 comp rx=0 tx=0).stop 2023-03-03T08:02:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.671+0000 7fa7c359e700 1 -- 172.21.15.98:0/133991896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c40cf570 msgr2=0x7fa7c40ca630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.671+0000 7fa7c359e700 1 --2- 172.21.15.98:0/133991896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c40cf570 0x7fa7c40ca630 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc00efc0 tx=0x7fa7bc00c3a0 comp rx=0 tx=0).stop 2023-03-03T08:02:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.672+0000 7fa7c359e700 1 -- 172.21.15.98:0/133991896 shutdown_connections 2023-03-03T08:02:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.672+0000 7fa7c359e700 1 --2- 172.21.15.98:0/133991896 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa7a40363f0 0x7fa7a40388a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.672+0000 7fa7c359e700 1 --2- 172.21.15.98:0/133991896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c40cf570 0x7fa7c40ca630 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.672+0000 7fa7c359e700 1 --2- 172.21.15.98:0/133991896 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7c40b5e70 0x7fa7c4153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.672+0000 7fa7c359e700 1 --2- 172.21.15.98:0/133991896 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7c40b8910 0x7fa7c40ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.672+0000 7fa7c359e700 1 -- 172.21.15.98:0/133991896 >> 172.21.15.98:0/133991896 conn(0x7fa7c406b880 msgr2=0x7fa7c415c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.672+0000 7fa7c359e700 1 -- 172.21.15.98:0/133991896 shutdown_connections 2023-03-03T08:02:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.672+0000 7fa7c359e700 1 -- 172.21.15.98:0/133991896 wait complete. 2023-03-03T08:02:08.676 INFO:teuthology.orchestra.run.smithi098.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-03-03T08:02:08.682 INFO:tasks.ceph.mds.d.smithi100.stderr: -7264> 2023-03-03T08:01:24.781+0000 7efd8d9f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T08:02:08.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-03-03T08:02:08.757 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2023-03-03T08:02:08.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.948+0000 7fdb7f7ff700 1 Processor -- start 2023-03-03T08:02:08.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.948+0000 7fdb7f7ff700 1 -- start start 2023-03-03T08:02:08.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.949+0000 7fdb7f7ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdb7815ca70 0x7fdb780cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:08.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.950+0000 7fdb7f7ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb780cf980 con 0x7fdb7806b4b0 2023-03-03T08:02:08.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.950+0000 7fdb7f7ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb780d1800 con 0x7fdb780d0270 2023-03-03T08:02:08.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.950+0000 7fdb7f7ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb780cd5b0 con 0x7fdb7815ca70 2023-03-03T08:02:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.950+0000 7fdb7cd9a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdb7806b4b0 0x7fdb780cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45524/0 (socket says 172.21.15.98:45524) 2023-03-03T08:02:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.950+0000 7fdb7cd9a700 1 -- 172.21.15.98:0/2801778725 learned_addr learned my addr 172.21.15.98:0/2801778725 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.950+0000 7fdb7d59b700 1 --2- 172.21.15.98:0/2801778725 >> v2:172.21.15.100:3300/0 conn(0x7fdb7815ca70 0x7fdb780cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:08.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.950+0000 7fdb6ffff700 1 -- 172.21.15.98:0/2801778725 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316751065 0 0) 0x7fdb780cf980 con 0x7fdb7806b4b0 2023-03-03T08:02:08.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.951+0000 7fdb6ffff700 1 -- 172.21.15.98:0/2801778725 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb640035b0 con 0x7fdb7806b4b0 2023-03-03T08:02:08.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.951+0000 7fdb7d59b700 1 -- 172.21.15.98:0/2801778725 >> v1:172.21.15.100:6789/0 conn(0x7fdb780d0270 legacy=0x7fdb780d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:08.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.951+0000 7fdb7d59b700 1 -- 172.21.15.98:0/2801778725 >> v1:172.21.15.98:6789/0 conn(0x7fdb7806b4b0 legacy=0x7fdb780cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:08.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.951+0000 7fdb7d59b700 1 -- 172.21.15.98:0/2801778725 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb780cd900 con 0x7fdb7815ca70 2023-03-03T08:02:08.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.951+0000 7fdb7d59b700 1 --2- 172.21.15.98:0/2801778725 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb7815ca70 0x7fdb780cf440 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fdb74009840 tx=0x7fdb74009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3de9e816979e238 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:08.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.951+0000 7fdb6ffff700 1 -- 172.21.15.98:0/2801778725 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb74011830 con 0x7fdb7815ca70 2023-03-03T08:02:08.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.951+0000 7fdb6ffff700 1 -- 172.21.15.98:0/2801778725 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb74011990 con 0x7fdb7815ca70 2023-03-03T08:02:08.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.951+0000 7fdb6ffff700 1 -- 172.21.15.98:0/2801778725 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb74011d20 con 0x7fdb7815ca70 2023-03-03T08:02:08.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.952+0000 7fdb7f7ff700 1 -- 172.21.15.98:0/2801778725 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb7815ca70 msgr2=0x7fdb780cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:08.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.952+0000 7fdb7f7ff700 1 --2- 172.21.15.98:0/2801778725 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb7815ca70 0x7fdb780cf440 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fdb74009840 tx=0x7fdb74009b50 comp rx=0 tx=0).stop 2023-03-03T08:02:08.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.952+0000 7fdb7f7ff700 1 -- 172.21.15.98:0/2801778725 shutdown_connections 2023-03-03T08:02:08.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.952+0000 7fdb7f7ff700 1 --2- 172.21.15.98:0/2801778725 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb7815ca70 0x7fdb780cf440 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:08.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.952+0000 7fdb7f7ff700 1 -- 172.21.15.98:0/2801778725 >> 172.21.15.98:0/2801778725 conn(0x7fdb78073e70 msgr2=0x7fdb7806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:08.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.952+0000 7fdb7f7ff700 1 -- 172.21.15.98:0/2801778725 shutdown_connections 2023-03-03T08:02:08.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.952+0000 7fdb7f7ff700 1 -- 172.21.15.98:0/2801778725 wait complete. 2023-03-03T08:02:08.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.954+0000 7fdb7f7ff700 1 Processor -- start 2023-03-03T08:02:08.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.954+0000 7fdb7f7ff700 1 -- start start 2023-03-03T08:02:08.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.955+0000 7fdb7f7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb7806b4b0 0x7fdb780b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:08.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.955+0000 7fdb7f7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb780d0270 0x7fdb7815dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:08.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.955+0000 7fdb7f7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7815ca70 0x7fdb7815e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:08.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.955+0000 7fdb7f7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb780cb4c0 con 0x7fdb7815ca70 2023-03-03T08:02:08.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.955+0000 7fdb7f7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb780cb7a0 con 0x7fdb780d0270 2023-03-03T08:02:08.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.956+0000 7fdb7f7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb780cba80 con 0x7fdb7806b4b0 2023-03-03T08:02:08.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.956+0000 7fdb7dd9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7815ca70 0x7fdb7815e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:08.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.956+0000 7fdb7dd9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7815ca70 0x7fdb7815e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35244/0 (socket says 172.21.15.98:35244) 2023-03-03T08:02:08.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.956+0000 7fdb7dd9c700 1 -- 172.21.15.98:0/1924793630 learned_addr learned my addr 172.21.15.98:0/1924793630 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:08.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.956+0000 7fdb7cd9a700 1 --2- 172.21.15.98:0/1924793630 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb780d0270 0x7fdb7815dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:08.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.956+0000 7fdb7d59b700 1 --2- 172.21.15.98:0/1924793630 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb7806b4b0 0x7fdb780b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:08.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.956+0000 7fdb7dd9c700 1 -- 172.21.15.98:0/1924793630 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb7806b4b0 msgr2=0x7fdb780b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:08.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.956+0000 7fdb7dd9c700 1 --2- 172.21.15.98:0/1924793630 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb7806b4b0 0x7fdb780b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:08.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.956+0000 7fdb7dd9c700 1 -- 172.21.15.98:0/1924793630 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb780d0270 msgr2=0x7fdb7815dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:08.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.956+0000 7fdb7dd9c700 1 --2- 172.21.15.98:0/1924793630 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb780d0270 0x7fdb7815dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:08.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.956+0000 7fdb7dd9c700 1 -- 172.21.15.98:0/1924793630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb740094a0 con 0x7fdb7815ca70 2023-03-03T08:02:08.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.956+0000 7fdb7d59b700 1 --2- 172.21.15.98:0/1924793630 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb7806b4b0 0x7fdb780b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:08.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.957+0000 7fdb7dd9c700 1 --2- 172.21.15.98:0/1924793630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7815ca70 0x7fdb7815e500 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000cb90 tx=0x7fdb7000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:08.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.957+0000 7fdb6dffb700 1 -- 172.21.15.98:0/1924793630 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb70003c00 con 0x7fdb7815ca70 2023-03-03T08:02:08.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.957+0000 7fdb6dffb700 1 -- 172.21.15.98:0/1924793630 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb70003d60 con 0x7fdb7815ca70 2023-03-03T08:02:08.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.957+0000 7fdb7f7ff700 1 -- 172.21.15.98:0/1924793630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb780d1aa0 con 0x7fdb7815ca70 2023-03-03T08:02:08.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.957+0000 7fdb6dffb700 1 -- 172.21.15.98:0/1924793630 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7000abb0 con 0x7fdb7815ca70 2023-03-03T08:02:08.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.957+0000 7fdb7f7ff700 1 -- 172.21.15.98:0/1924793630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb780cdbe0 con 0x7fdb7815ca70 2023-03-03T08:02:08.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.958+0000 7fdb7f7ff700 1 -- 172.21.15.98:0/1924793630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb7815ce40 con 0x7fdb7815ca70 2023-03-03T08:02:08.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.959+0000 7fdb6dffb700 1 -- 172.21.15.98:0/1924793630 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fdb70010050 con 0x7fdb7815ca70 2023-03-03T08:02:08.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.961+0000 7fdb6dffb700 1 --2- 172.21.15.98:0/1924793630 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdb5c068380 0x7fdb5c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:08.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.961+0000 7fdb6dffb700 1 -- 172.21.15.98:0/1924793630 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6707+0+0 (secure 0 0 0) 0x7fdb70085f60 con 0x7fdb7815ca70 2023-03-03T08:02:08.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.961+0000 7fdb7d59b700 1 --2- 172.21.15.98:0/1924793630 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdb5c068380 0x7fdb5c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:08.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.962+0000 7fdb7d59b700 1 --2- 172.21.15.98:0/1924793630 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdb5c068380 0x7fdb5c06a830 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdb740090b0 tx=0x7fdb74008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:08.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:08.964+0000 7fdb6dffb700 1 -- 172.21.15.98:0/1924793630 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdb70052450 con 0x7fdb7815ca70 2023-03-03T08:02:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.201+0000 7fdb7f7ff700 1 -- 172.21.15.98:0/1924793630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdb7804e6e0 con 0x7fdb7815ca70 2023-03-03T08:02:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.671+0000 7fdb6dffb700 1 -- 172.21.15.98:0/1924793630 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdb70055a70 con 0x7fdb7815ca70 2023-03-03T08:02:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.674+0000 7fdb7f7ff700 1 -- 172.21.15.98:0/1924793630 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdb5c068380 msgr2=0x7fdb5c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.674+0000 7fdb7f7ff700 1 --2- 172.21.15.98:0/1924793630 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdb5c068380 0x7fdb5c06a830 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdb740090b0 tx=0x7fdb74008030 comp rx=0 tx=0).stop 2023-03-03T08:02:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.674+0000 7fdb7f7ff700 1 -- 172.21.15.98:0/1924793630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7815ca70 msgr2=0x7fdb7815e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.674+0000 7fdb7f7ff700 1 --2- 172.21.15.98:0/1924793630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7815ca70 0x7fdb7815e500 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000cb90 tx=0x7fdb7000cf50 comp rx=0 tx=0).stop 2023-03-03T08:02:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.674+0000 7fdb7f7ff700 1 -- 172.21.15.98:0/1924793630 shutdown_connections 2023-03-03T08:02:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.674+0000 7fdb7f7ff700 1 --2- 172.21.15.98:0/1924793630 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdb5c068380 0x7fdb5c06a830 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.674+0000 7fdb7f7ff700 1 --2- 172.21.15.98:0/1924793630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7815ca70 0x7fdb7815e500 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.674+0000 7fdb7f7ff700 1 --2- 172.21.15.98:0/1924793630 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb7806b4b0 0x7fdb780b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.674+0000 7fdb7f7ff700 1 --2- 172.21.15.98:0/1924793630 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb780d0270 0x7fdb7815dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.674+0000 7fdb7f7ff700 1 -- 172.21.15.98:0/1924793630 >> 172.21.15.98:0/1924793630 conn(0x7fdb78073e70 msgr2=0x7fdb780b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.674+0000 7fdb7f7ff700 1 -- 172.21.15.98:0/1924793630 shutdown_connections 2023-03-03T08:02:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.674+0000 7fdb7f7ff700 1 -- 172.21.15.98:0/1924793630 wait complete. 2023-03-03T08:02:09.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-03-03T08:02:09.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.950+0000 7f730c1ff700 1 Processor -- start 2023-03-03T08:02:09.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.950+0000 7f730c1ff700 1 -- start start 2023-03-03T08:02:09.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.952+0000 7f730c1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73040cd770 0x7f73040b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:09.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.952+0000 7f730c1ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f730415bf70 0x7f730415c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:09.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.952+0000 7f730c1ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73040d0ac0 con 0x7f73040b8880 2023-03-03T08:02:09.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.952+0000 7f730c1ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73040cdbe0 con 0x7f730415bf70 2023-03-03T08:02:09.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.952+0000 7f730c1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73040cff70 con 0x7f73040cd770 2023-03-03T08:02:09.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.952+0000 7f730979a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73040cd770 0x7f73040b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.952+0000 7f730a79c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f730415bf70 0x7f730415c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.952+0000 7f730a79c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f730415bf70 0x7f730415c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:39498/0 (socket says 172.21.15.98:39498) 2023-03-03T08:02:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.953+0000 7f730a79c700 1 -- 172.21.15.98:0/947883605 learned_addr learned my addr 172.21.15.98:0/947883605 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.952+0000 7f730979a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73040cd770 0x7f73040b52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:54678/0 (socket says 172.21.15.98:54678) 2023-03-03T08:02:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.953+0000 7f730a79c700 1 -- 172.21.15.98:0/947883605 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73040cd770 msgr2=0x7f73040b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.953+0000 7f730a79c700 1 --2- 172.21.15.98:0/947883605 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73040cd770 0x7f73040b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.953+0000 7f730a79c700 1 -- 172.21.15.98:0/947883605 >> v1:172.21.15.100:6789/0 conn(0x7f73040b8880 legacy=0x7f73040b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:09.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.953+0000 7f730a79c700 1 -- 172.21.15.98:0/947883605 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73040d0250 con 0x7f730415bf70 2023-03-03T08:02:09.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.953+0000 7f730a79c700 1 --2- 172.21.15.98:0/947883605 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f730415bf70 0x7f730415c3c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f72fc009e50 tx=0x7f72fc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30903d87592ed4c8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:09.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.954+0000 7f7308f99700 1 -- 172.21.15.98:0/947883605 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc00e070 con 0x7f730415bf70 2023-03-03T08:02:09.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.954+0000 7f7308f99700 1 -- 172.21.15.98:0/947883605 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc00b910 con 0x7f730415bf70 2023-03-03T08:02:09.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.954+0000 7f7308f99700 1 -- 172.21.15.98:0/947883605 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc00bcc0 con 0x7f730415bf70 2023-03-03T08:02:09.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.954+0000 7f730c1ff700 1 -- 172.21.15.98:0/947883605 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f730415bf70 msgr2=0x7f730415c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:09.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.954+0000 7f730c1ff700 1 --2- 172.21.15.98:0/947883605 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f730415bf70 0x7f730415c3c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f72fc009e50 tx=0x7f72fc00b330 comp rx=0 tx=0).stop 2023-03-03T08:02:09.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.954+0000 7f730c1ff700 1 -- 172.21.15.98:0/947883605 shutdown_connections 2023-03-03T08:02:09.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.954+0000 7f730c1ff700 1 --2- 172.21.15.98:0/947883605 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f730415bf70 0x7f730415c3c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:09.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.954+0000 7f730c1ff700 1 --2- 172.21.15.98:0/947883605 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73040cd770 0x7f73040b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:09.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.954+0000 7f730c1ff700 1 -- 172.21.15.98:0/947883605 >> 172.21.15.98:0/947883605 conn(0x7f730406ad80 msgr2=0x7f730406b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:09.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.954+0000 7f730c1ff700 1 -- 172.21.15.98:0/947883605 shutdown_connections 2023-03-03T08:02:09.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.954+0000 7f730c1ff700 1 -- 172.21.15.98:0/947883605 wait complete. 2023-03-03T08:02:09.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.956+0000 7f730c1ff700 1 Processor -- start 2023-03-03T08:02:09.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.957+0000 7f730c1ff700 1 -- start start 2023-03-03T08:02:09.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.958+0000 7f730c1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73040b8880 0x7f730415d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:09.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.958+0000 7f730c1ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73040cd770 0x7f73040c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:09.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.958+0000 7f730c1ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f730415bf70 0x7f73041e5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:09.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.958+0000 7f730c1ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73041e6ad0 con 0x7f730415bf70 2023-03-03T08:02:09.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.958+0000 7f730c1ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73040c9580 con 0x7f73040cd770 2023-03-03T08:02:09.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.958+0000 7f730c1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730415c340 con 0x7f73040b8880 2023-03-03T08:02:09.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.958+0000 7f730a79c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f730415bf70 0x7f73041e5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:09.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.958+0000 7f730a79c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f730415bf70 0x7f73041e5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35258/0 (socket says 172.21.15.98:35258) 2023-03-03T08:02:09.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.958+0000 7f730a79c700 1 -- 172.21.15.98:0/3205331186 learned_addr learned my addr 172.21.15.98:0/3205331186 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:09.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.958+0000 7f7309f9b700 1 --2- 172.21.15.98:0/3205331186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73040b8880 0x7f730415d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:09.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.958+0000 7f730979a700 1 --2- 172.21.15.98:0/3205331186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73040cd770 0x7f73040c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.959+0000 7f730a79c700 1 -- 172.21.15.98:0/3205331186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73040b8880 msgr2=0x7f730415d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.959+0000 7f730a79c700 1 --2- 172.21.15.98:0/3205331186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73040b8880 0x7f730415d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.959+0000 7f730a79c700 1 -- 172.21.15.98:0/3205331186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73040cd770 msgr2=0x7f73040c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.959+0000 7f730a79c700 1 --2- 172.21.15.98:0/3205331186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73040cd770 0x7f73040c9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.959+0000 7f730a79c700 1 -- 172.21.15.98:0/3205331186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc009b30 con 0x7f730415bf70 2023-03-03T08:02:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.959+0000 7f730a79c700 1 --2- 172.21.15.98:0/3205331186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f730415bf70 0x7f73041e5d70 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f730408b830 tx=0x7f72fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.959+0000 7f72fa7fc700 1 -- 172.21.15.98:0/3205331186 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc004800 con 0x7f730415bf70 2023-03-03T08:02:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.959+0000 7f72fa7fc700 1 -- 172.21.15.98:0/3205331186 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc004960 con 0x7f730415bf70 2023-03-03T08:02:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.959+0000 7f730c1ff700 1 -- 172.21.15.98:0/3205331186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73040d0cf0 con 0x7f730415bf70 2023-03-03T08:02:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.959+0000 7f72fa7fc700 1 -- 172.21.15.98:0/3205331186 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc01c680 con 0x7f730415bf70 2023-03-03T08:02:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.959+0000 7f730c1ff700 1 -- 172.21.15.98:0/3205331186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73040d0530 con 0x7f730415bf70 2023-03-03T08:02:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.960+0000 7f730c1ff700 1 -- 172.21.15.98:0/3205331186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73040b8c50 con 0x7f730415bf70 2023-03-03T08:02:09.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.962+0000 7f72fa7fc700 1 -- 172.21.15.98:0/3205331186 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f72fc02c070 con 0x7f730415bf70 2023-03-03T08:02:09.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.964+0000 7f72fa7fc700 1 --2- 172.21.15.98:0/3205331186 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f72e8068380 0x7f72e806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:09.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.964+0000 7f72fa7fc700 1 -- 172.21.15.98:0/3205331186 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6707+0+0 (secure 0 0 0) 0x7f72fc087af0 con 0x7f730415bf70 2023-03-03T08:02:09.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.965+0000 7f7309f9b700 1 --2- 172.21.15.98:0/3205331186 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f72e8068380 0x7f72e806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:09.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.965+0000 7f7309f9b700 1 --2- 172.21.15.98:0/3205331186 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f72e8068380 0x7f72e806a830 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7300007fd0 tx=0x7f7300009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:09.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:09.968+0000 7f72fa7fc700 1 -- 172.21.15.98:0/3205331186 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f72fc053fe0 con 0x7f730415bf70 2023-03-03T08:02:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.159+0000 7f730c1ff700 1 -- 172.21.15.98:0/3205331186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f730404e6e0 con 0x7f730415bf70 2023-03-03T08:02:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.680+0000 7f72fa7fc700 1 -- 172.21.15.98:0/3205331186 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v81) v1 ==== 181+0+0 (secure 0 0 0) 0x7f72fc085d40 con 0x7f730415bf70 2023-03-03T08:02:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.684+0000 7f730c1ff700 1 -- 172.21.15.98:0/3205331186 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f72e8068380 msgr2=0x7f72e806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.684+0000 7f730c1ff700 1 --2- 172.21.15.98:0/3205331186 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f72e8068380 0x7f72e806a830 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7300007fd0 tx=0x7f7300009040 comp rx=0 tx=0).stop 2023-03-03T08:02:10.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.684+0000 7f730c1ff700 1 -- 172.21.15.98:0/3205331186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f730415bf70 msgr2=0x7f73041e5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:10.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.684+0000 7f730c1ff700 1 --2- 172.21.15.98:0/3205331186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f730415bf70 0x7f73041e5d70 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f730408b830 tx=0x7f72fc004ad0 comp rx=0 tx=0).stop 2023-03-03T08:02:10.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.684+0000 7f730c1ff700 1 -- 172.21.15.98:0/3205331186 shutdown_connections 2023-03-03T08:02:10.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.684+0000 7f730c1ff700 1 --2- 172.21.15.98:0/3205331186 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f72e8068380 0x7f72e806a830 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:10.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.684+0000 7f730c1ff700 1 --2- 172.21.15.98:0/3205331186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f730415bf70 0x7f73041e5d70 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:10.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.684+0000 7f730c1ff700 1 --2- 172.21.15.98:0/3205331186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73040b8880 0x7f730415d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:10.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.684+0000 7f730c1ff700 1 --2- 172.21.15.98:0/3205331186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73040cd770 0x7f73040c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:10.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.684+0000 7f730c1ff700 1 -- 172.21.15.98:0/3205331186 >> 172.21.15.98:0/3205331186 conn(0x7f730406ad80 msgr2=0x7f73040b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:10.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.685+0000 7f730c1ff700 1 -- 172.21.15.98:0/3205331186 shutdown_connections 2023-03-03T08:02:10.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.685+0000 7f730c1ff700 1 -- 172.21.15.98:0/3205331186 wait complete. 2023-03-03T08:02:10.689 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' removed 2023-03-03T08:02:10.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-03-03T08:02:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.965+0000 7f2a219ff700 1 Processor -- start 2023-03-03T08:02:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.966+0000 7f2a219ff700 1 -- start start 2023-03-03T08:02:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.967+0000 7f2a219ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a1c0b8880 0x7f2a1c0cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.967+0000 7f2a219ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c0d0ac0 con 0x7f2a1c0b4ba0 2023-03-03T08:02:10.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.967+0000 7f2a219ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c0cff70 con 0x7f2a1c0b5720 2023-03-03T08:02:10.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.967+0000 7f2a219ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c0d0250 con 0x7f2a1c0b8880 2023-03-03T08:02:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.967+0000 7f2a1a7fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2a1c0b4ba0 0x7f2a1c0b5010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45530/0 (socket says 172.21.15.98:45530) 2023-03-03T08:02:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.967+0000 7f2a1a7fc700 1 -- 172.21.15.98:0/3755751460 learned_addr learned my addr 172.21.15.98:0/3755751460 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.968+0000 7f2a1affd700 1 --2- 172.21.15.98:0/3755751460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a1c0b8880 0x7f2a1c0cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.968+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847023676 0 0) 0x7f2a1c0d0ac0 con 0x7f2a1c0b4ba0 2023-03-03T08:02:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.968+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a040035b0 con 0x7f2a1c0b4ba0 2023-03-03T08:02:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.968+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923108194 0 0) 0x7f2a1c0cff70 con 0x7f2a1c0b5720 2023-03-03T08:02:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.968+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a1c0d0ac0 con 0x7f2a1c0b5720 2023-03-03T08:02:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.968+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2256450260 0 0) 0x7f2a040035b0 con 0x7f2a1c0b4ba0 2023-03-03T08:02:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.968+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a1c0b8880 msgr2=0x7f2a1c0cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.968+0000 7f2a19ffb700 1 --2- 172.21.15.98:0/3755751460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a1c0b8880 0x7f2a1c0cd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.968+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 >> v1:172.21.15.100:6789/0 conn(0x7f2a1c0b5720 legacy=0x7f2a1c15bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.968+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c0d0530 con 0x7f2a1c0b4ba0 2023-03-03T08:02:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.968+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f2a10001670 con 0x7f2a1c0b4ba0 2023-03-03T08:02:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 >> v1:172.21.15.98:6789/0 conn(0x7f2a1c0b4ba0 legacy=0x7f2a1c0b5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a19ffb700 1 --2- 172.21.15.98:0/3755751460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a04003bc0 0x7f2a04003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a19ffb700 1 --2- 172.21.15.98:0/3755751460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a04004850 0x7f2a04004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a19ffb700 1 --2- 172.21.15.98:0/3755751460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a1c0b8880 0x7f2a04005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c0d0250 con 0x7f2a04003bc0 2023-03-03T08:02:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0400baf0 con 0x7f2a04004850 2023-03-03T08:02:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0400bcb0 con 0x7f2a1c0b8880 2023-03-03T08:02:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a1affd700 1 --2- 172.21.15.98:0/3755751460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a04003bc0 0x7f2a04003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a1a7fc700 1 --2- 172.21.15.98:0/3755751460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a04004850 0x7f2a04004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a1b7fe700 1 --2- 172.21.15.98:0/3755751460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a1c0b8880 0x7f2a04005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a1affd700 1 -- 172.21.15.98:0/3755751460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a1c0b8880 msgr2=0x7f2a04005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a1affd700 1 --2- 172.21.15.98:0/3755751460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a1c0b8880 0x7f2a04005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a1affd700 1 -- 172.21.15.98:0/3755751460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a04004850 msgr2=0x7f2a04004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a1affd700 1 --2- 172.21.15.98:0/3755751460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a04004850 0x7f2a04004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a1affd700 1 -- 172.21.15.98:0/3755751460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a040035b0 con 0x7f2a04003bc0 2023-03-03T08:02:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.969+0000 7f2a1b7fe700 1 --2- 172.21.15.98:0/3755751460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a1c0b8880 0x7f2a04005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:10.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.970+0000 7f2a1affd700 1 --2- 172.21.15.98:0/3755751460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a04003bc0 0x7f2a04003fb0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c009b60 tx=0x7f2a04005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee1021ed1234af2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:10.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.970+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c00e6c0 con 0x7f2a04003bc0 2023-03-03T08:02:10.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.970+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0c00e820 con 0x7f2a04003bc0 2023-03-03T08:02:10.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.970+0000 7f2a19ffb700 1 -- 172.21.15.98:0/3755751460 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c00eb90 con 0x7f2a04003bc0 2023-03-03T08:02:10.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.970+0000 7f2a219ff700 1 -- 172.21.15.98:0/3755751460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a04003bc0 msgr2=0x7f2a04003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:10.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.970+0000 7f2a219ff700 1 --2- 172.21.15.98:0/3755751460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a04003bc0 0x7f2a04003fb0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c009b60 tx=0x7f2a04005880 comp rx=0 tx=0).stop 2023-03-03T08:02:10.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.971+0000 7f2a1b7fe700 1 -- 172.21.15.98:0/3755751460 reap_dead start 2023-03-03T08:02:10.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.971+0000 7f2a219ff700 1 -- 172.21.15.98:0/3755751460 shutdown_connections 2023-03-03T08:02:10.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.971+0000 7f2a219ff700 1 -- 172.21.15.98:0/3755751460 >> 172.21.15.98:0/3755751460 conn(0x7f2a1c06ad80 msgr2=0x7f2a1c06b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.971+0000 7f2a219ff700 1 -- 172.21.15.98:0/3755751460 shutdown_connections 2023-03-03T08:02:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.971+0000 7f2a219ff700 1 -- 172.21.15.98:0/3755751460 wait complete. 2023-03-03T08:02:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.973+0000 7f2a219ff700 1 Processor -- start 2023-03-03T08:02:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.973+0000 7f2a219ff700 1 -- start start 2023-03-03T08:02:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.974+0000 7f2a219ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a04003bc0 0x7f2a1c0cbb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.974+0000 7f2a219ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a1c157bd0 0x7f2a1c0c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.974+0000 7f2a219ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a1c0ca1f0 0x7f2a1c0cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.974+0000 7f2a219ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c0c9580 con 0x7f2a1c0ca1f0 2023-03-03T08:02:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.974+0000 7f2a219ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c0caab0 con 0x7f2a04003bc0 2023-03-03T08:02:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.974+0000 7f2a219ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c0cad90 con 0x7f2a1c157bd0 2023-03-03T08:02:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.974+0000 7f2a1b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a1c0ca1f0 0x7f2a1c0cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.974+0000 7f2a1b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a1c0ca1f0 0x7f2a1c0cc910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35276/0 (socket says 172.21.15.98:35276) 2023-03-03T08:02:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.974+0000 7f2a1b7fe700 1 -- 172.21.15.98:0/2041188765 learned_addr learned my addr 172.21.15.98:0/2041188765 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.975+0000 7f2a1affd700 1 --2- 172.21.15.98:0/2041188765 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a04003bc0 0x7f2a1c0cbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.975+0000 7f2a1a7fc700 1 --2- 172.21.15.98:0/2041188765 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a1c157bd0 0x7f2a1c0c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.975+0000 7f2a1b7fe700 1 -- 172.21.15.98:0/2041188765 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a1c157bd0 msgr2=0x7f2a1c0c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.975+0000 7f2a1b7fe700 1 --2- 172.21.15.98:0/2041188765 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a1c157bd0 0x7f2a1c0c9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.975+0000 7f2a1b7fe700 1 -- 172.21.15.98:0/2041188765 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a04003bc0 msgr2=0x7f2a1c0cbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.975+0000 7f2a1b7fe700 1 --2- 172.21.15.98:0/2041188765 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a04003bc0 0x7f2a1c0cbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.975+0000 7f2a1b7fe700 1 -- 172.21.15.98:0/2041188765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c0b8c50 con 0x7f2a1c0ca1f0 2023-03-03T08:02:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.975+0000 7f2a1affd700 1 --2- 172.21.15.98:0/2041188765 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a04003bc0 0x7f2a1c0cbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.975+0000 7f2a1b7fe700 1 --2- 172.21.15.98:0/2041188765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a1c0ca1f0 0x7f2a1c0cc910 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f2a14001a90 tx=0x7f2a1400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.975+0000 7f2a03fff700 1 -- 172.21.15.98:0/2041188765 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a14003c10 con 0x7f2a1c0ca1f0 2023-03-03T08:02:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.976+0000 7f2a03fff700 1 -- 172.21.15.98:0/2041188765 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a14003d90 con 0x7f2a1c0ca1f0 2023-03-03T08:02:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.976+0000 7f2a03fff700 1 -- 172.21.15.98:0/2041188765 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a14005310 con 0x7f2a1c0ca1f0 2023-03-03T08:02:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.976+0000 7f2a219ff700 1 -- 172.21.15.98:0/2041188765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a1c0d0cf0 con 0x7f2a1c0ca1f0 2023-03-03T08:02:10.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.976+0000 7f2a219ff700 1 -- 172.21.15.98:0/2041188765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1c0cde70 con 0x7f2a1c0ca1f0 2023-03-03T08:02:10.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.977+0000 7f2a219ff700 1 -- 172.21.15.98:0/2041188765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a1c04e6e0 con 0x7f2a1c0ca1f0 2023-03-03T08:02:10.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.978+0000 7f2a03fff700 1 -- 172.21.15.98:0/2041188765 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f2a14005470 con 0x7f2a1c0ca1f0 2023-03-03T08:02:10.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.979+0000 7f2a03fff700 1 --2- 172.21.15.98:0/2041188765 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2a08068380 0x7f2a0806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:10.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.979+0000 7f2a03fff700 1 -- 172.21.15.98:0/2041188765 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6269+0+0 (secure 0 0 0) 0x7f2a14085ff0 con 0x7f2a1c0ca1f0 2023-03-03T08:02:10.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.980+0000 7f2a1affd700 1 --2- 172.21.15.98:0/2041188765 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2a08068380 0x7f2a0806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:10.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.981+0000 7f2a1affd700 1 --2- 172.21.15.98:0/2041188765 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2a08068380 0x7f2a0806a830 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c0090b0 tx=0x7f2a0c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:10.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:10.983+0000 7f2a03fff700 1 -- 172.21.15.98:0/2041188765 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2a140526c0 con 0x7f2a1c0ca1f0 2023-03-03T08:02:11.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.181+0000 7f2a219ff700 1 -- 172.21.15.98:0/2041188765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2a1c0b6690 con 0x7f2a1c0ca1f0 2023-03-03T08:02:11.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.683+0000 7f2a03fff700 1 -- 172.21.15.98:0/2041188765 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v82) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2a14001ce0 con 0x7f2a1c0ca1f0 2023-03-03T08:02:11.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.686+0000 7f2a219ff700 1 -- 172.21.15.98:0/2041188765 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2a08068380 msgr2=0x7f2a0806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:11.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.686+0000 7f2a219ff700 1 --2- 172.21.15.98:0/2041188765 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2a08068380 0x7f2a0806a830 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c0090b0 tx=0x7f2a0c008030 comp rx=0 tx=0).stop 2023-03-03T08:02:11.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.686+0000 7f2a219ff700 1 -- 172.21.15.98:0/2041188765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a1c0ca1f0 msgr2=0x7f2a1c0cc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:11.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.686+0000 7f2a219ff700 1 --2- 172.21.15.98:0/2041188765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a1c0ca1f0 0x7f2a1c0cc910 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f2a14001a90 tx=0x7f2a1400ef00 comp rx=0 tx=0).stop 2023-03-03T08:02:11.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.686+0000 7f2a219ff700 1 -- 172.21.15.98:0/2041188765 shutdown_connections 2023-03-03T08:02:11.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.686+0000 7f2a219ff700 1 --2- 172.21.15.98:0/2041188765 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2a08068380 0x7f2a0806a830 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:11.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.686+0000 7f2a219ff700 1 --2- 172.21.15.98:0/2041188765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a1c0ca1f0 0x7f2a1c0cc910 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:11.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.686+0000 7f2a219ff700 1 --2- 172.21.15.98:0/2041188765 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a04003bc0 0x7f2a1c0cbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:11.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.686+0000 7f2a219ff700 1 --2- 172.21.15.98:0/2041188765 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a1c157bd0 0x7f2a1c0c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:11.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.686+0000 7f2a219ff700 1 -- 172.21.15.98:0/2041188765 >> 172.21.15.98:0/2041188765 conn(0x7f2a1c06ad80 msgr2=0x7f2a1c0b3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:11.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.686+0000 7f2a219ff700 1 -- 172.21.15.98:0/2041188765 shutdown_connections 2023-03-03T08:02:11.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.686+0000 7f2a219ff700 1 -- 172.21.15.98:0/2041188765 wait complete. 2023-03-03T08:02:11.694 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' removed 2023-03-03T08:02:11.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-03-03T08:02:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.961+0000 7f2fc6bff700 1 Processor -- start 2023-03-03T08:02:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.961+0000 7f2fc6bff700 1 -- start start 2023-03-03T08:02:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.962+0000 7f2fc6bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2fc00b5310 0x7f2fc0167850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.962+0000 7f2fc6bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc00cd7d0 con 0x7f2fc00b8d10 2023-03-03T08:02:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.962+0000 7f2fc6bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc00b90e0 con 0x7f2fc00b26d0 2023-03-03T08:02:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.962+0000 7f2fc6bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc00cda20 con 0x7f2fc00b5310 2023-03-03T08:02:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.962+0000 7f2fc499b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2fc00b8d10 0x7f2fc00cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45532/0 (socket says 172.21.15.98:45532) 2023-03-03T08:02:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.962+0000 7f2fc499b700 1 -- 172.21.15.98:0/4236245139 learned_addr learned my addr 172.21.15.98:0/4236245139 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.962+0000 7f2fc519c700 1 --2- 172.21.15.98:0/4236245139 >> v2:172.21.15.98:3300/0 conn(0x7f2fc00b5310 0x7f2fc0167850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.963+0000 7f2fbf7fe700 1 -- 172.21.15.98:0/4236245139 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746389497 0 0) 0x7f2fc00cd7d0 con 0x7f2fc00b8d10 2023-03-03T08:02:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.963+0000 7f2fbf7fe700 1 -- 172.21.15.98:0/4236245139 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb00035b0 con 0x7f2fc00b8d10 2023-03-03T08:02:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.963+0000 7f2fc519c700 1 -- 172.21.15.98:0/4236245139 >> v1:172.21.15.100:6789/0 conn(0x7f2fc00b26d0 legacy=0x7f2fc00b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.963+0000 7f2fbf7fe700 1 -- 172.21.15.98:0/4236245139 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484418487 0 0) 0x7f2fc00b90e0 con 0x7f2fc00b26d0 2023-03-03T08:02:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.963+0000 7f2fc519c700 1 -- 172.21.15.98:0/4236245139 >> v1:172.21.15.98:6789/0 conn(0x7f2fc00b8d10 legacy=0x7f2fc00cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:11.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.963+0000 7f2fc519c700 1 -- 172.21.15.98:0/4236245139 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc00cdd00 con 0x7f2fc00b5310 2023-03-03T08:02:11.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.963+0000 7f2fc519c700 1 --2- 172.21.15.98:0/4236245139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc00b5310 0x7f2fc0167850 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8009d90 tx=0x7f2fb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ee489a9e0b73fc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:11.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.963+0000 7f2fbf7fe700 1 -- 172.21.15.98:0/4236245139 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb800e070 con 0x7f2fc00b5310 2023-03-03T08:02:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.964+0000 7f2fbf7fe700 1 -- 172.21.15.98:0/4236245139 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb800bb20 con 0x7f2fc00b5310 2023-03-03T08:02:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.964+0000 7f2fbf7fe700 1 -- 172.21.15.98:0/4236245139 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb800be90 con 0x7f2fc00b5310 2023-03-03T08:02:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.964+0000 7f2fc6bff700 1 -- 172.21.15.98:0/4236245139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc00b5310 msgr2=0x7f2fc0167850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.964+0000 7f2fc6bff700 1 --2- 172.21.15.98:0/4236245139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc00b5310 0x7f2fc0167850 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8009d90 tx=0x7f2fb800b330 comp rx=0 tx=0).stop 2023-03-03T08:02:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.964+0000 7f2fc6bff700 1 -- 172.21.15.98:0/4236245139 shutdown_connections 2023-03-03T08:02:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.964+0000 7f2fc6bff700 1 --2- 172.21.15.98:0/4236245139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc00b5310 0x7f2fc0167850 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.964+0000 7f2fc6bff700 1 -- 172.21.15.98:0/4236245139 >> 172.21.15.98:0/4236245139 conn(0x7f2fc0073e70 msgr2=0x7f2fc006ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.964+0000 7f2fc6bff700 1 -- 172.21.15.98:0/4236245139 shutdown_connections 2023-03-03T08:02:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.964+0000 7f2fc6bff700 1 -- 172.21.15.98:0/4236245139 wait complete. 2023-03-03T08:02:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.966+0000 7f2fc6bff700 1 Processor -- start 2023-03-03T08:02:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.966+0000 7f2fc6bff700 1 -- start start 2023-03-03T08:02:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.967+0000 7f2fc6bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fc00b26d0 0x7f2fc00ac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.967+0000 7f2fc6bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fc00b5310 0x7f2fc015adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.967+0000 7f2fc6bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc00b8d10 0x7f2fc00ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.967+0000 7f2fc6bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc00abdb0 con 0x7f2fc00b8d10 2023-03-03T08:02:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.967+0000 7f2fc6bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc00ac710 con 0x7f2fc00b5310 2023-03-03T08:02:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.967+0000 7f2fc6bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc00ab130 con 0x7f2fc00b26d0 2023-03-03T08:02:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.968+0000 7f2fc519c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc00b8d10 0x7f2fc00ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.968+0000 7f2fc519c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc00b8d10 0x7f2fc00ccfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35296/0 (socket says 172.21.15.98:35296) 2023-03-03T08:02:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.968+0000 7f2fc519c700 1 -- 172.21.15.98:0/3100156587 learned_addr learned my addr 172.21.15.98:0/3100156587 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.968+0000 7f2fbffff700 1 --2- 172.21.15.98:0/3100156587 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fc00b5310 0x7f2fc015adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.968+0000 7f2fc499b700 1 --2- 172.21.15.98:0/3100156587 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fc00b26d0 0x7f2fc00ac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.968+0000 7f2fc519c700 1 -- 172.21.15.98:0/3100156587 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fc00b26d0 msgr2=0x7f2fc00ac1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.968+0000 7f2fc519c700 1 --2- 172.21.15.98:0/3100156587 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fc00b26d0 0x7f2fc00ac1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.968+0000 7f2fc519c700 1 -- 172.21.15.98:0/3100156587 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fc00b5310 msgr2=0x7f2fc015adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.968+0000 7f2fc519c700 1 --2- 172.21.15.98:0/3100156587 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fc00b5310 0x7f2fc015adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.968+0000 7f2fc519c700 1 -- 172.21.15.98:0/3100156587 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb8009a70 con 0x7f2fc00b8d10 2023-03-03T08:02:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.969+0000 7f2fc519c700 1 --2- 172.21.15.98:0/3100156587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc00b8d10 0x7f2fc00ccfe0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8013000 tx=0x7f2fb8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.969+0000 7f2fbd7fa700 1 -- 172.21.15.98:0/3100156587 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb80047d0 con 0x7f2fc00b8d10 2023-03-03T08:02:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.969+0000 7f2fbd7fa700 1 -- 172.21.15.98:0/3100156587 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb8021070 con 0x7f2fc00b8d10 2023-03-03T08:02:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.969+0000 7f2fbd7fa700 1 -- 172.21.15.98:0/3100156587 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8026670 con 0x7f2fc00b8d10 2023-03-03T08:02:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.969+0000 7f2fc6bff700 1 -- 172.21.15.98:0/3100156587 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc00cdfe0 con 0x7f2fc00b8d10 2023-03-03T08:02:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.969+0000 7f2fc6bff700 1 -- 172.21.15.98:0/3100156587 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc00d0e80 con 0x7f2fc00b8d10 2023-03-03T08:02:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.970+0000 7f2fc6bff700 1 -- 172.21.15.98:0/3100156587 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc00b5fd0 con 0x7f2fc00b8d10 2023-03-03T08:02:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.971+0000 7f2fbd7fa700 1 -- 172.21.15.98:0/3100156587 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f2fb802f430 con 0x7f2fc00b8d10 2023-03-03T08:02:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.972+0000 7f2fbd7fa700 1 --2- 172.21.15.98:0/3100156587 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2fa4068380 0x7f2fa406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.973+0000 7f2fbd7fa700 1 -- 172.21.15.98:0/3100156587 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5855+0+0 (secure 0 0 0) 0x7f2fb8088080 con 0x7f2fc00b8d10 2023-03-03T08:02:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.973+0000 7f2fc499b700 1 --2- 172.21.15.98:0/3100156587 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2fa4068380 0x7f2fa406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.974+0000 7f2fc499b700 1 --2- 172.21.15.98:0/3100156587 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2fa4068380 0x7f2fa406a830 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4005910 tx=0x7f2fb4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:11.977+0000 7f2fbd7fa700 1 -- 172.21.15.98:0/3100156587 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2fb8054870 con 0x7f2fc00b8d10 2023-03-03T08:02:12.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.174+0000 7f2fc6bff700 1 -- 172.21.15.98:0/3100156587 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2fc00abef0 con 0x7f2fc00b8d10 2023-03-03T08:02:12.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.180+0000 7f2fbd7fa700 1 -- 172.21.15.98:0/3100156587 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2fb8004930 con 0x7f2fc00b8d10 2023-03-03T08:02:12.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.182+0000 7f2fc6bff700 1 -- 172.21.15.98:0/3100156587 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2fa4068380 msgr2=0x7f2fa406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:12.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.182+0000 7f2fc6bff700 1 --2- 172.21.15.98:0/3100156587 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2fa4068380 0x7f2fa406a830 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4005910 tx=0x7f2fb4009040 comp rx=0 tx=0).stop 2023-03-03T08:02:12.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.182+0000 7f2fc6bff700 1 -- 172.21.15.98:0/3100156587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc00b8d10 msgr2=0x7f2fc00ccfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:12.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.182+0000 7f2fc6bff700 1 --2- 172.21.15.98:0/3100156587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc00b8d10 0x7f2fc00ccfe0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8013000 tx=0x7f2fb8004ad0 comp rx=0 tx=0).stop 2023-03-03T08:02:12.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.182+0000 7f2fc6bff700 1 -- 172.21.15.98:0/3100156587 shutdown_connections 2023-03-03T08:02:12.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.182+0000 7f2fc6bff700 1 --2- 172.21.15.98:0/3100156587 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2fa4068380 0x7f2fa406a830 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:12.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.182+0000 7f2fc6bff700 1 --2- 172.21.15.98:0/3100156587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc00b8d10 0x7f2fc00ccfe0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:12.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.182+0000 7f2fc6bff700 1 --2- 172.21.15.98:0/3100156587 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fc00b26d0 0x7f2fc00ac1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:12.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.182+0000 7f2fc6bff700 1 --2- 172.21.15.98:0/3100156587 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fc00b5310 0x7f2fc015adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:12.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.182+0000 7f2fc6bff700 1 -- 172.21.15.98:0/3100156587 >> 172.21.15.98:0/3100156587 conn(0x7f2fc0073e70 msgr2=0x7f2fc00d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:12.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.182+0000 7f2fc6bff700 1 -- 172.21.15.98:0/3100156587 shutdown_connections 2023-03-03T08:02:12.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.182+0000 7f2fc6bff700 1 -- 172.21.15.98:0/3100156587 wait complete. 2023-03-03T08:02:12.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction' 2023-03-03T08:02:12.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.455+0000 7f84b759d700 1 Processor -- start 2023-03-03T08:02:12.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.455+0000 7f84b759d700 1 -- start start 2023-03-03T08:02:12.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b759d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f84b00b50d0 0x7f84b00cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:12.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b759d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f84b0160fa0 0x7f84b00d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:12.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b759d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b0161480 con 0x7f84b01461f0 2023-03-03T08:02:12.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b759d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b00d0cb0 con 0x7f84b0160fa0 2023-03-03T08:02:12.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b759d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b015cf30 con 0x7f84b00b50d0 2023-03-03T08:02:12.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b659b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f84b01461f0 0x7f84b00b49c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45544/0 (socket says 172.21.15.98:45544) 2023-03-03T08:02:12.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b659b700 1 -- 172.21.15.98:0/1661777090 learned_addr learned my addr 172.21.15.98:0/1661777090 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:12.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b6d9c700 1 --2- 172.21.15.98:0/1661777090 >> v2:172.21.15.98:3300/0 conn(0x7f84b0160fa0 0x7f84b00d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:12.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b5d9a700 1 --2- 172.21.15.98:0/1661777090 >> v2:172.21.15.100:3300/0 conn(0x7f84b00b50d0 0x7f84b00cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:12.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b5599700 1 -- 172.21.15.98:0/1661777090 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000495668 0 0) 0x7f84b0161480 con 0x7f84b01461f0 2023-03-03T08:02:12.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b5599700 1 -- 172.21.15.98:0/1661777090 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f849c0035b0 con 0x7f84b01461f0 2023-03-03T08:02:12.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b6d9c700 1 -- 172.21.15.98:0/1661777090 >> v2:172.21.15.100:3300/0 conn(0x7f84b00b50d0 msgr2=0x7f84b00cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:12.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b6d9c700 1 --2- 172.21.15.98:0/1661777090 >> v2:172.21.15.100:3300/0 conn(0x7f84b00b50d0 0x7f84b00cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:12.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b6d9c700 1 -- 172.21.15.98:0/1661777090 >> v1:172.21.15.98:6789/0 conn(0x7f84b01461f0 legacy=0x7f84b00b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:12.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b6d9c700 1 -- 172.21.15.98:0/1661777090 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b00cd6c0 con 0x7f84b0160fa0 2023-03-03T08:02:12.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.457+0000 7f84b6d9c700 1 --2- 172.21.15.98:0/1661777090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b0160fa0 0x7f84b00d10d0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f84b008bd40 tx=0x7f84a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2762f5a0864ccd98 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:12.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.458+0000 7f84b5599700 1 -- 172.21.15.98:0/1661777090 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a800e070 con 0x7f84b0160fa0 2023-03-03T08:02:12.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.458+0000 7f84b5599700 1 -- 172.21.15.98:0/1661777090 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a800bb20 con 0x7f84b0160fa0 2023-03-03T08:02:12.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.458+0000 7f84b5599700 1 -- 172.21.15.98:0/1661777090 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8012430 con 0x7f84b0160fa0 2023-03-03T08:02:12.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.458+0000 7f84b759d700 1 -- 172.21.15.98:0/1661777090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b0160fa0 msgr2=0x7f84b00d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:12.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.458+0000 7f84b759d700 1 --2- 172.21.15.98:0/1661777090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b0160fa0 0x7f84b00d10d0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f84b008bd40 tx=0x7f84a800b330 comp rx=0 tx=0).stop 2023-03-03T08:02:12.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.458+0000 7f84b759d700 1 -- 172.21.15.98:0/1661777090 shutdown_connections 2023-03-03T08:02:12.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.458+0000 7f84b759d700 1 --2- 172.21.15.98:0/1661777090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b0160fa0 0x7f84b00d10d0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:12.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.458+0000 7f84b759d700 1 --2- 172.21.15.98:0/1661777090 >> v2:172.21.15.100:3300/0 conn(0x7f84b00b50d0 0x7f84b00cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:12.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.458+0000 7f84b759d700 1 -- 172.21.15.98:0/1661777090 >> 172.21.15.98:0/1661777090 conn(0x7f84b006b900 msgr2=0x7f84b00cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:12.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.459+0000 7f84b759d700 1 -- 172.21.15.98:0/1661777090 shutdown_connections 2023-03-03T08:02:12.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.459+0000 7f84b759d700 1 -- 172.21.15.98:0/1661777090 wait complete. 2023-03-03T08:02:12.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.461+0000 7f84b759d700 1 Processor -- start 2023-03-03T08:02:12.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.461+0000 7f84b759d700 1 -- start start 2023-03-03T08:02:12.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b759d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b00b50d0 0x7f84b00c5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:12.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b759d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84b01461f0 0x7f84b015f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:12.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b759d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84b0160fa0 0x7f84b015bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:12.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b759d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00b74a0 con 0x7f84b00b50d0 2023-03-03T08:02:12.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b759d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00b6890 con 0x7f84b0160fa0 2023-03-03T08:02:12.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b759d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00b6b70 con 0x7f84b01461f0 2023-03-03T08:02:12.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b659b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b00b50d0 0x7f84b00c5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:12.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b659b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b00b50d0 0x7f84b00c5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35310/0 (socket says 172.21.15.98:35310) 2023-03-03T08:02:12.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b659b700 1 -- 172.21.15.98:0/513160048 learned_addr learned my addr 172.21.15.98:0/513160048 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:12.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b5d9a700 1 --2- 172.21.15.98:0/513160048 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84b01461f0 0x7f84b015f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:12.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b6d9c700 1 --2- 172.21.15.98:0/513160048 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84b0160fa0 0x7f84b015bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:12.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b659b700 1 -- 172.21.15.98:0/513160048 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84b01461f0 msgr2=0x7f84b015f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:12.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b659b700 1 --2- 172.21.15.98:0/513160048 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84b01461f0 0x7f84b015f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:12.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b659b700 1 -- 172.21.15.98:0/513160048 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84b0160fa0 msgr2=0x7f84b015bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:12.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b659b700 1 --2- 172.21.15.98:0/513160048 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84b0160fa0 0x7f84b015bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:12.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.462+0000 7f84b659b700 1 -- 172.21.15.98:0/513160048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a8009a70 con 0x7f84b00b50d0 2023-03-03T08:02:12.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.463+0000 7f84b659b700 1 --2- 172.21.15.98:0/513160048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b00b50d0 0x7f84b00c5d80 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00c700 tx=0x7f84ac00cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:12.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.463+0000 7f84a6ffd700 1 -- 172.21.15.98:0/513160048 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac005d70 con 0x7f84b00b50d0 2023-03-03T08:02:12.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.463+0000 7f84a6ffd700 1 -- 172.21.15.98:0/513160048 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ac0070d0 con 0x7f84b00b50d0 2023-03-03T08:02:12.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.463+0000 7f84a6ffd700 1 -- 172.21.15.98:0/513160048 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac007460 con 0x7f84b00b50d0 2023-03-03T08:02:12.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.463+0000 7f84b759d700 1 -- 172.21.15.98:0/513160048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b00cdc80 con 0x7f84b00b50d0 2023-03-03T08:02:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.463+0000 7f84b759d700 1 -- 172.21.15.98:0/513160048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b00cd9a0 con 0x7f84b00b50d0 2023-03-03T08:02:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.464+0000 7f84b759d700 1 -- 172.21.15.98:0/513160048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b01465c0 con 0x7f84b00b50d0 2023-03-03T08:02:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.466+0000 7f84a6ffd700 1 -- 172.21.15.98:0/513160048 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f84ac019030 con 0x7f84b00b50d0 2023-03-03T08:02:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.467+0000 7f84a6ffd700 1 --2- 172.21.15.98:0/513160048 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f84940683b0 0x7f849406a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:12.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.467+0000 7f84a6ffd700 1 -- 172.21.15.98:0/513160048 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5855+0+0 (secure 0 0 0) 0x7f84ac0851b0 con 0x7f84b00b50d0 2023-03-03T08:02:12.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.468+0000 7f84b5d9a700 1 --2- 172.21.15.98:0/513160048 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f84940683b0 0x7f849406a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:12.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.469+0000 7f84b5d9a700 1 --2- 172.21.15.98:0/513160048 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f84940683b0 0x7f849406a860 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f84b00d0190 tx=0x7f84a0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:12.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.472+0000 7f84a6ffd700 1 -- 172.21.15.98:0/513160048 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f84ac051980 con 0x7f84b00b50d0 2023-03-03T08:02:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:12.737+0000 7f84b759d700 1 -- 172.21.15.98:0/513160048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]} v 0) v1 -- 0x7f84b0068290 con 0x7f84b00b50d0 2023-03-03T08:02:13.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.699+0000 7f84a6ffd700 1 -- 172.21.15.98:0/513160048 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]}]=0 v356) v1 ==== 131+0+0 (secure 0 0 0) 0x7f84ac054fa0 con 0x7f84b00b50d0 2023-03-03T08:02:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.703+0000 7f84b759d700 1 -- 172.21.15.98:0/513160048 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f84940683b0 msgr2=0x7f849406a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.703+0000 7f84b759d700 1 --2- 172.21.15.98:0/513160048 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f84940683b0 0x7f849406a860 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f84b00d0190 tx=0x7f84a0008040 comp rx=0 tx=0).stop 2023-03-03T08:02:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.703+0000 7f84b759d700 1 -- 172.21.15.98:0/513160048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b00b50d0 msgr2=0x7f84b00c5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.703+0000 7f84b759d700 1 --2- 172.21.15.98:0/513160048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b00b50d0 0x7f84b00c5d80 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00c700 tx=0x7f84ac00cac0 comp rx=0 tx=0).stop 2023-03-03T08:02:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.703+0000 7f84b759d700 1 -- 172.21.15.98:0/513160048 shutdown_connections 2023-03-03T08:02:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.703+0000 7f84b759d700 1 --2- 172.21.15.98:0/513160048 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f84940683b0 0x7f849406a860 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.703+0000 7f84b759d700 1 --2- 172.21.15.98:0/513160048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b00b50d0 0x7f84b00c5d80 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.703+0000 7f84b759d700 1 --2- 172.21.15.98:0/513160048 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84b0160fa0 0x7f84b015bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.703+0000 7f84b759d700 1 --2- 172.21.15.98:0/513160048 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84b01461f0 0x7f84b015f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.703+0000 7f84b759d700 1 -- 172.21.15.98:0/513160048 >> 172.21.15.98:0/513160048 conn(0x7f84b006b900 msgr2=0x7f84b00b3600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.703+0000 7f84b759d700 1 -- 172.21.15.98:0/513160048 shutdown_connections 2023-03-03T08:02:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.703+0000 7f84b759d700 1 -- 172.21.15.98:0/513160048 wait complete. 2023-03-03T08:02:13.734 INFO:tasks.cephfs_test_runner:test_mgr_eviction (tasks.cephfs.test_volumes.TestMisc) ... ok 2023-03-03T08:02:13.735 INFO:tasks.cephfs_test_runner:Starting test: test_names_can_only_be_goodchars (tasks.cephfs.test_volumes.TestMisc) 2023-03-03T08:02:13.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars' 2023-03-03T08:02:13.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.984+0000 7f4c51bff700 1 Processor -- start 2023-03-03T08:02:13.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.984+0000 7f4c51bff700 1 -- start start 2023-03-03T08:02:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.985+0000 7f4c51bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4c4c0b8b90 0x7f4c4c0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.985+0000 7f4c51bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c4c0b5410 0x7f4c4c0d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.986+0000 7f4c51bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c4c0d0ef0 con 0x7f4c4c0b87c0 2023-03-03T08:02:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.986+0000 7f4c51bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c0d0600 con 0x7f4c4c0b5410 2023-03-03T08:02:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.986+0000 7f4c51bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c0cd970 con 0x7f4c4c0b8b90 2023-03-03T08:02:13.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.986+0000 7f4c4bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c4c0b5410 0x7f4c4c0d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:13.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.986+0000 7f4c4b7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4c4c0b87c0 0x7f4c4c0b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45556/0 (socket says 172.21.15.98:45556) 2023-03-03T08:02:13.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.986+0000 7f4c4b7fe700 1 -- 172.21.15.98:0/886816304 learned_addr learned my addr 172.21.15.98:0/886816304 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:13.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.986+0000 7f4c4affd700 1 --2- 172.21.15.98:0/886816304 >> v2:172.21.15.100:3300/0 conn(0x7f4c4c0b8b90 0x7f4c4c0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:13.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.986+0000 7f4c4a7fc700 1 -- 172.21.15.98:0/886816304 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693224074 0 0) 0x7f4c4c0d0ef0 con 0x7f4c4c0b87c0 2023-03-03T08:02:13.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.986+0000 7f4c4bfff700 1 -- 172.21.15.98:0/886816304 >> v2:172.21.15.100:3300/0 conn(0x7f4c4c0b8b90 msgr2=0x7f4c4c0cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:13.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.986+0000 7f4c4bfff700 1 --2- 172.21.15.98:0/886816304 >> v2:172.21.15.100:3300/0 conn(0x7f4c4c0b8b90 0x7f4c4c0cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:13.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.986+0000 7f4c4bfff700 1 -- 172.21.15.98:0/886816304 >> v1:172.21.15.98:6789/0 conn(0x7f4c4c0b87c0 legacy=0x7f4c4c0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:13.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.986+0000 7f4c4bfff700 1 -- 172.21.15.98:0/886816304 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c0cdc50 con 0x7f4c4c0b5410 2023-03-03T08:02:13.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.987+0000 7f4c4bfff700 1 --2- 172.21.15.98:0/886816304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0b5410 0x7f4c4c0d13d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c08c240 tx=0x7f4c4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a7f577232c2f81 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:13.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.987+0000 7f4c4a7fc700 1 -- 172.21.15.98:0/886816304 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4400e070 con 0x7f4c4c0b5410 2023-03-03T08:02:13.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.987+0000 7f4c4a7fc700 1 -- 172.21.15.98:0/886816304 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4400bb20 con 0x7f4c4c0b5410 2023-03-03T08:02:13.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.987+0000 7f4c4a7fc700 1 -- 172.21.15.98:0/886816304 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44012430 con 0x7f4c4c0b5410 2023-03-03T08:02:13.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.988+0000 7f4c51bff700 1 -- 172.21.15.98:0/886816304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0b5410 msgr2=0x7f4c4c0d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:13.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.988+0000 7f4c51bff700 1 --2- 172.21.15.98:0/886816304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0b5410 0x7f4c4c0d13d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c08c240 tx=0x7f4c4400b330 comp rx=0 tx=0).stop 2023-03-03T08:02:13.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.988+0000 7f4c51bff700 1 -- 172.21.15.98:0/886816304 shutdown_connections 2023-03-03T08:02:13.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.988+0000 7f4c51bff700 1 --2- 172.21.15.98:0/886816304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0b5410 0x7f4c4c0d13d0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:13.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.988+0000 7f4c51bff700 1 --2- 172.21.15.98:0/886816304 >> v2:172.21.15.100:3300/0 conn(0x7f4c4c0b8b90 0x7f4c4c0cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:13.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.988+0000 7f4c51bff700 1 -- 172.21.15.98:0/886816304 >> 172.21.15.98:0/886816304 conn(0x7f4c4c06aee0 msgr2=0x7f4c4c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:13.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.988+0000 7f4c51bff700 1 -- 172.21.15.98:0/886816304 shutdown_connections 2023-03-03T08:02:13.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.988+0000 7f4c51bff700 1 -- 172.21.15.98:0/886816304 wait complete. 2023-03-03T08:02:13.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.990+0000 7f4c51bff700 1 Processor -- start 2023-03-03T08:02:13.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.990+0000 7f4c51bff700 1 -- start start 2023-03-03T08:02:13.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.991+0000 7f4c51bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c4c0b5410 0x7f4c4c15c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:13.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.991+0000 7f4c51bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c4c0b87c0 0x7f4c4c15e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:13.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.991+0000 7f4c51bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0b8b90 0x7f4c4c0ab660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:13.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.991+0000 7f4c51bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0ab240 con 0x7f4c4c0b8b90 2023-03-03T08:02:13.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.991+0000 7f4c51bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0c9c70 con 0x7f4c4c0b87c0 2023-03-03T08:02:13.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.991+0000 7f4c51bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0c9f50 con 0x7f4c4c0b5410 2023-03-03T08:02:13.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.991+0000 7f4c4bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0b8b90 0x7f4c4c0ab660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:13.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.991+0000 7f4c4bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0b8b90 0x7f4c4c0ab660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35318/0 (socket says 172.21.15.98:35318) 2023-03-03T08:02:13.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.991+0000 7f4c4bfff700 1 -- 172.21.15.98:0/1193303651 learned_addr learned my addr 172.21.15.98:0/1193303651 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:13.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.992+0000 7f4c4affd700 1 --2- 172.21.15.98:0/1193303651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c4c0b87c0 0x7f4c4c15e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:13.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.992+0000 7f4c4b7fe700 1 --2- 172.21.15.98:0/1193303651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c4c0b5410 0x7f4c4c15c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:13.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.992+0000 7f4c4bfff700 1 -- 172.21.15.98:0/1193303651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c4c0b5410 msgr2=0x7f4c4c15c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:13.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.992+0000 7f4c4bfff700 1 --2- 172.21.15.98:0/1193303651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c4c0b5410 0x7f4c4c15c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.992+0000 7f4c4bfff700 1 -- 172.21.15.98:0/1193303651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c4c0b87c0 msgr2=0x7f4c4c15e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.992+0000 7f4c4bfff700 1 --2- 172.21.15.98:0/1193303651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c4c0b87c0 0x7f4c4c15e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.992+0000 7f4c4bfff700 1 -- 172.21.15.98:0/1193303651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c44009a70 con 0x7f4c4c0b8b90 2023-03-03T08:02:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.992+0000 7f4c4affd700 1 --2- 172.21.15.98:0/1193303651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c4c0b87c0 0x7f4c4c15e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.992+0000 7f4c4bfff700 1 --2- 172.21.15.98:0/1193303651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0b8b90 0x7f4c4c0ab660 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f4c44014040 tx=0x7f4c44004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.992+0000 7f4c33fff700 1 -- 172.21.15.98:0/1193303651 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44012430 con 0x7f4c4c0b8b90 2023-03-03T08:02:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.993+0000 7f4c33fff700 1 -- 172.21.15.98:0/1193303651 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c44020070 con 0x7f4c4c0b8b90 2023-03-03T08:02:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.993+0000 7f4c33fff700 1 -- 172.21.15.98:0/1193303651 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44004280 con 0x7f4c4c0b8b90 2023-03-03T08:02:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.993+0000 7f4c51bff700 1 -- 172.21.15.98:0/1193303651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4c0cdf30 con 0x7f4c4c0b8b90 2023-03-03T08:02:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.993+0000 7f4c51bff700 1 -- 172.21.15.98:0/1193303651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c073a80 con 0x7f4c4c0b8b90 2023-03-03T08:02:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.994+0000 7f4c51bff700 1 -- 172.21.15.98:0/1193303651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c4c0ce0c0 con 0x7f4c4c0b8b90 2023-03-03T08:02:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.995+0000 7f4c33fff700 1 -- 172.21.15.98:0/1193303651 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f4c4402e070 con 0x7f4c4c0b8b90 2023-03-03T08:02:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.996+0000 7f4c33fff700 1 --2- 172.21.15.98:0/1193303651 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4c38068380 0x7f4c3806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.996+0000 7f4c33fff700 1 -- 172.21.15.98:0/1193303651 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5855+0+0 (secure 0 0 0) 0x7f4c44022070 con 0x7f4c4c0b8b90 2023-03-03T08:02:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.997+0000 7f4c4b7fe700 1 --2- 172.21.15.98:0/1193303651 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4c38068380 0x7f4c3806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:13.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:13.998+0000 7f4c4b7fe700 1 --2- 172.21.15.98:0/1193303651 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4c38068380 0x7f4c3806a830 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c000f60 tx=0x7f4c3c007be0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:14.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.001+0000 7f4c33fff700 1 -- 172.21.15.98:0/1193303651 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4c44054230 con 0x7f4c4c0b8b90 2023-03-03T08:02:14.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.275+0000 7f4c51bff700 1 -- 172.21.15.98:0/1193303651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]} v 0) v1 -- 0x7f4c4c0729b0 con 0x7f4c4c0b8b90 2023-03-03T08:02:14.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.705+0000 7f4c33fff700 1 -- 172.21.15.98:0/1193303651 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]}]=0 v357) v1 ==== 149+0+0 (secure 0 0 0) 0x7f4c44057850 con 0x7f4c4c0b8b90 2023-03-03T08:02:14.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.707+0000 7f4c51bff700 1 -- 172.21.15.98:0/1193303651 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4c38068380 msgr2=0x7f4c3806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:14.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.707+0000 7f4c51bff700 1 --2- 172.21.15.98:0/1193303651 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4c38068380 0x7f4c3806a830 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c000f60 tx=0x7f4c3c007be0 comp rx=0 tx=0).stop 2023-03-03T08:02:14.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.708+0000 7f4c51bff700 1 -- 172.21.15.98:0/1193303651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0b8b90 msgr2=0x7f4c4c0ab660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:14.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.708+0000 7f4c51bff700 1 --2- 172.21.15.98:0/1193303651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0b8b90 0x7f4c4c0ab660 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f4c44014040 tx=0x7f4c44004ad0 comp rx=0 tx=0).stop 2023-03-03T08:02:14.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.708+0000 7f4c51bff700 1 -- 172.21.15.98:0/1193303651 shutdown_connections 2023-03-03T08:02:14.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.708+0000 7f4c51bff700 1 --2- 172.21.15.98:0/1193303651 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4c38068380 0x7f4c3806a830 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:14.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.708+0000 7f4c51bff700 1 --2- 172.21.15.98:0/1193303651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0b8b90 0x7f4c4c0ab660 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:14.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.708+0000 7f4c51bff700 1 --2- 172.21.15.98:0/1193303651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c4c0b5410 0x7f4c4c15c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:14.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.708+0000 7f4c51bff700 1 --2- 172.21.15.98:0/1193303651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c4c0b87c0 0x7f4c4c15e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:14.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.708+0000 7f4c51bff700 1 -- 172.21.15.98:0/1193303651 >> 172.21.15.98:0/1193303651 conn(0x7f4c4c06aee0 msgr2=0x7f4c4c0d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:14.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.708+0000 7f4c51bff700 1 -- 172.21.15.98:0/1193303651 shutdown_connections 2023-03-03T08:02:14.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.708+0000 7f4c51bff700 1 -- 172.21.15.98:0/1193303651 wait complete. 2023-03-03T08:02:14.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-03-03T08:02:14.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.981+0000 7fda19b9d700 1 Processor -- start 2023-03-03T08:02:14.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.982+0000 7fda19b9d700 1 -- start start 2023-03-03T08:02:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.983+0000 7fda19b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda141609b0 0x7fda14160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.983+0000 7fda19b9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fda140cf570 0x7fda1415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.983+0000 7fda19b9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda141612c0 con 0x7fda1415db30 2023-03-03T08:02:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.983+0000 7fda19b9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda140cebb0 con 0x7fda140cf570 2023-03-03T08:02:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.983+0000 7fda19b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda140cd780 con 0x7fda141609b0 2023-03-03T08:02:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.983+0000 7fda13fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fda140cf570 0x7fda1415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.983+0000 7fda13fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fda140cf570 0x7fda1415d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48086/0 (socket says 172.21.15.98:48086) 2023-03-03T08:02:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.984+0000 7fda13fff700 1 -- 172.21.15.98:0/1947999080 learned_addr learned my addr 172.21.15.98:0/1947999080 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.984+0000 7fda18b9b700 1 --2- 172.21.15.98:0/1947999080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda141609b0 0x7fda14160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:14.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.984+0000 7fda13fff700 1 -- 172.21.15.98:0/1947999080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda141609b0 msgr2=0x7fda14160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.984+0000 7fda13fff700 1 --2- 172.21.15.98:0/1947999080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda141609b0 0x7fda14160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.984+0000 7fda13fff700 1 -- 172.21.15.98:0/1947999080 >> v1:172.21.15.100:6789/0 conn(0x7fda1415db30 legacy=0x7fda140b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.984+0000 7fda13fff700 1 -- 172.21.15.98:0/1947999080 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda140cda60 con 0x7fda140cf570 2023-03-03T08:02:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.984+0000 7fda137fe700 1 -- 172.21.15.98:0/1947999080 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417252417 0 0) 0x7fda141612c0 con 0x7fda1415db30 2023-03-03T08:02:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.984+0000 7fda13fff700 1 --2- 172.21.15.98:0/1947999080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda140cf570 0x7fda1415d5f0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fda000098f0 tx=0x7fda00009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b33c486f28808220 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.985+0000 7fda137fe700 1 -- 172.21.15.98:0/1947999080 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda00011830 con 0x7fda140cf570 2023-03-03T08:02:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.985+0000 7fda137fe700 1 -- 172.21.15.98:0/1947999080 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda00011990 con 0x7fda140cf570 2023-03-03T08:02:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.985+0000 7fda137fe700 1 -- 172.21.15.98:0/1947999080 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda00011d20 con 0x7fda140cf570 2023-03-03T08:02:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.985+0000 7fda19b9d700 1 -- 172.21.15.98:0/1947999080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda140cf570 msgr2=0x7fda1415d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.985+0000 7fda19b9d700 1 --2- 172.21.15.98:0/1947999080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda140cf570 0x7fda1415d5f0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fda000098f0 tx=0x7fda00009c00 comp rx=0 tx=0).stop 2023-03-03T08:02:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.985+0000 7fda19b9d700 1 -- 172.21.15.98:0/1947999080 shutdown_connections 2023-03-03T08:02:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.985+0000 7fda19b9d700 1 --2- 172.21.15.98:0/1947999080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda141609b0 0x7fda14160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.985+0000 7fda19b9d700 1 --2- 172.21.15.98:0/1947999080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda140cf570 0x7fda1415d5f0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.985+0000 7fda19b9d700 1 -- 172.21.15.98:0/1947999080 >> 172.21.15.98:0/1947999080 conn(0x7fda1406b880 msgr2=0x7fda140cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:14.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.985+0000 7fda19b9d700 1 -- 172.21.15.98:0/1947999080 shutdown_connections 2023-03-03T08:02:14.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.986+0000 7fda19b9d700 1 -- 172.21.15.98:0/1947999080 wait complete. 2023-03-03T08:02:14.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.987+0000 7fda19b9d700 1 Processor -- start 2023-03-03T08:02:14.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.988+0000 7fda19b9d700 1 -- start start 2023-03-03T08:02:14.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda19b9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda140cf570 0x7fda1414c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:14.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda19b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda1415db30 0x7fda1415e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:14.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda19b9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda141609b0 0x7fda1415ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:14.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda19b9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1414cb20 con 0x7fda140cf570 2023-03-03T08:02:14.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda19b9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda140c8760 con 0x7fda141609b0 2023-03-03T08:02:14.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda19b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda140c8a40 con 0x7fda1415db30 2023-03-03T08:02:14.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda18b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda140cf570 0x7fda1414c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:14.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda18b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda140cf570 0x7fda1414c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48102/0 (socket says 172.21.15.98:48102) 2023-03-03T08:02:14.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda18b9b700 1 -- 172.21.15.98:0/407573186 learned_addr learned my addr 172.21.15.98:0/407573186 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:14.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda18b9b700 1 -- 172.21.15.98:0/407573186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda1415db30 msgr2=0x7fda1415e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:14.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda18b9b700 1 --2- 172.21.15.98:0/407573186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda1415db30 0x7fda1415e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:14.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda18b9b700 1 -- 172.21.15.98:0/407573186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda141609b0 msgr2=0x7fda1415ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:14.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda1939c700 1 --2- 172.21.15.98:0/407573186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda141609b0 0x7fda1415ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:14.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda18b9b700 1 --2- 172.21.15.98:0/407573186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda141609b0 0x7fda1415ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:14.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda18b9b700 1 -- 172.21.15.98:0/407573186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda00009550 con 0x7fda140cf570 2023-03-03T08:02:14.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.989+0000 7fda1939c700 1 --2- 172.21.15.98:0/407573186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda141609b0 0x7fda1415ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:14.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.990+0000 7fda18b9b700 1 --2- 172.21.15.98:0/407573186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda140cf570 0x7fda1414c5e0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fda0800e870 tx=0x7fda0800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:14.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.990+0000 7fda117fa700 1 -- 172.21.15.98:0/407573186 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0800ee70 con 0x7fda140cf570 2023-03-03T08:02:14.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.990+0000 7fda117fa700 1 -- 172.21.15.98:0/407573186 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0800c850 con 0x7fda140cf570 2023-03-03T08:02:14.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.990+0000 7fda117fa700 1 -- 172.21.15.98:0/407573186 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0800cbc0 con 0x7fda140cf570 2023-03-03T08:02:14.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.990+0000 7fda19b9d700 1 -- 172.21.15.98:0/407573186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda1415cdf0 con 0x7fda140cf570 2023-03-03T08:02:14.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.990+0000 7fda19b9d700 1 -- 172.21.15.98:0/407573186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda140cdd40 con 0x7fda140cf570 2023-03-03T08:02:14.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.991+0000 7fda19b9d700 1 -- 172.21.15.98:0/407573186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda1415d0d0 con 0x7fda140cf570 2023-03-03T08:02:14.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.992+0000 7fda117fa700 1 -- 172.21.15.98:0/407573186 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fda08019030 con 0x7fda140cf570 2023-03-03T08:02:14.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.994+0000 7fda117fa700 1 --2- 172.21.15.98:0/407573186 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd9f8068380 0x7fd9f806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:14.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.994+0000 7fda117fa700 1 -- 172.21.15.98:0/407573186 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5855+0+0 (secure 0 0 0) 0x7fda08085b70 con 0x7fda140cf570 2023-03-03T08:02:14.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.995+0000 7fda13fff700 1 --2- 172.21.15.98:0/407573186 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd9f8068380 0x7fd9f806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:14.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.996+0000 7fda13fff700 1 --2- 172.21.15.98:0/407573186 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd9f8068380 0x7fd9f806a830 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fda140d0da0 tx=0x7fda00010000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:14.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:14.998+0000 7fda117fa700 1 -- 172.21.15.98:0/407573186 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda080523c0 con 0x7fda140cf570 2023-03-03T08:02:15.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:15.190+0000 7fda19b9d700 1 -- 172.21.15.98:0/407573186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fda14068290 con 0x7fda140cf570 2023-03-03T08:02:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:15.195+0000 7fda117fa700 1 -- 172.21.15.98:0/407573186 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7fda08084120 con 0x7fda140cf570 2023-03-03T08:02:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:15.198+0000 7fda19b9d700 1 -- 172.21.15.98:0/407573186 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd9f8068380 msgr2=0x7fd9f806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:15.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:15.198+0000 7fda19b9d700 1 --2- 172.21.15.98:0/407573186 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd9f8068380 0x7fd9f806a830 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fda140d0da0 tx=0x7fda00010000 comp rx=0 tx=0).stop 2023-03-03T08:02:15.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:15.198+0000 7fda19b9d700 1 -- 172.21.15.98:0/407573186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda140cf570 msgr2=0x7fda1414c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:15.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:15.198+0000 7fda19b9d700 1 --2- 172.21.15.98:0/407573186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda140cf570 0x7fda1414c5e0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fda0800e870 tx=0x7fda0800ec30 comp rx=0 tx=0).stop 2023-03-03T08:02:15.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:15.198+0000 7fda19b9d700 1 -- 172.21.15.98:0/407573186 shutdown_connections 2023-03-03T08:02:15.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:15.198+0000 7fda19b9d700 1 --2- 172.21.15.98:0/407573186 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd9f8068380 0x7fd9f806a830 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:15.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:15.198+0000 7fda19b9d700 1 --2- 172.21.15.98:0/407573186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda140cf570 0x7fda1414c5e0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:15.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:15.198+0000 7fda19b9d700 1 --2- 172.21.15.98:0/407573186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda141609b0 0x7fda1415ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:15.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:15.198+0000 7fda19b9d700 1 --2- 172.21.15.98:0/407573186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda1415db30 0x7fda1415e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:15.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:15.198+0000 7fda19b9d700 1 -- 172.21.15.98:0/407573186 >> 172.21.15.98:0/407573186 conn(0x7fda1406b880 msgr2=0x7fda140d1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:15.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:15.198+0000 7fda19b9d700 1 -- 172.21.15.98:0/407573186 shutdown_connections 2023-03-03T08:02:15.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:15.198+0000 7fda19b9d700 1 -- 172.21.15.98:0/407573186 wait complete. 2023-03-03T08:02:15.226 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-03-03T08:02:15.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:02:15.357 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:02:15.494 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-03-03T08:02:15.494 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:02:15.495 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:02:15.563 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T08:02:15.563 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:02:15.564 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:02:15.632 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T08:02:15.633 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:02:15.633 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:02:15.700 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T08:02:15.701 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:02:15.701 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:02:15.770 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T08:02:15.771 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-03-03T08:02:15.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:02:16.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.013+0000 7f15bb59e700 1 Processor -- start 2023-03-03T08:02:16.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.013+0000 7f15bb59e700 1 -- start start 2023-03-03T08:02:16.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.015+0000 7f15bb59e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15bc146360 0x7f15bc160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:16.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.015+0000 7f15bb59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15bc0b4ab0 0x7f15bc0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:16.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.015+0000 7f15bb59e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f15bc0731e0 0x7f15bc073660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:16.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.015+0000 7f15bb59e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15bc0b5460 con 0x7f15bc0b4ab0 2023-03-03T08:02:16.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.015+0000 7f15bb59e700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15bc1614c0 con 0x7f15bc0731e0 2023-03-03T08:02:16.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.015+0000 7f15bb59e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc0d10f0 con 0x7f15bc146360 2023-03-03T08:02:16.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.015+0000 7f15b9d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15bc0b4ab0 0x7f15bc0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:16.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.015+0000 7f15b9d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15bc0b4ab0 0x7f15bc0b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48114/0 (socket says 172.21.15.98:48114) 2023-03-03T08:02:16.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.015+0000 7f15b9d9b700 1 -- 172.21.15.98:0/4089851239 learned_addr learned my addr 172.21.15.98:0/4089851239 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:16.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.015+0000 7f15bad9d700 1 --2- 172.21.15.98:0/4089851239 >> v2:172.21.15.100:3300/0 conn(0x7f15bc0731e0 0x7f15bc073660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:16.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.015+0000 7f15ba59c700 1 --2- 172.21.15.98:0/4089851239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15bc146360 0x7f15bc160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:16.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.016+0000 7f15b9d9b700 1 -- 172.21.15.98:0/4089851239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15bc146360 msgr2=0x7f15bc160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:16.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.016+0000 7f15b9d9b700 1 --2- 172.21.15.98:0/4089851239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15bc146360 0x7f15bc160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:16.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.016+0000 7f15b9d9b700 1 -- 172.21.15.98:0/4089851239 >> v2:172.21.15.100:3300/0 conn(0x7f15bc0731e0 msgr2=0x7f15bc073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:16.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.016+0000 7f15b9d9b700 1 --2- 172.21.15.98:0/4089851239 >> v2:172.21.15.100:3300/0 conn(0x7f15bc0731e0 0x7f15bc073660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:16.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.016+0000 7f15b9d9b700 1 -- 172.21.15.98:0/4089851239 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15bc0cdf20 con 0x7f15bc0b4ab0 2023-03-03T08:02:16.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.016+0000 7f15ba59c700 1 --2- 172.21.15.98:0/4089851239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15bc146360 0x7f15bc160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:16.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.016+0000 7f15b9d9b700 1 --2- 172.21.15.98:0/4089851239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15bc0b4ab0 0x7f15bc0b4f20 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f15bc0743c0 tx=0x7f15b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902e856afbbab15e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:16.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.016+0000 7f15b959a700 1 -- 172.21.15.98:0/4089851239 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b0011830 con 0x7f15bc0b4ab0 2023-03-03T08:02:16.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.016+0000 7f15b959a700 1 -- 172.21.15.98:0/4089851239 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b0011990 con 0x7f15bc0b4ab0 2023-03-03T08:02:16.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.017+0000 7f15b959a700 1 -- 172.21.15.98:0/4089851239 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b0011d20 con 0x7f15bc0b4ab0 2023-03-03T08:02:16.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.017+0000 7f15bb59e700 1 -- 172.21.15.98:0/4089851239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15bc0b4ab0 msgr2=0x7f15bc0b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:16.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.017+0000 7f15bb59e700 1 --2- 172.21.15.98:0/4089851239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15bc0b4ab0 0x7f15bc0b4f20 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f15bc0743c0 tx=0x7f15b0009bd0 comp rx=0 tx=0).stop 2023-03-03T08:02:16.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.017+0000 7f15bb59e700 1 -- 172.21.15.98:0/4089851239 shutdown_connections 2023-03-03T08:02:16.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.017+0000 7f15bb59e700 1 --2- 172.21.15.98:0/4089851239 >> v2:172.21.15.100:3300/0 conn(0x7f15bc0731e0 0x7f15bc073660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:16.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.017+0000 7f15bb59e700 1 --2- 172.21.15.98:0/4089851239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15bc146360 0x7f15bc160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:16.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.017+0000 7f15bb59e700 1 --2- 172.21.15.98:0/4089851239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15bc0b4ab0 0x7f15bc0b4f20 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:16.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.017+0000 7f15bb59e700 1 -- 172.21.15.98:0/4089851239 >> 172.21.15.98:0/4089851239 conn(0x7f15bc06b8e0 msgr2=0x7f15bc0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:16.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.017+0000 7f15bb59e700 1 -- 172.21.15.98:0/4089851239 shutdown_connections 2023-03-03T08:02:16.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.017+0000 7f15bb59e700 1 -- 172.21.15.98:0/4089851239 wait complete. 2023-03-03T08:02:16.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.019+0000 7f15bb59e700 1 Processor -- start 2023-03-03T08:02:16.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.020+0000 7f15bb59e700 1 -- start start 2023-03-03T08:02:16.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15bb59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15bc0731e0 0x7f15bc0caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:16.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15bb59e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15bc0b4ab0 0x7f15bc0ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:16.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15bb59e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15bc146360 0x7f15bc1f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:16.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15bb59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc0d1bc0 con 0x7f15bc0731e0 2023-03-03T08:02:16.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15bb59e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc15ced0 con 0x7f15bc146360 2023-03-03T08:02:16.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15bb59e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc15d1b0 con 0x7f15bc0b4ab0 2023-03-03T08:02:16.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15ba59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15bc0731e0 0x7f15bc0caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:16.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15ba59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15bc0731e0 0x7f15bc0caf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48116/0 (socket says 172.21.15.98:48116) 2023-03-03T08:02:16.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15ba59c700 1 -- 172.21.15.98:0/2989593267 learned_addr learned my addr 172.21.15.98:0/2989593267 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:16.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15bad9d700 1 --2- 172.21.15.98:0/2989593267 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15bc146360 0x7f15bc1f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:16.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15b9d9b700 1 --2- 172.21.15.98:0/2989593267 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15bc0b4ab0 0x7f15bc0ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:16.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15ba59c700 1 -- 172.21.15.98:0/2989593267 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15bc0b4ab0 msgr2=0x7f15bc0ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:16.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15ba59c700 1 --2- 172.21.15.98:0/2989593267 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15bc0b4ab0 0x7f15bc0ccb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:16.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15ba59c700 1 -- 172.21.15.98:0/2989593267 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15bc146360 msgr2=0x7f15bc1f4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:16.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15ba59c700 1 --2- 172.21.15.98:0/2989593267 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15bc146360 0x7f15bc1f4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:16.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.021+0000 7f15ba59c700 1 -- 172.21.15.98:0/2989593267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b0009550 con 0x7f15bc0731e0 2023-03-03T08:02:16.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.022+0000 7f15ba59c700 1 --2- 172.21.15.98:0/2989593267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15bc0731e0 0x7f15bc0caf00 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f15ac00e810 tx=0x7f15ac00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:16.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.022+0000 7f15aaffd700 1 -- 172.21.15.98:0/2989593267 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac013610 con 0x7f15bc0731e0 2023-03-03T08:02:16.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.022+0000 7f15aaffd700 1 -- 172.21.15.98:0/2989593267 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15ac013770 con 0x7f15bc0731e0 2023-03-03T08:02:16.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.022+0000 7f15aaffd700 1 -- 172.21.15.98:0/2989593267 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac013b20 con 0x7f15bc0731e0 2023-03-03T08:02:16.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.022+0000 7f15bb59e700 1 -- 172.21.15.98:0/2989593267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15bc0cd960 con 0x7f15bc0731e0 2023-03-03T08:02:16.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.022+0000 7f15bb59e700 1 -- 172.21.15.98:0/2989593267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15bc0cd680 con 0x7f15bc0731e0 2023-03-03T08:02:16.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.023+0000 7f15bb59e700 1 -- 172.21.15.98:0/2989593267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15bc0cdc40 con 0x7f15bc0731e0 2023-03-03T08:02:16.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.024+0000 7f15aaffd700 1 -- 172.21.15.98:0/2989593267 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f15ac023070 con 0x7f15bc0731e0 2023-03-03T08:02:16.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.025+0000 7f15aaffd700 1 --2- 172.21.15.98:0/2989593267 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f159c068380 0x7f159c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.026+0000 7f15aaffd700 1 -- 172.21.15.98:0/2989593267 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5855+0+0 (secure 0 0 0) 0x7f15ac085880 con 0x7f15bc0731e0 2023-03-03T08:02:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.026+0000 7f15b9d9b700 1 --2- 172.21.15.98:0/2989593267 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f159c068380 0x7f159c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.027+0000 7f15b9d9b700 1 --2- 172.21.15.98:0/2989593267 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f159c068380 0x7f159c06a830 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f15b0014040 tx=0x7f15b0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:16.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.029+0000 7f15aaffd700 1 -- 172.21.15.98:0/2989593267 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f15ac0520d0 con 0x7f15bc0731e0 2023-03-03T08:02:16.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.264+0000 7f15bb59e700 1 -- 172.21.15.98:0/2989593267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15bc15ddc0 con 0x7f15bc0731e0 2023-03-03T08:02:16.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.265+0000 7f15aaffd700 1 -- 172.21.15.98:0/2989593267 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+3532 (secure 0 0 0) 0x7f15ac021de0 con 0x7f15bc0731e0 2023-03-03T08:02:16.266 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:16.267 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":43,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6050,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/318891906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":318891906},{"type":"v1","addr":"172.21.15.98:6836","nonce":318891906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1280578660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1280578660},{"type":"v1","addr":"172.21.15.98:6837","nonce":1280578660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6906,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1395511883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1395511883},{"type":"v1","addr":"172.21.15.100:6834","nonce":1395511883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":7189,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3513389583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":3513389583},{"type":"v1","addr":"172.21.15.100:6835","nonce":3513389583}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[]} 2023-03-03T08:02:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.267+0000 7f15bb59e700 1 -- 172.21.15.98:0/2989593267 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f159c068380 msgr2=0x7f159c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.267+0000 7f15bb59e700 1 --2- 172.21.15.98:0/2989593267 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f159c068380 0x7f159c06a830 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f15b0014040 tx=0x7f15b0008030 comp rx=0 tx=0).stop 2023-03-03T08:02:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.267+0000 7f15bb59e700 1 -- 172.21.15.98:0/2989593267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15bc0731e0 msgr2=0x7f15bc0caf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.267+0000 7f15bb59e700 1 --2- 172.21.15.98:0/2989593267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15bc0731e0 0x7f15bc0caf00 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f15ac00e810 tx=0x7f15ac00eb20 comp rx=0 tx=0).stop 2023-03-03T08:02:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.268+0000 7f15bb59e700 1 -- 172.21.15.98:0/2989593267 shutdown_connections 2023-03-03T08:02:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.268+0000 7f15bb59e700 1 --2- 172.21.15.98:0/2989593267 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f159c068380 0x7f159c06a830 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.268+0000 7f15bb59e700 1 --2- 172.21.15.98:0/2989593267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15bc0731e0 0x7f15bc0caf00 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.268+0000 7f15bb59e700 1 --2- 172.21.15.98:0/2989593267 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15bc146360 0x7f15bc1f4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.268+0000 7f15bb59e700 1 --2- 172.21.15.98:0/2989593267 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f15bc0b4ab0 0x7f15bc0ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.268+0000 7f15bb59e700 1 -- 172.21.15.98:0/2989593267 >> 172.21.15.98:0/2989593267 conn(0x7f15bc06b8e0 msgr2=0x7f15bc0b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.268+0000 7f15bb59e700 1 -- 172.21.15.98:0/2989593267 shutdown_connections 2023-03-03T08:02:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.268+0000 7f15bb59e700 1 -- 172.21.15.98:0/2989593267 wait complete. 2023-03-03T08:02:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 43 2023-03-03T08:02:16.296 INFO:tasks.ceph.mds.a:Restarting daemon 2023-03-03T08:02:16.296 INFO:tasks.ceph.mds.a:Stopping old one... 2023-03-03T08:02:16.297 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-03-03T08:02:16.297 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:02:16.297 INFO:tasks.ceph.mds.c:Restarting daemon 2023-03-03T08:02:16.297 INFO:tasks.ceph.mds.c:Stopping old one... 2023-03-03T08:02:16.298 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-03-03T08:02:16.298 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:02:16.298 INFO:tasks.ceph.mds.b:Restarting daemon 2023-03-03T08:02:16.298 INFO:tasks.ceph.mds.b:Stopping old one... 2023-03-03T08:02:16.298 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-03-03T08:02:16.299 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:02:16.299 INFO:tasks.ceph.mds.d:Restarting daemon 2023-03-03T08:02:16.299 INFO:tasks.ceph.mds.d:Stopping old one... 2023-03-03T08:02:16.299 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-03-03T08:02:16.300 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:02:16.318 INFO:tasks.ceph.mds.a:Stopped 2023-03-03T08:02:16.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-03-03T08:02:16.320 INFO:tasks.ceph.mds.c:Stopped 2023-03-03T08:02:16.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-03-03T08:02:16.323 INFO:tasks.ceph.mds.b:Stopped 2023-03-03T08:02:16.323 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-03-03T08:02:16.324 INFO:tasks.ceph.mds.d:Stopped 2023-03-03T08:02:16.325 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-03-03T08:02:16.365 INFO:tasks.ceph.mds.a:Started 2023-03-03T08:02:16.366 DEBUG:teuthology.parallel:result is None 2023-03-03T08:02:16.370 INFO:tasks.ceph.mds.c:Started 2023-03-03T08:02:16.370 DEBUG:teuthology.parallel:result is None 2023-03-03T08:02:16.371 INFO:tasks.ceph.mds.b:Started 2023-03-03T08:02:16.371 DEBUG:teuthology.parallel:result is None 2023-03-03T08:02:16.375 INFO:tasks.ceph.mds.d:Started 2023-03-03T08:02:16.376 DEBUG:teuthology.parallel:result is None 2023-03-03T08:02:16.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-03-03T08:02:16.562 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2023-03-03T08:02:16.562 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-03-03T08:02:16.566 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2023-03-03T08:02:16.567 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2023-03-03T08:02:16.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.641+0000 7f50057ff700 1 Processor -- start 2023-03-03T08:02:16.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.641+0000 7f50057ff700 1 -- start start 2023-03-03T08:02:16.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.643+0000 7f50057ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500015ca70 0x7f50000cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:16.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.643+0000 7f50057ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50000cf980 con 0x7f500006b4b0 2023-03-03T08:02:16.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.643+0000 7f50057ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50000d1800 con 0x7f50000d0270 2023-03-03T08:02:16.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.643+0000 7f50057ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50000cd5b0 con 0x7f500015ca70 2023-03-03T08:02:16.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.643+0000 7f4ffe7fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f500006b4b0 0x7f50000cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58668/0 (socket says 172.21.15.98:58668) 2023-03-03T08:02:16.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.643+0000 7f4ffe7fc700 1 -- 172.21.15.98:0/2327078205 learned_addr learned my addr 172.21.15.98:0/2327078205 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:16.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.643+0000 7f4ffeffd700 1 --2- 172.21.15.98:0/2327078205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500015ca70 0x7f50000cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:16.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.644+0000 7f4ffdffb700 1 -- 172.21.15.98:0/2327078205 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579337079 0 0) 0x7f50000cf980 con 0x7f500006b4b0 2023-03-03T08:02:16.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.644+0000 7f4ffdffb700 1 -- 172.21.15.98:0/2327078205 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe80035b0 con 0x7f500006b4b0 2023-03-03T08:02:16.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.644+0000 7f4ffdffb700 1 -- 172.21.15.98:0/2327078205 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246721213 0 0) 0x7f50000d1800 con 0x7f50000d0270 2023-03-03T08:02:16.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.644+0000 7f4ffdffb700 1 -- 172.21.15.98:0/2327078205 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50000cf980 con 0x7f50000d0270 2023-03-03T08:02:16.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.644+0000 7f4ffeffd700 1 -- 172.21.15.98:0/2327078205 >> v1:172.21.15.100:6789/0 conn(0x7f50000d0270 legacy=0x7f50000d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:16.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.644+0000 7f4ffeffd700 1 -- 172.21.15.98:0/2327078205 >> v1:172.21.15.98:6789/0 conn(0x7f500006b4b0 legacy=0x7f50000cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:16.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.644+0000 7f4ffeffd700 1 -- 172.21.15.98:0/2327078205 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50000cd900 con 0x7f500015ca70 2023-03-03T08:02:16.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.644+0000 7f4ffeffd700 1 --2- 172.21.15.98:0/2327078205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500015ca70 0x7f50000cf440 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009dc0 tx=0x7f4ff000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64bcdfb0c511aa18 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:16.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.644+0000 7f4ffdffb700 1 -- 172.21.15.98:0/2327078205 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff000e070 con 0x7f500015ca70 2023-03-03T08:02:16.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.645+0000 7f4ffdffb700 1 -- 172.21.15.98:0/2327078205 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff000bb20 con 0x7f500015ca70 2023-03-03T08:02:16.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.645+0000 7f4ffdffb700 1 -- 172.21.15.98:0/2327078205 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0012430 con 0x7f500015ca70 2023-03-03T08:02:16.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.645+0000 7f50057ff700 1 -- 172.21.15.98:0/2327078205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500015ca70 msgr2=0x7f50000cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:16.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.645+0000 7f50057ff700 1 --2- 172.21.15.98:0/2327078205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500015ca70 0x7f50000cf440 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009dc0 tx=0x7f4ff000b330 comp rx=0 tx=0).stop 2023-03-03T08:02:16.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.645+0000 7f50057ff700 1 -- 172.21.15.98:0/2327078205 shutdown_connections 2023-03-03T08:02:16.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.645+0000 7f50057ff700 1 --2- 172.21.15.98:0/2327078205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500015ca70 0x7f50000cf440 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:16.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.645+0000 7f50057ff700 1 -- 172.21.15.98:0/2327078205 >> 172.21.15.98:0/2327078205 conn(0x7f5000073e70 msgr2=0x7f500006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:16.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.645+0000 7f50057ff700 1 -- 172.21.15.98:0/2327078205 shutdown_connections 2023-03-03T08:02:16.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.645+0000 7f50057ff700 1 -- 172.21.15.98:0/2327078205 wait complete. 2023-03-03T08:02:16.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.647+0000 7f50057ff700 1 Processor -- start 2023-03-03T08:02:16.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.648+0000 7f50057ff700 1 -- start start 2023-03-03T08:02:16.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.648+0000 7f50057ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500006b4b0 0x7f50000ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:16.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f50057ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50000d0270 0x7f50000b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:16.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f50057ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f500015ca70 0x7f50000b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:16.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f50057ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50000b5b00 con 0x7f50000d0270 2023-03-03T08:02:16.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f50057ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50000b4520 con 0x7f500015ca70 2023-03-03T08:02:16.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f50057ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50000b4800 con 0x7f500006b4b0 2023-03-03T08:02:16.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f4ffe7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50000d0270 0x7f50000b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:16.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f4ffeffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500006b4b0 0x7f50000ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:16.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f4ffeffd700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500006b4b0 0x7f50000ccfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:48416/0 (socket says 172.21.15.98:48416) 2023-03-03T08:02:16.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f4ffeffd700 1 -- 172.21.15.98:0/1230137331 learned_addr learned my addr 172.21.15.98:0/1230137331 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:16.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f4ffe7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50000d0270 0x7f50000b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48150/0 (socket says 172.21.15.98:48150) 2023-03-03T08:02:16.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f4fff7fe700 1 --2- 172.21.15.98:0/1230137331 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f500015ca70 0x7f50000b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:16.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f4ffeffd700 1 -- 172.21.15.98:0/1230137331 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f500015ca70 msgr2=0x7f50000b55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:16.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f4ffeffd700 1 --2- 172.21.15.98:0/1230137331 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f500015ca70 0x7f50000b55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:16.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f4ffeffd700 1 -- 172.21.15.98:0/1230137331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50000d0270 msgr2=0x7f50000b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:16.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f4ffeffd700 1 --2- 172.21.15.98:0/1230137331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50000d0270 0x7f50000b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:16.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f4ffeffd700 1 -- 172.21.15.98:0/1230137331 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff0009a70 con 0x7f500006b4b0 2023-03-03T08:02:16.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.649+0000 7f4ffe7fc700 1 --2- 172.21.15.98:0/1230137331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50000d0270 0x7f50000b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:16.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.650+0000 7f4ffeffd700 1 --2- 172.21.15.98:0/1230137331 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500006b4b0 0x7f50000ccfc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f50000b5d70 tx=0x7f4ff0004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:16.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.650+0000 7f4fe77fe700 1 -- 172.21.15.98:0/1230137331 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0012430 con 0x7f500006b4b0 2023-03-03T08:02:16.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.650+0000 7f4fe77fe700 1 -- 172.21.15.98:0/1230137331 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff0020070 con 0x7f500006b4b0 2023-03-03T08:02:16.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.650+0000 7f50057ff700 1 -- 172.21.15.98:0/1230137331 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50000d1aa0 con 0x7f500006b4b0 2023-03-03T08:02:16.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.650+0000 7f4fe77fe700 1 -- 172.21.15.98:0/1230137331 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff001ca70 con 0x7f500006b4b0 2023-03-03T08:02:16.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.650+0000 7f50057ff700 1 -- 172.21.15.98:0/1230137331 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50000cdbe0 con 0x7f500006b4b0 2023-03-03T08:02:16.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.651+0000 7f50057ff700 1 -- 172.21.15.98:0/1230137331 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500015ce40 con 0x7f500006b4b0 2023-03-03T08:02:16.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.653+0000 7f4fe77fe700 1 -- 172.21.15.98:0/1230137331 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f4ff001cbd0 con 0x7f500006b4b0 2023-03-03T08:02:16.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.654+0000 7f4fe77fe700 1 --2- 172.21.15.98:0/1230137331 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4fec068380 0x7f4fec06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:16.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.654+0000 7f4fe77fe700 1 -- 172.21.15.98:0/1230137331 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5855+0+0 (secure 0 0 0) 0x7f4ff0022070 con 0x7f500006b4b0 2023-03-03T08:02:16.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.654+0000 7f4ffe7fc700 1 --2- 172.21.15.98:0/1230137331 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4fec068380 0x7f4fec06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:16.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.655+0000 7f4ffe7fc700 1 --2- 172.21.15.98:0/1230137331 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4fec068380 0x7f4fec06a830 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f500015fae0 tx=0x7f4ff4007e00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:16.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.657+0000 7f4fe77fe700 1 -- 172.21.15.98:0/1230137331 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ff0054d00 con 0x7f500006b4b0 2023-03-03T08:02:16.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:16.869+0000 7f50057ff700 1 -- 172.21.15.98:0/1230137331 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f50000b4940 con 0x7f500006b4b0 2023-03-03T08:02:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:17.730+0000 7f4fe77fe700 1 -- 172.21.15.98:0/1230137331 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v87) v1 ==== 97+0+0 (secure 0 0 0) 0x7f50000b4940 con 0x7f500006b4b0 2023-03-03T08:02:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:17.733+0000 7f50057ff700 1 -- 172.21.15.98:0/1230137331 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4fec068380 msgr2=0x7f4fec06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:17.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:17.733+0000 7f50057ff700 1 --2- 172.21.15.98:0/1230137331 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4fec068380 0x7f4fec06a830 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f500015fae0 tx=0x7f4ff4007e00 comp rx=0 tx=0).stop 2023-03-03T08:02:17.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:17.733+0000 7f50057ff700 1 -- 172.21.15.98:0/1230137331 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500006b4b0 msgr2=0x7f50000ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:17.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:17.733+0000 7f50057ff700 1 --2- 172.21.15.98:0/1230137331 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500006b4b0 0x7f50000ccfc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f50000b5d70 tx=0x7f4ff0004ad0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:17.733+0000 7f50057ff700 1 -- 172.21.15.98:0/1230137331 shutdown_connections 2023-03-03T08:02:17.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:17.733+0000 7f50057ff700 1 --2- 172.21.15.98:0/1230137331 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f500015ca70 0x7f50000b55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:17.733+0000 7f50057ff700 1 --2- 172.21.15.98:0/1230137331 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500006b4b0 0x7f50000ccfc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:17.733+0000 7f50057ff700 1 --2- 172.21.15.98:0/1230137331 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4fec068380 0x7f4fec06a830 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:17.733+0000 7f50057ff700 1 --2- 172.21.15.98:0/1230137331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50000d0270 0x7f50000b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:17.733+0000 7f50057ff700 1 -- 172.21.15.98:0/1230137331 >> 172.21.15.98:0/1230137331 conn(0x7f5000073e70 msgr2=0x7f500015bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:17.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:17.733+0000 7f50057ff700 1 -- 172.21.15.98:0/1230137331 shutdown_connections 2023-03-03T08:02:17.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:17.733+0000 7f50057ff700 1 -- 172.21.15.98:0/1230137331 wait complete. 2023-03-03T08:02:17.736 INFO:teuthology.orchestra.run.smithi098.stderr: removed all blocklist entries 2023-03-03T08:02:17.764 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-03-03T08:02:18.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.010+0000 7f3afbdff700 1 Processor -- start 2023-03-03T08:02:18.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.010+0000 7f3afbdff700 1 -- start start 2023-03-03T08:02:18.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.011+0000 7f3afbdff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3af40b87c0 0x7f3af40cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:18.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.011+0000 7f3afbdff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af40d0ef0 con 0x7f3af40b5410 2023-03-03T08:02:18.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.011+0000 7f3afbdff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af40d0600 con 0x7f3af40b8b90 2023-03-03T08:02:18.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.011+0000 7f3afbdff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af40cd970 con 0x7f3af40b87c0 2023-03-03T08:02:18.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.012+0000 7f3afa39c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3af40b5410 0x7f3af40d13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58682/0 (socket says 172.21.15.98:58682) 2023-03-03T08:02:18.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.012+0000 7f3afa39c700 1 -- 172.21.15.98:0/1039709161 learned_addr learned my addr 172.21.15.98:0/1039709161 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:18.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.012+0000 7f3af9b9b700 1 --2- 172.21.15.98:0/1039709161 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3af40b87c0 0x7f3af40cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:18.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.012+0000 7f3af8b99700 1 -- 172.21.15.98:0/1039709161 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447621734 0 0) 0x7f3af40d0ef0 con 0x7f3af40b5410 2023-03-03T08:02:18.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.012+0000 7f3af8b99700 1 -- 172.21.15.98:0/1039709161 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ae00035b0 con 0x7f3af40b5410 2023-03-03T08:02:18.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3af8b99700 1 -- 172.21.15.98:0/1039709161 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 438697101 0 0) 0x7f3ae00035b0 con 0x7f3af40b5410 2023-03-03T08:02:18.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3af8b99700 1 -- 172.21.15.98:0/1039709161 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3af40b87c0 msgr2=0x7f3af40cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:18.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3af8b99700 1 --2- 172.21.15.98:0/1039709161 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3af40b87c0 0x7f3af40cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3af8b99700 1 -- 172.21.15.98:0/1039709161 >> v1:172.21.15.100:6789/0 conn(0x7f3af40b8b90 legacy=0x7f3af40b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:18.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3af8b99700 1 -- 172.21.15.98:0/1039709161 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af40cdc50 con 0x7f3af40b5410 2023-03-03T08:02:18.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3af8b99700 1 -- 172.21.15.98:0/1039709161 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f3aec0019f0 con 0x7f3af40b5410 2023-03-03T08:02:18.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3af8b99700 1 -- 172.21.15.98:0/1039709161 >> v1:172.21.15.98:6789/0 conn(0x7f3af40b5410 legacy=0x7f3af40d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:18.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3af8b99700 1 --2- 172.21.15.98:0/1039709161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae0003a30 0x7f3ae0003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:18.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3af8b99700 1 --2- 172.21.15.98:0/1039709161 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3af40b87c0 0x7f3ae0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:18.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3af8b99700 1 --2- 172.21.15.98:0/1039709161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ae0004d90 0x7f3ae0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:18.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3af8b99700 1 -- 172.21.15.98:0/1039709161 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af40cd970 con 0x7f3ae0003a30 2023-03-03T08:02:18.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3af8b99700 1 -- 172.21.15.98:0/1039709161 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae000baf0 con 0x7f3ae0004d90 2023-03-03T08:02:18.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3af8b99700 1 -- 172.21.15.98:0/1039709161 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae000bcb0 con 0x7f3af40b87c0 2023-03-03T08:02:18.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3af9b9b700 1 --2- 172.21.15.98:0/1039709161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae0003a30 0x7f3ae0003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:18.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3afa39c700 1 --2- 172.21.15.98:0/1039709161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ae0004d90 0x7f3ae0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:18.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.013+0000 7f3af939a700 1 --2- 172.21.15.98:0/1039709161 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3af40b87c0 0x7f3ae0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:18.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.014+0000 7f3af9b9b700 1 -- 172.21.15.98:0/1039709161 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3af40b87c0 msgr2=0x7f3ae0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:18.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.014+0000 7f3af9b9b700 1 --2- 172.21.15.98:0/1039709161 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3af40b87c0 0x7f3ae0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.014+0000 7f3af9b9b700 1 -- 172.21.15.98:0/1039709161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ae0004d90 msgr2=0x7f3ae0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:18.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.014+0000 7f3af9b9b700 1 --2- 172.21.15.98:0/1039709161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ae0004d90 0x7f3ae0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.014+0000 7f3af9b9b700 1 -- 172.21.15.98:0/1039709161 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae00035b0 con 0x7f3ae0003a30 2023-03-03T08:02:18.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.014+0000 7f3af9b9b700 1 --2- 172.21.15.98:0/1039709161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae0003a30 0x7f3ae0003e40 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f3af0009b60 tx=0x7f3ae0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aca7f86ddcb4f28 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:18.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.014+0000 7f3af8b99700 1 -- 172.21.15.98:0/1039709161 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af000e6c0 con 0x7f3ae0003a30 2023-03-03T08:02:18.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.014+0000 7f3af8b99700 1 -- 172.21.15.98:0/1039709161 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af000e820 con 0x7f3ae0003a30 2023-03-03T08:02:18.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.014+0000 7f3af8b99700 1 -- 172.21.15.98:0/1039709161 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af000eb90 con 0x7f3ae0003a30 2023-03-03T08:02:18.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.015+0000 7f3afbdff700 1 -- 172.21.15.98:0/1039709161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae0003a30 msgr2=0x7f3ae0003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:18.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.015+0000 7f3afbdff700 1 --2- 172.21.15.98:0/1039709161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae0003a30 0x7f3ae0003e40 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f3af0009b60 tx=0x7f3ae0005880 comp rx=0 tx=0).stop 2023-03-03T08:02:18.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.015+0000 7f3afa39c700 1 -- 172.21.15.98:0/1039709161 reap_dead start 2023-03-03T08:02:18.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.015+0000 7f3afbdff700 1 -- 172.21.15.98:0/1039709161 shutdown_connections 2023-03-03T08:02:18.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.015+0000 7f3afbdff700 1 -- 172.21.15.98:0/1039709161 >> 172.21.15.98:0/1039709161 conn(0x7f3af406aee0 msgr2=0x7f3af406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:18.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.015+0000 7f3afbdff700 1 -- 172.21.15.98:0/1039709161 shutdown_connections 2023-03-03T08:02:18.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.015+0000 7f3afbdff700 1 -- 172.21.15.98:0/1039709161 wait complete. 2023-03-03T08:02:18.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.017+0000 7f3afbdff700 1 Processor -- start 2023-03-03T08:02:18.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.017+0000 7f3afbdff700 1 -- start start 2023-03-03T08:02:18.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.018+0000 7f3afbdff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae0003a30 0x7f3af40c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:18.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.018+0000 7f3afbdff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3af40c7100 0x7f3af40c7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:18.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.018+0000 7f3afbdff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3af415ebc0 0x7f3af415f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:18.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.018+0000 7f3afbdff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af40c7ab0 con 0x7f3ae0003a30 2023-03-03T08:02:18.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.018+0000 7f3afbdff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af40cae50 con 0x7f3af40c7100 2023-03-03T08:02:18.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.018+0000 7f3afbdff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af415f920 con 0x7f3af415ebc0 2023-03-03T08:02:18.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.018+0000 7f3af9b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae0003a30 0x7f3af40c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:18.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.019+0000 7f3af9b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae0003a30 0x7f3af40c5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48182/0 (socket says 172.21.15.98:48182) 2023-03-03T08:02:18.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.019+0000 7f3af9b9b700 1 -- 172.21.15.98:0/4207678430 learned_addr learned my addr 172.21.15.98:0/4207678430 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:18.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.019+0000 7f3af9b9b700 1 -- 172.21.15.98:0/4207678430 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3af415ebc0 msgr2=0x7f3af415f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:18.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.019+0000 7f3af9b9b700 1 --2- 172.21.15.98:0/4207678430 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3af415ebc0 0x7f3af415f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.019+0000 7f3af9b9b700 1 -- 172.21.15.98:0/4207678430 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3af40c7100 msgr2=0x7f3af40c7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:18.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.019+0000 7f3af939a700 1 --2- 172.21.15.98:0/4207678430 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3af40c7100 0x7f3af40c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:18.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.019+0000 7f3af9b9b700 1 --2- 172.21.15.98:0/4207678430 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3af40c7100 0x7f3af40c7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.019+0000 7f3af9b9b700 1 -- 172.21.15.98:0/4207678430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af40cdf30 con 0x7f3ae0003a30 2023-03-03T08:02:18.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.019+0000 7f3af939a700 1 --2- 172.21.15.98:0/4207678430 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3af40c7100 0x7f3af40c7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:18.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.019+0000 7f3af9b9b700 1 --2- 172.21.15.98:0/4207678430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae0003a30 0x7f3af40c5f50 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f3ae000bac0 tx=0x7f3af00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:18.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.020+0000 7f3aea7fc700 1 -- 172.21.15.98:0/4207678430 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af000e6c0 con 0x7f3ae0003a30 2023-03-03T08:02:18.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.020+0000 7f3aea7fc700 1 -- 172.21.15.98:0/4207678430 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af0003ab0 con 0x7f3ae0003a30 2023-03-03T08:02:18.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.020+0000 7f3aea7fc700 1 -- 172.21.15.98:0/4207678430 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af0003e60 con 0x7f3ae0003a30 2023-03-03T08:02:18.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.020+0000 7f3afbdff700 1 -- 172.21.15.98:0/4207678430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af4073a80 con 0x7f3ae0003a30 2023-03-03T08:02:18.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.020+0000 7f3afbdff700 1 -- 172.21.15.98:0/4207678430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af40ce0c0 con 0x7f3ae0003a30 2023-03-03T08:02:18.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.021+0000 7f3afbdff700 1 -- 172.21.15.98:0/4207678430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af40729b0 con 0x7f3ae0003a30 2023-03-03T08:02:18.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.022+0000 7f3aea7fc700 1 -- 172.21.15.98:0/4207678430 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f3af0025070 con 0x7f3ae0003a30 2023-03-03T08:02:18.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.024+0000 7f3aea7fc700 1 --2- 172.21.15.98:0/4207678430 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3ad8068380 0x7f3ad806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:18.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.024+0000 7f3aea7fc700 1 -- 172.21.15.98:0/4207678430 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3af00872e0 con 0x7f3ae0003a30 2023-03-03T08:02:18.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.025+0000 7f3af939a700 1 --2- 172.21.15.98:0/4207678430 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3ad8068380 0x7f3ad806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:18.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.025+0000 7f3af939a700 1 --2- 172.21.15.98:0/4207678430 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3ad8068380 0x7f3ad806a830 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4000c00 tx=0x7f3ae4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:18.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.029+0000 7f3aea7fc700 1 -- 172.21.15.98:0/4207678430 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3af0053db0 con 0x7f3ae0003a30 2023-03-03T08:02:18.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.294+0000 7f3afbdff700 1 -- 172.21.15.98:0/4207678430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3af404e6e0 con 0x7f3ae0003a30 2023-03-03T08:02:18.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.295+0000 7f3aea7fc700 1 -- 172.21.15.98:0/4207678430 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5298 (secure 0 0 0) 0x7f3af0085b10 con 0x7f3ae0003a30 2023-03-03T08:02:18.297 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:18.297 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:02:18.297 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_dump": [ 2023-03-03T08:02:18.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.297 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.a", 2023-03-03T08:02:18.297 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkqFdgHxAAuwiGkaNnr5NplmhniDYAEQ==", 2023-03-03T08:02:18.297 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.298 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:02:18.298 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:02:18.298 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:02:18.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:02:18.298 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.299 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.b", 2023-03-03T08:02:18.299 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFk0x8kBhAA5JDo/wmI4650J9tUnvXVzw==", 2023-03-03T08:02:18.299 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.299 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:02:18.299 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:02:18.299 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:02:18.299 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:02:18.300 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.300 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.c", 2023-03-03T08:02:18.300 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkKoRwMBAAfkWGDUsqNgqfkscl0aBX1A==", 2023-03-03T08:02:18.300 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.300 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:02:18.300 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:02:18.301 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:02:18.301 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:02:18.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.301 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.d", 2023-03-03T08:02:18.301 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFkK9SKFxAAFSCozGfyRwrv46QPitRxTQ==", 2023-03-03T08:02:18.301 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.302 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:02:18.302 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:02:18.302 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:02:18.302 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:02:18.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.303 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.0", 2023-03-03T08:02:18.303 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDBpwFkTFzTERAAGHrFKuBlXIEfaYD7vs3ZHA==", 2023-03-03T08:02:18.303 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.303 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:02:18.303 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:02:18.303 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:02:18.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.304 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.1", 2023-03-03T08:02:18.304 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDEpwFkVWObLBAAW4acTvNbmscIe18NXVgRYg==", 2023-03-03T08:02:18.304 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.304 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:02:18.304 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:02:18.304 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:02:18.304 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.305 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.2", 2023-03-03T08:02:18.305 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDIpwFkK8lgChAAHVD5btBivUpcapSgDp2E0w==", 2023-03-03T08:02:18.305 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.305 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:02:18.305 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:02:18.306 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:02:18.306 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.306 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.3", 2023-03-03T08:02:18.306 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDLpwFksrb8IxAAN+CP1O3DJ1zDy+qyGNor2Q==", 2023-03-03T08:02:18.306 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.306 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:02:18.307 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:02:18.307 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:02:18.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.307 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.4", 2023-03-03T08:02:18.307 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDSpwFkkWGHIxAA4wejA0w05JPIUghEAVG/Rg==", 2023-03-03T08:02:18.307 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.308 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:02:18.308 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:02:18.308 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:02:18.308 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.308 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.5", 2023-03-03T08:02:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDWpwFknHmGAhAA6VM16luaykZfJEzr+yDZZA==", 2023-03-03T08:02:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:02:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:02:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:02:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.310 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.6", 2023-03-03T08:02:18.310 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDZpwFkcitNDRAAOz04YGTzx3Phenp0n8JeEA==", 2023-03-03T08:02:18.310 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.310 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:02:18.310 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:02:18.310 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:02:18.310 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.311 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.7", 2023-03-03T08:02:18.311 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDcpwFkrjmIFxAAbWeB2+8VQgyTplW+fk47OA==", 2023-03-03T08:02:18.311 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.311 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:02:18.311 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:02:18.311 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:02:18.311 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.312 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.0", 2023-03-03T08:02:18.312 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFk02lsJhAA0cH/evMnWdZN6DdBGCLjjg==", 2023-03-03T08:02:18.312 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.312 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:02:18.312 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow r", 2023-03-03T08:02:18.313 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rw pool=cephfs_data" 2023-03-03T08:02:18.313 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.313 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.1", 2023-03-03T08:02:18.313 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC9pwFkRqXrABAArm/01O5MtSTQHWQWKqNQEg==", 2023-03-03T08:02:18.313 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.313 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:02:18.314 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T08:02:18.314 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T08:02:18.314 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T08:02:18.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.314 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.2", 2023-03-03T08:02:18.314 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFkyKp3MRAA8Eh41TZNsO6/8VhAWCpNMQ==", 2023-03-03T08:02:18.315 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.315 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:02:18.315 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T08:02:18.315 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T08:02:18.315 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T08:02:18.315 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.316 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.3", 2023-03-03T08:02:18.316 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC9pwFk14bSCxAAy2nXc9G50W4lH5GtH84hpQ==", 2023-03-03T08:02:18.316 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.316 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:02:18.316 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T08:02:18.316 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T08:02:18.316 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T08:02:18.317 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.317 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.admin", 2023-03-03T08:02:18.317 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC6pwFkokmiEBAAFwi8rhPmk4Zyk95p760HDg==", 2023-03-03T08:02:18.317 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.317 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T08:02:18.317 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:02:18.318 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:02:18.318 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:02:18.318 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.318 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mgr.x", 2023-03-03T08:02:18.318 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkvdMQFBAAMi+x7psn5CDpd//d6jN2VA==", 2023-03-03T08:02:18.318 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.319 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T08:02:18.319 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile mgr", 2023-03-03T08:02:18.319 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:02:18.319 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:18.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:18.319 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mgr.y", 2023-03-03T08:02:18.320 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkfYO5BxAAFW/elbdoqVXBkUIKrAvYsg==", 2023-03-03T08:02:18.320 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:02:18.320 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T08:02:18.320 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile mgr", 2023-03-03T08:02:18.320 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:02:18.320 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.321 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:18.321 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:02:18.321 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:02:18.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.297+0000 7f3afbdff700 1 -- 172.21.15.98:0/4207678430 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3ad8068380 msgr2=0x7f3ad806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:18.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.297+0000 7f3afbdff700 1 --2- 172.21.15.98:0/4207678430 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3ad8068380 0x7f3ad806a830 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4000c00 tx=0x7f3ae4009040 comp rx=0 tx=0).stop 2023-03-03T08:02:18.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.298+0000 7f3afbdff700 1 -- 172.21.15.98:0/4207678430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae0003a30 msgr2=0x7f3af40c5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:18.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.298+0000 7f3afbdff700 1 --2- 172.21.15.98:0/4207678430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae0003a30 0x7f3af40c5f50 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f3ae000bac0 tx=0x7f3af00047c0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.298+0000 7f3afbdff700 1 -- 172.21.15.98:0/4207678430 shutdown_connections 2023-03-03T08:02:18.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.298+0000 7f3afbdff700 1 --2- 172.21.15.98:0/4207678430 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3ad8068380 0x7f3ad806a830 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.298+0000 7f3afbdff700 1 --2- 172.21.15.98:0/4207678430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae0003a30 0x7f3af40c5f50 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.298+0000 7f3afbdff700 1 --2- 172.21.15.98:0/4207678430 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3af415ebc0 0x7f3af415f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.298+0000 7f3afbdff700 1 --2- 172.21.15.98:0/4207678430 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3af40c7100 0x7f3af40c7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.298+0000 7f3afbdff700 1 -- 172.21.15.98:0/4207678430 >> 172.21.15.98:0/4207678430 conn(0x7f3af406aee0 msgr2=0x7f3af40b2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:18.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.298+0000 7f3afbdff700 1 -- 172.21.15.98:0/4207678430 shutdown_connections 2023-03-03T08:02:18.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.298+0000 7f3afbdff700 1 -- 172.21.15.98:0/4207678430 wait complete. 2023-03-03T08:02:18.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:02:18.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.568+0000 7f732adff700 1 Processor -- start 2023-03-03T08:02:18.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.568+0000 7f732adff700 1 -- start start 2023-03-03T08:02:18.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.570+0000 7f732adff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f73240d23e0 0x7f73240cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:18.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.570+0000 7f732adff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73240b5b20 con 0x7f73240b87c0 2023-03-03T08:02:18.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.570+0000 7f732adff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73240d0ef0 con 0x7f73240b8b90 2023-03-03T08:02:18.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.570+0000 7f732adff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73240d0600 con 0x7f73240d23e0 2023-03-03T08:02:18.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.570+0000 7f7328b9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f73240b87c0 0x7f73240b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58692/0 (socket says 172.21.15.98:58692) 2023-03-03T08:02:18.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.570+0000 7f7328b9b700 1 -- 172.21.15.98:0/805092326 learned_addr learned my addr 172.21.15.98:0/805092326 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:18.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.570+0000 7f732939c700 1 --2- 172.21.15.98:0/805092326 >> v2:172.21.15.98:3300/0 conn(0x7f73240d23e0 0x7f73240cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:18.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.571+0000 7f732939c700 1 -- 172.21.15.98:0/805092326 >> v1:172.21.15.100:6789/0 conn(0x7f73240b8b90 legacy=0x7f73240b5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:18.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.571+0000 7f732939c700 1 -- 172.21.15.98:0/805092326 >> v1:172.21.15.98:6789/0 conn(0x7f73240b87c0 legacy=0x7f73240b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:18.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.571+0000 7f732939c700 1 -- 172.21.15.98:0/805092326 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73240cd970 con 0x7f73240d23e0 2023-03-03T08:02:18.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.571+0000 7f73237fe700 1 -- 172.21.15.98:0/805092326 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400175090 0 0) 0x7f73240b5b20 con 0x7f73240b87c0 2023-03-03T08:02:18.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.571+0000 7f732939c700 1 --2- 172.21.15.98:0/805092326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73240d23e0 0x7f73240cffe0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f731c009d90 tx=0x7f731c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d908ca363c0a154 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:18.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.571+0000 7f73237fe700 1 -- 172.21.15.98:0/805092326 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c00e070 con 0x7f73240d23e0 2023-03-03T08:02:18.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.571+0000 7f73237fe700 1 -- 172.21.15.98:0/805092326 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731c00bbd0 con 0x7f73240d23e0 2023-03-03T08:02:18.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.571+0000 7f73237fe700 1 -- 172.21.15.98:0/805092326 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c012430 con 0x7f73240d23e0 2023-03-03T08:02:18.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.572+0000 7f732adff700 1 -- 172.21.15.98:0/805092326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73240d23e0 msgr2=0x7f73240cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:18.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.572+0000 7f732adff700 1 --2- 172.21.15.98:0/805092326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73240d23e0 0x7f73240cffe0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f731c009d90 tx=0x7f731c00b330 comp rx=0 tx=0).stop 2023-03-03T08:02:18.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.572+0000 7f732adff700 1 -- 172.21.15.98:0/805092326 shutdown_connections 2023-03-03T08:02:18.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.572+0000 7f732adff700 1 --2- 172.21.15.98:0/805092326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73240d23e0 0x7f73240cffe0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.572+0000 7f732adff700 1 -- 172.21.15.98:0/805092326 >> 172.21.15.98:0/805092326 conn(0x7f732406aee0 msgr2=0x7f732406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:18.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.572+0000 7f732adff700 1 -- 172.21.15.98:0/805092326 shutdown_connections 2023-03-03T08:02:18.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.572+0000 7f732adff700 1 -- 172.21.15.98:0/805092326 wait complete. 2023-03-03T08:02:18.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.574+0000 7f732adff700 1 Processor -- start 2023-03-03T08:02:18.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.574+0000 7f732adff700 1 -- start start 2023-03-03T08:02:18.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.575+0000 7f732adff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73240b87c0 0x7f7324150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:18.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.575+0000 7f732adff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73240b8b90 0x7f732415aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:18.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.575+0000 7f732adff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73240d23e0 0x7f732415af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:18.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.575+0000 7f732adff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73240cb410 con 0x7f73240d23e0 2023-03-03T08:02:18.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.575+0000 7f732adff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73240cc3c0 con 0x7f73240b87c0 2023-03-03T08:02:18.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.575+0000 7f732adff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73240cb130 con 0x7f73240b8b90 2023-03-03T08:02:18.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.576+0000 7f732939c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73240d23e0 0x7f732415af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:18.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.576+0000 7f732939c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73240d23e0 0x7f732415af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48188/0 (socket says 172.21.15.98:48188) 2023-03-03T08:02:18.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.576+0000 7f732939c700 1 -- 172.21.15.98:0/3877290793 learned_addr learned my addr 172.21.15.98:0/3877290793 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:18.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.576+0000 7f7328b9b700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73240b87c0 0x7f7324150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:18.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.576+0000 7f7323fff700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73240b8b90 0x7f732415aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:18.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.576+0000 7f732939c700 1 -- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73240b8b90 msgr2=0x7f732415aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:18.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.576+0000 7f732939c700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73240b8b90 0x7f732415aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.576+0000 7f732939c700 1 -- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73240b87c0 msgr2=0x7f7324150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:18.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.576+0000 7f732939c700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73240b87c0 0x7f7324150c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.576+0000 7f732939c700 1 -- 172.21.15.98:0/3877290793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c009a70 con 0x7f73240d23e0 2023-03-03T08:02:18.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.576+0000 7f7323fff700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73240b8b90 0x7f732415aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:18.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.576+0000 7f7328b9b700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73240b87c0 0x7f7324150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:18.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.576+0000 7f732939c700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73240d23e0 0x7f732415af60 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f731c014040 tx=0x7f731c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:18.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.577+0000 7f73217fa700 1 -- 172.21.15.98:0/3877290793 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c01b810 con 0x7f73240d23e0 2023-03-03T08:02:18.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.577+0000 7f73217fa700 1 -- 172.21.15.98:0/3877290793 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731c020070 con 0x7f73240d23e0 2023-03-03T08:02:18.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.577+0000 7f732adff700 1 -- 172.21.15.98:0/3877290793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7324073a80 con 0x7f73240d23e0 2023-03-03T08:02:18.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.577+0000 7f73217fa700 1 -- 172.21.15.98:0/3877290793 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c01bbb0 con 0x7f73240d23e0 2023-03-03T08:02:18.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.577+0000 7f732adff700 1 -- 172.21.15.98:0/3877290793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73240cdc50 con 0x7f73240d23e0 2023-03-03T08:02:18.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.578+0000 7f732adff700 1 -- 172.21.15.98:0/3877290793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73240cdf30 con 0x7f73240d23e0 2023-03-03T08:02:18.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.579+0000 7f73217fa700 1 -- 172.21.15.98:0/3877290793 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f731c01bd10 con 0x7f73240d23e0 2023-03-03T08:02:18.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.580+0000 7f73217fa700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7308068380 0x7f730806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:18.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.581+0000 7f73217fa700 1 -- 172.21.15.98:0/3877290793 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f731c022070 con 0x7f73240d23e0 2023-03-03T08:02:18.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.581+0000 7f7328b9b700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7308068380 0x7f730806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:18.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.582+0000 7f7328b9b700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7308068380 0x7f730806a830 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7318000f60 tx=0x7f7318009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:18.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.584+0000 7f73217fa700 1 -- 172.21.15.98:0/3877290793 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f731c055130 con 0x7f73240d23e0 2023-03-03T08:02:18.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.779+0000 7f732adff700 1 -- 172.21.15.98:0/3877290793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73240729b0 con 0x7f73240d23e0 2023-03-03T08:02:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.780+0000 7f73217fa700 1 -- 172.21.15.98:0/3877290793 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+14268 (secure 0 0 0) 0x7f731c058750 con 0x7f73240d23e0 2023-03-03T08:02:18.781 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:18.782 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":87,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:02:17.713549+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:02:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.783+0000 7f732adff700 1 -- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7308068380 msgr2=0x7f730806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.783+0000 7f732adff700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7308068380 0x7f730806a830 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7318000f60 tx=0x7f7318009040 comp rx=0 tx=0).stop 2023-03-03T08:02:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.783+0000 7f732adff700 1 -- 172.21.15.98:0/3877290793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73240d23e0 msgr2=0x7f732415af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.783+0000 7f732adff700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73240d23e0 0x7f732415af60 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f731c014040 tx=0x7f731c004ad0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.783+0000 7f732adff700 1 -- 172.21.15.98:0/3877290793 shutdown_connections 2023-03-03T08:02:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.783+0000 7f732adff700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7308068380 0x7f730806a830 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.783+0000 7f732adff700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73240d23e0 0x7f732415af60 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.783+0000 7f732adff700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73240b87c0 0x7f7324150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.783+0000 7f732adff700 1 --2- 172.21.15.98:0/3877290793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73240b8b90 0x7f732415aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.783+0000 7f732adff700 1 -- 172.21.15.98:0/3877290793 >> 172.21.15.98:0/3877290793 conn(0x7f732406aee0 msgr2=0x7f73240b1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:18.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.783+0000 7f732adff700 1 -- 172.21.15.98:0/3877290793 shutdown_connections 2023-03-03T08:02:18.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:18.783+0000 7f732adff700 1 -- 172.21.15.98:0/3877290793 wait complete. 2023-03-03T08:02:18.811 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:02:18.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:02:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.059+0000 7f141ffff700 1 Processor -- start 2023-03-03T08:02:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.060+0000 7f141ffff700 1 -- start start 2023-03-03T08:02:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.061+0000 7f141ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f14201609b0 0x7f1420160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.061+0000 7f141ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14200b4aa0 0x7f14200b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:19.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.061+0000 7f141ffff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14201612c0 con 0x7f14200cf570 2023-03-03T08:02:19.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.061+0000 7f141ffff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14200cebb0 con 0x7f14201609b0 2023-03-03T08:02:19.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.061+0000 7f141ffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14200b5460 con 0x7f14200b4aa0 2023-03-03T08:02:19.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.061+0000 7f141effd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f14201609b0 0x7f1420160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:19.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.061+0000 7f141e7fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f14200cf570 0x7f142015d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58694/0 (socket says 172.21.15.98:58694) 2023-03-03T08:02:19.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.061+0000 7f141e7fc700 1 -- 172.21.15.98:0/2261756534 learned_addr learned my addr 172.21.15.98:0/2261756534 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:19.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.062+0000 7f141f7fe700 1 --2- 172.21.15.98:0/2261756534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14200b4aa0 0x7f14200b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:19.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.062+0000 7f141dffb700 1 -- 172.21.15.98:0/2261756534 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572111089 0 0) 0x7f14201612c0 con 0x7f14200cf570 2023-03-03T08:02:19.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.062+0000 7f141effd700 1 -- 172.21.15.98:0/2261756534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14200b4aa0 msgr2=0x7f14200b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:19.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.062+0000 7f141effd700 1 --2- 172.21.15.98:0/2261756534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14200b4aa0 0x7f14200b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.062+0000 7f141effd700 1 -- 172.21.15.98:0/2261756534 >> v1:172.21.15.98:6789/0 conn(0x7f14200cf570 legacy=0x7f142015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.062+0000 7f141effd700 1 -- 172.21.15.98:0/2261756534 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142015dd00 con 0x7f14201609b0 2023-03-03T08:02:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.062+0000 7f141effd700 1 --2- 172.21.15.98:0/2261756534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14201609b0 0x7f1420160d80 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f1410009840 tx=0x7f1410009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27acea92740571d8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.063+0000 7f141dffb700 1 -- 172.21.15.98:0/2261756534 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1410011830 con 0x7f14201609b0 2023-03-03T08:02:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.063+0000 7f141dffb700 1 -- 172.21.15.98:0/2261756534 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1410011990 con 0x7f14201609b0 2023-03-03T08:02:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.063+0000 7f141dffb700 1 -- 172.21.15.98:0/2261756534 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1410011d20 con 0x7f14201609b0 2023-03-03T08:02:19.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.063+0000 7f141ffff700 1 -- 172.21.15.98:0/2261756534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14201609b0 msgr2=0x7f1420160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:19.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.063+0000 7f141ffff700 1 --2- 172.21.15.98:0/2261756534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14201609b0 0x7f1420160d80 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f1410009840 tx=0x7f1410009b50 comp rx=0 tx=0).stop 2023-03-03T08:02:19.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.063+0000 7f141ffff700 1 -- 172.21.15.98:0/2261756534 shutdown_connections 2023-03-03T08:02:19.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.063+0000 7f141ffff700 1 --2- 172.21.15.98:0/2261756534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14201609b0 0x7f1420160d80 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.063+0000 7f141ffff700 1 --2- 172.21.15.98:0/2261756534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14200b4aa0 0x7f14200b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.063+0000 7f141ffff700 1 -- 172.21.15.98:0/2261756534 >> 172.21.15.98:0/2261756534 conn(0x7f142006b880 msgr2=0x7f14200cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:19.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.063+0000 7f141ffff700 1 -- 172.21.15.98:0/2261756534 shutdown_connections 2023-03-03T08:02:19.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.063+0000 7f141ffff700 1 -- 172.21.15.98:0/2261756534 wait complete. 2023-03-03T08:02:19.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.065+0000 7f141ffff700 1 Processor -- start 2023-03-03T08:02:19.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.066+0000 7f141ffff700 1 -- start start 2023-03-03T08:02:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.067+0000 7f141ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14200b4aa0 0x7f14200ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.067+0000 7f141ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14200cf570 0x7f14200ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.067+0000 7f141ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14201609b0 0x7f14200b6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.067+0000 7f141ffff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14201506a0 con 0x7f14201609b0 2023-03-03T08:02:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.067+0000 7f141ffff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142014fa90 con 0x7f14200b4aa0 2023-03-03T08:02:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.067+0000 7f141ffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142014fd70 con 0x7f14200cf570 2023-03-03T08:02:19.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.067+0000 7f141f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14201609b0 0x7f14200b6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:19.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.067+0000 7f141f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14201609b0 0x7f14200b6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48204/0 (socket says 172.21.15.98:48204) 2023-03-03T08:02:19.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.067+0000 7f141f7fe700 1 -- 172.21.15.98:0/715059784 learned_addr learned my addr 172.21.15.98:0/715059784 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:19.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.067+0000 7f141effd700 1 --2- 172.21.15.98:0/715059784 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14200b4aa0 0x7f14200ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:19.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.067+0000 7f141e7fc700 1 --2- 172.21.15.98:0/715059784 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14200cf570 0x7f14200ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:19.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.068+0000 7f141f7fe700 1 -- 172.21.15.98:0/715059784 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14200cf570 msgr2=0x7f14200ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:19.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.068+0000 7f141f7fe700 1 --2- 172.21.15.98:0/715059784 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14200cf570 0x7f14200ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.068+0000 7f141f7fe700 1 -- 172.21.15.98:0/715059784 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14200b4aa0 msgr2=0x7f14200ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.068+0000 7f141f7fe700 1 --2- 172.21.15.98:0/715059784 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14200b4aa0 0x7f14200ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.068+0000 7f141f7fe700 1 -- 172.21.15.98:0/715059784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14100094a0 con 0x7f14201609b0 2023-03-03T08:02:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.068+0000 7f141e7fc700 1 --2- 172.21.15.98:0/715059784 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14200cf570 0x7f14200ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.068+0000 7f141f7fe700 1 --2- 172.21.15.98:0/715059784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14201609b0 0x7f14200b6d70 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f141800efc0 tx=0x7f141800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.068+0000 7f1407fff700 1 -- 172.21.15.98:0/715059784 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1418009ea0 con 0x7f14201609b0 2023-03-03T08:02:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.068+0000 7f1407fff700 1 -- 172.21.15.98:0/715059784 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141800f040 con 0x7f14201609b0 2023-03-03T08:02:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.068+0000 7f1407fff700 1 -- 172.21.15.98:0/715059784 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1418014480 con 0x7f14201609b0 2023-03-03T08:02:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.068+0000 7f141ffff700 1 -- 172.21.15.98:0/715059784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14200cda60 con 0x7f14201609b0 2023-03-03T08:02:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.068+0000 7f141ffff700 1 -- 172.21.15.98:0/715059784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14200cd780 con 0x7f14201609b0 2023-03-03T08:02:19.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.069+0000 7f141ffff700 1 -- 172.21.15.98:0/715059784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14200cdd40 con 0x7f14201609b0 2023-03-03T08:02:19.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.071+0000 7f1407fff700 1 -- 172.21.15.98:0/715059784 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f14180145e0 con 0x7f14201609b0 2023-03-03T08:02:19.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.072+0000 7f1407fff700 1 --2- 172.21.15.98:0/715059784 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f140c068380 0x7f140c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:19.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.072+0000 7f1407fff700 1 -- 172.21.15.98:0/715059784 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1418086790 con 0x7f14201609b0 2023-03-03T08:02:19.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.072+0000 7f141effd700 1 --2- 172.21.15.98:0/715059784 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f140c068380 0x7f140c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:19.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.073+0000 7f141effd700 1 --2- 172.21.15.98:0/715059784 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f140c068380 0x7f140c06a830 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f14100090b0 tx=0x7f141001c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:19.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.076+0000 7f1407fff700 1 -- 172.21.15.98:0/715059784 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1418053260 con 0x7f14201609b0 2023-03-03T08:02:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.266+0000 7f141ffff700 1 -- 172.21.15.98:0/715059784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1420068290 con 0x7f14201609b0 2023-03-03T08:02:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.267+0000 7f1407fff700 1 -- 172.21.15.98:0/715059784 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1418018020 con 0x7f14201609b0 2023-03-03T08:02:19.269 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:02:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.269+0000 7f141ffff700 1 -- 172.21.15.98:0/715059784 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f140c068380 msgr2=0x7f140c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.269+0000 7f141ffff700 1 --2- 172.21.15.98:0/715059784 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f140c068380 0x7f140c06a830 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f14100090b0 tx=0x7f141001c040 comp rx=0 tx=0).stop 2023-03-03T08:02:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.269+0000 7f141ffff700 1 -- 172.21.15.98:0/715059784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14201609b0 msgr2=0x7f14200b6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.269+0000 7f141ffff700 1 --2- 172.21.15.98:0/715059784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14201609b0 0x7f14200b6d70 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f141800efc0 tx=0x7f141800c3a0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.269+0000 7f141ffff700 1 -- 172.21.15.98:0/715059784 shutdown_connections 2023-03-03T08:02:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.269+0000 7f141ffff700 1 --2- 172.21.15.98:0/715059784 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f140c068380 0x7f140c06a830 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.269+0000 7f141ffff700 1 --2- 172.21.15.98:0/715059784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14201609b0 0x7f14200b6d70 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.269+0000 7f141ffff700 1 --2- 172.21.15.98:0/715059784 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14200b4aa0 0x7f14200ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.269+0000 7f141ffff700 1 --2- 172.21.15.98:0/715059784 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14200cf570 0x7f14200ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.269+0000 7f141ffff700 1 -- 172.21.15.98:0/715059784 >> 172.21.15.98:0/715059784 conn(0x7f142006b880 msgr2=0x7f14201525d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.270+0000 7f141ffff700 1 -- 172.21.15.98:0/715059784 shutdown_connections 2023-03-03T08:02:19.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.270+0000 7f141ffff700 1 -- 172.21.15.98:0/715059784 wait complete. 2023-03-03T08:02:19.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:02:19.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.545+0000 7fae657ff700 1 Processor -- start 2023-03-03T08:02:19.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.545+0000 7fae657ff700 1 -- start start 2023-03-03T08:02:19.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.546+0000 7fae657ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fae6015ca70 0x7fae600cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:19.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.546+0000 7fae657ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae6006b4b0 0x7fae600cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:19.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.546+0000 7fae657ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae600cf980 con 0x7fae600d00a0 2023-03-03T08:02:19.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.546+0000 7fae657ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae600cd5b0 con 0x7fae6006b4b0 2023-03-03T08:02:19.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.546+0000 7fae657ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae600d0540 con 0x7fae6015ca70 2023-03-03T08:02:19.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.547+0000 7fae5e7fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae6006b4b0 0x7fae600cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:19.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.547+0000 7fae5e7fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae6006b4b0 0x7fae600cfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48214/0 (socket says 172.21.15.98:48214) 2023-03-03T08:02:19.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.547+0000 7fae5e7fc700 1 -- 172.21.15.98:0/2374730647 learned_addr learned my addr 172.21.15.98:0/2374730647 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:19.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.547+0000 7fae5f7fe700 1 --1- 172.21.15.98:0/2374730647 >> v1:172.21.15.98:6789/0 conn(0x7fae600d00a0 0x7fae600b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58708/0 (socket says 172.21.15.98:58708) 2023-03-03T08:02:19.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.547+0000 7fae5effd700 1 --2- 172.21.15.98:0/2374730647 >> v2:172.21.15.100:3300/0 conn(0x7fae6015ca70 0x7fae600cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:19.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.547+0000 7fae5e7fc700 1 -- 172.21.15.98:0/2374730647 >> v2:172.21.15.100:3300/0 conn(0x7fae6015ca70 msgr2=0x7fae600cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:19.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.547+0000 7fae5e7fc700 1 --2- 172.21.15.98:0/2374730647 >> v2:172.21.15.100:3300/0 conn(0x7fae6015ca70 0x7fae600cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.547+0000 7fae5e7fc700 1 -- 172.21.15.98:0/2374730647 >> v1:172.21.15.98:6789/0 conn(0x7fae600d00a0 legacy=0x7fae600b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.547+0000 7fae5e7fc700 1 -- 172.21.15.98:0/2374730647 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae600d2fd0 con 0x7fae6006b4b0 2023-03-03T08:02:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.547+0000 7fae5e7fc700 1 --2- 172.21.15.98:0/2374730647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6006b4b0 0x7fae600cfb60 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fae6006c2c0 tx=0x7fae54009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b99202415b6b87c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.548+0000 7fae5dffb700 1 -- 172.21.15.98:0/2374730647 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54011830 con 0x7fae6006b4b0 2023-03-03T08:02:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.548+0000 7fae5dffb700 1 -- 172.21.15.98:0/2374730647 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae54011990 con 0x7fae6006b4b0 2023-03-03T08:02:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.548+0000 7fae5dffb700 1 -- 172.21.15.98:0/2374730647 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54011d20 con 0x7fae6006b4b0 2023-03-03T08:02:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.548+0000 7fae657ff700 1 -- 172.21.15.98:0/2374730647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6006b4b0 msgr2=0x7fae600cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.548+0000 7fae657ff700 1 --2- 172.21.15.98:0/2374730647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6006b4b0 0x7fae600cfb60 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fae6006c2c0 tx=0x7fae54009bd0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.548+0000 7fae657ff700 1 -- 172.21.15.98:0/2374730647 shutdown_connections 2023-03-03T08:02:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.548+0000 7fae657ff700 1 --2- 172.21.15.98:0/2374730647 >> v2:172.21.15.100:3300/0 conn(0x7fae6015ca70 0x7fae600cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.548+0000 7fae657ff700 1 --2- 172.21.15.98:0/2374730647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6006b4b0 0x7fae600cfb60 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.548+0000 7fae657ff700 1 -- 172.21.15.98:0/2374730647 >> 172.21.15.98:0/2374730647 conn(0x7fae60073e70 msgr2=0x7fae6006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.549+0000 7fae657ff700 1 -- 172.21.15.98:0/2374730647 shutdown_connections 2023-03-03T08:02:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.549+0000 7fae657ff700 1 -- 172.21.15.98:0/2374730647 wait complete. 2023-03-03T08:02:19.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.551+0000 7fae657ff700 1 Processor -- start 2023-03-03T08:02:19.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.552+0000 7fae657ff700 1 -- start start 2023-03-03T08:02:19.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.553+0000 7fae657ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6006b4b0 0x7fae601f4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:19.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.553+0000 7fae657ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae600d00a0 0x7fae600c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:19.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.553+0000 7fae657ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae6015ca70 0x7fae6015f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:19.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.553+0000 7fae657ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae601f5430 con 0x7fae6006b4b0 2023-03-03T08:02:19.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.553+0000 7fae657ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae601584c0 con 0x7fae6015ca70 2023-03-03T08:02:19.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.553+0000 7fae657ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6015cfe0 con 0x7fae600d00a0 2023-03-03T08:02:19.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.553+0000 7fae5effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6006b4b0 0x7fae601f4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:19.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.553+0000 7fae5effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6006b4b0 0x7fae601f4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48216/0 (socket says 172.21.15.98:48216) 2023-03-03T08:02:19.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.553+0000 7fae5effd700 1 -- 172.21.15.98:0/780862556 learned_addr learned my addr 172.21.15.98:0/780862556 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:19.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.554+0000 7fae5e7fc700 1 --2- 172.21.15.98:0/780862556 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae600d00a0 0x7fae600c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:19.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.554+0000 7fae5f7fe700 1 --2- 172.21.15.98:0/780862556 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae6015ca70 0x7fae6015f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:19.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.554+0000 7fae5effd700 1 -- 172.21.15.98:0/780862556 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae600d00a0 msgr2=0x7fae600c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:19.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.554+0000 7fae5effd700 1 --2- 172.21.15.98:0/780862556 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae600d00a0 0x7fae600c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.554+0000 7fae5effd700 1 -- 172.21.15.98:0/780862556 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae6015ca70 msgr2=0x7fae6015f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:19.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.554+0000 7fae5effd700 1 --2- 172.21.15.98:0/780862556 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae6015ca70 0x7fae6015f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.554+0000 7fae5effd700 1 -- 172.21.15.98:0/780862556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae54009550 con 0x7fae6006b4b0 2023-03-03T08:02:19.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.554+0000 7fae5f7fe700 1 --2- 172.21.15.98:0/780862556 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae6015ca70 0x7fae6015f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:19.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.554+0000 7fae5effd700 1 --2- 172.21.15.98:0/780862556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6006b4b0 0x7fae601f4ef0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fae5000e8a0 tx=0x7fae5000ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:19.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.554+0000 7fae477fe700 1 -- 172.21.15.98:0/780862556 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50009590 con 0x7fae6006b4b0 2023-03-03T08:02:19.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.554+0000 7fae477fe700 1 -- 172.21.15.98:0/780862556 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5000eea0 con 0x7fae6006b4b0 2023-03-03T08:02:19.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.554+0000 7fae477fe700 1 -- 172.21.15.98:0/780862556 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5000ca50 con 0x7fae6006b4b0 2023-03-03T08:02:19.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.554+0000 7fae657ff700 1 -- 172.21.15.98:0/780862556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae600cd900 con 0x7fae6006b4b0 2023-03-03T08:02:19.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.555+0000 7fae657ff700 1 -- 172.21.15.98:0/780862556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae600d1800 con 0x7fae6006b4b0 2023-03-03T08:02:19.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.556+0000 7fae657ff700 1 -- 172.21.15.98:0/780862556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae600cdbe0 con 0x7fae6006b4b0 2023-03-03T08:02:19.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.557+0000 7fae477fe700 1 -- 172.21.15.98:0/780862556 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fae50019030 con 0x7fae6006b4b0 2023-03-03T08:02:19.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.558+0000 7fae477fe700 1 --2- 172.21.15.98:0/780862556 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fae4c068380 0x7fae4c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:19.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.558+0000 7fae477fe700 1 -- 172.21.15.98:0/780862556 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fae500858b0 con 0x7fae6006b4b0 2023-03-03T08:02:19.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.559+0000 7fae5e7fc700 1 --2- 172.21.15.98:0/780862556 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fae4c068380 0x7fae4c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:19.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.560+0000 7fae5e7fc700 1 --2- 172.21.15.98:0/780862556 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fae4c068380 0x7fae4c06a830 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fae6006c2c0 tx=0x7fae54008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:19.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.562+0000 7fae477fe700 1 -- 172.21.15.98:0/780862556 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fae50052380 con 0x7fae6006b4b0 2023-03-03T08:02:19.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.756+0000 7fae657ff700 1 -- 172.21.15.98:0/780862556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fae6004e6e0 con 0x7fae6006b4b0 2023-03-03T08:02:19.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.757+0000 7fae477fe700 1 -- 172.21.15.98:0/780862556 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v87) v1 ==== 92+0+10 (secure 0 0 0) 0x7fae500559a0 con 0x7fae6006b4b0 2023-03-03T08:02:19.758 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:02:19.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.759+0000 7fae657ff700 1 -- 172.21.15.98:0/780862556 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fae4c068380 msgr2=0x7fae4c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:19.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.759+0000 7fae657ff700 1 --2- 172.21.15.98:0/780862556 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fae4c068380 0x7fae4c06a830 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fae6006c2c0 tx=0x7fae54008030 comp rx=0 tx=0).stop 2023-03-03T08:02:19.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.759+0000 7fae657ff700 1 -- 172.21.15.98:0/780862556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6006b4b0 msgr2=0x7fae601f4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:19.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.759+0000 7fae657ff700 1 --2- 172.21.15.98:0/780862556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6006b4b0 0x7fae601f4ef0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fae5000e8a0 tx=0x7fae5000ec60 comp rx=0 tx=0).stop 2023-03-03T08:02:19.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.760+0000 7fae657ff700 1 -- 172.21.15.98:0/780862556 shutdown_connections 2023-03-03T08:02:19.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.760+0000 7fae657ff700 1 --2- 172.21.15.98:0/780862556 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fae4c068380 0x7fae4c06a830 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.760+0000 7fae657ff700 1 --2- 172.21.15.98:0/780862556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae6006b4b0 0x7fae601f4ef0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.760+0000 7fae657ff700 1 --2- 172.21.15.98:0/780862556 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae6015ca70 0x7fae6015f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.760+0000 7fae657ff700 1 --2- 172.21.15.98:0/780862556 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae600d00a0 0x7fae600c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:19.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.760+0000 7fae657ff700 1 -- 172.21.15.98:0/780862556 >> 172.21.15.98:0/780862556 conn(0x7fae60073e70 msgr2=0x7fae600b80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:19.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.760+0000 7fae657ff700 1 -- 172.21.15.98:0/780862556 shutdown_connections 2023-03-03T08:02:19.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:19.760+0000 7fae657ff700 1 -- 172.21.15.98:0/780862556 wait complete. 2023-03-03T08:02:19.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-03-03T08:02:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.031+0000 7fb097fff700 1 Processor -- start 2023-03-03T08:02:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.031+0000 7fb097fff700 1 -- start start 2023-03-03T08:02:20.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.032+0000 7fb097fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0900d23e0 0x7fb0900cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.032+0000 7fb097fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0900b5b20 con 0x7fb0900b8b90 2023-03-03T08:02:20.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.032+0000 7fb097fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0900d0ef0 con 0x7fb0900b87c0 2023-03-03T08:02:20.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.032+0000 7fb097fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0900d0600 con 0x7fb0900d23e0 2023-03-03T08:02:20.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.033+0000 7fb09559a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb0900b8b90 0x7fb0900b5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58724/0 (socket says 172.21.15.98:58724) 2023-03-03T08:02:20.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.033+0000 7fb09559a700 1 -- 172.21.15.98:0/2903902298 learned_addr learned my addr 172.21.15.98:0/2903902298 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:20.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.033+0000 7fb09659c700 1 --2- 172.21.15.98:0/2903902298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0900d23e0 0x7fb0900cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:20.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.033+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594573206 0 0) 0x7fb0900b5b20 con 0x7fb0900b8b90 2023-03-03T08:02:20.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.033+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb07c0035b0 con 0x7fb0900b8b90 2023-03-03T08:02:20.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.033+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760934017 0 0) 0x7fb0900d0ef0 con 0x7fb0900b87c0 2023-03-03T08:02:20.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.033+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0900b5b20 con 0x7fb0900b87c0 2023-03-03T08:02:20.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.033+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1284465770 0 0) 0x7fb07c0035b0 con 0x7fb0900b8b90 2023-03-03T08:02:20.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.033+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0900d23e0 msgr2=0x7fb0900cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:20.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.033+0000 7fb094d99700 1 --2- 172.21.15.98:0/2903902298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0900d23e0 0x7fb0900cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.034+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 >> v1:172.21.15.100:6789/0 conn(0x7fb0900b87c0 legacy=0x7fb0900b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:20.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.034+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0900cd970 con 0x7fb0900b8b90 2023-03-03T08:02:20.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.034+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7fb080001670 con 0x7fb0900b8b90 2023-03-03T08:02:20.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.034+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 >> v1:172.21.15.98:6789/0 conn(0x7fb0900b8b90 legacy=0x7fb0900b5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:20.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.034+0000 7fb094d99700 1 --2- 172.21.15.98:0/2903902298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb07c003bc0 0x7fb07c003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.034+0000 7fb094d99700 1 --2- 172.21.15.98:0/2903902298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb07c004850 0x7fb07c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.034+0000 7fb094d99700 1 --2- 172.21.15.98:0/2903902298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0900d23e0 0x7fb07c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.034+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0900d0600 con 0x7fb07c004850 2023-03-03T08:02:20.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.034+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c00bac0 con 0x7fb07c003bc0 2023-03-03T08:02:20.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.034+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c00bc30 con 0x7fb0900d23e0 2023-03-03T08:02:20.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.034+0000 7fb09559a700 1 --2- 172.21.15.98:0/2903902298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb07c004850 0x7fb07c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:20.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.034+0000 7fb095d9b700 1 --2- 172.21.15.98:0/2903902298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb07c003bc0 0x7fb07c003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.034+0000 7fb09659c700 1 --2- 172.21.15.98:0/2903902298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0900d23e0 0x7fb07c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.035+0000 7fb09559a700 1 -- 172.21.15.98:0/2903902298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0900d23e0 msgr2=0x7fb07c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.035+0000 7fb09559a700 1 --2- 172.21.15.98:0/2903902298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0900d23e0 0x7fb07c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.035+0000 7fb09559a700 1 -- 172.21.15.98:0/2903902298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb07c003bc0 msgr2=0x7fb07c003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.035+0000 7fb09559a700 1 --2- 172.21.15.98:0/2903902298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb07c003bc0 0x7fb07c003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.035+0000 7fb09559a700 1 -- 172.21.15.98:0/2903902298 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07c0035b0 con 0x7fb07c004850 2023-03-03T08:02:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.035+0000 7fb09659c700 1 --2- 172.21.15.98:0/2903902298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0900d23e0 0x7fb07c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.035+0000 7fb09559a700 1 --2- 172.21.15.98:0/2903902298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb07c004850 0x7fb07c004cc0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fb0900b8fd0 tx=0x7fb07c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab2feba50176e73 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.035+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08000cd20 con 0x7fb07c004850 2023-03-03T08:02:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.035+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb080005960 con 0x7fb07c004850 2023-03-03T08:02:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.035+0000 7fb094d99700 1 -- 172.21.15.98:0/2903902298 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08000cea0 con 0x7fb07c004850 2023-03-03T08:02:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.036+0000 7fb097fff700 1 -- 172.21.15.98:0/2903902298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb07c004850 msgr2=0x7fb07c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.036+0000 7fb097fff700 1 --2- 172.21.15.98:0/2903902298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb07c004850 0x7fb07c004cc0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fb0900b8fd0 tx=0x7fb07c005880 comp rx=0 tx=0).stop 2023-03-03T08:02:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.036+0000 7fb09659c700 1 -- 172.21.15.98:0/2903902298 reap_dead start 2023-03-03T08:02:20.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.036+0000 7fb097fff700 1 -- 172.21.15.98:0/2903902298 shutdown_connections 2023-03-03T08:02:20.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.036+0000 7fb097fff700 1 -- 172.21.15.98:0/2903902298 >> 172.21.15.98:0/2903902298 conn(0x7fb09006aee0 msgr2=0x7fb09006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:20.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.036+0000 7fb097fff700 1 -- 172.21.15.98:0/2903902298 shutdown_connections 2023-03-03T08:02:20.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.036+0000 7fb097fff700 1 -- 172.21.15.98:0/2903902298 wait complete. 2023-03-03T08:02:20.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.039+0000 7fb097fff700 1 Processor -- start 2023-03-03T08:02:20.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.039+0000 7fb097fff700 1 -- start start 2023-03-03T08:02:20.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.040+0000 7fb097fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb07c004850 0x7fb090149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.040+0000 7fb097fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb090156660 0x7fb09015a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.041+0000 7fb097fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0900ca350 0x7fb090158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.041+0000 7fb097fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0900cb280 con 0x7fb090156660 2023-03-03T08:02:20.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.041+0000 7fb097fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0900cacc0 con 0x7fb07c004850 2023-03-03T08:02:20.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.041+0000 7fb097fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0900cafa0 con 0x7fb0900ca350 2023-03-03T08:02:20.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.041+0000 7fb09559a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb090156660 0x7fb09015a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:20.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.041+0000 7fb09559a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb090156660 0x7fb09015a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48222/0 (socket says 172.21.15.98:48222) 2023-03-03T08:02:20.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.041+0000 7fb09559a700 1 -- 172.21.15.98:0/4277737495 learned_addr learned my addr 172.21.15.98:0/4277737495 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:20.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.041+0000 7fb095d9b700 1 --2- 172.21.15.98:0/4277737495 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb07c004850 0x7fb090149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:20.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.041+0000 7fb09659c700 1 --2- 172.21.15.98:0/4277737495 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0900ca350 0x7fb090158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:20.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.041+0000 7fb09559a700 1 -- 172.21.15.98:0/4277737495 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0900ca350 msgr2=0x7fb090158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:20.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.041+0000 7fb09559a700 1 --2- 172.21.15.98:0/4277737495 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0900ca350 0x7fb090158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.041+0000 7fb09559a700 1 -- 172.21.15.98:0/4277737495 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb07c004850 msgr2=0x7fb090149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:20.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.041+0000 7fb09559a700 1 --2- 172.21.15.98:0/4277737495 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb07c004850 0x7fb090149f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.041+0000 7fb09559a700 1 -- 172.21.15.98:0/4277737495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb090073a80 con 0x7fb090156660 2023-03-03T08:02:20.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.041+0000 7fb095d9b700 1 --2- 172.21.15.98:0/4277737495 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb07c004850 0x7fb090149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:20.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.042+0000 7fb09559a700 1 --2- 172.21.15.98:0/4277737495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb090156660 0x7fb09015a6f0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fb07c004780 tx=0x7fb080001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:20.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.042+0000 7fb0867fc700 1 -- 172.21.15.98:0/4277737495 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080004c80 con 0x7fb090156660 2023-03-03T08:02:20.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.042+0000 7fb0867fc700 1 -- 172.21.15.98:0/4277737495 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb080004e00 con 0x7fb090156660 2023-03-03T08:02:20.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.042+0000 7fb097fff700 1 -- 172.21.15.98:0/4277737495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0900cdc50 con 0x7fb090156660 2023-03-03T08:02:20.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.042+0000 7fb0867fc700 1 -- 172.21.15.98:0/4277737495 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080007310 con 0x7fb090156660 2023-03-03T08:02:20.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.042+0000 7fb097fff700 1 -- 172.21.15.98:0/4277737495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0900cdf30 con 0x7fb090156660 2023-03-03T08:02:20.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.044+0000 7fb097fff700 1 -- 172.21.15.98:0/4277737495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0900729b0 con 0x7fb090156660 2023-03-03T08:02:20.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.045+0000 7fb0867fc700 1 -- 172.21.15.98:0/4277737495 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fb080024030 con 0x7fb090156660 2023-03-03T08:02:20.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.046+0000 7fb0867fc700 1 --2- 172.21.15.98:0/4277737495 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb074068380 0x7fb07406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.047+0000 7fb0867fc700 1 -- 172.21.15.98:0/4277737495 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb080086b50 con 0x7fb090156660 2023-03-03T08:02:20.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.047+0000 7fb095d9b700 1 --2- 172.21.15.98:0/4277737495 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb074068380 0x7fb07406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:20.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.048+0000 7fb095d9b700 1 --2- 172.21.15.98:0/4277737495 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb074068380 0x7fb07406a830 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb08c007fd0 tx=0x7fb08c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.051+0000 7fb0867fc700 1 -- 172.21.15.98:0/4277737495 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb080053690 con 0x7fb090156660 2023-03-03T08:02:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.245+0000 7fb097fff700 1 -- 172.21.15.98:0/4277737495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb09004e760 con 0x7fb090156660 2023-03-03T08:02:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.245+0000 7fb0867fc700 1 -- 172.21.15.98:0/4277737495 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v87) v1 ==== 84+0+126 (secure 0 0 0) 0x7fb080056cb0 con 0x7fb090156660 2023-03-03T08:02:20.247 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:20.247 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T08:02:20.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:20.247 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2023-03-03T08:02:20.247 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2023-03-03T08:02:20.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:20.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:20.248 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 2, 2023-03-03T08:02:20.248 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": "rbd" 2023-03-03T08:02:20.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:20.248 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T08:02:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.247+0000 7fb097fff700 1 -- 172.21.15.98:0/4277737495 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb074068380 msgr2=0x7fb07406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.247+0000 7fb097fff700 1 --2- 172.21.15.98:0/4277737495 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb074068380 0x7fb07406a830 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb08c007fd0 tx=0x7fb08c009040 comp rx=0 tx=0).stop 2023-03-03T08:02:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.247+0000 7fb097fff700 1 -- 172.21.15.98:0/4277737495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb090156660 msgr2=0x7fb09015a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.247+0000 7fb097fff700 1 --2- 172.21.15.98:0/4277737495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb090156660 0x7fb09015a6f0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fb07c004780 tx=0x7fb080001980 comp rx=0 tx=0).stop 2023-03-03T08:02:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.248+0000 7fb097fff700 1 -- 172.21.15.98:0/4277737495 shutdown_connections 2023-03-03T08:02:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.248+0000 7fb097fff700 1 --2- 172.21.15.98:0/4277737495 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0900ca350 0x7fb090158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.248+0000 7fb097fff700 1 --2- 172.21.15.98:0/4277737495 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb07c004850 0x7fb090149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.248+0000 7fb097fff700 1 --2- 172.21.15.98:0/4277737495 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb074068380 0x7fb07406a830 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.248+0000 7fb097fff700 1 --2- 172.21.15.98:0/4277737495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb090156660 0x7fb09015a6f0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.248+0000 7fb097fff700 1 -- 172.21.15.98:0/4277737495 >> 172.21.15.98:0/4277737495 conn(0x7fb09006aee0 msgr2=0x7fb0900d3010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.248+0000 7fb097fff700 1 -- 172.21.15.98:0/4277737495 shutdown_connections 2023-03-03T08:02:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.248+0000 7fb097fff700 1 -- 172.21.15.98:0/4277737495 wait complete. 2023-03-03T08:02:20.275 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-03-03T08:02:20.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-03-03T08:02:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.522+0000 7f584e19d700 1 Processor -- start 2023-03-03T08:02:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.523+0000 7f584e19d700 1 -- start start 2023-03-03T08:02:20.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.524+0000 7f584e19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5848073ed0 0x7f58480ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.524+0000 7f584e19d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58480b3c80 con 0x7f58480d0110 2023-03-03T08:02:20.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.524+0000 7f584e19d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58480b4370 con 0x7f58480b3550 2023-03-03T08:02:20.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.524+0000 7f584e19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58480b3ed0 con 0x7f5848073ed0 2023-03-03T08:02:20.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.524+0000 7f584c99a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f58480d0110 0x7f58480cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58738/0 (socket says 172.21.15.98:58738) 2023-03-03T08:02:20.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.524+0000 7f584c99a700 1 -- 172.21.15.98:0/3866637110 learned_addr learned my addr 172.21.15.98:0/3866637110 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:20.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.525+0000 7f584d99c700 1 --2- 172.21.15.98:0/3866637110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5848073ed0 0x7f58480ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:20.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.525+0000 7f583ffff700 1 -- 172.21.15.98:0/3866637110 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310822555 0 0) 0x7f58480b3c80 con 0x7f58480d0110 2023-03-03T08:02:20.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.525+0000 7f583ffff700 1 -- 172.21.15.98:0/3866637110 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58340035b0 con 0x7f58480d0110 2023-03-03T08:02:20.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.525+0000 7f583ffff700 1 -- 172.21.15.98:0/3866637110 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 212031691 0 0) 0x7f58340035b0 con 0x7f58480d0110 2023-03-03T08:02:20.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.525+0000 7f583ffff700 1 -- 172.21.15.98:0/3866637110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5848073ed0 msgr2=0x7f58480ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:20.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.525+0000 7f583ffff700 1 --2- 172.21.15.98:0/3866637110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5848073ed0 0x7f58480ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.525+0000 7f583ffff700 1 -- 172.21.15.98:0/3866637110 >> v1:172.21.15.100:6789/0 conn(0x7f58480b3550 legacy=0x7f58480cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:20.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.525+0000 7f583ffff700 1 -- 172.21.15.98:0/3866637110 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58480cd640 con 0x7f58480d0110 2023-03-03T08:02:20.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.525+0000 7f583ffff700 1 -- 172.21.15.98:0/3866637110 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f5838001670 con 0x7f58480d0110 2023-03-03T08:02:20.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.525+0000 7f583ffff700 1 -- 172.21.15.98:0/3866637110 >> v1:172.21.15.98:6789/0 conn(0x7f58480d0110 legacy=0x7f58480cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:20.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.525+0000 7f583ffff700 1 --2- 172.21.15.98:0/3866637110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5834003a30 0x7f5834003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.525+0000 7f583ffff700 1 --2- 172.21.15.98:0/3866637110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5848073ed0 0x7f5834004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.525+0000 7f583ffff700 1 --2- 172.21.15.98:0/3866637110 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5834004d90 0x7f5834005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.526+0000 7f583ffff700 1 -- 172.21.15.98:0/3866637110 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58480b3ed0 con 0x7f5834003a30 2023-03-03T08:02:20.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.526+0000 7f583ffff700 1 -- 172.21.15.98:0/3866637110 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583400ba10 con 0x7f5834004d90 2023-03-03T08:02:20.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.526+0000 7f583ffff700 1 -- 172.21.15.98:0/3866637110 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583400bba0 con 0x7f5848073ed0 2023-03-03T08:02:20.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.526+0000 7f584d19b700 1 --2- 172.21.15.98:0/3866637110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5834003a30 0x7f5834003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:20.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.526+0000 7f584c99a700 1 --2- 172.21.15.98:0/3866637110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5848073ed0 0x7f5834004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:20.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.526+0000 7f584d99c700 1 --2- 172.21.15.98:0/3866637110 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5834004d90 0x7f5834005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:20.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.526+0000 7f584d19b700 1 -- 172.21.15.98:0/3866637110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5848073ed0 msgr2=0x7f5834004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:20.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.526+0000 7f584d19b700 1 --2- 172.21.15.98:0/3866637110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5848073ed0 0x7f5834004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.526+0000 7f584d19b700 1 -- 172.21.15.98:0/3866637110 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5834004d90 msgr2=0x7f5834005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:20.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.526+0000 7f584d19b700 1 --2- 172.21.15.98:0/3866637110 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5834004d90 0x7f5834005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.526+0000 7f584d19b700 1 -- 172.21.15.98:0/3866637110 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58340035b0 con 0x7f5834003a30 2023-03-03T08:02:20.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.527+0000 7f584d19b700 1 --2- 172.21.15.98:0/3866637110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5834003a30 0x7f5834003e00 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f58440018d0 tx=0x7f5834005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f7c709cc7994655 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:20.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.527+0000 7f583ffff700 1 -- 172.21.15.98:0/3866637110 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584400c690 con 0x7f5834003a30 2023-03-03T08:02:20.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.527+0000 7f583ffff700 1 -- 172.21.15.98:0/3866637110 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584400c810 con 0x7f5834003a30 2023-03-03T08:02:20.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.527+0000 7f583ffff700 1 -- 172.21.15.98:0/3866637110 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584400cbc0 con 0x7f5834003a30 2023-03-03T08:02:20.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.527+0000 7f584e19d700 1 -- 172.21.15.98:0/3866637110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5834003a30 msgr2=0x7f5834003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:20.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.527+0000 7f584e19d700 1 --2- 172.21.15.98:0/3866637110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5834003a30 0x7f5834003e00 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f58440018d0 tx=0x7f5834005880 comp rx=0 tx=0).stop 2023-03-03T08:02:20.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.527+0000 7f584d99c700 1 -- 172.21.15.98:0/3866637110 reap_dead start 2023-03-03T08:02:20.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.528+0000 7f584e19d700 1 -- 172.21.15.98:0/3866637110 shutdown_connections 2023-03-03T08:02:20.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.528+0000 7f584e19d700 1 -- 172.21.15.98:0/3866637110 >> 172.21.15.98:0/3866637110 conn(0x7f5848073b00 msgr2=0x7f58480d2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:20.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.528+0000 7f584e19d700 1 -- 172.21.15.98:0/3866637110 shutdown_connections 2023-03-03T08:02:20.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.528+0000 7f584e19d700 1 -- 172.21.15.98:0/3866637110 wait complete. 2023-03-03T08:02:20.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.530+0000 7f584e19d700 1 Processor -- start 2023-03-03T08:02:20.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.530+0000 7f584e19d700 1 -- start start 2023-03-03T08:02:20.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.531+0000 7f584e19d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5834003a30 0x7f58480c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.531+0000 7f584e19d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f584815a270 0x7f5848159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.531+0000 7f584e19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f584806be60 0x7f584806c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.531+0000 7f584e19d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584806c840 con 0x7f5834003a30 2023-03-03T08:02:20.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.531+0000 7f584e19d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58480c7cf0 con 0x7f584815a270 2023-03-03T08:02:20.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.531+0000 7f584e19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58480c7fd0 con 0x7f584806be60 2023-03-03T08:02:20.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.531+0000 7f584d19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5834003a30 0x7f58480c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:20.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.531+0000 7f584d19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5834003a30 0x7f58480c8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48238/0 (socket says 172.21.15.98:48238) 2023-03-03T08:02:20.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.531+0000 7f584d19b700 1 -- 172.21.15.98:0/2722965364 learned_addr learned my addr 172.21.15.98:0/2722965364 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:20.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.532+0000 7f584d19b700 1 -- 172.21.15.98:0/2722965364 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f584806be60 msgr2=0x7f584806c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:20.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.532+0000 7f584d19b700 1 --2- 172.21.15.98:0/2722965364 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f584806be60 0x7f584806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.532+0000 7f584d19b700 1 -- 172.21.15.98:0/2722965364 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f584815a270 msgr2=0x7f5848159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:20.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.532+0000 7f584d19b700 1 --2- 172.21.15.98:0/2722965364 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f584815a270 0x7f5848159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.532+0000 7f584d19b700 1 -- 172.21.15.98:0/2722965364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58480d3d60 con 0x7f5834003a30 2023-03-03T08:02:20.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.532+0000 7f584d19b700 1 --2- 172.21.15.98:0/2722965364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5834003a30 0x7f58480c8d90 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f5834005800 tx=0x7f5844001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:20.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.532+0000 7f583dffb700 1 -- 172.21.15.98:0/2722965364 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584400cb80 con 0x7f5834003a30 2023-03-03T08:02:20.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.532+0000 7f583dffb700 1 -- 172.21.15.98:0/2722965364 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5844021070 con 0x7f5834003a30 2023-03-03T08:02:20.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.532+0000 7f583dffb700 1 -- 172.21.15.98:0/2722965364 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584400cee0 con 0x7f5834003a30 2023-03-03T08:02:20.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.533+0000 7f584e19d700 1 -- 172.21.15.98:0/2722965364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58480cd920 con 0x7f5834003a30 2023-03-03T08:02:20.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.533+0000 7f584e19d700 1 -- 172.21.15.98:0/2722965364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58480c8590 con 0x7f5834003a30 2023-03-03T08:02:20.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.534+0000 7f584e19d700 1 -- 172.21.15.98:0/2722965364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58480d3ef0 con 0x7f5834003a30 2023-03-03T08:02:20.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.535+0000 7f583dffb700 1 -- 172.21.15.98:0/2722965364 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f584402c020 con 0x7f5834003a30 2023-03-03T08:02:20.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.537+0000 7f583dffb700 1 --2- 172.21.15.98:0/2722965364 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f582c068380 0x7f582c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:20.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.537+0000 7f583dffb700 1 -- 172.21.15.98:0/2722965364 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5844086d60 con 0x7f5834003a30 2023-03-03T08:02:20.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.537+0000 7f584c99a700 1 --2- 172.21.15.98:0/2722965364 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f582c068380 0x7f582c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:20.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.538+0000 7f584c99a700 1 --2- 172.21.15.98:0/2722965364 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f582c068380 0x7f582c06a830 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f58380011e0 tx=0x7f5838001b40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:20.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.541+0000 7f583dffb700 1 -- 172.21.15.98:0/2722965364 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f58480d3ef0 con 0x7f5834003a30 2023-03-03T08:02:20.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.734+0000 7f584e19d700 1 -- 172.21.15.98:0/2722965364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f58480d3ef0 con 0x7f5834003a30 2023-03-03T08:02:20.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.854+0000 7f583dffb700 1 -- 172.21.15.98:0/2722965364 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v88) v1 ==== 138+0+0 (secure 0 0 0) 0x7f58480d3ef0 con 0x7f5834003a30 2023-03-03T08:02:20.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.858+0000 7f584e19d700 1 -- 172.21.15.98:0/2722965364 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f582c068380 msgr2=0x7f582c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:20.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.858+0000 7f584e19d700 1 --2- 172.21.15.98:0/2722965364 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f582c068380 0x7f582c06a830 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f58380011e0 tx=0x7f5838001b40 comp rx=0 tx=0).stop 2023-03-03T08:02:20.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.858+0000 7f584e19d700 1 -- 172.21.15.98:0/2722965364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5834003a30 msgr2=0x7f58480c8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:20.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.858+0000 7f584e19d700 1 --2- 172.21.15.98:0/2722965364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5834003a30 0x7f58480c8d90 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f5834005800 tx=0x7f5844001230 comp rx=0 tx=0).stop 2023-03-03T08:02:20.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.858+0000 7f584e19d700 1 -- 172.21.15.98:0/2722965364 shutdown_connections 2023-03-03T08:02:20.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.858+0000 7f584e19d700 1 --2- 172.21.15.98:0/2722965364 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f582c068380 0x7f582c06a830 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.858+0000 7f584e19d700 1 --2- 172.21.15.98:0/2722965364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5834003a30 0x7f58480c8d90 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.858+0000 7f584e19d700 1 --2- 172.21.15.98:0/2722965364 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f584806be60 0x7f584806c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.858+0000 7f584e19d700 1 --2- 172.21.15.98:0/2722965364 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f584815a270 0x7f5848159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.858+0000 7f584e19d700 1 -- 172.21.15.98:0/2722965364 >> 172.21.15.98:0/2722965364 conn(0x7f5848073b00 msgr2=0x7f58480ceb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.858+0000 7f584e19d700 1 -- 172.21.15.98:0/2722965364 shutdown_connections 2023-03-03T08:02:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:20.858+0000 7f584e19d700 1 -- 172.21.15.98:0/2722965364 wait complete. 2023-03-03T08:02:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2023-03-03T08:02:20.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-03-03T08:02:21.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.137+0000 7f476b3ff700 1 Processor -- start 2023-03-03T08:02:21.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.137+0000 7f476b3ff700 1 -- start start 2023-03-03T08:02:21.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.138+0000 7f476b3ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f476415ca70 0x7f47640cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.138+0000 7f476b3ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47640cf980 con 0x7f476406b4b0 2023-03-03T08:02:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.138+0000 7f476b3ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47640d1800 con 0x7f47640d0270 2023-03-03T08:02:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.138+0000 7f476b3ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47640cd5b0 con 0x7f476415ca70 2023-03-03T08:02:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.139+0000 7f476899a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f476406b4b0 0x7f47640cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58740/0 (socket says 172.21.15.98:58740) 2023-03-03T08:02:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.139+0000 7f476899a700 1 -- 172.21.15.98:0/144457908 learned_addr learned my addr 172.21.15.98:0/144457908 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:21.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.139+0000 7f476919b700 1 --2- 172.21.15.98:0/144457908 >> v2:172.21.15.100:3300/0 conn(0x7f476415ca70 0x7f47640cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:21.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.139+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094579020 0 0) 0x7f47640cf980 con 0x7f476406b4b0 2023-03-03T08:02:21.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.139+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47500035b0 con 0x7f476406b4b0 2023-03-03T08:02:21.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.139+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060453462 0 0) 0x7f47640d1800 con 0x7f47640d0270 2023-03-03T08:02:21.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.139+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47640cf980 con 0x7f47640d0270 2023-03-03T08:02:21.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.139+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1835240116 0 0) 0x7f47500035b0 con 0x7f476406b4b0 2023-03-03T08:02:21.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.140+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 >> v2:172.21.15.100:3300/0 conn(0x7f476415ca70 msgr2=0x7f47640cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:21.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.140+0000 7f475bfff700 1 --2- 172.21.15.98:0/144457908 >> v2:172.21.15.100:3300/0 conn(0x7f476415ca70 0x7f47640cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:21.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.140+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 >> v1:172.21.15.100:6789/0 conn(0x7f47640d0270 legacy=0x7f47640d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:21.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.140+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47640cd900 con 0x7f476406b4b0 2023-03-03T08:02:21.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.140+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f4754001670 con 0x7f476406b4b0 2023-03-03T08:02:21.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.140+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 >> v1:172.21.15.98:6789/0 conn(0x7f476406b4b0 legacy=0x7f47640cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:21.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.140+0000 7f475bfff700 1 --2- 172.21.15.98:0/144457908 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4750003b90 0x7f4750003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:21.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.140+0000 7f475bfff700 1 --2- 172.21.15.98:0/144457908 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4750004850 0x7f4750004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:21.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.140+0000 7f475bfff700 1 --2- 172.21.15.98:0/144457908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4750005200 0x7f475000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:21.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.140+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47640cd5b0 con 0x7f4750005200 2023-03-03T08:02:21.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.140+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475000c110 con 0x7f4750003b90 2023-03-03T08:02:21.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.140+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f475000c2d0 con 0x7f4750004850 2023-03-03T08:02:21.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.140+0000 7f476919b700 1 --2- 172.21.15.98:0/144457908 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4750003b90 0x7f4750003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:21.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.140+0000 7f476899a700 1 --2- 172.21.15.98:0/144457908 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4750004850 0x7f4750004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:21.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.140+0000 7f476999c700 1 --2- 172.21.15.98:0/144457908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4750005200 0x7f475000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:21.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.141+0000 7f476919b700 1 -- 172.21.15.98:0/144457908 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4750004850 msgr2=0x7f4750004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:21.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.141+0000 7f476919b700 1 --2- 172.21.15.98:0/144457908 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4750004850 0x7f4750004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:21.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.141+0000 7f476919b700 1 -- 172.21.15.98:0/144457908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4750005200 msgr2=0x7f475000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:21.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.141+0000 7f476919b700 1 --2- 172.21.15.98:0/144457908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4750005200 0x7f475000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:21.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.141+0000 7f476919b700 1 -- 172.21.15.98:0/144457908 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47640cd900 con 0x7f4750003b90 2023-03-03T08:02:21.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.141+0000 7f476999c700 1 -- 172.21.15.98:0/144457908 reap_dead start 2023-03-03T08:02:21.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.141+0000 7f476919b700 1 --2- 172.21.15.98:0/144457908 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4750003b90 0x7f4750003f80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4760009b30 tx=0x7f4750005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6faa380461e0bcf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:21.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.142+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476000e6c0 con 0x7f4750003b90 2023-03-03T08:02:21.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.142+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f476000e820 con 0x7f4750003b90 2023-03-03T08:02:21.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.142+0000 7f475bfff700 1 -- 172.21.15.98:0/144457908 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476000eb90 con 0x7f4750003b90 2023-03-03T08:02:21.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.142+0000 7f476b3ff700 1 -- 172.21.15.98:0/144457908 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4750003b90 msgr2=0x7f4750003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:21.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.142+0000 7f476b3ff700 1 --2- 172.21.15.98:0/144457908 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4750003b90 0x7f4750003f80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4760009b30 tx=0x7f4750005880 comp rx=0 tx=0).stop 2023-03-03T08:02:21.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.142+0000 7f476b3ff700 1 -- 172.21.15.98:0/144457908 shutdown_connections 2023-03-03T08:02:21.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.142+0000 7f476b3ff700 1 --2- 172.21.15.98:0/144457908 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4750003b90 0x7f4750003f80 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:21.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.142+0000 7f476b3ff700 1 -- 172.21.15.98:0/144457908 >> 172.21.15.98:0/144457908 conn(0x7f4764073e70 msgr2=0x7f476406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:21.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.142+0000 7f476b3ff700 1 -- 172.21.15.98:0/144457908 shutdown_connections 2023-03-03T08:02:21.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.142+0000 7f476b3ff700 1 -- 172.21.15.98:0/144457908 wait complete. 2023-03-03T08:02:21.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.144+0000 7f476b3ff700 1 Processor -- start 2023-03-03T08:02:21.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.145+0000 7f476b3ff700 1 -- start start 2023-03-03T08:02:21.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.146+0000 7f476b3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4750003b90 0x7f4764155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:21.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.146+0000 7f476b3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47640b2be0 0x7f47640b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:21.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.146+0000 7f476b3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47640b4910 0x7f47640b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:21.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.146+0000 7f476b3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47640b52f0 con 0x7f47640b4910 2023-03-03T08:02:21.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.146+0000 7f476b3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47640b3660 con 0x7f47640b2be0 2023-03-03T08:02:21.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.146+0000 7f476b3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47640b3940 con 0x7f4750003b90 2023-03-03T08:02:21.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.146+0000 7f476999c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47640b4910 0x7f47640b4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:21.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.146+0000 7f476999c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47640b4910 0x7f47640b4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48266/0 (socket says 172.21.15.98:48266) 2023-03-03T08:02:21.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.146+0000 7f476999c700 1 -- 172.21.15.98:0/2925635404 learned_addr learned my addr 172.21.15.98:0/2925635404 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:21.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.146+0000 7f476919b700 1 --2- 172.21.15.98:0/2925635404 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4750003b90 0x7f4764155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:21.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.146+0000 7f476899a700 1 --2- 172.21.15.98:0/2925635404 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47640b2be0 0x7f47640b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:21.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.146+0000 7f476999c700 1 -- 172.21.15.98:0/2925635404 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4750003b90 msgr2=0x7f4764155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:21.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.147+0000 7f476999c700 1 --2- 172.21.15.98:0/2925635404 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4750003b90 0x7f4764155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:21.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.147+0000 7f476999c700 1 -- 172.21.15.98:0/2925635404 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47640b2be0 msgr2=0x7f47640b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:21.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.147+0000 7f476999c700 1 --2- 172.21.15.98:0/2925635404 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47640b2be0 0x7f47640b3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:21.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.147+0000 7f476999c700 1 -- 172.21.15.98:0/2925635404 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47640d1aa0 con 0x7f47640b4910 2023-03-03T08:02:21.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.147+0000 7f476919b700 1 --2- 172.21.15.98:0/2925635404 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4750003b90 0x7f4764155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:21.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.147+0000 7f476999c700 1 --2- 172.21.15.98:0/2925635404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47640b4910 0x7f47640b4db0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f47640d0690 tx=0x7f475c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:21.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.147+0000 7f4759ffb700 1 -- 172.21.15.98:0/2925635404 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475c003cd0 con 0x7f47640b4910 2023-03-03T08:02:21.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.147+0000 7f4759ffb700 1 -- 172.21.15.98:0/2925635404 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475c003e30 con 0x7f47640b4910 2023-03-03T08:02:21.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.147+0000 7f4759ffb700 1 -- 172.21.15.98:0/2925635404 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475c00dbd0 con 0x7f47640b4910 2023-03-03T08:02:21.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.147+0000 7f476b3ff700 1 -- 172.21.15.98:0/2925635404 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47640cdbe0 con 0x7f47640b4910 2023-03-03T08:02:21.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.147+0000 7f476b3ff700 1 -- 172.21.15.98:0/2925635404 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f476415ce40 con 0x7f47640b4910 2023-03-03T08:02:21.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.148+0000 7f476b3ff700 1 -- 172.21.15.98:0/2925635404 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f476404e6e0 con 0x7f47640b4910 2023-03-03T08:02:21.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.150+0000 7f4759ffb700 1 -- 172.21.15.98:0/2925635404 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f475c00dd30 con 0x7f47640b4910 2023-03-03T08:02:21.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.151+0000 7f4759ffb700 1 --2- 172.21.15.98:0/2925635404 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4748068380 0x7f474806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:21.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.152+0000 7f4759ffb700 1 -- 172.21.15.98:0/2925635404 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5580+0+0 (secure 0 0 0) 0x7f475c0861a0 con 0x7f47640b4910 2023-03-03T08:02:21.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.152+0000 7f476919b700 1 --2- 172.21.15.98:0/2925635404 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4748068380 0x7f474806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:21.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.153+0000 7f476919b700 1 --2- 172.21.15.98:0/2925635404 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4748068380 0x7f474806a830 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f47600090b0 tx=0x7f476001e040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:21.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.156+0000 7f4759ffb700 1 -- 172.21.15.98:0/2925635404 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f475c052b00 con 0x7f47640b4910 2023-03-03T08:02:21.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.348+0000 7f476b3ff700 1 -- 172.21.15.98:0/2925635404 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f47640d15c0 con 0x7f47640b4910 2023-03-03T08:02:21.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.856+0000 7f4759ffb700 1 -- 172.21.15.98:0/2925635404 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v89) v1 ==== 170+0+0 (secure 0 0 0) 0x7f475c084860 con 0x7f47640b4910 2023-03-03T08:02:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.863+0000 7f476b3ff700 1 -- 172.21.15.98:0/2925635404 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4748068380 msgr2=0x7f474806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.863+0000 7f476b3ff700 1 --2- 172.21.15.98:0/2925635404 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4748068380 0x7f474806a830 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f47600090b0 tx=0x7f476001e040 comp rx=0 tx=0).stop 2023-03-03T08:02:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.863+0000 7f476b3ff700 1 -- 172.21.15.98:0/2925635404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47640b4910 msgr2=0x7f47640b4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.863+0000 7f476b3ff700 1 --2- 172.21.15.98:0/2925635404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47640b4910 0x7f47640b4db0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f47640d0690 tx=0x7f475c00d430 comp rx=0 tx=0).stop 2023-03-03T08:02:21.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.864+0000 7f476b3ff700 1 -- 172.21.15.98:0/2925635404 shutdown_connections 2023-03-03T08:02:21.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.864+0000 7f476b3ff700 1 --2- 172.21.15.98:0/2925635404 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4748068380 0x7f474806a830 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:21.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.864+0000 7f476b3ff700 1 --2- 172.21.15.98:0/2925635404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47640b4910 0x7f47640b4db0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:21.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.864+0000 7f476b3ff700 1 --2- 172.21.15.98:0/2925635404 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4750003b90 0x7f4764155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:21.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.864+0000 7f476b3ff700 1 --2- 172.21.15.98:0/2925635404 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47640b2be0 0x7f47640b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:21.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.864+0000 7f476b3ff700 1 -- 172.21.15.98:0/2925635404 >> 172.21.15.98:0/2925635404 conn(0x7f4764073e70 msgr2=0x7f47640b83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:21.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.866+0000 7f476b3ff700 1 -- 172.21.15.98:0/2925635404 shutdown_connections 2023-03-03T08:02:21.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:21.867+0000 7f476b3ff700 1 -- 172.21.15.98:0/2925635404 wait complete. 2023-03-03T08:02:21.877 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2023-03-03T08:02:21.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-03-03T08:02:22.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.152+0000 7fe3f7dff700 1 Processor -- start 2023-03-03T08:02:22.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.152+0000 7fe3f7dff700 1 -- start start 2023-03-03T08:02:22.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.153+0000 7fe3f7dff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe3f00d0270 0x7fe3f00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:22.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.153+0000 7fe3f7dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3f006b4b0 0x7fe3f00d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:22.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.153+0000 7fe3f7dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f00cf980 con 0x7fe3f015ca70 2023-03-03T08:02:22.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.153+0000 7fe3f7dff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f00cd5b0 con 0x7fe3f00d0270 2023-03-03T08:02:22.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.153+0000 7fe3f7dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f00d1800 con 0x7fe3f006b4b0 2023-03-03T08:02:22.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.154+0000 7fe3f5b9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe3f015ca70 0x7fe3f00cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58750/0 (socket says 172.21.15.98:58750) 2023-03-03T08:02:22.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.154+0000 7fe3f5b9b700 1 -- 172.21.15.98:0/3400179672 learned_addr learned my addr 172.21.15.98:0/3400179672 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:22.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.154+0000 7fe3f539a700 1 --2- 172.21.15.98:0/3400179672 >> v2:172.21.15.100:3300/0 conn(0x7fe3f00d0270 0x7fe3f00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:22.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.154+0000 7fe3f639c700 1 --2- 172.21.15.98:0/3400179672 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3f006b4b0 0x7fe3f00d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:22.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.154+0000 7fe3f4b99700 1 -- 172.21.15.98:0/3400179672 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781004673 0 0) 0x7fe3f00cf980 con 0x7fe3f015ca70 2023-03-03T08:02:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.154+0000 7fe3f4b99700 1 -- 172.21.15.98:0/3400179672 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3dc0035b0 con 0x7fe3f015ca70 2023-03-03T08:02:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.154+0000 7fe3f4b99700 1 -- 172.21.15.98:0/3400179672 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 138276426 0 0) 0x7fe3dc0035b0 con 0x7fe3f015ca70 2023-03-03T08:02:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.154+0000 7fe3f4b99700 1 -- 172.21.15.98:0/3400179672 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3f006b4b0 msgr2=0x7fe3f00d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.154+0000 7fe3f4b99700 1 --2- 172.21.15.98:0/3400179672 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3f006b4b0 0x7fe3f00d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.154+0000 7fe3f4b99700 1 -- 172.21.15.98:0/3400179672 >> v2:172.21.15.100:3300/0 conn(0x7fe3f00d0270 msgr2=0x7fe3f00cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.154+0000 7fe3f4b99700 1 --2- 172.21.15.98:0/3400179672 >> v2:172.21.15.100:3300/0 conn(0x7fe3f00d0270 0x7fe3f00cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.154+0000 7fe3f4b99700 1 -- 172.21.15.98:0/3400179672 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f00cd900 con 0x7fe3f015ca70 2023-03-03T08:02:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.154+0000 7fe3f4b99700 1 -- 172.21.15.98:0/3400179672 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7fe3ec001670 con 0x7fe3f015ca70 2023-03-03T08:02:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.154+0000 7fe3f4b99700 1 -- 172.21.15.98:0/3400179672 >> v1:172.21.15.98:6789/0 conn(0x7fe3f015ca70 legacy=0x7fe3f00cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.155+0000 7fe3f4b99700 1 --2- 172.21.15.98:0/3400179672 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3dc003a00 0x7fe3dc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.155+0000 7fe3f4b99700 1 --2- 172.21.15.98:0/3400179672 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3f006b4b0 0x7fe3dc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.155+0000 7fe3f4b99700 1 --2- 172.21.15.98:0/3400179672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3dc004d90 0x7fe3dc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.155+0000 7fe3f4b99700 1 -- 172.21.15.98:0/3400179672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f00cd5b0 con 0x7fe3dc004d90 2023-03-03T08:02:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.155+0000 7fe3f4b99700 1 -- 172.21.15.98:0/3400179672 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f00d1800 con 0x7fe3dc003a00 2023-03-03T08:02:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.155+0000 7fe3f4b99700 1 -- 172.21.15.98:0/3400179672 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc00ba10 con 0x7fe3f006b4b0 2023-03-03T08:02:22.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.155+0000 7fe3f539a700 1 --2- 172.21.15.98:0/3400179672 >> v2:172.21.15.100:3300/0 conn(0x7fe3f00d0270 0x7fe3f00cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:22.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.155+0000 7fe3f639c700 1 --2- 172.21.15.98:0/3400179672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3dc004d90 0x7fe3dc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:22.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.155+0000 7fe3f5b9b700 1 --2- 172.21.15.98:0/3400179672 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3dc003a00 0x7fe3dc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:22.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.155+0000 7fe3f639c700 1 -- 172.21.15.98:0/3400179672 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3f006b4b0 msgr2=0x7fe3dc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:22.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.155+0000 7fe3f639c700 1 --2- 172.21.15.98:0/3400179672 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3f006b4b0 0x7fe3dc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.155+0000 7fe3f639c700 1 -- 172.21.15.98:0/3400179672 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3dc003a00 msgr2=0x7fe3dc003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:22.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.155+0000 7fe3f639c700 1 --2- 172.21.15.98:0/3400179672 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3dc003a00 0x7fe3dc003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.155+0000 7fe3f639c700 1 -- 172.21.15.98:0/3400179672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc0035b0 con 0x7fe3dc004d90 2023-03-03T08:02:22.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.156+0000 7fe3f639c700 1 --2- 172.21.15.98:0/3400179672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3dc004d90 0x7fe3dc005200 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8009fc0 tx=0x7fe3dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995b3d4584bcaf70 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:22.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.156+0000 7fe3f4b99700 1 -- 172.21.15.98:0/3400179672 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e800ec30 con 0x7fe3dc004d90 2023-03-03T08:02:22.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.156+0000 7fe3f4b99700 1 -- 172.21.15.98:0/3400179672 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e800f040 con 0x7fe3dc004d90 2023-03-03T08:02:22.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.156+0000 7fe3f4b99700 1 -- 172.21.15.98:0/3400179672 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8014480 con 0x7fe3dc004d90 2023-03-03T08:02:22.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.156+0000 7fe3f7dff700 1 -- 172.21.15.98:0/3400179672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3dc004d90 msgr2=0x7fe3dc005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:22.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.156+0000 7fe3f7dff700 1 --2- 172.21.15.98:0/3400179672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3dc004d90 0x7fe3dc005200 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8009fc0 tx=0x7fe3dc005880 comp rx=0 tx=0).stop 2023-03-03T08:02:22.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.156+0000 7fe3f639c700 1 -- 172.21.15.98:0/3400179672 reap_dead start 2023-03-03T08:02:22.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.157+0000 7fe3f7dff700 1 -- 172.21.15.98:0/3400179672 shutdown_connections 2023-03-03T08:02:22.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.157+0000 7fe3f7dff700 1 -- 172.21.15.98:0/3400179672 >> 172.21.15.98:0/3400179672 conn(0x7fe3f0073e70 msgr2=0x7fe3f006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:22.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.157+0000 7fe3f7dff700 1 -- 172.21.15.98:0/3400179672 shutdown_connections 2023-03-03T08:02:22.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.157+0000 7fe3f7dff700 1 -- 172.21.15.98:0/3400179672 wait complete. 2023-03-03T08:02:22.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.159+0000 7fe3f7dff700 1 Processor -- start 2023-03-03T08:02:22.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.159+0000 7fe3f7dff700 1 -- start start 2023-03-03T08:02:22.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.160+0000 7fe3f7dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3dc004d90 0x7fe3f01e6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:22.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.160+0000 7fe3f7dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3f0156300 0x7fe3f0155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:22.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.160+0000 7fe3f7dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3f01556b0 0x7fe3f01433d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:22.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.160+0000 7fe3f7dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f0155b20 con 0x7fe3dc004d90 2023-03-03T08:02:22.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.160+0000 7fe3f7dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f00ca2f0 con 0x7fe3f01556b0 2023-03-03T08:02:22.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.160+0000 7fe3f7dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f00ca5d0 con 0x7fe3f0156300 2023-03-03T08:02:22.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.160+0000 7fe3f5b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3dc004d90 0x7fe3f01e6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:22.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.160+0000 7fe3f5b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3dc004d90 0x7fe3f01e6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48278/0 (socket says 172.21.15.98:48278) 2023-03-03T08:02:22.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.160+0000 7fe3f5b9b700 1 -- 172.21.15.98:0/2252561036 learned_addr learned my addr 172.21.15.98:0/2252561036 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:22.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.161+0000 7fe3f639c700 1 --2- 172.21.15.98:0/2252561036 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3f01556b0 0x7fe3f01433d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:22.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.161+0000 7fe3f539a700 1 --2- 172.21.15.98:0/2252561036 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3f0156300 0x7fe3f0155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:22.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.161+0000 7fe3f5b9b700 1 -- 172.21.15.98:0/2252561036 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3f0156300 msgr2=0x7fe3f0155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:22.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.161+0000 7fe3f5b9b700 1 --2- 172.21.15.98:0/2252561036 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3f0156300 0x7fe3f0155170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.161+0000 7fe3f5b9b700 1 -- 172.21.15.98:0/2252561036 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3f01556b0 msgr2=0x7fe3f01433d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:22.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.161+0000 7fe3f5b9b700 1 --2- 172.21.15.98:0/2252561036 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3f01556b0 0x7fe3f01433d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.161+0000 7fe3f5b9b700 1 -- 172.21.15.98:0/2252561036 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f00d1aa0 con 0x7fe3dc004d90 2023-03-03T08:02:22.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.161+0000 7fe3f5b9b700 1 --2- 172.21.15.98:0/2252561036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3dc004d90 0x7fe3f01e6da0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00e8c0 tx=0x7fe3ec00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:22.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.161+0000 7fe3e67fc700 1 -- 172.21.15.98:0/2252561036 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec00eec0 con 0x7fe3dc004d90 2023-03-03T08:02:22.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.162+0000 7fe3e67fc700 1 -- 172.21.15.98:0/2252561036 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ec00c7e0 con 0x7fe3dc004d90 2023-03-03T08:02:22.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.162+0000 7fe3e67fc700 1 -- 172.21.15.98:0/2252561036 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec00cb70 con 0x7fe3dc004d90 2023-03-03T08:02:22.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.162+0000 7fe3f7dff700 1 -- 172.21.15.98:0/2252561036 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f00cdbe0 con 0x7fe3dc004d90 2023-03-03T08:02:22.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.162+0000 7fe3f7dff700 1 -- 172.21.15.98:0/2252561036 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f015ce40 con 0x7fe3dc004d90 2023-03-03T08:02:22.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.163+0000 7fe3f7dff700 1 -- 172.21.15.98:0/2252561036 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f00b90f0 con 0x7fe3dc004d90 2023-03-03T08:02:22.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.164+0000 7fe3e67fc700 1 -- 172.21.15.98:0/2252561036 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fe3ec00ccd0 con 0x7fe3dc004d90 2023-03-03T08:02:22.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.166+0000 7fe3e67fc700 1 --2- 172.21.15.98:0/2252561036 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe3d4068380 0x7fe3d406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:22.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.166+0000 7fe3e67fc700 1 -- 172.21.15.98:0/2252561036 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5962+0+0 (secure 0 0 0) 0x7fe3ec085e60 con 0x7fe3dc004d90 2023-03-03T08:02:22.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.166+0000 7fe3f539a700 1 --2- 172.21.15.98:0/2252561036 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe3d4068380 0x7fe3d406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:22.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.167+0000 7fe3f539a700 1 --2- 172.21.15.98:0/2252561036 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe3d4068380 0x7fe3d406a830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0004820 tx=0x7fe3e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:22.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.170+0000 7fe3e67fc700 1 -- 172.21.15.98:0/2252561036 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe3ec052550 con 0x7fe3dc004d90 2023-03-03T08:02:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.403+0000 7fe3f7dff700 1 -- 172.21.15.98:0/2252561036 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe3f004e6e0 con 0x7fe3dc004d90 2023-03-03T08:02:22.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.410+0000 7fe3e67fc700 1 -- 172.21.15.98:0/2252561036 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v45) v1 ==== 174+0+0 (secure 0 0 0) 0x7fe3ec055b70 con 0x7fe3dc004d90 2023-03-03T08:02:22.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.429+0000 7fe3f7dff700 1 -- 172.21.15.98:0/2252561036 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe3d4068380 msgr2=0x7fe3d406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:22.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.429+0000 7fe3f7dff700 1 --2- 172.21.15.98:0/2252561036 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe3d4068380 0x7fe3d406a830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0004820 tx=0x7fe3e0008040 comp rx=0 tx=0).stop 2023-03-03T08:02:22.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.429+0000 7fe3f7dff700 1 -- 172.21.15.98:0/2252561036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3dc004d90 msgr2=0x7fe3f01e6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:22.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.429+0000 7fe3f7dff700 1 --2- 172.21.15.98:0/2252561036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3dc004d90 0x7fe3f01e6da0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00e8c0 tx=0x7fe3ec00ec80 comp rx=0 tx=0).stop 2023-03-03T08:02:22.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.433+0000 7fe3f7dff700 1 -- 172.21.15.98:0/2252561036 shutdown_connections 2023-03-03T08:02:22.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.433+0000 7fe3f7dff700 1 --2- 172.21.15.98:0/2252561036 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe3d4068380 0x7fe3d406a830 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.433+0000 7fe3f7dff700 1 --2- 172.21.15.98:0/2252561036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3dc004d90 0x7fe3f01e6da0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.433+0000 7fe3f7dff700 1 --2- 172.21.15.98:0/2252561036 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe3f01556b0 0x7fe3f01433d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.433+0000 7fe3f7dff700 1 --2- 172.21.15.98:0/2252561036 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3f0156300 0x7fe3f0155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.433+0000 7fe3f7dff700 1 -- 172.21.15.98:0/2252561036 >> 172.21.15.98:0/2252561036 conn(0x7fe3f0073e70 msgr2=0x7fe3f00d2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:22.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.438+0000 7fe3f7dff700 1 -- 172.21.15.98:0/2252561036 shutdown_connections 2023-03-03T08:02:22.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.438+0000 7fe3f7dff700 1 -- 172.21.15.98:0/2252561036 wait complete. 2023-03-03T08:02:22.444 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 13 and data pool 14 2023-03-03T08:02:22.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:02:22.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.716+0000 7f14c53ff700 1 Processor -- start 2023-03-03T08:02:22.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.716+0000 7f14c53ff700 1 -- start start 2023-03-03T08:02:22.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.718+0000 7f14c53ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c00b87c0 0x7f14c00cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:22.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.718+0000 7f14c53ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f14c00b5410 0x7f14c00d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:22.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.718+0000 7f14c53ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c00d0ef0 con 0x7f14c00b8b90 2023-03-03T08:02:22.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.718+0000 7f14c53ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c00d0600 con 0x7f14c00b5410 2023-03-03T08:02:22.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.718+0000 7f14c53ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c00cd970 con 0x7f14c00b87c0 2023-03-03T08:02:22.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.718+0000 7f14b7fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f14c00b8b90 0x7f14c00b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58756/0 (socket says 172.21.15.98:58756) 2023-03-03T08:02:22.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.718+0000 7f14b7fff700 1 -- 172.21.15.98:0/2835783682 learned_addr learned my addr 172.21.15.98:0/2835783682 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:22.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.718+0000 7f14bf7fe700 1 --2- 172.21.15.98:0/2835783682 >> v2:172.21.15.98:3300/0 conn(0x7f14c00b5410 0x7f14c00d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:22.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.718+0000 7f14beffd700 1 --2- 172.21.15.98:0/2835783682 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c00b87c0 0x7f14c00cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:22.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.719+0000 7f14be7fc700 1 -- 172.21.15.98:0/2835783682 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671437472 0 0) 0x7f14c00d0ef0 con 0x7f14c00b8b90 2023-03-03T08:02:22.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.719+0000 7f14be7fc700 1 -- 172.21.15.98:0/2835783682 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14ac0035b0 con 0x7f14c00b8b90 2023-03-03T08:02:22.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.719+0000 7f14bf7fe700 1 -- 172.21.15.98:0/2835783682 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c00b87c0 msgr2=0x7f14c00cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:22.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.719+0000 7f14bf7fe700 1 --2- 172.21.15.98:0/2835783682 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c00b87c0 0x7f14c00cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.719+0000 7f14bf7fe700 1 -- 172.21.15.98:0/2835783682 >> v1:172.21.15.98:6789/0 conn(0x7f14c00b8b90 legacy=0x7f14c00b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:22.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.719+0000 7f14bf7fe700 1 -- 172.21.15.98:0/2835783682 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c00cdc50 con 0x7f14c00b5410 2023-03-03T08:02:22.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.719+0000 7f14bf7fe700 1 --2- 172.21.15.98:0/2835783682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c00b5410 0x7f14c00d13d0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f14b8009d90 tx=0x7f14b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6bd459f2d5a8dd6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:22.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.719+0000 7f14be7fc700 1 -- 172.21.15.98:0/2835783682 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b800e070 con 0x7f14c00b5410 2023-03-03T08:02:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.719+0000 7f14be7fc700 1 -- 172.21.15.98:0/2835783682 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b800bb20 con 0x7f14c00b5410 2023-03-03T08:02:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.719+0000 7f14be7fc700 1 -- 172.21.15.98:0/2835783682 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b8012430 con 0x7f14c00b5410 2023-03-03T08:02:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.720+0000 7f14c53ff700 1 -- 172.21.15.98:0/2835783682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c00b5410 msgr2=0x7f14c00d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.720+0000 7f14c53ff700 1 --2- 172.21.15.98:0/2835783682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c00b5410 0x7f14c00d13d0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f14b8009d90 tx=0x7f14b800b330 comp rx=0 tx=0).stop 2023-03-03T08:02:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.720+0000 7f14c53ff700 1 -- 172.21.15.98:0/2835783682 shutdown_connections 2023-03-03T08:02:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.720+0000 7f14c53ff700 1 --2- 172.21.15.98:0/2835783682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c00b5410 0x7f14c00d13d0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.720+0000 7f14c53ff700 1 --2- 172.21.15.98:0/2835783682 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c00b87c0 0x7f14c00cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.720+0000 7f14c53ff700 1 -- 172.21.15.98:0/2835783682 >> 172.21.15.98:0/2835783682 conn(0x7f14c006aee0 msgr2=0x7f14c006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.720+0000 7f14c53ff700 1 -- 172.21.15.98:0/2835783682 shutdown_connections 2023-03-03T08:02:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.720+0000 7f14c53ff700 1 -- 172.21.15.98:0/2835783682 wait complete. 2023-03-03T08:02:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.722+0000 7f14c53ff700 1 Processor -- start 2023-03-03T08:02:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.722+0000 7f14c53ff700 1 -- start start 2023-03-03T08:02:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.723+0000 7f14c53ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c00b5410 0x7f14c00ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.723+0000 7f14c53ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c00b87c0 0x7f14c015e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.723+0000 7f14c53ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c00b8b90 0x7f14c00c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.723+0000 7f14c53ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c00acd20 con 0x7f14c00b8b90 2023-03-03T08:02:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.723+0000 7f14c53ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c00ab740 con 0x7f14c00b5410 2023-03-03T08:02:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.723+0000 7f14c53ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c00aba20 con 0x7f14c00b87c0 2023-03-03T08:02:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.724+0000 7f14bf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c00b8b90 0x7f14c00c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.724+0000 7f14bf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c00b8b90 0x7f14c00c8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48298/0 (socket says 172.21.15.98:48298) 2023-03-03T08:02:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.724+0000 7f14bf7fe700 1 -- 172.21.15.98:0/2504268357 learned_addr learned my addr 172.21.15.98:0/2504268357 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:22.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.724+0000 7f14beffd700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c00b5410 0x7f14c00ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:22.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.724+0000 7f14bf7fe700 1 -- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c00b87c0 msgr2=0x7f14c015e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:22.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.724+0000 7f14b7fff700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c00b87c0 0x7f14c015e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:22.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.724+0000 7f14bf7fe700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c00b87c0 0x7f14c015e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.724+0000 7f14bf7fe700 1 -- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c00b5410 msgr2=0x7f14c00ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:22.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.724+0000 7f14bf7fe700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c00b5410 0x7f14c00ac7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.724+0000 7f14bf7fe700 1 -- 172.21.15.98:0/2504268357 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b8009a70 con 0x7f14c00b8b90 2023-03-03T08:02:22.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.724+0000 7f14beffd700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c00b5410 0x7f14c00ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:22.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.724+0000 7f14b7fff700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c00b87c0 0x7f14c015e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:22.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.725+0000 7f14bf7fe700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c00b8b90 0x7f14c00c8f80 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f14b8014040 tx=0x7f14b8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:22.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.725+0000 7f14b77fe700 1 -- 172.21.15.98:0/2504268357 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b80129f0 con 0x7f14c00b8b90 2023-03-03T08:02:22.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.725+0000 7f14b77fe700 1 -- 172.21.15.98:0/2504268357 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b8020070 con 0x7f14c00b8b90 2023-03-03T08:02:22.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.725+0000 7f14b77fe700 1 -- 172.21.15.98:0/2504268357 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b8012db0 con 0x7f14c00b8b90 2023-03-03T08:02:22.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.725+0000 7f14c53ff700 1 -- 172.21.15.98:0/2504268357 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c00cdf30 con 0x7f14c00b8b90 2023-03-03T08:02:22.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.725+0000 7f14c53ff700 1 -- 172.21.15.98:0/2504268357 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c0073a80 con 0x7f14c00b8b90 2023-03-03T08:02:22.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.726+0000 7f14c53ff700 1 -- 172.21.15.98:0/2504268357 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c00ce0c0 con 0x7f14c00b8b90 2023-03-03T08:02:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.727+0000 7f14b77fe700 1 -- 172.21.15.98:0/2504268357 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f14b802e070 con 0x7f14c00b8b90 2023-03-03T08:02:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.729+0000 7f14b77fe700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f14a0068380 0x7f14a006a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.729+0000 7f14b77fe700 1 -- 172.21.15.98:0/2504268357 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f14b8022070 con 0x7f14c00b8b90 2023-03-03T08:02:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.729+0000 7f14beffd700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f14a0068380 0x7f14a006a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.730+0000 7f14beffd700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f14a0068380 0x7f14a006a830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009b60 tx=0x7f14b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:22.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.733+0000 7f14b77fe700 1 -- 172.21.15.98:0/2504268357 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f14b8054250 con 0x7f14c00b8b90 2023-03-03T08:02:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.928+0000 7f14c53ff700 1 -- 172.21.15.98:0/2504268357 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14c00729b0 con 0x7f14c00b8b90 2023-03-03T08:02:22.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.929+0000 7f14b77fe700 1 -- 172.21.15.98:0/2504268357 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+17935 (secure 0 0 0) 0x7f14b802c090 con 0x7f14c00b8b90 2023-03-03T08:02:22.931 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:22.931 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":90,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:02:22.404136+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:02:20.797579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-03-03T08:02:21.411899+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:02:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.932+0000 7f14c53ff700 1 -- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f14a0068380 msgr2=0x7f14a006a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.932+0000 7f14c53ff700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f14a0068380 0x7f14a006a830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009b60 tx=0x7f14b0008040 comp rx=0 tx=0).stop 2023-03-03T08:02:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.932+0000 7f14c53ff700 1 -- 172.21.15.98:0/2504268357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c00b8b90 msgr2=0x7f14c00c8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.932+0000 7f14c53ff700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c00b8b90 0x7f14c00c8f80 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f14b8014040 tx=0x7f14b8004ad0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.932+0000 7f14c53ff700 1 -- 172.21.15.98:0/2504268357 shutdown_connections 2023-03-03T08:02:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.932+0000 7f14c53ff700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f14a0068380 0x7f14a006a830 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.932+0000 7f14c53ff700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c00b8b90 0x7f14c00c8f80 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.932+0000 7f14c53ff700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c00b5410 0x7f14c00ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.932+0000 7f14c53ff700 1 --2- 172.21.15.98:0/2504268357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c00b87c0 0x7f14c015e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.932+0000 7f14c53ff700 1 -- 172.21.15.98:0/2504268357 >> 172.21.15.98:0/2504268357 conn(0x7f14c006aee0 msgr2=0x7f14c00b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.933+0000 7f14c53ff700 1 -- 172.21.15.98:0/2504268357 shutdown_connections 2023-03-03T08:02:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:22.933+0000 7f14c53ff700 1 -- 172.21.15.98:0/2504268357 wait complete. 2023-03-03T08:02:22.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:02:23.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.203+0000 7ff1ecdff700 1 Processor -- start 2023-03-03T08:02:23.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.203+0000 7ff1ecdff700 1 -- start start 2023-03-03T08:02:23.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.204+0000 7ff1ecdff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1e80cd8d0 0x7ff1e80b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.204+0000 7ff1ecdff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff1e80b6610 0x7ff1e80cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.204+0000 7ff1ecdff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e80d1a00 con 0x7ff1e815cc00 2023-03-03T08:02:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.204+0000 7ff1ecdff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e80ce040 con 0x7ff1e80b6610 2023-03-03T08:02:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.204+0000 7ff1ecdff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e8073be0 con 0x7ff1e80cd8d0 2023-03-03T08:02:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.205+0000 7ff1e659c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff1e815cc00 0x7ff1e80d27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58764/0 (socket says 172.21.15.98:58764) 2023-03-03T08:02:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.205+0000 7ff1e659c700 1 -- 172.21.15.98:0/4097741446 learned_addr learned my addr 172.21.15.98:0/4097741446 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.205+0000 7ff1e6d9d700 1 --2- 172.21.15.98:0/4097741446 >> v2:172.21.15.100:3300/0 conn(0x7ff1e80b6610 0x7ff1e80cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.205+0000 7ff1e5d9b700 1 --2- 172.21.15.98:0/4097741446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1e80cd8d0 0x7ff1e80b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.205+0000 7ff1e559a700 1 -- 172.21.15.98:0/4097741446 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229149067 0 0) 0x7ff1e80d1a00 con 0x7ff1e815cc00 2023-03-03T08:02:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.205+0000 7ff1e559a700 1 -- 172.21.15.98:0/4097741446 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1d00035b0 con 0x7ff1e815cc00 2023-03-03T08:02:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.205+0000 7ff1e6d9d700 1 -- 172.21.15.98:0/4097741446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1e80cd8d0 msgr2=0x7ff1e80b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.206+0000 7ff1e6d9d700 1 --2- 172.21.15.98:0/4097741446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1e80cd8d0 0x7ff1e80b60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.206+0000 7ff1e6d9d700 1 -- 172.21.15.98:0/4097741446 >> v1:172.21.15.98:6789/0 conn(0x7ff1e815cc00 legacy=0x7ff1e80d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.206+0000 7ff1e6d9d700 1 -- 172.21.15.98:0/4097741446 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e80d1ce0 con 0x7ff1e80b6610 2023-03-03T08:02:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.206+0000 7ff1e6d9d700 1 --2- 172.21.15.98:0/4097741446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1e80b6610 0x7ff1e80cf4d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009d90 tx=0x7ff1e000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4abd20ba626c38ac server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.206+0000 7ff1e559a700 1 -- 172.21.15.98:0/4097741446 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e000e070 con 0x7ff1e80b6610 2023-03-03T08:02:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.206+0000 7ff1e559a700 1 -- 172.21.15.98:0/4097741446 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e000bb20 con 0x7ff1e80b6610 2023-03-03T08:02:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.206+0000 7ff1e559a700 1 -- 172.21.15.98:0/4097741446 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0012430 con 0x7ff1e80b6610 2023-03-03T08:02:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.207+0000 7ff1ecdff700 1 -- 172.21.15.98:0/4097741446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1e80b6610 msgr2=0x7ff1e80cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.207+0000 7ff1ecdff700 1 --2- 172.21.15.98:0/4097741446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1e80b6610 0x7ff1e80cf4d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009d90 tx=0x7ff1e000b330 comp rx=0 tx=0).stop 2023-03-03T08:02:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.207+0000 7ff1ecdff700 1 -- 172.21.15.98:0/4097741446 shutdown_connections 2023-03-03T08:02:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.207+0000 7ff1ecdff700 1 --2- 172.21.15.98:0/4097741446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1e80b6610 0x7ff1e80cf4d0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.207+0000 7ff1ecdff700 1 --2- 172.21.15.98:0/4097741446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1e80cd8d0 0x7ff1e80b60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.207+0000 7ff1ecdff700 1 -- 172.21.15.98:0/4097741446 >> 172.21.15.98:0/4097741446 conn(0x7ff1e806aee0 msgr2=0x7ff1e806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.207+0000 7ff1ecdff700 1 -- 172.21.15.98:0/4097741446 shutdown_connections 2023-03-03T08:02:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.207+0000 7ff1ecdff700 1 -- 172.21.15.98:0/4097741446 wait complete. 2023-03-03T08:02:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.209+0000 7ff1ecdff700 1 Processor -- start 2023-03-03T08:02:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.209+0000 7ff1ecdff700 1 -- start start 2023-03-03T08:02:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.210+0000 7ff1ecdff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1e80b6610 0x7ff1e8154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.210+0000 7ff1ecdff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1e80cd8d0 0x7ff1e81532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.210+0000 7ff1ecdff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1e815cc00 0x7ff1e81556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.210+0000 7ff1ecdff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e815e7c0 con 0x7ff1e815cc00 2023-03-03T08:02:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.210+0000 7ff1ecdff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e80ca160 con 0x7ff1e80cd8d0 2023-03-03T08:02:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.210+0000 7ff1ecdff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e8154640 con 0x7ff1e80b6610 2023-03-03T08:02:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.211+0000 7ff1e6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1e815cc00 0x7ff1e81556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.211+0000 7ff1e6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1e815cc00 0x7ff1e81556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48312/0 (socket says 172.21.15.98:48312) 2023-03-03T08:02:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.211+0000 7ff1e6d9d700 1 -- 172.21.15.98:0/2191975496 learned_addr learned my addr 172.21.15.98:0/2191975496 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.211+0000 7ff1e659c700 1 --2- 172.21.15.98:0/2191975496 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1e80b6610 0x7ff1e8154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.211+0000 7ff1e5d9b700 1 --2- 172.21.15.98:0/2191975496 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1e80cd8d0 0x7ff1e81532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.211+0000 7ff1e6d9d700 1 -- 172.21.15.98:0/2191975496 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1e80b6610 msgr2=0x7ff1e8154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.211+0000 7ff1e6d9d700 1 --2- 172.21.15.98:0/2191975496 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1e80b6610 0x7ff1e8154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.211+0000 7ff1e6d9d700 1 -- 172.21.15.98:0/2191975496 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1e80cd8d0 msgr2=0x7ff1e81532d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.211+0000 7ff1e6d9d700 1 --2- 172.21.15.98:0/2191975496 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1e80cd8d0 0x7ff1e81532d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.211+0000 7ff1e6d9d700 1 -- 172.21.15.98:0/2191975496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e0009a70 con 0x7ff1e815cc00 2023-03-03T08:02:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.211+0000 7ff1e6d9d700 1 --2- 172.21.15.98:0/2191975496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1e815cc00 0x7ff1e81556e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7ff1e000b370 tx=0x7ff1e0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.212+0000 7ff1d6ffd700 1 -- 172.21.15.98:0/2191975496 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0012430 con 0x7ff1e815cc00 2023-03-03T08:02:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.212+0000 7ff1d6ffd700 1 -- 172.21.15.98:0/2191975496 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e0020070 con 0x7ff1e815cc00 2023-03-03T08:02:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.212+0000 7ff1ecdff700 1 -- 172.21.15.98:0/2191975496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e80d22a0 con 0x7ff1e815cc00 2023-03-03T08:02:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.212+0000 7ff1d6ffd700 1 -- 172.21.15.98:0/2191975496 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e00098f0 con 0x7ff1e815cc00 2023-03-03T08:02:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.212+0000 7ff1ecdff700 1 -- 172.21.15.98:0/2191975496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e80d1fc0 con 0x7ff1e815cc00 2023-03-03T08:02:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.213+0000 7ff1ecdff700 1 -- 172.21.15.98:0/2191975496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e80cdd40 con 0x7ff1e815cc00 2023-03-03T08:02:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.214+0000 7ff1d6ffd700 1 -- 172.21.15.98:0/2191975496 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7ff1e0004040 con 0x7ff1e815cc00 2023-03-03T08:02:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.216+0000 7ff1d6ffd700 1 --2- 172.21.15.98:0/2191975496 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff1c8068380 0x7ff1c806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.216+0000 7ff1d6ffd700 1 -- 172.21.15.98:0/2191975496 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff1e0022070 con 0x7ff1e815cc00 2023-03-03T08:02:23.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.216+0000 7ff1e659c700 1 --2- 172.21.15.98:0/2191975496 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff1c8068380 0x7ff1c806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:23.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.217+0000 7ff1e659c700 1 --2- 172.21.15.98:0/2191975496 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff1c8068380 0x7ff1c806a830 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8007fd0 tx=0x7ff1d8007dc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.220+0000 7ff1d6ffd700 1 -- 172.21.15.98:0/2191975496 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff1e00547c0 con 0x7ff1e815cc00 2023-03-03T08:02:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.417+0000 7ff1ecdff700 1 -- 172.21.15.98:0/2191975496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1e80729b0 con 0x7ff1e815cc00 2023-03-03T08:02:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.418+0000 7ff1d6ffd700 1 -- 172.21.15.98:0/2191975496 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+17922 (secure 0 0 0) 0x7ff1e0057de0 con 0x7ff1e815cc00 2023-03-03T08:02:23.420 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:23.420 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:02:23.406996+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:02:20.797579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-03-03T08:02:21.411899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:02:23.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.421+0000 7ff1ecdff700 1 -- 172.21.15.98:0/2191975496 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff1c8068380 msgr2=0x7ff1c806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:23.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.421+0000 7ff1ecdff700 1 --2- 172.21.15.98:0/2191975496 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff1c8068380 0x7ff1c806a830 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8007fd0 tx=0x7ff1d8007dc0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.421+0000 7ff1ecdff700 1 -- 172.21.15.98:0/2191975496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1e815cc00 msgr2=0x7ff1e81556e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:23.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.421+0000 7ff1ecdff700 1 --2- 172.21.15.98:0/2191975496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1e815cc00 0x7ff1e81556e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7ff1e000b370 tx=0x7ff1e0004ad0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.421+0000 7ff1ecdff700 1 -- 172.21.15.98:0/2191975496 shutdown_connections 2023-03-03T08:02:23.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.421+0000 7ff1ecdff700 1 --2- 172.21.15.98:0/2191975496 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff1c8068380 0x7ff1c806a830 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.421+0000 7ff1ecdff700 1 --2- 172.21.15.98:0/2191975496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1e815cc00 0x7ff1e81556e0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.421+0000 7ff1ecdff700 1 --2- 172.21.15.98:0/2191975496 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1e80b6610 0x7ff1e8154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.421+0000 7ff1ecdff700 1 --2- 172.21.15.98:0/2191975496 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1e80cd8d0 0x7ff1e81532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.421+0000 7ff1ecdff700 1 -- 172.21.15.98:0/2191975496 >> 172.21.15.98:0/2191975496 conn(0x7ff1e806aee0 msgr2=0x7ff1e8156190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.422+0000 7ff1ecdff700 1 -- 172.21.15.98:0/2191975496 shutdown_connections 2023-03-03T08:02:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.422+0000 7ff1ecdff700 1 -- 172.21.15.98:0/2191975496 wait complete. 2023-03-03T08:02:23.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-03-03T08:02:23.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.697+0000 7f73b159d700 1 Processor -- start 2023-03-03T08:02:23.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.697+0000 7f73b159d700 1 -- start start 2023-03-03T08:02:23.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.698+0000 7f73b159d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73ac1609b0 0x7f73ac160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:23.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.698+0000 7f73b159d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f73ac0cf570 0x7f73ac15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:23.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.698+0000 7f73b159d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f73ac15db30 0x7f73ac0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:23.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.698+0000 7f73b159d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ac0cebb0 con 0x7f73ac0cf570 2023-03-03T08:02:23.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.699+0000 7f73b159d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ac1612c0 con 0x7f73ac15db30 2023-03-03T08:02:23.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.699+0000 7f73b159d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac0cd780 con 0x7f73ac1609b0 2023-03-03T08:02:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.699+0000 7f73ab7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f73ac0cf570 0x7f73ac15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.699+0000 7f73ab7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f73ac0cf570 0x7f73ac15d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48322/0 (socket says 172.21.15.98:48322) 2023-03-03T08:02:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.699+0000 7f73ab7fe700 1 -- 172.21.15.98:0/2619513167 learned_addr learned my addr 172.21.15.98:0/2619513167 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.699+0000 7f73abfff700 1 --2- 172.21.15.98:0/2619513167 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73ac1609b0 0x7f73ac160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.699+0000 7f73b0d9c700 1 --2- 172.21.15.98:0/2619513167 >> v2:172.21.15.100:3300/0 conn(0x7f73ac15db30 0x7f73ac0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:23.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.699+0000 7f73ab7fe700 1 -- 172.21.15.98:0/2619513167 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73ac1609b0 msgr2=0x7f73ac160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:23.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.699+0000 7f73ab7fe700 1 --2- 172.21.15.98:0/2619513167 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73ac1609b0 0x7f73ac160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.699+0000 7f73ab7fe700 1 -- 172.21.15.98:0/2619513167 >> v2:172.21.15.100:3300/0 conn(0x7f73ac15db30 msgr2=0x7f73ac0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:23.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.699+0000 7f73ab7fe700 1 --2- 172.21.15.98:0/2619513167 >> v2:172.21.15.100:3300/0 conn(0x7f73ac15db30 0x7f73ac0b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.699+0000 7f73ab7fe700 1 -- 172.21.15.98:0/2619513167 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac0cda60 con 0x7f73ac0cf570 2023-03-03T08:02:23.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.700+0000 7f73ab7fe700 1 --2- 172.21.15.98:0/2619513167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac0cf570 0x7f73ac15d5f0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0d2be0 tx=0x7f73a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2feb41253c8edc4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:23.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.700+0000 7f73aaffd700 1 -- 172.21.15.98:0/2619513167 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0011830 con 0x7f73ac0cf570 2023-03-03T08:02:23.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.700+0000 7f73aaffd700 1 -- 172.21.15.98:0/2619513167 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a0011990 con 0x7f73ac0cf570 2023-03-03T08:02:23.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.700+0000 7f73aaffd700 1 -- 172.21.15.98:0/2619513167 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0011d20 con 0x7f73ac0cf570 2023-03-03T08:02:23.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.701+0000 7f73b159d700 1 -- 172.21.15.98:0/2619513167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac0cf570 msgr2=0x7f73ac15d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:23.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.701+0000 7f73b159d700 1 --2- 172.21.15.98:0/2619513167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac0cf570 0x7f73ac15d5f0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0d2be0 tx=0x7f73a0009bd0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.701+0000 7f73b159d700 1 -- 172.21.15.98:0/2619513167 shutdown_connections 2023-03-03T08:02:23.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.701+0000 7f73b159d700 1 --2- 172.21.15.98:0/2619513167 >> v2:172.21.15.100:3300/0 conn(0x7f73ac15db30 0x7f73ac0b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.701+0000 7f73b159d700 1 --2- 172.21.15.98:0/2619513167 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73ac1609b0 0x7f73ac160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.701+0000 7f73b159d700 1 --2- 172.21.15.98:0/2619513167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac0cf570 0x7f73ac15d5f0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.701+0000 7f73b159d700 1 -- 172.21.15.98:0/2619513167 >> 172.21.15.98:0/2619513167 conn(0x7f73ac06b880 msgr2=0x7f73ac0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:23.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.701+0000 7f73b159d700 1 -- 172.21.15.98:0/2619513167 shutdown_connections 2023-03-03T08:02:23.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.701+0000 7f73b159d700 1 -- 172.21.15.98:0/2619513167 wait complete. 2023-03-03T08:02:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.703+0000 7f73b159d700 1 Processor -- start 2023-03-03T08:02:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.703+0000 7f73b159d700 1 -- start start 2023-03-03T08:02:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.704+0000 7f73b159d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73ac0cf570 0x7f73ac14f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.704+0000 7f73b159d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac15db30 0x7f73ac1f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.704+0000 7f73b159d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73ac1609b0 0x7f73ac14d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.704+0000 7f73b159d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac15cc80 con 0x7f73ac15db30 2023-03-03T08:02:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.704+0000 7f73b159d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac0b3cf0 con 0x7f73ac1609b0 2023-03-03T08:02:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.704+0000 7f73b159d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac1f1ef0 con 0x7f73ac0cf570 2023-03-03T08:02:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.704+0000 7f73ab7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac15db30 0x7f73ac1f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.704+0000 7f73ab7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac15db30 0x7f73ac1f19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48326/0 (socket says 172.21.15.98:48326) 2023-03-03T08:02:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.704+0000 7f73ab7fe700 1 -- 172.21.15.98:0/2745237689 learned_addr learned my addr 172.21.15.98:0/2745237689 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.705+0000 7f73abfff700 1 --2- 172.21.15.98:0/2745237689 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73ac0cf570 0x7f73ac14f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.705+0000 7f73b0d9c700 1 --2- 172.21.15.98:0/2745237689 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73ac1609b0 0x7f73ac14d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.705+0000 7f73ab7fe700 1 -- 172.21.15.98:0/2745237689 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73ac0cf570 msgr2=0x7f73ac14f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.705+0000 7f73ab7fe700 1 --2- 172.21.15.98:0/2745237689 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73ac0cf570 0x7f73ac14f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.705+0000 7f73ab7fe700 1 -- 172.21.15.98:0/2745237689 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73ac1609b0 msgr2=0x7f73ac14d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.705+0000 7f73ab7fe700 1 --2- 172.21.15.98:0/2745237689 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73ac1609b0 0x7f73ac14d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.705+0000 7f73ab7fe700 1 -- 172.21.15.98:0/2745237689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a0009550 con 0x7f73ac15db30 2023-03-03T08:02:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.705+0000 7f73ab7fe700 1 --2- 172.21.15.98:0/2745237689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac15db30 0x7f73ac1f19b0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f73a0000c00 tx=0x7f73a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.706+0000 7f73a8ff9700 1 -- 172.21.15.98:0/2745237689 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0004240 con 0x7f73ac15db30 2023-03-03T08:02:23.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.706+0000 7f73a8ff9700 1 -- 172.21.15.98:0/2745237689 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a001f070 con 0x7f73ac15db30 2023-03-03T08:02:23.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.706+0000 7f73a8ff9700 1 -- 172.21.15.98:0/2745237689 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a00036a0 con 0x7f73ac15db30 2023-03-03T08:02:23.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.706+0000 7f73b159d700 1 -- 172.21.15.98:0/2745237689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ac15cdf0 con 0x7f73ac15db30 2023-03-03T08:02:23.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.706+0000 7f73b159d700 1 -- 172.21.15.98:0/2745237689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ac0cdd40 con 0x7f73ac15db30 2023-03-03T08:02:23.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.707+0000 7f73b159d700 1 -- 172.21.15.98:0/2745237689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73ac15d0d0 con 0x7f73ac15db30 2023-03-03T08:02:23.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.708+0000 7f73a8ff9700 1 -- 172.21.15.98:0/2745237689 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f73a002a020 con 0x7f73ac15db30 2023-03-03T08:02:23.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.709+0000 7f73a8ff9700 1 --2- 172.21.15.98:0/2745237689 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7398068380 0x7f739806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:23.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.710+0000 7f73a8ff9700 1 -- 172.21.15.98:0/2745237689 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f73a0086ff0 con 0x7f73ac15db30 2023-03-03T08:02:23.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.710+0000 7f73abfff700 1 --2- 172.21.15.98:0/2745237689 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7398068380 0x7f739806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:23.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.711+0000 7f73abfff700 1 --2- 172.21.15.98:0/2745237689 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7398068380 0x7f739806a830 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f739c009b40 tx=0x7f739c009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:23.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.713+0000 7f73a8ff9700 1 -- 172.21.15.98:0/2745237689 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f73a00b85e0 con 0x7f73ac15db30 2023-03-03T08:02:23.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:23.953+0000 7f73b159d700 1 -- 172.21.15.98:0/2745237689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f73ac0b3e30 con 0x7f73ac15db30 2023-03-03T08:02:24.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:24.873+0000 7f73a8ff9700 1 -- 172.21.15.98:0/2745237689 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v47) v1 ==== 118+0+0 (secure 0 0 0) 0x7f73a0015020 con 0x7f73ac15db30 2023-03-03T08:02:24.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:24.876+0000 7f73b159d700 1 -- 172.21.15.98:0/2745237689 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7398068380 msgr2=0x7f739806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:24.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:24.876+0000 7f73b159d700 1 --2- 172.21.15.98:0/2745237689 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7398068380 0x7f739806a830 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f739c009b40 tx=0x7f739c009210 comp rx=0 tx=0).stop 2023-03-03T08:02:24.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:24.876+0000 7f73b159d700 1 -- 172.21.15.98:0/2745237689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac15db30 msgr2=0x7f73ac1f19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:24.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:24.876+0000 7f73b159d700 1 --2- 172.21.15.98:0/2745237689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac15db30 0x7f73ac1f19b0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f73a0000c00 tx=0x7f73a0004750 comp rx=0 tx=0).stop 2023-03-03T08:02:24.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:24.876+0000 7f73b159d700 1 -- 172.21.15.98:0/2745237689 shutdown_connections 2023-03-03T08:02:24.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:24.876+0000 7f73b159d700 1 --2- 172.21.15.98:0/2745237689 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73ac1609b0 0x7f73ac14d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:24.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:24.876+0000 7f73b159d700 1 --2- 172.21.15.98:0/2745237689 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73ac0cf570 0x7f73ac14f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:24.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:24.876+0000 7f73b159d700 1 --2- 172.21.15.98:0/2745237689 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7398068380 0x7f739806a830 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:24.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:24.876+0000 7f73b159d700 1 --2- 172.21.15.98:0/2745237689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac15db30 0x7f73ac1f19b0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:24.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:24.876+0000 7f73b159d700 1 -- 172.21.15.98:0/2745237689 >> 172.21.15.98:0/2745237689 conn(0x7f73ac06b880 msgr2=0x7f73ac0d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:24.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:24.876+0000 7f73b159d700 1 -- 172.21.15.98:0/2745237689 shutdown_connections 2023-03-03T08:02:24.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:24.876+0000 7f73b159d700 1 -- 172.21.15.98:0/2745237689 wait complete. 2023-03-03T08:02:24.907 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-03-03T08:02:24.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-03-03T08:02:25.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.155+0000 7fdd9a19d700 1 Processor -- start 2023-03-03T08:02:25.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.155+0000 7fdd9a19d700 1 -- start start 2023-03-03T08:02:25.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.157+0000 7fdd9a19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd940cf570 0x7fdd9415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:25.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.157+0000 7fdd9a19d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdd9415db30 0x7fdd940b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:25.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.157+0000 7fdd9a19d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd940cd780 con 0x7fdd941609b0 2023-03-03T08:02:25.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.157+0000 7fdd9a19d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd940cebb0 con 0x7fdd9415db30 2023-03-03T08:02:25.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.157+0000 7fdd9a19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd940cda60 con 0x7fdd940cf570 2023-03-03T08:02:25.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.158+0000 7fdd9899a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd940cf570 0x7fdd9415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:25.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.158+0000 7fdd9999c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdd9415db30 0x7fdd940b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:25.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.158+0000 7fdd9919b700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fdd941609b0 0x7fdd94160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:33578/0 (socket says 172.21.15.98:33578) 2023-03-03T08:02:25.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.158+0000 7fdd9919b700 1 -- 172.21.15.98:0/2557667881 learned_addr learned my addr 172.21.15.98:0/2557667881 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:25.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.158+0000 7fdd9999c700 1 -- 172.21.15.98:0/2557667881 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd940cf570 msgr2=0x7fdd9415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:25.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.158+0000 7fdd9999c700 1 --2- 172.21.15.98:0/2557667881 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd940cf570 0x7fdd9415d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:25.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.158+0000 7fdd9999c700 1 -- 172.21.15.98:0/2557667881 >> v1:172.21.15.100:6789/0 conn(0x7fdd941609b0 legacy=0x7fdd94160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:25.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.158+0000 7fdd9999c700 1 -- 172.21.15.98:0/2557667881 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd940cdd40 con 0x7fdd9415db30 2023-03-03T08:02:25.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.158+0000 7fdd9999c700 1 --2- 172.21.15.98:0/2557667881 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd9415db30 0x7fdd940b4aa0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c009d90 tx=0x7fdd8c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad08efb51741a800 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:25.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.159+0000 7fdd8bfff700 1 -- 172.21.15.98:0/2557667881 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c00e070 con 0x7fdd9415db30 2023-03-03T08:02:25.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.159+0000 7fdd8bfff700 1 -- 172.21.15.98:0/2557667881 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8c00bb20 con 0x7fdd9415db30 2023-03-03T08:02:25.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.159+0000 7fdd8bfff700 1 -- 172.21.15.98:0/2557667881 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c012430 con 0x7fdd9415db30 2023-03-03T08:02:25.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.159+0000 7fdd9a19d700 1 -- 172.21.15.98:0/2557667881 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd9415db30 msgr2=0x7fdd940b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:25.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.159+0000 7fdd9a19d700 1 --2- 172.21.15.98:0/2557667881 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd9415db30 0x7fdd940b4aa0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c009d90 tx=0x7fdd8c00b330 comp rx=0 tx=0).stop 2023-03-03T08:02:25.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.159+0000 7fdd9a19d700 1 -- 172.21.15.98:0/2557667881 shutdown_connections 2023-03-03T08:02:25.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.159+0000 7fdd9a19d700 1 --2- 172.21.15.98:0/2557667881 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd9415db30 0x7fdd940b4aa0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:25.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.160+0000 7fdd9a19d700 1 --2- 172.21.15.98:0/2557667881 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd940cf570 0x7fdd9415d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:25.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.160+0000 7fdd9a19d700 1 -- 172.21.15.98:0/2557667881 >> 172.21.15.98:0/2557667881 conn(0x7fdd9406b880 msgr2=0x7fdd940cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:25.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.160+0000 7fdd9a19d700 1 -- 172.21.15.98:0/2557667881 shutdown_connections 2023-03-03T08:02:25.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.160+0000 7fdd9a19d700 1 -- 172.21.15.98:0/2557667881 wait complete. 2023-03-03T08:02:25.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.162+0000 7fdd9a19d700 1 Processor -- start 2023-03-03T08:02:25.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.162+0000 7fdd9a19d700 1 -- start start 2023-03-03T08:02:25.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.163+0000 7fdd9a19d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd940cf570 0x7fdd94145000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:25.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.163+0000 7fdd9a19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd9415db30 0x7fdd940b7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:25.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.163+0000 7fdd9a19d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd941609b0 0x7fdd94143340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:25.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.163+0000 7fdd9a19d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd94142bd0 con 0x7fdd940cf570 2023-03-03T08:02:25.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.163+0000 7fdd9a19d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd94142eb0 con 0x7fdd941609b0 2023-03-03T08:02:25.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.163+0000 7fdd9a19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd94145540 con 0x7fdd9415db30 2023-03-03T08:02:25.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.163+0000 7fdd9919b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd940cf570 0x7fdd94145000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:25.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.163+0000 7fdd9919b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd940cf570 0x7fdd94145000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43592/0 (socket says 172.21.15.98:43592) 2023-03-03T08:02:25.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.163+0000 7fdd9919b700 1 -- 172.21.15.98:0/3692523940 learned_addr learned my addr 172.21.15.98:0/3692523940 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:25.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.163+0000 7fdd9899a700 1 --2- 172.21.15.98:0/3692523940 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd9415db30 0x7fdd940b7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:25.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.163+0000 7fdd9999c700 1 --2- 172.21.15.98:0/3692523940 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd941609b0 0x7fdd94143340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:25.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.164+0000 7fdd9919b700 1 -- 172.21.15.98:0/3692523940 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd9415db30 msgr2=0x7fdd940b7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:25.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.164+0000 7fdd9919b700 1 --2- 172.21.15.98:0/3692523940 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd9415db30 0x7fdd940b7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:25.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.164+0000 7fdd9919b700 1 -- 172.21.15.98:0/3692523940 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd941609b0 msgr2=0x7fdd94143340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:25.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.164+0000 7fdd9919b700 1 --2- 172.21.15.98:0/3692523940 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd941609b0 0x7fdd94143340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:25.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.164+0000 7fdd9919b700 1 -- 172.21.15.98:0/3692523940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8c009a70 con 0x7fdd940cf570 2023-03-03T08:02:25.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.164+0000 7fdd9919b700 1 --2- 172.21.15.98:0/3692523940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd940cf570 0x7fdd94145000 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000c4b0 tx=0x7fdd9000c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:25.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.164+0000 7fdd89ffb700 1 -- 172.21.15.98:0/3692523940 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90005d70 con 0x7fdd940cf570 2023-03-03T08:02:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.164+0000 7fdd89ffb700 1 -- 172.21.15.98:0/3692523940 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd900070d0 con 0x7fdd940cf570 2023-03-03T08:02:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.164+0000 7fdd89ffb700 1 -- 172.21.15.98:0/3692523940 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90007440 con 0x7fdd940cf570 2023-03-03T08:02:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.164+0000 7fdd9a19d700 1 -- 172.21.15.98:0/3692523940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9415d0d0 con 0x7fdd940cf570 2023-03-03T08:02:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.164+0000 7fdd9a19d700 1 -- 172.21.15.98:0/3692523940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9415cdf0 con 0x7fdd940cf570 2023-03-03T08:02:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.166+0000 7fdd9a19d700 1 -- 172.21.15.98:0/3692523940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd940b4fe0 con 0x7fdd940cf570 2023-03-03T08:02:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.167+0000 7fdd89ffb700 1 -- 172.21.15.98:0/3692523940 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fdd90019030 con 0x7fdd940cf570 2023-03-03T08:02:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.168+0000 7fdd89ffb700 1 --2- 172.21.15.98:0/3692523940 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdd78068380 0x7fdd7806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.168+0000 7fdd89ffb700 1 -- 172.21.15.98:0/3692523940 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdd90084f90 con 0x7fdd940cf570 2023-03-03T08:02:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.169+0000 7fdd9899a700 1 --2- 172.21.15.98:0/3692523940 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdd78068380 0x7fdd7806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.170+0000 7fdd9899a700 1 --2- 172.21.15.98:0/3692523940 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdd78068380 0x7fdd7806a830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdd941f58e0 tx=0x7fdd84008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:25.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.172+0000 7fdd89ffb700 1 -- 172.21.15.98:0/3692523940 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdd9004b5e0 con 0x7fdd940cf570 2023-03-03T08:02:25.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.407+0000 7fdd9a19d700 1 -- 172.21.15.98:0/3692523940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fdd94068290 con 0x7fdd940cf570 2023-03-03T08:02:25.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.875+0000 7fdd89ffb700 1 -- 172.21.15.98:0/3692523940 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v48) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdd900170c0 con 0x7fdd940cf570 2023-03-03T08:02:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.878+0000 7fdd9a19d700 1 -- 172.21.15.98:0/3692523940 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdd78068380 msgr2=0x7fdd7806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.878+0000 7fdd9a19d700 1 --2- 172.21.15.98:0/3692523940 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdd78068380 0x7fdd7806a830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdd941f58e0 tx=0x7fdd84008040 comp rx=0 tx=0).stop 2023-03-03T08:02:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.878+0000 7fdd9a19d700 1 -- 172.21.15.98:0/3692523940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd940cf570 msgr2=0x7fdd94145000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.878+0000 7fdd9a19d700 1 --2- 172.21.15.98:0/3692523940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd940cf570 0x7fdd94145000 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000c4b0 tx=0x7fdd9000c870 comp rx=0 tx=0).stop 2023-03-03T08:02:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.878+0000 7fdd9a19d700 1 -- 172.21.15.98:0/3692523940 shutdown_connections 2023-03-03T08:02:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.878+0000 7fdd9a19d700 1 --2- 172.21.15.98:0/3692523940 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdd78068380 0x7fdd7806a830 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.878+0000 7fdd9a19d700 1 --2- 172.21.15.98:0/3692523940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd940cf570 0x7fdd94145000 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.878+0000 7fdd9a19d700 1 --2- 172.21.15.98:0/3692523940 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd941609b0 0x7fdd94143340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.878+0000 7fdd9a19d700 1 --2- 172.21.15.98:0/3692523940 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd9415db30 0x7fdd940b7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.878+0000 7fdd9a19d700 1 -- 172.21.15.98:0/3692523940 >> 172.21.15.98:0/3692523940 conn(0x7fdd9406b880 msgr2=0x7fdd940d1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.879+0000 7fdd9a19d700 1 -- 172.21.15.98:0/3692523940 shutdown_connections 2023-03-03T08:02:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:25.879+0000 7fdd9a19d700 1 -- 172.21.15.98:0/3692523940 wait complete. 2023-03-03T08:02:25.908 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-03-03T08:02:25.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:02:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.151+0000 7f10bb19d700 1 Processor -- start 2023-03-03T08:02:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.151+0000 7f10bb19d700 1 -- start start 2023-03-03T08:02:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.152+0000 7f10bb19d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f10b4146360 0x7f10b4160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.152+0000 7f10bb19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10b40b4ab0 0x7f10b40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.152+0000 7f10bb19d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f10b40731e0 0x7f10b4073660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.152+0000 7f10bb19d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b41614c0 con 0x7f10b40731e0 2023-03-03T08:02:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.152+0000 7f10bb19d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b40b5440 con 0x7f10b4146360 2023-03-03T08:02:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.152+0000 7f10bb19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b40d10f0 con 0x7f10b40b4ab0 2023-03-03T08:02:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.153+0000 7f10ba99c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f10b40731e0 0x7f10b4073660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.153+0000 7f10ba99c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f10b40731e0 0x7f10b4073660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43596/0 (socket says 172.21.15.98:43596) 2023-03-03T08:02:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.153+0000 7f10ba99c700 1 -- 172.21.15.98:0/1248445105 learned_addr learned my addr 172.21.15.98:0/1248445105 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.153+0000 7f10b999a700 1 --2- 172.21.15.98:0/1248445105 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10b40b4ab0 0x7f10b40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.153+0000 7f10ba19b700 1 --2- 172.21.15.98:0/1248445105 >> v2:172.21.15.100:3300/0 conn(0x7f10b4146360 0x7f10b4160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.153+0000 7f10ba99c700 1 -- 172.21.15.98:0/1248445105 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10b40b4ab0 msgr2=0x7f10b40b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.153+0000 7f10ba99c700 1 --2- 172.21.15.98:0/1248445105 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10b40b4ab0 0x7f10b40b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.153+0000 7f10ba99c700 1 -- 172.21.15.98:0/1248445105 >> v2:172.21.15.100:3300/0 conn(0x7f10b4146360 msgr2=0x7f10b4160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.153+0000 7f10ba99c700 1 --2- 172.21.15.98:0/1248445105 >> v2:172.21.15.100:3300/0 conn(0x7f10b4146360 0x7f10b4160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.153+0000 7f10ba99c700 1 -- 172.21.15.98:0/1248445105 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b40cdf20 con 0x7f10b40731e0 2023-03-03T08:02:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.153+0000 7f10ba99c700 1 --2- 172.21.15.98:0/1248445105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40731e0 0x7f10b4073660 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f10b40743f0 tx=0x7f10ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f2361793b1cd238 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:26.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.154+0000 7f10b9199700 1 -- 172.21.15.98:0/1248445105 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac00e070 con 0x7f10b40731e0 2023-03-03T08:02:26.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.154+0000 7f10b9199700 1 -- 172.21.15.98:0/1248445105 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10ac00b910 con 0x7f10b40731e0 2023-03-03T08:02:26.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.154+0000 7f10b9199700 1 -- 172.21.15.98:0/1248445105 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac00bcc0 con 0x7f10b40731e0 2023-03-03T08:02:26.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.154+0000 7f10bb19d700 1 -- 172.21.15.98:0/1248445105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40731e0 msgr2=0x7f10b4073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:26.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.154+0000 7f10bb19d700 1 --2- 172.21.15.98:0/1248445105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40731e0 0x7f10b4073660 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f10b40743f0 tx=0x7f10ac00b330 comp rx=0 tx=0).stop 2023-03-03T08:02:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.154+0000 7f10bb19d700 1 -- 172.21.15.98:0/1248445105 shutdown_connections 2023-03-03T08:02:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.154+0000 7f10bb19d700 1 --2- 172.21.15.98:0/1248445105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40731e0 0x7f10b4073660 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.154+0000 7f10bb19d700 1 --2- 172.21.15.98:0/1248445105 >> v2:172.21.15.100:3300/0 conn(0x7f10b4146360 0x7f10b4160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.154+0000 7f10bb19d700 1 --2- 172.21.15.98:0/1248445105 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10b40b4ab0 0x7f10b40b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.154+0000 7f10bb19d700 1 -- 172.21.15.98:0/1248445105 >> 172.21.15.98:0/1248445105 conn(0x7f10b406b8e0 msgr2=0x7f10b40cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.154+0000 7f10bb19d700 1 -- 172.21.15.98:0/1248445105 shutdown_connections 2023-03-03T08:02:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.155+0000 7f10bb19d700 1 -- 172.21.15.98:0/1248445105 wait complete. 2023-03-03T08:02:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.156+0000 7f10bb19d700 1 Processor -- start 2023-03-03T08:02:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.157+0000 7f10bb19d700 1 -- start start 2023-03-03T08:02:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10bb19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10b40731e0 0x7f10b41f6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10bb19d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40b4ab0 0x7f10b415d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10bb19d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10b4146360 0x7f10b40b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10bb19d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b415dbc0 con 0x7f10b40b4ab0 2023-03-03T08:02:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10bb19d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b415d8e0 con 0x7f10b4146360 2023-03-03T08:02:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10bb19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b40ca230 con 0x7f10b40731e0 2023-03-03T08:02:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10b999a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40b4ab0 0x7f10b415d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10b999a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40b4ab0 0x7f10b415d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43610/0 (socket says 172.21.15.98:43610) 2023-03-03T08:02:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10b999a700 1 -- 172.21.15.98:0/1417525983 learned_addr learned my addr 172.21.15.98:0/1417525983 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10ba19b700 1 --2- 172.21.15.98:0/1417525983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10b40731e0 0x7f10b41f6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10ba99c700 1 --2- 172.21.15.98:0/1417525983 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10b4146360 0x7f10b40b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10b999a700 1 -- 172.21.15.98:0/1417525983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10b40731e0 msgr2=0x7f10b41f6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10b999a700 1 --2- 172.21.15.98:0/1417525983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10b40731e0 0x7f10b41f6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10b999a700 1 -- 172.21.15.98:0/1417525983 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10b4146360 msgr2=0x7f10b40b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10b999a700 1 --2- 172.21.15.98:0/1417525983 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10b4146360 0x7f10b40b3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.158+0000 7f10b999a700 1 -- 172.21.15.98:0/1417525983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ac009b30 con 0x7f10b40b4ab0 2023-03-03T08:02:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.159+0000 7f10b999a700 1 --2- 172.21.15.98:0/1417525983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40b4ab0 0x7f10b415d020 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f10a400e870 tx=0x7f10a400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.159+0000 7f10aaffd700 1 -- 172.21.15.98:0/1417525983 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4009590 con 0x7f10b40b4ab0 2023-03-03T08:02:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.159+0000 7f10aaffd700 1 -- 172.21.15.98:0/1417525983 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a4013680 con 0x7f10b40b4ab0 2023-03-03T08:02:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.159+0000 7f10bb19d700 1 -- 172.21.15.98:0/1417525983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b40cd960 con 0x7f10b40b4ab0 2023-03-03T08:02:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.159+0000 7f10aaffd700 1 -- 172.21.15.98:0/1417525983 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a40139f0 con 0x7f10b40b4ab0 2023-03-03T08:02:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.159+0000 7f10bb19d700 1 -- 172.21.15.98:0/1417525983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b40cd680 con 0x7f10b40b4ab0 2023-03-03T08:02:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.160+0000 7f10bb19d700 1 -- 172.21.15.98:0/1417525983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b40cdc40 con 0x7f10b40b4ab0 2023-03-03T08:02:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.162+0000 7f10aaffd700 1 -- 172.21.15.98:0/1417525983 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f10a4019030 con 0x7f10b40b4ab0 2023-03-03T08:02:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.163+0000 7f10aaffd700 1 --2- 172.21.15.98:0/1417525983 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1098068380 0x7f109806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.163+0000 7f10aaffd700 1 -- 172.21.15.98:0/1417525983 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f10a4085ab0 con 0x7f10b40b4ab0 2023-03-03T08:02:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.164+0000 7f10ba19b700 1 --2- 172.21.15.98:0/1417525983 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1098068380 0x7f109806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.164+0000 7f10ba19b700 1 --2- 172.21.15.98:0/1417525983 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1098068380 0x7f109806a830 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f10b0009ac0 tx=0x7f10b0009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.167+0000 7f10aaffd700 1 -- 172.21.15.98:0/1417525983 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f10a40521d0 con 0x7f10b40b4ab0 2023-03-03T08:02:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.404+0000 7f10bb19d700 1 -- 172.21.15.98:0/1417525983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10b4068300 con 0x7f10b40b4ab0 2023-03-03T08:02:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.404+0000 7f10aaffd700 1 -- 172.21.15.98:0/1417525983 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4653 (secure 0 0 0) 0x7f10a40557f0 con 0x7f10b40b4ab0 2023-03-03T08:02:26.406 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:26.406 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2561725713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2561725713},{"type":"v1","addr":"172.21.15.98:6834","nonce":2561725713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1511797280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1511797280},{"type":"v1","addr":"172.21.15.100:6834","nonce":1511797280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/1843752855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":1843752855},{"type":"v1","addr":"172.21.15.100:6835","nonce":1843752855}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:02:22.404483+0000","modified":"2023-03-03T08:02:25.872537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7524":{"gid":7524,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6835/341265634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":341265634},{"type":"v1","addr":"172.21.15.98:6835","nonce":341265634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-03-03T08:02:26.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.407+0000 7f10bb19d700 1 -- 172.21.15.98:0/1417525983 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1098068380 msgr2=0x7f109806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:26.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.407+0000 7f10bb19d700 1 --2- 172.21.15.98:0/1417525983 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1098068380 0x7f109806a830 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f10b0009ac0 tx=0x7f10b0009210 comp rx=0 tx=0).stop 2023-03-03T08:02:26.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.407+0000 7f10bb19d700 1 -- 172.21.15.98:0/1417525983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40b4ab0 msgr2=0x7f10b415d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:26.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.407+0000 7f10bb19d700 1 --2- 172.21.15.98:0/1417525983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40b4ab0 0x7f10b415d020 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f10a400e870 tx=0x7f10a400eb80 comp rx=0 tx=0).stop 2023-03-03T08:02:26.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.407+0000 7f10bb19d700 1 -- 172.21.15.98:0/1417525983 shutdown_connections 2023-03-03T08:02:26.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.407+0000 7f10bb19d700 1 --2- 172.21.15.98:0/1417525983 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10b4146360 0x7f10b40b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.407+0000 7f10bb19d700 1 --2- 172.21.15.98:0/1417525983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10b40731e0 0x7f10b41f6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.407+0000 7f10bb19d700 1 --2- 172.21.15.98:0/1417525983 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1098068380 0x7f109806a830 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.407+0000 7f10bb19d700 1 --2- 172.21.15.98:0/1417525983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40b4ab0 0x7f10b415d020 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.407+0000 7f10bb19d700 1 -- 172.21.15.98:0/1417525983 >> 172.21.15.98:0/1417525983 conn(0x7f10b406b8e0 msgr2=0x7f10b40b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:26.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.407+0000 7f10bb19d700 1 -- 172.21.15.98:0/1417525983 shutdown_connections 2023-03-03T08:02:26.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.407+0000 7f10bb19d700 1 -- 172.21.15.98:0/1417525983 wait complete. 2023-03-03T08:02:26.411 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 48 2023-03-03T08:02:26.436 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:02:26.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.680+0000 7f26e5f9d700 1 Processor -- start 2023-03-03T08:02:26.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.680+0000 7f26e5f9d700 1 -- start start 2023-03-03T08:02:26.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.681+0000 7f26e5f9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f26e01595d0 0x7f26e00c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:26.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.681+0000 7f26e5f9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e006c7c0 0x7f26e01599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:26.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.682+0000 7f26e5f9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f26e0159ee0 0x7f26e015d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:26.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.682+0000 7f26e5f9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e00cd000 con 0x7f26e0159ee0 2023-03-03T08:02:26.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.682+0000 7f26e5f9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e006cd80 con 0x7f26e01595d0 2023-03-03T08:02:26.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.682+0000 7f26e5f9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e006d060 con 0x7f26e006c7c0 2023-03-03T08:02:26.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.682+0000 7f26e579c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f26e0159ee0 0x7f26e015d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:26.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.682+0000 7f26e579c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f26e0159ee0 0x7f26e015d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43624/0 (socket says 172.21.15.98:43624) 2023-03-03T08:02:26.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.682+0000 7f26e579c700 1 -- 172.21.15.98:0/2645596308 learned_addr learned my addr 172.21.15.98:0/2645596308 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:26.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.682+0000 7f26dffff700 1 --2- 172.21.15.98:0/2645596308 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e006c7c0 0x7f26e01599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:26.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.682+0000 7f26e4f9b700 1 --2- 172.21.15.98:0/2645596308 >> v2:172.21.15.100:3300/0 conn(0x7f26e01595d0 0x7f26e00c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:26.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.682+0000 7f26e579c700 1 -- 172.21.15.98:0/2645596308 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e006c7c0 msgr2=0x7f26e01599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:26.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.682+0000 7f26e579c700 1 --2- 172.21.15.98:0/2645596308 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e006c7c0 0x7f26e01599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.682+0000 7f26e579c700 1 -- 172.21.15.98:0/2645596308 >> v2:172.21.15.100:3300/0 conn(0x7f26e01595d0 msgr2=0x7f26e00c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:26.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.682+0000 7f26e579c700 1 --2- 172.21.15.98:0/2645596308 >> v2:172.21.15.100:3300/0 conn(0x7f26e01595d0 0x7f26e00c9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.683+0000 7f26e579c700 1 -- 172.21.15.98:0/2645596308 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e006d340 con 0x7f26e0159ee0 2023-03-03T08:02:26.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.683+0000 7f26e579c700 1 --2- 172.21.15.98:0/2645596308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e0159ee0 0x7f26e015d1f0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f26e006b3f0 tx=0x7f26d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8fad229323908 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:26.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.683+0000 7f26df7fe700 1 -- 172.21.15.98:0/2645596308 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d800e070 con 0x7f26e0159ee0 2023-03-03T08:02:26.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.683+0000 7f26df7fe700 1 -- 172.21.15.98:0/2645596308 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d800b910 con 0x7f26e0159ee0 2023-03-03T08:02:26.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.683+0000 7f26df7fe700 1 -- 172.21.15.98:0/2645596308 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d800bcc0 con 0x7f26e0159ee0 2023-03-03T08:02:26.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.684+0000 7f26e5f9d700 1 -- 172.21.15.98:0/2645596308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e0159ee0 msgr2=0x7f26e015d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:26.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.684+0000 7f26e5f9d700 1 --2- 172.21.15.98:0/2645596308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e0159ee0 0x7f26e015d1f0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f26e006b3f0 tx=0x7f26d800b330 comp rx=0 tx=0).stop 2023-03-03T08:02:26.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.684+0000 7f26e5f9d700 1 -- 172.21.15.98:0/2645596308 shutdown_connections 2023-03-03T08:02:26.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.684+0000 7f26e5f9d700 1 --2- 172.21.15.98:0/2645596308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e0159ee0 0x7f26e015d1f0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.684+0000 7f26e5f9d700 1 --2- 172.21.15.98:0/2645596308 >> v2:172.21.15.100:3300/0 conn(0x7f26e01595d0 0x7f26e00c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.684+0000 7f26e5f9d700 1 --2- 172.21.15.98:0/2645596308 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e006c7c0 0x7f26e01599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.684+0000 7f26e5f9d700 1 -- 172.21.15.98:0/2645596308 >> 172.21.15.98:0/2645596308 conn(0x7f26e00cf950 msgr2=0x7f26e00cb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:26.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.684+0000 7f26e5f9d700 1 -- 172.21.15.98:0/2645596308 shutdown_connections 2023-03-03T08:02:26.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.684+0000 7f26e5f9d700 1 -- 172.21.15.98:0/2645596308 wait complete. 2023-03-03T08:02:26.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.686+0000 7f26e5f9d700 1 Processor -- start 2023-03-03T08:02:26.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.686+0000 7f26e5f9d700 1 -- start start 2023-03-03T08:02:26.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.687+0000 7f26e5f9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e006c7c0 0x7f26e0154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:26.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.687+0000 7f26e5f9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e01595d0 0x7f26e00c7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:26.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.687+0000 7f26e5f9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26e0159ee0 0x7f26e01f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:26.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.687+0000 7f26e5f9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e01548c0 con 0x7f26e01595d0 2023-03-03T08:02:26.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.687+0000 7f26e5f9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00c7760 con 0x7f26e0159ee0 2023-03-03T08:02:26.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.687+0000 7f26e5f9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e015bda0 con 0x7f26e006c7c0 2023-03-03T08:02:26.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.687+0000 7f26dffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e01595d0 0x7f26e00c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:26.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.687+0000 7f26dffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e01595d0 0x7f26e00c7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43640/0 (socket says 172.21.15.98:43640) 2023-03-03T08:02:26.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.687+0000 7f26dffff700 1 -- 172.21.15.98:0/2855049032 learned_addr learned my addr 172.21.15.98:0/2855049032 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:26.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.688+0000 7f26e4f9b700 1 --2- 172.21.15.98:0/2855049032 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e006c7c0 0x7f26e0154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:26.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.688+0000 7f26e579c700 1 --2- 172.21.15.98:0/2855049032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26e0159ee0 0x7f26e01f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.688+0000 7f26dffff700 1 -- 172.21.15.98:0/2855049032 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e006c7c0 msgr2=0x7f26e0154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.688+0000 7f26dffff700 1 --2- 172.21.15.98:0/2855049032 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e006c7c0 0x7f26e0154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.688+0000 7f26dffff700 1 -- 172.21.15.98:0/2855049032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26e0159ee0 msgr2=0x7f26e01f5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.688+0000 7f26dffff700 1 --2- 172.21.15.98:0/2855049032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26e0159ee0 0x7f26e01f5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.688+0000 7f26dffff700 1 -- 172.21.15.98:0/2855049032 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d8009b30 con 0x7f26e01595d0 2023-03-03T08:02:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.688+0000 7f26dffff700 1 --2- 172.21.15.98:0/2855049032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e01595d0 0x7f26e00c7220 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f26cc00e930 tx=0x7f26cc00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.688+0000 7f26dd7fa700 1 -- 172.21.15.98:0/2855049032 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26cc009690 con 0x7f26e01595d0 2023-03-03T08:02:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.688+0000 7f26dd7fa700 1 -- 172.21.15.98:0/2855049032 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26cc00c850 con 0x7f26e01595d0 2023-03-03T08:02:26.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.688+0000 7f26e5f9d700 1 -- 172.21.15.98:0/2855049032 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e00ca260 con 0x7f26e01595d0 2023-03-03T08:02:26.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.689+0000 7f26dd7fa700 1 -- 172.21.15.98:0/2855049032 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26cc00cbe0 con 0x7f26e01595d0 2023-03-03T08:02:26.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.689+0000 7f26e5f9d700 1 -- 172.21.15.98:0/2855049032 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e006d620 con 0x7f26e01595d0 2023-03-03T08:02:26.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.690+0000 7f26e5f9d700 1 -- 172.21.15.98:0/2855049032 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e015a360 con 0x7f26e01595d0 2023-03-03T08:02:26.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.691+0000 7f26dd7fa700 1 -- 172.21.15.98:0/2855049032 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f26cc019030 con 0x7f26e01595d0 2023-03-03T08:02:26.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.692+0000 7f26dd7fa700 1 --2- 172.21.15.98:0/2855049032 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f26c4068380 0x7f26c406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:26.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.692+0000 7f26dd7fa700 1 -- 172.21.15.98:0/2855049032 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f26cc085cf0 con 0x7f26e01595d0 2023-03-03T08:02:26.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.693+0000 7f26e4f9b700 1 --2- 172.21.15.98:0/2855049032 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f26c4068380 0x7f26c406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:26.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.693+0000 7f26e4f9b700 1 --2- 172.21.15.98:0/2855049032 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f26c4068380 0x7f26c406a830 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009b70 tx=0x7f26d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:26.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.696+0000 7f26dd7fa700 1 -- 172.21.15.98:0/2855049032 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f26cc052470 con 0x7f26e01595d0 2023-03-03T08:02:26.716 INFO:tasks.ceph.mds.c.smithi098.stderr:2023-03-03T08:02:26.715+0000 7fe26a3f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T08:02:26.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.892+0000 7f26e5f9d700 1 -- 172.21.15.98:0/2855049032 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26e0068300 con 0x7f26e01595d0 2023-03-03T08:02:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.893+0000 7f26dd7fa700 1 -- 172.21.15.98:0/2855049032 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+17922 (secure 0 0 0) 0x7f26cc017090 con 0x7f26e01595d0 2023-03-03T08:02:26.895 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:26.895 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:02:23.406996+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:02:20.797579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-03-03T08:02:21.411899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:02:26.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.896+0000 7f26e5f9d700 1 -- 172.21.15.98:0/2855049032 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f26c4068380 msgr2=0x7f26c406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.896+0000 7f26e5f9d700 1 --2- 172.21.15.98:0/2855049032 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f26c4068380 0x7f26c406a830 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009b70 tx=0x7f26d4008040 comp rx=0 tx=0).stop 2023-03-03T08:02:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.896+0000 7f26e5f9d700 1 -- 172.21.15.98:0/2855049032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e01595d0 msgr2=0x7f26e00c7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.896+0000 7f26e5f9d700 1 --2- 172.21.15.98:0/2855049032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e01595d0 0x7f26e00c7220 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f26cc00e930 tx=0x7f26cc00ecf0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.896+0000 7f26e5f9d700 1 -- 172.21.15.98:0/2855049032 shutdown_connections 2023-03-03T08:02:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.896+0000 7f26e5f9d700 1 --2- 172.21.15.98:0/2855049032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26e0159ee0 0x7f26e01f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.896+0000 7f26e5f9d700 1 --2- 172.21.15.98:0/2855049032 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26e006c7c0 0x7f26e0154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.896+0000 7f26e5f9d700 1 --2- 172.21.15.98:0/2855049032 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f26c4068380 0x7f26c406a830 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.896+0000 7f26e5f9d700 1 --2- 172.21.15.98:0/2855049032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e01595d0 0x7f26e00c7220 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.896+0000 7f26e5f9d700 1 -- 172.21.15.98:0/2855049032 >> 172.21.15.98:0/2855049032 conn(0x7f26e00cf950 msgr2=0x7f26e00b0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.896+0000 7f26e5f9d700 1 -- 172.21.15.98:0/2855049032 shutdown_connections 2023-03-03T08:02:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:26.896+0000 7f26e5f9d700 1 -- 172.21.15.98:0/2855049032 wait complete. 2023-03-03T08:02:26.924 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-03-03T08:02:26.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:02:27.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.168+0000 7f3036dff700 1 Processor -- start 2023-03-03T08:02:27.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.168+0000 7f3036dff700 1 -- start start 2023-03-03T08:02:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.169+0000 7f3036dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f303015c8d0 0x7f30300b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:27.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.169+0000 7f3036dff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3030160ea0 0x7f30300b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:27.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.169+0000 7f3036dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30301613a0 con 0x7f30300b5710 2023-03-03T08:02:27.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.169+0000 7f3036dff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30300d0e50 con 0x7f3030160ea0 2023-03-03T08:02:27.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.169+0000 7f3036dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30300cd900 con 0x7f303015c8d0 2023-03-03T08:02:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.170+0000 7f302ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3030160ea0 0x7f30300b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.170+0000 7f302ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3030160ea0 0x7f30300b51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43648/0 (socket says 172.21.15.98:43648) 2023-03-03T08:02:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.170+0000 7f302ffff700 1 -- 172.21.15.98:0/2968318539 learned_addr learned my addr 172.21.15.98:0/2968318539 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.170+0000 7f3034b9b700 1 --2- 172.21.15.98:0/2968318539 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f303015c8d0 0x7f30300b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.170+0000 7f302ffff700 1 -- 172.21.15.98:0/2968318539 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f303015c8d0 msgr2=0x7f30300b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.170+0000 7f302ffff700 1 --2- 172.21.15.98:0/2968318539 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f303015c8d0 0x7f30300b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.170+0000 7f302ffff700 1 -- 172.21.15.98:0/2968318539 >> v1:172.21.15.100:6789/0 conn(0x7f30300b5710 legacy=0x7f30300cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.170+0000 7f302ffff700 1 -- 172.21.15.98:0/2968318539 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30300cdbe0 con 0x7f3030160ea0 2023-03-03T08:02:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.171+0000 7f302ffff700 1 --2- 172.21.15.98:0/2968318539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3030160ea0 0x7f30300b51d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f301c0098f0 tx=0x7f301c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d53a83fc949cf6c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.171+0000 7f302f7fe700 1 -- 172.21.15.98:0/2968318539 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c00b8a0 con 0x7f3030160ea0 2023-03-03T08:02:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.171+0000 7f302f7fe700 1 -- 172.21.15.98:0/2968318539 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c00ba00 con 0x7f3030160ea0 2023-03-03T08:02:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.171+0000 7f302f7fe700 1 -- 172.21.15.98:0/2968318539 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c00bd90 con 0x7f3030160ea0 2023-03-03T08:02:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.171+0000 7f3036dff700 1 -- 172.21.15.98:0/2968318539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3030160ea0 msgr2=0x7f30300b51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.171+0000 7f3036dff700 1 --2- 172.21.15.98:0/2968318539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3030160ea0 0x7f30300b51d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f301c0098f0 tx=0x7f301c009c00 comp rx=0 tx=0).stop 2023-03-03T08:02:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.171+0000 7f3036dff700 1 -- 172.21.15.98:0/2968318539 shutdown_connections 2023-03-03T08:02:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.171+0000 7f3036dff700 1 --2- 172.21.15.98:0/2968318539 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f303015c8d0 0x7f30300b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.171+0000 7f3036dff700 1 --2- 172.21.15.98:0/2968318539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3030160ea0 0x7f30300b51d0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.172+0000 7f3036dff700 1 -- 172.21.15.98:0/2968318539 >> 172.21.15.98:0/2968318539 conn(0x7f3030073e70 msgr2=0x7f303006ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.172+0000 7f3036dff700 1 -- 172.21.15.98:0/2968318539 shutdown_connections 2023-03-03T08:02:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.172+0000 7f3036dff700 1 -- 172.21.15.98:0/2968318539 wait complete. 2023-03-03T08:02:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.174+0000 7f3036dff700 1 Processor -- start 2023-03-03T08:02:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.174+0000 7f3036dff700 1 -- start start 2023-03-03T08:02:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.175+0000 7f3036dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30300b5710 0x7f303015ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.175+0000 7f3036dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f303015c8d0 0x7f3030155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.175+0000 7f3036dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3030160ea0 0x7f3030157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.175+0000 7f3036dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3030159580 con 0x7f3030160ea0 2023-03-03T08:02:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.175+0000 7f3036dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3030157790 con 0x7f30300b5710 2023-03-03T08:02:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.175+0000 7f3036dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303015d480 con 0x7f303015c8d0 2023-03-03T08:02:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.175+0000 7f303539c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3030160ea0 0x7f3030157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.175+0000 7f303539c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3030160ea0 0x7f3030157250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43658/0 (socket says 172.21.15.98:43658) 2023-03-03T08:02:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.175+0000 7f303539c700 1 -- 172.21.15.98:0/2134046260 learned_addr learned my addr 172.21.15.98:0/2134046260 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.175+0000 7f3034b9b700 1 --2- 172.21.15.98:0/2134046260 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30300b5710 0x7f303015ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.175+0000 7f302ffff700 1 --2- 172.21.15.98:0/2134046260 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f303015c8d0 0x7f3030155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.176+0000 7f303539c700 1 -- 172.21.15.98:0/2134046260 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f303015c8d0 msgr2=0x7f3030155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.176+0000 7f303539c700 1 --2- 172.21.15.98:0/2134046260 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f303015c8d0 0x7f3030155510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.176+0000 7f303539c700 1 -- 172.21.15.98:0/2134046260 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30300b5710 msgr2=0x7f303015ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.176+0000 7f303539c700 1 --2- 172.21.15.98:0/2134046260 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30300b5710 0x7f303015ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.176+0000 7f303539c700 1 -- 172.21.15.98:0/2134046260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c009550 con 0x7f3030160ea0 2023-03-03T08:02:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.176+0000 7f303539c700 1 --2- 172.21.15.98:0/2134046260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3030160ea0 0x7f3030157250 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f302800cb50 tx=0x7f302800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.176+0000 7f302d7fa700 1 -- 172.21.15.98:0/2134046260 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3028003c00 con 0x7f3030160ea0 2023-03-03T08:02:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.176+0000 7f302d7fa700 1 -- 172.21.15.98:0/2134046260 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3028003d60 con 0x7f3030160ea0 2023-03-03T08:02:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.176+0000 7f302d7fa700 1 -- 172.21.15.98:0/2134046260 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302800ab70 con 0x7f3030160ea0 2023-03-03T08:02:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.176+0000 7f3036dff700 1 -- 172.21.15.98:0/2134046260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303015cca0 con 0x7f3030160ea0 2023-03-03T08:02:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.176+0000 7f3036dff700 1 -- 172.21.15.98:0/2134046260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30300cdec0 con 0x7f3030160ea0 2023-03-03T08:02:27.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.177+0000 7f3036dff700 1 -- 172.21.15.98:0/2134046260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303006bb70 con 0x7f3030160ea0 2023-03-03T08:02:27.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.179+0000 7f302d7fa700 1 -- 172.21.15.98:0/2134046260 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f3028010050 con 0x7f3030160ea0 2023-03-03T08:02:27.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.180+0000 7f302d7fa700 1 --2- 172.21.15.98:0/2134046260 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3014068380 0x7f301406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:27.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.180+0000 7f302d7fa700 1 -- 172.21.15.98:0/2134046260 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3028085ca0 con 0x7f3030160ea0 2023-03-03T08:02:27.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.181+0000 7f3034b9b700 1 --2- 172.21.15.98:0/2134046260 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3014068380 0x7f301406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:27.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.182+0000 7f3034b9b700 1 --2- 172.21.15.98:0/2134046260 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3014068380 0x7f301406a830 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f30300cd8d0 tx=0x7f3024008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:27.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.184+0000 7f302d7fa700 1 -- 172.21.15.98:0/2134046260 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3028052420 con 0x7f3030160ea0 2023-03-03T08:02:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.370+0000 7f3036dff700 1 -- 172.21.15.98:0/2134046260 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f303004e6e0 con 0x7f3014068380 2023-03-03T08:02:27.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.382+0000 7f302d7fa700 1 -- 172.21.15.98:0/2134046260 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311138 (secure 0 0 0) 0x7f303004e6e0 con 0x7f3014068380 2023-03-03T08:02:27.387 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:27.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.387+0000 7f3036dff700 1 -- 172.21.15.98:0/2134046260 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3014068380 msgr2=0x7f301406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:27.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.387+0000 7f3036dff700 1 --2- 172.21.15.98:0/2134046260 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3014068380 0x7f301406a830 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f30300cd8d0 tx=0x7f3024008040 comp rx=0 tx=0).stop 2023-03-03T08:02:27.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.387+0000 7f3036dff700 1 -- 172.21.15.98:0/2134046260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3030160ea0 msgr2=0x7f3030157250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:27.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.387+0000 7f3036dff700 1 --2- 172.21.15.98:0/2134046260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3030160ea0 0x7f3030157250 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f302800cb50 tx=0x7f302800cf10 comp rx=0 tx=0).stop 2023-03-03T08:02:27.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.387+0000 7f3036dff700 1 -- 172.21.15.98:0/2134046260 shutdown_connections 2023-03-03T08:02:27.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.387+0000 7f3036dff700 1 --2- 172.21.15.98:0/2134046260 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3014068380 0x7f301406a830 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.387+0000 7f3036dff700 1 --2- 172.21.15.98:0/2134046260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3030160ea0 0x7f3030157250 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.387+0000 7f3036dff700 1 --2- 172.21.15.98:0/2134046260 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30300b5710 0x7f303015ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.387+0000 7f3036dff700 1 --2- 172.21.15.98:0/2134046260 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f303015c8d0 0x7f3030155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.387+0000 7f3036dff700 1 -- 172.21.15.98:0/2134046260 >> 172.21.15.98:0/2134046260 conn(0x7f3030073e70 msgr2=0x7f30300b6700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:27.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.387+0000 7f3036dff700 1 -- 172.21.15.98:0/2134046260 shutdown_connections 2023-03-03T08:02:27.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.387+0000 7f3036dff700 1 -- 172.21.15.98:0/2134046260 wait complete. 2023-03-03T08:02:27.392 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T08:02:27.415 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2023-03-03T08:02:26.164808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":223,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166,"ondisk_log_size":166,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":200,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":92248,"kb_used_data":9944,"kb_used_omap":0,"kb_used_meta":82240,"kb_avail":754882472,"statfs":{"total":773094113280,"available":772999651328,"internally_reserved":0,"allocated":10182656,"data_stored":5256208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":84213760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.310187"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595186+0000","last_change":"2023-03-03T08:02:22.430702+0000","last_active":"2023-03-03T08:02:23.595186+0000","last_peered":"2023-03-03T08:02:23.595186+0000","last_clean":"2023-03-03T08:02:23.595186+0000","last_became_active":"2023-03-03T08:02:22.430443+0000","last_became_peered":"2023-03-03T08:02:22.430443+0000","last_unstale":"2023-03-03T08:02:23.595186+0000","last_undegraded":"2023-03-03T08:02:23.595186+0000","last_fullsized":"2023-03-03T08:02:23.595186+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:00:48.464577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594708+0000","last_change":"2023-03-03T08:02:22.435774+0000","last_active":"2023-03-03T08:02:23.594708+0000","last_peered":"2023-03-03T08:02:23.594708+0000","last_clean":"2023-03-03T08:02:23.594708+0000","last_became_active":"2023-03-03T08:02:22.435450+0000","last_became_peered":"2023-03-03T08:02:22.435450+0000","last_unstale":"2023-03-03T08:02:23.594708+0000","last_undegraded":"2023-03-03T08:02:23.594708+0000","last_fullsized":"2023-03-03T08:02:23.594708+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:01:28.330059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594597+0000","last_change":"2023-03-03T08:02:22.422601+0000","last_active":"2023-03-03T08:02:23.594597+0000","last_peered":"2023-03-03T08:02:23.594597+0000","last_clean":"2023-03-03T08:02:23.594597+0000","last_became_active":"2023-03-03T08:02:22.422336+0000","last_became_peered":"2023-03-03T08:02:22.422336+0000","last_unstale":"2023-03-03T08:02:23.594597+0000","last_undegraded":"2023-03-03T08:02:23.594597+0000","last_fullsized":"2023-03-03T08:02:23.594597+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:56:41.737023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594520+0000","last_change":"2023-03-03T08:02:22.431276+0000","last_active":"2023-03-03T08:02:23.594520+0000","last_peered":"2023-03-03T08:02:23.594520+0000","last_clean":"2023-03-03T08:02:23.594520+0000","last_became_active":"2023-03-03T08:02:22.431011+0000","last_became_peered":"2023-03-03T08:02:22.431011+0000","last_unstale":"2023-03-03T08:02:23.594520+0000","last_undegraded":"2023-03-03T08:02:23.594520+0000","last_fullsized":"2023-03-03T08:02:23.594520+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:05:38.030389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.412835+0000","last_change":"2023-03-03T08:02:22.430589+0000","last_active":"2023-03-03T08:02:23.412835+0000","last_peered":"2023-03-03T08:02:23.412835+0000","last_clean":"2023-03-03T08:02:23.412835+0000","last_became_active":"2023-03-03T08:02:22.430316+0000","last_became_peered":"2023-03-03T08:02:22.430316+0000","last_unstale":"2023-03-03T08:02:23.412835+0000","last_undegraded":"2023-03-03T08:02:23.412835+0000","last_fullsized":"2023-03-03T08:02:23.412835+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:13:56.368129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594492+0000","last_change":"2023-03-03T08:02:22.433617+0000","last_active":"2023-03-03T08:02:23.594492+0000","last_peered":"2023-03-03T08:02:23.594492+0000","last_clean":"2023-03-03T08:02:23.594492+0000","last_became_active":"2023-03-03T08:02:22.431639+0000","last_became_peered":"2023-03-03T08:02:22.431639+0000","last_unstale":"2023-03-03T08:02:23.594492+0000","last_undegraded":"2023-03-03T08:02:23.594492+0000","last_fullsized":"2023-03-03T08:02:23.594492+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:31:12.347482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.411573+0000","last_change":"2023-03-03T08:02:22.433748+0000","last_active":"2023-03-03T08:02:23.411573+0000","last_peered":"2023-03-03T08:02:23.411573+0000","last_clean":"2023-03-03T08:02:23.411573+0000","last_became_active":"2023-03-03T08:02:22.433402+0000","last_became_peered":"2023-03-03T08:02:22.433402+0000","last_unstale":"2023-03-03T08:02:23.411573+0000","last_undegraded":"2023-03-03T08:02:23.411573+0000","last_fullsized":"2023-03-03T08:02:23.411573+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:00:48.464577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.596025+0000","last_change":"2023-03-03T08:02:22.427347+0000","last_active":"2023-03-03T08:02:23.596025+0000","last_peered":"2023-03-03T08:02:23.596025+0000","last_clean":"2023-03-03T08:02:23.596025+0000","last_became_active":"2023-03-03T08:02:22.426883+0000","last_became_peered":"2023-03-03T08:02:22.426883+0000","last_unstale":"2023-03-03T08:02:23.596025+0000","last_undegraded":"2023-03-03T08:02:23.596025+0000","last_fullsized":"2023-03-03T08:02:23.596025+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:12.558600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.413147+0000","last_change":"2023-03-03T08:02:22.424179+0000","last_active":"2023-03-03T08:02:23.413147+0000","last_peered":"2023-03-03T08:02:23.413147+0000","last_clean":"2023-03-03T08:02:23.413147+0000","last_became_active":"2023-03-03T08:02:22.423895+0000","last_became_peered":"2023-03-03T08:02:22.423895+0000","last_unstale":"2023-03-03T08:02:23.413147+0000","last_undegraded":"2023-03-03T08:02:23.413147+0000","last_fullsized":"2023-03-03T08:02:23.413147+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:20:04.574289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595081+0000","last_change":"2023-03-03T08:02:22.441027+0000","last_active":"2023-03-03T08:02:23.595081+0000","last_peered":"2023-03-03T08:02:23.595081+0000","last_clean":"2023-03-03T08:02:23.595081+0000","last_became_active":"2023-03-03T08:02:22.440624+0000","last_became_peered":"2023-03-03T08:02:22.440624+0000","last_unstale":"2023-03-03T08:02:23.595081+0000","last_undegraded":"2023-03-03T08:02:23.595081+0000","last_fullsized":"2023-03-03T08:02:23.595081+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:28:25.191765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.413287+0000","last_change":"2023-03-03T08:02:22.437088+0000","last_active":"2023-03-03T08:02:23.413287+0000","last_peered":"2023-03-03T08:02:23.413287+0000","last_clean":"2023-03-03T08:02:23.413287+0000","last_became_active":"2023-03-03T08:02:22.436567+0000","last_became_peered":"2023-03-03T08:02:22.436567+0000","last_unstale":"2023-03-03T08:02:23.413287+0000","last_undegraded":"2023-03-03T08:02:23.413287+0000","last_fullsized":"2023-03-03T08:02:23.413287+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:46:16.129844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594856+0000","last_change":"2023-03-03T08:02:22.431484+0000","last_active":"2023-03-03T08:02:23.594856+0000","last_peered":"2023-03-03T08:02:23.594856+0000","last_clean":"2023-03-03T08:02:23.594856+0000","last_became_active":"2023-03-03T08:02:22.431239+0000","last_became_peered":"2023-03-03T08:02:22.431239+0000","last_unstale":"2023-03-03T08:02:23.594856+0000","last_undegraded":"2023-03-03T08:02:23.594856+0000","last_fullsized":"2023-03-03T08:02:23.594856+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:21.374521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594202+0000","last_change":"2023-03-03T08:02:22.426282+0000","last_active":"2023-03-03T08:02:23.594202+0000","last_peered":"2023-03-03T08:02:23.594202+0000","last_clean":"2023-03-03T08:02:23.594202+0000","last_became_active":"2023-03-03T08:02:22.425689+0000","last_became_peered":"2023-03-03T08:02:22.425689+0000","last_unstale":"2023-03-03T08:02:23.594202+0000","last_undegraded":"2023-03-03T08:02:23.594202+0000","last_fullsized":"2023-03-03T08:02:23.594202+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:07.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.412890+0000","last_change":"2023-03-03T08:02:22.434995+0000","last_active":"2023-03-03T08:02:23.412890+0000","last_peered":"2023-03-03T08:02:23.412890+0000","last_clean":"2023-03-03T08:02:23.412890+0000","last_became_active":"2023-03-03T08:02:22.434627+0000","last_became_peered":"2023-03-03T08:02:22.434627+0000","last_unstale":"2023-03-03T08:02:23.412890+0000","last_undegraded":"2023-03-03T08:02:23.412890+0000","last_fullsized":"2023-03-03T08:02:23.412890+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:12:17.218987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.596250+0000","last_change":"2023-03-03T08:02:22.437244+0000","last_active":"2023-03-03T08:02:23.596250+0000","last_peered":"2023-03-03T08:02:23.596250+0000","last_clean":"2023-03-03T08:02:23.596250+0000","last_became_active":"2023-03-03T08:02:22.433987+0000","last_became_peered":"2023-03-03T08:02:22.433987+0000","last_unstale":"2023-03-03T08:02:23.596250+0000","last_undegraded":"2023-03-03T08:02:23.596250+0000","last_fullsized":"2023-03-03T08:02:23.596250+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:01:28.330059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595804+0000","last_change":"2023-03-03T08:02:22.426791+0000","last_active":"2023-03-03T08:02:23.595804+0000","last_peered":"2023-03-03T08:02:23.595804+0000","last_clean":"2023-03-03T08:02:23.595804+0000","last_became_active":"2023-03-03T08:02:22.426461+0000","last_became_peered":"2023-03-03T08:02:22.426461+0000","last_unstale":"2023-03-03T08:02:23.595804+0000","last_undegraded":"2023-03-03T08:02:23.595804+0000","last_fullsized":"2023-03-03T08:02:23.595804+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:54:03.035322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.596298+0000","last_change":"2023-03-03T08:02:22.435907+0000","last_active":"2023-03-03T08:02:23.596298+0000","last_peered":"2023-03-03T08:02:23.596298+0000","last_clean":"2023-03-03T08:02:23.596298+0000","last_became_active":"2023-03-03T08:02:22.432401+0000","last_became_peered":"2023-03-03T08:02:22.432401+0000","last_unstale":"2023-03-03T08:02:23.596298+0000","last_undegraded":"2023-03-03T08:02:23.596298+0000","last_fullsized":"2023-03-03T08:02:23.596298+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:04:41.141311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594707+0000","last_change":"2023-03-03T08:02:22.420788+0000","last_active":"2023-03-03T08:02:23.594707+0000","last_peered":"2023-03-03T08:02:23.594707+0000","last_clean":"2023-03-03T08:02:23.594707+0000","last_became_active":"2023-03-03T08:02:22.420496+0000","last_became_peered":"2023-03-03T08:02:22.420496+0000","last_unstale":"2023-03-03T08:02:23.594707+0000","last_undegraded":"2023-03-03T08:02:23.594707+0000","last_fullsized":"2023-03-03T08:02:23.594707+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:45.588606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.413111+0000","last_change":"2023-03-03T08:02:22.429036+0000","last_active":"2023-03-03T08:02:23.413111+0000","last_peered":"2023-03-03T08:02:23.413111+0000","last_clean":"2023-03-03T08:02:23.413111+0000","last_became_active":"2023-03-03T08:02:22.426987+0000","last_became_peered":"2023-03-03T08:02:22.426987+0000","last_unstale":"2023-03-03T08:02:23.413111+0000","last_undegraded":"2023-03-03T08:02:23.413111+0000","last_fullsized":"2023-03-03T08:02:23.413111+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:26:55.840281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594788+0000","last_change":"2023-03-03T08:02:22.433917+0000","last_active":"2023-03-03T08:02:23.594788+0000","last_peered":"2023-03-03T08:02:23.594788+0000","last_clean":"2023-03-03T08:02:23.594788+0000","last_became_active":"2023-03-03T08:02:22.433570+0000","last_became_peered":"2023-03-03T08:02:22.433570+0000","last_unstale":"2023-03-03T08:02:23.594788+0000","last_undegraded":"2023-03-03T08:02:23.594788+0000","last_fullsized":"2023-03-03T08:02:23.594788+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:05:19.334210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.412953+0000","last_change":"2023-03-03T08:02:22.425945+0000","last_active":"2023-03-03T08:02:23.412953+0000","last_peered":"2023-03-03T08:02:23.412953+0000","last_clean":"2023-03-03T08:02:23.412953+0000","last_became_active":"2023-03-03T08:02:22.421663+0000","last_became_peered":"2023-03-03T08:02:22.421663+0000","last_unstale":"2023-03-03T08:02:23.412953+0000","last_undegraded":"2023-03-03T08:02:23.412953+0000","last_fullsized":"2023-03-03T08:02:23.412953+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:07:17.564359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.412847+0000","last_change":"2023-03-03T08:02:22.434911+0000","last_active":"2023-03-03T08:02:23.412847+0000","last_peered":"2023-03-03T08:02:23.412847+0000","last_clean":"2023-03-03T08:02:23.412847+0000","last_became_active":"2023-03-03T08:02:22.434589+0000","last_became_peered":"2023-03-03T08:02:22.434589+0000","last_unstale":"2023-03-03T08:02:23.412847+0000","last_undegraded":"2023-03-03T08:02:23.412847+0000","last_fullsized":"2023-03-03T08:02:23.412847+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:34.930052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594007+0000","last_change":"2023-03-03T08:02:22.437187+0000","last_active":"2023-03-03T08:02:23.594007+0000","last_peered":"2023-03-03T08:02:23.594007+0000","last_clean":"2023-03-03T08:02:23.594007+0000","last_became_active":"2023-03-03T08:02:22.436843+0000","last_became_peered":"2023-03-03T08:02:22.436843+0000","last_unstale":"2023-03-03T08:02:23.594007+0000","last_undegraded":"2023-03-03T08:02:23.594007+0000","last_fullsized":"2023-03-03T08:02:23.594007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:00.941358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594266+0000","last_change":"2023-03-03T08:02:22.422598+0000","last_active":"2023-03-03T08:02:23.594266+0000","last_peered":"2023-03-03T08:02:23.594266+0000","last_clean":"2023-03-03T08:02:23.594266+0000","last_became_active":"2023-03-03T08:02:22.422326+0000","last_became_peered":"2023-03-03T08:02:22.422326+0000","last_unstale":"2023-03-03T08:02:23.594266+0000","last_undegraded":"2023-03-03T08:02:23.594266+0000","last_fullsized":"2023-03-03T08:02:23.594266+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:45:54.383085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594500+0000","last_change":"2023-03-03T08:02:22.435279+0000","last_active":"2023-03-03T08:02:23.594500+0000","last_peered":"2023-03-03T08:02:23.594500+0000","last_clean":"2023-03-03T08:02:23.594500+0000","last_became_active":"2023-03-03T08:02:22.434992+0000","last_became_peered":"2023-03-03T08:02:22.434992+0000","last_unstale":"2023-03-03T08:02:23.594500+0000","last_undegraded":"2023-03-03T08:02:23.594500+0000","last_fullsized":"2023-03-03T08:02:23.594500+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:52:46.445281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.413120+0000","last_change":"2023-03-03T08:02:22.431162+0000","last_active":"2023-03-03T08:02:23.413120+0000","last_peered":"2023-03-03T08:02:23.413120+0000","last_clean":"2023-03-03T08:02:23.413120+0000","last_became_active":"2023-03-03T08:02:22.430856+0000","last_became_peered":"2023-03-03T08:02:22.430856+0000","last_unstale":"2023-03-03T08:02:23.413120+0000","last_undegraded":"2023-03-03T08:02:23.413120+0000","last_fullsized":"2023-03-03T08:02:23.413120+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:47:11.815286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595766+0000","last_change":"2023-03-03T08:02:22.439980+0000","last_active":"2023-03-03T08:02:23.595766+0000","last_peered":"2023-03-03T08:02:23.595766+0000","last_clean":"2023-03-03T08:02:23.595766+0000","last_became_active":"2023-03-03T08:02:22.439690+0000","last_became_peered":"2023-03-03T08:02:22.439690+0000","last_unstale":"2023-03-03T08:02:23.595766+0000","last_undegraded":"2023-03-03T08:02:23.595766+0000","last_fullsized":"2023-03-03T08:02:23.595766+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:21:01.687797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594608+0000","last_change":"2023-03-03T08:02:22.427427+0000","last_active":"2023-03-03T08:02:23.594608+0000","last_peered":"2023-03-03T08:02:23.594608+0000","last_clean":"2023-03-03T08:02:23.594608+0000","last_became_active":"2023-03-03T08:02:22.427093+0000","last_became_peered":"2023-03-03T08:02:22.427093+0000","last_unstale":"2023-03-03T08:02:23.594608+0000","last_undegraded":"2023-03-03T08:02:23.594608+0000","last_fullsized":"2023-03-03T08:02:23.594608+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:39.345227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.411686+0000","last_change":"2023-03-03T08:02:22.431307+0000","last_active":"2023-03-03T08:02:23.411686+0000","last_peered":"2023-03-03T08:02:23.411686+0000","last_clean":"2023-03-03T08:02:23.411686+0000","last_became_active":"2023-03-03T08:02:22.430953+0000","last_became_peered":"2023-03-03T08:02:22.430953+0000","last_unstale":"2023-03-03T08:02:23.411686+0000","last_undegraded":"2023-03-03T08:02:23.411686+0000","last_fullsized":"2023-03-03T08:02:23.411686+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:23:35.028360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.413007+0000","last_change":"2023-03-03T08:02:22.422990+0000","last_active":"2023-03-03T08:02:23.413007+0000","last_peered":"2023-03-03T08:02:23.413007+0000","last_clean":"2023-03-03T08:02:23.413007+0000","last_became_active":"2023-03-03T08:02:22.422630+0000","last_became_peered":"2023-03-03T08:02:22.422630+0000","last_unstale":"2023-03-03T08:02:23.413007+0000","last_undegraded":"2023-03-03T08:02:23.413007+0000","last_fullsized":"2023-03-03T08:02:23.413007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:30:04.750090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.593819+0000","last_change":"2023-03-03T08:02:22.432833+0000","last_active":"2023-03-03T08:02:23.593819+0000","last_peered":"2023-03-03T08:02:23.593819+0000","last_clean":"2023-03-03T08:02:23.593819+0000","last_became_active":"2023-03-03T08:02:22.432549+0000","last_became_peered":"2023-03-03T08:02:22.432549+0000","last_unstale":"2023-03-03T08:02:23.593819+0000","last_undegraded":"2023-03-03T08:02:23.593819+0000","last_fullsized":"2023-03-03T08:02:23.593819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:56:41.737023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595692+0000","last_change":"2023-03-03T08:02:22.432104+0000","last_active":"2023-03-03T08:02:23.595692+0000","last_peered":"2023-03-03T08:02:23.595692+0000","last_clean":"2023-03-03T08:02:23.595692+0000","last_became_active":"2023-03-03T08:02:22.431815+0000","last_became_peered":"2023-03-03T08:02:22.431815+0000","last_unstale":"2023-03-03T08:02:23.595692+0000","last_undegraded":"2023-03-03T08:02:23.595692+0000","last_fullsized":"2023-03-03T08:02:23.595692+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:29:38.960491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594961+0000","last_change":"2023-03-03T08:02:22.432477+0000","last_active":"2023-03-03T08:02:23.594961+0000","last_peered":"2023-03-03T08:02:23.594961+0000","last_clean":"2023-03-03T08:02:23.594961+0000","last_became_active":"2023-03-03T08:02:22.431964+0000","last_became_peered":"2023-03-03T08:02:22.431964+0000","last_unstale":"2023-03-03T08:02:23.594961+0000","last_undegraded":"2023-03-03T08:02:23.594961+0000","last_fullsized":"2023-03-03T08:02:23.594961+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:35:35.596312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"90'4","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.446762+0000","last_change":"2023-03-03T08:02:21.879191+0000","last_active":"2023-03-03T08:02:22.446762+0000","last_peered":"2023-03-03T08:02:22.446762+0000","last_clean":"2023-03-03T08:02:22.446762+0000","last_became_active":"2023-03-03T08:02:21.878925+0000","last_became_peered":"2023-03-03T08:02:21.878925+0000","last_unstale":"2023-03-03T08:02:22.446762+0000","last_undegraded":"2023-03-03T08:02:22.446762+0000","last_fullsized":"2023-03-03T08:02:22.446762+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:06:27.575316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.596405+0000","last_change":"2023-03-03T08:02:22.435681+0000","last_active":"2023-03-03T08:02:23.596405+0000","last_peered":"2023-03-03T08:02:23.596405+0000","last_clean":"2023-03-03T08:02:23.596405+0000","last_became_active":"2023-03-03T08:02:22.435258+0000","last_became_peered":"2023-03-03T08:02:22.435258+0000","last_unstale":"2023-03-03T08:02:23.596405+0000","last_undegraded":"2023-03-03T08:02:23.596405+0000","last_fullsized":"2023-03-03T08:02:23.596405+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:10:30.503825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"90'1","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.440602+0000","last_change":"2023-03-03T08:02:21.874289+0000","last_active":"2023-03-03T08:02:22.440602+0000","last_peered":"2023-03-03T08:02:22.440602+0000","last_clean":"2023-03-03T08:02:22.440602+0000","last_became_active":"2023-03-03T08:02:21.873933+0000","last_became_peered":"2023-03-03T08:02:21.873933+0000","last_unstale":"2023-03-03T08:02:22.440602+0000","last_undegraded":"2023-03-03T08:02:22.440602+0000","last_fullsized":"2023-03-03T08:02:22.440602+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:11.554773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.411603+0000","last_change":"2023-03-03T08:02:22.430970+0000","last_active":"2023-03-03T08:02:23.411603+0000","last_peered":"2023-03-03T08:02:23.411603+0000","last_clean":"2023-03-03T08:02:23.411603+0000","last_became_active":"2023-03-03T08:02:22.430659+0000","last_became_peered":"2023-03-03T08:02:22.430659+0000","last_unstale":"2023-03-03T08:02:23.411603+0000","last_undegraded":"2023-03-03T08:02:23.411603+0000","last_fullsized":"2023-03-03T08:02:23.411603+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:11.514077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"90'2","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.439771+0000","last_change":"2023-03-03T08:02:21.870287+0000","last_active":"2023-03-03T08:02:22.439771+0000","last_peered":"2023-03-03T08:02:22.439771+0000","last_clean":"2023-03-03T08:02:22.439771+0000","last_became_active":"2023-03-03T08:02:21.870017+0000","last_became_peered":"2023-03-03T08:02:21.870017+0000","last_unstale":"2023-03-03T08:02:22.439771+0000","last_undegraded":"2023-03-03T08:02:22.439771+0000","last_fullsized":"2023-03-03T08:02:22.439771+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:11.554773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.596327+0000","last_change":"2023-03-03T08:02:22.441110+0000","last_active":"2023-03-03T08:02:23.596327+0000","last_peered":"2023-03-03T08:02:23.596327+0000","last_clean":"2023-03-03T08:02:23.596327+0000","last_became_active":"2023-03-03T08:02:22.440860+0000","last_became_peered":"2023-03-03T08:02:22.440860+0000","last_unstale":"2023-03-03T08:02:23.596327+0000","last_undegraded":"2023-03-03T08:02:23.596327+0000","last_fullsized":"2023-03-03T08:02:23.596327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:57:04.878329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"90'2","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.439508+0000","last_change":"2023-03-03T08:02:21.878531+0000","last_active":"2023-03-03T08:02:22.439508+0000","last_peered":"2023-03-03T08:02:22.439508+0000","last_clean":"2023-03-03T08:02:22.439508+0000","last_became_active":"2023-03-03T08:02:21.878208+0000","last_became_peered":"2023-03-03T08:02:21.878208+0000","last_unstale":"2023-03-03T08:02:22.439508+0000","last_undegraded":"2023-03-03T08:02:22.439508+0000","last_fullsized":"2023-03-03T08:02:22.439508+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:12:42.377302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595948+0000","last_change":"2023-03-03T08:02:22.433265+0000","last_active":"2023-03-03T08:02:23.595948+0000","last_peered":"2023-03-03T08:02:23.595948+0000","last_clean":"2023-03-03T08:02:23.595948+0000","last_became_active":"2023-03-03T08:02:22.432894+0000","last_became_peered":"2023-03-03T08:02:22.432894+0000","last_unstale":"2023-03-03T08:02:23.595948+0000","last_undegraded":"2023-03-03T08:02:23.595948+0000","last_fullsized":"2023-03-03T08:02:23.595948+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:03.788343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"90'1","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.440387+0000","last_change":"2023-03-03T08:02:21.871979+0000","last_active":"2023-03-03T08:02:22.440387+0000","last_peered":"2023-03-03T08:02:22.440387+0000","last_clean":"2023-03-03T08:02:22.440387+0000","last_became_active":"2023-03-03T08:02:21.871079+0000","last_became_peered":"2023-03-03T08:02:21.871079+0000","last_unstale":"2023-03-03T08:02:22.440387+0000","last_undegraded":"2023-03-03T08:02:22.440387+0000","last_fullsized":"2023-03-03T08:02:22.440387+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:55.500998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594380+0000","last_change":"2023-03-03T08:02:22.433798+0000","last_active":"2023-03-03T08:02:23.594380+0000","last_peered":"2023-03-03T08:02:23.594380+0000","last_clean":"2023-03-03T08:02:23.594380+0000","last_became_active":"2023-03-03T08:02:22.433494+0000","last_became_peered":"2023-03-03T08:02:22.433494+0000","last_unstale":"2023-03-03T08:02:23.594380+0000","last_undegraded":"2023-03-03T08:02:23.594380+0000","last_fullsized":"2023-03-03T08:02:23.594380+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:08.049210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"90'1","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.444249+0000","last_change":"2023-03-03T08:02:21.869557+0000","last_active":"2023-03-03T08:02:22.444249+0000","last_peered":"2023-03-03T08:02:22.444249+0000","last_clean":"2023-03-03T08:02:22.444249+0000","last_became_active":"2023-03-03T08:02:21.869249+0000","last_became_peered":"2023-03-03T08:02:21.869249+0000","last_unstale":"2023-03-03T08:02:22.444249+0000","last_undegraded":"2023-03-03T08:02:22.444249+0000","last_fullsized":"2023-03-03T08:02:22.444249+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:25:07.411525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594399+0000","last_change":"2023-03-03T08:02:22.433798+0000","last_active":"2023-03-03T08:02:23.594399+0000","last_peered":"2023-03-03T08:02:23.594399+0000","last_clean":"2023-03-03T08:02:23.594399+0000","last_became_active":"2023-03-03T08:02:22.433309+0000","last_became_peered":"2023-03-03T08:02:22.433309+0000","last_unstale":"2023-03-03T08:02:23.594399+0000","last_undegraded":"2023-03-03T08:02:23.594399+0000","last_fullsized":"2023-03-03T08:02:23.594399+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:16:15.247935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.410774+0000","last_change":"2023-03-03T08:02:21.870636+0000","last_active":"2023-03-03T08:02:22.410774+0000","last_peered":"2023-03-03T08:02:22.410774+0000","last_clean":"2023-03-03T08:02:22.410774+0000","last_became_active":"2023-03-03T08:02:21.870277+0000","last_became_peered":"2023-03-03T08:02:21.870277+0000","last_unstale":"2023-03-03T08:02:22.410774+0000","last_undegraded":"2023-03-03T08:02:22.410774+0000","last_fullsized":"2023-03-03T08:02:22.410774+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:05:18.330384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594297+0000","last_change":"2023-03-03T08:02:22.438281+0000","last_active":"2023-03-03T08:02:23.594297+0000","last_peered":"2023-03-03T08:02:23.594297+0000","last_clean":"2023-03-03T08:02:23.594297+0000","last_became_active":"2023-03-03T08:02:22.437991+0000","last_became_peered":"2023-03-03T08:02:22.437991+0000","last_unstale":"2023-03-03T08:02:23.594297+0000","last_undegraded":"2023-03-03T08:02:23.594297+0000","last_fullsized":"2023-03-03T08:02:23.594297+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:18:38.349625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.417767+0000","last_change":"2023-03-03T08:02:21.877758+0000","last_active":"2023-03-03T08:02:22.417767+0000","last_peered":"2023-03-03T08:02:22.417767+0000","last_clean":"2023-03-03T08:02:22.417767+0000","last_became_active":"2023-03-03T08:02:21.877460+0000","last_became_peered":"2023-03-03T08:02:21.877460+0000","last_unstale":"2023-03-03T08:02:22.417767+0000","last_undegraded":"2023-03-03T08:02:22.417767+0000","last_fullsized":"2023-03-03T08:02:22.417767+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:05:37.026563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.413046+0000","last_change":"2023-03-03T08:02:22.425333+0000","last_active":"2023-03-03T08:02:23.413046+0000","last_peered":"2023-03-03T08:02:23.413046+0000","last_clean":"2023-03-03T08:02:23.413046+0000","last_became_active":"2023-03-03T08:02:22.421632+0000","last_became_peered":"2023-03-03T08:02:22.421632+0000","last_unstale":"2023-03-03T08:02:23.413046+0000","last_undegraded":"2023-03-03T08:02:23.413046+0000","last_fullsized":"2023-03-03T08:02:23.413046+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:23:35.028360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.416440+0000","last_change":"2023-03-03T08:02:21.871439+0000","last_active":"2023-03-03T08:02:22.416440+0000","last_peered":"2023-03-03T08:02:22.416440+0000","last_clean":"2023-03-03T08:02:22.416440+0000","last_became_active":"2023-03-03T08:02:21.870955+0000","last_became_peered":"2023-03-03T08:02:21.870955+0000","last_unstale":"2023-03-03T08:02:22.416440+0000","last_undegraded":"2023-03-03T08:02:22.416440+0000","last_fullsized":"2023-03-03T08:02:22.416440+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:43:59.937532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.411713+0000","last_change":"2023-03-03T08:02:22.424754+0000","last_active":"2023-03-03T08:02:23.411713+0000","last_peered":"2023-03-03T08:02:23.411713+0000","last_clean":"2023-03-03T08:02:23.411713+0000","last_became_active":"2023-03-03T08:02:22.424452+0000","last_became_peered":"2023-03-03T08:02:22.424452+0000","last_unstale":"2023-03-03T08:02:23.411713+0000","last_undegraded":"2023-03-03T08:02:23.411713+0000","last_fullsized":"2023-03-03T08:02:23.411713+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:21:32.221194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.411881+0000","last_change":"2023-03-03T08:02:21.878213+0000","last_active":"2023-03-03T08:02:22.411881+0000","last_peered":"2023-03-03T08:02:22.411881+0000","last_clean":"2023-03-03T08:02:22.411881+0000","last_became_active":"2023-03-03T08:02:21.877934+0000","last_became_peered":"2023-03-03T08:02:21.877934+0000","last_unstale":"2023-03-03T08:02:22.411881+0000","last_undegraded":"2023-03-03T08:02:22.411881+0000","last_fullsized":"2023-03-03T08:02:22.411881+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:11.554773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594809+0000","last_change":"2023-03-03T08:02:22.431176+0000","last_active":"2023-03-03T08:02:23.594809+0000","last_peered":"2023-03-03T08:02:23.594809+0000","last_clean":"2023-03-03T08:02:23.594809+0000","last_became_active":"2023-03-03T08:02:22.430932+0000","last_became_peered":"2023-03-03T08:02:22.430932+0000","last_unstale":"2023-03-03T08:02:23.594809+0000","last_undegraded":"2023-03-03T08:02:23.594809+0000","last_fullsized":"2023-03-03T08:02:23.594809+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:11.514077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"90'3","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.447168+0000","last_change":"2023-03-03T08:02:21.878651+0000","last_active":"2023-03-03T08:02:22.447168+0000","last_peered":"2023-03-03T08:02:22.447168+0000","last_clean":"2023-03-03T08:02:22.447168+0000","last_became_active":"2023-03-03T08:02:21.878379+0000","last_became_peered":"2023-03-03T08:02:21.878379+0000","last_unstale":"2023-03-03T08:02:22.447168+0000","last_undegraded":"2023-03-03T08:02:22.447168+0000","last_fullsized":"2023-03-03T08:02:22.447168+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:04:25.058258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594398+0000","last_change":"2023-03-03T08:02:22.429360+0000","last_active":"2023-03-03T08:02:23.594398+0000","last_peered":"2023-03-03T08:02:23.594398+0000","last_clean":"2023-03-03T08:02:23.594398+0000","last_became_active":"2023-03-03T08:02:22.429108+0000","last_became_peered":"2023-03-03T08:02:22.429108+0000","last_unstale":"2023-03-03T08:02:23.594398+0000","last_undegraded":"2023-03-03T08:02:23.594398+0000","last_fullsized":"2023-03-03T08:02:23.594398+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:06:00.784892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"90'6","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.447872+0000","last_change":"2023-03-03T08:02:21.876477+0000","last_active":"2023-03-03T08:02:22.447872+0000","last_peered":"2023-03-03T08:02:22.447872+0000","last_clean":"2023-03-03T08:02:22.447872+0000","last_became_active":"2023-03-03T08:02:21.876153+0000","last_became_peered":"2023-03-03T08:02:21.876153+0000","last_unstale":"2023-03-03T08:02:22.447872+0000","last_undegraded":"2023-03-03T08:02:22.447872+0000","last_fullsized":"2023-03-03T08:02:22.447872+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:20:03.570463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.596042+0000","last_change":"2023-03-03T08:02:22.427140+0000","last_active":"2023-03-03T08:02:23.596042+0000","last_peered":"2023-03-03T08:02:23.596042+0000","last_clean":"2023-03-03T08:02:23.596042+0000","last_became_active":"2023-03-03T08:02:22.426570+0000","last_became_peered":"2023-03-03T08:02:22.426570+0000","last_unstale":"2023-03-03T08:02:23.596042+0000","last_undegraded":"2023-03-03T08:02:23.596042+0000","last_fullsized":"2023-03-03T08:02:23.596042+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:04:26.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"90'2","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.447529+0000","last_change":"2023-03-03T08:02:21.877740+0000","last_active":"2023-03-03T08:02:22.447529+0000","last_peered":"2023-03-03T08:02:22.447529+0000","last_clean":"2023-03-03T08:02:22.447529+0000","last_became_active":"2023-03-03T08:02:21.877334+0000","last_became_peered":"2023-03-03T08:02:21.877334+0000","last_unstale":"2023-03-03T08:02:22.447529+0000","last_undegraded":"2023-03-03T08:02:22.447529+0000","last_fullsized":"2023-03-03T08:02:22.447529+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:02.784516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.412761+0000","last_change":"2023-03-03T08:02:22.429658+0000","last_active":"2023-03-03T08:02:23.412761+0000","last_peered":"2023-03-03T08:02:23.412761+0000","last_clean":"2023-03-03T08:02:23.412761+0000","last_became_active":"2023-03-03T08:02:22.429355+0000","last_became_peered":"2023-03-03T08:02:22.429355+0000","last_unstale":"2023-03-03T08:02:23.412761+0000","last_undegraded":"2023-03-03T08:02:23.412761+0000","last_fullsized":"2023-03-03T08:02:23.412761+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:42.565751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.421281+0000","last_change":"2023-03-03T08:02:21.877446+0000","last_active":"2023-03-03T08:02:22.421281+0000","last_peered":"2023-03-03T08:02:22.421281+0000","last_clean":"2023-03-03T08:02:22.421281+0000","last_became_active":"2023-03-03T08:02:21.877142+0000","last_became_peered":"2023-03-03T08:02:21.877142+0000","last_unstale":"2023-03-03T08:02:22.421281+0000","last_undegraded":"2023-03-03T08:02:22.421281+0000","last_fullsized":"2023-03-03T08:02:22.421281+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:30:03.746263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594293+0000","last_change":"2023-03-03T08:02:22.441219+0000","last_active":"2023-03-03T08:02:23.594293+0000","last_peered":"2023-03-03T08:02:23.594293+0000","last_clean":"2023-03-03T08:02:23.594293+0000","last_became_active":"2023-03-03T08:02:22.440937+0000","last_became_peered":"2023-03-03T08:02:22.440937+0000","last_unstale":"2023-03-03T08:02:23.594293+0000","last_undegraded":"2023-03-03T08:02:23.594293+0000","last_fullsized":"2023-03-03T08:02:23.594293+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:39:31.044775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.416561+0000","last_change":"2023-03-03T08:02:21.875342+0000","last_active":"2023-03-03T08:02:22.416561+0000","last_peered":"2023-03-03T08:02:22.416561+0000","last_clean":"2023-03-03T08:02:22.416561+0000","last_became_active":"2023-03-03T08:02:21.874999+0000","last_became_peered":"2023-03-03T08:02:21.874999+0000","last_unstale":"2023-03-03T08:02:22.416561+0000","last_undegraded":"2023-03-03T08:02:22.416561+0000","last_fullsized":"2023-03-03T08:02:22.416561+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:14.083774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594646+0000","last_change":"2023-03-03T08:02:22.441976+0000","last_active":"2023-03-03T08:02:23.594646+0000","last_peered":"2023-03-03T08:02:23.594646+0000","last_clean":"2023-03-03T08:02:23.594646+0000","last_became_active":"2023-03-03T08:02:22.441582+0000","last_became_peered":"2023-03-03T08:02:22.441582+0000","last_unstale":"2023-03-03T08:02:23.594646+0000","last_undegraded":"2023-03-03T08:02:23.594646+0000","last_fullsized":"2023-03-03T08:02:23.594646+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:29:47.652120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.446476+0000","last_change":"2023-03-03T08:02:21.870898+0000","last_active":"2023-03-03T08:02:22.446476+0000","last_peered":"2023-03-03T08:02:22.446476+0000","last_clean":"2023-03-03T08:02:22.446476+0000","last_became_active":"2023-03-03T08:02:21.870586+0000","last_became_peered":"2023-03-03T08:02:21.870586+0000","last_unstale":"2023-03-03T08:02:22.446476+0000","last_undegraded":"2023-03-03T08:02:22.446476+0000","last_fullsized":"2023-03-03T08:02:22.446476+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:29:07.480349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594228+0000","last_change":"2023-03-03T08:02:22.429696+0000","last_active":"2023-03-03T08:02:23.594228+0000","last_peered":"2023-03-03T08:02:23.594228+0000","last_clean":"2023-03-03T08:02:23.594228+0000","last_became_active":"2023-03-03T08:02:22.429407+0000","last_became_peered":"2023-03-03T08:02:22.429407+0000","last_unstale":"2023-03-03T08:02:23.594228+0000","last_undegraded":"2023-03-03T08:02:23.594228+0000","last_fullsized":"2023-03-03T08:02:23.594228+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:09:13.117580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"90'2","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.440540+0000","last_change":"2023-03-03T08:02:21.875105+0000","last_active":"2023-03-03T08:02:22.440540+0000","last_peered":"2023-03-03T08:02:22.440540+0000","last_clean":"2023-03-03T08:02:22.440540+0000","last_became_active":"2023-03-03T08:02:21.874788+0000","last_became_peered":"2023-03-03T08:02:21.874788+0000","last_unstale":"2023-03-03T08:02:22.440540+0000","last_undegraded":"2023-03-03T08:02:22.440540+0000","last_fullsized":"2023-03-03T08:02:22.440540+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:05:18.330384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.596405+0000","last_change":"2023-03-03T08:02:22.436727+0000","last_active":"2023-03-03T08:02:23.596405+0000","last_peered":"2023-03-03T08:02:23.596405+0000","last_clean":"2023-03-03T08:02:23.596405+0000","last_became_active":"2023-03-03T08:02:22.432731+0000","last_became_peered":"2023-03-03T08:02:22.432731+0000","last_unstale":"2023-03-03T08:02:23.596405+0000","last_undegraded":"2023-03-03T08:02:23.596405+0000","last_fullsized":"2023-03-03T08:02:23.596405+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:50:52.137242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.413561+0000","last_change":"2023-03-03T08:02:21.874614+0000","last_active":"2023-03-03T08:02:22.413561+0000","last_peered":"2023-03-03T08:02:22.413561+0000","last_clean":"2023-03-03T08:02:22.413561+0000","last_became_active":"2023-03-03T08:02:21.874310+0000","last_became_peered":"2023-03-03T08:02:21.874310+0000","last_unstale":"2023-03-03T08:02:22.413561+0000","last_undegraded":"2023-03-03T08:02:22.413561+0000","last_fullsized":"2023-03-03T08:02:22.413561+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:21.946190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594760+0000","last_change":"2023-03-03T08:02:22.434939+0000","last_active":"2023-03-03T08:02:23.594760+0000","last_peered":"2023-03-03T08:02:23.594760+0000","last_clean":"2023-03-03T08:02:23.594760+0000","last_became_active":"2023-03-03T08:02:22.434169+0000","last_became_peered":"2023-03-03T08:02:22.434169+0000","last_unstale":"2023-03-03T08:02:23.594760+0000","last_undegraded":"2023-03-03T08:02:23.594760+0000","last_fullsized":"2023-03-03T08:02:23.594760+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:20:05.910132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.413479+0000","last_change":"2023-03-03T08:02:21.877730+0000","last_active":"2023-03-03T08:02:22.413479+0000","last_peered":"2023-03-03T08:02:22.413479+0000","last_clean":"2023-03-03T08:02:22.413479+0000","last_became_active":"2023-03-03T08:02:21.877447+0000","last_became_peered":"2023-03-03T08:02:21.877447+0000","last_unstale":"2023-03-03T08:02:22.413479+0000","last_undegraded":"2023-03-03T08:02:22.413479+0000","last_fullsized":"2023-03-03T08:02:22.413479+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:06.949059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594547+0000","last_change":"2023-03-03T08:02:22.424455+0000","last_active":"2023-03-03T08:02:23.594547+0000","last_peered":"2023-03-03T08:02:23.594547+0000","last_clean":"2023-03-03T08:02:23.594547+0000","last_became_active":"2023-03-03T08:02:22.424157+0000","last_became_peered":"2023-03-03T08:02:22.424157+0000","last_unstale":"2023-03-03T08:02:23.594547+0000","last_undegraded":"2023-03-03T08:02:23.594547+0000","last_fullsized":"2023-03-03T08:02:23.594547+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:18:38.349625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.421462+0000","last_change":"2023-03-03T08:02:21.879897+0000","last_active":"2023-03-03T08:02:22.421462+0000","last_peered":"2023-03-03T08:02:22.421462+0000","last_clean":"2023-03-03T08:02:22.421462+0000","last_became_active":"2023-03-03T08:02:21.879628+0000","last_became_peered":"2023-03-03T08:02:21.879628+0000","last_unstale":"2023-03-03T08:02:22.421462+0000","last_undegraded":"2023-03-03T08:02:22.421462+0000","last_fullsized":"2023-03-03T08:02:22.421462+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:40:24.235160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595330+0000","last_change":"2023-03-03T08:02:22.432956+0000","last_active":"2023-03-03T08:02:23.595330+0000","last_peered":"2023-03-03T08:02:23.595330+0000","last_clean":"2023-03-03T08:02:23.595330+0000","last_became_active":"2023-03-03T08:02:22.432701+0000","last_became_peered":"2023-03-03T08:02:22.432701+0000","last_unstale":"2023-03-03T08:02:23.595330+0000","last_undegraded":"2023-03-03T08:02:23.595330+0000","last_fullsized":"2023-03-03T08:02:23.595330+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:18:46.954636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.420995+0000","last_change":"2023-03-03T08:02:21.877397+0000","last_active":"2023-03-03T08:02:22.420995+0000","last_peered":"2023-03-03T08:02:22.420995+0000","last_clean":"2023-03-03T08:02:22.420995+0000","last_became_active":"2023-03-03T08:02:21.877124+0000","last_became_peered":"2023-03-03T08:02:21.877124+0000","last_unstale":"2023-03-03T08:02:22.420995+0000","last_undegraded":"2023-03-03T08:02:22.420995+0000","last_fullsized":"2023-03-03T08:02:22.420995+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:07:16.560533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595601+0000","last_change":"2023-03-03T08:02:22.430259+0000","last_active":"2023-03-03T08:02:23.595601+0000","last_peered":"2023-03-03T08:02:23.595601+0000","last_clean":"2023-03-03T08:02:23.595601+0000","last_became_active":"2023-03-03T08:02:22.429912+0000","last_became_peered":"2023-03-03T08:02:22.429912+0000","last_unstale":"2023-03-03T08:02:23.595601+0000","last_undegraded":"2023-03-03T08:02:23.595601+0000","last_fullsized":"2023-03-03T08:02:23.595601+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:21:32.221194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.414724+0000","last_change":"2023-03-03T08:02:21.868423+0000","last_active":"2023-03-03T08:02:22.414724+0000","last_peered":"2023-03-03T08:02:22.414724+0000","last_clean":"2023-03-03T08:02:22.414724+0000","last_became_active":"2023-03-03T08:02:21.868122+0000","last_became_peered":"2023-03-03T08:02:21.868122+0000","last_unstale":"2023-03-03T08:02:22.414724+0000","last_undegraded":"2023-03-03T08:02:22.414724+0000","last_fullsized":"2023-03-03T08:02:22.414724+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:27:52.756150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594073+0000","last_change":"2023-03-03T08:02:22.432093+0000","last_active":"2023-03-03T08:02:23.594073+0000","last_peered":"2023-03-03T08:02:23.594073+0000","last_clean":"2023-03-03T08:02:23.594073+0000","last_became_active":"2023-03-03T08:02:22.431777+0000","last_became_peered":"2023-03-03T08:02:22.431777+0000","last_unstale":"2023-03-03T08:02:23.594073+0000","last_undegraded":"2023-03-03T08:02:23.594073+0000","last_fullsized":"2023-03-03T08:02:23.594073+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:01:26.166146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.420496+0000","last_change":"2023-03-03T08:02:21.880896+0000","last_active":"2023-03-03T08:02:22.420496+0000","last_peered":"2023-03-03T08:02:22.420496+0000","last_clean":"2023-03-03T08:02:22.420496+0000","last_became_active":"2023-03-03T08:02:21.880537+0000","last_became_peered":"2023-03-03T08:02:21.880537+0000","last_unstale":"2023-03-03T08:02:22.420496+0000","last_undegraded":"2023-03-03T08:02:22.420496+0000","last_fullsized":"2023-03-03T08:02:22.420496+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:28.785586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595357+0000","last_change":"2023-03-03T08:02:22.444781+0000","last_active":"2023-03-03T08:02:23.595357+0000","last_peered":"2023-03-03T08:02:23.595357+0000","last_clean":"2023-03-03T08:02:23.595357+0000","last_became_active":"2023-03-03T08:02:22.444469+0000","last_became_peered":"2023-03-03T08:02:22.444469+0000","last_unstale":"2023-03-03T08:02:23.595357+0000","last_undegraded":"2023-03-03T08:02:23.595357+0000","last_fullsized":"2023-03-03T08:02:23.595357+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:22:38.933853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.417895+0000","last_change":"2023-03-03T08:02:21.878700+0000","last_active":"2023-03-03T08:02:22.417895+0000","last_peered":"2023-03-03T08:02:22.417895+0000","last_clean":"2023-03-03T08:02:22.417895+0000","last_became_active":"2023-03-03T08:02:21.878429+0000","last_became_peered":"2023-03-03T08:02:21.878429+0000","last_unstale":"2023-03-03T08:02:22.417895+0000","last_undegraded":"2023-03-03T08:02:22.417895+0000","last_fullsized":"2023-03-03T08:02:22.417895+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:38.341401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595201+0000","last_change":"2023-03-03T08:02:22.436418+0000","last_active":"2023-03-03T08:02:23.595201+0000","last_peered":"2023-03-03T08:02:23.595201+0000","last_clean":"2023-03-03T08:02:23.595201+0000","last_became_active":"2023-03-03T08:02:22.436168+0000","last_became_peered":"2023-03-03T08:02:22.436168+0000","last_unstale":"2023-03-03T08:02:23.595201+0000","last_undegraded":"2023-03-03T08:02:23.595201+0000","last_fullsized":"2023-03-03T08:02:23.595201+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:36:52.462067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.424333+0000","last_change":"2023-03-03T08:02:21.874537+0000","last_active":"2023-03-03T08:02:22.424333+0000","last_peered":"2023-03-03T08:02:22.424333+0000","last_clean":"2023-03-03T08:02:22.424333+0000","last_became_active":"2023-03-03T08:02:21.874287+0000","last_became_peered":"2023-03-03T08:02:21.874287+0000","last_unstale":"2023-03-03T08:02:22.424333+0000","last_undegraded":"2023-03-03T08:02:22.424333+0000","last_fullsized":"2023-03-03T08:02:22.424333+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:26:41.817800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.419690+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T08:02:22.419690+0000","last_peered":"2023-03-03T08:02:22.419690+0000","last_clean":"2023-03-03T08:02:22.419690+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T08:02:22.419690+0000","last_undegraded":"2023-03-03T08:02:22.419690+0000","last_fullsized":"2023-03-03T08:02:22.419690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.593940+0000","last_change":"2023-03-03T08:02:22.434463+0000","last_active":"2023-03-03T08:02:23.593940+0000","last_peered":"2023-03-03T08:02:23.593940+0000","last_clean":"2023-03-03T08:02:23.593940+0000","last_became_active":"2023-03-03T08:02:22.434172+0000","last_became_peered":"2023-03-03T08:02:22.434172+0000","last_unstale":"2023-03-03T08:02:23.593940+0000","last_undegraded":"2023-03-03T08:02:23.593940+0000","last_fullsized":"2023-03-03T08:02:23.593940+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:44:24.059066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.414103+0000","last_change":"2023-03-03T08:02:21.877158+0000","last_active":"2023-03-03T08:02:22.414103+0000","last_peered":"2023-03-03T08:02:22.414103+0000","last_clean":"2023-03-03T08:02:22.414103+0000","last_became_active":"2023-03-03T08:02:21.876773+0000","last_became_peered":"2023-03-03T08:02:21.876773+0000","last_unstale":"2023-03-03T08:02:22.414103+0000","last_undegraded":"2023-03-03T08:02:22.414103+0000","last_fullsized":"2023-03-03T08:02:22.414103+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:55.500998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.417890+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T08:02:22.417890+0000","last_peered":"2023-03-03T08:02:22.417890+0000","last_clean":"2023-03-03T08:02:22.417890+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T08:02:22.417890+0000","last_undegraded":"2023-03-03T08:02:22.417890+0000","last_fullsized":"2023-03-03T08:02:22.417890+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.593819+0000","last_change":"2023-03-03T08:02:22.430818+0000","last_active":"2023-03-03T08:02:23.593819+0000","last_peered":"2023-03-03T08:02:23.593819+0000","last_clean":"2023-03-03T08:02:23.593819+0000","last_became_active":"2023-03-03T08:02:22.430476+0000","last_became_peered":"2023-03-03T08:02:22.430476+0000","last_unstale":"2023-03-03T08:02:23.593819+0000","last_undegraded":"2023-03-03T08:02:23.593819+0000","last_fullsized":"2023-03-03T08:02:23.593819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:05:38.030389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.422265+0000","last_change":"2023-03-03T08:02:21.878320+0000","last_active":"2023-03-03T08:02:22.422265+0000","last_peered":"2023-03-03T08:02:22.422265+0000","last_clean":"2023-03-03T08:02:22.422265+0000","last_became_active":"2023-03-03T08:02:21.877985+0000","last_became_peered":"2023-03-03T08:02:21.877985+0000","last_unstale":"2023-03-03T08:02:22.422265+0000","last_undegraded":"2023-03-03T08:02:22.422265+0000","last_fullsized":"2023-03-03T08:02:22.422265+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:58:25.892217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.416981+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T08:02:22.416981+0000","last_peered":"2023-03-03T08:02:22.416981+0000","last_clean":"2023-03-03T08:02:22.416981+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T08:02:22.416981+0000","last_undegraded":"2023-03-03T08:02:22.416981+0000","last_fullsized":"2023-03-03T08:02:22.416981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594871+0000","last_change":"2023-03-03T08:02:22.432594+0000","last_active":"2023-03-03T08:02:23.594871+0000","last_peered":"2023-03-03T08:02:23.594871+0000","last_clean":"2023-03-03T08:02:23.594871+0000","last_became_active":"2023-03-03T08:02:22.432316+0000","last_became_peered":"2023-03-03T08:02:22.432316+0000","last_unstale":"2023-03-03T08:02:23.594871+0000","last_undegraded":"2023-03-03T08:02:23.594871+0000","last_fullsized":"2023-03-03T08:02:23.594871+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:21:10.960686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"90'1","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.439455+0000","last_change":"2023-03-03T08:02:21.879578+0000","last_active":"2023-03-03T08:02:22.439455+0000","last_peered":"2023-03-03T08:02:22.439455+0000","last_clean":"2023-03-03T08:02:22.439455+0000","last_became_active":"2023-03-03T08:02:21.879264+0000","last_became_peered":"2023-03-03T08:02:21.879264+0000","last_unstale":"2023-03-03T08:02:22.439455+0000","last_undegraded":"2023-03-03T08:02:22.439455+0000","last_fullsized":"2023-03-03T08:02:22.439455+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:37.978753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.417999+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T08:02:22.417999+0000","last_peered":"2023-03-03T08:02:22.417999+0000","last_clean":"2023-03-03T08:02:22.417999+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T08:02:22.417999+0000","last_undegraded":"2023-03-03T08:02:22.417999+0000","last_fullsized":"2023-03-03T08:02:22.417999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.411854+0000","last_change":"2023-03-03T08:02:22.433910+0000","last_active":"2023-03-03T08:02:23.411854+0000","last_peered":"2023-03-03T08:02:23.411854+0000","last_clean":"2023-03-03T08:02:23.411854+0000","last_became_active":"2023-03-03T08:02:22.433646+0000","last_became_peered":"2023-03-03T08:02:22.433646+0000","last_unstale":"2023-03-03T08:02:23.411854+0000","last_undegraded":"2023-03-03T08:02:23.411854+0000","last_fullsized":"2023-03-03T08:02:23.411854+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:45.588606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.416680+0000","last_change":"2023-03-03T08:02:21.873962+0000","last_active":"2023-03-03T08:02:22.416680+0000","last_peered":"2023-03-03T08:02:22.416680+0000","last_clean":"2023-03-03T08:02:22.416680+0000","last_became_active":"2023-03-03T08:02:21.873697+0000","last_became_peered":"2023-03-03T08:02:21.873697+0000","last_unstale":"2023-03-03T08:02:22.416680+0000","last_undegraded":"2023-03-03T08:02:22.416680+0000","last_fullsized":"2023-03-03T08:02:22.416680+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:37.978753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":266,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.418046+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T08:02:22.418046+0000","last_peered":"2023-03-03T08:02:22.418046+0000","last_clean":"2023-03-03T08:02:22.418046+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T08:02:22.418046+0000","last_undegraded":"2023-03-03T08:02:22.418046+0000","last_fullsized":"2023-03-03T08:02:22.418046+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.415422+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T08:02:22.415422+0000","last_peered":"2023-03-03T08:02:22.415422+0000","last_clean":"2023-03-03T08:02:22.415422+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T08:02:22.415422+0000","last_undegraded":"2023-03-03T08:02:22.415422+0000","last_fullsized":"2023-03-03T08:02:22.415422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594327+0000","last_change":"2023-03-03T08:02:22.432064+0000","last_active":"2023-03-03T08:02:23.594327+0000","last_peered":"2023-03-03T08:02:23.594327+0000","last_clean":"2023-03-03T08:02:23.594327+0000","last_became_active":"2023-03-03T08:02:22.431673+0000","last_became_peered":"2023-03-03T08:02:22.431673+0000","last_unstale":"2023-03-03T08:02:23.594327+0000","last_undegraded":"2023-03-03T08:02:23.594327+0000","last_fullsized":"2023-03-03T08:02:23.594327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:16.102515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"90'2","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.440917+0000","last_change":"2023-03-03T08:02:21.874486+0000","last_active":"2023-03-03T08:02:22.440917+0000","last_peered":"2023-03-03T08:02:22.440917+0000","last_clean":"2023-03-03T08:02:22.440917+0000","last_became_active":"2023-03-03T08:02:21.874149+0000","last_became_peered":"2023-03-03T08:02:21.874149+0000","last_unstale":"2023-03-03T08:02:22.440917+0000","last_undegraded":"2023-03-03T08:02:22.440917+0000","last_fullsized":"2023-03-03T08:02:22.440917+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:05:59.781066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.422063+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T08:02:22.422063+0000","last_peered":"2023-03-03T08:02:22.422063+0000","last_clean":"2023-03-03T08:02:22.422063+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T08:02:22.422063+0000","last_undegraded":"2023-03-03T08:02:22.422063+0000","last_fullsized":"2023-03-03T08:02:22.422063+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.413104+0000","last_change":"2023-03-03T08:02:22.431014+0000","last_active":"2023-03-03T08:02:23.413104+0000","last_peered":"2023-03-03T08:02:23.413104+0000","last_clean":"2023-03-03T08:02:23.413104+0000","last_became_active":"2023-03-03T08:02:22.430177+0000","last_became_peered":"2023-03-03T08:02:22.430177+0000","last_unstale":"2023-03-03T08:02:23.413104+0000","last_undegraded":"2023-03-03T08:02:23.413104+0000","last_fullsized":"2023-03-03T08:02:23.413104+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:21:32.221194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"90'1","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.440807+0000","last_change":"2023-03-03T08:02:21.878713+0000","last_active":"2023-03-03T08:02:22.440807+0000","last_peered":"2023-03-03T08:02:22.440807+0000","last_clean":"2023-03-03T08:02:22.440807+0000","last_became_active":"2023-03-03T08:02:21.878470+0000","last_became_peered":"2023-03-03T08:02:21.878470+0000","last_unstale":"2023-03-03T08:02:22.440807+0000","last_undegraded":"2023-03-03T08:02:22.440807+0000","last_fullsized":"2023-03-03T08:02:22.440807+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:38.341401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.414817+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T08:02:22.414817+0000","last_peered":"2023-03-03T08:02:22.414817+0000","last_clean":"2023-03-03T08:02:22.414817+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T08:02:22.414817+0000","last_undegraded":"2023-03-03T08:02:22.414817+0000","last_fullsized":"2023-03-03T08:02:22.414817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595896+0000","last_change":"2023-03-03T08:02:22.431946+0000","last_active":"2023-03-03T08:02:23.595896+0000","last_peered":"2023-03-03T08:02:23.595896+0000","last_clean":"2023-03-03T08:02:23.595896+0000","last_became_active":"2023-03-03T08:02:22.431388+0000","last_became_peered":"2023-03-03T08:02:22.431388+0000","last_unstale":"2023-03-03T08:02:23.595896+0000","last_undegraded":"2023-03-03T08:02:23.595896+0000","last_fullsized":"2023-03-03T08:02:23.595896+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:52.697811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"90'2","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.436052+0000","last_change":"2023-03-03T08:02:21.867808+0000","last_active":"2023-03-03T08:02:22.436052+0000","last_peered":"2023-03-03T08:02:22.436052+0000","last_clean":"2023-03-03T08:02:22.436052+0000","last_became_active":"2023-03-03T08:02:21.867524+0000","last_became_peered":"2023-03-03T08:02:21.867524+0000","last_unstale":"2023-03-03T08:02:22.436052+0000","last_undegraded":"2023-03-03T08:02:22.436052+0000","last_fullsized":"2023-03-03T08:02:22.436052+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:21.946190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.416817+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T08:02:22.416817+0000","last_peered":"2023-03-03T08:02:22.416817+0000","last_clean":"2023-03-03T08:02:22.416817+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T08:02:22.416817+0000","last_undegraded":"2023-03-03T08:02:22.416817+0000","last_fullsized":"2023-03-03T08:02:22.416817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574934,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11928,"kb_used_data":1680,"kb_used_omap":0,"kb_used_meta":10240,"kb_avail":94359912,"statfs":{"total":96636764160,"available":96624549888,"internally_reserved":0,"allocated":1720320,"data_stored":1099785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10485760},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.66100000000000003,"15min":0.70099999999999996},"min":{"1min":0.58499999999999996,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":1.546},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.67700000000000005,"15min":0.71799999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.78500000000000003,"5min":0.96499999999999997,"15min":1.702},"last":0.58199999999999996}]},{"osd":1,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.70799999999999996,"15min":1.024},"min":{"1min":0.52900000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.96299999999999997,"5min":1.111,"15min":6.6829999999999998},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.49399999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.84999999999999998,"5min":1.1890000000000001,"15min":1.944},"last":0.51900000000000002}]},{"osd":2,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.72799999999999998,"15min":0.71699999999999997},"min":{"1min":0.56599999999999995,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94499999999999995,"5min":1.224,"15min":1.252},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.74099999999999999,"15min":0.78400000000000003},"min":{"1min":0.57999999999999996,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.92100000000000004,"5min":1.2490000000000001,"15min":1.329},"last":0.496}]},{"osd":3,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71399999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.72099999999999997,"15min":0.77900000000000003},"min":{"1min":0.55200000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.92400000000000004,"5min":1.0640000000000001,"15min":1.294},"last":0.64500000000000002}]},{"osd":4,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.624,"15min":0.64900000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":0.76200000000000001,"5min":1.0049999999999999,"15min":1.077},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68799999999999994,"15min":0.64100000000000001},"min":{"1min":0.48199999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.99299999999999999,"5min":1.081,"15min":1.081},"last":0.41999999999999998}]},{"osd":5,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.70799999999999996,"15min":0.78000000000000003},"min":{"1min":0.53100000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80800000000000005,"5min":1.0409999999999999,"15min":1.3819999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.78600000000000003,"15min":0.81399999999999995},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.97999999999999998,"5min":1.2190000000000001,"15min":1.349},"last":0.67800000000000005}]},{"osd":6,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.79900000000000004,"15min":0.82299999999999995},"min":{"1min":0.59199999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79800000000000004,"15min":0.81599999999999995},"min":{"1min":0.63700000000000001,"5min":0.499,"15min":0.499},"max":{"1min":1.0329999999999999,"5min":1.135,"15min":1.194},"last":0.60499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574935,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10524,"kb_used_data":1108,"kb_used_omap":0,"kb_used_meta":9408,"kb_avail":94361316,"statfs":{"total":96636764160,"available":96625987584,"internally_reserved":0,"allocated":1134592,"data_stored":510478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9633792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.73099999999999998,"15min":0.69699999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.40100000000000002},"max":{"1min":1.016,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.71099999999999997,"15min":0.70199999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.55400000000000005}]},{"osd":1,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72399999999999998,"15min":0.68999999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0900000000000001,"5min":3.323,"15min":3.323},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.73499999999999999},"min":{"1min":0.55200000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89600000000000002,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.49199999999999999}]},{"osd":2,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.75800000000000001,"15min":0.73399999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.96699999999999997,"5min":1.54,"15min":1.54},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.74199999999999999,"15min":0.75600000000000001},"min":{"1min":0.52700000000000002,"5min":0.495,"15min":0.495},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.627}]},{"osd":3,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.80800000000000005,"15min":0.78800000000000003},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0309999999999999,"5min":4.8959999999999999,"15min":4.8959999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78400000000000003,"15min":0.77500000000000002},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.55600000000000005},"max":{"1min":0.90900000000000003,"5min":1.2270000000000001,"15min":1.343},"last":0.73699999999999999}]},{"osd":4,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.70799999999999996,"15min":0.88100000000000001},"min":{"1min":0.44800000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":4.0099999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.71299999999999997,"15min":0.89300000000000002},"min":{"1min":0.47499999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.98499999999999999,"5min":1.012,"15min":4.1859999999999999},"last":0.64100000000000001}]},{"osd":5,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.69199999999999995,"15min":0.76500000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.99299999999999999,"5min":1.03,"15min":1.6339999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.79800000000000004,"15min":0.82399999999999995},"min":{"1min":0.55400000000000005,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.056,"5min":1.202,"15min":1.429},"last":0.67000000000000004}]},{"osd":7,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.80500000000000005,"15min":0.83499999999999996},"min":{"1min":0.54700000000000004,"5min":0.54100000000000004,"15min":0.51900000000000002},"max":{"1min":1.071,"5min":1.107,"15min":1.452},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.78100000000000003,"15min":0.85699999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.1259999999999999,"5min":1.135,"15min":1.579},"last":0.69899999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10896,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":9792,"kb_avail":94360944,"statfs":{"total":96636764160,"available":96625606656,"internally_reserved":0,"allocated":1122304,"data_stored":509431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10027008},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46800000000000003,"15min":0.503},"min":{"1min":0.34300000000000003,"5min":0.307,"15min":0.307},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.4970000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.50800000000000001,"15min":0.623},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.74399999999999999,"5min":0.90200000000000002,"15min":3.972},"last":0.38800000000000001}]},{"osd":2,"last update":"Fri Mar 3 08:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.52500000000000002,"15min":0.89200000000000002},"min":{"1min":0.39900000000000002,"5min":0.34899999999999998,"15min":0.313},"max":{"1min":0.77100000000000002,"5min":0.78900000000000003,"15min":11.897},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59499999999999997,"15min":0.97099999999999997},"min":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78700000000000003,"5min":0.85199999999999998,"15min":11.804},"last":0.66200000000000003}]},{"osd":3,"last update":"Fri Mar 3 08:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58799999999999997,"15min":0.96099999999999997},"min":{"1min":0.434,"5min":0.371,"15min":0.371},"max":{"1min":0.85899999999999999,"5min":0.89500000000000002,"15min":11.843999999999999},"last":0.62},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55200000000000005,"15min":0.70499999999999996},"min":{"1min":0.45100000000000001,"5min":0.40000000000000002,"15min":0.36699999999999999},"max":{"1min":0.75600000000000001,"5min":0.96399999999999997,"15min":3.8740000000000001},"last":0.55300000000000005}]},{"osd":4,"last update":"Fri Mar 3 08:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.75800000000000001,"15min":0.84299999999999997},"min":{"1min":0.55100000000000005,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90000000000000002,"5min":1.1459999999999999,"15min":1.613},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81999999999999995,"15min":0.85499999999999998},"min":{"1min":0.67100000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.4630000000000001},"last":0.83799999999999997}]},{"osd":5,"last update":"Fri Mar 3 08:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.75700000000000001,"15min":0.89500000000000002},"min":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":3.8300000000000001},"last":0.746},{"interface":"front","average":{"1min":0.752,"5min":0.753,"15min":0.85099999999999998},"min":{"1min":0.622,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.008,"5min":1.0960000000000001,"15min":1.5469999999999999},"last":0.94699999999999995}]},{"osd":6,"last update":"Fri Mar 3 08:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.81399999999999995,"15min":0.96599999999999997},"min":{"1min":0.626,"5min":0.62,"15min":0.62},"max":{"1min":0.98299999999999998,"5min":1.173,"15min":3.907},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.81399999999999995,"15min":0.91700000000000004},"min":{"1min":0.67900000000000005,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.004,"5min":1.1819999999999999,"15min":1.5820000000000001},"last":0.72399999999999998}]},{"osd":7,"last update":"Fri Mar 3 08:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.79700000000000004,"15min":0.86699999999999999},"min":{"1min":0.59299999999999997,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0840000000000001,"5min":1.417,"15min":1.419},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80000000000000004,"15min":0.97699999999999998},"min":{"1min":0.65100000000000002,"5min":0.58199999999999996,"15min":0.57099999999999995},"max":{"1min":0.96799999999999997,"5min":1.3560000000000001,"15min":3.9399999999999999},"last":0.88600000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574934,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11748,"kb_used_data":1692,"kb_used_omap":0,"kb_used_meta":10048,"kb_avail":94360092,"statfs":{"total":96636764160,"available":96624734208,"internally_reserved":0,"allocated":1732608,"data_stored":1100948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10289152},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 08:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.43099999999999999,"15min":0.46300000000000002},"min":{"1min":0.32200000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56200000000000006,"5min":0.65000000000000002,"15min":0.89300000000000002},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.501,"15min":0.51500000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.64300000000000002,"5min":0.81999999999999995,"15min":0.871},"last":0.55000000000000004}]},{"osd":2,"last update":"Fri Mar 3 08:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.50800000000000001,"15min":0.54200000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.63500000000000001,"5min":0.77400000000000002,"15min":0.79100000000000004},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55800000000000005,"15min":0.58599999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.32800000000000001},"max":{"1min":0.72199999999999998,"5min":0.80800000000000005,"15min":1.1020000000000001},"last":0.44400000000000001}]},{"osd":3,"last update":"Fri Mar 3 08:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52300000000000002,"15min":0.54600000000000004},"min":{"1min":0.41699999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68799999999999994,"5min":0.71499999999999997,"15min":0.98499999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58699999999999997,"15min":0.63100000000000001},"min":{"1min":0.48299999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.69899999999999995,"5min":0.83499999999999996,"15min":1.2450000000000001},"last":0.59799999999999998}]},{"osd":4,"last update":"Fri Mar 3 08:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.79200000000000004,"15min":0.83399999999999996},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.242,"5min":1.242,"15min":1.5189999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.78100000000000003,"15min":0.81499999999999995},"min":{"1min":0.52400000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.218,"5min":1.218,"15min":1.605},"last":0.72199999999999998}]},{"osd":5,"last update":"Fri Mar 3 08:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78700000000000003,"15min":0.81599999999999995},"min":{"1min":0.58199999999999996,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.80200000000000005,"15min":0.81799999999999995},"min":{"1min":0.64200000000000002,"5min":0.60299999999999998,"15min":0.56599999999999995},"max":{"1min":0.99399999999999999,"5min":1.2070000000000001,"15min":1.286},"last":0.65100000000000002}]},{"osd":6,"last update":"Fri Mar 3 08:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.80700000000000005,"15min":0.86899999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1599999999999999,"5min":1.2470000000000001,"15min":1.623},"last":0.63},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80100000000000005,"15min":0.85399999999999998},"min":{"1min":0.60099999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.1240000000000001,"5min":1.137,"15min":1.5780000000000001},"last":0.77500000000000002}]},{"osd":7,"last update":"Fri Mar 3 08:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.80500000000000005,"15min":0.86899999999999999},"min":{"1min":0.59299999999999997,"5min":0.57599999999999996,"15min":0.57099999999999995},"max":{"1min":1.052,"5min":1.1020000000000001,"15min":1.6479999999999999},"last":0.752},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80800000000000005,"15min":0.85199999999999998},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.0840000000000001,"5min":1.1719999999999999,"15min":1.5509999999999999},"last":0.75700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574935,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10504,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":9408,"kb_avail":94361336,"statfs":{"total":96636764160,"available":96626008064,"internally_reserved":0,"allocated":1114112,"data_stored":508342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9633792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.42799999999999999,"15min":0.79700000000000004},"min":{"1min":0.307,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76500000000000001,"5min":0.78900000000000003,"15min":5.7889999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.497,"15min":0.69499999999999995},"min":{"1min":0.38100000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":2.6309999999999998},"last":0.40699999999999997}]},{"osd":1,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.50800000000000001,"15min":0.71199999999999997},"min":{"1min":0.33700000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":2.5209999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57999999999999996,"15min":0.93799999999999994},"min":{"1min":0.45700000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":5.8470000000000004},"last":0.54200000000000004}]},{"osd":3,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55200000000000005,"15min":1.1100000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85999999999999999,"5min":1.012,"15min":12.961},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56200000000000006,"15min":0.95699999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":6.1449999999999996},"last":0.59699999999999998}]},{"osd":4,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81799999999999995,"15min":1.2},"min":{"1min":0.60299999999999998,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1240000000000001,"5min":1.468,"15min":6.484},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84599999999999997,"15min":1.1839999999999999},"min":{"1min":0.66500000000000004,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.1519999999999999,"5min":1.1759999999999999,"15min":6.5510000000000002},"last":0.75800000000000001}]},{"osd":5,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.76600000000000001,"15min":1.111},"min":{"1min":0.59699999999999998,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.167,"5min":1.304,"15min":5.7119999999999997},"last":0.626},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.80200000000000005,"15min":1.163},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0629999999999999,"5min":1.1499999999999999,"15min":6.6369999999999996},"last":0.68899999999999995}]},{"osd":6,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83999999999999997,"15min":1.1719999999999999},"min":{"1min":0.61899999999999999,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.1739999999999999,"5min":1.393,"15min":6.625},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.84199999999999997,"15min":1.196},"min":{"1min":0.71199999999999997,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.137,"5min":1.347,"15min":5.7539999999999996},"last":0.72499999999999998}]},{"osd":7,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.78400000000000003,"15min":1.1539999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":6.7009999999999996},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.81299999999999994,"15min":1.145},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.107,"5min":1.107,"15min":6.6840000000000002},"last":0.70899999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574933,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14988,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":13888,"kb_avail":94356852,"statfs":{"total":96636764160,"available":96621416448,"internally_reserved":0,"allocated":1118208,"data_stored":509361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14221312},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42999999999999999,"15min":0.76800000000000002},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.56699999999999995,"5min":0.748,"15min":9.5820000000000007},"last":4.7050000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.50700000000000001,"15min":0.84099999999999997},"min":{"1min":0.377,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73999999999999999,"5min":0.83599999999999997,"15min":9.7840000000000007},"last":4.2910000000000004}]},{"osd":1,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48699999999999999,"15min":0.51800000000000002},"min":{"1min":0.375,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":0.66000000000000003,"5min":0.83799999999999997,"15min":1},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.53800000000000003,"15min":0.91700000000000004},"min":{"1min":0.36699999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73699999999999999,"5min":0.77400000000000002,"15min":9.702},"last":4.9199999999999999}]},{"osd":2,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.57699999999999996,"15min":0.88},"min":{"1min":0.46999999999999997,"5min":0.38,"15min":0.36499999999999999},"max":{"1min":1.026,"5min":1.026,"15min":8.3190000000000008},"last":3.5489999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.95399999999999996},"min":{"1min":0.49399999999999999,"5min":0.42099999999999999,"15min":0.41899999999999998},"max":{"1min":0.68600000000000005,"5min":0.93400000000000005,"15min":9.734},"last":0.98099999999999998}]},{"osd":4,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.755,"15min":1.097},"min":{"1min":0.61499999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.91900000000000004,"5min":1.1819999999999999,"15min":9.8629999999999995},"last":4.2519999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.80400000000000005,"15min":0.83599999999999997},"min":{"1min":0.58599999999999997,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.115,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":3.5110000000000001}]},{"osd":5,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80100000000000005,"15min":0.82599999999999996},"min":{"1min":0.66300000000000003,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.095,"5min":1.1899999999999999,"15min":1.3779999999999999},"last":3.5880000000000001},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.77700000000000002,"15min":1.1479999999999999},"min":{"1min":0.65200000000000002,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.96299999999999997,"5min":1.0860000000000001,"15min":9.6600000000000001},"last":1.9530000000000001}]},{"osd":6,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78600000000000003,"15min":1.161},"min":{"1min":0.65000000000000002,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.048,"5min":1.0600000000000001,"15min":9.8190000000000008},"last":4.415},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.79200000000000004,"15min":0.85499999999999998},"min":{"1min":0.65600000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0629999999999999,"5min":1.2609999999999999,"15min":1.609},"last":1.2609999999999999}]},{"osd":7,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":1.196},"min":{"1min":0.63300000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.0349999999999999,"5min":1.204,"15min":9.625},"last":1.018},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.81999999999999995,"15min":1.1719999999999999},"min":{"1min":0.67900000000000005,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.121,"5min":1.121,"15min":9.8949999999999996},"last":4.3259999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574933,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11156,"kb_used_data":1100,"kb_used_omap":0,"kb_used_meta":10048,"kb_avail":94360684,"statfs":{"total":96636764160,"available":96625340416,"internally_reserved":0,"allocated":1126400,"data_stored":509521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10289152},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.71899999999999997,"15min":0.93899999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.35599999999999998},"max":{"1min":0.874,"5min":1.054,"15min":7.1799999999999997},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70999999999999996,"15min":0.67400000000000004},"min":{"1min":0.55000000000000004,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.63600000000000001}]},{"osd":1,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.73699999999999999,"15min":1.0449999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.94899999999999995,"5min":1.153,"15min":9.5709999999999997},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.747,"15min":0.89600000000000002},"min":{"1min":0.53000000000000003,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.94599999999999995,"5min":1.153,"15min":5.266},"last":0.73299999999999998}]},{"osd":2,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.751,"15min":1.022},"min":{"1min":0.52900000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90900000000000003,"5min":1.034,"15min":8.4529999999999994},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.76100000000000001,"15min":0.96299999999999997},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.004,"5min":1.0880000000000001,"15min":6.6829999999999998},"last":0.66200000000000003}]},{"osd":3,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.55100000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.88,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.59099999999999997,"5min":0.58199999999999996,"15min":0.54300000000000004},"max":{"1min":0.98799999999999999,"5min":1.069,"15min":1.389},"last":0.75900000000000001}]},{"osd":5,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69699999999999995,"15min":0.73499999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84299999999999997,"5min":0.98799999999999999,"15min":1.0349999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.47999999999999998,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97799999999999998,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.54000000000000004}]},{"osd":6,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.76700000000000002,"15min":0.77900000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97399999999999998,"5min":1.032,"15min":1.032},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.79900000000000004,"15min":0.81599999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0489999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.83199999999999996}]},{"osd":7,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.77800000000000002,"15min":0.81599999999999995},"min":{"1min":0.57299999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.97299999999999998,"5min":1.119,"15min":1.679},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.84499999999999997,"15min":0.91600000000000004},"min":{"1min":0.60699999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.8340000000000001},"last":0.81299999999999994}]}]},{"osd":5,"up_from":13,"seq":55834574934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10504,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":9408,"kb_avail":94361336,"statfs":{"total":96636764160,"available":96626008064,"internally_reserved":0,"allocated":1114112,"data_stored":508342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9633792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66500000000000004,"15min":0.64400000000000002},"min":{"1min":0.432,"5min":0.318,"15min":0.318},"max":{"1min":0.90800000000000003,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.754,"15min":0.74299999999999999},"min":{"1min":0.52200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.96699999999999997,"5min":2.528,"15min":2.528},"last":0.59699999999999998}]},{"osd":1,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.72299999999999998,"15min":0.74099999999999999},"min":{"1min":0.45900000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.879,"5min":1.038,"15min":1.038},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.69699999999999995,"15min":0.64500000000000002},"min":{"1min":0.44800000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.86699999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63}]},{"osd":2,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.68000000000000005,"15min":0.67900000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82799999999999996,"5min":0.998,"15min":0.998},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.71399999999999997,"15min":0.71199999999999997},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.92400000000000004,"5min":1.024,"15min":1.321},"last":0.86599999999999999}]},{"osd":3,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.72199999999999998,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91800000000000004,"5min":1.024,"15min":1.024},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.75,"15min":0.73699999999999999},"min":{"1min":0.55700000000000005,"5min":0.501,"15min":0.501},"max":{"1min":0.995,"5min":1.012,"15min":1.012},"last":0.79500000000000004}]},{"osd":4,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.67900000000000005,"15min":0.66700000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.79000000000000004,"15min":0.78200000000000003},"min":{"1min":0.56399999999999995,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0109999999999999,"5min":1.079,"15min":1.079},"last":0.84299999999999997}]},{"osd":6,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.72099999999999997,"15min":0.78400000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.88300000000000001,"5min":1.012,"15min":1.071},"last":0.745},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71499999999999997,"15min":0.72199999999999998},"min":{"1min":0.48599999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89700000000000002,"5min":0.97099999999999997,"15min":1.0429999999999999},"last":0.67300000000000004}]},{"osd":7,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.748,"15min":0.754},"min":{"1min":0.52000000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.91000000000000003,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78600000000000003,"15min":0.82299999999999995},"min":{"1min":0.60099999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.94699999999999995,"5min":0.995,"15min":1.1080000000000001},"last":0.91500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:02:27.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:02:27.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.702+0000 7f94edbff700 1 Processor -- start 2023-03-03T08:02:27.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.703+0000 7f94edbff700 1 -- start start 2023-03-03T08:02:27.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.704+0000 7f94edbff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f94e80b8b90 0x7f94e80cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:27.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.704+0000 7f94edbff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f94e80b5410 0x7f94e80d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:27.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.704+0000 7f94edbff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e80d0ef0 con 0x7f94e80b87c0 2023-03-03T08:02:27.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.704+0000 7f94edbff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e80d0600 con 0x7f94e80b8b90 2023-03-03T08:02:27.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.704+0000 7f94edbff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e80cd970 con 0x7f94e80b5410 2023-03-03T08:02:27.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.704+0000 7f94e6ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f94e80b8b90 0x7f94e80cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:27.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.704+0000 7f94e6ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f94e80b8b90 0x7f94e80cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43664/0 (socket says 172.21.15.98:43664) 2023-03-03T08:02:27.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.705+0000 7f94e6ffd700 1 -- 172.21.15.98:0/477736996 learned_addr learned my addr 172.21.15.98:0/477736996 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:27.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.705+0000 7f94e7fff700 1 --2- 172.21.15.98:0/477736996 >> v2:172.21.15.100:3300/0 conn(0x7f94e80b5410 0x7f94e80d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:27.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.705+0000 7f94e6ffd700 1 -- 172.21.15.98:0/477736996 >> v2:172.21.15.100:3300/0 conn(0x7f94e80b5410 msgr2=0x7f94e80d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:27.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.705+0000 7f94e6ffd700 1 --2- 172.21.15.98:0/477736996 >> v2:172.21.15.100:3300/0 conn(0x7f94e80b5410 0x7f94e80d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.705+0000 7f94e6ffd700 1 -- 172.21.15.98:0/477736996 >> v1:172.21.15.100:6789/0 conn(0x7f94e80b87c0 legacy=0x7f94e80b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:27.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.705+0000 7f94e6ffd700 1 -- 172.21.15.98:0/477736996 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e80cdc50 con 0x7f94e80b8b90 2023-03-03T08:02:27.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.705+0000 7f94e6ffd700 1 --2- 172.21.15.98:0/477736996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e80b8b90 0x7f94e80cffe0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f94e80c6d50 tx=0x7f94dc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c083092db396459c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:27.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.706+0000 7f94e67fc700 1 -- 172.21.15.98:0/477736996 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc011830 con 0x7f94e80b8b90 2023-03-03T08:02:27.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.706+0000 7f94e67fc700 1 -- 172.21.15.98:0/477736996 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94dc011990 con 0x7f94e80b8b90 2023-03-03T08:02:27.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.706+0000 7f94e67fc700 1 -- 172.21.15.98:0/477736996 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc011d20 con 0x7f94e80b8b90 2023-03-03T08:02:27.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.706+0000 7f94edbff700 1 -- 172.21.15.98:0/477736996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e80b8b90 msgr2=0x7f94e80cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:27.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.706+0000 7f94edbff700 1 --2- 172.21.15.98:0/477736996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e80b8b90 0x7f94e80cffe0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f94e80c6d50 tx=0x7f94dc009bd0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.706+0000 7f94edbff700 1 -- 172.21.15.98:0/477736996 shutdown_connections 2023-03-03T08:02:27.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.706+0000 7f94edbff700 1 --2- 172.21.15.98:0/477736996 >> v2:172.21.15.100:3300/0 conn(0x7f94e80b5410 0x7f94e80d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.706+0000 7f94edbff700 1 --2- 172.21.15.98:0/477736996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e80b8b90 0x7f94e80cffe0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.706+0000 7f94edbff700 1 -- 172.21.15.98:0/477736996 >> 172.21.15.98:0/477736996 conn(0x7f94e806aee0 msgr2=0x7f94e806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:27.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.707+0000 7f94edbff700 1 -- 172.21.15.98:0/477736996 shutdown_connections 2023-03-03T08:02:27.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.707+0000 7f94edbff700 1 -- 172.21.15.98:0/477736996 wait complete. 2023-03-03T08:02:27.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.709+0000 7f94edbff700 1 Processor -- start 2023-03-03T08:02:27.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.709+0000 7f94edbff700 1 -- start start 2023-03-03T08:02:27.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.710+0000 7f94edbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94e80b5410 0x7f94e815e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:27.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.710+0000 7f94edbff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94e80b87c0 0x7f94e815dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:27.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.710+0000 7f94edbff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e80b8b90 0x7f94e81e5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:27.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.710+0000 7f94edbff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e81e5bd0 con 0x7f94e80b8b90 2023-03-03T08:02:27.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.710+0000 7f94edbff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e81e5e50 con 0x7f94e80b87c0 2023-03-03T08:02:27.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.710+0000 7f94edbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e81e6130 con 0x7f94e80b5410 2023-03-03T08:02:27.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.710+0000 7f94e7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e80b8b90 0x7f94e81e5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:27.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.710+0000 7f94e7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e80b8b90 0x7f94e81e5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43676/0 (socket says 172.21.15.98:43676) 2023-03-03T08:02:27.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.710+0000 7f94e7fff700 1 -- 172.21.15.98:0/592397667 learned_addr learned my addr 172.21.15.98:0/592397667 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:27.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.710+0000 7f94e6ffd700 1 --2- 172.21.15.98:0/592397667 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94e80b87c0 0x7f94e815dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:27.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.710+0000 7f94e77fe700 1 --2- 172.21.15.98:0/592397667 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94e80b5410 0x7f94e815e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:27.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.711+0000 7f94e7fff700 1 -- 172.21.15.98:0/592397667 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94e80b5410 msgr2=0x7f94e815e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:27.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.711+0000 7f94e7fff700 1 --2- 172.21.15.98:0/592397667 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94e80b5410 0x7f94e815e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.711+0000 7f94e7fff700 1 -- 172.21.15.98:0/592397667 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94e80b87c0 msgr2=0x7f94e815dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:27.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.711+0000 7f94e7fff700 1 --2- 172.21.15.98:0/592397667 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94e80b87c0 0x7f94e815dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.711+0000 7f94e7fff700 1 -- 172.21.15.98:0/592397667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94dc009550 con 0x7f94e80b8b90 2023-03-03T08:02:27.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.711+0000 7f94e77fe700 1 --2- 172.21.15.98:0/592397667 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94e80b5410 0x7f94e815e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:27.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.711+0000 7f94e7fff700 1 --2- 172.21.15.98:0/592397667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e80b8b90 0x7f94e81e5690 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f94e000efc0 tx=0x7f94e000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:27.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.711+0000 7f94cffff700 1 -- 172.21.15.98:0/592397667 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e000cd40 con 0x7f94e80b8b90 2023-03-03T08:02:27.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.711+0000 7f94cffff700 1 -- 172.21.15.98:0/592397667 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e000f040 con 0x7f94e80b8b90 2023-03-03T08:02:27.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.711+0000 7f94edbff700 1 -- 172.21.15.98:0/592397667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e80cdf30 con 0x7f94e80b8b90 2023-03-03T08:02:27.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.711+0000 7f94cffff700 1 -- 172.21.15.98:0/592397667 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e00146c0 con 0x7f94e80b8b90 2023-03-03T08:02:27.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.711+0000 7f94edbff700 1 -- 172.21.15.98:0/592397667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e8073a80 con 0x7f94e80b8b90 2023-03-03T08:02:27.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.713+0000 7f94edbff700 1 -- 172.21.15.98:0/592397667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e80ce0c0 con 0x7f94e80b8b90 2023-03-03T08:02:27.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.714+0000 7f94cffff700 1 -- 172.21.15.98:0/592397667 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f94e0014820 con 0x7f94e80b8b90 2023-03-03T08:02:27.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.715+0000 7f94cffff700 1 --2- 172.21.15.98:0/592397667 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f94d4068380 0x7f94d406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:27.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.715+0000 7f94cffff700 1 -- 172.21.15.98:0/592397667 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f94e0086740 con 0x7f94e80b8b90 2023-03-03T08:02:27.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.716+0000 7f94e77fe700 1 --2- 172.21.15.98:0/592397667 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f94d4068380 0x7f94d406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:27.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.716+0000 7f94e77fe700 1 --2- 172.21.15.98:0/592397667 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f94d4068380 0x7f94d406a830 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f94e80c6900 tx=0x7f94d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:27.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.719+0000 7f94cffff700 1 -- 172.21.15.98:0/592397667 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f94e0052fb0 con 0x7f94e80b8b90 2023-03-03T08:02:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.909+0000 7f94edbff700 1 -- 172.21.15.98:0/592397667 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f94e80729b0 con 0x7f94d4068380 2023-03-03T08:02:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.919+0000 7f94cffff700 1 -- 172.21.15.98:0/592397667 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311138 (secure 0 0 0) 0x7f94e80729b0 con 0x7f94d4068380 2023-03-03T08:02:27.924 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.924+0000 7f94edbff700 1 -- 172.21.15.98:0/592397667 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f94d4068380 msgr2=0x7f94d406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.924+0000 7f94edbff700 1 --2- 172.21.15.98:0/592397667 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f94d4068380 0x7f94d406a830 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f94e80c6900 tx=0x7f94d8009040 comp rx=0 tx=0).stop 2023-03-03T08:02:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.924+0000 7f94edbff700 1 -- 172.21.15.98:0/592397667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e80b8b90 msgr2=0x7f94e81e5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.924+0000 7f94edbff700 1 --2- 172.21.15.98:0/592397667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e80b8b90 0x7f94e81e5690 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f94e000efc0 tx=0x7f94e000c3a0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.924+0000 7f94edbff700 1 -- 172.21.15.98:0/592397667 shutdown_connections 2023-03-03T08:02:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.924+0000 7f94edbff700 1 --2- 172.21.15.98:0/592397667 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f94d4068380 0x7f94d406a830 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.924+0000 7f94edbff700 1 --2- 172.21.15.98:0/592397667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e80b8b90 0x7f94e81e5690 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.924+0000 7f94edbff700 1 --2- 172.21.15.98:0/592397667 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94e80b5410 0x7f94e815e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.924+0000 7f94edbff700 1 --2- 172.21.15.98:0/592397667 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94e80b87c0 0x7f94e815dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.924+0000 7f94edbff700 1 -- 172.21.15.98:0/592397667 >> 172.21.15.98:0/592397667 conn(0x7f94e806aee0 msgr2=0x7f94e80b2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.924+0000 7f94edbff700 1 -- 172.21.15.98:0/592397667 shutdown_connections 2023-03-03T08:02:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:27.924+0000 7f94edbff700 1 -- 172.21.15.98:0/592397667 wait complete. 2023-03-03T08:02:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T08:02:27.953 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2023-03-03T08:02:26.164808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":223,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166,"ondisk_log_size":166,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":200,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":92248,"kb_used_data":9944,"kb_used_omap":0,"kb_used_meta":82240,"kb_avail":754882472,"statfs":{"total":773094113280,"available":772999651328,"internally_reserved":0,"allocated":10182656,"data_stored":5256208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":84213760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.310187"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595186+0000","last_change":"2023-03-03T08:02:22.430702+0000","last_active":"2023-03-03T08:02:23.595186+0000","last_peered":"2023-03-03T08:02:23.595186+0000","last_clean":"2023-03-03T08:02:23.595186+0000","last_became_active":"2023-03-03T08:02:22.430443+0000","last_became_peered":"2023-03-03T08:02:22.430443+0000","last_unstale":"2023-03-03T08:02:23.595186+0000","last_undegraded":"2023-03-03T08:02:23.595186+0000","last_fullsized":"2023-03-03T08:02:23.595186+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:00:48.464577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594708+0000","last_change":"2023-03-03T08:02:22.435774+0000","last_active":"2023-03-03T08:02:23.594708+0000","last_peered":"2023-03-03T08:02:23.594708+0000","last_clean":"2023-03-03T08:02:23.594708+0000","last_became_active":"2023-03-03T08:02:22.435450+0000","last_became_peered":"2023-03-03T08:02:22.435450+0000","last_unstale":"2023-03-03T08:02:23.594708+0000","last_undegraded":"2023-03-03T08:02:23.594708+0000","last_fullsized":"2023-03-03T08:02:23.594708+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:01:28.330059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594597+0000","last_change":"2023-03-03T08:02:22.422601+0000","last_active":"2023-03-03T08:02:23.594597+0000","last_peered":"2023-03-03T08:02:23.594597+0000","last_clean":"2023-03-03T08:02:23.594597+0000","last_became_active":"2023-03-03T08:02:22.422336+0000","last_became_peered":"2023-03-03T08:02:22.422336+0000","last_unstale":"2023-03-03T08:02:23.594597+0000","last_undegraded":"2023-03-03T08:02:23.594597+0000","last_fullsized":"2023-03-03T08:02:23.594597+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:56:41.737023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594520+0000","last_change":"2023-03-03T08:02:22.431276+0000","last_active":"2023-03-03T08:02:23.594520+0000","last_peered":"2023-03-03T08:02:23.594520+0000","last_clean":"2023-03-03T08:02:23.594520+0000","last_became_active":"2023-03-03T08:02:22.431011+0000","last_became_peered":"2023-03-03T08:02:22.431011+0000","last_unstale":"2023-03-03T08:02:23.594520+0000","last_undegraded":"2023-03-03T08:02:23.594520+0000","last_fullsized":"2023-03-03T08:02:23.594520+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:05:38.030389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.412835+0000","last_change":"2023-03-03T08:02:22.430589+0000","last_active":"2023-03-03T08:02:23.412835+0000","last_peered":"2023-03-03T08:02:23.412835+0000","last_clean":"2023-03-03T08:02:23.412835+0000","last_became_active":"2023-03-03T08:02:22.430316+0000","last_became_peered":"2023-03-03T08:02:22.430316+0000","last_unstale":"2023-03-03T08:02:23.412835+0000","last_undegraded":"2023-03-03T08:02:23.412835+0000","last_fullsized":"2023-03-03T08:02:23.412835+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:13:56.368129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594492+0000","last_change":"2023-03-03T08:02:22.433617+0000","last_active":"2023-03-03T08:02:23.594492+0000","last_peered":"2023-03-03T08:02:23.594492+0000","last_clean":"2023-03-03T08:02:23.594492+0000","last_became_active":"2023-03-03T08:02:22.431639+0000","last_became_peered":"2023-03-03T08:02:22.431639+0000","last_unstale":"2023-03-03T08:02:23.594492+0000","last_undegraded":"2023-03-03T08:02:23.594492+0000","last_fullsized":"2023-03-03T08:02:23.594492+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:31:12.347482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.411573+0000","last_change":"2023-03-03T08:02:22.433748+0000","last_active":"2023-03-03T08:02:23.411573+0000","last_peered":"2023-03-03T08:02:23.411573+0000","last_clean":"2023-03-03T08:02:23.411573+0000","last_became_active":"2023-03-03T08:02:22.433402+0000","last_became_peered":"2023-03-03T08:02:22.433402+0000","last_unstale":"2023-03-03T08:02:23.411573+0000","last_undegraded":"2023-03-03T08:02:23.411573+0000","last_fullsized":"2023-03-03T08:02:23.411573+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:00:48.464577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.596025+0000","last_change":"2023-03-03T08:02:22.427347+0000","last_active":"2023-03-03T08:02:23.596025+0000","last_peered":"2023-03-03T08:02:23.596025+0000","last_clean":"2023-03-03T08:02:23.596025+0000","last_became_active":"2023-03-03T08:02:22.426883+0000","last_became_peered":"2023-03-03T08:02:22.426883+0000","last_unstale":"2023-03-03T08:02:23.596025+0000","last_undegraded":"2023-03-03T08:02:23.596025+0000","last_fullsized":"2023-03-03T08:02:23.596025+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:12.558600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.413147+0000","last_change":"2023-03-03T08:02:22.424179+0000","last_active":"2023-03-03T08:02:23.413147+0000","last_peered":"2023-03-03T08:02:23.413147+0000","last_clean":"2023-03-03T08:02:23.413147+0000","last_became_active":"2023-03-03T08:02:22.423895+0000","last_became_peered":"2023-03-03T08:02:22.423895+0000","last_unstale":"2023-03-03T08:02:23.413147+0000","last_undegraded":"2023-03-03T08:02:23.413147+0000","last_fullsized":"2023-03-03T08:02:23.413147+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:20:04.574289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595081+0000","last_change":"2023-03-03T08:02:22.441027+0000","last_active":"2023-03-03T08:02:23.595081+0000","last_peered":"2023-03-03T08:02:23.595081+0000","last_clean":"2023-03-03T08:02:23.595081+0000","last_became_active":"2023-03-03T08:02:22.440624+0000","last_became_peered":"2023-03-03T08:02:22.440624+0000","last_unstale":"2023-03-03T08:02:23.595081+0000","last_undegraded":"2023-03-03T08:02:23.595081+0000","last_fullsized":"2023-03-03T08:02:23.595081+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:28:25.191765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.413287+0000","last_change":"2023-03-03T08:02:22.437088+0000","last_active":"2023-03-03T08:02:23.413287+0000","last_peered":"2023-03-03T08:02:23.413287+0000","last_clean":"2023-03-03T08:02:23.413287+0000","last_became_active":"2023-03-03T08:02:22.436567+0000","last_became_peered":"2023-03-03T08:02:22.436567+0000","last_unstale":"2023-03-03T08:02:23.413287+0000","last_undegraded":"2023-03-03T08:02:23.413287+0000","last_fullsized":"2023-03-03T08:02:23.413287+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:46:16.129844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594856+0000","last_change":"2023-03-03T08:02:22.431484+0000","last_active":"2023-03-03T08:02:23.594856+0000","last_peered":"2023-03-03T08:02:23.594856+0000","last_clean":"2023-03-03T08:02:23.594856+0000","last_became_active":"2023-03-03T08:02:22.431239+0000","last_became_peered":"2023-03-03T08:02:22.431239+0000","last_unstale":"2023-03-03T08:02:23.594856+0000","last_undegraded":"2023-03-03T08:02:23.594856+0000","last_fullsized":"2023-03-03T08:02:23.594856+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:21.374521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594202+0000","last_change":"2023-03-03T08:02:22.426282+0000","last_active":"2023-03-03T08:02:23.594202+0000","last_peered":"2023-03-03T08:02:23.594202+0000","last_clean":"2023-03-03T08:02:23.594202+0000","last_became_active":"2023-03-03T08:02:22.425689+0000","last_became_peered":"2023-03-03T08:02:22.425689+0000","last_unstale":"2023-03-03T08:02:23.594202+0000","last_undegraded":"2023-03-03T08:02:23.594202+0000","last_fullsized":"2023-03-03T08:02:23.594202+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:07.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.412890+0000","last_change":"2023-03-03T08:02:22.434995+0000","last_active":"2023-03-03T08:02:23.412890+0000","last_peered":"2023-03-03T08:02:23.412890+0000","last_clean":"2023-03-03T08:02:23.412890+0000","last_became_active":"2023-03-03T08:02:22.434627+0000","last_became_peered":"2023-03-03T08:02:22.434627+0000","last_unstale":"2023-03-03T08:02:23.412890+0000","last_undegraded":"2023-03-03T08:02:23.412890+0000","last_fullsized":"2023-03-03T08:02:23.412890+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:12:17.218987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.596250+0000","last_change":"2023-03-03T08:02:22.437244+0000","last_active":"2023-03-03T08:02:23.596250+0000","last_peered":"2023-03-03T08:02:23.596250+0000","last_clean":"2023-03-03T08:02:23.596250+0000","last_became_active":"2023-03-03T08:02:22.433987+0000","last_became_peered":"2023-03-03T08:02:22.433987+0000","last_unstale":"2023-03-03T08:02:23.596250+0000","last_undegraded":"2023-03-03T08:02:23.596250+0000","last_fullsized":"2023-03-03T08:02:23.596250+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:01:28.330059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595804+0000","last_change":"2023-03-03T08:02:22.426791+0000","last_active":"2023-03-03T08:02:23.595804+0000","last_peered":"2023-03-03T08:02:23.595804+0000","last_clean":"2023-03-03T08:02:23.595804+0000","last_became_active":"2023-03-03T08:02:22.426461+0000","last_became_peered":"2023-03-03T08:02:22.426461+0000","last_unstale":"2023-03-03T08:02:23.595804+0000","last_undegraded":"2023-03-03T08:02:23.595804+0000","last_fullsized":"2023-03-03T08:02:23.595804+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:54:03.035322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.596298+0000","last_change":"2023-03-03T08:02:22.435907+0000","last_active":"2023-03-03T08:02:23.596298+0000","last_peered":"2023-03-03T08:02:23.596298+0000","last_clean":"2023-03-03T08:02:23.596298+0000","last_became_active":"2023-03-03T08:02:22.432401+0000","last_became_peered":"2023-03-03T08:02:22.432401+0000","last_unstale":"2023-03-03T08:02:23.596298+0000","last_undegraded":"2023-03-03T08:02:23.596298+0000","last_fullsized":"2023-03-03T08:02:23.596298+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:04:41.141311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594707+0000","last_change":"2023-03-03T08:02:22.420788+0000","last_active":"2023-03-03T08:02:23.594707+0000","last_peered":"2023-03-03T08:02:23.594707+0000","last_clean":"2023-03-03T08:02:23.594707+0000","last_became_active":"2023-03-03T08:02:22.420496+0000","last_became_peered":"2023-03-03T08:02:22.420496+0000","last_unstale":"2023-03-03T08:02:23.594707+0000","last_undegraded":"2023-03-03T08:02:23.594707+0000","last_fullsized":"2023-03-03T08:02:23.594707+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:45.588606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.413111+0000","last_change":"2023-03-03T08:02:22.429036+0000","last_active":"2023-03-03T08:02:23.413111+0000","last_peered":"2023-03-03T08:02:23.413111+0000","last_clean":"2023-03-03T08:02:23.413111+0000","last_became_active":"2023-03-03T08:02:22.426987+0000","last_became_peered":"2023-03-03T08:02:22.426987+0000","last_unstale":"2023-03-03T08:02:23.413111+0000","last_undegraded":"2023-03-03T08:02:23.413111+0000","last_fullsized":"2023-03-03T08:02:23.413111+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:26:55.840281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594788+0000","last_change":"2023-03-03T08:02:22.433917+0000","last_active":"2023-03-03T08:02:23.594788+0000","last_peered":"2023-03-03T08:02:23.594788+0000","last_clean":"2023-03-03T08:02:23.594788+0000","last_became_active":"2023-03-03T08:02:22.433570+0000","last_became_peered":"2023-03-03T08:02:22.433570+0000","last_unstale":"2023-03-03T08:02:23.594788+0000","last_undegraded":"2023-03-03T08:02:23.594788+0000","last_fullsized":"2023-03-03T08:02:23.594788+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:05:19.334210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.412953+0000","last_change":"2023-03-03T08:02:22.425945+0000","last_active":"2023-03-03T08:02:23.412953+0000","last_peered":"2023-03-03T08:02:23.412953+0000","last_clean":"2023-03-03T08:02:23.412953+0000","last_became_active":"2023-03-03T08:02:22.421663+0000","last_became_peered":"2023-03-03T08:02:22.421663+0000","last_unstale":"2023-03-03T08:02:23.412953+0000","last_undegraded":"2023-03-03T08:02:23.412953+0000","last_fullsized":"2023-03-03T08:02:23.412953+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:07:17.564359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.412847+0000","last_change":"2023-03-03T08:02:22.434911+0000","last_active":"2023-03-03T08:02:23.412847+0000","last_peered":"2023-03-03T08:02:23.412847+0000","last_clean":"2023-03-03T08:02:23.412847+0000","last_became_active":"2023-03-03T08:02:22.434589+0000","last_became_peered":"2023-03-03T08:02:22.434589+0000","last_unstale":"2023-03-03T08:02:23.412847+0000","last_undegraded":"2023-03-03T08:02:23.412847+0000","last_fullsized":"2023-03-03T08:02:23.412847+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:34.930052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594007+0000","last_change":"2023-03-03T08:02:22.437187+0000","last_active":"2023-03-03T08:02:23.594007+0000","last_peered":"2023-03-03T08:02:23.594007+0000","last_clean":"2023-03-03T08:02:23.594007+0000","last_became_active":"2023-03-03T08:02:22.436843+0000","last_became_peered":"2023-03-03T08:02:22.436843+0000","last_unstale":"2023-03-03T08:02:23.594007+0000","last_undegraded":"2023-03-03T08:02:23.594007+0000","last_fullsized":"2023-03-03T08:02:23.594007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:00.941358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594266+0000","last_change":"2023-03-03T08:02:22.422598+0000","last_active":"2023-03-03T08:02:23.594266+0000","last_peered":"2023-03-03T08:02:23.594266+0000","last_clean":"2023-03-03T08:02:23.594266+0000","last_became_active":"2023-03-03T08:02:22.422326+0000","last_became_peered":"2023-03-03T08:02:22.422326+0000","last_unstale":"2023-03-03T08:02:23.594266+0000","last_undegraded":"2023-03-03T08:02:23.594266+0000","last_fullsized":"2023-03-03T08:02:23.594266+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:45:54.383085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594500+0000","last_change":"2023-03-03T08:02:22.435279+0000","last_active":"2023-03-03T08:02:23.594500+0000","last_peered":"2023-03-03T08:02:23.594500+0000","last_clean":"2023-03-03T08:02:23.594500+0000","last_became_active":"2023-03-03T08:02:22.434992+0000","last_became_peered":"2023-03-03T08:02:22.434992+0000","last_unstale":"2023-03-03T08:02:23.594500+0000","last_undegraded":"2023-03-03T08:02:23.594500+0000","last_fullsized":"2023-03-03T08:02:23.594500+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:52:46.445281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.413120+0000","last_change":"2023-03-03T08:02:22.431162+0000","last_active":"2023-03-03T08:02:23.413120+0000","last_peered":"2023-03-03T08:02:23.413120+0000","last_clean":"2023-03-03T08:02:23.413120+0000","last_became_active":"2023-03-03T08:02:22.430856+0000","last_became_peered":"2023-03-03T08:02:22.430856+0000","last_unstale":"2023-03-03T08:02:23.413120+0000","last_undegraded":"2023-03-03T08:02:23.413120+0000","last_fullsized":"2023-03-03T08:02:23.413120+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:47:11.815286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595766+0000","last_change":"2023-03-03T08:02:22.439980+0000","last_active":"2023-03-03T08:02:23.595766+0000","last_peered":"2023-03-03T08:02:23.595766+0000","last_clean":"2023-03-03T08:02:23.595766+0000","last_became_active":"2023-03-03T08:02:22.439690+0000","last_became_peered":"2023-03-03T08:02:22.439690+0000","last_unstale":"2023-03-03T08:02:23.595766+0000","last_undegraded":"2023-03-03T08:02:23.595766+0000","last_fullsized":"2023-03-03T08:02:23.595766+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:21:01.687797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594608+0000","last_change":"2023-03-03T08:02:22.427427+0000","last_active":"2023-03-03T08:02:23.594608+0000","last_peered":"2023-03-03T08:02:23.594608+0000","last_clean":"2023-03-03T08:02:23.594608+0000","last_became_active":"2023-03-03T08:02:22.427093+0000","last_became_peered":"2023-03-03T08:02:22.427093+0000","last_unstale":"2023-03-03T08:02:23.594608+0000","last_undegraded":"2023-03-03T08:02:23.594608+0000","last_fullsized":"2023-03-03T08:02:23.594608+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:39.345227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.411686+0000","last_change":"2023-03-03T08:02:22.431307+0000","last_active":"2023-03-03T08:02:23.411686+0000","last_peered":"2023-03-03T08:02:23.411686+0000","last_clean":"2023-03-03T08:02:23.411686+0000","last_became_active":"2023-03-03T08:02:22.430953+0000","last_became_peered":"2023-03-03T08:02:22.430953+0000","last_unstale":"2023-03-03T08:02:23.411686+0000","last_undegraded":"2023-03-03T08:02:23.411686+0000","last_fullsized":"2023-03-03T08:02:23.411686+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:23:35.028360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.413007+0000","last_change":"2023-03-03T08:02:22.422990+0000","last_active":"2023-03-03T08:02:23.413007+0000","last_peered":"2023-03-03T08:02:23.413007+0000","last_clean":"2023-03-03T08:02:23.413007+0000","last_became_active":"2023-03-03T08:02:22.422630+0000","last_became_peered":"2023-03-03T08:02:22.422630+0000","last_unstale":"2023-03-03T08:02:23.413007+0000","last_undegraded":"2023-03-03T08:02:23.413007+0000","last_fullsized":"2023-03-03T08:02:23.413007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:30:04.750090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.593819+0000","last_change":"2023-03-03T08:02:22.432833+0000","last_active":"2023-03-03T08:02:23.593819+0000","last_peered":"2023-03-03T08:02:23.593819+0000","last_clean":"2023-03-03T08:02:23.593819+0000","last_became_active":"2023-03-03T08:02:22.432549+0000","last_became_peered":"2023-03-03T08:02:22.432549+0000","last_unstale":"2023-03-03T08:02:23.593819+0000","last_undegraded":"2023-03-03T08:02:23.593819+0000","last_fullsized":"2023-03-03T08:02:23.593819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:56:41.737023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595692+0000","last_change":"2023-03-03T08:02:22.432104+0000","last_active":"2023-03-03T08:02:23.595692+0000","last_peered":"2023-03-03T08:02:23.595692+0000","last_clean":"2023-03-03T08:02:23.595692+0000","last_became_active":"2023-03-03T08:02:22.431815+0000","last_became_peered":"2023-03-03T08:02:22.431815+0000","last_unstale":"2023-03-03T08:02:23.595692+0000","last_undegraded":"2023-03-03T08:02:23.595692+0000","last_fullsized":"2023-03-03T08:02:23.595692+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:29:38.960491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594961+0000","last_change":"2023-03-03T08:02:22.432477+0000","last_active":"2023-03-03T08:02:23.594961+0000","last_peered":"2023-03-03T08:02:23.594961+0000","last_clean":"2023-03-03T08:02:23.594961+0000","last_became_active":"2023-03-03T08:02:22.431964+0000","last_became_peered":"2023-03-03T08:02:22.431964+0000","last_unstale":"2023-03-03T08:02:23.594961+0000","last_undegraded":"2023-03-03T08:02:23.594961+0000","last_fullsized":"2023-03-03T08:02:23.594961+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:35:35.596312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"90'4","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.446762+0000","last_change":"2023-03-03T08:02:21.879191+0000","last_active":"2023-03-03T08:02:22.446762+0000","last_peered":"2023-03-03T08:02:22.446762+0000","last_clean":"2023-03-03T08:02:22.446762+0000","last_became_active":"2023-03-03T08:02:21.878925+0000","last_became_peered":"2023-03-03T08:02:21.878925+0000","last_unstale":"2023-03-03T08:02:22.446762+0000","last_undegraded":"2023-03-03T08:02:22.446762+0000","last_fullsized":"2023-03-03T08:02:22.446762+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:06:27.575316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.596405+0000","last_change":"2023-03-03T08:02:22.435681+0000","last_active":"2023-03-03T08:02:23.596405+0000","last_peered":"2023-03-03T08:02:23.596405+0000","last_clean":"2023-03-03T08:02:23.596405+0000","last_became_active":"2023-03-03T08:02:22.435258+0000","last_became_peered":"2023-03-03T08:02:22.435258+0000","last_unstale":"2023-03-03T08:02:23.596405+0000","last_undegraded":"2023-03-03T08:02:23.596405+0000","last_fullsized":"2023-03-03T08:02:23.596405+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:10:30.503825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"90'1","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.440602+0000","last_change":"2023-03-03T08:02:21.874289+0000","last_active":"2023-03-03T08:02:22.440602+0000","last_peered":"2023-03-03T08:02:22.440602+0000","last_clean":"2023-03-03T08:02:22.440602+0000","last_became_active":"2023-03-03T08:02:21.873933+0000","last_became_peered":"2023-03-03T08:02:21.873933+0000","last_unstale":"2023-03-03T08:02:22.440602+0000","last_undegraded":"2023-03-03T08:02:22.440602+0000","last_fullsized":"2023-03-03T08:02:22.440602+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:11.554773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.411603+0000","last_change":"2023-03-03T08:02:22.430970+0000","last_active":"2023-03-03T08:02:23.411603+0000","last_peered":"2023-03-03T08:02:23.411603+0000","last_clean":"2023-03-03T08:02:23.411603+0000","last_became_active":"2023-03-03T08:02:22.430659+0000","last_became_peered":"2023-03-03T08:02:22.430659+0000","last_unstale":"2023-03-03T08:02:23.411603+0000","last_undegraded":"2023-03-03T08:02:23.411603+0000","last_fullsized":"2023-03-03T08:02:23.411603+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:11.514077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"90'2","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.439771+0000","last_change":"2023-03-03T08:02:21.870287+0000","last_active":"2023-03-03T08:02:22.439771+0000","last_peered":"2023-03-03T08:02:22.439771+0000","last_clean":"2023-03-03T08:02:22.439771+0000","last_became_active":"2023-03-03T08:02:21.870017+0000","last_became_peered":"2023-03-03T08:02:21.870017+0000","last_unstale":"2023-03-03T08:02:22.439771+0000","last_undegraded":"2023-03-03T08:02:22.439771+0000","last_fullsized":"2023-03-03T08:02:22.439771+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:11.554773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.596327+0000","last_change":"2023-03-03T08:02:22.441110+0000","last_active":"2023-03-03T08:02:23.596327+0000","last_peered":"2023-03-03T08:02:23.596327+0000","last_clean":"2023-03-03T08:02:23.596327+0000","last_became_active":"2023-03-03T08:02:22.440860+0000","last_became_peered":"2023-03-03T08:02:22.440860+0000","last_unstale":"2023-03-03T08:02:23.596327+0000","last_undegraded":"2023-03-03T08:02:23.596327+0000","last_fullsized":"2023-03-03T08:02:23.596327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:57:04.878329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"90'2","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.439508+0000","last_change":"2023-03-03T08:02:21.878531+0000","last_active":"2023-03-03T08:02:22.439508+0000","last_peered":"2023-03-03T08:02:22.439508+0000","last_clean":"2023-03-03T08:02:22.439508+0000","last_became_active":"2023-03-03T08:02:21.878208+0000","last_became_peered":"2023-03-03T08:02:21.878208+0000","last_unstale":"2023-03-03T08:02:22.439508+0000","last_undegraded":"2023-03-03T08:02:22.439508+0000","last_fullsized":"2023-03-03T08:02:22.439508+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:12:42.377302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595948+0000","last_change":"2023-03-03T08:02:22.433265+0000","last_active":"2023-03-03T08:02:23.595948+0000","last_peered":"2023-03-03T08:02:23.595948+0000","last_clean":"2023-03-03T08:02:23.595948+0000","last_became_active":"2023-03-03T08:02:22.432894+0000","last_became_peered":"2023-03-03T08:02:22.432894+0000","last_unstale":"2023-03-03T08:02:23.595948+0000","last_undegraded":"2023-03-03T08:02:23.595948+0000","last_fullsized":"2023-03-03T08:02:23.595948+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:03.788343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"90'1","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.440387+0000","last_change":"2023-03-03T08:02:21.871979+0000","last_active":"2023-03-03T08:02:22.440387+0000","last_peered":"2023-03-03T08:02:22.440387+0000","last_clean":"2023-03-03T08:02:22.440387+0000","last_became_active":"2023-03-03T08:02:21.871079+0000","last_became_peered":"2023-03-03T08:02:21.871079+0000","last_unstale":"2023-03-03T08:02:22.440387+0000","last_undegraded":"2023-03-03T08:02:22.440387+0000","last_fullsized":"2023-03-03T08:02:22.440387+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:55.500998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594380+0000","last_change":"2023-03-03T08:02:22.433798+0000","last_active":"2023-03-03T08:02:23.594380+0000","last_peered":"2023-03-03T08:02:23.594380+0000","last_clean":"2023-03-03T08:02:23.594380+0000","last_became_active":"2023-03-03T08:02:22.433494+0000","last_became_peered":"2023-03-03T08:02:22.433494+0000","last_unstale":"2023-03-03T08:02:23.594380+0000","last_undegraded":"2023-03-03T08:02:23.594380+0000","last_fullsized":"2023-03-03T08:02:23.594380+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:08.049210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"90'1","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.444249+0000","last_change":"2023-03-03T08:02:21.869557+0000","last_active":"2023-03-03T08:02:22.444249+0000","last_peered":"2023-03-03T08:02:22.444249+0000","last_clean":"2023-03-03T08:02:22.444249+0000","last_became_active":"2023-03-03T08:02:21.869249+0000","last_became_peered":"2023-03-03T08:02:21.869249+0000","last_unstale":"2023-03-03T08:02:22.444249+0000","last_undegraded":"2023-03-03T08:02:22.444249+0000","last_fullsized":"2023-03-03T08:02:22.444249+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:25:07.411525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594399+0000","last_change":"2023-03-03T08:02:22.433798+0000","last_active":"2023-03-03T08:02:23.594399+0000","last_peered":"2023-03-03T08:02:23.594399+0000","last_clean":"2023-03-03T08:02:23.594399+0000","last_became_active":"2023-03-03T08:02:22.433309+0000","last_became_peered":"2023-03-03T08:02:22.433309+0000","last_unstale":"2023-03-03T08:02:23.594399+0000","last_undegraded":"2023-03-03T08:02:23.594399+0000","last_fullsized":"2023-03-03T08:02:23.594399+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:16:15.247935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.410774+0000","last_change":"2023-03-03T08:02:21.870636+0000","last_active":"2023-03-03T08:02:22.410774+0000","last_peered":"2023-03-03T08:02:22.410774+0000","last_clean":"2023-03-03T08:02:22.410774+0000","last_became_active":"2023-03-03T08:02:21.870277+0000","last_became_peered":"2023-03-03T08:02:21.870277+0000","last_unstale":"2023-03-03T08:02:22.410774+0000","last_undegraded":"2023-03-03T08:02:22.410774+0000","last_fullsized":"2023-03-03T08:02:22.410774+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:05:18.330384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594297+0000","last_change":"2023-03-03T08:02:22.438281+0000","last_active":"2023-03-03T08:02:23.594297+0000","last_peered":"2023-03-03T08:02:23.594297+0000","last_clean":"2023-03-03T08:02:23.594297+0000","last_became_active":"2023-03-03T08:02:22.437991+0000","last_became_peered":"2023-03-03T08:02:22.437991+0000","last_unstale":"2023-03-03T08:02:23.594297+0000","last_undegraded":"2023-03-03T08:02:23.594297+0000","last_fullsized":"2023-03-03T08:02:23.594297+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:18:38.349625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.417767+0000","last_change":"2023-03-03T08:02:21.877758+0000","last_active":"2023-03-03T08:02:22.417767+0000","last_peered":"2023-03-03T08:02:22.417767+0000","last_clean":"2023-03-03T08:02:22.417767+0000","last_became_active":"2023-03-03T08:02:21.877460+0000","last_became_peered":"2023-03-03T08:02:21.877460+0000","last_unstale":"2023-03-03T08:02:22.417767+0000","last_undegraded":"2023-03-03T08:02:22.417767+0000","last_fullsized":"2023-03-03T08:02:22.417767+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:05:37.026563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.413046+0000","last_change":"2023-03-03T08:02:22.425333+0000","last_active":"2023-03-03T08:02:23.413046+0000","last_peered":"2023-03-03T08:02:23.413046+0000","last_clean":"2023-03-03T08:02:23.413046+0000","last_became_active":"2023-03-03T08:02:22.421632+0000","last_became_peered":"2023-03-03T08:02:22.421632+0000","last_unstale":"2023-03-03T08:02:23.413046+0000","last_undegraded":"2023-03-03T08:02:23.413046+0000","last_fullsized":"2023-03-03T08:02:23.413046+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:23:35.028360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.416440+0000","last_change":"2023-03-03T08:02:21.871439+0000","last_active":"2023-03-03T08:02:22.416440+0000","last_peered":"2023-03-03T08:02:22.416440+0000","last_clean":"2023-03-03T08:02:22.416440+0000","last_became_active":"2023-03-03T08:02:21.870955+0000","last_became_peered":"2023-03-03T08:02:21.870955+0000","last_unstale":"2023-03-03T08:02:22.416440+0000","last_undegraded":"2023-03-03T08:02:22.416440+0000","last_fullsized":"2023-03-03T08:02:22.416440+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:43:59.937532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.411713+0000","last_change":"2023-03-03T08:02:22.424754+0000","last_active":"2023-03-03T08:02:23.411713+0000","last_peered":"2023-03-03T08:02:23.411713+0000","last_clean":"2023-03-03T08:02:23.411713+0000","last_became_active":"2023-03-03T08:02:22.424452+0000","last_became_peered":"2023-03-03T08:02:22.424452+0000","last_unstale":"2023-03-03T08:02:23.411713+0000","last_undegraded":"2023-03-03T08:02:23.411713+0000","last_fullsized":"2023-03-03T08:02:23.411713+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:21:32.221194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.411881+0000","last_change":"2023-03-03T08:02:21.878213+0000","last_active":"2023-03-03T08:02:22.411881+0000","last_peered":"2023-03-03T08:02:22.411881+0000","last_clean":"2023-03-03T08:02:22.411881+0000","last_became_active":"2023-03-03T08:02:21.877934+0000","last_became_peered":"2023-03-03T08:02:21.877934+0000","last_unstale":"2023-03-03T08:02:22.411881+0000","last_undegraded":"2023-03-03T08:02:22.411881+0000","last_fullsized":"2023-03-03T08:02:22.411881+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:11.554773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594809+0000","last_change":"2023-03-03T08:02:22.431176+0000","last_active":"2023-03-03T08:02:23.594809+0000","last_peered":"2023-03-03T08:02:23.594809+0000","last_clean":"2023-03-03T08:02:23.594809+0000","last_became_active":"2023-03-03T08:02:22.430932+0000","last_became_peered":"2023-03-03T08:02:22.430932+0000","last_unstale":"2023-03-03T08:02:23.594809+0000","last_undegraded":"2023-03-03T08:02:23.594809+0000","last_fullsized":"2023-03-03T08:02:23.594809+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:11.514077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"90'3","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.447168+0000","last_change":"2023-03-03T08:02:21.878651+0000","last_active":"2023-03-03T08:02:22.447168+0000","last_peered":"2023-03-03T08:02:22.447168+0000","last_clean":"2023-03-03T08:02:22.447168+0000","last_became_active":"2023-03-03T08:02:21.878379+0000","last_became_peered":"2023-03-03T08:02:21.878379+0000","last_unstale":"2023-03-03T08:02:22.447168+0000","last_undegraded":"2023-03-03T08:02:22.447168+0000","last_fullsized":"2023-03-03T08:02:22.447168+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:04:25.058258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594398+0000","last_change":"2023-03-03T08:02:22.429360+0000","last_active":"2023-03-03T08:02:23.594398+0000","last_peered":"2023-03-03T08:02:23.594398+0000","last_clean":"2023-03-03T08:02:23.594398+0000","last_became_active":"2023-03-03T08:02:22.429108+0000","last_became_peered":"2023-03-03T08:02:22.429108+0000","last_unstale":"2023-03-03T08:02:23.594398+0000","last_undegraded":"2023-03-03T08:02:23.594398+0000","last_fullsized":"2023-03-03T08:02:23.594398+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:06:00.784892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"90'6","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.447872+0000","last_change":"2023-03-03T08:02:21.876477+0000","last_active":"2023-03-03T08:02:22.447872+0000","last_peered":"2023-03-03T08:02:22.447872+0000","last_clean":"2023-03-03T08:02:22.447872+0000","last_became_active":"2023-03-03T08:02:21.876153+0000","last_became_peered":"2023-03-03T08:02:21.876153+0000","last_unstale":"2023-03-03T08:02:22.447872+0000","last_undegraded":"2023-03-03T08:02:22.447872+0000","last_fullsized":"2023-03-03T08:02:22.447872+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:20:03.570463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.596042+0000","last_change":"2023-03-03T08:02:22.427140+0000","last_active":"2023-03-03T08:02:23.596042+0000","last_peered":"2023-03-03T08:02:23.596042+0000","last_clean":"2023-03-03T08:02:23.596042+0000","last_became_active":"2023-03-03T08:02:22.426570+0000","last_became_peered":"2023-03-03T08:02:22.426570+0000","last_unstale":"2023-03-03T08:02:23.596042+0000","last_undegraded":"2023-03-03T08:02:23.596042+0000","last_fullsized":"2023-03-03T08:02:23.596042+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:04:26.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"90'2","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.447529+0000","last_change":"2023-03-03T08:02:21.877740+0000","last_active":"2023-03-03T08:02:22.447529+0000","last_peered":"2023-03-03T08:02:22.447529+0000","last_clean":"2023-03-03T08:02:22.447529+0000","last_became_active":"2023-03-03T08:02:21.877334+0000","last_became_peered":"2023-03-03T08:02:21.877334+0000","last_unstale":"2023-03-03T08:02:22.447529+0000","last_undegraded":"2023-03-03T08:02:22.447529+0000","last_fullsized":"2023-03-03T08:02:22.447529+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:02.784516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.412761+0000","last_change":"2023-03-03T08:02:22.429658+0000","last_active":"2023-03-03T08:02:23.412761+0000","last_peered":"2023-03-03T08:02:23.412761+0000","last_clean":"2023-03-03T08:02:23.412761+0000","last_became_active":"2023-03-03T08:02:22.429355+0000","last_became_peered":"2023-03-03T08:02:22.429355+0000","last_unstale":"2023-03-03T08:02:23.412761+0000","last_undegraded":"2023-03-03T08:02:23.412761+0000","last_fullsized":"2023-03-03T08:02:23.412761+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:58:42.565751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.421281+0000","last_change":"2023-03-03T08:02:21.877446+0000","last_active":"2023-03-03T08:02:22.421281+0000","last_peered":"2023-03-03T08:02:22.421281+0000","last_clean":"2023-03-03T08:02:22.421281+0000","last_became_active":"2023-03-03T08:02:21.877142+0000","last_became_peered":"2023-03-03T08:02:21.877142+0000","last_unstale":"2023-03-03T08:02:22.421281+0000","last_undegraded":"2023-03-03T08:02:22.421281+0000","last_fullsized":"2023-03-03T08:02:22.421281+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:30:03.746263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594293+0000","last_change":"2023-03-03T08:02:22.441219+0000","last_active":"2023-03-03T08:02:23.594293+0000","last_peered":"2023-03-03T08:02:23.594293+0000","last_clean":"2023-03-03T08:02:23.594293+0000","last_became_active":"2023-03-03T08:02:22.440937+0000","last_became_peered":"2023-03-03T08:02:22.440937+0000","last_unstale":"2023-03-03T08:02:23.594293+0000","last_undegraded":"2023-03-03T08:02:23.594293+0000","last_fullsized":"2023-03-03T08:02:23.594293+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:39:31.044775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.416561+0000","last_change":"2023-03-03T08:02:21.875342+0000","last_active":"2023-03-03T08:02:22.416561+0000","last_peered":"2023-03-03T08:02:22.416561+0000","last_clean":"2023-03-03T08:02:22.416561+0000","last_became_active":"2023-03-03T08:02:21.874999+0000","last_became_peered":"2023-03-03T08:02:21.874999+0000","last_unstale":"2023-03-03T08:02:22.416561+0000","last_undegraded":"2023-03-03T08:02:22.416561+0000","last_fullsized":"2023-03-03T08:02:22.416561+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:14.083774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594646+0000","last_change":"2023-03-03T08:02:22.441976+0000","last_active":"2023-03-03T08:02:23.594646+0000","last_peered":"2023-03-03T08:02:23.594646+0000","last_clean":"2023-03-03T08:02:23.594646+0000","last_became_active":"2023-03-03T08:02:22.441582+0000","last_became_peered":"2023-03-03T08:02:22.441582+0000","last_unstale":"2023-03-03T08:02:23.594646+0000","last_undegraded":"2023-03-03T08:02:23.594646+0000","last_fullsized":"2023-03-03T08:02:23.594646+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:29:47.652120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.446476+0000","last_change":"2023-03-03T08:02:21.870898+0000","last_active":"2023-03-03T08:02:22.446476+0000","last_peered":"2023-03-03T08:02:22.446476+0000","last_clean":"2023-03-03T08:02:22.446476+0000","last_became_active":"2023-03-03T08:02:21.870586+0000","last_became_peered":"2023-03-03T08:02:21.870586+0000","last_unstale":"2023-03-03T08:02:22.446476+0000","last_undegraded":"2023-03-03T08:02:22.446476+0000","last_fullsized":"2023-03-03T08:02:22.446476+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:29:07.480349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594228+0000","last_change":"2023-03-03T08:02:22.429696+0000","last_active":"2023-03-03T08:02:23.594228+0000","last_peered":"2023-03-03T08:02:23.594228+0000","last_clean":"2023-03-03T08:02:23.594228+0000","last_became_active":"2023-03-03T08:02:22.429407+0000","last_became_peered":"2023-03-03T08:02:22.429407+0000","last_unstale":"2023-03-03T08:02:23.594228+0000","last_undegraded":"2023-03-03T08:02:23.594228+0000","last_fullsized":"2023-03-03T08:02:23.594228+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:09:13.117580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"90'2","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.440540+0000","last_change":"2023-03-03T08:02:21.875105+0000","last_active":"2023-03-03T08:02:22.440540+0000","last_peered":"2023-03-03T08:02:22.440540+0000","last_clean":"2023-03-03T08:02:22.440540+0000","last_became_active":"2023-03-03T08:02:21.874788+0000","last_became_peered":"2023-03-03T08:02:21.874788+0000","last_unstale":"2023-03-03T08:02:22.440540+0000","last_undegraded":"2023-03-03T08:02:22.440540+0000","last_fullsized":"2023-03-03T08:02:22.440540+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:05:18.330384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.596405+0000","last_change":"2023-03-03T08:02:22.436727+0000","last_active":"2023-03-03T08:02:23.596405+0000","last_peered":"2023-03-03T08:02:23.596405+0000","last_clean":"2023-03-03T08:02:23.596405+0000","last_became_active":"2023-03-03T08:02:22.432731+0000","last_became_peered":"2023-03-03T08:02:22.432731+0000","last_unstale":"2023-03-03T08:02:23.596405+0000","last_undegraded":"2023-03-03T08:02:23.596405+0000","last_fullsized":"2023-03-03T08:02:23.596405+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:50:52.137242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.413561+0000","last_change":"2023-03-03T08:02:21.874614+0000","last_active":"2023-03-03T08:02:22.413561+0000","last_peered":"2023-03-03T08:02:22.413561+0000","last_clean":"2023-03-03T08:02:22.413561+0000","last_became_active":"2023-03-03T08:02:21.874310+0000","last_became_peered":"2023-03-03T08:02:21.874310+0000","last_unstale":"2023-03-03T08:02:22.413561+0000","last_undegraded":"2023-03-03T08:02:22.413561+0000","last_fullsized":"2023-03-03T08:02:22.413561+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:21.946190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594760+0000","last_change":"2023-03-03T08:02:22.434939+0000","last_active":"2023-03-03T08:02:23.594760+0000","last_peered":"2023-03-03T08:02:23.594760+0000","last_clean":"2023-03-03T08:02:23.594760+0000","last_became_active":"2023-03-03T08:02:22.434169+0000","last_became_peered":"2023-03-03T08:02:22.434169+0000","last_unstale":"2023-03-03T08:02:23.594760+0000","last_undegraded":"2023-03-03T08:02:23.594760+0000","last_fullsized":"2023-03-03T08:02:23.594760+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:20:05.910132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.413479+0000","last_change":"2023-03-03T08:02:21.877730+0000","last_active":"2023-03-03T08:02:22.413479+0000","last_peered":"2023-03-03T08:02:22.413479+0000","last_clean":"2023-03-03T08:02:22.413479+0000","last_became_active":"2023-03-03T08:02:21.877447+0000","last_became_peered":"2023-03-03T08:02:21.877447+0000","last_unstale":"2023-03-03T08:02:22.413479+0000","last_undegraded":"2023-03-03T08:02:22.413479+0000","last_fullsized":"2023-03-03T08:02:22.413479+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:06.949059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594547+0000","last_change":"2023-03-03T08:02:22.424455+0000","last_active":"2023-03-03T08:02:23.594547+0000","last_peered":"2023-03-03T08:02:23.594547+0000","last_clean":"2023-03-03T08:02:23.594547+0000","last_became_active":"2023-03-03T08:02:22.424157+0000","last_became_peered":"2023-03-03T08:02:22.424157+0000","last_unstale":"2023-03-03T08:02:23.594547+0000","last_undegraded":"2023-03-03T08:02:23.594547+0000","last_fullsized":"2023-03-03T08:02:23.594547+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:18:38.349625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.421462+0000","last_change":"2023-03-03T08:02:21.879897+0000","last_active":"2023-03-03T08:02:22.421462+0000","last_peered":"2023-03-03T08:02:22.421462+0000","last_clean":"2023-03-03T08:02:22.421462+0000","last_became_active":"2023-03-03T08:02:21.879628+0000","last_became_peered":"2023-03-03T08:02:21.879628+0000","last_unstale":"2023-03-03T08:02:22.421462+0000","last_undegraded":"2023-03-03T08:02:22.421462+0000","last_fullsized":"2023-03-03T08:02:22.421462+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:40:24.235160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595330+0000","last_change":"2023-03-03T08:02:22.432956+0000","last_active":"2023-03-03T08:02:23.595330+0000","last_peered":"2023-03-03T08:02:23.595330+0000","last_clean":"2023-03-03T08:02:23.595330+0000","last_became_active":"2023-03-03T08:02:22.432701+0000","last_became_peered":"2023-03-03T08:02:22.432701+0000","last_unstale":"2023-03-03T08:02:23.595330+0000","last_undegraded":"2023-03-03T08:02:23.595330+0000","last_fullsized":"2023-03-03T08:02:23.595330+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:18:46.954636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.420995+0000","last_change":"2023-03-03T08:02:21.877397+0000","last_active":"2023-03-03T08:02:22.420995+0000","last_peered":"2023-03-03T08:02:22.420995+0000","last_clean":"2023-03-03T08:02:22.420995+0000","last_became_active":"2023-03-03T08:02:21.877124+0000","last_became_peered":"2023-03-03T08:02:21.877124+0000","last_unstale":"2023-03-03T08:02:22.420995+0000","last_undegraded":"2023-03-03T08:02:22.420995+0000","last_fullsized":"2023-03-03T08:02:22.420995+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:07:16.560533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595601+0000","last_change":"2023-03-03T08:02:22.430259+0000","last_active":"2023-03-03T08:02:23.595601+0000","last_peered":"2023-03-03T08:02:23.595601+0000","last_clean":"2023-03-03T08:02:23.595601+0000","last_became_active":"2023-03-03T08:02:22.429912+0000","last_became_peered":"2023-03-03T08:02:22.429912+0000","last_unstale":"2023-03-03T08:02:23.595601+0000","last_undegraded":"2023-03-03T08:02:23.595601+0000","last_fullsized":"2023-03-03T08:02:23.595601+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:21:32.221194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.414724+0000","last_change":"2023-03-03T08:02:21.868423+0000","last_active":"2023-03-03T08:02:22.414724+0000","last_peered":"2023-03-03T08:02:22.414724+0000","last_clean":"2023-03-03T08:02:22.414724+0000","last_became_active":"2023-03-03T08:02:21.868122+0000","last_became_peered":"2023-03-03T08:02:21.868122+0000","last_unstale":"2023-03-03T08:02:22.414724+0000","last_undegraded":"2023-03-03T08:02:22.414724+0000","last_fullsized":"2023-03-03T08:02:22.414724+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:27:52.756150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594073+0000","last_change":"2023-03-03T08:02:22.432093+0000","last_active":"2023-03-03T08:02:23.594073+0000","last_peered":"2023-03-03T08:02:23.594073+0000","last_clean":"2023-03-03T08:02:23.594073+0000","last_became_active":"2023-03-03T08:02:22.431777+0000","last_became_peered":"2023-03-03T08:02:22.431777+0000","last_unstale":"2023-03-03T08:02:23.594073+0000","last_undegraded":"2023-03-03T08:02:23.594073+0000","last_fullsized":"2023-03-03T08:02:23.594073+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:01:26.166146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.420496+0000","last_change":"2023-03-03T08:02:21.880896+0000","last_active":"2023-03-03T08:02:22.420496+0000","last_peered":"2023-03-03T08:02:22.420496+0000","last_clean":"2023-03-03T08:02:22.420496+0000","last_became_active":"2023-03-03T08:02:21.880537+0000","last_became_peered":"2023-03-03T08:02:21.880537+0000","last_unstale":"2023-03-03T08:02:22.420496+0000","last_undegraded":"2023-03-03T08:02:22.420496+0000","last_fullsized":"2023-03-03T08:02:22.420496+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:28.785586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595357+0000","last_change":"2023-03-03T08:02:22.444781+0000","last_active":"2023-03-03T08:02:23.595357+0000","last_peered":"2023-03-03T08:02:23.595357+0000","last_clean":"2023-03-03T08:02:23.595357+0000","last_became_active":"2023-03-03T08:02:22.444469+0000","last_became_peered":"2023-03-03T08:02:22.444469+0000","last_unstale":"2023-03-03T08:02:23.595357+0000","last_undegraded":"2023-03-03T08:02:23.595357+0000","last_fullsized":"2023-03-03T08:02:23.595357+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:22:38.933853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.417895+0000","last_change":"2023-03-03T08:02:21.878700+0000","last_active":"2023-03-03T08:02:22.417895+0000","last_peered":"2023-03-03T08:02:22.417895+0000","last_clean":"2023-03-03T08:02:22.417895+0000","last_became_active":"2023-03-03T08:02:21.878429+0000","last_became_peered":"2023-03-03T08:02:21.878429+0000","last_unstale":"2023-03-03T08:02:22.417895+0000","last_undegraded":"2023-03-03T08:02:22.417895+0000","last_fullsized":"2023-03-03T08:02:22.417895+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:38.341401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595201+0000","last_change":"2023-03-03T08:02:22.436418+0000","last_active":"2023-03-03T08:02:23.595201+0000","last_peered":"2023-03-03T08:02:23.595201+0000","last_clean":"2023-03-03T08:02:23.595201+0000","last_became_active":"2023-03-03T08:02:22.436168+0000","last_became_peered":"2023-03-03T08:02:22.436168+0000","last_unstale":"2023-03-03T08:02:23.595201+0000","last_undegraded":"2023-03-03T08:02:23.595201+0000","last_fullsized":"2023-03-03T08:02:23.595201+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:36:52.462067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.424333+0000","last_change":"2023-03-03T08:02:21.874537+0000","last_active":"2023-03-03T08:02:22.424333+0000","last_peered":"2023-03-03T08:02:22.424333+0000","last_clean":"2023-03-03T08:02:22.424333+0000","last_became_active":"2023-03-03T08:02:21.874287+0000","last_became_peered":"2023-03-03T08:02:21.874287+0000","last_unstale":"2023-03-03T08:02:22.424333+0000","last_undegraded":"2023-03-03T08:02:22.424333+0000","last_fullsized":"2023-03-03T08:02:22.424333+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:26:41.817800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.419690+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T08:02:22.419690+0000","last_peered":"2023-03-03T08:02:22.419690+0000","last_clean":"2023-03-03T08:02:22.419690+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T08:02:22.419690+0000","last_undegraded":"2023-03-03T08:02:22.419690+0000","last_fullsized":"2023-03-03T08:02:22.419690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.593940+0000","last_change":"2023-03-03T08:02:22.434463+0000","last_active":"2023-03-03T08:02:23.593940+0000","last_peered":"2023-03-03T08:02:23.593940+0000","last_clean":"2023-03-03T08:02:23.593940+0000","last_became_active":"2023-03-03T08:02:22.434172+0000","last_became_peered":"2023-03-03T08:02:22.434172+0000","last_unstale":"2023-03-03T08:02:23.593940+0000","last_undegraded":"2023-03-03T08:02:23.593940+0000","last_fullsized":"2023-03-03T08:02:23.593940+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:44:24.059066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.414103+0000","last_change":"2023-03-03T08:02:21.877158+0000","last_active":"2023-03-03T08:02:22.414103+0000","last_peered":"2023-03-03T08:02:22.414103+0000","last_clean":"2023-03-03T08:02:22.414103+0000","last_became_active":"2023-03-03T08:02:21.876773+0000","last_became_peered":"2023-03-03T08:02:21.876773+0000","last_unstale":"2023-03-03T08:02:22.414103+0000","last_undegraded":"2023-03-03T08:02:22.414103+0000","last_fullsized":"2023-03-03T08:02:22.414103+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:15:55.500998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.417890+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T08:02:22.417890+0000","last_peered":"2023-03-03T08:02:22.417890+0000","last_clean":"2023-03-03T08:02:22.417890+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T08:02:22.417890+0000","last_undegraded":"2023-03-03T08:02:22.417890+0000","last_fullsized":"2023-03-03T08:02:22.417890+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.593819+0000","last_change":"2023-03-03T08:02:22.430818+0000","last_active":"2023-03-03T08:02:23.593819+0000","last_peered":"2023-03-03T08:02:23.593819+0000","last_clean":"2023-03-03T08:02:23.593819+0000","last_became_active":"2023-03-03T08:02:22.430476+0000","last_became_peered":"2023-03-03T08:02:22.430476+0000","last_unstale":"2023-03-03T08:02:23.593819+0000","last_undegraded":"2023-03-03T08:02:23.593819+0000","last_fullsized":"2023-03-03T08:02:23.593819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:05:38.030389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.422265+0000","last_change":"2023-03-03T08:02:21.878320+0000","last_active":"2023-03-03T08:02:22.422265+0000","last_peered":"2023-03-03T08:02:22.422265+0000","last_clean":"2023-03-03T08:02:22.422265+0000","last_became_active":"2023-03-03T08:02:21.877985+0000","last_became_peered":"2023-03-03T08:02:21.877985+0000","last_unstale":"2023-03-03T08:02:22.422265+0000","last_undegraded":"2023-03-03T08:02:22.422265+0000","last_fullsized":"2023-03-03T08:02:22.422265+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:58:25.892217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.416981+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T08:02:22.416981+0000","last_peered":"2023-03-03T08:02:22.416981+0000","last_clean":"2023-03-03T08:02:22.416981+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T08:02:22.416981+0000","last_undegraded":"2023-03-03T08:02:22.416981+0000","last_fullsized":"2023-03-03T08:02:22.416981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594871+0000","last_change":"2023-03-03T08:02:22.432594+0000","last_active":"2023-03-03T08:02:23.594871+0000","last_peered":"2023-03-03T08:02:23.594871+0000","last_clean":"2023-03-03T08:02:23.594871+0000","last_became_active":"2023-03-03T08:02:22.432316+0000","last_became_peered":"2023-03-03T08:02:22.432316+0000","last_unstale":"2023-03-03T08:02:23.594871+0000","last_undegraded":"2023-03-03T08:02:23.594871+0000","last_fullsized":"2023-03-03T08:02:23.594871+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:21:10.960686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"90'1","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.439455+0000","last_change":"2023-03-03T08:02:21.879578+0000","last_active":"2023-03-03T08:02:22.439455+0000","last_peered":"2023-03-03T08:02:22.439455+0000","last_clean":"2023-03-03T08:02:22.439455+0000","last_became_active":"2023-03-03T08:02:21.879264+0000","last_became_peered":"2023-03-03T08:02:21.879264+0000","last_unstale":"2023-03-03T08:02:22.439455+0000","last_undegraded":"2023-03-03T08:02:22.439455+0000","last_fullsized":"2023-03-03T08:02:22.439455+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:37.978753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.417999+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T08:02:22.417999+0000","last_peered":"2023-03-03T08:02:22.417999+0000","last_clean":"2023-03-03T08:02:22.417999+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T08:02:22.417999+0000","last_undegraded":"2023-03-03T08:02:22.417999+0000","last_fullsized":"2023-03-03T08:02:22.417999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.411854+0000","last_change":"2023-03-03T08:02:22.433910+0000","last_active":"2023-03-03T08:02:23.411854+0000","last_peered":"2023-03-03T08:02:23.411854+0000","last_clean":"2023-03-03T08:02:23.411854+0000","last_became_active":"2023-03-03T08:02:22.433646+0000","last_became_peered":"2023-03-03T08:02:22.433646+0000","last_unstale":"2023-03-03T08:02:23.411854+0000","last_undegraded":"2023-03-03T08:02:23.411854+0000","last_fullsized":"2023-03-03T08:02:23.411854+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:54:45.588606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.416680+0000","last_change":"2023-03-03T08:02:21.873962+0000","last_active":"2023-03-03T08:02:22.416680+0000","last_peered":"2023-03-03T08:02:22.416680+0000","last_clean":"2023-03-03T08:02:22.416680+0000","last_became_active":"2023-03-03T08:02:21.873697+0000","last_became_peered":"2023-03-03T08:02:21.873697+0000","last_unstale":"2023-03-03T08:02:22.416680+0000","last_undegraded":"2023-03-03T08:02:22.416680+0000","last_fullsized":"2023-03-03T08:02:22.416680+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:32:37.978753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":266,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.418046+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T08:02:22.418046+0000","last_peered":"2023-03-03T08:02:22.418046+0000","last_clean":"2023-03-03T08:02:22.418046+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T08:02:22.418046+0000","last_undegraded":"2023-03-03T08:02:22.418046+0000","last_fullsized":"2023-03-03T08:02:22.418046+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.415422+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T08:02:22.415422+0000","last_peered":"2023-03-03T08:02:22.415422+0000","last_clean":"2023-03-03T08:02:22.415422+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T08:02:22.415422+0000","last_undegraded":"2023-03-03T08:02:22.415422+0000","last_fullsized":"2023-03-03T08:02:22.415422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.594327+0000","last_change":"2023-03-03T08:02:22.432064+0000","last_active":"2023-03-03T08:02:23.594327+0000","last_peered":"2023-03-03T08:02:23.594327+0000","last_clean":"2023-03-03T08:02:23.594327+0000","last_became_active":"2023-03-03T08:02:22.431673+0000","last_became_peered":"2023-03-03T08:02:22.431673+0000","last_unstale":"2023-03-03T08:02:23.594327+0000","last_undegraded":"2023-03-03T08:02:23.594327+0000","last_fullsized":"2023-03-03T08:02:23.594327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:42:16.102515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"90'2","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.440917+0000","last_change":"2023-03-03T08:02:21.874486+0000","last_active":"2023-03-03T08:02:22.440917+0000","last_peered":"2023-03-03T08:02:22.440917+0000","last_clean":"2023-03-03T08:02:22.440917+0000","last_became_active":"2023-03-03T08:02:21.874149+0000","last_became_peered":"2023-03-03T08:02:21.874149+0000","last_unstale":"2023-03-03T08:02:22.440917+0000","last_undegraded":"2023-03-03T08:02:22.440917+0000","last_fullsized":"2023-03-03T08:02:22.440917+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:05:59.781066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.422063+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T08:02:22.422063+0000","last_peered":"2023-03-03T08:02:22.422063+0000","last_clean":"2023-03-03T08:02:22.422063+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T08:02:22.422063+0000","last_undegraded":"2023-03-03T08:02:22.422063+0000","last_fullsized":"2023-03-03T08:02:22.422063+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.413104+0000","last_change":"2023-03-03T08:02:22.431014+0000","last_active":"2023-03-03T08:02:23.413104+0000","last_peered":"2023-03-03T08:02:23.413104+0000","last_clean":"2023-03-03T08:02:23.413104+0000","last_became_active":"2023-03-03T08:02:22.430177+0000","last_became_peered":"2023-03-03T08:02:22.430177+0000","last_unstale":"2023-03-03T08:02:23.413104+0000","last_undegraded":"2023-03-03T08:02:23.413104+0000","last_fullsized":"2023-03-03T08:02:23.413104+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:21:32.221194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"90'1","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.440807+0000","last_change":"2023-03-03T08:02:21.878713+0000","last_active":"2023-03-03T08:02:22.440807+0000","last_peered":"2023-03-03T08:02:22.440807+0000","last_clean":"2023-03-03T08:02:22.440807+0000","last_became_active":"2023-03-03T08:02:21.878470+0000","last_became_peered":"2023-03-03T08:02:21.878470+0000","last_unstale":"2023-03-03T08:02:22.440807+0000","last_undegraded":"2023-03-03T08:02:22.440807+0000","last_fullsized":"2023-03-03T08:02:22.440807+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:38.341401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.414817+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T08:02:22.414817+0000","last_peered":"2023-03-03T08:02:22.414817+0000","last_clean":"2023-03-03T08:02:22.414817+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T08:02:22.414817+0000","last_undegraded":"2023-03-03T08:02:22.414817+0000","last_fullsized":"2023-03-03T08:02:22.414817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-03-03T08:02:23.595896+0000","last_change":"2023-03-03T08:02:22.431946+0000","last_active":"2023-03-03T08:02:23.595896+0000","last_peered":"2023-03-03T08:02:23.595896+0000","last_clean":"2023-03-03T08:02:23.595896+0000","last_became_active":"2023-03-03T08:02:22.431388+0000","last_became_peered":"2023-03-03T08:02:22.431388+0000","last_unstale":"2023-03-03T08:02:23.595896+0000","last_undegraded":"2023-03-03T08:02:23.595896+0000","last_fullsized":"2023-03-03T08:02:23.595896+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:21.851588+0000","last_clean_scrub_stamp":"2023-03-03T08:02:21.851588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:52.697811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"90'2","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.436052+0000","last_change":"2023-03-03T08:02:21.867808+0000","last_active":"2023-03-03T08:02:22.436052+0000","last_peered":"2023-03-03T08:02:22.436052+0000","last_clean":"2023-03-03T08:02:22.436052+0000","last_became_active":"2023-03-03T08:02:21.867524+0000","last_became_peered":"2023-03-03T08:02:21.867524+0000","last_unstale":"2023-03-03T08:02:22.436052+0000","last_undegraded":"2023-03-03T08:02:22.436052+0000","last_fullsized":"2023-03-03T08:02:22.436052+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:02:20.847762+0000","last_clean_scrub_stamp":"2023-03-03T08:02:20.847762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:21.946190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-03-03T08:02:22.416817+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T08:02:22.416817+0000","last_peered":"2023-03-03T08:02:22.416817+0000","last_clean":"2023-03-03T08:02:22.416817+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T08:02:22.416817+0000","last_undegraded":"2023-03-03T08:02:22.416817+0000","last_fullsized":"2023-03-03T08:02:22.416817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574934,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11928,"kb_used_data":1680,"kb_used_omap":0,"kb_used_meta":10240,"kb_avail":94359912,"statfs":{"total":96636764160,"available":96624549888,"internally_reserved":0,"allocated":1720320,"data_stored":1099785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10485760},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.66100000000000003,"15min":0.70099999999999996},"min":{"1min":0.58499999999999996,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":1.546},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.67700000000000005,"15min":0.71799999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.78500000000000003,"5min":0.96499999999999997,"15min":1.702},"last":0.58199999999999996}]},{"osd":1,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.70799999999999996,"15min":1.024},"min":{"1min":0.52900000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.96299999999999997,"5min":1.111,"15min":6.6829999999999998},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.49399999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.84999999999999998,"5min":1.1890000000000001,"15min":1.944},"last":0.51900000000000002}]},{"osd":2,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.72799999999999998,"15min":0.71699999999999997},"min":{"1min":0.56599999999999995,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94499999999999995,"5min":1.224,"15min":1.252},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.74099999999999999,"15min":0.78400000000000003},"min":{"1min":0.57999999999999996,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.92100000000000004,"5min":1.2490000000000001,"15min":1.329},"last":0.496}]},{"osd":3,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71399999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.72099999999999997,"15min":0.77900000000000003},"min":{"1min":0.55200000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.92400000000000004,"5min":1.0640000000000001,"15min":1.294},"last":0.64500000000000002}]},{"osd":4,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.624,"15min":0.64900000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":0.76200000000000001,"5min":1.0049999999999999,"15min":1.077},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68799999999999994,"15min":0.64100000000000001},"min":{"1min":0.48199999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.99299999999999999,"5min":1.081,"15min":1.081},"last":0.41999999999999998}]},{"osd":5,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.70799999999999996,"15min":0.78000000000000003},"min":{"1min":0.53100000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80800000000000005,"5min":1.0409999999999999,"15min":1.3819999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.78600000000000003,"15min":0.81399999999999995},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.97999999999999998,"5min":1.2190000000000001,"15min":1.349},"last":0.67800000000000005}]},{"osd":6,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.79900000000000004,"15min":0.82299999999999995},"min":{"1min":0.59199999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79800000000000004,"15min":0.81599999999999995},"min":{"1min":0.63700000000000001,"5min":0.499,"15min":0.499},"max":{"1min":1.0329999999999999,"5min":1.135,"15min":1.194},"last":0.60499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574935,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10524,"kb_used_data":1108,"kb_used_omap":0,"kb_used_meta":9408,"kb_avail":94361316,"statfs":{"total":96636764160,"available":96625987584,"internally_reserved":0,"allocated":1134592,"data_stored":510478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9633792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.73099999999999998,"15min":0.69699999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.40100000000000002},"max":{"1min":1.016,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.71099999999999997,"15min":0.70199999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.55400000000000005}]},{"osd":1,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72399999999999998,"15min":0.68999999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0900000000000001,"5min":3.323,"15min":3.323},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.73499999999999999},"min":{"1min":0.55200000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89600000000000002,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.49199999999999999}]},{"osd":2,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.75800000000000001,"15min":0.73399999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.96699999999999997,"5min":1.54,"15min":1.54},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.74199999999999999,"15min":0.75600000000000001},"min":{"1min":0.52700000000000002,"5min":0.495,"15min":0.495},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.627}]},{"osd":3,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.80800000000000005,"15min":0.78800000000000003},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0309999999999999,"5min":4.8959999999999999,"15min":4.8959999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78400000000000003,"15min":0.77500000000000002},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.55600000000000005},"max":{"1min":0.90900000000000003,"5min":1.2270000000000001,"15min":1.343},"last":0.73699999999999999}]},{"osd":4,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.70799999999999996,"15min":0.88100000000000001},"min":{"1min":0.44800000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":4.0099999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.71299999999999997,"15min":0.89300000000000002},"min":{"1min":0.47499999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.98499999999999999,"5min":1.012,"15min":4.1859999999999999},"last":0.64100000000000001}]},{"osd":5,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.69199999999999995,"15min":0.76500000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.99299999999999999,"5min":1.03,"15min":1.6339999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.79800000000000004,"15min":0.82399999999999995},"min":{"1min":0.55400000000000005,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.056,"5min":1.202,"15min":1.429},"last":0.67000000000000004}]},{"osd":7,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.80500000000000005,"15min":0.83499999999999996},"min":{"1min":0.54700000000000004,"5min":0.54100000000000004,"15min":0.51900000000000002},"max":{"1min":1.071,"5min":1.107,"15min":1.452},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.78100000000000003,"15min":0.85699999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.1259999999999999,"5min":1.135,"15min":1.579},"last":0.69899999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10896,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":9792,"kb_avail":94360944,"statfs":{"total":96636764160,"available":96625606656,"internally_reserved":0,"allocated":1122304,"data_stored":509431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10027008},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46800000000000003,"15min":0.503},"min":{"1min":0.34300000000000003,"5min":0.307,"15min":0.307},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.4970000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.50800000000000001,"15min":0.623},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.74399999999999999,"5min":0.90200000000000002,"15min":3.972},"last":0.38800000000000001}]},{"osd":2,"last update":"Fri Mar 3 08:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.52500000000000002,"15min":0.89200000000000002},"min":{"1min":0.39900000000000002,"5min":0.34899999999999998,"15min":0.313},"max":{"1min":0.77100000000000002,"5min":0.78900000000000003,"15min":11.897},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59499999999999997,"15min":0.97099999999999997},"min":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78700000000000003,"5min":0.85199999999999998,"15min":11.804},"last":0.66200000000000003}]},{"osd":3,"last update":"Fri Mar 3 08:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58799999999999997,"15min":0.96099999999999997},"min":{"1min":0.434,"5min":0.371,"15min":0.371},"max":{"1min":0.85899999999999999,"5min":0.89500000000000002,"15min":11.843999999999999},"last":0.62},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55200000000000005,"15min":0.70499999999999996},"min":{"1min":0.45100000000000001,"5min":0.40000000000000002,"15min":0.36699999999999999},"max":{"1min":0.75600000000000001,"5min":0.96399999999999997,"15min":3.8740000000000001},"last":0.55300000000000005}]},{"osd":4,"last update":"Fri Mar 3 08:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.75800000000000001,"15min":0.84299999999999997},"min":{"1min":0.55100000000000005,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90000000000000002,"5min":1.1459999999999999,"15min":1.613},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81999999999999995,"15min":0.85499999999999998},"min":{"1min":0.67100000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.4630000000000001},"last":0.83799999999999997}]},{"osd":5,"last update":"Fri Mar 3 08:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.75700000000000001,"15min":0.89500000000000002},"min":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":3.8300000000000001},"last":0.746},{"interface":"front","average":{"1min":0.752,"5min":0.753,"15min":0.85099999999999998},"min":{"1min":0.622,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.008,"5min":1.0960000000000001,"15min":1.5469999999999999},"last":0.94699999999999995}]},{"osd":6,"last update":"Fri Mar 3 08:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.81399999999999995,"15min":0.96599999999999997},"min":{"1min":0.626,"5min":0.62,"15min":0.62},"max":{"1min":0.98299999999999998,"5min":1.173,"15min":3.907},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.81399999999999995,"15min":0.91700000000000004},"min":{"1min":0.67900000000000005,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.004,"5min":1.1819999999999999,"15min":1.5820000000000001},"last":0.72399999999999998}]},{"osd":7,"last update":"Fri Mar 3 08:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.79700000000000004,"15min":0.86699999999999999},"min":{"1min":0.59299999999999997,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0840000000000001,"5min":1.417,"15min":1.419},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80000000000000004,"15min":0.97699999999999998},"min":{"1min":0.65100000000000002,"5min":0.58199999999999996,"15min":0.57099999999999995},"max":{"1min":0.96799999999999997,"5min":1.3560000000000001,"15min":3.9399999999999999},"last":0.88600000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574934,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11748,"kb_used_data":1692,"kb_used_omap":0,"kb_used_meta":10048,"kb_avail":94360092,"statfs":{"total":96636764160,"available":96624734208,"internally_reserved":0,"allocated":1732608,"data_stored":1100948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10289152},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 08:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.43099999999999999,"15min":0.46300000000000002},"min":{"1min":0.32200000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56200000000000006,"5min":0.65000000000000002,"15min":0.89300000000000002},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.501,"15min":0.51500000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.64300000000000002,"5min":0.81999999999999995,"15min":0.871},"last":0.55000000000000004}]},{"osd":2,"last update":"Fri Mar 3 08:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.50800000000000001,"15min":0.54200000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.63500000000000001,"5min":0.77400000000000002,"15min":0.79100000000000004},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55800000000000005,"15min":0.58599999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.32800000000000001},"max":{"1min":0.72199999999999998,"5min":0.80800000000000005,"15min":1.1020000000000001},"last":0.44400000000000001}]},{"osd":3,"last update":"Fri Mar 3 08:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52300000000000002,"15min":0.54600000000000004},"min":{"1min":0.41699999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68799999999999994,"5min":0.71499999999999997,"15min":0.98499999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58699999999999997,"15min":0.63100000000000001},"min":{"1min":0.48299999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.69899999999999995,"5min":0.83499999999999996,"15min":1.2450000000000001},"last":0.59799999999999998}]},{"osd":4,"last update":"Fri Mar 3 08:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.79200000000000004,"15min":0.83399999999999996},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.242,"5min":1.242,"15min":1.5189999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.78100000000000003,"15min":0.81499999999999995},"min":{"1min":0.52400000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.218,"5min":1.218,"15min":1.605},"last":0.72199999999999998}]},{"osd":5,"last update":"Fri Mar 3 08:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78700000000000003,"15min":0.81599999999999995},"min":{"1min":0.58199999999999996,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.80200000000000005,"15min":0.81799999999999995},"min":{"1min":0.64200000000000002,"5min":0.60299999999999998,"15min":0.56599999999999995},"max":{"1min":0.99399999999999999,"5min":1.2070000000000001,"15min":1.286},"last":0.65100000000000002}]},{"osd":6,"last update":"Fri Mar 3 08:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.80700000000000005,"15min":0.86899999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1599999999999999,"5min":1.2470000000000001,"15min":1.623},"last":0.63},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80100000000000005,"15min":0.85399999999999998},"min":{"1min":0.60099999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.1240000000000001,"5min":1.137,"15min":1.5780000000000001},"last":0.77500000000000002}]},{"osd":7,"last update":"Fri Mar 3 08:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.80500000000000005,"15min":0.86899999999999999},"min":{"1min":0.59299999999999997,"5min":0.57599999999999996,"15min":0.57099999999999995},"max":{"1min":1.052,"5min":1.1020000000000001,"15min":1.6479999999999999},"last":0.752},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80800000000000005,"15min":0.85199999999999998},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.0840000000000001,"5min":1.1719999999999999,"15min":1.5509999999999999},"last":0.75700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574935,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10504,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":9408,"kb_avail":94361336,"statfs":{"total":96636764160,"available":96626008064,"internally_reserved":0,"allocated":1114112,"data_stored":508342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9633792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.42799999999999999,"15min":0.79700000000000004},"min":{"1min":0.307,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76500000000000001,"5min":0.78900000000000003,"15min":5.7889999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.497,"15min":0.69499999999999995},"min":{"1min":0.38100000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":2.6309999999999998},"last":0.40699999999999997}]},{"osd":1,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.50800000000000001,"15min":0.71199999999999997},"min":{"1min":0.33700000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":2.5209999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57999999999999996,"15min":0.93799999999999994},"min":{"1min":0.45700000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":5.8470000000000004},"last":0.54200000000000004}]},{"osd":3,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55200000000000005,"15min":1.1100000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85999999999999999,"5min":1.012,"15min":12.961},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56200000000000006,"15min":0.95699999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":6.1449999999999996},"last":0.59699999999999998}]},{"osd":4,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81799999999999995,"15min":1.2},"min":{"1min":0.60299999999999998,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1240000000000001,"5min":1.468,"15min":6.484},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84599999999999997,"15min":1.1839999999999999},"min":{"1min":0.66500000000000004,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.1519999999999999,"5min":1.1759999999999999,"15min":6.5510000000000002},"last":0.75800000000000001}]},{"osd":5,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.76600000000000001,"15min":1.111},"min":{"1min":0.59699999999999998,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.167,"5min":1.304,"15min":5.7119999999999997},"last":0.626},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.80200000000000005,"15min":1.163},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0629999999999999,"5min":1.1499999999999999,"15min":6.6369999999999996},"last":0.68899999999999995}]},{"osd":6,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83999999999999997,"15min":1.1719999999999999},"min":{"1min":0.61899999999999999,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.1739999999999999,"5min":1.393,"15min":6.625},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.84199999999999997,"15min":1.196},"min":{"1min":0.71199999999999997,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.137,"5min":1.347,"15min":5.7539999999999996},"last":0.72499999999999998}]},{"osd":7,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.78400000000000003,"15min":1.1539999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":6.7009999999999996},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.81299999999999994,"15min":1.145},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.107,"5min":1.107,"15min":6.6840000000000002},"last":0.70899999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574933,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14988,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":13888,"kb_avail":94356852,"statfs":{"total":96636764160,"available":96621416448,"internally_reserved":0,"allocated":1118208,"data_stored":509361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14221312},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42999999999999999,"15min":0.76800000000000002},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.56699999999999995,"5min":0.748,"15min":9.5820000000000007},"last":4.7050000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.50700000000000001,"15min":0.84099999999999997},"min":{"1min":0.377,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73999999999999999,"5min":0.83599999999999997,"15min":9.7840000000000007},"last":4.2910000000000004}]},{"osd":1,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48699999999999999,"15min":0.51800000000000002},"min":{"1min":0.375,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":0.66000000000000003,"5min":0.83799999999999997,"15min":1},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.53800000000000003,"15min":0.91700000000000004},"min":{"1min":0.36699999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73699999999999999,"5min":0.77400000000000002,"15min":9.702},"last":4.9199999999999999}]},{"osd":2,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.57699999999999996,"15min":0.88},"min":{"1min":0.46999999999999997,"5min":0.38,"15min":0.36499999999999999},"max":{"1min":1.026,"5min":1.026,"15min":8.3190000000000008},"last":3.5489999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.95399999999999996},"min":{"1min":0.49399999999999999,"5min":0.42099999999999999,"15min":0.41899999999999998},"max":{"1min":0.68600000000000005,"5min":0.93400000000000005,"15min":9.734},"last":0.98099999999999998}]},{"osd":4,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.755,"15min":1.097},"min":{"1min":0.61499999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.91900000000000004,"5min":1.1819999999999999,"15min":9.8629999999999995},"last":4.2519999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.80400000000000005,"15min":0.83599999999999997},"min":{"1min":0.58599999999999997,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.115,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":3.5110000000000001}]},{"osd":5,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80100000000000005,"15min":0.82599999999999996},"min":{"1min":0.66300000000000003,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.095,"5min":1.1899999999999999,"15min":1.3779999999999999},"last":3.5880000000000001},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.77700000000000002,"15min":1.1479999999999999},"min":{"1min":0.65200000000000002,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.96299999999999997,"5min":1.0860000000000001,"15min":9.6600000000000001},"last":1.9530000000000001}]},{"osd":6,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78600000000000003,"15min":1.161},"min":{"1min":0.65000000000000002,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.048,"5min":1.0600000000000001,"15min":9.8190000000000008},"last":4.415},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.79200000000000004,"15min":0.85499999999999998},"min":{"1min":0.65600000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0629999999999999,"5min":1.2609999999999999,"15min":1.609},"last":1.2609999999999999}]},{"osd":7,"last update":"Fri Mar 3 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":1.196},"min":{"1min":0.63300000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.0349999999999999,"5min":1.204,"15min":9.625},"last":1.018},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.81999999999999995,"15min":1.1719999999999999},"min":{"1min":0.67900000000000005,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.121,"5min":1.121,"15min":9.8949999999999996},"last":4.3259999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574933,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11156,"kb_used_data":1100,"kb_used_omap":0,"kb_used_meta":10048,"kb_avail":94360684,"statfs":{"total":96636764160,"available":96625340416,"internally_reserved":0,"allocated":1126400,"data_stored":509521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10289152},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.71899999999999997,"15min":0.93899999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.35599999999999998},"max":{"1min":0.874,"5min":1.054,"15min":7.1799999999999997},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70999999999999996,"15min":0.67400000000000004},"min":{"1min":0.55000000000000004,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.63600000000000001}]},{"osd":1,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.73699999999999999,"15min":1.0449999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.94899999999999995,"5min":1.153,"15min":9.5709999999999997},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.747,"15min":0.89600000000000002},"min":{"1min":0.53000000000000003,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.94599999999999995,"5min":1.153,"15min":5.266},"last":0.73299999999999998}]},{"osd":2,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.751,"15min":1.022},"min":{"1min":0.52900000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90900000000000003,"5min":1.034,"15min":8.4529999999999994},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.76100000000000001,"15min":0.96299999999999997},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.004,"5min":1.0880000000000001,"15min":6.6829999999999998},"last":0.66200000000000003}]},{"osd":3,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.55100000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.88,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.59099999999999997,"5min":0.58199999999999996,"15min":0.54300000000000004},"max":{"1min":0.98799999999999999,"5min":1.069,"15min":1.389},"last":0.75900000000000001}]},{"osd":5,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69699999999999995,"15min":0.73499999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84299999999999997,"5min":0.98799999999999999,"15min":1.0349999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.47999999999999998,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97799999999999998,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.54000000000000004}]},{"osd":6,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.76700000000000002,"15min":0.77900000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97399999999999998,"5min":1.032,"15min":1.032},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.79900000000000004,"15min":0.81599999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0489999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.83199999999999996}]},{"osd":7,"last update":"Fri Mar 3 08:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.77800000000000002,"15min":0.81599999999999995},"min":{"1min":0.57299999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.97299999999999998,"5min":1.119,"15min":1.679},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.84499999999999997,"15min":0.91600000000000004},"min":{"1min":0.60699999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.8340000000000001},"last":0.81299999999999994}]}]},{"osd":5,"up_from":13,"seq":55834574934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10504,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":9408,"kb_avail":94361336,"statfs":{"total":96636764160,"available":96626008064,"internally_reserved":0,"allocated":1114112,"data_stored":508342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9633792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66500000000000004,"15min":0.64400000000000002},"min":{"1min":0.432,"5min":0.318,"15min":0.318},"max":{"1min":0.90800000000000003,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.754,"15min":0.74299999999999999},"min":{"1min":0.52200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.96699999999999997,"5min":2.528,"15min":2.528},"last":0.59699999999999998}]},{"osd":1,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.72299999999999998,"15min":0.74099999999999999},"min":{"1min":0.45900000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.879,"5min":1.038,"15min":1.038},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.69699999999999995,"15min":0.64500000000000002},"min":{"1min":0.44800000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.86699999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63}]},{"osd":2,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.68000000000000005,"15min":0.67900000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82799999999999996,"5min":0.998,"15min":0.998},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.71399999999999997,"15min":0.71199999999999997},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.92400000000000004,"5min":1.024,"15min":1.321},"last":0.86599999999999999}]},{"osd":3,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.72199999999999998,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91800000000000004,"5min":1.024,"15min":1.024},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.75,"15min":0.73699999999999999},"min":{"1min":0.55700000000000005,"5min":0.501,"15min":0.501},"max":{"1min":0.995,"5min":1.012,"15min":1.012},"last":0.79500000000000004}]},{"osd":4,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.67900000000000005,"15min":0.66700000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.79000000000000004,"15min":0.78200000000000003},"min":{"1min":0.56399999999999995,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0109999999999999,"5min":1.079,"15min":1.079},"last":0.84299999999999997}]},{"osd":6,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.72099999999999997,"15min":0.78400000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.88300000000000001,"5min":1.012,"15min":1.071},"last":0.745},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71499999999999997,"15min":0.72199999999999998},"min":{"1min":0.48599999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89700000000000002,"5min":0.97099999999999997,"15min":1.0429999999999999},"last":0.67300000000000004}]},{"osd":7,"last update":"Fri Mar 3 08:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.748,"15min":0.754},"min":{"1min":0.52000000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.91000000000000003,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78600000000000003,"15min":0.82299999999999995},"min":{"1min":0.60099999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.94699999999999995,"5min":0.995,"15min":1.1080000000000001},"last":0.91500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:02:27.963 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-03-03T08:02:27.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2023-03-03T08:02:28.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.242+0000 7f483539d700 1 Processor -- start 2023-03-03T08:02:28.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.243+0000 7f483539d700 1 -- start start 2023-03-03T08:02:28.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.244+0000 7f483539d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48300b4aa0 0x7f48300b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:28.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.244+0000 7f483539d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483015dd00 con 0x7f48301609b0 2023-03-03T08:02:28.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.244+0000 7f483539d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48300cd780 con 0x7f48300cf570 2023-03-03T08:02:28.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.244+0000 7f483539d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48300cebb0 con 0x7f48300b4aa0 2023-03-03T08:02:28.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.244+0000 7f482ffff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f48301609b0 0x7f4830160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34274/0 (socket says 172.21.15.98:34274) 2023-03-03T08:02:28.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.244+0000 7f482ffff700 1 -- 172.21.15.98:0/930937100 learned_addr learned my addr 172.21.15.98:0/930937100 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.245+0000 7f4834b9c700 1 --2- 172.21.15.98:0/930937100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48300b4aa0 0x7f48300b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.245+0000 7f482effd700 1 -- 172.21.15.98:0/930937100 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175767607 0 0) 0x7f483015dd00 con 0x7f48301609b0 2023-03-03T08:02:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.245+0000 7f482effd700 1 -- 172.21.15.98:0/930937100 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48180035b0 con 0x7f48301609b0 2023-03-03T08:02:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.245+0000 7f482effd700 1 -- 172.21.15.98:0/930937100 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3127384711 0 0) 0x7f48180035b0 con 0x7f48301609b0 2023-03-03T08:02:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.245+0000 7f482effd700 1 -- 172.21.15.98:0/930937100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48300b4aa0 msgr2=0x7f48300b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.245+0000 7f482effd700 1 --2- 172.21.15.98:0/930937100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48300b4aa0 0x7f48300b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.245+0000 7f482effd700 1 -- 172.21.15.98:0/930937100 >> v1:172.21.15.100:6789/0 conn(0x7f48300cf570 legacy=0x7f483015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.245+0000 7f482effd700 1 -- 172.21.15.98:0/930937100 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48300cda60 con 0x7f48301609b0 2023-03-03T08:02:28.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.245+0000 7f482effd700 1 -- 172.21.15.98:0/930937100 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f4820001670 con 0x7f48301609b0 2023-03-03T08:02:28.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.245+0000 7f482effd700 1 -- 172.21.15.98:0/930937100 >> v1:172.21.15.98:6789/0 conn(0x7f48301609b0 legacy=0x7f4830160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:28.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.245+0000 7f482effd700 1 --2- 172.21.15.98:0/930937100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48300b4aa0 0x7f4818003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:28.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.246+0000 7f482effd700 1 --2- 172.21.15.98:0/930937100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4818003f70 0x7f4818004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:28.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.246+0000 7f482effd700 1 --2- 172.21.15.98:0/930937100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4818004d90 0x7f4818005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:28.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.246+0000 7f482effd700 1 -- 172.21.15.98:0/930937100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481800ba10 con 0x7f4818004d90 2023-03-03T08:02:28.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.246+0000 7f482effd700 1 -- 172.21.15.98:0/930937100 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481800bb80 con 0x7f4818003f70 2023-03-03T08:02:28.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.246+0000 7f482effd700 1 -- 172.21.15.98:0/930937100 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481800bd40 con 0x7f48300b4aa0 2023-03-03T08:02:28.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.246+0000 7f4834b9c700 1 --2- 172.21.15.98:0/930937100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4818004d90 0x7f4818005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:28.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.246+0000 7f482ffff700 1 --2- 172.21.15.98:0/930937100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48300b4aa0 0x7f4818003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:28.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.246+0000 7f482f7fe700 1 --2- 172.21.15.98:0/930937100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4818003f70 0x7f4818004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:28.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.246+0000 7f4834b9c700 1 -- 172.21.15.98:0/930937100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48300b4aa0 msgr2=0x7f4818003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:28.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.246+0000 7f4834b9c700 1 --2- 172.21.15.98:0/930937100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48300b4aa0 0x7f4818003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:28.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.246+0000 7f4834b9c700 1 -- 172.21.15.98:0/930937100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4818003f70 msgr2=0x7f4818004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:28.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.246+0000 7f4834b9c700 1 --2- 172.21.15.98:0/930937100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4818003f70 0x7f4818004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.246+0000 7f4834b9c700 1 -- 172.21.15.98:0/930937100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48180035b0 con 0x7f4818004d90 2023-03-03T08:02:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.246+0000 7f482f7fe700 1 --2- 172.21.15.98:0/930937100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4818003f70 0x7f4818004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.247+0000 7f4834b9c700 1 --2- 172.21.15.98:0/930937100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4818004d90 0x7f4818005200 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f4828009fc0 tx=0x7f4818005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf05d5ef542cca93 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.247+0000 7f482effd700 1 -- 172.21.15.98:0/930937100 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482800ec30 con 0x7f4818004d90 2023-03-03T08:02:28.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.247+0000 7f482effd700 1 -- 172.21.15.98:0/930937100 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482800f040 con 0x7f4818004d90 2023-03-03T08:02:28.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.247+0000 7f482effd700 1 -- 172.21.15.98:0/930937100 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4828014480 con 0x7f4818004d90 2023-03-03T08:02:28.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.247+0000 7f483539d700 1 -- 172.21.15.98:0/930937100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4818004d90 msgr2=0x7f4818005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:28.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.247+0000 7f483539d700 1 --2- 172.21.15.98:0/930937100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4818004d90 0x7f4818005200 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f4828009fc0 tx=0x7f4818005880 comp rx=0 tx=0).stop 2023-03-03T08:02:28.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.247+0000 7f4834b9c700 1 -- 172.21.15.98:0/930937100 reap_dead start 2023-03-03T08:02:28.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.247+0000 7f483539d700 1 -- 172.21.15.98:0/930937100 shutdown_connections 2023-03-03T08:02:28.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.247+0000 7f483539d700 1 -- 172.21.15.98:0/930937100 >> 172.21.15.98:0/930937100 conn(0x7f483006b880 msgr2=0x7f48300cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:28.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.248+0000 7f483539d700 1 -- 172.21.15.98:0/930937100 shutdown_connections 2023-03-03T08:02:28.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.248+0000 7f483539d700 1 -- 172.21.15.98:0/930937100 wait complete. 2023-03-03T08:02:28.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.250+0000 7f483539d700 1 Processor -- start 2023-03-03T08:02:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.250+0000 7f483539d700 1 -- start start 2023-03-03T08:02:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.251+0000 7f483539d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4818004d90 0x7f4830151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.251+0000 7f483539d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48301e5a90 0x7f48300cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.251+0000 7f483539d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48300ccc60 0x7f483015e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.251+0000 7f483539d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48300b8e30 con 0x7f4818004d90 2023-03-03T08:02:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.251+0000 7f483539d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48300cb140 con 0x7f48301e5a90 2023-03-03T08:02:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.251+0000 7f483539d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48301e77b0 con 0x7f48300ccc60 2023-03-03T08:02:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.251+0000 7f482ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4818004d90 0x7f4830151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.251+0000 7f482ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4818004d90 0x7f4830151e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43694/0 (socket says 172.21.15.98:43694) 2023-03-03T08:02:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.251+0000 7f482ffff700 1 -- 172.21.15.98:0/1263497714 learned_addr learned my addr 172.21.15.98:0/1263497714 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.251+0000 7f4834b9c700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48300ccc60 0x7f483015e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.251+0000 7f482f7fe700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48301e5a90 0x7f48300cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:28.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.251+0000 7f482ffff700 1 -- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48300ccc60 msgr2=0x7f483015e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:28.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.251+0000 7f482ffff700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48300ccc60 0x7f483015e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:28.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.252+0000 7f482ffff700 1 -- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48301e5a90 msgr2=0x7f48300cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.252+0000 7f482ffff700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48301e5a90 0x7f48300cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.252+0000 7f482ffff700 1 -- 172.21.15.98:0/1263497714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483015cdf0 con 0x7f4818004d90 2023-03-03T08:02:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.252+0000 7f4834b9c700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48300ccc60 0x7f483015e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.252+0000 7f482f7fe700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48301e5a90 0x7f48300cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.252+0000 7f482ffff700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4818004d90 0x7f4830151e30 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f482000e8c0 tx=0x7f482000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.252+0000 7f482cff9700 1 -- 172.21.15.98:0/1263497714 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482000ee60 con 0x7f4818004d90 2023-03-03T08:02:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.252+0000 7f482cff9700 1 -- 172.21.15.98:0/1263497714 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482000c780 con 0x7f4818004d90 2023-03-03T08:02:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.252+0000 7f483539d700 1 -- 172.21.15.98:0/1263497714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48300cdd40 con 0x7f4818004d90 2023-03-03T08:02:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.252+0000 7f482cff9700 1 -- 172.21.15.98:0/1263497714 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482000cb30 con 0x7f4818004d90 2023-03-03T08:02:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.252+0000 7f483539d700 1 -- 172.21.15.98:0/1263497714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483015d0d0 con 0x7f4818004d90 2023-03-03T08:02:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.254+0000 7f483539d700 1 -- 172.21.15.98:0/1263497714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48300cb2b0 con 0x7f4818004d90 2023-03-03T08:02:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.255+0000 7f482cff9700 1 -- 172.21.15.98:0/1263497714 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f482000cc90 con 0x7f4818004d90 2023-03-03T08:02:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.256+0000 7f482cff9700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f481c068380 0x7f481c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.256+0000 7f482cff9700 1 -- 172.21.15.98:0/1263497714 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4820085db0 con 0x7f4818004d90 2023-03-03T08:02:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.256+0000 7f482f7fe700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f481c068380 0x7f481c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.257+0000 7f482f7fe700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f481c068380 0x7f481c06a830 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f48300cc4d0 tx=0x7f4824009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.259+0000 7f482cff9700 1 -- 172.21.15.98:0/1263497714 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f48300cb2b0 con 0x7f4818004d90 2023-03-03T08:02:28.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.523+0000 7f483539d700 1 -- 172.21.15.98:0/1263497714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7f48300cb2b0 con 0x7f4818004d90 2023-03-03T08:02:28.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.528+0000 7f482cff9700 1 -- 172.21.15.98:0/1263497714 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 184+0+0 (secure 0 0 0) 0x7f48300cb2b0 con 0x7f4818004d90 2023-03-03T08:02:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.531+0000 7f483539d700 1 -- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f481c068380 msgr2=0x7f481c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.531+0000 7f483539d700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f481c068380 0x7f481c06a830 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f48300cc4d0 tx=0x7f4824009040 comp rx=0 tx=0).stop 2023-03-03T08:02:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.531+0000 7f483539d700 1 -- 172.21.15.98:0/1263497714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4818004d90 msgr2=0x7f4830151e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.531+0000 7f483539d700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4818004d90 0x7f4830151e30 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f482000e8c0 tx=0x7f482000ec80 comp rx=0 tx=0).stop 2023-03-03T08:02:28.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.531+0000 7f483539d700 1 -- 172.21.15.98:0/1263497714 shutdown_connections 2023-03-03T08:02:28.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.531+0000 7f483539d700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f481c068380 0x7f481c06a830 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:28.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.531+0000 7f483539d700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4818004d90 0x7f4830151e30 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:28.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.531+0000 7f483539d700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48300ccc60 0x7f483015e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:28.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.531+0000 7f483539d700 1 --2- 172.21.15.98:0/1263497714 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48301e5a90 0x7f48300cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:28.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.531+0000 7f483539d700 1 -- 172.21.15.98:0/1263497714 >> 172.21.15.98:0/1263497714 conn(0x7f483006b880 msgr2=0x7f48300d3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:28.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.531+0000 7f483539d700 1 -- 172.21.15.98:0/1263497714 shutdown_connections 2023-03-03T08:02:28.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.531+0000 7f483539d700 1 -- 172.21.15.98:0/1263497714 wait complete. 2023-03-03T08:02:28.534 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.0 2023-03-03T08:02:28.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:02:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.802+0000 7f08573ff700 1 Processor -- start 2023-03-03T08:02:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.803+0000 7f08573ff700 1 -- start start 2023-03-03T08:02:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.804+0000 7f08573ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08500713e0 0x7f08500717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.804+0000 7f08573ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f085015e490 0x7f085015ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.804+0000 7f08573ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0850071cf0 con 0x7f085015b2a0 2023-03-03T08:02:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.804+0000 7f08573ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f085015b760 con 0x7f085015e490 2023-03-03T08:02:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.804+0000 7f08573ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f085015ba40 con 0x7f08500713e0 2023-03-03T08:02:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.804+0000 7f085499a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f085015e490 0x7f085015ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.805+0000 7f085599c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f085015b2a0 0x7f085006c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34278/0 (socket says 172.21.15.98:34278) 2023-03-03T08:02:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.805+0000 7f085599c700 1 -- 172.21.15.98:0/3170314411 learned_addr learned my addr 172.21.15.98:0/3170314411 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.805+0000 7f085519b700 1 --2- 172.21.15.98:0/3170314411 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08500713e0 0x7f08500717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.805+0000 7f085499a700 1 -- 172.21.15.98:0/3170314411 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08500713e0 msgr2=0x7f08500717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.805+0000 7f085499a700 1 --2- 172.21.15.98:0/3170314411 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08500713e0 0x7f08500717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.805+0000 7f085499a700 1 -- 172.21.15.98:0/3170314411 >> v1:172.21.15.98:6789/0 conn(0x7f085015b2a0 legacy=0x7f085006c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.805+0000 7f085499a700 1 -- 172.21.15.98:0/3170314411 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f085015bd20 con 0x7f085015e490 2023-03-03T08:02:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.805+0000 7f085519b700 1 --2- 172.21.15.98:0/3170314411 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08500713e0 0x7f08500717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.805+0000 7f085499a700 1 --2- 172.21.15.98:0/3170314411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085015e490 0x7f085015ad60 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f0840009830 tx=0x7f0840009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3693ffe702bbe90e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.806+0000 7f0847fff700 1 -- 172.21.15.98:0/3170314411 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840011830 con 0x7f085015e490 2023-03-03T08:02:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.806+0000 7f0847fff700 1 -- 172.21.15.98:0/3170314411 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0840011990 con 0x7f085015e490 2023-03-03T08:02:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.806+0000 7f0847fff700 1 -- 172.21.15.98:0/3170314411 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840011d40 con 0x7f085015e490 2023-03-03T08:02:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.806+0000 7f08573ff700 1 -- 172.21.15.98:0/3170314411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085015e490 msgr2=0x7f085015ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.806+0000 7f08573ff700 1 --2- 172.21.15.98:0/3170314411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085015e490 0x7f085015ad60 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f0840009830 tx=0x7f0840009b40 comp rx=0 tx=0).stop 2023-03-03T08:02:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.806+0000 7f08573ff700 1 -- 172.21.15.98:0/3170314411 shutdown_connections 2023-03-03T08:02:28.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.806+0000 7f08573ff700 1 --2- 172.21.15.98:0/3170314411 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08500713e0 0x7f08500717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:28.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.806+0000 7f08573ff700 1 --2- 172.21.15.98:0/3170314411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085015e490 0x7f085015ad60 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:28.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.806+0000 7f08573ff700 1 -- 172.21.15.98:0/3170314411 >> 172.21.15.98:0/3170314411 conn(0x7f08500b09f0 msgr2=0x7f08500b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:28.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.806+0000 7f08573ff700 1 -- 172.21.15.98:0/3170314411 shutdown_connections 2023-03-03T08:02:28.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.807+0000 7f08573ff700 1 -- 172.21.15.98:0/3170314411 wait complete. 2023-03-03T08:02:28.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.808+0000 7f08573ff700 1 Processor -- start 2023-03-03T08:02:28.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.809+0000 7f08573ff700 1 -- start start 2023-03-03T08:02:28.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.810+0000 7f08573ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08500713e0 0x7f0850145170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:28.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.810+0000 7f08573ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f085015b2a0 0x7f085015d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:28.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.810+0000 7f08573ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085015e490 0x7f08500c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:28.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.810+0000 7f08573ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08500cc9d0 con 0x7f085015e490 2023-03-03T08:02:28.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.810+0000 7f08573ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08500cd830 con 0x7f085015b2a0 2023-03-03T08:02:28.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.810+0000 7f08573ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08500ccc20 con 0x7f08500713e0 2023-03-03T08:02:28.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.810+0000 7f085599c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085015e490 0x7f08500c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:28.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.810+0000 7f085599c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085015e490 0x7f08500c9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43710/0 (socket says 172.21.15.98:43710) 2023-03-03T08:02:28.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.810+0000 7f085599c700 1 -- 172.21.15.98:0/3637783055 learned_addr learned my addr 172.21.15.98:0/3637783055 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:28.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.810+0000 7f085519b700 1 --2- 172.21.15.98:0/3637783055 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08500713e0 0x7f0850145170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:28.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.810+0000 7f085499a700 1 --2- 172.21.15.98:0/3637783055 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f085015b2a0 0x7f085015d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:28.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.811+0000 7f085599c700 1 -- 172.21.15.98:0/3637783055 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08500713e0 msgr2=0x7f0850145170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:28.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.811+0000 7f085599c700 1 --2- 172.21.15.98:0/3637783055 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08500713e0 0x7f0850145170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:28.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.811+0000 7f085599c700 1 -- 172.21.15.98:0/3637783055 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f085015b2a0 msgr2=0x7f085015d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:28.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.811+0000 7f085599c700 1 --2- 172.21.15.98:0/3637783055 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f085015b2a0 0x7f085015d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:28.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.811+0000 7f085599c700 1 -- 172.21.15.98:0/3637783055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0840009440 con 0x7f085015e490 2023-03-03T08:02:28.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.811+0000 7f085499a700 1 --2- 172.21.15.98:0/3637783055 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f085015b2a0 0x7f085015d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:28.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.811+0000 7f085599c700 1 --2- 172.21.15.98:0/3637783055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085015e490 0x7f08500c9430 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f084800cb20 tx=0x7f084800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:28.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.811+0000 7f0845ffb700 1 -- 172.21.15.98:0/3637783055 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084800aa60 con 0x7f085015e490 2023-03-03T08:02:28.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.811+0000 7f0845ffb700 1 -- 172.21.15.98:0/3637783055 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084800abc0 con 0x7f085015e490 2023-03-03T08:02:28.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.811+0000 7f0845ffb700 1 -- 172.21.15.98:0/3637783055 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848012400 con 0x7f085015e490 2023-03-03T08:02:28.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.811+0000 7f08573ff700 1 -- 172.21.15.98:0/3637783055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f085006c2d0 con 0x7f085015e490 2023-03-03T08:02:28.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.811+0000 7f08573ff700 1 -- 172.21.15.98:0/3637783055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f085006bff0 con 0x7f085015e490 2023-03-03T08:02:28.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.813+0000 7f08573ff700 1 -- 172.21.15.98:0/3637783055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085006cf00 con 0x7f085015e490 2023-03-03T08:02:28.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.814+0000 7f0845ffb700 1 -- 172.21.15.98:0/3637783055 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f0848019030 con 0x7f085015e490 2023-03-03T08:02:28.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.816+0000 7f0845ffb700 1 --2- 172.21.15.98:0/3637783055 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0834068380 0x7f083406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:28.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.816+0000 7f0845ffb700 1 -- 172.21.15.98:0/3637783055 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0848085880 con 0x7f085015e490 2023-03-03T08:02:28.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.816+0000 7f085519b700 1 --2- 172.21.15.98:0/3637783055 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0834068380 0x7f083406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:28.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.817+0000 7f085519b700 1 --2- 172.21.15.98:0/3637783055 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0834068380 0x7f083406a830 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f084c00c940 tx=0x7f084c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:28.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:28.820+0000 7f0845ffb700 1 -- 172.21.15.98:0/3637783055 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f084804b5e0 con 0x7f085015e490 2023-03-03T08:02:29.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.053+0000 7f08573ff700 1 -- 172.21.15.98:0/3637783055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f085004e6e0 con 0x7f085015e490 2023-03-03T08:02:29.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.054+0000 7f0845ffb700 1 -- 172.21.15.98:0/3637783055 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4653 (secure 0 0 0) 0x7f0848017090 con 0x7f085015e490 2023-03-03T08:02:29.056 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:29.056 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2561725713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2561725713},{"type":"v1","addr":"172.21.15.98:6834","nonce":2561725713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1511797280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1511797280},{"type":"v1","addr":"172.21.15.100:6834","nonce":1511797280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/1843752855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":1843752855},{"type":"v1","addr":"172.21.15.100:6835","nonce":1843752855}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:02:22.404483+0000","modified":"2023-03-03T08:02:25.872537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7524":{"gid":7524,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6835/341265634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":341265634},{"type":"v1","addr":"172.21.15.98:6835","nonce":341265634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-03-03T08:02:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.056+0000 7f08573ff700 1 -- 172.21.15.98:0/3637783055 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0834068380 msgr2=0x7f083406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.056+0000 7f08573ff700 1 --2- 172.21.15.98:0/3637783055 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0834068380 0x7f083406a830 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f084c00c940 tx=0x7f084c008040 comp rx=0 tx=0).stop 2023-03-03T08:02:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.056+0000 7f08573ff700 1 -- 172.21.15.98:0/3637783055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085015e490 msgr2=0x7f08500c9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.056+0000 7f08573ff700 1 --2- 172.21.15.98:0/3637783055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085015e490 0x7f08500c9430 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f084800cb20 tx=0x7f084800ce30 comp rx=0 tx=0).stop 2023-03-03T08:02:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.057+0000 7f08573ff700 1 -- 172.21.15.98:0/3637783055 shutdown_connections 2023-03-03T08:02:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.057+0000 7f08573ff700 1 --2- 172.21.15.98:0/3637783055 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0834068380 0x7f083406a830 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.057+0000 7f08573ff700 1 --2- 172.21.15.98:0/3637783055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085015e490 0x7f08500c9430 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.057+0000 7f08573ff700 1 --2- 172.21.15.98:0/3637783055 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08500713e0 0x7f0850145170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.057+0000 7f08573ff700 1 --2- 172.21.15.98:0/3637783055 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f085015b2a0 0x7f085015d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.057+0000 7f08573ff700 1 -- 172.21.15.98:0/3637783055 >> 172.21.15.98:0/3637783055 conn(0x7f08500b09f0 msgr2=0x7f085006c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.057+0000 7f08573ff700 1 -- 172.21.15.98:0/3637783055 shutdown_connections 2023-03-03T08:02:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.057+0000 7f08573ff700 1 -- 172.21.15.98:0/3637783055 wait complete. 2023-03-03T08:02:29.060 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 48 2023-03-03T08:02:29.488 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-03-03T08:02:22.404483+0000', 'modified': '2023-03-03T08:02:25.872537+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7524}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7524': {'gid': 7524, 'name': 'c', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.98:6835/341265634', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6833', 'nonce': 341265634}, {'type': 'v1', 'addr': '172.21.15.98:6835', 'nonce': 341265634}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-03-03T08:02:29.489 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-03-03T08:02:29.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-03-03T08:02:29.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.732+0000 7f1e115ff700 1 Processor -- start 2023-03-03T08:02:29.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.733+0000 7f1e115ff700 1 -- start start 2023-03-03T08:02:29.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.734+0000 7f1e115ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1e0c15ca70 0x7f1e0c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.734+0000 7f1e115ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0c0cf980 con 0x7f1e0c06b4b0 2023-03-03T08:02:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.734+0000 7f1e115ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0c0d1800 con 0x7f1e0c0d0270 2023-03-03T08:02:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.734+0000 7f1e115ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e0c0cd5b0 con 0x7f1e0c15ca70 2023-03-03T08:02:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.734+0000 7f1e0a7fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1e0c06b4b0 0x7f1e0c0cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34292/0 (socket says 172.21.15.98:34292) 2023-03-03T08:02:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.734+0000 7f1e0a7fc700 1 -- 172.21.15.98:0/705849643 learned_addr learned my addr 172.21.15.98:0/705849643 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.735+0000 7f1e0affd700 1 --2- 172.21.15.98:0/705849643 >> v2:172.21.15.100:3300/0 conn(0x7f1e0c15ca70 0x7f1e0c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.735+0000 7f1e09ffb700 1 -- 172.21.15.98:0/705849643 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682069679 0 0) 0x7f1e0c0cf980 con 0x7f1e0c06b4b0 2023-03-03T08:02:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.735+0000 7f1e09ffb700 1 -- 172.21.15.98:0/705849643 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1df40035b0 con 0x7f1e0c06b4b0 2023-03-03T08:02:29.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.735+0000 7f1e09ffb700 1 -- 172.21.15.98:0/705849643 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287864863 0 0) 0x7f1e0c0d1800 con 0x7f1e0c0d0270 2023-03-03T08:02:29.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.735+0000 7f1e09ffb700 1 -- 172.21.15.98:0/705849643 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e0c0cf980 con 0x7f1e0c0d0270 2023-03-03T08:02:29.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.735+0000 7f1e09ffb700 1 -- 172.21.15.98:0/705849643 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1515654975 0 0) 0x7f1df40035b0 con 0x7f1e0c06b4b0 2023-03-03T08:02:29.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.735+0000 7f1e0affd700 1 -- 172.21.15.98:0/705849643 >> v1:172.21.15.100:6789/0 conn(0x7f1e0c0d0270 legacy=0x7f1e0c0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:29.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.735+0000 7f1e0affd700 1 -- 172.21.15.98:0/705849643 >> v1:172.21.15.98:6789/0 conn(0x7f1e0c06b4b0 legacy=0x7f1e0c0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:29.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.735+0000 7f1e0affd700 1 -- 172.21.15.98:0/705849643 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c0cd900 con 0x7f1e0c15ca70 2023-03-03T08:02:29.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.736+0000 7f1e0affd700 1 --2- 172.21.15.98:0/705849643 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e0c15ca70 0x7f1e0c0cf440 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009840 tx=0x7f1dfc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f2ac27763dfbeaa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:29.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.736+0000 7f1e09ffb700 1 -- 172.21.15.98:0/705849643 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e00001670 con 0x7f1e0c15ca70 2023-03-03T08:02:29.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.736+0000 7f1e09ffb700 1 -- 172.21.15.98:0/705849643 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dfc004030 con 0x7f1e0c15ca70 2023-03-03T08:02:29.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.736+0000 7f1e09ffb700 1 -- 172.21.15.98:0/705849643 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc0043c0 con 0x7f1e0c15ca70 2023-03-03T08:02:29.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.736+0000 7f1e115ff700 1 -- 172.21.15.98:0/705849643 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e0c15ca70 msgr2=0x7f1e0c0cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:29.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.736+0000 7f1e115ff700 1 --2- 172.21.15.98:0/705849643 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e0c15ca70 0x7f1e0c0cf440 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009840 tx=0x7f1dfc009b50 comp rx=0 tx=0).stop 2023-03-03T08:02:29.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.737+0000 7f1e115ff700 1 -- 172.21.15.98:0/705849643 shutdown_connections 2023-03-03T08:02:29.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.737+0000 7f1e115ff700 1 --2- 172.21.15.98:0/705849643 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e0c15ca70 0x7f1e0c0cf440 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.737+0000 7f1e115ff700 1 -- 172.21.15.98:0/705849643 >> 172.21.15.98:0/705849643 conn(0x7f1e0c073e70 msgr2=0x7f1e0c06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:29.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.737+0000 7f1e115ff700 1 -- 172.21.15.98:0/705849643 shutdown_connections 2023-03-03T08:02:29.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.737+0000 7f1e115ff700 1 -- 172.21.15.98:0/705849643 wait complete. 2023-03-03T08:02:29.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.739+0000 7f1e115ff700 1 Processor -- start 2023-03-03T08:02:29.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.739+0000 7f1e115ff700 1 -- start start 2023-03-03T08:02:29.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.740+0000 7f1e115ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e0c06b4b0 0x7f1e0c150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.740+0000 7f1e115ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e0c0d0270 0x7f1e0c15e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.740+0000 7f1e115ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0c15ca70 0x7f1e0c0b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.740+0000 7f1e115ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c15d1c0 con 0x7f1e0c15ca70 2023-03-03T08:02:29.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.740+0000 7f1e115ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c15eab0 con 0x7f1e0c06b4b0 2023-03-03T08:02:29.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.740+0000 7f1e115ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c0ca2f0 con 0x7f1e0c0d0270 2023-03-03T08:02:29.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.740+0000 7f1e0b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0c15ca70 0x7f1e0c0b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.740+0000 7f1e0b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0c15ca70 0x7f1e0c0b3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43712/0 (socket says 172.21.15.98:43712) 2023-03-03T08:02:29.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.740+0000 7f1e0b7fe700 1 -- 172.21.15.98:0/3222188898 learned_addr learned my addr 172.21.15.98:0/3222188898 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:29.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.740+0000 7f1e0affd700 1 --2- 172.21.15.98:0/3222188898 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e0c06b4b0 0x7f1e0c150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.740+0000 7f1e0a7fc700 1 --2- 172.21.15.98:0/3222188898 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e0c0d0270 0x7f1e0c15e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.741+0000 7f1e0b7fe700 1 -- 172.21.15.98:0/3222188898 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e0c0d0270 msgr2=0x7f1e0c15e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:29.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.741+0000 7f1e0b7fe700 1 --2- 172.21.15.98:0/3222188898 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e0c0d0270 0x7f1e0c15e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.741+0000 7f1e0b7fe700 1 -- 172.21.15.98:0/3222188898 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e0c06b4b0 msgr2=0x7f1e0c150f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:29.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.741+0000 7f1e0b7fe700 1 --2- 172.21.15.98:0/3222188898 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e0c06b4b0 0x7f1e0c150f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.741+0000 7f1e0b7fe700 1 -- 172.21.15.98:0/3222188898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dfc0094a0 con 0x7f1e0c15ca70 2023-03-03T08:02:29.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.741+0000 7f1e0a7fc700 1 --2- 172.21.15.98:0/3222188898 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e0c0d0270 0x7f1e0c15e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:29.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.741+0000 7f1e0b7fe700 1 --2- 172.21.15.98:0/3222188898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0c15ca70 0x7f1e0c0b3f10 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400cbc0 tx=0x7f1e0400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:29.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.741+0000 7f1df37fe700 1 -- 172.21.15.98:0/3222188898 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e04003c00 con 0x7f1e0c15ca70 2023-03-03T08:02:29.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.741+0000 7f1df37fe700 1 -- 172.21.15.98:0/3222188898 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e04003d80 con 0x7f1e0c15ca70 2023-03-03T08:02:29.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.741+0000 7f1df37fe700 1 -- 172.21.15.98:0/3222188898 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0400ab40 con 0x7f1e0c15ca70 2023-03-03T08:02:29.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.741+0000 7f1e115ff700 1 -- 172.21.15.98:0/3222188898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e0c0d1aa0 con 0x7f1e0c15ca70 2023-03-03T08:02:29.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.742+0000 7f1e115ff700 1 -- 172.21.15.98:0/3222188898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0c0cdbe0 con 0x7f1e0c15ca70 2023-03-03T08:02:29.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.744+0000 7f1df37fe700 1 -- 172.21.15.98:0/3222188898 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f1e04010050 con 0x7f1e0c15ca70 2023-03-03T08:02:29.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.746+0000 7f1df37fe700 1 --2- 172.21.15.98:0/3222188898 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1df8068380 0x7f1df806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.747+0000 7f1df37fe700 1 -- 172.21.15.98:0/3222188898 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1e04085c70 con 0x7f1e0c15ca70 2023-03-03T08:02:29.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.747+0000 7f1e0affd700 1 --2- 172.21.15.98:0/3222188898 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1df8068380 0x7f1df806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.748+0000 7f1e0affd700 1 --2- 172.21.15.98:0/3222188898 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1df8068380 0x7f1df806a830 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0090b0 tx=0x7f1dfc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:29.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.748+0000 7f1e24f9bb80 1 Processor -- start 2023-03-03T08:02:29.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.749+0000 7f1e24f9bb80 1 -- start start 2023-03-03T08:02:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e24f9bb80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e05a70 0x559ac6e05e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e24f9bb80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e04cd0 0x559ac6e54030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e24f9bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e54570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e24f9bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac6d7b4f0 con 0x559ac6e00560 2023-03-03T08:02:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e24f9bb80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac6cf0b60 con 0x559ac6e04cd0 2023-03-03T08:02:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e24f9bb80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ac6d5fad0 con 0x559ac6e05a70 2023-03-03T08:02:29.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e0b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e54570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e0b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e54570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43720/0 (socket says 172.21.15.98:43720) 2023-03-03T08:02:29.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e0b7fe700 1 -- 172.21.15.98:0/4172363079 learned_addr learned my addr 172.21.15.98:0/4172363079 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:29.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e0a7fc700 1 --2- 172.21.15.98:0/4172363079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e04cd0 0x559ac6e54030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e0affd700 1 --2- 172.21.15.98:0/4172363079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e05a70 0x559ac6e05e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e0b7fe700 1 -- 172.21.15.98:0/4172363079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e05a70 msgr2=0x559ac6e05e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e0b7fe700 1 --2- 172.21.15.98:0/4172363079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e05a70 0x559ac6e05e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e0b7fe700 1 -- 172.21.15.98:0/4172363079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e04cd0 msgr2=0x559ac6e54030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e0b7fe700 1 --2- 172.21.15.98:0/4172363079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e04cd0 0x559ac6e54030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e0b7fe700 1 -- 172.21.15.98:0/4172363079 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ac6b4d540 con 0x559ac6e00560 2023-03-03T08:02:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.750+0000 7f1e0a7fc700 1 --2- 172.21.15.98:0/4172363079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e04cd0 0x559ac6e54030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.751+0000 7f1e0b7fe700 1 --2- 172.21.15.98:0/4172363079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e54570 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f1e04084240 tx=0x7f1e0405e2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=774ecb88ceefb33a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:29.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.751+0000 7f1df0ff9700 1 -- 172.21.15.98:0/4172363079 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e04055540 con 0x559ac6e00560 2023-03-03T08:02:29.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.751+0000 7f1df0ff9700 1 -- 172.21.15.98:0/4172363079 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e040556a0 con 0x559ac6e00560 2023-03-03T08:02:29.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.751+0000 7f1df0ff9700 1 -- 172.21.15.98:0/4172363079 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e04055810 con 0x559ac6e00560 2023-03-03T08:02:29.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.752+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/4172363079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 msgr2=0x559ac6e54570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:29.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.752+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/4172363079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e54570 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f1e04084240 tx=0x7f1e0405e2c0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.752+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/4172363079 shutdown_connections 2023-03-03T08:02:29.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.752+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/4172363079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e54570 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.752+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/4172363079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e05a70 0x559ac6e05e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.752+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/4172363079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e04cd0 0x559ac6e54030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.752+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/4172363079 >> 172.21.15.98:0/4172363079 conn(0x559ac6e03c30 msgr2=0x559ac6e045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:29.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.752+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/4172363079 shutdown_connections 2023-03-03T08:02:29.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.752+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/4172363079 wait complete. 2023-03-03T08:02:29.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.753+0000 7f1e24f9bb80 1 -- start start 2023-03-03T08:02:29.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.753+0000 7f1e24f9bb80 1 Processor -- start 2023-03-03T08:02:29.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.754+0000 7f1e24f9bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e5d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.754+0000 7f1e24f9bb80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e04cd0 0x559ac6e31fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.754+0000 7f1e24f9bb80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e05a70 0x559ac6e48110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.754+0000 7f1e24f9bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac6d58e40 con 0x559ac6e00560 2023-03-03T08:02:29.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.754+0000 7f1e24f9bb80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac6b4e140 con 0x559ac6e05a70 2023-03-03T08:02:29.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.754+0000 7f1e24f9bb80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ac6e5bff0 con 0x559ac6e04cd0 2023-03-03T08:02:29.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.754+0000 7f1e24f9bb80 10 client.0 _ll_register_callbacks cb 0x559ac6cf71c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T08:02:29.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.755+0000 7f1e0affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e5d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.755+0000 7f1e0affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e5d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43722/0 (socket says 172.21.15.98:43722) 2023-03-03T08:02:29.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.755+0000 7f1e0affd700 1 -- 172.21.15.98:0/2814915847 learned_addr learned my addr 172.21.15.98:0/2814915847 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:29.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.755+0000 7f1e0a7fc700 1 --2- 172.21.15.98:0/2814915847 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e04cd0 0x559ac6e31fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.755+0000 7f1e0b7fe700 1 --2- 172.21.15.98:0/2814915847 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e05a70 0x559ac6e48110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.755+0000 7f1e0affd700 1 -- 172.21.15.98:0/2814915847 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e04cd0 msgr2=0x559ac6e31fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.755+0000 7f1e0affd700 1 --2- 172.21.15.98:0/2814915847 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e04cd0 0x559ac6e31fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.755+0000 7f1e0affd700 1 -- 172.21.15.98:0/2814915847 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e05a70 msgr2=0x559ac6e48110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.755+0000 7f1e0affd700 1 --2- 172.21.15.98:0/2814915847 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e05a70 0x559ac6e48110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.755+0000 7f1e0affd700 1 -- 172.21.15.98:0/2814915847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0c0cdbe0 con 0x559ac6e00560 2023-03-03T08:02:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.755+0000 7f1e0affd700 1 --2- 172.21.15.98:0/2814915847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e5d8e0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009ea0 tx=0x7f1dfc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef8debe0deca276 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.755+0000 7f1de9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-03-03T08:02:29.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.756+0000 7f1de9ffb700 1 -- 172.21.15.98:0/2814915847 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc02d070 con 0x559ac6e00560 2023-03-03T08:02:29.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.756+0000 7f1de9ffb700 1 -- 172.21.15.98:0/2814915847 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1dfc011630 con 0x559ac6e00560 2023-03-03T08:02:29.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.756+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2814915847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559ac6df8c50 con 0x559ac6e00560 2023-03-03T08:02:29.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.756+0000 7f1de9ffb700 1 -- 172.21.15.98:0/2814915847 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1dfc020070 con 0x559ac6e00560 2023-03-03T08:02:29.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.756+0000 7f1e24f9bb80 10 client.7668 fetch_fsmap learned FSMap version 48 2023-03-03T08:02:29.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.756+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2814915847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x559ac6b87b10 con 0x559ac6e00560 2023-03-03T08:02:29.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.756+0000 7f1de9ffb700 1 -- 172.21.15.98:0/2814915847 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559ac6df8c50 con 0x559ac6e00560 2023-03-03T08:02:29.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.756+0000 7f1de9ffb700 1 -- 172.21.15.98:0/2814915847 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 48) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f1dfc0117f0 con 0x559ac6e00560 2023-03-03T08:02:29.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.756+0000 7f1e24f9bb80 10 client.7668 fetch_fsmap finished waiting for FSMap version 48 2023-03-03T08:02:29.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.756+0000 7f1e24f9bb80 10 client.7668 resolve_mds: resolved name 'c' to daemon mds.c 2023-03-03T08:02:29.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.757+0000 7f1e24f9bb80 20 client.7668 populate_metadata read hostname 'smithi098' 2023-03-03T08:02:29.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.757+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/2814915847 >> [v2:172.21.15.98:6833/341265634,v1:172.21.15.98:6835/341265634] conn(0x559ac6e4e3f0 0x559ac6e4c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.757+0000 7f1e24f9bb80 4 client.7668 mds_command: new command op to 7524 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-03T08:02:29.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.757+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2814915847 --> [v2:172.21.15.98:6833/341265634,v1:172.21.15.98:6835/341265634] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559ac6d52020 con 0x559ac6e4e3f0 2023-03-03T08:02:29.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.757+0000 7f1e0a7fc700 1 --2- 172.21.15.98:0/2814915847 >> [v2:172.21.15.98:6833/341265634,v1:172.21.15.98:6835/341265634] conn(0x559ac6e4e3f0 0x559ac6e4c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.757+0000 7f1de9ffb700 10 client.7668 ms_handle_connect on v2:172.21.15.98:6833/341265634 2023-03-03T08:02:29.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.757+0000 7f1e0a7fc700 1 --2- 172.21.15.98:0/2814915847 >> [v2:172.21.15.98:6833/341265634,v1:172.21.15.98:6835/341265634] conn(0x559ac6e4e3f0 0x559ac6e4c550 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=756dac6f10f265c8 server_cookie=e38d0b6c73a18e69 in_seq=0 out_seq=0 2023-03-03T08:02:29.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.758+0000 7f1de9ffb700 1 -- 172.21.15.98:0/2814915847 <== mds.0 v2:172.21.15.98:6833/341265634 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x559ac6d52020 con 0x559ac6e4e3f0 2023-03-03T08:02:29.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.758+0000 7f1de9ffb700 10 client.7668 handle_command_reply: tid=0 2023-03-03T08:02:29.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.758+0000 7f1e24f9bb80 1 client.7668 shutdown 2023-03-03T08:02:29.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.759+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2814915847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 msgr2=0x559ac6e5d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:29.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.759+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/2814915847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e5d8e0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009ea0 tx=0x7f1dfc0047c0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.759+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2814915847 shutdown_connections 2023-03-03T08:02:29.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.759+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/2814915847 >> [v2:172.21.15.98:6833/341265634,v1:172.21.15.98:6835/341265634] conn(0x559ac6e4e3f0 0x559ac6e4c550 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.759+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/2814915847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e5d8e0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.759+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/2814915847 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e05a70 0x559ac6e48110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.759+0000 7f1de9ffb700 0 client.7668 ms_handle_reset on v2:172.21.15.98:6833/341265634 2023-03-03T08:02:29.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.759+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/2814915847 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e04cd0 0x559ac6e31fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.759+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2814915847 >> 172.21.15.98:0/2814915847 conn(0x559ac6e03c30 msgr2=0x559ac6e063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:29.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.759+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2814915847 shutdown_connections 2023-03-03T08:02:29.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.759+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2814915847 wait complete. 2023-03-03T08:02:29.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.759+0000 7f1e24f9bb80 20 client.7668 trim_cache size 0 max 16384 2023-03-03T08:02:29.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.777+0000 7f1e24f9bb80 1 Processor -- start 2023-03-03T08:02:29.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.777+0000 7f1e24f9bb80 1 -- start start 2023-03-03T08:02:29.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.778+0000 7f1e24f9bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e2dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.778+0000 7f1e24f9bb80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e04cd0 0x559ac6e5c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.778+0000 7f1e24f9bb80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e05a70 0x559ac6eb5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.778+0000 7f1e24f9bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac6e5d090 con 0x559ac6e00560 2023-03-03T08:02:29.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.778+0000 7f1e24f9bb80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac6e06620 con 0x559ac6e04cd0 2023-03-03T08:02:29.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.778+0000 7f1e24f9bb80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ac6e06db0 con 0x559ac6e05a70 2023-03-03T08:02:29.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.778+0000 7f1e0affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e2dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.778+0000 7f1e0affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e2dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43736/0 (socket says 172.21.15.98:43736) 2023-03-03T08:02:29.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.779+0000 7f1e0affd700 1 -- 172.21.15.98:0/933413876 learned_addr learned my addr 172.21.15.98:0/933413876 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:29.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.779+0000 7f1e0affd700 1 -- 172.21.15.98:0/933413876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e05a70 msgr2=0x559ac6eb5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:29.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.779+0000 7f1e0b7fe700 1 --2- 172.21.15.98:0/933413876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e05a70 0x559ac6eb5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.779+0000 7f1e0a7fc700 1 --2- 172.21.15.98:0/933413876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e04cd0 0x559ac6e5c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.779+0000 7f1e0affd700 1 --2- 172.21.15.98:0/933413876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e05a70 0x559ac6eb5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.779+0000 7f1e0affd700 1 -- 172.21.15.98:0/933413876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e04cd0 msgr2=0x559ac6e5c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:29.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.779+0000 7f1e0affd700 1 --2- 172.21.15.98:0/933413876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e04cd0 0x559ac6e5c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.779+0000 7f1e0affd700 1 -- 172.21.15.98:0/933413876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ac6d20230 con 0x559ac6e00560 2023-03-03T08:02:29.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.779+0000 7f1e0a7fc700 1 --2- 172.21.15.98:0/933413876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e04cd0 0x559ac6e5c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:29.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.779+0000 7f1e0affd700 1 --2- 172.21.15.98:0/933413876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e2dea0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc004310 tx=0x7f1dfc02d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1863e8a1acec7f80 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:29.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.779+0000 7f1df17fa700 1 -- 172.21.15.98:0/933413876 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc020040 con 0x559ac6e00560 2023-03-03T08:02:29.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.779+0000 7f1df17fa700 1 -- 172.21.15.98:0/933413876 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dfc006e00 con 0x559ac6e00560 2023-03-03T08:02:29.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.779+0000 7f1df17fa700 1 -- 172.21.15.98:0/933413876 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc011870 con 0x559ac6e00560 2023-03-03T08:02:29.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.780+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/933413876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 msgr2=0x559ac6e2dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:29.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.780+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/933413876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e2dea0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc004310 tx=0x7f1dfc02d000 comp rx=0 tx=0).stop 2023-03-03T08:02:29.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.780+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/933413876 shutdown_connections 2023-03-03T08:02:29.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.780+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/933413876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e00560 0x559ac6e2dea0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.780+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/933413876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e05a70 0x559ac6eb5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.780+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/933413876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e04cd0 0x559ac6e5c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.780+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/933413876 >> 172.21.15.98:0/933413876 conn(0x559ac6e03c30 msgr2=0x559ac6e04090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:29.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.780+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/933413876 shutdown_connections 2023-03-03T08:02:29.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.780+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/933413876 wait complete. 2023-03-03T08:02:29.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.781+0000 7f1e24f9bb80 1 -- start start 2023-03-03T08:02:29.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.781+0000 7f1e24f9bb80 1 Processor -- start 2023-03-03T08:02:29.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.782+0000 7f1e24f9bb80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e00560 0x559ac6e2c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.782+0000 7f1e24f9bb80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e04cd0 0x559ac6dff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.782+0000 7f1e24f9bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e05a70 0x559ac6dff800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.782+0000 7f1e24f9bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac6e4ae90 con 0x559ac6e05a70 2023-03-03T08:02:29.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.782+0000 7f1e24f9bb80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac6e61a10 con 0x559ac6e00560 2023-03-03T08:02:29.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.782+0000 7f1e24f9bb80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ac6e62010 con 0x559ac6e04cd0 2023-03-03T08:02:29.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.782+0000 7f1e24f9bb80 10 client.0 _ll_register_callbacks cb 0x559ac6cf71c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T08:02:29.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.782+0000 7f1e0b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e05a70 0x559ac6dff800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.783+0000 7f1e0b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e05a70 0x559ac6dff800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43748/0 (socket says 172.21.15.98:43748) 2023-03-03T08:02:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.783+0000 7f1e0b7fe700 1 -- 172.21.15.98:0/2013953726 learned_addr learned my addr 172.21.15.98:0/2013953726 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.783+0000 7f1e0a7fc700 1 --2- 172.21.15.98:0/2013953726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e04cd0 0x559ac6dff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.783+0000 7f1e0b7fe700 1 -- 172.21.15.98:0/2013953726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e04cd0 msgr2=0x559ac6dff2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.783+0000 7f1e0affd700 1 --2- 172.21.15.98:0/2013953726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e00560 0x559ac6e2c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.783+0000 7f1e0b7fe700 1 --2- 172.21.15.98:0/2013953726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e04cd0 0x559ac6dff2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.783+0000 7f1e0b7fe700 1 -- 172.21.15.98:0/2013953726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e00560 msgr2=0x559ac6e2c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.783+0000 7f1e0b7fe700 1 --2- 172.21.15.98:0/2013953726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e00560 0x559ac6e2c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.783+0000 7f1e0b7fe700 1 -- 172.21.15.98:0/2013953726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559ac6b87b10 con 0x559ac6e05a70 2023-03-03T08:02:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.783+0000 7f1e0a7fc700 1 --2- 172.21.15.98:0/2013953726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e04cd0 0x559ac6dff2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.783+0000 7f1e0affd700 1 --2- 172.21.15.98:0/2013953726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e00560 0x559ac6e2c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.783+0000 7f1e0b7fe700 1 --2- 172.21.15.98:0/2013953726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e05a70 0x559ac6dff800 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f1e04055d50 tx=0x7f1e0401d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d473fe61c4b718be server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.783+0000 7f1de9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-03-03T08:02:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.783+0000 7f1de9ffb700 1 -- 172.21.15.98:0/2013953726 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e04055810 con 0x559ac6e05a70 2023-03-03T08:02:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.784+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2013953726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1e040556a0 con 0x559ac6e05a70 2023-03-03T08:02:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.784+0000 7f1de9ffb700 1 -- 172.21.15.98:0/2013953726 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1e04063070 con 0x559ac6e05a70 2023-03-03T08:02:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.784+0000 7f1de9ffb700 1 -- 172.21.15.98:0/2013953726 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e04055540 con 0x559ac6e05a70 2023-03-03T08:02:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.784+0000 7f1e24f9bb80 10 client.7674 fetch_fsmap learned FSMap version 48 2023-03-03T08:02:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.784+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2013953726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x559ac6b70990 con 0x559ac6e05a70 2023-03-03T08:02:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.784+0000 7f1de9ffb700 1 -- 172.21.15.98:0/2013953726 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e04072070 con 0x559ac6e05a70 2023-03-03T08:02:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.784+0000 7f1de9ffb700 1 -- 172.21.15.98:0/2013953726 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 48) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f1e04051e40 con 0x559ac6e05a70 2023-03-03T08:02:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.784+0000 7f1e24f9bb80 10 client.7674 fetch_fsmap finished waiting for FSMap version 48 2023-03-03T08:02:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.784+0000 7f1e24f9bb80 10 client.7674 resolve_mds: resolved name 'c' to daemon mds.c 2023-03-03T08:02:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.784+0000 7f1e24f9bb80 20 client.7674 populate_metadata read hostname 'smithi098' 2023-03-03T08:02:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.784+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/2013953726 >> [v2:172.21.15.98:6833/341265634,v1:172.21.15.98:6835/341265634] conn(0x559ac6e4b380 0x559ac6e4b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.784+0000 7f1e24f9bb80 4 client.7674 mds_command: new command op to 7524 tid=0[{"prefix": "status"}] 2023-03-03T08:02:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.784+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2013953726 --> [v2:172.21.15.98:6833/341265634,v1:172.21.15.98:6835/341265634] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559ac6ba8ca0 con 0x559ac6e4b380 2023-03-03T08:02:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.784+0000 7f1e0affd700 1 --2- 172.21.15.98:0/2013953726 >> [v2:172.21.15.98:6833/341265634,v1:172.21.15.98:6835/341265634] conn(0x559ac6e4b380 0x559ac6e4b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.785+0000 7f1e0affd700 1 --2- 172.21.15.98:0/2013953726 >> [v2:172.21.15.98:6833/341265634,v1:172.21.15.98:6835/341265634] conn(0x559ac6e4b380 0x559ac6e4b750 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2158d192172f27 server_cookie=12db9e31d56fa573 in_seq=0 out_seq=0 2023-03-03T08:02:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.785+0000 7f1de9ffb700 10 client.7674 ms_handle_connect on v2:172.21.15.98:6833/341265634 2023-03-03T08:02:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.785+0000 7f1de9ffb700 1 -- 172.21.15.98:0/2013953726 <== mds.0 v2:172.21.15.98:6833/341265634 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+317 (crc 0 0 0) 0x559ac6ba8ca0 con 0x559ac6e4b380 2023-03-03T08:02:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.785+0000 7f1de9ffb700 10 client.7674 handle_command_reply: tid=0 2023-03-03T08:02:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.785+0000 7f1e24f9bb80 1 client.7674 shutdown 2023-03-03T08:02:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.785+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2013953726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e05a70 msgr2=0x559ac6dff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.785+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/2013953726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e05a70 0x559ac6dff800 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f1e04055d50 tx=0x7f1e0401d070 comp rx=0 tx=0).stop 2023-03-03T08:02:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.785+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2013953726 shutdown_connections 2023-03-03T08:02:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.785+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/2013953726 >> [v2:172.21.15.98:6833/341265634,v1:172.21.15.98:6835/341265634] conn(0x559ac6e4b380 0x559ac6e4b750 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.786+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/2013953726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ac6e05a70 0x559ac6dff800 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.786+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/2013953726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559ac6e00560 0x559ac6e2c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.786+0000 7f1de9ffb700 0 client.7674 ms_handle_reset on v2:172.21.15.98:6833/341265634 2023-03-03T08:02:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.786+0000 7f1e24f9bb80 1 --2- 172.21.15.98:0/2013953726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559ac6e04cd0 0x559ac6dff2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.786+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2013953726 >> 172.21.15.98:0/2013953726 conn(0x559ac6e03c30 msgr2=0x559ac6e326d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.786+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2013953726 shutdown_connections 2023-03-03T08:02:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.786+0000 7f1e24f9bb80 1 -- 172.21.15.98:0/2013953726 wait complete. 2023-03-03T08:02:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.786+0000 7f1e24f9bb80 20 client.7674 trim_cache size 0 max 16384 2023-03-03T08:02:29.795 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:02:29.795 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "fbbd322f-236a-44f8-96c5-be6963846c9b", 2023-03-03T08:02:29.795 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2023-03-03T08:02:29.795 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 7524, 2023-03-03T08:02:29.796 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-03-03T08:02:29.796 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-03-03T08:02:29.796 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-03-03T08:02:29.796 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 7.378392034, 2023-03-03T08:02:29.796 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 48, 2023-03-03T08:02:29.796 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 91, 2023-03-03T08:02:29.796 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 91, 2023-03-03T08:02:29.796 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 13.21900114 2023-03-03T08:02:29.797 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:02:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.786+0000 7f1e115ff700 1 -- 172.21.15.98:0/3222188898 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1df8068380 msgr2=0x7f1df806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.787+0000 7f1e115ff700 1 --2- 172.21.15.98:0/3222188898 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1df8068380 0x7f1df806a830 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0090b0 tx=0x7f1dfc008030 comp rx=0 tx=0).stop 2023-03-03T08:02:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.787+0000 7f1e115ff700 1 -- 172.21.15.98:0/3222188898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0c15ca70 msgr2=0x7f1e0c0b3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.787+0000 7f1e115ff700 1 --2- 172.21.15.98:0/3222188898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0c15ca70 0x7f1e0c0b3f10 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400cbc0 tx=0x7f1e0400ced0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.787+0000 7f1e115ff700 1 -- 172.21.15.98:0/3222188898 shutdown_connections 2023-03-03T08:02:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.787+0000 7f1e115ff700 1 --2- 172.21.15.98:0/3222188898 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1df8068380 0x7f1df806a830 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.787+0000 7f1e115ff700 1 --2- 172.21.15.98:0/3222188898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0c15ca70 0x7f1e0c0b3f10 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.787+0000 7f1e115ff700 1 --2- 172.21.15.98:0/3222188898 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e0c06b4b0 0x7f1e0c150f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.787+0000 7f1e115ff700 1 --2- 172.21.15.98:0/3222188898 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e0c0d0270 0x7f1e0c15e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.787+0000 7f1e115ff700 1 -- 172.21.15.98:0/3222188898 >> 172.21.15.98:0/3222188898 conn(0x7f1e0c073e70 msgr2=0x7f1e0c0d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.787+0000 7f1e115ff700 1 -- 172.21.15.98:0/3222188898 shutdown_connections 2023-03-03T08:02:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:29.787+0000 7f1e115ff700 1 -- 172.21.15.98:0/3222188898 wait complete. 2023-03-03T08:02:29.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T08:02:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.061+0000 7f664ffff700 1 Processor -- start 2023-03-03T08:02:30.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.061+0000 7f664ffff700 1 -- start start 2023-03-03T08:02:30.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.063+0000 7f664ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6650160f80 0x7f66500d1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:30.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.063+0000 7f664ffff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66500d10f0 con 0x7f6650146360 2023-03-03T08:02:30.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.063+0000 7f664ffff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66500cdf20 con 0x7f66500b51c0 2023-03-03T08:02:30.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.063+0000 7f664ffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66500cd680 con 0x7f6650160f80 2023-03-03T08:02:30.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.063+0000 7f664effd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6650146360 0x7f66500b4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34308/0 (socket says 172.21.15.98:34308) 2023-03-03T08:02:30.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.063+0000 7f664effd700 1 -- 172.21.15.98:0/2825002390 learned_addr learned my addr 172.21.15.98:0/2825002390 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:30.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664f7fe700 1 --2- 172.21.15.98:0/2825002390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6650160f80 0x7f66500d1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:30.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301082615 0 0) 0x7f66500d10f0 con 0x7f6650146360 2023-03-03T08:02:30.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f663c0035b0 con 0x7f6650146360 2023-03-03T08:02:30.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569438663 0 0) 0x7f66500cdf20 con 0x7f66500b51c0 2023-03-03T08:02:30.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66500d10f0 con 0x7f66500b51c0 2023-03-03T08:02:30.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3946920127 0 0) 0x7f663c0035b0 con 0x7f6650146360 2023-03-03T08:02:30.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6650160f80 msgr2=0x7f66500d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:30.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 --2- 172.21.15.98:0/2825002390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6650160f80 0x7f66500d1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:30.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 >> v1:172.21.15.100:6789/0 conn(0x7f66500b51c0 legacy=0x7f66500d1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:30.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66500cd960 con 0x7f6650146360 2023-03-03T08:02:30.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f6640001670 con 0x7f6650146360 2023-03-03T08:02:30.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 >> v1:172.21.15.98:6789/0 conn(0x7f6650146360 legacy=0x7f66500b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:30.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 --2- 172.21.15.98:0/2825002390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f663c003bc0 0x7f663c003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:30.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 --2- 172.21.15.98:0/2825002390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6650160f80 0x7f663c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:30.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 --2- 172.21.15.98:0/2825002390 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f663c004d90 0x7f663c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:30.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66500cd680 con 0x7f663c003bc0 2023-03-03T08:02:30.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c00bba0 con 0x7f663c004d90 2023-03-03T08:02:30.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.064+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663c00bd10 con 0x7f6650160f80 2023-03-03T08:02:30.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.065+0000 7f664effd700 1 --2- 172.21.15.98:0/2825002390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f663c003bc0 0x7f663c003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:30.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.065+0000 7f664f7fe700 1 --2- 172.21.15.98:0/2825002390 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f663c004d90 0x7f663c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:30.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.065+0000 7f6647fff700 1 --2- 172.21.15.98:0/2825002390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6650160f80 0x7f663c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:30.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.065+0000 7f664effd700 1 -- 172.21.15.98:0/2825002390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6650160f80 msgr2=0x7f663c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:30.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.065+0000 7f664effd700 1 --2- 172.21.15.98:0/2825002390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6650160f80 0x7f663c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:30.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.065+0000 7f664effd700 1 -- 172.21.15.98:0/2825002390 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f663c004d90 msgr2=0x7f663c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:30.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.065+0000 7f664effd700 1 --2- 172.21.15.98:0/2825002390 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f663c004d90 0x7f663c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:30.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.065+0000 7f664effd700 1 -- 172.21.15.98:0/2825002390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c0035b0 con 0x7f663c003bc0 2023-03-03T08:02:30.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.065+0000 7f664effd700 1 --2- 172.21.15.98:0/2825002390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f663c003bc0 0x7f663c003fb0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f664000c0e0 tx=0x7f663c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb27b503cb1ee0db server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:30.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.066+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664000cc80 con 0x7f663c003bc0 2023-03-03T08:02:30.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.066+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664000cde0 con 0x7f663c003bc0 2023-03-03T08:02:30.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.066+0000 7f664e7fc700 1 -- 172.21.15.98:0/2825002390 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640005ba0 con 0x7f663c003bc0 2023-03-03T08:02:30.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.066+0000 7f664ffff700 1 -- 172.21.15.98:0/2825002390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f663c003bc0 msgr2=0x7f663c003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:30.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.066+0000 7f664ffff700 1 --2- 172.21.15.98:0/2825002390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f663c003bc0 0x7f663c003fb0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f664000c0e0 tx=0x7f663c005880 comp rx=0 tx=0).stop 2023-03-03T08:02:30.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.066+0000 7f664f7fe700 1 -- 172.21.15.98:0/2825002390 reap_dead start 2023-03-03T08:02:30.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.066+0000 7f664ffff700 1 -- 172.21.15.98:0/2825002390 shutdown_connections 2023-03-03T08:02:30.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.066+0000 7f664ffff700 1 -- 172.21.15.98:0/2825002390 >> 172.21.15.98:0/2825002390 conn(0x7f665006b8e0 msgr2=0x7f66500cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:30.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.066+0000 7f664ffff700 1 -- 172.21.15.98:0/2825002390 shutdown_connections 2023-03-03T08:02:30.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.066+0000 7f664ffff700 1 -- 172.21.15.98:0/2825002390 wait complete. 2023-03-03T08:02:30.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.069+0000 7f664ffff700 1 Processor -- start 2023-03-03T08:02:30.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.070+0000 7f664ffff700 1 -- start start 2023-03-03T08:02:30.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.071+0000 7f664ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f663c003bc0 0x7f6650159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:30.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.071+0000 7f664ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66500c7620 0x7f66500c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:30.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.071+0000 7f664ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66500b44a0 0x7f66500b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:30.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.071+0000 7f664ffff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66500b4e50 con 0x7f663c003bc0 2023-03-03T08:02:30.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.071+0000 7f664ffff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665015e7b0 con 0x7f66500b44a0 2023-03-03T08:02:30.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.071+0000 7f664ffff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66500c80c0 con 0x7f66500c7620 2023-03-03T08:02:30.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.071+0000 7f664effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f663c003bc0 0x7f6650159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:30.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.071+0000 7f664effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f663c003bc0 0x7f6650159f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43778/0 (socket says 172.21.15.98:43778) 2023-03-03T08:02:30.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.071+0000 7f664effd700 1 -- 172.21.15.98:0/3931769726 learned_addr learned my addr 172.21.15.98:0/3931769726 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:30.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.071+0000 7f6647fff700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66500c7620 0x7f66500c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:30.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.071+0000 7f664f7fe700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66500b44a0 0x7f66500b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:30.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.072+0000 7f664effd700 1 -- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66500c7620 msgr2=0x7f66500c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:30.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.072+0000 7f664effd700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66500c7620 0x7f66500c7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:30.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.072+0000 7f664effd700 1 -- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66500b44a0 msgr2=0x7f66500b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:30.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.072+0000 7f664effd700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66500b44a0 0x7f66500b4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:30.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.072+0000 7f664effd700 1 -- 172.21.15.98:0/3931769726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66500d0d80 con 0x7f663c003bc0 2023-03-03T08:02:30.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.072+0000 7f6647fff700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66500c7620 0x7f66500c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:30.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.072+0000 7f664f7fe700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66500b44a0 0x7f66500b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:30.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.072+0000 7f664effd700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f663c003bc0 0x7f6650159f90 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f6640001eb0 tx=0x7f6640001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:30.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.072+0000 7f66477fe700 1 -- 172.21.15.98:0/3931769726 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640004c80 con 0x7f663c003bc0 2023-03-03T08:02:30.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.072+0000 7f66477fe700 1 -- 172.21.15.98:0/3931769726 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6640004de0 con 0x7f663c003bc0 2023-03-03T08:02:30.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.072+0000 7f664ffff700 1 -- 172.21.15.98:0/3931769726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66500cdc40 con 0x7f663c003bc0 2023-03-03T08:02:30.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.072+0000 7f66477fe700 1 -- 172.21.15.98:0/3931769726 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66400072f0 con 0x7f663c003bc0 2023-03-03T08:02:30.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.073+0000 7f664ffff700 1 -- 172.21.15.98:0/3931769726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66501613c0 con 0x7f663c003bc0 2023-03-03T08:02:30.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.074+0000 7f664ffff700 1 -- 172.21.15.98:0/3931769726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66500cf7b0 con 0x7f663c003bc0 2023-03-03T08:02:30.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.075+0000 7f66477fe700 1 -- 172.21.15.98:0/3931769726 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f6640010070 con 0x7f663c003bc0 2023-03-03T08:02:30.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.077+0000 7f66477fe700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6630068380 0x7f663006a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:30.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.077+0000 7f66477fe700 1 -- 172.21.15.98:0/3931769726 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6640086dc0 con 0x7f663c003bc0 2023-03-03T08:02:30.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.078+0000 7f6647fff700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6630068380 0x7f663006a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:30.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.079+0000 7f6647fff700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6630068380 0x7f663006a830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6638000c00 tx=0x7f6638009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:30.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.081+0000 7f66477fe700 1 -- 172.21.15.98:0/3931769726 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f66400535b0 con 0x7f663c003bc0 2023-03-03T08:02:30.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.316+0000 7f664ffff700 1 -- 172.21.15.98:0/3931769726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6650068300 con 0x7f663c003bc0 2023-03-03T08:02:30.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.316+0000 7f66477fe700 1 -- 172.21.15.98:0/3931769726 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v48) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6640017070 con 0x7f663c003bc0 2023-03-03T08:02:30.318 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T08:02:30.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.319+0000 7f664ffff700 1 -- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6630068380 msgr2=0x7f663006a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:30.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.319+0000 7f664ffff700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6630068380 0x7f663006a830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6638000c00 tx=0x7f6638009040 comp rx=0 tx=0).stop 2023-03-03T08:02:30.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.319+0000 7f664ffff700 1 -- 172.21.15.98:0/3931769726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f663c003bc0 msgr2=0x7f6650159f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:30.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.319+0000 7f664ffff700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f663c003bc0 0x7f6650159f90 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f6640001eb0 tx=0x7f6640001930 comp rx=0 tx=0).stop 2023-03-03T08:02:30.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.319+0000 7f664ffff700 1 -- 172.21.15.98:0/3931769726 shutdown_connections 2023-03-03T08:02:30.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.319+0000 7f664ffff700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6630068380 0x7f663006a830 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:30.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.319+0000 7f664ffff700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f663c003bc0 0x7f6650159f90 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:30.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.319+0000 7f664ffff700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66500b44a0 0x7f66500b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:30.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.319+0000 7f664ffff700 1 --2- 172.21.15.98:0/3931769726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66500c7620 0x7f66500c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:30.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.319+0000 7f664ffff700 1 -- 172.21.15.98:0/3931769726 >> 172.21.15.98:0/3931769726 conn(0x7f665006b8e0 msgr2=0x7f66500b7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:30.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.319+0000 7f664ffff700 1 -- 172.21.15.98:0/3931769726 shutdown_connections 2023-03-03T08:02:30.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:30.319+0000 7f664ffff700 1 -- 172.21.15.98:0/3931769726 wait complete. 2023-03-03T08:02:30.347 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T08:02:30.348 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T08:02:30.348 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2023-03-03T08:02:30.348 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-03-03T08:02:30.349 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-03-03T08:02:30.349 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T08:02:30.349 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-03-03T08:02:30.349 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T08:02:30.350 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T08:02:30.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:30.350 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2023-03-03T08:02:30.420 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T08:02:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T08:02:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T08:02:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:02:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-03-03T08:02:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:02:30.421 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:02:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:02:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2023-03-03T08:02:30.422 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:02:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:02:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2023-03-03T08:02:30.423 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:02:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2023-03-03T08:02:30.423 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T08:02:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2023-03-03T08:02:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T08:02:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 41805sec preferred_lft 41805sec 2023-03-03T08:02:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2023-03-03T08:02:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:02:30.424 INFO:teuthology.orchestra.run.smithi098.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T08:02:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 46:3f:9e:55:37:22 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:02:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-03-03T08:02:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:02:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::1857:c2ff:fe19:346f/64 scope link 2023-03-03T08:02:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:02:30.425 INFO:teuthology.orchestra.run.smithi098.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:02:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 46:3f:9e:55:37:22 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-03-03T08:02:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::443f:9eff:fe55:3722/64 scope link 2023-03-03T08:02:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:02:30.425 INFO:teuthology.orchestra.run.smithi098.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:02:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether d6:19:97:a6:7c:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-03-03T08:02:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::d419:97ff:fea6:7c83/64 scope link 2023-03-03T08:02:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:02:30.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:30.427 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T08:02:30.526 INFO:teuthology.orchestra.run.smithi098.stdout:1 2023-03-03T08:02:30.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:30.531 DEBUG:teuthology.orchestra.run.smithi098:> route 2023-03-03T08:02:30.602 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2023-03-03T08:02:30.603 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T08:02:30.603 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T08:02:30.603 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T08:02:30.603 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T08:02:30.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:30.605 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-03-03T08:02:30.605 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-03-03T08:02:30.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T08:02:30.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T08:02:30.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T08:02:30.606 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-03-03T08:02:30.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:30.847 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-03-03T08:02:30.916 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-03-03T08:02:30.917 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-03-03T08:02:30.918 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T08:02:30.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:30.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2023-03-03T08:02:31.022 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-03-03T08:02:31.022 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:02:31.023 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-03-03T08:02:31.090 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-03-03T08:02:31.091 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:02:31.092 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-03-03T08:02:31.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T08:02:31.261 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:02:31.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:02:31.359 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:02:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:02:31.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:31.364 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T08:02:31.432 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-03-03T08:02:31.433 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/) 2023-03-03T08:02:31.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T08:02:31.542 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:02:31.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:02:31.645 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:02:31.649 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:02:31.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:31.650 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T08:02:31.662 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:2023-03-03T08:02:31.661+0000 7fec5e1625c0 -1 init, newargv = 0x55b91322cfb0 newargc=15 2023-03-03T08:02:31.663 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[144617]: starting ceph client 2023-03-03T08:02:31.672 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[144617]: starting fuse 2023-03-03T08:02:31.724 INFO:teuthology.orchestra.run.smithi098.stdout:42 2023-03-03T08:02:31.725 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2023-03-03T08:02:31.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T08:02:31.726 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2023-03-03T08:02:31.727 DEBUG:teuthology.orchestra.run.smithi098:> import re 2023-03-03T08:02:31.727 DEBUG:teuthology.orchestra.run.smithi098:> import os 2023-03-03T08:02:31.727 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2023-03-03T08:02:31.728 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:02:31.728 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2023-03-03T08:02:31.728 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-03-03T08:02:31.729 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2023-03-03T08:02:31.729 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-03-03T08:02:31.729 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:02:31.730 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2023-03-03T08:02:31.730 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2023-03-03T08:02:31.730 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2023-03-03T08:02:31.730 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2023-03-03T08:02:31.731 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:02:31.731 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2023-03-03T08:02:31.731 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T08:02:31.732 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T08:02:31.732 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T08:02:31.732 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2023-03-03T08:02:31.733 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2023-03-03T08:02:31.733 DEBUG:teuthology.orchestra.run.smithi098:> return f 2023-03-03T08:02:31.733 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T08:02:31.733 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:02:31.734 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.0")) 2023-03-03T08:02:31.734 DEBUG:teuthology.orchestra.run.smithi098:> ' 2023-03-03T08:02:31.948 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.0.144617.asok 2023-03-03T08:02:32.125 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.144617.asok 2023-03-03T08:02:32.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:32.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.144617.asok status 2023-03-03T08:02:32.396 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:02:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2023-03-03T08:02:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T08:02:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T08:02:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "0", 2023-03-03T08:02:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2023-03-03T08:02:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-03-03T08:02:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "144617", 2023-03-03T08:02:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2023-03-03T08:02:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2023-03-03T08:02:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2023-03-03T08:02:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 7686, 2023-03-03T08:02:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2023-03-03T08:02:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2023-03-03T08:02:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2023-03-03T08:02:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 7686 2023-03-03T08:02:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T08:02:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T08:02:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:02:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 693507664 2023-03-03T08:02:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T08:02:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T08:02:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:02:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 693507664 2023-03-03T08:02:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:02:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.7686 192.168.0.1:0/693507664", 2023-03-03T08:02:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/693507664", 2023-03-03T08:02:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2023-03-03T08:02:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 48, 2023-03-03T08:02:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 91, 2023-03-03T08:02:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2023-03-03T08:02:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2023-03-03T08:02:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2023-03-03T08:02:32.401 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:02:32.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:32.418 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2023-03-03T08:02:32.418 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.0 2023-03-03T08:02:32.487 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2023-03-03T08:02:32.488 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-03-03T08:02:32.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:32.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-03-03T08:02:32.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T08:02:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.834+0000 7fad127ff700 1 Processor -- start 2023-03-03T08:02:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.835+0000 7fad127ff700 1 -- start start 2023-03-03T08:02:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.836+0000 7fad127ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c0b87c0 0x7fad0c0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.836+0000 7fad127ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad0c0d0ef0 con 0x7fad0c0b8b90 2023-03-03T08:02:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.836+0000 7fad127ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad0c0d0600 con 0x7fad0c0b5410 2023-03-03T08:02:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.836+0000 7fad127ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c0cd970 con 0x7fad0c0b87c0 2023-03-03T08:02:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.837+0000 7fad03fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fad0c0b8b90 0x7fad0c0b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34312/0 (socket says 172.21.15.98:34312) 2023-03-03T08:02:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.837+0000 7fad03fff700 1 -- 172.21.15.98:0/913446854 learned_addr learned my addr 172.21.15.98:0/913446854 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.837+0000 7fad0bfff700 1 --2- 172.21.15.98:0/913446854 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c0b87c0 0x7fad0c0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.837+0000 7fad0b7fe700 1 -- 172.21.15.98:0/913446854 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080873244 0 0) 0x7fad0c0d0ef0 con 0x7fad0c0b8b90 2023-03-03T08:02:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.837+0000 7fad0b7fe700 1 -- 172.21.15.98:0/913446854 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facf80035b0 con 0x7fad0c0b8b90 2023-03-03T08:02:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.837+0000 7fad0b7fe700 1 -- 172.21.15.98:0/913446854 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024308348 0 0) 0x7fad0c0d0600 con 0x7fad0c0b5410 2023-03-03T08:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.837+0000 7fad0b7fe700 1 -- 172.21.15.98:0/913446854 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad0c0d0ef0 con 0x7fad0c0b5410 2023-03-03T08:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.837+0000 7fad0bfff700 1 -- 172.21.15.98:0/913446854 >> v1:172.21.15.100:6789/0 conn(0x7fad0c0b5410 legacy=0x7fad0c0d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.837+0000 7fad0bfff700 1 -- 172.21.15.98:0/913446854 >> v1:172.21.15.98:6789/0 conn(0x7fad0c0b8b90 legacy=0x7fad0c0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.837+0000 7fad0bfff700 1 -- 172.21.15.98:0/913446854 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0c0cdc50 con 0x7fad0c0b87c0 2023-03-03T08:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.838+0000 7fad0bfff700 1 --2- 172.21.15.98:0/913446854 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c0b87c0 0x7fad0c0cffe0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7facfc009870 tx=0x7facfc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78b7538134bb2c01 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.838+0000 7fad0b7fe700 1 -- 172.21.15.98:0/913446854 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc011830 con 0x7fad0c0b87c0 2023-03-03T08:02:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.838+0000 7fad0b7fe700 1 -- 172.21.15.98:0/913446854 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facfc011990 con 0x7fad0c0b87c0 2023-03-03T08:02:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.838+0000 7fad0b7fe700 1 -- 172.21.15.98:0/913446854 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc011d20 con 0x7fad0c0b87c0 2023-03-03T08:02:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.838+0000 7fad127ff700 1 -- 172.21.15.98:0/913446854 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c0b87c0 msgr2=0x7fad0c0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.838+0000 7fad127ff700 1 --2- 172.21.15.98:0/913446854 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c0b87c0 0x7fad0c0cffe0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7facfc009870 tx=0x7facfc009b80 comp rx=0 tx=0).stop 2023-03-03T08:02:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.839+0000 7fad127ff700 1 -- 172.21.15.98:0/913446854 shutdown_connections 2023-03-03T08:02:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.839+0000 7fad127ff700 1 --2- 172.21.15.98:0/913446854 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c0b87c0 0x7fad0c0cffe0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.839+0000 7fad127ff700 1 -- 172.21.15.98:0/913446854 >> 172.21.15.98:0/913446854 conn(0x7fad0c06aee0 msgr2=0x7fad0c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.839+0000 7fad127ff700 1 -- 172.21.15.98:0/913446854 shutdown_connections 2023-03-03T08:02:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.839+0000 7fad127ff700 1 -- 172.21.15.98:0/913446854 wait complete. 2023-03-03T08:02:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.842+0000 7fad127ff700 1 Processor -- start 2023-03-03T08:02:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.842+0000 7fad127ff700 1 -- start start 2023-03-03T08:02:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.843+0000 7fad127ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0c0b5410 0x7fad0c0cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.843+0000 7fad127ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c0b87c0 0x7fad0c0cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.843+0000 7fad127ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad0c0b8b90 0x7fad0c153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.843+0000 7fad127ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c15eeb0 con 0x7fad0c0b5410 2023-03-03T08:02:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.843+0000 7fad127ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c15e2c0 con 0x7fad0c0b8b90 2023-03-03T08:02:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.843+0000 7fad127ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c15f2d0 con 0x7fad0c0b87c0 2023-03-03T08:02:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.844+0000 7fad0bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0c0b5410 0x7fad0c0cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.844+0000 7fad0bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0c0b5410 0x7fad0c0cac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43782/0 (socket says 172.21.15.98:43782) 2023-03-03T08:02:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.844+0000 7fad0bfff700 1 -- 172.21.15.98:0/3536224531 learned_addr learned my addr 172.21.15.98:0/3536224531 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.844+0000 7fad03fff700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c0b87c0 0x7fad0c0cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.844+0000 7fad10d9c700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad0c0b8b90 0x7fad0c153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.844+0000 7fad0bfff700 1 -- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c0b87c0 msgr2=0x7fad0c0cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.844+0000 7fad0bfff700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c0b87c0 0x7fad0c0cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.844+0000 7fad0bfff700 1 -- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad0c0b8b90 msgr2=0x7fad0c153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.844+0000 7fad0bfff700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad0c0b8b90 0x7fad0c153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.844+0000 7fad0bfff700 1 -- 172.21.15.98:0/3536224531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facfc0094a0 con 0x7fad0c0b5410 2023-03-03T08:02:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.844+0000 7fad10d9c700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad0c0b8b90 0x7fad0c153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.844+0000 7fad03fff700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c0b87c0 0x7fad0c0cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.844+0000 7fad0bfff700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0c0b5410 0x7fad0c0cac70 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7facfc0097c0 tx=0x7facfc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.845+0000 7fad097fa700 1 -- 172.21.15.98:0/3536224531 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc01aa70 con 0x7fad0c0b5410 2023-03-03T08:02:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.845+0000 7fad097fa700 1 -- 172.21.15.98:0/3536224531 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facfc01abd0 con 0x7fad0c0b5410 2023-03-03T08:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.845+0000 7fad127ff700 1 -- 172.21.15.98:0/3536224531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad0c0cdf30 con 0x7fad0c0b5410 2023-03-03T08:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.845+0000 7fad097fa700 1 -- 172.21.15.98:0/3536224531 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc0243f0 con 0x7fad0c0b5410 2023-03-03T08:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.845+0000 7fad127ff700 1 -- 172.21.15.98:0/3536224531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad0c073a80 con 0x7fad0c0b5410 2023-03-03T08:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.846+0000 7fad127ff700 1 -- 172.21.15.98:0/3536224531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad0c0ce0c0 con 0x7fad0c0b5410 2023-03-03T08:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.847+0000 7fad097fa700 1 -- 172.21.15.98:0/3536224531 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7facfc00f050 con 0x7fad0c0b5410 2023-03-03T08:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.849+0000 7fad097fa700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7facec068380 0x7facec06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.849+0000 7fad097fa700 1 -- 172.21.15.98:0/3536224531 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7facfc087100 con 0x7fad0c0b5410 2023-03-03T08:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.849+0000 7fad03fff700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7facec068380 0x7facec06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.850+0000 7fad03fff700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7facec068380 0x7facec06a830 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7facf4000f60 tx=0x7facf4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:32.853+0000 7fad097fa700 1 -- 172.21.15.98:0/3536224531 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7facfc053880 con 0x7fad0c0b5410 2023-03-03T08:02:33.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:33.087+0000 7fad127ff700 1 -- 172.21.15.98:0/3536224531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fad0c0729b0 con 0x7fad0c0b5410 2023-03-03T08:02:33.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:33.087+0000 7fad097fa700 1 -- 172.21.15.98:0/3536224531 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v48) v1 ==== 53+0+73 (secure 0 0 0) 0x7facfc015070 con 0x7fad0c0b5410 2023-03-03T08:02:33.089 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T08:02:33.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:33.090+0000 7fad127ff700 1 -- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7facec068380 msgr2=0x7facec06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:33.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:33.090+0000 7fad127ff700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7facec068380 0x7facec06a830 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7facf4000f60 tx=0x7facf4009040 comp rx=0 tx=0).stop 2023-03-03T08:02:33.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:33.090+0000 7fad127ff700 1 -- 172.21.15.98:0/3536224531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0c0b5410 msgr2=0x7fad0c0cac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:33.090+0000 7fad127ff700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0c0b5410 0x7fad0c0cac70 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7facfc0097c0 tx=0x7facfc004750 comp rx=0 tx=0).stop 2023-03-03T08:02:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:33.090+0000 7fad127ff700 1 -- 172.21.15.98:0/3536224531 shutdown_connections 2023-03-03T08:02:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:33.090+0000 7fad127ff700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7facec068380 0x7facec06a830 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:33.090+0000 7fad127ff700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad0c0b5410 0x7fad0c0cac70 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:33.090+0000 7fad127ff700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad0c0b8b90 0x7fad0c153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:33.090+0000 7fad127ff700 1 --2- 172.21.15.98:0/3536224531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c0b87c0 0x7fad0c0cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:33.090+0000 7fad127ff700 1 -- 172.21.15.98:0/3536224531 >> 172.21.15.98:0/3536224531 conn(0x7fad0c06aee0 msgr2=0x7fad0c0b3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:33.090+0000 7fad127ff700 1 -- 172.21.15.98:0/3536224531 shutdown_connections 2023-03-03T08:02:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:33.090+0000 7fad127ff700 1 -- 172.21.15.98:0/3536224531 wait complete. 2023-03-03T08:02:33.118 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T08:02:33.118 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T08:02:33.119 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2023-03-03T08:02:33.119 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-03-03T08:02:33.119 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-03-03T08:02:33.119 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T08:02:33.120 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-03-03T08:02:33.120 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T08:02:33.120 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-03-03T08:02:33.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:33.121 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2023-03-03T08:02:33.195 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T08:02:33.195 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T08:02:33.196 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T08:02:33.196 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:02:33.196 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2023-03-03T08:02:33.196 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:02:33.196 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:02:33.196 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:02:33.196 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2023-03-03T08:02:33.197 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:02:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:02:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2023-03-03T08:02:33.197 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:02:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:02:33.197 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T08:02:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:02:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T08:02:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 41807sec preferred_lft 41807sec 2023-03-03T08:02:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2023-03-03T08:02:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:02:33.198 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T08:02:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ee:78:63:4a:ea:32 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:02:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-03-03T08:02:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:02:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::9406:dcff:fe40:a4a4/64 scope link 2023-03-03T08:02:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:02:33.199 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:02:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:5e:bd:c1:84:4d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-03-03T08:02:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f45e:bdff:fec1:844d/64 scope link 2023-03-03T08:02:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:02:33.199 INFO:teuthology.orchestra.run.smithi100.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:02:33.200 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ee:78:63:4a:ea:32 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-03-03T08:02:33.200 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec78:63ff:fe4a:ea32/64 scope link 2023-03-03T08:02:33.200 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:02:33.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:33.202 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T08:02:33.300 INFO:teuthology.orchestra.run.smithi100.stdout:1 2023-03-03T08:02:33.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:33.305 DEBUG:teuthology.orchestra.run.smithi100:> route 2023-03-03T08:02:33.376 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2023-03-03T08:02:33.376 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T08:02:33.377 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T08:02:33.377 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T08:02:33.377 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T08:02:33.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:33.378 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2023-03-03T08:02:33.379 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2023-03-03T08:02:33.379 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T08:02:33.379 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T08:02:33.380 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T08:02:33.380 DEBUG:teuthology.orchestra.run.smithi100:> ') 2023-03-03T08:02:33.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:02:33.524 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:02:33.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:33.630 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2023-03-03T08:02:33.697 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-03-03T08:02:33.698 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-03-03T08:02:33.699 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-03-03T08:02:33.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:33.699 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2023-03-03T08:02:33.802 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-03-03T08:02:33.802 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:02:33.803 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-03-03T08:02:33.872 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-03-03T08:02:33.872 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:02:33.873 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-03-03T08:02:33.939 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T08:02:34.042 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:02:34.042 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:02:34.139 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:02:34.143 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:02:34.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:34.146 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T08:02:34.215 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-03-03T08:02:34.216 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_mountpoint=/) 2023-03-03T08:02:34.221 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T08:02:34.325 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:02:34.326 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:02:34.430 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:02:34.435 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:02:34.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:34.437 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T08:02:34.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2023-03-03T08:02:34.450+0000 7f4f4b6135c0 -1 init, newargv = 0x55ba65676d90 newargc=15 2023-03-03T08:02:34.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[120896]: starting ceph client 2023-03-03T08:02:34.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[120896]: starting fuse 2023-03-03T08:02:34.514 INFO:teuthology.orchestra.run.smithi100.stdout:42 2023-03-03T08:02:34.515 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2023-03-03T08:02:34.515 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T08:02:34.516 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2023-03-03T08:02:34.516 DEBUG:teuthology.orchestra.run.smithi100:> import re 2023-03-03T08:02:34.516 DEBUG:teuthology.orchestra.run.smithi100:> import os 2023-03-03T08:02:34.517 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2023-03-03T08:02:34.517 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:02:34.517 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2023-03-03T08:02:34.518 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-03-03T08:02:34.518 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2023-03-03T08:02:34.518 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-03-03T08:02:34.518 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:02:34.519 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2023-03-03T08:02:34.519 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2023-03-03T08:02:34.520 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2023-03-03T08:02:34.520 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2023-03-03T08:02:34.520 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:02:34.521 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2023-03-03T08:02:34.521 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T08:02:34.521 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T08:02:34.522 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T08:02:34.522 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2023-03-03T08:02:34.522 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2023-03-03T08:02:34.522 DEBUG:teuthology.orchestra.run.smithi100:> return f 2023-03-03T08:02:34.523 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T08:02:34.523 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:02:34.523 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.1")) 2023-03-03T08:02:34.524 DEBUG:teuthology.orchestra.run.smithi100:> ' 2023-03-03T08:02:34.740 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.1.120896.asok 2023-03-03T08:02:34.918 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.120896.asok 2023-03-03T08:02:34.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:34.919 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.120896.asok status 2023-03-03T08:02:35.191 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2023-03-03T08:02:35.191 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2023-03-03T08:02:35.191 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T08:02:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T08:02:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "1", 2023-03-03T08:02:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2023-03-03T08:02:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-03-03T08:02:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "120896", 2023-03-03T08:02:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2023-03-03T08:02:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:02:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2023-03-03T08:02:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2023-03-03T08:02:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 6557, 2023-03-03T08:02:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2023-03-03T08:02:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2023-03-03T08:02:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2023-03-03T08:02:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 6557 2023-03-03T08:02:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:02:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T08:02:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T08:02:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:02:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2807628791 2023-03-03T08:02:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: } 2023-03-03T08:02:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:02:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T08:02:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T08:02:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:02:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2807628791 2023-03-03T08:02:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:02:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.6557 192.168.0.1:0/2807628791", 2023-03-03T08:02:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.1:0/2807628791", 2023-03-03T08:02:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2023-03-03T08:02:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 48, 2023-03-03T08:02:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 91, 2023-03-03T08:02:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2023-03-03T08:02:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2023-03-03T08:02:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2023-03-03T08:02:35.196 INFO:teuthology.orchestra.run.smithi100.stdout:} 2023-03-03T08:02:35.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:35.212 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2023-03-03T08:02:35.212 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2023-03-03T08:02:35.282 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2023-03-03T08:02:35.282 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-03-03T08:02:35.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:35.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-03-03T08:02:35.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-03-03T08:02:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.644+0000 7f331bfff700 1 Processor -- start 2023-03-03T08:02:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.645+0000 7f331bfff700 1 -- start start 2023-03-03T08:02:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.646+0000 7f331bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f331c0d25e0 0x7f331c0cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.646+0000 7f331bfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f331c0b5130 0x7f331c0d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.646+0000 7f331bfff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331c160ef0 con 0x7f331c06b750 2023-03-03T08:02:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.646+0000 7f331bfff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f331c161420 con 0x7f331c0d25e0 2023-03-03T08:02:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.646+0000 7f331bfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331c161140 con 0x7f331c0b5130 2023-03-03T08:02:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.646+0000 7f331affd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f331c0d25e0 0x7f331c0cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.646+0000 7f3313fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f331c06b750 0x7f331c0b4a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44038/0 (socket says 172.21.15.98:44038) 2023-03-03T08:02:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.647+0000 7f3313fff700 1 -- 172.21.15.98:0/3989397345 learned_addr learned my addr 172.21.15.98:0/3989397345 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.647+0000 7f331b7fe700 1 --2- 172.21.15.98:0/3989397345 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f331c0b5130 0x7f331c0d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.647+0000 7f331affd700 1 -- 172.21.15.98:0/3989397345 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f331c0b5130 msgr2=0x7f331c0d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.647+0000 7f331affd700 1 --2- 172.21.15.98:0/3989397345 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f331c0b5130 0x7f331c0d0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.647+0000 7f331affd700 1 -- 172.21.15.98:0/3989397345 >> v1:172.21.15.98:6789/0 conn(0x7f331c06b750 legacy=0x7f331c0b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.647+0000 7f331affd700 1 -- 172.21.15.98:0/3989397345 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331c0cd720 con 0x7f331c0d25e0 2023-03-03T08:02:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.647+0000 7f331affd700 1 --2- 172.21.15.98:0/3989397345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c0d25e0 0x7f331c0cf2d0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f330c009840 tx=0x7f330c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71ab778799f7186 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.648+0000 7f331a7fc700 1 -- 172.21.15.98:0/3989397345 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c011830 con 0x7f331c0d25e0 2023-03-03T08:02:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.648+0000 7f331a7fc700 1 -- 172.21.15.98:0/3989397345 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330c011990 con 0x7f331c0d25e0 2023-03-03T08:02:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.648+0000 7f331a7fc700 1 -- 172.21.15.98:0/3989397345 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c011d20 con 0x7f331c0d25e0 2023-03-03T08:02:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.648+0000 7f331bfff700 1 -- 172.21.15.98:0/3989397345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c0d25e0 msgr2=0x7f331c0cf2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.648+0000 7f331bfff700 1 --2- 172.21.15.98:0/3989397345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c0d25e0 0x7f331c0cf2d0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f330c009840 tx=0x7f330c009b50 comp rx=0 tx=0).stop 2023-03-03T08:02:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.648+0000 7f331bfff700 1 -- 172.21.15.98:0/3989397345 shutdown_connections 2023-03-03T08:02:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.648+0000 7f331bfff700 1 --2- 172.21.15.98:0/3989397345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c0d25e0 0x7f331c0cf2d0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.648+0000 7f331bfff700 1 --2- 172.21.15.98:0/3989397345 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f331c0b5130 0x7f331c0d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.648+0000 7f331bfff700 1 -- 172.21.15.98:0/3989397345 >> 172.21.15.98:0/3989397345 conn(0x7f331c0d3be0 msgr2=0x7f331c06ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.648+0000 7f331bfff700 1 -- 172.21.15.98:0/3989397345 shutdown_connections 2023-03-03T08:02:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.648+0000 7f331bfff700 1 -- 172.21.15.98:0/3989397345 wait complete. 2023-03-03T08:02:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.650+0000 7f331bfff700 1 Processor -- start 2023-03-03T08:02:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.651+0000 7f331bfff700 1 -- start start 2023-03-03T08:02:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c06b750 0x7f331c1f2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331bfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f331c0b5130 0x7f331c0cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331bfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f331c0d25e0 0x7f331c0cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331bfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c1f26a0 con 0x7f331c06b750 2023-03-03T08:02:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331bfff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c0c5be0 con 0x7f331c0d25e0 2023-03-03T08:02:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331bfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331c15d0e0 con 0x7f331c0b5130 2023-03-03T08:02:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c06b750 0x7f331c1f2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c06b750 0x7f331c1f2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60100/0 (socket says 172.21.15.98:60100) 2023-03-03T08:02:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331affd700 1 -- 172.21.15.98:0/1977996913 learned_addr learned my addr 172.21.15.98:0/1977996913 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f3313fff700 1 --2- 172.21.15.98:0/1977996913 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f331c0b5130 0x7f331c0cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331b7fe700 1 --2- 172.21.15.98:0/1977996913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f331c0d25e0 0x7f331c0cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331affd700 1 -- 172.21.15.98:0/1977996913 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f331c0b5130 msgr2=0x7f331c0cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331affd700 1 --2- 172.21.15.98:0/1977996913 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f331c0b5130 0x7f331c0cb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331affd700 1 -- 172.21.15.98:0/1977996913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f331c0d25e0 msgr2=0x7f331c0cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331affd700 1 --2- 172.21.15.98:0/1977996913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f331c0d25e0 0x7f331c0cba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331affd700 1 -- 172.21.15.98:0/1977996913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c0094a0 con 0x7f331c06b750 2023-03-03T08:02:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.652+0000 7f331b7fe700 1 --2- 172.21.15.98:0/1977996913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f331c0d25e0 0x7f331c0cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.653+0000 7f331affd700 1 --2- 172.21.15.98:0/1977996913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c06b750 0x7f331c1f2160 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f330c0090b0 tx=0x7f330c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.653+0000 7f33137fe700 1 -- 172.21.15.98:0/1977996913 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c0036a0 con 0x7f331c06b750 2023-03-03T08:02:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.653+0000 7f33137fe700 1 -- 172.21.15.98:0/1977996913 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330c01f070 con 0x7f331c06b750 2023-03-03T08:02:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.653+0000 7f33137fe700 1 -- 172.21.15.98:0/1977996913 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c003a00 con 0x7f331c06b750 2023-03-03T08:02:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.653+0000 7f331bfff700 1 -- 172.21.15.98:0/1977996913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331c1466a0 con 0x7f331c06b750 2023-03-03T08:02:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.653+0000 7f331bfff700 1 -- 172.21.15.98:0/1977996913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331c0cda00 con 0x7f331c06b750 2023-03-03T08:02:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.654+0000 7f331bfff700 1 -- 172.21.15.98:0/1977996913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331c0d29b0 con 0x7f331c06b750 2023-03-03T08:02:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.655+0000 7f33137fe700 1 -- 172.21.15.98:0/1977996913 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f330c00f050 con 0x7f331c06b750 2023-03-03T08:02:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.657+0000 7f33137fe700 1 --2- 172.21.15.98:0/1977996913 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f32fc0683b0 0x7f32fc06a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.657+0000 7f33137fe700 1 -- 172.21.15.98:0/1977996913 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f330c021070 con 0x7f331c06b750 2023-03-03T08:02:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.657+0000 7f3313fff700 1 --2- 172.21.15.98:0/1977996913 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f32fc0683b0 0x7f32fc06a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.658+0000 7f3313fff700 1 --2- 172.21.15.98:0/1977996913 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f32fc0683b0 0x7f32fc06a860 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f331c1f5c60 tx=0x7f3304009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:35.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.661+0000 7f33137fe700 1 -- 172.21.15.98:0/1977996913 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f330c053f10 con 0x7f331c06b750 2023-03-03T08:02:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:35.896+0000 7f331bfff700 1 -- 172.21.15.98:0/1977996913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f331c068290 con 0x7f331c06b750 2023-03-03T08:02:36.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.558+0000 7f33137fe700 1 -- 172.21.15.98:0/1977996913 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v49) v1 ==== 136+0+0 (secure 0 0 0) 0x7f330c085c70 con 0x7f331c06b750 2023-03-03T08:02:36.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.561+0000 7f331bfff700 1 -- 172.21.15.98:0/1977996913 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f32fc0683b0 msgr2=0x7f32fc06a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.561+0000 7f331bfff700 1 --2- 172.21.15.98:0/1977996913 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f32fc0683b0 0x7f32fc06a860 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f331c1f5c60 tx=0x7f3304009040 comp rx=0 tx=0).stop 2023-03-03T08:02:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.561+0000 7f331bfff700 1 -- 172.21.15.98:0/1977996913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c06b750 msgr2=0x7f331c1f2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.561+0000 7f331bfff700 1 --2- 172.21.15.98:0/1977996913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c06b750 0x7f331c1f2160 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f330c0090b0 tx=0x7f330c004750 comp rx=0 tx=0).stop 2023-03-03T08:02:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.561+0000 7f331bfff700 1 -- 172.21.15.98:0/1977996913 shutdown_connections 2023-03-03T08:02:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.561+0000 7f331bfff700 1 --2- 172.21.15.98:0/1977996913 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f32fc0683b0 0x7f32fc06a860 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.561+0000 7f331bfff700 1 --2- 172.21.15.98:0/1977996913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c06b750 0x7f331c1f2160 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.561+0000 7f331bfff700 1 --2- 172.21.15.98:0/1977996913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f331c0d25e0 0x7f331c0cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:36.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.561+0000 7f331bfff700 1 --2- 172.21.15.98:0/1977996913 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f331c0b5130 0x7f331c0cb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:36.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.561+0000 7f331bfff700 1 -- 172.21.15.98:0/1977996913 >> 172.21.15.98:0/1977996913 conn(0x7f331c0d3be0 msgr2=0x7f331c0b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:36.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.561+0000 7f331bfff700 1 -- 172.21.15.98:0/1977996913 shutdown_connections 2023-03-03T08:02:36.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.561+0000 7f331bfff700 1 -- 172.21.15.98:0/1977996913 wait complete. 2023-03-03T08:02:36.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-03-03T08:02:36.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.836+0000 7f2d77dff700 1 Processor -- start 2023-03-03T08:02:36.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.837+0000 7f2d77dff700 1 -- start start 2023-03-03T08:02:36.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.838+0000 7f2d77dff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d7015ca70 0x7f2d700cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:36.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.838+0000 7f2d77dff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2d7006b4b0 0x7f2d700cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:36.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.838+0000 7f2d77dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d700cf980 con 0x7f2d700d00a0 2023-03-03T08:02:36.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.838+0000 7f2d77dff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d700cd5b0 con 0x7f2d7015ca70 2023-03-03T08:02:36.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.838+0000 7f2d77dff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d700d0540 con 0x7f2d7006b4b0 2023-03-03T08:02:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.838+0000 7f2d75b9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d7015ca70 0x7f2d700cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.839+0000 7f2d75b9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d7015ca70 0x7f2d700cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60108/0 (socket says 172.21.15.98:60108) 2023-03-03T08:02:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.839+0000 7f2d75b9b700 1 -- 172.21.15.98:0/399690508 learned_addr learned my addr 172.21.15.98:0/399690508 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.839+0000 7f2d7539a700 1 --2- 172.21.15.98:0/399690508 >> v2:172.21.15.100:3300/0 conn(0x7f2d7006b4b0 0x7f2d700cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.839+0000 7f2d75b9b700 1 -- 172.21.15.98:0/399690508 >> v2:172.21.15.100:3300/0 conn(0x7f2d7006b4b0 msgr2=0x7f2d700cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:36.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.839+0000 7f2d75b9b700 1 --2- 172.21.15.98:0/399690508 >> v2:172.21.15.100:3300/0 conn(0x7f2d7006b4b0 0x7f2d700cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:36.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.839+0000 7f2d75b9b700 1 -- 172.21.15.98:0/399690508 >> v1:172.21.15.100:6789/0 conn(0x7f2d700d00a0 legacy=0x7f2d700b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:36.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.839+0000 7f2d75b9b700 1 -- 172.21.15.98:0/399690508 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d700d2fd0 con 0x7f2d7015ca70 2023-03-03T08:02:36.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.839+0000 7f2d7539a700 1 --2- 172.21.15.98:0/399690508 >> v2:172.21.15.100:3300/0 conn(0x7f2d7006b4b0 0x7f2d700cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:36.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.839+0000 7f2d75b9b700 1 --2- 172.21.15.98:0/399690508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d7015ca70 0x7f2d700cf440 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2d70160560 tx=0x7f2d6c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e068f4de95137b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:36.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.840+0000 7f2d74b99700 1 -- 172.21.15.98:0/399690508 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c004230 con 0x7f2d7015ca70 2023-03-03T08:02:36.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.840+0000 7f2d74b99700 1 -- 172.21.15.98:0/399690508 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c004390 con 0x7f2d7015ca70 2023-03-03T08:02:36.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.840+0000 7f2d74b99700 1 -- 172.21.15.98:0/399690508 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c0036a0 con 0x7f2d7015ca70 2023-03-03T08:02:36.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.840+0000 7f2d77dff700 1 -- 172.21.15.98:0/399690508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d7015ca70 msgr2=0x7f2d700cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.840+0000 7f2d77dff700 1 --2- 172.21.15.98:0/399690508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d7015ca70 0x7f2d700cf440 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2d70160560 tx=0x7f2d6c009bd0 comp rx=0 tx=0).stop 2023-03-03T08:02:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.840+0000 7f2d77dff700 1 -- 172.21.15.98:0/399690508 shutdown_connections 2023-03-03T08:02:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.840+0000 7f2d77dff700 1 --2- 172.21.15.98:0/399690508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d7015ca70 0x7f2d700cf440 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.840+0000 7f2d77dff700 1 --2- 172.21.15.98:0/399690508 >> v2:172.21.15.100:3300/0 conn(0x7f2d7006b4b0 0x7f2d700cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.840+0000 7f2d77dff700 1 -- 172.21.15.98:0/399690508 >> 172.21.15.98:0/399690508 conn(0x7f2d70073e70 msgr2=0x7f2d7006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.840+0000 7f2d77dff700 1 -- 172.21.15.98:0/399690508 shutdown_connections 2023-03-03T08:02:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.841+0000 7f2d77dff700 1 -- 172.21.15.98:0/399690508 wait complete. 2023-03-03T08:02:36.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.842+0000 7f2d77dff700 1 Processor -- start 2023-03-03T08:02:36.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.843+0000 7f2d77dff700 1 -- start start 2023-03-03T08:02:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.844+0000 7f2d77dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d7006b4b0 0x7f2d70154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.844+0000 7f2d77dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2d700d00a0 0x7f2d701548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.844+0000 7f2d77dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d7015ca70 0x7f2d7015f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.844+0000 7f2d77dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d70154e00 con 0x7f2d7015ca70 2023-03-03T08:02:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.844+0000 7f2d77dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d700caca0 con 0x7f2d700d00a0 2023-03-03T08:02:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.844+0000 7f2d77dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7015fc70 con 0x7f2d7006b4b0 2023-03-03T08:02:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.844+0000 7f2d7639c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d7015ca70 0x7f2d7015f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:36.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.844+0000 7f2d7639c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d7015ca70 0x7f2d7015f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60124/0 (socket says 172.21.15.98:60124) 2023-03-03T08:02:36.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.844+0000 7f2d7639c700 1 -- 172.21.15.98:0/2650551634 learned_addr learned my addr 172.21.15.98:0/2650551634 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:36.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.844+0000 7f2d7539a700 1 --2- 172.21.15.98:0/2650551634 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2d700d00a0 0x7f2d701548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:36.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.844+0000 7f2d75b9b700 1 --2- 172.21.15.98:0/2650551634 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d7006b4b0 0x7f2d70154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:36.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.845+0000 7f2d7639c700 1 -- 172.21.15.98:0/2650551634 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d7006b4b0 msgr2=0x7f2d70154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.845+0000 7f2d7639c700 1 --2- 172.21.15.98:0/2650551634 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d7006b4b0 0x7f2d70154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.845+0000 7f2d7639c700 1 -- 172.21.15.98:0/2650551634 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2d700d00a0 msgr2=0x7f2d701548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.845+0000 7f2d7639c700 1 --2- 172.21.15.98:0/2650551634 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2d700d00a0 0x7f2d701548c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.845+0000 7f2d7639c700 1 -- 172.21.15.98:0/2650551634 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c009550 con 0x7f2d7015ca70 2023-03-03T08:02:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.845+0000 7f2d7639c700 1 --2- 172.21.15.98:0/2650551634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d7015ca70 0x7f2d7015f420 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800cc10 tx=0x7f2d6800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.845+0000 7f2d667fc700 1 -- 172.21.15.98:0/2650551634 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6800aa60 con 0x7f2d7015ca70 2023-03-03T08:02:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.845+0000 7f2d667fc700 1 -- 172.21.15.98:0/2650551634 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6800abc0 con 0x7f2d7015ca70 2023-03-03T08:02:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.845+0000 7f2d77dff700 1 -- 172.21.15.98:0/2650551634 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d700cd900 con 0x7f2d7015ca70 2023-03-03T08:02:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.845+0000 7f2d667fc700 1 -- 172.21.15.98:0/2650551634 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d68012420 con 0x7f2d7015ca70 2023-03-03T08:02:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.846+0000 7f2d77dff700 1 -- 172.21.15.98:0/2650551634 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d700d1800 con 0x7f2d7015ca70 2023-03-03T08:02:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.847+0000 7f2d77dff700 1 -- 172.21.15.98:0/2650551634 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d700cdbe0 con 0x7f2d7015ca70 2023-03-03T08:02:36.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.848+0000 7f2d667fc700 1 -- 172.21.15.98:0/2650551634 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f2d68010050 con 0x7f2d7015ca70 2023-03-03T08:02:36.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.849+0000 7f2d667fc700 1 --2- 172.21.15.98:0/2650551634 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2d54068380 0x7f2d5406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:36.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.849+0000 7f2d667fc700 1 -- 172.21.15.98:0/2650551634 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2d68085fb0 con 0x7f2d7015ca70 2023-03-03T08:02:36.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.850+0000 7f2d75b9b700 1 --2- 172.21.15.98:0/2650551634 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2d54068380 0x7f2d5406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:36.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.851+0000 7f2d75b9b700 1 --2- 172.21.15.98:0/2650551634 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2d54068380 0x7f2d5406a830 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c009130 tx=0x7f2d6c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:36.853+0000 7f2d667fc700 1 -- 172.21.15.98:0/2650551634 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2d680526d0 con 0x7f2d7015ca70 2023-03-03T08:02:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.072+0000 7f2d77dff700 1 -- 172.21.15.98:0/2650551634 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d7006c110 con 0x7f2d54068380 2023-03-03T08:02:37.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.074+0000 7f2d667fc700 1 -- 172.21.15.98:0/2650551634 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f2d7006c110 con 0x7f2d54068380 2023-03-03T08:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T08:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephfs" 2023-03-03T08:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T08:02:37.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.077+0000 7f2d77dff700 1 -- 172.21.15.98:0/2650551634 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2d54068380 msgr2=0x7f2d5406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.077+0000 7f2d77dff700 1 --2- 172.21.15.98:0/2650551634 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2d54068380 0x7f2d5406a830 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c009130 tx=0x7f2d6c008030 comp rx=0 tx=0).stop 2023-03-03T08:02:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.077+0000 7f2d77dff700 1 -- 172.21.15.98:0/2650551634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d7015ca70 msgr2=0x7f2d7015f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.077+0000 7f2d77dff700 1 --2- 172.21.15.98:0/2650551634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d7015ca70 0x7f2d7015f420 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800cc10 tx=0x7f2d6800cf20 comp rx=0 tx=0).stop 2023-03-03T08:02:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.077+0000 7f2d77dff700 1 -- 172.21.15.98:0/2650551634 shutdown_connections 2023-03-03T08:02:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.077+0000 7f2d77dff700 1 --2- 172.21.15.98:0/2650551634 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2d54068380 0x7f2d5406a830 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.077+0000 7f2d77dff700 1 --2- 172.21.15.98:0/2650551634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d7015ca70 0x7f2d7015f420 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.077+0000 7f2d77dff700 1 --2- 172.21.15.98:0/2650551634 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d7006b4b0 0x7f2d70154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.077+0000 7f2d77dff700 1 --2- 172.21.15.98:0/2650551634 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2d700d00a0 0x7f2d701548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.077+0000 7f2d77dff700 1 -- 172.21.15.98:0/2650551634 >> 172.21.15.98:0/2650551634 conn(0x7f2d70073e70 msgr2=0x7f2d701e52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.077+0000 7f2d77dff700 1 -- 172.21.15.98:0/2650551634 shutdown_connections 2023-03-03T08:02:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.078+0000 7f2d77dff700 1 -- 172.21.15.98:0/2650551634 wait complete. 2023-03-03T08:02:37.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-03-03T08:02:37.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.353+0000 7fdc6b3ff700 1 Processor -- start 2023-03-03T08:02:37.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.353+0000 7fdc6b3ff700 1 -- start start 2023-03-03T08:02:37.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.354+0000 7fdc6b3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc6415ca70 0x7fdc640cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:37.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.354+0000 7fdc6b3ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdc6406b4b0 0x7fdc640cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:37.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.355+0000 7fdc6b3ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdc640d00a0 0x7fdc640b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:37.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.355+0000 7fdc6b3ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc640cd5b0 con 0x7fdc640d00a0 2023-03-03T08:02:37.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.355+0000 7fdc6b3ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc640cf980 con 0x7fdc6406b4b0 2023-03-03T08:02:37.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.355+0000 7fdc6b3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc640d0510 con 0x7fdc6415ca70 2023-03-03T08:02:37.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.355+0000 7fdc6999c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdc640d00a0 0x7fdc640b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:37.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.355+0000 7fdc6999c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdc640d00a0 0x7fdc640b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60130/0 (socket says 172.21.15.98:60130) 2023-03-03T08:02:37.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.355+0000 7fdc6999c700 1 -- 172.21.15.98:0/1401903687 learned_addr learned my addr 172.21.15.98:0/1401903687 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:37.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.355+0000 7fdc6899a700 1 --2- 172.21.15.98:0/1401903687 >> v2:172.21.15.100:3300/0 conn(0x7fdc6406b4b0 0x7fdc640cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:37.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.355+0000 7fdc6919b700 1 --2- 172.21.15.98:0/1401903687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc6415ca70 0x7fdc640cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:37.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.356+0000 7fdc6999c700 1 -- 172.21.15.98:0/1401903687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc6415ca70 msgr2=0x7fdc640cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.356+0000 7fdc6999c700 1 --2- 172.21.15.98:0/1401903687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc6415ca70 0x7fdc640cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.356+0000 7fdc6999c700 1 -- 172.21.15.98:0/1401903687 >> v2:172.21.15.100:3300/0 conn(0x7fdc6406b4b0 msgr2=0x7fdc640cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.356+0000 7fdc6999c700 1 --2- 172.21.15.98:0/1401903687 >> v2:172.21.15.100:3300/0 conn(0x7fdc6406b4b0 0x7fdc640cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.356+0000 7fdc6999c700 1 -- 172.21.15.98:0/1401903687 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc640d1800 con 0x7fdc640d00a0 2023-03-03T08:02:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.356+0000 7fdc6999c700 1 --2- 172.21.15.98:0/1401903687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc640d00a0 0x7fdc640b4c00 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fdc640b5d40 tx=0x7fdc5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d383266f4097942e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.356+0000 7fdc5bfff700 1 -- 172.21.15.98:0/1401903687 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c00e070 con 0x7fdc640d00a0 2023-03-03T08:02:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.356+0000 7fdc5bfff700 1 -- 172.21.15.98:0/1401903687 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5c00b8d0 con 0x7fdc640d00a0 2023-03-03T08:02:37.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.356+0000 7fdc5bfff700 1 -- 172.21.15.98:0/1401903687 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c00bc80 con 0x7fdc640d00a0 2023-03-03T08:02:37.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.357+0000 7fdc6b3ff700 1 -- 172.21.15.98:0/1401903687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc640d00a0 msgr2=0x7fdc640b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:37.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.357+0000 7fdc6b3ff700 1 --2- 172.21.15.98:0/1401903687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc640d00a0 0x7fdc640b4c00 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fdc640b5d40 tx=0x7fdc5c00b330 comp rx=0 tx=0).stop 2023-03-03T08:02:37.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.357+0000 7fdc6b3ff700 1 -- 172.21.15.98:0/1401903687 shutdown_connections 2023-03-03T08:02:37.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.357+0000 7fdc6b3ff700 1 --2- 172.21.15.98:0/1401903687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc640d00a0 0x7fdc640b4c00 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.357+0000 7fdc6b3ff700 1 --2- 172.21.15.98:0/1401903687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc6415ca70 0x7fdc640cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.357+0000 7fdc6b3ff700 1 --2- 172.21.15.98:0/1401903687 >> v2:172.21.15.100:3300/0 conn(0x7fdc6406b4b0 0x7fdc640cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.357+0000 7fdc6b3ff700 1 -- 172.21.15.98:0/1401903687 >> 172.21.15.98:0/1401903687 conn(0x7fdc64073e70 msgr2=0x7fdc6406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:37.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.357+0000 7fdc6b3ff700 1 -- 172.21.15.98:0/1401903687 shutdown_connections 2023-03-03T08:02:37.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.357+0000 7fdc6b3ff700 1 -- 172.21.15.98:0/1401903687 wait complete. 2023-03-03T08:02:37.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.359+0000 7fdc6b3ff700 1 Processor -- start 2023-03-03T08:02:37.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.359+0000 7fdc6b3ff700 1 -- start start 2023-03-03T08:02:37.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.360+0000 7fdc6b3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc6406b4b0 0x7fdc64157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:37.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.360+0000 7fdc6b3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc640d00a0 0x7fdc6415f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:37.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.360+0000 7fdc6b3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc6415ca70 0x7fdc6415ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:37.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.360+0000 7fdc6b3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc640cd100 con 0x7fdc640d00a0 2023-03-03T08:02:37.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.360+0000 7fdc6b3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc641f2300 con 0x7fdc6406b4b0 2023-03-03T08:02:37.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.360+0000 7fdc6b3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc640b2d80 con 0x7fdc6415ca70 2023-03-03T08:02:37.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.361+0000 7fdc6899a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc640d00a0 0x7fdc6415f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:37.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.361+0000 7fdc6899a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc640d00a0 0x7fdc6415f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60136/0 (socket says 172.21.15.98:60136) 2023-03-03T08:02:37.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.361+0000 7fdc6899a700 1 -- 172.21.15.98:0/1124348367 learned_addr learned my addr 172.21.15.98:0/1124348367 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:37.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.361+0000 7fdc6919b700 1 --2- 172.21.15.98:0/1124348367 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc6406b4b0 0x7fdc64157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:37.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.361+0000 7fdc6999c700 1 --2- 172.21.15.98:0/1124348367 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc6415ca70 0x7fdc6415ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:37.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.361+0000 7fdc6899a700 1 -- 172.21.15.98:0/1124348367 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc6415ca70 msgr2=0x7fdc6415ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:37.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.361+0000 7fdc6899a700 1 --2- 172.21.15.98:0/1124348367 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc6415ca70 0x7fdc6415ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.361+0000 7fdc6899a700 1 -- 172.21.15.98:0/1124348367 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc6406b4b0 msgr2=0x7fdc64157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:37.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.361+0000 7fdc6899a700 1 --2- 172.21.15.98:0/1124348367 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc6406b4b0 0x7fdc64157a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.361+0000 7fdc6899a700 1 -- 172.21.15.98:0/1124348367 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c009b30 con 0x7fdc640d00a0 2023-03-03T08:02:37.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.362+0000 7fdc6899a700 1 --2- 172.21.15.98:0/1124348367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc640d00a0 0x7fdc6415f600 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fdc5400e980 tx=0x7fdc5400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:37.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.362+0000 7fdc59ffb700 1 -- 172.21.15.98:0/1124348367 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc54009710 con 0x7fdc640d00a0 2023-03-03T08:02:37.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.362+0000 7fdc59ffb700 1 -- 172.21.15.98:0/1124348367 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5400c690 con 0x7fdc640d00a0 2023-03-03T08:02:37.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.362+0000 7fdc59ffb700 1 -- 172.21.15.98:0/1124348367 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5400ca20 con 0x7fdc640d00a0 2023-03-03T08:02:37.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.362+0000 7fdc6b3ff700 1 -- 172.21.15.98:0/1124348367 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc640cdbe0 con 0x7fdc640d00a0 2023-03-03T08:02:37.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.362+0000 7fdc6b3ff700 1 -- 172.21.15.98:0/1124348367 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc640cd900 con 0x7fdc640d00a0 2023-03-03T08:02:37.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.363+0000 7fdc6b3ff700 1 -- 172.21.15.98:0/1124348367 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc640d1aa0 con 0x7fdc640d00a0 2023-03-03T08:02:37.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.364+0000 7fdc59ffb700 1 -- 172.21.15.98:0/1124348367 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fdc54019030 con 0x7fdc640d00a0 2023-03-03T08:02:37.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.366+0000 7fdc59ffb700 1 --2- 172.21.15.98:0/1124348367 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdc48068380 0x7fdc4806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:37.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.366+0000 7fdc59ffb700 1 -- 172.21.15.98:0/1124348367 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdc54085c30 con 0x7fdc640d00a0 2023-03-03T08:02:37.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.366+0000 7fdc6919b700 1 --2- 172.21.15.98:0/1124348367 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdc48068380 0x7fdc4806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:37.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.367+0000 7fdc6919b700 1 --2- 172.21.15.98:0/1124348367 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdc48068380 0x7fdc4806a830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdc60006b20 tx=0x7fdc60008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:37.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.369+0000 7fdc59ffb700 1 -- 172.21.15.98:0/1124348367 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdc540523b0 con 0x7fdc640d00a0 2023-03-03T08:02:37.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.565+0000 7fdc6b3ff700 1 -- 172.21.15.98:0/1124348367 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdc640ccba0 con 0x7fdc640d00a0 2023-03-03T08:02:37.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.566+0000 7fdc59ffb700 1 -- 172.21.15.98:0/1124348367 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdc540559d0 con 0x7fdc640d00a0 2023-03-03T08:02:37.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.568+0000 7fdc6b3ff700 1 -- 172.21.15.98:0/1124348367 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdc48068380 msgr2=0x7fdc4806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:37.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.568+0000 7fdc6b3ff700 1 --2- 172.21.15.98:0/1124348367 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdc48068380 0x7fdc4806a830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdc60006b20 tx=0x7fdc60008040 comp rx=0 tx=0).stop 2023-03-03T08:02:37.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.568+0000 7fdc6b3ff700 1 -- 172.21.15.98:0/1124348367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc640d00a0 msgr2=0x7fdc6415f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:37.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.568+0000 7fdc6b3ff700 1 --2- 172.21.15.98:0/1124348367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc640d00a0 0x7fdc6415f600 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fdc5400e980 tx=0x7fdc5400ec90 comp rx=0 tx=0).stop 2023-03-03T08:02:37.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.569+0000 7fdc6b3ff700 1 -- 172.21.15.98:0/1124348367 shutdown_connections 2023-03-03T08:02:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.569+0000 7fdc6b3ff700 1 --2- 172.21.15.98:0/1124348367 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc6415ca70 0x7fdc6415ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.569+0000 7fdc6b3ff700 1 --2- 172.21.15.98:0/1124348367 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc6406b4b0 0x7fdc64157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.569+0000 7fdc6b3ff700 1 --2- 172.21.15.98:0/1124348367 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdc48068380 0x7fdc4806a830 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.569+0000 7fdc6b3ff700 1 --2- 172.21.15.98:0/1124348367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc640d00a0 0x7fdc6415f600 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.569+0000 7fdc6b3ff700 1 -- 172.21.15.98:0/1124348367 >> 172.21.15.98:0/1124348367 conn(0x7fdc64073e70 msgr2=0x7fdc640b5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.569+0000 7fdc6b3ff700 1 -- 172.21.15.98:0/1124348367 shutdown_connections 2023-03-03T08:02:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.569+0000 7fdc6b3ff700 1 -- 172.21.15.98:0/1124348367 wait complete. 2023-03-03T08:02:37.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume create 'abcd@#' 2023-03-03T08:02:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.844+0000 7f78781ff700 1 Processor -- start 2023-03-03T08:02:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.844+0000 7f78781ff700 1 -- start start 2023-03-03T08:02:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.846+0000 7f78781ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f787006b4b0 0x7f78700cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.846+0000 7f78781ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78700cf980 con 0x7f78700d0270 2023-03-03T08:02:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.846+0000 7f78781ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78700d1800 con 0x7f787015ca70 2023-03-03T08:02:37.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.846+0000 7f78781ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78700cd5b0 con 0x7f787006b4b0 2023-03-03T08:02:37.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.846+0000 7f787579a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f78700d0270 0x7f78700d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44046/0 (socket says 172.21.15.98:44046) 2023-03-03T08:02:37.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.846+0000 7f787579a700 1 -- 172.21.15.98:0/2756023473 learned_addr learned my addr 172.21.15.98:0/2756023473 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:37.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.846+0000 7f787679c700 1 --2- 172.21.15.98:0/2756023473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f787006b4b0 0x7f78700cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:37.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.847+0000 7f7874f99700 1 -- 172.21.15.98:0/2756023473 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613928083 0 0) 0x7f78700cf980 con 0x7f78700d0270 2023-03-03T08:02:37.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.847+0000 7f7874f99700 1 -- 172.21.15.98:0/2756023473 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f785c0035b0 con 0x7f78700d0270 2023-03-03T08:02:37.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.847+0000 7f7874f99700 1 -- 172.21.15.98:0/2756023473 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579457079 0 0) 0x7f78700d1800 con 0x7f787015ca70 2023-03-03T08:02:37.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.847+0000 7f7874f99700 1 -- 172.21.15.98:0/2756023473 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78700cf980 con 0x7f787015ca70 2023-03-03T08:02:37.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.847+0000 7f787679c700 1 -- 172.21.15.98:0/2756023473 >> v1:172.21.15.100:6789/0 conn(0x7f787015ca70 legacy=0x7f78700cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:37.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.847+0000 7f787679c700 1 -- 172.21.15.98:0/2756023473 >> v1:172.21.15.98:6789/0 conn(0x7f78700d0270 legacy=0x7f78700d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:37.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.847+0000 7f787679c700 1 -- 172.21.15.98:0/2756023473 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78700cd900 con 0x7f787006b4b0 2023-03-03T08:02:37.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.847+0000 7f787679c700 1 --2- 172.21.15.98:0/2756023473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f787006b4b0 0x7f78700cf440 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7868009dc0 tx=0x7f786800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0cdd5a00fd3f682 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:37.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.848+0000 7f7874f99700 1 -- 172.21.15.98:0/2756023473 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786800e070 con 0x7f787006b4b0 2023-03-03T08:02:37.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.848+0000 7f7874f99700 1 -- 172.21.15.98:0/2756023473 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786800bb20 con 0x7f787006b4b0 2023-03-03T08:02:37.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.848+0000 7f7874f99700 1 -- 172.21.15.98:0/2756023473 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7868012430 con 0x7f787006b4b0 2023-03-03T08:02:37.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.848+0000 7f78781ff700 1 -- 172.21.15.98:0/2756023473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f787006b4b0 msgr2=0x7f78700cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:37.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.848+0000 7f78781ff700 1 --2- 172.21.15.98:0/2756023473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f787006b4b0 0x7f78700cf440 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7868009dc0 tx=0x7f786800b330 comp rx=0 tx=0).stop 2023-03-03T08:02:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.848+0000 7f78781ff700 1 -- 172.21.15.98:0/2756023473 shutdown_connections 2023-03-03T08:02:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.848+0000 7f78781ff700 1 --2- 172.21.15.98:0/2756023473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f787006b4b0 0x7f78700cf440 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.848+0000 7f78781ff700 1 -- 172.21.15.98:0/2756023473 >> 172.21.15.98:0/2756023473 conn(0x7f7870073e70 msgr2=0x7f787006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.848+0000 7f78781ff700 1 -- 172.21.15.98:0/2756023473 shutdown_connections 2023-03-03T08:02:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.849+0000 7f78781ff700 1 -- 172.21.15.98:0/2756023473 wait complete. 2023-03-03T08:02:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.851+0000 7f78781ff700 1 Processor -- start 2023-03-03T08:02:37.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.851+0000 7f78781ff700 1 -- start start 2023-03-03T08:02:37.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.852+0000 7f78781ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787006b4b0 0x7f78700ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.852+0000 7f78781ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78700d0270 0x7f7870155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.852+0000 7f78781ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f787015ca70 0x7f7870156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.852+0000 7f78781ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78701579a0 con 0x7f787006b4b0 2023-03-03T08:02:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.852+0000 7f78781ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787014ede0 con 0x7f787015ca70 2023-03-03T08:02:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.852+0000 7f78781ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7870156900 con 0x7f78700d0270 2023-03-03T08:02:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.852+0000 7f7875f9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787006b4b0 0x7f78700ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.852+0000 7f7875f9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787006b4b0 0x7f78700ccb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60142/0 (socket says 172.21.15.98:60142) 2023-03-03T08:02:37.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.852+0000 7f7875f9b700 1 -- 172.21.15.98:0/4207263269 learned_addr learned my addr 172.21.15.98:0/4207263269 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:37.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.852+0000 7f787679c700 1 --2- 172.21.15.98:0/4207263269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f787015ca70 0x7f7870156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:37.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.852+0000 7f787579a700 1 --2- 172.21.15.98:0/4207263269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78700d0270 0x7f7870155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:37.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.853+0000 7f7875f9b700 1 -- 172.21.15.98:0/4207263269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78700d0270 msgr2=0x7f7870155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:37.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.853+0000 7f7875f9b700 1 --2- 172.21.15.98:0/4207263269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78700d0270 0x7f7870155d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.853+0000 7f7875f9b700 1 -- 172.21.15.98:0/4207263269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f787015ca70 msgr2=0x7f7870156260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:37.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.853+0000 7f7875f9b700 1 --2- 172.21.15.98:0/4207263269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f787015ca70 0x7f7870156260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:37.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.853+0000 7f7875f9b700 1 -- 172.21.15.98:0/4207263269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7868009a70 con 0x7f787006b4b0 2023-03-03T08:02:37.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.853+0000 7f7875f9b700 1 --2- 172.21.15.98:0/4207263269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787006b4b0 0x7f78700ccb00 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f786c00c520 tx=0x7f786c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:37.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.853+0000 7f78667fc700 1 -- 172.21.15.98:0/4207263269 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c0052d0 con 0x7f787006b4b0 2023-03-03T08:02:37.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.853+0000 7f78667fc700 1 -- 172.21.15.98:0/4207263269 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786c005430 con 0x7f787006b4b0 2023-03-03T08:02:37.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.853+0000 7f78667fc700 1 -- 172.21.15.98:0/4207263269 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c0038c0 con 0x7f787006b4b0 2023-03-03T08:02:37.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.853+0000 7f78781ff700 1 -- 172.21.15.98:0/4207263269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78700d1aa0 con 0x7f787006b4b0 2023-03-03T08:02:37.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.854+0000 7f78781ff700 1 -- 172.21.15.98:0/4207263269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78700cdbe0 con 0x7f787006b4b0 2023-03-03T08:02:37.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.855+0000 7f78781ff700 1 -- 172.21.15.98:0/4207263269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f787015ce40 con 0x7f787006b4b0 2023-03-03T08:02:37.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.856+0000 7f78667fc700 1 -- 172.21.15.98:0/4207263269 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f786c019030 con 0x7f787006b4b0 2023-03-03T08:02:37.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.858+0000 7f78667fc700 1 --2- 172.21.15.98:0/4207263269 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7854068380 0x7f785406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:37.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.858+0000 7f78667fc700 1 -- 172.21.15.98:0/4207263269 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f786c085130 con 0x7f787006b4b0 2023-03-03T08:02:37.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.859+0000 7f787579a700 1 --2- 172.21.15.98:0/4207263269 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7854068380 0x7f785406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:37.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.859+0000 7f787579a700 1 --2- 172.21.15.98:0/4207263269 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7854068380 0x7f785406a830 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7870158e10 tx=0x7f7860007dc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:37.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:37.862+0000 7f78667fc700 1 -- 172.21.15.98:0/4207263269 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f786c051820 con 0x7f787006b4b0 2023-03-03T08:02:38.085 INFO:teuthology.orchestra.run.smithi098.stderr:Invalid command: invalid chars #@ in abcd@# 2023-03-03T08:02:38.086 INFO:teuthology.orchestra.run.smithi098.stderr:fs volume create [] : Create a CephFS volume 2023-03-03T08:02:38.086 INFO:teuthology.orchestra.run.smithi098.stderr:Error EINVAL: invalid command 2023-03-03T08:02:38.101 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-03-03T08:02:38.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume create testvol 2023-03-03T08:02:38.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.348+0000 7f9b1bfff700 1 Processor -- start 2023-03-03T08:02:38.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.348+0000 7f9b1bfff700 1 -- start start 2023-03-03T08:02:38.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.350+0000 7f9b1bfff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9b1c0b4aa0 0x7f9b1c0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:38.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.350+0000 7f9b1bfff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1c15dd00 con 0x7f9b1c0cf570 2023-03-03T08:02:38.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.350+0000 7f9b1bfff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1c0cd780 con 0x7f9b1c1609b0 2023-03-03T08:02:38.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.350+0000 7f9b1bfff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b1c0cebb0 con 0x7f9b1c0b4aa0 2023-03-03T08:02:38.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.350+0000 7f9b1a7fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9b1c0cf570 0x7f9b1c15d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44062/0 (socket says 172.21.15.98:44062) 2023-03-03T08:02:38.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.350+0000 7f9b1a7fc700 1 -- 172.21.15.98:0/1832890685 learned_addr learned my addr 172.21.15.98:0/1832890685 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:38.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.350+0000 7f9b1b7fe700 1 --2- 172.21.15.98:0/1832890685 >> v2:172.21.15.100:3300/0 conn(0x7f9b1c0b4aa0 0x7f9b1c0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:38.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.351+0000 7f9b19ffb700 1 -- 172.21.15.98:0/1832890685 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485412521 0 0) 0x7f9b1c15dd00 con 0x7f9b1c0cf570 2023-03-03T08:02:38.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.351+0000 7f9b19ffb700 1 -- 172.21.15.98:0/1832890685 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b040035b0 con 0x7f9b1c0cf570 2023-03-03T08:02:38.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.351+0000 7f9b19ffb700 1 -- 172.21.15.98:0/1832890685 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037748205 0 0) 0x7f9b1c0cd780 con 0x7f9b1c1609b0 2023-03-03T08:02:38.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.351+0000 7f9b19ffb700 1 -- 172.21.15.98:0/1832890685 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b1c15dd00 con 0x7f9b1c1609b0 2023-03-03T08:02:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.351+0000 7f9b1b7fe700 1 -- 172.21.15.98:0/1832890685 >> v1:172.21.15.100:6789/0 conn(0x7f9b1c1609b0 legacy=0x7f9b1c160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.351+0000 7f9b1b7fe700 1 -- 172.21.15.98:0/1832890685 >> v1:172.21.15.98:6789/0 conn(0x7f9b1c0cf570 legacy=0x7f9b1c15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.351+0000 7f9b19ffb700 1 -- 172.21.15.98:0/1832890685 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2915620107 0 0) 0x7f9b040035b0 con 0x7f9b1c0cf570 2023-03-03T08:02:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.351+0000 7f9b1b7fe700 1 -- 172.21.15.98:0/1832890685 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1c0cda60 con 0x7f9b1c0b4aa0 2023-03-03T08:02:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.351+0000 7f9b1b7fe700 1 --2- 172.21.15.98:0/1832890685 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9b1c0b4aa0 0x7f9b1c0b4f20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9b14009d90 tx=0x7f9b1400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64a0cc78df1e73cb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.352+0000 7f9b19ffb700 1 -- 172.21.15.98:0/1832890685 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1400e070 con 0x7f9b1c0b4aa0 2023-03-03T08:02:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.352+0000 7f9b19ffb700 1 -- 172.21.15.98:0/1832890685 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b14003a20 con 0x7f9b1c0b4aa0 2023-03-03T08:02:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.352+0000 7f9b19ffb700 1 -- 172.21.15.98:0/1832890685 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b10001670 con 0x7f9b1c0b4aa0 2023-03-03T08:02:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.352+0000 7f9b1bfff700 1 -- 172.21.15.98:0/1832890685 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9b1c0b4aa0 msgr2=0x7f9b1c0b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.352+0000 7f9b1bfff700 1 --2- 172.21.15.98:0/1832890685 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9b1c0b4aa0 0x7f9b1c0b4f20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9b14009d90 tx=0x7f9b1400b330 comp rx=0 tx=0).stop 2023-03-03T08:02:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.352+0000 7f9b1bfff700 1 -- 172.21.15.98:0/1832890685 shutdown_connections 2023-03-03T08:02:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.352+0000 7f9b1bfff700 1 --2- 172.21.15.98:0/1832890685 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9b1c0b4aa0 0x7f9b1c0b4f20 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.352+0000 7f9b1bfff700 1 -- 172.21.15.98:0/1832890685 >> 172.21.15.98:0/1832890685 conn(0x7f9b1c06b880 msgr2=0x7f9b1c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:38.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.352+0000 7f9b1bfff700 1 -- 172.21.15.98:0/1832890685 shutdown_connections 2023-03-03T08:02:38.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.353+0000 7f9b1bfff700 1 -- 172.21.15.98:0/1832890685 wait complete. 2023-03-03T08:02:38.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.355+0000 7f9b1bfff700 1 Processor -- start 2023-03-03T08:02:38.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.356+0000 7f9b1bfff700 1 -- start start 2023-03-03T08:02:38.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.357+0000 7f9b1bfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b1c0b4aa0 0x7f9b1c0b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:38.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.357+0000 7f9b1bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b1c0cf570 0x7f9b1c1f5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:38.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.357+0000 7f9b1bfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9b1c1609b0 0x7f9b1c0c89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:38.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.357+0000 7f9b1bfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c0b84a0 con 0x7f9b1c0cf570 2023-03-03T08:02:38.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.357+0000 7f9b1bfff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c0d2100 con 0x7f9b1c1609b0 2023-03-03T08:02:38.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.357+0000 7f9b1bfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c0b8df0 con 0x7f9b1c0b4aa0 2023-03-03T08:02:38.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.357+0000 7f9b1a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b1c0cf570 0x7f9b1c1f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:38.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.357+0000 7f9b1a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b1c0cf570 0x7f9b1c1f5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60156/0 (socket says 172.21.15.98:60156) 2023-03-03T08:02:38.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.357+0000 7f9b1a7fc700 1 -- 172.21.15.98:0/592794847 learned_addr learned my addr 172.21.15.98:0/592794847 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:38.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.357+0000 7f9b1b7fe700 1 --2- 172.21.15.98:0/592794847 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9b1c1609b0 0x7f9b1c0c89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:38.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.357+0000 7f9b1affd700 1 --2- 172.21.15.98:0/592794847 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b1c0b4aa0 0x7f9b1c0b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:38.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.358+0000 7f9b1a7fc700 1 -- 172.21.15.98:0/592794847 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b1c0b4aa0 msgr2=0x7f9b1c0b88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:38.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.358+0000 7f9b1a7fc700 1 --2- 172.21.15.98:0/592794847 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b1c0b4aa0 0x7f9b1c0b88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:38.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.358+0000 7f9b1a7fc700 1 -- 172.21.15.98:0/592794847 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9b1c1609b0 msgr2=0x7f9b1c0c89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:38.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.358+0000 7f9b1a7fc700 1 --2- 172.21.15.98:0/592794847 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9b1c1609b0 0x7f9b1c0c89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:38.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.358+0000 7f9b1a7fc700 1 -- 172.21.15.98:0/592794847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b14009a70 con 0x7f9b1c0cf570 2023-03-03T08:02:38.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.358+0000 7f9b1a7fc700 1 --2- 172.21.15.98:0/592794847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b1c0cf570 0x7f9b1c1f5570 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f9b1000c750 tx=0x7f9b1000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:38.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.358+0000 7f9b037fe700 1 -- 172.21.15.98:0/592794847 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b10004c80 con 0x7f9b1c0cf570 2023-03-03T08:02:38.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.358+0000 7f9b037fe700 1 -- 172.21.15.98:0/592794847 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b10004e00 con 0x7f9b1c0cf570 2023-03-03T08:02:38.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.358+0000 7f9b037fe700 1 -- 172.21.15.98:0/592794847 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b10007310 con 0x7f9b1c0cf570 2023-03-03T08:02:38.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.358+0000 7f9b1bfff700 1 -- 172.21.15.98:0/592794847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b1c15cdf0 con 0x7f9b1c0cf570 2023-03-03T08:02:38.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.359+0000 7f9b1bfff700 1 -- 172.21.15.98:0/592794847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b1c0cdd40 con 0x7f9b1c0cf570 2023-03-03T08:02:38.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.360+0000 7f9b1bfff700 1 -- 172.21.15.98:0/592794847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b1c15d0d0 con 0x7f9b1c0cf570 2023-03-03T08:02:38.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.361+0000 7f9b037fe700 1 -- 172.21.15.98:0/592794847 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f9b1001a020 con 0x7f9b1c0cf570 2023-03-03T08:02:38.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.362+0000 7f9b037fe700 1 --2- 172.21.15.98:0/592794847 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9b08068380 0x7f9b0806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:38.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.362+0000 7f9b037fe700 1 -- 172.21.15.98:0/592794847 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9b10085940 con 0x7f9b1c0cf570 2023-03-03T08:02:38.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.363+0000 7f9b1affd700 1 --2- 172.21.15.98:0/592794847 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9b08068380 0x7f9b0806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:38.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.364+0000 7f9b1affd700 1 --2- 172.21.15.98:0/592794847 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9b08068380 0x7f9b0806a830 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0011e0 tx=0x7f9b0c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:38.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.366+0000 7f9b037fe700 1 -- 172.21.15.98:0/592794847 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9b100520c0 con 0x7f9b1c0cf570 2023-03-03T08:02:38.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:38.593+0000 7f9b1bfff700 1 -- 172.21.15.98:0/592794847 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "testvol", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b1c06bf00 con 0x7f9b08068380 2023-03-03T08:02:40.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.615+0000 7f9b037fe700 1 -- 172.21.15.98:0/592794847 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 Volume created successfully (no MDS daemons created)) v1 ==== 60+0+0 (secure 0 0 0) 0x7f9b1c06bf00 con 0x7f9b08068380 2023-03-03T08:02:40.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.620+0000 7f9b1bfff700 1 -- 172.21.15.98:0/592794847 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9b08068380 msgr2=0x7f9b0806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:40.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.620+0000 7f9b1bfff700 1 --2- 172.21.15.98:0/592794847 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9b08068380 0x7f9b0806a830 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0011e0 tx=0x7f9b0c009040 comp rx=0 tx=0).stop 2023-03-03T08:02:40.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.620+0000 7f9b1bfff700 1 -- 172.21.15.98:0/592794847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b1c0cf570 msgr2=0x7f9b1c1f5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:40.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.620+0000 7f9b1bfff700 1 --2- 172.21.15.98:0/592794847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b1c0cf570 0x7f9b1c1f5570 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f9b1000c750 tx=0x7f9b1000ca60 comp rx=0 tx=0).stop 2023-03-03T08:02:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.620+0000 7f9b1bfff700 1 -- 172.21.15.98:0/592794847 shutdown_connections 2023-03-03T08:02:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.620+0000 7f9b1bfff700 1 --2- 172.21.15.98:0/592794847 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9b1c1609b0 0x7f9b1c0c89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.620+0000 7f9b1bfff700 1 --2- 172.21.15.98:0/592794847 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9b1c0b4aa0 0x7f9b1c0b88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.620+0000 7f9b1bfff700 1 --2- 172.21.15.98:0/592794847 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9b08068380 0x7f9b0806a830 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.620+0000 7f9b1bfff700 1 --2- 172.21.15.98:0/592794847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b1c0cf570 0x7f9b1c1f5570 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.620+0000 7f9b1bfff700 1 -- 172.21.15.98:0/592794847 >> 172.21.15.98:0/592794847 conn(0x7f9b1c06b880 msgr2=0x7f9b1c152950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.620+0000 7f9b1bfff700 1 -- 172.21.15.98:0/592794847 shutdown_connections 2023-03-03T08:02:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.620+0000 7f9b1bfff700 1 -- 172.21.15.98:0/592794847 wait complete. 2023-03-03T08:02:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:Volume created successfully (no MDS daemons created) 2023-03-03T08:02:40.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create testvol 'abcd@#' 2023-03-03T08:02:40.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.909+0000 7f062afff700 1 Processor -- start 2023-03-03T08:02:40.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.909+0000 7f062afff700 1 -- start start 2023-03-03T08:02:40.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.910+0000 7f062afff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06240b26d0 0x7f06240b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:40.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.910+0000 7f062afff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f06240b5140 0x7f06240d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:40.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.911+0000 7f062afff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06240cd7d0 con 0x7f06240b8d10 2023-03-03T08:02:40.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.911+0000 7f062afff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06240b55c0 con 0x7f06240b5140 2023-03-03T08:02:40.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.911+0000 7f062afff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06240b90e0 con 0x7f06240b26d0 2023-03-03T08:02:40.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.911+0000 7f0628d9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f06240b8d10 0x7f06240cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44078/0 (socket says 172.21.15.98:44078) 2023-03-03T08:02:40.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.911+0000 7f0628d9b700 1 -- 172.21.15.98:0/2279342447 learned_addr learned my addr 172.21.15.98:0/2279342447 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.911+0000 7f062959c700 1 --2- 172.21.15.98:0/2279342447 >> v2:172.21.15.98:3300/0 conn(0x7f06240b5140 0x7f06240d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.911+0000 7f0623fff700 1 --2- 172.21.15.98:0/2279342447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06240b26d0 0x7f06240b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.911+0000 7f06237fe700 1 -- 172.21.15.98:0/2279342447 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842923319 0 0) 0x7f06240cd7d0 con 0x7f06240b8d10 2023-03-03T08:02:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.911+0000 7f06237fe700 1 -- 172.21.15.98:0/2279342447 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06140035b0 con 0x7f06240b8d10 2023-03-03T08:02:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.912+0000 7f062959c700 1 -- 172.21.15.98:0/2279342447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06240b26d0 msgr2=0x7f06240b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.912+0000 7f062959c700 1 --2- 172.21.15.98:0/2279342447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06240b26d0 0x7f06240b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.912+0000 7f062959c700 1 -- 172.21.15.98:0/2279342447 >> v1:172.21.15.98:6789/0 conn(0x7f06240b8d10 legacy=0x7f06240cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.912+0000 7f062959c700 1 -- 172.21.15.98:0/2279342447 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06240cda20 con 0x7f06240b5140 2023-03-03T08:02:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.912+0000 7f062959c700 1 --2- 172.21.15.98:0/2279342447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06240b5140 0x7f06240d13a0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f061c009d90 tx=0x7f061c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c70b8ce07288d06 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.912+0000 7f06237fe700 1 -- 172.21.15.98:0/2279342447 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c00e070 con 0x7f06240b5140 2023-03-03T08:02:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.912+0000 7f06237fe700 1 -- 172.21.15.98:0/2279342447 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061c00bb20 con 0x7f06240b5140 2023-03-03T08:02:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.912+0000 7f06237fe700 1 -- 172.21.15.98:0/2279342447 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c012430 con 0x7f06240b5140 2023-03-03T08:02:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.913+0000 7f062afff700 1 -- 172.21.15.98:0/2279342447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06240b5140 msgr2=0x7f06240d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.913+0000 7f062afff700 1 --2- 172.21.15.98:0/2279342447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06240b5140 0x7f06240d13a0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f061c009d90 tx=0x7f061c00b330 comp rx=0 tx=0).stop 2023-03-03T08:02:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.913+0000 7f062afff700 1 -- 172.21.15.98:0/2279342447 shutdown_connections 2023-03-03T08:02:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.913+0000 7f062afff700 1 --2- 172.21.15.98:0/2279342447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06240b5140 0x7f06240d13a0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.913+0000 7f062afff700 1 --2- 172.21.15.98:0/2279342447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06240b26d0 0x7f06240b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.913+0000 7f062afff700 1 -- 172.21.15.98:0/2279342447 >> 172.21.15.98:0/2279342447 conn(0x7f0624073e70 msgr2=0x7f062406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.913+0000 7f062afff700 1 -- 172.21.15.98:0/2279342447 shutdown_connections 2023-03-03T08:02:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.913+0000 7f062afff700 1 -- 172.21.15.98:0/2279342447 wait complete. 2023-03-03T08:02:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.916+0000 7f062afff700 1 Processor -- start 2023-03-03T08:02:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.916+0000 7f062afff700 1 -- start start 2023-03-03T08:02:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.917+0000 7f062afff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06240b26d0 0x7f06240cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.918+0000 7f062afff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06240b5140 0x7f062415cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.918+0000 7f062afff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06240b8d10 0x7f062414a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.918+0000 7f062afff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06240cc840 con 0x7f06240b8d10 2023-03-03T08:02:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.918+0000 7f062afff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0624154b30 con 0x7f06240b5140 2023-03-03T08:02:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.918+0000 7f062afff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06240cbbc0 con 0x7f06240b26d0 2023-03-03T08:02:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.918+0000 7f062959c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06240b8d10 0x7f062414a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.918+0000 7f062959c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06240b8d10 0x7f062414a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60170/0 (socket says 172.21.15.98:60170) 2023-03-03T08:02:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.918+0000 7f062959c700 1 -- 172.21.15.98:0/1603293908 learned_addr learned my addr 172.21.15.98:0/1603293908 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.918+0000 7f0628d9b700 1 --2- 172.21.15.98:0/1603293908 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06240b26d0 0x7f06240cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.918+0000 7f0623fff700 1 --2- 172.21.15.98:0/1603293908 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06240b5140 0x7f062415cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.918+0000 7f062959c700 1 -- 172.21.15.98:0/1603293908 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06240b26d0 msgr2=0x7f06240cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.918+0000 7f062959c700 1 --2- 172.21.15.98:0/1603293908 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06240b26d0 0x7f06240cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.918+0000 7f062959c700 1 -- 172.21.15.98:0/1603293908 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06240b5140 msgr2=0x7f062415cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.918+0000 7f062959c700 1 --2- 172.21.15.98:0/1603293908 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06240b5140 0x7f062415cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.919+0000 7f062959c700 1 -- 172.21.15.98:0/1603293908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061c009a70 con 0x7f06240b8d10 2023-03-03T08:02:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.919+0000 7f0628d9b700 1 --2- 172.21.15.98:0/1603293908 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06240b26d0 0x7f06240cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.919+0000 7f062959c700 1 --2- 172.21.15.98:0/1603293908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06240b8d10 0x7f062414a690 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f061c014040 tx=0x7f061c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.919+0000 7f06217fa700 1 -- 172.21.15.98:0/1603293908 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c0129f0 con 0x7f06240b8d10 2023-03-03T08:02:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.919+0000 7f06217fa700 1 -- 172.21.15.98:0/1603293908 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061c020070 con 0x7f06240b8d10 2023-03-03T08:02:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.919+0000 7f06217fa700 1 -- 172.21.15.98:0/1603293908 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c012db0 con 0x7f06240b8d10 2023-03-03T08:02:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.919+0000 7f062afff700 1 -- 172.21.15.98:0/1603293908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06240d0e80 con 0x7f06240b8d10 2023-03-03T08:02:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.919+0000 7f062afff700 1 -- 172.21.15.98:0/1603293908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06240cdd00 con 0x7f06240b8d10 2023-03-03T08:02:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.920+0000 7f062afff700 1 -- 172.21.15.98:0/1603293908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06240cdfe0 con 0x7f06240b8d10 2023-03-03T08:02:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.922+0000 7f06217fa700 1 -- 172.21.15.98:0/1603293908 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f061c02e070 con 0x7f06240b8d10 2023-03-03T08:02:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.923+0000 7f06217fa700 1 --2- 172.21.15.98:0/1603293908 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0608068380 0x7f060806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.923+0000 7f06217fa700 1 -- 172.21.15.98:0/1603293908 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6896+0+0 (secure 0 0 0) 0x7f061c022070 con 0x7f06240b8d10 2023-03-03T08:02:40.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.923+0000 7f0628d9b700 1 --2- 172.21.15.98:0/1603293908 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0608068380 0x7f060806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:40.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.924+0000 7f0628d9b700 1 --2- 172.21.15.98:0/1603293908 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0608068380 0x7f060806a830 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0618007fd0 tx=0x7f0618007dc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:40.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:40.928+0000 7f06217fa700 1 -- 172.21.15.98:0/1603293908 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f061c0540b0 con 0x7f06240b8d10 2023-03-03T08:02:41.147 INFO:teuthology.orchestra.run.smithi098.stderr:Invalid command: invalid chars @# in abcd@# 2023-03-03T08:02:41.147 INFO:teuthology.orchestra.run.smithi098.stderr:fs subvolumegroup create [] [] [] [] [] : Create a CephFS subvolume group in a volume, and optionally, with a specific data pool layout, and a specific numeric mode 2023-03-03T08:02:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:Error EINVAL: invalid command 2023-03-03T08:02:41.164 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-03-03T08:02:41.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create testvol 'abcd@#' 2023-03-03T08:02:41.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.410+0000 7fe4f73ff700 1 Processor -- start 2023-03-03T08:02:41.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.410+0000 7fe4f73ff700 1 -- start start 2023-03-03T08:02:41.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.412+0000 7fe4f73ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe4f015dd20 0x7fe4f00b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:41.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.412+0000 7fe4f73ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f015e170 con 0x7fe4f00b0800 2023-03-03T08:02:41.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.412+0000 7fe4f73ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f00b1110 con 0x7fe4f00cd5c0 2023-03-03T08:02:41.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.412+0000 7fe4f73ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f00c9d50 con 0x7fe4f015dd20 2023-03-03T08:02:41.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.412+0000 7fe4f519b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe4f00b0800 0x7fe4f00c93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44092/0 (socket says 172.21.15.98:44092) 2023-03-03T08:02:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.412+0000 7fe4f519b700 1 -- 172.21.15.98:0/1244113389 learned_addr learned my addr 172.21.15.98:0/1244113389 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.412+0000 7fe4f499a700 1 --2- 172.21.15.98:0/1244113389 >> v2:172.21.15.100:3300/0 conn(0x7fe4f015dd20 0x7fe4f00b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.413+0000 7fe4e7fff700 1 -- 172.21.15.98:0/1244113389 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3302888458 0 0) 0x7fe4f015e170 con 0x7fe4f00b0800 2023-03-03T08:02:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.413+0000 7fe4e7fff700 1 -- 172.21.15.98:0/1244113389 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4dc0035b0 con 0x7fe4f00b0800 2023-03-03T08:02:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.413+0000 7fe4e7fff700 1 -- 172.21.15.98:0/1244113389 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828205075 0 0) 0x7fe4f00b1110 con 0x7fe4f00cd5c0 2023-03-03T08:02:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.413+0000 7fe4e7fff700 1 -- 172.21.15.98:0/1244113389 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f015e170 con 0x7fe4f00cd5c0 2023-03-03T08:02:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.413+0000 7fe4f499a700 1 -- 172.21.15.98:0/1244113389 >> v1:172.21.15.100:6789/0 conn(0x7fe4f00cd5c0 legacy=0x7fe4f01599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.413+0000 7fe4f499a700 1 -- 172.21.15.98:0/1244113389 >> v1:172.21.15.98:6789/0 conn(0x7fe4f00b0800 legacy=0x7fe4f00c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.413+0000 7fe4f499a700 1 -- 172.21.15.98:0/1244113389 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f00ca030 con 0x7fe4f015dd20 2023-03-03T08:02:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.413+0000 7fe4f499a700 1 --2- 172.21.15.98:0/1244113389 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4f015dd20 0x7fe4f00b0bd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009840 tx=0x7fe4e0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=714b6f5ad9cb7dc3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.413+0000 7fe4e7fff700 1 -- 172.21.15.98:0/1244113389 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0011830 con 0x7fe4f015dd20 2023-03-03T08:02:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.414+0000 7fe4e7fff700 1 -- 172.21.15.98:0/1244113389 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e0011990 con 0x7fe4f015dd20 2023-03-03T08:02:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.414+0000 7fe4e7fff700 1 -- 172.21.15.98:0/1244113389 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0011d20 con 0x7fe4f015dd20 2023-03-03T08:02:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.414+0000 7fe4f73ff700 1 -- 172.21.15.98:0/1244113389 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4f015dd20 msgr2=0x7fe4f00b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.414+0000 7fe4f73ff700 1 --2- 172.21.15.98:0/1244113389 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4f015dd20 0x7fe4f00b0bd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009840 tx=0x7fe4e0009b50 comp rx=0 tx=0).stop 2023-03-03T08:02:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.414+0000 7fe4f73ff700 1 -- 172.21.15.98:0/1244113389 shutdown_connections 2023-03-03T08:02:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.414+0000 7fe4f73ff700 1 --2- 172.21.15.98:0/1244113389 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4f015dd20 0x7fe4f00b0bd0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.414+0000 7fe4f73ff700 1 -- 172.21.15.98:0/1244113389 >> 172.21.15.98:0/1244113389 conn(0x7fe4f00cb800 msgr2=0x7fe4f00cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.414+0000 7fe4f73ff700 1 -- 172.21.15.98:0/1244113389 shutdown_connections 2023-03-03T08:02:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.414+0000 7fe4f73ff700 1 -- 172.21.15.98:0/1244113389 wait complete. 2023-03-03T08:02:41.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.416+0000 7fe4f73ff700 1 Processor -- start 2023-03-03T08:02:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.417+0000 7fe4f73ff700 1 -- start start 2023-03-03T08:02:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.418+0000 7fe4f73ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4f00b0800 0x7fe4f01553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.418+0000 7fe4f73ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4f00cd5c0 0x7fe4f014f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.418+0000 7fe4f73ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4f015dd20 0x7fe4f014ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.418+0000 7fe4f73ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f0157fd0 con 0x7fe4f015dd20 2023-03-03T08:02:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.418+0000 7fe4f73ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f01558e0 con 0x7fe4f00cd5c0 2023-03-03T08:02:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.418+0000 7fe4f73ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f01e31d0 con 0x7fe4f00b0800 2023-03-03T08:02:41.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.418+0000 7fe4f519b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4f00b0800 0x7fe4f01553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:41.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.418+0000 7fe4f599c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4f015dd20 0x7fe4f014ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:41.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.418+0000 7fe4f519b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4f00b0800 0x7fe4f01553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:44270/0 (socket says 172.21.15.98:44270) 2023-03-03T08:02:41.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.418+0000 7fe4f519b700 1 -- 172.21.15.98:0/3455017241 learned_addr learned my addr 172.21.15.98:0/3455017241 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:41.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.418+0000 7fe4f499a700 1 --2- 172.21.15.98:0/3455017241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4f00cd5c0 0x7fe4f014f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.419+0000 7fe4f519b700 1 -- 172.21.15.98:0/3455017241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4f00cd5c0 msgr2=0x7fe4f014f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.419+0000 7fe4f519b700 1 --2- 172.21.15.98:0/3455017241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4f00cd5c0 0x7fe4f014f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.419+0000 7fe4f519b700 1 -- 172.21.15.98:0/3455017241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4f015dd20 msgr2=0x7fe4f014ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.419+0000 7fe4f519b700 1 --2- 172.21.15.98:0/3455017241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4f015dd20 0x7fe4f014ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.419+0000 7fe4f519b700 1 -- 172.21.15.98:0/3455017241 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e00094a0 con 0x7fe4f00b0800 2023-03-03T08:02:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.419+0000 7fe4f599c700 1 --2- 172.21.15.98:0/3455017241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4f015dd20 0x7fe4f014ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.419+0000 7fe4f519b700 1 --2- 172.21.15.98:0/3455017241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4f00b0800 0x7fe4f01553a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec00c730 tx=0x7fe4ec00ca40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.419+0000 7fe4e5ffb700 1 -- 172.21.15.98:0/3455017241 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec005d70 con 0x7fe4f00b0800 2023-03-03T08:02:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.419+0000 7fe4e5ffb700 1 -- 172.21.15.98:0/3455017241 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec0070d0 con 0x7fe4f00b0800 2023-03-03T08:02:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.419+0000 7fe4e5ffb700 1 -- 172.21.15.98:0/3455017241 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec007440 con 0x7fe4f00b0800 2023-03-03T08:02:41.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.419+0000 7fe4f73ff700 1 -- 172.21.15.98:0/3455017241 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f00cd0a0 con 0x7fe4f00b0800 2023-03-03T08:02:41.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.420+0000 7fe4f73ff700 1 -- 172.21.15.98:0/3455017241 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f00ccdc0 con 0x7fe4f00b0800 2023-03-03T08:02:41.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.421+0000 7fe4f73ff700 1 -- 172.21.15.98:0/3455017241 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f00ca310 con 0x7fe4f00b0800 2023-03-03T08:02:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.422+0000 7fe4e5ffb700 1 -- 172.21.15.98:0/3455017241 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fe4ec019030 con 0x7fe4f00b0800 2023-03-03T08:02:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.423+0000 7fe4e5ffb700 1 --2- 172.21.15.98:0/3455017241 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe4d4068380 0x7fe4d406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.423+0000 7fe4e5ffb700 1 -- 172.21.15.98:0/3455017241 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6896+0+0 (secure 0 0 0) 0x7fe4ec0852c0 con 0x7fe4f00b0800 2023-03-03T08:02:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.424+0000 7fe4f499a700 1 --2- 172.21.15.98:0/3455017241 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe4d4068380 0x7fe4d406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.424+0000 7fe4f499a700 1 --2- 172.21.15.98:0/3455017241 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe4d4068380 0x7fe4d406a830 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009fd0 tx=0x7fe4e001c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:41.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.429+0000 7fe4e5ffb700 1 -- 172.21.15.98:0/3455017241 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe4ec04b5e0 con 0x7fe4f00b0800 2023-03-03T08:02:41.667 INFO:teuthology.orchestra.run.smithi098.stderr:Invalid command: invalid chars @# in abcd@# 2023-03-03T08:02:41.668 INFO:teuthology.orchestra.run.smithi098.stderr:fs subvolume create [] [] [] [] [] [] [--namespace-isolated] : Create a CephFS subvolume in a volume, and optionally, with a specific size (in bytes), a specific data pool layout, a specific mode, in a specific subvolume group and in separate RADOS namespace 2023-03-03T08:02:41.668 INFO:teuthology.orchestra.run.smithi098.stderr:Error EINVAL: invalid command 2023-03-03T08:02:41.683 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-03-03T08:02:41.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm testvol --yes-i-really-mean-it 2023-03-03T08:02:41.717 INFO:tasks.ceph.mds.d.smithi100.stderr:2023-03-03T08:02:41.716+0000 7f225d9f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T08:02:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.933+0000 7fa987fff700 1 Processor -- start 2023-03-03T08:02:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.933+0000 7fa987fff700 1 -- start start 2023-03-03T08:02:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.934+0000 7fa987fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa9881609b0 0x7fa988160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:41.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.934+0000 7fa987fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9880b4aa0 0x7fa9880b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:41.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.934+0000 7fa987fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9881612c0 con 0x7fa9880cf570 2023-03-03T08:02:41.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.934+0000 7fa987fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9880cebb0 con 0x7fa9881609b0 2023-03-03T08:02:41.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.934+0000 7fa987fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9880b5460 con 0x7fa9880b4aa0 2023-03-03T08:02:41.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.935+0000 7fa97ffff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa9880cf570 0x7fa98815d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44094/0 (socket says 172.21.15.98:44094) 2023-03-03T08:02:41.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.935+0000 7fa97ffff700 1 -- 172.21.15.98:0/1233864774 learned_addr learned my addr 172.21.15.98:0/1233864774 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:41.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.935+0000 7fa9877fe700 1 --2- 172.21.15.98:0/1233864774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9880b4aa0 0x7fa9880b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:41.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.935+0000 7fa986ffd700 1 --2- 172.21.15.98:0/1233864774 >> v2:172.21.15.100:3300/0 conn(0x7fa9881609b0 0x7fa988160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.935+0000 7fa9867fc700 1 -- 172.21.15.98:0/1233864774 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943774509 0 0) 0x7fa9881612c0 con 0x7fa9880cf570 2023-03-03T08:02:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.935+0000 7fa9867fc700 1 -- 172.21.15.98:0/1233864774 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9740035b0 con 0x7fa9880cf570 2023-03-03T08:02:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.935+0000 7fa9877fe700 1 -- 172.21.15.98:0/1233864774 >> v2:172.21.15.100:3300/0 conn(0x7fa9881609b0 msgr2=0x7fa988160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.935+0000 7fa9877fe700 1 --2- 172.21.15.98:0/1233864774 >> v2:172.21.15.100:3300/0 conn(0x7fa9881609b0 0x7fa988160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.935+0000 7fa9877fe700 1 -- 172.21.15.98:0/1233864774 >> v1:172.21.15.98:6789/0 conn(0x7fa9880cf570 legacy=0x7fa98815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.935+0000 7fa9877fe700 1 -- 172.21.15.98:0/1233864774 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98815dd00 con 0x7fa9880b4aa0 2023-03-03T08:02:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.936+0000 7fa9877fe700 1 --2- 172.21.15.98:0/1233864774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9880b4aa0 0x7fa9880b4f20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa980009d90 tx=0x7fa98000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=362d9989fec0bac3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.936+0000 7fa9867fc700 1 -- 172.21.15.98:0/1233864774 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98000e070 con 0x7fa9880b4aa0 2023-03-03T08:02:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.936+0000 7fa9867fc700 1 -- 172.21.15.98:0/1233864774 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98000bb20 con 0x7fa9880b4aa0 2023-03-03T08:02:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.936+0000 7fa9867fc700 1 -- 172.21.15.98:0/1233864774 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980012430 con 0x7fa9880b4aa0 2023-03-03T08:02:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.937+0000 7fa987fff700 1 -- 172.21.15.98:0/1233864774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9880b4aa0 msgr2=0x7fa9880b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.937+0000 7fa987fff700 1 --2- 172.21.15.98:0/1233864774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9880b4aa0 0x7fa9880b4f20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa980009d90 tx=0x7fa98000b330 comp rx=0 tx=0).stop 2023-03-03T08:02:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.937+0000 7fa987fff700 1 -- 172.21.15.98:0/1233864774 shutdown_connections 2023-03-03T08:02:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.937+0000 7fa987fff700 1 --2- 172.21.15.98:0/1233864774 >> v2:172.21.15.100:3300/0 conn(0x7fa9881609b0 0x7fa988160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.937+0000 7fa987fff700 1 --2- 172.21.15.98:0/1233864774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9880b4aa0 0x7fa9880b4f20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:41.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.937+0000 7fa987fff700 1 -- 172.21.15.98:0/1233864774 >> 172.21.15.98:0/1233864774 conn(0x7fa98806b880 msgr2=0x7fa9880cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:41.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.937+0000 7fa987fff700 1 -- 172.21.15.98:0/1233864774 shutdown_connections 2023-03-03T08:02:41.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.937+0000 7fa987fff700 1 -- 172.21.15.98:0/1233864774 wait complete. 2023-03-03T08:02:41.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.939+0000 7fa987fff700 1 Processor -- start 2023-03-03T08:02:41.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.939+0000 7fa987fff700 1 -- start start 2023-03-03T08:02:41.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.940+0000 7fa987fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9880b4aa0 0x7fa98814f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:41.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.940+0000 7fa987fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9880cf570 0x7fa9880c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:41.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.940+0000 7fa987fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9881609b0 0x7fa98815f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:41.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.940+0000 7fa987fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9880c7090 con 0x7fa9880b4aa0 2023-03-03T08:02:41.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.940+0000 7fa987fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9880c72c0 con 0x7fa9881609b0 2023-03-03T08:02:41.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.940+0000 7fa987fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9880c75a0 con 0x7fa9880cf570 2023-03-03T08:02:41.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.940+0000 7fa986ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9880b4aa0 0x7fa98814f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:41.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.941+0000 7fa986ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9880b4aa0 0x7fa98814f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60186/0 (socket says 172.21.15.98:60186) 2023-03-03T08:02:41.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.941+0000 7fa986ffd700 1 -- 172.21.15.98:0/1271656534 learned_addr learned my addr 172.21.15.98:0/1271656534 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:41.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.941+0000 7fa97ffff700 1 --2- 172.21.15.98:0/1271656534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9880cf570 0x7fa9880c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:41.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.941+0000 7fa9877fe700 1 --2- 172.21.15.98:0/1271656534 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9881609b0 0x7fa98815f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:41.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.941+0000 7fa986ffd700 1 -- 172.21.15.98:0/1271656534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9880cf570 msgr2=0x7fa9880c6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:41.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.941+0000 7fa986ffd700 1 --2- 172.21.15.98:0/1271656534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9880cf570 0x7fa9880c6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:41.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.941+0000 7fa986ffd700 1 -- 172.21.15.98:0/1271656534 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9881609b0 msgr2=0x7fa98815f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:41.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.941+0000 7fa986ffd700 1 --2- 172.21.15.98:0/1271656534 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9881609b0 0x7fa98815f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:41.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.941+0000 7fa986ffd700 1 -- 172.21.15.98:0/1271656534 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa980009a70 con 0x7fa9880b4aa0 2023-03-03T08:02:41.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.941+0000 7fa97ffff700 1 --2- 172.21.15.98:0/1271656534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9880cf570 0x7fa9880c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:41.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.941+0000 7fa986ffd700 1 --2- 172.21.15.98:0/1271656534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9880b4aa0 0x7fa98814f530 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fa97800e870 tx=0x7fa97800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:41.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.942+0000 7fa984ff9700 1 -- 172.21.15.98:0/1271656534 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa978009590 con 0x7fa9880b4aa0 2023-03-03T08:02:41.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.942+0000 7fa984ff9700 1 -- 172.21.15.98:0/1271656534 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97800ee70 con 0x7fa9880b4aa0 2023-03-03T08:02:41.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.942+0000 7fa984ff9700 1 -- 172.21.15.98:0/1271656534 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97800ca50 con 0x7fa9880b4aa0 2023-03-03T08:02:41.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.942+0000 7fa987fff700 1 -- 172.21.15.98:0/1271656534 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9880cda60 con 0x7fa9880b4aa0 2023-03-03T08:02:41.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.942+0000 7fa987fff700 1 -- 172.21.15.98:0/1271656534 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9880cd780 con 0x7fa9880b4aa0 2023-03-03T08:02:41.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.943+0000 7fa987fff700 1 -- 172.21.15.98:0/1271656534 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9880cdd40 con 0x7fa9880b4aa0 2023-03-03T08:02:41.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.945+0000 7fa984ff9700 1 -- 172.21.15.98:0/1271656534 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fa978019030 con 0x7fa9880b4aa0 2023-03-03T08:02:41.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.946+0000 7fa984ff9700 1 --2- 172.21.15.98:0/1271656534 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa968068380 0x7fa96806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:41.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.946+0000 7fa984ff9700 1 -- 172.21.15.98:0/1271656534 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6896+0+0 (secure 0 0 0) 0x7fa978085f70 con 0x7fa9880b4aa0 2023-03-03T08:02:41.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.947+0000 7fa97ffff700 1 --2- 172.21.15.98:0/1271656534 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa968068380 0x7fa96806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:41.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.947+0000 7fa97ffff700 1 --2- 172.21.15.98:0/1271656534 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa968068380 0x7fa96806a830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa9700011e0 tx=0x7fa970009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:41.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:41.950+0000 7fa984ff9700 1 -- 172.21.15.98:0/1271656534 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa9780523d0 con 0x7fa9880b4aa0 2023-03-03T08:02:42.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:42.169+0000 7fa987fff700 1 -- 172.21.15.98:0/1271656534 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "testvol", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9880c81b0 con 0x7fa968068380 2023-03-03T08:02:42.613 INFO:tasks.ceph.mon.a.smithi098.stderr:2023-03-03T08:02:42.612+0000 7fb71abf9700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-03-03T08:02:42.624 INFO:tasks.ceph.mds.d.smithi100.stderr: -24> 2023-03-03T08:02:41.716+0000 7f225d9f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T08:02:42.682 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2023-03-03T08:02:45.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:45.635+0000 7fa984ff9700 1 -- 172.21.15.98:0/1271656534 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+78 (secure 0 0 0) 0x7fa9880c81b0 con 0x7fa968068380 2023-03-03T08:02:45.637 INFO:teuthology.orchestra.run.smithi098.stdout:metadata pool: cephfs.testvol.meta data pool: ['cephfs.testvol.data'] removed 2023-03-03T08:02:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:45.640+0000 7fa987fff700 1 -- 172.21.15.98:0/1271656534 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa968068380 msgr2=0x7fa96806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:45.640+0000 7fa987fff700 1 --2- 172.21.15.98:0/1271656534 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa968068380 0x7fa96806a830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa9700011e0 tx=0x7fa970009040 comp rx=0 tx=0).stop 2023-03-03T08:02:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:45.640+0000 7fa987fff700 1 -- 172.21.15.98:0/1271656534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9880b4aa0 msgr2=0x7fa98814f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:45.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:45.640+0000 7fa987fff700 1 --2- 172.21.15.98:0/1271656534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9880b4aa0 0x7fa98814f530 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fa97800e870 tx=0x7fa97800ec30 comp rx=0 tx=0).stop 2023-03-03T08:02:45.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:45.640+0000 7fa987fff700 1 -- 172.21.15.98:0/1271656534 shutdown_connections 2023-03-03T08:02:45.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:45.640+0000 7fa987fff700 1 --2- 172.21.15.98:0/1271656534 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa968068380 0x7fa96806a830 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:45.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:45.640+0000 7fa987fff700 1 --2- 172.21.15.98:0/1271656534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9880b4aa0 0x7fa98814f530 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:45.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:45.640+0000 7fa987fff700 1 --2- 172.21.15.98:0/1271656534 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9881609b0 0x7fa98815f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:45.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:45.640+0000 7fa987fff700 1 --2- 172.21.15.98:0/1271656534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9880cf570 0x7fa9880c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:45.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:45.640+0000 7fa987fff700 1 -- 172.21.15.98:0/1271656534 >> 172.21.15.98:0/1271656534 conn(0x7fa98806b880 msgr2=0x7fa9880d13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:45.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:45.640+0000 7fa987fff700 1 -- 172.21.15.98:0/1271656534 shutdown_connections 2023-03-03T08:02:45.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:45.640+0000 7fa987fff700 1 -- 172.21.15.98:0/1271656534 wait complete. 2023-03-03T08:02:45.672 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-03-03T08:02:45.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:02:45.808 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:02:45.957 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-03-03T08:02:45.958 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:02:45.958 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:02:46.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:02:46.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.268+0000 7facbefff700 1 Processor -- start 2023-03-03T08:02:46.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.269+0000 7facbefff700 1 -- start start 2023-03-03T08:02:46.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.270+0000 7facbefff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7facb80b87c0 0x7facb80cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:46.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.270+0000 7facbefff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb80d0ef0 con 0x7facb80b5410 2023-03-03T08:02:46.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.270+0000 7facbefff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb80d0600 con 0x7facb80b8b90 2023-03-03T08:02:46.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.270+0000 7facbefff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb80cd970 con 0x7facb80b87c0 2023-03-03T08:02:46.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.271+0000 7facbd59c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7facb80b5410 0x7facb80d13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37214/0 (socket says 172.21.15.98:37214) 2023-03-03T08:02:46.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.271+0000 7facbd59c700 1 -- 172.21.15.98:0/1255259962 learned_addr learned my addr 172.21.15.98:0/1255259962 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:46.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.271+0000 7facbcd9b700 1 --2- 172.21.15.98:0/1255259962 >> v2:172.21.15.100:3300/0 conn(0x7facb80b87c0 0x7facb80cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:46.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.271+0000 7facb77fe700 1 -- 172.21.15.98:0/1255259962 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154027535 0 0) 0x7facb80d0ef0 con 0x7facb80b5410 2023-03-03T08:02:46.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.271+0000 7facb77fe700 1 -- 172.21.15.98:0/1255259962 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faca80035b0 con 0x7facb80b5410 2023-03-03T08:02:46.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.271+0000 7facb77fe700 1 -- 172.21.15.98:0/1255259962 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658957231 0 0) 0x7facb80d0600 con 0x7facb80b8b90 2023-03-03T08:02:46.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.271+0000 7facb77fe700 1 -- 172.21.15.98:0/1255259962 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb80d0ef0 con 0x7facb80b8b90 2023-03-03T08:02:46.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.271+0000 7facbcd9b700 1 -- 172.21.15.98:0/1255259962 >> v1:172.21.15.100:6789/0 conn(0x7facb80b8b90 legacy=0x7facb80b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:46.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.271+0000 7facbcd9b700 1 -- 172.21.15.98:0/1255259962 >> v1:172.21.15.98:6789/0 conn(0x7facb80b5410 legacy=0x7facb80d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:46.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.271+0000 7facb77fe700 1 -- 172.21.15.98:0/1255259962 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2528891792 0 0) 0x7faca80035b0 con 0x7facb80b5410 2023-03-03T08:02:46.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.271+0000 7facbcd9b700 1 -- 172.21.15.98:0/1255259962 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb80cdc50 con 0x7facb80b87c0 2023-03-03T08:02:46.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.272+0000 7facbcd9b700 1 --2- 172.21.15.98:0/1255259962 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facb80b87c0 0x7facb80cffe0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7facac009840 tx=0x7facac009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7201f457f49b66a0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:46.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.272+0000 7facb77fe700 1 -- 172.21.15.98:0/1255259962 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac011830 con 0x7facb80b87c0 2023-03-03T08:02:46.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.272+0000 7facb77fe700 1 -- 172.21.15.98:0/1255259962 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facac011990 con 0x7facb80b87c0 2023-03-03T08:02:46.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.272+0000 7facb77fe700 1 -- 172.21.15.98:0/1255259962 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac011d20 con 0x7facb80b87c0 2023-03-03T08:02:46.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.273+0000 7facbefff700 1 -- 172.21.15.98:0/1255259962 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facb80b87c0 msgr2=0x7facb80cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:46.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.273+0000 7facbefff700 1 --2- 172.21.15.98:0/1255259962 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facb80b87c0 0x7facb80cffe0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7facac009840 tx=0x7facac009b50 comp rx=0 tx=0).stop 2023-03-03T08:02:46.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.273+0000 7facbefff700 1 -- 172.21.15.98:0/1255259962 shutdown_connections 2023-03-03T08:02:46.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.273+0000 7facbefff700 1 --2- 172.21.15.98:0/1255259962 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facb80b87c0 0x7facb80cffe0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.273+0000 7facbefff700 1 -- 172.21.15.98:0/1255259962 >> 172.21.15.98:0/1255259962 conn(0x7facb806aee0 msgr2=0x7facb806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:46.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.273+0000 7facbefff700 1 -- 172.21.15.98:0/1255259962 shutdown_connections 2023-03-03T08:02:46.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.273+0000 7facbefff700 1 -- 172.21.15.98:0/1255259962 wait complete. 2023-03-03T08:02:46.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.275+0000 7facbefff700 1 Processor -- start 2023-03-03T08:02:46.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.275+0000 7facbefff700 1 -- start start 2023-03-03T08:02:46.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.276+0000 7facbefff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb80b5410 0x7facb814ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:46.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.276+0000 7facbefff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facb80b87c0 0x7facb8152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:46.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.276+0000 7facbefff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb80b8b90 0x7facb81f1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:46.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.276+0000 7facbefff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb8152760 con 0x7facb80b8b90 2023-03-03T08:02:46.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.276+0000 7facbefff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb80c6d80 con 0x7facb80b87c0 2023-03-03T08:02:46.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.276+0000 7facbefff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb80c78c0 con 0x7facb80b5410 2023-03-03T08:02:46.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.277+0000 7facbd59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb80b8b90 0x7facb81f1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:46.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.277+0000 7facbd59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb80b8b90 0x7facb81f1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52772/0 (socket says 172.21.15.98:52772) 2023-03-03T08:02:46.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.277+0000 7facbd59c700 1 -- 172.21.15.98:0/3251987149 learned_addr learned my addr 172.21.15.98:0/3251987149 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:46.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.277+0000 7facb7fff700 1 --2- 172.21.15.98:0/3251987149 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facb80b87c0 0x7facb8152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:46.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.277+0000 7facbcd9b700 1 --2- 172.21.15.98:0/3251987149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb80b5410 0x7facb814ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:46.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.277+0000 7facbd59c700 1 -- 172.21.15.98:0/3251987149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb80b5410 msgr2=0x7facb814ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:46.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.277+0000 7facbd59c700 1 --2- 172.21.15.98:0/3251987149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb80b5410 0x7facb814ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.277+0000 7facbd59c700 1 -- 172.21.15.98:0/3251987149 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facb80b87c0 msgr2=0x7facb8152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:46.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.277+0000 7facbd59c700 1 --2- 172.21.15.98:0/3251987149 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facb80b87c0 0x7facb8152220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.277+0000 7facbd59c700 1 -- 172.21.15.98:0/3251987149 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac0094a0 con 0x7facb80b8b90 2023-03-03T08:02:46.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.277+0000 7facb7fff700 1 --2- 172.21.15.98:0/3251987149 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facb80b87c0 0x7facb8152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:46.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.277+0000 7facbd59c700 1 --2- 172.21.15.98:0/3251987149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb80b8b90 0x7facb81f1920 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7facb000cad0 tx=0x7facb000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:46.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.278+0000 7facb57fa700 1 -- 172.21.15.98:0/3251987149 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb000a9e0 con 0x7facb80b8b90 2023-03-03T08:02:46.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.278+0000 7facb57fa700 1 -- 172.21.15.98:0/3251987149 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb000ab40 con 0x7facb80b8b90 2023-03-03T08:02:46.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.278+0000 7facb57fa700 1 -- 172.21.15.98:0/3251987149 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb0012400 con 0x7facb80b8b90 2023-03-03T08:02:46.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.278+0000 7facbefff700 1 -- 172.21.15.98:0/3251987149 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb80cdf30 con 0x7facb80b8b90 2023-03-03T08:02:46.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.278+0000 7facbefff700 1 -- 172.21.15.98:0/3251987149 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb8073a80 con 0x7facb80b8b90 2023-03-03T08:02:46.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.279+0000 7facbefff700 1 -- 172.21.15.98:0/3251987149 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb80ce0c0 con 0x7facb80b8b90 2023-03-03T08:02:46.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.280+0000 7facb57fa700 1 -- 172.21.15.98:0/3251987149 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7facb0010070 con 0x7facb80b8b90 2023-03-03T08:02:46.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.281+0000 7facb57fa700 1 --2- 172.21.15.98:0/3251987149 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fac9c068380 0x7fac9c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:46.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.282+0000 7facb57fa700 1 -- 172.21.15.98:0/3251987149 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7facb0085e70 con 0x7facb80b8b90 2023-03-03T08:02:46.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.282+0000 7facbcd9b700 1 --2- 172.21.15.98:0/3251987149 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fac9c068380 0x7fac9c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:46.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.283+0000 7facbcd9b700 1 --2- 172.21.15.98:0/3251987149 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fac9c068380 0x7fac9c06a830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7facac009fd0 tx=0x7facac01c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:46.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.286+0000 7facb57fa700 1 -- 172.21.15.98:0/3251987149 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7facb0052590 con 0x7facb80b8b90 2023-03-03T08:02:46.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.486+0000 7facbefff700 1 -- 172.21.15.98:0/3251987149 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facb80729b0 con 0x7facb80b8b90 2023-03-03T08:02:46.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.488+0000 7facb57fa700 1 -- 172.21.15.98:0/3251987149 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+18053 (secure 0 0 0) 0x7facb0055bb0 con 0x7facb80b8b90 2023-03-03T08:02:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:46.490 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":98,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:02:45.624487+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:02:20.797579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-03-03T08:02:21.411899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:6835/1843752855":"2023-03-04T08:02:42.175049+0000","172.21.15.100:6833/1843752855":"2023-03-04T08:02:42.175049+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:02:46.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.491+0000 7facbefff700 1 -- 172.21.15.98:0/3251987149 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fac9c068380 msgr2=0x7fac9c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:46.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.491+0000 7facbefff700 1 --2- 172.21.15.98:0/3251987149 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fac9c068380 0x7fac9c06a830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7facac009fd0 tx=0x7facac01c040 comp rx=0 tx=0).stop 2023-03-03T08:02:46.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.491+0000 7facbefff700 1 -- 172.21.15.98:0/3251987149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb80b8b90 msgr2=0x7facb81f1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:46.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.491+0000 7facbefff700 1 --2- 172.21.15.98:0/3251987149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb80b8b90 0x7facb81f1920 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7facb000cad0 tx=0x7facb000cde0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.491+0000 7facbefff700 1 -- 172.21.15.98:0/3251987149 shutdown_connections 2023-03-03T08:02:46.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.491+0000 7facbefff700 1 --2- 172.21.15.98:0/3251987149 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fac9c068380 0x7fac9c06a830 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.491+0000 7facbefff700 1 --2- 172.21.15.98:0/3251987149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb80b8b90 0x7facb81f1920 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.491+0000 7facbefff700 1 --2- 172.21.15.98:0/3251987149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb80b5410 0x7facb814ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.491+0000 7facbefff700 1 --2- 172.21.15.98:0/3251987149 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facb80b87c0 0x7facb8152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.491+0000 7facbefff700 1 -- 172.21.15.98:0/3251987149 >> 172.21.15.98:0/3251987149 conn(0x7facb806aee0 msgr2=0x7facb806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:46.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.491+0000 7facbefff700 1 -- 172.21.15.98:0/3251987149 shutdown_connections 2023-03-03T08:02:46.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.491+0000 7facbefff700 1 -- 172.21.15.98:0/3251987149 wait complete. 2023-03-03T08:02:46.521 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T08:02:20.797579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T08:02:21.411899+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:02:46.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:02:46.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.770+0000 7f4e9e5ff700 1 Processor -- start 2023-03-03T08:02:46.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.770+0000 7f4e9e5ff700 1 -- start start 2023-03-03T08:02:46.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.771+0000 7f4e9e5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e980b0800 0x7f4e9815dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:46.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.771+0000 7f4e9e5ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4e980cd5c0 0x7f4e981599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:46.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.771+0000 7f4e9e5ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e980c9d50 con 0x7f4e980c93d0 2023-03-03T08:02:46.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.772+0000 7f4e9e5ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e980ca030 con 0x7f4e980cd5c0 2023-03-03T08:02:46.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.772+0000 7f4e9e5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e980ccdc0 con 0x7f4e980b0800 2023-03-03T08:02:46.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.772+0000 7f4e9cb9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4e980cd5c0 0x7f4e981599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:46.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.772+0000 7f4e8ffff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4e980c93d0 0x7f4e980b0bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37224/0 (socket says 172.21.15.98:37224) 2023-03-03T08:02:46.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.772+0000 7f4e8ffff700 1 -- 172.21.15.98:0/1023715246 learned_addr learned my addr 172.21.15.98:0/1023715246 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:46.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.772+0000 7f4e97fff700 1 --2- 172.21.15.98:0/1023715246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e980b0800 0x7f4e9815dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:46.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.772+0000 7f4e9cb9c700 1 -- 172.21.15.98:0/1023715246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e980b0800 msgr2=0x7f4e9815dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:46.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.772+0000 7f4e9cb9c700 1 --2- 172.21.15.98:0/1023715246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e980b0800 0x7f4e9815dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.772+0000 7f4e9cb9c700 1 -- 172.21.15.98:0/1023715246 >> v1:172.21.15.98:6789/0 conn(0x7f4e980c93d0 legacy=0x7f4e980b0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:46.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.772+0000 7f4e9cb9c700 1 -- 172.21.15.98:0/1023715246 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e980cd0a0 con 0x7f4e980cd5c0 2023-03-03T08:02:46.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.772+0000 7f4e977fe700 1 -- 172.21.15.98:0/1023715246 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132403495 0 0) 0x7f4e980c9d50 con 0x7f4e980c93d0 2023-03-03T08:02:46.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.773+0000 7f4e9cb9c700 1 --2- 172.21.15.98:0/1023715246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e980cd5c0 0x7f4e981599a0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f4e90009d90 tx=0x7f4e9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42bd12f98b5e73ad server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:46.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.773+0000 7f4e977fe700 1 -- 172.21.15.98:0/1023715246 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9000e070 con 0x7f4e980cd5c0 2023-03-03T08:02:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.773+0000 7f4e977fe700 1 -- 172.21.15.98:0/1023715246 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9000bbd0 con 0x7f4e980cd5c0 2023-03-03T08:02:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.773+0000 7f4e977fe700 1 -- 172.21.15.98:0/1023715246 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e90012430 con 0x7f4e980cd5c0 2023-03-03T08:02:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.774+0000 7f4e9e5ff700 1 -- 172.21.15.98:0/1023715246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e980cd5c0 msgr2=0x7f4e981599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.774+0000 7f4e9e5ff700 1 --2- 172.21.15.98:0/1023715246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e980cd5c0 0x7f4e981599a0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f4e90009d90 tx=0x7f4e9000b330 comp rx=0 tx=0).stop 2023-03-03T08:02:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.774+0000 7f4e9e5ff700 1 -- 172.21.15.98:0/1023715246 shutdown_connections 2023-03-03T08:02:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.774+0000 7f4e9e5ff700 1 --2- 172.21.15.98:0/1023715246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e980cd5c0 0x7f4e981599a0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.774+0000 7f4e9e5ff700 1 --2- 172.21.15.98:0/1023715246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e980b0800 0x7f4e9815dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.774+0000 7f4e9e5ff700 1 -- 172.21.15.98:0/1023715246 >> 172.21.15.98:0/1023715246 conn(0x7f4e980cb800 msgr2=0x7f4e980cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:46.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.774+0000 7f4e9e5ff700 1 -- 172.21.15.98:0/1023715246 shutdown_connections 2023-03-03T08:02:46.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.774+0000 7f4e9e5ff700 1 -- 172.21.15.98:0/1023715246 wait complete. 2023-03-03T08:02:46.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.776+0000 7f4e9e5ff700 1 Processor -- start 2023-03-03T08:02:46.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.776+0000 7f4e9e5ff700 1 -- start start 2023-03-03T08:02:46.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.777+0000 7f4e9e5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e980b0800 0x7f4e98157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:46.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.777+0000 7f4e9e5ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4e980c93d0 0x7f4e98153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:46.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.777+0000 7f4e9e5ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e980cd5c0 0x7f4e9806b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:46.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.777+0000 7f4e9e5ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e981f6000 con 0x7f4e980cd5c0 2023-03-03T08:02:46.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.777+0000 7f4e9e5ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e98153fe0 con 0x7f4e980c93d0 2023-03-03T08:02:46.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.777+0000 7f4e9e5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e98154e20 con 0x7f4e980b0800 2023-03-03T08:02:46.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.777+0000 7f4e9cb9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e980cd5c0 0x7f4e9806b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:46.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.777+0000 7f4e9cb9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e980cd5c0 0x7f4e9806b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52796/0 (socket says 172.21.15.98:52796) 2023-03-03T08:02:46.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.777+0000 7f4e9cb9c700 1 -- 172.21.15.98:0/4248021496 learned_addr learned my addr 172.21.15.98:0/4248021496 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:46.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.777+0000 7f4e97fff700 1 --2- 172.21.15.98:0/4248021496 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e980b0800 0x7f4e98157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:46.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.778+0000 7f4e8ffff700 1 --2- 172.21.15.98:0/4248021496 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4e980c93d0 0x7f4e98153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:46.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.778+0000 7f4e9cb9c700 1 -- 172.21.15.98:0/4248021496 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e980b0800 msgr2=0x7f4e98157fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:46.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.778+0000 7f4e9cb9c700 1 --2- 172.21.15.98:0/4248021496 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e980b0800 0x7f4e98157fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.778+0000 7f4e9cb9c700 1 -- 172.21.15.98:0/4248021496 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4e980c93d0 msgr2=0x7f4e98153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:46.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.778+0000 7f4e9cb9c700 1 --2- 172.21.15.98:0/4248021496 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4e980c93d0 0x7f4e98153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.778+0000 7f4e9cb9c700 1 -- 172.21.15.98:0/4248021496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e90009a70 con 0x7f4e980cd5c0 2023-03-03T08:02:46.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.778+0000 7f4e8ffff700 1 --2- 172.21.15.98:0/4248021496 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4e980c93d0 0x7f4e98153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:46.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.778+0000 7f4e9cb9c700 1 --2- 172.21.15.98:0/4248021496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e980cd5c0 0x7f4e9806b820 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000b730 tx=0x7f4e90004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:46.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.779+0000 7f4e957fa700 1 -- 172.21.15.98:0/4248021496 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e90012430 con 0x7f4e980cd5c0 2023-03-03T08:02:46.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.779+0000 7f4e957fa700 1 -- 172.21.15.98:0/4248021496 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e90020070 con 0x7f4e980cd5c0 2023-03-03T08:02:46.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.779+0000 7f4e957fa700 1 -- 172.21.15.98:0/4248021496 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e90004280 con 0x7f4e980cd5c0 2023-03-03T08:02:46.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.779+0000 7f4e9e5ff700 1 -- 172.21.15.98:0/4248021496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e980ca310 con 0x7f4e980cd5c0 2023-03-03T08:02:46.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.779+0000 7f4e9e5ff700 1 -- 172.21.15.98:0/4248021496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e980c9820 con 0x7f4e980cd5c0 2023-03-03T08:02:46.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.780+0000 7f4e9e5ff700 1 -- 172.21.15.98:0/4248021496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e980ca4a0 con 0x7f4e980cd5c0 2023-03-03T08:02:46.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.781+0000 7f4e957fa700 1 -- 172.21.15.98:0/4248021496 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f4e900043e0 con 0x7f4e980cd5c0 2023-03-03T08:02:46.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.782+0000 7f4e957fa700 1 --2- 172.21.15.98:0/4248021496 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4e78068380 0x7f4e7806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:46.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.783+0000 7f4e957fa700 1 -- 172.21.15.98:0/4248021496 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4e90022070 con 0x7f4e980cd5c0 2023-03-03T08:02:46.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.783+0000 7f4e97fff700 1 --2- 172.21.15.98:0/4248021496 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4e78068380 0x7f4e7806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:46.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.784+0000 7f4e97fff700 1 --2- 172.21.15.98:0/4248021496 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4e78068380 0x7f4e7806a830 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4e88006b20 tx=0x7f4e88008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:46.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.786+0000 7f4e957fa700 1 -- 172.21.15.98:0/4248021496 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4e900548e0 con 0x7f4e980cd5c0 2023-03-03T08:02:46.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.981+0000 7f4e9e5ff700 1 -- 172.21.15.98:0/4248021496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4e98154fb0 con 0x7f4e980cd5c0 2023-03-03T08:02:46.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.981+0000 7f4e957fa700 1 -- 172.21.15.98:0/4248021496 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4e90086640 con 0x7f4e980cd5c0 2023-03-03T08:02:46.984 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:02:46.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.984+0000 7f4e9e5ff700 1 -- 172.21.15.98:0/4248021496 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4e78068380 msgr2=0x7f4e7806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.984+0000 7f4e9e5ff700 1 --2- 172.21.15.98:0/4248021496 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4e78068380 0x7f4e7806a830 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4e88006b20 tx=0x7f4e88008040 comp rx=0 tx=0).stop 2023-03-03T08:02:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.984+0000 7f4e9e5ff700 1 -- 172.21.15.98:0/4248021496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e980cd5c0 msgr2=0x7f4e9806b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.984+0000 7f4e9e5ff700 1 --2- 172.21.15.98:0/4248021496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e980cd5c0 0x7f4e9806b820 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000b730 tx=0x7f4e90004ad0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.984+0000 7f4e9e5ff700 1 -- 172.21.15.98:0/4248021496 shutdown_connections 2023-03-03T08:02:46.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.984+0000 7f4e9e5ff700 1 --2- 172.21.15.98:0/4248021496 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4e78068380 0x7f4e7806a830 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.984+0000 7f4e9e5ff700 1 --2- 172.21.15.98:0/4248021496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e980cd5c0 0x7f4e9806b820 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.984+0000 7f4e9e5ff700 1 --2- 172.21.15.98:0/4248021496 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e980b0800 0x7f4e98157fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.984+0000 7f4e9e5ff700 1 --2- 172.21.15.98:0/4248021496 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4e980c93d0 0x7f4e98153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:46.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.984+0000 7f4e9e5ff700 1 -- 172.21.15.98:0/4248021496 >> 172.21.15.98:0/4248021496 conn(0x7f4e980cb800 msgr2=0x7f4e980afad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:46.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.984+0000 7f4e9e5ff700 1 -- 172.21.15.98:0/4248021496 shutdown_connections 2023-03-03T08:02:46.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:46.984+0000 7f4e9e5ff700 1 -- 172.21.15.98:0/4248021496 wait complete. 2023-03-03T08:02:47.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:02:47.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.261+0000 7fd80ad9d700 1 Processor -- start 2023-03-03T08:02:47.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.262+0000 7fd80ad9d700 1 -- start start 2023-03-03T08:02:47.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.263+0000 7fd80ad9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd8040cf570 0x7fd80415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:47.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.263+0000 7fd80ad9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd80415db30 0x7fd8040b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:47.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.263+0000 7fd80ad9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8040cd780 con 0x7fd8041609b0 2023-03-03T08:02:47.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.263+0000 7fd80ad9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8040cebb0 con 0x7fd80415db30 2023-03-03T08:02:47.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.263+0000 7fd80ad9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8040cda60 con 0x7fd8040cf570 2023-03-03T08:02:47.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.263+0000 7fd80a59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd80415db30 0x7fd8040b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:47.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.263+0000 7fd80959a700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd8040cf570 0x7fd80415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:47.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.263+0000 7fd80a59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd80415db30 0x7fd8040b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52810/0 (socket says 172.21.15.98:52810) 2023-03-03T08:02:47.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.263+0000 7fd80a59c700 1 -- 172.21.15.98:0/3788206284 learned_addr learned my addr 172.21.15.98:0/3788206284 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:47.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.263+0000 7fd809d9b700 1 --1- 172.21.15.98:0/3788206284 >> v1:172.21.15.100:6789/0 conn(0x7fd8041609b0 0x7fd804160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:50276/0 (socket says 172.21.15.98:50276) 2023-03-03T08:02:47.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.264+0000 7fd80a59c700 1 -- 172.21.15.98:0/3788206284 >> v2:172.21.15.100:3300/0 conn(0x7fd8040cf570 msgr2=0x7fd80415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:47.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.264+0000 7fd80a59c700 1 --2- 172.21.15.98:0/3788206284 >> v2:172.21.15.100:3300/0 conn(0x7fd8040cf570 0x7fd80415d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.264+0000 7fd80a59c700 1 -- 172.21.15.98:0/3788206284 >> v1:172.21.15.100:6789/0 conn(0x7fd8041609b0 legacy=0x7fd804160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:47.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.264+0000 7fd80a59c700 1 -- 172.21.15.98:0/3788206284 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8040cdd40 con 0x7fd80415db30 2023-03-03T08:02:47.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.264+0000 7fd80a59c700 1 --2- 172.21.15.98:0/3788206284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd80415db30 0x7fd8040b4aa0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fd8040cf0c0 tx=0x7fd7fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deabfabe5f096b76 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:47.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.264+0000 7fd808d99700 1 -- 172.21.15.98:0/3788206284 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc00e070 con 0x7fd80415db30 2023-03-03T08:02:47.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.265+0000 7fd808d99700 1 -- 172.21.15.98:0/3788206284 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7fc00b910 con 0x7fd80415db30 2023-03-03T08:02:47.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.265+0000 7fd808d99700 1 -- 172.21.15.98:0/3788206284 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc00bcc0 con 0x7fd80415db30 2023-03-03T08:02:47.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.265+0000 7fd80ad9d700 1 -- 172.21.15.98:0/3788206284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd80415db30 msgr2=0x7fd8040b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:47.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.265+0000 7fd80ad9d700 1 --2- 172.21.15.98:0/3788206284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd80415db30 0x7fd8040b4aa0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fd8040cf0c0 tx=0x7fd7fc00b330 comp rx=0 tx=0).stop 2023-03-03T08:02:47.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.265+0000 7fd80ad9d700 1 -- 172.21.15.98:0/3788206284 shutdown_connections 2023-03-03T08:02:47.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.265+0000 7fd80ad9d700 1 --2- 172.21.15.98:0/3788206284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd80415db30 0x7fd8040b4aa0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.265+0000 7fd80ad9d700 1 --2- 172.21.15.98:0/3788206284 >> v2:172.21.15.100:3300/0 conn(0x7fd8040cf570 0x7fd80415d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.265+0000 7fd80ad9d700 1 -- 172.21.15.98:0/3788206284 >> 172.21.15.98:0/3788206284 conn(0x7fd80406b880 msgr2=0x7fd8040cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:47.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.265+0000 7fd80ad9d700 1 -- 172.21.15.98:0/3788206284 shutdown_connections 2023-03-03T08:02:47.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.265+0000 7fd80ad9d700 1 -- 172.21.15.98:0/3788206284 wait complete. 2023-03-03T08:02:47.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.267+0000 7fd80ad9d700 1 Processor -- start 2023-03-03T08:02:47.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.268+0000 7fd80ad9d700 1 -- start start 2023-03-03T08:02:47.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.269+0000 7fd80ad9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8040cf570 0x7fd8040cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:47.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.269+0000 7fd80ad9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd80415db30 0x7fd8041482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:47.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.269+0000 7fd80ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8041609b0 0x7fd8040d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:47.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.269+0000 7fd80ad9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8040b7430 con 0x7fd8041609b0 2023-03-03T08:02:47.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.269+0000 7fd80ad9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8040b7710 con 0x7fd8040cf570 2023-03-03T08:02:47.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.269+0000 7fd80ad9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8040b79f0 con 0x7fd80415db30 2023-03-03T08:02:47.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.269+0000 7fd80a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8041609b0 0x7fd8040d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:47.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.269+0000 7fd80a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8041609b0 0x7fd8040d17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52820/0 (socket says 172.21.15.98:52820) 2023-03-03T08:02:47.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.269+0000 7fd80a59c700 1 -- 172.21.15.98:0/492713530 learned_addr learned my addr 172.21.15.98:0/492713530 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:47.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.269+0000 7fd809d9b700 1 --2- 172.21.15.98:0/492713530 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8040cf570 0x7fd8040cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:47.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.269+0000 7fd80959a700 1 --2- 172.21.15.98:0/492713530 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd80415db30 0x7fd8041482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:47.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.270+0000 7fd80a59c700 1 -- 172.21.15.98:0/492713530 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd80415db30 msgr2=0x7fd8041482e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:47.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.270+0000 7fd80a59c700 1 --2- 172.21.15.98:0/492713530 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd80415db30 0x7fd8041482e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.270+0000 7fd80a59c700 1 -- 172.21.15.98:0/492713530 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8040cf570 msgr2=0x7fd8040cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:47.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.270+0000 7fd80a59c700 1 --2- 172.21.15.98:0/492713530 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8040cf570 0x7fd8040cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.270+0000 7fd80a59c700 1 -- 172.21.15.98:0/492713530 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7fc009b30 con 0x7fd8041609b0 2023-03-03T08:02:47.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.270+0000 7fd80a59c700 1 --2- 172.21.15.98:0/492713530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8041609b0 0x7fd8040d17d0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00b370 tx=0x7fd7fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:47.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.270+0000 7fd7fa7fc700 1 -- 172.21.15.98:0/492713530 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc0099b0 con 0x7fd8041609b0 2023-03-03T08:02:47.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.270+0000 7fd7fa7fc700 1 -- 172.21.15.98:0/492713530 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7fc0092e0 con 0x7fd8041609b0 2023-03-03T08:02:47.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.270+0000 7fd80ad9d700 1 -- 172.21.15.98:0/492713530 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd80415d0d0 con 0x7fd8041609b0 2023-03-03T08:02:47.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.270+0000 7fd7fa7fc700 1 -- 172.21.15.98:0/492713530 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc01c660 con 0x7fd8041609b0 2023-03-03T08:02:47.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.271+0000 7fd80ad9d700 1 -- 172.21.15.98:0/492713530 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd80415cdf0 con 0x7fd8041609b0 2023-03-03T08:02:47.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.272+0000 7fd80ad9d700 1 -- 172.21.15.98:0/492713530 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80415d5f0 con 0x7fd8041609b0 2023-03-03T08:02:47.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.273+0000 7fd7fa7fc700 1 -- 172.21.15.98:0/492713530 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fd7fc022030 con 0x7fd8041609b0 2023-03-03T08:02:47.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.275+0000 7fd7fa7fc700 1 --2- 172.21.15.98:0/492713530 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd7e8068380 0x7fd7e806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:47.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.275+0000 7fd7fa7fc700 1 -- 172.21.15.98:0/492713530 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd7fc087700 con 0x7fd8041609b0 2023-03-03T08:02:47.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.275+0000 7fd809d9b700 1 --2- 172.21.15.98:0/492713530 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd7e8068380 0x7fd7e806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:47.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.276+0000 7fd809d9b700 1 --2- 172.21.15.98:0/492713530 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd7e8068380 0x7fd7e806a830 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd800000f60 tx=0x7fd800007d10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:47.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.278+0000 7fd7fa7fc700 1 -- 172.21.15.98:0/492713530 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd7fc053e20 con 0x7fd8041609b0 2023-03-03T08:02:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.472+0000 7fd80ad9d700 1 -- 172.21.15.98:0/492713530 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd804067d20 con 0x7fd8041609b0 2023-03-03T08:02:47.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.472+0000 7fd7fa7fc700 1 -- 172.21.15.98:0/492713530 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v98) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd7fc085b80 con 0x7fd8041609b0 2023-03-03T08:02:47.474 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:02:47.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.475+0000 7fd80ad9d700 1 -- 172.21.15.98:0/492713530 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd7e8068380 msgr2=0x7fd7e806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:47.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.475+0000 7fd80ad9d700 1 --2- 172.21.15.98:0/492713530 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd7e8068380 0x7fd7e806a830 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd800000f60 tx=0x7fd800007d10 comp rx=0 tx=0).stop 2023-03-03T08:02:47.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.475+0000 7fd80ad9d700 1 -- 172.21.15.98:0/492713530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8041609b0 msgr2=0x7fd8040d17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:47.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.475+0000 7fd80ad9d700 1 --2- 172.21.15.98:0/492713530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8041609b0 0x7fd8040d17d0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00b370 tx=0x7fd7fc004ad0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.475+0000 7fd80ad9d700 1 -- 172.21.15.98:0/492713530 shutdown_connections 2023-03-03T08:02:47.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.475+0000 7fd80ad9d700 1 --2- 172.21.15.98:0/492713530 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd7e8068380 0x7fd7e806a830 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.475+0000 7fd80ad9d700 1 --2- 172.21.15.98:0/492713530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8041609b0 0x7fd8040d17d0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.475+0000 7fd80ad9d700 1 --2- 172.21.15.98:0/492713530 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8040cf570 0x7fd8040cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.475+0000 7fd80ad9d700 1 --2- 172.21.15.98:0/492713530 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd80415db30 0x7fd8041482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.475+0000 7fd80ad9d700 1 -- 172.21.15.98:0/492713530 >> 172.21.15.98:0/492713530 conn(0x7fd80406b880 msgr2=0x7fd8040731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:47.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.475+0000 7fd80ad9d700 1 -- 172.21.15.98:0/492713530 shutdown_connections 2023-03-03T08:02:47.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.475+0000 7fd80ad9d700 1 -- 172.21.15.98:0/492713530 wait complete. 2023-03-03T08:02:47.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T08:02:47.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.753+0000 7f62b0fff700 1 Processor -- start 2023-03-03T08:02:47.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.754+0000 7f62b0fff700 1 -- start start 2023-03-03T08:02:47.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.755+0000 7f62b0fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ac0af220 0x7f62ac06a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:47.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.755+0000 7f62b0fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f62ac06a870 0x7f62ac0cec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:47.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.755+0000 7f62b0fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f62ac0cf170 0x7f62ac0b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:47.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.755+0000 7f62b0fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62ac0cfd00 con 0x7f62ac0cf170 2023-03-03T08:02:47.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.755+0000 7f62b0fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62ac06acc0 con 0x7f62ac06a870 2023-03-03T08:02:47.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.755+0000 7f62b0fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0cacb0 con 0x7f62ac0af220 2023-03-03T08:02:47.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.755+0000 7f62aad9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f62ac0cf170 0x7f62ac0b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:47.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.755+0000 7f62aad9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f62ac0cf170 0x7f62ac0b18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52836/0 (socket says 172.21.15.98:52836) 2023-03-03T08:02:47.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.755+0000 7f62aad9d700 1 -- 172.21.15.98:0/517127945 learned_addr learned my addr 172.21.15.98:0/517127945 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:47.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.756+0000 7f62a9d9b700 1 --2- 172.21.15.98:0/517127945 >> v2:172.21.15.100:3300/0 conn(0x7f62ac06a870 0x7f62ac0cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:47.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.756+0000 7f62aa59c700 1 --2- 172.21.15.98:0/517127945 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ac0af220 0x7f62ac06a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:47.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.756+0000 7f62aad9d700 1 -- 172.21.15.98:0/517127945 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ac0af220 msgr2=0x7f62ac06a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:47.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.756+0000 7f62aad9d700 1 --2- 172.21.15.98:0/517127945 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ac0af220 0x7f62ac06a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.756+0000 7f62aad9d700 1 -- 172.21.15.98:0/517127945 >> v2:172.21.15.100:3300/0 conn(0x7f62ac06a870 msgr2=0x7f62ac0cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:47.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.756+0000 7f62aad9d700 1 --2- 172.21.15.98:0/517127945 >> v2:172.21.15.100:3300/0 conn(0x7f62ac06a870 0x7f62ac0cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.756+0000 7f62aad9d700 1 -- 172.21.15.98:0/517127945 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac0cfa20 con 0x7f62ac0cf170 2023-03-03T08:02:47.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.756+0000 7f62aad9d700 1 --2- 172.21.15.98:0/517127945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62ac0cf170 0x7f62ac0b18f0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0b2970 tx=0x7f62a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3508ecd865c7f9c8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:47.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.757+0000 7f62a959a700 1 -- 172.21.15.98:0/517127945 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a400e070 con 0x7f62ac0cf170 2023-03-03T08:02:47.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.757+0000 7f62a959a700 1 -- 172.21.15.98:0/517127945 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a400b910 con 0x7f62ac0cf170 2023-03-03T08:02:47.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.757+0000 7f62a959a700 1 -- 172.21.15.98:0/517127945 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a400bcc0 con 0x7f62ac0cf170 2023-03-03T08:02:47.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.757+0000 7f62b0fff700 1 -- 172.21.15.98:0/517127945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62ac0cf170 msgr2=0x7f62ac0b18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:47.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.757+0000 7f62b0fff700 1 --2- 172.21.15.98:0/517127945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62ac0cf170 0x7f62ac0b18f0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0b2970 tx=0x7f62a400b330 comp rx=0 tx=0).stop 2023-03-03T08:02:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.757+0000 7f62b0fff700 1 -- 172.21.15.98:0/517127945 shutdown_connections 2023-03-03T08:02:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.757+0000 7f62b0fff700 1 --2- 172.21.15.98:0/517127945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62ac0cf170 0x7f62ac0b18f0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.757+0000 7f62b0fff700 1 --2- 172.21.15.98:0/517127945 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ac0af220 0x7f62ac06a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.757+0000 7f62b0fff700 1 --2- 172.21.15.98:0/517127945 >> v2:172.21.15.100:3300/0 conn(0x7f62ac06a870 0x7f62ac0cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.757+0000 7f62b0fff700 1 -- 172.21.15.98:0/517127945 >> 172.21.15.98:0/517127945 conn(0x7f62ac0cc8f0 msgr2=0x7f62ac0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.758+0000 7f62b0fff700 1 -- 172.21.15.98:0/517127945 shutdown_connections 2023-03-03T08:02:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.758+0000 7f62b0fff700 1 -- 172.21.15.98:0/517127945 wait complete. 2023-03-03T08:02:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.760+0000 7f62b0fff700 1 Processor -- start 2023-03-03T08:02:47.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.761+0000 7f62b0fff700 1 -- start start 2023-03-03T08:02:47.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.762+0000 7f62b0fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f62ac06a870 0x7f62ac151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:47.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.762+0000 7f62b0fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ac0af220 0x7f62ac0c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:47.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.762+0000 7f62b0fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62ac0cf170 0x7f62ac1f20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:47.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.762+0000 7f62b0fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0c7480 con 0x7f62ac0cf170 2023-03-03T08:02:47.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.762+0000 7f62b0fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0b5790 con 0x7f62ac06a870 2023-03-03T08:02:47.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.762+0000 7f62b0fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0b5a70 con 0x7f62ac0af220 2023-03-03T08:02:47.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.762+0000 7f62aad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62ac0cf170 0x7f62ac1f20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:47.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.763+0000 7f62aad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62ac0cf170 0x7f62ac1f20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52848/0 (socket says 172.21.15.98:52848) 2023-03-03T08:02:47.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.763+0000 7f62aad9d700 1 -- 172.21.15.98:0/1545659542 learned_addr learned my addr 172.21.15.98:0/1545659542 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:47.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.763+0000 7f62aa59c700 1 --2- 172.21.15.98:0/1545659542 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f62ac06a870 0x7f62ac151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:47.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.763+0000 7f62a9d9b700 1 --2- 172.21.15.98:0/1545659542 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ac0af220 0x7f62ac0c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:47.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.763+0000 7f62aad9d700 1 -- 172.21.15.98:0/1545659542 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ac0af220 msgr2=0x7f62ac0c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:47.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.763+0000 7f62aad9d700 1 --2- 172.21.15.98:0/1545659542 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ac0af220 0x7f62ac0c6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.763+0000 7f62aad9d700 1 -- 172.21.15.98:0/1545659542 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f62ac06a870 msgr2=0x7f62ac151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:47.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.763+0000 7f62aad9d700 1 --2- 172.21.15.98:0/1545659542 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f62ac06a870 0x7f62ac151cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.763+0000 7f62aad9d700 1 -- 172.21.15.98:0/1545659542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a4009b30 con 0x7f62ac0cf170 2023-03-03T08:02:47.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.763+0000 7f62a9d9b700 1 --2- 172.21.15.98:0/1545659542 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ac0af220 0x7f62ac0c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:47.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.763+0000 7f62aad9d700 1 --2- 172.21.15.98:0/1545659542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62ac0cf170 0x7f62ac1f20b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f62ac066f00 tx=0x7f62a4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:47.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.764+0000 7f629affd700 1 -- 172.21.15.98:0/1545659542 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a40099b0 con 0x7f62ac0cf170 2023-03-03T08:02:47.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.764+0000 7f629affd700 1 -- 172.21.15.98:0/1545659542 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a400ce80 con 0x7f62ac0cf170 2023-03-03T08:02:47.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.764+0000 7f62b0fff700 1 -- 172.21.15.98:0/1545659542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ac0ca940 con 0x7f62ac0cf170 2023-03-03T08:02:47.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.764+0000 7f629affd700 1 -- 172.21.15.98:0/1545659542 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a4004490 con 0x7f62ac0cf170 2023-03-03T08:02:47.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.764+0000 7f62b0fff700 1 -- 172.21.15.98:0/1545659542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ac0ca660 con 0x7f62ac0cf170 2023-03-03T08:02:47.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.765+0000 7f62b0fff700 1 -- 172.21.15.98:0/1545659542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ac0af5f0 con 0x7f62ac0cf170 2023-03-03T08:02:47.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.766+0000 7f629affd700 1 -- 172.21.15.98:0/1545659542 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f62a402b070 con 0x7f62ac0cf170 2023-03-03T08:02:47.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.767+0000 7f629affd700 1 --2- 172.21.15.98:0/1545659542 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f628c068380 0x7f628c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:47.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.767+0000 7f629affd700 1 -- 172.21.15.98:0/1545659542 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f62a4086fe0 con 0x7f62ac0cf170 2023-03-03T08:02:47.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.768+0000 7f62aa59c700 1 --2- 172.21.15.98:0/1545659542 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f628c068380 0x7f628c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:47.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.769+0000 7f62aa59c700 1 --2- 172.21.15.98:0/1545659542 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f628c068380 0x7f628c06a830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0b2940 tx=0x7f629c009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:47.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.772+0000 7f629affd700 1 -- 172.21.15.98:0/1545659542 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f62a4053700 con 0x7f62ac0cf170 2023-03-03T08:02:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.962+0000 7f62b0fff700 1 -- 172.21.15.98:0/1545659542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f62ac04e6e0 con 0x7f62ac0cf170 2023-03-03T08:02:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.963+0000 7f629affd700 1 -- 172.21.15.98:0/1545659542 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7f62a401f020 con 0x7f62ac0cf170 2023-03-03T08:02:47.964 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T08:02:47.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.965+0000 7f62b0fff700 1 -- 172.21.15.98:0/1545659542 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f628c068380 msgr2=0x7f628c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:47.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.965+0000 7f62b0fff700 1 --2- 172.21.15.98:0/1545659542 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f628c068380 0x7f628c06a830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0b2940 tx=0x7f629c009210 comp rx=0 tx=0).stop 2023-03-03T08:02:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.965+0000 7f62b0fff700 1 -- 172.21.15.98:0/1545659542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62ac0cf170 msgr2=0x7f62ac1f20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.965+0000 7f62b0fff700 1 --2- 172.21.15.98:0/1545659542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62ac0cf170 0x7f62ac1f20b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f62ac066f00 tx=0x7f62a4004ad0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.965+0000 7f62b0fff700 1 -- 172.21.15.98:0/1545659542 shutdown_connections 2023-03-03T08:02:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.965+0000 7f62b0fff700 1 --2- 172.21.15.98:0/1545659542 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f628c068380 0x7f628c06a830 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.965+0000 7f62b0fff700 1 --2- 172.21.15.98:0/1545659542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62ac0cf170 0x7f62ac1f20b0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.965+0000 7f62b0fff700 1 --2- 172.21.15.98:0/1545659542 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f62ac06a870 0x7f62ac151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.965+0000 7f62b0fff700 1 --2- 172.21.15.98:0/1545659542 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ac0af220 0x7f62ac0c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.965+0000 7f62b0fff700 1 -- 172.21.15.98:0/1545659542 >> 172.21.15.98:0/1545659542 conn(0x7f62ac0cc8f0 msgr2=0x7f62ac06dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.965+0000 7f62b0fff700 1 -- 172.21.15.98:0/1545659542 shutdown_connections 2023-03-03T08:02:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:47.965+0000 7f62b0fff700 1 -- 172.21.15.98:0/1545659542 wait complete. 2023-03-03T08:02:47.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T08:02:48.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.241+0000 7f0d779ff700 1 Processor -- start 2023-03-03T08:02:48.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.241+0000 7f0d779ff700 1 -- start start 2023-03-03T08:02:48.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.242+0000 7f0d779ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0d7006b4b0 0x7f0d700cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.242+0000 7f0d779ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d700cf980 con 0x7f0d7015ca70 2023-03-03T08:02:48.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.242+0000 7f0d779ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d700d1800 con 0x7f0d700d0270 2023-03-03T08:02:48.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.242+0000 7f0d779ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d700cd5b0 con 0x7f0d7006b4b0 2023-03-03T08:02:48.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.243+0000 7f0d7579b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0d7015ca70 0x7f0d700cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37240/0 (socket says 172.21.15.98:37240) 2023-03-03T08:02:48.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.243+0000 7f0d7579b700 1 -- 172.21.15.98:0/210360665 learned_addr learned my addr 172.21.15.98:0/210360665 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:48.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.243+0000 7f0d67fff700 1 -- 172.21.15.98:0/210360665 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123810352 0 0) 0x7f0d700cf980 con 0x7f0d7015ca70 2023-03-03T08:02:48.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.243+0000 7f0d75f9c700 1 --2- 172.21.15.98:0/210360665 >> v2:172.21.15.100:3300/0 conn(0x7f0d7006b4b0 0x7f0d700cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.243+0000 7f0d67fff700 1 -- 172.21.15.98:0/210360665 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d5c0035b0 con 0x7f0d7015ca70 2023-03-03T08:02:48.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.243+0000 7f0d67fff700 1 -- 172.21.15.98:0/210360665 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2307680326 0 0) 0x7f0d5c0035b0 con 0x7f0d7015ca70 2023-03-03T08:02:48.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.243+0000 7f0d67fff700 1 -- 172.21.15.98:0/210360665 >> v2:172.21.15.100:3300/0 conn(0x7f0d7006b4b0 msgr2=0x7f0d700cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:48.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.243+0000 7f0d67fff700 1 --2- 172.21.15.98:0/210360665 >> v2:172.21.15.100:3300/0 conn(0x7f0d7006b4b0 0x7f0d700cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.243+0000 7f0d67fff700 1 -- 172.21.15.98:0/210360665 >> v1:172.21.15.100:6789/0 conn(0x7f0d700d0270 legacy=0x7f0d700d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:48.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.244+0000 7f0d67fff700 1 -- 172.21.15.98:0/210360665 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d700cd900 con 0x7f0d7015ca70 2023-03-03T08:02:48.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.244+0000 7f0d67fff700 1 -- 172.21.15.98:0/210360665 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f0d6c001670 con 0x7f0d7015ca70 2023-03-03T08:02:48.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.244+0000 7f0d67fff700 1 -- 172.21.15.98:0/210360665 >> v1:172.21.15.98:6789/0 conn(0x7f0d7015ca70 legacy=0x7f0d700cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:48.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.244+0000 7f0d67fff700 1 --2- 172.21.15.98:0/210360665 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d5c003a50 0x7f0d5c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.244+0000 7f0d67fff700 1 --2- 172.21.15.98:0/210360665 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d5c004850 0x7f0d5c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.244+0000 7f0d67fff700 1 --2- 172.21.15.98:0/210360665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d5c005250 0x7f0d5c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.244+0000 7f0d67fff700 1 -- 172.21.15.98:0/210360665 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d700cd5b0 con 0x7f0d5c005250 2023-03-03T08:02:48.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.244+0000 7f0d67fff700 1 -- 172.21.15.98:0/210360665 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c00c160 con 0x7f0d5c004850 2023-03-03T08:02:48.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.244+0000 7f0d67fff700 1 -- 172.21.15.98:0/210360665 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c00c320 con 0x7f0d5c003a50 2023-03-03T08:02:48.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.244+0000 7f0d75f9c700 1 --2- 172.21.15.98:0/210360665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d5c005250 0x7f0d5c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.244+0000 7f0d74f9a700 1 --2- 172.21.15.98:0/210360665 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d5c004850 0x7f0d5c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.244+0000 7f0d7579b700 1 --2- 172.21.15.98:0/210360665 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d5c003a50 0x7f0d5c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.245+0000 7f0d75f9c700 1 -- 172.21.15.98:0/210360665 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d5c003a50 msgr2=0x7f0d5c003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:48.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.245+0000 7f0d75f9c700 1 --2- 172.21.15.98:0/210360665 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d5c003a50 0x7f0d5c003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.245+0000 7f0d75f9c700 1 -- 172.21.15.98:0/210360665 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d5c004850 msgr2=0x7f0d5c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:48.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.245+0000 7f0d75f9c700 1 --2- 172.21.15.98:0/210360665 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d5c004850 0x7f0d5c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.245+0000 7f0d75f9c700 1 -- 172.21.15.98:0/210360665 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c0035b0 con 0x7f0d5c005250 2023-03-03T08:02:48.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.245+0000 7f0d75f9c700 1 -- 172.21.15.98:0/210360665 reap_dead start 2023-03-03T08:02:48.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.245+0000 7f0d75f9c700 1 --2- 172.21.15.98:0/210360665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d5c005250 0x7f0d5c00ba10 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f0d68009fc0 tx=0x7f0d5c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b4ee83305d0624 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:48.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.245+0000 7f0d67fff700 1 -- 172.21.15.98:0/210360665 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68003a20 con 0x7f0d5c005250 2023-03-03T08:02:48.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.245+0000 7f0d67fff700 1 -- 172.21.15.98:0/210360665 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6800f040 con 0x7f0d5c005250 2023-03-03T08:02:48.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.245+0000 7f0d67fff700 1 -- 172.21.15.98:0/210360665 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68003de0 con 0x7f0d5c005250 2023-03-03T08:02:48.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.246+0000 7f0d779ff700 1 -- 172.21.15.98:0/210360665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d5c005250 msgr2=0x7f0d5c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:48.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.246+0000 7f0d779ff700 1 --2- 172.21.15.98:0/210360665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d5c005250 0x7f0d5c00ba10 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f0d68009fc0 tx=0x7f0d5c005880 comp rx=0 tx=0).stop 2023-03-03T08:02:48.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.246+0000 7f0d779ff700 1 -- 172.21.15.98:0/210360665 shutdown_connections 2023-03-03T08:02:48.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.246+0000 7f0d779ff700 1 --2- 172.21.15.98:0/210360665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d5c005250 0x7f0d5c00ba10 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.246+0000 7f0d779ff700 1 -- 172.21.15.98:0/210360665 >> 172.21.15.98:0/210360665 conn(0x7f0d70073e70 msgr2=0x7f0d7006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:48.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.246+0000 7f0d779ff700 1 -- 172.21.15.98:0/210360665 shutdown_connections 2023-03-03T08:02:48.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.246+0000 7f0d779ff700 1 -- 172.21.15.98:0/210360665 wait complete. 2023-03-03T08:02:48.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.248+0000 7f0d779ff700 1 Processor -- start 2023-03-03T08:02:48.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.248+0000 7f0d779ff700 1 -- start start 2023-03-03T08:02:48.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.249+0000 7f0d779ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d5c005250 0x7f0d700b5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.249+0000 7f0d779ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d70154a50 0x7f0d701f5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.249+0000 7f0d779ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d701f5ca0 0x7f0d700d3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.249+0000 7f0d779ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d700b4980 con 0x7f0d70154a50 2023-03-03T08:02:48.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.249+0000 7f0d779ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d701f61b0 con 0x7f0d5c005250 2023-03-03T08:02:48.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.249+0000 7f0d779ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d701f6490 con 0x7f0d701f5ca0 2023-03-03T08:02:48.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.250+0000 7f0d74f9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d70154a50 0x7f0d701f5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.250+0000 7f0d74f9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d70154a50 0x7f0d701f5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52872/0 (socket says 172.21.15.98:52872) 2023-03-03T08:02:48.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.250+0000 7f0d74f9a700 1 -- 172.21.15.98:0/3266010214 learned_addr learned my addr 172.21.15.98:0/3266010214 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:48.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.250+0000 7f0d7579b700 1 --2- 172.21.15.98:0/3266010214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d5c005250 0x7f0d700b5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.250+0000 7f0d75f9c700 1 --2- 172.21.15.98:0/3266010214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d701f5ca0 0x7f0d700d3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.250+0000 7f0d74f9a700 1 -- 172.21.15.98:0/3266010214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d701f5ca0 msgr2=0x7f0d700d3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:48.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.250+0000 7f0d74f9a700 1 --2- 172.21.15.98:0/3266010214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d701f5ca0 0x7f0d700d3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.250+0000 7f0d74f9a700 1 -- 172.21.15.98:0/3266010214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d5c005250 msgr2=0x7f0d700b5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:48.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.250+0000 7f0d74f9a700 1 --2- 172.21.15.98:0/3266010214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d5c005250 0x7f0d700b5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.250+0000 7f0d74f9a700 1 -- 172.21.15.98:0/3266010214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d700d1aa0 con 0x7f0d70154a50 2023-03-03T08:02:48.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.250+0000 7f0d75f9c700 1 --2- 172.21.15.98:0/3266010214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d701f5ca0 0x7f0d700d3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:48.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.250+0000 7f0d74f9a700 1 --2- 172.21.15.98:0/3266010214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d70154a50 0x7f0d701f5760 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f0d6000f5a0 tx=0x7f0d6000f960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:48.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.251+0000 7f0d65ffb700 1 -- 172.21.15.98:0/3266010214 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d60005320 con 0x7f0d70154a50 2023-03-03T08:02:48.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.251+0000 7f0d65ffb700 1 -- 172.21.15.98:0/3266010214 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d60005480 con 0x7f0d70154a50 2023-03-03T08:02:48.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.251+0000 7f0d65ffb700 1 -- 172.21.15.98:0/3266010214 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d60006a80 con 0x7f0d70154a50 2023-03-03T08:02:48.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.251+0000 7f0d779ff700 1 -- 172.21.15.98:0/3266010214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d700cdbe0 con 0x7f0d70154a50 2023-03-03T08:02:48.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.251+0000 7f0d779ff700 1 -- 172.21.15.98:0/3266010214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7015ce40 con 0x7f0d70154a50 2023-03-03T08:02:48.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.252+0000 7f0d779ff700 1 -- 172.21.15.98:0/3266010214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d700b58d0 con 0x7f0d70154a50 2023-03-03T08:02:48.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.253+0000 7f0d65ffb700 1 -- 172.21.15.98:0/3266010214 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f0d60006be0 con 0x7f0d70154a50 2023-03-03T08:02:48.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.254+0000 7f0d65ffb700 1 --2- 172.21.15.98:0/3266010214 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0d54068380 0x7f0d5406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.255+0000 7f0d65ffb700 1 -- 172.21.15.98:0/3266010214 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0d60085170 con 0x7f0d70154a50 2023-03-03T08:02:48.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.255+0000 7f0d7579b700 1 --2- 172.21.15.98:0/3266010214 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0d54068380 0x7f0d5406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.256+0000 7f0d7579b700 1 --2- 172.21.15.98:0/3266010214 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0d54068380 0x7f0d5406a830 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c001050 tx=0x7f0d6c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:48.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.258+0000 7f0d65ffb700 1 -- 172.21.15.98:0/3266010214 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0d6004d5e0 con 0x7f0d70154a50 2023-03-03T08:02:48.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.449+0000 7f0d779ff700 1 -- 172.21.15.98:0/3266010214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0d7004e6e0 con 0x7f0d70154a50 2023-03-03T08:02:48.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.449+0000 7f0d65ffb700 1 -- 172.21.15.98:0/3266010214 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0d60051a30 con 0x7f0d70154a50 2023-03-03T08:02:48.451 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T08:02:48.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.452+0000 7f0d779ff700 1 -- 172.21.15.98:0/3266010214 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0d54068380 msgr2=0x7f0d5406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:48.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.452+0000 7f0d779ff700 1 --2- 172.21.15.98:0/3266010214 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0d54068380 0x7f0d5406a830 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c001050 tx=0x7f0d6c009040 comp rx=0 tx=0).stop 2023-03-03T08:02:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.452+0000 7f0d779ff700 1 -- 172.21.15.98:0/3266010214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d70154a50 msgr2=0x7f0d701f5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.452+0000 7f0d779ff700 1 --2- 172.21.15.98:0/3266010214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d70154a50 0x7f0d701f5760 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f0d6000f5a0 tx=0x7f0d6000f960 comp rx=0 tx=0).stop 2023-03-03T08:02:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.452+0000 7f0d779ff700 1 -- 172.21.15.98:0/3266010214 shutdown_connections 2023-03-03T08:02:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.452+0000 7f0d779ff700 1 --2- 172.21.15.98:0/3266010214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d701f5ca0 0x7f0d700d3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.452+0000 7f0d779ff700 1 --2- 172.21.15.98:0/3266010214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d5c005250 0x7f0d700b5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.452+0000 7f0d779ff700 1 --2- 172.21.15.98:0/3266010214 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0d54068380 0x7f0d5406a830 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.452+0000 7f0d779ff700 1 --2- 172.21.15.98:0/3266010214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d70154a50 0x7f0d701f5760 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.452+0000 7f0d779ff700 1 -- 172.21.15.98:0/3266010214 >> 172.21.15.98:0/3266010214 conn(0x7f0d70073e70 msgr2=0x7f0d700b6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:48.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.452+0000 7f0d779ff700 1 -- 172.21.15.98:0/3266010214 shutdown_connections 2023-03-03T08:02:48.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.452+0000 7f0d779ff700 1 -- 172.21.15.98:0/3266010214 wait complete. 2023-03-03T08:02:48.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T08:02:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.728+0000 7ff932dff700 1 Processor -- start 2023-03-03T08:02:48.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.728+0000 7ff932dff700 1 -- start start 2023-03-03T08:02:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.730+0000 7ff932dff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff92c15ca70 0x7ff92c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.730+0000 7ff932dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff92c0d0270 0x7ff92c0d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.730+0000 7ff932dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92c0cf980 con 0x7ff92c06b4b0 2023-03-03T08:02:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.730+0000 7ff932dff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff92c0cd5b0 con 0x7ff92c15ca70 2023-03-03T08:02:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.730+0000 7ff932dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0d1800 con 0x7ff92c0d0270 2023-03-03T08:02:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.730+0000 7ff92bfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff92c06b4b0 0x7ff92c0cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37256/0 (socket says 172.21.15.98:37256) 2023-03-03T08:02:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.730+0000 7ff92bfff700 1 -- 172.21.15.98:0/4206561676 learned_addr learned my addr 172.21.15.98:0/4206561676 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.730+0000 7ff93139c700 1 --2- 172.21.15.98:0/4206561676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff92c0d0270 0x7ff92c0d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.730+0000 7ff930b9b700 1 --2- 172.21.15.98:0/4206561676 >> v2:172.21.15.100:3300/0 conn(0x7ff92c15ca70 0x7ff92c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 -- 172.21.15.98:0/4206561676 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722570765 0 0) 0x7ff92c0cf980 con 0x7ff92c06b4b0 2023-03-03T08:02:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 -- 172.21.15.98:0/4206561676 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff91c0035b0 con 0x7ff92c06b4b0 2023-03-03T08:02:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 -- 172.21.15.98:0/4206561676 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3937355722 0 0) 0x7ff91c0035b0 con 0x7ff92c06b4b0 2023-03-03T08:02:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 -- 172.21.15.98:0/4206561676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff92c0d0270 msgr2=0x7ff92c0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 --2- 172.21.15.98:0/4206561676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff92c0d0270 0x7ff92c0d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 -- 172.21.15.98:0/4206561676 >> v2:172.21.15.100:3300/0 conn(0x7ff92c15ca70 msgr2=0x7ff92c0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 --2- 172.21.15.98:0/4206561676 >> v2:172.21.15.100:3300/0 conn(0x7ff92c15ca70 0x7ff92c0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 -- 172.21.15.98:0/4206561676 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c0cd900 con 0x7ff92c06b4b0 2023-03-03T08:02:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 -- 172.21.15.98:0/4206561676 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7ff918001670 con 0x7ff92c06b4b0 2023-03-03T08:02:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 -- 172.21.15.98:0/4206561676 >> v1:172.21.15.98:6789/0 conn(0x7ff92c06b4b0 legacy=0x7ff92c0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 --2- 172.21.15.98:0/4206561676 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91c003a00 0x7ff91c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 --2- 172.21.15.98:0/4206561676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff91c004430 0x7ff91c0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 --2- 172.21.15.98:0/4206561676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff91c005000 0x7ff91c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 -- 172.21.15.98:0/4206561676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0cd5b0 con 0x7ff91c004430 2023-03-03T08:02:48.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 -- 172.21.15.98:0/4206561676 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0d1800 con 0x7ff91c003a00 2023-03-03T08:02:48.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92b7fe700 1 -- 172.21.15.98:0/4206561676 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c00bf50 con 0x7ff91c005000 2023-03-03T08:02:48.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff930b9b700 1 --2- 172.21.15.98:0/4206561676 >> v2:172.21.15.100:3300/0 conn(0x7ff92c15ca70 0x7ff92c0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:48.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff93139c700 1 --2- 172.21.15.98:0/4206561676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff92c0d0270 0x7ff92c0d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:48.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.731+0000 7ff92bfff700 1 --2- 172.21.15.98:0/4206561676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff91c004430 0x7ff91c0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.732+0000 7ff93139c700 1 --2- 172.21.15.98:0/4206561676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff91c005000 0x7ff91c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.732+0000 7ff930b9b700 1 --2- 172.21.15.98:0/4206561676 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91c003a00 0x7ff91c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.732+0000 7ff92bfff700 1 -- 172.21.15.98:0/4206561676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff91c005000 msgr2=0x7ff91c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:48.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.732+0000 7ff92bfff700 1 --2- 172.21.15.98:0/4206561676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff91c005000 0x7ff91c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.732+0000 7ff92bfff700 1 -- 172.21.15.98:0/4206561676 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91c003a00 msgr2=0x7ff91c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:48.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.732+0000 7ff92bfff700 1 --2- 172.21.15.98:0/4206561676 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91c003a00 0x7ff91c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.732+0000 7ff92bfff700 1 -- 172.21.15.98:0/4206561676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c0035b0 con 0x7ff91c004430 2023-03-03T08:02:48.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.732+0000 7ff92bfff700 1 --2- 172.21.15.98:0/4206561676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff91c004430 0x7ff91c0048f0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7ff91800c2f0 tx=0x7ff91c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8efb58d7f38f9f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.732+0000 7ff92b7fe700 1 -- 172.21.15.98:0/4206561676 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91800ce30 con 0x7ff91c004430 2023-03-03T08:02:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.733+0000 7ff92b7fe700 1 -- 172.21.15.98:0/4206561676 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff918005960 con 0x7ff91c004430 2023-03-03T08:02:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.733+0000 7ff92b7fe700 1 -- 172.21.15.98:0/4206561676 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff918005d10 con 0x7ff91c004430 2023-03-03T08:02:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.733+0000 7ff932dff700 1 -- 172.21.15.98:0/4206561676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff91c004430 msgr2=0x7ff91c0048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.733+0000 7ff932dff700 1 --2- 172.21.15.98:0/4206561676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff91c004430 0x7ff91c0048f0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7ff91800c2f0 tx=0x7ff91c005880 comp rx=0 tx=0).stop 2023-03-03T08:02:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.733+0000 7ff93139c700 1 -- 172.21.15.98:0/4206561676 reap_dead start 2023-03-03T08:02:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.733+0000 7ff932dff700 1 -- 172.21.15.98:0/4206561676 shutdown_connections 2023-03-03T08:02:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.733+0000 7ff932dff700 1 -- 172.21.15.98:0/4206561676 >> 172.21.15.98:0/4206561676 conn(0x7ff92c073e70 msgr2=0x7ff92c06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.733+0000 7ff932dff700 1 -- 172.21.15.98:0/4206561676 shutdown_connections 2023-03-03T08:02:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.733+0000 7ff932dff700 1 -- 172.21.15.98:0/4206561676 wait complete. 2023-03-03T08:02:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.735+0000 7ff932dff700 1 Processor -- start 2023-03-03T08:02:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.736+0000 7ff932dff700 1 -- start start 2023-03-03T08:02:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff932dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91c004430 0x7ff92c0cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff932dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff92c0b3ad0 0x7ff92c151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff932dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff92c0b2be0 0x7ff92c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff932dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c152d40 con 0x7ff92c0b3ad0 2023-03-03T08:02:48.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff932dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0b4050 con 0x7ff91c004430 2023-03-03T08:02:48.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff932dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0b4330 con 0x7ff92c0b2be0 2023-03-03T08:02:48.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff92bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff92c0b3ad0 0x7ff92c151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff92bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff92c0b3ad0 0x7ff92c151600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52880/0 (socket says 172.21.15.98:52880) 2023-03-03T08:02:48.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff92bfff700 1 -- 172.21.15.98:0/511147747 learned_addr learned my addr 172.21.15.98:0/511147747 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:48.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff930b9b700 1 --2- 172.21.15.98:0/511147747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91c004430 0x7ff92c0cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff93139c700 1 --2- 172.21.15.98:0/511147747 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff92c0b2be0 0x7ff92c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff92bfff700 1 -- 172.21.15.98:0/511147747 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff92c0b2be0 msgr2=0x7ff92c0b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:48.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff92bfff700 1 --2- 172.21.15.98:0/511147747 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff92c0b2be0 0x7ff92c0b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff92bfff700 1 -- 172.21.15.98:0/511147747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91c004430 msgr2=0x7ff92c0cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:48.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff92bfff700 1 --2- 172.21.15.98:0/511147747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91c004430 0x7ff92c0cb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.737+0000 7ff92bfff700 1 -- 172.21.15.98:0/511147747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c0d1aa0 con 0x7ff92c0b3ad0 2023-03-03T08:02:48.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.738+0000 7ff92bfff700 1 --2- 172.21.15.98:0/511147747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff92c0b3ad0 0x7ff92c151600 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7ff91c005790 tx=0x7ff918001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:48.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.738+0000 7ff9297fa700 1 -- 172.21.15.98:0/511147747 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff918007c20 con 0x7ff92c0b3ad0 2023-03-03T08:02:48.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.738+0000 7ff9297fa700 1 -- 172.21.15.98:0/511147747 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff918004870 con 0x7ff92c0b3ad0 2023-03-03T08:02:48.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.738+0000 7ff9297fa700 1 -- 172.21.15.98:0/511147747 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff918004c00 con 0x7ff92c0b3ad0 2023-03-03T08:02:48.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.738+0000 7ff932dff700 1 -- 172.21.15.98:0/511147747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92c0cdbe0 con 0x7ff92c0b3ad0 2023-03-03T08:02:48.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.738+0000 7ff932dff700 1 -- 172.21.15.98:0/511147747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92c15ce40 con 0x7ff92c0b3ad0 2023-03-03T08:02:48.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.739+0000 7ff932dff700 1 -- 172.21.15.98:0/511147747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92c04e6e0 con 0x7ff92c0b3ad0 2023-03-03T08:02:48.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.741+0000 7ff9297fa700 1 -- 172.21.15.98:0/511147747 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7ff91802e070 con 0x7ff92c0b3ad0 2023-03-03T08:02:48.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.742+0000 7ff9297fa700 1 --2- 172.21.15.98:0/511147747 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff910068380 0x7ff91006a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:48.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.742+0000 7ff9297fa700 1 -- 172.21.15.98:0/511147747 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff9180875a0 con 0x7ff92c0b3ad0 2023-03-03T08:02:48.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.742+0000 7ff930b9b700 1 --2- 172.21.15.98:0/511147747 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff910068380 0x7ff91006a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:48.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.743+0000 7ff930b9b700 1 --2- 172.21.15.98:0/511147747 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff910068380 0x7ff91006a830 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff9200045f0 tx=0x7ff920008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:48.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.746+0000 7ff9297fa700 1 -- 172.21.15.98:0/511147747 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff918053ca0 con 0x7ff92c0b3ad0 2023-03-03T08:02:48.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.954+0000 7ff932dff700 1 -- 172.21.15.98:0/511147747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff92c0d1540 con 0x7ff92c0b3ad0 2023-03-03T08:02:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.954+0000 7ff9297fa700 1 -- 172.21.15.98:0/511147747 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v98) v1 ==== 80+0+124 (secure 0 0 0) 0x7ff918085a20 con 0x7ff92c0b3ad0 2023-03-03T08:02:48.956 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.100:6835/1843752855 2023-03-04T08:02:42.175049+0000 2023-03-03T08:02:48.957 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.100:6833/1843752855 2023-03-04T08:02:42.175049+0000 2023-03-03T08:02:48.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.957+0000 7ff932dff700 1 -- 172.21.15.98:0/511147747 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff910068380 msgr2=0x7ff91006a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:48.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.957+0000 7ff932dff700 1 --2- 172.21.15.98:0/511147747 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff910068380 0x7ff91006a830 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff9200045f0 tx=0x7ff920008040 comp rx=0 tx=0).stop 2023-03-03T08:02:48.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.957+0000 7ff932dff700 1 -- 172.21.15.98:0/511147747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff92c0b3ad0 msgr2=0x7ff92c151600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:48.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.957+0000 7ff932dff700 1 --2- 172.21.15.98:0/511147747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff92c0b3ad0 0x7ff92c151600 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7ff91c005790 tx=0x7ff918001a00 comp rx=0 tx=0).stop 2023-03-03T08:02:48.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.957+0000 7ff932dff700 1 -- 172.21.15.98:0/511147747 shutdown_connections 2023-03-03T08:02:48.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.957+0000 7ff932dff700 1 --2- 172.21.15.98:0/511147747 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff92c0b2be0 0x7ff92c0b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.957+0000 7ff932dff700 1 --2- 172.21.15.98:0/511147747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff91c004430 0x7ff92c0cb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.957+0000 7ff932dff700 1 --2- 172.21.15.98:0/511147747 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff910068380 0x7ff91006a830 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.957+0000 7ff932dff700 1 --2- 172.21.15.98:0/511147747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff92c0b3ad0 0x7ff92c151600 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.957+0000 7ff932dff700 1 -- 172.21.15.98:0/511147747 >> 172.21.15.98:0/511147747 conn(0x7ff92c073e70 msgr2=0x7ff92c0b6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.957+0000 7ff932dff700 1 -- 172.21.15.98:0/511147747 shutdown_connections 2023-03-03T08:02:48.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:48.957+0000 7ff932dff700 1 -- 172.21.15.98:0/511147747 wait complete. 2023-03-03T08:02:48.961 INFO:teuthology.orchestra.run.smithi098.stderr:listed 2 entries 2023-03-03T08:02:48.987 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2023-03-03T08:02:48.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:48.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-03-03T08:02:49.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[144617]: fuse finished with error 0 and tester_r 0 2023-03-03T08:02:49.090 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T08:02:49.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:49.091 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-03-03T08:02:49.264 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:02:49.265 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:02:49.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:02:49.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.582+0000 7f683ed9d700 1 Processor -- start 2023-03-03T08:02:49.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.582+0000 7f683ed9d700 1 -- start start 2023-03-03T08:02:49.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.583+0000 7f683ed9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f68380b51c0 0x7f6838160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:49.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.584+0000 7f683ed9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f68380d1580 0x7f68380d1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:49.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.584+0000 7f683ed9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68381614c0 con 0x7f6838146360 2023-03-03T08:02:49.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.584+0000 7f683ed9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68380d10f0 con 0x7f68380b51c0 2023-03-03T08:02:49.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.584+0000 7f683ed9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68380cdf20 con 0x7f68380d1580 2023-03-03T08:02:49.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.584+0000 7f683d59a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f68380b51c0 0x7f6838160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:49.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.584+0000 7f683d59a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f68380b51c0 0x7f6838160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52888/0 (socket says 172.21.15.98:52888) 2023-03-03T08:02:49.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.584+0000 7f683d59a700 1 -- 172.21.15.98:0/1373502145 learned_addr learned my addr 172.21.15.98:0/1373502145 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.584+0000 7f683e59c700 1 --2- 172.21.15.98:0/1373502145 >> v2:172.21.15.100:3300/0 conn(0x7f68380d1580 0x7f68380d1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.584+0000 7f683d59a700 1 -- 172.21.15.98:0/1373502145 >> v2:172.21.15.100:3300/0 conn(0x7f68380d1580 msgr2=0x7f68380d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.584+0000 7f683d59a700 1 --2- 172.21.15.98:0/1373502145 >> v2:172.21.15.100:3300/0 conn(0x7f68380d1580 0x7f68380d1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.584+0000 7f683d59a700 1 -- 172.21.15.98:0/1373502145 >> v1:172.21.15.100:6789/0 conn(0x7f6838146360 legacy=0x7f68380b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.584+0000 7f683d59a700 1 -- 172.21.15.98:0/1373502145 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68380cd680 con 0x7f68380b51c0 2023-03-03T08:02:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.584+0000 7f683e59c700 1 --2- 172.21.15.98:0/1373502145 >> v2:172.21.15.100:3300/0 conn(0x7f68380d1580 0x7f68380d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.585+0000 7f683d59a700 1 --2- 172.21.15.98:0/1373502145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68380b51c0 0x7f6838160f80 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f68380d2fe0 tx=0x7f6828009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf6ea606551ee5d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.585+0000 7f683cd99700 1 -- 172.21.15.98:0/1373502145 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6828004230 con 0x7f68380b51c0 2023-03-03T08:02:49.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.585+0000 7f683cd99700 1 -- 172.21.15.98:0/1373502145 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6828004390 con 0x7f68380b51c0 2023-03-03T08:02:49.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.585+0000 7f683cd99700 1 -- 172.21.15.98:0/1373502145 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68280036a0 con 0x7f68380b51c0 2023-03-03T08:02:49.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.586+0000 7f683ed9d700 1 -- 172.21.15.98:0/1373502145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68380b51c0 msgr2=0x7f6838160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:49.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.586+0000 7f683ed9d700 1 --2- 172.21.15.98:0/1373502145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68380b51c0 0x7f6838160f80 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f68380d2fe0 tx=0x7f6828009bd0 comp rx=0 tx=0).stop 2023-03-03T08:02:49.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.586+0000 7f683ed9d700 1 -- 172.21.15.98:0/1373502145 shutdown_connections 2023-03-03T08:02:49.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.586+0000 7f683ed9d700 1 --2- 172.21.15.98:0/1373502145 >> v2:172.21.15.100:3300/0 conn(0x7f68380d1580 0x7f68380d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:49.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.586+0000 7f683ed9d700 1 --2- 172.21.15.98:0/1373502145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68380b51c0 0x7f6838160f80 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:49.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.586+0000 7f683ed9d700 1 -- 172.21.15.98:0/1373502145 >> 172.21.15.98:0/1373502145 conn(0x7f683806b8e0 msgr2=0x7f68380cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:49.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.586+0000 7f683ed9d700 1 -- 172.21.15.98:0/1373502145 shutdown_connections 2023-03-03T08:02:49.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.586+0000 7f683ed9d700 1 -- 172.21.15.98:0/1373502145 wait complete. 2023-03-03T08:02:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.588+0000 7f683ed9d700 1 Processor -- start 2023-03-03T08:02:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.588+0000 7f683ed9d700 1 -- start start 2023-03-03T08:02:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.589+0000 7f683ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68380b51c0 0x7f68380c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.589+0000 7f683ed9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68380d1580 0x7f68380c7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.589+0000 7f683ed9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6838146360 0x7f68380c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.589+0000 7f683ed9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68380c7080 con 0x7f68380b51c0 2023-03-03T08:02:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.589+0000 7f683ed9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68380cad10 con 0x7f68380d1580 2023-03-03T08:02:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.589+0000 7f683ed9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68380ca100 con 0x7f6838146360 2023-03-03T08:02:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.590+0000 7f683dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68380b51c0 0x7f68380c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.590+0000 7f683dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68380b51c0 0x7f68380c5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52900/0 (socket says 172.21.15.98:52900) 2023-03-03T08:02:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.590+0000 7f683dd9b700 1 -- 172.21.15.98:0/3310299413 learned_addr learned my addr 172.21.15.98:0/3310299413 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.590+0000 7f683d59a700 1 --2- 172.21.15.98:0/3310299413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68380d1580 0x7f68380c7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.590+0000 7f683e59c700 1 --2- 172.21.15.98:0/3310299413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6838146360 0x7f68380c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.590+0000 7f683dd9b700 1 -- 172.21.15.98:0/3310299413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6838146360 msgr2=0x7f68380c7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.590+0000 7f683dd9b700 1 --2- 172.21.15.98:0/3310299413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6838146360 0x7f68380c7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.590+0000 7f683dd9b700 1 -- 172.21.15.98:0/3310299413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68380d1580 msgr2=0x7f68380c7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.590+0000 7f683dd9b700 1 --2- 172.21.15.98:0/3310299413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68380d1580 0x7f68380c7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.590+0000 7f683dd9b700 1 -- 172.21.15.98:0/3310299413 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6828009550 con 0x7f68380b51c0 2023-03-03T08:02:49.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.590+0000 7f683dd9b700 1 --2- 172.21.15.98:0/3310299413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68380b51c0 0x7f68380c5d00 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f683400c4d0 tx=0x7f683400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:49.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.591+0000 7f682e7fc700 1 -- 172.21.15.98:0/3310299413 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6834011790 con 0x7f68380b51c0 2023-03-03T08:02:49.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.591+0000 7f682e7fc700 1 -- 172.21.15.98:0/3310299413 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68340118f0 con 0x7f68380b51c0 2023-03-03T08:02:49.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.591+0000 7f683ed9d700 1 -- 172.21.15.98:0/3310299413 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68380cdc40 con 0x7f68380b51c0 2023-03-03T08:02:49.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.591+0000 7f682e7fc700 1 -- 172.21.15.98:0/3310299413 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6834011ca0 con 0x7f68380b51c0 2023-03-03T08:02:49.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.591+0000 7f683ed9d700 1 -- 172.21.15.98:0/3310299413 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68380cd960 con 0x7f68380b51c0 2023-03-03T08:02:49.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.592+0000 7f683ed9d700 1 -- 172.21.15.98:0/3310299413 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68380d0d80 con 0x7f68380b51c0 2023-03-03T08:02:49.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.593+0000 7f682e7fc700 1 -- 172.21.15.98:0/3310299413 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f6834010050 con 0x7f68380b51c0 2023-03-03T08:02:49.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.594+0000 7f682e7fc700 1 --2- 172.21.15.98:0/3310299413 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f681c068380 0x7f681c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:49.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.595+0000 7f682e7fc700 1 -- 172.21.15.98:0/3310299413 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6834085ab0 con 0x7f68380b51c0 2023-03-03T08:02:49.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.595+0000 7f683d59a700 1 --2- 172.21.15.98:0/3310299413 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f681c068380 0x7f681c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:49.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.596+0000 7f683d59a700 1 --2- 172.21.15.98:0/3310299413 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f681c068380 0x7f681c06a830 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f68380b55b0 tx=0x7f6828008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:49.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.599+0000 7f682e7fc700 1 -- 172.21.15.98:0/3310299413 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f68340521d0 con 0x7f68380b51c0 2023-03-03T08:02:49.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.794+0000 7f683ed9d700 1 -- 172.21.15.98:0/3310299413 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6838068300 con 0x7f68380b51c0 2023-03-03T08:02:49.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.795+0000 7f682e7fc700 1 -- 172.21.15.98:0/3310299413 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+18053 (secure 0 0 0) 0x7f68340557f0 con 0x7f68380b51c0 2023-03-03T08:02:49.797 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:49.797 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":98,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:02:45.624487+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:02:20.797579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-03-03T08:02:21.411899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:6835/1843752855":"2023-03-04T08:02:42.175049+0000","172.21.15.100:6833/1843752855":"2023-03-04T08:02:42.175049+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:02:49.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.797+0000 7f683ed9d700 1 -- 172.21.15.98:0/3310299413 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f681c068380 msgr2=0x7f681c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:49.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.797+0000 7f683ed9d700 1 --2- 172.21.15.98:0/3310299413 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f681c068380 0x7f681c06a830 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f68380b55b0 tx=0x7f6828008030 comp rx=0 tx=0).stop 2023-03-03T08:02:49.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.797+0000 7f683ed9d700 1 -- 172.21.15.98:0/3310299413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68380b51c0 msgr2=0x7f68380c5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:49.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.797+0000 7f683ed9d700 1 --2- 172.21.15.98:0/3310299413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68380b51c0 0x7f68380c5d00 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f683400c4d0 tx=0x7f683400c7e0 comp rx=0 tx=0).stop 2023-03-03T08:02:49.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.798+0000 7f683ed9d700 1 -- 172.21.15.98:0/3310299413 shutdown_connections 2023-03-03T08:02:49.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.798+0000 7f683ed9d700 1 --2- 172.21.15.98:0/3310299413 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f681c068380 0x7f681c06a830 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:49.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.798+0000 7f683ed9d700 1 --2- 172.21.15.98:0/3310299413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68380b51c0 0x7f68380c5d00 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:49.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.798+0000 7f683ed9d700 1 --2- 172.21.15.98:0/3310299413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6838146360 0x7f68380c7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:49.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.798+0000 7f683ed9d700 1 --2- 172.21.15.98:0/3310299413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68380d1580 0x7f68380c7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:49.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.798+0000 7f683ed9d700 1 -- 172.21.15.98:0/3310299413 >> 172.21.15.98:0/3310299413 conn(0x7f683806b8e0 msgr2=0x7f68380b8160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:49.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.798+0000 7f683ed9d700 1 -- 172.21.15.98:0/3310299413 shutdown_connections 2023-03-03T08:02:49.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:49.798+0000 7f683ed9d700 1 -- 172.21.15.98:0/3310299413 wait complete. 2023-03-03T08:02:49.827 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T08:02:20.797579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T08:02:21.411899+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:02:49.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:02:50.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.074+0000 7fd43c3ff700 1 Processor -- start 2023-03-03T08:02:50.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.074+0000 7fd43c3ff700 1 -- start start 2023-03-03T08:02:50.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.075+0000 7fd43c3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd43415ca70 0x7fd4340cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:50.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.075+0000 7fd43c3ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd43406b4b0 0x7fd4340cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:50.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.075+0000 7fd43c3ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd4340d00a0 0x7fd4340b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:50.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.075+0000 7fd43c3ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4340cd5b0 con 0x7fd4340d00a0 2023-03-03T08:02:50.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.076+0000 7fd43c3ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4340cf980 con 0x7fd43406b4b0 2023-03-03T08:02:50.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.076+0000 7fd43c3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4340d0510 con 0x7fd43415ca70 2023-03-03T08:02:50.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.076+0000 7fd43a99c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd4340d00a0 0x7fd4340b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:50.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.076+0000 7fd43a99c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd4340d00a0 0x7fd4340b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52914/0 (socket says 172.21.15.98:52914) 2023-03-03T08:02:50.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.076+0000 7fd43a99c700 1 -- 172.21.15.98:0/3962533892 learned_addr learned my addr 172.21.15.98:0/3962533892 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:50.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.076+0000 7fd43a19b700 1 --2- 172.21.15.98:0/3962533892 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd43415ca70 0x7fd4340cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:50.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.076+0000 7fd43999a700 1 --2- 172.21.15.98:0/3962533892 >> v2:172.21.15.100:3300/0 conn(0x7fd43406b4b0 0x7fd4340cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:50.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.076+0000 7fd43a99c700 1 -- 172.21.15.98:0/3962533892 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd43415ca70 msgr2=0x7fd4340cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:50.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.076+0000 7fd43a99c700 1 --2- 172.21.15.98:0/3962533892 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd43415ca70 0x7fd4340cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.076+0000 7fd43a99c700 1 -- 172.21.15.98:0/3962533892 >> v2:172.21.15.100:3300/0 conn(0x7fd43406b4b0 msgr2=0x7fd4340cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:50.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.076+0000 7fd43a99c700 1 --2- 172.21.15.98:0/3962533892 >> v2:172.21.15.100:3300/0 conn(0x7fd43406b4b0 0x7fd4340cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.076+0000 7fd43a99c700 1 -- 172.21.15.98:0/3962533892 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4340d1800 con 0x7fd4340d00a0 2023-03-03T08:02:50.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.077+0000 7fd43a99c700 1 --2- 172.21.15.98:0/3962533892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4340d00a0 0x7fd4340b4c00 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fd4340b5d40 tx=0x7fd42c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee82cf4061e1b0ef server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:50.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.077+0000 7fd439199700 1 -- 172.21.15.98:0/3962533892 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c00e070 con 0x7fd4340d00a0 2023-03-03T08:02:50.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.077+0000 7fd439199700 1 -- 172.21.15.98:0/3962533892 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42c00b8d0 con 0x7fd4340d00a0 2023-03-03T08:02:50.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.077+0000 7fd439199700 1 -- 172.21.15.98:0/3962533892 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c00bc80 con 0x7fd4340d00a0 2023-03-03T08:02:50.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.078+0000 7fd43c3ff700 1 -- 172.21.15.98:0/3962533892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4340d00a0 msgr2=0x7fd4340b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:50.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.078+0000 7fd43c3ff700 1 --2- 172.21.15.98:0/3962533892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4340d00a0 0x7fd4340b4c00 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fd4340b5d40 tx=0x7fd42c00b330 comp rx=0 tx=0).stop 2023-03-03T08:02:50.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.078+0000 7fd43c3ff700 1 -- 172.21.15.98:0/3962533892 shutdown_connections 2023-03-03T08:02:50.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.078+0000 7fd43c3ff700 1 --2- 172.21.15.98:0/3962533892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4340d00a0 0x7fd4340b4c00 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.078+0000 7fd43c3ff700 1 --2- 172.21.15.98:0/3962533892 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd43415ca70 0x7fd4340cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.078+0000 7fd43c3ff700 1 --2- 172.21.15.98:0/3962533892 >> v2:172.21.15.100:3300/0 conn(0x7fd43406b4b0 0x7fd4340cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.078+0000 7fd43c3ff700 1 -- 172.21.15.98:0/3962533892 >> 172.21.15.98:0/3962533892 conn(0x7fd434073e70 msgr2=0x7fd43406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.078+0000 7fd43c3ff700 1 -- 172.21.15.98:0/3962533892 shutdown_connections 2023-03-03T08:02:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.078+0000 7fd43c3ff700 1 -- 172.21.15.98:0/3962533892 wait complete. 2023-03-03T08:02:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.080+0000 7fd43c3ff700 1 Processor -- start 2023-03-03T08:02:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.080+0000 7fd43c3ff700 1 -- start start 2023-03-03T08:02:50.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.081+0000 7fd43c3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd43406b4b0 0x7fd434155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:50.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.081+0000 7fd43c3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4340d00a0 0x7fd4340b6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:50.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.081+0000 7fd43c3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd43415ca70 0x7fd434156f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.081+0000 7fd43c3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4340b2f20 con 0x7fd4340d00a0 2023-03-03T08:02:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.081+0000 7fd43c3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4340b74c0 con 0x7fd43406b4b0 2023-03-03T08:02:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.081+0000 7fd43c3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4340b77a0 con 0x7fd43415ca70 2023-03-03T08:02:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.082+0000 7fd43999a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4340d00a0 0x7fd4340b6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.082+0000 7fd43999a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4340d00a0 0x7fd4340b6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52928/0 (socket says 172.21.15.98:52928) 2023-03-03T08:02:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.082+0000 7fd43999a700 1 -- 172.21.15.98:0/1898722884 learned_addr learned my addr 172.21.15.98:0/1898722884 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.082+0000 7fd43a19b700 1 --2- 172.21.15.98:0/1898722884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd43406b4b0 0x7fd434155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.082+0000 7fd43a99c700 1 --2- 172.21.15.98:0/1898722884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd43415ca70 0x7fd434156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.082+0000 7fd43999a700 1 -- 172.21.15.98:0/1898722884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd43415ca70 msgr2=0x7fd434156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.082+0000 7fd43999a700 1 --2- 172.21.15.98:0/1898722884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd43415ca70 0x7fd434156f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.082+0000 7fd43999a700 1 -- 172.21.15.98:0/1898722884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd43406b4b0 msgr2=0x7fd434155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.082+0000 7fd43999a700 1 --2- 172.21.15.98:0/1898722884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd43406b4b0 0x7fd434155740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.082+0000 7fd43999a700 1 -- 172.21.15.98:0/1898722884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42c009b30 con 0x7fd4340d00a0 2023-03-03T08:02:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.083+0000 7fd43999a700 1 --2- 172.21.15.98:0/1898722884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4340d00a0 0x7fd4340b6ae0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fd42400e980 tx=0x7fd42400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.083+0000 7fd42affd700 1 -- 172.21.15.98:0/1898722884 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd424009710 con 0x7fd4340d00a0 2023-03-03T08:02:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.083+0000 7fd42affd700 1 -- 172.21.15.98:0/1898722884 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42400c690 con 0x7fd4340d00a0 2023-03-03T08:02:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.083+0000 7fd42affd700 1 -- 172.21.15.98:0/1898722884 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42400ca20 con 0x7fd4340d00a0 2023-03-03T08:02:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.083+0000 7fd43c3ff700 1 -- 172.21.15.98:0/1898722884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4340cdbe0 con 0x7fd4340d00a0 2023-03-03T08:02:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.083+0000 7fd43c3ff700 1 -- 172.21.15.98:0/1898722884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4340cd900 con 0x7fd4340d00a0 2023-03-03T08:02:50.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.084+0000 7fd43c3ff700 1 -- 172.21.15.98:0/1898722884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4340d1aa0 con 0x7fd4340d00a0 2023-03-03T08:02:50.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.085+0000 7fd42affd700 1 -- 172.21.15.98:0/1898722884 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fd424019030 con 0x7fd4340d00a0 2023-03-03T08:02:50.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.087+0000 7fd42affd700 1 --2- 172.21.15.98:0/1898722884 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd418068380 0x7fd41806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:50.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.087+0000 7fd42affd700 1 -- 172.21.15.98:0/1898722884 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd424085c50 con 0x7fd4340d00a0 2023-03-03T08:02:50.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.087+0000 7fd43a19b700 1 --2- 172.21.15.98:0/1898722884 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd418068380 0x7fd41806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:50.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.088+0000 7fd43a19b700 1 --2- 172.21.15.98:0/1898722884 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd418068380 0x7fd41806a830 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd434157f00 tx=0x7fd430009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:50.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.091+0000 7fd42affd700 1 -- 172.21.15.98:0/1898722884 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd424052370 con 0x7fd4340d00a0 2023-03-03T08:02:50.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.285+0000 7fd43c3ff700 1 -- 172.21.15.98:0/1898722884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd43404e6e0 con 0x7fd4340d00a0 2023-03-03T08:02:50.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.285+0000 7fd42affd700 1 -- 172.21.15.98:0/1898722884 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd424055990 con 0x7fd4340d00a0 2023-03-03T08:02:50.288 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:02:50.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.287+0000 7fd43c3ff700 1 -- 172.21.15.98:0/1898722884 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd418068380 msgr2=0x7fd41806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:50.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.287+0000 7fd43c3ff700 1 --2- 172.21.15.98:0/1898722884 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd418068380 0x7fd41806a830 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd434157f00 tx=0x7fd430009210 comp rx=0 tx=0).stop 2023-03-03T08:02:50.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.288+0000 7fd43c3ff700 1 -- 172.21.15.98:0/1898722884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4340d00a0 msgr2=0x7fd4340b6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:50.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.288+0000 7fd43c3ff700 1 --2- 172.21.15.98:0/1898722884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4340d00a0 0x7fd4340b6ae0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fd42400e980 tx=0x7fd42400ec90 comp rx=0 tx=0).stop 2023-03-03T08:02:50.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.288+0000 7fd43c3ff700 1 -- 172.21.15.98:0/1898722884 shutdown_connections 2023-03-03T08:02:50.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.288+0000 7fd43c3ff700 1 --2- 172.21.15.98:0/1898722884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd43415ca70 0x7fd434156f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.288+0000 7fd43c3ff700 1 --2- 172.21.15.98:0/1898722884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd43406b4b0 0x7fd434155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.288+0000 7fd43c3ff700 1 --2- 172.21.15.98:0/1898722884 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd418068380 0x7fd41806a830 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.288+0000 7fd43c3ff700 1 --2- 172.21.15.98:0/1898722884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4340d00a0 0x7fd4340b6ae0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.288+0000 7fd43c3ff700 1 -- 172.21.15.98:0/1898722884 >> 172.21.15.98:0/1898722884 conn(0x7fd434073e70 msgr2=0x7fd4340b7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:50.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.288+0000 7fd43c3ff700 1 -- 172.21.15.98:0/1898722884 shutdown_connections 2023-03-03T08:02:50.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.288+0000 7fd43c3ff700 1 -- 172.21.15.98:0/1898722884 wait complete. 2023-03-03T08:02:50.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:02:50.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.564+0000 7fb74f19d700 1 Processor -- start 2023-03-03T08:02:50.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.564+0000 7fb74f19d700 1 -- start start 2023-03-03T08:02:50.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.566+0000 7fb74f19d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb7481609b0 0x7fb748160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:50.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.566+0000 7fb74f19d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7481612c0 con 0x7fb7480cf570 2023-03-03T08:02:50.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.566+0000 7fb74f19d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74815dd00 con 0x7fb7480b4aa0 2023-03-03T08:02:50.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.566+0000 7fb74f19d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7480cebb0 con 0x7fb7481609b0 2023-03-03T08:02:50.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.566+0000 7fb74d99a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb7480cf570 0x7fb74815d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37258/0 (socket says 172.21.15.98:37258) 2023-03-03T08:02:50.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.566+0000 7fb74d99a700 1 -- 172.21.15.98:0/3496938363 learned_addr learned my addr 172.21.15.98:0/3496938363 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:50.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.566+0000 7fb74e19b700 1 --2- 172.21.15.98:0/3496938363 >> v2:172.21.15.100:3300/0 conn(0x7fb7481609b0 0x7fb748160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:50.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.567+0000 7fb74d199700 1 -- 172.21.15.98:0/3496938363 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422859707 0 0) 0x7fb7481612c0 con 0x7fb7480cf570 2023-03-03T08:02:50.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.567+0000 7fb74d199700 1 -- 172.21.15.98:0/3496938363 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7340035b0 con 0x7fb7480cf570 2023-03-03T08:02:50.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.567+0000 7fb74e19b700 1 -- 172.21.15.98:0/3496938363 >> v1:172.21.15.100:6789/0 conn(0x7fb7480b4aa0 legacy=0x7fb7480d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:50.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.567+0000 7fb74e19b700 1 -- 172.21.15.98:0/3496938363 >> v1:172.21.15.98:6789/0 conn(0x7fb7480cf570 legacy=0x7fb74815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:50.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.567+0000 7fb74e19b700 1 -- 172.21.15.98:0/3496938363 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7480cd780 con 0x7fb7481609b0 2023-03-03T08:02:50.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.567+0000 7fb74e19b700 1 --2- 172.21.15.98:0/3496938363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7481609b0 0x7fb748160d80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb744009840 tx=0x7fb744009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b505f6e7e114ddf0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.568+0000 7fb74d199700 1 -- 172.21.15.98:0/3496938363 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb744011830 con 0x7fb7481609b0 2023-03-03T08:02:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.568+0000 7fb74d199700 1 -- 172.21.15.98:0/3496938363 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb744011990 con 0x7fb7481609b0 2023-03-03T08:02:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.568+0000 7fb74d199700 1 -- 172.21.15.98:0/3496938363 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb744011d00 con 0x7fb7481609b0 2023-03-03T08:02:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.568+0000 7fb74f19d700 1 -- 172.21.15.98:0/3496938363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7481609b0 msgr2=0x7fb748160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.568+0000 7fb74f19d700 1 --2- 172.21.15.98:0/3496938363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7481609b0 0x7fb748160d80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb744009840 tx=0x7fb744009b50 comp rx=0 tx=0).stop 2023-03-03T08:02:50.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.568+0000 7fb74f19d700 1 -- 172.21.15.98:0/3496938363 shutdown_connections 2023-03-03T08:02:50.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.568+0000 7fb74f19d700 1 --2- 172.21.15.98:0/3496938363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7481609b0 0x7fb748160d80 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.568+0000 7fb74f19d700 1 -- 172.21.15.98:0/3496938363 >> 172.21.15.98:0/3496938363 conn(0x7fb74806b880 msgr2=0x7fb7480cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:50.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.568+0000 7fb74f19d700 1 -- 172.21.15.98:0/3496938363 shutdown_connections 2023-03-03T08:02:50.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.568+0000 7fb74f19d700 1 -- 172.21.15.98:0/3496938363 wait complete. 2023-03-03T08:02:50.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.570+0000 7fb74f19d700 1 Processor -- start 2023-03-03T08:02:50.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.571+0000 7fb74f19d700 1 -- start start 2023-03-03T08:02:50.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74f19d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7480b4aa0 0x7fb74814c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:50.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74f19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7480cf570 0x7fb7480b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:50.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74f19d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7481609b0 0x7fb748151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:50.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74f19d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7480ca0f0 con 0x7fb7480b4aa0 2023-03-03T08:02:50.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74f19d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74814c5d0 con 0x7fb7481609b0 2023-03-03T08:02:50.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74f19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7480ca990 con 0x7fb7480cf570 2023-03-03T08:02:50.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74e19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7480b4aa0 0x7fb74814c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:50.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74e19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7480b4aa0 0x7fb74814c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52932/0 (socket says 172.21.15.98:52932) 2023-03-03T08:02:50.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74e19b700 1 -- 172.21.15.98:0/866559471 learned_addr learned my addr 172.21.15.98:0/866559471 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:50.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74e99c700 1 --2- 172.21.15.98:0/866559471 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7481609b0 0x7fb748151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:50.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74d99a700 1 --2- 172.21.15.98:0/866559471 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7480cf570 0x7fb7480b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:50.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74e19b700 1 -- 172.21.15.98:0/866559471 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7480cf570 msgr2=0x7fb7480b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:50.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74e19b700 1 --2- 172.21.15.98:0/866559471 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7480cf570 0x7fb7480b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74e19b700 1 -- 172.21.15.98:0/866559471 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7481609b0 msgr2=0x7fb748151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74e19b700 1 --2- 172.21.15.98:0/866559471 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7481609b0 0x7fb748151870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.572+0000 7fb74e19b700 1 -- 172.21.15.98:0/866559471 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7440094a0 con 0x7fb7480b4aa0 2023-03-03T08:02:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.573+0000 7fb74d99a700 1 --2- 172.21.15.98:0/866559471 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7480cf570 0x7fb7480b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.573+0000 7fb74e19b700 1 --2- 172.21.15.98:0/866559471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7480b4aa0 0x7fb74814c090 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fb744000c00 tx=0x7fb744004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.573+0000 7fb73effd700 1 -- 172.21.15.98:0/866559471 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7440038a0 con 0x7fb7480b4aa0 2023-03-03T08:02:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.573+0000 7fb73effd700 1 -- 172.21.15.98:0/866559471 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74401e070 con 0x7fb7480b4aa0 2023-03-03T08:02:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.573+0000 7fb73effd700 1 -- 172.21.15.98:0/866559471 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb744003c00 con 0x7fb7480b4aa0 2023-03-03T08:02:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.573+0000 7fb74f19d700 1 -- 172.21.15.98:0/866559471 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7480cdd40 con 0x7fb7480b4aa0 2023-03-03T08:02:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.573+0000 7fb74f19d700 1 -- 172.21.15.98:0/866559471 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7480cda60 con 0x7fb7480b4aa0 2023-03-03T08:02:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.575+0000 7fb74f19d700 1 -- 172.21.15.98:0/866559471 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74815cdf0 con 0x7fb7480b4aa0 2023-03-03T08:02:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.576+0000 7fb73effd700 1 -- 172.21.15.98:0/866559471 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fb74400f050 con 0x7fb7480b4aa0 2023-03-03T08:02:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.578+0000 7fb73effd700 1 --2- 172.21.15.98:0/866559471 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb72c068380 0x7fb72c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.578+0000 7fb73effd700 1 -- 172.21.15.98:0/866559471 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb744086e10 con 0x7fb7480b4aa0 2023-03-03T08:02:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.578+0000 7fb74d99a700 1 --2- 172.21.15.98:0/866559471 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb72c068380 0x7fb72c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.579+0000 7fb74d99a700 1 --2- 172.21.15.98:0/866559471 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb72c068380 0x7fb72c06a830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb7380011e0 tx=0x7fb738009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.582+0000 7fb73effd700 1 -- 172.21.15.98:0/866559471 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb74408a030 con 0x7fb7480b4aa0 2023-03-03T08:02:50.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.777+0000 7fb74f19d700 1 -- 172.21.15.98:0/866559471 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb748068290 con 0x7fb7480b4aa0 2023-03-03T08:02:50.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.778+0000 7fb73effd700 1 -- 172.21.15.98:0/866559471 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v98) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb744020020 con 0x7fb7480b4aa0 2023-03-03T08:02:50.780 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:02:50.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.780+0000 7fb74f19d700 1 -- 172.21.15.98:0/866559471 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb72c068380 msgr2=0x7fb72c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:50.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.780+0000 7fb74f19d700 1 --2- 172.21.15.98:0/866559471 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb72c068380 0x7fb72c06a830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb7380011e0 tx=0x7fb738009040 comp rx=0 tx=0).stop 2023-03-03T08:02:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.780+0000 7fb74f19d700 1 -- 172.21.15.98:0/866559471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7480b4aa0 msgr2=0x7fb74814c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.780+0000 7fb74f19d700 1 --2- 172.21.15.98:0/866559471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7480b4aa0 0x7fb74814c090 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fb744000c00 tx=0x7fb744004750 comp rx=0 tx=0).stop 2023-03-03T08:02:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.780+0000 7fb74f19d700 1 -- 172.21.15.98:0/866559471 shutdown_connections 2023-03-03T08:02:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.780+0000 7fb74f19d700 1 --2- 172.21.15.98:0/866559471 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb72c068380 0x7fb72c06a830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.780+0000 7fb74f19d700 1 --2- 172.21.15.98:0/866559471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7480b4aa0 0x7fb74814c090 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.780+0000 7fb74f19d700 1 --2- 172.21.15.98:0/866559471 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb7481609b0 0x7fb748151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.780+0000 7fb74f19d700 1 --2- 172.21.15.98:0/866559471 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb7480cf570 0x7fb7480b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:50.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.780+0000 7fb74f19d700 1 -- 172.21.15.98:0/866559471 >> 172.21.15.98:0/866559471 conn(0x7fb74806b880 msgr2=0x7fb748153120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:50.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.781+0000 7fb74f19d700 1 -- 172.21.15.98:0/866559471 shutdown_connections 2023-03-03T08:02:50.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:50.781+0000 7fb74f19d700 1 -- 172.21.15.98:0/866559471 wait complete. 2023-03-03T08:02:50.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T08:02:51.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.056+0000 7ff8c33ff700 1 Processor -- start 2023-03-03T08:02:51.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.056+0000 7ff8c33ff700 1 -- start start 2023-03-03T08:02:51.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.058+0000 7ff8c33ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff8bc0b5310 0x7ff8bc167850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.058+0000 7ff8c33ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8bc0cd7d0 con 0x7ff8bc0b26d0 2023-03-03T08:02:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.058+0000 7ff8c33ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8bc0b90e0 con 0x7ff8bc0b8d10 2023-03-03T08:02:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.058+0000 7ff8c33ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8bc0cda20 con 0x7ff8bc0b5310 2023-03-03T08:02:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.058+0000 7ff8c099a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff8bc0b26d0 0x7ff8bc0b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37274/0 (socket says 172.21.15.98:37274) 2023-03-03T08:02:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.058+0000 7ff8c099a700 1 -- 172.21.15.98:0/3029884946 learned_addr learned my addr 172.21.15.98:0/3029884946 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.058+0000 7ff8c199c700 1 --2- 172.21.15.98:0/3029884946 >> v2:172.21.15.100:3300/0 conn(0x7ff8bc0b5310 0x7ff8bc167850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:51.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.058+0000 7ff8b3fff700 1 -- 172.21.15.98:0/3029884946 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613587067 0 0) 0x7ff8bc0cd7d0 con 0x7ff8bc0b26d0 2023-03-03T08:02:51.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.059+0000 7ff8b3fff700 1 -- 172.21.15.98:0/3029884946 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a80035b0 con 0x7ff8bc0b26d0 2023-03-03T08:02:51.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.059+0000 7ff8b3fff700 1 -- 172.21.15.98:0/3029884946 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699265219 0 0) 0x7ff8bc0b90e0 con 0x7ff8bc0b8d10 2023-03-03T08:02:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.059+0000 7ff8b3fff700 1 -- 172.21.15.98:0/3029884946 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8bc0cd7d0 con 0x7ff8bc0b8d10 2023-03-03T08:02:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.059+0000 7ff8b3fff700 1 -- 172.21.15.98:0/3029884946 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1581808794 0 0) 0x7ff8a80035b0 con 0x7ff8bc0b26d0 2023-03-03T08:02:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.059+0000 7ff8c199c700 1 -- 172.21.15.98:0/3029884946 >> v1:172.21.15.100:6789/0 conn(0x7ff8bc0b8d10 legacy=0x7ff8bc0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.059+0000 7ff8c199c700 1 -- 172.21.15.98:0/3029884946 >> v1:172.21.15.98:6789/0 conn(0x7ff8bc0b26d0 legacy=0x7ff8bc0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.059+0000 7ff8c199c700 1 -- 172.21.15.98:0/3029884946 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8bc0cdd00 con 0x7ff8bc0b5310 2023-03-03T08:02:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.059+0000 7ff8c199c700 1 --2- 172.21.15.98:0/3029884946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8bc0b5310 0x7ff8bc167850 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009d90 tx=0x7ff8b400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66100a9c36129607 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.060+0000 7ff8b3fff700 1 -- 172.21.15.98:0/3029884946 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b400e070 con 0x7ff8bc0b5310 2023-03-03T08:02:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.060+0000 7ff8b3fff700 1 -- 172.21.15.98:0/3029884946 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b4003a20 con 0x7ff8bc0b5310 2023-03-03T08:02:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.060+0000 7ff8b3fff700 1 -- 172.21.15.98:0/3029884946 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac001670 con 0x7ff8bc0b5310 2023-03-03T08:02:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.060+0000 7ff8c33ff700 1 -- 172.21.15.98:0/3029884946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8bc0b5310 msgr2=0x7ff8bc167850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.060+0000 7ff8c33ff700 1 --2- 172.21.15.98:0/3029884946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8bc0b5310 0x7ff8bc167850 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009d90 tx=0x7ff8b400b330 comp rx=0 tx=0).stop 2023-03-03T08:02:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.060+0000 7ff8c33ff700 1 -- 172.21.15.98:0/3029884946 shutdown_connections 2023-03-03T08:02:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.060+0000 7ff8c33ff700 1 --2- 172.21.15.98:0/3029884946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8bc0b5310 0x7ff8bc167850 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.060+0000 7ff8c33ff700 1 -- 172.21.15.98:0/3029884946 >> 172.21.15.98:0/3029884946 conn(0x7ff8bc073e70 msgr2=0x7ff8bc06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.060+0000 7ff8c33ff700 1 -- 172.21.15.98:0/3029884946 shutdown_connections 2023-03-03T08:02:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.060+0000 7ff8c33ff700 1 -- 172.21.15.98:0/3029884946 wait complete. 2023-03-03T08:02:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.062+0000 7ff8c33ff700 1 Processor -- start 2023-03-03T08:02:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.063+0000 7ff8c33ff700 1 -- start start 2023-03-03T08:02:51.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.063+0000 7ff8c33ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8bc0b26d0 0x7ff8bc15e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:51.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.063+0000 7ff8c33ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8bc0b5310 0x7ff8bc0cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.063+0000 7ff8c33ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8bc0b8d10 0x7ff8bc0cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.063+0000 7ff8c33ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc158d10 con 0x7ff8bc0b5310 2023-03-03T08:02:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.064+0000 7ff8c33ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc157c70 con 0x7ff8bc0b8d10 2023-03-03T08:02:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.064+0000 7ff8c33ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc157f50 con 0x7ff8bc0b26d0 2023-03-03T08:02:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.064+0000 7ff8c099a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8bc0b5310 0x7ff8bc0cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.064+0000 7ff8c099a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8bc0b5310 0x7ff8bc0cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52948/0 (socket says 172.21.15.98:52948) 2023-03-03T08:02:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.064+0000 7ff8c099a700 1 -- 172.21.15.98:0/2632966505 learned_addr learned my addr 172.21.15.98:0/2632966505 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.064+0000 7ff8c119b700 1 --2- 172.21.15.98:0/2632966505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8bc0b26d0 0x7ff8bc15e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.064+0000 7ff8c199c700 1 --2- 172.21.15.98:0/2632966505 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8bc0b8d10 0x7ff8bc0cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.064+0000 7ff8c099a700 1 -- 172.21.15.98:0/2632966505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8bc0b26d0 msgr2=0x7ff8bc15e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.064+0000 7ff8c099a700 1 --2- 172.21.15.98:0/2632966505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8bc0b26d0 0x7ff8bc15e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.064+0000 7ff8c099a700 1 -- 172.21.15.98:0/2632966505 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8bc0b8d10 msgr2=0x7ff8bc0cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.064+0000 7ff8c099a700 1 --2- 172.21.15.98:0/2632966505 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8bc0b8d10 0x7ff8bc0cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.064+0000 7ff8c099a700 1 -- 172.21.15.98:0/2632966505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b4009a70 con 0x7ff8bc0b5310 2023-03-03T08:02:51.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.064+0000 7ff8c199c700 1 --2- 172.21.15.98:0/2632966505 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8bc0b8d10 0x7ff8bc0cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:51.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.065+0000 7ff8c099a700 1 --2- 172.21.15.98:0/2632966505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8bc0b5310 0x7ff8bc0cc420 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00c500 tx=0x7ff8ac00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:51.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.065+0000 7ff8b1ffb700 1 -- 172.21.15.98:0/2632966505 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac004c80 con 0x7ff8bc0b5310 2023-03-03T08:02:51.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.065+0000 7ff8b1ffb700 1 -- 172.21.15.98:0/2632966505 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ac004de0 con 0x7ff8bc0b5310 2023-03-03T08:02:51.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.065+0000 7ff8b1ffb700 1 -- 172.21.15.98:0/2632966505 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac007310 con 0x7ff8bc0b5310 2023-03-03T08:02:51.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.065+0000 7ff8c33ff700 1 -- 172.21.15.98:0/2632966505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8bc0cdfe0 con 0x7ff8bc0b5310 2023-03-03T08:02:51.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.065+0000 7ff8c33ff700 1 -- 172.21.15.98:0/2632966505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8bc0d0e80 con 0x7ff8bc0b5310 2023-03-03T08:02:51.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.066+0000 7ff8c33ff700 1 -- 172.21.15.98:0/2632966505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8bc0b5fd0 con 0x7ff8bc0b5310 2023-03-03T08:02:51.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.067+0000 7ff8b1ffb700 1 -- 172.21.15.98:0/2632966505 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7ff8ac01a020 con 0x7ff8bc0b5310 2023-03-03T08:02:51.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.069+0000 7ff8b1ffb700 1 --2- 172.21.15.98:0/2632966505 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff8a00683b0 0x7ff8a006a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:51.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.069+0000 7ff8b1ffb700 1 -- 172.21.15.98:0/2632966505 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff8ac085b00 con 0x7ff8bc0b5310 2023-03-03T08:02:51.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.069+0000 7ff8c119b700 1 --2- 172.21.15.98:0/2632966505 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff8a00683b0 0x7ff8a006a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:51.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.070+0000 7ff8c119b700 1 --2- 172.21.15.98:0/2632966505 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff8a00683b0 0x7ff8a006a860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff8b80011e0 tx=0x7ff8b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:51.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.072+0000 7ff8b1ffb700 1 -- 172.21.15.98:0/2632966505 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff8ac052190 con 0x7ff8bc0b5310 2023-03-03T08:02:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.265+0000 7ff8c33ff700 1 -- 172.21.15.98:0/2632966505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff8bc04e6e0 con 0x7ff8bc0b5310 2023-03-03T08:02:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.266+0000 7ff8b1ffb700 1 -- 172.21.15.98:0/2632966505 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff8ac0557b0 con 0x7ff8bc0b5310 2023-03-03T08:02:51.268 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T08:02:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.268+0000 7ff8c33ff700 1 -- 172.21.15.98:0/2632966505 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff8a00683b0 msgr2=0x7ff8a006a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.268+0000 7ff8c33ff700 1 --2- 172.21.15.98:0/2632966505 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff8a00683b0 0x7ff8a006a860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff8b80011e0 tx=0x7ff8b8009040 comp rx=0 tx=0).stop 2023-03-03T08:02:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.268+0000 7ff8c33ff700 1 -- 172.21.15.98:0/2632966505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8bc0b5310 msgr2=0x7ff8bc0cc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.268+0000 7ff8c33ff700 1 --2- 172.21.15.98:0/2632966505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8bc0b5310 0x7ff8bc0cc420 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00c500 tx=0x7ff8ac00c8c0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.268+0000 7ff8c33ff700 1 -- 172.21.15.98:0/2632966505 shutdown_connections 2023-03-03T08:02:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.268+0000 7ff8c33ff700 1 --2- 172.21.15.98:0/2632966505 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8bc0b8d10 0x7ff8bc0cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.268+0000 7ff8c33ff700 1 --2- 172.21.15.98:0/2632966505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8bc0b26d0 0x7ff8bc15e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.268+0000 7ff8c33ff700 1 --2- 172.21.15.98:0/2632966505 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff8a00683b0 0x7ff8a006a860 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.268+0000 7ff8c33ff700 1 --2- 172.21.15.98:0/2632966505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8bc0b5310 0x7ff8bc0cc420 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.268+0000 7ff8c33ff700 1 -- 172.21.15.98:0/2632966505 >> 172.21.15.98:0/2632966505 conn(0x7ff8bc073e70 msgr2=0x7ff8bc06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.268+0000 7ff8c33ff700 1 -- 172.21.15.98:0/2632966505 shutdown_connections 2023-03-03T08:02:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.268+0000 7ff8c33ff700 1 -- 172.21.15.98:0/2632966505 wait complete. 2023-03-03T08:02:51.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T08:02:51.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.542+0000 7fd8763ff700 1 Processor -- start 2023-03-03T08:02:51.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.542+0000 7fd8763ff700 1 -- start start 2023-03-03T08:02:51.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.543+0000 7fd8763ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd8700b8d10 0x7fd8700cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.543+0000 7fd8763ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8700b5310 0x7fd8700d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.543+0000 7fd8763ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8700cd7d0 con 0x7fd8700b26d0 2023-03-03T08:02:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.543+0000 7fd8763ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8700b90e0 con 0x7fd8700b8d10 2023-03-03T08:02:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.543+0000 7fd8763ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8700cda20 con 0x7fd8700b5310 2023-03-03T08:02:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.544+0000 7fd86f7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd8700b26d0 0x7fd8700b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37276/0 (socket says 172.21.15.98:37276) 2023-03-03T08:02:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.544+0000 7fd86f7fe700 1 -- 172.21.15.98:0/2496059546 learned_addr learned my addr 172.21.15.98:0/2496059546 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.544+0000 7fd86ffff700 1 --2- 172.21.15.98:0/2496059546 >> v2:172.21.15.100:3300/0 conn(0x7fd8700b8d10 0x7fd8700cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.544+0000 7fd87499c700 1 --2- 172.21.15.98:0/2496059546 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8700b5310 0x7fd8700d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.544+0000 7fd86effd700 1 -- 172.21.15.98:0/2496059546 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120232627 0 0) 0x7fd8700cd7d0 con 0x7fd8700b26d0 2023-03-03T08:02:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.544+0000 7fd86effd700 1 -- 172.21.15.98:0/2496059546 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8580035b0 con 0x7fd8700b26d0 2023-03-03T08:02:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.545+0000 7fd86ffff700 1 -- 172.21.15.98:0/2496059546 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8700b5310 msgr2=0x7fd8700d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.545+0000 7fd86ffff700 1 --2- 172.21.15.98:0/2496059546 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8700b5310 0x7fd8700d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.545+0000 7fd86effd700 1 -- 172.21.15.98:0/2496059546 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 273009279 0 0) 0x7fd8580035b0 con 0x7fd8700b26d0 2023-03-03T08:02:51.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.545+0000 7fd86ffff700 1 -- 172.21.15.98:0/2496059546 >> v1:172.21.15.98:6789/0 conn(0x7fd8700b26d0 legacy=0x7fd8700b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:51.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.545+0000 7fd86ffff700 1 -- 172.21.15.98:0/2496059546 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8700cdd00 con 0x7fd8700b8d10 2023-03-03T08:02:51.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.545+0000 7fd86ffff700 1 --2- 172.21.15.98:0/2496059546 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8700b8d10 0x7fd8700cffc0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd860009840 tx=0x7fd860009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed1d9fe4b365eae8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:51.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.545+0000 7fd86effd700 1 -- 172.21.15.98:0/2496059546 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864001670 con 0x7fd8700b8d10 2023-03-03T08:02:51.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.545+0000 7fd86effd700 1 -- 172.21.15.98:0/2496059546 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd860004030 con 0x7fd8700b8d10 2023-03-03T08:02:51.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.545+0000 7fd86effd700 1 -- 172.21.15.98:0/2496059546 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8600043c0 con 0x7fd8700b8d10 2023-03-03T08:02:51.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.546+0000 7fd8763ff700 1 -- 172.21.15.98:0/2496059546 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8700b8d10 msgr2=0x7fd8700cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:51.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.546+0000 7fd8763ff700 1 --2- 172.21.15.98:0/2496059546 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8700b8d10 0x7fd8700cffc0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd860009840 tx=0x7fd860009b50 comp rx=0 tx=0).stop 2023-03-03T08:02:51.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.546+0000 7fd8763ff700 1 -- 172.21.15.98:0/2496059546 shutdown_connections 2023-03-03T08:02:51.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.546+0000 7fd8763ff700 1 --2- 172.21.15.98:0/2496059546 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8700b8d10 0x7fd8700cffc0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.546+0000 7fd8763ff700 1 --2- 172.21.15.98:0/2496059546 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8700b5310 0x7fd8700d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.546+0000 7fd8763ff700 1 -- 172.21.15.98:0/2496059546 >> 172.21.15.98:0/2496059546 conn(0x7fd870073e70 msgr2=0x7fd87006ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.546+0000 7fd8763ff700 1 -- 172.21.15.98:0/2496059546 shutdown_connections 2023-03-03T08:02:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.546+0000 7fd8763ff700 1 -- 172.21.15.98:0/2496059546 wait complete. 2023-03-03T08:02:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.548+0000 7fd8763ff700 1 Processor -- start 2023-03-03T08:02:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.548+0000 7fd8763ff700 1 -- start start 2023-03-03T08:02:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.549+0000 7fd8763ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8700b26d0 0x7fd870151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.549+0000 7fd8763ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8700b5310 0x7fd8700b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.549+0000 7fd8763ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8700b8d10 0x7fd8700b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.549+0000 7fd8763ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87015dfa0 con 0x7fd8700b5310 2023-03-03T08:02:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.549+0000 7fd8763ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8701f23c0 con 0x7fd8700b26d0 2023-03-03T08:02:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.549+0000 7fd8763ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8701f26a0 con 0x7fd8700b8d10 2023-03-03T08:02:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.550+0000 7fd86f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8700b5310 0x7fd8700b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.550+0000 7fd86ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8700b26d0 0x7fd870151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.550+0000 7fd86ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8700b26d0 0x7fd870151360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:50356/0 (socket says 172.21.15.98:50356) 2023-03-03T08:02:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.550+0000 7fd86ffff700 1 -- 172.21.15.98:0/2446496119 learned_addr learned my addr 172.21.15.98:0/2446496119 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.550+0000 7fd87499c700 1 --2- 172.21.15.98:0/2446496119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8700b8d10 0x7fd8700b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.550+0000 7fd86f7fe700 1 -- 172.21.15.98:0/2446496119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8700b8d10 msgr2=0x7fd8700b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.550+0000 7fd86f7fe700 1 --2- 172.21.15.98:0/2446496119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8700b8d10 0x7fd8700b4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.550+0000 7fd86f7fe700 1 -- 172.21.15.98:0/2446496119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8700b26d0 msgr2=0x7fd870151360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.550+0000 7fd86f7fe700 1 --2- 172.21.15.98:0/2446496119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8700b26d0 0x7fd870151360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.550+0000 7fd86f7fe700 1 -- 172.21.15.98:0/2446496119 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8600094a0 con 0x7fd8700b5310 2023-03-03T08:02:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.550+0000 7fd86ffff700 1 --2- 172.21.15.98:0/2446496119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8700b26d0 0x7fd870151360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.551+0000 7fd86f7fe700 1 --2- 172.21.15.98:0/2446496119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8700b5310 0x7fd8700b4010 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fd86400c750 tx=0x7fd86400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:51.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.551+0000 7fd86cff9700 1 -- 172.21.15.98:0/2446496119 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864004c80 con 0x7fd8700b5310 2023-03-03T08:02:51.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.551+0000 7fd86cff9700 1 -- 172.21.15.98:0/2446496119 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd864004de0 con 0x7fd8700b5310 2023-03-03T08:02:51.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.551+0000 7fd86cff9700 1 -- 172.21.15.98:0/2446496119 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864007310 con 0x7fd8700b5310 2023-03-03T08:02:51.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.551+0000 7fd8763ff700 1 -- 172.21.15.98:0/2446496119 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8700cdfe0 con 0x7fd8700b5310 2023-03-03T08:02:51.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.551+0000 7fd8763ff700 1 -- 172.21.15.98:0/2446496119 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8700d0e80 con 0x7fd8700b5310 2023-03-03T08:02:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.552+0000 7fd8763ff700 1 -- 172.21.15.98:0/2446496119 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8700d0500 con 0x7fd8700b5310 2023-03-03T08:02:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.554+0000 7fd86cff9700 1 -- 172.21.15.98:0/2446496119 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fd86401a020 con 0x7fd8700b5310 2023-03-03T08:02:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.555+0000 7fd86cff9700 1 --2- 172.21.15.98:0/2446496119 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd85c068380 0x7fd85c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.555+0000 7fd86cff9700 1 -- 172.21.15.98:0/2446496119 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd8640858f0 con 0x7fd8700b5310 2023-03-03T08:02:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.556+0000 7fd86ffff700 1 --2- 172.21.15.98:0/2446496119 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd85c068380 0x7fd85c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.556+0000 7fd86ffff700 1 --2- 172.21.15.98:0/2446496119 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd85c068380 0x7fd85c06a830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd860000c00 tx=0x7fd860008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:51.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.559+0000 7fd86cff9700 1 -- 172.21.15.98:0/2446496119 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd8640520c0 con 0x7fd8700b5310 2023-03-03T08:02:51.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.751+0000 7fd8763ff700 1 -- 172.21.15.98:0/2446496119 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd87004e6e0 con 0x7fd8700b5310 2023-03-03T08:02:51.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.752+0000 7fd86cff9700 1 -- 172.21.15.98:0/2446496119 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd864083d70 con 0x7fd8700b5310 2023-03-03T08:02:51.753 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T08:02:51.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.754+0000 7fd8763ff700 1 -- 172.21.15.98:0/2446496119 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd85c068380 msgr2=0x7fd85c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:51.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.754+0000 7fd8763ff700 1 --2- 172.21.15.98:0/2446496119 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd85c068380 0x7fd85c06a830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd860000c00 tx=0x7fd860008030 comp rx=0 tx=0).stop 2023-03-03T08:02:51.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.754+0000 7fd8763ff700 1 -- 172.21.15.98:0/2446496119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8700b5310 msgr2=0x7fd8700b4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:51.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.754+0000 7fd8763ff700 1 --2- 172.21.15.98:0/2446496119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8700b5310 0x7fd8700b4010 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fd86400c750 tx=0x7fd86400ca60 comp rx=0 tx=0).stop 2023-03-03T08:02:51.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.754+0000 7fd8763ff700 1 -- 172.21.15.98:0/2446496119 shutdown_connections 2023-03-03T08:02:51.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.754+0000 7fd8763ff700 1 --2- 172.21.15.98:0/2446496119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8700b8d10 0x7fd8700b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.754+0000 7fd8763ff700 1 --2- 172.21.15.98:0/2446496119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8700b26d0 0x7fd870151360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.754+0000 7fd8763ff700 1 --2- 172.21.15.98:0/2446496119 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd85c068380 0x7fd85c06a830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.754+0000 7fd8763ff700 1 --2- 172.21.15.98:0/2446496119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8700b5310 0x7fd8700b4010 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:51.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.754+0000 7fd8763ff700 1 -- 172.21.15.98:0/2446496119 >> 172.21.15.98:0/2446496119 conn(0x7fd870073e70 msgr2=0x7fd8700d1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:51.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.754+0000 7fd8763ff700 1 -- 172.21.15.98:0/2446496119 shutdown_connections 2023-03-03T08:02:51.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:51.755+0000 7fd8763ff700 1 -- 172.21.15.98:0/2446496119 wait complete. 2023-03-03T08:02:51.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T08:02:52.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.027+0000 7f041e99d700 1 Processor -- start 2023-03-03T08:02:52.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.028+0000 7f041e99d700 1 -- start start 2023-03-03T08:02:52.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.029+0000 7f041e99d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f04181609b0 0x7f0418160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:52.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.029+0000 7f041e99d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04180b4aa0 0x7f04180b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:52.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.029+0000 7f041e99d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04181612c0 con 0x7f04180cf570 2023-03-03T08:02:52.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.029+0000 7f041e99d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04180cebb0 con 0x7f04181609b0 2023-03-03T08:02:52.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.029+0000 7f041e99d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04180b5460 con 0x7f04180b4aa0 2023-03-03T08:02:52.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.030+0000 7f041d99b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f04181609b0 0x7f0418160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:52.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.030+0000 7f041e19c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04180b4aa0 0x7f04180b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:52.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.030+0000 7f041d19a700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f04180cf570 0x7f041815d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:50306/0 (socket says 172.21.15.98:50306) 2023-03-03T08:02:52.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.030+0000 7f041d19a700 1 -- 172.21.15.98:0/4015085300 learned_addr learned my addr 172.21.15.98:0/4015085300 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:52.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.030+0000 7f041c999700 1 -- 172.21.15.98:0/4015085300 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947537649 0 0) 0x7f04181612c0 con 0x7f04180cf570 2023-03-03T08:02:52.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.030+0000 7f041c999700 1 -- 172.21.15.98:0/4015085300 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04040035b0 con 0x7f04180cf570 2023-03-03T08:02:52.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.030+0000 7f041e19c700 1 -- 172.21.15.98:0/4015085300 >> v2:172.21.15.100:3300/0 conn(0x7f04181609b0 msgr2=0x7f0418160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:52.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.030+0000 7f041e19c700 1 --2- 172.21.15.98:0/4015085300 >> v2:172.21.15.100:3300/0 conn(0x7f04181609b0 0x7f0418160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:52.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.030+0000 7f041e19c700 1 -- 172.21.15.98:0/4015085300 >> v1:172.21.15.100:6789/0 conn(0x7f04180cf570 legacy=0x7f041815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:52.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.030+0000 7f041e19c700 1 -- 172.21.15.98:0/4015085300 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041815dd00 con 0x7f04180b4aa0 2023-03-03T08:02:52.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.030+0000 7f041d99b700 1 --2- 172.21.15.98:0/4015085300 >> v2:172.21.15.100:3300/0 conn(0x7f04181609b0 0x7f0418160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:52.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.031+0000 7f041e19c700 1 --2- 172.21.15.98:0/4015085300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04180b4aa0 0x7f04180b4f20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0410009d90 tx=0x7f041000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f0905729666d453 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:52.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.031+0000 7f041c999700 1 -- 172.21.15.98:0/4015085300 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041000e070 con 0x7f04180b4aa0 2023-03-03T08:02:52.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.031+0000 7f041c999700 1 -- 172.21.15.98:0/4015085300 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041000bb20 con 0x7f04180b4aa0 2023-03-03T08:02:52.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.031+0000 7f041c999700 1 -- 172.21.15.98:0/4015085300 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0410012430 con 0x7f04180b4aa0 2023-03-03T08:02:52.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.031+0000 7f041e99d700 1 -- 172.21.15.98:0/4015085300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04180b4aa0 msgr2=0x7f04180b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:52.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.031+0000 7f041e99d700 1 --2- 172.21.15.98:0/4015085300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04180b4aa0 0x7f04180b4f20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0410009d90 tx=0x7f041000b330 comp rx=0 tx=0).stop 2023-03-03T08:02:52.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.032+0000 7f041e99d700 1 -- 172.21.15.98:0/4015085300 shutdown_connections 2023-03-03T08:02:52.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.032+0000 7f041e99d700 1 --2- 172.21.15.98:0/4015085300 >> v2:172.21.15.100:3300/0 conn(0x7f04181609b0 0x7f0418160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:52.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.032+0000 7f041e99d700 1 --2- 172.21.15.98:0/4015085300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04180b4aa0 0x7f04180b4f20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:52.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.032+0000 7f041e99d700 1 -- 172.21.15.98:0/4015085300 >> 172.21.15.98:0/4015085300 conn(0x7f041806b880 msgr2=0x7f04180cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:52.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.032+0000 7f041e99d700 1 -- 172.21.15.98:0/4015085300 shutdown_connections 2023-03-03T08:02:52.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.032+0000 7f041e99d700 1 -- 172.21.15.98:0/4015085300 wait complete. 2023-03-03T08:02:52.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.034+0000 7f041e99d700 1 Processor -- start 2023-03-03T08:02:52.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.034+0000 7f041e99d700 1 -- start start 2023-03-03T08:02:52.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.035+0000 7f041e99d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04180b4aa0 0x7f04181507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:52.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.035+0000 7f041e99d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04180cf570 0x7f04180c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:52.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.035+0000 7f041e99d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04181609b0 0x7f04180b7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:52.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.035+0000 7f041e99d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0418150cf0 con 0x7f04180b4aa0 2023-03-03T08:02:52.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.035+0000 7f041e99d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041815aaf0 con 0x7f04180cf570 2023-03-03T08:02:52.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.035+0000 7f041e99d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041815add0 con 0x7f04181609b0 2023-03-03T08:02:52.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.035+0000 7f041d99b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04180b4aa0 0x7f04181507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:52.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.035+0000 7f041d99b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04180b4aa0 0x7f04181507b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52962/0 (socket says 172.21.15.98:52962) 2023-03-03T08:02:52.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.035+0000 7f041d99b700 1 -- 172.21.15.98:0/1459811662 learned_addr learned my addr 172.21.15.98:0/1459811662 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:52.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.036+0000 7f041e19c700 1 --2- 172.21.15.98:0/1459811662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04181609b0 0x7f04180b7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:52.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.036+0000 7f041d19a700 1 --2- 172.21.15.98:0/1459811662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04180cf570 0x7f04180c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:52.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.036+0000 7f041d99b700 1 -- 172.21.15.98:0/1459811662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04181609b0 msgr2=0x7f04180b7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:52.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.036+0000 7f041d99b700 1 --2- 172.21.15.98:0/1459811662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04181609b0 0x7f04180b7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:52.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.036+0000 7f041d99b700 1 -- 172.21.15.98:0/1459811662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04180cf570 msgr2=0x7f04180c7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:52.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.036+0000 7f041d99b700 1 --2- 172.21.15.98:0/1459811662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04180cf570 0x7f04180c7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:52.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.036+0000 7f041d99b700 1 -- 172.21.15.98:0/1459811662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04140094a0 con 0x7f04180b4aa0 2023-03-03T08:02:52.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.036+0000 7f041d99b700 1 --2- 172.21.15.98:0/1459811662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04180b4aa0 0x7f04181507b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f041400ebe0 tx=0x7f0414009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:52.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.036+0000 7f040e7fc700 1 -- 172.21.15.98:0/1459811662 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041400c8b0 con 0x7f04180b4aa0 2023-03-03T08:02:52.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.036+0000 7f040e7fc700 1 -- 172.21.15.98:0/1459811662 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041400ca10 con 0x7f04180b4aa0 2023-03-03T08:02:52.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.036+0000 7f041e99d700 1 -- 172.21.15.98:0/1459811662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0410009a70 con 0x7f04180b4aa0 2023-03-03T08:02:52.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.036+0000 7f040e7fc700 1 -- 172.21.15.98:0/1459811662 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041400cda0 con 0x7f04180b4aa0 2023-03-03T08:02:52.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.037+0000 7f041e99d700 1 -- 172.21.15.98:0/1459811662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04180cda60 con 0x7f04180b4aa0 2023-03-03T08:02:52.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.038+0000 7f041e99d700 1 -- 172.21.15.98:0/1459811662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04180cd780 con 0x7f04180b4aa0 2023-03-03T08:02:52.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.039+0000 7f040e7fc700 1 -- 172.21.15.98:0/1459811662 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f0414019030 con 0x7f04180b4aa0 2023-03-03T08:02:52.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.040+0000 7f040e7fc700 1 --2- 172.21.15.98:0/1459811662 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f03fc068380 0x7f03fc06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:52.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.040+0000 7f040e7fc700 1 -- 172.21.15.98:0/1459811662 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0414085dd0 con 0x7f04180b4aa0 2023-03-03T08:02:52.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.041+0000 7f041d19a700 1 --2- 172.21.15.98:0/1459811662 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f03fc068380 0x7f03fc06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:52.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.041+0000 7f041d19a700 1 --2- 172.21.15.98:0/1459811662 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f03fc068380 0x7f03fc06a830 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f041815f710 tx=0x7f0408009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:52.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.044+0000 7f040e7fc700 1 -- 172.21.15.98:0/1459811662 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f04140524f0 con 0x7f04180b4aa0 2023-03-03T08:02:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.251+0000 7f041e99d700 1 -- 172.21.15.98:0/1459811662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f04180cdd40 con 0x7f04180b4aa0 2023-03-03T08:02:52.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.251+0000 7f040e7fc700 1 -- 172.21.15.98:0/1459811662 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v98) v1 ==== 80+0+124 (secure 0 0 0) 0x7f0414017090 con 0x7f04180b4aa0 2023-03-03T08:02:52.253 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.100:6835/1843752855 2023-03-04T08:02:42.175049+0000 2023-03-03T08:02:52.253 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.100:6833/1843752855 2023-03-04T08:02:42.175049+0000 2023-03-03T08:02:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.254+0000 7f041e99d700 1 -- 172.21.15.98:0/1459811662 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f03fc068380 msgr2=0x7f03fc06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.254+0000 7f041e99d700 1 --2- 172.21.15.98:0/1459811662 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f03fc068380 0x7f03fc06a830 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f041815f710 tx=0x7f0408009040 comp rx=0 tx=0).stop 2023-03-03T08:02:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.254+0000 7f041e99d700 1 -- 172.21.15.98:0/1459811662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04180b4aa0 msgr2=0x7f04181507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.254+0000 7f041e99d700 1 --2- 172.21.15.98:0/1459811662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04180b4aa0 0x7f04181507b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f041400ebe0 tx=0x7f0414009e20 comp rx=0 tx=0).stop 2023-03-03T08:02:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.254+0000 7f041e99d700 1 -- 172.21.15.98:0/1459811662 shutdown_connections 2023-03-03T08:02:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.254+0000 7f041e99d700 1 --2- 172.21.15.98:0/1459811662 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f03fc068380 0x7f03fc06a830 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.254+0000 7f041e99d700 1 --2- 172.21.15.98:0/1459811662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04180b4aa0 0x7f04181507b0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.254+0000 7f041e99d700 1 --2- 172.21.15.98:0/1459811662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04181609b0 0x7f04180b7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.254+0000 7f041e99d700 1 --2- 172.21.15.98:0/1459811662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04180cf570 0x7f04180c7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.254+0000 7f041e99d700 1 -- 172.21.15.98:0/1459811662 >> 172.21.15.98:0/1459811662 conn(0x7f041806b880 msgr2=0x7f04180d0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.254+0000 7f041e99d700 1 -- 172.21.15.98:0/1459811662 shutdown_connections 2023-03-03T08:02:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.254+0000 7f041e99d700 1 -- 172.21.15.98:0/1459811662 wait complete. 2023-03-03T08:02:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:listed 2 entries 2023-03-03T08:02:52.282 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2023-03-03T08:02:52.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:52.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-03-03T08:02:52.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[120896]: fuse finished with error 0 and tester_r 0 2023-03-03T08:02:52.389 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T08:02:52.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:52.390 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-03-03T08:02:52.461 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:02:52.462 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:02:52.530 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T08:02:52.531 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-03-03T08:02:52.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:52.531 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-03-03T08:02:52.600 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-03-03T08:02:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:02:52.603 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:02:52.604 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:02:52.674 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T08:02:52.675 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2023-03-03T08:02:52.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:02:52.675 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-03-03T08:02:52.742 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-03-03T08:02:52.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:02:52.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:02:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.987+0000 7f89adfff700 1 Processor -- start 2023-03-03T08:02:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.988+0000 7f89adfff700 1 -- start start 2023-03-03T08:02:52.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.989+0000 7f89adfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f89a80b87c0 0x7f89a80cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:52.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.989+0000 7f89adfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89a80b5410 0x7f89a80d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.989+0000 7f89adfff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a80d0ef0 con 0x7f89a80b8b90 2023-03-03T08:02:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.989+0000 7f89adfff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a80d0600 con 0x7f89a80b87c0 2023-03-03T08:02:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.989+0000 7f89adfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a80cd970 con 0x7f89a80b5410 2023-03-03T08:02:52.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.989+0000 7f89a77fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f89a80b87c0 0x7f89a80cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:52.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.990+0000 7f89a6ffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f89a80b8b90 0x7f89a80b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37284/0 (socket says 172.21.15.98:37284) 2023-03-03T08:02:52.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.990+0000 7f89a6ffd700 1 -- 172.21.15.98:0/1398840452 learned_addr learned my addr 172.21.15.98:0/1398840452 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:52.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.990+0000 7f89a7fff700 1 --2- 172.21.15.98:0/1398840452 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89a80b5410 0x7f89a80d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:52.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.990+0000 7f89a67fc700 1 -- 172.21.15.98:0/1398840452 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662096578 0 0) 0x7f89a80d0ef0 con 0x7f89a80b8b90 2023-03-03T08:02:52.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.990+0000 7f89a67fc700 1 -- 172.21.15.98:0/1398840452 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89900035b0 con 0x7f89a80b8b90 2023-03-03T08:02:52.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.990+0000 7f89a77fe700 1 -- 172.21.15.98:0/1398840452 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89a80b5410 msgr2=0x7f89a80d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:52.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.990+0000 7f89a77fe700 1 --2- 172.21.15.98:0/1398840452 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89a80b5410 0x7f89a80d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:52.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.990+0000 7f89a77fe700 1 -- 172.21.15.98:0/1398840452 >> v1:172.21.15.98:6789/0 conn(0x7f89a80b8b90 legacy=0x7f89a80b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:52.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.990+0000 7f89a77fe700 1 -- 172.21.15.98:0/1398840452 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a80cdc50 con 0x7f89a80b87c0 2023-03-03T08:02:52.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.990+0000 7f89a77fe700 1 --2- 172.21.15.98:0/1398840452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a80b87c0 0x7f89a80cffe0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f8998009840 tx=0x7f8998009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398cb6fda4519dc1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.991+0000 7f89a67fc700 1 -- 172.21.15.98:0/1398840452 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8998011830 con 0x7f89a80b87c0 2023-03-03T08:02:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.991+0000 7f89a67fc700 1 -- 172.21.15.98:0/1398840452 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8998011990 con 0x7f89a80b87c0 2023-03-03T08:02:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.991+0000 7f89a67fc700 1 -- 172.21.15.98:0/1398840452 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8998011d20 con 0x7f89a80b87c0 2023-03-03T08:02:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.991+0000 7f89adfff700 1 -- 172.21.15.98:0/1398840452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a80b87c0 msgr2=0x7f89a80cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.991+0000 7f89adfff700 1 --2- 172.21.15.98:0/1398840452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a80b87c0 0x7f89a80cffe0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f8998009840 tx=0x7f8998009b50 comp rx=0 tx=0).stop 2023-03-03T08:02:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.991+0000 7f89adfff700 1 -- 172.21.15.98:0/1398840452 shutdown_connections 2023-03-03T08:02:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.991+0000 7f89adfff700 1 --2- 172.21.15.98:0/1398840452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a80b87c0 0x7f89a80cffe0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:52.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.991+0000 7f89adfff700 1 --2- 172.21.15.98:0/1398840452 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89a80b5410 0x7f89a80d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:52.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.991+0000 7f89adfff700 1 -- 172.21.15.98:0/1398840452 >> 172.21.15.98:0/1398840452 conn(0x7f89a806aee0 msgr2=0x7f89a806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:52.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.991+0000 7f89adfff700 1 -- 172.21.15.98:0/1398840452 shutdown_connections 2023-03-03T08:02:52.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.992+0000 7f89adfff700 1 -- 172.21.15.98:0/1398840452 wait complete. 2023-03-03T08:02:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.994+0000 7f89adfff700 1 Processor -- start 2023-03-03T08:02:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.994+0000 7f89adfff700 1 -- start start 2023-03-03T08:02:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.995+0000 7f89adfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a80b5410 0x7f89a80ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.995+0000 7f89adfff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89a80b87c0 0x7f89a815a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.995+0000 7f89adfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89a80b8b90 0x7f89a8159330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.995+0000 7f89adfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a815aaa0 con 0x7f89a80b5410 2023-03-03T08:02:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.995+0000 7f89adfff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a815d070 con 0x7f89a80b8b90 2023-03-03T08:02:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.995+0000 7f89adfff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a815acd0 con 0x7f89a80b87c0 2023-03-03T08:02:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.995+0000 7f89a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a80b5410 0x7f89a80ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.995+0000 7f89a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a80b5410 0x7f89a80ca350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52978/0 (socket says 172.21.15.98:52978) 2023-03-03T08:02:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.995+0000 7f89a77fe700 1 -- 172.21.15.98:0/3601975599 learned_addr learned my addr 172.21.15.98:0/3601975599 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.996+0000 7f89a7fff700 1 --2- 172.21.15.98:0/3601975599 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89a80b8b90 0x7f89a8159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.996+0000 7f89a6ffd700 1 --2- 172.21.15.98:0/3601975599 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89a80b87c0 0x7f89a815a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.996+0000 7f89a77fe700 1 -- 172.21.15.98:0/3601975599 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89a80b87c0 msgr2=0x7f89a815a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.996+0000 7f89a77fe700 1 --2- 172.21.15.98:0/3601975599 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89a80b87c0 0x7f89a815a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.996+0000 7f89a77fe700 1 -- 172.21.15.98:0/3601975599 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89a80b8b90 msgr2=0x7f89a8159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.996+0000 7f89a77fe700 1 --2- 172.21.15.98:0/3601975599 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89a80b8b90 0x7f89a8159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.996+0000 7f89a77fe700 1 -- 172.21.15.98:0/3601975599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89980094a0 con 0x7f89a80b5410 2023-03-03T08:02:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.996+0000 7f89a77fe700 1 --2- 172.21.15.98:0/3601975599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a80b5410 0x7f89a80ca350 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f89980090b0 tx=0x7f8998004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.996+0000 7f898ffff700 1 -- 172.21.15.98:0/3601975599 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89980036a0 con 0x7f89a80b5410 2023-03-03T08:02:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.996+0000 7f898ffff700 1 -- 172.21.15.98:0/3601975599 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899801f070 con 0x7f89a80b5410 2023-03-03T08:02:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.996+0000 7f898ffff700 1 -- 172.21.15.98:0/3601975599 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8998003a40 con 0x7f89a80b5410 2023-03-03T08:02:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.996+0000 7f89adfff700 1 -- 172.21.15.98:0/3601975599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a80cdf30 con 0x7f89a80b5410 2023-03-03T08:02:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.997+0000 7f89adfff700 1 -- 172.21.15.98:0/3601975599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a8073a80 con 0x7f89a80b5410 2023-03-03T08:02:53.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.998+0000 7f89adfff700 1 -- 172.21.15.98:0/3601975599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a80ce0c0 con 0x7f89a80b5410 2023-03-03T08:02:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:52.999+0000 7f898ffff700 1 -- 172.21.15.98:0/3601975599 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f899800f050 con 0x7f89a80b5410 2023-03-03T08:02:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.001+0000 7f898ffff700 1 --2- 172.21.15.98:0/3601975599 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8994068380 0x7f899406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.001+0000 7f898ffff700 1 -- 172.21.15.98:0/3601975599 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8998021070 con 0x7f89a80b5410 2023-03-03T08:02:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.001+0000 7f89a6ffd700 1 --2- 172.21.15.98:0/3601975599 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8994068380 0x7f899406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.002+0000 7f89a6ffd700 1 --2- 172.21.15.98:0/3601975599 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8994068380 0x7f899406a830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f89a80b1660 tx=0x7f899c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:53.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.006+0000 7f898ffff700 1 -- 172.21.15.98:0/3601975599 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8998053db0 con 0x7f89a80b5410 2023-03-03T08:02:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.241+0000 7f89adfff700 1 -- 172.21.15.98:0/3601975599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89a80729b0 con 0x7f89a80b5410 2023-03-03T08:02:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.241+0000 7f898ffff700 1 -- 172.21.15.98:0/3601975599 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f8998085b10 con 0x7f89a80b5410 2023-03-03T08:02:53.243 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:53.243 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6602,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/463617820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":463617820},{"type":"v1","addr":"172.21.15.100:6835","nonce":463617820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2561725713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2561725713},{"type":"v1","addr":"172.21.15.98:6834","nonce":2561725713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1511797280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1511797280},{"type":"v1","addr":"172.21.15.100:6834","nonce":1511797280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:02:22.404483+0000","modified":"2023-03-03T08:02:25.872537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7524":{"gid":7524,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6835/341265634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":341265634},{"type":"v1","addr":"172.21.15.98:6835","nonce":341265634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-03-03T08:02:53.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.244+0000 7f89adfff700 1 -- 172.21.15.98:0/3601975599 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8994068380 msgr2=0x7f899406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:53.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.244+0000 7f89adfff700 1 --2- 172.21.15.98:0/3601975599 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8994068380 0x7f899406a830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f89a80b1660 tx=0x7f899c009040 comp rx=0 tx=0).stop 2023-03-03T08:02:53.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.244+0000 7f89adfff700 1 -- 172.21.15.98:0/3601975599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a80b5410 msgr2=0x7f89a80ca350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:53.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.244+0000 7f89adfff700 1 --2- 172.21.15.98:0/3601975599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a80b5410 0x7f89a80ca350 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f89980090b0 tx=0x7f8998004750 comp rx=0 tx=0).stop 2023-03-03T08:02:53.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.244+0000 7f89adfff700 1 -- 172.21.15.98:0/3601975599 shutdown_connections 2023-03-03T08:02:53.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.244+0000 7f89adfff700 1 --2- 172.21.15.98:0/3601975599 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8994068380 0x7f899406a830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:53.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.244+0000 7f89adfff700 1 --2- 172.21.15.98:0/3601975599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a80b5410 0x7f89a80ca350 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:53.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.244+0000 7f89adfff700 1 --2- 172.21.15.98:0/3601975599 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89a80b8b90 0x7f89a8159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:53.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.244+0000 7f89adfff700 1 --2- 172.21.15.98:0/3601975599 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89a80b87c0 0x7f89a815a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:53.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.244+0000 7f89adfff700 1 -- 172.21.15.98:0/3601975599 >> 172.21.15.98:0/3601975599 conn(0x7f89a806aee0 msgr2=0x7f89a80b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:53.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.244+0000 7f89adfff700 1 -- 172.21.15.98:0/3601975599 shutdown_connections 2023-03-03T08:02:53.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.244+0000 7f89adfff700 1 -- 172.21.15.98:0/3601975599 wait complete. 2023-03-03T08:02:53.248 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 54 2023-03-03T08:02:53.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:02:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.517+0000 7f07f43ff700 1 Processor -- start 2023-03-03T08:02:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.517+0000 7f07f43ff700 1 -- start start 2023-03-03T08:02:53.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.518+0000 7f07f43ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f07ec0b8b90 0x7f07ec0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:53.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.518+0000 7f07f43ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ec0d0ef0 con 0x7f07ec0b87c0 2023-03-03T08:02:53.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.518+0000 7f07f43ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ec0d0600 con 0x7f07ec0b5410 2023-03-03T08:02:53.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.518+0000 7f07f43ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07ec0cd970 con 0x7f07ec0b8b90 2023-03-03T08:02:53.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.519+0000 7f07f219b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f07ec0b87c0 0x7f07ec0b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37298/0 (socket says 172.21.15.98:37298) 2023-03-03T08:02:53.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.519+0000 7f07f219b700 1 -- 172.21.15.98:0/1714490829 learned_addr learned my addr 172.21.15.98:0/1714490829 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:53.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.519+0000 7f07f199a700 1 --2- 172.21.15.98:0/1714490829 >> v2:172.21.15.100:3300/0 conn(0x7f07ec0b8b90 0x7f07ec0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:53.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.519+0000 7f07f1199700 1 -- 172.21.15.98:0/1714490829 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860149159 0 0) 0x7f07ec0d0ef0 con 0x7f07ec0b87c0 2023-03-03T08:02:53.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.519+0000 7f07f1199700 1 -- 172.21.15.98:0/1714490829 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07d80035b0 con 0x7f07ec0b87c0 2023-03-03T08:02:53.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.519+0000 7f07f199a700 1 -- 172.21.15.98:0/1714490829 >> v1:172.21.15.100:6789/0 conn(0x7f07ec0b5410 legacy=0x7f07ec0d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:53.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.519+0000 7f07f199a700 1 -- 172.21.15.98:0/1714490829 >> v1:172.21.15.98:6789/0 conn(0x7f07ec0b87c0 legacy=0x7f07ec0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:53.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.519+0000 7f07f199a700 1 -- 172.21.15.98:0/1714490829 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ec0cdc50 con 0x7f07ec0b8b90 2023-03-03T08:02:53.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.520+0000 7f07f199a700 1 --2- 172.21.15.98:0/1714490829 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07ec0b8b90 0x7f07ec0cffe0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f07dc009840 tx=0x7f07dc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b372c9a79c64151 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:53.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.520+0000 7f07f1199700 1 -- 172.21.15.98:0/1714490829 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc011830 con 0x7f07ec0b8b90 2023-03-03T08:02:53.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.520+0000 7f07f1199700 1 -- 172.21.15.98:0/1714490829 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07dc011990 con 0x7f07ec0b8b90 2023-03-03T08:02:53.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.520+0000 7f07f1199700 1 -- 172.21.15.98:0/1714490829 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc011d20 con 0x7f07ec0b8b90 2023-03-03T08:02:53.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.521+0000 7f07f43ff700 1 -- 172.21.15.98:0/1714490829 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07ec0b8b90 msgr2=0x7f07ec0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:53.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.521+0000 7f07f43ff700 1 --2- 172.21.15.98:0/1714490829 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07ec0b8b90 0x7f07ec0cffe0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f07dc009840 tx=0x7f07dc009b50 comp rx=0 tx=0).stop 2023-03-03T08:02:53.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.521+0000 7f07f43ff700 1 -- 172.21.15.98:0/1714490829 shutdown_connections 2023-03-03T08:02:53.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.521+0000 7f07f43ff700 1 --2- 172.21.15.98:0/1714490829 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07ec0b8b90 0x7f07ec0cffe0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:53.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.521+0000 7f07f43ff700 1 -- 172.21.15.98:0/1714490829 >> 172.21.15.98:0/1714490829 conn(0x7f07ec06aee0 msgr2=0x7f07ec06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:53.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.521+0000 7f07f43ff700 1 -- 172.21.15.98:0/1714490829 shutdown_connections 2023-03-03T08:02:53.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.521+0000 7f07f43ff700 1 -- 172.21.15.98:0/1714490829 wait complete. 2023-03-03T08:02:53.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.523+0000 7f07f43ff700 1 Processor -- start 2023-03-03T08:02:53.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.523+0000 7f07f43ff700 1 -- start start 2023-03-03T08:02:53.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.524+0000 7f07f43ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07ec0b5410 0x7f07ec15af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:53.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.524+0000 7f07f43ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07ec0b87c0 0x7f07ec0ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:53.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.524+0000 7f07f43ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07ec0b8b90 0x7f07ec1e6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:53.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.524+0000 7f07f43ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec0cba90 con 0x7f07ec0b5410 2023-03-03T08:02:53.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.524+0000 7f07f43ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec15b4b0 con 0x7f07ec0b87c0 2023-03-03T08:02:53.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.524+0000 7f07f43ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec0cae10 con 0x7f07ec0b8b90 2023-03-03T08:02:53.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.525+0000 7f07f219b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07ec0b5410 0x7f07ec15af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:53.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.525+0000 7f07f219b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07ec0b5410 0x7f07ec15af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52980/0 (socket says 172.21.15.98:52980) 2023-03-03T08:02:53.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.525+0000 7f07f219b700 1 -- 172.21.15.98:0/1580041115 learned_addr learned my addr 172.21.15.98:0/1580041115 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:53.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.525+0000 7f07f199a700 1 --2- 172.21.15.98:0/1580041115 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07ec0b87c0 0x7f07ec0ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:53.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.525+0000 7f07f299c700 1 --2- 172.21.15.98:0/1580041115 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07ec0b8b90 0x7f07ec1e6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:53.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.525+0000 7f07f219b700 1 -- 172.21.15.98:0/1580041115 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07ec0b8b90 msgr2=0x7f07ec1e6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:53.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.525+0000 7f07f219b700 1 --2- 172.21.15.98:0/1580041115 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07ec0b8b90 0x7f07ec1e6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:53.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.525+0000 7f07f219b700 1 -- 172.21.15.98:0/1580041115 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07ec0b87c0 msgr2=0x7f07ec0ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:53.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.525+0000 7f07f219b700 1 --2- 172.21.15.98:0/1580041115 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07ec0b87c0 0x7f07ec0ca350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:53.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.525+0000 7f07f219b700 1 -- 172.21.15.98:0/1580041115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07dc0094a0 con 0x7f07ec0b5410 2023-03-03T08:02:53.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.525+0000 7f07f219b700 1 --2- 172.21.15.98:0/1580041115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07ec0b5410 0x7f07ec15af70 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f07e800c700 tx=0x7f07e800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:53.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.526+0000 7f07e2ffd700 1 -- 172.21.15.98:0/1580041115 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8005d70 con 0x7f07ec0b5410 2023-03-03T08:02:53.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.526+0000 7f07e2ffd700 1 -- 172.21.15.98:0/1580041115 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e80070d0 con 0x7f07ec0b5410 2023-03-03T08:02:53.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.526+0000 7f07f43ff700 1 -- 172.21.15.98:0/1580041115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07ec0cdf30 con 0x7f07ec0b5410 2023-03-03T08:02:53.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.526+0000 7f07e2ffd700 1 -- 172.21.15.98:0/1580041115 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8007460 con 0x7f07ec0b5410 2023-03-03T08:02:53.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.526+0000 7f07f43ff700 1 -- 172.21.15.98:0/1580041115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ec073a80 con 0x7f07ec0b5410 2023-03-03T08:02:53.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.527+0000 7f07f43ff700 1 -- 172.21.15.98:0/1580041115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07ec0ce0c0 con 0x7f07ec0b5410 2023-03-03T08:02:53.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.528+0000 7f07e2ffd700 1 -- 172.21.15.98:0/1580041115 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f07e8019030 con 0x7f07ec0b5410 2023-03-03T08:02:53.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.530+0000 7f07e2ffd700 1 --2- 172.21.15.98:0/1580041115 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f07d0068380 0x7f07d006a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:53.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.531+0000 7f07e2ffd700 1 -- 172.21.15.98:0/1580041115 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f07e8085210 con 0x7f07ec0b5410 2023-03-03T08:02:53.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.531+0000 7f07f199a700 1 --2- 172.21.15.98:0/1580041115 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f07d0068380 0x7f07d006a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:53.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.532+0000 7f07f199a700 1 --2- 172.21.15.98:0/1580041115 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f07d0068380 0x7f07d006a830 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f07ec0b8fd0 tx=0x7f07dc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:53.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.534+0000 7f07e2ffd700 1 -- 172.21.15.98:0/1580041115 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f07e8051930 con 0x7f07ec0b5410 2023-03-03T08:02:53.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.728+0000 7f07f43ff700 1 -- 172.21.15.98:0/1580041115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07ec0729b0 con 0x7f07ec0b5410 2023-03-03T08:02:53.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.729+0000 7f07e2ffd700 1 -- 172.21.15.98:0/1580041115 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+18053 (secure 0 0 0) 0x7f07e8083690 con 0x7f07ec0b5410 2023-03-03T08:02:53.731 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:53.731 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":98,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:02:45.624487+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:02:20.797579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-03-03T08:02:21.411899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:6835/1843752855":"2023-03-04T08:02:42.175049+0000","172.21.15.100:6833/1843752855":"2023-03-04T08:02:42.175049+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:02:53.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.732+0000 7f07f43ff700 1 -- 172.21.15.98:0/1580041115 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f07d0068380 msgr2=0x7f07d006a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:53.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.732+0000 7f07f43ff700 1 --2- 172.21.15.98:0/1580041115 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f07d0068380 0x7f07d006a830 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f07ec0b8fd0 tx=0x7f07dc008030 comp rx=0 tx=0).stop 2023-03-03T08:02:53.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.732+0000 7f07f43ff700 1 -- 172.21.15.98:0/1580041115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07ec0b5410 msgr2=0x7f07ec15af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:53.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.732+0000 7f07f43ff700 1 --2- 172.21.15.98:0/1580041115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07ec0b5410 0x7f07ec15af70 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f07e800c700 tx=0x7f07e800ca10 comp rx=0 tx=0).stop 2023-03-03T08:02:53.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.732+0000 7f07f43ff700 1 -- 172.21.15.98:0/1580041115 shutdown_connections 2023-03-03T08:02:53.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.732+0000 7f07f43ff700 1 --2- 172.21.15.98:0/1580041115 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f07d0068380 0x7f07d006a830 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:53.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.732+0000 7f07f43ff700 1 --2- 172.21.15.98:0/1580041115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07ec0b5410 0x7f07ec15af70 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:53.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.732+0000 7f07f43ff700 1 --2- 172.21.15.98:0/1580041115 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07ec0b8b90 0x7f07ec1e6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:53.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.732+0000 7f07f43ff700 1 --2- 172.21.15.98:0/1580041115 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07ec0b87c0 0x7f07ec0ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:53.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.732+0000 7f07f43ff700 1 -- 172.21.15.98:0/1580041115 >> 172.21.15.98:0/1580041115 conn(0x7f07ec06aee0 msgr2=0x7f07ec0b7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:53.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.732+0000 7f07f43ff700 1 -- 172.21.15.98:0/1580041115 shutdown_connections 2023-03-03T08:02:53.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:53.732+0000 7f07f43ff700 1 -- 172.21.15.98:0/1580041115 wait complete. 2023-03-03T08:02:53.760 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T08:02:20.797579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T08:02:21.411899+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:02:53.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:02:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.009+0000 7f5e2119d700 1 Processor -- start 2023-03-03T08:02:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.010+0000 7f5e2119d700 1 -- start start 2023-03-03T08:02:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.011+0000 7f5e2119d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5e1c0cf570 0x7f5e1c15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.011+0000 7f5e2119d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5e1c15db30 0x7f5e1c0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.011+0000 7f5e2119d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1c0cd780 con 0x7f5e1c1609b0 2023-03-03T08:02:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.011+0000 7f5e2119d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e1c0cebb0 con 0x7f5e1c0cf570 2023-03-03T08:02:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.011+0000 7f5e2119d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e1c0cda60 con 0x7f5e1c15db30 2023-03-03T08:02:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.011+0000 7f5e1bfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5e1c1609b0 0x7f5e1c160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37308/0 (socket says 172.21.15.98:37308) 2023-03-03T08:02:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.011+0000 7f5e1bfff700 1 -- 172.21.15.98:0/127174840 learned_addr learned my addr 172.21.15.98:0/127174840 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.011+0000 7f5e1b7fe700 1 --2- 172.21.15.98:0/127174840 >> v2:172.21.15.98:3300/0 conn(0x7f5e1c0cf570 0x7f5e1c15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.012+0000 7f5e2099c700 1 --2- 172.21.15.98:0/127174840 >> v2:172.21.15.100:3300/0 conn(0x7f5e1c15db30 0x7f5e1c0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.012+0000 7f5e1affd700 1 -- 172.21.15.98:0/127174840 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441442714 0 0) 0x7f5e1c0cd780 con 0x7f5e1c1609b0 2023-03-03T08:02:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.012+0000 7f5e1affd700 1 -- 172.21.15.98:0/127174840 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e040035b0 con 0x7f5e1c1609b0 2023-03-03T08:02:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.012+0000 7f5e1b7fe700 1 -- 172.21.15.98:0/127174840 >> v2:172.21.15.100:3300/0 conn(0x7f5e1c15db30 msgr2=0x7f5e1c0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.012+0000 7f5e1b7fe700 1 --2- 172.21.15.98:0/127174840 >> v2:172.21.15.100:3300/0 conn(0x7f5e1c15db30 0x7f5e1c0b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.012+0000 7f5e1b7fe700 1 -- 172.21.15.98:0/127174840 >> v1:172.21.15.98:6789/0 conn(0x7f5e1c1609b0 legacy=0x7f5e1c160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.012+0000 7f5e1b7fe700 1 -- 172.21.15.98:0/127174840 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c0cdd40 con 0x7f5e1c0cf570 2023-03-03T08:02:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.012+0000 7f5e1b7fe700 1 --2- 172.21.15.98:0/127174840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e1c0cf570 0x7f5e1c15d5f0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0ca080 tx=0x7f5e10009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227620a368d97ec3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.013+0000 7f5e1affd700 1 -- 172.21.15.98:0/127174840 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e10011830 con 0x7f5e1c0cf570 2023-03-03T08:02:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.013+0000 7f5e1affd700 1 -- 172.21.15.98:0/127174840 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e10011990 con 0x7f5e1c0cf570 2023-03-03T08:02:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.013+0000 7f5e1affd700 1 -- 172.21.15.98:0/127174840 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e10011d20 con 0x7f5e1c0cf570 2023-03-03T08:02:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.013+0000 7f5e2119d700 1 -- 172.21.15.98:0/127174840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e1c0cf570 msgr2=0x7f5e1c15d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.013+0000 7f5e2119d700 1 --2- 172.21.15.98:0/127174840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e1c0cf570 0x7f5e1c15d5f0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0ca080 tx=0x7f5e10009b20 comp rx=0 tx=0).stop 2023-03-03T08:02:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.013+0000 7f5e2119d700 1 -- 172.21.15.98:0/127174840 shutdown_connections 2023-03-03T08:02:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.013+0000 7f5e2119d700 1 --2- 172.21.15.98:0/127174840 >> v2:172.21.15.100:3300/0 conn(0x7f5e1c15db30 0x7f5e1c0b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.013+0000 7f5e2119d700 1 --2- 172.21.15.98:0/127174840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e1c0cf570 0x7f5e1c15d5f0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.013+0000 7f5e2119d700 1 -- 172.21.15.98:0/127174840 >> 172.21.15.98:0/127174840 conn(0x7f5e1c06b880 msgr2=0x7f5e1c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.013+0000 7f5e2119d700 1 -- 172.21.15.98:0/127174840 shutdown_connections 2023-03-03T08:02:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.013+0000 7f5e2119d700 1 -- 172.21.15.98:0/127174840 wait complete. 2023-03-03T08:02:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.015+0000 7f5e2119d700 1 Processor -- start 2023-03-03T08:02:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.016+0000 7f5e2119d700 1 -- start start 2023-03-03T08:02:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e2119d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e1c0cf570 0x7f5e1c1508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e2119d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e1c15db30 0x7f5e1c151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e2119d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e1c1609b0 0x7f5e1c14c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e2119d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c151fd0 con 0x7f5e1c1609b0 2023-03-03T08:02:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e2119d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c14ccb0 con 0x7f5e1c0cf570 2023-03-03T08:02:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e2119d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c0c73e0 con 0x7f5e1c15db30 2023-03-03T08:02:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e2099c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e1c1609b0 0x7f5e1c14c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e2099c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e1c1609b0 0x7f5e1c14c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53004/0 (socket says 172.21.15.98:53004) 2023-03-03T08:02:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e2099c700 1 -- 172.21.15.98:0/2952419123 learned_addr learned my addr 172.21.15.98:0/2952419123 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e1bfff700 1 --2- 172.21.15.98:0/2952419123 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e1c0cf570 0x7f5e1c1508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e1b7fe700 1 --2- 172.21.15.98:0/2952419123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e1c15db30 0x7f5e1c151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e2099c700 1 -- 172.21.15.98:0/2952419123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e1c15db30 msgr2=0x7f5e1c151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e2099c700 1 --2- 172.21.15.98:0/2952419123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e1c15db30 0x7f5e1c151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e2099c700 1 -- 172.21.15.98:0/2952419123 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e1c0cf570 msgr2=0x7f5e1c1508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e2099c700 1 --2- 172.21.15.98:0/2952419123 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e1c0cf570 0x7f5e1c1508d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.017+0000 7f5e2099c700 1 -- 172.21.15.98:0/2952419123 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e100094a0 con 0x7f5e1c1609b0 2023-03-03T08:02:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.018+0000 7f5e2099c700 1 --2- 172.21.15.98:0/2952419123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e1c1609b0 0x7f5e1c14c770 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f5e1400ef90 tx=0x7f5e1400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.018+0000 7f5e18ff9700 1 -- 172.21.15.98:0/2952419123 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1400cc90 con 0x7f5e1c1609b0 2023-03-03T08:02:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.018+0000 7f5e18ff9700 1 -- 172.21.15.98:0/2952419123 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1400f040 con 0x7f5e1c1609b0 2023-03-03T08:02:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.018+0000 7f5e18ff9700 1 -- 172.21.15.98:0/2952419123 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e14014480 con 0x7f5e1c1609b0 2023-03-03T08:02:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.018+0000 7f5e2119d700 1 -- 172.21.15.98:0/2952419123 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1c15d0d0 con 0x7f5e1c1609b0 2023-03-03T08:02:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.018+0000 7f5e2119d700 1 -- 172.21.15.98:0/2952419123 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1c15cdf0 con 0x7f5e1c1609b0 2023-03-03T08:02:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.019+0000 7f5e2119d700 1 -- 172.21.15.98:0/2952419123 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e1c0b4fe0 con 0x7f5e1c1609b0 2023-03-03T08:02:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.021+0000 7f5e18ff9700 1 -- 172.21.15.98:0/2952419123 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f5e140145e0 con 0x7f5e1c1609b0 2023-03-03T08:02:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.023+0000 7f5e18ff9700 1 --2- 172.21.15.98:0/2952419123 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5e080683b0 0x7f5e0806a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.023+0000 7f5e18ff9700 1 -- 172.21.15.98:0/2952419123 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5e14086a80 con 0x7f5e1c1609b0 2023-03-03T08:02:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.023+0000 7f5e1bfff700 1 --2- 172.21.15.98:0/2952419123 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5e080683b0 0x7f5e0806a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.024+0000 7f5e1bfff700 1 --2- 172.21.15.98:0/2952419123 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5e080683b0 0x7f5e0806a860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c007fd0 tx=0x7f5e0c007dc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.027+0000 7f5e18ff9700 1 -- 172.21.15.98:0/2952419123 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5e140531a0 con 0x7f5e1c1609b0 2023-03-03T08:02:54.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.219+0000 7f5e2119d700 1 -- 172.21.15.98:0/2952419123 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5e1c0c7550 con 0x7f5e1c1609b0 2023-03-03T08:02:54.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.219+0000 7f5e18ff9700 1 -- 172.21.15.98:0/2952419123 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5e140180a0 con 0x7f5e1c1609b0 2023-03-03T08:02:54.221 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:02:54.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.222+0000 7f5e2119d700 1 -- 172.21.15.98:0/2952419123 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5e080683b0 msgr2=0x7f5e0806a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:54.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.222+0000 7f5e2119d700 1 --2- 172.21.15.98:0/2952419123 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5e080683b0 0x7f5e0806a860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c007fd0 tx=0x7f5e0c007dc0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.222+0000 7f5e2119d700 1 -- 172.21.15.98:0/2952419123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e1c1609b0 msgr2=0x7f5e1c14c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:54.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.222+0000 7f5e2119d700 1 --2- 172.21.15.98:0/2952419123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e1c1609b0 0x7f5e1c14c770 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f5e1400ef90 tx=0x7f5e1400c3a0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.222+0000 7f5e2119d700 1 -- 172.21.15.98:0/2952419123 shutdown_connections 2023-03-03T08:02:54.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.222+0000 7f5e2119d700 1 --2- 172.21.15.98:0/2952419123 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5e080683b0 0x7f5e0806a860 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.222+0000 7f5e2119d700 1 --2- 172.21.15.98:0/2952419123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e1c1609b0 0x7f5e1c14c770 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.222+0000 7f5e2119d700 1 --2- 172.21.15.98:0/2952419123 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e1c0cf570 0x7f5e1c1508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.222+0000 7f5e2119d700 1 --2- 172.21.15.98:0/2952419123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e1c15db30 0x7f5e1c151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.222+0000 7f5e2119d700 1 -- 172.21.15.98:0/2952419123 >> 172.21.15.98:0/2952419123 conn(0x7f5e1c06b880 msgr2=0x7f5e1c0d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:54.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.222+0000 7f5e2119d700 1 -- 172.21.15.98:0/2952419123 shutdown_connections 2023-03-03T08:02:54.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.222+0000 7f5e2119d700 1 -- 172.21.15.98:0/2952419123 wait complete. 2023-03-03T08:02:54.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:02:54.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.500+0000 7feb2d3ff700 1 Processor -- start 2023-03-03T08:02:54.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.501+0000 7feb2d3ff700 1 -- start start 2023-03-03T08:02:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.502+0000 7feb2d3ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb2815ca70 0x7feb280cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.502+0000 7feb2d3ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7feb280d0270 0x7feb280d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.502+0000 7feb2d3ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb280cf980 con 0x7feb2806b4b0 2023-03-03T08:02:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.502+0000 7feb2d3ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb280cd5b0 con 0x7feb2815ca70 2023-03-03T08:02:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.502+0000 7feb2d3ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb280d1800 con 0x7feb280d0270 2023-03-03T08:02:54.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.502+0000 7feb26ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb2815ca70 0x7feb280cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:54.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.502+0000 7feb26ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb2815ca70 0x7feb280cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53006/0 (socket says 172.21.15.98:53006) 2023-03-03T08:02:54.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.502+0000 7feb277fe700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7feb280d0270 0x7feb280d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:54.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.502+0000 7feb26ffd700 1 -- 172.21.15.98:0/2443940411 learned_addr learned my addr 172.21.15.98:0/2443940411 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:54.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.503+0000 7feb26ffd700 1 -- 172.21.15.98:0/2443940411 >> v2:172.21.15.100:3300/0 conn(0x7feb280d0270 msgr2=0x7feb280d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:54.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.503+0000 7feb26ffd700 1 --2- 172.21.15.98:0/2443940411 >> v2:172.21.15.100:3300/0 conn(0x7feb280d0270 0x7feb280d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.503+0000 7feb26ffd700 1 -- 172.21.15.98:0/2443940411 >> v1:172.21.15.100:6789/0 conn(0x7feb2806b4b0 legacy=0x7feb280cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:54.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.503+0000 7feb26ffd700 1 -- 172.21.15.98:0/2443940411 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb280cd900 con 0x7feb2815ca70 2023-03-03T08:02:54.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.503+0000 7feb26ffd700 1 --2- 172.21.15.98:0/2443940411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb2815ca70 0x7feb280cf440 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7feb280ca200 tx=0x7feb18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03467e64053be10 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:54.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.503+0000 7feb25ffb700 1 -- 172.21.15.98:0/2443940411 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18011830 con 0x7feb2815ca70 2023-03-03T08:02:54.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.504+0000 7feb25ffb700 1 -- 172.21.15.98:0/2443940411 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb18011990 con 0x7feb2815ca70 2023-03-03T08:02:54.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.504+0000 7feb25ffb700 1 -- 172.21.15.98:0/2443940411 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18011d20 con 0x7feb2815ca70 2023-03-03T08:02:54.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.504+0000 7feb2d3ff700 1 -- 172.21.15.98:0/2443940411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb2815ca70 msgr2=0x7feb280cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:54.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.504+0000 7feb2d3ff700 1 --2- 172.21.15.98:0/2443940411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb2815ca70 0x7feb280cf440 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7feb280ca200 tx=0x7feb18009bd0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.504+0000 7feb2d3ff700 1 -- 172.21.15.98:0/2443940411 shutdown_connections 2023-03-03T08:02:54.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.504+0000 7feb2d3ff700 1 --2- 172.21.15.98:0/2443940411 >> v2:172.21.15.100:3300/0 conn(0x7feb280d0270 0x7feb280d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.504+0000 7feb2d3ff700 1 --2- 172.21.15.98:0/2443940411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb2815ca70 0x7feb280cf440 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.504+0000 7feb2d3ff700 1 -- 172.21.15.98:0/2443940411 >> 172.21.15.98:0/2443940411 conn(0x7feb28073e70 msgr2=0x7feb2806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:54.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.504+0000 7feb2d3ff700 1 -- 172.21.15.98:0/2443940411 shutdown_connections 2023-03-03T08:02:54.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.504+0000 7feb2d3ff700 1 -- 172.21.15.98:0/2443940411 wait complete. 2023-03-03T08:02:54.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.506+0000 7feb2d3ff700 1 Processor -- start 2023-03-03T08:02:54.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.506+0000 7feb2d3ff700 1 -- start start 2023-03-03T08:02:54.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.507+0000 7feb2d3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb2806b4b0 0x7feb280cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:54.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.507+0000 7feb2d3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb280d0270 0x7feb28156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:54.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.507+0000 7feb2d3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb2815ca70 0x7feb2815f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:54.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.507+0000 7feb2d3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb281f3300 con 0x7feb2815ca70 2023-03-03T08:02:54.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.507+0000 7feb2d3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb281f23f0 con 0x7feb280d0270 2023-03-03T08:02:54.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.507+0000 7feb2d3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb281f2680 con 0x7feb2806b4b0 2023-03-03T08:02:54.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.508+0000 7feb277fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb2815ca70 0x7feb2815f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:54.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.508+0000 7feb277fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb2815ca70 0x7feb2815f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53016/0 (socket says 172.21.15.98:53016) 2023-03-03T08:02:54.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.508+0000 7feb277fe700 1 -- 172.21.15.98:0/789110871 learned_addr learned my addr 172.21.15.98:0/789110871 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:54.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.508+0000 7feb26ffd700 1 --2- 172.21.15.98:0/789110871 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb2806b4b0 0x7feb280cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:54.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.508+0000 7feb267fc700 1 --2- 172.21.15.98:0/789110871 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb280d0270 0x7feb28156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:54.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.508+0000 7feb277fe700 1 -- 172.21.15.98:0/789110871 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb2806b4b0 msgr2=0x7feb280cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:54.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.508+0000 7feb277fe700 1 --2- 172.21.15.98:0/789110871 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb2806b4b0 0x7feb280cb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.508+0000 7feb277fe700 1 -- 172.21.15.98:0/789110871 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb280d0270 msgr2=0x7feb28156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:54.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.508+0000 7feb277fe700 1 --2- 172.21.15.98:0/789110871 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb280d0270 0x7feb28156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.508+0000 7feb277fe700 1 -- 172.21.15.98:0/789110871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb18009550 con 0x7feb2815ca70 2023-03-03T08:02:54.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.509+0000 7feb277fe700 1 --2- 172.21.15.98:0/789110871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb2815ca70 0x7feb2815f820 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7feb2000ef90 tx=0x7feb20009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:54.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.509+0000 7feb0f7fe700 1 -- 172.21.15.98:0/789110871 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20009b20 con 0x7feb2815ca70 2023-03-03T08:02:54.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.509+0000 7feb0f7fe700 1 -- 172.21.15.98:0/789110871 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2000f040 con 0x7feb2815ca70 2023-03-03T08:02:54.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.509+0000 7feb0f7fe700 1 -- 172.21.15.98:0/789110871 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2000cc70 con 0x7feb2815ca70 2023-03-03T08:02:54.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.509+0000 7feb2d3ff700 1 -- 172.21.15.98:0/789110871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb280d1aa0 con 0x7feb2815ca70 2023-03-03T08:02:54.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.509+0000 7feb2d3ff700 1 -- 172.21.15.98:0/789110871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb280cdbe0 con 0x7feb2815ca70 2023-03-03T08:02:54.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.510+0000 7feb2d3ff700 1 -- 172.21.15.98:0/789110871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2815ce40 con 0x7feb2815ca70 2023-03-03T08:02:54.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.511+0000 7feb0f7fe700 1 -- 172.21.15.98:0/789110871 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7feb20014480 con 0x7feb2815ca70 2023-03-03T08:02:54.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.513+0000 7feb0f7fe700 1 --2- 172.21.15.98:0/789110871 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7feb14068380 0x7feb1406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:54.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.513+0000 7feb0f7fe700 1 -- 172.21.15.98:0/789110871 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7feb20086930 con 0x7feb2815ca70 2023-03-03T08:02:54.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.513+0000 7feb26ffd700 1 --2- 172.21.15.98:0/789110871 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7feb14068380 0x7feb1406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:54.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.514+0000 7feb26ffd700 1 --2- 172.21.15.98:0/789110871 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7feb14068380 0x7feb1406a830 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7feb18009c10 tx=0x7feb18008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:54.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.516+0000 7feb0f7fe700 1 -- 172.21.15.98:0/789110871 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feb20052ff0 con 0x7feb2815ca70 2023-03-03T08:02:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.709+0000 7feb2d3ff700 1 -- 172.21.15.98:0/789110871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7feb2804e6e0 con 0x7feb2815ca70 2023-03-03T08:02:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.709+0000 7feb0f7fe700 1 -- 172.21.15.98:0/789110871 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v98) v1 ==== 92+0+10 (secure 0 0 0) 0x7feb200180a0 con 0x7feb2815ca70 2023-03-03T08:02:54.712 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:02:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.712+0000 7feb2d3ff700 1 -- 172.21.15.98:0/789110871 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7feb14068380 msgr2=0x7feb1406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.712+0000 7feb2d3ff700 1 --2- 172.21.15.98:0/789110871 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7feb14068380 0x7feb1406a830 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7feb18009c10 tx=0x7feb18008030 comp rx=0 tx=0).stop 2023-03-03T08:02:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.712+0000 7feb2d3ff700 1 -- 172.21.15.98:0/789110871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb2815ca70 msgr2=0x7feb2815f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:54.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.712+0000 7feb2d3ff700 1 --2- 172.21.15.98:0/789110871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb2815ca70 0x7feb2815f820 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7feb2000ef90 tx=0x7feb20009f60 comp rx=0 tx=0).stop 2023-03-03T08:02:54.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.712+0000 7feb2d3ff700 1 -- 172.21.15.98:0/789110871 shutdown_connections 2023-03-03T08:02:54.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.712+0000 7feb2d3ff700 1 --2- 172.21.15.98:0/789110871 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7feb14068380 0x7feb1406a830 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.712+0000 7feb2d3ff700 1 --2- 172.21.15.98:0/789110871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb2815ca70 0x7feb2815f820 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.712+0000 7feb2d3ff700 1 --2- 172.21.15.98:0/789110871 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb2806b4b0 0x7feb280cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.712+0000 7feb2d3ff700 1 --2- 172.21.15.98:0/789110871 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb280d0270 0x7feb28156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.712+0000 7feb2d3ff700 1 -- 172.21.15.98:0/789110871 >> 172.21.15.98:0/789110871 conn(0x7feb28073e70 msgr2=0x7feb2815ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:54.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.712+0000 7feb2d3ff700 1 -- 172.21.15.98:0/789110871 shutdown_connections 2023-03-03T08:02:54.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.712+0000 7feb2d3ff700 1 -- 172.21.15.98:0/789110871 wait complete. 2023-03-03T08:02:54.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T08:02:54.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.988+0000 7f07809ff700 1 Processor -- start 2023-03-03T08:02:54.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.988+0000 7f07809ff700 1 -- start start 2023-03-03T08:02:54.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.989+0000 7f07809ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f077c0b8d10 0x7f077c0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:54.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.989+0000 7f07809ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f077c0b26d0 0x7f077c0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:54.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.989+0000 7f07809ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f077c0b5140 0x7f077c160520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:54.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.989+0000 7f07809ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f077c0b55c0 con 0x7f077c0b5140 2023-03-03T08:02:54.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.989+0000 7f07809ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f077c0cd7d0 con 0x7f077c0b26d0 2023-03-03T08:02:54.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.989+0000 7f07809ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f077c0b90e0 con 0x7f077c0b8d10 2023-03-03T08:02:54.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.990+0000 7f077ad9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f077c0b5140 0x7f077c160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:54.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.990+0000 7f077ad9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f077c0b5140 0x7f077c160520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34172/0 (socket says 172.21.15.98:34172) 2023-03-03T08:02:54.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.990+0000 7f077ad9d700 1 -- 172.21.15.98:0/1322367901 learned_addr learned my addr 172.21.15.98:0/1322367901 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:54.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.990+0000 7f077a59c700 1 --2- 172.21.15.98:0/1322367901 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f077c0b8d10 0x7f077c0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.990+0000 7f0779d9b700 1 --2- 172.21.15.98:0/1322367901 >> v2:172.21.15.100:3300/0 conn(0x7f077c0b26d0 0x7f077c0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.990+0000 7f077ad9d700 1 -- 172.21.15.98:0/1322367901 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f077c0b8d10 msgr2=0x7f077c0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.990+0000 7f077ad9d700 1 --2- 172.21.15.98:0/1322367901 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f077c0b8d10 0x7f077c0cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.990+0000 7f077ad9d700 1 -- 172.21.15.98:0/1322367901 >> v2:172.21.15.100:3300/0 conn(0x7f077c0b26d0 msgr2=0x7f077c0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.990+0000 7f077ad9d700 1 --2- 172.21.15.98:0/1322367901 >> v2:172.21.15.100:3300/0 conn(0x7f077c0b26d0 0x7f077c0b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.990+0000 7f077ad9d700 1 -- 172.21.15.98:0/1322367901 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077c0cda20 con 0x7f077c0b5140 2023-03-03T08:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.990+0000 7f0779d9b700 1 --2- 172.21.15.98:0/1322367901 >> v2:172.21.15.100:3300/0 conn(0x7f077c0b26d0 0x7f077c0b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.991+0000 7f077ad9d700 1 --2- 172.21.15.98:0/1322367901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f077c0b5140 0x7f077c160520 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f077c0d4210 tx=0x7f077400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83dcd7061fe08c89 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.991+0000 7f077959a700 1 -- 172.21.15.98:0/1322367901 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077400e070 con 0x7f077c0b5140 2023-03-03T08:02:54.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.991+0000 7f077959a700 1 -- 172.21.15.98:0/1322367901 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077400b9c0 con 0x7f077c0b5140 2023-03-03T08:02:54.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.991+0000 7f077959a700 1 -- 172.21.15.98:0/1322367901 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077400bd70 con 0x7f077c0b5140 2023-03-03T08:02:54.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.991+0000 7f07809ff700 1 -- 172.21.15.98:0/1322367901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f077c0b5140 msgr2=0x7f077c160520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:54.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.991+0000 7f07809ff700 1 --2- 172.21.15.98:0/1322367901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f077c0b5140 0x7f077c160520 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f077c0d4210 tx=0x7f077400b330 comp rx=0 tx=0).stop 2023-03-03T08:02:54.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.991+0000 7f07809ff700 1 -- 172.21.15.98:0/1322367901 shutdown_connections 2023-03-03T08:02:54.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.991+0000 7f07809ff700 1 --2- 172.21.15.98:0/1322367901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f077c0b5140 0x7f077c160520 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.992+0000 7f07809ff700 1 --2- 172.21.15.98:0/1322367901 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f077c0b8d10 0x7f077c0cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.992+0000 7f07809ff700 1 --2- 172.21.15.98:0/1322367901 >> v2:172.21.15.100:3300/0 conn(0x7f077c0b26d0 0x7f077c0b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:54.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.992+0000 7f07809ff700 1 -- 172.21.15.98:0/1322367901 >> 172.21.15.98:0/1322367901 conn(0x7f077c073e70 msgr2=0x7f077c06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:54.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.992+0000 7f07809ff700 1 -- 172.21.15.98:0/1322367901 shutdown_connections 2023-03-03T08:02:54.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.992+0000 7f07809ff700 1 -- 172.21.15.98:0/1322367901 wait complete. 2023-03-03T08:02:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.994+0000 7f07809ff700 1 Processor -- start 2023-03-03T08:02:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.994+0000 7f07809ff700 1 -- start start 2023-03-03T08:02:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.995+0000 7f07809ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f077c0b26d0 0x7f077c0ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.995+0000 7f07809ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f077c0b5140 0x7f077c0d1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.995+0000 7f07809ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f077c0b8d10 0x7f077c14d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.995+0000 7f07809ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077c1f5a10 con 0x7f077c0b26d0 2023-03-03T08:02:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.995+0000 7f07809ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077c1f5cf0 con 0x7f077c0b8d10 2023-03-03T08:02:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.995+0000 7f07809ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f077c1f5fd0 con 0x7f077c0b5140 2023-03-03T08:02:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.995+0000 7f077a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f077c0b26d0 0x7f077c0ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.995+0000 7f077a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f077c0b26d0 0x7f077c0ca250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34178/0 (socket says 172.21.15.98:34178) 2023-03-03T08:02:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.995+0000 7f077a59c700 1 -- 172.21.15.98:0/3497518225 learned_addr learned my addr 172.21.15.98:0/3497518225 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.996+0000 7f0779d9b700 1 --2- 172.21.15.98:0/3497518225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f077c0b5140 0x7f077c0d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.996+0000 7f077ad9d700 1 --2- 172.21.15.98:0/3497518225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f077c0b8d10 0x7f077c14d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.996+0000 7f077a59c700 1 -- 172.21.15.98:0/3497518225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f077c0b5140 msgr2=0x7f077c0d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:55.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.996+0000 7f077a59c700 1 --2- 172.21.15.98:0/3497518225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f077c0b5140 0x7f077c0d1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.996+0000 7f077a59c700 1 -- 172.21.15.98:0/3497518225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f077c0b8d10 msgr2=0x7f077c14d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:55.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.996+0000 7f077a59c700 1 --2- 172.21.15.98:0/3497518225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f077c0b8d10 0x7f077c14d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.996+0000 7f077a59c700 1 -- 172.21.15.98:0/3497518225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0774009b30 con 0x7f077c0b26d0 2023-03-03T08:02:55.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.996+0000 7f077ad9d700 1 --2- 172.21.15.98:0/3497518225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f077c0b8d10 0x7f077c14d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:55.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.996+0000 7f077a59c700 1 --2- 172.21.15.98:0/3497518225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f077c0b26d0 0x7f077c0ca250 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f076c00d800 tx=0x7f076c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:55.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.996+0000 7f076affd700 1 -- 172.21.15.98:0/3497518225 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c0125f0 con 0x7f077c0b26d0 2023-03-03T08:02:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.996+0000 7f076affd700 1 -- 172.21.15.98:0/3497518225 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c012750 con 0x7f077c0b26d0 2023-03-03T08:02:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.996+0000 7f076affd700 1 -- 172.21.15.98:0/3497518225 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c012b00 con 0x7f077c0b26d0 2023-03-03T08:02:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.996+0000 7f07809ff700 1 -- 172.21.15.98:0/3497518225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077c0d0e80 con 0x7f077c0b26d0 2023-03-03T08:02:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.997+0000 7f07809ff700 1 -- 172.21.15.98:0/3497518225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077c0cdd00 con 0x7f077c0b26d0 2023-03-03T08:02:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.998+0000 7f07809ff700 1 -- 172.21.15.98:0/3497518225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f077c0cdfe0 con 0x7f077c0b26d0 2023-03-03T08:02:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:54.999+0000 7f076affd700 1 -- 172.21.15.98:0/3497518225 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f076c012c60 con 0x7f077c0b26d0 2023-03-03T08:02:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.000+0000 7f076affd700 1 --2- 172.21.15.98:0/3497518225 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f075c0683b0 0x7f075c06a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.000+0000 7f076affd700 1 -- 172.21.15.98:0/3497518225 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f076c086240 con 0x7f077c0b26d0 2023-03-03T08:02:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.001+0000 7f0779d9b700 1 --2- 172.21.15.98:0/3497518225 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f075c0683b0 0x7f075c06a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.001+0000 7f0779d9b700 1 --2- 172.21.15.98:0/3497518225 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f075c0683b0 0x7f075c06a860 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0770006fd0 tx=0x7f0770008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:55.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.004+0000 7f076affd700 1 -- 172.21.15.98:0/3497518225 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f076c052960 con 0x7f077c0b26d0 2023-03-03T08:02:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.196+0000 7f07809ff700 1 -- 172.21.15.98:0/3497518225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f077c15e1f0 con 0x7f077c0b26d0 2023-03-03T08:02:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.196+0000 7f076affd700 1 -- 172.21.15.98:0/3497518225 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7f077c15e1f0 con 0x7f077c0b26d0 2023-03-03T08:02:55.198 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T08:02:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.199+0000 7f07809ff700 1 -- 172.21.15.98:0/3497518225 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f075c0683b0 msgr2=0x7f075c06a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.199+0000 7f07809ff700 1 --2- 172.21.15.98:0/3497518225 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f075c0683b0 0x7f075c06a860 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0770006fd0 tx=0x7f0770008040 comp rx=0 tx=0).stop 2023-03-03T08:02:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.199+0000 7f07809ff700 1 -- 172.21.15.98:0/3497518225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f077c0b26d0 msgr2=0x7f077c0ca250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.199+0000 7f07809ff700 1 --2- 172.21.15.98:0/3497518225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f077c0b26d0 0x7f077c0ca250 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f076c00d800 tx=0x7f076c00dbc0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.199+0000 7f07809ff700 1 -- 172.21.15.98:0/3497518225 shutdown_connections 2023-03-03T08:02:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.199+0000 7f07809ff700 1 --2- 172.21.15.98:0/3497518225 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f075c0683b0 0x7f075c06a860 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.199+0000 7f07809ff700 1 --2- 172.21.15.98:0/3497518225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f077c0b26d0 0x7f077c0ca250 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.199+0000 7f07809ff700 1 --2- 172.21.15.98:0/3497518225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f077c0b8d10 0x7f077c14d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.199+0000 7f07809ff700 1 --2- 172.21.15.98:0/3497518225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f077c0b5140 0x7f077c0d1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.199+0000 7f07809ff700 1 -- 172.21.15.98:0/3497518225 >> 172.21.15.98:0/3497518225 conn(0x7f077c073e70 msgr2=0x7f077c0b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.199+0000 7f07809ff700 1 -- 172.21.15.98:0/3497518225 shutdown_connections 2023-03-03T08:02:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.199+0000 7f07809ff700 1 -- 172.21.15.98:0/3497518225 wait complete. 2023-03-03T08:02:55.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T08:02:55.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.474+0000 7feb705ff700 1 Processor -- start 2023-03-03T08:02:55.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.474+0000 7feb705ff700 1 -- start start 2023-03-03T08:02:55.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.475+0000 7feb705ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb6815ca70 0x7feb680cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.475+0000 7feb705ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb680d0270 0x7feb680d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.475+0000 7feb705ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb680cf980 con 0x7feb6806b4b0 2023-03-03T08:02:55.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.475+0000 7feb705ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb680cd5b0 con 0x7feb6815ca70 2023-03-03T08:02:55.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.475+0000 7feb705ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb680d1800 con 0x7feb680d0270 2023-03-03T08:02:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.476+0000 7feb6e39b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb6815ca70 0x7feb680cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.476+0000 7feb6e39b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb6815ca70 0x7feb680cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34188/0 (socket says 172.21.15.98:34188) 2023-03-03T08:02:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.476+0000 7feb6e39b700 1 -- 172.21.15.98:0/3927667026 learned_addr learned my addr 172.21.15.98:0/3927667026 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.476+0000 7feb6db9a700 1 --1- 172.21.15.98:0/3927667026 >> v1:172.21.15.98:6789/0 conn(0x7feb6806b4b0 0x7feb680cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38616/0 (socket says 172.21.15.98:38616) 2023-03-03T08:02:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.476+0000 7feb6eb9c700 1 --2- 172.21.15.98:0/3927667026 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb680d0270 0x7feb680d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.476+0000 7feb6e39b700 1 -- 172.21.15.98:0/3927667026 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb680d0270 msgr2=0x7feb680d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.476+0000 7feb6e39b700 1 --2- 172.21.15.98:0/3927667026 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb680d0270 0x7feb680d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.476+0000 7feb6e39b700 1 -- 172.21.15.98:0/3927667026 >> v1:172.21.15.98:6789/0 conn(0x7feb6806b4b0 legacy=0x7feb680cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.476+0000 7feb6e39b700 1 -- 172.21.15.98:0/3927667026 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb680cd900 con 0x7feb6815ca70 2023-03-03T08:02:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.477+0000 7feb6e39b700 1 --2- 172.21.15.98:0/3927667026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb6815ca70 0x7feb680cf440 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7feb640098f0 tx=0x7feb64009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5eab60b111a0bf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.477+0000 7feb6d399700 1 -- 172.21.15.98:0/3927667026 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb64011830 con 0x7feb6815ca70 2023-03-03T08:02:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.477+0000 7feb6d399700 1 -- 172.21.15.98:0/3927667026 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb64011990 con 0x7feb6815ca70 2023-03-03T08:02:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.477+0000 7feb6d399700 1 -- 172.21.15.98:0/3927667026 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb64011d20 con 0x7feb6815ca70 2023-03-03T08:02:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.477+0000 7feb705ff700 1 -- 172.21.15.98:0/3927667026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb6815ca70 msgr2=0x7feb680cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.477+0000 7feb705ff700 1 --2- 172.21.15.98:0/3927667026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb6815ca70 0x7feb680cf440 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7feb640098f0 tx=0x7feb64009c00 comp rx=0 tx=0).stop 2023-03-03T08:02:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.478+0000 7feb705ff700 1 -- 172.21.15.98:0/3927667026 shutdown_connections 2023-03-03T08:02:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.478+0000 7feb705ff700 1 --2- 172.21.15.98:0/3927667026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb6815ca70 0x7feb680cf440 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.478+0000 7feb705ff700 1 --2- 172.21.15.98:0/3927667026 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb680d0270 0x7feb680d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.478+0000 7feb705ff700 1 -- 172.21.15.98:0/3927667026 >> 172.21.15.98:0/3927667026 conn(0x7feb68073e70 msgr2=0x7feb6806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.478+0000 7feb705ff700 1 -- 172.21.15.98:0/3927667026 shutdown_connections 2023-03-03T08:02:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.478+0000 7feb705ff700 1 -- 172.21.15.98:0/3927667026 wait complete. 2023-03-03T08:02:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.480+0000 7feb705ff700 1 Processor -- start 2023-03-03T08:02:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.480+0000 7feb705ff700 1 -- start start 2023-03-03T08:02:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.481+0000 7feb705ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb6806b4b0 0x7feb681f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.481+0000 7feb705ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb680d0270 0x7feb68153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.481+0000 7feb705ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb6815ca70 0x7feb6815e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.481+0000 7feb705ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb680ca250 con 0x7feb6815ca70 2023-03-03T08:02:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.481+0000 7feb705ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb680cadd0 con 0x7feb680d0270 2023-03-03T08:02:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.481+0000 7feb705ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb681f3b00 con 0x7feb6806b4b0 2023-03-03T08:02:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.481+0000 7feb6eb9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb6815ca70 0x7feb6815e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.481+0000 7feb6eb9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb6815ca70 0x7feb6815e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34194/0 (socket says 172.21.15.98:34194) 2023-03-03T08:02:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.481+0000 7feb6eb9c700 1 -- 172.21.15.98:0/684164476 learned_addr learned my addr 172.21.15.98:0/684164476 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.481+0000 7feb6e39b700 1 --2- 172.21.15.98:0/684164476 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb6806b4b0 0x7feb681f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.481+0000 7feb6db9a700 1 --2- 172.21.15.98:0/684164476 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb680d0270 0x7feb68153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.482+0000 7feb6eb9c700 1 -- 172.21.15.98:0/684164476 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb6806b4b0 msgr2=0x7feb681f35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.482+0000 7feb6eb9c700 1 --2- 172.21.15.98:0/684164476 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb6806b4b0 0x7feb681f35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.482+0000 7feb6eb9c700 1 -- 172.21.15.98:0/684164476 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb680d0270 msgr2=0x7feb68153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.482+0000 7feb6eb9c700 1 --2- 172.21.15.98:0/684164476 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb680d0270 0x7feb68153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.482+0000 7feb6eb9c700 1 -- 172.21.15.98:0/684164476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb64009550 con 0x7feb6815ca70 2023-03-03T08:02:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.482+0000 7feb6eb9c700 1 --2- 172.21.15.98:0/684164476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb6815ca70 0x7feb6815e4a0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7feb6000efc0 tx=0x7feb6000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.482+0000 7feb5effd700 1 -- 172.21.15.98:0/684164476 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb60009ba0 con 0x7feb6815ca70 2023-03-03T08:02:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.482+0000 7feb5effd700 1 -- 172.21.15.98:0/684164476 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6000f040 con 0x7feb6815ca70 2023-03-03T08:02:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.482+0000 7feb5effd700 1 -- 172.21.15.98:0/684164476 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb60014480 con 0x7feb6815ca70 2023-03-03T08:02:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.482+0000 7feb705ff700 1 -- 172.21.15.98:0/684164476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb680d1aa0 con 0x7feb6815ca70 2023-03-03T08:02:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.482+0000 7feb705ff700 1 -- 172.21.15.98:0/684164476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb680cdbe0 con 0x7feb6815ca70 2023-03-03T08:02:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.483+0000 7feb705ff700 1 -- 172.21.15.98:0/684164476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb6815ce40 con 0x7feb6815ca70 2023-03-03T08:02:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.485+0000 7feb5effd700 1 -- 172.21.15.98:0/684164476 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7feb600145e0 con 0x7feb6815ca70 2023-03-03T08:02:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.486+0000 7feb5effd700 1 --2- 172.21.15.98:0/684164476 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7feb4c068380 0x7feb4c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.486+0000 7feb5effd700 1 -- 172.21.15.98:0/684164476 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7feb60086870 con 0x7feb6815ca70 2023-03-03T08:02:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.486+0000 7feb6e39b700 1 --2- 172.21.15.98:0/684164476 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7feb4c068380 0x7feb4c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.487+0000 7feb6e39b700 1 --2- 172.21.15.98:0/684164476 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7feb4c068380 0x7feb4c06a830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feb64009c40 tx=0x7feb64008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.490+0000 7feb5effd700 1 -- 172.21.15.98:0/684164476 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feb60052f90 con 0x7feb6815ca70 2023-03-03T08:02:55.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.680+0000 7feb705ff700 1 -- 172.21.15.98:0/684164476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feb680caf40 con 0x7feb6815ca70 2023-03-03T08:02:55.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.681+0000 7feb5effd700 1 -- 172.21.15.98:0/684164476 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7feb60084cf0 con 0x7feb6815ca70 2023-03-03T08:02:55.683 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T08:02:55.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.683+0000 7feb705ff700 1 -- 172.21.15.98:0/684164476 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7feb4c068380 msgr2=0x7feb4c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:55.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.683+0000 7feb705ff700 1 --2- 172.21.15.98:0/684164476 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7feb4c068380 0x7feb4c06a830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feb64009c40 tx=0x7feb64008030 comp rx=0 tx=0).stop 2023-03-03T08:02:55.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.683+0000 7feb705ff700 1 -- 172.21.15.98:0/684164476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb6815ca70 msgr2=0x7feb6815e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:55.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.683+0000 7feb705ff700 1 --2- 172.21.15.98:0/684164476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb6815ca70 0x7feb6815e4a0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7feb6000efc0 tx=0x7feb6000c3a0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.684+0000 7feb705ff700 1 -- 172.21.15.98:0/684164476 shutdown_connections 2023-03-03T08:02:55.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.684+0000 7feb705ff700 1 --2- 172.21.15.98:0/684164476 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7feb4c068380 0x7feb4c06a830 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.684+0000 7feb705ff700 1 --2- 172.21.15.98:0/684164476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb6815ca70 0x7feb6815e4a0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.684+0000 7feb705ff700 1 --2- 172.21.15.98:0/684164476 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb6806b4b0 0x7feb681f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.684+0000 7feb705ff700 1 --2- 172.21.15.98:0/684164476 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb680d0270 0x7feb68153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.684+0000 7feb705ff700 1 -- 172.21.15.98:0/684164476 >> 172.21.15.98:0/684164476 conn(0x7feb68073e70 msgr2=0x7feb680d2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:55.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.684+0000 7feb705ff700 1 -- 172.21.15.98:0/684164476 shutdown_connections 2023-03-03T08:02:55.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.684+0000 7feb705ff700 1 -- 172.21.15.98:0/684164476 wait complete. 2023-03-03T08:02:55.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:02:55.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.956+0000 7f4da39ff700 1 Processor -- start 2023-03-03T08:02:55.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.956+0000 7f4da39ff700 1 -- start start 2023-03-03T08:02:55.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.958+0000 7f4da39ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c15cc00 0x7f4d9c0cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.958+0000 7f4da39ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4d9c0d2c30 0x7f4d9c0cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.958+0000 7f4da39ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9c0cde10 con 0x7f4d9c0d27c0 2023-03-03T08:02:55.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.958+0000 7f4da39ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d9c0ce040 con 0x7f4d9c0d2c30 2023-03-03T08:02:55.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.958+0000 7f4da39ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c0d30b0 con 0x7f4d9c15cc00 2023-03-03T08:02:55.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.958+0000 7f4da1f9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4d9c0d2c30 0x7f4d9c0cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.958+0000 7f4da1f9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4d9c0d2c30 0x7f4d9c0cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34204/0 (socket says 172.21.15.98:34204) 2023-03-03T08:02:55.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.958+0000 7f4da1f9c700 1 -- 172.21.15.98:0/4012619865 learned_addr learned my addr 172.21.15.98:0/4012619865 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:55.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.958+0000 7f4da179b700 1 --2- 172.21.15.98:0/4012619865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c15cc00 0x7f4d9c0cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.959+0000 7f4da1f9c700 1 -- 172.21.15.98:0/4012619865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c15cc00 msgr2=0x7f4d9c0cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:55.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.959+0000 7f4da1f9c700 1 --2- 172.21.15.98:0/4012619865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c15cc00 0x7f4d9c0cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.959+0000 7f4da1f9c700 1 -- 172.21.15.98:0/4012619865 >> v1:172.21.15.100:6789/0 conn(0x7f4d9c0d27c0 legacy=0x7f4d9c0b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.959+0000 7f4da1f9c700 1 -- 172.21.15.98:0/4012619865 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c0d1a00 con 0x7f4d9c0d2c30 2023-03-03T08:02:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.959+0000 7f4da1f9c700 1 --2- 172.21.15.98:0/4012619865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c0d2c30 0x7f4d9c0cf4d0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f4d94009e50 tx=0x7f4d9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e07ec111c2c88d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.959+0000 7f4d93fff700 1 -- 172.21.15.98:0/4012619865 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9400e070 con 0x7f4d9c0d2c30 2023-03-03T08:02:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.959+0000 7f4d93fff700 1 -- 172.21.15.98:0/4012619865 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9400b910 con 0x7f4d9c0d2c30 2023-03-03T08:02:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.959+0000 7f4d93fff700 1 -- 172.21.15.98:0/4012619865 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9400bcc0 con 0x7f4d9c0d2c30 2023-03-03T08:02:55.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.960+0000 7f4da39ff700 1 -- 172.21.15.98:0/4012619865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c0d2c30 msgr2=0x7f4d9c0cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:55.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.960+0000 7f4da39ff700 1 --2- 172.21.15.98:0/4012619865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c0d2c30 0x7f4d9c0cf4d0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f4d94009e50 tx=0x7f4d9400b330 comp rx=0 tx=0).stop 2023-03-03T08:02:55.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.960+0000 7f4da39ff700 1 -- 172.21.15.98:0/4012619865 shutdown_connections 2023-03-03T08:02:55.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.960+0000 7f4da39ff700 1 --2- 172.21.15.98:0/4012619865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c0d2c30 0x7f4d9c0cf4d0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.960+0000 7f4da39ff700 1 --2- 172.21.15.98:0/4012619865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c15cc00 0x7f4d9c0cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.960+0000 7f4da39ff700 1 -- 172.21.15.98:0/4012619865 >> 172.21.15.98:0/4012619865 conn(0x7f4d9c06aee0 msgr2=0x7f4d9c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:55.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.960+0000 7f4da39ff700 1 -- 172.21.15.98:0/4012619865 shutdown_connections 2023-03-03T08:02:55.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.960+0000 7f4da39ff700 1 -- 172.21.15.98:0/4012619865 wait complete. 2023-03-03T08:02:55.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.962+0000 7f4da39ff700 1 Processor -- start 2023-03-03T08:02:55.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.962+0000 7f4da39ff700 1 -- start start 2023-03-03T08:02:55.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.963+0000 7f4da39ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c0d27c0 0x7f4d9c14f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.963+0000 7f4da39ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c0d2c30 0x7f4d9c0cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.963+0000 7f4da39ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d9c15cc00 0x7f4d9c0cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.963+0000 7f4da39ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c0cd610 con 0x7f4d9c0d2c30 2023-03-03T08:02:55.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.963+0000 7f4da39ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c0cd050 con 0x7f4d9c15cc00 2023-03-03T08:02:55.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.963+0000 7f4da39ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c0cd330 con 0x7f4d9c0d27c0 2023-03-03T08:02:55.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.964+0000 7f4da0f9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c0d2c30 0x7f4d9c0cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.964+0000 7f4da0f9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c0d2c30 0x7f4d9c0cc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34206/0 (socket says 172.21.15.98:34206) 2023-03-03T08:02:55.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.964+0000 7f4da0f9a700 1 -- 172.21.15.98:0/3067261822 learned_addr learned my addr 172.21.15.98:0/3067261822 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:55.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.964+0000 7f4da179b700 1 --2- 172.21.15.98:0/3067261822 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c0d27c0 0x7f4d9c14f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.964+0000 7f4da1f9c700 1 --2- 172.21.15.98:0/3067261822 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d9c15cc00 0x7f4d9c0cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.964+0000 7f4da0f9a700 1 -- 172.21.15.98:0/3067261822 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c0d27c0 msgr2=0x7f4d9c14f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:55.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.964+0000 7f4da0f9a700 1 --2- 172.21.15.98:0/3067261822 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c0d27c0 0x7f4d9c14f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.964+0000 7f4da0f9a700 1 -- 172.21.15.98:0/3067261822 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d9c15cc00 msgr2=0x7f4d9c0cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.964+0000 7f4da0f9a700 1 --2- 172.21.15.98:0/3067261822 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d9c15cc00 0x7f4d9c0cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.964+0000 7f4da0f9a700 1 -- 172.21.15.98:0/3067261822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d94009b30 con 0x7f4d9c0d2c30 2023-03-03T08:02:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.965+0000 7f4da0f9a700 1 --2- 172.21.15.98:0/3067261822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c0d2c30 0x7f4d9c0cc490 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c00c450 tx=0x7f4d8c00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.965+0000 7f4d91ffb700 1 -- 172.21.15.98:0/3067261822 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c0052d0 con 0x7f4d9c0d2c30 2023-03-03T08:02:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.965+0000 7f4d91ffb700 1 -- 172.21.15.98:0/3067261822 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8c005430 con 0x7f4d9c0d2c30 2023-03-03T08:02:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.965+0000 7f4d91ffb700 1 -- 172.21.15.98:0/3067261822 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c0038c0 con 0x7f4d9c0d2c30 2023-03-03T08:02:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.965+0000 7f4da39ff700 1 -- 172.21.15.98:0/3067261822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d9c0d1fc0 con 0x7f4d9c0d2c30 2023-03-03T08:02:55.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.965+0000 7f4da39ff700 1 -- 172.21.15.98:0/3067261822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d9c0d1ce0 con 0x7f4d9c0d2c30 2023-03-03T08:02:55.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.966+0000 7f4da39ff700 1 -- 172.21.15.98:0/3067261822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d9c0d22a0 con 0x7f4d9c0d2c30 2023-03-03T08:02:55.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.967+0000 7f4d91ffb700 1 -- 172.21.15.98:0/3067261822 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f4d8c019030 con 0x7f4d9c0d2c30 2023-03-03T08:02:55.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.969+0000 7f4d91ffb700 1 --2- 172.21.15.98:0/3067261822 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4d80068380 0x7f4d8006a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:55.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.970+0000 7f4d91ffb700 1 -- 172.21.15.98:0/3067261822 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4d8c084fc0 con 0x7f4d9c0d2c30 2023-03-03T08:02:55.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.970+0000 7f4da179b700 1 --2- 172.21.15.98:0/3067261822 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4d80068380 0x7f4d8006a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:55.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.971+0000 7f4da179b700 1 --2- 172.21.15.98:0/3067261822 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4d80068380 0x7f4d8006a830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009be0 tx=0x7f4d98009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:55.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:55.973+0000 7f4d91ffb700 1 -- 172.21.15.98:0/3067261822 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4d8c0516e0 con 0x7f4d9c0d2c30 2023-03-03T08:02:56.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.206+0000 7f4da39ff700 1 -- 172.21.15.98:0/3067261822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d9c0729b0 con 0x7f4d9c0d2c30 2023-03-03T08:02:56.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.207+0000 7f4d91ffb700 1 -- 172.21.15.98:0/3067261822 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f4d8c017090 con 0x7f4d9c0d2c30 2023-03-03T08:02:56.208 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:56.209 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6602,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/463617820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":463617820},{"type":"v1","addr":"172.21.15.100:6835","nonce":463617820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2561725713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2561725713},{"type":"v1","addr":"172.21.15.98:6834","nonce":2561725713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1511797280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1511797280},{"type":"v1","addr":"172.21.15.100:6834","nonce":1511797280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:02:22.404483+0000","modified":"2023-03-03T08:02:25.872537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7524":{"gid":7524,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6835/341265634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":341265634},{"type":"v1","addr":"172.21.15.98:6835","nonce":341265634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-03-03T08:02:56.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.209+0000 7f4da39ff700 1 -- 172.21.15.98:0/3067261822 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4d80068380 msgr2=0x7f4d8006a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:56.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.209+0000 7f4da39ff700 1 --2- 172.21.15.98:0/3067261822 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4d80068380 0x7f4d8006a830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009be0 tx=0x7f4d98009210 comp rx=0 tx=0).stop 2023-03-03T08:02:56.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.209+0000 7f4da39ff700 1 -- 172.21.15.98:0/3067261822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c0d2c30 msgr2=0x7f4d9c0cc490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:56.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.209+0000 7f4da39ff700 1 --2- 172.21.15.98:0/3067261822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c0d2c30 0x7f4d9c0cc490 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c00c450 tx=0x7f4d8c00c760 comp rx=0 tx=0).stop 2023-03-03T08:02:56.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.210+0000 7f4da39ff700 1 -- 172.21.15.98:0/3067261822 shutdown_connections 2023-03-03T08:02:56.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.210+0000 7f4da39ff700 1 --2- 172.21.15.98:0/3067261822 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d9c15cc00 0x7f4d9c0cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:56.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.210+0000 7f4da39ff700 1 --2- 172.21.15.98:0/3067261822 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d9c0d27c0 0x7f4d9c14f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:56.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.210+0000 7f4da39ff700 1 --2- 172.21.15.98:0/3067261822 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4d80068380 0x7f4d8006a830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:56.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.210+0000 7f4da39ff700 1 --2- 172.21.15.98:0/3067261822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d9c0d2c30 0x7f4d9c0cc490 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:56.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.210+0000 7f4da39ff700 1 -- 172.21.15.98:0/3067261822 >> 172.21.15.98:0/3067261822 conn(0x7f4d9c06aee0 msgr2=0x7f4d9c15bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:56.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.210+0000 7f4da39ff700 1 -- 172.21.15.98:0/3067261822 shutdown_connections 2023-03-03T08:02:56.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.210+0000 7f4da39ff700 1 -- 172.21.15.98:0/3067261822 wait complete. 2023-03-03T08:02:56.215 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 54 2023-03-03T08:02:56.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:02:56.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.483+0000 7fade47ff700 1 Processor -- start 2023-03-03T08:02:56.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.483+0000 7fade47ff700 1 -- start start 2023-03-03T08:02:56.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.485+0000 7fade47ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faddc0b8b90 0x7faddc0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:56.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.485+0000 7fade47ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faddc0d0ef0 con 0x7faddc0b87c0 2023-03-03T08:02:56.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.485+0000 7fade47ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faddc0d0600 con 0x7faddc0b5410 2023-03-03T08:02:56.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.485+0000 7fade47ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faddc0cd970 con 0x7faddc0b8b90 2023-03-03T08:02:56.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.485+0000 7fade1d9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faddc0b8b90 0x7faddc0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:56.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.485+0000 7fade259b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7faddc0b87c0 0x7faddc0b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38618/0 (socket says 172.21.15.98:38618) 2023-03-03T08:02:56.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.485+0000 7fade259b700 1 -- 172.21.15.98:0/2861403264 learned_addr learned my addr 172.21.15.98:0/2861403264 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:56.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.485+0000 7fade1d9a700 1 -- 172.21.15.98:0/2861403264 >> v1:172.21.15.100:6789/0 conn(0x7faddc0b5410 legacy=0x7faddc0d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:56.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.485+0000 7fade1d9a700 1 -- 172.21.15.98:0/2861403264 >> v1:172.21.15.98:6789/0 conn(0x7faddc0b87c0 legacy=0x7faddc0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:56.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.485+0000 7fade1d9a700 1 -- 172.21.15.98:0/2861403264 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faddc0cdc50 con 0x7faddc0b8b90 2023-03-03T08:02:56.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.486+0000 7fade1d9a700 1 --2- 172.21.15.98:0/2861403264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faddc0b8b90 0x7faddc0cffe0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009840 tx=0x7fadcc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e839c548b38e73 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:56.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.486+0000 7fade1599700 1 -- 172.21.15.98:0/2861403264 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc004230 con 0x7faddc0b8b90 2023-03-03T08:02:56.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.486+0000 7fade1599700 1 -- 172.21.15.98:0/2861403264 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadcc004390 con 0x7faddc0b8b90 2023-03-03T08:02:56.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.486+0000 7fade1599700 1 -- 172.21.15.98:0/2861403264 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc0036a0 con 0x7faddc0b8b90 2023-03-03T08:02:56.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.487+0000 7fade47ff700 1 -- 172.21.15.98:0/2861403264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faddc0b8b90 msgr2=0x7faddc0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:56.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.487+0000 7fade47ff700 1 --2- 172.21.15.98:0/2861403264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faddc0b8b90 0x7faddc0cffe0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009840 tx=0x7fadcc009b50 comp rx=0 tx=0).stop 2023-03-03T08:02:56.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.487+0000 7fade47ff700 1 -- 172.21.15.98:0/2861403264 shutdown_connections 2023-03-03T08:02:56.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.487+0000 7fade47ff700 1 --2- 172.21.15.98:0/2861403264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faddc0b8b90 0x7faddc0cffe0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:56.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.487+0000 7fade47ff700 1 -- 172.21.15.98:0/2861403264 >> 172.21.15.98:0/2861403264 conn(0x7faddc06aee0 msgr2=0x7faddc06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:56.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.487+0000 7fade47ff700 1 -- 172.21.15.98:0/2861403264 shutdown_connections 2023-03-03T08:02:56.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.487+0000 7fade47ff700 1 -- 172.21.15.98:0/2861403264 wait complete. 2023-03-03T08:02:56.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.489+0000 7fade47ff700 1 Processor -- start 2023-03-03T08:02:56.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.489+0000 7fade47ff700 1 -- start start 2023-03-03T08:02:56.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.490+0000 7fade47ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faddc0b5410 0x7faddc1f5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:56.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.490+0000 7fade47ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faddc0b87c0 0x7faddc1f5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:56.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.490+0000 7fade47ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faddc0b8b90 0x7faddc1f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:56.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.490+0000 7fade47ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc0cd240 con 0x7faddc0b8b90 2023-03-03T08:02:56.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.490+0000 7fade47ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc1f34c0 con 0x7faddc0b87c0 2023-03-03T08:02:56.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.490+0000 7fade47ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faddc1f37a0 con 0x7faddc0b5410 2023-03-03T08:02:56.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.491+0000 7fade2d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faddc0b8b90 0x7faddc1f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:56.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.491+0000 7fade2d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faddc0b8b90 0x7faddc1f2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34216/0 (socket says 172.21.15.98:34216) 2023-03-03T08:02:56.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.491+0000 7fade2d9c700 1 -- 172.21.15.98:0/4192693328 learned_addr learned my addr 172.21.15.98:0/4192693328 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:56.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.491+0000 7fade1d9a700 1 --2- 172.21.15.98:0/4192693328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faddc0b87c0 0x7faddc1f5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:56.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.491+0000 7fade259b700 1 --2- 172.21.15.98:0/4192693328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faddc0b5410 0x7faddc1f5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:56.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.491+0000 7fade2d9c700 1 -- 172.21.15.98:0/4192693328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faddc0b5410 msgr2=0x7faddc1f5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:56.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.491+0000 7fade2d9c700 1 --2- 172.21.15.98:0/4192693328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faddc0b5410 0x7faddc1f5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:56.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.491+0000 7fade2d9c700 1 -- 172.21.15.98:0/4192693328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faddc0b87c0 msgr2=0x7faddc1f5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:56.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.491+0000 7fade2d9c700 1 --2- 172.21.15.98:0/4192693328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faddc0b87c0 0x7faddc1f5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:56.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.491+0000 7fade2d9c700 1 -- 172.21.15.98:0/4192693328 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc0094a0 con 0x7faddc0b8b90 2023-03-03T08:02:56.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.491+0000 7fade1d9a700 1 --2- 172.21.15.98:0/4192693328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faddc0b87c0 0x7faddc1f5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:56.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.491+0000 7fade2d9c700 1 --2- 172.21.15.98:0/4192693328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faddc0b8b90 0x7faddc1f2260 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fadd400b990 tx=0x7fadd400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:56.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.492+0000 7fadd2ffd700 1 -- 172.21.15.98:0/4192693328 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd400c9d0 con 0x7faddc0b8b90 2023-03-03T08:02:56.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.492+0000 7fadd2ffd700 1 -- 172.21.15.98:0/4192693328 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd400cb30 con 0x7faddc0b8b90 2023-03-03T08:02:56.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.492+0000 7fadd2ffd700 1 -- 172.21.15.98:0/4192693328 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd4014440 con 0x7faddc0b8b90 2023-03-03T08:02:56.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.492+0000 7fade47ff700 1 -- 172.21.15.98:0/4192693328 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faddc0cdf30 con 0x7faddc0b8b90 2023-03-03T08:02:56.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.492+0000 7fade47ff700 1 -- 172.21.15.98:0/4192693328 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faddc073a80 con 0x7faddc0b8b90 2023-03-03T08:02:56.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.493+0000 7fade47ff700 1 -- 172.21.15.98:0/4192693328 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faddc0ce0c0 con 0x7faddc0b8b90 2023-03-03T08:02:56.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.494+0000 7fadd2ffd700 1 -- 172.21.15.98:0/4192693328 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fadd401b030 con 0x7faddc0b8b90 2023-03-03T08:02:56.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.495+0000 7fadd2ffd700 1 --2- 172.21.15.98:0/4192693328 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fadc0068380 0x7fadc006a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:56.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.496+0000 7fadd2ffd700 1 -- 172.21.15.98:0/4192693328 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fadd4085aa0 con 0x7faddc0b8b90 2023-03-03T08:02:56.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.496+0000 7fade259b700 1 --2- 172.21.15.98:0/4192693328 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fadc0068380 0x7fadc006a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:56.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.497+0000 7fade259b700 1 --2- 172.21.15.98:0/4192693328 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fadc0068380 0x7fadc006a830 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fadd8000f60 tx=0x7fadd8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:56.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.499+0000 7fadd2ffd700 1 -- 172.21.15.98:0/4192693328 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fadd40521c0 con 0x7faddc0b8b90 2023-03-03T08:02:56.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.695+0000 7fade47ff700 1 -- 172.21.15.98:0/4192693328 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faddc0729b0 con 0x7faddc0b8b90 2023-03-03T08:02:56.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.696+0000 7fadd2ffd700 1 -- 172.21.15.98:0/4192693328 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+18053 (secure 0 0 0) 0x7fadd4019090 con 0x7faddc0b8b90 2023-03-03T08:02:56.698 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:56.698 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":98,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:02:45.624487+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:02:20.797579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-03-03T08:02:21.411899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:6835/1843752855":"2023-03-04T08:02:42.175049+0000","172.21.15.100:6833/1843752855":"2023-03-04T08:02:42.175049+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:02:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.699+0000 7fade47ff700 1 -- 172.21.15.98:0/4192693328 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fadc0068380 msgr2=0x7fadc006a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.699+0000 7fade47ff700 1 --2- 172.21.15.98:0/4192693328 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fadc0068380 0x7fadc006a830 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fadd8000f60 tx=0x7fadd8009040 comp rx=0 tx=0).stop 2023-03-03T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.699+0000 7fade47ff700 1 -- 172.21.15.98:0/4192693328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faddc0b8b90 msgr2=0x7faddc1f2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.699+0000 7fade47ff700 1 --2- 172.21.15.98:0/4192693328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faddc0b8b90 0x7faddc1f2260 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fadd400b990 tx=0x7fadd400bd50 comp rx=0 tx=0).stop 2023-03-03T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.699+0000 7fade47ff700 1 -- 172.21.15.98:0/4192693328 shutdown_connections 2023-03-03T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.699+0000 7fade47ff700 1 --2- 172.21.15.98:0/4192693328 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fadc0068380 0x7fadc006a830 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.699+0000 7fade47ff700 1 --2- 172.21.15.98:0/4192693328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faddc0b8b90 0x7faddc1f2260 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.699+0000 7fade47ff700 1 --2- 172.21.15.98:0/4192693328 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faddc0b5410 0x7faddc1f5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.699+0000 7fade47ff700 1 --2- 172.21.15.98:0/4192693328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faddc0b87c0 0x7faddc1f5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.699+0000 7fade47ff700 1 -- 172.21.15.98:0/4192693328 >> 172.21.15.98:0/4192693328 conn(0x7faddc06aee0 msgr2=0x7faddc0b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.699+0000 7fade47ff700 1 -- 172.21.15.98:0/4192693328 shutdown_connections 2023-03-03T08:02:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.699+0000 7fade47ff700 1 -- 172.21.15.98:0/4192693328 wait complete. 2023-03-03T08:02:56.728 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-03-03T08:02:56.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:02:56.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.972+0000 7f88af5ff700 1 Processor -- start 2023-03-03T08:02:56.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.972+0000 7f88af5ff700 1 -- start start 2023-03-03T08:02:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.973+0000 7f88af5ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f88a80b8b90 0x7f88a80cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.974+0000 7f88af5ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a80d0ef0 con 0x7f88a80b5410 2023-03-03T08:02:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.974+0000 7f88af5ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a80d0600 con 0x7f88a80b87c0 2023-03-03T08:02:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.974+0000 7f88af5ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a80cd970 con 0x7f88a80b8b90 2023-03-03T08:02:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.974+0000 7f88acb9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f88a80b8b90 0x7f88a80cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.974+0000 7f88acb9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f88a80b8b90 0x7f88a80cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34224/0 (socket says 172.21.15.98:34224) 2023-03-03T08:02:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.974+0000 7f88acb9a700 1 -- 172.21.15.98:0/3256412902 learned_addr learned my addr 172.21.15.98:0/3256412902 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.974+0000 7f88acb9a700 1 -- 172.21.15.98:0/3256412902 >> v1:172.21.15.100:6789/0 conn(0x7f88a80b87c0 legacy=0x7f88a80b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.974+0000 7f88acb9a700 1 -- 172.21.15.98:0/3256412902 >> v1:172.21.15.98:6789/0 conn(0x7f88a80b5410 legacy=0x7f88a80d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.975+0000 7f88acb9a700 1 -- 172.21.15.98:0/3256412902 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a80cdc50 con 0x7f88a80b8b90 2023-03-03T08:02:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.974+0000 7f889ffff700 1 -- 172.21.15.98:0/3256412902 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508999028 0 0) 0x7f88a80d0ef0 con 0x7f88a80b5410 2023-03-03T08:02:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.975+0000 7f88acb9a700 1 --2- 172.21.15.98:0/3256412902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a80b8b90 0x7f88a80cffe0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f88980098f0 tx=0x7f8898009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd6428983fd6191 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.975+0000 7f889ffff700 1 -- 172.21.15.98:0/3256412902 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889800b8a0 con 0x7f88a80b8b90 2023-03-03T08:02:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.975+0000 7f889ffff700 1 -- 172.21.15.98:0/3256412902 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889800ba00 con 0x7f88a80b8b90 2023-03-03T08:02:56.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.975+0000 7f889ffff700 1 -- 172.21.15.98:0/3256412902 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889800bd90 con 0x7f88a80b8b90 2023-03-03T08:02:56.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.976+0000 7f88af5ff700 1 -- 172.21.15.98:0/3256412902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a80b8b90 msgr2=0x7f88a80cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:56.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.976+0000 7f88af5ff700 1 --2- 172.21.15.98:0/3256412902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a80b8b90 0x7f88a80cffe0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f88980098f0 tx=0x7f8898009c00 comp rx=0 tx=0).stop 2023-03-03T08:02:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.976+0000 7f88af5ff700 1 -- 172.21.15.98:0/3256412902 shutdown_connections 2023-03-03T08:02:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.976+0000 7f88af5ff700 1 --2- 172.21.15.98:0/3256412902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a80b8b90 0x7f88a80cffe0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.976+0000 7f88af5ff700 1 -- 172.21.15.98:0/3256412902 >> 172.21.15.98:0/3256412902 conn(0x7f88a806aee0 msgr2=0x7f88a806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.976+0000 7f88af5ff700 1 -- 172.21.15.98:0/3256412902 shutdown_connections 2023-03-03T08:02:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.976+0000 7f88af5ff700 1 -- 172.21.15.98:0/3256412902 wait complete. 2023-03-03T08:02:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.978+0000 7f88af5ff700 1 Processor -- start 2023-03-03T08:02:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.978+0000 7f88af5ff700 1 -- start start 2023-03-03T08:02:56.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.979+0000 7f88af5ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88a80b5410 0x7f88a8152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:56.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.979+0000 7f88af5ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88a80b87c0 0x7f88a81532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:56.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.979+0000 7f88af5ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a80b8b90 0x7f88a80c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:56.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.979+0000 7f88af5ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a81530a0 con 0x7f88a80b8b90 2023-03-03T08:02:56.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.979+0000 7f88af5ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80c83b0 con 0x7f88a80b87c0 2023-03-03T08:02:56.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.979+0000 7f88af5ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80c8690 con 0x7f88a80b5410 2023-03-03T08:02:56.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.980+0000 7f88adb9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a80b8b90 0x7f88a80c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:56.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.980+0000 7f88adb9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a80b8b90 0x7f88a80c9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34240/0 (socket says 172.21.15.98:34240) 2023-03-03T08:02:56.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.980+0000 7f88adb9c700 1 -- 172.21.15.98:0/1608373335 learned_addr learned my addr 172.21.15.98:0/1608373335 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:56.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.980+0000 7f88acb9a700 1 --2- 172.21.15.98:0/1608373335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88a80b87c0 0x7f88a81532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:56.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.980+0000 7f88ad39b700 1 --2- 172.21.15.98:0/1608373335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88a80b5410 0x7f88a8152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:56.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.980+0000 7f88adb9c700 1 -- 172.21.15.98:0/1608373335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88a80b5410 msgr2=0x7f88a8152b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:56.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.980+0000 7f88adb9c700 1 --2- 172.21.15.98:0/1608373335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88a80b5410 0x7f88a8152b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:56.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.980+0000 7f88adb9c700 1 -- 172.21.15.98:0/1608373335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88a80b87c0 msgr2=0x7f88a81532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:56.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.980+0000 7f88adb9c700 1 --2- 172.21.15.98:0/1608373335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88a80b87c0 0x7f88a81532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:56.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.980+0000 7f88adb9c700 1 -- 172.21.15.98:0/1608373335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8898009550 con 0x7f88a80b8b90 2023-03-03T08:02:56.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.980+0000 7f88adb9c700 1 --2- 172.21.15.98:0/1608373335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a80b8b90 0x7f88a80c9450 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f88a000c7a0 tx=0x7f88a000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:56.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.980+0000 7f889dffb700 1 -- 172.21.15.98:0/1608373335 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0003e30 con 0x7f88a80b8b90 2023-03-03T08:02:56.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.981+0000 7f889dffb700 1 -- 172.21.15.98:0/1608373335 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a0005070 con 0x7f88a80b8b90 2023-03-03T08:02:56.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.981+0000 7f889dffb700 1 -- 172.21.15.98:0/1608373335 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0005420 con 0x7f88a80b8b90 2023-03-03T08:02:56.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.981+0000 7f88af5ff700 1 -- 172.21.15.98:0/1608373335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a80cdf30 con 0x7f88a80b8b90 2023-03-03T08:02:56.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.981+0000 7f88af5ff700 1 -- 172.21.15.98:0/1608373335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a8073a80 con 0x7f88a80b8b90 2023-03-03T08:02:56.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.982+0000 7f88af5ff700 1 -- 172.21.15.98:0/1608373335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a80ce0c0 con 0x7f88a80b8b90 2023-03-03T08:02:56.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.983+0000 7f889dffb700 1 -- 172.21.15.98:0/1608373335 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f88a0019030 con 0x7f88a80b8b90 2023-03-03T08:02:56.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.984+0000 7f889dffb700 1 --2- 172.21.15.98:0/1608373335 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f888c068380 0x7f888c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:56.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.984+0000 7f889dffb700 1 -- 172.21.15.98:0/1608373335 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f88a00852d0 con 0x7f88a80b8b90 2023-03-03T08:02:56.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.985+0000 7f88ad39b700 1 --2- 172.21.15.98:0/1608373335 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f888c068380 0x7f888c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:56.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.986+0000 7f88ad39b700 1 --2- 172.21.15.98:0/1608373335 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f888c068380 0x7f888c06a830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f88a80c8080 tx=0x7f88a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:56.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:56.988+0000 7f889dffb700 1 -- 172.21.15.98:0/1608373335 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f88a0051ae0 con 0x7f88a80b8b90 2023-03-03T08:02:57.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.221+0000 7f88af5ff700 1 -- 172.21.15.98:0/1608373335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88a80729b0 con 0x7f88a80b8b90 2023-03-03T08:02:57.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.222+0000 7f889dffb700 1 -- 172.21.15.98:0/1608373335 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f88a0017090 con 0x7f88a80b8b90 2023-03-03T08:02:57.224 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:57.224 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6602,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/463617820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":463617820},{"type":"v1","addr":"172.21.15.100:6835","nonce":463617820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2561725713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2561725713},{"type":"v1","addr":"172.21.15.98:6834","nonce":2561725713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1511797280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1511797280},{"type":"v1","addr":"172.21.15.100:6834","nonce":1511797280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:02:22.404483+0000","modified":"2023-03-03T08:02:25.872537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7524":{"gid":7524,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6835/341265634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":341265634},{"type":"v1","addr":"172.21.15.98:6835","nonce":341265634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-03-03T08:02:57.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.225+0000 7f88af5ff700 1 -- 172.21.15.98:0/1608373335 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f888c068380 msgr2=0x7f888c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:57.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.225+0000 7f88af5ff700 1 --2- 172.21.15.98:0/1608373335 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f888c068380 0x7f888c06a830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f88a80c8080 tx=0x7f88a4009040 comp rx=0 tx=0).stop 2023-03-03T08:02:57.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.225+0000 7f88af5ff700 1 -- 172.21.15.98:0/1608373335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a80b8b90 msgr2=0x7f88a80c9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:57.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.225+0000 7f88af5ff700 1 --2- 172.21.15.98:0/1608373335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a80b8b90 0x7f88a80c9450 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f88a000c7a0 tx=0x7f88a000cb60 comp rx=0 tx=0).stop 2023-03-03T08:02:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.225+0000 7f88af5ff700 1 -- 172.21.15.98:0/1608373335 shutdown_connections 2023-03-03T08:02:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.225+0000 7f88af5ff700 1 --2- 172.21.15.98:0/1608373335 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f888c068380 0x7f888c06a830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.225+0000 7f88af5ff700 1 --2- 172.21.15.98:0/1608373335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a80b8b90 0x7f88a80c9450 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.225+0000 7f88af5ff700 1 --2- 172.21.15.98:0/1608373335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88a80b5410 0x7f88a8152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.225+0000 7f88af5ff700 1 --2- 172.21.15.98:0/1608373335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88a80b87c0 0x7f88a81532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.225+0000 7f88af5ff700 1 -- 172.21.15.98:0/1608373335 >> 172.21.15.98:0/1608373335 conn(0x7f88a806aee0 msgr2=0x7f88a80b2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.225+0000 7f88af5ff700 1 -- 172.21.15.98:0/1608373335 shutdown_connections 2023-03-03T08:02:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.225+0000 7f88af5ff700 1 -- 172.21.15.98:0/1608373335 wait complete. 2023-03-03T08:02:57.228 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 54 2023-03-03T08:02:57.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:02:57.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.496+0000 7f7dc7bff700 1 Processor -- start 2023-03-03T08:02:57.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.496+0000 7f7dc7bff700 1 -- start start 2023-03-03T08:02:57.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.497+0000 7f7dc7bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7dc00b87c0 0x7f7dc00cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.497+0000 7f7dc7bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7dc00b5410 0x7f7dc00d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.498+0000 7f7dc7bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc00d0ef0 con 0x7f7dc00b8b90 2023-03-03T08:02:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.498+0000 7f7dc7bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc00d0600 con 0x7f7dc00b5410 2023-03-03T08:02:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.498+0000 7f7dc7bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc00cd970 con 0x7f7dc00b87c0 2023-03-03T08:02:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.498+0000 7f7dc619c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7dc00b5410 0x7f7dc00d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.498+0000 7f7dc619c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7dc00b5410 0x7f7dc00d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34254/0 (socket says 172.21.15.98:34254) 2023-03-03T08:02:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.498+0000 7f7dc619c700 1 -- 172.21.15.98:0/229277723 learned_addr learned my addr 172.21.15.98:0/229277723 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.498+0000 7f7dc599b700 1 --2- 172.21.15.98:0/229277723 >> v2:172.21.15.100:3300/0 conn(0x7f7dc00b87c0 0x7f7dc00cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.498+0000 7f7dc619c700 1 -- 172.21.15.98:0/229277723 >> v2:172.21.15.100:3300/0 conn(0x7f7dc00b87c0 msgr2=0x7f7dc00cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.498+0000 7f7dc619c700 1 --2- 172.21.15.98:0/229277723 >> v2:172.21.15.100:3300/0 conn(0x7f7dc00b87c0 0x7f7dc00cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.498+0000 7f7dc619c700 1 -- 172.21.15.98:0/229277723 >> v1:172.21.15.100:6789/0 conn(0x7f7dc00b8b90 legacy=0x7f7dc00b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.499+0000 7f7dc619c700 1 -- 172.21.15.98:0/229277723 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc00cdc50 con 0x7f7dc00b5410 2023-03-03T08:02:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.499+0000 7f7dc619c700 1 --2- 172.21.15.98:0/229277723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc00b5410 0x7f7dc00d13d0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0073840 tx=0x7f7db800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b1b3524823747e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.499+0000 7f7dc4999700 1 -- 172.21.15.98:0/229277723 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db800e070 con 0x7f7dc00b5410 2023-03-03T08:02:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.499+0000 7f7dc4999700 1 -- 172.21.15.98:0/229277723 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db8003a20 con 0x7f7dc00b5410 2023-03-03T08:02:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.499+0000 7f7dc4999700 1 -- 172.21.15.98:0/229277723 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db8003dd0 con 0x7f7dc00b5410 2023-03-03T08:02:57.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.500+0000 7f7dc7bff700 1 -- 172.21.15.98:0/229277723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc00b5410 msgr2=0x7f7dc00d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:57.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.500+0000 7f7dc7bff700 1 --2- 172.21.15.98:0/229277723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc00b5410 0x7f7dc00d13d0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0073840 tx=0x7f7db800b330 comp rx=0 tx=0).stop 2023-03-03T08:02:57.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.500+0000 7f7dc7bff700 1 -- 172.21.15.98:0/229277723 shutdown_connections 2023-03-03T08:02:57.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.500+0000 7f7dc7bff700 1 --2- 172.21.15.98:0/229277723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc00b5410 0x7f7dc00d13d0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:57.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.500+0000 7f7dc7bff700 1 --2- 172.21.15.98:0/229277723 >> v2:172.21.15.100:3300/0 conn(0x7f7dc00b87c0 0x7f7dc00cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:57.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.500+0000 7f7dc7bff700 1 -- 172.21.15.98:0/229277723 >> 172.21.15.98:0/229277723 conn(0x7f7dc006aee0 msgr2=0x7f7dc006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:57.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.500+0000 7f7dc7bff700 1 -- 172.21.15.98:0/229277723 shutdown_connections 2023-03-03T08:02:57.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.500+0000 7f7dc7bff700 1 -- 172.21.15.98:0/229277723 wait complete. 2023-03-03T08:02:57.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.502+0000 7f7dc7bff700 1 Processor -- start 2023-03-03T08:02:57.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.502+0000 7f7dc7bff700 1 -- start start 2023-03-03T08:02:57.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.503+0000 7f7dc7bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc00b5410 0x7f7dc01e5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:57.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.503+0000 7f7dc7bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7dc00b87c0 0x7f7dc01e61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:57.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.503+0000 7f7dc7bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7dc00b8b90 0x7f7dc00cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:57.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.503+0000 7f7dc7bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc01e7610 con 0x7f7dc00b5410 2023-03-03T08:02:57.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.503+0000 7f7dc7bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc01e6a00 con 0x7f7dc00b87c0 2023-03-03T08:02:57.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.503+0000 7f7dc7bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc01e6ce0 con 0x7f7dc00b8b90 2023-03-03T08:02:57.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.503+0000 7f7dc599b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc00b5410 0x7f7dc01e5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:57.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.504+0000 7f7dc599b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc00b5410 0x7f7dc01e5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34266/0 (socket says 172.21.15.98:34266) 2023-03-03T08:02:57.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.504+0000 7f7dc599b700 1 -- 172.21.15.98:0/2011525165 learned_addr learned my addr 172.21.15.98:0/2011525165 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:57.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.504+0000 7f7dc619c700 1 --2- 172.21.15.98:0/2011525165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7dc00b8b90 0x7f7dc00cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:57.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.504+0000 7f7dc519a700 1 --2- 172.21.15.98:0/2011525165 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7dc00b87c0 0x7f7dc01e61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:57.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.504+0000 7f7dc599b700 1 -- 172.21.15.98:0/2011525165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7dc00b8b90 msgr2=0x7f7dc00cb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:57.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.504+0000 7f7dc599b700 1 --2- 172.21.15.98:0/2011525165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7dc00b8b90 0x7f7dc00cb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:57.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.504+0000 7f7dc599b700 1 -- 172.21.15.98:0/2011525165 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7dc00b87c0 msgr2=0x7f7dc01e61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:57.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.504+0000 7f7dc599b700 1 --2- 172.21.15.98:0/2011525165 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7dc00b87c0 0x7f7dc01e61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:57.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.504+0000 7f7dc599b700 1 -- 172.21.15.98:0/2011525165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db8009b30 con 0x7f7dc00b5410 2023-03-03T08:02:57.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.504+0000 7f7dc519a700 1 --2- 172.21.15.98:0/2011525165 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7dc00b87c0 0x7f7dc01e61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:02:57.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.504+0000 7f7dc599b700 1 --2- 172.21.15.98:0/2011525165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc00b5410 0x7f7dc01e5c60 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc00e810 tx=0x7f7dbc00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:57.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.505+0000 7f7db67fc700 1 -- 172.21.15.98:0/2011525165 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc013610 con 0x7f7dc00b5410 2023-03-03T08:02:57.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.505+0000 7f7db67fc700 1 -- 172.21.15.98:0/2011525165 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dbc013770 con 0x7f7dc00b5410 2023-03-03T08:02:57.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.505+0000 7f7db67fc700 1 -- 172.21.15.98:0/2011525165 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc013b20 con 0x7f7dc00b5410 2023-03-03T08:02:57.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.505+0000 7f7dc7bff700 1 -- 172.21.15.98:0/2011525165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc00cdf30 con 0x7f7dc00b5410 2023-03-03T08:02:57.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.505+0000 7f7dc7bff700 1 -- 172.21.15.98:0/2011525165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc0073a80 con 0x7f7dc00b5410 2023-03-03T08:02:57.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.506+0000 7f7dc7bff700 1 -- 172.21.15.98:0/2011525165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc00ce0c0 con 0x7f7dc00b5410 2023-03-03T08:02:57.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.507+0000 7f7db67fc700 1 -- 172.21.15.98:0/2011525165 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f7dbc023070 con 0x7f7dc00b5410 2023-03-03T08:02:57.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.508+0000 7f7db67fc700 1 --2- 172.21.15.98:0/2011525165 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7da4068380 0x7f7da406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:57.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.508+0000 7f7db67fc700 1 -- 172.21.15.98:0/2011525165 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7dbc085ba0 con 0x7f7dc00b5410 2023-03-03T08:02:57.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.509+0000 7f7dc519a700 1 --2- 172.21.15.98:0/2011525165 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7da4068380 0x7f7da406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:57.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.510+0000 7f7dc519a700 1 --2- 172.21.15.98:0/2011525165 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7da4068380 0x7f7da406a830 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7db0006fd0 tx=0x7f7db0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:57.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.512+0000 7f7db67fc700 1 -- 172.21.15.98:0/2011525165 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7dbc052240 con 0x7f7dc00b5410 2023-03-03T08:02:57.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.751+0000 7f7dc7bff700 1 -- 172.21.15.98:0/2011525165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7dc00729b0 con 0x7f7dc00b5410 2023-03-03T08:02:57.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.752+0000 7f7db67fc700 1 -- 172.21.15.98:0/2011525165 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f7dbc059e60 con 0x7f7dc00b5410 2023-03-03T08:02:57.753 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:57.753 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6602,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/463617820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":463617820},{"type":"v1","addr":"172.21.15.100:6835","nonce":463617820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2561725713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2561725713},{"type":"v1","addr":"172.21.15.98:6834","nonce":2561725713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1511797280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1511797280},{"type":"v1","addr":"172.21.15.100:6834","nonce":1511797280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:02:22.404483+0000","modified":"2023-03-03T08:02:25.872537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7524":{"gid":7524,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6835/341265634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":341265634},{"type":"v1","addr":"172.21.15.98:6835","nonce":341265634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-03-03T08:02:57.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.754+0000 7f7dc7bff700 1 -- 172.21.15.98:0/2011525165 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7da4068380 msgr2=0x7f7da406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:57.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.754+0000 7f7dc7bff700 1 --2- 172.21.15.98:0/2011525165 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7da4068380 0x7f7da406a830 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7db0006fd0 tx=0x7f7db0008040 comp rx=0 tx=0).stop 2023-03-03T08:02:57.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.754+0000 7f7dc7bff700 1 -- 172.21.15.98:0/2011525165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc00b5410 msgr2=0x7f7dc01e5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:57.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.754+0000 7f7dc7bff700 1 --2- 172.21.15.98:0/2011525165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc00b5410 0x7f7dc01e5c60 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc00e810 tx=0x7f7dbc00ebd0 comp rx=0 tx=0).stop 2023-03-03T08:02:57.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.754+0000 7f7dc7bff700 1 -- 172.21.15.98:0/2011525165 shutdown_connections 2023-03-03T08:02:57.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.754+0000 7f7dc7bff700 1 --2- 172.21.15.98:0/2011525165 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7da4068380 0x7f7da406a830 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:57.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.754+0000 7f7dc7bff700 1 --2- 172.21.15.98:0/2011525165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc00b5410 0x7f7dc01e5c60 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:57.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.754+0000 7f7dc7bff700 1 --2- 172.21.15.98:0/2011525165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7dc00b8b90 0x7f7dc00cb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:57.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.754+0000 7f7dc7bff700 1 --2- 172.21.15.98:0/2011525165 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7dc00b87c0 0x7f7dc01e61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:57.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.754+0000 7f7dc7bff700 1 -- 172.21.15.98:0/2011525165 >> 172.21.15.98:0/2011525165 conn(0x7f7dc006aee0 msgr2=0x7f7dc00b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:57.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.754+0000 7f7dc7bff700 1 -- 172.21.15.98:0/2011525165 shutdown_connections 2023-03-03T08:02:57.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:57.754+0000 7f7dc7bff700 1 -- 172.21.15.98:0/2011525165 wait complete. 2023-03-03T08:02:57.757 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 54 2023-03-03T08:02:57.782 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:02:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.026+0000 7f72ffbff700 1 Processor -- start 2023-03-03T08:02:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.026+0000 7f72ffbff700 1 -- start start 2023-03-03T08:02:58.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.027+0000 7f72ffbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72f80b87c0 0x7f72f80cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.027+0000 7f72ffbff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f72f80b5410 0x7f72f80d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.028+0000 7f72ffbff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f80d0ef0 con 0x7f72f80b8b90 2023-03-03T08:02:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.028+0000 7f72ffbff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f80d0600 con 0x7f72f80b5410 2023-03-03T08:02:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.028+0000 7f72ffbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80cd970 con 0x7f72f80b87c0 2023-03-03T08:02:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.028+0000 7f72fe19c700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f72f80b5410 0x7f72f80d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.028+0000 7f72fd19a700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f72f80b8b90 0x7f72f80b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:37854/0 (socket says 172.21.15.98:37854) 2023-03-03T08:02:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.028+0000 7f72fd19a700 1 -- 172.21.15.98:0/2367511416 learned_addr learned my addr 172.21.15.98:0/2367511416 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:02:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.028+0000 7f72fd99b700 1 --2- 172.21.15.98:0/2367511416 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72f80b87c0 0x7f72f80cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.029+0000 7f72fe19c700 1 -- 172.21.15.98:0/2367511416 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72f80b87c0 msgr2=0x7f72f80cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.029+0000 7f72fe19c700 1 --2- 172.21.15.98:0/2367511416 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72f80b87c0 0x7f72f80cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.029+0000 7f72fe19c700 1 -- 172.21.15.98:0/2367511416 >> v1:172.21.15.100:6789/0 conn(0x7f72f80b8b90 legacy=0x7f72f80b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.029+0000 7f72fe19c700 1 -- 172.21.15.98:0/2367511416 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f80cdc50 con 0x7f72f80b5410 2023-03-03T08:02:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.029+0000 7f72fc999700 1 -- 172.21.15.98:0/2367511416 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637366543 0 0) 0x7f72f80d0ef0 con 0x7f72f80b8b90 2023-03-03T08:02:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.029+0000 7f72fe19c700 1 --2- 172.21.15.98:0/2367511416 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72f80b5410 0x7f72f80d13d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009d90 tx=0x7f72f000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=258ad2e4640c4bb6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.029+0000 7f72fc999700 1 -- 172.21.15.98:0/2367511416 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f000e070 con 0x7f72f80b5410 2023-03-03T08:02:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.029+0000 7f72fc999700 1 -- 172.21.15.98:0/2367511416 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f000bb20 con 0x7f72f80b5410 2023-03-03T08:02:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.029+0000 7f72fc999700 1 -- 172.21.15.98:0/2367511416 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f0012430 con 0x7f72f80b5410 2023-03-03T08:02:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.030+0000 7f72ffbff700 1 -- 172.21.15.98:0/2367511416 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72f80b5410 msgr2=0x7f72f80d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.030+0000 7f72ffbff700 1 --2- 172.21.15.98:0/2367511416 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72f80b5410 0x7f72f80d13d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009d90 tx=0x7f72f000b330 comp rx=0 tx=0).stop 2023-03-03T08:02:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.030+0000 7f72ffbff700 1 -- 172.21.15.98:0/2367511416 shutdown_connections 2023-03-03T08:02:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.030+0000 7f72ffbff700 1 --2- 172.21.15.98:0/2367511416 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72f80b5410 0x7f72f80d13d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.030+0000 7f72ffbff700 1 --2- 172.21.15.98:0/2367511416 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72f80b87c0 0x7f72f80cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.030+0000 7f72ffbff700 1 -- 172.21.15.98:0/2367511416 >> 172.21.15.98:0/2367511416 conn(0x7f72f806aee0 msgr2=0x7f72f806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.030+0000 7f72ffbff700 1 -- 172.21.15.98:0/2367511416 shutdown_connections 2023-03-03T08:02:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.030+0000 7f72ffbff700 1 -- 172.21.15.98:0/2367511416 wait complete. 2023-03-03T08:02:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.032+0000 7f72ffbff700 1 Processor -- start 2023-03-03T08:02:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.032+0000 7f72ffbff700 1 -- start start 2023-03-03T08:02:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.033+0000 7f72ffbff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72f80b5410 0x7f72f814e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.033+0000 7f72ffbff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72f80b87c0 0x7f72f80cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.033+0000 7f72ffbff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72f80b8b90 0x7f72f80ab0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.033+0000 7f72ffbff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80aacb0 con 0x7f72f80b8b90 2023-03-03T08:02:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.033+0000 7f72ffbff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80aa030 con 0x7f72f80b87c0 2023-03-03T08:02:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.033+0000 7f72ffbff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80aa310 con 0x7f72f80b5410 2023-03-03T08:02:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.034+0000 7f72fe19c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72f80b8b90 0x7f72f80ab0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.034+0000 7f72fe19c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72f80b8b90 0x7f72f80ab0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34276/0 (socket says 172.21.15.98:34276) 2023-03-03T08:02:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.034+0000 7f72fe19c700 1 -- 172.21.15.98:0/2191041871 learned_addr learned my addr 172.21.15.98:0/2191041871 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.034+0000 7f72fd19a700 1 --2- 172.21.15.98:0/2191041871 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72f80b87c0 0x7f72f80cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.034+0000 7f72fd99b700 1 --2- 172.21.15.98:0/2191041871 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72f80b5410 0x7f72f814e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.034+0000 7f72fe19c700 1 -- 172.21.15.98:0/2191041871 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72f80b5410 msgr2=0x7f72f814e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.034+0000 7f72fe19c700 1 --2- 172.21.15.98:0/2191041871 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72f80b5410 0x7f72f814e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.034+0000 7f72fe19c700 1 -- 172.21.15.98:0/2191041871 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72f80b87c0 msgr2=0x7f72f80cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.034+0000 7f72fe19c700 1 --2- 172.21.15.98:0/2191041871 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72f80b87c0 0x7f72f80cce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.034+0000 7f72fe19c700 1 -- 172.21.15.98:0/2191041871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f0009a70 con 0x7f72f80b8b90 2023-03-03T08:02:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.035+0000 7f72fe19c700 1 --2- 172.21.15.98:0/2191041871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72f80b8b90 0x7f72f80ab0d0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f72f80d0a60 tx=0x7f72f0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.035+0000 7f72ea7fc700 1 -- 172.21.15.98:0/2191041871 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f0012430 con 0x7f72f80b8b90 2023-03-03T08:02:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.035+0000 7f72ea7fc700 1 -- 172.21.15.98:0/2191041871 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f0020070 con 0x7f72f80b8b90 2023-03-03T08:02:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.035+0000 7f72ffbff700 1 -- 172.21.15.98:0/2191041871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f80cdf30 con 0x7f72f80b8b90 2023-03-03T08:02:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.035+0000 7f72ea7fc700 1 -- 172.21.15.98:0/2191041871 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f0004280 con 0x7f72f80b8b90 2023-03-03T08:02:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.035+0000 7f72ffbff700 1 -- 172.21.15.98:0/2191041871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f8073a80 con 0x7f72f80b8b90 2023-03-03T08:02:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.036+0000 7f72ffbff700 1 -- 172.21.15.98:0/2191041871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f80ce0c0 con 0x7f72f80b8b90 2023-03-03T08:02:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.038+0000 7f72ea7fc700 1 -- 172.21.15.98:0/2191041871 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f72f00043e0 con 0x7f72f80b8b90 2023-03-03T08:02:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.039+0000 7f72ea7fc700 1 --2- 172.21.15.98:0/2191041871 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f72dc068380 0x7f72dc06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.039+0000 7f72ea7fc700 1 -- 172.21.15.98:0/2191041871 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f72f0022070 con 0x7f72f80b8b90 2023-03-03T08:02:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.039+0000 7f72fd99b700 1 --2- 172.21.15.98:0/2191041871 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f72dc068380 0x7f72dc06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.040+0000 7f72fd99b700 1 --2- 172.21.15.98:0/2191041871 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f72dc068380 0x7f72dc06a830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009be0 tx=0x7f72f4009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.043+0000 7f72ea7fc700 1 -- 172.21.15.98:0/2191041871 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f72f0054810 con 0x7f72f80b8b90 2023-03-03T08:02:58.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.239+0000 7f72ffbff700 1 -- 172.21.15.98:0/2191041871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72f80729b0 con 0x7f72f80b8b90 2023-03-03T08:02:58.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.240+0000 7f72ea7fc700 1 -- 172.21.15.98:0/2191041871 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+18053 (secure 0 0 0) 0x7f72f0057e30 con 0x7f72f80b8b90 2023-03-03T08:02:58.242 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:02:58.242 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":98,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:02:45.624487+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:02:20.797579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-03-03T08:02:21.411899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:6835/1843752855":"2023-03-04T08:02:42.175049+0000","172.21.15.100:6833/1843752855":"2023-03-04T08:02:42.175049+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:02:58.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.243+0000 7f72ffbff700 1 -- 172.21.15.98:0/2191041871 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f72dc068380 msgr2=0x7f72dc06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:58.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.243+0000 7f72ffbff700 1 --2- 172.21.15.98:0/2191041871 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f72dc068380 0x7f72dc06a830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009be0 tx=0x7f72f4009210 comp rx=0 tx=0).stop 2023-03-03T08:02:58.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.243+0000 7f72ffbff700 1 -- 172.21.15.98:0/2191041871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72f80b8b90 msgr2=0x7f72f80ab0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:58.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.243+0000 7f72ffbff700 1 --2- 172.21.15.98:0/2191041871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72f80b8b90 0x7f72f80ab0d0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f72f80d0a60 tx=0x7f72f0004ad0 comp rx=0 tx=0).stop 2023-03-03T08:02:58.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.243+0000 7f72ffbff700 1 -- 172.21.15.98:0/2191041871 shutdown_connections 2023-03-03T08:02:58.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.243+0000 7f72ffbff700 1 --2- 172.21.15.98:0/2191041871 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f72dc068380 0x7f72dc06a830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:58.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.243+0000 7f72ffbff700 1 --2- 172.21.15.98:0/2191041871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72f80b8b90 0x7f72f80ab0d0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:58.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.243+0000 7f72ffbff700 1 --2- 172.21.15.98:0/2191041871 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72f80b5410 0x7f72f814e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:58.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.243+0000 7f72ffbff700 1 --2- 172.21.15.98:0/2191041871 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72f80b87c0 0x7f72f80cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:58.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.243+0000 7f72ffbff700 1 -- 172.21.15.98:0/2191041871 >> 172.21.15.98:0/2191041871 conn(0x7f72f806aee0 msgr2=0x7f72f80b3680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:58.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.243+0000 7f72ffbff700 1 -- 172.21.15.98:0/2191041871 shutdown_connections 2023-03-03T08:02:58.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.243+0000 7f72ffbff700 1 -- 172.21.15.98:0/2191041871 wait complete. 2023-03-03T08:02:58.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-03-03T08:02:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.515+0000 7f25435ff700 1 Processor -- start 2023-03-03T08:02:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.516+0000 7f25435ff700 1 -- start start 2023-03-03T08:02:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.517+0000 7f25435ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f253c15ca70 0x7f253c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.517+0000 7f25435ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f253c06b4b0 0x7f253c0cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.517+0000 7f25435ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f253c0cf980 con 0x7f253c0d00a0 2023-03-03T08:02:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.517+0000 7f25435ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f253c0cd5b0 con 0x7f253c06b4b0 2023-03-03T08:02:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.517+0000 7f25435ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253c0d0510 con 0x7f253c15ca70 2023-03-03T08:02:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.517+0000 7f2540b9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f253c06b4b0 0x7f253c0cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.518+0000 7f2540b9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f253c06b4b0 0x7f253c0cfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34286/0 (socket says 172.21.15.98:34286) 2023-03-03T08:02:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.518+0000 7f2540b9a700 1 -- 172.21.15.98:0/535787178 learned_addr learned my addr 172.21.15.98:0/535787178 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.518+0000 7f254139b700 1 --2- 172.21.15.98:0/535787178 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f253c15ca70 0x7f253c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.518+0000 7f2540b9a700 1 -- 172.21.15.98:0/535787178 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f253c15ca70 msgr2=0x7f253c0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.518+0000 7f2540b9a700 1 --2- 172.21.15.98:0/535787178 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f253c15ca70 0x7f253c0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.518+0000 7f2540b9a700 1 -- 172.21.15.98:0/535787178 >> v1:172.21.15.100:6789/0 conn(0x7f253c0d00a0 legacy=0x7f253c0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.518+0000 7f2540b9a700 1 -- 172.21.15.98:0/535787178 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c0d2fd0 con 0x7f253c06b4b0 2023-03-03T08:02:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.518+0000 7f2540b9a700 1 --2- 172.21.15.98:0/535787178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f253c06b4b0 0x7f253c0cfb60 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f252c0098f0 tx=0x7f252c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc14f78058b9bee2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.519+0000 7f2533fff700 1 -- 172.21.15.98:0/535787178 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252c00b8a0 con 0x7f253c06b4b0 2023-03-03T08:02:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.519+0000 7f2533fff700 1 -- 172.21.15.98:0/535787178 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f252c00ba00 con 0x7f253c06b4b0 2023-03-03T08:02:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.519+0000 7f2533fff700 1 -- 172.21.15.98:0/535787178 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252c00bd90 con 0x7f253c06b4b0 2023-03-03T08:02:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.519+0000 7f25435ff700 1 -- 172.21.15.98:0/535787178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f253c06b4b0 msgr2=0x7f253c0cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.519+0000 7f25435ff700 1 --2- 172.21.15.98:0/535787178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f253c06b4b0 0x7f253c0cfb60 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f252c0098f0 tx=0x7f252c009c00 comp rx=0 tx=0).stop 2023-03-03T08:02:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.519+0000 7f25435ff700 1 -- 172.21.15.98:0/535787178 shutdown_connections 2023-03-03T08:02:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.519+0000 7f25435ff700 1 --2- 172.21.15.98:0/535787178 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f253c15ca70 0x7f253c0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.519+0000 7f25435ff700 1 --2- 172.21.15.98:0/535787178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f253c06b4b0 0x7f253c0cfb60 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.519+0000 7f25435ff700 1 -- 172.21.15.98:0/535787178 >> 172.21.15.98:0/535787178 conn(0x7f253c073e70 msgr2=0x7f253c06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.519+0000 7f25435ff700 1 -- 172.21.15.98:0/535787178 shutdown_connections 2023-03-03T08:02:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.520+0000 7f25435ff700 1 -- 172.21.15.98:0/535787178 wait complete. 2023-03-03T08:02:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.521+0000 7f25435ff700 1 Processor -- start 2023-03-03T08:02:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.522+0000 7f25435ff700 1 -- start start 2023-03-03T08:02:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f25435ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f253c06b4b0 0x7f253c15f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f25435ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f253c0d00a0 0x7f253c15f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f25435ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f253c15ca70 0x7f253c15ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f25435ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c15af40 con 0x7f253c06b4b0 2023-03-03T08:02:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f25435ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c15db00 con 0x7f253c0d00a0 2023-03-03T08:02:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f25435ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253c15dde0 con 0x7f253c15ca70 2023-03-03T08:02:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f254139b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f253c06b4b0 0x7f253c15f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f254139b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f253c06b4b0 0x7f253c15f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34294/0 (socket says 172.21.15.98:34294) 2023-03-03T08:02:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f254139b700 1 -- 172.21.15.98:0/1285027790 learned_addr learned my addr 172.21.15.98:0/1285027790 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f2540b9a700 1 --2- 172.21.15.98:0/1285027790 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f253c0d00a0 0x7f253c15f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f2541b9c700 1 --2- 172.21.15.98:0/1285027790 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f253c15ca70 0x7f253c15ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f254139b700 1 -- 172.21.15.98:0/1285027790 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f253c15ca70 msgr2=0x7f253c15ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f254139b700 1 --2- 172.21.15.98:0/1285027790 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f253c15ca70 0x7f253c15ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f254139b700 1 -- 172.21.15.98:0/1285027790 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f253c0d00a0 msgr2=0x7f253c15f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f254139b700 1 --2- 172.21.15.98:0/1285027790 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f253c0d00a0 0x7f253c15f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f254139b700 1 -- 172.21.15.98:0/1285027790 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252c009550 con 0x7f253c06b4b0 2023-03-03T08:02:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.523+0000 7f2540b9a700 1 --2- 172.21.15.98:0/1285027790 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f253c0d00a0 0x7f253c15f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:58.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.524+0000 7f254139b700 1 --2- 172.21.15.98:0/1285027790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f253c06b4b0 0x7f253c15f230 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f253800d800 tx=0x7f253800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:58.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.524+0000 7f2531ffb700 1 -- 172.21.15.98:0/1285027790 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25380125f0 con 0x7f253c06b4b0 2023-03-03T08:02:58.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.524+0000 7f2531ffb700 1 -- 172.21.15.98:0/1285027790 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2538012750 con 0x7f253c06b4b0 2023-03-03T08:02:58.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.524+0000 7f2531ffb700 1 -- 172.21.15.98:0/1285027790 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538012b00 con 0x7f253c06b4b0 2023-03-03T08:02:58.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.524+0000 7f25435ff700 1 -- 172.21.15.98:0/1285027790 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253c0cd900 con 0x7f253c06b4b0 2023-03-03T08:02:58.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.524+0000 7f25435ff700 1 -- 172.21.15.98:0/1285027790 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253c0d1800 con 0x7f253c06b4b0 2023-03-03T08:02:58.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.525+0000 7f25435ff700 1 -- 172.21.15.98:0/1285027790 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f253c0cdbe0 con 0x7f253c06b4b0 2023-03-03T08:02:58.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.527+0000 7f2531ffb700 1 -- 172.21.15.98:0/1285027790 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f2538012c60 con 0x7f253c06b4b0 2023-03-03T08:02:58.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.528+0000 7f2531ffb700 1 --2- 172.21.15.98:0/1285027790 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2520068380 0x7f252006a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:58.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.528+0000 7f2531ffb700 1 -- 172.21.15.98:0/1285027790 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2538086390 con 0x7f253c06b4b0 2023-03-03T08:02:58.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.529+0000 7f2540b9a700 1 --2- 172.21.15.98:0/1285027790 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2520068380 0x7f252006a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:58.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.529+0000 7f2540b9a700 1 --2- 172.21.15.98:0/1285027790 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2520068380 0x7f252006a830 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f252c0098f0 tx=0x7f252c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:58.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.532+0000 7f2531ffb700 1 -- 172.21.15.98:0/1285027790 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2538052ab0 con 0x7f253c06b4b0 2023-03-03T08:02:58.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:58.772+0000 7f25435ff700 1 -- 172.21.15.98:0/1285027790 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f253c04e6e0 con 0x7f253c06b4b0 2023-03-03T08:02:59.693 INFO:tasks.ceph.mon.a.smithi098.stderr:2023-03-03T08:02:59.692+0000 7fb71abf9700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-03-03T08:02:59.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.700+0000 7f2531ffb700 1 -- 172.21.15.98:0/1285027790 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v55) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2538084810 con 0x7f253c06b4b0 2023-03-03T08:02:59.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.703+0000 7f25435ff700 1 -- 172.21.15.98:0/1285027790 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2520068380 msgr2=0x7f252006a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:59.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.703+0000 7f25435ff700 1 --2- 172.21.15.98:0/1285027790 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2520068380 0x7f252006a830 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f252c0098f0 tx=0x7f252c008030 comp rx=0 tx=0).stop 2023-03-03T08:02:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.703+0000 7f25435ff700 1 -- 172.21.15.98:0/1285027790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f253c06b4b0 msgr2=0x7f253c15f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.703+0000 7f25435ff700 1 --2- 172.21.15.98:0/1285027790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f253c06b4b0 0x7f253c15f230 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f253800d800 tx=0x7f253800dbc0 comp rx=0 tx=0).stop 2023-03-03T08:02:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.703+0000 7f25435ff700 1 -- 172.21.15.98:0/1285027790 shutdown_connections 2023-03-03T08:02:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.703+0000 7f25435ff700 1 --2- 172.21.15.98:0/1285027790 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2520068380 0x7f252006a830 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.703+0000 7f25435ff700 1 --2- 172.21.15.98:0/1285027790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f253c06b4b0 0x7f253c15f230 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.703+0000 7f25435ff700 1 --2- 172.21.15.98:0/1285027790 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f253c15ca70 0x7f253c15ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:59.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.703+0000 7f25435ff700 1 --2- 172.21.15.98:0/1285027790 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f253c0d00a0 0x7f253c15f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:59.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.703+0000 7f25435ff700 1 -- 172.21.15.98:0/1285027790 >> 172.21.15.98:0/1285027790 conn(0x7f253c073e70 msgr2=0x7f253c0d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:59.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.703+0000 7f25435ff700 1 -- 172.21.15.98:0/1285027790 shutdown_connections 2023-03-03T08:02:59.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.703+0000 7f25435ff700 1 -- 172.21.15.98:0/1285027790 wait complete. 2023-03-03T08:02:59.706 INFO:teuthology.orchestra.run.smithi098.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-03-03T08:02:59.709 INFO:tasks.ceph.mds.c.smithi098.stderr: -1677> 2023-03-03T08:02:26.715+0000 7fe26a3f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T08:02:59.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-03-03T08:02:59.773 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-03-03T08:02:59.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.985+0000 7fdb66fff700 1 Processor -- start 2023-03-03T08:02:59.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.985+0000 7fdb66fff700 1 -- start start 2023-03-03T08:02:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.986+0000 7fdb66fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdb6015ca70 0x7fdb600cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.986+0000 7fdb66fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdb600d0270 0x7fdb600d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.986+0000 7fdb66fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb600cf980 con 0x7fdb6006b4b0 2023-03-03T08:02:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.987+0000 7fdb66fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb600cd5b0 con 0x7fdb6015ca70 2023-03-03T08:02:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.987+0000 7fdb66fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb600d1800 con 0x7fdb600d0270 2023-03-03T08:02:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.987+0000 7fdb64d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdb6015ca70 0x7fdb600cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.987+0000 7fdb64d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdb6015ca70 0x7fdb600cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34316/0 (socket says 172.21.15.98:34316) 2023-03-03T08:02:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.987+0000 7fdb64d9b700 1 -- 172.21.15.98:0/2935401900 learned_addr learned my addr 172.21.15.98:0/2935401900 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.987+0000 7fdb6559c700 1 --2- 172.21.15.98:0/2935401900 >> v2:172.21.15.100:3300/0 conn(0x7fdb600d0270 0x7fdb600d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.987+0000 7fdb64d9b700 1 -- 172.21.15.98:0/2935401900 >> v2:172.21.15.100:3300/0 conn(0x7fdb600d0270 msgr2=0x7fdb600d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.987+0000 7fdb64d9b700 1 --2- 172.21.15.98:0/2935401900 >> v2:172.21.15.100:3300/0 conn(0x7fdb600d0270 0x7fdb600d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.987+0000 7fdb64d9b700 1 -- 172.21.15.98:0/2935401900 >> v1:172.21.15.100:6789/0 conn(0x7fdb6006b4b0 legacy=0x7fdb600cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.987+0000 7fdb64d9b700 1 -- 172.21.15.98:0/2935401900 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb600cd900 con 0x7fdb6015ca70 2023-03-03T08:02:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.988+0000 7fdb64d9b700 1 --2- 172.21.15.98:0/2935401900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb6015ca70 0x7fdb600cf440 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fdb600ca200 tx=0x7fdb54009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591cac1d6e5e3b08 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.988+0000 7fdb5f7fe700 1 -- 172.21.15.98:0/2935401900 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb54011830 con 0x7fdb6015ca70 2023-03-03T08:02:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.988+0000 7fdb5f7fe700 1 -- 172.21.15.98:0/2935401900 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb54011990 con 0x7fdb6015ca70 2023-03-03T08:02:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.988+0000 7fdb5f7fe700 1 -- 172.21.15.98:0/2935401900 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb54011d20 con 0x7fdb6015ca70 2023-03-03T08:02:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.988+0000 7fdb66fff700 1 -- 172.21.15.98:0/2935401900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb6015ca70 msgr2=0x7fdb600cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.988+0000 7fdb66fff700 1 --2- 172.21.15.98:0/2935401900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb6015ca70 0x7fdb600cf440 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fdb600ca200 tx=0x7fdb54009bd0 comp rx=0 tx=0).stop 2023-03-03T08:02:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.989+0000 7fdb66fff700 1 -- 172.21.15.98:0/2935401900 shutdown_connections 2023-03-03T08:02:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.989+0000 7fdb66fff700 1 --2- 172.21.15.98:0/2935401900 >> v2:172.21.15.100:3300/0 conn(0x7fdb600d0270 0x7fdb600d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.989+0000 7fdb66fff700 1 --2- 172.21.15.98:0/2935401900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb6015ca70 0x7fdb600cf440 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.989+0000 7fdb66fff700 1 -- 172.21.15.98:0/2935401900 >> 172.21.15.98:0/2935401900 conn(0x7fdb60073e70 msgr2=0x7fdb6006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.989+0000 7fdb66fff700 1 -- 172.21.15.98:0/2935401900 shutdown_connections 2023-03-03T08:02:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.989+0000 7fdb66fff700 1 -- 172.21.15.98:0/2935401900 wait complete. 2023-03-03T08:02:59.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.991+0000 7fdb66fff700 1 Processor -- start 2023-03-03T08:02:59.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.991+0000 7fdb66fff700 1 -- start start 2023-03-03T08:02:59.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.992+0000 7fdb66fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb6006b4b0 0x7fdb600cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:59.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.992+0000 7fdb66fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb600d0270 0x7fdb60156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.992+0000 7fdb66fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb6015ca70 0x7fdb6015f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.992+0000 7fdb66fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb601f3300 con 0x7fdb6015ca70 2023-03-03T08:02:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.992+0000 7fdb66fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb601f23f0 con 0x7fdb6006b4b0 2023-03-03T08:02:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.992+0000 7fdb66fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb601f2680 con 0x7fdb600d0270 2023-03-03T08:02:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.992+0000 7fdb6559c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb6015ca70 0x7fdb6015f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.992+0000 7fdb6559c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb6015ca70 0x7fdb6015f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34330/0 (socket says 172.21.15.98:34330) 2023-03-03T08:02:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.992+0000 7fdb6559c700 1 -- 172.21.15.98:0/2859863140 learned_addr learned my addr 172.21.15.98:0/2859863140 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:02:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.992+0000 7fdb5ffff700 1 --2- 172.21.15.98:0/2859863140 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb600d0270 0x7fdb60156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.993+0000 7fdb64d9b700 1 --2- 172.21.15.98:0/2859863140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb6006b4b0 0x7fdb600cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.993+0000 7fdb6559c700 1 -- 172.21.15.98:0/2859863140 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb600d0270 msgr2=0x7fdb60156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.993+0000 7fdb6559c700 1 --2- 172.21.15.98:0/2859863140 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb600d0270 0x7fdb60156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.993+0000 7fdb6559c700 1 -- 172.21.15.98:0/2859863140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb6006b4b0 msgr2=0x7fdb600cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.993+0000 7fdb6559c700 1 --2- 172.21.15.98:0/2859863140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb6006b4b0 0x7fdb600cb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.993+0000 7fdb6559c700 1 -- 172.21.15.98:0/2859863140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb54009550 con 0x7fdb6015ca70 2023-03-03T08:02:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.993+0000 7fdb6559c700 1 --2- 172.21.15.98:0/2859863140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb6015ca70 0x7fdb6015f820 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fdb5800ef90 tx=0x7fdb58009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.993+0000 7fdb5d7fa700 1 -- 172.21.15.98:0/2859863140 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb58009b20 con 0x7fdb6015ca70 2023-03-03T08:02:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.994+0000 7fdb5d7fa700 1 -- 172.21.15.98:0/2859863140 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5800f040 con 0x7fdb6015ca70 2023-03-03T08:02:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.994+0000 7fdb5d7fa700 1 -- 172.21.15.98:0/2859863140 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5800cc70 con 0x7fdb6015ca70 2023-03-03T08:02:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.994+0000 7fdb66fff700 1 -- 172.21.15.98:0/2859863140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb600d1aa0 con 0x7fdb6015ca70 2023-03-03T08:02:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.994+0000 7fdb66fff700 1 -- 172.21.15.98:0/2859863140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb600cdbe0 con 0x7fdb6015ca70 2023-03-03T08:02:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.995+0000 7fdb66fff700 1 -- 172.21.15.98:0/2859863140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6015ce40 con 0x7fdb6015ca70 2023-03-03T08:02:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.996+0000 7fdb5d7fa700 1 -- 172.21.15.98:0/2859863140 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fdb58014480 con 0x7fdb6015ca70 2023-03-03T08:02:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.997+0000 7fdb5d7fa700 1 --2- 172.21.15.98:0/2859863140 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdb44068380 0x7fdb4406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.998+0000 7fdb5d7fa700 1 -- 172.21.15.98:0/2859863140 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6234+0+0 (secure 0 0 0) 0x7fdb58086920 con 0x7fdb6015ca70 2023-03-03T08:03:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.998+0000 7fdb64d9b700 1 --2- 172.21.15.98:0/2859863140 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdb44068380 0x7fdb4406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:02:59.999+0000 7fdb64d9b700 1 --2- 172.21.15.98:0/2859863140 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdb44068380 0x7fdb4406a830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009c10 tx=0x7fdb54008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:00.002+0000 7fdb5d7fa700 1 -- 172.21.15.98:0/2859863140 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdb58052f90 con 0x7fdb6015ca70 2023-03-03T08:03:00.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:00.237+0000 7fdb66fff700 1 -- 172.21.15.98:0/2859863140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdb6004e6e0 con 0x7fdb6015ca70 2023-03-03T08:03:00.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:00.702+0000 7fdb5d7fa700 1 -- 172.21.15.98:0/2859863140 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdb580180a0 con 0x7fdb6015ca70 2023-03-03T08:03:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:00.705+0000 7fdb66fff700 1 -- 172.21.15.98:0/2859863140 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdb44068380 msgr2=0x7fdb4406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:00.705+0000 7fdb66fff700 1 --2- 172.21.15.98:0/2859863140 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdb44068380 0x7fdb4406a830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009c10 tx=0x7fdb54008030 comp rx=0 tx=0).stop 2023-03-03T08:03:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:00.705+0000 7fdb66fff700 1 -- 172.21.15.98:0/2859863140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb6015ca70 msgr2=0x7fdb6015f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:00.705+0000 7fdb66fff700 1 --2- 172.21.15.98:0/2859863140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb6015ca70 0x7fdb6015f820 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fdb5800ef90 tx=0x7fdb58009f60 comp rx=0 tx=0).stop 2023-03-03T08:03:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:00.705+0000 7fdb66fff700 1 -- 172.21.15.98:0/2859863140 shutdown_connections 2023-03-03T08:03:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:00.705+0000 7fdb66fff700 1 --2- 172.21.15.98:0/2859863140 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fdb44068380 0x7fdb4406a830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:00.705+0000 7fdb66fff700 1 --2- 172.21.15.98:0/2859863140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb6015ca70 0x7fdb6015f820 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:00.705+0000 7fdb66fff700 1 --2- 172.21.15.98:0/2859863140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb6006b4b0 0x7fdb600cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:00.705+0000 7fdb66fff700 1 --2- 172.21.15.98:0/2859863140 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb600d0270 0x7fdb60156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:00.706+0000 7fdb66fff700 1 -- 172.21.15.98:0/2859863140 >> 172.21.15.98:0/2859863140 conn(0x7fdb60073e70 msgr2=0x7fdb6015ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:00.706+0000 7fdb66fff700 1 -- 172.21.15.98:0/2859863140 shutdown_connections 2023-03-03T08:03:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:00.706+0000 7fdb66fff700 1 -- 172.21.15.98:0/2859863140 wait complete. 2023-03-03T08:03:00.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-03-03T08:03:01.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.027+0000 7fea3f19d700 1 Processor -- start 2023-03-03T08:03:01.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.028+0000 7fea3f19d700 1 -- start start 2023-03-03T08:03:01.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.029+0000 7fea3f19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea380d25e0 0x7fea380cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:01.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.029+0000 7fea3f19d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fea3806b750 0x7fea380b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:01.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.029+0000 7fea3f19d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea38160ef0 con 0x7fea380b4f60 2023-03-03T08:03:01.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.029+0000 7fea3f19d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea380b53e0 con 0x7fea3806b750 2023-03-03T08:03:01.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.029+0000 7fea3f19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea38161420 con 0x7fea380d25e0 2023-03-03T08:03:01.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.030+0000 7fea3d99a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fea3806b750 0x7fea380b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:01.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.030+0000 7fea3d99a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fea3806b750 0x7fea380b4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34338/0 (socket says 172.21.15.98:34338) 2023-03-03T08:03:01.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.030+0000 7fea3d99a700 1 -- 172.21.15.98:0/3053331061 learned_addr learned my addr 172.21.15.98:0/3053331061 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:01.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.030+0000 7fea3e19b700 1 --2- 172.21.15.98:0/3053331061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea380d25e0 0x7fea380cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:01.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.030+0000 7fea3d99a700 1 -- 172.21.15.98:0/3053331061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea380d25e0 msgr2=0x7fea380cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:01.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.030+0000 7fea3d99a700 1 --2- 172.21.15.98:0/3053331061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea380d25e0 0x7fea380cf2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:01.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.030+0000 7fea3d99a700 1 -- 172.21.15.98:0/3053331061 >> v1:172.21.15.100:6789/0 conn(0x7fea380b4f60 legacy=0x7fea380731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:01.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.030+0000 7fea3d99a700 1 -- 172.21.15.98:0/3053331061 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea38161140 con 0x7fea3806b750 2023-03-03T08:03:01.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.031+0000 7fea3d99a700 1 --2- 172.21.15.98:0/3053331061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea3806b750 0x7fea380b4a20 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fea28009920 tx=0x7fea28009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25dc4426a8b34d0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:01.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.031+0000 7fea3d199700 1 -- 172.21.15.98:0/3053331061 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28011830 con 0x7fea3806b750 2023-03-03T08:03:01.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.031+0000 7fea3d199700 1 -- 172.21.15.98:0/3053331061 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea28011990 con 0x7fea3806b750 2023-03-03T08:03:01.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.031+0000 7fea3d199700 1 -- 172.21.15.98:0/3053331061 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28011d20 con 0x7fea3806b750 2023-03-03T08:03:01.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.031+0000 7fea3f19d700 1 -- 172.21.15.98:0/3053331061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea3806b750 msgr2=0x7fea380b4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:01.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.031+0000 7fea3f19d700 1 --2- 172.21.15.98:0/3053331061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea3806b750 0x7fea380b4a20 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fea28009920 tx=0x7fea28009c30 comp rx=0 tx=0).stop 2023-03-03T08:03:01.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.032+0000 7fea3f19d700 1 -- 172.21.15.98:0/3053331061 shutdown_connections 2023-03-03T08:03:01.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.032+0000 7fea3f19d700 1 --2- 172.21.15.98:0/3053331061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea380d25e0 0x7fea380cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:01.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.032+0000 7fea3f19d700 1 --2- 172.21.15.98:0/3053331061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea3806b750 0x7fea380b4a20 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:01.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.032+0000 7fea3f19d700 1 -- 172.21.15.98:0/3053331061 >> 172.21.15.98:0/3053331061 conn(0x7fea380d3be0 msgr2=0x7fea3806ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:01.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.032+0000 7fea3f19d700 1 -- 172.21.15.98:0/3053331061 shutdown_connections 2023-03-03T08:03:01.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.032+0000 7fea3f19d700 1 -- 172.21.15.98:0/3053331061 wait complete. 2023-03-03T08:03:01.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.034+0000 7fea3f19d700 1 Processor -- start 2023-03-03T08:03:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.034+0000 7fea3f19d700 1 -- start start 2023-03-03T08:03:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.035+0000 7fea3f19d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea3806b750 0x7fea381f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.035+0000 7fea3f19d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fea380b4f60 0x7fea380b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.035+0000 7fea3f19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea380d25e0 0x7fea380b4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.035+0000 7fea3f19d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea381f2150 con 0x7fea3806b750 2023-03-03T08:03:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.035+0000 7fea3f19d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea380c4690 con 0x7fea380b4f60 2023-03-03T08:03:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.035+0000 7fea3f19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea380c4970 con 0x7fea380d25e0 2023-03-03T08:03:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.035+0000 7fea3e19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea3806b750 0x7fea381f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.035+0000 7fea3e19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea3806b750 0x7fea381f1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34352/0 (socket says 172.21.15.98:34352) 2023-03-03T08:03:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.035+0000 7fea3e19b700 1 -- 172.21.15.98:0/2250012650 learned_addr learned my addr 172.21.15.98:0/2250012650 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.035+0000 7fea3d99a700 1 --2- 172.21.15.98:0/2250012650 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fea380b4f60 0x7fea380b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.035+0000 7fea3e99c700 1 --2- 172.21.15.98:0/2250012650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea380d25e0 0x7fea380b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.036+0000 7fea3e19b700 1 -- 172.21.15.98:0/2250012650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea380d25e0 msgr2=0x7fea380b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:01.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.036+0000 7fea3e19b700 1 --2- 172.21.15.98:0/2250012650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea380d25e0 0x7fea380b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:01.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.036+0000 7fea3e19b700 1 -- 172.21.15.98:0/2250012650 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fea380b4f60 msgr2=0x7fea380b3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:01.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.036+0000 7fea3e19b700 1 --2- 172.21.15.98:0/2250012650 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fea380b4f60 0x7fea380b3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:01.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.036+0000 7fea3e19b700 1 -- 172.21.15.98:0/2250012650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea28009550 con 0x7fea3806b750 2023-03-03T08:03:01.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.036+0000 7fea3e19b700 1 --2- 172.21.15.98:0/2250012650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea3806b750 0x7fea381f1c10 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fea3400e870 tx=0x7fea3400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:01.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.036+0000 7fea2effd700 1 -- 172.21.15.98:0/2250012650 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea34009590 con 0x7fea3806b750 2023-03-03T08:03:01.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.036+0000 7fea2effd700 1 -- 172.21.15.98:0/2250012650 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea34013680 con 0x7fea3806b750 2023-03-03T08:03:01.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.036+0000 7fea3f19d700 1 -- 172.21.15.98:0/2250012650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea380cda00 con 0x7fea3806b750 2023-03-03T08:03:01.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.036+0000 7fea2effd700 1 -- 172.21.15.98:0/2250012650 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea340139f0 con 0x7fea3806b750 2023-03-03T08:03:01.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.036+0000 7fea3f19d700 1 -- 172.21.15.98:0/2250012650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea380cd720 con 0x7fea3806b750 2023-03-03T08:03:01.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.038+0000 7fea3f19d700 1 -- 172.21.15.98:0/2250012650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea381466a0 con 0x7fea3806b750 2023-03-03T08:03:01.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.039+0000 7fea2effd700 1 -- 172.21.15.98:0/2250012650 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fea34019030 con 0x7fea3806b750 2023-03-03T08:03:01.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.040+0000 7fea2effd700 1 --2- 172.21.15.98:0/2250012650 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fea1c068380 0x7fea1c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:01.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.040+0000 7fea2effd700 1 -- 172.21.15.98:0/2250012650 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6350+0+0 (secure 0 0 0) 0x7fea34085c60 con 0x7fea3806b750 2023-03-03T08:03:01.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.040+0000 7fea3d99a700 1 --2- 172.21.15.98:0/2250012650 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fea1c068380 0x7fea1c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:01.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.041+0000 7fea3d99a700 1 --2- 172.21.15.98:0/2250012650 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fea1c068380 0x7fea1c06a830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fea28014040 tx=0x7fea28008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:01.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.044+0000 7fea2effd700 1 -- 172.21.15.98:0/2250012650 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fea340522c0 con 0x7fea3806b750 2023-03-03T08:03:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.241+0000 7fea3f19d700 1 -- 172.21.15.98:0/2250012650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fea38068290 con 0x7fea3806b750 2023-03-03T08:03:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.797+0000 7fea2effd700 1 -- 172.21.15.98:0/2250012650 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v101) v1 ==== 181+0+0 (secure 0 0 0) 0x7fea340558e0 con 0x7fea3806b750 2023-03-03T08:03:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.801+0000 7fea3f19d700 1 -- 172.21.15.98:0/2250012650 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fea1c068380 msgr2=0x7fea1c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.801+0000 7fea3f19d700 1 --2- 172.21.15.98:0/2250012650 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fea1c068380 0x7fea1c06a830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fea28014040 tx=0x7fea28008030 comp rx=0 tx=0).stop 2023-03-03T08:03:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.801+0000 7fea3f19d700 1 -- 172.21.15.98:0/2250012650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea3806b750 msgr2=0x7fea381f1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.801+0000 7fea3f19d700 1 --2- 172.21.15.98:0/2250012650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea3806b750 0x7fea381f1c10 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fea3400e870 tx=0x7fea3400eb80 comp rx=0 tx=0).stop 2023-03-03T08:03:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.801+0000 7fea3f19d700 1 -- 172.21.15.98:0/2250012650 shutdown_connections 2023-03-03T08:03:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.801+0000 7fea3f19d700 1 --2- 172.21.15.98:0/2250012650 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fea1c068380 0x7fea1c06a830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.801+0000 7fea3f19d700 1 --2- 172.21.15.98:0/2250012650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea3806b750 0x7fea381f1c10 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.801+0000 7fea3f19d700 1 --2- 172.21.15.98:0/2250012650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea380d25e0 0x7fea380b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.801+0000 7fea3f19d700 1 --2- 172.21.15.98:0/2250012650 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fea380b4f60 0x7fea380b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.801+0000 7fea3f19d700 1 -- 172.21.15.98:0/2250012650 >> 172.21.15.98:0/2250012650 conn(0x7fea380d3be0 msgr2=0x7fea380d02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.801+0000 7fea3f19d700 1 -- 172.21.15.98:0/2250012650 shutdown_connections 2023-03-03T08:03:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:01.802+0000 7fea3f19d700 1 -- 172.21.15.98:0/2250012650 wait complete. 2023-03-03T08:03:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' removed 2023-03-03T08:03:01.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-03-03T08:03:02.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.084+0000 7f98ce99d700 1 Processor -- start 2023-03-03T08:03:02.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.085+0000 7f98ce99d700 1 -- start start 2023-03-03T08:03:02.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.086+0000 7f98ce99d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f98c80b3550 0x7f98c80cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:02.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.086+0000 7f98ce99d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f98c80cee50 0x7f98c80cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:02.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.086+0000 7f98ce99d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c80b3c80 con 0x7f98c8073ed0 2023-03-03T08:03:02.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.086+0000 7f98ce99d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c80b4370 con 0x7f98c80b3550 2023-03-03T08:03:02.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.086+0000 7f98ce99d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c80b3ed0 con 0x7f98c80cee50 2023-03-03T08:03:02.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.086+0000 7f98cd99b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f98c80b3550 0x7f98c80cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:02.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.086+0000 7f98cd19a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f98c8073ed0 0x7f98c8160780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38656/0 (socket says 172.21.15.98:38656) 2023-03-03T08:03:02.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.087+0000 7f98cd19a700 1 -- 172.21.15.98:0/552964976 learned_addr learned my addr 172.21.15.98:0/552964976 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:02.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.087+0000 7f98ce19c700 1 --2- 172.21.15.98:0/552964976 >> v2:172.21.15.100:3300/0 conn(0x7f98c80cee50 0x7f98c80cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:02.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.087+0000 7f98cd99b700 1 -- 172.21.15.98:0/552964976 >> v2:172.21.15.100:3300/0 conn(0x7f98c80cee50 msgr2=0x7f98c80cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:02.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.087+0000 7f98cd99b700 1 --2- 172.21.15.98:0/552964976 >> v2:172.21.15.100:3300/0 conn(0x7f98c80cee50 0x7f98c80cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:02.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.087+0000 7f98cd99b700 1 -- 172.21.15.98:0/552964976 >> v1:172.21.15.98:6789/0 conn(0x7f98c8073ed0 legacy=0x7f98c8160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:02.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.087+0000 7f98cd99b700 1 -- 172.21.15.98:0/552964976 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c80cd640 con 0x7f98c80b3550 2023-03-03T08:03:02.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.087+0000 7f98cd99b700 1 --2- 172.21.15.98:0/552964976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c80b3550 0x7f98c80cfa00 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f98c8161820 tx=0x7f98c4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7eec55fba07a58d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:02.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.088+0000 7f98cc999700 1 -- 172.21.15.98:0/552964976 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4011830 con 0x7f98c80b3550 2023-03-03T08:03:02.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.088+0000 7f98cc999700 1 -- 172.21.15.98:0/552964976 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c4011990 con 0x7f98c80b3550 2023-03-03T08:03:02.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.088+0000 7f98cc999700 1 -- 172.21.15.98:0/552964976 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4011d20 con 0x7f98c80b3550 2023-03-03T08:03:02.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.088+0000 7f98ce99d700 1 -- 172.21.15.98:0/552964976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c80b3550 msgr2=0x7f98c80cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:02.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.088+0000 7f98ce99d700 1 --2- 172.21.15.98:0/552964976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c80b3550 0x7f98c80cfa00 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f98c8161820 tx=0x7f98c4009b20 comp rx=0 tx=0).stop 2023-03-03T08:03:02.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.088+0000 7f98ce99d700 1 -- 172.21.15.98:0/552964976 shutdown_connections 2023-03-03T08:03:02.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.088+0000 7f98ce99d700 1 --2- 172.21.15.98:0/552964976 >> v2:172.21.15.100:3300/0 conn(0x7f98c80cee50 0x7f98c80cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:02.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.088+0000 7f98ce99d700 1 --2- 172.21.15.98:0/552964976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c80b3550 0x7f98c80cfa00 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:02.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.088+0000 7f98ce99d700 1 -- 172.21.15.98:0/552964976 >> 172.21.15.98:0/552964976 conn(0x7f98c8073b00 msgr2=0x7f98c80d2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:02.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.088+0000 7f98ce99d700 1 -- 172.21.15.98:0/552964976 shutdown_connections 2023-03-03T08:03:02.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.088+0000 7f98ce99d700 1 -- 172.21.15.98:0/552964976 wait complete. 2023-03-03T08:03:02.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.090+0000 7f98ce99d700 1 Processor -- start 2023-03-03T08:03:02.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.091+0000 7f98ce99d700 1 -- start start 2023-03-03T08:03:02.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98ce99d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c8073ed0 0x7f98c81f6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:02.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98ce99d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98c80b3550 0x7f98c80b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:02.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98ce99d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98c80cee50 0x7f98c80cc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:02.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98ce99d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c81f6070 con 0x7f98c8073ed0 2023-03-03T08:03:02.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98ce99d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c8161070 con 0x7f98c80b3550 2023-03-03T08:03:02.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98ce99d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c81f5a60 con 0x7f98c80cee50 2023-03-03T08:03:02.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98cd99b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c8073ed0 0x7f98c81f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:02.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98cd99b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c8073ed0 0x7f98c81f6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34372/0 (socket says 172.21.15.98:34372) 2023-03-03T08:03:02.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98cd99b700 1 -- 172.21.15.98:0/112959617 learned_addr learned my addr 172.21.15.98:0/112959617 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:02.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98ce19c700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98c80cee50 0x7f98c80cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:02.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98cd19a700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98c80b3550 0x7f98c80b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:02.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98cd99b700 1 -- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98c80cee50 msgr2=0x7f98c80cc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:02.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98cd99b700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98c80cee50 0x7f98c80cc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:02.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98cd99b700 1 -- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98c80b3550 msgr2=0x7f98c80b0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:02.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98cd99b700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98c80b3550 0x7f98c80b0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:02.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98cd99b700 1 -- 172.21.15.98:0/112959617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c40094a0 con 0x7f98c8073ed0 2023-03-03T08:03:02.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98cd19a700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98c80b3550 0x7f98c80b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.092+0000 7f98ce19c700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98c80cee50 0x7f98c80cc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.093+0000 7f98cd99b700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c8073ed0 0x7f98c81f6500 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009b60 tx=0x7f98c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.093+0000 7f98be7fc700 1 -- 172.21.15.98:0/112959617 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c40036a0 con 0x7f98c8073ed0 2023-03-03T08:03:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.093+0000 7f98be7fc700 1 -- 172.21.15.98:0/112959617 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c401f070 con 0x7f98c8073ed0 2023-03-03T08:03:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.093+0000 7f98ce99d700 1 -- 172.21.15.98:0/112959617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c80d3d60 con 0x7f98c8073ed0 2023-03-03T08:03:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.093+0000 7f98be7fc700 1 -- 172.21.15.98:0/112959617 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4003a60 con 0x7f98c8073ed0 2023-03-03T08:03:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.093+0000 7f98ce99d700 1 -- 172.21.15.98:0/112959617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c80cd920 con 0x7f98c8073ed0 2023-03-03T08:03:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.094+0000 7f98ce99d700 1 -- 172.21.15.98:0/112959617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c80d3ef0 con 0x7f98c8073ed0 2023-03-03T08:03:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.096+0000 7f98be7fc700 1 -- 172.21.15.98:0/112959617 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f98c400f050 con 0x7f98c8073ed0 2023-03-03T08:03:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.097+0000 7f98be7fc700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98ac068380 0x7f98ac06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.097+0000 7f98be7fc700 1 -- 172.21.15.98:0/112959617 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f98c4021070 con 0x7f98c8073ed0 2023-03-03T08:03:02.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.098+0000 7f98cd19a700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98ac068380 0x7f98ac06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:02.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.098+0000 7f98cd19a700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98ac068380 0x7f98ac06a830 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f98b8007fd0 tx=0x7f98b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:02.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.101+0000 7f98be7fc700 1 -- 172.21.15.98:0/112959617 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f98c80d3ef0 con 0x7f98c8073ed0 2023-03-03T08:03:02.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.291+0000 7f98ce99d700 1 -- 172.21.15.98:0/112959617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f98c80d3ef0 con 0x7f98c8073ed0 2023-03-03T08:03:02.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.808+0000 7f98be7fc700 1 -- 172.21.15.98:0/112959617 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v102) v1 ==== 169+0+0 (secure 0 0 0) 0x7f98c80d3ef0 con 0x7f98c8073ed0 2023-03-03T08:03:02.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.811+0000 7f98ce99d700 1 -- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98ac068380 msgr2=0x7f98ac06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:02.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.811+0000 7f98ce99d700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98ac068380 0x7f98ac06a830 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f98b8007fd0 tx=0x7f98b8009040 comp rx=0 tx=0).stop 2023-03-03T08:03:02.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.811+0000 7f98ce99d700 1 -- 172.21.15.98:0/112959617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c8073ed0 msgr2=0x7f98c81f6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:02.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.811+0000 7f98ce99d700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c8073ed0 0x7f98c81f6500 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009b60 tx=0x7f98c4004750 comp rx=0 tx=0).stop 2023-03-03T08:03:02.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.811+0000 7f98ce99d700 1 -- 172.21.15.98:0/112959617 shutdown_connections 2023-03-03T08:03:02.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.811+0000 7f98ce99d700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98ac068380 0x7f98ac06a830 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:02.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.811+0000 7f98ce99d700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c8073ed0 0x7f98c81f6500 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:02.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.811+0000 7f98ce99d700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98c80cee50 0x7f98c80cc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:02.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.811+0000 7f98ce99d700 1 --2- 172.21.15.98:0/112959617 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98c80b3550 0x7f98c80b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:02.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.811+0000 7f98ce99d700 1 -- 172.21.15.98:0/112959617 >> 172.21.15.98:0/112959617 conn(0x7f98c8073b00 msgr2=0x7f98c80d1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:02.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.811+0000 7f98ce99d700 1 -- 172.21.15.98:0/112959617 shutdown_connections 2023-03-03T08:03:02.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:02.811+0000 7f98ce99d700 1 -- 172.21.15.98:0/112959617 wait complete. 2023-03-03T08:03:02.815 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' removed 2023-03-03T08:03:02.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-03-03T08:03:03.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.087+0000 7f2ab4bff700 1 Processor -- start 2023-03-03T08:03:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.087+0000 7f2ab4bff700 1 -- start start 2023-03-03T08:03:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.089+0000 7f2ab4bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2ab00b8d10 0x7f2ab00cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.089+0000 7f2ab4bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2ab00b26d0 0x7f2ab00b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.089+0000 7f2ab4bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab00cd7d0 con 0x7f2ab00b5140 2023-03-03T08:03:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.089+0000 7f2ab4bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab00b55c0 con 0x7f2ab00b8d10 2023-03-03T08:03:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.089+0000 7f2ab4bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab00b90e0 con 0x7f2ab00b26d0 2023-03-03T08:03:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.089+0000 7f2aae59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2ab00b8d10 0x7f2ab00cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.089+0000 7f2aae59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2ab00b8d10 0x7f2ab00cffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34388/0 (socket says 172.21.15.98:34388) 2023-03-03T08:03:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.089+0000 7f2aae59c700 1 -- 172.21.15.98:0/951412963 learned_addr learned my addr 172.21.15.98:0/951412963 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.090+0000 7f2aadd9b700 1 --2- 172.21.15.98:0/951412963 >> v2:172.21.15.100:3300/0 conn(0x7f2ab00b26d0 0x7f2ab00b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:03.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.090+0000 7f2aae59c700 1 -- 172.21.15.98:0/951412963 >> v2:172.21.15.100:3300/0 conn(0x7f2ab00b26d0 msgr2=0x7f2ab00b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:03.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.090+0000 7f2aae59c700 1 --2- 172.21.15.98:0/951412963 >> v2:172.21.15.100:3300/0 conn(0x7f2ab00b26d0 0x7f2ab00b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.090+0000 7f2aae59c700 1 -- 172.21.15.98:0/951412963 >> v1:172.21.15.100:6789/0 conn(0x7f2ab00b5140 legacy=0x7f2ab0160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:03.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.090+0000 7f2aae59c700 1 -- 172.21.15.98:0/951412963 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab00cda20 con 0x7f2ab00b8d10 2023-03-03T08:03:03.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.090+0000 7f2aae59c700 1 --2- 172.21.15.98:0/951412963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ab00b8d10 0x7f2ab00cffc0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f2ab008bfa0 tx=0x7f2aa0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=534af88070d6f2ae server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:03.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.091+0000 7f2aad59a700 1 -- 172.21.15.98:0/951412963 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa000b870 con 0x7f2ab00b8d10 2023-03-03T08:03:03.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.091+0000 7f2aad59a700 1 -- 172.21.15.98:0/951412963 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa000b9d0 con 0x7f2ab00b8d10 2023-03-03T08:03:03.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.091+0000 7f2aad59a700 1 -- 172.21.15.98:0/951412963 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa000bd60 con 0x7f2ab00b8d10 2023-03-03T08:03:03.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.091+0000 7f2ab4bff700 1 -- 172.21.15.98:0/951412963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ab00b8d10 msgr2=0x7f2ab00cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:03.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.091+0000 7f2ab4bff700 1 --2- 172.21.15.98:0/951412963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ab00b8d10 0x7f2ab00cffc0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f2ab008bfa0 tx=0x7f2aa0009bd0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.091+0000 7f2ab4bff700 1 -- 172.21.15.98:0/951412963 shutdown_connections 2023-03-03T08:03:03.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.091+0000 7f2ab4bff700 1 --2- 172.21.15.98:0/951412963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ab00b8d10 0x7f2ab00cffc0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.091+0000 7f2ab4bff700 1 --2- 172.21.15.98:0/951412963 >> v2:172.21.15.100:3300/0 conn(0x7f2ab00b26d0 0x7f2ab00b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.091+0000 7f2ab4bff700 1 -- 172.21.15.98:0/951412963 >> 172.21.15.98:0/951412963 conn(0x7f2ab0073e70 msgr2=0x7f2ab006ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:03.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.091+0000 7f2ab4bff700 1 -- 172.21.15.98:0/951412963 shutdown_connections 2023-03-03T08:03:03.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.092+0000 7f2ab4bff700 1 -- 172.21.15.98:0/951412963 wait complete. 2023-03-03T08:03:03.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.094+0000 7f2ab4bff700 1 Processor -- start 2023-03-03T08:03:03.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.094+0000 7f2ab4bff700 1 -- start start 2023-03-03T08:03:03.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.095+0000 7f2ab4bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ab00b26d0 0x7f2ab00ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:03.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.095+0000 7f2ab4bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ab00b5140 0x7f2ab00ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:03.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.095+0000 7f2ab4bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ab00b8d10 0x7f2ab01f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:03.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.095+0000 7f2ab4bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab01566c0 con 0x7f2ab00b8d10 2023-03-03T08:03:03.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.095+0000 7f2ab4bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab00caca0 con 0x7f2ab00b26d0 2023-03-03T08:03:03.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.095+0000 7f2ab4bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab0152790 con 0x7f2ab00b5140 2023-03-03T08:03:03.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.095+0000 7f2aae59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ab00b26d0 0x7f2ab00ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:03.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.095+0000 7f2aae59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ab00b26d0 0x7f2ab00ca760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:47592/0 (socket says 172.21.15.98:47592) 2023-03-03T08:03:03.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.095+0000 7f2aae59c700 1 -- 172.21.15.98:0/1390921578 learned_addr learned my addr 172.21.15.98:0/1390921578 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:03.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.095+0000 7f2aaed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ab00b8d10 0x7f2ab01f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:03.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.095+0000 7f2aadd9b700 1 --2- 172.21.15.98:0/1390921578 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ab00b5140 0x7f2ab00ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:03.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.096+0000 7f2aae59c700 1 -- 172.21.15.98:0/1390921578 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ab00b5140 msgr2=0x7f2ab00ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:03.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.096+0000 7f2aae59c700 1 --2- 172.21.15.98:0/1390921578 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ab00b5140 0x7f2ab00ccd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.096+0000 7f2aae59c700 1 -- 172.21.15.98:0/1390921578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ab00b8d10 msgr2=0x7f2ab01f5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:03.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.096+0000 7f2aae59c700 1 --2- 172.21.15.98:0/1390921578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ab00b8d10 0x7f2ab01f5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.096+0000 7f2aae59c700 1 -- 172.21.15.98:0/1390921578 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa0009550 con 0x7f2ab00b26d0 2023-03-03T08:03:03.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.096+0000 7f2aaed9d700 1 --2- 172.21.15.98:0/1390921578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ab00b8d10 0x7f2ab01f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:03:03.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.096+0000 7f2aae59c700 1 --2- 172.21.15.98:0/1390921578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ab00b26d0 0x7f2ab00ca760 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0000c00 tx=0x7f2aa0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:03.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.096+0000 7f2a9effd700 1 -- 172.21.15.98:0/1390921578 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0004310 con 0x7f2ab00b26d0 2023-03-03T08:03:03.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.097+0000 7f2a9effd700 1 -- 172.21.15.98:0/1390921578 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa001e070 con 0x7f2ab00b26d0 2023-03-03T08:03:03.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.097+0000 7f2ab4bff700 1 -- 172.21.15.98:0/1390921578 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab00d0e80 con 0x7f2ab00b26d0 2023-03-03T08:03:03.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.097+0000 7f2a9effd700 1 -- 172.21.15.98:0/1390921578 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa00036a0 con 0x7f2ab00b26d0 2023-03-03T08:03:03.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.097+0000 7f2ab4bff700 1 -- 172.21.15.98:0/1390921578 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab00cdd00 con 0x7f2ab00b26d0 2023-03-03T08:03:03.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.098+0000 7f2ab4bff700 1 -- 172.21.15.98:0/1390921578 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab00cdfe0 con 0x7f2ab00b26d0 2023-03-03T08:03:03.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.099+0000 7f2a9effd700 1 -- 172.21.15.98:0/1390921578 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f2aa0029020 con 0x7f2ab00b26d0 2023-03-03T08:03:03.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.100+0000 7f2a9effd700 1 --2- 172.21.15.98:0/1390921578 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2a90068380 0x7f2a9006a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:03.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.100+0000 7f2a9effd700 1 -- 172.21.15.98:0/1390921578 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f2aa0086fb0 con 0x7f2ab00b26d0 2023-03-03T08:03:03.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.101+0000 7f2aadd9b700 1 --2- 172.21.15.98:0/1390921578 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2a90068380 0x7f2a9006a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:03.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.101+0000 7f2aadd9b700 1 --2- 172.21.15.98:0/1390921578 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2a90068380 0x7f2a9006a830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2ab01f5c70 tx=0x7f2aa4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:03.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.104+0000 7f2a9effd700 1 -- 172.21.15.98:0/1390921578 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2aa0053a50 con 0x7f2ab00b26d0 2023-03-03T08:03:03.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.295+0000 7f2ab4bff700 1 -- 172.21.15.98:0/1390921578 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2ab015eac0 con 0x7f2ab00b26d0 2023-03-03T08:03:03.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.302+0000 7f2a9effd700 1 -- 172.21.15.98:0/1390921578 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2aa0057070 con 0x7f2ab00b26d0 2023-03-03T08:03:03.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.304+0000 7f2ab4bff700 1 -- 172.21.15.98:0/1390921578 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2a90068380 msgr2=0x7f2a9006a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:03.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.304+0000 7f2ab4bff700 1 --2- 172.21.15.98:0/1390921578 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2a90068380 0x7f2a9006a830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2ab01f5c70 tx=0x7f2aa4008040 comp rx=0 tx=0).stop 2023-03-03T08:03:03.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.304+0000 7f2ab4bff700 1 -- 172.21.15.98:0/1390921578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ab00b26d0 msgr2=0x7f2ab00ca760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:03.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.304+0000 7f2ab4bff700 1 --2- 172.21.15.98:0/1390921578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ab00b26d0 0x7f2ab00ca760 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0000c00 tx=0x7f2aa0004750 comp rx=0 tx=0).stop 2023-03-03T08:03:03.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.304+0000 7f2ab4bff700 1 -- 172.21.15.98:0/1390921578 shutdown_connections 2023-03-03T08:03:03.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.304+0000 7f2ab4bff700 1 --2- 172.21.15.98:0/1390921578 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2a90068380 0x7f2a9006a830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.304+0000 7f2ab4bff700 1 --2- 172.21.15.98:0/1390921578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ab00b8d10 0x7f2ab01f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.304+0000 7f2ab4bff700 1 --2- 172.21.15.98:0/1390921578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ab00b26d0 0x7f2ab00ca760 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.304+0000 7f2ab4bff700 1 --2- 172.21.15.98:0/1390921578 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ab00b5140 0x7f2ab00ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.304+0000 7f2ab4bff700 1 -- 172.21.15.98:0/1390921578 >> 172.21.15.98:0/1390921578 conn(0x7f2ab0073e70 msgr2=0x7f2ab00b2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:03.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.304+0000 7f2ab4bff700 1 -- 172.21.15.98:0/1390921578 shutdown_connections 2023-03-03T08:03:03.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.304+0000 7f2ab4bff700 1 -- 172.21.15.98:0/1390921578 wait complete. 2023-03-03T08:03:03.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars' 2023-03-03T08:03:03.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.574+0000 7f583edff700 1 Processor -- start 2023-03-03T08:03:03.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.574+0000 7f583edff700 1 -- start start 2023-03-03T08:03:03.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.576+0000 7f583edff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f583815cc00 0x7f58380cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:03.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.576+0000 7f583edff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f58380d27c0 0x7f58380d2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:03.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.576+0000 7f583edff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58380cde10 con 0x7f58380b60d0 2023-03-03T08:03:03.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.576+0000 7f583edff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58380ce040 con 0x7f58380d27c0 2023-03-03T08:03:03.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.576+0000 7f583edff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5838073be0 con 0x7f583815cc00 2023-03-03T08:03:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.576+0000 7f5837fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f58380d27c0 0x7f58380d2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.576+0000 7f5837fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f58380d27c0 0x7f58380d2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34408/0 (socket says 172.21.15.98:34408) 2023-03-03T08:03:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.576+0000 7f5837fff700 1 -- 172.21.15.98:0/4159934742 learned_addr learned my addr 172.21.15.98:0/4159934742 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.577+0000 7f583cb9b700 1 --2- 172.21.15.98:0/4159934742 >> v2:172.21.15.100:3300/0 conn(0x7f583815cc00 0x7f58380cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.577+0000 7f5837fff700 1 -- 172.21.15.98:0/4159934742 >> v2:172.21.15.100:3300/0 conn(0x7f583815cc00 msgr2=0x7f58380cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:03.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.577+0000 7f5837fff700 1 --2- 172.21.15.98:0/4159934742 >> v2:172.21.15.100:3300/0 conn(0x7f583815cc00 0x7f58380cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.577+0000 7f5837fff700 1 -- 172.21.15.98:0/4159934742 >> v1:172.21.15.100:6789/0 conn(0x7f58380b60d0 legacy=0x7f58380b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:03.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.577+0000 7f5837fff700 1 -- 172.21.15.98:0/4159934742 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58380d1a00 con 0x7f58380d27c0 2023-03-03T08:03:03.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.577+0000 7f583cb9b700 1 --2- 172.21.15.98:0/4159934742 >> v2:172.21.15.100:3300/0 conn(0x7f583815cc00 0x7f58380cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:03.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.577+0000 7f5837fff700 1 --2- 172.21.15.98:0/4159934742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58380d27c0 0x7f58380d2c60 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f583808c200 tx=0x7f5824009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c01f87f4abb2bfbc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:03.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.577+0000 7f58377fe700 1 -- 172.21.15.98:0/4159934742 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582400b870 con 0x7f58380d27c0 2023-03-03T08:03:03.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.578+0000 7f58377fe700 1 -- 172.21.15.98:0/4159934742 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582400b9d0 con 0x7f58380d27c0 2023-03-03T08:03:03.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.578+0000 7f58377fe700 1 -- 172.21.15.98:0/4159934742 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582400bd60 con 0x7f58380d27c0 2023-03-03T08:03:03.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.578+0000 7f583edff700 1 -- 172.21.15.98:0/4159934742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58380d27c0 msgr2=0x7f58380d2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:03.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.578+0000 7f583edff700 1 --2- 172.21.15.98:0/4159934742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58380d27c0 0x7f58380d2c60 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f583808c200 tx=0x7f5824009bd0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.578+0000 7f583edff700 1 -- 172.21.15.98:0/4159934742 shutdown_connections 2023-03-03T08:03:03.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.578+0000 7f583edff700 1 --2- 172.21.15.98:0/4159934742 >> v2:172.21.15.100:3300/0 conn(0x7f583815cc00 0x7f58380cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.578+0000 7f583edff700 1 --2- 172.21.15.98:0/4159934742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58380d27c0 0x7f58380d2c60 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.578+0000 7f583edff700 1 -- 172.21.15.98:0/4159934742 >> 172.21.15.98:0/4159934742 conn(0x7f583806aee0 msgr2=0x7f583806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:03.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.578+0000 7f583edff700 1 -- 172.21.15.98:0/4159934742 shutdown_connections 2023-03-03T08:03:03.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.578+0000 7f583edff700 1 -- 172.21.15.98:0/4159934742 wait complete. 2023-03-03T08:03:03.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.580+0000 7f583edff700 1 Processor -- start 2023-03-03T08:03:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.581+0000 7f583edff700 1 -- start start 2023-03-03T08:03:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.582+0000 7f583edff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58380b60d0 0x7f58380ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.582+0000 7f583edff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58380d27c0 0x7f583815e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.582+0000 7f583edff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f583815cc00 0x7f58380b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.582+0000 7f583edff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58380c8370 con 0x7f583815cc00 2023-03-03T08:03:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.582+0000 7f583edff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58380c77c0 con 0x7f58380d27c0 2023-03-03T08:03:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.582+0000 7f583edff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58380cd330 con 0x7f58380b60d0 2023-03-03T08:03:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.582+0000 7f583d39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f583815cc00 0x7f58380b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:03.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.582+0000 7f583d39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f583815cc00 0x7f58380b4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34412/0 (socket says 172.21.15.98:34412) 2023-03-03T08:03:03.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.582+0000 7f583d39c700 1 -- 172.21.15.98:0/1562431531 learned_addr learned my addr 172.21.15.98:0/1562431531 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:03.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.582+0000 7f5837fff700 1 --2- 172.21.15.98:0/1562431531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58380d27c0 0x7f583815e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:03.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.582+0000 7f583cb9b700 1 --2- 172.21.15.98:0/1562431531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58380b60d0 0x7f58380ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:03.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.583+0000 7f583d39c700 1 -- 172.21.15.98:0/1562431531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58380b60d0 msgr2=0x7f58380ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:03.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.583+0000 7f583d39c700 1 --2- 172.21.15.98:0/1562431531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58380b60d0 0x7f58380ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.583+0000 7f583d39c700 1 -- 172.21.15.98:0/1562431531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58380d27c0 msgr2=0x7f583815e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:03.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.583+0000 7f583d39c700 1 --2- 172.21.15.98:0/1562431531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58380d27c0 0x7f583815e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:03.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.583+0000 7f583d39c700 1 -- 172.21.15.98:0/1562431531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5824009550 con 0x7f583815cc00 2023-03-03T08:03:03.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.583+0000 7f583d39c700 1 --2- 172.21.15.98:0/1562431531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f583815cc00 0x7f58380b4590 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f583000cb50 tx=0x7f583000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:03.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.583+0000 7f58357fa700 1 -- 172.21.15.98:0/1562431531 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830003c00 con 0x7f583815cc00 2023-03-03T08:03:03.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.583+0000 7f58357fa700 1 -- 172.21.15.98:0/1562431531 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5830003d60 con 0x7f583815cc00 2023-03-03T08:03:03.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.583+0000 7f58357fa700 1 -- 172.21.15.98:0/1562431531 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583000aa20 con 0x7f583815cc00 2023-03-03T08:03:03.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.583+0000 7f583edff700 1 -- 172.21.15.98:0/1562431531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58380d1fc0 con 0x7f583815cc00 2023-03-03T08:03:03.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.584+0000 7f583edff700 1 -- 172.21.15.98:0/1562431531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58380d1ce0 con 0x7f583815cc00 2023-03-03T08:03:03.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.585+0000 7f583edff700 1 -- 172.21.15.98:0/1562431531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58380d22a0 con 0x7f583815cc00 2023-03-03T08:03:03.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.586+0000 7f58357fa700 1 -- 172.21.15.98:0/1562431531 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f5830010050 con 0x7f583815cc00 2023-03-03T08:03:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.587+0000 7f58357fa700 1 --2- 172.21.15.98:0/1562431531 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f581c068380 0x7f581c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.587+0000 7f58357fa700 1 -- 172.21.15.98:0/1562431531 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f5830085a50 con 0x7f583815cc00 2023-03-03T08:03:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.588+0000 7f583cb9b700 1 --2- 172.21.15.98:0/1562431531 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f581c068380 0x7f581c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.588+0000 7f583cb9b700 1 --2- 172.21.15.98:0/1562431531 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f581c068380 0x7f581c06a830 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f582c0099d0 tx=0x7f582c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.591+0000 7f58357fa700 1 -- 172.21.15.98:0/1562431531 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f58300523f0 con 0x7f583815cc00 2023-03-03T08:03:03.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:03:03.711 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:03:03.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:03.867+0000 7f583edff700 1 -- 172.21.15.98:0/1562431531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]} v 0) v1 -- 0x7f58380729b0 con 0x7f583815cc00 2023-03-03T08:03:04.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:04.813+0000 7f58357fa700 1 -- 172.21.15.98:0/1562431531 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]}]=0 v404) v1 ==== 146+0+0 (secure 0 0 0) 0x7f58300841d0 con 0x7f583815cc00 2023-03-03T08:03:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:04.816+0000 7f583edff700 1 -- 172.21.15.98:0/1562431531 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f581c068380 msgr2=0x7f581c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:04.816+0000 7f583edff700 1 --2- 172.21.15.98:0/1562431531 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f581c068380 0x7f581c06a830 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f582c0099d0 tx=0x7f582c008040 comp rx=0 tx=0).stop 2023-03-03T08:03:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:04.816+0000 7f583edff700 1 -- 172.21.15.98:0/1562431531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f583815cc00 msgr2=0x7f58380b4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:04.816+0000 7f583edff700 1 --2- 172.21.15.98:0/1562431531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f583815cc00 0x7f58380b4590 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f583000cb50 tx=0x7f583000ce60 comp rx=0 tx=0).stop 2023-03-03T08:03:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:04.816+0000 7f583edff700 1 -- 172.21.15.98:0/1562431531 shutdown_connections 2023-03-03T08:03:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:04.816+0000 7f583edff700 1 --2- 172.21.15.98:0/1562431531 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f581c068380 0x7f581c06a830 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:04.816+0000 7f583edff700 1 --2- 172.21.15.98:0/1562431531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f583815cc00 0x7f58380b4590 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:04.816+0000 7f583edff700 1 --2- 172.21.15.98:0/1562431531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58380b60d0 0x7f58380ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:04.816+0000 7f583edff700 1 --2- 172.21.15.98:0/1562431531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58380d27c0 0x7f583815e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:04.816+0000 7f583edff700 1 -- 172.21.15.98:0/1562431531 >> 172.21.15.98:0/1562431531 conn(0x7f583806aee0 msgr2=0x7f583815be70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:04.817+0000 7f583edff700 1 -- 172.21.15.98:0/1562431531 shutdown_connections 2023-03-03T08:03:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:04.817+0000 7f583edff700 1 -- 172.21.15.98:0/1562431531 wait complete. 2023-03-03T08:03:04.848 INFO:tasks.cephfs_test_runner:test_names_can_only_be_goodchars (tasks.cephfs.test_volumes.TestMisc) ... ok 2023-03-03T08:03:04.849 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) 2023-03-03T08:03:04.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity' 2023-03-03T08:03:05.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.135+0000 7fb020dff700 1 Processor -- start 2023-03-03T08:03:05.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.135+0000 7fb020dff700 1 -- start start 2023-03-03T08:03:05.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.137+0000 7fb020dff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb01c0d23e0 0x7fb01c0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:05.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.137+0000 7fb020dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c0b5b20 con 0x7fb01c0b87c0 2023-03-03T08:03:05.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.137+0000 7fb020dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c0d0ef0 con 0x7fb01c0b8b90 2023-03-03T08:03:05.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.137+0000 7fb020dff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb01c0d0600 con 0x7fb01c0d23e0 2023-03-03T08:03:05.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.137+0000 7fb01a59c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb01c0b87c0 0x7fb01c0b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41956/0 (socket says 172.21.15.98:41956) 2023-03-03T08:03:05.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.137+0000 7fb01a59c700 1 -- 172.21.15.98:0/1996716207 learned_addr learned my addr 172.21.15.98:0/1996716207 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:05.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.137+0000 7fb01ad9d700 1 --2- 172.21.15.98:0/1996716207 >> v2:172.21.15.100:3300/0 conn(0x7fb01c0d23e0 0x7fb01c0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:05.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.137+0000 7fb01959a700 1 -- 172.21.15.98:0/1996716207 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377830101 0 0) 0x7fb01c0b5b20 con 0x7fb01c0b87c0 2023-03-03T08:03:05.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.138+0000 7fb01959a700 1 -- 172.21.15.98:0/1996716207 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0040035b0 con 0x7fb01c0b87c0 2023-03-03T08:03:05.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.138+0000 7fb01959a700 1 -- 172.21.15.98:0/1996716207 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835665912 0 0) 0x7fb01c0d0ef0 con 0x7fb01c0b8b90 2023-03-03T08:03:05.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.138+0000 7fb01959a700 1 -- 172.21.15.98:0/1996716207 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb01c0b5b20 con 0x7fb01c0b8b90 2023-03-03T08:03:05.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.138+0000 7fb01959a700 1 -- 172.21.15.98:0/1996716207 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 83743197 0 0) 0x7fb0040035b0 con 0x7fb01c0b87c0 2023-03-03T08:03:05.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.138+0000 7fb01ad9d700 1 -- 172.21.15.98:0/1996716207 >> v1:172.21.15.100:6789/0 conn(0x7fb01c0b8b90 legacy=0x7fb01c0b5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:05.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.138+0000 7fb01ad9d700 1 -- 172.21.15.98:0/1996716207 >> v1:172.21.15.98:6789/0 conn(0x7fb01c0b87c0 legacy=0x7fb01c0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:05.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.138+0000 7fb01ad9d700 1 -- 172.21.15.98:0/1996716207 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c0cd970 con 0x7fb01c0d23e0 2023-03-03T08:03:05.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.138+0000 7fb01ad9d700 1 --2- 172.21.15.98:0/1996716207 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb01c0d23e0 0x7fb01c0cffe0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb014009d90 tx=0x7fb01400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5013ad49b75e0c7b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:05.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.139+0000 7fb01959a700 1 -- 172.21.15.98:0/1996716207 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01400e070 con 0x7fb01c0d23e0 2023-03-03T08:03:05.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.139+0000 7fb01959a700 1 -- 172.21.15.98:0/1996716207 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb014003a20 con 0x7fb01c0d23e0 2023-03-03T08:03:05.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.139+0000 7fb01959a700 1 -- 172.21.15.98:0/1996716207 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c001670 con 0x7fb01c0d23e0 2023-03-03T08:03:05.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.139+0000 7fb020dff700 1 -- 172.21.15.98:0/1996716207 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb01c0d23e0 msgr2=0x7fb01c0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:05.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.139+0000 7fb020dff700 1 --2- 172.21.15.98:0/1996716207 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb01c0d23e0 0x7fb01c0cffe0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb014009d90 tx=0x7fb01400b330 comp rx=0 tx=0).stop 2023-03-03T08:03:05.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.139+0000 7fb020dff700 1 -- 172.21.15.98:0/1996716207 shutdown_connections 2023-03-03T08:03:05.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.139+0000 7fb020dff700 1 --2- 172.21.15.98:0/1996716207 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb01c0d23e0 0x7fb01c0cffe0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:05.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.139+0000 7fb020dff700 1 -- 172.21.15.98:0/1996716207 >> 172.21.15.98:0/1996716207 conn(0x7fb01c06aee0 msgr2=0x7fb01c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:05.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.139+0000 7fb020dff700 1 -- 172.21.15.98:0/1996716207 shutdown_connections 2023-03-03T08:03:05.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.139+0000 7fb020dff700 1 -- 172.21.15.98:0/1996716207 wait complete. 2023-03-03T08:03:05.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.141+0000 7fb020dff700 1 Processor -- start 2023-03-03T08:03:05.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.142+0000 7fb020dff700 1 -- start start 2023-03-03T08:03:05.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb020dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb01c0b87c0 0x7fb01c14a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:05.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb020dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb01c0b8b90 0x7fb01c1f5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:05.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb020dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb01c0d23e0 0x7fb01c1f5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:05.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb020dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c14add0 con 0x7fb01c0b87c0 2023-03-03T08:03:05.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb020dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c1497f0 con 0x7fb01c0b8b90 2023-03-03T08:03:05.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb020dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c149ad0 con 0x7fb01c0d23e0 2023-03-03T08:03:05.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb01a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb01c0b87c0 0x7fb01c14a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:05.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb01a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb01c0b87c0 0x7fb01c14a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51050/0 (socket says 172.21.15.98:51050) 2023-03-03T08:03:05.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb01a59c700 1 -- 172.21.15.98:0/3458782509 learned_addr learned my addr 172.21.15.98:0/3458782509 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:05.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb019d9b700 1 --2- 172.21.15.98:0/3458782509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb01c0b8b90 0x7fb01c1f5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:05.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb01ad9d700 1 --2- 172.21.15.98:0/3458782509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb01c0d23e0 0x7fb01c1f5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:05.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb01a59c700 1 -- 172.21.15.98:0/3458782509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb01c0d23e0 msgr2=0x7fb01c1f5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:05.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb01a59c700 1 --2- 172.21.15.98:0/3458782509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb01c0d23e0 0x7fb01c1f5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:05.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb01a59c700 1 -- 172.21.15.98:0/3458782509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb01c0b8b90 msgr2=0x7fb01c1f5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:05.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb01a59c700 1 --2- 172.21.15.98:0/3458782509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb01c0b8b90 0x7fb01c1f5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:05.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.143+0000 7fb01a59c700 1 -- 172.21.15.98:0/3458782509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb014009a70 con 0x7fb01c0b87c0 2023-03-03T08:03:05.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.144+0000 7fb01a59c700 1 --2- 172.21.15.98:0/3458782509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb01c0b87c0 0x7fb01c14a890 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00c750 tx=0x7fb00c00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:05.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.144+0000 7fb00affd700 1 -- 172.21.15.98:0/3458782509 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c004c80 con 0x7fb01c0b87c0 2023-03-03T08:03:05.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.144+0000 7fb00affd700 1 -- 172.21.15.98:0/3458782509 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00c004de0 con 0x7fb01c0b87c0 2023-03-03T08:03:05.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.144+0000 7fb00affd700 1 -- 172.21.15.98:0/3458782509 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c0072d0 con 0x7fb01c0b87c0 2023-03-03T08:03:05.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.144+0000 7fb020dff700 1 -- 172.21.15.98:0/3458782509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01c073a80 con 0x7fb01c0b87c0 2023-03-03T08:03:05.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.144+0000 7fb020dff700 1 -- 172.21.15.98:0/3458782509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01c0cdc50 con 0x7fb01c0b87c0 2023-03-03T08:03:05.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.145+0000 7fb020dff700 1 -- 172.21.15.98:0/3458782509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01c0cdf30 con 0x7fb01c0b87c0 2023-03-03T08:03:05.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.147+0000 7fb00affd700 1 -- 172.21.15.98:0/3458782509 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fb00c01a020 con 0x7fb01c0b87c0 2023-03-03T08:03:05.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.148+0000 7fb00affd700 1 --2- 172.21.15.98:0/3458782509 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7faffc068380 0x7faffc06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:05.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.148+0000 7fb00affd700 1 -- 172.21.15.98:0/3458782509 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fb00c0857c0 con 0x7fb01c0b87c0 2023-03-03T08:03:05.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.149+0000 7fb019d9b700 1 --2- 172.21.15.98:0/3458782509 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7faffc068380 0x7faffc06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:05.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.150+0000 7fb019d9b700 1 --2- 172.21.15.98:0/3458782509 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7faffc068380 0x7faffc06a830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0cfbd0 tx=0x7fb010009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:05.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.152+0000 7fb00affd700 1 -- 172.21.15.98:0/3458782509 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb00c0521e0 con 0x7fb01c0b87c0 2023-03-03T08:03:05.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.416+0000 7fb020dff700 1 -- 172.21.15.98:0/3458782509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]} v 0) v1 -- 0x7fb01c0729b0 con 0x7fb01c0b87c0 2023-03-03T08:03:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.815+0000 7fb00affd700 1 -- 172.21.15.98:0/3458782509 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]}]=0 v405) v1 ==== 152+0+0 (secure 0 0 0) 0x7fb00c083f40 con 0x7fb01c0b87c0 2023-03-03T08:03:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.818+0000 7fb020dff700 1 -- 172.21.15.98:0/3458782509 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7faffc068380 msgr2=0x7faffc06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.818+0000 7fb020dff700 1 --2- 172.21.15.98:0/3458782509 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7faffc068380 0x7faffc06a830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0cfbd0 tx=0x7fb010009040 comp rx=0 tx=0).stop 2023-03-03T08:03:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.818+0000 7fb020dff700 1 -- 172.21.15.98:0/3458782509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb01c0b87c0 msgr2=0x7fb01c14a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.818+0000 7fb020dff700 1 --2- 172.21.15.98:0/3458782509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb01c0b87c0 0x7fb01c14a890 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00c750 tx=0x7fb00c00ca60 comp rx=0 tx=0).stop 2023-03-03T08:03:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.818+0000 7fb020dff700 1 -- 172.21.15.98:0/3458782509 shutdown_connections 2023-03-03T08:03:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.818+0000 7fb020dff700 1 --2- 172.21.15.98:0/3458782509 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7faffc068380 0x7faffc06a830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.818+0000 7fb020dff700 1 --2- 172.21.15.98:0/3458782509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb01c0b87c0 0x7fb01c14a890 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.818+0000 7fb020dff700 1 --2- 172.21.15.98:0/3458782509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb01c0d23e0 0x7fb01c1f5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.818+0000 7fb020dff700 1 --2- 172.21.15.98:0/3458782509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb01c0b8b90 0x7fb01c1f5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.818+0000 7fb020dff700 1 -- 172.21.15.98:0/3458782509 >> 172.21.15.98:0/3458782509 conn(0x7fb01c06aee0 msgr2=0x7fb01c0b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.818+0000 7fb020dff700 1 -- 172.21.15.98:0/3458782509 shutdown_connections 2023-03-03T08:03:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:05.818+0000 7fb020dff700 1 -- 172.21.15.98:0/3458782509 wait complete. 2023-03-03T08:03:05.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-03-03T08:03:06.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.092+0000 7f208e3ff700 1 Processor -- start 2023-03-03T08:03:06.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.092+0000 7f208e3ff700 1 -- start start 2023-03-03T08:03:06.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.093+0000 7f208e3ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f20880b8d10 0x7f20880cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:06.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.094+0000 7f208e3ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20880b5310 0x7f20880d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:06.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.094+0000 7f208e3ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20880cd7d0 con 0x7f20880b26d0 2023-03-03T08:03:06.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.094+0000 7f208e3ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20880b90e0 con 0x7f20880b5310 2023-03-03T08:03:06.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.094+0000 7f208e3ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20880cda20 con 0x7f20880b8d10 2023-03-03T08:03:06.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.094+0000 7f208c99c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20880b5310 0x7f20880d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:06.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.094+0000 7f208c99c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20880b5310 0x7f20880d13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51062/0 (socket says 172.21.15.98:51062) 2023-03-03T08:03:06.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.094+0000 7f208c99c700 1 -- 172.21.15.98:0/2467901841 learned_addr learned my addr 172.21.15.98:0/2467901841 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:06.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.094+0000 7f2087fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f20880b8d10 0x7f20880cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:06.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.094+0000 7f208c99c700 1 -- 172.21.15.98:0/2467901841 >> v2:172.21.15.100:3300/0 conn(0x7f20880b8d10 msgr2=0x7f20880cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:06.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.094+0000 7f208c99c700 1 --2- 172.21.15.98:0/2467901841 >> v2:172.21.15.100:3300/0 conn(0x7f20880b8d10 0x7f20880cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:06.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.094+0000 7f208c99c700 1 -- 172.21.15.98:0/2467901841 >> v1:172.21.15.100:6789/0 conn(0x7f20880b26d0 legacy=0x7f20880b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:06.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.094+0000 7f208c99c700 1 -- 172.21.15.98:0/2467901841 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20880cdd00 con 0x7f20880b5310 2023-03-03T08:03:06.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.095+0000 7f208c99c700 1 --2- 172.21.15.98:0/2467901841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20880b5310 0x7f20880d13a0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f208808c040 tx=0x7f208000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aeb4d117074fb4b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:06.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.095+0000 7f2086ffd700 1 -- 172.21.15.98:0/2467901841 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208000e070 con 0x7f20880b5310 2023-03-03T08:03:06.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.095+0000 7f2086ffd700 1 -- 172.21.15.98:0/2467901841 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208000b910 con 0x7f20880b5310 2023-03-03T08:03:06.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.095+0000 7f2086ffd700 1 -- 172.21.15.98:0/2467901841 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208000bcc0 con 0x7f20880b5310 2023-03-03T08:03:06.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.096+0000 7f208e3ff700 1 -- 172.21.15.98:0/2467901841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20880b5310 msgr2=0x7f20880d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:06.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.096+0000 7f208e3ff700 1 --2- 172.21.15.98:0/2467901841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20880b5310 0x7f20880d13a0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f208808c040 tx=0x7f208000b330 comp rx=0 tx=0).stop 2023-03-03T08:03:06.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.096+0000 7f208e3ff700 1 -- 172.21.15.98:0/2467901841 shutdown_connections 2023-03-03T08:03:06.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.096+0000 7f208e3ff700 1 --2- 172.21.15.98:0/2467901841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20880b5310 0x7f20880d13a0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:06.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.096+0000 7f208e3ff700 1 --2- 172.21.15.98:0/2467901841 >> v2:172.21.15.100:3300/0 conn(0x7f20880b8d10 0x7f20880cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:06.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.096+0000 7f208e3ff700 1 -- 172.21.15.98:0/2467901841 >> 172.21.15.98:0/2467901841 conn(0x7f2088073e70 msgr2=0x7f208806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:06.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.096+0000 7f208e3ff700 1 -- 172.21.15.98:0/2467901841 shutdown_connections 2023-03-03T08:03:06.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.096+0000 7f208e3ff700 1 -- 172.21.15.98:0/2467901841 wait complete. 2023-03-03T08:03:06.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.098+0000 7f208e3ff700 1 Processor -- start 2023-03-03T08:03:06.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.098+0000 7f208e3ff700 1 -- start start 2023-03-03T08:03:06.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.099+0000 7f208e3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20880b26d0 0x7f2088156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:06.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.099+0000 7f208e3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20880b5310 0x7f2088151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:06.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.099+0000 7f208e3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20880b8d10 0x7f20880b75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:06.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.099+0000 7f208e3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2088152ab0 con 0x7f20880b26d0 2023-03-03T08:03:06.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.099+0000 7f208e3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2088158b40 con 0x7f20880b5310 2023-03-03T08:03:06.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.099+0000 7f208e3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2088159870 con 0x7f20880b8d10 2023-03-03T08:03:06.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.099+0000 7f2087fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20880b26d0 0x7f2088156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:06.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.100+0000 7f2087fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20880b26d0 0x7f2088156540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51076/0 (socket says 172.21.15.98:51076) 2023-03-03T08:03:06.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.100+0000 7f2087fff700 1 -- 172.21.15.98:0/935604475 learned_addr learned my addr 172.21.15.98:0/935604475 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:06.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.100+0000 7f208c99c700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20880b8d10 0x7f20880b75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:06.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.100+0000 7f2087fff700 1 -- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20880b8d10 msgr2=0x7f20880b75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:06.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.100+0000 7f2087fff700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20880b8d10 0x7f20880b75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:06.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.100+0000 7f20877fe700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20880b5310 0x7f2088151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:06.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.100+0000 7f2087fff700 1 -- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20880b5310 msgr2=0x7f2088151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:06.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.100+0000 7f2087fff700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20880b5310 0x7f2088151920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:06.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.100+0000 7f2087fff700 1 -- 172.21.15.98:0/935604475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2080009b30 con 0x7f20880b26d0 2023-03-03T08:03:06.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.100+0000 7f208c99c700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20880b8d10 0x7f20880b75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:03:06.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.100+0000 7f20877fe700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20880b5310 0x7f2088151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:03:06.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.100+0000 7f2087fff700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20880b26d0 0x7f2088156540 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f207800e9d0 tx=0x7f207800ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:06.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.100+0000 7f2084ff9700 1 -- 172.21.15.98:0/935604475 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2078009730 con 0x7f20880b26d0 2023-03-03T08:03:06.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.101+0000 7f2084ff9700 1 -- 172.21.15.98:0/935604475 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207800c850 con 0x7f20880b26d0 2023-03-03T08:03:06.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.101+0000 7f2084ff9700 1 -- 172.21.15.98:0/935604475 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207800cbe0 con 0x7f20880b26d0 2023-03-03T08:03:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.101+0000 7f208e3ff700 1 -- 172.21.15.98:0/935604475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20880cdfe0 con 0x7f20880b26d0 2023-03-03T08:03:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.101+0000 7f208e3ff700 1 -- 172.21.15.98:0/935604475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20880d0e80 con 0x7f20880b26d0 2023-03-03T08:03:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.102+0000 7f208e3ff700 1 -- 172.21.15.98:0/935604475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20880d0500 con 0x7f20880b26d0 2023-03-03T08:03:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.103+0000 7f2084ff9700 1 -- 172.21.15.98:0/935604475 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f2078019030 con 0x7f20880b26d0 2023-03-03T08:03:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.104+0000 7f2084ff9700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2074068380 0x7f207406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:06.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.104+0000 7f2084ff9700 1 -- 172.21.15.98:0/935604475 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f2078085b70 con 0x7f20880b26d0 2023-03-03T08:03:06.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.105+0000 7f20877fe700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2074068380 0x7f207406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:06.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.106+0000 7f20877fe700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2074068380 0x7f207406a830 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2088159a70 tx=0x7f207c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:06.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.108+0000 7f2084ff9700 1 -- 172.21.15.98:0/935604475 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2078052590 con 0x7f20880b26d0 2023-03-03T08:03:06.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.299+0000 7f208e3ff700 1 -- 172.21.15.98:0/935604475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f208804e6e0 con 0x7f20880b26d0 2023-03-03T08:03:06.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.304+0000 7f2084ff9700 1 -- 172.21.15.98:0/935604475 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f20780170c0 con 0x7f20880b26d0 2023-03-03T08:03:06.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.307+0000 7f208e3ff700 1 -- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2074068380 msgr2=0x7f207406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:06.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.307+0000 7f208e3ff700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2074068380 0x7f207406a830 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2088159a70 tx=0x7f207c009040 comp rx=0 tx=0).stop 2023-03-03T08:03:06.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.307+0000 7f208e3ff700 1 -- 172.21.15.98:0/935604475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20880b26d0 msgr2=0x7f2088156540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:06.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.307+0000 7f208e3ff700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20880b26d0 0x7f2088156540 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f207800e9d0 tx=0x7f207800ed90 comp rx=0 tx=0).stop 2023-03-03T08:03:06.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.307+0000 7f208e3ff700 1 -- 172.21.15.98:0/935604475 shutdown_connections 2023-03-03T08:03:06.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.307+0000 7f208e3ff700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2074068380 0x7f207406a830 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:06.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.307+0000 7f208e3ff700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20880b26d0 0x7f2088156540 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:06.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.307+0000 7f208e3ff700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20880b8d10 0x7f20880b75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:06.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.307+0000 7f208e3ff700 1 --2- 172.21.15.98:0/935604475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20880b5310 0x7f2088151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:06.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.307+0000 7f208e3ff700 1 -- 172.21.15.98:0/935604475 >> 172.21.15.98:0/935604475 conn(0x7f2088073e70 msgr2=0x7f208806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:06.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.307+0000 7f208e3ff700 1 -- 172.21.15.98:0/935604475 shutdown_connections 2023-03-03T08:03:06.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:06.307+0000 7f208e3ff700 1 -- 172.21.15.98:0/935604475 wait complete. 2023-03-03T08:03:06.336 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-03-03T08:03:06.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:03:06.470 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:03:06.616 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-03-03T08:03:06.617 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:03:06.617 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:03:06.685 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T08:03:06.686 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:03:06.686 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:03:06.755 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T08:03:06.756 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:03:06.756 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:03:06.825 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349160k,nr_inodes=4087290,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275604k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-03T08:03:06.826 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-03-03T08:03:06.826 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:03:06.894 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16349000k,nr_inodes=4087250,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275572k,mode=700,uid=1000,gid=1268 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-03-03T08:03:06.895 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-03-03T08:03:06.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:03:07.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.138+0000 7f202e1ff700 1 Processor -- start 2023-03-03T08:03:07.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.139+0000 7f202e1ff700 1 -- start start 2023-03-03T08:03:07.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.140+0000 7f202e1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20280b8b90 0x7f20280cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:07.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.140+0000 7f202e1ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f20280b5410 0x7f20280d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:07.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.140+0000 7f202e1ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20280d0ef0 con 0x7f20280b87c0 2023-03-03T08:03:07.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.140+0000 7f202e1ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20280d0600 con 0x7f20280b5410 2023-03-03T08:03:07.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.140+0000 7f202e1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20280cd970 con 0x7f20280b8b90 2023-03-03T08:03:07.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.141+0000 7f2027fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f20280b5410 0x7f20280d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:07.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.141+0000 7f20277fe700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f20280b87c0 0x7f20280b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:60090/0 (socket says 172.21.15.98:60090) 2023-03-03T08:03:07.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.141+0000 7f20277fe700 1 -- 172.21.15.98:0/1883050620 learned_addr learned my addr 172.21.15.98:0/1883050620 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:07.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.141+0000 7f2026ffd700 1 --2- 172.21.15.98:0/1883050620 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20280b8b90 0x7f20280cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:07.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.141+0000 7f2027fff700 1 -- 172.21.15.98:0/1883050620 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20280b8b90 msgr2=0x7f20280cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:07.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.141+0000 7f2027fff700 1 --2- 172.21.15.98:0/1883050620 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20280b8b90 0x7f20280cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:07.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.141+0000 7f2027fff700 1 -- 172.21.15.98:0/1883050620 >> v1:172.21.15.100:6789/0 conn(0x7f20280b87c0 legacy=0x7f20280b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:07.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.141+0000 7f2027fff700 1 -- 172.21.15.98:0/1883050620 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20280cdc50 con 0x7f20280b5410 2023-03-03T08:03:07.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.142+0000 7f2027fff700 1 --2- 172.21.15.98:0/1883050620 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20280b5410 0x7f20280d13d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2020009d90 tx=0x7f202000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b51650b943c50910 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:07.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.142+0000 7f20267fc700 1 -- 172.21.15.98:0/1883050620 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202000e070 con 0x7f20280b5410 2023-03-03T08:03:07.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.142+0000 7f20267fc700 1 -- 172.21.15.98:0/1883050620 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202000bb20 con 0x7f20280b5410 2023-03-03T08:03:07.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.142+0000 7f20267fc700 1 -- 172.21.15.98:0/1883050620 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020012430 con 0x7f20280b5410 2023-03-03T08:03:07.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.142+0000 7f202e1ff700 1 -- 172.21.15.98:0/1883050620 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20280b5410 msgr2=0x7f20280d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:07.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.142+0000 7f202e1ff700 1 --2- 172.21.15.98:0/1883050620 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20280b5410 0x7f20280d13d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2020009d90 tx=0x7f202000b330 comp rx=0 tx=0).stop 2023-03-03T08:03:07.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.143+0000 7f202e1ff700 1 -- 172.21.15.98:0/1883050620 shutdown_connections 2023-03-03T08:03:07.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.143+0000 7f202e1ff700 1 --2- 172.21.15.98:0/1883050620 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20280b5410 0x7f20280d13d0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:07.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.143+0000 7f202e1ff700 1 --2- 172.21.15.98:0/1883050620 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20280b8b90 0x7f20280cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:07.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.143+0000 7f202e1ff700 1 -- 172.21.15.98:0/1883050620 >> 172.21.15.98:0/1883050620 conn(0x7f202806aee0 msgr2=0x7f202806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:07.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.143+0000 7f202e1ff700 1 -- 172.21.15.98:0/1883050620 shutdown_connections 2023-03-03T08:03:07.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.143+0000 7f202e1ff700 1 -- 172.21.15.98:0/1883050620 wait complete. 2023-03-03T08:03:07.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.145+0000 7f202e1ff700 1 Processor -- start 2023-03-03T08:03:07.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.145+0000 7f202e1ff700 1 -- start start 2023-03-03T08:03:07.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.146+0000 7f202e1ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20280b5410 0x7f202815c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:07.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.146+0000 7f202e1ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20280b87c0 0x7f20280ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:07.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.146+0000 7f202e1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20280b8b90 0x7f20280c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:07.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.146+0000 7f202e1ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20280cb830 con 0x7f20280b5410 2023-03-03T08:03:07.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.146+0000 7f202e1ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20280cacc0 con 0x7f20280b87c0 2023-03-03T08:03:07.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.146+0000 7f202e1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20280cafa0 con 0x7f20280b8b90 2023-03-03T08:03:07.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.146+0000 7f20277fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20280b5410 0x7f202815c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:07.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.146+0000 7f20277fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20280b5410 0x7f202815c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51088/0 (socket says 172.21.15.98:51088) 2023-03-03T08:03:07.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.146+0000 7f20277fe700 1 -- 172.21.15.98:0/1886787431 learned_addr learned my addr 172.21.15.98:0/1886787431 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:07.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.147+0000 7f2026ffd700 1 --2- 172.21.15.98:0/1886787431 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20280b87c0 0x7f20280ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:07.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.147+0000 7f2027fff700 1 --2- 172.21.15.98:0/1886787431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20280b8b90 0x7f20280c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:07.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.147+0000 7f20277fe700 1 -- 172.21.15.98:0/1886787431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20280b8b90 msgr2=0x7f20280c83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:07.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.147+0000 7f20277fe700 1 --2- 172.21.15.98:0/1886787431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20280b8b90 0x7f20280c83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:07.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.147+0000 7f20277fe700 1 -- 172.21.15.98:0/1886787431 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20280b87c0 msgr2=0x7f20280ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:07.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.147+0000 7f20277fe700 1 --2- 172.21.15.98:0/1886787431 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20280b87c0 0x7f20280ca350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:07.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.147+0000 7f20277fe700 1 -- 172.21.15.98:0/1886787431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2020009a70 con 0x7f20280b5410 2023-03-03T08:03:07.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.147+0000 7f20277fe700 1 --2- 172.21.15.98:0/1886787431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20280b5410 0x7f202815c9b0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f201800c520 tx=0x7f201800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:07.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.148+0000 7f200ffff700 1 -- 172.21.15.98:0/1886787431 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2018005d70 con 0x7f20280b5410 2023-03-03T08:03:07.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.148+0000 7f200ffff700 1 -- 172.21.15.98:0/1886787431 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20180070d0 con 0x7f20280b5410 2023-03-03T08:03:07.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.148+0000 7f200ffff700 1 -- 172.21.15.98:0/1886787431 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2018007440 con 0x7f20280b5410 2023-03-03T08:03:07.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.148+0000 7f202e1ff700 1 -- 172.21.15.98:0/1886787431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20280cdf30 con 0x7f20280b5410 2023-03-03T08:03:07.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.148+0000 7f202e1ff700 1 -- 172.21.15.98:0/1886787431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2028073a80 con 0x7f20280b5410 2023-03-03T08:03:07.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.149+0000 7f202e1ff700 1 -- 172.21.15.98:0/1886787431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20280ce0c0 con 0x7f20280b5410 2023-03-03T08:03:07.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.150+0000 7f200ffff700 1 -- 172.21.15.98:0/1886787431 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f2018019030 con 0x7f20280b5410 2023-03-03T08:03:07.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.152+0000 7f200ffff700 1 --2- 172.21.15.98:0/1886787431 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2014068380 0x7f201406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:07.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.152+0000 7f200ffff700 1 -- 172.21.15.98:0/1886787431 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f2018084cc0 con 0x7f20280b5410 2023-03-03T08:03:07.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.152+0000 7f2026ffd700 1 --2- 172.21.15.98:0/1886787431 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2014068380 0x7f201406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:07.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.153+0000 7f2026ffd700 1 --2- 172.21.15.98:0/1886787431 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2014068380 0x7f201406a830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f202815cef0 tx=0x7f201c009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:07.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.155+0000 7f200ffff700 1 -- 172.21.15.98:0/1886787431 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f201804b5e0 con 0x7f20280b5410 2023-03-03T08:03:07.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.390+0000 7f202e1ff700 1 -- 172.21.15.98:0/1886787431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20280729b0 con 0x7f20280b5410 2023-03-03T08:03:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.390+0000 7f200ffff700 1 -- 172.21.15.98:0/1886787431 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+3532 (secure 0 0 0) 0x7f2018017090 con 0x7f20280b5410 2023-03-03T08:03:07.392 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:03:07.393 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":56,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6602,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/463617820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":463617820},{"type":"v1","addr":"172.21.15.100:6835","nonce":463617820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2561725713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2561725713},{"type":"v1","addr":"172.21.15.98:6834","nonce":2561725713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/1511797280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":1511797280},{"type":"v1","addr":"172.21.15.100:6834","nonce":1511797280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3399768807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":3399768807},{"type":"v1","addr":"172.21.15.98:6835","nonce":3399768807}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":56}],"filesystems":[]} 2023-03-03T08:03:07.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.393+0000 7f202e1ff700 1 -- 172.21.15.98:0/1886787431 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2014068380 msgr2=0x7f201406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:07.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.393+0000 7f202e1ff700 1 --2- 172.21.15.98:0/1886787431 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2014068380 0x7f201406a830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f202815cef0 tx=0x7f201c009210 comp rx=0 tx=0).stop 2023-03-03T08:03:07.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.393+0000 7f202e1ff700 1 -- 172.21.15.98:0/1886787431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20280b5410 msgr2=0x7f202815c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:07.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.393+0000 7f202e1ff700 1 --2- 172.21.15.98:0/1886787431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20280b5410 0x7f202815c9b0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f201800c520 tx=0x7f201800c830 comp rx=0 tx=0).stop 2023-03-03T08:03:07.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.393+0000 7f202e1ff700 1 -- 172.21.15.98:0/1886787431 shutdown_connections 2023-03-03T08:03:07.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.393+0000 7f202e1ff700 1 --2- 172.21.15.98:0/1886787431 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2014068380 0x7f201406a830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:07.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.393+0000 7f202e1ff700 1 --2- 172.21.15.98:0/1886787431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20280b5410 0x7f202815c9b0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:07.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.393+0000 7f202e1ff700 1 --2- 172.21.15.98:0/1886787431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20280b8b90 0x7f20280c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:07.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.393+0000 7f202e1ff700 1 --2- 172.21.15.98:0/1886787431 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20280b87c0 0x7f20280ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:07.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.393+0000 7f202e1ff700 1 -- 172.21.15.98:0/1886787431 >> 172.21.15.98:0/1886787431 conn(0x7f202806aee0 msgr2=0x7f202815f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:07.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.393+0000 7f202e1ff700 1 -- 172.21.15.98:0/1886787431 shutdown_connections 2023-03-03T08:03:07.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.393+0000 7f202e1ff700 1 -- 172.21.15.98:0/1886787431 wait complete. 2023-03-03T08:03:07.398 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 56 2023-03-03T08:03:07.422 INFO:tasks.ceph.mds.a:Restarting daemon 2023-03-03T08:03:07.422 INFO:tasks.ceph.mds.a:Stopping old one... 2023-03-03T08:03:07.423 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-03-03T08:03:07.423 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:03:07.423 INFO:tasks.ceph.mds.c:Restarting daemon 2023-03-03T08:03:07.423 INFO:tasks.ceph.mds.c:Stopping old one... 2023-03-03T08:03:07.423 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-03-03T08:03:07.424 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:03:07.424 INFO:tasks.ceph.mds.b:Restarting daemon 2023-03-03T08:03:07.424 INFO:tasks.ceph.mds.b:Stopping old one... 2023-03-03T08:03:07.425 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-03-03T08:03:07.425 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:03:07.425 INFO:tasks.ceph.mds.d:Restarting daemon 2023-03-03T08:03:07.425 INFO:tasks.ceph.mds.d:Stopping old one... 2023-03-03T08:03:07.426 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-03-03T08:03:07.426 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:03:07.444 INFO:tasks.ceph.mds.c:Stopped 2023-03-03T08:03:07.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-03-03T08:03:07.445 INFO:tasks.ceph.mds.a:Stopped 2023-03-03T08:03:07.446 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-03-03T08:03:07.454 INFO:tasks.ceph.mds.b:Stopped 2023-03-03T08:03:07.454 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-03-03T08:03:07.455 INFO:tasks.ceph.mds.d:Stopped 2023-03-03T08:03:07.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-03-03T08:03:07.490 INFO:tasks.ceph.mds.c:Started 2023-03-03T08:03:07.490 DEBUG:teuthology.parallel:result is None 2023-03-03T08:03:07.493 INFO:tasks.ceph.mds.a:Started 2023-03-03T08:03:07.494 DEBUG:teuthology.parallel:result is None 2023-03-03T08:03:07.500 INFO:tasks.ceph.mds.b:Started 2023-03-03T08:03:07.500 DEBUG:teuthology.parallel:result is None 2023-03-03T08:03:07.504 INFO:tasks.ceph.mds.d:Started 2023-03-03T08:03:07.504 DEBUG:teuthology.parallel:result is None 2023-03-03T08:03:07.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-03-03T08:03:07.685 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-03-03T08:03:07.689 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2023-03-03T08:03:07.696 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2023-03-03T08:03:07.697 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2023-03-03T08:03:07.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.765+0000 7f3c90bff700 1 Processor -- start 2023-03-03T08:03:07.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.766+0000 7f3c90bff700 1 -- start start 2023-03-03T08:03:07.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.767+0000 7f3c90bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c8c0d0270 0x7f3c8c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:07.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.767+0000 7f3c90bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8c0cf980 con 0x7f3c8c06b4b0 2023-03-03T08:03:07.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.767+0000 7f3c90bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8c0d1800 con 0x7f3c8c15ca70 2023-03-03T08:03:07.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.767+0000 7f3c90bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c0cd5b0 con 0x7f3c8c0d0270 2023-03-03T08:03:07.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.767+0000 7f3c83fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c8c0d0270 0x7f3c8c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:07.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.767+0000 7f3c8ad9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3c8c06b4b0 0x7f3c8c0d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41976/0 (socket says 172.21.15.98:41976) 2023-03-03T08:03:07.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.767+0000 7f3c8ad9d700 1 -- 172.21.15.98:0/735702987 learned_addr learned my addr 172.21.15.98:0/735702987 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:07.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.767+0000 7f3c8a59c700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f3c8c15ca70 0x7f3c8c0cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:60106/0 (socket says 172.21.15.98:60106) 2023-03-03T08:03:07.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.768+0000 7f3c83fff700 1 -- 172.21.15.98:0/735702987 >> v1:172.21.15.100:6789/0 conn(0x7f3c8c15ca70 legacy=0x7f3c8c0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:07.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.768+0000 7f3c89d9b700 1 -- 172.21.15.98:0/735702987 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671308301 0 0) 0x7f3c8c0d1800 con 0x7f3c8c15ca70 2023-03-03T08:03:07.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.768+0000 7f3c83fff700 1 -- 172.21.15.98:0/735702987 >> v1:172.21.15.98:6789/0 conn(0x7f3c8c06b4b0 legacy=0x7f3c8c0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:07.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.768+0000 7f3c83fff700 1 -- 172.21.15.98:0/735702987 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8c0cd900 con 0x7f3c8c0d0270 2023-03-03T08:03:07.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.768+0000 7f3c83fff700 1 --2- 172.21.15.98:0/735702987 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c8c0d0270 0x7f3c8c0cf440 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3c74009870 tx=0x7f3c74009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77bb1ea8dcc91ae9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:07.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.769+0000 7f3c89d9b700 1 -- 172.21.15.98:0/735702987 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74011830 con 0x7f3c8c0d0270 2023-03-03T08:03:07.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.769+0000 7f3c89d9b700 1 -- 172.21.15.98:0/735702987 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c74011990 con 0x7f3c8c0d0270 2023-03-03T08:03:07.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.769+0000 7f3c89d9b700 1 -- 172.21.15.98:0/735702987 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74011d20 con 0x7f3c8c0d0270 2023-03-03T08:03:07.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.769+0000 7f3c90bff700 1 -- 172.21.15.98:0/735702987 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c8c0d0270 msgr2=0x7f3c8c0cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:07.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.769+0000 7f3c90bff700 1 --2- 172.21.15.98:0/735702987 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c8c0d0270 0x7f3c8c0cf440 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3c74009870 tx=0x7f3c74009b80 comp rx=0 tx=0).stop 2023-03-03T08:03:07.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.769+0000 7f3c90bff700 1 -- 172.21.15.98:0/735702987 shutdown_connections 2023-03-03T08:03:07.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.769+0000 7f3c90bff700 1 --2- 172.21.15.98:0/735702987 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c8c0d0270 0x7f3c8c0cf440 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:07.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.769+0000 7f3c90bff700 1 -- 172.21.15.98:0/735702987 >> 172.21.15.98:0/735702987 conn(0x7f3c8c073e70 msgr2=0x7f3c8c06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:07.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.769+0000 7f3c90bff700 1 -- 172.21.15.98:0/735702987 shutdown_connections 2023-03-03T08:03:07.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.769+0000 7f3c90bff700 1 -- 172.21.15.98:0/735702987 wait complete. 2023-03-03T08:03:07.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.771+0000 7f3c90bff700 1 Processor -- start 2023-03-03T08:03:07.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.772+0000 7f3c90bff700 1 -- start start 2023-03-03T08:03:07.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c90bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c8c06b4b0 0x7f3c8c0ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:07.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c90bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c8c0d0270 0x7f3c8c15fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:07.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c90bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c8c15ca70 0x7f3c8c0cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:07.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c90bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c0cd400 con 0x7f3c8c06b4b0 2023-03-03T08:03:07.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c90bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c1f22d0 con 0x7f3c8c15ca70 2023-03-03T08:03:07.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c90bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c0cc880 con 0x7f3c8c0d0270 2023-03-03T08:03:07.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c8a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c8c06b4b0 0x7f3c8c0ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:07.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c8a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c8c06b4b0 0x7f3c8c0ca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51120/0 (socket says 172.21.15.98:51120) 2023-03-03T08:03:07.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c8a59c700 1 -- 172.21.15.98:0/44609030 learned_addr learned my addr 172.21.15.98:0/44609030 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:07.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c83fff700 1 --2- 172.21.15.98:0/44609030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c8c0d0270 0x7f3c8c15fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:07.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c8ad9d700 1 --2- 172.21.15.98:0/44609030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c8c15ca70 0x7f3c8c0cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:07.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c8a59c700 1 -- 172.21.15.98:0/44609030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c8c0d0270 msgr2=0x7f3c8c15fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:07.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c8a59c700 1 --2- 172.21.15.98:0/44609030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c8c0d0270 0x7f3c8c15fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:07.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c8a59c700 1 -- 172.21.15.98:0/44609030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c8c15ca70 msgr2=0x7f3c8c0cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:07.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c8a59c700 1 --2- 172.21.15.98:0/44609030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c8c15ca70 0x7f3c8c0cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:07.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.773+0000 7f3c8a59c700 1 -- 172.21.15.98:0/44609030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c740094a0 con 0x7f3c8c06b4b0 2023-03-03T08:03:07.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.774+0000 7f3c8a59c700 1 --2- 172.21.15.98:0/44609030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c8c06b4b0 0x7f3c8c0ca680 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c00c4d0 tx=0x7f3c7c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:07.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.774+0000 7f3c82ffd700 1 -- 172.21.15.98:0/44609030 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c004c80 con 0x7f3c8c06b4b0 2023-03-03T08:03:07.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.774+0000 7f3c82ffd700 1 -- 172.21.15.98:0/44609030 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7c004de0 con 0x7f3c8c06b4b0 2023-03-03T08:03:07.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.774+0000 7f3c90bff700 1 -- 172.21.15.98:0/44609030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8c0d1aa0 con 0x7f3c8c06b4b0 2023-03-03T08:03:07.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.774+0000 7f3c82ffd700 1 -- 172.21.15.98:0/44609030 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c007310 con 0x7f3c8c06b4b0 2023-03-03T08:03:07.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.774+0000 7f3c90bff700 1 -- 172.21.15.98:0/44609030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8c0cdbe0 con 0x7f3c8c06b4b0 2023-03-03T08:03:07.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.775+0000 7f3c90bff700 1 -- 172.21.15.98:0/44609030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8c15ce40 con 0x7f3c8c06b4b0 2023-03-03T08:03:07.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.777+0000 7f3c82ffd700 1 -- 172.21.15.98:0/44609030 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f3c7c01a020 con 0x7f3c8c06b4b0 2023-03-03T08:03:07.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.778+0000 7f3c82ffd700 1 --2- 172.21.15.98:0/44609030 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3c6c068380 0x7f3c6c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:07.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.778+0000 7f3c82ffd700 1 -- 172.21.15.98:0/44609030 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f3c7c085500 con 0x7f3c8c06b4b0 2023-03-03T08:03:07.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.778+0000 7f3c83fff700 1 --2- 172.21.15.98:0/44609030 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3c6c068380 0x7f3c6c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:07.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.779+0000 7f3c83fff700 1 --2- 172.21.15.98:0/44609030 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3c6c068380 0x7f3c6c06a830 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3c74009870 tx=0x7f3c74008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:07.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.782+0000 7f3c82ffd700 1 -- 172.21.15.98:0/44609030 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3c7c051f20 con 0x7f3c8c06b4b0 2023-03-03T08:03:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:07.991+0000 7f3c90bff700 1 -- 172.21.15.98:0/44609030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3c8c04e6e0 con 0x7f3c8c06b4b0 2023-03-03T08:03:08.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:08.831+0000 7f3c82ffd700 1 -- 172.21.15.98:0/44609030 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v107) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3c7c083c80 con 0x7f3c8c06b4b0 2023-03-03T08:03:08.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:08.834+0000 7f3c90bff700 1 -- 172.21.15.98:0/44609030 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3c6c068380 msgr2=0x7f3c6c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:08.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:08.834+0000 7f3c90bff700 1 --2- 172.21.15.98:0/44609030 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3c6c068380 0x7f3c6c06a830 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3c74009870 tx=0x7f3c74008030 comp rx=0 tx=0).stop 2023-03-03T08:03:08.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:08.834+0000 7f3c90bff700 1 -- 172.21.15.98:0/44609030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c8c06b4b0 msgr2=0x7f3c8c0ca680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:08.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:08.834+0000 7f3c90bff700 1 --2- 172.21.15.98:0/44609030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c8c06b4b0 0x7f3c8c0ca680 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c00c4d0 tx=0x7f3c7c00c7e0 comp rx=0 tx=0).stop 2023-03-03T08:03:08.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:08.834+0000 7f3c90bff700 1 -- 172.21.15.98:0/44609030 shutdown_connections 2023-03-03T08:03:08.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:08.834+0000 7f3c90bff700 1 --2- 172.21.15.98:0/44609030 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f3c6c068380 0x7f3c6c06a830 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:08.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:08.834+0000 7f3c90bff700 1 --2- 172.21.15.98:0/44609030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c8c06b4b0 0x7f3c8c0ca680 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:08.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:08.834+0000 7f3c90bff700 1 --2- 172.21.15.98:0/44609030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c8c15ca70 0x7f3c8c0cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:08.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:08.834+0000 7f3c90bff700 1 --2- 172.21.15.98:0/44609030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c8c0d0270 0x7f3c8c15fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:08.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:08.834+0000 7f3c90bff700 1 -- 172.21.15.98:0/44609030 >> 172.21.15.98:0/44609030 conn(0x7f3c8c073e70 msgr2=0x7f3c8c0d3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:08.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:08.834+0000 7f3c90bff700 1 -- 172.21.15.98:0/44609030 shutdown_connections 2023-03-03T08:03:08.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:08.834+0000 7f3c90bff700 1 -- 172.21.15.98:0/44609030 wait complete. 2023-03-03T08:03:08.837 INFO:teuthology.orchestra.run.smithi098.stderr: removed all blocklist entries 2023-03-03T08:03:08.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-03-03T08:03:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.105+0000 7f8d6599d700 1 Processor -- start 2023-03-03T08:03:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.106+0000 7f8d6599d700 1 -- start start 2023-03-03T08:03:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.107+0000 7f8d6599d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d60146360 0x7f8d60160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.107+0000 7f8d6599d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8d600b4ab0 0x7f8d600b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.107+0000 7f8d6599d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d601614c0 con 0x7f8d600731e0 2023-03-03T08:03:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.107+0000 7f8d6599d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d600b5460 con 0x7f8d600b4ab0 2023-03-03T08:03:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.107+0000 7f8d6599d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d600d10f0 con 0x7f8d60146360 2023-03-03T08:03:09.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.107+0000 7f8d5ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8d600b4ab0 0x7f8d600b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:09.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.108+0000 7f8d5ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8d600b4ab0 0x7f8d600b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51122/0 (socket says 172.21.15.98:51122) 2023-03-03T08:03:09.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.108+0000 7f8d5ffff700 1 -- 172.21.15.98:0/3509167415 learned_addr learned my addr 172.21.15.98:0/3509167415 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:09.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.108+0000 7f8d6499b700 1 --2- 172.21.15.98:0/3509167415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d60146360 0x7f8d60160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:09.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.108+0000 7f8d5ffff700 1 -- 172.21.15.98:0/3509167415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d60146360 msgr2=0x7f8d60160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:09.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.108+0000 7f8d5ffff700 1 --2- 172.21.15.98:0/3509167415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d60146360 0x7f8d60160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.108+0000 7f8d5ffff700 1 -- 172.21.15.98:0/3509167415 >> v1:172.21.15.100:6789/0 conn(0x7f8d600731e0 legacy=0x7f8d60073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:09.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.108+0000 7f8d5ffff700 1 -- 172.21.15.98:0/3509167415 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d600cdf20 con 0x7f8d600b4ab0 2023-03-03T08:03:09.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.108+0000 7f8d5ffff700 1 --2- 172.21.15.98:0/3509167415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d600b4ab0 0x7f8d600b4f20 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c0098f0 tx=0x7f8d4c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280fd9ba30ffca28 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:09.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.109+0000 7f8d5f7fe700 1 -- 172.21.15.98:0/3509167415 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c004230 con 0x7f8d600b4ab0 2023-03-03T08:03:09.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.109+0000 7f8d5f7fe700 1 -- 172.21.15.98:0/3509167415 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4c004390 con 0x7f8d600b4ab0 2023-03-03T08:03:09.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.109+0000 7f8d5f7fe700 1 -- 172.21.15.98:0/3509167415 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c0036a0 con 0x7f8d600b4ab0 2023-03-03T08:03:09.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.109+0000 7f8d6599d700 1 -- 172.21.15.98:0/3509167415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d600b4ab0 msgr2=0x7f8d600b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:09.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.109+0000 7f8d6599d700 1 --2- 172.21.15.98:0/3509167415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d600b4ab0 0x7f8d600b4f20 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c0098f0 tx=0x7f8d4c009c00 comp rx=0 tx=0).stop 2023-03-03T08:03:09.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.109+0000 7f8d6599d700 1 -- 172.21.15.98:0/3509167415 shutdown_connections 2023-03-03T08:03:09.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.109+0000 7f8d6599d700 1 --2- 172.21.15.98:0/3509167415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d60146360 0x7f8d60160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.109+0000 7f8d6599d700 1 --2- 172.21.15.98:0/3509167415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d600b4ab0 0x7f8d600b4f20 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.109+0000 7f8d6599d700 1 -- 172.21.15.98:0/3509167415 >> 172.21.15.98:0/3509167415 conn(0x7f8d6006b8e0 msgr2=0x7f8d600cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:09.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.109+0000 7f8d6599d700 1 -- 172.21.15.98:0/3509167415 shutdown_connections 2023-03-03T08:03:09.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.109+0000 7f8d6599d700 1 -- 172.21.15.98:0/3509167415 wait complete. 2023-03-03T08:03:09.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.112+0000 7f8d6599d700 1 Processor -- start 2023-03-03T08:03:09.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.113+0000 7f8d6599d700 1 -- start start 2023-03-03T08:03:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.114+0000 7f8d6599d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d600731e0 0x7f8d6015a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.114+0000 7f8d6599d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d600b4ab0 0x7f8d600b8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.114+0000 7f8d6599d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d60146360 0x7f8d600c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.114+0000 7f8d6599d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d600c6440 con 0x7f8d600731e0 2023-03-03T08:03:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.114+0000 7f8d6599d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d600caef0 con 0x7f8d60146360 2023-03-03T08:03:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.114+0000 7f8d6599d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d600cb1d0 con 0x7f8d600b4ab0 2023-03-03T08:03:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.114+0000 7f8d6499b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d600731e0 0x7f8d6015a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.114+0000 7f8d6499b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d600731e0 0x7f8d6015a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51134/0 (socket says 172.21.15.98:51134) 2023-03-03T08:03:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.114+0000 7f8d6499b700 1 -- 172.21.15.98:0/4108450411 learned_addr learned my addr 172.21.15.98:0/4108450411 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:09.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.114+0000 7f8d5ffff700 1 --2- 172.21.15.98:0/4108450411 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d600b4ab0 0x7f8d600b8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:09.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.114+0000 7f8d6519c700 1 --2- 172.21.15.98:0/4108450411 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d60146360 0x7f8d600c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:09.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.115+0000 7f8d6499b700 1 -- 172.21.15.98:0/4108450411 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d600b4ab0 msgr2=0x7f8d600b8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:09.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.115+0000 7f8d6499b700 1 --2- 172.21.15.98:0/4108450411 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d600b4ab0 0x7f8d600b8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.115+0000 7f8d6499b700 1 -- 172.21.15.98:0/4108450411 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d60146360 msgr2=0x7f8d600c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:09.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.115+0000 7f8d6499b700 1 --2- 172.21.15.98:0/4108450411 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d60146360 0x7f8d600c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.115+0000 7f8d6499b700 1 -- 172.21.15.98:0/4108450411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c009550 con 0x7f8d600731e0 2023-03-03T08:03:09.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.115+0000 7f8d6519c700 1 --2- 172.21.15.98:0/4108450411 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d60146360 0x7f8d600c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:09.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.115+0000 7f8d6499b700 1 --2- 172.21.15.98:0/4108450411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d600731e0 0x7f8d6015a490 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f8d5400d800 tx=0x7f8d5400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:09.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.115+0000 7f8d5d7fa700 1 -- 172.21.15.98:0/4108450411 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d540125f0 con 0x7f8d600731e0 2023-03-03T08:03:09.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.115+0000 7f8d5d7fa700 1 -- 172.21.15.98:0/4108450411 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d54012750 con 0x7f8d600731e0 2023-03-03T08:03:09.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.115+0000 7f8d6599d700 1 -- 172.21.15.98:0/4108450411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d600cd960 con 0x7f8d600731e0 2023-03-03T08:03:09.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.115+0000 7f8d5d7fa700 1 -- 172.21.15.98:0/4108450411 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54012b00 con 0x7f8d600731e0 2023-03-03T08:03:09.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.116+0000 7f8d6599d700 1 -- 172.21.15.98:0/4108450411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d600cd680 con 0x7f8d600731e0 2023-03-03T08:03:09.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.117+0000 7f8d6599d700 1 -- 172.21.15.98:0/4108450411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d600cdc40 con 0x7f8d600731e0 2023-03-03T08:03:09.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.118+0000 7f8d5d7fa700 1 -- 172.21.15.98:0/4108450411 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f8d54012c60 con 0x7f8d600731e0 2023-03-03T08:03:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.119+0000 7f8d5d7fa700 1 --2- 172.21.15.98:0/4108450411 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8d44068380 0x7f8d4406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.119+0000 7f8d5d7fa700 1 -- 172.21.15.98:0/4108450411 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8d54085f60 con 0x7f8d600731e0 2023-03-03T08:03:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.120+0000 7f8d5ffff700 1 --2- 172.21.15.98:0/4108450411 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8d44068380 0x7f8d4406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:09.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.121+0000 7f8d5ffff700 1 --2- 172.21.15.98:0/4108450411 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8d44068380 0x7f8d4406a830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c009130 tx=0x7f8d4c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:09.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.123+0000 7f8d5d7fa700 1 -- 172.21.15.98:0/4108450411 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8d540529a0 con 0x7f8d600731e0 2023-03-03T08:03:09.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.386+0000 7f8d6599d700 1 -- 172.21.15.98:0/4108450411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8d60068300 con 0x7f8d600731e0 2023-03-03T08:03:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.386+0000 7f8d5d7fa700 1 -- 172.21.15.98:0/4108450411 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5298 (secure 0 0 0) 0x7f8d54055fc0 con 0x7f8d600731e0 2023-03-03T08:03:09.388 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:03:09.388 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:03:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_dump": [ 2023-03-03T08:03:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.a", 2023-03-03T08:03:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkqFdgHxAAuwiGkaNnr5NplmhniDYAEQ==", 2023-03-03T08:03:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:03:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:03:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:03:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:03:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.b", 2023-03-03T08:03:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFk0x8kBhAA5JDo/wmI4650J9tUnvXVzw==", 2023-03-03T08:03:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:03:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:03:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:03:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:03:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.c", 2023-03-03T08:03:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkKoRwMBAAfkWGDUsqNgqfkscl0aBX1A==", 2023-03-03T08:03:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:03:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:03:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:03:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:03:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.d", 2023-03-03T08:03:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFkK9SKFxAAFSCozGfyRwrv46QPitRxTQ==", 2023-03-03T08:03:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:03:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:03:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:03:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:03:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.0", 2023-03-03T08:03:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDBpwFkTFzTERAAGHrFKuBlXIEfaYD7vs3ZHA==", 2023-03-03T08:03:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:03:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:03:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:03:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.1", 2023-03-03T08:03:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDEpwFkVWObLBAAW4acTvNbmscIe18NXVgRYg==", 2023-03-03T08:03:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:03:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:03:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:03:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.2", 2023-03-03T08:03:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDIpwFkK8lgChAAHVD5btBivUpcapSgDp2E0w==", 2023-03-03T08:03:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:03:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:03:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:03:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.3", 2023-03-03T08:03:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDLpwFksrb8IxAAN+CP1O3DJ1zDy+qyGNor2Q==", 2023-03-03T08:03:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:03:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:03:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:03:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.4", 2023-03-03T08:03:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDSpwFkkWGHIxAA4wejA0w05JPIUghEAVG/Rg==", 2023-03-03T08:03:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:03:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:03:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:03:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.5", 2023-03-03T08:03:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDWpwFknHmGAhAA6VM16luaykZfJEzr+yDZZA==", 2023-03-03T08:03:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:03:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:03:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:03:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.6", 2023-03-03T08:03:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDZpwFkcitNDRAAOz04YGTzx3Phenp0n8JeEA==", 2023-03-03T08:03:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:03:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:03:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:03:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.7", 2023-03-03T08:03:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDcpwFkrjmIFxAAbWeB2+8VQgyTplW+fk47OA==", 2023-03-03T08:03:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-03-03T08:03:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-03-03T08:03:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:03:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.0", 2023-03-03T08:03:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFk02lsJhAA0cH/evMnWdZN6DdBGCLjjg==", 2023-03-03T08:03:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:03:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow r", 2023-03-03T08:03:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rw pool=cephfs_data" 2023-03-03T08:03:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.1", 2023-03-03T08:03:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC9pwFkRqXrABAArm/01O5MtSTQHWQWKqNQEg==", 2023-03-03T08:03:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:03:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T08:03:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T08:03:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T08:03:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.2", 2023-03-03T08:03:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC8pwFkyKp3MRAA8Eh41TZNsO6/8VhAWCpNMQ==", 2023-03-03T08:03:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:03:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T08:03:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T08:03:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T08:03:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.3", 2023-03-03T08:03:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC9pwFk14bSCxAAy2nXc9G50W4lH5GtH84hpQ==", 2023-03-03T08:03:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-03-03T08:03:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-03-03T08:03:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-03-03T08:03:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-03-03T08:03:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.admin", 2023-03-03T08:03:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC6pwFkokmiEBAAFwi8rhPmk4Zyk95p760HDg==", 2023-03-03T08:03:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T08:03:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-03-03T08:03:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-03-03T08:03:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:03:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mgr.x", 2023-03-03T08:03:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkvdMQFBAAMi+x7psn5CDpd//d6jN2VA==", 2023-03-03T08:03:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T08:03:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile mgr", 2023-03-03T08:03:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:03:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mgr.y", 2023-03-03T08:03:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7pwFkfYO5BxAAFW/elbdoqVXBkUIKrAvYsg==", 2023-03-03T08:03:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-03-03T08:03:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-03-03T08:03:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile mgr", 2023-03-03T08:03:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-03-03T08:03:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-03-03T08:03:09.414 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:03:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.389+0000 7f8d6599d700 1 -- 172.21.15.98:0/4108450411 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8d44068380 msgr2=0x7f8d4406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.389+0000 7f8d6599d700 1 --2- 172.21.15.98:0/4108450411 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8d44068380 0x7f8d4406a830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c009130 tx=0x7f8d4c008030 comp rx=0 tx=0).stop 2023-03-03T08:03:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.389+0000 7f8d6599d700 1 -- 172.21.15.98:0/4108450411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d600731e0 msgr2=0x7f8d6015a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.389+0000 7f8d6599d700 1 --2- 172.21.15.98:0/4108450411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d600731e0 0x7f8d6015a490 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f8d5400d800 tx=0x7f8d5400db10 comp rx=0 tx=0).stop 2023-03-03T08:03:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.389+0000 7f8d6599d700 1 -- 172.21.15.98:0/4108450411 shutdown_connections 2023-03-03T08:03:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.389+0000 7f8d6599d700 1 --2- 172.21.15.98:0/4108450411 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8d44068380 0x7f8d4406a830 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.389+0000 7f8d6599d700 1 --2- 172.21.15.98:0/4108450411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d600731e0 0x7f8d6015a490 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.389+0000 7f8d6599d700 1 --2- 172.21.15.98:0/4108450411 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d60146360 0x7f8d600c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.389+0000 7f8d6599d700 1 --2- 172.21.15.98:0/4108450411 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d600b4ab0 0x7f8d600b8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.389+0000 7f8d6599d700 1 -- 172.21.15.98:0/4108450411 >> 172.21.15.98:0/4108450411 conn(0x7f8d6006b8e0 msgr2=0x7f8d60073620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.389+0000 7f8d6599d700 1 -- 172.21.15.98:0/4108450411 shutdown_connections 2023-03-03T08:03:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.390+0000 7f8d6599d700 1 -- 172.21.15.98:0/4108450411 wait complete. 2023-03-03T08:03:09.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:03:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.661+0000 7f31151ff700 1 Processor -- start 2023-03-03T08:03:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.661+0000 7f31151ff700 1 -- start start 2023-03-03T08:03:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.662+0000 7f31151ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31100b8b90 0x7f31100cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.662+0000 7f31151ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f31100b5410 0x7f31100d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.662+0000 7f31151ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31100d0ef0 con 0x7f31100b87c0 2023-03-03T08:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.662+0000 7f31151ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31100d0600 con 0x7f31100b5410 2023-03-03T08:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.662+0000 7f31151ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31100cd970 con 0x7f31100b8b90 2023-03-03T08:03:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.663+0000 7f310f59e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f31100b5410 0x7f31100d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.663+0000 7f310f59e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f31100b5410 0x7f31100d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:59296/0 (socket says 172.21.15.98:59296) 2023-03-03T08:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.663+0000 7f310f59e700 1 -- 172.21.15.98:0/1892165030 learned_addr learned my addr 172.21.15.98:0/1892165030 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.663+0000 7f310e59c700 1 --2- 172.21.15.98:0/1892165030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31100b8b90 0x7f31100cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.663+0000 7f310f59e700 1 -- 172.21.15.98:0/1892165030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31100b8b90 msgr2=0x7f31100cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.663+0000 7f310f59e700 1 --2- 172.21.15.98:0/1892165030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31100b8b90 0x7f31100cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.664+0000 7f310f59e700 1 -- 172.21.15.98:0/1892165030 >> v1:172.21.15.100:6789/0 conn(0x7f31100b87c0 legacy=0x7f31100b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.664+0000 7f310f59e700 1 -- 172.21.15.98:0/1892165030 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31100cdc50 con 0x7f31100b5410 2023-03-03T08:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.664+0000 7f310e59c700 1 --2- 172.21.15.98:0/1892165030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31100b8b90 0x7f31100cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.664+0000 7f310dd9b700 1 -- 172.21.15.98:0/1892165030 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515442870 0 0) 0x7f31100d0ef0 con 0x7f31100b87c0 2023-03-03T08:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.664+0000 7f310f59e700 1 --2- 172.21.15.98:0/1892165030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31100b5410 0x7f31100d13d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3108009e50 tx=0x7f310800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a323cc5c79f3fc2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.664+0000 7f310dd9b700 1 -- 172.21.15.98:0/1892165030 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310800e070 con 0x7f31100b5410 2023-03-03T08:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.664+0000 7f310dd9b700 1 -- 172.21.15.98:0/1892165030 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310800b910 con 0x7f31100b5410 2023-03-03T08:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.664+0000 7f310dd9b700 1 -- 172.21.15.98:0/1892165030 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310800bcc0 con 0x7f31100b5410 2023-03-03T08:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.665+0000 7f31151ff700 1 -- 172.21.15.98:0/1892165030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31100b5410 msgr2=0x7f31100d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.665+0000 7f31151ff700 1 --2- 172.21.15.98:0/1892165030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31100b5410 0x7f31100d13d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3108009e50 tx=0x7f310800b330 comp rx=0 tx=0).stop 2023-03-03T08:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.665+0000 7f31151ff700 1 -- 172.21.15.98:0/1892165030 shutdown_connections 2023-03-03T08:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.665+0000 7f31151ff700 1 --2- 172.21.15.98:0/1892165030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31100b5410 0x7f31100d13d0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.665+0000 7f31151ff700 1 --2- 172.21.15.98:0/1892165030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31100b8b90 0x7f31100cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.665+0000 7f31151ff700 1 -- 172.21.15.98:0/1892165030 >> 172.21.15.98:0/1892165030 conn(0x7f311006aee0 msgr2=0x7f311006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.665+0000 7f31151ff700 1 -- 172.21.15.98:0/1892165030 shutdown_connections 2023-03-03T08:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.665+0000 7f31151ff700 1 -- 172.21.15.98:0/1892165030 wait complete. 2023-03-03T08:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.667+0000 7f31151ff700 1 Processor -- start 2023-03-03T08:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.667+0000 7f31151ff700 1 -- start start 2023-03-03T08:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.668+0000 7f31151ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31100b5410 0x7f31100a9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.668+0000 7f31151ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31100b87c0 0x7f311014bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.668+0000 7f31151ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31100b8b90 0x7f311014c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.668+0000 7f31151ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31100ccca0 con 0x7f31100b8b90 2023-03-03T08:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.668+0000 7f31151ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311014f810 con 0x7f31100b87c0 2023-03-03T08:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.668+0000 7f31151ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31100cc090 con 0x7f31100b5410 2023-03-03T08:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.668+0000 7f310f59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31100b8b90 0x7f311014c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.669+0000 7f310f59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31100b8b90 0x7f311014c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51146/0 (socket says 172.21.15.98:51146) 2023-03-03T08:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.669+0000 7f310f59e700 1 -- 172.21.15.98:0/1873698269 learned_addr learned my addr 172.21.15.98:0/1873698269 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.669+0000 7f310e59c700 1 --2- 172.21.15.98:0/1873698269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31100b87c0 0x7f311014bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.669+0000 7f310ed9d700 1 --2- 172.21.15.98:0/1873698269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31100b5410 0x7f31100a9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.669+0000 7f310f59e700 1 -- 172.21.15.98:0/1873698269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31100b5410 msgr2=0x7f31100a9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.669+0000 7f310f59e700 1 --2- 172.21.15.98:0/1873698269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31100b5410 0x7f31100a9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.669+0000 7f310f59e700 1 -- 172.21.15.98:0/1873698269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31100b87c0 msgr2=0x7f311014bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.669+0000 7f310f59e700 1 --2- 172.21.15.98:0/1873698269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31100b87c0 0x7f311014bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.669+0000 7f310f59e700 1 -- 172.21.15.98:0/1873698269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31040094a0 con 0x7f31100b8b90 2023-03-03T08:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.669+0000 7f310f59e700 1 --2- 172.21.15.98:0/1873698269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31100b8b90 0x7f311014c300 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f310800b680 tx=0x7f3108004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.670+0000 7f30f77fe700 1 -- 172.21.15.98:0/1873698269 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31080048b0 con 0x7f31100b8b90 2023-03-03T08:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.670+0000 7f30f77fe700 1 -- 172.21.15.98:0/1873698269 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31080099b0 con 0x7f31100b8b90 2023-03-03T08:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.670+0000 7f31151ff700 1 -- 172.21.15.98:0/1873698269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3108009b30 con 0x7f31100b8b90 2023-03-03T08:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.670+0000 7f30f77fe700 1 -- 172.21.15.98:0/1873698269 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310801c680 con 0x7f31100b8b90 2023-03-03T08:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.670+0000 7f31151ff700 1 -- 172.21.15.98:0/1873698269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31100cdf30 con 0x7f31100b8b90 2023-03-03T08:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.671+0000 7f31151ff700 1 -- 172.21.15.98:0/1873698269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31100ce0c0 con 0x7f31100b8b90 2023-03-03T08:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.672+0000 7f30f77fe700 1 -- 172.21.15.98:0/1873698269 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f310802c070 con 0x7f31100b8b90 2023-03-03T08:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.673+0000 7f30f77fe700 1 --2- 172.21.15.98:0/1873698269 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f30fc068380 0x7f30fc06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.674+0000 7f30f77fe700 1 -- 172.21.15.98:0/1873698269 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f31080874d0 con 0x7f31100b8b90 2023-03-03T08:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.674+0000 7f310ed9d700 1 --2- 172.21.15.98:0/1873698269 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f30fc068380 0x7f30fc06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.675+0000 7f310ed9d700 1 --2- 172.21.15.98:0/1873698269 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f30fc068380 0x7f30fc06a830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3100007fd0 tx=0x7f3100009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.677+0000 7f30f77fe700 1 -- 172.21.15.98:0/1873698269 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3108053f40 con 0x7f31100b8b90 2023-03-03T08:03:09.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.874+0000 7f31151ff700 1 -- 172.21.15.98:0/1873698269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31100729b0 con 0x7f31100b8b90 2023-03-03T08:03:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.875+0000 7f30f77fe700 1 -- 172.21.15.98:0/1873698269 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+14273 (secure 0 0 0) 0x7f3108057560 con 0x7f31100b8b90 2023-03-03T08:03:09.877 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:03:09.877 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":107,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:03:08.821686+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:03:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.878+0000 7f31151ff700 1 -- 172.21.15.98:0/1873698269 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f30fc068380 msgr2=0x7f30fc06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.878+0000 7f31151ff700 1 --2- 172.21.15.98:0/1873698269 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f30fc068380 0x7f30fc06a830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3100007fd0 tx=0x7f3100009040 comp rx=0 tx=0).stop 2023-03-03T08:03:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.878+0000 7f31151ff700 1 -- 172.21.15.98:0/1873698269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31100b8b90 msgr2=0x7f311014c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.878+0000 7f31151ff700 1 --2- 172.21.15.98:0/1873698269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31100b8b90 0x7f311014c300 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f310800b680 tx=0x7f3108004ad0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.878+0000 7f31151ff700 1 -- 172.21.15.98:0/1873698269 shutdown_connections 2023-03-03T08:03:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.878+0000 7f31151ff700 1 --2- 172.21.15.98:0/1873698269 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f30fc068380 0x7f30fc06a830 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.878+0000 7f31151ff700 1 --2- 172.21.15.98:0/1873698269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31100b8b90 0x7f311014c300 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.878+0000 7f31151ff700 1 --2- 172.21.15.98:0/1873698269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31100b5410 0x7f31100a9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.878+0000 7f31151ff700 1 --2- 172.21.15.98:0/1873698269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31100b87c0 0x7f311014bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:09.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.878+0000 7f31151ff700 1 -- 172.21.15.98:0/1873698269 >> 172.21.15.98:0/1873698269 conn(0x7f311006aee0 msgr2=0x7f31100d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:09.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.878+0000 7f31151ff700 1 -- 172.21.15.98:0/1873698269 shutdown_connections 2023-03-03T08:03:09.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:09.878+0000 7f31151ff700 1 -- 172.21.15.98:0/1873698269 wait complete. 2023-03-03T08:03:09.907 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:03:09.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:03:10.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.153+0000 7f82c09ff700 1 Processor -- start 2023-03-03T08:03:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.154+0000 7f82c09ff700 1 -- start start 2023-03-03T08:03:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.155+0000 7f82c09ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f82bc0b8d10 0x7f82bc0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.155+0000 7f82c09ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82bc0b26d0 0x7f82bc0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:10.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.155+0000 7f82c09ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc0cd7d0 con 0x7f82bc0b5140 2023-03-03T08:03:10.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.155+0000 7f82c09ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82bc0b55c0 con 0x7f82bc0b8d10 2023-03-03T08:03:10.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.155+0000 7f82c09ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc0b90e0 con 0x7f82bc0b26d0 2023-03-03T08:03:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.156+0000 7f82bad9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f82bc0b5140 0x7f82bc160520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41986/0 (socket says 172.21.15.98:41986) 2023-03-03T08:03:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.156+0000 7f82bad9d700 1 -- 172.21.15.98:0/4070597168 learned_addr learned my addr 172.21.15.98:0/4070597168 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.156+0000 7f82ba59c700 1 --2- 172.21.15.98:0/4070597168 >> v2:172.21.15.100:3300/0 conn(0x7f82bc0b8d10 0x7f82bc0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.156+0000 7f82b9d9b700 1 --2- 172.21.15.98:0/4070597168 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82bc0b26d0 0x7f82bc0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.156+0000 7f82b959a700 1 -- 172.21.15.98:0/4070597168 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566994486 0 0) 0x7f82bc0cd7d0 con 0x7f82bc0b5140 2023-03-03T08:03:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.156+0000 7f82b959a700 1 -- 172.21.15.98:0/4070597168 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a40035b0 con 0x7f82bc0b5140 2023-03-03T08:03:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.156+0000 7f82ba59c700 1 -- 172.21.15.98:0/4070597168 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82bc0b26d0 msgr2=0x7f82bc0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.156+0000 7f82ba59c700 1 --2- 172.21.15.98:0/4070597168 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82bc0b26d0 0x7f82bc0b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.156+0000 7f82ba59c700 1 -- 172.21.15.98:0/4070597168 >> v1:172.21.15.98:6789/0 conn(0x7f82bc0b5140 legacy=0x7f82bc160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.156+0000 7f82ba59c700 1 -- 172.21.15.98:0/4070597168 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc0cda20 con 0x7f82bc0b8d10 2023-03-03T08:03:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.157+0000 7f82ba59c700 1 --2- 172.21.15.98:0/4070597168 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82bc0b8d10 0x7f82bc0cffc0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f82ac009840 tx=0x7f82ac009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de2e011c43e15c21 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.157+0000 7f82b959a700 1 -- 172.21.15.98:0/4070597168 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac011830 con 0x7f82bc0b8d10 2023-03-03T08:03:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.157+0000 7f82b959a700 1 -- 172.21.15.98:0/4070597168 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ac011990 con 0x7f82bc0b8d10 2023-03-03T08:03:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.157+0000 7f82b959a700 1 -- 172.21.15.98:0/4070597168 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac011d20 con 0x7f82bc0b8d10 2023-03-03T08:03:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.158+0000 7f82c09ff700 1 -- 172.21.15.98:0/4070597168 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82bc0b8d10 msgr2=0x7f82bc0cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.158+0000 7f82c09ff700 1 --2- 172.21.15.98:0/4070597168 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82bc0b8d10 0x7f82bc0cffc0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f82ac009840 tx=0x7f82ac009b50 comp rx=0 tx=0).stop 2023-03-03T08:03:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.158+0000 7f82c09ff700 1 -- 172.21.15.98:0/4070597168 shutdown_connections 2023-03-03T08:03:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.158+0000 7f82c09ff700 1 --2- 172.21.15.98:0/4070597168 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82bc0b8d10 0x7f82bc0cffc0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.158+0000 7f82c09ff700 1 --2- 172.21.15.98:0/4070597168 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82bc0b26d0 0x7f82bc0b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.158+0000 7f82c09ff700 1 -- 172.21.15.98:0/4070597168 >> 172.21.15.98:0/4070597168 conn(0x7f82bc073e70 msgr2=0x7f82bc06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.158+0000 7f82c09ff700 1 -- 172.21.15.98:0/4070597168 shutdown_connections 2023-03-03T08:03:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.158+0000 7f82c09ff700 1 -- 172.21.15.98:0/4070597168 wait complete. 2023-03-03T08:03:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.160+0000 7f82c09ff700 1 Processor -- start 2023-03-03T08:03:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.160+0000 7f82c09ff700 1 -- start start 2023-03-03T08:03:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.161+0000 7f82c09ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82bc0b26d0 0x7f82bc151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.161+0000 7f82c09ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82bc0b5140 0x7f82bc0b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.161+0000 7f82c09ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82bc0b8d10 0x7f82bc0b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.161+0000 7f82c09ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc1f25f0 con 0x7f82bc0b8d10 2023-03-03T08:03:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.161+0000 7f82c09ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc15f0a0 con 0x7f82bc0b5140 2023-03-03T08:03:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.161+0000 7f82c09ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc15f380 con 0x7f82bc0b26d0 2023-03-03T08:03:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.162+0000 7f82bad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82bc0b8d10 0x7f82bc0b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.162+0000 7f82bad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82bc0b8d10 0x7f82bc0b40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51150/0 (socket says 172.21.15.98:51150) 2023-03-03T08:03:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.162+0000 7f82bad9d700 1 -- 172.21.15.98:0/1353791975 learned_addr learned my addr 172.21.15.98:0/1353791975 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.162+0000 7f82ba59c700 1 --2- 172.21.15.98:0/1353791975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82bc0b26d0 0x7f82bc151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.162+0000 7f82b9d9b700 1 --2- 172.21.15.98:0/1353791975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82bc0b5140 0x7f82bc0b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:10.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.162+0000 7f82bad9d700 1 -- 172.21.15.98:0/1353791975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82bc0b26d0 msgr2=0x7f82bc151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:10.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.162+0000 7f82bad9d700 1 --2- 172.21.15.98:0/1353791975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82bc0b26d0 0x7f82bc151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.162+0000 7f82bad9d700 1 -- 172.21.15.98:0/1353791975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82bc0b5140 msgr2=0x7f82bc0b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.162+0000 7f82bad9d700 1 --2- 172.21.15.98:0/1353791975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82bc0b5140 0x7f82bc0b3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.162+0000 7f82bad9d700 1 -- 172.21.15.98:0/1353791975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac0094a0 con 0x7f82bc0b8d10 2023-03-03T08:03:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.162+0000 7f82ba59c700 1 --2- 172.21.15.98:0/1353791975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82bc0b26d0 0x7f82bc151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.162+0000 7f82bad9d700 1 --2- 172.21.15.98:0/1353791975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82bc0b8d10 0x7f82bc0b40b0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f82b400caa0 tx=0x7f82b400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.163+0000 7f82aaffd700 1 -- 172.21.15.98:0/1353791975 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b400aa80 con 0x7f82bc0b8d10 2023-03-03T08:03:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.163+0000 7f82aaffd700 1 -- 172.21.15.98:0/1353791975 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b400abe0 con 0x7f82bc0b8d10 2023-03-03T08:03:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.163+0000 7f82c09ff700 1 -- 172.21.15.98:0/1353791975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82bc0d0e80 con 0x7f82bc0b8d10 2023-03-03T08:03:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.163+0000 7f82aaffd700 1 -- 172.21.15.98:0/1353791975 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b4012400 con 0x7f82bc0b8d10 2023-03-03T08:03:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.163+0000 7f82c09ff700 1 -- 172.21.15.98:0/1353791975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82bc0cdd00 con 0x7f82bc0b8d10 2023-03-03T08:03:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.164+0000 7f82c09ff700 1 -- 172.21.15.98:0/1353791975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82bc0cdfe0 con 0x7f82bc0b8d10 2023-03-03T08:03:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.165+0000 7f82aaffd700 1 -- 172.21.15.98:0/1353791975 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f82b4019030 con 0x7f82bc0b8d10 2023-03-03T08:03:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.166+0000 7f82aaffd700 1 --2- 172.21.15.98:0/1353791975 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f829c068380 0x7f829c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.167+0000 7f82aaffd700 1 -- 172.21.15.98:0/1353791975 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f82b4085660 con 0x7f82bc0b8d10 2023-03-03T08:03:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.167+0000 7f82ba59c700 1 --2- 172.21.15.98:0/1353791975 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f829c068380 0x7f829c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:10.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.168+0000 7f82ba59c700 1 --2- 172.21.15.98:0/1353791975 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f829c068380 0x7f829c06a830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f82ac000c00 tx=0x7f82ac008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:10.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.170+0000 7f82aaffd700 1 -- 172.21.15.98:0/1353791975 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f82b40520d0 con 0x7f82bc0b8d10 2023-03-03T08:03:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.361+0000 7f82c09ff700 1 -- 172.21.15.98:0/1353791975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f82bc0cf4d0 con 0x7f82bc0b8d10 2023-03-03T08:03:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.362+0000 7f82aaffd700 1 -- 172.21.15.98:0/1353791975 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f82b4083e90 con 0x7f82bc0b8d10 2023-03-03T08:03:10.363 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:03:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.364+0000 7f82c09ff700 1 -- 172.21.15.98:0/1353791975 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f829c068380 msgr2=0x7f829c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.364+0000 7f82c09ff700 1 --2- 172.21.15.98:0/1353791975 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f829c068380 0x7f829c06a830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f82ac000c00 tx=0x7f82ac008030 comp rx=0 tx=0).stop 2023-03-03T08:03:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.364+0000 7f82c09ff700 1 -- 172.21.15.98:0/1353791975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82bc0b8d10 msgr2=0x7f82bc0b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.364+0000 7f82c09ff700 1 --2- 172.21.15.98:0/1353791975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82bc0b8d10 0x7f82bc0b40b0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f82b400caa0 tx=0x7f82b400ce60 comp rx=0 tx=0).stop 2023-03-03T08:03:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.364+0000 7f82c09ff700 1 -- 172.21.15.98:0/1353791975 shutdown_connections 2023-03-03T08:03:10.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.364+0000 7f82c09ff700 1 --2- 172.21.15.98:0/1353791975 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f829c068380 0x7f829c06a830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.364+0000 7f82c09ff700 1 --2- 172.21.15.98:0/1353791975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82bc0b8d10 0x7f82bc0b40b0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.364+0000 7f82c09ff700 1 --2- 172.21.15.98:0/1353791975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82bc0b26d0 0x7f82bc151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.364+0000 7f82c09ff700 1 --2- 172.21.15.98:0/1353791975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82bc0b5140 0x7f82bc0b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.364+0000 7f82c09ff700 1 -- 172.21.15.98:0/1353791975 >> 172.21.15.98:0/1353791975 conn(0x7f82bc073e70 msgr2=0x7f82bc0d3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:10.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.365+0000 7f82c09ff700 1 -- 172.21.15.98:0/1353791975 shutdown_connections 2023-03-03T08:03:10.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.365+0000 7f82c09ff700 1 -- 172.21.15.98:0/1353791975 wait complete. 2023-03-03T08:03:10.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:03:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.640+0000 7fb65219d700 1 Processor -- start 2023-03-03T08:03:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.641+0000 7fb65219d700 1 -- start start 2023-03-03T08:03:10.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.642+0000 7fb65219d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb64c0cf570 0x7fb64c15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:10.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.642+0000 7fb65219d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb64c15db30 0x7fb64c0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:10.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.642+0000 7fb65219d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64c0cd780 con 0x7fb64c1609b0 2023-03-03T08:03:10.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.642+0000 7fb65219d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb64c0cebb0 con 0x7fb64c15db30 2023-03-03T08:03:10.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.642+0000 7fb65219d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c0cda60 con 0x7fb64c0cf570 2023-03-03T08:03:10.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.642+0000 7fb65199c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb64c15db30 0x7fb64c0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:10.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.643+0000 7fb65119b700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fb64c1609b0 0x7fb64c160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:60142/0 (socket says 172.21.15.98:60142) 2023-03-03T08:03:10.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.643+0000 7fb65119b700 1 -- 172.21.15.98:0/2665705540 learned_addr learned my addr 172.21.15.98:0/2665705540 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:10.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.643+0000 7fb65099a700 1 --2- 172.21.15.98:0/2665705540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb64c0cf570 0x7fb64c15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:10.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.643+0000 7fb65199c700 1 -- 172.21.15.98:0/2665705540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb64c0cf570 msgr2=0x7fb64c15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:10.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.643+0000 7fb65199c700 1 --2- 172.21.15.98:0/2665705540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb64c0cf570 0x7fb64c15d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.643+0000 7fb65199c700 1 -- 172.21.15.98:0/2665705540 >> v1:172.21.15.100:6789/0 conn(0x7fb64c1609b0 legacy=0x7fb64c160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:10.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.643+0000 7fb65199c700 1 -- 172.21.15.98:0/2665705540 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64c0cdd40 con 0x7fb64c15db30 2023-03-03T08:03:10.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.643+0000 7fb643fff700 1 -- 172.21.15.98:0/2665705540 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339907422 0 0) 0x7fb64c0cd780 con 0x7fb64c1609b0 2023-03-03T08:03:10.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.643+0000 7fb65199c700 1 --2- 172.21.15.98:0/2665705540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c15db30 0x7fb64c0b4aa0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fb644009d90 tx=0x7fb64400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a17e03692a2e6a31 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:10.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.644+0000 7fb643fff700 1 -- 172.21.15.98:0/2665705540 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64400e070 con 0x7fb64c15db30 2023-03-03T08:03:10.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.644+0000 7fb643fff700 1 -- 172.21.15.98:0/2665705540 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64400bb20 con 0x7fb64c15db30 2023-03-03T08:03:10.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.644+0000 7fb643fff700 1 -- 172.21.15.98:0/2665705540 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644012430 con 0x7fb64c15db30 2023-03-03T08:03:10.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.644+0000 7fb65219d700 1 -- 172.21.15.98:0/2665705540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c15db30 msgr2=0x7fb64c0b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:10.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.644+0000 7fb65219d700 1 --2- 172.21.15.98:0/2665705540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c15db30 0x7fb64c0b4aa0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fb644009d90 tx=0x7fb64400b330 comp rx=0 tx=0).stop 2023-03-03T08:03:10.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.644+0000 7fb65219d700 1 -- 172.21.15.98:0/2665705540 shutdown_connections 2023-03-03T08:03:10.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.644+0000 7fb65219d700 1 --2- 172.21.15.98:0/2665705540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c15db30 0x7fb64c0b4aa0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.644+0000 7fb65219d700 1 --2- 172.21.15.98:0/2665705540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb64c0cf570 0x7fb64c15d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.644+0000 7fb65219d700 1 -- 172.21.15.98:0/2665705540 >> 172.21.15.98:0/2665705540 conn(0x7fb64c06b880 msgr2=0x7fb64c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:10.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.644+0000 7fb65219d700 1 -- 172.21.15.98:0/2665705540 shutdown_connections 2023-03-03T08:03:10.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.644+0000 7fb65219d700 1 -- 172.21.15.98:0/2665705540 wait complete. 2023-03-03T08:03:10.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.646+0000 7fb65219d700 1 Processor -- start 2023-03-03T08:03:10.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.647+0000 7fb65219d700 1 -- start start 2023-03-03T08:03:10.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.647+0000 7fb65219d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb64c0cf570 0x7fb64c1f15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:10.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.648+0000 7fb65219d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb64c15db30 0x7fb64c15be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:10.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.648+0000 7fb65219d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c1609b0 0x7fb64c15c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:10.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.648+0000 7fb65219d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c152010 con 0x7fb64c1609b0 2023-03-03T08:03:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.648+0000 7fb65219d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c1e6910 con 0x7fb64c15db30 2023-03-03T08:03:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.648+0000 7fb65219d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c1e6bf0 con 0x7fb64c0cf570 2023-03-03T08:03:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.648+0000 7fb65199c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c1609b0 0x7fb64c15c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.648+0000 7fb65199c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c1609b0 0x7fb64c15c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51172/0 (socket says 172.21.15.98:51172) 2023-03-03T08:03:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.648+0000 7fb65199c700 1 -- 172.21.15.98:0/4117959652 learned_addr learned my addr 172.21.15.98:0/4117959652 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.648+0000 7fb65119b700 1 --2- 172.21.15.98:0/4117959652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb64c0cf570 0x7fb64c1f15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.648+0000 7fb65099a700 1 --2- 172.21.15.98:0/4117959652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb64c15db30 0x7fb64c15be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.648+0000 7fb65199c700 1 -- 172.21.15.98:0/4117959652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb64c0cf570 msgr2=0x7fb64c1f15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.648+0000 7fb65199c700 1 --2- 172.21.15.98:0/4117959652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb64c0cf570 0x7fb64c1f15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.648+0000 7fb65199c700 1 -- 172.21.15.98:0/4117959652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb64c15db30 msgr2=0x7fb64c15be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.648+0000 7fb65199c700 1 --2- 172.21.15.98:0/4117959652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb64c15db30 0x7fb64c15be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.648+0000 7fb65199c700 1 -- 172.21.15.98:0/4117959652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb644009a70 con 0x7fb64c1609b0 2023-03-03T08:03:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.649+0000 7fb65199c700 1 --2- 172.21.15.98:0/4117959652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c1609b0 0x7fb64c15c380 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fb64c074380 tx=0x7fb644004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.649+0000 7fb641ffb700 1 -- 172.21.15.98:0/4117959652 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6440098f0 con 0x7fb64c1609b0 2023-03-03T08:03:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.649+0000 7fb641ffb700 1 -- 172.21.15.98:0/4117959652 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb644020070 con 0x7fb64c1609b0 2023-03-03T08:03:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.649+0000 7fb641ffb700 1 -- 172.21.15.98:0/4117959652 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6440276a0 con 0x7fb64c1609b0 2023-03-03T08:03:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.649+0000 7fb65219d700 1 -- 172.21.15.98:0/4117959652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64c15d0d0 con 0x7fb64c1609b0 2023-03-03T08:03:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.649+0000 7fb65219d700 1 -- 172.21.15.98:0/4117959652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64c15cdf0 con 0x7fb64c1609b0 2023-03-03T08:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.650+0000 7fb65219d700 1 -- 172.21.15.98:0/4117959652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb64c0b4fe0 con 0x7fb64c1609b0 2023-03-03T08:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.651+0000 7fb641ffb700 1 -- 172.21.15.98:0/4117959652 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fb64402e070 con 0x7fb64c1609b0 2023-03-03T08:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.653+0000 7fb641ffb700 1 --2- 172.21.15.98:0/4117959652 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb630068380 0x7fb63006a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.653+0000 7fb641ffb700 1 -- 172.21.15.98:0/4117959652 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb644022070 con 0x7fb64c1609b0 2023-03-03T08:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.654+0000 7fb65119b700 1 --2- 172.21.15.98:0/4117959652 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb630068380 0x7fb63006a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.655+0000 7fb65119b700 1 --2- 172.21.15.98:0/4117959652 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb630068380 0x7fb63006a830 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb648000f60 tx=0x7fb648007c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:10.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.658+0000 7fb641ffb700 1 -- 172.21.15.98:0/4117959652 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb6440542a0 con 0x7fb64c1609b0 2023-03-03T08:03:10.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.849+0000 7fb65219d700 1 -- 172.21.15.98:0/4117959652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb64c068290 con 0x7fb64c1609b0 2023-03-03T08:03:10.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.849+0000 7fb641ffb700 1 -- 172.21.15.98:0/4117959652 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v107) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb64402c020 con 0x7fb64c1609b0 2023-03-03T08:03:10.851 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:03:10.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.852+0000 7fb65219d700 1 -- 172.21.15.98:0/4117959652 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb630068380 msgr2=0x7fb63006a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:10.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.852+0000 7fb65219d700 1 --2- 172.21.15.98:0/4117959652 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb630068380 0x7fb63006a830 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb648000f60 tx=0x7fb648007c90 comp rx=0 tx=0).stop 2023-03-03T08:03:10.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.852+0000 7fb65219d700 1 -- 172.21.15.98:0/4117959652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c1609b0 msgr2=0x7fb64c15c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:10.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.852+0000 7fb65219d700 1 --2- 172.21.15.98:0/4117959652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c1609b0 0x7fb64c15c380 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fb64c074380 tx=0x7fb644004ad0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.852+0000 7fb65219d700 1 -- 172.21.15.98:0/4117959652 shutdown_connections 2023-03-03T08:03:10.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.852+0000 7fb65219d700 1 --2- 172.21.15.98:0/4117959652 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fb630068380 0x7fb63006a830 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.852+0000 7fb65219d700 1 --2- 172.21.15.98:0/4117959652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c1609b0 0x7fb64c15c380 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.852+0000 7fb65219d700 1 --2- 172.21.15.98:0/4117959652 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb64c0cf570 0x7fb64c1f15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.852+0000 7fb65219d700 1 --2- 172.21.15.98:0/4117959652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb64c15db30 0x7fb64c15be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:10.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.852+0000 7fb65219d700 1 -- 172.21.15.98:0/4117959652 >> 172.21.15.98:0/4117959652 conn(0x7fb64c06b880 msgr2=0x7fb64c15e670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:10.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.852+0000 7fb65219d700 1 -- 172.21.15.98:0/4117959652 shutdown_connections 2023-03-03T08:03:10.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:10.852+0000 7fb65219d700 1 -- 172.21.15.98:0/4117959652 wait complete. 2023-03-03T08:03:10.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-03-03T08:03:11.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.125+0000 7fd589dff700 1 Processor -- start 2023-03-03T08:03:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.125+0000 7fd589dff700 1 -- start start 2023-03-03T08:03:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.126+0000 7fd589dff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd58415cc00 0x7fd5840cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.127+0000 7fd589dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5840cde10 con 0x7fd5840d2c60 2023-03-03T08:03:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.127+0000 7fd589dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5840d1a00 con 0x7fd5840d27c0 2023-03-03T08:03:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.127+0000 7fd589dff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5840ce040 con 0x7fd58415cc00 2023-03-03T08:03:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.127+0000 7fd583fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd5840d2c60 0x7fd5840cf4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41996/0 (socket says 172.21.15.98:41996) 2023-03-03T08:03:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.127+0000 7fd583fff700 1 -- 172.21.15.98:0/1496229907 learned_addr learned my addr 172.21.15.98:0/1496229907 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.127+0000 7fd5837fe700 1 --2- 172.21.15.98:0/1496229907 >> v2:172.21.15.100:3300/0 conn(0x7fd58415cc00 0x7fd5840cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.127+0000 7fd5827fc700 1 -- 172.21.15.98:0/1496229907 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005210434 0 0) 0x7fd5840cde10 con 0x7fd5840d2c60 2023-03-03T08:03:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.127+0000 7fd5827fc700 1 -- 172.21.15.98:0/1496229907 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd56c0035b0 con 0x7fd5840d2c60 2023-03-03T08:03:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.128+0000 7fd5827fc700 1 -- 172.21.15.98:0/1496229907 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2044554409 0 0) 0x7fd56c0035b0 con 0x7fd5840d2c60 2023-03-03T08:03:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.128+0000 7fd5827fc700 1 -- 172.21.15.98:0/1496229907 >> v2:172.21.15.100:3300/0 conn(0x7fd58415cc00 msgr2=0x7fd5840cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.128+0000 7fd5827fc700 1 --2- 172.21.15.98:0/1496229907 >> v2:172.21.15.100:3300/0 conn(0x7fd58415cc00 0x7fd5840cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.128+0000 7fd5827fc700 1 -- 172.21.15.98:0/1496229907 >> v1:172.21.15.100:6789/0 conn(0x7fd5840d27c0 legacy=0x7fd5840b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.128+0000 7fd5827fc700 1 -- 172.21.15.98:0/1496229907 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5840d1ce0 con 0x7fd5840d2c60 2023-03-03T08:03:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.128+0000 7fd5827fc700 1 -- 172.21.15.98:0/1496229907 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7fd57c0019f0 con 0x7fd5840d2c60 2023-03-03T08:03:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.128+0000 7fd5837fe700 1 --2- 172.21.15.98:0/1496229907 >> v2:172.21.15.100:3300/0 conn(0x7fd58415cc00 0x7fd5840cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:03:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.128+0000 7fd5827fc700 1 -- 172.21.15.98:0/1496229907 >> v1:172.21.15.98:6789/0 conn(0x7fd5840d2c60 legacy=0x7fd5840cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.128+0000 7fd5827fc700 1 --2- 172.21.15.98:0/1496229907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd56c003a00 0x7fd56c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.128+0000 7fd5827fc700 1 --2- 172.21.15.98:0/1496229907 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd56c004850 0x7fd56c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.128+0000 7fd5827fc700 1 --2- 172.21.15.98:0/1496229907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd56c005200 0x7fd56c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.128+0000 7fd5827fc700 1 -- 172.21.15.98:0/1496229907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5840ce040 con 0x7fd56c003a00 2023-03-03T08:03:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.128+0000 7fd5827fc700 1 -- 172.21.15.98:0/1496229907 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c00c060 con 0x7fd56c005200 2023-03-03T08:03:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.128+0000 7fd5827fc700 1 -- 172.21.15.98:0/1496229907 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c00c220 con 0x7fd56c004850 2023-03-03T08:03:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.128+0000 7fd5837fe700 1 --2- 172.21.15.98:0/1496229907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd56c003a00 0x7fd56c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.129+0000 7fd5837fe700 1 -- 172.21.15.98:0/1496229907 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd56c004850 msgr2=0x7fd56c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.129+0000 7fd5837fe700 1 --2- 172.21.15.98:0/1496229907 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd56c004850 0x7fd56c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.129+0000 7fd5837fe700 1 -- 172.21.15.98:0/1496229907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd56c005200 msgr2=0x7fd56c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.129+0000 7fd583fff700 1 --2- 172.21.15.98:0/1496229907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd56c005200 0x7fd56c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.129+0000 7fd5837fe700 1 --2- 172.21.15.98:0/1496229907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd56c005200 0x7fd56c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.129+0000 7fd5837fe700 1 -- 172.21.15.98:0/1496229907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c0035b0 con 0x7fd56c003a00 2023-03-03T08:03:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.129+0000 7fd583fff700 1 --2- 172.21.15.98:0/1496229907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd56c005200 0x7fd56c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:03:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.129+0000 7fd583fff700 1 -- 172.21.15.98:0/1496229907 reap_dead start 2023-03-03T08:03:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.129+0000 7fd5837fe700 1 --2- 172.21.15.98:0/1496229907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd56c003a00 0x7fd56c003dd0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fd574009d30 tx=0x7fd56c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70664fcf586bfc05 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.129+0000 7fd5827fc700 1 -- 172.21.15.98:0/1496229907 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd574009e80 con 0x7fd56c003a00 2023-03-03T08:03:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.129+0000 7fd5827fc700 1 -- 172.21.15.98:0/1496229907 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57400e9a0 con 0x7fd56c003a00 2023-03-03T08:03:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.129+0000 7fd5827fc700 1 -- 172.21.15.98:0/1496229907 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57400ed10 con 0x7fd56c003a00 2023-03-03T08:03:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.130+0000 7fd589dff700 1 -- 172.21.15.98:0/1496229907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd56c003a00 msgr2=0x7fd56c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.130+0000 7fd589dff700 1 --2- 172.21.15.98:0/1496229907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd56c003a00 0x7fd56c003dd0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fd574009d30 tx=0x7fd56c005880 comp rx=0 tx=0).stop 2023-03-03T08:03:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.130+0000 7fd589dff700 1 -- 172.21.15.98:0/1496229907 shutdown_connections 2023-03-03T08:03:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.130+0000 7fd589dff700 1 --2- 172.21.15.98:0/1496229907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd56c003a00 0x7fd56c003dd0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.130+0000 7fd589dff700 1 -- 172.21.15.98:0/1496229907 >> 172.21.15.98:0/1496229907 conn(0x7fd58406aee0 msgr2=0x7fd58406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.130+0000 7fd589dff700 1 -- 172.21.15.98:0/1496229907 shutdown_connections 2023-03-03T08:03:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.130+0000 7fd589dff700 1 -- 172.21.15.98:0/1496229907 wait complete. 2023-03-03T08:03:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.132+0000 7fd589dff700 1 Processor -- start 2023-03-03T08:03:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.133+0000 7fd589dff700 1 -- start start 2023-03-03T08:03:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd589dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd56c003a00 0x7fd5840b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd589dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5840b3520 0x7fd58414f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd589dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd58415cfd0 0x7fd58415d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd589dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58415d990 con 0x7fd56c003a00 2023-03-03T08:03:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd589dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5840b4360 con 0x7fd5840b3520 2023-03-03T08:03:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd589dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5840b4640 con 0x7fd58415cfd0 2023-03-03T08:03:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd5837fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd56c003a00 0x7fd5840b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd5837fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd56c003a00 0x7fd5840b3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51180/0 (socket says 172.21.15.98:51180) 2023-03-03T08:03:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd5837fe700 1 -- 172.21.15.98:0/3055756852 learned_addr learned my addr 172.21.15.98:0/3055756852 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd582ffd700 1 --2- 172.21.15.98:0/3055756852 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5840b3520 0x7fd58414f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd583fff700 1 --2- 172.21.15.98:0/3055756852 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd58415cfd0 0x7fd58415d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd5837fe700 1 -- 172.21.15.98:0/3055756852 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd58415cfd0 msgr2=0x7fd58415d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd5837fe700 1 --2- 172.21.15.98:0/3055756852 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd58415cfd0 0x7fd58415d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd5837fe700 1 -- 172.21.15.98:0/3055756852 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5840b3520 msgr2=0x7fd58414f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd5837fe700 1 --2- 172.21.15.98:0/3055756852 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5840b3520 0x7fd58414f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.134+0000 7fd5837fe700 1 -- 172.21.15.98:0/3055756852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5840d22a0 con 0x7fd56c003a00 2023-03-03T08:03:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.135+0000 7fd5837fe700 1 --2- 172.21.15.98:0/3055756852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd56c003a00 0x7fd5840b3d80 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fd5740090b0 tx=0x7fd5740047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.135+0000 7fd56bfff700 1 -- 172.21.15.98:0/3055756852 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd574009e80 con 0x7fd56c003a00 2023-03-03T08:03:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.135+0000 7fd56bfff700 1 -- 172.21.15.98:0/3055756852 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57401c450 con 0x7fd56c003a00 2023-03-03T08:03:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.135+0000 7fd56bfff700 1 -- 172.21.15.98:0/3055756852 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57401c800 con 0x7fd56c003a00 2023-03-03T08:03:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.135+0000 7fd589dff700 1 -- 172.21.15.98:0/3055756852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5840d1fc0 con 0x7fd56c003a00 2023-03-03T08:03:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.135+0000 7fd589dff700 1 -- 172.21.15.98:0/3055756852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5840cd8d0 con 0x7fd56c003a00 2023-03-03T08:03:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.137+0000 7fd589dff700 1 -- 172.21.15.98:0/3055756852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5840729b0 con 0x7fd56c003a00 2023-03-03T08:03:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.138+0000 7fd56bfff700 1 -- 172.21.15.98:0/3055756852 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fd57402d030 con 0x7fd56c003a00 2023-03-03T08:03:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.139+0000 7fd56bfff700 1 --2- 172.21.15.98:0/3055756852 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd570068380 0x7fd57006a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.139+0000 7fd56bfff700 1 -- 172.21.15.98:0/3055756852 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd574087130 con 0x7fd56c003a00 2023-03-03T08:03:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.140+0000 7fd582ffd700 1 --2- 172.21.15.98:0/3055756852 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd570068380 0x7fd57006a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.141+0000 7fd582ffd700 1 --2- 172.21.15.98:0/3055756852 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd570068380 0x7fd57006a830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd5780018b0 tx=0x7fd578009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.143+0000 7fd56bfff700 1 -- 172.21.15.98:0/3055756852 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd574053c00 con 0x7fd56c003a00 2023-03-03T08:03:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.338+0000 7fd589dff700 1 -- 172.21.15.98:0/3055756852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd5841512a0 con 0x7fd56c003a00 2023-03-03T08:03:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.338+0000 7fd56bfff700 1 -- 172.21.15.98:0/3055756852 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v107) v1 ==== 84+0+126 (secure 0 0 0) 0x7fd574057220 con 0x7fd56c003a00 2023-03-03T08:03:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:03:11.340 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T08:03:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2023-03-03T08:03:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2023-03-03T08:03:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 2, 2023-03-03T08:03:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": "rbd" 2023-03-03T08:03:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:11.341 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T08:03:11.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.341+0000 7fd589dff700 1 -- 172.21.15.98:0/3055756852 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd570068380 msgr2=0x7fd57006a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:11.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.341+0000 7fd589dff700 1 --2- 172.21.15.98:0/3055756852 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd570068380 0x7fd57006a830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd5780018b0 tx=0x7fd578009040 comp rx=0 tx=0).stop 2023-03-03T08:03:11.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.341+0000 7fd589dff700 1 -- 172.21.15.98:0/3055756852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd56c003a00 msgr2=0x7fd5840b3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.341+0000 7fd589dff700 1 --2- 172.21.15.98:0/3055756852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd56c003a00 0x7fd5840b3d80 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fd5740090b0 tx=0x7fd5740047c0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.341+0000 7fd589dff700 1 -- 172.21.15.98:0/3055756852 shutdown_connections 2023-03-03T08:03:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.341+0000 7fd589dff700 1 --2- 172.21.15.98:0/3055756852 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd570068380 0x7fd57006a830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.341+0000 7fd589dff700 1 --2- 172.21.15.98:0/3055756852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd56c003a00 0x7fd5840b3d80 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.341+0000 7fd589dff700 1 --2- 172.21.15.98:0/3055756852 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd58415cfd0 0x7fd58415d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.341+0000 7fd589dff700 1 --2- 172.21.15.98:0/3055756852 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5840b3520 0x7fd58414f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.341+0000 7fd589dff700 1 -- 172.21.15.98:0/3055756852 >> 172.21.15.98:0/3055756852 conn(0x7fd58406aee0 msgr2=0x7fd5840b6140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.341+0000 7fd589dff700 1 -- 172.21.15.98:0/3055756852 shutdown_connections 2023-03-03T08:03:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.341+0000 7fd589dff700 1 -- 172.21.15.98:0/3055756852 wait complete. 2023-03-03T08:03:11.369 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-03-03T08:03:11.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-03-03T08:03:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.618+0000 7f75ee19d700 1 Processor -- start 2023-03-03T08:03:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.619+0000 7f75ee19d700 1 -- start start 2023-03-03T08:03:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.620+0000 7f75ee19d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f75e80d0110 0x7f75e80cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.620+0000 7f75ee19d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f75e8073ed0 0x7f75e80ce380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.620+0000 7f75ee19d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e80b3c80 con 0x7f75e80b3550 2023-03-03T08:03:11.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.620+0000 7f75ee19d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e80b4370 con 0x7f75e8073ed0 2023-03-03T08:03:11.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.620+0000 7f75ee19d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e80b3ed0 con 0x7f75e80d0110 2023-03-03T08:03:11.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.620+0000 7f75ed99c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f75e8073ed0 0x7f75e80ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:11.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.620+0000 7f75ed99c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f75e8073ed0 0x7f75e80ce380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51190/0 (socket says 172.21.15.98:51190) 2023-03-03T08:03:11.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.621+0000 7f75ed99c700 1 -- 172.21.15.98:0/1675335084 learned_addr learned my addr 172.21.15.98:0/1675335084 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:11.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.621+0000 7f75ec99a700 1 --2- 172.21.15.98:0/1675335084 >> v2:172.21.15.100:3300/0 conn(0x7f75e80d0110 0x7f75e80cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:11.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.621+0000 7f75ed99c700 1 -- 172.21.15.98:0/1675335084 >> v2:172.21.15.100:3300/0 conn(0x7f75e80d0110 msgr2=0x7f75e80cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:11.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.621+0000 7f75ed99c700 1 --2- 172.21.15.98:0/1675335084 >> v2:172.21.15.100:3300/0 conn(0x7f75e80d0110 0x7f75e80cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.621+0000 7f75ed99c700 1 -- 172.21.15.98:0/1675335084 >> v1:172.21.15.100:6789/0 conn(0x7f75e80b3550 legacy=0x7f75e80cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.621+0000 7f75ed99c700 1 -- 172.21.15.98:0/1675335084 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e80cd640 con 0x7f75e8073ed0 2023-03-03T08:03:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.621+0000 7f75ed99c700 1 --2- 172.21.15.98:0/1675335084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75e8073ed0 0x7f75e80ce380 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f75e8161820 tx=0x7f75e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4facbae46bc44d6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.622+0000 7f75dffff700 1 -- 172.21.15.98:0/1675335084 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e000e070 con 0x7f75e8073ed0 2023-03-03T08:03:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.622+0000 7f75dffff700 1 -- 172.21.15.98:0/1675335084 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e000b910 con 0x7f75e8073ed0 2023-03-03T08:03:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.622+0000 7f75dffff700 1 -- 172.21.15.98:0/1675335084 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e000bcc0 con 0x7f75e8073ed0 2023-03-03T08:03:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.622+0000 7f75ee19d700 1 -- 172.21.15.98:0/1675335084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75e8073ed0 msgr2=0x7f75e80ce380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.622+0000 7f75ee19d700 1 --2- 172.21.15.98:0/1675335084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75e8073ed0 0x7f75e80ce380 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f75e8161820 tx=0x7f75e000b330 comp rx=0 tx=0).stop 2023-03-03T08:03:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.622+0000 7f75ee19d700 1 -- 172.21.15.98:0/1675335084 shutdown_connections 2023-03-03T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.622+0000 7f75ee19d700 1 --2- 172.21.15.98:0/1675335084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75e8073ed0 0x7f75e80ce380 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.622+0000 7f75ee19d700 1 --2- 172.21.15.98:0/1675335084 >> v2:172.21.15.100:3300/0 conn(0x7f75e80d0110 0x7f75e80cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.622+0000 7f75ee19d700 1 -- 172.21.15.98:0/1675335084 >> 172.21.15.98:0/1675335084 conn(0x7f75e8073b00 msgr2=0x7f75e80d2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.622+0000 7f75ee19d700 1 -- 172.21.15.98:0/1675335084 shutdown_connections 2023-03-03T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.622+0000 7f75ee19d700 1 -- 172.21.15.98:0/1675335084 wait complete. 2023-03-03T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.624+0000 7f75ee19d700 1 Processor -- start 2023-03-03T08:03:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.624+0000 7f75ee19d700 1 -- start start 2023-03-03T08:03:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.625+0000 7f75ee19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75e8073ed0 0x7f75e80b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.625+0000 7f75ee19d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75e80b3550 0x7f75e81f3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.625+0000 7f75ee19d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75e80d0110 0x7f75e80b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.625+0000 7f75ee19d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e80c9590 con 0x7f75e80b3550 2023-03-03T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.625+0000 7f75ee19d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e80afca0 con 0x7f75e80d0110 2023-03-03T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.625+0000 7f75ee19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e80afeb0 con 0x7f75e8073ed0 2023-03-03T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.626+0000 7f75ec99a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75e80b3550 0x7f75e81f3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.626+0000 7f75ed19b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75e8073ed0 0x7f75e80b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.626+0000 7f75ec99a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75e80b3550 0x7f75e81f3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51204/0 (socket says 172.21.15.98:51204) 2023-03-03T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.626+0000 7f75ec99a700 1 -- 172.21.15.98:0/804038998 learned_addr learned my addr 172.21.15.98:0/804038998 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.626+0000 7f75ed99c700 1 --2- 172.21.15.98:0/804038998 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75e80d0110 0x7f75e80b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.626+0000 7f75ec99a700 1 -- 172.21.15.98:0/804038998 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75e8073ed0 msgr2=0x7f75e80b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.626+0000 7f75ec99a700 1 --2- 172.21.15.98:0/804038998 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75e8073ed0 0x7f75e80b23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.626+0000 7f75ec99a700 1 -- 172.21.15.98:0/804038998 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75e80d0110 msgr2=0x7f75e80b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.626+0000 7f75ec99a700 1 --2- 172.21.15.98:0/804038998 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75e80d0110 0x7f75e80b0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.626+0000 7f75ec99a700 1 -- 172.21.15.98:0/804038998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e0009b30 con 0x7f75e80b3550 2023-03-03T08:03:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.627+0000 7f75ec99a700 1 --2- 172.21.15.98:0/804038998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75e80b3550 0x7f75e81f3680 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f75d800e870 tx=0x7f75d800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.627+0000 7f75ddffb700 1 -- 172.21.15.98:0/804038998 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d8009590 con 0x7f75e80b3550 2023-03-03T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.627+0000 7f75ddffb700 1 -- 172.21.15.98:0/804038998 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d8013680 con 0x7f75e80b3550 2023-03-03T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.627+0000 7f75ee19d700 1 -- 172.21.15.98:0/804038998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e80d3d60 con 0x7f75e80b3550 2023-03-03T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.627+0000 7f75ddffb700 1 -- 172.21.15.98:0/804038998 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d80139f0 con 0x7f75e80b3550 2023-03-03T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.627+0000 7f75ee19d700 1 -- 172.21.15.98:0/804038998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e80cd920 con 0x7f75e80b3550 2023-03-03T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.628+0000 7f75ee19d700 1 -- 172.21.15.98:0/804038998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e80d3ef0 con 0x7f75e80b3550 2023-03-03T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.630+0000 7f75ddffb700 1 -- 172.21.15.98:0/804038998 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f75d8019030 con 0x7f75e80b3550 2023-03-03T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.631+0000 7f75ddffb700 1 --2- 172.21.15.98:0/804038998 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f75cc068380 0x7f75cc06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.631+0000 7f75ddffb700 1 -- 172.21.15.98:0/804038998 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f75d8085730 con 0x7f75e80b3550 2023-03-03T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.631+0000 7f75ed19b700 1 --2- 172.21.15.98:0/804038998 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f75cc068380 0x7f75cc06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.632+0000 7f75ed19b700 1 --2- 172.21.15.98:0/804038998 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f75cc068380 0x7f75cc06a830 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f75e80b8ee0 tx=0x7f75e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.635+0000 7f75ddffb700 1 -- 172.21.15.98:0/804038998 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f75e80d3ef0 con 0x7f75e80b3550 2023-03-03T08:03:11.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.828+0000 7f75ee19d700 1 -- 172.21.15.98:0/804038998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f75e80d3ef0 con 0x7f75e80b3550 2023-03-03T08:03:11.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.946+0000 7f75ddffb700 1 -- 172.21.15.98:0/804038998 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v108) v1 ==== 138+0+0 (secure 0 0 0) 0x7f75e80d3ef0 con 0x7f75e80b3550 2023-03-03T08:03:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.950+0000 7f75ee19d700 1 -- 172.21.15.98:0/804038998 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f75cc068380 msgr2=0x7f75cc06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.950+0000 7f75ee19d700 1 --2- 172.21.15.98:0/804038998 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f75cc068380 0x7f75cc06a830 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f75e80b8ee0 tx=0x7f75e4009040 comp rx=0 tx=0).stop 2023-03-03T08:03:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.950+0000 7f75ee19d700 1 -- 172.21.15.98:0/804038998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75e80b3550 msgr2=0x7f75e81f3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.950+0000 7f75ee19d700 1 --2- 172.21.15.98:0/804038998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75e80b3550 0x7f75e81f3680 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f75d800e870 tx=0x7f75d800eb80 comp rx=0 tx=0).stop 2023-03-03T08:03:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.950+0000 7f75ee19d700 1 -- 172.21.15.98:0/804038998 shutdown_connections 2023-03-03T08:03:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.950+0000 7f75ee19d700 1 --2- 172.21.15.98:0/804038998 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75e80d0110 0x7f75e80b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.950+0000 7f75ee19d700 1 --2- 172.21.15.98:0/804038998 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75e8073ed0 0x7f75e80b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.950+0000 7f75ee19d700 1 --2- 172.21.15.98:0/804038998 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f75cc068380 0x7f75cc06a830 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.950+0000 7f75ee19d700 1 --2- 172.21.15.98:0/804038998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75e80b3550 0x7f75e81f3680 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.950+0000 7f75ee19d700 1 -- 172.21.15.98:0/804038998 >> 172.21.15.98:0/804038998 conn(0x7f75e8073b00 msgr2=0x7f75e806b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.950+0000 7f75ee19d700 1 -- 172.21.15.98:0/804038998 shutdown_connections 2023-03-03T08:03:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:11.951+0000 7f75ee19d700 1 -- 172.21.15.98:0/804038998 wait complete. 2023-03-03T08:03:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2023-03-03T08:03:11.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-03-03T08:03:12.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.233+0000 7f5adc3ff700 1 Processor -- start 2023-03-03T08:03:12.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.233+0000 7f5adc3ff700 1 -- start start 2023-03-03T08:03:12.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.234+0000 7f5adc3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ad415ca70 0x7f5ad40cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:12.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.234+0000 7f5adc3ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad40cf980 con 0x7f5ad406b4b0 2023-03-03T08:03:12.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.234+0000 7f5adc3ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad40d1800 con 0x7f5ad40d0270 2023-03-03T08:03:12.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.234+0000 7f5adc3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad40cd5b0 con 0x7f5ad415ca70 2023-03-03T08:03:12.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.235+0000 7f5ad999a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5ad406b4b0 0x7f5ad40cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42004/0 (socket says 172.21.15.98:42004) 2023-03-03T08:03:12.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.235+0000 7f5ad999a700 1 -- 172.21.15.98:0/1285225004 learned_addr learned my addr 172.21.15.98:0/1285225004 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:12.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.235+0000 7f5ada19b700 1 --2- 172.21.15.98:0/1285225004 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ad415ca70 0x7f5ad40cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:12.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.235+0000 7f5ad9199700 1 -- 172.21.15.98:0/1285225004 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118741434 0 0) 0x7f5ad40cf980 con 0x7f5ad406b4b0 2023-03-03T08:03:12.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.235+0000 7f5ad9199700 1 -- 172.21.15.98:0/1285225004 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac00035b0 con 0x7f5ad406b4b0 2023-03-03T08:03:12.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.235+0000 7f5ad9199700 1 -- 172.21.15.98:0/1285225004 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628256795 0 0) 0x7f5ad40d1800 con 0x7f5ad40d0270 2023-03-03T08:03:12.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.235+0000 7f5ad9199700 1 -- 172.21.15.98:0/1285225004 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad40cf980 con 0x7f5ad40d0270 2023-03-03T08:03:12.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.236+0000 7f5ad9199700 1 -- 172.21.15.98:0/1285225004 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 889248693 0 0) 0x7f5ac00035b0 con 0x7f5ad406b4b0 2023-03-03T08:03:12.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.236+0000 7f5ada19b700 1 -- 172.21.15.98:0/1285225004 >> v1:172.21.15.100:6789/0 conn(0x7f5ad40d0270 legacy=0x7f5ad40d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:12.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.236+0000 7f5ada19b700 1 -- 172.21.15.98:0/1285225004 >> v1:172.21.15.98:6789/0 conn(0x7f5ad406b4b0 legacy=0x7f5ad40cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:12.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.236+0000 7f5ada19b700 1 -- 172.21.15.98:0/1285225004 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad40cd900 con 0x7f5ad415ca70 2023-03-03T08:03:12.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.236+0000 7f5ada19b700 1 --2- 172.21.15.98:0/1285225004 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ad415ca70 0x7f5ad40cf440 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0009870 tx=0x7f5ad0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87e041e1888bc69a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:12.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.236+0000 7f5ad9199700 1 -- 172.21.15.98:0/1285225004 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4001670 con 0x7f5ad415ca70 2023-03-03T08:03:12.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.236+0000 7f5ad9199700 1 -- 172.21.15.98:0/1285225004 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad0004030 con 0x7f5ad415ca70 2023-03-03T08:03:12.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.236+0000 7f5ad9199700 1 -- 172.21.15.98:0/1285225004 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad00043c0 con 0x7f5ad415ca70 2023-03-03T08:03:12.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.237+0000 7f5adc3ff700 1 -- 172.21.15.98:0/1285225004 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ad415ca70 msgr2=0x7f5ad40cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:12.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.237+0000 7f5adc3ff700 1 --2- 172.21.15.98:0/1285225004 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ad415ca70 0x7f5ad40cf440 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0009870 tx=0x7f5ad0009b80 comp rx=0 tx=0).stop 2023-03-03T08:03:12.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.237+0000 7f5adc3ff700 1 -- 172.21.15.98:0/1285225004 shutdown_connections 2023-03-03T08:03:12.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.237+0000 7f5adc3ff700 1 --2- 172.21.15.98:0/1285225004 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ad415ca70 0x7f5ad40cf440 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:12.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.237+0000 7f5adc3ff700 1 -- 172.21.15.98:0/1285225004 >> 172.21.15.98:0/1285225004 conn(0x7f5ad4073e70 msgr2=0x7f5ad406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:12.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.237+0000 7f5adc3ff700 1 -- 172.21.15.98:0/1285225004 shutdown_connections 2023-03-03T08:03:12.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.237+0000 7f5adc3ff700 1 -- 172.21.15.98:0/1285225004 wait complete. 2023-03-03T08:03:12.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.239+0000 7f5adc3ff700 1 Processor -- start 2023-03-03T08:03:12.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.239+0000 7f5adc3ff700 1 -- start start 2023-03-03T08:03:12.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.240+0000 7f5adc3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ad406b4b0 0x7f5ad415a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:12.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.240+0000 7f5adc3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ad40d0270 0x7f5ad41550d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:12.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.240+0000 7f5adc3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ad415ca70 0x7f5ad40b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:12.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.240+0000 7f5adc3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad415e9f0 con 0x7f5ad415ca70 2023-03-03T08:03:12.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.240+0000 7f5adc3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad415e710 con 0x7f5ad40d0270 2023-03-03T08:03:12.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.240+0000 7f5adc3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad4159db0 con 0x7f5ad406b4b0 2023-03-03T08:03:12.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.241+0000 7f5ada19b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ad406b4b0 0x7f5ad415a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:12.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.241+0000 7f5ada99c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ad415ca70 0x7f5ad40b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:12.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.241+0000 7f5ad999a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ad40d0270 0x7f5ad41550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:12.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.241+0000 7f5ada19b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ad406b4b0 0x7f5ad415a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:34240/0 (socket says 172.21.15.98:34240) 2023-03-03T08:03:12.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.241+0000 7f5ada19b700 1 -- 172.21.15.98:0/3798017380 learned_addr learned my addr 172.21.15.98:0/3798017380 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:12.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.241+0000 7f5ada99c700 1 -- 172.21.15.98:0/3798017380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ad406b4b0 msgr2=0x7f5ad415a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:12.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.241+0000 7f5ada99c700 1 --2- 172.21.15.98:0/3798017380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ad406b4b0 0x7f5ad415a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:12.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.241+0000 7f5ada99c700 1 -- 172.21.15.98:0/3798017380 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ad40d0270 msgr2=0x7f5ad41550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:12.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.241+0000 7f5ada99c700 1 --2- 172.21.15.98:0/3798017380 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ad40d0270 0x7f5ad41550d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:12.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.241+0000 7f5ada99c700 1 -- 172.21.15.98:0/3798017380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad00094a0 con 0x7f5ad415ca70 2023-03-03T08:03:12.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.242+0000 7f5ada99c700 1 --2- 172.21.15.98:0/3798017380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ad415ca70 0x7f5ad40b2be0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f5acc00cbc0 tx=0x7f5acc00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:12.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.242+0000 7f5acaffd700 1 -- 172.21.15.98:0/3798017380 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5acc003c00 con 0x7f5ad415ca70 2023-03-03T08:03:12.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.242+0000 7f5acaffd700 1 -- 172.21.15.98:0/3798017380 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5acc003d60 con 0x7f5ad415ca70 2023-03-03T08:03:12.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.242+0000 7f5adc3ff700 1 -- 172.21.15.98:0/3798017380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad40d1aa0 con 0x7f5ad415ca70 2023-03-03T08:03:12.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.242+0000 7f5acaffd700 1 -- 172.21.15.98:0/3798017380 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5acc00aa80 con 0x7f5ad415ca70 2023-03-03T08:03:12.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.242+0000 7f5adc3ff700 1 -- 172.21.15.98:0/3798017380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad40cdbe0 con 0x7f5ad415ca70 2023-03-03T08:03:12.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.243+0000 7f5adc3ff700 1 -- 172.21.15.98:0/3798017380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad415ce40 con 0x7f5ad415ca70 2023-03-03T08:03:12.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.245+0000 7f5acaffd700 1 -- 172.21.15.98:0/3798017380 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f5acc010050 con 0x7f5ad415ca70 2023-03-03T08:03:12.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.247+0000 7f5acaffd700 1 --2- 172.21.15.98:0/3798017380 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5ab8068380 0x7f5ab806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:12.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.247+0000 7f5acaffd700 1 -- 172.21.15.98:0/3798017380 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5580+0+0 (secure 0 0 0) 0x7f5acc085a90 con 0x7f5ad415ca70 2023-03-03T08:03:12.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.248+0000 7f5ada19b700 1 --2- 172.21.15.98:0/3798017380 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5ab8068380 0x7f5ab806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:12.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.249+0000 7f5ada19b700 1 --2- 172.21.15.98:0/3798017380 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5ab8068380 0x7f5ab806a830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0000c00 tx=0x7f5ad0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:12.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.252+0000 7f5acaffd700 1 -- 172.21.15.98:0/3798017380 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5acc052370 con 0x7f5ad415ca70 2023-03-03T08:03:12.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.444+0000 7f5adc3ff700 1 -- 172.21.15.98:0/3798017380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5ad404e6e0 con 0x7f5ad415ca70 2023-03-03T08:03:12.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.953+0000 7f5acaffd700 1 -- 172.21.15.98:0/3798017380 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v109) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5acc084150 con 0x7f5ad415ca70 2023-03-03T08:03:12.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.963+0000 7f5adc3ff700 1 -- 172.21.15.98:0/3798017380 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5ab8068380 msgr2=0x7f5ab806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:12.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.963+0000 7f5adc3ff700 1 --2- 172.21.15.98:0/3798017380 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5ab8068380 0x7f5ab806a830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0000c00 tx=0x7f5ad0008030 comp rx=0 tx=0).stop 2023-03-03T08:03:12.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.963+0000 7f5adc3ff700 1 -- 172.21.15.98:0/3798017380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ad415ca70 msgr2=0x7f5ad40b2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:12.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.963+0000 7f5adc3ff700 1 --2- 172.21.15.98:0/3798017380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ad415ca70 0x7f5ad40b2be0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f5acc00cbc0 tx=0x7f5acc00ced0 comp rx=0 tx=0).stop 2023-03-03T08:03:12.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.964+0000 7f5adc3ff700 1 -- 172.21.15.98:0/3798017380 shutdown_connections 2023-03-03T08:03:12.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.964+0000 7f5adc3ff700 1 --2- 172.21.15.98:0/3798017380 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5ab8068380 0x7f5ab806a830 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:12.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.964+0000 7f5adc3ff700 1 --2- 172.21.15.98:0/3798017380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ad415ca70 0x7f5ad40b2be0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:12.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.964+0000 7f5adc3ff700 1 --2- 172.21.15.98:0/3798017380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ad406b4b0 0x7f5ad415a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:12.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.964+0000 7f5adc3ff700 1 --2- 172.21.15.98:0/3798017380 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ad40d0270 0x7f5ad41550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:12.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.964+0000 7f5adc3ff700 1 -- 172.21.15.98:0/3798017380 >> 172.21.15.98:0/3798017380 conn(0x7f5ad4073e70 msgr2=0x7f5ad406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:12.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.970+0000 7f5adc3ff700 1 -- 172.21.15.98:0/3798017380 shutdown_connections 2023-03-03T08:03:12.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:12.971+0000 7f5adc3ff700 1 -- 172.21.15.98:0/3798017380 wait complete. 2023-03-03T08:03:12.976 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2023-03-03T08:03:13.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-03-03T08:03:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.249+0000 7f441f9ff700 1 Processor -- start 2023-03-03T08:03:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.249+0000 7f441f9ff700 1 -- start start 2023-03-03T08:03:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.250+0000 7f441f9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44180b8d10 0x7f44180cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.250+0000 7f441f9ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f44180b26d0 0x7f44180b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.250+0000 7f441f9ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44180cd7d0 con 0x7f44180b5140 2023-03-03T08:03:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.250+0000 7f441f9ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44180b55c0 con 0x7f44180b26d0 2023-03-03T08:03:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.250+0000 7f441f9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44180b90e0 con 0x7f44180b8d10 2023-03-03T08:03:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.250+0000 7f441cf9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f44180b26d0 0x7f44180b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.251+0000 7f441cf9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f44180b26d0 0x7f44180b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51232/0 (socket says 172.21.15.98:51232) 2023-03-03T08:03:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.251+0000 7f441cf9a700 1 -- 172.21.15.98:0/3525673707 learned_addr learned my addr 172.21.15.98:0/3525673707 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.251+0000 7f441d79b700 1 --2- 172.21.15.98:0/3525673707 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44180b8d10 0x7f44180cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.251+0000 7f441cf9a700 1 -- 172.21.15.98:0/3525673707 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44180b8d10 msgr2=0x7f44180cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.251+0000 7f441cf9a700 1 --2- 172.21.15.98:0/3525673707 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44180b8d10 0x7f44180cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.251+0000 7f441cf9a700 1 -- 172.21.15.98:0/3525673707 >> v1:172.21.15.98:6789/0 conn(0x7f44180b5140 legacy=0x7f4418160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.251+0000 7f441cf9a700 1 -- 172.21.15.98:0/3525673707 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44180cda20 con 0x7f44180b26d0 2023-03-03T08:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.251+0000 7f441d79b700 1 --2- 172.21.15.98:0/3525673707 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44180b8d10 0x7f44180cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.251+0000 7f441cf9a700 1 --2- 172.21.15.98:0/3525673707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180b26d0 0x7f44180b4c00 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f44080098f0 tx=0x7f4408009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcffb7e37d0867e9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.252+0000 7f440ffff700 1 -- 172.21.15.98:0/3525673707 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440800b8a0 con 0x7f44180b26d0 2023-03-03T08:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.252+0000 7f440ffff700 1 -- 172.21.15.98:0/3525673707 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440800ba00 con 0x7f44180b26d0 2023-03-03T08:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.252+0000 7f440ffff700 1 -- 172.21.15.98:0/3525673707 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440800bd90 con 0x7f44180b26d0 2023-03-03T08:03:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.252+0000 7f441f9ff700 1 -- 172.21.15.98:0/3525673707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180b26d0 msgr2=0x7f44180b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.252+0000 7f441f9ff700 1 --2- 172.21.15.98:0/3525673707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180b26d0 0x7f44180b4c00 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f44080098f0 tx=0x7f4408009c00 comp rx=0 tx=0).stop 2023-03-03T08:03:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.252+0000 7f441f9ff700 1 -- 172.21.15.98:0/3525673707 shutdown_connections 2023-03-03T08:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.252+0000 7f441f9ff700 1 --2- 172.21.15.98:0/3525673707 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44180b8d10 0x7f44180cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.252+0000 7f441f9ff700 1 --2- 172.21.15.98:0/3525673707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180b26d0 0x7f44180b4c00 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.252+0000 7f441f9ff700 1 -- 172.21.15.98:0/3525673707 >> 172.21.15.98:0/3525673707 conn(0x7f4418073e70 msgr2=0x7f441806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.253+0000 7f441f9ff700 1 -- 172.21.15.98:0/3525673707 shutdown_connections 2023-03-03T08:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.253+0000 7f441f9ff700 1 -- 172.21.15.98:0/3525673707 wait complete. 2023-03-03T08:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.255+0000 7f441f9ff700 1 Processor -- start 2023-03-03T08:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.255+0000 7f441f9ff700 1 -- start start 2023-03-03T08:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.256+0000 7f441f9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44180b26d0 0x7f44181e7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.256+0000 7f441f9ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180b5140 0x7f441815e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.256+0000 7f441f9ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44180b8d10 0x7f441815a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.256+0000 7f441f9ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44181e7b90 con 0x7f44180b5140 2023-03-03T08:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.256+0000 7f441f9ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441815e600 con 0x7f44180b8d10 2023-03-03T08:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.256+0000 7f441f9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44181e89d0 con 0x7f44180b26d0 2023-03-03T08:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.256+0000 7f441cf9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180b5140 0x7f441815e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.256+0000 7f441cf9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180b5140 0x7f441815e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51240/0 (socket says 172.21.15.98:51240) 2023-03-03T08:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.256+0000 7f441cf9a700 1 -- 172.21.15.98:0/2764776964 learned_addr learned my addr 172.21.15.98:0/2764776964 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.256+0000 7f441d79b700 1 --2- 172.21.15.98:0/2764776964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44180b26d0 0x7f44181e7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.256+0000 7f441df9c700 1 --2- 172.21.15.98:0/2764776964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44180b8d10 0x7f441815a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.257+0000 7f441cf9a700 1 -- 172.21.15.98:0/2764776964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44180b26d0 msgr2=0x7f44181e7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.257+0000 7f441cf9a700 1 --2- 172.21.15.98:0/2764776964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44180b26d0 0x7f44181e7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.257+0000 7f441cf9a700 1 -- 172.21.15.98:0/2764776964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44180b8d10 msgr2=0x7f441815a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.257+0000 7f441cf9a700 1 --2- 172.21.15.98:0/2764776964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44180b8d10 0x7f441815a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.257+0000 7f441cf9a700 1 -- 172.21.15.98:0/2764776964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4408009550 con 0x7f44180b5140 2023-03-03T08:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.257+0000 7f441df9c700 1 --2- 172.21.15.98:0/2764776964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44180b8d10 0x7f441815a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.257+0000 7f441cf9a700 1 --2- 172.21.15.98:0/2764776964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180b5140 0x7f441815e0c0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f4408009c40 tx=0x7f4408004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.257+0000 7f440dffb700 1 -- 172.21.15.98:0/2764776964 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4408004590 con 0x7f44180b5140 2023-03-03T08:03:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.257+0000 7f440dffb700 1 -- 172.21.15.98:0/2764776964 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440801e070 con 0x7f44180b5140 2023-03-03T08:03:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.257+0000 7f441f9ff700 1 -- 172.21.15.98:0/2764776964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44180d0e80 con 0x7f44180b5140 2023-03-03T08:03:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.257+0000 7f440dffb700 1 -- 172.21.15.98:0/2764776964 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44080038e0 con 0x7f44180b5140 2023-03-03T08:03:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.258+0000 7f441f9ff700 1 -- 172.21.15.98:0/2764776964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44180cdd00 con 0x7f44180b5140 2023-03-03T08:03:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.259+0000 7f441f9ff700 1 -- 172.21.15.98:0/2764776964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44180cdfe0 con 0x7f44180b5140 2023-03-03T08:03:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.260+0000 7f440dffb700 1 -- 172.21.15.98:0/2764776964 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f440800f050 con 0x7f44180b5140 2023-03-03T08:03:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.262+0000 7f440dffb700 1 --2- 172.21.15.98:0/2764776964 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f43fc068380 0x7f43fc06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.262+0000 7f440dffb700 1 -- 172.21.15.98:0/2764776964 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f4408020070 con 0x7f44180b5140 2023-03-03T08:03:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.263+0000 7f441d79b700 1 --2- 172.21.15.98:0/2764776964 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f43fc068380 0x7f43fc06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.263+0000 7f441d79b700 1 --2- 172.21.15.98:0/2764776964 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f43fc068380 0x7f43fc06a830 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f441815adf0 tx=0x7f4414008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.267+0000 7f440dffb700 1 -- 172.21.15.98:0/2764776964 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4408053c40 con 0x7f44180b5140 2023-03-03T08:03:13.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.505+0000 7f441f9ff700 1 -- 172.21.15.98:0/2764776964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f441806b810 con 0x7f44180b5140 2023-03-03T08:03:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.511+0000 7f440dffb700 1 -- 172.21.15.98:0/2764776964 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v58) v1 ==== 174+0+0 (secure 0 0 0) 0x7f4408006d80 con 0x7f44180b5140 2023-03-03T08:03:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.522+0000 7f441f9ff700 1 -- 172.21.15.98:0/2764776964 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f43fc068380 msgr2=0x7f43fc06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.522+0000 7f441f9ff700 1 --2- 172.21.15.98:0/2764776964 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f43fc068380 0x7f43fc06a830 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f441815adf0 tx=0x7f4414008040 comp rx=0 tx=0).stop 2023-03-03T08:03:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.522+0000 7f441f9ff700 1 -- 172.21.15.98:0/2764776964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180b5140 msgr2=0x7f441815e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.522+0000 7f441f9ff700 1 --2- 172.21.15.98:0/2764776964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180b5140 0x7f441815e0c0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f4408009c40 tx=0x7f4408004750 comp rx=0 tx=0).stop 2023-03-03T08:03:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.522+0000 7f441f9ff700 1 -- 172.21.15.98:0/2764776964 shutdown_connections 2023-03-03T08:03:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.522+0000 7f441f9ff700 1 --2- 172.21.15.98:0/2764776964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44180b8d10 0x7f441815a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.522+0000 7f441f9ff700 1 --2- 172.21.15.98:0/2764776964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44180b26d0 0x7f44181e7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.522+0000 7f441f9ff700 1 --2- 172.21.15.98:0/2764776964 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f43fc068380 0x7f43fc06a830 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f441815adf0 tx=0x7f4414008040 comp rx=0 tx=0).stop 2023-03-03T08:03:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.522+0000 7f441f9ff700 1 --2- 172.21.15.98:0/2764776964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180b5140 0x7f441815e0c0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.522+0000 7f441f9ff700 1 -- 172.21.15.98:0/2764776964 >> 172.21.15.98:0/2764776964 conn(0x7f4418073e70 msgr2=0x7f441806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.523+0000 7f441f9ff700 1 -- 172.21.15.98:0/2764776964 shutdown_connections 2023-03-03T08:03:13.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.523+0000 7f441f9ff700 1 -- 172.21.15.98:0/2764776964 wait complete. 2023-03-03T08:03:13.530 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 17 and data pool 18 2023-03-03T08:03:13.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:03:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.807+0000 7fc5c9fff700 1 Processor -- start 2023-03-03T08:03:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.807+0000 7fc5c9fff700 1 -- start start 2023-03-03T08:03:13.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.808+0000 7fc5c9fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc5c40b8b90 0x7fc5c40cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.808+0000 7fc5c9fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5c40b5410 0x7fc5c40d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.808+0000 7fc5c9fff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c40d0ef0 con 0x7fc5c40b87c0 2023-03-03T08:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.808+0000 7fc5c9fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c40d0600 con 0x7fc5c40b5410 2023-03-03T08:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.808+0000 7fc5c9fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c40cd970 con 0x7fc5c40b8b90 2023-03-03T08:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.809+0000 7fc5c3fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5c40b5410 0x7fc5c40d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.809+0000 7fc5c3fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5c40b5410 0x7fc5c40d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51244/0 (socket says 172.21.15.98:51244) 2023-03-03T08:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.809+0000 7fc5c3fff700 1 -- 172.21.15.98:0/459340750 learned_addr learned my addr 172.21.15.98:0/459340750 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.809+0000 7fc5bbfff700 1 --2- 172.21.15.98:0/459340750 >> v2:172.21.15.100:3300/0 conn(0x7fc5c40b8b90 0x7fc5c40cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.809+0000 7fc5c3fff700 1 -- 172.21.15.98:0/459340750 >> v2:172.21.15.100:3300/0 conn(0x7fc5c40b8b90 msgr2=0x7fc5c40cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.809+0000 7fc5c3fff700 1 --2- 172.21.15.98:0/459340750 >> v2:172.21.15.100:3300/0 conn(0x7fc5c40b8b90 0x7fc5c40cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.809+0000 7fc5c3fff700 1 -- 172.21.15.98:0/459340750 >> v1:172.21.15.100:6789/0 conn(0x7fc5c40b87c0 legacy=0x7fc5c40b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.809+0000 7fc5c3fff700 1 -- 172.21.15.98:0/459340750 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c40cdc50 con 0x7fc5c40b5410 2023-03-03T08:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.809+0000 7fc5c3fff700 1 --2- 172.21.15.98:0/459340750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5c40b5410 0x7fc5c40d13d0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fc5c408c240 tx=0x7fc5bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3411078543c635cc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.810+0000 7fc5c2ffd700 1 -- 172.21.15.98:0/459340750 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc00e070 con 0x7fc5c40b5410 2023-03-03T08:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.810+0000 7fc5c2ffd700 1 -- 172.21.15.98:0/459340750 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5bc00b910 con 0x7fc5c40b5410 2023-03-03T08:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.810+0000 7fc5c2ffd700 1 -- 172.21.15.98:0/459340750 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc00bcc0 con 0x7fc5c40b5410 2023-03-03T08:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.810+0000 7fc5c9fff700 1 -- 172.21.15.98:0/459340750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5c40b5410 msgr2=0x7fc5c40d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.810+0000 7fc5c9fff700 1 --2- 172.21.15.98:0/459340750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5c40b5410 0x7fc5c40d13d0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fc5c408c240 tx=0x7fc5bc00b330 comp rx=0 tx=0).stop 2023-03-03T08:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.810+0000 7fc5c9fff700 1 -- 172.21.15.98:0/459340750 shutdown_connections 2023-03-03T08:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.810+0000 7fc5c9fff700 1 --2- 172.21.15.98:0/459340750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5c40b5410 0x7fc5c40d13d0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.810+0000 7fc5c9fff700 1 --2- 172.21.15.98:0/459340750 >> v2:172.21.15.100:3300/0 conn(0x7fc5c40b8b90 0x7fc5c40cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.810+0000 7fc5c9fff700 1 -- 172.21.15.98:0/459340750 >> 172.21.15.98:0/459340750 conn(0x7fc5c406aee0 msgr2=0x7fc5c406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.810+0000 7fc5c9fff700 1 -- 172.21.15.98:0/459340750 shutdown_connections 2023-03-03T08:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.811+0000 7fc5c9fff700 1 -- 172.21.15.98:0/459340750 wait complete. 2023-03-03T08:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.812+0000 7fc5c9fff700 1 Processor -- start 2023-03-03T08:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.813+0000 7fc5c9fff700 1 -- start start 2023-03-03T08:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5c9fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c40b5410 0x7fc5c414f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5c9fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5c40b87c0 0x7fc5c40aae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5c9fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5c40b8b90 0x7fc5c40ac420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5c9fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c414ff00 con 0x7fc5c40b87c0 2023-03-03T08:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5c9fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40cac60 con 0x7fc5c40b8b90 2023-03-03T08:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5c9fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40ab780 con 0x7fc5c40b5410 2023-03-03T08:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5bbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5c40b87c0 0x7fc5c40aae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5bbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5c40b87c0 0x7fc5c40aae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51250/0 (socket says 172.21.15.98:51250) 2023-03-03T08:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5bbfff700 1 -- 172.21.15.98:0/1880265033 learned_addr learned my addr 172.21.15.98:0/1880265033 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5c37fe700 1 --2- 172.21.15.98:0/1880265033 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c40b5410 0x7fc5c414f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5c3fff700 1 --2- 172.21.15.98:0/1880265033 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5c40b8b90 0x7fc5c40ac420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5bbfff700 1 -- 172.21.15.98:0/1880265033 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c40b5410 msgr2=0x7fc5c414f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5bbfff700 1 --2- 172.21.15.98:0/1880265033 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c40b5410 0x7fc5c414f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5bbfff700 1 -- 172.21.15.98:0/1880265033 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5c40b8b90 msgr2=0x7fc5c40ac420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5bbfff700 1 --2- 172.21.15.98:0/1880265033 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5c40b8b90 0x7fc5c40ac420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5bbfff700 1 -- 172.21.15.98:0/1880265033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc009b30 con 0x7fc5c40b87c0 2023-03-03T08:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.814+0000 7fc5c3fff700 1 --2- 172.21.15.98:0/1880265033 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5c40b8b90 0x7fc5c40ac420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.815+0000 7fc5bbfff700 1 --2- 172.21.15.98:0/1880265033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5c40b87c0 0x7fc5c40aae50 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00e7f0 tx=0x7fc5ac00ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.815+0000 7fc5c0ff9700 1 -- 172.21.15.98:0/1880265033 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac009550 con 0x7fc5c40b87c0 2023-03-03T08:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.815+0000 7fc5c0ff9700 1 -- 172.21.15.98:0/1880265033 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ac013680 con 0x7fc5c40b87c0 2023-03-03T08:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.815+0000 7fc5c9fff700 1 -- 172.21.15.98:0/1880265033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c40cdf30 con 0x7fc5c40b87c0 2023-03-03T08:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.815+0000 7fc5c0ff9700 1 -- 172.21.15.98:0/1880265033 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac013a10 con 0x7fc5c40b87c0 2023-03-03T08:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.815+0000 7fc5c9fff700 1 -- 172.21.15.98:0/1880265033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c4073a80 con 0x7fc5c40b87c0 2023-03-03T08:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.816+0000 7fc5c9fff700 1 -- 172.21.15.98:0/1880265033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c40ce0c0 con 0x7fc5c40b87c0 2023-03-03T08:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.818+0000 7fc5c0ff9700 1 -- 172.21.15.98:0/1880265033 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fc5ac019030 con 0x7fc5c40b87c0 2023-03-03T08:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.820+0000 7fc5c0ff9700 1 --2- 172.21.15.98:0/1880265033 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc5a4068380 0x7fc5a406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.820+0000 7fc5c0ff9700 1 -- 172.21.15.98:0/1880265033 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc5ac085c60 con 0x7fc5c40b87c0 2023-03-03T08:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.820+0000 7fc5c37fe700 1 --2- 172.21.15.98:0/1880265033 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc5a4068380 0x7fc5a406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.821+0000 7fc5c37fe700 1 --2- 172.21.15.98:0/1880265033 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc5a4068380 0x7fc5a406a830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4000f60 tx=0x7fc5b4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:13.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:13.824+0000 7fc5c0ff9700 1 -- 172.21.15.98:0/1880265033 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc5ac0523e0 con 0x7fc5c40b87c0 2023-03-03T08:03:14.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.017+0000 7fc5c9fff700 1 -- 172.21.15.98:0/1880265033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5c40b9290 con 0x7fc5c40b87c0 2023-03-03T08:03:14.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.019+0000 7fc5c0ff9700 1 -- 172.21.15.98:0/1880265033 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+18002 (secure 0 0 0) 0x7fc5ac055a00 con 0x7fc5c40b87c0 2023-03-03T08:03:14.020 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:03:14.020 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":110,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:03:13.506474+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:03:11.891955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-03-03T08:03:12.507916+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:03:14.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.022+0000 7fc5c9fff700 1 -- 172.21.15.98:0/1880265033 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc5a4068380 msgr2=0x7fc5a406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:14.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.022+0000 7fc5c9fff700 1 --2- 172.21.15.98:0/1880265033 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc5a4068380 0x7fc5a406a830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4000f60 tx=0x7fc5b4009040 comp rx=0 tx=0).stop 2023-03-03T08:03:14.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.022+0000 7fc5c9fff700 1 -- 172.21.15.98:0/1880265033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5c40b87c0 msgr2=0x7fc5c40aae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:14.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.022+0000 7fc5c9fff700 1 --2- 172.21.15.98:0/1880265033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5c40b87c0 0x7fc5c40aae50 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00e7f0 tx=0x7fc5ac00ebb0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.022+0000 7fc5c9fff700 1 -- 172.21.15.98:0/1880265033 shutdown_connections 2023-03-03T08:03:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.022+0000 7fc5c9fff700 1 --2- 172.21.15.98:0/1880265033 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5c40b8b90 0x7fc5c40ac420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.022+0000 7fc5c9fff700 1 --2- 172.21.15.98:0/1880265033 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c40b5410 0x7fc5c414f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.022+0000 7fc5c9fff700 1 --2- 172.21.15.98:0/1880265033 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc5a4068380 0x7fc5a406a830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.022+0000 7fc5c9fff700 1 --2- 172.21.15.98:0/1880265033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5c40b87c0 0x7fc5c40aae50 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.022+0000 7fc5c9fff700 1 -- 172.21.15.98:0/1880265033 >> 172.21.15.98:0/1880265033 conn(0x7fc5c406aee0 msgr2=0x7fc5c415c530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.022+0000 7fc5c9fff700 1 -- 172.21.15.98:0/1880265033 shutdown_connections 2023-03-03T08:03:14.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.022+0000 7fc5c9fff700 1 -- 172.21.15.98:0/1880265033 wait complete. 2023-03-03T08:03:14.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:03:14.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.292+0000 7fe5afdff700 1 Processor -- start 2023-03-03T08:03:14.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.293+0000 7fe5afdff700 1 -- start start 2023-03-03T08:03:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.294+0000 7fe5afdff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe5a815cc00 0x7fe5a80cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.294+0000 7fe5afdff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe5a80d27c0 0x7fe5a80d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.294+0000 7fe5afdff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe5a80b60d0 0x7fe5a80b6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.294+0000 7fe5afdff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a80ce040 con 0x7fe5a80b60d0 2023-03-03T08:03:14.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.294+0000 7fe5afdff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a8073be0 con 0x7fe5a815cc00 2023-03-03T08:03:14.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.294+0000 7fe5afdff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a80cde10 con 0x7fe5a80d27c0 2023-03-03T08:03:14.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.295+0000 7fe5ae39c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe5a80b60d0 0x7fe5a80b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.295+0000 7fe5ae39c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe5a80b60d0 0x7fe5a80b6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51252/0 (socket says 172.21.15.98:51252) 2023-03-03T08:03:14.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.295+0000 7fe5ae39c700 1 -- 172.21.15.98:0/1562592059 learned_addr learned my addr 172.21.15.98:0/1562592059 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:14.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.295+0000 7fe5ad39a700 1 --2- 172.21.15.98:0/1562592059 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe5a80d27c0 0x7fe5a80d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.295+0000 7fe5adb9b700 1 --2- 172.21.15.98:0/1562592059 >> v2:172.21.15.100:3300/0 conn(0x7fe5a815cc00 0x7fe5a80cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.295+0000 7fe5ae39c700 1 -- 172.21.15.98:0/1562592059 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe5a80d27c0 msgr2=0x7fe5a80d2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:14.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.295+0000 7fe5ae39c700 1 --2- 172.21.15.98:0/1562592059 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe5a80d27c0 0x7fe5a80d2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.295+0000 7fe5ae39c700 1 -- 172.21.15.98:0/1562592059 >> v2:172.21.15.100:3300/0 conn(0x7fe5a815cc00 msgr2=0x7fe5a80cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:14.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.295+0000 7fe5ae39c700 1 --2- 172.21.15.98:0/1562592059 >> v2:172.21.15.100:3300/0 conn(0x7fe5a815cc00 0x7fe5a80cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.295+0000 7fe5ae39c700 1 -- 172.21.15.98:0/1562592059 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a80d1a00 con 0x7fe5a80b60d0 2023-03-03T08:03:14.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.295+0000 7fe5ae39c700 1 --2- 172.21.15.98:0/1562592059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5a80b60d0 0x7fe5a80b6550 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fe5a808c200 tx=0x7fe5a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62ff6cfb52d96ca server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:14.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.296+0000 7fe5acb99700 1 -- 172.21.15.98:0/1562592059 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a000e070 con 0x7fe5a80b60d0 2023-03-03T08:03:14.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.296+0000 7fe5acb99700 1 -- 172.21.15.98:0/1562592059 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a000b9c0 con 0x7fe5a80b60d0 2023-03-03T08:03:14.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.296+0000 7fe5acb99700 1 -- 172.21.15.98:0/1562592059 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a000bd70 con 0x7fe5a80b60d0 2023-03-03T08:03:14.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.296+0000 7fe5afdff700 1 -- 172.21.15.98:0/1562592059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5a80b60d0 msgr2=0x7fe5a80b6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:14.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.296+0000 7fe5afdff700 1 --2- 172.21.15.98:0/1562592059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5a80b60d0 0x7fe5a80b6550 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fe5a808c200 tx=0x7fe5a000b330 comp rx=0 tx=0).stop 2023-03-03T08:03:14.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.296+0000 7fe5afdff700 1 -- 172.21.15.98:0/1562592059 shutdown_connections 2023-03-03T08:03:14.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.296+0000 7fe5afdff700 1 --2- 172.21.15.98:0/1562592059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5a80b60d0 0x7fe5a80b6550 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.296+0000 7fe5afdff700 1 --2- 172.21.15.98:0/1562592059 >> v2:172.21.15.100:3300/0 conn(0x7fe5a815cc00 0x7fe5a80cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.296+0000 7fe5afdff700 1 --2- 172.21.15.98:0/1562592059 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe5a80d27c0 0x7fe5a80d2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.296+0000 7fe5afdff700 1 -- 172.21.15.98:0/1562592059 >> 172.21.15.98:0/1562592059 conn(0x7fe5a806aee0 msgr2=0x7fe5a806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:14.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.296+0000 7fe5afdff700 1 -- 172.21.15.98:0/1562592059 shutdown_connections 2023-03-03T08:03:14.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.297+0000 7fe5afdff700 1 -- 172.21.15.98:0/1562592059 wait complete. 2023-03-03T08:03:14.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.299+0000 7fe5afdff700 1 Processor -- start 2023-03-03T08:03:14.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.299+0000 7fe5afdff700 1 -- start start 2023-03-03T08:03:14.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.300+0000 7fe5afdff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe5a80b60d0 0x7fe5a8158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.300+0000 7fe5afdff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5a80d27c0 0x7fe5a815b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.300+0000 7fe5afdff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe5a815cc00 0x7fe5a815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.300+0000 7fe5afdff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a81f6510 con 0x7fe5a80d27c0 2023-03-03T08:03:14.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.300+0000 7fe5afdff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a80cd480 con 0x7fe5a80b60d0 2023-03-03T08:03:14.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.300+0000 7fe5afdff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a80cc6c0 con 0x7fe5a815cc00 2023-03-03T08:03:14.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.300+0000 7fe5ad39a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5a80d27c0 0x7fe5a815b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.300+0000 7fe5ad39a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5a80d27c0 0x7fe5a815b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51264/0 (socket says 172.21.15.98:51264) 2023-03-03T08:03:14.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.300+0000 7fe5ad39a700 1 -- 172.21.15.98:0/3578702982 learned_addr learned my addr 172.21.15.98:0/3578702982 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:14.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.301+0000 7fe5ad39a700 1 -- 172.21.15.98:0/3578702982 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe5a815cc00 msgr2=0x7fe5a815bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:14.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.301+0000 7fe5adb9b700 1 --2- 172.21.15.98:0/3578702982 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe5a80b60d0 0x7fe5a8158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.301+0000 7fe5ad39a700 1 --2- 172.21.15.98:0/3578702982 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe5a815cc00 0x7fe5a815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.301+0000 7fe5ad39a700 1 -- 172.21.15.98:0/3578702982 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe5a80b60d0 msgr2=0x7fe5a8158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:14.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.301+0000 7fe5ad39a700 1 --2- 172.21.15.98:0/3578702982 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe5a80b60d0 0x7fe5a8158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.301+0000 7fe5ad39a700 1 -- 172.21.15.98:0/3578702982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a0009b30 con 0x7fe5a80d27c0 2023-03-03T08:03:14.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.301+0000 7fe5ad39a700 1 --2- 172.21.15.98:0/3578702982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5a80d27c0 0x7fe5a815b7a0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fe59800d800 tx=0x7fe59800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.301+0000 7fe59e7fc700 1 -- 172.21.15.98:0/3578702982 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5980125f0 con 0x7fe5a80d27c0 2023-03-03T08:03:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.301+0000 7fe59e7fc700 1 -- 172.21.15.98:0/3578702982 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe598012750 con 0x7fe5a80d27c0 2023-03-03T08:03:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.301+0000 7fe59e7fc700 1 -- 172.21.15.98:0/3578702982 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598012ae0 con 0x7fe5a80d27c0 2023-03-03T08:03:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.301+0000 7fe5afdff700 1 -- 172.21.15.98:0/3578702982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a80d1fc0 con 0x7fe5a80d27c0 2023-03-03T08:03:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.301+0000 7fe5afdff700 1 -- 172.21.15.98:0/3578702982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a80d1ce0 con 0x7fe5a80d27c0 2023-03-03T08:03:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.303+0000 7fe5afdff700 1 -- 172.21.15.98:0/3578702982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a80d22a0 con 0x7fe5a80d27c0 2023-03-03T08:03:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.304+0000 7fe59e7fc700 1 -- 172.21.15.98:0/3578702982 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fe598012c40 con 0x7fe5a80d27c0 2023-03-03T08:03:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.305+0000 7fe59e7fc700 1 --2- 172.21.15.98:0/3578702982 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe58c068380 0x7fe58c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.305+0000 7fe59e7fc700 1 -- 172.21.15.98:0/3578702982 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe598086260 con 0x7fe5a80d27c0 2023-03-03T08:03:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.306+0000 7fe5adb9b700 1 --2- 172.21.15.98:0/3578702982 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe58c068380 0x7fe58c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.307+0000 7fe5adb9b700 1 --2- 172.21.15.98:0/3578702982 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe58c068380 0x7fe58c06a830 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4009950 tx=0x7fe5a4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:14.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.309+0000 7fe59e7fc700 1 -- 172.21.15.98:0/3578702982 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe5980529e0 con 0x7fe5a80d27c0 2023-03-03T08:03:14.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.501+0000 7fe5afdff700 1 -- 172.21.15.98:0/3578702982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5a80729b0 con 0x7fe5a80d27c0 2023-03-03T08:03:14.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.503+0000 7fe59e7fc700 1 -- 172.21.15.98:0/3578702982 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+18002 (secure 0 0 0) 0x7fe598012ed0 con 0x7fe5a80d27c0 2023-03-03T08:03:14.504 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:03:14.504 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":110,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:03:13.506474+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:03:11.891955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-03-03T08:03:12.507916+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:03:14.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.505+0000 7fe5afdff700 1 -- 172.21.15.98:0/3578702982 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe58c068380 msgr2=0x7fe58c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.505+0000 7fe5afdff700 1 --2- 172.21.15.98:0/3578702982 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe58c068380 0x7fe58c06a830 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4009950 tx=0x7fe5a4008040 comp rx=0 tx=0).stop 2023-03-03T08:03:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.505+0000 7fe5afdff700 1 -- 172.21.15.98:0/3578702982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5a80d27c0 msgr2=0x7fe5a815b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.505+0000 7fe5afdff700 1 --2- 172.21.15.98:0/3578702982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5a80d27c0 0x7fe5a815b7a0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fe59800d800 tx=0x7fe59800dbc0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.505+0000 7fe5afdff700 1 -- 172.21.15.98:0/3578702982 shutdown_connections 2023-03-03T08:03:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.505+0000 7fe5afdff700 1 --2- 172.21.15.98:0/3578702982 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe5a815cc00 0x7fe5a815bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.505+0000 7fe5afdff700 1 --2- 172.21.15.98:0/3578702982 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe5a80b60d0 0x7fe5a8158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.505+0000 7fe5afdff700 1 --2- 172.21.15.98:0/3578702982 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe58c068380 0x7fe58c06a830 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.505+0000 7fe5afdff700 1 --2- 172.21.15.98:0/3578702982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5a80d27c0 0x7fe5a815b7a0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.505+0000 7fe5afdff700 1 -- 172.21.15.98:0/3578702982 >> 172.21.15.98:0/3578702982 conn(0x7fe5a806aee0 msgr2=0x7fe5a80b6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:14.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.505+0000 7fe5afdff700 1 -- 172.21.15.98:0/3578702982 shutdown_connections 2023-03-03T08:03:14.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.506+0000 7fe5afdff700 1 -- 172.21.15.98:0/3578702982 wait complete. 2023-03-03T08:03:14.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-03-03T08:03:14.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.783+0000 7f93d19ff700 1 Processor -- start 2023-03-03T08:03:14.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.783+0000 7f93d19ff700 1 -- start start 2023-03-03T08:03:14.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.784+0000 7f93d19ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93cc0b8d10 0x7f93cc0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.785+0000 7f93d19ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f93cc0b5310 0x7f93cc0d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.785+0000 7f93d19ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93cc0cd7d0 con 0x7f93cc0b26d0 2023-03-03T08:03:14.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.785+0000 7f93d19ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93cc0b90e0 con 0x7f93cc0b8d10 2023-03-03T08:03:14.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.785+0000 7f93d19ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93cc0cda20 con 0x7f93cc0b5310 2023-03-03T08:03:14.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.785+0000 7f93caffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93cc0b8d10 0x7f93cc0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.785+0000 7f93caffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93cc0b8d10 0x7f93cc0cffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51268/0 (socket says 172.21.15.98:51268) 2023-03-03T08:03:14.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.785+0000 7f93caffd700 1 -- 172.21.15.98:0/3165182368 learned_addr learned my addr 172.21.15.98:0/3165182368 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:14.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.785+0000 7f93ca7fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f93cc0b26d0 0x7f93cc0b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42026/0 (socket says 172.21.15.98:42026) 2023-03-03T08:03:14.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.785+0000 7f93cb7fe700 1 --2- 172.21.15.98:0/3165182368 >> v2:172.21.15.100:3300/0 conn(0x7f93cc0b5310 0x7f93cc0d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.785+0000 7f93caffd700 1 -- 172.21.15.98:0/3165182368 >> v2:172.21.15.100:3300/0 conn(0x7f93cc0b5310 msgr2=0x7f93cc0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.786+0000 7f93caffd700 1 --2- 172.21.15.98:0/3165182368 >> v2:172.21.15.100:3300/0 conn(0x7f93cc0b5310 0x7f93cc0d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.786+0000 7f93caffd700 1 -- 172.21.15.98:0/3165182368 >> v1:172.21.15.98:6789/0 conn(0x7f93cc0b26d0 legacy=0x7f93cc0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.786+0000 7f93caffd700 1 -- 172.21.15.98:0/3165182368 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93cc0cdd00 con 0x7f93cc0b8d10 2023-03-03T08:03:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.786+0000 7f93c9ffb700 1 -- 172.21.15.98:0/3165182368 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728100807 0 0) 0x7f93cc0cd7d0 con 0x7f93cc0b26d0 2023-03-03T08:03:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.786+0000 7f93caffd700 1 --2- 172.21.15.98:0/3165182368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc0b8d10 0x7f93cc0cffc0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f93cc06b740 tx=0x7f93bc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58e4e2f0d281221 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.786+0000 7f93c9ffb700 1 -- 172.21.15.98:0/3165182368 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc00b870 con 0x7f93cc0b8d10 2023-03-03T08:03:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.786+0000 7f93c9ffb700 1 -- 172.21.15.98:0/3165182368 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93bc00b9d0 con 0x7f93cc0b8d10 2023-03-03T08:03:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.786+0000 7f93c9ffb700 1 -- 172.21.15.98:0/3165182368 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc00bd60 con 0x7f93cc0b8d10 2023-03-03T08:03:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.787+0000 7f93d19ff700 1 -- 172.21.15.98:0/3165182368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc0b8d10 msgr2=0x7f93cc0cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.787+0000 7f93d19ff700 1 --2- 172.21.15.98:0/3165182368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc0b8d10 0x7f93cc0cffc0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f93cc06b740 tx=0x7f93bc009bd0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.787+0000 7f93d19ff700 1 -- 172.21.15.98:0/3165182368 shutdown_connections 2023-03-03T08:03:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.787+0000 7f93d19ff700 1 --2- 172.21.15.98:0/3165182368 >> v2:172.21.15.100:3300/0 conn(0x7f93cc0b5310 0x7f93cc0d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.787+0000 7f93d19ff700 1 --2- 172.21.15.98:0/3165182368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc0b8d10 0x7f93cc0cffc0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.787+0000 7f93d19ff700 1 -- 172.21.15.98:0/3165182368 >> 172.21.15.98:0/3165182368 conn(0x7f93cc073e70 msgr2=0x7f93cc06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:14.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.787+0000 7f93d19ff700 1 -- 172.21.15.98:0/3165182368 shutdown_connections 2023-03-03T08:03:14.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.787+0000 7f93d19ff700 1 -- 172.21.15.98:0/3165182368 wait complete. 2023-03-03T08:03:14.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.789+0000 7f93d19ff700 1 Processor -- start 2023-03-03T08:03:14.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.789+0000 7f93d19ff700 1 -- start start 2023-03-03T08:03:14.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.790+0000 7f93d19ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93cc0b26d0 0x7f93cc15e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.790+0000 7f93d19ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc0b5310 0x7f93cc1564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.790+0000 7f93d19ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93cc0b8d10 0x7f93cc156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.790+0000 7f93d19ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc15f4a0 con 0x7f93cc0b5310 2023-03-03T08:03:14.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.790+0000 7f93d19ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc143410 con 0x7f93cc0b8d10 2023-03-03T08:03:14.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.790+0000 7f93d19ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc15e580 con 0x7f93cc0b26d0 2023-03-03T08:03:14.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.790+0000 7f93ca7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc0b5310 0x7f93cc1564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.790+0000 7f93ca7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc0b5310 0x7f93cc1564d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51282/0 (socket says 172.21.15.98:51282) 2023-03-03T08:03:14.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.790+0000 7f93ca7fc700 1 -- 172.21.15.98:0/1387975909 learned_addr learned my addr 172.21.15.98:0/1387975909 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:14.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.791+0000 7f93caffd700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93cc0b26d0 0x7f93cc15e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.791+0000 7f93cb7fe700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93cc0b8d10 0x7f93cc156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.791+0000 7f93ca7fc700 1 -- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93cc0b26d0 msgr2=0x7f93cc15e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:14.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.791+0000 7f93ca7fc700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93cc0b26d0 0x7f93cc15e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.791+0000 7f93ca7fc700 1 -- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93cc0b8d10 msgr2=0x7f93cc156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:14.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.791+0000 7f93ca7fc700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93cc0b8d10 0x7f93cc156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.791+0000 7f93ca7fc700 1 -- 172.21.15.98:0/1387975909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93bc009550 con 0x7f93cc0b5310 2023-03-03T08:03:14.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.791+0000 7f93cb7fe700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93cc0b8d10 0x7f93cc156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:03:14.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.791+0000 7f93caffd700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93cc0b26d0 0x7f93cc15e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:03:14.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.791+0000 7f93ca7fc700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc0b5310 0x7f93cc1564d0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f93c000c550 tx=0x7f93c000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.791+0000 7f93b3fff700 1 -- 172.21.15.98:0/1387975909 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c0004c80 con 0x7f93cc0b5310 2023-03-03T08:03:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.791+0000 7f93b3fff700 1 -- 172.21.15.98:0/1387975909 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c0004e00 con 0x7f93cc0b5310 2023-03-03T08:03:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.791+0000 7f93d19ff700 1 -- 172.21.15.98:0/1387975909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93cc0cdfe0 con 0x7f93cc0b5310 2023-03-03T08:03:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.791+0000 7f93b3fff700 1 -- 172.21.15.98:0/1387975909 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c0007310 con 0x7f93cc0b5310 2023-03-03T08:03:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.792+0000 7f93d19ff700 1 -- 172.21.15.98:0/1387975909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93cc0d0e80 con 0x7f93cc0b5310 2023-03-03T08:03:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.793+0000 7f93d19ff700 1 -- 172.21.15.98:0/1387975909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93cc0d0500 con 0x7f93cc0b5310 2023-03-03T08:03:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.794+0000 7f93b3fff700 1 -- 172.21.15.98:0/1387975909 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f93c001a020 con 0x7f93cc0b5310 2023-03-03T08:03:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.795+0000 7f93b3fff700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f93b8068380 0x7f93b806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.795+0000 7f93b3fff700 1 -- 172.21.15.98:0/1387975909 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f93c0085a20 con 0x7f93cc0b5310 2023-03-03T08:03:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.796+0000 7f93caffd700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f93b8068380 0x7f93b806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.796+0000 7f93caffd700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f93b8068380 0x7f93b806a830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009c10 tx=0x7f93bc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:14.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:14.799+0000 7f93b3fff700 1 -- 172.21.15.98:0/1387975909 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f93c0052140 con 0x7f93cc0b5310 2023-03-03T08:03:15.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:15.040+0000 7f93d19ff700 1 -- 172.21.15.98:0/1387975909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f93cc04e6e0 con 0x7f93cc0b5310 2023-03-03T08:03:15.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:15.968+0000 7f93b3fff700 1 -- 172.21.15.98:0/1387975909 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v61) v1 ==== 118+0+0 (secure 0 0 0) 0x7f93c0085020 con 0x7f93cc0b5310 2023-03-03T08:03:15.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:15.971+0000 7f93d19ff700 1 -- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f93b8068380 msgr2=0x7f93b806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:15.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:15.971+0000 7f93d19ff700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f93b8068380 0x7f93b806a830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009c10 tx=0x7f93bc008030 comp rx=0 tx=0).stop 2023-03-03T08:03:15.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:15.971+0000 7f93d19ff700 1 -- 172.21.15.98:0/1387975909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc0b5310 msgr2=0x7f93cc1564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:15.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:15.971+0000 7f93d19ff700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc0b5310 0x7f93cc1564d0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f93c000c550 tx=0x7f93c000c910 comp rx=0 tx=0).stop 2023-03-03T08:03:15.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:15.972+0000 7f93d19ff700 1 -- 172.21.15.98:0/1387975909 shutdown_connections 2023-03-03T08:03:15.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:15.972+0000 7f93d19ff700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93cc0b8d10 0x7f93cc156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:15.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:15.972+0000 7f93d19ff700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93cc0b26d0 0x7f93cc15e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:15.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:15.972+0000 7f93d19ff700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f93b8068380 0x7f93b806a830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:15.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:15.972+0000 7f93d19ff700 1 --2- 172.21.15.98:0/1387975909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc0b5310 0x7f93cc1564d0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:15.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:15.972+0000 7f93d19ff700 1 -- 172.21.15.98:0/1387975909 >> 172.21.15.98:0/1387975909 conn(0x7f93cc073e70 msgr2=0x7f93cc0b3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:15.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:15.972+0000 7f93d19ff700 1 -- 172.21.15.98:0/1387975909 shutdown_connections 2023-03-03T08:03:15.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:15.972+0000 7f93d19ff700 1 -- 172.21.15.98:0/1387975909 wait complete. 2023-03-03T08:03:16.002 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-03-03T08:03:16.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-03-03T08:03:16.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.248+0000 7f71625ff700 1 Processor -- start 2023-03-03T08:03:16.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.248+0000 7f71625ff700 1 -- start start 2023-03-03T08:03:16.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.250+0000 7f71625ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f715c15dd20 0x7f715c0b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:16.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.250+0000 7f71625ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f715c0cd5c0 0x7f715c1599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:16.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.250+0000 7f71625ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c15e170 con 0x7f715c0b0800 2023-03-03T08:03:16.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.250+0000 7f71625ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715c0b1110 con 0x7f715c15dd20 2023-03-03T08:03:16.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.250+0000 7f71625ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715c0c9d50 con 0x7f715c0cd5c0 2023-03-03T08:03:16.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.250+0000 7f715b7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f715c15dd20 0x7f715c0b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:16.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.250+0000 7f715b7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f715c15dd20 0x7f715c0b0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60138/0 (socket says 172.21.15.98:60138) 2023-03-03T08:03:16.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.250+0000 7f715b7fe700 1 -- 172.21.15.98:0/394704146 learned_addr learned my addr 172.21.15.98:0/394704146 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:16.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.250+0000 7f7160b9c700 1 --2- 172.21.15.98:0/394704146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f715c0cd5c0 0x7f715c1599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.251+0000 7f715b7fe700 1 -- 172.21.15.98:0/394704146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f715c0cd5c0 msgr2=0x7f715c1599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.251+0000 7f715b7fe700 1 --2- 172.21.15.98:0/394704146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f715c0cd5c0 0x7f715c1599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.251+0000 7f715b7fe700 1 -- 172.21.15.98:0/394704146 >> v1:172.21.15.100:6789/0 conn(0x7f715c0b0800 legacy=0x7f715c0c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.251+0000 7f715b7fe700 1 -- 172.21.15.98:0/394704146 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c0ca030 con 0x7f715c15dd20 2023-03-03T08:03:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.251+0000 7f715b7fe700 1 --2- 172.21.15.98:0/394704146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f715c15dd20 0x7f715c0b0bd0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f71500098f0 tx=0x7f7150009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ca9852ac82eb76 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.251+0000 7f715affd700 1 -- 172.21.15.98:0/394704146 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150011830 con 0x7f715c15dd20 2023-03-03T08:03:16.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.251+0000 7f715affd700 1 -- 172.21.15.98:0/394704146 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7150011990 con 0x7f715c15dd20 2023-03-03T08:03:16.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.251+0000 7f715affd700 1 -- 172.21.15.98:0/394704146 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150011d20 con 0x7f715c15dd20 2023-03-03T08:03:16.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.252+0000 7f71625ff700 1 -- 172.21.15.98:0/394704146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f715c15dd20 msgr2=0x7f715c0b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:16.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.252+0000 7f71625ff700 1 --2- 172.21.15.98:0/394704146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f715c15dd20 0x7f715c0b0bd0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f71500098f0 tx=0x7f7150009c00 comp rx=0 tx=0).stop 2023-03-03T08:03:16.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.252+0000 7f71625ff700 1 -- 172.21.15.98:0/394704146 shutdown_connections 2023-03-03T08:03:16.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.252+0000 7f71625ff700 1 --2- 172.21.15.98:0/394704146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f715c0cd5c0 0x7f715c1599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:16.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.252+0000 7f71625ff700 1 --2- 172.21.15.98:0/394704146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f715c15dd20 0x7f715c0b0bd0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:16.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.252+0000 7f71625ff700 1 -- 172.21.15.98:0/394704146 >> 172.21.15.98:0/394704146 conn(0x7f715c0cb800 msgr2=0x7f715c0cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:16.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.252+0000 7f71625ff700 1 -- 172.21.15.98:0/394704146 shutdown_connections 2023-03-03T08:03:16.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.252+0000 7f71625ff700 1 -- 172.21.15.98:0/394704146 wait complete. 2023-03-03T08:03:16.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.254+0000 7f71625ff700 1 Processor -- start 2023-03-03T08:03:16.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.254+0000 7f71625ff700 1 -- start start 2023-03-03T08:03:16.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.255+0000 7f71625ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f715c0b0800 0x7f715c0c28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:16.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.255+0000 7f71625ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f715c0cd5c0 0x7f715c14ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:16.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.255+0000 7f71625ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f715c15dd20 0x7f715c0c3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:16.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.255+0000 7f71625ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c0c2df0 con 0x7f715c0b0800 2023-03-03T08:03:16.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.255+0000 7f71625ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c0ae090 con 0x7f715c0cd5c0 2023-03-03T08:03:16.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.255+0000 7f71625ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715c0ad480 con 0x7f715c15dd20 2023-03-03T08:03:16.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.256+0000 7f715bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f715c0b0800 0x7f715c0c28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:16.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.256+0000 7f715bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f715c0b0800 0x7f715c0c28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60150/0 (socket says 172.21.15.98:60150) 2023-03-03T08:03:16.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.256+0000 7f715bfff700 1 -- 172.21.15.98:0/1357250101 learned_addr learned my addr 172.21.15.98:0/1357250101 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:16.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.256+0000 7f7160b9c700 1 --2- 172.21.15.98:0/1357250101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f715c15dd20 0x7f715c0c3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:16.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.256+0000 7f715b7fe700 1 --2- 172.21.15.98:0/1357250101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f715c0cd5c0 0x7f715c14ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:16.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.256+0000 7f715bfff700 1 -- 172.21.15.98:0/1357250101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f715c15dd20 msgr2=0x7f715c0c3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:16.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.256+0000 7f715bfff700 1 --2- 172.21.15.98:0/1357250101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f715c15dd20 0x7f715c0c3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:16.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.256+0000 7f715bfff700 1 -- 172.21.15.98:0/1357250101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f715c0cd5c0 msgr2=0x7f715c14ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:16.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.256+0000 7f715bfff700 1 --2- 172.21.15.98:0/1357250101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f715c0cd5c0 0x7f715c14ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:16.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.256+0000 7f715bfff700 1 -- 172.21.15.98:0/1357250101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7150009550 con 0x7f715c0b0800 2023-03-03T08:03:16.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.257+0000 7f715bfff700 1 --2- 172.21.15.98:0/1357250101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f715c0b0800 0x7f715c0c28b0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f714c00c510 tx=0x7f714c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:16.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.257+0000 7f7158ff9700 1 -- 172.21.15.98:0/1357250101 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c0052d0 con 0x7f715c0b0800 2023-03-03T08:03:16.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.257+0000 7f7158ff9700 1 -- 172.21.15.98:0/1357250101 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c005430 con 0x7f715c0b0800 2023-03-03T08:03:16.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.257+0000 7f7158ff9700 1 -- 172.21.15.98:0/1357250101 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c0038c0 con 0x7f715c0b0800 2023-03-03T08:03:16.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.257+0000 7f71625ff700 1 -- 172.21.15.98:0/1357250101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715c0cd0a0 con 0x7f715c0b0800 2023-03-03T08:03:16.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.257+0000 7f71625ff700 1 -- 172.21.15.98:0/1357250101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715c0ccdc0 con 0x7f715c0b0800 2023-03-03T08:03:16.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.258+0000 7f71625ff700 1 -- 172.21.15.98:0/1357250101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715c0ca310 con 0x7f715c0b0800 2023-03-03T08:03:16.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.260+0000 7f7158ff9700 1 -- 172.21.15.98:0/1357250101 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f714c019030 con 0x7f715c0b0800 2023-03-03T08:03:16.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.261+0000 7f7158ff9700 1 --2- 172.21.15.98:0/1357250101 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7148068380 0x7f714806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:16.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.261+0000 7f7158ff9700 1 -- 172.21.15.98:0/1357250101 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f714c085130 con 0x7f715c0b0800 2023-03-03T08:03:16.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.261+0000 7f715b7fe700 1 --2- 172.21.15.98:0/1357250101 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7148068380 0x7f714806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:16.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.262+0000 7f715b7fe700 1 --2- 172.21.15.98:0/1357250101 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7148068380 0x7f714806a830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7150014040 tx=0x7f7150008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:16.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.265+0000 7f7158ff9700 1 -- 172.21.15.98:0/1357250101 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f714c0518b0 con 0x7f715c0b0800 2023-03-03T08:03:16.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.500+0000 7f71625ff700 1 -- 172.21.15.98:0/1357250101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f715c04e6e0 con 0x7f715c0b0800 2023-03-03T08:03:16.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.977+0000 7f7158ff9700 1 -- 172.21.15.98:0/1357250101 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v62) v1 ==== 122+0+0 (secure 0 0 0) 0x7f714c083610 con 0x7f715c0b0800 2023-03-03T08:03:16.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.980+0000 7f71625ff700 1 -- 172.21.15.98:0/1357250101 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7148068380 msgr2=0x7f714806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:16.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.980+0000 7f71625ff700 1 --2- 172.21.15.98:0/1357250101 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7148068380 0x7f714806a830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7150014040 tx=0x7f7150008030 comp rx=0 tx=0).stop 2023-03-03T08:03:16.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.980+0000 7f71625ff700 1 -- 172.21.15.98:0/1357250101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f715c0b0800 msgr2=0x7f715c0c28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:16.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.980+0000 7f71625ff700 1 --2- 172.21.15.98:0/1357250101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f715c0b0800 0x7f715c0c28b0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f714c00c510 tx=0x7f714c00c8d0 comp rx=0 tx=0).stop 2023-03-03T08:03:16.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.980+0000 7f71625ff700 1 -- 172.21.15.98:0/1357250101 shutdown_connections 2023-03-03T08:03:16.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.980+0000 7f71625ff700 1 --2- 172.21.15.98:0/1357250101 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7148068380 0x7f714806a830 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:16.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.980+0000 7f71625ff700 1 --2- 172.21.15.98:0/1357250101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f715c0b0800 0x7f715c0c28b0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:16.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.980+0000 7f71625ff700 1 --2- 172.21.15.98:0/1357250101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f715c15dd20 0x7f715c0c3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:16.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.980+0000 7f71625ff700 1 --2- 172.21.15.98:0/1357250101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f715c0cd5c0 0x7f715c14ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:16.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.980+0000 7f71625ff700 1 -- 172.21.15.98:0/1357250101 >> 172.21.15.98:0/1357250101 conn(0x7f715c0cb800 msgr2=0x7f715c153260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:16.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.981+0000 7f71625ff700 1 -- 172.21.15.98:0/1357250101 shutdown_connections 2023-03-03T08:03:16.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:16.981+0000 7f71625ff700 1 -- 172.21.15.98:0/1357250101 wait complete. 2023-03-03T08:03:17.012 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-03-03T08:03:17.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:03:17.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.256+0000 7ffafd7ff700 1 Processor -- start 2023-03-03T08:03:17.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.256+0000 7ffafd7ff700 1 -- start start 2023-03-03T08:03:17.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.258+0000 7ffafd7ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ffaf80cd8d0 0x7ffaf80b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:17.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.258+0000 7ffafd7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80b6610 0x7ffaf80cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:17.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.258+0000 7ffafd7ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf80cdd70 con 0x7ffaf815cc00 2023-03-03T08:03:17.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.258+0000 7ffafd7ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf80ce040 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.258+0000 7ffafd7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf8073be0 con 0x7ffaf80b6610 2023-03-03T08:03:17.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.258+0000 7ffaf67fc700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ffaf80cd8d0 0x7ffaf80b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:17.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.258+0000 7ffaf77fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80b6610 0x7ffaf80cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:17.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.258+0000 7ffaf67fc700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ffaf80cd8d0 0x7ffaf80b60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:58076/0 (socket says 172.21.15.98:58076) 2023-03-03T08:03:17.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.258+0000 7ffaf67fc700 1 -- 172.21.15.98:0/937821263 learned_addr learned my addr 172.21.15.98:0/937821263 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.259+0000 7ffaf67fc700 1 -- 172.21.15.98:0/937821263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80b6610 msgr2=0x7ffaf80cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.259+0000 7ffaf67fc700 1 --2- 172.21.15.98:0/937821263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80b6610 0x7ffaf80cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.259+0000 7ffaf67fc700 1 -- 172.21.15.98:0/937821263 >> v1:172.21.15.100:6789/0 conn(0x7ffaf815cc00 legacy=0x7ffaf80d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.259+0000 7ffaf67fc700 1 -- 172.21.15.98:0/937821263 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf80d1a00 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.259+0000 7ffaf67fc700 1 --2- 172.21.15.98:0/937821263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffaf80cd8d0 0x7ffaf80b60d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0098f0 tx=0x7ffaec009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1d9a5e188cfbf6d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.260+0000 7ffaf5ffb700 1 -- 172.21.15.98:0/937821263 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec011830 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.260+0000 7ffaf5ffb700 1 -- 172.21.15.98:0/937821263 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaec011990 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.260+0000 7ffaf5ffb700 1 -- 172.21.15.98:0/937821263 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec011d20 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.260+0000 7ffafd7ff700 1 -- 172.21.15.98:0/937821263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffaf80cd8d0 msgr2=0x7ffaf80b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.260+0000 7ffafd7ff700 1 --2- 172.21.15.98:0/937821263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffaf80cd8d0 0x7ffaf80b60d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0098f0 tx=0x7ffaec009c00 comp rx=0 tx=0).stop 2023-03-03T08:03:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.260+0000 7ffafd7ff700 1 -- 172.21.15.98:0/937821263 shutdown_connections 2023-03-03T08:03:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.260+0000 7ffafd7ff700 1 --2- 172.21.15.98:0/937821263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80b6610 0x7ffaf80cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.260+0000 7ffafd7ff700 1 --2- 172.21.15.98:0/937821263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffaf80cd8d0 0x7ffaf80b60d0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:17.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.260+0000 7ffafd7ff700 1 -- 172.21.15.98:0/937821263 >> 172.21.15.98:0/937821263 conn(0x7ffaf806aee0 msgr2=0x7ffaf806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:17.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.260+0000 7ffafd7ff700 1 -- 172.21.15.98:0/937821263 shutdown_connections 2023-03-03T08:03:17.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.260+0000 7ffafd7ff700 1 -- 172.21.15.98:0/937821263 wait complete. 2023-03-03T08:03:17.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.262+0000 7ffafd7ff700 1 Processor -- start 2023-03-03T08:03:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.262+0000 7ffafd7ff700 1 -- start start 2023-03-03T08:03:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.263+0000 7ffafd7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80b6610 0x7ffaf80c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.263+0000 7ffafd7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaf80cd8d0 0x7ffaf813b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.263+0000 7ffafd7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffaf815cc00 0x7ffaf80b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.263+0000 7ffafd7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80cc300 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.263+0000 7ffafd7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf815ddc0 con 0x7ffaf815cc00 2023-03-03T08:03:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.263+0000 7ffafd7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf815d820 con 0x7ffaf80b6610 2023-03-03T08:03:17.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.264+0000 7ffaf67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaf80cd8d0 0x7ffaf813b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:17.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.264+0000 7ffaf67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaf80cd8d0 0x7ffaf813b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60166/0 (socket says 172.21.15.98:60166) 2023-03-03T08:03:17.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.264+0000 7ffaf67fc700 1 -- 172.21.15.98:0/1741749263 learned_addr learned my addr 172.21.15.98:0/1741749263 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:17.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.264+0000 7ffaf6ffd700 1 --2- 172.21.15.98:0/1741749263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80b6610 0x7ffaf80c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:17.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.264+0000 7ffaf77fe700 1 --2- 172.21.15.98:0/1741749263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffaf815cc00 0x7ffaf80b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:17.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.264+0000 7ffaf67fc700 1 -- 172.21.15.98:0/1741749263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80b6610 msgr2=0x7ffaf80c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:17.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.264+0000 7ffaf67fc700 1 --2- 172.21.15.98:0/1741749263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80b6610 0x7ffaf80c9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:17.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.264+0000 7ffaf67fc700 1 -- 172.21.15.98:0/1741749263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffaf815cc00 msgr2=0x7ffaf80b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:17.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.264+0000 7ffaf67fc700 1 --2- 172.21.15.98:0/1741749263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffaf815cc00 0x7ffaf80b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:17.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.264+0000 7ffaf67fc700 1 -- 172.21.15.98:0/1741749263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec009550 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.265+0000 7ffaf67fc700 1 --2- 172.21.15.98:0/1741749263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaf80cd8d0 0x7ffaf813b6c0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7ffaec009c40 tx=0x7ffaec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:17.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.265+0000 7ffadf7fe700 1 -- 172.21.15.98:0/1741749263 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec004400 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.265+0000 7ffadf7fe700 1 -- 172.21.15.98:0/1741749263 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaec01f070 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.265+0000 7ffadf7fe700 1 -- 172.21.15.98:0/1741749263 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec0038c0 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.265+0000 7ffafd7ff700 1 -- 172.21.15.98:0/1741749263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf80d1fc0 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.265+0000 7ffafd7ff700 1 -- 172.21.15.98:0/1741749263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf80d1ce0 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.266+0000 7ffafd7ff700 1 -- 172.21.15.98:0/1741749263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf80d22a0 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.268+0000 7ffadf7fe700 1 -- 172.21.15.98:0/1741749263 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7ffaec02a020 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.269+0000 7ffadf7fe700 1 --2- 172.21.15.98:0/1741749263 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ffae4068380 0x7ffae406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:17.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.269+0000 7ffadf7fe700 1 -- 172.21.15.98:0/1741749263 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ffaec087420 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.269+0000 7ffaf6ffd700 1 --2- 172.21.15.98:0/1741749263 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ffae4068380 0x7ffae406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:17.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.270+0000 7ffaf6ffd700 1 --2- 172.21.15.98:0/1741749263 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ffae4068380 0x7ffae406a830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffae8007fd0 tx=0x7ffae8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:17.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.273+0000 7ffadf7fe700 1 -- 172.21.15.98:0/1741749263 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffaec053ba0 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.507+0000 7ffafd7ff700 1 -- 172.21.15.98:0/1741749263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffaf80729b0 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.508+0000 7ffadf7fe700 1 -- 172.21.15.98:0/1741749263 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4656 (secure 0 0 0) 0x7ffaec085900 con 0x7ffaf80cd8d0 2023-03-03T08:03:17.511 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:03:17.512 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/2563396918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":2563396918},{"type":"v1","addr":"172.21.15.100:6834","nonce":2563396918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7992,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2374561485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2374561485},{"type":"v1","addr":"172.21.15.98:6834","nonce":2374561485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7995,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2695503624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":2695503624},{"type":"v1","addr":"172.21.15.98:6835","nonce":2695503624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:03:13.506818+0000","modified":"2023-03-03T08:03:16.969687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8001},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8001":{"gid":8001,"name":"d","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/2660525287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":2660525287},{"type":"v1","addr":"172.21.15.100:6835","nonce":2660525287}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-03-03T08:03:17.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.511+0000 7ffafd7ff700 1 -- 172.21.15.98:0/1741749263 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ffae4068380 msgr2=0x7ffae406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:17.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.511+0000 7ffafd7ff700 1 --2- 172.21.15.98:0/1741749263 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ffae4068380 0x7ffae406a830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffae8007fd0 tx=0x7ffae8009040 comp rx=0 tx=0).stop 2023-03-03T08:03:17.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.511+0000 7ffafd7ff700 1 -- 172.21.15.98:0/1741749263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaf80cd8d0 msgr2=0x7ffaf813b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:17.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.511+0000 7ffafd7ff700 1 --2- 172.21.15.98:0/1741749263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaf80cd8d0 0x7ffaf813b6c0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7ffaec009c40 tx=0x7ffaec004750 comp rx=0 tx=0).stop 2023-03-03T08:03:17.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.511+0000 7ffafd7ff700 1 -- 172.21.15.98:0/1741749263 shutdown_connections 2023-03-03T08:03:17.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.511+0000 7ffafd7ff700 1 --2- 172.21.15.98:0/1741749263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffaf815cc00 0x7ffaf80b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:17.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.511+0000 7ffafd7ff700 1 --2- 172.21.15.98:0/1741749263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80b6610 0x7ffaf80c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:17.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.511+0000 7ffafd7ff700 1 --2- 172.21.15.98:0/1741749263 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ffae4068380 0x7ffae406a830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:17.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.511+0000 7ffafd7ff700 1 --2- 172.21.15.98:0/1741749263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaf80cd8d0 0x7ffaf813b6c0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:17.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.511+0000 7ffafd7ff700 1 -- 172.21.15.98:0/1741749263 >> 172.21.15.98:0/1741749263 conn(0x7ffaf806aee0 msgr2=0x7ffaf8159910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:17.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.511+0000 7ffafd7ff700 1 -- 172.21.15.98:0/1741749263 shutdown_connections 2023-03-03T08:03:17.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.511+0000 7ffafd7ff700 1 -- 172.21.15.98:0/1741749263 wait complete. 2023-03-03T08:03:17.517 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 62 2023-03-03T08:03:17.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:03:17.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.787+0000 7f727e9ff700 1 Processor -- start 2023-03-03T08:03:17.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.787+0000 7f727e9ff700 1 -- start start 2023-03-03T08:03:17.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.788+0000 7f727e9ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f72780b87c0 0x7f72780cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:17.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.789+0000 7f727e9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72780b8b90 0x7f72780b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:17.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.789+0000 7f727e9ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72780d0ef0 con 0x7f72780b5240 2023-03-03T08:03:17.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.789+0000 7f727e9ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72780d0600 con 0x7f72780b87c0 2023-03-03T08:03:17.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.789+0000 7f727e9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72780cd970 con 0x7f72780b8b90 2023-03-03T08:03:17.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.789+0000 7f7277fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f72780b87c0 0x7f72780cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:17.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.789+0000 7f72777fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72780b8b90 0x7f72780b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:17.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.789+0000 7f72777fe700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72780b8b90 0x7f72780b4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:51554/0 (socket says 172.21.15.98:51554) 2023-03-03T08:03:17.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.789+0000 7f72777fe700 1 -- 172.21.15.98:0/3964648248 learned_addr learned my addr 172.21.15.98:0/3964648248 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:17.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.790+0000 7f72777fe700 1 -- 172.21.15.98:0/3964648248 >> v2:172.21.15.100:3300/0 conn(0x7f72780b87c0 msgr2=0x7f72780cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:17.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.790+0000 7f72777fe700 1 --2- 172.21.15.98:0/3964648248 >> v2:172.21.15.100:3300/0 conn(0x7f72780b87c0 0x7f72780cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:17.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.790+0000 7f72777fe700 1 -- 172.21.15.98:0/3964648248 >> v1:172.21.15.100:6789/0 conn(0x7f72780b5240 legacy=0x7f72781605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:17.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.790+0000 7f72777fe700 1 -- 172.21.15.98:0/3964648248 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72780cdc50 con 0x7f72780b8b90 2023-03-03T08:03:17.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.790+0000 7f7277fff700 1 --2- 172.21.15.98:0/3964648248 >> v2:172.21.15.100:3300/0 conn(0x7f72780b87c0 0x7f72780cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:03:17.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.790+0000 7f72777fe700 1 --2- 172.21.15.98:0/3964648248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72780b8b90 0x7f72780b4d00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f726c0098f0 tx=0x7f726c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c66367e836fe171 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:17.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.790+0000 7f7276ffd700 1 -- 172.21.15.98:0/3964648248 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c011830 con 0x7f72780b8b90 2023-03-03T08:03:17.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.791+0000 7f7276ffd700 1 -- 172.21.15.98:0/3964648248 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726c011990 con 0x7f72780b8b90 2023-03-03T08:03:17.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.791+0000 7f7276ffd700 1 -- 172.21.15.98:0/3964648248 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c011d20 con 0x7f72780b8b90 2023-03-03T08:03:17.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.791+0000 7f727e9ff700 1 -- 172.21.15.98:0/3964648248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72780b8b90 msgr2=0x7f72780b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:17.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.791+0000 7f727e9ff700 1 --2- 172.21.15.98:0/3964648248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72780b8b90 0x7f72780b4d00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f726c0098f0 tx=0x7f726c009c00 comp rx=0 tx=0).stop 2023-03-03T08:03:17.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.791+0000 7f727e9ff700 1 -- 172.21.15.98:0/3964648248 shutdown_connections 2023-03-03T08:03:17.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.791+0000 7f727e9ff700 1 --2- 172.21.15.98:0/3964648248 >> v2:172.21.15.100:3300/0 conn(0x7f72780b87c0 0x7f72780cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:17.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.791+0000 7f727e9ff700 1 --2- 172.21.15.98:0/3964648248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72780b8b90 0x7f72780b4d00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:17.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.791+0000 7f727e9ff700 1 -- 172.21.15.98:0/3964648248 >> 172.21.15.98:0/3964648248 conn(0x7f727806aee0 msgr2=0x7f727806b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:17.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.791+0000 7f727e9ff700 1 -- 172.21.15.98:0/3964648248 shutdown_connections 2023-03-03T08:03:17.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.791+0000 7f727e9ff700 1 -- 172.21.15.98:0/3964648248 wait complete. 2023-03-03T08:03:17.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.793+0000 7f727e9ff700 1 Processor -- start 2023-03-03T08:03:17.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.794+0000 7f727e9ff700 1 -- start start 2023-03-03T08:03:17.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.795+0000 7f727e9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72780b5240 0x7f72780c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:17.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.795+0000 7f727e9ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72780b87c0 0x7f72780c8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:17.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.795+0000 7f727e9ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72780b8b90 0x7f727815b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:17.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.795+0000 7f727e9ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72780c9740 con 0x7f72780b87c0 2023-03-03T08:03:17.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.795+0000 7f727e9ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727815bee0 con 0x7f72780b8b90 2023-03-03T08:03:17.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.795+0000 7f727e9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72780c7390 con 0x7f72780b5240 2023-03-03T08:03:17.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.795+0000 7f72777fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72780b87c0 0x7f72780c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:17.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.795+0000 7f72777fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72780b87c0 0x7f72780c8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60172/0 (socket says 172.21.15.98:60172) 2023-03-03T08:03:17.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.795+0000 7f72777fe700 1 -- 172.21.15.98:0/573137205 learned_addr learned my addr 172.21.15.98:0/573137205 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:17.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.795+0000 7f7277fff700 1 --2- 172.21.15.98:0/573137205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72780b5240 0x7f72780c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:17.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.796+0000 7f727cf9c700 1 --2- 172.21.15.98:0/573137205 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72780b8b90 0x7f727815b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:17.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.796+0000 7f72777fe700 1 -- 172.21.15.98:0/573137205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72780b5240 msgr2=0x7f72780c6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:17.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.796+0000 7f72777fe700 1 --2- 172.21.15.98:0/573137205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72780b5240 0x7f72780c6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:17.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.796+0000 7f72777fe700 1 -- 172.21.15.98:0/573137205 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72780b8b90 msgr2=0x7f727815b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:17.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.796+0000 7f72777fe700 1 --2- 172.21.15.98:0/573137205 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72780b8b90 0x7f727815b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:17.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.796+0000 7f72777fe700 1 -- 172.21.15.98:0/573137205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72680094a0 con 0x7f72780b87c0 2023-03-03T08:03:17.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.796+0000 7f727cf9c700 1 --2- 172.21.15.98:0/573137205 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72780b8b90 0x7f727815b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:03:17.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.796+0000 7f72777fe700 1 --2- 172.21.15.98:0/573137205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72780b87c0 0x7f72780c8430 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f726c009c40 tx=0x7f726c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:17.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.796+0000 7f7274ff9700 1 -- 172.21.15.98:0/573137205 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c0036a0 con 0x7f72780b87c0 2023-03-03T08:03:17.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.796+0000 7f7274ff9700 1 -- 172.21.15.98:0/573137205 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726c01e070 con 0x7f72780b87c0 2023-03-03T08:03:17.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.796+0000 7f727e9ff700 1 -- 172.21.15.98:0/573137205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f726c009550 con 0x7f72780b87c0 2023-03-03T08:03:17.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.796+0000 7f7274ff9700 1 -- 172.21.15.98:0/573137205 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c003a60 con 0x7f72780b87c0 2023-03-03T08:03:17.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.797+0000 7f727e9ff700 1 -- 172.21.15.98:0/573137205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72780cdf30 con 0x7f72780b87c0 2023-03-03T08:03:17.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.798+0000 7f727e9ff700 1 -- 172.21.15.98:0/573137205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72780ce0c0 con 0x7f72780b87c0 2023-03-03T08:03:17.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.799+0000 7f7274ff9700 1 -- 172.21.15.98:0/573137205 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f726c00f050 con 0x7f72780b87c0 2023-03-03T08:03:17.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.800+0000 7f7274ff9700 1 --2- 172.21.15.98:0/573137205 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7264068380 0x7f726406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:17.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.800+0000 7f7274ff9700 1 -- 172.21.15.98:0/573137205 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f726c086cf0 con 0x7f72780b87c0 2023-03-03T08:03:17.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.801+0000 7f7277fff700 1 --2- 172.21.15.98:0/573137205 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7264068380 0x7f726406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:17.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.802+0000 7f7277fff700 1 --2- 172.21.15.98:0/573137205 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7264068380 0x7f726406a830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7268009d00 tx=0x7f7268008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:17.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:17.805+0000 7f7274ff9700 1 -- 172.21.15.98:0/573137205 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f726c053470 con 0x7f72780b87c0 2023-03-03T08:03:18.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.001+0000 7f727e9ff700 1 -- 172.21.15.98:0/573137205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72780729b0 con 0x7f72780b87c0 2023-03-03T08:03:18.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.003+0000 7f7274ff9700 1 -- 172.21.15.98:0/573137205 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+17989 (secure 0 0 0) 0x7f726c015090 con 0x7f72780b87c0 2023-03-03T08:03:18.005 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:03:18.006 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":111,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:03:14.509774+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:03:11.891955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-03-03T08:03:12.507916+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:03:18.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.005+0000 7f727e9ff700 1 -- 172.21.15.98:0/573137205 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7264068380 msgr2=0x7f726406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:18.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.005+0000 7f727e9ff700 1 --2- 172.21.15.98:0/573137205 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7264068380 0x7f726406a830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7268009d00 tx=0x7f7268008040 comp rx=0 tx=0).stop 2023-03-03T08:03:18.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.005+0000 7f727e9ff700 1 -- 172.21.15.98:0/573137205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72780b87c0 msgr2=0x7f72780c8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:18.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.005+0000 7f727e9ff700 1 --2- 172.21.15.98:0/573137205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72780b87c0 0x7f72780c8430 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f726c009c40 tx=0x7f726c004750 comp rx=0 tx=0).stop 2023-03-03T08:03:18.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.006+0000 7f727e9ff700 1 -- 172.21.15.98:0/573137205 shutdown_connections 2023-03-03T08:03:18.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.006+0000 7f727e9ff700 1 --2- 172.21.15.98:0/573137205 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f72780b8b90 0x7f727815b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.006+0000 7f727e9ff700 1 --2- 172.21.15.98:0/573137205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f72780b5240 0x7f72780c6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.006+0000 7f727e9ff700 1 --2- 172.21.15.98:0/573137205 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7264068380 0x7f726406a830 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.006+0000 7f727e9ff700 1 --2- 172.21.15.98:0/573137205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72780b87c0 0x7f72780c8430 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.006+0000 7f727e9ff700 1 -- 172.21.15.98:0/573137205 >> 172.21.15.98:0/573137205 conn(0x7f727806aee0 msgr2=0x7f72780b34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:18.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.006+0000 7f727e9ff700 1 -- 172.21.15.98:0/573137205 shutdown_connections 2023-03-03T08:03:18.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.006+0000 7f727e9ff700 1 -- 172.21.15.98:0/573137205 wait complete. 2023-03-03T08:03:18.036 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-03-03T08:03:18.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:03:18.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.278+0000 7fc698bff700 1 Processor -- start 2023-03-03T08:03:18.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.279+0000 7fc698bff700 1 -- start start 2023-03-03T08:03:18.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.280+0000 7fc698bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc6940b87c0 0x7fc6940cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:18.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.280+0000 7fc698bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc6940b8b90 0x7fc6940b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:18.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.280+0000 7fc698bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6940d0ef0 con 0x7fc6940b5240 2023-03-03T08:03:18.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.280+0000 7fc698bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6940d0600 con 0x7fc6940b87c0 2023-03-03T08:03:18.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.280+0000 7fc698bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6940cd970 con 0x7fc6940b8b90 2023-03-03T08:03:18.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.280+0000 7fc69259c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc6940b87c0 0x7fc6940cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:18.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.280+0000 7fc692d9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc6940b5240 0x7fc6941605b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42680/0 (socket says 172.21.15.98:42680) 2023-03-03T08:03:18.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.280+0000 7fc692d9d700 1 -- 172.21.15.98:0/3381061972 learned_addr learned my addr 172.21.15.98:0/3381061972 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:18.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.281+0000 7fc691d9b700 1 --2- 172.21.15.98:0/3381061972 >> v2:172.21.15.100:3300/0 conn(0x7fc6940b8b90 0x7fc6940b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:18.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.281+0000 7fc69259c700 1 -- 172.21.15.98:0/3381061972 >> v2:172.21.15.100:3300/0 conn(0x7fc6940b8b90 msgr2=0x7fc6940b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:18.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.281+0000 7fc69259c700 1 --2- 172.21.15.98:0/3381061972 >> v2:172.21.15.100:3300/0 conn(0x7fc6940b8b90 0x7fc6940b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.281+0000 7fc69259c700 1 -- 172.21.15.98:0/3381061972 >> v1:172.21.15.98:6789/0 conn(0x7fc6940b5240 legacy=0x7fc6941605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:18.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.281+0000 7fc69259c700 1 -- 172.21.15.98:0/3381061972 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6940cdc50 con 0x7fc6940b87c0 2023-03-03T08:03:18.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.281+0000 7fc69159a700 1 -- 172.21.15.98:0/3381061972 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412349361 0 0) 0x7fc6940d0ef0 con 0x7fc6940b5240 2023-03-03T08:03:18.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.281+0000 7fc69259c700 1 --2- 172.21.15.98:0/3381061972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6940b87c0 0x7fc6940cffe0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fc69408c240 tx=0x7fc684009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8498d97e8b7ced9f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:18.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.282+0000 7fc69159a700 1 -- 172.21.15.98:0/3381061972 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc684011830 con 0x7fc6940b87c0 2023-03-03T08:03:18.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.282+0000 7fc69159a700 1 -- 172.21.15.98:0/3381061972 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc684011990 con 0x7fc6940b87c0 2023-03-03T08:03:18.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.282+0000 7fc69159a700 1 -- 172.21.15.98:0/3381061972 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc684011d20 con 0x7fc6940b87c0 2023-03-03T08:03:18.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.282+0000 7fc698bff700 1 -- 172.21.15.98:0/3381061972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6940b87c0 msgr2=0x7fc6940cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:18.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.282+0000 7fc698bff700 1 --2- 172.21.15.98:0/3381061972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6940b87c0 0x7fc6940cffe0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fc69408c240 tx=0x7fc684009b20 comp rx=0 tx=0).stop 2023-03-03T08:03:18.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.282+0000 7fc698bff700 1 -- 172.21.15.98:0/3381061972 shutdown_connections 2023-03-03T08:03:18.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.282+0000 7fc698bff700 1 --2- 172.21.15.98:0/3381061972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6940b87c0 0x7fc6940cffe0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.282+0000 7fc698bff700 1 --2- 172.21.15.98:0/3381061972 >> v2:172.21.15.100:3300/0 conn(0x7fc6940b8b90 0x7fc6940b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.282+0000 7fc698bff700 1 -- 172.21.15.98:0/3381061972 >> 172.21.15.98:0/3381061972 conn(0x7fc69406aee0 msgr2=0x7fc69406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:18.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.282+0000 7fc698bff700 1 -- 172.21.15.98:0/3381061972 shutdown_connections 2023-03-03T08:03:18.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.283+0000 7fc698bff700 1 -- 172.21.15.98:0/3381061972 wait complete. 2023-03-03T08:03:18.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.284+0000 7fc698bff700 1 Processor -- start 2023-03-03T08:03:18.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.285+0000 7fc698bff700 1 -- start start 2023-03-03T08:03:18.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.286+0000 7fc698bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6940b5240 0x7fc6940a9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:18.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.286+0000 7fc698bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6940b87c0 0x7fc6941f5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:18.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.286+0000 7fc698bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6940b8b90 0x7fc69415b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:18.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.286+0000 7fc698bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6940cb480 con 0x7fc6940b8b90 2023-03-03T08:03:18.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.286+0000 7fc698bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6940cb760 con 0x7fc6940b5240 2023-03-03T08:03:18.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.286+0000 7fc698bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6940cba40 con 0x7fc6940b87c0 2023-03-03T08:03:18.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.286+0000 7fc692d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6940b8b90 0x7fc69415b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:18.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.286+0000 7fc692d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6940b8b90 0x7fc69415b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60192/0 (socket says 172.21.15.98:60192) 2023-03-03T08:03:18.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.286+0000 7fc692d9d700 1 -- 172.21.15.98:0/2439455317 learned_addr learned my addr 172.21.15.98:0/2439455317 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:18.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.286+0000 7fc69259c700 1 --2- 172.21.15.98:0/2439455317 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6940b5240 0x7fc6940a9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:18.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.286+0000 7fc691d9b700 1 --2- 172.21.15.98:0/2439455317 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6940b87c0 0x7fc6941f5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:18.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.287+0000 7fc692d9d700 1 -- 172.21.15.98:0/2439455317 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6940b87c0 msgr2=0x7fc6941f5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:18.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.287+0000 7fc692d9d700 1 --2- 172.21.15.98:0/2439455317 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6940b87c0 0x7fc6941f5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.287+0000 7fc692d9d700 1 -- 172.21.15.98:0/2439455317 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6940b5240 msgr2=0x7fc6940a9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:18.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.287+0000 7fc692d9d700 1 --2- 172.21.15.98:0/2439455317 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6940b5240 0x7fc6940a9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.287+0000 7fc692d9d700 1 -- 172.21.15.98:0/2439455317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6840094a0 con 0x7fc6940b8b90 2023-03-03T08:03:18.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.287+0000 7fc692d9d700 1 --2- 172.21.15.98:0/2439455317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6940b8b90 0x7fc69415b3e0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fc68c002230 tx=0x7fc68c00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:18.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.287+0000 7fc682ffd700 1 -- 172.21.15.98:0/2439455317 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c00aac0 con 0x7fc6940b8b90 2023-03-03T08:03:18.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.287+0000 7fc682ffd700 1 -- 172.21.15.98:0/2439455317 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc68c00ac40 con 0x7fc6940b8b90 2023-03-03T08:03:18.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.287+0000 7fc682ffd700 1 -- 172.21.15.98:0/2439455317 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c0070d0 con 0x7fc6940b8b90 2023-03-03T08:03:18.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.287+0000 7fc698bff700 1 -- 172.21.15.98:0/2439455317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6940cdf30 con 0x7fc6940b8b90 2023-03-03T08:03:18.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.287+0000 7fc698bff700 1 -- 172.21.15.98:0/2439455317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc694073a80 con 0x7fc6940b8b90 2023-03-03T08:03:18.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.288+0000 7fc698bff700 1 -- 172.21.15.98:0/2439455317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6940ce0c0 con 0x7fc6940b8b90 2023-03-03T08:03:18.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.290+0000 7fc682ffd700 1 -- 172.21.15.98:0/2439455317 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fc68c010070 con 0x7fc6940b8b90 2023-03-03T08:03:18.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.291+0000 7fc682ffd700 1 --2- 172.21.15.98:0/2439455317 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc674068380 0x7fc67406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:18.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.291+0000 7fc682ffd700 1 -- 172.21.15.98:0/2439455317 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc68c085640 con 0x7fc6940b8b90 2023-03-03T08:03:18.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.291+0000 7fc69259c700 1 --2- 172.21.15.98:0/2439455317 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc674068380 0x7fc67406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:18.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.292+0000 7fc69259c700 1 --2- 172.21.15.98:0/2439455317 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc674068380 0x7fc67406a830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc6840090b0 tx=0x7fc684008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:18.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.294+0000 7fc682ffd700 1 -- 172.21.15.98:0/2439455317 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc68c051dc0 con 0x7fc6940b8b90 2023-03-03T08:03:18.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.486+0000 7fc698bff700 1 -- 172.21.15.98:0/2439455317 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc6940729b0 con 0x7fc674068380 2023-03-03T08:03:18.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.499+0000 7fc682ffd700 1 -- 172.21.15.98:0/2439455317 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311616 (secure 0 0 0) 0x7fc6940729b0 con 0x7fc674068380 2023-03-03T08:03:18.505 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:03:18.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.504+0000 7fc698bff700 1 -- 172.21.15.98:0/2439455317 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc674068380 msgr2=0x7fc67406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:18.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.504+0000 7fc698bff700 1 --2- 172.21.15.98:0/2439455317 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc674068380 0x7fc67406a830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc6840090b0 tx=0x7fc684008030 comp rx=0 tx=0).stop 2023-03-03T08:03:18.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.504+0000 7fc698bff700 1 -- 172.21.15.98:0/2439455317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6940b8b90 msgr2=0x7fc69415b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:18.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.504+0000 7fc698bff700 1 --2- 172.21.15.98:0/2439455317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6940b8b90 0x7fc69415b3e0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fc68c002230 tx=0x7fc68c00cf50 comp rx=0 tx=0).stop 2023-03-03T08:03:18.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.504+0000 7fc698bff700 1 -- 172.21.15.98:0/2439455317 shutdown_connections 2023-03-03T08:03:18.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.504+0000 7fc698bff700 1 --2- 172.21.15.98:0/2439455317 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc674068380 0x7fc67406a830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.504+0000 7fc698bff700 1 --2- 172.21.15.98:0/2439455317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6940b8b90 0x7fc69415b3e0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.504+0000 7fc698bff700 1 --2- 172.21.15.98:0/2439455317 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6940b5240 0x7fc6940a9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.504+0000 7fc698bff700 1 --2- 172.21.15.98:0/2439455317 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6940b87c0 0x7fc6941f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.504+0000 7fc698bff700 1 -- 172.21.15.98:0/2439455317 >> 172.21.15.98:0/2439455317 conn(0x7fc69406aee0 msgr2=0x7fc69415f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:18.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.504+0000 7fc698bff700 1 -- 172.21.15.98:0/2439455317 shutdown_connections 2023-03-03T08:03:18.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.504+0000 7fc698bff700 1 -- 172.21.15.98:0/2439455317 wait complete. 2023-03-03T08:03:18.509 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T08:03:18.533 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2023-03-03T08:03:18.178288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":223,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166,"ondisk_log_size":166,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":131,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":106356,"kb_used_data":11956,"kb_used_omap":0,"kb_used_meta":94336,"kb_avail":754868364,"statfs":{"total":773094113280,"available":772985204736,"internally_reserved":0,"allocated":12242944,"data_stored":6291001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":96600064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001589"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523119+0000","last_change":"2023-03-03T08:03:13.533486+0000","last_active":"2023-03-03T08:03:14.523119+0000","last_peered":"2023-03-03T08:03:14.523119+0000","last_clean":"2023-03-03T08:03:14.523119+0000","last_became_active":"2023-03-03T08:03:13.533163+0000","last_became_peered":"2023-03-03T08:03:13.533163+0000","last_unstale":"2023-03-03T08:03:14.523119+0000","last_undegraded":"2023-03-03T08:03:14.523119+0000","last_fullsized":"2023-03-03T08:03:14.523119+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:19:12.616918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518473+0000","last_change":"2023-03-03T08:03:13.538123+0000","last_active":"2023-03-03T08:03:14.518473+0000","last_peered":"2023-03-03T08:03:14.518473+0000","last_clean":"2023-03-03T08:03:14.518473+0000","last_became_active":"2023-03-03T08:03:13.537834+0000","last_became_peered":"2023-03-03T08:03:13.537834+0000","last_unstale":"2023-03-03T08:03:14.518473+0000","last_undegraded":"2023-03-03T08:03:14.518473+0000","last_fullsized":"2023-03-03T08:03:14.518473+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:06:03.384487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518646+0000","last_change":"2023-03-03T08:03:13.541184+0000","last_active":"2023-03-03T08:03:14.518646+0000","last_peered":"2023-03-03T08:03:14.518646+0000","last_clean":"2023-03-03T08:03:14.518646+0000","last_became_active":"2023-03-03T08:03:13.540768+0000","last_became_peered":"2023-03-03T08:03:13.540768+0000","last_unstale":"2023-03-03T08:03:14.518646+0000","last_undegraded":"2023-03-03T08:03:14.518646+0000","last_fullsized":"2023-03-03T08:03:14.518646+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:34:21.624584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504541+0000","last_change":"2023-03-03T08:03:13.536126+0000","last_active":"2023-03-03T08:03:15.504541+0000","last_peered":"2023-03-03T08:03:15.504541+0000","last_clean":"2023-03-03T08:03:15.504541+0000","last_became_active":"2023-03-03T08:03:13.535846+0000","last_became_peered":"2023-03-03T08:03:13.535846+0000","last_unstale":"2023-03-03T08:03:15.504541+0000","last_undegraded":"2023-03-03T08:03:15.504541+0000","last_fullsized":"2023-03-03T08:03:15.504541+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:01:17.733840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523134+0000","last_change":"2023-03-03T08:03:13.537294+0000","last_active":"2023-03-03T08:03:14.523134+0000","last_peered":"2023-03-03T08:03:14.523134+0000","last_clean":"2023-03-03T08:03:14.523134+0000","last_became_active":"2023-03-03T08:03:13.537046+0000","last_became_peered":"2023-03-03T08:03:13.537046+0000","last_unstale":"2023-03-03T08:03:14.523134+0000","last_undegraded":"2023-03-03T08:03:14.523134+0000","last_fullsized":"2023-03-03T08:03:14.523134+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:58:37.748545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608860+0000","last_change":"2023-03-03T08:03:13.541951+0000","last_active":"2023-03-03T08:03:15.608860+0000","last_peered":"2023-03-03T08:03:15.608860+0000","last_clean":"2023-03-03T08:03:15.608860+0000","last_became_active":"2023-03-03T08:03:13.541648+0000","last_became_peered":"2023-03-03T08:03:13.541648+0000","last_unstale":"2023-03-03T08:03:15.608860+0000","last_undegraded":"2023-03-03T08:03:15.608860+0000","last_fullsized":"2023-03-03T08:03:15.608860+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:45:32.972695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608534+0000","last_change":"2023-03-03T08:03:13.544582+0000","last_active":"2023-03-03T08:03:15.608534+0000","last_peered":"2023-03-03T08:03:15.608534+0000","last_clean":"2023-03-03T08:03:15.608534+0000","last_became_active":"2023-03-03T08:03:13.544316+0000","last_became_peered":"2023-03-03T08:03:13.544316+0000","last_unstale":"2023-03-03T08:03:15.608534+0000","last_undegraded":"2023-03-03T08:03:15.608534+0000","last_fullsized":"2023-03-03T08:03:15.608534+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:22:36.589665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518017+0000","last_change":"2023-03-03T08:03:13.541072+0000","last_active":"2023-03-03T08:03:14.518017+0000","last_peered":"2023-03-03T08:03:14.518017+0000","last_clean":"2023-03-03T08:03:14.518017+0000","last_became_active":"2023-03-03T08:03:13.540790+0000","last_became_peered":"2023-03-03T08:03:13.540790+0000","last_unstale":"2023-03-03T08:03:14.518017+0000","last_undegraded":"2023-03-03T08:03:14.518017+0000","last_fullsized":"2023-03-03T08:03:14.518017+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:53.717879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504857+0000","last_change":"2023-03-03T08:03:13.531573+0000","last_active":"2023-03-03T08:03:15.504857+0000","last_peered":"2023-03-03T08:03:15.504857+0000","last_clean":"2023-03-03T08:03:15.504857+0000","last_became_active":"2023-03-03T08:03:13.527902+0000","last_became_peered":"2023-03-03T08:03:13.527902+0000","last_unstale":"2023-03-03T08:03:15.504857+0000","last_undegraded":"2023-03-03T08:03:15.504857+0000","last_fullsized":"2023-03-03T08:03:15.504857+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:44:39.592199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898648+0000","last_change":"2023-03-03T08:03:13.525841+0000","last_active":"2023-03-03T08:03:14.898648+0000","last_peered":"2023-03-03T08:03:14.898648+0000","last_clean":"2023-03-03T08:03:14.898648+0000","last_became_active":"2023-03-03T08:03:13.525451+0000","last_became_peered":"2023-03-03T08:03:13.525451+0000","last_unstale":"2023-03-03T08:03:14.898648+0000","last_undegraded":"2023-03-03T08:03:14.898648+0000","last_fullsized":"2023-03-03T08:03:14.898648+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:49:23.485411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523375+0000","last_change":"2023-03-03T08:03:13.536401+0000","last_active":"2023-03-03T08:03:14.523375+0000","last_peered":"2023-03-03T08:03:14.523375+0000","last_clean":"2023-03-03T08:03:14.523375+0000","last_became_active":"2023-03-03T08:03:13.536115+0000","last_became_peered":"2023-03-03T08:03:13.536115+0000","last_unstale":"2023-03-03T08:03:14.523375+0000","last_undegraded":"2023-03-03T08:03:14.523375+0000","last_fullsized":"2023-03-03T08:03:14.523375+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:34:21.624584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608999+0000","last_change":"2023-03-03T08:03:13.529974+0000","last_active":"2023-03-03T08:03:15.608999+0000","last_peered":"2023-03-03T08:03:15.608999+0000","last_clean":"2023-03-03T08:03:15.608999+0000","last_became_active":"2023-03-03T08:03:13.529630+0000","last_became_peered":"2023-03-03T08:03:13.529630+0000","last_unstale":"2023-03-03T08:03:15.608999+0000","last_undegraded":"2023-03-03T08:03:15.608999+0000","last_fullsized":"2023-03-03T08:03:15.608999+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:43:14.168233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898763+0000","last_change":"2023-03-03T08:03:13.535865+0000","last_active":"2023-03-03T08:03:14.898763+0000","last_peered":"2023-03-03T08:03:14.898763+0000","last_clean":"2023-03-03T08:03:14.898763+0000","last_became_active":"2023-03-03T08:03:13.535536+0000","last_became_peered":"2023-03-03T08:03:13.535536+0000","last_unstale":"2023-03-03T08:03:14.898763+0000","last_undegraded":"2023-03-03T08:03:14.898763+0000","last_fullsized":"2023-03-03T08:03:14.898763+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:59:02.639151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518296+0000","last_change":"2023-03-03T08:03:13.542647+0000","last_active":"2023-03-03T08:03:14.518296+0000","last_peered":"2023-03-03T08:03:14.518296+0000","last_clean":"2023-03-03T08:03:14.518296+0000","last_became_active":"2023-03-03T08:03:13.542151+0000","last_became_peered":"2023-03-03T08:03:13.542151+0000","last_unstale":"2023-03-03T08:03:14.518296+0000","last_undegraded":"2023-03-03T08:03:14.518296+0000","last_fullsized":"2023-03-03T08:03:14.518296+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:58:37.748545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.522827+0000","last_change":"2023-03-03T08:03:13.536251+0000","last_active":"2023-03-03T08:03:14.522827+0000","last_peered":"2023-03-03T08:03:14.522827+0000","last_clean":"2023-03-03T08:03:14.522827+0000","last_became_active":"2023-03-03T08:03:13.535962+0000","last_became_peered":"2023-03-03T08:03:13.535962+0000","last_unstale":"2023-03-03T08:03:14.522827+0000","last_undegraded":"2023-03-03T08:03:14.522827+0000","last_fullsized":"2023-03-03T08:03:14.522827+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:09.173678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608239+0000","last_change":"2023-03-03T08:03:13.529261+0000","last_active":"2023-03-03T08:03:15.608239+0000","last_peered":"2023-03-03T08:03:15.608239+0000","last_clean":"2023-03-03T08:03:15.608239+0000","last_became_active":"2023-03-03T08:03:13.528288+0000","last_became_peered":"2023-03-03T08:03:13.528288+0000","last_unstale":"2023-03-03T08:03:15.608239+0000","last_undegraded":"2023-03-03T08:03:15.608239+0000","last_fullsized":"2023-03-03T08:03:15.608239+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:12:29.525071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898680+0000","last_change":"2023-03-03T08:03:13.536371+0000","last_active":"2023-03-03T08:03:14.898680+0000","last_peered":"2023-03-03T08:03:14.898680+0000","last_clean":"2023-03-03T08:03:14.898680+0000","last_became_active":"2023-03-03T08:03:13.535954+0000","last_became_peered":"2023-03-03T08:03:13.535954+0000","last_unstale":"2023-03-03T08:03:14.898680+0000","last_undegraded":"2023-03-03T08:03:14.898680+0000","last_fullsized":"2023-03-03T08:03:14.898680+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:20:02.083571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518500+0000","last_change":"2023-03-03T08:03:13.537731+0000","last_active":"2023-03-03T08:03:14.518500+0000","last_peered":"2023-03-03T08:03:14.518500+0000","last_clean":"2023-03-03T08:03:14.518500+0000","last_became_active":"2023-03-03T08:03:13.537363+0000","last_became_peered":"2023-03-03T08:03:13.537363+0000","last_unstale":"2023-03-03T08:03:14.518500+0000","last_undegraded":"2023-03-03T08:03:14.518500+0000","last_fullsized":"2023-03-03T08:03:14.518500+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:00:50.026977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609070+0000","last_change":"2023-03-03T08:03:13.546570+0000","last_active":"2023-03-03T08:03:15.609070+0000","last_peered":"2023-03-03T08:03:15.609070+0000","last_clean":"2023-03-03T08:03:15.609070+0000","last_became_active":"2023-03-03T08:03:13.545993+0000","last_became_peered":"2023-03-03T08:03:13.545993+0000","last_unstale":"2023-03-03T08:03:15.609070+0000","last_undegraded":"2023-03-03T08:03:15.609070+0000","last_fullsized":"2023-03-03T08:03:15.609070+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:13:53.802081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608167+0000","last_change":"2023-03-03T08:03:13.524630+0000","last_active":"2023-03-03T08:03:15.608167+0000","last_peered":"2023-03-03T08:03:15.608167+0000","last_clean":"2023-03-03T08:03:15.608167+0000","last_became_active":"2023-03-03T08:03:13.524384+0000","last_became_peered":"2023-03-03T08:03:13.524384+0000","last_unstale":"2023-03-03T08:03:15.608167+0000","last_undegraded":"2023-03-03T08:03:15.608167+0000","last_fullsized":"2023-03-03T08:03:15.608167+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:26:10.759906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504755+0000","last_change":"2023-03-03T08:03:13.532766+0000","last_active":"2023-03-03T08:03:15.504755+0000","last_peered":"2023-03-03T08:03:15.504755+0000","last_clean":"2023-03-03T08:03:15.504755+0000","last_became_active":"2023-03-03T08:03:13.532440+0000","last_became_peered":"2023-03-03T08:03:13.532440+0000","last_unstale":"2023-03-03T08:03:15.504755+0000","last_undegraded":"2023-03-03T08:03:15.504755+0000","last_fullsized":"2023-03-03T08:03:15.504755+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:53.717879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523174+0000","last_change":"2023-03-03T08:03:13.542532+0000","last_active":"2023-03-03T08:03:14.523174+0000","last_peered":"2023-03-03T08:03:14.523174+0000","last_clean":"2023-03-03T08:03:14.523174+0000","last_became_active":"2023-03-03T08:03:13.542252+0000","last_became_peered":"2023-03-03T08:03:13.542252+0000","last_unstale":"2023-03-03T08:03:14.523174+0000","last_undegraded":"2023-03-03T08:03:14.523174+0000","last_fullsized":"2023-03-03T08:03:14.523174+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:01:17.733840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504405+0000","last_change":"2023-03-03T08:03:13.535248+0000","last_active":"2023-03-03T08:03:15.504405+0000","last_peered":"2023-03-03T08:03:15.504405+0000","last_clean":"2023-03-03T08:03:15.504405+0000","last_became_active":"2023-03-03T08:03:13.534988+0000","last_became_peered":"2023-03-03T08:03:13.534988+0000","last_unstale":"2023-03-03T08:03:15.504405+0000","last_undegraded":"2023-03-03T08:03:15.504405+0000","last_fullsized":"2023-03-03T08:03:15.504405+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:16:44.241182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608456+0000","last_change":"2023-03-03T08:03:13.540902+0000","last_active":"2023-03-03T08:03:15.608456+0000","last_peered":"2023-03-03T08:03:15.608456+0000","last_clean":"2023-03-03T08:03:15.608456+0000","last_became_active":"2023-03-03T08:03:13.540200+0000","last_became_peered":"2023-03-03T08:03:13.540200+0000","last_unstale":"2023-03-03T08:03:15.608456+0000","last_undegraded":"2023-03-03T08:03:15.608456+0000","last_fullsized":"2023-03-03T08:03:15.608456+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:00:50.026977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609556+0000","last_change":"2023-03-03T08:03:13.536855+0000","last_active":"2023-03-03T08:03:15.609556+0000","last_peered":"2023-03-03T08:03:15.609556+0000","last_clean":"2023-03-03T08:03:15.609556+0000","last_became_active":"2023-03-03T08:03:13.536559+0000","last_became_peered":"2023-03-03T08:03:13.536559+0000","last_unstale":"2023-03-03T08:03:15.609556+0000","last_undegraded":"2023-03-03T08:03:15.609556+0000","last_fullsized":"2023-03-03T08:03:15.609556+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:53.717879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608418+0000","last_change":"2023-03-03T08:03:13.525143+0000","last_active":"2023-03-03T08:03:15.608418+0000","last_peered":"2023-03-03T08:03:15.608418+0000","last_clean":"2023-03-03T08:03:15.608418+0000","last_became_active":"2023-03-03T08:03:13.524843+0000","last_became_peered":"2023-03-03T08:03:13.524843+0000","last_unstale":"2023-03-03T08:03:15.608418+0000","last_undegraded":"2023-03-03T08:03:15.608418+0000","last_fullsized":"2023-03-03T08:03:15.608418+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:30.932557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609003+0000","last_change":"2023-03-03T08:03:13.546267+0000","last_active":"2023-03-03T08:03:15.609003+0000","last_peered":"2023-03-03T08:03:15.609003+0000","last_clean":"2023-03-03T08:03:15.609003+0000","last_became_active":"2023-03-03T08:03:13.545593+0000","last_became_peered":"2023-03-03T08:03:13.545593+0000","last_unstale":"2023-03-03T08:03:15.609003+0000","last_undegraded":"2023-03-03T08:03:15.609003+0000","last_fullsized":"2023-03-03T08:03:15.609003+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:05:16.636308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608899+0000","last_change":"2023-03-03T08:03:13.536872+0000","last_active":"2023-03-03T08:03:15.608899+0000","last_peered":"2023-03-03T08:03:15.608899+0000","last_clean":"2023-03-03T08:03:15.608899+0000","last_became_active":"2023-03-03T08:03:13.536515+0000","last_became_peered":"2023-03-03T08:03:13.536515+0000","last_unstale":"2023-03-03T08:03:15.608899+0000","last_undegraded":"2023-03-03T08:03:15.608899+0000","last_fullsized":"2023-03-03T08:03:15.608899+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:04:11.932849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608270+0000","last_change":"2023-03-03T08:03:13.524788+0000","last_active":"2023-03-03T08:03:15.608270+0000","last_peered":"2023-03-03T08:03:15.608270+0000","last_clean":"2023-03-03T08:03:15.608270+0000","last_became_active":"2023-03-03T08:03:13.524050+0000","last_became_peered":"2023-03-03T08:03:13.524050+0000","last_unstale":"2023-03-03T08:03:15.608270+0000","last_undegraded":"2023-03-03T08:03:15.608270+0000","last_fullsized":"2023-03-03T08:03:15.608270+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:58.868299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518337+0000","last_change":"2023-03-03T08:03:13.542700+0000","last_active":"2023-03-03T08:03:14.518337+0000","last_peered":"2023-03-03T08:03:14.518337+0000","last_clean":"2023-03-03T08:03:14.518337+0000","last_became_active":"2023-03-03T08:03:13.542466+0000","last_became_peered":"2023-03-03T08:03:13.542466+0000","last_unstale":"2023-03-03T08:03:14.518337+0000","last_undegraded":"2023-03-03T08:03:14.518337+0000","last_fullsized":"2023-03-03T08:03:14.518337+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:01:17.733840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608633+0000","last_change":"2023-03-03T08:03:13.528885+0000","last_active":"2023-03-03T08:03:15.608633+0000","last_peered":"2023-03-03T08:03:15.608633+0000","last_clean":"2023-03-03T08:03:15.608633+0000","last_became_active":"2023-03-03T08:03:13.528550+0000","last_became_peered":"2023-03-03T08:03:13.528550+0000","last_unstale":"2023-03-03T08:03:15.608633+0000","last_undegraded":"2023-03-03T08:03:15.608633+0000","last_fullsized":"2023-03-03T08:03:15.608633+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:17:00.514431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608492+0000","last_change":"2023-03-03T08:03:13.540714+0000","last_active":"2023-03-03T08:03:15.608492+0000","last_peered":"2023-03-03T08:03:15.608492+0000","last_clean":"2023-03-03T08:03:15.608492+0000","last_became_active":"2023-03-03T08:03:13.539742+0000","last_became_peered":"2023-03-03T08:03:13.539742+0000","last_unstale":"2023-03-03T08:03:15.608492+0000","last_undegraded":"2023-03-03T08:03:15.608492+0000","last_fullsized":"2023-03-03T08:03:15.608492+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:41:08.450685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898700+0000","last_change":"2023-03-03T08:03:13.526045+0000","last_active":"2023-03-03T08:03:14.898700+0000","last_peered":"2023-03-03T08:03:14.898700+0000","last_clean":"2023-03-03T08:03:14.898700+0000","last_became_active":"2023-03-03T08:03:13.525794+0000","last_became_peered":"2023-03-03T08:03:13.525794+0000","last_unstale":"2023-03-03T08:03:14.898700+0000","last_undegraded":"2023-03-03T08:03:14.898700+0000","last_fullsized":"2023-03-03T08:03:14.898700+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:52.761809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"110'4","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608918+0000","last_change":"2023-03-03T08:03:12.972426+0000","last_active":"2023-03-03T08:03:15.608918+0000","last_peered":"2023-03-03T08:03:15.608918+0000","last_clean":"2023-03-03T08:03:15.608918+0000","last_became_active":"2023-03-03T08:03:12.972074+0000","last_became_peered":"2023-03-03T08:03:12.972074+0000","last_unstale":"2023-03-03T08:03:15.608918+0000","last_undegraded":"2023-03-03T08:03:15.608918+0000","last_fullsized":"2023-03-03T08:03:15.608918+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:22:52.548029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608577+0000","last_change":"2023-03-03T08:03:13.526231+0000","last_active":"2023-03-03T08:03:15.608577+0000","last_peered":"2023-03-03T08:03:15.608577+0000","last_clean":"2023-03-03T08:03:15.608577+0000","last_became_active":"2023-03-03T08:03:13.525969+0000","last_became_peered":"2023-03-03T08:03:13.525969+0000","last_unstale":"2023-03-03T08:03:15.608577+0000","last_undegraded":"2023-03-03T08:03:15.608577+0000","last_fullsized":"2023-03-03T08:03:15.608577+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:42:50.562879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898775+0000","last_change":"2023-03-03T08:03:12.973103+0000","last_active":"2023-03-03T08:03:14.898775+0000","last_peered":"2023-03-03T08:03:14.898775+0000","last_clean":"2023-03-03T08:03:14.898775+0000","last_became_active":"2023-03-03T08:03:12.972819+0000","last_became_peered":"2023-03-03T08:03:12.972819+0000","last_unstale":"2023-03-03T08:03:14.898775+0000","last_undegraded":"2023-03-03T08:03:14.898775+0000","last_fullsized":"2023-03-03T08:03:14.898775+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:59:52.623541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609229+0000","last_change":"2023-03-03T08:03:13.532277+0000","last_active":"2023-03-03T08:03:15.609229+0000","last_peered":"2023-03-03T08:03:15.609229+0000","last_clean":"2023-03-03T08:03:15.609229+0000","last_became_active":"2023-03-03T08:03:13.531948+0000","last_became_peered":"2023-03-03T08:03:13.531948+0000","last_unstale":"2023-03-03T08:03:15.609229+0000","last_undegraded":"2023-03-03T08:03:15.609229+0000","last_fullsized":"2023-03-03T08:03:15.609229+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:50:59.265078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504599+0000","last_change":"2023-03-03T08:03:12.970249+0000","last_active":"2023-03-03T08:03:15.504599+0000","last_peered":"2023-03-03T08:03:15.504599+0000","last_clean":"2023-03-03T08:03:15.504599+0000","last_became_active":"2023-03-03T08:03:12.969836+0000","last_became_peered":"2023-03-03T08:03:12.969836+0000","last_unstale":"2023-03-03T08:03:15.504599+0000","last_undegraded":"2023-03-03T08:03:15.504599+0000","last_fullsized":"2023-03-03T08:03:15.504599+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:11.950736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504881+0000","last_change":"2023-03-03T08:03:13.532840+0000","last_active":"2023-03-03T08:03:15.504881+0000","last_peered":"2023-03-03T08:03:15.504881+0000","last_clean":"2023-03-03T08:03:15.504881+0000","last_became_active":"2023-03-03T08:03:13.531307+0000","last_became_peered":"2023-03-03T08:03:13.531307+0000","last_unstale":"2023-03-03T08:03:15.504881+0000","last_undegraded":"2023-03-03T08:03:15.504881+0000","last_fullsized":"2023-03-03T08:03:15.504881+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:10:15.395992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898809+0000","last_change":"2023-03-03T08:03:12.974193+0000","last_active":"2023-03-03T08:03:14.898809+0000","last_peered":"2023-03-03T08:03:14.898809+0000","last_clean":"2023-03-03T08:03:14.898809+0000","last_became_active":"2023-03-03T08:03:12.973951+0000","last_became_peered":"2023-03-03T08:03:12.973951+0000","last_unstale":"2023-03-03T08:03:14.898809+0000","last_undegraded":"2023-03-03T08:03:14.898809+0000","last_fullsized":"2023-03-03T08:03:14.898809+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:48:22.334467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898396+0000","last_change":"2023-03-03T08:03:13.538795+0000","last_active":"2023-03-03T08:03:14.898396+0000","last_peered":"2023-03-03T08:03:14.898396+0000","last_clean":"2023-03-03T08:03:14.898396+0000","last_became_active":"2023-03-03T08:03:13.538554+0000","last_became_peered":"2023-03-03T08:03:13.538554+0000","last_unstale":"2023-03-03T08:03:14.898396+0000","last_undegraded":"2023-03-03T08:03:14.898396+0000","last_fullsized":"2023-03-03T08:03:14.898396+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:15:57.749701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523272+0000","last_change":"2023-03-03T08:03:12.976886+0000","last_active":"2023-03-03T08:03:14.523272+0000","last_peered":"2023-03-03T08:03:14.523272+0000","last_clean":"2023-03-03T08:03:14.523272+0000","last_became_active":"2023-03-03T08:03:12.976580+0000","last_became_peered":"2023-03-03T08:03:12.976580+0000","last_unstale":"2023-03-03T08:03:14.523272+0000","last_undegraded":"2023-03-03T08:03:14.523272+0000","last_fullsized":"2023-03-03T08:03:14.523272+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:17:37.014803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.522826+0000","last_change":"2023-03-03T08:03:13.538527+0000","last_active":"2023-03-03T08:03:14.522826+0000","last_peered":"2023-03-03T08:03:14.522826+0000","last_clean":"2023-03-03T08:03:14.522826+0000","last_became_active":"2023-03-03T08:03:13.537453+0000","last_became_peered":"2023-03-03T08:03:13.537453+0000","last_unstale":"2023-03-03T08:03:14.522826+0000","last_undegraded":"2023-03-03T08:03:14.522826+0000","last_fullsized":"2023-03-03T08:03:14.522826+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:16:44.241182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608957+0000","last_change":"2023-03-03T08:03:12.976434+0000","last_active":"2023-03-03T08:03:15.608957+0000","last_peered":"2023-03-03T08:03:15.608957+0000","last_clean":"2023-03-03T08:03:15.608957+0000","last_became_active":"2023-03-03T08:03:12.976090+0000","last_became_peered":"2023-03-03T08:03:12.976090+0000","last_unstale":"2023-03-03T08:03:15.608957+0000","last_undegraded":"2023-03-03T08:03:15.608957+0000","last_fullsized":"2023-03-03T08:03:15.608957+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:50:09.187662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518416+0000","last_change":"2023-03-03T08:03:13.532246+0000","last_active":"2023-03-03T08:03:14.518416+0000","last_peered":"2023-03-03T08:03:14.518416+0000","last_clean":"2023-03-03T08:03:14.518416+0000","last_became_active":"2023-03-03T08:03:13.531955+0000","last_became_peered":"2023-03-03T08:03:13.531955+0000","last_unstale":"2023-03-03T08:03:14.518416+0000","last_undegraded":"2023-03-03T08:03:14.518416+0000","last_fullsized":"2023-03-03T08:03:14.518416+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:50:59.265078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609174+0000","last_change":"2023-03-03T08:03:12.975571+0000","last_active":"2023-03-03T08:03:15.609174+0000","last_peered":"2023-03-03T08:03:15.609174+0000","last_clean":"2023-03-03T08:03:15.609174+0000","last_became_active":"2023-03-03T08:03:12.975216+0000","last_became_peered":"2023-03-03T08:03:12.975216+0000","last_unstale":"2023-03-03T08:03:15.609174+0000","last_undegraded":"2023-03-03T08:03:15.609174+0000","last_fullsized":"2023-03-03T08:03:15.609174+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:32:08.429902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898609+0000","last_change":"2023-03-03T08:03:13.526953+0000","last_active":"2023-03-03T08:03:14.898609+0000","last_peered":"2023-03-03T08:03:14.898609+0000","last_clean":"2023-03-03T08:03:14.898609+0000","last_became_active":"2023-03-03T08:03:13.526445+0000","last_became_peered":"2023-03-03T08:03:13.526445+0000","last_unstale":"2023-03-03T08:03:14.898609+0000","last_undegraded":"2023-03-03T08:03:14.898609+0000","last_fullsized":"2023-03-03T08:03:14.898609+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:32:09.433165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608423+0000","last_change":"2023-03-03T08:03:12.971410+0000","last_active":"2023-03-03T08:03:15.608423+0000","last_peered":"2023-03-03T08:03:15.608423+0000","last_clean":"2023-03-03T08:03:15.608423+0000","last_became_active":"2023-03-03T08:03:12.971161+0000","last_became_peered":"2023-03-03T08:03:12.971161+0000","last_unstale":"2023-03-03T08:03:15.608423+0000","last_undegraded":"2023-03-03T08:03:15.608423+0000","last_fullsized":"2023-03-03T08:03:15.608423+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:42:52.028315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609429+0000","last_change":"2023-03-03T08:03:13.528209+0000","last_active":"2023-03-03T08:03:15.609429+0000","last_peered":"2023-03-03T08:03:15.609429+0000","last_clean":"2023-03-03T08:03:15.609429+0000","last_became_active":"2023-03-03T08:03:13.527936+0000","last_became_peered":"2023-03-03T08:03:13.527936+0000","last_unstale":"2023-03-03T08:03:15.609429+0000","last_undegraded":"2023-03-03T08:03:15.609429+0000","last_fullsized":"2023-03-03T08:03:15.609429+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:06:03.384487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898837+0000","last_change":"2023-03-03T08:03:12.971968+0000","last_active":"2023-03-03T08:03:14.898837+0000","last_peered":"2023-03-03T08:03:14.898837+0000","last_clean":"2023-03-03T08:03:14.898837+0000","last_became_active":"2023-03-03T08:03:12.971706+0000","last_became_peered":"2023-03-03T08:03:12.971706+0000","last_unstale":"2023-03-03T08:03:14.898837+0000","last_undegraded":"2023-03-03T08:03:14.898837+0000","last_fullsized":"2023-03-03T08:03:14.898837+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:29.821562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523306+0000","last_change":"2023-03-03T08:03:13.545174+0000","last_active":"2023-03-03T08:03:14.523306+0000","last_peered":"2023-03-03T08:03:14.523306+0000","last_clean":"2023-03-03T08:03:14.523306+0000","last_became_active":"2023-03-03T08:03:13.544886+0000","last_became_peered":"2023-03-03T08:03:13.544886+0000","last_unstale":"2023-03-03T08:03:14.523306+0000","last_undegraded":"2023-03-03T08:03:14.523306+0000","last_fullsized":"2023-03-03T08:03:14.523306+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:10:15.395992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608499+0000","last_change":"2023-03-03T08:03:12.967798+0000","last_active":"2023-03-03T08:03:15.608499+0000","last_peered":"2023-03-03T08:03:15.608499+0000","last_clean":"2023-03-03T08:03:15.608499+0000","last_became_active":"2023-03-03T08:03:12.966918+0000","last_became_peered":"2023-03-03T08:03:12.966918+0000","last_unstale":"2023-03-03T08:03:15.608499+0000","last_undegraded":"2023-03-03T08:03:15.608499+0000","last_fullsized":"2023-03-03T08:03:15.608499+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:45.293155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608829+0000","last_change":"2023-03-03T08:03:13.539258+0000","last_active":"2023-03-03T08:03:15.608829+0000","last_peered":"2023-03-03T08:03:15.608829+0000","last_clean":"2023-03-03T08:03:15.608829+0000","last_became_active":"2023-03-03T08:03:13.538957+0000","last_became_peered":"2023-03-03T08:03:13.538957+0000","last_unstale":"2023-03-03T08:03:15.608829+0000","last_undegraded":"2023-03-03T08:03:15.608829+0000","last_fullsized":"2023-03-03T08:03:15.608829+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:35:57.846808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504839+0000","last_change":"2023-03-03T08:03:12.975961+0000","last_active":"2023-03-03T08:03:15.504839+0000","last_peered":"2023-03-03T08:03:15.504839+0000","last_clean":"2023-03-03T08:03:15.504839+0000","last_became_active":"2023-03-03T08:03:12.975608+0000","last_became_peered":"2023-03-03T08:03:12.975608+0000","last_unstale":"2023-03-03T08:03:15.504839+0000","last_undegraded":"2023-03-03T08:03:15.504839+0000","last_fullsized":"2023-03-03T08:03:15.504839+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:35:52.551259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609674+0000","last_change":"2023-03-03T08:03:13.543808+0000","last_active":"2023-03-03T08:03:15.609674+0000","last_peered":"2023-03-03T08:03:15.609674+0000","last_clean":"2023-03-03T08:03:15.609674+0000","last_became_active":"2023-03-03T08:03:13.543443+0000","last_became_peered":"2023-03-03T08:03:13.543443+0000","last_unstale":"2023-03-03T08:03:15.609674+0000","last_undegraded":"2023-03-03T08:03:15.609674+0000","last_fullsized":"2023-03-03T08:03:15.609674+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:44:27.769287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504498+0000","last_change":"2023-03-03T08:03:12.971813+0000","last_active":"2023-03-03T08:03:15.504498+0000","last_peered":"2023-03-03T08:03:15.504498+0000","last_clean":"2023-03-03T08:03:15.504498+0000","last_became_active":"2023-03-03T08:03:12.971395+0000","last_became_peered":"2023-03-03T08:03:12.971395+0000","last_unstale":"2023-03-03T08:03:15.504498+0000","last_undegraded":"2023-03-03T08:03:15.504498+0000","last_fullsized":"2023-03-03T08:03:15.504498+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:20:01.080308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608341+0000","last_change":"2023-03-03T08:03:13.537183+0000","last_active":"2023-03-03T08:03:15.608341+0000","last_peered":"2023-03-03T08:03:15.608341+0000","last_clean":"2023-03-03T08:03:15.608341+0000","last_became_active":"2023-03-03T08:03:13.536912+0000","last_became_peered":"2023-03-03T08:03:13.536912+0000","last_unstale":"2023-03-03T08:03:15.608341+0000","last_undegraded":"2023-03-03T08:03:15.608341+0000","last_fullsized":"2023-03-03T08:03:15.608341+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:56:50.934181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608566+0000","last_change":"2023-03-03T08:03:12.972074+0000","last_active":"2023-03-03T08:03:15.608566+0000","last_peered":"2023-03-03T08:03:15.608566+0000","last_clean":"2023-03-03T08:03:15.608566+0000","last_became_active":"2023-03-03T08:03:12.971734+0000","last_became_peered":"2023-03-03T08:03:12.971734+0000","last_unstale":"2023-03-03T08:03:15.608566+0000","last_undegraded":"2023-03-03T08:03:15.608566+0000","last_fullsized":"2023-03-03T08:03:15.608566+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:57:17.100976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518089+0000","last_change":"2023-03-03T08:03:13.540445+0000","last_active":"2023-03-03T08:03:14.518089+0000","last_peered":"2023-03-03T08:03:14.518089+0000","last_clean":"2023-03-03T08:03:14.518089+0000","last_became_active":"2023-03-03T08:03:13.540166+0000","last_became_peered":"2023-03-03T08:03:13.540166+0000","last_unstale":"2023-03-03T08:03:14.518089+0000","last_undegraded":"2023-03-03T08:03:14.518089+0000","last_fullsized":"2023-03-03T08:03:14.518089+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:09.173678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504313+0000","last_change":"2023-03-03T08:03:12.973378+0000","last_active":"2023-03-03T08:03:15.504313+0000","last_peered":"2023-03-03T08:03:15.504313+0000","last_clean":"2023-03-03T08:03:15.504313+0000","last_became_active":"2023-03-03T08:03:12.972932+0000","last_became_peered":"2023-03-03T08:03:12.972932+0000","last_unstale":"2023-03-03T08:03:15.504313+0000","last_undegraded":"2023-03-03T08:03:15.504313+0000","last_fullsized":"2023-03-03T08:03:15.504313+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:51.758546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518449+0000","last_change":"2023-03-03T08:03:13.542448+0000","last_active":"2023-03-03T08:03:14.518449+0000","last_peered":"2023-03-03T08:03:14.518449+0000","last_clean":"2023-03-03T08:03:14.518449+0000","last_became_active":"2023-03-03T08:03:13.541743+0000","last_became_peered":"2023-03-03T08:03:13.541743+0000","last_unstale":"2023-03-03T08:03:14.518449+0000","last_undegraded":"2023-03-03T08:03:14.518449+0000","last_fullsized":"2023-03-03T08:03:14.518449+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:10:15.395992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518280+0000","last_change":"2023-03-03T08:03:12.968898+0000","last_active":"2023-03-03T08:03:14.518280+0000","last_peered":"2023-03-03T08:03:14.518280+0000","last_clean":"2023-03-03T08:03:14.518280+0000","last_became_active":"2023-03-03T08:03:12.968439+0000","last_became_peered":"2023-03-03T08:03:12.968439+0000","last_unstale":"2023-03-03T08:03:14.518280+0000","last_undegraded":"2023-03-03T08:03:14.518280+0000","last_fullsized":"2023-03-03T08:03:14.518280+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:49:22.482147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609353+0000","last_change":"2023-03-03T08:03:13.529057+0000","last_active":"2023-03-03T08:03:15.609353+0000","last_peered":"2023-03-03T08:03:15.609353+0000","last_clean":"2023-03-03T08:03:15.609353+0000","last_became_active":"2023-03-03T08:03:13.528577+0000","last_became_peered":"2023-03-03T08:03:13.528577+0000","last_unstale":"2023-03-03T08:03:15.609353+0000","last_undegraded":"2023-03-03T08:03:15.609353+0000","last_fullsized":"2023-03-03T08:03:15.609353+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:17:38.018066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504230+0000","last_change":"2023-03-03T08:03:12.973948+0000","last_active":"2023-03-03T08:03:15.504230+0000","last_peered":"2023-03-03T08:03:15.504230+0000","last_clean":"2023-03-03T08:03:15.504230+0000","last_became_active":"2023-03-03T08:03:12.973579+0000","last_became_peered":"2023-03-03T08:03:12.973579+0000","last_unstale":"2023-03-03T08:03:15.504230+0000","last_undegraded":"2023-03-03T08:03:15.504230+0000","last_fullsized":"2023-03-03T08:03:15.504230+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:15:56.746437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608915+0000","last_change":"2023-03-03T08:03:13.527986+0000","last_active":"2023-03-03T08:03:15.608915+0000","last_peered":"2023-03-03T08:03:15.608915+0000","last_clean":"2023-03-03T08:03:15.608915+0000","last_became_active":"2023-03-03T08:03:13.527647+0000","last_became_peered":"2023-03-03T08:03:13.527647+0000","last_unstale":"2023-03-03T08:03:15.608915+0000","last_undegraded":"2023-03-03T08:03:15.608915+0000","last_fullsized":"2023-03-03T08:03:15.608915+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:27:11.786506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609109+0000","last_change":"2023-03-03T08:03:12.972891+0000","last_active":"2023-03-03T08:03:15.609109+0000","last_peered":"2023-03-03T08:03:15.609109+0000","last_clean":"2023-03-03T08:03:15.609109+0000","last_became_active":"2023-03-03T08:03:12.972367+0000","last_became_peered":"2023-03-03T08:03:12.972367+0000","last_unstale":"2023-03-03T08:03:15.609109+0000","last_undegraded":"2023-03-03T08:03:15.609109+0000","last_fullsized":"2023-03-03T08:03:15.609109+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:34.969668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898647+0000","last_change":"2023-03-03T08:03:13.538318+0000","last_active":"2023-03-03T08:03:14.898647+0000","last_peered":"2023-03-03T08:03:14.898647+0000","last_clean":"2023-03-03T08:03:14.898647+0000","last_became_active":"2023-03-03T08:03:13.538088+0000","last_became_peered":"2023-03-03T08:03:13.538088+0000","last_unstale":"2023-03-03T08:03:14.898647+0000","last_undegraded":"2023-03-03T08:03:14.898647+0000","last_fullsized":"2023-03-03T08:03:14.898647+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:12.954000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608936+0000","last_change":"2023-03-03T08:03:12.971078+0000","last_active":"2023-03-03T08:03:15.608936+0000","last_peered":"2023-03-03T08:03:15.608936+0000","last_clean":"2023-03-03T08:03:15.608936+0000","last_became_active":"2023-03-03T08:03:12.970784+0000","last_became_peered":"2023-03-03T08:03:12.970784+0000","last_unstale":"2023-03-03T08:03:15.608936+0000","last_undegraded":"2023-03-03T08:03:15.608936+0000","last_fullsized":"2023-03-03T08:03:15.608936+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:07:16.503347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504788+0000","last_change":"2023-03-03T08:03:13.534724+0000","last_active":"2023-03-03T08:03:15.504788+0000","last_peered":"2023-03-03T08:03:15.504788+0000","last_clean":"2023-03-03T08:03:15.504788+0000","last_became_active":"2023-03-03T08:03:13.534419+0000","last_became_peered":"2023-03-03T08:03:13.534419+0000","last_unstale":"2023-03-03T08:03:15.504788+0000","last_undegraded":"2023-03-03T08:03:15.504788+0000","last_fullsized":"2023-03-03T08:03:15.504788+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:34:21.624584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609104+0000","last_change":"2023-03-03T08:03:12.968475+0000","last_active":"2023-03-03T08:03:15.609104+0000","last_peered":"2023-03-03T08:03:15.609104+0000","last_clean":"2023-03-03T08:03:15.609104+0000","last_became_active":"2023-03-03T08:03:12.968141+0000","last_became_peered":"2023-03-03T08:03:12.968141+0000","last_unstale":"2023-03-03T08:03:15.609104+0000","last_undegraded":"2023-03-03T08:03:15.609104+0000","last_fullsized":"2023-03-03T08:03:15.609104+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:03:49.445758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608998+0000","last_change":"2023-03-03T08:03:13.529103+0000","last_active":"2023-03-03T08:03:15.608998+0000","last_peered":"2023-03-03T08:03:15.608998+0000","last_clean":"2023-03-03T08:03:15.608998+0000","last_became_active":"2023-03-03T08:03:13.528804+0000","last_became_peered":"2023-03-03T08:03:13.528804+0000","last_unstale":"2023-03-03T08:03:15.608998+0000","last_undegraded":"2023-03-03T08:03:15.608998+0000","last_fullsized":"2023-03-03T08:03:15.608998+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:26:45.045000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609004+0000","last_change":"2023-03-03T08:03:12.968634+0000","last_active":"2023-03-03T08:03:15.609004+0000","last_peered":"2023-03-03T08:03:15.609004+0000","last_clean":"2023-03-03T08:03:15.609004+0000","last_became_active":"2023-03-03T08:03:12.967878+0000","last_became_peered":"2023-03-03T08:03:12.967878+0000","last_unstale":"2023-03-03T08:03:15.609004+0000","last_undegraded":"2023-03-03T08:03:15.609004+0000","last_fullsized":"2023-03-03T08:03:15.609004+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:32:21.108432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608520+0000","last_change":"2023-03-03T08:03:13.529496+0000","last_active":"2023-03-03T08:03:15.608520+0000","last_peered":"2023-03-03T08:03:15.608520+0000","last_clean":"2023-03-03T08:03:15.608520+0000","last_became_active":"2023-03-03T08:03:13.529086+0000","last_became_peered":"2023-03-03T08:03:13.529086+0000","last_unstale":"2023-03-03T08:03:15.608520+0000","last_undegraded":"2023-03-03T08:03:15.608520+0000","last_fullsized":"2023-03-03T08:03:15.608520+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:12:50.605693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523256+0000","last_change":"2023-03-03T08:03:12.973745+0000","last_active":"2023-03-03T08:03:14.523256+0000","last_peered":"2023-03-03T08:03:14.523256+0000","last_clean":"2023-03-03T08:03:14.523256+0000","last_became_active":"2023-03-03T08:03:12.973471+0000","last_became_peered":"2023-03-03T08:03:12.973471+0000","last_unstale":"2023-03-03T08:03:14.523256+0000","last_undegraded":"2023-03-03T08:03:14.523256+0000","last_fullsized":"2023-03-03T08:03:14.523256+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:39:17.607622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518229+0000","last_change":"2023-03-03T08:03:13.540474+0000","last_active":"2023-03-03T08:03:14.518229+0000","last_peered":"2023-03-03T08:03:14.518229+0000","last_clean":"2023-03-03T08:03:14.518229+0000","last_became_active":"2023-03-03T08:03:13.540230+0000","last_became_peered":"2023-03-03T08:03:13.540230+0000","last_unstale":"2023-03-03T08:03:14.518229+0000","last_undegraded":"2023-03-03T08:03:14.518229+0000","last_fullsized":"2023-03-03T08:03:14.518229+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:16:44.241182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523306+0000","last_change":"2023-03-03T08:03:12.977628+0000","last_active":"2023-03-03T08:03:14.523306+0000","last_peered":"2023-03-03T08:03:14.523306+0000","last_clean":"2023-03-03T08:03:14.523306+0000","last_became_active":"2023-03-03T08:03:12.976889+0000","last_became_peered":"2023-03-03T08:03:12.976889+0000","last_unstale":"2023-03-03T08:03:14.523306+0000","last_undegraded":"2023-03-03T08:03:14.523306+0000","last_fullsized":"2023-03-03T08:03:14.523306+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:44:26.766023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504326+0000","last_change":"2023-03-03T08:03:13.528119+0000","last_active":"2023-03-03T08:03:15.504326+0000","last_peered":"2023-03-03T08:03:15.504326+0000","last_clean":"2023-03-03T08:03:15.504326+0000","last_became_active":"2023-03-03T08:03:13.527830+0000","last_became_peered":"2023-03-03T08:03:13.527830+0000","last_unstale":"2023-03-03T08:03:15.504326+0000","last_undegraded":"2023-03-03T08:03:15.504326+0000","last_fullsized":"2023-03-03T08:03:15.504326+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:09.173678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608632+0000","last_change":"2023-03-03T08:03:12.970900+0000","last_active":"2023-03-03T08:03:15.608632+0000","last_peered":"2023-03-03T08:03:15.608632+0000","last_clean":"2023-03-03T08:03:15.608632+0000","last_became_active":"2023-03-03T08:03:12.970633+0000","last_became_peered":"2023-03-03T08:03:12.970633+0000","last_unstale":"2023-03-03T08:03:15.608632+0000","last_undegraded":"2023-03-03T08:03:15.608632+0000","last_fullsized":"2023-03-03T08:03:15.608632+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:56:15.442953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898833+0000","last_change":"2023-03-03T08:03:13.544190+0000","last_active":"2023-03-03T08:03:14.898833+0000","last_peered":"2023-03-03T08:03:14.898833+0000","last_clean":"2023-03-03T08:03:14.898833+0000","last_became_active":"2023-03-03T08:03:13.543942+0000","last_became_peered":"2023-03-03T08:03:13.543942+0000","last_unstale":"2023-03-03T08:03:14.898833+0000","last_undegraded":"2023-03-03T08:03:14.898833+0000","last_fullsized":"2023-03-03T08:03:14.898833+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:17:36.836414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504233+0000","last_change":"2023-03-03T08:03:12.971112+0000","last_active":"2023-03-03T08:03:15.504233+0000","last_peered":"2023-03-03T08:03:15.504233+0000","last_clean":"2023-03-03T08:03:15.504233+0000","last_became_active":"2023-03-03T08:03:12.970796+0000","last_became_peered":"2023-03-03T08:03:12.970796+0000","last_unstale":"2023-03-03T08:03:15.504233+0000","last_undegraded":"2023-03-03T08:03:15.504233+0000","last_fullsized":"2023-03-03T08:03:15.504233+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:35:13.046397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609598+0000","last_change":"2023-03-03T08:03:13.529245+0000","last_active":"2023-03-03T08:03:15.609598+0000","last_peered":"2023-03-03T08:03:15.609598+0000","last_clean":"2023-03-03T08:03:15.609598+0000","last_became_active":"2023-03-03T08:03:13.528965+0000","last_became_peered":"2023-03-03T08:03:13.528965+0000","last_unstale":"2023-03-03T08:03:15.609598+0000","last_undegraded":"2023-03-03T08:03:15.609598+0000","last_fullsized":"2023-03-03T08:03:15.609598+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:05:48.872734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"110'6","reported_seq":17,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518662+0000","last_change":"2023-03-03T08:03:12.971469+0000","last_active":"2023-03-03T08:03:14.518662+0000","last_peered":"2023-03-03T08:03:14.518662+0000","last_clean":"2023-03-03T08:03:14.518662+0000","last_became_active":"2023-03-03T08:03:12.971204+0000","last_became_peered":"2023-03-03T08:03:12.971204+0000","last_unstale":"2023-03-03T08:03:14.518662+0000","last_undegraded":"2023-03-03T08:03:14.518662+0000","last_fullsized":"2023-03-03T08:03:14.518662+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:32:08.429902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.529120+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T08:03:13.529120+0000","last_peered":"2023-03-03T08:03:13.529120+0000","last_clean":"2023-03-03T08:03:13.529120+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T08:03:13.529120+0000","last_undegraded":"2023-03-03T08:03:13.529120+0000","last_fullsized":"2023-03-03T08:03:13.529120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518573+0000","last_change":"2023-03-03T08:03:13.541671+0000","last_active":"2023-03-03T08:03:14.518573+0000","last_peered":"2023-03-03T08:03:14.518573+0000","last_clean":"2023-03-03T08:03:14.518573+0000","last_became_active":"2023-03-03T08:03:13.541383+0000","last_became_peered":"2023-03-03T08:03:13.541383+0000","last_unstale":"2023-03-03T08:03:14.518573+0000","last_undegraded":"2023-03-03T08:03:14.518573+0000","last_fullsized":"2023-03-03T08:03:14.518573+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:44:39.592199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"110'3","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609634+0000","last_change":"2023-03-03T08:03:12.975920+0000","last_active":"2023-03-03T08:03:15.609634+0000","last_peered":"2023-03-03T08:03:15.609634+0000","last_clean":"2023-03-03T08:03:15.609634+0000","last_became_active":"2023-03-03T08:03:12.975541+0000","last_became_peered":"2023-03-03T08:03:12.975541+0000","last_unstale":"2023-03-03T08:03:15.609634+0000","last_undegraded":"2023-03-03T08:03:15.609634+0000","last_fullsized":"2023-03-03T08:03:15.609634+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:48:02.934456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.515328+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T08:03:13.515328+0000","last_peered":"2023-03-03T08:03:13.515328+0000","last_clean":"2023-03-03T08:03:13.515328+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T08:03:13.515328+0000","last_undegraded":"2023-03-03T08:03:13.515328+0000","last_fullsized":"2023-03-03T08:03:13.515328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609538+0000","last_change":"2023-03-03T08:03:13.542375+0000","last_active":"2023-03-03T08:03:15.609538+0000","last_peered":"2023-03-03T08:03:15.609538+0000","last_clean":"2023-03-03T08:03:15.609538+0000","last_became_active":"2023-03-03T08:03:13.542095+0000","last_became_peered":"2023-03-03T08:03:13.542095+0000","last_unstale":"2023-03-03T08:03:15.609538+0000","last_undegraded":"2023-03-03T08:03:15.609538+0000","last_fullsized":"2023-03-03T08:03:15.609538+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:24:32.147518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609316+0000","last_change":"2023-03-03T08:03:12.975337+0000","last_active":"2023-03-03T08:03:15.609316+0000","last_peered":"2023-03-03T08:03:15.609316+0000","last_clean":"2023-03-03T08:03:15.609316+0000","last_became_active":"2023-03-03T08:03:12.975038+0000","last_became_peered":"2023-03-03T08:03:12.975038+0000","last_unstale":"2023-03-03T08:03:15.609316+0000","last_undegraded":"2023-03-03T08:03:15.609316+0000","last_fullsized":"2023-03-03T08:03:15.609316+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:31.956827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.515322+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T08:03:13.515322+0000","last_peered":"2023-03-03T08:03:13.515322+0000","last_clean":"2023-03-03T08:03:13.515322+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T08:03:13.515322+0000","last_undegraded":"2023-03-03T08:03:13.515322+0000","last_fullsized":"2023-03-03T08:03:13.515322+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608583+0000","last_change":"2023-03-03T08:03:13.546953+0000","last_active":"2023-03-03T08:03:15.608583+0000","last_peered":"2023-03-03T08:03:15.608583+0000","last_clean":"2023-03-03T08:03:15.608583+0000","last_became_active":"2023-03-03T08:03:13.546621+0000","last_became_peered":"2023-03-03T08:03:13.546621+0000","last_unstale":"2023-03-03T08:03:15.608583+0000","last_undegraded":"2023-03-03T08:03:15.608583+0000","last_fullsized":"2023-03-03T08:03:15.608583+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:28:07.879188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518442+0000","last_change":"2023-03-03T08:03:12.971334+0000","last_active":"2023-03-03T08:03:14.518442+0000","last_peered":"2023-03-03T08:03:14.518442+0000","last_clean":"2023-03-03T08:03:14.518442+0000","last_became_active":"2023-03-03T08:03:12.971083+0000","last_became_peered":"2023-03-03T08:03:12.971083+0000","last_unstale":"2023-03-03T08:03:14.518442+0000","last_undegraded":"2023-03-03T08:03:14.518442+0000","last_fullsized":"2023-03-03T08:03:14.518442+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:17:35.833151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.516906+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T08:03:13.516906+0000","last_peered":"2023-03-03T08:03:13.516906+0000","last_clean":"2023-03-03T08:03:13.516906+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T08:03:13.516906+0000","last_undegraded":"2023-03-03T08:03:13.516906+0000","last_fullsized":"2023-03-03T08:03:13.516906+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504823+0000","last_change":"2023-03-03T08:03:13.532910+0000","last_active":"2023-03-03T08:03:15.504823+0000","last_peered":"2023-03-03T08:03:15.504823+0000","last_clean":"2023-03-03T08:03:15.504823+0000","last_became_active":"2023-03-03T08:03:13.532587+0000","last_became_peered":"2023-03-03T08:03:13.532587+0000","last_unstale":"2023-03-03T08:03:15.504823+0000","last_undegraded":"2023-03-03T08:03:15.504823+0000","last_fullsized":"2023-03-03T08:03:15.504823+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:06:03.384487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"110'8","reported_seq":19,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518102+0000","last_change":"2023-03-03T08:03:12.972059+0000","last_active":"2023-03-03T08:03:14.518102+0000","last_peered":"2023-03-03T08:03:14.518102+0000","last_clean":"2023-03-03T08:03:14.518102+0000","last_became_active":"2023-03-03T08:03:12.971734+0000","last_became_peered":"2023-03-03T08:03:12.971734+0000","last_unstale":"2023-03-03T08:03:14.518102+0000","last_undegraded":"2023-03-03T08:03:14.518102+0000","last_fullsized":"2023-03-03T08:03:14.518102+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:29:54.191181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":284,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.517085+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T08:03:13.517085+0000","last_peered":"2023-03-03T08:03:13.517085+0000","last_clean":"2023-03-03T08:03:13.517085+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T08:03:13.517085+0000","last_undegraded":"2023-03-03T08:03:13.517085+0000","last_fullsized":"2023-03-03T08:03:13.517085+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":88,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.517798+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T08:03:13.517798+0000","last_peered":"2023-03-03T08:03:13.517798+0000","last_clean":"2023-03-03T08:03:13.517798+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T08:03:13.517798+0000","last_undegraded":"2023-03-03T08:03:13.517798+0000","last_fullsized":"2023-03-03T08:03:13.517798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898497+0000","last_change":"2023-03-03T08:03:13.543981+0000","last_active":"2023-03-03T08:03:14.898497+0000","last_peered":"2023-03-03T08:03:14.898497+0000","last_clean":"2023-03-03T08:03:14.898497+0000","last_became_active":"2023-03-03T08:03:13.543705+0000","last_became_peered":"2023-03-03T08:03:13.543705+0000","last_unstale":"2023-03-03T08:03:14.898497+0000","last_undegraded":"2023-03-03T08:03:14.898497+0000","last_fullsized":"2023-03-03T08:03:14.898497+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:29:55.194445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518507+0000","last_change":"2023-03-03T08:03:12.975860+0000","last_active":"2023-03-03T08:03:14.518507+0000","last_peered":"2023-03-03T08:03:14.518507+0000","last_clean":"2023-03-03T08:03:14.518507+0000","last_became_active":"2023-03-03T08:03:12.975589+0000","last_became_peered":"2023-03-03T08:03:12.975589+0000","last_unstale":"2023-03-03T08:03:14.518507+0000","last_undegraded":"2023-03-03T08:03:14.518507+0000","last_fullsized":"2023-03-03T08:03:14.518507+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:51.758546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":83,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.514684+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T08:03:13.514684+0000","last_peered":"2023-03-03T08:03:13.514684+0000","last_clean":"2023-03-03T08:03:13.514684+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T08:03:13.514684+0000","last_undegraded":"2023-03-03T08:03:13.514684+0000","last_fullsized":"2023-03-03T08:03:13.514684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608226+0000","last_change":"2023-03-03T08:03:13.534135+0000","last_active":"2023-03-03T08:03:15.608226+0000","last_peered":"2023-03-03T08:03:15.608226+0000","last_clean":"2023-03-03T08:03:15.608226+0000","last_became_active":"2023-03-03T08:03:13.533898+0000","last_became_peered":"2023-03-03T08:03:13.533898+0000","last_unstale":"2023-03-03T08:03:15.608226+0000","last_undegraded":"2023-03-03T08:03:15.608226+0000","last_fullsized":"2023-03-03T08:03:15.608226+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:10:09.472093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518195+0000","last_change":"2023-03-03T08:03:12.968683+0000","last_active":"2023-03-03T08:03:14.518195+0000","last_peered":"2023-03-03T08:03:14.518195+0000","last_clean":"2023-03-03T08:03:14.518195+0000","last_became_active":"2023-03-03T08:03:12.967877+0000","last_became_peered":"2023-03-03T08:03:12.967877+0000","last_unstale":"2023-03-03T08:03:14.518195+0000","last_undegraded":"2023-03-03T08:03:14.518195+0000","last_fullsized":"2023-03-03T08:03:14.518195+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:59:01.635888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":88,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.517732+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T08:03:13.517732+0000","last_peered":"2023-03-03T08:03:13.517732+0000","last_clean":"2023-03-03T08:03:13.517732+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T08:03:13.517732+0000","last_undegraded":"2023-03-03T08:03:13.517732+0000","last_fullsized":"2023-03-03T08:03:13.517732+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608733+0000","last_change":"2023-03-03T08:03:13.549078+0000","last_active":"2023-03-03T08:03:15.608733+0000","last_peered":"2023-03-03T08:03:15.608733+0000","last_clean":"2023-03-03T08:03:15.608733+0000","last_became_active":"2023-03-03T08:03:13.548802+0000","last_became_peered":"2023-03-03T08:03:13.548802+0000","last_unstale":"2023-03-03T08:03:15.608733+0000","last_undegraded":"2023-03-03T08:03:15.608733+0000","last_fullsized":"2023-03-03T08:03:15.608733+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:56:53.272507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608982+0000","last_change":"2023-03-03T08:03:12.971765+0000","last_active":"2023-03-03T08:03:15.608982+0000","last_peered":"2023-03-03T08:03:15.608982+0000","last_clean":"2023-03-03T08:03:15.608982+0000","last_became_active":"2023-03-03T08:03:12.971428+0000","last_became_peered":"2023-03-03T08:03:12.971428+0000","last_unstale":"2023-03-03T08:03:15.608982+0000","last_undegraded":"2023-03-03T08:03:15.608982+0000","last_fullsized":"2023-03-03T08:03:15.608982+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:52:05.550981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.514807+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T08:03:13.514807+0000","last_peered":"2023-03-03T08:03:13.514807+0000","last_clean":"2023-03-03T08:03:13.514807+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T08:03:13.514807+0000","last_undegraded":"2023-03-03T08:03:13.514807+0000","last_fullsized":"2023-03-03T08:03:13.514807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574944,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":16788,"kb_used_data":1932,"kb_used_omap":0,"kb_used_meta":14848,"kb_avail":94355052,"statfs":{"total":96636764160,"available":96619573248,"internally_reserved":0,"allocated":1978368,"data_stored":1225680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":15204352},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:02:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66600000000000004,"15min":0.69599999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.88700000000000001,"5min":0.91700000000000004,"15min":1.546},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.66800000000000004,"15min":0.71199999999999997},"min":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.92900000000000005,"5min":0.96499999999999997,"15min":1.702},"last":0.64200000000000002}]},{"osd":1,"last update":"Fri Mar 3 08:02:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.71999999999999997,"15min":0.99199999999999999},"min":{"1min":0.48899999999999999,"5min":0.434,"15min":0.41699999999999998},"max":{"1min":0.94699999999999995,"5min":1.111,"15min":6.6829999999999998},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68300000000000005,"15min":0.746},"min":{"1min":0.47199999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.87,"5min":1.1890000000000001,"15min":1.944},"last":0.89200000000000002}]},{"osd":2,"last update":"Fri Mar 3 08:02:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72199999999999998,"15min":0.71499999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.87,"5min":1.224,"15min":1.252},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.74099999999999999,"15min":0.77900000000000003},"min":{"1min":0.496,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0229999999999999,"5min":1.2490000000000001,"15min":1.329},"last":1.0129999999999999}]},{"osd":3,"last update":"Fri Mar 3 08:02:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91500000000000004,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.73099999999999998,"15min":0.77500000000000002},"min":{"1min":0.52500000000000002,"5min":0.51000000000000001,"15min":0.46000000000000002},"max":{"1min":0.97299999999999998,"5min":1.014,"15min":1.294},"last":1.04}]},{"osd":4,"last update":"Fri Mar 3 08:02:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60999999999999999,"15min":0.64600000000000002},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41099999999999998},"max":{"1min":0.93300000000000005,"5min":1.0049999999999999,"15min":1.077},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.68200000000000005,"15min":0.64400000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.40899999999999997},"max":{"1min":0.95899999999999996,"5min":1.081,"15min":1.081},"last":0.77400000000000002}]},{"osd":5,"last update":"Fri Mar 3 08:02:40 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.71499999999999997,"15min":0.77500000000000002},"min":{"1min":0.50700000000000001,"5min":0.40699999999999997,"15min":0.39100000000000001},"max":{"1min":1.0009999999999999,"5min":1.036,"15min":1.3819999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.77300000000000002,"15min":0.81000000000000005},"min":{"1min":0.57399999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0009999999999999,"5min":1.2190000000000001,"15min":1.349},"last":0.70299999999999996}]},{"osd":6,"last update":"Fri Mar 3 08:02:40 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.79300000000000004,"15min":0.82199999999999995},"min":{"1min":0.56200000000000006,"5min":0.56000000000000005,"15min":0.54100000000000004},"max":{"1min":1.081,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.999},{"interface":"front","average":{"1min":0.749,"5min":0.79500000000000004,"15min":0.81100000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0169999999999999,"5min":1.135,"15min":1.194},"last":0.91100000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574945,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11276,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":9920,"kb_avail":94360564,"statfs":{"total":96636764160,"available":96625217536,"internally_reserved":0,"allocated":1380352,"data_stored":640752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10158080},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73499999999999999,"15min":0.69999999999999996},"min":{"1min":0.42899999999999999,"5min":0.42299999999999999,"15min":0.40100000000000002},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.4550000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.878,"5min":0.751,"15min":0.71399999999999997},"min":{"1min":0.46200000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.2169999999999996,"5min":5.2169999999999996,"15min":5.2169999999999996},"last":0.67700000000000005}]},{"osd":1,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.71699999999999997,"15min":0.69699999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":3.323},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.746,"15min":0.73399999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.48199999999999998},"max":{"1min":2.105,"5min":2.105,"15min":2.105},"last":0.76500000000000001}]},{"osd":2,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.76300000000000001,"15min":0.74099999999999999},"min":{"1min":0.51000000000000001,"5min":0.5,"15min":0.5},"max":{"1min":3.956,"5min":3.956,"15min":3.956},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.54900000000000004,"5min":0.495,"15min":0.495},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.80200000000000005,"15min":0.80100000000000005},"min":{"1min":0.53500000000000003,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":4.2270000000000003,"5min":4.2270000000000003,"15min":4.8959999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.80700000000000005,"15min":0.78400000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":4.0049999999999999,"5min":4.0049999999999999,"15min":4.0049999999999999},"last":0.75700000000000001}]},{"osd":4,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.70699999999999996,"15min":0.86799999999999999},"min":{"1min":0.45000000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.1349999999999998,"5min":2.1349999999999998,"15min":4.0099999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.75,"15min":0.89000000000000001},"min":{"1min":0.504,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":5.3029999999999999,"5min":5.3029999999999999,"15min":5.3029999999999999},"last":0.623}]},{"osd":5,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.70799999999999996,"15min":0.76400000000000001},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.347,"5min":2.347,"15min":2.347},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.79100000000000004,"15min":0.81999999999999995},"min":{"1min":0.501,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.58999999999999997}]},{"osd":7,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.83299999999999996,"15min":0.84499999999999997},"min":{"1min":0.63,"5min":0.54100000000000004,"15min":0.51900000000000002},"max":{"1min":5.2629999999999999,"5min":5.2629999999999999,"15min":5.2629999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.78900000000000003,"15min":0.85299999999999998},"min":{"1min":0.50600000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.532,"5min":2.532,"15min":2.532},"last":0.70999999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11664,"kb_used_data":1352,"kb_used_omap":0,"kb_used_meta":10304,"kb_avail":94360176,"statfs":{"total":96636764160,"available":96624820224,"internally_reserved":0,"allocated":1384448,"data_stored":641227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10551296},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46899999999999997,"15min":0.498},"min":{"1min":0.32300000000000001,"5min":0.308,"15min":0.307},"max":{"1min":0.69399999999999995,"5min":1.0289999999999999,"15min":1.4970000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51300000000000001,"15min":0.60999999999999999},"min":{"1min":0.34799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.68500000000000005,"5min":0.90200000000000002,"15min":3.972},"last":0.33500000000000002}]},{"osd":2,"last update":"Fri Mar 3 08:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52600000000000002,"15min":0.85299999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.313},"max":{"1min":0.64200000000000002,"5min":0.78900000000000003,"15min":11.897},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59699999999999998,"15min":0.93200000000000005},"min":{"1min":0.46200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78700000000000003,"5min":0.85199999999999998,"15min":11.804},"last":0.503}]},{"osd":3,"last update":"Fri Mar 3 08:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58999999999999997,"15min":0.92300000000000004},"min":{"1min":0.42299999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.752,"5min":0.89500000000000002,"15min":11.843999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55800000000000005,"15min":0.68899999999999995},"min":{"1min":0.40799999999999997,"5min":0.40000000000000002,"15min":0.36699999999999999},"max":{"1min":0.80800000000000005,"5min":0.96399999999999997,"15min":3.8740000000000001},"last":0.54200000000000004}]},{"osd":4,"last update":"Fri Mar 3 08:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.753,"15min":0.83399999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.39900000000000002},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.613},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.81299999999999994,"15min":0.84899999999999998},"min":{"1min":0.624,"5min":0.624,"15min":0.53800000000000003},"max":{"1min":1.0429999999999999,"5min":1.127,"15min":1.4630000000000001},"last":0.99099999999999999}]},{"osd":5,"last update":"Fri Mar 3 08:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.753,"15min":0.88100000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0189999999999999,"5min":1.0620000000000001,"15min":3.8300000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.749,"15min":0.83999999999999997},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.47499999999999998},"max":{"1min":0.96199999999999997,"5min":1.008,"15min":1.5469999999999999},"last":0.82199999999999995}]},{"osd":6,"last update":"Fri Mar 3 08:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.81100000000000005,"15min":0.94999999999999996},"min":{"1min":0.64300000000000002,"5min":0.62,"15min":0.62},"max":{"1min":1.028,"5min":1.173,"15min":3.907},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.80200000000000005,"15min":0.90400000000000003},"min":{"1min":0.59799999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1779999999999999,"5min":1.1819999999999999,"15min":1.5820000000000001},"last":0.876}]},{"osd":7,"last update":"Fri Mar 3 08:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.79300000000000004,"15min":0.85899999999999999},"min":{"1min":0.60899999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.123,"5min":1.417,"15min":1.419},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.80400000000000005,"15min":0.95799999999999996},"min":{"1min":0.58399999999999996,"5min":0.58199999999999996,"15min":0.57099999999999995},"max":{"1min":1.153,"5min":1.3560000000000001,"15min":3.9399999999999999},"last":0.89100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574944,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":16592,"kb_used_data":1928,"kb_used_omap":0,"kb_used_meta":14656,"kb_avail":94355248,"statfs":{"total":96636764160,"available":96619773952,"internally_reserved":0,"allocated":1974272,"data_stored":1230576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":15007744},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 08:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44,"15min":0.46200000000000002},"min":{"1min":0.32900000000000001,"5min":0.30399999999999999,"15min":0.23999999999999999},"max":{"1min":0.63,"5min":0.65000000000000002,"15min":0.89300000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.35599999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.75900000000000001,"5min":0.81999999999999995,"15min":0.871},"last":0.46100000000000002}]},{"osd":2,"last update":"Fri Mar 3 08:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51100000000000001,"15min":0.54100000000000004},"min":{"1min":0.39200000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.66000000000000003,"5min":0.77400000000000002,"15min":0.79100000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.41499999999999998,"5min":0.39100000000000001,"15min":0.32800000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":1.1020000000000001},"last":0.50800000000000001}]},{"osd":3,"last update":"Fri Mar 3 08:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53300000000000003,"15min":0.54700000000000004},"min":{"1min":0.371,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70099999999999996,"5min":0.71499999999999997,"15min":0.98499999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58899999999999997,"15min":0.627},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.71299999999999997,"5min":0.83499999999999996,"15min":1.2450000000000001},"last":0.55900000000000005}]},{"osd":4,"last update":"Fri Mar 3 08:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.78700000000000003,"15min":0.83099999999999996},"min":{"1min":0.61299999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.1120000000000001,"5min":1.242,"15min":1.5189999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77300000000000002,"15min":0.81100000000000005},"min":{"1min":0.53800000000000003,"5min":0.505,"15min":0.505},"max":{"1min":0.999,"5min":1.218,"15min":1.605},"last":0.67900000000000005}]},{"osd":5,"last update":"Fri Mar 3 08:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.77700000000000002,"15min":0.81200000000000006},"min":{"1min":0.67600000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.93899999999999995,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.78800000000000003,"15min":0.81499999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.056,"5min":1.2070000000000001,"15min":1.286},"last":0.81299999999999994}]},{"osd":6,"last update":"Fri Mar 3 08:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79300000000000004,"15min":0.86099999999999999},"min":{"1min":0.625,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.95399999999999996,"5min":1.2470000000000001,"15min":1.623},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.79000000000000004,"15min":0.84699999999999998},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1319999999999999,"5min":1.137,"15min":1.5780000000000001},"last":0.83199999999999996}]},{"osd":7,"last update":"Fri Mar 3 08:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.80900000000000005,"15min":0.86499999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.57099999999999995},"max":{"1min":1.0569999999999999,"5min":1.1020000000000001,"15min":1.6479999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.79900000000000004,"15min":0.84699999999999998},"min":{"1min":0.59699999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1,"5min":1.1719999999999999,"15min":1.5509999999999999},"last":0.79800000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574945,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11196,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":9856,"kb_avail":94360644,"statfs":{"total":96636764160,"available":96625299456,"internally_reserved":0,"allocated":1363968,"data_stored":633533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10092544},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.432,"15min":0.76100000000000001},"min":{"1min":0.32400000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71199999999999997,"5min":0.78900000000000003,"15min":5.7889999999999997},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.5,"15min":0.67400000000000004},"min":{"1min":0.40699999999999997,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59999999999999998,"5min":0.93700000000000006,"15min":2.6309999999999998},"last":0.434}]},{"osd":1,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51000000000000001,"15min":0.69199999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67100000000000004,"5min":0.90400000000000003,"15min":2.5209999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58699999999999997,"15min":0.90200000000000002},"min":{"1min":0.438,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.745,"5min":0.98399999999999999,"15min":5.8470000000000004},"last":0.53000000000000003}]},{"osd":3,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56299999999999994,"15min":1.0549999999999999},"min":{"1min":0.38700000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.71099999999999997,"5min":1.012,"15min":12.961},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.57599999999999996,"15min":0.92000000000000004},"min":{"1min":0.47999999999999998,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.78400000000000003,"5min":0.92100000000000004,"15min":6.1449999999999996},"last":0.497}]},{"osd":4,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.79700000000000004,"15min":1.155},"min":{"1min":0.51400000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.89200000000000002,"5min":1.468,"15min":6.484},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.83299999999999996,"15min":1.147},"min":{"1min":0.623,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.032,"5min":1.1759999999999999,"15min":6.5510000000000002},"last":0.67400000000000004}]},{"osd":5,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.755,"15min":1.073},"min":{"1min":0.56699999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.88800000000000001,"5min":1.304,"15min":5.7119999999999997},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.79100000000000004,"15min":1.1240000000000001},"min":{"1min":0.57399999999999995,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0049999999999999,"5min":1.1499999999999999,"15min":6.6369999999999996},"last":0.51600000000000001}]},{"osd":6,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.82799999999999996,"15min":1.1339999999999999},"min":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.0129999999999999,"5min":1.393,"15min":6.625},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.83099999999999996,"15min":1.1559999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":0.97499999999999998,"5min":1.347,"15min":5.7539999999999996},"last":0.752}]},{"osd":7,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.78100000000000003,"15min":1.1160000000000001},"min":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.92500000000000004,"5min":1.0980000000000001,"15min":6.7009999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.81000000000000005,"15min":1.1100000000000001},"min":{"1min":0.59999999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":0.96399999999999997,"5min":1.107,"15min":6.6840000000000002},"last":0.57699999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574943,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15708,"kb_used_data":1364,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":94356132,"statfs":{"total":96636764160,"available":96620679168,"internally_reserved":0,"allocated":1396736,"data_stored":642787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.47399999999999998,"15min":0.748},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.7050000000000001,"5min":4.7050000000000001,"15min":9.5820000000000007},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.55000000000000004,"15min":0.82099999999999995},"min":{"1min":0.34300000000000003,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":9.7840000000000007},"last":0.57899999999999996}]},{"osd":1,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.504,"15min":0.51900000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.58799999999999997,"15min":0.89400000000000002},"min":{"1min":0.36899999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":4.9199999999999999,"5min":4.9199999999999999,"15min":9.702},"last":0.52500000000000002}]},{"osd":2,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.61699999999999999,"15min":0.85999999999999999},"min":{"1min":0.42499999999999999,"5min":0.41499999999999998,"15min":0.36499999999999999},"max":{"1min":3.5489999999999999,"5min":3.5489999999999999,"15min":8.3190000000000008},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.61399999999999999,"15min":0.92200000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.41899999999999998},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":9.734},"last":0.55100000000000005}]},{"osd":4,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.80000000000000004,"15min":1.0800000000000001},"min":{"1min":0.61199999999999999,"5min":0.54400000000000004,"15min":0.51100000000000001},"max":{"1min":4.2519999999999998,"5min":4.2519999999999998,"15min":9.8629999999999995},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.79800000000000004,"15min":0.83999999999999997},"min":{"1min":0.61199999999999999,"5min":0.55700000000000005,"15min":0.55100000000000005},"max":{"1min":3.5110000000000001,"5min":3.5110000000000001,"15min":3.5110000000000001},"last":0.70999999999999996}]},{"osd":5,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.82099999999999995,"15min":0.83299999999999996},"min":{"1min":0.65400000000000003,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.5880000000000001,"5min":3.5880000000000001,"15min":3.5880000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.79900000000000004,"15min":1.1200000000000001},"min":{"1min":0.628,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.7040000000000002,"5min":2.7040000000000002,"15min":9.6600000000000001},"last":0.71999999999999997}]},{"osd":6,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":0.84599999999999997,"15min":1.1459999999999999},"min":{"1min":0.60699999999999998,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":4.423,"5min":4.423,"15min":9.8190000000000008},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.79600000000000004,"15min":0.85099999999999998},"min":{"1min":0.626,"5min":0.60299999999999998,"15min":0.51800000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.609},"last":0.88600000000000001}]},{"osd":7,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.81299999999999994,"15min":1.1619999999999999},"min":{"1min":0.67000000000000004,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.226,"5min":2.226,"15min":9.625},"last":0.879},{"interface":"front","average":{"1min":1.0609999999999999,"5min":0.85799999999999998,"15min":1.153},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":4.3259999999999996,"5min":4.3259999999999996,"15min":9.8949999999999996},"last":0.80500000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574943,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11928,"kb_used_data":1360,"kb_used_omap":0,"kb_used_meta":10560,"kb_avail":94359912,"statfs":{"total":96636764160,"available":96624549888,"internally_reserved":0,"allocated":1392640,"data_stored":642753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10813440},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71399999999999997,"15min":0.91600000000000004},"min":{"1min":0.52500000000000002,"5min":0.39700000000000002,"15min":0.35599999999999998},"max":{"1min":0.94099999999999995,"5min":1.0049999999999999,"15min":7.1799999999999997},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.70499999999999996,"15min":0.67500000000000004},"min":{"1min":0.47999999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91500000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.70199999999999996}]},{"osd":1,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.71699999999999997,"15min":1.0109999999999999},"min":{"1min":0.47399999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.88900000000000001,"5min":0.94899999999999995,"15min":9.5709999999999997},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.748,"15min":0.878},"min":{"1min":0.496,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.92500000000000004,"5min":1.153,"15min":5.266},"last":0.89200000000000002}]},{"osd":2,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.74099999999999999,"15min":0.99099999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.98399999999999999,"5min":1.034,"15min":8.4529999999999994},"last":1.024},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.752,"15min":0.93999999999999995},"min":{"1min":0.55700000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88200000000000001,"5min":1.018,"15min":6.6829999999999998},"last":0.81499999999999995}]},{"osd":3,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.54000000000000004,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.872,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.78200000000000003,"15min":0.79800000000000004},"min":{"1min":0.59399999999999997,"5min":0.58199999999999996,"15min":0.54300000000000004},"max":{"1min":0.95899999999999996,"5min":0.99299999999999999,"15min":1.389},"last":0.77800000000000002}]},{"osd":5,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69099999999999995,"15min":0.72999999999999998},"min":{"1min":0.46700000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1,"5min":1,"15min":1.0349999999999999},"last":1.002},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.70899999999999996,"15min":0.71899999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.88700000000000001,"5min":0.998,"15min":1.0640000000000001},"last":0.93600000000000005}]},{"osd":6,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.76600000000000001,"15min":0.77800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.996,"5min":1.032,"15min":1.032},"last":1.077},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.78800000000000003,"15min":0.81200000000000006},"min":{"1min":0.52600000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.032,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":1.204}]},{"osd":7,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.76100000000000001,"15min":0.80800000000000005},"min":{"1min":0.51000000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.95799999999999996,"5min":1.0609999999999999,"15min":1.679},"last":1.0569999999999999},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.83999999999999997,"15min":0.90700000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0169999999999999,"5min":1.1599999999999999,"15min":1.8340000000000001},"last":1.1899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574944,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11204,"kb_used_data":1340,"kb_used_omap":0,"kb_used_meta":9856,"kb_avail":94360636,"statfs":{"total":96636764160,"available":96625291264,"internally_reserved":0,"allocated":1372160,"data_stored":633693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10092544},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.44900000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.76700000000000002,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.751,"15min":0.74099999999999999},"min":{"1min":0.46800000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0600000000000001,"5min":2.528,"15min":2.528},"last":0.77400000000000002}]},{"osd":1,"last update":"Fri Mar 3 08:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.72299999999999998,"15min":0.73599999999999999},"min":{"1min":0.48599999999999999,"5min":0.45900000000000002,"15min":0.45700000000000002},"max":{"1min":0.93600000000000005,"5min":1.038,"15min":1.038},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.69799999999999995,"15min":0.64900000000000002},"min":{"1min":0.46000000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.75800000000000001}]},{"osd":2,"last update":"Fri Mar 3 08:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.48999999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70999999999999996,"15min":0.71199999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.86599999999999999,"5min":1.024,"15min":1.321},"last":0.80600000000000005}]},{"osd":3,"last update":"Fri Mar 3 08:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.71699999999999997,"15min":0.69699999999999995},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.999,"5min":1.024,"15min":1.024},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.749,"15min":0.73699999999999999},"min":{"1min":0.53500000000000003,"5min":0.501,"15min":0.501},"max":{"1min":0.93100000000000005,"5min":1.012,"15min":1.012},"last":0.70699999999999996}]},{"osd":4,"last update":"Fri Mar 3 08:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.67100000000000004,"15min":0.66600000000000004},"min":{"1min":0.41399999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83599999999999997,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.61499999999999999,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.94299999999999995,"5min":1.079,"15min":1.079},"last":0.83099999999999996}]},{"osd":6,"last update":"Fri Mar 3 08:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72499999999999998,"15min":0.77800000000000002},"min":{"1min":0.48199999999999998,"5min":0.442,"15min":0.442},"max":{"1min":0.98299999999999998,"5min":1.012,"15min":1.071},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.71799999999999997,"15min":0.72199999999999998},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.91400000000000003,"5min":0.97099999999999997,"15min":1.0429999999999999},"last":0.79300000000000004}]},{"osd":7,"last update":"Fri Mar 3 08:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.75,"15min":0.75600000000000001},"min":{"1min":0.498,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0269999999999999,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.78900000000000003,"15min":0.82099999999999995},"min":{"1min":0.54900000000000004,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.1080000000000001},"last":0.85699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:03:18.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:03:18.565 INFO:tasks.ceph.mds.d.smithi100.stderr:2023-03-03T08:03:18.563+0000 7f5504df1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T08:03:18.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.815+0000 7f7fb8dff700 1 Processor -- start 2023-03-03T08:03:18.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.815+0000 7f7fb8dff700 1 -- start start 2023-03-03T08:03:18.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.817+0000 7f7fb8dff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7fb40b8b90 0x7f7fb40cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:18.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.817+0000 7f7fb8dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fb40b5410 0x7f7fb40d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:18.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.817+0000 7f7fb8dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb40d0ef0 con 0x7f7fb40b87c0 2023-03-03T08:03:18.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.817+0000 7f7fb8dff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb40d0600 con 0x7f7fb40b8b90 2023-03-03T08:03:18.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.817+0000 7f7fb8dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb40cd970 con 0x7f7fb40b5410 2023-03-03T08:03:18.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.817+0000 7f7fb2d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fb40b5410 0x7f7fb40d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:18.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.817+0000 7f7fb1d9b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7fb40b8b90 0x7f7fb40cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:18.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.818+0000 7f7fb2d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fb40b5410 0x7f7fb40d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:51576/0 (socket says 172.21.15.98:51576) 2023-03-03T08:03:18.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.818+0000 7f7fb1d9b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7fb40b8b90 0x7f7fb40cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:58114/0 (socket says 172.21.15.98:58114) 2023-03-03T08:03:18.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.818+0000 7f7fb2d9d700 1 -- 172.21.15.98:0/220757009 learned_addr learned my addr 172.21.15.98:0/220757009 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:18.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.818+0000 7f7fb1d9b700 1 -- 172.21.15.98:0/220757009 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fb40b5410 msgr2=0x7f7fb40d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:18.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.818+0000 7f7fb1d9b700 1 --2- 172.21.15.98:0/220757009 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fb40b5410 0x7f7fb40d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.818+0000 7f7fb1d9b700 1 -- 172.21.15.98:0/220757009 >> v1:172.21.15.100:6789/0 conn(0x7f7fb40b87c0 legacy=0x7f7fb40b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:18.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.818+0000 7f7fb1d9b700 1 -- 172.21.15.98:0/220757009 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb40cdc50 con 0x7f7fb40b8b90 2023-03-03T08:03:18.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.818+0000 7f7fb1d9b700 1 --2- 172.21.15.98:0/220757009 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fb40b8b90 0x7f7fb40cffe0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009840 tx=0x7f7fa8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c78401c439132bcc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:18.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.819+0000 7f7fb159a700 1 -- 172.21.15.98:0/220757009 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8011830 con 0x7f7fb40b8b90 2023-03-03T08:03:18.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.819+0000 7f7fb159a700 1 -- 172.21.15.98:0/220757009 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa8011990 con 0x7f7fb40b8b90 2023-03-03T08:03:18.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.819+0000 7f7fb159a700 1 -- 172.21.15.98:0/220757009 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8011d20 con 0x7f7fb40b8b90 2023-03-03T08:03:18.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.819+0000 7f7fb8dff700 1 -- 172.21.15.98:0/220757009 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fb40b8b90 msgr2=0x7f7fb40cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:18.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.819+0000 7f7fb8dff700 1 --2- 172.21.15.98:0/220757009 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fb40b8b90 0x7f7fb40cffe0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009840 tx=0x7f7fa8009b50 comp rx=0 tx=0).stop 2023-03-03T08:03:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.820+0000 7f7fb8dff700 1 -- 172.21.15.98:0/220757009 shutdown_connections 2023-03-03T08:03:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.820+0000 7f7fb8dff700 1 --2- 172.21.15.98:0/220757009 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fb40b5410 0x7f7fb40d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.820+0000 7f7fb8dff700 1 --2- 172.21.15.98:0/220757009 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fb40b8b90 0x7f7fb40cffe0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.820+0000 7f7fb8dff700 1 -- 172.21.15.98:0/220757009 >> 172.21.15.98:0/220757009 conn(0x7f7fb406aee0 msgr2=0x7f7fb406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.820+0000 7f7fb8dff700 1 -- 172.21.15.98:0/220757009 shutdown_connections 2023-03-03T08:03:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.820+0000 7f7fb8dff700 1 -- 172.21.15.98:0/220757009 wait complete. 2023-03-03T08:03:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.822+0000 7f7fb8dff700 1 Processor -- start 2023-03-03T08:03:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.822+0000 7f7fb8dff700 1 -- start start 2023-03-03T08:03:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.823+0000 7f7fb8dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb40b5410 0x7f7fb415e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.823+0000 7f7fb8dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fb40b87c0 0x7f7fb4156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.823+0000 7f7fb8dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fb40b8b90 0x7f7fb415bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.823+0000 7f7fb8dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb415e4f0 con 0x7f7fb40b5410 2023-03-03T08:03:18.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.823+0000 7f7fb8dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb40ca350 con 0x7f7fb40b87c0 2023-03-03T08:03:18.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.823+0000 7f7fb8dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb40ca630 con 0x7f7fb40b8b90 2023-03-03T08:03:18.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.823+0000 7f7fb259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb40b5410 0x7f7fb415e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:18.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.823+0000 7f7fb259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb40b5410 0x7f7fb415e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60200/0 (socket says 172.21.15.98:60200) 2023-03-03T08:03:18.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.823+0000 7f7fb259c700 1 -- 172.21.15.98:0/4076631125 learned_addr learned my addr 172.21.15.98:0/4076631125 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:18.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.823+0000 7f7fb1d9b700 1 --2- 172.21.15.98:0/4076631125 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fb40b87c0 0x7f7fb4156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:18.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.823+0000 7f7fb2d9d700 1 --2- 172.21.15.98:0/4076631125 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fb40b8b90 0x7f7fb415bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:18.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.824+0000 7f7fb259c700 1 -- 172.21.15.98:0/4076631125 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fb40b8b90 msgr2=0x7f7fb415bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:18.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.824+0000 7f7fb259c700 1 --2- 172.21.15.98:0/4076631125 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fb40b8b90 0x7f7fb415bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.824+0000 7f7fb259c700 1 -- 172.21.15.98:0/4076631125 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fb40b87c0 msgr2=0x7f7fb4156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:18.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.824+0000 7f7fb259c700 1 --2- 172.21.15.98:0/4076631125 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fb40b87c0 0x7f7fb4156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:18.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.824+0000 7f7fb259c700 1 -- 172.21.15.98:0/4076631125 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa80094a0 con 0x7f7fb40b5410 2023-03-03T08:03:18.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.824+0000 7f7fb259c700 1 --2- 172.21.15.98:0/4076631125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb40b5410 0x7f7fb415e7d0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400c4e0 tx=0x7f7fa400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:18.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.824+0000 7f7fa2ffd700 1 -- 172.21.15.98:0/4076631125 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa40052d0 con 0x7f7fb40b5410 2023-03-03T08:03:18.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.824+0000 7f7fa2ffd700 1 -- 172.21.15.98:0/4076631125 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa4005430 con 0x7f7fb40b5410 2023-03-03T08:03:18.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.824+0000 7f7fb8dff700 1 -- 172.21.15.98:0/4076631125 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb40cdf30 con 0x7f7fb40b5410 2023-03-03T08:03:18.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.824+0000 7f7fa2ffd700 1 -- 172.21.15.98:0/4076631125 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa40038c0 con 0x7f7fb40b5410 2023-03-03T08:03:18.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.825+0000 7f7fb8dff700 1 -- 172.21.15.98:0/4076631125 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb4073a80 con 0x7f7fb40b5410 2023-03-03T08:03:18.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.826+0000 7f7fb8dff700 1 -- 172.21.15.98:0/4076631125 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb40ce0c0 con 0x7f7fb40b5410 2023-03-03T08:03:18.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.827+0000 7f7fa2ffd700 1 -- 172.21.15.98:0/4076631125 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f7fa4019030 con 0x7f7fb40b5410 2023-03-03T08:03:18.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.828+0000 7f7fa2ffd700 1 --2- 172.21.15.98:0/4076631125 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7f94068380 0x7f7f9406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:18.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.828+0000 7f7fa2ffd700 1 -- 172.21.15.98:0/4076631125 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7fa4084fe0 con 0x7f7fb40b5410 2023-03-03T08:03:18.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.829+0000 7f7fb1d9b700 1 --2- 172.21.15.98:0/4076631125 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7f94068380 0x7f7f9406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:18.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.830+0000 7f7fb1d9b700 1 --2- 172.21.15.98:0/4076631125 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7f94068380 0x7f7f9406a830 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7fb40b8fd0 tx=0x7f7fa801c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:18.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:18.833+0000 7f7fa2ffd700 1 -- 172.21.15.98:0/4076631125 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7fa40517a0 con 0x7f7fb40b5410 2023-03-03T08:03:19.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.024+0000 7f7fb8dff700 1 -- 172.21.15.98:0/4076631125 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7fb40729b0 con 0x7f7f94068380 2023-03-03T08:03:19.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.035+0000 7f7fa2ffd700 1 -- 172.21.15.98:0/4076631125 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311616 (secure 0 0 0) 0x7f7fb40729b0 con 0x7f7f94068380 2023-03-03T08:03:19.043 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:03:19.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.040+0000 7f7fb8dff700 1 -- 172.21.15.98:0/4076631125 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7f94068380 msgr2=0x7f7f9406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:19.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.040+0000 7f7fb8dff700 1 --2- 172.21.15.98:0/4076631125 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7f94068380 0x7f7f9406a830 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7fb40b8fd0 tx=0x7f7fa801c040 comp rx=0 tx=0).stop 2023-03-03T08:03:19.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.040+0000 7f7fb8dff700 1 -- 172.21.15.98:0/4076631125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb40b5410 msgr2=0x7f7fb415e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:19.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.040+0000 7f7fb8dff700 1 --2- 172.21.15.98:0/4076631125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb40b5410 0x7f7fb415e7d0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400c4e0 tx=0x7f7fa400c7f0 comp rx=0 tx=0).stop 2023-03-03T08:03:19.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.040+0000 7f7fb8dff700 1 -- 172.21.15.98:0/4076631125 shutdown_connections 2023-03-03T08:03:19.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.040+0000 7f7fb8dff700 1 --2- 172.21.15.98:0/4076631125 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7f94068380 0x7f7f9406a830 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:19.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.040+0000 7f7fb8dff700 1 --2- 172.21.15.98:0/4076631125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb40b5410 0x7f7fb415e7d0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:19.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.040+0000 7f7fb8dff700 1 --2- 172.21.15.98:0/4076631125 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fb40b8b90 0x7f7fb415bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:19.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.040+0000 7f7fb8dff700 1 --2- 172.21.15.98:0/4076631125 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fb40b87c0 0x7f7fb4156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:19.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.040+0000 7f7fb8dff700 1 -- 172.21.15.98:0/4076631125 >> 172.21.15.98:0/4076631125 conn(0x7f7fb406aee0 msgr2=0x7f7fb40b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:19.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.040+0000 7f7fb8dff700 1 -- 172.21.15.98:0/4076631125 shutdown_connections 2023-03-03T08:03:19.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.040+0000 7f7fb8dff700 1 -- 172.21.15.98:0/4076631125 wait complete. 2023-03-03T08:03:19.048 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-03-03T08:03:19.070 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2023-03-03T08:03:18.178288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":223,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166,"ondisk_log_size":166,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":131,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":106356,"kb_used_data":11956,"kb_used_omap":0,"kb_used_meta":94336,"kb_avail":754868364,"statfs":{"total":773094113280,"available":772985204736,"internally_reserved":0,"allocated":12242944,"data_stored":6291001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":96600064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001589"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523119+0000","last_change":"2023-03-03T08:03:13.533486+0000","last_active":"2023-03-03T08:03:14.523119+0000","last_peered":"2023-03-03T08:03:14.523119+0000","last_clean":"2023-03-03T08:03:14.523119+0000","last_became_active":"2023-03-03T08:03:13.533163+0000","last_became_peered":"2023-03-03T08:03:13.533163+0000","last_unstale":"2023-03-03T08:03:14.523119+0000","last_undegraded":"2023-03-03T08:03:14.523119+0000","last_fullsized":"2023-03-03T08:03:14.523119+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:19:12.616918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518473+0000","last_change":"2023-03-03T08:03:13.538123+0000","last_active":"2023-03-03T08:03:14.518473+0000","last_peered":"2023-03-03T08:03:14.518473+0000","last_clean":"2023-03-03T08:03:14.518473+0000","last_became_active":"2023-03-03T08:03:13.537834+0000","last_became_peered":"2023-03-03T08:03:13.537834+0000","last_unstale":"2023-03-03T08:03:14.518473+0000","last_undegraded":"2023-03-03T08:03:14.518473+0000","last_fullsized":"2023-03-03T08:03:14.518473+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:06:03.384487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518646+0000","last_change":"2023-03-03T08:03:13.541184+0000","last_active":"2023-03-03T08:03:14.518646+0000","last_peered":"2023-03-03T08:03:14.518646+0000","last_clean":"2023-03-03T08:03:14.518646+0000","last_became_active":"2023-03-03T08:03:13.540768+0000","last_became_peered":"2023-03-03T08:03:13.540768+0000","last_unstale":"2023-03-03T08:03:14.518646+0000","last_undegraded":"2023-03-03T08:03:14.518646+0000","last_fullsized":"2023-03-03T08:03:14.518646+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:34:21.624584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504541+0000","last_change":"2023-03-03T08:03:13.536126+0000","last_active":"2023-03-03T08:03:15.504541+0000","last_peered":"2023-03-03T08:03:15.504541+0000","last_clean":"2023-03-03T08:03:15.504541+0000","last_became_active":"2023-03-03T08:03:13.535846+0000","last_became_peered":"2023-03-03T08:03:13.535846+0000","last_unstale":"2023-03-03T08:03:15.504541+0000","last_undegraded":"2023-03-03T08:03:15.504541+0000","last_fullsized":"2023-03-03T08:03:15.504541+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:01:17.733840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523134+0000","last_change":"2023-03-03T08:03:13.537294+0000","last_active":"2023-03-03T08:03:14.523134+0000","last_peered":"2023-03-03T08:03:14.523134+0000","last_clean":"2023-03-03T08:03:14.523134+0000","last_became_active":"2023-03-03T08:03:13.537046+0000","last_became_peered":"2023-03-03T08:03:13.537046+0000","last_unstale":"2023-03-03T08:03:14.523134+0000","last_undegraded":"2023-03-03T08:03:14.523134+0000","last_fullsized":"2023-03-03T08:03:14.523134+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:58:37.748545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608860+0000","last_change":"2023-03-03T08:03:13.541951+0000","last_active":"2023-03-03T08:03:15.608860+0000","last_peered":"2023-03-03T08:03:15.608860+0000","last_clean":"2023-03-03T08:03:15.608860+0000","last_became_active":"2023-03-03T08:03:13.541648+0000","last_became_peered":"2023-03-03T08:03:13.541648+0000","last_unstale":"2023-03-03T08:03:15.608860+0000","last_undegraded":"2023-03-03T08:03:15.608860+0000","last_fullsized":"2023-03-03T08:03:15.608860+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:45:32.972695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608534+0000","last_change":"2023-03-03T08:03:13.544582+0000","last_active":"2023-03-03T08:03:15.608534+0000","last_peered":"2023-03-03T08:03:15.608534+0000","last_clean":"2023-03-03T08:03:15.608534+0000","last_became_active":"2023-03-03T08:03:13.544316+0000","last_became_peered":"2023-03-03T08:03:13.544316+0000","last_unstale":"2023-03-03T08:03:15.608534+0000","last_undegraded":"2023-03-03T08:03:15.608534+0000","last_fullsized":"2023-03-03T08:03:15.608534+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:22:36.589665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518017+0000","last_change":"2023-03-03T08:03:13.541072+0000","last_active":"2023-03-03T08:03:14.518017+0000","last_peered":"2023-03-03T08:03:14.518017+0000","last_clean":"2023-03-03T08:03:14.518017+0000","last_became_active":"2023-03-03T08:03:13.540790+0000","last_became_peered":"2023-03-03T08:03:13.540790+0000","last_unstale":"2023-03-03T08:03:14.518017+0000","last_undegraded":"2023-03-03T08:03:14.518017+0000","last_fullsized":"2023-03-03T08:03:14.518017+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:53.717879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504857+0000","last_change":"2023-03-03T08:03:13.531573+0000","last_active":"2023-03-03T08:03:15.504857+0000","last_peered":"2023-03-03T08:03:15.504857+0000","last_clean":"2023-03-03T08:03:15.504857+0000","last_became_active":"2023-03-03T08:03:13.527902+0000","last_became_peered":"2023-03-03T08:03:13.527902+0000","last_unstale":"2023-03-03T08:03:15.504857+0000","last_undegraded":"2023-03-03T08:03:15.504857+0000","last_fullsized":"2023-03-03T08:03:15.504857+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:44:39.592199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898648+0000","last_change":"2023-03-03T08:03:13.525841+0000","last_active":"2023-03-03T08:03:14.898648+0000","last_peered":"2023-03-03T08:03:14.898648+0000","last_clean":"2023-03-03T08:03:14.898648+0000","last_became_active":"2023-03-03T08:03:13.525451+0000","last_became_peered":"2023-03-03T08:03:13.525451+0000","last_unstale":"2023-03-03T08:03:14.898648+0000","last_undegraded":"2023-03-03T08:03:14.898648+0000","last_fullsized":"2023-03-03T08:03:14.898648+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:49:23.485411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523375+0000","last_change":"2023-03-03T08:03:13.536401+0000","last_active":"2023-03-03T08:03:14.523375+0000","last_peered":"2023-03-03T08:03:14.523375+0000","last_clean":"2023-03-03T08:03:14.523375+0000","last_became_active":"2023-03-03T08:03:13.536115+0000","last_became_peered":"2023-03-03T08:03:13.536115+0000","last_unstale":"2023-03-03T08:03:14.523375+0000","last_undegraded":"2023-03-03T08:03:14.523375+0000","last_fullsized":"2023-03-03T08:03:14.523375+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:34:21.624584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608999+0000","last_change":"2023-03-03T08:03:13.529974+0000","last_active":"2023-03-03T08:03:15.608999+0000","last_peered":"2023-03-03T08:03:15.608999+0000","last_clean":"2023-03-03T08:03:15.608999+0000","last_became_active":"2023-03-03T08:03:13.529630+0000","last_became_peered":"2023-03-03T08:03:13.529630+0000","last_unstale":"2023-03-03T08:03:15.608999+0000","last_undegraded":"2023-03-03T08:03:15.608999+0000","last_fullsized":"2023-03-03T08:03:15.608999+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:43:14.168233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898763+0000","last_change":"2023-03-03T08:03:13.535865+0000","last_active":"2023-03-03T08:03:14.898763+0000","last_peered":"2023-03-03T08:03:14.898763+0000","last_clean":"2023-03-03T08:03:14.898763+0000","last_became_active":"2023-03-03T08:03:13.535536+0000","last_became_peered":"2023-03-03T08:03:13.535536+0000","last_unstale":"2023-03-03T08:03:14.898763+0000","last_undegraded":"2023-03-03T08:03:14.898763+0000","last_fullsized":"2023-03-03T08:03:14.898763+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:59:02.639151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518296+0000","last_change":"2023-03-03T08:03:13.542647+0000","last_active":"2023-03-03T08:03:14.518296+0000","last_peered":"2023-03-03T08:03:14.518296+0000","last_clean":"2023-03-03T08:03:14.518296+0000","last_became_active":"2023-03-03T08:03:13.542151+0000","last_became_peered":"2023-03-03T08:03:13.542151+0000","last_unstale":"2023-03-03T08:03:14.518296+0000","last_undegraded":"2023-03-03T08:03:14.518296+0000","last_fullsized":"2023-03-03T08:03:14.518296+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:58:37.748545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.522827+0000","last_change":"2023-03-03T08:03:13.536251+0000","last_active":"2023-03-03T08:03:14.522827+0000","last_peered":"2023-03-03T08:03:14.522827+0000","last_clean":"2023-03-03T08:03:14.522827+0000","last_became_active":"2023-03-03T08:03:13.535962+0000","last_became_peered":"2023-03-03T08:03:13.535962+0000","last_unstale":"2023-03-03T08:03:14.522827+0000","last_undegraded":"2023-03-03T08:03:14.522827+0000","last_fullsized":"2023-03-03T08:03:14.522827+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:09.173678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608239+0000","last_change":"2023-03-03T08:03:13.529261+0000","last_active":"2023-03-03T08:03:15.608239+0000","last_peered":"2023-03-03T08:03:15.608239+0000","last_clean":"2023-03-03T08:03:15.608239+0000","last_became_active":"2023-03-03T08:03:13.528288+0000","last_became_peered":"2023-03-03T08:03:13.528288+0000","last_unstale":"2023-03-03T08:03:15.608239+0000","last_undegraded":"2023-03-03T08:03:15.608239+0000","last_fullsized":"2023-03-03T08:03:15.608239+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:12:29.525071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898680+0000","last_change":"2023-03-03T08:03:13.536371+0000","last_active":"2023-03-03T08:03:14.898680+0000","last_peered":"2023-03-03T08:03:14.898680+0000","last_clean":"2023-03-03T08:03:14.898680+0000","last_became_active":"2023-03-03T08:03:13.535954+0000","last_became_peered":"2023-03-03T08:03:13.535954+0000","last_unstale":"2023-03-03T08:03:14.898680+0000","last_undegraded":"2023-03-03T08:03:14.898680+0000","last_fullsized":"2023-03-03T08:03:14.898680+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:20:02.083571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518500+0000","last_change":"2023-03-03T08:03:13.537731+0000","last_active":"2023-03-03T08:03:14.518500+0000","last_peered":"2023-03-03T08:03:14.518500+0000","last_clean":"2023-03-03T08:03:14.518500+0000","last_became_active":"2023-03-03T08:03:13.537363+0000","last_became_peered":"2023-03-03T08:03:13.537363+0000","last_unstale":"2023-03-03T08:03:14.518500+0000","last_undegraded":"2023-03-03T08:03:14.518500+0000","last_fullsized":"2023-03-03T08:03:14.518500+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:00:50.026977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609070+0000","last_change":"2023-03-03T08:03:13.546570+0000","last_active":"2023-03-03T08:03:15.609070+0000","last_peered":"2023-03-03T08:03:15.609070+0000","last_clean":"2023-03-03T08:03:15.609070+0000","last_became_active":"2023-03-03T08:03:13.545993+0000","last_became_peered":"2023-03-03T08:03:13.545993+0000","last_unstale":"2023-03-03T08:03:15.609070+0000","last_undegraded":"2023-03-03T08:03:15.609070+0000","last_fullsized":"2023-03-03T08:03:15.609070+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:13:53.802081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608167+0000","last_change":"2023-03-03T08:03:13.524630+0000","last_active":"2023-03-03T08:03:15.608167+0000","last_peered":"2023-03-03T08:03:15.608167+0000","last_clean":"2023-03-03T08:03:15.608167+0000","last_became_active":"2023-03-03T08:03:13.524384+0000","last_became_peered":"2023-03-03T08:03:13.524384+0000","last_unstale":"2023-03-03T08:03:15.608167+0000","last_undegraded":"2023-03-03T08:03:15.608167+0000","last_fullsized":"2023-03-03T08:03:15.608167+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:26:10.759906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504755+0000","last_change":"2023-03-03T08:03:13.532766+0000","last_active":"2023-03-03T08:03:15.504755+0000","last_peered":"2023-03-03T08:03:15.504755+0000","last_clean":"2023-03-03T08:03:15.504755+0000","last_became_active":"2023-03-03T08:03:13.532440+0000","last_became_peered":"2023-03-03T08:03:13.532440+0000","last_unstale":"2023-03-03T08:03:15.504755+0000","last_undegraded":"2023-03-03T08:03:15.504755+0000","last_fullsized":"2023-03-03T08:03:15.504755+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:53.717879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523174+0000","last_change":"2023-03-03T08:03:13.542532+0000","last_active":"2023-03-03T08:03:14.523174+0000","last_peered":"2023-03-03T08:03:14.523174+0000","last_clean":"2023-03-03T08:03:14.523174+0000","last_became_active":"2023-03-03T08:03:13.542252+0000","last_became_peered":"2023-03-03T08:03:13.542252+0000","last_unstale":"2023-03-03T08:03:14.523174+0000","last_undegraded":"2023-03-03T08:03:14.523174+0000","last_fullsized":"2023-03-03T08:03:14.523174+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:01:17.733840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504405+0000","last_change":"2023-03-03T08:03:13.535248+0000","last_active":"2023-03-03T08:03:15.504405+0000","last_peered":"2023-03-03T08:03:15.504405+0000","last_clean":"2023-03-03T08:03:15.504405+0000","last_became_active":"2023-03-03T08:03:13.534988+0000","last_became_peered":"2023-03-03T08:03:13.534988+0000","last_unstale":"2023-03-03T08:03:15.504405+0000","last_undegraded":"2023-03-03T08:03:15.504405+0000","last_fullsized":"2023-03-03T08:03:15.504405+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:16:44.241182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608456+0000","last_change":"2023-03-03T08:03:13.540902+0000","last_active":"2023-03-03T08:03:15.608456+0000","last_peered":"2023-03-03T08:03:15.608456+0000","last_clean":"2023-03-03T08:03:15.608456+0000","last_became_active":"2023-03-03T08:03:13.540200+0000","last_became_peered":"2023-03-03T08:03:13.540200+0000","last_unstale":"2023-03-03T08:03:15.608456+0000","last_undegraded":"2023-03-03T08:03:15.608456+0000","last_fullsized":"2023-03-03T08:03:15.608456+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:00:50.026977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609556+0000","last_change":"2023-03-03T08:03:13.536855+0000","last_active":"2023-03-03T08:03:15.609556+0000","last_peered":"2023-03-03T08:03:15.609556+0000","last_clean":"2023-03-03T08:03:15.609556+0000","last_became_active":"2023-03-03T08:03:13.536559+0000","last_became_peered":"2023-03-03T08:03:13.536559+0000","last_unstale":"2023-03-03T08:03:15.609556+0000","last_undegraded":"2023-03-03T08:03:15.609556+0000","last_fullsized":"2023-03-03T08:03:15.609556+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:53.717879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608418+0000","last_change":"2023-03-03T08:03:13.525143+0000","last_active":"2023-03-03T08:03:15.608418+0000","last_peered":"2023-03-03T08:03:15.608418+0000","last_clean":"2023-03-03T08:03:15.608418+0000","last_became_active":"2023-03-03T08:03:13.524843+0000","last_became_peered":"2023-03-03T08:03:13.524843+0000","last_unstale":"2023-03-03T08:03:15.608418+0000","last_undegraded":"2023-03-03T08:03:15.608418+0000","last_fullsized":"2023-03-03T08:03:15.608418+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:08:30.932557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609003+0000","last_change":"2023-03-03T08:03:13.546267+0000","last_active":"2023-03-03T08:03:15.609003+0000","last_peered":"2023-03-03T08:03:15.609003+0000","last_clean":"2023-03-03T08:03:15.609003+0000","last_became_active":"2023-03-03T08:03:13.545593+0000","last_became_peered":"2023-03-03T08:03:13.545593+0000","last_unstale":"2023-03-03T08:03:15.609003+0000","last_undegraded":"2023-03-03T08:03:15.609003+0000","last_fullsized":"2023-03-03T08:03:15.609003+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:05:16.636308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608899+0000","last_change":"2023-03-03T08:03:13.536872+0000","last_active":"2023-03-03T08:03:15.608899+0000","last_peered":"2023-03-03T08:03:15.608899+0000","last_clean":"2023-03-03T08:03:15.608899+0000","last_became_active":"2023-03-03T08:03:13.536515+0000","last_became_peered":"2023-03-03T08:03:13.536515+0000","last_unstale":"2023-03-03T08:03:15.608899+0000","last_undegraded":"2023-03-03T08:03:15.608899+0000","last_fullsized":"2023-03-03T08:03:15.608899+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:04:11.932849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608270+0000","last_change":"2023-03-03T08:03:13.524788+0000","last_active":"2023-03-03T08:03:15.608270+0000","last_peered":"2023-03-03T08:03:15.608270+0000","last_clean":"2023-03-03T08:03:15.608270+0000","last_became_active":"2023-03-03T08:03:13.524050+0000","last_became_peered":"2023-03-03T08:03:13.524050+0000","last_unstale":"2023-03-03T08:03:15.608270+0000","last_undegraded":"2023-03-03T08:03:15.608270+0000","last_fullsized":"2023-03-03T08:03:15.608270+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:58.868299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518337+0000","last_change":"2023-03-03T08:03:13.542700+0000","last_active":"2023-03-03T08:03:14.518337+0000","last_peered":"2023-03-03T08:03:14.518337+0000","last_clean":"2023-03-03T08:03:14.518337+0000","last_became_active":"2023-03-03T08:03:13.542466+0000","last_became_peered":"2023-03-03T08:03:13.542466+0000","last_unstale":"2023-03-03T08:03:14.518337+0000","last_undegraded":"2023-03-03T08:03:14.518337+0000","last_fullsized":"2023-03-03T08:03:14.518337+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:01:17.733840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608633+0000","last_change":"2023-03-03T08:03:13.528885+0000","last_active":"2023-03-03T08:03:15.608633+0000","last_peered":"2023-03-03T08:03:15.608633+0000","last_clean":"2023-03-03T08:03:15.608633+0000","last_became_active":"2023-03-03T08:03:13.528550+0000","last_became_peered":"2023-03-03T08:03:13.528550+0000","last_unstale":"2023-03-03T08:03:15.608633+0000","last_undegraded":"2023-03-03T08:03:15.608633+0000","last_fullsized":"2023-03-03T08:03:15.608633+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:17:00.514431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608492+0000","last_change":"2023-03-03T08:03:13.540714+0000","last_active":"2023-03-03T08:03:15.608492+0000","last_peered":"2023-03-03T08:03:15.608492+0000","last_clean":"2023-03-03T08:03:15.608492+0000","last_became_active":"2023-03-03T08:03:13.539742+0000","last_became_peered":"2023-03-03T08:03:13.539742+0000","last_unstale":"2023-03-03T08:03:15.608492+0000","last_undegraded":"2023-03-03T08:03:15.608492+0000","last_fullsized":"2023-03-03T08:03:15.608492+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:41:08.450685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898700+0000","last_change":"2023-03-03T08:03:13.526045+0000","last_active":"2023-03-03T08:03:14.898700+0000","last_peered":"2023-03-03T08:03:14.898700+0000","last_clean":"2023-03-03T08:03:14.898700+0000","last_became_active":"2023-03-03T08:03:13.525794+0000","last_became_peered":"2023-03-03T08:03:13.525794+0000","last_unstale":"2023-03-03T08:03:14.898700+0000","last_undegraded":"2023-03-03T08:03:14.898700+0000","last_fullsized":"2023-03-03T08:03:14.898700+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:52.761809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"110'4","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608918+0000","last_change":"2023-03-03T08:03:12.972426+0000","last_active":"2023-03-03T08:03:15.608918+0000","last_peered":"2023-03-03T08:03:15.608918+0000","last_clean":"2023-03-03T08:03:15.608918+0000","last_became_active":"2023-03-03T08:03:12.972074+0000","last_became_peered":"2023-03-03T08:03:12.972074+0000","last_unstale":"2023-03-03T08:03:15.608918+0000","last_undegraded":"2023-03-03T08:03:15.608918+0000","last_fullsized":"2023-03-03T08:03:15.608918+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:22:52.548029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608577+0000","last_change":"2023-03-03T08:03:13.526231+0000","last_active":"2023-03-03T08:03:15.608577+0000","last_peered":"2023-03-03T08:03:15.608577+0000","last_clean":"2023-03-03T08:03:15.608577+0000","last_became_active":"2023-03-03T08:03:13.525969+0000","last_became_peered":"2023-03-03T08:03:13.525969+0000","last_unstale":"2023-03-03T08:03:15.608577+0000","last_undegraded":"2023-03-03T08:03:15.608577+0000","last_fullsized":"2023-03-03T08:03:15.608577+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:42:50.562879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898775+0000","last_change":"2023-03-03T08:03:12.973103+0000","last_active":"2023-03-03T08:03:14.898775+0000","last_peered":"2023-03-03T08:03:14.898775+0000","last_clean":"2023-03-03T08:03:14.898775+0000","last_became_active":"2023-03-03T08:03:12.972819+0000","last_became_peered":"2023-03-03T08:03:12.972819+0000","last_unstale":"2023-03-03T08:03:14.898775+0000","last_undegraded":"2023-03-03T08:03:14.898775+0000","last_fullsized":"2023-03-03T08:03:14.898775+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:59:52.623541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609229+0000","last_change":"2023-03-03T08:03:13.532277+0000","last_active":"2023-03-03T08:03:15.609229+0000","last_peered":"2023-03-03T08:03:15.609229+0000","last_clean":"2023-03-03T08:03:15.609229+0000","last_became_active":"2023-03-03T08:03:13.531948+0000","last_became_peered":"2023-03-03T08:03:13.531948+0000","last_unstale":"2023-03-03T08:03:15.609229+0000","last_undegraded":"2023-03-03T08:03:15.609229+0000","last_fullsized":"2023-03-03T08:03:15.609229+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:50:59.265078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504599+0000","last_change":"2023-03-03T08:03:12.970249+0000","last_active":"2023-03-03T08:03:15.504599+0000","last_peered":"2023-03-03T08:03:15.504599+0000","last_clean":"2023-03-03T08:03:15.504599+0000","last_became_active":"2023-03-03T08:03:12.969836+0000","last_became_peered":"2023-03-03T08:03:12.969836+0000","last_unstale":"2023-03-03T08:03:15.504599+0000","last_undegraded":"2023-03-03T08:03:15.504599+0000","last_fullsized":"2023-03-03T08:03:15.504599+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:11.950736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504881+0000","last_change":"2023-03-03T08:03:13.532840+0000","last_active":"2023-03-03T08:03:15.504881+0000","last_peered":"2023-03-03T08:03:15.504881+0000","last_clean":"2023-03-03T08:03:15.504881+0000","last_became_active":"2023-03-03T08:03:13.531307+0000","last_became_peered":"2023-03-03T08:03:13.531307+0000","last_unstale":"2023-03-03T08:03:15.504881+0000","last_undegraded":"2023-03-03T08:03:15.504881+0000","last_fullsized":"2023-03-03T08:03:15.504881+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:10:15.395992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898809+0000","last_change":"2023-03-03T08:03:12.974193+0000","last_active":"2023-03-03T08:03:14.898809+0000","last_peered":"2023-03-03T08:03:14.898809+0000","last_clean":"2023-03-03T08:03:14.898809+0000","last_became_active":"2023-03-03T08:03:12.973951+0000","last_became_peered":"2023-03-03T08:03:12.973951+0000","last_unstale":"2023-03-03T08:03:14.898809+0000","last_undegraded":"2023-03-03T08:03:14.898809+0000","last_fullsized":"2023-03-03T08:03:14.898809+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:48:22.334467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898396+0000","last_change":"2023-03-03T08:03:13.538795+0000","last_active":"2023-03-03T08:03:14.898396+0000","last_peered":"2023-03-03T08:03:14.898396+0000","last_clean":"2023-03-03T08:03:14.898396+0000","last_became_active":"2023-03-03T08:03:13.538554+0000","last_became_peered":"2023-03-03T08:03:13.538554+0000","last_unstale":"2023-03-03T08:03:14.898396+0000","last_undegraded":"2023-03-03T08:03:14.898396+0000","last_fullsized":"2023-03-03T08:03:14.898396+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:15:57.749701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523272+0000","last_change":"2023-03-03T08:03:12.976886+0000","last_active":"2023-03-03T08:03:14.523272+0000","last_peered":"2023-03-03T08:03:14.523272+0000","last_clean":"2023-03-03T08:03:14.523272+0000","last_became_active":"2023-03-03T08:03:12.976580+0000","last_became_peered":"2023-03-03T08:03:12.976580+0000","last_unstale":"2023-03-03T08:03:14.523272+0000","last_undegraded":"2023-03-03T08:03:14.523272+0000","last_fullsized":"2023-03-03T08:03:14.523272+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:17:37.014803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.522826+0000","last_change":"2023-03-03T08:03:13.538527+0000","last_active":"2023-03-03T08:03:14.522826+0000","last_peered":"2023-03-03T08:03:14.522826+0000","last_clean":"2023-03-03T08:03:14.522826+0000","last_became_active":"2023-03-03T08:03:13.537453+0000","last_became_peered":"2023-03-03T08:03:13.537453+0000","last_unstale":"2023-03-03T08:03:14.522826+0000","last_undegraded":"2023-03-03T08:03:14.522826+0000","last_fullsized":"2023-03-03T08:03:14.522826+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:16:44.241182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608957+0000","last_change":"2023-03-03T08:03:12.976434+0000","last_active":"2023-03-03T08:03:15.608957+0000","last_peered":"2023-03-03T08:03:15.608957+0000","last_clean":"2023-03-03T08:03:15.608957+0000","last_became_active":"2023-03-03T08:03:12.976090+0000","last_became_peered":"2023-03-03T08:03:12.976090+0000","last_unstale":"2023-03-03T08:03:15.608957+0000","last_undegraded":"2023-03-03T08:03:15.608957+0000","last_fullsized":"2023-03-03T08:03:15.608957+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:50:09.187662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518416+0000","last_change":"2023-03-03T08:03:13.532246+0000","last_active":"2023-03-03T08:03:14.518416+0000","last_peered":"2023-03-03T08:03:14.518416+0000","last_clean":"2023-03-03T08:03:14.518416+0000","last_became_active":"2023-03-03T08:03:13.531955+0000","last_became_peered":"2023-03-03T08:03:13.531955+0000","last_unstale":"2023-03-03T08:03:14.518416+0000","last_undegraded":"2023-03-03T08:03:14.518416+0000","last_fullsized":"2023-03-03T08:03:14.518416+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:50:59.265078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609174+0000","last_change":"2023-03-03T08:03:12.975571+0000","last_active":"2023-03-03T08:03:15.609174+0000","last_peered":"2023-03-03T08:03:15.609174+0000","last_clean":"2023-03-03T08:03:15.609174+0000","last_became_active":"2023-03-03T08:03:12.975216+0000","last_became_peered":"2023-03-03T08:03:12.975216+0000","last_unstale":"2023-03-03T08:03:15.609174+0000","last_undegraded":"2023-03-03T08:03:15.609174+0000","last_fullsized":"2023-03-03T08:03:15.609174+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:32:08.429902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898609+0000","last_change":"2023-03-03T08:03:13.526953+0000","last_active":"2023-03-03T08:03:14.898609+0000","last_peered":"2023-03-03T08:03:14.898609+0000","last_clean":"2023-03-03T08:03:14.898609+0000","last_became_active":"2023-03-03T08:03:13.526445+0000","last_became_peered":"2023-03-03T08:03:13.526445+0000","last_unstale":"2023-03-03T08:03:14.898609+0000","last_undegraded":"2023-03-03T08:03:14.898609+0000","last_fullsized":"2023-03-03T08:03:14.898609+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:32:09.433165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608423+0000","last_change":"2023-03-03T08:03:12.971410+0000","last_active":"2023-03-03T08:03:15.608423+0000","last_peered":"2023-03-03T08:03:15.608423+0000","last_clean":"2023-03-03T08:03:15.608423+0000","last_became_active":"2023-03-03T08:03:12.971161+0000","last_became_peered":"2023-03-03T08:03:12.971161+0000","last_unstale":"2023-03-03T08:03:15.608423+0000","last_undegraded":"2023-03-03T08:03:15.608423+0000","last_fullsized":"2023-03-03T08:03:15.608423+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:42:52.028315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609429+0000","last_change":"2023-03-03T08:03:13.528209+0000","last_active":"2023-03-03T08:03:15.609429+0000","last_peered":"2023-03-03T08:03:15.609429+0000","last_clean":"2023-03-03T08:03:15.609429+0000","last_became_active":"2023-03-03T08:03:13.527936+0000","last_became_peered":"2023-03-03T08:03:13.527936+0000","last_unstale":"2023-03-03T08:03:15.609429+0000","last_undegraded":"2023-03-03T08:03:15.609429+0000","last_fullsized":"2023-03-03T08:03:15.609429+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:06:03.384487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898837+0000","last_change":"2023-03-03T08:03:12.971968+0000","last_active":"2023-03-03T08:03:14.898837+0000","last_peered":"2023-03-03T08:03:14.898837+0000","last_clean":"2023-03-03T08:03:14.898837+0000","last_became_active":"2023-03-03T08:03:12.971706+0000","last_became_peered":"2023-03-03T08:03:12.971706+0000","last_unstale":"2023-03-03T08:03:14.898837+0000","last_undegraded":"2023-03-03T08:03:14.898837+0000","last_fullsized":"2023-03-03T08:03:14.898837+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:29.821562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523306+0000","last_change":"2023-03-03T08:03:13.545174+0000","last_active":"2023-03-03T08:03:14.523306+0000","last_peered":"2023-03-03T08:03:14.523306+0000","last_clean":"2023-03-03T08:03:14.523306+0000","last_became_active":"2023-03-03T08:03:13.544886+0000","last_became_peered":"2023-03-03T08:03:13.544886+0000","last_unstale":"2023-03-03T08:03:14.523306+0000","last_undegraded":"2023-03-03T08:03:14.523306+0000","last_fullsized":"2023-03-03T08:03:14.523306+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:10:15.395992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608499+0000","last_change":"2023-03-03T08:03:12.967798+0000","last_active":"2023-03-03T08:03:15.608499+0000","last_peered":"2023-03-03T08:03:15.608499+0000","last_clean":"2023-03-03T08:03:15.608499+0000","last_became_active":"2023-03-03T08:03:12.966918+0000","last_became_peered":"2023-03-03T08:03:12.966918+0000","last_unstale":"2023-03-03T08:03:15.608499+0000","last_undegraded":"2023-03-03T08:03:15.608499+0000","last_fullsized":"2023-03-03T08:03:15.608499+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:45.293155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608829+0000","last_change":"2023-03-03T08:03:13.539258+0000","last_active":"2023-03-03T08:03:15.608829+0000","last_peered":"2023-03-03T08:03:15.608829+0000","last_clean":"2023-03-03T08:03:15.608829+0000","last_became_active":"2023-03-03T08:03:13.538957+0000","last_became_peered":"2023-03-03T08:03:13.538957+0000","last_unstale":"2023-03-03T08:03:15.608829+0000","last_undegraded":"2023-03-03T08:03:15.608829+0000","last_fullsized":"2023-03-03T08:03:15.608829+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:35:57.846808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504839+0000","last_change":"2023-03-03T08:03:12.975961+0000","last_active":"2023-03-03T08:03:15.504839+0000","last_peered":"2023-03-03T08:03:15.504839+0000","last_clean":"2023-03-03T08:03:15.504839+0000","last_became_active":"2023-03-03T08:03:12.975608+0000","last_became_peered":"2023-03-03T08:03:12.975608+0000","last_unstale":"2023-03-03T08:03:15.504839+0000","last_undegraded":"2023-03-03T08:03:15.504839+0000","last_fullsized":"2023-03-03T08:03:15.504839+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:35:52.551259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609674+0000","last_change":"2023-03-03T08:03:13.543808+0000","last_active":"2023-03-03T08:03:15.609674+0000","last_peered":"2023-03-03T08:03:15.609674+0000","last_clean":"2023-03-03T08:03:15.609674+0000","last_became_active":"2023-03-03T08:03:13.543443+0000","last_became_peered":"2023-03-03T08:03:13.543443+0000","last_unstale":"2023-03-03T08:03:15.609674+0000","last_undegraded":"2023-03-03T08:03:15.609674+0000","last_fullsized":"2023-03-03T08:03:15.609674+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:44:27.769287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504498+0000","last_change":"2023-03-03T08:03:12.971813+0000","last_active":"2023-03-03T08:03:15.504498+0000","last_peered":"2023-03-03T08:03:15.504498+0000","last_clean":"2023-03-03T08:03:15.504498+0000","last_became_active":"2023-03-03T08:03:12.971395+0000","last_became_peered":"2023-03-03T08:03:12.971395+0000","last_unstale":"2023-03-03T08:03:15.504498+0000","last_undegraded":"2023-03-03T08:03:15.504498+0000","last_fullsized":"2023-03-03T08:03:15.504498+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:20:01.080308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608341+0000","last_change":"2023-03-03T08:03:13.537183+0000","last_active":"2023-03-03T08:03:15.608341+0000","last_peered":"2023-03-03T08:03:15.608341+0000","last_clean":"2023-03-03T08:03:15.608341+0000","last_became_active":"2023-03-03T08:03:13.536912+0000","last_became_peered":"2023-03-03T08:03:13.536912+0000","last_unstale":"2023-03-03T08:03:15.608341+0000","last_undegraded":"2023-03-03T08:03:15.608341+0000","last_fullsized":"2023-03-03T08:03:15.608341+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:56:50.934181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608566+0000","last_change":"2023-03-03T08:03:12.972074+0000","last_active":"2023-03-03T08:03:15.608566+0000","last_peered":"2023-03-03T08:03:15.608566+0000","last_clean":"2023-03-03T08:03:15.608566+0000","last_became_active":"2023-03-03T08:03:12.971734+0000","last_became_peered":"2023-03-03T08:03:12.971734+0000","last_unstale":"2023-03-03T08:03:15.608566+0000","last_undegraded":"2023-03-03T08:03:15.608566+0000","last_fullsized":"2023-03-03T08:03:15.608566+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:57:17.100976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518089+0000","last_change":"2023-03-03T08:03:13.540445+0000","last_active":"2023-03-03T08:03:14.518089+0000","last_peered":"2023-03-03T08:03:14.518089+0000","last_clean":"2023-03-03T08:03:14.518089+0000","last_became_active":"2023-03-03T08:03:13.540166+0000","last_became_peered":"2023-03-03T08:03:13.540166+0000","last_unstale":"2023-03-03T08:03:14.518089+0000","last_undegraded":"2023-03-03T08:03:14.518089+0000","last_fullsized":"2023-03-03T08:03:14.518089+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:09.173678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504313+0000","last_change":"2023-03-03T08:03:12.973378+0000","last_active":"2023-03-03T08:03:15.504313+0000","last_peered":"2023-03-03T08:03:15.504313+0000","last_clean":"2023-03-03T08:03:15.504313+0000","last_became_active":"2023-03-03T08:03:12.972932+0000","last_became_peered":"2023-03-03T08:03:12.972932+0000","last_unstale":"2023-03-03T08:03:15.504313+0000","last_undegraded":"2023-03-03T08:03:15.504313+0000","last_fullsized":"2023-03-03T08:03:15.504313+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:51.758546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518449+0000","last_change":"2023-03-03T08:03:13.542448+0000","last_active":"2023-03-03T08:03:14.518449+0000","last_peered":"2023-03-03T08:03:14.518449+0000","last_clean":"2023-03-03T08:03:14.518449+0000","last_became_active":"2023-03-03T08:03:13.541743+0000","last_became_peered":"2023-03-03T08:03:13.541743+0000","last_unstale":"2023-03-03T08:03:14.518449+0000","last_undegraded":"2023-03-03T08:03:14.518449+0000","last_fullsized":"2023-03-03T08:03:14.518449+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:10:15.395992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518280+0000","last_change":"2023-03-03T08:03:12.968898+0000","last_active":"2023-03-03T08:03:14.518280+0000","last_peered":"2023-03-03T08:03:14.518280+0000","last_clean":"2023-03-03T08:03:14.518280+0000","last_became_active":"2023-03-03T08:03:12.968439+0000","last_became_peered":"2023-03-03T08:03:12.968439+0000","last_unstale":"2023-03-03T08:03:14.518280+0000","last_undegraded":"2023-03-03T08:03:14.518280+0000","last_fullsized":"2023-03-03T08:03:14.518280+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:49:22.482147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609353+0000","last_change":"2023-03-03T08:03:13.529057+0000","last_active":"2023-03-03T08:03:15.609353+0000","last_peered":"2023-03-03T08:03:15.609353+0000","last_clean":"2023-03-03T08:03:15.609353+0000","last_became_active":"2023-03-03T08:03:13.528577+0000","last_became_peered":"2023-03-03T08:03:13.528577+0000","last_unstale":"2023-03-03T08:03:15.609353+0000","last_undegraded":"2023-03-03T08:03:15.609353+0000","last_fullsized":"2023-03-03T08:03:15.609353+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:17:38.018066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504230+0000","last_change":"2023-03-03T08:03:12.973948+0000","last_active":"2023-03-03T08:03:15.504230+0000","last_peered":"2023-03-03T08:03:15.504230+0000","last_clean":"2023-03-03T08:03:15.504230+0000","last_became_active":"2023-03-03T08:03:12.973579+0000","last_became_peered":"2023-03-03T08:03:12.973579+0000","last_unstale":"2023-03-03T08:03:15.504230+0000","last_undegraded":"2023-03-03T08:03:15.504230+0000","last_fullsized":"2023-03-03T08:03:15.504230+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:15:56.746437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608915+0000","last_change":"2023-03-03T08:03:13.527986+0000","last_active":"2023-03-03T08:03:15.608915+0000","last_peered":"2023-03-03T08:03:15.608915+0000","last_clean":"2023-03-03T08:03:15.608915+0000","last_became_active":"2023-03-03T08:03:13.527647+0000","last_became_peered":"2023-03-03T08:03:13.527647+0000","last_unstale":"2023-03-03T08:03:15.608915+0000","last_undegraded":"2023-03-03T08:03:15.608915+0000","last_fullsized":"2023-03-03T08:03:15.608915+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:27:11.786506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609109+0000","last_change":"2023-03-03T08:03:12.972891+0000","last_active":"2023-03-03T08:03:15.609109+0000","last_peered":"2023-03-03T08:03:15.609109+0000","last_clean":"2023-03-03T08:03:15.609109+0000","last_became_active":"2023-03-03T08:03:12.972367+0000","last_became_peered":"2023-03-03T08:03:12.972367+0000","last_unstale":"2023-03-03T08:03:15.609109+0000","last_undegraded":"2023-03-03T08:03:15.609109+0000","last_fullsized":"2023-03-03T08:03:15.609109+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:34.969668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898647+0000","last_change":"2023-03-03T08:03:13.538318+0000","last_active":"2023-03-03T08:03:14.898647+0000","last_peered":"2023-03-03T08:03:14.898647+0000","last_clean":"2023-03-03T08:03:14.898647+0000","last_became_active":"2023-03-03T08:03:13.538088+0000","last_became_peered":"2023-03-03T08:03:13.538088+0000","last_unstale":"2023-03-03T08:03:14.898647+0000","last_undegraded":"2023-03-03T08:03:14.898647+0000","last_fullsized":"2023-03-03T08:03:14.898647+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:12.954000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608936+0000","last_change":"2023-03-03T08:03:12.971078+0000","last_active":"2023-03-03T08:03:15.608936+0000","last_peered":"2023-03-03T08:03:15.608936+0000","last_clean":"2023-03-03T08:03:15.608936+0000","last_became_active":"2023-03-03T08:03:12.970784+0000","last_became_peered":"2023-03-03T08:03:12.970784+0000","last_unstale":"2023-03-03T08:03:15.608936+0000","last_undegraded":"2023-03-03T08:03:15.608936+0000","last_fullsized":"2023-03-03T08:03:15.608936+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:07:16.503347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504788+0000","last_change":"2023-03-03T08:03:13.534724+0000","last_active":"2023-03-03T08:03:15.504788+0000","last_peered":"2023-03-03T08:03:15.504788+0000","last_clean":"2023-03-03T08:03:15.504788+0000","last_became_active":"2023-03-03T08:03:13.534419+0000","last_became_peered":"2023-03-03T08:03:13.534419+0000","last_unstale":"2023-03-03T08:03:15.504788+0000","last_undegraded":"2023-03-03T08:03:15.504788+0000","last_fullsized":"2023-03-03T08:03:15.504788+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:34:21.624584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609104+0000","last_change":"2023-03-03T08:03:12.968475+0000","last_active":"2023-03-03T08:03:15.609104+0000","last_peered":"2023-03-03T08:03:15.609104+0000","last_clean":"2023-03-03T08:03:15.609104+0000","last_became_active":"2023-03-03T08:03:12.968141+0000","last_became_peered":"2023-03-03T08:03:12.968141+0000","last_unstale":"2023-03-03T08:03:15.609104+0000","last_undegraded":"2023-03-03T08:03:15.609104+0000","last_fullsized":"2023-03-03T08:03:15.609104+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:03:49.445758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608998+0000","last_change":"2023-03-03T08:03:13.529103+0000","last_active":"2023-03-03T08:03:15.608998+0000","last_peered":"2023-03-03T08:03:15.608998+0000","last_clean":"2023-03-03T08:03:15.608998+0000","last_became_active":"2023-03-03T08:03:13.528804+0000","last_became_peered":"2023-03-03T08:03:13.528804+0000","last_unstale":"2023-03-03T08:03:15.608998+0000","last_undegraded":"2023-03-03T08:03:15.608998+0000","last_fullsized":"2023-03-03T08:03:15.608998+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:26:45.045000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609004+0000","last_change":"2023-03-03T08:03:12.968634+0000","last_active":"2023-03-03T08:03:15.609004+0000","last_peered":"2023-03-03T08:03:15.609004+0000","last_clean":"2023-03-03T08:03:15.609004+0000","last_became_active":"2023-03-03T08:03:12.967878+0000","last_became_peered":"2023-03-03T08:03:12.967878+0000","last_unstale":"2023-03-03T08:03:15.609004+0000","last_undegraded":"2023-03-03T08:03:15.609004+0000","last_fullsized":"2023-03-03T08:03:15.609004+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:32:21.108432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608520+0000","last_change":"2023-03-03T08:03:13.529496+0000","last_active":"2023-03-03T08:03:15.608520+0000","last_peered":"2023-03-03T08:03:15.608520+0000","last_clean":"2023-03-03T08:03:15.608520+0000","last_became_active":"2023-03-03T08:03:13.529086+0000","last_became_peered":"2023-03-03T08:03:13.529086+0000","last_unstale":"2023-03-03T08:03:15.608520+0000","last_undegraded":"2023-03-03T08:03:15.608520+0000","last_fullsized":"2023-03-03T08:03:15.608520+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:12:50.605693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523256+0000","last_change":"2023-03-03T08:03:12.973745+0000","last_active":"2023-03-03T08:03:14.523256+0000","last_peered":"2023-03-03T08:03:14.523256+0000","last_clean":"2023-03-03T08:03:14.523256+0000","last_became_active":"2023-03-03T08:03:12.973471+0000","last_became_peered":"2023-03-03T08:03:12.973471+0000","last_unstale":"2023-03-03T08:03:14.523256+0000","last_undegraded":"2023-03-03T08:03:14.523256+0000","last_fullsized":"2023-03-03T08:03:14.523256+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:39:17.607622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518229+0000","last_change":"2023-03-03T08:03:13.540474+0000","last_active":"2023-03-03T08:03:14.518229+0000","last_peered":"2023-03-03T08:03:14.518229+0000","last_clean":"2023-03-03T08:03:14.518229+0000","last_became_active":"2023-03-03T08:03:13.540230+0000","last_became_peered":"2023-03-03T08:03:13.540230+0000","last_unstale":"2023-03-03T08:03:14.518229+0000","last_undegraded":"2023-03-03T08:03:14.518229+0000","last_fullsized":"2023-03-03T08:03:14.518229+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:16:44.241182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.523306+0000","last_change":"2023-03-03T08:03:12.977628+0000","last_active":"2023-03-03T08:03:14.523306+0000","last_peered":"2023-03-03T08:03:14.523306+0000","last_clean":"2023-03-03T08:03:14.523306+0000","last_became_active":"2023-03-03T08:03:12.976889+0000","last_became_peered":"2023-03-03T08:03:12.976889+0000","last_unstale":"2023-03-03T08:03:14.523306+0000","last_undegraded":"2023-03-03T08:03:14.523306+0000","last_fullsized":"2023-03-03T08:03:14.523306+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:44:26.766023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504326+0000","last_change":"2023-03-03T08:03:13.528119+0000","last_active":"2023-03-03T08:03:15.504326+0000","last_peered":"2023-03-03T08:03:15.504326+0000","last_clean":"2023-03-03T08:03:15.504326+0000","last_became_active":"2023-03-03T08:03:13.527830+0000","last_became_peered":"2023-03-03T08:03:13.527830+0000","last_unstale":"2023-03-03T08:03:15.504326+0000","last_undegraded":"2023-03-03T08:03:15.504326+0000","last_fullsized":"2023-03-03T08:03:15.504326+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:09.173678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608632+0000","last_change":"2023-03-03T08:03:12.970900+0000","last_active":"2023-03-03T08:03:15.608632+0000","last_peered":"2023-03-03T08:03:15.608632+0000","last_clean":"2023-03-03T08:03:15.608632+0000","last_became_active":"2023-03-03T08:03:12.970633+0000","last_became_peered":"2023-03-03T08:03:12.970633+0000","last_unstale":"2023-03-03T08:03:15.608632+0000","last_undegraded":"2023-03-03T08:03:15.608632+0000","last_fullsized":"2023-03-03T08:03:15.608632+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:56:15.442953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898833+0000","last_change":"2023-03-03T08:03:13.544190+0000","last_active":"2023-03-03T08:03:14.898833+0000","last_peered":"2023-03-03T08:03:14.898833+0000","last_clean":"2023-03-03T08:03:14.898833+0000","last_became_active":"2023-03-03T08:03:13.543942+0000","last_became_peered":"2023-03-03T08:03:13.543942+0000","last_unstale":"2023-03-03T08:03:14.898833+0000","last_undegraded":"2023-03-03T08:03:14.898833+0000","last_fullsized":"2023-03-03T08:03:14.898833+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:17:36.836414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504233+0000","last_change":"2023-03-03T08:03:12.971112+0000","last_active":"2023-03-03T08:03:15.504233+0000","last_peered":"2023-03-03T08:03:15.504233+0000","last_clean":"2023-03-03T08:03:15.504233+0000","last_became_active":"2023-03-03T08:03:12.970796+0000","last_became_peered":"2023-03-03T08:03:12.970796+0000","last_unstale":"2023-03-03T08:03:15.504233+0000","last_undegraded":"2023-03-03T08:03:15.504233+0000","last_fullsized":"2023-03-03T08:03:15.504233+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:35:13.046397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609598+0000","last_change":"2023-03-03T08:03:13.529245+0000","last_active":"2023-03-03T08:03:15.609598+0000","last_peered":"2023-03-03T08:03:15.609598+0000","last_clean":"2023-03-03T08:03:15.609598+0000","last_became_active":"2023-03-03T08:03:13.528965+0000","last_became_peered":"2023-03-03T08:03:13.528965+0000","last_unstale":"2023-03-03T08:03:15.609598+0000","last_undegraded":"2023-03-03T08:03:15.609598+0000","last_fullsized":"2023-03-03T08:03:15.609598+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:05:48.872734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"110'6","reported_seq":17,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518662+0000","last_change":"2023-03-03T08:03:12.971469+0000","last_active":"2023-03-03T08:03:14.518662+0000","last_peered":"2023-03-03T08:03:14.518662+0000","last_clean":"2023-03-03T08:03:14.518662+0000","last_became_active":"2023-03-03T08:03:12.971204+0000","last_became_peered":"2023-03-03T08:03:12.971204+0000","last_unstale":"2023-03-03T08:03:14.518662+0000","last_undegraded":"2023-03-03T08:03:14.518662+0000","last_fullsized":"2023-03-03T08:03:14.518662+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:32:08.429902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.529120+0000","last_change":"2023-03-03T07:55:40.502916+0000","last_active":"2023-03-03T08:03:13.529120+0000","last_peered":"2023-03-03T08:03:13.529120+0000","last_clean":"2023-03-03T08:03:13.529120+0000","last_became_active":"2023-03-03T07:55:40.502495+0000","last_became_peered":"2023-03-03T07:55:40.502495+0000","last_unstale":"2023-03-03T08:03:13.529120+0000","last_undegraded":"2023-03-03T08:03:13.529120+0000","last_fullsized":"2023-03-03T08:03:13.529120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518573+0000","last_change":"2023-03-03T08:03:13.541671+0000","last_active":"2023-03-03T08:03:14.518573+0000","last_peered":"2023-03-03T08:03:14.518573+0000","last_clean":"2023-03-03T08:03:14.518573+0000","last_became_active":"2023-03-03T08:03:13.541383+0000","last_became_peered":"2023-03-03T08:03:13.541383+0000","last_unstale":"2023-03-03T08:03:14.518573+0000","last_undegraded":"2023-03-03T08:03:14.518573+0000","last_fullsized":"2023-03-03T08:03:14.518573+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:44:39.592199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"110'3","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609634+0000","last_change":"2023-03-03T08:03:12.975920+0000","last_active":"2023-03-03T08:03:15.609634+0000","last_peered":"2023-03-03T08:03:15.609634+0000","last_clean":"2023-03-03T08:03:15.609634+0000","last_became_active":"2023-03-03T08:03:12.975541+0000","last_became_peered":"2023-03-03T08:03:12.975541+0000","last_unstale":"2023-03-03T08:03:15.609634+0000","last_undegraded":"2023-03-03T08:03:15.609634+0000","last_fullsized":"2023-03-03T08:03:15.609634+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:48:02.934456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.515328+0000","last_change":"2023-03-03T07:55:40.500523+0000","last_active":"2023-03-03T08:03:13.515328+0000","last_peered":"2023-03-03T08:03:13.515328+0000","last_clean":"2023-03-03T08:03:13.515328+0000","last_became_active":"2023-03-03T07:55:40.500202+0000","last_became_peered":"2023-03-03T07:55:40.500202+0000","last_unstale":"2023-03-03T08:03:13.515328+0000","last_undegraded":"2023-03-03T08:03:13.515328+0000","last_fullsized":"2023-03-03T08:03:13.515328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609538+0000","last_change":"2023-03-03T08:03:13.542375+0000","last_active":"2023-03-03T08:03:15.609538+0000","last_peered":"2023-03-03T08:03:15.609538+0000","last_clean":"2023-03-03T08:03:15.609538+0000","last_became_active":"2023-03-03T08:03:13.542095+0000","last_became_peered":"2023-03-03T08:03:13.542095+0000","last_unstale":"2023-03-03T08:03:15.609538+0000","last_undegraded":"2023-03-03T08:03:15.609538+0000","last_fullsized":"2023-03-03T08:03:15.609538+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:24:32.147518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.609316+0000","last_change":"2023-03-03T08:03:12.975337+0000","last_active":"2023-03-03T08:03:15.609316+0000","last_peered":"2023-03-03T08:03:15.609316+0000","last_clean":"2023-03-03T08:03:15.609316+0000","last_became_active":"2023-03-03T08:03:12.975038+0000","last_became_peered":"2023-03-03T08:03:12.975038+0000","last_unstale":"2023-03-03T08:03:15.609316+0000","last_undegraded":"2023-03-03T08:03:15.609316+0000","last_fullsized":"2023-03-03T08:03:15.609316+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:31.956827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.515322+0000","last_change":"2023-03-03T07:55:41.517978+0000","last_active":"2023-03-03T08:03:13.515322+0000","last_peered":"2023-03-03T08:03:13.515322+0000","last_clean":"2023-03-03T08:03:13.515322+0000","last_became_active":"2023-03-03T07:55:41.517664+0000","last_became_peered":"2023-03-03T07:55:41.517664+0000","last_unstale":"2023-03-03T08:03:13.515322+0000","last_undegraded":"2023-03-03T08:03:13.515322+0000","last_fullsized":"2023-03-03T08:03:13.515322+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608583+0000","last_change":"2023-03-03T08:03:13.546953+0000","last_active":"2023-03-03T08:03:15.608583+0000","last_peered":"2023-03-03T08:03:15.608583+0000","last_clean":"2023-03-03T08:03:15.608583+0000","last_became_active":"2023-03-03T08:03:13.546621+0000","last_became_peered":"2023-03-03T08:03:13.546621+0000","last_unstale":"2023-03-03T08:03:15.608583+0000","last_undegraded":"2023-03-03T08:03:15.608583+0000","last_fullsized":"2023-03-03T08:03:15.608583+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:28:07.879188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518442+0000","last_change":"2023-03-03T08:03:12.971334+0000","last_active":"2023-03-03T08:03:14.518442+0000","last_peered":"2023-03-03T08:03:14.518442+0000","last_clean":"2023-03-03T08:03:14.518442+0000","last_became_active":"2023-03-03T08:03:12.971083+0000","last_became_peered":"2023-03-03T08:03:12.971083+0000","last_unstale":"2023-03-03T08:03:14.518442+0000","last_undegraded":"2023-03-03T08:03:14.518442+0000","last_fullsized":"2023-03-03T08:03:14.518442+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:17:35.833151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.516906+0000","last_change":"2023-03-03T07:55:40.499063+0000","last_active":"2023-03-03T08:03:13.516906+0000","last_peered":"2023-03-03T08:03:13.516906+0000","last_clean":"2023-03-03T08:03:13.516906+0000","last_became_active":"2023-03-03T07:55:40.498654+0000","last_became_peered":"2023-03-03T07:55:40.498654+0000","last_unstale":"2023-03-03T08:03:13.516906+0000","last_undegraded":"2023-03-03T08:03:13.516906+0000","last_fullsized":"2023-03-03T08:03:13.516906+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.504823+0000","last_change":"2023-03-03T08:03:13.532910+0000","last_active":"2023-03-03T08:03:15.504823+0000","last_peered":"2023-03-03T08:03:15.504823+0000","last_clean":"2023-03-03T08:03:15.504823+0000","last_became_active":"2023-03-03T08:03:13.532587+0000","last_became_peered":"2023-03-03T08:03:13.532587+0000","last_unstale":"2023-03-03T08:03:15.504823+0000","last_undegraded":"2023-03-03T08:03:15.504823+0000","last_fullsized":"2023-03-03T08:03:15.504823+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:06:03.384487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"110'8","reported_seq":19,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518102+0000","last_change":"2023-03-03T08:03:12.972059+0000","last_active":"2023-03-03T08:03:14.518102+0000","last_peered":"2023-03-03T08:03:14.518102+0000","last_clean":"2023-03-03T08:03:14.518102+0000","last_became_active":"2023-03-03T08:03:12.971734+0000","last_became_peered":"2023-03-03T08:03:12.971734+0000","last_unstale":"2023-03-03T08:03:14.518102+0000","last_undegraded":"2023-03-03T08:03:14.518102+0000","last_fullsized":"2023-03-03T08:03:14.518102+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:29:54.191181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":284,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.517085+0000","last_change":"2023-03-03T07:55:33.490983+0000","last_active":"2023-03-03T08:03:13.517085+0000","last_peered":"2023-03-03T08:03:13.517085+0000","last_clean":"2023-03-03T08:03:13.517085+0000","last_became_active":"2023-03-03T07:55:33.490379+0000","last_became_peered":"2023-03-03T07:55:33.490379+0000","last_unstale":"2023-03-03T08:03:13.517085+0000","last_undegraded":"2023-03-03T08:03:13.517085+0000","last_fullsized":"2023-03-03T08:03:13.517085+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:32.437128+0000","last_clean_scrub_stamp":"2023-03-03T07:55:32.437128+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:16:01.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":88,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.517798+0000","last_change":"2023-03-03T07:55:40.499029+0000","last_active":"2023-03-03T08:03:13.517798+0000","last_peered":"2023-03-03T08:03:13.517798+0000","last_clean":"2023-03-03T08:03:13.517798+0000","last_became_active":"2023-03-03T07:55:40.498469+0000","last_became_peered":"2023-03-03T07:55:40.498469+0000","last_unstale":"2023-03-03T08:03:13.517798+0000","last_undegraded":"2023-03-03T08:03:13.517798+0000","last_fullsized":"2023-03-03T08:03:13.517798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:16:35.804050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.898497+0000","last_change":"2023-03-03T08:03:13.543981+0000","last_active":"2023-03-03T08:03:14.898497+0000","last_peered":"2023-03-03T08:03:14.898497+0000","last_clean":"2023-03-03T08:03:14.898497+0000","last_became_active":"2023-03-03T08:03:13.543705+0000","last_became_peered":"2023-03-03T08:03:13.543705+0000","last_unstale":"2023-03-03T08:03:14.898497+0000","last_undegraded":"2023-03-03T08:03:14.898497+0000","last_fullsized":"2023-03-03T08:03:14.898497+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:29:55.194445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518507+0000","last_change":"2023-03-03T08:03:12.975860+0000","last_active":"2023-03-03T08:03:14.518507+0000","last_peered":"2023-03-03T08:03:14.518507+0000","last_clean":"2023-03-03T08:03:14.518507+0000","last_became_active":"2023-03-03T08:03:12.975589+0000","last_became_peered":"2023-03-03T08:03:12.975589+0000","last_unstale":"2023-03-03T08:03:14.518507+0000","last_undegraded":"2023-03-03T08:03:14.518507+0000","last_fullsized":"2023-03-03T08:03:14.518507+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:41:51.758546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":83,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.514684+0000","last_change":"2023-03-03T07:55:40.684421+0000","last_active":"2023-03-03T08:03:13.514684+0000","last_peered":"2023-03-03T08:03:13.514684+0000","last_clean":"2023-03-03T08:03:13.514684+0000","last_became_active":"2023-03-03T07:55:40.684040+0000","last_became_peered":"2023-03-03T07:55:40.684040+0000","last_unstale":"2023-03-03T08:03:13.514684+0000","last_undegraded":"2023-03-03T08:03:13.514684+0000","last_fullsized":"2023-03-03T08:03:13.514684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:34:54.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608226+0000","last_change":"2023-03-03T08:03:13.534135+0000","last_active":"2023-03-03T08:03:15.608226+0000","last_peered":"2023-03-03T08:03:15.608226+0000","last_clean":"2023-03-03T08:03:15.608226+0000","last_became_active":"2023-03-03T08:03:13.533898+0000","last_became_peered":"2023-03-03T08:03:13.533898+0000","last_unstale":"2023-03-03T08:03:15.608226+0000","last_undegraded":"2023-03-03T08:03:15.608226+0000","last_fullsized":"2023-03-03T08:03:15.608226+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:10:09.472093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:14.518195+0000","last_change":"2023-03-03T08:03:12.968683+0000","last_active":"2023-03-03T08:03:14.518195+0000","last_peered":"2023-03-03T08:03:14.518195+0000","last_clean":"2023-03-03T08:03:14.518195+0000","last_became_active":"2023-03-03T08:03:12.967877+0000","last_became_peered":"2023-03-03T08:03:12.967877+0000","last_unstale":"2023-03-03T08:03:14.518195+0000","last_undegraded":"2023-03-03T08:03:14.518195+0000","last_fullsized":"2023-03-03T08:03:14.518195+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:59:01.635888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":88,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.517732+0000","last_change":"2023-03-03T07:55:40.499389+0000","last_active":"2023-03-03T08:03:13.517732+0000","last_peered":"2023-03-03T08:03:13.517732+0000","last_clean":"2023-03-03T08:03:13.517732+0000","last_became_active":"2023-03-03T07:55:40.498913+0000","last_became_peered":"2023-03-03T07:55:40.498913+0000","last_unstale":"2023-03-03T08:03:13.517732+0000","last_undegraded":"2023-03-03T08:03:13.517732+0000","last_fullsized":"2023-03-03T08:03:13.517732+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:48:17.703619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608733+0000","last_change":"2023-03-03T08:03:13.549078+0000","last_active":"2023-03-03T08:03:15.608733+0000","last_peered":"2023-03-03T08:03:15.608733+0000","last_clean":"2023-03-03T08:03:15.608733+0000","last_became_active":"2023-03-03T08:03:13.548802+0000","last_became_peered":"2023-03-03T08:03:13.548802+0000","last_unstale":"2023-03-03T08:03:15.608733+0000","last_undegraded":"2023-03-03T08:03:15.608733+0000","last_fullsized":"2023-03-03T08:03:15.608733+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:12.945386+0000","last_clean_scrub_stamp":"2023-03-03T08:03:12.945386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:56:53.272507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-03-03T08:03:15.608982+0000","last_change":"2023-03-03T08:03:12.971765+0000","last_active":"2023-03-03T08:03:15.608982+0000","last_peered":"2023-03-03T08:03:15.608982+0000","last_clean":"2023-03-03T08:03:15.608982+0000","last_became_active":"2023-03-03T08:03:12.971428+0000","last_became_peered":"2023-03-03T08:03:12.971428+0000","last_unstale":"2023-03-03T08:03:15.608982+0000","last_undegraded":"2023-03-03T08:03:15.608982+0000","last_fullsized":"2023-03-03T08:03:15.608982+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:03:11.942123+0000","last_clean_scrub_stamp":"2023-03-03T08:03:11.942123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:52:05.550981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-03-03T08:03:13.514807+0000","last_change":"2023-03-03T07:55:41.517299+0000","last_active":"2023-03-03T08:03:13.514807+0000","last_peered":"2023-03-03T08:03:13.514807+0000","last_clean":"2023-03-03T08:03:13.514807+0000","last_became_active":"2023-03-03T07:55:41.516979+0000","last_became_peered":"2023-03-03T07:55:41.516979+0000","last_unstale":"2023-03-03T08:03:13.514807+0000","last_undegraded":"2023-03-03T08:03:13.514807+0000","last_fullsized":"2023-03-03T08:03:13.514807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T07:55:39.477269+0000","last_clean_scrub_stamp":"2023-03-03T07:55:39.477269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:31:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574944,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":16788,"kb_used_data":1932,"kb_used_omap":0,"kb_used_meta":14848,"kb_avail":94355052,"statfs":{"total":96636764160,"available":96619573248,"internally_reserved":0,"allocated":1978368,"data_stored":1225680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":15204352},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:02:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66600000000000004,"15min":0.69599999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.88700000000000001,"5min":0.91700000000000004,"15min":1.546},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.66800000000000004,"15min":0.71199999999999997},"min":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.92900000000000005,"5min":0.96499999999999997,"15min":1.702},"last":0.64200000000000002}]},{"osd":1,"last update":"Fri Mar 3 08:02:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.71999999999999997,"15min":0.99199999999999999},"min":{"1min":0.48899999999999999,"5min":0.434,"15min":0.41699999999999998},"max":{"1min":0.94699999999999995,"5min":1.111,"15min":6.6829999999999998},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68300000000000005,"15min":0.746},"min":{"1min":0.47199999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.87,"5min":1.1890000000000001,"15min":1.944},"last":0.89200000000000002}]},{"osd":2,"last update":"Fri Mar 3 08:02:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72199999999999998,"15min":0.71499999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.87,"5min":1.224,"15min":1.252},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.74099999999999999,"15min":0.77900000000000003},"min":{"1min":0.496,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0229999999999999,"5min":1.2490000000000001,"15min":1.329},"last":1.0129999999999999}]},{"osd":3,"last update":"Fri Mar 3 08:02:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91500000000000004,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.73099999999999998,"15min":0.77500000000000002},"min":{"1min":0.52500000000000002,"5min":0.51000000000000001,"15min":0.46000000000000002},"max":{"1min":0.97299999999999998,"5min":1.014,"15min":1.294},"last":1.04}]},{"osd":4,"last update":"Fri Mar 3 08:02:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60999999999999999,"15min":0.64600000000000002},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41099999999999998},"max":{"1min":0.93300000000000005,"5min":1.0049999999999999,"15min":1.077},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.68200000000000005,"15min":0.64400000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.40899999999999997},"max":{"1min":0.95899999999999996,"5min":1.081,"15min":1.081},"last":0.77400000000000002}]},{"osd":5,"last update":"Fri Mar 3 08:02:40 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.71499999999999997,"15min":0.77500000000000002},"min":{"1min":0.50700000000000001,"5min":0.40699999999999997,"15min":0.39100000000000001},"max":{"1min":1.0009999999999999,"5min":1.036,"15min":1.3819999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.77300000000000002,"15min":0.81000000000000005},"min":{"1min":0.57399999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0009999999999999,"5min":1.2190000000000001,"15min":1.349},"last":0.70299999999999996}]},{"osd":6,"last update":"Fri Mar 3 08:02:40 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.79300000000000004,"15min":0.82199999999999995},"min":{"1min":0.56200000000000006,"5min":0.56000000000000005,"15min":0.54100000000000004},"max":{"1min":1.081,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.999},{"interface":"front","average":{"1min":0.749,"5min":0.79500000000000004,"15min":0.81100000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0169999999999999,"5min":1.135,"15min":1.194},"last":0.91100000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574945,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11276,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":9920,"kb_avail":94360564,"statfs":{"total":96636764160,"available":96625217536,"internally_reserved":0,"allocated":1380352,"data_stored":640752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10158080},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73499999999999999,"15min":0.69999999999999996},"min":{"1min":0.42899999999999999,"5min":0.42299999999999999,"15min":0.40100000000000002},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.4550000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.878,"5min":0.751,"15min":0.71399999999999997},"min":{"1min":0.46200000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.2169999999999996,"5min":5.2169999999999996,"15min":5.2169999999999996},"last":0.67700000000000005}]},{"osd":1,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.71699999999999997,"15min":0.69699999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":3.323},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.746,"15min":0.73399999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.48199999999999998},"max":{"1min":2.105,"5min":2.105,"15min":2.105},"last":0.76500000000000001}]},{"osd":2,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.76300000000000001,"15min":0.74099999999999999},"min":{"1min":0.51000000000000001,"5min":0.5,"15min":0.5},"max":{"1min":3.956,"5min":3.956,"15min":3.956},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.54900000000000004,"5min":0.495,"15min":0.495},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.80200000000000005,"15min":0.80100000000000005},"min":{"1min":0.53500000000000003,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":4.2270000000000003,"5min":4.2270000000000003,"15min":4.8959999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.80700000000000005,"15min":0.78400000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":4.0049999999999999,"5min":4.0049999999999999,"15min":4.0049999999999999},"last":0.75700000000000001}]},{"osd":4,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.70699999999999996,"15min":0.86799999999999999},"min":{"1min":0.45000000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.1349999999999998,"5min":2.1349999999999998,"15min":4.0099999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.75,"15min":0.89000000000000001},"min":{"1min":0.504,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":5.3029999999999999,"5min":5.3029999999999999,"15min":5.3029999999999999},"last":0.623}]},{"osd":5,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.70799999999999996,"15min":0.76400000000000001},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.347,"5min":2.347,"15min":2.347},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.79100000000000004,"15min":0.81999999999999995},"min":{"1min":0.501,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.58999999999999997}]},{"osd":7,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.83299999999999996,"15min":0.84499999999999997},"min":{"1min":0.63,"5min":0.54100000000000004,"15min":0.51900000000000002},"max":{"1min":5.2629999999999999,"5min":5.2629999999999999,"15min":5.2629999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.78900000000000003,"15min":0.85299999999999998},"min":{"1min":0.50600000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.532,"5min":2.532,"15min":2.532},"last":0.70999999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11664,"kb_used_data":1352,"kb_used_omap":0,"kb_used_meta":10304,"kb_avail":94360176,"statfs":{"total":96636764160,"available":96624820224,"internally_reserved":0,"allocated":1384448,"data_stored":641227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10551296},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46899999999999997,"15min":0.498},"min":{"1min":0.32300000000000001,"5min":0.308,"15min":0.307},"max":{"1min":0.69399999999999995,"5min":1.0289999999999999,"15min":1.4970000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51300000000000001,"15min":0.60999999999999999},"min":{"1min":0.34799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.68500000000000005,"5min":0.90200000000000002,"15min":3.972},"last":0.33500000000000002}]},{"osd":2,"last update":"Fri Mar 3 08:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52600000000000002,"15min":0.85299999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.313},"max":{"1min":0.64200000000000002,"5min":0.78900000000000003,"15min":11.897},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59699999999999998,"15min":0.93200000000000005},"min":{"1min":0.46200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78700000000000003,"5min":0.85199999999999998,"15min":11.804},"last":0.503}]},{"osd":3,"last update":"Fri Mar 3 08:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58999999999999997,"15min":0.92300000000000004},"min":{"1min":0.42299999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.752,"5min":0.89500000000000002,"15min":11.843999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55800000000000005,"15min":0.68899999999999995},"min":{"1min":0.40799999999999997,"5min":0.40000000000000002,"15min":0.36699999999999999},"max":{"1min":0.80800000000000005,"5min":0.96399999999999997,"15min":3.8740000000000001},"last":0.54200000000000004}]},{"osd":4,"last update":"Fri Mar 3 08:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.753,"15min":0.83399999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.39900000000000002},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.613},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.81299999999999994,"15min":0.84899999999999998},"min":{"1min":0.624,"5min":0.624,"15min":0.53800000000000003},"max":{"1min":1.0429999999999999,"5min":1.127,"15min":1.4630000000000001},"last":0.99099999999999999}]},{"osd":5,"last update":"Fri Mar 3 08:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.753,"15min":0.88100000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0189999999999999,"5min":1.0620000000000001,"15min":3.8300000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.749,"15min":0.83999999999999997},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.47499999999999998},"max":{"1min":0.96199999999999997,"5min":1.008,"15min":1.5469999999999999},"last":0.82199999999999995}]},{"osd":6,"last update":"Fri Mar 3 08:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.81100000000000005,"15min":0.94999999999999996},"min":{"1min":0.64300000000000002,"5min":0.62,"15min":0.62},"max":{"1min":1.028,"5min":1.173,"15min":3.907},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.80200000000000005,"15min":0.90400000000000003},"min":{"1min":0.59799999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1779999999999999,"5min":1.1819999999999999,"15min":1.5820000000000001},"last":0.876}]},{"osd":7,"last update":"Fri Mar 3 08:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.79300000000000004,"15min":0.85899999999999999},"min":{"1min":0.60899999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.123,"5min":1.417,"15min":1.419},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.80400000000000005,"15min":0.95799999999999996},"min":{"1min":0.58399999999999996,"5min":0.58199999999999996,"15min":0.57099999999999995},"max":{"1min":1.153,"5min":1.3560000000000001,"15min":3.9399999999999999},"last":0.89100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574944,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":16592,"kb_used_data":1928,"kb_used_omap":0,"kb_used_meta":14656,"kb_avail":94355248,"statfs":{"total":96636764160,"available":96619773952,"internally_reserved":0,"allocated":1974272,"data_stored":1230576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":15007744},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 3 08:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44,"15min":0.46200000000000002},"min":{"1min":0.32900000000000001,"5min":0.30399999999999999,"15min":0.23999999999999999},"max":{"1min":0.63,"5min":0.65000000000000002,"15min":0.89300000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.35599999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.75900000000000001,"5min":0.81999999999999995,"15min":0.871},"last":0.46100000000000002}]},{"osd":2,"last update":"Fri Mar 3 08:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51100000000000001,"15min":0.54100000000000004},"min":{"1min":0.39200000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.66000000000000003,"5min":0.77400000000000002,"15min":0.79100000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.41499999999999998,"5min":0.39100000000000001,"15min":0.32800000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":1.1020000000000001},"last":0.50800000000000001}]},{"osd":3,"last update":"Fri Mar 3 08:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53300000000000003,"15min":0.54700000000000004},"min":{"1min":0.371,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70099999999999996,"5min":0.71499999999999997,"15min":0.98499999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58899999999999997,"15min":0.627},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.71299999999999997,"5min":0.83499999999999996,"15min":1.2450000000000001},"last":0.55900000000000005}]},{"osd":4,"last update":"Fri Mar 3 08:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.78700000000000003,"15min":0.83099999999999996},"min":{"1min":0.61299999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.1120000000000001,"5min":1.242,"15min":1.5189999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77300000000000002,"15min":0.81100000000000005},"min":{"1min":0.53800000000000003,"5min":0.505,"15min":0.505},"max":{"1min":0.999,"5min":1.218,"15min":1.605},"last":0.67900000000000005}]},{"osd":5,"last update":"Fri Mar 3 08:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.77700000000000002,"15min":0.81200000000000006},"min":{"1min":0.67600000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.93899999999999995,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.78800000000000003,"15min":0.81499999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.056,"5min":1.2070000000000001,"15min":1.286},"last":0.81299999999999994}]},{"osd":6,"last update":"Fri Mar 3 08:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79300000000000004,"15min":0.86099999999999999},"min":{"1min":0.625,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.95399999999999996,"5min":1.2470000000000001,"15min":1.623},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.79000000000000004,"15min":0.84699999999999998},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1319999999999999,"5min":1.137,"15min":1.5780000000000001},"last":0.83199999999999996}]},{"osd":7,"last update":"Fri Mar 3 08:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.80900000000000005,"15min":0.86499999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.57099999999999995},"max":{"1min":1.0569999999999999,"5min":1.1020000000000001,"15min":1.6479999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.79900000000000004,"15min":0.84699999999999998},"min":{"1min":0.59699999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1,"5min":1.1719999999999999,"15min":1.5509999999999999},"last":0.79800000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574945,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11196,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":9856,"kb_avail":94360644,"statfs":{"total":96636764160,"available":96625299456,"internally_reserved":0,"allocated":1363968,"data_stored":633533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10092544},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.432,"15min":0.76100000000000001},"min":{"1min":0.32400000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71199999999999997,"5min":0.78900000000000003,"15min":5.7889999999999997},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.5,"15min":0.67400000000000004},"min":{"1min":0.40699999999999997,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59999999999999998,"5min":0.93700000000000006,"15min":2.6309999999999998},"last":0.434}]},{"osd":1,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51000000000000001,"15min":0.69199999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67100000000000004,"5min":0.90400000000000003,"15min":2.5209999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58699999999999997,"15min":0.90200000000000002},"min":{"1min":0.438,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.745,"5min":0.98399999999999999,"15min":5.8470000000000004},"last":0.53000000000000003}]},{"osd":3,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56299999999999994,"15min":1.0549999999999999},"min":{"1min":0.38700000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.71099999999999997,"5min":1.012,"15min":12.961},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.57599999999999996,"15min":0.92000000000000004},"min":{"1min":0.47999999999999998,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.78400000000000003,"5min":0.92100000000000004,"15min":6.1449999999999996},"last":0.497}]},{"osd":4,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.79700000000000004,"15min":1.155},"min":{"1min":0.51400000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.89200000000000002,"5min":1.468,"15min":6.484},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.83299999999999996,"15min":1.147},"min":{"1min":0.623,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.032,"5min":1.1759999999999999,"15min":6.5510000000000002},"last":0.67400000000000004}]},{"osd":5,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.755,"15min":1.073},"min":{"1min":0.56699999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.88800000000000001,"5min":1.304,"15min":5.7119999999999997},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.79100000000000004,"15min":1.1240000000000001},"min":{"1min":0.57399999999999995,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0049999999999999,"5min":1.1499999999999999,"15min":6.6369999999999996},"last":0.51600000000000001}]},{"osd":6,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.82799999999999996,"15min":1.1339999999999999},"min":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.0129999999999999,"5min":1.393,"15min":6.625},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.83099999999999996,"15min":1.1559999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":0.97499999999999998,"5min":1.347,"15min":5.7539999999999996},"last":0.752}]},{"osd":7,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.78100000000000003,"15min":1.1160000000000001},"min":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.92500000000000004,"5min":1.0980000000000001,"15min":6.7009999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.81000000000000005,"15min":1.1100000000000001},"min":{"1min":0.59999999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":0.96399999999999997,"5min":1.107,"15min":6.6840000000000002},"last":0.57699999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574943,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15708,"kb_used_data":1364,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":94356132,"statfs":{"total":96636764160,"available":96620679168,"internally_reserved":0,"allocated":1396736,"data_stored":642787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.47399999999999998,"15min":0.748},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.7050000000000001,"5min":4.7050000000000001,"15min":9.5820000000000007},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.55000000000000004,"15min":0.82099999999999995},"min":{"1min":0.34300000000000003,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":9.7840000000000007},"last":0.57899999999999996}]},{"osd":1,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.504,"15min":0.51900000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.58799999999999997,"15min":0.89400000000000002},"min":{"1min":0.36899999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":4.9199999999999999,"5min":4.9199999999999999,"15min":9.702},"last":0.52500000000000002}]},{"osd":2,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.61699999999999999,"15min":0.85999999999999999},"min":{"1min":0.42499999999999999,"5min":0.41499999999999998,"15min":0.36499999999999999},"max":{"1min":3.5489999999999999,"5min":3.5489999999999999,"15min":8.3190000000000008},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.61399999999999999,"15min":0.92200000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.41899999999999998},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":9.734},"last":0.55100000000000005}]},{"osd":4,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.80000000000000004,"15min":1.0800000000000001},"min":{"1min":0.61199999999999999,"5min":0.54400000000000004,"15min":0.51100000000000001},"max":{"1min":4.2519999999999998,"5min":4.2519999999999998,"15min":9.8629999999999995},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.79800000000000004,"15min":0.83999999999999997},"min":{"1min":0.61199999999999999,"5min":0.55700000000000005,"15min":0.55100000000000005},"max":{"1min":3.5110000000000001,"5min":3.5110000000000001,"15min":3.5110000000000001},"last":0.70999999999999996}]},{"osd":5,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.82099999999999995,"15min":0.83299999999999996},"min":{"1min":0.65400000000000003,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.5880000000000001,"5min":3.5880000000000001,"15min":3.5880000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.79900000000000004,"15min":1.1200000000000001},"min":{"1min":0.628,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.7040000000000002,"5min":2.7040000000000002,"15min":9.6600000000000001},"last":0.71999999999999997}]},{"osd":6,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":0.84599999999999997,"15min":1.1459999999999999},"min":{"1min":0.60699999999999998,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":4.423,"5min":4.423,"15min":9.8190000000000008},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.79600000000000004,"15min":0.85099999999999998},"min":{"1min":0.626,"5min":0.60299999999999998,"15min":0.51800000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.609},"last":0.88600000000000001}]},{"osd":7,"last update":"Fri Mar 3 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.81299999999999994,"15min":1.1619999999999999},"min":{"1min":0.67000000000000004,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.226,"5min":2.226,"15min":9.625},"last":0.879},{"interface":"front","average":{"1min":1.0609999999999999,"5min":0.85799999999999998,"15min":1.153},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":4.3259999999999996,"5min":4.3259999999999996,"15min":9.8949999999999996},"last":0.80500000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574943,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11928,"kb_used_data":1360,"kb_used_omap":0,"kb_used_meta":10560,"kb_avail":94359912,"statfs":{"total":96636764160,"available":96624549888,"internally_reserved":0,"allocated":1392640,"data_stored":642753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10813440},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71399999999999997,"15min":0.91600000000000004},"min":{"1min":0.52500000000000002,"5min":0.39700000000000002,"15min":0.35599999999999998},"max":{"1min":0.94099999999999995,"5min":1.0049999999999999,"15min":7.1799999999999997},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.70499999999999996,"15min":0.67500000000000004},"min":{"1min":0.47999999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91500000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.70199999999999996}]},{"osd":1,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.71699999999999997,"15min":1.0109999999999999},"min":{"1min":0.47399999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.88900000000000001,"5min":0.94899999999999995,"15min":9.5709999999999997},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.748,"15min":0.878},"min":{"1min":0.496,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.92500000000000004,"5min":1.153,"15min":5.266},"last":0.89200000000000002}]},{"osd":2,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.74099999999999999,"15min":0.99099999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.98399999999999999,"5min":1.034,"15min":8.4529999999999994},"last":1.024},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.752,"15min":0.93999999999999995},"min":{"1min":0.55700000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88200000000000001,"5min":1.018,"15min":6.6829999999999998},"last":0.81499999999999995}]},{"osd":3,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.54000000000000004,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.872,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.78200000000000003,"15min":0.79800000000000004},"min":{"1min":0.59399999999999997,"5min":0.58199999999999996,"15min":0.54300000000000004},"max":{"1min":0.95899999999999996,"5min":0.99299999999999999,"15min":1.389},"last":0.77800000000000002}]},{"osd":5,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69099999999999995,"15min":0.72999999999999998},"min":{"1min":0.46700000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1,"5min":1,"15min":1.0349999999999999},"last":1.002},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.70899999999999996,"15min":0.71899999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.88700000000000001,"5min":0.998,"15min":1.0640000000000001},"last":0.93600000000000005}]},{"osd":6,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.76600000000000001,"15min":0.77800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.996,"5min":1.032,"15min":1.032},"last":1.077},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.78800000000000003,"15min":0.81200000000000006},"min":{"1min":0.52600000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.032,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":1.204}]},{"osd":7,"last update":"Fri Mar 3 08:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.76100000000000001,"15min":0.80800000000000005},"min":{"1min":0.51000000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.95799999999999996,"5min":1.0609999999999999,"15min":1.679},"last":1.0569999999999999},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.83999999999999997,"15min":0.90700000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0169999999999999,"5min":1.1599999999999999,"15min":1.8340000000000001},"last":1.1899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574944,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11204,"kb_used_data":1340,"kb_used_omap":0,"kb_used_meta":9856,"kb_avail":94360636,"statfs":{"total":96636764160,"available":96625291264,"internally_reserved":0,"allocated":1372160,"data_stored":633693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10092544},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 3 08:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.44900000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.76700000000000002,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.751,"15min":0.74099999999999999},"min":{"1min":0.46800000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0600000000000001,"5min":2.528,"15min":2.528},"last":0.77400000000000002}]},{"osd":1,"last update":"Fri Mar 3 08:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.72299999999999998,"15min":0.73599999999999999},"min":{"1min":0.48599999999999999,"5min":0.45900000000000002,"15min":0.45700000000000002},"max":{"1min":0.93600000000000005,"5min":1.038,"15min":1.038},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.69799999999999995,"15min":0.64900000000000002},"min":{"1min":0.46000000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.75800000000000001}]},{"osd":2,"last update":"Fri Mar 3 08:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.48999999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70999999999999996,"15min":0.71199999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.86599999999999999,"5min":1.024,"15min":1.321},"last":0.80600000000000005}]},{"osd":3,"last update":"Fri Mar 3 08:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.71699999999999997,"15min":0.69699999999999995},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.999,"5min":1.024,"15min":1.024},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.749,"15min":0.73699999999999999},"min":{"1min":0.53500000000000003,"5min":0.501,"15min":0.501},"max":{"1min":0.93100000000000005,"5min":1.012,"15min":1.012},"last":0.70699999999999996}]},{"osd":4,"last update":"Fri Mar 3 08:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.67100000000000004,"15min":0.66600000000000004},"min":{"1min":0.41399999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83599999999999997,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.61499999999999999,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.94299999999999995,"5min":1.079,"15min":1.079},"last":0.83099999999999996}]},{"osd":6,"last update":"Fri Mar 3 08:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72499999999999998,"15min":0.77800000000000002},"min":{"1min":0.48199999999999998,"5min":0.442,"15min":0.442},"max":{"1min":0.98299999999999998,"5min":1.012,"15min":1.071},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.71799999999999997,"15min":0.72199999999999998},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.91400000000000003,"5min":0.97099999999999997,"15min":1.0429999999999999},"last":0.79300000000000004}]},{"osd":7,"last update":"Fri Mar 3 08:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.75,"15min":0.75600000000000001},"min":{"1min":0.498,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0269999999999999,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.78900000000000003,"15min":0.82099999999999995},"min":{"1min":0.54900000000000004,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.1080000000000001},"last":0.85699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:03:19.078 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-03-03T08:03:19.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2023-03-03T08:03:19.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.360+0000 7f1d4f3ff700 1 Processor -- start 2023-03-03T08:03:19.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.360+0000 7f1d4f3ff700 1 -- start start 2023-03-03T08:03:19.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.361+0000 7f1d4f3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4815ca70 0x7f1d480cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:19.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.362+0000 7f1d4f3ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d480cf980 con 0x7f1d480d0270 2023-03-03T08:03:19.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.362+0000 7f1d4f3ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d480d1800 con 0x7f1d4806b4b0 2023-03-03T08:03:19.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.362+0000 7f1d4f3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d480cd5b0 con 0x7f1d4815ca70 2023-03-03T08:03:19.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.362+0000 7f1d4d99c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1d480d0270 0x7f1d480d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42688/0 (socket says 172.21.15.98:42688) 2023-03-03T08:03:19.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.362+0000 7f1d4d99c700 1 -- 172.21.15.98:0/903979017 learned_addr learned my addr 172.21.15.98:0/903979017 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:19.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.362+0000 7f1d4d19b700 1 --2- 172.21.15.98:0/903979017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4815ca70 0x7f1d480cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:19.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.362+0000 7f1d3ffff700 1 -- 172.21.15.98:0/903979017 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581762253 0 0) 0x7f1d480cf980 con 0x7f1d480d0270 2023-03-03T08:03:19.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.362+0000 7f1d3ffff700 1 -- 172.21.15.98:0/903979017 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d340035b0 con 0x7f1d480d0270 2023-03-03T08:03:19.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.363+0000 7f1d3ffff700 1 -- 172.21.15.98:0/903979017 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582346869 0 0) 0x7f1d480d1800 con 0x7f1d4806b4b0 2023-03-03T08:03:19.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.363+0000 7f1d3ffff700 1 -- 172.21.15.98:0/903979017 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d480cf980 con 0x7f1d4806b4b0 2023-03-03T08:03:19.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.363+0000 7f1d4d19b700 1 -- 172.21.15.98:0/903979017 >> v1:172.21.15.100:6789/0 conn(0x7f1d4806b4b0 legacy=0x7f1d480cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:19.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.363+0000 7f1d4d19b700 1 -- 172.21.15.98:0/903979017 >> v1:172.21.15.98:6789/0 conn(0x7f1d480d0270 legacy=0x7f1d480d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:19.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.363+0000 7f1d4d19b700 1 -- 172.21.15.98:0/903979017 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d480cd900 con 0x7f1d4815ca70 2023-03-03T08:03:19.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.363+0000 7f1d4d19b700 1 --2- 172.21.15.98:0/903979017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4815ca70 0x7f1d480cf440 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1d44009870 tx=0x7f1d44009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edfcf2edbb718767 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:19.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.363+0000 7f1d3ffff700 1 -- 172.21.15.98:0/903979017 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d44011830 con 0x7f1d4815ca70 2023-03-03T08:03:19.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.363+0000 7f1d3ffff700 1 -- 172.21.15.98:0/903979017 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d44011990 con 0x7f1d4815ca70 2023-03-03T08:03:19.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.363+0000 7f1d3ffff700 1 -- 172.21.15.98:0/903979017 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d44011d20 con 0x7f1d4815ca70 2023-03-03T08:03:19.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.364+0000 7f1d4f3ff700 1 -- 172.21.15.98:0/903979017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4815ca70 msgr2=0x7f1d480cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:19.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.364+0000 7f1d4f3ff700 1 --2- 172.21.15.98:0/903979017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4815ca70 0x7f1d480cf440 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1d44009870 tx=0x7f1d44009b80 comp rx=0 tx=0).stop 2023-03-03T08:03:19.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.364+0000 7f1d4f3ff700 1 -- 172.21.15.98:0/903979017 shutdown_connections 2023-03-03T08:03:19.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.364+0000 7f1d4f3ff700 1 --2- 172.21.15.98:0/903979017 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4815ca70 0x7f1d480cf440 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:19.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.364+0000 7f1d4f3ff700 1 -- 172.21.15.98:0/903979017 >> 172.21.15.98:0/903979017 conn(0x7f1d48073e70 msgr2=0x7f1d4806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:19.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.364+0000 7f1d4f3ff700 1 -- 172.21.15.98:0/903979017 shutdown_connections 2023-03-03T08:03:19.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.364+0000 7f1d4f3ff700 1 -- 172.21.15.98:0/903979017 wait complete. 2023-03-03T08:03:19.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.366+0000 7f1d4f3ff700 1 Processor -- start 2023-03-03T08:03:19.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.366+0000 7f1d4f3ff700 1 -- start start 2023-03-03T08:03:19.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.367+0000 7f1d4f3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d4806b4b0 0x7f1d4815ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:19.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.367+0000 7f1d4f3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d480d0270 0x7f1d4814c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:19.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.367+0000 7f1d4f3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4815ca70 0x7f1d481f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:19.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.367+0000 7f1d4f3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4815df10 con 0x7f1d4806b4b0 2023-03-03T08:03:19.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.367+0000 7f1d4f3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d480ca630 con 0x7f1d480d0270 2023-03-03T08:03:19.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.367+0000 7f1d4f3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4815e190 con 0x7f1d4815ca70 2023-03-03T08:03:19.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.368+0000 7f1d4d19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d4806b4b0 0x7f1d4815ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:19.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.368+0000 7f1d4d19b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d4806b4b0 0x7f1d4815ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60208/0 (socket says 172.21.15.98:60208) 2023-03-03T08:03:19.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.368+0000 7f1d4d19b700 1 -- 172.21.15.98:0/1759562007 learned_addr learned my addr 172.21.15.98:0/1759562007 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:19.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.368+0000 7f1d4d99c700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4815ca70 0x7f1d481f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:19.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.368+0000 7f1d4c99a700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d480d0270 0x7f1d4814c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:19.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.368+0000 7f1d4d19b700 1 -- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4815ca70 msgr2=0x7f1d481f58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:19.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.368+0000 7f1d4d19b700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4815ca70 0x7f1d481f58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:19.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.368+0000 7f1d4d19b700 1 -- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d480d0270 msgr2=0x7f1d4814c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:19.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.368+0000 7f1d4d19b700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d480d0270 0x7f1d4814c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:19.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.368+0000 7f1d4d19b700 1 -- 172.21.15.98:0/1759562007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d440094a0 con 0x7f1d4806b4b0 2023-03-03T08:03:19.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.368+0000 7f1d4d99c700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4815ca70 0x7f1d481f58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:19.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.368+0000 7f1d4c99a700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d480d0270 0x7f1d4814c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:19.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.369+0000 7f1d4d19b700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d4806b4b0 0x7f1d4815ec70 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f1d440090b0 tx=0x7f1d44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:19.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.369+0000 7f1d3dffb700 1 -- 172.21.15.98:0/1759562007 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4401a860 con 0x7f1d4806b4b0 2023-03-03T08:03:19.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.369+0000 7f1d3dffb700 1 -- 172.21.15.98:0/1759562007 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d4401a9c0 con 0x7f1d4806b4b0 2023-03-03T08:03:19.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.369+0000 7f1d3dffb700 1 -- 172.21.15.98:0/1759562007 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4401ad30 con 0x7f1d4806b4b0 2023-03-03T08:03:19.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.369+0000 7f1d4f3ff700 1 -- 172.21.15.98:0/1759562007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d480d1aa0 con 0x7f1d4806b4b0 2023-03-03T08:03:19.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.369+0000 7f1d4f3ff700 1 -- 172.21.15.98:0/1759562007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d480cdbe0 con 0x7f1d4806b4b0 2023-03-03T08:03:19.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.370+0000 7f1d4f3ff700 1 -- 172.21.15.98:0/1759562007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4815ce40 con 0x7f1d4806b4b0 2023-03-03T08:03:19.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.372+0000 7f1d3dffb700 1 -- 172.21.15.98:0/1759562007 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f1d4400f050 con 0x7f1d4806b4b0 2023-03-03T08:03:19.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.373+0000 7f1d3dffb700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1d2c068380 0x7f1d2c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:19.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.373+0000 7f1d3dffb700 1 -- 172.21.15.98:0/1759562007 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1d44086c20 con 0x7f1d4806b4b0 2023-03-03T08:03:19.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.373+0000 7f1d4c99a700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1d2c068380 0x7f1d2c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:19.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.374+0000 7f1d4c99a700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1d2c068380 0x7f1d2c06a830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1d38007fd0 tx=0x7f1d38009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:19.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.377+0000 7f1d3dffb700 1 -- 172.21.15.98:0/1759562007 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1d44053450 con 0x7f1d4806b4b0 2023-03-03T08:03:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.642+0000 7f1d4f3ff700 1 -- 172.21.15.98:0/1759562007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7f1d4804e6e0 con 0x7f1d4806b4b0 2023-03-03T08:03:19.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.646+0000 7f1d3dffb700 1 -- 172.21.15.98:0/1759562007 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 184+0+0 (secure 0 0 0) 0x7f1d440290e0 con 0x7f1d4806b4b0 2023-03-03T08:03:19.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.649+0000 7f1d4f3ff700 1 -- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1d2c068380 msgr2=0x7f1d2c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.649+0000 7f1d4f3ff700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1d2c068380 0x7f1d2c06a830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1d38007fd0 tx=0x7f1d38009040 comp rx=0 tx=0).stop 2023-03-03T08:03:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.649+0000 7f1d4f3ff700 1 -- 172.21.15.98:0/1759562007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d4806b4b0 msgr2=0x7f1d4815ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.649+0000 7f1d4f3ff700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d4806b4b0 0x7f1d4815ec70 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f1d440090b0 tx=0x7f1d44004750 comp rx=0 tx=0).stop 2023-03-03T08:03:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.649+0000 7f1d4f3ff700 1 -- 172.21.15.98:0/1759562007 shutdown_connections 2023-03-03T08:03:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.649+0000 7f1d4f3ff700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f1d2c068380 0x7f1d2c06a830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.649+0000 7f1d4f3ff700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d4806b4b0 0x7f1d4815ec70 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.649+0000 7f1d4f3ff700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4815ca70 0x7f1d481f58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.649+0000 7f1d4f3ff700 1 --2- 172.21.15.98:0/1759562007 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d480d0270 0x7f1d4814c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.649+0000 7f1d4f3ff700 1 -- 172.21.15.98:0/1759562007 >> 172.21.15.98:0/1759562007 conn(0x7f1d48073e70 msgr2=0x7f1d480b5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.649+0000 7f1d4f3ff700 1 -- 172.21.15.98:0/1759562007 shutdown_connections 2023-03-03T08:03:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.649+0000 7f1d4f3ff700 1 -- 172.21.15.98:0/1759562007 wait complete. 2023-03-03T08:03:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.0 2023-03-03T08:03:19.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:03:19.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.921+0000 7f82631ff700 1 Processor -- start 2023-03-03T08:03:19.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.921+0000 7f82631ff700 1 -- start start 2023-03-03T08:03:19.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.922+0000 7f82631ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f825c15cc00 0x7f825c0cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:19.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.922+0000 7f82631ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c0cde10 con 0x7f825c0d27c0 2023-03-03T08:03:19.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.922+0000 7f82631ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c0d1a00 con 0x7f825c0d2c60 2023-03-03T08:03:19.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.922+0000 7f82631ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f825c0ce040 con 0x7f825c15cc00 2023-03-03T08:03:19.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.923+0000 7f825bfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f825c0d27c0 0x7f825c0b60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42692/0 (socket says 172.21.15.98:42692) 2023-03-03T08:03:19.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.923+0000 7f825bfff700 1 -- 172.21.15.98:0/2795534186 learned_addr learned my addr 172.21.15.98:0/2795534186 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:19.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.923+0000 7f8260f9b700 1 --2- 172.21.15.98:0/2795534186 >> v2:172.21.15.100:3300/0 conn(0x7f825c15cc00 0x7f825c0cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:19.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.923+0000 7f825b7fe700 1 -- 172.21.15.98:0/2795534186 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851295326 0 0) 0x7f825c0cde10 con 0x7f825c0d27c0 2023-03-03T08:03:19.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.923+0000 7f825b7fe700 1 -- 172.21.15.98:0/2795534186 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f824c0035b0 con 0x7f825c0d27c0 2023-03-03T08:03:19.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.923+0000 7f8260f9b700 1 -- 172.21.15.98:0/2795534186 >> v1:172.21.15.100:6789/0 conn(0x7f825c0d2c60 legacy=0x7f825c0cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:19.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.923+0000 7f825b7fe700 1 -- 172.21.15.98:0/2795534186 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556555767 0 0) 0x7f825c0d1a00 con 0x7f825c0d2c60 2023-03-03T08:03:19.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.923+0000 7f8260f9b700 1 -- 172.21.15.98:0/2795534186 >> v1:172.21.15.98:6789/0 conn(0x7f825c0d27c0 legacy=0x7f825c0b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:19.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.924+0000 7f8260f9b700 1 -- 172.21.15.98:0/2795534186 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c0d1ce0 con 0x7f825c15cc00 2023-03-03T08:03:19.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.924+0000 7f8260f9b700 1 --2- 172.21.15.98:0/2795534186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f825c15cc00 0x7f825c0cd8d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8250009840 tx=0x7f8250009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1db2b3dd07ea1ac server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:19.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.924+0000 7f825b7fe700 1 -- 172.21.15.98:0/2795534186 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8250011830 con 0x7f825c15cc00 2023-03-03T08:03:19.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.924+0000 7f825b7fe700 1 -- 172.21.15.98:0/2795534186 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8250011990 con 0x7f825c15cc00 2023-03-03T08:03:19.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.924+0000 7f825b7fe700 1 -- 172.21.15.98:0/2795534186 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8250011d20 con 0x7f825c15cc00 2023-03-03T08:03:19.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.925+0000 7f82631ff700 1 -- 172.21.15.98:0/2795534186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f825c15cc00 msgr2=0x7f825c0cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:19.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.925+0000 7f82631ff700 1 --2- 172.21.15.98:0/2795534186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f825c15cc00 0x7f825c0cd8d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8250009840 tx=0x7f8250009b50 comp rx=0 tx=0).stop 2023-03-03T08:03:19.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.925+0000 7f82631ff700 1 -- 172.21.15.98:0/2795534186 shutdown_connections 2023-03-03T08:03:19.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.925+0000 7f82631ff700 1 --2- 172.21.15.98:0/2795534186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f825c15cc00 0x7f825c0cd8d0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:19.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.925+0000 7f82631ff700 1 -- 172.21.15.98:0/2795534186 >> 172.21.15.98:0/2795534186 conn(0x7f825c06aee0 msgr2=0x7f825c06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:19.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.925+0000 7f82631ff700 1 -- 172.21.15.98:0/2795534186 shutdown_connections 2023-03-03T08:03:19.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.925+0000 7f82631ff700 1 -- 172.21.15.98:0/2795534186 wait complete. 2023-03-03T08:03:19.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.927+0000 7f82631ff700 1 Processor -- start 2023-03-03T08:03:19.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.927+0000 7f82631ff700 1 -- start start 2023-03-03T08:03:19.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.928+0000 7f82631ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f825c0d27c0 0x7f825c15fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:19.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.928+0000 7f82631ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f825c0d2c60 0x7f825c15cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:19.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.928+0000 7f82631ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f825c15cc00 0x7f825c15d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:19.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.928+0000 7f82631ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825c15ee40 con 0x7f825c0d2c60 2023-03-03T08:03:19.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.928+0000 7f82631ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825c1431f0 con 0x7f825c15cc00 2023-03-03T08:03:19.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.928+0000 7f82631ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825c142f10 con 0x7f825c0d27c0 2023-03-03T08:03:19.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.928+0000 7f825bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f825c0d2c60 0x7f825c15cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:19.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.929+0000 7f825bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f825c0d2c60 0x7f825c15cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60220/0 (socket says 172.21.15.98:60220) 2023-03-03T08:03:19.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.929+0000 7f825bfff700 1 -- 172.21.15.98:0/1618629688 learned_addr learned my addr 172.21.15.98:0/1618629688 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:19.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.929+0000 7f826179c700 1 --2- 172.21.15.98:0/1618629688 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f825c15cc00 0x7f825c15d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:19.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.929+0000 7f8260f9b700 1 --2- 172.21.15.98:0/1618629688 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f825c0d27c0 0x7f825c15fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:19.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.929+0000 7f825bfff700 1 -- 172.21.15.98:0/1618629688 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f825c0d27c0 msgr2=0x7f825c15fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:19.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.929+0000 7f825bfff700 1 --2- 172.21.15.98:0/1618629688 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f825c0d27c0 0x7f825c15fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:19.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.929+0000 7f825bfff700 1 -- 172.21.15.98:0/1618629688 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f825c15cc00 msgr2=0x7f825c15d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:19.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.929+0000 7f825bfff700 1 --2- 172.21.15.98:0/1618629688 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f825c15cc00 0x7f825c15d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:19.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.929+0000 7f825bfff700 1 -- 172.21.15.98:0/1618629688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82500094a0 con 0x7f825c0d2c60 2023-03-03T08:03:19.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.929+0000 7f8260f9b700 1 --2- 172.21.15.98:0/1618629688 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f825c0d27c0 0x7f825c15fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:19.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.929+0000 7f825bfff700 1 --2- 172.21.15.98:0/1618629688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f825c0d2c60 0x7f825c15cfd0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f824800c450 tx=0x7f824800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:19.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.930+0000 7f82597fa700 1 -- 172.21.15.98:0/1618629688 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8248005d70 con 0x7f825c0d2c60 2023-03-03T08:03:19.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.930+0000 7f82597fa700 1 -- 172.21.15.98:0/1618629688 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82480070d0 con 0x7f825c0d2c60 2023-03-03T08:03:19.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.930+0000 7f82631ff700 1 -- 172.21.15.98:0/1618629688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f825c0d22a0 con 0x7f825c0d2c60 2023-03-03T08:03:19.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.930+0000 7f82597fa700 1 -- 172.21.15.98:0/1618629688 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8248007440 con 0x7f825c0d2c60 2023-03-03T08:03:19.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.930+0000 7f82631ff700 1 -- 172.21.15.98:0/1618629688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f825c0d1fc0 con 0x7f825c0d2c60 2023-03-03T08:03:19.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.931+0000 7f82631ff700 1 -- 172.21.15.98:0/1618629688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f825c0b67e0 con 0x7f825c0d2c60 2023-03-03T08:03:19.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.932+0000 7f82597fa700 1 -- 172.21.15.98:0/1618629688 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f8248019030 con 0x7f825c0d2c60 2023-03-03T08:03:19.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.934+0000 7f82597fa700 1 --2- 172.21.15.98:0/1618629688 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8240068380 0x7f824006a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:19.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.934+0000 7f82597fa700 1 -- 172.21.15.98:0/1618629688 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8248084f20 con 0x7f825c0d2c60 2023-03-03T08:03:19.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.935+0000 7f8260f9b700 1 --2- 172.21.15.98:0/1618629688 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8240068380 0x7f824006a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:19.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.935+0000 7f8260f9b700 1 --2- 172.21.15.98:0/1618629688 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8240068380 0x7f824006a830 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8250009fd0 tx=0x7f825001c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:19.938+0000 7f82597fa700 1 -- 172.21.15.98:0/1618629688 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f824804b5e0 con 0x7f825c0d2c60 2023-03-03T08:03:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.171+0000 7f82631ff700 1 -- 172.21.15.98:0/1618629688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f825c0729b0 con 0x7f825c0d2c60 2023-03-03T08:03:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.171+0000 7f82597fa700 1 -- 172.21.15.98:0/1618629688 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4656 (secure 0 0 0) 0x7f8248017020 con 0x7f825c0d2c60 2023-03-03T08:03:20.174 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:03:20.174 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6834/2563396918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":2563396918},{"type":"v1","addr":"172.21.15.100:6834","nonce":2563396918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7992,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2374561485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2374561485},{"type":"v1","addr":"172.21.15.98:6834","nonce":2374561485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7995,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2695503624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":2695503624},{"type":"v1","addr":"172.21.15.98:6835","nonce":2695503624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:03:13.506818+0000","modified":"2023-03-03T08:03:16.969687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8001},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8001":{"gid":8001,"name":"d","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/2660525287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6833","nonce":2660525287},{"type":"v1","addr":"172.21.15.100:6835","nonce":2660525287}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-03-03T08:03:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.174+0000 7f82631ff700 1 -- 172.21.15.98:0/1618629688 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8240068380 msgr2=0x7f824006a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.174+0000 7f82631ff700 1 --2- 172.21.15.98:0/1618629688 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8240068380 0x7f824006a830 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8250009fd0 tx=0x7f825001c040 comp rx=0 tx=0).stop 2023-03-03T08:03:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.174+0000 7f82631ff700 1 -- 172.21.15.98:0/1618629688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f825c0d2c60 msgr2=0x7f825c15cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.174+0000 7f82631ff700 1 --2- 172.21.15.98:0/1618629688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f825c0d2c60 0x7f825c15cfd0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f824800c450 tx=0x7f824800c760 comp rx=0 tx=0).stop 2023-03-03T08:03:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.174+0000 7f82631ff700 1 -- 172.21.15.98:0/1618629688 shutdown_connections 2023-03-03T08:03:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.174+0000 7f82631ff700 1 --2- 172.21.15.98:0/1618629688 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f825c15cc00 0x7f825c15d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.174+0000 7f82631ff700 1 --2- 172.21.15.98:0/1618629688 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f825c0d27c0 0x7f825c15fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.174+0000 7f82631ff700 1 --2- 172.21.15.98:0/1618629688 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8240068380 0x7f824006a830 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.174+0000 7f82631ff700 1 --2- 172.21.15.98:0/1618629688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f825c0d2c60 0x7f825c15cfd0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.174+0000 7f82631ff700 1 -- 172.21.15.98:0/1618629688 >> 172.21.15.98:0/1618629688 conn(0x7f825c06aee0 msgr2=0x7f825c0b6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.174+0000 7f82631ff700 1 -- 172.21.15.98:0/1618629688 shutdown_connections 2023-03-03T08:03:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.174+0000 7f82631ff700 1 -- 172.21.15.98:0/1618629688 wait complete. 2023-03-03T08:03:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 62 2023-03-03T08:03:20.607 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-03-03T08:03:13.506818+0000', 'modified': '2023-03-03T08:03:16.969687+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8001}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8001': {'gid': 8001, 'name': 'd', 'rank': 0, 'incarnation': 58, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.100:6835/2660525287', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6833', 'nonce': 2660525287}, {'type': 'v1', 'addr': '172.21.15.100:6835', 'nonce': 2660525287}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-03-03T08:03:20.607 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-03-03T08:03:20.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-03-03T08:03:20.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.851+0000 7f29477ff700 1 Processor -- start 2023-03-03T08:03:20.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.851+0000 7f29477ff700 1 -- start start 2023-03-03T08:03:20.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.853+0000 7f29477ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29400d0270 0x7f29400cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.853+0000 7f29477ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f294006b4b0 0x7f29400d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.853+0000 7f29477ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29400cf980 con 0x7f294015ca70 2023-03-03T08:03:20.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.853+0000 7f29477ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29400cd5b0 con 0x7f294006b4b0 2023-03-03T08:03:20.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.853+0000 7f29477ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29400d1800 con 0x7f29400d0270 2023-03-03T08:03:20.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.853+0000 7f2945d9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f294006b4b0 0x7f29400d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.853+0000 7f2945d9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f294006b4b0 0x7f29400d1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60228/0 (socket says 172.21.15.98:60228) 2023-03-03T08:03:20.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.853+0000 7f2945d9c700 1 -- 172.21.15.98:0/1020936237 learned_addr learned my addr 172.21.15.98:0/1020936237 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:20.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.853+0000 7f2944d9a700 1 --2- 172.21.15.98:0/1020936237 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29400d0270 0x7f29400cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.853+0000 7f2945d9c700 1 -- 172.21.15.98:0/1020936237 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29400d0270 msgr2=0x7f29400cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:20.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.853+0000 7f2945d9c700 1 --2- 172.21.15.98:0/1020936237 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29400d0270 0x7f29400cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.854+0000 7f2945d9c700 1 -- 172.21.15.98:0/1020936237 >> v1:172.21.15.100:6789/0 conn(0x7f294015ca70 legacy=0x7f29400cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:20.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.854+0000 7f2945d9c700 1 -- 172.21.15.98:0/1020936237 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29400cd900 con 0x7f294006b4b0 2023-03-03T08:03:20.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.854+0000 7f2945d9c700 1 --2- 172.21.15.98:0/1020936237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294006b4b0 0x7f29400d1fc0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f2938009e50 tx=0x7f293800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8909a8d00e2f943c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:20.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.854+0000 7f2937fff700 1 -- 172.21.15.98:0/1020936237 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293800e070 con 0x7f294006b4b0 2023-03-03T08:03:20.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.854+0000 7f2937fff700 1 -- 172.21.15.98:0/1020936237 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293800b910 con 0x7f294006b4b0 2023-03-03T08:03:20.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.854+0000 7f2937fff700 1 -- 172.21.15.98:0/1020936237 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293800bcc0 con 0x7f294006b4b0 2023-03-03T08:03:20.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.855+0000 7f29477ff700 1 -- 172.21.15.98:0/1020936237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294006b4b0 msgr2=0x7f29400d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:20.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.855+0000 7f29477ff700 1 --2- 172.21.15.98:0/1020936237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294006b4b0 0x7f29400d1fc0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f2938009e50 tx=0x7f293800b330 comp rx=0 tx=0).stop 2023-03-03T08:03:20.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.855+0000 7f29477ff700 1 -- 172.21.15.98:0/1020936237 shutdown_connections 2023-03-03T08:03:20.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.855+0000 7f29477ff700 1 --2- 172.21.15.98:0/1020936237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294006b4b0 0x7f29400d1fc0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.855+0000 7f29477ff700 1 --2- 172.21.15.98:0/1020936237 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29400d0270 0x7f29400cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.855+0000 7f29477ff700 1 -- 172.21.15.98:0/1020936237 >> 172.21.15.98:0/1020936237 conn(0x7f2940073e70 msgr2=0x7f294006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:20.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.855+0000 7f29477ff700 1 -- 172.21.15.98:0/1020936237 shutdown_connections 2023-03-03T08:03:20.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.855+0000 7f29477ff700 1 -- 172.21.15.98:0/1020936237 wait complete. 2023-03-03T08:03:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.857+0000 7f29477ff700 1 Processor -- start 2023-03-03T08:03:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.857+0000 7f29477ff700 1 -- start start 2023-03-03T08:03:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.858+0000 7f29477ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294006b4b0 0x7f29400b7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.858+0000 7f29477ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29400d0270 0x7f29400ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.858+0000 7f29477ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294015ca70 0x7f2940151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.858+0000 7f29477ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29400cb5b0 con 0x7f294006b4b0 2023-03-03T08:03:20.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.858+0000 7f29477ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29400c7eb0 con 0x7f29400d0270 2023-03-03T08:03:20.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.858+0000 7f29477ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29400cad10 con 0x7f294015ca70 2023-03-03T08:03:20.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.858+0000 7f294559b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294006b4b0 0x7f29400b7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.858+0000 7f294559b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294006b4b0 0x7f29400b7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60238/0 (socket says 172.21.15.98:60238) 2023-03-03T08:03:20.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.858+0000 7f294559b700 1 -- 172.21.15.98:0/2120234797 learned_addr learned my addr 172.21.15.98:0/2120234797 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:20.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.859+0000 7f2945d9c700 1 --2- 172.21.15.98:0/2120234797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294015ca70 0x7f2940151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.859+0000 7f2944d9a700 1 --2- 172.21.15.98:0/2120234797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29400d0270 0x7f29400ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.859+0000 7f294559b700 1 -- 172.21.15.98:0/2120234797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294015ca70 msgr2=0x7f2940151320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.859+0000 7f294559b700 1 --2- 172.21.15.98:0/2120234797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294015ca70 0x7f2940151320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.859+0000 7f294559b700 1 -- 172.21.15.98:0/2120234797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29400d0270 msgr2=0x7f29400ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.859+0000 7f294559b700 1 --2- 172.21.15.98:0/2120234797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29400d0270 0x7f29400ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.859+0000 7f294559b700 1 -- 172.21.15.98:0/2120234797 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2938009b30 con 0x7f294006b4b0 2023-03-03T08:03:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.859+0000 7f2944d9a700 1 --2- 172.21.15.98:0/2120234797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29400d0270 0x7f29400ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.859+0000 7f294559b700 1 --2- 172.21.15.98:0/2120234797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294006b4b0 0x7f29400b7d00 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f293c00c4e0 tx=0x7f293c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.859+0000 7f2935ffb700 1 -- 172.21.15.98:0/2120234797 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293c0052d0 con 0x7f294006b4b0 2023-03-03T08:03:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.859+0000 7f2935ffb700 1 -- 172.21.15.98:0/2120234797 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293c005430 con 0x7f294006b4b0 2023-03-03T08:03:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.859+0000 7f2935ffb700 1 -- 172.21.15.98:0/2120234797 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293c0038c0 con 0x7f294006b4b0 2023-03-03T08:03:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.859+0000 7f29477ff700 1 -- 172.21.15.98:0/2120234797 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29400d1aa0 con 0x7f294006b4b0 2023-03-03T08:03:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.860+0000 7f29477ff700 1 -- 172.21.15.98:0/2120234797 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29400cdbe0 con 0x7f294006b4b0 2023-03-03T08:03:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.864+0000 7f2935ffb700 1 -- 172.21.15.98:0/2120234797 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f293c019030 con 0x7f294006b4b0 2023-03-03T08:03:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.865+0000 7f2935ffb700 1 --2- 172.21.15.98:0/2120234797 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2924068380 0x7f292406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.865+0000 7f2935ffb700 1 -- 172.21.15.98:0/2120234797 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f293c084fd0 con 0x7f294006b4b0 2023-03-03T08:03:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.866+0000 7f2944d9a700 1 --2- 172.21.15.98:0/2120234797 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2924068380 0x7f292406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.866+0000 7f2944d9a700 1 --2- 172.21.15.98:0/2120234797 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2924068380 0x7f292406a830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2940159d70 tx=0x7f2930008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.867+0000 7f295b0adb80 1 Processor -- start 2023-03-03T08:03:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.867+0000 7f295b0adb80 1 -- start start 2023-03-03T08:03:20.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f295b0adb80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778fda70 0x55ff778fde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f295b0adb80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778fccd0 0x55ff7794c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f295b0adb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778f8560 0x55ff7794c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f295b0adb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff778734f0 con 0x55ff778f8560 2023-03-03T08:03:20.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f295b0adb80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff777e8b60 con 0x55ff778fccd0 2023-03-03T08:03:20.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f295b0adb80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ff77857ad0 con 0x55ff778fda70 2023-03-03T08:03:20.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f2945d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778f8560 0x55ff7794c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f294559b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778fda70 0x55ff778fde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f2945d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778f8560 0x55ff7794c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60250/0 (socket says 172.21.15.98:60250) 2023-03-03T08:03:20.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f2945d9c700 1 -- 172.21.15.98:0/138679306 learned_addr learned my addr 172.21.15.98:0/138679306 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:20.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f2944d9a700 1 --2- 172.21.15.98:0/138679306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778fccd0 0x55ff7794c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f2945d9c700 1 -- 172.21.15.98:0/138679306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778fda70 msgr2=0x55ff778fde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:20.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f2945d9c700 1 --2- 172.21.15.98:0/138679306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778fda70 0x55ff778fde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f2945d9c700 1 -- 172.21.15.98:0/138679306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778fccd0 msgr2=0x55ff7794c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:20.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f2945d9c700 1 --2- 172.21.15.98:0/138679306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778fccd0 0x55ff7794c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.868+0000 7f2945d9c700 1 -- 172.21.15.98:0/138679306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff77645540 con 0x55ff778f8560 2023-03-03T08:03:20.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.869+0000 7f2945d9c700 1 --2- 172.21.15.98:0/138679306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778f8560 0x55ff7794c570 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f293800b680 tx=0x7f2938010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c34b539fffa6588 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:20.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.869+0000 7f2922ffd700 1 -- 172.21.15.98:0/138679306 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2938003a20 con 0x55ff778f8560 2023-03-03T08:03:20.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.869+0000 7f2922ffd700 1 -- 172.21.15.98:0/138679306 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2938003b80 con 0x55ff778f8560 2023-03-03T08:03:20.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.869+0000 7f2922ffd700 1 -- 172.21.15.98:0/138679306 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2938004040 con 0x55ff778f8560 2023-03-03T08:03:20.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.870+0000 7f295b0adb80 1 -- 172.21.15.98:0/138679306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778f8560 msgr2=0x55ff7794c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:20.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.870+0000 7f295b0adb80 1 --2- 172.21.15.98:0/138679306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778f8560 0x55ff7794c570 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f293800b680 tx=0x7f2938010000 comp rx=0 tx=0).stop 2023-03-03T08:03:20.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.870+0000 7f295b0adb80 1 -- 172.21.15.98:0/138679306 shutdown_connections 2023-03-03T08:03:20.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.870+0000 7f295b0adb80 1 --2- 172.21.15.98:0/138679306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778f8560 0x55ff7794c570 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.870+0000 7f295b0adb80 1 --2- 172.21.15.98:0/138679306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778fda70 0x55ff778fde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.870+0000 7f295b0adb80 1 --2- 172.21.15.98:0/138679306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778fccd0 0x55ff7794c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.870+0000 7f295b0adb80 1 -- 172.21.15.98:0/138679306 >> 172.21.15.98:0/138679306 conn(0x55ff778fbc30 msgr2=0x55ff778fc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:20.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.870+0000 7f295b0adb80 1 -- 172.21.15.98:0/138679306 shutdown_connections 2023-03-03T08:03:20.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.870+0000 7f295b0adb80 1 -- 172.21.15.98:0/138679306 wait complete. 2023-03-03T08:03:20.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.871+0000 7f295b0adb80 1 -- start start 2023-03-03T08:03:20.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.871+0000 7f295b0adb80 1 Processor -- start 2023-03-03T08:03:20.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.872+0000 7f295b0adb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778f8560 0x55ff7792bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.872+0000 7f295b0adb80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778fccd0 0x55ff77946c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.872+0000 7f295b0adb80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778fda70 0x55ff77919820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.872+0000 7f295b0adb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff77850e40 con 0x55ff778f8560 2023-03-03T08:03:20.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.872+0000 7f295b0adb80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff77646140 con 0x55ff778fccd0 2023-03-03T08:03:20.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.872+0000 7f295b0adb80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ff77940fc0 con 0x55ff778fda70 2023-03-03T08:03:20.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.872+0000 7f295b0adb80 10 client.0 _ll_register_callbacks cb 0x55ff777ef1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T08:03:20.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.873+0000 7f294559b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778f8560 0x55ff7792bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.873+0000 7f294559b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778f8560 0x55ff7792bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60254/0 (socket says 172.21.15.98:60254) 2023-03-03T08:03:20.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.873+0000 7f294559b700 1 -- 172.21.15.98:0/339907307 learned_addr learned my addr 172.21.15.98:0/339907307 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:20.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.873+0000 7f2945d9c700 1 --2- 172.21.15.98:0/339907307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778fda70 0x55ff77919820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.873+0000 7f2944d9a700 1 --2- 172.21.15.98:0/339907307 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778fccd0 0x55ff77946c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.873+0000 7f294559b700 1 -- 172.21.15.98:0/339907307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778fda70 msgr2=0x55ff77919820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:20.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.873+0000 7f294559b700 1 --2- 172.21.15.98:0/339907307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778fda70 0x55ff77919820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.873+0000 7f294559b700 1 -- 172.21.15.98:0/339907307 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778fccd0 msgr2=0x55ff77946c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:20.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.873+0000 7f294559b700 1 --2- 172.21.15.98:0/339907307 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778fccd0 0x55ff77946c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.873+0000 7f294559b700 1 -- 172.21.15.98:0/339907307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293800be70 con 0x55ff778f8560 2023-03-03T08:03:20.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.873+0000 7f294559b700 1 --2- 172.21.15.98:0/339907307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778f8560 0x55ff7792bf40 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f293c00a040 tx=0x7f293c05d530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8323389cee28c4cb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:20.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.873+0000 7f291bfff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-03-03T08:03:20.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.874+0000 7f291bfff700 1 -- 172.21.15.98:0/339907307 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293c068de0 con 0x55ff778f8560 2023-03-03T08:03:20.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.874+0000 7f291bfff700 1 -- 172.21.15.98:0/339907307 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f293c05d0a0 con 0x55ff778f8560 2023-03-03T08:03:20.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.874+0000 7f295b0adb80 1 -- 172.21.15.98:0/339907307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ff778f0c50 con 0x55ff778f8560 2023-03-03T08:03:20.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.874+0000 7f291bfff700 1 -- 172.21.15.98:0/339907307 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7f293c05b070 con 0x55ff778f8560 2023-03-03T08:03:20.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.874+0000 7f295b0adb80 10 client.8124 fetch_fsmap learned FSMap version 62 2023-03-03T08:03:20.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.874+0000 7f295b0adb80 1 -- 172.21.15.98:0/339907307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0x55ff7767fb10 con 0x55ff778f8560 2023-03-03T08:03:20.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.874+0000 7f291bfff700 1 -- 172.21.15.98:0/339907307 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ff778f0c50 con 0x55ff778f8560 2023-03-03T08:03:20.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.874+0000 7f291bfff700 1 -- 172.21.15.98:0/339907307 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 62) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f293c083170 con 0x55ff778f8560 2023-03-03T08:03:20.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.874+0000 7f295b0adb80 10 client.8124 fetch_fsmap finished waiting for FSMap version 62 2023-03-03T08:03:20.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.874+0000 7f295b0adb80 10 client.8124 resolve_mds: resolved name 'd' to daemon mds.d 2023-03-03T08:03:20.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.874+0000 7f295b0adb80 20 client.8124 populate_metadata read hostname 'smithi098' 2023-03-03T08:03:20.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.874+0000 7f295b0adb80 1 --2- 172.21.15.98:0/339907307 >> [v2:172.21.15.100:6833/2660525287,v1:172.21.15.100:6835/2660525287] conn(0x55ff77929c80 0x55ff7792a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.874+0000 7f295b0adb80 4 client.8124 mds_command: new command op to 8001 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-03T08:03:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.874+0000 7f295b0adb80 1 -- 172.21.15.98:0/339907307 --> [v2:172.21.15.100:6833/2660525287,v1:172.21.15.100:6835/2660525287] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ff7784a020 con 0x55ff77929c80 2023-03-03T08:03:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.875+0000 7f2944d9a700 1 --2- 172.21.15.98:0/339907307 >> [v2:172.21.15.100:6833/2660525287,v1:172.21.15.100:6835/2660525287] conn(0x55ff77929c80 0x55ff7792a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.876+0000 7f2944d9a700 1 --2- 172.21.15.98:0/339907307 >> [v2:172.21.15.100:6833/2660525287,v1:172.21.15.100:6835/2660525287] conn(0x55ff77929c80 0x55ff7792a050 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2697330686d1be43 server_cookie=f1c37955699b9d57 in_seq=0 out_seq=0 2023-03-03T08:03:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.876+0000 7f291bfff700 10 client.8124 ms_handle_connect on v2:172.21.15.100:6833/2660525287 2023-03-03T08:03:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.877+0000 7f291bfff700 1 -- 172.21.15.98:0/339907307 <== mds.0 v2:172.21.15.100:6833/2660525287 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55ff7784a020 con 0x55ff77929c80 2023-03-03T08:03:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.877+0000 7f291bfff700 10 client.8124 handle_command_reply: tid=0 2023-03-03T08:03:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.877+0000 7f295b0adb80 1 client.8124 shutdown 2023-03-03T08:03:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.877+0000 7f295b0adb80 1 -- 172.21.15.98:0/339907307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778f8560 msgr2=0x55ff7792bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.877+0000 7f295b0adb80 1 --2- 172.21.15.98:0/339907307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778f8560 0x55ff7792bf40 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f293c00a040 tx=0x7f293c05d530 comp rx=0 tx=0).stop 2023-03-03T08:03:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.877+0000 7f295b0adb80 1 -- 172.21.15.98:0/339907307 shutdown_connections 2023-03-03T08:03:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.877+0000 7f295b0adb80 1 --2- 172.21.15.98:0/339907307 >> [v2:172.21.15.100:6833/2660525287,v1:172.21.15.100:6835/2660525287] conn(0x55ff77929c80 0x55ff7792a050 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.877+0000 7f295b0adb80 1 --2- 172.21.15.98:0/339907307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778f8560 0x55ff7792bf40 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.877+0000 7f295b0adb80 1 --2- 172.21.15.98:0/339907307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778fda70 0x55ff77919820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.877+0000 7f295b0adb80 1 --2- 172.21.15.98:0/339907307 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778fccd0 0x55ff77946c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.877+0000 7f291bfff700 0 client.8124 ms_handle_reset on v2:172.21.15.100:6833/2660525287 2023-03-03T08:03:20.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.877+0000 7f295b0adb80 1 -- 172.21.15.98:0/339907307 >> 172.21.15.98:0/339907307 conn(0x55ff778fbc30 msgr2=0x55ff778f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:20.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.877+0000 7f295b0adb80 1 -- 172.21.15.98:0/339907307 shutdown_connections 2023-03-03T08:03:20.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.877+0000 7f295b0adb80 1 -- 172.21.15.98:0/339907307 wait complete. 2023-03-03T08:03:20.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.878+0000 7f295b0adb80 20 client.8124 trim_cache size 0 max 16384 2023-03-03T08:03:20.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.896+0000 7f295b0adb80 1 Processor -- start 2023-03-03T08:03:20.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.896+0000 7f295b0adb80 1 -- start start 2023-03-03T08:03:20.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.897+0000 7f295b0adb80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778f8560 0x55ff778fe660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.897+0000 7f295b0adb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778fccd0 0x55ff778febd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.897+0000 7f295b0adb80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778fda70 0x55ff778fc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.897+0000 7f295b0adb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff778f7c80 con 0x55ff778fccd0 2023-03-03T08:03:20.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.897+0000 7f295b0adb80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff778f3450 con 0x55ff778fda70 2023-03-03T08:03:20.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.897+0000 7f295b0adb80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ff778fab00 con 0x55ff778f8560 2023-03-03T08:03:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.897+0000 7f2944d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778fccd0 0x55ff778febd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.897+0000 7f2944d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778fccd0 0x55ff778febd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60266/0 (socket says 172.21.15.98:60266) 2023-03-03T08:03:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.897+0000 7f2944d9a700 1 -- 172.21.15.98:0/1478648031 learned_addr learned my addr 172.21.15.98:0/1478648031 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.898+0000 7f2945d9c700 1 --2- 172.21.15.98:0/1478648031 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778fda70 0x55ff778fc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.898+0000 7f294559b700 1 --2- 172.21.15.98:0/1478648031 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778f8560 0x55ff778fe660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.898+0000 7f2944d9a700 1 -- 172.21.15.98:0/1478648031 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778f8560 msgr2=0x55ff778fe660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.898+0000 7f2944d9a700 1 --2- 172.21.15.98:0/1478648031 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778f8560 0x55ff778fe660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.898+0000 7f2944d9a700 1 -- 172.21.15.98:0/1478648031 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778fda70 msgr2=0x55ff778fc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.898+0000 7f2944d9a700 1 --2- 172.21.15.98:0/1478648031 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778fda70 0x55ff778fc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.898+0000 7f2944d9a700 1 -- 172.21.15.98:0/1478648031 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff77818230 con 0x55ff778fccd0 2023-03-03T08:03:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.898+0000 7f2944d9a700 1 --2- 172.21.15.98:0/1478648031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778fccd0 0x55ff778febd0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f2930009650 tx=0x7f2930017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb808b9d7ac2324 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.898+0000 7f29237fe700 1 -- 172.21.15.98:0/1478648031 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2930028040 con 0x55ff778fccd0 2023-03-03T08:03:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.898+0000 7f29237fe700 1 -- 172.21.15.98:0/1478648031 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293002a380 con 0x55ff778fccd0 2023-03-03T08:03:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.898+0000 7f29237fe700 1 -- 172.21.15.98:0/1478648031 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2930015b00 con 0x55ff778fccd0 2023-03-03T08:03:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.899+0000 7f295b0adb80 1 -- 172.21.15.98:0/1478648031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778fccd0 msgr2=0x55ff778febd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.899+0000 7f295b0adb80 1 --2- 172.21.15.98:0/1478648031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778fccd0 0x55ff778febd0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f2930009650 tx=0x7f2930017000 comp rx=0 tx=0).stop 2023-03-03T08:03:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.899+0000 7f295b0adb80 1 -- 172.21.15.98:0/1478648031 shutdown_connections 2023-03-03T08:03:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.899+0000 7f295b0adb80 1 --2- 172.21.15.98:0/1478648031 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778fda70 0x55ff778fc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.899+0000 7f295b0adb80 1 --2- 172.21.15.98:0/1478648031 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778f8560 0x55ff778fe660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.899+0000 7f295b0adb80 1 --2- 172.21.15.98:0/1478648031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778fccd0 0x55ff778febd0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.899+0000 7f295b0adb80 1 -- 172.21.15.98:0/1478648031 >> 172.21.15.98:0/1478648031 conn(0x55ff778fbc30 msgr2=0x55ff778f4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.899+0000 7f295b0adb80 1 -- 172.21.15.98:0/1478648031 shutdown_connections 2023-03-03T08:03:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.899+0000 7f295b0adb80 1 -- 172.21.15.98:0/1478648031 wait complete. 2023-03-03T08:03:20.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.900+0000 7f295b0adb80 1 -- start start 2023-03-03T08:03:20.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.900+0000 7f295b0adb80 1 Processor -- start 2023-03-03T08:03:20.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.901+0000 7f295b0adb80 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778f8560 0x55ff77990e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.901+0000 7f295b0adb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778fccd0 0x55ff779abcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.901+0000 7f295b0adb80 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778fda70 0x55ff77928400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.901+0000 7f295b0adb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff77928ed0 con 0x55ff778fccd0 2023-03-03T08:03:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.901+0000 7f295b0adb80 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff7792ac50 con 0x55ff778f8560 2023-03-03T08:03:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.901+0000 7f295b0adb80 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ff779ab840 con 0x55ff778fda70 2023-03-03T08:03:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.901+0000 7f295b0adb80 10 client.0 _ll_register_callbacks cb 0x55ff777ef1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T08:03:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.902+0000 7f2944d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778fccd0 0x55ff779abcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.902+0000 7f2944d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778fccd0 0x55ff779abcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60276/0 (socket says 172.21.15.98:60276) 2023-03-03T08:03:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.902+0000 7f2944d9a700 1 -- 172.21.15.98:0/1123392887 learned_addr learned my addr 172.21.15.98:0/1123392887 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.902+0000 7f294559b700 1 --2- 172.21.15.98:0/1123392887 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778f8560 0x55ff77990e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.902+0000 7f2945d9c700 1 --2- 172.21.15.98:0/1123392887 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778fda70 0x55ff77928400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.902+0000 7f2944d9a700 1 -- 172.21.15.98:0/1123392887 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778fda70 msgr2=0x55ff77928400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.902+0000 7f2944d9a700 1 --2- 172.21.15.98:0/1123392887 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778fda70 0x55ff77928400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.902+0000 7f2944d9a700 1 -- 172.21.15.98:0/1123392887 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778f8560 msgr2=0x55ff77990e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.902+0000 7f2944d9a700 1 --2- 172.21.15.98:0/1123392887 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778f8560 0x55ff77990e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.902+0000 7f2944d9a700 1 -- 172.21.15.98:0/1123392887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2930011ab0 con 0x55ff778fccd0 2023-03-03T08:03:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.902+0000 7f294559b700 1 --2- 172.21.15.98:0/1123392887 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778f8560 0x55ff77990e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.902+0000 7f2944d9a700 1 --2- 172.21.15.98:0/1123392887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778fccd0 0x55ff779abcd0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f294006ac70 tx=0x7f2930004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a20294b6270f23 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.902+0000 7f291bfff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-03-03T08:03:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.903+0000 7f291bfff700 1 -- 172.21.15.98:0/1123392887 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2930028050 con 0x55ff778fccd0 2023-03-03T08:03:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.903+0000 7f295b0adb80 1 -- 172.21.15.98:0/1123392887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2938003b80 con 0x55ff778fccd0 2023-03-03T08:03:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.903+0000 7f291bfff700 1 -- 172.21.15.98:0/1123392887 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f293001b040 con 0x55ff778fccd0 2023-03-03T08:03:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.903+0000 7f291bfff700 1 -- 172.21.15.98:0/1123392887 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2930015dd0 con 0x55ff778fccd0 2023-03-03T08:03:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.903+0000 7f295b0adb80 10 client.8130 fetch_fsmap learned FSMap version 62 2023-03-03T08:03:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.903+0000 7f295b0adb80 1 -- 172.21.15.98:0/1123392887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0x55ff77668990 con 0x55ff778fccd0 2023-03-03T08:03:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.903+0000 7f291bfff700 1 -- 172.21.15.98:0/1123392887 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2938003b80 con 0x55ff778fccd0 2023-03-03T08:03:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.903+0000 7f291bfff700 1 -- 172.21.15.98:0/1123392887 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 62) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f293002a7e0 con 0x55ff778fccd0 2023-03-03T08:03:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.903+0000 7f295b0adb80 10 client.8130 fetch_fsmap finished waiting for FSMap version 62 2023-03-03T08:03:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.903+0000 7f295b0adb80 10 client.8130 resolve_mds: resolved name 'd' to daemon mds.d 2023-03-03T08:03:20.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.903+0000 7f295b0adb80 20 client.8130 populate_metadata read hostname 'smithi098' 2023-03-03T08:03:20.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.903+0000 7f295b0adb80 1 --2- 172.21.15.98:0/1123392887 >> [v2:172.21.15.100:6833/2660525287,v1:172.21.15.100:6835/2660525287] conn(0x55ff778fc000 0x55ff778fc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:20.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.903+0000 7f295b0adb80 4 client.8130 mds_command: new command op to 8001 tid=0[{"prefix": "status"}] 2023-03-03T08:03:20.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.903+0000 7f295b0adb80 1 -- 172.21.15.98:0/1123392887 --> [v2:172.21.15.100:6833/2660525287,v1:172.21.15.100:6835/2660525287] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ff776a0ca0 con 0x55ff778fc000 2023-03-03T08:03:20.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.904+0000 7f294559b700 1 --2- 172.21.15.98:0/1123392887 >> [v2:172.21.15.100:6833/2660525287,v1:172.21.15.100:6835/2660525287] conn(0x55ff778fc000 0x55ff778fc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:20.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.904+0000 7f294559b700 1 --2- 172.21.15.98:0/1123392887 >> [v2:172.21.15.100:6833/2660525287,v1:172.21.15.100:6835/2660525287] conn(0x55ff778fc000 0x55ff778fc3d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=705d17861df41a3c server_cookie=9af58913f0214646 in_seq=0 out_seq=0 2023-03-03T08:03:20.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.904+0000 7f291bfff700 10 client.8130 ms_handle_connect on v2:172.21.15.100:6833/2660525287 2023-03-03T08:03:20.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.905+0000 7f291bfff700 1 -- 172.21.15.98:0/1123392887 <== mds.0 v2:172.21.15.100:6833/2660525287 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55ff776a0ca0 con 0x55ff778fc000 2023-03-03T08:03:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.905+0000 7f291bfff700 10 client.8130 handle_command_reply: tid=0 2023-03-03T08:03:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.905+0000 7f295b0adb80 1 client.8130 shutdown 2023-03-03T08:03:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.905+0000 7f295b0adb80 1 -- 172.21.15.98:0/1123392887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778fccd0 msgr2=0x55ff779abcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.905+0000 7f295b0adb80 1 --2- 172.21.15.98:0/1123392887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778fccd0 0x55ff779abcd0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f294006ac70 tx=0x7f2930004750 comp rx=0 tx=0).stop 2023-03-03T08:03:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.905+0000 7f295b0adb80 1 -- 172.21.15.98:0/1123392887 shutdown_connections 2023-03-03T08:03:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.905+0000 7f295b0adb80 1 --2- 172.21.15.98:0/1123392887 >> [v2:172.21.15.100:6833/2660525287,v1:172.21.15.100:6835/2660525287] conn(0x55ff778fc000 0x55ff778fc3d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.905+0000 7f295b0adb80 1 --2- 172.21.15.98:0/1123392887 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ff778fda70 0x55ff77928400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.905+0000 7f295b0adb80 1 --2- 172.21.15.98:0/1123392887 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ff778f8560 0x55ff77990e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.905+0000 7f291bfff700 0 client.8130 ms_handle_reset on v2:172.21.15.100:6833/2660525287 2023-03-03T08:03:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.905+0000 7f295b0adb80 1 --2- 172.21.15.98:0/1123392887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ff778fccd0 0x55ff779abcd0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.905+0000 7f295b0adb80 1 -- 172.21.15.98:0/1123392887 >> 172.21.15.98:0/1123392887 conn(0x55ff778fbc30 msgr2=0x55ff778fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.905+0000 7f295b0adb80 1 -- 172.21.15.98:0/1123392887 shutdown_connections 2023-03-03T08:03:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.905+0000 7f295b0adb80 1 -- 172.21.15.98:0/1123392887 wait complete. 2023-03-03T08:03:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.906+0000 7f295b0adb80 20 client.8130 trim_cache size 0 max 16384 2023-03-03T08:03:20.915 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:03:20.915 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "fbbd322f-236a-44f8-96c5-be6963846c9b", 2023-03-03T08:03:20.915 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2023-03-03T08:03:20.916 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 8001, 2023-03-03T08:03:20.916 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-03-03T08:03:20.916 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-03-03T08:03:20.916 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-03-03T08:03:20.916 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 7.3963911529999997, 2023-03-03T08:03:20.916 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 62, 2023-03-03T08:03:20.916 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 111, 2023-03-03T08:03:20.917 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 111, 2023-03-03T08:03:20.917 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 13.200334135 2023-03-03T08:03:20.917 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:03:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.906+0000 7f29477ff700 1 -- 172.21.15.98:0/2120234797 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2924068380 msgr2=0x7f292406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.906+0000 7f29477ff700 1 --2- 172.21.15.98:0/2120234797 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2924068380 0x7f292406a830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2940159d70 tx=0x7f2930008040 comp rx=0 tx=0).stop 2023-03-03T08:03:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.906+0000 7f29477ff700 1 -- 172.21.15.98:0/2120234797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294006b4b0 msgr2=0x7f29400b7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.906+0000 7f29477ff700 1 --2- 172.21.15.98:0/2120234797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294006b4b0 0x7f29400b7d00 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f293c00c4e0 tx=0x7f293c00c7f0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.906+0000 7f29477ff700 1 -- 172.21.15.98:0/2120234797 shutdown_connections 2023-03-03T08:03:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.906+0000 7f29477ff700 1 --2- 172.21.15.98:0/2120234797 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2924068380 0x7f292406a830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.906+0000 7f29477ff700 1 --2- 172.21.15.98:0/2120234797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294006b4b0 0x7f29400b7d00 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.906+0000 7f29477ff700 1 --2- 172.21.15.98:0/2120234797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294015ca70 0x7f2940151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.906+0000 7f29477ff700 1 --2- 172.21.15.98:0/2120234797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29400d0270 0x7f29400ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.906+0000 7f29477ff700 1 -- 172.21.15.98:0/2120234797 >> 172.21.15.98:0/2120234797 conn(0x7f2940073e70 msgr2=0x7f29400b5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.906+0000 7f29477ff700 1 -- 172.21.15.98:0/2120234797 shutdown_connections 2023-03-03T08:03:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:20.906+0000 7f29477ff700 1 -- 172.21.15.98:0/2120234797 wait complete. 2023-03-03T08:03:20.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T08:03:21.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.179+0000 7f77e95ff700 1 Processor -- start 2023-03-03T08:03:21.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.179+0000 7f77e95ff700 1 -- start start 2023-03-03T08:03:21.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.180+0000 7f77e95ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77e40b87c0 0x7f77e40cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:21.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.180+0000 7f77e95ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f77e40b5410 0x7f77e40d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:21.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.180+0000 7f77e95ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e40d0ef0 con 0x7f77e40b8b90 2023-03-03T08:03:21.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.180+0000 7f77e95ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e40d0600 con 0x7f77e40b5410 2023-03-03T08:03:21.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.180+0000 7f77e95ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40cd970 con 0x7f77e40b87c0 2023-03-03T08:03:21.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.181+0000 7f77e27fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f77e40b8b90 0x7f77e40b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42698/0 (socket says 172.21.15.98:42698) 2023-03-03T08:03:21.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.181+0000 7f77e27fc700 1 -- 172.21.15.98:0/1787180935 learned_addr learned my addr 172.21.15.98:0/1787180935 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:21.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.181+0000 7f77e37fe700 1 --2- 172.21.15.98:0/1787180935 >> v2:172.21.15.100:3300/0 conn(0x7f77e40b5410 0x7f77e40d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:21.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.181+0000 7f77e2ffd700 1 --2- 172.21.15.98:0/1787180935 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77e40b87c0 0x7f77e40cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:21.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.181+0000 7f77e1ffb700 1 -- 172.21.15.98:0/1787180935 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674018628 0 0) 0x7f77e40d0ef0 con 0x7f77e40b8b90 2023-03-03T08:03:21.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.181+0000 7f77e1ffb700 1 -- 172.21.15.98:0/1787180935 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77cc0035b0 con 0x7f77e40b8b90 2023-03-03T08:03:21.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.182+0000 7f77e37fe700 1 -- 172.21.15.98:0/1787180935 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77e40b87c0 msgr2=0x7f77e40cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:21.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.182+0000 7f77e1ffb700 1 -- 172.21.15.98:0/1787180935 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3455082144 0 0) 0x7f77cc0035b0 con 0x7f77e40b8b90 2023-03-03T08:03:21.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.182+0000 7f77e37fe700 1 --2- 172.21.15.98:0/1787180935 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77e40b87c0 0x7f77e40cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:21.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.182+0000 7f77e37fe700 1 -- 172.21.15.98:0/1787180935 >> v1:172.21.15.98:6789/0 conn(0x7f77e40b8b90 legacy=0x7f77e40b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:21.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.182+0000 7f77e37fe700 1 -- 172.21.15.98:0/1787180935 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e40cdc50 con 0x7f77e40b5410 2023-03-03T08:03:21.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.182+0000 7f77e37fe700 1 --2- 172.21.15.98:0/1787180935 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f77e40b5410 0x7f77e40d13d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009d90 tx=0x7f77dc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77f385c249d8f502 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:21.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.182+0000 7f77e1ffb700 1 -- 172.21.15.98:0/1787180935 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc00e070 con 0x7f77e40b5410 2023-03-03T08:03:21.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.182+0000 7f77e1ffb700 1 -- 172.21.15.98:0/1787180935 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77dc003a20 con 0x7f77e40b5410 2023-03-03T08:03:21.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.182+0000 7f77e1ffb700 1 -- 172.21.15.98:0/1787180935 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d8001670 con 0x7f77e40b5410 2023-03-03T08:03:21.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.183+0000 7f77e95ff700 1 -- 172.21.15.98:0/1787180935 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f77e40b5410 msgr2=0x7f77e40d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:21.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.183+0000 7f77e95ff700 1 --2- 172.21.15.98:0/1787180935 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f77e40b5410 0x7f77e40d13d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009d90 tx=0x7f77dc00b330 comp rx=0 tx=0).stop 2023-03-03T08:03:21.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.183+0000 7f77e95ff700 1 -- 172.21.15.98:0/1787180935 shutdown_connections 2023-03-03T08:03:21.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.183+0000 7f77e95ff700 1 --2- 172.21.15.98:0/1787180935 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f77e40b5410 0x7f77e40d13d0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:21.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.183+0000 7f77e95ff700 1 --2- 172.21.15.98:0/1787180935 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77e40b87c0 0x7f77e40cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:21.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.183+0000 7f77e95ff700 1 -- 172.21.15.98:0/1787180935 >> 172.21.15.98:0/1787180935 conn(0x7f77e406aee0 msgr2=0x7f77e406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:21.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.183+0000 7f77e95ff700 1 -- 172.21.15.98:0/1787180935 shutdown_connections 2023-03-03T08:03:21.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.183+0000 7f77e95ff700 1 -- 172.21.15.98:0/1787180935 wait complete. 2023-03-03T08:03:21.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.185+0000 7f77e95ff700 1 Processor -- start 2023-03-03T08:03:21.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.185+0000 7f77e95ff700 1 -- start start 2023-03-03T08:03:21.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.186+0000 7f77e95ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77e40b5410 0x7f77e4157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:21.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.186+0000 7f77e95ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f77e40b87c0 0x7f77e4158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:21.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.186+0000 7f77e95ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77e40b8b90 0x7f77e40d1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:21.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.186+0000 7f77e95ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e4158910 con 0x7f77e40b5410 2023-03-03T08:03:21.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.186+0000 7f77e95ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e4158bf0 con 0x7f77e40b87c0 2023-03-03T08:03:21.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.186+0000 7f77e95ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e4158ed0 con 0x7f77e40b8b90 2023-03-03T08:03:21.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.187+0000 7f77e2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77e40b5410 0x7f77e4157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:21.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.187+0000 7f77e2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77e40b5410 0x7f77e4157840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60280/0 (socket says 172.21.15.98:60280) 2023-03-03T08:03:21.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.187+0000 7f77e2ffd700 1 -- 172.21.15.98:0/4079869315 learned_addr learned my addr 172.21.15.98:0/4079869315 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:21.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.187+0000 7f77e27fc700 1 --2- 172.21.15.98:0/4079869315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f77e40b87c0 0x7f77e4158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:21.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.187+0000 7f77e37fe700 1 --2- 172.21.15.98:0/4079869315 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77e40b8b90 0x7f77e40d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:21.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.187+0000 7f77e2ffd700 1 -- 172.21.15.98:0/4079869315 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77e40b8b90 msgr2=0x7f77e40d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:21.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.187+0000 7f77e2ffd700 1 --2- 172.21.15.98:0/4079869315 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77e40b8b90 0x7f77e40d1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:21.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.187+0000 7f77e2ffd700 1 -- 172.21.15.98:0/4079869315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f77e40b87c0 msgr2=0x7f77e4158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:21.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.187+0000 7f77e2ffd700 1 --2- 172.21.15.98:0/4079869315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f77e40b87c0 0x7f77e4158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:21.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.187+0000 7f77e2ffd700 1 -- 172.21.15.98:0/4079869315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77dc009a70 con 0x7f77e40b5410 2023-03-03T08:03:21.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.187+0000 7f77e37fe700 1 --2- 172.21.15.98:0/4079869315 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77e40b8b90 0x7f77e40d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:21.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.188+0000 7f77e2ffd700 1 --2- 172.21.15.98:0/4079869315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77e40b5410 0x7f77e4157840 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f77d400e9b0 tx=0x7f77d400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:21.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.188+0000 7f77cb7fe700 1 -- 172.21.15.98:0/4079869315 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d4009710 con 0x7f77e40b5410 2023-03-03T08:03:21.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.188+0000 7f77cb7fe700 1 -- 172.21.15.98:0/4079869315 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d400c850 con 0x7f77e40b5410 2023-03-03T08:03:21.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.188+0000 7f77e95ff700 1 -- 172.21.15.98:0/4079869315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e40cdf30 con 0x7f77e40b5410 2023-03-03T08:03:21.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.188+0000 7f77cb7fe700 1 -- 172.21.15.98:0/4079869315 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d400cbc0 con 0x7f77e40b5410 2023-03-03T08:03:21.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.188+0000 7f77e95ff700 1 -- 172.21.15.98:0/4079869315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e4073a80 con 0x7f77e40b5410 2023-03-03T08:03:21.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.189+0000 7f77e95ff700 1 -- 172.21.15.98:0/4079869315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e40ce0c0 con 0x7f77e40b5410 2023-03-03T08:03:21.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.190+0000 7f77cb7fe700 1 -- 172.21.15.98:0/4079869315 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f77d4019030 con 0x7f77e40b5410 2023-03-03T08:03:21.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.192+0000 7f77cb7fe700 1 --2- 172.21.15.98:0/4079869315 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f77d0068380 0x7f77d006a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:21.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.192+0000 7f77cb7fe700 1 -- 172.21.15.98:0/4079869315 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f77d4085d40 con 0x7f77e40b5410 2023-03-03T08:03:21.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.192+0000 7f77e27fc700 1 --2- 172.21.15.98:0/4079869315 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f77d0068380 0x7f77d006a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:21.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.193+0000 7f77e27fc700 1 --2- 172.21.15.98:0/4079869315 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f77d0068380 0x7f77d006a830 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f77e41f55f0 tx=0x7f77d8007ba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:21.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.196+0000 7f77cb7fe700 1 -- 172.21.15.98:0/4079869315 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f77d40524c0 con 0x7f77e40b5410 2023-03-03T08:03:21.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.431+0000 7f77e95ff700 1 -- 172.21.15.98:0/4079869315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f77e40729b0 con 0x7f77e40b5410 2023-03-03T08:03:21.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.432+0000 7f77cb7fe700 1 -- 172.21.15.98:0/4079869315 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v62) v1 ==== 53+0+73 (secure 0 0 0) 0x7f77d4017070 con 0x7f77e40b5410 2023-03-03T08:03:21.434 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T08:03:21.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.434+0000 7f77e95ff700 1 -- 172.21.15.98:0/4079869315 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f77d0068380 msgr2=0x7f77d006a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:21.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.434+0000 7f77e95ff700 1 --2- 172.21.15.98:0/4079869315 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f77d0068380 0x7f77d006a830 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f77e41f55f0 tx=0x7f77d8007ba0 comp rx=0 tx=0).stop 2023-03-03T08:03:21.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.434+0000 7f77e95ff700 1 -- 172.21.15.98:0/4079869315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77e40b5410 msgr2=0x7f77e4157840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:21.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.434+0000 7f77e95ff700 1 --2- 172.21.15.98:0/4079869315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77e40b5410 0x7f77e4157840 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f77d400e9b0 tx=0x7f77d400ecc0 comp rx=0 tx=0).stop 2023-03-03T08:03:21.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.434+0000 7f77e95ff700 1 -- 172.21.15.98:0/4079869315 shutdown_connections 2023-03-03T08:03:21.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.434+0000 7f77e95ff700 1 --2- 172.21.15.98:0/4079869315 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f77d0068380 0x7f77d006a830 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:21.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.434+0000 7f77e95ff700 1 --2- 172.21.15.98:0/4079869315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77e40b5410 0x7f77e4157840 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:21.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.434+0000 7f77e95ff700 1 --2- 172.21.15.98:0/4079869315 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f77e40b8b90 0x7f77e40d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:21.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.434+0000 7f77e95ff700 1 --2- 172.21.15.98:0/4079869315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f77e40b87c0 0x7f77e4158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:21.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.435+0000 7f77e95ff700 1 -- 172.21.15.98:0/4079869315 >> 172.21.15.98:0/4079869315 conn(0x7f77e406aee0 msgr2=0x7f77e40b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:21.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.435+0000 7f77e95ff700 1 -- 172.21.15.98:0/4079869315 shutdown_connections 2023-03-03T08:03:21.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:21.435+0000 7f77e95ff700 1 -- 172.21.15.98:0/4079869315 wait complete. 2023-03-03T08:03:21.463 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T08:03:21.464 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T08:03:21.464 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2023-03-03T08:03:21.464 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-03-03T08:03:21.465 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-03-03T08:03:21.465 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T08:03:21.465 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-03-03T08:03:21.465 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T08:03:21.466 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T08:03:21.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:21.467 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2023-03-03T08:03:21.537 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T08:03:21.538 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T08:03:21.538 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T08:03:21.538 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:03:21.538 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-03-03T08:03:21.539 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:03:21.539 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:03:21.539 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:03:21.539 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2023-03-03T08:03:21.539 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:03:21.539 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:03:21.539 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2023-03-03T08:03:21.540 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:03:21.540 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2023-03-03T08:03:21.541 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T08:03:21.541 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2023-03-03T08:03:21.541 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T08:03:21.541 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 41754sec preferred_lft 41754sec 2023-03-03T08:03:21.541 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2023-03-03T08:03:21.542 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:03:21.542 INFO:teuthology.orchestra.run.smithi098.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T08:03:21.542 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 46:3f:9e:55:37:22 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:03:21.542 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-03-03T08:03:21.542 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:03:21.543 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::1857:c2ff:fe19:346f/64 scope link 2023-03-03T08:03:21.543 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:03:21.543 INFO:teuthology.orchestra.run.smithi098.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:03:21.543 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 46:3f:9e:55:37:22 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-03-03T08:03:21.543 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::443f:9eff:fe55:3722/64 scope link 2023-03-03T08:03:21.543 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:03:21.543 INFO:teuthology.orchestra.run.smithi098.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:03:21.544 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether d6:19:97:a6:7c:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-03-03T08:03:21.544 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::d419:97ff:fea6:7c83/64 scope link 2023-03-03T08:03:21.544 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:03:21.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:21.546 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T08:03:21.642 INFO:teuthology.orchestra.run.smithi098.stdout:1 2023-03-03T08:03:21.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:21.648 DEBUG:teuthology.orchestra.run.smithi098:> route 2023-03-03T08:03:21.719 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2023-03-03T08:03:21.720 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T08:03:21.720 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T08:03:21.720 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T08:03:21.720 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T08:03:21.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:21.722 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-03-03T08:03:21.722 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-03-03T08:03:21.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T08:03:21.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T08:03:21.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T08:03:21.724 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-03-03T08:03:21.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:21.964 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-03-03T08:03:22.032 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-03-03T08:03:22.033 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-03-03T08:03:22.034 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T08:03:22.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:22.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2023-03-03T08:03:22.136 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-03-03T08:03:22.136 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:03:22.137 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-03-03T08:03:22.205 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-03-03T08:03:22.205 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:03:22.206 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-03-03T08:03:22.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T08:03:22.374 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:03:22.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:03:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:03:22.475 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:03:22.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:22.476 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T08:03:22.544 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-03-03T08:03:22.545 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/) 2023-03-03T08:03:22.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-03-03T08:03:22.655 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:03:22.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:03:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:03:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:03:22.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:22.760 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-03-03T08:03:22.773 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:2023-03-03T08:03:22.771+0000 7ff5b36945c0 -1 init, newargv = 0x55ac979cb110 newargc=15 2023-03-03T08:03:22.773 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[148567]: starting ceph client 2023-03-03T08:03:22.784 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[148567]: starting fuse 2023-03-03T08:03:22.835 INFO:teuthology.orchestra.run.smithi098.stdout:42 2023-03-03T08:03:22.836 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2023-03-03T08:03:22.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T08:03:22.837 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2023-03-03T08:03:22.837 DEBUG:teuthology.orchestra.run.smithi098:> import re 2023-03-03T08:03:22.837 DEBUG:teuthology.orchestra.run.smithi098:> import os 2023-03-03T08:03:22.838 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2023-03-03T08:03:22.838 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:03:22.838 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2023-03-03T08:03:22.839 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-03-03T08:03:22.839 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2023-03-03T08:03:22.839 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-03-03T08:03:22.839 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:03:22.840 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2023-03-03T08:03:22.840 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2023-03-03T08:03:22.840 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2023-03-03T08:03:22.841 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2023-03-03T08:03:22.841 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:03:22.842 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2023-03-03T08:03:22.842 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T08:03:22.843 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T08:03:22.844 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T08:03:22.844 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2023-03-03T08:03:22.845 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2023-03-03T08:03:22.845 DEBUG:teuthology.orchestra.run.smithi098:> return f 2023-03-03T08:03:22.846 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T08:03:22.847 DEBUG:teuthology.orchestra.run.smithi098:> 2023-03-03T08:03:22.848 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.0")) 2023-03-03T08:03:22.848 DEBUG:teuthology.orchestra.run.smithi098:> ' 2023-03-03T08:03:23.063 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.0.148567.asok 2023-03-03T08:03:23.241 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.148567.asok 2023-03-03T08:03:23.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:23.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.148567.asok status 2023-03-03T08:03:23.509 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:03:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2023-03-03T08:03:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T08:03:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T08:03:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "0", 2023-03-03T08:03:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2023-03-03T08:03:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-03-03T08:03:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "148567", 2023-03-03T08:03:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2023-03-03T08:03:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2023-03-03T08:03:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2023-03-03T08:03:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 8142, 2023-03-03T08:03:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2023-03-03T08:03:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2023-03-03T08:03:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2023-03-03T08:03:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 8142 2023-03-03T08:03:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T08:03:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T08:03:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:03:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1753905538 2023-03-03T08:03:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-03-03T08:03:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-03-03T08:03:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:03:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1753905538 2023-03-03T08:03:23.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:23.516 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.8142 192.168.0.1:0/1753905538", 2023-03-03T08:03:23.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/1753905538", 2023-03-03T08:03:23.516 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2023-03-03T08:03:23.517 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 62, 2023-03-03T08:03:23.517 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 111, 2023-03-03T08:03:23.517 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2023-03-03T08:03:23.517 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2023-03-03T08:03:23.518 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2023-03-03T08:03:23.518 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:03:23.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:23.531 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2023-03-03T08:03:23.531 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.0 2023-03-03T08:03:23.601 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2023-03-03T08:03:23.602 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-03-03T08:03:23.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:23.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-03-03T08:03:23.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T08:03:23.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.948+0000 7f91e7bff700 1 Processor -- start 2023-03-03T08:03:23.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.949+0000 7f91e7bff700 1 -- start start 2023-03-03T08:03:23.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.950+0000 7f91e7bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f91e00b8b90 0x7f91e00cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:23.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.950+0000 7f91e7bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f91e00b5410 0x7f91e00d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:23.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.950+0000 7f91e7bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e00d0ef0 con 0x7f91e00b87c0 2023-03-03T08:03:23.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.950+0000 7f91e7bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e00d0600 con 0x7f91e00b8b90 2023-03-03T08:03:23.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.950+0000 7f91e7bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e00cd970 con 0x7f91e00b5410 2023-03-03T08:03:23.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.950+0000 7f91e519a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f91e00b8b90 0x7f91e00cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:23.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.950+0000 7f91e599b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f91e00b87c0 0x7f91e00b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42714/0 (socket says 172.21.15.98:42714) 2023-03-03T08:03:23.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.950+0000 7f91e599b700 1 -- 172.21.15.98:0/2774868432 learned_addr learned my addr 172.21.15.98:0/2774868432 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:23.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.951+0000 7f91e619c700 1 --2- 172.21.15.98:0/2774868432 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f91e00b5410 0x7f91e00d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:23.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.951+0000 7f91e519a700 1 -- 172.21.15.98:0/2774868432 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f91e00b5410 msgr2=0x7f91e00d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:23.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.951+0000 7f91e519a700 1 --2- 172.21.15.98:0/2774868432 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f91e00b5410 0x7f91e00d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:23.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.951+0000 7f91e519a700 1 -- 172.21.15.98:0/2774868432 >> v1:172.21.15.98:6789/0 conn(0x7f91e00b87c0 legacy=0x7f91e00b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:23.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.951+0000 7f91e519a700 1 -- 172.21.15.98:0/2774868432 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e00cdc50 con 0x7f91e00b8b90 2023-03-03T08:03:23.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.951+0000 7f91e4999700 1 -- 172.21.15.98:0/2774868432 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339178205 0 0) 0x7f91e00d0ef0 con 0x7f91e00b87c0 2023-03-03T08:03:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.951+0000 7f91e519a700 1 --2- 172.21.15.98:0/2774868432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91e00b8b90 0x7f91e00cffe0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009840 tx=0x7f91d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439dfe4ae4ccecfd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.952+0000 7f91e4999700 1 -- 172.21.15.98:0/2774868432 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d0011830 con 0x7f91e00b8b90 2023-03-03T08:03:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.952+0000 7f91e4999700 1 -- 172.21.15.98:0/2774868432 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d0011990 con 0x7f91e00b8b90 2023-03-03T08:03:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.952+0000 7f91e4999700 1 -- 172.21.15.98:0/2774868432 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d0011d20 con 0x7f91e00b8b90 2023-03-03T08:03:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.952+0000 7f91e7bff700 1 -- 172.21.15.98:0/2774868432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91e00b8b90 msgr2=0x7f91e00cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.952+0000 7f91e7bff700 1 --2- 172.21.15.98:0/2774868432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91e00b8b90 0x7f91e00cffe0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009840 tx=0x7f91d0009b50 comp rx=0 tx=0).stop 2023-03-03T08:03:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.952+0000 7f91e7bff700 1 -- 172.21.15.98:0/2774868432 shutdown_connections 2023-03-03T08:03:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.952+0000 7f91e7bff700 1 --2- 172.21.15.98:0/2774868432 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f91e00b5410 0x7f91e00d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.952+0000 7f91e7bff700 1 --2- 172.21.15.98:0/2774868432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91e00b8b90 0x7f91e00cffe0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.952+0000 7f91e7bff700 1 -- 172.21.15.98:0/2774868432 >> 172.21.15.98:0/2774868432 conn(0x7f91e006aee0 msgr2=0x7f91e006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.952+0000 7f91e7bff700 1 -- 172.21.15.98:0/2774868432 shutdown_connections 2023-03-03T08:03:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.952+0000 7f91e7bff700 1 -- 172.21.15.98:0/2774868432 wait complete. 2023-03-03T08:03:23.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.954+0000 7f91e7bff700 1 Processor -- start 2023-03-03T08:03:23.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.955+0000 7f91e7bff700 1 -- start start 2023-03-03T08:03:23.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.955+0000 7f91e7bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91e00b5410 0x7f91e015abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:23.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.955+0000 7f91e7bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f91e00b87c0 0x7f91e00cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.956+0000 7f91e7bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f91e00b8b90 0x7f91e00aa840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.956+0000 7f91e7bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e015b0f0 con 0x7f91e00b5410 2023-03-03T08:03:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.956+0000 7f91e7bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e00ca7f0 con 0x7f91e00b87c0 2023-03-03T08:03:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.956+0000 7f91e7bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e00caad0 con 0x7f91e00b8b90 2023-03-03T08:03:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.956+0000 7f91e599b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91e00b5410 0x7f91e015abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.956+0000 7f91e599b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91e00b5410 0x7f91e015abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60296/0 (socket says 172.21.15.98:60296) 2023-03-03T08:03:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.956+0000 7f91e599b700 1 -- 172.21.15.98:0/641084227 learned_addr learned my addr 172.21.15.98:0/641084227 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.956+0000 7f91e619c700 1 --2- 172.21.15.98:0/641084227 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f91e00b8b90 0x7f91e00aa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.956+0000 7f91e519a700 1 --2- 172.21.15.98:0/641084227 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f91e00b87c0 0x7f91e00cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.956+0000 7f91e599b700 1 -- 172.21.15.98:0/641084227 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f91e00b8b90 msgr2=0x7f91e00aa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.956+0000 7f91e599b700 1 --2- 172.21.15.98:0/641084227 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f91e00b8b90 0x7f91e00aa840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.956+0000 7f91e599b700 1 -- 172.21.15.98:0/641084227 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f91e00b87c0 msgr2=0x7f91e00cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.956+0000 7f91e599b700 1 --2- 172.21.15.98:0/641084227 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f91e00b87c0 0x7f91e00cb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.956+0000 7f91e599b700 1 -- 172.21.15.98:0/641084227 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d00094a0 con 0x7f91e00b5410 2023-03-03T08:03:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.957+0000 7f91e599b700 1 --2- 172.21.15.98:0/641084227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91e00b5410 0x7f91e015abb0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0011e0 tx=0x7f91dc00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.957+0000 7f91d67fc700 1 -- 172.21.15.98:0/641084227 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc004c80 con 0x7f91e00b5410 2023-03-03T08:03:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.957+0000 7f91d67fc700 1 -- 172.21.15.98:0/641084227 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91dc004e00 con 0x7f91e00b5410 2023-03-03T08:03:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.957+0000 7f91d67fc700 1 -- 172.21.15.98:0/641084227 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc007310 con 0x7f91e00b5410 2023-03-03T08:03:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.957+0000 7f91e7bff700 1 -- 172.21.15.98:0/641084227 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e00cdf30 con 0x7f91e00b5410 2023-03-03T08:03:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.957+0000 7f91e7bff700 1 -- 172.21.15.98:0/641084227 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e0073a80 con 0x7f91e00b5410 2023-03-03T08:03:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.958+0000 7f91e7bff700 1 -- 172.21.15.98:0/641084227 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e00ce0c0 con 0x7f91e00b5410 2023-03-03T08:03:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.960+0000 7f91d67fc700 1 -- 172.21.15.98:0/641084227 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f91dc01a020 con 0x7f91e00b5410 2023-03-03T08:03:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.962+0000 7f91d67fc700 1 --2- 172.21.15.98:0/641084227 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f91c4068380 0x7f91c406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.962+0000 7f91d67fc700 1 -- 172.21.15.98:0/641084227 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f91dc0858c0 con 0x7f91e00b5410 2023-03-03T08:03:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.962+0000 7f91e519a700 1 --2- 172.21.15.98:0/641084227 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f91c4068380 0x7f91c406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.963+0000 7f91e519a700 1 --2- 172.21.15.98:0/641084227 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f91c4068380 0x7f91c406a830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f91e00b8fd0 tx=0x7f91d001c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:23.966+0000 7f91d67fc700 1 -- 172.21.15.98:0/641084227 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f91dc051fe0 con 0x7f91e00b5410 2023-03-03T08:03:24.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:24.200+0000 7f91e7bff700 1 -- 172.21.15.98:0/641084227 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f91e00729b0 con 0x7f91e00b5410 2023-03-03T08:03:24.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:24.201+0000 7f91d67fc700 1 -- 172.21.15.98:0/641084227 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v62) v1 ==== 53+0+73 (secure 0 0 0) 0x7f91dc083da0 con 0x7f91e00b5410 2023-03-03T08:03:24.203 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T08:03:24.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:24.203+0000 7f91e7bff700 1 -- 172.21.15.98:0/641084227 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f91c4068380 msgr2=0x7f91c406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:24.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:24.203+0000 7f91e7bff700 1 --2- 172.21.15.98:0/641084227 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f91c4068380 0x7f91c406a830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f91e00b8fd0 tx=0x7f91d001c040 comp rx=0 tx=0).stop 2023-03-03T08:03:24.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:24.203+0000 7f91e7bff700 1 -- 172.21.15.98:0/641084227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91e00b5410 msgr2=0x7f91e015abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:24.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:24.203+0000 7f91e7bff700 1 --2- 172.21.15.98:0/641084227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91e00b5410 0x7f91e015abb0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0011e0 tx=0x7f91dc00ca30 comp rx=0 tx=0).stop 2023-03-03T08:03:24.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:24.203+0000 7f91e7bff700 1 -- 172.21.15.98:0/641084227 shutdown_connections 2023-03-03T08:03:24.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:24.203+0000 7f91e7bff700 1 --2- 172.21.15.98:0/641084227 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f91c4068380 0x7f91c406a830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:24.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:24.203+0000 7f91e7bff700 1 --2- 172.21.15.98:0/641084227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91e00b5410 0x7f91e015abb0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:24.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:24.203+0000 7f91e7bff700 1 --2- 172.21.15.98:0/641084227 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f91e00b8b90 0x7f91e00aa840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:24.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:24.203+0000 7f91e7bff700 1 --2- 172.21.15.98:0/641084227 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f91e00b87c0 0x7f91e00cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:24.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:24.203+0000 7f91e7bff700 1 -- 172.21.15.98:0/641084227 >> 172.21.15.98:0/641084227 conn(0x7f91e006aee0 msgr2=0x7f91e00b7820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:24.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:24.204+0000 7f91e7bff700 1 -- 172.21.15.98:0/641084227 shutdown_connections 2023-03-03T08:03:24.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:24.204+0000 7f91e7bff700 1 -- 172.21.15.98:0/641084227 wait complete. 2023-03-03T08:03:24.234 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T08:03:24.235 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T08:03:24.235 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2023-03-03T08:03:24.235 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-03-03T08:03:24.236 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-03-03T08:03:24.236 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T08:03:24.236 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-03-03T08:03:24.236 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T08:03:24.237 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-03-03T08:03:24.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:24.238 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2023-03-03T08:03:24.312 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T08:03:24.313 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T08:03:24.313 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T08:03:24.313 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:03:24.313 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2023-03-03T08:03:24.313 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:03:24.314 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:03:24.314 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:03:24.314 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2023-03-03T08:03:24.314 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:03:24.314 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:03:24.314 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2023-03-03T08:03:24.315 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:03:24.315 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:03:24.315 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T08:03:24.315 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:03:24.315 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T08:03:24.315 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 41756sec preferred_lft 41756sec 2023-03-03T08:03:24.315 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2023-03-03T08:03:24.315 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:03:24.316 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-03-03T08:03:24.316 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ee:78:63:4a:ea:32 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:03:24.316 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-03-03T08:03:24.316 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:03:24.316 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::9406:dcff:fe40:a4a4/64 scope link 2023-03-03T08:03:24.316 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:03:24.317 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:03:24.317 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:5e:bd:c1:84:4d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-03-03T08:03:24.317 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f45e:bdff:fec1:844d/64 scope link 2023-03-03T08:03:24.317 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:03:24.317 INFO:teuthology.orchestra.run.smithi100.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-03-03T08:03:24.317 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ee:78:63:4a:ea:32 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-03-03T08:03:24.318 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec78:63ff:fe4a:ea32/64 scope link 2023-03-03T08:03:24.318 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:03:24.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:24.320 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T08:03:24.418 INFO:teuthology.orchestra.run.smithi100.stdout:1 2023-03-03T08:03:24.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:24.424 DEBUG:teuthology.orchestra.run.smithi100:> route 2023-03-03T08:03:24.496 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2023-03-03T08:03:24.497 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T08:03:24.497 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T08:03:24.497 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T08:03:24.498 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T08:03:24.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:24.499 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2023-03-03T08:03:24.500 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2023-03-03T08:03:24.501 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T08:03:24.501 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T08:03:24.502 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T08:03:24.503 DEBUG:teuthology.orchestra.run.smithi100:> ') 2023-03-03T08:03:24.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:24.747 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2023-03-03T08:03:24.815 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-03-03T08:03:24.815 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-03-03T08:03:24.816 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-03-03T08:03:24.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:24.817 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2023-03-03T08:03:24.918 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-03-03T08:03:24.919 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:03:24.919 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-03-03T08:03:24.987 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-03-03T08:03:24.988 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:03:24.989 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-03-03T08:03:25.056 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T08:03:25.158 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:03:25.159 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:03:25.257 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:03:25.261 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:03:25.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:25.262 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T08:03:25.331 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-03-03T08:03:25.332 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_mountpoint=/) 2023-03-03T08:03:25.337 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2023-03-03T08:03:25.443 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-03T08:03:25.444 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-03T08:03:25.543 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-03T08:03:25.548 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-03T08:03:25.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:25.549 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2023-03-03T08:03:25.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2023-03-03T08:03:25.559+0000 7fb582c695c0 -1 init, newargv = 0x563d2bb40dc0 newargc=15 2023-03-03T08:03:25.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[121779]: starting ceph client 2023-03-03T08:03:25.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[121779]: starting fuse 2023-03-03T08:03:25.625 INFO:teuthology.orchestra.run.smithi100.stdout:42 2023-03-03T08:03:25.626 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2023-03-03T08:03:25.627 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-03T08:03:25.627 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2023-03-03T08:03:25.627 DEBUG:teuthology.orchestra.run.smithi100:> import re 2023-03-03T08:03:25.628 DEBUG:teuthology.orchestra.run.smithi100:> import os 2023-03-03T08:03:25.628 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2023-03-03T08:03:25.628 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:03:25.629 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2023-03-03T08:03:25.629 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-03-03T08:03:25.629 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2023-03-03T08:03:25.630 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-03-03T08:03:25.630 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:03:25.630 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2023-03-03T08:03:25.630 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2023-03-03T08:03:25.631 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2023-03-03T08:03:25.631 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2023-03-03T08:03:25.631 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:03:25.632 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2023-03-03T08:03:25.632 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-03T08:03:25.632 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-03T08:03:25.633 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-03T08:03:25.633 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2023-03-03T08:03:25.634 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2023-03-03T08:03:25.634 DEBUG:teuthology.orchestra.run.smithi100:> return f 2023-03-03T08:03:25.634 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-03T08:03:25.635 DEBUG:teuthology.orchestra.run.smithi100:> 2023-03-03T08:03:25.635 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.1")) 2023-03-03T08:03:25.635 DEBUG:teuthology.orchestra.run.smithi100:> ' 2023-03-03T08:03:25.851 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.1.121779.asok 2023-03-03T08:03:26.029 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.121779.asok 2023-03-03T08:03:26.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:26.030 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.121779.asok status 2023-03-03T08:03:26.300 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2023-03-03T08:03:26.301 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2023-03-03T08:03:26.301 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "472aaef4ec3b3a2c29f73ca67854785a910f1f9b", 2023-03-03T08:03:26.301 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev)", 2023-03-03T08:03:26.301 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "1", 2023-03-03T08:03:26.301 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2023-03-03T08:03:26.301 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-03-03T08:03:26.302 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "121779", 2023-03-03T08:03:26.302 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2023-03-03T08:03:26.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:03:26.302 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2023-03-03T08:03:26.302 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2023-03-03T08:03:26.302 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 6881, 2023-03-03T08:03:26.302 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2023-03-03T08:03:26.303 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2023-03-03T08:03:26.303 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2023-03-03T08:03:26.303 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 6881 2023-03-03T08:03:26.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:03:26.303 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T08:03:26.303 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T08:03:26.304 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:03:26.304 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1503700276 2023-03-03T08:03:26.304 INFO:teuthology.orchestra.run.smithi100.stdout: } 2023-03-03T08:03:26.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:03:26.304 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2023-03-03T08:03:26.304 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2023-03-03T08:03:26.304 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2023-03-03T08:03:26.304 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1503700276 2023-03-03T08:03:26.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2023-03-03T08:03:26.305 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.6881 192.168.0.1:0/1503700276", 2023-03-03T08:03:26.305 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.1:0/1503700276", 2023-03-03T08:03:26.305 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2023-03-03T08:03:26.305 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 62, 2023-03-03T08:03:26.305 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 111, 2023-03-03T08:03:26.305 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2023-03-03T08:03:26.306 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2023-03-03T08:03:26.306 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2023-03-03T08:03:26.306 INFO:teuthology.orchestra.run.smithi100.stdout:} 2023-03-03T08:03:26.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:26.322 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2023-03-03T08:03:26.323 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2023-03-03T08:03:26.392 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2023-03-03T08:03:26.392 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-03-03T08:03:26.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:26.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-03-03T08:03:26.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-03-03T08:03:26.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.751+0000 7f825b3ff700 1 Processor -- start 2023-03-03T08:03:26.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.751+0000 7f825b3ff700 1 -- start start 2023-03-03T08:03:26.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.752+0000 7f825b3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82540cd770 0x7f82540b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:26.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.753+0000 7f825b3ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82540d0ac0 con 0x7f825415bf70 2023-03-03T08:03:26.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.753+0000 7f825b3ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82540cdbe0 con 0x7f82540b8880 2023-03-03T08:03:26.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.753+0000 7f825b3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82540cff70 con 0x7f82540cd770 2023-03-03T08:03:26.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.753+0000 7f825999c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f825415bf70 0x7f825415c3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45084/0 (socket says 172.21.15.98:45084) 2023-03-03T08:03:26.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.753+0000 7f825999c700 1 -- 172.21.15.98:0/1111992679 learned_addr learned my addr 172.21.15.98:0/1111992679 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:26.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.753+0000 7f825899a700 1 --2- 172.21.15.98:0/1111992679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82540cd770 0x7f82540b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:26.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.753+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098554461 0 0) 0x7f82540d0ac0 con 0x7f825415bf70 2023-03-03T08:03:26.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82400035b0 con 0x7f825415bf70 2023-03-03T08:03:26.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835678735 0 0) 0x7f82540cdbe0 con 0x7f82540b8880 2023-03-03T08:03:26.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82540d0ac0 con 0x7f82540b8880 2023-03-03T08:03:26.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 606492214 0 0) 0x7f82400035b0 con 0x7f825415bf70 2023-03-03T08:03:26.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82540cd770 msgr2=0x7f82540b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:26.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 --2- 172.21.15.98:0/1111992679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82540cd770 0x7f82540b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:26.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 >> v1:172.21.15.100:6789/0 conn(0x7f82540b8880 legacy=0x7f82540b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:26.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82540d0250 con 0x7f825415bf70 2023-03-03T08:03:26.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7f824c0019f0 con 0x7f825415bf70 2023-03-03T08:03:26.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 >> v1:172.21.15.98:6789/0 conn(0x7f825415bf70 legacy=0x7f825415c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:26.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 --2- 172.21.15.98:0/1111992679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8240003bc0 0x7f8240003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:26.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 --2- 172.21.15.98:0/1111992679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8240004850 0x7f8240004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:26.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 --2- 172.21.15.98:0/1111992679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82540cd770 0x7f8240005260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:26.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82540cff70 con 0x7f8240003bc0 2023-03-03T08:03:26.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824000bba0 con 0x7f8240004850 2023-03-03T08:03:26.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824000bd60 con 0x7f82540cd770 2023-03-03T08:03:26.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.754+0000 7f825919b700 1 --2- 172.21.15.98:0/1111992679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8240003bc0 0x7f8240003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:26.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.755+0000 7f825899a700 1 --2- 172.21.15.98:0/1111992679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8240004850 0x7f8240004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:26.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.755+0000 7f825999c700 1 --2- 172.21.15.98:0/1111992679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82540cd770 0x7f8240005260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:26.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.755+0000 7f825899a700 1 -- 172.21.15.98:0/1111992679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82540cd770 msgr2=0x7f8240005260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:26.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.755+0000 7f825899a700 1 --2- 172.21.15.98:0/1111992679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82540cd770 0x7f8240005260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:26.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.755+0000 7f825899a700 1 -- 172.21.15.98:0/1111992679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8240003bc0 msgr2=0x7f8240003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:26.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.755+0000 7f825899a700 1 --2- 172.21.15.98:0/1111992679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8240003bc0 0x7f8240003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:26.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.755+0000 7f825899a700 1 -- 172.21.15.98:0/1111992679 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82400035b0 con 0x7f8240004850 2023-03-03T08:03:26.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.755+0000 7f825919b700 1 --2- 172.21.15.98:0/1111992679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8240003bc0 0x7f8240003fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:03:26.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.755+0000 7f825899a700 1 --2- 172.21.15.98:0/1111992679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8240004850 0x7f8240004cc0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8244009b60 tx=0x7f8240005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8426cc420dbf242d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:26.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.756+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824400e6c0 con 0x7f8240004850 2023-03-03T08:03:26.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.756+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824400e820 con 0x7f8240004850 2023-03-03T08:03:26.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.756+0000 7f824bfff700 1 -- 172.21.15.98:0/1111992679 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824400eb90 con 0x7f8240004850 2023-03-03T08:03:26.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.756+0000 7f825b3ff700 1 -- 172.21.15.98:0/1111992679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8240004850 msgr2=0x7f8240004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:26.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.756+0000 7f825b3ff700 1 --2- 172.21.15.98:0/1111992679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8240004850 0x7f8240004cc0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8244009b60 tx=0x7f8240005880 comp rx=0 tx=0).stop 2023-03-03T08:03:26.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.756+0000 7f825999c700 1 -- 172.21.15.98:0/1111992679 reap_dead start 2023-03-03T08:03:26.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.756+0000 7f825b3ff700 1 -- 172.21.15.98:0/1111992679 shutdown_connections 2023-03-03T08:03:26.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.756+0000 7f825b3ff700 1 -- 172.21.15.98:0/1111992679 >> 172.21.15.98:0/1111992679 conn(0x7f825406ad80 msgr2=0x7f825406b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:26.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.756+0000 7f825b3ff700 1 -- 172.21.15.98:0/1111992679 shutdown_connections 2023-03-03T08:03:26.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.756+0000 7f825b3ff700 1 -- 172.21.15.98:0/1111992679 wait complete. 2023-03-03T08:03:26.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.758+0000 7f825b3ff700 1 Processor -- start 2023-03-03T08:03:26.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.759+0000 7f825b3ff700 1 -- start start 2023-03-03T08:03:26.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.760+0000 7f825b3ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8240004850 0x7f8254150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:26.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.760+0000 7f825b3ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f825415db60 0x7f825414fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:26.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.760+0000 7f825b3ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82540ccb80 0x7f82540ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:26.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.760+0000 7f825b3ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82541504d0 con 0x7f82540ccb80 2023-03-03T08:03:26.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.760+0000 7f825b3ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82540cbdc0 con 0x7f825415db60 2023-03-03T08:03:26.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.760+0000 7f825b3ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82540cc0a0 con 0x7f8240004850 2023-03-03T08:03:26.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.760+0000 7f825999c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82540ccb80 0x7f82540ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.760+0000 7f825919b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8240004850 0x7f8254150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.760+0000 7f825899a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f825415db60 0x7f825414fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.760+0000 7f825919b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8240004850 0x7f8254150660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:45642/0 (socket says 172.21.15.98:45642) 2023-03-03T08:03:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.760+0000 7f825919b700 1 -- 172.21.15.98:0/2587880811 learned_addr learned my addr 172.21.15.98:0/2587880811 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.761+0000 7f825899a700 1 -- 172.21.15.98:0/2587880811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8240004850 msgr2=0x7f8254150660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.761+0000 7f825899a700 1 --2- 172.21.15.98:0/2587880811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8240004850 0x7f8254150660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.761+0000 7f825899a700 1 -- 172.21.15.98:0/2587880811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82540ccb80 msgr2=0x7f82540ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.761+0000 7f825899a700 1 --2- 172.21.15.98:0/2587880811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82540ccb80 0x7f82540ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.761+0000 7f825899a700 1 -- 172.21.15.98:0/2587880811 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82540d0cf0 con 0x7f825415db60 2023-03-03T08:03:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.761+0000 7f825999c700 1 --2- 172.21.15.98:0/2587880811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82540ccb80 0x7f82540ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:03:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.761+0000 7f825899a700 1 --2- 172.21.15.98:0/2587880811 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f825415db60 0x7f825414fad0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8244019000 tx=0x7f82440047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.761+0000 7f8249ffb700 1 -- 172.21.15.98:0/2587880811 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824400e6c0 con 0x7f825415db60 2023-03-03T08:03:26.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.761+0000 7f8249ffb700 1 -- 172.21.15.98:0/2587880811 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82440038e0 con 0x7f825415db60 2023-03-03T08:03:26.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.761+0000 7f8249ffb700 1 -- 172.21.15.98:0/2587880811 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244003c90 con 0x7f825415db60 2023-03-03T08:03:26.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.761+0000 7f825b3ff700 1 -- 172.21.15.98:0/2587880811 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82540d0530 con 0x7f825415db60 2023-03-03T08:03:26.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.762+0000 7f825b3ff700 1 -- 172.21.15.98:0/2587880811 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82540b8c50 con 0x7f825415db60 2023-03-03T08:03:26.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.763+0000 7f825b3ff700 1 -- 172.21.15.98:0/2587880811 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82540cbdc0 con 0x7f825415db60 2023-03-03T08:03:26.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.764+0000 7f8249ffb700 1 -- 172.21.15.98:0/2587880811 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f8244025070 con 0x7f825415db60 2023-03-03T08:03:26.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.765+0000 7f8249ffb700 1 --2- 172.21.15.98:0/2587880811 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8238068380 0x7f823806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:26.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.765+0000 7f8249ffb700 1 -- 172.21.15.98:0/2587880811 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8244087410 con 0x7f825415db60 2023-03-03T08:03:26.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.765+0000 7f825919b700 1 --2- 172.21.15.98:0/2587880811 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8238068380 0x7f823806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:26.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.766+0000 7f825919b700 1 --2- 172.21.15.98:0/2587880811 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8238068380 0x7f823806a830 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f82500015a0 tx=0x7f8250009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:26.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:26.770+0000 7f8249ffb700 1 -- 172.21.15.98:0/2587880811 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8244053b90 con 0x7f825415db60 2023-03-03T08:03:27.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.003+0000 7f825b3ff700 1 -- 172.21.15.98:0/2587880811 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f82540cd530 con 0x7f825415db60 2023-03-03T08:03:27.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.575+0000 7f8249ffb700 1 -- 172.21.15.98:0/2587880811 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v63) v1 ==== 136+0+0 (secure 0 0 0) 0x7f82440858f0 con 0x7f825415db60 2023-03-03T08:03:27.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.577+0000 7f825b3ff700 1 -- 172.21.15.98:0/2587880811 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8238068380 msgr2=0x7f823806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:27.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.577+0000 7f825b3ff700 1 --2- 172.21.15.98:0/2587880811 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8238068380 0x7f823806a830 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f82500015a0 tx=0x7f8250009040 comp rx=0 tx=0).stop 2023-03-03T08:03:27.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.577+0000 7f825b3ff700 1 -- 172.21.15.98:0/2587880811 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f825415db60 msgr2=0x7f825414fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:27.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.577+0000 7f825b3ff700 1 --2- 172.21.15.98:0/2587880811 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f825415db60 0x7f825414fad0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8244019000 tx=0x7f82440047c0 comp rx=0 tx=0).stop 2023-03-03T08:03:27.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.578+0000 7f825b3ff700 1 -- 172.21.15.98:0/2587880811 shutdown_connections 2023-03-03T08:03:27.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.578+0000 7f825b3ff700 1 --2- 172.21.15.98:0/2587880811 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8238068380 0x7f823806a830 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:27.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.578+0000 7f825b3ff700 1 --2- 172.21.15.98:0/2587880811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82540ccb80 0x7f82540ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:27.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.578+0000 7f825b3ff700 1 --2- 172.21.15.98:0/2587880811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8240004850 0x7f8254150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:27.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.578+0000 7f825b3ff700 1 --2- 172.21.15.98:0/2587880811 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f825415db60 0x7f825414fad0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:27.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.578+0000 7f825b3ff700 1 -- 172.21.15.98:0/2587880811 >> 172.21.15.98:0/2587880811 conn(0x7f825406ad80 msgr2=0x7f82540b5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:27.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.578+0000 7f825b3ff700 1 -- 172.21.15.98:0/2587880811 shutdown_connections 2023-03-03T08:03:27.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.578+0000 7f825b3ff700 1 -- 172.21.15.98:0/2587880811 wait complete. 2023-03-03T08:03:27.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-03-03T08:03:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.854+0000 7f44433ff700 1 Processor -- start 2023-03-03T08:03:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.854+0000 7f44433ff700 1 -- start start 2023-03-03T08:03:27.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.856+0000 7f44433ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f443c06b4b0 0x7f443c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:27.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.856+0000 7f44433ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f443c0cf980 con 0x7f443c0d0270 2023-03-03T08:03:27.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.856+0000 7f44433ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f443c0d1800 con 0x7f443c15ca70 2023-03-03T08:03:27.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.856+0000 7f44433ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f443c0cd5b0 con 0x7f443c06b4b0 2023-03-03T08:03:27.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.856+0000 7f444099a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f443c0d0270 0x7f443c0d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45090/0 (socket says 172.21.15.98:45090) 2023-03-03T08:03:27.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.856+0000 7f444099a700 1 -- 172.21.15.98:0/2209445159 learned_addr learned my addr 172.21.15.98:0/2209445159 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:27.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.856+0000 7f444199c700 1 --2- 172.21.15.98:0/2209445159 >> v2:172.21.15.100:3300/0 conn(0x7f443c06b4b0 0x7f443c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:27.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.856+0000 7f4433fff700 1 -- 172.21.15.98:0/2209445159 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813136923 0 0) 0x7f443c0cf980 con 0x7f443c0d0270 2023-03-03T08:03:27.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.857+0000 7f4433fff700 1 -- 172.21.15.98:0/2209445159 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44280035b0 con 0x7f443c0d0270 2023-03-03T08:03:27.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.857+0000 7f444199c700 1 -- 172.21.15.98:0/2209445159 >> v1:172.21.15.100:6789/0 conn(0x7f443c15ca70 legacy=0x7f443c0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:27.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.857+0000 7f444199c700 1 -- 172.21.15.98:0/2209445159 >> v1:172.21.15.98:6789/0 conn(0x7f443c0d0270 legacy=0x7f443c0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:27.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.857+0000 7f444199c700 1 -- 172.21.15.98:0/2209445159 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c0cd900 con 0x7f443c06b4b0 2023-03-03T08:03:27.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.857+0000 7f444199c700 1 --2- 172.21.15.98:0/2209445159 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f443c06b4b0 0x7f443c0cf440 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4434009d90 tx=0x7f443400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26a8ff8f27a0b74a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:27.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.858+0000 7f4433fff700 1 -- 172.21.15.98:0/2209445159 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443400e070 con 0x7f443c06b4b0 2023-03-03T08:03:27.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.858+0000 7f4433fff700 1 -- 172.21.15.98:0/2209445159 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443400bb20 con 0x7f443c06b4b0 2023-03-03T08:03:27.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.858+0000 7f4433fff700 1 -- 172.21.15.98:0/2209445159 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443400be90 con 0x7f443c06b4b0 2023-03-03T08:03:27.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.858+0000 7f44433ff700 1 -- 172.21.15.98:0/2209445159 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f443c06b4b0 msgr2=0x7f443c0cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:27.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.858+0000 7f44433ff700 1 --2- 172.21.15.98:0/2209445159 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f443c06b4b0 0x7f443c0cf440 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4434009d90 tx=0x7f443400b330 comp rx=0 tx=0).stop 2023-03-03T08:03:27.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.858+0000 7f44433ff700 1 -- 172.21.15.98:0/2209445159 shutdown_connections 2023-03-03T08:03:27.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.858+0000 7f44433ff700 1 --2- 172.21.15.98:0/2209445159 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f443c06b4b0 0x7f443c0cf440 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:27.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.858+0000 7f44433ff700 1 -- 172.21.15.98:0/2209445159 >> 172.21.15.98:0/2209445159 conn(0x7f443c073e70 msgr2=0x7f443c06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:27.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.858+0000 7f44433ff700 1 -- 172.21.15.98:0/2209445159 shutdown_connections 2023-03-03T08:03:27.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.859+0000 7f44433ff700 1 -- 172.21.15.98:0/2209445159 wait complete. 2023-03-03T08:03:27.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.860+0000 7f44433ff700 1 Processor -- start 2023-03-03T08:03:27.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.861+0000 7f44433ff700 1 -- start start 2023-03-03T08:03:27.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.862+0000 7f44433ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f443c06b4b0 0x7f443c153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:27.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.862+0000 7f44433ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f443c0d0270 0x7f443c0b7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:27.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.862+0000 7f44433ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f443c15ca70 0x7f443c0b7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:27.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.862+0000 7f44433ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c15fed0 con 0x7f443c0d0270 2023-03-03T08:03:27.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.862+0000 7f44433ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c154280 con 0x7f443c15ca70 2023-03-03T08:03:27.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.862+0000 7f44433ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443c15f250 con 0x7f443c06b4b0 2023-03-03T08:03:27.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.862+0000 7f444099a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f443c0d0270 0x7f443c0b7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:27.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.862+0000 7f444099a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f443c0d0270 0x7f443c0b7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47190/0 (socket says 172.21.15.98:47190) 2023-03-03T08:03:27.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.862+0000 7f444099a700 1 -- 172.21.15.98:0/2086571253 learned_addr learned my addr 172.21.15.98:0/2086571253 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:27.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.862+0000 7f444119b700 1 --2- 172.21.15.98:0/2086571253 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f443c06b4b0 0x7f443c153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:27.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.862+0000 7f444199c700 1 --2- 172.21.15.98:0/2086571253 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f443c15ca70 0x7f443c0b7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:27.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.863+0000 7f444099a700 1 -- 172.21.15.98:0/2086571253 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f443c06b4b0 msgr2=0x7f443c153d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:27.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.863+0000 7f444099a700 1 --2- 172.21.15.98:0/2086571253 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f443c06b4b0 0x7f443c153d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:27.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.863+0000 7f444099a700 1 -- 172.21.15.98:0/2086571253 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f443c15ca70 msgr2=0x7f443c0b7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:27.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.863+0000 7f444099a700 1 --2- 172.21.15.98:0/2086571253 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f443c15ca70 0x7f443c0b7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:27.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.863+0000 7f444099a700 1 -- 172.21.15.98:0/2086571253 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4434009a70 con 0x7f443c0d0270 2023-03-03T08:03:27.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.863+0000 7f444099a700 1 --2- 172.21.15.98:0/2086571253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f443c0d0270 0x7f443c0b7800 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f442c00c4f0 tx=0x7f442c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:27.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.863+0000 7f4431ffb700 1 -- 172.21.15.98:0/2086571253 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c005d70 con 0x7f443c0d0270 2023-03-03T08:03:27.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.863+0000 7f4431ffb700 1 -- 172.21.15.98:0/2086571253 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442c0070d0 con 0x7f443c0d0270 2023-03-03T08:03:27.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.863+0000 7f4431ffb700 1 -- 172.21.15.98:0/2086571253 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c007440 con 0x7f443c0d0270 2023-03-03T08:03:27.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.863+0000 7f44433ff700 1 -- 172.21.15.98:0/2086571253 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443c0d1aa0 con 0x7f443c0d0270 2023-03-03T08:03:27.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.864+0000 7f44433ff700 1 -- 172.21.15.98:0/2086571253 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443c0cdbe0 con 0x7f443c0d0270 2023-03-03T08:03:27.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.865+0000 7f44433ff700 1 -- 172.21.15.98:0/2086571253 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f443c15ce40 con 0x7f443c0d0270 2023-03-03T08:03:27.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.866+0000 7f4431ffb700 1 -- 172.21.15.98:0/2086571253 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f442c019030 con 0x7f443c0d0270 2023-03-03T08:03:27.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.867+0000 7f4431ffb700 1 --2- 172.21.15.98:0/2086571253 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4420068380 0x7f442006a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:27.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.867+0000 7f4431ffb700 1 -- 172.21.15.98:0/2086571253 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f442c084f70 con 0x7f443c0d0270 2023-03-03T08:03:27.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.868+0000 7f444119b700 1 --2- 172.21.15.98:0/2086571253 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4420068380 0x7f442006a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:27.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.868+0000 7f444119b700 1 --2- 172.21.15.98:0/2086571253 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4420068380 0x7f442006a830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4438007fd0 tx=0x7f4438009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:27.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:27.872+0000 7f4431ffb700 1 -- 172.21.15.98:0/2086571253 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f442c04b5e0 con 0x7f443c0d0270 2023-03-03T08:03:28.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.088+0000 7f44433ff700 1 -- 172.21.15.98:0/2086571253 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f443c04e6e0 con 0x7f4420068380 2023-03-03T08:03:28.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.090+0000 7f4431ffb700 1 -- 172.21.15.98:0/2086571253 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f443c04e6e0 con 0x7f4420068380 2023-03-03T08:03:28.092 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T08:03:28.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:28.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephfs" 2023-03-03T08:03:28.093 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:28.093 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T08:03:28.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.092+0000 7f44433ff700 1 -- 172.21.15.98:0/2086571253 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4420068380 msgr2=0x7f442006a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:28.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.092+0000 7f44433ff700 1 --2- 172.21.15.98:0/2086571253 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4420068380 0x7f442006a830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4438007fd0 tx=0x7f4438009040 comp rx=0 tx=0).stop 2023-03-03T08:03:28.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.092+0000 7f44433ff700 1 -- 172.21.15.98:0/2086571253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f443c0d0270 msgr2=0x7f443c0b7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:28.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.092+0000 7f44433ff700 1 --2- 172.21.15.98:0/2086571253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f443c0d0270 0x7f443c0b7800 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f442c00c4f0 tx=0x7f442c00c800 comp rx=0 tx=0).stop 2023-03-03T08:03:28.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.092+0000 7f44433ff700 1 -- 172.21.15.98:0/2086571253 shutdown_connections 2023-03-03T08:03:28.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.092+0000 7f44433ff700 1 --2- 172.21.15.98:0/2086571253 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f443c15ca70 0x7f443c0b7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:28.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.092+0000 7f44433ff700 1 --2- 172.21.15.98:0/2086571253 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f443c06b4b0 0x7f443c153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:28.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.092+0000 7f44433ff700 1 --2- 172.21.15.98:0/2086571253 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f4420068380 0x7f442006a830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:28.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.092+0000 7f44433ff700 1 --2- 172.21.15.98:0/2086571253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f443c0d0270 0x7f443c0b7800 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:28.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.092+0000 7f44433ff700 1 -- 172.21.15.98:0/2086571253 >> 172.21.15.98:0/2086571253 conn(0x7f443c073e70 msgr2=0x7f443c0b5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:28.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.093+0000 7f44433ff700 1 -- 172.21.15.98:0/2086571253 shutdown_connections 2023-03-03T08:03:28.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.093+0000 7f44433ff700 1 -- 172.21.15.98:0/2086571253 wait complete. 2023-03-03T08:03:28.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-03-03T08:03:28.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.369+0000 7f47bf59e700 1 Processor -- start 2023-03-03T08:03:28.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.369+0000 7f47bf59e700 1 -- start start 2023-03-03T08:03:28.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.370+0000 7f47bf59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f47c00cf570 0x7f47c015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.370+0000 7f47bf59e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c00cd780 con 0x7f47c015db30 2023-03-03T08:03:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.371+0000 7f47bf59e700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c00cda60 con 0x7f47c01609b0 2023-03-03T08:03:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.371+0000 7f47bf59e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c00cebb0 con 0x7f47c00cf570 2023-03-03T08:03:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.371+0000 7f47bed9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f47c015db30 0x7f47c00b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45104/0 (socket says 172.21.15.98:45104) 2023-03-03T08:03:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.371+0000 7f47bed9d700 1 -- 172.21.15.98:0/678729274 learned_addr learned my addr 172.21.15.98:0/678729274 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.371+0000 7f47bdd9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f47c00cf570 0x7f47c015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.371+0000 7f47bdd9b700 1 -- 172.21.15.98:0/678729274 >> v1:172.21.15.100:6789/0 conn(0x7f47c01609b0 legacy=0x7f47c0160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.372+0000 7f47bdd9b700 1 -- 172.21.15.98:0/678729274 >> v1:172.21.15.98:6789/0 conn(0x7f47c015db30 legacy=0x7f47c00b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.372+0000 7f47bdd9b700 1 -- 172.21.15.98:0/678729274 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c00cdd40 con 0x7f47c00cf570 2023-03-03T08:03:28.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.372+0000 7f47bd59a700 1 -- 172.21.15.98:0/678729274 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889932298 0 0) 0x7f47c00cd780 con 0x7f47c015db30 2023-03-03T08:03:28.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.372+0000 7f47bdd9b700 1 --2- 172.21.15.98:0/678729274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c00cf570 0x7f47c015d5f0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f47b40097c0 tx=0x7f47b4009ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc890e4021ebeeba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:28.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.372+0000 7f47bd59a700 1 -- 172.21.15.98:0/678729274 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4004230 con 0x7f47c00cf570 2023-03-03T08:03:28.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.372+0000 7f47bd59a700 1 -- 172.21.15.98:0/678729274 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b4004390 con 0x7f47c00cf570 2023-03-03T08:03:28.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.372+0000 7f47bd59a700 1 -- 172.21.15.98:0/678729274 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b40036a0 con 0x7f47c00cf570 2023-03-03T08:03:28.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.373+0000 7f47bf59e700 1 -- 172.21.15.98:0/678729274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c00cf570 msgr2=0x7f47c015d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:28.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.373+0000 7f47bf59e700 1 --2- 172.21.15.98:0/678729274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c00cf570 0x7f47c015d5f0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f47b40097c0 tx=0x7f47b4009ad0 comp rx=0 tx=0).stop 2023-03-03T08:03:28.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.373+0000 7f47bf59e700 1 -- 172.21.15.98:0/678729274 shutdown_connections 2023-03-03T08:03:28.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.373+0000 7f47bf59e700 1 --2- 172.21.15.98:0/678729274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c00cf570 0x7f47c015d5f0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:28.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.373+0000 7f47bf59e700 1 -- 172.21.15.98:0/678729274 >> 172.21.15.98:0/678729274 conn(0x7f47c006b880 msgr2=0x7f47c00cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:28.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.373+0000 7f47bf59e700 1 -- 172.21.15.98:0/678729274 shutdown_connections 2023-03-03T08:03:28.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.373+0000 7f47bf59e700 1 -- 172.21.15.98:0/678729274 wait complete. 2023-03-03T08:03:28.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.375+0000 7f47bf59e700 1 Processor -- start 2023-03-03T08:03:28.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.375+0000 7f47bf59e700 1 -- start start 2023-03-03T08:03:28.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.376+0000 7f47bf59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c00cf570 0x7f47c00d1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:28.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.376+0000 7f47bf59e700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47c015db30 0x7f47c014f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:28.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.376+0000 7f47bf59e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47c01609b0 0x7f47c0150e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:28.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.376+0000 7f47bf59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c00b71b0 con 0x7f47c00cf570 2023-03-03T08:03:28.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.376+0000 7f47bf59e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c00b7490 con 0x7f47c01609b0 2023-03-03T08:03:28.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.376+0000 7f47bf59e700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c00b7770 con 0x7f47c015db30 2023-03-03T08:03:28.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.376+0000 7f47be59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c00cf570 0x7f47c00d1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:28.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.376+0000 7f47be59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c00cf570 0x7f47c00d1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47210/0 (socket says 172.21.15.98:47210) 2023-03-03T08:03:28.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.377+0000 7f47be59c700 1 -- 172.21.15.98:0/542798720 learned_addr learned my addr 172.21.15.98:0/542798720 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:28.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.377+0000 7f47bed9d700 1 --2- 172.21.15.98:0/542798720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47c01609b0 0x7f47c0150e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:28.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.377+0000 7f47bdd9b700 1 --2- 172.21.15.98:0/542798720 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47c015db30 0x7f47c014f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:28.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.377+0000 7f47be59c700 1 -- 172.21.15.98:0/542798720 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47c015db30 msgr2=0x7f47c014f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:28.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.377+0000 7f47be59c700 1 --2- 172.21.15.98:0/542798720 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47c015db30 0x7f47c014f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:28.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.377+0000 7f47be59c700 1 -- 172.21.15.98:0/542798720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47c01609b0 msgr2=0x7f47c0150e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:28.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.377+0000 7f47be59c700 1 --2- 172.21.15.98:0/542798720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47c01609b0 0x7f47c0150e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:28.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.377+0000 7f47be59c700 1 -- 172.21.15.98:0/542798720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b40094a0 con 0x7f47c00cf570 2023-03-03T08:03:28.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.377+0000 7f47bdd9b700 1 --2- 172.21.15.98:0/542798720 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47c015db30 0x7f47c014f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:03:28.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.377+0000 7f47be59c700 1 --2- 172.21.15.98:0/542798720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c00cf570 0x7f47c00d1060 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f47b000c480 tx=0x7f47b000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:28.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.377+0000 7f47aeffd700 1 -- 172.21.15.98:0/542798720 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b00054e0 con 0x7f47c00cf570 2023-03-03T08:03:28.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.378+0000 7f47aeffd700 1 -- 172.21.15.98:0/542798720 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b0003680 con 0x7f47c00cf570 2023-03-03T08:03:28.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.378+0000 7f47aeffd700 1 -- 172.21.15.98:0/542798720 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b0003a30 con 0x7f47c00cf570 2023-03-03T08:03:28.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.378+0000 7f47bf59e700 1 -- 172.21.15.98:0/542798720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c015d0d0 con 0x7f47c00cf570 2023-03-03T08:03:28.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.378+0000 7f47bf59e700 1 -- 172.21.15.98:0/542798720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c015cdf0 con 0x7f47c00cf570 2023-03-03T08:03:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.379+0000 7f47bf59e700 1 -- 172.21.15.98:0/542798720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c0068290 con 0x7f47c00cf570 2023-03-03T08:03:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.380+0000 7f47aeffd700 1 -- 172.21.15.98:0/542798720 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f47b0019030 con 0x7f47c00cf570 2023-03-03T08:03:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.381+0000 7f47aeffd700 1 --2- 172.21.15.98:0/542798720 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f47a00683b0 0x7f47a006a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.381+0000 7f47aeffd700 1 -- 172.21.15.98:0/542798720 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f47b00852a0 con 0x7f47c00cf570 2023-03-03T08:03:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.382+0000 7f47bdd9b700 1 --2- 172.21.15.98:0/542798720 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f47a00683b0 0x7f47a006a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:28.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.383+0000 7f47bdd9b700 1 --2- 172.21.15.98:0/542798720 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f47a00683b0 0x7f47a006a860 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f47c015a7d0 tx=0x7f47b4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:28.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.386+0000 7f47aeffd700 1 -- 172.21.15.98:0/542798720 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f47b0051990 con 0x7f47c00cf570 2023-03-03T08:03:28.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.580+0000 7f47bf59e700 1 -- 172.21.15.98:0/542798720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f47c00d0430 con 0x7f47c00cf570 2023-03-03T08:03:28.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.581+0000 7f47aeffd700 1 -- 172.21.15.98:0/542798720 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f47b00170c0 con 0x7f47c00cf570 2023-03-03T08:03:28.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.584+0000 7f47bf59e700 1 -- 172.21.15.98:0/542798720 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f47a00683b0 msgr2=0x7f47a006a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:28.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.584+0000 7f47bf59e700 1 --2- 172.21.15.98:0/542798720 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f47a00683b0 0x7f47a006a860 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f47c015a7d0 tx=0x7f47b4008030 comp rx=0 tx=0).stop 2023-03-03T08:03:28.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.584+0000 7f47bf59e700 1 -- 172.21.15.98:0/542798720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c00cf570 msgr2=0x7f47c00d1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:28.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.584+0000 7f47bf59e700 1 --2- 172.21.15.98:0/542798720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c00cf570 0x7f47c00d1060 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f47b000c480 tx=0x7f47b000c840 comp rx=0 tx=0).stop 2023-03-03T08:03:28.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.584+0000 7f47bf59e700 1 -- 172.21.15.98:0/542798720 shutdown_connections 2023-03-03T08:03:28.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.584+0000 7f47bf59e700 1 --2- 172.21.15.98:0/542798720 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f47a00683b0 0x7f47a006a860 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:28.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.584+0000 7f47bf59e700 1 --2- 172.21.15.98:0/542798720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c00cf570 0x7f47c00d1060 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:28.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.584+0000 7f47bf59e700 1 --2- 172.21.15.98:0/542798720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47c01609b0 0x7f47c0150e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:28.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.584+0000 7f47bf59e700 1 --2- 172.21.15.98:0/542798720 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47c015db30 0x7f47c014f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:28.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.584+0000 7f47bf59e700 1 -- 172.21.15.98:0/542798720 >> 172.21.15.98:0/542798720 conn(0x7f47c006b880 msgr2=0x7f47c00731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:28.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.584+0000 7f47bf59e700 1 -- 172.21.15.98:0/542798720 shutdown_connections 2023-03-03T08:03:28.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:28.584+0000 7f47bf59e700 1 -- 172.21.15.98:0/542798720 wait complete. 2023-03-03T08:03:28.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:28.614 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4) 2023-03-03T08:03:28.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:28.734 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/fake) 2023-03-03T08:03:28.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:28.857 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2023-03-03T08:03:28.931 INFO:teuthology.orchestra.run.smithi098.stdout:cephfs_data 2023-03-03T08:03:28.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:28.932 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4) 2023-03-03T08:03:29.042 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:03:29.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvolume_0000000000508309/.meta 2023-03-03T08:03:29.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000508309 2023-03-03T08:03:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.436+0000 7fc3803ff700 1 Processor -- start 2023-03-03T08:03:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.436+0000 7fc3803ff700 1 -- start start 2023-03-03T08:03:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.437+0000 7fc3803ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3780d0270 0x7fc3780cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.438+0000 7fc3803ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc37806b4b0 0x7fc3780d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.438+0000 7fc3803ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3780cf980 con 0x7fc37815ca70 2023-03-03T08:03:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.438+0000 7fc3803ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3780cd5b0 con 0x7fc37806b4b0 2023-03-03T08:03:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.438+0000 7fc3803ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780d1800 con 0x7fc3780d0270 2023-03-03T08:03:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.438+0000 7fc37e19b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc37815ca70 0x7fc3780cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45116/0 (socket says 172.21.15.98:45116) 2023-03-03T08:03:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.438+0000 7fc37e19b700 1 -- 172.21.15.98:0/2348504742 learned_addr learned my addr 172.21.15.98:0/2348504742 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.438+0000 7fc37e99c700 1 --2- 172.21.15.98:0/2348504742 >> v2:172.21.15.100:3300/0 conn(0x7fc37806b4b0 0x7fc3780d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.438+0000 7fc37d99a700 1 --2- 172.21.15.98:0/2348504742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3780d0270 0x7fc3780cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.438+0000 7fc37d199700 1 -- 172.21.15.98:0/2348504742 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626275652 0 0) 0x7fc3780cf980 con 0x7fc37815ca70 2023-03-03T08:03:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.438+0000 7fc37d199700 1 -- 172.21.15.98:0/2348504742 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3640035b0 con 0x7fc37815ca70 2023-03-03T08:03:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37d199700 1 -- 172.21.15.98:0/2348504742 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3834011255 0 0) 0x7fc3640035b0 con 0x7fc37815ca70 2023-03-03T08:03:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37d199700 1 -- 172.21.15.98:0/2348504742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3780d0270 msgr2=0x7fc3780cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37d199700 1 --2- 172.21.15.98:0/2348504742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3780d0270 0x7fc3780cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37d199700 1 -- 172.21.15.98:0/2348504742 >> v2:172.21.15.100:3300/0 conn(0x7fc37806b4b0 msgr2=0x7fc3780d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37d199700 1 --2- 172.21.15.98:0/2348504742 >> v2:172.21.15.100:3300/0 conn(0x7fc37806b4b0 0x7fc3780d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37d199700 1 -- 172.21.15.98:0/2348504742 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3780cd900 con 0x7fc37815ca70 2023-03-03T08:03:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37d199700 1 -- 172.21.15.98:0/2348504742 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7fc374001670 con 0x7fc37815ca70 2023-03-03T08:03:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37d199700 1 -- 172.21.15.98:0/2348504742 >> v1:172.21.15.98:6789/0 conn(0x7fc37815ca70 legacy=0x7fc3780cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37d199700 1 --2- 172.21.15.98:0/2348504742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3780d0270 0x7fc364003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37d199700 1 --2- 172.21.15.98:0/2348504742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc364003fd0 0x7fc364004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37d199700 1 --2- 172.21.15.98:0/2348504742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc364004d90 0x7fc364005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37d199700 1 -- 172.21.15.98:0/2348504742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780cd5b0 con 0x7fc364004d90 2023-03-03T08:03:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37d199700 1 -- 172.21.15.98:0/2348504742 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780d1800 con 0x7fc364003fd0 2023-03-03T08:03:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37d199700 1 -- 172.21.15.98:0/2348504742 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36400ba90 con 0x7fc3780d0270 2023-03-03T08:03:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37e99c700 1 --2- 172.21.15.98:0/2348504742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc364004d90 0x7fc364005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37e19b700 1 --2- 172.21.15.98:0/2348504742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3780d0270 0x7fc364003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.439+0000 7fc37d99a700 1 --2- 172.21.15.98:0/2348504742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc364003fd0 0x7fc364004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.440+0000 7fc37e99c700 1 -- 172.21.15.98:0/2348504742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3780d0270 msgr2=0x7fc364003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.440+0000 7fc37e99c700 1 --2- 172.21.15.98:0/2348504742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3780d0270 0x7fc364003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.440+0000 7fc37e99c700 1 -- 172.21.15.98:0/2348504742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc364003fd0 msgr2=0x7fc364004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.440+0000 7fc37e99c700 1 --2- 172.21.15.98:0/2348504742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc364003fd0 0x7fc364004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.440+0000 7fc37e99c700 1 -- 172.21.15.98:0/2348504742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3640035b0 con 0x7fc364004d90 2023-03-03T08:03:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.440+0000 7fc37d99a700 1 --2- 172.21.15.98:0/2348504742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc364003fd0 0x7fc364004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.440+0000 7fc37e99c700 1 --2- 172.21.15.98:0/2348504742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc364004d90 0x7fc364005200 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fc370009fc0 tx=0x7fc364005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f81f526fc063105 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.440+0000 7fc37d199700 1 -- 172.21.15.98:0/2348504742 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37000ec30 con 0x7fc364004d90 2023-03-03T08:03:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.440+0000 7fc37d199700 1 -- 172.21.15.98:0/2348504742 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37000f040 con 0x7fc364004d90 2023-03-03T08:03:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.440+0000 7fc37d199700 1 -- 172.21.15.98:0/2348504742 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc370014480 con 0x7fc364004d90 2023-03-03T08:03:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.441+0000 7fc3803ff700 1 -- 172.21.15.98:0/2348504742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc364004d90 msgr2=0x7fc364005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.441+0000 7fc3803ff700 1 --2- 172.21.15.98:0/2348504742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc364004d90 0x7fc364005200 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fc370009fc0 tx=0x7fc364005880 comp rx=0 tx=0).stop 2023-03-03T08:03:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.441+0000 7fc37e99c700 1 -- 172.21.15.98:0/2348504742 reap_dead start 2023-03-03T08:03:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.441+0000 7fc3803ff700 1 -- 172.21.15.98:0/2348504742 shutdown_connections 2023-03-03T08:03:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.441+0000 7fc3803ff700 1 -- 172.21.15.98:0/2348504742 >> 172.21.15.98:0/2348504742 conn(0x7fc378073e70 msgr2=0x7fc37806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.441+0000 7fc3803ff700 1 -- 172.21.15.98:0/2348504742 shutdown_connections 2023-03-03T08:03:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.441+0000 7fc3803ff700 1 -- 172.21.15.98:0/2348504742 wait complete. 2023-03-03T08:03:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.443+0000 7fc3803ff700 1 Processor -- start 2023-03-03T08:03:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.443+0000 7fc3803ff700 1 -- start start 2023-03-03T08:03:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.444+0000 7fc3803ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc364004d90 0x7fc3781e6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.444+0000 7fc3803ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc378155130 0x7fc3781555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.444+0000 7fc3803ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc37815d3a0 0x7fc37815d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.444+0000 7fc3803ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780b41c0 con 0x7fc37815d3a0 2023-03-03T08:03:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.444+0000 7fc3803ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780b4350 con 0x7fc378155130 2023-03-03T08:03:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.444+0000 7fc3803ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc378155ae0 con 0x7fc364004d90 2023-03-03T08:03:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.445+0000 7fc37e99c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc37815d3a0 0x7fc37815d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.445+0000 7fc37e99c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc37815d3a0 0x7fc37815d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47218/0 (socket says 172.21.15.98:47218) 2023-03-03T08:03:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.445+0000 7fc37e99c700 1 -- 172.21.15.98:0/3745775710 learned_addr learned my addr 172.21.15.98:0/3745775710 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.445+0000 7fc37d99a700 1 --2- 172.21.15.98:0/3745775710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc378155130 0x7fc3781555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.445+0000 7fc37e19b700 1 --2- 172.21.15.98:0/3745775710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc364004d90 0x7fc3781e6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.445+0000 7fc37e99c700 1 -- 172.21.15.98:0/3745775710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc364004d90 msgr2=0x7fc3781e6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.445+0000 7fc37e99c700 1 --2- 172.21.15.98:0/3745775710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc364004d90 0x7fc3781e6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.445+0000 7fc37e99c700 1 -- 172.21.15.98:0/3745775710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc378155130 msgr2=0x7fc3781555a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.445+0000 7fc37e99c700 1 --2- 172.21.15.98:0/3745775710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc378155130 0x7fc3781555a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.445+0000 7fc37e99c700 1 -- 172.21.15.98:0/3745775710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3780d1aa0 con 0x7fc37815d3a0 2023-03-03T08:03:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.446+0000 7fc37e99c700 1 --2- 172.21.15.98:0/3745775710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc37815d3a0 0x7fc37815d7f0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fc370016000 tx=0x7fc370009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.446+0000 7fc36effd700 1 -- 172.21.15.98:0/3745775710 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc370007000 con 0x7fc37815d3a0 2023-03-03T08:03:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.446+0000 7fc36effd700 1 -- 172.21.15.98:0/3745775710 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3700159d0 con 0x7fc37815d3a0 2023-03-03T08:03:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.446+0000 7fc36effd700 1 -- 172.21.15.98:0/3745775710 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc370015dc0 con 0x7fc37815d3a0 2023-03-03T08:03:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.446+0000 7fc3803ff700 1 -- 172.21.15.98:0/3745775710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3780cdbe0 con 0x7fc37815d3a0 2023-03-03T08:03:29.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.446+0000 7fc3803ff700 1 -- 172.21.15.98:0/3745775710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37815ce40 con 0x7fc37815d3a0 2023-03-03T08:03:29.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.447+0000 7fc3803ff700 1 -- 172.21.15.98:0/3745775710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3780b44c0 con 0x7fc37815d3a0 2023-03-03T08:03:29.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.448+0000 7fc36effd700 1 -- 172.21.15.98:0/3745775710 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7fc37002d030 con 0x7fc37815d3a0 2023-03-03T08:03:29.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.449+0000 7fc36effd700 1 --2- 172.21.15.98:0/3745775710 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc35c0683b0 0x7fc35c06a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:29.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.449+0000 7fc36effd700 1 -- 172.21.15.98:0/3745775710 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc3700873e0 con 0x7fc37815d3a0 2023-03-03T08:03:29.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.450+0000 7fc37e19b700 1 --2- 172.21.15.98:0/3745775710 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc35c0683b0 0x7fc35c06a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:29.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.451+0000 7fc37e19b700 1 --2- 172.21.15.98:0/3745775710 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc35c0683b0 0x7fc35c06a860 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc374001eb0 tx=0x7fc374007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:29.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.453+0000 7fc36effd700 1 -- 172.21.15.98:0/3745775710 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc3780b44c0 con 0x7fc37815d3a0 2023-03-03T08:03:29.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.685+0000 7fc3803ff700 1 -- 172.21.15.98:0/3745775710 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000508309", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3780b44c0 con 0x7fc35c0683b0 2023-03-03T08:03:29.695 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:03:29.694+0000 7fcbb3bf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:03:29.696 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:03:29.694+0000 7fcbb3bf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:03:29.696 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:03:29.694+0000 7fcbb3bf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:03:29.696 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:03:29.694+0000 7fcbb3bf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:03:29.696 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:03:29.694+0000 7fcbb3bf9700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:03:29.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.730+0000 7fc36effd700 1 -- 172.21.15.98:0/3745775710 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7fc3780b44c0 con 0x7fc35c0683b0 2023-03-03T08:03:29.733 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4 2023-03-03T08:03:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.733+0000 7fc3803ff700 1 -- 172.21.15.98:0/3745775710 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc35c0683b0 msgr2=0x7fc35c06a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.733+0000 7fc3803ff700 1 --2- 172.21.15.98:0/3745775710 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc35c0683b0 0x7fc35c06a860 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc374001eb0 tx=0x7fc374007040 comp rx=0 tx=0).stop 2023-03-03T08:03:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.733+0000 7fc3803ff700 1 -- 172.21.15.98:0/3745775710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc37815d3a0 msgr2=0x7fc37815d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.733+0000 7fc3803ff700 1 --2- 172.21.15.98:0/3745775710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc37815d3a0 0x7fc37815d7f0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fc370016000 tx=0x7fc370009750 comp rx=0 tx=0).stop 2023-03-03T08:03:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.734+0000 7fc3803ff700 1 -- 172.21.15.98:0/3745775710 shutdown_connections 2023-03-03T08:03:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.734+0000 7fc3803ff700 1 --2- 172.21.15.98:0/3745775710 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc35c0683b0 0x7fc35c06a860 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.734+0000 7fc3803ff700 1 --2- 172.21.15.98:0/3745775710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc37815d3a0 0x7fc37815d7f0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.734+0000 7fc3803ff700 1 --2- 172.21.15.98:0/3745775710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc364004d90 0x7fc3781e6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.734+0000 7fc3803ff700 1 --2- 172.21.15.98:0/3745775710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc378155130 0x7fc3781555a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.734+0000 7fc3803ff700 1 -- 172.21.15.98:0/3745775710 >> 172.21.15.98:0/3745775710 conn(0x7fc378073e70 msgr2=0x7fc3780d2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.734+0000 7fc3803ff700 1 -- 172.21.15.98:0/3745775710 shutdown_connections 2023-03-03T08:03:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:29.734+0000 7fc3803ff700 1 -- 172.21.15.98:0/3745775710 wait complete. 2023-03-03T08:03:29.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs 2023-03-03T08:03:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.008+0000 7f73b65ff700 1 Processor -- start 2023-03-03T08:03:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.008+0000 7f73b65ff700 1 -- start start 2023-03-03T08:03:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.009+0000 7f73b65ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b00d0270 0x7f73b00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.010+0000 7f73b65ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b00cf980 con 0x7f73b006b4b0 2023-03-03T08:03:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.010+0000 7f73b65ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b00d1800 con 0x7f73b015ca70 2023-03-03T08:03:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.010+0000 7f73b65ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00cd5b0 con 0x7f73b00d0270 2023-03-03T08:03:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.010+0000 7f73b4b9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f73b006b4b0 0x7f73b00d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45128/0 (socket says 172.21.15.98:45128) 2023-03-03T08:03:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.010+0000 7f73b4b9c700 1 -- 172.21.15.98:0/2911427505 learned_addr learned my addr 172.21.15.98:0/2911427505 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.010+0000 7f73af7fe700 1 --2- 172.21.15.98:0/2911427505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b00d0270 0x7f73b00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.010+0000 7f73aeffd700 1 -- 172.21.15.98:0/2911427505 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314096627 0 0) 0x7f73b00cf980 con 0x7f73b006b4b0 2023-03-03T08:03:30.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.010+0000 7f73aeffd700 1 -- 172.21.15.98:0/2911427505 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73980035b0 con 0x7f73b006b4b0 2023-03-03T08:03:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.011+0000 7f73aeffd700 1 -- 172.21.15.98:0/2911427505 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504305116 0 0) 0x7f73b00d1800 con 0x7f73b015ca70 2023-03-03T08:03:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.011+0000 7f73aeffd700 1 -- 172.21.15.98:0/2911427505 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73b00cf980 con 0x7f73b015ca70 2023-03-03T08:03:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.011+0000 7f73af7fe700 1 -- 172.21.15.98:0/2911427505 >> v1:172.21.15.100:6789/0 conn(0x7f73b015ca70 legacy=0x7f73b00cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.011+0000 7f73af7fe700 1 -- 172.21.15.98:0/2911427505 >> v1:172.21.15.98:6789/0 conn(0x7f73b006b4b0 legacy=0x7f73b00d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.011+0000 7f73af7fe700 1 -- 172.21.15.98:0/2911427505 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b00cd900 con 0x7f73b00d0270 2023-03-03T08:03:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.011+0000 7f73af7fe700 1 --2- 172.21.15.98:0/2911427505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b00d0270 0x7f73b00cf440 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009870 tx=0x7f73a4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9918979656b03d96 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.011+0000 7f73aeffd700 1 -- 172.21.15.98:0/2911427505 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a4004230 con 0x7f73b00d0270 2023-03-03T08:03:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.011+0000 7f73aeffd700 1 -- 172.21.15.98:0/2911427505 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a4004390 con 0x7f73b00d0270 2023-03-03T08:03:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.011+0000 7f73aeffd700 1 -- 172.21.15.98:0/2911427505 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a40036a0 con 0x7f73b00d0270 2023-03-03T08:03:30.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.012+0000 7f73b65ff700 1 -- 172.21.15.98:0/2911427505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b00d0270 msgr2=0x7f73b00cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:30.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.012+0000 7f73b65ff700 1 --2- 172.21.15.98:0/2911427505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b00d0270 0x7f73b00cf440 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009870 tx=0x7f73a4009b80 comp rx=0 tx=0).stop 2023-03-03T08:03:30.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.012+0000 7f73b65ff700 1 -- 172.21.15.98:0/2911427505 shutdown_connections 2023-03-03T08:03:30.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.012+0000 7f73b65ff700 1 --2- 172.21.15.98:0/2911427505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b00d0270 0x7f73b00cf440 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:30.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.012+0000 7f73b65ff700 1 -- 172.21.15.98:0/2911427505 >> 172.21.15.98:0/2911427505 conn(0x7f73b0073e70 msgr2=0x7f73b006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.012+0000 7f73b65ff700 1 -- 172.21.15.98:0/2911427505 shutdown_connections 2023-03-03T08:03:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.012+0000 7f73b65ff700 1 -- 172.21.15.98:0/2911427505 wait complete. 2023-03-03T08:03:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.014+0000 7f73b65ff700 1 Processor -- start 2023-03-03T08:03:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.014+0000 7f73b65ff700 1 -- start start 2023-03-03T08:03:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.015+0000 7f73b65ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b006b4b0 0x7f73b014df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.015+0000 7f73b65ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73b00d0270 0x7f73b00c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.015+0000 7f73b65ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b015ca70 0x7f73b00b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.015+0000 7f73b65ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b01f1dd0 con 0x7f73b006b4b0 2023-03-03T08:03:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.015+0000 7f73b65ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00ca250 con 0x7f73b00d0270 2023-03-03T08:03:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.015+0000 7f73b65ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00ca4a0 con 0x7f73b015ca70 2023-03-03T08:03:30.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.016+0000 7f73affff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b006b4b0 0x7f73b014df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:30.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.016+0000 7f73affff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b006b4b0 0x7f73b014df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47222/0 (socket says 172.21.15.98:47222) 2023-03-03T08:03:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.016+0000 7f73affff700 1 -- 172.21.15.98:0/3594392553 learned_addr learned my addr 172.21.15.98:0/3594392553 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.016+0000 7f73af7fe700 1 --2- 172.21.15.98:0/3594392553 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73b00d0270 0x7f73b00c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.016+0000 7f73b4b9c700 1 --2- 172.21.15.98:0/3594392553 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b015ca70 0x7f73b00b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.016+0000 7f73affff700 1 -- 172.21.15.98:0/3594392553 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b015ca70 msgr2=0x7f73b00b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.016+0000 7f73affff700 1 --2- 172.21.15.98:0/3594392553 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b015ca70 0x7f73b00b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.016+0000 7f73affff700 1 -- 172.21.15.98:0/3594392553 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73b00d0270 msgr2=0x7f73b00c76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.016+0000 7f73affff700 1 --2- 172.21.15.98:0/3594392553 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73b00d0270 0x7f73b00c76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.016+0000 7f73affff700 1 -- 172.21.15.98:0/3594392553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a40094a0 con 0x7f73b006b4b0 2023-03-03T08:03:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.016+0000 7f73b4b9c700 1 --2- 172.21.15.98:0/3594392553 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b015ca70 0x7f73b00b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:03:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.016+0000 7f73affff700 1 --2- 172.21.15.98:0/3594392553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b006b4b0 0x7f73b014df50 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f73a000c530 tx=0x7f73a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.016+0000 7f73acff9700 1 -- 172.21.15.98:0/3594392553 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a00054e0 con 0x7f73b006b4b0 2023-03-03T08:03:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.017+0000 7f73acff9700 1 -- 172.21.15.98:0/3594392553 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a0003680 con 0x7f73b006b4b0 2023-03-03T08:03:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.017+0000 7f73acff9700 1 -- 172.21.15.98:0/3594392553 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0003a30 con 0x7f73b006b4b0 2023-03-03T08:03:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.017+0000 7f73b65ff700 1 -- 172.21.15.98:0/3594392553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b00d1aa0 con 0x7f73b006b4b0 2023-03-03T08:03:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.017+0000 7f73b65ff700 1 -- 172.21.15.98:0/3594392553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b00cdbe0 con 0x7f73b006b4b0 2023-03-03T08:03:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.018+0000 7f73b65ff700 1 -- 172.21.15.98:0/3594392553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b015ce40 con 0x7f73b006b4b0 2023-03-03T08:03:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.019+0000 7f73acff9700 1 -- 172.21.15.98:0/3594392553 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f73a0019030 con 0x7f73b006b4b0 2023-03-03T08:03:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.020+0000 7f73acff9700 1 --2- 172.21.15.98:0/3594392553 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f739c068380 0x7f739c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.020+0000 7f73acff9700 1 -- 172.21.15.98:0/3594392553 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f73a00850b0 con 0x7f73b006b4b0 2023-03-03T08:03:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.021+0000 7f73af7fe700 1 --2- 172.21.15.98:0/3594392553 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f739c068380 0x7f739c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.022+0000 7f73af7fe700 1 --2- 172.21.15.98:0/3594392553 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f739c068380 0x7f739c06a830 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009870 tx=0x7f73a401c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.024+0000 7f73acff9700 1 -- 172.21.15.98:0/3594392553 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f73a0051830 con 0x7f73b006b4b0 2023-03-03T08:03:30.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.244+0000 7f73b65ff700 1 -- 172.21.15.98:0/3594392553 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""]}) v1 -- 0x7f73b004e6e0 con 0x7f739c068380 2023-03-03T08:03:30.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.250+0000 7f73acff9700 1 -- 172.21.15.98:0/3594392553 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+61 (secure 0 0 0) 0x7f73b004e6e0 con 0x7f739c068380 2023-03-03T08:03:30.251 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T08:03:30.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:30.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subvolume_0000000000508309" 2023-03-03T08:03:30.252 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:30.252 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T08:03:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.252+0000 7f73b65ff700 1 -- 172.21.15.98:0/3594392553 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f739c068380 msgr2=0x7f739c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.252+0000 7f73b65ff700 1 --2- 172.21.15.98:0/3594392553 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f739c068380 0x7f739c06a830 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009870 tx=0x7f73a401c040 comp rx=0 tx=0).stop 2023-03-03T08:03:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.252+0000 7f73b65ff700 1 -- 172.21.15.98:0/3594392553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b006b4b0 msgr2=0x7f73b014df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.252+0000 7f73b65ff700 1 --2- 172.21.15.98:0/3594392553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b006b4b0 0x7f73b014df50 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f73a000c530 tx=0x7f73a000c8f0 comp rx=0 tx=0).stop 2023-03-03T08:03:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.252+0000 7f73b65ff700 1 -- 172.21.15.98:0/3594392553 shutdown_connections 2023-03-03T08:03:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.252+0000 7f73b65ff700 1 --2- 172.21.15.98:0/3594392553 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f739c068380 0x7f739c06a830 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.252+0000 7f73b65ff700 1 --2- 172.21.15.98:0/3594392553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b006b4b0 0x7f73b014df50 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.252+0000 7f73b65ff700 1 --2- 172.21.15.98:0/3594392553 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b015ca70 0x7f73b00b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.252+0000 7f73b65ff700 1 --2- 172.21.15.98:0/3594392553 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73b00d0270 0x7f73b00c76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.252+0000 7f73b65ff700 1 -- 172.21.15.98:0/3594392553 >> 172.21.15.98:0/3594392553 conn(0x7f73b0073e70 msgr2=0x7f73b015a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.252+0000 7f73b65ff700 1 -- 172.21.15.98:0/3594392553 shutdown_connections 2023-03-03T08:03:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.252+0000 7f73b65ff700 1 -- 172.21.15.98:0/3594392553 wait complete. 2023-03-03T08:03:30.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvolume_0000000000508309 2023-03-03T08:03:30.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.528+0000 7f6bf6dff700 1 Processor -- start 2023-03-03T08:03:30.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.528+0000 7f6bf6dff700 1 -- start start 2023-03-03T08:03:30.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.529+0000 7f6bf6dff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6bf00d0270 0x7f6bf00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:30.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.529+0000 7f6bf6dff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6bf006b4b0 0x7f6bf00d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:30.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.530+0000 7f6bf6dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf00cf980 con 0x7f6bf015ca70 2023-03-03T08:03:30.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.530+0000 7f6bf6dff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bf00cd5b0 con 0x7f6bf006b4b0 2023-03-03T08:03:30.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.530+0000 7f6bf6dff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bf00d1800 con 0x7f6bf00d0270 2023-03-03T08:03:30.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.530+0000 7f6bf539c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6bf006b4b0 0x7f6bf00d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:30.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.530+0000 7f6bf539c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6bf006b4b0 0x7f6bf00d1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47236/0 (socket says 172.21.15.98:47236) 2023-03-03T08:03:30.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.530+0000 7f6bf539c700 1 -- 172.21.15.98:0/776962167 learned_addr learned my addr 172.21.15.98:0/776962167 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:30.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.530+0000 7f6beffff700 1 --2- 172.21.15.98:0/776962167 >> v2:172.21.15.100:3300/0 conn(0x7f6bf00d0270 0x7f6bf00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:30.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.530+0000 7f6bf539c700 1 -- 172.21.15.98:0/776962167 >> v2:172.21.15.100:3300/0 conn(0x7f6bf00d0270 msgr2=0x7f6bf00cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:30.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.530+0000 7f6bf539c700 1 --2- 172.21.15.98:0/776962167 >> v2:172.21.15.100:3300/0 conn(0x7f6bf00d0270 0x7f6bf00cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:30.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.531+0000 7f6bf539c700 1 -- 172.21.15.98:0/776962167 >> v1:172.21.15.100:6789/0 conn(0x7f6bf015ca70 legacy=0x7f6bf00cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:30.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.531+0000 7f6bf539c700 1 -- 172.21.15.98:0/776962167 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf00cd900 con 0x7f6bf006b4b0 2023-03-03T08:03:30.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.531+0000 7f6bef7fe700 1 -- 172.21.15.98:0/776962167 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353634069 0 0) 0x7f6bf00cf980 con 0x7f6bf015ca70 2023-03-03T08:03:30.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.531+0000 7f6bf539c700 1 --2- 172.21.15.98:0/776962167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf006b4b0 0x7f6bf00d1fc0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0160560 tx=0x7f6be800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356c48d05a9077de server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:30.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.531+0000 7f6bef7fe700 1 -- 172.21.15.98:0/776962167 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be800e070 con 0x7f6bf006b4b0 2023-03-03T08:03:30.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.531+0000 7f6bef7fe700 1 -- 172.21.15.98:0/776962167 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be800b910 con 0x7f6bf006b4b0 2023-03-03T08:03:30.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.531+0000 7f6bef7fe700 1 -- 172.21.15.98:0/776962167 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be800bcc0 con 0x7f6bf006b4b0 2023-03-03T08:03:30.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.532+0000 7f6bf6dff700 1 -- 172.21.15.98:0/776962167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf006b4b0 msgr2=0x7f6bf00d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:30.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.532+0000 7f6bf6dff700 1 --2- 172.21.15.98:0/776962167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf006b4b0 0x7f6bf00d1fc0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0160560 tx=0x7f6be800b330 comp rx=0 tx=0).stop 2023-03-03T08:03:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.532+0000 7f6bf6dff700 1 -- 172.21.15.98:0/776962167 shutdown_connections 2023-03-03T08:03:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.532+0000 7f6bf6dff700 1 --2- 172.21.15.98:0/776962167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf006b4b0 0x7f6bf00d1fc0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.532+0000 7f6bf6dff700 1 --2- 172.21.15.98:0/776962167 >> v2:172.21.15.100:3300/0 conn(0x7f6bf00d0270 0x7f6bf00cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.532+0000 7f6bf6dff700 1 -- 172.21.15.98:0/776962167 >> 172.21.15.98:0/776962167 conn(0x7f6bf0073e70 msgr2=0x7f6bf006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.532+0000 7f6bf6dff700 1 -- 172.21.15.98:0/776962167 shutdown_connections 2023-03-03T08:03:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.532+0000 7f6bf6dff700 1 -- 172.21.15.98:0/776962167 wait complete. 2023-03-03T08:03:30.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.534+0000 7f6bf6dff700 1 Processor -- start 2023-03-03T08:03:30.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.534+0000 7f6bf6dff700 1 -- start start 2023-03-03T08:03:30.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.535+0000 7f6bf6dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf006b4b0 0x7f6bf00b4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:30.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.535+0000 7f6bf6dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bf00d0270 0x7f6bf00b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:30.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.535+0000 7f6bf6dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6bf015ca70 0x7f6bf00b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:30.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.535+0000 7f6bf6dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf015ff60 con 0x7f6bf006b4b0 2023-03-03T08:03:30.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.535+0000 7f6bf6dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf00b2cf0 con 0x7f6bf00d0270 2023-03-03T08:03:30.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.535+0000 7f6bf6dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf00b2fd0 con 0x7f6bf015ca70 2023-03-03T08:03:30.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.535+0000 7f6bf4b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf006b4b0 0x7f6bf00b4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:30.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.536+0000 7f6bf4b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf006b4b0 0x7f6bf00b4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47252/0 (socket says 172.21.15.98:47252) 2023-03-03T08:03:30.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.536+0000 7f6bf4b9b700 1 -- 172.21.15.98:0/1055396595 learned_addr learned my addr 172.21.15.98:0/1055396595 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:30.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.536+0000 7f6bf539c700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6bf015ca70 0x7f6bf00b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:30.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.536+0000 7f6beffff700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bf00d0270 0x7f6bf00b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:30.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.536+0000 7f6bf4b9b700 1 -- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6bf015ca70 msgr2=0x7f6bf00b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:30.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.536+0000 7f6bf4b9b700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6bf015ca70 0x7f6bf00b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:30.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.536+0000 7f6bf4b9b700 1 -- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bf00d0270 msgr2=0x7f6bf00b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:30.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.536+0000 7f6bf4b9b700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bf00d0270 0x7f6bf00b3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.536+0000 7f6bf4b9b700 1 -- 172.21.15.98:0/1055396595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be8009b30 con 0x7f6bf006b4b0 2023-03-03T08:03:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.536+0000 7f6beffff700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bf00d0270 0x7f6bf00b3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.536+0000 7f6bf539c700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6bf015ca70 0x7f6bf00b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.536+0000 7f6bf4b9b700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf006b4b0 0x7f6bf00b4c70 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f6be40011e0 tx=0x7f6be400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.537+0000 7f6bed7fa700 1 -- 172.21.15.98:0/1055396595 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be40054e0 con 0x7f6bf006b4b0 2023-03-03T08:03:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.537+0000 7f6bed7fa700 1 -- 172.21.15.98:0/1055396595 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be4003680 con 0x7f6bf006b4b0 2023-03-03T08:03:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.537+0000 7f6bf6dff700 1 -- 172.21.15.98:0/1055396595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf00d1aa0 con 0x7f6bf006b4b0 2023-03-03T08:03:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.537+0000 7f6bed7fa700 1 -- 172.21.15.98:0/1055396595 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be4003a30 con 0x7f6bf006b4b0 2023-03-03T08:03:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.537+0000 7f6bf6dff700 1 -- 172.21.15.98:0/1055396595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf00cdbe0 con 0x7f6bf006b4b0 2023-03-03T08:03:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.538+0000 7f6bf6dff700 1 -- 172.21.15.98:0/1055396595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf015ce40 con 0x7f6bf006b4b0 2023-03-03T08:03:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.539+0000 7f6bed7fa700 1 -- 172.21.15.98:0/1055396595 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 86495+0+0 (secure 0 0 0) 0x7f6be4019030 con 0x7f6bf006b4b0 2023-03-03T08:03:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.541+0000 7f6bed7fa700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6bd4068380 0x7f6bd406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.541+0000 7f6bed7fa700 1 -- 172.21.15.98:0/1055396595 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6be40851b0 con 0x7f6bf006b4b0 2023-03-03T08:03:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.541+0000 7f6beffff700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6bd4068380 0x7f6bd406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:30.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.542+0000 7f6beffff700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6bd4068380 0x7f6bd406a830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0158160 tx=0x7f6bdc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:30.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.545+0000 7f6bed7fa700 1 -- 172.21.15.98:0/1055396595 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6be4051930 con 0x7f6bf006b4b0 2023-03-03T08:03:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.598+0000 7f6bed7fa700 1 -- 172.21.15.98:0/1055396595 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 16) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f6be4010070 con 0x7f6bf006b4b0 2023-03-03T08:03:30.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:30.776+0000 7f6bf6dff700 1 -- 172.21.15.98:0/1055396595 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvolume_0000000000508309", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bf004e6e0 con 0x7f6bd4068380 2023-03-03T08:03:31.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.060+0000 7f6bed7fa700 1 -- 172.21.15.98:0/1055396595 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+684 (secure 0 0 0) 0x7f6bf004e6e0 con 0x7f6bd4068380 2023-03-03T08:03:31.063 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:03:31.063 INFO:teuthology.orchestra.run.smithi098.stdout: "atime": "2023-03-03 08:03:28", 2023-03-03T08:03:31.063 INFO:teuthology.orchestra.run.smithi098.stdout: "bytes_pcent": "undefined", 2023-03-03T08:03:31.063 INFO:teuthology.orchestra.run.smithi098.stdout: "bytes_quota": "infinite", 2023-03-03T08:03:31.063 INFO:teuthology.orchestra.run.smithi098.stdout: "bytes_used": 0, 2023-03-03T08:03:31.064 INFO:teuthology.orchestra.run.smithi098.stdout: "created_at": "2023-03-03 08:03:28", 2023-03-03T08:03:31.064 INFO:teuthology.orchestra.run.smithi098.stdout: "ctime": "2023-03-03 08:03:29", 2023-03-03T08:03:31.064 INFO:teuthology.orchestra.run.smithi098.stdout: "data_pool": "cephfs_data", 2023-03-03T08:03:31.064 INFO:teuthology.orchestra.run.smithi098.stdout: "features": [ 2023-03-03T08:03:31.064 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot-clone", 2023-03-03T08:03:31.065 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot-autoprotect" 2023-03-03T08:03:31.065 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:03:31.065 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 0, 2023-03-03T08:03:31.065 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": 16877, 2023-03-03T08:03:31.065 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_addrs": [ 2023-03-03T08:03:31.065 INFO:teuthology.orchestra.run.smithi098.stdout: "172.21.15.98:6789", 2023-03-03T08:03:31.065 INFO:teuthology.orchestra.run.smithi098.stdout: "172.21.15.100:6789", 2023-03-03T08:03:31.065 INFO:teuthology.orchestra.run.smithi098.stdout: "172.21.15.100:6790" 2023-03-03T08:03:31.066 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:03:31.066 INFO:teuthology.orchestra.run.smithi098.stdout: "mtime": "2023-03-03 08:03:28", 2023-03-03T08:03:31.066 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4", 2023-03-03T08:03:31.066 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_namespace": "", 2023-03-03T08:03:31.066 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "complete", 2023-03-03T08:03:31.066 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "subvolume", 2023-03-03T08:03:31.066 INFO:teuthology.orchestra.run.smithi098.stdout: "uid": 0 2023-03-03T08:03:31.067 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:03:31.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.063+0000 7f6bf6dff700 1 -- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6bd4068380 msgr2=0x7f6bd406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:31.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.063+0000 7f6bf6dff700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6bd4068380 0x7f6bd406a830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0158160 tx=0x7f6bdc008040 comp rx=0 tx=0).stop 2023-03-03T08:03:31.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.063+0000 7f6bf6dff700 1 -- 172.21.15.98:0/1055396595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf006b4b0 msgr2=0x7f6bf00b4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:31.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.063+0000 7f6bf6dff700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf006b4b0 0x7f6bf00b4c70 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f6be40011e0 tx=0x7f6be400cae0 comp rx=0 tx=0).stop 2023-03-03T08:03:31.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.063+0000 7f6bf6dff700 1 -- 172.21.15.98:0/1055396595 shutdown_connections 2023-03-03T08:03:31.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.063+0000 7f6bf6dff700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6bd4068380 0x7f6bd406a830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:31.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.063+0000 7f6bf6dff700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf006b4b0 0x7f6bf00b4c70 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:31.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.063+0000 7f6bf6dff700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6bf015ca70 0x7f6bf00b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:31.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.063+0000 7f6bf6dff700 1 --2- 172.21.15.98:0/1055396595 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bf00d0270 0x7f6bf00b3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:31.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.063+0000 7f6bf6dff700 1 -- 172.21.15.98:0/1055396595 >> 172.21.15.98:0/1055396595 conn(0x7f6bf0073e70 msgr2=0x7f6bf00b88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:31.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.063+0000 7f6bf6dff700 1 -- 172.21.15.98:0/1055396595 shutdown_connections 2023-03-03T08:03:31.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.063+0000 7f6bf6dff700 1 -- 172.21.15.98:0/1055396595 wait complete. 2023-03-03T08:03:31.092 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume resize cephfs subvolume_0000000000508309 10485760 2023-03-03T08:03:31.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.341+0000 7f4074bff700 1 Processor -- start 2023-03-03T08:03:31.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.341+0000 7f4074bff700 1 -- start start 2023-03-03T08:03:31.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.343+0000 7f4074bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f407015d010 0x7f40700b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:31.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.343+0000 7f4074bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40700b26e0 con 0x7f407015cc40 2023-03-03T08:03:31.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.343+0000 7f4074bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40701617b0 con 0x7f40700b22c0 2023-03-03T08:03:31.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.343+0000 7f4074bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f407006b900 con 0x7f407015d010 2023-03-03T08:03:31.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.343+0000 7f406e59c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f407015cc40 0x7f40700cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45138/0 (socket says 172.21.15.98:45138) 2023-03-03T08:03:31.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.343+0000 7f406e59c700 1 -- 172.21.15.98:0/751690242 learned_addr learned my addr 172.21.15.98:0/751690242 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:31.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.343+0000 7f406dd9b700 1 --2- 172.21.15.98:0/751690242 >> v2:172.21.15.98:3300/0 conn(0x7f407015d010 0x7f40700b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:31.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.344+0000 7f406dd9b700 1 -- 172.21.15.98:0/751690242 >> v1:172.21.15.100:6789/0 conn(0x7f40700b22c0 legacy=0x7f40700d0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:31.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.344+0000 7f406dd9b700 1 -- 172.21.15.98:0/751690242 >> v1:172.21.15.98:6789/0 conn(0x7f407015cc40 legacy=0x7f40700cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:31.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.344+0000 7f406dd9b700 1 -- 172.21.15.98:0/751690242 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40701609f0 con 0x7f407015d010 2023-03-03T08:03:31.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.344+0000 7f406dd9b700 1 --2- 172.21.15.98:0/751690242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f407015d010 0x7f40700b1d80 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f4064009840 tx=0x7f4064009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9d6fd201bd9550 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:31.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.344+0000 7f406d59a700 1 -- 172.21.15.98:0/751690242 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4064004230 con 0x7f407015d010 2023-03-03T08:03:31.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.344+0000 7f406d59a700 1 -- 172.21.15.98:0/751690242 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4064004390 con 0x7f407015d010 2023-03-03T08:03:31.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.344+0000 7f406d59a700 1 -- 172.21.15.98:0/751690242 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40640036a0 con 0x7f407015d010 2023-03-03T08:03:31.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.345+0000 7f4074bff700 1 -- 172.21.15.98:0/751690242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f407015d010 msgr2=0x7f40700b1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:31.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.345+0000 7f4074bff700 1 --2- 172.21.15.98:0/751690242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f407015d010 0x7f40700b1d80 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f4064009840 tx=0x7f4064009b50 comp rx=0 tx=0).stop 2023-03-03T08:03:31.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.345+0000 7f4074bff700 1 -- 172.21.15.98:0/751690242 shutdown_connections 2023-03-03T08:03:31.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.345+0000 7f4074bff700 1 --2- 172.21.15.98:0/751690242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f407015d010 0x7f40700b1d80 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:31.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.345+0000 7f4074bff700 1 -- 172.21.15.98:0/751690242 >> 172.21.15.98:0/751690242 conn(0x7f4070073e10 msgr2=0x7f40700d31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:31.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.345+0000 7f4074bff700 1 -- 172.21.15.98:0/751690242 shutdown_connections 2023-03-03T08:03:31.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.345+0000 7f4074bff700 1 -- 172.21.15.98:0/751690242 wait complete. 2023-03-03T08:03:31.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.347+0000 7f4074bff700 1 Processor -- start 2023-03-03T08:03:31.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.347+0000 7f4074bff700 1 -- start start 2023-03-03T08:03:31.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.348+0000 7f4074bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40700b22c0 0x7f40701e6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:31.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.348+0000 7f4074bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f407015cc40 0x7f40701e6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:31.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.348+0000 7f4074bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f407015d010 0x7f4070145450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:31.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.348+0000 7f4074bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40700c9820 con 0x7f407015cc40 2023-03-03T08:03:31.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.348+0000 7f4074bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4070156d40 con 0x7f40700b22c0 2023-03-03T08:03:31.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.348+0000 7f4074bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f407015fa20 con 0x7f407015d010 2023-03-03T08:03:31.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.349+0000 7f406e59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40700b22c0 0x7f40701e6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:31.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.349+0000 7f406dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f407015cc40 0x7f40701e6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:31.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.349+0000 7f406dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f407015cc40 0x7f40701e6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47276/0 (socket says 172.21.15.98:47276) 2023-03-03T08:03:31.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.349+0000 7f406dd9b700 1 -- 172.21.15.98:0/2069402361 learned_addr learned my addr 172.21.15.98:0/2069402361 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:31.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.349+0000 7f406ed9d700 1 --2- 172.21.15.98:0/2069402361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f407015d010 0x7f4070145450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:31.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.349+0000 7f406dd9b700 1 -- 172.21.15.98:0/2069402361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f407015d010 msgr2=0x7f4070145450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:31.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.349+0000 7f406dd9b700 1 --2- 172.21.15.98:0/2069402361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f407015d010 0x7f4070145450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:31.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.349+0000 7f406dd9b700 1 -- 172.21.15.98:0/2069402361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40700b22c0 msgr2=0x7f40701e6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:31.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.349+0000 7f406dd9b700 1 --2- 172.21.15.98:0/2069402361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40700b22c0 0x7f40701e6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:31.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.349+0000 7f406dd9b700 1 -- 172.21.15.98:0/2069402361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40640094a0 con 0x7f407015cc40 2023-03-03T08:03:31.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.350+0000 7f406dd9b700 1 --2- 172.21.15.98:0/2069402361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f407015cc40 0x7f40701e6bd0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f4064004000 tx=0x7f4064004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:31.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.350+0000 7f405effd700 1 -- 172.21.15.98:0/2069402361 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4064003cd0 con 0x7f407015cc40 2023-03-03T08:03:31.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.350+0000 7f405effd700 1 -- 172.21.15.98:0/2069402361 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406401f070 con 0x7f407015cc40 2023-03-03T08:03:31.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.350+0000 7f4074bff700 1 -- 172.21.15.98:0/2069402361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4070160fb0 con 0x7f407015cc40 2023-03-03T08:03:31.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.350+0000 7f405effd700 1 -- 172.21.15.98:0/2069402361 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4064011870 con 0x7f407015cc40 2023-03-03T08:03:31.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.350+0000 7f4074bff700 1 -- 172.21.15.98:0/2069402361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4070160cd0 con 0x7f407015cc40 2023-03-03T08:03:31.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.351+0000 7f4074bff700 1 -- 172.21.15.98:0/2069402361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4070161290 con 0x7f407015cc40 2023-03-03T08:03:31.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.353+0000 7f405effd700 1 -- 172.21.15.98:0/2069402361 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f406402a020 con 0x7f407015cc40 2023-03-03T08:03:31.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.354+0000 7f405effd700 1 --2- 172.21.15.98:0/2069402361 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f40500683c0 0x7f405006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:31.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.354+0000 7f405effd700 1 -- 172.21.15.98:0/2069402361 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4064087480 con 0x7f407015cc40 2023-03-03T08:03:31.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.354+0000 7f406e59c700 1 --2- 172.21.15.98:0/2069402361 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f40500683c0 0x7f405006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:31.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.355+0000 7f406e59c700 1 --2- 172.21.15.98:0/2069402361 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f40500683c0 0x7f405006a870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4060000f60 tx=0x7f4060004a50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:31.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.358+0000 7f405effd700 1 -- 172.21.15.98:0/2069402361 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4064053b20 con 0x7f407015cc40 2023-03-03T08:03:31.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.587+0000 7f4074bff700 1 -- 172.21.15.98:0/2069402361 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume resize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000508309", "new_size": "10485760", "target": ["mon-mgr", ""]}) v1 -- 0x7f4070161290 con 0x7f40500683c0 2023-03-03T08:03:31.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.603+0000 7f405effd700 1 -- 172.21.15.98:0/2069402361 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+128 (secure 0 0 0) 0x7f4070161290 con 0x7f40500683c0 2023-03-03T08:03:31.606 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T08:03:31.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:31.606 INFO:teuthology.orchestra.run.smithi098.stdout: "bytes_used": 0 2023-03-03T08:03:31.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:31.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:31.606 INFO:teuthology.orchestra.run.smithi098.stdout: "bytes_quota": 10485760 2023-03-03T08:03:31.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:31.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:31.607 INFO:teuthology.orchestra.run.smithi098.stdout: "bytes_pcent": "0.00" 2023-03-03T08:03:31.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:31.607 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T08:03:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.606+0000 7f4074bff700 1 -- 172.21.15.98:0/2069402361 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f40500683c0 msgr2=0x7f405006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.606+0000 7f4074bff700 1 --2- 172.21.15.98:0/2069402361 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f40500683c0 0x7f405006a870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4060000f60 tx=0x7f4060004a50 comp rx=0 tx=0).stop 2023-03-03T08:03:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.606+0000 7f4074bff700 1 -- 172.21.15.98:0/2069402361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f407015cc40 msgr2=0x7f40701e6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.606+0000 7f4074bff700 1 --2- 172.21.15.98:0/2069402361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f407015cc40 0x7f40701e6bd0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f4064004000 tx=0x7f4064004750 comp rx=0 tx=0).stop 2023-03-03T08:03:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.606+0000 7f4074bff700 1 -- 172.21.15.98:0/2069402361 shutdown_connections 2023-03-03T08:03:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.606+0000 7f4074bff700 1 --2- 172.21.15.98:0/2069402361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f407015d010 0x7f4070145450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.606+0000 7f4074bff700 1 --2- 172.21.15.98:0/2069402361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40700b22c0 0x7f40701e6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.606+0000 7f4074bff700 1 --2- 172.21.15.98:0/2069402361 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f40500683c0 0x7f405006a870 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.606+0000 7f4074bff700 1 --2- 172.21.15.98:0/2069402361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f407015cc40 0x7f40701e6bd0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.606+0000 7f4074bff700 1 -- 172.21.15.98:0/2069402361 >> 172.21.15.98:0/2069402361 conn(0x7f4070073e10 msgr2=0x7f4070158a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.606+0000 7f4074bff700 1 -- 172.21.15.98:0/2069402361 shutdown_connections 2023-03-03T08:03:31.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.606+0000 7f4074bff700 1 -- 172.21.15.98:0/2069402361 wait complete. 2023-03-03T08:03:31.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvolume_0000000000508309 2023-03-03T08:03:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.880+0000 7f5a8259d700 1 Processor -- start 2023-03-03T08:03:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.881+0000 7f5a8259d700 1 -- start start 2023-03-03T08:03:31.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.882+0000 7f5a8259d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5a7c0b4aa0 0x7f5a7c0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:31.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.882+0000 7f5a8259d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7c15dd00 con 0x7f5a7c0cf570 2023-03-03T08:03:31.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.882+0000 7f5a8259d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7c0cd780 con 0x7f5a7c1609b0 2023-03-03T08:03:31.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.882+0000 7f5a8259d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a7c0cebb0 con 0x7f5a7c0b4aa0 2023-03-03T08:03:31.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.883+0000 7f5a80d9a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5a7c0cf570 0x7f5a7c15d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45146/0 (socket says 172.21.15.98:45146) 2023-03-03T08:03:31.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.883+0000 7f5a80d9a700 1 -- 172.21.15.98:0/1442441920 learned_addr learned my addr 172.21.15.98:0/1442441920 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:31.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.883+0000 7f5a81d9c700 1 --2- 172.21.15.98:0/1442441920 >> v2:172.21.15.98:3300/0 conn(0x7f5a7c0b4aa0 0x7f5a7c0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:31.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.883+0000 7f5a81d9c700 1 -- 172.21.15.98:0/1442441920 >> v1:172.21.15.100:6789/0 conn(0x7f5a7c1609b0 legacy=0x7f5a7c160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:31.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.883+0000 7f5a81d9c700 1 -- 172.21.15.98:0/1442441920 >> v1:172.21.15.98:6789/0 conn(0x7f5a7c0cf570 legacy=0x7f5a7c15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.883+0000 7f5a73fff700 1 -- 172.21.15.98:0/1442441920 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643515899 0 0) 0x7f5a7c0cd780 con 0x7f5a7c1609b0 2023-03-03T08:03:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.883+0000 7f5a81d9c700 1 -- 172.21.15.98:0/1442441920 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7c0cda60 con 0x7f5a7c0b4aa0 2023-03-03T08:03:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.884+0000 7f5a81d9c700 1 --2- 172.21.15.98:0/1442441920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a7c0b4aa0 0x7f5a7c0b4f20 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f5a74009d90 tx=0x7f5a7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c60e03ba6c098a4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.884+0000 7f5a73fff700 1 -- 172.21.15.98:0/1442441920 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7400e070 con 0x7f5a7c0b4aa0 2023-03-03T08:03:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.884+0000 7f5a73fff700 1 -- 172.21.15.98:0/1442441920 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7400bad0 con 0x7f5a7c0b4aa0 2023-03-03T08:03:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.884+0000 7f5a73fff700 1 -- 172.21.15.98:0/1442441920 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7400be80 con 0x7f5a7c0b4aa0 2023-03-03T08:03:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.884+0000 7f5a8259d700 1 -- 172.21.15.98:0/1442441920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a7c0b4aa0 msgr2=0x7f5a7c0b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.884+0000 7f5a8259d700 1 --2- 172.21.15.98:0/1442441920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a7c0b4aa0 0x7f5a7c0b4f20 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f5a74009d90 tx=0x7f5a7400b330 comp rx=0 tx=0).stop 2023-03-03T08:03:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.885+0000 7f5a8259d700 1 -- 172.21.15.98:0/1442441920 shutdown_connections 2023-03-03T08:03:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.885+0000 7f5a8259d700 1 --2- 172.21.15.98:0/1442441920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a7c0b4aa0 0x7f5a7c0b4f20 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:31.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.885+0000 7f5a8259d700 1 -- 172.21.15.98:0/1442441920 >> 172.21.15.98:0/1442441920 conn(0x7f5a7c06b880 msgr2=0x7f5a7c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:31.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.885+0000 7f5a8259d700 1 -- 172.21.15.98:0/1442441920 shutdown_connections 2023-03-03T08:03:31.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.885+0000 7f5a8259d700 1 -- 172.21.15.98:0/1442441920 wait complete. 2023-03-03T08:03:31.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.887+0000 7f5a8259d700 1 Processor -- start 2023-03-03T08:03:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.887+0000 7f5a8259d700 1 -- start start 2023-03-03T08:03:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.888+0000 7f5a8259d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a7c0b4aa0 0x7f5a7c150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:31.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.888+0000 7f5a8259d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a7c0cf570 0x7f5a7c0cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:31.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.888+0000 7f5a8259d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5a7c1609b0 0x7f5a7c0cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:31.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.888+0000 7f5a8259d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c15df20 con 0x7f5a7c0cf570 2023-03-03T08:03:31.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.888+0000 7f5a8259d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c15e1e0 con 0x7f5a7c1609b0 2023-03-03T08:03:31.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.888+0000 7f5a8259d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c0ccef0 con 0x7f5a7c0b4aa0 2023-03-03T08:03:31.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.888+0000 7f5a80d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a7c0cf570 0x7f5a7c0cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:31.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.888+0000 7f5a80d9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a7c0cf570 0x7f5a7c0cc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47304/0 (socket says 172.21.15.98:47304) 2023-03-03T08:03:31.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.888+0000 7f5a80d9a700 1 -- 172.21.15.98:0/3097544120 learned_addr learned my addr 172.21.15.98:0/3097544120 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.889+0000 7f5a81d9c700 1 --2- 172.21.15.98:0/3097544120 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5a7c1609b0 0x7f5a7c0cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.889+0000 7f5a8159b700 1 --2- 172.21.15.98:0/3097544120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a7c0b4aa0 0x7f5a7c150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.889+0000 7f5a80d9a700 1 -- 172.21.15.98:0/3097544120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a7c0b4aa0 msgr2=0x7f5a7c150ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.889+0000 7f5a80d9a700 1 --2- 172.21.15.98:0/3097544120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a7c0b4aa0 0x7f5a7c150ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.889+0000 7f5a80d9a700 1 -- 172.21.15.98:0/3097544120 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5a7c1609b0 msgr2=0x7f5a7c0cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.889+0000 7f5a80d9a700 1 --2- 172.21.15.98:0/3097544120 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5a7c1609b0 0x7f5a7c0cc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.889+0000 7f5a80d9a700 1 -- 172.21.15.98:0/3097544120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a74009a70 con 0x7f5a7c0cf570 2023-03-03T08:03:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.889+0000 7f5a8159b700 1 --2- 172.21.15.98:0/3097544120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a7c0b4aa0 0x7f5a7c150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:03:31.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.889+0000 7f5a80d9a700 1 --2- 172.21.15.98:0/3097544120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a7c0cf570 0x7f5a7c0cc470 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c00c780 tx=0x7f5a6c00cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:31.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.889+0000 7f5a71ffb700 1 -- 172.21.15.98:0/3097544120 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c004c80 con 0x7f5a7c0cf570 2023-03-03T08:03:31.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.889+0000 7f5a71ffb700 1 -- 172.21.15.98:0/3097544120 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6c004de0 con 0x7f5a7c0cf570 2023-03-03T08:03:31.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.889+0000 7f5a71ffb700 1 -- 172.21.15.98:0/3097544120 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c007310 con 0x7f5a7c0cf570 2023-03-03T08:03:31.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.890+0000 7f5a8259d700 1 -- 172.21.15.98:0/3097544120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7c15cdf0 con 0x7f5a7c0cf570 2023-03-03T08:03:31.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.890+0000 7f5a8259d700 1 -- 172.21.15.98:0/3097544120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7c0cdd40 con 0x7f5a7c0cf570 2023-03-03T08:03:31.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.891+0000 7f5a8259d700 1 -- 172.21.15.98:0/3097544120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7c15d0d0 con 0x7f5a7c0cf570 2023-03-03T08:03:31.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.892+0000 7f5a71ffb700 1 -- 172.21.15.98:0/3097544120 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f5a6c010070 con 0x7f5a7c0cf570 2023-03-03T08:03:31.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.893+0000 7f5a71ffb700 1 --2- 172.21.15.98:0/3097544120 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5a600683c0 0x7f5a6006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:31.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.893+0000 7f5a71ffb700 1 -- 172.21.15.98:0/3097544120 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5a6c084fe0 con 0x7f5a7c0cf570 2023-03-03T08:03:31.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.894+0000 7f5a8159b700 1 --2- 172.21.15.98:0/3097544120 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5a600683c0 0x7f5a6006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:31.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.894+0000 7f5a8159b700 1 --2- 172.21.15.98:0/3097544120 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5a600683c0 0x7f5a6006a870 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5a78000f60 tx=0x7f5a78009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:31.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:31.897+0000 7f5a71ffb700 1 -- 172.21.15.98:0/3097544120 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5a6c0515f0 con 0x7f5a7c0cf570 2023-03-03T08:03:32.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.121+0000 7f5a8259d700 1 -- 172.21.15.98:0/3097544120 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvolume_0000000000508309", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a7c068290 con 0x7f5a600683c0 2023-03-03T08:03:32.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.135+0000 7f5a71ffb700 1 -- 172.21.15.98:0/3097544120 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+677 (secure 0 0 0) 0x7f5a7c068290 con 0x7f5a600683c0 2023-03-03T08:03:32.137 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:03:32.137 INFO:teuthology.orchestra.run.smithi098.stdout: "atime": "2023-03-03 08:03:28", 2023-03-03T08:03:32.137 INFO:teuthology.orchestra.run.smithi098.stdout: "bytes_pcent": "0.00", 2023-03-03T08:03:32.137 INFO:teuthology.orchestra.run.smithi098.stdout: "bytes_quota": 10485760, 2023-03-03T08:03:32.137 INFO:teuthology.orchestra.run.smithi098.stdout: "bytes_used": 0, 2023-03-03T08:03:32.137 INFO:teuthology.orchestra.run.smithi098.stdout: "created_at": "2023-03-03 08:03:28", 2023-03-03T08:03:32.138 INFO:teuthology.orchestra.run.smithi098.stdout: "ctime": "2023-03-03 08:03:31", 2023-03-03T08:03:32.138 INFO:teuthology.orchestra.run.smithi098.stdout: "data_pool": "cephfs_data", 2023-03-03T08:03:32.138 INFO:teuthology.orchestra.run.smithi098.stdout: "features": [ 2023-03-03T08:03:32.138 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot-clone", 2023-03-03T08:03:32.138 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot-autoprotect" 2023-03-03T08:03:32.138 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:03:32.138 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 0, 2023-03-03T08:03:32.138 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": 16877, 2023-03-03T08:03:32.139 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_addrs": [ 2023-03-03T08:03:32.139 INFO:teuthology.orchestra.run.smithi098.stdout: "172.21.15.98:6789", 2023-03-03T08:03:32.139 INFO:teuthology.orchestra.run.smithi098.stdout: "172.21.15.100:6789", 2023-03-03T08:03:32.139 INFO:teuthology.orchestra.run.smithi098.stdout: "172.21.15.100:6790" 2023-03-03T08:03:32.139 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:03:32.139 INFO:teuthology.orchestra.run.smithi098.stdout: "mtime": "2023-03-03 08:03:28", 2023-03-03T08:03:32.139 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4", 2023-03-03T08:03:32.139 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_namespace": "", 2023-03-03T08:03:32.140 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "complete", 2023-03-03T08:03:32.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "subvolume", 2023-03-03T08:03:32.140 INFO:teuthology.orchestra.run.smithi098.stdout: "uid": 0 2023-03-03T08:03:32.140 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:03:32.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.136+0000 7f5a8259d700 1 -- 172.21.15.98:0/3097544120 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5a600683c0 msgr2=0x7f5a6006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:32.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.136+0000 7f5a8259d700 1 --2- 172.21.15.98:0/3097544120 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5a600683c0 0x7f5a6006a870 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5a78000f60 tx=0x7f5a78009040 comp rx=0 tx=0).stop 2023-03-03T08:03:32.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.136+0000 7f5a8259d700 1 -- 172.21.15.98:0/3097544120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a7c0cf570 msgr2=0x7f5a7c0cc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:32.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.136+0000 7f5a8259d700 1 --2- 172.21.15.98:0/3097544120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a7c0cf570 0x7f5a7c0cc470 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c00c780 tx=0x7f5a6c00cb40 comp rx=0 tx=0).stop 2023-03-03T08:03:32.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.137+0000 7f5a8259d700 1 -- 172.21.15.98:0/3097544120 shutdown_connections 2023-03-03T08:03:32.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.137+0000 7f5a8259d700 1 --2- 172.21.15.98:0/3097544120 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5a7c1609b0 0x7f5a7c0cc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.137+0000 7f5a8259d700 1 --2- 172.21.15.98:0/3097544120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5a7c0b4aa0 0x7f5a7c150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.137+0000 7f5a8259d700 1 --2- 172.21.15.98:0/3097544120 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f5a600683c0 0x7f5a6006a870 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.137+0000 7f5a8259d700 1 --2- 172.21.15.98:0/3097544120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a7c0cf570 0x7f5a7c0cc470 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.137+0000 7f5a8259d700 1 -- 172.21.15.98:0/3097544120 >> 172.21.15.98:0/3097544120 conn(0x7f5a7c06b880 msgr2=0x7f5a7c0d2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:32.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.137+0000 7f5a8259d700 1 -- 172.21.15.98:0/3097544120 shutdown_connections 2023-03-03T08:03:32.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.137+0000 7f5a8259d700 1 -- 172.21.15.98:0/3097544120 wait complete. 2023-03-03T08:03:32.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000508309 --mode 777 --uid 1000 --gid 1000 2023-03-03T08:03:32.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.418+0000 7f7a81bff700 1 Processor -- start 2023-03-03T08:03:32.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.418+0000 7f7a81bff700 1 -- start start 2023-03-03T08:03:32.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.420+0000 7f7a81bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7a7c15ca70 0x7f7a7c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:32.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.420+0000 7f7a81bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7a7c06b4b0 0x7f7a7c0cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:32.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.420+0000 7f7a81bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a7c0d00a0 0x7f7a7c0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:32.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.420+0000 7f7a81bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a7c0cd5b0 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.420+0000 7f7a81bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a7c0cf980 con 0x7f7a7c15ca70 2023-03-03T08:03:32.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.420+0000 7f7a81bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c0d0510 con 0x7f7a7c0d00a0 2023-03-03T08:03:32.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.420+0000 7f7a73fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7a7c06b4b0 0x7f7a7c0cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:32.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.420+0000 7f7a73fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7a7c06b4b0 0x7f7a7c0cfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47316/0 (socket says 172.21.15.98:47316) 2023-03-03T08:03:32.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.420+0000 7f7a73fff700 1 -- 172.21.15.98:0/2970908307 learned_addr learned my addr 172.21.15.98:0/2970908307 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:32.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.420+0000 7f7a7b7fe700 1 --2- 172.21.15.98:0/2970908307 >> v2:172.21.15.100:3300/0 conn(0x7f7a7c15ca70 0x7f7a7c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:32.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.420+0000 7f7a7bfff700 1 --2- 172.21.15.98:0/2970908307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a7c0d00a0 0x7f7a7c0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:32.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.421+0000 7f7a73fff700 1 -- 172.21.15.98:0/2970908307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a7c0d00a0 msgr2=0x7f7a7c0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:32.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.421+0000 7f7a73fff700 1 --2- 172.21.15.98:0/2970908307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a7c0d00a0 0x7f7a7c0b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.421+0000 7f7a73fff700 1 -- 172.21.15.98:0/2970908307 >> v2:172.21.15.100:3300/0 conn(0x7f7a7c15ca70 msgr2=0x7f7a7c0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:32.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.421+0000 7f7a73fff700 1 --2- 172.21.15.98:0/2970908307 >> v2:172.21.15.100:3300/0 conn(0x7f7a7c15ca70 0x7f7a7c0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.421+0000 7f7a73fff700 1 -- 172.21.15.98:0/2970908307 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c0d1800 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.421+0000 7f7a73fff700 1 --2- 172.21.15.98:0/2970908307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a7c06b4b0 0x7f7a7c0cfb60 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c06c2c0 tx=0x7f7a64009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63fca585afbcd7e5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:32.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.421+0000 7f7a7affd700 1 -- 172.21.15.98:0/2970908307 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64011830 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.422+0000 7f7a7affd700 1 -- 172.21.15.98:0/2970908307 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a64011990 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.422+0000 7f7a7affd700 1 -- 172.21.15.98:0/2970908307 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64011d20 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.422+0000 7f7a81bff700 1 -- 172.21.15.98:0/2970908307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a7c06b4b0 msgr2=0x7f7a7c0cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:32.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.422+0000 7f7a81bff700 1 --2- 172.21.15.98:0/2970908307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a7c06b4b0 0x7f7a7c0cfb60 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c06c2c0 tx=0x7f7a64009c00 comp rx=0 tx=0).stop 2023-03-03T08:03:32.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.422+0000 7f7a81bff700 1 -- 172.21.15.98:0/2970908307 shutdown_connections 2023-03-03T08:03:32.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.422+0000 7f7a81bff700 1 --2- 172.21.15.98:0/2970908307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a7c0d00a0 0x7f7a7c0b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.422+0000 7f7a81bff700 1 --2- 172.21.15.98:0/2970908307 >> v2:172.21.15.100:3300/0 conn(0x7f7a7c15ca70 0x7f7a7c0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.422+0000 7f7a81bff700 1 --2- 172.21.15.98:0/2970908307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a7c06b4b0 0x7f7a7c0cfb60 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.422+0000 7f7a81bff700 1 -- 172.21.15.98:0/2970908307 >> 172.21.15.98:0/2970908307 conn(0x7f7a7c073e70 msgr2=0x7f7a7c06ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:32.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.422+0000 7f7a81bff700 1 -- 172.21.15.98:0/2970908307 shutdown_connections 2023-03-03T08:03:32.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.422+0000 7f7a81bff700 1 -- 172.21.15.98:0/2970908307 wait complete. 2023-03-03T08:03:32.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.424+0000 7f7a81bff700 1 Processor -- start 2023-03-03T08:03:32.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.425+0000 7f7a81bff700 1 -- start start 2023-03-03T08:03:32.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a81bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a7c06b4b0 0x7f7a7c1f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:32.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a81bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7a7c0d00a0 0x7f7a7c0caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a81bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a7c15ca70 0x7f7a7c0cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a81bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c0cbf80 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a81bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c0cc260 con 0x7f7a7c0d00a0 2023-03-03T08:03:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a81bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c0ca3f0 con 0x7f7a7c15ca70 2023-03-03T08:03:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a7b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a7c06b4b0 0x7f7a7c1f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a7b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a7c06b4b0 0x7f7a7c1f1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47330/0 (socket says 172.21.15.98:47330) 2023-03-03T08:03:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a7b7fe700 1 -- 172.21.15.98:0/404920627 learned_addr learned my addr 172.21.15.98:0/404920627 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a73fff700 1 --2- 172.21.15.98:0/404920627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7a7c0d00a0 0x7f7a7c0caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:32.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a7bfff700 1 --2- 172.21.15.98:0/404920627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a7c15ca70 0x7f7a7c0cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:32.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a7b7fe700 1 -- 172.21.15.98:0/404920627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a7c15ca70 msgr2=0x7f7a7c0cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:32.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a7b7fe700 1 --2- 172.21.15.98:0/404920627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a7c15ca70 0x7f7a7c0cb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a7b7fe700 1 -- 172.21.15.98:0/404920627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7a7c0d00a0 msgr2=0x7f7a7c0caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:32.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a7b7fe700 1 --2- 172.21.15.98:0/404920627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7a7c0d00a0 0x7f7a7c0caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a7b7fe700 1 -- 172.21.15.98:0/404920627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a64009550 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.426+0000 7f7a73fff700 1 --2- 172.21.15.98:0/404920627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7a7c0d00a0 0x7f7a7c0caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:03:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.427+0000 7f7a7b7fe700 1 --2- 172.21.15.98:0/404920627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a7c06b4b0 0x7f7a7c1f1ea0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00e870 tx=0x7f7a6c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.427+0000 7f7a78ff9700 1 -- 172.21.15.98:0/404920627 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c009590 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.427+0000 7f7a78ff9700 1 -- 172.21.15.98:0/404920627 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6c013680 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.427+0000 7f7a81bff700 1 -- 172.21.15.98:0/404920627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7c0cdbe0 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.427+0000 7f7a78ff9700 1 -- 172.21.15.98:0/404920627 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c0139f0 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.427+0000 7f7a81bff700 1 -- 172.21.15.98:0/404920627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7c0cd900 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.428+0000 7f7a81bff700 1 -- 172.21.15.98:0/404920627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7c0d1a80 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.430+0000 7f7a78ff9700 1 -- 172.21.15.98:0/404920627 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f7a6c019030 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.431+0000 7f7a78ff9700 1 --2- 172.21.15.98:0/404920627 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7a5c0683c0 0x7f7a5c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.431+0000 7f7a78ff9700 1 -- 172.21.15.98:0/404920627 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7a6c085b70 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.431+0000 7f7a73fff700 1 --2- 172.21.15.98:0/404920627 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7a5c0683c0 0x7f7a5c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:32.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.432+0000 7f7a73fff700 1 --2- 172.21.15.98:0/404920627 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7a5c0683c0 0x7f7a5c06a870 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c06c2c0 tx=0x7f7a64008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:32.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.434+0000 7f7a78ff9700 1 -- 172.21.15.98:0/404920627 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7a6c0521b0 con 0x7f7a7c06b4b0 2023-03-03T08:03:32.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.662+0000 7f7a81bff700 1 -- 172.21.15.98:0/404920627 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000508309", "mode": "777", "uid": 1000, "gid": 1000, "target": ["mon-mgr", ""]}) v1 -- 0x7f7a7c0ca230 con 0x7f7a5c0683c0 2023-03-03T08:03:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.682+0000 7f7a78ff9700 1 -- 172.21.15.98:0/404920627 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7a7c0ca230 con 0x7f7a5c0683c0 2023-03-03T08:03:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.684+0000 7f7a81bff700 1 -- 172.21.15.98:0/404920627 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7a5c0683c0 msgr2=0x7f7a5c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.684+0000 7f7a81bff700 1 --2- 172.21.15.98:0/404920627 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7a5c0683c0 0x7f7a5c06a870 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c06c2c0 tx=0x7f7a64008030 comp rx=0 tx=0).stop 2023-03-03T08:03:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.684+0000 7f7a81bff700 1 -- 172.21.15.98:0/404920627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a7c06b4b0 msgr2=0x7f7a7c1f1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.684+0000 7f7a81bff700 1 --2- 172.21.15.98:0/404920627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a7c06b4b0 0x7f7a7c1f1ea0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00e870 tx=0x7f7a6c00eb80 comp rx=0 tx=0).stop 2023-03-03T08:03:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.684+0000 7f7a81bff700 1 -- 172.21.15.98:0/404920627 shutdown_connections 2023-03-03T08:03:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.684+0000 7f7a81bff700 1 --2- 172.21.15.98:0/404920627 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f7a5c0683c0 0x7f7a5c06a870 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.684+0000 7f7a81bff700 1 --2- 172.21.15.98:0/404920627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a7c06b4b0 0x7f7a7c1f1ea0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.684+0000 7f7a81bff700 1 --2- 172.21.15.98:0/404920627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a7c15ca70 0x7f7a7c0cb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.684+0000 7f7a81bff700 1 --2- 172.21.15.98:0/404920627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7a7c0d00a0 0x7f7a7c0caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.684+0000 7f7a81bff700 1 -- 172.21.15.98:0/404920627 >> 172.21.15.98:0/404920627 conn(0x7f7a7c073e70 msgr2=0x7f7a7c0b7750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.684+0000 7f7a81bff700 1 -- 172.21.15.98:0/404920627 shutdown_connections 2023-03-03T08:03:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.684+0000 7f7a81bff700 1 -- 172.21.15.98:0/404920627 wait complete. 2023-03-03T08:03:32.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000508309 2023-03-03T08:03:32.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.961+0000 7f6d6e1ff700 1 Processor -- start 2023-03-03T08:03:32.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.961+0000 7f6d6e1ff700 1 -- start start 2023-03-03T08:03:32.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.962+0000 7f6d6e1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d6815ca70 0x7f6d680cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:32.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.962+0000 7f6d6e1ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6d6806b4b0 0x7f6d680cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:32.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.962+0000 7f6d6e1ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d680cf980 con 0x7f6d680d00a0 2023-03-03T08:03:32.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.962+0000 7f6d6e1ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d680cd5b0 con 0x7f6d6806b4b0 2023-03-03T08:03:32.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.962+0000 7f6d6e1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d680d0510 con 0x7f6d6815ca70 2023-03-03T08:03:32.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.963+0000 7f6d66ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6d6806b4b0 0x7f6d680cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:32.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.963+0000 7f6d67fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6d680d00a0 0x7f6d680b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45154/0 (socket says 172.21.15.98:45154) 2023-03-03T08:03:32.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.963+0000 7f6d67fff700 1 -- 172.21.15.98:0/2711439321 learned_addr learned my addr 172.21.15.98:0/2711439321 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:32.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.963+0000 7f6d677fe700 1 --2- 172.21.15.98:0/2711439321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d6815ca70 0x7f6d680cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:32.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.963+0000 7f6d667fc700 1 -- 172.21.15.98:0/2711439321 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947362871 0 0) 0x7f6d680cf980 con 0x7f6d680d00a0 2023-03-03T08:03:32.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.963+0000 7f6d667fc700 1 -- 172.21.15.98:0/2711439321 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d500035b0 con 0x7f6d680d00a0 2023-03-03T08:03:32.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.964+0000 7f6d66ffd700 1 -- 172.21.15.98:0/2711439321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d6815ca70 msgr2=0x7f6d680cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:32.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.964+0000 7f6d66ffd700 1 --2- 172.21.15.98:0/2711439321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d6815ca70 0x7f6d680cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.964+0000 7f6d66ffd700 1 -- 172.21.15.98:0/2711439321 >> v1:172.21.15.98:6789/0 conn(0x7f6d680d00a0 legacy=0x7f6d680b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:32.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.964+0000 7f6d66ffd700 1 -- 172.21.15.98:0/2711439321 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d680d2fd0 con 0x7f6d6806b4b0 2023-03-03T08:03:32.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.964+0000 7f6d677fe700 1 --2- 172.21.15.98:0/2711439321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d6815ca70 0x7f6d680cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:03:32.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.964+0000 7f6d66ffd700 1 --2- 172.21.15.98:0/2711439321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d6806b4b0 0x7f6d680cfb60 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c009840 tx=0x7f6d5c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a05490259e410a0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:32.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.964+0000 7f6d667fc700 1 -- 172.21.15.98:0/2711439321 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c011830 con 0x7f6d6806b4b0 2023-03-03T08:03:32.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.964+0000 7f6d667fc700 1 -- 172.21.15.98:0/2711439321 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5c011990 con 0x7f6d6806b4b0 2023-03-03T08:03:32.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.964+0000 7f6d667fc700 1 -- 172.21.15.98:0/2711439321 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c011d20 con 0x7f6d6806b4b0 2023-03-03T08:03:32.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.965+0000 7f6d6e1ff700 1 -- 172.21.15.98:0/2711439321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d6806b4b0 msgr2=0x7f6d680cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:32.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.965+0000 7f6d6e1ff700 1 --2- 172.21.15.98:0/2711439321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d6806b4b0 0x7f6d680cfb60 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c009840 tx=0x7f6d5c009b50 comp rx=0 tx=0).stop 2023-03-03T08:03:32.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.965+0000 7f6d6e1ff700 1 -- 172.21.15.98:0/2711439321 shutdown_connections 2023-03-03T08:03:32.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.965+0000 7f6d6e1ff700 1 --2- 172.21.15.98:0/2711439321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d6815ca70 0x7f6d680cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.965+0000 7f6d6e1ff700 1 --2- 172.21.15.98:0/2711439321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d6806b4b0 0x7f6d680cfb60 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.965+0000 7f6d6e1ff700 1 -- 172.21.15.98:0/2711439321 >> 172.21.15.98:0/2711439321 conn(0x7f6d68073e70 msgr2=0x7f6d6806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:32.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.965+0000 7f6d6e1ff700 1 -- 172.21.15.98:0/2711439321 shutdown_connections 2023-03-03T08:03:32.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.965+0000 7f6d6e1ff700 1 -- 172.21.15.98:0/2711439321 wait complete. 2023-03-03T08:03:32.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.967+0000 7f6d6e1ff700 1 Processor -- start 2023-03-03T08:03:32.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.967+0000 7f6d6e1ff700 1 -- start start 2023-03-03T08:03:32.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.968+0000 7f6d6e1ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d6806b4b0 0x7f6d6815f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:32.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.968+0000 7f6d6e1ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d680d00a0 0x7f6d6815f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:32.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.968+0000 7f6d6e1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d6815ca70 0x7f6d6815d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:32.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.968+0000 7f6d6e1ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6815ff60 con 0x7f6d6806b4b0 2023-03-03T08:03:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.968+0000 7f6d6e1ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6815a580 con 0x7f6d680d00a0 2023-03-03T08:03:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.968+0000 7f6d6e1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d680cbb50 con 0x7f6d6815ca70 2023-03-03T08:03:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.968+0000 7f6d677fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d6806b4b0 0x7f6d6815f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.969+0000 7f6d677fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d6806b4b0 0x7f6d6815f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47348/0 (socket says 172.21.15.98:47348) 2023-03-03T08:03:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.969+0000 7f6d677fe700 1 -- 172.21.15.98:0/3102745762 learned_addr learned my addr 172.21.15.98:0/3102745762 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.969+0000 7f6d66ffd700 1 --2- 172.21.15.98:0/3102745762 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d680d00a0 0x7f6d6815f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.969+0000 7f6d67fff700 1 --2- 172.21.15.98:0/3102745762 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d6815ca70 0x7f6d6815d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.969+0000 7f6d677fe700 1 -- 172.21.15.98:0/3102745762 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d6815ca70 msgr2=0x7f6d6815d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.969+0000 7f6d677fe700 1 --2- 172.21.15.98:0/3102745762 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d6815ca70 0x7f6d6815d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.969+0000 7f6d677fe700 1 -- 172.21.15.98:0/3102745762 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d680d00a0 msgr2=0x7f6d6815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.969+0000 7f6d677fe700 1 --2- 172.21.15.98:0/3102745762 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d680d00a0 0x7f6d6815f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.969+0000 7f6d677fe700 1 -- 172.21.15.98:0/3102745762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d580094a0 con 0x7f6d6806b4b0 2023-03-03T08:03:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.969+0000 7f6d677fe700 1 --2- 172.21.15.98:0/3102745762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d6806b4b0 0x7f6d6815f2f0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f6d5800ebe0 tx=0x7f6d58009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:32.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.969+0000 7f6d4ffff700 1 -- 172.21.15.98:0/3102745762 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5800c910 con 0x7f6d6806b4b0 2023-03-03T08:03:32.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.970+0000 7f6d4ffff700 1 -- 172.21.15.98:0/3102745762 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5800ca70 con 0x7f6d6806b4b0 2023-03-03T08:03:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.970+0000 7f6d6e1ff700 1 -- 172.21.15.98:0/3102745762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5c0094a0 con 0x7f6d6806b4b0 2023-03-03T08:03:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.970+0000 7f6d4ffff700 1 -- 172.21.15.98:0/3102745762 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5800cde0 con 0x7f6d6806b4b0 2023-03-03T08:03:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.970+0000 7f6d6e1ff700 1 -- 172.21.15.98:0/3102745762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d680cd900 con 0x7f6d6806b4b0 2023-03-03T08:03:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.971+0000 7f6d6e1ff700 1 -- 172.21.15.98:0/3102745762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d680d1800 con 0x7f6d6806b4b0 2023-03-03T08:03:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.972+0000 7f6d4ffff700 1 -- 172.21.15.98:0/3102745762 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f6d58019030 con 0x7f6d6806b4b0 2023-03-03T08:03:32.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.973+0000 7f6d4ffff700 1 --2- 172.21.15.98:0/3102745762 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6d540683c0 0x7f6d5406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:32.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.973+0000 7f6d4ffff700 1 -- 172.21.15.98:0/3102745762 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6d58085fe0 con 0x7f6d6806b4b0 2023-03-03T08:03:32.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.974+0000 7f6d66ffd700 1 --2- 172.21.15.98:0/3102745762 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6d540683c0 0x7f6d5406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:32.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.975+0000 7f6d66ffd700 1 --2- 172.21.15.98:0/3102745762 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6d540683c0 0x7f6d5406a870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c009840 tx=0x7f6d5c01c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:32.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:32.977+0000 7f6d4ffff700 1 -- 172.21.15.98:0/3102745762 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6d58052600 con 0x7f6d6806b4b0 2023-03-03T08:03:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:33.201+0000 7f6d6e1ff700 1 -- 172.21.15.98:0/3102745762 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000508309", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d680cb9a0 con 0x7f6d540683c0 2023-03-03T08:03:33.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:33.215+0000 7f6d4ffff700 1 -- 172.21.15.98:0/3102745762 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7f6d680cb9a0 con 0x7f6d540683c0 2023-03-03T08:03:33.217 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4 2023-03-03T08:03:33.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:33.217+0000 7f6d6e1ff700 1 -- 172.21.15.98:0/3102745762 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6d540683c0 msgr2=0x7f6d5406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:33.217+0000 7f6d6e1ff700 1 --2- 172.21.15.98:0/3102745762 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6d540683c0 0x7f6d5406a870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c009840 tx=0x7f6d5c01c040 comp rx=0 tx=0).stop 2023-03-03T08:03:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:33.217+0000 7f6d6e1ff700 1 -- 172.21.15.98:0/3102745762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d6806b4b0 msgr2=0x7f6d6815f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:33.217+0000 7f6d6e1ff700 1 --2- 172.21.15.98:0/3102745762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d6806b4b0 0x7f6d6815f2f0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f6d5800ebe0 tx=0x7f6d58009ec0 comp rx=0 tx=0).stop 2023-03-03T08:03:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:33.217+0000 7f6d6e1ff700 1 -- 172.21.15.98:0/3102745762 shutdown_connections 2023-03-03T08:03:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:33.217+0000 7f6d6e1ff700 1 --2- 172.21.15.98:0/3102745762 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f6d540683c0 0x7f6d5406a870 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:33.217+0000 7f6d6e1ff700 1 --2- 172.21.15.98:0/3102745762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d6806b4b0 0x7f6d6815f2f0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:33.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:33.217+0000 7f6d6e1ff700 1 --2- 172.21.15.98:0/3102745762 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6d6815ca70 0x7f6d6815d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:33.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:33.217+0000 7f6d6e1ff700 1 --2- 172.21.15.98:0/3102745762 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d680d00a0 0x7f6d6815f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:33.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:33.217+0000 7f6d6e1ff700 1 -- 172.21.15.98:0/3102745762 >> 172.21.15.98:0/3102745762 conn(0x7f6d68073e70 msgr2=0x7f6d680cf400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:33.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:33.217+0000 7f6d6e1ff700 1 -- 172.21.15.98:0/3102745762 shutdown_connections 2023-03-03T08:03:33.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:33.217+0000 7f6d6e1ff700 1 -- 172.21.15.98:0/3102745762 wait complete. 2023-03-03T08:03:33.247 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000508309 with 8 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4 2023-03-03T08:03:33.247 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:03:33.248 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:03:33.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:33.317 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2023-03-03T08:03:33.411 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-03-03T08:03:33.411 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-03-03T08:03:33.411 INFO:teuthology.orchestra.run.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215744 s, 48.6 MB/s 2023-03-03T08:03:33.413 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:03:33.413 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:03:33.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:33.485 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2023-03-03T08:03:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-03-03T08:03:33.587 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-03-03T08:03:33.587 INFO:teuthology.orchestra.run.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241211 s, 43.5 MB/s 2023-03-03T08:03:33.589 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:03:33.589 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:03:33.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:33.659 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2023-03-03T08:03:33.761 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-03-03T08:03:33.761 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-03-03T08:03:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239287 s, 43.8 MB/s 2023-03-03T08:03:33.763 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:03:33.763 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:03:33.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:33.832 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2023-03-03T08:03:33.933 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-03-03T08:03:33.934 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-03-03T08:03:33.934 INFO:teuthology.orchestra.run.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226381 s, 46.3 MB/s 2023-03-03T08:03:33.935 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:03:33.936 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:03:33.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:03:33.945 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:03:34.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:34.008 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2023-03-03T08:03:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-03-03T08:03:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-03-03T08:03:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235046 s, 44.6 MB/s 2023-03-03T08:03:34.150 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:03:34.151 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:03:34.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:34.219 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2023-03-03T08:03:34.319 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-03-03T08:03:34.319 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-03-03T08:03:34.320 INFO:teuthology.orchestra.run.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237158 s, 44.2 MB/s 2023-03-03T08:03:34.321 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:03:34.321 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:03:34.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:34.429 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2023-03-03T08:03:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-03-03T08:03:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-03-03T08:03:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203538 s, 51.5 MB/s 2023-03-03T08:03:34.527 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:03:34.528 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:03:34.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:34.596 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2023-03-03T08:03:34.695 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-03-03T08:03:34.695 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-03-03T08:03:34.696 INFO:teuthology.orchestra.run.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208963 s, 50.2 MB/s 2023-03-03T08:03:34.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvolume_0000000000508309 snapshot_0000000000029275 2023-03-03T08:03:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.947+0000 7f9ac259d700 1 Processor -- start 2023-03-03T08:03:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.948+0000 7f9ac259d700 1 -- start start 2023-03-03T08:03:34.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.949+0000 7f9ac259d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9abc0b3550 0x7f9abc0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:34.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.949+0000 7f9ac259d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9abc0cee50 0x7f9abc0cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:34.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.949+0000 7f9ac259d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9abc0b3c80 con 0x7f9abc073ed0 2023-03-03T08:03:34.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.949+0000 7f9ac259d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9abc0b4370 con 0x7f9abc0cee50 2023-03-03T08:03:34.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.949+0000 7f9ac259d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9abc0b3ed0 con 0x7f9abc0b3550 2023-03-03T08:03:34.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.950+0000 7f9ac159b700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9abc0b3550 0x7f9abc0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:34.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.950+0000 7f9ac0d9a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9abc073ed0 0x7f9abc160780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44656/0 (socket says 172.21.15.98:44656) 2023-03-03T08:03:34.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.950+0000 7f9ac1d9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9abc0cee50 0x7f9abc0cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:34.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.950+0000 7f9ac0d9a700 1 -- 172.21.15.98:0/69090616 learned_addr learned my addr 172.21.15.98:0/69090616 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.950+0000 7f9ac1d9c700 1 -- 172.21.15.98:0/69090616 >> v2:172.21.15.100:3300/0 conn(0x7f9abc0b3550 msgr2=0x7f9abc0cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.950+0000 7f9ac1d9c700 1 --2- 172.21.15.98:0/69090616 >> v2:172.21.15.100:3300/0 conn(0x7f9abc0b3550 0x7f9abc0cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.950+0000 7f9ac1d9c700 1 -- 172.21.15.98:0/69090616 >> v1:172.21.15.98:6789/0 conn(0x7f9abc073ed0 legacy=0x7f9abc160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.950+0000 7f9ac1d9c700 1 -- 172.21.15.98:0/69090616 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc0cd640 con 0x7f9abc0cee50 2023-03-03T08:03:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.950+0000 7f9ac1d9c700 1 --2- 172.21.15.98:0/69090616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9abc0cee50 0x7f9abc0cff40 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f9abc06a6e0 tx=0x7f9ab400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f2c26bdfaa2cba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.951+0000 7f9ab3fff700 1 -- 172.21.15.98:0/69090616 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab400e070 con 0x7f9abc0cee50 2023-03-03T08:03:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.951+0000 7f9ab3fff700 1 -- 172.21.15.98:0/69090616 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab400bb20 con 0x7f9abc0cee50 2023-03-03T08:03:34.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.951+0000 7f9ab3fff700 1 -- 172.21.15.98:0/69090616 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4012430 con 0x7f9abc0cee50 2023-03-03T08:03:34.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.951+0000 7f9ac259d700 1 -- 172.21.15.98:0/69090616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9abc0cee50 msgr2=0x7f9abc0cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:34.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.951+0000 7f9ac259d700 1 --2- 172.21.15.98:0/69090616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9abc0cee50 0x7f9abc0cff40 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f9abc06a6e0 tx=0x7f9ab400b330 comp rx=0 tx=0).stop 2023-03-03T08:03:34.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.951+0000 7f9ac259d700 1 -- 172.21.15.98:0/69090616 shutdown_connections 2023-03-03T08:03:34.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.951+0000 7f9ac259d700 1 --2- 172.21.15.98:0/69090616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9abc0cee50 0x7f9abc0cff40 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:34.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.951+0000 7f9ac259d700 1 --2- 172.21.15.98:0/69090616 >> v2:172.21.15.100:3300/0 conn(0x7f9abc0b3550 0x7f9abc0cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:34.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.951+0000 7f9ac259d700 1 -- 172.21.15.98:0/69090616 >> 172.21.15.98:0/69090616 conn(0x7f9abc073b00 msgr2=0x7f9abc0d2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:34.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.952+0000 7f9ac259d700 1 -- 172.21.15.98:0/69090616 shutdown_connections 2023-03-03T08:03:34.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.952+0000 7f9ac259d700 1 -- 172.21.15.98:0/69090616 wait complete. 2023-03-03T08:03:34.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.954+0000 7f9ac259d700 1 Processor -- start 2023-03-03T08:03:34.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.954+0000 7f9ac259d700 1 -- start start 2023-03-03T08:03:34.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.955+0000 7f9ac259d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9abc073ed0 0x7f9abc158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:34.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.955+0000 7f9ac259d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9abc0b3550 0x7f9abc15a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:34.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.955+0000 7f9ac259d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9abc0cee50 0x7f9abc15b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.955+0000 7f9ac259d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc159df0 con 0x7f9abc0cee50 2023-03-03T08:03:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.955+0000 7f9ac259d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc159170 con 0x7f9abc073ed0 2023-03-03T08:03:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.955+0000 7f9ac259d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc159450 con 0x7f9abc0b3550 2023-03-03T08:03:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.955+0000 7f9ac1d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9abc0cee50 0x7f9abc15b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.955+0000 7f9ac1d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9abc0cee50 0x7f9abc15b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55992/0 (socket says 172.21.15.98:55992) 2023-03-03T08:03:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.955+0000 7f9ac1d9c700 1 -- 172.21.15.98:0/2974923509 learned_addr learned my addr 172.21.15.98:0/2974923509 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:34.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.956+0000 7f9ac159b700 1 --2- 172.21.15.98:0/2974923509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9abc073ed0 0x7f9abc158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:34.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.956+0000 7f9ac0d9a700 1 --2- 172.21.15.98:0/2974923509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9abc0b3550 0x7f9abc15a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:34.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.956+0000 7f9ac1d9c700 1 -- 172.21.15.98:0/2974923509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9abc0b3550 msgr2=0x7f9abc15a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:34.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.956+0000 7f9ac1d9c700 1 --2- 172.21.15.98:0/2974923509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9abc0b3550 0x7f9abc15a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:34.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.956+0000 7f9ac1d9c700 1 -- 172.21.15.98:0/2974923509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9abc073ed0 msgr2=0x7f9abc158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:34.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.956+0000 7f9ac1d9c700 1 --2- 172.21.15.98:0/2974923509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9abc073ed0 0x7f9abc158470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:34.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.956+0000 7f9ac1d9c700 1 -- 172.21.15.98:0/2974923509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab4009a70 con 0x7f9abc0cee50 2023-03-03T08:03:34.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.956+0000 7f9ac159b700 1 --2- 172.21.15.98:0/2974923509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9abc073ed0 0x7f9abc158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:34.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.956+0000 7f9ac1d9c700 1 --2- 172.21.15.98:0/2974923509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9abc0cee50 0x7f9abc15b680 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4014040 tx=0x7f9ab4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:34.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.956+0000 7f9ab1ffb700 1 -- 172.21.15.98:0/2974923509 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4012430 con 0x7f9abc0cee50 2023-03-03T08:03:34.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.956+0000 7f9ab1ffb700 1 -- 172.21.15.98:0/2974923509 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab4020070 con 0x7f9abc0cee50 2023-03-03T08:03:34.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.957+0000 7f9ab1ffb700 1 -- 172.21.15.98:0/2974923509 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4004280 con 0x7f9abc0cee50 2023-03-03T08:03:34.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.957+0000 7f9ac259d700 1 -- 172.21.15.98:0/2974923509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9abc0d3d60 con 0x7f9abc0cee50 2023-03-03T08:03:34.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.957+0000 7f9ac259d700 1 -- 172.21.15.98:0/2974923509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9abc0cd920 con 0x7f9abc0cee50 2023-03-03T08:03:34.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.958+0000 7f9ac259d700 1 -- 172.21.15.98:0/2974923509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9abc0d3ef0 con 0x7f9abc0cee50 2023-03-03T08:03:34.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.959+0000 7f9ab1ffb700 1 -- 172.21.15.98:0/2974923509 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f9ab402e070 con 0x7f9abc0cee50 2023-03-03T08:03:34.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.960+0000 7f9ab1ffb700 1 --2- 172.21.15.98:0/2974923509 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9aa00683c0 0x7f9aa006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:34.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.960+0000 7f9ab1ffb700 1 -- 172.21.15.98:0/2974923509 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9ab4022070 con 0x7f9abc0cee50 2023-03-03T08:03:34.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.961+0000 7f9ac159b700 1 --2- 172.21.15.98:0/2974923509 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9aa00683c0 0x7f9aa006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:34.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.961+0000 7f9ac159b700 1 --2- 172.21.15.98:0/2974923509 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9aa00683c0 0x7f9aa006a870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009c30 tx=0x7f9ab8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:34.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:34.964+0000 7f9ab1ffb700 1 -- 172.21.15.98:0/2974923509 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9abc0d3ef0 con 0x7f9abc0cee50 2023-03-03T08:03:35.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.188+0000 7f9ac259d700 1 -- 172.21.15.98:0/2974923509 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000508309", "snap_name": "snapshot_0000000000029275", "target": ["mon-mgr", ""]}) v1 -- 0x7f9abc0cb340 con 0x7f9aa00683c0 2023-03-03T08:03:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.213+0000 7f9ab1ffb700 1 -- 172.21.15.98:0/2974923509 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9abc0cb340 con 0x7f9aa00683c0 2023-03-03T08:03:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.216+0000 7f9ac259d700 1 -- 172.21.15.98:0/2974923509 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9aa00683c0 msgr2=0x7f9aa006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.216+0000 7f9ac259d700 1 --2- 172.21.15.98:0/2974923509 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9aa00683c0 0x7f9aa006a870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009c30 tx=0x7f9ab8008040 comp rx=0 tx=0).stop 2023-03-03T08:03:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.216+0000 7f9ac259d700 1 -- 172.21.15.98:0/2974923509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9abc0cee50 msgr2=0x7f9abc15b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.216+0000 7f9ac259d700 1 --2- 172.21.15.98:0/2974923509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9abc0cee50 0x7f9abc15b680 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4014040 tx=0x7f9ab4004ad0 comp rx=0 tx=0).stop 2023-03-03T08:03:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.216+0000 7f9ac259d700 1 -- 172.21.15.98:0/2974923509 shutdown_connections 2023-03-03T08:03:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.216+0000 7f9ac259d700 1 --2- 172.21.15.98:0/2974923509 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9aa00683c0 0x7f9aa006a870 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.216+0000 7f9ac259d700 1 --2- 172.21.15.98:0/2974923509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9abc0cee50 0x7f9abc15b680 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:35.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.216+0000 7f9ac259d700 1 --2- 172.21.15.98:0/2974923509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9abc073ed0 0x7f9abc158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:35.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.216+0000 7f9ac259d700 1 --2- 172.21.15.98:0/2974923509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9abc0b3550 0x7f9abc15a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:35.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.216+0000 7f9ac259d700 1 -- 172.21.15.98:0/2974923509 >> 172.21.15.98:0/2974923509 conn(0x7f9abc073b00 msgr2=0x7f9abc0cfca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:35.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.216+0000 7f9ac259d700 1 -- 172.21.15.98:0/2974923509 shutdown_connections 2023-03-03T08:03:35.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.216+0000 7f9ac259d700 1 -- 172.21.15.98:0/2974923509 wait complete. 2023-03-03T08:03:35.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvolume_0000000000508309 snapshot_0000000000029275 clone_0000000001033876 2023-03-03T08:03:35.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.495+0000 7f0555f9d700 1 Processor -- start 2023-03-03T08:03:35.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.496+0000 7f0555f9d700 1 -- start start 2023-03-03T08:03:35.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.497+0000 7f0555f9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f05500cf570 0x7f055015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:35.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.497+0000 7f0555f9d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05500cd780 con 0x7f05501609b0 2023-03-03T08:03:35.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.497+0000 7f0555f9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05500cda60 con 0x7f055015db30 2023-03-03T08:03:35.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.497+0000 7f0555f9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05500cebb0 con 0x7f05500cf570 2023-03-03T08:03:35.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.497+0000 7f0554f9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f05501609b0 0x7f0550160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44662/0 (socket says 172.21.15.98:44662) 2023-03-03T08:03:35.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.497+0000 7f0554f9b700 1 -- 172.21.15.98:0/2008121388 learned_addr learned my addr 172.21.15.98:0/2008121388 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:35.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.497+0000 7f054ffff700 1 --2- 172.21.15.98:0/2008121388 >> v2:172.21.15.98:3300/0 conn(0x7f05500cf570 0x7f055015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:35.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.498+0000 7f054ffff700 1 -- 172.21.15.98:0/2008121388 >> v1:172.21.15.100:6789/0 conn(0x7f055015db30 legacy=0x7f05500b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:35.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.498+0000 7f054ffff700 1 -- 172.21.15.98:0/2008121388 >> v1:172.21.15.98:6789/0 conn(0x7f05501609b0 legacy=0x7f0550160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:35.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.498+0000 7f054ffff700 1 -- 172.21.15.98:0/2008121388 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05500cdd40 con 0x7f05500cf570 2023-03-03T08:03:35.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.498+0000 7f054f7fe700 1 -- 172.21.15.98:0/2008121388 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974018402 0 0) 0x7f05500cd780 con 0x7f05501609b0 2023-03-03T08:03:35.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.498+0000 7f054ffff700 1 --2- 172.21.15.98:0/2008121388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05500cf570 0x7f055015d5f0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f053c009840 tx=0x7f053c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f7abcd7e85a471 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:35.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.499+0000 7f054f7fe700 1 -- 172.21.15.98:0/2008121388 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c011830 con 0x7f05500cf570 2023-03-03T08:03:35.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.499+0000 7f054f7fe700 1 -- 172.21.15.98:0/2008121388 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c011990 con 0x7f05500cf570 2023-03-03T08:03:35.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.499+0000 7f054f7fe700 1 -- 172.21.15.98:0/2008121388 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c011d20 con 0x7f05500cf570 2023-03-03T08:03:35.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.499+0000 7f0555f9d700 1 -- 172.21.15.98:0/2008121388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05500cf570 msgr2=0x7f055015d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:35.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.499+0000 7f0555f9d700 1 --2- 172.21.15.98:0/2008121388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05500cf570 0x7f055015d5f0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f053c009840 tx=0x7f053c009b50 comp rx=0 tx=0).stop 2023-03-03T08:03:35.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.499+0000 7f0555f9d700 1 -- 172.21.15.98:0/2008121388 shutdown_connections 2023-03-03T08:03:35.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.499+0000 7f0555f9d700 1 --2- 172.21.15.98:0/2008121388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05500cf570 0x7f055015d5f0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:35.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.499+0000 7f0555f9d700 1 -- 172.21.15.98:0/2008121388 >> 172.21.15.98:0/2008121388 conn(0x7f055006b880 msgr2=0x7f05500cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:35.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.499+0000 7f0555f9d700 1 -- 172.21.15.98:0/2008121388 shutdown_connections 2023-03-03T08:03:35.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.501+0000 7f0555f9d700 1 -- 172.21.15.98:0/2008121388 wait complete. 2023-03-03T08:03:35.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.501+0000 7f0555f9d700 1 Processor -- start 2023-03-03T08:03:35.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.502+0000 7f0555f9d700 1 -- start start 2023-03-03T08:03:35.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f0555f9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05500cf570 0x7f055014e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:35.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f0555f9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f055015db30 0x7f05500c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:35.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f0555f9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05501609b0 0x7f05500c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:35.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f0555f9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055014edd0 con 0x7f05501609b0 2023-03-03T08:03:35.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f0555f9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05500d1920 con 0x7f055015db30 2023-03-03T08:03:35.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f0555f9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05500c7b20 con 0x7f05500cf570 2023-03-03T08:03:35.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f055579c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05501609b0 0x7f05500c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:35.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f055579c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05501609b0 0x7f05500c8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56010/0 (socket says 172.21.15.98:56010) 2023-03-03T08:03:35.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f055579c700 1 -- 172.21.15.98:0/208821475 learned_addr learned my addr 172.21.15.98:0/208821475 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:35.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f0554f9b700 1 --2- 172.21.15.98:0/208821475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05500cf570 0x7f055014e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:35.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f054ffff700 1 --2- 172.21.15.98:0/208821475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f055015db30 0x7f05500c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:35.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f055579c700 1 -- 172.21.15.98:0/208821475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05500cf570 msgr2=0x7f055014e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:35.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f055579c700 1 --2- 172.21.15.98:0/208821475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05500cf570 0x7f055014e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f055579c700 1 -- 172.21.15.98:0/208821475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f055015db30 msgr2=0x7f05500c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f055579c700 1 --2- 172.21.15.98:0/208821475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f055015db30 0x7f05500c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.503+0000 7f055579c700 1 -- 172.21.15.98:0/208821475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c0094a0 con 0x7f05501609b0 2023-03-03T08:03:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.504+0000 7f055579c700 1 --2- 172.21.15.98:0/208821475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05501609b0 0x7f05500c8220 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f054800cb90 tx=0x7f054800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.504+0000 7f054d7fa700 1 -- 172.21.15.98:0/208821475 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0548003c00 con 0x7f05501609b0 2023-03-03T08:03:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.504+0000 7f054d7fa700 1 -- 172.21.15.98:0/208821475 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0548003d60 con 0x7f05501609b0 2023-03-03T08:03:35.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.504+0000 7f0555f9d700 1 -- 172.21.15.98:0/208821475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f055015d0d0 con 0x7f05501609b0 2023-03-03T08:03:35.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.504+0000 7f054d7fa700 1 -- 172.21.15.98:0/208821475 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054800abb0 con 0x7f05501609b0 2023-03-03T08:03:35.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.504+0000 7f0555f9d700 1 -- 172.21.15.98:0/208821475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f055015cdf0 con 0x7f05501609b0 2023-03-03T08:03:35.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.505+0000 7f0555f9d700 1 -- 172.21.15.98:0/208821475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05500b51b0 con 0x7f05501609b0 2023-03-03T08:03:35.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.507+0000 7f054d7fa700 1 -- 172.21.15.98:0/208821475 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f0548010050 con 0x7f05501609b0 2023-03-03T08:03:35.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.508+0000 7f054d7fa700 1 --2- 172.21.15.98:0/208821475 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f05340683c0 0x7f053406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:35.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.508+0000 7f054d7fa700 1 -- 172.21.15.98:0/208821475 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0548085e20 con 0x7f05501609b0 2023-03-03T08:03:35.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.508+0000 7f0554f9b700 1 --2- 172.21.15.98:0/208821475 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f05340683c0 0x7f053406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:35.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.509+0000 7f0554f9b700 1 --2- 172.21.15.98:0/208821475 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f05340683c0 0x7f053406a870 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0544000f60 tx=0x7f0544007db0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:35.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.511+0000 7f054d7fa700 1 -- 172.21.15.98:0/208821475 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f05480524c0 con 0x7f05501609b0 2023-03-03T08:03:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.744+0000 7f0555f9d700 1 -- 172.21.15.98:0/208821475 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvolume_0000000000508309", "snap_name": "snapshot_0000000000029275", "target_sub_name": "clone_0000000001033876", "target": ["mon-mgr", ""]}) v1 -- 0x7f0550068290 con 0x7f05340683c0 2023-03-03T08:03:35.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.834+0000 7f054d7fa700 1 -- 172.21.15.98:0/208821475 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0550068290 con 0x7f05340683c0 2023-03-03T08:03:35.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.837+0000 7f0555f9d700 1 -- 172.21.15.98:0/208821475 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f05340683c0 msgr2=0x7f053406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:35.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.837+0000 7f0555f9d700 1 --2- 172.21.15.98:0/208821475 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f05340683c0 0x7f053406a870 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0544000f60 tx=0x7f0544007db0 comp rx=0 tx=0).stop 2023-03-03T08:03:35.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.837+0000 7f0555f9d700 1 -- 172.21.15.98:0/208821475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05501609b0 msgr2=0x7f05500c8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:35.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.837+0000 7f0555f9d700 1 --2- 172.21.15.98:0/208821475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05501609b0 0x7f05500c8220 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f054800cb90 tx=0x7f054800cf50 comp rx=0 tx=0).stop 2023-03-03T08:03:35.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.837+0000 7f0555f9d700 1 -- 172.21.15.98:0/208821475 shutdown_connections 2023-03-03T08:03:35.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.837+0000 7f0555f9d700 1 --2- 172.21.15.98:0/208821475 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f05340683c0 0x7f053406a870 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:35.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.837+0000 7f0555f9d700 1 --2- 172.21.15.98:0/208821475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05501609b0 0x7f05500c8220 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:35.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.837+0000 7f0555f9d700 1 --2- 172.21.15.98:0/208821475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05500cf570 0x7f055014e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:35.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.837+0000 7f0555f9d700 1 --2- 172.21.15.98:0/208821475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f055015db30 0x7f05500c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:35.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.837+0000 7f0555f9d700 1 -- 172.21.15.98:0/208821475 >> 172.21.15.98:0/208821475 conn(0x7f055006b880 msgr2=0x7f055014f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:35.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.837+0000 7f0555f9d700 1 -- 172.21.15.98:0/208821475 shutdown_connections 2023-03-03T08:03:35.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:35.837+0000 7f0555f9d700 1 -- 172.21.15.98:0/208821475 wait complete. 2023-03-03T08:03:35.840 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:03:35.838+0000 7fcb843cf700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:03:35.840 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:03:35.838+0000 7fcb843cf700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:03:35.840 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:03:35.838+0000 7fcb843cf700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:03:35.840 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:03:35.838+0000 7fcb843cf700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:03:35.841 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:03:35.838+0000 7fcb843cf700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:03:35.860 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:03:35.858+0000 7fcb83bce700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:03:35.860 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:03:35.858+0000 7fcb83bce700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:03:35.860 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:03:35.858+0000 7fcb83bce700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:03:35.860 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:03:35.858+0000 7fcb83bce700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:03:35.861 INFO:tasks.ceph.mgr.x.smithi100.stderr:2023-03-03T08:03:35.858+0000 7fcb83bce700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-03T08:03:35.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs clone_0000000001033876 2023-03-03T08:03:36.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.129+0000 7f99219ff700 1 Processor -- start 2023-03-03T08:03:36.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.129+0000 7f99219ff700 1 -- start start 2023-03-03T08:03:36.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.130+0000 7f99219ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f991c0b8d10 0x7f991c0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:36.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.130+0000 7f99219ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f991c0b5310 0x7f991c0d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:36.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.130+0000 7f99219ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991c0cd7d0 con 0x7f991c0b26d0 2023-03-03T08:03:36.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.130+0000 7f99219ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f991c0b90e0 con 0x7f991c0b8d10 2023-03-03T08:03:36.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.130+0000 7f99219ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991c0cda20 con 0x7f991c0b5310 2023-03-03T08:03:36.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.130+0000 7f991affd700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f991c0b8d10 0x7f991c0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.131+0000 7f9913fff700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f991c0b26d0 0x7f991c0b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:38838/0 (socket says 172.21.15.98:38838) 2023-03-03T08:03:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.131+0000 7f9913fff700 1 -- 172.21.15.98:0/1375462863 learned_addr learned my addr 172.21.15.98:0/1375462863 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.131+0000 7f991b7fe700 1 --2- 172.21.15.98:0/1375462863 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f991c0b5310 0x7f991c0d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.131+0000 7f991affd700 1 -- 172.21.15.98:0/1375462863 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f991c0b5310 msgr2=0x7f991c0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.131+0000 7f991affd700 1 --2- 172.21.15.98:0/1375462863 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f991c0b5310 0x7f991c0d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.131+0000 7f991affd700 1 -- 172.21.15.98:0/1375462863 >> v1:172.21.15.100:6789/0 conn(0x7f991c0b26d0 legacy=0x7f991c0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.131+0000 7f991affd700 1 -- 172.21.15.98:0/1375462863 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c0cdd00 con 0x7f991c0b8d10 2023-03-03T08:03:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.131+0000 7f991affd700 1 --2- 172.21.15.98:0/1375462863 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f991c0b8d10 0x7f991c0cffc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f990c009840 tx=0x7f990c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9fcd48843da66fd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.132+0000 7f991a7fc700 1 -- 172.21.15.98:0/1375462863 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c011830 con 0x7f991c0b8d10 2023-03-03T08:03:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.132+0000 7f991a7fc700 1 -- 172.21.15.98:0/1375462863 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990c011990 con 0x7f991c0b8d10 2023-03-03T08:03:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.132+0000 7f991a7fc700 1 -- 172.21.15.98:0/1375462863 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c011d20 con 0x7f991c0b8d10 2023-03-03T08:03:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.132+0000 7f99219ff700 1 -- 172.21.15.98:0/1375462863 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f991c0b8d10 msgr2=0x7f991c0cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.132+0000 7f99219ff700 1 --2- 172.21.15.98:0/1375462863 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f991c0b8d10 0x7f991c0cffc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f990c009840 tx=0x7f990c009b50 comp rx=0 tx=0).stop 2023-03-03T08:03:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.132+0000 7f99219ff700 1 -- 172.21.15.98:0/1375462863 shutdown_connections 2023-03-03T08:03:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.132+0000 7f99219ff700 1 --2- 172.21.15.98:0/1375462863 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f991c0b8d10 0x7f991c0cffc0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.132+0000 7f99219ff700 1 --2- 172.21.15.98:0/1375462863 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f991c0b5310 0x7f991c0d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.132+0000 7f99219ff700 1 -- 172.21.15.98:0/1375462863 >> 172.21.15.98:0/1375462863 conn(0x7f991c073e70 msgr2=0x7f991c06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.132+0000 7f99219ff700 1 -- 172.21.15.98:0/1375462863 shutdown_connections 2023-03-03T08:03:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.132+0000 7f99219ff700 1 -- 172.21.15.98:0/1375462863 wait complete. 2023-03-03T08:03:36.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.135+0000 7f99219ff700 1 Processor -- start 2023-03-03T08:03:36.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.135+0000 7f99219ff700 1 -- start start 2023-03-03T08:03:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.136+0000 7f99219ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f991c0b26d0 0x7f991c151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.136+0000 7f99219ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f991c0b5310 0x7f991c1f2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.136+0000 7f99219ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f991c0b8d10 0x7f991c0cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.136+0000 7f99219ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c1f2ee0 con 0x7f991c0b5310 2023-03-03T08:03:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.136+0000 7f99219ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c1f31c0 con 0x7f991c0b26d0 2023-03-03T08:03:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.136+0000 7f99219ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991c1f34a0 con 0x7f991c0b8d10 2023-03-03T08:03:36.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.136+0000 7f991affd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f991c0b26d0 0x7f991c151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:36.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.136+0000 7f991affd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f991c0b26d0 0x7f991c151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:59116/0 (socket says 172.21.15.98:59116) 2023-03-03T08:03:36.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.136+0000 7f991affd700 1 -- 172.21.15.98:0/447459627 learned_addr learned my addr 172.21.15.98:0/447459627 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:36.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.136+0000 7f9913fff700 1 --2- 172.21.15.98:0/447459627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f991c0b5310 0x7f991c1f2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.136+0000 7f991b7fe700 1 --2- 172.21.15.98:0/447459627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f991c0b8d10 0x7f991c0cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.137+0000 7f991affd700 1 -- 172.21.15.98:0/447459627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f991c0b8d10 msgr2=0x7f991c0cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.137+0000 7f991affd700 1 --2- 172.21.15.98:0/447459627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f991c0b8d10 0x7f991c0cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.137+0000 7f991affd700 1 -- 172.21.15.98:0/447459627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f991c0b5310 msgr2=0x7f991c1f2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.137+0000 7f991affd700 1 --2- 172.21.15.98:0/447459627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f991c0b5310 0x7f991c1f2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.137+0000 7f991affd700 1 -- 172.21.15.98:0/447459627 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c0094a0 con 0x7f991c0b26d0 2023-03-03T08:03:36.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.137+0000 7f991affd700 1 --2- 172.21.15.98:0/447459627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f991c0b26d0 0x7f991c151820 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f990c009fd0 tx=0x7f990c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:36.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.138+0000 7f9918ff9700 1 -- 172.21.15.98:0/447459627 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c0036a0 con 0x7f991c0b26d0 2023-03-03T08:03:36.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.138+0000 7f9918ff9700 1 -- 172.21.15.98:0/447459627 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990c01f070 con 0x7f991c0b26d0 2023-03-03T08:03:36.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.138+0000 7f9918ff9700 1 -- 172.21.15.98:0/447459627 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c003a60 con 0x7f991c0b26d0 2023-03-03T08:03:36.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.138+0000 7f99219ff700 1 -- 172.21.15.98:0/447459627 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991c0cdfe0 con 0x7f991c0b26d0 2023-03-03T08:03:36.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.138+0000 7f99219ff700 1 -- 172.21.15.98:0/447459627 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f991c0d0e80 con 0x7f991c0b26d0 2023-03-03T08:03:36.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.139+0000 7f99219ff700 1 -- 172.21.15.98:0/447459627 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f991c0d0500 con 0x7f991c0b26d0 2023-03-03T08:03:36.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.140+0000 7f9918ff9700 1 -- 172.21.15.98:0/447459627 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 16) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f990c00f050 con 0x7f991c0b26d0 2023-03-03T08:03:36.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.141+0000 7f9918ff9700 1 --2- 172.21.15.98:0/447459627 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98fc0683c0 0x7f98fc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:36.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.141+0000 7f9918ff9700 1 -- 172.21.15.98:0/447459627 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f990c021070 con 0x7f991c0b26d0 2023-03-03T08:03:36.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.142+0000 7f9913fff700 1 --2- 172.21.15.98:0/447459627 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98fc0683c0 0x7f98fc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:36.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.142+0000 7f9913fff700 1 --2- 172.21.15.98:0/447459627 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98fc0683c0 0x7f98fc06a870 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f991c151d60 tx=0x7f9904009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:36.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.147+0000 7f9918ff9700 1 -- 172.21.15.98:0/447459627 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f990c053d90 con 0x7f991c0b26d0 2023-03-03T08:03:36.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.383+0000 7f99219ff700 1 -- 172.21.15.98:0/447459627 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "clone_0000000001033876", "target": ["mon-mgr", ""]}) v1 -- 0x7f991c1f3300 con 0x7f98fc0683c0 2023-03-03T08:03:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:36.623+0000 7f9918ff9700 1 -- 172.21.15.98:0/447459627 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mgrmap(e 17) v1 ==== 86615+0+0 (secure 0 0 0) 0x7f990c054160 con 0x7f991c0b26d0 2023-03-03T08:03:37.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.192+0000 7f9918ff9700 1 -- 172.21.15.98:0/447459627 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f991c1f3300 con 0x7f98fc0683c0 2023-03-03T08:03:37.194 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:03:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "status": { 2023-03-03T08:03:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "complete" 2023-03-03T08:03:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:37.194 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:03:37.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.195+0000 7f99219ff700 1 -- 172.21.15.98:0/447459627 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98fc0683c0 msgr2=0x7f98fc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:37.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.195+0000 7f99219ff700 1 --2- 172.21.15.98:0/447459627 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98fc0683c0 0x7f98fc06a870 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f991c151d60 tx=0x7f9904009040 comp rx=0 tx=0).stop 2023-03-03T08:03:37.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.195+0000 7f99219ff700 1 -- 172.21.15.98:0/447459627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f991c0b26d0 msgr2=0x7f991c151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:37.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.195+0000 7f99219ff700 1 --2- 172.21.15.98:0/447459627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f991c0b26d0 0x7f991c151820 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f990c009fd0 tx=0x7f990c004750 comp rx=0 tx=0).stop 2023-03-03T08:03:37.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.195+0000 7f99219ff700 1 -- 172.21.15.98:0/447459627 shutdown_connections 2023-03-03T08:03:37.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.195+0000 7f99219ff700 1 --2- 172.21.15.98:0/447459627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f991c0b8d10 0x7f991c0cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:37.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.195+0000 7f99219ff700 1 --2- 172.21.15.98:0/447459627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f991c0b26d0 0x7f991c151820 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:37.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.195+0000 7f99219ff700 1 --2- 172.21.15.98:0/447459627 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98fc0683c0 0x7f98fc06a870 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:37.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.195+0000 7f99219ff700 1 --2- 172.21.15.98:0/447459627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f991c0b5310 0x7f991c1f2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:37.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.195+0000 7f99219ff700 1 -- 172.21.15.98:0/447459627 >> 172.21.15.98:0/447459627 conn(0x7f991c073e70 msgr2=0x7f991c06ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:37.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.195+0000 7f99219ff700 1 -- 172.21.15.98:0/447459627 shutdown_connections 2023-03-03T08:03:37.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.195+0000 7f99219ff700 1 -- 172.21.15.98:0/447459627 wait complete. 2023-03-03T08:03:37.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:37.225 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/clone_0000000001033876/.meta) 2023-03-03T08:03:37.330 INFO:teuthology.orchestra.run.smithi098.stdout:[GLOBAL] 2023-03-03T08:03:37.331 INFO:teuthology.orchestra.run.smithi098.stdout:version = 2 2023-03-03T08:03:37.331 INFO:teuthology.orchestra.run.smithi098.stdout:type = clone 2023-03-03T08:03:37.331 INFO:teuthology.orchestra.run.smithi098.stdout:path = /volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c 2023-03-03T08:03:37.331 INFO:teuthology.orchestra.run.smithi098.stdout:state = complete 2023-03-03T08:03:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:03:37.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000508309 2023-03-03T08:03:37.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.588+0000 7ff20999d700 1 Processor -- start 2023-03-03T08:03:37.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.588+0000 7ff20999d700 1 -- start start 2023-03-03T08:03:37.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.589+0000 7ff20999d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2041609b0 0x7ff204160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:37.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.589+0000 7ff20999d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2040b4aa0 0x7ff2040b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:37.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.589+0000 7ff20999d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2041612c0 con 0x7ff2040cf570 2023-03-03T08:03:37.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.589+0000 7ff20999d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2040cebb0 con 0x7ff2041609b0 2023-03-03T08:03:37.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.589+0000 7ff20999d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040b5460 con 0x7ff2040b4aa0 2023-03-03T08:03:37.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.590+0000 7ff203fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff2040cf570 0x7ff20415d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44668/0 (socket says 172.21.15.98:44668) 2023-03-03T08:03:37.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.590+0000 7ff203fff700 1 -- 172.21.15.98:0/2238463671 learned_addr learned my addr 172.21.15.98:0/2238463671 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:37.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.590+0000 7ff20899b700 1 --2- 172.21.15.98:0/2238463671 >> v2:172.21.15.98:3300/0 conn(0x7ff2041609b0 0x7ff204160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:37.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.590+0000 7ff20919c700 1 --2- 172.21.15.98:0/2238463671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2040b4aa0 0x7ff2040b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:37.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.590+0000 7ff20899b700 1 -- 172.21.15.98:0/2238463671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2040b4aa0 msgr2=0x7ff2040b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:37.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.590+0000 7ff20899b700 1 --2- 172.21.15.98:0/2238463671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2040b4aa0 0x7ff2040b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:37.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.590+0000 7ff20899b700 1 -- 172.21.15.98:0/2238463671 >> v1:172.21.15.98:6789/0 conn(0x7ff2040cf570 legacy=0x7ff20415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:37.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.590+0000 7ff20899b700 1 -- 172.21.15.98:0/2238463671 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20415dd00 con 0x7ff2041609b0 2023-03-03T08:03:37.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.591+0000 7ff20899b700 1 --2- 172.21.15.98:0/2238463671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2041609b0 0x7ff204160d80 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8009840 tx=0x7ff1f8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2811c70eb0b754 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:37.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.591+0000 7ff2037fe700 1 -- 172.21.15.98:0/2238463671 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8011830 con 0x7ff2041609b0 2023-03-03T08:03:37.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.591+0000 7ff2037fe700 1 -- 172.21.15.98:0/2238463671 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f8011990 con 0x7ff2041609b0 2023-03-03T08:03:37.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.591+0000 7ff2037fe700 1 -- 172.21.15.98:0/2238463671 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8011d20 con 0x7ff2041609b0 2023-03-03T08:03:37.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.591+0000 7ff20999d700 1 -- 172.21.15.98:0/2238463671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2041609b0 msgr2=0x7ff204160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:37.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.591+0000 7ff20999d700 1 --2- 172.21.15.98:0/2238463671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2041609b0 0x7ff204160d80 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8009840 tx=0x7ff1f8009b50 comp rx=0 tx=0).stop 2023-03-03T08:03:37.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.592+0000 7ff20999d700 1 -- 172.21.15.98:0/2238463671 shutdown_connections 2023-03-03T08:03:37.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.592+0000 7ff20999d700 1 --2- 172.21.15.98:0/2238463671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2041609b0 0x7ff204160d80 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:37.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.592+0000 7ff20999d700 1 --2- 172.21.15.98:0/2238463671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2040b4aa0 0x7ff2040b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:37.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.592+0000 7ff20999d700 1 -- 172.21.15.98:0/2238463671 >> 172.21.15.98:0/2238463671 conn(0x7ff20406b880 msgr2=0x7ff2040cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:37.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.592+0000 7ff20999d700 1 -- 172.21.15.98:0/2238463671 shutdown_connections 2023-03-03T08:03:37.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.592+0000 7ff20999d700 1 -- 172.21.15.98:0/2238463671 wait complete. 2023-03-03T08:03:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.594+0000 7ff20999d700 1 Processor -- start 2023-03-03T08:03:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.594+0000 7ff20999d700 1 -- start start 2023-03-03T08:03:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.595+0000 7ff20999d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2040b4aa0 0x7ff2041f19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.595+0000 7ff20999d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2040cf570 0x7ff2040b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:37.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.595+0000 7ff20999d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2041609b0 0x7ff2040b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:37.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.595+0000 7ff20999d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2041f3300 con 0x7ff2040cf570 2023-03-03T08:03:37.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.595+0000 7ff20999d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040b8e30 con 0x7ff2041609b0 2023-03-03T08:03:37.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.595+0000 7ff20999d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040cd2b0 con 0x7ff2040b4aa0 2023-03-03T08:03:37.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.595+0000 7ff203fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2040cf570 0x7ff2040b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:37.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.595+0000 7ff203fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2040cf570 0x7ff2040b1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56022/0 (socket says 172.21.15.98:56022) 2023-03-03T08:03:37.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.595+0000 7ff203fff700 1 -- 172.21.15.98:0/2225165256 learned_addr learned my addr 172.21.15.98:0/2225165256 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:37.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.595+0000 7ff20899b700 1 --2- 172.21.15.98:0/2225165256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2040b4aa0 0x7ff2041f19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:37.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.595+0000 7ff20919c700 1 --2- 172.21.15.98:0/2225165256 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2041609b0 0x7ff2040b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:37.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.596+0000 7ff203fff700 1 -- 172.21.15.98:0/2225165256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2040b4aa0 msgr2=0x7ff2041f19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:37.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.596+0000 7ff203fff700 1 --2- 172.21.15.98:0/2225165256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2040b4aa0 0x7ff2041f19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:37.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.596+0000 7ff203fff700 1 -- 172.21.15.98:0/2225165256 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2041609b0 msgr2=0x7ff2040b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:37.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.596+0000 7ff203fff700 1 --2- 172.21.15.98:0/2225165256 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2041609b0 0x7ff2040b2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:37.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.596+0000 7ff203fff700 1 -- 172.21.15.98:0/2225165256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f80094a0 con 0x7ff2040cf570 2023-03-03T08:03:37.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.596+0000 7ff20899b700 1 --2- 172.21.15.98:0/2225165256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2040b4aa0 0x7ff2041f19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:03:37.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.596+0000 7ff203fff700 1 --2- 172.21.15.98:0/2225165256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2040cf570 0x7ff2040b1c30 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000c520 tx=0x7ff1f000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:37.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.596+0000 7ff2017fa700 1 -- 172.21.15.98:0/2225165256 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0005d70 con 0x7ff2040cf570 2023-03-03T08:03:37.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.597+0000 7ff2017fa700 1 -- 172.21.15.98:0/2225165256 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f00070d0 con 0x7ff2040cf570 2023-03-03T08:03:37.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.597+0000 7ff20999d700 1 -- 172.21.15.98:0/2225165256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2040cda60 con 0x7ff2040cf570 2023-03-03T08:03:37.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.597+0000 7ff2017fa700 1 -- 172.21.15.98:0/2225165256 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0007440 con 0x7ff2040cf570 2023-03-03T08:03:37.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.597+0000 7ff20999d700 1 -- 172.21.15.98:0/2225165256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2040cd780 con 0x7ff2040cf570 2023-03-03T08:03:37.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.598+0000 7ff20999d700 1 -- 172.21.15.98:0/2225165256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2040cdd40 con 0x7ff2040cf570 2023-03-03T08:03:37.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.599+0000 7ff2017fa700 1 -- 172.21.15.98:0/2225165256 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 86615+0+0 (secure 0 0 0) 0x7ff1f0019030 con 0x7ff2040cf570 2023-03-03T08:03:37.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.600+0000 7ff2017fa700 1 --2- 172.21.15.98:0/2225165256 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff1e8068450 0x7ff1e806a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:37.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.600+0000 7ff2017fa700 1 -- 172.21.15.98:0/2225165256 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff1f0084ff0 con 0x7ff2040cf570 2023-03-03T08:03:37.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.601+0000 7ff20899b700 1 --2- 172.21.15.98:0/2225165256 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff1e8068450 0x7ff1e806a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:37.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.602+0000 7ff20899b700 1 --2- 172.21.15.98:0/2225165256 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff1e8068450 0x7ff1e806a900 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8000c00 tx=0x7ff1f801c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:37.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.604+0000 7ff2017fa700 1 -- 172.21.15.98:0/2225165256 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff1f004b5e0 con 0x7ff2040cf570 2023-03-03T08:03:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.831+0000 7ff20999d700 1 -- 172.21.15.98:0/2225165256 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000508309", "target": ["mon-mgr", ""]}) v1 -- 0x7ff204068290 con 0x7ff1e8068450 2023-03-03T08:03:37.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.849+0000 7ff2017fa700 1 -- 172.21.15.98:0/2225165256 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7ff204068290 con 0x7ff1e8068450 2023-03-03T08:03:37.851 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4 2023-03-03T08:03:37.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.852+0000 7ff20999d700 1 -- 172.21.15.98:0/2225165256 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff1e8068450 msgr2=0x7ff1e806a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.852+0000 7ff20999d700 1 --2- 172.21.15.98:0/2225165256 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff1e8068450 0x7ff1e806a900 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8000c00 tx=0x7ff1f801c040 comp rx=0 tx=0).stop 2023-03-03T08:03:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.852+0000 7ff20999d700 1 -- 172.21.15.98:0/2225165256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2040cf570 msgr2=0x7ff2040b1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.852+0000 7ff20999d700 1 --2- 172.21.15.98:0/2225165256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2040cf570 0x7ff2040b1c30 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000c520 tx=0x7ff1f000c830 comp rx=0 tx=0).stop 2023-03-03T08:03:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.852+0000 7ff20999d700 1 -- 172.21.15.98:0/2225165256 shutdown_connections 2023-03-03T08:03:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.852+0000 7ff20999d700 1 --2- 172.21.15.98:0/2225165256 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff2041609b0 0x7ff2040b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.852+0000 7ff20999d700 1 --2- 172.21.15.98:0/2225165256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2040b4aa0 0x7ff2041f19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:37.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.852+0000 7ff20999d700 1 --2- 172.21.15.98:0/2225165256 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7ff1e8068450 0x7ff1e806a900 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:37.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.852+0000 7ff20999d700 1 --2- 172.21.15.98:0/2225165256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2040cf570 0x7ff2040b1c30 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:37.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.852+0000 7ff20999d700 1 -- 172.21.15.98:0/2225165256 >> 172.21.15.98:0/2225165256 conn(0x7ff20406b880 msgr2=0x7ff2040b4270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:37.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.852+0000 7ff20999d700 1 -- 172.21.15.98:0/2225165256 shutdown_connections 2023-03-03T08:03:37.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:37.852+0000 7ff20999d700 1 -- 172.21.15.98:0/2225165256 wait complete. 2023-03-03T08:03:37.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs clone_0000000001033876 2023-03-03T08:03:38.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.130+0000 7f58e87ff700 1 Processor -- start 2023-03-03T08:03:38.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.130+0000 7f58e87ff700 1 -- start start 2023-03-03T08:03:38.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.132+0000 7f58e87ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f58e015ca70 0x7f58e00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:38.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.132+0000 7f58e87ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e00cf980 con 0x7f58e00d0270 2023-03-03T08:03:38.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.132+0000 7f58e87ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e00d1800 con 0x7f58e006b4b0 2023-03-03T08:03:38.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.132+0000 7f58e87ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58e00cd5b0 con 0x7f58e015ca70 2023-03-03T08:03:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.132+0000 7f58e6d9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f58e00d0270 0x7f58e00d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44670/0 (socket says 172.21.15.98:44670) 2023-03-03T08:03:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.132+0000 7f58e6d9c700 1 -- 172.21.15.98:0/3274469259 learned_addr learned my addr 172.21.15.98:0/3274469259 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.132+0000 7f58e659b700 1 --2- 172.21.15.98:0/3274469259 >> v2:172.21.15.100:3300/0 conn(0x7f58e015ca70 0x7f58e00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.133+0000 7f58e5599700 1 -- 172.21.15.98:0/3274469259 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618063653 0 0) 0x7f58e00cf980 con 0x7f58e00d0270 2023-03-03T08:03:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.133+0000 7f58e5599700 1 -- 172.21.15.98:0/3274469259 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58cc0035b0 con 0x7f58e00d0270 2023-03-03T08:03:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.133+0000 7f58e5599700 1 -- 172.21.15.98:0/3274469259 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387211242 0 0) 0x7f58e00d1800 con 0x7f58e006b4b0 2023-03-03T08:03:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.133+0000 7f58e5599700 1 -- 172.21.15.98:0/3274469259 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e00cf980 con 0x7f58e006b4b0 2023-03-03T08:03:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.133+0000 7f58e659b700 1 -- 172.21.15.98:0/3274469259 >> v1:172.21.15.100:6789/0 conn(0x7f58e006b4b0 legacy=0x7f58e00cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.133+0000 7f58e659b700 1 -- 172.21.15.98:0/3274469259 >> v1:172.21.15.98:6789/0 conn(0x7f58e00d0270 legacy=0x7f58e00d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.133+0000 7f58e659b700 1 -- 172.21.15.98:0/3274469259 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e00cd900 con 0x7f58e015ca70 2023-03-03T08:03:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.133+0000 7f58e659b700 1 --2- 172.21.15.98:0/3274469259 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58e015ca70 0x7f58e00cf440 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f58dc009840 tx=0x7f58dc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74d51f89d4c07bfa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.134+0000 7f58e5599700 1 -- 172.21.15.98:0/3274469259 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc011830 con 0x7f58e015ca70 2023-03-03T08:03:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.134+0000 7f58e5599700 1 -- 172.21.15.98:0/3274469259 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58dc011990 con 0x7f58e015ca70 2023-03-03T08:03:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.134+0000 7f58e5599700 1 -- 172.21.15.98:0/3274469259 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc011d20 con 0x7f58e015ca70 2023-03-03T08:03:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.134+0000 7f58e87ff700 1 -- 172.21.15.98:0/3274469259 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58e015ca70 msgr2=0x7f58e00cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.134+0000 7f58e87ff700 1 --2- 172.21.15.98:0/3274469259 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58e015ca70 0x7f58e00cf440 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f58dc009840 tx=0x7f58dc009b50 comp rx=0 tx=0).stop 2023-03-03T08:03:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.134+0000 7f58e87ff700 1 -- 172.21.15.98:0/3274469259 shutdown_connections 2023-03-03T08:03:38.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.134+0000 7f58e87ff700 1 --2- 172.21.15.98:0/3274469259 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58e015ca70 0x7f58e00cf440 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:38.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.134+0000 7f58e87ff700 1 -- 172.21.15.98:0/3274469259 >> 172.21.15.98:0/3274469259 conn(0x7f58e0073e70 msgr2=0x7f58e006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:38.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.134+0000 7f58e87ff700 1 -- 172.21.15.98:0/3274469259 shutdown_connections 2023-03-03T08:03:38.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.135+0000 7f58e87ff700 1 -- 172.21.15.98:0/3274469259 wait complete. 2023-03-03T08:03:38.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.136+0000 7f58e87ff700 1 Processor -- start 2023-03-03T08:03:38.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.137+0000 7f58e87ff700 1 -- start start 2023-03-03T08:03:38.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e87ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58e006b4b0 0x7f58e0158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:38.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e87ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58e00d0270 0x7f58e00ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:38.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e87ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58e015ca70 0x7f58e00ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:38.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e87ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e015f9a0 con 0x7f58e006b4b0 2023-03-03T08:03:38.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e87ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e00cada0 con 0x7f58e015ca70 2023-03-03T08:03:38.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e87ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e00cb080 con 0x7f58e00d0270 2023-03-03T08:03:38.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e659b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58e006b4b0 0x7f58e0158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:38.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e659b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58e006b4b0 0x7f58e0158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56028/0 (socket says 172.21.15.98:56028) 2023-03-03T08:03:38.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e659b700 1 -- 172.21.15.98:0/852900708 learned_addr learned my addr 172.21.15.98:0/852900708 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:38.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e6d9c700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58e015ca70 0x7f58e00ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:38.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e5d9a700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58e00d0270 0x7f58e00ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:38.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e659b700 1 -- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58e00d0270 msgr2=0x7f58e00ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:38.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e659b700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58e00d0270 0x7f58e00ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:38.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e659b700 1 -- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58e015ca70 msgr2=0x7f58e00ca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:38.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e659b700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58e015ca70 0x7f58e00ca790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:38.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.138+0000 7f58e659b700 1 -- 172.21.15.98:0/852900708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc0094a0 con 0x7f58e006b4b0 2023-03-03T08:03:38.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.139+0000 7f58e6d9c700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58e015ca70 0x7f58e00ca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:03:38.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.139+0000 7f58e5d9a700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58e00d0270 0x7f58e00ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:03:38.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.139+0000 7f58e659b700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58e006b4b0 0x7f58e0158750 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f58dc0090b0 tx=0x7f58dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:38.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.139+0000 7f58d6ffd700 1 -- 172.21.15.98:0/852900708 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc01a860 con 0x7f58e006b4b0 2023-03-03T08:03:38.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.139+0000 7f58d6ffd700 1 -- 172.21.15.98:0/852900708 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58dc01a9c0 con 0x7f58e006b4b0 2023-03-03T08:03:38.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.139+0000 7f58e87ff700 1 -- 172.21.15.98:0/852900708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e00d1aa0 con 0x7f58e006b4b0 2023-03-03T08:03:38.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.139+0000 7f58d6ffd700 1 -- 172.21.15.98:0/852900708 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc01ad50 con 0x7f58e006b4b0 2023-03-03T08:03:38.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.139+0000 7f58e87ff700 1 -- 172.21.15.98:0/852900708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e00cdbe0 con 0x7f58e006b4b0 2023-03-03T08:03:38.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.140+0000 7f58e87ff700 1 -- 172.21.15.98:0/852900708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e015ce40 con 0x7f58e006b4b0 2023-03-03T08:03:38.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.142+0000 7f58d6ffd700 1 -- 172.21.15.98:0/852900708 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 86615+0+0 (secure 0 0 0) 0x7f58dc00f050 con 0x7f58e006b4b0 2023-03-03T08:03:38.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.143+0000 7f58d6ffd700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f58c4068450 0x7f58c406a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.143+0000 7f58d6ffd700 1 -- 172.21.15.98:0/852900708 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f58dc086d70 con 0x7f58e006b4b0 2023-03-03T08:03:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.143+0000 7f58e5d9a700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f58c4068450 0x7f58c406a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.144+0000 7f58e5d9a700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f58c4068450 0x7f58c406a900 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f58e01e8a00 tx=0x7f58d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:38.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.147+0000 7f58d6ffd700 1 -- 172.21.15.98:0/852900708 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f58dc053460 con 0x7f58e006b4b0 2023-03-03T08:03:38.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.369+0000 7f58e87ff700 1 -- 172.21.15.98:0/852900708 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "clone_0000000001033876", "target": ["mon-mgr", ""]}) v1 -- 0x7f58e004e6e0 con 0x7f58c4068450 2023-03-03T08:03:38.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.376+0000 7f58d6ffd700 1 -- 172.21.15.98:0/852900708 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+78 (secure 0 0 0) 0x7f58e004e6e0 con 0x7f58c4068450 2023-03-03T08:03:38.378 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c 2023-03-03T08:03:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.379+0000 7f58e87ff700 1 -- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f58c4068450 msgr2=0x7f58c406a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.379+0000 7f58e87ff700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f58c4068450 0x7f58c406a900 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f58e01e8a00 tx=0x7f58d0009040 comp rx=0 tx=0).stop 2023-03-03T08:03:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.379+0000 7f58e87ff700 1 -- 172.21.15.98:0/852900708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58e006b4b0 msgr2=0x7f58e0158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.379+0000 7f58e87ff700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58e006b4b0 0x7f58e0158750 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f58dc0090b0 tx=0x7f58dc004750 comp rx=0 tx=0).stop 2023-03-03T08:03:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.379+0000 7f58e87ff700 1 -- 172.21.15.98:0/852900708 shutdown_connections 2023-03-03T08:03:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.379+0000 7f58e87ff700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f58c4068450 0x7f58c406a900 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.379+0000 7f58e87ff700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58e006b4b0 0x7f58e0158750 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.379+0000 7f58e87ff700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58e015ca70 0x7f58e00ca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.379+0000 7f58e87ff700 1 --2- 172.21.15.98:0/852900708 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58e00d0270 0x7f58e00ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.379+0000 7f58e87ff700 1 -- 172.21.15.98:0/852900708 >> 172.21.15.98:0/852900708 conn(0x7f58e0073e70 msgr2=0x7f58e00d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:38.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.379+0000 7f58e87ff700 1 -- 172.21.15.98:0/852900708 shutdown_connections 2023-03-03T08:03:38.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.379+0000 7f58e87ff700 1 -- 172.21.15.98:0/852900708 wait complete. 2023-03-03T08:03:38.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:38.408 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275) 2023-03-03T08:03:38.492 INFO:teuthology.orchestra.run.smithi098.stdout:9 2023-03-03T08:03:38.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:38.493 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c) 2023-03-03T08:03:38.573 INFO:teuthology.orchestra.run.smithi098.stdout:9 2023-03-03T08:03:38.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs clone_0000000001033876 2023-03-03T08:03:38.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.823+0000 7facba79d700 1 Processor -- start 2023-03-03T08:03:38.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.824+0000 7facba79d700 1 -- start start 2023-03-03T08:03:38.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.825+0000 7facba79d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7facb40cf570 0x7facb415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:38.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.825+0000 7facba79d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb415db30 0x7facb40b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:38.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.825+0000 7facba79d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb40cd780 con 0x7facb41609b0 2023-03-03T08:03:38.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.825+0000 7facba79d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb40cebb0 con 0x7facb40cf570 2023-03-03T08:03:38.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.825+0000 7facba79d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb40cda60 con 0x7facb415db30 2023-03-03T08:03:38.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.826+0000 7facb9f9c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb415db30 0x7facb40b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:38.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.826+0000 7facb9f9c700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb415db30 0x7facb40b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:58864/0 (socket says 172.21.15.98:58864) 2023-03-03T08:03:38.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.826+0000 7facb9f9c700 1 -- 172.21.15.98:0/1168607777 learned_addr learned my addr 172.21.15.98:0/1168607777 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:38.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.826+0000 7facb8f9a700 1 --2- 172.21.15.98:0/1168607777 >> v2:172.21.15.100:3300/0 conn(0x7facb40cf570 0x7facb415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:38.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.826+0000 7facabfff700 1 -- 172.21.15.98:0/1168607777 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181391202 0 0) 0x7facb40cd780 con 0x7facb41609b0 2023-03-03T08:03:38.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.826+0000 7facabfff700 1 -- 172.21.15.98:0/1168607777 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faca00035b0 con 0x7facb41609b0 2023-03-03T08:03:38.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.826+0000 7facb9f9c700 1 -- 172.21.15.98:0/1168607777 >> v2:172.21.15.100:3300/0 conn(0x7facb40cf570 msgr2=0x7facb415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:38.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.826+0000 7facb9f9c700 1 --2- 172.21.15.98:0/1168607777 >> v2:172.21.15.100:3300/0 conn(0x7facb40cf570 0x7facb415d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:38.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.826+0000 7facb9f9c700 1 -- 172.21.15.98:0/1168607777 >> v1:172.21.15.100:6789/0 conn(0x7facb41609b0 legacy=0x7facb4160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:38.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.826+0000 7facb9f9c700 1 -- 172.21.15.98:0/1168607777 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb40cdd40 con 0x7facb415db30 2023-03-03T08:03:38.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.827+0000 7facb9f9c700 1 --2- 172.21.15.98:0/1168607777 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb415db30 0x7facb40b4aa0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7facac009e50 tx=0x7facac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f87de660416da41 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:38.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.827+0000 7facabfff700 1 -- 172.21.15.98:0/1168607777 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac00e070 con 0x7facb415db30 2023-03-03T08:03:38.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.827+0000 7facabfff700 1 -- 172.21.15.98:0/1168607777 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facac00b910 con 0x7facb415db30 2023-03-03T08:03:38.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.827+0000 7facabfff700 1 -- 172.21.15.98:0/1168607777 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac00bcc0 con 0x7facb415db30 2023-03-03T08:03:38.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.828+0000 7facba79d700 1 -- 172.21.15.98:0/1168607777 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb415db30 msgr2=0x7facb40b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:38.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.828+0000 7facba79d700 1 --2- 172.21.15.98:0/1168607777 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb415db30 0x7facb40b4aa0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7facac009e50 tx=0x7facac00b330 comp rx=0 tx=0).stop 2023-03-03T08:03:38.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.828+0000 7facba79d700 1 -- 172.21.15.98:0/1168607777 shutdown_connections 2023-03-03T08:03:38.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.828+0000 7facba79d700 1 --2- 172.21.15.98:0/1168607777 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb415db30 0x7facb40b4aa0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:38.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.828+0000 7facba79d700 1 --2- 172.21.15.98:0/1168607777 >> v2:172.21.15.100:3300/0 conn(0x7facb40cf570 0x7facb415d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:38.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.828+0000 7facba79d700 1 -- 172.21.15.98:0/1168607777 >> 172.21.15.98:0/1168607777 conn(0x7facb406b880 msgr2=0x7facb40cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:38.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.828+0000 7facba79d700 1 -- 172.21.15.98:0/1168607777 shutdown_connections 2023-03-03T08:03:38.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.828+0000 7facba79d700 1 -- 172.21.15.98:0/1168607777 wait complete. 2023-03-03T08:03:38.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.830+0000 7facba79d700 1 Processor -- start 2023-03-03T08:03:38.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.830+0000 7facba79d700 1 -- start start 2023-03-03T08:03:38.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.831+0000 7facba79d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb40cf570 0x7facb40b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:38.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.831+0000 7facba79d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facb415db30 0x7facb40c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:38.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.831+0000 7facba79d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb41609b0 0x7facb40c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:38.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.831+0000 7facba79d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb40c6740 con 0x7facb40cf570 2023-03-03T08:03:38.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.831+0000 7facba79d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb40c8950 con 0x7facb415db30 2023-03-03T08:03:38.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.831+0000 7facba79d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb40c6920 con 0x7facb41609b0 2023-03-03T08:03:38.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.831+0000 7facb979b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb40cf570 0x7facb40b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:38.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.831+0000 7facb979b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb40cf570 0x7facb40b37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56042/0 (socket says 172.21.15.98:56042) 2023-03-03T08:03:38.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.831+0000 7facb979b700 1 -- 172.21.15.98:0/223555669 learned_addr learned my addr 172.21.15.98:0/223555669 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:38.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.832+0000 7facb9f9c700 1 --2- 172.21.15.98:0/223555669 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb41609b0 0x7facb40c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:38.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.832+0000 7facb8f9a700 1 --2- 172.21.15.98:0/223555669 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facb415db30 0x7facb40c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:38.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.832+0000 7facb979b700 1 -- 172.21.15.98:0/223555669 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb41609b0 msgr2=0x7facb40c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:38.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.832+0000 7facb979b700 1 --2- 172.21.15.98:0/223555669 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb41609b0 0x7facb40c8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:38.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.832+0000 7facb979b700 1 -- 172.21.15.98:0/223555669 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facb415db30 msgr2=0x7facb40c7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:38.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.832+0000 7facb979b700 1 --2- 172.21.15.98:0/223555669 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facb415db30 0x7facb40c7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:38.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.832+0000 7facb979b700 1 -- 172.21.15.98:0/223555669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac009b30 con 0x7facb40cf570 2023-03-03T08:03:38.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.832+0000 7facb979b700 1 --2- 172.21.15.98:0/223555669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb40cf570 0x7facb40b37c0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7facb000c500 tx=0x7facb000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:38.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.832+0000 7faca9ffb700 1 -- 172.21.15.98:0/223555669 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00052d0 con 0x7facb40cf570 2023-03-03T08:03:38.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.832+0000 7faca9ffb700 1 -- 172.21.15.98:0/223555669 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb0005450 con 0x7facb40cf570 2023-03-03T08:03:38.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.833+0000 7facba79d700 1 -- 172.21.15.98:0/223555669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb415d0d0 con 0x7facb40cf570 2023-03-03T08:03:38.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.833+0000 7faca9ffb700 1 -- 172.21.15.98:0/223555669 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00038c0 con 0x7facb40cf570 2023-03-03T08:03:38.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.833+0000 7facba79d700 1 -- 172.21.15.98:0/223555669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb415cdf0 con 0x7facb40cf570 2023-03-03T08:03:38.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.834+0000 7facba79d700 1 -- 172.21.15.98:0/223555669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb40b4fe0 con 0x7facb40cf570 2023-03-03T08:03:38.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.835+0000 7faca9ffb700 1 -- 172.21.15.98:0/223555669 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 86615+0+0 (secure 0 0 0) 0x7facb0019030 con 0x7facb40cf570 2023-03-03T08:03:38.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.837+0000 7faca9ffb700 1 --2- 172.21.15.98:0/223555669 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fac98068450 0x7fac9806a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:38.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.837+0000 7faca9ffb700 1 -- 172.21.15.98:0/223555669 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7facb0085110 con 0x7facb40cf570 2023-03-03T08:03:38.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.838+0000 7facb8f9a700 1 --2- 172.21.15.98:0/223555669 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fac98068450 0x7fac9806a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:38.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.838+0000 7facb8f9a700 1 --2- 172.21.15.98:0/223555669 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fac98068450 0x7fac9806a900 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7facb40b8a80 tx=0x7faca4009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:38.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:38.841+0000 7faca9ffb700 1 -- 172.21.15.98:0/223555669 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7facb0051800 con 0x7facb40cf570 2023-03-03T08:03:39.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:39.066+0000 7facba79d700 1 -- 172.21.15.98:0/223555669 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "clone_0000000001033876", "target": ["mon-mgr", ""]}) v1 -- 0x7facb4068290 con 0x7fac98068450 2023-03-03T08:03:39.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:39.077+0000 7faca9ffb700 1 -- 172.21.15.98:0/223555669 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+712 (secure 0 0 0) 0x7facb4068290 con 0x7fac98068450 2023-03-03T08:03:39.078 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:03:39.079 INFO:teuthology.orchestra.run.smithi098.stdout: "atime": "2023-03-03 08:03:28", 2023-03-03T08:03:39.079 INFO:teuthology.orchestra.run.smithi098.stdout: "bytes_pcent": "80.00", 2023-03-03T08:03:39.079 INFO:teuthology.orchestra.run.smithi098.stdout: "bytes_quota": 10485760, 2023-03-03T08:03:39.079 INFO:teuthology.orchestra.run.smithi098.stdout: "bytes_used": 8388608, 2023-03-03T08:03:39.079 INFO:teuthology.orchestra.run.smithi098.stdout: "created_at": "2023-03-03 08:03:35", 2023-03-03T08:03:39.079 INFO:teuthology.orchestra.run.smithi098.stdout: "ctime": "2023-03-03 08:03:36", 2023-03-03T08:03:39.079 INFO:teuthology.orchestra.run.smithi098.stdout: "data_pool": "cephfs_data", 2023-03-03T08:03:39.080 INFO:teuthology.orchestra.run.smithi098.stdout: "features": [ 2023-03-03T08:03:39.080 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot-clone", 2023-03-03T08:03:39.080 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot-autoprotect", 2023-03-03T08:03:39.080 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot-retention" 2023-03-03T08:03:39.080 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:03:39.080 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 1000, 2023-03-03T08:03:39.080 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": 16895, 2023-03-03T08:03:39.080 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_addrs": [ 2023-03-03T08:03:39.081 INFO:teuthology.orchestra.run.smithi098.stdout: "172.21.15.98:6789", 2023-03-03T08:03:39.081 INFO:teuthology.orchestra.run.smithi098.stdout: "172.21.15.100:6789", 2023-03-03T08:03:39.081 INFO:teuthology.orchestra.run.smithi098.stdout: "172.21.15.100:6790" 2023-03-03T08:03:39.081 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-03-03T08:03:39.081 INFO:teuthology.orchestra.run.smithi098.stdout: "mtime": "2023-03-03 08:03:34", 2023-03-03T08:03:39.081 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c", 2023-03-03T08:03:39.081 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_namespace": "", 2023-03-03T08:03:39.082 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "complete", 2023-03-03T08:03:39.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "clone", 2023-03-03T08:03:39.082 INFO:teuthology.orchestra.run.smithi098.stdout: "uid": 1000 2023-03-03T08:03:39.082 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:03:39.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:39.079+0000 7facba79d700 1 -- 172.21.15.98:0/223555669 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fac98068450 msgr2=0x7fac9806a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:39.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:39.079+0000 7facba79d700 1 --2- 172.21.15.98:0/223555669 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fac98068450 0x7fac9806a900 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7facb40b8a80 tx=0x7faca4009210 comp rx=0 tx=0).stop 2023-03-03T08:03:39.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:39.079+0000 7facba79d700 1 -- 172.21.15.98:0/223555669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb40cf570 msgr2=0x7facb40b37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:39.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:39.079+0000 7facba79d700 1 --2- 172.21.15.98:0/223555669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb40cf570 0x7facb40b37c0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7facb000c500 tx=0x7facb000c810 comp rx=0 tx=0).stop 2023-03-03T08:03:39.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:39.079+0000 7facba79d700 1 -- 172.21.15.98:0/223555669 shutdown_connections 2023-03-03T08:03:39.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:39.079+0000 7facba79d700 1 --2- 172.21.15.98:0/223555669 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fac98068450 0x7fac9806a900 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:39.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:39.079+0000 7facba79d700 1 --2- 172.21.15.98:0/223555669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb40cf570 0x7facb40b37c0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:39.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:39.079+0000 7facba79d700 1 --2- 172.21.15.98:0/223555669 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facb41609b0 0x7facb40c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:39.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:39.079+0000 7facba79d700 1 --2- 172.21.15.98:0/223555669 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facb415db30 0x7facb40c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:39.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:39.079+0000 7facba79d700 1 -- 172.21.15.98:0/223555669 >> 172.21.15.98:0/223555669 conn(0x7facb406b880 msgr2=0x7facb4152010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:39.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:39.079+0000 7facba79d700 1 -- 172.21.15.98:0/223555669 shutdown_connections 2023-03-03T08:03:39.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:39.079+0000 7facba79d700 1 -- 172.21.15.98:0/223555669 wait complete. 2023-03-03T08:03:39.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:39.108 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275) 2023-03-03T08:03:39.191 INFO:teuthology.orchestra.run.smithi098.stdout:10485760 2023-03-03T08:03:39.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:39.192 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275) 2023-03-03T08:03:39.273 INFO:teuthology.orchestra.run.smithi098.stdout:cephfs_data 2023-03-03T08:03:39.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:39.274 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool_namespace volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275) 2023-03-03T08:03:39.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:39.358 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec find volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275) 2023-03-03T08:03:39.443 INFO:teuthology.orchestra.run.smithi098.stdout:volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275 2023-03-03T08:03:39.444 INFO:teuthology.orchestra.run.smithi098.stdout:volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.2 2023-03-03T08:03:39.444 INFO:teuthology.orchestra.run.smithi098.stdout:volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.5 2023-03-03T08:03:39.444 INFO:teuthology.orchestra.run.smithi098.stdout:volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.4 2023-03-03T08:03:39.444 INFO:teuthology.orchestra.run.smithi098.stdout:volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.7 2023-03-03T08:03:39.444 INFO:teuthology.orchestra.run.smithi098.stdout:volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.6 2023-03-03T08:03:39.445 INFO:teuthology.orchestra.run.smithi098.stdout:volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.0 2023-03-03T08:03:39.445 INFO:teuthology.orchestra.run.smithi098.stdout:volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.3 2023-03-03T08:03:39.445 INFO:teuthology.orchestra.run.smithi098.stdout:volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.1 2023-03-03T08:03:39.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:39.446 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275) 2023-03-03T08:03:39.518 INFO:teuthology.orchestra.run.smithi098.stdout:41ff 2023-03-03T08:03:39.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:39.520 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/) 2023-03-03T08:03:39.590 INFO:teuthology.orchestra.run.smithi098.stdout:41ff 2023-03-03T08:03:39.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:39.593 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275) 2023-03-03T08:03:39.667 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:39.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:39.669 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/) 2023-03-03T08:03:39.741 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:39.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:39.742 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275) 2023-03-03T08:03:39.815 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:39.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:39.817 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/) 2023-03-03T08:03:39.887 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:39.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:39.889 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275) 2023-03-03T08:03:39.966 INFO:teuthology.orchestra.run.smithi098.stdout:1677830614 2023-03-03T08:03:39.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:39.967 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/) 2023-03-03T08:03:40.037 INFO:teuthology.orchestra.run.smithi098.stdout:1677830614 2023-03-03T08:03:40.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:40.040 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.2) 2023-03-03T08:03:40.114 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:40.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:40.116 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.2) 2023-03-03T08:03:40.197 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:40.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:40.199 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.2) 2023-03-03T08:03:40.272 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:40.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:40.274 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.2) 2023-03-03T08:03:40.344 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:40.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:40.347 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.2) 2023-03-03T08:03:40.419 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:40.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:40.422 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.2) 2023-03-03T08:03:40.493 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:40.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:40.495 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.2) 2023-03-03T08:03:40.568 INFO:teuthology.orchestra.run.smithi098.stdout:1677830613 2023-03-03T08:03:40.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:40.570 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.2) 2023-03-03T08:03:40.641 INFO:teuthology.orchestra.run.smithi098.stdout:1677830613 2023-03-03T08:03:40.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:40.643 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.5) 2023-03-03T08:03:40.717 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:40.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:40.719 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.5) 2023-03-03T08:03:40.799 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:40.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:40.801 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.5) 2023-03-03T08:03:40.874 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:40.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:40.876 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.5) 2023-03-03T08:03:40.947 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:40.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:40.949 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.5) 2023-03-03T08:03:41.022 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:41.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:41.024 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.5) 2023-03-03T08:03:41.094 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:41.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:41.096 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.5) 2023-03-03T08:03:41.169 INFO:teuthology.orchestra.run.smithi098.stdout:1677830614 2023-03-03T08:03:41.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:41.172 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.5) 2023-03-03T08:03:41.241 INFO:teuthology.orchestra.run.smithi098.stdout:1677830614 2023-03-03T08:03:41.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:41.243 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.4) 2023-03-03T08:03:41.315 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:41.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:41.318 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.4) 2023-03-03T08:03:41.397 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:41.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:41.400 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.4) 2023-03-03T08:03:41.473 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:41.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:41.527 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.4) 2023-03-03T08:03:41.597 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:41.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:41.599 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.4) 2023-03-03T08:03:41.672 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:41.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:41.676 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.4) 2023-03-03T08:03:41.747 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:41.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:41.749 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.4) 2023-03-03T08:03:41.822 INFO:teuthology.orchestra.run.smithi098.stdout:1677830614 2023-03-03T08:03:41.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:41.825 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.4) 2023-03-03T08:03:41.895 INFO:teuthology.orchestra.run.smithi098.stdout:1677830614 2023-03-03T08:03:41.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:41.897 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.7) 2023-03-03T08:03:41.969 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:41.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:41.972 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.7) 2023-03-03T08:03:42.047 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:42.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:42.049 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.7) 2023-03-03T08:03:42.121 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:42.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:42.123 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.7) 2023-03-03T08:03:42.194 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:42.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:42.196 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.7) 2023-03-03T08:03:42.268 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:42.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:42.270 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.7) 2023-03-03T08:03:42.341 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:42.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:42.344 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.7) 2023-03-03T08:03:42.416 INFO:teuthology.orchestra.run.smithi098.stdout:1677830614 2023-03-03T08:03:42.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:42.419 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.7) 2023-03-03T08:03:42.489 INFO:teuthology.orchestra.run.smithi098.stdout:1677830614 2023-03-03T08:03:42.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:42.491 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.6) 2023-03-03T08:03:42.563 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:42.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:42.566 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.6) 2023-03-03T08:03:42.637 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:42.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:42.640 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.6) 2023-03-03T08:03:42.712 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:42.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:42.714 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.6) 2023-03-03T08:03:42.785 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:42.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:42.788 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.6) 2023-03-03T08:03:42.861 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:42.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:42.863 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.6) 2023-03-03T08:03:42.933 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:42.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:42.935 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.6) 2023-03-03T08:03:43.008 INFO:teuthology.orchestra.run.smithi098.stdout:1677830614 2023-03-03T08:03:43.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:43.010 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.6) 2023-03-03T08:03:43.079 INFO:teuthology.orchestra.run.smithi098.stdout:1677830614 2023-03-03T08:03:43.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:43.081 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.0) 2023-03-03T08:03:43.153 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:43.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:43.155 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.0) 2023-03-03T08:03:43.226 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:43.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:43.228 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.0) 2023-03-03T08:03:43.300 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:43.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:43.302 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.0) 2023-03-03T08:03:43.374 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:43.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:43.376 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.0) 2023-03-03T08:03:43.448 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:43.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:43.450 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.0) 2023-03-03T08:03:43.520 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:43.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:43.523 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.0) 2023-03-03T08:03:43.595 INFO:teuthology.orchestra.run.smithi098.stdout:1677830613 2023-03-03T08:03:43.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:43.597 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.0) 2023-03-03T08:03:43.668 INFO:teuthology.orchestra.run.smithi098.stdout:1677830613 2023-03-03T08:03:43.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:43.670 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.3) 2023-03-03T08:03:43.742 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:43.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:43.744 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.3) 2023-03-03T08:03:43.815 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:43.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:43.817 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.3) 2023-03-03T08:03:43.890 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:43.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:43.892 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.3) 2023-03-03T08:03:43.962 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:43.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:43.964 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.3) 2023-03-03T08:03:44.037 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:44.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:44.039 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.3) 2023-03-03T08:03:44.109 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:44.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:44.111 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.3) 2023-03-03T08:03:44.184 INFO:teuthology.orchestra.run.smithi098.stdout:1677830613 2023-03-03T08:03:44.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:44.186 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.3) 2023-03-03T08:03:44.256 INFO:teuthology.orchestra.run.smithi098.stdout:1677830613 2023-03-03T08:03:44.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:44.258 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.1) 2023-03-03T08:03:44.330 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:44.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:44.332 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.1) 2023-03-03T08:03:44.403 INFO:teuthology.orchestra.run.smithi098.stdout:81b4 2023-03-03T08:03:44.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:44.405 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.1) 2023-03-03T08:03:44.483 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:44.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:44.485 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.1) 2023-03-03T08:03:44.555 INFO:teuthology.orchestra.run.smithi098.stdout:1000 2023-03-03T08:03:44.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:44.558 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.1) 2023-03-03T08:03:44.629 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:44.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:44.631 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.1) 2023-03-03T08:03:44.702 INFO:teuthology.orchestra.run.smithi098.stdout:1268 2023-03-03T08:03:44.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:44.704 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvolume_0000000000508309/21bf31ac-367f-43c6-9c19-c70479123ed4/.snap/snapshot_0000000000029275/subvolume_file.1) 2023-03-03T08:03:44.777 INFO:teuthology.orchestra.run.smithi098.stdout:1677830613 2023-03-03T08:03:44.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:44.779 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/clone_0000000001033876/b56138d0-2975-43b4-9395-2ba93f46141c/subvolume_file.1) 2023-03-03T08:03:44.849 INFO:teuthology.orchestra.run.smithi098.stdout:1677830613 2023-03-03T08:03:44.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvolume_0000000000508309 fake clone_0000000001033877 2023-03-03T08:03:45.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.101+0000 7fc6a79ff700 1 Processor -- start 2023-03-03T08:03:45.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.101+0000 7fc6a79ff700 1 -- start start 2023-03-03T08:03:45.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.103+0000 7fc6a79ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6a00d0270 0x7fc6a00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:45.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.103+0000 7fc6a79ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc6a006b4b0 0x7fc6a00d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:45.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.103+0000 7fc6a79ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a00cf980 con 0x7fc6a015ca70 2023-03-03T08:03:45.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.103+0000 7fc6a79ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6a00cd5b0 con 0x7fc6a006b4b0 2023-03-03T08:03:45.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.103+0000 7fc6a79ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a00d1800 con 0x7fc6a00d0270 2023-03-03T08:03:45.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.103+0000 7fc6a579b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc6a015ca70 0x7fc6a00cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51814/0 (socket says 172.21.15.98:51814) 2023-03-03T08:03:45.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.103+0000 7fc6a579b700 1 -- 172.21.15.98:0/4127170043 learned_addr learned my addr 172.21.15.98:0/4127170043 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:45.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.103+0000 7fc6a5f9c700 1 --2- 172.21.15.98:0/4127170043 >> v2:172.21.15.98:3300/0 conn(0x7fc6a006b4b0 0x7fc6a00d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:45.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.104+0000 7fc6a4f9a700 1 --2- 172.21.15.98:0/4127170043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6a00d0270 0x7fc6a00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:45.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.104+0000 7fc697fff700 1 -- 172.21.15.98:0/4127170043 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724392551 0 0) 0x7fc6a00cf980 con 0x7fc6a015ca70 2023-03-03T08:03:45.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.104+0000 7fc697fff700 1 -- 172.21.15.98:0/4127170043 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc68c0035b0 con 0x7fc6a015ca70 2023-03-03T08:03:45.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.104+0000 7fc6a5f9c700 1 -- 172.21.15.98:0/4127170043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6a00d0270 msgr2=0x7fc6a00cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:45.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.104+0000 7fc6a5f9c700 1 --2- 172.21.15.98:0/4127170043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6a00d0270 0x7fc6a00cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:45.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.104+0000 7fc6a5f9c700 1 -- 172.21.15.98:0/4127170043 >> v1:172.21.15.98:6789/0 conn(0x7fc6a015ca70 legacy=0x7fc6a00cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:45.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.104+0000 7fc6a5f9c700 1 -- 172.21.15.98:0/4127170043 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a00cd900 con 0x7fc6a006b4b0 2023-03-03T08:03:45.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.104+0000 7fc6a5f9c700 1 --2- 172.21.15.98:0/4127170043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6a006b4b0 0x7fc6a00d1fc0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fc698009d90 tx=0x7fc69800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02253a9babf517a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:45.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.105+0000 7fc697fff700 1 -- 172.21.15.98:0/4127170043 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69800e070 con 0x7fc6a006b4b0 2023-03-03T08:03:45.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.105+0000 7fc697fff700 1 -- 172.21.15.98:0/4127170043 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69800bb20 con 0x7fc6a006b4b0 2023-03-03T08:03:45.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.105+0000 7fc697fff700 1 -- 172.21.15.98:0/4127170043 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69800be90 con 0x7fc6a006b4b0 2023-03-03T08:03:45.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.105+0000 7fc6a79ff700 1 -- 172.21.15.98:0/4127170043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6a006b4b0 msgr2=0x7fc6a00d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:45.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.105+0000 7fc6a79ff700 1 --2- 172.21.15.98:0/4127170043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6a006b4b0 0x7fc6a00d1fc0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fc698009d90 tx=0x7fc69800b330 comp rx=0 tx=0).stop 2023-03-03T08:03:45.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.105+0000 7fc6a79ff700 1 -- 172.21.15.98:0/4127170043 shutdown_connections 2023-03-03T08:03:45.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.105+0000 7fc6a79ff700 1 --2- 172.21.15.98:0/4127170043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6a006b4b0 0x7fc6a00d1fc0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:45.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.105+0000 7fc6a79ff700 1 --2- 172.21.15.98:0/4127170043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6a00d0270 0x7fc6a00cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:45.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.105+0000 7fc6a79ff700 1 -- 172.21.15.98:0/4127170043 >> 172.21.15.98:0/4127170043 conn(0x7fc6a0073e70 msgr2=0x7fc6a006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:45.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.105+0000 7fc6a79ff700 1 -- 172.21.15.98:0/4127170043 shutdown_connections 2023-03-03T08:03:45.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.105+0000 7fc6a79ff700 1 -- 172.21.15.98:0/4127170043 wait complete. 2023-03-03T08:03:45.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.107+0000 7fc6a79ff700 1 Processor -- start 2023-03-03T08:03:45.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.108+0000 7fc6a79ff700 1 -- start start 2023-03-03T08:03:45.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.109+0000 7fc6a79ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6a006b4b0 0x7fc6a0149760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:45.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.109+0000 7fc6a79ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6a00d0270 0x7fc6a00b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:45.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.109+0000 7fc6a79ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6a015ca70 0x7fc6a00b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:45.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.109+0000 7fc6a79ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a0149ca0 con 0x7fc6a00d0270 2023-03-03T08:03:45.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.109+0000 7fc6a79ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a01f23c0 con 0x7fc6a015ca70 2023-03-03T08:03:45.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.109+0000 7fc6a79ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a01f26a0 con 0x7fc6a006b4b0 2023-03-03T08:03:45.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.109+0000 7fc6a4f9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6a00d0270 0x7fc6a00b4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:45.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.109+0000 7fc6a4f9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6a00d0270 0x7fc6a00b4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38006/0 (socket says 172.21.15.98:38006) 2023-03-03T08:03:45.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.109+0000 7fc6a4f9a700 1 -- 172.21.15.98:0/3925359966 learned_addr learned my addr 172.21.15.98:0/3925359966 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:45.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.109+0000 7fc6a579b700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6a006b4b0 0x7fc6a0149760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:45.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.109+0000 7fc6a5f9c700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6a015ca70 0x7fc6a00b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:45.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.109+0000 7fc6a4f9a700 1 -- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6a006b4b0 msgr2=0x7fc6a0149760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:45.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.109+0000 7fc6a4f9a700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6a006b4b0 0x7fc6a0149760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:45.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.109+0000 7fc6a4f9a700 1 -- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6a015ca70 msgr2=0x7fc6a00b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:45.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.109+0000 7fc6a4f9a700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6a015ca70 0x7fc6a00b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:45.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.110+0000 7fc6a4f9a700 1 -- 172.21.15.98:0/3925359966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc698009a70 con 0x7fc6a00d0270 2023-03-03T08:03:45.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.110+0000 7fc6a579b700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6a006b4b0 0x7fc6a0149760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:45.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.110+0000 7fc6a5f9c700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6a015ca70 0x7fc6a00b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:45.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.110+0000 7fc6a4f9a700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6a00d0270 0x7fc6a00b4db0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fc69000e870 tx=0x7fc69000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:45.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.110+0000 7fc695ffb700 1 -- 172.21.15.98:0/3925359966 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69000c640 con 0x7fc6a00d0270 2023-03-03T08:03:45.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.110+0000 7fc695ffb700 1 -- 172.21.15.98:0/3925359966 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69000c7a0 con 0x7fc6a00d0270 2023-03-03T08:03:45.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.110+0000 7fc695ffb700 1 -- 172.21.15.98:0/3925359966 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69000cb10 con 0x7fc6a00d0270 2023-03-03T08:03:45.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.110+0000 7fc6a79ff700 1 -- 172.21.15.98:0/3925359966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a00d1aa0 con 0x7fc6a00d0270 2023-03-03T08:03:45.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.110+0000 7fc6a79ff700 1 -- 172.21.15.98:0/3925359966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a00cdbe0 con 0x7fc6a00d0270 2023-03-03T08:03:45.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.112+0000 7fc6a79ff700 1 -- 172.21.15.98:0/3925359966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a015ce40 con 0x7fc6a00d0270 2023-03-03T08:03:45.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.113+0000 7fc695ffb700 1 -- 172.21.15.98:0/3925359966 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 86615+0+0 (secure 0 0 0) 0x7fc690019030 con 0x7fc6a00d0270 2023-03-03T08:03:45.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.114+0000 7fc695ffb700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc684068450 0x7fc68406a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:45.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.114+0000 7fc695ffb700 1 -- 172.21.15.98:0/3925359966 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc690085ba0 con 0x7fc6a00d0270 2023-03-03T08:03:45.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.114+0000 7fc6a579b700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc684068450 0x7fc68406a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:45.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.115+0000 7fc6a579b700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc684068450 0x7fc68406a900 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc69c005910 tx=0x7fc69c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:45.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.118+0000 7fc695ffb700 1 -- 172.21.15.98:0/3925359966 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc690052380 con 0x7fc6a00d0270 2023-03-03T08:03:45.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.349+0000 7fc6a79ff700 1 -- 172.21.15.98:0/3925359966 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvolume_0000000000508309", "snap_name": "fake", "target_sub_name": "clone_0000000001033877", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6a004e6e0 con 0x7fc684068450 2023-03-03T08:03:45.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.428+0000 7fc695ffb700 1 -- 172.21.15.98:0/3925359966 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc6a004e6e0 con 0x7fc684068450 2023-03-03T08:03:45.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.430+0000 7fc6a79ff700 1 -- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc684068450 msgr2=0x7fc68406a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:45.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.430+0000 7fc6a79ff700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc684068450 0x7fc68406a900 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc69c005910 tx=0x7fc69c009040 comp rx=0 tx=0).stop 2023-03-03T08:03:45.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.430+0000 7fc6a79ff700 1 -- 172.21.15.98:0/3925359966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6a00d0270 msgr2=0x7fc6a00b4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:45.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.430+0000 7fc6a79ff700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6a00d0270 0x7fc6a00b4db0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fc69000e870 tx=0x7fc69000ec30 comp rx=0 tx=0).stop 2023-03-03T08:03:45.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.430+0000 7fc6a79ff700 1 -- 172.21.15.98:0/3925359966 shutdown_connections 2023-03-03T08:03:45.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.430+0000 7fc6a79ff700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6a015ca70 0x7fc6a00b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:45.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.430+0000 7fc6a79ff700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6a006b4b0 0x7fc6a0149760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:45.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.430+0000 7fc6a79ff700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc684068450 0x7fc68406a900 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:45.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.430+0000 7fc6a79ff700 1 --2- 172.21.15.98:0/3925359966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6a00d0270 0x7fc6a00b4db0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:45.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.430+0000 7fc6a79ff700 1 -- 172.21.15.98:0/3925359966 >> 172.21.15.98:0/3925359966 conn(0x7fc6a0073e70 msgr2=0x7fc6a00b6830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:45.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.431+0000 7fc6a79ff700 1 -- 172.21.15.98:0/3925359966 shutdown_connections 2023-03-03T08:03:45.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.431+0000 7fc6a79ff700 1 -- 172.21.15.98:0/3925359966 wait complete. 2023-03-03T08:03:45.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs clone_0000000001033877 2023-03-03T08:03:45.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.711+0000 7fd244dff700 1 Processor -- start 2023-03-03T08:03:45.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.711+0000 7fd244dff700 1 -- start start 2023-03-03T08:03:45.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.712+0000 7fd244dff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd2400b5310 0x7fd240167850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:45.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.712+0000 7fd244dff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2400cd7d0 con 0x7fd2400b26d0 2023-03-03T08:03:45.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.712+0000 7fd244dff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2400b90e0 con 0x7fd2400b8d10 2023-03-03T08:03:45.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.712+0000 7fd244dff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2400cda20 con 0x7fd2400b5310 2023-03-03T08:03:45.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.713+0000 7fd23dd9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd2400b26d0 0x7fd2400b4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51830/0 (socket says 172.21.15.98:51830) 2023-03-03T08:03:45.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.713+0000 7fd23dd9b700 1 -- 172.21.15.98:0/965308625 learned_addr learned my addr 172.21.15.98:0/965308625 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:45.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.713+0000 7fd23ed9d700 1 --2- 172.21.15.98:0/965308625 >> v2:172.21.15.98:3300/0 conn(0x7fd2400b5310 0x7fd240167850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:45.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.713+0000 7fd23d59a700 1 -- 172.21.15.98:0/965308625 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350423499 0 0) 0x7fd2400cd7d0 con 0x7fd2400b26d0 2023-03-03T08:03:45.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.713+0000 7fd23d59a700 1 -- 172.21.15.98:0/965308625 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2280035b0 con 0x7fd2400b26d0 2023-03-03T08:03:45.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.713+0000 7fd23ed9d700 1 -- 172.21.15.98:0/965308625 >> v1:172.21.15.100:6789/0 conn(0x7fd2400b8d10 legacy=0x7fd2400cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:45.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.713+0000 7fd23ed9d700 1 -- 172.21.15.98:0/965308625 >> v1:172.21.15.98:6789/0 conn(0x7fd2400b26d0 legacy=0x7fd2400b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:45.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.714+0000 7fd23ed9d700 1 -- 172.21.15.98:0/965308625 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2400cdd00 con 0x7fd2400b5310 2023-03-03T08:03:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.714+0000 7fd23ed9d700 1 --2- 172.21.15.98:0/965308625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400b5310 0x7fd240167850 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fd238009d90 tx=0x7fd23800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=271aeb2c94206bc6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.714+0000 7fd23d59a700 1 -- 172.21.15.98:0/965308625 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23800e070 con 0x7fd2400b5310 2023-03-03T08:03:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.714+0000 7fd23d59a700 1 -- 172.21.15.98:0/965308625 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23800bb20 con 0x7fd2400b5310 2023-03-03T08:03:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.714+0000 7fd23d59a700 1 -- 172.21.15.98:0/965308625 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd238012430 con 0x7fd2400b5310 2023-03-03T08:03:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.715+0000 7fd244dff700 1 -- 172.21.15.98:0/965308625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400b5310 msgr2=0x7fd240167850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.715+0000 7fd244dff700 1 --2- 172.21.15.98:0/965308625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400b5310 0x7fd240167850 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fd238009d90 tx=0x7fd23800b330 comp rx=0 tx=0).stop 2023-03-03T08:03:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.715+0000 7fd244dff700 1 -- 172.21.15.98:0/965308625 shutdown_connections 2023-03-03T08:03:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.715+0000 7fd244dff700 1 --2- 172.21.15.98:0/965308625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400b5310 0x7fd240167850 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.715+0000 7fd244dff700 1 -- 172.21.15.98:0/965308625 >> 172.21.15.98:0/965308625 conn(0x7fd240073e70 msgr2=0x7fd24006ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.715+0000 7fd244dff700 1 -- 172.21.15.98:0/965308625 shutdown_connections 2023-03-03T08:03:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.715+0000 7fd244dff700 1 -- 172.21.15.98:0/965308625 wait complete. 2023-03-03T08:03:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.717+0000 7fd244dff700 1 Processor -- start 2023-03-03T08:03:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.717+0000 7fd244dff700 1 -- start start 2023-03-03T08:03:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.718+0000 7fd244dff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2400b26d0 0x7fd24015b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.718+0000 7fd244dff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400b5310 0x7fd2400cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.718+0000 7fd244dff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2400b8d10 0x7fd24015e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.718+0000 7fd244dff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24015f100 con 0x7fd2400b5310 2023-03-03T08:03:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.718+0000 7fd244dff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2400ca2f0 con 0x7fd2400b26d0 2023-03-03T08:03:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.718+0000 7fd244dff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2400ca5d0 con 0x7fd2400b8d10 2023-03-03T08:03:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.719+0000 7fd23dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400b5310 0x7fd2400cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:45.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.719+0000 7fd23dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400b5310 0x7fd2400cadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38022/0 (socket says 172.21.15.98:38022) 2023-03-03T08:03:45.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.719+0000 7fd23e59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2400b26d0 0x7fd24015b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:45.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.719+0000 7fd23dd9b700 1 -- 172.21.15.98:0/117502340 learned_addr learned my addr 172.21.15.98:0/117502340 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:45.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.719+0000 7fd23ed9d700 1 --2- 172.21.15.98:0/117502340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2400b8d10 0x7fd24015e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:45.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.719+0000 7fd23dd9b700 1 -- 172.21.15.98:0/117502340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2400b8d10 msgr2=0x7fd24015e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:45.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.719+0000 7fd23dd9b700 1 --2- 172.21.15.98:0/117502340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2400b8d10 0x7fd24015e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:45.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.719+0000 7fd23dd9b700 1 -- 172.21.15.98:0/117502340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2400b26d0 msgr2=0x7fd24015b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:45.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.719+0000 7fd23dd9b700 1 --2- 172.21.15.98:0/117502340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2400b26d0 0x7fd24015b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:45.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.719+0000 7fd23dd9b700 1 -- 172.21.15.98:0/117502340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd238009a70 con 0x7fd2400b5310 2023-03-03T08:03:45.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.719+0000 7fd23dd9b700 1 --2- 172.21.15.98:0/117502340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400b5310 0x7fd2400cadd0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fd234001eb0 tx=0x7fd23400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:45.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.720+0000 7fd22effd700 1 -- 172.21.15.98:0/117502340 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd234005d70 con 0x7fd2400b5310 2023-03-03T08:03:45.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.720+0000 7fd22effd700 1 -- 172.21.15.98:0/117502340 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2340070d0 con 0x7fd2400b5310 2023-03-03T08:03:45.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.720+0000 7fd22effd700 1 -- 172.21.15.98:0/117502340 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd234007460 con 0x7fd2400b5310 2023-03-03T08:03:45.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.720+0000 7fd244dff700 1 -- 172.21.15.98:0/117502340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2400cdfe0 con 0x7fd2400b5310 2023-03-03T08:03:45.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.720+0000 7fd244dff700 1 -- 172.21.15.98:0/117502340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2400d0e80 con 0x7fd2400b5310 2023-03-03T08:03:45.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.721+0000 7fd244dff700 1 -- 172.21.15.98:0/117502340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2400b5fd0 con 0x7fd2400b5310 2023-03-03T08:03:45.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.722+0000 7fd22effd700 1 -- 172.21.15.98:0/117502340 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 86615+0+0 (secure 0 0 0) 0x7fd234019030 con 0x7fd2400b5310 2023-03-03T08:03:45.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.724+0000 7fd22effd700 1 --2- 172.21.15.98:0/117502340 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd220068450 0x7fd22006a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:45.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.724+0000 7fd22effd700 1 -- 172.21.15.98:0/117502340 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd234085160 con 0x7fd2400b5310 2023-03-03T08:03:45.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.724+0000 7fd23e59c700 1 --2- 172.21.15.98:0/117502340 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd220068450 0x7fd22006a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:45.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.725+0000 7fd23e59c700 1 --2- 172.21.15.98:0/117502340 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd220068450 0x7fd22006a900 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd24015e720 tx=0x7fd230009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:45.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.727+0000 7fd22effd700 1 -- 172.21.15.98:0/117502340 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd234051850 con 0x7fd2400b5310 2023-03-03T08:03:45.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:45.959+0000 7fd244dff700 1 -- 172.21.15.98:0/117502340 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "clone_0000000001033877", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2400caaf0 con 0x7fd220068450 2023-03-03T08:03:47.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:47.728+0000 7fd22effd700 1 -- 172.21.15.98:0/117502340 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+168 (secure 0 0 0) 0x7fd2400caaf0 con 0x7fd220068450 2023-03-03T08:03:47.730 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:03:47.730 INFO:teuthology.orchestra.run.smithi098.stdout: "status": { 2023-03-03T08:03:47.730 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "pending", 2023-03-03T08:03:47.730 INFO:teuthology.orchestra.run.smithi098.stdout: "source": { 2023-03-03T08:03:47.730 INFO:teuthology.orchestra.run.smithi098.stdout: "volume": "cephfs", 2023-03-03T08:03:47.731 INFO:teuthology.orchestra.run.smithi098.stdout: "subvolume": "subvolume_0000000000508309", 2023-03-03T08:03:47.731 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot": "fake" 2023-03-03T08:03:47.731 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:47.731 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:47.731 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:03:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:47.731+0000 7fd244dff700 1 -- 172.21.15.98:0/117502340 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd220068450 msgr2=0x7fd22006a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:47.731+0000 7fd244dff700 1 --2- 172.21.15.98:0/117502340 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd220068450 0x7fd22006a900 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd24015e720 tx=0x7fd230009040 comp rx=0 tx=0).stop 2023-03-03T08:03:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:47.731+0000 7fd244dff700 1 -- 172.21.15.98:0/117502340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400b5310 msgr2=0x7fd2400cadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:47.731+0000 7fd244dff700 1 --2- 172.21.15.98:0/117502340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400b5310 0x7fd2400cadd0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fd234001eb0 tx=0x7fd23400c7b0 comp rx=0 tx=0).stop 2023-03-03T08:03:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:47.732+0000 7fd244dff700 1 -- 172.21.15.98:0/117502340 shutdown_connections 2023-03-03T08:03:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:47.732+0000 7fd244dff700 1 --2- 172.21.15.98:0/117502340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2400b8d10 0x7fd24015e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:47.732+0000 7fd244dff700 1 --2- 172.21.15.98:0/117502340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2400b26d0 0x7fd24015b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:47.732+0000 7fd244dff700 1 --2- 172.21.15.98:0/117502340 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fd220068450 0x7fd22006a900 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:47.732+0000 7fd244dff700 1 --2- 172.21.15.98:0/117502340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400b5310 0x7fd2400cadd0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:47.732+0000 7fd244dff700 1 -- 172.21.15.98:0/117502340 >> 172.21.15.98:0/117502340 conn(0x7fd240073e70 msgr2=0x7fd2400b3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:47.732+0000 7fd244dff700 1 -- 172.21.15.98:0/117502340 shutdown_connections 2023-03-03T08:03:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:47.732+0000 7fd244dff700 1 -- 172.21.15.98:0/117502340 wait complete. 2023-03-03T08:03:48.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs clone_0000000001033877 2023-03-03T08:03:49.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.020+0000 7fa82f59d700 1 Processor -- start 2023-03-03T08:03:49.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.021+0000 7fa82f59d700 1 -- start start 2023-03-03T08:03:49.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.022+0000 7fa82f59d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa8281609b0 0x7fa828160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:49.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.022+0000 7fa82f59d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa8280cf570 0x7fa82815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:49.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.022+0000 7fa82f59d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8281612c0 con 0x7fa82815db30 2023-03-03T08:03:49.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.022+0000 7fa82f59d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8280cebb0 con 0x7fa8281609b0 2023-03-03T08:03:49.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.022+0000 7fa82f59d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8280cd780 con 0x7fa8280cf570 2023-03-03T08:03:49.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.022+0000 7fa82ed9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa82815db30 0x7fa8280b4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51846/0 (socket says 172.21.15.98:51846) 2023-03-03T08:03:49.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.022+0000 7fa82ed9c700 1 -- 172.21.15.98:0/2416931512 learned_addr learned my addr 172.21.15.98:0/2416931512 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:49.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.023+0000 7fa82dd9a700 1 --2- 172.21.15.98:0/2416931512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa8280cf570 0x7fa82815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:49.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.023+0000 7fa82e59b700 1 --2- 172.21.15.98:0/2416931512 >> v2:172.21.15.100:3300/0 conn(0x7fa8281609b0 0x7fa828160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:49.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.023+0000 7fa82d599700 1 -- 172.21.15.98:0/2416931512 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758568039 0 0) 0x7fa8281612c0 con 0x7fa82815db30 2023-03-03T08:03:49.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.023+0000 7fa82d599700 1 -- 172.21.15.98:0/2416931512 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8140035b0 con 0x7fa82815db30 2023-03-03T08:03:49.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.023+0000 7fa82dd9a700 1 -- 172.21.15.98:0/2416931512 >> v2:172.21.15.100:3300/0 conn(0x7fa8281609b0 msgr2=0x7fa828160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:49.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.023+0000 7fa82dd9a700 1 --2- 172.21.15.98:0/2416931512 >> v2:172.21.15.100:3300/0 conn(0x7fa8281609b0 0x7fa828160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:49.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.023+0000 7fa82dd9a700 1 -- 172.21.15.98:0/2416931512 >> v1:172.21.15.98:6789/0 conn(0x7fa82815db30 legacy=0x7fa8280b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:49.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.023+0000 7fa82dd9a700 1 -- 172.21.15.98:0/2416931512 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8280cda60 con 0x7fa8280cf570 2023-03-03T08:03:49.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.024+0000 7fa82dd9a700 1 --2- 172.21.15.98:0/2416931512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa8280cf570 0x7fa82815d5f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa818009840 tx=0x7fa818009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3dd7efab6049fc7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:49.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.024+0000 7fa82d599700 1 -- 172.21.15.98:0/2416931512 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818011830 con 0x7fa8280cf570 2023-03-03T08:03:49.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.024+0000 7fa82d599700 1 -- 172.21.15.98:0/2416931512 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa818011990 con 0x7fa8280cf570 2023-03-03T08:03:49.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.024+0000 7fa82d599700 1 -- 172.21.15.98:0/2416931512 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818011d20 con 0x7fa8280cf570 2023-03-03T08:03:49.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.025+0000 7fa82f59d700 1 -- 172.21.15.98:0/2416931512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa8280cf570 msgr2=0x7fa82815d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:49.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.025+0000 7fa82f59d700 1 --2- 172.21.15.98:0/2416931512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa8280cf570 0x7fa82815d5f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa818009840 tx=0x7fa818009b50 comp rx=0 tx=0).stop 2023-03-03T08:03:49.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.025+0000 7fa82f59d700 1 -- 172.21.15.98:0/2416931512 shutdown_connections 2023-03-03T08:03:49.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.025+0000 7fa82f59d700 1 --2- 172.21.15.98:0/2416931512 >> v2:172.21.15.100:3300/0 conn(0x7fa8281609b0 0x7fa828160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:49.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.025+0000 7fa82f59d700 1 --2- 172.21.15.98:0/2416931512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa8280cf570 0x7fa82815d5f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:49.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.025+0000 7fa82f59d700 1 -- 172.21.15.98:0/2416931512 >> 172.21.15.98:0/2416931512 conn(0x7fa82806b880 msgr2=0x7fa8280cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:49.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.025+0000 7fa82f59d700 1 -- 172.21.15.98:0/2416931512 shutdown_connections 2023-03-03T08:03:49.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.025+0000 7fa82f59d700 1 -- 172.21.15.98:0/2416931512 wait complete. 2023-03-03T08:03:49.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.028+0000 7fa82f59d700 1 Processor -- start 2023-03-03T08:03:49.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.028+0000 7fa82f59d700 1 -- start start 2023-03-03T08:03:49.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.029+0000 7fa82f59d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa8280cf570 0x7fa8281443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:49.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.029+0000 7fa82f59d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa82815db30 0x7fa8280cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.029+0000 7fa82f59d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8281609b0 0x7fa8281f2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.029+0000 7fa82f59d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8281448f0 con 0x7fa8281609b0 2023-03-03T08:03:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.029+0000 7fa82f59d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8280b81c0 con 0x7fa8280cf570 2023-03-03T08:03:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.029+0000 7fa82f59d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8280b84a0 con 0x7fa82815db30 2023-03-03T08:03:49.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.030+0000 7fa82ed9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8281609b0 0x7fa8281f2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:49.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.030+0000 7fa82ed9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8281609b0 0x7fa8281f2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38036/0 (socket says 172.21.15.98:38036) 2023-03-03T08:03:49.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.030+0000 7fa82ed9c700 1 -- 172.21.15.98:0/2866857929 learned_addr learned my addr 172.21.15.98:0/2866857929 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.030+0000 7fa82dd9a700 1 --2- 172.21.15.98:0/2866857929 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa82815db30 0x7fa8280cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.030+0000 7fa82e59b700 1 --2- 172.21.15.98:0/2866857929 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa8280cf570 0x7fa8281443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.030+0000 7fa82ed9c700 1 -- 172.21.15.98:0/2866857929 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa82815db30 msgr2=0x7fa8280cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.030+0000 7fa82ed9c700 1 --2- 172.21.15.98:0/2866857929 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa82815db30 0x7fa8280cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.030+0000 7fa82ed9c700 1 -- 172.21.15.98:0/2866857929 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa8280cf570 msgr2=0x7fa8281443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.030+0000 7fa82ed9c700 1 --2- 172.21.15.98:0/2866857929 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa8280cf570 0x7fa8281443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.030+0000 7fa82ed9c700 1 -- 172.21.15.98:0/2866857929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8180094a0 con 0x7fa8281609b0 2023-03-03T08:03:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.031+0000 7fa82ed9c700 1 --2- 172.21.15.98:0/2866857929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8281609b0 0x7fa8281f2300 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fa82000cb40 tx=0x7fa82000ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.031+0000 7fa81effd700 1 -- 172.21.15.98:0/2866857929 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82000a9e0 con 0x7fa8281609b0 2023-03-03T08:03:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.031+0000 7fa81effd700 1 -- 172.21.15.98:0/2866857929 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82000ab40 con 0x7fa8281609b0 2023-03-03T08:03:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.031+0000 7fa82f59d700 1 -- 172.21.15.98:0/2866857929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82815cdf0 con 0x7fa8281609b0 2023-03-03T08:03:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.031+0000 7fa81effd700 1 -- 172.21.15.98:0/2866857929 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa820012400 con 0x7fa8281609b0 2023-03-03T08:03:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.031+0000 7fa82f59d700 1 -- 172.21.15.98:0/2866857929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8280cdd40 con 0x7fa8281609b0 2023-03-03T08:03:49.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.032+0000 7fa82f59d700 1 -- 172.21.15.98:0/2866857929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82815d0d0 con 0x7fa8281609b0 2023-03-03T08:03:49.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.033+0000 7fa81effd700 1 -- 172.21.15.98:0/2866857929 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 86615+0+0 (secure 0 0 0) 0x7fa820019030 con 0x7fa8281609b0 2023-03-03T08:03:49.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.035+0000 7fa81effd700 1 --2- 172.21.15.98:0/2866857929 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa80c068450 0x7fa80c06a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:49.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.035+0000 7fa81effd700 1 -- 172.21.15.98:0/2866857929 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa8200858c0 con 0x7fa8281609b0 2023-03-03T08:03:49.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.035+0000 7fa82e59b700 1 --2- 172.21.15.98:0/2866857929 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa80c068450 0x7fa80c06a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:49.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.036+0000 7fa82e59b700 1 --2- 172.21.15.98:0/2866857929 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa80c068450 0x7fa80c06a900 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa8281f3480 tx=0x7fa824009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:49.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.039+0000 7fa81effd700 1 -- 172.21.15.98:0/2866857929 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa82004b5e0 con 0x7fa8281609b0 2023-03-03T08:03:49.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:49.272+0000 7fa82f59d700 1 -- 172.21.15.98:0/2866857929 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "clone_0000000001033877", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8280b2a80 con 0x7fa80c068450 2023-03-03T08:03:52.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:52.721+0000 7fa81effd700 1 -- 172.21.15.98:0/2866857929 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fa8280b2a80 con 0x7fa80c068450 2023-03-03T08:03:52.723 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:03:52.723 INFO:teuthology.orchestra.run.smithi098.stdout: "status": { 2023-03-03T08:03:52.723 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "complete" 2023-03-03T08:03:52.723 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:52.724 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:03:52.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:52.724+0000 7fa82f59d700 1 -- 172.21.15.98:0/2866857929 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa80c068450 msgr2=0x7fa80c06a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:52.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:52.724+0000 7fa82f59d700 1 --2- 172.21.15.98:0/2866857929 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa80c068450 0x7fa80c06a900 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa8281f3480 tx=0x7fa824009210 comp rx=0 tx=0).stop 2023-03-03T08:03:52.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:52.724+0000 7fa82f59d700 1 -- 172.21.15.98:0/2866857929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8281609b0 msgr2=0x7fa8281f2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:52.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:52.724+0000 7fa82f59d700 1 --2- 172.21.15.98:0/2866857929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8281609b0 0x7fa8281f2300 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fa82000cb40 tx=0x7fa82000ce50 comp rx=0 tx=0).stop 2023-03-03T08:03:52.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:52.724+0000 7fa82f59d700 1 -- 172.21.15.98:0/2866857929 shutdown_connections 2023-03-03T08:03:52.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:52.724+0000 7fa82f59d700 1 --2- 172.21.15.98:0/2866857929 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa80c068450 0x7fa80c06a900 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:52.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:52.724+0000 7fa82f59d700 1 --2- 172.21.15.98:0/2866857929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8281609b0 0x7fa8281f2300 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:52.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:52.725+0000 7fa82f59d700 1 --2- 172.21.15.98:0/2866857929 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa8280cf570 0x7fa8281443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:52.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:52.725+0000 7fa82f59d700 1 --2- 172.21.15.98:0/2866857929 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa82815db30 0x7fa8280cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:52.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:52.725+0000 7fa82f59d700 1 -- 172.21.15.98:0/2866857929 >> 172.21.15.98:0/2866857929 conn(0x7fa82806b880 msgr2=0x7fa8280d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:52.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:52.725+0000 7fa82f59d700 1 -- 172.21.15.98:0/2866857929 shutdown_connections 2023-03-03T08:03:52.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:52.725+0000 7fa82f59d700 1 -- 172.21.15.98:0/2866857929 wait complete. 2023-03-03T08:03:52.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:03:52.756 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/clone_0000000001033877/.meta) 2023-03-03T08:03:52.861 INFO:teuthology.orchestra.run.smithi098.stdout:[GLOBAL] 2023-03-03T08:03:52.861 INFO:teuthology.orchestra.run.smithi098.stdout:version = 2 2023-03-03T08:03:52.861 INFO:teuthology.orchestra.run.smithi098.stdout:type = clone 2023-03-03T08:03:52.861 INFO:teuthology.orchestra.run.smithi098.stdout:path = /volumes/_nogroup/clone_0000000001033877/0eaa77e3-623d-4f22-a7e1-126ad14f45db 2023-03-03T08:03:52.862 INFO:teuthology.orchestra.run.smithi098.stdout:state = complete 2023-03-03T08:03:52.862 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:03:52.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot info cephfs subvolume_0000000000508309 snapshot_0000000000029275 2023-03-03T08:03:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.117+0000 7f00fa1ff700 1 Processor -- start 2023-03-03T08:03:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.118+0000 7f00fa1ff700 1 -- start start 2023-03-03T08:03:53.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.119+0000 7f00fa1ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f00f40cd770 0x7f00f40b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:53.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.119+0000 7f00fa1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00f415bf70 0x7f00f415c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:53.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.119+0000 7f00fa1ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f40d0ac0 con 0x7f00f40b8880 2023-03-03T08:03:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.119+0000 7f00fa1ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f40cdbe0 con 0x7f00f40cd770 2023-03-03T08:03:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.119+0000 7f00fa1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f40cff70 con 0x7f00f415bf70 2023-03-03T08:03:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.119+0000 7f00f37fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f00f40b8880 0x7f00f40b4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51856/0 (socket says 172.21.15.98:51856) 2023-03-03T08:03:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.119+0000 7f00f37fe700 1 -- 172.21.15.98:0/2472758367 learned_addr learned my addr 172.21.15.98:0/2472758367 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.119+0000 7f00f2ffd700 1 --2- 172.21.15.98:0/2472758367 >> v2:172.21.15.98:3300/0 conn(0x7f00f40cd770 0x7f00f40b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.120+0000 7f00f3fff700 1 --2- 172.21.15.98:0/2472758367 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00f415bf70 0x7f00f415c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.120+0000 7f00f27fc700 1 -- 172.21.15.98:0/2472758367 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588570096 0 0) 0x7f00f40d0ac0 con 0x7f00f40b8880 2023-03-03T08:03:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.120+0000 7f00f2ffd700 1 -- 172.21.15.98:0/2472758367 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00f415bf70 msgr2=0x7f00f415c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.120+0000 7f00f2ffd700 1 --2- 172.21.15.98:0/2472758367 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00f415bf70 0x7f00f415c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.120+0000 7f00f2ffd700 1 -- 172.21.15.98:0/2472758367 >> v1:172.21.15.98:6789/0 conn(0x7f00f40b8880 legacy=0x7f00f40b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.120+0000 7f00f2ffd700 1 -- 172.21.15.98:0/2472758367 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f40d0250 con 0x7f00f40cd770 2023-03-03T08:03:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.120+0000 7f00f3fff700 1 --2- 172.21.15.98:0/2472758367 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00f415bf70 0x7f00f415c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.120+0000 7f00f2ffd700 1 --2- 172.21.15.98:0/2472758367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f40cd770 0x7f00f40b52b0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009840 tx=0x7f00e8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4650e57fb8d3e414 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.120+0000 7f00f27fc700 1 -- 172.21.15.98:0/2472758367 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8011830 con 0x7f00f40cd770 2023-03-03T08:03:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.121+0000 7f00f27fc700 1 -- 172.21.15.98:0/2472758367 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e8011990 con 0x7f00f40cd770 2023-03-03T08:03:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.121+0000 7f00f27fc700 1 -- 172.21.15.98:0/2472758367 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8011d20 con 0x7f00f40cd770 2023-03-03T08:03:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.121+0000 7f00fa1ff700 1 -- 172.21.15.98:0/2472758367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f40cd770 msgr2=0x7f00f40b52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.121+0000 7f00fa1ff700 1 --2- 172.21.15.98:0/2472758367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f40cd770 0x7f00f40b52b0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009840 tx=0x7f00e8009b50 comp rx=0 tx=0).stop 2023-03-03T08:03:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.121+0000 7f00fa1ff700 1 -- 172.21.15.98:0/2472758367 shutdown_connections 2023-03-03T08:03:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.121+0000 7f00fa1ff700 1 --2- 172.21.15.98:0/2472758367 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00f415bf70 0x7f00f415c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.121+0000 7f00fa1ff700 1 --2- 172.21.15.98:0/2472758367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f40cd770 0x7f00f40b52b0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.121+0000 7f00fa1ff700 1 -- 172.21.15.98:0/2472758367 >> 172.21.15.98:0/2472758367 conn(0x7f00f406ad80 msgr2=0x7f00f406b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.121+0000 7f00fa1ff700 1 -- 172.21.15.98:0/2472758367 shutdown_connections 2023-03-03T08:03:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.121+0000 7f00fa1ff700 1 -- 172.21.15.98:0/2472758367 wait complete. 2023-03-03T08:03:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.124+0000 7f00fa1ff700 1 Processor -- start 2023-03-03T08:03:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.124+0000 7f00fa1ff700 1 -- start start 2023-03-03T08:03:53.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.126+0000 7f00fa1ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00f40b8880 0x7f00f4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:53.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.126+0000 7f00fa1ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00f40cd770 0x7f00f4159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:53.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.126+0000 7f00fa1ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f415bf70 0x7f00f4159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:53.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.126+0000 7f00fa1ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f40b55d0 con 0x7f00f415bf70 2023-03-03T08:03:53.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.126+0000 7f00fa1ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f415a3f0 con 0x7f00f40b8880 2023-03-03T08:03:53.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.126+0000 7f00fa1ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f41f23a0 con 0x7f00f40cd770 2023-03-03T08:03:53.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.126+0000 7f00f3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f415bf70 0x7f00f4159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:53.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.126+0000 7f00f3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f415bf70 0x7f00f4159eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38046/0 (socket says 172.21.15.98:38046) 2023-03-03T08:03:53.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.126+0000 7f00f3fff700 1 -- 172.21.15.98:0/158628789 learned_addr learned my addr 172.21.15.98:0/158628789 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:53.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.126+0000 7f00f2ffd700 1 --2- 172.21.15.98:0/158628789 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00f40cd770 0x7f00f4159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:53.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.126+0000 7f00f37fe700 1 --2- 172.21.15.98:0/158628789 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00f40b8880 0x7f00f4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:53.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.127+0000 7f00f3fff700 1 -- 172.21.15.98:0/158628789 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00f40cd770 msgr2=0x7f00f4159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:53.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.127+0000 7f00f3fff700 1 --2- 172.21.15.98:0/158628789 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00f40cd770 0x7f00f4159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.127+0000 7f00f3fff700 1 -- 172.21.15.98:0/158628789 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00f40b8880 msgr2=0x7f00f4155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:53.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.127+0000 7f00f3fff700 1 --2- 172.21.15.98:0/158628789 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00f40b8880 0x7f00f4155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.127+0000 7f00f3fff700 1 -- 172.21.15.98:0/158628789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e80094a0 con 0x7f00f415bf70 2023-03-03T08:03:53.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.127+0000 7f00f3fff700 1 --2- 172.21.15.98:0/158628789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f415bf70 0x7f00f4159eb0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009fa0 tx=0x7f00ec00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:53.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.127+0000 7f00dbfff700 1 -- 172.21.15.98:0/158628789 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec00ddb0 con 0x7f00f415bf70 2023-03-03T08:03:53.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.127+0000 7f00dbfff700 1 -- 172.21.15.98:0/158628789 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ec00f040 con 0x7f00f415bf70 2023-03-03T08:03:53.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.127+0000 7f00dbfff700 1 -- 172.21.15.98:0/158628789 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec003c60 con 0x7f00f415bf70 2023-03-03T08:03:53.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.127+0000 7f00fa1ff700 1 -- 172.21.15.98:0/158628789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f40d0cf0 con 0x7f00f415bf70 2023-03-03T08:03:53.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.128+0000 7f00fa1ff700 1 -- 172.21.15.98:0/158628789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f40d0530 con 0x7f00f415bf70 2023-03-03T08:03:53.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.129+0000 7f00fa1ff700 1 -- 172.21.15.98:0/158628789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f40b8c50 con 0x7f00f415bf70 2023-03-03T08:03:53.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.130+0000 7f00dbfff700 1 -- 172.21.15.98:0/158628789 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 86615+0+0 (secure 0 0 0) 0x7f00ec01c030 con 0x7f00f415bf70 2023-03-03T08:03:53.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.131+0000 7f00dbfff700 1 --2- 172.21.15.98:0/158628789 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f00e0068450 0x7f00e006a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:53.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.131+0000 7f00dbfff700 1 -- 172.21.15.98:0/158628789 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f00ec085e20 con 0x7f00f415bf70 2023-03-03T08:03:53.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.132+0000 7f00f37fe700 1 --2- 172.21.15.98:0/158628789 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f00e0068450 0x7f00e006a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:53.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.132+0000 7f00f37fe700 1 --2- 172.21.15.98:0/158628789 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f00e0068450 0x7f00e006a900 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f00e4000f60 tx=0x7f00e4007e70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:53.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.135+0000 7f00dbfff700 1 -- 172.21.15.98:0/158628789 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f00ec052510 con 0x7f00f415bf70 2023-03-03T08:03:53.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.363+0000 7f00fa1ff700 1 -- 172.21.15.98:0/158628789 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot info", "vol_name": "cephfs", "sub_name": "subvolume_0000000000508309", "snap_name": "snapshot_0000000000029275", "target": ["mon-mgr", ""]}) v1 -- 0x7f00f40ce4f0 con 0x7f00e0068450 2023-03-03T08:03:53.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.383+0000 7f00dbfff700 1 -- 172.21.15.98:0/158628789 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+115 (secure 0 0 0) 0x7f00f40ce4f0 con 0x7f00e0068450 2023-03-03T08:03:53.385 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-03-03T08:03:53.385 INFO:teuthology.orchestra.run.smithi098.stdout: "created_at": "2023-03-03 08:03:35.208439", 2023-03-03T08:03:53.385 INFO:teuthology.orchestra.run.smithi098.stdout: "data_pool": "cephfs_data", 2023-03-03T08:03:53.385 INFO:teuthology.orchestra.run.smithi098.stdout: "has_pending_clones": "no" 2023-03-03T08:03:53.385 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-03-03T08:03:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.385+0000 7f00fa1ff700 1 -- 172.21.15.98:0/158628789 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f00e0068450 msgr2=0x7f00e006a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.385+0000 7f00fa1ff700 1 --2- 172.21.15.98:0/158628789 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f00e0068450 0x7f00e006a900 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f00e4000f60 tx=0x7f00e4007e70 comp rx=0 tx=0).stop 2023-03-03T08:03:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.385+0000 7f00fa1ff700 1 -- 172.21.15.98:0/158628789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f415bf70 msgr2=0x7f00f4159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.385+0000 7f00fa1ff700 1 --2- 172.21.15.98:0/158628789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f415bf70 0x7f00f4159eb0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009fa0 tx=0x7f00ec00d3b0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.385+0000 7f00fa1ff700 1 -- 172.21.15.98:0/158628789 shutdown_connections 2023-03-03T08:03:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.385+0000 7f00fa1ff700 1 --2- 172.21.15.98:0/158628789 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f00e0068450 0x7f00e006a900 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.386+0000 7f00fa1ff700 1 --2- 172.21.15.98:0/158628789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f415bf70 0x7f00f4159eb0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.386+0000 7f00fa1ff700 1 --2- 172.21.15.98:0/158628789 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00f40b8880 0x7f00f4155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.386+0000 7f00fa1ff700 1 --2- 172.21.15.98:0/158628789 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00f40cd770 0x7f00f4159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.386+0000 7f00fa1ff700 1 -- 172.21.15.98:0/158628789 >> 172.21.15.98:0/158628789 conn(0x7f00f406ad80 msgr2=0x7f00f40d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:53.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.386+0000 7f00fa1ff700 1 -- 172.21.15.98:0/158628789 shutdown_connections 2023-03-03T08:03:53.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.386+0000 7f00fa1ff700 1 -- 172.21.15.98:0/158628789 wait complete. 2023-03-03T08:03:53.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs subvolume_0000000000508309 2023-03-03T08:03:53.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.666+0000 7f0e767ff700 1 Processor -- start 2023-03-03T08:03:53.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.666+0000 7f0e767ff700 1 -- start start 2023-03-03T08:03:53.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.668+0000 7f0e767ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e7015d010 0x7f0e700b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:53.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.668+0000 7f0e767ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0e700b22c0 0x7f0e700d0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:53.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.668+0000 7f0e767ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e701617b0 con 0x7f0e7015cc40 2023-03-03T08:03:53.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.668+0000 7f0e767ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e700b2740 con 0x7f0e700b22c0 2023-03-03T08:03:53.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.668+0000 7f0e767ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7006b900 con 0x7f0e7015d010 2023-03-03T08:03:53.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.668+0000 7f0e6ffff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0e7015cc40 0x7f0e700cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51858/0 (socket says 172.21.15.98:51858) 2023-03-03T08:03:53.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.668+0000 7f0e6ffff700 1 -- 172.21.15.98:0/3270734388 learned_addr learned my addr 172.21.15.98:0/3270734388 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:53.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.668+0000 7f0e67fff700 1 --2- 172.21.15.98:0/3270734388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e7015d010 0x7f0e700b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:53.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.668+0000 7f0e74d9c700 1 --2- 172.21.15.98:0/3270734388 >> v2:172.21.15.100:3300/0 conn(0x7f0e700b22c0 0x7f0e700d0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:53.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.669+0000 7f0e74d9c700 1 -- 172.21.15.98:0/3270734388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e7015d010 msgr2=0x7f0e700b1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:53.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.669+0000 7f0e74d9c700 1 --2- 172.21.15.98:0/3270734388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e7015d010 0x7f0e700b1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.669+0000 7f0e74d9c700 1 -- 172.21.15.98:0/3270734388 >> v1:172.21.15.98:6789/0 conn(0x7f0e7015cc40 legacy=0x7f0e700cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:53.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.669+0000 7f0e74d9c700 1 -- 172.21.15.98:0/3270734388 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e701609f0 con 0x7f0e700b22c0 2023-03-03T08:03:53.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.669+0000 7f0e6f7fe700 1 -- 172.21.15.98:0/3270734388 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379709293 0 0) 0x7f0e701617b0 con 0x7f0e7015cc40 2023-03-03T08:03:53.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.670+0000 7f0e74d9c700 1 --2- 172.21.15.98:0/3270734388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e700b22c0 0x7f0e700d0f30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0e68009d90 tx=0x7f0e6800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40b5dca8447d0989 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:53.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.670+0000 7f0e6f7fe700 1 -- 172.21.15.98:0/3270734388 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6800e070 con 0x7f0e700b22c0 2023-03-03T08:03:53.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.670+0000 7f0e6f7fe700 1 -- 172.21.15.98:0/3270734388 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e6800bb20 con 0x7f0e700b22c0 2023-03-03T08:03:53.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.670+0000 7f0e6f7fe700 1 -- 172.21.15.98:0/3270734388 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e68012430 con 0x7f0e700b22c0 2023-03-03T08:03:53.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.671+0000 7f0e767ff700 1 -- 172.21.15.98:0/3270734388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e700b22c0 msgr2=0x7f0e700d0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:53.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.671+0000 7f0e767ff700 1 --2- 172.21.15.98:0/3270734388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e700b22c0 0x7f0e700d0f30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0e68009d90 tx=0x7f0e6800b330 comp rx=0 tx=0).stop 2023-03-03T08:03:53.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.671+0000 7f0e767ff700 1 -- 172.21.15.98:0/3270734388 shutdown_connections 2023-03-03T08:03:53.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.671+0000 7f0e767ff700 1 --2- 172.21.15.98:0/3270734388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e700b22c0 0x7f0e700d0f30 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.671+0000 7f0e767ff700 1 --2- 172.21.15.98:0/3270734388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e7015d010 0x7f0e700b1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.671+0000 7f0e767ff700 1 -- 172.21.15.98:0/3270734388 >> 172.21.15.98:0/3270734388 conn(0x7f0e70073e10 msgr2=0x7f0e700d31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:53.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.671+0000 7f0e767ff700 1 -- 172.21.15.98:0/3270734388 shutdown_connections 2023-03-03T08:03:53.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.671+0000 7f0e767ff700 1 -- 172.21.15.98:0/3270734388 wait complete. 2023-03-03T08:03:53.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.674+0000 7f0e767ff700 1 Processor -- start 2023-03-03T08:03:53.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.674+0000 7f0e767ff700 1 -- start start 2023-03-03T08:03:53.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.675+0000 7f0e767ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e700b22c0 0x7f0e700c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:53.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.675+0000 7f0e767ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e7015cc40 0x7f0e700b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:53.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.676+0000 7f0e767ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e7015d010 0x7f0e7015df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:53.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.676+0000 7f0e767ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e700b51d0 con 0x7f0e7015d010 2023-03-03T08:03:53.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.676+0000 7f0e767ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e700c9760 con 0x7f0e7015cc40 2023-03-03T08:03:53.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.676+0000 7f0e767ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e700b4550 con 0x7f0e700b22c0 2023-03-03T08:03:53.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.676+0000 7f0e6ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e700b22c0 0x7f0e700c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:53.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.676+0000 7f0e6ffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e700b22c0 0x7f0e700c9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:43630/0 (socket says 172.21.15.98:43630) 2023-03-03T08:03:53.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.676+0000 7f0e6ffff700 1 -- 172.21.15.98:0/30649415 learned_addr learned my addr 172.21.15.98:0/30649415 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:53.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.676+0000 7f0e67fff700 1 --2- 172.21.15.98:0/30649415 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e7015cc40 0x7f0e700b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:53.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.676+0000 7f0e74d9c700 1 --2- 172.21.15.98:0/30649415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e7015d010 0x7f0e7015df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:53.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.676+0000 7f0e74d9c700 1 -- 172.21.15.98:0/30649415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e700b22c0 msgr2=0x7f0e700c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:53.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.676+0000 7f0e74d9c700 1 --2- 172.21.15.98:0/30649415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e700b22c0 0x7f0e700c9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.676+0000 7f0e74d9c700 1 -- 172.21.15.98:0/30649415 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e7015cc40 msgr2=0x7f0e700b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.676+0000 7f0e74d9c700 1 --2- 172.21.15.98:0/30649415 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e7015cc40 0x7f0e700b3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.676+0000 7f0e74d9c700 1 -- 172.21.15.98:0/30649415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e68009a70 con 0x7f0e7015d010 2023-03-03T08:03:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.677+0000 7f0e74d9c700 1 --2- 172.21.15.98:0/30649415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e7015d010 0x7f0e7015df00 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f0e68016000 tx=0x7f0e68004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.677+0000 7f0e6d7fa700 1 -- 172.21.15.98:0/30649415 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e68012430 con 0x7f0e7015d010 2023-03-03T08:03:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.677+0000 7f0e6d7fa700 1 -- 172.21.15.98:0/30649415 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e68020070 con 0x7f0e7015d010 2023-03-03T08:03:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.677+0000 7f0e767ff700 1 -- 172.21.15.98:0/30649415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e70160fb0 con 0x7f0e7015d010 2023-03-03T08:03:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.677+0000 7f0e6d7fa700 1 -- 172.21.15.98:0/30649415 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e68004280 con 0x7f0e7015d010 2023-03-03T08:03:53.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.677+0000 7f0e767ff700 1 -- 172.21.15.98:0/30649415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e70160cd0 con 0x7f0e7015d010 2023-03-03T08:03:53.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.679+0000 7f0e767ff700 1 -- 172.21.15.98:0/30649415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e70161290 con 0x7f0e7015d010 2023-03-03T08:03:53.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.680+0000 7f0e6d7fa700 1 -- 172.21.15.98:0/30649415 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 86615+0+0 (secure 0 0 0) 0x7f0e6802e070 con 0x7f0e7015d010 2023-03-03T08:03:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.681+0000 7f0e6d7fa700 1 --2- 172.21.15.98:0/30649415 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0e50068450 0x7f0e5006a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.681+0000 7f0e6d7fa700 1 -- 172.21.15.98:0/30649415 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0e68022070 con 0x7f0e7015d010 2023-03-03T08:03:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.681+0000 7f0e6ffff700 1 --2- 172.21.15.98:0/30649415 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0e50068450 0x7f0e5006a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.682+0000 7f0e6ffff700 1 --2- 172.21.15.98:0/30649415 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0e50068450 0x7f0e5006a900 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0e60000f60 tx=0x7f0e60007ea0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:53.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.685+0000 7f0e6d7fa700 1 -- 172.21.15.98:0/30649415 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0e68054170 con 0x7f0e7015d010 2023-03-03T08:03:53.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.912+0000 7f0e767ff700 1 -- 172.21.15.98:0/30649415 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "subvolume_0000000000508309", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e700b5310 con 0x7f0e50068450 2023-03-03T08:03:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.929+0000 7f0e6d7fa700 1 -- 172.21.15.98:0/30649415 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+96 (secure 0 0 0) 0x7f0e700b5310 con 0x7f0e50068450 2023-03-03T08:03:53.930 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-03-03T08:03:53.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:53.931 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fake" 2023-03-03T08:03:53.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-03-03T08:03:53.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-03-03T08:03:53.931 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_0000000000029275" 2023-03-03T08:03:53.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-03-03T08:03:53.931 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-03-03T08:03:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.931+0000 7f0e767ff700 1 -- 172.21.15.98:0/30649415 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0e50068450 msgr2=0x7f0e5006a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.931+0000 7f0e767ff700 1 --2- 172.21.15.98:0/30649415 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0e50068450 0x7f0e5006a900 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0e60000f60 tx=0x7f0e60007ea0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.931+0000 7f0e767ff700 1 -- 172.21.15.98:0/30649415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e7015d010 msgr2=0x7f0e7015df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.931+0000 7f0e767ff700 1 --2- 172.21.15.98:0/30649415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e7015d010 0x7f0e7015df00 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f0e68016000 tx=0x7f0e68004ad0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.931+0000 7f0e767ff700 1 -- 172.21.15.98:0/30649415 shutdown_connections 2023-03-03T08:03:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.931+0000 7f0e767ff700 1 --2- 172.21.15.98:0/30649415 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f0e50068450 0x7f0e5006a900 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.931+0000 7f0e767ff700 1 --2- 172.21.15.98:0/30649415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e7015d010 0x7f0e7015df00 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.931+0000 7f0e767ff700 1 --2- 172.21.15.98:0/30649415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e700b22c0 0x7f0e700c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.931+0000 7f0e767ff700 1 --2- 172.21.15.98:0/30649415 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e7015cc40 0x7f0e700b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:53.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.931+0000 7f0e767ff700 1 -- 172.21.15.98:0/30649415 >> 172.21.15.98:0/30649415 conn(0x7f0e70073e10 msgr2=0x7f0e70156fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:53.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.931+0000 7f0e767ff700 1 -- 172.21.15.98:0/30649415 shutdown_connections 2023-03-03T08:03:53.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:53.932+0000 7f0e767ff700 1 -- 172.21.15.98:0/30649415 wait complete. 2023-03-03T08:03:53.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvolume_0000000000508309 snapshot_0000000000029275 2023-03-03T08:03:54.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.209+0000 7f97c5fff700 1 Processor -- start 2023-03-03T08:03:54.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.209+0000 7f97c5fff700 1 -- start start 2023-03-03T08:03:54.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.210+0000 7f97c5fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f97c00b8880 0x7f97c00cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:54.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.210+0000 7f97c5fff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f97c00b4ba0 0x7f97c00b5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:54.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.210+0000 7f97c5fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97c00b5550 0x7f97c01604c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:54.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.210+0000 7f97c5fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c00d0ac0 con 0x7f97c00b8880 2023-03-03T08:03:54.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.210+0000 7f97c5fff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c00b59d0 con 0x7f97c00b4ba0 2023-03-03T08:03:54.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.210+0000 7f97c5fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00cff70 con 0x7f97c00b5550 2023-03-03T08:03:54.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.211+0000 7f97bf7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f97c00b8880 0x7f97c00cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:54.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.211+0000 7f97bf7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f97c00b8880 0x7f97c00cd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38058/0 (socket says 172.21.15.98:38058) 2023-03-03T08:03:54.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.211+0000 7f97bf7fe700 1 -- 172.21.15.98:0/1209759509 learned_addr learned my addr 172.21.15.98:0/1209759509 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:54.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.211+0000 7f97bffff700 1 --2- 172.21.15.98:0/1209759509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97c00b5550 0x7f97c01604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:54.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.211+0000 7f97beffd700 1 --2- 172.21.15.98:0/1209759509 >> v2:172.21.15.100:3300/0 conn(0x7f97c00b4ba0 0x7f97c00b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:54.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.211+0000 7f97bf7fe700 1 -- 172.21.15.98:0/1209759509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97c00b5550 msgr2=0x7f97c01604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:54.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.211+0000 7f97bf7fe700 1 --2- 172.21.15.98:0/1209759509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97c00b5550 0x7f97c01604c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.211+0000 7f97bf7fe700 1 -- 172.21.15.98:0/1209759509 >> v2:172.21.15.100:3300/0 conn(0x7f97c00b4ba0 msgr2=0x7f97c00b5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:54.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.211+0000 7f97bf7fe700 1 --2- 172.21.15.98:0/1209759509 >> v2:172.21.15.100:3300/0 conn(0x7f97c00b4ba0 0x7f97c00b5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.211+0000 7f97bf7fe700 1 -- 172.21.15.98:0/1209759509 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c00d0250 con 0x7f97c00b8880 2023-03-03T08:03:54.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.211+0000 7f97beffd700 1 --2- 172.21.15.98:0/1209759509 >> v2:172.21.15.100:3300/0 conn(0x7f97c00b4ba0 0x7f97c00b5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:54.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.212+0000 7f97bf7fe700 1 --2- 172.21.15.98:0/1209759509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97c00b8880 0x7f97c00cd770 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f97c006a8a0 tx=0x7f97b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8edb9b1fac07dec server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:54.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.212+0000 7f97be7fc700 1 -- 172.21.15.98:0/1209759509 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b000b870 con 0x7f97c00b8880 2023-03-03T08:03:54.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.212+0000 7f97be7fc700 1 -- 172.21.15.98:0/1209759509 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b000b9d0 con 0x7f97c00b8880 2023-03-03T08:03:54.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.212+0000 7f97be7fc700 1 -- 172.21.15.98:0/1209759509 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b000bd60 con 0x7f97c00b8880 2023-03-03T08:03:54.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.213+0000 7f97c5fff700 1 -- 172.21.15.98:0/1209759509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97c00b8880 msgr2=0x7f97c00cd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:54.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.213+0000 7f97c5fff700 1 --2- 172.21.15.98:0/1209759509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97c00b8880 0x7f97c00cd770 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f97c006a8a0 tx=0x7f97b0009bd0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.213+0000 7f97c5fff700 1 -- 172.21.15.98:0/1209759509 shutdown_connections 2023-03-03T08:03:54.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.213+0000 7f97c5fff700 1 --2- 172.21.15.98:0/1209759509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97c00b5550 0x7f97c01604c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.213+0000 7f97c5fff700 1 --2- 172.21.15.98:0/1209759509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97c00b8880 0x7f97c00cd770 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.213+0000 7f97c5fff700 1 --2- 172.21.15.98:0/1209759509 >> v2:172.21.15.100:3300/0 conn(0x7f97c00b4ba0 0x7f97c00b5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.213+0000 7f97c5fff700 1 -- 172.21.15.98:0/1209759509 >> 172.21.15.98:0/1209759509 conn(0x7f97c006ad80 msgr2=0x7f97c006b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:54.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.213+0000 7f97c5fff700 1 -- 172.21.15.98:0/1209759509 shutdown_connections 2023-03-03T08:03:54.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.213+0000 7f97c5fff700 1 -- 172.21.15.98:0/1209759509 wait complete. 2023-03-03T08:03:54.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.215+0000 7f97c5fff700 1 Processor -- start 2023-03-03T08:03:54.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.215+0000 7f97c5fff700 1 -- start start 2023-03-03T08:03:54.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.216+0000 7f97c5fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97c00b4ba0 0x7f97c015a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:54.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.216+0000 7f97c5fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97c00b5550 0x7f97c00ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:54.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.216+0000 7f97c5fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97c00b8880 0x7f97c00c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:54.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.216+0000 7f97c5fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00cb5e0 con 0x7f97c00b4ba0 2023-03-03T08:03:54.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.216+0000 7f97c5fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00ca730 con 0x7f97c00b8880 2023-03-03T08:03:54.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.216+0000 7f97c5fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00ca960 con 0x7f97c00b5550 2023-03-03T08:03:54.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.216+0000 7f97bf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97c00b4ba0 0x7f97c015a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:54.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.217+0000 7f97bf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97c00b4ba0 0x7f97c015a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38066/0 (socket says 172.21.15.98:38066) 2023-03-03T08:03:54.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.217+0000 7f97bf7fe700 1 -- 172.21.15.98:0/2239435188 learned_addr learned my addr 172.21.15.98:0/2239435188 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:54.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.217+0000 7f97bffff700 1 --2- 172.21.15.98:0/2239435188 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97c00b8880 0x7f97c00c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:54.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.217+0000 7f97beffd700 1 --2- 172.21.15.98:0/2239435188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97c00b5550 0x7f97c00ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:54.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.217+0000 7f97bf7fe700 1 -- 172.21.15.98:0/2239435188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97c00b5550 msgr2=0x7f97c00ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:54.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.217+0000 7f97bf7fe700 1 --2- 172.21.15.98:0/2239435188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97c00b5550 0x7f97c00ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.217+0000 7f97bf7fe700 1 -- 172.21.15.98:0/2239435188 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97c00b8880 msgr2=0x7f97c00c8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:54.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.217+0000 7f97bf7fe700 1 --2- 172.21.15.98:0/2239435188 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97c00b8880 0x7f97c00c8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.217+0000 7f97bf7fe700 1 -- 172.21.15.98:0/2239435188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b0009550 con 0x7f97c00b4ba0 2023-03-03T08:03:54.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.217+0000 7f97bf7fe700 1 --2- 172.21.15.98:0/2239435188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97c00b4ba0 0x7f97c015a490 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f97b0000c00 tx=0x7f97b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:54.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.217+0000 7f97a7fff700 1 -- 172.21.15.98:0/2239435188 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b0004540 con 0x7f97c00b4ba0 2023-03-03T08:03:54.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.218+0000 7f97a7fff700 1 -- 172.21.15.98:0/2239435188 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b00036a0 con 0x7f97c00b4ba0 2023-03-03T08:03:54.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.218+0000 7f97a7fff700 1 -- 172.21.15.98:0/2239435188 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b0003a50 con 0x7f97c00b4ba0 2023-03-03T08:03:54.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.218+0000 7f97c5fff700 1 -- 172.21.15.98:0/2239435188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c00d0cf0 con 0x7f97c00b4ba0 2023-03-03T08:03:54.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.218+0000 7f97c5fff700 1 -- 172.21.15.98:0/2239435188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c00d0530 con 0x7f97c00b4ba0 2023-03-03T08:03:54.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.219+0000 7f97c5fff700 1 -- 172.21.15.98:0/2239435188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c00b8c50 con 0x7f97c00b4ba0 2023-03-03T08:03:54.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.220+0000 7f97a7fff700 1 -- 172.21.15.98:0/2239435188 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 86615+0+0 (secure 0 0 0) 0x7f97b000f050 con 0x7f97c00b4ba0 2023-03-03T08:03:54.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.221+0000 7f97a7fff700 1 --2- 172.21.15.98:0/2239435188 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f97ac068450 0x7f97ac06a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:54.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.221+0000 7f97a7fff700 1 -- 172.21.15.98:0/2239435188 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f97b0087780 con 0x7f97c00b4ba0 2023-03-03T08:03:54.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.222+0000 7f97beffd700 1 --2- 172.21.15.98:0/2239435188 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f97ac068450 0x7f97ac06a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:54.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.222+0000 7f97beffd700 1 --2- 172.21.15.98:0/2239435188 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f97ac068450 0x7f97ac06a900 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f97c00cd360 tx=0x7f97b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:54.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.225+0000 7f97a7fff700 1 -- 172.21.15.98:0/2239435188 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f97b0053e70 con 0x7f97c00b4ba0 2023-03-03T08:03:54.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.455+0000 7f97c5fff700 1 -- 172.21.15.98:0/2239435188 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000508309", "snap_name": "snapshot_0000000000029275", "target": ["mon-mgr", ""]}) v1 -- 0x7f97c004e6e0 con 0x7f97ac068450 2023-03-03T08:03:54.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.491+0000 7f97a7fff700 1 -- 172.21.15.98:0/2239435188 <== mgr.4107 v2:172.21.15.100:6836/108673 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f97c004e6e0 con 0x7f97ac068450 2023-03-03T08:03:54.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.494+0000 7f97c5fff700 1 -- 172.21.15.98:0/2239435188 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f97ac068450 msgr2=0x7f97ac06a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:54.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.494+0000 7f97c5fff700 1 --2- 172.21.15.98:0/2239435188 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f97ac068450 0x7f97ac06a900 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f97c00cd360 tx=0x7f97b4008040 comp rx=0 tx=0).stop 2023-03-03T08:03:54.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.494+0000 7f97c5fff700 1 -- 172.21.15.98:0/2239435188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97c00b4ba0 msgr2=0x7f97c015a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:54.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.494+0000 7f97c5fff700 1 --2- 172.21.15.98:0/2239435188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97c00b4ba0 0x7f97c015a490 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f97b0000c00 tx=0x7f97b0004750 comp rx=0 tx=0).stop 2023-03-03T08:03:54.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.494+0000 7f97c5fff700 1 -- 172.21.15.98:0/2239435188 shutdown_connections 2023-03-03T08:03:54.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.494+0000 7f97c5fff700 1 --2- 172.21.15.98:0/2239435188 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f97ac068450 0x7f97ac06a900 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.494+0000 7f97c5fff700 1 --2- 172.21.15.98:0/2239435188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97c00b4ba0 0x7f97c015a490 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.494+0000 7f97c5fff700 1 --2- 172.21.15.98:0/2239435188 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97c00b8880 0x7f97c00c8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.494+0000 7f97c5fff700 1 --2- 172.21.15.98:0/2239435188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97c00b5550 0x7f97c00ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.494+0000 7f97c5fff700 1 -- 172.21.15.98:0/2239435188 >> 172.21.15.98:0/2239435188 conn(0x7f97c006ad80 msgr2=0x7f97c00d2730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:54.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.494+0000 7f97c5fff700 1 -- 172.21.15.98:0/2239435188 shutdown_connections 2023-03-03T08:03:54.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.495+0000 7f97c5fff700 1 -- 172.21.15.98:0/2239435188 wait complete. 2023-03-03T08:03:54.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvolume_0000000000508309 fake 2023-03-03T08:03:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.775+0000 7fcbfc99d700 1 Processor -- start 2023-03-03T08:03:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.775+0000 7fcbfc99d700 1 -- start start 2023-03-03T08:03:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.776+0000 7fcbfc99d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbf80b3550 0x7fcbf80cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.776+0000 7fcbfc99d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf80b3c80 con 0x7fcbf80cee50 2023-03-03T08:03:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.776+0000 7fcbfc99d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf81615a0 con 0x7fcbf8073ed0 2023-03-03T08:03:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.776+0000 7fcbfc99d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf80b4370 con 0x7fcbf80b3550 2023-03-03T08:03:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.777+0000 7fcbf7fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcbf80cee50 0x7fcbf8160e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51860/0 (socket says 172.21.15.98:51860) 2023-03-03T08:03:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.777+0000 7fcbf7fff700 1 -- 172.21.15.98:0/3543169243 learned_addr learned my addr 172.21.15.98:0/3543169243 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:03:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.777+0000 7fcbf77fe700 1 --2- 172.21.15.98:0/3543169243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbf80b3550 0x7fcbf80cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.777+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659918518 0 0) 0x7fcbf80b3c80 con 0x7fcbf80cee50 2023-03-03T08:03:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.777+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe00035b0 con 0x7fcbf80cee50 2023-03-03T08:03:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.777+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360821325 0 0) 0x7fcbf81615a0 con 0x7fcbf8073ed0 2023-03-03T08:03:54.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf80b3c80 con 0x7fcbf8073ed0 2023-03-03T08:03:54.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1379746520 0 0) 0x7fcbe00035b0 con 0x7fcbf80cee50 2023-03-03T08:03:54.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbf80b3550 msgr2=0x7fcbf80cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:54.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf67fc700 1 --2- 172.21.15.98:0/3543169243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbf80b3550 0x7fcbf80cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 >> v1:172.21.15.100:6789/0 conn(0x7fcbf8073ed0 legacy=0x7fcbf8160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:54.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf80b3ed0 con 0x7fcbf80cee50 2023-03-03T08:03:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3408812025 0 0) 0x7fcbf00019f0 con 0x7fcbf80cee50 2023-03-03T08:03:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf77fe700 1 --2- 172.21.15.98:0/3543169243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbf80b3550 0x7fcbf80cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:03:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 >> v1:172.21.15.98:6789/0 conn(0x7fcbf80cee50 legacy=0x7fcbf8160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf67fc700 1 --2- 172.21.15.98:0/3543169243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbe0003bc0 0x7fcbe0003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf67fc700 1 --2- 172.21.15.98:0/3543169243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcbe0004850 0x7fcbe0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf67fc700 1 --2- 172.21.15.98:0/3543169243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbf80b3550 0x7fcbe0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf80b4370 con 0x7fcbe0003bc0 2023-03-03T08:03:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe000bac0 con 0x7fcbe0004850 2023-03-03T08:03:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe000bc50 con 0x7fcbf80b3550 2023-03-03T08:03:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.778+0000 7fcbf77fe700 1 --2- 172.21.15.98:0/3543169243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbe0003bc0 0x7fcbe0003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.779+0000 7fcbf77fe700 1 -- 172.21.15.98:0/3543169243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbf80b3550 msgr2=0x7fcbe0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.779+0000 7fcbf77fe700 1 --2- 172.21.15.98:0/3543169243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbf80b3550 0x7fcbe0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.779+0000 7fcbf77fe700 1 -- 172.21.15.98:0/3543169243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcbe0004850 msgr2=0x7fcbe0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.779+0000 7fcbf77fe700 1 --2- 172.21.15.98:0/3543169243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcbe0004850 0x7fcbe0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.779+0000 7fcbf77fe700 1 -- 172.21.15.98:0/3543169243 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe80094a0 con 0x7fcbe0003bc0 2023-03-03T08:03:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.779+0000 7fcbf77fe700 1 --2- 172.21.15.98:0/3543169243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbe0003bc0 0x7fcbe0003fb0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009d60 tx=0x7fcbe0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bb1aa93eba77607 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.779+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe800e9a0 con 0x7fcbe0003bc0 2023-03-03T08:03:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.779+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe800eb00 con 0x7fcbe0003bc0 2023-03-03T08:03:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.779+0000 7fcbf67fc700 1 -- 172.21.15.98:0/3543169243 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe800ee90 con 0x7fcbe0003bc0 2023-03-03T08:03:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.780+0000 7fcbfc99d700 1 -- 172.21.15.98:0/3543169243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbe0003bc0 msgr2=0x7fcbe0003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.780+0000 7fcbfc99d700 1 --2- 172.21.15.98:0/3543169243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbe0003bc0 0x7fcbe0003fb0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009d60 tx=0x7fcbe0005880 comp rx=0 tx=0).stop 2023-03-03T08:03:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.780+0000 7fcbf7fff700 1 -- 172.21.15.98:0/3543169243 reap_dead start 2023-03-03T08:03:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.780+0000 7fcbfc99d700 1 -- 172.21.15.98:0/3543169243 shutdown_connections 2023-03-03T08:03:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.780+0000 7fcbfc99d700 1 -- 172.21.15.98:0/3543169243 >> 172.21.15.98:0/3543169243 conn(0x7fcbf8073b00 msgr2=0x7fcbf80d2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.780+0000 7fcbfc99d700 1 -- 172.21.15.98:0/3543169243 shutdown_connections 2023-03-03T08:03:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.780+0000 7fcbfc99d700 1 -- 172.21.15.98:0/3543169243 wait complete. 2023-03-03T08:03:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.782+0000 7fcbfc99d700 1 Processor -- start 2023-03-03T08:03:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.782+0000 7fcbfc99d700 1 -- start start 2023-03-03T08:03:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.783+0000 7fcbfc99d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbe0003bc0 0x7fcbf80c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.783+0000 7fcbfc99d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcbf80b6370 0x7fcbf80b7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.783+0000 7fcbfc99d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbf80b7fd0 0x7fcbf8145f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.783+0000 7fcbfc99d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf80b8da0 con 0x7fcbf80b7fd0 2023-03-03T08:03:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.783+0000 7fcbfc99d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf80cb400 con 0x7fcbf80b6370 2023-03-03T08:03:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.783+0000 7fcbfc99d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf80b8500 con 0x7fcbe0003bc0 2023-03-03T08:03:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.783+0000 7fcbf7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbf80b7fd0 0x7fcbf8145f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.783+0000 7fcbf7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbf80b7fd0 0x7fcbf8145f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38098/0 (socket says 172.21.15.98:38098) 2023-03-03T08:03:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.783+0000 7fcbf7fff700 1 -- 172.21.15.98:0/915097959 learned_addr learned my addr 172.21.15.98:0/915097959 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:03:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.784+0000 7fcbf6ffd700 1 --2- 172.21.15.98:0/915097959 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcbf80b6370 0x7fcbf80b7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.784+0000 7fcbf77fe700 1 --2- 172.21.15.98:0/915097959 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbe0003bc0 0x7fcbf80c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.784+0000 7fcbf7fff700 1 -- 172.21.15.98:0/915097959 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbe0003bc0 msgr2=0x7fcbf80c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.784+0000 7fcbf7fff700 1 --2- 172.21.15.98:0/915097959 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbe0003bc0 0x7fcbf80c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.784+0000 7fcbf7fff700 1 -- 172.21.15.98:0/915097959 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcbf80b6370 msgr2=0x7fcbf80b7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.784+0000 7fcbf7fff700 1 --2- 172.21.15.98:0/915097959 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcbf80b6370 0x7fcbf80b7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.784+0000 7fcbf7fff700 1 -- 172.21.15.98:0/915097959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf80cd920 con 0x7fcbf80b7fd0 2023-03-03T08:03:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.784+0000 7fcbf77fe700 1 --2- 172.21.15.98:0/915097959 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbe0003bc0 0x7fcbf80c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:03:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.784+0000 7fcbf7fff700 1 --2- 172.21.15.98:0/915097959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbf80b7fd0 0x7fcbf8145f00 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fcbf808d750 tx=0x7fcbf000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.784+0000 7fcbdffff700 1 -- 172.21.15.98:0/915097959 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf000ab00 con 0x7fcbf80b7fd0 2023-03-03T08:03:54.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.784+0000 7fcbdffff700 1 -- 172.21.15.98:0/915097959 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf000ac60 con 0x7fcbf80b7fd0 2023-03-03T08:03:54.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.784+0000 7fcbdffff700 1 -- 172.21.15.98:0/915097959 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf00070d0 con 0x7fcbf80b7fd0 2023-03-03T08:03:54.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.784+0000 7fcbfc99d700 1 -- 172.21.15.98:0/915097959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf80cd640 con 0x7fcbf80b7fd0 2023-03-03T08:03:54.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.785+0000 7fcbfc99d700 1 -- 172.21.15.98:0/915097959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf80d3d60 con 0x7fcbf80b7fd0 2023-03-03T08:03:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.786+0000 7fcbfc99d700 1 -- 172.21.15.98:0/915097959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf8068290 con 0x7fcbf80b7fd0 2023-03-03T08:03:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.787+0000 7fcbdffff700 1 -- 172.21.15.98:0/915097959 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 86615+0+0 (secure 0 0 0) 0x7fcbf0010070 con 0x7fcbf80b7fd0 2023-03-03T08:03:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.788+0000 7fcbdffff700 1 --2- 172.21.15.98:0/915097959 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fcbe4068450 0x7fcbe406a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.788+0000 7fcbdffff700 1 -- 172.21.15.98:0/915097959 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fcbf00857d0 con 0x7fcbf80b7fd0 2023-03-03T08:03:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.789+0000 7fcbf77fe700 1 --2- 172.21.15.98:0/915097959 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fcbe4068450 0x7fcbe406a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.790+0000 7fcbf77fe700 1 --2- 172.21.15.98:0/915097959 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fcbe4068450 0x7fcbe406a900 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8000c00 tx=0x7fcbe8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:54.792+0000 7fcbdffff700 1 -- 172.21.15.98:0/915097959 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcbf0051ec0 con 0x7fcbf80b7fd0 2023-03-03T08:03:55.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:03:55.021+0000 7fcbfc99d700 1 -- 172.21.15.98:0/915097959 --> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000508309", "snap_name": "fake", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbf80d05a0 con 0x7fcbe4068450 2023-03-03T08:03:55.041 INFO:tasks.ceph.mds.d.smithi100.stderr:2023-03-03T08:03:55.040+0000 7f55075f6700 -1 mds.0.cache.den(0x10000000002 .meta.tmp) newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta.tmp [5,head] auth NULL (dn xlock x=1 by 0x56157da0d400) (dversion lock w=1 last_client=7876) pv=158 v=148 ap=2 ino=(nil) state=1610612736 | request=1 lock=2 inodepin=0 dirty=1 authpin=1 clientlease=0 0x56157d9d7680] 2023-03-03T08:03:55.041 INFO:tasks.ceph.mds.d.smithi100.stderr:2023-03-03T08:03:55.040+0000 7f55075f6700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta.tmp [5,head] auth NULL (dn xlock x=1 by 0x56157da0d400) (dversion lock w=1 last_client=7876) pv=158 v=148 ap=2 ino=(nil) state=1610612736 | request=1 lock=2 inodepin=0 dirty=1 authpin=1 clientlease=0 0x56157d9d7680] 2023-03-03T08:03:55.042 INFO:tasks.ceph.mds.d.smithi100.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7f55075f6700 time 2023-03-03T08:03:55.040951+0000 2023-03-03T08:03:55.042 INFO:tasks.ceph.mds.d.smithi100.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:03:55.043 INFO:tasks.ceph.mds.d.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:03:55.043 INFO:tasks.ceph.mds.d.smithi100.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f5517aa941b] 2023-03-03T08:03:55.043 INFO:tasks.ceph.mds.d.smithi100.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x56157ab21794] 2023-03-03T08:03:55.043 INFO:tasks.ceph.mds.d.smithi100.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x56157a97ada7] 2023-03-03T08:03:55.043 INFO:tasks.ceph.mds.d.smithi100.stderr: 4: (Server::handle_client_openc(boost::intrusive_ptr&)+0xab6) [0x56157a937fa6] 2023-03-03T08:03:55.043 INFO:tasks.ceph.mds.d.smithi100.stderr: 5: (Server::handle_client_request(boost::intrusive_ptr const&)+0x6d5) [0x56157a963a15] 2023-03-03T08:03:55.043 INFO:tasks.ceph.mds.d.smithi100.stderr: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xf4) [0x56157a968454] 2023-03-03T08:03:55.044 INFO:tasks.ceph.mds.d.smithi100.stderr: 7: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x56157a8b2a63] 2023-03-03T08:03:55.044 INFO:tasks.ceph.mds.d.smithi100.stderr: 8: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56157a8b307c] 2023-03-03T08:03:55.044 INFO:tasks.ceph.mds.d.smithi100.stderr: 9: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56157a89c6af] 2023-03-03T08:03:55.044 INFO:tasks.ceph.mds.d.smithi100.stderr: 10: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5517d208d8] 2023-03-03T08:03:55.044 INFO:tasks.ceph.mds.d.smithi100.stderr: 11: (DispatchQueue::entry()+0x50f) [0x7f5517d1da7f] 2023-03-03T08:03:55.044 INFO:tasks.ceph.mds.d.smithi100.stderr: 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5517de3171] 2023-03-03T08:03:55.044 INFO:tasks.ceph.mds.d.smithi100.stderr: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f5515e081ca] 2023-03-03T08:03:55.045 INFO:tasks.ceph.mds.d.smithi100.stderr: 14: clone() 2023-03-03T08:03:55.045 INFO:tasks.ceph.mds.d.smithi100.stderr:2023-03-03T08:03:55.041+0000 7f55075f6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7f55075f6700 time 2023-03-03T08:03:55.040951+0000 2023-03-03T08:03:55.045 INFO:tasks.ceph.mds.d.smithi100.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:03:55.045 INFO:tasks.ceph.mds.d.smithi100.stderr: 2023-03-03T08:03:55.045 INFO:tasks.ceph.mds.d.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:03:55.045 INFO:tasks.ceph.mds.d.smithi100.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f5517aa941b] 2023-03-03T08:03:55.045 INFO:tasks.ceph.mds.d.smithi100.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x56157ab21794] 2023-03-03T08:03:55.046 INFO:tasks.ceph.mds.d.smithi100.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x56157a97ada7] 2023-03-03T08:03:55.046 INFO:tasks.ceph.mds.d.smithi100.stderr: 4: (Server::handle_client_openc(boost::intrusive_ptr&)+0xab6) [0x56157a937fa6] 2023-03-03T08:03:55.046 INFO:tasks.ceph.mds.d.smithi100.stderr: 5: (Server::handle_client_request(boost::intrusive_ptr const&)+0x6d5) [0x56157a963a15] 2023-03-03T08:03:55.046 INFO:tasks.ceph.mds.d.smithi100.stderr: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xf4) [0x56157a968454] 2023-03-03T08:03:55.046 INFO:tasks.ceph.mds.d.smithi100.stderr: 7: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x56157a8b2a63] 2023-03-03T08:03:55.046 INFO:tasks.ceph.mds.d.smithi100.stderr: 8: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56157a8b307c] 2023-03-03T08:03:55.046 INFO:tasks.ceph.mds.d.smithi100.stderr: 9: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56157a89c6af] 2023-03-03T08:03:55.047 INFO:tasks.ceph.mds.d.smithi100.stderr: 10: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5517d208d8] 2023-03-03T08:03:55.047 INFO:tasks.ceph.mds.d.smithi100.stderr: 11: (DispatchQueue::entry()+0x50f) [0x7f5517d1da7f] 2023-03-03T08:03:55.047 INFO:tasks.ceph.mds.d.smithi100.stderr: 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5517de3171] 2023-03-03T08:03:55.047 INFO:tasks.ceph.mds.d.smithi100.stderr: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f5515e081ca] 2023-03-03T08:03:55.047 INFO:tasks.ceph.mds.d.smithi100.stderr: 14: clone() 2023-03-03T08:03:55.047 INFO:tasks.ceph.mds.d.smithi100.stderr: 2023-03-03T08:03:55.047 INFO:tasks.ceph.mds.d.smithi100.stderr:*** Caught signal (Aborted) ** 2023-03-03T08:03:55.048 INFO:tasks.ceph.mds.d.smithi100.stderr: in thread 7f55075f6700 thread_name:ms_dispatch 2023-03-03T08:03:55.048 INFO:tasks.ceph.mds.d.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:03:55.048 INFO:tasks.ceph.mds.d.smithi100.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f5515e12cf0] 2023-03-03T08:03:55.048 INFO:tasks.ceph.mds.d.smithi100.stderr: 2: gsignal() 2023-03-03T08:03:55.049 INFO:tasks.ceph.mds.d.smithi100.stderr: 3: abort() 2023-03-03T08:03:55.049 INFO:tasks.ceph.mds.d.smithi100.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7f5517aa94db] 2023-03-03T08:03:55.049 INFO:tasks.ceph.mds.d.smithi100.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x56157ab21794] 2023-03-03T08:03:55.049 INFO:tasks.ceph.mds.d.smithi100.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x56157a97ada7] 2023-03-03T08:03:55.049 INFO:tasks.ceph.mds.d.smithi100.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xab6) [0x56157a937fa6] 2023-03-03T08:03:55.049 INFO:tasks.ceph.mds.d.smithi100.stderr: 8: (Server::handle_client_request(boost::intrusive_ptr const&)+0x6d5) [0x56157a963a15] 2023-03-03T08:03:55.050 INFO:tasks.ceph.mds.d.smithi100.stderr: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xf4) [0x56157a968454] 2023-03-03T08:03:55.050 INFO:tasks.ceph.mds.d.smithi100.stderr: 10: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x56157a8b2a63] 2023-03-03T08:03:55.050 INFO:tasks.ceph.mds.d.smithi100.stderr: 11: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56157a8b307c] 2023-03-03T08:03:55.050 INFO:tasks.ceph.mds.d.smithi100.stderr: 12: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56157a89c6af] 2023-03-03T08:03:55.050 INFO:tasks.ceph.mds.d.smithi100.stderr: 13: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5517d208d8] 2023-03-03T08:03:55.050 INFO:tasks.ceph.mds.d.smithi100.stderr: 14: (DispatchQueue::entry()+0x50f) [0x7f5517d1da7f] 2023-03-03T08:03:55.050 INFO:tasks.ceph.mds.d.smithi100.stderr: 15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5517de3171] 2023-03-03T08:03:55.051 INFO:tasks.ceph.mds.d.smithi100.stderr: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f5515e081ca] 2023-03-03T08:03:55.051 INFO:tasks.ceph.mds.d.smithi100.stderr: 17: clone() 2023-03-03T08:03:55.051 INFO:tasks.ceph.mds.d.smithi100.stderr:2023-03-03T08:03:55.043+0000 7f55075f6700 -1 *** Caught signal (Aborted) ** 2023-03-03T08:03:55.051 INFO:tasks.ceph.mds.d.smithi100.stderr: in thread 7f55075f6700 thread_name:ms_dispatch 2023-03-03T08:03:55.051 INFO:tasks.ceph.mds.d.smithi100.stderr: 2023-03-03T08:03:55.051 INFO:tasks.ceph.mds.d.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:03:55.051 INFO:tasks.ceph.mds.d.smithi100.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f5515e12cf0] 2023-03-03T08:03:55.052 INFO:tasks.ceph.mds.d.smithi100.stderr: 2: gsignal() 2023-03-03T08:03:55.052 INFO:tasks.ceph.mds.d.smithi100.stderr: 3: abort() 2023-03-03T08:03:55.052 INFO:tasks.ceph.mds.d.smithi100.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7f5517aa94db] 2023-03-03T08:03:55.052 INFO:tasks.ceph.mds.d.smithi100.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x56157ab21794] 2023-03-03T08:03:55.052 INFO:tasks.ceph.mds.d.smithi100.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x56157a97ada7] 2023-03-03T08:03:55.052 INFO:tasks.ceph.mds.d.smithi100.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xab6) [0x56157a937fa6] 2023-03-03T08:03:55.052 INFO:tasks.ceph.mds.d.smithi100.stderr: 8: (Server::handle_client_request(boost::intrusive_ptr const&)+0x6d5) [0x56157a963a15] 2023-03-03T08:03:55.053 INFO:tasks.ceph.mds.d.smithi100.stderr: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xf4) [0x56157a968454] 2023-03-03T08:03:55.053 INFO:tasks.ceph.mds.d.smithi100.stderr: 10: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x56157a8b2a63] 2023-03-03T08:03:55.053 INFO:tasks.ceph.mds.d.smithi100.stderr: 11: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56157a8b307c] 2023-03-03T08:03:55.053 INFO:tasks.ceph.mds.d.smithi100.stderr: 12: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56157a89c6af] 2023-03-03T08:03:55.053 INFO:tasks.ceph.mds.d.smithi100.stderr: 13: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5517d208d8] 2023-03-03T08:03:55.053 INFO:tasks.ceph.mds.d.smithi100.stderr: 14: (DispatchQueue::entry()+0x50f) [0x7f5517d1da7f] 2023-03-03T08:03:55.054 INFO:tasks.ceph.mds.d.smithi100.stderr: 15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5517de3171] 2023-03-03T08:03:55.054 INFO:tasks.ceph.mds.d.smithi100.stderr: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f5515e081ca] 2023-03-03T08:03:55.054 INFO:tasks.ceph.mds.d.smithi100.stderr: 17: clone() 2023-03-03T08:03:55.054 INFO:tasks.ceph.mds.d.smithi100.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-03T08:03:55.054 INFO:tasks.ceph.mds.d.smithi100.stderr: 2023-03-03T08:03:55.080 INFO:tasks.ceph.mds.d.smithi100.stderr: -4> 2023-03-03T08:03:55.040+0000 7f55075f6700 -1 mds.0.cache.den(0x10000000002 .meta.tmp) newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta.tmp [5,head] auth NULL (dn xlock x=1 by 0x56157da0d400) (dversion lock w=1 last_client=7876) pv=158 v=148 ap=2 ino=(nil) state=1610612736 | request=1 lock=2 inodepin=0 dirty=1 authpin=1 clientlease=0 0x56157d9d7680] 2023-03-03T08:03:55.081 INFO:tasks.ceph.mds.d.smithi100.stderr: -2> 2023-03-03T08:03:55.040+0000 7f55075f6700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta.tmp [5,head] auth NULL (dn xlock x=1 by 0x56157da0d400) (dversion lock w=1 last_client=7876) pv=158 v=148 ap=2 ino=(nil) state=1610612736 | request=1 lock=2 inodepin=0 dirty=1 authpin=1 clientlease=0 0x56157d9d7680] 2023-03-03T08:03:55.081 INFO:tasks.ceph.mds.d.smithi100.stderr: -1> 2023-03-03T08:03:55.041+0000 7f55075f6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7f55075f6700 time 2023-03-03T08:03:55.040951+0000 2023-03-03T08:03:55.081 INFO:tasks.ceph.mds.d.smithi100.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:03:55.081 INFO:tasks.ceph.mds.d.smithi100.stderr: 2023-03-03T08:03:55.081 INFO:tasks.ceph.mds.d.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:03:55.081 INFO:tasks.ceph.mds.d.smithi100.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f5517aa941b] 2023-03-03T08:03:55.082 INFO:tasks.ceph.mds.d.smithi100.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x56157ab21794] 2023-03-03T08:03:55.082 INFO:tasks.ceph.mds.d.smithi100.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x56157a97ada7] 2023-03-03T08:03:55.082 INFO:tasks.ceph.mds.d.smithi100.stderr: 4: (Server::handle_client_openc(boost::intrusive_ptr&)+0xab6) [0x56157a937fa6] 2023-03-03T08:03:55.082 INFO:tasks.ceph.mds.d.smithi100.stderr: 5: (Server::handle_client_request(boost::intrusive_ptr const&)+0x6d5) [0x56157a963a15] 2023-03-03T08:03:55.082 INFO:tasks.ceph.mds.d.smithi100.stderr: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xf4) [0x56157a968454] 2023-03-03T08:03:55.082 INFO:tasks.ceph.mds.d.smithi100.stderr: 7: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x56157a8b2a63] 2023-03-03T08:03:55.082 INFO:tasks.ceph.mds.d.smithi100.stderr: 8: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56157a8b307c] 2023-03-03T08:03:55.083 INFO:tasks.ceph.mds.d.smithi100.stderr: 9: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56157a89c6af] 2023-03-03T08:03:55.083 INFO:tasks.ceph.mds.d.smithi100.stderr: 10: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5517d208d8] 2023-03-03T08:03:55.083 INFO:tasks.ceph.mds.d.smithi100.stderr: 11: (DispatchQueue::entry()+0x50f) [0x7f5517d1da7f] 2023-03-03T08:03:55.083 INFO:tasks.ceph.mds.d.smithi100.stderr: 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5517de3171] 2023-03-03T08:03:55.083 INFO:tasks.ceph.mds.d.smithi100.stderr: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f5515e081ca] 2023-03-03T08:03:55.083 INFO:tasks.ceph.mds.d.smithi100.stderr: 14: clone() 2023-03-03T08:03:55.083 INFO:tasks.ceph.mds.d.smithi100.stderr: 2023-03-03T08:03:55.084 INFO:tasks.ceph.mds.d.smithi100.stderr: 0> 2023-03-03T08:03:55.043+0000 7f55075f6700 -1 *** Caught signal (Aborted) ** 2023-03-03T08:03:55.084 INFO:tasks.ceph.mds.d.smithi100.stderr: in thread 7f55075f6700 thread_name:ms_dispatch 2023-03-03T08:03:55.084 INFO:tasks.ceph.mds.d.smithi100.stderr: 2023-03-03T08:03:55.084 INFO:tasks.ceph.mds.d.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:03:55.084 INFO:tasks.ceph.mds.d.smithi100.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f5515e12cf0] 2023-03-03T08:03:55.084 INFO:tasks.ceph.mds.d.smithi100.stderr: 2: gsignal() 2023-03-03T08:03:55.084 INFO:tasks.ceph.mds.d.smithi100.stderr: 3: abort() 2023-03-03T08:03:55.085 INFO:tasks.ceph.mds.d.smithi100.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7f5517aa94db] 2023-03-03T08:03:55.085 INFO:tasks.ceph.mds.d.smithi100.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x56157ab21794] 2023-03-03T08:03:55.085 INFO:tasks.ceph.mds.d.smithi100.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x56157a97ada7] 2023-03-03T08:03:55.085 INFO:tasks.ceph.mds.d.smithi100.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xab6) [0x56157a937fa6] 2023-03-03T08:03:55.085 INFO:tasks.ceph.mds.d.smithi100.stderr: 8: (Server::handle_client_request(boost::intrusive_ptr const&)+0x6d5) [0x56157a963a15] 2023-03-03T08:03:55.085 INFO:tasks.ceph.mds.d.smithi100.stderr: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xf4) [0x56157a968454] 2023-03-03T08:03:55.085 INFO:tasks.ceph.mds.d.smithi100.stderr: 10: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x56157a8b2a63] 2023-03-03T08:03:55.086 INFO:tasks.ceph.mds.d.smithi100.stderr: 11: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56157a8b307c] 2023-03-03T08:03:55.086 INFO:tasks.ceph.mds.d.smithi100.stderr: 12: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56157a89c6af] 2023-03-03T08:03:55.086 INFO:tasks.ceph.mds.d.smithi100.stderr: 13: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5517d208d8] 2023-03-03T08:03:55.086 INFO:tasks.ceph.mds.d.smithi100.stderr: 14: (DispatchQueue::entry()+0x50f) [0x7f5517d1da7f] 2023-03-03T08:03:55.086 INFO:tasks.ceph.mds.d.smithi100.stderr: 15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5517de3171] 2023-03-03T08:03:55.086 INFO:tasks.ceph.mds.d.smithi100.stderr: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f5515e081ca] 2023-03-03T08:03:55.086 INFO:tasks.ceph.mds.d.smithi100.stderr: 17: clone() 2023-03-03T08:03:55.087 INFO:tasks.ceph.mds.d.smithi100.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-03T08:03:55.087 INFO:tasks.ceph.mds.d.smithi100.stderr: 2023-03-03T08:03:55.098 INFO:tasks.ceph.mds.d.smithi100.stderr: -4> 2023-03-03T08:03:55.040+0000 7f55075f6700 -1 mds.0.cache.den(0x10000000002 .meta.tmp) newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta.tmp [5,head] auth NULL (dn xlock x=1 by 0x56157da0d400) (dversion lock w=1 last_client=7876) pv=158 v=148 ap=2 ino=(nil) state=1610612736 | request=1 lock=2 inodepin=0 dirty=1 authpin=1 clientlease=0 0x56157d9d7680] 2023-03-03T08:03:55.098 INFO:tasks.ceph.mds.d.smithi100.stderr: -2> 2023-03-03T08:03:55.040+0000 7f55075f6700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta.tmp [5,head] auth NULL (dn xlock x=1 by 0x56157da0d400) (dversion lock w=1 last_client=7876) pv=158 v=148 ap=2 ino=(nil) state=1610612736 | request=1 lock=2 inodepin=0 dirty=1 authpin=1 clientlease=0 0x56157d9d7680] 2023-03-03T08:03:55.098 INFO:tasks.ceph.mds.d.smithi100.stderr: -1> 2023-03-03T08:03:55.041+0000 7f55075f6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7f55075f6700 time 2023-03-03T08:03:55.040951+0000 2023-03-03T08:03:55.099 INFO:tasks.ceph.mds.d.smithi100.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:03:55.099 INFO:tasks.ceph.mds.d.smithi100.stderr: 2023-03-03T08:03:55.099 INFO:tasks.ceph.mds.d.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:03:55.099 INFO:tasks.ceph.mds.d.smithi100.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f5517aa941b] 2023-03-03T08:03:55.099 INFO:tasks.ceph.mds.d.smithi100.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x56157ab21794] 2023-03-03T08:03:55.099 INFO:tasks.ceph.mds.d.smithi100.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x56157a97ada7] 2023-03-03T08:03:55.099 INFO:tasks.ceph.mds.d.smithi100.stderr: 4: (Server::handle_client_openc(boost::intrusive_ptr&)+0xab6) [0x56157a937fa6] 2023-03-03T08:03:55.100 INFO:tasks.ceph.mds.d.smithi100.stderr: 5: (Server::handle_client_request(boost::intrusive_ptr const&)+0x6d5) [0x56157a963a15] 2023-03-03T08:03:55.100 INFO:tasks.ceph.mds.d.smithi100.stderr: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xf4) [0x56157a968454] 2023-03-03T08:03:55.100 INFO:tasks.ceph.mds.d.smithi100.stderr: 7: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x56157a8b2a63] 2023-03-03T08:03:55.100 INFO:tasks.ceph.mds.d.smithi100.stderr: 8: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56157a8b307c] 2023-03-03T08:03:55.100 INFO:tasks.ceph.mds.d.smithi100.stderr: 9: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56157a89c6af] 2023-03-03T08:03:55.100 INFO:tasks.ceph.mds.d.smithi100.stderr: 10: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5517d208d8] 2023-03-03T08:03:55.100 INFO:tasks.ceph.mds.d.smithi100.stderr: 11: (DispatchQueue::entry()+0x50f) [0x7f5517d1da7f] 2023-03-03T08:03:55.101 INFO:tasks.ceph.mds.d.smithi100.stderr: 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5517de3171] 2023-03-03T08:03:55.101 INFO:tasks.ceph.mds.d.smithi100.stderr: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f5515e081ca] 2023-03-03T08:03:55.101 INFO:tasks.ceph.mds.d.smithi100.stderr: 14: clone() 2023-03-03T08:03:55.101 INFO:tasks.ceph.mds.d.smithi100.stderr: 2023-03-03T08:03:55.101 INFO:tasks.ceph.mds.d.smithi100.stderr: 0> 2023-03-03T08:03:55.043+0000 7f55075f6700 -1 *** Caught signal (Aborted) ** 2023-03-03T08:03:55.101 INFO:tasks.ceph.mds.d.smithi100.stderr: in thread 7f55075f6700 thread_name:ms_dispatch 2023-03-03T08:03:55.102 INFO:tasks.ceph.mds.d.smithi100.stderr: 2023-03-03T08:03:55.102 INFO:tasks.ceph.mds.d.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:03:55.102 INFO:tasks.ceph.mds.d.smithi100.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f5515e12cf0] 2023-03-03T08:03:55.102 INFO:tasks.ceph.mds.d.smithi100.stderr: 2: gsignal() 2023-03-03T08:03:55.103 INFO:tasks.ceph.mds.d.smithi100.stderr: 3: abort() 2023-03-03T08:03:55.103 INFO:tasks.ceph.mds.d.smithi100.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7f5517aa94db] 2023-03-03T08:03:55.103 INFO:tasks.ceph.mds.d.smithi100.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x56157ab21794] 2023-03-03T08:03:55.103 INFO:tasks.ceph.mds.d.smithi100.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x56157a97ada7] 2023-03-03T08:03:55.103 INFO:tasks.ceph.mds.d.smithi100.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xab6) [0x56157a937fa6] 2023-03-03T08:03:55.104 INFO:tasks.ceph.mds.d.smithi100.stderr: 8: (Server::handle_client_request(boost::intrusive_ptr const&)+0x6d5) [0x56157a963a15] 2023-03-03T08:03:55.104 INFO:tasks.ceph.mds.d.smithi100.stderr: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xf4) [0x56157a968454] 2023-03-03T08:03:55.104 INFO:tasks.ceph.mds.d.smithi100.stderr: 10: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x56157a8b2a63] 2023-03-03T08:03:55.104 INFO:tasks.ceph.mds.d.smithi100.stderr: 11: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56157a8b307c] 2023-03-03T08:03:55.104 INFO:tasks.ceph.mds.d.smithi100.stderr: 12: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56157a89c6af] 2023-03-03T08:03:55.105 INFO:tasks.ceph.mds.d.smithi100.stderr: 13: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5517d208d8] 2023-03-03T08:03:55.105 INFO:tasks.ceph.mds.d.smithi100.stderr: 14: (DispatchQueue::entry()+0x50f) [0x7f5517d1da7f] 2023-03-03T08:03:55.105 INFO:tasks.ceph.mds.d.smithi100.stderr: 15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5517de3171] 2023-03-03T08:03:55.105 INFO:tasks.ceph.mds.d.smithi100.stderr: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f5515e081ca] 2023-03-03T08:03:55.105 INFO:tasks.ceph.mds.d.smithi100.stderr: 17: clone() 2023-03-03T08:03:55.106 INFO:tasks.ceph.mds.d.smithi100.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-03T08:03:55.106 INFO:tasks.ceph.mds.d.smithi100.stderr: 2023-03-03T08:03:55.526 INFO:tasks.ceph.mds.d.smithi100.stderr:daemon-helper: command crashed with signal 6 2023-03-03T08:03:57.906 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-03-03T08:04:04.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:04:04.310 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:04:04.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~7s 2023-03-03T08:04:11.129 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~13s 2023-03-03T08:04:15.594 INFO:tasks.ceph.mds.a.smithi098.stderr:2023-03-03T08:04:15.592+0000 7f8c395ea700 -1 mds.0.cache.den(0x10000000002 .meta) newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x55f8653a0f00] 2023-03-03T08:04:15.594 INFO:tasks.ceph.mds.a.smithi098.stderr:2023-03-03T08:04:15.592+0000 7f8c395ea700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x55f8653a0f00] 2023-03-03T08:04:15.595 INFO:tasks.ceph.mds.a.smithi098.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7f8c395ea700 time 2023-03-03T08:04:15.593561+0000 2023-03-03T08:04:15.595 INFO:tasks.ceph.mds.a.smithi098.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:04:15.595 INFO:tasks.ceph.mds.a.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:15.595 INFO:tasks.ceph.mds.a.smithi098.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f8c4faa941b] 2023-03-03T08:04:15.596 INFO:tasks.ceph.mds.a.smithi098.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x55f861c15794] 2023-03-03T08:04:15.596 INFO:tasks.ceph.mds.a.smithi098.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f861a6eda7] 2023-03-03T08:04:15.596 INFO:tasks.ceph.mds.a.smithi098.stderr: 4: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f861ba2c95] 2023-03-03T08:04:15.596 INFO:tasks.ceph.mds.a.smithi098.stderr: 5: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f861b70c70] 2023-03-03T08:04:15.596 INFO:tasks.ceph.mds.a.smithi098.stderr: 6: (MDSContext::complete(int)+0x5f) [0x55f861cf22ff] 2023-03-03T08:04:15.596 INFO:tasks.ceph.mds.a.smithi098.stderr: 7: (MDSIOContextBase::complete(int)+0x524) [0x55f861cf2a84] 2023-03-03T08:04:15.596 INFO:tasks.ceph.mds.a.smithi098.stderr: 8: (Filer::C_Probe::finish(int)+0xbb) [0x55f861db0fab] 2023-03-03T08:04:15.597 INFO:tasks.ceph.mds.a.smithi098.stderr: 9: (Context::complete(int)+0xd) [0x55f86199213d] 2023-03-03T08:04:15.597 INFO:tasks.ceph.mds.a.smithi098.stderr: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7f8c4fb4c33d] 2023-03-03T08:04:15.597 INFO:tasks.ceph.mds.a.smithi098.stderr: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f8c4de081ca] 2023-03-03T08:04:15.597 INFO:tasks.ceph.mds.a.smithi098.stderr: 12: clone() 2023-03-03T08:04:15.597 INFO:tasks.ceph.mds.a.smithi098.stderr:2023-03-03T08:04:15.594+0000 7f8c395ea700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7f8c395ea700 time 2023-03-03T08:04:15.593561+0000 2023-03-03T08:04:15.597 INFO:tasks.ceph.mds.a.smithi098.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:04:15.597 INFO:tasks.ceph.mds.a.smithi098.stderr: 2023-03-03T08:04:15.598 INFO:tasks.ceph.mds.a.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:15.598 INFO:tasks.ceph.mds.a.smithi098.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f8c4faa941b] 2023-03-03T08:04:15.598 INFO:tasks.ceph.mds.a.smithi098.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x55f861c15794] 2023-03-03T08:04:15.598 INFO:tasks.ceph.mds.a.smithi098.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f861a6eda7] 2023-03-03T08:04:15.598 INFO:tasks.ceph.mds.a.smithi098.stderr: 4: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f861ba2c95] 2023-03-03T08:04:15.598 INFO:tasks.ceph.mds.a.smithi098.stderr: 5: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f861b70c70] 2023-03-03T08:04:15.598 INFO:tasks.ceph.mds.a.smithi098.stderr: 6: (MDSContext::complete(int)+0x5f) [0x55f861cf22ff] 2023-03-03T08:04:15.598 INFO:tasks.ceph.mds.a.smithi098.stderr: 7: (MDSIOContextBase::complete(int)+0x524) [0x55f861cf2a84] 2023-03-03T08:04:15.599 INFO:tasks.ceph.mds.a.smithi098.stderr: 8: (Filer::C_Probe::finish(int)+0xbb) [0x55f861db0fab] 2023-03-03T08:04:15.599 INFO:tasks.ceph.mds.a.smithi098.stderr: 9: (Context::complete(int)+0xd) [0x55f86199213d] 2023-03-03T08:04:15.599 INFO:tasks.ceph.mds.a.smithi098.stderr: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7f8c4fb4c33d] 2023-03-03T08:04:15.599 INFO:tasks.ceph.mds.a.smithi098.stderr: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f8c4de081ca] 2023-03-03T08:04:15.599 INFO:tasks.ceph.mds.a.smithi098.stderr: 12: clone() 2023-03-03T08:04:15.599 INFO:tasks.ceph.mds.a.smithi098.stderr: 2023-03-03T08:04:15.599 INFO:tasks.ceph.mds.a.smithi098.stderr:*** Caught signal (Aborted) ** 2023-03-03T08:04:15.600 INFO:tasks.ceph.mds.a.smithi098.stderr: in thread 7f8c395ea700 thread_name:MR_Finisher 2023-03-03T08:04:15.600 INFO:tasks.ceph.mds.a.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:15.600 INFO:tasks.ceph.mds.a.smithi098.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8c4de12cf0] 2023-03-03T08:04:15.600 INFO:tasks.ceph.mds.a.smithi098.stderr: 2: gsignal() 2023-03-03T08:04:15.600 INFO:tasks.ceph.mds.a.smithi098.stderr: 3: abort() 2023-03-03T08:04:15.601 INFO:tasks.ceph.mds.a.smithi098.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7f8c4faa94db] 2023-03-03T08:04:15.601 INFO:tasks.ceph.mds.a.smithi098.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x55f861c15794] 2023-03-03T08:04:15.601 INFO:tasks.ceph.mds.a.smithi098.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f861a6eda7] 2023-03-03T08:04:15.601 INFO:tasks.ceph.mds.a.smithi098.stderr: 7: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f861ba2c95] 2023-03-03T08:04:15.601 INFO:tasks.ceph.mds.a.smithi098.stderr: 8: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f861b70c70] 2023-03-03T08:04:15.601 INFO:tasks.ceph.mds.a.smithi098.stderr: 9: (MDSContext::complete(int)+0x5f) [0x55f861cf22ff] 2023-03-03T08:04:15.601 INFO:tasks.ceph.mds.a.smithi098.stderr: 10: (MDSIOContextBase::complete(int)+0x524) [0x55f861cf2a84] 2023-03-03T08:04:15.602 INFO:tasks.ceph.mds.a.smithi098.stderr: 11: (Filer::C_Probe::finish(int)+0xbb) [0x55f861db0fab] 2023-03-03T08:04:15.602 INFO:tasks.ceph.mds.a.smithi098.stderr: 12: (Context::complete(int)+0xd) [0x55f86199213d] 2023-03-03T08:04:15.602 INFO:tasks.ceph.mds.a.smithi098.stderr: 13: (Finisher::finisher_thread_entry()+0x18d) [0x7f8c4fb4c33d] 2023-03-03T08:04:15.602 INFO:tasks.ceph.mds.a.smithi098.stderr: 14: /lib64/libpthread.so.0(+0x81ca) [0x7f8c4de081ca] 2023-03-03T08:04:15.602 INFO:tasks.ceph.mds.a.smithi098.stderr: 15: clone() 2023-03-03T08:04:15.602 INFO:tasks.ceph.mds.a.smithi098.stderr:2023-03-03T08:04:15.596+0000 7f8c395ea700 -1 *** Caught signal (Aborted) ** 2023-03-03T08:04:15.602 INFO:tasks.ceph.mds.a.smithi098.stderr: in thread 7f8c395ea700 thread_name:MR_Finisher 2023-03-03T08:04:15.603 INFO:tasks.ceph.mds.a.smithi098.stderr: 2023-03-03T08:04:15.603 INFO:tasks.ceph.mds.a.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:15.603 INFO:tasks.ceph.mds.a.smithi098.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8c4de12cf0] 2023-03-03T08:04:15.603 INFO:tasks.ceph.mds.a.smithi098.stderr: 2: gsignal() 2023-03-03T08:04:15.603 INFO:tasks.ceph.mds.a.smithi098.stderr: 3: abort() 2023-03-03T08:04:15.603 INFO:tasks.ceph.mds.a.smithi098.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7f8c4faa94db] 2023-03-03T08:04:15.603 INFO:tasks.ceph.mds.a.smithi098.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x55f861c15794] 2023-03-03T08:04:15.604 INFO:tasks.ceph.mds.a.smithi098.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f861a6eda7] 2023-03-03T08:04:15.604 INFO:tasks.ceph.mds.a.smithi098.stderr: 7: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f861ba2c95] 2023-03-03T08:04:15.604 INFO:tasks.ceph.mds.a.smithi098.stderr: 8: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f861b70c70] 2023-03-03T08:04:15.604 INFO:tasks.ceph.mds.a.smithi098.stderr: 9: (MDSContext::complete(int)+0x5f) [0x55f861cf22ff] 2023-03-03T08:04:15.604 INFO:tasks.ceph.mds.a.smithi098.stderr: 10: (MDSIOContextBase::complete(int)+0x524) [0x55f861cf2a84] 2023-03-03T08:04:15.604 INFO:tasks.ceph.mds.a.smithi098.stderr: 11: (Filer::C_Probe::finish(int)+0xbb) [0x55f861db0fab] 2023-03-03T08:04:15.604 INFO:tasks.ceph.mds.a.smithi098.stderr: 12: (Context::complete(int)+0xd) [0x55f86199213d] 2023-03-03T08:04:15.604 INFO:tasks.ceph.mds.a.smithi098.stderr: 13: (Finisher::finisher_thread_entry()+0x18d) [0x7f8c4fb4c33d] 2023-03-03T08:04:15.605 INFO:tasks.ceph.mds.a.smithi098.stderr: 14: /lib64/libpthread.so.0(+0x81ca) [0x7f8c4de081ca] 2023-03-03T08:04:15.605 INFO:tasks.ceph.mds.a.smithi098.stderr: 15: clone() 2023-03-03T08:04:15.605 INFO:tasks.ceph.mds.a.smithi098.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-03T08:04:15.605 INFO:tasks.ceph.mds.a.smithi098.stderr: 2023-03-03T08:04:15.625 INFO:tasks.ceph.mds.a.smithi098.stderr: -4> 2023-03-03T08:04:15.592+0000 7f8c395ea700 -1 mds.0.cache.den(0x10000000002 .meta) newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x55f8653a0f00] 2023-03-03T08:04:15.625 INFO:tasks.ceph.mds.a.smithi098.stderr: -2> 2023-03-03T08:04:15.592+0000 7f8c395ea700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x55f8653a0f00] 2023-03-03T08:04:15.625 INFO:tasks.ceph.mds.a.smithi098.stderr: -1> 2023-03-03T08:04:15.594+0000 7f8c395ea700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7f8c395ea700 time 2023-03-03T08:04:15.593561+0000 2023-03-03T08:04:15.625 INFO:tasks.ceph.mds.a.smithi098.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:04:15.625 INFO:tasks.ceph.mds.a.smithi098.stderr: 2023-03-03T08:04:15.626 INFO:tasks.ceph.mds.a.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:15.626 INFO:tasks.ceph.mds.a.smithi098.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f8c4faa941b] 2023-03-03T08:04:15.626 INFO:tasks.ceph.mds.a.smithi098.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x55f861c15794] 2023-03-03T08:04:15.626 INFO:tasks.ceph.mds.a.smithi098.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f861a6eda7] 2023-03-03T08:04:15.626 INFO:tasks.ceph.mds.a.smithi098.stderr: 4: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f861ba2c95] 2023-03-03T08:04:15.626 INFO:tasks.ceph.mds.a.smithi098.stderr: 5: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f861b70c70] 2023-03-03T08:04:15.626 INFO:tasks.ceph.mds.a.smithi098.stderr: 6: (MDSContext::complete(int)+0x5f) [0x55f861cf22ff] 2023-03-03T08:04:15.627 INFO:tasks.ceph.mds.a.smithi098.stderr: 7: (MDSIOContextBase::complete(int)+0x524) [0x55f861cf2a84] 2023-03-03T08:04:15.627 INFO:tasks.ceph.mds.a.smithi098.stderr: 8: (Filer::C_Probe::finish(int)+0xbb) [0x55f861db0fab] 2023-03-03T08:04:15.627 INFO:tasks.ceph.mds.a.smithi098.stderr: 9: (Context::complete(int)+0xd) [0x55f86199213d] 2023-03-03T08:04:15.627 INFO:tasks.ceph.mds.a.smithi098.stderr: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7f8c4fb4c33d] 2023-03-03T08:04:15.627 INFO:tasks.ceph.mds.a.smithi098.stderr: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f8c4de081ca] 2023-03-03T08:04:15.627 INFO:tasks.ceph.mds.a.smithi098.stderr: 12: clone() 2023-03-03T08:04:15.627 INFO:tasks.ceph.mds.a.smithi098.stderr: 2023-03-03T08:04:15.627 INFO:tasks.ceph.mds.a.smithi098.stderr: 0> 2023-03-03T08:04:15.596+0000 7f8c395ea700 -1 *** Caught signal (Aborted) ** 2023-03-03T08:04:15.628 INFO:tasks.ceph.mds.a.smithi098.stderr: in thread 7f8c395ea700 thread_name:MR_Finisher 2023-03-03T08:04:15.628 INFO:tasks.ceph.mds.a.smithi098.stderr: 2023-03-03T08:04:15.628 INFO:tasks.ceph.mds.a.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:15.628 INFO:tasks.ceph.mds.a.smithi098.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8c4de12cf0] 2023-03-03T08:04:15.628 INFO:tasks.ceph.mds.a.smithi098.stderr: 2: gsignal() 2023-03-03T08:04:15.628 INFO:tasks.ceph.mds.a.smithi098.stderr: 3: abort() 2023-03-03T08:04:15.628 INFO:tasks.ceph.mds.a.smithi098.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7f8c4faa94db] 2023-03-03T08:04:15.629 INFO:tasks.ceph.mds.a.smithi098.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x55f861c15794] 2023-03-03T08:04:15.629 INFO:tasks.ceph.mds.a.smithi098.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f861a6eda7] 2023-03-03T08:04:15.629 INFO:tasks.ceph.mds.a.smithi098.stderr: 7: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f861ba2c95] 2023-03-03T08:04:15.629 INFO:tasks.ceph.mds.a.smithi098.stderr: 8: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f861b70c70] 2023-03-03T08:04:15.629 INFO:tasks.ceph.mds.a.smithi098.stderr: 9: (MDSContext::complete(int)+0x5f) [0x55f861cf22ff] 2023-03-03T08:04:15.629 INFO:tasks.ceph.mds.a.smithi098.stderr: 10: (MDSIOContextBase::complete(int)+0x524) [0x55f861cf2a84] 2023-03-03T08:04:15.629 INFO:tasks.ceph.mds.a.smithi098.stderr: 11: (Filer::C_Probe::finish(int)+0xbb) [0x55f861db0fab] 2023-03-03T08:04:15.629 INFO:tasks.ceph.mds.a.smithi098.stderr: 12: (Context::complete(int)+0xd) [0x55f86199213d] 2023-03-03T08:04:15.630 INFO:tasks.ceph.mds.a.smithi098.stderr: 13: (Finisher::finisher_thread_entry()+0x18d) [0x7f8c4fb4c33d] 2023-03-03T08:04:15.630 INFO:tasks.ceph.mds.a.smithi098.stderr: 14: /lib64/libpthread.so.0(+0x81ca) [0x7f8c4de081ca] 2023-03-03T08:04:15.630 INFO:tasks.ceph.mds.a.smithi098.stderr: 15: clone() 2023-03-03T08:04:15.630 INFO:tasks.ceph.mds.a.smithi098.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-03T08:04:15.630 INFO:tasks.ceph.mds.a.smithi098.stderr: 2023-03-03T08:04:15.637 INFO:tasks.ceph.mds.a.smithi098.stderr: -4> 2023-03-03T08:04:15.592+0000 7f8c395ea700 -1 mds.0.cache.den(0x10000000002 .meta) newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x55f8653a0f00] 2023-03-03T08:04:15.637 INFO:tasks.ceph.mds.a.smithi098.stderr: -2> 2023-03-03T08:04:15.592+0000 7f8c395ea700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x55f8653a0f00] 2023-03-03T08:04:15.637 INFO:tasks.ceph.mds.a.smithi098.stderr: -1> 2023-03-03T08:04:15.594+0000 7f8c395ea700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7f8c395ea700 time 2023-03-03T08:04:15.593561+0000 2023-03-03T08:04:15.637 INFO:tasks.ceph.mds.a.smithi098.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:04:15.637 INFO:tasks.ceph.mds.a.smithi098.stderr: 2023-03-03T08:04:15.637 INFO:tasks.ceph.mds.a.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:15.638 INFO:tasks.ceph.mds.a.smithi098.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f8c4faa941b] 2023-03-03T08:04:15.638 INFO:tasks.ceph.mds.a.smithi098.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x55f861c15794] 2023-03-03T08:04:15.638 INFO:tasks.ceph.mds.a.smithi098.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f861a6eda7] 2023-03-03T08:04:15.638 INFO:tasks.ceph.mds.a.smithi098.stderr: 4: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f861ba2c95] 2023-03-03T08:04:15.638 INFO:tasks.ceph.mds.a.smithi098.stderr: 5: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f861b70c70] 2023-03-03T08:04:15.638 INFO:tasks.ceph.mds.a.smithi098.stderr: 6: (MDSContext::complete(int)+0x5f) [0x55f861cf22ff] 2023-03-03T08:04:15.638 INFO:tasks.ceph.mds.a.smithi098.stderr: 7: (MDSIOContextBase::complete(int)+0x524) [0x55f861cf2a84] 2023-03-03T08:04:15.639 INFO:tasks.ceph.mds.a.smithi098.stderr: 8: (Filer::C_Probe::finish(int)+0xbb) [0x55f861db0fab] 2023-03-03T08:04:15.639 INFO:tasks.ceph.mds.a.smithi098.stderr: 9: (Context::complete(int)+0xd) [0x55f86199213d] 2023-03-03T08:04:15.639 INFO:tasks.ceph.mds.a.smithi098.stderr: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7f8c4fb4c33d] 2023-03-03T08:04:15.639 INFO:tasks.ceph.mds.a.smithi098.stderr: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f8c4de081ca] 2023-03-03T08:04:15.639 INFO:tasks.ceph.mds.a.smithi098.stderr: 12: clone() 2023-03-03T08:04:15.639 INFO:tasks.ceph.mds.a.smithi098.stderr: 2023-03-03T08:04:15.639 INFO:tasks.ceph.mds.a.smithi098.stderr: 0> 2023-03-03T08:04:15.596+0000 7f8c395ea700 -1 *** Caught signal (Aborted) ** 2023-03-03T08:04:15.639 INFO:tasks.ceph.mds.a.smithi098.stderr: in thread 7f8c395ea700 thread_name:MR_Finisher 2023-03-03T08:04:15.640 INFO:tasks.ceph.mds.a.smithi098.stderr: 2023-03-03T08:04:15.640 INFO:tasks.ceph.mds.a.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:15.640 INFO:tasks.ceph.mds.a.smithi098.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8c4de12cf0] 2023-03-03T08:04:15.640 INFO:tasks.ceph.mds.a.smithi098.stderr: 2: gsignal() 2023-03-03T08:04:15.640 INFO:tasks.ceph.mds.a.smithi098.stderr: 3: abort() 2023-03-03T08:04:15.640 INFO:tasks.ceph.mds.a.smithi098.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7f8c4faa94db] 2023-03-03T08:04:15.640 INFO:tasks.ceph.mds.a.smithi098.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x55f861c15794] 2023-03-03T08:04:15.641 INFO:tasks.ceph.mds.a.smithi098.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f861a6eda7] 2023-03-03T08:04:15.641 INFO:tasks.ceph.mds.a.smithi098.stderr: 7: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f861ba2c95] 2023-03-03T08:04:15.641 INFO:tasks.ceph.mds.a.smithi098.stderr: 8: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f861b70c70] 2023-03-03T08:04:15.641 INFO:tasks.ceph.mds.a.smithi098.stderr: 9: (MDSContext::complete(int)+0x5f) [0x55f861cf22ff] 2023-03-03T08:04:15.641 INFO:tasks.ceph.mds.a.smithi098.stderr: 10: (MDSIOContextBase::complete(int)+0x524) [0x55f861cf2a84] 2023-03-03T08:04:15.641 INFO:tasks.ceph.mds.a.smithi098.stderr: 11: (Filer::C_Probe::finish(int)+0xbb) [0x55f861db0fab] 2023-03-03T08:04:15.641 INFO:tasks.ceph.mds.a.smithi098.stderr: 12: (Context::complete(int)+0xd) [0x55f86199213d] 2023-03-03T08:04:15.641 INFO:tasks.ceph.mds.a.smithi098.stderr: 13: (Finisher::finisher_thread_entry()+0x18d) [0x7f8c4fb4c33d] 2023-03-03T08:04:15.642 INFO:tasks.ceph.mds.a.smithi098.stderr: 14: /lib64/libpthread.so.0(+0x81ca) [0x7f8c4de081ca] 2023-03-03T08:04:15.642 INFO:tasks.ceph.mds.a.smithi098.stderr: 15: clone() 2023-03-03T08:04:15.642 INFO:tasks.ceph.mds.a.smithi098.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-03T08:04:15.642 INFO:tasks.ceph.mds.a.smithi098.stderr: 2023-03-03T08:04:15.944 INFO:tasks.ceph.mds.a.smithi098.stderr:daemon-helper: command crashed with signal 6 2023-03-03T08:04:17.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2023-03-03T08:04:17.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~20s 2023-03-03T08:04:24.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~7s 2023-03-03T08:04:24.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~26s 2023-03-03T08:04:30.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~13s 2023-03-03T08:04:30.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~33s 2023-03-03T08:04:34.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:04:34.543 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:04:35.596 INFO:tasks.ceph.mds.c.smithi098.stderr:2023-03-03T08:04:35.594+0000 7fdfff9ea700 -1 mds.0.cache.den(0x10000000002 .meta) newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x563e53fe2f00] 2023-03-03T08:04:35.596 INFO:tasks.ceph.mds.c.smithi098.stderr:2023-03-03T08:04:35.594+0000 7fdfff9ea700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x563e53fe2f00] 2023-03-03T08:04:35.597 INFO:tasks.ceph.mds.c.smithi098.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7fdfff9ea700 time 2023-03-03T08:04:35.595608+0000 2023-03-03T08:04:35.597 INFO:tasks.ceph.mds.c.smithi098.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:04:35.597 INFO:tasks.ceph.mds.c.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:35.598 INFO:tasks.ceph.mds.c.smithi098.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7fe015ea941b] 2023-03-03T08:04:35.598 INFO:tasks.ceph.mds.c.smithi098.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x563e50a66794] 2023-03-03T08:04:35.598 INFO:tasks.ceph.mds.c.smithi098.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x563e508bfda7] 2023-03-03T08:04:35.598 INFO:tasks.ceph.mds.c.smithi098.stderr: 4: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x563e509f3c95] 2023-03-03T08:04:35.599 INFO:tasks.ceph.mds.c.smithi098.stderr: 5: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x563e509c1c70] 2023-03-03T08:04:35.599 INFO:tasks.ceph.mds.c.smithi098.stderr: 6: (MDSContext::complete(int)+0x5f) [0x563e50b432ff] 2023-03-03T08:04:35.599 INFO:tasks.ceph.mds.c.smithi098.stderr: 7: (MDSIOContextBase::complete(int)+0x524) [0x563e50b43a84] 2023-03-03T08:04:35.599 INFO:tasks.ceph.mds.c.smithi098.stderr: 8: (Filer::C_Probe::finish(int)+0xbb) [0x563e50c01fab] 2023-03-03T08:04:35.600 INFO:tasks.ceph.mds.c.smithi098.stderr: 9: (Context::complete(int)+0xd) [0x563e507e313d] 2023-03-03T08:04:35.600 INFO:tasks.ceph.mds.c.smithi098.stderr: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7fe015f4c33d] 2023-03-03T08:04:35.600 INFO:tasks.ceph.mds.c.smithi098.stderr: 11: /lib64/libpthread.so.0(+0x81ca) [0x7fe0142081ca] 2023-03-03T08:04:35.601 INFO:tasks.ceph.mds.c.smithi098.stderr: 12: clone() 2023-03-03T08:04:35.601 INFO:tasks.ceph.mds.c.smithi098.stderr:2023-03-03T08:04:35.596+0000 7fdfff9ea700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7fdfff9ea700 time 2023-03-03T08:04:35.595608+0000 2023-03-03T08:04:35.601 INFO:tasks.ceph.mds.c.smithi098.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:04:35.601 INFO:tasks.ceph.mds.c.smithi098.stderr: 2023-03-03T08:04:35.602 INFO:tasks.ceph.mds.c.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:35.602 INFO:tasks.ceph.mds.c.smithi098.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7fe015ea941b] 2023-03-03T08:04:35.602 INFO:tasks.ceph.mds.c.smithi098.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x563e50a66794] 2023-03-03T08:04:35.603 INFO:tasks.ceph.mds.c.smithi098.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x563e508bfda7] 2023-03-03T08:04:35.603 INFO:tasks.ceph.mds.c.smithi098.stderr: 4: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x563e509f3c95] 2023-03-03T08:04:35.603 INFO:tasks.ceph.mds.c.smithi098.stderr: 5: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x563e509c1c70] 2023-03-03T08:04:35.603 INFO:tasks.ceph.mds.c.smithi098.stderr: 6: (MDSContext::complete(int)+0x5f) [0x563e50b432ff] 2023-03-03T08:04:35.604 INFO:tasks.ceph.mds.c.smithi098.stderr: 7: (MDSIOContextBase::complete(int)+0x524) [0x563e50b43a84] 2023-03-03T08:04:35.604 INFO:tasks.ceph.mds.c.smithi098.stderr: 8: (Filer::C_Probe::finish(int)+0xbb) [0x563e50c01fab] 2023-03-03T08:04:35.604 INFO:tasks.ceph.mds.c.smithi098.stderr: 9: (Context::complete(int)+0xd) [0x563e507e313d] 2023-03-03T08:04:35.604 INFO:tasks.ceph.mds.c.smithi098.stderr: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7fe015f4c33d] 2023-03-03T08:04:35.605 INFO:tasks.ceph.mds.c.smithi098.stderr: 11: /lib64/libpthread.so.0(+0x81ca) [0x7fe0142081ca] 2023-03-03T08:04:35.605 INFO:tasks.ceph.mds.c.smithi098.stderr: 12: clone() 2023-03-03T08:04:35.605 INFO:tasks.ceph.mds.c.smithi098.stderr: 2023-03-03T08:04:35.606 INFO:tasks.ceph.mds.c.smithi098.stderr:*** Caught signal (Aborted) ** 2023-03-03T08:04:35.606 INFO:tasks.ceph.mds.c.smithi098.stderr: in thread 7fdfff9ea700 thread_name:MR_Finisher 2023-03-03T08:04:35.607 INFO:tasks.ceph.mds.c.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:35.607 INFO:tasks.ceph.mds.c.smithi098.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fe014212cf0] 2023-03-03T08:04:35.607 INFO:tasks.ceph.mds.c.smithi098.stderr: 2: gsignal() 2023-03-03T08:04:35.607 INFO:tasks.ceph.mds.c.smithi098.stderr: 3: abort() 2023-03-03T08:04:35.607 INFO:tasks.ceph.mds.c.smithi098.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7fe015ea94db] 2023-03-03T08:04:35.608 INFO:tasks.ceph.mds.c.smithi098.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x563e50a66794] 2023-03-03T08:04:35.608 INFO:tasks.ceph.mds.c.smithi098.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x563e508bfda7] 2023-03-03T08:04:35.608 INFO:tasks.ceph.mds.c.smithi098.stderr: 7: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x563e509f3c95] 2023-03-03T08:04:35.608 INFO:tasks.ceph.mds.c.smithi098.stderr: 8: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x563e509c1c70] 2023-03-03T08:04:35.608 INFO:tasks.ceph.mds.c.smithi098.stderr: 9: (MDSContext::complete(int)+0x5f) [0x563e50b432ff] 2023-03-03T08:04:35.608 INFO:tasks.ceph.mds.c.smithi098.stderr: 10: (MDSIOContextBase::complete(int)+0x524) [0x563e50b43a84] 2023-03-03T08:04:35.609 INFO:tasks.ceph.mds.c.smithi098.stderr: 11: (Filer::C_Probe::finish(int)+0xbb) [0x563e50c01fab] 2023-03-03T08:04:35.609 INFO:tasks.ceph.mds.c.smithi098.stderr: 12: (Context::complete(int)+0xd) [0x563e507e313d] 2023-03-03T08:04:35.609 INFO:tasks.ceph.mds.c.smithi098.stderr: 13: (Finisher::finisher_thread_entry()+0x18d) [0x7fe015f4c33d] 2023-03-03T08:04:35.609 INFO:tasks.ceph.mds.c.smithi098.stderr: 14: /lib64/libpthread.so.0(+0x81ca) [0x7fe0142081ca] 2023-03-03T08:04:35.609 INFO:tasks.ceph.mds.c.smithi098.stderr: 15: clone() 2023-03-03T08:04:35.610 INFO:tasks.ceph.mds.c.smithi098.stderr:2023-03-03T08:04:35.598+0000 7fdfff9ea700 -1 *** Caught signal (Aborted) ** 2023-03-03T08:04:35.610 INFO:tasks.ceph.mds.c.smithi098.stderr: in thread 7fdfff9ea700 thread_name:MR_Finisher 2023-03-03T08:04:35.610 INFO:tasks.ceph.mds.c.smithi098.stderr: 2023-03-03T08:04:35.610 INFO:tasks.ceph.mds.c.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:35.610 INFO:tasks.ceph.mds.c.smithi098.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fe014212cf0] 2023-03-03T08:04:35.611 INFO:tasks.ceph.mds.c.smithi098.stderr: 2: gsignal() 2023-03-03T08:04:35.611 INFO:tasks.ceph.mds.c.smithi098.stderr: 3: abort() 2023-03-03T08:04:35.611 INFO:tasks.ceph.mds.c.smithi098.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7fe015ea94db] 2023-03-03T08:04:35.611 INFO:tasks.ceph.mds.c.smithi098.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x563e50a66794] 2023-03-03T08:04:35.611 INFO:tasks.ceph.mds.c.smithi098.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x563e508bfda7] 2023-03-03T08:04:35.612 INFO:tasks.ceph.mds.c.smithi098.stderr: 7: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x563e509f3c95] 2023-03-03T08:04:35.612 INFO:tasks.ceph.mds.c.smithi098.stderr: 8: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x563e509c1c70] 2023-03-03T08:04:35.612 INFO:tasks.ceph.mds.c.smithi098.stderr: 9: (MDSContext::complete(int)+0x5f) [0x563e50b432ff] 2023-03-03T08:04:35.612 INFO:tasks.ceph.mds.c.smithi098.stderr: 10: (MDSIOContextBase::complete(int)+0x524) [0x563e50b43a84] 2023-03-03T08:04:35.612 INFO:tasks.ceph.mds.c.smithi098.stderr: 11: (Filer::C_Probe::finish(int)+0xbb) [0x563e50c01fab] 2023-03-03T08:04:35.612 INFO:tasks.ceph.mds.c.smithi098.stderr: 12: (Context::complete(int)+0xd) [0x563e507e313d] 2023-03-03T08:04:35.613 INFO:tasks.ceph.mds.c.smithi098.stderr: 13: (Finisher::finisher_thread_entry()+0x18d) [0x7fe015f4c33d] 2023-03-03T08:04:35.613 INFO:tasks.ceph.mds.c.smithi098.stderr: 14: /lib64/libpthread.so.0(+0x81ca) [0x7fe0142081ca] 2023-03-03T08:04:35.613 INFO:tasks.ceph.mds.c.smithi098.stderr: 15: clone() 2023-03-03T08:04:35.613 INFO:tasks.ceph.mds.c.smithi098.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-03T08:04:35.613 INFO:tasks.ceph.mds.c.smithi098.stderr: 2023-03-03T08:04:35.626 INFO:tasks.ceph.mds.c.smithi098.stderr: -7> 2023-03-03T08:04:35.594+0000 7fdfff9ea700 -1 mds.0.cache.den(0x10000000002 .meta) newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x563e53fe2f00] 2023-03-03T08:04:35.626 INFO:tasks.ceph.mds.c.smithi098.stderr: -5> 2023-03-03T08:04:35.594+0000 7fdfff9ea700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x563e53fe2f00] 2023-03-03T08:04:35.626 INFO:tasks.ceph.mds.c.smithi098.stderr: -1> 2023-03-03T08:04:35.596+0000 7fdfff9ea700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7fdfff9ea700 time 2023-03-03T08:04:35.595608+0000 2023-03-03T08:04:35.627 INFO:tasks.ceph.mds.c.smithi098.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:04:35.627 INFO:tasks.ceph.mds.c.smithi098.stderr: 2023-03-03T08:04:35.627 INFO:tasks.ceph.mds.c.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:35.628 INFO:tasks.ceph.mds.c.smithi098.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7fe015ea941b] 2023-03-03T08:04:35.628 INFO:tasks.ceph.mds.c.smithi098.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x563e50a66794] 2023-03-03T08:04:35.628 INFO:tasks.ceph.mds.c.smithi098.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x563e508bfda7] 2023-03-03T08:04:35.628 INFO:tasks.ceph.mds.c.smithi098.stderr: 4: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x563e509f3c95] 2023-03-03T08:04:35.628 INFO:tasks.ceph.mds.c.smithi098.stderr: 5: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x563e509c1c70] 2023-03-03T08:04:35.628 INFO:tasks.ceph.mds.c.smithi098.stderr: 6: (MDSContext::complete(int)+0x5f) [0x563e50b432ff] 2023-03-03T08:04:35.628 INFO:tasks.ceph.mds.c.smithi098.stderr: 7: (MDSIOContextBase::complete(int)+0x524) [0x563e50b43a84] 2023-03-03T08:04:35.629 INFO:tasks.ceph.mds.c.smithi098.stderr: 8: (Filer::C_Probe::finish(int)+0xbb) [0x563e50c01fab] 2023-03-03T08:04:35.629 INFO:tasks.ceph.mds.c.smithi098.stderr: 9: (Context::complete(int)+0xd) [0x563e507e313d] 2023-03-03T08:04:35.629 INFO:tasks.ceph.mds.c.smithi098.stderr: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7fe015f4c33d] 2023-03-03T08:04:35.629 INFO:tasks.ceph.mds.c.smithi098.stderr: 11: /lib64/libpthread.so.0(+0x81ca) [0x7fe0142081ca] 2023-03-03T08:04:35.629 INFO:tasks.ceph.mds.c.smithi098.stderr: 12: clone() 2023-03-03T08:04:35.629 INFO:tasks.ceph.mds.c.smithi098.stderr: 2023-03-03T08:04:35.630 INFO:tasks.ceph.mds.c.smithi098.stderr: 0> 2023-03-03T08:04:35.598+0000 7fdfff9ea700 -1 *** Caught signal (Aborted) ** 2023-03-03T08:04:35.630 INFO:tasks.ceph.mds.c.smithi098.stderr: in thread 7fdfff9ea700 thread_name:MR_Finisher 2023-03-03T08:04:35.630 INFO:tasks.ceph.mds.c.smithi098.stderr: 2023-03-03T08:04:35.630 INFO:tasks.ceph.mds.c.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:35.631 INFO:tasks.ceph.mds.c.smithi098.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fe014212cf0] 2023-03-03T08:04:35.631 INFO:tasks.ceph.mds.c.smithi098.stderr: 2: gsignal() 2023-03-03T08:04:35.632 INFO:tasks.ceph.mds.c.smithi098.stderr: 3: abort() 2023-03-03T08:04:35.632 INFO:tasks.ceph.mds.c.smithi098.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7fe015ea94db] 2023-03-03T08:04:35.632 INFO:tasks.ceph.mds.c.smithi098.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x563e50a66794] 2023-03-03T08:04:35.633 INFO:tasks.ceph.mds.c.smithi098.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x563e508bfda7] 2023-03-03T08:04:35.633 INFO:tasks.ceph.mds.c.smithi098.stderr: 7: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x563e509f3c95] 2023-03-03T08:04:35.634 INFO:tasks.ceph.mds.c.smithi098.stderr: 8: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x563e509c1c70] 2023-03-03T08:04:35.634 INFO:tasks.ceph.mds.c.smithi098.stderr: 9: (MDSContext::complete(int)+0x5f) [0x563e50b432ff] 2023-03-03T08:04:35.634 INFO:tasks.ceph.mds.c.smithi098.stderr: 10: (MDSIOContextBase::complete(int)+0x524) [0x563e50b43a84] 2023-03-03T08:04:35.634 INFO:tasks.ceph.mds.c.smithi098.stderr: 11: (Filer::C_Probe::finish(int)+0xbb) [0x563e50c01fab] 2023-03-03T08:04:35.635 INFO:tasks.ceph.mds.c.smithi098.stderr: 12: (Context::complete(int)+0xd) [0x563e507e313d] 2023-03-03T08:04:35.635 INFO:tasks.ceph.mds.c.smithi098.stderr: 13: (Finisher::finisher_thread_entry()+0x18d) [0x7fe015f4c33d] 2023-03-03T08:04:35.635 INFO:tasks.ceph.mds.c.smithi098.stderr: 14: /lib64/libpthread.so.0(+0x81ca) [0x7fe0142081ca] 2023-03-03T08:04:35.635 INFO:tasks.ceph.mds.c.smithi098.stderr: 15: clone() 2023-03-03T08:04:35.636 INFO:tasks.ceph.mds.c.smithi098.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-03T08:04:35.636 INFO:tasks.ceph.mds.c.smithi098.stderr: 2023-03-03T08:04:35.638 INFO:tasks.ceph.mds.c.smithi098.stderr: -7> 2023-03-03T08:04:35.594+0000 7fdfff9ea700 -1 mds.0.cache.den(0x10000000002 .meta) newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x563e53fe2f00] 2023-03-03T08:04:35.638 INFO:tasks.ceph.mds.c.smithi098.stderr: -5> 2023-03-03T08:04:35.594+0000 7fdfff9ea700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x563e53fe2f00] 2023-03-03T08:04:35.638 INFO:tasks.ceph.mds.c.smithi098.stderr: -1> 2023-03-03T08:04:35.596+0000 7fdfff9ea700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7fdfff9ea700 time 2023-03-03T08:04:35.595608+0000 2023-03-03T08:04:35.638 INFO:tasks.ceph.mds.c.smithi098.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:04:35.638 INFO:tasks.ceph.mds.c.smithi098.stderr: 2023-03-03T08:04:35.639 INFO:tasks.ceph.mds.c.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:35.639 INFO:tasks.ceph.mds.c.smithi098.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7fe015ea941b] 2023-03-03T08:04:35.639 INFO:tasks.ceph.mds.c.smithi098.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x563e50a66794] 2023-03-03T08:04:35.639 INFO:tasks.ceph.mds.c.smithi098.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x563e508bfda7] 2023-03-03T08:04:35.639 INFO:tasks.ceph.mds.c.smithi098.stderr: 4: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x563e509f3c95] 2023-03-03T08:04:35.639 INFO:tasks.ceph.mds.c.smithi098.stderr: 5: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x563e509c1c70] 2023-03-03T08:04:35.640 INFO:tasks.ceph.mds.c.smithi098.stderr: 6: (MDSContext::complete(int)+0x5f) [0x563e50b432ff] 2023-03-03T08:04:35.640 INFO:tasks.ceph.mds.c.smithi098.stderr: 7: (MDSIOContextBase::complete(int)+0x524) [0x563e50b43a84] 2023-03-03T08:04:35.640 INFO:tasks.ceph.mds.c.smithi098.stderr: 8: (Filer::C_Probe::finish(int)+0xbb) [0x563e50c01fab] 2023-03-03T08:04:35.640 INFO:tasks.ceph.mds.c.smithi098.stderr: 9: (Context::complete(int)+0xd) [0x563e507e313d] 2023-03-03T08:04:35.640 INFO:tasks.ceph.mds.c.smithi098.stderr: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7fe015f4c33d] 2023-03-03T08:04:35.641 INFO:tasks.ceph.mds.c.smithi098.stderr: 11: /lib64/libpthread.so.0(+0x81ca) [0x7fe0142081ca] 2023-03-03T08:04:35.641 INFO:tasks.ceph.mds.c.smithi098.stderr: 12: clone() 2023-03-03T08:04:35.641 INFO:tasks.ceph.mds.c.smithi098.stderr: 2023-03-03T08:04:35.641 INFO:tasks.ceph.mds.c.smithi098.stderr: 0> 2023-03-03T08:04:35.598+0000 7fdfff9ea700 -1 *** Caught signal (Aborted) ** 2023-03-03T08:04:35.642 INFO:tasks.ceph.mds.c.smithi098.stderr: in thread 7fdfff9ea700 thread_name:MR_Finisher 2023-03-03T08:04:35.642 INFO:tasks.ceph.mds.c.smithi098.stderr: 2023-03-03T08:04:35.642 INFO:tasks.ceph.mds.c.smithi098.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:35.642 INFO:tasks.ceph.mds.c.smithi098.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fe014212cf0] 2023-03-03T08:04:35.642 INFO:tasks.ceph.mds.c.smithi098.stderr: 2: gsignal() 2023-03-03T08:04:35.642 INFO:tasks.ceph.mds.c.smithi098.stderr: 3: abort() 2023-03-03T08:04:35.642 INFO:tasks.ceph.mds.c.smithi098.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7fe015ea94db] 2023-03-03T08:04:35.643 INFO:tasks.ceph.mds.c.smithi098.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x563e50a66794] 2023-03-03T08:04:35.643 INFO:tasks.ceph.mds.c.smithi098.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x563e508bfda7] 2023-03-03T08:04:35.643 INFO:tasks.ceph.mds.c.smithi098.stderr: 7: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x563e509f3c95] 2023-03-03T08:04:35.643 INFO:tasks.ceph.mds.c.smithi098.stderr: 8: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x563e509c1c70] 2023-03-03T08:04:35.643 INFO:tasks.ceph.mds.c.smithi098.stderr: 9: (MDSContext::complete(int)+0x5f) [0x563e50b432ff] 2023-03-03T08:04:35.643 INFO:tasks.ceph.mds.c.smithi098.stderr: 10: (MDSIOContextBase::complete(int)+0x524) [0x563e50b43a84] 2023-03-03T08:04:35.644 INFO:tasks.ceph.mds.c.smithi098.stderr: 11: (Filer::C_Probe::finish(int)+0xbb) [0x563e50c01fab] 2023-03-03T08:04:35.644 INFO:tasks.ceph.mds.c.smithi098.stderr: 12: (Context::complete(int)+0xd) [0x563e507e313d] 2023-03-03T08:04:35.644 INFO:tasks.ceph.mds.c.smithi098.stderr: 13: (Finisher::finisher_thread_entry()+0x18d) [0x7fe015f4c33d] 2023-03-03T08:04:35.644 INFO:tasks.ceph.mds.c.smithi098.stderr: 14: /lib64/libpthread.so.0(+0x81ca) [0x7fe0142081ca] 2023-03-03T08:04:35.644 INFO:tasks.ceph.mds.c.smithi098.stderr: 15: clone() 2023-03-03T08:04:35.644 INFO:tasks.ceph.mds.c.smithi098.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-03T08:04:35.644 INFO:tasks.ceph.mds.c.smithi098.stderr: 2023-03-03T08:04:35.969 INFO:tasks.ceph.mds.c.smithi098.stderr:daemon-helper: command crashed with signal 6 2023-03-03T08:04:37.075 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~20s 2023-03-03T08:04:37.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2023-03-03T08:04:37.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~39s 2023-03-03T08:04:43.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~26s 2023-03-03T08:04:43.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~6s 2023-03-03T08:04:43.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~46s 2023-03-03T08:04:49.896 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~32s 2023-03-03T08:04:49.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~13s 2023-03-03T08:04:49.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~52s 2023-03-03T08:04:55.602 INFO:tasks.ceph.mds.b.smithi100.stderr:2023-03-03T08:04:55.600+0000 7f73fe5ea700 -1 mds.0.cache.den(0x10000000002 .meta) newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x55f0c3072f00] 2023-03-03T08:04:55.602 INFO:tasks.ceph.mds.b.smithi100.stderr:2023-03-03T08:04:55.600+0000 7f73fe5ea700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x55f0c3072f00] 2023-03-03T08:04:55.603 INFO:tasks.ceph.mds.b.smithi100.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7f73fe5ea700 time 2023-03-03T08:04:55.601591+0000 2023-03-03T08:04:55.603 INFO:tasks.ceph.mds.b.smithi100.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:04:55.603 INFO:tasks.ceph.mds.b.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:55.603 INFO:tasks.ceph.mds.b.smithi100.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f7414aa941b] 2023-03-03T08:04:55.604 INFO:tasks.ceph.mds.b.smithi100.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x55f0bfd70794] 2023-03-03T08:04:55.604 INFO:tasks.ceph.mds.b.smithi100.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f0bfbc9da7] 2023-03-03T08:04:55.604 INFO:tasks.ceph.mds.b.smithi100.stderr: 4: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f0bfcfdc95] 2023-03-03T08:04:55.604 INFO:tasks.ceph.mds.b.smithi100.stderr: 5: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f0bfccbc70] 2023-03-03T08:04:55.604 INFO:tasks.ceph.mds.b.smithi100.stderr: 6: (MDSContext::complete(int)+0x5f) [0x55f0bfe4d2ff] 2023-03-03T08:04:55.604 INFO:tasks.ceph.mds.b.smithi100.stderr: 7: (MDSIOContextBase::complete(int)+0x524) [0x55f0bfe4da84] 2023-03-03T08:04:55.604 INFO:tasks.ceph.mds.b.smithi100.stderr: 8: (Filer::C_Probe::finish(int)+0xbb) [0x55f0bff0bfab] 2023-03-03T08:04:55.605 INFO:tasks.ceph.mds.b.smithi100.stderr: 9: (Context::complete(int)+0xd) [0x55f0bfaed13d] 2023-03-03T08:04:55.605 INFO:tasks.ceph.mds.b.smithi100.stderr: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7f7414b4c33d] 2023-03-03T08:04:55.605 INFO:tasks.ceph.mds.b.smithi100.stderr: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f7412e081ca] 2023-03-03T08:04:55.605 INFO:tasks.ceph.mds.b.smithi100.stderr: 12: clone() 2023-03-03T08:04:55.605 INFO:tasks.ceph.mds.b.smithi100.stderr:2023-03-03T08:04:55.602+0000 7f73fe5ea700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7f73fe5ea700 time 2023-03-03T08:04:55.601591+0000 2023-03-03T08:04:55.605 INFO:tasks.ceph.mds.b.smithi100.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:04:55.605 INFO:tasks.ceph.mds.b.smithi100.stderr: 2023-03-03T08:04:55.606 INFO:tasks.ceph.mds.b.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:55.606 INFO:tasks.ceph.mds.b.smithi100.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f7414aa941b] 2023-03-03T08:04:55.606 INFO:tasks.ceph.mds.b.smithi100.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x55f0bfd70794] 2023-03-03T08:04:55.606 INFO:tasks.ceph.mds.b.smithi100.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f0bfbc9da7] 2023-03-03T08:04:55.606 INFO:tasks.ceph.mds.b.smithi100.stderr: 4: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f0bfcfdc95] 2023-03-03T08:04:55.606 INFO:tasks.ceph.mds.b.smithi100.stderr: 5: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f0bfccbc70] 2023-03-03T08:04:55.607 INFO:tasks.ceph.mds.b.smithi100.stderr: 6: (MDSContext::complete(int)+0x5f) [0x55f0bfe4d2ff] 2023-03-03T08:04:55.607 INFO:tasks.ceph.mds.b.smithi100.stderr: 7: (MDSIOContextBase::complete(int)+0x524) [0x55f0bfe4da84] 2023-03-03T08:04:55.607 INFO:tasks.ceph.mds.b.smithi100.stderr: 8: (Filer::C_Probe::finish(int)+0xbb) [0x55f0bff0bfab] 2023-03-03T08:04:55.607 INFO:tasks.ceph.mds.b.smithi100.stderr: 9: (Context::complete(int)+0xd) [0x55f0bfaed13d] 2023-03-03T08:04:55.607 INFO:tasks.ceph.mds.b.smithi100.stderr: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7f7414b4c33d] 2023-03-03T08:04:55.607 INFO:tasks.ceph.mds.b.smithi100.stderr: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f7412e081ca] 2023-03-03T08:04:55.607 INFO:tasks.ceph.mds.b.smithi100.stderr: 12: clone() 2023-03-03T08:04:55.607 INFO:tasks.ceph.mds.b.smithi100.stderr: 2023-03-03T08:04:55.608 INFO:tasks.ceph.mds.b.smithi100.stderr:*** Caught signal (Aborted) ** 2023-03-03T08:04:55.608 INFO:tasks.ceph.mds.b.smithi100.stderr: in thread 7f73fe5ea700 thread_name:MR_Finisher 2023-03-03T08:04:55.608 INFO:tasks.ceph.mds.b.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:55.609 INFO:tasks.ceph.mds.b.smithi100.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7412e12cf0] 2023-03-03T08:04:55.609 INFO:tasks.ceph.mds.b.smithi100.stderr: 2: gsignal() 2023-03-03T08:04:55.609 INFO:tasks.ceph.mds.b.smithi100.stderr: 3: abort() 2023-03-03T08:04:55.609 INFO:tasks.ceph.mds.b.smithi100.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7f7414aa94db] 2023-03-03T08:04:55.609 INFO:tasks.ceph.mds.b.smithi100.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x55f0bfd70794] 2023-03-03T08:04:55.609 INFO:tasks.ceph.mds.b.smithi100.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f0bfbc9da7] 2023-03-03T08:04:55.609 INFO:tasks.ceph.mds.b.smithi100.stderr: 7: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f0bfcfdc95] 2023-03-03T08:04:55.610 INFO:tasks.ceph.mds.b.smithi100.stderr: 8: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f0bfccbc70] 2023-03-03T08:04:55.610 INFO:tasks.ceph.mds.b.smithi100.stderr: 9: (MDSContext::complete(int)+0x5f) [0x55f0bfe4d2ff] 2023-03-03T08:04:55.610 INFO:tasks.ceph.mds.b.smithi100.stderr: 10: (MDSIOContextBase::complete(int)+0x524) [0x55f0bfe4da84] 2023-03-03T08:04:55.610 INFO:tasks.ceph.mds.b.smithi100.stderr: 11: (Filer::C_Probe::finish(int)+0xbb) [0x55f0bff0bfab] 2023-03-03T08:04:55.610 INFO:tasks.ceph.mds.b.smithi100.stderr: 12: (Context::complete(int)+0xd) [0x55f0bfaed13d] 2023-03-03T08:04:55.610 INFO:tasks.ceph.mds.b.smithi100.stderr: 13: (Finisher::finisher_thread_entry()+0x18d) [0x7f7414b4c33d] 2023-03-03T08:04:55.610 INFO:tasks.ceph.mds.b.smithi100.stderr: 14: /lib64/libpthread.so.0(+0x81ca) [0x7f7412e081ca] 2023-03-03T08:04:55.610 INFO:tasks.ceph.mds.b.smithi100.stderr: 15: clone() 2023-03-03T08:04:55.611 INFO:tasks.ceph.mds.b.smithi100.stderr:2023-03-03T08:04:55.604+0000 7f73fe5ea700 -1 *** Caught signal (Aborted) ** 2023-03-03T08:04:55.611 INFO:tasks.ceph.mds.b.smithi100.stderr: in thread 7f73fe5ea700 thread_name:MR_Finisher 2023-03-03T08:04:55.611 INFO:tasks.ceph.mds.b.smithi100.stderr: 2023-03-03T08:04:55.611 INFO:tasks.ceph.mds.b.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:55.611 INFO:tasks.ceph.mds.b.smithi100.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7412e12cf0] 2023-03-03T08:04:55.611 INFO:tasks.ceph.mds.b.smithi100.stderr: 2: gsignal() 2023-03-03T08:04:55.611 INFO:tasks.ceph.mds.b.smithi100.stderr: 3: abort() 2023-03-03T08:04:55.612 INFO:tasks.ceph.mds.b.smithi100.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7f7414aa94db] 2023-03-03T08:04:55.612 INFO:tasks.ceph.mds.b.smithi100.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x55f0bfd70794] 2023-03-03T08:04:55.612 INFO:tasks.ceph.mds.b.smithi100.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f0bfbc9da7] 2023-03-03T08:04:55.612 INFO:tasks.ceph.mds.b.smithi100.stderr: 7: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f0bfcfdc95] 2023-03-03T08:04:55.612 INFO:tasks.ceph.mds.b.smithi100.stderr: 8: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f0bfccbc70] 2023-03-03T08:04:55.612 INFO:tasks.ceph.mds.b.smithi100.stderr: 9: (MDSContext::complete(int)+0x5f) [0x55f0bfe4d2ff] 2023-03-03T08:04:55.612 INFO:tasks.ceph.mds.b.smithi100.stderr: 10: (MDSIOContextBase::complete(int)+0x524) [0x55f0bfe4da84] 2023-03-03T08:04:55.612 INFO:tasks.ceph.mds.b.smithi100.stderr: 11: (Filer::C_Probe::finish(int)+0xbb) [0x55f0bff0bfab] 2023-03-03T08:04:55.613 INFO:tasks.ceph.mds.b.smithi100.stderr: 12: (Context::complete(int)+0xd) [0x55f0bfaed13d] 2023-03-03T08:04:55.613 INFO:tasks.ceph.mds.b.smithi100.stderr: 13: (Finisher::finisher_thread_entry()+0x18d) [0x7f7414b4c33d] 2023-03-03T08:04:55.613 INFO:tasks.ceph.mds.b.smithi100.stderr: 14: /lib64/libpthread.so.0(+0x81ca) [0x7f7412e081ca] 2023-03-03T08:04:55.613 INFO:tasks.ceph.mds.b.smithi100.stderr: 15: clone() 2023-03-03T08:04:55.613 INFO:tasks.ceph.mds.b.smithi100.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-03T08:04:55.613 INFO:tasks.ceph.mds.b.smithi100.stderr: 2023-03-03T08:04:55.632 INFO:tasks.ceph.mds.b.smithi100.stderr: -5> 2023-03-03T08:04:55.600+0000 7f73fe5ea700 -1 mds.0.cache.den(0x10000000002 .meta) newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x55f0c3072f00] 2023-03-03T08:04:55.633 INFO:tasks.ceph.mds.b.smithi100.stderr: -3> 2023-03-03T08:04:55.600+0000 7f73fe5ea700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x55f0c3072f00] 2023-03-03T08:04:55.633 INFO:tasks.ceph.mds.b.smithi100.stderr: -1> 2023-03-03T08:04:55.602+0000 7f73fe5ea700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7f73fe5ea700 time 2023-03-03T08:04:55.601591+0000 2023-03-03T08:04:55.633 INFO:tasks.ceph.mds.b.smithi100.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:04:55.633 INFO:tasks.ceph.mds.b.smithi100.stderr: 2023-03-03T08:04:55.633 INFO:tasks.ceph.mds.b.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:55.633 INFO:tasks.ceph.mds.b.smithi100.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f7414aa941b] 2023-03-03T08:04:55.633 INFO:tasks.ceph.mds.b.smithi100.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x55f0bfd70794] 2023-03-03T08:04:55.634 INFO:tasks.ceph.mds.b.smithi100.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f0bfbc9da7] 2023-03-03T08:04:55.634 INFO:tasks.ceph.mds.b.smithi100.stderr: 4: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f0bfcfdc95] 2023-03-03T08:04:55.634 INFO:tasks.ceph.mds.b.smithi100.stderr: 5: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f0bfccbc70] 2023-03-03T08:04:55.634 INFO:tasks.ceph.mds.b.smithi100.stderr: 6: (MDSContext::complete(int)+0x5f) [0x55f0bfe4d2ff] 2023-03-03T08:04:55.634 INFO:tasks.ceph.mds.b.smithi100.stderr: 7: (MDSIOContextBase::complete(int)+0x524) [0x55f0bfe4da84] 2023-03-03T08:04:55.634 INFO:tasks.ceph.mds.b.smithi100.stderr: 8: (Filer::C_Probe::finish(int)+0xbb) [0x55f0bff0bfab] 2023-03-03T08:04:55.634 INFO:tasks.ceph.mds.b.smithi100.stderr: 9: (Context::complete(int)+0xd) [0x55f0bfaed13d] 2023-03-03T08:04:55.635 INFO:tasks.ceph.mds.b.smithi100.stderr: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7f7414b4c33d] 2023-03-03T08:04:55.635 INFO:tasks.ceph.mds.b.smithi100.stderr: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f7412e081ca] 2023-03-03T08:04:55.635 INFO:tasks.ceph.mds.b.smithi100.stderr: 12: clone() 2023-03-03T08:04:55.635 INFO:tasks.ceph.mds.b.smithi100.stderr: 2023-03-03T08:04:55.635 INFO:tasks.ceph.mds.b.smithi100.stderr: 0> 2023-03-03T08:04:55.604+0000 7f73fe5ea700 -1 *** Caught signal (Aborted) ** 2023-03-03T08:04:55.635 INFO:tasks.ceph.mds.b.smithi100.stderr: in thread 7f73fe5ea700 thread_name:MR_Finisher 2023-03-03T08:04:55.635 INFO:tasks.ceph.mds.b.smithi100.stderr: 2023-03-03T08:04:55.636 INFO:tasks.ceph.mds.b.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:55.636 INFO:tasks.ceph.mds.b.smithi100.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7412e12cf0] 2023-03-03T08:04:55.636 INFO:tasks.ceph.mds.b.smithi100.stderr: 2: gsignal() 2023-03-03T08:04:55.636 INFO:tasks.ceph.mds.b.smithi100.stderr: 3: abort() 2023-03-03T08:04:55.636 INFO:tasks.ceph.mds.b.smithi100.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7f7414aa94db] 2023-03-03T08:04:55.636 INFO:tasks.ceph.mds.b.smithi100.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x55f0bfd70794] 2023-03-03T08:04:55.636 INFO:tasks.ceph.mds.b.smithi100.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f0bfbc9da7] 2023-03-03T08:04:55.637 INFO:tasks.ceph.mds.b.smithi100.stderr: 7: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f0bfcfdc95] 2023-03-03T08:04:55.637 INFO:tasks.ceph.mds.b.smithi100.stderr: 8: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f0bfccbc70] 2023-03-03T08:04:55.637 INFO:tasks.ceph.mds.b.smithi100.stderr: 9: (MDSContext::complete(int)+0x5f) [0x55f0bfe4d2ff] 2023-03-03T08:04:55.637 INFO:tasks.ceph.mds.b.smithi100.stderr: 10: (MDSIOContextBase::complete(int)+0x524) [0x55f0bfe4da84] 2023-03-03T08:04:55.637 INFO:tasks.ceph.mds.b.smithi100.stderr: 11: (Filer::C_Probe::finish(int)+0xbb) [0x55f0bff0bfab] 2023-03-03T08:04:55.637 INFO:tasks.ceph.mds.b.smithi100.stderr: 12: (Context::complete(int)+0xd) [0x55f0bfaed13d] 2023-03-03T08:04:55.637 INFO:tasks.ceph.mds.b.smithi100.stderr: 13: (Finisher::finisher_thread_entry()+0x18d) [0x7f7414b4c33d] 2023-03-03T08:04:55.638 INFO:tasks.ceph.mds.b.smithi100.stderr: 14: /lib64/libpthread.so.0(+0x81ca) [0x7f7412e081ca] 2023-03-03T08:04:55.638 INFO:tasks.ceph.mds.b.smithi100.stderr: 15: clone() 2023-03-03T08:04:55.638 INFO:tasks.ceph.mds.b.smithi100.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-03T08:04:55.638 INFO:tasks.ceph.mds.b.smithi100.stderr: 2023-03-03T08:04:55.644 INFO:tasks.ceph.mds.b.smithi100.stderr: -5> 2023-03-03T08:04:55.600+0000 7f73fe5ea700 -1 mds.0.cache.den(0x10000000002 .meta) newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x55f0c3072f00] 2023-03-03T08:04:55.644 INFO:tasks.ceph.mds.b.smithi100.stderr: -3> 2023-03-03T08:04:55.600+0000 7f73fe5ea700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/volumes/_nogroup/subvolume_0000000000508309/.meta [5,head] auth (dversion lock) pv=156 v=154 ino=0x100000007e2 state=1610612736 | inodepin=1 dirty=1 0x55f0c3072f00] 2023-03-03T08:04:55.644 INFO:tasks.ceph.mds.b.smithi100.stderr: -1> 2023-03-03T08:04:55.602+0000 7f73fe5ea700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7f73fe5ea700 time 2023-03-03T08:04:55.601591+0000 2023-03-03T08:04:55.645 INFO:tasks.ceph.mds.b.smithi100.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2739-g472aaef4/rpm/el8/BUILD/ceph-18.0.0-2739-g472aaef4/src/mds/CDentry.cc: 715: ceph_abort_msg("abort() called") 2023-03-03T08:04:55.645 INFO:tasks.ceph.mds.b.smithi100.stderr: 2023-03-03T08:04:55.645 INFO:tasks.ceph.mds.b.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:55.645 INFO:tasks.ceph.mds.b.smithi100.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f7414aa941b] 2023-03-03T08:04:55.645 INFO:tasks.ceph.mds.b.smithi100.stderr: 2: (CDentry::check_corruption(bool)+0x614) [0x55f0bfd70794] 2023-03-03T08:04:55.645 INFO:tasks.ceph.mds.b.smithi100.stderr: 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f0bfbc9da7] 2023-03-03T08:04:55.646 INFO:tasks.ceph.mds.b.smithi100.stderr: 4: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f0bfcfdc95] 2023-03-03T08:04:55.646 INFO:tasks.ceph.mds.b.smithi100.stderr: 5: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f0bfccbc70] 2023-03-03T08:04:55.646 INFO:tasks.ceph.mds.b.smithi100.stderr: 6: (MDSContext::complete(int)+0x5f) [0x55f0bfe4d2ff] 2023-03-03T08:04:55.646 INFO:tasks.ceph.mds.b.smithi100.stderr: 7: (MDSIOContextBase::complete(int)+0x524) [0x55f0bfe4da84] 2023-03-03T08:04:55.646 INFO:tasks.ceph.mds.b.smithi100.stderr: 8: (Filer::C_Probe::finish(int)+0xbb) [0x55f0bff0bfab] 2023-03-03T08:04:55.646 INFO:tasks.ceph.mds.b.smithi100.stderr: 9: (Context::complete(int)+0xd) [0x55f0bfaed13d] 2023-03-03T08:04:55.646 INFO:tasks.ceph.mds.b.smithi100.stderr: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7f7414b4c33d] 2023-03-03T08:04:55.647 INFO:tasks.ceph.mds.b.smithi100.stderr: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f7412e081ca] 2023-03-03T08:04:55.647 INFO:tasks.ceph.mds.b.smithi100.stderr: 12: clone() 2023-03-03T08:04:55.647 INFO:tasks.ceph.mds.b.smithi100.stderr: 2023-03-03T08:04:55.647 INFO:tasks.ceph.mds.b.smithi100.stderr: 0> 2023-03-03T08:04:55.604+0000 7f73fe5ea700 -1 *** Caught signal (Aborted) ** 2023-03-03T08:04:55.647 INFO:tasks.ceph.mds.b.smithi100.stderr: in thread 7f73fe5ea700 thread_name:MR_Finisher 2023-03-03T08:04:55.647 INFO:tasks.ceph.mds.b.smithi100.stderr: 2023-03-03T08:04:55.647 INFO:tasks.ceph.mds.b.smithi100.stderr: ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 2023-03-03T08:04:55.648 INFO:tasks.ceph.mds.b.smithi100.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7412e12cf0] 2023-03-03T08:04:55.648 INFO:tasks.ceph.mds.b.smithi100.stderr: 2: gsignal() 2023-03-03T08:04:55.648 INFO:tasks.ceph.mds.b.smithi100.stderr: 3: abort() 2023-03-03T08:04:55.648 INFO:tasks.ceph.mds.b.smithi100.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7f7414aa94db] 2023-03-03T08:04:55.648 INFO:tasks.ceph.mds.b.smithi100.stderr: 5: (CDentry::check_corruption(bool)+0x614) [0x55f0bfd70794] 2023-03-03T08:04:55.648 INFO:tasks.ceph.mds.b.smithi100.stderr: 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x55f0bfbc9da7] 2023-03-03T08:04:55.648 INFO:tasks.ceph.mds.b.smithi100.stderr: 7: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x855) [0x55f0bfcfdc95] 2023-03-03T08:04:55.649 INFO:tasks.ceph.mds.b.smithi100.stderr: 8: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x4f0) [0x55f0bfccbc70] 2023-03-03T08:04:55.649 INFO:tasks.ceph.mds.b.smithi100.stderr: 9: (MDSContext::complete(int)+0x5f) [0x55f0bfe4d2ff] 2023-03-03T08:04:55.649 INFO:tasks.ceph.mds.b.smithi100.stderr: 10: (MDSIOContextBase::complete(int)+0x524) [0x55f0bfe4da84] 2023-03-03T08:04:55.649 INFO:tasks.ceph.mds.b.smithi100.stderr: 11: (Filer::C_Probe::finish(int)+0xbb) [0x55f0bff0bfab] 2023-03-03T08:04:55.649 INFO:tasks.ceph.mds.b.smithi100.stderr: 12: (Context::complete(int)+0xd) [0x55f0bfaed13d] 2023-03-03T08:04:55.649 INFO:tasks.ceph.mds.b.smithi100.stderr: 13: (Finisher::finisher_thread_entry()+0x18d) [0x7f7414b4c33d] 2023-03-03T08:04:55.649 INFO:tasks.ceph.mds.b.smithi100.stderr: 14: /lib64/libpthread.so.0(+0x81ca) [0x7f7412e081ca] 2023-03-03T08:04:55.650 INFO:tasks.ceph.mds.b.smithi100.stderr: 15: clone() 2023-03-03T08:04:55.650 INFO:tasks.ceph.mds.b.smithi100.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-03T08:04:55.650 INFO:tasks.ceph.mds.b.smithi100.stderr: 2023-03-03T08:04:56.018 INFO:tasks.ceph.mds.b.smithi100.stderr:daemon-helper: command crashed with signal 6 2023-03-03T08:04:56.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~39s 2023-03-03T08:04:56.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~19s 2023-03-03T08:04:56.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2023-03-03T08:04:56.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~59s 2023-03-03T08:05:02.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~45s 2023-03-03T08:05:02.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~26s 2023-03-03T08:05:02.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~6s 2023-03-03T08:05:02.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~65s 2023-03-03T08:05:04.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:05:04.699 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:05:08.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~51s 2023-03-03T08:05:08.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~32s 2023-03-03T08:05:08.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~13s 2023-03-03T08:05:08.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~71s 2023-03-03T08:05:15.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~58s 2023-03-03T08:05:15.164 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~38s 2023-03-03T08:05:15.164 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~19s 2023-03-03T08:05:15.164 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~78s 2023-03-03T08:05:21.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~64s 2023-03-03T08:05:21.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~44s 2023-03-03T08:05:21.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~25s 2023-03-03T08:05:21.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~84s 2023-03-03T08:05:27.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~70s 2023-03-03T08:05:27.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~51s 2023-03-03T08:05:27.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~32s 2023-03-03T08:05:27.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~90s 2023-03-03T08:05:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:32.645+0000 7fcbdffff700 1 -- 172.21.15.98:0/915097959 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 18) v1 ==== 86575+0+0 (secure 0 0 0) 0x7fcbf0052290 con 0x7fcbf80b7fd0 2023-03-03T08:05:34.096 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~77s 2023-03-03T08:05:34.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~57s 2023-03-03T08:05:34.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~38s 2023-03-03T08:05:34.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~96s 2023-03-03T08:05:34.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:05:34.851 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:05:40.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~83s 2023-03-03T08:05:40.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~63s 2023-03-03T08:05:40.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~44s 2023-03-03T08:05:40.409 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~103s 2023-03-03T08:05:46.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~89s 2023-03-03T08:05:46.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~70s 2023-03-03T08:05:46.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~51s 2023-03-03T08:05:46.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~109s 2023-03-03T08:05:53.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~96s 2023-03-03T08:05:53.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~76s 2023-03-03T08:05:53.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~57s 2023-03-03T08:05:53.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~115s 2023-03-03T08:05:54.647 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-03-03T08:05:54.650 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-03-03T08:05:54.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:05:54.788 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-03-03T08:05:54.925 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-03-03T08:05:54.926 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:05:54.926 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:05:54.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:05:55.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.236+0000 7fedd91ff700 1 Processor -- start 2023-03-03T08:05:55.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.237+0000 7fedd91ff700 1 -- start start 2023-03-03T08:05:55.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.238+0000 7fedd91ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedd40b87c0 0x7fedd40cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:55.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.238+0000 7fedd91ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fedd40b5410 0x7fedd40d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:55.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.238+0000 7fedd91ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd40d0ef0 con 0x7fedd40b8b90 2023-03-03T08:05:55.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.238+0000 7fedd91ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd40d0600 con 0x7fedd40b5410 2023-03-03T08:05:55.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.238+0000 7fedd91ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40cd970 con 0x7fedd40b87c0 2023-03-03T08:05:55.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.239+0000 7fedd2d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedd40b87c0 0x7fedd40cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:55.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.239+0000 7fedd359e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fedd40b5410 0x7fedd40d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:55.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.239+0000 7fedd2d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedd40b87c0 0x7fedd40cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:39662/0 (socket says 172.21.15.98:39662) 2023-03-03T08:05:55.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.239+0000 7fedd259c700 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fedd40b8b90 0x7fedd40b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.98:51526/0 (socket says 172.21.15.98:51526) 2023-03-03T08:05:55.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.239+0000 7fedd359e700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fedd40b5410 0x7fedd40d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.98:58070/0 (socket says 172.21.15.98:58070) 2023-03-03T08:05:55.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.239+0000 7fedd2d9d700 1 -- 172.21.15.98:0/2120068451 learned_addr learned my addr 172.21.15.98:0/2120068451 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:05:55.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.239+0000 7fedd359e700 1 -- 172.21.15.98:0/2120068451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedd40b87c0 msgr2=0x7fedd40cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:05:55.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.239+0000 7fedd359e700 1 --2- 172.21.15.98:0/2120068451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedd40b87c0 0x7fedd40cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.239+0000 7fedd359e700 1 -- 172.21.15.98:0/2120068451 >> v1:172.21.15.100:6789/0 conn(0x7fedd40b8b90 legacy=0x7fedd40b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:55.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.239+0000 7fedd359e700 1 -- 172.21.15.98:0/2120068451 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd40cdc50 con 0x7fedd40b5410 2023-03-03T08:05:55.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.240+0000 7fedd359e700 1 --2- 172.21.15.98:0/2120068451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedd40b5410 0x7fedd40d13d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fedcc009d90 tx=0x7fedcc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34ad073ec4ba0e8b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:55.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.240+0000 7fedd1d9b700 1 -- 172.21.15.98:0/2120068451 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc00e070 con 0x7fedd40b5410 2023-03-03T08:05:55.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.240+0000 7fedd1d9b700 1 -- 172.21.15.98:0/2120068451 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedcc00bb20 con 0x7fedd40b5410 2023-03-03T08:05:55.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.240+0000 7fedd1d9b700 1 -- 172.21.15.98:0/2120068451 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc012430 con 0x7fedd40b5410 2023-03-03T08:05:55.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.241+0000 7fedd91ff700 1 -- 172.21.15.98:0/2120068451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedd40b5410 msgr2=0x7fedd40d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:55.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.241+0000 7fedd91ff700 1 --2- 172.21.15.98:0/2120068451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedd40b5410 0x7fedd40d13d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fedcc009d90 tx=0x7fedcc00b330 comp rx=0 tx=0).stop 2023-03-03T08:05:55.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.241+0000 7fedd91ff700 1 -- 172.21.15.98:0/2120068451 shutdown_connections 2023-03-03T08:05:55.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.241+0000 7fedd91ff700 1 --2- 172.21.15.98:0/2120068451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedd40b5410 0x7fedd40d13d0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.241+0000 7fedd91ff700 1 --2- 172.21.15.98:0/2120068451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedd40b87c0 0x7fedd40cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.241+0000 7fedd91ff700 1 -- 172.21.15.98:0/2120068451 >> 172.21.15.98:0/2120068451 conn(0x7fedd406aee0 msgr2=0x7fedd406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:05:55.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.241+0000 7fedd91ff700 1 -- 172.21.15.98:0/2120068451 shutdown_connections 2023-03-03T08:05:55.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.241+0000 7fedd91ff700 1 -- 172.21.15.98:0/2120068451 wait complete. 2023-03-03T08:05:55.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.243+0000 7fedd91ff700 1 Processor -- start 2023-03-03T08:05:55.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.243+0000 7fedd91ff700 1 -- start start 2023-03-03T08:05:55.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.244+0000 7fedd91ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedd40b5410 0x7fedd40c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:55.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.244+0000 7fedd91ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd40b87c0 0x7fedd40c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:55.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.244+0000 7fedd91ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedd40b8b90 0x7fedd40c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:55.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.244+0000 7fedd91ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd41e5000 con 0x7fedd40b87c0 2023-03-03T08:05:55.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.244+0000 7fedd91ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd41e5250 con 0x7fedd40b8b90 2023-03-03T08:05:55.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.244+0000 7fedd91ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd41e5530 con 0x7fedd40b5410 2023-03-03T08:05:55.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.244+0000 7fedd259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd40b87c0 0x7fedd40c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:55.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.244+0000 7fedd2d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedd40b5410 0x7fedd40c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:55.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.244+0000 7fedd259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd40b87c0 0x7fedd40c6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60272/0 (socket says 172.21.15.98:60272) 2023-03-03T08:05:55.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.244+0000 7fedd2d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedd40b5410 0x7fedd40c9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.98:39670/0 (socket says 172.21.15.98:39670) 2023-03-03T08:05:55.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.244+0000 7fedd259c700 1 -- 172.21.15.98:0/4087793628 learned_addr learned my addr 172.21.15.98:0/4087793628 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:05:55.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.244+0000 7fedd359e700 1 --2- 172.21.15.98:0/4087793628 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedd40b8b90 0x7fedd40c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:55.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.245+0000 7fedd2d9d700 1 -- 172.21.15.98:0/4087793628 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedd40b8b90 msgr2=0x7fedd40c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:55.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.245+0000 7fedd2d9d700 1 --2- 172.21.15.98:0/4087793628 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedd40b8b90 0x7fedd40c72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.245+0000 7fedd2d9d700 1 -- 172.21.15.98:0/4087793628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd40b87c0 msgr2=0x7fedd40c6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:55.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.245+0000 7fedd2d9d700 1 --2- 172.21.15.98:0/4087793628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd40b87c0 0x7fedd40c6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.245+0000 7fedd2d9d700 1 -- 172.21.15.98:0/4087793628 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedcc009a70 con 0x7fedd40b5410 2023-03-03T08:05:55.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.245+0000 7fedd259c700 1 --2- 172.21.15.98:0/4087793628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd40b87c0 0x7fedd40c6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:05:55.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.245+0000 7fedd2d9d700 1 --2- 172.21.15.98:0/4087793628 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedd40b5410 0x7fedd40c9860 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fedc400e8e0 tx=0x7fedc400ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:55.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.245+0000 7fedbb7fe700 1 -- 172.21.15.98:0/4087793628 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc4013680 con 0x7fedd40b5410 2023-03-03T08:05:55.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.246+0000 7fedbb7fe700 1 -- 172.21.15.98:0/4087793628 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc40137e0 con 0x7fedd40b5410 2023-03-03T08:05:55.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.246+0000 7fedd91ff700 1 -- 172.21.15.98:0/4087793628 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd40cdf30 con 0x7fedd40b5410 2023-03-03T08:05:55.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.246+0000 7fedbb7fe700 1 -- 172.21.15.98:0/4087793628 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc4013b70 con 0x7fedd40b5410 2023-03-03T08:05:55.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.246+0000 7fedd91ff700 1 -- 172.21.15.98:0/4087793628 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd4073a80 con 0x7fedd40b5410 2023-03-03T08:05:55.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.247+0000 7fedd91ff700 1 -- 172.21.15.98:0/4087793628 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd40ce0c0 con 0x7fedd40b5410 2023-03-03T08:05:55.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.248+0000 7fedbb7fe700 1 -- 172.21.15.98:0/4087793628 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 18) v1 ==== 86575+0+0 (secure 0 0 0) 0x7fedc4019030 con 0x7fedd40b5410 2023-03-03T08:05:55.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.249+0000 7fedbb7fe700 1 --2- 172.21.15.98:0/4087793628 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fedc0068410 0x7fedc006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:55.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.250+0000 7fedbb7fe700 1 -- 172.21.15.98:0/4087793628 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7fedc4086230 con 0x7fedd40b5410 2023-03-03T08:05:55.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.250+0000 7fedd259c700 1 --2- 172.21.15.98:0/4087793628 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fedc0068410 0x7fedc006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:55.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.251+0000 7fedd259c700 1 --2- 172.21.15.98:0/4087793628 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fedc0068410 0x7fedc006a8c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fedd40c9100 tx=0x7fedc8007f90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:55.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.253+0000 7fedbb7fe700 1 -- 172.21.15.98:0/4087793628 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fedc40527a0 con 0x7fedd40b5410 2023-03-03T08:05:55.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.445+0000 7fedd91ff700 1 -- 172.21.15.98:0/4087793628 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedd40729b0 con 0x7fedd40b5410 2023-03-03T08:05:55.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.447+0000 7fedbb7fe700 1 -- 172.21.15.98:0/4087793628 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+18380 (secure 0 0 0) 0x7fedc4055dc0 con 0x7fedd40b5410 2023-03-03T08:05:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:05:55.448 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":114,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:04:52.583483+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:03:11.891955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-03-03T08:03:12.507916+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6832/2374561485":"2023-03-04T08:04:52.583414+0000","172.21.15.98:6835/2695503624":"2023-03-04T08:04:32.578044+0000","172.21.15.98:6833/2695503624":"2023-03-04T08:04:32.578044+0000","172.21.15.98:6834/2374561485":"2023-03-04T08:04:52.583414+0000","172.21.15.100:6835/2660525287":"2023-03-04T08:04:12.571647+0000","172.21.15.100:6833/2660525287":"2023-03-04T08:04:12.571647+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:05:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.449+0000 7fedd91ff700 1 -- 172.21.15.98:0/4087793628 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fedc0068410 msgr2=0x7fedc006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.449+0000 7fedd91ff700 1 --2- 172.21.15.98:0/4087793628 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fedc0068410 0x7fedc006a8c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fedd40c9100 tx=0x7fedc8007f90 comp rx=0 tx=0).stop 2023-03-03T08:05:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.449+0000 7fedd91ff700 1 -- 172.21.15.98:0/4087793628 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedd40b5410 msgr2=0x7fedd40c9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.449+0000 7fedd91ff700 1 --2- 172.21.15.98:0/4087793628 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedd40b5410 0x7fedd40c9860 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fedc400e8e0 tx=0x7fedc400ebf0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.449+0000 7fedd91ff700 1 -- 172.21.15.98:0/4087793628 shutdown_connections 2023-03-03T08:05:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.449+0000 7fedd91ff700 1 --2- 172.21.15.98:0/4087793628 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedd40b8b90 0x7fedd40c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.449+0000 7fedd91ff700 1 --2- 172.21.15.98:0/4087793628 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedd40b5410 0x7fedd40c9860 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.449+0000 7fedd91ff700 1 --2- 172.21.15.98:0/4087793628 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fedc0068410 0x7fedc006a8c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.449+0000 7fedd91ff700 1 --2- 172.21.15.98:0/4087793628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd40b87c0 0x7fedd40c6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.449+0000 7fedd91ff700 1 -- 172.21.15.98:0/4087793628 >> 172.21.15.98:0/4087793628 conn(0x7fedd406aee0 msgr2=0x7fedd40b7180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:05:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.449+0000 7fedd91ff700 1 -- 172.21.15.98:0/4087793628 shutdown_connections 2023-03-03T08:05:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.449+0000 7fedd91ff700 1 -- 172.21.15.98:0/4087793628 wait complete. 2023-03-03T08:05:55.478 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T08:03:11.891955+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T08:03:12.507916+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:05:55.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:05:55.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.725+0000 7f928119d700 1 Processor -- start 2023-03-03T08:05:55.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.726+0000 7f928119d700 1 -- start start 2023-03-03T08:05:55.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.727+0000 7f928119d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f927c1609b0 0x7f927c160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:55.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.727+0000 7f928119d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f927c0b4aa0 0x7f927c0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:55.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.727+0000 7f928119d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927c1612c0 con 0x7f927c0cf570 2023-03-03T08:05:55.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.727+0000 7f928119d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f927c0cebb0 con 0x7f927c1609b0 2023-03-03T08:05:55.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.727+0000 7f928119d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f927c0b5460 con 0x7f927c0b4aa0 2023-03-03T08:05:55.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.727+0000 7f927b7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f927c0cf570 0x7f927c15d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48736/0 (socket says 172.21.15.98:48736) 2023-03-03T08:05:55.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.727+0000 7f927b7fe700 1 -- 172.21.15.98:0/2239786333 learned_addr learned my addr 172.21.15.98:0/2239786333 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:05:55.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.727+0000 7f927bfff700 1 --2- 172.21.15.98:0/2239786333 >> v2:172.21.15.98:3300/0 conn(0x7f927c1609b0 0x7f927c160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:55.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.728+0000 7f928099c700 1 --2- 172.21.15.98:0/2239786333 >> v2:172.21.15.100:3300/0 conn(0x7f927c0b4aa0 0x7f927c0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:55.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.728+0000 7f927affd700 1 -- 172.21.15.98:0/2239786333 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820406733 0 0) 0x7f927c1612c0 con 0x7f927c0cf570 2023-03-03T08:05:55.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.728+0000 7f927affd700 1 -- 172.21.15.98:0/2239786333 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92640035b0 con 0x7f927c0cf570 2023-03-03T08:05:55.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.728+0000 7f927bfff700 1 -- 172.21.15.98:0/2239786333 >> v2:172.21.15.100:3300/0 conn(0x7f927c0b4aa0 msgr2=0x7f927c0b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:05:55.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.728+0000 7f927bfff700 1 --2- 172.21.15.98:0/2239786333 >> v2:172.21.15.100:3300/0 conn(0x7f927c0b4aa0 0x7f927c0b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.728+0000 7f927bfff700 1 -- 172.21.15.98:0/2239786333 >> v1:172.21.15.98:6789/0 conn(0x7f927c0cf570 legacy=0x7f927c15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:55.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.728+0000 7f927bfff700 1 -- 172.21.15.98:0/2239786333 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927c15dd00 con 0x7f927c1609b0 2023-03-03T08:05:55.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.728+0000 7f927bfff700 1 --2- 172.21.15.98:0/2239786333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f927c1609b0 0x7f927c160d80 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f927c06d4d0 tx=0x7f926c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de4105658491f44e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:55.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.729+0000 7f927affd700 1 -- 172.21.15.98:0/2239786333 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c011830 con 0x7f927c1609b0 2023-03-03T08:05:55.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.729+0000 7f927affd700 1 -- 172.21.15.98:0/2239786333 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926c011990 con 0x7f927c1609b0 2023-03-03T08:05:55.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.729+0000 7f927affd700 1 -- 172.21.15.98:0/2239786333 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c011d20 con 0x7f927c1609b0 2023-03-03T08:05:55.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.729+0000 7f928119d700 1 -- 172.21.15.98:0/2239786333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f927c1609b0 msgr2=0x7f927c160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:55.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.729+0000 7f928119d700 1 --2- 172.21.15.98:0/2239786333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f927c1609b0 0x7f927c160d80 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f927c06d4d0 tx=0x7f926c009b20 comp rx=0 tx=0).stop 2023-03-03T08:05:55.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.729+0000 7f928119d700 1 -- 172.21.15.98:0/2239786333 shutdown_connections 2023-03-03T08:05:55.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.729+0000 7f928119d700 1 --2- 172.21.15.98:0/2239786333 >> v2:172.21.15.100:3300/0 conn(0x7f927c0b4aa0 0x7f927c0b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.729+0000 7f928119d700 1 --2- 172.21.15.98:0/2239786333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f927c1609b0 0x7f927c160d80 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.729+0000 7f928119d700 1 -- 172.21.15.98:0/2239786333 >> 172.21.15.98:0/2239786333 conn(0x7f927c06b880 msgr2=0x7f927c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:05:55.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.729+0000 7f928119d700 1 -- 172.21.15.98:0/2239786333 shutdown_connections 2023-03-03T08:05:55.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.729+0000 7f928119d700 1 -- 172.21.15.98:0/2239786333 wait complete. 2023-03-03T08:05:55.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.731+0000 7f928119d700 1 Processor -- start 2023-03-03T08:05:55.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.732+0000 7f928119d700 1 -- start start 2023-03-03T08:05:55.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.733+0000 7f928119d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f927c0b4aa0 0x7f927c14f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:55.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.733+0000 7f928119d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f927c0cf570 0x7f927c1503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:55.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.733+0000 7f928119d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f927c1609b0 0x7f927c14e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:55.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.733+0000 7f928119d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c150c20 con 0x7f927c1609b0 2023-03-03T08:05:55.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.733+0000 7f928119d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c150f00 con 0x7f927c0b4aa0 2023-03-03T08:05:55.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.733+0000 7f928119d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927c15aab0 con 0x7f927c0cf570 2023-03-03T08:05:55.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.733+0000 7f928099c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f927c1609b0 0x7f927c14e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:55.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.733+0000 7f928099c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f927c1609b0 0x7f927c14e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60294/0 (socket says 172.21.15.98:60294) 2023-03-03T08:05:55.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.733+0000 7f928099c700 1 -- 172.21.15.98:0/512054087 learned_addr learned my addr 172.21.15.98:0/512054087 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:05:55.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.733+0000 7f927bfff700 1 --2- 172.21.15.98:0/512054087 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f927c0b4aa0 0x7f927c14f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:55.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.733+0000 7f927b7fe700 1 --2- 172.21.15.98:0/512054087 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f927c0cf570 0x7f927c1503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:55.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.734+0000 7f928099c700 1 -- 172.21.15.98:0/512054087 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f927c0cf570 msgr2=0x7f927c1503e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:55.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.734+0000 7f928099c700 1 --2- 172.21.15.98:0/512054087 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f927c0cf570 0x7f927c1503e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.734+0000 7f928099c700 1 -- 172.21.15.98:0/512054087 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f927c0b4aa0 msgr2=0x7f927c14f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:55.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.734+0000 7f928099c700 1 --2- 172.21.15.98:0/512054087 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f927c0b4aa0 0x7f927c14f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.734+0000 7f928099c700 1 -- 172.21.15.98:0/512054087 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c0094a0 con 0x7f927c1609b0 2023-03-03T08:05:55.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.734+0000 7f927bfff700 1 --2- 172.21.15.98:0/512054087 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f927c0b4aa0 0x7f927c14f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:05:55.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.734+0000 7f928099c700 1 --2- 172.21.15.98:0/512054087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f927c1609b0 0x7f927c14e6e0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f927400efc0 tx=0x7f927400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:55.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.734+0000 7f9278ff9700 1 -- 172.21.15.98:0/512054087 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927400cd40 con 0x7f927c1609b0 2023-03-03T08:05:55.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.734+0000 7f9278ff9700 1 -- 172.21.15.98:0/512054087 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927400f040 con 0x7f927c1609b0 2023-03-03T08:05:55.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.734+0000 7f928119d700 1 -- 172.21.15.98:0/512054087 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927c0cda60 con 0x7f927c1609b0 2023-03-03T08:05:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.734+0000 7f9278ff9700 1 -- 172.21.15.98:0/512054087 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92740146c0 con 0x7f927c1609b0 2023-03-03T08:05:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.734+0000 7f928119d700 1 -- 172.21.15.98:0/512054087 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927c0cd780 con 0x7f927c1609b0 2023-03-03T08:05:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.736+0000 7f928119d700 1 -- 172.21.15.98:0/512054087 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f927c0cdd40 con 0x7f927c1609b0 2023-03-03T08:05:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.737+0000 7f9278ff9700 1 -- 172.21.15.98:0/512054087 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 86575+0+0 (secure 0 0 0) 0x7f9274014820 con 0x7f927c1609b0 2023-03-03T08:05:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.739+0000 7f9278ff9700 1 --2- 172.21.15.98:0/512054087 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9268068410 0x7f926806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.739+0000 7f9278ff9700 1 -- 172.21.15.98:0/512054087 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f9274086b10 con 0x7f927c1609b0 2023-03-03T08:05:55.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.740+0000 7f927bfff700 1 --2- 172.21.15.98:0/512054087 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9268068410 0x7f926806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:55.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.740+0000 7f927bfff700 1 --2- 172.21.15.98:0/512054087 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9268068410 0x7f926806a8c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f926c0090b0 tx=0x7f926c01c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:55.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.744+0000 7f9278ff9700 1 -- 172.21.15.98:0/512054087 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9274053080 con 0x7f927c1609b0 2023-03-03T08:05:55.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.937+0000 7f928119d700 1 -- 172.21.15.98:0/512054087 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f927c15afa0 con 0x7f927c1609b0 2023-03-03T08:05:55.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.938+0000 7f9278ff9700 1 -- 172.21.15.98:0/512054087 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9274018020 con 0x7f927c1609b0 2023-03-03T08:05:55.939 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:05:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.940+0000 7f928119d700 1 -- 172.21.15.98:0/512054087 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9268068410 msgr2=0x7f926806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.940+0000 7f928119d700 1 --2- 172.21.15.98:0/512054087 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9268068410 0x7f926806a8c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f926c0090b0 tx=0x7f926c01c040 comp rx=0 tx=0).stop 2023-03-03T08:05:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.940+0000 7f928119d700 1 -- 172.21.15.98:0/512054087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f927c1609b0 msgr2=0x7f927c14e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.940+0000 7f928119d700 1 --2- 172.21.15.98:0/512054087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f927c1609b0 0x7f927c14e6e0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f927400efc0 tx=0x7f927400c3a0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.940+0000 7f928119d700 1 -- 172.21.15.98:0/512054087 shutdown_connections 2023-03-03T08:05:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.940+0000 7f928119d700 1 --2- 172.21.15.98:0/512054087 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9268068410 0x7f926806a8c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.940+0000 7f928119d700 1 --2- 172.21.15.98:0/512054087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f927c1609b0 0x7f927c14e6e0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.940+0000 7f928119d700 1 --2- 172.21.15.98:0/512054087 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f927c0b4aa0 0x7f927c14f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.940+0000 7f928119d700 1 --2- 172.21.15.98:0/512054087 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f927c0cf570 0x7f927c1503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.940+0000 7f928119d700 1 -- 172.21.15.98:0/512054087 >> 172.21.15.98:0/512054087 conn(0x7f927c06b880 msgr2=0x7f927c15f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:05:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.940+0000 7f928119d700 1 -- 172.21.15.98:0/512054087 shutdown_connections 2023-03-03T08:05:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:55.941+0000 7f928119d700 1 -- 172.21.15.98:0/512054087 wait complete. 2023-03-03T08:05:55.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:05:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.214+0000 7f8b9139d700 1 Processor -- start 2023-03-03T08:05:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.215+0000 7f8b9139d700 1 -- start start 2023-03-03T08:05:56.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.216+0000 7f8b9139d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b8c0cf570 0x7f8b8c15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:56.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.216+0000 7f8b9139d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8b8c15db30 0x7f8b8c0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:56.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.216+0000 7f8b9139d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8c0cd780 con 0x7f8b8c1609b0 2023-03-03T08:05:56.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.216+0000 7f8b9139d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8c0cebb0 con 0x7f8b8c15db30 2023-03-03T08:05:56.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.216+0000 7f8b9139d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0cda60 con 0x7f8b8c0cf570 2023-03-03T08:05:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.216+0000 7f8b8bfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8b8c1609b0 0x7f8b8c160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48738/0 (socket says 172.21.15.98:48738) 2023-03-03T08:05:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.216+0000 7f8b90b9c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8b8c15db30 0x7f8b8c0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.216+0000 7f8b8bfff700 1 -- 172.21.15.98:0/3948033193 learned_addr learned my addr 172.21.15.98:0/3948033193 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:05:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.216+0000 7f8b8b7fe700 1 --2- 172.21.15.98:0/3948033193 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b8c0cf570 0x7f8b8c15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.217+0000 7f8b90b9c700 1 -- 172.21.15.98:0/3948033193 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b8c0cf570 msgr2=0x7f8b8c15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:05:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.217+0000 7f8b90b9c700 1 --2- 172.21.15.98:0/3948033193 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b8c0cf570 0x7f8b8c15d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.217+0000 7f8b90b9c700 1 -- 172.21.15.98:0/3948033193 >> v1:172.21.15.98:6789/0 conn(0x7f8b8c1609b0 legacy=0x7f8b8c160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.217+0000 7f8b90b9c700 1 -- 172.21.15.98:0/3948033193 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8c0cdd40 con 0x7f8b8c15db30 2023-03-03T08:05:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.217+0000 7f8b8affd700 1 -- 172.21.15.98:0/3948033193 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560449026 0 0) 0x7f8b8c0cd780 con 0x7f8b8c1609b0 2023-03-03T08:05:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.217+0000 7f8b90b9c700 1 --2- 172.21.15.98:0/3948033193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b8c15db30 0x7f8b8c0b4aa0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f8b84009d90 tx=0x7f8b8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f37b234264bd95b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.217+0000 7f8b8affd700 1 -- 172.21.15.98:0/3948033193 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8400e070 con 0x7f8b8c15db30 2023-03-03T08:05:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.218+0000 7f8b8affd700 1 -- 172.21.15.98:0/3948033193 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8400bb20 con 0x7f8b8c15db30 2023-03-03T08:05:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.218+0000 7f8b8affd700 1 -- 172.21.15.98:0/3948033193 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b84012430 con 0x7f8b8c15db30 2023-03-03T08:05:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.218+0000 7f8b9139d700 1 -- 172.21.15.98:0/3948033193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b8c15db30 msgr2=0x7f8b8c0b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.218+0000 7f8b9139d700 1 --2- 172.21.15.98:0/3948033193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b8c15db30 0x7f8b8c0b4aa0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f8b84009d90 tx=0x7f8b8400b330 comp rx=0 tx=0).stop 2023-03-03T08:05:56.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.218+0000 7f8b9139d700 1 -- 172.21.15.98:0/3948033193 shutdown_connections 2023-03-03T08:05:56.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.218+0000 7f8b9139d700 1 --2- 172.21.15.98:0/3948033193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b8c15db30 0x7f8b8c0b4aa0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.218+0000 7f8b9139d700 1 --2- 172.21.15.98:0/3948033193 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b8c0cf570 0x7f8b8c15d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.218+0000 7f8b9139d700 1 -- 172.21.15.98:0/3948033193 >> 172.21.15.98:0/3948033193 conn(0x7f8b8c06b880 msgr2=0x7f8b8c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:05:56.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.218+0000 7f8b9139d700 1 -- 172.21.15.98:0/3948033193 shutdown_connections 2023-03-03T08:05:56.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.218+0000 7f8b9139d700 1 -- 172.21.15.98:0/3948033193 wait complete. 2023-03-03T08:05:56.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.220+0000 7f8b9139d700 1 Processor -- start 2023-03-03T08:05:56.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.220+0000 7f8b9139d700 1 -- start start 2023-03-03T08:05:56.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.221+0000 7f8b9139d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b8c0cf570 0x7f8b8c1f15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:56.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.221+0000 7f8b9139d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b8c15db30 0x7f8b8c15be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:56.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.221+0000 7f8b9139d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b8c1609b0 0x7f8b8c15c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:56.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.221+0000 7f8b9139d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c152010 con 0x7f8b8c1609b0 2023-03-03T08:05:56.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.221+0000 7f8b9139d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c1e6910 con 0x7f8b8c15db30 2023-03-03T08:05:56.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.221+0000 7f8b9139d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c1e6bf0 con 0x7f8b8c0cf570 2023-03-03T08:05:56.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.222+0000 7f8b90b9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b8c1609b0 0x7f8b8c15c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.222+0000 7f8b90b9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b8c1609b0 0x7f8b8c15c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60318/0 (socket says 172.21.15.98:60318) 2023-03-03T08:05:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.222+0000 7f8b90b9c700 1 -- 172.21.15.98:0/3316173343 learned_addr learned my addr 172.21.15.98:0/3316173343 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:05:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.222+0000 7f8b8bfff700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b8c0cf570 0x7f8b8c1f15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.222+0000 7f8b90b9c700 1 -- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b8c0cf570 msgr2=0x7f8b8c1f15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.222+0000 7f8b90b9c700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b8c0cf570 0x7f8b8c1f15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.222+0000 7f8b90b9c700 1 -- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b8c15db30 msgr2=0x7f8b8c15be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.222+0000 7f8b8b7fe700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b8c15db30 0x7f8b8c15be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.222+0000 7f8b90b9c700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b8c15db30 0x7f8b8c15be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.222+0000 7f8b90b9c700 1 -- 172.21.15.98:0/3316173343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b84009a70 con 0x7f8b8c1609b0 2023-03-03T08:05:56.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.222+0000 7f8b8bfff700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b8c0cf570 0x7f8b8c1f15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:05:56.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.222+0000 7f8b8b7fe700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b8c15db30 0x7f8b8c15be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:05:56.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.223+0000 7f8b90b9c700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b8c1609b0 0x7f8b8c15c380 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c074380 tx=0x7f8b84004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:56.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.223+0000 7f8b88ff9700 1 -- 172.21.15.98:0/3316173343 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b840098f0 con 0x7f8b8c1609b0 2023-03-03T08:05:56.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.223+0000 7f8b88ff9700 1 -- 172.21.15.98:0/3316173343 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b84020070 con 0x7f8b8c1609b0 2023-03-03T08:05:56.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.223+0000 7f8b88ff9700 1 -- 172.21.15.98:0/3316173343 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b840276a0 con 0x7f8b8c1609b0 2023-03-03T08:05:56.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.223+0000 7f8b9139d700 1 -- 172.21.15.98:0/3316173343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b8c15d0d0 con 0x7f8b8c1609b0 2023-03-03T08:05:56.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.224+0000 7f8b9139d700 1 -- 172.21.15.98:0/3316173343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8c15cdf0 con 0x7f8b8c1609b0 2023-03-03T08:05:56.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.225+0000 7f8b9139d700 1 -- 172.21.15.98:0/3316173343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b8c0b4fe0 con 0x7f8b8c1609b0 2023-03-03T08:05:56.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.226+0000 7f8b88ff9700 1 -- 172.21.15.98:0/3316173343 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 86575+0+0 (secure 0 0 0) 0x7f8b8402e070 con 0x7f8b8c1609b0 2023-03-03T08:05:56.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.227+0000 7f8b88ff9700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8b78068410 0x7f8b7806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:56.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.227+0000 7f8b88ff9700 1 -- 172.21.15.98:0/3316173343 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f8b84022070 con 0x7f8b8c1609b0 2023-03-03T08:05:56.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.228+0000 7f8b8bfff700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8b78068410 0x7f8b7806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:56.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.228+0000 7f8b8bfff700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8b78068410 0x7f8b7806a8c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c007fd0 tx=0x7f8b7c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:56.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.231+0000 7f8b88ff9700 1 -- 172.21.15.98:0/3316173343 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b840542f0 con 0x7f8b8c1609b0 2023-03-03T08:05:56.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.422+0000 7f8b9139d700 1 -- 172.21.15.98:0/3316173343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8b8c068290 con 0x7f8b8c1609b0 2023-03-03T08:05:56.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.423+0000 7f8b88ff9700 1 -- 172.21.15.98:0/3316173343 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v114) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8b8402c020 con 0x7f8b8c1609b0 2023-03-03T08:05:56.425 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:05:56.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.426+0000 7f8b9139d700 1 -- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8b78068410 msgr2=0x7f8b7806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:56.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.426+0000 7f8b9139d700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8b78068410 0x7f8b7806a8c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c007fd0 tx=0x7f8b7c009040 comp rx=0 tx=0).stop 2023-03-03T08:05:56.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.426+0000 7f8b9139d700 1 -- 172.21.15.98:0/3316173343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b8c1609b0 msgr2=0x7f8b8c15c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:56.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.426+0000 7f8b9139d700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b8c1609b0 0x7f8b8c15c380 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c074380 tx=0x7f8b84004ad0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.426+0000 7f8b9139d700 1 -- 172.21.15.98:0/3316173343 shutdown_connections 2023-03-03T08:05:56.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.426+0000 7f8b9139d700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8b78068410 0x7f8b7806a8c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.426+0000 7f8b9139d700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b8c1609b0 0x7f8b8c15c380 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.426+0000 7f8b9139d700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b8c0cf570 0x7f8b8c1f15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.426+0000 7f8b9139d700 1 --2- 172.21.15.98:0/3316173343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b8c15db30 0x7f8b8c15be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.426+0000 7f8b9139d700 1 -- 172.21.15.98:0/3316173343 >> 172.21.15.98:0/3316173343 conn(0x7f8b8c06b880 msgr2=0x7f8b8c15e670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:05:56.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.426+0000 7f8b9139d700 1 -- 172.21.15.98:0/3316173343 shutdown_connections 2023-03-03T08:05:56.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.426+0000 7f8b9139d700 1 -- 172.21.15.98:0/3316173343 wait complete. 2023-03-03T08:05:56.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T08:05:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.702+0000 7efff4d9d700 1 Processor -- start 2023-03-03T08:05:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.702+0000 7efff4d9d700 1 -- start start 2023-03-03T08:05:56.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.703+0000 7efff4d9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efff01609b0 0x7efff0160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:56.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.704+0000 7efff4d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efff00b4aa0 0x7efff00b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:56.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.704+0000 7efff4d9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff01612c0 con 0x7efff00cf570 2023-03-03T08:05:56.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.704+0000 7efff4d9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efff00cebb0 con 0x7efff00b4aa0 2023-03-03T08:05:56.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.704+0000 7efff4d9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efff00b5460 con 0x7efff01609b0 2023-03-03T08:05:56.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.704+0000 7effeffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efff00b4aa0 0x7efff00b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:56.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.704+0000 7effeffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efff00b4aa0 0x7efff00b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60330/0 (socket says 172.21.15.98:60330) 2023-03-03T08:05:56.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.704+0000 7effeffff700 1 -- 172.21.15.98:0/2875403478 learned_addr learned my addr 172.21.15.98:0/2875403478 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:05:56.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.704+0000 7effef7fe700 1 --2- 172.21.15.98:0/2875403478 >> v2:172.21.15.100:3300/0 conn(0x7efff01609b0 0x7efff0160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:56.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.705+0000 7effeffff700 1 -- 172.21.15.98:0/2875403478 >> v2:172.21.15.100:3300/0 conn(0x7efff01609b0 msgr2=0x7efff0160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:05:56.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.705+0000 7effeffff700 1 --2- 172.21.15.98:0/2875403478 >> v2:172.21.15.100:3300/0 conn(0x7efff01609b0 0x7efff0160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.705+0000 7effeffff700 1 -- 172.21.15.98:0/2875403478 >> v1:172.21.15.100:6789/0 conn(0x7efff00cf570 legacy=0x7efff015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:56.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.705+0000 7effeffff700 1 -- 172.21.15.98:0/2875403478 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff015dd00 con 0x7efff00b4aa0 2023-03-03T08:05:56.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.705+0000 7effee7fc700 1 -- 172.21.15.98:0/2875403478 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388264718 0 0) 0x7efff01612c0 con 0x7efff00cf570 2023-03-03T08:05:56.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.705+0000 7effeffff700 1 --2- 172.21.15.98:0/2875403478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efff00b4aa0 0x7efff00b4f20 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7efff00742b0 tx=0x7effe800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82dcad07c0572c71 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:56.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.705+0000 7effee7fc700 1 -- 172.21.15.98:0/2875403478 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe800e070 con 0x7efff00b4aa0 2023-03-03T08:05:56.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.705+0000 7effee7fc700 1 -- 172.21.15.98:0/2875403478 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe800b910 con 0x7efff00b4aa0 2023-03-03T08:05:56.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.705+0000 7effee7fc700 1 -- 172.21.15.98:0/2875403478 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe800bcc0 con 0x7efff00b4aa0 2023-03-03T08:05:56.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.706+0000 7efff4d9d700 1 -- 172.21.15.98:0/2875403478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efff00b4aa0 msgr2=0x7efff00b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:56.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.706+0000 7efff4d9d700 1 --2- 172.21.15.98:0/2875403478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efff00b4aa0 0x7efff00b4f20 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7efff00742b0 tx=0x7effe800b330 comp rx=0 tx=0).stop 2023-03-03T08:05:56.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.706+0000 7efff4d9d700 1 -- 172.21.15.98:0/2875403478 shutdown_connections 2023-03-03T08:05:56.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.706+0000 7efff4d9d700 1 --2- 172.21.15.98:0/2875403478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efff00b4aa0 0x7efff00b4f20 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.706+0000 7efff4d9d700 1 --2- 172.21.15.98:0/2875403478 >> v2:172.21.15.100:3300/0 conn(0x7efff01609b0 0x7efff0160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.706+0000 7efff4d9d700 1 -- 172.21.15.98:0/2875403478 >> 172.21.15.98:0/2875403478 conn(0x7efff006b880 msgr2=0x7efff00cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:05:56.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.706+0000 7efff4d9d700 1 -- 172.21.15.98:0/2875403478 shutdown_connections 2023-03-03T08:05:56.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.706+0000 7efff4d9d700 1 -- 172.21.15.98:0/2875403478 wait complete. 2023-03-03T08:05:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.708+0000 7efff4d9d700 1 Processor -- start 2023-03-03T08:05:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.708+0000 7efff4d9d700 1 -- start start 2023-03-03T08:05:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.709+0000 7efff4d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efff00b4aa0 0x7efff0148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.709+0000 7efff4d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efff00cf570 0x7efff01f5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.709+0000 7efff4d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efff01609b0 0x7efff00d16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.709+0000 7efff4d9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff00b7090 con 0x7efff01609b0 2023-03-03T08:05:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.709+0000 7efff4d9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff00b7610 con 0x7efff00cf570 2023-03-03T08:05:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.709+0000 7efff4d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff01f63d0 con 0x7efff00b4aa0 2023-03-03T08:05:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.710+0000 7effeffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efff01609b0 0x7efff00d16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.710+0000 7effeffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efff01609b0 0x7efff00d16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60338/0 (socket says 172.21.15.98:60338) 2023-03-03T08:05:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.710+0000 7effeffff700 1 -- 172.21.15.98:0/3033294563 learned_addr learned my addr 172.21.15.98:0/3033294563 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:05:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.710+0000 7effef7fe700 1 --2- 172.21.15.98:0/3033294563 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efff00b4aa0 0x7efff0148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.710+0000 7effeeffd700 1 --2- 172.21.15.98:0/3033294563 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efff00cf570 0x7efff01f5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:56.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.710+0000 7effeffff700 1 -- 172.21.15.98:0/3033294563 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efff00b4aa0 msgr2=0x7efff0148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:56.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.710+0000 7effeffff700 1 --2- 172.21.15.98:0/3033294563 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efff00b4aa0 0x7efff0148eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.710+0000 7effeffff700 1 -- 172.21.15.98:0/3033294563 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efff00cf570 msgr2=0x7efff01f5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:56.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.710+0000 7effeffff700 1 --2- 172.21.15.98:0/3033294563 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efff00cf570 0x7efff01f5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.710+0000 7effeffff700 1 -- 172.21.15.98:0/3033294563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe8009b30 con 0x7efff01609b0 2023-03-03T08:05:56.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.710+0000 7effeffff700 1 --2- 172.21.15.98:0/3033294563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efff01609b0 0x7efff00d16a0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7efff0074380 tx=0x7effe8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:56.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.711+0000 7effd7fff700 1 -- 172.21.15.98:0/3033294563 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe80099b0 con 0x7efff01609b0 2023-03-03T08:05:56.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.711+0000 7effd7fff700 1 -- 172.21.15.98:0/3033294563 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe80092e0 con 0x7efff01609b0 2023-03-03T08:05:56.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.711+0000 7efff4d9d700 1 -- 172.21.15.98:0/3033294563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff00cda60 con 0x7efff01609b0 2023-03-03T08:05:56.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.711+0000 7effd7fff700 1 -- 172.21.15.98:0/3033294563 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe801c660 con 0x7efff01609b0 2023-03-03T08:05:56.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.711+0000 7efff4d9d700 1 -- 172.21.15.98:0/3033294563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff00cd780 con 0x7efff01609b0 2023-03-03T08:05:56.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.712+0000 7efff4d9d700 1 -- 172.21.15.98:0/3033294563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff00cdd40 con 0x7efff01609b0 2023-03-03T08:05:56.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.713+0000 7effd7fff700 1 -- 172.21.15.98:0/3033294563 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 86575+0+0 (secure 0 0 0) 0x7effe8022030 con 0x7efff01609b0 2023-03-03T08:05:56.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.715+0000 7effd7fff700 1 --2- 172.21.15.98:0/3033294563 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7effdc068410 0x7effdc06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:56.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.715+0000 7effd7fff700 1 -- 172.21.15.98:0/3033294563 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7effe80878a0 con 0x7efff01609b0 2023-03-03T08:05:56.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.716+0000 7effef7fe700 1 --2- 172.21.15.98:0/3033294563 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7effdc068410 0x7effdc06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:56.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.717+0000 7effef7fe700 1 --2- 172.21.15.98:0/3033294563 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7effdc068410 0x7effdc06a8c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7effe0009ad0 tx=0x7effe0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:56.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.720+0000 7effd7fff700 1 -- 172.21.15.98:0/3033294563 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7effe8053db0 con 0x7efff01609b0 2023-03-03T08:05:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.909+0000 7efff4d9d700 1 -- 172.21.15.98:0/3033294563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efff00b77a0 con 0x7efff01609b0 2023-03-03T08:05:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.910+0000 7effd7fff700 1 -- 172.21.15.98:0/3033294563 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7effe80573d0 con 0x7efff01609b0 2023-03-03T08:05:56.911 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T08:05:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.912+0000 7efff4d9d700 1 -- 172.21.15.98:0/3033294563 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7effdc068410 msgr2=0x7effdc06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.912+0000 7efff4d9d700 1 --2- 172.21.15.98:0/3033294563 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7effdc068410 0x7effdc06a8c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7effe0009ad0 tx=0x7effe0008040 comp rx=0 tx=0).stop 2023-03-03T08:05:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.912+0000 7efff4d9d700 1 -- 172.21.15.98:0/3033294563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efff01609b0 msgr2=0x7efff00d16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.912+0000 7efff4d9d700 1 --2- 172.21.15.98:0/3033294563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efff01609b0 0x7efff00d16a0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7efff0074380 tx=0x7effe8004ad0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.912+0000 7efff4d9d700 1 -- 172.21.15.98:0/3033294563 shutdown_connections 2023-03-03T08:05:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.912+0000 7efff4d9d700 1 --2- 172.21.15.98:0/3033294563 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7effdc068410 0x7effdc06a8c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.912+0000 7efff4d9d700 1 --2- 172.21.15.98:0/3033294563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efff01609b0 0x7efff00d16a0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.912+0000 7efff4d9d700 1 --2- 172.21.15.98:0/3033294563 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efff00b4aa0 0x7efff0148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.912+0000 7efff4d9d700 1 --2- 172.21.15.98:0/3033294563 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efff00cf570 0x7efff01f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.913+0000 7efff4d9d700 1 -- 172.21.15.98:0/3033294563 >> 172.21.15.98:0/3033294563 conn(0x7efff006b880 msgr2=0x7efff01530b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:05:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.913+0000 7efff4d9d700 1 -- 172.21.15.98:0/3033294563 shutdown_connections 2023-03-03T08:05:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:56.913+0000 7efff4d9d700 1 -- 172.21.15.98:0/3033294563 wait complete. 2023-03-03T08:05:56.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T08:05:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.187+0000 7fa477fff700 1 Processor -- start 2023-03-03T08:05:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.187+0000 7fa477fff700 1 -- start start 2023-03-03T08:05:57.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.188+0000 7fa477fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4700d0270 0x7fa4700cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:57.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.188+0000 7fa477fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa47006b4b0 0x7fa4700d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:57.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.188+0000 7fa477fff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4700cf980 con 0x7fa47015ca70 2023-03-03T08:05:57.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.188+0000 7fa477fff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4700cd5b0 con 0x7fa47006b4b0 2023-03-03T08:05:57.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.188+0000 7fa477fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4700d1800 con 0x7fa4700d0270 2023-03-03T08:05:57.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.189+0000 7fa475d9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa47015ca70 0x7fa4700cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48744/0 (socket says 172.21.15.98:48744) 2023-03-03T08:05:57.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.189+0000 7fa475d9b700 1 -- 172.21.15.98:0/916678786 learned_addr learned my addr 172.21.15.98:0/916678786 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:05:57.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.189+0000 7fa47659c700 1 --2- 172.21.15.98:0/916678786 >> v2:172.21.15.98:3300/0 conn(0x7fa47006b4b0 0x7fa4700d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:57.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.189+0000 7fa47559a700 1 --2- 172.21.15.98:0/916678786 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4700d0270 0x7fa4700cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:57.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.189+0000 7fa47659c700 1 -- 172.21.15.98:0/916678786 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4700d0270 msgr2=0x7fa4700cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:05:57.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.189+0000 7fa47659c700 1 --2- 172.21.15.98:0/916678786 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4700d0270 0x7fa4700cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.189+0000 7fa47659c700 1 -- 172.21.15.98:0/916678786 >> v1:172.21.15.98:6789/0 conn(0x7fa47015ca70 legacy=0x7fa4700cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:57.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.189+0000 7fa47659c700 1 -- 172.21.15.98:0/916678786 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4700cd900 con 0x7fa47006b4b0 2023-03-03T08:05:57.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.189+0000 7fa474d99700 1 -- 172.21.15.98:0/916678786 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140012466 0 0) 0x7fa4700cf980 con 0x7fa47015ca70 2023-03-03T08:05:57.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.190+0000 7fa47659c700 1 --2- 172.21.15.98:0/916678786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47006b4b0 0x7fa4700d1fc0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fa468009d90 tx=0x7fa46800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93fe1cf350475c3c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:57.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.190+0000 7fa474d99700 1 -- 172.21.15.98:0/916678786 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46800e070 con 0x7fa47006b4b0 2023-03-03T08:05:57.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.190+0000 7fa474d99700 1 -- 172.21.15.98:0/916678786 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46800bb20 con 0x7fa47006b4b0 2023-03-03T08:05:57.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.190+0000 7fa474d99700 1 -- 172.21.15.98:0/916678786 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa468012430 con 0x7fa47006b4b0 2023-03-03T08:05:57.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.190+0000 7fa477fff700 1 -- 172.21.15.98:0/916678786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47006b4b0 msgr2=0x7fa4700d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:57.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.190+0000 7fa477fff700 1 --2- 172.21.15.98:0/916678786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47006b4b0 0x7fa4700d1fc0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fa468009d90 tx=0x7fa46800b330 comp rx=0 tx=0).stop 2023-03-03T08:05:57.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.191+0000 7fa477fff700 1 -- 172.21.15.98:0/916678786 shutdown_connections 2023-03-03T08:05:57.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.191+0000 7fa477fff700 1 --2- 172.21.15.98:0/916678786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47006b4b0 0x7fa4700d1fc0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.191+0000 7fa477fff700 1 --2- 172.21.15.98:0/916678786 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4700d0270 0x7fa4700cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.191+0000 7fa477fff700 1 -- 172.21.15.98:0/916678786 >> 172.21.15.98:0/916678786 conn(0x7fa470073e70 msgr2=0x7fa47006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:05:57.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.191+0000 7fa477fff700 1 -- 172.21.15.98:0/916678786 shutdown_connections 2023-03-03T08:05:57.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.191+0000 7fa477fff700 1 -- 172.21.15.98:0/916678786 wait complete. 2023-03-03T08:05:57.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.193+0000 7fa477fff700 1 Processor -- start 2023-03-03T08:05:57.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.193+0000 7fa477fff700 1 -- start start 2023-03-03T08:05:57.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.194+0000 7fa477fff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa47006b4b0 0x7fa4700b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:57.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.194+0000 7fa477fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4700d0270 0x7fa47014b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:57.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.194+0000 7fa477fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47015ca70 0x7fa4701f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:57.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.194+0000 7fa477fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47014bb50 con 0x7fa47015ca70 2023-03-03T08:05:57.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.194+0000 7fa477fff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4701f3280 con 0x7fa4700d0270 2023-03-03T08:05:57.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.194+0000 7fa477fff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4700b54d0 con 0x7fa47006b4b0 2023-03-03T08:05:57.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.194+0000 7fa47659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47015ca70 0x7fa4701f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:57.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.194+0000 7fa475d9b700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa47006b4b0 0x7fa4700b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:57.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.194+0000 7fa47659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47015ca70 0x7fa4701f2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60346/0 (socket says 172.21.15.98:60346) 2023-03-03T08:05:57.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.194+0000 7fa47659c700 1 -- 172.21.15.98:0/2005043789 learned_addr learned my addr 172.21.15.98:0/2005043789 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:05:57.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.194+0000 7fa47559a700 1 --2- 172.21.15.98:0/2005043789 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4700d0270 0x7fa47014b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:57.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.195+0000 7fa47659c700 1 -- 172.21.15.98:0/2005043789 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa47006b4b0 msgr2=0x7fa4700b4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:57.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.195+0000 7fa47659c700 1 --2- 172.21.15.98:0/2005043789 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa47006b4b0 0x7fa4700b4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.195+0000 7fa47659c700 1 -- 172.21.15.98:0/2005043789 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4700d0270 msgr2=0x7fa47014b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:57.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.195+0000 7fa47659c700 1 --2- 172.21.15.98:0/2005043789 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4700d0270 0x7fa47014b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.195+0000 7fa47659c700 1 -- 172.21.15.98:0/2005043789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa468009a70 con 0x7fa47015ca70 2023-03-03T08:05:57.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.195+0000 7fa47659c700 1 --2- 172.21.15.98:0/2005043789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47015ca70 0x7fa4701f2d40 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fa4700b5d70 tx=0x7fa468004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:57.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.195+0000 7fa4667fc700 1 -- 172.21.15.98:0/2005043789 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa468012430 con 0x7fa47015ca70 2023-03-03T08:05:57.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.195+0000 7fa4667fc700 1 -- 172.21.15.98:0/2005043789 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa468020070 con 0x7fa47015ca70 2023-03-03T08:05:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.195+0000 7fa4667fc700 1 -- 172.21.15.98:0/2005043789 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa468004280 con 0x7fa47015ca70 2023-03-03T08:05:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.195+0000 7fa477fff700 1 -- 172.21.15.98:0/2005043789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4700d1aa0 con 0x7fa47015ca70 2023-03-03T08:05:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.196+0000 7fa477fff700 1 -- 172.21.15.98:0/2005043789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4700cdbe0 con 0x7fa47015ca70 2023-03-03T08:05:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.197+0000 7fa477fff700 1 -- 172.21.15.98:0/2005043789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47015ce40 con 0x7fa47015ca70 2023-03-03T08:05:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.198+0000 7fa4667fc700 1 -- 172.21.15.98:0/2005043789 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 86575+0+0 (secure 0 0 0) 0x7fa46802e070 con 0x7fa47015ca70 2023-03-03T08:05:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.199+0000 7fa4667fc700 1 --2- 172.21.15.98:0/2005043789 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa454068410 0x7fa45406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.199+0000 7fa4667fc700 1 -- 172.21.15.98:0/2005043789 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7fa468022070 con 0x7fa47015ca70 2023-03-03T08:05:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.200+0000 7fa475d9b700 1 --2- 172.21.15.98:0/2005043789 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa454068410 0x7fa45406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:57.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.200+0000 7fa475d9b700 1 --2- 172.21.15.98:0/2005043789 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa454068410 0x7fa45406a8c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa46c007fd0 tx=0x7fa46c007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:57.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.203+0000 7fa4667fc700 1 -- 172.21.15.98:0/2005043789 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa468054100 con 0x7fa47015ca70 2023-03-03T08:05:57.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.394+0000 7fa477fff700 1 -- 172.21.15.98:0/2005043789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa47004e6e0 con 0x7fa47015ca70 2023-03-03T08:05:57.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.394+0000 7fa4667fc700 1 -- 172.21.15.98:0/2005043789 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa46802c020 con 0x7fa47015ca70 2023-03-03T08:05:57.396 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T08:05:57.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.397+0000 7fa477fff700 1 -- 172.21.15.98:0/2005043789 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa454068410 msgr2=0x7fa45406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:57.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.397+0000 7fa477fff700 1 --2- 172.21.15.98:0/2005043789 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa454068410 0x7fa45406a8c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa46c007fd0 tx=0x7fa46c007de0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.397+0000 7fa477fff700 1 -- 172.21.15.98:0/2005043789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47015ca70 msgr2=0x7fa4701f2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:57.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.397+0000 7fa477fff700 1 --2- 172.21.15.98:0/2005043789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47015ca70 0x7fa4701f2d40 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fa4700b5d70 tx=0x7fa468004ad0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.397+0000 7fa477fff700 1 -- 172.21.15.98:0/2005043789 shutdown_connections 2023-03-03T08:05:57.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.397+0000 7fa477fff700 1 --2- 172.21.15.98:0/2005043789 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa454068410 0x7fa45406a8c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.397+0000 7fa477fff700 1 --2- 172.21.15.98:0/2005043789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47015ca70 0x7fa4701f2d40 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.397+0000 7fa477fff700 1 --2- 172.21.15.98:0/2005043789 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa47006b4b0 0x7fa4700b4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.397+0000 7fa477fff700 1 --2- 172.21.15.98:0/2005043789 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4700d0270 0x7fa47014b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.397+0000 7fa477fff700 1 -- 172.21.15.98:0/2005043789 >> 172.21.15.98:0/2005043789 conn(0x7fa470073e70 msgr2=0x7fa4700cf400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:05:57.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.397+0000 7fa477fff700 1 -- 172.21.15.98:0/2005043789 shutdown_connections 2023-03-03T08:05:57.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.397+0000 7fa477fff700 1 -- 172.21.15.98:0/2005043789 wait complete. 2023-03-03T08:05:57.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T08:05:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.676+0000 7f9908d9d700 1 Processor -- start 2023-03-03T08:05:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.676+0000 7f9908d9d700 1 -- start start 2023-03-03T08:05:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.678+0000 7f9908d9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f99041609b0 0x7f9904160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.678+0000 7f9908d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99040cf570 0x7f990415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.678+0000 7f9908d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f990415db30 0x7f99040b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.678+0000 7f9908d9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99040cebb0 con 0x7f990415db30 2023-03-03T08:05:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.678+0000 7f9908d9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99041612c0 con 0x7f99041609b0 2023-03-03T08:05:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.678+0000 7f9908d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99040cd780 con 0x7f99040cf570 2023-03-03T08:05:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.678+0000 7f9903fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f990415db30 0x7f99040b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.678+0000 7f9903fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f990415db30 0x7f99040b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60362/0 (socket says 172.21.15.98:60362) 2023-03-03T08:05:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.678+0000 7f9903fff700 1 -- 172.21.15.98:0/455689614 learned_addr learned my addr 172.21.15.98:0/455689614 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:05:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.679+0000 7f9902ffd700 1 --2- 172.21.15.98:0/455689614 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99040cf570 0x7f990415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.679+0000 7f99037fe700 1 --2- 172.21.15.98:0/455689614 >> v2:172.21.15.100:3300/0 conn(0x7f99041609b0 0x7f9904160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.679+0000 7f9903fff700 1 -- 172.21.15.98:0/455689614 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99040cf570 msgr2=0x7f990415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:05:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.679+0000 7f9903fff700 1 --2- 172.21.15.98:0/455689614 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99040cf570 0x7f990415d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.679+0000 7f9903fff700 1 -- 172.21.15.98:0/455689614 >> v2:172.21.15.100:3300/0 conn(0x7f99041609b0 msgr2=0x7f9904160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:05:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.679+0000 7f9903fff700 1 --2- 172.21.15.98:0/455689614 >> v2:172.21.15.100:3300/0 conn(0x7f99041609b0 0x7f9904160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.679+0000 7f9903fff700 1 -- 172.21.15.98:0/455689614 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99040cda60 con 0x7f990415db30 2023-03-03T08:05:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.679+0000 7f99037fe700 1 --2- 172.21.15.98:0/455689614 >> v2:172.21.15.100:3300/0 conn(0x7f99041609b0 0x7f9904160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:05:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.679+0000 7f9903fff700 1 --2- 172.21.15.98:0/455689614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f990415db30 0x7f99040b4aa0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f99040742b0 tx=0x7f98fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71e8b0defff14a9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.680+0000 7f99027fc700 1 -- 172.21.15.98:0/455689614 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc00e070 con 0x7f990415db30 2023-03-03T08:05:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.680+0000 7f99027fc700 1 -- 172.21.15.98:0/455689614 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc00b910 con 0x7f990415db30 2023-03-03T08:05:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.680+0000 7f99027fc700 1 -- 172.21.15.98:0/455689614 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc00bcc0 con 0x7f990415db30 2023-03-03T08:05:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.680+0000 7f9908d9d700 1 -- 172.21.15.98:0/455689614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f990415db30 msgr2=0x7f99040b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.680+0000 7f9908d9d700 1 --2- 172.21.15.98:0/455689614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f990415db30 0x7f99040b4aa0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f99040742b0 tx=0x7f98fc00b330 comp rx=0 tx=0).stop 2023-03-03T08:05:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.680+0000 7f9908d9d700 1 -- 172.21.15.98:0/455689614 shutdown_connections 2023-03-03T08:05:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.680+0000 7f9908d9d700 1 --2- 172.21.15.98:0/455689614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f990415db30 0x7f99040b4aa0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.680+0000 7f9908d9d700 1 --2- 172.21.15.98:0/455689614 >> v2:172.21.15.100:3300/0 conn(0x7f99041609b0 0x7f9904160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.680+0000 7f9908d9d700 1 --2- 172.21.15.98:0/455689614 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99040cf570 0x7f990415d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.680+0000 7f9908d9d700 1 -- 172.21.15.98:0/455689614 >> 172.21.15.98:0/455689614 conn(0x7f990406b880 msgr2=0x7f99040cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:05:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.680+0000 7f9908d9d700 1 -- 172.21.15.98:0/455689614 shutdown_connections 2023-03-03T08:05:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.681+0000 7f9908d9d700 1 -- 172.21.15.98:0/455689614 wait complete. 2023-03-03T08:05:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.683+0000 7f9908d9d700 1 Processor -- start 2023-03-03T08:05:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.683+0000 7f9908d9d700 1 -- start start 2023-03-03T08:05:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.684+0000 7f9908d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99040cf570 0x7f99040c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.684+0000 7f9908d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f990415db30 0x7f99040b5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.684+0000 7f9908d9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99041609b0 0x7f99040b6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.684+0000 7f9908d9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99040c6350 con 0x7f99040cf570 2023-03-03T08:05:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.684+0000 7f9908d9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99040c6980 con 0x7f990415db30 2023-03-03T08:05:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.684+0000 7f9908d9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99040c6f40 con 0x7f99041609b0 2023-03-03T08:05:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.684+0000 7f99037fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99040cf570 0x7f99040c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.684+0000 7f99037fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99040cf570 0x7f99040c5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60374/0 (socket says 172.21.15.98:60374) 2023-03-03T08:05:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.684+0000 7f99037fe700 1 -- 172.21.15.98:0/2090129976 learned_addr learned my addr 172.21.15.98:0/2090129976 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:05:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.684+0000 7f9902ffd700 1 --2- 172.21.15.98:0/2090129976 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f990415db30 0x7f99040b5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.684+0000 7f9903fff700 1 --2- 172.21.15.98:0/2090129976 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99041609b0 0x7f99040b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.685+0000 7f99037fe700 1 -- 172.21.15.98:0/2090129976 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99041609b0 msgr2=0x7f99040b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.685+0000 7f99037fe700 1 --2- 172.21.15.98:0/2090129976 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99041609b0 0x7f99040b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.685+0000 7f99037fe700 1 -- 172.21.15.98:0/2090129976 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f990415db30 msgr2=0x7f99040b5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.685+0000 7f99037fe700 1 --2- 172.21.15.98:0/2090129976 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f990415db30 0x7f99040b5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.685+0000 7f99037fe700 1 -- 172.21.15.98:0/2090129976 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc009b30 con 0x7f99040cf570 2023-03-03T08:05:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.685+0000 7f9903fff700 1 --2- 172.21.15.98:0/2090129976 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99041609b0 0x7f99040b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:05:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.685+0000 7f99037fe700 1 --2- 172.21.15.98:0/2090129976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99040cf570 0x7f99040c5e10 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f98f400e840 tx=0x7f98f400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.685+0000 7f98ebfff700 1 -- 172.21.15.98:0/2090129976 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f400c020 con 0x7f99040cf570 2023-03-03T08:05:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.685+0000 7f98ebfff700 1 -- 172.21.15.98:0/2090129976 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f4004250 con 0x7f99040cf570 2023-03-03T08:05:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.685+0000 7f9908d9d700 1 -- 172.21.15.98:0/2090129976 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f990415cdf0 con 0x7f99040cf570 2023-03-03T08:05:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.685+0000 7f98ebfff700 1 -- 172.21.15.98:0/2090129976 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f40036a0 con 0x7f99040cf570 2023-03-03T08:05:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.686+0000 7f9908d9d700 1 -- 172.21.15.98:0/2090129976 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99040cdd40 con 0x7f99040cf570 2023-03-03T08:05:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.686+0000 7f9908d9d700 1 -- 172.21.15.98:0/2090129976 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990415d0d0 con 0x7f99040cf570 2023-03-03T08:05:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.688+0000 7f98ebfff700 1 -- 172.21.15.98:0/2090129976 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 86575+0+0 (secure 0 0 0) 0x7f98f4003800 con 0x7f99040cf570 2023-03-03T08:05:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.690+0000 7f98ebfff700 1 --2- 172.21.15.98:0/2090129976 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98f0068410 0x7f98f006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.690+0000 7f98ebfff700 1 -- 172.21.15.98:0/2090129976 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f98f4085cf0 con 0x7f99040cf570 2023-03-03T08:05:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.690+0000 7f9902ffd700 1 --2- 172.21.15.98:0/2090129976 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98f0068410 0x7f98f006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.691+0000 7f9902ffd700 1 --2- 172.21.15.98:0/2090129976 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98f0068410 0x7f98f006a8c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f99040b2830 tx=0x7f98f8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.694+0000 7f98ebfff700 1 -- 172.21.15.98:0/2090129976 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f98f4052260 con 0x7f99040cf570 2023-03-03T08:05:57.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.901+0000 7f9908d9d700 1 -- 172.21.15.98:0/2090129976 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f99040c7220 con 0x7f99040cf570 2023-03-03T08:05:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.901+0000 7f98ebfff700 1 -- 172.21.15.98:0/2090129976 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v114) v1 ==== 80+0+368 (secure 0 0 0) 0x7f98f400f040 con 0x7f99040cf570 2023-03-03T08:05:57.903 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6832/2374561485 2023-03-04T08:04:52.583414+0000 2023-03-03T08:05:57.903 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6835/2695503624 2023-03-04T08:04:32.578044+0000 2023-03-03T08:05:57.903 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6833/2695503624 2023-03-04T08:04:32.578044+0000 2023-03-03T08:05:57.903 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6834/2374561485 2023-03-04T08:04:52.583414+0000 2023-03-03T08:05:57.903 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.100:6835/2660525287 2023-03-04T08:04:12.571647+0000 2023-03-03T08:05:57.904 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.100:6833/2660525287 2023-03-04T08:04:12.571647+0000 2023-03-03T08:05:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.904+0000 7f9908d9d700 1 -- 172.21.15.98:0/2090129976 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98f0068410 msgr2=0x7f98f006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.904+0000 7f9908d9d700 1 --2- 172.21.15.98:0/2090129976 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98f0068410 0x7f98f006a8c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f99040b2830 tx=0x7f98f8008040 comp rx=0 tx=0).stop 2023-03-03T08:05:57.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.904+0000 7f9908d9d700 1 -- 172.21.15.98:0/2090129976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99040cf570 msgr2=0x7f99040c5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:57.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.904+0000 7f9908d9d700 1 --2- 172.21.15.98:0/2090129976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99040cf570 0x7f99040c5e10 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f98f400e840 tx=0x7f98f400ec00 comp rx=0 tx=0).stop 2023-03-03T08:05:57.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.904+0000 7f9908d9d700 1 -- 172.21.15.98:0/2090129976 shutdown_connections 2023-03-03T08:05:57.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.904+0000 7f9908d9d700 1 --2- 172.21.15.98:0/2090129976 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f98f0068410 0x7f98f006a8c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.904+0000 7f9908d9d700 1 --2- 172.21.15.98:0/2090129976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99040cf570 0x7f99040c5e10 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.904+0000 7f9908d9d700 1 --2- 172.21.15.98:0/2090129976 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99041609b0 0x7f99040b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.904+0000 7f9908d9d700 1 --2- 172.21.15.98:0/2090129976 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f990415db30 0x7f99040b5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:57.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.904+0000 7f9908d9d700 1 -- 172.21.15.98:0/2090129976 >> 172.21.15.98:0/2090129976 conn(0x7f990406b880 msgr2=0x7f99040d0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:05:57.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.904+0000 7f9908d9d700 1 -- 172.21.15.98:0/2090129976 shutdown_connections 2023-03-03T08:05:57.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:05:57.904+0000 7f9908d9d700 1 -- 172.21.15.98:0/2090129976 wait complete. 2023-03-03T08:05:57.908 INFO:teuthology.orchestra.run.smithi098.stderr:listed 6 entries 2023-03-03T08:05:57.933 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2023-03-03T08:05:57.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:05:57.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-03-03T08:05:58.028 INFO:teuthology.orchestra.run.smithi098.stderr:fusermount: failed to unmount /home/ubuntu/cephtest/mnt.0: Device or resource busy 2023-03-03T08:05:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:05:58.034 INFO:tasks.cephfs.fuse_mount:Failed to unmount ceph-fuse on ubuntu@smithi098.front.sepia.ceph.com, aborting... 2023-03-03T08:05:58.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:05:58.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo PATH=/usr/sbin:$PATH lsof ; ps auxf 2023-03-03T08:05:59.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~102s 2023-03-03T08:05:59.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~82s 2023-03-03T08:05:59.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~63s 2023-03-03T08:05:59.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~122s 2023-03-03T08:06:05.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:06:05.048 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:06:05.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~108s 2023-03-03T08:06:05.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~89s 2023-03-03T08:06:05.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~69s 2023-03-03T08:06:05.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~128s 2023-03-03T08:06:11.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~115s 2023-03-03T08:06:11.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~95s 2023-03-03T08:06:11.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~76s 2023-03-03T08:06:11.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~134s 2023-03-03T08:06:18.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~121s 2023-03-03T08:06:18.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~101s 2023-03-03T08:06:18.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~82s 2023-03-03T08:06:18.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~141s 2023-03-03T08:06:24.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~127s 2023-03-03T08:06:24.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~108s 2023-03-03T08:06:24.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~88s 2023-03-03T08:06:24.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~147s 2023-03-03T08:06:30.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~133s 2023-03-03T08:06:30.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~114s 2023-03-03T08:06:30.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~95s 2023-03-03T08:06:30.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~153s 2023-03-03T08:06:35.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:06:35.254 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:06:37.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~140s 2023-03-03T08:06:37.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~120s 2023-03-03T08:06:37.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~101s 2023-03-03T08:06:37.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~160s 2023-03-03T08:06:43.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~146s 2023-03-03T08:06:43.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~127s 2023-03-03T08:06:43.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~107s 2023-03-03T08:06:43.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~166s 2023-03-03T08:06:49.832 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~152s 2023-03-03T08:06:49.832 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~133s 2023-03-03T08:06:49.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~114s 2023-03-03T08:06:49.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~172s 2023-03-03T08:06:56.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~159s 2023-03-03T08:06:56.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~139s 2023-03-03T08:06:56.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~120s 2023-03-03T08:06:56.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~179s 2023-03-03T08:07:02.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~165s 2023-03-03T08:07:02.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~145s 2023-03-03T08:07:02.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~126s 2023-03-03T08:07:02.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~185s 2023-03-03T08:07:05.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:07:05.448 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:07:08.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~171s 2023-03-03T08:07:08.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~152s 2023-03-03T08:07:08.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~133s 2023-03-03T08:07:08.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~191s 2023-03-03T08:07:15.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~178s 2023-03-03T08:07:15.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~158s 2023-03-03T08:07:15.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~139s 2023-03-03T08:07:15.078 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~197s 2023-03-03T08:07:21.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~184s 2023-03-03T08:07:21.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~164s 2023-03-03T08:07:21.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~145s 2023-03-03T08:07:21.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~204s 2023-03-03T08:07:27.697 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~190s 2023-03-03T08:07:27.698 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~171s 2023-03-03T08:07:27.698 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~151s 2023-03-03T08:07:27.698 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~210s 2023-03-03T08:07:34.008 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~197s 2023-03-03T08:07:34.008 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~177s 2023-03-03T08:07:34.008 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~158s 2023-03-03T08:07:34.009 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~216s 2023-03-03T08:07:35.806 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:07:35.813 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:07:40.316 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~203s 2023-03-03T08:07:40.316 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~183s 2023-03-03T08:07:40.317 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~164s 2023-03-03T08:07:40.317 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~223s 2023-03-03T08:07:46.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~209s 2023-03-03T08:07:46.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~190s 2023-03-03T08:07:46.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~170s 2023-03-03T08:07:46.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~229s 2023-03-03T08:07:52.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~215s 2023-03-03T08:07:52.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~196s 2023-03-03T08:07:52.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~177s 2023-03-03T08:07:52.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~235s 2023-03-03T08:07:59.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~222s 2023-03-03T08:07:59.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~202s 2023-03-03T08:07:59.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~183s 2023-03-03T08:07:59.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~242s 2023-03-03T08:08:05.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~228s 2023-03-03T08:08:05.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~209s 2023-03-03T08:08:05.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~189s 2023-03-03T08:08:05.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~248s 2023-03-03T08:08:05.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:08:05.952 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:08:11.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~234s 2023-03-03T08:08:11.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~215s 2023-03-03T08:08:11.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~196s 2023-03-03T08:08:11.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~254s 2023-03-03T08:08:18.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~241s 2023-03-03T08:08:18.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~221s 2023-03-03T08:08:18.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~202s 2023-03-03T08:08:18.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~261s 2023-03-03T08:08:24.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~247s 2023-03-03T08:08:24.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~228s 2023-03-03T08:08:24.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~208s 2023-03-03T08:08:24.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~267s 2023-03-03T08:08:30.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~253s 2023-03-03T08:08:30.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~234s 2023-03-03T08:08:30.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~215s 2023-03-03T08:08:30.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~273s 2023-03-03T08:08:36.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:08:36.109 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:08:37.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~260s 2023-03-03T08:08:37.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~240s 2023-03-03T08:08:37.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~221s 2023-03-03T08:08:37.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~280s 2023-03-03T08:08:43.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~266s 2023-03-03T08:08:43.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~246s 2023-03-03T08:08:43.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~227s 2023-03-03T08:08:43.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~286s 2023-03-03T08:08:49.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~272s 2023-03-03T08:08:49.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~253s 2023-03-03T08:08:49.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~234s 2023-03-03T08:08:49.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~292s 2023-03-03T08:08:56.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~279s 2023-03-03T08:08:56.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~259s 2023-03-03T08:08:56.040 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~240s 2023-03-03T08:08:56.040 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~298s 2023-03-03T08:09:02.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~285s 2023-03-03T08:09:02.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~265s 2023-03-03T08:09:02.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~246s 2023-03-03T08:09:02.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~305s 2023-03-03T08:09:02.351 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-03-03T08:09:02.351 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:09:02.352 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:09:02.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:09:02.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.674+0000 7fc5853ff700 1 Processor -- start 2023-03-03T08:09:02.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.675+0000 7fc5853ff700 1 -- start start 2023-03-03T08:09:02.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.676+0000 7fc5853ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc580159010 0x7fc5800c9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:02.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.676+0000 7fc5853ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58015d9e0 con 0x7fc5800cc110 2023-03-03T08:09:02.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.676+0000 7fc5853ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5801597e0 con 0x7fc580070ff0 2023-03-03T08:09:02.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.676+0000 7fc5853ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc580159a80 con 0x7fc580159010 2023-03-03T08:09:02.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.677+0000 7fc57effd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc580159010 0x7fc5800c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:02.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.677+0000 7fc577fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc5800cc110 0x7fc58015cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52944/0 (socket says 172.21.15.98:52944) 2023-03-03T08:09:02.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.677+0000 7fc577fff700 1 -- 172.21.15.98:0/2444679616 learned_addr learned my addr 172.21.15.98:0/2444679616 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:09:02.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.677+0000 7fc57effd700 1 -- 172.21.15.98:0/2444679616 >> v1:172.21.15.100:6789/0 conn(0x7fc580070ff0 legacy=0x7fc58015d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:02.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.677+0000 7fc57e7fc700 1 -- 172.21.15.98:0/2444679616 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234095882 0 0) 0x7fc58015d9e0 con 0x7fc5800cc110 2023-03-03T08:09:02.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.677+0000 7fc57effd700 1 -- 172.21.15.98:0/2444679616 >> v1:172.21.15.98:6789/0 conn(0x7fc5800cc110 legacy=0x7fc58015cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:02.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.677+0000 7fc57effd700 1 -- 172.21.15.98:0/2444679616 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc580159d60 con 0x7fc580159010 2023-03-03T08:09:02.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.678+0000 7fc57effd700 1 --2- 172.21.15.98:0/2444679616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580159010 0x7fc5800c9450 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fc570009840 tx=0x7fc570009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7190b90be2b53eb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:02.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.678+0000 7fc57e7fc700 1 -- 172.21.15.98:0/2444679616 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570011830 con 0x7fc580159010 2023-03-03T08:09:02.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.678+0000 7fc57e7fc700 1 -- 172.21.15.98:0/2444679616 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc570011990 con 0x7fc580159010 2023-03-03T08:09:02.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.678+0000 7fc57e7fc700 1 -- 172.21.15.98:0/2444679616 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570011d20 con 0x7fc580159010 2023-03-03T08:09:02.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.678+0000 7fc5853ff700 1 -- 172.21.15.98:0/2444679616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580159010 msgr2=0x7fc5800c9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:02.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.678+0000 7fc5853ff700 1 --2- 172.21.15.98:0/2444679616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580159010 0x7fc5800c9450 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fc570009840 tx=0x7fc570009b50 comp rx=0 tx=0).stop 2023-03-03T08:09:02.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.679+0000 7fc5853ff700 1 -- 172.21.15.98:0/2444679616 shutdown_connections 2023-03-03T08:09:02.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.679+0000 7fc5853ff700 1 --2- 172.21.15.98:0/2444679616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580159010 0x7fc5800c9450 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:02.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.679+0000 7fc5853ff700 1 -- 172.21.15.98:0/2444679616 >> 172.21.15.98:0/2444679616 conn(0x7fc5800b09f0 msgr2=0x7fc5800b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:02.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.679+0000 7fc5853ff700 1 -- 172.21.15.98:0/2444679616 shutdown_connections 2023-03-03T08:09:02.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.679+0000 7fc5853ff700 1 -- 172.21.15.98:0/2444679616 wait complete. 2023-03-03T08:09:02.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.681+0000 7fc5853ff700 1 Processor -- start 2023-03-03T08:09:02.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.681+0000 7fc5853ff700 1 -- start start 2023-03-03T08:09:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.682+0000 7fc5853ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc580070ff0 0x7fc580146b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.682+0000 7fc5853ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5800cc110 0x7fc5800c1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.682+0000 7fc5853ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580159010 0x7fc580141950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.682+0000 7fc5853ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5800cc960 con 0x7fc580159010 2023-03-03T08:09:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.682+0000 7fc5853ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5800ce720 con 0x7fc5800cc110 2023-03-03T08:09:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.682+0000 7fc5853ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5800cea00 con 0x7fc580070ff0 2023-03-03T08:09:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.682+0000 7fc57f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580159010 0x7fc580141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.682+0000 7fc57f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580159010 0x7fc580141950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54836/0 (socket says 172.21.15.98:54836) 2023-03-03T08:09:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.682+0000 7fc57f7fe700 1 -- 172.21.15.98:0/3006690710 learned_addr learned my addr 172.21.15.98:0/3006690710 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.682+0000 7fc57effd700 1 --2- 172.21.15.98:0/3006690710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc580070ff0 0x7fc580146b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.683+0000 7fc577fff700 1 --2- 172.21.15.98:0/3006690710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5800cc110 0x7fc5800c1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.683+0000 7fc57f7fe700 1 -- 172.21.15.98:0/3006690710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc580070ff0 msgr2=0x7fc580146b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.683+0000 7fc57f7fe700 1 --2- 172.21.15.98:0/3006690710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc580070ff0 0x7fc580146b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.683+0000 7fc57f7fe700 1 -- 172.21.15.98:0/3006690710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5800cc110 msgr2=0x7fc5800c1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.683+0000 7fc57f7fe700 1 --2- 172.21.15.98:0/3006690710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5800cc110 0x7fc5800c1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.683+0000 7fc57f7fe700 1 -- 172.21.15.98:0/3006690710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5700094a0 con 0x7fc580159010 2023-03-03T08:09:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.683+0000 7fc57f7fe700 1 --2- 172.21.15.98:0/3006690710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580159010 0x7fc580141950 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fc57800cc50 tx=0x7fc57800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.683+0000 7fc5777fe700 1 -- 172.21.15.98:0/3006690710 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578003c00 con 0x7fc580159010 2023-03-03T08:09:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.684+0000 7fc5777fe700 1 -- 172.21.15.98:0/3006690710 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc578003d60 con 0x7fc580159010 2023-03-03T08:09:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.684+0000 7fc5777fe700 1 -- 172.21.15.98:0/3006690710 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57800aab0 con 0x7fc580159010 2023-03-03T08:09:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.684+0000 7fc5853ff700 1 -- 172.21.15.98:0/3006690710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58015a320 con 0x7fc580159010 2023-03-03T08:09:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.684+0000 7fc5853ff700 1 -- 172.21.15.98:0/3006690710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58015a040 con 0x7fc580159010 2023-03-03T08:09:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.685+0000 7fc5853ff700 1 -- 172.21.15.98:0/3006690710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5800cc740 con 0x7fc580159010 2023-03-03T08:09:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.686+0000 7fc5777fe700 1 -- 172.21.15.98:0/3006690710 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fc578010050 con 0x7fc580159010 2023-03-03T08:09:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.687+0000 7fc5777fe700 1 --2- 172.21.15.98:0/3006690710 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc5600683c0 0x7fc56006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.687+0000 7fc5777fe700 1 -- 172.21.15.98:0/3006690710 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7fc578085df0 con 0x7fc580159010 2023-03-03T08:09:02.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.688+0000 7fc57effd700 1 --2- 172.21.15.98:0/3006690710 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc5600683c0 0x7fc56006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:02.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.689+0000 7fc57effd700 1 --2- 172.21.15.98:0/3006690710 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc5600683c0 0x7fc56006a870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc5700090b0 tx=0x7fc57001c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.692+0000 7fc5777fe700 1 -- 172.21.15.98:0/3006690710 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc5780523b0 con 0x7fc580159010 2023-03-03T08:09:02.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.886+0000 7fc5853ff700 1 -- 172.21.15.98:0/3006690710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc58004e6e0 con 0x7fc580159010 2023-03-03T08:09:02.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.887+0000 7fc5777fe700 1 -- 172.21.15.98:0/3006690710 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+18380 (secure 0 0 0) 0x7fc5780559d0 con 0x7fc580159010 2023-03-03T08:09:02.889 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:09:02.889 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":114,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:04:52.583483+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:03:11.891955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-03-03T08:03:12.507916+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6832/2374561485":"2023-03-04T08:04:52.583414+0000","172.21.15.98:6835/2695503624":"2023-03-04T08:04:32.578044+0000","172.21.15.98:6833/2695503624":"2023-03-04T08:04:32.578044+0000","172.21.15.98:6834/2374561485":"2023-03-04T08:04:52.583414+0000","172.21.15.100:6835/2660525287":"2023-03-04T08:04:12.571647+0000","172.21.15.100:6833/2660525287":"2023-03-04T08:04:12.571647+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:09:02.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.890+0000 7fc5853ff700 1 -- 172.21.15.98:0/3006690710 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc5600683c0 msgr2=0x7fc56006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:02.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.890+0000 7fc5853ff700 1 --2- 172.21.15.98:0/3006690710 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc5600683c0 0x7fc56006a870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc5700090b0 tx=0x7fc57001c040 comp rx=0 tx=0).stop 2023-03-03T08:09:02.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.890+0000 7fc5853ff700 1 -- 172.21.15.98:0/3006690710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580159010 msgr2=0x7fc580141950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:02.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.890+0000 7fc5853ff700 1 --2- 172.21.15.98:0/3006690710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580159010 0x7fc580141950 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fc57800cc50 tx=0x7fc57800cf60 comp rx=0 tx=0).stop 2023-03-03T08:09:02.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.890+0000 7fc5853ff700 1 -- 172.21.15.98:0/3006690710 shutdown_connections 2023-03-03T08:09:02.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.890+0000 7fc5853ff700 1 --2- 172.21.15.98:0/3006690710 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fc5600683c0 0x7fc56006a870 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:02.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.890+0000 7fc5853ff700 1 --2- 172.21.15.98:0/3006690710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580159010 0x7fc580141950 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:02.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.890+0000 7fc5853ff700 1 --2- 172.21.15.98:0/3006690710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc580070ff0 0x7fc580146b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:02.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.890+0000 7fc5853ff700 1 --2- 172.21.15.98:0/3006690710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5800cc110 0x7fc5800c1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:02.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.890+0000 7fc5853ff700 1 -- 172.21.15.98:0/3006690710 >> 172.21.15.98:0/3006690710 conn(0x7fc5800b09f0 msgr2=0x7fc580150a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:02.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.890+0000 7fc5853ff700 1 -- 172.21.15.98:0/3006690710 shutdown_connections 2023-03-03T08:09:02.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:02.890+0000 7fc5853ff700 1 -- 172.21.15.98:0/3006690710 wait complete. 2023-03-03T08:09:02.920 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T08:03:11.891955+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T08:03:12.507916+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:09:02.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:09:03.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.207+0000 7fab4b9ff700 1 Processor -- start 2023-03-03T08:09:03.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.207+0000 7fab4b9ff700 1 -- start start 2023-03-03T08:09:03.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.209+0000 7fab4b9ff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab440d0270 0x7fab440cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:03.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.209+0000 7fab4b9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab4406b4b0 0x7fab440d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:03.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.209+0000 7fab4b9ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab440cf980 con 0x7fab4415ca70 2023-03-03T08:09:03.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.209+0000 7fab4b9ff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab440cd5b0 con 0x7fab440d0270 2023-03-03T08:09:03.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.209+0000 7fab4b9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab440d1800 con 0x7fab4406b4b0 2023-03-03T08:09:03.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.209+0000 7fab48f9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab440d0270 0x7fab440cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:03.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.209+0000 7fab48f9a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab440d0270 0x7fab440cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54852/0 (socket says 172.21.15.98:54852) 2023-03-03T08:09:03.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.209+0000 7fab48f9a700 1 -- 172.21.15.98:0/181963053 learned_addr learned my addr 172.21.15.98:0/181963053 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.209+0000 7fab49f9c700 1 --2- 172.21.15.98:0/181963053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab4406b4b0 0x7fab440d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.210+0000 7fab3bfff700 1 -- 172.21.15.98:0/181963053 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216010229 0 0) 0x7fab440cf980 con 0x7fab4415ca70 2023-03-03T08:09:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.210+0000 7fab3bfff700 1 -- 172.21.15.98:0/181963053 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab300035b0 con 0x7fab4415ca70 2023-03-03T08:09:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.210+0000 7fab48f9a700 1 -- 172.21.15.98:0/181963053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab4406b4b0 msgr2=0x7fab440d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:09:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.210+0000 7fab48f9a700 1 --2- 172.21.15.98:0/181963053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab4406b4b0 0x7fab440d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.210+0000 7fab48f9a700 1 -- 172.21.15.98:0/181963053 >> v1:172.21.15.100:6789/0 conn(0x7fab4415ca70 legacy=0x7fab440cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.210+0000 7fab48f9a700 1 -- 172.21.15.98:0/181963053 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab440cd900 con 0x7fab440d0270 2023-03-03T08:09:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.210+0000 7fab49f9c700 1 --2- 172.21.15.98:0/181963053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab4406b4b0 0x7fab440d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:09:03.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.210+0000 7fab48f9a700 1 --2- 172.21.15.98:0/181963053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab440d0270 0x7fab440cf440 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fab340098f0 tx=0x7fab34009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fccc784234a8a02b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:03.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.210+0000 7fab3bfff700 1 -- 172.21.15.98:0/181963053 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34011830 con 0x7fab440d0270 2023-03-03T08:09:03.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.210+0000 7fab3bfff700 1 -- 172.21.15.98:0/181963053 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab34011990 con 0x7fab440d0270 2023-03-03T08:09:03.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.210+0000 7fab3bfff700 1 -- 172.21.15.98:0/181963053 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34011d20 con 0x7fab440d0270 2023-03-03T08:09:03.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.211+0000 7fab4b9ff700 1 -- 172.21.15.98:0/181963053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab440d0270 msgr2=0x7fab440cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:03.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.211+0000 7fab4b9ff700 1 --2- 172.21.15.98:0/181963053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab440d0270 0x7fab440cf440 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fab340098f0 tx=0x7fab34009c00 comp rx=0 tx=0).stop 2023-03-03T08:09:03.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.211+0000 7fab4b9ff700 1 -- 172.21.15.98:0/181963053 shutdown_connections 2023-03-03T08:09:03.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.211+0000 7fab4b9ff700 1 --2- 172.21.15.98:0/181963053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab4406b4b0 0x7fab440d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.211+0000 7fab4b9ff700 1 --2- 172.21.15.98:0/181963053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab440d0270 0x7fab440cf440 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.211+0000 7fab4b9ff700 1 -- 172.21.15.98:0/181963053 >> 172.21.15.98:0/181963053 conn(0x7fab44073e70 msgr2=0x7fab4406ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:03.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.211+0000 7fab4b9ff700 1 -- 172.21.15.98:0/181963053 shutdown_connections 2023-03-03T08:09:03.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.211+0000 7fab4b9ff700 1 -- 172.21.15.98:0/181963053 wait complete. 2023-03-03T08:09:03.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.213+0000 7fab4b9ff700 1 Processor -- start 2023-03-03T08:09:03.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.213+0000 7fab4b9ff700 1 -- start start 2023-03-03T08:09:03.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.214+0000 7fab4b9ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab4406b4b0 0x7fab4414f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:03.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.214+0000 7fab4b9ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fab440d0270 0x7fab440c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:03.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.214+0000 7fab4b9ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4415ca70 0x7fab440b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:03.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.214+0000 7fab4b9ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab440c6c90 con 0x7fab4415ca70 2023-03-03T08:09:03.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.214+0000 7fab4b9ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab440cc900 con 0x7fab440d0270 2023-03-03T08:09:03.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.214+0000 7fab4b9ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab440ccbe0 con 0x7fab4406b4b0 2023-03-03T08:09:03.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.215+0000 7fab49f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4415ca70 0x7fab440b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:03.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.215+0000 7fab49f9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4415ca70 0x7fab440b78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54858/0 (socket says 172.21.15.98:54858) 2023-03-03T08:09:03.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.215+0000 7fab49f9c700 1 -- 172.21.15.98:0/1623551275 learned_addr learned my addr 172.21.15.98:0/1623551275 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:03.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.215+0000 7fab4979b700 1 --2- 172.21.15.98:0/1623551275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab4406b4b0 0x7fab4414f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:03.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.215+0000 7fab48f9a700 1 --2- 172.21.15.98:0/1623551275 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fab440d0270 0x7fab440c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.215+0000 7fab49f9c700 1 -- 172.21.15.98:0/1623551275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab4406b4b0 msgr2=0x7fab4414f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.215+0000 7fab49f9c700 1 --2- 172.21.15.98:0/1623551275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab4406b4b0 0x7fab4414f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.215+0000 7fab49f9c700 1 -- 172.21.15.98:0/1623551275 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fab440d0270 msgr2=0x7fab440c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.215+0000 7fab49f9c700 1 --2- 172.21.15.98:0/1623551275 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fab440d0270 0x7fab440c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.215+0000 7fab49f9c700 1 -- 172.21.15.98:0/1623551275 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c009a70 con 0x7fab4415ca70 2023-03-03T08:09:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.215+0000 7fab48f9a700 1 --2- 172.21.15.98:0/1623551275 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fab440d0270 0x7fab440c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:09:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.215+0000 7fab49f9c700 1 --2- 172.21.15.98:0/1623551275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4415ca70 0x7fab440b78d0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00efc0 tx=0x7fab3c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:03.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.216+0000 7fab39ffb700 1 -- 172.21.15.98:0/1623551275 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c00cb40 con 0x7fab4415ca70 2023-03-03T08:09:03.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.216+0000 7fab39ffb700 1 -- 172.21.15.98:0/1623551275 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3c00f040 con 0x7fab4415ca70 2023-03-03T08:09:03.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.216+0000 7fab4b9ff700 1 -- 172.21.15.98:0/1623551275 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab34009550 con 0x7fab4415ca70 2023-03-03T08:09:03.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.216+0000 7fab39ffb700 1 -- 172.21.15.98:0/1623551275 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c003a20 con 0x7fab4415ca70 2023-03-03T08:09:03.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.216+0000 7fab4b9ff700 1 -- 172.21.15.98:0/1623551275 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab440d1aa0 con 0x7fab4415ca70 2023-03-03T08:09:03.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.217+0000 7fab4b9ff700 1 -- 172.21.15.98:0/1623551275 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab440cdbe0 con 0x7fab4415ca70 2023-03-03T08:09:03.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.218+0000 7fab39ffb700 1 -- 172.21.15.98:0/1623551275 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fab3c014480 con 0x7fab4415ca70 2023-03-03T08:09:03.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.219+0000 7fab39ffb700 1 --2- 172.21.15.98:0/1623551275 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fab280683c0 0x7fab2806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:03.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.220+0000 7fab39ffb700 1 -- 172.21.15.98:0/1623551275 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7fab3c086a60 con 0x7fab4415ca70 2023-03-03T08:09:03.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.220+0000 7fab4979b700 1 --2- 172.21.15.98:0/1623551275 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fab280683c0 0x7fab2806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:03.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.221+0000 7fab4979b700 1 --2- 172.21.15.98:0/1623551275 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fab280683c0 0x7fab2806a870 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fab400011e0 tx=0x7fab40009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:03.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.223+0000 7fab39ffb700 1 -- 172.21.15.98:0/1623551275 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fab3c052fc0 con 0x7fab4415ca70 2023-03-03T08:09:03.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.415+0000 7fab4b9ff700 1 -- 172.21.15.98:0/1623551275 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab4415ce40 con 0x7fab4415ca70 2023-03-03T08:09:03.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.415+0000 7fab39ffb700 1 -- 172.21.15.98:0/1623551275 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab3c009cd0 con 0x7fab4415ca70 2023-03-03T08:09:03.417 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:09:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.418+0000 7fab4b9ff700 1 -- 172.21.15.98:0/1623551275 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fab280683c0 msgr2=0x7fab2806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.418+0000 7fab4b9ff700 1 --2- 172.21.15.98:0/1623551275 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fab280683c0 0x7fab2806a870 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fab400011e0 tx=0x7fab40009040 comp rx=0 tx=0).stop 2023-03-03T08:09:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.418+0000 7fab4b9ff700 1 -- 172.21.15.98:0/1623551275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4415ca70 msgr2=0x7fab440b78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.418+0000 7fab4b9ff700 1 --2- 172.21.15.98:0/1623551275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4415ca70 0x7fab440b78d0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00efc0 tx=0x7fab3c00c3a0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.418+0000 7fab4b9ff700 1 -- 172.21.15.98:0/1623551275 shutdown_connections 2023-03-03T08:09:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.418+0000 7fab4b9ff700 1 --2- 172.21.15.98:0/1623551275 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fab280683c0 0x7fab2806a870 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.418+0000 7fab4b9ff700 1 --2- 172.21.15.98:0/1623551275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4415ca70 0x7fab440b78d0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.418+0000 7fab4b9ff700 1 --2- 172.21.15.98:0/1623551275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab4406b4b0 0x7fab4414f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.418+0000 7fab4b9ff700 1 --2- 172.21.15.98:0/1623551275 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fab440d0270 0x7fab440c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.418+0000 7fab4b9ff700 1 -- 172.21.15.98:0/1623551275 >> 172.21.15.98:0/1623551275 conn(0x7fab44073e70 msgr2=0x7fab440b8850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.418+0000 7fab4b9ff700 1 -- 172.21.15.98:0/1623551275 shutdown_connections 2023-03-03T08:09:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.419+0000 7fab4b9ff700 1 -- 172.21.15.98:0/1623551275 wait complete. 2023-03-03T08:09:03.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:09:03.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.694+0000 7f9fa2b9d700 1 Processor -- start 2023-03-03T08:09:03.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.694+0000 7f9fa2b9d700 1 -- start start 2023-03-03T08:09:03.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.696+0000 7f9fa2b9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9f9c0b5e70 0x7f9f9c0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.696+0000 7f9fa2b9d700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9f9c0cf570 0x7f9f9c0b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.696+0000 7f9fa2b9d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c15ce00 con 0x7f9f9c0b8910 2023-03-03T08:09:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.696+0000 7f9fa2b9d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f9c0d0bf0 con 0x7f9f9c0b5e70 2023-03-03T08:09:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.696+0000 7f9fa2b9d700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f9c0b8ce0 con 0x7f9f9c0cf570 2023-03-03T08:09:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.696+0000 7f9fa139a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9f9c0b5e70 0x7f9f9c0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.696+0000 7f9fa139a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9f9c0b5e70 0x7f9f9c0b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54874/0 (socket says 172.21.15.98:54874) 2023-03-03T08:09:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.696+0000 7f9fa139a700 1 -- 172.21.15.98:0/3569885707 learned_addr learned my addr 172.21.15.98:0/3569885707 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.696+0000 7f9fa239c700 1 --2- 172.21.15.98:0/3569885707 >> v2:172.21.15.100:3300/0 conn(0x7f9f9c0cf570 0x7f9f9c0b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.696+0000 7f9fa139a700 1 -- 172.21.15.98:0/3569885707 >> v2:172.21.15.100:3300/0 conn(0x7f9f9c0cf570 msgr2=0x7f9f9c0b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:09:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.697+0000 7f9fa139a700 1 --2- 172.21.15.98:0/3569885707 >> v2:172.21.15.100:3300/0 conn(0x7f9f9c0cf570 0x7f9f9c0b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.697+0000 7f9fa139a700 1 -- 172.21.15.98:0/3569885707 >> v1:172.21.15.100:6789/0 conn(0x7f9f9c0b8910 legacy=0x7f9f9c1604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.697+0000 7f9fa139a700 1 -- 172.21.15.98:0/3569885707 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c160cf0 con 0x7f9f9c0b5e70 2023-03-03T08:09:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.697+0000 7f9fa239c700 1 --2- 172.21.15.98:0/3569885707 >> v2:172.21.15.100:3300/0 conn(0x7f9f9c0cf570 0x7f9f9c0b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:09:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.697+0000 7f9fa139a700 1 --2- 172.21.15.98:0/3569885707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f9c0b5e70 0x7f9f9c0b4aa0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0742d0 tx=0x7f9f8c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c6a83f5d858a48 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.697+0000 7f9fa0b99700 1 -- 172.21.15.98:0/3569885707 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c00b870 con 0x7f9f9c0b5e70 2023-03-03T08:09:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.697+0000 7f9fa0b99700 1 -- 172.21.15.98:0/3569885707 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8c00b9d0 con 0x7f9f9c0b5e70 2023-03-03T08:09:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.697+0000 7f9fa0b99700 1 -- 172.21.15.98:0/3569885707 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c00bd60 con 0x7f9f9c0b5e70 2023-03-03T08:09:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.698+0000 7f9fa2b9d700 1 -- 172.21.15.98:0/3569885707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f9c0b5e70 msgr2=0x7f9f9c0b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.698+0000 7f9fa2b9d700 1 --2- 172.21.15.98:0/3569885707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f9c0b5e70 0x7f9f9c0b4aa0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0742d0 tx=0x7f9f8c009bd0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.698+0000 7f9fa2b9d700 1 -- 172.21.15.98:0/3569885707 shutdown_connections 2023-03-03T08:09:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.698+0000 7f9fa2b9d700 1 --2- 172.21.15.98:0/3569885707 >> v2:172.21.15.100:3300/0 conn(0x7f9f9c0cf570 0x7f9f9c0b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.698+0000 7f9fa2b9d700 1 --2- 172.21.15.98:0/3569885707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f9c0b5e70 0x7f9f9c0b4aa0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.698+0000 7f9fa2b9d700 1 -- 172.21.15.98:0/3569885707 >> 172.21.15.98:0/3569885707 conn(0x7f9f9c06b880 msgr2=0x7f9f9c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.698+0000 7f9fa2b9d700 1 -- 172.21.15.98:0/3569885707 shutdown_connections 2023-03-03T08:09:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.698+0000 7f9fa2b9d700 1 -- 172.21.15.98:0/3569885707 wait complete. 2023-03-03T08:09:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.701+0000 7f9fa2b9d700 1 Processor -- start 2023-03-03T08:09:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.701+0000 7f9fa2b9d700 1 -- start start 2023-03-03T08:09:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.702+0000 7f9fa2b9d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f9c0b5e70 0x7f9f9c0b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.702+0000 7f9fa2b9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f9c0b8910 0x7f9f9c0b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.702+0000 7f9fa2b9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f9c0cf570 0x7f9f9c1505e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.702+0000 7f9fa2b9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c151690 con 0x7f9f9c0b8910 2023-03-03T08:09:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.702+0000 7f9fa2b9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c1e62c0 con 0x7f9f9c0cf570 2023-03-03T08:09:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.702+0000 7f9fa2b9d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c0b34c0 con 0x7f9f9c0b5e70 2023-03-03T08:09:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.703+0000 7f9fa139a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f9c0b8910 0x7f9f9c0b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.703+0000 7f9fa139a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f9c0b8910 0x7f9f9c0b4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54884/0 (socket says 172.21.15.98:54884) 2023-03-03T08:09:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.703+0000 7f9fa139a700 1 -- 172.21.15.98:0/571677140 learned_addr learned my addr 172.21.15.98:0/571677140 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.703+0000 7f9fa1b9b700 1 --2- 172.21.15.98:0/571677140 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f9c0b5e70 0x7f9f9c0b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.703+0000 7f9fa239c700 1 --2- 172.21.15.98:0/571677140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f9c0cf570 0x7f9f9c1505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.703+0000 7f9fa139a700 1 -- 172.21.15.98:0/571677140 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f9c0b5e70 msgr2=0x7f9f9c0b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.703+0000 7f9fa139a700 1 --2- 172.21.15.98:0/571677140 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f9c0b5e70 0x7f9f9c0b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.703+0000 7f9fa139a700 1 -- 172.21.15.98:0/571677140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f9c0cf570 msgr2=0x7f9f9c1505e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.703+0000 7f9fa139a700 1 --2- 172.21.15.98:0/571677140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f9c0cf570 0x7f9f9c1505e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.703+0000 7f9fa139a700 1 -- 172.21.15.98:0/571677140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c009550 con 0x7f9f9c0b8910 2023-03-03T08:09:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.703+0000 7f9fa239c700 1 --2- 172.21.15.98:0/571677140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f9c0cf570 0x7f9f9c1505e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:09:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.703+0000 7f9fa139a700 1 --2- 172.21.15.98:0/571677140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f9c0b8910 0x7f9f9c0b4ec0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c000c00 tx=0x7f9f8c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.704+0000 7f9f927fc700 1 -- 172.21.15.98:0/571677140 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c004520 con 0x7f9f9c0b8910 2023-03-03T08:09:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.704+0000 7f9f927fc700 1 -- 172.21.15.98:0/571677140 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8c01e070 con 0x7f9f9c0b8910 2023-03-03T08:09:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.704+0000 7f9f927fc700 1 -- 172.21.15.98:0/571677140 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c0038a0 con 0x7f9f9c0b8910 2023-03-03T08:09:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.704+0000 7f9fa2b9d700 1 -- 172.21.15.98:0/571677140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9c1612b0 con 0x7f9f9c0b8910 2023-03-03T08:09:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.704+0000 7f9fa2b9d700 1 -- 172.21.15.98:0/571677140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9c160fd0 con 0x7f9f9c0b8910 2023-03-03T08:09:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.705+0000 7f9fa2b9d700 1 -- 172.21.15.98:0/571677140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9c0d0eb0 con 0x7f9f9c0b8910 2023-03-03T08:09:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.706+0000 7f9f927fc700 1 -- 172.21.15.98:0/571677140 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f9f8c00f050 con 0x7f9f9c0b8910 2023-03-03T08:09:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.707+0000 7f9f927fc700 1 --2- 172.21.15.98:0/571677140 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9f800683c0 0x7f9f8006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.707+0000 7f9f927fc700 1 -- 172.21.15.98:0/571677140 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f9f8c020070 con 0x7f9f9c0b8910 2023-03-03T08:09:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.708+0000 7f9fa1b9b700 1 --2- 172.21.15.98:0/571677140 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9f800683c0 0x7f9f8006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.709+0000 7f9fa1b9b700 1 --2- 172.21.15.98:0/571677140 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9f800683c0 0x7f9f8006a870 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9f98000f60 tx=0x7f9f98009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.711+0000 7f9f927fc700 1 -- 172.21.15.98:0/571677140 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9f8c053b10 con 0x7f9f9c0b8910 2023-03-03T08:09:03.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.906+0000 7f9fa2b9d700 1 -- 172.21.15.98:0/571677140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9f9c068290 con 0x7f9f9c0b8910 2023-03-03T08:09:03.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.906+0000 7f9f927fc700 1 -- 172.21.15.98:0/571677140 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v114) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9f8c057130 con 0x7f9f9c0b8910 2023-03-03T08:09:03.908 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:09:03.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.909+0000 7f9fa2b9d700 1 -- 172.21.15.98:0/571677140 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9f800683c0 msgr2=0x7f9f8006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:03.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.909+0000 7f9fa2b9d700 1 --2- 172.21.15.98:0/571677140 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9f800683c0 0x7f9f8006a870 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9f98000f60 tx=0x7f9f98009040 comp rx=0 tx=0).stop 2023-03-03T08:09:03.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.909+0000 7f9fa2b9d700 1 -- 172.21.15.98:0/571677140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f9c0b8910 msgr2=0x7f9f9c0b4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:03.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.909+0000 7f9fa2b9d700 1 --2- 172.21.15.98:0/571677140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f9c0b8910 0x7f9f9c0b4ec0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c000c00 tx=0x7f9f8c004750 comp rx=0 tx=0).stop 2023-03-03T08:09:03.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.909+0000 7f9fa2b9d700 1 -- 172.21.15.98:0/571677140 shutdown_connections 2023-03-03T08:09:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.909+0000 7f9fa2b9d700 1 --2- 172.21.15.98:0/571677140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f9c0cf570 0x7f9f9c1505e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.909+0000 7f9fa2b9d700 1 --2- 172.21.15.98:0/571677140 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f9c0b5e70 0x7f9f9c0b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.909+0000 7f9fa2b9d700 1 --2- 172.21.15.98:0/571677140 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9f800683c0 0x7f9f8006a870 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.909+0000 7f9fa2b9d700 1 --2- 172.21.15.98:0/571677140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f9c0b8910 0x7f9f9c0b4ec0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.909+0000 7f9fa2b9d700 1 -- 172.21.15.98:0/571677140 >> 172.21.15.98:0/571677140 conn(0x7f9f9c06b880 msgr2=0x7f9f9c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.909+0000 7f9fa2b9d700 1 -- 172.21.15.98:0/571677140 shutdown_connections 2023-03-03T08:09:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:03.909+0000 7f9fa2b9d700 1 -- 172.21.15.98:0/571677140 wait complete. 2023-03-03T08:09:03.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T08:09:04.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.182+0000 7f8cff7ff700 1 Processor -- start 2023-03-03T08:09:04.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.182+0000 7f8cff7ff700 1 -- start start 2023-03-03T08:09:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.184+0000 7f8cff7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cf80b26d0 0x7f8cf80b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.184+0000 7f8cff7ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf80cd7d0 con 0x7f8cf80b5140 2023-03-03T08:09:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.184+0000 7f8cff7ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf80b90e0 con 0x7f8cf80b8d10 2023-03-03T08:09:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.184+0000 7f8cff7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf80b55c0 con 0x7f8cf80b26d0 2023-03-03T08:09:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.184+0000 7f8cfcd9a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cf80b26d0 0x7f8cf80b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.184+0000 7f8cfdd9c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8cf80b5140 0x7f8cf80d13a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52946/0 (socket says 172.21.15.98:52946) 2023-03-03T08:09:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.184+0000 7f8cfdd9c700 1 -- 172.21.15.98:0/2151740386 learned_addr learned my addr 172.21.15.98:0/2151740386 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:09:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.185+0000 7f8ceffff700 1 -- 172.21.15.98:0/2151740386 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448792052 0 0) 0x7f8cf80cd7d0 con 0x7f8cf80b5140 2023-03-03T08:09:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.185+0000 7f8cfcd9a700 1 -- 172.21.15.98:0/2151740386 >> v1:172.21.15.100:6789/0 conn(0x7f8cf80b8d10 legacy=0x7f8cf80cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.185+0000 7f8cfcd9a700 1 -- 172.21.15.98:0/2151740386 >> v1:172.21.15.98:6789/0 conn(0x7f8cf80b5140 legacy=0x7f8cf80d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.185+0000 7f8cfcd9a700 1 -- 172.21.15.98:0/2151740386 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf80cda20 con 0x7f8cf80b26d0 2023-03-03T08:09:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.185+0000 7f8cfcd9a700 1 --2- 172.21.15.98:0/2151740386 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cf80b26d0 0x7f8cf80b4c00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8ce80097f0 tx=0x7f8ce8009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0cb6d8253da83d3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.186+0000 7f8ceffff700 1 -- 172.21.15.98:0/2151740386 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8004230 con 0x7f8cf80b26d0 2023-03-03T08:09:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.186+0000 7f8ceffff700 1 -- 172.21.15.98:0/2151740386 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce8004390 con 0x7f8cf80b26d0 2023-03-03T08:09:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.186+0000 7f8ceffff700 1 -- 172.21.15.98:0/2151740386 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce80036a0 con 0x7f8cf80b26d0 2023-03-03T08:09:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.186+0000 7f8cff7ff700 1 -- 172.21.15.98:0/2151740386 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cf80b26d0 msgr2=0x7f8cf80b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.186+0000 7f8cff7ff700 1 --2- 172.21.15.98:0/2151740386 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cf80b26d0 0x7f8cf80b4c00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8ce80097f0 tx=0x7f8ce8009b00 comp rx=0 tx=0).stop 2023-03-03T08:09:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.186+0000 7f8cff7ff700 1 -- 172.21.15.98:0/2151740386 shutdown_connections 2023-03-03T08:09:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.186+0000 7f8cff7ff700 1 --2- 172.21.15.98:0/2151740386 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cf80b26d0 0x7f8cf80b4c00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.186+0000 7f8cff7ff700 1 -- 172.21.15.98:0/2151740386 >> 172.21.15.98:0/2151740386 conn(0x7f8cf8073e70 msgr2=0x7f8cf806ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.186+0000 7f8cff7ff700 1 -- 172.21.15.98:0/2151740386 shutdown_connections 2023-03-03T08:09:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.186+0000 7f8cff7ff700 1 -- 172.21.15.98:0/2151740386 wait complete. 2023-03-03T08:09:04.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.188+0000 7f8cff7ff700 1 Processor -- start 2023-03-03T08:09:04.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.188+0000 7f8cff7ff700 1 -- start start 2023-03-03T08:09:04.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.189+0000 7f8cff7ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cf80b26d0 0x7f8cf8145c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:04.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.189+0000 7f8cff7ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cf80b5140 0x7f8cf81461a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:04.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.189+0000 7f8cff7ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cf80b8d10 0x7f8cf81466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:04.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.189+0000 7f8cff7ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf80c8f90 con 0x7f8cf80b5140 2023-03-03T08:09:04.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.189+0000 7f8cff7ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf80c86c0 con 0x7f8cf80b26d0 2023-03-03T08:09:04.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.189+0000 7f8cff7ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf80ca7c0 con 0x7f8cf80b8d10 2023-03-03T08:09:04.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.190+0000 7f8cfcd9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cf80b5140 0x7f8cf81461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:04.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.190+0000 7f8cfcd9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cf80b5140 0x7f8cf81461a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54886/0 (socket says 172.21.15.98:54886) 2023-03-03T08:09:04.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.190+0000 7f8cfcd9a700 1 -- 172.21.15.98:0/284293591 learned_addr learned my addr 172.21.15.98:0/284293591 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:04.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.190+0000 7f8cfdd9c700 1 --2- 172.21.15.98:0/284293591 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cf80b8d10 0x7f8cf81466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:04.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.190+0000 7f8cfd59b700 1 --2- 172.21.15.98:0/284293591 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cf80b26d0 0x7f8cf8145c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:04.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.190+0000 7f8cfcd9a700 1 -- 172.21.15.98:0/284293591 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cf80b8d10 msgr2=0x7f8cf81466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:04.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.190+0000 7f8cfcd9a700 1 --2- 172.21.15.98:0/284293591 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cf80b8d10 0x7f8cf81466e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:04.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.190+0000 7f8cfcd9a700 1 -- 172.21.15.98:0/284293591 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cf80b26d0 msgr2=0x7f8cf8145c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:04.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.190+0000 7f8cfcd9a700 1 --2- 172.21.15.98:0/284293591 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cf80b26d0 0x7f8cf8145c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:04.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.190+0000 7f8cfcd9a700 1 -- 172.21.15.98:0/284293591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce80094a0 con 0x7f8cf80b5140 2023-03-03T08:09:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.191+0000 7f8cfcd9a700 1 --2- 172.21.15.98:0/284293591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cf80b5140 0x7f8cf81461a0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f8ce80090b0 tx=0x7f8ce8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.191+0000 7f8cedffb700 1 -- 172.21.15.98:0/284293591 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8003c10 con 0x7f8cf80b5140 2023-03-03T08:09:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.191+0000 7f8cedffb700 1 -- 172.21.15.98:0/284293591 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce801f070 con 0x7f8cf80b5140 2023-03-03T08:09:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.191+0000 7f8cedffb700 1 -- 172.21.15.98:0/284293591 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8011630 con 0x7f8cf80b5140 2023-03-03T08:09:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.191+0000 7f8cff7ff700 1 -- 172.21.15.98:0/284293591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf80d0e80 con 0x7f8cf80b5140 2023-03-03T08:09:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.191+0000 7f8cff7ff700 1 -- 172.21.15.98:0/284293591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf80cdd00 con 0x7f8cf80b5140 2023-03-03T08:09:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.192+0000 7f8cff7ff700 1 -- 172.21.15.98:0/284293591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf80cdfe0 con 0x7f8cf80b5140 2023-03-03T08:09:04.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.193+0000 7f8cedffb700 1 -- 172.21.15.98:0/284293591 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f8ce802a020 con 0x7f8cf80b5140 2023-03-03T08:09:04.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.195+0000 7f8cedffb700 1 --2- 172.21.15.98:0/284293591 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8cdc0683c0 0x7f8cdc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:04.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.195+0000 7f8cedffb700 1 -- 172.21.15.98:0/284293591 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f8ce8087500 con 0x7f8cf80b5140 2023-03-03T08:09:04.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.195+0000 7f8cfd59b700 1 --2- 172.21.15.98:0/284293591 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8cdc0683c0 0x7f8cdc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:04.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.196+0000 7f8cfd59b700 1 --2- 172.21.15.98:0/284293591 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8cdc0683c0 0x7f8cdc06a870 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8cf80cbc40 tx=0x7f8cf4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:04.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.198+0000 7f8cedffb700 1 -- 172.21.15.98:0/284293591 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8ce8053ac0 con 0x7f8cf80b5140 2023-03-03T08:09:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.389+0000 7f8cff7ff700 1 -- 172.21.15.98:0/284293591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8cf80cdfe0 con 0x7f8cf80b5140 2023-03-03T08:09:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.389+0000 7f8cedffb700 1 -- 172.21.15.98:0/284293591 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8ce80570e0 con 0x7f8cf80b5140 2023-03-03T08:09:04.391 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T08:09:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.392+0000 7f8cff7ff700 1 -- 172.21.15.98:0/284293591 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8cdc0683c0 msgr2=0x7f8cdc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.392+0000 7f8cff7ff700 1 --2- 172.21.15.98:0/284293591 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8cdc0683c0 0x7f8cdc06a870 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8cf80cbc40 tx=0x7f8cf4009040 comp rx=0 tx=0).stop 2023-03-03T08:09:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.392+0000 7f8cff7ff700 1 -- 172.21.15.98:0/284293591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cf80b5140 msgr2=0x7f8cf81461a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.392+0000 7f8cff7ff700 1 --2- 172.21.15.98:0/284293591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cf80b5140 0x7f8cf81461a0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f8ce80090b0 tx=0x7f8ce8004750 comp rx=0 tx=0).stop 2023-03-03T08:09:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.392+0000 7f8cff7ff700 1 -- 172.21.15.98:0/284293591 shutdown_connections 2023-03-03T08:09:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.392+0000 7f8cff7ff700 1 --2- 172.21.15.98:0/284293591 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cf80b8d10 0x7f8cf81466e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.392+0000 7f8cff7ff700 1 --2- 172.21.15.98:0/284293591 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cf80b26d0 0x7f8cf8145c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.392+0000 7f8cff7ff700 1 --2- 172.21.15.98:0/284293591 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8cdc0683c0 0x7f8cdc06a870 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.392+0000 7f8cff7ff700 1 --2- 172.21.15.98:0/284293591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cf80b5140 0x7f8cf81461a0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:04.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.392+0000 7f8cff7ff700 1 -- 172.21.15.98:0/284293591 >> 172.21.15.98:0/284293591 conn(0x7f8cf8073e70 msgr2=0x7f8cf80d12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:04.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.392+0000 7f8cff7ff700 1 -- 172.21.15.98:0/284293591 shutdown_connections 2023-03-03T08:09:04.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.392+0000 7f8cff7ff700 1 -- 172.21.15.98:0/284293591 wait complete. 2023-03-03T08:09:04.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T08:09:04.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.665+0000 7f37063ff700 1 Processor -- start 2023-03-03T08:09:04.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.665+0000 7f37063ff700 1 -- start start 2023-03-03T08:09:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.666+0000 7f37063ff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f370006b4b0 0x7f37000cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.666+0000 7f37063ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37000cf980 con 0x7f37000d0270 2023-03-03T08:09:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.666+0000 7f37063ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37000d1800 con 0x7f370015ca70 2023-03-03T08:09:04.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.666+0000 7f37063ff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37000cd5b0 con 0x7f370006b4b0 2023-03-03T08:09:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.666+0000 7f36ff7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f37000d0270 0x7f37000d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52952/0 (socket says 172.21.15.98:52952) 2023-03-03T08:09:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.666+0000 7f36ff7fe700 1 -- 172.21.15.98:0/2699681557 learned_addr learned my addr 172.21.15.98:0/2699681557 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:09:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.667+0000 7f370499c700 1 --2- 172.21.15.98:0/2699681557 >> v2:172.21.15.100:3300/0 conn(0x7f370006b4b0 0x7f37000cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.667+0000 7f36feffd700 1 -- 172.21.15.98:0/2699681557 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665297128 0 0) 0x7f37000cf980 con 0x7f37000d0270 2023-03-03T08:09:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.667+0000 7f36feffd700 1 -- 172.21.15.98:0/2699681557 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e80035b0 con 0x7f37000d0270 2023-03-03T08:09:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.667+0000 7f36feffd700 1 -- 172.21.15.98:0/2699681557 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975852396 0 0) 0x7f37000d1800 con 0x7f370015ca70 2023-03-03T08:09:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.667+0000 7f36feffd700 1 -- 172.21.15.98:0/2699681557 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37000cf980 con 0x7f370015ca70 2023-03-03T08:09:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.667+0000 7f36feffd700 1 -- 172.21.15.98:0/2699681557 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2113801031 0 0) 0x7f36e80035b0 con 0x7f37000d0270 2023-03-03T08:09:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.667+0000 7f370499c700 1 -- 172.21.15.98:0/2699681557 >> v1:172.21.15.100:6789/0 conn(0x7f370015ca70 legacy=0x7f37000cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.667+0000 7f370499c700 1 -- 172.21.15.98:0/2699681557 >> v1:172.21.15.98:6789/0 conn(0x7f37000d0270 legacy=0x7f37000d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.667+0000 7f370499c700 1 -- 172.21.15.98:0/2699681557 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37000cd900 con 0x7f370006b4b0 2023-03-03T08:09:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.668+0000 7f370499c700 1 --2- 172.21.15.98:0/2699681557 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f370006b4b0 0x7f37000cf440 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f36f8009d90 tx=0x7f36f800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b263800c7b6cbeb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.668+0000 7f36feffd700 1 -- 172.21.15.98:0/2699681557 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f800e070 con 0x7f370006b4b0 2023-03-03T08:09:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.668+0000 7f36feffd700 1 -- 172.21.15.98:0/2699681557 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f8003a20 con 0x7f370006b4b0 2023-03-03T08:09:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.668+0000 7f36feffd700 1 -- 172.21.15.98:0/2699681557 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f4001670 con 0x7f370006b4b0 2023-03-03T08:09:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.668+0000 7f37063ff700 1 -- 172.21.15.98:0/2699681557 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f370006b4b0 msgr2=0x7f37000cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.668+0000 7f37063ff700 1 --2- 172.21.15.98:0/2699681557 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f370006b4b0 0x7f37000cf440 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f36f8009d90 tx=0x7f36f800b330 comp rx=0 tx=0).stop 2023-03-03T08:09:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.669+0000 7f37063ff700 1 -- 172.21.15.98:0/2699681557 shutdown_connections 2023-03-03T08:09:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.669+0000 7f37063ff700 1 --2- 172.21.15.98:0/2699681557 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f370006b4b0 0x7f37000cf440 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.669+0000 7f37063ff700 1 -- 172.21.15.98:0/2699681557 >> 172.21.15.98:0/2699681557 conn(0x7f3700073e70 msgr2=0x7f370006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.669+0000 7f37063ff700 1 -- 172.21.15.98:0/2699681557 shutdown_connections 2023-03-03T08:09:04.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.669+0000 7f37063ff700 1 -- 172.21.15.98:0/2699681557 wait complete. 2023-03-03T08:09:04.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.671+0000 7f37063ff700 1 Processor -- start 2023-03-03T08:09:04.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.671+0000 7f37063ff700 1 -- start start 2023-03-03T08:09:04.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.672+0000 7f37063ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f370006b4b0 0x7f37000b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:04.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.672+0000 7f37063ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f37000d0270 0x7f3700157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:04.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.672+0000 7f37063ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f370015ca70 0x7f370015ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:04.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.672+0000 7f37063ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370015e3b0 con 0x7f370006b4b0 2023-03-03T08:09:04.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.672+0000 7f37063ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37000b39b0 con 0x7f37000d0270 2023-03-03T08:09:04.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.672+0000 7f37063ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37000b3c90 con 0x7f370015ca70 2023-03-03T08:09:04.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.672+0000 7f36fffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f370006b4b0 0x7f37000b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:04.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.672+0000 7f36fffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f370006b4b0 0x7f37000b4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54896/0 (socket says 172.21.15.98:54896) 2023-03-03T08:09:04.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.672+0000 7f36fffff700 1 -- 172.21.15.98:0/1360882070 learned_addr learned my addr 172.21.15.98:0/1360882070 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:04.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.673+0000 7f36ff7fe700 1 --2- 172.21.15.98:0/1360882070 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f37000d0270 0x7f3700157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.673+0000 7f370499c700 1 --2- 172.21.15.98:0/1360882070 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f370015ca70 0x7f370015ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.673+0000 7f36fffff700 1 -- 172.21.15.98:0/1360882070 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f370015ca70 msgr2=0x7f370015ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.673+0000 7f36fffff700 1 --2- 172.21.15.98:0/1360882070 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f370015ca70 0x7f370015ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.673+0000 7f36fffff700 1 -- 172.21.15.98:0/1360882070 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f37000d0270 msgr2=0x7f3700157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.673+0000 7f36fffff700 1 --2- 172.21.15.98:0/1360882070 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f37000d0270 0x7f3700157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.673+0000 7f36fffff700 1 -- 172.21.15.98:0/1360882070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f8009a70 con 0x7f370006b4b0 2023-03-03T08:09:04.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.673+0000 7f36fffff700 1 --2- 172.21.15.98:0/1360882070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f370006b4b0 0x7f37000b4b10 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f36f000c500 tx=0x7f36f000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:04.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.673+0000 7f36fcff9700 1 -- 172.21.15.98:0/1360882070 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f00052d0 con 0x7f370006b4b0 2023-03-03T08:09:04.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.673+0000 7f36fcff9700 1 -- 172.21.15.98:0/1360882070 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f0005450 con 0x7f370006b4b0 2023-03-03T08:09:04.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.673+0000 7f36fcff9700 1 -- 172.21.15.98:0/1360882070 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f00038c0 con 0x7f370006b4b0 2023-03-03T08:09:04.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.673+0000 7f37063ff700 1 -- 172.21.15.98:0/1360882070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37000d1aa0 con 0x7f370006b4b0 2023-03-03T08:09:04.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.674+0000 7f37063ff700 1 -- 172.21.15.98:0/1360882070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37000cdbe0 con 0x7f370006b4b0 2023-03-03T08:09:04.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.675+0000 7f37063ff700 1 -- 172.21.15.98:0/1360882070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f370015ce40 con 0x7f370006b4b0 2023-03-03T08:09:04.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.676+0000 7f36fcff9700 1 -- 172.21.15.98:0/1360882070 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f36f0019030 con 0x7f370006b4b0 2023-03-03T08:09:04.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.677+0000 7f36fcff9700 1 --2- 172.21.15.98:0/1360882070 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f36ec0683c0 0x7f36ec06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:04.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.677+0000 7f36fcff9700 1 -- 172.21.15.98:0/1360882070 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f36f0085360 con 0x7f370006b4b0 2023-03-03T08:09:04.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.678+0000 7f36ff7fe700 1 --2- 172.21.15.98:0/1360882070 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f36ec0683c0 0x7f36ec06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:04.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.678+0000 7f36ff7fe700 1 --2- 172.21.15.98:0/1360882070 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f36ec0683c0 0x7f36ec06a870 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f37000cd480 tx=0x7f36f4001c20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:04.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.681+0000 7f36fcff9700 1 -- 172.21.15.98:0/1360882070 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f36f0051920 con 0x7f370006b4b0 2023-03-03T08:09:04.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.872+0000 7f37063ff700 1 -- 172.21.15.98:0/1360882070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f37000d0ba0 con 0x7f370006b4b0 2023-03-03T08:09:04.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.872+0000 7f36fcff9700 1 -- 172.21.15.98:0/1360882070 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7f36f0054f40 con 0x7f370006b4b0 2023-03-03T08:09:04.874 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T08:09:04.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.875+0000 7f37063ff700 1 -- 172.21.15.98:0/1360882070 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f36ec0683c0 msgr2=0x7f36ec06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:04.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.875+0000 7f37063ff700 1 --2- 172.21.15.98:0/1360882070 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f36ec0683c0 0x7f36ec06a870 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f37000cd480 tx=0x7f36f4001c20 comp rx=0 tx=0).stop 2023-03-03T08:09:04.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.875+0000 7f37063ff700 1 -- 172.21.15.98:0/1360882070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f370006b4b0 msgr2=0x7f37000b4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:04.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.875+0000 7f37063ff700 1 --2- 172.21.15.98:0/1360882070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f370006b4b0 0x7f37000b4b10 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f36f000c500 tx=0x7f36f000c8c0 comp rx=0 tx=0).stop 2023-03-03T08:09:04.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.875+0000 7f37063ff700 1 -- 172.21.15.98:0/1360882070 shutdown_connections 2023-03-03T08:09:04.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.875+0000 7f37063ff700 1 --2- 172.21.15.98:0/1360882070 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f36ec0683c0 0x7f36ec06a870 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:04.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.875+0000 7f37063ff700 1 --2- 172.21.15.98:0/1360882070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f370006b4b0 0x7f37000b4b10 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:04.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.875+0000 7f37063ff700 1 --2- 172.21.15.98:0/1360882070 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f370015ca70 0x7f370015ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:04.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.875+0000 7f37063ff700 1 --2- 172.21.15.98:0/1360882070 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f37000d0270 0x7f3700157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:04.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.875+0000 7f37063ff700 1 -- 172.21.15.98:0/1360882070 >> 172.21.15.98:0/1360882070 conn(0x7f3700073e70 msgr2=0x7f37000b7720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:04.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.875+0000 7f37063ff700 1 -- 172.21.15.98:0/1360882070 shutdown_connections 2023-03-03T08:09:04.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:04.875+0000 7f37063ff700 1 -- 172.21.15.98:0/1360882070 wait complete. 2023-03-03T08:09:04.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T08:09:05.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.147+0000 7f8b6efff700 1 Processor -- start 2023-03-03T08:09:05.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.147+0000 7f8b6efff700 1 -- start start 2023-03-03T08:09:05.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.149+0000 7f8b6efff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8b6806b4b0 0x7f8b680cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:05.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.149+0000 7f8b6efff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b680cf980 con 0x7f8b680d0270 2023-03-03T08:09:05.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.149+0000 7f8b6efff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b680d1800 con 0x7f8b6815ca70 2023-03-03T08:09:05.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.149+0000 7f8b6efff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b680cd5b0 con 0x7f8b6806b4b0 2023-03-03T08:09:05.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.149+0000 7f8b67fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8b680d0270 0x7f8b680d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38660/0 (socket says 172.21.15.98:38660) 2023-03-03T08:09:05.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.149+0000 7f8b67fff700 1 -- 172.21.15.98:0/3270681183 learned_addr learned my addr 172.21.15.98:0/3270681183 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:09:05.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.149+0000 7f8b6d59c700 1 --2- 172.21.15.98:0/3270681183 >> v2:172.21.15.98:3300/0 conn(0x7f8b6806b4b0 0x7f8b680cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:05.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.150+0000 7f8b677fe700 1 -- 172.21.15.98:0/3270681183 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720195650 0 0) 0x7f8b680cf980 con 0x7f8b680d0270 2023-03-03T08:09:05.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.150+0000 7f8b677fe700 1 -- 172.21.15.98:0/3270681183 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b580035b0 con 0x7f8b680d0270 2023-03-03T08:09:05.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.150+0000 7f8b6d59c700 1 -- 172.21.15.98:0/3270681183 >> v1:172.21.15.100:6789/0 conn(0x7f8b6815ca70 legacy=0x7f8b680cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:05.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.150+0000 7f8b677fe700 1 -- 172.21.15.98:0/3270681183 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599988894 0 0) 0x7f8b680d1800 con 0x7f8b6815ca70 2023-03-03T08:09:05.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.150+0000 7f8b6d59c700 1 -- 172.21.15.98:0/3270681183 >> v1:172.21.15.98:6789/0 conn(0x7f8b680d0270 legacy=0x7f8b680d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:05.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.150+0000 7f8b6d59c700 1 -- 172.21.15.98:0/3270681183 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b680cd900 con 0x7f8b6806b4b0 2023-03-03T08:09:05.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.150+0000 7f8b6d59c700 1 --2- 172.21.15.98:0/3270681183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b6806b4b0 0x7f8b680cf440 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f8b60009d90 tx=0x7f8b6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86ec6237f6d41ba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:05.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.150+0000 7f8b677fe700 1 -- 172.21.15.98:0/3270681183 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6000e070 con 0x7f8b6806b4b0 2023-03-03T08:09:05.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.150+0000 7f8b677fe700 1 -- 172.21.15.98:0/3270681183 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b6000bb20 con 0x7f8b6806b4b0 2023-03-03T08:09:05.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.150+0000 7f8b677fe700 1 -- 172.21.15.98:0/3270681183 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6000bed0 con 0x7f8b6806b4b0 2023-03-03T08:09:05.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.151+0000 7f8b6efff700 1 -- 172.21.15.98:0/3270681183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b6806b4b0 msgr2=0x7f8b680cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:05.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.151+0000 7f8b6efff700 1 --2- 172.21.15.98:0/3270681183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b6806b4b0 0x7f8b680cf440 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f8b60009d90 tx=0x7f8b6000b330 comp rx=0 tx=0).stop 2023-03-03T08:09:05.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.151+0000 7f8b6efff700 1 -- 172.21.15.98:0/3270681183 shutdown_connections 2023-03-03T08:09:05.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.151+0000 7f8b6efff700 1 --2- 172.21.15.98:0/3270681183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b6806b4b0 0x7f8b680cf440 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:05.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.151+0000 7f8b6efff700 1 -- 172.21.15.98:0/3270681183 >> 172.21.15.98:0/3270681183 conn(0x7f8b68073e70 msgr2=0x7f8b6806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:05.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.151+0000 7f8b6efff700 1 -- 172.21.15.98:0/3270681183 shutdown_connections 2023-03-03T08:09:05.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.151+0000 7f8b6efff700 1 -- 172.21.15.98:0/3270681183 wait complete. 2023-03-03T08:09:05.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.153+0000 7f8b6efff700 1 Processor -- start 2023-03-03T08:09:05.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.153+0000 7f8b6efff700 1 -- start start 2023-03-03T08:09:05.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.154+0000 7f8b6efff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b6806b4b0 0x7f8b6815f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:05.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.154+0000 7f8b6efff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b680d0270 0x7f8b680b6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:05.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.154+0000 7f8b6efff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b6815ca70 0x7f8b6815d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.154+0000 7f8b6efff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b681f57d0 con 0x7f8b680d0270 2023-03-03T08:09:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.154+0000 7f8b6efff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6815f7a0 con 0x7f8b6806b4b0 2023-03-03T08:09:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.154+0000 7f8b6efff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6815fa80 con 0x7f8b6815ca70 2023-03-03T08:09:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.155+0000 7f8b67fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b680d0270 0x7f8b680b6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.155+0000 7f8b67fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b680d0270 0x7f8b680b6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44940/0 (socket says 172.21.15.98:44940) 2023-03-03T08:09:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.155+0000 7f8b67fff700 1 -- 172.21.15.98:0/2218575796 learned_addr learned my addr 172.21.15.98:0/2218575796 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.155+0000 7f8b6cd9b700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b6806b4b0 0x7f8b6815f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.155+0000 7f8b6d59c700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b6815ca70 0x7f8b6815d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:05.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.155+0000 7f8b67fff700 1 -- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b6815ca70 msgr2=0x7f8b6815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:05.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.155+0000 7f8b67fff700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b6815ca70 0x7f8b6815d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:05.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.155+0000 7f8b67fff700 1 -- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b6806b4b0 msgr2=0x7f8b6815f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:05.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.155+0000 7f8b67fff700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b6806b4b0 0x7f8b6815f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:05.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.155+0000 7f8b67fff700 1 -- 172.21.15.98:0/2218575796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b60009a70 con 0x7f8b680d0270 2023-03-03T08:09:05.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.155+0000 7f8b6d59c700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b6815ca70 0x7f8b6815d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:09:05.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.155+0000 7f8b6cd9b700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b6806b4b0 0x7f8b6815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:09:05.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.155+0000 7f8b67fff700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b680d0270 0x7f8b680b6790 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400c500 tx=0x7f8b5400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:05.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.155+0000 7f8b657fa700 1 -- 172.21.15.98:0/2218575796 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54005e30 con 0x7f8b680d0270 2023-03-03T08:09:05.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.156+0000 7f8b657fa700 1 -- 172.21.15.98:0/2218575796 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b540070d0 con 0x7f8b680d0270 2023-03-03T08:09:05.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.156+0000 7f8b6efff700 1 -- 172.21.15.98:0/2218575796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b680d1aa0 con 0x7f8b680d0270 2023-03-03T08:09:05.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.156+0000 7f8b657fa700 1 -- 172.21.15.98:0/2218575796 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54007480 con 0x7f8b680d0270 2023-03-03T08:09:05.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.156+0000 7f8b6efff700 1 -- 172.21.15.98:0/2218575796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b680cdbe0 con 0x7f8b680d0270 2023-03-03T08:09:05.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.157+0000 7f8b6efff700 1 -- 172.21.15.98:0/2218575796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b6815ce40 con 0x7f8b680d0270 2023-03-03T08:09:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.158+0000 7f8b657fa700 1 -- 172.21.15.98:0/2218575796 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f8b54019030 con 0x7f8b680d0270 2023-03-03T08:09:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.159+0000 7f8b657fa700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8b4c0683c0 0x7f8b4c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.159+0000 7f8b657fa700 1 -- 172.21.15.98:0/2218575796 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f8b54085390 con 0x7f8b680d0270 2023-03-03T08:09:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.160+0000 7f8b6cd9b700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8b4c0683c0 0x7f8b4c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.160+0000 7f8b6cd9b700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8b4c0683c0 0x7f8b4c06a870 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c000f60 tx=0x7f8b5c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.163+0000 7f8b657fa700 1 -- 172.21.15.98:0/2218575796 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b54051950 con 0x7f8b680d0270 2023-03-03T08:09:05.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.370+0000 7f8b6efff700 1 -- 172.21.15.98:0/2218575796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8b681f6710 con 0x7f8b680d0270 2023-03-03T08:09:05.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.370+0000 7f8b657fa700 1 -- 172.21.15.98:0/2218575796 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v114) v1 ==== 80+0+368 (secure 0 0 0) 0x7f8b54017180 con 0x7f8b680d0270 2023-03-03T08:09:05.372 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6832/2374561485 2023-03-04T08:04:52.583414+0000 2023-03-03T08:09:05.372 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6835/2695503624 2023-03-04T08:04:32.578044+0000 2023-03-03T08:09:05.372 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6833/2695503624 2023-03-04T08:04:32.578044+0000 2023-03-03T08:09:05.372 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6834/2374561485 2023-03-04T08:04:52.583414+0000 2023-03-03T08:09:05.373 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.100:6835/2660525287 2023-03-04T08:04:12.571647+0000 2023-03-03T08:09:05.373 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.100:6833/2660525287 2023-03-04T08:04:12.571647+0000 2023-03-03T08:09:05.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.373+0000 7f8b6efff700 1 -- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8b4c0683c0 msgr2=0x7f8b4c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:05.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.373+0000 7f8b6efff700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8b4c0683c0 0x7f8b4c06a870 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c000f60 tx=0x7f8b5c009040 comp rx=0 tx=0).stop 2023-03-03T08:09:05.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.373+0000 7f8b6efff700 1 -- 172.21.15.98:0/2218575796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b680d0270 msgr2=0x7f8b680b6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:05.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.373+0000 7f8b6efff700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b680d0270 0x7f8b680b6790 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400c500 tx=0x7f8b5400c8c0 comp rx=0 tx=0).stop 2023-03-03T08:09:05.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.373+0000 7f8b6efff700 1 -- 172.21.15.98:0/2218575796 shutdown_connections 2023-03-03T08:09:05.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.373+0000 7f8b6efff700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b6815ca70 0x7f8b6815d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:05.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.373+0000 7f8b6efff700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b6806b4b0 0x7f8b6815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:05.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.373+0000 7f8b6efff700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f8b4c0683c0 0x7f8b4c06a870 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:05.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.373+0000 7f8b6efff700 1 --2- 172.21.15.98:0/2218575796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b680d0270 0x7f8b680b6790 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:05.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.373+0000 7f8b6efff700 1 -- 172.21.15.98:0/2218575796 >> 172.21.15.98:0/2218575796 conn(0x7f8b68073e70 msgr2=0x7f8b6815a740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:05.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.373+0000 7f8b6efff700 1 -- 172.21.15.98:0/2218575796 shutdown_connections 2023-03-03T08:09:05.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.373+0000 7f8b6efff700 1 -- 172.21.15.98:0/2218575796 wait complete. 2023-03-03T08:09:05.377 INFO:teuthology.orchestra.run.smithi098.stderr:listed 6 entries 2023-03-03T08:09:05.401 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-03-03T08:09:05.402 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:09:05.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:09:05.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.713+0000 7f9e3a19d700 1 Processor -- start 2023-03-03T08:09:05.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.713+0000 7f9e3a19d700 1 -- start start 2023-03-03T08:09:05.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.715+0000 7f9e3a19d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e340b51c0 0x7f9e34160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:05.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.715+0000 7f9e3a19d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e341614c0 con 0x7f9e340d1580 2023-03-03T08:09:05.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.715+0000 7f9e3a19d700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e340d10f0 con 0x7f9e34146360 2023-03-03T08:09:05.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.715+0000 7f9e3a19d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e340cdf20 con 0x7f9e340b51c0 2023-03-03T08:09:05.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.715+0000 7f9e3899a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e340b51c0 0x7f9e34160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:05.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.715+0000 7f9e3999c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9e340d1580 0x7f9e340d1980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38670/0 (socket says 172.21.15.98:38670) 2023-03-03T08:09:05.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.715+0000 7f9e3999c700 1 -- 172.21.15.98:0/466887790 learned_addr learned my addr 172.21.15.98:0/466887790 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:09:05.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.716+0000 7f9e3899a700 1 -- 172.21.15.98:0/466887790 >> v1:172.21.15.100:6789/0 conn(0x7f9e34146360 legacy=0x7f9e340b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:05.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.716+0000 7f9e3899a700 1 -- 172.21.15.98:0/466887790 >> v1:172.21.15.98:6789/0 conn(0x7f9e340d1580 legacy=0x7f9e340d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:05.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.716+0000 7f9e3899a700 1 -- 172.21.15.98:0/466887790 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e340cd680 con 0x7f9e340b51c0 2023-03-03T08:09:05.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.716+0000 7f9e2bfff700 1 -- 172.21.15.98:0/466887790 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322211807 0 0) 0x7f9e2c001490 con 0x7f9e340d1580 2023-03-03T08:09:05.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.716+0000 7f9e3899a700 1 --2- 172.21.15.98:0/466887790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e340b51c0 0x7f9e34160f80 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009840 tx=0x7f9e24009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7438c59aa5bbb41 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:05.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.716+0000 7f9e2bfff700 1 -- 172.21.15.98:0/466887790 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e24011830 con 0x7f9e340b51c0 2023-03-03T08:09:05.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.716+0000 7f9e2bfff700 1 -- 172.21.15.98:0/466887790 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e24011990 con 0x7f9e340b51c0 2023-03-03T08:09:05.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.716+0000 7f9e2bfff700 1 -- 172.21.15.98:0/466887790 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e24011d20 con 0x7f9e340b51c0 2023-03-03T08:09:05.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.717+0000 7f9e3a19d700 1 -- 172.21.15.98:0/466887790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e340b51c0 msgr2=0x7f9e34160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:05.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.717+0000 7f9e3a19d700 1 --2- 172.21.15.98:0/466887790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e340b51c0 0x7f9e34160f80 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009840 tx=0x7f9e24009b50 comp rx=0 tx=0).stop 2023-03-03T08:09:05.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.717+0000 7f9e3a19d700 1 -- 172.21.15.98:0/466887790 shutdown_connections 2023-03-03T08:09:05.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.717+0000 7f9e3a19d700 1 --2- 172.21.15.98:0/466887790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e340b51c0 0x7f9e34160f80 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:05.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.717+0000 7f9e3a19d700 1 -- 172.21.15.98:0/466887790 >> 172.21.15.98:0/466887790 conn(0x7f9e3406b8e0 msgr2=0x7f9e340cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:05.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.717+0000 7f9e3a19d700 1 -- 172.21.15.98:0/466887790 shutdown_connections 2023-03-03T08:09:05.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.717+0000 7f9e3a19d700 1 -- 172.21.15.98:0/466887790 wait complete. 2023-03-03T08:09:05.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.719+0000 7f9e3a19d700 1 Processor -- start 2023-03-03T08:09:05.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.719+0000 7f9e3a19d700 1 -- start start 2023-03-03T08:09:05.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.720+0000 7f9e3a19d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e340b51c0 0x7f9e341f5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:05.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.720+0000 7f9e3a19d700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e340d1580 0x7f9e340c7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:05.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.720+0000 7f9e3a19d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e34146360 0x7f9e34159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:05.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.720+0000 7f9e3a19d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e340c8d10 con 0x7f9e34146360 2023-03-03T08:09:05.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.720+0000 7f9e3a19d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e341f6040 con 0x7f9e340b51c0 2023-03-03T08:09:05.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.720+0000 7f9e3a19d700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e340c8140 con 0x7f9e340d1580 2023-03-03T08:09:05.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.721+0000 7f9e3999c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e34146360 0x7f9e34159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:05.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.721+0000 7f9e3999c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e34146360 0x7f9e34159990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44962/0 (socket says 172.21.15.98:44962) 2023-03-03T08:09:05.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.721+0000 7f9e3999c700 1 -- 172.21.15.98:0/3704540398 learned_addr learned my addr 172.21.15.98:0/3704540398 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:05.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.721+0000 7f9e3919b700 1 --2- 172.21.15.98:0/3704540398 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e340b51c0 0x7f9e341f5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:05.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.721+0000 7f9e3899a700 1 --2- 172.21.15.98:0/3704540398 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e340d1580 0x7f9e340c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:05.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.721+0000 7f9e3999c700 1 -- 172.21.15.98:0/3704540398 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e340d1580 msgr2=0x7f9e340c7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:05.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.721+0000 7f9e3999c700 1 --2- 172.21.15.98:0/3704540398 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e340d1580 0x7f9e340c7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:05.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.721+0000 7f9e3999c700 1 -- 172.21.15.98:0/3704540398 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e340b51c0 msgr2=0x7f9e341f5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:05.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.721+0000 7f9e3999c700 1 --2- 172.21.15.98:0/3704540398 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e340b51c0 0x7f9e341f5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:05.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.721+0000 7f9e3999c700 1 -- 172.21.15.98:0/3704540398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e240094a0 con 0x7f9e34146360 2023-03-03T08:09:05.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.721+0000 7f9e3919b700 1 --2- 172.21.15.98:0/3704540398 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e340b51c0 0x7f9e341f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:09:05.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.722+0000 7f9e3999c700 1 --2- 172.21.15.98:0/3704540398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e34146360 0x7f9e34159990 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c00ccd0 tx=0x7f9e2c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:05.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.722+0000 7f9e29ffb700 1 -- 172.21.15.98:0/3704540398 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c00ad40 con 0x7f9e34146360 2023-03-03T08:09:05.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.722+0000 7f9e29ffb700 1 -- 172.21.15.98:0/3704540398 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2c0070d0 con 0x7f9e34146360 2023-03-03T08:09:05.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.722+0000 7f9e29ffb700 1 -- 172.21.15.98:0/3704540398 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c007480 con 0x7f9e34146360 2023-03-03T08:09:05.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.722+0000 7f9e3a19d700 1 -- 172.21.15.98:0/3704540398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e340cdc40 con 0x7f9e34146360 2023-03-03T08:09:05.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.722+0000 7f9e3a19d700 1 -- 172.21.15.98:0/3704540398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e340cd960 con 0x7f9e34146360 2023-03-03T08:09:05.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.723+0000 7f9e3a19d700 1 -- 172.21.15.98:0/3704540398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e340d0d80 con 0x7f9e34146360 2023-03-03T08:09:05.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.725+0000 7f9e29ffb700 1 -- 172.21.15.98:0/3704540398 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f9e2c010070 con 0x7f9e34146360 2023-03-03T08:09:05.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.726+0000 7f9e29ffb700 1 --2- 172.21.15.98:0/3704540398 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9e180683c0 0x7f9e1806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:05.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.726+0000 7f9e29ffb700 1 -- 172.21.15.98:0/3704540398 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f9e2c085880 con 0x7f9e34146360 2023-03-03T08:09:05.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.726+0000 7f9e3919b700 1 --2- 172.21.15.98:0/3704540398 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9e180683c0 0x7f9e1806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:05.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.727+0000 7f9e3919b700 1 --2- 172.21.15.98:0/3704540398 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9e180683c0 0x7f9e1806a870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9e30000f60 tx=0x7f9e30009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:05.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.729+0000 7f9e29ffb700 1 -- 172.21.15.98:0/3704540398 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9e2c051e40 con 0x7f9e34146360 2023-03-03T08:09:05.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.920+0000 7f9e3a19d700 1 -- 172.21.15.98:0/3704540398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e340c8ec0 con 0x7f9e34146360 2023-03-03T08:09:05.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.921+0000 7f9e29ffb700 1 -- 172.21.15.98:0/3704540398 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+18380 (secure 0 0 0) 0x7f9e340c8ec0 con 0x7f9e34146360 2023-03-03T08:09:05.923 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-03-03T08:09:05.923 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":114,"fsid":"fbbd322f-236a-44f8-96c5-be6963846c9b","created":"2023-03-03T07:55:17.653932+0000","modified":"2023-03-03T08:04:52.583483+0000","last_up_change":"2023-03-03T07:55:31.421636+0000","last_in_change":"2023-03-03T07:55:21.740708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T07:55:32.263759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T07:55:38.678356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-03-03T08:03:11.891955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-03-03T08:03:12.507916+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18707639-bc5c-49a9-b8b6-d563deea2dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":109121},{"type":"v1","addr":"172.21.15.98:6825","nonce":109121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":109121},{"type":"v1","addr":"172.21.15.98:6827","nonce":109121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":109121},{"type":"v1","addr":"172.21.15.98:6831","nonce":109121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":109121},{"type":"v1","addr":"172.21.15.98:6829","nonce":109121}]},"public_addr":"172.21.15.98:6825/109121","cluster_addr":"172.21.15.98:6827/109121","heartbeat_back_addr":"172.21.15.98:6831/109121","heartbeat_front_addr":"172.21.15.98:6829/109121","state":["exists","up"]},{"osd":1,"uuid":"6e6ec2b2-e35f-4d13-a06a-a6f58af34e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":109118},{"type":"v1","addr":"172.21.15.98:6801","nonce":109118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":109118},{"type":"v1","addr":"172.21.15.98:6803","nonce":109118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":109118},{"type":"v1","addr":"172.21.15.98:6807","nonce":109118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":109118},{"type":"v1","addr":"172.21.15.98:6805","nonce":109118}]},"public_addr":"172.21.15.98:6801/109118","cluster_addr":"172.21.15.98:6803/109118","heartbeat_back_addr":"172.21.15.98:6807/109118","heartbeat_front_addr":"172.21.15.98:6805/109118","state":["exists","up"]},{"osd":2,"uuid":"8e6fc347-1bd9-4381-8410-2e0110bd5052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":109122},{"type":"v1","addr":"172.21.15.98:6809","nonce":109122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":109122},{"type":"v1","addr":"172.21.15.98:6811","nonce":109122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":109122},{"type":"v1","addr":"172.21.15.98:6815","nonce":109122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":109122},{"type":"v1","addr":"172.21.15.98:6813","nonce":109122}]},"public_addr":"172.21.15.98:6809/109122","cluster_addr":"172.21.15.98:6811/109122","heartbeat_back_addr":"172.21.15.98:6815/109122","heartbeat_front_addr":"172.21.15.98:6813/109122","state":["exists","up"]},{"osd":3,"uuid":"cca208ef-d27f-4daf-b3c8-8bf22b693bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":109119},{"type":"v1","addr":"172.21.15.98:6817","nonce":109119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":109119},{"type":"v1","addr":"172.21.15.98:6819","nonce":109119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":109119},{"type":"v1","addr":"172.21.15.98:6823","nonce":109119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":109119},{"type":"v1","addr":"172.21.15.98:6821","nonce":109119}]},"public_addr":"172.21.15.98:6817/109119","cluster_addr":"172.21.15.98:6819/109119","heartbeat_back_addr":"172.21.15.98:6823/109119","heartbeat_front_addr":"172.21.15.98:6821/109119","state":["exists","up"]},{"osd":4,"uuid":"8f10aa62-7454-4d09-a38c-0ed613912eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6819","nonce":109381},{"type":"v1","addr":"172.21.15.100:6823","nonce":109381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":109381},{"type":"v1","addr":"172.21.15.100:6827","nonce":109381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":109381},{"type":"v1","addr":"172.21.15.100:6831","nonce":109381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":109381},{"type":"v1","addr":"172.21.15.100:6829","nonce":109381}]},"public_addr":"172.21.15.100:6823/109381","cluster_addr":"172.21.15.100:6827/109381","heartbeat_back_addr":"172.21.15.100:6831/109381","heartbeat_front_addr":"172.21.15.100:6829/109381","state":["exists","up"]},{"osd":5,"uuid":"265a0fef-4906-46a8-bf8e-505bdb8c3180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":109379},{"type":"v1","addr":"172.21.15.100:6810","nonce":109379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":109379},{"type":"v1","addr":"172.21.15.100:6813","nonce":109379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":109379},{"type":"v1","addr":"172.21.15.100:6820","nonce":109379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":109379},{"type":"v1","addr":"172.21.15.100:6816","nonce":109379}]},"public_addr":"172.21.15.100:6810/109379","cluster_addr":"172.21.15.100:6813/109379","heartbeat_back_addr":"172.21.15.100:6820/109379","heartbeat_front_addr":"172.21.15.100:6816/109379","state":["exists","up"]},{"osd":6,"uuid":"77a7054d-959a-437e-9c4b-ed58e78b58b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":109382},{"type":"v1","addr":"172.21.15.100:6812","nonce":109382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":109382},{"type":"v1","addr":"172.21.15.100:6817","nonce":109382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":109382},{"type":"v1","addr":"172.21.15.100:6826","nonce":109382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":109382},{"type":"v1","addr":"172.21.15.100:6822","nonce":109382}]},"public_addr":"172.21.15.100:6812/109382","cluster_addr":"172.21.15.100:6817/109382","heartbeat_back_addr":"172.21.15.100:6826/109382","heartbeat_front_addr":"172.21.15.100:6822/109382","state":["exists","up"]},{"osd":7,"uuid":"9701bb5b-c322-4341-9dec-4b4176c1fbbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":109380},{"type":"v1","addr":"172.21.15.100:6801","nonce":109380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":109380},{"type":"v1","addr":"172.21.15.100:6803","nonce":109380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":109380},{"type":"v1","addr":"172.21.15.100:6807","nonce":109380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":109380},{"type":"v1","addr":"172.21.15.100:6805","nonce":109380}]},"public_addr":"172.21.15.100:6801/109380","cluster_addr":"172.21.15.100:6803/109380","heartbeat_back_addr":"172.21.15.100:6807/109380","heartbeat_front_addr":"172.21.15.100:6805/109380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.546248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.536105+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.525594+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.515710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.657547+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.625695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.618480+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T07:55:27.668049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6832/2374561485":"2023-03-04T08:04:52.583414+0000","172.21.15.98:6835/2695503624":"2023-03-04T08:04:32.578044+0000","172.21.15.98:6833/2695503624":"2023-03-04T08:04:32.578044+0000","172.21.15.98:6834/2374561485":"2023-03-04T08:04:52.583414+0000","172.21.15.100:6835/2660525287":"2023-03-04T08:04:12.571647+0000","172.21.15.100:6833/2660525287":"2023-03-04T08:04:12.571647+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:09:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.924+0000 7f9e3a19d700 1 -- 172.21.15.98:0/3704540398 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9e180683c0 msgr2=0x7f9e1806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.924+0000 7f9e3a19d700 1 --2- 172.21.15.98:0/3704540398 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9e180683c0 0x7f9e1806a870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9e30000f60 tx=0x7f9e30009040 comp rx=0 tx=0).stop 2023-03-03T08:09:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.924+0000 7f9e3a19d700 1 -- 172.21.15.98:0/3704540398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e34146360 msgr2=0x7f9e34159990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.924+0000 7f9e3a19d700 1 --2- 172.21.15.98:0/3704540398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e34146360 0x7f9e34159990 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c00ccd0 tx=0x7f9e2c00a340 comp rx=0 tx=0).stop 2023-03-03T08:09:05.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.924+0000 7f9e3a19d700 1 -- 172.21.15.98:0/3704540398 shutdown_connections 2023-03-03T08:09:05.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.924+0000 7f9e3a19d700 1 --2- 172.21.15.98:0/3704540398 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f9e180683c0 0x7f9e1806a870 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:05.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.924+0000 7f9e3a19d700 1 --2- 172.21.15.98:0/3704540398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e34146360 0x7f9e34159990 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:05.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.924+0000 7f9e3a19d700 1 --2- 172.21.15.98:0/3704540398 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e340b51c0 0x7f9e341f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:05.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.924+0000 7f9e3a19d700 1 --2- 172.21.15.98:0/3704540398 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e340d1580 0x7f9e340c7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:05.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.924+0000 7f9e3a19d700 1 -- 172.21.15.98:0/3704540398 >> 172.21.15.98:0/3704540398 conn(0x7f9e3406b8e0 msgr2=0x7f9e3415afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:05.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.924+0000 7f9e3a19d700 1 -- 172.21.15.98:0/3704540398 shutdown_connections 2023-03-03T08:09:05.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:05.924+0000 7f9e3a19d700 1 -- 172.21.15.98:0/3704540398 wait complete. 2023-03-03T08:09:05.953 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T07:55:32.263759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T07:55:38.678356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T08:03:11.891955+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T08:03:12.507916+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:09:05.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:09:06.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.199+0000 7facf89ff700 1 Processor -- start 2023-03-03T08:09:06.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.199+0000 7facf89ff700 1 -- start start 2023-03-03T08:09:06.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.200+0000 7facf89ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facf40b26d0 0x7facf40b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:06.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.201+0000 7facf89ff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf40cd7d0 con 0x7facf40b8d10 2023-03-03T08:09:06.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.201+0000 7facf89ff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf40b90e0 con 0x7facf40b5140 2023-03-03T08:09:06.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.201+0000 7facf89ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf40b55c0 con 0x7facf40b26d0 2023-03-03T08:09:06.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.201+0000 7facf259c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7facf40b8d10 0x7facf40cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38686/0 (socket says 172.21.15.98:38686) 2023-03-03T08:09:06.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.201+0000 7facf259c700 1 -- 172.21.15.98:0/1072596205 learned_addr learned my addr 172.21.15.98:0/1072596205 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:09:06.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.201+0000 7facf1d9b700 1 --2- 172.21.15.98:0/1072596205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facf40b26d0 0x7facf40b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:06.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.202+0000 7facf159a700 1 -- 172.21.15.98:0/1072596205 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022257462 0 0) 0x7facf40cd7d0 con 0x7facf40b8d10 2023-03-03T08:09:06.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.202+0000 7facf159a700 1 -- 172.21.15.98:0/1072596205 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facdc0035b0 con 0x7facf40b8d10 2023-03-03T08:09:06.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.202+0000 7facf1d9b700 1 -- 172.21.15.98:0/1072596205 >> v1:172.21.15.100:6789/0 conn(0x7facf40b5140 legacy=0x7facf40d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:06.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.202+0000 7facf159a700 1 -- 172.21.15.98:0/1072596205 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121672834 0 0) 0x7facf40b90e0 con 0x7facf40b5140 2023-03-03T08:09:06.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.202+0000 7facf1d9b700 1 -- 172.21.15.98:0/1072596205 >> v1:172.21.15.98:6789/0 conn(0x7facf40b8d10 legacy=0x7facf40cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:06.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.202+0000 7facf1d9b700 1 -- 172.21.15.98:0/1072596205 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf40cda20 con 0x7facf40b26d0 2023-03-03T08:09:06.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.202+0000 7facf1d9b700 1 --2- 172.21.15.98:0/1072596205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facf40b26d0 0x7facf40b4c00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7face8009870 tx=0x7face8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95df0a3c74b85043 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:06.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.203+0000 7facf159a700 1 -- 172.21.15.98:0/1072596205 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face8011830 con 0x7facf40b26d0 2023-03-03T08:09:06.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.203+0000 7facf159a700 1 -- 172.21.15.98:0/1072596205 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face8011990 con 0x7facf40b26d0 2023-03-03T08:09:06.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.203+0000 7facf159a700 1 -- 172.21.15.98:0/1072596205 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face8011d00 con 0x7facf40b26d0 2023-03-03T08:09:06.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.203+0000 7facf89ff700 1 -- 172.21.15.98:0/1072596205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facf40b26d0 msgr2=0x7facf40b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:06.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.203+0000 7facf89ff700 1 --2- 172.21.15.98:0/1072596205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facf40b26d0 0x7facf40b4c00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7face8009870 tx=0x7face8009b80 comp rx=0 tx=0).stop 2023-03-03T08:09:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.203+0000 7facf89ff700 1 -- 172.21.15.98:0/1072596205 shutdown_connections 2023-03-03T08:09:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.203+0000 7facf89ff700 1 --2- 172.21.15.98:0/1072596205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facf40b26d0 0x7facf40b4c00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.203+0000 7facf89ff700 1 -- 172.21.15.98:0/1072596205 >> 172.21.15.98:0/1072596205 conn(0x7facf4073e70 msgr2=0x7facf406ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.203+0000 7facf89ff700 1 -- 172.21.15.98:0/1072596205 shutdown_connections 2023-03-03T08:09:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.204+0000 7facf89ff700 1 -- 172.21.15.98:0/1072596205 wait complete. 2023-03-03T08:09:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.205+0000 7facf89ff700 1 Processor -- start 2023-03-03T08:09:06.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.206+0000 7facf89ff700 1 -- start start 2023-03-03T08:09:06.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf89ff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facf40b26d0 0x7facf415faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:06.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf89ff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facf40b5140 0x7facf4159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:06.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf89ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facf40b8d10 0x7facf40ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:06.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf89ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf40c8b00 con 0x7facf40b8d10 2023-03-03T08:09:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf89ff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf415a210 con 0x7facf40b5140 2023-03-03T08:09:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf89ff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf40c7ef0 con 0x7facf40b26d0 2023-03-03T08:09:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf2d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facf40b8d10 0x7facf40ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf2d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facf40b8d10 0x7facf40ca250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44972/0 (socket says 172.21.15.98:44972) 2023-03-03T08:09:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf2d9d700 1 -- 172.21.15.98:0/566392397 learned_addr learned my addr 172.21.15.98:0/566392397 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf259c700 1 --2- 172.21.15.98:0/566392397 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facf40b26d0 0x7facf415faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf1d9b700 1 --2- 172.21.15.98:0/566392397 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facf40b5140 0x7facf4159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf2d9d700 1 -- 172.21.15.98:0/566392397 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facf40b26d0 msgr2=0x7facf415faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf2d9d700 1 --2- 172.21.15.98:0/566392397 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facf40b26d0 0x7facf415faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf2d9d700 1 -- 172.21.15.98:0/566392397 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facf40b5140 msgr2=0x7facf4159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf2d9d700 1 --2- 172.21.15.98:0/566392397 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facf40b5140 0x7facf4159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.207+0000 7facf2d9d700 1 -- 172.21.15.98:0/566392397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face80094a0 con 0x7facf40b8d10 2023-03-03T08:09:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.208+0000 7facf2d9d700 1 --2- 172.21.15.98:0/566392397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facf40b8d10 0x7facf40ca250 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7facec00c760 tx=0x7facec00ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.208+0000 7face2ffd700 1 -- 172.21.15.98:0/566392397 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec003e30 con 0x7facf40b8d10 2023-03-03T08:09:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.208+0000 7face2ffd700 1 -- 172.21.15.98:0/566392397 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facec005070 con 0x7facf40b8d10 2023-03-03T08:09:06.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.208+0000 7face2ffd700 1 -- 172.21.15.98:0/566392397 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec005400 con 0x7facf40b8d10 2023-03-03T08:09:06.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.208+0000 7facf89ff700 1 -- 172.21.15.98:0/566392397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf40d0e80 con 0x7facf40b8d10 2023-03-03T08:09:06.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.208+0000 7facf89ff700 1 -- 172.21.15.98:0/566392397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf40cdd00 con 0x7facf40b8d10 2023-03-03T08:09:06.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.209+0000 7facf89ff700 1 -- 172.21.15.98:0/566392397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf40cdfe0 con 0x7facf40b8d10 2023-03-03T08:09:06.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.211+0000 7face2ffd700 1 -- 172.21.15.98:0/566392397 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86535+0+0 (secure 0 0 0) 0x7facec010070 con 0x7facf40b8d10 2023-03-03T08:09:06.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.212+0000 7face2ffd700 1 --2- 172.21.15.98:0/566392397 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7facd40683c0 0x7facd406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.213+0000 7face2ffd700 1 -- 172.21.15.98:0/566392397 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7facec085a90 con 0x7facf40b8d10 2023-03-03T08:09:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.213+0000 7facf259c700 1 --2- 172.21.15.98:0/566392397 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7facd40683c0 0x7facd406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:06.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.214+0000 7facf259c700 1 --2- 172.21.15.98:0/566392397 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7facd40683c0 0x7facd406a870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7face4007fd0 tx=0x7face4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:06.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.216+0000 7face2ffd700 1 -- 172.21.15.98:0/566392397 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7facec052050 con 0x7facf40b8d10 2023-03-03T08:09:06.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.406+0000 7facf89ff700 1 -- 172.21.15.98:0/566392397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7facf404e6e0 con 0x7facf40b8d10 2023-03-03T08:09:06.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.406+0000 7face2ffd700 1 -- 172.21.15.98:0/566392397 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7facec055670 con 0x7facf40b8d10 2023-03-03T08:09:06.408 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-03-03T08:09:06.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.409+0000 7facf89ff700 1 -- 172.21.15.98:0/566392397 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7facd40683c0 msgr2=0x7facd406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:06.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.409+0000 7facf89ff700 1 --2- 172.21.15.98:0/566392397 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7facd40683c0 0x7facd406a870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7face4007fd0 tx=0x7face4009040 comp rx=0 tx=0).stop 2023-03-03T08:09:06.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.409+0000 7facf89ff700 1 -- 172.21.15.98:0/566392397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facf40b8d10 msgr2=0x7facf40ca250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:06.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.409+0000 7facf89ff700 1 --2- 172.21.15.98:0/566392397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facf40b8d10 0x7facf40ca250 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7facec00c760 tx=0x7facec00ca70 comp rx=0 tx=0).stop 2023-03-03T08:09:06.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.409+0000 7facf89ff700 1 -- 172.21.15.98:0/566392397 shutdown_connections 2023-03-03T08:09:06.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.409+0000 7facf89ff700 1 --2- 172.21.15.98:0/566392397 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7facd40683c0 0x7facd406a870 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:06.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.409+0000 7facf89ff700 1 --2- 172.21.15.98:0/566392397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facf40b8d10 0x7facf40ca250 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:06.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.409+0000 7facf89ff700 1 --2- 172.21.15.98:0/566392397 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7facf40b26d0 0x7facf415faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:06.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.409+0000 7facf89ff700 1 --2- 172.21.15.98:0/566392397 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7facf40b5140 0x7facf4159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:06.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.409+0000 7facf89ff700 1 -- 172.21.15.98:0/566392397 >> 172.21.15.98:0/566392397 conn(0x7facf4073e70 msgr2=0x7facf40d12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:06.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.409+0000 7facf89ff700 1 -- 172.21.15.98:0/566392397 shutdown_connections 2023-03-03T08:09:06.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.409+0000 7facf89ff700 1 -- 172.21.15.98:0/566392397 wait complete. 2023-03-03T08:09:06.437 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:09:06.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:09:06.489 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:09:06.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.690+0000 7f2ff6bff700 1 Processor -- start 2023-03-03T08:09:06.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.690+0000 7f2ff6bff700 1 -- start start 2023-03-03T08:09:06.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.691+0000 7f2ff6bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ff015ca70 0x7f2ff00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:06.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.691+0000 7f2ff6bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff00cf980 con 0x7f2ff00d0270 2023-03-03T08:09:06.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.691+0000 7f2ff6bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff00d1800 con 0x7f2ff006b4b0 2023-03-03T08:09:06.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.691+0000 7f2ff6bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff00cd5b0 con 0x7f2ff015ca70 2023-03-03T08:09:06.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.692+0000 7f2ff519c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2ff00d0270 0x7f2ff00d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38692/0 (socket says 172.21.15.98:38692) 2023-03-03T08:09:06.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.692+0000 7f2ff519c700 1 -- 172.21.15.98:0/4273841106 learned_addr learned my addr 172.21.15.98:0/4273841106 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:09:06.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.692+0000 7f2ff499b700 1 --2- 172.21.15.98:0/4273841106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ff015ca70 0x7f2ff00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:06.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.692+0000 7f2fef7fe700 1 -- 172.21.15.98:0/4273841106 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266782848 0 0) 0x7f2ff00cf980 con 0x7f2ff00d0270 2023-03-03T08:09:06.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.692+0000 7f2fef7fe700 1 -- 172.21.15.98:0/4273841106 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fe00035b0 con 0x7f2ff00d0270 2023-03-03T08:09:06.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.692+0000 7f2fef7fe700 1 -- 172.21.15.98:0/4273841106 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307042834 0 0) 0x7f2ff00d1800 con 0x7f2ff006b4b0 2023-03-03T08:09:06.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.692+0000 7f2fef7fe700 1 -- 172.21.15.98:0/4273841106 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff00cf980 con 0x7f2ff006b4b0 2023-03-03T08:09:06.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.692+0000 7f2ff499b700 1 -- 172.21.15.98:0/4273841106 >> v1:172.21.15.100:6789/0 conn(0x7f2ff006b4b0 legacy=0x7f2ff00cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:06.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.692+0000 7f2ff499b700 1 -- 172.21.15.98:0/4273841106 >> v1:172.21.15.98:6789/0 conn(0x7f2ff00d0270 legacy=0x7f2ff00d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:06.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.692+0000 7f2ff499b700 1 -- 172.21.15.98:0/4273841106 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff00cd900 con 0x7f2ff015ca70 2023-03-03T08:09:06.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.693+0000 7f2ff499b700 1 --2- 172.21.15.98:0/4273841106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ff015ca70 0x7f2ff00cf440 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4009870 tx=0x7f2fe4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3acbc855466c68d3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:06.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.693+0000 7f2fef7fe700 1 -- 172.21.15.98:0/4273841106 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4011830 con 0x7f2ff015ca70 2023-03-03T08:09:06.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.693+0000 7f2fef7fe700 1 -- 172.21.15.98:0/4273841106 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe4011990 con 0x7f2ff015ca70 2023-03-03T08:09:06.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.693+0000 7f2fef7fe700 1 -- 172.21.15.98:0/4273841106 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4011d20 con 0x7f2ff015ca70 2023-03-03T08:09:06.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.694+0000 7f2ff6bff700 1 -- 172.21.15.98:0/4273841106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ff015ca70 msgr2=0x7f2ff00cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:06.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.694+0000 7f2ff6bff700 1 --2- 172.21.15.98:0/4273841106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ff015ca70 0x7f2ff00cf440 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4009870 tx=0x7f2fe4009b80 comp rx=0 tx=0).stop 2023-03-03T08:09:06.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.694+0000 7f2ff6bff700 1 -- 172.21.15.98:0/4273841106 shutdown_connections 2023-03-03T08:09:06.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.694+0000 7f2ff6bff700 1 --2- 172.21.15.98:0/4273841106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ff015ca70 0x7f2ff00cf440 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:06.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.694+0000 7f2ff6bff700 1 -- 172.21.15.98:0/4273841106 >> 172.21.15.98:0/4273841106 conn(0x7f2ff0073e70 msgr2=0x7f2ff006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:06.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.694+0000 7f2ff6bff700 1 -- 172.21.15.98:0/4273841106 shutdown_connections 2023-03-03T08:09:06.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.694+0000 7f2ff6bff700 1 -- 172.21.15.98:0/4273841106 wait complete. 2023-03-03T08:09:06.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.696+0000 7f2ff6bff700 1 Processor -- start 2023-03-03T08:09:06.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.696+0000 7f2ff6bff700 1 -- start start 2023-03-03T08:09:06.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.697+0000 7f2ff6bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ff006b4b0 0x7f2ff015ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:06.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.697+0000 7f2ff6bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ff00d0270 0x7f2ff014c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:06.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.697+0000 7f2ff6bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff015ca70 0x7f2ff01f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:06.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.697+0000 7f2ff6bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff015df10 con 0x7f2ff015ca70 2023-03-03T08:09:06.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.697+0000 7f2ff6bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff00ca630 con 0x7f2ff006b4b0 2023-03-03T08:09:06.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.697+0000 7f2ff6bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff015e190 con 0x7f2ff00d0270 2023-03-03T08:09:06.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.698+0000 7f2ff519c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff015ca70 0x7f2ff01f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:06.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.698+0000 7f2feffff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ff00d0270 0x7f2ff014c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:06.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.698+0000 7f2ff499b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ff006b4b0 0x7f2ff015ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:06.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.698+0000 7f2ff519c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff015ca70 0x7f2ff01f58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44984/0 (socket says 172.21.15.98:44984) 2023-03-03T08:09:06.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.698+0000 7f2ff519c700 1 -- 172.21.15.98:0/1380168823 learned_addr learned my addr 172.21.15.98:0/1380168823 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:06.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.698+0000 7f2ff519c700 1 -- 172.21.15.98:0/1380168823 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ff00d0270 msgr2=0x7f2ff014c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:06.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.698+0000 7f2ff519c700 1 --2- 172.21.15.98:0/1380168823 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ff00d0270 0x7f2ff014c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:06.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.698+0000 7f2ff519c700 1 -- 172.21.15.98:0/1380168823 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ff006b4b0 msgr2=0x7f2ff015ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.698+0000 7f2ff519c700 1 --2- 172.21.15.98:0/1380168823 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ff006b4b0 0x7f2ff015ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.698+0000 7f2ff519c700 1 -- 172.21.15.98:0/1380168823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe40094a0 con 0x7f2ff015ca70 2023-03-03T08:09:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.698+0000 7f2ff519c700 1 --2- 172.21.15.98:0/1380168823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff015ca70 0x7f2ff01f58b0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800cb40 tx=0x7f2fe800ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.699+0000 7f2fed7fa700 1 -- 172.21.15.98:0/1380168823 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe800a9e0 con 0x7f2ff015ca70 2023-03-03T08:09:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.699+0000 7f2fed7fa700 1 -- 172.21.15.98:0/1380168823 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe800ab40 con 0x7f2ff015ca70 2023-03-03T08:09:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.699+0000 7f2fed7fa700 1 -- 172.21.15.98:0/1380168823 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8012400 con 0x7f2ff015ca70 2023-03-03T08:09:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.699+0000 7f2ff6bff700 1 -- 172.21.15.98:0/1380168823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff00d1aa0 con 0x7f2ff015ca70 2023-03-03T08:09:06.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.699+0000 7f2ff6bff700 1 -- 172.21.15.98:0/1380168823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff00cdbe0 con 0x7f2ff015ca70 2023-03-03T08:09:06.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.700+0000 7f2ff6bff700 1 -- 172.21.15.98:0/1380168823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff015ce40 con 0x7f2ff015ca70 2023-03-03T08:09:06.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.702+0000 7f2fed7fa700 1 -- 172.21.15.98:0/1380168823 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f2fe8019030 con 0x7f2ff015ca70 2023-03-03T08:09:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.703+0000 7f2fed7fa700 1 --2- 172.21.15.98:0/1380168823 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2fd40683c0 0x7f2fd406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.703+0000 7f2fed7fa700 1 -- 172.21.15.98:0/1380168823 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f2fe8085980 con 0x7f2ff015ca70 2023-03-03T08:09:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.703+0000 7f2ff499b700 1 --2- 172.21.15.98:0/1380168823 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2fd40683c0 0x7f2fd406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.704+0000 7f2ff499b700 1 --2- 172.21.15.98:0/1380168823 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2fd40683c0 0x7f2fd406a870 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40097c0 tx=0x7f2fe401c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:06.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.709+0000 7f2fed7fa700 1 -- 172.21.15.98:0/1380168823 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2fe804b5e0 con 0x7f2ff015ca70 2023-03-03T08:09:06.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.901+0000 7f2ff6bff700 1 -- 172.21.15.98:0/1380168823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2ff004e6e0 con 0x7f2ff015ca70 2023-03-03T08:09:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.901+0000 7f2fed7fa700 1 -- 172.21.15.98:0/1380168823 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v114) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2fe8017020 con 0x7f2ff015ca70 2023-03-03T08:09:06.903 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-03-03T08:09:06.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.904+0000 7f2ff6bff700 1 -- 172.21.15.98:0/1380168823 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2fd40683c0 msgr2=0x7f2fd406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:06.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.904+0000 7f2ff6bff700 1 --2- 172.21.15.98:0/1380168823 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2fd40683c0 0x7f2fd406a870 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40097c0 tx=0x7f2fe401c040 comp rx=0 tx=0).stop 2023-03-03T08:09:06.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.904+0000 7f2ff6bff700 1 -- 172.21.15.98:0/1380168823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff015ca70 msgr2=0x7f2ff01f58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:06.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.904+0000 7f2ff6bff700 1 --2- 172.21.15.98:0/1380168823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff015ca70 0x7f2ff01f58b0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800cb40 tx=0x7f2fe800ce50 comp rx=0 tx=0).stop 2023-03-03T08:09:06.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.904+0000 7f2ff6bff700 1 -- 172.21.15.98:0/1380168823 shutdown_connections 2023-03-03T08:09:06.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.904+0000 7f2ff6bff700 1 --2- 172.21.15.98:0/1380168823 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f2fd40683c0 0x7f2fd406a870 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:06.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.904+0000 7f2ff6bff700 1 --2- 172.21.15.98:0/1380168823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff015ca70 0x7f2ff01f58b0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:06.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.904+0000 7f2ff6bff700 1 --2- 172.21.15.98:0/1380168823 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2ff006b4b0 0x7f2ff015ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:06.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.904+0000 7f2ff6bff700 1 --2- 172.21.15.98:0/1380168823 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ff00d0270 0x7f2ff014c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:06.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.904+0000 7f2ff6bff700 1 -- 172.21.15.98:0/1380168823 >> 172.21.15.98:0/1380168823 conn(0x7f2ff0073e70 msgr2=0x7f2ff00b5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:06.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.904+0000 7f2ff6bff700 1 -- 172.21.15.98:0/1380168823 shutdown_connections 2023-03-03T08:09:06.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:06.904+0000 7f2ff6bff700 1 -- 172.21.15.98:0/1380168823 wait complete. 2023-03-03T08:09:06.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T08:09:07.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.178+0000 7fa3e9bff700 1 Processor -- start 2023-03-03T08:09:07.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.178+0000 7fa3e9bff700 1 -- start start 2023-03-03T08:09:07.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.179+0000 7fa3e9bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa3e40b8760 0x7fa3e40cd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:07.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.179+0000 7fa3e9bff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa3e40b8b30 0x7fa3e40b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:07.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.179+0000 7fa3e9bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e40b55d0 con 0x7fa3e40b5170 2023-03-03T08:09:07.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.179+0000 7fa3e9bff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e40d07b0 con 0x7fa3e40b8b30 2023-03-03T08:09:07.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.179+0000 7fa3e9bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e40d1290 con 0x7fa3e40b8760 2023-03-03T08:09:07.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.180+0000 7fa3e2ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa3e40b8b30 0x7fa3e40b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:07.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.180+0000 7fa3e2ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa3e40b8b30 0x7fa3e40b4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45000/0 (socket says 172.21.15.98:45000) 2023-03-03T08:09:07.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.180+0000 7fa3e2ffd700 1 -- 172.21.15.98:0/3798450841 learned_addr learned my addr 172.21.15.98:0/3798450841 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:07.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.180+0000 7fa3e37fe700 1 --2- 172.21.15.98:0/3798450841 >> v2:172.21.15.100:3300/0 conn(0x7fa3e40b8760 0x7fa3e40cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:07.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.180+0000 7fa3e2ffd700 1 -- 172.21.15.98:0/3798450841 >> v2:172.21.15.100:3300/0 conn(0x7fa3e40b8760 msgr2=0x7fa3e40cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:09:07.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.180+0000 7fa3e2ffd700 1 --2- 172.21.15.98:0/3798450841 >> v2:172.21.15.100:3300/0 conn(0x7fa3e40b8760 0x7fa3e40cd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.180+0000 7fa3e2ffd700 1 -- 172.21.15.98:0/3798450841 >> v1:172.21.15.100:6789/0 conn(0x7fa3e40b5170 legacy=0x7fa3e41605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:07.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.180+0000 7fa3e2ffd700 1 -- 172.21.15.98:0/3798450841 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e40b5a10 con 0x7fa3e40b8b30 2023-03-03T08:09:07.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.181+0000 7fa3e2ffd700 1 --2- 172.21.15.98:0/3798450841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3e40b8b30 0x7fa3e40b4c30 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40b5d40 tx=0x7fa3d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e1096cd0e2fbc5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:07.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.181+0000 7fa3e27fc700 1 -- 172.21.15.98:0/3798450841 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d800b870 con 0x7fa3e40b8b30 2023-03-03T08:09:07.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.181+0000 7fa3e27fc700 1 -- 172.21.15.98:0/3798450841 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d800b9d0 con 0x7fa3e40b8b30 2023-03-03T08:09:07.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.181+0000 7fa3e27fc700 1 -- 172.21.15.98:0/3798450841 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d800bd60 con 0x7fa3e40b8b30 2023-03-03T08:09:07.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.182+0000 7fa3e9bff700 1 -- 172.21.15.98:0/3798450841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3e40b8b30 msgr2=0x7fa3e40b4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:07.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.182+0000 7fa3e9bff700 1 --2- 172.21.15.98:0/3798450841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3e40b8b30 0x7fa3e40b4c30 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40b5d40 tx=0x7fa3d8009bd0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.182+0000 7fa3e9bff700 1 -- 172.21.15.98:0/3798450841 shutdown_connections 2023-03-03T08:09:07.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.182+0000 7fa3e9bff700 1 --2- 172.21.15.98:0/3798450841 >> v2:172.21.15.100:3300/0 conn(0x7fa3e40b8760 0x7fa3e40cd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.182+0000 7fa3e9bff700 1 --2- 172.21.15.98:0/3798450841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3e40b8b30 0x7fa3e40b4c30 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.182+0000 7fa3e9bff700 1 -- 172.21.15.98:0/3798450841 >> 172.21.15.98:0/3798450841 conn(0x7fa3e4073e70 msgr2=0x7fa3e406adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:07.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.182+0000 7fa3e9bff700 1 -- 172.21.15.98:0/3798450841 shutdown_connections 2023-03-03T08:09:07.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.182+0000 7fa3e9bff700 1 -- 172.21.15.98:0/3798450841 wait complete. 2023-03-03T08:09:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.184+0000 7fa3e9bff700 1 Processor -- start 2023-03-03T08:09:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.184+0000 7fa3e9bff700 1 -- start start 2023-03-03T08:09:07.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.185+0000 7fa3e9bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3e40b5170 0x7fa3e415fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:07.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.185+0000 7fa3e9bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3e40b8760 0x7fa3e40a9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:07.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.185+0000 7fa3e9bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3e40b8b30 0x7fa3e414e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:07.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.185+0000 7fa3e9bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e415ff40 con 0x7fa3e40b8760 2023-03-03T08:09:07.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.185+0000 7fa3e9bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e41f31e0 con 0x7fa3e40b5170 2023-03-03T08:09:07.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.185+0000 7fa3e9bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e41f2940 con 0x7fa3e40b8b30 2023-03-03T08:09:07.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.185+0000 7fa3e2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3e40b8760 0x7fa3e40a9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.185+0000 7fa3e2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3e40b8760 0x7fa3e40a9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45008/0 (socket says 172.21.15.98:45008) 2023-03-03T08:09:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.185+0000 7fa3e2ffd700 1 -- 172.21.15.98:0/1795851666 learned_addr learned my addr 172.21.15.98:0/1795851666 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.185+0000 7fa3e37fe700 1 --2- 172.21.15.98:0/1795851666 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3e40b5170 0x7fa3e415fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.185+0000 7fa3e3fff700 1 --2- 172.21.15.98:0/1795851666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3e40b8b30 0x7fa3e414e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:07.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.186+0000 7fa3e2ffd700 1 -- 172.21.15.98:0/1795851666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3e40b8b30 msgr2=0x7fa3e414e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:07.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.186+0000 7fa3e2ffd700 1 --2- 172.21.15.98:0/1795851666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3e40b8b30 0x7fa3e414e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.186+0000 7fa3e2ffd700 1 -- 172.21.15.98:0/1795851666 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3e40b5170 msgr2=0x7fa3e415fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:07.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.186+0000 7fa3e2ffd700 1 --2- 172.21.15.98:0/1795851666 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3e40b5170 0x7fa3e415fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.186+0000 7fa3e2ffd700 1 -- 172.21.15.98:0/1795851666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d8009550 con 0x7fa3e40b8760 2023-03-03T08:09:07.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.186+0000 7fa3e2ffd700 1 --2- 172.21.15.98:0/1795851666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3e40b8760 0x7fa3e40a9b60 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8000c00 tx=0x7fa3d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:07.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.186+0000 7fa3cbfff700 1 -- 172.21.15.98:0/1795851666 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d8004540 con 0x7fa3e40b8760 2023-03-03T08:09:07.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.186+0000 7fa3cbfff700 1 -- 172.21.15.98:0/1795851666 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d80036a0 con 0x7fa3e40b8760 2023-03-03T08:09:07.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.186+0000 7fa3cbfff700 1 -- 172.21.15.98:0/1795851666 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d8003a50 con 0x7fa3e40b8760 2023-03-03T08:09:07.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.186+0000 7fa3e9bff700 1 -- 172.21.15.98:0/1795851666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e40d01f0 con 0x7fa3e40b8760 2023-03-03T08:09:07.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.187+0000 7fa3e9bff700 1 -- 172.21.15.98:0/1795851666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e40cff10 con 0x7fa3e40b8760 2023-03-03T08:09:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.188+0000 7fa3e9bff700 1 -- 172.21.15.98:0/1795851666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e40d04d0 con 0x7fa3e40b8760 2023-03-03T08:09:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.189+0000 7fa3cbfff700 1 -- 172.21.15.98:0/1795851666 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fa3d800f050 con 0x7fa3e40b8760 2023-03-03T08:09:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.190+0000 7fa3cbfff700 1 --2- 172.21.15.98:0/1795851666 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa3d00683c0 0x7fa3d006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.190+0000 7fa3cbfff700 1 -- 172.21.15.98:0/1795851666 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7fa3d8087860 con 0x7fa3e40b8760 2023-03-03T08:09:07.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.191+0000 7fa3e37fe700 1 --2- 172.21.15.98:0/1795851666 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa3d00683c0 0x7fa3d006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:07.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.191+0000 7fa3e37fe700 1 --2- 172.21.15.98:0/1795851666 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa3d00683c0 0x7fa3d006a870 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40b4800 tx=0x7fa3d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:07.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.194+0000 7fa3cbfff700 1 -- 172.21.15.98:0/1795851666 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa3d8053e20 con 0x7fa3e40b8760 2023-03-03T08:09:07.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.389+0000 7fa3e9bff700 1 -- 172.21.15.98:0/1795851666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa3e404e6e0 con 0x7fa3e40b8760 2023-03-03T08:09:07.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.390+0000 7fa3cbfff700 1 -- 172.21.15.98:0/1795851666 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa3d8057440 con 0x7fa3e40b8760 2023-03-03T08:09:07.391 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-03-03T08:09:07.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.392+0000 7fa3e9bff700 1 -- 172.21.15.98:0/1795851666 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa3d00683c0 msgr2=0x7fa3d006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:07.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.392+0000 7fa3e9bff700 1 --2- 172.21.15.98:0/1795851666 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa3d00683c0 0x7fa3d006a870 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40b4800 tx=0x7fa3d4008040 comp rx=0 tx=0).stop 2023-03-03T08:09:07.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.392+0000 7fa3e9bff700 1 -- 172.21.15.98:0/1795851666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3e40b8760 msgr2=0x7fa3e40a9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:07.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.392+0000 7fa3e9bff700 1 --2- 172.21.15.98:0/1795851666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3e40b8760 0x7fa3e40a9b60 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8000c00 tx=0x7fa3d8004750 comp rx=0 tx=0).stop 2023-03-03T08:09:07.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.392+0000 7fa3e9bff700 1 -- 172.21.15.98:0/1795851666 shutdown_connections 2023-03-03T08:09:07.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.392+0000 7fa3e9bff700 1 --2- 172.21.15.98:0/1795851666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3e40b8b30 0x7fa3e414e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.392+0000 7fa3e9bff700 1 --2- 172.21.15.98:0/1795851666 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3e40b5170 0x7fa3e415fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.392+0000 7fa3e9bff700 1 --2- 172.21.15.98:0/1795851666 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fa3d00683c0 0x7fa3d006a870 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.392+0000 7fa3e9bff700 1 --2- 172.21.15.98:0/1795851666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3e40b8760 0x7fa3e40a9b60 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.392+0000 7fa3e9bff700 1 -- 172.21.15.98:0/1795851666 >> 172.21.15.98:0/1795851666 conn(0x7fa3e4073e70 msgr2=0x7fa3e40b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:07.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.392+0000 7fa3e9bff700 1 -- 172.21.15.98:0/1795851666 shutdown_connections 2023-03-03T08:09:07.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.392+0000 7fa3e9bff700 1 -- 172.21.15.98:0/1795851666 wait complete. 2023-03-03T08:09:07.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T08:09:07.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.666+0000 7f0787bff700 1 Processor -- start 2023-03-03T08:09:07.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.666+0000 7f0787bff700 1 -- start start 2023-03-03T08:09:07.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.667+0000 7f0787bff700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f078015ca70 0x7f07800cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:07.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.667+0000 7f0787bff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07800cf980 con 0x7f07800d0270 2023-03-03T08:09:07.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.668+0000 7f0787bff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07800d1800 con 0x7f078006b4b0 2023-03-03T08:09:07.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.668+0000 7f0787bff700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07800cd5b0 con 0x7f078015ca70 2023-03-03T08:09:07.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.668+0000 7f078619c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f07800d0270 0x7f07800d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38694/0 (socket says 172.21.15.98:38694) 2023-03-03T08:09:07.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.668+0000 7f078619c700 1 -- 172.21.15.98:0/2248246117 learned_addr learned my addr 172.21.15.98:0/2248246117 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-03-03T08:09:07.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.668+0000 7f078599b700 1 --2- 172.21.15.98:0/2248246117 >> v2:172.21.15.100:3300/0 conn(0x7f078015ca70 0x7f07800cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.668+0000 7f0784999700 1 -- 172.21.15.98:0/2248246117 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232173058 0 0) 0x7f07800cf980 con 0x7f07800d0270 2023-03-03T08:09:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.668+0000 7f0784999700 1 -- 172.21.15.98:0/2248246117 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f076c0035b0 con 0x7f07800d0270 2023-03-03T08:09:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.668+0000 7f0784999700 1 -- 172.21.15.98:0/2248246117 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475443410 0 0) 0x7f07800d1800 con 0x7f078006b4b0 2023-03-03T08:09:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.669+0000 7f0784999700 1 -- 172.21.15.98:0/2248246117 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07800cf980 con 0x7f078006b4b0 2023-03-03T08:09:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.669+0000 7f078599b700 1 -- 172.21.15.98:0/2248246117 >> v1:172.21.15.100:6789/0 conn(0x7f078006b4b0 legacy=0x7f07800cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.669+0000 7f078599b700 1 -- 172.21.15.98:0/2248246117 >> v1:172.21.15.98:6789/0 conn(0x7f07800d0270 legacy=0x7f07800d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.669+0000 7f078599b700 1 -- 172.21.15.98:0/2248246117 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07800cd900 con 0x7f078015ca70 2023-03-03T08:09:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.669+0000 7f078599b700 1 --2- 172.21.15.98:0/2248246117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f078015ca70 0x7f07800cf440 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f077c009840 tx=0x7f077c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27417e1c5c7773a8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.669+0000 7f0784999700 1 -- 172.21.15.98:0/2248246117 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c011830 con 0x7f078015ca70 2023-03-03T08:09:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.669+0000 7f0784999700 1 -- 172.21.15.98:0/2248246117 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077c011990 con 0x7f078015ca70 2023-03-03T08:09:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.669+0000 7f0784999700 1 -- 172.21.15.98:0/2248246117 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c011d20 con 0x7f078015ca70 2023-03-03T08:09:07.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.670+0000 7f0787bff700 1 -- 172.21.15.98:0/2248246117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f078015ca70 msgr2=0x7f07800cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:07.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.670+0000 7f0787bff700 1 --2- 172.21.15.98:0/2248246117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f078015ca70 0x7f07800cf440 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f077c009840 tx=0x7f077c009b50 comp rx=0 tx=0).stop 2023-03-03T08:09:07.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.670+0000 7f0787bff700 1 -- 172.21.15.98:0/2248246117 shutdown_connections 2023-03-03T08:09:07.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.670+0000 7f0787bff700 1 --2- 172.21.15.98:0/2248246117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f078015ca70 0x7f07800cf440 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.670+0000 7f0787bff700 1 -- 172.21.15.98:0/2248246117 >> 172.21.15.98:0/2248246117 conn(0x7f0780073e70 msgr2=0x7f078006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:07.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.670+0000 7f0787bff700 1 -- 172.21.15.98:0/2248246117 shutdown_connections 2023-03-03T08:09:07.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.670+0000 7f0787bff700 1 -- 172.21.15.98:0/2248246117 wait complete. 2023-03-03T08:09:07.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.673+0000 7f0787bff700 1 Processor -- start 2023-03-03T08:09:07.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.673+0000 7f0787bff700 1 -- start start 2023-03-03T08:09:07.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.675+0000 7f0787bff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078006b4b0 0x7f0780158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:07.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.675+0000 7f0787bff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07800d0270 0x7f07800ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:07.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.675+0000 7f0787bff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078015ca70 0x7f07800ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:07.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.675+0000 7f0787bff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078015f9a0 con 0x7f078015ca70 2023-03-03T08:09:07.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.675+0000 7f0787bff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07800cada0 con 0x7f07800d0270 2023-03-03T08:09:07.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.675+0000 7f0787bff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07800cb080 con 0x7f078006b4b0 2023-03-03T08:09:07.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.675+0000 7f078619c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078015ca70 0x7f07800ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:07.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.675+0000 7f078619c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078015ca70 0x7f07800ca790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45016/0 (socket says 172.21.15.98:45016) 2023-03-03T08:09:07.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.675+0000 7f078619c700 1 -- 172.21.15.98:0/2738981263 learned_addr learned my addr 172.21.15.98:0/2738981263 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:07.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.675+0000 7f078599b700 1 --2- 172.21.15.98:0/2738981263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078006b4b0 0x7f0780158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:07.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.676+0000 7f078519a700 1 --2- 172.21.15.98:0/2738981263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07800d0270 0x7f07800ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:07.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.676+0000 7f078619c700 1 -- 172.21.15.98:0/2738981263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078006b4b0 msgr2=0x7f0780158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:07.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.676+0000 7f078619c700 1 --2- 172.21.15.98:0/2738981263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078006b4b0 0x7f0780158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.676+0000 7f078619c700 1 -- 172.21.15.98:0/2738981263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07800d0270 msgr2=0x7f07800ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:07.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.676+0000 7f078619c700 1 --2- 172.21.15.98:0/2738981263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07800d0270 0x7f07800ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.676+0000 7f078619c700 1 -- 172.21.15.98:0/2738981263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077c0094a0 con 0x7f078015ca70 2023-03-03T08:09:07.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.676+0000 7f078519a700 1 --2- 172.21.15.98:0/2738981263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07800d0270 0x7f07800ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:09:07.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.676+0000 7f078619c700 1 --2- 172.21.15.98:0/2738981263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078015ca70 0x7f07800ca790 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f077800cbc0 tx=0x7f077800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:07.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.676+0000 7f07767fc700 1 -- 172.21.15.98:0/2738981263 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077800aaf0 con 0x7f078015ca70 2023-03-03T08:09:07.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.676+0000 7f07767fc700 1 -- 172.21.15.98:0/2738981263 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077800ac50 con 0x7f078015ca70 2023-03-03T08:09:07.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.676+0000 7f07767fc700 1 -- 172.21.15.98:0/2738981263 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0778012400 con 0x7f078015ca70 2023-03-03T08:09:07.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.676+0000 7f0787bff700 1 -- 172.21.15.98:0/2738981263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07800d1aa0 con 0x7f078015ca70 2023-03-03T08:09:07.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.676+0000 7f0787bff700 1 -- 172.21.15.98:0/2738981263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07800cdbe0 con 0x7f078015ca70 2023-03-03T08:09:07.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.678+0000 7f0787bff700 1 -- 172.21.15.98:0/2738981263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f078015ce40 con 0x7f078015ca70 2023-03-03T08:09:07.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.679+0000 7f07767fc700 1 -- 172.21.15.98:0/2738981263 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86535+0+0 (secure 0 0 0) 0x7f0778019030 con 0x7f078015ca70 2023-03-03T08:09:07.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.680+0000 7f07767fc700 1 --2- 172.21.15.98:0/2738981263 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f07640683c0 0x7f076406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.680+0000 7f07767fc700 1 -- 172.21.15.98:0/2738981263 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f0778085a90 con 0x7f078015ca70 2023-03-03T08:09:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.681+0000 7f078599b700 1 --2- 172.21.15.98:0/2738981263 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f07640683c0 0x7f076406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.682+0000 7f078599b700 1 --2- 172.21.15.98:0/2738981263 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f07640683c0 0x7f076406a870 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f077c009fd0 tx=0x7f077c01c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:07.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.685+0000 7f07767fc700 1 -- 172.21.15.98:0/2738981263 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f077804b5e0 con 0x7f078015ca70 2023-03-03T08:09:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.875+0000 7f0787bff700 1 -- 172.21.15.98:0/2738981263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f078004e6e0 con 0x7f078015ca70 2023-03-03T08:09:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.876+0000 7f07767fc700 1 -- 172.21.15.98:0/2738981263 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0778017020 con 0x7f078015ca70 2023-03-03T08:09:07.877 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-03-03T08:09:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.878+0000 7f0787bff700 1 -- 172.21.15.98:0/2738981263 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f07640683c0 msgr2=0x7f076406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.878+0000 7f0787bff700 1 --2- 172.21.15.98:0/2738981263 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f07640683c0 0x7f076406a870 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f077c009fd0 tx=0x7f077c01c040 comp rx=0 tx=0).stop 2023-03-03T08:09:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.878+0000 7f0787bff700 1 -- 172.21.15.98:0/2738981263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078015ca70 msgr2=0x7f07800ca790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.878+0000 7f0787bff700 1 --2- 172.21.15.98:0/2738981263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078015ca70 0x7f07800ca790 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f077800cbc0 tx=0x7f077800cf80 comp rx=0 tx=0).stop 2023-03-03T08:09:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.878+0000 7f0787bff700 1 -- 172.21.15.98:0/2738981263 shutdown_connections 2023-03-03T08:09:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.878+0000 7f0787bff700 1 --2- 172.21.15.98:0/2738981263 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7f07640683c0 0x7f076406a870 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.878+0000 7f0787bff700 1 --2- 172.21.15.98:0/2738981263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078015ca70 0x7f07800ca790 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.878+0000 7f0787bff700 1 --2- 172.21.15.98:0/2738981263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f078006b4b0 0x7f0780158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.878+0000 7f0787bff700 1 --2- 172.21.15.98:0/2738981263 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07800d0270 0x7f07800ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:07.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.878+0000 7f0787bff700 1 -- 172.21.15.98:0/2738981263 >> 172.21.15.98:0/2738981263 conn(0x7f0780073e70 msgr2=0x7f07800d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:07.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.879+0000 7f0787bff700 1 -- 172.21.15.98:0/2738981263 shutdown_connections 2023-03-03T08:09:07.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:07.879+0000 7f0787bff700 1 -- 172.21.15.98:0/2738981263 wait complete. 2023-03-03T08:09:07.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-03T08:09:08.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.151+0000 7fe75fdff700 1 Processor -- start 2023-03-03T08:09:08.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.151+0000 7fe75fdff700 1 -- start start 2023-03-03T08:09:08.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.153+0000 7fe75fdff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe75815ca70 0x7fe7580cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:08.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.153+0000 7fe75fdff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe75806b4b0 0x7fe7580cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:08.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.153+0000 7fe75fdff700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7580cf980 con 0x7fe7580d00a0 2023-03-03T08:09:08.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.153+0000 7fe75fdff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7580cd5b0 con 0x7fe75815ca70 2023-03-03T08:09:08.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.153+0000 7fe75fdff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7580d0510 con 0x7fe75806b4b0 2023-03-03T08:09:08.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.153+0000 7fe75db9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe75815ca70 0x7fe7580cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:08.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.153+0000 7fe75db9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe75815ca70 0x7fe7580cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45022/0 (socket says 172.21.15.98:45022) 2023-03-03T08:09:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.153+0000 7fe75db9b700 1 -- 172.21.15.98:0/1702957206 learned_addr learned my addr 172.21.15.98:0/1702957206 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.153+0000 7fe75d39a700 1 --2- 172.21.15.98:0/1702957206 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe75806b4b0 0x7fe7580cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.154+0000 7fe75db9b700 1 -- 172.21.15.98:0/1702957206 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe75806b4b0 msgr2=0x7fe7580cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:09:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.154+0000 7fe75db9b700 1 --2- 172.21.15.98:0/1702957206 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe75806b4b0 0x7fe7580cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.154+0000 7fe75db9b700 1 -- 172.21.15.98:0/1702957206 >> v1:172.21.15.100:6789/0 conn(0x7fe7580d00a0 legacy=0x7fe7580b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.154+0000 7fe75db9b700 1 -- 172.21.15.98:0/1702957206 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7580d2fd0 con 0x7fe75815ca70 2023-03-03T08:09:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.154+0000 7fe75db9b700 1 --2- 172.21.15.98:0/1702957206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75815ca70 0x7fe7580cf440 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fe7540098f0 tx=0x7fe754009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606a261b30307ff2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.154+0000 7fe75cb99700 1 -- 172.21.15.98:0/1702957206 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75400b8a0 con 0x7fe75815ca70 2023-03-03T08:09:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.154+0000 7fe75cb99700 1 -- 172.21.15.98:0/1702957206 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75400ba00 con 0x7fe75815ca70 2023-03-03T08:09:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.154+0000 7fe75cb99700 1 -- 172.21.15.98:0/1702957206 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75400bd90 con 0x7fe75815ca70 2023-03-03T08:09:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.155+0000 7fe75fdff700 1 -- 172.21.15.98:0/1702957206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75815ca70 msgr2=0x7fe7580cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.155+0000 7fe75fdff700 1 --2- 172.21.15.98:0/1702957206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75815ca70 0x7fe7580cf440 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fe7540098f0 tx=0x7fe754009c00 comp rx=0 tx=0).stop 2023-03-03T08:09:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.155+0000 7fe75fdff700 1 -- 172.21.15.98:0/1702957206 shutdown_connections 2023-03-03T08:09:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.155+0000 7fe75fdff700 1 --2- 172.21.15.98:0/1702957206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75815ca70 0x7fe7580cf440 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.155+0000 7fe75fdff700 1 --2- 172.21.15.98:0/1702957206 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe75806b4b0 0x7fe7580cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.155+0000 7fe75fdff700 1 -- 172.21.15.98:0/1702957206 >> 172.21.15.98:0/1702957206 conn(0x7fe758073e70 msgr2=0x7fe75806ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.155+0000 7fe75fdff700 1 -- 172.21.15.98:0/1702957206 shutdown_connections 2023-03-03T08:09:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.155+0000 7fe75fdff700 1 -- 172.21.15.98:0/1702957206 wait complete. 2023-03-03T08:09:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.157+0000 7fe75fdff700 1 Processor -- start 2023-03-03T08:09:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.157+0000 7fe75fdff700 1 -- start start 2023-03-03T08:09:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.158+0000 7fe75fdff700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe75806b4b0 0x7fe7581e5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.158+0000 7fe75fdff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe7580d00a0 0x7fe7580cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.159+0000 7fe75fdff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75815ca70 0x7fe75814b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.159+0000 7fe75fdff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7580cba80 con 0x7fe75815ca70 2023-03-03T08:09:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.159+0000 7fe75fdff700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7580cc450 con 0x7fe7580d00a0 2023-03-03T08:09:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.159+0000 7fe75fdff700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7580cb150 con 0x7fe75806b4b0 2023-03-03T08:09:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.159+0000 7fe75e39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75815ca70 0x7fe75814b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.159+0000 7fe75e39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75815ca70 0x7fe75814b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45024/0 (socket says 172.21.15.98:45024) 2023-03-03T08:09:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.159+0000 7fe75e39c700 1 -- 172.21.15.98:0/3271767956 learned_addr learned my addr 172.21.15.98:0/3271767956 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-03-03T08:09:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.159+0000 7fe75db9b700 1 --2- 172.21.15.98:0/3271767956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe75806b4b0 0x7fe7581e5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.159+0000 7fe75d39a700 1 --2- 172.21.15.98:0/3271767956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe7580d00a0 0x7fe7580cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.159+0000 7fe75e39c700 1 -- 172.21.15.98:0/3271767956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe75806b4b0 msgr2=0x7fe7581e5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.159+0000 7fe75e39c700 1 --2- 172.21.15.98:0/3271767956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe75806b4b0 0x7fe7581e5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.159+0000 7fe75e39c700 1 -- 172.21.15.98:0/3271767956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe7580d00a0 msgr2=0x7fe7580cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.159+0000 7fe75e39c700 1 --2- 172.21.15.98:0/3271767956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe7580d00a0 0x7fe7580cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.159+0000 7fe75e39c700 1 -- 172.21.15.98:0/3271767956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe754009550 con 0x7fe75815ca70 2023-03-03T08:09:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.160+0000 7fe75e39c700 1 --2- 172.21.15.98:0/3271767956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75815ca70 0x7fe75814b400 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fe75000cbc0 tx=0x7fe75000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.160+0000 7fe74e7fc700 1 -- 172.21.15.98:0/3271767956 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750003c00 con 0x7fe75815ca70 2023-03-03T08:09:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.160+0000 7fe74e7fc700 1 -- 172.21.15.98:0/3271767956 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe750003d60 con 0x7fe75815ca70 2023-03-03T08:09:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.160+0000 7fe74e7fc700 1 -- 172.21.15.98:0/3271767956 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75000aae0 con 0x7fe75815ca70 2023-03-03T08:09:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.160+0000 7fe75fdff700 1 -- 172.21.15.98:0/3271767956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7580cd900 con 0x7fe75815ca70 2023-03-03T08:09:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.160+0000 7fe75fdff700 1 -- 172.21.15.98:0/3271767956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7580d1800 con 0x7fe75815ca70 2023-03-03T08:09:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.161+0000 7fe75fdff700 1 -- 172.21.15.98:0/3271767956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7580cdbe0 con 0x7fe75815ca70 2023-03-03T08:09:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.163+0000 7fe74e7fc700 1 -- 172.21.15.98:0/3271767956 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86535+0+0 (secure 0 0 0) 0x7fe750010050 con 0x7fe75815ca70 2023-03-03T08:09:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.164+0000 7fe74e7fc700 1 --2- 172.21.15.98:0/3271767956 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe73c0683c0 0x7fe73c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:09:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.165+0000 7fe74e7fc700 1 -- 172.21.15.98:0/3271767956 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6320+0+0 (secure 0 0 0) 0x7fe750085e00 con 0x7fe75815ca70 2023-03-03T08:09:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.165+0000 7fe75db9b700 1 --2- 172.21.15.98:0/3271767956 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe73c0683c0 0x7fe73c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:09:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.166+0000 7fe75db9b700 1 --2- 172.21.15.98:0/3271767956 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe73c0683c0 0x7fe73c06a870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe754009c40 tx=0x7fe754008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:09:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.169+0000 7fe74e7fc700 1 -- 172.21.15.98:0/3271767956 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe7500523c0 con 0x7fe75815ca70 2023-03-03T08:09:08.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.376+0000 7fe75fdff700 1 -- 172.21.15.98:0/3271767956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe75808b040 con 0x7fe75815ca70 2023-03-03T08:09:08.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.376+0000 7fe74e7fc700 1 -- 172.21.15.98:0/3271767956 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v114) v1 ==== 80+0+368 (secure 0 0 0) 0x7fe7500559e0 con 0x7fe75815ca70 2023-03-03T08:09:08.378 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6832/2374561485 2023-03-04T08:04:52.583414+0000 2023-03-03T08:09:08.378 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6835/2695503624 2023-03-04T08:04:32.578044+0000 2023-03-03T08:09:08.378 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6833/2695503624 2023-03-04T08:04:32.578044+0000 2023-03-03T08:09:08.378 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6834/2374561485 2023-03-04T08:04:52.583414+0000 2023-03-03T08:09:08.378 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.100:6835/2660525287 2023-03-04T08:04:12.571647+0000 2023-03-03T08:09:08.378 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.100:6833/2660525287 2023-03-04T08:04:12.571647+0000 2023-03-03T08:09:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.379+0000 7fe75fdff700 1 -- 172.21.15.98:0/3271767956 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe73c0683c0 msgr2=0x7fe73c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.379+0000 7fe75fdff700 1 --2- 172.21.15.98:0/3271767956 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe73c0683c0 0x7fe73c06a870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe754009c40 tx=0x7fe754008030 comp rx=0 tx=0).stop 2023-03-03T08:09:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.379+0000 7fe75fdff700 1 -- 172.21.15.98:0/3271767956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75815ca70 msgr2=0x7fe75814b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:09:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.379+0000 7fe75fdff700 1 --2- 172.21.15.98:0/3271767956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75815ca70 0x7fe75814b400 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fe75000cbc0 tx=0x7fe75000ced0 comp rx=0 tx=0).stop 2023-03-03T08:09:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.379+0000 7fe75fdff700 1 -- 172.21.15.98:0/3271767956 shutdown_connections 2023-03-03T08:09:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.379+0000 7fe75fdff700 1 --2- 172.21.15.98:0/3271767956 >> [v2:172.21.15.100:6836/108673,v1:172.21.15.100:6837/108673] conn(0x7fe73c0683c0 0x7fe73c06a870 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.379+0000 7fe75fdff700 1 --2- 172.21.15.98:0/3271767956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe75815ca70 0x7fe75814b400 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.379+0000 7fe75fdff700 1 --2- 172.21.15.98:0/3271767956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe75806b4b0 0x7fe7581e5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.379+0000 7fe75fdff700 1 --2- 172.21.15.98:0/3271767956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe7580d00a0 0x7fe7580cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:09:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.379+0000 7fe75fdff700 1 -- 172.21.15.98:0/3271767956 >> 172.21.15.98:0/3271767956 conn(0x7fe758073e70 msgr2=0x7fe7580d3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:09:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.379+0000 7fe75fdff700 1 -- 172.21.15.98:0/3271767956 shutdown_connections 2023-03-03T08:09:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-03-03T08:09:08.379+0000 7fe75fdff700 1 -- 172.21.15.98:0/3271767956 wait complete. 2023-03-03T08:09:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:listed 6 entries 2023-03-03T08:09:08.407 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2023-03-03T08:09:08.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:09:08.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-03-03T08:09:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:fusermount: failed to unmount /home/ubuntu/cephtest/mnt.0: Device or resource busy 2023-03-03T08:09:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:09:08.508 INFO:tasks.cephfs.fuse_mount:Failed to unmount ceph-fuse on ubuntu@smithi098.front.sepia.ceph.com, aborting... 2023-03-03T08:09:08.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:09:08.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo PATH=/usr/sbin:$PATH lsof ; ps auxf 2023-03-03T08:09:36.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:09:36.718 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:10:06.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:10:06.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:10:37.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:10:37.034 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:10:58.046 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2023-03-03T08:10:58.048 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2023-03-03T08:11:07.176 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:11:07.183 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:11:37.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:11:37.396 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:12:07.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:12:07.704 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:12:37.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:12:37.848 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:13:08.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:13:08.046 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:13:38.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:13:38.251 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:14:08.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:14:08.467 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:14:08.515 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2023-03-03T08:14:08.515 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2023-03-03T08:14:38.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:14:38.615 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:15:08.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:15:08.841 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:15:38.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:15:38.987 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:16:09.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:16:09.184 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:16:39.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:16:39.555 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:17:09.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:17:09.769 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:17:39.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:17:39.975 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:18:10.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:18:10.138 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:18:40.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:18:40.354 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:19:10.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:19:10.493 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:19:40.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:19:40.691 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:20:10.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:20:10.910 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:20:41.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:20:41.144 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:21:11.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:21:11.417 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:21:41.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:21:41.557 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:22:11.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:22:11.778 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:22:41.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:22:41.993 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:23:12.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:23:12.165 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:23:42.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:23:42.311 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:24:12.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:24:12.452 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:24:42.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:24:42.600 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:25:12.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:25:12.839 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:25:43.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:25:43.013 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:26:13.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:26:13.170 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:26:43.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:26:43.343 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:27:13.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:27:13.555 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:27:43.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:27:43.709 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:28:13.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:28:13.910 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:28:44.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:28:44.108 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:29:14.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:29:14.256 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:29:44.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:29:44.396 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:30:14.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:30:14.594 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:30:44.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:30:44.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:31:15.081 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:31:15.088 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:31:45.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:31:45.268 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:32:15.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:32:15.507 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:32:45.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:32:45.713 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:33:15.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:33:15.861 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:33:46.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:33:46.085 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:34:16.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:34:16.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:34:46.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:34:46.437 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:35:16.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:35:16.676 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:35:46.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:35:46.873 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:36:17.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:36:17.103 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:36:47.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:36:47.279 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:37:17.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:37:17.420 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:37:47.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:37:47.575 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:38:17.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:38:17.747 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:38:47.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:38:47.895 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:39:18.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:39:18.109 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:39:48.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:39:48.262 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:40:18.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:40:18.403 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:40:48.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:40:48.634 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:41:18.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:41:18.873 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:41:49.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:41:49.095 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:19.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:19.240 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:49.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:49.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:43:19.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:43:19.516 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:43:49.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:43:49.742 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:19.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:19.885 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:50.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:50.116 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:45:20.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:45:20.446 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:45:50.662 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:45:50.668 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:46:20.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:46:20.808 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:46:50.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:46:50.948 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:21.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:21.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:51.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:51.339 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:21.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:21.474 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:51.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:51.709 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:21.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:21.853 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:51.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:52.000 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:22.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:22.255 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:52.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:52.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:51:22.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:51:22.608 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:51:52.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:51:52.831 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:23.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:23.053 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:53.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:53.198 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:53:23.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:53:23.346 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:53:53.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:53:53.487 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:23.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:23.630 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:53.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:53.777 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:23.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:23.988 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:54.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:54.195 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:24.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:24.415 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:54.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:54.588 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:24.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:24.794 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:55.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:55.009 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:58:25.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:58:25.240 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:58:55.464 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:58:55.470 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:25.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:25.658 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:55.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:55.863 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:26.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:26.081 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:56.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:56.297 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:26.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:26.467 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:56.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:56.689 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:26.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:26.915 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:57.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:57.060 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:27.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:27.205 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:57.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:57.352 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:04:27.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:04:27.554 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:04:57.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:04:57.944 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:28.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:28.161 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:58.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:58.389 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:28.631 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:28.637 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:58.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:58.783 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:28.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:28.929 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:59.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:59.188 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:29.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:29.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:59.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:59.560 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:29.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:29.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:59.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:59.921 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:10:30.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:10:30.067 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:00.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:00.296 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:30.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:30.619 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:00.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:00.764 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:30.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:30.974 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:01.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:01.117 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:31.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:31.330 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:14:01.462 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:14:01.468 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:14:31.607 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:14:31.614 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:01.839 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:01.845 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:31.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:31.984 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:16:02.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:16:02.196 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:16:32.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:16:32.410 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:02.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:02.565 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:32.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:32.779 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:02.911 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:02.917 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:33.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:33.081 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:03.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:03.228 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:33.433 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:33.440 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:03.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:03.629 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:33.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:33.844 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:21:04.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:21:04.018 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:21:34.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:21:34.261 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:04.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:04.410 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:34.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:34.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:04.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:04.854 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:34.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:34.999 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:24:05.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:24:05.307 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:24:35.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:24:35.583 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:05.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:05.794 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:36.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:36.008 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:06.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:06.206 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:36.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:36.353 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:06.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:06.492 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:36.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:36.697 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:06.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:06.903 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:37.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:37.043 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:29:07.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:29:07.257 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:29:37.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:29:37.487 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:07.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:07.807 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:38.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:38.016 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:31:08.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:31:08.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:31:38.351 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:31:38.357 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:32:08.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:32:08.577 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:32:38.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:32:38.808 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:09.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:09.112 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:39.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:39.267 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:09.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:09.605 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:39.806 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:39.812 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:35:10.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:35:10.035 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:35:40.173 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:35:40.179 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:10.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:10.426 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:40.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:40.640 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:10.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:10.835 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:41.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:41.049 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:11.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:11.212 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:41.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:41.360 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:11.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:11.583 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:41.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:42.003 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:12.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:12.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:42.369 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:42.376 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:12.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:12.523 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:42.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:42.731 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:42:12.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:42:12.959 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:42:43.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:42:43.165 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:13.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:13.303 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:43.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:43.634 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:13.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:13.841 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:44.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:44.060 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:45:14.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:45:14.433 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:45:44.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:45:44.579 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:14.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:14.781 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:45.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:45.028 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:15.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:15.241 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:45.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:45.398 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:48:15.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:48:15.595 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:48:45.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:48:45.819 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:15.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:15.974 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:46.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:46.188 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:16.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:16.326 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:46.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:46.567 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:51:16.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:51:16.712 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:51:46.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:51:46.922 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:52:17.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:52:17.116 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:52:47.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:52:47.367 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:17.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:17.589 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:47.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:47.735 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:17.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:17.958 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:48.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:48.348 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:55:18.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:55:18.494 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:55:48.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:55:48.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:18.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:18.906 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:49.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:49.110 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:19.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:19.308 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:49.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:49.564 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:58:19.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:58:19.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:58:49.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:58:49.887 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:20.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:20.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:50.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:50.256 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:20.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:20.520 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:50.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:50.657 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:20.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:20.889 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:51.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:51.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:02:21.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:02:21.275 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:02:51.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:02:51.422 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:21.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:21.621 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:51.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:51.767 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:21.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:21.986 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:52.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:52.163 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:22.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:22.390 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:52.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:52.608 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:22.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:22.757 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:52.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:52.953 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:07:23.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:07:23.099 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:07:53.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:07:53.315 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:23.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:23.520 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:53.661 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:53.667 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:23.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:23.806 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:54.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:54.187 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:24.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:24.385 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:54.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:54.565 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:11:24.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:11:24.709 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:11:54.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:11:54.851 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:25.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:25.062 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:55.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:55.209 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:25.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:25.424 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:55.647 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:55.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:25.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:25.800 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:56.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:56.189 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:26.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:26.328 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:56.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:56.523 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:26.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:26.721 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:56.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:56.861 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:27.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:27.081 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:57.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:57.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:27.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:27.499 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:57.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:57.713 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:19:27.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:19:27.872 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:19:58.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:19:58.070 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:28.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:28.300 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:58.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:58.541 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:28.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:28.746 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:59.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:59.051 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:22:29.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:22:29.207 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:22:59.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:22:59.420 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:29.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:29.584 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:59.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:59.789 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:24:29.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:24:29.936 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:25:00.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:25:00.152 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:25:30.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:25:30.401 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:00.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:00.627 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:30.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:30.790 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:27:00.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:27:00.971 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:27:31.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:27:31.334 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:01.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:01.473 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:31.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:31.686 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:02.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:02.117 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:32.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:32.263 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:02.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:02.484 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:32.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:32.642 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:31:02.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:31:02.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:31:32.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:31:32.937 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:03.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:03.080 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:33.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:33.276 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:33:03.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:33:03.516 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:33:33.662 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:33:33.668 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:03.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:03.825 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:33.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:33.964 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:35:04.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:35:04.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:35:34.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:35:34.349 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:04.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:04.496 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:34.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:34.727 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:04.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:04.868 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:35.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:35.087 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:05.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:05.326 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:35.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:35.472 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:39:05.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:39:05.711 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:39:35.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:39:35.982 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:06.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:06.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:36.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:36.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:06.416 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:06.423 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:36.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:36.645 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:07.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:07.035 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:37.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:37.192 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:07.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:07.420 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:37.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:37.626 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:07.759 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:07.765 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:37.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:37.975 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:08.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:08.244 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:38.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:38.404 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:46:08.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:46:08.616 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:46:38.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:46:38.912 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:09.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:09.060 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:39.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:39.205 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:09.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:09.346 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:39.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:39.568 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:09.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:09.815 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:39.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:39.970 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:50:10.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:50:10.209 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:50:40.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:50:40.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:10.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:10.535 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:40.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:40.750 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:10.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:10.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:41.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:41.163 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:11.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:11.394 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:41.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:41.545 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:54:11.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:54:11.776 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:54:41.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:54:41.919 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:12.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:12.070 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:42.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:42.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:12.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:12.431 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:42.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:42.627 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:12.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:12.773 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:42.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:43.004 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:13.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:13.217 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:43.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:43.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:59:13.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:59:13.586 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:59:43.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:59:43.818 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:13.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:13.971 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:44.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:44.210 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:14.446 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:14.453 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:44.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:44.643 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:14.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:14.876 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:45.009 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:45.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:03:15.232 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:03:15.238 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:03:45.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:03:45.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:15.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:15.557 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:45.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:45.770 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:15.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:15.968 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:46.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:46.291 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:16.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:16.521 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:46.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:46.664 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:07:16.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:07:16.886 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:07:47.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:07:47.122 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:17.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:17.269 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:47.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:47.444 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:17.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:17.606 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:47.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:47.837 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:10:18.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:10:18.184 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:10:48.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:10:48.337 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:11:18.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:11:18.554 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:11:48.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:11:48.705 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:18.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:18.969 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:49.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:49.114 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:19.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:19.311 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:49.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:49.493 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:14:19.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:14:19.757 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:14:49.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:14:49.904 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:20.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:20.143 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:50.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:50.288 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:20.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:20.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:50.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:50.674 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:20.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:20.828 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:51.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:51.023 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:21.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:21.255 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:51.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:51.510 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:19:21.666 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:19:21.673 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:19:51.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:19:51.819 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:20:21.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:20:21.999 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:20:52.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:20:52.246 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:22.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:22.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:52.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:52.702 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:22:22.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:22:22.870 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:22:53.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:22:53.084 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:23.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:23.302 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:53.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:53.512 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:23.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:23.655 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:53.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:53.813 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:25:23.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:25:23.952 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:25:54.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:25:54.147 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:24.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:24.319 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:54.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:54.525 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:27:24.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:27:24.665 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:27:54.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:27:54.899 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:25.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:25.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:55.351 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:55.358 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:25.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:25.513 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:55.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:55.702 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:30:25.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:30:25.953 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:30:56.103 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:30:56.109 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:26.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:26.263 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:56.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:56.419 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:26.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:26.565 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:56.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:56.720 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:26.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:26.868 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:57.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:57.033 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:34:27.164 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:34:27.170 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:34:57.586 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:34:57.592 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:27.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:27.731 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:57.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:57.952 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:28.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:28.090 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:58.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:58.308 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:28.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:28.484 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:58.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:58.677 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:28.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:28.818 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:59.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:59.048 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:29.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:29.227 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:59.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:59.450 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:29.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:29.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:59.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:59.852 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:41:30.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:41:30.083 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:00.216 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:00.222 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:30.446 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:30.453 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:00.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:00.597 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:30.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:30.790 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:44:01.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:44:01.079 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:44:31.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:44:31.228 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:01.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:01.381 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:31.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:31.595 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:46:01.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:46:01.759 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:46:31.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:46:31.899 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:02.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:02.113 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:32.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:32.260 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:02.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:02.407 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:32.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:32.697 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:02.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:02.833 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:33.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:33.042 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:03.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:03.478 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:33.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:33.649 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:03.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:03.863 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:34.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:34.057 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:52:04.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:52:04.223 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:52:34.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:52:34.563 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:04.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:04.701 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:34.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:34.915 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:05.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:05.319 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:35.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:35.449 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:05.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:05.673 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:35.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:35.827 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:06.009 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:06.016 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:36.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:36.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:06.405 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:06.412 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:36.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:36.556 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:58:06.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:58:06.696 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:58:36.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:58:36.833 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:07.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:07.047 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:37.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:37.202 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:00:07.410 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:00:07.416 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:00:37.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:00:37.568 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:07.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:07.785 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:37.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:37.988 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:08.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:08.145 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:38.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:38.359 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:03:08.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:03:08.607 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:03:38.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:03:38.760 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:08.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:08.959 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:39.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:39.097 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:09.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:09.285 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:39.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:39.506 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:06:09.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:06:09.696 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:06:39.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:06:39.854 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:07:10.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:07:10.047 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:07:40.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:07:40.202 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:10.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:10.425 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:40.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:40.635 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:10.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:10.784 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:40.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:40.973 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:10:11.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:10:11.124 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:10:41.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:10:41.334 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:11.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:11.532 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:41.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:41.671 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:11.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:11.833 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:42.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:42.048 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:12.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:12.254 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:42.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:42.465 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:12.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:12.604 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:42.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:42.826 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:15:12.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:15:12.981 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:15:43.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:15:43.131 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:13.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:13.341 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:43.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:43.498 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:13.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:13.811 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:43.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:44.001 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:14.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:14.147 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:44.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:44.373 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:14.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:14.716 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:44.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:44.863 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:15.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:15.068 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:45.300 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:45.307 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:15.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:15.453 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:45.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:45.692 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:15.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:15.905 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:46.103 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:46.110 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:16.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:16.316 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:46.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:46.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:16.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:16.685 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:46.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:46.879 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:17.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:17.156 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:47.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:47.298 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:26:17.431 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:26:17.437 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:26:47.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:26:47.783 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:17.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:17.923 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:48.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:48.127 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:18.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:18.367 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:48.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:48.555 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:29:18.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:29:18.702 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:29:48.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:29:48.908 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:19.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:19.061 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:49.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:49.254 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:19.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:19.602 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:49.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:49.825 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:19.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:19.989 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:50.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:50.213 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:20.344 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:20.350 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:50.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:50.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:34:20.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:34:20.695 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:34:50.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:34:50.840 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:21.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:21.032 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:51.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:51.193 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:36:21.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:36:21.332 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:36:51.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:36:51.571 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:21.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:21.796 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:52.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:52.043 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:22.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:22.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:52.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:52.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:39:22.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:39:22.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:39:52.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:39:52.699 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:22.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:22.848 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:53.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:53.071 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:41:23.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:41:23.209 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:41:53.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:41:53.389 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:23.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:23.747 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:53.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:53.978 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:24.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:24.157 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:54.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:54.418 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:44:24.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:44:24.574 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:44:54.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:44:54.794 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:25.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:25.209 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:55.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:55.354 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:25.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:25.569 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:55.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:55.714 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:25.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:25.961 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:56.171 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:56.177 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:48:26.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:48:26.329 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:48:56.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:48:56.539 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:49:26.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:49:26.689 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:49:56.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:49:56.891 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:27.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:27.107 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:57.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:57.287 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:27.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:27.494 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:57.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:57.656 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:27.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:27.961 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:58.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:58.107 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:53:28.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:53:28.241 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:53:58.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:53:58.550 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:28.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:28.696 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:58.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:58.904 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:29.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:29.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:59.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:59.399 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:56:29.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:56:29.542 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:56:59.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:56:59.774 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:57:29.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:57:29.919 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:58:00.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:58:00.149 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:58:30.306 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:58:30.312 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:00.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:00.459 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:30.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:30.761 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:00:00.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:00:00.966 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:00:31.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:00:31.183 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:01.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:01.361 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:31.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:31.518 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:01.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:01.683 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:31.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:31.862 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:03:02.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:03:02.074 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:03:32.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:03:32.229 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:02.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:02.451 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:32.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:32.597 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:02.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:02.752 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:32.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:32.891 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:03.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:03.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:33.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:33.609 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:03.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:03.749 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:33.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:33.903 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:04.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:04.133 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:34.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:34.508 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:09:04.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:09:04.706 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:09:34.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:09:34.895 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:05.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:05.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:35.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:35.370 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:05.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:05.590 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:35.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:35.779 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:12:05.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:12:05.960 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:12:36.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:12:36.172 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:06.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:06.377 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:36.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:36.599 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:14:06.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:14:06.745 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:14:36.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:14:36.975 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:07.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:07.126 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:37.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:37.273 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:07.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:07.483 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:37.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:37.661 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:07.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:07.891 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:38.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:38.246 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:08.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:08.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:38.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:38.689 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:19:08.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:19:08.828 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:19:39.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:19:39.060 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:09.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:09.200 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:39.433 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:39.439 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:09.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:09.639 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:39.779 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:39.786 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:10.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:10.125 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:40.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:40.263 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:10.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:10.488 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:40.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:40.722 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:24:10.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:24:10.961 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:24:41.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:24:41.115 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:11.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:11.262 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:41.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:41.401 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:26:11.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:26:11.599 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:26:41.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:26:41.820 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:11.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:11.958 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:42.175 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:42.181 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:12.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:12.384 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:42.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:42.597 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:29:12.782 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:29:12.788 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:29:42.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:29:42.928 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:13.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:13.114 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:43.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:43.319 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:13.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:13.557 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:43.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:43.784 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:32:13.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:32:13.948 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:32:44.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:32:44.187 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:14.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:14.336 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:44.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:44.534 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:34:14.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:34:14.721 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:34:44.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:34:44.869 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:15.009 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:15.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:45.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:45.220 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:36:15.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:36:15.368 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:36:45.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:36:45.526 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:15.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:15.764 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:45.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:45.974 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:16.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:16.112 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:46.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:46.292 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:39:16.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:39:16.447 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:39:46.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:39:46.720 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:16.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:16.941 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:47.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:47.126 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:41:17.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:41:17.533 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:41:47.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:41:47.755 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:42:17.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:42:17.979 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:42:48.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:42:48.142 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:18.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:18.347 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:48.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:48.495 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:44:18.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:44:18.717 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:44:48.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:44:48.856 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:19.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:19.100 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:49.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:49.394 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:46:19.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:46:19.616 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:46:49.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:46:49.822 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:20.021 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:20.028 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:50.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:50.217 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:20.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:20.373 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:50.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:50.586 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:20.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:20.806 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:51.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:51.012 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:21.219 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:21.225 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:51.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:51.380 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:21.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:21.518 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:51.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:51.741 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:21.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:21.937 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:52.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:52.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:53:22.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:53:22.297 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:53:52.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:53:52.462 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:22.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:22.684 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:53.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:53.082 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:23.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:23.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:53.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:53.569 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:56:23.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:56:23.776 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:56:53.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:56:54.005 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:57:24.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:57:24.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:57:54.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:57:54.400 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:24.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:24.546 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:54.765 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:54.771 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:24.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:24.994 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:55.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:55.217 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:25.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:25.425 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:55.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:55.632 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:01:25.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:01:25.852 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:01:56.067 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:01:56.074 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:26.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:26.288 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:56.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:56.433 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:26.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:26.580 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:56.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:56.793 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:27.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:27.024 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:57.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:57.230 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:27.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:27.450 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:57.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:57.606 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:06:27.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:06:27.836 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:06:58.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:06:58.058 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:28.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:28.271 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:58.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:58.610 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:28.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:28.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:59.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:59.095 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:29.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:29.314 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:59.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:59.589 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:10:29.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:10:29.747 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:10:59.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:10:59.992 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:11:30.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:11:30.211 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:00.405 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:00.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:30.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:30.633 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:00.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:00.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:31.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:31.011 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:01.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:01.240 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:31.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:31.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:01.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:01.525 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:31.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:31.671 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:16:01.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:16:01.809 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:16:32.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:16:32.064 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:02.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:02.268 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:32.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:32.498 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:02.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:02.644 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:32.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:32.858 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:19:03.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:19:03.080 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:19:33.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:19:33.310 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:03.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:03.515 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:33.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:33.732 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:03.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:03.883 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:34.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:34.097 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:04.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:04.407 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:34.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:34.624 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:04.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:04.801 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:34.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:34.942 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:05.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:05.091 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:35.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:35.236 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:05.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:05.384 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:35.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:35.540 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:05.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:05.704 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:35.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:35.861 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:27:05.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:27:05.998 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:27:36.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:27:36.246 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:06.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:06.390 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:36.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:36.600 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:06.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:06.797 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:36.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:36.938 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:07.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:07.152 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:37.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:37.382 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:31:07.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:31:07.579 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:31:37.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:31:37.726 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:08.009 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:08.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:38.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:38.162 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:08.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:08.368 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:38.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:38.599 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:08.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:08.813 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:38.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:38.955 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:35:09.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:35:09.110 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:35:39.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:35:39.306 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:09.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:09.519 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:39.869 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:39.875 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:37:10.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:37:10.028 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:37:40.219 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:37:40.226 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:10.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:10.452 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:40.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:40.657 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:10.876 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:10.882 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:41.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:41.077 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:40:11.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:40:11.232 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:40:41.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:40:41.463 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:11.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:11.753 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:41.927 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:41.933 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:42:12.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:42:12.090 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:42:42.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:42:42.221 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:12.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:12.433 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:42.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:42.646 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:44:12.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:44:12.884 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:44:43.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:44:43.056 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:13.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:13.270 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:43.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:43.466 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:13.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:13.712 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:43.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:43.858 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:13.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:14.005 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:44.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:44.152 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:14.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:14.350 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:44.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:44.570 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:14.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:14.767 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:44.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:44.973 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:50:15.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:50:15.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:50:45.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:50:45.366 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:15.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:15.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:45.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:45.780 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:15.990 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:15.996 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:46.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:46.237 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:16.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:16.383 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:46.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:46.529 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:54:16.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:54:16.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:54:46.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:54:46.972 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:17.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:17.174 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:47.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:47.317 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:17.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:17.486 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:47.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:47.716 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:17.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:17.939 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:48.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:48.154 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:18.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:18.293 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:48.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:48.513 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:18.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:18.729 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:48.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:48.966 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:19.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:19.213 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:49.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:49.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:19.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:19.564 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:49.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:49.804 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:20.009 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:20.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:50.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:50.162 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:20.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:20.394 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:50.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:50.542 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:20.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:20.770 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:50.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:51.004 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:21.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:21.201 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:51.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:51.431 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:21.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:21.639 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:51.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:51.878 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:07:22.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:07:22.108 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:07:52.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:07:52.341 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:22.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:22.493 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:52.687 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:52.694 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:23.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:23.021 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:53.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:53.224 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:23.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:23.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:53.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:53.561 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:23.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:23.715 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:53.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:53.937 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:12:24.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:12:24.142 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:12:54.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:12:54.285 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:24.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:24.451 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:54.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:54.632 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:24.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:24.804 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:55.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:55.009 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:15:25.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:15:25.229 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:15:55.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:15:55.418 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:25.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:25.765 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:55.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:55.919 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:17:26.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:17:26.149 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:17:56.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:17:56.322 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:26.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:26.625 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:56.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:56.785 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:19:27.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:19:27.013 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:19:57.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:19:57.242 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:27.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:27.456 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:57.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:57.603 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:21:27.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:21:27.749 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:21:57.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:21:57.905 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:28.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:28.132 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:58.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:58.477 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:28.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:28.692 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:58.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:58.835 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:24:28.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:24:28.970 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:24:59.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:24:59.177 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:29.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:29.392 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:59.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:59.605 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:26:29.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:26:29.759 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:26:59.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:26:59.965 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:27:30.171 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:27:30.177 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:28:00.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:28:00.400 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:28:30.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:28:30.588 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:00.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:00.784 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:30.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:30.948 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:01.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:01.117 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:31.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:31.332 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:01.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:01.496 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:31.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:31.692 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:32:01.900 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:32:01.906 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:32:32.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:32:32.078 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:02.289 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:02.295 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:32.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:32.456 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:02.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:02.688 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:32.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:32.909 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:35:03.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:35:03.127 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:35:33.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:35:33.352 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:03.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:03.563 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:33.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:33.779 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:03.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:03.985 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:34.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:34.198 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:04.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:04.429 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:34.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:34.676 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:04.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:04.905 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:35.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:35.043 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:05.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:05.185 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:35.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:35.504 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:41:05.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:41:05.658 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:41:35.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:41:35.887 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:06.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:06.068 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:36.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:36.273 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:06.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:06.414 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:36.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:36.644 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:44:06.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:44:06.874 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:44:37.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:44:37.129 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:07.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:07.300 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:37.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:37.513 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:07.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:07.715 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:37.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:37.955 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:08.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:08.137 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:38.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:38.577 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:48:08.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:48:08.724 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:48:38.939 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:48:38.946 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:09.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:09.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:39.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:39.389 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:09.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:09.610 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:39.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:39.811 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:51:09.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:51:10.000 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:51:40.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:51:40.366 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:10.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:10.524 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:40.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:40.680 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:10.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:10.892 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:41.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:41.040 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:54:11.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:54:11.271 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:54:41.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:54:41.585 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:11.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:11.808 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:42.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:42.014 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:12.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:12.228 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:42.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:42.379 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:12.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:12.517 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:42.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:42.731 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:12.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:12.961 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:43.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:43.169 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:13.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:13.492 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:43.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:43.713 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:00:13.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:00:13.928 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:00:44.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:00:44.159 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:01:14.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:01:14.382 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:01:44.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:01:44.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:14.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:14.752 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:44.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:44.909 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:03:15.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:03:15.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:03:45.381 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:03:45.387 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:15.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:15.535 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:45.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:45.680 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:05:15.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:05:15.897 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:05:46.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:05:46.102 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:16.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:16.249 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:46.416 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:46.422 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:16.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:16.622 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:46.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:46.778 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:17.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:17.032 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:47.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:47.370 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:17.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:17.509 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:47.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:47.741 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:10:17.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:10:17.977 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:10:48.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:10:48.191 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:18.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:18.384 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:48.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:48.544 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:18.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:18.780 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:48.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:48.923 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:19.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:19.136 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:49.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:49.274 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:14:19.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:14:19.496 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:14:49.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:14:49.636 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:15:19.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:15:19.866 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:15:50.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:15:50.077 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:20.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:20.236 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:50.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:50.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:20.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:20.619 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:50.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:50.849 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:18:20.997 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:18:21.003 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:18:51.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:18:51.218 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:21.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:21.357 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:51.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:51.586 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:20:21.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:20:21.724 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:20:51.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:20:51.904 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:22.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:22.138 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:52.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:52.314 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:22:22.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:22:22.519 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:22:52.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:22:52.665 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:22.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:22.879 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:53.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:53.024 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:23.232 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:23.238 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:53.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:53.440 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:23.642 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:23.648 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:53.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:53.836 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:23.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:24.001 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:54.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:54.221 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:27:24.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:27:24.621 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:27:54.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:27:54.818 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:28:25.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:28:25.040 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:28:55.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:28:55.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:25.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:25.477 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:55.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:55.717 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:25.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:25.939 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:56.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:56.078 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:31:26.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:31:26.309 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:31:56.448 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:31:56.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:26.662 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:26.669 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:56.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:56.882 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:33:27.081 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:33:27.088 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:33:57.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:33:57.240 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:27.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:27.462 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:57.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:57.684 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:27.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:27.831 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:57.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:57.972 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:36:28.169 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:36:28.175 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:36:58.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:36:58.389 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:28.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:28.607 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:58.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:58.832 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:38:28.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:38:28.976 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:38:59.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:38:59.183 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:29.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:29.422 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:59.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:59.577 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:40:29.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:40:29.781 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:41:00.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:41:00.298 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:41:30.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:41:30.437 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:00.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:00.589 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:30.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:30.819 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:43:01.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:43:01.047 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:43:31.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:43:31.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:01.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:01.409 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:31.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:31.556 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:45:01.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:45:01.702 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:45:31.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:45:31.901 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:02.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:02.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:32.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:32.235 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:02.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:02.377 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:32.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:32.601 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:02.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:02.878 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:33.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:33.122 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:03.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:03.336 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:33.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:33.483 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:50:03.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:50:03.678 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:50:33.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:50:33.903 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:04.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:04.138 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:34.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:34.360 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:04.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:04.590 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:34.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:34.844 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:53:04.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:53:05.000 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:53:35.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:53:35.148 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:05.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:05.367 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:35.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:35.505 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:05.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:05.724 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:35.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:35.939 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:56:06.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:56:06.163 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:56:36.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:56:36.302 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:57:06.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:57:06.448 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:57:36.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:57:36.594 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:06.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:06.802 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:37.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:37.012 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:07.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:07.217 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:37.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:37.363 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:00:07.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:00:07.590 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:00:37.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:00:37.806 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:07.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:07.971 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:38.186 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:38.192 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:02:08.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:02:08.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:02:38.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:02:38.580 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:08.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:08.732 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:38.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:38.939 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:09.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:09.169 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:39.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:39.404 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:09.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:09.633 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:39.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:39.765 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:09.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:09.903 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:40.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:40.125 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:07:10.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:07:10.356 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:07:40.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:07:40.501 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:08:10.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:08:10.698 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:08:40.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:08:40.920 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:11.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:11.065 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:41.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:41.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:10:11.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:10:11.410 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:10:41.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:10:41.640 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:11:11.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:11:11.797 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:11:41.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:11:41.943 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:12:12.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:12:12.164 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:12:42.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:12:42.388 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:12.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:12.532 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:42.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:42.742 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:12.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:12.891 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:43.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:43.103 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:15:13.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:15:13.326 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:15:43.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:15:43.531 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:13.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:13.678 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:43.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:43.870 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:14.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:14.185 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:44.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:44.415 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:18:14.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:18:14.615 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:18:44.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:18:44.817 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:14.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:14.968 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:45.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:45.188 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:20:15.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:20:15.410 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:20:45.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:20:45.641 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:21:15.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:21:15.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:21:45.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:21:45.969 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:16.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:16.115 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:46.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:46.356 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:23:16.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:23:16.561 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:23:46.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:23:46.700 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:16.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:16.910 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:47.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:47.149 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:25:17.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:25:17.379 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:25:47.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:25:47.601 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:26:17.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:26:17.747 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:26:47.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:26:47.978 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:27:18.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:27:18.117 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:27:48.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:27:48.426 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:28:18.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:28:18.637 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:28:48.782 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:28:48.789 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:29:18.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:29:18.997 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:29:49.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:29:49.147 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:30:19.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:30:19.354 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:30:49.561 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:30:49.568 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:31:20.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:31:20.008 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:31:50.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:31:50.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:32:20.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:32:20.369 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:32:50.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:32:50.526 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:33:20.764 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:33:20.770 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:33:50.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:33:50.904 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:34:21.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:34:21.136 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:34:51.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:34:51.335 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:35:21.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:35:21.491 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:35:51.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:35:51.718 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:36:21.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:36:21.917 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:36:52.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:36:52.149 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:37:22.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:37:22.293 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:37:52.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:37:52.490 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:38:22.730 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:38:22.736 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:38:52.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:38:52.941 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:39:23.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:39:23.172 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:39:53.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:39:53.318 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:40:23.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:40:23.457 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:40:53.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:40:53.604 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:41:23.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:41:23.759 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:41:53.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:41:53.904 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:42:24.103 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:42:24.110 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:42:54.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:42:54.298 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:43:24.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:43:24.521 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:43:54.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:43:54.719 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:44:24.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:44:24.938 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:44:55.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:44:55.185 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:45:25.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:45:25.374 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:45:55.585 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:45:55.592 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:46:25.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:46:25.839 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:46:56.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:46:56.056 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:47:26.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:47:26.193 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:47:56.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:47:56.381 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:48:26.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:48:26.622 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:48:56.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:48:56.828 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:49:27.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:49:27.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:49:57.186 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:49:57.192 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:50:27.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:50:27.420 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:50:57.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:50:57.570 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:51:27.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:51:27.774 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:51:57.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:51:57.979 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:52:28.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:52:28.167 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:52:58.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:52:58.391 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:53:28.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:53:28.537 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:53:58.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:53:58.760 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:54:28.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:54:28.989 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:54:59.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:54:59.205 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:55:29.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:55:29.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:55:59.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:55:59.587 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:56:29.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:56:29.805 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:57:00.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:57:00.030 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:57:30.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:57:30.241 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:58:00.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:58:00.463 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:58:30.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:58:30.611 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:59:00.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:59:00.809 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:59:31.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:59:31.022 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:00:01.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:00:01.260 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:00:31.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:00:31.400 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:01:01.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:01:01.545 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:01:31.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:01:31.683 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:02:01.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:02:01.905 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:02:32.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:02:32.126 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:03:02.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:03:02.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:03:32.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:03:32.495 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:04:02.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:04:02.641 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:04:32.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:04:32.889 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:05:03.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:05:03.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:05:33.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:05:33.358 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:06:03.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:06:03.496 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:06:33.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:06:33.711 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:07:03.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:07:03.925 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:07:34.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:07:34.156 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:08:04.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:08:04.371 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:08:34.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:08:34.568 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:09:04.745 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:09:04.752 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:09:34.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:09:34.987 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:10:05.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:10:05.184 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:10:35.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:10:35.382 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:11:05.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:11:05.521 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:11:35.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:11:35.718 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:12:05.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:12:05.864 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:12:36.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:12:36.077 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:13:06.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:13:06.291 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:13:36.431 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:13:36.437 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:14:06.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:14:06.667 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:14:36.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:14:36.814 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:15:06.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:15:07.002 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:15:37.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:15:37.249 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:16:07.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:16:07.462 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:16:37.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:16:37.609 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:17:07.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:17:07.765 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:17:37.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:17:37.903 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:18:08.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:18:08.108 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:18:38.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:18:38.323 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:19:08.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:19:08.543 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:19:38.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:19:38.696 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:20:08.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:20:08.886 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:20:39.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:20:39.225 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:21:09.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:21:09.363 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:21:39.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:21:39.577 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:22:09.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:22:09.774 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:22:39.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:22:39.929 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:23:10.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:23:10.146 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:23:40.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:23:40.380 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:24:10.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:24:10.806 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:24:40.939 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:24:40.945 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:25:11.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:25:11.172 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:25:41.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:25:41.382 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:26:11.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:26:11.595 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:26:41.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:26:41.743 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:27:11.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:27:11.958 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:27:42.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:27:42.156 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:28:12.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:28:12.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:28:42.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:28:42.600 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:29:12.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:29:12.753 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:29:42.927 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:29:42.933 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:30:13.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:30:13.076 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:30:43.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:30:43.294 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:31:13.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:31:13.450 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:31:43.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:31:43.651 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:32:13.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:32:13.841 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:32:43.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:32:43.989 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:33:14.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:33:14.127 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:33:44.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:33:44.331 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:34:14.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:34:14.537 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:34:44.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:34:44.735 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:35:14.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:35:14.874 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:35:45.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:35:45.071 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:36:15.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:36:15.217 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:36:45.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:36:45.450 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:37:15.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:37:15.594 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:37:45.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:37:45.736 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:38:15.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:38:15.879 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:38:46.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:38:46.034 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:39:16.168 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:39:16.175 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:39:46.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:39:46.319 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:40:16.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:40:16.466 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:40:46.607 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:40:46.614 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:41:16.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:41:16.825 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:41:47.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:41:47.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:42:17.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:42:17.229 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:42:47.453 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:42:47.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:43:17.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:43:17.681 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:43:47.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:43:47.920 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:44:18.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:44:18.068 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:44:48.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:44:48.279 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:45:18.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:45:18.483 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:45:48.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:45:48.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:46:18.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:46:18.944 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:46:49.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:46:49.125 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:47:19.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:47:19.297 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:47:49.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:47:49.504 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:48:19.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:48:19.731 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:48:49.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:48:49.945 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:49:20.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:49:20.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:49:50.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:49:50.341 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:50:20.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:50:20.546 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:50:50.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:50:50.777 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:51:21.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:51:21.058 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:51:51.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:51:51.204 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:52:21.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:52:21.353 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:52:51.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:52:51.574 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:53:21.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:53:21.796 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:53:51.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:53:51.985 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:54:22.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:54:22.209 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:54:52.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:54:52.380 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:55:22.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:55:22.595 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:55:52.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:55:52.804 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:56:23.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:56:23.010 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:56:53.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:56:53.221 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:57:23.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:57:23.535 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:57:53.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:57:53.742 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:58:23.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:58:23.963 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:58:54.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:58:54.176 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:59:24.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:59:24.320 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:59:54.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:59:54.489 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:00:24.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:00:24.686 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:00:54.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:00:54.910 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:01:25.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:01:25.193 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:01:55.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:01:55.341 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:02:25.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:02:25.502 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:02:55.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:02:55.724 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:03:25.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:03:25.922 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:03:56.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:03:56.136 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:04:26.369 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:04:26.375 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:04:56.521 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:04:56.528 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:05:26.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:05:26.752 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:05:56.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:05:56.976 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:06:27.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:06:27.181 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:06:57.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:06:57.390 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:07:27.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:07:27.529 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:07:57.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:07:57.766 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:08:27.990 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:08:27.996 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:08:58.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:08:58.210 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:09:28.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:09:28.418 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:09:58.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:09:58.587 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:10:28.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:10:28.810 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:10:58.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:10:59.000 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:11:29.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:11:29.197 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:11:59.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:11:59.414 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:12:29.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:12:29.575 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:12:59.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:12:59.805 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:13:30.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:13:30.010 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:14:00.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:14:00.233 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:14:30.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:14:30.431 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:15:00.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:15:00.619 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:15:30.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:15:30.850 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:16:01.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:16:01.065 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:16:31.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:16:31.270 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:17:01.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:17:01.476 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:17:31.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:17:31.619 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:18:01.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:18:01.844 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:18:32.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:18:32.050 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:19:02.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:19:02.249 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:19:32.455 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:19:32.462 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:20:02.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:20:02.685 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:20:32.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:20:32.890 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:21:03.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:21:03.121 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:21:33.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:21:33.285 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:22:03.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:22:03.482 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:22:33.622 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:22:33.629 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:23:03.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:23:03.775 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:23:33.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:23:33.982 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:24:04.173 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:24:04.179 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:24:34.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:24:34.395 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:25:04.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:25:04.600 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:25:34.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:25:34.750 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:26:04.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:26:04.970 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:26:35.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:26:35.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:27:05.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:27:05.350 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:27:35.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:27:35.563 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:28:05.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:28:05.760 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:28:35.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:28:35.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:29:06.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:29:06.198 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:29:36.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:29:36.395 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:30:06.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:30:06.603 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:30:36.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:30:36.823 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:31:07.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:31:07.023 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:31:37.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:31:37.218 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:32:07.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:32:07.440 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:32:37.631 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:32:37.639 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:33:07.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:33:07.852 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:33:38.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:33:38.058 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:34:08.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:34:08.291 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:34:38.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:34:38.478 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:35:08.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:35:08.620 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:35:38.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:35:38.786 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:36:08.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:36:08.929 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:36:24.664 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2023-03-03T19:36:24.665 DEBUG:teuthology.task.console_log:Killing console logger for smithi098 2023-03-03T19:36:24.666 DEBUG:teuthology.task.console_log:Killing console logger for smithi100 2023-03-03T19:36:24.667 DEBUG:teuthology.exit:Finished running handlers